2023-10-30T11:48:06.160 INFO:root:teuthology version: 0.0.1.dev202+gbef9b3e 2023-10-30T11:48:06.175 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_api_tests} --owner scheduled_vallariag@teuthology --verbose --archive /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989 --name vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi -- /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/orig.config.yaml 2023-10-30T11:48:06.223 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T11:48:06.303 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989 branch: wip-vallariag-centos9-only description: rados:monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_api_tests} email: val.agl002@gmail.com first_in_suite: false job_id: '7440989' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vallariag-centos9-only ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 003372b105ed2deff4dd01cf2ce736f137365e86 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 003372b105ed2deff4dd01cf2ce736f137365e86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-xml-scanner sha1: 397e85a5f8ee6b984077e94184404370497237a9 owner: scheduled_vallariag@teuthology priority: 80 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 4758 sha1: 003372b105ed2deff4dd01cf2ce736f137365e86 sleep_before_teardown: 0 suite: rados:monthrash suite_branch: wip-xml-scanner suite_path: /home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa suite_relpath: qa suite_repo: https://github.com/VallariAg/ceph.git suite_sha1: 397e85a5f8ee6b984077e94184404370497237a9 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/jATN70vGGBL54z9Cu8g6a1rsSO8iHlokG3RH+faOTkoEZuvVRihXNsRF42NBzTBt0Pz6tSWC90Tt7c2Ko0kA= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZKfXPJ1kTzkgnwWEh9xKGaopLAhExAfdfXrJKba4ufD5+QyJGrpSOAwDeMmT0mp2LxAeaX0erLyhdetBDjU1o= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: unittest-xml-scanner teuthology_sha1: bef9b3ee0aeb26a72d4f9af72f05102582ed658c timestamp: 2023-10-30_03:55:12 tube: smithi user: vallariag verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T11:48:06.303 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa; will attempt to use it 2023-10-30T11:48:06.395 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa/tasks 2023-10-30T11:48:06.396 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T11:48:06.398 INFO:teuthology.task.internal:Checking packages... 2023-10-30T11:48:06.637 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '003372b105ed2deff4dd01cf2ce736f137365e86' 2023-10-30T11:48:06.638 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T11:48:06.638 INFO:teuthology.packaging:ref: None 2023-10-30T11:48:06.638 INFO:teuthology.packaging:tag: None 2023-10-30T11:48:06.639 INFO:teuthology.packaging:branch: wip-vallariag-centos9-only 2023-10-30T11:48:06.639 INFO:teuthology.packaging:sha1: 003372b105ed2deff4dd01cf2ce736f137365e86 2023-10-30T11:48:06.639 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vallariag-centos9-only 2023-10-30T11:48:06.792 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6851.g003372b1 2023-10-30T11:48:06.793 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T11:48:06.799 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T11:48:06.800 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T11:48:06.805 INFO:teuthology.task.internal:Saving configuration 2023-10-30T11:48:06.819 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T11:48:06.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T11:48:06.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:42:17.500390', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/jATN70vGGBL54z9Cu8g6a1rsSO8iHlokG3RH+faOTkoEZuvVRihXNsRF42NBzTBt0Pz6tSWC90Tt7c2Ko0kA='} 2023-10-30T11:48:06.873 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:42:17.501905', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZKfXPJ1kTzkgnwWEh9xKGaopLAhExAfdfXrJKba4ufD5+QyJGrpSOAwDeMmT0mp2LxAeaX0erLyhdetBDjU1o='} 2023-10-30T11:48:06.873 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T11:48:06.879 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-10-30T11:48:06.880 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-10-30T11:48:06.880 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T11:48:06.982 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa4418c2b80>, signals=[15]) 2023-10-30T11:48:06.983 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T11:48:06.989 INFO:teuthology.task.internal:Opening connections... 2023-10-30T11:48:06.989 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T11:48:06.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:48:07.134 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T11:48:07.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:48:07.219 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T11:48:07.226 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2023-10-30T11:48:07.244 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2023-10-30T11:48:07.245 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2023-10-30T11:48:07.300 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2023-10-30T11:48:07.301 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2023-10-30T11:48:07.301 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2023-10-30T11:48:07.301 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2023-10-30T11:48:07.301 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2023-10-30T11:48:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2023-10-30T11:48:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T11:48:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2023-10-30T11:48:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2023-10-30T11:48:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T11:48:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2023-10-30T11:48:07.303 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T11:48:07.303 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T11:48:07.303 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T11:48:07.304 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-10-30T11:48:07.326 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2023-10-30T11:48:07.344 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2023-10-30T11:48:07.344 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2023-10-30T11:48:07.401 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2023-10-30T11:48:07.402 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2023-10-30T11:48:07.402 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2023-10-30T11:48:07.402 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2023-10-30T11:48:07.402 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2023-10-30T11:48:07.403 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2023-10-30T11:48:07.403 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-30T11:48:07.403 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2023-10-30T11:48:07.403 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2023-10-30T11:48:07.403 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-30T11:48:07.404 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2023-10-30T11:48:07.404 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-30T11:48:07.404 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-30T11:48:07.404 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-30T11:48:07.405 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2023-10-30T11:48:07.435 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T11:48:07.444 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T11:48:07.449 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T11:48:07.449 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2023-10-30T11:48:07.453 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2023-10-30T11:48:07.470 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T11:48:07.476 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T11:48:07.477 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2023-10-30T11:48:07.510 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2023-10-30T11:48:07.536 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T11:48:07.594 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T11:48:07.647 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-30T11:48:07.648 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-30T11:48:07.648 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T11:48:07.649 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T11:48:07.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:48:07.665 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-30T11:48:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:48:07.667 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-10-30T11:48:07.722 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-361.el9.x86_64 2023-10-30T11:48:07.722 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.14.0-361.el9.x86_64 2023-10-30T11:48:07.722 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2023-10-30T11:48:07.725 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-361.el9.x86_64 2023-10-30T11:48:07.725 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-361.el9.x86_64 2023-10-30T11:48:07.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2023-10-30T11:48:09.967 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-10-30T11:48:10.870 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 16 MB/s | 6.3 MB 00:00 2023-10-30T11:48:11.309 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2023-10-30T11:48:12.187 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-10-30T11:48:12.833 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 7.9 MB 00:01 2023-10-30T11:48:15.544 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 7.9 MB 00:03 2023-10-30T11:48:18.317 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 18 MB 00:04 2023-10-30T11:48:18.611 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 18 MB 00:02 2023-10-30T11:48:22.318 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 10 MB/s | 5.7 MB 00:00 2023-10-30T11:48:22.670 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 5.7 MB 00:00 2023-10-30T11:48:23.777 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2023-10-30T11:48:24.196 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-10-30T11:48:25.581 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 11 MB/s | 19 MB 00:01 2023-10-30T11:48:30.303 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2023-10-30T11:48:32.460 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T11:48:32.460 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T11:48:32.565 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:48:32.572 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:48:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-30T11:48:32.572 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:48:32.573 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:48:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T11:48:32.573 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:48:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T11:48:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T11:48:32.574 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T11:48:32.574 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:48:32.574 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:48:32.574 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:48:32.574 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2023-10-30T11:48:32.575 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:48:32.575 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 95 M 2023-10-30T11:48:32.576 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 120 M 2023-10-30T11:48:32.576 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:48:33.938 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 2.0 MB/s | 19 MB 00:09 2023-10-30T11:48:36.804 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 1.3 MB/s | 5.3 MB 00:04 2023-10-30T11:48:38.404 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2023-10-30T11:48:39.096 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 3.1 MB/s | 20 MB 00:06 2023-10-30T11:48:40.510 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-30T11:48:40.511 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-30T11:48:40.609 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:48:40.620 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:48:40.620 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2023-10-30T11:48:40.621 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:48:40.621 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:48:40.621 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-30T11:48:40.621 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:48:40.621 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-30T11:48:40.622 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-30T11:48:40.622 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-30T11:48:40.622 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:48:40.622 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:48:40.622 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:48:40.623 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2023-10-30T11:48:40.623 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:48:40.623 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 95 M 2023-10-30T11:48:40.624 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 120 M 2023-10-30T11:48:40.624 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:48:40.846 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 4.6 MB/s | 38 MB 00:08 2023-10-30T11:48:42.031 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 3.9 MB/s | 5.3 MB 00:01 2023-10-30T11:48:42.806 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 9.4 MB/s | 20 MB 00:02 2023-10-30T11:48:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 12 MB/s | 38 MB 00:03 2023-10-30T11:48:43.854 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 4.5 MB/s | 32 MB 00:07 2023-10-30T11:48:43.854 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:48:43.855 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.4 MB/s | 95 MB 00:11 2023-10-30T11:48:44.156 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 15 MB/s | 32 MB 00:02 2023-10-30T11:48:44.159 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:48:44.159 INFO:teuthology.orchestra.run.smithi118.stdout:Total 27 MB/s | 95 MB 00:03 2023-10-30T11:48:44.756 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:48:45.002 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:48:45.187 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:48:45.188 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:48:45.369 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:48:45.369 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:48:46.552 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:48:46.553 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:48:46.680 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:48:46.681 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:48:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:48:49.342 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:48:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:48:49.996 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:48:50.064 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:48:50.122 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:48:52.826 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:48:52.979 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:48:53.008 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:48:53.160 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:48:57.582 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:48:57.764 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:48:58.119 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:48:58.234 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:02.379 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:03.210 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:31.015 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:31.058 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:31.483 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:31.483 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:49:31.483 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:49:31.483 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:49:31.907 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:31.907 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:49:31.907 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:49:31.908 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T11:49:31.908 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T11:49:31.908 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T11:49:31.908 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T11:49:31.908 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:49:31.909 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:49:32.427 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:49:32.560 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:49:32.561 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:49:32.561 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T11:49:32.561 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T11:49:32.562 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T11:49:32.562 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-30T11:49:32.562 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T11:49:32.563 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:49:32.680 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:49:32.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:49:32.681 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2023-10-30T11:49:32.727 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2023-10-30T11:49:32.727 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2023-10-30T11:49:32.728 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2023-10-30T11:49:32.728 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 68.el9 2023-10-30T11:49:32.728 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2023-10-30T11:49:32.728 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T11:49:32.728 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2023-10-30T11:49:32.729 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8264254 2023-10-30T11:49:32.729 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2023-10-30T11:49:32.729 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T11:49:32.729 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T11:49:32.729 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T11:49:32.730 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T11:49:32.730 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2023-10-30T11:49:32.730 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2023-10-30T11:49:32.730 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T11:49:32.730 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2023-10-30T11:49:32.731 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2023-10-30T11:49:32.731 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:49:32.731 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T11:49:32.731 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T11:49:32.731 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T11:49:32.732 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2023-10-30T11:49:32.732 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:49:32.732 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T11:49:32.733 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T11:49:32.733 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 5.14.0-378.el9.x86_64 2023-10-30T11:49:32.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T11:49:33.274 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2023-10-30T11:49:33.767 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:33.811 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:34.239 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:34.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-30T11:49:34.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-30T11:49:34.239 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-30T11:49:34.661 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T11:49:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:done 2023-10-30T11:49:34.696 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2023-10-30T11:49:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-30T11:49:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:49:34.698 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:49:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-30T11:49:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-30T11:49:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-30T11:49:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-30T11:49:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:49:34.700 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:49:34.714 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.F9lPVQ47fm 2023-10-30T11:49:34.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.F9lPVQ47fm 2023-10-30T11:49:34.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.F9lPVQ47fm 2023-10-30T11:49:34.924 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.F9lPVQ47fm is 6KB 2023-10-30T11:49:34.972 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.F9lPVQ47fm 2023-10-30T11:49:34.986 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T11:49:35.051 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T11:49:35.051 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T11:49:35.051 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T11:49:35.051 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T11:49:35.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T11:49:35.195 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2023-10-30T11:49:35.214 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:49:35.367 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:49:35.368 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:49:35.368 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-30T11:49:35.368 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-30T11:49:35.368 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-30T11:49:35.368 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-30T11:49:35.369 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2023-10-30T11:49:35.369 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-10-30T11:49:35.490 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-30T11:49:35.491 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-30T11:49:35.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2023-10-30T11:49:35.539 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2023-10-30T11:49:35.540 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2023-10-30T11:49:35.540 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.06 2023-10-30T11:49:35.540 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 68.el9 2023-10-30T11:49:35.540 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2023-10-30T11:49:35.541 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-30T11:49:35.541 INFO:teuthology.orchestra.run.smithi050.stdout:Group : Unspecified 2023-10-30T11:49:35.541 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 8264254 2023-10-30T11:49:35.541 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2023-10-30T11:49:35.541 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-30T11:49:35.542 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-30T11:49:35.542 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-30T11:49:35.542 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-30T11:49:35.542 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : builder@centos.org 2023-10-30T11:49:35.543 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2023-10-30T11:49:35.543 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-30T11:49:35.543 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2023-10-30T11:49:35.543 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2023-10-30T11:49:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:49:35.544 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-30T11:49:35.544 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-30T11:49:35.544 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-30T11:49:35.544 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2023-10-30T11:49:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:49:35.545 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2023-10-30T11:49:35.546 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-30T11:49:35.546 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 5.14.0-378.el9.x86_64 2023-10-30T11:49:35.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-30T11:49:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2023-10-30T11:49:37.410 INFO:teuthology.orchestra.run.smithi050.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-30T11:49:37.454 INFO:teuthology.orchestra.run.smithi050.stderr:done 2023-10-30T11:49:37.456 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-10-30T11:49:37.472 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.rutILL56rx 2023-10-30T11:49:37.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rutILL56rx 2023-10-30T11:49:37.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.rutILL56rx 2023-10-30T11:49:37.684 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.rutILL56rx is 6KB 2023-10-30T11:49:37.704 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.rutILL56rx 2023-10-30T11:49:37.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2023-10-30T11:49:37.788 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-30T11:49:37.788 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-30T11:49:37.788 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-30T11:49:37.789 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-30T11:49:37.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-30T11:49:37.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2023-10-30T11:50:05.222 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-30T11:50:05.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-30T11:50:05.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:50:07.956 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-30T11:50:07.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-30T11:50:07.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:50:39.738 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-30T11:50:48.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-30T11:50:48.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:51:07.978 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T11:51:16.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-30T11:51:16.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:51:17.405 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-10-30T11:51:17.769 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-30T11:51:17.769 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-30T11:51:17.769 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-10-30T11:51:17.783 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-378.el9.x86_64 2023-10-30T11:51:17.784 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T11:51:17.784 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T11:51:17.784 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T11:51:18.785 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-30T11:51:18.785 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T11:51:18.854 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2023-10-30T11:51:18.890 DEBUG:teuthology.parallel:result is None 2023-10-30T11:51:48.750 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T11:52:00.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-30T11:52:00.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:52:01.199 DEBUG:teuthology.orchestra.run.smithi118:> true 2023-10-30T11:52:01.562 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-30T11:52:01.563 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-378.el9.x86_64"... 2023-10-30T11:52:01.563 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-30T11:52:01.577 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-378.el9.x86_64 2023-10-30T11:52:01.578 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-30T11:52:01.578 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T11:52:01.578 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-30T11:52:02.579 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-30T11:52:02.580 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-30T11:52:02.662 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2023-10-30T11:52:02.692 DEBUG:teuthology.parallel:result is None 2023-10-30T11:52:02.693 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T11:52:02.700 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T11:52:02.701 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T11:52:02.704 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T11:52:02.721 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T11:52:02.733 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T11:52:02.742 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T11:52:02.743 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T11:52:02.763 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T11:52:02.794 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T11:52:02.801 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T11:52:02.801 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T11:52:02.829 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T11:52:02.867 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:02.867 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:02.881 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:02.884 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T11:52:02.886 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T11:52:02.894 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T11:52:02.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T11:52:02.929 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T11:52:02.960 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T11:52:02.983 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T11:52:02.984 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T11:52:03.003 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T11:52:03.019 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:03.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:03.151 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:03.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:03.282 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T11:52:03.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T11:52:03.353 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:03.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T11:52:03.448 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:03.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T11:52:03.577 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T11:52:03.578 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T11:52:03.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2023-10-30T11:52:03.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2023-10-30T11:52:03.704 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T11:52:03.728 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T11:52:04.133 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T11:52:04.141 INFO:teuthology.task.internal:Starting timer... 2023-10-30T11:52:04.141 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T11:52:04.191 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T11:52:04.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2023-10-30T11:52:04.295 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2023-10-30T11:52:04.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2023-10-30T11:52:04.341 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2023-10-30T11:52:04.343 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T11:52:04.343 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2023-10-30T11:52:04.366 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2023-10-30T11:52:04.367 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2023-10-30T11:52:04.404 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2023-10-30T11:52:04.405 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2023-10-30T11:52:04.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T11:52:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:52:04.440 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T11:52:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T11:52:04.472 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T11:52:04.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2023-10-30T11:52:04.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2023-10-30T11:52:04.548 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T11:52:04.558 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-30T11:52:04.563 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-30T11:52:04.717 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T11:52:04.732 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T11:52:04.734 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T11:57:42.204 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2023-10-30T11:57:42.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-30T11:57:42.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:57:42.306 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-10-30T11:57:42.377 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-30T11:57:42.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-30T11:57:42.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T11:57:42.461 DEBUG:teuthology.orchestra.run.smithi118:> true 2023-10-30T11:57:42.528 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-30T11:57:42.529 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T11:57:42.548 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T11:57:42.549 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T11:57:42.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T11:57:42.552 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T11:57:42.553 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T11:57:42.586 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T11:57:42.599 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-30T11:57:42.606 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T11:57:42.616 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-30T11:57:42.643 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2023-10-30T11:57:42.644 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2023-10-30T11:57:42.658 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2023-10-30T11:57:42.660 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2023-10-30T11:57:42.675 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T11:57:42.680 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-30T11:57:42.690 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T11:57:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-30T11:57:42.734 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2023-10-30T11:57:42.737 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T11:57:42.738 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2023-10-30T11:57:42.738 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.738 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.738 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.738 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2023-10-30T11:57:42.751 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T11:57:42.751 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-10-30T11:57:42.752 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.752 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.752 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.752 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T11:57:42.753 INFO:teuthology.run_tasks:Running task install... 2023-10-30T11:57:42.762 DEBUG:teuthology.task.install:project ceph 2023-10-30T11:57:42.763 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '003372b105ed2deff4dd01cf2ce736f137365e86'}} 2023-10-30T11:57:42.763 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '003372b105ed2deff4dd01cf2ce736f137365e86'} 2023-10-30T11:57:42.763 INFO:teuthology.task.install:Using flavor: default 2023-10-30T11:57:42.836 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-30T11:57:42.836 INFO:teuthology.task.install:extra packages: [] 2023-10-30T11:57:42.837 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '003372b105ed2deff4dd01cf2ce736f137365e86', 'tag': None, 'wait_for_package': False} 2023-10-30T11:57:42.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=003372b105ed2deff4dd01cf2ce736f137365e86 2023-10-30T11:57:42.839 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '003372b105ed2deff4dd01cf2ce736f137365e86', 'tag': None, 'wait_for_package': False} 2023-10-30T11:57:42.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=003372b105ed2deff4dd01cf2ce736f137365e86 2023-10-30T11:57:42.986 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/ 2023-10-30T11:57:42.986 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6851.g003372b1 2023-10-30T11:57:43.051 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/ 2023-10-30T11:57:43.052 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6851.g003372b1 2023-10-30T11:57:43.100 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T11:57:43.100 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T11:57:43.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T11:57:43.129 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-30T11:57:43.129 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/003372b105ed2deff4dd01cf2ce736f137365e86/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T11:57:43.197 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T11:57:43.222 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vallariag-centos9-only/003372b105ed2deff4dd01cf2ce736f137365e86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-30T11:57:43.223 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T11:57:43.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-30T11:57:43.256 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-30T11:57:43.256 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/003372b105ed2deff4dd01cf2ce736f137365e86/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-30T11:57:43.277 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T11:57:43.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-30T11:57:43.342 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2023-10-30T11:57:43.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2023-10-30T11:57:43.415 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-30T11:57:43.445 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2023-10-30T11:57:43.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2023-10-30T11:57:43.574 INFO:teuthology.orchestra.run.smithi118.stdout:72 files removed 2023-10-30T11:57:43.600 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2023-10-30T11:57:43.704 INFO:teuthology.orchestra.run.smithi050.stdout:72 files removed 2023-10-30T11:57:43.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2023-10-30T11:57:44.338 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 185 kB/s | 78 kB 00:00 2023-10-30T11:57:44.424 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 218 kB/s | 78 kB 00:00 2023-10-30T11:57:44.747 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-10-30T11:57:44.784 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 31 kB/s | 9.4 kB 00:00 2023-10-30T11:57:45.131 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2023-10-30T11:57:45.159 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2023-10-30T11:57:45.492 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 19 MB/s | 6.3 MB 00:00 2023-10-30T11:57:45.528 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - AppStream 19 MB/s | 6.3 MB 00:00 2023-10-30T11:57:46.850 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2023-10-30T11:57:46.878 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2023-10-30T11:57:47.811 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 7.9 MB 00:00 2023-10-30T11:57:49.657 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 7.9 MB 00:02 2023-10-30T11:57:53.962 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 18 MB 00:05 2023-10-30T11:57:54.717 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 18 MB 00:04 2023-10-30T11:57:58.840 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.7 MB 00:01 2023-10-30T11:57:59.443 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.7 MB 00:01 2023-10-30T11:58:00.147 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 15 kB 00:00 2023-10-30T11:58:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 15 kB 00:00 2023-10-30T11:58:02.962 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 19 MB 00:02 2023-10-30T11:58:07.678 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2023-10-30T11:58:10.064 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:58:10.080 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:58:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:58:10.080 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:58:10.081 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:58:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:18.0.0-6851.g003372b1.el9 ceph 8.6 M 2023-10-30T11:58:10.081 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2023-10-30T11:58:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 3.3 M 2023-10-30T11:58:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 3.0 M 2023-10-30T11:58:10.082 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:58:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options 2023-10-30T11:58:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T11:58:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:18.0.0-6851.g003372b1.el9 ceph 5.2 M 2023-10-30T11:58:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:18.0.0-6851.g003372b1.el9 ceph 19 M 2023-10-30T11:58:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:18.0.0-6851.g003372b1.el9 ceph 24 k 2023-10-30T11:58:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T11:58:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T11:58:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T11:58:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T11:58:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 716 k 2023-10-30T11:58:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T11:58:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T11:58:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T11:58:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 473 k 2023-10-30T11:58:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T11:58:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 4.8 M 2023-10-30T11:58:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T11:58:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T11:58:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T11:58:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse 2023-10-30T11:58:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 2:18.0.0-6851.g003372b1.el9 ceph 45 k 2023-10-30T11:58:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:18.0.0-6851.g003372b1.el9 ceph 127 k 2023-10-30T11:58:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:18.0.0-6851.g003372b1.el9 ceph 161 k 2023-10-30T11:58:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T11:58:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:18.0.0-6851.g003372b1.el9 ceph 320 k 2023-10-30T11:58:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:18.0.0-6851.g003372b1.el9 ceph 298 k 2023-10-30T11:58:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:18.0.0-6851.g003372b1.el9 ceph 99 k 2023-10-30T11:58:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T11:58:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T11:58:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:10.088 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:58:10.088 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:58:10.088 INFO:teuthology.orchestra.run.smithi118.stdout:Install 28 Packages 2023-10-30T11:58:10.088 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2023-10-30T11:58:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:10.089 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 55 M 2023-10-30T11:58:10.089 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:58:12.571 INFO:teuthology.orchestra.run.smithi118.stdout:(1/30): ceph-base-18.0.0-6851.g003372b1.el9.x86 2.4 MB/s | 5.2 MB 00:02 2023-10-30T11:58:12.663 INFO:teuthology.orchestra.run.smithi118.stdout:(2/30): ceph-selinux-18.0.0-6851.g003372b1.el9. 263 kB/s | 24 kB 00:00 2023-10-30T11:58:13.130 INFO:teuthology.orchestra.run.smithi118.stdout:(3/30): libcephfs2-18.0.0-6851.g003372b1.el9.x8 1.5 MB/s | 716 kB 00:00 2023-10-30T11:58:13.222 INFO:teuthology.orchestra.run.smithi118.stdout:(4/30): libradosstriper1-18.0.0-6851.g003372b1. 5.0 MB/s | 473 kB 00:00 2023-10-30T11:58:13.872 INFO:teuthology.orchestra.run.smithi118.stdout:(5/30): librgw2-18.0.0-6851.g003372b1.el9.x86_6 7.3 MB/s | 4.8 MB 00:00 2023-10-30T11:58:13.973 INFO:teuthology.orchestra.run.smithi118.stdout:(6/30): python3-ceph-argparse-18.0.0-6851.g0033 451 kB/s | 45 kB 00:00 2023-10-30T11:58:14.106 INFO:teuthology.orchestra.run.smithi118.stdout:(7/30): python3-ceph-common-18.0.0-6851.g003372 950 kB/s | 127 kB 00:00 2023-10-30T11:58:14.240 INFO:teuthology.orchestra.run.smithi118.stdout:(8/30): python3-cephfs-18.0.0-6851.g003372b1.el 1.2 MB/s | 161 kB 00:00 2023-10-30T11:58:14.357 INFO:teuthology.orchestra.run.smithi118.stdout:(9/30): python3-rados-18.0.0-6851.g003372b1.el9 2.7 MB/s | 320 kB 00:00 2023-10-30T11:58:14.466 INFO:teuthology.orchestra.run.smithi118.stdout:(10/30): ceph-radosgw-18.0.0-6851.g003372b1.el9 2.1 MB/s | 8.6 MB 00:04 2023-10-30T11:58:14.490 INFO:teuthology.orchestra.run.smithi118.stdout:(11/30): python3-rbd-18.0.0-6851.g003372b1.el9. 2.2 MB/s | 298 kB 00:00 2023-10-30T11:58:14.600 INFO:teuthology.orchestra.run.smithi118.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 976 kB/s | 106 kB 00:00 2023-10-30T11:58:14.625 INFO:teuthology.orchestra.run.smithi118.stdout:(13/30): python3-rgw-18.0.0-6851.g003372b1.el9. 627 kB/s | 99 kB 00:00 2023-10-30T11:58:14.650 INFO:teuthology.orchestra.run.smithi118.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-10-30T11:58:14.684 INFO:teuthology.orchestra.run.smithi118.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-10-30T11:58:14.709 INFO:teuthology.orchestra.run.smithi118.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-30T11:58:14.726 INFO:teuthology.orchestra.run.smithi118.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 41 kB 00:00 2023-10-30T11:58:14.784 INFO:teuthology.orchestra.run.smithi118.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 605 kB/s | 45 kB 00:00 2023-10-30T11:58:15.152 INFO:teuthology.orchestra.run.smithi118.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 260 kB/s | 111 kB 00:00 2023-10-30T11:58:15.277 INFO:teuthology.orchestra.run.smithi118.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 663 kB/s | 309 kB 00:00 2023-10-30T11:58:15.372 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 19 MB 00:15 2023-10-30T11:58:15.377 INFO:teuthology.orchestra.run.smithi118.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 244 kB/s | 24 kB 00:00 2023-10-30T11:58:15.502 INFO:teuthology.orchestra.run.smithi118.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2023-10-30T11:58:15.619 INFO:teuthology.orchestra.run.smithi118.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2023-10-30T11:58:15.828 INFO:teuthology.orchestra.run.smithi118.stdout:(24/30): ceph-common-18.0.0-6851.g003372b1.el9. 3.5 MB/s | 19 MB 00:05 2023-10-30T11:58:15.895 INFO:teuthology.orchestra.run.smithi118.stdout:(25/30): libarrow-9.0.0-9.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2023-10-30T11:58:15.962 INFO:teuthology.orchestra.run.smithi118.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.4 MB/s | 837 kB 00:00 2023-10-30T11:58:16.171 INFO:teuthology.orchestra.run.smithi118.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-10-30T11:58:16.304 INFO:teuthology.orchestra.run.smithi118.stdout:(28/30): re2-20211101-3.el9.x86_64.rpm 403 kB/s | 192 kB 00:00 2023-10-30T11:58:16.538 INFO:teuthology.orchestra.run.smithi118.stdout:(29/30): librados2-18.0.0-6851.g003372b1.el9.x8 5.7 MB/s | 3.3 MB 00:00 2023-10-30T11:58:17.121 INFO:teuthology.orchestra.run.smithi118.stdout:(30/30): librbd1-18.0.0-6851.g003372b1.el9.x86_ 2.9 MB/s | 3.0 MB 00:01 2023-10-30T11:58:17.128 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:17.128 INFO:teuthology.orchestra.run.smithi118.stdout:Total 7.8 MB/s | 55 MB 00:07 2023-10-30T11:58:17.140 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:58:17.177 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:58:17.178 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:58:17.669 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:58:17.670 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:58:19.060 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:58:19.117 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T11:58:19.167 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T11:58:19.227 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T11:58:19.811 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T11:58:19.859 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:18.0.0-6851.g003372b1.el9.x86_64 5/32 2023-10-30T11:58:19.932 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:18.0.0-6851.g003372b1.el9.x86_64 5/32 2023-10-30T11:58:20.019 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 6/32 2023-10-30T11:58:20.085 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 7/32 2023-10-30T11:58:20.106 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-10-30T11:58:20.140 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 7/32 2023-10-30T11:58:20.175 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T11:58:20.219 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:18.0.0-6851.g003372b1.el9. 9/32 2023-10-30T11:58:20.904 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 10/32 2023-10-30T11:58:21.098 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_6 11/32 2023-10-30T11:58:21.145 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_6 11/32 2023-10-30T11:58:21.255 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T11:58:21.295 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T11:58:21.344 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T11:58:21.376 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T11:58:21.403 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T11:58:21.439 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T11:58:21.472 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:21.774 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T11:58:21.802 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 20/32 2023-10-30T11:58:21.875 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 20/32 2023-10-30T11:58:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 21/32 2023-10-30T11:58:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T11:58:22.419 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T11:58:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 24/32 2023-10-30T11:58:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 24/32 2023-10-30T11:58:22.585 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 25/32 2023-10-30T11:58:22.600 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:58:22.611 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:58:22.611 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:58:22.611 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:58:22.611 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:58:22.612 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:18.0.0-6851.g003372b1.el9 ceph 8.6 M 2023-10-30T11:58:22.612 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2023-10-30T11:58:22.612 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 3.3 M 2023-10-30T11:58:22.612 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 3.0 M 2023-10-30T11:58:22.612 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:58:22.613 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options 2023-10-30T11:58:22.613 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-30T11:58:22.613 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:18.0.0-6851.g003372b1.el9 ceph 5.2 M 2023-10-30T11:58:22.613 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:18.0.0-6851.g003372b1.el9 ceph 19 M 2023-10-30T11:58:22.613 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:18.0.0-6851.g003372b1.el9 ceph 24 k 2023-10-30T11:58:22.614 INFO:teuthology.orchestra.run.smithi050.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-30T11:58:22.614 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-30T11:58:22.614 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-30T11:58:22.614 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-30T11:58:22.614 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 716 k 2023-10-30T11:58:22.615 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-30T11:58:22.615 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-30T11:58:22.615 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-30T11:58:22.615 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 473 k 2023-10-30T11:58:22.615 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-30T11:58:22.615 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:18.0.0-6851.g003372b1.el9 ceph 4.8 M 2023-10-30T11:58:22.616 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-30T11:58:22.616 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-30T11:58:22.616 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-30T11:58:22.616 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse 2023-10-30T11:58:22.616 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 2:18.0.0-6851.g003372b1.el9 ceph 45 k 2023-10-30T11:58:22.617 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:18.0.0-6851.g003372b1.el9 ceph 127 k 2023-10-30T11:58:22.617 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:18.0.0-6851.g003372b1.el9 ceph 161 k 2023-10-30T11:58:22.617 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-30T11:58:22.617 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:18.0.0-6851.g003372b1.el9 ceph 320 k 2023-10-30T11:58:22.617 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:18.0.0-6851.g003372b1.el9 ceph 298 k 2023-10-30T11:58:22.618 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:18.0.0-6851.g003372b1.el9 ceph 99 k 2023-10-30T11:58:22.618 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-30T11:58:22.618 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-30T11:58:22.618 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:22.618 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:58:22.618 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:58:22.619 INFO:teuthology.orchestra.run.smithi050.stdout:Install 28 Packages 2023-10-30T11:58:22.619 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2023-10-30T11:58:22.619 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:18.0.0-6851.g003372b1.el9.x8 26/32 2023-10-30T11:58:22.620 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 55 M 2023-10-30T11:58:22.620 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:58:24.574 INFO:teuthology.orchestra.run.smithi050.stdout:(1/30): ceph-base-18.0.0-6851.g003372b1.el9.x86 3.5 MB/s | 5.2 MB 00:01 2023-10-30T11:58:24.649 INFO:teuthology.orchestra.run.smithi050.stdout:(2/30): ceph-selinux-18.0.0-6851.g003372b1.el9. 321 kB/s | 24 kB 00:00 2023-10-30T11:58:24.824 INFO:teuthology.orchestra.run.smithi050.stdout:(3/30): libcephfs2-18.0.0-6851.g003372b1.el9.x8 4.0 MB/s | 716 kB 00:00 2023-10-30T11:58:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 27/32 2023-10-30T11:58:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 27/32 2023-10-30T11:58:24.949 INFO:teuthology.orchestra.run.smithi050.stdout:(4/30): libradosstriper1-18.0.0-6851.g003372b1. 3.7 MB/s | 473 kB 00:00 2023-10-30T11:58:25.469 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 27/32 2023-10-30T11:58:25.532 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 28/32 2023-10-30T11:58:25.616 INFO:teuthology.orchestra.run.smithi050.stdout:(5/30): librgw2-18.0.0-6851.g003372b1.el9.x86_6 7.2 MB/s | 4.8 MB 00:00 2023-10-30T11:58:25.619 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 28/32 2023-10-30T11:58:25.619 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T11:58:25.619 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T11:58:25.620 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:25.658 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 29/32 2023-10-30T11:58:25.683 INFO:teuthology.orchestra.run.smithi050.stdout:(6/30): python3-ceph-argparse-18.0.0-6851.g0033 678 kB/s | 45 kB 00:00 2023-10-30T11:58:25.750 INFO:teuthology.orchestra.run.smithi050.stdout:(7/30): python3-ceph-common-18.0.0-6851.g003372 1.9 MB/s | 127 kB 00:00 2023-10-30T11:58:25.817 INFO:teuthology.orchestra.run.smithi050.stdout:(8/30): python3-cephfs-18.0.0-6851.g003372b1.el 2.4 MB/s | 161 kB 00:00 2023-10-30T11:58:25.884 INFO:teuthology.orchestra.run.smithi050.stdout:(9/30): python3-rados-18.0.0-6851.g003372b1.el9 4.7 MB/s | 320 kB 00:00 2023-10-30T11:58:25.959 INFO:teuthology.orchestra.run.smithi050.stdout:(10/30): python3-rbd-18.0.0-6851.g003372b1.el9. 3.9 MB/s | 298 kB 00:00 2023-10-30T11:58:26.026 INFO:teuthology.orchestra.run.smithi050.stdout:(11/30): python3-rgw-18.0.0-6851.g003372b1.el9. 1.5 MB/s | 99 kB 00:00 2023-10-30T11:58:26.118 INFO:teuthology.orchestra.run.smithi050.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.1 MB/s | 106 kB 00:00 2023-10-30T11:58:26.161 INFO:teuthology.orchestra.run.smithi050.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2023-10-30T11:58:26.211 INFO:teuthology.orchestra.run.smithi050.stdout:(14/30): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-10-30T11:58:26.244 INFO:teuthology.orchestra.run.smithi050.stdout:(15/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2023-10-30T11:58:26.270 INFO:teuthology.orchestra.run.smithi050.stdout:(16/30): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2023-10-30T11:58:26.370 INFO:teuthology.orchestra.run.smithi050.stdout:(17/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 453 kB/s | 45 kB 00:00 2023-10-30T11:58:26.821 INFO:teuthology.orchestra.run.smithi050.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 246 kB/s | 111 kB 00:00 2023-10-30T11:58:27.079 INFO:teuthology.orchestra.run.smithi050.stdout:(19/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2023-10-30T11:58:27.613 INFO:teuthology.orchestra.run.smithi050.stdout:(20/30): ceph-radosgw-18.0.0-6851.g003372b1.el9 1.9 MB/s | 8.6 MB 00:04 2023-10-30T11:58:27.680 INFO:teuthology.orchestra.run.smithi050.stdout:(21/30): libarrow-9.0.0-9.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2023-10-30T11:58:27.813 INFO:teuthology.orchestra.run.smithi050.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 365 kB/s | 49 kB 00:00 2023-10-30T11:58:27.922 INFO:teuthology.orchestra.run.smithi050.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2023-10-30T11:58:27.947 INFO:teuthology.orchestra.run.smithi050.stdout:(24/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 73 kB/s | 24 kB 00:00 2023-10-30T11:58:28.055 INFO:teuthology.orchestra.run.smithi050.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 6.1 MB/s | 837 kB 00:00 2023-10-30T11:58:28.206 INFO:teuthology.orchestra.run.smithi050.stdout:(26/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-10-30T11:58:28.323 INFO:teuthology.orchestra.run.smithi050.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 510 kB/s | 192 kB 00:00 2023-10-30T11:58:28.831 INFO:teuthology.orchestra.run.smithi050.stdout:(28/30): librados2-18.0.0-6851.g003372b1.el9.x8 5.3 MB/s | 3.3 MB 00:00 2023-10-30T11:58:29.490 INFO:teuthology.orchestra.run.smithi050.stdout:(29/30): librbd1-18.0.0-6851.g003372b1.el9.x86_ 2.6 MB/s | 3.0 MB 00:01 2023-10-30T11:58:30.807 INFO:teuthology.orchestra.run.smithi050.stdout:(30/30): ceph-common-18.0.0-6851.g003372b1.el9. 2.5 MB/s | 19 MB 00:07 2023-10-30T11:58:30.810 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:30.811 INFO:teuthology.orchestra.run.smithi050.stdout:Total 6.7 MB/s | 55 MB 00:08 2023-10-30T11:58:30.823 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:58:30.866 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:58:30.866 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:58:31.357 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:58:31.358 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:58:32.916 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:58:32.978 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-30T11:58:33.033 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-10-30T11:58:33.097 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-10-30T11:58:33.435 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-10-30T11:58:33.486 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:18.0.0-6851.g003372b1.el9.x86_64 5/32 2023-10-30T11:58:33.578 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:18.0.0-6851.g003372b1.el9.x86_64 5/32 2023-10-30T11:58:33.674 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 6/32 2023-10-30T11:58:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 7/32 2023-10-30T11:58:33.742 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 7/32 2023-10-30T11:58:33.774 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-10-30T11:58:33.818 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:18.0.0-6851.g003372b1.el9. 9/32 2023-10-30T11:58:33.886 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 10/32 2023-10-30T11:58:33.910 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_6 11/32 2023-10-30T11:58:33.968 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_6 11/32 2023-10-30T11:58:34.077 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-30T11:58:34.110 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-30T11:58:34.150 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-30T11:58:34.183 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-30T11:58:34.210 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-30T11:58:34.246 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-30T11:58:34.279 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:34.596 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-30T11:58:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 20/32 2023-10-30T11:58:34.724 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 20/32 2023-10-30T11:58:34.755 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 21/32 2023-10-30T11:58:34.791 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-30T11:58:35.014 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 29/32 2023-10-30T11:58:35.015 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2023-10-30T11:58:35.015 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2023-10-30T11:58:35.015 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2023-10-30T11:58:35.015 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2023-10-30T11:58:35.016 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2023-10-30T11:58:35.016 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2023-10-30T11:58:35.016 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2023-10-30T11:58:35.016 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:35.240 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-30T11:58:35.270 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 24/32 2023-10-30T11:58:35.341 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 24/32 2023-10-30T11:58:35.421 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 25/32 2023-10-30T11:58:35.472 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:18.0.0-6851.g003372b1.el9.x8 26/32 2023-10-30T11:58:35.800 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 30/32 2023-10-30T11:58:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 30/32 2023-10-30T11:58:35.824 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:58:35.825 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T11:58:35.825 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:35.825 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:35.911 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 1/32 2023-10-30T11:58:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 2/32 2023-10-30T11:58:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 3/32 2023-10-30T11:58:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 4/32 2023-10-30T11:58:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 5/32 2023-10-30T11:58:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_6 6/32 2023-10-30T11:58:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 7/32 2023-10-30T11:58:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6851.g003372b1.el9. 8/32 2023-10-30T11:58:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:18.0.0-6851.g003372b1.el9.x8 9/32 2023-10-30T11:58:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 10/32 2023-10-30T11:58:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 11/32 2023-10-30T11:58:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 12/32 2023-10-30T11:58:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 13/32 2023-10-30T11:58:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T11:58:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T11:58:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T11:58:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T11:58:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T11:58:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T11:58:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T11:58:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T11:58:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T11:58:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T11:58:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T11:58:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T11:58:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T11:58:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T11:58:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:18.0.0-6851.g003372b1.el9.x86_64 29/32 2023-10-30T11:58:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T11:58:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 31/32 2023-10-30T11:58:37.685 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 27/32 2023-10-30T11:58:37.724 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 27/32 2023-10-30T11:58:38.047 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:38.047 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:38.047 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2023-10-30T11:58:38.047 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.048 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.048 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:58:38.048 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T11:58:38.048 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.048 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.049 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.049 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.049 INFO:teuthology.orchestra.run.smithi118.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T11:58:38.049 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T11:58:38.049 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T11:58:38.050 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T11:58:38.050 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.050 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T11:58:38.050 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T11:58:38.050 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T11:58:38.051 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.051 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T11:58:38.051 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.051 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T11:58:38.051 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T11:58:38.052 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T11:58:38.052 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.052 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.052 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.052 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T11:58:38.053 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.053 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.053 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:38.054 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T11:58:38.054 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T11:58:38.054 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:38.054 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:58:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 27/32 2023-10-30T11:58:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 28/32 2023-10-30T11:58:38.404 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2023-10-30T11:58:38.563 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 28/32 2023-10-30T11:58:38.563 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-30T11:58:38.563 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-30T11:58:38.564 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 29/32 2023-10-30T11:58:38.947 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:31 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:58:39.097 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:58:39.097 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:58:39.098 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:58:39.098 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:58:39.098 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:58:39.098 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:18.0.0-6851.g003372b1.el9 ceph 41 M 2023-10-30T11:58:39.098 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:58:39.099 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T11:58:39.099 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:18.0.0-6851.g003372b1.el9 ceph 164 k 2023-10-30T11:58:39.099 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T11:58:39.099 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T11:58:39.099 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T11:58:39.100 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:39.100 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:58:39.100 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:58:39.100 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2023-10-30T11:58:39.100 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:39.101 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 42 M 2023-10-30T11:58:39.101 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 172 M 2023-10-30T11:58:39.101 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:58:39.366 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2023-10-30T11:58:39.408 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2023-10-30T11:58:39.449 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-10-30T11:58:39.500 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): libcephsqlite-18.0.0-6851.g003372b1.el9. 599 kB/s | 164 kB 00:00 2023-10-30T11:58:39.650 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 933 kB/s | 187 kB 00:00 2023-10-30T11:58:41.692 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-18.0.0-6851.g003372b1.el9.x86_ 17 MB/s | 41 MB 00:02 2023-10-30T11:58:41.695 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:41.695 INFO:teuthology.orchestra.run.smithi118.stdout:Total 16 MB/s | 42 MB 00:02 2023-10-30T11:58:41.708 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:58:41.730 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:58:41.730 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:58:42.095 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:58:42.095 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:58:42.620 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:58:42.706 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T11:58:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T11:58:42.779 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:58:42.825 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T11:58:42.854 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 5/6 2023-10-30T11:58:46.698 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 5/6 2023-10-30T11:58:46.777 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 6/6 2023-10-30T11:58:48.606 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 29/32 2023-10-30T11:58:48.606 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2023-10-30T11:58:48.606 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2023-10-30T11:58:48.606 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2023-10-30T11:58:48.607 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2023-10-30T11:58:48.607 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2023-10-30T11:58:48.607 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2023-10-30T11:58:48.607 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2023-10-30T11:58:48.607 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 6/6 2023-10-30T11:58:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 1/6 2023-10-30T11:58:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 2/6 2023-10-30T11:58:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:58:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T11:58:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T11:58:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 30/32 2023-10-30T11:58:49.470 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 30/32 2023-10-30T11:58:49.470 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:58:49.471 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-30T11:58:49.471 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:49.471 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-30T11:58:49.471 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:49.556 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-30T11:58:49.556 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T11:58:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:49.648 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:58:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.6-15.el9.x86_64 2023-10-30T11:58:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T11:58:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T11:58:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T11:58:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:49.650 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:58:49.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2023-10-30T11:58:50.485 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:43 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:58:50.628 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:58:50.630 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2023-10-30T11:58:50.630 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:58:50.631 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2023-10-30T11:58:50.631 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:58:50.631 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:18.0.0-6851.g003372b1.el9 ceph 6.5 k 2023-10-30T11:58:50.632 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:58:50.632 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:18.0.0-6851.g003372b1.el9 ceph 2.1 M 2023-10-30T11:58:50.632 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:18.0.0-6851.g003372b1.el9 ceph 1.5 M 2023-10-30T11:58:50.632 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 243 k 2023-10-30T11:58:50.633 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:18.0.0-6851.g003372b1.el9 ceph 4.4 M 2023-10-30T11:58:50.633 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:18.0.0-6851.g003372b1.el9 ceph 18 M 2023-10-30T11:58:50.633 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T11:58:50.633 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T11:58:50.634 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T11:58:50.634 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T11:58:50.634 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T11:58:50.634 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T11:58:50.636 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T11:58:50.636 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T11:58:50.636 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T11:58:50.636 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T11:58:50.637 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T11:58:50.637 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T11:58:50.637 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T11:58:50.637 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T11:58:50.637 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T11:58:50.638 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T11:58:50.638 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T11:58:50.638 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T11:58:50.638 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T11:58:50.639 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T11:58:50.639 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T11:58:50.639 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T11:58:50.639 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T11:58:50.639 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:50.640 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:58:50.640 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2023-10-30T11:58:50.640 INFO:teuthology.orchestra.run.smithi118.stdout:Install 33 Packages 2023-10-30T11:58:50.641 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:58:50.642 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 30 M 2023-10-30T11:58:50.642 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 110 M 2023-10-30T11:58:50.642 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:58:50.812 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:50.813 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 1/32 2023-10-30T11:58:50.813 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 2/32 2023-10-30T11:58:50.813 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 3/32 2023-10-30T11:58:50.813 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 4/32 2023-10-30T11:58:50.813 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 5/32 2023-10-30T11:58:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_6 6/32 2023-10-30T11:58:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 7/32 2023-10-30T11:58:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6851.g003372b1.el9. 8/32 2023-10-30T11:58:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:18.0.0-6851.g003372b1.el9.x8 9/32 2023-10-30T11:58:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 10/32 2023-10-30T11:58:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 11/32 2023-10-30T11:58:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 12/32 2023-10-30T11:58:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 13/32 2023-10-30T11:58:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-30T11:58:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-30T11:58:50.816 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-30T11:58:50.816 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-30T11:58:50.816 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-30T11:58:50.816 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-30T11:58:50.816 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-30T11:58:50.816 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-30T11:58:50.817 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-30T11:58:50.817 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-30T11:58:50.817 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-30T11:58:50.817 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-30T11:58:50.817 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-30T11:58:50.818 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-30T11:58:50.818 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-30T11:58:50.818 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:18.0.0-6851.g003372b1.el9.x86_64 29/32 2023-10-30T11:58:50.818 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-30T11:58:50.818 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 31/32 2023-10-30T11:58:51.162 INFO:teuthology.orchestra.run.smithi118.stdout:(1/33): ceph-18.0.0-6851.g003372b1.el9.x86_64.r 29 kB/s | 6.5 kB 00:00 2023-10-30T11:58:51.846 INFO:teuthology.orchestra.run.smithi118.stdout:(2/33): ceph-mds-18.0.0-6851.g003372b1.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2023-10-30T11:58:51.879 INFO:teuthology.orchestra.run.smithi118.stdout:(3/33): ceph-mgr-18.0.0-6851.g003372b1.el9.x86_ 1.5 MB/s | 1.5 MB 00:00 2023-10-30T11:58:52.024 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-30T11:58:52.025 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:52.025 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2023-10-30T11:58:52.025 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.026 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.026 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:58:52.027 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-30T11:58:52.027 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.027 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.028 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.028 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.028 INFO:teuthology.orchestra.run.smithi050.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-30T11:58:52.028 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-30T11:58:52.029 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-30T11:58:52.029 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-30T11:58:52.029 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.029 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-30T11:58:52.030 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-30T11:58:52.030 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-30T11:58:52.030 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.031 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-30T11:58:52.031 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.031 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-30T11:58:52.032 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-30T11:58:52.032 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-30T11:58:52.032 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.032 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.032 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.033 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-30T11:58:52.033 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.033 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.033 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:58:52.034 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-3.el9.x86_64 2023-10-30T11:58:52.034 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-30T11:58:52.034 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:52.035 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:58:52.046 INFO:teuthology.orchestra.run.smithi118.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6851.g0033 1.4 MB/s | 243 kB 00:00 2023-10-30T11:58:52.156 INFO:teuthology.orchestra.run.smithi118.stdout:(5/33): ceph-mon-18.0.0-6851.g003372b1.el9.x86_ 4.4 MB/s | 4.4 MB 00:00 2023-10-30T11:58:52.264 INFO:teuthology.orchestra.run.smithi118.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 381 kB/s | 32 kB 00:00 2023-10-30T11:58:52.289 INFO:teuthology.orchestra.run.smithi118.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 596 kB/s | 144 kB 00:00 2023-10-30T11:58:52.314 INFO:teuthology.orchestra.run.smithi118.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2023-10-30T11:58:52.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2023-10-30T11:58:52.348 INFO:teuthology.orchestra.run.smithi118.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-10-30T11:58:52.373 INFO:teuthology.orchestra.run.smithi118.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2023-10-30T11:58:52.399 INFO:teuthology.orchestra.run.smithi118.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-10-30T11:58:52.423 INFO:teuthology.orchestra.run.smithi118.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-30T11:58:52.449 INFO:teuthology.orchestra.run.smithi118.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-10-30T11:58:52.558 INFO:teuthology.orchestra.run.smithi118.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2023-10-30T11:58:52.650 INFO:teuthology.orchestra.run.smithi118.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 5.4 MB/s | 1.2 MB 00:00 2023-10-30T11:58:52.675 INFO:teuthology.orchestra.run.smithi118.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2023-10-30T11:58:52.717 INFO:teuthology.orchestra.run.smithi118.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2023-10-30T11:58:52.760 INFO:teuthology.orchestra.run.smithi118.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2023-10-30T11:58:52.784 INFO:teuthology.orchestra.run.smithi118.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2023-10-30T11:58:52.810 INFO:teuthology.orchestra.run.smithi118.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2023-10-30T11:58:52.835 INFO:teuthology.orchestra.run.smithi118.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2023-10-30T11:58:52.852 INFO:teuthology.orchestra.run.smithi118.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 463 kB/s | 19 kB 00:00 2023-10-30T11:58:52.869 INFO:teuthology.orchestra.run.smithi118.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 579 kB/s | 20 kB 00:00 2023-10-30T11:58:52.894 INFO:teuthology.orchestra.run.smithi118.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2023-10-30T11:58:52.895 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:32 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:58:52.921 INFO:teuthology.orchestra.run.smithi118.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2023-10-30T11:58:52.953 INFO:teuthology.orchestra.run.smithi118.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2023-10-30T11:58:52.979 INFO:teuthology.orchestra.run.smithi118.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2023-10-30T11:58:53.009 INFO:teuthology.orchestra.run.smithi118.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2023-10-30T11:58:53.029 INFO:teuthology.orchestra.run.smithi118.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2023-10-30T11:58:53.041 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:58:53.042 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:58:53.042 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:58:53.042 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:58:53.042 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:58:53.043 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:18.0.0-6851.g003372b1.el9 ceph 41 M 2023-10-30T11:58:53.043 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:58:53.043 INFO:teuthology.orchestra.run.smithi050.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-30T11:58:53.043 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:18.0.0-6851.g003372b1.el9 ceph 164 k 2023-10-30T11:58:53.043 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-30T11:58:53.043 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-30T11:58:53.044 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-30T11:58:53.044 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:53.044 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:58:53.044 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:58:53.044 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-10-30T11:58:53.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:58:53.045 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 42 M 2023-10-30T11:58:53.045 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 172 M 2023-10-30T11:58:53.046 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:58:53.054 INFO:teuthology.orchestra.run.smithi118.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-10-30T11:58:53.113 INFO:teuthology.orchestra.run.smithi118.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2023-10-30T11:58:53.138 INFO:teuthology.orchestra.run.smithi118.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2023-10-30T11:58:53.402 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-10-30T11:58:53.452 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-10-30T11:58:53.477 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-10-30T11:58:53.810 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): libcephsqlite-18.0.0-6851.g003372b1.el9. 294 kB/s | 164 kB 00:00 2023-10-30T11:58:53.994 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 361 kB/s | 187 kB 00:00 2023-10-30T11:58:56.094 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): ceph-test-18.0.0-6851.g003372b1.el9.x86_ 14 MB/s | 41 MB 00:02 2023-10-30T11:58:56.097 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:58:56.097 INFO:teuthology.orchestra.run.smithi050.stdout:Total 14 MB/s | 42 MB 00:03 2023-10-30T11:58:56.112 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:58:56.136 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:58:56.136 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:58:56.514 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:58:56.515 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:58:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:58:57.155 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-10-30T11:58:57.196 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-10-30T11:58:57.237 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:58:57.278 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-10-30T11:58:57.306 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 5/6 2023-10-30T11:59:00.246 INFO:teuthology.orchestra.run.smithi118.stdout:(33/33): ceph-osd-18.0.0-6851.g003372b1.el9.x86 2.1 MB/s | 18 MB 00:08 2023-10-30T11:59:00.249 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:00.250 INFO:teuthology.orchestra.run.smithi118.stdout:Total 3.1 MB/s | 30 MB 00:09 2023-10-30T11:59:00.283 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:00.325 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:00.326 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:01.334 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 5/6 2023-10-30T11:59:01.407 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 6/6 2023-10-30T11:59:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:01.990 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T11:59:02.022 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T11:59:02.055 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T11:59:02.146 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T11:59:02.234 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T11:59:02.316 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T11:59:02.362 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T11:59:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T11:59:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T11:59:02.523 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T11:59:02.572 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:59:02.639 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:59:02.836 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T11:59:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T11:59:02.931 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T11:59:02.965 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T11:59:03.037 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T11:59:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T11:59:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T11:59:03.222 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T11:59:03.264 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T11:59:03.298 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T11:59:03.499 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T11:59:03.580 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 6/6 2023-10-30T11:59:03.581 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 1/6 2023-10-30T11:59:03.581 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 2/6 2023-10-30T11:59:03.581 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-10-30T11:59:03.581 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-10-30T11:59:03.582 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-10-30T11:59:03.630 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T11:59:03.702 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T11:59:03.883 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T11:59:03.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-10-30T11:59:03.959 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:03.959 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:03.959 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:03.959 INFO:teuthology.orchestra.run.smithi050.stdout: jq-1.6-15.el9.x86_64 2023-10-30T11:59:03.959 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:03.960 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-30T11:59:03.960 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-30T11:59:03.960 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-30T11:59:03.960 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:03.960 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:04.006 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:59:04.158 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6851.g003372b1.el9. 28/33 2023-10-30T11:59:04.185 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 29/33 2023-10-30T11:59:04.210 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 29/33 2023-10-30T11:59:04.210 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:04.210 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T11:59:04.210 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:59:04.211 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:59:04.211 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:04.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2023-10-30T11:59:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:44 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2023-10-30T11:59:04.939 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2023-10-30T11:59:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:04.940 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:18.0.0-6851.g003372b1.el9 ceph 6.5 k 2023-10-30T11:59:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:59:04.940 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:18.0.0-6851.g003372b1.el9 ceph 2.1 M 2023-10-30T11:59:04.941 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:18.0.0-6851.g003372b1.el9 ceph 1.5 M 2023-10-30T11:59:04.941 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 243 k 2023-10-30T11:59:04.941 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:18.0.0-6851.g003372b1.el9 ceph 4.4 M 2023-10-30T11:59:04.941 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:18.0.0-6851.g003372b1.el9 ceph 18 M 2023-10-30T11:59:04.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-30T11:59:04.942 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-30T11:59:04.942 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-30T11:59:04.942 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-30T11:59:04.942 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-30T11:59:04.942 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-30T11:59:04.943 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-30T11:59:04.943 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-30T11:59:04.943 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-30T11:59:04.943 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-30T11:59:04.943 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-30T11:59:04.943 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-30T11:59:04.944 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-30T11:59:04.944 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-30T11:59:04.944 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-30T11:59:04.944 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-30T11:59:04.944 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-30T11:59:04.945 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-30T11:59:04.945 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-30T11:59:04.945 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-30T11:59:04.945 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-30T11:59:04.945 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-30T11:59:04.946 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-30T11:59:04.946 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-30T11:59:04.946 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-30T11:59:04.946 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-30T11:59:04.946 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-30T11:59:04.947 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2023-10-30T11:59:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:Install 33 Packages 2023-10-30T11:59:04.947 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 30 M 2023-10-30T11:59:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 110 M 2023-10-30T11:59:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:05.505 INFO:teuthology.orchestra.run.smithi050.stdout:(1/33): ceph-18.0.0-6851.g003372b1.el9.x86_64.r 37 kB/s | 6.5 kB 00:00 2023-10-30T11:59:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 30/33 2023-10-30T11:59:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 30/33 2023-10-30T11:59:05.826 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:05.827 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T11:59:05.827 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:05.827 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:05.947 INFO:teuthology.orchestra.run.smithi050.stdout:(2/33): ceph-mds-18.0.0-6851.g003372b1.el9.x86_ 3.4 MB/s | 2.1 MB 00:00 2023-10-30T11:59:06.222 INFO:teuthology.orchestra.run.smithi050.stdout:(3/33): ceph-mon-18.0.0-6851.g003372b1.el9.x86_ 6.1 MB/s | 4.4 MB 00:00 2023-10-30T11:59:06.264 INFO:teuthology.orchestra.run.smithi050.stdout:(4/33): ceph-mgr-18.0.0-6851.g003372b1.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2023-10-30T11:59:06.265 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 31/33 2023-10-30T11:59:06.289 INFO:teuthology.orchestra.run.smithi050.stdout:(5/33): ceph-mgr-modules-core-18.0.0-6851.g0033 3.5 MB/s | 243 kB 00:00 2023-10-30T11:59:06.289 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 31/33 2023-10-30T11:59:06.290 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:06.290 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T11:59:06.290 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:06.291 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:06.291 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:06.381 INFO:teuthology.orchestra.run.smithi050.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 472 kB/s | 32 kB 00:00 2023-10-30T11:59:06.406 INFO:teuthology.orchestra.run.smithi050.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2023-10-30T11:59:06.423 INFO:teuthology.orchestra.run.smithi050.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 920 kB/s | 39 kB 00:00 2023-10-30T11:59:06.449 INFO:teuthology.orchestra.run.smithi050.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-10-30T11:59:06.466 INFO:teuthology.orchestra.run.smithi050.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2023-10-30T11:59:06.483 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 32/33 2023-10-30T11:59:06.491 INFO:teuthology.orchestra.run.smithi050.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-10-30T11:59:06.506 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 32/33 2023-10-30T11:59:06.506 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:06.506 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T11:59:06.506 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:06.506 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:06.507 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:06.516 INFO:teuthology.orchestra.run.smithi050.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-10-30T11:59:06.542 INFO:teuthology.orchestra.run.smithi050.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-10-30T11:59:06.564 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:18.0.0-6851.g003372b1.el9.x86_64 33/33 2023-10-30T11:59:06.650 INFO:teuthology.orchestra.run.smithi050.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2023-10-30T11:59:06.742 INFO:teuthology.orchestra.run.smithi050.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 5.4 MB/s | 1.2 MB 00:00 2023-10-30T11:59:07.184 INFO:teuthology.orchestra.run.smithi050.stdout:(16/33): ceph-osd-18.0.0-6851.g003372b1.el9.x86 14 MB/s | 18 MB 00:01 2023-10-30T11:59:07.209 INFO:teuthology.orchestra.run.smithi050.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 78 kB/s | 43 kB 00:00 2023-10-30T11:59:07.335 INFO:teuthology.orchestra.run.smithi050.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2023-10-30T11:59:07.418 INFO:teuthology.orchestra.run.smithi050.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 264 kB/s | 172 kB 00:00 2023-10-30T11:59:07.460 INFO:teuthology.orchestra.run.smithi050.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2023-10-30T11:59:07.527 INFO:teuthology.orchestra.run.smithi050.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2023-10-30T11:59:07.586 INFO:teuthology.orchestra.run.smithi050.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2023-10-30T11:59:07.636 INFO:teuthology.orchestra.run.smithi050.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2023-10-30T11:59:07.761 INFO:teuthology.orchestra.run.smithi050.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 264 kB/s | 46 kB 00:00 2023-10-30T11:59:07.788 INFO:teuthology.orchestra.run.smithi050.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 524 kB/s | 79 kB 00:00 2023-10-30T11:59:07.854 INFO:teuthology.orchestra.run.smithi050.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 536 kB/s | 358 kB 00:00 2023-10-30T11:59:07.896 INFO:teuthology.orchestra.run.smithi050.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2023-10-30T11:59:07.970 INFO:teuthology.orchestra.run.smithi050.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 770 kB/s | 90 kB 00:00 2023-10-30T11:59:08.012 INFO:teuthology.orchestra.run.smithi050.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2023-10-30T11:59:08.071 INFO:teuthology.orchestra.run.smithi050.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 879 kB/s | 272 kB 00:00 2023-10-30T11:59:08.113 INFO:teuthology.orchestra.run.smithi050.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2023-10-30T11:59:08.180 INFO:teuthology.orchestra.run.smithi050.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2023-10-30T11:59:08.280 INFO:teuthology.orchestra.run.smithi050.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2023-10-30T11:59:08.287 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:08.287 INFO:teuthology.orchestra.run.smithi050.stdout:Total 9.0 MB/s | 30 MB 00:03 2023-10-30T11:59:08.320 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:08.353 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:18.0.0-6851.g003372b1.el9.x86_64 33/33 2023-10-30T11:59:08.353 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:18.0.0-6851.g003372b1.el9.x86_64 1/33 2023-10-30T11:59:08.353 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 2/33 2023-10-30T11:59:08.353 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 3/33 2023-10-30T11:59:08.353 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 4/33 2023-10-30T11:59:08.354 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 5/33 2023-10-30T11:59:08.354 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6851.g003372b1.el9. 6/33 2023-10-30T11:59:08.354 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T11:59:08.354 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T11:59:08.354 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T11:59:08.355 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T11:59:08.355 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:59:08.355 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:59:08.355 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T11:59:08.355 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T11:59:08.356 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T11:59:08.356 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T11:59:08.356 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T11:59:08.356 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T11:59:08.356 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T11:59:08.356 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T11:59:08.357 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T11:59:08.357 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T11:59:08.357 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T11:59:08.357 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T11:59:08.357 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T11:59:08.358 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T11:59:08.358 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:59:08.359 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T11:59:08.359 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T11:59:08.359 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T11:59:08.359 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T11:59:08.359 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T11:59:08.365 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:08.365 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:08.751 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:08.751 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:09.372 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T11:59:09.373 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:09.373 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:09.373 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:09.373 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:09.374 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:09.374 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:09.374 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:09.374 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:09.374 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T11:59:09.375 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T11:59:09.375 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T11:59:09.375 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T11:59:09.375 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T11:59:09.375 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T11:59:09.376 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T11:59:09.376 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T11:59:09.376 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T11:59:09.376 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T11:59:09.376 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T11:59:09.377 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T11:59:09.377 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T11:59:09.377 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T11:59:09.377 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T11:59:09.377 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T11:59:09.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T11:59:09.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T11:59:09.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T11:59:09.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T11:59:09.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T11:59:09.379 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T11:59:09.379 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T11:59:09.379 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T11:59:09.379 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T11:59:09.379 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T11:59:09.380 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T11:59:09.380 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:09.380 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:09.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2023-10-30T11:59:10.165 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:10.224 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-30T11:59:10.258 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-30T11:59:10.291 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-30T11:59:10.318 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:03 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:10.388 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-30T11:59:10.408 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:10.456 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:10.457 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:10.457 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:10.463 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-30T11:59:10.511 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-30T11:59:10.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2023-10-30T11:59:10.548 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-30T11:59:10.628 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-30T11:59:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-30T11:59:10.725 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-30T11:59:10.784 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:59:10.856 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:59:11.053 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-30T11:59:11.065 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:04 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:11.104 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-30T11:59:11.151 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-30T11:59:11.186 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-30T11:59:11.204 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:11.206 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:59:11.206 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:11.206 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 270 k 2023-10-30T11:59:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:11.207 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:11.207 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:11.207 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T11:59:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:11.208 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 270 k 2023-10-30T11:59:11.208 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 265 k 2023-10-30T11:59:11.208 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:11.215 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-30T11:59:11.297 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-30T11:59:11.334 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-30T11:59:11.368 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-30T11:59:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-30T11:59:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-30T11:59:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-30T11:59:11.549 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-18.0.0-6851.g003372b1.el9.noarch.rpm 788 kB/s | 270 kB 00:00 2023-10-30T11:59:11.549 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:11.550 INFO:teuthology.orchestra.run.smithi118.stdout:Total 785 kB/s | 270 kB 00:00 2023-10-30T11:59:11.550 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:11.555 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:11.555 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:11.558 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:11.558 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:11.607 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-30T11:59:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-30T11:59:11.828 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-30T11:59:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:59:12.013 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:12.095 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:12.118 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6851.g003372b1.el9. 28/33 2023-10-30T11:59:12.146 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 29/33 2023-10-30T11:59:12.171 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 29/33 2023-10-30T11:59:12.172 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:12.172 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-30T11:59:12.172 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:59:12.172 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-30T11:59:12.172 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:12.445 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:12.723 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:12.723 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:12.723 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:12.723 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:12.724 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:12.724 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:12.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T11:59:13.498 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:06 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:13.639 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:13.640 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:13.640 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-30T11:59:13.641 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:13.641 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:13.641 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6851.g003372b1.el9 ceph 141 k 2023-10-30T11:59:13.641 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:13.641 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:13.642 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:13.642 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T11:59:13.642 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:13.642 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 141 k 2023-10-30T11:59:13.642 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 400 k 2023-10-30T11:59:13.643 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:13.797 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 30/33 2023-10-30T11:59:13.825 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 30/33 2023-10-30T11:59:13.826 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:13.826 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-30T11:59:13.826 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:13.826 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-30T11:59:13.827 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:14.148 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-18.0.0-6851.g003372 279 kB/s | 141 kB 00:00 2023-10-30T11:59:14.149 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:14.149 INFO:teuthology.orchestra.run.smithi118.stdout:Total 278 kB/s | 141 kB 00:00 2023-10-30T11:59:14.150 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:14.157 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:14.157 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:14.208 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:14.209 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 31/33 2023-10-30T11:59:14.323 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 31/33 2023-10-30T11:59:14.323 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:14.323 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-30T11:59:14.323 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:14.324 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-30T11:59:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:14.365 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:14.424 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6851.g003372b1. 1/1 2023-10-30T11:59:14.448 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6851.g003372b1. 1/1 2023-10-30T11:59:14.448 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:14.449 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T11:59:14.449 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:14.532 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 32/33 2023-10-30T11:59:14.556 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 32/33 2023-10-30T11:59:14.556 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:14.556 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-30T11:59:14.557 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:14.557 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-30T11:59:14.557 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:14.624 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:18.0.0-6851.g003372b1.el9.x86_64 33/33 2023-10-30T11:59:15.379 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6851.g003372b1. 1/1 2023-10-30T11:59:15.379 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:15.379 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:15.380 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:15.380 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:15.380 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:15.659 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2023-10-30T11:59:16.224 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:09 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:16.316 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:16.365 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:16.366 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:16.367 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:16.405 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:18.0.0-6851.g003372b1.el9.x86_64 33/33 2023-10-30T11:59:16.405 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:18.0.0-6851.g003372b1.el9.x86_64 1/33 2023-10-30T11:59:16.406 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 2/33 2023-10-30T11:59:16.406 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 3/33 2023-10-30T11:59:16.406 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 4/33 2023-10-30T11:59:16.406 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 5/33 2023-10-30T11:59:16.406 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6851.g003372b1.el9. 6/33 2023-10-30T11:59:16.407 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-30T11:59:16.407 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-30T11:59:16.407 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-30T11:59:16.407 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-30T11:59:16.407 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-30T11:59:16.408 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-30T11:59:16.408 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-30T11:59:16.408 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-30T11:59:16.408 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-30T11:59:16.408 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-30T11:59:16.408 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-30T11:59:16.409 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-30T11:59:16.409 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-30T11:59:16.409 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-30T11:59:16.409 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-30T11:59:16.409 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-30T11:59:16.410 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-30T11:59:16.410 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-30T11:59:16.410 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-30T11:59:16.411 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-30T11:59:16.411 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-30T11:59:16.411 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-30T11:59:16.412 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-30T11:59:16.412 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-30T11:59:16.412 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-30T11:59:16.412 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-30T11:59:16.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T11:59:16.977 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:09 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:17.117 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:17.118 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:17.119 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:59:17.119 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:17.119 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:17.119 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 1.8 M 2023-10-30T11:59:17.119 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:59:17.119 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 23 k 2023-10-30T11:59:17.120 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 15 k 2023-10-30T11:59:17.120 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T11:59:17.120 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T11:59:17.120 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T11:59:17.120 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2023-10-30T11:59:17.121 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T11:59:17.121 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:17.121 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:17.121 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:17.121 INFO:teuthology.orchestra.run.smithi118.stdout:Install 7 Packages 2023-10-30T11:59:17.122 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:17.122 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.1 M 2023-10-30T11:59:17.122 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 8.6 M 2023-10-30T11:59:17.123 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:17.184 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-30T11:59:17.185 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:17.185 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:17.185 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:17.185 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:17.186 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:17.186 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:17.186 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:17.186 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:17.186 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-30T11:59:17.187 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-30T11:59:17.187 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-30T11:59:17.187 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-30T11:59:17.187 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-30T11:59:17.187 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-30T11:59:17.187 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-30T11:59:17.188 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-30T11:59:17.188 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-30T11:59:17.188 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-30T11:59:17.188 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-30T11:59:17.188 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-30T11:59:17.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-30T11:59:17.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-30T11:59:17.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-30T11:59:17.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2023-10-30T11:59:17.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-30T11:59:17.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-30T11:59:17.190 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-30T11:59:17.190 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-30T11:59:17.190 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-30T11:59:17.190 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-30T11:59:17.190 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-30T11:59:17.191 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-30T11:59:17.191 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-30T11:59:17.191 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-30T11:59:17.191 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-30T11:59:17.192 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:17.192 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:17.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2023-10-30T11:59:17.736 INFO:teuthology.orchestra.run.smithi118.stdout:(1/7): ceph-prometheus-alerts-18.0.0-6851.g0033 68 kB/s | 15 kB 00:00 2023-10-30T11:59:17.895 INFO:teuthology.orchestra.run.smithi118.stdout:(2/7): ceph-grafana-dashboards-18.0.0-6851.g003 61 kB/s | 23 kB 00:00 2023-10-30T11:59:17.920 INFO:teuthology.orchestra.run.smithi118.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 49 kB/s | 9.0 kB 00:00 2023-10-30T11:59:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 231 kB/s | 31 kB 00:00 2023-10-30T11:59:18.098 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:58 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:18.112 INFO:teuthology.orchestra.run.smithi118.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 188 kB/s | 41 kB 00:00 2023-10-30T11:59:18.192 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:18.243 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:18.244 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T11:59:18.244 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:18.246 INFO:teuthology.orchestra.run.smithi118.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6851.g003372b1 2.4 MB/s | 1.8 MB 00:00 2023-10-30T11:59:18.271 INFO:teuthology.orchestra.run.smithi118.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 869 kB/s | 188 kB 00:00 2023-10-30T11:59:18.272 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:18.273 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.8 MB/s | 2.1 MB 00:01 2023-10-30T11:59:18.273 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:18.281 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:18.282 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:18.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2023-10-30T11:59:18.341 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:18.342 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:18.641 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:18.706 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T11:59:18.740 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T11:59:18.794 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T11:59:18.835 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T11:59:18.864 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6851.g003372b1.el9.n 5/7 2023-10-30T11:59:18.885 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:58 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:19.031 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:19.032 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:19.032 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:19.033 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:19.033 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:19.034 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 270 k 2023-10-30T11:59:19.034 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:19.034 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:19.035 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:19.035 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T11:59:19.035 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:19.036 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 270 k 2023-10-30T11:59:19.036 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 265 k 2023-10-30T11:59:19.037 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:19.105 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6851.g003372b1.el9. 6/7 2023-10-30T11:59:19.142 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarc 7/7 2023-10-30T11:59:19.405 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-18.0.0-6851.g003372b1.el9.noarch.rpm 724 kB/s | 270 kB 00:00 2023-10-30T11:59:19.406 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:19.407 INFO:teuthology.orchestra.run.smithi050.stdout:Total 721 kB/s | 270 kB 00:00 2023-10-30T11:59:19.407 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:19.412 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:19.412 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:19.415 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:19.415 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:19.492 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:19.685 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarc 7/7 2023-10-30T11:59:19.685 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6851.g003372b1.el9. 1/7 2023-10-30T11:59:19.685 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarc 2/7 2023-10-30T11:59:19.686 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6851.g003372b1.el9.n 3/7 2023-10-30T11:59:19.686 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T11:59:19.686 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T11:59:19.686 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T11:59:19.931 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:19.996 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T11:59:19.997 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:19.997 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:19.997 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:19.997 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:19.997 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:19.997 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T11:59:19.998 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T11:59:19.998 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T11:59:19.998 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T11:59:19.998 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:19.998 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:20.013 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:20.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T11:59:20.358 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:20.697 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:20.698 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:20.698 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:20.772 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:13 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:20.915 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:20.916 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2023-10-30T11:59:20.916 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:59:20.916 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2023-10-30T11:59:20.916 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:20.917 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 7.4 M 2023-10-30T11:59:20.917 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:59:20.917 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T11:59:20.917 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T11:59:20.917 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T11:59:20.918 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T11:59:20.918 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T11:59:20.918 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T11:59:20.918 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T11:59:20.918 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T11:59:20.919 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T11:59:20.919 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T11:59:20.919 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T11:59:20.919 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:20.919 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:20.920 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2023-10-30T11:59:20.920 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2023-10-30T11:59:20.920 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:20.921 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 37 M 2023-10-30T11:59:20.921 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 209 M 2023-10-30T11:59:20.921 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:20.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2023-10-30T11:59:21.045 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 249 kB/s | 32 kB 00:00 2023-10-30T11:59:21.079 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2023-10-30T11:59:21.112 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T11:59:21.221 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2023-10-30T11:59:21.355 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 205 kB 00:00 2023-10-30T11:59:21.421 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-10-30T11:59:21.558 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:01 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:21.655 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2023-10-30T11:59:21.672 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 876 kB/s | 217 kB 00:00 2023-10-30T11:59:21.703 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:21.704 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:21.704 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-30T11:59:21.705 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:21.705 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:21.705 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6851.g003372b1.el9 ceph 141 k 2023-10-30T11:59:21.705 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:21.705 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:21.705 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:21.706 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T11:59:21.706 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:21.706 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 141 k 2023-10-30T11:59:21.706 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 400 k 2023-10-30T11:59:21.707 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:21.806 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2023-10-30T11:59:22.020 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-18.0.0-6851.g003372 450 kB/s | 141 kB 00:00 2023-10-30T11:59:22.021 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:22.021 INFO:teuthology.orchestra.run.smithi050.stdout:Total 448 kB/s | 141 kB 00:00 2023-10-30T11:59:22.022 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:22.028 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:22.029 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:22.084 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:22.085 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:22.189 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2023-10-30T11:59:22.214 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 480 kB/s | 195 kB 00:00 2023-10-30T11:59:22.248 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:22.312 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6851.g003372b1. 1/1 2023-10-30T11:59:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6851.g003372b1. 1/1 2023-10-30T11:59:22.338 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:22.338 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-30T11:59:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:22.390 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 5.0 MB/s | 7.4 MB 00:01 2023-10-30T11:59:22.392 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:22.393 INFO:teuthology.orchestra.run.smithi118.stdout:Total 25 MB/s | 37 MB 00:01 2023-10-30T11:59:22.393 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:22.415 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:22.415 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:22.829 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:22.829 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:23.407 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6851.g003372b1. 1/1 2023-10-30T11:59:23.407 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:23.407 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:23.408 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:23.408 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:23.408 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:23.614 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T11:59:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T11:59:23.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2023-10-30T11:59:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T11:59:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:23.988 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T11:59:24.247 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:04 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:24.340 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:24.392 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:24.393 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T11:59:24.393 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:24.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2023-10-30T11:59:24.717 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T11:59:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:25.028 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:05 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:25.172 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:25.173 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:25.173 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:25.174 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:25.174 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:25.174 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 1.8 M 2023-10-30T11:59:25.174 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:59:25.174 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 23 k 2023-10-30T11:59:25.175 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 15 k 2023-10-30T11:59:25.175 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-30T11:59:25.175 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-30T11:59:25.175 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-30T11:59:25.175 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2023-10-30T11:59:25.176 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-30T11:59:25.176 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:25.176 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:25.176 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:25.176 INFO:teuthology.orchestra.run.smithi050.stdout:Install 7 Packages 2023-10-30T11:59:25.177 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:25.177 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.1 M 2023-10-30T11:59:25.177 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 8.6 M 2023-10-30T11:59:25.178 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:25.532 INFO:teuthology.orchestra.run.smithi050.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6851.g003 139 kB/s | 23 kB 00:00 2023-10-30T11:59:25.549 INFO:teuthology.orchestra.run.smithi050.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6851.g0033 82 kB/s | 15 kB 00:00 2023-10-30T11:59:25.774 INFO:teuthology.orchestra.run.smithi050.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 38 kB/s | 9.0 kB 00:00 2023-10-30T11:59:25.866 INFO:teuthology.orchestra.run.smithi050.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 129 kB/s | 41 kB 00:00 2023-10-30T11:59:25.883 INFO:teuthology.orchestra.run.smithi050.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 282 kB/s | 31 kB 00:00 2023-10-30T11:59:25.984 INFO:teuthology.orchestra.run.smithi050.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6851.g003372b1 2.9 MB/s | 1.8 MB 00:00 2023-10-30T11:59:26.084 INFO:teuthology.orchestra.run.smithi050.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 868 kB/s | 188 kB 00:00 2023-10-30T11:59:26.087 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:26.087 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.3 MB/s | 2.1 MB 00:00 2023-10-30T11:59:26.088 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:26.096 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:26.097 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:26.121 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T11:59:26.170 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:26.170 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:26.484 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:26.552 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-30T11:59:26.586 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-30T11:59:26.634 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-30T11:59:26.664 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-30T11:59:26.685 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6851.g003372b1.el9.n 5/7 2023-10-30T11:59:26.929 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6851.g003372b1.el9. 6/7 2023-10-30T11:59:26.962 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarc 7/7 2023-10-30T11:59:27.150 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T11:59:27.181 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6851.g00337 12/12 2023-10-30T11:59:27.540 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarc 7/7 2023-10-30T11:59:27.541 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6851.g003372b1.el9. 1/7 2023-10-30T11:59:27.541 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarc 2/7 2023-10-30T11:59:27.541 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6851.g003372b1.el9.n 3/7 2023-10-30T11:59:27.541 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-30T11:59:27.541 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-30T11:59:27.542 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-30T11:59:27.856 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-30T11:59:27.857 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:27.857 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:27.857 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:27.857 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:27.857 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:27.858 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-30T11:59:27.858 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-30T11:59:27.858 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-30T11:59:27.858 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-30T11:59:27.858 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:27.859 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:28.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-30T11:59:28.634 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:08 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:28.780 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:28.781 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-30T11:59:28.781 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:28.781 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-30T11:59:28.782 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 7.4 M 2023-10-30T11:59:28.782 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:59:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-30T11:59:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-30T11:59:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-30T11:59:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-30T11:59:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-30T11:59:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-30T11:59:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-30T11:59:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-30T11:59:28.784 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-30T11:59:28.784 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-30T11:59:28.784 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-30T11:59:28.784 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:28.784 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:28.784 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-30T11:59:28.785 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2023-10-30T11:59:28.785 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:28.785 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 37 M 2023-10-30T11:59:28.786 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 209 M 2023-10-30T11:59:28.786 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:28.915 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 241 kB/s | 32 kB 00:00 2023-10-30T11:59:28.949 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 531 kB/s | 17 kB 00:00 2023-10-30T11:59:28.982 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-30T11:59:29.041 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-30T11:59:29.108 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.0 MB/s | 205 kB 00:00 2023-10-30T11:59:29.175 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-10-30T11:59:29.259 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2023-10-30T11:59:29.284 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2023-10-30T11:59:29.359 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 11 MB/s | 803 kB 00:00 2023-10-30T11:59:29.526 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-10-30T11:59:29.551 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-10-30T11:59:29.868 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 6.8 MB/s | 7.4 MB 00:01 2023-10-30T11:59:29.870 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:29.871 INFO:teuthology.orchestra.run.smithi050.stdout:Total 34 MB/s | 37 MB 00:01 2023-10-30T11:59:29.871 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:29.894 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:29.895 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:29.943 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6851.g00337 12/12 2023-10-30T11:59:29.943 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6851.g00337 1/12 2023-10-30T11:59:29.944 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T11:59:29.944 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:29.944 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T11:59:29.944 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:29.945 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:29.945 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T11:59:29.945 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T11:59:29.945 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:29.945 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T11:59:29.946 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T11:59:30.345 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:30.346 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:30.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T11:59:30.431 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:30.431 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:30.431 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:30.431 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T11:59:30.432 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T11:59:30.432 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T11:59:30.432 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:30.432 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:30.432 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T11:59:30.433 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T11:59:30.433 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T11:59:30.433 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:30.433 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:30.433 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T11:59:30.434 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:30.434 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:30.740 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2023-10-30T11:59:31.067 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-30T11:59:31.206 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-30T11:59:31.290 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:24 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:31.293 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:31.335 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-30T11:59:31.431 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:31.432 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2023-10-30T11:59:31.432 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:59:31.432 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2023-10-30T11:59:31.432 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:31.433 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 48 k 2023-10-30T11:59:31.433 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:59:31.433 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T11:59:31.433 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T11:59:31.433 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T11:59:31.434 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T11:59:31.434 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T11:59:31.434 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T11:59:31.434 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T11:59:31.435 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T11:59:31.435 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T11:59:31.435 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T11:59:31.435 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T11:59:31.435 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T11:59:31.436 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:31.436 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:31.436 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2023-10-30T11:59:31.436 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2023-10-30T11:59:31.436 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:31.437 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.0 M 2023-10-30T11:59:31.437 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2023-10-30T11:59:31.438 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:31.534 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:31.566 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-30T11:59:31.924 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 24 kB 00:00 2023-10-30T11:59:31.949 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2023-10-30T11:59:31.991 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-10-30T11:59:32.016 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-30T11:59:32.041 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T11:59:32.067 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-10-30T11:59:32.209 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2023-10-30T11:59:32.234 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): ceph-mgr-rook-18.0.0-6851.g003372b1.el9 111 kB/s | 48 kB 00:00 2023-10-30T11:59:32.259 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2023-10-30T11:59:32.301 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-google-auth-2.23.3-1.el9.noarc 2.1 MB/s | 197 kB 00:00 2023-10-30T11:59:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-30T11:59:32.368 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.0 MB/s | 90 kB 00:00 2023-10-30T11:59:32.440 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:32.470 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 280 kB/s | 59 kB 00:00 2023-10-30T11:59:32.521 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2023-10-30T11:59:32.525 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:32.525 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2023-10-30T11:59:32.525 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:32.538 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:32.538 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:32.634 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:32.635 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:33.181 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:33.342 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T11:59:33.397 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T11:59:33.452 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T11:59:33.489 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T11:59:33.531 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T11:59:33.624 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T11:59:33.742 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T11:59:33.748 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-30T11:59:33.799 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T11:59:34.522 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T11:59:34.599 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T11:59:34.631 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T11:59:34.676 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T11:59:34.700 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 13/13 2023-10-30T11:59:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-30T11:59:34.849 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6851.g00337 12/12 2023-10-30T11:59:35.702 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 13/13 2023-10-30T11:59:35.703 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 1/13 2023-10-30T11:59:35.703 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T11:59:35.703 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T11:59:35.703 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T11:59:35.704 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T11:59:35.704 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T11:59:35.704 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T11:59:35.704 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T11:59:35.704 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T11:59:35.705 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T11:59:35.705 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T11:59:35.705 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T11:59:36.086 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T11:59:36.087 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:36.087 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:36.087 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:36.087 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T11:59:36.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T11:59:36.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T11:59:36.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T11:59:36.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T11:59:36.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T11:59:36.088 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T11:59:36.089 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T11:59:36.089 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T11:59:36.089 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T11:59:36.089 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T11:59:36.089 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T11:59:36.090 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:36.090 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:36.327 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T11:59:36.868 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:29 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:37.007 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:37.008 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2023-10-30T11:59:37.008 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:59:37.008 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2023-10-30T11:59:37.008 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:37.008 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 132 k 2023-10-30T11:59:37.009 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:59:37.009 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T11:59:37.009 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T11:59:37.009 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T11:59:37.009 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T11:59:37.010 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T11:59:37.010 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:37.010 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:37.010 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2023-10-30T11:59:37.010 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2023-10-30T11:59:37.011 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:37.011 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 6.8 M 2023-10-30T11:59:37.011 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 33 M 2023-10-30T11:59:37.012 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:37.216 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-10-30T11:59:37.350 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-10-30T11:59:37.375 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6851.g003372b1.e 423 kB/s | 132 kB 00:00 2023-10-30T11:59:37.453 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6851.g00337 12/12 2023-10-30T11:59:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6851.g00337 1/12 2023-10-30T11:59:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-30T11:59:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-30T11:59:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-30T11:59:37.455 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-30T11:59:37.455 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-30T11:59:37.455 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-30T11:59:37.455 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-30T11:59:37.455 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-30T11:59:37.455 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-30T11:59:37.456 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-30T11:59:37.559 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2023-10-30T11:59:37.576 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 257 kB/s | 58 kB 00:00 2023-10-30T11:59:37.592 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 235 kB/s | 51 kB 00:00 2023-10-30T11:59:37.596 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:37.596 INFO:teuthology.orchestra.run.smithi118.stdout:Total 12 MB/s | 6.8 MB 00:00 2023-10-30T11:59:37.596 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:37.605 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:37.605 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:37.693 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:37.693 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:37.897 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-30T11:59:37.898 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:37.898 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:37.898 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:37.898 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-30T11:59:37.899 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-30T11:59:37.899 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-30T11:59:37.899 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:37.899 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-30T11:59:37.899 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-30T11:59:37.900 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-30T11:59:37.900 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-30T11:59:37.900 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:37.900 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-30T11:59:37.900 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-30T11:59:37.900 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:37.901 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:37.986 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:38.149 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T11:59:38.194 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T11:59:38.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2023-10-30T11:59:38.711 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T11:59:38.797 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T11:59:38.811 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:18 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:38.858 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T11:59:38.882 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 6/6 2023-10-30T11:59:38.957 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:38.958 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-30T11:59:38.959 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:38.959 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-30T11:59:38.959 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 48 k 2023-10-30T11:59:38.960 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:59:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-30T11:59:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-30T11:59:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-30T11:59:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-30T11:59:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-30T11:59:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-30T11:59:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-30T11:59:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-30T11:59:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-30T11:59:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-30T11:59:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-30T11:59:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-30T11:59:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:38.964 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:38.964 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-30T11:59:38.964 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2023-10-30T11:59:38.964 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:38.965 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.0 M 2023-10-30T11:59:38.966 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2023-10-30T11:59:38.966 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:39.159 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 24 kB 00:00 2023-10-30T11:59:39.184 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2023-10-30T11:59:39.226 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-10-30T11:59:39.251 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2023-10-30T11:59:39.276 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-30T11:59:39.302 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 851 kB/s | 43 kB 00:00 2023-10-30T11:59:39.496 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): ceph-mgr-rook-18.0.0-6851.g003372b1.el9 104 kB/s | 48 kB 00:00 2023-10-30T11:59:39.702 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2023-10-30T11:59:39.763 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 6/6 2023-10-30T11:59:39.764 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/6 2023-10-30T11:59:39.764 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T11:59:39.764 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T11:59:39.764 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T11:59:39.764 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T11:59:39.794 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 62 kB/s | 32 kB 00:00 2023-10-30T11:59:40.020 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 263 kB/s | 59 kB 00:00 2023-10-30T11:59:40.068 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T11:59:40.069 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:40.069 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:40.069 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:40.069 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T11:59:40.069 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T11:59:40.070 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T11:59:40.070 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T11:59:40.070 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T11:59:40.070 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:40.070 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:40.195 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 282 kB/s | 197 kB 00:00 2023-10-30T11:59:40.220 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 448 kB/s | 90 kB 00:00 2023-10-30T11:59:40.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2023-10-30T11:59:40.370 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-10-30T11:59:40.376 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:40.376 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-10-30T11:59:40.376 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:40.392 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:40.392 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:40.507 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:40.509 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:40.859 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:33 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:40.998 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:40.999 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:40.999 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:59:40.999 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:41.000 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:41.000 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:18.0.0-6851.g003372b1.el9 ceph 856 k 2023-10-30T11:59:41.000 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2023-10-30T11:59:41.000 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T11:59:41.001 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:41.001 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:41.001 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:41.001 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2023-10-30T11:59:41.001 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:41.002 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 934 k 2023-10-30T11:59:41.002 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 2.7 M 2023-10-30T11:59:41.002 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:41.112 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 703 kB/s | 79 kB 00:00 2023-10-30T11:59:41.170 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:41.342 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-30T11:59:41.473 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-30T11:59:41.529 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-30T11:59:41.564 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-30T11:59:41.607 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-30T11:59:41.710 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-30T11:59:41.813 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-30T11:59:41.867 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-30T11:59:42.187 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-fuse-18.0.0-6851.g003372b1.el9.x86_ 721 kB/s | 856 kB 00:01 2023-10-30T11:59:42.188 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:42.189 INFO:teuthology.orchestra.run.smithi118.stdout:Total 785 kB/s | 934 kB 00:01 2023-10-30T11:59:42.189 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:42.210 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:42.210 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:42.286 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:42.287 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:42.480 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:42.618 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T11:59:42.634 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-30T11:59:42.679 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-30T11:59:42.686 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 2/2 2023-10-30T11:59:42.712 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-30T11:59:42.759 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-30T11:59:42.782 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 13/13 2023-10-30T11:59:43.402 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 2/2 2023-10-30T11:59:43.403 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/2 2023-10-30T11:59:43.591 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 13/13 2023-10-30T11:59:43.591 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 1/13 2023-10-30T11:59:43.592 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-30T11:59:43.592 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-30T11:59:43.592 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-30T11:59:43.592 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-30T11:59:43.592 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-30T11:59:43.592 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-30T11:59:43.593 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-30T11:59:43.593 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-30T11:59:43.593 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-30T11:59:43.593 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-30T11:59:43.593 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-30T11:59:43.693 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T11:59:43.694 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:43.694 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:43.694 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T11:59:43.694 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:43.694 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:43.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-volume 2023-10-30T11:59:43.972 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-30T11:59:43.973 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:43.973 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:43.973 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:43.973 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-30T11:59:43.973 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-30T11:59:43.974 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-30T11:59:43.974 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-30T11:59:43.974 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-30T11:59:43.974 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-30T11:59:43.974 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-30T11:59:43.975 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-30T11:59:43.975 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-30T11:59:43.975 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-30T11:59:43.975 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-30T11:59:43.975 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-30T11:59:43.976 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:43.976 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:44.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2023-10-30T11:59:44.490 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:44.637 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:44.638 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2023-10-30T11:59:44.638 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:44.638 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 262 k 2023-10-30T11:59:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:44.639 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:44.639 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:44.639 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T11:59:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:44.640 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 262 k 2023-10-30T11:59:44.640 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 1.2 M 2023-10-30T11:59:44.640 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:44.777 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:24 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:44.925 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:44.925 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-30T11:59:44.926 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:44.926 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-30T11:59:44.926 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:44.926 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 132 k 2023-10-30T11:59:44.926 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:59:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-30T11:59:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-30T11:59:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-30T11:59:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-30T11:59:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-30T11:59:44.928 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:44.928 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:44.928 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-30T11:59:44.928 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-10-30T11:59:44.928 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:44.929 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 6.8 M 2023-10-30T11:59:44.929 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2023-10-30T11:59:44.929 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:44.985 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-volume-18.0.0-6851.g003372b1.el9.noarch.rp 757 kB/s | 262 kB 00:00 2023-10-30T11:59:44.985 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:44.986 INFO:teuthology.orchestra.run.smithi118.stdout:Total 753 kB/s | 262 kB 00:00 2023-10-30T11:59:44.987 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:44.994 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:44.995 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:45.030 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:45.031 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:45.138 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-30T11:59:45.244 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:45.280 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-10-30T11:59:45.303 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:45.320 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:45.320 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:45.321 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T11:59:45.321 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:45.480 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 291 kB/s | 58 kB 00:00 2023-10-30T11:59:45.497 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): ceph-mgr-cephadm-18.0.0-6851.g003372b1.e 262 kB/s | 132 kB 00:00 2023-10-30T11:59:45.522 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2023-10-30T11:59:45.547 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2023-10-30T11:59:45.551 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:45.551 INFO:teuthology.orchestra.run.smithi050.stdout:Total 11 MB/s | 6.8 MB 00:00 2023-10-30T11:59:45.552 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:45.562 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:45.562 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:45.658 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:45.658 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:46.136 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-30T11:59:46.200 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-30T11:59:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:46.416 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:46.417 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-30T11:59:46.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2023-10-30T11:59:46.845 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-30T11:59:46.907 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-30T11:59:46.936 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 6/6 2023-10-30T11:59:47.317 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:40 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:47.459 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:47.459 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-30T11:59:47.460 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:47.460 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:18.0.0-6851.g003372b1.el9 ceph 126 k 2023-10-30T11:59:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:47.461 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:47.461 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:47.461 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T11:59:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:47.461 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 126 k 2023-10-30T11:59:47.462 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 456 k 2023-10-30T11:59:47.462 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:47.745 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 6/6 2023-10-30T11:59:47.745 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 1/6 2023-10-30T11:59:47.745 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-30T11:59:47.746 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-30T11:59:47.746 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-30T11:59:47.746 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-30T11:59:47.825 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-18.0.0-6851.g003372b1.el9.x86_64 347 kB/s | 126 kB 00:00 2023-10-30T11:59:47.826 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:47.826 INFO:teuthology.orchestra.run.smithi118.stdout:Total 346 kB/s | 126 kB 00:00 2023-10-30T11:59:47.826 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:47.831 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:47.831 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:47.887 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:47.887 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:48.032 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:48.052 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-30T11:59:48.053 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:48.053 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:48.053 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:48.053 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-30T11:59:48.053 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-30T11:59:48.054 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-30T11:59:48.054 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-30T11:59:48.054 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-30T11:59:48.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:48.054 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:48.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2023-10-30T11:59:48.579 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:48.875 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:48.875 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:48.875 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:48.876 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:48.876 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:48.876 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:48.893 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:28 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:49.037 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:49.039 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:49.039 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:49.039 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:49.040 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:49.040 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:18.0.0-6851.g003372b1.el9 ceph 856 k 2023-10-30T11:59:49.040 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-30T11:59:49.040 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-30T11:59:49.040 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:49.041 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:49.041 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:49.041 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2023-10-30T11:59:49.041 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:49.041 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 934 k 2023-10-30T11:59:49.042 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.7 M 2023-10-30T11:59:49.042 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:49.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2023-10-30T11:59:49.144 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 775 kB/s | 79 kB 00:00 2023-10-30T11:59:49.698 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:49.790 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:49.840 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:49.841 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:49.841 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:49.875 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-18.0.0-6851.g003372b1.el9.x86_ 1.0 MB/s | 856 kB 00:00 2023-10-30T11:59:49.876 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:49.877 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.1 MB/s | 934 kB 00:00 2023-10-30T11:59:49.877 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:49.900 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:49.901 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:49.904 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2023-10-30T11:59:49.982 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:49.983 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:50.189 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-30T11:59:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 2/2 2023-10-30T11:59:50.445 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:43 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:50.586 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:50.587 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:50.587 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-30T11:59:50.587 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:50.588 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:50.588 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:18.0.0-6851.g003372b1.el9 ceph 31 k 2023-10-30T11:59:50.588 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:50.588 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:50.588 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:50.589 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T11:59:50.589 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:50.589 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 31 k 2023-10-30T11:59:50.589 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 139 k 2023-10-30T11:59:50.589 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:50.796 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-18.0.0-6851.g003372b1.el9.x86_6 149 kB/s | 31 kB 00:00 2023-10-30T11:59:50.797 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:50.797 INFO:teuthology.orchestra.run.smithi118.stdout:Total 148 kB/s | 31 kB 00:00 2023-10-30T11:59:50.797 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:50.800 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:50.801 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:50.823 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:50.823 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:50.905 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:51.029 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 2/2 2023-10-30T11:59:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/2 2023-10-30T11:59:51.416 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-30T11:59:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:51.487 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-30T11:59:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:51.487 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:51.689 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:51.690 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:51.690 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:51.690 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:51.690 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:51.691 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:51.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2023-10-30T11:59:51.869 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2023-10-30T11:59:52.313 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:32 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:52.410 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:45 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:52.460 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-30T11:59:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:18.0.0-6851.g003372b1.el9 ceph-noarch 262 k 2023-10-30T11:59:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:52.462 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:52.462 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:52.462 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T11:59:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:52.463 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 262 k 2023-10-30T11:59:52.463 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.2 M 2023-10-30T11:59:52.463 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:52.501 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:52.551 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:52.552 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:52.552 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:52.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2023-10-30T11:59:52.871 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-volume-18.0.0-6851.g003372b1.el9.noarch.rp 640 kB/s | 262 kB 00:00 2023-10-30T11:59:52.872 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:52.872 INFO:teuthology.orchestra.run.smithi050.stdout:Total 638 kB/s | 262 kB 00:00 2023-10-30T11:59:52.872 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:52.881 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:52.881 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:52.917 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:52.918 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:53.142 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:53.168 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:46 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:53.222 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:53.223 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T11:59:53.223 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-30T11:59:53.223 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:53.260 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:53.310 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:53.310 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:53.311 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:53.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2023-10-30T11:59:53.925 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:46 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:54.016 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:54.065 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:54.065 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:54.066 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:54.130 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2023-10-30T11:59:54.225 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 1/1 2023-10-30T11:59:54.226 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:54.226 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:54.226 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:18.0.0-6851.g003372b1.el9.noarch 2023-10-30T11:59:54.227 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:54.227 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:54.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2023-10-30T11:59:54.676 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:47 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:54.767 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:54.816 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:54.817 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:54.817 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:54.894 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2023-10-30T11:59:55.116 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:35 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:55.263 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:55.263 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-30T11:59:55.263 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:55.264 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:18.0.0-6851.g003372b1.el9 ceph 126 k 2023-10-30T11:59:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:55.264 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:55.265 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:55.265 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T11:59:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:55.265 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 126 k 2023-10-30T11:59:55.265 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2023-10-30T11:59:55.266 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:55.432 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:48 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:55.510 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-18.0.0-6851.g003372b1.el9.x86_64 513 kB/s | 126 kB 00:00 2023-10-30T11:59:55.511 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:55.511 INFO:teuthology.orchestra.run.smithi050.stdout:Total 511 kB/s | 126 kB 00:00 2023-10-30T11:59:55.511 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:55.517 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:55.517 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:55.523 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:55.573 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:55.574 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:55.574 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:55.576 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:55.577 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:55.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2023-10-30T11:59:55.731 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:55.844 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:56.194 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:49 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:56.277 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:56.285 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:56.335 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:56.335 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-30T11:59:56.336 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:56.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2023-10-30T11:59:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:56.601 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:56.602 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:56.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2023-10-30T11:59:56.952 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:49 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:57.094 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:57.095 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:57.095 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2023-10-30T11:59:57.095 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:57.095 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:57.095 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:18.0.0-6851.g003372b1.el9 ceph 85 k 2023-10-30T11:59:57.096 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:57.096 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:57.096 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:57.096 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T11:59:57.096 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:57.097 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 85 k 2023-10-30T11:59:57.097 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 227 k 2023-10-30T11:59:57.097 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:57.333 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-18.0.0-6851.g003372b1.el9.x86_64.rpm 357 kB/s | 85 kB 00:00 2023-10-30T11:59:57.334 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:57.334 INFO:teuthology.orchestra.run.smithi118.stdout:Total 355 kB/s | 85 kB 00:00 2023-10-30T11:59:57.335 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T11:59:57.341 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T11:59:57.341 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T11:59:57.395 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T11:59:57.395 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T11:59:57.431 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:37 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:57.525 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T11:59:57.549 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T11:59:57.578 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:57.579 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T11:59:57.579 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:57.630 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:57.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2023-10-30T11:59:58.032 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:58.213 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:38 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T11:59:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:58.334 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T11:59:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:58.335 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:58.335 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T11:59:58.360 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T11:59:58.361 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:58.362 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-30T11:59:58.362 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:58.362 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T11:59:58.363 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:18.0.0-6851.g003372b1.el9 ceph 31 k 2023-10-30T11:59:58.363 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:58.363 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T11:59:58.363 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T11:59:58.364 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T11:59:58.364 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:58.364 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 k 2023-10-30T11:59:58.364 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 139 k 2023-10-30T11:59:58.364 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T11:59:58.531 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-18.0.0-6851.g003372b1.el9.x86_6 184 kB/s | 31 kB 00:00 2023-10-30T11:59:58.531 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T11:59:58.534 INFO:teuthology.orchestra.run.smithi050.stdout:Total 183 kB/s | 31 kB 00:00 2023-10-30T11:59:58.534 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T11:59:58.535 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T11:59:58.535 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T11:59:58.558 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T11:59:58.559 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T11:59:58.569 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2023-10-30T11:59:58.653 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T11:59:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:59.114 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:52 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T11:59:59.190 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:59.256 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T11:59:59.257 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:59.257 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2023-10-30T11:59:59.257 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:59.257 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T11:59:59.258 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:18.0.0-6851.g003372b1.el9 ceph 3.0 M 2023-10-30T11:59:59.258 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:59.258 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T11:59:59.258 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T11:59:59.258 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T11:59:59.259 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T11:59:59.259 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.0 M 2023-10-30T11:59:59.259 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 12 M 2023-10-30T11:59:59.259 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T11:59:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T11:59:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:59.478 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T11:59:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T11:59:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T11:59:59.479 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T11:59:59.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2023-10-30T12:00:00.249 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:40 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:00.343 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T12:00:00.396 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:00.397 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T12:00:00.397 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:00.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2023-10-30T12:00:00.979 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-18.0.0-6851.g003372b1.el9.x86_64.rpm 1.8 MB/s | 3.0 MB 00:01 2023-10-30T12:00:00.980 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:00.980 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.8 MB/s | 3.0 MB 00:01 2023-10-30T12:00:00.980 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T12:00:00.990 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T12:00:00.990 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T12:00:01.050 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:01.063 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T12:00:01.064 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T12:00:01.144 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T12:00:01.197 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:01.198 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T12:00:01.198 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:01.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2023-10-30T12:00:01.474 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T12:00:01.546 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:01.573 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:01.573 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:00:01.573 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T12:00:01.574 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:00:01.574 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:00:01.574 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:01.841 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:41 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:01.935 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T12:00:01.988 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:01.989 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T12:00:01.989 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:02.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2023-10-30T12:00:02.629 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:42 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:02.719 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T12:00:02.774 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:02.775 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T12:00:02.775 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:02.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2023-10-30T12:00:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:02.889 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T12:00:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T12:00:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:02.889 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T12:00:03.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2023-10-30T12:00:03.403 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:43 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:03.498 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T12:00:03.550 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:03.551 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T12:00:03.551 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:03.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2023-10-30T12:00:03.670 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:56 ago on Mon 30 Oct 2023 11:58:07 AM UTC. 2023-10-30T12:00:03.810 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-30T12:00:03.811 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T12:00:03.812 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2023-10-30T12:00:03.812 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T12:00:03.812 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2023-10-30T12:00:03.812 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:18.0.0-6851.g003372b1.el9 ceph 169 k 2023-10-30T12:00:03.812 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:03.813 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2023-10-30T12:00:03.813 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2023-10-30T12:00:03.813 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2023-10-30T12:00:03.813 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:03.813 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 169 k 2023-10-30T12:00:03.814 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 490 k 2023-10-30T12:00:03.814 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2023-10-30T12:00:04.190 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:44 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:04.285 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:18.0.0-6851.g003372b1.el9.x86_64 is already installed. 2023-10-30T12:00:04.337 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:04.338 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-30T12:00:04.339 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:04.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2023-10-30T12:00:04.416 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-18.0.0-6851.g003372b1.el9.x86_64.rpm 280 kB/s | 169 kB 00:00 2023-10-30T12:00:04.417 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:04.418 INFO:teuthology.orchestra.run.smithi118.stdout:Total 279 kB/s | 169 kB 00:00 2023-10-30T12:00:04.418 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2023-10-30T12:00:04.425 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2023-10-30T12:00:04.425 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2023-10-30T12:00:04.481 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2023-10-30T12:00:04.481 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2023-10-30T12:00:04.627 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2023-10-30T12:00:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:04.981 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:44 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:05.129 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:05.131 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:05.131 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-30T12:00:05.131 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:05.132 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T12:00:05.132 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:18.0.0-6851.g003372b1.el9 ceph 85 k 2023-10-30T12:00:05.132 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T12:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T12:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 85 k 2023-10-30T12:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 227 k 2023-10-30T12:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T12:00:05.162 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:05.484 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:05.485 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:05.485 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2023-10-30T12:00:05.486 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T12:00:05.486 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:05.486 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-30T12:00:05.567 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-18.0.0-6851.g003372b1.el9.x86_64.rpm 195 kB/s | 85 kB 00:00 2023-10-30T12:00:05.567 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:05.568 INFO:teuthology.orchestra.run.smithi050.stdout:Total 194 kB/s | 85 kB 00:00 2023-10-30T12:00:05.568 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T12:00:05.575 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T12:00:05.575 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T12:00:05.634 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T12:00:05.635 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T12:00:05.698 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T12:00:05.894 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:06.267 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:06.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:06.555 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:06.555 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T12:00:06.555 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T12:00:06.555 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:06.556 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:06.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2023-10-30T12:00:07.330 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:47 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:07.478 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:07.479 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:07.479 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-30T12:00:07.479 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:07.479 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T12:00:07.479 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:18.0.0-6851.g003372b1.el9 ceph 3.0 M 2023-10-30T12:00:07.480 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:07.480 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T12:00:07.480 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:07.480 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T12:00:07.481 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:07.481 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.0 M 2023-10-30T12:00:07.481 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 12 M 2023-10-30T12:00:07.481 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T12:00:09.727 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-18.0.0-6851.g003372b1.el9.x86_64.rpm 1.3 MB/s | 3.0 MB 00:02 2023-10-30T12:00:09.728 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:09.729 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.3 MB/s | 3.0 MB 00:02 2023-10-30T12:00:09.729 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T12:00:09.741 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T12:00:09.741 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T12:00:09.820 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T12:00:09.821 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T12:00:10.257 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T12:00:10.328 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:10.353 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:10.353 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-30T12:00:10.353 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-30T12:00:10.354 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:00:10.354 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-30T12:00:10.354 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:11.594 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:11.594 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:11.594 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T12:00:11.594 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T12:00:11.594 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:11.595 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:11.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2023-10-30T12:00:12.444 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:52 ago on Mon 30 Oct 2023 11:58:20 AM UTC. 2023-10-30T12:00:12.591 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-30T12:00:12.592 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:12.592 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-30T12:00:12.592 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:12.592 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-30T12:00:12.592 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:18.0.0-6851.g003372b1.el9 ceph 169 k 2023-10-30T12:00:12.593 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:12.593 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-30T12:00:12.593 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-30T12:00:12.593 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-30T12:00:12.593 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:12.593 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 169 k 2023-10-30T12:00:12.594 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 490 k 2023-10-30T12:00:12.594 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-30T12:00:12.923 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-18.0.0-6851.g003372b1.el9.x86_64.rpm 511 kB/s | 169 kB 00:00 2023-10-30T12:00:12.924 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-30T12:00:12.924 INFO:teuthology.orchestra.run.smithi050.stdout:Total 509 kB/s | 169 kB 00:00 2023-10-30T12:00:12.924 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-30T12:00:12.932 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-30T12:00:12.932 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-30T12:00:12.993 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-30T12:00:12.993 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-30T12:00:13.142 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-30T12:00:13.232 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:13.707 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:14.016 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 1/1 2023-10-30T12:00:14.016 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:14.017 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-30T12:00:14.017 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:18.0.0-6851.g003372b1.el9.x86_64 2023-10-30T12:00:14.017 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:14.017 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-30T12:00:14.244 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:14.245 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=003372b105ed2deff4dd01cf2ce736f137365e86 2023-10-30T12:00:14.478 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T12:00:14.499 INFO:teuthology.orchestra.run.smithi050.stdout:18.0.0-6851.g003372b1.el9 2023-10-30T12:00:14.500 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6851.g003372b1.el9 2023-10-30T12:00:14.500 INFO:teuthology.task.install:The correct ceph version 18.0.0-6851.g003372b1 is installed. 2023-10-30T12:00:14.501 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=003372b105ed2deff4dd01cf2ce736f137365e86 2023-10-30T12:00:14.638 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-30T12:00:14.664 INFO:teuthology.orchestra.run.smithi118.stdout:18.0.0-6851.g003372b1.el9 2023-10-30T12:00:14.664 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6851.g003372b1.el9 2023-10-30T12:00:14.665 INFO:teuthology.task.install:The correct ceph version 18.0.0-6851.g003372b1 is installed. 2023-10-30T12:00:14.665 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T12:00:14.681 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:14.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:00:14.714 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:14.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:00:14.750 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T12:00:14.751 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:14.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:00:14.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:00:14.900 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:14.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:00:14.932 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:00:15.001 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T12:00:15.001 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:15.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:00:15.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:00:15.132 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:15.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:00:15.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:00:15.234 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T12:00:15.234 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:15.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:00:15.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:00:15.336 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:15.336 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:00:15.367 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:00:15.438 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T12:00:15.731 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T12:00:15.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2023-10-30T12:00:15.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2023-10-30T12:00:15.765 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T12:00:15.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:00:15.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:00:15.835 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T12:00:15.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:00:15.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:00:15.917 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T12:00:15.917 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T12:00:15.918 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989', 'branch': 'wip-vallariag-centos9-only', 'description': 'rados:monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_api_tests}', 'email': 'val.agl002@gmail.com', 'first_in_suite': False, 'job_id': '7440989', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-vallariag-centos9-only'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '003372b105ed2deff4dd01cf2ce736f137365e86'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '003372b105ed2deff4dd01cf2ce736f137365e86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-xml-scanner', 'sha1': '397e85a5f8ee6b984077e94184404370497237a9'}}, 'owner': 'scheduled_vallariag@teuthology', 'priority': 80, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 4758, 'sha1': '003372b105ed2deff4dd01cf2ce736f137365e86', 'sleep_before_teardown': 0, 'suite': 'rados:monthrash', 'suite_branch': 'wip-xml-scanner', 'suite_path': '/home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/VallariAg/ceph.git', 'suite_sha1': '397e85a5f8ee6b984077e94184404370497237a9', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/jATN70vGGBL54z9Cu8g6a1rsSO8iHlokG3RH+faOTkoEZuvVRihXNsRF42NBzTBt0Pz6tSWC90Tt7c2Ko0kA=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZKfXPJ1kTzkgnwWEh9xKGaopLAhExAfdfXrJKba4ufD5+QyJGrpSOAwDeMmT0mp2LxAeaX0erLyhdetBDjU1o='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'unittest-xml-scanner', 'teuthology_sha1': 'bef9b3ee0aeb26a72d4f9af72f05102582ed658c', 'timestamp': '2023-10-30_03:55:12', 'tube': 'smithi', 'user': 'vallariag', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T12:00:15.918 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:00:15.950 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:00:15.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:00:16.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:00:16.033 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:16.034 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:00:16.089 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:00:16.089 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2023-10-30T12:00:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:00:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:16.146 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-10-30T12:00:16.146 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:16.146 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:16.146 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-30 12:00:10.606897729 +0000 2023-10-30T12:00:16.147 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-30 11:57:18.251752501 +0000 2023-10-30T12:00:16.147 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-30 11:57:18.251752501 +0000 2023-10-30T12:00:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-30 11:57:18.251752501 +0000 2023-10-30T12:00:16.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:00:16.214 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-30T12:00:16.214 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-30T12:00:16.215 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.8388e-05 s, 5.2 MB/s 2023-10-30T12:00:16.216 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:00:16.271 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2023-10-30T12:00:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:00:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:16.327 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-10-30T12:00:16.327 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:16.327 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:16.327 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-30 12:00:10.607897704 +0000 2023-10-30T12:00:16.328 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-30 11:57:18.778738943 +0000 2023-10-30T12:00:16.328 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-30 11:57:18.778738943 +0000 2023-10-30T12:00:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-30 11:57:18.778738943 +0000 2023-10-30T12:00:16.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:00:16.395 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-30T12:00:16.396 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-30T12:00:16.396 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.13e-05 s, 6.3 MB/s 2023-10-30T12:00:16.397 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:00:16.453 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2023-10-30T12:00:16.508 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:00:16.509 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:16.509 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-10-30T12:00:16.509 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:16.509 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:16.509 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-30 12:00:10.607897704 +0000 2023-10-30T12:00:16.510 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-30 11:57:19.356724073 +0000 2023-10-30T12:00:16.510 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-30 11:57:19.356724073 +0000 2023-10-30T12:00:16.510 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-30 11:57:19.356724073 +0000 2023-10-30T12:00:16.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:00:16.577 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-30T12:00:16.578 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-30T12:00:16.578 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.3184e-05 s, 5.5 MB/s 2023-10-30T12:00:16.579 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:00:16.635 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2023-10-30T12:00:16.689 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:00:16.690 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:16.690 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-10-30T12:00:16.690 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:16.690 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:16.691 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-30 12:00:10.608897680 +0000 2023-10-30T12:00:16.691 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-30 11:57:19.901710051 +0000 2023-10-30T12:00:16.691 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-30 11:57:19.901710051 +0000 2023-10-30T12:00:16.691 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-30 11:57:19.901710051 +0000 2023-10-30T12:00:16.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:00:16.759 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-30T12:00:16.759 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-30T12:00:16.759 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.7697e-05 s, 5.8 MB/s 2023-10-30T12:00:16.760 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:00:16.816 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:16.817 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:16.817 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:00:16.835 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:00:16.836 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2023-10-30T12:00:16.893 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:00:16.893 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:16.894 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-10-30T12:00:16.894 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:16.894 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:16.894 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-30 12:00:01.804831857 +0000 2023-10-30T12:00:16.894 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-30 11:57:12.908352190 +0000 2023-10-30T12:00:16.895 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-30 11:57:12.908352190 +0000 2023-10-30T12:00:16.895 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2023-10-30 11:57:12.908352190 +0000 2023-10-30T12:00:16.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:00:16.963 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-30T12:00:16.963 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-30T12:00:16.963 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000119183 s, 4.3 MB/s 2023-10-30T12:00:16.964 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:00:17.022 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2023-10-30T12:00:17.079 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:00:17.080 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:17.080 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-10-30T12:00:17.080 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:17.081 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:17.081 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-30 12:00:01.804831857 +0000 2023-10-30T12:00:17.081 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-30 11:57:13.412340846 +0000 2023-10-30T12:00:17.081 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-30 11:57:13.412340846 +0000 2023-10-30T12:00:17.081 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2023-10-30 11:57:13.412340846 +0000 2023-10-30T12:00:17.082 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:00:17.147 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-30T12:00:17.147 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-30T12:00:17.148 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000108812 s, 4.7 MB/s 2023-10-30T12:00:17.149 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:00:17.206 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2023-10-30T12:00:17.262 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:00:17.262 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:17.262 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-30T12:00:17.263 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:17.263 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:17.263 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-30 12:00:01.805831835 +0000 2023-10-30T12:00:17.263 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-30 11:57:13.925329298 +0000 2023-10-30T12:00:17.263 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-30 11:57:13.925329298 +0000 2023-10-30T12:00:17.264 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2023-10-30 11:57:13.925329298 +0000 2023-10-30T12:00:17.264 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:00:17.328 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-30T12:00:17.328 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-30T12:00:17.328 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 7.5768e-05 s, 6.8 MB/s 2023-10-30T12:00:17.329 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:00:17.385 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2023-10-30T12:00:17.440 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:00:17.440 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:00:17.441 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-10-30T12:00:17.441 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:00:17.441 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2023-10-30T12:00:17.441 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-30 12:00:01.805831835 +0000 2023-10-30T12:00:17.441 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-30 11:57:14.412318336 +0000 2023-10-30T12:00:17.441 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-30 11:57:14.412318336 +0000 2023-10-30T12:00:17.442 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2023-10-30 11:57:14.412318336 +0000 2023-10-30T12:00:17.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:00:17.506 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-30T12:00:17.507 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-30T12:00:17.507 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000122481 s, 4.2 MB/s 2023-10-30T12:00:17.508 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:00:17.564 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:17.564 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-10-30T12:00:17.565 INFO:tasks.ceph:Generating config... 2023-10-30T12:00:17.587 INFO:tasks.ceph:[client] debug monc = 20 2023-10-30T12:00:17.587 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T12:00:17.588 INFO:tasks.ceph:[global] debug ms = 1 2023-10-30T12:00:17.588 INFO:tasks.ceph:[global] debug objecter = 20 2023-10-30T12:00:17.588 INFO:tasks.ceph:[global] debug rados = 20 2023-10-30T12:00:17.588 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-30T12:00:17.589 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-10-30T12:00:17.589 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-30T12:00:17.589 INFO:tasks.ceph:[global] ms type = async 2023-10-30T12:00:17.590 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T12:00:17.590 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T12:00:17.591 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T12:00:17.591 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T12:00:17.591 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T12:00:17.591 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2023-10-30T12:00:17.592 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-30T12:00:17.592 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-30T12:00:17.592 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-30T12:00:17.592 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-10-30T12:00:17.593 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-10-30T12:00:17.593 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-10-30T12:00:17.593 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-10-30T12:00:17.594 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-10-30T12:00:17.594 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T12:00:17.594 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T12:00:17.594 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T12:00:17.595 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T12:00:17.595 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-10-30T12:00:17.595 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-30T12:00:17.595 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-30T12:00:17.596 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T12:00:17.596 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T12:00:17.596 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-30T12:00:17.597 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T12:00:17.597 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T12:00:17.597 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T12:00:17.597 INFO:tasks.ceph:[osd] osd class default list = * 2023-10-30T12:00:17.597 INFO:tasks.ceph:[osd] osd class load list = * 2023-10-30T12:00:17.598 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-10-30T12:00:17.598 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-10-30T12:00:17.598 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T12:00:17.599 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-10-30T12:00:17.599 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-10-30T12:00:17.599 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-10-30T12:00:17.599 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T12:00:17.600 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-10-30T12:00:17.600 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-10-30T12:00:17.600 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T12:00:17.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T12:00:17.664 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T12:00:17.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T12:00:17.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:00:17.750 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.50'), ('mon.c', '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]'), ('mon.b', '172.21.15.118')] 2023-10-30T12:00:17.751 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],172.21.15.118', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-10-30T12:00:17.752 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:17.752 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:00:17.808 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --addv c '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]' --add b 172.21.15.118 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:17.895 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:17.895 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:17.895 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = pacific 2023-10-30T12:00:17.896 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2023-10-30T12:00:17.896 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:17.896 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:17.896 INFO:teuthology.orchestra.run.smithi050.stdout:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:17.896 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 16 (pacific) 2023-10-30T12:00:17.897 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2023-10-30T12:00:17.897 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:17.897 INFO:teuthology.orchestra.run.smithi050.stdout:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:17.897 INFO:teuthology.orchestra.run.smithi050.stdout:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:17.897 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T12:00:17.899 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:00:17.915 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 87b22564-9727-4955-ac7b-d9a79464690c... 2023-10-30T12:00:17.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:00:17.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:00:18.010 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2023-10-30T12:00:18.011 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2023-10-30T12:00:18.011 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:00:18.011 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2023-10-30T12:00:18.011 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.011 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2023-10-30T12:00:18.012 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.012 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2023-10-30T12:00:18.012 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.012 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2023-10-30T12:00:18.012 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2023-10-30T12:00:18.013 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.013 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2023-10-30T12:00:18.013 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2023-10-30T12:00:18.013 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.013 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:00:18.014 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2023-10-30T12:00:18.014 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.014 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:00:18.014 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:00:18.014 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.014 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2023-10-30T12:00:18.015 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.015 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2023-10-30T12:00:18.015 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2023-10-30T12:00:18.015 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:00:18.015 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:00:18.016 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2023-10-30T12:00:18.016 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:00:18.016 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2023-10-30T12:00:18.016 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:00:18.016 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:00:18.016 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2023-10-30T12:00:18.017 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.017 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:00:18.017 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.017 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2023-10-30T12:00:18.017 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.018 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2023-10-30T12:00:18.018 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.018 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2023-10-30T12:00:18.018 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2023-10-30T12:00:18.018 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2023-10-30T12:00:18.019 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = "172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],172.21.15.118" 2023-10-30T12:00:18.019 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-30T12:00:18.019 INFO:teuthology.orchestra.run.smithi118.stdout: debug objecter = 20 2023-10-30T12:00:18.019 INFO:teuthology.orchestra.run.smithi118.stdout: debug rados = 20 2023-10-30T12:00:18.019 INFO:teuthology.orchestra.run.smithi118.stdout: mon client directed command retry = 5 2023-10-30T12:00:18.019 INFO:teuthology.orchestra.run.smithi118.stdout: mon election default strategy = 1 2023-10-30T12:00:18.020 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject socket failures = 5000 2023-10-30T12:00:18.020 INFO:teuthology.orchestra.run.smithi118.stdout: ms type = async 2023-10-30T12:00:18.020 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:18.020 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.020 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2023-10-30T12:00:18.020 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2023-10-30T12:00:18.021 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.021 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2023-10-30T12:00:18.021 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2023-10-30T12:00:18.021 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:00:18.021 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.022 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2023-10-30T12:00:18.022 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2023-10-30T12:00:18.022 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.022 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2023-10-30T12:00:18.022 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2023-10-30T12:00:18.023 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2023-10-30T12:00:18.023 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.023 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2023-10-30T12:00:18.023 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2023-10-30T12:00:18.023 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.023 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:00:18.024 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.024 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2023-10-30T12:00:18.024 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.024 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2023-10-30T12:00:18.024 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.025 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:00:18.025 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2023-10-30T12:00:18.025 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.025 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2023-10-30T12:00:18.025 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2023-10-30T12:00:18.025 INFO:teuthology.orchestra.run.smithi118.stdout: bdev async discard = True 2023-10-30T12:00:18.026 INFO:teuthology.orchestra.run.smithi118.stdout: bdev enable discard = True 2023-10-30T12:00:18.026 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2023-10-30T12:00:18.026 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2023-10-30T12:00:18.026 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore zero block detection = True 2023-10-30T12:00:18.026 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 20 2023-10-30T12:00:18.027 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 20 2023-10-30T12:00:18.027 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-30T12:00:18.027 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2023-10-30T12:00:18.027 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 10 2023-10-30T12:00:18.027 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:00:18.027 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2023-10-30T12:00:18.028 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:00:18.028 INFO:teuthology.orchestra.run.smithi118.stdout: osd class default list = * 2023-10-30T12:00:18.028 INFO:teuthology.orchestra.run.smithi118.stdout: osd class load list = * 2023-10-30T12:00:18.028 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify cached snaps = True 2023-10-30T12:00:18.028 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify missing on start = True 2023-10-30T12:00:18.029 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:00:18.029 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock override recovery settings = True 2023-10-30T12:00:18.029 INFO:teuthology.orchestra.run.smithi118.stdout: osd memory target = 1610612736 2023-10-30T12:00:18.029 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2023-10-30T12:00:18.029 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue = debug_random 2023-10-30T12:00:18.029 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue cut off = debug_random 2023-10-30T12:00:18.030 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.030 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2023-10-30T12:00:18.030 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-30T12:00:18.030 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2023-10-30T12:00:18.030 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-30T12:00:18.031 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-30T12:00:18.031 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:00:18.031 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:00:18.031 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2023-10-30T12:00:18.031 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.031 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2023-10-30T12:00:18.032 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-30T12:00:18.032 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-30T12:00:18.032 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2023-10-30T12:00:18.032 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-30T12:00:18.032 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2023-10-30T12:00:18.033 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2023-10-30T12:00:18.033 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:00:18.033 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2023-10-30T12:00:18.033 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2023-10-30T12:00:18.033 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:00:18.033 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.034 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:00:18.034 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:00:18.034 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:00:18.034 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.034 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:00:18.035 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:00:18.035 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:00:18.035 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.035 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2023-10-30T12:00:18.035 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:00:18.036 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.036 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:00:18.036 INFO:teuthology.orchestra.run.smithi118.stdout: mon min osdmap epochs = 25 2023-10-30T12:00:18.036 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune interval = 2 2023-10-30T12:00:18.036 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune min = 15 2023-10-30T12:00:18.037 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune txsize = 2 2023-10-30T12:00:18.037 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub inject crc mismatch = 0.01 2023-10-30T12:00:18.037 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub inject missing keys = 0.05 2023-10-30T12:00:18.037 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub interval = 300 2023-10-30T12:00:18.037 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on pool no app = False 2023-10-30T12:00:18.037 INFO:teuthology.orchestra.run.smithi118.stdout: paxos service trim min = 5 2023-10-30T12:00:18.038 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-30T12:00:18.038 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2023-10-30T12:00:18.038 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2023-10-30T12:00:18.038 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2023-10-30T12:00:18.038 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2023-10-30T12:00:18.039 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:00:18.039 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:00:18.039 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2023-10-30T12:00:18.039 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-30T12:00:18.039 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2023-10-30T12:00:18.039 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.c] 2023-10-30T12:00:18.040 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.b] 2023-10-30T12:00:18.041 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2023-10-30T12:00:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2023-10-30T12:00:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:00:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2023-10-30T12:00:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2023-10-30T12:00:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2023-10-30T12:00:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2023-10-30T12:00:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2023-10-30T12:00:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2023-10-30T12:00:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2023-10-30T12:00:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:00:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2023-10-30T12:00:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:00:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:00:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2023-10-30T12:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2023-10-30T12:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2023-10-30T12:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2023-10-30T12:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2023-10-30T12:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2023-10-30T12:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2023-10-30T12:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2023-10-30T12:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2023-10-30T12:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2023-10-30T12:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2023-10-30T12:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],172.21.15.118" 2023-10-30T12:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-30T12:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: debug objecter = 20 2023-10-30T12:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: debug rados = 20 2023-10-30T12:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2023-10-30T12:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: mon election default strategy = 1 2023-10-30T12:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 5000 2023-10-30T12:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: ms type = async 2023-10-30T12:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2023-10-30T12:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2023-10-30T12:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2023-10-30T12:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2023-10-30T12:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2023-10-30T12:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2023-10-30T12:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2023-10-30T12:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2023-10-30T12:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2023-10-30T12:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2023-10-30T12:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2023-10-30T12:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2023-10-30T12:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2023-10-30T12:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2023-10-30T12:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2023-10-30T12:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2023-10-30T12:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: bdev async discard = True 2023-10-30T12:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: bdev enable discard = True 2023-10-30T12:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2023-10-30T12:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2023-10-30T12:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore zero block detection = True 2023-10-30T12:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2023-10-30T12:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2023-10-30T12:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-30T12:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2023-10-30T12:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2023-10-30T12:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2023-10-30T12:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: osd class default list = * 2023-10-30T12:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: osd class load list = * 2023-10-30T12:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify cached snaps = True 2023-10-30T12:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify missing on start = True 2023-10-30T12:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock override recovery settings = True 2023-10-30T12:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: osd memory target = 1610612736 2023-10-30T12:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2023-10-30T12:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue = debug_random 2023-10-30T12:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue cut off = debug_random 2023-10-30T12:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2023-10-30T12:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-30T12:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2023-10-30T12:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-10-30T12:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-30T12:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2023-10-30T12:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2023-10-30T12:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-30T12:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-10-30T12:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2023-10-30T12:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-30T12:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2023-10-30T12:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2023-10-30T12:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2023-10-30T12:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2023-10-30T12:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2023-10-30T12:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: mon min osdmap epochs = 25 2023-10-30T12:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune interval = 2 2023-10-30T12:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune min = 15 2023-10-30T12:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune txsize = 2 2023-10-30T12:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub inject crc mismatch = 0.01 2023-10-30T12:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub inject missing keys = 0.05 2023-10-30T12:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub interval = 300 2023-10-30T12:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on pool no app = False 2023-10-30T12:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: paxos service trim min = 5 2023-10-30T12:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2023-10-30T12:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2023-10-30T12:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2023-10-30T12:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2023-10-30T12:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2023-10-30T12:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-30T12:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2023-10-30T12:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2023-10-30T12:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2023-10-30T12:00:18.071 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T12:00:18.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T12:00:18.141 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T12:00:18.141 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:18.142 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T12:00:18.156 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:18.157 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T12:00:18.211 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:00:18.212 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:18.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:00:18.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:00:18.298 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:18.298 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:18.353 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:00:18.353 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:18.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:00:18.354 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:00:18.395 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:18.395 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:18.450 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T12:00:18.450 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T12:00:18.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:00:18.552 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:00:18.555 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T12:00:18.555 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T12:00:18.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T12:00:18.597 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T12:00:18.612 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T12:00:18.613 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-10-30T12:00:18.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T12:00:18.644 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:18.645 INFO:tasks.ceph:role: osd.0 2023-10-30T12:00:18.645 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:00:18.645 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:00:18.715 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:00:18.715 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:00:18.715 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:00:18.715 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:00:18.716 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:00:18.716 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:00:18.716 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:00:18.716 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:00:18.716 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:00:18.717 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:00:18.747 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-30T12:00:18.749 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-30T12:00:18.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T12:00:18.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T12:00:18.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T12:00:18.969 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:18.970 INFO:tasks.ceph:role: osd.1 2023-10-30T12:00:18.970 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:00:18.970 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:00:19.039 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:00:19.040 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:00:19.040 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:00:19.040 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:00:19.040 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:00:19.041 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:00:19.041 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:00:19.041 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:00:19.041 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:00:19.041 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:00:19.072 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-30T12:00:19.074 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-30T12:00:19.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T12:00:19.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T12:00:19.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T12:00:19.293 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:19.293 INFO:tasks.ceph:role: osd.2 2023-10-30T12:00:19.294 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:00:19.294 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:00:19.363 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:00:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:00:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:00:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:00:19.364 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:00:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:00:19.364 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:00:19.365 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:00:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:00:19.365 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:00:19.396 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-30T12:00:19.398 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-30T12:00:19.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T12:00:19.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T12:00:19.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:19.659+0000 7f8c05214500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T12:00:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:19.660+0000 7f8c05214500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T12:00:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:19.660+0000 7f8c05214500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:00:19.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:19.660+0000 7f8c05214500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:00:19.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:19.660+0000 7f8c05214500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:00:19.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:19.660+0000 7f8c05214500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T12:00:22.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T12:00:22.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:22.632+0000 7f9984e0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T12:00:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:22.632+0000 7f9984e0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T12:00:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:22.632+0000 7f9984e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:00:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:22.632+0000 7f9984e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:00:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:22.632+0000 7f9984e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:00:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:22.633+0000 7f9984e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T12:00:25.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T12:00:25.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:25.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:25.614+0000 7f75d3c14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T12:00:25.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:25.614+0000 7f75d3c14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T12:00:25.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:25.614+0000 7f75d3c14500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:00:25.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:25.614+0000 7f75d3c14500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:00:25.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:25.614+0000 7f75d3c14500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:00:25.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:25.615+0000 7f75d3c14500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T12:00:28.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T12:00:28.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T12:00:28.532 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:28.533 INFO:tasks.ceph:role: osd.3 2023-10-30T12:00:28.533 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:00:28.533 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:00:28.602 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:00:28.603 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:00:28.603 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:00:28.603 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:00:28.603 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:00:28.603 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:00:28.604 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:00:28.604 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:00:28.604 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:00:28.604 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:00:28.635 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2023-10-30T12:00:28.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2023-10-30T12:00:28.637 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-10-30T12:00:28.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T12:00:28.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T12:00:28.850 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:28.853 INFO:tasks.ceph:role: osd.4 2023-10-30T12:00:28.854 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:00:28.854 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:00:28.916 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:00:28.917 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:00:28.917 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:00:28.917 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:00:28.917 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:00:28.918 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:00:28.918 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:00:28.918 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:00:28.918 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:00:28.918 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:00:28.949 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2023-10-30T12:00:28.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2023-10-30T12:00:28.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-10-30T12:00:29.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T12:00:29.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T12:00:29.161 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-10-30T12:00:29.162 INFO:tasks.ceph:role: osd.5 2023-10-30T12:00:29.162 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:00:29.163 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:00:29.228 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:00:29.229 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:00:29.229 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:00:29.229 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-30T12:00:29.229 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:00:29.229 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:00:29.230 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:00:29.230 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-30T12:00:29.230 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:00:29.230 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:00:29.261 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2023-10-30T12:00:29.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2023-10-30T12:00:29.263 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-10-30T12:00:29.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T12:00:29.405 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:29.504 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:29.502+0000 7f2d9ca0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T12:00:29.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:29.502+0000 7f2d9ca0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T12:00:29.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:29.503+0000 7f2d9ca0e500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:00:29.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:29.503+0000 7f2d9ca0e500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:00:29.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:29.503+0000 7f2d9ca0e500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:00:29.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:29.503+0000 7f2d9ca0e500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T12:00:32.349 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T12:00:32.382 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:32.477 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:32.475+0000 7fc9ec025500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T12:00:32.477 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:32.475+0000 7fc9ec025500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T12:00:32.477 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:32.475+0000 7fc9ec025500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:00:32.477 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:32.475+0000 7fc9ec025500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:00:32.478 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:32.475+0000 7fc9ec025500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:00:32.478 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:32.475+0000 7fc9ec025500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T12:00:35.327 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T12:00:35.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:35.448 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:35.446+0000 7f6291216500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T12:00:35.449 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:35.446+0000 7f6291216500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T12:00:35.449 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:35.446+0000 7f6291216500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:00:35.449 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:35.446+0000 7f6291216500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:00:35.449 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:35.446+0000 7f6291216500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:00:35.449 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:35.447+0000 7f6291216500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T12:00:38.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T12:00:38.323 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T12:00:38.323 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:38.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T12:00:38.361 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:38.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T12:00:38.436 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:38.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T12:00:38.509 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:38.509 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T12:00:38.541 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:38.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T12:00:38.612 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:38.613 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T12:00:38.683 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:38.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T12:00:38.753 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:38.753 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T12:00:38.809 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T12:00:38.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:00:38.812 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:00:38.853 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2023-10-30T12:00:38.853 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDTmj9lUvJmJxAA0ayP7/A4DuuM1mh/A9ABcA== 2023-10-30T12:00:38.853 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2023-10-30T12:00:38.853 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDWmj9lEVTDJRAANemb8SqAeQP+bdQ3zdynyg== 2023-10-30T12:00:38.853 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2023-10-30T12:00:38.854 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDZmj9lUMGwJBAAfILNjbg2mPd6m4oR+dt3Uw== 2023-10-30T12:00:38.854 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2023-10-30T12:00:38.854 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDSmj9lBq7qIBAA8Ys/ZW8ZoSUP/Mebny+FrA== 2023-10-30T12:00:38.854 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2023-10-30T12:00:38.854 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDdmj9lYs8EHhAAECrRgKkCBFwBL99Lk6p6bQ== 2023-10-30T12:00:38.855 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2023-10-30T12:00:38.855 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDgmj9lSE9dHBAAPnpCZPNyqmc7AT//6U2Fjw== 2023-10-30T12:00:38.855 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2023-10-30T12:00:38.855 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDjmj9lkWWtGhAAAeHppxY4e7zXfzCAHRrguw== 2023-10-30T12:00:38.855 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2023-10-30T12:00:38.856 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDSmj9l3FSZIxAAdc1+wcWEFJttQelyrpBFNA== 2023-10-30T12:00:38.876 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2023-10-30T12:00:38.876 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDTmj9lUvJmJxAA0ayP7/A4DuuM1mh/A9ABcA== 2023-10-30T12:00:38.877 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2023-10-30T12:00:38.877 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDWmj9lEVTDJRAANemb8SqAeQP+bdQ3zdynyg== 2023-10-30T12:00:38.877 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2023-10-30T12:00:38.877 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDZmj9lUMGwJBAAfILNjbg2mPd6m4oR+dt3Uw== 2023-10-30T12:00:38.877 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2023-10-30T12:00:38.878 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDSmj9lBq7qIBAA8Ys/ZW8ZoSUP/Mebny+FrA== 2023-10-30T12:00:38.878 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2023-10-30T12:00:38.878 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDdmj9lYs8EHhAAECrRgKkCBFwBL99Lk6p6bQ== 2023-10-30T12:00:38.878 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.4] 2023-10-30T12:00:38.878 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDgmj9lSE9dHBAAPnpCZPNyqmc7AT//6U2Fjw== 2023-10-30T12:00:38.879 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.5] 2023-10-30T12:00:38.879 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDjmj9lkWWtGhAAAeHppxY4e7zXfzCAHRrguw== 2023-10-30T12:00:38.879 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2023-10-30T12:00:38.879 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQDSmj9l3FSZIxAAdc1+wcWEFJttQelyrpBFNA== 2023-10-30T12:00:38.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:38.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:38.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:38.977 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.035 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:00:39.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:00:39.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.203 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:00:39.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:00:39.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:00:39.372 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T12:00:39.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T12:00:39.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:00:39.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T12:00:39.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T12:00:39.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:00:39.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T12:00:39.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T12:00:39.766 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:00:39.865 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T12:00:39.892 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:39.895 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:00:39.946 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T12:00:39.947 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:00:39.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:00:39.950 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:00:39.950 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:00:39.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:00:39.953 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:00:39.953 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:00:39.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:00:39.989 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:00:39.989 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T12:00:39.990 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T12:00:39.990 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-30T12:00:39.992 INFO:tasks.ceph.mgr.x:Started 2023-10-30T12:00:39.993 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T12:00:39.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T12:00:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.099+0000 7fbff50fc640 10 monclient: get_monmap_and_config 2023-10-30T12:00:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.099+0000 7fbff50fc640 10 monclient: build_initial_monmap 2023-10-30T12:00:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.099+0000 7fbff50fc640 10 monclient: monmap: 2023-10-30T12:00:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:40.100326+0000 2023-10-30T12:00:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:40.100326+0000 2023-10-30T12:00:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:40.106 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.099+0000 7fbff50fc640 1 Processor -- start 2023-10-30T12:00:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.099+0000 7fbff50fc640 1 -- start start 2023-10-30T12:00:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.099+0000 7fbff50fc640 10 monclient: init 2023-10-30T12:00:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:00:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbff01327f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 0x7fbff013aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient(hunting): picked mon.noname-a con 0x7fbff0099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 0x7fbff013d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient(hunting): picked mon.noname-c con 0x7fbff013afa0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff00a1620 con 0x7fbff01327f0 2023-10-30T12:00:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff009a6b0 con 0x7fbff0099a40 2023-10-30T12:00:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff00a1b20 con 0x7fbff013afa0 2023-10-30T12:00:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbff50fc640 10 monclient(hunting): authenticate will time out at 885.283369s 2023-10-30T12:00:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 0x7fbff013aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfeed9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbff01327f0 0x7fbff0132bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34404/0 (socket says 172.21.15.50:34404) 2023-10-30T12:00:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 0x7fbff013aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38530/0 (socket says 172.21.15.50:38530) 2023-10-30T12:00:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfeed9d640 1 -- 172.21.15.50:0/1852822038 learned_addr learned my addr 172.21.15.50:0/1852822038 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 10 monclient(hunting): get_auth_request con 0x7fbff0099a40 auth_method 0 2023-10-30T12:00:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 1 -- 172.21.15.50:0/1852822038 >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 msgr2=0x7fbff013aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:00:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 1 -- 172.21.15.50:0/1852822038 >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 msgr2=0x7fbff013aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:00:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 1 --2- 172.21.15.50:0/1852822038 >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 0x7fbff013aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:00:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 1 --2- 172.21.15.50:0/1852822038 >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 0x7fbff013d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfee59c640 1 --2- 172.21.15.50:0/1852822038 >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 0x7fbff013aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:00:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 10 monclient(hunting): get_auth_request con 0x7fbff013afa0 auth_method 0 2023-10-30T12:00:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 1 -- 172.21.15.50:0/1852822038 >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 msgr2=0x7fbff013d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:00:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 1 -- 172.21.15.50:0/1852822038 >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 msgr2=0x7fbff013d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:00:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 1 --2- 172.21.15.50:0/1852822038 >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 0x7fbff013d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:00:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.100+0000 7fbfef59e640 1 --2- 172.21.15.50:0/1852822038 >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 0x7fbff013d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:00:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.286+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844540113 0 0) 0x7fbff00a1620 con 0x7fbff01327f0 2023-10-30T12:00:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.286+0000 7fbfedd9b640 10 monclient(hunting): _negotiate 2023-10-30T12:00:40.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.286+0000 7fbfedd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.286+0000 7fbfedd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.286+0000 7fbfedd9b640 10 monclient(hunting): my global_id is 4098 2023-10-30T12:00:40.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.286+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd8003520 con 0x7fbff01327f0 2023-10-30T12:00:40.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1648361380 0 0) 0x7fbfd8003520 con 0x7fbff01327f0 2023-10-30T12:00:40.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 msgr2=0x7fbff013d390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:00:40.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 --2- 172.21.15.50:0/1852822038 >> v2:172.21.15.118:3300/0 conn(0x7fbff013afa0 0x7fbff013d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 msgr2=0x7fbff013aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:00:40.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 --2- 172.21.15.50:0/1852822038 >> v2:172.21.15.50:3300/0 conn(0x7fbff0099a40 0x7fbff013aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:00:40.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:00:40.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:40.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:00:40.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff009a0f0 con 0x7fbff01327f0 2023-10-30T12:00:40.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: _finish_auth 0 2023-10-30T12:00:40.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:00:40.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:40.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fbfe0001670 con 0x7fbff01327f0 2023-10-30T12:00:40.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:00:40.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: dump: 2023-10-30T12:00:40.302 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:40.302 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.304 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.304 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.304 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:40.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:00:40.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:40.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 >> v1:172.21.15.50:6789/0 conn(0x7fbff01327f0 legacy=0x7fbff0132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbfd8003970 0x7fbfd8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): picked mon.b con 0x7fbfd8003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbfd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): picked mon.a con 0x7fbfd80047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbfd80051f0 0x7fbfd800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): picked mon.c con 0x7fbfd80051f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff009a6b0 con 0x7fbfd80047c0 2023-10-30T12:00:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff00a1b20 con 0x7fbfd8003970 2023-10-30T12:00:40.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd800c1d0 con 0x7fbfd80051f0 2023-10-30T12:00:40.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfedd9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:00:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbfd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): get_auth_request con 0x7fbfd80047c0 auth_method 0 2023-10-30T12:00:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:40.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbfd8003970 0x7fbfd8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbfd80051f0 0x7fbfd800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:40.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 10 monclient(hunting): get_auth_request con 0x7fbfd80051f0 auth_method 0 2023-10-30T12:00:40.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 10 monclient(hunting): get_auth_request con 0x7fbfd8003970 auth_method 0 2023-10-30T12:00:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:40.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:40.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:40.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-10-30T12:00:40.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 1 -- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbfd80051f0 msgr2=0x7fbfd800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:40.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbfd80051f0 0x7fbfd800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 1 -- 172.21.15.50:0/1852822038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbfd8003970 msgr2=0x7fbfd8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:40.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbfd8003970 0x7fbfd8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient: _finish_hunting 0 2023-10-30T12:00:40.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 1 monclient: found mon.a 2023-10-30T12:00:40.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:40.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:40.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 1 -- 172.21.15.50:0/1852822038 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd8003520 con 0x7fbfd80047c0 2023-10-30T12:00:40.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient: _finish_auth 0 2023-10-30T12:00:40.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 10 monclient: _check_auth_tickets 2023-10-30T12:00:40.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfee59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:40.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbfef59e640 1 -- 172.21.15.50:0/1852822038 reap_dead start 2023-10-30T12:00:40.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbff50fc640 5 monclient: authenticate success, global_id 4098 2023-10-30T12:00:40.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.287+0000 7fbff50fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:40.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfee59c640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbfd8004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400b9e0 tx=0x7fbfd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddc6ff22f054a22 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:40.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4032620 con 0x7fbfd80047c0 2023-10-30T12:00:40.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:40.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 10 monclient: dump: 2023-10-30T12:00:40.323 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:40.324 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.324 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.325 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.325 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.325 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:40.326 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.326 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.327 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.327 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4032780 con 0x7fbfd80047c0 2023-10-30T12:00:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbff50fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbff50fc640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbff50fc640 4 set_mon_vals no callback set 2023-10-30T12:00:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 1 -- 172.21.15.50:0/1852822038 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4032b30 con 0x7fbfd80047c0 2023-10-30T12:00:40.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:40.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.289+0000 7fbfedd9b640 10 monclient: dump: 2023-10-30T12:00:40.331 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:40.331 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.331 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:40.333 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.333 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.333 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.334 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 10 monclient: shutdown 2023-10-30T12:00:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 1 -- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 msgr2=0x7fbfd8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbfd8004c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400b9e0 tx=0x7fbfd80057f0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 1 -- 172.21.15.50:0/1852822038 shutdown_connections 2023-10-30T12:00:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 1 --2- 172.21.15.50:0/1852822038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbfd8004c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 1 -- 172.21.15.50:0/1852822038 >> 172.21.15.50:0/1852822038 conn(0x7fbff00a82f0 msgr2=0x7fbff00a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 1 -- 172.21.15.50:0/1852822038 shutdown_connections 2023-10-30T12:00:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.290+0000 7fbff50fc640 1 -- 172.21.15.50:0/1852822038 wait complete. 2023-10-30T12:00:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 10 monclient: build_initial_monmap 2023-10-30T12:00:40.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 10 monclient: monmap: 2023-10-30T12:00:40.338 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:40.339 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:40.339 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:40.339 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:40.339 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:40.341 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:40.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 1 librados: starting msgr at 2023-10-30T12:00:40.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 1 librados: starting objecter 2023-10-30T12:00:40.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 1 Processor -- start 2023-10-30T12:00:40.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 1 -- start start 2023-10-30T12:00:40.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 1 librados: setting wanted keys 2023-10-30T12:00:40.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 1 librados: calling monclient init 2023-10-30T12:00:40.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.291+0000 7fbff50fc640 10 monclient: init 2023-10-30T12:00:40.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:40.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbff01315f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd80047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff01ca720 0x7fbff0131b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): picked mon.noname-b con 0x7fbff01ca720 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbff0136410 0x7fbff01368c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): picked mon.noname-c con 0x7fbff0136410 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0138260 con 0x7fbfd80047c0 2023-10-30T12:00:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff013a1a0 con 0x7fbff01ca720 2023-10-30T12:00:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff01cb3d0 con 0x7fbff0136410 2023-10-30T12:00:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbff50fc640 10 monclient(hunting): authenticate will time out at 885.475447s 2023-10-30T12:00:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbff01315f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbff0136410 0x7fbff01368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbff01315f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38554/0 (socket says 172.21.15.50:38554) 2023-10-30T12:00:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 1 -- 172.21.15.50:0/7180133 learned_addr learned my addr 172.21.15.50:0/7180133 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 10 monclient(hunting): get_auth_request con 0x7fbfd80047c0 auth_method 0 2023-10-30T12:00:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff01ca720 0x7fbff0131b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): get_auth_request con 0x7fbff0136410 auth_method 0 2023-10-30T12:00:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 10 monclient(hunting): get_auth_request con 0x7fbff01ca720 auth_method 0 2023-10-30T12:00:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:40.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:40.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2023-10-30T12:00:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 1 -- 172.21.15.50:0/7180133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff01ca720 msgr2=0x7fbff0131b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff01ca720 0x7fbff0131b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 1 -- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 msgr2=0x7fbff01315f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbff01315f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient: _finish_hunting 0 2023-10-30T12:00:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 1 monclient: found mon.noname-c 2023-10-30T12:00:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfef59e640 1 -- 172.21.15.50:0/7180133 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff00a3d90 con 0x7fbff0136410 2023-10-30T12:00:40.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.292+0000 7fbfeed9d640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbff01315f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:40.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfef59e640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbff0136410 0x7fbff01368c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbff00a7390 tx=0x7fbfe8007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:40.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 client.?.objecter ms_handle_connect 0x7fbff0136410 2023-10-30T12:00:40.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:40.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 1 -- 172.21.15.50:0/7180133 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8014610 con 0x7fbff0136410 2023-10-30T12:00:40.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:40.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: dump: 2023-10-30T12:00:40.355 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:40.355 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.357 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 1 -- 172.21.15.50:0/7180133 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe8014790 con 0x7fbff0136410 2023-10-30T12:00:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 5 monclient: authenticate success, global_id 4103 2023-10-30T12:00:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 10 monclient: _renew_subs 2023-10-30T12:00:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff0139be0 con 0x7fbff0136410 2023-10-30T12:00:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 1 -- 172.21.15.50:0/7180133 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8014b40 con 0x7fbff0136410 2023-10-30T12:00:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 monclient: dump: 2023-10-30T12:00:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.361 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 10 client.4103.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 10 monclient: _renew_subs 2023-10-30T12:00:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff0136e00 con 0x7fbff0136410 2023-10-30T12:00:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbff50fc640 1 librados: init done 2023-10-30T12:00:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfed59a640 4 set_mon_vals no callback set 2023-10-30T12:00:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 1 -- 172.21.15.50:0/7180133 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbfe8014ca0 con 0x7fbff0136410 2023-10-30T12:00:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 1 -- 172.21.15.50:0/7180133 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbfe801d9f0 con 0x7fbff0136410 2023-10-30T12:00:40.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 10 client.4103.objecter ms_dispatch 0x7fbff0000b90 osd_map(1..1 src has 1..1) v4 2023-10-30T12:00:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 3 client.4103.objecter handle_osd_map got epochs [1,1] > 0 2023-10-30T12:00:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 3 client.4103.objecter handle_osd_map decoding full epoch 1 2023-10-30T12:00:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.293+0000 7fbfd77fe640 20 client.4103.objecter dump_active .. 0 homeless 2023-10-30T12:00:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.294+0000 7fbff50fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.294+0000 7fbff50fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.294+0000 7fbff50fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.294+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff013a2e0 con 0x7fbff0136410 2023-10-30T12:00:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.296+0000 7fbfd77fe640 1 -- 172.21.15.50:0/7180133 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fbff013a2e0 con 0x7fbff0136410 2023-10-30T12:00:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.296+0000 7fbfd77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.296+0000 7fbfd77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.361+0000 7fbff50fc640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:00:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.361+0000 7fbff50fc640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:00:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.361+0000 7fbff50fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.361+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fbff00a75c0 con 0x7fbff0136410 2023-10-30T12:00:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.364+0000 7fbfd77fe640 1 -- 172.21.15.50:0/7180133 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbfe8007e20 con 0x7fbff0136410 2023-10-30T12:00:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.364+0000 7fbfd77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-10-30T12:00:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.364+0000 7fbfd77fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-10-30T12:00:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2023-10-30T12:00:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 10 librados: watch_flush enter 2023-10-30T12:00:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 10 librados: watch_flush exit 2023-10-30T12:00:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 20 client.4103.objecter shutdown clearing up homeless session... 2023-10-30T12:00:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 10 client.4103.objecter successfully canceled tick 2023-10-30T12:00:40.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 10 monclient: shutdown 2023-10-30T12:00:40.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:40.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbff0136410 msgr2=0x7fbff01368c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbff0136410 0x7fbff01368c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbff00a7390 tx=0x7fbfe8007700 comp rx=0 tx=0).stop 2023-10-30T12:00:40.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 shutdown_connections 2023-10-30T12:00:40.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbff0136410 0x7fbff01368c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbff01ca720 0x7fbff0131b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 --2- 172.21.15.50:0/7180133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfd80047c0 0x7fbff01315f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 >> 172.21.15.50:0/7180133 conn(0x7fbff00a82f0 msgr2=0x7fbff00a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 shutdown_connections 2023-10-30T12:00:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 -- 172.21.15.50:0/7180133 wait complete. 2023-10-30T12:00:40.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:40.365+0000 7fbff50fc640 1 librados: shutdown 2023-10-30T12:00:40.377 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T12:00:40.377 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T12:00:40.378 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:40.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T12:00:40.408 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:40.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T12:00:40.479 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-30T12:00:40.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T12:00:40.548 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:40.549 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T12:00:40.576 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:40.577 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T12:00:40.643 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:00:40.643 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T12:00:40.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new dba8cbf1-bc2e-4420-957f-11ec607e814e 0 2023-10-30T12:00:40.786 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:40.783+0000 7f3dd2056040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:00:40.839 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:40.836+0000 7f3dd2056040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:00:40.866 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.845+0000 7f06df31d640 10 monclient: get_monmap_and_config 2023-10-30T12:00:40.867 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.846+0000 7f06df31d640 10 monclient: build_initial_monmap 2023-10-30T12:00:40.867 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.846+0000 7f06df31d640 10 monclient: monmap: 2023-10-30T12:00:40.868 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:40.868 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.868 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:40.847847+0000 2023-10-30T12:00:40.869 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:40.847847+0000 2023-10-30T12:00:40.869 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:40.869 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:40.870 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:40.870 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:40.870 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:40.871 INFO:teuthology.orchestra.run.smithi118.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:40.871 INFO:teuthology.orchestra.run.smithi118.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:40.872 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.846+0000 7f06df31d640 1 Processor -- start 2023-10-30T12:00:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.846+0000 7f06df31d640 1 -- start start 2023-10-30T12:00:40.873 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.846+0000 7f06df31d640 10 monclient: init 2023-10-30T12:00:40.873 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:40.874 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-10-30T12:00:40.874 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06d81327f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:40.875 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f06d8099a40 0x7f06d813aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.875 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient(hunting): picked mon.noname-a con 0x7f06d8099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:40.875 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f06d813afa0 0x7f06d813d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.876 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient(hunting): picked mon.noname-c con 0x7f06d813afa0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:40.876 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d80a1620 con 0x7f06d81327f0 2023-10-30T12:00:40.877 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.877 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d809a6b0 con 0x7f06d8099a40 2023-10-30T12:00:40.877 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.878 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d80a1b20 con 0x7f06d813afa0 2023-10-30T12:00:40.878 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:40.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06df31d640 10 monclient(hunting): authenticate will time out at 872.185465s 2023-10-30T12:00:40.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f06d813afa0 0x7f06d813d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f06d813afa0 0x7f06d813d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51494/0 (socket says 172.21.15.118:51494) 2023-10-30T12:00:40.880 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 -- 172.21.15.118:0/635714229 learned_addr learned my addr 172.21.15.118:0/635714229 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:40.880 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): get_auth_request con 0x7f06d813afa0 auth_method 0 2023-10-30T12:00:40.880 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd0b9640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f06d81327f0 0x7f06d8132bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:55716/0 (socket says 172.21.15.118:55716) 2023-10-30T12:00:40.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dc8b8640 1 --2- 172.21.15.118:0/635714229 >> v2:172.21.15.50:3300/0 conn(0x7f06d8099a40 0x7f06d813aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:40.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dc8b8640 10 monclient(hunting): get_auth_request con 0x7f06d8099a40 auth_method 0 2023-10-30T12:00:40.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dc8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dc8b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dc8b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2023-10-30T12:00:40.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 -- 172.21.15.118:0/635714229 >> v2:172.21.15.50:3300/0 conn(0x7f06d8099a40 msgr2=0x7f06d813aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:40.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 --2- 172.21.15.118:0/635714229 >> v2:172.21.15.50:3300/0 conn(0x7f06d8099a40 0x7f06d813aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 -- 172.21.15.118:0/635714229 >> v1:172.21.15.118:6789/0 conn(0x7f06d81327f0 legacy=0x7f06d8132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient: _finish_hunting 0 2023-10-30T12:00:40.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 monclient: found mon.noname-c 2023-10-30T12:00:40.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:40.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:40.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dd8ba640 1 -- 172.21.15.118:0/635714229 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d809a0f0 con 0x7f06d813afa0 2023-10-30T12:00:40.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.847+0000 7f06dc8b8640 1 --2- 172.21.15.118:0/635714229 >> v2:172.21.15.50:3300/0 conn(0x7f06d8099a40 0x7f06d813aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:40.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06dd8ba640 1 --2- 172.21.15.118:0/635714229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d813afa0 0x7f06d813d390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06d8071fa0 tx=0x7f06d002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5696f92a4c7217 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:40.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 1 -- 172.21.15.118:0/635714229 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d002c9f0 con 0x7f06d813afa0 2023-10-30T12:00:40.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:40.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: dump: 2023-10-30T12:00:40.887 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:40.888 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.888 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.888 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.888 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.888 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:40.889 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.889 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.889 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.889 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:40.889 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: _finish_auth 0 2023-10-30T12:00:40.889 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:40.890 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:40.890 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 1 -- 172.21.15.118:0/635714229 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d002cb50 con 0x7f06d813afa0 2023-10-30T12:00:40.890 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06df31d640 5 monclient: authenticate success, global_id 4108 2023-10-30T12:00:40.890 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06df31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:40.890 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:40.891 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06df31d640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:40.891 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 1 -- 172.21.15.118:0/635714229 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d00343f0 con 0x7f06d813afa0 2023-10-30T12:00:40.891 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.891 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06df31d640 4 set_mon_vals no callback set 2023-10-30T12:00:40.891 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:40.892 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.848+0000 7f06cffff640 10 monclient: dump: 2023-10-30T12:00:40.892 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:40.892 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.892 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.892 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.893 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.893 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:40.893 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.893 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.893 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.894 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:40.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 10 monclient: shutdown 2023-10-30T12:00:40.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:40.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 -- 172.21.15.118:0/635714229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d813afa0 msgr2=0x7f06d813d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 --2- 172.21.15.118:0/635714229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d813afa0 0x7f06d813d390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06d8071fa0 tx=0x7f06d002be50 comp rx=0 tx=0).stop 2023-10-30T12:00:40.895 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 -- 172.21.15.118:0/635714229 shutdown_connections 2023-10-30T12:00:40.895 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 --2- 172.21.15.118:0/635714229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d813afa0 0x7f06d813d390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.895 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 --2- 172.21.15.118:0/635714229 >> v2:172.21.15.50:3300/0 conn(0x7f06d8099a40 0x7f06d813aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.895 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 -- 172.21.15.118:0/635714229 >> 172.21.15.118:0/635714229 conn(0x7f06d80a82f0 msgr2=0x7f06d80a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:40.895 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 -- 172.21.15.118:0/635714229 shutdown_connections 2023-10-30T12:00:40.896 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 -- 172.21.15.118:0/635714229 wait complete. 2023-10-30T12:00:40.896 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 10 monclient: build_initial_monmap 2023-10-30T12:00:40.896 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 10 monclient: monmap: 2023-10-30T12:00:40.896 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:40.896 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:40.897 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2023-10-30T12:00:40.897 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2023-10-30T12:00:40.897 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:40.897 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:40.897 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:40.897 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:40.898 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:40.898 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:40.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 librados: starting msgr at 2023-10-30T12:00:40.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.849+0000 7f06df31d640 1 librados: starting objecter 2023-10-30T12:00:40.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 Processor -- start 2023-10-30T12:00:40.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 -- start start 2023-10-30T12:00:40.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 librados: setting wanted keys 2023-10-30T12:00:40.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 librados: calling monclient init 2023-10-30T12:00:40.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient: init 2023-10-30T12:00:40.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:40.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:40.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d8099a40 0x7f06d8134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): picked mon.noname-b con 0x7f06d8099a40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:40.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d81327f0 0x7f06d8135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): picked mon.noname-c con 0x7f06d81327f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:40.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d813afa0 0x7f06d8135970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:40.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): picked mon.noname-a con 0x7f06d813afa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:40.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d81386c0 con 0x7f06d813afa0 2023-10-30T12:00:40.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d8138f60 con 0x7f06d8099a40 2023-10-30T12:00:40.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:40.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d81377d0 con 0x7f06d81327f0 2023-10-30T12:00:40.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:40.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06df31d640 10 monclient(hunting): authenticate will time out at 872.188751s 2023-10-30T12:00:40.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d8099a40 0x7f06d8134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d8099a40 0x7f06d8134ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51504/0 (socket says 172.21.15.118:51504) 2023-10-30T12:00:40.903 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 -- 172.21.15.118:0/631102939 learned_addr learned my addr 172.21.15.118:0/631102939 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:40.903 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): get_auth_request con 0x7f06d8099a40 auth_method 0 2023-10-30T12:00:40.903 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.903 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.903 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d81327f0 0x7f06d8135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d813afa0 0x7f06d8135970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:40.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:40.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 10 monclient(hunting): get_auth_request con 0x7f06d81327f0 auth_method 0 2023-10-30T12:00:40.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 10 monclient(hunting): get_auth_request con 0x7f06d813afa0 auth_method 0 2023-10-30T12:00:40.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:40.906 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:40.906 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:40.906 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.906 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.906 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:40.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:40.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:40.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dc8b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:40.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2023-10-30T12:00:40.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 -- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d81327f0 msgr2=0x7f06d8135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.908 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d81327f0 0x7f06d8135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.908 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 -- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d813afa0 msgr2=0x7f06d8135970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.908 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d813afa0 0x7f06d8135970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.908 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient: _finish_hunting 0 2023-10-30T12:00:40.908 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 monclient: found mon.noname-b 2023-10-30T12:00:40.908 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:40.909 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:40.909 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.850+0000 7f06dd0b9640 1 -- 172.21.15.118:0/631102939 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d0009b30 con 0x7f06d8099a40 2023-10-30T12:00:40.909 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06dd0b9640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d8099a40 0x7f06d8134ef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f06c800c4d0 tx=0x7f06c800c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:40.909 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 client.?.objecter ms_handle_connect 0x7f06d8099a40 2023-10-30T12:00:40.909 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:40.910 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 1 -- 172.21.15.118:0/631102939 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8007480 con 0x7f06d8099a40 2023-10-30T12:00:40.910 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.910 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:40.910 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: dump: 2023-10-30T12:00:40.910 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:40.911 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.911 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.911 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.911 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.911 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:40.912 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.912 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.912 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.912 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:40.912 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:40.912 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:40.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:40.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 5 monclient: authenticate success, global_id 4114 2023-10-30T12:00:40.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 10 monclient: _renew_subs 2023-10-30T12:00:40.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 1 -- 172.21.15.118:0/631102939 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c80075e0 con 0x7f06d8099a40 2023-10-30T12:00:40.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:40.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d80a17f0 con 0x7f06d8099a40 2023-10-30T12:00:40.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:40.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 1 -- 172.21.15.118:0/631102939 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c80049e0 con 0x7f06d8099a40 2023-10-30T12:00:40.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:40.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:40.915 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 monclient: dump: 2023-10-30T12:00:40.915 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:40.915 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:40.915 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.915 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:40.915 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:40.916 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:40.916 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:40.916 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:40.916 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:40.916 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:40.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cf7fe640 4 set_mon_vals no callback set 2023-10-30T12:00:40.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:40.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 10 monclient: _renew_subs 2023-10-30T12:00:40.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:40.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 1 -- 172.21.15.118:0/631102939 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f06c801c3b0 con 0x7f06d8099a40 2023-10-30T12:00:40.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d8138c80 con 0x7f06d8099a40 2023-10-30T12:00:40.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06df31d640 1 librados: init done 2023-10-30T12:00:40.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 1 -- 172.21.15.118:0/631102939 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f06c801cbd0 con 0x7f06d8099a40 2023-10-30T12:00:40.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 10 client.4114.objecter ms_dispatch 0x7f06d8000b90 osd_map(2..2 src has 1..2) v4 2023-10-30T12:00:40.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2023-10-30T12:00:40.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2023-10-30T12:00:40.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.851+0000 7f06cdffb640 20 client.4114.objecter dump_active .. 0 homeless 2023-10-30T12:00:40.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.852+0000 7f06df31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:40.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.852+0000 7f06df31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:40.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.852+0000 7f06df31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:40.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.852+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d8138900 con 0x7f06d8099a40 2023-10-30T12:00:40.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.854+0000 7f06cdffb640 1 -- 172.21.15.118:0/631102939 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f06c8018070 con 0x7f06d8099a40 2023-10-30T12:00:40.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.854+0000 7f06cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:40.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.854+0000 7f06cdffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:40.927 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.924+0000 7f06df31d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dba8cbf1-bc2e-4420-957f-11ec607e814e", "id": 0}] 2023-10-30T12:00:40.927 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.924+0000 7f06df31d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dba8cbf1-bc2e-4420-957f-11ec607e814e", "id": 0}] 2023-10-30T12:00:40.927 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.924+0000 7f06df31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:40.928 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.924+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dba8cbf1-bc2e-4420-957f-11ec607e814e", "id": 0} v 0) v1 -- 0x7f06d80a47b0 con 0x7f06d8099a40 2023-10-30T12:00:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06cdffb640 1 -- 172.21.15.118:0/631102939 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dba8cbf1-bc2e-4420-957f-11ec607e814e", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f06c801ce60 con 0x7f06d8099a40 2023-10-30T12:00:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dba8cbf1-bc2e-4420-957f-11ec607e814e", "id": 0}] 2023-10-30T12:00:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06cdffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:40.932 INFO:teuthology.orchestra.run.smithi118.stdout:0 2023-10-30T12:00:40.933 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 10 librados: watch_flush enter 2023-10-30T12:00:40.933 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 10 librados: watch_flush exit 2023-10-30T12:00:40.933 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 20 client.4114.objecter shutdown clearing up homeless session... 2023-10-30T12:00:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 10 client.4114.objecter successfully canceled tick 2023-10-30T12:00:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 10 monclient: shutdown 2023-10-30T12:00:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d8099a40 msgr2=0x7f06d8134ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d8099a40 0x7f06d8134ef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f06c800c4d0 tx=0x7f06c800c990 comp rx=0 tx=0).stop 2023-10-30T12:00:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 shutdown_connections 2023-10-30T12:00:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d813afa0 0x7f06d8135970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d81327f0 0x7f06d8135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 1 --2- 172.21.15.118:0/631102939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06d8099a40 0x7f06d8134ef0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.929+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 >> 172.21.15.118:0/631102939 conn(0x7f06d80a82f0 msgr2=0x7f06d80a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:40.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.930+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 shutdown_connections 2023-10-30T12:00:40.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.930+0000 7f06df31d640 1 -- 172.21.15.118:0/631102939 wait complete. 2023-10-30T12:00:40.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:40.930+0000 7f06df31d640 1 librados: shutdown 2023-10-30T12:00:40.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new c13ebbef-aa1a-4da7-b5f6-26b02e43955c 1 2023-10-30T12:00:41.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.077+0000 7f4042773640 10 monclient: get_monmap_and_config 2023-10-30T12:00:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.077+0000 7f4042773640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.077+0000 7f4042773640 10 monclient: monmap: 2023-10-30T12:00:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:41.078934+0000 2023-10-30T12:00:41.081 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:41.078934+0000 2023-10-30T12:00:41.081 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.081 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.081 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:41.081 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:41.082 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:41.082 INFO:teuthology.orchestra.run.smithi118.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:41.082 INFO:teuthology.orchestra.run.smithi118.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:41.082 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.097 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.077+0000 7f4042773640 1 Processor -- start 2023-10-30T12:00:41.098 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.077+0000 7f4042773640 1 -- start start 2023-10-30T12:00:41.098 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.077+0000 7f4042773640 10 monclient: init 2023-10-30T12:00:41.098 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.098 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-10-30T12:00:41.099 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f403c1327f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:41.099 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f403c099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:41.099 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c13c180 0x7f403c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.099 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient(hunting): picked mon.noname-b con 0x7f403c13c180 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.100 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0a1620 con 0x7f403c1327f0 2023-10-30T12:00:41.100 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0a1b20 con 0x7f403c099a40 2023-10-30T12:00:41.100 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.100 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c09a6b0 con 0x7f403c13c180 2023-10-30T12:00:41.100 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.101 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4042773640 10 monclient(hunting): authenticate will time out at 872.416557s 2023-10-30T12:00:41.101 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403b7fe640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f403c099a40 0x7f403c13aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:55720/0 (socket says 172.21.15.118:55720) 2023-10-30T12:00:41.101 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403b7fe640 1 -- 172.21.15.118:0/2107440481 learned_addr learned my addr 172.21.15.118:0/2107440481 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-30T12:00:41.101 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 1 --2- 172.21.15.118:0/2107440481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c13c180 0x7f403c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.101 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): get_auth_request con 0x7f403c13c180 auth_method 0 2023-10-30T12:00:41.102 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.102 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.102 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.102 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.102 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.103 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.103 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 1 -- 172.21.15.118:0/2107440481 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685150063 0 0) 0x7f403c0a1b20 con 0x7f403c099a40 2023-10-30T12:00:41.103 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): _negotiate 2023-10-30T12:00:41.103 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.103 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.103 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): my global_id is 4117 2023-10-30T12:00:41.104 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 1 -- 172.21.15.118:0/2107440481 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4024003520 con 0x7f403c099a40 2023-10-30T12:00:41.104 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 1 -- 172.21.15.118:0/2107440481 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760065215 0 0) 0x7f403c0a1620 con 0x7f403c1327f0 2023-10-30T12:00:41.104 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): _negotiate 2023-10-30T12:00:41.104 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.105 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.105 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 10 monclient(hunting): my global_id is 4116 2023-10-30T12:00:41.105 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f403affd640 1 -- 172.21.15.118:0/2107440481 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c0a1b20 con 0x7f403c1327f0 2023-10-30T12:00:41.106 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2023-10-30T12:00:41.106 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 1 -- 172.21.15.118:0/2107440481 >> v1:172.21.15.118:6789/0 conn(0x7f403c099a40 legacy=0x7f403c13aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.106 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 1 -- 172.21.15.118:0/2107440481 >> v1:172.21.15.50:6789/0 conn(0x7f403c1327f0 legacy=0x7f403c132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.107 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.107 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 1 monclient: found mon.noname-b 2023-10-30T12:00:41.107 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.107 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:41.107 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.078+0000 7f4040d10640 1 -- 172.21.15.118:0/2107440481 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c09a0f0 con 0x7f403c13c180 2023-10-30T12:00:41.108 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f4040d10640 1 --2- 172.21.15.118:0/2107440481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c13c180 0x7f403c13e570 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4034009dc0 tx=0x7f403402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=455689b6d37d7ff9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.108 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 1 -- 172.21.15.118:0/2107440481 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403402cbb0 con 0x7f403c13c180 2023-10-30T12:00:41.108 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.108 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:41.108 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: dump: 2023-10-30T12:00:41.109 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.109 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.109 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.109 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.109 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.109 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.110 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.110 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.110 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.110 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.110 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.111 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.111 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.111 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 1 -- 172.21.15.118:0/2107440481 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403402cd10 con 0x7f403c13c180 2023-10-30T12:00:41.111 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f4042773640 5 monclient: authenticate success, global_id 4109 2023-10-30T12:00:41.111 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f4042773640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:41.111 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.112 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f4042773640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:41.112 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 1 -- 172.21.15.118:0/2107440481 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034034660 con 0x7f403c13c180 2023-10-30T12:00:41.112 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.112 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f4042773640 4 set_mon_vals no callback set 2023-10-30T12:00:41.112 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:41.113 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.079+0000 7f403affd640 10 monclient: dump: 2023-10-30T12:00:41.113 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.113 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.113 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.113 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.114 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.114 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.114 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.114 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.114 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.115 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.115 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 10 monclient: shutdown 2023-10-30T12:00:41.115 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.115 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 1 -- 172.21.15.118:0/2107440481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c13c180 msgr2=0x7f403c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.115 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 1 --2- 172.21.15.118:0/2107440481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c13c180 0x7f403c13e570 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4034009dc0 tx=0x7f403402be00 comp rx=0 tx=0).stop 2023-10-30T12:00:41.116 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 1 -- 172.21.15.118:0/2107440481 shutdown_connections 2023-10-30T12:00:41.116 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 1 --2- 172.21.15.118:0/2107440481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c13c180 0x7f403c13e570 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.116 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 1 -- 172.21.15.118:0/2107440481 >> 172.21.15.118:0/2107440481 conn(0x7f403c0a82f0 msgr2=0x7f403c0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.117 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 1 -- 172.21.15.118:0/2107440481 shutdown_connections 2023-10-30T12:00:41.117 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 1 -- 172.21.15.118:0/2107440481 wait complete. 2023-10-30T12:00:41.117 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.080+0000 7f4042773640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.117 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient: monmap: 2023-10-30T12:00:41.117 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.118 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:41.118 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2023-10-30T12:00:41.118 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2023-10-30T12:00:41.118 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.119 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.119 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:41.119 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:41.119 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:41.119 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.119 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 librados: starting msgr at 2023-10-30T12:00:41.120 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 librados: starting objecter 2023-10-30T12:00:41.120 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 Processor -- start 2023-10-30T12:00:41.120 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 -- start start 2023-10-30T12:00:41.120 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 librados: setting wanted keys 2023-10-30T12:00:41.120 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 librados: calling monclient init 2023-10-30T12:00:41.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient: init 2023-10-30T12:00:41.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:00:41.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 0x7f403c131210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.122 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient(hunting): picked mon.noname-c con 0x7f403c099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.122 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f403c131750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.122 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient(hunting): picked mon.noname-a con 0x7f403c1327f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:41.123 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 0x7f403c131c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.123 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient(hunting): picked mon.noname-b con 0x7f403c13c180 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:41.124 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.124 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.081+0000 7f4042773640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1388d0 con 0x7f403c1327f0 2023-10-30T12:00:41.124 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.125 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c1385e0 con 0x7f403c13c180 2023-10-30T12:00:41.125 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.125 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0a17f0 con 0x7f403c099a40 2023-10-30T12:00:41.126 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.126 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient(hunting): authenticate will time out at 872.420212s 2023-10-30T12:00:41.126 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 0x7f403c131c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.127 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 0x7f403c131c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51506/0 (socket says 172.21.15.118:51506) 2023-10-30T12:00:41.127 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 -- 172.21.15.118:0/2403525692 learned_addr learned my addr 172.21.15.118:0/2403525692 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.127 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): get_auth_request con 0x7f403c13c180 auth_method 0 2023-10-30T12:00:41.128 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.128 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.129 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.129 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.129 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.130 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.130 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403b7fe640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f403c131750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.130 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403bfff640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 0x7f403c131210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.131 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403b7fe640 10 monclient(hunting): get_auth_request con 0x7f403c1327f0 auth_method 0 2023-10-30T12:00:41.131 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.131 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.131 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.132 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403bfff640 10 monclient(hunting): get_auth_request con 0x7f403c099a40 auth_method 0 2023-10-30T12:00:41.132 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.132 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-10-30T12:00:41.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 msgr2=0x7f403c131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.134 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 0x7f403c131210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.134 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 msgr2=0x7f403c131750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.134 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f403c131750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.135 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.135 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 monclient: found mon.noname-b 2023-10-30T12:00:41.135 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.136 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.136 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4034009a70 con 0x7f403c13c180 2023-10-30T12:00:41.136 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403b7fe640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f403c131750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:41.137 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4040d10640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 0x7f403c131c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4034037040 tx=0x7f4034004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.137 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 client.?.objecter ms_handle_connect 0x7f403c13c180 2023-10-30T12:00:41.137 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:41.137 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034034ea0 con 0x7f403c13c180 2023-10-30T12:00:41.138 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.138 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:41.138 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: dump: 2023-10-30T12:00:41.138 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.139 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.139 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.139 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.139 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.139 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.139 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.140 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.140 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.140 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.140 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.140 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.141 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.141 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4034040650 con 0x7f403c13c180 2023-10-30T12:00:41.141 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 5 monclient: authenticate success, global_id 4120 2023-10-30T12:00:41.141 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient: _renew_subs 2023-10-30T12:00:41.141 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.141 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c13b830 con 0x7f403c13c180 2023-10-30T12:00:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034040a00 con 0x7f403c13c180 2023-10-30T12:00:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4038ff9640 10 monclient: dump: 2023-10-30T12:00:41.143 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.143 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.143 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.143 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.143 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.144 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.144 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.144 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.144 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.144 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.144 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f403a7fc640 4 set_mon_vals no callback set 2023-10-30T12:00:41.145 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:41.145 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient: _renew_subs 2023-10-30T12:00:41.145 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.145 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c139ff0 con 0x7f403c13c180 2023-10-30T12:00:41.145 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.082+0000 7f4042773640 1 librados: init done 2023-10-30T12:00:41.145 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4034049400 con 0x7f403c13c180 2023-10-30T12:00:41.146 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f403403e7b0 con 0x7f403c13c180 2023-10-30T12:00:41.146 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 10 client.4120.objecter ms_dispatch 0x7f403c000b90 osd_map(3..3 src has 1..3) v4 2023-10-30T12:00:41.146 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2023-10-30T12:00:41.146 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2023-10-30T12:00:41.146 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 20 client.4120.objecter dump_active .. 0 homeless 2023-10-30T12:00:41.147 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4042773640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.147 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4042773640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.147 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4042773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.147 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c073a00 con 0x7f403c13c180 2023-10-30T12:00:41.147 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4040d10640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 msgr2=0x7f403c131c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:00:41.148 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4040d10640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 msgr2=0x7f403c131c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:00:41.148 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4040d10640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 0x7f403c131c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4034037040 tx=0x7f4034004a40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:00:41.148 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4040d10640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 0x7f403c131c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4034037040 tx=0x7f4034004a40 comp rx=0 tx=0).stop 2023-10-30T12:00:41.148 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:41.148 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.148 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 msgr2=0x7f403c131c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:00:41.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f403c13c180 0x7f403c131c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:41.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.083+0000 7f4038ff9640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4028006970 0x7f4028008da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 monclient(hunting): picked mon.b con 0x7f4028006970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:41.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f402800b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 monclient(hunting): picked mon.a con 0x7f403c1327f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:41.150 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 0x7f402800e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.150 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 monclient(hunting): picked mon.c con 0x7f403c099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.150 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.150 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402800edb0 con 0x7f403c1327f0 2023-10-30T12:00:41.150 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.151 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402800eef0 con 0x7f4028006970 2023-10-30T12:00:41.151 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.151 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402800f0b0 con 0x7f403c099a40 2023-10-30T12:00:41.151 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.151 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4028006970 0x7f4028008da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): get_auth_request con 0x7f4028006970 auth_method 0 2023-10-30T12:00:41.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:41.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 0x7f402800e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f402800b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 10 monclient(hunting): get_auth_request con 0x7f403c099a40 auth_method 0 2023-10-30T12:00:41.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:41.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 10 monclient(hunting): get_auth_request con 0x7f403c1327f0 auth_method 0 2023-10-30T12:00:41.154 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.154 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.154 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:41.154 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.154 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.155 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:41.155 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.155 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.155 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4040d10640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:41.155 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.156 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.156 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:41.156 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2023-10-30T12:00:41.156 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 msgr2=0x7f402800e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.156 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 0x7f402800e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.156 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 msgr2=0x7f402800b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.157 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f402800b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.157 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.157 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 monclient: found mon.b 2023-10-30T12:00:41.157 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.157 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.158 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f402800f270 con 0x7f4028006970 2023-10-30T12:00:41.158 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.158 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.158 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f402c00f040 con 0x7f4028006970 2023-10-30T12:00:41.158 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.158 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.159 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.159 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 client.4120.objecter ms_handle_connect 0x7f4028006970 2023-10-30T12:00:41.159 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f4038ff9640 10 client.4120.objecter resend_mon_ops 2023-10-30T12:00:41.160 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.084+0000 7f403bfff640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4028006970 0x7f4028008da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f402c001a80 tx=0x7f4034030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.160 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c007ea0 con 0x7f4028006970 2023-10-30T12:00:41.160 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4038ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.161 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4038ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.161 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4038ff9640 10 monclient: dump: 2023-10-30T12:00:41.161 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.162 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.162 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.166 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.167 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.167 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.167 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.168 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.168 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.168 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c001ca0 con 0x7f4028006970 2023-10-30T12:00:41.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4038ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.170 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f402c00eaa0 con 0x7f4028006970 2023-10-30T12:00:41.170 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.086+0000 7f4039ffb640 4 set_mon_vals no callback set 2023-10-30T12:00:41.170 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.088+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f402c01a020 con 0x7f4028006970 2023-10-30T12:00:41.171 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.088+0000 7f4038ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.171 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.088+0000 7f4038ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:41.172 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.157+0000 7f4042773640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c13ebbef-aa1a-4da7-b5f6-26b02e43955c", "id": 1}] 2023-10-30T12:00:41.173 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.157+0000 7f4042773640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c13ebbef-aa1a-4da7-b5f6-26b02e43955c", "id": 1}] 2023-10-30T12:00:41.173 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.157+0000 7f4042773640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.173 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.157+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c13ebbef-aa1a-4da7-b5f6-26b02e43955c", "id": 1} v 0) v1 -- 0x7f403c04e1d0 con 0x7f4028006970 2023-10-30T12:00:41.174 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4038ff9640 1 -- 172.21.15.118:0/2403525692 <== mon.1 v2:172.21.15.118:3300/0 5 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c13ebbef-aa1a-4da7-b5f6-26b02e43955c", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f402c001330 con 0x7f4028006970 2023-10-30T12:00:41.174 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4038ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c13ebbef-aa1a-4da7-b5f6-26b02e43955c", "id": 1}] 2023-10-30T12:00:41.174 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4038ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:41.174 INFO:teuthology.orchestra.run.smithi118.stdout:1 2023-10-30T12:00:41.176 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 10 librados: watch_flush enter 2023-10-30T12:00:41.176 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 10 librados: watch_flush exit 2023-10-30T12:00:41.177 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 20 client.4120.objecter shutdown clearing up homeless session... 2023-10-30T12:00:41.177 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 10 client.4120.objecter successfully canceled tick 2023-10-30T12:00:41.177 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 10 monclient: shutdown 2023-10-30T12:00:41.177 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.177 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4028006970 msgr2=0x7f4028008da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.177 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4028006970 0x7f4028008da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f402c001a80 tx=0x7f4034030000 comp rx=0 tx=0).stop 2023-10-30T12:00:41.178 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 shutdown_connections 2023-10-30T12:00:41.178 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f403c099a40 0x7f402800e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.178 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.161+0000 7f4042773640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f403c1327f0 0x7f402800b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.178 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.162+0000 7f4042773640 1 --2- 172.21.15.118:0/2403525692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4028006970 0x7f4028008da0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.179 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.162+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 >> 172.21.15.118:0/2403525692 conn(0x7f403c0a82f0 msgr2=0x7f403c0a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.179 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.162+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 shutdown_connections 2023-10-30T12:00:41.179 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.162+0000 7f4042773640 1 -- 172.21.15.118:0/2403525692 wait complete. 2023-10-30T12:00:41.179 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.162+0000 7f4042773640 1 librados: shutdown 2023-10-30T12:00:41.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 23116074-425b-4664-a1be-0e40bba62fcb 2 2023-10-30T12:00:41.279 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.277+0000 7f2ab14d0640 10 monclient: get_monmap_and_config 2023-10-30T12:00:41.280 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.277+0000 7f2ab14d0640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.280 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.277+0000 7f2ab14d0640 10 monclient: monmap: 2023-10-30T12:00:41.280 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.280 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.281 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:41.279052+0000 2023-10-30T12:00:41.281 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:41.279052+0000 2023-10-30T12:00:41.281 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.281 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.281 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:41.282 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:41.282 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:41.282 INFO:teuthology.orchestra.run.smithi118.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:41.282 INFO:teuthology.orchestra.run.smithi118.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:41.282 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.290 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.277+0000 7f2ab14d0640 1 Processor -- start 2023-10-30T12:00:41.290 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.277+0000 7f2ab14d0640 1 -- start start 2023-10-30T12:00:41.291 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.277+0000 7f2ab14d0640 10 monclient: init 2023-10-30T12:00:41.291 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.291 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-10-30T12:00:41.291 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2aac1327f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:41.291 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2aac099a40 0x7f2aac13aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2aac099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac13afa0 0x7f2aac13d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2aac13afa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0a1620 con 0x7f2aac1327f0 2023-10-30T12:00:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac09a6b0 con 0x7f2aac099a40 2023-10-30T12:00:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0a1b20 con 0x7f2aac13afa0 2023-10-30T12:00:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2ab14d0640 10 monclient(hunting): authenticate will time out at 872.616559s 2023-10-30T12:00:41.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2aac099a40 0x7f2aac13aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2aac099a40 0x7f2aac13aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51518/0 (socket says 172.21.15.118:51518) 2023-10-30T12:00:41.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 -- 172.21.15.118:0/1450522379 learned_addr learned my addr 172.21.15.118:0/1450522379 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): get_auth_request con 0x7f2aac099a40 auth_method 0 2023-10-30T12:00:41.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.295 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.295 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.295 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.295 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.296 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.296 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aab7fe640 1 --2- 172.21.15.118:0/1450522379 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac13afa0 0x7f2aac13d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.296 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2023-10-30T12:00:41.297 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 -- 172.21.15.118:0/1450522379 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac13afa0 msgr2=0x7f2aac13d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:41.297 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 --2- 172.21.15.118:0/1450522379 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac13afa0 0x7f2aac13d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.297 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 -- 172.21.15.118:0/1450522379 >> v1:172.21.15.50:6789/0 conn(0x7f2aac1327f0 legacy=0x7f2aac132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:41.297 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.298 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 monclient: found mon.noname-c 2023-10-30T12:00:41.298 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.298 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.298 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aaa7fc640 1 -- 172.21.15.118:0/1450522379 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac09a0f0 con 0x7f2aac099a40 2023-10-30T12:00:41.298 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aab7fe640 10 monclient: get_auth_request con 0x7f2aac13afa0 auth_method 0 2023-10-30T12:00:41.299 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.278+0000 7f2aab7fe640 1 --2- 172.21.15.118:0/1450522379 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac13afa0 0x7f2aac13d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:41.299 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aaa7fc640 1 --2- 172.21.15.118:0/1450522379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac13aa60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009870 tx=0x7f2a9c02b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1e9c8488f13c224 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.299 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 1 -- 172.21.15.118:0/1450522379 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c031740 con 0x7f2aac099a40 2023-10-30T12:00:41.299 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.299 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:41.299 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: dump: 2023-10-30T12:00:41.300 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.300 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.300 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.300 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.300 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.301 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.302 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.302 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.302 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.302 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2ab14d0640 5 monclient: authenticate success, global_id 4123 2023-10-30T12:00:41.302 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 1 -- 172.21.15.118:0/1450522379 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c0318a0 con 0x7f2aac099a40 2023-10-30T12:00:41.303 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2ab14d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:41.303 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.303 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2ab14d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:41.303 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2ab14d0640 4 set_mon_vals no callback set 2023-10-30T12:00:41.303 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 1 -- 172.21.15.118:0/1450522379 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c031c30 con 0x7f2aac099a40 2023-10-30T12:00:41.303 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.304 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.304 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.279+0000 7f2aa9ffb640 10 monclient: dump: 2023-10-30T12:00:41.304 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.304 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.304 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.305 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.305 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.305 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.305 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.305 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.306 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.306 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.306 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 10 monclient: shutdown 2023-10-30T12:00:41.306 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.306 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 -- 172.21.15.118:0/1450522379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 msgr2=0x7f2aac13aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.307 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 --2- 172.21.15.118:0/1450522379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac13aa60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009870 tx=0x7f2a9c02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.307 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 -- 172.21.15.118:0/1450522379 shutdown_connections 2023-10-30T12:00:41.307 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 --2- 172.21.15.118:0/1450522379 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac13afa0 0x7f2aac13d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.307 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 --2- 172.21.15.118:0/1450522379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac13aa60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.307 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 -- 172.21.15.118:0/1450522379 >> 172.21.15.118:0/1450522379 conn(0x7f2aac0a82f0 msgr2=0x7f2aac0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.308 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 -- 172.21.15.118:0/1450522379 shutdown_connections 2023-10-30T12:00:41.308 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 1 -- 172.21.15.118:0/1450522379 wait complete. 2023-10-30T12:00:41.308 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.308 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.280+0000 7f2ab14d0640 10 monclient: monmap: 2023-10-30T12:00:41.308 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.309 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:41.309 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2023-10-30T12:00:41.309 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2023-10-30T12:00:41.309 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.309 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.309 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:41.310 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:41.310 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:41.310 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.310 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 librados: starting msgr at 2023-10-30T12:00:41.311 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 librados: starting objecter 2023-10-30T12:00:41.311 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 Processor -- start 2023-10-30T12:00:41.311 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 -- start start 2023-10-30T12:00:41.311 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 librados: setting wanted keys 2023-10-30T12:00:41.311 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 librados: calling monclient init 2023-10-30T12:00:41.312 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient: init 2023-10-30T12:00:41.312 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.312 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:41.312 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac1ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.312 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2aac099a40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:41.313 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac1327f0 0x7f2aac136150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.313 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2aac1327f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.313 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2aac13afa0 0x7f2aac136690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.313 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2aac13afa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:41.313 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.314 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac1317b0 con 0x7f2aac13afa0 2023-10-30T12:00:41.314 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.314 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0a17f0 con 0x7f2aac099a40 2023-10-30T12:00:41.314 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.314 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0a4280 con 0x7f2aac1327f0 2023-10-30T12:00:41.315 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.315 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2ab14d0640 10 monclient(hunting): authenticate will time out at 872.620109s 2023-10-30T12:00:41.315 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.281+0000 7f2aaaffd640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac1ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.315 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac1ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51522/0 (socket says 172.21.15.118:51522) 2023-10-30T12:00:41.315 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 -- 172.21.15.118:0/480483483 learned_addr learned my addr 172.21.15.118:0/480483483 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.315 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): get_auth_request con 0x7f2aac099a40 auth_method 0 2023-10-30T12:00:41.316 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.316 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.316 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.316 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.316 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.317 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.317 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aab7fe640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2aac13afa0 0x7f2aac136690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.317 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2023-10-30T12:00:41.317 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaa7fc640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac1327f0 0x7f2aac136150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.317 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 -- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac1327f0 msgr2=0x7f2aac136150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.318 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac1327f0 0x7f2aac136150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.318 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 -- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2aac13afa0 msgr2=0x7f2aac136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.318 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2aac13afa0 0x7f2aac136690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.318 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.318 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 monclient: found mon.noname-b 2023-10-30T12:00:41.319 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.319 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.319 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 -- 172.21.15.118:0/480483483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0094d0 con 0x7f2aac099a40 2023-10-30T12:00:41.319 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aaaffd640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac1ca840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c430 tx=0x7f2aa000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.319 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 client.?.objecter ms_handle_connect 0x7f2aac099a40 2023-10-30T12:00:41.320 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:41.320 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 1 -- 172.21.15.118:0/480483483 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0011790 con 0x7f2aac099a40 2023-10-30T12:00:41.320 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.320 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:41.320 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: dump: 2023-10-30T12:00:41.321 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.321 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.321 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.321 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.321 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.321 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.322 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.322 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.322 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.322 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.322 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 1 -- 172.21.15.118:0/480483483 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa00118f0 con 0x7f2aac099a40 2023-10-30T12:00:41.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 5 monclient: authenticate success, global_id 4126 2023-10-30T12:00:41.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 10 monclient: _renew_subs 2023-10-30T12:00:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac138760 con 0x7f2aac099a40 2023-10-30T12:00:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 1 -- 172.21.15.118:0/480483483 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0011ce0 con 0x7f2aac099a40 2023-10-30T12:00:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 monclient: dump: 2023-10-30T12:00:41.325 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.325 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.325 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.326 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.326 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.326 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.326 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.326 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.327 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.327 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.327 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:41.327 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2aa97fa640 4 set_mon_vals no callback set 2023-10-30T12:00:41.327 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 10 monclient: _renew_subs 2023-10-30T12:00:41.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac13a170 con 0x7f2aac099a40 2023-10-30T12:00:41.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2ab14d0640 1 librados: init done 2023-10-30T12:00:41.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 1 -- 172.21.15.118:0/480483483 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2aa001c760 con 0x7f2aac099a40 2023-10-30T12:00:41.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 1 -- 172.21.15.118:0/480483483 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f2aa0012b60 con 0x7f2aac099a40 2023-10-30T12:00:41.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 10 client.4126.objecter ms_dispatch 0x7f2aac000b90 osd_map(4..4 src has 1..4) v4 2023-10-30T12:00:41.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2023-10-30T12:00:41.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2023-10-30T12:00:41.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.282+0000 7f2a937fe640 20 client.4126.objecter dump_active .. 0 homeless 2023-10-30T12:00:41.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.283+0000 7f2ab14d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.283+0000 7f2ab14d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.283+0000 7f2ab14d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.283+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac04e1d0 con 0x7f2aac099a40 2023-10-30T12:00:41.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.285+0000 7f2a937fe640 1 -- 172.21.15.118:0/480483483 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2aa0012d40 con 0x7f2aac099a40 2023-10-30T12:00:41.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.285+0000 7f2a937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.285+0000 7f2a937fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:41.379 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.376+0000 7f2ab14d0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "23116074-425b-4664-a1be-0e40bba62fcb", "id": 2}] 2023-10-30T12:00:41.379 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.376+0000 7f2ab14d0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "23116074-425b-4664-a1be-0e40bba62fcb", "id": 2}] 2023-10-30T12:00:41.379 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.376+0000 7f2ab14d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.379 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.376+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "23116074-425b-4664-a1be-0e40bba62fcb", "id": 2} v 0) v1 -- 0x7f2aac074510 con 0x7f2aac099a40 2023-10-30T12:00:41.382 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2a937fe640 1 -- 172.21.15.118:0/480483483 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23116074-425b-4664-a1be-0e40bba62fcb", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2aa0012740 con 0x7f2aac099a40 2023-10-30T12:00:41.382 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2a937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "23116074-425b-4664-a1be-0e40bba62fcb", "id": 2}] 2023-10-30T12:00:41.382 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2a937fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:41.383 INFO:teuthology.orchestra.run.smithi118.stdout:2 2023-10-30T12:00:41.384 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 10 librados: watch_flush enter 2023-10-30T12:00:41.384 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 10 librados: watch_flush exit 2023-10-30T12:00:41.384 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 20 client.4126.objecter shutdown clearing up homeless session... 2023-10-30T12:00:41.385 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 10 client.4126.objecter successfully canceled tick 2023-10-30T12:00:41.385 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 10 monclient: shutdown 2023-10-30T12:00:41.385 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.385 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 msgr2=0x7f2aac1ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.385 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.380+0000 7f2ab14d0640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac1ca840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c430 tx=0x7f2aa000c8f0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.386 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 shutdown_connections 2023-10-30T12:00:41.386 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2aac13afa0 0x7f2aac136690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.386 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2aac1327f0 0x7f2aac136150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.386 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 --2- 172.21.15.118:0/480483483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2aac099a40 0x7f2aac1ca840 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.386 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 >> 172.21.15.118:0/480483483 conn(0x7f2aac0a82f0 msgr2=0x7f2aac0a3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.387 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 shutdown_connections 2023-10-30T12:00:41.387 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 -- 172.21.15.118:0/480483483 wait complete. 2023-10-30T12:00:41.387 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.381+0000 7f2ab14d0640 1 librados: shutdown 2023-10-30T12:00:41.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 515e513c-cb67-4495-ba25-68cb35215c58 3 2023-10-30T12:00:41.409 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.407+0000 7f3dd2056040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:00:41.465 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.462+0000 7f3dd2056040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:00:41.493 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.490+0000 7fc0ad333640 10 monclient: get_monmap_and_config 2023-10-30T12:00:41.493 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.490+0000 7fc0ad333640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.493 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.490+0000 7fc0ad333640 10 monclient: monmap: 2023-10-30T12:00:41.493 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.493 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:41.492274+0000 2023-10-30T12:00:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:41.492274+0000 2023-10-30T12:00:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:41.495 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.504 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.490+0000 7fc0ad333640 1 Processor -- start 2023-10-30T12:00:41.504 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 1 -- start start 2023-10-30T12:00:41.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient: init 2023-10-30T12:00:41.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T12:00:41.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc0a8069de0 0x7fc0a806b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient(hunting): picked mon.noname-c con 0x7fc0a8069de0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:41.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0a806a1c0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:41.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0a813b0e0 0x7fc0a813d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient(hunting): picked mon.noname-a con 0x7fc0a813b0e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:41.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a8071180 con 0x7fc0a806a1c0 2023-10-30T12:00:41.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a80a5240 con 0x7fc0a813b0e0 2023-10-30T12:00:41.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a809e060 con 0x7fc0a8069de0 2023-10-30T12:00:41.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0ad333640 10 monclient(hunting): authenticate will time out at 872.829753s 2023-10-30T12:00:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc0a8069de0 0x7fc0a806b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a67fc640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc0a806a1c0 0x7fc0a806a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:50116/0 (socket says 172.21.15.118:50116) 2023-10-30T12:00:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a67fc640 1 -- 172.21.15.118:0/35922626 learned_addr learned my addr 172.21.15.118:0/35922626 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-30T12:00:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): get_auth_request con 0x7fc0a8069de0 auth_method 0 2023-10-30T12:00:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2023-10-30T12:00:41.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 1 -- 172.21.15.118:0/35922626 >> v2:172.21.15.50:3300/0 conn(0x7fc0a813b0e0 msgr2=0x7fc0a813d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:41.510 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a77fe640 1 --2- 172.21.15.118:0/35922626 >> v2:172.21.15.50:3300/0 conn(0x7fc0a813b0e0 0x7fc0a813d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.510 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 1 --2- 172.21.15.118:0/35922626 >> v2:172.21.15.50:3300/0 conn(0x7fc0a813b0e0 0x7fc0a813d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.510 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a5ffb640 1 -- 172.21.15.118:0/35922626 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340009466 0 0) 0x7fc0a8071180 con 0x7fc0a806a1c0 2023-10-30T12:00:41.510 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 1 -- 172.21.15.118:0/35922626 >> v1:172.21.15.118:6789/0 conn(0x7fc0a806a1c0 legacy=0x7fc0a806a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.510 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.491+0000 7fc0a6ffd640 1 monclient: found mon.noname-c 2023-10-30T12:00:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a6ffd640 1 -- 172.21.15.118:0/35922626 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80a4a40 con 0x7fc0a8069de0 2023-10-30T12:00:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a77fe640 10 monclient: get_auth_request con 0x7fc0a813b0e0 auth_method 0 2023-10-30T12:00:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a77fe640 1 --2- 172.21.15.118:0/35922626 >> v2:172.21.15.50:3300/0 conn(0x7fc0a813b0e0 0x7fc0a813d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a6ffd640 1 --2- 172.21.15.118:0/35922626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a8069de0 0x7fc0a806b120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80a3430 tx=0x7fc09802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a49da894441937 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 1 -- 172.21.15.118:0/35922626 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098031710 con 0x7fc0a8069de0 2023-10-30T12:00:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: dump: 2023-10-30T12:00:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.515 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0ad333640 5 monclient: authenticate success, global_id 4129 2023-10-30T12:00:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 1 -- 172.21.15.118:0/35922626 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09802f050 con 0x7fc0a8069de0 2023-10-30T12:00:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0ad333640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0ad333640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0ad333640 4 set_mon_vals no callback set 2023-10-30T12:00:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 1 -- 172.21.15.118:0/35922626 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc098031a90 con 0x7fc0a8069de0 2023-10-30T12:00:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.492+0000 7fc0a5ffb640 10 monclient: dump: 2023-10-30T12:00:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.519 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 10 monclient: shutdown 2023-10-30T12:00:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.520 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 -- 172.21.15.118:0/35922626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a8069de0 msgr2=0x7fc0a806b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.520 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 --2- 172.21.15.118:0/35922626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a8069de0 0x7fc0a806b120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80a3430 tx=0x7fc09802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.520 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 -- 172.21.15.118:0/35922626 shutdown_connections 2023-10-30T12:00:41.520 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 --2- 172.21.15.118:0/35922626 >> v2:172.21.15.50:3300/0 conn(0x7fc0a813b0e0 0x7fc0a813d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.520 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 --2- 172.21.15.118:0/35922626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a8069de0 0x7fc0a806b120 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.520 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 -- 172.21.15.118:0/35922626 >> 172.21.15.118:0/35922626 conn(0x7fc0a80a6340 msgr2=0x7fc0a80a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.521 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 -- 172.21.15.118:0/35922626 shutdown_connections 2023-10-30T12:00:41.521 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 1 -- 172.21.15.118:0/35922626 wait complete. 2023-10-30T12:00:41.521 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.521 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.493+0000 7fc0ad333640 10 monclient: monmap: 2023-10-30T12:00:41.521 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.522 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:41.522 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2023-10-30T12:00:41.522 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2023-10-30T12:00:41.522 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.522 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.523 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:41.523 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:41.523 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:41.523 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.523 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 librados: starting msgr at 2023-10-30T12:00:41.524 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 librados: starting objecter 2023-10-30T12:00:41.524 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 Processor -- start 2023-10-30T12:00:41.524 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 -- start start 2023-10-30T12:00:41.524 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 librados: setting wanted keys 2023-10-30T12:00:41.524 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 librados: calling monclient init 2023-10-30T12:00:41.524 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient: init 2023-10-30T12:00:41.525 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.525 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:00:41.525 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0a8069de0 0x7fc0a8139440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.525 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): picked mon.noname-c con 0x7fc0a8069de0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.525 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0a806a1c0 0x7fc0a81c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.526 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): picked mon.noname-a con 0x7fc0a806a1c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:41.526 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a813b0e0 0x7fc0a8135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.526 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): picked mon.noname-b con 0x7fc0a813b0e0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:41.526 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.526 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a813a030 con 0x7fc0a806a1c0 2023-10-30T12:00:41.526 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.527 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a811e9f0 con 0x7fc0a813b0e0 2023-10-30T12:00:41.527 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.527 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a811ecd0 con 0x7fc0a8069de0 2023-10-30T12:00:41.527 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.527 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0ad333640 10 monclient(hunting): authenticate will time out at 872.833105s 2023-10-30T12:00:41.528 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.494+0000 7fc0a77fe640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a813b0e0 0x7fc0a8135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.528 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a813b0e0 0x7fc0a8135610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40920/0 (socket says 172.21.15.118:40920) 2023-10-30T12:00:41.528 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 -- 172.21.15.118:0/3992579585 learned_addr learned my addr 172.21.15.118:0/3992579585 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.528 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): get_auth_request con 0x7fc0a813b0e0 auth_method 0 2023-10-30T12:00:41.528 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.528 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.529 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.529 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.529 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.529 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.529 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a67fc640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0a806a1c0 0x7fc0a81c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.530 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2023-10-30T12:00:41.530 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a6ffd640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0a8069de0 0x7fc0a8139440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.530 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 -- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0a8069de0 msgr2=0x7fc0a8139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.530 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0a8069de0 0x7fc0a8139440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.530 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 -- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0a806a1c0 msgr2=0x7fc0a81c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.531 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0a806a1c0 0x7fc0a81c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.531 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.531 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 monclient: found mon.noname-b 2023-10-30T12:00:41.531 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.531 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.531 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 -- 172.21.15.118:0/3992579585 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc098009400 con 0x7fc0a813b0e0 2023-10-30T12:00:41.532 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a67fc640 10 monclient: get_auth_request con 0x7fc0a806a1c0 auth_method 0 2023-10-30T12:00:41.532 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a67fc640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0a806a1c0 0x7fc0a81c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:41.532 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a77fe640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a813b0e0 0x7fc0a8135610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000cef0 tx=0x7fc0a000a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.532 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 client.?.objecter ms_handle_connect 0x7fc0a813b0e0 2023-10-30T12:00:41.532 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:41.533 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 1 -- 172.21.15.118:0/3992579585 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a000d840 con 0x7fc0a813b0e0 2023-10-30T12:00:41.533 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.533 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:41.533 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: dump: 2023-10-30T12:00:41.533 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.533 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.534 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.534 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.534 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.534 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.534 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.535 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.535 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.535 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.535 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.535 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.536 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.536 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 5 monclient: authenticate success, global_id 4135 2023-10-30T12:00:41.536 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 1 -- 172.21.15.118:0/3992579585 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a000d9c0 con 0x7fc0a813b0e0 2023-10-30T12:00:41.536 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.536 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 1 -- 172.21.15.118:0/3992579585 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a000dd70 con 0x7fc0a813b0e0 2023-10-30T12:00:41.537 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.537 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.537 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 monclient: dump: 2023-10-30T12:00:41.537 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.537 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.538 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.538 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.538 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.538 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.538 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.539 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.539 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.539 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.539 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 10 monclient: _renew_subs 2023-10-30T12:00:41.539 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0a57fa640 4 set_mon_vals no callback set 2023-10-30T12:00:41.540 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.540 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a811efb0 con 0x7fc0a813b0e0 2023-10-30T12:00:41.540 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:41.540 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 10 monclient: _renew_subs 2023-10-30T12:00:41.540 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.540 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a8135b50 con 0x7fc0a813b0e0 2023-10-30T12:00:41.541 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc0ad333640 1 librados: init done 2023-10-30T12:00:41.541 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 1 -- 172.21.15.118:0/3992579585 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc0a001f740 con 0x7fc0a813b0e0 2023-10-30T12:00:41.541 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 1 -- 172.21.15.118:0/3992579585 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fc0a0015c40 con 0x7fc0a813b0e0 2023-10-30T12:00:41.541 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 10 client.4135.objecter ms_dispatch 0x7fc0a8000b90 osd_map(5..5 src has 1..5) v4 2023-10-30T12:00:41.541 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2023-10-30T12:00:41.541 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.495+0000 7fc08f7fe640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2023-10-30T12:00:41.542 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.496+0000 7fc08f7fe640 20 client.4135.objecter dump_active .. 0 homeless 2023-10-30T12:00:41.542 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.497+0000 7fc0ad333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.542 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.497+0000 7fc0ad333640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.542 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.497+0000 7fc0ad333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.542 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.497+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a80a5430 con 0x7fc0a813b0e0 2023-10-30T12:00:41.543 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.498+0000 7fc08f7fe640 1 -- 172.21.15.118:0/3992579585 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc0a001a070 con 0x7fc0a813b0e0 2023-10-30T12:00:41.543 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.499+0000 7fc08f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.543 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.499+0000 7fc08f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:41.543 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.510+0000 7f3dd2056040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:00:41.568 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.565+0000 7f3dd2056040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:00:41.574 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.570+0000 7fc0ad333640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "515e513c-cb67-4495-ba25-68cb35215c58", "id": 3}] 2023-10-30T12:00:41.574 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.570+0000 7fc0ad333640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "515e513c-cb67-4495-ba25-68cb35215c58", "id": 3}] 2023-10-30T12:00:41.575 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.570+0000 7fc0ad333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.575 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.570+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "515e513c-cb67-4495-ba25-68cb35215c58", "id": 3} v 0) v1 -- 0x7fc0a804e1d0 con 0x7fc0a813b0e0 2023-10-30T12:00:41.576 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.573+0000 7fc08f7fe640 1 -- 172.21.15.118:0/3992579585 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "515e513c-cb67-4495-ba25-68cb35215c58", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc0a0015e50 con 0x7fc0a813b0e0 2023-10-30T12:00:41.576 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.573+0000 7fc08f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "515e513c-cb67-4495-ba25-68cb35215c58", "id": 3}] 2023-10-30T12:00:41.576 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.573+0000 7fc08f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:41.577 INFO:teuthology.orchestra.run.smithi118.stdout:3 2023-10-30T12:00:41.578 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 10 librados: watch_flush enter 2023-10-30T12:00:41.578 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 10 librados: watch_flush exit 2023-10-30T12:00:41.578 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 20 client.4135.objecter shutdown clearing up homeless session... 2023-10-30T12:00:41.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 10 client.4135.objecter successfully canceled tick 2023-10-30T12:00:41.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 10 monclient: shutdown 2023-10-30T12:00:41.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a813b0e0 msgr2=0x7fc0a8135610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a813b0e0 0x7fc0a8135610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000cef0 tx=0x7fc0a000a710 comp rx=0 tx=0).stop 2023-10-30T12:00:41.579 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 shutdown_connections 2023-10-30T12:00:41.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0a813b0e0 0x7fc0a8135610 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0a806a1c0 0x7fc0a81c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 --2- 172.21.15.118:0/3992579585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0a8069de0 0x7fc0a8139440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 >> 172.21.15.118:0/3992579585 conn(0x7fc0a80a6340 msgr2=0x7fc0a813c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.580 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 shutdown_connections 2023-10-30T12:00:41.581 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 -- 172.21.15.118:0/3992579585 wait complete. 2023-10-30T12:00:41.581 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.574+0000 7fc0ad333640 1 librados: shutdown 2023-10-30T12:00:41.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new fa9df8af-676c-49c6-9888-0e330fbbc4ac 4 2023-10-30T12:00:41.612 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.609+0000 7f3dd2056040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:00:41.714 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.708+0000 7f79ec439640 10 monclient: get_monmap_and_config 2023-10-30T12:00:41.715 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.708+0000 7f79ec439640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.715 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.708+0000 7f79ec439640 10 monclient: monmap: 2023-10-30T12:00:41.715 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.715 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.716 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:41.709938+0000 2023-10-30T12:00:41.716 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:41.709938+0000 2023-10-30T12:00:41.716 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.716 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.716 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:41.716 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:41.717 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:41.717 INFO:teuthology.orchestra.run.smithi118.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:41.717 INFO:teuthology.orchestra.run.smithi118.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:41.717 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.717 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.708+0000 7f79ec439640 1 Processor -- start 2023-10-30T12:00:41.718 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.708+0000 7f79ec439640 1 -- start start 2023-10-30T12:00:41.718 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.708+0000 7f79ec439640 10 monclient: init 2023-10-30T12:00:41.718 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.718 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T12:00:41.718 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f79e41327f0 0x7f79e4132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.719 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): picked mon.noname-c con 0x7f79e41327f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:41.719 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e4135700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.719 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): picked mon.noname-b con 0x7f79e4099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.719 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f79e4135c40 0x7f79e4136090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.719 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): picked mon.noname-a con 0x7f79e4135c40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:41.720 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.720 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e41365d0 con 0x7f79e4135c40 2023-10-30T12:00:41.720 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.720 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e409a6b0 con 0x7f79e41327f0 2023-10-30T12:00:41.721 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.721 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40a1620 con 0x7f79e4099a40 2023-10-30T12:00:41.721 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.721 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ec439640 10 monclient(hunting): authenticate will time out at 873.047529s 2023-10-30T12:00:41.722 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f79e41327f0 0x7f79e4132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.722 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f79e41327f0 0x7f79e4132bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40934/0 (socket says 172.21.15.118:40934) 2023-10-30T12:00:41.722 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 -- 172.21.15.118:0/1543599157 learned_addr learned my addr 172.21.15.118:0/1543599157 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.722 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): get_auth_request con 0x7f79e41327f0 auth_method 0 2023-10-30T12:00:41.722 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.723 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.723 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.723 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.724 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.724 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.724 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79e99d4640 1 --2- 172.21.15.118:0/1543599157 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e4135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.725 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea9d6640 1 --2- 172.21.15.118:0/1543599157 >> v2:172.21.15.50:3300/0 conn(0x7f79e4135c40 0x7f79e4136090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.725 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79e99d4640 10 monclient(hunting): get_auth_request con 0x7f79e4099a40 auth_method 0 2023-10-30T12:00:41.725 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79e99d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.726 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79e99d4640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.726 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79e99d4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.726 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea9d6640 10 monclient(hunting): get_auth_request con 0x7f79e4135c40 auth_method 0 2023-10-30T12:00:41.727 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea9d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.727 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea9d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.727 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea9d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.728 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2023-10-30T12:00:41.728 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 -- 172.21.15.118:0/1543599157 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 msgr2=0x7f79e4135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:41.728 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 --2- 172.21.15.118:0/1543599157 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e4135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.729 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 -- 172.21.15.118:0/1543599157 >> v2:172.21.15.50:3300/0 conn(0x7f79e4135c40 msgr2=0x7f79e4136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:41.729 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 --2- 172.21.15.118:0/1543599157 >> v2:172.21.15.50:3300/0 conn(0x7f79e4135c40 0x7f79e4136090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.729 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.729 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 monclient: found mon.noname-c 2023-10-30T12:00:41.730 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.730 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.731 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.709+0000 7f79ea1d5640 1 -- 172.21.15.118:0/1543599157 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40a1b20 con 0x7f79e41327f0 2023-10-30T12:00:41.732 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79ea1d5640 1 --2- 172.21.15.118:0/1543599157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e41327f0 0x7f79e4132bd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79e40a8a50 tx=0x7f79e002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a9f219f2034b3a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.732 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 1 -- 172.21.15.118:0/1543599157 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0031750 con 0x7f79e41327f0 2023-10-30T12:00:41.732 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.732 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:41.732 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: dump: 2023-10-30T12:00:41.733 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.733 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.733 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.733 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.733 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.733 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.734 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.734 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.734 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.734 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.734 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.735 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.735 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.735 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 1 -- 172.21.15.118:0/1543599157 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e00318b0 con 0x7f79e41327f0 2023-10-30T12:00:41.735 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79ec439640 5 monclient: authenticate success, global_id 4138 2023-10-30T12:00:41.735 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79ec439640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:41.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79ec439640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:41.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 1 -- 172.21.15.118:0/1543599157 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0031c40 con 0x7f79e41327f0 2023-10-30T12:00:41.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79ec439640 4 set_mon_vals no callback set 2023-10-30T12:00:41.737 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.737 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.710+0000 7f79e91d3640 10 monclient: dump: 2023-10-30T12:00:41.737 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.737 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.737 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.738 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.738 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.738 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.738 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.738 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.738 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.739 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.739 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 10 monclient: shutdown 2023-10-30T12:00:41.739 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.739 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 -- 172.21.15.118:0/1543599157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e41327f0 msgr2=0x7f79e4132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.739 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 --2- 172.21.15.118:0/1543599157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e41327f0 0x7f79e4132bd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79e40a8a50 tx=0x7f79e002b7f0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.740 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 -- 172.21.15.118:0/1543599157 shutdown_connections 2023-10-30T12:00:41.740 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 --2- 172.21.15.118:0/1543599157 >> v2:172.21.15.50:3300/0 conn(0x7f79e4135c40 0x7f79e4136090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.740 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 --2- 172.21.15.118:0/1543599157 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e4135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.740 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 --2- 172.21.15.118:0/1543599157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e41327f0 0x7f79e4132bd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.740 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 -- 172.21.15.118:0/1543599157 >> 172.21.15.118:0/1543599157 conn(0x7f79e40a82f0 msgr2=0x7f79e40a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.741 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 -- 172.21.15.118:0/1543599157 shutdown_connections 2023-10-30T12:00:41.741 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.711+0000 7f79ec439640 1 -- 172.21.15.118:0/1543599157 wait complete. 2023-10-30T12:00:41.741 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.741 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient: monmap: 2023-10-30T12:00:41.741 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.742 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:41.742 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2023-10-30T12:00:41.742 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2023-10-30T12:00:41.742 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.742 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.743 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:41.743 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:41.743 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:41.743 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.743 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 librados: starting msgr at 2023-10-30T12:00:41.744 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 librados: starting objecter 2023-10-30T12:00:41.744 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 Processor -- start 2023-10-30T12:00:41.744 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 -- start start 2023-10-30T12:00:41.744 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 librados: setting wanted keys 2023-10-30T12:00:41.744 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 librados: calling monclient init 2023-10-30T12:00:41.744 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient: init 2023-10-30T12:00:41.749 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.749 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:00:41.749 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e409e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.749 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): picked mon.noname-c con 0x7f79e4099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79e41327f0 0x7f79e409ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): picked mon.noname-a con 0x7f79e41327f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:41.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e4135c40 0x7f79e409efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): picked mon.noname-b con 0x7f79e4135c40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:41.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.751 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40a1860 con 0x7f79e41327f0 2023-10-30T12:00:41.751 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.751 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e411ed70 con 0x7f79e4135c40 2023-10-30T12:00:41.751 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.751 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40a4110 con 0x7f79e4099a40 2023-10-30T12:00:41.752 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.752 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ec439640 10 monclient(hunting): authenticate will time out at 873.051029s 2023-10-30T12:00:41.752 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ea9d6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e4135c40 0x7f79e409efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.752 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ea9d6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e4135c40 0x7f79e409efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40938/0 (socket says 172.21.15.118:40938) 2023-10-30T12:00:41.752 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ea9d6640 1 -- 172.21.15.118:0/210777052 learned_addr learned my addr 172.21.15.118:0/210777052 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ea9d6640 10 monclient(hunting): get_auth_request con 0x7f79e4135c40 auth_method 0 2023-10-30T12:00:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ea9d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ea9d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.712+0000 7f79ea9d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.754 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.754 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79e99d4640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79e41327f0 0x7f79e409ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.754 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e409e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.754 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79e99d4640 10 monclient(hunting): get_auth_request con 0x7f79e41327f0 auth_method 0 2023-10-30T12:00:41.754 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79e99d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.755 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79e99d4640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.755 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79e99d4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.755 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 10 monclient(hunting): get_auth_request con 0x7f79e4099a40 auth_method 0 2023-10-30T12:00:41.755 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.756 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.756 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.756 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 1 -- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 msgr2=0x7f79e409e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:00:41.756 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 1 -- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 msgr2=0x7f79e409e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:00:41.756 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e409e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:00:41.757 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2023-10-30T12:00:41.757 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea1d5640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e409e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:00:41.757 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 1 -- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 msgr2=0x7f79e409e540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:00:41.757 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e409e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.757 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 1 -- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79e41327f0 msgr2=0x7f79e409ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.757 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79e41327f0 0x7f79e409ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.758 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.758 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 1 monclient: found mon.noname-b 2023-10-30T12:00:41.758 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.758 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.758 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 1 -- 172.21.15.118:0/210777052 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e00094d0 con 0x7f79e4135c40 2023-10-30T12:00:41.759 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79e99d4640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79e41327f0 0x7f79e409ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:41.759 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ea9d6640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e4135c40 0x7f79e409efc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79dc007e40 tx=0x7f79dc00e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.759 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 client.?.objecter ms_handle_connect 0x7f79e4135c40 2023-10-30T12:00:41.759 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:41.759 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 1 -- 172.21.15.118:0/210777052 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc014620 con 0x7f79e4135c40 2023-10-30T12:00:41.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:41.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: dump: 2023-10-30T12:00:41.760 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.760 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.761 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.761 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.761 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.761 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.761 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.762 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.762 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.762 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.762 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.762 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 1 -- 172.21.15.118:0/210777052 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc00ee90 con 0x7f79e4135c40 2023-10-30T12:00:41.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 5 monclient: authenticate success, global_id 4141 2023-10-30T12:00:41.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 10 monclient: _renew_subs 2023-10-30T12:00:41.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.763 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e411ea90 con 0x7f79e4135c40 2023-10-30T12:00:41.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 1 -- 172.21.15.118:0/210777052 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc0149e0 con 0x7f79e4135c40 2023-10-30T12:00:41.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.764 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 monclient: dump: 2023-10-30T12:00:41.765 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.765 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.765 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.765 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.765 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.765 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.766 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.766 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.766 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.766 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.766 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79e89d2640 4 set_mon_vals no callback set 2023-10-30T12:00:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 10 monclient: _renew_subs 2023-10-30T12:00:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e411e050 con 0x7f79e4135c40 2023-10-30T12:00:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79ec439640 1 librados: init done 2023-10-30T12:00:41.768 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 1 -- 172.21.15.118:0/210777052 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f79dc01e460 con 0x7f79e4135c40 2023-10-30T12:00:41.768 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 1 -- 172.21.15.118:0/210777052 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f79dc01d8d0 con 0x7f79e4135c40 2023-10-30T12:00:41.768 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 10 client.4141.objecter ms_dispatch 0x7f79e4000b90 osd_map(6..6 src has 1..6) v4 2023-10-30T12:00:41.768 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2023-10-30T12:00:41.768 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2023-10-30T12:00:41.768 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.713+0000 7f79daffd640 20 client.4141.objecter dump_active .. 0 homeless 2023-10-30T12:00:41.769 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.714+0000 7f79ec439640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.769 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.714+0000 7f79ec439640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.769 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.714+0000 7f79ec439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.769 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.714+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e404e1d0 con 0x7f79e4135c40 2023-10-30T12:00:41.769 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.716+0000 7f79daffd640 1 -- 172.21.15.118:0/210777052 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f79dc01dc60 con 0x7f79e4135c40 2023-10-30T12:00:41.770 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.716+0000 7f79daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.770 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.716+0000 7f3dd2056040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-30T12:00:41.771 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.716+0000 7f79daffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:41.790 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.787+0000 7f79ec439640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fa9df8af-676c-49c6-9888-0e330fbbc4ac", "id": 4}] 2023-10-30T12:00:41.790 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.787+0000 7f79ec439640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fa9df8af-676c-49c6-9888-0e330fbbc4ac", "id": 4}] 2023-10-30T12:00:41.790 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.787+0000 7f79ec439640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.790 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.787+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fa9df8af-676c-49c6-9888-0e330fbbc4ac", "id": 4} v 0) v1 -- 0x7f79e4074510 con 0x7f79e4135c40 2023-10-30T12:00:41.791 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.788+0000 7f3dd2056040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:00:41.793 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.790+0000 7f79daffd640 1 -- 172.21.15.118:0/210777052 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fa9df8af-676c-49c6-9888-0e330fbbc4ac", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f79dc01de00 con 0x7f79e4135c40 2023-10-30T12:00:41.793 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.790+0000 7f79daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fa9df8af-676c-49c6-9888-0e330fbbc4ac", "id": 4}] 2023-10-30T12:00:41.793 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.790+0000 7f79daffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:41.793 INFO:teuthology.orchestra.run.smithi118.stdout:4 2023-10-30T12:00:41.795 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 10 librados: watch_flush enter 2023-10-30T12:00:41.795 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 10 librados: watch_flush exit 2023-10-30T12:00:41.795 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 20 client.4141.objecter shutdown clearing up homeless session... 2023-10-30T12:00:41.795 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 10 client.4141.objecter successfully canceled tick 2023-10-30T12:00:41.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 10 monclient: shutdown 2023-10-30T12:00:41.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e4135c40 msgr2=0x7f79e409efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e4135c40 0x7f79e409efc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79dc007e40 tx=0x7f79dc00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 shutdown_connections 2023-10-30T12:00:41.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79e4135c40 0x7f79e409efc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79e41327f0 0x7f79e409ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 --2- 172.21.15.118:0/210777052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f79e4099a40 0x7f79e409e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 >> 172.21.15.118:0/210777052 conn(0x7f79e40a82f0 msgr2=0x7f79e4131a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 shutdown_connections 2023-10-30T12:00:41.798 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 -- 172.21.15.118:0/210777052 wait complete. 2023-10-30T12:00:41.798 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.791+0000 7f79ec439640 1 librados: shutdown 2023-10-30T12:00:41.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 79b07bf0-2944-4413-b6e1-23ecbc533add 5 2023-10-30T12:00:41.840 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.837+0000 7f3dd2056040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:00:41.895 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.892+0000 7f3dd2056040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:00:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.899+0000 7fd47d1b6640 10 monclient: get_monmap_and_config 2023-10-30T12:00:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.899+0000 7fd47d1b6640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.899+0000 7fd47d1b6640 10 monclient: monmap: 2023-10-30T12:00:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:41.901616+0000 2023-10-30T12:00:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:41.901616+0000 2023-10-30T12:00:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:41.905 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 1 Processor -- start 2023-10-30T12:00:41.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 1 -- start start 2023-10-30T12:00:41.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient: init 2023-10-30T12:00:41.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.913 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-10-30T12:00:41.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4781327f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:41.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd478099a40 0x7fd47813aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient(hunting): picked mon.noname-a con 0x7fd478099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:41.914 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd47813afa0 0x7fd47813d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.915 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient(hunting): picked mon.noname-b con 0x7fd47813afa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.915 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780a1620 con 0x7fd4781327f0 2023-10-30T12:00:41.915 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.915 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47809a6b0 con 0x7fd478099a40 2023-10-30T12:00:41.915 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.916 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780a1b20 con 0x7fd47813afa0 2023-10-30T12:00:41.916 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.916 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.900+0000 7fd47d1b6640 10 monclient(hunting): authenticate will time out at 873.239017s 2023-10-30T12:00:41.916 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd47813afa0 0x7fd47813d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.916 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd478099a40 0x7fd47813aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd47813afa0 0x7fd47813d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.118:60714/0 (socket says 172.21.15.118:60714) 2023-10-30T12:00:41.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd478099a40 0x7fd47813aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.118:45412/0 (socket says 172.21.15.118:45412) 2023-10-30T12:00:41.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 -- 172.21.15.118:0/4209623809 learned_addr learned my addr 172.21.15.118:0/4209623809 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): get_auth_request con 0x7fd47813afa0 auth_method 0 2023-10-30T12:00:41.917 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd476d9d640 1 --1- 172.21.15.118:0/4209623809 >> v1:172.21.15.50:6789/0 conn(0x7fd4781327f0 0x7fd478132bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.118:59552/0 (socket says 172.21.15.118:59552) 2023-10-30T12:00:41.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 10 monclient(hunting): get_auth_request con 0x7fd478099a40 auth_method 0 2023-10-30T12:00:41.918 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.919 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2023-10-30T12:00:41.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 -- 172.21.15.118:0/4209623809 >> v2:172.21.15.50:3300/0 conn(0x7fd478099a40 msgr2=0x7fd47813aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:41.920 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 --2- 172.21.15.118:0/4209623809 >> v2:172.21.15.50:3300/0 conn(0x7fd478099a40 0x7fd47813aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.921 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 -- 172.21.15.118:0/4209623809 >> v1:172.21.15.50:6789/0 conn(0x7fd4781327f0 legacy=0x7fd478132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.921 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.921 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 monclient: found mon.noname-b 2023-10-30T12:00:41.921 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.921 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:41.921 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47759e640 1 -- 172.21.15.118:0/4209623809 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47809a0f0 con 0x7fd47813afa0 2023-10-30T12:00:41.922 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.901+0000 7fd47659c640 1 --2- 172.21.15.118:0/4209623809 >> v2:172.21.15.50:3300/0 conn(0x7fd478099a40 0x7fd47813aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:41.922 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd47759e640 1 --2- 172.21.15.118:0/4209623809 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd47813afa0 0x7fd47813d390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd470009e50 tx=0x7fd47002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d06c5563af120e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.922 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 1 -- 172.21.15.118:0/4209623809 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47002c9e0 con 0x7fd47813afa0 2023-10-30T12:00:41.922 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.923 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:41.923 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: dump: 2023-10-30T12:00:41.923 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.923 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.923 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.923 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.924 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.924 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.924 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.924 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.924 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.924 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.925 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.925 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.925 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.925 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd47d1b6640 5 monclient: authenticate success, global_id 4118 2023-10-30T12:00:41.925 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 1 -- 172.21.15.118:0/4209623809 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47002cb40 con 0x7fd47813afa0 2023-10-30T12:00:41.926 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd47d1b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:41.926 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.926 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 1 -- 172.21.15.118:0/4209623809 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4700343f0 con 0x7fd47813afa0 2023-10-30T12:00:41.926 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd47d1b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:41.926 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd47d1b6640 4 set_mon_vals no callback set 2023-10-30T12:00:41.927 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.927 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:41.927 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.902+0000 7fd475d9b640 10 monclient: dump: 2023-10-30T12:00:41.927 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.927 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.928 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.928 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.928 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.928 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.928 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.929 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.929 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.929 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.929 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 10 monclient: shutdown 2023-10-30T12:00:41.929 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.930 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4209623809 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd47813afa0 msgr2=0x7fd47813d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.930 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 --2- 172.21.15.118:0/4209623809 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd47813afa0 0x7fd47813d390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd470009e50 tx=0x7fd47002be40 comp rx=0 tx=0).stop 2023-10-30T12:00:41.930 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4209623809 shutdown_connections 2023-10-30T12:00:41.930 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 --2- 172.21.15.118:0/4209623809 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd47813afa0 0x7fd47813d390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.930 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 --2- 172.21.15.118:0/4209623809 >> v2:172.21.15.50:3300/0 conn(0x7fd478099a40 0x7fd47813aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.930 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4209623809 >> 172.21.15.118:0/4209623809 conn(0x7fd4780a82f0 msgr2=0x7fd4780a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.931 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4209623809 shutdown_connections 2023-10-30T12:00:41.931 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4209623809 wait complete. 2023-10-30T12:00:41.931 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 10 monclient: build_initial_monmap 2023-10-30T12:00:41.931 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.903+0000 7fd47d1b6640 10 monclient: monmap: 2023-10-30T12:00:41.931 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 0 2023-10-30T12:00:41.932 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:41.932 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 0.000000 2023-10-30T12:00:41.932 INFO:teuthology.orchestra.run.smithi118.stderr:created 0.000000 2023-10-30T12:00:41.932 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:41.932 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.932 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:41.933 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:41.933 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:41.933 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.933 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 librados: starting msgr at 2023-10-30T12:00:41.933 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 librados: starting objecter 2023-10-30T12:00:41.933 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 Processor -- start 2023-10-30T12:00:41.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 -- start start 2023-10-30T12:00:41.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 librados: setting wanted keys 2023-10-30T12:00:41.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 librados: calling monclient init 2023-10-30T12:00:41.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient: init 2023-10-30T12:00:41.934 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:41.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:00:41.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd478099a40 0x7fd478139560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): picked mon.noname-c con 0x7fd478099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:41.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4781327f0 0x7fd478130530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.935 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): picked mon.noname-b con 0x7fd4781327f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:41.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd47813afa0 0x7fd478130a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:41.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): picked mon.noname-a con 0x7fd47813afa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:41.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd478130fb0 con 0x7fd47813afa0 2023-10-30T12:00:41.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.936 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780a3fe0 con 0x7fd4781327f0 2023-10-30T12:00:41.937 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:41.937 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780a17f0 con 0x7fd478099a40 2023-10-30T12:00:41.937 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:41.937 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47d1b6640 10 monclient(hunting): authenticate will time out at 873.242996s 2023-10-30T12:00:41.937 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47659c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4781327f0 0x7fd478130530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.938 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47659c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4781327f0 0x7fd478130530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40952/0 (socket says 172.21.15.118:40952) 2023-10-30T12:00:41.938 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47659c640 1 -- 172.21.15.118:0/4264249866 learned_addr learned my addr 172.21.15.118:0/4264249866 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:00:41.938 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47659c640 10 monclient(hunting): get_auth_request con 0x7fd4781327f0 auth_method 0 2023-10-30T12:00:41.938 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:41.938 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:41.939 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.904+0000 7fd47659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:41.939 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:41.939 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:41.939 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:41.939 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47759e640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd47813afa0 0x7fd478130a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.940 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd476d9d640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd478099a40 0x7fd478139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:41.940 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2023-10-30T12:00:41.940 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 1 -- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd478099a40 msgr2=0x7fd478139560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.940 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd478099a40 0x7fd478139560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.940 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 1 -- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd47813afa0 msgr2=0x7fd478130a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd47813afa0 0x7fd478130a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 10 monclient: _finish_hunting 0 2023-10-30T12:00:41.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 1 monclient: found mon.noname-b 2023-10-30T12:00:41.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:41.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.942 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 1 -- 172.21.15.118:0/4264249866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c009400 con 0x7fd4781327f0 2023-10-30T12:00:41.942 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47759e640 10 monclient: get_auth_request con 0x7fd47813afa0 auth_method 0 2023-10-30T12:00:41.942 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47759e640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd47813afa0 0x7fd478130a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:41.942 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47659c640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4781327f0 0x7fd478130530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00e960 tx=0x7fd46c00ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:41.942 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 client.?.objecter ms_handle_connect 0x7fd4781327f0 2023-10-30T12:00:41.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:41.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 1 -- 172.21.15.118:0/4264249866 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00ca00 con 0x7fd4781327f0 2023-10-30T12:00:41.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:41.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: dump: 2023-10-30T12:00:41.943 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.944 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.944 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.944 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.944 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.944 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.945 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.945 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.945 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.945 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.945 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:41.945 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:41.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:41.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 1 -- 172.21.15.118:0/4264249866 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c00cb60 con 0x7fd4781327f0 2023-10-30T12:00:41.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 5 monclient: authenticate success, global_id 4144 2023-10-30T12:00:41.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 10 monclient: _renew_subs 2023-10-30T12:00:41.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd470009b30 con 0x7fd4781327f0 2023-10-30T12:00:41.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:41.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 1 -- 172.21.15.118:0/4264249866 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c013470 con 0x7fd4781327f0 2023-10-30T12:00:41.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:41.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:41.948 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 monclient: dump: 2023-10-30T12:00:41.948 INFO:teuthology.orchestra.run.smithi118.stderr:epoch 1 2023-10-30T12:00:41.948 INFO:teuthology.orchestra.run.smithi118.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:41.948 INFO:teuthology.orchestra.run.smithi118.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.948 INFO:teuthology.orchestra.run.smithi118.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:41.948 INFO:teuthology.orchestra.run.smithi118.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:41.949 INFO:teuthology.orchestra.run.smithi118.stderr:election_strategy: 1 2023-10-30T12:00:41.949 INFO:teuthology.orchestra.run.smithi118.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:41.949 INFO:teuthology.orchestra.run.smithi118.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:41.949 INFO:teuthology.orchestra.run.smithi118.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:41.949 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:00:41.950 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:41.950 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 10 monclient: _renew_subs 2023-10-30T12:00:41.950 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.950 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4781373c0 con 0x7fd4781327f0 2023-10-30T12:00:41.950 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47d1b6640 1 librados: init done 2023-10-30T12:00:41.951 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd47559a640 4 set_mon_vals no callback set 2023-10-30T12:00:41.951 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 1 -- 172.21.15.118:0/4264249866 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd46c005c90 con 0x7fd4781327f0 2023-10-30T12:00:41.951 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 1 -- 172.21.15.118:0/4264249866 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fd46c013b50 con 0x7fd4781327f0 2023-10-30T12:00:41.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 10 client.4144.objecter ms_dispatch 0x7fd478000b90 osd_map(7..7 src has 1..7) v4 2023-10-30T12:00:41.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2023-10-30T12:00:41.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2023-10-30T12:00:41.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.905+0000 7fd45f7fe640 20 client.4144.objecter dump_active .. 0 homeless 2023-10-30T12:00:41.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.906+0000 7fd47d1b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.906+0000 7fd47d1b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.906+0000 7fd47d1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.906+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47804e1d0 con 0x7fd4781327f0 2023-10-30T12:00:41.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.908+0000 7fd45f7fe640 1 -- 172.21.15.118:0/4264249866 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd46c021060 con 0x7fd4781327f0 2023-10-30T12:00:41.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.908+0000 7fd45f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:41.954 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.908+0000 7fd45f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:41.954 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:41.942+0000 7f3dd2056040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:00:41.981 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.978+0000 7fd47d1b6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "79b07bf0-2944-4413-b6e1-23ecbc533add", "id": 5}] 2023-10-30T12:00:41.981 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.978+0000 7fd47d1b6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "79b07bf0-2944-4413-b6e1-23ecbc533add", "id": 5}] 2023-10-30T12:00:41.981 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.978+0000 7fd47d1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:41.981 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.978+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "79b07bf0-2944-4413-b6e1-23ecbc533add", "id": 5} v 0) v1 -- 0x7fd478074510 con 0x7fd4781327f0 2023-10-30T12:00:41.984 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.981+0000 7fd45f7fe640 1 -- 172.21.15.118:0/4264249866 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79b07bf0-2944-4413-b6e1-23ecbc533add", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd46c02a0b0 con 0x7fd4781327f0 2023-10-30T12:00:41.984 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.981+0000 7fd45f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "79b07bf0-2944-4413-b6e1-23ecbc533add", "id": 5}] 2023-10-30T12:00:41.984 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.981+0000 7fd45f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:41.985 INFO:teuthology.orchestra.run.smithi118.stdout:5 2023-10-30T12:00:41.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 10 librados: watch_flush enter 2023-10-30T12:00:41.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 10 librados: watch_flush exit 2023-10-30T12:00:41.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 20 client.4144.objecter shutdown clearing up homeless session... 2023-10-30T12:00:41.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 10 client.4144.objecter successfully canceled tick 2023-10-30T12:00:41.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 10 monclient: shutdown 2023-10-30T12:00:41.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:41.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4781327f0 msgr2=0x7fd478130530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:41.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4781327f0 0x7fd478130530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00e960 tx=0x7fd46c00ee20 comp rx=0 tx=0).stop 2023-10-30T12:00:41.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 shutdown_connections 2023-10-30T12:00:41.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd47813afa0 0x7fd478130a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd4781327f0 0x7fd478130530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 --2- 172.21.15.118:0/4264249866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd478099a40 0x7fd478139560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:41.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 >> 172.21.15.118:0/4264249866 conn(0x7fd4780a82f0 msgr2=0x7fd478131390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:41.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 shutdown_connections 2023-10-30T12:00:41.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 -- 172.21.15.118:0/4264249866 wait complete. 2023-10-30T12:00:41.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-30T12:00:41.982+0000 7fd47d1b6640 1 librados: shutdown 2023-10-30T12:00:41.995 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T12:00:41.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-30T12:00:41.998 INFO:tasks.ceph.osd.0:Started 2023-10-30T12:00:41.998 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T12:00:41.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-30T12:00:42.001 INFO:tasks.ceph.osd.1:Started 2023-10-30T12:00:42.001 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T12:00:42.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-30T12:00:42.004 INFO:tasks.ceph.osd.2:Started 2023-10-30T12:00:42.004 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T12:00:42.004 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-30T12:00:42.037 INFO:tasks.ceph.osd.3:Started 2023-10-30T12:00:42.038 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T12:00:42.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-30T12:00:42.041 INFO:tasks.ceph.osd.4:Started 2023-10-30T12:00:42.041 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T12:00:42.042 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-30T12:00:42.043 INFO:tasks.ceph.osd.5:Started 2023-10-30T12:00:42.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:00:42.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.158+0000 7f538fd05640 10 monclient: get_monmap_and_config 2023-10-30T12:00:42.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.158+0000 7f538fd05640 10 monclient: build_initial_monmap 2023-10-30T12:00:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.158+0000 7f538fd05640 10 monclient: monmap: 2023-10-30T12:00:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:42.159718+0000 2023-10-30T12:00:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:42.159718+0000 2023-10-30T12:00:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:42.163 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:42.163 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:42.163 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:42.163 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:42.163 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:42.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 Processor -- start 2023-10-30T12:00:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 -- start start 2023-10-30T12:00:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient: init 2023-10-30T12:00:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:00:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5388131b20 0x7f5388132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient(hunting): picked mon.noname-a con 0x7f5388131b20 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient(hunting): picked mon.noname-b con 0x7f5388071bc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5388097410 0x7f53880978a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient(hunting): picked mon.noname-c con 0x7f5388097410 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538809a9b0 con 0x7f5388131b20 2023-10-30T12:00:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5388131f00 con 0x7f5388097410 2023-10-30T12:00:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.159+0000 7f538fd05640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538809d920 con 0x7f5388071bc0 2023-10-30T12:00:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538fd05640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538fd05640 10 monclient(hunting): authenticate will time out at 887.343017s 2023-10-30T12:00:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5388131b20 0x7f5388132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47146/0 (socket says 172.21.15.50:47146) 2023-10-30T12:00:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 learned_addr learned my addr 172.21.15.50:0/2101940839 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): get_auth_request con 0x7f5388071bc0 auth_method 0 2023-10-30T12:00:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 10 monclient(hunting): get_auth_request con 0x7f5388131b20 auth_method 0 2023-10-30T12:00:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 1 --2- 172.21.15.50:0/2101940839 >> v2:172.21.15.118:3300/0 conn(0x7f5388097410 0x7f53880978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 10 monclient(hunting): get_auth_request con 0x7f5388097410 auth_method 0 2023-10-30T12:00:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2023-10-30T12:00:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 >> v2:172.21.15.118:3300/0 conn(0x7f5388097410 msgr2=0x7f53880978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> v2:172.21.15.118:3300/0 conn(0x7f5388097410 0x7f53880978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 >> v2:172.21.15.50:3300/0 conn(0x7f5388131b20 msgr2=0x7f5388132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> v2:172.21.15.50:3300/0 conn(0x7f5388131b20 0x7f5388132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient: _finish_hunting 0 2023-10-30T12:00:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 monclient: found mon.noname-b 2023-10-30T12:00:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538809a0b0 con 0x7f5388071bc0 2023-10-30T12:00:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.160+0000 7f538daa1640 1 --2- 172.21.15.50:0/2101940839 >> v2:172.21.15.50:3300/0 conn(0x7f5388131b20 0x7f5388132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.161+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 msgr2=0x7f5388071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2023-10-30T12:00:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.161+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f5388072590 tx=0x7f537802bba0 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2023-10-30T12:00:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.161+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:00:42.331 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.328+0000 7f3dd2056040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-30T12:00:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538d2a0640 10 monclient: get_auth_request con 0x7f5388071bc0 auth_method 0 2023-10-30T12:00:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538d2a0640 0 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-30T12:00:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: ms_handle_reset current mon [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 msgr2=0x7f5388071fc0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2023-10-30T12:00:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5388071bc0 0x7f5388071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: _add_conns ranks=[1,0] 2023-10-30T12:00:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5374003500 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5374007490 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537400b440 con 0x7f5374007490 2023-10-30T12:00:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537400c640 con 0x7f5374003500 2023-10-30T12:00:42.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:42.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:42.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970914550 0 0) 0x7f537400b440 con 0x7f5374007490 2023-10-30T12:00:42.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): _negotiate 2023-10-30T12:00:42.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> v1:172.21.15.50:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f537400b5d0 con 0x7f5374007490 2023-10-30T12:00:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672085181 0 0) 0x7f537400c640 con 0x7f5374003500 2023-10-30T12:00:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): _negotiate 2023-10-30T12:00:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> v1:172.21.15.118:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f537400b440 con 0x7f5374003500 2023-10-30T12:00:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2123592243 0 0) 0x7f537400b5d0 con 0x7f5374007490 2023-10-30T12:00:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 >> v1:172.21.15.118:6789/0 conn(0x7f5374003500 legacy=0x7f5374005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: _finish_hunting 0 2023-10-30T12:00:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:00:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:00:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537400d900 con 0x7f5374007490 2023-10-30T12:00:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: _finish_auth 0 2023-10-30T12:00:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: _check_auth_tickets 2023-10-30T12:00:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f537802c8d0 con 0x7f5374007490 2023-10-30T12:00:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538fd05640 5 monclient: authenticate success, global_id 4127 2023-10-30T12:00:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538fd05640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:00:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.362+0000 7f538ca9f640 10 monclient: dump: 2023-10-30T12:00:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:42.382 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:00:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 >> v1:172.21.15.50:6789/0 conn(0x7f5374007490 legacy=0x7f5374009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 1 -- 172.21.15.50:0/2101940839 reap_dead start 2023-10-30T12:00:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f537400bcc0 0x7f537400c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): picked mon.b con 0x7f537400bcc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5374005a50 0x7f537400c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): picked mon.c con 0x7f5374005a50 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:42.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400cd10 0x7f537400d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): picked mon.a con 0x7f537400cd10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:42.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5374006350 con 0x7f537400cd10 2023-10-30T12:00:42.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53740064c0 con 0x7f537400bcc0 2023-10-30T12:00:42.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5374006680 con 0x7f5374005a50 2023-10-30T12:00:42.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:42.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:00:42.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538fd05640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:42.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5374005a50 0x7f537400c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400cd10 0x7f537400d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f537400bcc0 0x7f537400c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): get_auth_request con 0x7f5374005a50 auth_method 0 2023-10-30T12:00:42.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:42.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 10 monclient(hunting): get_auth_request con 0x7f537400bcc0 auth_method 0 2023-10-30T12:00:42.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 10 monclient(hunting): get_auth_request con 0x7f537400cd10 auth_method 0 2023-10-30T12:00:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient(hunting): handle_auth_done global_id 4127 payload 293 2023-10-30T12:00:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f537400bcc0 msgr2=0x7f537400c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:42.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f537400bcc0 0x7f537400c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400cd10 msgr2=0x7f537400d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400cd10 0x7f537400d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient: _finish_hunting 0 2023-10-30T12:00:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 monclient: found mon.c 2023-10-30T12:00:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 -- 172.21.15.50:0/2101940839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537400b5d0 con 0x7f5374005a50 2023-10-30T12:00:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient: _finish_auth 0 2023-10-30T12:00:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 10 monclient: _check_auth_tickets 2023-10-30T12:00:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:42.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538daa1640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f537400bcc0 0x7f537400c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:42.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538d2a0640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5374005a50 0x7f537400c7d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5388072590 tx=0x7f537400b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6ad2df8ac9a285 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:42.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53780326e0 con 0x7f5374005a50 2023-10-30T12:00:42.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: dump: 2023-10-30T12:00:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.396 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.396 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:42.396 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:42.396 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:42.397 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:42.397 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:42.397 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:42.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378032840 con 0x7f5374005a50 2023-10-30T12:00:42.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538fd05640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:42.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:42.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538fd05640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:42.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538fd05640 4 set_mon_vals no callback set 2023-10-30T12:00:42.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 1 -- 172.21.15.50:0/2101940839 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378032bb0 con 0x7f5374005a50 2023-10-30T12:00:42.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:42.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:42.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.363+0000 7f538ca9f640 10 monclient: dump: 2023-10-30T12:00:42.399 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:42.399 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:42.400 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.400 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.400 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:42.400 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:42.401 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:42.401 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:42.401 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:42.401 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:42.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 10 monclient: shutdown 2023-10-30T12:00:42.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:42.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 -- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5374005a50 msgr2=0x7f537400c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:42.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5374005a50 0x7f537400c7d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5388072590 tx=0x7f537400b7d0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 -- 172.21.15.50:0/2101940839 shutdown_connections 2023-10-30T12:00:42.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400cd10 0x7f537400d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5374005a50 0x7f537400c7d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 --2- 172.21.15.50:0/2101940839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f537400bcc0 0x7f537400c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 -- 172.21.15.50:0/2101940839 >> 172.21.15.50:0/2101940839 conn(0x7f53880a05c0 msgr2=0x7f538809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:42.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 -- 172.21.15.50:0/2101940839 shutdown_connections 2023-10-30T12:00:42.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.364+0000 7f538fd05640 1 -- 172.21.15.50:0/2101940839 wait complete. 2023-10-30T12:00:42.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 10 monclient: build_initial_monmap 2023-10-30T12:00:42.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 10 monclient: monmap: 2023-10-30T12:00:42.404 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:42.404 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:42.405 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:42.405 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:42.405 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:42.405 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:42.405 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:42.406 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:42.406 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:42.406 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:42.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 1 librados: starting msgr at 2023-10-30T12:00:42.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 1 librados: starting objecter 2023-10-30T12:00:42.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 1 Processor -- start 2023-10-30T12:00:42.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 1 -- start start 2023-10-30T12:00:42.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 1 librados: setting wanted keys 2023-10-30T12:00:42.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 1 librados: calling monclient init 2023-10-30T12:00:42.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.365+0000 7f538fd05640 10 monclient: init 2023-10-30T12:00:42.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:42.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:42.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5374005a50 0x7f5388129680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): picked mon.noname-b con 0x7f5374005a50 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400bcc0 0x7f5388129bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): picked mon.noname-a con 0x7f537400bcc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f537400cd10 0x7f538812a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): picked mon.noname-c con 0x7f537400cd10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538812b600 con 0x7f537400bcc0 2023-10-30T12:00:42.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880964d0 con 0x7f5374005a50 2023-10-30T12:00:42.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:42.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880993c0 con 0x7f537400cd10 2023-10-30T12:00:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538fd05640 10 monclient(hunting): authenticate will time out at 887.549279s 2023-10-30T12:00:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400bcc0 0x7f5388129bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f537400cd10 0x7f538812a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400bcc0 0x7f5388129bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38602/0 (socket says 172.21.15.50:38602) 2023-10-30T12:00:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 1 -- 172.21.15.50:0/663613153 learned_addr learned my addr 172.21.15.50:0/663613153 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): get_auth_request con 0x7f537400cd10 auth_method 0 2023-10-30T12:00:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 10 monclient(hunting): get_auth_request con 0x7f537400bcc0 auth_method 0 2023-10-30T12:00:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5374005a50 0x7f5388129680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 10 monclient(hunting): get_auth_request con 0x7f5374005a50 auth_method 0 2023-10-30T12:00:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:42.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:42.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538daa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:42.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient(hunting): handle_auth_done global_id 4130 payload 824 2023-10-30T12:00:42.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 -- 172.21.15.50:0/663613153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5374005a50 msgr2=0x7f5388129680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:42.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5374005a50 0x7f5388129680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 -- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400bcc0 msgr2=0x7f5388129bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:42.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400bcc0 0x7f5388129bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient: _finish_hunting 0 2023-10-30T12:00:42.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 monclient: found mon.noname-c 2023-10-30T12:00:42.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:42.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 -- 172.21.15.50:0/663613153 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538812cd20 con 0x7f537400cd10 2023-10-30T12:00:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538d2a0640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400bcc0 0x7f5388129bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f538e2a2640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f537400cd10 0x7f538812a100 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f538806bb10 tx=0x7f5380011f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f537e7fc640 10 client.?.objecter ms_handle_connect 0x7f537400cd10 2023-10-30T12:00:42.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.366+0000 7f537e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:42.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 1 -- 172.21.15.50:0/663613153 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380010040 con 0x7f537400cd10 2023-10-30T12:00:42.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:42.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:42.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: dump: 2023-10-30T12:00:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:42.422 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 1 -- 172.21.15.50:0/663613153 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380009e20 con 0x7f537400cd10 2023-10-30T12:00:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 5 monclient: authenticate success, global_id 4130 2023-10-30T12:00:42.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 10 monclient: _renew_subs 2023-10-30T12:00:42.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:42.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538812c480 con 0x7f537400cd10 2023-10-30T12:00:42.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:42.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 1 -- 172.21.15.50:0/663613153 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380003bd0 con 0x7f537400cd10 2023-10-30T12:00:42.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:42.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:42.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 monclient: dump: 2023-10-30T12:00:42.425 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:42.427 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:42.427 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:42.427 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:42.427 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:42.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 10 monclient: _renew_subs 2023-10-30T12:00:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537ffff640 4 set_mon_vals no callback set 2023-10-30T12:00:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5388131700 con 0x7f537400cd10 2023-10-30T12:00:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f538fd05640 1 librados: init done 2023-10-30T12:00:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 1 -- 172.21.15.50:0/663613153 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5380004330 con 0x7f537400cd10 2023-10-30T12:00:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 1 -- 172.21.15.50:0/663613153 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f5380016bd0 con 0x7f537400cd10 2023-10-30T12:00:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 10 client.4130.objecter ms_dispatch 0x7f5388000b90 osd_map(8..8 src has 1..8) v4 2023-10-30T12:00:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 3 client.4130.objecter handle_osd_map got epochs [8,8] > 0 2023-10-30T12:00:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 3 client.4130.objecter handle_osd_map decoding full epoch 8 2023-10-30T12:00:42.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.367+0000 7f537e7fc640 20 client.4130.objecter dump_active .. 0 homeless 2023-10-30T12:00:42.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.368+0000 7f538fd05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:42.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.368+0000 7f538fd05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:42.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.368+0000 7f538fd05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:42.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.368+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538804e160 con 0x7f537400cd10 2023-10-30T12:00:42.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.370+0000 7f537e7fc640 1 -- 172.21.15.50:0/663613153 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f538001a020 con 0x7f537400cd10 2023-10-30T12:00:42.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.370+0000 7f537e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:42.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.370+0000 7f537e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:42.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.430+0000 7f538fd05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:42.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.430+0000 7f538fd05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:42.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.430+0000 7f538fd05640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:42.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.430+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f538806f260 con 0x7f537400cd10 2023-10-30T12:00:42.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f537e7fc640 1 -- 172.21.15.50:0/663613153 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7f5380003d30 con 0x7f537400cd10 2023-10-30T12:00:42.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f537e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:42.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f537e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:42.434 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:42.435 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","created":"2023-10-30T12:00:40.280079+0000","modified":"2023-10-30T12:00:41.981126+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:00:41.981126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dba8cbf1-bc2e-4420-957f-11ec607e814e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c13ebbef-aa1a-4da7-b5f6-26b02e43955c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"23116074-425b-4664-a1be-0e40bba62fcb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"515e513c-cb67-4495-ba25-68cb35215c58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9df8af-676c-49c6-9888-0e330fbbc4ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"79b07bf0-2944-4413-b6e1-23ecbc533add","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:00:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 10 librados: watch_flush enter 2023-10-30T12:00:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 10 librados: watch_flush exit 2023-10-30T12:00:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 20 client.4130.objecter shutdown clearing up homeless session... 2023-10-30T12:00:42.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 10 client.4130.objecter successfully canceled tick 2023-10-30T12:00:42.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 10 monclient: shutdown 2023-10-30T12:00:42.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:42.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f537400cd10 msgr2=0x7f538812a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:42.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.431+0000 7f538fd05640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f537400cd10 0x7f538812a100 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f538806bb10 tx=0x7f5380011f90 comp rx=0 tx=0).stop 2023-10-30T12:00:42.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 shutdown_connections 2023-10-30T12:00:42.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f537400cd10 0x7f538812a100 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f537400bcc0 0x7f5388129bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 --2- 172.21.15.50:0/663613153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5374005a50 0x7f5388129680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:42.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 >> 172.21.15.50:0/663613153 conn(0x7f53880a05c0 msgr2=0x7f53880a2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:42.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 shutdown_connections 2023-10-30T12:00:42.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 -- 172.21.15.50:0/663613153 wait complete. 2023-10-30T12:00:42.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:42.432+0000 7f538fd05640 1 librados: shutdown 2023-10-30T12:00:42.443 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T12:00:42.444 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T12:00:42.467 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.465+0000 7f3dd2056040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:00:42.581 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.579+0000 7f3dd2056040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:00:42.611 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-10-30T12:00:42.610+0000 7f11b9665500 -1 Falling back to public interface 2023-10-30T12:00:42.619 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-10-30T12:00:42.617+0000 7f1aa2592500 -1 Falling back to public interface 2023-10-30T12:00:42.623 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-10-30T12:00:42.621+0000 7f916a8da500 -1 Falling back to public interface 2023-10-30T12:00:42.626 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.623+0000 7f3dd2056040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:00:42.638 INFO:tasks.ceph.osd.5.smithi118.stderr:2023-10-30T12:00:42.635+0000 7f1f6cc61500 -1 Falling back to public interface 2023-10-30T12:00:42.641 INFO:tasks.ceph.osd.3.smithi118.stderr:2023-10-30T12:00:42.638+0000 7fcdbd600500 -1 Falling back to public interface 2023-10-30T12:00:42.641 INFO:tasks.ceph.osd.4.smithi118.stderr:2023-10-30T12:00:42.638+0000 7fc107a2b500 -1 Falling back to public interface 2023-10-30T12:00:42.642 INFO:tasks.ceph.osd.3.smithi118.stderr:2023-10-30T12:00:42.639+0000 7fcdbc5a7640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6809/0: (98) Address already in use 2023-10-30T12:00:42.725 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.722+0000 7f3dd2056040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:00:42.819 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.816+0000 7f3dd2056040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:00:42.959 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.957+0000 7f3dd2056040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:00:43.002 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:42.999+0000 7f3dd2056040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:00:43.048 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:00:43.140 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:43.137+0000 7f3dd2056040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:00:43.150 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.148+0000 7f112fe4f640 10 monclient: get_monmap_and_config 2023-10-30T12:00:43.151 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.148+0000 7f112fe4f640 10 monclient: build_initial_monmap 2023-10-30T12:00:43.151 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.148+0000 7f112fe4f640 10 monclient: monmap: 2023-10-30T12:00:43.151 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2023-10-30T12:00:43.151 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:43.151 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:43.150023+0000 2023-10-30T12:00:43.152 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:43.150023+0000 2023-10-30T12:00:43.152 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:43.152 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:43.152 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:43.152 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:43.153 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:43.153 INFO:teuthology.misc.health.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:43.153 INFO:teuthology.misc.health.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:43.153 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:43.160 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.149+0000 7f112fe4f640 1 Processor -- start 2023-10-30T12:00:43.161 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.149+0000 7f112fe4f640 1 -- start start 2023-10-30T12:00:43.161 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.149+0000 7f112fe4f640 10 monclient: init 2023-10-30T12:00:43.161 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:43.161 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-10-30T12:00:43.162 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11280a4bf0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:43.162 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1128099a40 0x7f112813aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:43.162 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient(hunting): picked mon.noname-c con 0x7f1128099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:43.162 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11280a6810 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:43.162 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11280a4fd0 con 0x7f11280a4bf0 2023-10-30T12:00:43.163 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11280a76e0 con 0x7f11280a6810 2023-10-30T12:00:43.163 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:43.163 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11280a7300 con 0x7f1128099a40 2023-10-30T12:00:43.163 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:43.163 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112fe4f640 10 monclient(hunting): authenticate will time out at 888.333447s 2023-10-30T12:00:43.164 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112dbeb640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f11280a4bf0 0x7f11280a6100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34442/0 (socket says 172.21.15.50:34442) 2023-10-30T12:00:43.164 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112dbeb640 1 -- 172.21.15.50:0/2796058215 learned_addr learned my addr 172.21.15.50:0/2796058215 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:43.164 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 1 --2- 172.21.15.50:0/2796058215 >> v2:172.21.15.118:3300/0 conn(0x7f1128099a40 0x7f112813aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:43.164 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 10 monclient(hunting): get_auth_request con 0x7f1128099a40 auth_method 0 2023-10-30T12:00:43.164 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:43.164 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:43.165 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:43.165 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319478658 0 0) 0x7f11280a4fd0 con 0x7f11280a4bf0 2023-10-30T12:00:43.165 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112cbe9640 10 monclient(hunting): _negotiate 2023-10-30T12:00:43.165 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112cbe9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:43.165 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112cbe9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:43.166 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112cbe9640 10 monclient(hunting): my global_id is 4164 2023-10-30T12:00:43.166 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1114003520 con 0x7f11280a4bf0 2023-10-30T12:00:43.166 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:43.166 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:43.166 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.150+0000 7f112d3ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:43.167 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891747900 0 0) 0x7f11280a76e0 con 0x7f11280a6810 2023-10-30T12:00:43.167 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 10 monclient(hunting): _negotiate 2023-10-30T12:00:43.167 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:43.167 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:43.167 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 10 monclient(hunting): my global_id is 4180 2023-10-30T12:00:43.167 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11280a4fd0 con 0x7f11280a6810 2023-10-30T12:00:43.168 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 10 monclient(hunting): handle_auth_done global_id 4177 payload 274 2023-10-30T12:00:43.168 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 1 -- 172.21.15.50:0/2796058215 >> v1:172.21.15.118:6789/0 conn(0x7f11280a6810 legacy=0x7f112813d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:43.168 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 1 -- 172.21.15.50:0/2796058215 >> v1:172.21.15.50:6789/0 conn(0x7f11280a4bf0 legacy=0x7f11280a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:43.168 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801478885 0 0) 0x7f1114003520 con 0x7f11280a4bf0 2023-10-30T12:00:43.168 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 10 monclient: _finish_hunting 0 2023-10-30T12:00:43.169 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 1 monclient: found mon.noname-c 2023-10-30T12:00:43.169 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:43.169 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:43.169 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 1 -- 172.21.15.50:0/2796058215 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11280a5ca0 con 0x7f1128099a40 2023-10-30T12:00:43.169 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112cbe9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:43.169 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.151+0000 7f112d3ea640 1 --2- 172.21.15.50:0/2796058215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1128099a40 0x7f112813aab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f11180097a0 tx=0x7f111802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ff4408d19a4285e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:43.170 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124001690 con 0x7f1128099a40 2023-10-30T12:00:43.170 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:43.170 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:43.170 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: dump: 2023-10-30T12:00:43.170 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:43.171 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:43.171 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.171 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.171 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:43.171 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:43.172 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:43.172 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:43.172 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:43.172 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:43.172 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: _finish_auth 0 2023-10-30T12:00:43.172 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: _check_auth_tickets 2023-10-30T12:00:43.173 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:43.173 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 5 monclient: authenticate success, global_id 4177 2023-10-30T12:00:43.173 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111802e050 con 0x7f1128099a40 2023-10-30T12:00:43.173 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:43.173 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:43.174 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 4 set_mon_vals no callback set 2023-10-30T12:00:43.174 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 1 -- 172.21.15.50:0/2796058215 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118030730 con 0x7f1128099a40 2023-10-30T12:00:43.174 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:43.174 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:43.174 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112cbe9640 10 monclient: dump: 2023-10-30T12:00:43.175 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:43.175 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:43.175 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.175 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.175 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:43.175 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:43.176 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:43.176 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:43.176 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:43.176 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:43.176 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 10 monclient: shutdown 2023-10-30T12:00:43.177 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:43.177 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 1 -- 172.21.15.50:0/2796058215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1128099a40 msgr2=0x7f112813aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:43.177 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 1 --2- 172.21.15.50:0/2796058215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1128099a40 0x7f112813aab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f11180097a0 tx=0x7f111802b710 comp rx=0 tx=0).stop 2023-10-30T12:00:43.177 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 1 -- 172.21.15.50:0/2796058215 shutdown_connections 2023-10-30T12:00:43.177 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 1 --2- 172.21.15.50:0/2796058215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1128099a40 0x7f112813aab0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:43.178 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 1 -- 172.21.15.50:0/2796058215 >> 172.21.15.50:0/2796058215 conn(0x7f11280a80a0 msgr2=0x7f11280a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:43.178 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 1 -- 172.21.15.50:0/2796058215 shutdown_connections 2023-10-30T12:00:43.178 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.152+0000 7f112fe4f640 1 -- 172.21.15.50:0/2796058215 wait complete. 2023-10-30T12:00:43.178 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 10 monclient: build_initial_monmap 2023-10-30T12:00:43.178 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 10 monclient: monmap: 2023-10-30T12:00:43.178 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2023-10-30T12:00:43.179 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:43.179 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:43.179 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2023-10-30T12:00:43.179 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:43.179 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:43.180 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:43.180 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:43.180 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:43.180 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:43.180 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 1 librados: starting msgr at 2023-10-30T12:00:43.180 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 1 librados: starting objecter 2023-10-30T12:00:43.181 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 1 Processor -- start 2023-10-30T12:00:43.181 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 1 -- start start 2023-10-30T12:00:43.181 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 1 librados: setting wanted keys 2023-10-30T12:00:43.181 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 1 librados: calling monclient init 2023-10-30T12:00:43.181 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.153+0000 7f112fe4f640 10 monclient: init 2023-10-30T12:00:43.182 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:43.182 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:00:43.182 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11280a4bf0 0x7f112809d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:43.182 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): picked mon.noname-c con 0x7f11280a4bf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:43.182 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11280a6810 0x7f112809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:43.183 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): picked mon.noname-a con 0x7f11280a6810 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:43.183 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f112809fc60 0x7f112809e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:43.183 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): picked mon.noname-b con 0x7f112809fc60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:43.183 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:43.183 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11281336a0 con 0x7f11280a6810 2023-10-30T12:00:43.183 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:43.184 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11280a18d0 con 0x7f112809fc60 2023-10-30T12:00:43.184 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:43.184 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112812c3f0 con 0x7f11280a4bf0 2023-10-30T12:00:43.184 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:43.184 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112fe4f640 10 monclient(hunting): authenticate will time out at 888.337274s 2023-10-30T12:00:43.185 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11280a6810 0x7f112809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:43.185 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11280a4bf0 0x7f112809d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:43.185 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11280a6810 0x7f112809dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38618/0 (socket says 172.21.15.50:38618) 2023-10-30T12:00:43.185 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 -- 172.21.15.50:0/1682803869 learned_addr learned my addr 172.21.15.50:0/1682803869 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:43.185 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): get_auth_request con 0x7f11280a6810 auth_method 0 2023-10-30T12:00:43.185 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:43.186 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:43.186 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:43.186 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 10 monclient(hunting): get_auth_request con 0x7f11280a4bf0 auth_method 0 2023-10-30T12:00:43.186 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:43.186 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:43.187 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:43.187 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f112809fc60 0x7f112809e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:43.187 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 10 monclient(hunting): get_auth_request con 0x7f112809fc60 auth_method 0 2023-10-30T12:00:43.187 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:43.187 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:43.188 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:43.188 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:43.188 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:43.188 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:43.188 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:43.189 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:43.189 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112dbeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:43.189 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:43.189 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:43.189 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:43.189 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient(hunting): handle_auth_done global_id 4167 payload 824 2023-10-30T12:00:43.190 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 -- 172.21.15.50:0/1682803869 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11280a4bf0 msgr2=0x7f112809d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:43.190 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11280a4bf0 0x7f112809d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:43.190 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 -- 172.21.15.50:0/1682803869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f112809fc60 msgr2=0x7f112809e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:43.190 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f112809fc60 0x7f112809e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:43.190 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient: _finish_hunting 0 2023-10-30T12:00:43.191 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 monclient: found mon.noname-a 2023-10-30T12:00:43.191 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:43.191 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:43.191 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 -- 172.21.15.50:0/1682803869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1118009400 con 0x7f11280a6810 2023-10-30T12:00:43.191 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112e3ec640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f112809fc60 0x7f112809e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:43.192 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f112d3ea640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11280a6810 0x7f112809dcb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1118035040 tx=0x7f11180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:43.192 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f111e7fc640 10 client.?.objecter ms_handle_connect 0x7f11280a6810 2023-10-30T12:00:43.192 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.154+0000 7f111e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:43.192 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 1 -- 172.21.15.50:0/1682803869 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118004220 con 0x7f11280a6810 2023-10-30T12:00:43.192 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:43.193 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:43.193 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: dump: 2023-10-30T12:00:43.193 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:43.193 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:43.193 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.194 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.194 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:43.194 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:43.194 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:43.194 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:43.195 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:43.195 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:43.195 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:43.195 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:43.195 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:43.195 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 1 -- 172.21.15.50:0/1682803869 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1118004380 con 0x7f11280a6810 2023-10-30T12:00:43.196 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 5 monclient: authenticate success, global_id 4167 2023-10-30T12:00:43.196 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 10 monclient: _renew_subs 2023-10-30T12:00:43.196 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:43.196 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11281333c0 con 0x7f11280a6810 2023-10-30T12:00:43.196 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:43.197 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 1 -- 172.21.15.50:0/1682803869 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111803c8c0 con 0x7f11280a6810 2023-10-30T12:00:43.197 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:43.197 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:43.197 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:43.197 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 monclient: dump: 2023-10-30T12:00:43.197 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:43.198 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:43.198 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.198 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:43.198 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:43.198 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:43.199 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:43.199 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:43.199 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:43.199 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:43.199 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 10 monclient: _renew_subs 2023-10-30T12:00:43.200 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111ffff640 4 set_mon_vals no callback set 2023-10-30T12:00:43.200 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:43.200 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112809e730 con 0x7f11280a6810 2023-10-30T12:00:43.200 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f112fe4f640 1 librados: init done 2023-10-30T12:00:43.200 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 1 -- 172.21.15.50:0/1682803869 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1118039040 con 0x7f11280a6810 2023-10-30T12:00:43.200 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 1 -- 172.21.15.50:0/1682803869 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f1118003a70 con 0x7f11280a6810 2023-10-30T12:00:43.201 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 10 client.4167.objecter ms_dispatch 0x7f1128000b90 osd_map(8..8 src has 1..8) v4 2023-10-30T12:00:43.201 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 3 client.4167.objecter handle_osd_map got epochs [8,8] > 0 2023-10-30T12:00:43.201 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 3 client.4167.objecter handle_osd_map decoding full epoch 8 2023-10-30T12:00:43.201 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.155+0000 7f111e7fc640 20 client.4167.objecter dump_active .. 0 homeless 2023-10-30T12:00:43.201 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.156+0000 7f112fe4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:43.202 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.156+0000 7f112fe4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:43.202 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.156+0000 7f112fe4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:43.202 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.156+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112804e160 con 0x7f11280a6810 2023-10-30T12:00:43.202 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.158+0000 7f111e7fc640 1 -- 172.21.15.50:0/1682803869 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f111804c060 con 0x7f11280a6810 2023-10-30T12:00:43.202 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.158+0000 7f111e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:43.203 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.158+0000 7f111e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:43.221 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.219+0000 7f112fe4f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:43.221 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.219+0000 7f112fe4f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:43.221 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.219+0000 7f112fe4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:43.222 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.219+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11281338e0 con 0x7f11280a6810 2023-10-30T12:00:43.222 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f111e7fc640 1 -- 172.21.15.50:0/1682803869 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7f1118043070 con 0x7f11280a6810 2023-10-30T12:00:43.222 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f111e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:43.223 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f111e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:43.223 INFO:teuthology.misc.health.smithi050.stdout: 2023-10-30T12:00:43.223 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":8,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","created":"2023-10-30T12:00:40.280079+0000","modified":"2023-10-30T12:00:41.981126+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:00:41.981126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dba8cbf1-bc2e-4420-957f-11ec607e814e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c13ebbef-aa1a-4da7-b5f6-26b02e43955c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"23116074-425b-4664-a1be-0e40bba62fcb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"515e513c-cb67-4495-ba25-68cb35215c58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9df8af-676c-49c6-9888-0e330fbbc4ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"79b07bf0-2944-4413-b6e1-23ecbc533add","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:00:43.224 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 10 librados: watch_flush enter 2023-10-30T12:00:43.225 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 10 librados: watch_flush exit 2023-10-30T12:00:43.225 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 20 client.4167.objecter shutdown clearing up homeless session... 2023-10-30T12:00:43.225 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 10 client.4167.objecter successfully canceled tick 2023-10-30T12:00:43.225 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 10 monclient: shutdown 2023-10-30T12:00:43.225 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:43.226 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11280a6810 msgr2=0x7f112809dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:43.226 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11280a6810 0x7f112809dcb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1118035040 tx=0x7f11180046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:43.226 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 shutdown_connections 2023-10-30T12:00:43.227 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f112809fc60 0x7f112809e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:43.227 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11280a6810 0x7f112809dcb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:43.227 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 --2- 172.21.15.50:0/1682803869 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11280a4bf0 0x7f112809d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:43.227 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 >> 172.21.15.50:0/1682803869 conn(0x7f11280a80a0 msgr2=0x7f112812b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:43.228 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 shutdown_connections 2023-10-30T12:00:43.228 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 -- 172.21.15.50:0/1682803869 wait complete. 2023-10-30T12:00:43.228 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:43.220+0000 7f112fe4f640 1 librados: shutdown 2023-10-30T12:00:43.231 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-10-30T12:00:43.950 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:43.947+0000 7f3dd2056040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:00:44.219 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:44.217+0000 7f3dd2056040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:00:44.270 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:44.268+0000 7f3dd2056040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:00:44.373 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:44.370+0000 7f3dd2056040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:00:45.399 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:45.396+0000 7f3d734bc640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T12:00:45.399 INFO:tasks.ceph.mgr.x.smithi118.stderr:2023-10-30T12:00:45.396+0000 7f3d734bc640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T12:00:46.047 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-10-30T12:00:46.046+0000 7f11b9665500 -1 osd.2 0 log_to_monitors true 2023-10-30T12:00:46.079 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-10-30T12:00:46.078+0000 7f1aa2592500 -1 osd.1 0 log_to_monitors true 2023-10-30T12:00:46.110 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-10-30T12:00:46.110+0000 7f916a8da500 -1 osd.0 0 log_to_monitors true 2023-10-30T12:00:46.171 INFO:tasks.ceph.osd.5.smithi118.stderr:2023-10-30T12:00:46.168+0000 7f1f6cc61500 -1 osd.5 0 log_to_monitors true 2023-10-30T12:00:46.212 INFO:tasks.ceph.osd.3.smithi118.stderr:2023-10-30T12:00:46.210+0000 7fcdbd600500 -1 osd.3 0 log_to_monitors true 2023-10-30T12:00:46.218 INFO:tasks.ceph.osd.4.smithi118.stderr:2023-10-30T12:00:46.216+0000 7fc107a2b500 -1 osd.4 0 log_to_monitors true 2023-10-30T12:00:47.410 INFO:tasks.ceph.osd.5.smithi118.stderr:2023-10-30T12:00:47.406+0000 7f1f683ca640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T12:00:47.410 INFO:tasks.ceph.osd.4.smithi118.stderr:2023-10-30T12:00:47.406+0000 7fc1027fc640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T12:00:47.411 INFO:tasks.ceph.osd.3.smithi118.stderr:2023-10-30T12:00:47.406+0000 7fcdb8d69640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T12:00:47.416 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-10-30T12:00:47.415+0000 7f1a9dcfb640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T12:00:47.417 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-10-30T12:00:47.416+0000 7f916787d640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T12:00:47.417 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-10-30T12:00:47.416+0000 7f11b4dce640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T12:00:47.421 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-10-30T12:00:47.419+0000 7f1a96b2e640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:00:47.422 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-10-30T12:00:47.420+0000 7f915ee76640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:00:47.422 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-10-30T12:00:47.420+0000 7f11ad400640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:00:47.422 INFO:tasks.ceph.osd.3.smithi118.stderr:2023-10-30T12:00:47.420+0000 7fcdb1b9c640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:00:47.423 INFO:tasks.ceph.osd.5.smithi118.stderr:2023-10-30T12:00:47.420+0000 7f1f611fd640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:00:47.423 INFO:tasks.ceph.osd.4.smithi118.stderr:2023-10-30T12:00:47.420+0000 7fc0fb62f640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:00:49.836 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:00:49.938 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.936+0000 7fdf6d01f640 10 monclient: get_monmap_and_config 2023-10-30T12:00:49.938 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.936+0000 7fdf6d01f640 10 monclient: build_initial_monmap 2023-10-30T12:00:49.938 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.936+0000 7fdf6d01f640 10 monclient: monmap: 2023-10-30T12:00:49.939 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2023-10-30T12:00:49.939 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:49.939 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:49.937398+0000 2023-10-30T12:00:49.939 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:49.937398+0000 2023-10-30T12:00:49.939 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:49.940 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:49.940 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:49.940 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:49.940 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:49.940 INFO:teuthology.misc.health.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:49.941 INFO:teuthology.misc.health.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:49.941 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:49.948 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 1 Processor -- start 2023-10-30T12:00:49.948 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 1 -- start start 2023-10-30T12:00:49.948 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient: init 2023-10-30T12:00:49.948 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:49.949 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:00:49.949 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdf680a1230 0x7fdf6809e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:49.949 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient(hunting): picked mon.noname-c con 0x7fdf680a1230 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:49.949 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf680a1610 0x7fdf680a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:49.949 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient(hunting): picked mon.noname-b con 0x7fdf680a1610 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:49.949 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf6813a0d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:49.950 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf68071560 con 0x7fdf6813a0d0 2023-10-30T12:00:49.950 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:49.950 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6806b040 con 0x7fdf680a1230 2023-10-30T12:00:49.950 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:49.950 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6809def0 con 0x7fdf680a1610 2023-10-30T12:00:49.951 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:49.951 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf6d01f640 10 monclient(hunting): authenticate will time out at 895.120633s 2023-10-30T12:00:49.951 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf680a1610 0x7fdf680a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:49.951 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf680a1610 0x7fdf680a1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35470/0 (socket says 172.21.15.50:35470) 2023-10-30T12:00:49.951 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 1 -- 172.21.15.50:0/2247908323 learned_addr learned my addr 172.21.15.50:0/2247908323 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:49.952 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 10 monclient(hunting): get_auth_request con 0x7fdf680a1610 auth_method 0 2023-10-30T12:00:49.952 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:49.952 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:49.952 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:49.952 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf66d9d640 1 --2- 172.21.15.50:0/2247908323 >> v2:172.21.15.118:3300/0 conn(0x7fdf680a1230 0x7fdf6809e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:49.952 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:49.953 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:49.953 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:49.953 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf66d9d640 10 monclient(hunting): get_auth_request con 0x7fdf680a1230 auth_method 0 2023-10-30T12:00:49.953 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf66d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:49.953 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf66d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:49.954 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.937+0000 7fdf66d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:49.954 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf66d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:49.954 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf66d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:49.954 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf66d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:49.954 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2023-10-30T12:00:49.955 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 1 -- 172.21.15.50:0/2247908323 >> v2:172.21.15.118:3300/0 conn(0x7fdf680a1230 msgr2=0x7fdf6809e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:49.955 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 1 --2- 172.21.15.50:0/2247908323 >> v2:172.21.15.118:3300/0 conn(0x7fdf680a1230 0x7fdf6809e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:49.955 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 1 -- 172.21.15.50:0/2247908323 >> v1:172.21.15.118:6789/0 conn(0x7fdf6813a0d0 legacy=0x7fdf6813c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:49.955 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:49.955 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 1 monclient: found mon.noname-b 2023-10-30T12:00:49.956 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:49.956 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:49.956 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 1 -- 172.21.15.50:0/2247908323 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6806b550 con 0x7fdf680a1610 2023-10-30T12:00:49.956 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.938+0000 7fdf5ffff640 1 --2- 172.21.15.50:0/2247908323 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf680a1610 0x7fdf680a1ac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009870 tx=0x7fdf5002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e4939b979c89b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:49.956 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 1 -- 172.21.15.50:0/2247908323 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf500317c0 con 0x7fdf680a1610 2023-10-30T12:00:49.957 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:49.957 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:49.957 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: dump: 2023-10-30T12:00:49.957 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:49.957 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:49.957 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.958 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.958 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:49.958 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:49.958 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:49.958 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:49.959 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:49.959 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:49.959 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: _finish_auth 0 2023-10-30T12:00:49.959 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: _check_auth_tickets 2023-10-30T12:00:49.959 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:49.959 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 5 monclient: authenticate success, global_id 4169 2023-10-30T12:00:49.960 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:49.960 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 1 -- 172.21.15.50:0/2247908323 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50031920 con 0x7fdf680a1610 2023-10-30T12:00:49.960 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:49.960 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:49.960 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 4 set_mon_vals no callback set 2023-10-30T12:00:49.961 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 1 -- 172.21.15.50:0/2247908323 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50031cb0 con 0x7fdf680a1610 2023-10-30T12:00:49.961 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:49.961 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:49.961 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6659c640 10 monclient: dump: 2023-10-30T12:00:49.961 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:49.962 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:49.962 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.962 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.962 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:49.962 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:49.962 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:49.963 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:49.963 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:49.963 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:49.963 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 10 monclient: shutdown 2023-10-30T12:00:49.963 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:49.964 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 -- 172.21.15.50:0/2247908323 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf680a1610 msgr2=0x7fdf680a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:49.964 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/2247908323 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf680a1610 0x7fdf680a1ac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009870 tx=0x7fdf5002b860 comp rx=0 tx=0).stop 2023-10-30T12:00:49.964 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 -- 172.21.15.50:0/2247908323 shutdown_connections 2023-10-30T12:00:49.964 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/2247908323 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf680a1610 0x7fdf680a1ac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:49.964 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/2247908323 >> v2:172.21.15.118:3300/0 conn(0x7fdf680a1230 0x7fdf6809e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:49.965 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 -- 172.21.15.50:0/2247908323 >> 172.21.15.50:0/2247908323 conn(0x7fdf680a2cd0 msgr2=0x7fdf680a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:49.965 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 -- 172.21.15.50:0/2247908323 shutdown_connections 2023-10-30T12:00:49.965 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.939+0000 7fdf6d01f640 1 -- 172.21.15.50:0/2247908323 wait complete. 2023-10-30T12:00:49.965 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 10 monclient: build_initial_monmap 2023-10-30T12:00:49.965 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 10 monclient: monmap: 2023-10-30T12:00:49.965 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2023-10-30T12:00:49.966 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:49.966 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:49.966 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2023-10-30T12:00:49.966 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:49.966 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:49.967 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:49.967 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:49.967 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:49.967 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:49.967 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 1 librados: starting msgr at 2023-10-30T12:00:49.968 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 1 librados: starting objecter 2023-10-30T12:00:49.968 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 1 Processor -- start 2023-10-30T12:00:49.968 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 1 -- start start 2023-10-30T12:00:49.968 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 1 librados: setting wanted keys 2023-10-30T12:00:49.968 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 1 librados: calling monclient init 2023-10-30T12:00:49.969 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.940+0000 7fdf6d01f640 10 monclient: init 2023-10-30T12:00:49.969 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:49.969 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:49.969 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf680a1230 0x7fdf681ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:49.969 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): picked mon.noname-b con 0x7fdf680a1230 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:49.970 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf680a1610 0x7fdf68122660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:49.970 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): picked mon.noname-a con 0x7fdf680a1610 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:49.970 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf6813a0d0 0x7fdf68122ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:49.970 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): picked mon.noname-c con 0x7fdf6813a0d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:49.970 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:49.971 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf68126690 con 0x7fdf680a1610 2023-10-30T12:00:49.971 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:49.971 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf68070710 con 0x7fdf680a1230 2023-10-30T12:00:49.971 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:49.971 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6806b220 con 0x7fdf6813a0d0 2023-10-30T12:00:49.972 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:49.972 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6d01f640 10 monclient(hunting): authenticate will time out at 895.124359s 2023-10-30T12:00:49.972 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf6813a0d0 0x7fdf68122ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:49.972 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf680a1610 0x7fdf68122660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:49.972 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf6813a0d0 0x7fdf68122ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35482/0 (socket says 172.21.15.50:35482) 2023-10-30T12:00:49.972 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 -- 172.21.15.50:0/759464642 learned_addr learned my addr 172.21.15.50:0/759464642 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:49.973 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): get_auth_request con 0x7fdf6813a0d0 auth_method 0 2023-10-30T12:00:49.973 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:49.973 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:49.973 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:49.973 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 10 monclient(hunting): get_auth_request con 0x7fdf680a1610 auth_method 0 2023-10-30T12:00:49.974 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:49.974 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:49.974 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:49.974 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:49.974 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:49.974 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:49.975 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:49.975 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:49.975 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:49.975 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf680a1230 0x7fdf681ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:49.975 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 10 monclient(hunting): get_auth_request con 0x7fdf680a1230 auth_method 0 2023-10-30T12:00:49.976 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:49.976 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:49.976 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:49.976 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:49.976 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:49.977 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf66d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:49.977 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient(hunting): handle_auth_done global_id 4172 payload 824 2023-10-30T12:00:49.977 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 -- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf680a1230 msgr2=0x7fdf681ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:49.977 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf680a1230 0x7fdf681ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:49.977 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 -- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf680a1610 msgr2=0x7fdf68122660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:49.978 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf680a1610 0x7fdf68122660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:49.978 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient: _finish_hunting 0 2023-10-30T12:00:49.978 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 monclient: found mon.noname-c 2023-10-30T12:00:49.978 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:49.978 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:49.979 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf6759e640 1 -- 172.21.15.50:0/759464642 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf500094d0 con 0x7fdf6813a0d0 2023-10-30T12:00:49.979 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.941+0000 7fdf5ffff640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf680a1610 0x7fdf68122660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:49.979 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6759e640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf6813a0d0 0x7fdf68122ba0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000c730 tx=0x7fdf6000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:49.979 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 client.?.objecter ms_handle_connect 0x7fdf6813a0d0 2023-10-30T12:00:49.979 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:49.980 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 1 -- 172.21.15.50:0/759464642 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60003b70 con 0x7fdf6813a0d0 2023-10-30T12:00:49.980 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:49.980 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:49.980 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: dump: 2023-10-30T12:00:49.980 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:49.980 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:49.981 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.981 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.981 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:49.981 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:49.981 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:49.982 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:49.982 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:49.982 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:49.982 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:49.982 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:49.983 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:49.983 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 1 -- 172.21.15.50:0/759464642 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf60003cd0 con 0x7fdf6813a0d0 2023-10-30T12:00:49.983 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 5 monclient: authenticate success, global_id 4172 2023-10-30T12:00:49.983 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 10 monclient: _renew_subs 2023-10-30T12:00:49.983 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:49.983 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf681bf1c0 con 0x7fdf6813a0d0 2023-10-30T12:00:49.984 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:49.984 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 1 -- 172.21.15.50:0/759464642 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6000aa90 con 0x7fdf6813a0d0 2023-10-30T12:00:49.984 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:49.984 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:49.984 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf5f7fe640 10 monclient: dump: 2023-10-30T12:00:49.985 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2023-10-30T12:00:49.985 INFO:teuthology.misc.health.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:49.985 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.985 INFO:teuthology.misc.health.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:49.985 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:49.985 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:49.986 INFO:teuthology.misc.health.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:49.986 INFO:teuthology.misc.health.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:49.986 INFO:teuthology.misc.health.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:49.986 INFO:teuthology.misc.health.smithi050.stderr: 2023-10-30T12:00:49.986 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf65d9b640 4 set_mon_vals no callback set 2023-10-30T12:00:49.987 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 10 client.4172.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:49.987 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 10 monclient: _renew_subs 2023-10-30T12:00:49.987 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:49.987 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf68125df0 con 0x7fdf6813a0d0 2023-10-30T12:00:49.987 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.942+0000 7fdf6d01f640 1 librados: init done 2023-10-30T12:00:49.988 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf5f7fe640 1 -- 172.21.15.50:0/759464642 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 4) v1 ==== 44799+0+0 (secure 0 0 0) 0x7fdf6000abf0 con 0x7fdf6813a0d0 2023-10-30T12:00:49.988 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf5f7fe640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf48039260 0x7fdf4803b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:49.988 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf6d01f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:49.988 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf6d01f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:49.988 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf6d01f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:49.989 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf5f7fe640 1 -- 172.21.15.50:0/759464642 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3982+0+0 (secure 0 0 0) 0x7fdf60049fe0 con 0x7fdf6813a0d0 2023-10-30T12:00:49.989 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf5f7fe640 10 client.4172.objecter ms_dispatch 0x7fdf68000b90 osd_map(12..12 src has 1..12) v4 2023-10-30T12:00:49.989 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf5f7fe640 3 client.4172.objecter handle_osd_map got epochs [12,12] > 0 2023-10-30T12:00:49.989 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf5f7fe640 3 client.4172.objecter handle_osd_map decoding full epoch 12 2023-10-30T12:00:49.989 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6804e160 con 0x7fdf6813a0d0 2023-10-30T12:00:49.989 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf5f7fe640 20 client.4172.objecter dump_active .. 0 homeless 2023-10-30T12:00:49.990 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf66d9d640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf48039260 0x7fdf4803b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:49.990 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.943+0000 7fdf66d9d640 10 monclient: get_auth_request con 0x7fdf48039260 auth_method 0 2023-10-30T12:00:49.990 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.944+0000 7fdf5f7fe640 10 client.4172.objecter ms_handle_connect 0x7fdf48039260 2023-10-30T12:00:49.991 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.944+0000 7fdf66d9d640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf48039260 0x7fdf4803b720 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009aa0 tx=0x7fdf58009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:49.991 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.948+0000 7fdf5f7fe640 1 -- 172.21.15.50:0/759464642 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf60003e40 con 0x7fdf6813a0d0 2023-10-30T12:00:49.991 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.948+0000 7fdf5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:49.991 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:49.948+0000 7fdf5f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:50.126 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.125+0000 7fdf6d01f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:50.127 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.125+0000 7fdf6d01f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:50.127 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.125+0000 7fdf6d01f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:50.127 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.125+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf68126310 con 0x7fdf6813a0d0 2023-10-30T12:00:50.128 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.125+0000 7fdf5f7fe640 1 -- 172.21.15.50:0/759464642 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+10419 (secure 0 0 0) 0x7fdf6001a360 con 0x7fdf6813a0d0 2023-10-30T12:00:50.128 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.125+0000 7fdf5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:50.128 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.125+0000 7fdf5f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:50.129 INFO:teuthology.misc.health.smithi050.stdout: 2023-10-30T12:00:50.129 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":12,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","created":"2023-10-30T12:00:40.280079+0000","modified":"2023-10-30T12:00:49.409444+0000","last_up_change":"2023-10-30T12:00:48.401197+0000","last_in_change":"2023-10-30T12:00:41.981126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:00:48.447131+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dba8cbf1-bc2e-4420-957f-11ec607e814e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6817","nonce":47190190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6819","nonce":47190190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6823","nonce":47190190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6821","nonce":47190190}]},"public_addr":"172.21.15.50:6817/47190190","cluster_addr":"172.21.15.50:6819/47190190","heartbeat_back_addr":"172.21.15.50:6823/47190190","heartbeat_front_addr":"172.21.15.50:6821/47190190","state":["exists","up"]},{"osd":1,"uuid":"c13ebbef-aa1a-4da7-b5f6-26b02e43955c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6809","nonce":659398918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6811","nonce":659398918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6815","nonce":659398918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6813","nonce":659398918}]},"public_addr":"172.21.15.50:6809/659398918","cluster_addr":"172.21.15.50:6811/659398918","heartbeat_back_addr":"172.21.15.50:6815/659398918","heartbeat_front_addr":"172.21.15.50:6813/659398918","state":["exists","up"]},{"osd":2,"uuid":"23116074-425b-4664-a1be-0e40bba62fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6801","nonce":3518075216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6803","nonce":3518075216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6807","nonce":3518075216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6805","nonce":3518075216}]},"public_addr":"172.21.15.50:6801/3518075216","cluster_addr":"172.21.15.50:6803/3518075216","heartbeat_back_addr":"172.21.15.50:6807/3518075216","heartbeat_front_addr":"172.21.15.50:6805/3518075216","state":["exists","up"]},{"osd":3,"uuid":"515e513c-cb67-4495-ba25-68cb35215c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6810","nonce":1009584072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6814","nonce":1009584072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6822","nonce":1009584072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6818","nonce":1009584072}]},"public_addr":"172.21.15.118:6810/1009584072","cluster_addr":"172.21.15.118:6814/1009584072","heartbeat_back_addr":"172.21.15.118:6822/1009584072","heartbeat_front_addr":"172.21.15.118:6818/1009584072","state":["exists","up"]},{"osd":4,"uuid":"fa9df8af-676c-49c6-9888-0e330fbbc4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6811","nonce":4272127357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6815","nonce":4272127357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6823","nonce":4272127357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6819","nonce":4272127357}]},"public_addr":"172.21.15.118:6811/4272127357","cluster_addr":"172.21.15.118:6815/4272127357","heartbeat_back_addr":"172.21.15.118:6823/4272127357","heartbeat_front_addr":"172.21.15.118:6819/4272127357","state":["exists","up"]},{"osd":5,"uuid":"79b07bf0-2944-4413-b6e1-23ecbc533add","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6801","nonce":1399177807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6803","nonce":1399177807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6807","nonce":1399177807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6805","nonce":1399177807}]},"public_addr":"172.21.15.118:6801/1399177807","cluster_addr":"172.21.15.118:6803/1399177807","heartbeat_back_addr":"172.21.15.118:6807/1399177807","heartbeat_front_addr":"172.21.15.118:6805/1399177807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.083230+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.120938+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.049176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.192079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:00:50.130 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 10 librados: watch_flush enter 2023-10-30T12:00:50.131 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 10 librados: watch_flush exit 2023-10-30T12:00:50.131 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 20 client.4172.objecter shutdown clearing up homeless session... 2023-10-30T12:00:50.131 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 10 client.4172.objecter successfully canceled tick 2023-10-30T12:00:50.131 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf48039260 msgr2=0x7fdf4803b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:50.131 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf48039260 0x7fdf4803b720 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009aa0 tx=0x7fdf58009190 comp rx=0 tx=0).stop 2023-10-30T12:00:50.132 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 10 monclient: shutdown 2023-10-30T12:00:50.132 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:50.132 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf6813a0d0 msgr2=0x7fdf68122ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:50.132 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf6813a0d0 0x7fdf68122ba0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000c730 tx=0x7fdf6000cbf0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.132 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 shutdown_connections 2023-10-30T12:00:50.132 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf48039260 0x7fdf4803b720 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.133 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf6813a0d0 0x7fdf68122ba0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.133 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf680a1610 0x7fdf68122660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.133 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 --2- 172.21.15.50:0/759464642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf680a1230 0x7fdf681ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.133 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 >> 172.21.15.50:0/759464642 conn(0x7fdf680a2cd0 msgr2=0x7fdf6806e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:50.133 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 shutdown_connections 2023-10-30T12:00:50.133 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 -- 172.21.15.50:0/759464642 wait complete. 2023-10-30T12:00:50.134 INFO:teuthology.misc.health.smithi050.stderr:2023-10-30T12:00:50.127+0000 7fdf6d01f640 1 librados: shutdown 2023-10-30T12:00:50.138 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-10-30T12:00:50.139 INFO:tasks.ceph:Creating RBD pool 2023-10-30T12:00:50.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-30T12:00:50.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.242+0000 7f6e0df90640 10 monclient: get_monmap_and_config 2023-10-30T12:00:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.242+0000 7f6e0df90640 10 monclient: build_initial_monmap 2023-10-30T12:00:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.242+0000 7f6e0df90640 10 monclient: monmap: 2023-10-30T12:00:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:50.243137+0000 2023-10-30T12:00:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:50.243137+0000 2023-10-30T12:00:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:50.246 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:50.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.242+0000 7f6e0df90640 1 Processor -- start 2023-10-30T12:00:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.242+0000 7f6e0df90640 1 -- start start 2023-10-30T12:00:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.242+0000 7f6e0df90640 10 monclient: init 2023-10-30T12:00:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-10-30T12:00:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e081327f0 0x7f6e08132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:50.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): picked mon.noname-b con 0x7f6e081327f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:50.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6e08099a40 0x7f6e08135700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:50.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): picked mon.noname-a con 0x7f6e08099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:50.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6e08135c40 0x7f6e08136090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:50.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): picked mon.noname-c con 0x7f6e08135c40 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e081365d0 con 0x7f6e08099a40 2023-10-30T12:00:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0809a6b0 con 0x7f6e08135c40 2023-10-30T12:00:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080a1620 con 0x7f6e081327f0 2023-10-30T12:00:50.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:50.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e0df90640 10 monclient(hunting): authenticate will time out at 895.426295s 2023-10-30T12:00:50.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e081327f0 0x7f6e08132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:50.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e081327f0 0x7f6e08132bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35494/0 (socket says 172.21.15.50:35494) 2023-10-30T12:00:50.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 -- 172.21.15.50:0/40675594 learned_addr learned my addr 172.21.15.50:0/40675594 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:50.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): get_auth_request con 0x7f6e081327f0 auth_method 0 2023-10-30T12:00:50.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:50.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:50.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:50.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 1 --2- 172.21.15.50:0/40675594 >> v2:172.21.15.50:3300/0 conn(0x7f6e08099a40 0x7f6e08135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:50.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 10 monclient(hunting): get_auth_request con 0x7f6e08099a40 auth_method 0 2023-10-30T12:00:50.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:50.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:50.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:50.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:50.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:50.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:50.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:50.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:50.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:50.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e07fff640 1 --2- 172.21.15.50:0/40675594 >> v2:172.21.15.118:3300/0 conn(0x7f6e08135c40 0x7f6e08136090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:50.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e07fff640 10 monclient(hunting): get_auth_request con 0x7f6e08135c40 auth_method 0 2023-10-30T12:00:50.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:50.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e07fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:50.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e07fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:50.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient(hunting): handle_auth_done global_id 4175 payload 274 2023-10-30T12:00:50.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 -- 172.21.15.50:0/40675594 >> v2:172.21.15.118:3300/0 conn(0x7f6e08135c40 msgr2=0x7f6e08136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:50.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 --2- 172.21.15.50:0/40675594 >> v2:172.21.15.118:3300/0 conn(0x7f6e08135c40 0x7f6e08136090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 -- 172.21.15.50:0/40675594 >> v2:172.21.15.50:3300/0 conn(0x7f6e08099a40 msgr2=0x7f6e08135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:50.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 --2- 172.21.15.50:0/40675594 >> v2:172.21.15.50:3300/0 conn(0x7f6e08099a40 0x7f6e08135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:50.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 monclient: found mon.noname-b 2023-10-30T12:00:50.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:50.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:50.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.243+0000 7f6e077fe640 1 -- 172.21.15.50:0/40675594 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e080a1b20 con 0x7f6e081327f0 2023-10-30T12:00:50.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e077fe640 1 --2- 172.21.15.50:0/40675594 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e081327f0 0x7f6e08132bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6e080a7990 tx=0x7f6df802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8474015e263dd294 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:50.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 1 -- 172.21.15.50:0/40675594 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8031740 con 0x7f6e081327f0 2023-10-30T12:00:50.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:50.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:50.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: dump: 2023-10-30T12:00:50.266 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:50.266 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:50.266 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.266 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.266 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:50.267 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 1 -- 172.21.15.50:0/40675594 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df80318a0 con 0x7f6e081327f0 2023-10-30T12:00:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e0df90640 5 monclient: authenticate success, global_id 4175 2023-10-30T12:00:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e0df90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:50.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:50.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 1 -- 172.21.15.50:0/40675594 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8031c30 con 0x7f6e081327f0 2023-10-30T12:00:50.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:50.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:50.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e067fc640 10 monclient: dump: 2023-10-30T12:00:50.269 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:50.271 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e0df90640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:50.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.244+0000 7f6e0df90640 4 set_mon_vals no callback set 2023-10-30T12:00:50.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 10 monclient: shutdown 2023-10-30T12:00:50.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:50.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 -- 172.21.15.50:0/40675594 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e081327f0 msgr2=0x7f6e08132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:50.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 --2- 172.21.15.50:0/40675594 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e081327f0 0x7f6e08132bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6e080a7990 tx=0x7f6df802b7e0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 -- 172.21.15.50:0/40675594 shutdown_connections 2023-10-30T12:00:50.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 --2- 172.21.15.50:0/40675594 >> v2:172.21.15.118:3300/0 conn(0x7f6e08135c40 0x7f6e08136090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 --2- 172.21.15.50:0/40675594 >> v2:172.21.15.50:3300/0 conn(0x7f6e08099a40 0x7f6e08135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 --2- 172.21.15.50:0/40675594 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e081327f0 0x7f6e08132bd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 -- 172.21.15.50:0/40675594 >> 172.21.15.50:0/40675594 conn(0x7f6e080a82f0 msgr2=0x7f6e080a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:50.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 -- 172.21.15.50:0/40675594 shutdown_connections 2023-10-30T12:00:50.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 -- 172.21.15.50:0/40675594 wait complete. 2023-10-30T12:00:50.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 10 monclient: build_initial_monmap 2023-10-30T12:00:50.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 10 monclient: monmap: 2023-10-30T12:00:50.274 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:50.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:50.275 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:50.275 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:50.275 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:50.275 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:50.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:50.276 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:50.276 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:50.276 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:50.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 librados: starting msgr at 2023-10-30T12:00:50.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.245+0000 7f6e0df90640 1 librados: starting objecter 2023-10-30T12:00:50.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 Processor -- start 2023-10-30T12:00:50.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 -- start start 2023-10-30T12:00:50.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 librados: setting wanted keys 2023-10-30T12:00:50.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 librados: calling monclient init 2023-10-30T12:00:50.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient: init 2023-10-30T12:00:50.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:50.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:50.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e08099a40 0x7f6e0811e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:50.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): picked mon.noname-a con 0x7f6e08099a40 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:50.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e081327f0 0x7f6e0811ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:50.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): picked mon.noname-b con 0x7f6e081327f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:50.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e08135c40 0x7f6e0811f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:50.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): picked mon.noname-c con 0x7f6e08135c40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:50.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:50.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08120cc0 con 0x7f6e08099a40 2023-10-30T12:00:50.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:50.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08121e00 con 0x7f6e081327f0 2023-10-30T12:00:50.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:50.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081223c0 con 0x7f6e08135c40 2023-10-30T12:00:50.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:50.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e0df90640 10 monclient(hunting): authenticate will time out at 895.429724s 2023-10-30T12:00:50.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e08099a40 0x7f6e0811e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:50.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e08099a40 0x7f6e0811e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37924/0 (socket says 172.21.15.50:37924) 2023-10-30T12:00:50.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 1 -- 172.21.15.50:0/1715052851 learned_addr learned my addr 172.21.15.50:0/1715052851 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:50.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 10 monclient(hunting): get_auth_request con 0x7f6e08099a40 auth_method 0 2023-10-30T12:00:50.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e08135c40 0x7f6e0811f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:50.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:50.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:50.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:50.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 10 monclient(hunting): get_auth_request con 0x7f6e08135c40 auth_method 0 2023-10-30T12:00:50.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:50.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:50.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:50.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:50.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:50.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:50.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:50.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:50.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.246+0000 7f6e07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:50.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e06ffd640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e081327f0 0x7f6e0811ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:50.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e06ffd640 10 monclient(hunting): get_auth_request con 0x7f6e081327f0 auth_method 0 2023-10-30T12:00:50.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:50.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e06ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:50.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e06ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:50.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 10 monclient(hunting): handle_auth_done global_id 4209 payload 824 2023-10-30T12:00:50.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 1 -- 172.21.15.50:0/1715052851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e08135c40 msgr2=0x7f6e0811f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:50.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e08135c40 0x7f6e0811f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 1 -- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e081327f0 msgr2=0x7f6e0811ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:50.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e081327f0 0x7f6e0811ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:50.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:50.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 1 monclient: found mon.noname-a 2023-10-30T12:00:50.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:50.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:50.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 1 -- 172.21.15.50:0/1715052851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80094d0 con 0x7f6e08099a40 2023-10-30T12:00:50.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e077fe640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e08099a40 0x7f6e0811e780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6df802b820 tx=0x7f6df80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:50.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 client.?.objecter ms_handle_connect 0x7f6e08099a40 2023-10-30T12:00:50.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:50.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 1 -- 172.21.15.50:0/1715052851 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8033a40 con 0x7f6e08099a40 2023-10-30T12:00:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: dump: 2023-10-30T12:00:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:50.290 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: _finish_auth 0 2023-10-30T12:00:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:50.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 1 -- 172.21.15.50:0/1715052851 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8033bc0 con 0x7f6e08099a40 2023-10-30T12:00:50.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 5 monclient: authenticate success, global_id 4209 2023-10-30T12:00:50.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 10 monclient: _renew_subs 2023-10-30T12:00:50.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:50.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e080a1850 con 0x7f6e08099a40 2023-10-30T12:00:50.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:50.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 1 -- 172.21.15.50:0/1715052851 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df803c400 con 0x7f6e08099a40 2023-10-30T12:00:50.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:50.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:50.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:50.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6deffff640 10 monclient: dump: 2023-10-30T12:00:50.293 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:50.293 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:50.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:50.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:50.294 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:50.294 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:50.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:50.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:50.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:50.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e05ffb640 4 set_mon_vals no callback set 2023-10-30T12:00:50.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 10 monclient: _renew_subs 2023-10-30T12:00:50.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:50.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e08120010 con 0x7f6e08099a40 2023-10-30T12:00:50.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.247+0000 7f6e0df90640 1 librados: init done 2023-10-30T12:00:50.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6deffff640 1 -- 172.21.15.50:0/1715052851 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) v1 ==== 44799+0+0 (secure 0 0 0) 0x7f6df803a040 con 0x7f6e08099a40 2023-10-30T12:00:50.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6e0df90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:50.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6e0df90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:50.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6e0df90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:50.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6deffff640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df4039260 0x7f6df403b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:50.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e08121f70 con 0x7f6e08099a40 2023-10-30T12:00:50.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6deffff640 1 -- 172.21.15.50:0/1715052851 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3982+0+0 (secure 0 0 0) 0x7f6df806bc40 con 0x7f6e08099a40 2023-10-30T12:00:50.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6deffff640 10 client.4209.objecter ms_dispatch 0x7f6e08000b90 osd_map(12..12 src has 1..12) v4 2023-10-30T12:00:50.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6deffff640 3 client.4209.objecter handle_osd_map got epochs [12,12] > 0 2023-10-30T12:00:50.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6deffff640 3 client.4209.objecter handle_osd_map decoding full epoch 12 2023-10-30T12:00:50.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6deffff640 20 client.4209.objecter dump_active .. 0 homeless 2023-10-30T12:00:50.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6e06ffd640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df4039260 0x7f6df403b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:50.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.248+0000 7f6e06ffd640 10 monclient: get_auth_request con 0x7f6df4039260 auth_method 0 2023-10-30T12:00:50.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.249+0000 7f6deffff640 10 client.4209.objecter ms_handle_connect 0x7f6df4039260 2023-10-30T12:00:50.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.249+0000 7f6e06ffd640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df4039260 0x7f6df403b720 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e0811fca0 tx=0x7f6dfc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:50.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.253+0000 7f6deffff640 1 -- 172.21.15.50:0/1715052851 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6df802f080 con 0x7f6e08099a40 2023-10-30T12:00:50.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.253+0000 7f6deffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:50.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.253+0000 7f6deffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:50.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.433+0000 7f6e0df90640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:00:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.433+0000 7f6e0df90640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:00:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.433+0000 7f6e0df90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:50.433+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f6e0804e1d0 con 0x7f6e08099a40 2023-10-30T12:00:51.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.469+0000 7f6deffff640 1 -- 172.21.15.50:0/1715052851 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f6df8038070 con 0x7f6e08099a40 2023-10-30T12:00:51.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.469+0000 7f6deffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-10-30T12:00:51.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.469+0000 7f6deffff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-10-30T12:00:51.472 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2023-10-30T12:00:51.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 10 librados: watch_flush enter 2023-10-30T12:00:51.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 10 librados: watch_flush exit 2023-10-30T12:00:51.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 20 client.4209.objecter shutdown clearing up homeless session... 2023-10-30T12:00:51.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 10 client.4209.objecter successfully canceled tick 2023-10-30T12:00:51.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df4039260 msgr2=0x7f6df403b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:51.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df4039260 0x7f6df403b720 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e0811fca0 tx=0x7f6dfc008040 comp rx=0 tx=0).stop 2023-10-30T12:00:51.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 10 monclient: shutdown 2023-10-30T12:00:51.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:51.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e08099a40 msgr2=0x7f6e0811e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e08099a40 0x7f6e0811e780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6df802b820 tx=0x7f6df80046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 shutdown_connections 2023-10-30T12:00:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df4039260 0x7f6df403b720 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e08135c40 0x7f6e0811f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e081327f0 0x7f6e0811ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 --2- 172.21.15.50:0/1715052851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e08099a40 0x7f6e0811e780 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 >> 172.21.15.50:0/1715052851 conn(0x7f6e080a82f0 msgr2=0x7f6e080a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:51.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 shutdown_connections 2023-10-30T12:00:51.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 -- 172.21.15.50:0/1715052851 wait complete. 2023-10-30T12:00:51.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.472+0000 7f6e0df90640 1 librados: shutdown 2023-10-30T12:00:51.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-30T12:00:51.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.589+0000 7f511a0da640 10 monclient: get_monmap_and_config 2023-10-30T12:00:51.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.589+0000 7f511a0da640 10 monclient: build_initial_monmap 2023-10-30T12:00:51.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.589+0000 7f511a0da640 10 monclient: monmap: 2023-10-30T12:00:51.592 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:51.592 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:51.592 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:51.590687+0000 2023-10-30T12:00:51.592 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:51.590687+0000 2023-10-30T12:00:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:51.594 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:51.594 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:51.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 1 Processor -- start 2023-10-30T12:00:51.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 1 -- start start 2023-10-30T12:00:51.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient: init 2023-10-30T12:00:51.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:51.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-10-30T12:00:51.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f51141327f0 0x7f5114132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:51.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient(hunting): picked mon.noname-c con 0x7f51141327f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5114099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5114135e10 0x7f511413da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient(hunting): picked mon.noname-b con 0x7f5114135e10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51140a1620 con 0x7f5114099a40 2023-10-30T12:00:51.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:51.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511409a6b0 con 0x7f51141327f0 2023-10-30T12:00:51.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:51.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51140a1b20 con 0x7f5114135e10 2023-10-30T12:00:51.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:51.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f511a0da640 10 monclient(hunting): authenticate will time out at 896.773858s 2023-10-30T12:00:51.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.590+0000 7f5112ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5114099a40 0x7f5114135700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60768/0 (socket says 172.21.15.50:60768) 2023-10-30T12:00:51.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f5112ffd640 1 -- 172.21.15.50:0/3268770030 learned_addr learned my addr 172.21.15.50:0/3268770030 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:51.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 1 --2- 172.21.15.50:0/3268770030 >> v2:172.21.15.118:3300/0 conn(0x7f51141327f0 0x7f5114132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:51.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f5113fff640 1 --2- 172.21.15.50:0/3268770030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5114135e10 0x7f511413da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:51.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): get_auth_request con 0x7f51141327f0 auth_method 0 2023-10-30T12:00:51.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:51.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:51.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:51.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f5113fff640 10 monclient(hunting): get_auth_request con 0x7f5114135e10 auth_method 0 2023-10-30T12:00:51.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f5113fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:51.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f5113fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:51.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f5113fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:51.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:51.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:51.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:51.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51127fc640 1 -- 172.21.15.50:0/3268770030 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805296946 0 0) 0x7f51140a1620 con 0x7f5114099a40 2023-10-30T12:00:51.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51127fc640 10 monclient(hunting): _negotiate 2023-10-30T12:00:51.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51127fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51127fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51127fc640 10 monclient(hunting): my global_id is 4221 2023-10-30T12:00:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51127fc640 1 -- 172.21.15.50:0/3268770030 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50fc003520 con 0x7f5114099a40 2023-10-30T12:00:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2023-10-30T12:00:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 1 -- 172.21.15.50:0/3268770030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5114135e10 msgr2=0x7f511413da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 1 --2- 172.21.15.50:0/3268770030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5114135e10 0x7f511413da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 1 -- 172.21.15.50:0/3268770030 >> v1:172.21.15.50:6789/0 conn(0x7f5114099a40 legacy=0x7f5114135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 1 monclient: found mon.noname-c 2023-10-30T12:00:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f51137fe640 1 -- 172.21.15.50:0/3268770030 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114133110 con 0x7f51141327f0 2023-10-30T12:00:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.591+0000 7f5113fff640 1 --2- 172.21.15.50:0/3268770030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5114135e10 0x7f511413da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51137fe640 1 --2- 172.21.15.50:0/3268770030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51141327f0 0x7f5114132bd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f51080097a0 tx=0x7f510802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a87bc080874a20a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 1 -- 172.21.15.50:0/3268770030 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510802f050 con 0x7f51141327f0 2023-10-30T12:00:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: dump: 2023-10-30T12:00:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:51.615 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 1 -- 172.21.15.50:0/3268770030 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108031700 con 0x7f51141327f0 2023-10-30T12:00:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 1 -- 172.21.15.50:0/3268770030 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108031a90 con 0x7f51141327f0 2023-10-30T12:00:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f511a0da640 5 monclient: authenticate success, global_id 4246 2023-10-30T12:00:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f511a0da640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f511a0da640 4 set_mon_vals no callback set 2023-10-30T12:00:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.592+0000 7f51127fc640 10 monclient: dump: 2023-10-30T12:00:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:51.619 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 10 monclient: shutdown 2023-10-30T12:00:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 -- 172.21.15.50:0/3268770030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51141327f0 msgr2=0x7f5114132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 --2- 172.21.15.50:0/3268770030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51141327f0 0x7f5114132bd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f51080097a0 tx=0x7f510802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 -- 172.21.15.50:0/3268770030 shutdown_connections 2023-10-30T12:00:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 --2- 172.21.15.50:0/3268770030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5114135e10 0x7f511413da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 --2- 172.21.15.50:0/3268770030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51141327f0 0x7f5114132bd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 -- 172.21.15.50:0/3268770030 >> 172.21.15.50:0/3268770030 conn(0x7f51140a82f0 msgr2=0x7f51140a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 -- 172.21.15.50:0/3268770030 shutdown_connections 2023-10-30T12:00:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 -- 172.21.15.50:0/3268770030 wait complete. 2023-10-30T12:00:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 10 monclient: build_initial_monmap 2023-10-30T12:00:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 10 monclient: monmap: 2023-10-30T12:00:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:51.623 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 librados: starting msgr at 2023-10-30T12:00:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.593+0000 7f511a0da640 1 librados: starting objecter 2023-10-30T12:00:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 Processor -- start 2023-10-30T12:00:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 -- start start 2023-10-30T12:00:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 librados: setting wanted keys 2023-10-30T12:00:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 librados: calling monclient init 2023-10-30T12:00:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient: init 2023-10-30T12:00:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5114099a40 0x7f51141ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): picked mon.noname-b con 0x7f5114099a40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51141327f0 0x7f51141ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): picked mon.noname-c con 0x7f51141327f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5114135e10 0x7f511411e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): picked mon.noname-a con 0x7f5114135e10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114122660 con 0x7f5114135e10 2023-10-30T12:00:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51141228b0 con 0x7f5114099a40 2023-10-30T12:00:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5114121120 con 0x7f51141327f0 2023-10-30T12:00:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f511a0da640 10 monclient(hunting): authenticate will time out at 896.777773s 2023-10-30T12:00:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5113fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5114135e10 0x7f511411e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5113fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5114135e10 0x7f511411e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37936/0 (socket says 172.21.15.50:37936) 2023-10-30T12:00:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5112ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51141327f0 0x7f51141ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5113fff640 1 -- 172.21.15.50:0/4057504667 learned_addr learned my addr 172.21.15.50:0/4057504667 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5113fff640 10 monclient(hunting): get_auth_request con 0x7f5114135e10 auth_method 0 2023-10-30T12:00:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5113fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5113fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5113fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f51137fe640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5114099a40 0x7f51141ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f51137fe640 10 monclient(hunting): get_auth_request con 0x7f5114099a40 auth_method 0 2023-10-30T12:00:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f51137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f51137fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f51137fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5112ffd640 10 monclient(hunting): get_auth_request con 0x7f51141327f0 auth_method 0 2023-10-30T12:00:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5112ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.594+0000 7f5112ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f5113fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f5113fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f5113fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 10 monclient(hunting): handle_auth_done global_id 4249 payload 824 2023-10-30T12:00:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 1 -- 172.21.15.50:0/4057504667 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51141327f0 msgr2=0x7f51141ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51141327f0 0x7f51141ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 1 -- 172.21.15.50:0/4057504667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5114135e10 msgr2=0x7f511411e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5114135e10 0x7f511411e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 1 monclient: found mon.noname-b 2023-10-30T12:00:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 1 -- 172.21.15.50:0/4057504667 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5108009400 con 0x7f5114099a40 2023-10-30T12:00:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f51137fe640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5114099a40 0x7f51141ca2d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f51140a5530 tx=0x7f51080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 client.?.objecter ms_handle_connect 0x7f5114099a40 2023-10-30T12:00:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 1 -- 172.21.15.50:0/4057504667 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108031500 con 0x7f5114099a40 2023-10-30T12:00:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: dump: 2023-10-30T12:00:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:51.637 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: _finish_auth 0 2023-10-30T12:00:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 1 -- 172.21.15.50:0/4057504667 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108033a30 con 0x7f5114099a40 2023-10-30T12:00:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 1 -- 172.21.15.50:0/4057504667 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108033de0 con 0x7f5114099a40 2023-10-30T12:00:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 5 monclient: authenticate success, global_id 4249 2023-10-30T12:00:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 10 monclient: _renew_subs 2023-10-30T12:00:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51140a17f0 con 0x7f5114099a40 2023-10-30T12:00:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f5111ffb640 4 set_mon_vals no callback set 2023-10-30T12:00:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f50fbfff640 10 monclient: dump: 2023-10-30T12:00:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:51.642 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 10 monclient: _renew_subs 2023-10-30T12:00:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51141204e0 con 0x7f5114099a40 2023-10-30T12:00:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.595+0000 7f511a0da640 1 librados: init done 2023-10-30T12:00:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f50fbfff640 1 -- 172.21.15.50:0/4057504667 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 44799+0+0 (secure 0 0 0) 0x7f510803a040 con 0x7f5114099a40 2023-10-30T12:00:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f511a0da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f511a0da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f511a0da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5114122130 con 0x7f5114099a40 2023-10-30T12:00:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f50fbfff640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5100039260 0x7f510003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f50fbfff640 1 -- 172.21.15.50:0/4057504667 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4335+0+0 (secure 0 0 0) 0x7f510806b640 con 0x7f5114099a40 2023-10-30T12:00:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f50fbfff640 10 client.4249.objecter ms_dispatch 0x7f5114000b90 osd_map(14..14 src has 1..14) v4 2023-10-30T12:00:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f50fbfff640 3 client.4249.objecter handle_osd_map got epochs [14,14] > 0 2023-10-30T12:00:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f50fbfff640 3 client.4249.objecter handle_osd_map decoding full epoch 14 2023-10-30T12:00:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f50fbfff640 20 client.4249.objecter dump_active .. 0 homeless 2023-10-30T12:00:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f5112ffd640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5100039260 0x7f510003b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.596+0000 7f5112ffd640 10 monclient: get_auth_request con 0x7f5100039260 auth_method 0 2023-10-30T12:00:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.597+0000 7f5112ffd640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5100039260 0x7f510003b720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f51040011e0 tx=0x7f5104007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.597+0000 7f50fbfff640 10 client.4249.objecter ms_handle_connect 0x7f5100039260 2023-10-30T12:00:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.601+0000 7f50fbfff640 1 -- 172.21.15.50:0/4057504667 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5108045360 con 0x7f5114099a40 2023-10-30T12:00:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.601+0000 7f50fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.601+0000 7f50fbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:51.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.782+0000 7f511a0da640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:00:51.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.782+0000 7f511a0da640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:00:51.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.782+0000 7f511a0da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:51.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:51.782+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5114122410 con 0x7f5114099a40 2023-10-30T12:00:52.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.473+0000 7f50fbfff640 1 -- 172.21.15.50:0/4057504667 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5108042070 con 0x7f5114099a40 2023-10-30T12:00:52.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.473+0000 7f50fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-10-30T12:00:52.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.473+0000 7f50fbfff640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-10-30T12:00:52.476 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-30T12:00:52.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 10 librados: watch_flush enter 2023-10-30T12:00:52.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 10 librados: watch_flush exit 2023-10-30T12:00:52.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 20 client.4249.objecter shutdown clearing up homeless session... 2023-10-30T12:00:52.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 10 client.4249.objecter successfully canceled tick 2023-10-30T12:00:52.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5100039260 msgr2=0x7f510003b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5100039260 0x7f510003b720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f51040011e0 tx=0x7f5104007d30 comp rx=0 tx=0).stop 2023-10-30T12:00:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 10 monclient: shutdown 2023-10-30T12:00:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5114099a40 msgr2=0x7f51141ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.478+0000 7f511a0da640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5114099a40 0x7f51141ca2d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f51140a5530 tx=0x7f51080046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.479+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 shutdown_connections 2023-10-30T12:00:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.479+0000 7f511a0da640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5100039260 0x7f510003b720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.479+0000 7f511a0da640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5114135e10 0x7f511411e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.479+0000 7f511a0da640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51141327f0 0x7f51141ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.479+0000 7f511a0da640 1 --2- 172.21.15.50:0/4057504667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5114099a40 0x7f51141ca2d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.479+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 >> 172.21.15.50:0/4057504667 conn(0x7f51140a82f0 msgr2=0x7f51140a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.480+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 shutdown_connections 2023-10-30T12:00:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.480+0000 7f511a0da640 1 -- 172.21.15.50:0/4057504667 wait complete. 2023-10-30T12:00:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.480+0000 7f511a0da640 1 librados: shutdown 2023-10-30T12:00:52.494 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T12:00:52.495 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T12:00:52.495 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T12:00:52.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T12:00:52.496 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T12:00:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.606+0000 7fc757eda640 10 monclient: get_monmap_and_config 2023-10-30T12:00:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.606+0000 7fc757eda640 10 monclient: build_initial_monmap 2023-10-30T12:00:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.606+0000 7fc757eda640 10 monclient: monmap: 2023-10-30T12:00:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:52.609 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:52.609 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:52.607562+0000 2023-10-30T12:00:52.609 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:52.607562+0000 2023-10-30T12:00:52.609 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:52.609 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.610 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:52.610 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:52.610 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:52.610 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:52.610 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:52.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 1 Processor -- start 2023-10-30T12:00:52.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 1 -- start start 2023-10-30T12:00:52.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient: init 2023-10-30T12:00:52.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:52.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-10-30T12:00:52.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc7500a2dd0 0x7fc7500a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient(hunting): picked mon.noname-b con 0x7fc7500a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:52.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc75006a8e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:52.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc750071400 0x7fc75013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient(hunting): picked mon.noname-c con 0x7fc750071400 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:52.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500a31b0 con 0x7fc75006a8e0 2023-10-30T12:00:52.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75006a6a0 con 0x7fc750071400 2023-10-30T12:00:52.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75006be90 con 0x7fc7500a2dd0 2023-10-30T12:00:52.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:52.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc757eda640 10 monclient(hunting): authenticate will time out at 897.790752s 2023-10-30T12:00:52.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc755c76640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc7500a2dd0 0x7fc7500a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc755c76640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc7500a2dd0 0x7fc7500a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35530/0 (socket says 172.21.15.50:35530) 2023-10-30T12:00:52.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc755c76640 1 -- 172.21.15.50:0/3810244143 learned_addr learned my addr 172.21.15.50:0/3810244143 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:52.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc755c76640 10 monclient(hunting): get_auth_request con 0x7fc7500a2dd0 auth_method 0 2023-10-30T12:00:52.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc755c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc755c76640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.607+0000 7fc755c76640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:52.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:52.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:52.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:52.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc756477640 1 --2- 172.21.15.50:0/3810244143 >> v2:172.21.15.118:3300/0 conn(0x7fc750071400 0x7fc75013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc756477640 10 monclient(hunting): get_auth_request con 0x7fc750071400 auth_method 0 2023-10-30T12:00:52.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc756477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc756477640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc756477640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:52.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 10 monclient(hunting): handle_auth_done global_id 4196 payload 274 2023-10-30T12:00:52.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 1 -- 172.21.15.50:0/3810244143 >> v2:172.21.15.118:3300/0 conn(0x7fc750071400 msgr2=0x7fc75013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:52.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 1 --2- 172.21.15.50:0/3810244143 >> v2:172.21.15.118:3300/0 conn(0x7fc750071400 0x7fc75013cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 1 -- 172.21.15.50:0/3810244143 >> v1:172.21.15.118:6789/0 conn(0x7fc75006a8e0 legacy=0x7fc7500a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:52.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 10 monclient: _finish_hunting 0 2023-10-30T12:00:52.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 1 monclient: found mon.noname-b 2023-10-30T12:00:52.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:52.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:52.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 1 -- 172.21.15.50:0/3810244143 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75009e190 con 0x7fc7500a2dd0 2023-10-30T12:00:52.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.608+0000 7fc755c76640 1 --2- 172.21.15.50:0/3810244143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc7500a2dd0 0x7fc7500a46c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009e80 tx=0x7fc74c02bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99926d4ad24361c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:52.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 1 -- 172.21.15.50:0/3810244143 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c02c930 con 0x7fc7500a2dd0 2023-10-30T12:00:52.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:52.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:52.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: dump: 2023-10-30T12:00:52.626 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:52.627 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:52.627 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.627 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.627 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:52.627 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.629 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:52.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:52.629 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:52.629 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: _finish_auth 0 2023-10-30T12:00:52.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: _check_auth_tickets 2023-10-30T12:00:52.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:52.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 5 monclient: authenticate success, global_id 4196 2023-10-30T12:00:52.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:52.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 1 -- 172.21.15.50:0/3810244143 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c02ca90 con 0x7fc7500a2dd0 2023-10-30T12:00:52.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:52.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:52.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 4 set_mon_vals no callback set 2023-10-30T12:00:52.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 1 -- 172.21.15.50:0/3810244143 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c02ce20 con 0x7fc7500a2dd0 2023-10-30T12:00:52.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:52.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:52.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc754c74640 10 monclient: dump: 2023-10-30T12:00:52.632 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:52.632 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:52.632 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.633 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.633 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:52.633 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.633 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:52.633 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:52.634 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:52.634 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 10 monclient: shutdown 2023-10-30T12:00:52.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:52.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 -- 172.21.15.50:0/3810244143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc7500a2dd0 msgr2=0x7fc7500a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 --2- 172.21.15.50:0/3810244143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc7500a2dd0 0x7fc7500a46c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009e80 tx=0x7fc74c02bd90 comp rx=0 tx=0).stop 2023-10-30T12:00:52.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 -- 172.21.15.50:0/3810244143 shutdown_connections 2023-10-30T12:00:52.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 --2- 172.21.15.50:0/3810244143 >> v2:172.21.15.118:3300/0 conn(0x7fc750071400 0x7fc75013cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 --2- 172.21.15.50:0/3810244143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc7500a2dd0 0x7fc7500a46c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 -- 172.21.15.50:0/3810244143 >> 172.21.15.50:0/3810244143 conn(0x7fc7500a22b0 msgr2=0x7fc7500a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:52.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 -- 172.21.15.50:0/3810244143 shutdown_connections 2023-10-30T12:00:52.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.609+0000 7fc757eda640 1 -- 172.21.15.50:0/3810244143 wait complete. 2023-10-30T12:00:52.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 10 monclient: build_initial_monmap 2023-10-30T12:00:52.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 10 monclient: monmap: 2023-10-30T12:00:52.636 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:52.636 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:52.636 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:52.637 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:52.637 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:52.637 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.637 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:52.637 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:52.638 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:52.638 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 1 librados: starting msgr at 2023-10-30T12:00:52.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 1 librados: starting objecter 2023-10-30T12:00:52.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 1 Processor -- start 2023-10-30T12:00:52.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 1 -- start start 2023-10-30T12:00:52.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 1 librados: setting wanted keys 2023-10-30T12:00:52.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 1 librados: calling monclient init 2023-10-30T12:00:52.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.610+0000 7fc757eda640 10 monclient: init 2023-10-30T12:00:52.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:52.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:00:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc750071400 0x7fc7501ba900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): picked mon.noname-c con 0x7fc750071400 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7500a2dd0 0x7fc7501a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): picked mon.noname-b con 0x7fc7500a2dd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7501a47e0 0x7fc7501a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): picked mon.noname-a con 0x7fc7501a47e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7501a8350 con 0x7fc7501a47e0 2023-10-30T12:00:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7501bbb80 con 0x7fc7500a2dd0 2023-10-30T12:00:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7501cb040 con 0x7fc750071400 2023-10-30T12:00:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc757eda640 10 monclient(hunting): authenticate will time out at 897.794340s 2023-10-30T12:00:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc750071400 0x7fc7501ba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc750071400 0x7fc7501ba900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35544/0 (socket says 172.21.15.50:35544) 2023-10-30T12:00:52.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 -- 172.21.15.50:0/2247206984 learned_addr learned my addr 172.21.15.50:0/2247206984 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:52.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): get_auth_request con 0x7fc750071400 auth_method 0 2023-10-30T12:00:52.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:52.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7501a47e0 0x7fc7501a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 10 monclient(hunting): get_auth_request con 0x7fc7501a47e0 auth_method 0 2023-10-30T12:00:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient(hunting): handle_auth_done global_id 4199 payload 824 2023-10-30T12:00:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 -- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7500a2dd0 msgr2=0x7fc7501a42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755475640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7500a2dd0 0x7fc7501a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7500a2dd0 0x7fc7501a42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 -- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7501a47e0 msgr2=0x7fc7501a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7501a47e0 0x7fc7501a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient: _finish_hunting 0 2023-10-30T12:00:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 monclient: found mon.noname-c 2023-10-30T12:00:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 -- 172.21.15.50:0/2247206984 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c009b30 con 0x7fc750071400 2023-10-30T12:00:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc756477640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7501a47e0 0x7fc7501a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc755c76640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc750071400 0x7fc7501ba900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036040 tx=0x7fc74c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc7467fc640 10 client.?.objecter ms_handle_connect 0x7fc750071400 2023-10-30T12:00:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.611+0000 7fc7467fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 1 -- 172.21.15.50:0/2247206984 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c03e450 con 0x7fc750071400 2023-10-30T12:00:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: dump: 2023-10-30T12:00:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:52.650 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.650 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:52.650 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:52.650 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:52.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:52.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:52.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:52.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 1 -- 172.21.15.50:0/2247206984 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c03e5b0 con 0x7fc750071400 2023-10-30T12:00:52.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 5 monclient: authenticate success, global_id 4199 2023-10-30T12:00:52.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 10 monclient: _renew_subs 2023-10-30T12:00:52.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:52.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc750070990 con 0x7fc750071400 2023-10-30T12:00:52.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:52.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 1 -- 172.21.15.50:0/2247206984 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c03e960 con 0x7fc750071400 2023-10-30T12:00:52.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:52.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:52.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 10 monclient: dump: 2023-10-30T12:00:52.653 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:52.653 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:52.653 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.654 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.654 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:52.654 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.654 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:52.654 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:52.655 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:52.655 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:52.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 10 monclient: _renew_subs 2023-10-30T12:00:52.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc747fff640 4 set_mon_vals no callback set 2023-10-30T12:00:52.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:52.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7501a7d00 con 0x7fc750071400 2023-10-30T12:00:52.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc757eda640 1 librados: init done 2023-10-30T12:00:52.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.612+0000 7fc7467fc640 1 -- 172.21.15.50:0/2247206984 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc74c03eac0 con 0x7fc750071400 2023-10-30T12:00:52.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc757eda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:52.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc757eda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:52.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc757eda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:52.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75004e160 con 0x7fc750071400 2023-10-30T12:00:52.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc7467fc640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7340392f0 0x7fc73403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc7467fc640 1 -- 172.21.15.50:0/2247206984 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fc74c06d290 con 0x7fc750071400 2023-10-30T12:00:52.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc7467fc640 10 client.4199.objecter ms_dispatch 0x7fc750000b90 osd_map(15..15 src has 1..15) v4 2023-10-30T12:00:52.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc7467fc640 3 client.4199.objecter handle_osd_map got epochs [15,15] > 0 2023-10-30T12:00:52.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc7467fc640 3 client.4199.objecter handle_osd_map decoding full epoch 15 2023-10-30T12:00:52.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc7467fc640 20 client.4199.objecter dump_active .. 0 homeless 2023-10-30T12:00:52.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc755475640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7340392f0 0x7fc73403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc755475640 10 monclient: get_auth_request con 0x7fc7340392f0 auth_method 0 2023-10-30T12:00:52.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc7467fc640 10 client.4199.objecter ms_handle_connect 0x7fc7340392f0 2023-10-30T12:00:52.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.613+0000 7fc755475640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7340392f0 0x7fc73403b7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc7501a6d00 tx=0x7fc740008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:52.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.618+0000 7fc7467fc640 1 -- 172.21.15.50:0/2247206984 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc74c03c350 con 0x7fc750071400 2023-10-30T12:00:52.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.618+0000 7fc7467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:52.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.618+0000 7fc7467fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:52.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.839+0000 7fc757eda640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:00:52.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.839+0000 7fc757eda640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:00:52.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.839+0000 7fc757eda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:52.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.839+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc7501a5ef0 con 0x7fc750071400 2023-10-30T12:00:52.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.841+0000 7fc7467fc640 1 -- 172.21.15.50:0/2247206984 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+86181 (secure 0 0 0) 0x7fc74c043090 con 0x7fc750071400 2023-10-30T12:00:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.841+0000 7fc7467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-10-30T12:00:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.841+0000 7fc7467fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:52.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 10 librados: watch_flush enter 2023-10-30T12:00:52.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 10 librados: watch_flush exit 2023-10-30T12:00:52.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 20 client.4199.objecter shutdown clearing up homeless session... 2023-10-30T12:00:52.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 10 client.4199.objecter successfully canceled tick 2023-10-30T12:00:52.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7340392f0 msgr2=0x7fc73403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7340392f0 0x7fc73403b7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc7501a6d00 tx=0x7fc740008040 comp rx=0 tx=0).stop 2023-10-30T12:00:52.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 10 monclient: shutdown 2023-10-30T12:00:52.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:52.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc750071400 msgr2=0x7fc7501ba900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc750071400 0x7fc7501ba900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036040 tx=0x7fc74c004a40 comp rx=0 tx=0).stop 2023-10-30T12:00:52.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 shutdown_connections 2023-10-30T12:00:52.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7340392f0 0x7fc73403b7b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7501a47e0 0x7fc7501a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7500a2dd0 0x7fc7501a42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 --2- 172.21.15.50:0/2247206984 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc750071400 0x7fc7501ba900 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 >> 172.21.15.50:0/2247206984 conn(0x7fc7500a22b0 msgr2=0x7fc75006dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:52.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 shutdown_connections 2023-10-30T12:00:52.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 -- 172.21.15.50:0/2247206984 wait complete. 2023-10-30T12:00:52.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.843+0000 7fc757eda640 1 librados: shutdown 2023-10-30T12:00:52.855 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":1859404137},{"type":"v1","addr":"172.21.15.118:6825","nonce":1859404137}]},"active_addr":"172.21.15.118:6825/1859404137","active_change":"2023-10-30T12:00:44.386529+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":2334612971}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":552849019}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":2518850359}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3341793726}]}]} 2023-10-30T12:00:52.859 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T12:00:52.860 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T12:00:52.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:00:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.963+0000 7f931097d640 10 monclient: get_monmap_and_config 2023-10-30T12:00:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.963+0000 7f931097d640 10 monclient: build_initial_monmap 2023-10-30T12:00:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.963+0000 7f931097d640 10 monclient: monmap: 2023-10-30T12:00:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:52.964907+0000 2023-10-30T12:00:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:52.964907+0000 2023-10-30T12:00:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:52.968 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:52.968 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:52.968 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.964+0000 7f931097d640 1 Processor -- start 2023-10-30T12:00:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.964+0000 7f931097d640 1 -- start start 2023-10-30T12:00:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.964+0000 7f931097d640 10 monclient: init 2023-10-30T12:00:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-10-30T12:00:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f930c0a6070 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f930c0a57e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f930c13bfd0 0x7f930c13e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient(hunting): picked mon.noname-c con 0x7f930c13bfd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c09a270 con 0x7f930c0a57e0 2023-10-30T12:00:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c099f90 con 0x7f930c0a6070 2023-10-30T12:00:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c09a550 con 0x7f930c13bfd0 2023-10-30T12:00:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f931097d640 10 monclient(hunting): authenticate will time out at 898.148201s 2023-10-30T12:00:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f9309d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f930c0a57e0 0x7f930c13a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60778/0 (socket says 172.21.15.50:60778) 2023-10-30T12:00:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f9309d9b640 1 -- 172.21.15.50:0/954202970 learned_addr learned my addr 172.21.15.50:0/954202970 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 1 --2- 172.21.15.50:0/954202970 >> v2:172.21.15.118:3300/0 conn(0x7f930c13bfd0 0x7f930c13e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 10 monclient(hunting): get_auth_request con 0x7f930c13bfd0 auth_method 0 2023-10-30T12:00:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325131250 0 0) 0x7f930c09a270 con 0x7f930c0a57e0 2023-10-30T12:00:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): _negotiate 2023-10-30T12:00:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): my global_id is 4230 2023-10-30T12:00:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f4003520 con 0x7f930c0a57e0 2023-10-30T12:00:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3719773978 0 0) 0x7f92f4003520 con 0x7f930c0a57e0 2023-10-30T12:00:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 >> v2:172.21.15.118:3300/0 conn(0x7f930c13bfd0 msgr2=0x7f930c13e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 --2- 172.21.15.50:0/954202970 >> v2:172.21.15.118:3300/0 conn(0x7f930c13bfd0 0x7f930c13e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 >> v1:172.21.15.118:6789/0 conn(0x7f930c0a6070 legacy=0x7f930c0a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: _finish_hunting 0 2023-10-30T12:00:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:00:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:00:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c09a830 con 0x7f930c0a57e0 2023-10-30T12:00:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: _finish_auth 0 2023-10-30T12:00:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: _check_auth_tickets 2023-10-30T12:00:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f9300001690 con 0x7f930c0a57e0 2023-10-30T12:00:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:00:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: dump: 2023-10-30T12:00:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:52.987 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:00:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 >> v1:172.21.15.50:6789/0 conn(0x7f930c0a57e0 legacy=0x7f930c13a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f40039c0 0x7f92f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): picked mon.b con 0x7f92f40039c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f92f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): picked mon.c con 0x7f92f40047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92f40052a0 0x7f92f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): picked mon.a con 0x7f92f40052a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c09a550 con 0x7f92f40052a0 2023-10-30T12:00:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f400c1a0 con 0x7f92f40039c0 2023-10-30T12:00:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f400c360 con 0x7f92f40047c0 2023-10-30T12:00:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f930959a640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:00:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.965+0000 7f9309d9b640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f92f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92f40052a0 0x7f92f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): get_auth_request con 0x7f92f40047c0 auth_method 0 2023-10-30T12:00:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f40039c0 0x7f92f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 10 monclient(hunting): get_auth_request con 0x7f92f40052a0 auth_method 0 2023-10-30T12:00:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 10 monclient(hunting): get_auth_request con 0x7f92f40039c0 auth_method 0 2023-10-30T12:00:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:52.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:52.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:52.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:52.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:52.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:52.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:52.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:52.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:52.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient(hunting): handle_auth_done global_id 4230 payload 293 2023-10-30T12:00:52.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 1 -- 172.21.15.50:0/954202970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f40039c0 msgr2=0x7f92f4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:52.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f40039c0 0x7f92f4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 1 -- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92f40052a0 msgr2=0x7f92f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:52.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92f40052a0 0x7f92f400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:52.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:00:52.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 1 monclient: found mon.c 2023-10-30T12:00:52.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 1 -- 172.21.15.50:0/954202970 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4003520 con 0x7f92f40047c0 2023-10-30T12:00:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient: _finish_auth 0 2023-10-30T12:00:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:00:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f9309d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f931097d640 5 monclient: authenticate success, global_id 4230 2023-10-30T12:00:52.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f931097d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:52.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930a59c640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92f40039c0 0x7f92f4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:52.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92f40052a0 0x7f92f400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:52.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.966+0000 7f930ad9d640 1 -- 172.21.15.50:0/954202970 reap_dead start 2023-10-30T12:00:52.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f9309d9b640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f92f4004c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f930000c160 tx=0x7f92f40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6ab599fbc69bee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300005ad0 con 0x7f92f40047c0 2023-10-30T12:00:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 10 monclient: dump: 2023-10-30T12:00:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.000 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.000 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.000 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.000 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.000 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.000 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.001 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.001 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:53.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9300005c30 con 0x7f92f40047c0 2023-10-30T12:00:53.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 4 set_mon_vals no callback set 2023-10-30T12:00:53.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 1 -- 172.21.15.50:0/954202970 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300007070 con 0x7f92f40047c0 2023-10-30T12:00:53.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:53.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f930959a640 10 monclient: dump: 2023-10-30T12:00:53.003 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.003 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.003 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.004 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.004 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.004 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.004 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 10 monclient: shutdown 2023-10-30T12:00:53.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 1 -- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 msgr2=0x7f92f4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f92f4004c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f930000c160 tx=0x7f92f40057f0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 1 -- 172.21.15.50:0/954202970 shutdown_connections 2023-10-30T12:00:53.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 1 --2- 172.21.15.50:0/954202970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f92f4004c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 1 -- 172.21.15.50:0/954202970 >> 172.21.15.50:0/954202970 conn(0x7f930c0a81d0 msgr2=0x7f930c0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 1 -- 172.21.15.50:0/954202970 shutdown_connections 2023-10-30T12:00:53.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.967+0000 7f931097d640 1 -- 172.21.15.50:0/954202970 wait complete. 2023-10-30T12:00:53.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 10 monclient: monmap: 2023-10-30T12:00:53.007 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.007 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.007 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.007 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.007 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.007 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.008 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.008 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.008 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.008 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 1 librados: starting msgr at 2023-10-30T12:00:53.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 1 librados: starting objecter 2023-10-30T12:00:53.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 1 Processor -- start 2023-10-30T12:00:53.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 1 -- start start 2023-10-30T12:00:53.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 1 librados: setting wanted keys 2023-10-30T12:00:53.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 1 librados: calling monclient init 2023-10-30T12:00:53.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.968+0000 7f931097d640 10 monclient: init 2023-10-30T12:00:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:00:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f930c1ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): picked mon.noname-c con 0x7f92f40047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f930c1cab50 0x7f930c09c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): picked mon.noname-a con 0x7f930c1cab50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f930c1cafd0 0x7f930c09cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): picked mon.noname-b con 0x7f930c1cafd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0a1690 con 0x7f930c1cab50 2023-10-30T12:00:53.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c1337e0 con 0x7f930c1cafd0 2023-10-30T12:00:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0a06c0 con 0x7f92f40047c0 2023-10-30T12:00:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f931097d640 10 monclient(hunting): authenticate will time out at 898.152423s 2023-10-30T12:00:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f930c1ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f930c1cab50 0x7f930c09c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f930c1ca610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35572/0 (socket says 172.21.15.50:35572) 2023-10-30T12:00:53.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f930c1cab50 0x7f930c09c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37966/0 (socket says 172.21.15.50:37966) 2023-10-30T12:00:53.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 -- 172.21.15.50:0/2785437409 learned_addr learned my addr 172.21.15.50:0/2785437409 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): get_auth_request con 0x7f92f40047c0 auth_method 0 2023-10-30T12:00:53.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 10 monclient(hunting): get_auth_request con 0x7f930c1cab50 auth_method 0 2023-10-30T12:00:53.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930ad9d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f930c1cafd0 0x7f930c09cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930ad9d640 10 monclient(hunting): get_auth_request con 0x7f930c1cafd0 auth_method 0 2023-10-30T12:00:53.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2023-10-30T12:00:53.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 -- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f930c1cafd0 msgr2=0x7f930c09cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f930c1cafd0 0x7f930c09cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 -- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f930c1cab50 msgr2=0x7f930c09c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f930c1cab50 0x7f930c09c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 monclient: found mon.noname-c 2023-10-30T12:00:53.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930a59c640 1 -- 172.21.15.50:0/2785437409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c09f480 con 0x7f92f40047c0 2023-10-30T12:00:53.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f930ad9d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f930c1cafd0 0x7f930c09cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:53.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.969+0000 7f9309d9b640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f930c1cab50 0x7f930c09c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f930a59c640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f930c1ca610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f92fc007ce0 tx=0x7f92fc0021d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 client.?.objecter ms_handle_connect 0x7f92f40047c0 2023-10-30T12:00:53.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:53.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 1 -- 172.21.15.50:0/2785437409 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc003b80 con 0x7f92f40047c0 2023-10-30T12:00:53.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:53.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: dump: 2023-10-30T12:00:53.020 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.020 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.021 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.021 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.021 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.021 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.021 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.021 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.022 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.022 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 1 -- 172.21.15.50:0/2785437409 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc003ce0 con 0x7f92f40047c0 2023-10-30T12:00:53.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 5 monclient: authenticate success, global_id 4202 2023-10-30T12:00:53.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 10 monclient: _renew_subs 2023-10-30T12:00:53.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c09fe20 con 0x7f92f40047c0 2023-10-30T12:00:53.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 1 -- 172.21.15.50:0/2785437409 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc00db50 con 0x7f92f40047c0 2023-10-30T12:00:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f92faffd640 10 monclient: dump: 2023-10-30T12:00:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.025 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.025 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.025 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.025 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.025 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.026 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.026 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.026 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f9308d99640 4 set_mon_vals no callback set 2023-10-30T12:00:53.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 10 client.4202.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 10 monclient: _renew_subs 2023-10-30T12:00:53.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c0a0ae0 con 0x7f92f40047c0 2023-10-30T12:00:53.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.970+0000 7f931097d640 1 librados: init done 2023-10-30T12:00:53.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f92faffd640 1 -- 172.21.15.50:0/2785437409 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f92fc00dcb0 con 0x7f92f40047c0 2023-10-30T12:00:53.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f931097d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f931097d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f931097d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f92faffd640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92ec039320 0x7f92ec03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c0a0cd0 con 0x7f92f40047c0 2023-10-30T12:00:53.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f92faffd640 1 -- 172.21.15.50:0/2785437409 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f92fc04a640 con 0x7f92f40047c0 2023-10-30T12:00:53.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f92faffd640 10 client.4202.objecter ms_dispatch 0x7f930c000b90 osd_map(15..15 src has 1..15) v4 2023-10-30T12:00:53.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f92faffd640 3 client.4202.objecter handle_osd_map got epochs [15,15] > 0 2023-10-30T12:00:53.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f92faffd640 3 client.4202.objecter handle_osd_map decoding full epoch 15 2023-10-30T12:00:53.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f92faffd640 20 client.4202.objecter dump_active .. 0 homeless 2023-10-30T12:00:53.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f9309d9b640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92ec039320 0x7f92ec03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.971+0000 7f9309d9b640 10 monclient: get_auth_request con 0x7f92ec039320 auth_method 0 2023-10-30T12:00:53.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.972+0000 7f9309d9b640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92ec039320 0x7f92ec03b7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9300034000 tx=0x7f9300039040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.972+0000 7f92faffd640 10 client.4202.objecter ms_handle_connect 0x7f92ec039320 2023-10-30T12:00:53.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.976+0000 7f92faffd640 1 -- 172.21.15.50:0/2785437409 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f92fc012760 con 0x7f92f40047c0 2023-10-30T12:00:53.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.976+0000 7f92faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:52.976+0000 7f92faffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:53.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.155+0000 7f931097d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:53.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.155+0000 7f931097d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:53.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.155+0000 7f931097d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.155+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f930c04e160 con 0x7f92f40047c0 2023-10-30T12:00:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.156+0000 7f92faffd640 1 -- 172.21.15.50:0/2785437409 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12167 (secure 0 0 0) 0x7f92fc022e80 con 0x7f92f40047c0 2023-10-30T12:00:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.156+0000 7f92faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:53.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.156+0000 7f92faffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:53.159 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:53.160 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":15,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","created":"2023-10-30T12:00:40.280079+0000","modified":"2023-10-30T12:00:52.470683+0000","last_up_change":"2023-10-30T12:00:48.401197+0000","last_in_change":"2023-10-30T12:00:41.981126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:00:48.447131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:00:50.466441+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dba8cbf1-bc2e-4420-957f-11ec607e814e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6817","nonce":47190190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6819","nonce":47190190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6823","nonce":47190190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6821","nonce":47190190}]},"public_addr":"172.21.15.50:6817/47190190","cluster_addr":"172.21.15.50:6819/47190190","heartbeat_back_addr":"172.21.15.50:6823/47190190","heartbeat_front_addr":"172.21.15.50:6821/47190190","state":["exists","up"]},{"osd":1,"uuid":"c13ebbef-aa1a-4da7-b5f6-26b02e43955c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6809","nonce":659398918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6811","nonce":659398918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6815","nonce":659398918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6813","nonce":659398918}]},"public_addr":"172.21.15.50:6809/659398918","cluster_addr":"172.21.15.50:6811/659398918","heartbeat_back_addr":"172.21.15.50:6815/659398918","heartbeat_front_addr":"172.21.15.50:6813/659398918","state":["exists","up"]},{"osd":2,"uuid":"23116074-425b-4664-a1be-0e40bba62fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6801","nonce":3518075216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6803","nonce":3518075216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6807","nonce":3518075216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6805","nonce":3518075216}]},"public_addr":"172.21.15.50:6801/3518075216","cluster_addr":"172.21.15.50:6803/3518075216","heartbeat_back_addr":"172.21.15.50:6807/3518075216","heartbeat_front_addr":"172.21.15.50:6805/3518075216","state":["exists","up"]},{"osd":3,"uuid":"515e513c-cb67-4495-ba25-68cb35215c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6810","nonce":1009584072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6814","nonce":1009584072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6822","nonce":1009584072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6818","nonce":1009584072}]},"public_addr":"172.21.15.118:6810/1009584072","cluster_addr":"172.21.15.118:6814/1009584072","heartbeat_back_addr":"172.21.15.118:6822/1009584072","heartbeat_front_addr":"172.21.15.118:6818/1009584072","state":["exists","up"]},{"osd":4,"uuid":"fa9df8af-676c-49c6-9888-0e330fbbc4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6811","nonce":4272127357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6815","nonce":4272127357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6823","nonce":4272127357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6819","nonce":4272127357}]},"public_addr":"172.21.15.118:6811/4272127357","cluster_addr":"172.21.15.118:6815/4272127357","heartbeat_back_addr":"172.21.15.118:6823/4272127357","heartbeat_front_addr":"172.21.15.118:6819/4272127357","state":["exists","up"]},{"osd":5,"uuid":"79b07bf0-2944-4413-b6e1-23ecbc533add","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6801","nonce":1399177807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6803","nonce":1399177807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6807","nonce":1399177807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6805","nonce":1399177807}]},"public_addr":"172.21.15.118:6801/1399177807","cluster_addr":"172.21.15.118:6803/1399177807","heartbeat_back_addr":"172.21.15.118:6807/1399177807","heartbeat_front_addr":"172.21.15.118:6805/1399177807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.083230+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.120938+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.049176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.192079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:00:53.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 10 librados: watch_flush enter 2023-10-30T12:00:53.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 10 librados: watch_flush exit 2023-10-30T12:00:53.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 20 client.4202.objecter shutdown clearing up homeless session... 2023-10-30T12:00:53.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 10 client.4202.objecter successfully canceled tick 2023-10-30T12:00:53.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92ec039320 msgr2=0x7f92ec03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92ec039320 0x7f92ec03b7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9300034000 tx=0x7f9300039040 comp rx=0 tx=0).stop 2023-10-30T12:00:53.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 10 monclient: shutdown 2023-10-30T12:00:53.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 msgr2=0x7f930c1ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f930c1ca610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f92fc007ce0 tx=0x7f92fc0021d0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 shutdown_connections 2023-10-30T12:00:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92ec039320 0x7f92ec03b7e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f930c1cafd0 0x7f930c09cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f930c1cab50 0x7f930c09c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 --2- 172.21.15.50:0/2785437409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92f40047c0 0x7f930c1ca610 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 >> 172.21.15.50:0/2785437409 conn(0x7f930c0a81d0 msgr2=0x7f930c1339f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 shutdown_connections 2023-10-30T12:00:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 -- 172.21.15.50:0/2785437409 wait complete. 2023-10-30T12:00:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.158+0000 7f931097d640 1 librados: shutdown 2023-10-30T12:00:53.171 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T12:00:53.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:00:53.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.274+0000 7fe08e36c640 10 monclient: get_monmap_and_config 2023-10-30T12:00:53.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.274+0000 7fe08e36c640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.274+0000 7fe08e36c640 10 monclient: monmap: 2023-10-30T12:00:53.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.276 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.277 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:53.275257+0000 2023-10-30T12:00:53.277 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:53.275257+0000 2023-10-30T12:00:53.277 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.277 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.277 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:53.277 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:53.278 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:53.278 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:53.278 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:53.278 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.274+0000 7fe08e36c640 1 Processor -- start 2023-10-30T12:00:53.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.274+0000 7fe08e36c640 1 -- start start 2023-10-30T12:00:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.274+0000 7fe08e36c640 10 monclient: init 2023-10-30T12:00:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:00:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0880a6070 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe0880a57e0 0x7fe08813a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0880a57e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe08813adf0 0x7fe08813d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient(hunting): picked mon.noname-a con 0x7fe08813adf0 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08809a270 con 0x7fe0880a6070 2023-10-30T12:00:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe088099f90 con 0x7fe08813adf0 2023-10-30T12:00:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08809a550 con 0x7fe0880a57e0 2023-10-30T12:00:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08e36c640 10 monclient(hunting): authenticate will time out at 898.458440s 2023-10-30T12:00:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe08813adf0 0x7fe08813d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe08813adf0 0x7fe08813d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37978/0 (socket says 172.21.15.50:37978) 2023-10-30T12:00:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 -- 172.21.15.50:0/4064869041 learned_addr learned my addr 172.21.15.50:0/4064869041 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): get_auth_request con 0x7fe08813adf0 auth_method 0 2023-10-30T12:00:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe0877fe640 1 --2- 172.21.15.50:0/4064869041 >> v2:172.21.15.118:3300/0 conn(0x7fe0880a57e0 0x7fe08813a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe0877fe640 10 monclient(hunting): get_auth_request con 0x7fe0880a57e0 auth_method 0 2023-10-30T12:00:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe0877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe0877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe0877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2023-10-30T12:00:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 -- 172.21.15.50:0/4064869041 >> v2:172.21.15.118:3300/0 conn(0x7fe0880a57e0 msgr2=0x7fe08813a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 --2- 172.21.15.50:0/4064869041 >> v2:172.21.15.118:3300/0 conn(0x7fe0880a57e0 0x7fe08813a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 -- 172.21.15.50:0/4064869041 >> v1:172.21.15.118:6789/0 conn(0x7fe0880a6070 legacy=0x7fe0880a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 monclient: found mon.noname-a 2023-10-30T12:00:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.275+0000 7fe08c909640 1 -- 172.21.15.50:0/4064869041 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08809a830 con 0x7fe08813adf0 2023-10-30T12:00:53.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe08c909640 1 --2- 172.21.15.50:0/4064869041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe08813adf0 0x7fe08813d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe0880a20a0 tx=0x7fe08002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8453632aec08df3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 1 -- 172.21.15.50:0/4064869041 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08002c930 con 0x7fe08813adf0 2023-10-30T12:00:53.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:53.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: dump: 2023-10-30T12:00:53.294 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.294 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.294 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.296 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe08e36c640 5 monclient: authenticate success, global_id 4236 2023-10-30T12:00:53.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe08e36c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:53.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 1 -- 172.21.15.50:0/4064869041 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08002ca90 con 0x7fe08813adf0 2023-10-30T12:00:53.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe08e36c640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe08e36c640 4 set_mon_vals no callback set 2023-10-30T12:00:53.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 1 -- 172.21.15.50:0/4064869041 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08002ce20 con 0x7fe08813adf0 2023-10-30T12:00:53.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:53.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.276+0000 7fe086ffd640 10 monclient: dump: 2023-10-30T12:00:53.298 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.298 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.299 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.299 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.299 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.299 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.299 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.300 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.300 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.300 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 10 monclient: shutdown 2023-10-30T12:00:53.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 -- 172.21.15.50:0/4064869041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe08813adf0 msgr2=0x7fe08813d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 --2- 172.21.15.50:0/4064869041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe08813adf0 0x7fe08813d1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe0880a20a0 tx=0x7fe08002bd90 comp rx=0 tx=0).stop 2023-10-30T12:00:53.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 -- 172.21.15.50:0/4064869041 shutdown_connections 2023-10-30T12:00:53.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 --2- 172.21.15.50:0/4064869041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe08813adf0 0x7fe08813d1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 --2- 172.21.15.50:0/4064869041 >> v2:172.21.15.118:3300/0 conn(0x7fe0880a57e0 0x7fe08813a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 -- 172.21.15.50:0/4064869041 >> 172.21.15.50:0/4064869041 conn(0x7fe0880a81d0 msgr2=0x7fe0880a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 -- 172.21.15.50:0/4064869041 shutdown_connections 2023-10-30T12:00:53.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 1 -- 172.21.15.50:0/4064869041 wait complete. 2023-10-30T12:00:53.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.277+0000 7fe08e36c640 10 monclient: monmap: 2023-10-30T12:00:53.302 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.302 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.303 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.303 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.303 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.303 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.303 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.303 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.304 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.304 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 librados: starting msgr at 2023-10-30T12:00:53.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 librados: starting objecter 2023-10-30T12:00:53.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 Processor -- start 2023-10-30T12:00:53.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 -- start start 2023-10-30T12:00:53.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 librados: setting wanted keys 2023-10-30T12:00:53.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 librados: calling monclient init 2023-10-30T12:00:53.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 10 monclient: init 2023-10-30T12:00:53.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:53.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0880a57e0 0x7fe0881ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0880a57e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0880a6070 0x7fe088121a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.278+0000 7fe08e36c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0880a6070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe08813adf0 0x7fe088121f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient(hunting): picked mon.noname-c con 0x7fe08813adf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881251d0 con 0x7fe0880a57e0 2023-10-30T12:00:53.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08813a5b0 con 0x7fe0880a6070 2023-10-30T12:00:53.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0880a1690 con 0x7fe08813adf0 2023-10-30T12:00:53.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient(hunting): authenticate will time out at 898.462040s 2023-10-30T12:00:53.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0880a57e0 0x7fe0881ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe08813adf0 0x7fe088121f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0880a57e0 0x7fe0881ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37994/0 (socket says 172.21.15.50:37994) 2023-10-30T12:00:53.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 -- 172.21.15.50:0/3923337130 learned_addr learned my addr 172.21.15.50:0/3923337130 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe08813adf0 0x7fe088121f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35584/0 (socket says 172.21.15.50:35584) 2023-10-30T12:00:53.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): get_auth_request con 0x7fe0880a57e0 auth_method 0 2023-10-30T12:00:53.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 10 monclient(hunting): get_auth_request con 0x7fe08813adf0 auth_method 0 2023-10-30T12:00:53.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0880a6070 0x7fe088121a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08c909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 10 monclient(hunting): get_auth_request con 0x7fe0880a6070 auth_method 0 2023-10-30T12:00:53.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2023-10-30T12:00:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 -- 172.21.15.50:0/3923337130 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe08813adf0 msgr2=0x7fe088121f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe08813adf0 0x7fe088121f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 -- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0880a6070 msgr2=0x7fe088121a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0880a6070 0x7fe088121a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 monclient: found mon.noname-a 2023-10-30T12:00:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 -- 172.21.15.50:0/3923337130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe080009b30 con 0x7fe0880a57e0 2023-10-30T12:00:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe087fff640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0880a57e0 0x7fe0881ca690 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe07000c430 tx=0x7fe07000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 client.?.objecter ms_handle_connect 0x7fe0880a57e0 2023-10-30T12:00:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 1 -- 172.21.15.50:0/3923337130 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070011790 con 0x7fe0880a57e0 2023-10-30T12:00:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: dump: 2023-10-30T12:00:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.318 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 1 -- 172.21.15.50:0/3923337130 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0700118f0 con 0x7fe0880a57e0 2023-10-30T12:00:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 5 monclient: authenticate success, global_id 4239 2023-10-30T12:00:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient: _renew_subs 2023-10-30T12:00:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe088125a70 con 0x7fe0880a57e0 2023-10-30T12:00:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 1 -- 172.21.15.50:0/3923337130 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070011ca0 con 0x7fe0880a57e0 2023-10-30T12:00:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe084ff9640 10 monclient: dump: 2023-10-30T12:00:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.322 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe0867fc640 4 set_mon_vals no callback set 2023-10-30T12:00:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient: _renew_subs 2023-10-30T12:00:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0881396f0 con 0x7fe0880a57e0 2023-10-30T12:00:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.279+0000 7fe08e36c640 1 librados: init done 2023-10-30T12:00:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe084ff9640 1 -- 172.21.15.50:0/3923337130 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe07001c460 con 0x7fe0880a57e0 2023-10-30T12:00:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe08e36c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe08e36c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe08e36c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08804e160 con 0x7fe0880a57e0 2023-10-30T12:00:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe084ff9640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0680392f0 0x7fe06803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe084ff9640 1 -- 172.21.15.50:0/3923337130 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fe07004a480 con 0x7fe0880a57e0 2023-10-30T12:00:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe084ff9640 10 client.4239.objecter ms_dispatch 0x7fe088000b90 osd_map(15..15 src has 1..15) v4 2023-10-30T12:00:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe084ff9640 3 client.4239.objecter handle_osd_map got epochs [15,15] > 0 2023-10-30T12:00:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe084ff9640 3 client.4239.objecter handle_osd_map decoding full epoch 15 2023-10-30T12:00:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.280+0000 7fe084ff9640 20 client.4239.objecter dump_active .. 0 homeless 2023-10-30T12:00:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.281+0000 7fe0877fe640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0680392f0 0x7fe06803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.281+0000 7fe0877fe640 10 monclient: get_auth_request con 0x7fe0680392f0 auth_method 0 2023-10-30T12:00:53.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.281+0000 7fe084ff9640 10 client.4239.objecter ms_handle_connect 0x7fe0680392f0 2023-10-30T12:00:53.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.281+0000 7fe0877fe640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0680392f0 0x7fe06803b7b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe0880a18e0 tx=0x7fe078008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.285+0000 7fe084ff9640 1 -- 172.21.15.50:0/3923337130 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe0700106b0 con 0x7fe0880a57e0 2023-10-30T12:00:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.285+0000 7fe084ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.285+0000 7fe084ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:53.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.464+0000 7fe08e36c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:53.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.464+0000 7fe08e36c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:53.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.464+0000 7fe08e36c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.464+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe088074580 con 0x7fe0880a57e0 2023-10-30T12:00:53.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.465+0000 7fe084ff9640 1 -- 172.21.15.50:0/3923337130 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12167 (secure 0 0 0) 0x7fe07001a360 con 0x7fe0880a57e0 2023-10-30T12:00:53.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.465+0000 7fe084ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:53.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.465+0000 7fe084ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:53.469 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":15,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","created":"2023-10-30T12:00:40.280079+0000","modified":"2023-10-30T12:00:52.470683+0000","last_up_change":"2023-10-30T12:00:48.401197+0000","last_in_change":"2023-10-30T12:00:41.981126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:00:48.447131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:00:50.466441+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dba8cbf1-bc2e-4420-957f-11ec607e814e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6817","nonce":47190190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6819","nonce":47190190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6823","nonce":47190190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6821","nonce":47190190}]},"public_addr":"172.21.15.50:6817/47190190","cluster_addr":"172.21.15.50:6819/47190190","heartbeat_back_addr":"172.21.15.50:6823/47190190","heartbeat_front_addr":"172.21.15.50:6821/47190190","state":["exists","up"]},{"osd":1,"uuid":"c13ebbef-aa1a-4da7-b5f6-26b02e43955c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6809","nonce":659398918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6811","nonce":659398918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6815","nonce":659398918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6813","nonce":659398918}]},"public_addr":"172.21.15.50:6809/659398918","cluster_addr":"172.21.15.50:6811/659398918","heartbeat_back_addr":"172.21.15.50:6815/659398918","heartbeat_front_addr":"172.21.15.50:6813/659398918","state":["exists","up"]},{"osd":2,"uuid":"23116074-425b-4664-a1be-0e40bba62fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6801","nonce":3518075216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6803","nonce":3518075216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6807","nonce":3518075216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6805","nonce":3518075216}]},"public_addr":"172.21.15.50:6801/3518075216","cluster_addr":"172.21.15.50:6803/3518075216","heartbeat_back_addr":"172.21.15.50:6807/3518075216","heartbeat_front_addr":"172.21.15.50:6805/3518075216","state":["exists","up"]},{"osd":3,"uuid":"515e513c-cb67-4495-ba25-68cb35215c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6810","nonce":1009584072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6814","nonce":1009584072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6822","nonce":1009584072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6818","nonce":1009584072}]},"public_addr":"172.21.15.118:6810/1009584072","cluster_addr":"172.21.15.118:6814/1009584072","heartbeat_back_addr":"172.21.15.118:6822/1009584072","heartbeat_front_addr":"172.21.15.118:6818/1009584072","state":["exists","up"]},{"osd":4,"uuid":"fa9df8af-676c-49c6-9888-0e330fbbc4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6811","nonce":4272127357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6815","nonce":4272127357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6823","nonce":4272127357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6819","nonce":4272127357}]},"public_addr":"172.21.15.118:6811/4272127357","cluster_addr":"172.21.15.118:6815/4272127357","heartbeat_back_addr":"172.21.15.118:6823/4272127357","heartbeat_front_addr":"172.21.15.118:6819/4272127357","state":["exists","up"]},{"osd":5,"uuid":"79b07bf0-2944-4413-b6e1-23ecbc533add","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6801","nonce":1399177807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6803","nonce":1399177807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6807","nonce":1399177807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6805","nonce":1399177807}]},"public_addr":"172.21.15.118:6801/1399177807","cluster_addr":"172.21.15.118:6803/1399177807","heartbeat_back_addr":"172.21.15.118:6807/1399177807","heartbeat_front_addr":"172.21.15.118:6805/1399177807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.083230+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.120938+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.049176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.192079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:00:53.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 10 librados: watch_flush enter 2023-10-30T12:00:53.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 10 librados: watch_flush exit 2023-10-30T12:00:53.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 20 client.4239.objecter shutdown clearing up homeless session... 2023-10-30T12:00:53.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 10 client.4239.objecter successfully canceled tick 2023-10-30T12:00:53.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0680392f0 msgr2=0x7fe06803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0680392f0 0x7fe06803b7b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe0880a18e0 tx=0x7fe078008040 comp rx=0 tx=0).stop 2023-10-30T12:00:53.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 10 monclient: shutdown 2023-10-30T12:00:53.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0880a57e0 msgr2=0x7fe0881ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0880a57e0 0x7fe0881ca690 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe07000c430 tx=0x7fe07000c8f0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 shutdown_connections 2023-10-30T12:00:53.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0680392f0 0x7fe06803b7b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe08813adf0 0x7fe088121f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe0880a6070 0x7fe088121a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 --2- 172.21.15.50:0/3923337130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0880a57e0 0x7fe0881ca690 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 >> 172.21.15.50:0/3923337130 conn(0x7fe0880a81d0 msgr2=0x7fe08809e280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 shutdown_connections 2023-10-30T12:00:53.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 -- 172.21.15.50:0/3923337130 wait complete. 2023-10-30T12:00:53.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.467+0000 7fe08e36c640 1 librados: shutdown 2023-10-30T12:00:53.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T12:00:53.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T12:00:53.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T12:00:53.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T12:00:53.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T12:00:53.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T12:00:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.604+0000 7f758238a640 10 monclient: get_monmap_and_config 2023-10-30T12:00:53.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.604+0000 7f43750bb640 10 monclient: get_monmap_and_config 2023-10-30T12:00:53.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.604+0000 7f43750bb640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.604+0000 7f43750bb640 10 monclient: monmap: 2023-10-30T12:00:53.607 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:53.605758+0000 2023-10-30T12:00:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:53.605758+0000 2023-10-30T12:00:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:53.609 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:53.609 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:53.609 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:53.609 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:53.609 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.604+0000 7f758238a640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.604+0000 7f758238a640 10 monclient: monmap: 2023-10-30T12:00:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:53.605758+0000 2023-10-30T12:00:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:53.605758+0000 2023-10-30T12:00:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:53.612 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:53.612 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:53.612 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:53.612 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:53.612 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f43750bb640 1 Processor -- start 2023-10-30T12:00:53.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f43750bb640 1 -- start start 2023-10-30T12:00:53.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f43750bb640 10 monclient: init 2023-10-30T12:00:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f758238a640 1 Processor -- start 2023-10-30T12:00:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f758238a640 1 -- start start 2023-10-30T12:00:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f758238a640 10 monclient: init 2023-10-30T12:00:53.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f43750bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f43750bb640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-10-30T12:00:53.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.605+0000 7f43750bb640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4370071c00 0x7f437009a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 10 monclient(hunting): picked mon.noname-a con 0x7f4370071c00 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:53.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-10-30T12:00:53.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f757c0a8010 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:53.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f757c099a40 0x7f757c13aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4370097fe0 0x7f437009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient(hunting): picked mon.noname-a con 0x7f757c099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:53.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 10 monclient(hunting): picked mon.noname-c con 0x7f4370097fe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:53.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f757c132da0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:53.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0a7190 con 0x7f757c0a8010 2023-10-30T12:00:53.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0a83f0 con 0x7f757c132da0 2023-10-30T12:00:53.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c0a7600 con 0x7f757c099a40 2023-10-30T12:00:53.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f758238a640 10 monclient(hunting): authenticate will time out at 898.789419s 2023-10-30T12:00:53.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f437009afa0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:53.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4370097600 con 0x7f437009afa0 2023-10-30T12:00:53.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43700978e0 con 0x7f4370071c00 2023-10-30T12:00:53.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437009bf50 con 0x7f4370097fe0 2023-10-30T12:00:53.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f757c099a40 0x7f757c13aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f757c099a40 0x7f757c13aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37998/0 (socket says 172.21.15.50:37998) 2023-10-30T12:00:53.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f43750bb640 10 monclient(hunting): authenticate will time out at 898.789646s 2023-10-30T12:00:53.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 1 -- 172.21.15.50:0/2820957418 learned_addr learned my addr 172.21.15.50:0/2820957418 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 10 monclient(hunting): get_auth_request con 0x7f757c099a40 auth_method 0 2023-10-30T12:00:53.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757bfff640 1 --1- 172.21.15.50:0/2820957418 >> v1:172.21.15.50:6789/0 conn(0x7f757c0a8010 0x7f757c132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60790/0 (socket says 172.21.15.50:60790) 2023-10-30T12:00:53.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436e59c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4370097fe0 0x7f437009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436ed9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4370071c00 0x7f437009a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f757b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436e59c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4370097fe0 0x7f437009aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:59062/0 (socket says 172.21.15.50:59062) 2023-10-30T12:00:53.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436ed9d640 10 monclient(hunting): get_auth_request con 0x7f4370071c00 auth_method 0 2023-10-30T12:00:53.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436e59c640 1 -- 172.21.15.50:0/316752631 learned_addr learned my addr 172.21.15.50:0/316752631 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.606+0000 7f436e59c640 10 monclient(hunting): get_auth_request con 0x7f4370097fe0 auth_method 0 2023-10-30T12:00:53.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2023-10-30T12:00:53.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 1 -- 172.21.15.50:0/2820957418 >> v1:172.21.15.118:6789/0 conn(0x7f757c132da0 legacy=0x7f757c13cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 1 -- 172.21.15.50:0/2820957418 >> v1:172.21.15.50:6789/0 conn(0x7f757c0a8010 legacy=0x7f757c132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:00:53.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f757b7fe640 1 -- 172.21.15.50:0/2820957418 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c09a440 con 0x7f757c099a40 2023-10-30T12:00:53.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2023-10-30T12:00:53.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 1 -- 172.21.15.50:0/316752631 >> v2:172.21.15.118:3300/0 conn(0x7f4370097fe0 msgr2=0x7f437009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 1 --2- 172.21.15.50:0/316752631 >> v2:172.21.15.118:3300/0 conn(0x7f4370097fe0 0x7f437009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 1 -- 172.21.15.50:0/316752631 >> v1:172.21.15.50:6789/0 conn(0x7f437009afa0 legacy=0x7f4370138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 1 monclient: found mon.noname-a 2023-10-30T12:00:53.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436ed9d640 1 -- 172.21.15.50:0/316752631 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437009d9b0 con 0x7f4370071c00 2023-10-30T12:00:53.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.607+0000 7f436e59c640 1 --2- 172.21.15.50:0/316752631 >> v2:172.21.15.118:3300/0 conn(0x7f4370097fe0 0x7f437009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:53.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757b7fe640 1 --2- 172.21.15.50:0/2820957418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c099a40 0x7f757c13aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7570009870 tx=0x7f757002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34875eac74bc9f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436ed9d640 1 --2- 172.21.15.50:0/316752631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4370071c00 0x7f437009a150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f437006bb30 tx=0x7f436002fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54be6aecefc38d08 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 1 -- 172.21.15.50:0/2820957418 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75700317e0 con 0x7f757c099a40 2023-10-30T12:00:53.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 1 -- 172.21.15.50:0/316752631 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360030980 con 0x7f4370071c00 2023-10-30T12:00:53.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: dump: 2023-10-30T12:00:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.639 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 1 -- 172.21.15.50:0/2820957418 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7570031940 con 0x7f757c099a40 2023-10-30T12:00:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 1 -- 172.21.15.50:0/2820957418 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570031cd0 con 0x7f757c099a40 2023-10-30T12:00:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:53.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f757affd640 10 monclient: dump: 2023-10-30T12:00:53.641 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.641 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.641 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.641 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.642 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.642 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.642 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.642 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.643 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f758238a640 5 monclient: authenticate success, global_id 4242 2023-10-30T12:00:53.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f758238a640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f758238a640 4 set_mon_vals no callback set 2023-10-30T12:00:53.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: dump: 2023-10-30T12:00:53.645 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.645 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.645 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.645 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.646 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.646 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.646 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.646 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.646 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.647 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 1 -- 172.21.15.50:0/316752631 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4360030ae0 con 0x7f4370071c00 2023-10-30T12:00:53.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f43750bb640 5 monclient: authenticate success, global_id 4245 2023-10-30T12:00:53.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f43750bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:53.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f43750bb640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f43750bb640 4 set_mon_vals no callback set 2023-10-30T12:00:53.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 1 -- 172.21.15.50:0/316752631 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360030e70 con 0x7f4370071c00 2023-10-30T12:00:53.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:53.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.608+0000 7f436dd9b640 10 monclient: dump: 2023-10-30T12:00:53.649 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.649 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.650 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.650 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 10 monclient: shutdown 2023-10-30T12:00:53.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 1 -- 172.21.15.50:0/2820957418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c099a40 msgr2=0x7f757c13aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 1 --2- 172.21.15.50:0/2820957418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c099a40 0x7f757c13aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7570009870 tx=0x7f757002b880 comp rx=0 tx=0).stop 2023-10-30T12:00:53.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 1 -- 172.21.15.50:0/2820957418 shutdown_connections 2023-10-30T12:00:53.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 1 --2- 172.21.15.50:0/2820957418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c099a40 0x7f757c13aa10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 1 -- 172.21.15.50:0/2820957418 >> 172.21.15.50:0/2820957418 conn(0x7f757c0a4a10 msgr2=0x7f757c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 10 monclient: shutdown 2023-10-30T12:00:53.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 1 -- 172.21.15.50:0/2820957418 shutdown_connections 2023-10-30T12:00:53.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f758238a640 1 -- 172.21.15.50:0/2820957418 wait complete. 2023-10-30T12:00:53.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 -- 172.21.15.50:0/316752631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4370071c00 msgr2=0x7f437009a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 --2- 172.21.15.50:0/316752631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4370071c00 0x7f437009a150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f437006bb30 tx=0x7f436002fbd0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 -- 172.21.15.50:0/316752631 shutdown_connections 2023-10-30T12:00:53.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 --2- 172.21.15.50:0/316752631 >> v2:172.21.15.118:3300/0 conn(0x7f4370097fe0 0x7f437009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 --2- 172.21.15.50:0/316752631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4370071c00 0x7f437009a150 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 -- 172.21.15.50:0/316752631 >> 172.21.15.50:0/316752631 conn(0x7f43700a3980 msgr2=0x7f43700a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 -- 172.21.15.50:0/316752631 shutdown_connections 2023-10-30T12:00:53.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.609+0000 7f43750bb640 1 -- 172.21.15.50:0/316752631 wait complete. 2023-10-30T12:00:53.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.610+0000 7f758238a640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.610+0000 7f758238a640 10 monclient: monmap: 2023-10-30T12:00:53.660 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.660 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.660 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.661 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.661 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.661 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.661 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.661 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.661 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.662 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.610+0000 7f43750bb640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.610+0000 7f43750bb640 10 monclient: monmap: 2023-10-30T12:00:53.663 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.663 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.663 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.663 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.664 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.664 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.664 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.664 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.664 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.664 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 librados: starting msgr at 2023-10-30T12:00:53.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 librados: starting objecter 2023-10-30T12:00:53.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f758238a640 1 librados: starting msgr at 2023-10-30T12:00:53.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f758238a640 1 librados: starting objecter 2023-10-30T12:00:53.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f758238a640 1 Processor -- start 2023-10-30T12:00:53.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 Processor -- start 2023-10-30T12:00:53.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 -- start start 2023-10-30T12:00:53.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 librados: setting wanted keys 2023-10-30T12:00:53.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 librados: calling monclient init 2023-10-30T12:00:53.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient: init 2023-10-30T12:00:53.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f758238a640 1 -- start start 2023-10-30T12:00:53.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f758238a640 1 librados: setting wanted keys 2023-10-30T12:00:53.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f758238a640 1 librados: calling monclient init 2023-10-30T12:00:53.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f758238a640 10 monclient: init 2023-10-30T12:00:53.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:53.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4370071c00 0x7f43701ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): picked mon.noname-b con 0x7f4370071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:00:53.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c0a8010 0x7f757c136790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4370097fe0 0x7f43701ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): picked mon.noname-c con 0x7f4370097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f437009afa0 0x7f4370070560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): picked mon.noname-a con 0x7f437009afa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437008d2c0 con 0x7f437009afa0 2023-10-30T12:00:53.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437009cbc0 con 0x7f4370071c00 2023-10-30T12:00:53.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437008d880 con 0x7f4370097fe0 2023-10-30T12:00:53.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.611+0000 7f43750bb640 10 monclient(hunting): authenticate will time out at 898.794937s 2023-10-30T12:00:53.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4370071c00 0x7f43701ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f437009afa0 0x7f4370070560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4370071c00 0x7f43701ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:59066/0 (socket says 172.21.15.50:59066) 2023-10-30T12:00:53.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 1 -- 172.21.15.50:0/4252039707 learned_addr learned my addr 172.21.15.50:0/4252039707 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 10 monclient(hunting): get_auth_request con 0x7f4370071c00 auth_method 0 2023-10-30T12:00:53.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4370097fe0 0x7f43701ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 10 monclient(hunting): get_auth_request con 0x7f437009afa0 auth_method 0 2023-10-30T12:00:53.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 10 monclient(hunting): get_auth_request con 0x7f4370097fe0 auth_method 0 2023-10-30T12:00:53.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 10 monclient(hunting): picked mon.noname-a con 0x7f757c0a8010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f757bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c0a8010 0x7f757c136790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f757bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c0a8010 0x7f757c136790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38020/0 (socket says 172.21.15.50:38020) 2023-10-30T12:00:53.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f757bfff640 1 -- 172.21.15.50:0/365728940 learned_addr learned my addr 172.21.15.50:0/365728940 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f757c132da0 0x7f757c136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 10 monclient(hunting): picked mon.noname-c con 0x7f757c132da0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f757c137210 0x7f757c1310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 10 monclient(hunting): picked mon.noname-b con 0x7f757c137210 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f757b7fe640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f757c132da0 0x7f757c136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f758238a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f758238a640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c1397a0 con 0x7f757c0a8010 2023-10-30T12:00:53.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f758238a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f758238a640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c138920 con 0x7f757c137210 2023-10-30T12:00:53.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f757c137210 0x7f757c1310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.612+0000 7f436e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2023-10-30T12:00:53.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f758238a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 1 -- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4370097fe0 msgr2=0x7f43701ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f758238a640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c13db20 con 0x7f757c132da0 2023-10-30T12:00:53.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f758238a640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f758238a640 10 monclient(hunting): authenticate will time out at 898.796230s 2023-10-30T12:00:53.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 10 monclient(hunting): get_auth_request con 0x7f757c0a8010 auth_method 0 2023-10-30T12:00:53.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): get_auth_request con 0x7f757c132da0 auth_method 0 2023-10-30T12:00:53.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 10 monclient(hunting): get_auth_request con 0x7f757c137210 auth_method 0 2023-10-30T12:00:53.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2023-10-30T12:00:53.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 1 -- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f757c137210 msgr2=0x7f757c1310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f757c137210 0x7f757c1310e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 1 -- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c0a8010 msgr2=0x7f757c136790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c0a8010 0x7f757c136790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 1 monclient: found mon.noname-c 2023-10-30T12:00:53.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757b7fe640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75700094d0 con 0x7f757c132da0 2023-10-30T12:00:53.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f7580927640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f757c137210 0x7f757c1310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f757bfff640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f757c0a8010 0x7f757c136790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4370097fe0 0x7f43701ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 1 -- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f437009afa0 msgr2=0x7f4370070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f437009afa0 0x7f4370070560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 1 monclient: found mon.noname-b 2023-10-30T12:00:53.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f757b7fe640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f757c132da0 0x7f757c136cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7570033000 tx=0x7f75700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 client.?.objecter ms_handle_connect 0x7f757c132da0 2023-10-30T12:00:53.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:53.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436ed9d640 1 -- 172.21.15.50:0/4252039707 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600093a0 con 0x7f4370071c00 2023-10-30T12:00:53.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.613+0000 7f436f59e640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f437009afa0 0x7f4370070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f436e59c640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4370097fe0 0x7f43701ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f436ed9d640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4370071c00 0x7f43701ce980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4360009880 tx=0x7f43600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 client.?.objecter ms_handle_connect 0x7f4370071c00 2023-10-30T12:00:53.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:53.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75700329c0 con 0x7f757c132da0 2023-10-30T12:00:53.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360013560 con 0x7f4370071c00 2023-10-30T12:00:53.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:53.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: dump: 2023-10-30T12:00:53.701 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.702 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.702 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.702 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.702 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.703 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.703 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43600136e0 con 0x7f4370071c00 2023-10-30T12:00:53.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436000f6b0 con 0x7f4370071c00 2023-10-30T12:00:53.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:53.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43577fe640 10 monclient: dump: 2023-10-30T12:00:53.705 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.705 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.706 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.706 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.706 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.706 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.706 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 5 monclient: authenticate success, global_id 4273 2023-10-30T12:00:53.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 10 monclient: _renew_subs 2023-10-30T12:00:53.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437008be70 con 0x7f4370071c00 2023-10-30T12:00:53.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 10 monclient: _renew_subs 2023-10-30T12:00:53.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437008b4f0 con 0x7f4370071c00 2023-10-30T12:00:53.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f43750bb640 1 librados: init done 2023-10-30T12:00:53.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:53.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: dump: 2023-10-30T12:00:53.710 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.710 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.710 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.711 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.711 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.711 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.711 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.711 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.712 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.712 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7570032b40 con 0x7f757c132da0 2023-10-30T12:00:53.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 5 monclient: authenticate success, global_id 4211 2023-10-30T12:00:53.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 10 monclient: _renew_subs 2023-10-30T12:00:53.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c13aa30 con 0x7f757c132da0 2023-10-30T12:00:53.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f436d59a640 4 set_mon_vals no callback set 2023-10-30T12:00:53.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75700453f0 con 0x7f757c132da0 2023-10-30T12:00:53.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:53.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f7578ff9640 10 monclient: dump: 2023-10-30T12:00:53.716 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.716 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.716 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.716 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.717 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.717 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.717 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.717 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.717 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.717 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 10 monclient: _renew_subs 2023-10-30T12:00:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c0a18d0 con 0x7f757c132da0 2023-10-30T12:00:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f758238a640 1 librados: init done 2023-10-30T12:00:53.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f43750bb640 10 client.4273.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f43750bb640 15 client.4273.objecter _session_command_op_assign -1 1 2023-10-30T12:00:53.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f43750bb640 20 client.4273.objecter _get_session osd=-1 returning homeless 2023-10-30T12:00:53.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f43750bb640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f43750bb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4370071fe0 con 0x7f4370071c00 2023-10-30T12:00:53.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.614+0000 7f757a7fc640 4 set_mon_vals no callback set 2023-10-30T12:00:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f75627fc640 10 client.4211.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f75627fc640 15 client.4211.objecter _session_command_op_assign -1 1 2023-10-30T12:00:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f75627fc640 20 client.4211.objecter _get_session osd=-1 returning homeless 2023-10-30T12:00:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f75627fc640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f75627fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f757c09aa00 con 0x7f757c132da0 2023-10-30T12:00:53.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.616+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f757004d3f0 con 0x7f757c132da0 2023-10-30T12:00:53.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f436000f810 con 0x7f4370071c00 2023-10-30T12:00:53.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7568039320 0x7f756803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f435c0392f0 0x7f435c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f436006c220 con 0x7f4370071c00 2023-10-30T12:00:53.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f757006c450 con 0x7f757c132da0 2023-10-30T12:00:53.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 10 client.4211.objecter ms_dispatch 0x7f757c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:53.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 3 client.4211.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:53.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 3 client.4211.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:53.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 10 client.4211.objecter checking command 1 2023-10-30T12:00:53.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f756803f130 0x7f7568041550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 20 client.4211.objecter _get_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 20 client.4211.objecter put_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 15 client.4211.objecter _session_command_op_remove -1 1 2023-10-30T12:00:53.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 20 client.4211.objecter _get_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 20 client.4211.objecter get_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 15 client.4211.objecter _session_command_op_assign 2 1 2023-10-30T12:00:53.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 20 client.4211.objecter put_session s=0x7f756803ea60 osd=2 4 2023-10-30T12:00:53.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 10 client.4211.objecter _send_command 1 2023-10-30T12:00:53.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7568041c00 con 0x7f756803f130 2023-10-30T12:00:53.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 20 client.4211.objecter dump_active .. 0 homeless 2023-10-30T12:00:53.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f757006c870 con 0x7f757c132da0 2023-10-30T12:00:53.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7578ff9640 10 monclient: handle_get_version_reply finishing 1 version 16 2023-10-30T12:00:53.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 10 client.4273.objecter ms_dispatch 0x7f4370000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:53.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 3 client.4273.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:53.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 3 client.4273.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:53.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 10 client.4273.objecter checking command 1 2023-10-30T12:00:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f757bfff640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7568039320 0x7f756803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f435c03f100 0x7f435c041520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 20 client.4273.objecter _get_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 20 client.4273.objecter put_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 15 client.4273.objecter _session_command_op_remove -1 1 2023-10-30T12:00:53.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 20 client.4273.objecter _get_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 20 client.4273.objecter get_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 15 client.4273.objecter _session_command_op_assign 1 1 2023-10-30T12:00:53.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 20 client.4273.objecter put_session s=0x7f435c03ea30 osd=1 4 2023-10-30T12:00:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 10 client.4273.objecter _send_command 1 2023-10-30T12:00:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 --> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435c041bd0 con 0x7f435c03f100 2023-10-30T12:00:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 20 client.4273.objecter dump_active .. 0 homeless 2023-10-30T12:00:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f436000fdf0 con 0x7f4370071c00 2023-10-30T12:00:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f43577fe640 10 monclient: handle_get_version_reply finishing 1 version 16 2023-10-30T12:00:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7580927640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f756803f130 0x7f7568041550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f7580927640 10 monclient: get_auth_request con 0x7f756803f130 auth_method 0 2023-10-30T12:00:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f757bfff640 10 monclient: get_auth_request con 0x7f7568039320 auth_method 0 2023-10-30T12:00:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f436e59c640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f435c0392f0 0x7f435c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f436f59e640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f435c03f100 0x7f435c041520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.617+0000 7f436f59e640 10 monclient: get_auth_request con 0x7f435c03f100 auth_method 0 2023-10-30T12:00:53.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f436e59c640 10 monclient: get_auth_request con 0x7f435c0392f0 auth_method 0 2023-10-30T12:00:53.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f7580927640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f756803f130 0x7f7568041550 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f7578ff9640 10 client.4211.objecter ms_handle_connect 0x7f756803f130 2023-10-30T12:00:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f7578ff9640 10 client.4211.objecter ms_handle_connect 0x7f7568039320 2023-10-30T12:00:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f43577fe640 10 client.4273.objecter ms_handle_connect 0x7f435c03f100 2023-10-30T12:00:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f436f59e640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f435c03f100 0x7f435c041520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f757bfff640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7568039320 0x7f756803b7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f756c000f60 tx=0x7f756c007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f43577fe640 10 client.4273.objecter ms_handle_connect 0x7f435c0392f0 2023-10-30T12:00:53.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.618+0000 7f436e59c640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f435c0392f0 0x7f435c03b7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4370097210 tx=0x7f4364008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== osd.2 v2:172.21.15.50:6800/3518075216 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f7568041c00 con 0x7f756803f130 2023-10-30T12:00:53.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f7578ff9640 10 client.4211.objecter ms_dispatch 0x7f757c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:00:53.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f7578ff9640 10 client.4211.objecter _finish_command 1 = system:0 2023-10-30T12:00:53.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f7578ff9640 20 client.4211.objecter put_session s=0x7f756803ea60 osd=2 4 2023-10-30T12:00:53.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f7578ff9640 15 client.4211.objecter _session_command_op_remove 2 1 2023-10-30T12:00:53.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== osd.1 v2:172.21.15.50:6808/659398918 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f435c041bd0 con 0x7f435c03f100 2023-10-30T12:00:53.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f43577fe640 10 client.4273.objecter ms_dispatch 0x7f4370000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:00:53.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f43577fe640 10 client.4273.objecter _finish_command 1 = system:0 2023-10-30T12:00:53.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f43577fe640 20 client.4273.objecter put_session s=0x7f435c03ea30 osd=1 4 2023-10-30T12:00:53.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.619+0000 7f43577fe640 15 client.4273.objecter _session_command_op_remove 1 1 2023-10-30T12:00:53.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 10 client.4211.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:00:53.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 15 client.4211.objecter _session_command_op_assign -1 2 2023-10-30T12:00:53.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 20 client.4211.objecter _get_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 20 client.4211.objecter put_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 20 client.4211.objecter _get_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 15 client.4211.objecter _session_command_op_remove -1 2 2023-10-30T12:00:53.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 20 client.4211.objecter get_session s=0x7f756803ea60 osd=2 3 2023-10-30T12:00:53.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 15 client.4211.objecter _session_command_op_assign 2 2 2023-10-30T12:00:53.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 20 client.4211.objecter put_session s=0x7f756803ea60 osd=2 4 2023-10-30T12:00:53.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 10 client.4211.objecter _send_command 2 2023-10-30T12:00:53.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 --> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f757c09a720 con 0x7f756803f130 2023-10-30T12:00:53.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f7578ff9640 1 -- 172.21.15.50:0/365728940 <== osd.2 v2:172.21.15.50:6800/3518075216 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f75740045f0 con 0x7f756803f130 2023-10-30T12:00:53.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 10 client.4273.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:00:53.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 15 client.4273.objecter _session_command_op_assign -1 2 2023-10-30T12:00:53.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 20 client.4273.objecter _get_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 20 client.4273.objecter put_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 20 client.4273.objecter _get_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 15 client.4273.objecter _session_command_op_remove -1 2 2023-10-30T12:00:53.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 20 client.4273.objecter get_session s=0x7f435c03ea30 osd=1 3 2023-10-30T12:00:53.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 15 client.4273.objecter _session_command_op_assign 1 2 2023-10-30T12:00:53.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 20 client.4273.objecter put_session s=0x7f435c03ea30 osd=1 4 2023-10-30T12:00:53.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 10 client.4273.objecter _send_command 2 2023-10-30T12:00:53.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 --> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f437004e160 con 0x7f435c03f100 2023-10-30T12:00:53.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f7578ff9640 10 client.4211.objecter ms_dispatch 0x7f757c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:00:53.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f7578ff9640 10 client.4211.objecter _finish_command 2 = system:0 2023-10-30T12:00:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f7578ff9640 20 client.4211.objecter put_session s=0x7f756803ea60 osd=2 4 2023-10-30T12:00:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.633+0000 7f7578ff9640 15 client.4211.objecter _session_command_op_remove 2 2 2023-10-30T12:00:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43577fe640 1 -- 172.21.15.50:0/4252039707 <== osd.1 v2:172.21.15.50:6808/659398918 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f437004e160 con 0x7f435c03f100 2023-10-30T12:00:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43577fe640 10 client.4273.objecter ms_dispatch 0x7f4370000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:00:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43577fe640 10 client.4273.objecter _finish_command 2 = system:0 2023-10-30T12:00:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43577fe640 20 client.4273.objecter put_session s=0x7f435c03ea30 osd=1 4 2023-10-30T12:00:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43577fe640 15 client.4273.objecter _session_command_op_remove 1 2 2023-10-30T12:00:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 10 librados: watch_flush enter 2023-10-30T12:00:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 10 librados: watch_flush exit 2023-10-30T12:00:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 10 client.4211.objecter close_session for osd.2 2023-10-30T12:00:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f756803f130 msgr2=0x7f7568041550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f756803f130 0x7f7568041550 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 20 client.4211.objecter put_session s=0x7f756803ea60 osd=2 1 2023-10-30T12:00:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 20 client.4211.objecter shutdown clearing up homeless session... 2023-10-30T12:00:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 10 client.4211.objecter successfully canceled tick 2023-10-30T12:00:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7568039320 msgr2=0x7f756803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7568039320 0x7f756803b7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f756c000f60 tx=0x7f756c007f00 comp rx=0 tx=0).stop 2023-10-30T12:00:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 10 monclient: shutdown 2023-10-30T12:00:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f757c132da0 msgr2=0x7f757c136cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 --2- 172.21.15.50:0/365728940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f757c132da0 0x7f757c136cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7570033000 tx=0x7f75700046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f7580927640 1 -- 172.21.15.50:0/365728940 reap_dead start 2023-10-30T12:00:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 shutdown_connections 2023-10-30T12:00:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 10 librados: watch_flush enter 2023-10-30T12:00:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 10 librados: watch_flush exit 2023-10-30T12:00:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 10 client.4273.objecter close_session for osd.1 2023-10-30T12:00:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f435c03f100 msgr2=0x7f435c041520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f435c03f100 0x7f435c041520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 >> 172.21.15.50:0/365728940 conn(0x7f757c0a4a10 msgr2=0x7f757c069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 20 client.4273.objecter put_session s=0x7f435c03ea30 osd=1 1 2023-10-30T12:00:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 20 client.4273.objecter shutdown clearing up homeless session... 2023-10-30T12:00:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 10 client.4273.objecter successfully canceled tick 2023-10-30T12:00:53.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f435c0392f0 msgr2=0x7f435c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f435c0392f0 0x7f435c03b7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4370097210 tx=0x7f4364008040 comp rx=0 tx=0).stop 2023-10-30T12:00:53.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 shutdown_connections 2023-10-30T12:00:53.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 10 monclient: shutdown 2023-10-30T12:00:53.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4370071c00 msgr2=0x7f43701ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.634+0000 7f43750bb640 1 --2- 172.21.15.50:0/4252039707 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4370071c00 0x7f43701ce980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4360009880 tx=0x7f43600046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f75627fc640 1 -- 172.21.15.50:0/365728940 wait complete. 2023-10-30T12:00:53.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f436f59e640 1 -- 172.21.15.50:0/4252039707 reap_dead start 2023-10-30T12:00:53.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 shutdown_connections 2023-10-30T12:00:53.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 >> 172.21.15.50:0/4252039707 conn(0x7f43700a3980 msgr2=0x7f43701b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 shutdown_connections 2023-10-30T12:00:53.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f43750bb640 1 -- 172.21.15.50:0/4252039707 wait complete. 2023-10-30T12:00:53.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f75627fc640 1 librados: shutdown 2023-10-30T12:00:53.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.635+0000 7f43750bb640 1 librados: shutdown 2023-10-30T12:00:53.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.649+0000 7f7cbff73640 10 monclient: get_monmap_and_config 2023-10-30T12:00:53.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.649+0000 7f7cbff73640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.649+0000 7f7cbff73640 10 monclient: monmap: 2023-10-30T12:00:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:53.650543+0000 2023-10-30T12:00:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:53.650543+0000 2023-10-30T12:00:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:53.762 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.649+0000 7fcee00e1640 10 monclient: get_monmap_and_config 2023-10-30T12:00:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.649+0000 7fcee00e1640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.649+0000 7fcee00e1640 10 monclient: monmap: 2023-10-30T12:00:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:53.650513+0000 2023-10-30T12:00:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:53.650513+0000 2023-10-30T12:00:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:53.765 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7fcee00e1640 1 Processor -- start 2023-10-30T12:00:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 1 Processor -- start 2023-10-30T12:00:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7fcee00e1640 1 -- start start 2023-10-30T12:00:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7fcee00e1640 10 monclient: init 2023-10-30T12:00:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 1 -- start start 2023-10-30T12:00:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 10 monclient: init 2023-10-30T12:00:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7fcee00e1640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7fcee00e1640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:00:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-10-30T12:00:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cb80a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7fcee00e1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fced80a8010 0x7fced8132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7fcee00e1640 10 monclient(hunting): picked mon.noname-a con 0x7fced80a8010 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fced8099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 10 monclient(hunting): picked mon.noname-b con 0x7f7cb80a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.650+0000 7f7cbff73640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7cb813af40 0x7f7cb813d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb813af40 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb80a4190 con 0x7f7cb80a2ae0 2023-10-30T12:00:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb80a1630 con 0x7f7cb813af40 2023-10-30T12:00:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb809a2e0 con 0x7f7cb80a2450 2023-10-30T12:00:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbff73640 10 monclient(hunting): authenticate will time out at 898.834097s 2023-10-30T12:00:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fced8132bd0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80a7190 con 0x7fced8099a40 2023-10-30T12:00:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80a83f0 con 0x7fced8132bd0 2023-10-30T12:00:53.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.775 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:53.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:00:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced80a7600 con 0x7fced80a8010 2023-10-30T12:00:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fced80a8010 0x7fced8132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcee00e1640 10 monclient(hunting): authenticate will time out at 898.834339s 2023-10-30T12:00:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fced80a8010 0x7fced8132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38028/0 (socket says 172.21.15.50:38028) 2023-10-30T12:00:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 1 -- 172.21.15.50:0/1149780609 learned_addr learned my addr 172.21.15.50:0/1149780609 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): get_auth_request con 0x7fced80a8010 auth_method 0 2023-10-30T12:00:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedd67c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fced8099a40 0x7fced81356d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60818/0 (socket says 172.21.15.50:60818) 2023-10-30T12:00:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb813aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35618/0 (socket says 172.21.15.50:35618) 2023-10-30T12:00:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 -- 172.21.15.50:0/1491607957 learned_addr learned my addr 172.21.15.50:0/1491607957 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): get_auth_request con 0x7f7cb80a2450 auth_method 0 2023-10-30T12:00:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient(hunting): handle_auth_done global_id 4260 payload 274 2023-10-30T12:00:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 1 -- 172.21.15.50:0/1149780609 >> v1:172.21.15.118:6789/0 conn(0x7fced8132bd0 legacy=0x7fced8135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2023-10-30T12:00:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 -- 172.21.15.50:0/1491607957 >> v2:172.21.15.118:3300/0 conn(0x7f7cb813af40 msgr2=0x7f7cb813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 --2- 172.21.15.50:0/1491607957 >> v2:172.21.15.118:3300/0 conn(0x7f7cb813af40 0x7f7cb813d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 -- 172.21.15.50:0/1491607957 >> v1:172.21.15.118:6789/0 conn(0x7f7cb80a2ae0 legacy=0x7f7cb80a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 1 -- 172.21.15.50:0/1149780609 >> v1:172.21.15.50:6789/0 conn(0x7fced8099a40 legacy=0x7fced81356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 1 monclient: found mon.noname-a 2023-10-30T12:00:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7fcedde7d640 1 -- 172.21.15.50:0/1149780609 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced81364f0 con 0x7fced80a8010 2023-10-30T12:00:53.786 INFO:teuthology.orchestra.run.smithi050.stdout:47244640258 2023-10-30T12:00:53.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:00:53.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 monclient: found mon.noname-b 2023-10-30T12:00:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.651+0000 7f7cbd50e640 1 -- 172.21.15.50:0/1491607957 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb809a8a0 con 0x7f7cb80a2450 2023-10-30T12:00:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.652+0000 7fcedde7d640 1 --2- 172.21.15.50:0/1149780609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced80a8010 0x7fced8132690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fced4009870 tx=0x7fced402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6d7c973c690330 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.652+0000 7f7cbd50e640 1 --2- 172.21.15.50:0/1491607957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb813aa00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009870 tx=0x7f7ca802b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f2601a753fab13 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.652+0000 7fcedce7b640 1 -- 172.21.15.50:0/1149780609 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4031860 con 0x7fced80a8010 2023-10-30T12:00:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.652+0000 7fcedce7b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.652+0000 7fcedce7b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.652+0000 7fcedce7b640 10 monclient: dump: 2023-10-30T12:00:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.792 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 5 monclient: authenticate success, global_id 4260 2023-10-30T12:00:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 1 -- 172.21.15.50:0/1149780609 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced40319c0 con 0x7fced80a8010 2023-10-30T12:00:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 1 -- 172.21.15.50:0/1149780609 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4031d50 con 0x7fced80a8010 2023-10-30T12:00:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 4 set_mon_vals no callback set 2023-10-30T12:00:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 1 -- 172.21.15.50:0/1491607957 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8031750 con 0x7f7cb80a2450 2023-10-30T12:00:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: dump: 2023-10-30T12:00:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.798 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 1 -- 172.21.15.50:0/1491607957 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca80318b0 con 0x7f7cb80a2450 2023-10-30T12:00:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 1 -- 172.21.15.50:0/1491607957 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8031c40 con 0x7f7cb80a2450 2023-10-30T12:00:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcedce7b640 10 monclient: dump: 2023-10-30T12:00:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.801 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbcd0d640 10 monclient: dump: 2023-10-30T12:00:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.804 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 5 monclient: authenticate success, global_id 4214 2023-10-30T12:00:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 4 set_mon_vals no callback set 2023-10-30T12:00:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 10 monclient: shutdown 2023-10-30T12:00:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 1 -- 172.21.15.50:0/1149780609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced80a8010 msgr2=0x7fced8132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 1 --2- 172.21.15.50:0/1149780609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced80a8010 0x7fced8132690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fced4009870 tx=0x7fced402b900 comp rx=0 tx=0).stop 2023-10-30T12:00:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 1 -- 172.21.15.50:0/1149780609 shutdown_connections 2023-10-30T12:00:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 1 --2- 172.21.15.50:0/1149780609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced80a8010 0x7fced8132690 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 1 -- 172.21.15.50:0/1149780609 >> 172.21.15.50:0/1149780609 conn(0x7fced80a4a10 msgr2=0x7fced80a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 10 monclient: shutdown 2023-10-30T12:00:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 1 -- 172.21.15.50:0/1491607957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 msgr2=0x7f7cb813aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 1 --2- 172.21.15.50:0/1491607957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb813aa00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009870 tx=0x7f7ca802b7f0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7fcee00e1640 1 -- 172.21.15.50:0/1149780609 shutdown_connections 2023-10-30T12:00:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 1 -- 172.21.15.50:0/1491607957 shutdown_connections 2023-10-30T12:00:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.653+0000 7f7cbff73640 1 --2- 172.21.15.50:0/1491607957 >> v2:172.21.15.118:3300/0 conn(0x7f7cb813af40 0x7f7cb813d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7f7cbff73640 1 --2- 172.21.15.50:0/1491607957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb813aa00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7f7cbff73640 1 -- 172.21.15.50:0/1491607957 >> 172.21.15.50:0/1491607957 conn(0x7f7cb80a80b0 msgr2=0x7f7cb80a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7fcee00e1640 1 -- 172.21.15.50:0/1149780609 wait complete. 2023-10-30T12:00:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7f7cbff73640 1 -- 172.21.15.50:0/1491607957 shutdown_connections 2023-10-30T12:00:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7f7cbff73640 1 -- 172.21.15.50:0/1491607957 wait complete. 2023-10-30T12:00:53.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7fcee00e1640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7fcee00e1640 10 monclient: monmap: 2023-10-30T12:00:53.811 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.811 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.811 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.812 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.812 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.812 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.812 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.812 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.813 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7f7cbff73640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7f7cbff73640 10 monclient: monmap: 2023-10-30T12:00:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.816 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7fcee00e1640 1 librados: starting msgr at 2023-10-30T12:00:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.654+0000 7fcee00e1640 1 librados: starting objecter 2023-10-30T12:00:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 1 Processor -- start 2023-10-30T12:00:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 librados: starting msgr at 2023-10-30T12:00:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 librados: starting objecter 2023-10-30T12:00:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fc07317d640 10 monclient: get_monmap_and_config 2023-10-30T12:00:53.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fc07317d640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 1 -- start start 2023-10-30T12:00:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fc07317d640 10 monclient: monmap: 2023-10-30T12:00:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:53.656115+0000 2023-10-30T12:00:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:53.656115+0000 2023-10-30T12:00:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.820 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.820 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:53.820 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:53.820 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:53.820 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:53.821 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:53.821 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 1 librados: setting wanted keys 2023-10-30T12:00:53.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 1 librados: calling monclient init 2023-10-30T12:00:53.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 10 monclient: init 2023-10-30T12:00:53.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 Processor -- start 2023-10-30T12:00:53.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 -- start start 2023-10-30T12:00:53.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f4ada67c640 10 monclient: get_monmap_and_config 2023-10-30T12:00:53.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f4ada67c640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f4ada67c640 10 monclient: monmap: 2023-10-30T12:00:53.824 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.824 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.824 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:53.656354+0000 2023-10-30T12:00:53.824 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:53.656354+0000 2023-10-30T12:00:53.824 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.825 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.825 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:53.825 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:53.825 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:53.825 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:53.825 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:53.826 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 librados: setting wanted keys 2023-10-30T12:00:53.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 librados: calling monclient init 2023-10-30T12:00:53.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient: init 2023-10-30T12:00:53.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:00:53.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb81321e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb80a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb80a2ae0 0x7f7cb8132720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): picked mon.noname-a con 0x7f7cb80a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb813af40 0x7f7cb81370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): picked mon.noname-b con 0x7f7cb813af40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb81cb180 con 0x7f7cb80a2ae0 2023-10-30T12:00:53.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb80a1790 con 0x7f7cb813af40 2023-10-30T12:00:53.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb813a870 con 0x7f7cb80a2450 2023-10-30T12:00:53.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbff73640 10 monclient(hunting): authenticate will time out at 898.838775s 2023-10-30T12:00:53.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbdd0f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb81321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:00:53.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbdd0f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb81321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35626/0 (socket says 172.21.15.50:35626) 2023-10-30T12:00:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbdd0f640 1 -- 172.21.15.50:0/1296179123 learned_addr learned my addr 172.21.15.50:0/1296179123 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbdd0f640 10 monclient(hunting): get_auth_request con 0x7f7cb80a2450 auth_method 0 2023-10-30T12:00:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbdd0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbdd0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7f7cbdd0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.655+0000 7fcee00e1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fced80a8010 0x7fced81ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): picked mon.noname-c con 0x7fced80a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb80a2ae0 0x7f7cb8132720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 10 monclient(hunting): get_auth_request con 0x7f7cb80a2ae0 auth_method 0 2023-10-30T12:00:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb813af40 0x7f7cb81370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fced8132bd0 0x7fced81cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fc07317d640 1 Processor -- start 2023-10-30T12:00:53.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): get_auth_request con 0x7f7cb813af40 auth_method 0 2023-10-30T12:00:53.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbdd0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fced80a8010 0x7fced81ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbdd0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbdd0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fc07317d640 1 -- start start 2023-10-30T12:00:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fc07317d640 10 monclient: init 2023-10-30T12:00:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): picked mon.noname-b con 0x7fced8132bd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fced80a8010 0x7fced81ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35638/0 (socket says 172.21.15.50:35638) 2023-10-30T12:00:53.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 1 -- 172.21.15.50:0/3686367470 learned_addr learned my addr 172.21.15.50:0/3686367470 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced8122a90 0x7fced811e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedd67c640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fced8132bd0 0x7fced81cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f4ada67c640 1 Processor -- start 2023-10-30T12:00:53.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2023-10-30T12:00:53.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 -- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 msgr2=0x7f7cb81321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb81321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 -- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb80a2ae0 msgr2=0x7f7cb8132720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb80a2ae0 0x7f7cb8132720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 monclient: found mon.noname-b 2023-10-30T12:00:53.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 -- 172.21.15.50:0/1296179123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca80094d0 con 0x7f7cb813af40 2023-10-30T12:00:53.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcede67e640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced8122a90 0x7fced811e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbdd0f640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb80a2450 0x7f7cb81321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): picked mon.noname-a con 0x7fced8122a90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced81220b0 con 0x7fced8122a90 2023-10-30T12:00:53.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced80a18d0 con 0x7fced8132bd0 2023-10-30T12:00:53.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced8122670 con 0x7fced80a8010 2023-10-30T12:00:53.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbd50e640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb80a2ae0 0x7f7cb8132720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcee00e1640 10 monclient(hunting): authenticate will time out at 898.839828s 2023-10-30T12:00:53.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 10 monclient(hunting): get_auth_request con 0x7fced80a8010 auth_method 0 2023-10-30T12:00:53.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f4ada67c640 1 -- start start 2023-10-30T12:00:53.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f4ada67c640 10 monclient: init 2023-10-30T12:00:53.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedd67c640 10 monclient(hunting): get_auth_request con 0x7fced8132bd0 auth_method 0 2023-10-30T12:00:53.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedd67c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedd67c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedd67c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcede67e640 10 monclient(hunting): get_auth_request con 0x7fced8122a90 auth_method 0 2023-10-30T12:00:53.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcede67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcede67e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcede67e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fcedde7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cbe510640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb813af40 0x7f7cb81370c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8072170 tx=0x7f7cb0009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cae7fc640 10 client.?.objecter ms_handle_connect 0x7f7cb813af40 2023-10-30T12:00:53.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7f7cae7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.656+0000 7fc07317d640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:00:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb000cdf0 con 0x7f7cb813af40 2023-10-30T12:00:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc06c0a8010 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: dump: 2023-10-30T12:00:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.858 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.858 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.858 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.858 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.858 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc06c099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:53.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc06c132da0 0x7fc06c13e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 10 monclient(hunting): picked mon.noname-c con 0x7fc06c132da0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:53.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06c0a7190 con 0x7fc06c099a40 2023-10-30T12:00:53.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb0014440 con 0x7f7cb813af40 2023-10-30T12:00:53.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0014830 con 0x7f7cb813af40 2023-10-30T12:00:53.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:53.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cae7fc640 10 monclient: dump: 2023-10-30T12:00:53.861 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.862 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.862 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.862 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.862 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.862 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.863 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.863 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.863 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.863 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06c0a83f0 con 0x7fc06c0a8010 2023-10-30T12:00:53.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06c0a7600 con 0x7fc06c132da0 2023-10-30T12:00:53.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07317d640 10 monclient(hunting): authenticate will time out at 898.840197s 2023-10-30T12:00:53.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 5 monclient: authenticate success, global_id 4282 2023-10-30T12:00:53.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 10 monclient: _renew_subs 2023-10-30T12:00:53.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7caffff640 4 set_mon_vals no callback set 2023-10-30T12:00:53.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcede67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcede67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcede67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb811e8b0 con 0x7f7cb813af40 2023-10-30T12:00:53.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-10-30T12:00:53.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 10 monclient: _renew_subs 2023-10-30T12:00:53.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb8139b50 con 0x7f7cb813af40 2023-10-30T12:00:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f7cbff73640 1 librados: init done 2023-10-30T12:00:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ad40a8010 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4ad4099a40 0x7f4ad413aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient(hunting): picked mon.noname-a con 0x7f4ad4099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4ad4132da0 0x7f4ad413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient(hunting): picked mon.noname-c con 0x7f4ad4132da0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40a7190 con 0x7f4ad40a8010 2023-10-30T12:00:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40a83f0 con 0x7f4ad4099a40 2023-10-30T12:00:53.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2023-10-30T12:00:53.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 1 -- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fced8132bd0 msgr2=0x7fced81cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fced8132bd0 0x7fced81cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 1 -- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced8122a90 msgr2=0x7fced811e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc070f19640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fc06c0a8010 0x7fc06c132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:34242/0 (socket says 172.21.15.50:34242) 2023-10-30T12:00:53.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc070f19640 1 -- 172.21.15.50:0/4014022647 learned_addr learned my addr 172.21.15.50:0/4014022647 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:53.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40a7600 con 0x7f4ad4132da0 2023-10-30T12:00:53.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ada67c640 10 monclient(hunting): authenticate will time out at 898.840461s 2023-10-30T12:00:53.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07171a640 1 --2- 172.21.15.50:0/4014022647 >> v2:172.21.15.118:3300/0 conn(0x7fc06c132da0 0x7fc06c13e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad3fff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f4ad40a8010 0x7f4ad4132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:34248/0 (socket says 172.21.15.50:34248) 2023-10-30T12:00:53.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad3fff640 1 -- 172.21.15.50:0/936146000 learned_addr learned my addr 172.21.15.50:0/936146000 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:53.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad37fe640 1 --2- 172.21.15.50:0/936146000 >> v2:172.21.15.50:3300/0 conn(0x7f4ad4099a40 0x7f4ad413aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced8122a90 0x7fced811e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 1 monclient: found mon.noname-c 2023-10-30T12:00:53.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcedde7d640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced40094d0 con 0x7fced80a8010 2023-10-30T12:00:53.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fcede67e640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fced8122a90 0x7fced811e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07171a640 10 monclient(hunting): get_auth_request con 0x7fc06c132da0 auth_method 0 2023-10-30T12:00:53.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad8c19640 1 --2- 172.21.15.50:0/936146000 >> v2:172.21.15.118:3300/0 conn(0x7f4ad4132da0 0x7f4ad413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad37fe640 10 monclient(hunting): get_auth_request con 0x7f4ad4099a40 auth_method 0 2023-10-30T12:00:53.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07171a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07171a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc07171a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc06b7fe640 1 -- 172.21.15.50:0/4014022647 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451806543 0 0) 0x7fc06c0a83f0 con 0x7fc06c0a8010 2023-10-30T12:00:53.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc06b7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:00:53.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc06b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc06b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc06b7fe640 10 monclient(hunting): my global_id is 4288 2023-10-30T12:00:53.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7fc06b7fe640 1 -- 172.21.15.50:0/4014022647 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc058003520 con 0x7fc06c0a8010 2023-10-30T12:00:53.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.657+0000 7f4ad8c19640 10 monclient(hunting): get_auth_request con 0x7f4ad4132da0 auth_method 0 2023-10-30T12:00:53.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992345875 0 0) 0x7f4ad40a7190 con 0x7f4ad40a8010 2023-10-30T12:00:53.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:00:53.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): my global_id is 4294 2023-10-30T12:00:53.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4abc003520 con 0x7f4ad40a8010 2023-10-30T12:00:53.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-10-30T12:00:53.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 1 -- 172.21.15.50:0/4014022647 >> v1:172.21.15.118:6789/0 conn(0x7fc06c0a8010 legacy=0x7fc06c132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc06b7fe640 1 -- 172.21.15.50:0/4014022647 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2742233079 0 0) 0x7fc058003520 con 0x7fc06c0a8010 2023-10-30T12:00:53.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 1 -- 172.21.15.50:0/4014022647 >> v1:172.21.15.50:6789/0 conn(0x7fc06c099a40 legacy=0x7fc06c13aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 1 monclient: found mon.noname-c 2023-10-30T12:00:53.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc07171a640 1 -- 172.21.15.50:0/4014022647 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c09a440 con 0x7fc06c132da0 2023-10-30T12:00:53.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcedde7d640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fced80a8010 0x7fced81ca830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fced4009870 tx=0x7fced40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 10 client.?.objecter ms_handle_connect 0x7fced80a8010 2023-10-30T12:00:53.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:53.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fc06b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:53.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1301514643 0 0) 0x7f4abc003520 con 0x7f4ad40a8010 2023-10-30T12:00:53.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 >> v2:172.21.15.118:3300/0 conn(0x7f4ad4132da0 msgr2=0x7f4ad413cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 --2- 172.21.15.50:0/936146000 >> v2:172.21.15.118:3300/0 conn(0x7f4ad4132da0 0x7f4ad413cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 >> v2:172.21.15.50:3300/0 conn(0x7f4ad4099a40 msgr2=0x7f4ad413aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7cb001d3b0 con 0x7f7cb813af40 2023-10-30T12:00:53.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4031810 con 0x7fced80a8010 2023-10-30T12:00:53.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:53.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 10 monclient: dump: 2023-10-30T12:00:53.891 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.891 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.891 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.891 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.892 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.892 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.892 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.892 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.892 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.892 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 5 monclient: authenticate success, global_id 4220 2023-10-30T12:00:53.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 10 monclient: _renew_subs 2023-10-30T12:00:53.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 --2- 172.21.15.50:0/936146000 >> v2:172.21.15.50:3300/0 conn(0x7f4ad4099a40 0x7f4ad413aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:00:53.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:00:53.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad409a440 con 0x7f4ad40a8010 2023-10-30T12:00:53.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced8135ba0 con 0x7fced80a8010 2023-10-30T12:00:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f4ac40019f0 con 0x7f4ad40a8010 2023-10-30T12:00:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:00:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: dump: 2023-10-30T12:00:53.898 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.898 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.898 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.898 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.899 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.899 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.899 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.899 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.899 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.899 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:00:53.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 >> v1:172.21.15.118:6789/0 conn(0x7f4ad40a8010 legacy=0x7f4ad4132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:53.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4abc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): picked mon.a con 0x7f4abc003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4abc0047c0 0x7f4abc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): picked mon.b con 0x7f4abc0047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc005180 0x7f4abc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): picked mon.c con 0x7f4abc005180 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40a83f0 con 0x7f4abc003970 2023-10-30T12:00:53.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40a7600 con 0x7f4abc0047c0 2023-10-30T12:00:53.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc00c100 con 0x7f4abc005180 2023-10-30T12:00:53.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad2ffd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:00:53.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 1 --2- 172.21.15.50:0/936146000 >> v2:172.21.15.118:3300/0 conn(0x7f4ad4132da0 0x7f4ad413cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:53.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 10 monclient: _renew_subs 2023-10-30T12:00:53.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbff73640 10 client.4282.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbff73640 15 client.4282.objecter _session_command_op_assign -1 1 2023-10-30T12:00:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbff73640 20 client.4282.objecter _get_session osd=-1 returning homeless 2023-10-30T12:00:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbff73640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced8120230 con 0x7fced80a8010 2023-10-30T12:00:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbff73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7cb811e360 con 0x7f7cb813af40 2023-10-30T12:00:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c9c0392f0 0x7f7c9c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f7cb004a700 con 0x7f7cb813af40 2023-10-30T12:00:53.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 10 client.4282.objecter ms_dispatch 0x7f7cb8000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:53.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 3 client.4282.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:53.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 3 client.4282.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad3fff640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4abc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad3fff640 10 monclient(hunting): get_auth_request con 0x7f4abc003970 auth_method 0 2023-10-30T12:00:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc005180 0x7f4abc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 10 monclient(hunting): get_auth_request con 0x7f4abc005180 auth_method 0 2023-10-30T12:00:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f4ad8c19640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:53.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 10 client.4282.objecter checking command 1 2023-10-30T12:00:53.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7f7c9c03f100 0x7f7c9c041520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad8c19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad8c19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad8c19640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:53.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient(hunting): handle_auth_done global_id 4294 payload 293 2023-10-30T12:00:53.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 1 -- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc005180 msgr2=0x7f4abc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 20 client.4282.objecter _get_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 20 client.4282.objecter put_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 15 client.4282.objecter _session_command_op_remove -1 1 2023-10-30T12:00:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 20 client.4282.objecter _get_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:53.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 20 client.4282.objecter get_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:53.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 15 client.4282.objecter _session_command_op_assign 0 1 2023-10-30T12:00:53.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 20 client.4282.objecter put_session s=0x7f7c9c03ea30 osd=0 4 2023-10-30T12:00:53.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 10 client.4282.objecter _send_command 1 2023-10-30T12:00:53.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 --> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c9c041bd0 con 0x7f7c9c03f100 2023-10-30T12:00:53.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cae7fc640 20 client.4282.objecter dump_active .. 0 homeless 2023-10-30T12:00:53.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbdd0f640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c9c0392f0 0x7f7c9c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.658+0000 7f7cbdd0f640 10 monclient: get_auth_request con 0x7f7c9c0392f0 auth_method 0 2023-10-30T12:00:53.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad37fe640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4abc0047c0 0x7f4abc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f7cbd50e640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7f7c9c03f100 0x7f7c9c041520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f7cbd50e640 10 monclient: get_auth_request con 0x7f7c9c03f100 auth_method 0 2023-10-30T12:00:53.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc005180 0x7f4abc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 1 -- 172.21.15.50:0/936146000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4abc0047c0 msgr2=0x7f4abc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:53.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4abc0047c0 0x7f4abc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 1 monclient: found mon.a 2023-10-30T12:00:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 1 -- 172.21.15.50:0/936146000 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad409a440 con 0x7f4abc003970 2023-10-30T12:00:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad8c19640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc005180 0x7f4abc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad8c19640 1 -- 172.21.15.50:0/936146000 reap_dead start 2023-10-30T12:00:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cb001be80 con 0x7f7cb813af40 2023-10-30T12:00:53.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f7cae7fc640 10 monclient: handle_get_version_reply finishing 1 version 16 2023-10-30T12:00:53.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fc07171a640 1 --2- 172.21.15.50:0/4014022647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c13e140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009d90 tx=0x7fc05c02bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f61d0b9181549b25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f7cae7fc640 10 client.4282.objecter ms_handle_connect 0x7f7c9c03f100 2023-10-30T12:00:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f7cbd50e640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7f7c9c03f100 0x7f7c9c041520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad37fe640 10 monclient: get_auth_request con 0x7f4abc0047c0 auth_method 0 2023-10-30T12:00:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ada67c640 5 monclient: authenticate success, global_id 4294 2023-10-30T12:00:53.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ada67c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:53.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f4ad37fe640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4abc0047c0 0x7f4abc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:53.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcee00e1640 1 librados: init done 2023-10-30T12:00:53.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced40443f0 con 0x7fced80a8010 2023-10-30T12:00:53.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcece7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4044760 con 0x7fced80a8010 2023-10-30T12:00:53.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcece7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:53.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcece7fc640 10 monclient: dump: 2023-10-30T12:00:53.924 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.924 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.924 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.924 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.925 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.925 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.925 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.925 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.925 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.925 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7fcecffff640 4 set_mon_vals no callback set 2023-10-30T12:00:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 1 -- 172.21.15.50:0/4014022647 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064001670 con 0x7fc06c132da0 2023-10-30T12:00:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:53.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fced4044900 con 0x7fced80a8010 2023-10-30T12:00:53.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: dump: 2023-10-30T12:00:53.928 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.928 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.928 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.928 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.928 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.928 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.929 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.929 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.929 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.929 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 1 -- 172.21.15.50:0/4014022647 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c02c850 con 0x7fc06c132da0 2023-10-30T12:00:53.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 1 -- 172.21.15.50:0/4014022647 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c02cbe0 con 0x7fc06c132da0 2023-10-30T12:00:53.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:53.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc06b7fe640 10 monclient: dump: 2023-10-30T12:00:53.931 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.931 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.932 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.932 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.932 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.932 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.932 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.933 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.933 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.933 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.659+0000 7f7cbdd0f640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c9c0392f0 0x7f7c9c03b7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40043e0 tx=0x7f7cb4007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f7cae7fc640 10 client.4282.objecter ms_handle_connect 0x7f7c9c0392f0 2023-10-30T12:00:53.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc07317d640 5 monclient: authenticate success, global_id 4291 2023-10-30T12:00:53.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc07317d640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fc07317d640 4 set_mon_vals no callback set 2023-10-30T12:00:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcee00e1640 10 client.4220.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcee00e1640 15 client.4220.objecter _session_command_op_assign -1 1 2023-10-30T12:00:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcee00e1640 20 client.4220.objecter _get_session osd=-1 returning homeless 2023-10-30T12:00:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcee00e1640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcee00e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fced809aa00 con 0x7fced80a8010 2023-10-30T12:00:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcebc0392f0 0x7fcebc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad3fff640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4abc003d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400a8d0 tx=0x7f4abc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f747d1d8bb288383 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fced406d580 con 0x7fced80a8010 2023-10-30T12:00:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 10 client.4220.objecter ms_dispatch 0x7fced8000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 3 client.4220.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 3 client.4220.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 10 client.4220.objecter checking command 1 2023-10-30T12:00:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7fcebc03f100 0x7fcebc041520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 20 client.4220.objecter _get_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 20 client.4220.objecter put_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 15 client.4220.objecter _session_command_op_remove -1 1 2023-10-30T12:00:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 20 client.4220.objecter _get_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 20 client.4220.objecter get_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 15 client.4220.objecter _session_command_op_assign 5 1 2023-10-30T12:00:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 20 client.4220.objecter put_session s=0x7fcebc03ea30 osd=5 4 2023-10-30T12:00:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 10 client.4220.objecter _send_command 1 2023-10-30T12:00:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcebc041bd0 con 0x7fcebc03f100 2023-10-30T12:00:53.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 20 client.4220.objecter dump_active .. 0 homeless 2023-10-30T12:00:53.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fced4044bb0 con 0x7fced80a8010 2023-10-30T12:00:53.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcece7fc640 10 monclient: handle_get_version_reply finishing 1 version 16 2023-10-30T12:00:53.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcedd67c640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcebc0392f0 0x7fcebc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== osd.0 v2:172.21.15.50:6816/47190190 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f7c9c041bd0 con 0x7f7c9c03f100 2023-10-30T12:00:53.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f7cae7fc640 10 client.4282.objecter ms_dispatch 0x7f7cb8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:00:53.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f7cae7fc640 10 client.4282.objecter _finish_command 1 = system:0 2023-10-30T12:00:53.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f7cae7fc640 20 client.4282.objecter put_session s=0x7f7c9c03ea30 osd=0 4 2023-10-30T12:00:53.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f7cae7fc640 15 client.4282.objecter _session_command_op_remove 0 1 2023-10-30T12:00:53.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4034630 con 0x7f4abc003970 2023-10-30T12:00:53.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:53.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 10 monclient: dump: 2023-10-30T12:00:53.943 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.944 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.944 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.944 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.944 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.944 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.945 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.945 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.945 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.945 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ada67c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:53.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac402f070 con 0x7f4abc003970 2023-10-30T12:00:53.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7fcedd67c640 10 monclient: get_auth_request con 0x7fcebc0392f0 auth_method 0 2023-10-30T12:00:53.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ada67c640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:53.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ada67c640 4 set_mon_vals no callback set 2023-10-30T12:00:53.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 1 -- 172.21.15.50:0/936146000 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4034630 con 0x7f4abc003970 2023-10-30T12:00:53.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:53.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.660+0000 7f4ad2ffd640 10 monclient: dump: 2023-10-30T12:00:53.948 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.948 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.948 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.948 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.948 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.949 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.949 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.949 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.949 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.949 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fcede67e640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7fcebc03f100 0x7fcebc041520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fcede67e640 10 monclient: get_auth_request con 0x7fcebc03f100 auth_method 0 2023-10-30T12:00:53.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 10 monclient: shutdown 2023-10-30T12:00:53.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 1 -- 172.21.15.50:0/4014022647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 msgr2=0x7fc06c13e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 1 --2- 172.21.15.50:0/4014022647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c13e140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009d90 tx=0x7fc05c02bd20 comp rx=0 tx=0).stop 2023-10-30T12:00:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 1 -- 172.21.15.50:0/4014022647 shutdown_connections 2023-10-30T12:00:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 1 --2- 172.21.15.50:0/4014022647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c13e140 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 1 -- 172.21.15.50:0/4014022647 >> 172.21.15.50:0/4014022647 conn(0x7fc06c0a4a10 msgr2=0x7fc06c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 1 -- 172.21.15.50:0/4014022647 shutdown_connections 2023-10-30T12:00:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 1 -- 172.21.15.50:0/4014022647 wait complete. 2023-10-30T12:00:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fcece7fc640 10 client.4220.objecter ms_handle_connect 0x7fcebc0392f0 2023-10-30T12:00:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fcede67e640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7fcebc03f100 0x7fcebc041520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fcece7fc640 10 client.4220.objecter ms_handle_connect 0x7fcebc03f100 2023-10-30T12:00:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fcedd67c640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcebc0392f0 0x7fcebc03b7b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fced8131520 tx=0x7fcec8007c80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 10 monclient: shutdown 2023-10-30T12:00:53.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:53.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 1 -- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 msgr2=0x7f4abc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4abc003d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400a8d0 tx=0x7f4abc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 1 -- 172.21.15.50:0/936146000 shutdown_connections 2023-10-30T12:00:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 1 --2- 172.21.15.50:0/936146000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4abc003d90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 1 -- 172.21.15.50:0/936146000 >> 172.21.15.50:0/936146000 conn(0x7f4ad40a4a10 msgr2=0x7f4ad40a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 1 -- 172.21.15.50:0/936146000 shutdown_connections 2023-10-30T12:00:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7f4ada67c640 1 -- 172.21.15.50:0/936146000 wait complete. 2023-10-30T12:00:53.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.661+0000 7fc07317d640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 10 monclient: monmap: 2023-10-30T12:00:53.957 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.957 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.957 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.958 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.958 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.958 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.958 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.958 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.959 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.959 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 1 librados: starting msgr at 2023-10-30T12:00:53.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 1 librados: starting objecter 2023-10-30T12:00:53.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== osd.5 v2:172.21.15.118:6800/1399177807 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fcebc041bd0 con 0x7fcebc03f100 2023-10-30T12:00:53.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fcece7fc640 10 client.4220.objecter ms_dispatch 0x7fced8000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:00:53.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fcece7fc640 10 client.4220.objecter _finish_command 1 = system:0 2023-10-30T12:00:53.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fcece7fc640 20 client.4220.objecter put_session s=0x7fcebc03ea30 osd=5 4 2023-10-30T12:00:53.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fcece7fc640 15 client.4220.objecter _session_command_op_remove 5 1 2023-10-30T12:00:53.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 1 Processor -- start 2023-10-30T12:00:53.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 1 -- start start 2023-10-30T12:00:53.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 1 librados: setting wanted keys 2023-10-30T12:00:53.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 1 librados: calling monclient init 2023-10-30T12:00:53.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7fc07317d640 10 monclient: init 2023-10-30T12:00:53.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7f4ada67c640 10 monclient: build_initial_monmap 2023-10-30T12:00:53.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7f4ada67c640 10 monclient: monmap: 2023-10-30T12:00:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:53.964 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.964 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:53.964 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:53.964 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:53.964 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7f4ada67c640 1 librados: starting msgr at 2023-10-30T12:00:53.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7f4ada67c640 1 librados: starting objecter 2023-10-30T12:00:53.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.662+0000 7f4ada67c640 1 Processor -- start 2023-10-30T12:00:53.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 -- start start 2023-10-30T12:00:53.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 librados: setting wanted keys 2023-10-30T12:00:53.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 librados: calling monclient init 2023-10-30T12:00:53.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient: init 2023-10-30T12:00:53.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:53.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc06c0a8010 0x7fc06c0a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): picked mon.noname-a con 0x7fc06c0a8010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c1367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): picked mon.noname-b con 0x7fc06c132da0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc06c136d00 0x7fc06c1371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc06bfff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c1367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07171a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc06c136d00 0x7fc06c1371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc06bfff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c1367c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:59122/0 (socket says 172.21.15.50:59122) 2023-10-30T12:00:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07171a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc06c136d00 0x7fc06c1371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35644/0 (socket says 172.21.15.50:35644) 2023-10-30T12:00:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc070f19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc06c0a8010 0x7fc06c0a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc070f19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc06c0a8010 0x7fc06c0a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38066/0 (socket says 172.21.15.50:38066) 2023-10-30T12:00:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): picked mon.noname-c con 0x7fc06c136d00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c139d70 con 0x7fc06c0a8010 2023-10-30T12:00:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c1396b0 con 0x7fc06c132da0 2023-10-30T12:00:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c13d5d0 con 0x7fc06c136d00 2023-10-30T12:00:53.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc07317d640 10 monclient(hunting): authenticate will time out at 898.846857s 2023-10-30T12:00:53.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc06bfff640 1 -- 172.21.15.50:0/3091845062 learned_addr learned my addr 172.21.15.50:0/3091845062 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc06bfff640 10 monclient(hunting): get_auth_request con 0x7fc06c132da0 auth_method 0 2023-10-30T12:00:53.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7fc06bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:53.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 10 monclient(hunting): get_auth_request con 0x7fc06c136d00 auth_method 0 2023-10-30T12:00:53.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:00:53.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4ad409f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient(hunting): picked mon.noname-a con 0x7f4abc003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:53.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ad409fad0 0x7f4ad409ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient(hunting): picked mon.noname-c con 0x7f4ad409fad0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:53.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ad40a0470 0x7f4ad41d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:53.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient(hunting): picked mon.noname-b con 0x7f4ad40a0470 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:53.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad41200d0 con 0x7f4abc003970 2023-10-30T12:00:53.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad411ebb0 con 0x7f4ad40a0470 2023-10-30T12:00:53.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:53.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40a18d0 con 0x7f4ad409fad0 2023-10-30T12:00:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ada67c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ad3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4ad409f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ada67c640 10 monclient(hunting): authenticate will time out at 898.847023s 2023-10-30T12:00:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.663+0000 7f4ad37fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ad409fad0 0x7f4ad409ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4ad409f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38070/0 (socket says 172.21.15.50:38070) 2023-10-30T12:00:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad3fff640 1 -- 172.21.15.50:0/3618028076 learned_addr learned my addr 172.21.15.50:0/3618028076 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad3fff640 10 monclient(hunting): get_auth_request con 0x7f4abc003970 auth_method 0 2023-10-30T12:00:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): get_auth_request con 0x7f4ad409fad0 auth_method 0 2023-10-30T12:00:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad8c19640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ad40a0470 0x7f4ad41d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad8c19640 10 monclient(hunting): get_auth_request con 0x7f4ad40a0470 auth_method 0 2023-10-30T12:00:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad8c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad8c19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad8c19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2023-10-30T12:00:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 1 -- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ad40a0470 msgr2=0x7f4ad41d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ad40a0470 0x7f4ad41d05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 1 -- 172.21.15.50:0/3618028076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 msgr2=0x7f4ad409f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:53.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:53.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4ad409f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 1 monclient: found mon.noname-c 2023-10-30T12:00:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 1 -- 172.21.15.50:0/3618028076 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad411dfb0 con 0x7f4ad409fad0 2023-10-30T12:00:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2023-10-30T12:00:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 1 -- 172.21.15.50:0/3091845062 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc06c136d00 msgr2=0x7fc06c1371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad3fff640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc003970 0x7f4ad409f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc06c136d00 0x7fc06c1371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 1 -- 172.21.15.50:0/3091845062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc06c0a8010 msgr2=0x7fc06c0a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc06c0a8010 0x7fc06c0a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 1 monclient: found mon.noname-b 2023-10-30T12:00:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc06bfff640 1 -- 172.21.15.50:0/3091845062 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c009a70 con 0x7fc06c132da0 2023-10-30T12:00:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc070f19640 10 monclient: get_auth_request con 0x7fc06c0a8010 auth_method 0 2023-10-30T12:00:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc070f19640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc06c0a8010 0x7fc06c0a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7fc07171a640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc06c136d00 0x7fc06c1371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad37fe640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ad409fad0 0x7f4ad409ff30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4136790 tx=0x7f4ac8010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 client.?.objecter ms_handle_connect 0x7f4ad409fad0 2023-10-30T12:00:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac800e920 con 0x7f4ad409fad0 2023-10-30T12:00:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:53.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: dump: 2023-10-30T12:00:53.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.994 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: _finish_auth 0 2023-10-30T12:00:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:00:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac800ea80 con 0x7f4ad409fad0 2023-10-30T12:00:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac800edf0 con 0x7f4ad409fad0 2023-10-30T12:00:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad0ff9640 10 monclient: dump: 2023-10-30T12:00:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:53.997 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ad27fc640 4 set_mon_vals no callback set 2023-10-30T12:00:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ada67c640 5 monclient: authenticate success, global_id 4223 2023-10-30T12:00:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ada67c640 10 monclient: _renew_subs 2023-10-30T12:00:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ada67c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.664+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad4131400 con 0x7f4ad409fad0 2023-10-30T12:00:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7f4ada67c640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7f4ada67c640 10 monclient: _renew_subs 2023-10-30T12:00:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7f4ada67c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad411fa30 con 0x7f4ad409fad0 2023-10-30T12:00:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7f4ada67c640 1 librados: init done 2023-10-30T12:00:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 client.?.objecter ms_handle_connect 0x7fc06c132da0 2023-10-30T12:00:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc06bfff640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c1367c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc05400c3c0 tx=0x7fc05400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054005220 con 0x7fc06c132da0 2023-10-30T12:00:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: dump: 2023-10-30T12:00:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 5 monclient: authenticate success, global_id 4300 2023-10-30T12:00:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 10 monclient: _renew_subs 2023-10-30T12:00:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06c0a18d0 con 0x7fc06c132da0 2023-10-30T12:00:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054005380 con 0x7fc06c132da0 2023-10-30T12:00:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 10 monclient: _renew_subs 2023-10-30T12:00:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06c1393d0 con 0x7fc06c132da0 2023-10-30T12:00:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc07317d640 1 librados: init done 2023-10-30T12:00:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054005220 con 0x7fc06c132da0 2023-10-30T12:00:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc0697fa640 10 monclient: dump: 2023-10-30T12:00:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.009 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7fc06affd640 4 set_mon_vals no callback set 2023-10-30T12:00:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7f4ada67c640 10 client.4223.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.665+0000 7f4ada67c640 15 client.4223.objecter _session_command_op_assign -1 1 2023-10-30T12:00:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ada67c640 20 client.4223.objecter _get_session osd=-1 returning homeless 2023-10-30T12:00:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ada67c640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ada67c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4ad409aa00 con 0x7f4ad409fad0 2023-10-30T12:00:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4ac801c400 con 0x7f4ad409fad0 2023-10-30T12:00:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ac0039340 0x7f4ac003b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f4ac804a780 con 0x7f4ad409fad0 2023-10-30T12:00:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 10 client.4223.objecter ms_dispatch 0x7f4ad4000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 3 client.4223.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 3 client.4223.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 10 client.4223.objecter checking command 1 2023-10-30T12:00:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f4ac003f150 0x7f4ac0041570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 20 client.4223.objecter _get_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 20 client.4223.objecter put_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 15 client.4223.objecter _session_command_op_remove -1 1 2023-10-30T12:00:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 20 client.4223.objecter _get_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 20 client.4223.objecter get_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 15 client.4223.objecter _session_command_op_assign 4 1 2023-10-30T12:00:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 20 client.4223.objecter put_session s=0x7f4ac003ea80 osd=4 4 2023-10-30T12:00:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 10 client.4223.objecter _send_command 1 2023-10-30T12:00:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 --> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ac0041c20 con 0x7f4ac003f150 2023-10-30T12:00:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 20 client.4223.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ac804cd10 con 0x7f4ad409fad0 2023-10-30T12:00:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad0ff9640 10 monclient: handle_get_version_reply finishing 1 version 16 2023-10-30T12:00:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad3fff640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ac0039340 0x7f4ac003b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad3fff640 10 monclient: get_auth_request con 0x7f4ac0039340 auth_method 0 2023-10-30T12:00:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7fc07317d640 10 client.4300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7fc07317d640 15 client.4300.objecter _session_command_op_assign -1 1 2023-10-30T12:00:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7fc07317d640 20 client.4300.objecter _get_session osd=-1 returning homeless 2023-10-30T12:00:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7fc07317d640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7fc07317d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc06c09aa00 con 0x7fc06c132da0 2023-10-30T12:00:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad8c19640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f4ac003f150 0x7f4ac0041570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.666+0000 7f4ad8c19640 10 monclient: get_auth_request con 0x7f4ac003f150 auth_method 0 2023-10-30T12:00:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc054007030 con 0x7fc06c132da0 2023-10-30T12:00:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7f4ad0ff9640 10 client.4223.objecter ms_handle_connect 0x7f4ac0039340 2023-10-30T12:00:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7f4ad3fff640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ac0039340 0x7f4ac003b800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400a8d0 tx=0x7f4ac402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc04c0393b0 0x7fc04c03b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fc0540494c0 con 0x7fc06c132da0 2023-10-30T12:00:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 10 client.4300.objecter ms_dispatch 0x7fc06c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 3 client.4300.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 3 client.4300.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7f4ad0ff9640 10 client.4223.objecter ms_handle_connect 0x7f4ac003f150 2023-10-30T12:00:54.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7f4ad8c19640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f4ac003f150 0x7f4ac0041570 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 10 client.4300.objecter checking command 1 2023-10-30T12:00:54.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fc04c03f1c0 0x7fc04c0415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 20 client.4300.objecter _get_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 20 client.4300.objecter put_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 15 client.4300.objecter _session_command_op_remove -1 1 2023-10-30T12:00:54.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 20 client.4300.objecter _get_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 20 client.4300.objecter get_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 15 client.4300.objecter _session_command_op_assign 3 1 2023-10-30T12:00:54.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 20 client.4300.objecter put_session s=0x7fc04c03eaf0 osd=3 4 2023-10-30T12:00:54.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 10 client.4300.objecter _send_command 1 2023-10-30T12:00:54.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc04c041c90 con 0x7fc04c03f1c0 2023-10-30T12:00:54.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 20 client.4300.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc054005220 con 0x7fc06c132da0 2023-10-30T12:00:54.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 10 monclient: handle_get_version_reply finishing 1 version 16 2023-10-30T12:00:54.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc07171a640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fc04c03f1c0 0x7fc04c0415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc07171a640 10 monclient: get_auth_request con 0x7fc04c03f1c0 auth_method 0 2023-10-30T12:00:54.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc0697fa640 10 client.4300.objecter ms_handle_connect 0x7fc04c03f1c0 2023-10-30T12:00:54.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.667+0000 7fc07171a640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fc04c03f1c0 0x7fc04c0415e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== osd.4 v2:172.21.15.118:6809/4272127357 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4ac0041c20 con 0x7f4ac003f150 2023-10-30T12:00:54.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7f4ad0ff9640 10 client.4223.objecter ms_dispatch 0x7f4ad4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:00:54.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7f4ad0ff9640 10 client.4223.objecter _finish_command 1 = system:0 2023-10-30T12:00:54.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7f4ad0ff9640 20 client.4223.objecter put_session s=0x7f4ac003ea80 osd=4 4 2023-10-30T12:00:54.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7f4ad0ff9640 15 client.4223.objecter _session_command_op_remove 4 1 2023-10-30T12:00:54.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== osd.3 v2:172.21.15.118:6808/1009584072 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc04c041c90 con 0x7fc04c03f1c0 2023-10-30T12:00:54.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7fc0697fa640 10 client.4300.objecter ms_dispatch 0x7fc06c000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:00:54.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7fc0697fa640 10 client.4300.objecter _finish_command 1 = system:0 2023-10-30T12:00:54.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7fc0697fa640 20 client.4300.objecter put_session s=0x7fc04c03eaf0 osd=3 4 2023-10-30T12:00:54.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.668+0000 7fc0697fa640 15 client.4300.objecter _session_command_op_remove 3 1 2023-10-30T12:00:54.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.677+0000 7f7cbff73640 10 client.4282.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:00:54.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 15 client.4282.objecter _session_command_op_assign -1 2 2023-10-30T12:00:54.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 client.4282.objecter _get_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:54.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 client.4282.objecter put_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:54.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 client.4282.objecter _get_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:54.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 15 client.4282.objecter _session_command_op_remove -1 2 2023-10-30T12:00:54.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 client.4282.objecter get_session s=0x7f7c9c03ea30 osd=0 3 2023-10-30T12:00:54.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 15 client.4282.objecter _session_command_op_assign 0 2 2023-10-30T12:00:54.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 client.4282.objecter put_session s=0x7f7c9c03ea30 osd=0 4 2023-10-30T12:00:54.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 10 client.4282.objecter _send_command 2 2023-10-30T12:00:54.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 --> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7cb809a5c0 con 0x7f7c9c03f100 2023-10-30T12:00:54.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cae7fc640 1 -- 172.21.15.50:0/1296179123 <== osd.0 v2:172.21.15.50:6816/47190190 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7ca8031ef0 con 0x7f7c9c03f100 2023-10-30T12:00:54.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cae7fc640 10 client.4282.objecter ms_dispatch 0x7f7cb8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:00:54.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cae7fc640 10 client.4282.objecter _finish_command 2 = system:0 2023-10-30T12:00:54.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cae7fc640 20 client.4282.objecter put_session s=0x7f7c9c03ea30 osd=0 4 2023-10-30T12:00:54.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cae7fc640 15 client.4282.objecter _session_command_op_remove 0 2 2023-10-30T12:00:54.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 10 librados: watch_flush enter 2023-10-30T12:00:54.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 10 librados: watch_flush exit 2023-10-30T12:00:54.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 10 client.4282.objecter close_session for osd.0 2023-10-30T12:00:54.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7f7c9c03f100 msgr2=0x7f7c9c041520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7f7c9c03f100 0x7f7c9c041520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 client.4282.objecter put_session s=0x7f7c9c03ea30 osd=0 1 2023-10-30T12:00:54.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 client.4282.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 10 client.4282.objecter successfully canceled tick 2023-10-30T12:00:54.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c9c0392f0 msgr2=0x7f7c9c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c9c0392f0 0x7f7c9c03b7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40043e0 tx=0x7f7cb4007ef0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 10 monclient: shutdown 2023-10-30T12:00:54.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb813af40 msgr2=0x7f7cb81370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 --2- 172.21.15.50:0/1296179123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb813af40 0x7f7cb81370c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8072170 tx=0x7f7cb0009ed0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbe510640 1 -- 172.21.15.50:0/1296179123 reap_dead start 2023-10-30T12:00:54.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 shutdown_connections 2023-10-30T12:00:54.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 >> 172.21.15.50:0/1296179123 conn(0x7f7cb80a80b0 msgr2=0x7f7cb809b490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7fc070f19640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc04c0393b0 0x7fc04c03b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 shutdown_connections 2023-10-30T12:00:54.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.678+0000 7fc070f19640 10 monclient: get_auth_request con 0x7fc04c0393b0 auth_method 0 2023-10-30T12:00:54.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.679+0000 7f7cbff73640 1 -- 172.21.15.50:0/1296179123 wait complete. 2023-10-30T12:00:54.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.679+0000 7f7cbff73640 1 librados: shutdown 2023-10-30T12:00:54.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.679+0000 7fc0697fa640 10 client.4300.objecter ms_handle_connect 0x7fc04c0393b0 2023-10-30T12:00:54.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.679+0000 7fc070f19640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc04c0393b0 0x7fc04c03b870 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc0640011e0 tx=0x7fc064009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 10 client.4223.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:00:54.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 15 client.4223.objecter _session_command_op_assign -1 2 2023-10-30T12:00:54.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 20 client.4223.objecter _get_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 20 client.4223.objecter put_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 20 client.4223.objecter _get_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 15 client.4223.objecter _session_command_op_remove -1 2 2023-10-30T12:00:54.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 20 client.4223.objecter get_session s=0x7f4ac003ea80 osd=4 3 2023-10-30T12:00:54.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 15 client.4223.objecter _session_command_op_assign 4 2 2023-10-30T12:00:54.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 20 client.4223.objecter put_session s=0x7f4ac003ea80 osd=4 4 2023-10-30T12:00:54.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 10 client.4223.objecter _send_command 2 2023-10-30T12:00:54.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.682+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 --> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4ad409a720 con 0x7f4ac003f150 2023-10-30T12:00:54.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ad0ff9640 1 -- 172.21.15.50:0/3618028076 <== osd.4 v2:172.21.15.118:6809/4272127357 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4acc0095c0 con 0x7f4ac003f150 2023-10-30T12:00:54.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ad0ff9640 10 client.4223.objecter ms_dispatch 0x7f4ad4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:00:54.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ad0ff9640 10 client.4223.objecter _finish_command 2 = system:0 2023-10-30T12:00:54.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ad0ff9640 20 client.4223.objecter put_session s=0x7f4ac003ea80 osd=4 4 2023-10-30T12:00:54.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ad0ff9640 15 client.4223.objecter _session_command_op_remove 4 2 2023-10-30T12:00:54.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 10 librados: watch_flush enter 2023-10-30T12:00:54.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 10 librados: watch_flush exit 2023-10-30T12:00:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 10 client.4223.objecter close_session for osd.4 2023-10-30T12:00:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f4ac003f150 msgr2=0x7f4ac0041570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f4ac003f150 0x7f4ac0041570 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 20 client.4223.objecter put_session s=0x7f4ac003ea80 osd=4 1 2023-10-30T12:00:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 20 client.4223.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 10 client.4223.objecter successfully canceled tick 2023-10-30T12:00:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ac0039340 msgr2=0x7f4ac003b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ac0039340 0x7f4ac003b800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400a8d0 tx=0x7f4ac402f000 comp rx=0 tx=0).stop 2023-10-30T12:00:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 10 monclient: shutdown 2023-10-30T12:00:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ad409fad0 msgr2=0x7f4ad409ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 --2- 172.21.15.50:0/3618028076 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ad409fad0 0x7f4ad409ff30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4136790 tx=0x7f4ac8010c20 comp rx=0 tx=0).stop 2023-10-30T12:00:54.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ad8c19640 1 -- 172.21.15.50:0/3618028076 reap_dead start 2023-10-30T12:00:54.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 shutdown_connections 2023-10-30T12:00:54.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 >> 172.21.15.50:0/3618028076 conn(0x7f4ad40a4a10 msgr2=0x7f4ad4069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 shutdown_connections 2023-10-30T12:00:54.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 -- 172.21.15.50:0/3618028076 wait complete. 2023-10-30T12:00:54.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.683+0000 7f4ada67c640 1 librados: shutdown 2023-10-30T12:00:54.049 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:54.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:00:54.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 10 client.4220.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:00:54.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 15 client.4220.objecter _session_command_op_assign -1 2 2023-10-30T12:00:54.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 20 client.4220.objecter _get_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:54.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 20 client.4220.objecter put_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:54.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 20 client.4220.objecter _get_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:54.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 15 client.4220.objecter _session_command_op_remove -1 2 2023-10-30T12:00:54.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 20 client.4220.objecter get_session s=0x7fcebc03ea30 osd=5 3 2023-10-30T12:00:54.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 15 client.4220.objecter _session_command_op_assign 5 2 2023-10-30T12:00:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 20 client.4220.objecter put_session s=0x7fcebc03ea30 osd=5 4 2023-10-30T12:00:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 10 client.4220.objecter _send_command 2 2023-10-30T12:00:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 --> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fced809a720 con 0x7fcebc03f100 2023-10-30T12:00:54.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcece7fc640 1 -- 172.21.15.50:0/3686367470 <== osd.5 v2:172.21.15.118:6800/1399177807 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fced0001f10 con 0x7fcebc03f100 2023-10-30T12:00:54.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcece7fc640 10 client.4220.objecter ms_dispatch 0x7fced8000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:00:54.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcece7fc640 10 client.4220.objecter _finish_command 2 = system:0 2023-10-30T12:00:54.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcece7fc640 20 client.4220.objecter put_session s=0x7fcebc03ea30 osd=5 4 2023-10-30T12:00:54.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.692+0000 7fcece7fc640 15 client.4220.objecter _session_command_op_remove 5 2 2023-10-30T12:00:54.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 10 librados: watch_flush enter 2023-10-30T12:00:54.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 10 librados: watch_flush exit 2023-10-30T12:00:54.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 10 client.4220.objecter close_session for osd.5 2023-10-30T12:00:54.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7fcebc03f100 msgr2=0x7fcebc041520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7fcebc03f100 0x7fcebc041520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 20 client.4220.objecter put_session s=0x7fcebc03ea30 osd=5 1 2023-10-30T12:00:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 20 client.4220.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 10 client.4220.objecter successfully canceled tick 2023-10-30T12:00:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcebc0392f0 msgr2=0x7fcebc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcebc0392f0 0x7fcebc03b7b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fced8131520 tx=0x7fcec8007c80 comp rx=0 tx=0).stop 2023-10-30T12:00:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 10 monclient: shutdown 2023-10-30T12:00:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fced80a8010 msgr2=0x7fced81ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 --2- 172.21.15.50:0/3686367470 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fced80a8010 0x7fced81ca830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fced4009870 tx=0x7fced40046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcede67e640 1 -- 172.21.15.50:0/3686367470 reap_dead start 2023-10-30T12:00:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 shutdown_connections 2023-10-30T12:00:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 >> 172.21.15.50:0/3686367470 conn(0x7fced80a4a10 msgr2=0x7fced8135db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 shutdown_connections 2023-10-30T12:00:54.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 -- 172.21.15.50:0/3686367470 wait complete. 2023-10-30T12:00:54.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.693+0000 7fcee00e1640 1 librados: shutdown 2023-10-30T12:00:54.059 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:54.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:00:54.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 10 client.4300.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-10-30T12:00:54.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 15 client.4300.objecter _session_command_op_assign -1 2 2023-10-30T12:00:54.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 20 client.4300.objecter _get_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 20 client.4300.objecter put_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 20 client.4300.objecter _get_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 15 client.4300.objecter _session_command_op_remove -1 2 2023-10-30T12:00:54.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 20 client.4300.objecter get_session s=0x7fc04c03eaf0 osd=3 3 2023-10-30T12:00:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 15 client.4300.objecter _session_command_op_assign 3 2 2023-10-30T12:00:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 20 client.4300.objecter put_session s=0x7fc04c03eaf0 osd=3 4 2023-10-30T12:00:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 10 client.4300.objecter _send_command 2 2023-10-30T12:00:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.699+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc06c09a720 con 0x7fc04c03f1c0 2023-10-30T12:00:54.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc0697fa640 1 -- 172.21.15.50:0/3091845062 <== osd.3 v2:172.21.15.118:6808/1009584072 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc05c034680 con 0x7fc04c03f1c0 2023-10-30T12:00:54.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc0697fa640 10 client.4300.objecter ms_dispatch 0x7fc06c000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:00:54.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc0697fa640 10 client.4300.objecter _finish_command 2 = system:0 2023-10-30T12:00:54.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc0697fa640 20 client.4300.objecter put_session s=0x7fc04c03eaf0 osd=3 4 2023-10-30T12:00:54.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc0697fa640 15 client.4300.objecter _session_command_op_remove 3 2 2023-10-30T12:00:54.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 10 librados: watch_flush enter 2023-10-30T12:00:54.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 10 librados: watch_flush exit 2023-10-30T12:00:54.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 10 client.4300.objecter close_session for osd.3 2023-10-30T12:00:54.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fc04c03f1c0 msgr2=0x7fc04c0415e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fc04c03f1c0 0x7fc04c0415e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 20 client.4300.objecter put_session s=0x7fc04c03eaf0 osd=3 1 2023-10-30T12:00:54.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 20 client.4300.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 10 client.4300.objecter successfully canceled tick 2023-10-30T12:00:54.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc04c0393b0 msgr2=0x7fc04c03b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc04c0393b0 0x7fc04c03b870 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc0640011e0 tx=0x7fc064009040 comp rx=0 tx=0).stop 2023-10-30T12:00:54.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 10 monclient: shutdown 2023-10-30T12:00:54.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 msgr2=0x7fc06c1367c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 --2- 172.21.15.50:0/3091845062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc06c132da0 0x7fc06c1367c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc05400c3c0 tx=0x7fc05400c880 comp rx=0 tx=0).stop 2023-10-30T12:00:54.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07171a640 1 -- 172.21.15.50:0/3091845062 reap_dead start 2023-10-30T12:00:54.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 shutdown_connections 2023-10-30T12:00:54.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 >> 172.21.15.50:0/3091845062 conn(0x7fc06c0a4a10 msgr2=0x7fc06c0a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 shutdown_connections 2023-10-30T12:00:54.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 -- 172.21.15.50:0/3091845062 wait complete. 2023-10-30T12:00:54.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:53.700+0000 7fc07317d640 1 librados: shutdown 2023-10-30T12:00:54.071 INFO:teuthology.orchestra.run.smithi050.stdout:47244640258 2023-10-30T12:00:54.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:00:54.072 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:54.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:00:54.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.214+0000 7f1a85ce5640 10 monclient: get_monmap_and_config 2023-10-30T12:00:54.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.214+0000 7f1a85ce5640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.214+0000 7f1a85ce5640 10 monclient: monmap: 2023-10-30T12:00:54.216 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:54.215119+0000 2023-10-30T12:00:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:54.215119+0000 2023-10-30T12:00:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.218 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:54.218 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:54.218 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:54.218 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:54.218 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:54.219 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.214+0000 7f1a85ce5640 1 Processor -- start 2023-10-30T12:00:54.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 1 -- start start 2023-10-30T12:00:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 10 monclient: init 2023-10-30T12:00:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-10-30T12:00:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a800a8010 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80099a40 0x7f1a8013aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 10 monclient(hunting): picked mon.noname-b con 0x7f1a80099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a80132da0 0x7f1a8013cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a85ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f1a80132da0 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a7effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80099a40 0x7f1a8013aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a7effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80099a40 0x7f1a8013aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35668/0 (socket says 172.21.15.50:35668) 2023-10-30T12:00:54.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a7ffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a80132da0 0x7f1a8013cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.215+0000 7f1a7effd640 1 -- 172.21.15.50:0/1085791947 learned_addr learned my addr 172.21.15.50:0/1085791947 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8009a210 con 0x7f1a800a8010 2023-10-30T12:00:54.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a85ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a800a7190 con 0x7f1a80132da0 2023-10-30T12:00:54.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a85ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a800a83f0 con 0x7f1a80099a40 2023-10-30T12:00:54.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a85ce5640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a85ce5640 10 monclient(hunting): authenticate will time out at 899.399044s 2023-10-30T12:00:54.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): get_auth_request con 0x7f1a80099a40 auth_method 0 2023-10-30T12:00:54.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 10 monclient(hunting): get_auth_request con 0x7f1a80132da0 auth_method 0 2023-10-30T12:00:54.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-10-30T12:00:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 1 -- 172.21.15.50:0/1085791947 >> v2:172.21.15.50:3300/0 conn(0x7f1a80132da0 msgr2=0x7f1a8013cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 1 --2- 172.21.15.50:0/1085791947 >> v2:172.21.15.50:3300/0 conn(0x7f1a80132da0 0x7f1a8013cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 1 -- 172.21.15.50:0/1085791947 >> v1:172.21.15.118:6789/0 conn(0x7f1a800a8010 legacy=0x7f1a80132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 1 monclient: found mon.noname-b 2023-10-30T12:00:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7effd640 1 -- 172.21.15.50:0/1085791947 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a800a7600 con 0x7f1a80099a40 2023-10-30T12:00:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.216+0000 7f1a7ffff640 1 --2- 172.21.15.50:0/1085791947 >> v2:172.21.15.50:3300/0 conn(0x7f1a80132da0 0x7f1a8013cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:54.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7effd640 1 --2- 172.21.15.50:0/1085791947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80099a40 0x7f1a8013aa10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009870 tx=0x7f1a7402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8223bcb5e7e7bdff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7fdbcfe61640 10 monclient: get_monmap_and_config 2023-10-30T12:00:54.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/1085791947 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a740317c0 con 0x7f1a80099a40 2023-10-30T12:00:54.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:54.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: dump: 2023-10-30T12:00:54.233 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.233 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.233 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.233 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.233 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.234 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.234 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.234 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.234 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.234 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a85ce5640 5 monclient: authenticate success, global_id 4229 2023-10-30T12:00:54.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a85ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:54.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7fdbcfe61640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7fdbcfe61640 10 monclient: monmap: 2023-10-30T12:00:54.236 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.236 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.236 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:54.218829+0000 2023-10-30T12:00:54.237 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:54.218829+0000 2023-10-30T12:00:54.237 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.237 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.237 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:54.237 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:54.238 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:54.238 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:54.238 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:54.238 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/1085791947 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74031920 con 0x7f1a80099a40 2023-10-30T12:00:54.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/1085791947 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74031cb0 con 0x7f1a80099a40 2023-10-30T12:00:54.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:54.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.217+0000 7f1a7e7fc640 10 monclient: dump: 2023-10-30T12:00:54.240 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.240 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.240 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.241 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.241 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.241 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.241 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.241 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.242 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.242 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.218+0000 7f1a85ce5640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:54.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.218+0000 7f1a85ce5640 4 set_mon_vals no callback set 2023-10-30T12:00:54.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.218+0000 7fdbcfe61640 1 Processor -- start 2023-10-30T12:00:54.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.218+0000 7fdbcfe61640 1 -- start start 2023-10-30T12:00:54.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.218+0000 7fdbcfe61640 10 monclient: init 2023-10-30T12:00:54.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 10 monclient: shutdown 2023-10-30T12:00:54.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80099a40 msgr2=0x7f1a8013aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/1085791947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80099a40 0x7f1a8013aa10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009870 tx=0x7f1a7402b860 comp rx=0 tx=0).stop 2023-10-30T12:00:54.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 shutdown_connections 2023-10-30T12:00:54.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/1085791947 >> v2:172.21.15.50:3300/0 conn(0x7f1a80132da0 0x7f1a8013cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/1085791947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80099a40 0x7f1a8013aa10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 >> 172.21.15.50:0/1085791947 conn(0x7f1a800a4a10 msgr2=0x7f1a800a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 shutdown_connections 2023-10-30T12:00:54.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:00:54.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc80a8010 0x7fdbc8132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc80a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7f1a85ce5640 1 -- 172.21.15.50:0/1085791947 wait complete. 2023-10-30T12:00:54.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdbc8099a40 0x7fdbc8132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc8099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:54.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbc81356d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:54.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc80a7190 con 0x7fdbc81356d0 2023-10-30T12:00:54.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc8133110 con 0x7fdbc8099a40 2023-10-30T12:00:54.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80a83f0 con 0x7fdbc80a8010 2023-10-30T12:00:54.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcdbfd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc80a8010 0x7fdbc8132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcdbfd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc80a8010 0x7fdbc8132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35672/0 (socket says 172.21.15.50:35672) 2023-10-30T12:00:54.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcdbfd640 1 -- 172.21.15.50:0/4017316138 learned_addr learned my addr 172.21.15.50:0/4017316138 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcfe61640 10 monclient(hunting): authenticate will time out at 899.402956s 2023-10-30T12:00:54.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcdbfd640 10 monclient(hunting): get_auth_request con 0x7fdbc80a8010 auth_method 0 2023-10-30T12:00:54.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcdbfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcdbfd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcdbfd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.219+0000 7fdbcd3fc640 1 --2- 172.21.15.50:0/4017316138 >> v2:172.21.15.118:3300/0 conn(0x7fdbc8099a40 0x7fdbc8132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): get_auth_request con 0x7fdbc8099a40 auth_method 0 2023-10-30T12:00:54.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcdbfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcdbfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcdbfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2023-10-30T12:00:54.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 1 -- 172.21.15.50:0/4017316138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc80a8010 msgr2=0x7fdbc8132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:54.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 1 --2- 172.21.15.50:0/4017316138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc80a8010 0x7fdbc8132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 1 -- 172.21.15.50:0/4017316138 >> v1:172.21.15.118:6789/0 conn(0x7fdbc81356d0 legacy=0x7fdbc8135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 1 monclient: found mon.noname-c 2023-10-30T12:00:54.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcd3fc640 1 -- 172.21.15.50:0/4017316138 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80a7600 con 0x7fdbc8099a40 2023-10-30T12:00:54.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7fdbcdbfd640 1 --2- 172.21.15.50:0/4017316138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc80a8010 0x7fdbc8132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:54.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7f1a85ce5640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7f1a85ce5640 10 monclient: monmap: 2023-10-30T12:00:54.257 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.257 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:54.257 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:54.257 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:54.257 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.258 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.258 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:54.258 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:54.258 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:54.258 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7f1a85ce5640 1 librados: starting msgr at 2023-10-30T12:00:54.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.220+0000 7f1a85ce5640 1 librados: starting objecter 2023-10-30T12:00:54.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 1 Processor -- start 2023-10-30T12:00:54.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbcd3fc640 1 --2- 172.21.15.50:0/4017316138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc8099a40 0x7fdbc8132bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80097a0 tx=0x7fdbb802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0ac1022be01a798 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 1 -- start start 2023-10-30T12:00:54.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 1 librados: setting wanted keys 2023-10-30T12:00:54.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 1 librados: calling monclient init 2023-10-30T12:00:54.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 1 -- 172.21.15.50:0/4017316138 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80316f0 con 0x7fdbc8099a40 2023-10-30T12:00:54.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: dump: 2023-10-30T12:00:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.264 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 1 -- 172.21.15.50:0/4017316138 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb8031850 con 0x7fdbc8099a40 2023-10-30T12:00:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 1 -- 172.21.15.50:0/4017316138 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8031be0 con 0x7fdbc8099a40 2023-10-30T12:00:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbccbfb640 10 monclient: dump: 2023-10-30T12:00:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.267 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.267 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.267 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.267 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.267 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 10 monclient: init 2023-10-30T12:00:54.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbcfe61640 5 monclient: authenticate success, global_id 4309 2023-10-30T12:00:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbcfe61640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7fdbcfe61640 4 set_mon_vals no callback set 2023-10-30T12:00:54.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:54.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a800a8010 0x7f1a80131430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 10 monclient(hunting): picked mon.noname-b con 0x7f1a800a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.221+0000 7f1a85ce5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80132da0 0x7f1a80131970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 10 monclient(hunting): picked mon.noname-c con 0x7f1a80132da0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80132da0 0x7f1a80131970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80132da0 0x7f1a80131970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35680/0 (socket says 172.21.15.50:35680) 2023-10-30T12:00:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8009af20 0x7f1a8009b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 1 -- 172.21.15.50:0/797216654 learned_addr learned my addr 172.21.15.50:0/797216654 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a800a8010 0x7f1a80131430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8009af20 0x7f1a8009b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f1a8009af20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8009d0f0 con 0x7f1a8009af20 2023-10-30T12:00:54.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8009e5a0 con 0x7f1a800a8010 2023-10-30T12:00:54.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8009e7f0 con 0x7f1a80132da0 2023-10-30T12:00:54.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a85ce5640 10 monclient(hunting): authenticate will time out at 899.405445s 2023-10-30T12:00:54.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 10 monclient(hunting): get_auth_request con 0x7f1a80132da0 auth_method 0 2023-10-30T12:00:54.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): get_auth_request con 0x7f1a800a8010 auth_method 0 2023-10-30T12:00:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 10 monclient(hunting): get_auth_request con 0x7f1a8009af20 auth_method 0 2023-10-30T12:00:54.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 10 monclient: shutdown 2023-10-30T12:00:54.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 1 -- 172.21.15.50:0/4017316138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc8099a40 msgr2=0x7fdbc8132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/4017316138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc8099a40 0x7fdbc8132bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80097a0 tx=0x7fdbb802b790 comp rx=0 tx=0).stop 2023-10-30T12:00:54.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 1 -- 172.21.15.50:0/4017316138 shutdown_connections 2023-10-30T12:00:54.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/4017316138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc8099a40 0x7fdbc8132bd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/4017316138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc80a8010 0x7fdbc8132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7fdbcfe61640 1 -- 172.21.15.50:0/4017316138 >> 172.21.15.50:0/4017316138 conn(0x7fdbc80a4a10 msgr2=0x7fdbc80a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2023-10-30T12:00:54.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 1 -- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80132da0 msgr2=0x7f1a80131970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80132da0 0x7f1a80131970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 1 -- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8009af20 msgr2=0x7f1a8009b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8009af20 0x7f1a8009b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 1 monclient: found mon.noname-b 2023-10-30T12:00:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.222+0000 7f1a7f7fe640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a78009a70 con 0x7f1a800a8010 2023-10-30T12:00:54.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a7ffff640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8009af20 0x7f1a8009b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:54.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7fdbcfe61640 1 -- 172.21.15.50:0/4017316138 shutdown_connections 2023-10-30T12:00:54.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a7f7fe640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a800a8010 0x7f1a80131430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000c390 tx=0x7f1a7000c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 client.?.objecter ms_handle_connect 0x7f1a800a8010 2023-10-30T12:00:54.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:54.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7fdbcfe61640 1 -- 172.21.15.50:0/4017316138 wait complete. 2023-10-30T12:00:54.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 1 -- 172.21.15.50:0/797216654 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a700052d0 con 0x7f1a800a8010 2023-10-30T12:00:54.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:54.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: dump: 2023-10-30T12:00:54.286 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.286 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.287 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.287 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.287 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.287 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.287 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.287 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.288 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.288 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 1 -- 172.21.15.50:0/797216654 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a70005430 con 0x7f1a800a8010 2023-10-30T12:00:54.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 5 monclient: authenticate success, global_id 4315 2023-10-30T12:00:54.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 10 monclient: _renew_subs 2023-10-30T12:00:54.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a740094d0 con 0x7f1a800a8010 2023-10-30T12:00:54.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 1 -- 172.21.15.50:0/797216654 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a70003830 con 0x7f1a800a8010 2023-10-30T12:00:54.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:54.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a67fff640 10 monclient: dump: 2023-10-30T12:00:54.290 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.291 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.291 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.291 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.291 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.291 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.291 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.292 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.292 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.292 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 10 monclient: _renew_subs 2023-10-30T12:00:54.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a7d7fa640 4 set_mon_vals no callback set 2023-10-30T12:00:54.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8013cab0 con 0x7f1a800a8010 2023-10-30T12:00:54.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.223+0000 7f1a85ce5640 1 librados: init done 2023-10-30T12:00:54.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.224+0000 7fdbcfe61640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.224+0000 7fdbcfe61640 10 monclient: monmap: 2023-10-30T12:00:54.294 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.294 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:54.296 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:54.296 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.224+0000 7fdbcfe61640 1 librados: starting msgr at 2023-10-30T12:00:54.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.224+0000 7fdbcfe61640 1 librados: starting objecter 2023-10-30T12:00:54.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.224+0000 7f1a67fff640 1 -- 172.21.15.50:0/797216654 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1a70003990 con 0x7f1a800a8010 2023-10-30T12:00:54.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 Processor -- start 2023-10-30T12:00:54.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a67fff640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a85ce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a85ce5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a85ce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a67fff640 1 -- 172.21.15.50:0/797216654 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f1a70049980 con 0x7f1a800a8010 2023-10-30T12:00:54.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a67fff640 10 client.4315.objecter ms_dispatch 0x7f1a80000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a67fff640 3 client.4315.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a67fff640 3 client.4315.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8009e3e0 con 0x7f1a800a8010 2023-10-30T12:00:54.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 -- start start 2023-10-30T12:00:54.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 librados: setting wanted keys 2023-10-30T12:00:54.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 librados: calling monclient init 2023-10-30T12:00:54.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a67fff640 20 client.4315.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient: init 2023-10-30T12:00:54.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a7effd640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7f1a7effd640 10 monclient: get_auth_request con 0x7f1a6c0392f0 auth_method 0 2023-10-30T12:00:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc80a8010 0x7fdbc81ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc80a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc81356d0 0x7fdbc81caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc81356d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:54.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc811e600 0x7fdbc811ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc811e600 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8121d80 con 0x7fdbc81356d0 2023-10-30T12:00:54.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8122420 con 0x7fdbc80a8010 2023-10-30T12:00:54.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8120f00 con 0x7fdbc811e600 2023-10-30T12:00:54.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.225+0000 7fdbcfe61640 10 monclient(hunting): authenticate will time out at 899.408907s 2023-10-30T12:00:54.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc81356d0 0x7fdbc81caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7f1a7effd640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1a740090b0 tx=0x7f1a7403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7f1a67fff640 10 client.4315.objecter ms_handle_connect 0x7f1a6c0392f0 2023-10-30T12:00:54.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc81356d0 0x7fdbc81caa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38092/0 (socket says 172.21.15.50:38092) 2023-10-30T12:00:54.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 1 -- 172.21.15.50:0/840625749 learned_addr learned my addr 172.21.15.50:0/840625749 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 10 monclient(hunting): get_auth_request con 0x7fdbc81356d0 auth_method 0 2023-10-30T12:00:54.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc811e600 0x7fdbc811ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc80a8010 0x7fdbc81ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): get_auth_request con 0x7fdbc811e600 auth_method 0 2023-10-30T12:00:54.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 10 monclient(hunting): get_auth_request con 0x7fdbc80a8010 auth_method 0 2023-10-30T12:00:54.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcdbfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2023-10-30T12:00:54.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 -- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc80a8010 msgr2=0x7fdbc81ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc80a8010 0x7fdbc81ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 -- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc81356d0 msgr2=0x7fdbc81caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc81356d0 0x7fdbc81caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 monclient: found mon.noname-c 2023-10-30T12:00:54.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 -- 172.21.15.50:0/840625749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40094d0 con 0x7fdbc811e600 2023-10-30T12:00:54.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbcd3fc640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc81356d0 0x7fdbc81caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:54.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbce3fe640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc811e600 0x7fdbc811ea90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdbc000c730 tx=0x7fdbc000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbbe7fc640 10 client.?.objecter ms_handle_connect 0x7fdbc811e600 2023-10-30T12:00:54.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbbe7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:54.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbbe7fc640 1 -- 172.21.15.50:0/840625749 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0006d50 con 0x7fdbc811e600 2023-10-30T12:00:54.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.226+0000 7fdbbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:54.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: dump: 2023-10-30T12:00:54.318 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.318 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.318 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.319 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.319 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.319 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.319 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.319 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.319 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.320 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 1 -- 172.21.15.50:0/840625749 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc0003970 con 0x7fdbc811e600 2023-10-30T12:00:54.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 1 -- 172.21.15.50:0/840625749 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0003d60 con 0x7fdbc811e600 2023-10-30T12:00:54.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:54.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbe7fc640 10 monclient: dump: 2023-10-30T12:00:54.321 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.322 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.322 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.322 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.322 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.322 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.323 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.323 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.323 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.323 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbbffff640 4 set_mon_vals no callback set 2023-10-30T12:00:54.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 5 monclient: authenticate success, global_id 4238 2023-10-30T12:00:54.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 10 monclient: _renew_subs 2023-10-30T12:00:54.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb8009400 con 0x7fdbc811e600 2023-10-30T12:00:54.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 10 monclient: _renew_subs 2023-10-30T12:00:54.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc8134cc0 con 0x7fdbc811e600 2023-10-30T12:00:54.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.227+0000 7fdbcfe61640 1 librados: init done 2023-10-30T12:00:54.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbbe7fc640 1 -- 172.21.15.50:0/840625749 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdbc00052e0 con 0x7fdbc811e600 2023-10-30T12:00:54.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbcfe61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbcfe61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbcfe61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc8122260 con 0x7fdbc811e600 2023-10-30T12:00:54.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbbe7fc640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbac0392f0 0x7fdbac03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbbe7fc640 1 -- 172.21.15.50:0/840625749 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fdbc004a120 con 0x7fdbc811e600 2023-10-30T12:00:54.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbbe7fc640 10 client.4238.objecter ms_dispatch 0x7fdbc8000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbbe7fc640 3 client.4238.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbbe7fc640 3 client.4238.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbbe7fc640 20 client.4238.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbcdbfd640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbac0392f0 0x7fdbac03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.228+0000 7fdbcdbfd640 10 monclient: get_auth_request con 0x7fdbac0392f0 auth_method 0 2023-10-30T12:00:54.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.229+0000 7fdbbe7fc640 10 client.4238.objecter ms_handle_connect 0x7fdbac0392f0 2023-10-30T12:00:54.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.229+0000 7fdbcdbfd640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbac0392f0 0x7fdbac03b7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4006ca0 tx=0x7fdbc4009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.230+0000 7f1a67fff640 1 -- 172.21.15.50:0/797216654 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a8009e3e0 con 0x7f1a800a8010 2023-10-30T12:00:54.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.230+0000 7f1a67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.230+0000 7f1a67fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:54.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.235+0000 7fdbbe7fc640 1 -- 172.21.15.50:0/840625749 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdbc00072c0 con 0x7fdbc811e600 2023-10-30T12:00:54.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.235+0000 7fdbbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.235+0000 7fdbbe7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:54.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.256+0000 7f58659b0640 10 monclient: get_monmap_and_config 2023-10-30T12:00:54.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.256+0000 7f58659b0640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.256+0000 7f58659b0640 10 monclient: monmap: 2023-10-30T12:00:54.332 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.332 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.332 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:54.257766+0000 2023-10-30T12:00:54.332 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:54.257766+0000 2023-10-30T12:00:54.333 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.333 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.333 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:54.333 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:54.333 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:54.334 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.257+0000 7f58659b0640 1 Processor -- start 2023-10-30T12:00:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.257+0000 7f58659b0640 1 -- start start 2023-10-30T12:00:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.257+0000 7f58659b0640 10 monclient: init 2023-10-30T12:00:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.257+0000 7f58659b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.257+0000 7f58659b0640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:00:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.257+0000 7f58659b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58600a8010 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.257+0000 7f58659b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5860099a40 0x7f586013aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 10 monclient(hunting): picked mon.noname-b con 0x7f5860099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5860132da0 0x7f586013cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 10 monclient(hunting): picked mon.noname-a con 0x7f5860132da0 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586009a210 con 0x7f58600a8010 2023-10-30T12:00:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58600a7190 con 0x7f5860132da0 2023-10-30T12:00:54.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58600a83f0 con 0x7f5860099a40 2023-10-30T12:00:54.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f58659b0640 10 monclient(hunting): authenticate will time out at 899.441123s 2023-10-30T12:00:54.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5860099a40 0x7f586013aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585effd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f58600a8010 0x7f5860132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60834/0 (socket says 172.21.15.50:60834) 2023-10-30T12:00:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5860099a40 0x7f586013aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35690/0 (socket says 172.21.15.50:35690) 2023-10-30T12:00:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 -- 172.21.15.50:0/2198116857 learned_addr learned my addr 172.21.15.50:0/2198116857 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): get_auth_request con 0x7f5860099a40 auth_method 0 2023-10-30T12:00:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 1 --2- 172.21.15.50:0/2198116857 >> v2:172.21.15.50:3300/0 conn(0x7f5860132da0 0x7f586013cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 10 monclient(hunting): get_auth_request con 0x7f5860132da0 auth_method 0 2023-10-30T12:00:54.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-10-30T12:00:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 -- 172.21.15.50:0/2198116857 >> v2:172.21.15.50:3300/0 conn(0x7f5860132da0 msgr2=0x7f586013cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 --2- 172.21.15.50:0/2198116857 >> v2:172.21.15.50:3300/0 conn(0x7f5860132da0 0x7f586013cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 -- 172.21.15.50:0/2198116857 >> v1:172.21.15.50:6789/0 conn(0x7f58600a8010 legacy=0x7f5860132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 monclient: found mon.noname-b 2023-10-30T12:00:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.258+0000 7f585e7fc640 1 -- 172.21.15.50:0/2198116857 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600a7600 con 0x7f5860099a40 2023-10-30T12:00:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.259+0000 7f8641083640 10 monclient: get_monmap_and_config 2023-10-30T12:00:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.259+0000 7f8641083640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.259+0000 7f8641083640 10 monclient: monmap: 2023-10-30T12:00:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:54.260317+0000 2023-10-30T12:00:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:54.260317+0000 2023-10-30T12:00:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:54.346 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585e7fc640 1 --2- 172.21.15.50:0/2198116857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5860099a40 0x7f586013aa10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5854009870 tx=0x7f585402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b3e638a8d387b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 1 Processor -- start 2023-10-30T12:00:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 1 -- 172.21.15.50:0/2198116857 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58540317c0 con 0x7f5860099a40 2023-10-30T12:00:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: dump: 2023-10-30T12:00:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.349 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 1 -- 172.21.15.50:0/2198116857 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5854031920 con 0x7f5860099a40 2023-10-30T12:00:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 1 -- 172.21.15.50:0/2198116857 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5854031cb0 con 0x7f5860099a40 2023-10-30T12:00:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f58659b0640 5 monclient: authenticate success, global_id 4241 2023-10-30T12:00:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f58659b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f58659b0640 4 set_mon_vals no callback set 2023-10-30T12:00:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f585dffb640 10 monclient: dump: 2023-10-30T12:00:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.354 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 1 -- start start 2023-10-30T12:00:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient: init 2023-10-30T12:00:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T12:00:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8634029b70 0x7f8634027890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient(hunting): picked mon.noname-c con 0x7f8634029b70 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8634022550 0x7f8634027dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient(hunting): picked mon.noname-b con 0x7f8634022550 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86340bbea0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8634028600 con 0x7f86340bbea0 2023-10-30T12:00:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8634024d50 con 0x7f8634029b70 2023-10-30T12:00:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8634029f50 con 0x7f8634022550 2023-10-30T12:00:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.260+0000 7f8641083640 10 monclient(hunting): authenticate will time out at 899.443961s 2023-10-30T12:00:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 10 monclient: shutdown 2023-10-30T12:00:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 -- 172.21.15.50:0/2198116857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5860099a40 msgr2=0x7f586013aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 --2- 172.21.15.50:0/2198116857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5860099a40 0x7f586013aa10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5854009870 tx=0x7f585402b860 comp rx=0 tx=0).stop 2023-10-30T12:00:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 -- 172.21.15.50:0/2198116857 shutdown_connections 2023-10-30T12:00:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 --2- 172.21.15.50:0/2198116857 >> v2:172.21.15.50:3300/0 conn(0x7f5860132da0 0x7f586013cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 --2- 172.21.15.50:0/2198116857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5860099a40 0x7f586013aa10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 -- 172.21.15.50:0/2198116857 >> 172.21.15.50:0/2198116857 conn(0x7f58600a4a10 msgr2=0x7f58600a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f86340bbea0 0x7f86340bc330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60850/0 (socket says 172.21.15.50:60850) 2023-10-30T12:00:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863bfff640 1 -- 172.21.15.50:0/720566516 learned_addr learned my addr 172.21.15.50:0/720566516 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 1 --2- 172.21.15.50:0/720566516 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8634022550 0x7f8634027dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): get_auth_request con 0x7f8634022550 auth_method 0 2023-10-30T12:00:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863b7fe640 1 --2- 172.21.15.50:0/720566516 >> v2:172.21.15.118:3300/0 conn(0x7f8634029b70 0x7f8634027890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863b7fe640 10 monclient(hunting): get_auth_request con 0x7f8634029b70 auth_method 0 2023-10-30T12:00:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863a7fc640 1 -- 172.21.15.50:0/720566516 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113359101 0 0) 0x7f8630001490 con 0x7f86340bbea0 2023-10-30T12:00:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863a7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:00:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863a7fc640 10 monclient(hunting): my global_id is 4296 2023-10-30T12:00:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863a7fc640 1 -- 172.21.15.50:0/720566516 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8620003520 con 0x7f86340bbea0 2023-10-30T12:00:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 -- 172.21.15.50:0/2198116857 shutdown_connections 2023-10-30T12:00:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f58659b0640 1 -- 172.21.15.50:0/2198116857 wait complete. 2023-10-30T12:00:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2023-10-30T12:00:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 1 -- 172.21.15.50:0/720566516 >> v2:172.21.15.118:3300/0 conn(0x7f8634029b70 msgr2=0x7f8634027890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 1 --2- 172.21.15.50:0/720566516 >> v2:172.21.15.118:3300/0 conn(0x7f8634029b70 0x7f8634027890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 1 -- 172.21.15.50:0/720566516 >> v1:172.21.15.50:6789/0 conn(0x7f86340bbea0 legacy=0x7f86340bc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 1 monclient: found mon.noname-b 2023-10-30T12:00:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.261+0000 7f863affd640 1 -- 172.21.15.50:0/720566516 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634022ae0 con 0x7f8634022550 2023-10-30T12:00:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 10 monclient: monmap: 2023-10-30T12:00:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:54.369 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 1 librados: starting msgr at 2023-10-30T12:00:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 1 librados: starting objecter 2023-10-30T12:00:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 1 Processor -- start 2023-10-30T12:00:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 1 -- start start 2023-10-30T12:00:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 1 librados: setting wanted keys 2023-10-30T12:00:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 1 librados: calling monclient init 2023-10-30T12:00:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f58659b0640 10 monclient: init 2023-10-30T12:00:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.262+0000 7f863affd640 1 --2- 172.21.15.50:0/720566516 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8634022550 0x7f8634027dd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8634022cd0 tx=0x7f862c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ebd1980e6b82c89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 1 -- 172.21.15.50:0/720566516 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c031780 con 0x7f8634022550 2023-10-30T12:00:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: dump: 2023-10-30T12:00:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 1 -- 172.21.15.50:0/720566516 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c0318e0 con 0x7f8634022550 2023-10-30T12:00:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 1 -- 172.21.15.50:0/720566516 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c031c70 con 0x7f8634022550 2023-10-30T12:00:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f863a7fc640 10 monclient: dump: 2023-10-30T12:00:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.377 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:00:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58600a8010 0x7f58601ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 10 monclient(hunting): picked mon.noname-c con 0x7f58600a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5860132da0 0x7f58601cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 10 monclient(hunting): picked mon.noname-b con 0x7f5860132da0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5860135ce0 0x7f5860136140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f585effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58600a8010 0x7f58601ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f585effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58600a8010 0x7f58601ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35708/0 (socket says 172.21.15.50:35708) 2023-10-30T12:00:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5860135ce0 0x7f5860136140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f585e7fc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5860132da0 0x7f58601cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5860135ce0 0x7f5860136140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38116/0 (socket says 172.21.15.50:38116) 2023-10-30T12:00:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f585f7fe640 1 -- 172.21.15.50:0/3530807228 learned_addr learned my addr 172.21.15.50:0/3530807228 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 10 monclient(hunting): picked mon.noname-a con 0x7f5860135ce0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5860139b70 con 0x7f5860135ce0 2023-10-30T12:00:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f58659b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5860138650 con 0x7f5860132da0 2023-10-30T12:00:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f58659b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58600a18d0 con 0x7f58600a8010 2023-10-30T12:00:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f58659b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f58659b0640 10 monclient(hunting): authenticate will time out at 899.447029s 2023-10-30T12:00:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): get_auth_request con 0x7f5860135ce0 auth_method 0 2023-10-30T12:00:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585e7fc640 10 monclient(hunting): get_auth_request con 0x7f5860132da0 auth_method 0 2023-10-30T12:00:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585effd640 10 monclient(hunting): get_auth_request con 0x7f58600a8010 auth_method 0 2023-10-30T12:00:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f8641083640 5 monclient: authenticate success, global_id 4244 2023-10-30T12:00:54.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f8641083640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:54.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.263+0000 7f8641083640 4 set_mon_vals no callback set 2023-10-30T12:00:54.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 10 monclient: shutdown 2023-10-30T12:00:54.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 -- 172.21.15.50:0/720566516 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8634022550 msgr2=0x7f8634027dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 --2- 172.21.15.50:0/720566516 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8634022550 0x7f8634027dd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8634022cd0 tx=0x7f862c02b820 comp rx=0 tx=0).stop 2023-10-30T12:00:54.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 -- 172.21.15.50:0/720566516 shutdown_connections 2023-10-30T12:00:54.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 --2- 172.21.15.50:0/720566516 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8634022550 0x7f8634027dd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 --2- 172.21.15.50:0/720566516 >> v2:172.21.15.118:3300/0 conn(0x7f8634029b70 0x7f8634027890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 -- 172.21.15.50:0/720566516 >> 172.21.15.50:0/720566516 conn(0x7f8634006850 msgr2=0x7f8634006c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 -- 172.21.15.50:0/720566516 shutdown_connections 2023-10-30T12:00:54.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2023-10-30T12:00:54.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 1 -- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58600a8010 msgr2=0x7f58601ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58600a8010 0x7f58601ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 1 -- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5860132da0 msgr2=0x7f58601cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f8641083640 1 -- 172.21.15.50:0/720566516 wait complete. 2023-10-30T12:00:54.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5860132da0 0x7f58601cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f18afa4f640 10 monclient: get_monmap_and_config 2023-10-30T12:00:54.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f18afa4f640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f18afa4f640 10 monclient: monmap: 2023-10-30T12:00:54.391 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.391 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.392 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:54.266388+0000 2023-10-30T12:00:54.392 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:54.266388+0000 2023-10-30T12:00:54.392 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.392 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.392 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:54.393 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:54.393 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:54.393 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:54.393 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:54.393 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f18afa4f640 1 Processor -- start 2023-10-30T12:00:54.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:00:54.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58540094d0 con 0x7f5860135ce0 2023-10-30T12:00:54.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f585f7fe640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5860135ce0 0x7f5860136140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f585800efc0 tx=0x7f585800c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f5847fff640 10 client.?.objecter ms_handle_connect 0x7f5860135ce0 2023-10-30T12:00:54.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f5847fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:54.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f5847fff640 1 -- 172.21.15.50:0/3530807228 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858014480 con 0x7f5860135ce0 2023-10-30T12:00:54.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.264+0000 7f5847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:54.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: dump: 2023-10-30T12:00:54.396 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.396 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.396 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.396 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.397 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.397 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.397 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.397 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.397 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.398 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 1 -- 172.21.15.50:0/3530807228 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585800f040 con 0x7f5860135ce0 2023-10-30T12:00:54.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 5 monclient: authenticate success, global_id 4299 2023-10-30T12:00:54.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 10 monclient: _renew_subs 2023-10-30T12:00:54.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5860139d50 con 0x7f5860135ce0 2023-10-30T12:00:54.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f585d7fa640 4 set_mon_vals no callback set 2023-10-30T12:00:54.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f585effd640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58600a8010 0x7f58601ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:54.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 10 monclient: _renew_subs 2023-10-30T12:00:54.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58601394b0 con 0x7f5860135ce0 2023-10-30T12:00:54.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f58659b0640 1 librados: init done 2023-10-30T12:00:54.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 1 -- 172.21.15.50:0/3530807228 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5858014820 con 0x7f5860135ce0 2023-10-30T12:00:54.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:54.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f5847fff640 10 monclient: dump: 2023-10-30T12:00:54.402 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.402 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.402 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.403 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.403 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f8641083640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f8641083640 10 monclient: monmap: 2023-10-30T12:00:54.404 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.404 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:54.405 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:54.405 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:54.405 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.405 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.405 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:54.406 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:54.406 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:54.406 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f8641083640 1 librados: starting msgr at 2023-10-30T12:00:54.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f8641083640 1 librados: starting objecter 2023-10-30T12:00:54.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.265+0000 7f8641083640 1 Processor -- start 2023-10-30T12:00:54.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f8641083640 1 -- start start 2023-10-30T12:00:54.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f8641083640 1 librados: setting wanted keys 2023-10-30T12:00:54.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f8641083640 1 librados: calling monclient init 2023-10-30T12:00:54.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f8641083640 10 monclient: init 2023-10-30T12:00:54.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f88bee46640 10 monclient: get_monmap_and_config 2023-10-30T12:00:54.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f88bee46640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f88bee46640 10 monclient: monmap: 2023-10-30T12:00:54.408 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.408 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:54.267614+0000 2023-10-30T12:00:54.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:54.267614+0000 2023-10-30T12:00:54.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.409 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:54.409 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:54.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:54.410 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:54.410 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:54.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f58659b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.266+0000 7f58659b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f58659b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f5847fff640 1 -- 172.21.15.50:0/3530807228 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5858014980 con 0x7f5860135ce0 2023-10-30T12:00:54.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586004e160 con 0x7f5860135ce0 2023-10-30T12:00:54.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:54.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8634029b70 0x7f8634151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): picked mon.noname-a con 0x7f8634029b70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:54.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86340bbea0 0x7f8634151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): picked mon.noname-b con 0x7f86340bbea0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:54.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f86340a4dc0 0x7f86340a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): picked mon.noname-c con 0x7f86340a4dc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86340a8be0 con 0x7f8634029b70 2023-10-30T12:00:54.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86340a8e30 con 0x7f86340bbea0 2023-10-30T12:00:54.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86340bbbf0 con 0x7f86340a4dc0 2023-10-30T12:00:54.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f8641083640 10 monclient(hunting): authenticate will time out at 899.450354s 2023-10-30T12:00:54.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f5847fff640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 1 -- start start 2023-10-30T12:00:54.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient: init 2023-10-30T12:00:54.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f86340a4dc0 0x7f86340a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f86340a4dc0 0x7f86340a5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35710/0 (socket says 172.21.15.50:35710) 2023-10-30T12:00:54.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 1 -- 172.21.15.50:0/4142666354 learned_addr learned my addr 172.21.15.50:0/4142666354 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 10 monclient(hunting): get_auth_request con 0x7f86340a4dc0 auth_method 0 2023-10-30T12:00:54.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f5847fff640 1 -- 172.21.15.50:0/3530807228 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f585804b3f0 con 0x7f5860135ce0 2023-10-30T12:00:54.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f863bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f88bee46640 1 Processor -- start 2023-10-30T12:00:54.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f5847fff640 10 client.4299.objecter ms_dispatch 0x7f5860000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f585effd640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f585effd640 10 monclient: get_auth_request con 0x7f584c0392f0 auth_method 0 2023-10-30T12:00:54.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f5847fff640 3 client.4299.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f5847fff640 3 client.4299.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-10-30T12:00:54.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18a809d4f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:54.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f5847fff640 20 client.4299.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18a809abd0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:54.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2023-10-30T12:00:54.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 1 -- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86340bbea0 msgr2=0x7f8634151540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:00:54.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86340bbea0 0x7f8634151540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 1 -- 172.21.15.50:0/4142666354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8634029b70 msgr2=0x7f8634151000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:00:54.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8634029b70 0x7f8634151000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 1 monclient: found mon.noname-c 2023-10-30T12:00:54.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 1 -- 172.21.15.50:0/4142666354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c009400 con 0x7f86340a4dc0 2023-10-30T12:00:54.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a81407e0 0x7f18a8142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient(hunting): picked mon.noname-b con 0x7f18a81407e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a8098a00 con 0x7f18a809abd0 2023-10-30T12:00:54.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a8099ee0 con 0x7f18a809d4f0 2023-10-30T12:00:54.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f863bfff640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f86340a4dc0 0x7f86340a5250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f863000cba0 tx=0x7f863000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f8638ff9640 10 client.?.objecter ms_handle_connect 0x7f86340a4dc0 2023-10-30T12:00:54.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f8638ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a8098380 con 0x7f18a81407e0 2023-10-30T12:00:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f18afa4f640 10 monclient(hunting): authenticate will time out at 899.450982s 2023-10-30T12:00:54.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a81407e0 0x7f18a8142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f88bee46640 1 -- start start 2023-10-30T12:00:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.267+0000 7f88bee46640 10 monclient: init 2023-10-30T12:00:54.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a81407e0 0x7f18a8142bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35712/0 (socket says 172.21.15.50:35712) 2023-10-30T12:00:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 1 -- 172.21.15.50:0/3712944628 learned_addr learned my addr 172.21.15.50:0/3712944628 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): get_auth_request con 0x7f18a81407e0 auth_method 0 2023-10-30T12:00:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2023-10-30T12:00:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 1 -- 172.21.15.50:0/3712944628 >> v1:172.21.15.118:6789/0 conn(0x7f18a809d4f0 legacy=0x7f18a8097560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T12:00:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 1 -- 172.21.15.50:0/3712944628 >> v1:172.21.15.50:6789/0 conn(0x7f18a809abd0 legacy=0x7f18a8097c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:54.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 1 monclient: found mon.noname-b 2023-10-30T12:00:54.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f88bee46640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f88bee46640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:00:54.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f18adfec640 1 -- 172.21.15.50:0/3712944628 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8099c00 con 0x7f18a81407e0 2023-10-30T12:00:54.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f88bee46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88b8070bb0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:54.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f88bee46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88b8139ff0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:54.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f88bee46640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f88b813a420 0x7f88b8142430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f88bee46640 10 monclient(hunting): picked mon.noname-c con 0x7f88b813a420 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:54.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.268+0000 7f88bee46640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b806a770 con 0x7f88b8139ff0 2023-10-30T12:00:54.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bee46640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b806abe0 con 0x7f88b8070bb0 2023-10-30T12:00:54.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bee46640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bee46640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b80a3300 con 0x7f88b813a420 2023-10-30T12:00:54.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bee46640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bee46640 10 monclient(hunting): authenticate will time out at 899.452042s 2023-10-30T12:00:54.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bcbe2640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f88b8070bb0 0x7f88b8070f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:34290/0 (socket says 172.21.15.50:34290) 2023-10-30T12:00:54.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bcbe2640 1 -- 172.21.15.50:0/867057182 learned_addr learned my addr 172.21.15.50:0/867057182 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:54.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 1 --2- 172.21.15.50:0/867057182 >> v2:172.21.15.118:3300/0 conn(0x7f88b813a420 0x7f88b8142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): get_auth_request con 0x7f88b813a420 auth_method 0 2023-10-30T12:00:54.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient(hunting): handle_auth_done global_id 4327 payload 274 2023-10-30T12:00:54.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 1 -- 172.21.15.50:0/867057182 >> v1:172.21.15.118:6789/0 conn(0x7f88b8070bb0 legacy=0x7f88b8070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 1 -- 172.21.15.50:0/867057182 >> v1:172.21.15.50:6789/0 conn(0x7f88b8139ff0 legacy=0x7f88b813ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:54.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f585effd640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5850000f60 tx=0x7f5850009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f5847fff640 10 client.4299.objecter ms_handle_connect 0x7f584c0392f0 2023-10-30T12:00:54.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 1 -- 172.21.15.50:0/4142666354 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863000ad40 con 0x7f86340a4dc0 2023-10-30T12:00:54.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:54.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: dump: 2023-10-30T12:00:54.440 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.440 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.441 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.441 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.441 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.441 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.441 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.441 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.442 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.442 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 1 -- 172.21.15.50:0/4142666354 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8630012400 con 0x7f86340a4dc0 2023-10-30T12:00:54.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8641083640 5 monclient: authenticate success, global_id 4247 2023-10-30T12:00:54.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 1 -- 172.21.15.50:0/4142666354 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863000aea0 con 0x7f86340a4dc0 2023-10-30T12:00:54.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:54.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f8638ff9640 10 monclient: dump: 2023-10-30T12:00:54.444 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.444 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.444 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.444 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.445 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.445 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.445 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.445 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.445 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.445 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f18adfec640 1 --2- 172.21.15.50:0/3712944628 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a81407e0 0x7f18a8142bd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009e80 tx=0x7f18a002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63c203d09e0a7a57 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 1 monclient: found mon.noname-c 2023-10-30T12:00:54.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f88bd3e3640 1 -- 172.21.15.50:0/867057182 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b806de80 con 0x7f88b813a420 2023-10-30T12:00:54.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.269+0000 7f86397fa640 4 set_mon_vals no callback set 2023-10-30T12:00:54.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 10 monclient: _renew_subs 2023-10-30T12:00:54.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86340a7700 con 0x7f86340a4dc0 2023-10-30T12:00:54.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 10 monclient: _renew_subs 2023-10-30T12:00:54.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86340a6a60 con 0x7f86340a4dc0 2023-10-30T12:00:54.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 1 -- 172.21.15.50:0/3712944628 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a002c930 con 0x7f18a81407e0 2023-10-30T12:00:54.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:54.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: dump: 2023-10-30T12:00:54.450 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.451 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.451 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.451 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.451 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.451 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.451 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.452 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.452 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.452 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 1 -- 172.21.15.50:0/3712944628 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a002ca90 con 0x7f18a81407e0 2023-10-30T12:00:54.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 1 -- 172.21.15.50:0/3712944628 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a002ce20 con 0x7f18a81407e0 2023-10-30T12:00:54.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:54.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f189bfff640 10 monclient: dump: 2023-10-30T12:00:54.454 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.454 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.454 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.454 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.454 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.455 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.455 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.455 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.455 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.455 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f18afa4f640 5 monclient: authenticate success, global_id 4250 2023-10-30T12:00:54.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f18afa4f640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:54.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f18afa4f640 4 set_mon_vals no callback set 2023-10-30T12:00:54.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f8641083640 1 librados: init done 2023-10-30T12:00:54.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f88bd3e3640 1 --2- 172.21.15.50:0/867057182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b813a420 0x7f88b8142430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009d90 tx=0x7f88b002bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37090fb10ba6d7ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f88b77fe640 1 -- 172.21.15.50:0/867057182 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b002cb60 con 0x7f88b813a420 2023-10-30T12:00:54.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f88b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f88b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:54.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.270+0000 7f88b77fe640 10 monclient: dump: 2023-10-30T12:00:54.458 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.458 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.458 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.458 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.459 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.459 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.459 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.459 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.459 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.459 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 1 -- 172.21.15.50:0/867057182 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b002ccc0 con 0x7f88b813a420 2023-10-30T12:00:54.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 1 -- 172.21.15.50:0/867057182 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0034640 con 0x7f88b813a420 2023-10-30T12:00:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88b77fe640 10 monclient: dump: 2023-10-30T12:00:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.462 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.462 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.462 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.462 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.462 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.463 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.463 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.463 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88bee46640 5 monclient: authenticate success, global_id 4327 2023-10-30T12:00:54.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88bee46640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:54.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88bee46640 4 set_mon_vals no callback set 2023-10-30T12:00:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f18afa4f640 10 monclient: shutdown 2023-10-30T12:00:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f18afa4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f18afa4f640 1 -- 172.21.15.50:0/3712944628 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a81407e0 msgr2=0x7f18a8142bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f18afa4f640 1 --2- 172.21.15.50:0/3712944628 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a81407e0 0x7f18a8142bd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009e80 tx=0x7f18a002bd90 comp rx=0 tx=0).stop 2023-10-30T12:00:54.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f18afa4f640 1 -- 172.21.15.50:0/3712944628 shutdown_connections 2023-10-30T12:00:54.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f18afa4f640 1 --2- 172.21.15.50:0/3712944628 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a81407e0 0x7f18a8142bd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f18afa4f640 1 -- 172.21.15.50:0/3712944628 >> 172.21.15.50:0/3712944628 conn(0x7f18a80a3980 msgr2=0x7f18a80a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8638ff9640 1 -- 172.21.15.50:0/4142666354 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8630007070 con 0x7f86340a4dc0 2023-10-30T12:00:54.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8638ff9640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f863c0b2ad0 0x7f863c0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8641083640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88bee46640 10 monclient: shutdown 2023-10-30T12:00:54.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88bee46640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88bee46640 1 -- 172.21.15.50:0/867057182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b813a420 msgr2=0x7f88b8142430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f88bee46640 1 --2- 172.21.15.50:0/867057182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b813a420 0x7f88b8142430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009d90 tx=0x7f88b002bdb0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8641083640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8641083640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f18afa4f640 1 -- 172.21.15.50:0/3712944628 shutdown_connections 2023-10-30T12:00:54.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f18afa4f640 1 -- 172.21.15.50:0/3712944628 wait complete. 2023-10-30T12:00:54.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f88bee46640 1 -- 172.21.15.50:0/867057182 shutdown_connections 2023-10-30T12:00:54.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f88bee46640 1 --2- 172.21.15.50:0/867057182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b813a420 0x7f88b8142430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f88bee46640 1 -- 172.21.15.50:0/867057182 >> 172.21.15.50:0/867057182 conn(0x7f88b80a6260 msgr2=0x7f88b80a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8638ff9640 1 -- 172.21.15.50:0/4142666354 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f863004a170 con 0x7f86340a4dc0 2023-10-30T12:00:54.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8638ff9640 10 client.4247.objecter ms_dispatch 0x7f863c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8638ff9640 3 client.4247.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8638ff9640 3 client.4247.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.271+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8634028600 con 0x7f86340a4dc0 2023-10-30T12:00:54.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f8638ff9640 20 client.4247.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f863b7fe640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f863c0b2ad0 0x7f863c0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f863b7fe640 10 monclient: get_auth_request con 0x7f863c0b2ad0 auth_method 0 2023-10-30T12:00:54.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f88bee46640 1 -- 172.21.15.50:0/867057182 shutdown_connections 2023-10-30T12:00:54.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f88bee46640 1 -- 172.21.15.50:0/867057182 wait complete. 2023-10-30T12:00:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f863b7fe640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f863c0b2ad0 0x7f863c0b4f90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8628009870 tx=0x7f8628008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f8638ff9640 10 client.4247.objecter ms_handle_connect 0x7f863c0b2ad0 2023-10-30T12:00:54.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f18afa4f640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.272+0000 7f18afa4f640 10 monclient: monmap: 2023-10-30T12:00:54.476 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.476 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:54.476 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:54.477 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:54.477 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.477 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.477 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:54.477 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:54.477 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:54.478 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f18afa4f640 1 librados: starting msgr at 2023-10-30T12:00:54.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f18afa4f640 1 librados: starting objecter 2023-10-30T12:00:54.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f18afa4f640 1 Processor -- start 2023-10-30T12:00:54.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 10 monclient: build_initial_monmap 2023-10-30T12:00:54.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 10 monclient: monmap: 2023-10-30T12:00:54.479 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:54.479 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:54.479 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:54.481 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f18afa4f640 1 -- start start 2023-10-30T12:00:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f18afa4f640 1 librados: setting wanted keys 2023-10-30T12:00:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f18afa4f640 1 librados: calling monclient init 2023-10-30T12:00:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f18afa4f640 10 monclient: init 2023-10-30T12:00:54.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 1 librados: starting msgr at 2023-10-30T12:00:54.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 1 librados: starting objecter 2023-10-30T12:00:54.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 1 Processor -- start 2023-10-30T12:00:54.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 1 -- start start 2023-10-30T12:00:54.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 1 librados: setting wanted keys 2023-10-30T12:00:54.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 1 librados: calling monclient init 2023-10-30T12:00:54.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.273+0000 7f88bee46640 10 monclient: init 2023-10-30T12:00:54.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:00:54.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a809abd0 0x7f18a81cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): picked mon.noname-c con 0x7f18a809abd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a809abd0 0x7f18a81cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a809abd0 0x7f18a81cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35728/0 (socket says 172.21.15.50:35728) 2023-10-30T12:00:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 1 -- 172.21.15.50:0/1287713945 learned_addr learned my addr 172.21.15.50:0/1287713945 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a809d4f0 0x7f18a8070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): picked mon.noname-b con 0x7f18a809d4f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a81407e0 0x7f18a8070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): picked mon.noname-a con 0x7f18a81407e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:54.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a8129650 con 0x7f18a81407e0 2023-10-30T12:00:54.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a81c58d0 con 0x7f18a809d4f0 2023-10-30T12:00:54.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a812a500 con 0x7f18a809abd0 2023-10-30T12:00:54.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18afa4f640 10 monclient(hunting): authenticate will time out at 899.457384s 2023-10-30T12:00:54.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): get_auth_request con 0x7f18a809abd0 auth_method 0 2023-10-30T12:00:54.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a81407e0 0x7f18a8070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 10 monclient(hunting): get_auth_request con 0x7f18a81407e0 auth_method 0 2023-10-30T12:00:54.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f88bee46640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:54.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f88bee46640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:54.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88b8070bb0 0x7f88b813f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): picked mon.noname-a con 0x7f88b8070bb0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:54.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8139ff0 0x7f88b81ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): picked mon.noname-b con 0x7f88b8139ff0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:54.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88b813a420 0x7f88b81cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18adfec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2023-10-30T12:00:54.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.274+0000 7f18ad7eb640 1 -- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a809d4f0 msgr2=0x7f18a8070430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:00:54.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a809d4f0 0x7f18a8070430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 1 -- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a81407e0 msgr2=0x7f18a8070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): picked mon.noname-c con 0x7f88b813a420 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b813d880 con 0x7f88b8070bb0 2023-10-30T12:00:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b813f270 con 0x7f88b8139ff0 2023-10-30T12:00:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a81407e0 0x7f18a8070970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 1 monclient: found mon.noname-c 2023-10-30T12:00:54.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f18ad7eb640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0009b30 con 0x7f18a809abd0 2023-10-30T12:00:54.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:54.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b8071700 con 0x7f88b813a420 2023-10-30T12:00:54.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:54.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bee46640 10 monclient(hunting): authenticate will time out at 899.458285s 2023-10-30T12:00:54.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88b8070bb0 0x7f88b813f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88b8070bb0 0x7f88b813f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38142/0 (socket says 172.21.15.50:38142) 2023-10-30T12:00:54.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 -- 172.21.15.50:0/2265528722 learned_addr learned my addr 172.21.15.50:0/2265528722 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:54.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): get_auth_request con 0x7f88b8070bb0 auth_method 0 2023-10-30T12:00:54.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:54.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:54.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:54.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:54.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:54.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:54.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-10-30T12:00:54.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 -- 172.21.15.50:0/2265528722 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88b813a420 msgr2=0x7f88b81cf050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:00:54.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88b813a420 0x7f88b81cf050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 -- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8139ff0 msgr2=0x7f88b81ceb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:00:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8139ff0 0x7f88b81ceb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient: _finish_hunting 0 2023-10-30T12:00:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 monclient: found mon.noname-a 2023-10-30T12:00:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.275+0000 7f88bcbe2640 1 -- 172.21.15.50:0/2265528722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b0009a70 con 0x7f88b8070bb0 2023-10-30T12:00:54.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bcbe2640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88b8070bb0 0x7f88b813f600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c4d0 tx=0x7f88a800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 client.?.objecter ms_handle_connect 0x7f88b8070bb0 2023-10-30T12:00:54.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:54.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f18adfec640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a81407e0 0x7f18a8070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:54.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 1 -- 172.21.15.50:0/2265528722 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8007540 con 0x7f88b8070bb0 2023-10-30T12:00:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: dump: 2023-10-30T12:00:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.509 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f18ad7eb640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a809abd0 0x7f18a81cf040 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b470 tx=0x7f18a400b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 1 -- 172.21.15.50:0/2265528722 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a80057a0 con 0x7f88b8070bb0 2023-10-30T12:00:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 1 -- 172.21.15.50:0/2265528722 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8005b10 con 0x7f88b8070bb0 2023-10-30T12:00:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b57fa640 10 monclient: dump: 2023-10-30T12:00:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.513 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 5 monclient: authenticate success, global_id 4305 2023-10-30T12:00:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 10 monclient: _renew_subs 2023-10-30T12:00:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88b67fc640 4 set_mon_vals no callback set 2023-10-30T12:00:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b8142000 con 0x7f88b8070bb0 2023-10-30T12:00:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 10 monclient: _renew_subs 2023-10-30T12:00:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b81cf590 con 0x7f88b8070bb0 2023-10-30T12:00:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f88bee46640 1 librados: init done 2023-10-30T12:00:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f1899ffb640 10 client.?.objecter ms_handle_connect 0x7f18a809abd0 2023-10-30T12:00:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f5847fff640 1 -- 172.21.15.50:0/3530807228 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5858007d90 con 0x7f5860135ce0 2023-10-30T12:00:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f5847fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f5847fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.276+0000 7f1899ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:54.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.277+0000 7f88b57fa640 1 -- 172.21.15.50:0/2265528722 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f88a8005c70 con 0x7f88b8070bb0 2023-10-30T12:00:54.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88bee46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88bee46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88bee46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b804e160 con 0x7f88b8070bb0 2023-10-30T12:00:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b57fa640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f88a40392f0 0x7f88a403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b57fa640 1 -- 172.21.15.50:0/2265528722 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f88a8049940 con 0x7f88b8070bb0 2023-10-30T12:00:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b57fa640 10 client.4305.objecter ms_dispatch 0x7f88b8000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b57fa640 3 client.4305.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b57fa640 3 client.4305.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b57fa640 20 client.4305.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b7fff640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f88a40392f0 0x7f88a403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b7fff640 10 monclient: get_auth_request con 0x7f88a40392f0 auth_method 0 2023-10-30T12:00:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b57fa640 10 client.4305.objecter ms_handle_connect 0x7f88a40392f0 2023-10-30T12:00:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.278+0000 7f88b7fff640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f88a40392f0 0x7f88a403b7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88b813e870 tx=0x7f88ac0079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f8638ff9640 1 -- 172.21.15.50:0/4142666354 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8634028600 con 0x7f86340a4dc0 2023-10-30T12:00:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f8638ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f8638ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 1 -- 172.21.15.50:0/1287713945 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a40059c0 con 0x7f18a809abd0 2023-10-30T12:00:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 10 monclient: dump: 2023-10-30T12:00:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.526 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 1 -- 172.21.15.50:0/1287713945 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a4005b20 con 0x7f18a809abd0 2023-10-30T12:00:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.279+0000 7f1899ffb640 1 -- 172.21.15.50:0/1287713945 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4011690 con 0x7f18a809abd0 2023-10-30T12:00:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.280+0000 7f1899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.280+0000 7f1899ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.280+0000 7f1899ffb640 10 monclient: dump: 2023-10-30T12:00:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:54.530 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.280+0000 7f189affd640 4 set_mon_vals no callback set 2023-10-30T12:00:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.280+0000 7f18afa4f640 5 monclient: authenticate success, global_id 4253 2023-10-30T12:00:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.280+0000 7f18afa4f640 10 monclient: _renew_subs 2023-10-30T12:00:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.280+0000 7f18afa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.281+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a8097240 con 0x7f18a809abd0 2023-10-30T12:00:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.281+0000 7f18afa4f640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.281+0000 7f18afa4f640 10 monclient: _renew_subs 2023-10-30T12:00:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.281+0000 7f18afa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.281+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a812ada0 con 0x7f18a809abd0 2023-10-30T12:00:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.281+0000 7f18afa4f640 1 librados: init done 2023-10-30T12:00:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.282+0000 7f18afa4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.283+0000 7f18afa4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.283+0000 7f18afa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.283+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a809ca90 con 0x7f18a809abd0 2023-10-30T12:00:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.283+0000 7f1899ffb640 1 -- 172.21.15.50:0/1287713945 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f18a40117f0 con 0x7f18a809abd0 2023-10-30T12:00:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.283+0000 7f1899ffb640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f188c039340 0x7f188c03b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f1899ffb640 1 -- 172.21.15.50:0/1287713945 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f18a404a490 con 0x7f18a809abd0 2023-10-30T12:00:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f1899ffb640 10 client.4253.objecter ms_dispatch 0x7f18a8000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f1899ffb640 3 client.4253.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f1899ffb640 3 client.4253.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f1899ffb640 20 client.4253.objecter dump_active .. 0 homeless 2023-10-30T12:00:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f18acfea640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f188c039340 0x7f188c03b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f18acfea640 10 monclient: get_auth_request con 0x7f188c039340 auth_method 0 2023-10-30T12:00:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.284+0000 7f18acfea640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f188c039340 0x7f188c03b800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18a806a490 tx=0x7f189c007990 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.285+0000 7f1899ffb640 10 client.4253.objecter ms_handle_connect 0x7f188c039340 2023-10-30T12:00:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.287+0000 7f88b57fa640 1 -- 172.21.15.50:0/2265528722 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f88a8007e60 con 0x7f88b8070bb0 2023-10-30T12:00:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.288+0000 7f88b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.288+0000 7f88b57fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:54.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.293+0000 7f1899ffb640 1 -- 172.21.15.50:0/1287713945 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f18a400f340 con 0x7f18a809abd0 2023-10-30T12:00:54.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.293+0000 7f1899ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:54.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.293+0000 7f1899ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.433+0000 7fdbcfe61640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:00:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.433+0000 7fdbcfe61640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:00:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.433+0000 7fdbcfe61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.433+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdbc804e160 con 0x7fdbc811e600 2023-10-30T12:00:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.434+0000 7fdbbe7fc640 1 -- 172.21.15.50:0/840625749 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdbc001a3b0 con 0x7fdbc811e600 2023-10-30T12:00:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.434+0000 7fdbbe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:00:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.434+0000 7fdbbe7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:54.539 INFO:teuthology.orchestra.run.smithi050.stdout:47244640257 2023-10-30T12:00:54.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 10 librados: watch_flush enter 2023-10-30T12:00:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 10 librados: watch_flush exit 2023-10-30T12:00:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 20 client.4238.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 10 client.4238.objecter successfully canceled tick 2023-10-30T12:00:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbac0392f0 msgr2=0x7fdbac03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbac0392f0 0x7fdbac03b7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4006ca0 tx=0x7fdbc4009240 comp rx=0 tx=0).stop 2023-10-30T12:00:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 10 monclient: shutdown 2023-10-30T12:00:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc811e600 msgr2=0x7fdbc811ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc811e600 0x7fdbc811ea90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdbc000c730 tx=0x7fdbc000cbf0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 shutdown_connections 2023-10-30T12:00:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbac0392f0 0x7fdbac03b7b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbc811e600 0x7fdbc811ea90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbc81356d0 0x7fdbc81caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 --2- 172.21.15.50:0/840625749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbc80a8010 0x7fdbc81ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.436+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 >> 172.21.15.50:0/840625749 conn(0x7fdbc80a4a10 msgr2=0x7fdbc81332c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.437+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 shutdown_connections 2023-10-30T12:00:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.437+0000 7fdbcfe61640 1 -- 172.21.15.50:0/840625749 wait complete. 2023-10-30T12:00:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.437+0000 7fdbcfe61640 1 librados: shutdown 2023-10-30T12:00:54.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.5 2023-10-30T12:00:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.461+0000 7f8641083640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:00:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.461+0000 7f8641083640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:00:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.461+0000 7f8641083640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.461+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f86340a8f70 con 0x7f86340a4dc0 2023-10-30T12:00:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.461+0000 7f8638ff9640 1 -- 172.21.15.50:0/4142666354 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f86340a8f70 con 0x7f86340a4dc0 2023-10-30T12:00:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.461+0000 7f8638ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:00:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.461+0000 7f8638ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:54.546 INFO:teuthology.orchestra.run.smithi050.stdout:47244640258 2023-10-30T12:00:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 10 librados: watch_flush enter 2023-10-30T12:00:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 10 librados: watch_flush exit 2023-10-30T12:00:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 20 client.4247.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 10 client.4247.objecter successfully canceled tick 2023-10-30T12:00:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f863c0b2ad0 msgr2=0x7f863c0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f863c0b2ad0 0x7f863c0b4f90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8628009870 tx=0x7f8628008040 comp rx=0 tx=0).stop 2023-10-30T12:00:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 10 monclient: shutdown 2023-10-30T12:00:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f86340a4dc0 msgr2=0x7f86340a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f86340a4dc0 0x7f86340a5250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f863000cba0 tx=0x7f863000a430 comp rx=0 tx=0).stop 2023-10-30T12:00:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 shutdown_connections 2023-10-30T12:00:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f863c0b2ad0 0x7f863c0b4f90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f86340a4dc0 0x7f86340a5250 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f86340bbea0 0x7f8634151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 --2- 172.21.15.50:0/4142666354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8634029b70 0x7f8634151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.463+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 >> 172.21.15.50:0/4142666354 conn(0x7f8634006850 msgr2=0x7f86340b9290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.464+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 shutdown_connections 2023-10-30T12:00:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.464+0000 7f8641083640 1 -- 172.21.15.50:0/4142666354 wait complete. 2023-10-30T12:00:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.464+0000 7f8641083640 1 librados: shutdown 2023-10-30T12:00:54.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2023-10-30T12:00:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.605+0000 7f1a85ce5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:00:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.605+0000 7f1a85ce5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:00:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.605+0000 7f1a85ce5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:00:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.605+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1a8009e930 con 0x7f1a800a8010 2023-10-30T12:00:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.606+0000 7f1a67fff640 1 -- 172.21.15.50:0/797216654 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1a8009e930 con 0x7f1a800a8010 2023-10-30T12:00:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.606+0000 7f1a67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-10-30T12:00:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.606+0000 7f1a67fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:54.609 INFO:teuthology.orchestra.run.smithi050.stdout:47244640258 2023-10-30T12:00:54.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 10 librados: watch_flush enter 2023-10-30T12:00:54.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 10 librados: watch_flush exit 2023-10-30T12:00:54.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 20 client.4315.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 10 client.4315.objecter successfully canceled tick 2023-10-30T12:00:54.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 msgr2=0x7f1a6c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1a740090b0 tx=0x7f1a7403d040 comp rx=0 tx=0).stop 2023-10-30T12:00:54.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 10 monclient: shutdown 2023-10-30T12:00:54.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a800a8010 msgr2=0x7f1a80131430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a800a8010 0x7f1a80131430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000c390 tx=0x7f1a7000c790 comp rx=0 tx=0).stop 2023-10-30T12:00:54.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 shutdown_connections 2023-10-30T12:00:54.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8009af20 0x7f1a8009b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a80132da0 0x7f1a80131970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 --2- 172.21.15.50:0/797216654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a800a8010 0x7f1a80131430 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 >> 172.21.15.50:0/797216654 conn(0x7f1a800a4a10 msgr2=0x7f1a80069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 shutdown_connections 2023-10-30T12:00:54.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.608+0000 7f1a85ce5640 1 -- 172.21.15.50:0/797216654 wait complete. 2023-10-30T12:00:54.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.609+0000 7f1a85ce5640 1 librados: shutdown 2023-10-30T12:00:54.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.2 2023-10-30T12:00:54.621 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.645+0000 7f88bee46640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:00:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.645+0000 7f88bee46640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:00:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.645+0000 7f88bee46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.645+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f88b80745a0 con 0x7f88b8070bb0 2023-10-30T12:00:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.645+0000 7f88b57fa640 1 -- 172.21.15.50:0/2265528722 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88a8017020 con 0x7f88b8070bb0 2023-10-30T12:00:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.645+0000 7f88b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-10-30T12:00:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.645+0000 7f88b57fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:54.649 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 10 librados: watch_flush enter 2023-10-30T12:00:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 10 librados: watch_flush exit 2023-10-30T12:00:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 20 client.4305.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 10 client.4305.objecter successfully canceled tick 2023-10-30T12:00:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f88a40392f0 msgr2=0x7f88a403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f88a40392f0 0x7f88a403b7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88b813e870 tx=0x7f88ac0079b0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 10 monclient: shutdown 2023-10-30T12:00:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88b8070bb0 msgr2=0x7f88b813f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88b8070bb0 0x7f88b813f600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c4d0 tx=0x7f88a800c990 comp rx=0 tx=0).stop 2023-10-30T12:00:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 shutdown_connections 2023-10-30T12:00:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f88a40392f0 0x7f88a403b7b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88b813a420 0x7f88b81cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88b8139ff0 0x7f88b81ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 --2- 172.21.15.50:0/2265528722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88b8070bb0 0x7f88b813f600 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 >> 172.21.15.50:0/2265528722 conn(0x7f88b80a6260 msgr2=0x7f88b806f160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 shutdown_connections 2023-10-30T12:00:54.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 -- 172.21.15.50:0/2265528722 wait complete. 2023-10-30T12:00:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.648+0000 7f88bee46640 1 librados: shutdown 2023-10-30T12:00:54.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2023-10-30T12:00:54.660 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.676+0000 7f58659b0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:00:54.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.676+0000 7f58659b0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:00:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.676+0000 7f58659b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.676+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f58600745a0 con 0x7f5860135ce0 2023-10-30T12:00:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.677+0000 7f5847fff640 1 -- 172.21.15.50:0/3530807228 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5858018070 con 0x7f5860135ce0 2023-10-30T12:00:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.677+0000 7f5847fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-10-30T12:00:54.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.677+0000 7f5847fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:54.680 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:54.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 10 librados: watch_flush enter 2023-10-30T12:00:54.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 10 librados: watch_flush exit 2023-10-30T12:00:54.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 20 client.4299.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 10 client.4299.objecter successfully canceled tick 2023-10-30T12:00:54.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 msgr2=0x7f584c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5850000f60 tx=0x7f5850009040 comp rx=0 tx=0).stop 2023-10-30T12:00:54.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 10 monclient: shutdown 2023-10-30T12:00:54.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5860135ce0 msgr2=0x7f5860136140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5860135ce0 0x7f5860136140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f585800efc0 tx=0x7f585800c550 comp rx=0 tx=0).stop 2023-10-30T12:00:54.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 shutdown_connections 2023-10-30T12:00:54.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5860135ce0 0x7f5860136140 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5860132da0 0x7f58601cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 --2- 172.21.15.50:0/3530807228 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58600a8010 0x7f58601ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 >> 172.21.15.50:0/3530807228 conn(0x7f58600a4a10 msgr2=0x7f5860069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 shutdown_connections 2023-10-30T12:00:54.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 -- 172.21.15.50:0/3530807228 wait complete. 2023-10-30T12:00:54.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.679+0000 7f58659b0640 1 librados: shutdown 2023-10-30T12:00:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.686+0000 7f18afa4f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:00:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.686+0000 7f18afa4f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:00:54.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.686+0000 7f18afa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:54.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.686+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f18a804e160 con 0x7f18a809abd0 2023-10-30T12:00:54.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.686+0000 7f1899ffb640 1 -- 172.21.15.50:0/1287713945 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f18a4018b50 con 0x7f18a809abd0 2023-10-30T12:00:54.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.687+0000 7f1899ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-10-30T12:00:54.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.687+0000 7f1899ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:54.689 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:54.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.688+0000 7f18afa4f640 10 librados: watch_flush enter 2023-10-30T12:00:54.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.688+0000 7f18afa4f640 10 librados: watch_flush exit 2023-10-30T12:00:54.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.688+0000 7f18afa4f640 20 client.4253.objecter shutdown clearing up homeless session... 2023-10-30T12:00:54.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.688+0000 7f18afa4f640 10 client.4253.objecter successfully canceled tick 2023-10-30T12:00:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.688+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f188c039340 msgr2=0x7f188c03b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.688+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f188c039340 0x7f188c03b800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18a806a490 tx=0x7f189c007990 comp rx=0 tx=0).stop 2023-10-30T12:00:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 10 monclient: shutdown 2023-10-30T12:00:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:54.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a809abd0 msgr2=0x7f18a81cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a809abd0 0x7f18a81cf040 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b470 tx=0x7f18a400b930 comp rx=0 tx=0).stop 2023-10-30T12:00:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 shutdown_connections 2023-10-30T12:00:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f188c039340 0x7f188c03b800 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a81407e0 0x7f18a8070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a809d4f0 0x7f18a8070430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 --2- 172.21.15.50:0/1287713945 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18a809abd0 0x7f18a81cf040 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:54.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 >> 172.21.15.50:0/1287713945 conn(0x7f18a80a3980 msgr2=0x7f18a80a1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:54.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 shutdown_connections 2023-10-30T12:00:54.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 -- 172.21.15.50:0/1287713945 wait complete. 2023-10-30T12:00:54.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:54.689+0000 7f18afa4f640 1 librados: shutdown 2023-10-30T12:00:54.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-10-30T12:00:54.695 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:54.700 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2023-10-30T12:00:54.701 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:55.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:00:55.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:00:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.663+0000 7fd155bb7640 10 monclient: get_monmap_and_config 2023-10-30T12:00:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.663+0000 7fd155bb7640 10 monclient: build_initial_monmap 2023-10-30T12:00:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.663+0000 7fd155bb7640 10 monclient: monmap: 2023-10-30T12:00:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:55.664372+0000 2023-10-30T12:00:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:55.664372+0000 2023-10-30T12:00:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:55.668 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:55.668 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 1 Processor -- start 2023-10-30T12:00:55.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 1 -- start start 2023-10-30T12:00:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient: init 2023-10-30T12:00:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:00:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd1500a8010 0x7fd150132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient(hunting): picked mon.noname-a con 0x7fd1500a8010 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd150099a40 0x7fd150132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient(hunting): picked mon.noname-c con 0x7fd150099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1501356d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1500a7190 con 0x7fd1501356d0 2023-10-30T12:00:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd150133110 con 0x7fd1500a8010 2023-10-30T12:00:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1500a83f0 con 0x7fd150099a40 2023-10-30T12:00:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd155bb7640 10 monclient(hunting): authenticate will time out at 900.847677s 2023-10-30T12:00:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd1500a8010 0x7fd150132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd1500a8010 0x7fd150132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38148/0 (socket says 172.21.15.50:38148) 2023-10-30T12:00:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 1 -- 172.21.15.50:0/1437385767 learned_addr learned my addr 172.21.15.50:0/1437385767 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 10 monclient(hunting): get_auth_request con 0x7fd1500a8010 auth_method 0 2023-10-30T12:00:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.664+0000 7fd14effd640 1 --2- 172.21.15.50:0/1437385767 >> v2:172.21.15.118:3300/0 conn(0x7fd150099a40 0x7fd150132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14effd640 10 monclient(hunting): get_auth_request con 0x7fd150099a40 auth_method 0 2023-10-30T12:00:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2023-10-30T12:00:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 1 -- 172.21.15.50:0/1437385767 >> v2:172.21.15.118:3300/0 conn(0x7fd150099a40 msgr2=0x7fd150132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 1 --2- 172.21.15.50:0/1437385767 >> v2:172.21.15.118:3300/0 conn(0x7fd150099a40 0x7fd150132bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 1 -- 172.21.15.50:0/1437385767 >> v1:172.21.15.118:6789/0 conn(0x7fd1501356d0 legacy=0x7fd150135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:00:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 1 -- 172.21.15.50:0/1437385767 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1500a7600 con 0x7fd1500a8010 2023-10-30T12:00:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.665+0000 7fd14f7fe640 1 --2- 172.21.15.50:0/1437385767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1500a8010 0x7fd150132690 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1500a18a0 tx=0x7fd14002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378613d328e7b1e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 1 -- 172.21.15.50:0/1437385767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1400317e0 con 0x7fd1500a8010 2023-10-30T12:00:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: dump: 2023-10-30T12:00:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.687 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 1 -- 172.21.15.50:0/1437385767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd140031940 con 0x7fd1500a8010 2023-10-30T12:00:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 5 monclient: authenticate success, global_id 4308 2023-10-30T12:00:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 4 set_mon_vals no callback set 2023-10-30T12:00:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 1 -- 172.21.15.50:0/1437385767 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140031cd0 con 0x7fd1500a8010 2023-10-30T12:00:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd14e7fc640 10 monclient: dump: 2023-10-30T12:00:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.691 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 10 monclient: shutdown 2023-10-30T12:00:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 -- 172.21.15.50:0/1437385767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1500a8010 msgr2=0x7fd150132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 --2- 172.21.15.50:0/1437385767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1500a8010 0x7fd150132690 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1500a18a0 tx=0x7fd14002b880 comp rx=0 tx=0).stop 2023-10-30T12:00:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 -- 172.21.15.50:0/1437385767 shutdown_connections 2023-10-30T12:00:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 --2- 172.21.15.50:0/1437385767 >> v2:172.21.15.118:3300/0 conn(0x7fd150099a40 0x7fd150132bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 --2- 172.21.15.50:0/1437385767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1500a8010 0x7fd150132690 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 -- 172.21.15.50:0/1437385767 >> 172.21.15.50:0/1437385767 conn(0x7fd1500a4a10 msgr2=0x7fd1500a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 -- 172.21.15.50:0/1437385767 shutdown_connections 2023-10-30T12:00:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.666+0000 7fd155bb7640 1 -- 172.21.15.50:0/1437385767 wait complete. 2023-10-30T12:00:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.667+0000 7fd155bb7640 10 monclient: build_initial_monmap 2023-10-30T12:00:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.667+0000 7fd155bb7640 10 monclient: monmap: 2023-10-30T12:00:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:55.696 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.667+0000 7fd155bb7640 1 librados: starting msgr at 2023-10-30T12:00:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.667+0000 7fd155bb7640 1 librados: starting objecter 2023-10-30T12:00:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.667+0000 7fd155bb7640 1 Processor -- start 2023-10-30T12:00:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 -- start start 2023-10-30T12:00:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 librados: setting wanted keys 2023-10-30T12:00:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 librados: calling monclient init 2023-10-30T12:00:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient: init 2023-10-30T12:00:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1500a8010 0x7fd1501ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): picked mon.noname-b con 0x7fd1500a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1501356d0 0x7fd1501cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): picked mon.noname-a con 0x7fd1501356d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd15011e430 0x7fd15011e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): picked mon.noname-c con 0x7fd15011e430 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd150120c70 con 0x7fd1501356d0 2023-10-30T12:00:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd150121db0 con 0x7fd1500a8010 2023-10-30T12:00:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1500a18d0 con 0x7fd15011e430 2023-10-30T12:00:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd155bb7640 10 monclient(hunting): authenticate will time out at 900.851624s 2023-10-30T12:00:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1501356d0 0x7fd1501cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1501356d0 0x7fd1501cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38158/0 (socket says 172.21.15.50:38158) 2023-10-30T12:00:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 1 -- 172.21.15.50:0/2322244397 learned_addr learned my addr 172.21.15.50:0/2322244397 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 10 monclient(hunting): get_auth_request con 0x7fd1501356d0 auth_method 0 2023-10-30T12:00:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd15011e430 0x7fd15011e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1500a8010 0x7fd1501ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 10 monclient(hunting): get_auth_request con 0x7fd15011e430 auth_method 0 2023-10-30T12:00:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 10 monclient(hunting): get_auth_request con 0x7fd1500a8010 auth_method 0 2023-10-30T12:00:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.668+0000 7fd14f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2023-10-30T12:00:55.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 1 -- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd15011e430 msgr2=0x7fd15011e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd15011e430 0x7fd15011e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 1 -- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1500a8010 msgr2=0x7fd1501ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1500a8010 0x7fd1501ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:55.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 1 monclient: found mon.noname-a 2023-10-30T12:00:55.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:55.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:55.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 1 -- 172.21.15.50:0/2322244397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400094d0 con 0x7fd1501356d0 2023-10-30T12:00:55.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14effd640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1501356d0 0x7fd1501cad70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd13800efc0 tx=0x7fd13800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:55.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 client.?.objecter ms_handle_connect 0x7fd1501356d0 2023-10-30T12:00:55.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:55.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 1 -- 172.21.15.50:0/2322244397 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138009b00 con 0x7fd1501356d0 2023-10-30T12:00:55.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:55.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: dump: 2023-10-30T12:00:55.711 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.712 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.712 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.712 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.712 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.712 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.712 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.713 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.713 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.713 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: _finish_auth 0 2023-10-30T12:00:55.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:55.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:55.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 1 -- 172.21.15.50:0/2322244397 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13800f040 con 0x7fd1501356d0 2023-10-30T12:00:55.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 5 monclient: authenticate success, global_id 4311 2023-10-30T12:00:55.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 10 monclient: _renew_subs 2023-10-30T12:00:55.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:55.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd150122370 con 0x7fd1501356d0 2023-10-30T12:00:55.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:55.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 1 -- 172.21.15.50:0/2322244397 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1380146c0 con 0x7fd1501356d0 2023-10-30T12:00:55.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:55.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd147fff640 10 monclient: dump: 2023-10-30T12:00:55.715 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.716 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.716 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.716 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.716 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.716 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.717 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.717 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.717 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.717 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd14dffb640 4 set_mon_vals no callback set 2023-10-30T12:00:55.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:55.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 10 monclient: _renew_subs 2023-10-30T12:00:55.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:55.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd150121ad0 con 0x7fd1501356d0 2023-10-30T12:00:55.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.669+0000 7fd155bb7640 1 librados: init done 2023-10-30T12:00:55.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd147fff640 1 -- 172.21.15.50:0/2322244397 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd138014820 con 0x7fd1501356d0 2023-10-30T12:00:55.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd147fff640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd130039320 0x7fd13003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd155bb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:55.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd155bb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:55.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd155bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:55.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd147fff640 1 -- 172.21.15.50:0/2322244397 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fd13804b1c0 con 0x7fd1501356d0 2023-10-30T12:00:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd147fff640 10 client.4311.objecter ms_dispatch 0x7fd150000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd147fff640 3 client.4311.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd147fff640 3 client.4311.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15004e160 con 0x7fd1501356d0 2023-10-30T12:00:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd147fff640 20 client.4311.objecter dump_active .. 0 homeless 2023-10-30T12:00:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd14f7fe640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd130039320 0x7fd13003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.670+0000 7fd14f7fe640 10 monclient: get_auth_request con 0x7fd130039320 auth_method 0 2023-10-30T12:00:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.671+0000 7fd14f7fe640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd130039320 0x7fd13003b7e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd140036000 tx=0x7fd14003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.671+0000 7fd147fff640 10 client.4311.objecter ms_handle_connect 0x7fd130039320 2023-10-30T12:00:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.675+0000 7fd147fff640 1 -- 172.21.15.50:0/2322244397 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd138007920 con 0x7fd1501356d0 2023-10-30T12:00:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.675+0000 7fd147fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:55.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.675+0000 7fd147fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:55.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.701+0000 7fe86467c640 10 monclient: get_monmap_and_config 2023-10-30T12:00:55.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.701+0000 7fe86467c640 10 monclient: build_initial_monmap 2023-10-30T12:00:55.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.701+0000 7fe86467c640 10 monclient: monmap: 2023-10-30T12:00:55.730 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:55.730 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.731 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:55.702728+0000 2023-10-30T12:00:55.731 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:55.702728+0000 2023-10-30T12:00:55.731 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:55.731 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.731 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:55.731 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:55.732 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:55.732 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:55.732 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:55.732 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.702+0000 7fe86467c640 1 Processor -- start 2023-10-30T12:00:55.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.702+0000 7fe86467c640 1 -- start start 2023-10-30T12:00:55.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.702+0000 7fe86467c640 10 monclient: init 2023-10-30T12:00:55.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.702+0000 7fe86467c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:55.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.702+0000 7fe86467c640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:00:55.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.702+0000 7fe86467c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe85c0a8010 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:55.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe85c099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:55.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe85c132da0 0x7fe85c13e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 10 monclient(hunting): picked mon.noname-c con 0x7fe85c132da0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:55.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c0a7190 con 0x7fe85c099a40 2023-10-30T12:00:55.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c0a83f0 con 0x7fe85c0a8010 2023-10-30T12:00:55.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85c0a7600 con 0x7fe85c132da0 2023-10-30T12:00:55.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:55.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe86467c640 10 monclient(hunting): authenticate will time out at 900.886114s 2023-10-30T12:00:55.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861c17640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe85c099a40 0x7fe85c13aa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60876/0 (socket says 172.21.15.50:60876) 2023-10-30T12:00:55.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861c17640 1 -- 172.21.15.50:0/2167061738 learned_addr learned my addr 172.21.15.50:0/2167061738 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:55.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 1 --2- 172.21.15.50:0/2167061738 >> v2:172.21.15.118:3300/0 conn(0x7fe85c132da0 0x7fe85c13e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): get_auth_request con 0x7fe85c132da0 auth_method 0 2023-10-30T12:00:55.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861416640 1 -- 172.21.15.50:0/2167061738 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952378761 0 0) 0x7fe85c0a7190 con 0x7fe85c099a40 2023-10-30T12:00:55.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861416640 10 monclient(hunting): _negotiate 2023-10-30T12:00:55.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861416640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861416640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861416640 10 monclient(hunting): my global_id is 4314 2023-10-30T12:00:55.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe861416640 1 -- 172.21.15.50:0/2167061738 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe848003520 con 0x7fe85c099a40 2023-10-30T12:00:55.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2023-10-30T12:00:55.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 1 -- 172.21.15.50:0/2167061738 >> v1:172.21.15.118:6789/0 conn(0x7fe85c0a8010 legacy=0x7fe85c132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 1 -- 172.21.15.50:0/2167061738 >> v1:172.21.15.50:6789/0 conn(0x7fe85c099a40 legacy=0x7fe85c13aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient: _finish_hunting 0 2023-10-30T12:00:55.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 1 monclient: found mon.noname-c 2023-10-30T12:00:55.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:55.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:00:55.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.703+0000 7fe862c19640 1 -- 172.21.15.50:0/2167061738 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c09a440 con 0x7fe85c132da0 2023-10-30T12:00:55.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe862c19640 1 --2- 172.21.15.50:0/2167061738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c132da0 0x7fe85c13e140 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe854009d90 tx=0x7fe85402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=669b12ec0383edb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:55.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 1 -- 172.21.15.50:0/2167061738 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85402cbd0 con 0x7fe85c132da0 2023-10-30T12:00:55.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:55.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: dump: 2023-10-30T12:00:55.741 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.741 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.741 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.742 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.742 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.742 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.742 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.742 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.743 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.743 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: _finish_auth 0 2023-10-30T12:00:55.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: _check_auth_tickets 2023-10-30T12:00:55.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:55.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe86467c640 5 monclient: authenticate success, global_id 4342 2023-10-30T12:00:55.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 1 -- 172.21.15.50:0/2167061738 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85402cd30 con 0x7fe85c132da0 2023-10-30T12:00:55.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe86467c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:55.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:55.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe86467c640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:55.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe86467c640 4 set_mon_vals no callback set 2023-10-30T12:00:55.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 1 -- 172.21.15.50:0/2167061738 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85402cea0 con 0x7fe85c132da0 2023-10-30T12:00:55.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:55.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.704+0000 7fe861416640 10 monclient: dump: 2023-10-30T12:00:55.745 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.745 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.746 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.746 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.746 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.746 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.746 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.747 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 10 monclient: shutdown 2023-10-30T12:00:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:55.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 -- 172.21.15.50:0/2167061738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c132da0 msgr2=0x7fe85c13e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 --2- 172.21.15.50:0/2167061738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c132da0 0x7fe85c13e140 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe854009d90 tx=0x7fe85402be20 comp rx=0 tx=0).stop 2023-10-30T12:00:55.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 -- 172.21.15.50:0/2167061738 shutdown_connections 2023-10-30T12:00:55.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 --2- 172.21.15.50:0/2167061738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c132da0 0x7fe85c13e140 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 -- 172.21.15.50:0/2167061738 >> 172.21.15.50:0/2167061738 conn(0x7fe85c0a4a10 msgr2=0x7fe85c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:55.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 -- 172.21.15.50:0/2167061738 shutdown_connections 2023-10-30T12:00:55.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 -- 172.21.15.50:0/2167061738 wait complete. 2023-10-30T12:00:55.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 10 monclient: build_initial_monmap 2023-10-30T12:00:55.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 10 monclient: monmap: 2023-10-30T12:00:55.749 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:55.749 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:55.751 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 librados: starting msgr at 2023-10-30T12:00:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.705+0000 7fe86467c640 1 librados: starting objecter 2023-10-30T12:00:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 Processor -- start 2023-10-30T12:00:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 -- start start 2023-10-30T12:00:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 librados: setting wanted keys 2023-10-30T12:00:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 librados: calling monclient init 2023-10-30T12:00:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient: init 2023-10-30T12:00:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c0a8010 0x7fe85c12dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): picked mon.noname-b con 0x7fe85c0a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe85c132da0 0x7fe85c12e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): picked mon.noname-c con 0x7fe85c132da0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe85c1310c0 0x7fe85c12e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): picked mon.noname-a con 0x7fe85c1310c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0a18d0 con 0x7fe85c1310c0 2023-10-30T12:00:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c136370 con 0x7fe85c0a8010 2023-10-30T12:00:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c13dac0 con 0x7fe85c132da0 2023-10-30T12:00:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe86467c640 10 monclient(hunting): authenticate will time out at 900.889580s 2023-10-30T12:00:55.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe85c132da0 0x7fe85c12e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe85c1310c0 0x7fe85c12e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe85c1310c0 0x7fe85c12e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38162/0 (socket says 172.21.15.50:38162) 2023-10-30T12:00:55.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 1 -- 172.21.15.50:0/385568181 learned_addr learned my addr 172.21.15.50:0/385568181 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:55.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 10 monclient(hunting): get_auth_request con 0x7fe85c1310c0 auth_method 0 2023-10-30T12:00:55.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 10 monclient(hunting): get_auth_request con 0x7fe85c132da0 auth_method 0 2023-10-30T12:00:55.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862c19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe861c17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862418640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c0a8010 0x7fe85c12dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862418640 10 monclient(hunting): get_auth_request con 0x7fe85c0a8010 auth_method 0 2023-10-30T12:00:55.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862418640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862418640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.706+0000 7fe862418640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe862418640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe862418640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe862418640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2023-10-30T12:00:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 1 -- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c0a8010 msgr2=0x7fe85c12dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c0a8010 0x7fe85c12dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 1 -- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe85c1310c0 msgr2=0x7fe85c12e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe85c1310c0 0x7fe85c12e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 10 monclient: _finish_hunting 0 2023-10-30T12:00:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 1 monclient: found mon.noname-c 2023-10-30T12:00:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 1 -- 172.21.15.50:0/385568181 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe854009a70 con 0x7fe85c132da0 2023-10-30T12:00:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe862c19640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe85c1310c0 0x7fe85c12e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe861c17640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe85c132da0 0x7fe85c12e100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe85c0a5230 tx=0x7fe84c001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 client.?.objecter ms_handle_connect 0x7fe85c132da0 2023-10-30T12:00:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 1 -- 172.21.15.50:0/385568181 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c005ce0 con 0x7fe85c132da0 2023-10-30T12:00:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: dump: 2023-10-30T12:00:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.766 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 5 monclient: authenticate success, global_id 4259 2023-10-30T12:00:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 1 -- 172.21.15.50:0/385568181 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c005e40 con 0x7fe85c132da0 2023-10-30T12:00:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 10 monclient: _renew_subs 2023-10-30T12:00:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c135e60 con 0x7fe85c132da0 2023-10-30T12:00:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 1 -- 172.21.15.50:0/385568181 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0072d0 con 0x7fe85c132da0 2023-10-30T12:00:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe852ffd640 10 monclient: dump: 2023-10-30T12:00:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:55.771 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe860c15640 4 set_mon_vals no callback set 2023-10-30T12:00:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 10 monclient: _renew_subs 2023-10-30T12:00:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c1307c0 con 0x7fe85c132da0 2023-10-30T12:00:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.707+0000 7fe86467c640 1 librados: init done 2023-10-30T12:00:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe852ffd640 1 -- 172.21.15.50:0/385568181 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe84c0074f0 con 0x7fe85c132da0 2023-10-30T12:00:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe86467c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe86467c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe86467c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c04e160 con 0x7fe85c132da0 2023-10-30T12:00:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe852ffd640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe8400392f0 0x7fe84003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe852ffd640 1 -- 172.21.15.50:0/385568181 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fe84c049900 con 0x7fe85c132da0 2023-10-30T12:00:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe852ffd640 10 client.4259.objecter ms_dispatch 0x7fe85c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe852ffd640 3 client.4259.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe852ffd640 3 client.4259.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe852ffd640 20 client.4259.objecter dump_active .. 0 homeless 2023-10-30T12:00:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe862418640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe8400392f0 0x7fe84003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.708+0000 7fe862418640 10 monclient: get_auth_request con 0x7fe8400392f0 auth_method 0 2023-10-30T12:00:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.709+0000 7fe862418640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe8400392f0 0x7fe84003b7b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe858000f60 tx=0x7fe858009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.709+0000 7fe852ffd640 10 client.4259.objecter ms_handle_connect 0x7fe8400392f0 2023-10-30T12:00:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.713+0000 7fe852ffd640 1 -- 172.21.15.50:0/385568181 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe84c019d20 con 0x7fe85c132da0 2023-10-30T12:00:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.713+0000 7fe852ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.713+0000 7fe852ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:55.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.855+0000 7fd155bb7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:00:55.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.855+0000 7fd155bb7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:00:55.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.855+0000 7fd155bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:55.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.855+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd15009a4d0 con 0x7fd1501356d0 2023-10-30T12:00:55.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.855+0000 7fd147fff640 1 -- 172.21.15.50:0/2322244397 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd138040090 con 0x7fd1501356d0 2023-10-30T12:00:55.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.855+0000 7fd147fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-10-30T12:00:55.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.855+0000 7fd147fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:55.858 INFO:teuthology.orchestra.run.smithi050.stdout:47244640258 2023-10-30T12:00:55.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 10 librados: watch_flush enter 2023-10-30T12:00:55.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 10 librados: watch_flush exit 2023-10-30T12:00:55.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 20 client.4311.objecter shutdown clearing up homeless session... 2023-10-30T12:00:55.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 10 client.4311.objecter successfully canceled tick 2023-10-30T12:00:55.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd130039320 msgr2=0x7fd13003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd130039320 0x7fd13003b7e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd140036000 tx=0x7fd14003d040 comp rx=0 tx=0).stop 2023-10-30T12:00:55.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 10 monclient: shutdown 2023-10-30T12:00:55.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:55.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1501356d0 msgr2=0x7fd1501cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1501356d0 0x7fd1501cad70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd13800efc0 tx=0x7fd13800c490 comp rx=0 tx=0).stop 2023-10-30T12:00:55.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 shutdown_connections 2023-10-30T12:00:55.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd130039320 0x7fd13003b7e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd15011e430 0x7fd15011e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1501356d0 0x7fd1501cad70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 --2- 172.21.15.50:0/2322244397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1500a8010 0x7fd1501ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 >> 172.21.15.50:0/2322244397 conn(0x7fd1500a4a10 msgr2=0x7fd1501332c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:55.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 shutdown_connections 2023-10-30T12:00:55.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 -- 172.21.15.50:0/2322244397 wait complete. 2023-10-30T12:00:55.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.857+0000 7fd155bb7640 1 librados: shutdown 2023-10-30T12:00:55.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.5 2023-10-30T12:00:55.870 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.889+0000 7fe86467c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:00:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.889+0000 7fe86467c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:00:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.889+0000 7fe86467c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.889+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe85c0745a0 con 0x7fe85c132da0 2023-10-30T12:00:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.890+0000 7fe852ffd640 1 -- 172.21.15.50:0/385568181 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe84c017070 con 0x7fe85c132da0 2023-10-30T12:00:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.890+0000 7fe852ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-10-30T12:00:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.890+0000 7fe852ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:55.893 INFO:teuthology.orchestra.run.smithi050.stdout:47244640259 2023-10-30T12:00:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 10 librados: watch_flush enter 2023-10-30T12:00:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 10 librados: watch_flush exit 2023-10-30T12:00:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 20 client.4259.objecter shutdown clearing up homeless session... 2023-10-30T12:00:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 10 client.4259.objecter successfully canceled tick 2023-10-30T12:00:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe8400392f0 msgr2=0x7fe84003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe8400392f0 0x7fe84003b7b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe858000f60 tx=0x7fe858009040 comp rx=0 tx=0).stop 2023-10-30T12:00:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 10 monclient: shutdown 2023-10-30T12:00:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe85c132da0 msgr2=0x7fe85c12e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe85c132da0 0x7fe85c12e100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe85c0a5230 tx=0x7fe84c001e70 comp rx=0 tx=0).stop 2023-10-30T12:00:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 shutdown_connections 2023-10-30T12:00:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe8400392f0 0x7fe84003b7b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe85c1310c0 0x7fe85c12e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe85c132da0 0x7fe85c12e100 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 --2- 172.21.15.50:0/385568181 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe85c0a8010 0x7fe85c12dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 >> 172.21.15.50:0/385568181 conn(0x7fe85c0a4a10 msgr2=0x7fe85c0a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 shutdown_connections 2023-10-30T12:00:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 -- 172.21.15.50:0/385568181 wait complete. 2023-10-30T12:00:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:55.892+0000 7fe86467c640 1 librados: shutdown 2023-10-30T12:00:55.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2023-10-30T12:00:55.904 DEBUG:teuthology.parallel:result is None 2023-10-30T12:00:55.905 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:00:55.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:00:56.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.048+0000 7f4c07f90640 10 monclient: get_monmap_and_config 2023-10-30T12:00:56.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.048+0000 7f4c07f90640 10 monclient: build_initial_monmap 2023-10-30T12:00:56.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.048+0000 7f4c07f90640 10 monclient: monmap: 2023-10-30T12:00:56.051 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:56.051 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.051 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:56.049511+0000 2023-10-30T12:00:56.051 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:56.049511+0000 2023-10-30T12:00:56.051 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:56.053 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.048+0000 7f4c07f90640 1 Processor -- start 2023-10-30T12:00:56.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 1 -- start start 2023-10-30T12:00:56.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient: init 2023-10-30T12:00:56.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:56.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:00:56.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4c000a7000 0x7f4c00135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient(hunting): picked mon.noname-c con 0x7f4c000a7000 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:56.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c0006abe0 0x7f4c001360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient(hunting): picked mon.noname-b con 0x7f4c0006abe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:56.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c000a4460 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:56.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c000a73e0 con 0x7f4c000a4460 2023-10-30T12:00:56.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c00136600 con 0x7f4c000a7000 2023-10-30T12:00:56.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0009a000 con 0x7f4c0006abe0 2023-10-30T12:00:56.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:56.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c07f90640 10 monclient(hunting): authenticate will time out at 901.232678s 2023-10-30T12:00:56.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c0006abe0 0x7f4c001360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c0006abe0 0x7f4c001360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35758/0 (socket says 172.21.15.50:35758) 2023-10-30T12:00:56.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 1 -- 172.21.15.50:0/3531898082 learned_addr learned my addr 172.21.15.50:0/3531898082 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 10 monclient(hunting): get_auth_request con 0x7f4c0006abe0 auth_method 0 2023-10-30T12:00:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.049+0000 7f4c0552b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c05d2c640 1 --2- 172.21.15.50:0/3531898082 >> v2:172.21.15.118:3300/0 conn(0x7f4c000a7000 0x7f4c00135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-10-30T12:00:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 1 -- 172.21.15.50:0/3531898082 >> v2:172.21.15.118:3300/0 conn(0x7f4c000a7000 msgr2=0x7f4c00135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 1 --2- 172.21.15.50:0/3531898082 >> v2:172.21.15.118:3300/0 conn(0x7f4c000a7000 0x7f4c00135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 1 -- 172.21.15.50:0/3531898082 >> v1:172.21.15.118:6789/0 conn(0x7f4c000a4460 legacy=0x7f4c000a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:56.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 10 monclient: _finish_hunting 0 2023-10-30T12:00:56.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 1 monclient: found mon.noname-b 2023-10-30T12:00:56.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:56.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 1 -- 172.21.15.50:0/3531898082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0009a2e0 con 0x7f4c0006abe0 2023-10-30T12:00:56.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c05d2c640 10 monclient: get_auth_request con 0x7f4c000a7000 auth_method 0 2023-10-30T12:00:56.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c05d2c640 1 --2- 172.21.15.50:0/3531898082 >> v2:172.21.15.118:3300/0 conn(0x7f4c000a7000 0x7f4c00135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:56.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c0552b640 1 --2- 172.21.15.50:0/3531898082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c0006abe0 0x7f4c001360c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009870 tx=0x7f4bf402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2509a19da27cf951 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c04d2a640 1 -- 172.21.15.50:0/3531898082 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40317c0 con 0x7f4c0006abe0 2023-10-30T12:00:56.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.050+0000 7f4c04d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:56.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: dump: 2023-10-30T12:00:56.068 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.069 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.069 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.069 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.069 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.070 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.070 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: _finish_auth 0 2023-10-30T12:00:56.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: _check_auth_tickets 2023-10-30T12:00:56.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:56.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 5 monclient: authenticate success, global_id 4262 2023-10-30T12:00:56.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 1 -- 172.21.15.50:0/3531898082 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf4031920 con 0x7f4c0006abe0 2023-10-30T12:00:56.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:56.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:56.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:56.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 4 set_mon_vals no callback set 2023-10-30T12:00:56.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 1 -- 172.21.15.50:0/3531898082 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4031cb0 con 0x7f4c0006abe0 2023-10-30T12:00:56.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:56.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c04d2a640 10 monclient: dump: 2023-10-30T12:00:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 10 monclient: shutdown 2023-10-30T12:00:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 -- 172.21.15.50:0/3531898082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c0006abe0 msgr2=0x7f4c001360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 --2- 172.21.15.50:0/3531898082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c0006abe0 0x7f4c001360c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009870 tx=0x7f4bf402b860 comp rx=0 tx=0).stop 2023-10-30T12:00:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 -- 172.21.15.50:0/3531898082 shutdown_connections 2023-10-30T12:00:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 --2- 172.21.15.50:0/3531898082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c0006abe0 0x7f4c001360c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 --2- 172.21.15.50:0/3531898082 >> v2:172.21.15.118:3300/0 conn(0x7f4c000a7000 0x7f4c00135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 -- 172.21.15.50:0/3531898082 >> 172.21.15.50:0/3531898082 conn(0x7f4c00066000 msgr2=0x7f4c000a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 -- 172.21.15.50:0/3531898082 shutdown_connections 2023-10-30T12:00:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.051+0000 7f4c07f90640 1 -- 172.21.15.50:0/3531898082 wait complete. 2023-10-30T12:00:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient: build_initial_monmap 2023-10-30T12:00:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient: monmap: 2023-10-30T12:00:56.077 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:56.077 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:56.077 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:56.077 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:56.077 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:56.078 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.078 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:56.078 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:56.078 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:56.078 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 librados: starting msgr at 2023-10-30T12:00:56.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 librados: starting objecter 2023-10-30T12:00:56.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 Processor -- start 2023-10-30T12:00:56.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 -- start start 2023-10-30T12:00:56.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 librados: setting wanted keys 2023-10-30T12:00:56.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 librados: calling monclient init 2023-10-30T12:00:56.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient: init 2023-10-30T12:00:56.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:56.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:56.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c0006abe0 0x7f4c001ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): picked mon.noname-b con 0x7f4c0006abe0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:56.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c000a4460 0x7f4c00131550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): picked mon.noname-c con 0x7f4c000a4460 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:56.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c000a7000 0x7f4c00131a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): picked mon.noname-a con 0x7f4c000a7000 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:56.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0011e970 con 0x7f4c000a7000 2023-10-30T12:00:56.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c001cb060 con 0x7f4c0006abe0 2023-10-30T12:00:56.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000a0350 con 0x7f4c000a4460 2023-10-30T12:00:56.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.052+0000 7f4c07f90640 10 monclient(hunting): authenticate will time out at 901.235980s 2023-10-30T12:00:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c000a4460 0x7f4c00131550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c000a4460 0x7f4c00131550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35760/0 (socket says 172.21.15.50:35760) 2023-10-30T12:00:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c000a7000 0x7f4c00131a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 -- 172.21.15.50:0/1976528127 learned_addr learned my addr 172.21.15.50:0/1976528127 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): get_auth_request con 0x7f4c000a4460 auth_method 0 2023-10-30T12:00:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 10 monclient(hunting): get_auth_request con 0x7f4c000a7000 auth_method 0 2023-10-30T12:00:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0652d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c05d2c640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c0006abe0 0x7f4c001ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c05d2c640 10 monclient(hunting): get_auth_request con 0x7f4c0006abe0 auth_method 0 2023-10-30T12:00:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c05d2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c05d2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c05d2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2023-10-30T12:00:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 -- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c0006abe0 msgr2=0x7f4c001ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c0006abe0 0x7f4c001ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 -- 172.21.15.50:0/1976528127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c000a7000 msgr2=0x7f4c00131a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c000a7000 0x7f4c00131a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient: _finish_hunting 0 2023-10-30T12:00:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 monclient: found mon.noname-c 2023-10-30T12:00:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 -- 172.21.15.50:0/1976528127 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf40094d0 con 0x7f4c000a4460 2023-10-30T12:00:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c0552b640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c000a4460 0x7f4c00131550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4000c00 tx=0x7f4bf40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f4c000a4460 2023-10-30T12:00:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 1 -- 172.21.15.50:0/1976528127 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40329a0 con 0x7f4c000a4460 2023-10-30T12:00:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:56.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: dump: 2023-10-30T12:00:56.090 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.090 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.091 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.091 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.091 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.091 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.091 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.091 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.092 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:56.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:56.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:56.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 1 -- 172.21.15.50:0/1976528127 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf4032b20 con 0x7f4c000a4460 2023-10-30T12:00:56.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 5 monclient: authenticate success, global_id 4265 2023-10-30T12:00:56.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 10 monclient: _renew_subs 2023-10-30T12:00:56.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c00132490 con 0x7f4c000a4460 2023-10-30T12:00:56.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:56.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 1 -- 172.21.15.50:0/1976528127 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40453f0 con 0x7f4c000a4460 2023-10-30T12:00:56.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:56.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:56.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfe7fc640 10 monclient: dump: 2023-10-30T12:00:56.094 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.095 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.095 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.095 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.095 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.096 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4bfffff640 4 set_mon_vals no callback set 2023-10-30T12:00:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 10 monclient: _renew_subs 2023-10-30T12:00:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0011e690 con 0x7f4c000a4460 2023-10-30T12:00:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.053+0000 7f4c07f90640 1 librados: init done 2023-10-30T12:00:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4bfe7fc640 1 -- 172.21.15.50:0/1976528127 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4bf404d3f0 con 0x7f4c000a4460 2023-10-30T12:00:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4bfe7fc640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bec039320 0x7f4bec03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4bfe7fc640 1 -- 172.21.15.50:0/1976528127 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f4bf406c390 con 0x7f4c000a4460 2023-10-30T12:00:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4bfe7fc640 10 client.4265.objecter ms_dispatch 0x7f4c00000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4bfe7fc640 3 client.4265.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4bfe7fc640 3 client.4265.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4bfe7fc640 20 client.4265.objecter dump_active .. 0 homeless 2023-10-30T12:00:56.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4c07f90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4c07f90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4c07f90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0011eab0 con 0x7f4c000a4460 2023-10-30T12:00:56.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4c05d2c640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bec039320 0x7f4bec03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.054+0000 7f4c05d2c640 10 monclient: get_auth_request con 0x7f4bec039320 auth_method 0 2023-10-30T12:00:56.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.055+0000 7f4c05d2c640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bec039320 0x7f4bec03b7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00075b0 tx=0x7f4bf0007540 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.055+0000 7f4bfe7fc640 10 client.4265.objecter ms_handle_connect 0x7f4bec039320 2023-10-30T12:00:56.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.059+0000 7f4bfe7fc640 1 -- 172.21.15.50:0/1976528127 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4c0011eab0 con 0x7f4c000a4460 2023-10-30T12:00:56.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.059+0000 7f4bfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.059+0000 7f4bfe7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:56.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.230+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c000a1790 con 0x7f4bec039320 2023-10-30T12:00:56.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.232+0000 7f4bfe7fc640 1 -- 172.21.15.50:0/1976528127 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39456 (secure 0 0 0) 0x7f4c000a1790 con 0x7f4bec039320 2023-10-30T12:00:56.235 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:56.235 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:00:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 10 librados: watch_flush enter 2023-10-30T12:00:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 10 librados: watch_flush exit 2023-10-30T12:00:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 20 client.4265.objecter shutdown clearing up homeless session... 2023-10-30T12:00:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 10 client.4265.objecter successfully canceled tick 2023-10-30T12:00:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bec039320 msgr2=0x7f4bec03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bec039320 0x7f4bec03b7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00075b0 tx=0x7f4bf0007540 comp rx=0 tx=0).stop 2023-10-30T12:00:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 10 monclient: shutdown 2023-10-30T12:00:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c000a4460 msgr2=0x7f4c00131550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c000a4460 0x7f4c00131550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4000c00 tx=0x7f4bf40046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 shutdown_connections 2023-10-30T12:00:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bec039320 0x7f4bec03b7e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c000a7000 0x7f4c00131a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c000a4460 0x7f4c00131550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 --2- 172.21.15.50:0/1976528127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c0006abe0 0x7f4c001ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.234+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 >> 172.21.15.50:0/1976528127 conn(0x7f4c00066000 msgr2=0x7f4c001bd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:56.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.235+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 shutdown_connections 2023-10-30T12:00:56.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.235+0000 7f4c07f90640 1 -- 172.21.15.50:0/1976528127 wait complete. 2023-10-30T12:00:56.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.235+0000 7f4c07f90640 1 librados: shutdown 2023-10-30T12:00:56.247 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-10-30T12:00:54.393161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163976,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067064,"statfs":{"total":579820584960,"available":579652673536,"internally_reserved":0,"allocated":1900544,"data_stored":1314104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":165927591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475189+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:00:53.475189+0000","last_peered":"2023-10-30T12:00:53.475189+0000","last_clean":"2023-10-30T12:00:53.475189+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:00:53.475189+0000","last_undegraded":"2023-10-30T12:00:53.475189+0000","last_fullsized":"2023-10-30T12:00:53.475189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475012+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:00:53.475012+0000","last_peered":"2023-10-30T12:00:53.475012+0000","last_clean":"2023-10-30T12:00:53.475012+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:00:53.475012+0000","last_undegraded":"2023-10-30T12:00:53.475012+0000","last_fullsized":"2023-10-30T12:00:53.475012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583189+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:00:53.583189+0000","last_peered":"2023-10-30T12:00:53.583189+0000","last_clean":"2023-10-30T12:00:53.583189+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:00:53.583189+0000","last_undegraded":"2023-10-30T12:00:53.583189+0000","last_fullsized":"2023-10-30T12:00:53.583189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583724+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:00:53.583724+0000","last_peered":"2023-10-30T12:00:53.583724+0000","last_clean":"2023-10-30T12:00:53.583724+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:00:53.583724+0000","last_undegraded":"2023-10-30T12:00:53.583724+0000","last_fullsized":"2023-10-30T12:00:53.583724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475112+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:00:53.475112+0000","last_peered":"2023-10-30T12:00:53.475112+0000","last_clean":"2023-10-30T12:00:53.475112+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:00:53.475112+0000","last_undegraded":"2023-10-30T12:00:53.475112+0000","last_fullsized":"2023-10-30T12:00:53.475112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583196+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:00:53.583196+0000","last_peered":"2023-10-30T12:00:53.583196+0000","last_clean":"2023-10-30T12:00:53.583196+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:00:53.583196+0000","last_undegraded":"2023-10-30T12:00:53.583196+0000","last_fullsized":"2023-10-30T12:00:53.583196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":163,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475070+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:00:53.475070+0000","last_peered":"2023-10-30T12:00:53.475070+0000","last_clean":"2023-10-30T12:00:53.475070+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:00:53.475070+0000","last_undegraded":"2023-10-30T12:00:53.475070+0000","last_fullsized":"2023-10-30T12:00:53.475070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":42708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":42708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":571636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]}]},{"osd":2,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":42708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":42708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":571636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:00:56.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:00:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.351+0000 7efd6d34c640 10 monclient: get_monmap_and_config 2023-10-30T12:00:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.351+0000 7efd6d34c640 10 monclient: build_initial_monmap 2023-10-30T12:00:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.351+0000 7efd6d34c640 10 monclient: monmap: 2023-10-30T12:00:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:56.352840+0000 2023-10-30T12:00:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:56.352840+0000 2023-10-30T12:00:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:56.356 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:56.356 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:56.356 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 1 Processor -- start 2023-10-30T12:00:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 1 -- start start 2023-10-30T12:00:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient: init 2023-10-30T12:00:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-10-30T12:00:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efd680a7000 0x7efd68135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient(hunting): picked mon.noname-c con 0x7efd680a7000 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd6806abe0 0x7efd681360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient(hunting): picked mon.noname-a con 0x7efd6806abe0 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd680a4460 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd680a73e0 con 0x7efd680a4460 2023-10-30T12:00:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd68136600 con 0x7efd6806abe0 2023-10-30T12:00:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6809a000 con 0x7efd680a7000 2023-10-30T12:00:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.352+0000 7efd6d34c640 10 monclient(hunting): authenticate will time out at 901.535911s 2023-10-30T12:00:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd6806abe0 0x7efd681360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd6806abe0 0x7efd681360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38174/0 (socket says 172.21.15.50:38174) 2023-10-30T12:00:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 -- 172.21.15.50:0/3974278730 learned_addr learned my addr 172.21.15.50:0/3974278730 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): get_auth_request con 0x7efd6806abe0 auth_method 0 2023-10-30T12:00:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd66ffd640 1 --2- 172.21.15.50:0/3974278730 >> v2:172.21.15.118:3300/0 conn(0x7efd680a7000 0x7efd68135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd66ffd640 10 monclient(hunting): get_auth_request con 0x7efd680a7000 auth_method 0 2023-10-30T12:00:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2023-10-30T12:00:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 -- 172.21.15.50:0/3974278730 >> v2:172.21.15.118:3300/0 conn(0x7efd680a7000 msgr2=0x7efd68135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 --2- 172.21.15.50:0/3974278730 >> v2:172.21.15.118:3300/0 conn(0x7efd680a7000 0x7efd68135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 -- 172.21.15.50:0/3974278730 >> v1:172.21.15.50:6789/0 conn(0x7efd680a4460 legacy=0x7efd680a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient: _finish_hunting 0 2023-10-30T12:00:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 monclient: found mon.noname-a 2023-10-30T12:00:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd65ffb640 1 -- 172.21.15.50:0/3974278730 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650082007 0 0) 0x7efd680a73e0 con 0x7efd680a4460 2023-10-30T12:00:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd667fc640 1 -- 172.21.15.50:0/3974278730 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6809a2e0 con 0x7efd6806abe0 2023-10-30T12:00:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.353+0000 7efd65ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd667fc640 1 --2- 172.21.15.50:0/3974278730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681360c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd680a76b0 tx=0x7efd5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00a2cb917d65a3d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 1 -- 172.21.15.50:0/3974278730 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c031870 con 0x7efd6806abe0 2023-10-30T12:00:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: dump: 2023-10-30T12:00:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.374 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:56.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:56.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 1 -- 172.21.15.50:0/3974278730 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c0319d0 con 0x7efd6806abe0 2023-10-30T12:00:56.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 5 monclient: authenticate success, global_id 4323 2023-10-30T12:00:56.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:56.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:56.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:56.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 4 set_mon_vals no callback set 2023-10-30T12:00:56.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 1 -- 172.21.15.50:0/3974278730 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c031d60 con 0x7efd6806abe0 2023-10-30T12:00:56.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:56.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd65ffb640 10 monclient: dump: 2023-10-30T12:00:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.378 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 10 monclient: shutdown 2023-10-30T12:00:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 -- 172.21.15.50:0/3974278730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 msgr2=0x7efd681360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 --2- 172.21.15.50:0/3974278730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681360c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd680a76b0 tx=0x7efd5c02b910 comp rx=0 tx=0).stop 2023-10-30T12:00:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 -- 172.21.15.50:0/3974278730 shutdown_connections 2023-10-30T12:00:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 --2- 172.21.15.50:0/3974278730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681360c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 --2- 172.21.15.50:0/3974278730 >> v2:172.21.15.118:3300/0 conn(0x7efd680a7000 0x7efd68135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 -- 172.21.15.50:0/3974278730 >> 172.21.15.50:0/3974278730 conn(0x7efd68066000 msgr2=0x7efd680a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 -- 172.21.15.50:0/3974278730 shutdown_connections 2023-10-30T12:00:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.354+0000 7efd6d34c640 1 -- 172.21.15.50:0/3974278730 wait complete. 2023-10-30T12:00:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 10 monclient: build_initial_monmap 2023-10-30T12:00:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 10 monclient: monmap: 2023-10-30T12:00:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:56.382 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 1 librados: starting msgr at 2023-10-30T12:00:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 1 librados: starting objecter 2023-10-30T12:00:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 1 Processor -- start 2023-10-30T12:00:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 1 -- start start 2023-10-30T12:00:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 1 librados: setting wanted keys 2023-10-30T12:00:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 1 librados: calling monclient init 2023-10-30T12:00:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.355+0000 7efd6d34c640 10 monclient: init 2023-10-30T12:00:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:00:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): picked mon.noname-a con 0x7efd6806abe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:56.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd680a4460 0x7efd68132500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): picked mon.noname-c con 0x7efd680a4460 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:56.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd680a7000 0x7efd68132a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): picked mon.noname-b con 0x7efd680a7000 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:56.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd68120910 con 0x7efd6806abe0 2023-10-30T12:00:56.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd68120bf0 con 0x7efd680a7000 2023-10-30T12:00:56.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd68120ed0 con 0x7efd680a4460 2023-10-30T12:00:56.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:56.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd6d34c640 10 monclient(hunting): authenticate will time out at 901.539370s 2023-10-30T12:00:56.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd680a4460 0x7efd68132500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38186/0 (socket says 172.21.15.50:38186) 2023-10-30T12:00:56.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 1 -- 172.21.15.50:0/16538851 learned_addr learned my addr 172.21.15.50:0/16538851 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:56.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): get_auth_request con 0x7efd680a4460 auth_method 0 2023-10-30T12:00:56.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 10 monclient(hunting): get_auth_request con 0x7efd6806abe0 auth_method 0 2023-10-30T12:00:56.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd677fe640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd680a7000 0x7efd68132a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2023-10-30T12:00:56.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 -- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd680a7000 msgr2=0x7efd68132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd680a7000 0x7efd68132a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 -- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 msgr2=0x7efd681ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient: _finish_hunting 0 2023-10-30T12:00:56.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 monclient: found mon.noname-c 2023-10-30T12:00:56.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:56.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 -- 172.21.15.50:0/16538851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c0094d0 con 0x7efd680a4460 2023-10-30T12:00:56.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd66ffd640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:56.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd667fc640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd680a4460 0x7efd68132500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efd5c033000 tx=0x7efd5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd4f7fe640 10 client.?.objecter ms_handle_connect 0x7efd680a4460 2023-10-30T12:00:56.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.356+0000 7efd4f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:56.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 1 -- 172.21.15.50:0/16538851 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c03ac60 con 0x7efd680a4460 2023-10-30T12:00:56.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:56.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: dump: 2023-10-30T12:00:56.394 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.394 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.394 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.394 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.394 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.394 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.395 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.395 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.395 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.395 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:56.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:56.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:56.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 1 -- 172.21.15.50:0/16538851 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c03ade0 con 0x7efd680a4460 2023-10-30T12:00:56.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:56.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 1 -- 172.21.15.50:0/16538851 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c044630 con 0x7efd680a4460 2023-10-30T12:00:56.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 5 monclient: authenticate success, global_id 4268 2023-10-30T12:00:56.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:56.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 10 monclient: dump: 2023-10-30T12:00:56.397 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.397 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.398 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.398 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.398 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.398 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.398 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.399 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.399 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.399 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 10 monclient: _renew_subs 2023-10-30T12:00:56.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd680a09b0 con 0x7efd680a4460 2023-10-30T12:00:56.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd657fa640 4 set_mon_vals no callback set 2023-10-30T12:00:56.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:56.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 10 monclient: _renew_subs 2023-10-30T12:00:56.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6809ff90 con 0x7efd680a4460 2023-10-30T12:00:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd6d34c640 1 librados: init done 2023-10-30T12:00:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.357+0000 7efd4f7fe640 1 -- 172.21.15.50:0/16538851 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efd5c0303c0 con 0x7efd680a4460 2023-10-30T12:00:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd6d34c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd4f7fe640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd540392f0 0x7efd5403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd6d34c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd6d34c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:56.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6804e160 con 0x7efd680a4460 2023-10-30T12:00:56.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd4f7fe640 1 -- 172.21.15.50:0/16538851 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7efd5c02d040 con 0x7efd680a4460 2023-10-30T12:00:56.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd4f7fe640 10 client.4268.objecter ms_dispatch 0x7efd68000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:56.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd4f7fe640 3 client.4268.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:56.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd4f7fe640 3 client.4268.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:56.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd4f7fe640 20 client.4268.objecter dump_active .. 0 homeless 2023-10-30T12:00:56.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd66ffd640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd540392f0 0x7efd5403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.358+0000 7efd66ffd640 10 monclient: get_auth_request con 0x7efd540392f0 auth_method 0 2023-10-30T12:00:56.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.359+0000 7efd4f7fe640 10 client.4268.objecter ms_handle_connect 0x7efd540392f0 2023-10-30T12:00:56.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.359+0000 7efd66ffd640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd540392f0 0x7efd5403b7b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd58006fd0 tx=0x7efd58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.362+0000 7efd4f7fe640 1 -- 172.21.15.50:0/16538851 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efd5c035080 con 0x7efd680a4460 2023-10-30T12:00:56.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.362+0000 7efd4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.362+0000 7efd4f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.538+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd68120a80 con 0x7efd540392f0 2023-10-30T12:00:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.540+0000 7efd4f7fe640 1 -- 172.21.15.50:0/16538851 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40580 (secure 0 0 0) 0x7efd68120a80 con 0x7efd540392f0 2023-10-30T12:00:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:56.543 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:00:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 10 librados: watch_flush enter 2023-10-30T12:00:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 10 librados: watch_flush exit 2023-10-30T12:00:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 20 client.4268.objecter shutdown clearing up homeless session... 2023-10-30T12:00:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 10 client.4268.objecter successfully canceled tick 2023-10-30T12:00:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd540392f0 msgr2=0x7efd5403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd540392f0 0x7efd5403b7b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd58006fd0 tx=0x7efd58008040 comp rx=0 tx=0).stop 2023-10-30T12:00:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 10 monclient: shutdown 2023-10-30T12:00:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd680a4460 msgr2=0x7efd68132500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd680a4460 0x7efd68132500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efd5c033000 tx=0x7efd5c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 shutdown_connections 2023-10-30T12:00:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd540392f0 0x7efd5403b7b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd680a7000 0x7efd68132a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd680a4460 0x7efd68132500 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 --2- 172.21.15.50:0/16538851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd6806abe0 0x7efd681ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 >> 172.21.15.50:0/16538851 conn(0x7efd68066000 msgr2=0x7efd680a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 shutdown_connections 2023-10-30T12:00:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.542+0000 7efd6d34c640 1 -- 172.21.15.50:0/16538851 wait complete. 2023-10-30T12:00:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.543+0000 7efd6d34c640 1 librados: shutdown 2023-10-30T12:00:56.555 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-10-30T12:00:56.393533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164072,"kb_used_data":1928,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066968,"statfs":{"total":579820584960,"available":579652575232,"internally_reserved":0,"allocated":1974272,"data_stored":1343408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":165927591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000371"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475189+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:00:53.475189+0000","last_peered":"2023-10-30T12:00:53.475189+0000","last_clean":"2023-10-30T12:00:53.475189+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:00:53.475189+0000","last_undegraded":"2023-10-30T12:00:53.475189+0000","last_fullsized":"2023-10-30T12:00:53.475189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475012+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:00:53.475012+0000","last_peered":"2023-10-30T12:00:53.475012+0000","last_clean":"2023-10-30T12:00:53.475012+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:00:53.475012+0000","last_undegraded":"2023-10-30T12:00:53.475012+0000","last_fullsized":"2023-10-30T12:00:53.475012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583189+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:00:53.583189+0000","last_peered":"2023-10-30T12:00:53.583189+0000","last_clean":"2023-10-30T12:00:53.583189+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:00:53.583189+0000","last_undegraded":"2023-10-30T12:00:53.583189+0000","last_fullsized":"2023-10-30T12:00:53.583189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583724+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:00:53.583724+0000","last_peered":"2023-10-30T12:00:53.583724+0000","last_clean":"2023-10-30T12:00:53.583724+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:00:53.583724+0000","last_undegraded":"2023-10-30T12:00:53.583724+0000","last_fullsized":"2023-10-30T12:00:53.583724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475112+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:00:53.475112+0000","last_peered":"2023-10-30T12:00:53.475112+0000","last_clean":"2023-10-30T12:00:53.475112+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:00:53.475112+0000","last_undegraded":"2023-10-30T12:00:53.475112+0000","last_fullsized":"2023-10-30T12:00:53.475112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583196+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:00:53.583196+0000","last_peered":"2023-10-30T12:00:53.583196+0000","last_clean":"2023-10-30T12:00:53.583196+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:00:53.583196+0000","last_undegraded":"2023-10-30T12:00:53.583196+0000","last_fullsized":"2023-10-30T12:00:53.583196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":163,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475070+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:00:53.475070+0000","last_peered":"2023-10-30T12:00:53.475070+0000","last_clean":"2023-10-30T12:00:53.475070+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:00:53.475070+0000","last_undegraded":"2023-10-30T12:00:53.475070+0000","last_fullsized":"2023-10-30T12:00:53.475070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":576520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":576520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:00:56.556 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:00:56.556 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T12:00:56.556 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T12:00:56.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T12:00:56.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.660+0000 7fd2754b4640 10 monclient: get_monmap_and_config 2023-10-30T12:00:56.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.660+0000 7fd2754b4640 10 monclient: build_initial_monmap 2023-10-30T12:00:56.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.660+0000 7fd2754b4640 10 monclient: monmap: 2023-10-30T12:00:56.663 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:56.663 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.663 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:56.661421+0000 2023-10-30T12:00:56.663 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:56.661421+0000 2023-10-30T12:00:56.663 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:56.665 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:56.665 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.660+0000 7fd2754b4640 1 Processor -- start 2023-10-30T12:00:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.660+0000 7fd2754b4640 1 -- start start 2023-10-30T12:00:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.660+0000 7fd2754b4640 10 monclient: init 2023-10-30T12:00:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:00:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd2700a2dd0 0x7fd2700a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient(hunting): picked mon.noname-a con 0x7fd2700a2dd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd27006a8e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd270071400 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2700a31b0 con 0x7fd27006a8e0 2023-10-30T12:00:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27006a6a0 con 0x7fd270071400 2023-10-30T12:00:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27006be90 con 0x7fd2700a2dd0 2023-10-30T12:00:56.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:56.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd2754b4640 10 monclient(hunting): authenticate will time out at 901.844447s 2023-10-30T12:00:56.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd2700a2dd0 0x7fd2700a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26e7fc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd27006a8e0 0x7fd2700a4c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60900/0 (socket says 172.21.15.50:60900) 2023-10-30T12:00:56.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26e7fc640 1 -- 172.21.15.50:0/3955023920 learned_addr learned my addr 172.21.15.50:0/3955023920 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): get_auth_request con 0x7fd2700a2dd0 auth_method 0 2023-10-30T12:00:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2023-10-30T12:00:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 1 -- 172.21.15.50:0/3955023920 >> v1:172.21.15.118:6789/0 conn(0x7fd270071400 legacy=0x7fd27013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 1 -- 172.21.15.50:0/3955023920 >> v1:172.21.15.50:6789/0 conn(0x7fd27006a8e0 legacy=0x7fd2700a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 1 monclient: found mon.noname-a 2023-10-30T12:00:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26dffb640 1 -- 172.21.15.50:0/3955023920 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608613117 0 0) 0x7fd2700a31b0 con 0x7fd27006a8e0 2023-10-30T12:00:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26effd640 1 -- 172.21.15.50:0/3955023920 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27009e190 con 0x7fd2700a2dd0 2023-10-30T12:00:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.661+0000 7fd26dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:56.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26effd640 1 --2- 172.21.15.50:0/3955023920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2700a2dd0 0x7fd2700a46c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd260009d90 tx=0x7fd26002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1e1bcba8efe910 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 1 -- 172.21.15.50:0/3955023920 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26002cb40 con 0x7fd2700a2dd0 2023-10-30T12:00:56.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:56.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: dump: 2023-10-30T12:00:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.681 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.681 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.681 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:56.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:56.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:56.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd2754b4640 5 monclient: authenticate success, global_id 4332 2023-10-30T12:00:56.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd2754b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:56.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 1 -- 172.21.15.50:0/3955023920 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26002cca0 con 0x7fd2700a2dd0 2023-10-30T12:00:56.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:56.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd2754b4640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:56.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd2754b4640 4 set_mon_vals no callback set 2023-10-30T12:00:56.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 1 -- 172.21.15.50:0/3955023920 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260034660 con 0x7fd2700a2dd0 2023-10-30T12:00:56.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.662+0000 7fd26dffb640 10 monclient: dump: 2023-10-30T12:00:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.685 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 10 monclient: shutdown 2023-10-30T12:00:56.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:56.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 1 -- 172.21.15.50:0/3955023920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2700a2dd0 msgr2=0x7fd2700a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 1 --2- 172.21.15.50:0/3955023920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2700a2dd0 0x7fd2700a46c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd260009d90 tx=0x7fd26002bd90 comp rx=0 tx=0).stop 2023-10-30T12:00:56.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 1 -- 172.21.15.50:0/3955023920 shutdown_connections 2023-10-30T12:00:56.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 1 --2- 172.21.15.50:0/3955023920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2700a2dd0 0x7fd2700a46c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 1 -- 172.21.15.50:0/3955023920 >> 172.21.15.50:0/3955023920 conn(0x7fd2700a22b0 msgr2=0x7fd2700a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:56.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 1 -- 172.21.15.50:0/3955023920 shutdown_connections 2023-10-30T12:00:56.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 1 -- 172.21.15.50:0/3955023920 wait complete. 2023-10-30T12:00:56.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 10 monclient: build_initial_monmap 2023-10-30T12:00:56.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.663+0000 7fd2754b4640 10 monclient: monmap: 2023-10-30T12:00:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:56.689 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 librados: starting msgr at 2023-10-30T12:00:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 librados: starting objecter 2023-10-30T12:00:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 Processor -- start 2023-10-30T12:00:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 -- start start 2023-10-30T12:00:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 librados: setting wanted keys 2023-10-30T12:00:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 librados: calling monclient init 2023-10-30T12:00:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 10 monclient: init 2023-10-30T12:00:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd270071400 0x7fd2701c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 10 monclient(hunting): picked mon.noname-a con 0x7fd270071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2700a2dd0 0x7fd2701c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 10 monclient(hunting): picked mon.noname-b con 0x7fd2700a2dd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2701c70f0 0x7fd2701ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 10 monclient(hunting): picked mon.noname-c con 0x7fd2701c70f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.664+0000 7fd2754b4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27006b130 con 0x7fd270071400 2023-10-30T12:00:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2701a8cf0 con 0x7fd2700a2dd0 2023-10-30T12:00:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2700706b0 con 0x7fd2701c70f0 2023-10-30T12:00:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient(hunting): authenticate will time out at 901.848043s 2023-10-30T12:00:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd270071400 0x7fd2701c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd270071400 0x7fd2701c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38198/0 (socket says 172.21.15.50:38198) 2023-10-30T12:00:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 -- 172.21.15.50:0/57087615 learned_addr learned my addr 172.21.15.50:0/57087615 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): get_auth_request con 0x7fd270071400 auth_method 0 2023-10-30T12:00:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2701c70f0 0x7fd2701ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 10 monclient(hunting): get_auth_request con 0x7fd2701c70f0 auth_method 0 2023-10-30T12:00:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2700a2dd0 0x7fd2701c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 10 monclient(hunting): get_auth_request con 0x7fd2700a2dd0 auth_method 0 2023-10-30T12:00:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2023-10-30T12:00:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 -- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2701c70f0 msgr2=0x7fd2701ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2701c70f0 0x7fd2701ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 -- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2700a2dd0 msgr2=0x7fd2701c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2700a2dd0 0x7fd2701c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient: _finish_hunting 0 2023-10-30T12:00:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 monclient: found mon.noname-a 2023-10-30T12:00:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 -- 172.21.15.50:0/57087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd260009a70 con 0x7fd270071400 2023-10-30T12:00:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26effd640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd270071400 0x7fd2701c3f30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd260009eb0 tx=0x7fd260004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 client.?.objecter ms_handle_connect 0x7fd270071400 2023-10-30T12:00:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 1 -- 172.21.15.50:0/57087615 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260003b90 con 0x7fd270071400 2023-10-30T12:00:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: dump: 2023-10-30T12:00:56.702 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.702 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.702 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.702 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.702 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.703 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.703 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 1 -- 172.21.15.50:0/57087615 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd260003cf0 con 0x7fd270071400 2023-10-30T12:00:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 5 monclient: authenticate success, global_id 4338 2023-10-30T12:00:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 1 -- 172.21.15.50:0/57087615 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260004270 con 0x7fd270071400 2023-10-30T12:00:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient: _renew_subs 2023-10-30T12:00:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2701ab050 con 0x7fd270071400 2023-10-30T12:00:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:56.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:56.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2677fe640 10 monclient: dump: 2023-10-30T12:00:56.706 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:56.706 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:56.707 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:56.708 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd26d7fa640 4 set_mon_vals no callback set 2023-10-30T12:00:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient: _renew_subs 2023-10-30T12:00:56.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:56.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2701a9570 con 0x7fd270071400 2023-10-30T12:00:56.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.665+0000 7fd2754b4640 1 librados: init done 2023-10-30T12:00:56.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.666+0000 7fd2677fe640 1 -- 172.21.15.50:0/57087615 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd2600043d0 con 0x7fd270071400 2023-10-30T12:00:56.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.666+0000 7fd2754b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.666+0000 7fd2754b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.666+0000 7fd2754b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:56.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.666+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2700675d0 con 0x7fd270071400 2023-10-30T12:00:56.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd2677fe640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2500392f0 0x7fd25003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:56.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd2677fe640 1 -- 172.21.15.50:0/57087615 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fd26006d620 con 0x7fd270071400 2023-10-30T12:00:56.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd2677fe640 10 client.4338.objecter ms_dispatch 0x7fd270000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:56.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd2677fe640 3 client.4338.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:56.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd2677fe640 3 client.4338.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:56.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd2677fe640 20 client.4338.objecter dump_active .. 0 homeless 2023-10-30T12:00:56.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd26e7fc640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2500392f0 0x7fd25003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:56.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd26e7fc640 10 monclient: get_auth_request con 0x7fd2500392f0 auth_method 0 2023-10-30T12:00:56.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd26e7fc640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2500392f0 0x7fd25003b7b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2701cb1b0 tx=0x7fd258007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:56.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.667+0000 7fd2677fe640 10 client.4338.objecter ms_handle_connect 0x7fd2500392f0 2023-10-30T12:00:56.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.671+0000 7fd2677fe640 1 -- 172.21.15.50:0/57087615 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd260004860 con 0x7fd270071400 2023-10-30T12:00:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.671+0000 7fd2677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:56.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.671+0000 7fd2677fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:56.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.884+0000 7fd2754b4640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-10-30T12:00:56.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.884+0000 7fd2754b4640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-10-30T12:00:56.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.884+0000 7fd2754b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:56.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.884+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd27004e160 con 0x7fd270071400 2023-10-30T12:00:56.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.885+0000 7fd2677fe640 1 -- 172.21.15.50:0/57087615 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fd260039070 con 0x7fd270071400 2023-10-30T12:00:56.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.885+0000 7fd2677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-10-30T12:00:56.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.885+0000 7fd2677fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:56.888 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:56.888 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T12:00:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 10 librados: watch_flush enter 2023-10-30T12:00:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 10 librados: watch_flush exit 2023-10-30T12:00:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 20 client.4338.objecter shutdown clearing up homeless session... 2023-10-30T12:00:56.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 10 client.4338.objecter successfully canceled tick 2023-10-30T12:00:56.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2500392f0 msgr2=0x7fd25003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2500392f0 0x7fd25003b7b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2701cb1b0 tx=0x7fd258007d50 comp rx=0 tx=0).stop 2023-10-30T12:00:56.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 10 monclient: shutdown 2023-10-30T12:00:56.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:56.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd270071400 msgr2=0x7fd2701c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:56.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd270071400 0x7fd2701c3f30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd260009eb0 tx=0x7fd260004a40 comp rx=0 tx=0).stop 2023-10-30T12:00:56.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 shutdown_connections 2023-10-30T12:00:56.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2500392f0 0x7fd25003b7b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2701c70f0 0x7fd2701ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2700a2dd0 0x7fd2701c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 --2- 172.21.15.50:0/57087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd270071400 0x7fd2701c3f30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:56.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.886+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 >> 172.21.15.50:0/57087615 conn(0x7fd2700a22b0 msgr2=0x7fd27006e790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:56.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.887+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 shutdown_connections 2023-10-30T12:00:56.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.887+0000 7fd2754b4640 1 -- 172.21.15.50:0/57087615 wait complete. 2023-10-30T12:00:56.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:56.887+0000 7fd2754b4640 1 librados: shutdown 2023-10-30T12:00:56.899 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T12:00:56.899 INFO:teuthology.run_tasks:Running task mon_thrash... 2023-10-30T12:00:56.949 INFO:tasks.mon_thrash:Beginning mon_thrash... 2023-10-30T12:00:56.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:00:57.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.055+0000 7f4164b97640 10 monclient: get_monmap_and_config 2023-10-30T12:00:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.055+0000 7f4164b97640 10 monclient: build_initial_monmap 2023-10-30T12:00:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.055+0000 7f4164b97640 10 monclient: monmap: 2023-10-30T12:00:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:57.056145+0000 2023-10-30T12:00:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:57.056145+0000 2023-10-30T12:00:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:57.059 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.055+0000 7f4164b97640 1 Processor -- start 2023-10-30T12:00:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.055+0000 7f4164b97640 1 -- start start 2023-10-30T12:00:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.055+0000 7f4164b97640 10 monclient: init 2023-10-30T12:00:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T12:00:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f41600a4b60 0x7f41601326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient(hunting): picked mon.noname-a con 0x7f41600a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4160099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4160132be0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41600a4f40 con 0x7f4160132be0 2023-10-30T12:00:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41600a4150 con 0x7f4160099a40 2023-10-30T12:00:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41600a3d70 con 0x7f41600a4b60 2023-10-30T12:00:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f4164b97640 10 monclient(hunting): authenticate will time out at 902.239340s 2023-10-30T12:00:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f41600a4b60 0x7f41601326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415ed9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4160132be0 0x7f41600a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60912/0 (socket says 172.21.15.50:60912) 2023-10-30T12:00:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415ed9d640 1 -- 172.21.15.50:0/2449284707 learned_addr learned my addr 172.21.15.50:0/2449284707 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): get_auth_request con 0x7f41600a4b60 auth_method 0 2023-10-30T12:00:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2023-10-30T12:00:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 1 -- 172.21.15.50:0/2449284707 >> v1:172.21.15.118:6789/0 conn(0x7f4160099a40 legacy=0x7f41600a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 1 -- 172.21.15.50:0/2449284707 >> v1:172.21.15.50:6789/0 conn(0x7f4160132be0 legacy=0x7f41600a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:00:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 1 monclient: found mon.noname-a 2023-10-30T12:00:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415e59c640 1 -- 172.21.15.50:0/2449284707 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41600a3170 con 0x7f41600a4b60 2023-10-30T12:00:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415d59a640 1 -- 172.21.15.50:0/2449284707 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265094846 0 0) 0x7f41600a4f40 con 0x7f4160132be0 2023-10-30T12:00:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.056+0000 7f415d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415e59c640 1 --2- 172.21.15.50:0/2449284707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f41601326a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f41580097a0 tx=0x7f415802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62577f25b35c9e75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 1 -- 172.21.15.50:0/2449284707 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415802f050 con 0x7f41600a4b60 2023-10-30T12:00:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: dump: 2023-10-30T12:00:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.076 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: _finish_auth 0 2023-10-30T12:00:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:00:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f4164b97640 5 monclient: authenticate success, global_id 4341 2023-10-30T12:00:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f4164b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 1 -- 172.21.15.50:0/2449284707 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4158031700 con 0x7f41600a4b60 2023-10-30T12:00:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f4164b97640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f4164b97640 4 set_mon_vals no callback set 2023-10-30T12:00:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 1 -- 172.21.15.50:0/2449284707 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158031a90 con 0x7f41600a4b60 2023-10-30T12:00:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.057+0000 7f415d59a640 10 monclient: dump: 2023-10-30T12:00:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.081 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 10 monclient: shutdown 2023-10-30T12:00:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 1 -- 172.21.15.50:0/2449284707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 msgr2=0x7f41601326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 1 --2- 172.21.15.50:0/2449284707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f41601326a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f41580097a0 tx=0x7f415802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 1 -- 172.21.15.50:0/2449284707 shutdown_connections 2023-10-30T12:00:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 1 --2- 172.21.15.50:0/2449284707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f41601326a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 1 -- 172.21.15.50:0/2449284707 >> 172.21.15.50:0/2449284707 conn(0x7f41600a8060 msgr2=0x7f41600a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 1 -- 172.21.15.50:0/2449284707 shutdown_connections 2023-10-30T12:00:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.058+0000 7f4164b97640 1 -- 172.21.15.50:0/2449284707 wait complete. 2023-10-30T12:00:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient: build_initial_monmap 2023-10-30T12:00:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient: monmap: 2023-10-30T12:00:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:57.085 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 librados: starting msgr at 2023-10-30T12:00:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 librados: starting objecter 2023-10-30T12:00:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 Processor -- start 2023-10-30T12:00:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 -- start start 2023-10-30T12:00:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 librados: setting wanted keys 2023-10-30T12:00:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 librados: calling monclient init 2023-10-30T12:00:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient: init 2023-10-30T12:00:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:00:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f4160136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): picked mon.noname-a con 0x7f41600a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4160132be0 0x7f4160137290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): picked mon.noname-c con 0x7f4160132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41601377d0 0x7f41601d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): picked mon.noname-b con 0x7f41601377d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600a1940 con 0x7f41600a4b60 2023-10-30T12:00:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416013a590 con 0x7f41601377d0 2023-10-30T12:00:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416013d7c0 con 0x7f4160132be0 2023-10-30T12:00:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f4164b97640 10 monclient(hunting): authenticate will time out at 902.242892s 2023-10-30T12:00:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f415e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f4160136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f415e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f4160136d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38220/0 (socket says 172.21.15.50:38220) 2023-10-30T12:00:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f415e59c640 1 -- 172.21.15.50:0/1388285102 learned_addr learned my addr 172.21.15.50:0/1388285102 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f415e59c640 10 monclient(hunting): get_auth_request con 0x7f41600a4b60 auth_method 0 2023-10-30T12:00:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.059+0000 7f415dd9b640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4160132be0 0x7f4160137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 10 monclient(hunting): get_auth_request con 0x7f4160132be0 auth_method 0 2023-10-30T12:00:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415ed9d640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41601377d0 0x7f41601d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415ed9d640 10 monclient(hunting): get_auth_request con 0x7f41601377d0 auth_method 0 2023-10-30T12:00:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2023-10-30T12:00:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 1 -- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4160132be0 msgr2=0x7f4160137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4160132be0 0x7f4160137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 1 -- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41601377d0 msgr2=0x7f41601d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41601377d0 0x7f41601d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:00:57.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 1 monclient: found mon.noname-a 2023-10-30T12:00:57.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:57.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:57.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 1 -- 172.21.15.50:0/1388285102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158009400 con 0x7f41600a4b60 2023-10-30T12:00:57.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415dd9b640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4160132be0 0x7f4160137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:57.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415e59c640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f4160136d50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41580098c0 tx=0x7f41580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 client.?.objecter ms_handle_connect 0x7f41600a4b60 2023-10-30T12:00:57.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:57.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 1 -- 172.21.15.50:0/1388285102 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158033c30 con 0x7f41600a4b60 2023-10-30T12:00:57.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:57.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: dump: 2023-10-30T12:00:57.097 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.097 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.097 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.097 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.097 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.098 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.098 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.098 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.098 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.099 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: _finish_auth 0 2023-10-30T12:00:57.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:57.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:57.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 5 monclient: authenticate success, global_id 4347 2023-10-30T12:00:57.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 1 -- 172.21.15.50:0/1388285102 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4158033dd0 con 0x7f41600a4b60 2023-10-30T12:00:57.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:57.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 10 monclient: _renew_subs 2023-10-30T12:00:57.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 1 -- 172.21.15.50:0/1388285102 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158004270 con 0x7f41600a4b60 2023-10-30T12:00:57.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:57.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41600a03a0 con 0x7f41600a4b60 2023-10-30T12:00:57.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:57.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f414effd640 10 monclient: dump: 2023-10-30T12:00:57.101 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.102 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.102 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.102 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.102 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.102 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.102 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.103 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.103 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.103 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f415cd99640 4 set_mon_vals no callback set 2023-10-30T12:00:57.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:57.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 10 monclient: _renew_subs 2023-10-30T12:00:57.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:57.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4160138780 con 0x7f41600a4b60 2023-10-30T12:00:57.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.060+0000 7f4164b97640 1 librados: init done 2023-10-30T12:00:57.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f414effd640 1 -- 172.21.15.50:0/1388285102 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f415803a070 con 0x7f41600a4b60 2023-10-30T12:00:57.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f4164b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f4164b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f4164b97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:57.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41600739a0 con 0x7f41600a4b60 2023-10-30T12:00:57.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f414effd640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f41400392f0 0x7f414003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f414effd640 1 -- 172.21.15.50:0/1388285102 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f415806c240 con 0x7f41600a4b60 2023-10-30T12:00:57.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f414effd640 10 client.4347.objecter ms_dispatch 0x7f4160000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:57.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f414effd640 3 client.4347.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:57.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f414effd640 3 client.4347.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:57.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.061+0000 7f414effd640 20 client.4347.objecter dump_active .. 0 homeless 2023-10-30T12:00:57.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.062+0000 7f415dd9b640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f41400392f0 0x7f414003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.062+0000 7f415dd9b640 10 monclient: get_auth_request con 0x7f41400392f0 auth_method 0 2023-10-30T12:00:57.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.062+0000 7f415dd9b640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f41400392f0 0x7f414003b7b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4160138420 tx=0x7f4154009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.062+0000 7f414effd640 10 client.4347.objecter ms_handle_connect 0x7f41400392f0 2023-10-30T12:00:57.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.066+0000 7f414effd640 1 -- 172.21.15.50:0/1388285102 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41580043d0 con 0x7f41600a4b60 2023-10-30T12:00:57.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.066+0000 7f414effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.066+0000 7f414effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:57.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.246+0000 7f4164b97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:57.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.246+0000 7f4164b97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:57.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.246+0000 7f4164b97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:57.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.246+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f416004e160 con 0x7f41600a4b60 2023-10-30T12:00:57.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.247+0000 7f414effd640 1 -- 172.21.15.50:0/1388285102 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12154 (secure 0 0 0) 0x7f4158038020 con 0x7f41600a4b60 2023-10-30T12:00:57.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.247+0000 7f414effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:00:57.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.247+0000 7f414effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:57.250 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:57.250 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":16,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","created":"2023-10-30T12:00:40.280079+0000","modified":"2023-10-30T12:00:53.472055+0000","last_up_change":"2023-10-30T12:00:48.401197+0000","last_in_change":"2023-10-30T12:00:41.981126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:00:48.447131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:00:50.466441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dba8cbf1-bc2e-4420-957f-11ec607e814e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6817","nonce":47190190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6819","nonce":47190190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6823","nonce":47190190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6821","nonce":47190190}]},"public_addr":"172.21.15.50:6817/47190190","cluster_addr":"172.21.15.50:6819/47190190","heartbeat_back_addr":"172.21.15.50:6823/47190190","heartbeat_front_addr":"172.21.15.50:6821/47190190","state":["exists","up"]},{"osd":1,"uuid":"c13ebbef-aa1a-4da7-b5f6-26b02e43955c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6809","nonce":659398918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6811","nonce":659398918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6815","nonce":659398918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6813","nonce":659398918}]},"public_addr":"172.21.15.50:6809/659398918","cluster_addr":"172.21.15.50:6811/659398918","heartbeat_back_addr":"172.21.15.50:6815/659398918","heartbeat_front_addr":"172.21.15.50:6813/659398918","state":["exists","up"]},{"osd":2,"uuid":"23116074-425b-4664-a1be-0e40bba62fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6801","nonce":3518075216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6803","nonce":3518075216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6807","nonce":3518075216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6805","nonce":3518075216}]},"public_addr":"172.21.15.50:6801/3518075216","cluster_addr":"172.21.15.50:6803/3518075216","heartbeat_back_addr":"172.21.15.50:6807/3518075216","heartbeat_front_addr":"172.21.15.50:6805/3518075216","state":["exists","up"]},{"osd":3,"uuid":"515e513c-cb67-4495-ba25-68cb35215c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6810","nonce":1009584072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6814","nonce":1009584072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6822","nonce":1009584072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6818","nonce":1009584072}]},"public_addr":"172.21.15.118:6810/1009584072","cluster_addr":"172.21.15.118:6814/1009584072","heartbeat_back_addr":"172.21.15.118:6822/1009584072","heartbeat_front_addr":"172.21.15.118:6818/1009584072","state":["exists","up"]},{"osd":4,"uuid":"fa9df8af-676c-49c6-9888-0e330fbbc4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6811","nonce":4272127357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6815","nonce":4272127357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6823","nonce":4272127357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6819","nonce":4272127357}]},"public_addr":"172.21.15.118:6811/4272127357","cluster_addr":"172.21.15.118:6815/4272127357","heartbeat_back_addr":"172.21.15.118:6823/4272127357","heartbeat_front_addr":"172.21.15.118:6819/4272127357","state":["exists","up"]},{"osd":5,"uuid":"79b07bf0-2944-4413-b6e1-23ecbc533add","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6801","nonce":1399177807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6803","nonce":1399177807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6807","nonce":1399177807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6805","nonce":1399177807}]},"public_addr":"172.21.15.118:6801/1399177807","cluster_addr":"172.21.15.118:6803/1399177807","heartbeat_back_addr":"172.21.15.118:6807/1399177807","heartbeat_front_addr":"172.21.15.118:6805/1399177807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.083230+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.120938+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.049176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.192079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:00:57.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 10 librados: watch_flush enter 2023-10-30T12:00:57.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 10 librados: watch_flush exit 2023-10-30T12:00:57.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 20 client.4347.objecter shutdown clearing up homeless session... 2023-10-30T12:00:57.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 10 client.4347.objecter successfully canceled tick 2023-10-30T12:00:57.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f41400392f0 msgr2=0x7f414003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f41400392f0 0x7f414003b7b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4160138420 tx=0x7f4154009060 comp rx=0 tx=0).stop 2023-10-30T12:00:57.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 10 monclient: shutdown 2023-10-30T12:00:57.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:57.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 msgr2=0x7f4160136d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f4160136d50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41580098c0 tx=0x7f41580046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 shutdown_connections 2023-10-30T12:00:57.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f41400392f0 0x7f414003b7b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41601377d0 0x7f41601d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4160132be0 0x7f4160137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 --2- 172.21.15.50:0/1388285102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41600a4b60 0x7f4160136d50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.248+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 >> 172.21.15.50:0/1388285102 conn(0x7f41600a8060 msgr2=0x7f41600a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:57.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.249+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 shutdown_connections 2023-10-30T12:00:57.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.249+0000 7f4164b97640 1 -- 172.21.15.50:0/1388285102 wait complete. 2023-10-30T12:00:57.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.249+0000 7f4164b97640 1 librados: shutdown 2023-10-30T12:00:57.261 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:00:48.447131+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-30T12:00:50.466441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:00:57.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:00:57.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.368+0000 7f8ea3cda640 10 monclient: get_monmap_and_config 2023-10-30T12:00:57.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.368+0000 7f8ea3cda640 10 monclient: build_initial_monmap 2023-10-30T12:00:57.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.368+0000 7f8ea3cda640 10 monclient: monmap: 2023-10-30T12:00:57.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:57.371 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.371 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:57.369127+0000 2023-10-30T12:00:57.371 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:57.369127+0000 2023-10-30T12:00:57.371 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:57.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.372 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:57.372 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:57.372 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:57.372 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:57.373 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:57.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.368+0000 7f8ea3cda640 1 Processor -- start 2023-10-30T12:00:57.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.368+0000 7f8ea3cda640 1 -- start start 2023-10-30T12:00:57.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.368+0000 7f8ea3cda640 10 monclient: init 2023-10-30T12:00:57.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T12:00:57.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e9c1327f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:57.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e9c099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:00:57.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8e9c13c180 0x7f8e9c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9c13c180 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:57.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a1620 con 0x7f8e9c1327f0 2023-10-30T12:00:57.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a1b20 con 0x7f8e9c099a40 2023-10-30T12:00:57.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9c09a6b0 con 0x7f8e9c13c180 2023-10-30T12:00:57.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea3cda640 10 monclient(hunting): authenticate will time out at 902.552408s 2023-10-30T12:00:57.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea1a76640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8e9c1327f0 0x7f8e9c132bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60916/0 (socket says 172.21.15.50:60916) 2023-10-30T12:00:57.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea1a76640 1 -- 172.21.15.50:0/354583090 learned_addr learned my addr 172.21.15.50:0/354583090 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:57.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264283640 0 0) 0x7f8e9c0a1620 con 0x7f8e9c1327f0 2023-10-30T12:00:57.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 10 monclient(hunting): _negotiate 2023-10-30T12:00:57.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 10 monclient(hunting): my global_id is 4350 2023-10-30T12:00:57.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e84003520 con 0x7f8e9c1327f0 2023-10-30T12:00:57.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 1 --2- 172.21.15.50:0/354583090 >> v2:172.21.15.118:3300/0 conn(0x7f8e9c13c180 0x7f8e9c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 10 monclient(hunting): get_auth_request con 0x7f8e9c13c180 auth_method 0 2023-10-30T12:00:57.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271679918 0 0) 0x7f8e84003520 con 0x7f8e9c1327f0 2023-10-30T12:00:57.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 >> v2:172.21.15.118:3300/0 conn(0x7f8e9c13c180 msgr2=0x7f8e9c13e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:57.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 --2- 172.21.15.50:0/354583090 >> v2:172.21.15.118:3300/0 conn(0x7f8e9c13c180 0x7f8e9c13e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 >> v1:172.21.15.118:6789/0 conn(0x7f8e9c099a40 legacy=0x7f8e9c13aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 10 monclient: _finish_hunting 0 2023-10-30T12:00:57.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:00:57.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:57.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:00:57.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.369+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c09a0f0 con 0x7f8e9c1327f0 2023-10-30T12:00:57.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient: _finish_auth 0 2023-10-30T12:00:57.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient: _check_auth_tickets 2023-10-30T12:00:57.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:57.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea3cda640 5 monclient: authenticate success, global_id 4350 2023-10-30T12:00:57.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f8e94001670 con 0x7f8e9c1327f0 2023-10-30T12:00:57.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:00:57.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient: dump: 2023-10-30T12:00:57.392 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.392 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.392 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.392 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.392 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.392 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.393 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.393 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.393 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.393 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:00:57.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 >> v1:172.21.15.50:6789/0 conn(0x7f8e9c1327f0 legacy=0x7f8e9c132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:00:57.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e84003970 0x7f8e84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): picked mon.c con 0x7f8e84003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:57.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e840047c0 0x7f8e84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): picked mon.a con 0x7f8e840047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:57.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e840052a0 0x7f8e8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): picked mon.b con 0x7f8e840052a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:57.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c09a6b0 con 0x7f8e840047c0 2023-10-30T12:00:57.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8400c170 con 0x7f8e840052a0 2023-10-30T12:00:57.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8400c330 con 0x7f8e84003970 2023-10-30T12:00:57.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea0a74640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:00:57.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea3cda640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:57.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e84003970 0x7f8e84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): get_auth_request con 0x7f8e84003970 auth_method 0 2023-10-30T12:00:57.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e840047c0 0x7f8e84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:57.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 10 monclient(hunting): get_auth_request con 0x7f8e840047c0 auth_method 0 2023-10-30T12:00:57.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:57.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e840052a0 0x7f8e8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:57.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 10 monclient(hunting): get_auth_request con 0x7f8e840052a0 auth_method 0 2023-10-30T12:00:57.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:57.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1275640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:57.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:57.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient(hunting): handle_auth_done global_id 4350 payload 293 2023-10-30T12:00:57.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 1 -- 172.21.15.50:0/354583090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e840052a0 msgr2=0x7f8e8400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:57.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e840052a0 0x7f8e8400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 1 -- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e840047c0 msgr2=0x7f8e84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:57.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e840047c0 0x7f8e84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient: _finish_hunting 0 2023-10-30T12:00:57.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 1 monclient: found mon.c 2023-10-30T12:00:57.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:57.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 1 -- 172.21.15.50:0/354583090 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84003520 con 0x7f8e84003970 2023-10-30T12:00:57.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient: _finish_auth 0 2023-10-30T12:00:57.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 10 monclient: _check_auth_tickets 2023-10-30T12:00:57.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea1a76640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:57.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.370+0000 7f8ea2277640 1 -- 172.21.15.50:0/354583090 reap_dead start 2023-10-30T12:00:57.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e84003970 0x7f8e84003d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400c210 tx=0x7f8e840057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a1d50f36ce85a1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94005ad0 con 0x7f8e84003970 2023-10-30T12:00:57.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:57.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 10 monclient: dump: 2023-10-30T12:00:57.405 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.405 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.405 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.405 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.406 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.406 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.406 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.406 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.406 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.407 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:57.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e94005c30 con 0x7f8e84003970 2023-10-30T12:00:57.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:57.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:57.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 4 set_mon_vals no callback set 2023-10-30T12:00:57.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 1 -- 172.21.15.50:0/354583090 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9402ca20 con 0x7f8e84003970 2023-10-30T12:00:57.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:57.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea0a74640 10 monclient: dump: 2023-10-30T12:00:57.409 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.410 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 10 monclient: shutdown 2023-10-30T12:00:57.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:57.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 1 -- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e84003970 msgr2=0x7f8e84003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e84003970 0x7f8e84003d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400c210 tx=0x7f8e840057f0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 1 -- 172.21.15.50:0/354583090 shutdown_connections 2023-10-30T12:00:57.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/354583090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e84003970 0x7f8e84003d90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.371+0000 7f8ea3cda640 1 -- 172.21.15.50:0/354583090 >> 172.21.15.50:0/354583090 conn(0x7f8e9c0a82f0 msgr2=0x7f8e9c0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:57.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.372+0000 7f8ea3cda640 1 -- 172.21.15.50:0/354583090 shutdown_connections 2023-10-30T12:00:57.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.372+0000 7f8ea3cda640 1 -- 172.21.15.50:0/354583090 wait complete. 2023-10-30T12:00:57.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.372+0000 7f8ea3cda640 10 monclient: build_initial_monmap 2023-10-30T12:00:57.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.372+0000 7f8ea3cda640 10 monclient: monmap: 2023-10-30T12:00:57.412 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:57.413 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:57.413 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:57.413 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:57.413 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:57.413 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.414 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:57.414 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:57.414 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:57.414 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.372+0000 7f8ea3cda640 1 librados: starting msgr at 2023-10-30T12:00:57.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.372+0000 7f8ea3cda640 1 librados: starting objecter 2023-10-30T12:00:57.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.372+0000 7f8ea3cda640 1 Processor -- start 2023-10-30T12:00:57.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 -- start start 2023-10-30T12:00:57.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 librados: setting wanted keys 2023-10-30T12:00:57.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 librados: calling monclient init 2023-10-30T12:00:57.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient: init 2023-10-30T12:00:57.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:57.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e84003970 0x7f8e9c131210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): picked mon.noname-b con 0x7f8e84003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:57.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e9c131750 0x7f8e9c131bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9c131750 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:57.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e9c135f20 0x7f8e9c1363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9c135f20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:57.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c1393e0 con 0x7f8e9c131750 2023-10-30T12:00:57.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c0a17f0 con 0x7f8e84003970 2023-10-30T12:00:57.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c13b8a0 con 0x7f8e9c135f20 2023-10-30T12:00:57.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea3cda640 10 monclient(hunting): authenticate will time out at 902.556519s 2023-10-30T12:00:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e9c131750 0x7f8e9c131bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e9c135f20 0x7f8e9c1363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e9c131750 0x7f8e9c131bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38240/0 (socket says 172.21.15.50:38240) 2023-10-30T12:00:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): get_auth_request con 0x7f8e9c135f20 auth_method 0 2023-10-30T12:00:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 1 -- 172.21.15.50:0/876833132 learned_addr learned my addr 172.21.15.50:0/876833132 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 10 monclient(hunting): get_auth_request con 0x7f8e9c131750 auth_method 0 2023-10-30T12:00:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e84003970 0x7f8e9c131210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1a76640 10 monclient(hunting): get_auth_request con 0x7f8e84003970 auth_method 0 2023-10-30T12:00:57.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1a76640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1a76640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2023-10-30T12:00:57.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 1 -- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e84003970 msgr2=0x7f8e9c131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e84003970 0x7f8e9c131210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 1 -- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e9c131750 msgr2=0x7f8e9c131bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e9c131750 0x7f8e9c131bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient: _finish_hunting 0 2023-10-30T12:00:57.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 1 monclient: found mon.noname-c 2023-10-30T12:00:57.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:57.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea2277640 1 -- 172.21.15.50:0/876833132 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c1396f0 con 0x7f8e9c135f20 2023-10-30T12:00:57.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e84003970 0x7f8e9c131210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:57.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.373+0000 7f8ea1275640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e9c131750 0x7f8e9c131bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:57.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea2277640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e9c135f20 0x7f8e9c1363a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0a7390 tx=0x7f8e8c00f580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 client.?.objecter ms_handle_connect 0x7f8e9c135f20 2023-10-30T12:00:57.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:57.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 1 -- 172.21.15.50:0/876833132 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c00e040 con 0x7f8e9c135f20 2023-10-30T12:00:57.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:57.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: dump: 2023-10-30T12:00:57.427 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.427 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.427 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.427 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.427 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.428 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.428 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.428 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.428 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.428 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:57.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:57.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:57.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 1 -- 172.21.15.50:0/876833132 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c00f040 con 0x7f8e9c135f20 2023-10-30T12:00:57.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 5 monclient: authenticate success, global_id 4277 2023-10-30T12:00:57.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:57.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 10 monclient: _renew_subs 2023-10-30T12:00:57.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 1 -- 172.21.15.50:0/876833132 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c003bf0 con 0x7f8e9c135f20 2023-10-30T12:00:57.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c1399d0 con 0x7f8e9c135f20 2023-10-30T12:00:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9bfff640 4 set_mon_vals no callback set 2023-10-30T12:00:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8e9a7fc640 10 monclient: dump: 2023-10-30T12:00:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.433 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 10 monclient: _renew_subs 2023-10-30T12:00:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c13a720 con 0x7f8e9c135f20 2023-10-30T12:00:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.374+0000 7f8ea3cda640 1 librados: init done 2023-10-30T12:00:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 1 -- 172.21.15.50:0/876833132 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8e8c003d50 con 0x7f8e9c135f20 2023-10-30T12:00:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8ea3cda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8ea3cda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8ea3cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e880392f0 0x7f8e8803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c0a64e0 con 0x7f8e9c135f20 2023-10-30T12:00:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 1 -- 172.21.15.50:0/876833132 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f8e8c04b090 con 0x7f8e9c135f20 2023-10-30T12:00:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 10 client.4277.objecter ms_dispatch 0x7f8e9c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 3 client.4277.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 3 client.4277.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 20 client.4277.objecter dump_active .. 0 homeless 2023-10-30T12:00:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e880392f0 0x7f8e8803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8ea1a76640 10 monclient: get_auth_request con 0x7f8e880392f0 auth_method 0 2023-10-30T12:00:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8e9a7fc640 10 client.4277.objecter ms_handle_connect 0x7f8e880392f0 2023-10-30T12:00:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.375+0000 7f8ea1a76640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e880392f0 0x7f8e8803b7b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e94034000 tx=0x7f8e94009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.379+0000 7f8e9a7fc640 1 -- 172.21.15.50:0/876833132 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8e8c004030 con 0x7f8e9c135f20 2023-10-30T12:00:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.379+0000 7f8e9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.379+0000 7f8e9a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.555+0000 7f8ea3cda640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2023-10-30T12:00:57.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.555+0000 7f8ea3cda640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2023-10-30T12:00:57.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.555+0000 7f8ea3cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.555+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e9c04e1d0 con 0x7f8e9c135f20 2023-10-30T12:00:57.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.555+0000 7f8e9a7fc640 1 -- 172.21.15.50:0/876833132 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e8c01c580 con 0x7f8e9c135f20 2023-10-30T12:00:57.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.555+0000 7f8e9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2023-10-30T12:00:57.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.555+0000 7f8e9a7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:57.559 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-10-30T12:00:57.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 10 librados: watch_flush enter 2023-10-30T12:00:57.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 10 librados: watch_flush exit 2023-10-30T12:00:57.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 20 client.4277.objecter shutdown clearing up homeless session... 2023-10-30T12:00:57.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 10 client.4277.objecter successfully canceled tick 2023-10-30T12:00:57.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e880392f0 msgr2=0x7f8e8803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e880392f0 0x7f8e8803b7b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e94034000 tx=0x7f8e94009030 comp rx=0 tx=0).stop 2023-10-30T12:00:57.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 10 monclient: shutdown 2023-10-30T12:00:57.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:57.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e9c135f20 msgr2=0x7f8e9c1363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e9c135f20 0x7f8e9c1363a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0a7390 tx=0x7f8e8c00f580 comp rx=0 tx=0).stop 2023-10-30T12:00:57.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 shutdown_connections 2023-10-30T12:00:57.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e880392f0 0x7f8e8803b7b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e9c135f20 0x7f8e9c1363a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e9c131750 0x7f8e9c131bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.557+0000 7f8ea3cda640 1 --2- 172.21.15.50:0/876833132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e84003970 0x7f8e9c131210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.558+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 >> 172.21.15.50:0/876833132 conn(0x7f8e9c0a82f0 msgr2=0x7f8e9c0a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:57.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.558+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 shutdown_connections 2023-10-30T12:00:57.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.558+0000 7f8ea3cda640 1 -- 172.21.15.50:0/876833132 wait complete. 2023-10-30T12:00:57.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.558+0000 7f8ea3cda640 1 librados: shutdown 2023-10-30T12:00:57.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-30T12:00:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.676+0000 7fbf54d00640 10 monclient: get_monmap_and_config 2023-10-30T12:00:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.676+0000 7fbf54d00640 10 monclient: build_initial_monmap 2023-10-30T12:00:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.676+0000 7fbf54d00640 10 monclient: monmap: 2023-10-30T12:00:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:57.677427+0000 2023-10-30T12:00:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:57.677427+0000 2023-10-30T12:00:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:57.680 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.676+0000 7fbf54d00640 1 Processor -- start 2023-10-30T12:00:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.676+0000 7fbf54d00640 1 -- start start 2023-10-30T12:00:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.676+0000 7fbf54d00640 10 monclient: init 2023-10-30T12:00:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-10-30T12:00:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf5009bb70 0x7fbf50098c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5009bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf5009bf50 0x7fbf5009c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5009bf50 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf50132810 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5009d720 con 0x7fbf50132810 2023-10-30T12:00:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf50097c60 con 0x7fbf5009bf50 2023-10-30T12:00:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5009d040 con 0x7fbf5009bb70 2023-10-30T12:00:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf54d00640 10 monclient(hunting): authenticate will time out at 902.860569s 2023-10-30T12:00:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf5009bf50 0x7fbf5009c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf5009bb70 0x7fbf50098c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf5009bf50 0x7fbf5009c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42196/0 (socket says 172.21.15.50:42196) 2023-10-30T12:00:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4ed9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbf50132810 0x7fbf50132ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35232/0 (socket says 172.21.15.50:35232) 2023-10-30T12:00:57.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 1 -- 172.21.15.50:0/4170776356 learned_addr learned my addr 172.21.15.50:0/4170776356 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:57.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): get_auth_request con 0x7fbf5009bf50 auth_method 0 2023-10-30T12:00:57.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 10 monclient(hunting): get_auth_request con 0x7fbf5009bb70 auth_method 0 2023-10-30T12:00:57.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.677+0000 7fbf4dd9b640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2023-10-30T12:00:57.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 1 -- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf5009bb70 msgr2=0x7fbf50098c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:57.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4d59a640 1 -- 172.21.15.50:0/4170776356 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591367882 0 0) 0x7fbf5009d720 con 0x7fbf50132810 2023-10-30T12:00:57.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 1 --2- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf5009bb70 0x7fbf50098c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 1 -- 172.21.15.50:0/4170776356 >> v1:172.21.15.50:6789/0 conn(0x7fbf50132810 legacy=0x7fbf50132ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:00:57.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 1 monclient: found mon.noname-a 2023-10-30T12:00:57.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:57.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:57.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 1 -- 172.21.15.50:0/4170776356 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50097980 con 0x7fbf5009bf50 2023-10-30T12:00:57.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4e59c640 1 --2- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf5009bb70 0x7fbf50098c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:57.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:57.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4dd9b640 1 --2- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf5009c370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009870 tx=0x7fbf4002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d465cfb5febb96f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4d59a640 1 -- 172.21.15.50:0/4170776356 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf400317c0 con 0x7fbf5009bf50 2023-10-30T12:00:57.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:57.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4d59a640 10 monclient: dump: 2023-10-30T12:00:57.697 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.697 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.698 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.698 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.698 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.698 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.698 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.699 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.699 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.699 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.678+0000 7fbf4d59a640 10 monclient: _finish_auth 0 2023-10-30T12:00:57.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:00:57.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:57.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 5 monclient: authenticate success, global_id 4356 2023-10-30T12:00:57.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:57.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 1 -- 172.21.15.50:0/4170776356 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf40031920 con 0x7fbf5009bf50 2023-10-30T12:00:57.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:57.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:57.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 1 -- 172.21.15.50:0/4170776356 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40031cb0 con 0x7fbf5009bf50 2023-10-30T12:00:57.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 4 set_mon_vals no callback set 2023-10-30T12:00:57.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:57.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf4d59a640 10 monclient: dump: 2023-10-30T12:00:57.702 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.702 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.702 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.702 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.702 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.703 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.703 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 10 monclient: shutdown 2023-10-30T12:00:57.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:57.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 -- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 msgr2=0x7fbf5009c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 --2- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf5009c370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009870 tx=0x7fbf4002b860 comp rx=0 tx=0).stop 2023-10-30T12:00:57.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 -- 172.21.15.50:0/4170776356 shutdown_connections 2023-10-30T12:00:57.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 --2- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf5009c370 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 --2- 172.21.15.50:0/4170776356 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf5009bb70 0x7fbf50098c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 -- 172.21.15.50:0/4170776356 >> 172.21.15.50:0/4170776356 conn(0x7fbf500a0480 msgr2=0x7fbf500a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 -- 172.21.15.50:0/4170776356 shutdown_connections 2023-10-30T12:00:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.679+0000 7fbf54d00640 1 -- 172.21.15.50:0/4170776356 wait complete. 2023-10-30T12:00:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 10 monclient: build_initial_monmap 2023-10-30T12:00:57.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 10 monclient: monmap: 2023-10-30T12:00:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:57.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:57.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 1 librados: starting msgr at 2023-10-30T12:00:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 1 librados: starting objecter 2023-10-30T12:00:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 1 Processor -- start 2023-10-30T12:00:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 1 -- start start 2023-10-30T12:00:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 1 librados: setting wanted keys 2023-10-30T12:00:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 1 librados: calling monclient init 2023-10-30T12:00:57.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.680+0000 7fbf54d00640 10 monclient: init 2023-10-30T12:00:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5009bb70 0x7fbf5013d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5009bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:57.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf501c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5009bf50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf50132810 0x7fbf501ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): picked mon.noname-c con 0x7fbf50132810 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5013b920 con 0x7fbf5009bf50 2023-10-30T12:00:57.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5013d070 con 0x7fbf5009bb70 2023-10-30T12:00:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5013ca40 con 0x7fbf50132810 2023-10-30T12:00:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient(hunting): authenticate will time out at 902.864148s 2023-10-30T12:00:57.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf501c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf50132810 0x7fbf501ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf501c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42202/0 (socket says 172.21.15.50:42202) 2023-10-30T12:00:57.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): get_auth_request con 0x7fbf50132810 auth_method 0 2023-10-30T12:00:57.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 1 -- 172.21.15.50:0/550583275 learned_addr learned my addr 172.21.15.50:0/550583275 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:57.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 10 monclient(hunting): get_auth_request con 0x7fbf5009bf50 auth_method 0 2023-10-30T12:00:57.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2023-10-30T12:00:57.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4e59c640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5009bb70 0x7fbf5013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 -- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5009bb70 msgr2=0x7fbf5013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5009bb70 0x7fbf5013d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 -- 172.21.15.50:0/550583275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 msgr2=0x7fbf501c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf501c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:57.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 monclient: found mon.noname-c 2023-10-30T12:00:57.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:57.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 -- 172.21.15.50:0/550583275 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf38009400 con 0x7fbf50132810 2023-10-30T12:00:57.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4e59c640 10 monclient: get_auth_request con 0x7fbf5009bb70 auth_method 0 2023-10-30T12:00:57.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4e59c640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5009bb70 0x7fbf5013d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:00:57.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4ed9d640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf50132810 0x7fbf501ca500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c6d0 tx=0x7fbf4800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 client.?.objecter ms_handle_connect 0x7fbf50132810 2023-10-30T12:00:57.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:57.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 1 -- 172.21.15.50:0/550583275 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48004d60 con 0x7fbf50132810 2023-10-30T12:00:57.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:57.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: dump: 2023-10-30T12:00:57.718 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.719 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.719 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.719 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.719 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.719 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.720 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.720 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.720 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.720 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 1 -- 172.21.15.50:0/550583275 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4800a640 con 0x7fbf50132810 2023-10-30T12:00:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 1 -- 172.21.15.50:0/550583275 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800aa30 con 0x7fbf50132810 2023-10-30T12:00:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 5 monclient: authenticate success, global_id 4283 2023-10-30T12:00:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf46ffd640 10 monclient: dump: 2023-10-30T12:00:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:57.724 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:57.724 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient: _renew_subs 2023-10-30T12:00:57.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf4cd99640 4 set_mon_vals no callback set 2023-10-30T12:00:57.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf400094d0 con 0x7fbf50132810 2023-10-30T12:00:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.681+0000 7fbf54d00640 10 monclient: _renew_subs 2023-10-30T12:00:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.682+0000 7fbf54d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.682+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5013aa10 con 0x7fbf50132810 2023-10-30T12:00:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.682+0000 7fbf54d00640 1 librados: init done 2023-10-30T12:00:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.682+0000 7fbf46ffd640 1 -- 172.21.15.50:0/550583275 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbf480070d0 con 0x7fbf50132810 2023-10-30T12:00:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.682+0000 7fbf46ffd640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf300392f0 0x7fbf3003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf46ffd640 1 -- 172.21.15.50:0/550583275 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fbf48049c60 con 0x7fbf50132810 2023-10-30T12:00:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf46ffd640 10 client.4283.objecter ms_dispatch 0x7fbf50000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf46ffd640 3 client.4283.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf46ffd640 3 client.4283.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf46ffd640 20 client.4283.objecter dump_active .. 0 homeless 2023-10-30T12:00:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf54d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf54d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf54d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf500a1bb0 con 0x7fbf50132810 2023-10-30T12:00:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf4e59c640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf300392f0 0x7fbf3003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf4e59c640 10 monclient: get_auth_request con 0x7fbf300392f0 auth_method 0 2023-10-30T12:00:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf46ffd640 10 client.4283.objecter ms_handle_connect 0x7fbf300392f0 2023-10-30T12:00:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.683+0000 7fbf4e59c640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf300392f0 0x7fbf3003b7b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009c50 tx=0x7fbf38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.687+0000 7fbf46ffd640 1 -- 172.21.15.50:0/550583275 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf48018020 con 0x7fbf50132810 2023-10-30T12:00:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.687+0000 7fbf46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.687+0000 7fbf46ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.861+0000 7fbf54d00640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2023-10-30T12:00:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.861+0000 7fbf54d00640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2023-10-30T12:00:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.861+0000 7fbf54d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.861+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbf5009c830 con 0x7fbf50132810 2023-10-30T12:00:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.862+0000 7fbf46ffd640 1 -- 172.21.15.50:0/550583275 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbf4801b360 con 0x7fbf50132810 2023-10-30T12:00:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.862+0000 7fbf46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2023-10-30T12:00:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.862+0000 7fbf46ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:57.865 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 8 2023-10-30T12:00:57.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 10 librados: watch_flush enter 2023-10-30T12:00:57.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 10 librados: watch_flush exit 2023-10-30T12:00:57.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 20 client.4283.objecter shutdown clearing up homeless session... 2023-10-30T12:00:57.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 10 client.4283.objecter successfully canceled tick 2023-10-30T12:00:57.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf300392f0 msgr2=0x7fbf3003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf300392f0 0x7fbf3003b7b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009c50 tx=0x7fbf38008040 comp rx=0 tx=0).stop 2023-10-30T12:00:57.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 10 monclient: shutdown 2023-10-30T12:00:57.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:57.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf50132810 msgr2=0x7fbf501ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:57.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf50132810 0x7fbf501ca500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c6d0 tx=0x7fbf4800cb90 comp rx=0 tx=0).stop 2023-10-30T12:00:57.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 shutdown_connections 2023-10-30T12:00:57.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf300392f0 0x7fbf3003b7b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf50132810 0x7fbf501ca500 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf5009bf50 0x7fbf501c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 --2- 172.21.15.50:0/550583275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5009bb70 0x7fbf5013d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:57.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.863+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 >> 172.21.15.50:0/550583275 conn(0x7fbf500a0480 msgr2=0x7fbf5006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:57.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.864+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 shutdown_connections 2023-10-30T12:00:57.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.864+0000 7fbf54d00640 1 -- 172.21.15.50:0/550583275 wait complete. 2023-10-30T12:00:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.864+0000 7fbf54d00640 1 librados: shutdown 2023-10-30T12:00:57.876 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2023-10-30T12:00:57.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:00:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.979+0000 7f5f7d96e640 10 monclient: get_monmap_and_config 2023-10-30T12:00:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.979+0000 7f5f7d96e640 10 monclient: build_initial_monmap 2023-10-30T12:00:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.979+0000 7f5f7d96e640 10 monclient: monmap: 2023-10-30T12:00:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:57.980679+0000 2023-10-30T12:00:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:57.980679+0000 2023-10-30T12:00:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:57.985 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:57.985 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:57.985 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 1 Processor -- start 2023-10-30T12:00:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 1 -- start start 2023-10-30T12:00:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient: init 2023-10-30T12:00:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:00:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f780a2dd0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f7806a8e0 0x7f5f780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f7806a8e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5f78071400 0x7f5f7813cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f78071400 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f780a31b0 con 0x7f5f780a2dd0 2023-10-30T12:00:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7806a6a0 con 0x7f5f78071400 2023-10-30T12:00:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7806be90 con 0x7f5f7806a8e0 2023-10-30T12:00:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f7d96e640 10 monclient(hunting): authenticate will time out at 903.163897s 2023-10-30T12:00:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.980+0000 7f5f6ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f7806a8e0 0x7f5f780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f76ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5f780a2dd0 0x7f5f780a46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35242/0 (socket says 172.21.15.50:35242) 2023-10-30T12:00:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f76ffd640 1 -- 172.21.15.50:0/4192573584 learned_addr learned my addr 172.21.15.50:0/4192573584 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): get_auth_request con 0x7f5f7806a8e0 auth_method 0 2023-10-30T12:00:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f777fe640 1 --2- 172.21.15.50:0/4192573584 >> v2:172.21.15.118:3300/0 conn(0x7f5f78071400 0x7f5f7813cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f777fe640 10 monclient(hunting): get_auth_request con 0x7f5f78071400 auth_method 0 2023-10-30T12:00:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f767fc640 1 -- 172.21.15.50:0/4192573584 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227376192 0 0) 0x7f5f780a31b0 con 0x7f5f780a2dd0 2023-10-30T12:00:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f767fc640 10 monclient(hunting): _negotiate 2023-10-30T12:00:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f767fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f767fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f767fc640 10 monclient(hunting): my global_id is 4365 2023-10-30T12:00:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f767fc640 1 -- 172.21.15.50:0/4192573584 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f64003520 con 0x7f5f780a2dd0 2023-10-30T12:00:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2023-10-30T12:00:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 1 -- 172.21.15.50:0/4192573584 >> v2:172.21.15.118:3300/0 conn(0x7f5f78071400 msgr2=0x7f5f7813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 1 --2- 172.21.15.50:0/4192573584 >> v2:172.21.15.118:3300/0 conn(0x7f5f78071400 0x7f5f7813cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 1 -- 172.21.15.50:0/4192573584 >> v1:172.21.15.50:6789/0 conn(0x7f5f780a2dd0 legacy=0x7f5f780a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 1 monclient: found mon.noname-b 2023-10-30T12:00:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.981+0000 7f5f6ffff640 1 -- 172.21.15.50:0/4192573584 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7809e190 con 0x7f5f7806a8e0 2023-10-30T12:00:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f6ffff640 1 --2- 172.21.15.50:0/4192573584 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f7806a8e0 0x7f5f780a4dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5f600097a0 tx=0x7f5f6002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=450e1ceb2ee1ea5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 1 -- 172.21.15.50:0/4192573584 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6002f050 con 0x7f5f7806a8e0 2023-10-30T12:00:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: dump: 2023-10-30T12:00:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.003 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: _finish_auth 0 2023-10-30T12:00:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: _check_auth_tickets 2023-10-30T12:00:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 1 -- 172.21.15.50:0/4192573584 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f60031700 con 0x7f5f7806a8e0 2023-10-30T12:00:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 5 monclient: authenticate success, global_id 4286 2023-10-30T12:00:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 1 -- 172.21.15.50:0/4192573584 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f60031a90 con 0x7f5f7806a8e0 2023-10-30T12:00:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 4 set_mon_vals no callback set 2023-10-30T12:00:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f767fc640 10 monclient: dump: 2023-10-30T12:00:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.007 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 10 monclient: shutdown 2023-10-30T12:00:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 -- 172.21.15.50:0/4192573584 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f7806a8e0 msgr2=0x7f5f780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/4192573584 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f7806a8e0 0x7f5f780a4dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5f600097a0 tx=0x7f5f6002b7a0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 -- 172.21.15.50:0/4192573584 shutdown_connections 2023-10-30T12:00:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/4192573584 >> v2:172.21.15.118:3300/0 conn(0x7f5f78071400 0x7f5f7813cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/4192573584 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f7806a8e0 0x7f5f780a4dd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 -- 172.21.15.50:0/4192573584 >> 172.21.15.50:0/4192573584 conn(0x7f5f780a22b0 msgr2=0x7f5f780a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 -- 172.21.15.50:0/4192573584 shutdown_connections 2023-10-30T12:00:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.982+0000 7f5f7d96e640 1 -- 172.21.15.50:0/4192573584 wait complete. 2023-10-30T12:00:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 10 monclient: build_initial_monmap 2023-10-30T12:00:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 10 monclient: monmap: 2023-10-30T12:00:58.010 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:58.010 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:58.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:58.010 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:58.010 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:58.011 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 1 librados: starting msgr at 2023-10-30T12:00:58.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 1 librados: starting objecter 2023-10-30T12:00:58.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 1 Processor -- start 2023-10-30T12:00:58.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 1 -- start start 2023-10-30T12:00:58.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 1 librados: setting wanted keys 2023-10-30T12:00:58.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 1 librados: calling monclient init 2023-10-30T12:00:58.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.983+0000 7f5f7d96e640 10 monclient: init 2023-10-30T12:00:58.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:58.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:58.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f78071400 0x7f5f781ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f78071400 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f780a2dd0 0x7f5f781caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f780a2dd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f781226d0 0x7f5f78122b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f781226d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7813c520 con 0x7f5f780a2dd0 2023-10-30T12:00:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78124de0 con 0x7f5f78071400 2023-10-30T12:00:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7806b0a0 con 0x7f5f781226d0 2023-10-30T12:00:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient(hunting): authenticate will time out at 903.167162s 2023-10-30T12:00:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f780a2dd0 0x7f5f781caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f781226d0 0x7f5f78122b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f781226d0 0x7f5f78122b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42978/0 (socket says 172.21.15.50:42978) 2023-10-30T12:00:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f780a2dd0 0x7f5f781caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42212/0 (socket says 172.21.15.50:42212) 2023-10-30T12:00:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 -- 172.21.15.50:0/1069209011 learned_addr learned my addr 172.21.15.50:0/1069209011 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): get_auth_request con 0x7f5f781226d0 auth_method 0 2023-10-30T12:00:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 10 monclient(hunting): get_auth_request con 0x7f5f780a2dd0 auth_method 0 2023-10-30T12:00:58.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f78071400 0x7f5f781ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 10 monclient(hunting): get_auth_request con 0x7f5f78071400 auth_method 0 2023-10-30T12:00:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2023-10-30T12:00:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 -- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f78071400 msgr2=0x7f5f781ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f78071400 0x7f5f781ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 -- 172.21.15.50:0/1069209011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f780a2dd0 msgr2=0x7f5f781caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f780a2dd0 0x7f5f781caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient: _finish_hunting 0 2023-10-30T12:00:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 monclient: found mon.noname-c 2023-10-30T12:00:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 -- 172.21.15.50:0/1069209011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f60009400 con 0x7f5f781226d0 2023-10-30T12:00:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f777fe640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f781226d0 0x7f5f78122b60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f70007e10 tx=0x7f5f7000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 client.?.objecter ms_handle_connect 0x7f5f781226d0 2023-10-30T12:00:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 1 -- 172.21.15.50:0/1069209011 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70014620 con 0x7f5f781226d0 2023-10-30T12:00:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: dump: 2023-10-30T12:00:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.026 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.026 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: _finish_auth 0 2023-10-30T12:00:58.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:00:58.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 1 -- 172.21.15.50:0/1069209011 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f700147a0 con 0x7f5f781226d0 2023-10-30T12:00:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 5 monclient: authenticate success, global_id 4289 2023-10-30T12:00:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient: _renew_subs 2023-10-30T12:00:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f780705b0 con 0x7f5f781226d0 2023-10-30T12:00:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 1 -- 172.21.15.50:0/1069209011 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70014b50 con 0x7f5f781226d0 2023-10-30T12:00:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f74ff9640 10 monclient: dump: 2023-10-30T12:00:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.984+0000 7f5f75ffb640 4 set_mon_vals no callback set 2023-10-30T12:00:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f7d96e640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f7d96e640 10 monclient: _renew_subs 2023-10-30T12:00:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f7d96e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f78125ea0 con 0x7f5f781226d0 2023-10-30T12:00:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f7d96e640 1 librados: init done 2023-10-30T12:00:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f74ff9640 1 -- 172.21.15.50:0/1069209011 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5f70014cb0 con 0x7f5f781226d0 2023-10-30T12:00:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f74ff9640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f580392f0 0x7f5f5803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.985+0000 7f5f7d96e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f7d96e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f7d96e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f74ff9640 1 -- 172.21.15.50:0/1069209011 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f5f7004ae60 con 0x7f5f781226d0 2023-10-30T12:00:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f74ff9640 10 client.4289.objecter ms_dispatch 0x7f5f78000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f74ff9640 3 client.4289.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f74ff9640 3 client.4289.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7804e160 con 0x7f5f781226d0 2023-10-30T12:00:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f74ff9640 20 client.4289.objecter dump_active .. 0 homeless 2023-10-30T12:00:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f76ffd640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f580392f0 0x7f5f5803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f76ffd640 10 monclient: get_auth_request con 0x7f5f580392f0 auth_method 0 2023-10-30T12:00:58.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f74ff9640 10 client.4289.objecter ms_handle_connect 0x7f5f580392f0 2023-10-30T12:00:58.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.986+0000 7f5f76ffd640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f580392f0 0x7f5f5803b7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f680011e0 tx=0x7f5f68007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.990+0000 7f5f74ff9640 1 -- 172.21.15.50:0/1069209011 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f700122e0 con 0x7f5f781226d0 2023-10-30T12:00:58.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.990+0000 7f5f74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:57.990+0000 7f5f74ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:58.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.161+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f780a5310 con 0x7f5f580392f0 2023-10-30T12:00:58.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.163+0000 7f5f74ff9640 1 -- 172.21.15.50:0/1069209011 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40580 (secure 0 0 0) 0x7f5f680014d0 con 0x7f5f580392f0 2023-10-30T12:00:58.166 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:58.167 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:00:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 10 librados: watch_flush enter 2023-10-30T12:00:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 10 librados: watch_flush exit 2023-10-30T12:00:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 20 client.4289.objecter shutdown clearing up homeless session... 2023-10-30T12:00:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 10 client.4289.objecter successfully canceled tick 2023-10-30T12:00:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f580392f0 msgr2=0x7f5f5803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f580392f0 0x7f5f5803b7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f680011e0 tx=0x7f5f68007d30 comp rx=0 tx=0).stop 2023-10-30T12:00:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 10 monclient: shutdown 2023-10-30T12:00:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f781226d0 msgr2=0x7f5f78122b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f781226d0 0x7f5f78122b60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f70007e10 tx=0x7f5f7000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 shutdown_connections 2023-10-30T12:00:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f580392f0 0x7f5f5803b7b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f781226d0 0x7f5f78122b60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f780a2dd0 0x7f5f781caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 --2- 172.21.15.50:0/1069209011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f78071400 0x7f5f781ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 >> 172.21.15.50:0/1069209011 conn(0x7f5f780a22b0 msgr2=0x7f5f7806e790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 shutdown_connections 2023-10-30T12:00:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 -- 172.21.15.50:0/1069209011 wait complete. 2023-10-30T12:00:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.165+0000 7f5f7d96e640 1 librados: shutdown 2023-10-30T12:00:58.177 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-10-30T12:00:56.393533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164072,"kb_used_data":1928,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066968,"statfs":{"total":579820584960,"available":579652575232,"internally_reserved":0,"allocated":1974272,"data_stored":1343408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":165927591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000371"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475189+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:00:53.475189+0000","last_peered":"2023-10-30T12:00:53.475189+0000","last_clean":"2023-10-30T12:00:53.475189+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:00:53.475189+0000","last_undegraded":"2023-10-30T12:00:53.475189+0000","last_fullsized":"2023-10-30T12:00:53.475189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475012+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:00:53.475012+0000","last_peered":"2023-10-30T12:00:53.475012+0000","last_clean":"2023-10-30T12:00:53.475012+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:00:53.475012+0000","last_undegraded":"2023-10-30T12:00:53.475012+0000","last_fullsized":"2023-10-30T12:00:53.475012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583189+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:00:53.583189+0000","last_peered":"2023-10-30T12:00:53.583189+0000","last_clean":"2023-10-30T12:00:53.583189+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:00:53.583189+0000","last_undegraded":"2023-10-30T12:00:53.583189+0000","last_fullsized":"2023-10-30T12:00:53.583189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583724+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:00:53.583724+0000","last_peered":"2023-10-30T12:00:53.583724+0000","last_clean":"2023-10-30T12:00:53.583724+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:00:53.583724+0000","last_undegraded":"2023-10-30T12:00:53.583724+0000","last_fullsized":"2023-10-30T12:00:53.583724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475112+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:00:53.475112+0000","last_peered":"2023-10-30T12:00:53.475112+0000","last_clean":"2023-10-30T12:00:53.475112+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:00:53.475112+0000","last_undegraded":"2023-10-30T12:00:53.475112+0000","last_fullsized":"2023-10-30T12:00:53.475112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583196+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:00:53.583196+0000","last_peered":"2023-10-30T12:00:53.583196+0000","last_clean":"2023-10-30T12:00:53.583196+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:00:53.583196+0000","last_undegraded":"2023-10-30T12:00:53.583196+0000","last_fullsized":"2023-10-30T12:00:53.583196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":163,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475070+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:00:53.475070+0000","last_peered":"2023-10-30T12:00:53.475070+0000","last_clean":"2023-10-30T12:00:53.475070+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:00:53.475070+0000","last_undegraded":"2023-10-30T12:00:53.475070+0000","last_fullsized":"2023-10-30T12:00:53.475070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":576520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":576520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:00:58.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:00:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.282+0000 7f8a848f2640 10 monclient: get_monmap_and_config 2023-10-30T12:00:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.282+0000 7f8a848f2640 10 monclient: build_initial_monmap 2023-10-30T12:00:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.282+0000 7f8a848f2640 10 monclient: monmap: 2023-10-30T12:00:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:58.283184+0000 2023-10-30T12:00:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:58.283184+0000 2023-10-30T12:00:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:58.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.282+0000 7f8a848f2640 1 Processor -- start 2023-10-30T12:00:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.282+0000 7f8a848f2640 1 -- start start 2023-10-30T12:00:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.282+0000 7f8a848f2640 10 monclient: init 2023-10-30T12:00:58.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:58.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T12:00:58.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8a800a6070 0x7f8a800a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient(hunting): picked mon.noname-c con 0x7f8a800a6070 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:58.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a800a6450 0x7f8a80135590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient(hunting): picked mon.noname-a con 0x7f8a800a6450 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:58.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a80135ad0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:58.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a8009a270 con 0x7f8a80135ad0 2023-10-30T12:00:58.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a80099f90 con 0x7f8a800a6450 2023-10-30T12:00:58.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8009a550 con 0x7f8a800a6070 2023-10-30T12:00:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a848f2640 10 monclient(hunting): authenticate will time out at 903.466405s 2023-10-30T12:00:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a800a6450 0x7f8a80135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a800a6450 0x7f8a80135590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42220/0 (socket says 172.21.15.50:42220) 2023-10-30T12:00:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 -- 172.21.15.50:0/3740764702 learned_addr learned my addr 172.21.15.50:0/3740764702 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): get_auth_request con 0x7f8a800a6450 auth_method 0 2023-10-30T12:00:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7e59c640 1 --2- 172.21.15.50:0/3740764702 >> v2:172.21.15.118:3300/0 conn(0x7f8a800a6070 0x7f8a800a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7e59c640 10 monclient(hunting): get_auth_request con 0x7f8a800a6070 auth_method 0 2023-10-30T12:00:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2023-10-30T12:00:58.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 -- 172.21.15.50:0/3740764702 >> v2:172.21.15.118:3300/0 conn(0x7f8a800a6070 msgr2=0x7f8a800a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:58.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 --2- 172.21.15.50:0/3740764702 >> v2:172.21.15.118:3300/0 conn(0x7f8a800a6070 0x7f8a800a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 -- 172.21.15.50:0/3740764702 >> v1:172.21.15.50:6789/0 conn(0x7f8a80135ad0 legacy=0x7f8a8013c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:00:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 monclient: found mon.noname-a 2023-10-30T12:00:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.283+0000 7f8a7dd9b640 1 -- 172.21.15.50:0/3740764702 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8009a830 con 0x7f8a800a6450 2023-10-30T12:00:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7dd9b640 1 --2- 172.21.15.50:0/3740764702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6450 0x7f8a80135590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a800a4ee0 tx=0x7f8a6802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd397bf2dd0ea0a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 1 -- 172.21.15.50:0/3740764702 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a680317e0 con 0x7f8a800a6450 2023-10-30T12:00:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: dump: 2023-10-30T12:00:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.304 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.304 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.304 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.304 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.304 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: _finish_auth 0 2023-10-30T12:00:58.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:00:58.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:58.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 1 -- 172.21.15.50:0/3740764702 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a68031940 con 0x7f8a800a6450 2023-10-30T12:00:58.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a848f2640 5 monclient: authenticate success, global_id 4371 2023-10-30T12:00:58.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a848f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:58.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:58.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a848f2640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:58.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 1 -- 172.21.15.50:0/3740764702 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68031cd0 con 0x7f8a800a6450 2023-10-30T12:00:58.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a848f2640 4 set_mon_vals no callback set 2023-10-30T12:00:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.284+0000 7f8a7d59a640 10 monclient: dump: 2023-10-30T12:00:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.309 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.309 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 10 monclient: shutdown 2023-10-30T12:00:58.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:58.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 -- 172.21.15.50:0/3740764702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6450 msgr2=0x7f8a80135590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3740764702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6450 0x7f8a80135590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a800a4ee0 tx=0x7f8a6802b880 comp rx=0 tx=0).stop 2023-10-30T12:00:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 -- 172.21.15.50:0/3740764702 shutdown_connections 2023-10-30T12:00:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3740764702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6450 0x7f8a80135590 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3740764702 >> v2:172.21.15.118:3300/0 conn(0x7f8a800a6070 0x7f8a800a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 -- 172.21.15.50:0/3740764702 >> 172.21.15.50:0/3740764702 conn(0x7f8a800a81d0 msgr2=0x7f8a800a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:58.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 -- 172.21.15.50:0/3740764702 shutdown_connections 2023-10-30T12:00:58.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.285+0000 7f8a848f2640 1 -- 172.21.15.50:0/3740764702 wait complete. 2023-10-30T12:00:58.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient: build_initial_monmap 2023-10-30T12:00:58.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient: monmap: 2023-10-30T12:00:58.311 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:58.312 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:58.312 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:58.312 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:58.312 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:58.312 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.313 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:58.313 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:58.313 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:58.313 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 librados: starting msgr at 2023-10-30T12:00:58.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 librados: starting objecter 2023-10-30T12:00:58.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 Processor -- start 2023-10-30T12:00:58.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 -- start start 2023-10-30T12:00:58.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 librados: setting wanted keys 2023-10-30T12:00:58.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 librados: calling monclient init 2023-10-30T12:00:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient: init 2023-10-30T12:00:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6070 0x7f8a801ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): picked mon.noname-a con 0x7f8a800a6070 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a800a6450 0x7f8a801cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): picked mon.noname-b con 0x7f8a800a6450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a80135ad0 0x7f8a8011e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): picked mon.noname-c con 0x7f8a80135ad0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80122180 con 0x7f8a800a6070 2023-10-30T12:00:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80120c40 con 0x7f8a800a6450 2023-10-30T12:00:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801325c0 con 0x7f8a80135ad0 2023-10-30T12:00:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a848f2640 10 monclient(hunting): authenticate will time out at 903.469860s 2023-10-30T12:00:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6070 0x7f8a801ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6070 0x7f8a801ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42232/0 (socket says 172.21.15.50:42232) 2023-10-30T12:00:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7e59c640 1 -- 172.21.15.50:0/3385374031 learned_addr learned my addr 172.21.15.50:0/3385374031 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7e59c640 10 monclient(hunting): get_auth_request con 0x7f8a800a6070 auth_method 0 2023-10-30T12:00:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7ed9d640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a80135ad0 0x7f8a8011e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.286+0000 7f8a7e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7ed9d640 10 monclient(hunting): get_auth_request con 0x7f8a80135ad0 auth_method 0 2023-10-30T12:00:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a800a6450 0x7f8a801cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 10 monclient(hunting): get_auth_request con 0x7f8a800a6450 auth_method 0 2023-10-30T12:00:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2023-10-30T12:00:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 1 -- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a80135ad0 msgr2=0x7f8a8011e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a80135ad0 0x7f8a8011e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 1 -- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a800a6450 msgr2=0x7f8a801cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a800a6450 0x7f8a801cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:00:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 1 monclient: found mon.noname-a 2023-10-30T12:00:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 1 -- 172.21.15.50:0/3385374031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a680094d0 con 0x7f8a800a6070 2023-10-30T12:00:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7dd9b640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a800a6450 0x7f8a801cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7e59c640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6070 0x7f8a801ca6f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800d830 tx=0x7f8a7800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 client.?.objecter ms_handle_connect 0x7f8a800a6070 2023-10-30T12:00:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 1 -- 172.21.15.50:0/3385374031 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800b6c0 con 0x7f8a800a6070 2023-10-30T12:00:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: dump: 2023-10-30T12:00:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.328 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 1 -- 172.21.15.50:0/3385374031 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7800b840 con 0x7f8a800a6070 2023-10-30T12:00:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 5 monclient: authenticate success, global_id 4374 2023-10-30T12:00:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 10 monclient: _renew_subs 2023-10-30T12:00:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800a1690 con 0x7f8a800a6070 2023-10-30T12:00:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 1 -- 172.21.15.50:0/3385374031 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800bbf0 con 0x7f8a800a6070 2023-10-30T12:00:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a76ffd640 10 monclient: dump: 2023-10-30T12:00:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.332 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a7cd99640 4 set_mon_vals no callback set 2023-10-30T12:00:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 10 monclient: _renew_subs 2023-10-30T12:00:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a80121ac0 con 0x7f8a800a6070 2023-10-30T12:00:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.287+0000 7f8a848f2640 1 librados: init done 2023-10-30T12:00:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a76ffd640 1 -- 172.21.15.50:0/3385374031 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8a7801e460 con 0x7f8a800a6070 2023-10-30T12:00:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a848f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a848f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a848f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8004e160 con 0x7f8a800a6070 2023-10-30T12:00:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a76ffd640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a76ffd640 1 -- 172.21.15.50:0/3385374031 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f8a7804ad60 con 0x7f8a800a6070 2023-10-30T12:00:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a76ffd640 10 client.4374.objecter ms_dispatch 0x7f8a80000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a76ffd640 3 client.4374.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a76ffd640 3 client.4374.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a76ffd640 20 client.4374.objecter dump_active .. 0 homeless 2023-10-30T12:00:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a7dd9b640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.288+0000 7f8a7dd9b640 10 monclient: get_auth_request con 0x7f8a600392f0 auth_method 0 2023-10-30T12:00:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.289+0000 7f8a76ffd640 10 client.4374.objecter ms_handle_connect 0x7f8a600392f0 2023-10-30T12:00:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.289+0000 7f8a7dd9b640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a68036000 tx=0x7f8a6803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.293+0000 7f8a76ffd640 1 -- 172.21.15.50:0/3385374031 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8a7801c950 con 0x7f8a800a6070 2023-10-30T12:00:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.293+0000 7f8a76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.293+0000 7f8a76ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.463+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a800a6900 con 0x7f8a600392f0 2023-10-30T12:00:58.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.465+0000 7f8a76ffd640 1 -- 172.21.15.50:0/3385374031 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40580 (secure 0 0 0) 0x7f8a68031bb0 con 0x7f8a600392f0 2023-10-30T12:00:58.467 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:00:58.468 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:00:58.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 10 librados: watch_flush enter 2023-10-30T12:00:58.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 10 librados: watch_flush exit 2023-10-30T12:00:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 20 client.4374.objecter shutdown clearing up homeless session... 2023-10-30T12:00:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 10 client.4374.objecter successfully canceled tick 2023-10-30T12:00:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 msgr2=0x7f8a6003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a68036000 tx=0x7f8a6803d040 comp rx=0 tx=0).stop 2023-10-30T12:00:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 10 monclient: shutdown 2023-10-30T12:00:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6070 msgr2=0x7f8a801ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6070 0x7f8a801ca6f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800d830 tx=0x7f8a7800dcf0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 shutdown_connections 2023-10-30T12:00:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a80135ad0 0x7f8a8011e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a800a6450 0x7f8a801cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 --2- 172.21.15.50:0/3385374031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a800a6070 0x7f8a801ca6f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 >> 172.21.15.50:0/3385374031 conn(0x7f8a800a81d0 msgr2=0x7f8a800a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:58.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 shutdown_connections 2023-10-30T12:00:58.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 -- 172.21.15.50:0/3385374031 wait complete. 2023-10-30T12:00:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.467+0000 7f8a848f2640 1 librados: shutdown 2023-10-30T12:00:58.479 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-10-30T12:00:58.393912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164072,"kb_used_data":1928,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066968,"statfs":{"total":579820584960,"available":579652575232,"internally_reserved":0,"allocated":1974272,"data_stored":1343408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":165927591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475189+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:00:53.475189+0000","last_peered":"2023-10-30T12:00:53.475189+0000","last_clean":"2023-10-30T12:00:53.475189+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:00:53.475189+0000","last_undegraded":"2023-10-30T12:00:53.475189+0000","last_fullsized":"2023-10-30T12:00:53.475189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475012+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:00:53.475012+0000","last_peered":"2023-10-30T12:00:53.475012+0000","last_clean":"2023-10-30T12:00:53.475012+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:00:53.475012+0000","last_undegraded":"2023-10-30T12:00:53.475012+0000","last_fullsized":"2023-10-30T12:00:53.475012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475063+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:00:53.475063+0000","last_peered":"2023-10-30T12:00:53.475063+0000","last_clean":"2023-10-30T12:00:53.475063+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:00:53.475063+0000","last_undegraded":"2023-10-30T12:00:53.475063+0000","last_fullsized":"2023-10-30T12:00:53.475063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583189+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:00:53.583189+0000","last_peered":"2023-10-30T12:00:53.583189+0000","last_clean":"2023-10-30T12:00:53.583189+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:00:53.583189+0000","last_undegraded":"2023-10-30T12:00:53.583189+0000","last_fullsized":"2023-10-30T12:00:53.583189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583724+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:00:53.583724+0000","last_peered":"2023-10-30T12:00:53.583724+0000","last_clean":"2023-10-30T12:00:53.583724+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:00:53.583724+0000","last_undegraded":"2023-10-30T12:00:53.583724+0000","last_fullsized":"2023-10-30T12:00:53.583724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475112+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:00:53.475112+0000","last_peered":"2023-10-30T12:00:53.475112+0000","last_clean":"2023-10-30T12:00:53.475112+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:00:53.475112+0000","last_undegraded":"2023-10-30T12:00:53.475112+0000","last_fullsized":"2023-10-30T12:00:53.475112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.583196+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:00:53.583196+0000","last_peered":"2023-10-30T12:00:53.583196+0000","last_clean":"2023-10-30T12:00:53.583196+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:00:53.583196+0000","last_undegraded":"2023-10-30T12:00:53.583196+0000","last_fullsized":"2023-10-30T12:00:53.583196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":163,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-30T12:00:53.475070+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:00:53.475070+0000","last_peered":"2023-10-30T12:00:53.475070+0000","last_clean":"2023-10-30T12:00:53.475070+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:00:53.475070+0000","last_undegraded":"2023-10-30T12:00:53.475070+0000","last_fullsized":"2023-10-30T12:00:53.475070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":576520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":47592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":576520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:00:58.480 INFO:tasks.mon_thrash.ceph_manager:clean! 2023-10-30T12:00:58.481 DEBUG:tasks.mon_thrash:Yielding 2023-10-30T12:00:58.481 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T12:00:58.518 INFO:tasks.workunit:Pulling workunits from ref 397e85a5f8ee6b984077e94184404370497237a9 2023-10-30T12:00:58.519 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T12:00:58.519 DEBUG:teuthology.orchestra.run.smithi118:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:00:58.520 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2023-10-30T12:00:58.520 INFO:tasks.mon_thrash.mon_thrasher:seed: 1698667258, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2023-10-30T12:00:58.520 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:00:58.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:00:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:00:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-30T12:00:58.539 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:00:58.595 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-30T12:00:58.596 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-30T12:00:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.625+0000 7fc36d14e640 10 monclient: get_monmap_and_config 2023-10-30T12:00:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.625+0000 7fc36d14e640 10 monclient: build_initial_monmap 2023-10-30T12:00:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.625+0000 7fc36d14e640 10 monclient: monmap: 2023-10-30T12:00:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:58.626510+0000 2023-10-30T12:00:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:58.626510+0000 2023-10-30T12:00:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:58.629 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.625+0000 7fc36d14e640 1 Processor -- start 2023-10-30T12:00:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 1 -- start start 2023-10-30T12:00:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient: init 2023-10-30T12:00:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T12:00:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc3680a2dd0 0x7fc3680a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3680a2dd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc36806a8e0 0x7fc3680a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): picked mon.noname-a con 0x7fc36806a8e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc368071400 0x7fc3680a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): picked mon.noname-b con 0x7fc368071400 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3680a31b0 con 0x7fc36806a8e0 2023-10-30T12:00:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36806a6a0 con 0x7fc3680a2dd0 2023-10-30T12:00:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36806be90 con 0x7fc368071400 2023-10-30T12:00:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36d14e640 10 monclient(hunting): authenticate will time out at 903.809494s 2023-10-30T12:00:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc368071400 0x7fc3680a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc36806a8e0 0x7fc3680a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc368071400 0x7fc3680a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42994/0 (socket says 172.21.15.50:42994) 2023-10-30T12:00:58.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 -- 172.21.15.50:0/1189565272 learned_addr learned my addr 172.21.15.50:0/1189565272 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:58.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): get_auth_request con 0x7fc368071400 auth_method 0 2023-10-30T12:00:58.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc36806a8e0 0x7fc3680a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42234/0 (socket says 172.21.15.50:42234) 2023-10-30T12:00:58.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 10 monclient(hunting): get_auth_request con 0x7fc36806a8e0 auth_method 0 2023-10-30T12:00:58.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 1 --2- 172.21.15.50:0/1189565272 >> v2:172.21.15.118:3300/0 conn(0x7fc3680a2dd0 0x7fc3680a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 10 monclient(hunting): get_auth_request con 0x7fc3680a2dd0 auth_method 0 2023-10-30T12:00:58.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2023-10-30T12:00:58.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 -- 172.21.15.50:0/1189565272 >> v2:172.21.15.118:3300/0 conn(0x7fc3680a2dd0 msgr2=0x7fc3680a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:58.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 --2- 172.21.15.50:0/1189565272 >> v2:172.21.15.118:3300/0 conn(0x7fc3680a2dd0 0x7fc3680a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 -- 172.21.15.50:0/1189565272 >> v2:172.21.15.50:3300/0 conn(0x7fc36806a8e0 msgr2=0x7fc3680a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:58.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 --2- 172.21.15.50:0/1189565272 >> v2:172.21.15.50:3300/0 conn(0x7fc36806a8e0 0x7fc3680a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 10 monclient: _finish_hunting 0 2023-10-30T12:00:58.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 1 monclient: found mon.noname-b 2023-10-30T12:00:58.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.626+0000 7fc36759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:58.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:58.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36759e640 1 -- 172.21.15.50:0/1189565272 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36809e190 con 0x7fc368071400 2023-10-30T12:00:58.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36659c640 1 --2- 172.21.15.50:0/1189565272 >> v2:172.21.15.50:3300/0 conn(0x7fc36806a8e0 0x7fc3680a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:58.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36759e640 1 --2- 172.21.15.50:0/1189565272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc368071400 0x7fc3680a5140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc36806af70 tx=0x7fc36002bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d35c1f533cb738d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 1 -- 172.21.15.50:0/1189565272 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36002c960 con 0x7fc368071400 2023-10-30T12:00:58.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:58.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: dump: 2023-10-30T12:00:58.649 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.649 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.649 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.649 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.649 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.649 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.650 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.650 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.650 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.650 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: _finish_auth 0 2023-10-30T12:00:58.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:00:58.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:58.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 1 -- 172.21.15.50:0/1189565272 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36002cac0 con 0x7fc368071400 2023-10-30T12:00:58.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36d14e640 5 monclient: authenticate success, global_id 4295 2023-10-30T12:00:58.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36d14e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:58.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:58.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36d14e640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:58.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 1 -- 172.21.15.50:0/1189565272 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36002ce50 con 0x7fc368071400 2023-10-30T12:00:58.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc36d14e640 4 set_mon_vals no callback set 2023-10-30T12:00:58.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:58.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.627+0000 7fc365d9b640 10 monclient: dump: 2023-10-30T12:00:58.653 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.653 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.653 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.654 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.654 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.654 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.654 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.654 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.655 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.655 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 10 monclient: shutdown 2023-10-30T12:00:58.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:58.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 -- 172.21.15.50:0/1189565272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc368071400 msgr2=0x7fc3680a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 --2- 172.21.15.50:0/1189565272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc368071400 0x7fc3680a5140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc36806af70 tx=0x7fc36002bdc0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 -- 172.21.15.50:0/1189565272 shutdown_connections 2023-10-30T12:00:58.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 --2- 172.21.15.50:0/1189565272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc368071400 0x7fc3680a5140 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 --2- 172.21.15.50:0/1189565272 >> v2:172.21.15.50:3300/0 conn(0x7fc36806a8e0 0x7fc3680a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 --2- 172.21.15.50:0/1189565272 >> v2:172.21.15.118:3300/0 conn(0x7fc3680a2dd0 0x7fc3680a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 -- 172.21.15.50:0/1189565272 >> 172.21.15.50:0/1189565272 conn(0x7fc3680a22b0 msgr2=0x7fc3680a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 -- 172.21.15.50:0/1189565272 shutdown_connections 2023-10-30T12:00:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.628+0000 7fc36d14e640 1 -- 172.21.15.50:0/1189565272 wait complete. 2023-10-30T12:00:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient: build_initial_monmap 2023-10-30T12:00:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient: monmap: 2023-10-30T12:00:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:58.659 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 librados: starting msgr at 2023-10-30T12:00:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 librados: starting objecter 2023-10-30T12:00:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 Processor -- start 2023-10-30T12:00:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 -- start start 2023-10-30T12:00:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 librados: setting wanted keys 2023-10-30T12:00:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 librados: calling monclient init 2023-10-30T12:00:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient: init 2023-10-30T12:00:58.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:58.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:00:58.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc368071400 0x7fc3681ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient(hunting): picked mon.noname-a con 0x7fc368071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:58.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3680a2dd0 0x7fc36813b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3680a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:58.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc36813b780 0x7fc368122b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.629+0000 7fc36d14e640 10 monclient(hunting): picked mon.noname-b con 0x7fc36813b780 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:58.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc368126340 con 0x7fc368071400 2023-10-30T12:00:58.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681ca8b0 con 0x7fc36813b780 2023-10-30T12:00:58.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:58.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681251d0 con 0x7fc3680a2dd0 2023-10-30T12:00:58.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:58.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient(hunting): authenticate will time out at 903.813101s 2023-10-30T12:00:58.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3680a2dd0 0x7fc36813b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc368071400 0x7fc3681ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3680a2dd0 0x7fc36813b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43010/0 (socket says 172.21.15.50:43010) 2023-10-30T12:00:58.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 1 -- 172.21.15.50:0/2861189291 learned_addr learned my addr 172.21.15.50:0/2861189291 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:58.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 10 monclient(hunting): get_auth_request con 0x7fc3680a2dd0 auth_method 0 2023-10-30T12:00:58.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): get_auth_request con 0x7fc368071400 auth_method 0 2023-10-30T12:00:58.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc36813b780 0x7fc368122b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 10 monclient(hunting): get_auth_request con 0x7fc36813b780 auth_method 0 2023-10-30T12:00:58.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:58.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:58.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:58.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:58.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:58.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:58.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2023-10-30T12:00:58.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 -- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3680a2dd0 msgr2=0x7fc36813b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3680a2dd0 0x7fc36813b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 -- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc36813b780 msgr2=0x7fc368122b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc36813b780 0x7fc368122b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:58.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 monclient: found mon.noname-a 2023-10-30T12:00:58.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:58.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 -- 172.21.15.50:0/2861189291 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358009400 con 0x7fc368071400 2023-10-30T12:00:58.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36659c640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3680a2dd0 0x7fc36813b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:58.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc366d9d640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc368071400 0x7fc3681ca370 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00e850 tx=0x7fc35c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 client.?.objecter ms_handle_connect 0x7fc368071400 2023-10-30T12:00:58.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:58.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 1 -- 172.21.15.50:0/2861189291 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0095b0 con 0x7fc368071400 2023-10-30T12:00:58.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:58.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: dump: 2023-10-30T12:00:58.671 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.671 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.672 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.672 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.672 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.672 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.673 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.673 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.673 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:58.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:58.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:58.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 5 monclient: authenticate success, global_id 4380 2023-10-30T12:00:58.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 1 -- 172.21.15.50:0/2861189291 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c00c940 con 0x7fc368071400 2023-10-30T12:00:58.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient: _renew_subs 2023-10-30T12:00:58.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc360009b30 con 0x7fc368071400 2023-10-30T12:00:58.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:58.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 1 -- 172.21.15.50:0/2861189291 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c00ccd0 con 0x7fc368071400 2023-10-30T12:00:58.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:58.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:58.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc34f7fe640 10 monclient: dump: 2023-10-30T12:00:58.676 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:58.676 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:58.676 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.676 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:58.676 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:58.676 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:58.677 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:58.677 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:58.677 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:58.677 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:58.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:58.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient: _renew_subs 2023-10-30T12:00:58.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc368070370 con 0x7fc368071400 2023-10-30T12:00:58.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36559a640 4 set_mon_vals no callback set 2023-10-30T12:00:58.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.630+0000 7fc36d14e640 1 librados: init done 2023-10-30T12:00:58.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.631+0000 7fc34f7fe640 1 -- 172.21.15.50:0/2861189291 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc35c013470 con 0x7fc368071400 2023-10-30T12:00:58.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.631+0000 7fc36d14e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.631+0000 7fc36d14e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.631+0000 7fc36d14e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.631+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36813bfa0 con 0x7fc368071400 2023-10-30T12:00:58.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.631+0000 7fc34f7fe640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3540392f0 0x7fc35403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:58.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc34f7fe640 1 -- 172.21.15.50:0/2861189291 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fc35c04a5e0 con 0x7fc368071400 2023-10-30T12:00:58.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc34f7fe640 10 client.4380.objecter ms_dispatch 0x7fc368000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:58.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc34f7fe640 3 client.4380.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:58.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc34f7fe640 3 client.4380.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:58.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc34f7fe640 20 client.4380.objecter dump_active .. 0 homeless 2023-10-30T12:00:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc36659c640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3540392f0 0x7fc35403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc36659c640 10 monclient: get_auth_request con 0x7fc3540392f0 auth_method 0 2023-10-30T12:00:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc36659c640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3540392f0 0x7fc35403b7b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc368124410 tx=0x7fc358009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.632+0000 7fc34f7fe640 10 client.4380.objecter ms_handle_connect 0x7fc3540392f0 2023-10-30T12:00:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.636+0000 7fc34f7fe640 1 -- 172.21.15.50:0/2861189291 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc35c013720 con 0x7fc368071400 2023-10-30T12:00:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.636+0000 7fc34f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.636+0000 7fc34f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:58.683 INFO:tasks.workunit:timeout=3h 2023-10-30T12:00:58.684 INFO:tasks.workunit:cleanup=True 2023-10-30T12:00:58.684 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/VallariAg/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 397e85a5f8ee6b984077e94184404370497237a9 2023-10-30T12:00:58.741 INFO:tasks.workunit.client.0.smithi118.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T12:00:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.883+0000 7fc36d14e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:00:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.883+0000 7fc36d14e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:00:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.883+0000 7fc36d14e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.883+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc36804e160 con 0x7fc368071400 2023-10-30T12:00:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.883+0000 7fc34f7fe640 1 -- 172.21.15.50:0/2861189291 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fc35c01b3b0 con 0x7fc368071400 2023-10-30T12:00:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.883+0000 7fc34f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:00:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.883+0000 7fc34f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:58.887 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:00:58.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 10 librados: watch_flush enter 2023-10-30T12:00:58.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 10 librados: watch_flush exit 2023-10-30T12:00:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 20 client.4380.objecter shutdown clearing up homeless session... 2023-10-30T12:00:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 10 client.4380.objecter successfully canceled tick 2023-10-30T12:00:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3540392f0 msgr2=0x7fc35403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3540392f0 0x7fc35403b7b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc368124410 tx=0x7fc358009190 comp rx=0 tx=0).stop 2023-10-30T12:00:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 10 monclient: shutdown 2023-10-30T12:00:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc368071400 msgr2=0x7fc3681ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc368071400 0x7fc3681ca370 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00e850 tx=0x7fc35c00ed10 comp rx=0 tx=0).stop 2023-10-30T12:00:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 shutdown_connections 2023-10-30T12:00:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3540392f0 0x7fc35403b7b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc36813b780 0x7fc368122b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3680a2dd0 0x7fc36813b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 --2- 172.21.15.50:0/2861189291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc368071400 0x7fc3681ca370 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 >> 172.21.15.50:0/2861189291 conn(0x7fc3680a22b0 msgr2=0x7fc36806e530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 shutdown_connections 2023-10-30T12:00:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 -- 172.21.15.50:0/2861189291 wait complete. 2023-10-30T12:00:58.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:58.885+0000 7fc36d14e640 1 librados: shutdown 2023-10-30T12:00:58.898 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:00:58.898 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:00:58.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:00:58.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:00:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.002+0000 7fde62b5e640 10 monclient: get_monmap_and_config 2023-10-30T12:00:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.002+0000 7fde62b5e640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.002+0000 7fde62b5e640 10 monclient: monmap: 2023-10-30T12:00:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:59.003833+0000 2023-10-30T12:00:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:59.003833+0000 2023-10-30T12:00:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:59.007 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:59.007 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:59.007 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 1 Processor -- start 2023-10-30T12:00:59.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 1 -- start start 2023-10-30T12:00:59.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 10 monclient: init 2023-10-30T12:00:59.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T12:00:59.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fde5c096540 0x7fde5c096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 10 monclient(hunting): picked mon.noname-c con 0x7fde5c096540 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde5c096e60 0x7fde5c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.003+0000 7fde62b5e640 10 monclient(hunting): picked mon.noname-a con 0x7fde5c096e60 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde5c09afa0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5c09bb40 con 0x7fde5c09afa0 2023-10-30T12:00:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5c09bd30 con 0x7fde5c096e60 2023-10-30T12:00:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5c09c400 con 0x7fde5c096540 2023-10-30T12:00:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde62b5e640 10 monclient(hunting): authenticate will time out at 904.187044s 2023-10-30T12:00:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde5c096e60 0x7fde5c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde610fb640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fde5c09afa0 0x7fde5c09b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35264/0 (socket says 172.21.15.50:35264) 2023-10-30T12:00:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde610fb640 1 -- 172.21.15.50:0/337131260 learned_addr learned my addr 172.21.15.50:0/337131260 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): get_auth_request con 0x7fde5c096e60 auth_method 0 2023-10-30T12:00:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde608fa640 1 --2- 172.21.15.50:0/337131260 >> v2:172.21.15.118:3300/0 conn(0x7fde5c096540 0x7fde5c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde608fa640 10 monclient(hunting): get_auth_request con 0x7fde5c096540 auth_method 0 2023-10-30T12:00:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde608fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde608fa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde608fa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2023-10-30T12:00:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 1 -- 172.21.15.50:0/337131260 >> v2:172.21.15.118:3300/0 conn(0x7fde5c096540 msgr2=0x7fde5c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 1 --2- 172.21.15.50:0/337131260 >> v2:172.21.15.118:3300/0 conn(0x7fde5c096540 0x7fde5c096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 1 -- 172.21.15.50:0/337131260 >> v1:172.21.15.50:6789/0 conn(0x7fde5c09afa0 legacy=0x7fde5c09b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 1 monclient: found mon.noname-a 2023-10-30T12:00:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.004+0000 7fde53fff640 1 -- 172.21.15.50:0/337131260 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c09c120 con 0x7fde5c096e60 2023-10-30T12:00:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde53fff640 1 --2- 172.21.15.50:0/337131260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c09aa60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fde5c06afd0 tx=0x7fde4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6735579a22dadf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 1 -- 172.21.15.50:0/337131260 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c02e070 con 0x7fde5c096e60 2023-10-30T12:00:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:59.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: dump: 2023-10-30T12:00:59.024 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.024 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.024 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.026 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 1 -- 172.21.15.50:0/337131260 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c0315a0 con 0x7fde5c096e60 2023-10-30T12:00:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 1 -- 172.21.15.50:0/337131260 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c031930 con 0x7fde5c096e60 2023-10-30T12:00:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 5 monclient: authenticate success, global_id 4383 2023-10-30T12:00:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 4 set_mon_vals no callback set 2023-10-30T12:00:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde537fe640 10 monclient: dump: 2023-10-30T12:00:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.029 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 10 monclient: shutdown 2023-10-30T12:00:59.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 -- 172.21.15.50:0/337131260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 msgr2=0x7fde5c09aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 --2- 172.21.15.50:0/337131260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c09aa60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fde5c06afd0 tx=0x7fde4c02b840 comp rx=0 tx=0).stop 2023-10-30T12:00:59.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 -- 172.21.15.50:0/337131260 shutdown_connections 2023-10-30T12:00:59.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 --2- 172.21.15.50:0/337131260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c09aa60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 --2- 172.21.15.50:0/337131260 >> v2:172.21.15.118:3300/0 conn(0x7fde5c096540 0x7fde5c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 -- 172.21.15.50:0/337131260 >> 172.21.15.50:0/337131260 conn(0x7fde5c0a3980 msgr2=0x7fde5c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 -- 172.21.15.50:0/337131260 shutdown_connections 2023-10-30T12:00:59.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.005+0000 7fde62b5e640 1 -- 172.21.15.50:0/337131260 wait complete. 2023-10-30T12:00:59.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 10 monclient: monmap: 2023-10-30T12:00:59.032 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:59.033 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:59.033 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:59.033 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.033 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.034 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:59.034 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:59.034 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:59.034 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 1 librados: starting msgr at 2023-10-30T12:00:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 1 librados: starting objecter 2023-10-30T12:00:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 1 Processor -- start 2023-10-30T12:00:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 1 -- start start 2023-10-30T12:00:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 1 librados: setting wanted keys 2023-10-30T12:00:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 1 librados: calling monclient init 2023-10-30T12:00:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.006+0000 7fde62b5e640 10 monclient: init 2023-10-30T12:00:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde5c096540 0x7fde5c070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): picked mon.noname-b con 0x7fde5c096540 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): picked mon.noname-a con 0x7fde5c096e60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde5c09afa0 0x7fde5c070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): picked mon.noname-c con 0x7fde5c09afa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c0930e0 con 0x7fde5c096e60 2023-10-30T12:00:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c09a080 con 0x7fde5c096540 2023-10-30T12:00:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c098a00 con 0x7fde5c09afa0 2023-10-30T12:00:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde62b5e640 10 monclient(hunting): authenticate will time out at 904.190432s 2023-10-30T12:00:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde5c09afa0 0x7fde5c070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c070680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42256/0 (socket says 172.21.15.50:42256) 2023-10-30T12:00:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 -- 172.21.15.50:0/1767608556 learned_addr learned my addr 172.21.15.50:0/1767608556 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): get_auth_request con 0x7fde5c096e60 auth_method 0 2023-10-30T12:00:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 10 monclient(hunting): get_auth_request con 0x7fde5c09afa0 auth_method 0 2023-10-30T12:00:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde5c096540 0x7fde5c070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 10 monclient(hunting): get_auth_request con 0x7fde5c096540 auth_method 0 2023-10-30T12:00:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2023-10-30T12:00:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde5c09afa0 msgr2=0x7fde5c070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde5c09afa0 0x7fde5c070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde5c096540 msgr2=0x7fde5c070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde5c096540 0x7fde5c070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 monclient: found mon.noname-a 2023-10-30T12:00:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde53fff640 1 -- 172.21.15.50:0/1767608556 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c009400 con 0x7fde5c096e60 2023-10-30T12:00:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.007+0000 7fde610fb640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde5c09afa0 0x7fde5c070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde53fff640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c070680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde4c009890 tx=0x7fde4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 client.?.objecter ms_handle_connect 0x7fde5c096e60 2023-10-30T12:00:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c032840 con 0x7fde5c096e60 2023-10-30T12:00:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: dump: 2023-10-30T12:00:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.049 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c0329a0 con 0x7fde5c096e60 2023-10-30T12:00:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 5 monclient: authenticate success, global_id 4389 2023-10-30T12:00:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 10 monclient: _renew_subs 2023-10-30T12:00:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c093e60 con 0x7fde5c096e60 2023-10-30T12:00:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c032d50 con 0x7fde5c096e60 2023-10-30T12:00:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde517fa640 10 monclient: dump: 2023-10-30T12:00:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.053 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 10 monclient: _renew_subs 2023-10-30T12:00:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c098260 con 0x7fde5c096e60 2023-10-30T12:00:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde52ffd640 4 set_mon_vals no callback set 2023-10-30T12:00:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.008+0000 7fde62b5e640 1 librados: init done 2023-10-30T12:00:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fde4c03b3f0 con 0x7fde5c096e60 2023-10-30T12:00:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde62b5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde62b5e640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c097b70 0x7fde5c1ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde62b5e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c098f80 con 0x7fde5c097b70 2023-10-30T12:00:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c097b70 0x7fde5c1ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde480392f0 0x7fde4803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient: get_auth_request con 0x7fde5c097b70 auth_method 0 2023-10-30T12:00:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fde4c06c400 con 0x7fde5c096e60 2023-10-30T12:00:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 10 client.4389.objecter ms_dispatch 0x7fde5c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 3 client.4389.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 3 client.4389.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 20 client.4389.objecter dump_active .. 0 homeless 2023-10-30T12:00:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2023-10-30T12:00:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 1 -- 172.21.15.50:0/1767608556 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde5c09d3a0 con 0x7fde5c097b70 2023-10-30T12:00:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde610fb640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde480392f0 0x7fde4803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde610fb640 10 monclient: get_auth_request con 0x7fde480392f0 auth_method 0 2023-10-30T12:00:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 10 client.4389.objecter ms_handle_connect 0x7fde5c097b70 2023-10-30T12:00:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 10 client.4389.objecter resend_mon_ops 2023-10-30T12:00:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde608fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c097b70 0x7fde5c1ca720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde580107f0 tx=0x7fde58010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58005ba0 con 0x7fde5c097b70 2023-10-30T12:00:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.009+0000 7fde517fa640 10 monclient: dump: 2023-10-30T12:00:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.061 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde610fb640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde480392f0 0x7fde4803b7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0975a0 tx=0x7fde54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 10 client.4389.objecter ms_handle_connect 0x7fde480392f0 2023-10-30T12:00:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fde58005d00 con 0x7fde5c097b70 2023-10-30T12:00:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c097b70 msgr2=0x7fde5c1ca720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c097b70 0x7fde5c1ca720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde580107f0 tx=0x7fde58010cb0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c097b70 msgr2=0x7fde5c1ca720 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:00:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.010+0000 7fde517fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c097b70 0x7fde5c1ca720 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde580107f0 tx=0x7fde58010cb0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde62b5e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:00:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde62b5e640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c1cac60 0x7fde5c1cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde62b5e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c073ff0 con 0x7fde5c1cac60 2023-10-30T12:00:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c1cac60 0x7fde5c1cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient: get_auth_request con 0x7fde5c1cac60 auth_method 0 2023-10-30T12:00:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2023-10-30T12:00:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.016+0000 7fde608fa640 1 -- 172.21.15.50:0/1767608556 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fde5c1cb5b0 con 0x7fde5c1cac60 2023-10-30T12:00:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 10 client.4389.objecter ms_handle_connect 0x7fde5c1cac60 2023-10-30T12:00:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 10 client.4389.objecter resend_mon_ops 2023-10-30T12:00:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde608fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c1cac60 0x7fde5c1cb070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fde580107f0 tx=0x7fde580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58025070 con 0x7fde5c1cac60 2023-10-30T12:00:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 10 monclient: dump: 2023-10-30T12:00:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.071 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fde58005e70 con 0x7fde5c1cac60 2023-10-30T12:00:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:00:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c1cac60 msgr2=0x7fde5c1cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c1cac60 0x7fde5c1cb070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fde580107f0 tx=0x7fde580046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c1cac60 msgr2=0x7fde5c1cb070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:00:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde517fa640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c1cac60 0x7fde5c1cb070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.073 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:00:59.073 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:00:59.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:00:59.073 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:00:59.073 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 4, 2023-10-30T12:00:59.074 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:00:59.074 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:00:59.074 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:00:59.074 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:00:59.074 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.075 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 18, 2023-10-30T12:00:59.075 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:00:59.075 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:00:59.075 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:00:59.075 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.076 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.076 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.076 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.076 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.076 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.077 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.077 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.077 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.077 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.077 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.077 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:00:59.078 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:00:59.078 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.078 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.078 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.078 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.079 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.079 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.079 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.079 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.079 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.079 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.080 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.080 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:00:59.080 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:00:59.080 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:00:59.080 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:00:59.081 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:00:59.081 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:00:59.081 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:00:59.081 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:00:59.081 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:00:59.082 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:00:59.082 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:00:59.082 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:00:59.082 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:00:59.082 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:00:59.082 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:00:59.083 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:00:59.083 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:00:59.083 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.083 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.083 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.084 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.084 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.084 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.084 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.084 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.084 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.085 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.085 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.085 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:00:59.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.085 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:00:59.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:00:59.086 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:00:59.086 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.086 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.087 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.087 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:00:59.087 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.088 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.088 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:00:59.088 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.088 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.089 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:00:59.089 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:00:59.089 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.089 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.089 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:00:59.090 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:00:59.090 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.090 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.091 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.091 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:00:59.091 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.092 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.092 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:00:59.092 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.093 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.093 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:00:59.093 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:00:59.093 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.093 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.094 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:00:59.094 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:00:59.095 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.095 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.095 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.095 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:00:59.095 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.096 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.096 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:00:59.096 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.097 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.097 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:00:59.097 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:00:59.097 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.098 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.098 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.098 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:00:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:00:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:00:59.100 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.100 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.100 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:00:59.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.100 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.100 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.101 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:00:59.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.101 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.101 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:00:59.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.102 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.102 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.102 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:00:59.102 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.102 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.103 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:00:59.103 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:00:59.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 10 librados: watch_flush enter 2023-10-30T12:00:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 10 librados: watch_flush exit 2023-10-30T12:00:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 20 client.4389.objecter shutdown clearing up homeless session... 2023-10-30T12:00:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 10 client.4389.objecter successfully canceled tick 2023-10-30T12:00:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde480392f0 msgr2=0x7fde4803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde480392f0 0x7fde4803b7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0975a0 tx=0x7fde54009040 comp rx=0 tx=0).stop 2023-10-30T12:00:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 10 monclient: shutdown 2023-10-30T12:00:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 msgr2=0x7fde5c070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c070680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde4c009890 tx=0x7fde4c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde610fb640 1 -- 172.21.15.50:0/1767608556 reap_dead start 2023-10-30T12:00:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 shutdown_connections 2023-10-30T12:00:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 --2- 172.21.15.50:0/1767608556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5c096e60 0x7fde5c070680 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 >> 172.21.15.50:0/1767608556 conn(0x7fde5c0a3980 msgr2=0x7fde5c1b9910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 shutdown_connections 2023-10-30T12:00:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 -- 172.21.15.50:0/1767608556 wait complete. 2023-10-30T12:00:59.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.017+0000 7fde62b5e640 1 librados: shutdown 2023-10-30T12:00:59.108 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:00:59.108 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:00:59.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:00:59.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.214+0000 7f990c275640 10 monclient: get_monmap_and_config 2023-10-30T12:00:59.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.214+0000 7f990c275640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.214+0000 7f990c275640 10 monclient: monmap: 2023-10-30T12:00:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:59.215221+0000 2023-10-30T12:00:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:59.215221+0000 2023-10-30T12:00:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.217 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.218 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:59.218 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:59.218 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:59.218 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:59.219 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:59.219 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.214+0000 7f990c275640 1 Processor -- start 2023-10-30T12:00:59.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.214+0000 7f990c275640 1 -- start start 2023-10-30T12:00:59.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.214+0000 7f990c275640 10 monclient: init 2023-10-30T12:00:59.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-10-30T12:00:59.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99040a8010 0x7f9904132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient(hunting): picked mon.noname-b con 0x7f99040a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9904099a40 0x7f9904132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient(hunting): picked mon.noname-a con 0x7f9904099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:00:59.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99041356d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:59.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99040a7190 con 0x7f99041356d0 2023-10-30T12:00:59.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9904133110 con 0x7f9904099a40 2023-10-30T12:00:59.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040a83f0 con 0x7f99040a8010 2023-10-30T12:00:59.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990c275640 10 monclient(hunting): authenticate will time out at 904.398429s 2023-10-30T12:00:59.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99040a8010 0x7f9904132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9904099a40 0x7f9904132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9904099a40 0x7f9904132bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42296/0 (socket says 172.21.15.50:42296) 2023-10-30T12:00:59.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99040a8010 0x7f9904132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43014/0 (socket says 172.21.15.50:43014) 2023-10-30T12:00:59.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 1 -- 172.21.15.50:0/1807668941 learned_addr learned my addr 172.21.15.50:0/1807668941 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:59.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 10 monclient(hunting): get_auth_request con 0x7f9904099a40 auth_method 0 2023-10-30T12:00:59.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): get_auth_request con 0x7f99040a8010 auth_method 0 2023-10-30T12:00:59.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2023-10-30T12:00:59.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 1 -- 172.21.15.50:0/1807668941 >> v2:172.21.15.50:3300/0 conn(0x7f9904099a40 msgr2=0x7f9904132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:59.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 1 --2- 172.21.15.50:0/1807668941 >> v2:172.21.15.50:3300/0 conn(0x7f9904099a40 0x7f9904132bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 1 -- 172.21.15.50:0/1807668941 >> v1:172.21.15.50:6789/0 conn(0x7f99041356d0 legacy=0x7f9904135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 1 monclient: found mon.noname-b 2023-10-30T12:00:59.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:59.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f990a011640 1 -- 172.21.15.50:0/1807668941 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040a7600 con 0x7f99040a8010 2023-10-30T12:00:59.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.215+0000 7f9909810640 1 --2- 172.21.15.50:0/1807668941 >> v2:172.21.15.50:3300/0 conn(0x7f9904099a40 0x7f9904132bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:59.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990a011640 1 --2- 172.21.15.50:0/1807668941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99040a8010 0x7f9904132690 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99000097a0 tx=0x7f990002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc19e98f6205643d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 1 -- 172.21.15.50:0/1807668941 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99000316f0 con 0x7f99040a8010 2023-10-30T12:00:59.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:59.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: dump: 2023-10-30T12:00:59.236 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.236 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.237 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.237 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.237 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.237 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.237 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.238 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.238 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.238 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 1 -- 172.21.15.50:0/1807668941 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9900031850 con 0x7f99040a8010 2023-10-30T12:00:59.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990c275640 5 monclient: authenticate success, global_id 4304 2023-10-30T12:00:59.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990c275640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:59.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990c275640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:59.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 1 -- 172.21.15.50:0/1807668941 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900031be0 con 0x7f99040a8010 2023-10-30T12:00:59.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990c275640 4 set_mon_vals no callback set 2023-10-30T12:00:59.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:59.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.216+0000 7f990900f640 10 monclient: dump: 2023-10-30T12:00:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.242 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 10 monclient: shutdown 2023-10-30T12:00:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 -- 172.21.15.50:0/1807668941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99040a8010 msgr2=0x7f9904132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 --2- 172.21.15.50:0/1807668941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99040a8010 0x7f9904132690 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99000097a0 tx=0x7f990002b790 comp rx=0 tx=0).stop 2023-10-30T12:00:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 -- 172.21.15.50:0/1807668941 shutdown_connections 2023-10-30T12:00:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 --2- 172.21.15.50:0/1807668941 >> v2:172.21.15.50:3300/0 conn(0x7f9904099a40 0x7f9904132bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 --2- 172.21.15.50:0/1807668941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99040a8010 0x7f9904132690 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 -- 172.21.15.50:0/1807668941 >> 172.21.15.50:0/1807668941 conn(0x7f99040a4a10 msgr2=0x7f99040a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 -- 172.21.15.50:0/1807668941 shutdown_connections 2023-10-30T12:00:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.217+0000 7f990c275640 1 -- 172.21.15.50:0/1807668941 wait complete. 2023-10-30T12:00:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient: monmap: 2023-10-30T12:00:59.245 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.245 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:59.245 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:59.245 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:59.245 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.246 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.246 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:59.246 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:59.246 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:59.246 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 librados: starting msgr at 2023-10-30T12:00:59.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 librados: starting objecter 2023-10-30T12:00:59.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 Processor -- start 2023-10-30T12:00:59.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 -- start start 2023-10-30T12:00:59.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 librados: setting wanted keys 2023-10-30T12:00:59.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 librados: calling monclient init 2023-10-30T12:00:59.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient: init 2023-10-30T12:00:59.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:00:59.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99040a8010 0x7f99041ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): picked mon.noname-b con 0x7f99040a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:59.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99041356d0 0x7f99041cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): picked mon.noname-c con 0x7f99041356d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f990411e5c0 0x7f990411ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): picked mon.noname-a con 0x7f990411e5c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:59.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9904121d90 con 0x7f990411e5c0 2023-10-30T12:00:59.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9904120f10 con 0x7f99040a8010 2023-10-30T12:00:59.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040a18d0 con 0x7f99041356d0 2023-10-30T12:00:59.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.218+0000 7f990c275640 10 monclient(hunting): authenticate will time out at 904.401994s 2023-10-30T12:00:59.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99041356d0 0x7f99041cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f990411e5c0 0x7f990411ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99041356d0 0x7f99041cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43016/0 (socket says 172.21.15.50:43016) 2023-10-30T12:00:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f990411e5c0 0x7f990411ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42306/0 (socket says 172.21.15.50:42306) 2023-10-30T12:00:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 1 -- 172.21.15.50:0/2833483433 learned_addr learned my addr 172.21.15.50:0/2833483433 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 10 monclient(hunting): get_auth_request con 0x7f99041356d0 auth_method 0 2023-10-30T12:00:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): get_auth_request con 0x7f990411e5c0 auth_method 0 2023-10-30T12:00:59.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a011640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99040a8010 0x7f99041ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a011640 10 monclient(hunting): get_auth_request con 0x7f99040a8010 auth_method 0 2023-10-30T12:00:59.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a011640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a011640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient(hunting): handle_auth_done global_id 4404 payload 824 2023-10-30T12:00:59.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99041356d0 msgr2=0x7f99041cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99041356d0 0x7f99041cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99040a8010 msgr2=0x7f99041ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99040a8010 0x7f99041ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 monclient: found mon.noname-a 2023-10-30T12:00:59.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 -- 172.21.15.50:0/2833483433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f40094d0 con 0x7f990411e5c0 2023-10-30T12:00:59.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f9909810640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f99041356d0 0x7f99041cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:00:59.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a011640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99040a8010 0x7f99041ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:00:59.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990a812640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f990411e5c0 0x7f990411ea50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00c690 tx=0x7f98fc00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 client.?.objecter ms_handle_connect 0x7f990411e5c0 2023-10-30T12:00:59.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:59.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc003b70 con 0x7f990411e5c0 2023-10-30T12:00:59.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:59.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: dump: 2023-10-30T12:00:59.259 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.261 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.261 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.261 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc003cd0 con 0x7f990411e5c0 2023-10-30T12:00:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 5 monclient: authenticate success, global_id 4404 2023-10-30T12:00:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 10 monclient: _renew_subs 2023-10-30T12:00:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9900009400 con 0x7f990411e5c0 2023-10-30T12:00:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc005310 con 0x7f990411e5c0 2023-10-30T12:00:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f98faffd640 10 monclient: dump: 2023-10-30T12:00:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.265 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990880e640 4 set_mon_vals no callback set 2023-10-30T12:00:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 10 monclient: _renew_subs 2023-10-30T12:00:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9904120260 con 0x7f990411e5c0 2023-10-30T12:00:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.219+0000 7f990c275640 1 librados: init done 2023-10-30T12:00:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990c275640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990c275640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9904099a40 0x7f990411ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9904073fd0 con 0x7f9904099a40 2023-10-30T12:00:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f98fc005470 con 0x7f990411e5c0 2023-10-30T12:00:59.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9904099a40 0x7f990411ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient: get_auth_request con 0x7f9904099a40 auth_method 0 2023-10-30T12:00:59.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f98faffd640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98e8039320 0x7f98e803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2023-10-30T12:00:59.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f98fc04a1c0 con 0x7f990411e5c0 2023-10-30T12:00:59.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f98faffd640 10 client.4404.objecter ms_dispatch 0x7f9904000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:59.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f98faffd640 3 client.4404.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:59.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f98faffd640 3 client.4404.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:59.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.220+0000 7f990a011640 1 -- 172.21.15.50:0/2833483433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f990409a720 con 0x7f9904099a40 2023-10-30T12:00:59.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 20 client.4404.objecter dump_active .. 0 homeless 2023-10-30T12:00:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f9909810640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98e8039320 0x7f98e803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f9909810640 10 monclient: get_auth_request con 0x7f98e8039320 auth_method 0 2023-10-30T12:00:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 client.4404.objecter ms_handle_connect 0x7f9904099a40 2023-10-30T12:00:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 client.4404.objecter resend_mon_ops 2023-10-30T12:00:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f990a011640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9904099a40 0x7f990411ef90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9904099ed0 tx=0x7f99000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900034680 con 0x7f9904099a40 2023-10-30T12:00:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 monclient: dump: 2023-10-30T12:00:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.274 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.274 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.274 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f99000347e0 con 0x7f9904099a40 2023-10-30T12:00:59.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9904099a40 msgr2=0x7f990411ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9904099a40 0x7f990411ef90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9904099ed0 tx=0x7f99000046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9904099a40 msgr2=0x7f990411ef90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:00:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9904099a40 0x7f990411ef90 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9904099ed0 tx=0x7f99000046c0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f98faffd640 10 client.4404.objecter ms_handle_connect 0x7f98e8039320 2023-10-30T12:00:59.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.221+0000 7f9909810640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98e8039320 0x7f98e803b7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9904132130 tx=0x7f98f4009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990c275640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:00:59.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990c275640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f990411f4d0 0x7f990409df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9904134da0 con 0x7f990411f4d0 2023-10-30T12:00:59.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990a011640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f990411f4d0 0x7f990409df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990a011640 10 monclient: get_auth_request con 0x7f990411f4d0 auth_method 0 2023-10-30T12:00:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990a011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990a011640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.227+0000 7f990a011640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990a011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990a011640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2023-10-30T12:00:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990a011640 1 -- 172.21.15.50:0/2833483433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f990409a4b0 con 0x7f990411f4d0 2023-10-30T12:00:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 10 client.4404.objecter ms_handle_connect 0x7f990411f4d0 2023-10-30T12:00:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 10 client.4404.objecter resend_mon_ops 2023-10-30T12:00:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990a011640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f990411f4d0 0x7f990409df40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99000097a0 tx=0x7f9900031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990004d070 con 0x7f990411f4d0 2023-10-30T12:00:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 10 monclient: dump: 2023-10-30T12:00:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.283 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f9900003610 con 0x7f990411f4d0 2023-10-30T12:00:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:00:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f990411f4d0 msgr2=0x7f990409df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f990411f4d0 0x7f990409df40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99000097a0 tx=0x7f9900031000 comp rx=0 tx=0).stop 2023-10-30T12:00:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f990411f4d0 msgr2=0x7f990409df40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:00:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f98faffd640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f990411f4d0 0x7f990409df40 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99000097a0 tx=0x7f9900031000 comp rx=0 tx=0).stop 2023-10-30T12:00:59.285 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:00:59.285 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:00:59.285 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:00:59.286 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:00:59.286 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 4, 2023-10-30T12:00:59.286 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:00:59.286 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:00:59.286 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:00:59.287 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:00:59.287 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.287 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 18, 2023-10-30T12:00:59.287 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:00:59.287 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:00:59.287 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:00:59.288 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.288 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.288 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.288 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.288 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.289 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.289 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.289 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.290 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.290 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.290 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:00:59.290 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:00:59.290 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.291 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.291 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.291 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.291 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.291 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.292 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.292 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.292 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.292 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.292 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.293 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:00:59.293 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:00:59.293 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:00:59.293 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:00:59.294 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:00:59.294 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:00:59.294 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:00:59.294 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:00:59.294 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:00:59.295 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:00:59.295 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:00:59.295 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:00:59.295 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:00:59.295 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:00:59.296 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:00:59.296 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:00:59.296 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:00:59.296 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.296 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.296 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.297 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.297 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.297 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.297 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.297 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.298 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.298 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.298 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.298 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:00:59.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.299 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:00:59.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.299 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:00:59.299 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:00:59.299 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:00:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:00:59.301 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.302 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:00:59.302 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:00:59.302 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.302 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.303 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.303 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:00:59.303 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:00:59.304 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.304 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:00:59.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.305 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:00:59.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.306 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:00:59.307 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:00:59.307 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.307 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.307 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.308 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:00:59.308 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:00:59.308 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.309 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:00:59.309 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.310 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:00:59.310 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.310 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.310 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.311 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:00:59.311 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:00:59.311 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.311 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.312 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.312 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.312 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:00:59.312 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:00:59.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.313 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.313 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.313 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:00:59.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.314 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.314 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:00:59.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.314 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.314 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.314 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:00:59.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.315 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.315 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:00:59.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.315 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.316 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.316 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:00:59.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.316 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.316 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:00:59.317 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:00:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 10 librados: watch_flush enter 2023-10-30T12:00:59.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 10 librados: watch_flush exit 2023-10-30T12:00:59.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 20 client.4404.objecter shutdown clearing up homeless session... 2023-10-30T12:00:59.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 10 client.4404.objecter successfully canceled tick 2023-10-30T12:00:59.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98e8039320 msgr2=0x7f98e803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98e8039320 0x7f98e803b7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9904132130 tx=0x7f98f4009240 comp rx=0 tx=0).stop 2023-10-30T12:00:59.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 10 monclient: shutdown 2023-10-30T12:00:59.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f990411e5c0 msgr2=0x7f990411ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990c275640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f990411e5c0 0x7f990411ea50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00c690 tx=0x7f98fc00cb50 comp rx=0 tx=0).stop 2023-10-30T12:00:59.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.228+0000 7f990a812640 1 -- 172.21.15.50:0/2833483433 reap_dead start 2023-10-30T12:00:59.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.229+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 shutdown_connections 2023-10-30T12:00:59.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.229+0000 7f990c275640 1 --2- 172.21.15.50:0/2833483433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f990411e5c0 0x7f990411ea50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.229+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 >> 172.21.15.50:0/2833483433 conn(0x7f99040a4a10 msgr2=0x7f99041332c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.229+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 shutdown_connections 2023-10-30T12:00:59.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.229+0000 7f990c275640 1 -- 172.21.15.50:0/2833483433 wait complete. 2023-10-30T12:00:59.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.229+0000 7f990c275640 1 librados: shutdown 2023-10-30T12:00:59.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:00:59.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:00:59.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:00:59.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.427+0000 7fc85b94c640 10 monclient: get_monmap_and_config 2023-10-30T12:00:59.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.427+0000 7fc85b94c640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.427+0000 7fc85b94c640 10 monclient: monmap: 2023-10-30T12:00:59.430 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.430 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.430 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:59.428537+0000 2023-10-30T12:00:59.430 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:59.428537+0000 2023-10-30T12:00:59.430 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.431 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.431 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:59.431 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:59.431 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:59.431 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:59.432 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:59.432 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 1 Processor -- start 2023-10-30T12:00:59.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 1 -- start start 2023-10-30T12:00:59.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient: init 2023-10-30T12:00:59.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-10-30T12:00:59.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc854071c00 0x7fc85409a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient(hunting): picked mon.noname-b con 0x7fc854071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc854097fe0 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:59.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc85409b170 0x7fc854139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient(hunting): picked mon.noname-c con 0x7fc85409b170 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:59.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc854097600 con 0x7fc854097fe0 2023-10-30T12:00:59.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8540978e0 con 0x7fc85409b170 2023-10-30T12:00:59.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85409bf50 con 0x7fc854071c00 2023-10-30T12:00:59.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc85b94c640 10 monclient(hunting): authenticate will time out at 904.611728s 2023-10-30T12:00:59.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc854071c00 0x7fc85409a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc858ee7640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc854097fe0 0x7fc85409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35290/0 (socket says 172.21.15.50:35290) 2023-10-30T12:00:59.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc858ee7640 1 -- 172.21.15.50:0/805970 learned_addr learned my addr 172.21.15.50:0/805970 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:59.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 10 monclient(hunting): get_auth_request con 0x7fc854071c00 auth_method 0 2023-10-30T12:00:59.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.428+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc859ee9640 1 --2- 172.21.15.50:0/805970 >> v2:172.21.15.118:3300/0 conn(0x7fc85409b170 0x7fc854139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc859ee9640 10 monclient(hunting): get_auth_request con 0x7fc85409b170 auth_method 0 2023-10-30T12:00:59.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc859ee9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc859ee9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc859ee9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2023-10-30T12:00:59.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 1 -- 172.21.15.50:0/805970 >> v2:172.21.15.118:3300/0 conn(0x7fc85409b170 msgr2=0x7fc854139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:59.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 1 --2- 172.21.15.50:0/805970 >> v2:172.21.15.118:3300/0 conn(0x7fc85409b170 0x7fc854139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 1 -- 172.21.15.50:0/805970 >> v1:172.21.15.50:6789/0 conn(0x7fc854097fe0 legacy=0x7fc85409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 1 monclient: found mon.noname-b 2023-10-30T12:00:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:00:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc853fff640 1 -- 172.21.15.50:0/805970 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625548784 0 0) 0x7fc854097600 con 0x7fc854097fe0 2023-10-30T12:00:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 1 -- 172.21.15.50:0/805970 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85409d9b0 con 0x7fc854071c00 2023-10-30T12:00:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc853fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:00:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.429+0000 7fc8596e8640 1 --2- 172.21.15.50:0/805970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc854071c00 0x7fc85409a150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc844009de0 tx=0x7fc84402bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=992f1e276411e72 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 1 -- 172.21.15.50:0/805970 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84402cb70 con 0x7fc854071c00 2023-10-30T12:00:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:00:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: dump: 2023-10-30T12:00:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.451 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 1 -- 172.21.15.50:0/805970 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84402ccd0 con 0x7fc854071c00 2023-10-30T12:00:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 5 monclient: authenticate success, global_id 4316 2023-10-30T12:00:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 1 -- 172.21.15.50:0/805970 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844034660 con 0x7fc854071c00 2023-10-30T12:00:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 4 set_mon_vals no callback set 2023-10-30T12:00:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc853fff640 10 monclient: dump: 2023-10-30T12:00:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.456 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 10 monclient: shutdown 2023-10-30T12:00:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 -- 172.21.15.50:0/805970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc854071c00 msgr2=0x7fc85409a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 --2- 172.21.15.50:0/805970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc854071c00 0x7fc85409a150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc844009de0 tx=0x7fc84402bdc0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 -- 172.21.15.50:0/805970 shutdown_connections 2023-10-30T12:00:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 --2- 172.21.15.50:0/805970 >> v2:172.21.15.118:3300/0 conn(0x7fc85409b170 0x7fc854139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 --2- 172.21.15.50:0/805970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc854071c00 0x7fc85409a150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 -- 172.21.15.50:0/805970 >> 172.21.15.50:0/805970 conn(0x7fc8540a3980 msgr2=0x7fc8540a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 -- 172.21.15.50:0/805970 shutdown_connections 2023-10-30T12:00:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.430+0000 7fc85b94c640 1 -- 172.21.15.50:0/805970 wait complete. 2023-10-30T12:00:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 10 monclient: monmap: 2023-10-30T12:00:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:59.460 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 1 librados: starting msgr at 2023-10-30T12:00:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 1 librados: starting objecter 2023-10-30T12:00:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 1 Processor -- start 2023-10-30T12:00:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 1 -- start start 2023-10-30T12:00:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 1 librados: setting wanted keys 2023-10-30T12:00:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 1 librados: calling monclient init 2023-10-30T12:00:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.431+0000 7fc85b94c640 10 monclient: init 2023-10-30T12:00:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:00:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854071c00 0x7fc8541c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): picked mon.noname-b con 0x7fc854071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc854097fe0 0x7fc854133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): picked mon.noname-a con 0x7fc854097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc85409b170 0x7fc854134210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): picked mon.noname-c con 0x7fc85409b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85409cc30 con 0x7fc854097fe0 2023-10-30T12:00:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc854070f30 con 0x7fc854071c00 2023-10-30T12:00:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc854128f10 con 0x7fc85409b170 2023-10-30T12:00:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient(hunting): authenticate will time out at 904.615198s 2023-10-30T12:00:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc854097fe0 0x7fc854133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc85409b170 0x7fc854134210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc854097fe0 0x7fc854133cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42322/0 (socket says 172.21.15.50:42322) 2023-10-30T12:00:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 1 -- 172.21.15.50:0/904265931 learned_addr learned my addr 172.21.15.50:0/904265931 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 10 monclient(hunting): get_auth_request con 0x7fc854097fe0 auth_method 0 2023-10-30T12:00:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc85409b170 0x7fc854134210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43048/0 (socket says 172.21.15.50:43048) 2023-10-30T12:00:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): get_auth_request con 0x7fc85409b170 auth_method 0 2023-10-30T12:00:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc858ee7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc8596e8640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854071c00 0x7fc8541c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc8596e8640 10 monclient(hunting): get_auth_request con 0x7fc854071c00 auth_method 0 2023-10-30T12:00:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc8596e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc8596e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc8596e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2023-10-30T12:00:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854071c00 msgr2=0x7fc8541c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854071c00 0x7fc8541c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc854097fe0 msgr2=0x7fc854133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc854097fe0 0x7fc854133cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 monclient: found mon.noname-c 2023-10-30T12:00:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:00:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 -- 172.21.15.50:0/904265931 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844009a70 con 0x7fc85409b170 2023-10-30T12:00:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc859ee9640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc85409b170 0x7fc854134210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc84c007850 tx=0x7fc84c007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 client.?.objecter ms_handle_connect 0x7fc85409b170 2023-10-30T12:00:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c014070 con 0x7fc85409b170 2023-10-30T12:00:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:00:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: dump: 2023-10-30T12:00:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.474 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 5 monclient: authenticate success, global_id 4319 2023-10-30T12:00:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c0096a0 con 0x7fc85409b170 2023-10-30T12:00:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c009a30 con 0x7fc85409b170 2023-10-30T12:00:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient: _renew_subs 2023-10-30T12:00:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc854129450 con 0x7fc85409b170 2023-10-30T12:00:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc8537fe640 4 set_mon_vals no callback set 2023-10-30T12:00:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:00:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.432+0000 7fc851ffb640 10 monclient: dump: 2023-10-30T12:00:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.479 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 10 monclient: _renew_subs 2023-10-30T12:00:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:00:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85413b900 con 0x7fc85409b170 2023-10-30T12:00:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 1 librados: init done 2023-10-30T12:00:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8541c84e0 0x7fc854138e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc84c018070 con 0x7fc85409b170 2023-10-30T12:00:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.433+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc854073ff0 con 0x7fc8541c84e0 2023-10-30T12:00:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc840039320 0x7fc84003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fc84c04dae0 con 0x7fc85409b170 2023-10-30T12:00:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8541c84e0 0x7fc854138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 10 client.4319.objecter ms_dispatch 0x7fc854000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 3 client.4319.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 3 client.4319.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient: get_auth_request con 0x7fc8541c84e0 auth_method 0 2023-10-30T12:00:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 20 client.4319.objecter dump_active .. 0 homeless 2023-10-30T12:00:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc858ee7640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc840039320 0x7fc84003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc858ee7640 10 monclient: get_auth_request con 0x7fc840039320 auth_method 0 2023-10-30T12:00:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2023-10-30T12:00:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 1 -- 172.21.15.50:0/904265931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc854071fe0 con 0x7fc8541c84e0 2023-10-30T12:00:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 10 client.4319.objecter ms_handle_connect 0x7fc840039320 2023-10-30T12:00:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc858ee7640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc840039320 0x7fc84003b7e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc85413bd90 tx=0x7fc848007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 10 client.4319.objecter ms_handle_connect 0x7fc8541c84e0 2023-10-30T12:00:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc851ffb640 10 client.4319.objecter resend_mon_ops 2023-10-30T12:00:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.434+0000 7fc8596e8640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8541c84e0 0x7fc854138e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc85409fe20 tx=0x7fc844004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844003b90 con 0x7fc8541c84e0 2023-10-30T12:00:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 10 monclient: dump: 2023-10-30T12:00:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.488 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc844003cf0 con 0x7fc8541c84e0 2023-10-30T12:00:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8541c84e0 msgr2=0x7fc854138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8541c84e0 0x7fc854138e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc85409fe20 tx=0x7fc844004a40 comp rx=0 tx=0).stop 2023-10-30T12:00:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8541c84e0 msgr2=0x7fc854138e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:00:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.435+0000 7fc851ffb640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8541c84e0 0x7fc854138e00 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc85409fe20 tx=0x7fc844004a40 comp rx=0 tx=0).stop 2023-10-30T12:00:59.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc85b94c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:00:59.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc85b94c640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854139340 0x7fc854139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc85b94c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc854139c90 con 0x7fc854139340 2023-10-30T12:00:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc8596e8640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854139340 0x7fc854139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc8596e8640 10 monclient: get_auth_request con 0x7fc854139340 auth_method 0 2023-10-30T12:00:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc8596e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc8596e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.441+0000 7fc8596e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc8596e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc8596e8640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2023-10-30T12:00:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc8596e8640 1 -- 172.21.15.50:0/904265931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc854129690 con 0x7fc854139340 2023-10-30T12:00:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc8596e8640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854139340 0x7fc854139750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc84402be00 tx=0x7fc844031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 10 client.4319.objecter ms_handle_connect 0x7fc854139340 2023-10-30T12:00:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 10 client.4319.objecter resend_mon_ops 2023-10-30T12:00:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844052850 con 0x7fc854139340 2023-10-30T12:00:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:00:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 10 monclient: dump: 2023-10-30T12:00:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.497 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fc844004890 con 0x7fc854139340 2023-10-30T12:00:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:00:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854139340 msgr2=0x7fc854139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854139340 0x7fc854139750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc84402be00 tx=0x7fc844031000 comp rx=0 tx=0).stop 2023-10-30T12:00:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854139340 msgr2=0x7fc854139750 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:00:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.442+0000 7fc851ffb640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc854139340 0x7fc854139750 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc84402be00 tx=0x7fc844031000 comp rx=0 tx=0).stop 2023-10-30T12:00:59.498 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:00:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:00:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:00:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:00:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 4, 2023-10-30T12:00:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:00:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:00:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:00:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:00:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 19, 2023-10-30T12:00:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:00:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:00:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:00:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:00:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:00:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:00:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:00:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:00:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:00:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:00:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:00:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:00:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:00:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:00:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:00:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:00:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:00:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:00:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:00:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:00:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:00:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:00:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:00:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:00:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:00:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:00:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:00:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:00:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:00:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:00:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:00:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:00:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:00:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:00:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:00:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:00:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:00:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:00:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:00:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:00:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:00:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:00:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:00:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:00:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:00:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:00:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:00:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:00:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:00:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:00:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:00:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:00:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:00:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:00:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:00:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:00:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:00:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:00:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:00:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:00:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:00:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:00:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:00:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:00:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:00:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:00:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:00:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:00:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:00:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:00:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:00:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:00:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:00:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:00:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:00:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:00:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:00:59.531 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:00:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 10 librados: watch_flush enter 2023-10-30T12:00:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 10 librados: watch_flush exit 2023-10-30T12:00:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 20 client.4319.objecter shutdown clearing up homeless session... 2023-10-30T12:00:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 10 client.4319.objecter successfully canceled tick 2023-10-30T12:00:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc840039320 msgr2=0x7fc84003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc840039320 0x7fc84003b7e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc85413bd90 tx=0x7fc848007e10 comp rx=0 tx=0).stop 2023-10-30T12:00:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 10 monclient: shutdown 2023-10-30T12:00:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc85409b170 msgr2=0x7fc854134210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc859ee9640 1 -- 172.21.15.50:0/904265931 reap_dead start 2023-10-30T12:00:59.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc85409b170 0x7fc854134210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc84c007850 tx=0x7fc84c007d10 comp rx=0 tx=0).stop 2023-10-30T12:00:59.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 shutdown_connections 2023-10-30T12:00:59.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 --2- 172.21.15.50:0/904265931 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc85409b170 0x7fc854134210 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 >> 172.21.15.50:0/904265931 conn(0x7fc8540a3980 msgr2=0x7fc854136cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 shutdown_connections 2023-10-30T12:00:59.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 -- 172.21.15.50:0/904265931 wait complete. 2023-10-30T12:00:59.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.443+0000 7fc85b94c640 1 librados: shutdown 2023-10-30T12:00:59.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:00:59.537 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-10-30T12:00:59.537 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:00:59.537 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-10-30T12:00:59.538 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:00:59.538 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:00:59.538 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:00:59.548 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:00:59.548 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:00:59.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:00:59.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.691+0000 7f12a909b640 10 monclient: get_monmap_and_config 2023-10-30T12:00:59.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.691+0000 7f12a909b640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.692+0000 7f12a909b640 10 monclient: monmap: 2023-10-30T12:00:59.694 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.694 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.694 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:59.693061+0000 2023-10-30T12:00:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:59.693061+0000 2023-10-30T12:00:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:00:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:00:59.696 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:00:59.696 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:00:59.696 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:00:59.696 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.692+0000 7f12a909b640 1 Processor -- start 2023-10-30T12:00:59.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.692+0000 7f12a909b640 1 -- start start 2023-10-30T12:00:59.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.692+0000 7f12a909b640 10 monclient: init 2023-10-30T12:00:59.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-10-30T12:00:59.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12a40a0080 addr v1:172.21.15.50:6789/0 2023-10-30T12:00:59.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f12a40720b0 0x7f12a413ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient(hunting): picked mon.noname-c con 0x7f12a40720b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:00:59.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 0x7f12a413d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient(hunting): picked mon.noname-b con 0x7f12a4073a30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a409ccc0 con 0x7f12a40a0080 2023-10-30T12:00:59.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a4073620 con 0x7f12a40720b0 2023-10-30T12:00:59.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4071e90 con 0x7f12a4073a30 2023-10-30T12:00:59.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 10 monclient(hunting): authenticate will time out at 904.876118s 2023-10-30T12:00:59.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a359e640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 msgr2=0x7f12a413d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:00:59.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a359e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 0x7f12a413d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:00:59.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a2d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f12a40a0080 0x7f12a40a0460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35304/0 (socket says 172.21.15.50:35304) 2023-10-30T12:00:59.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1276869648 learned_addr learned my addr 172.21.15.50:0/1276869648 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:00:59.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460604859 0 0) 0x7f12a409ccc0 con 0x7f12a40a0080 2023-10-30T12:00:59.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:00:59.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 1 --2- 172.21.15.50:0/1276869648 >> v2:172.21.15.118:3300/0 conn(0x7f12a40720b0 0x7f12a413ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): my global_id is 4413 2023-10-30T12:00:59.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128c003520 con 0x7f12a40a0080 2023-10-30T12:00:59.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 10 monclient(hunting): get_auth_request con 0x7f12a40720b0 auth_method 0 2023-10-30T12:00:59.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297224440 0 0) 0x7f128c003520 con 0x7f12a40a0080 2023-10-30T12:00:59.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 msgr2=0x7f12a413d280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:00:59.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 0x7f12a413d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 >> v2:172.21.15.118:3300/0 conn(0x7f12a40720b0 msgr2=0x7f12a413ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:59.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 --2- 172.21.15.50:0/1276869648 >> v2:172.21.15.118:3300/0 conn(0x7f12a40720b0 0x7f12a413ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:00:59.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:00:59.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a409c2b0 con 0x7f12a40a0080 2023-10-30T12:00:59.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 5 monclient: authenticate success, global_id 4413 2023-10-30T12:00:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f12940017f0 con 0x7f12a40a0080 2023-10-30T12:00:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:00:59.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: dump: 2023-10-30T12:00:59.717 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.717 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.717 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.717 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.718 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:00:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 >> v1:172.21.15.50:6789/0 conn(0x7f12a40a0080 legacy=0x7f12a40a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:00:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f128c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): picked mon.a con 0x7f128c003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:59.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 0x7f128c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): picked mon.c con 0x7f12a4073a30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f128c004d00 0x7f128c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): picked mon.b con 0x7f128c004d00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4073620 con 0x7f128c003970 2023-10-30T12:00:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4071e90 con 0x7f128c004d00 2023-10-30T12:00:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128c00bb90 con 0x7f12a4073a30 2023-10-30T12:00:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a1d9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:00:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 1 -- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 msgr2=0x7f128c0047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:00:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a909b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a259c640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 0x7f128c0047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:00:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.693+0000 7f12a2d9d640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f128c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): get_auth_request con 0x7f128c003970 auth_method 0 2023-10-30T12:00:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f128c004d00 0x7f128c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 10 monclient(hunting): get_auth_request con 0x7f128c004d00 auth_method 0 2023-10-30T12:00:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:00:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:00:59.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_done global_id 4413 payload 293 2023-10-30T12:00:59.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 msgr2=0x7f128c0047c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:00:59.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4073a30 0x7f128c0047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1276869648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f128c004d00 msgr2=0x7f128c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:00:59.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f128c004d00 0x7f128c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 1 monclient: found mon.a 2023-10-30T12:00:59.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1276869648 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c003520 con 0x7f128c003970 2023-10-30T12:00:59.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.694+0000 7f12a2d9d640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f128c003d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f129400c190 tx=0x7f128c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e50efff8e7b30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294007270 con 0x7f128c003970 2023-10-30T12:00:59.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 10 monclient: dump: 2023-10-30T12:00:59.729 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.731 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12940073f0 con 0x7f128c003970 2023-10-30T12:00:59.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:00:59.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 1 -- 172.21.15.50:0/1276869648 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129402c8a0 con 0x7f128c003970 2023-10-30T12:00:59.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 10 monclient: get_monmap_and_config success 2023-10-30T12:00:59.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 4 set_mon_vals no callback set 2023-10-30T12:00:59.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a1d9b640 10 monclient: dump: 2023-10-30T12:00:59.733 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.733 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.734 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.734 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.734 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.734 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.734 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.734 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.735 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.735 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 10 monclient: shutdown 2023-10-30T12:00:59.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 1 -- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 msgr2=0x7f128c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 1 --2- 172.21.15.50:0/1276869648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f128c003d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f129400c190 tx=0x7f128c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a359e640 1 -- 172.21.15.50:0/1276869648 reap_dead start 2023-10-30T12:00:59.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 1 -- 172.21.15.50:0/1276869648 shutdown_connections 2023-10-30T12:00:59.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 1 -- 172.21.15.50:0/1276869648 >> 172.21.15.50:0/1276869648 conn(0x7f12a40a4870 msgr2=0x7f12a40a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 1 -- 172.21.15.50:0/1276869648 shutdown_connections 2023-10-30T12:00:59.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.695+0000 7f12a909b640 1 -- 172.21.15.50:0/1276869648 wait complete. 2023-10-30T12:00:59.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 10 monclient: build_initial_monmap 2023-10-30T12:00:59.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 10 monclient: monmap: 2023-10-30T12:00:59.737 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:00:59.737 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:00:59.737 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:00:59.738 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:00:59.738 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:00:59.738 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.738 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:00:59.738 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:00:59.739 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:00:59.739 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 1 librados: starting msgr at 2023-10-30T12:00:59.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 1 librados: starting objecter 2023-10-30T12:00:59.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 1 Processor -- start 2023-10-30T12:00:59.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 1 -- start start 2023-10-30T12:00:59.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 1 librados: setting wanted keys 2023-10-30T12:00:59.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 1 librados: calling monclient init 2023-10-30T12:00:59.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.696+0000 7f12a909b640 10 monclient: init 2023-10-30T12:00:59.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient: _reopen_session rank -1 2023-10-30T12:00:59.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:00:59.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f12a41ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): picked mon.noname-a con 0x7f128c003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:00:59.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a4090d10 0x7f12a4096730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): picked mon.noname-b con 0x7f12a4090d10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:00:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4096c70 0x7f12a41c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): picked mon.noname-c con 0x7f12a4096c70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:00:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a40919a0 con 0x7f128c003970 2023-10-30T12:00:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4091c10 con 0x7f12a4090d10 2023-10-30T12:00:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:00:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4091ef0 con 0x7f12a4096c70 2023-10-30T12:00:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): _renew_subs 2023-10-30T12:00:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a909b640 10 monclient(hunting): authenticate will time out at 904.880456s 2023-10-30T12:00:59.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a359e640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4096c70 msgr2=0x7f12a41c6c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:00:59.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a359e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4096c70 0x7f12a41c6c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:00:59.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f12a41ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f12a41ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42350/0 (socket says 172.21.15.50:42350) 2023-10-30T12:00:59.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1049821891 learned_addr learned my addr 172.21.15.50:0/1049821891 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:00:59.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): get_auth_request con 0x7f128c003970 auth_method 0 2023-10-30T12:00:59.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:00:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:00:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:00:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a259c640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a4090d10 0x7f12a4096730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a259c640 10 monclient(hunting): get_auth_request con 0x7f12a4090d10 auth_method 0 2023-10-30T12:00:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:00:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:00:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:00:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2023-10-30T12:00:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1049821891 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4096c70 msgr2=0x7f12a41c6c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:00:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4096c70 0x7f12a41c6c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a4090d10 msgr2=0x7f12a4096730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a4090d10 0x7f12a4096730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:00:59.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 monclient: found mon.noname-a 2023-10-30T12:00:59.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:00:59.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.697+0000 7f12a2d9d640 1 -- 172.21.15.50:0/1049821891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a40921d0 con 0x7f128c003970 2023-10-30T12:00:59.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a2d9d640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f12a41ca120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f129400c2b0 tx=0x7f1294001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 client.?.objecter ms_handle_connect 0x7f128c003970 2023-10-30T12:00:59.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:00:59.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 1 -- 172.21.15.50:0/1049821891 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129403c040 con 0x7f128c003970 2023-10-30T12:00:59.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:00:59.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: dump: 2023-10-30T12:00:59.750 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.750 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.751 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.751 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.751 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.751 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.751 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.752 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.752 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.752 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:00:59.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:00:59.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:00:59.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 5 monclient: authenticate success, global_id 4416 2023-10-30T12:00:59.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 1 -- 172.21.15.50:0/1049821891 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294007280 con 0x7f128c003970 2023-10-30T12:00:59.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 10 monclient: _renew_subs 2023-10-30T12:00:59.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a41cab80 con 0x7f128c003970 2023-10-30T12:00:59.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:00:59.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 1 -- 172.21.15.50:0/1049821891 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12940059e0 con 0x7f128c003970 2023-10-30T12:00:59.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:00:59.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:00:59.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f128b7fe640 10 monclient: dump: 2023-10-30T12:00:59.755 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:00:59.755 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:00:59.755 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.755 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:00:59.755 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:00:59.756 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:00:59.756 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:00:59.756 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:00:59.756 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:00:59.756 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:00:59.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:00:59.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 10 monclient: _renew_subs 2023-10-30T12:00:59.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a159a640 4 set_mon_vals no callback set 2023-10-30T12:00:59.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a41c7160 con 0x7f128c003970 2023-10-30T12:00:59.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.698+0000 7f12a909b640 1 librados: init done 2023-10-30T12:00:59.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f128b7fe640 1 -- 172.21.15.50:0/1049821891 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1294032070 con 0x7f128c003970 2023-10-30T12:00:59.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f12a909b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f12a909b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f12a909b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a404e160 con 0x7f128c003970 2023-10-30T12:00:59.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f128b7fe640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12900392a0 0x7f129003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:00:59.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f128b7fe640 1 -- 172.21.15.50:0/1049821891 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f129406b5f0 con 0x7f128c003970 2023-10-30T12:00:59.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f128b7fe640 10 client.4416.objecter ms_dispatch 0x7f12a4000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:00:59.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f128b7fe640 3 client.4416.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:00:59.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f128b7fe640 3 client.4416.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:00:59.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f128b7fe640 20 client.4416.objecter dump_active .. 0 homeless 2023-10-30T12:00:59.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f12a259c640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12900392a0 0x7f129003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:00:59.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.699+0000 7f12a259c640 10 monclient: get_auth_request con 0x7f12900392a0 auth_method 0 2023-10-30T12:00:59.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.700+0000 7f128b7fe640 10 client.4416.objecter ms_handle_connect 0x7f12900392a0 2023-10-30T12:00:59.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.700+0000 7f12a259c640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12900392a0 0x7f129003b760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12a4091110 tx=0x7f1298008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:00:59.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.704+0000 7f128b7fe640 1 -- 172.21.15.50:0/1049821891 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12940343d0 con 0x7f128c003970 2023-10-30T12:00:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.704+0000 7f128b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:00:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.704+0000 7f128b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:00:59.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.946+0000 7f12a909b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:00:59.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.946+0000 7f12a909b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:00:59.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.946+0000 7f12a909b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:00:59.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.946+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f12a409d910 con 0x7f128c003970 2023-10-30T12:00:59.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.946+0000 7f128b7fe640 1 -- 172.21.15.50:0/1049821891 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f1294033b20 con 0x7f128c003970 2023-10-30T12:00:59.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.946+0000 7f128b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:00:59.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.946+0000 7f128b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:00:59.950 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:00:59.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 10 librados: watch_flush enter 2023-10-30T12:00:59.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 10 librados: watch_flush exit 2023-10-30T12:00:59.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 20 client.4416.objecter shutdown clearing up homeless session... 2023-10-30T12:00:59.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 10 client.4416.objecter successfully canceled tick 2023-10-30T12:00:59.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12900392a0 msgr2=0x7f129003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12900392a0 0x7f129003b760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12a4091110 tx=0x7f1298008040 comp rx=0 tx=0).stop 2023-10-30T12:00:59.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 10 monclient: shutdown 2023-10-30T12:00:59.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:00:59.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 msgr2=0x7f12a41ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:00:59.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f12a41ca120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f129400c2b0 tx=0x7f1294001b80 comp rx=0 tx=0).stop 2023-10-30T12:00:59.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 shutdown_connections 2023-10-30T12:00:59.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12900392a0 0x7f129003b760 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a4096c70 0x7f12a41c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a4090d10 0x7f12a4096730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 --2- 172.21.15.50:0/1049821891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f128c003970 0x7f12a41ca120 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:00:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 >> 172.21.15.50:0/1049821891 conn(0x7f12a40a4870 msgr2=0x7f12a41bcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:00:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 shutdown_connections 2023-10-30T12:00:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 -- 172.21.15.50:0/1049821891 wait complete. 2023-10-30T12:00:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:00:59.948+0000 7f12a909b640 1 librados: shutdown 2023-10-30T12:01:02.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:03.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.078+0000 7fa1a4325640 10 monclient: get_monmap_and_config 2023-10-30T12:01:03.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.078+0000 7fa1a4325640 10 monclient: build_initial_monmap 2023-10-30T12:01:03.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.078+0000 7fa1a4325640 10 monclient: monmap: 2023-10-30T12:01:03.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:03.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:03.082 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:03.080029+0000 2023-10-30T12:01:03.082 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:03.080029+0000 2023-10-30T12:01:03.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:03.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:03.082 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:03.083 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:03.083 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:03.083 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:03.083 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:03.084 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.079+0000 7fa1a4325640 1 Processor -- start 2023-10-30T12:01:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.079+0000 7fa1a4325640 1 -- start start 2023-10-30T12:01:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.079+0000 7fa1a4325640 10 monclient: init 2023-10-30T12:01:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-10-30T12:01:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c0a2dd0 0x7fa19c0a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c0a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa19c06a8e0 0x7fa19c0a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c06a8e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa19c071400 0x7fa19c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c071400 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa19c0a31b0 con 0x7fa19c071400 2023-10-30T12:01:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa19c06a6a0 con 0x7fa19c06a8e0 2023-10-30T12:01:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c06be90 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a4325640 10 monclient(hunting): authenticate will time out at 908.263274s 2023-10-30T12:01:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a20c1640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c0a2dd0 msgr2=0x7fa19c0a46c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:03.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a20c1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c0a2dd0 0x7fa19c0a46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:03.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa19c071400 0x7fa19c0a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:03.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa19c071400 0x7fa19c0a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42358/0 (socket says 172.21.15.50:42358) 2023-10-30T12:01:03.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa19c06a8e0 0x7fa19c0a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:03.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 -- 172.21.15.50:0/2623194085 learned_addr learned my addr 172.21.15.50:0/2623194085 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:03.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): get_auth_request con 0x7fa19c071400 auth_method 0 2023-10-30T12:01:03.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:03.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:03.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:03.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 10 monclient(hunting): get_auth_request con 0x7fa19c06a8e0 auth_method 0 2023-10-30T12:01:03.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:03.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:03.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:03.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:03.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:03.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:03.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:03.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:03.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a18c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:03.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient(hunting): handle_auth_done global_id 4419 payload 274 2023-10-30T12:01:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 -- 172.21.15.50:0/2623194085 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c0a2dd0 msgr2=0x7fa19c0a46c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 --2- 172.21.15.50:0/2623194085 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c0a2dd0 0x7fa19c0a46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 -- 172.21.15.50:0/2623194085 >> v2:172.21.15.118:3300/0 conn(0x7fa19c06a8e0 msgr2=0x7fa19c0a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 --2- 172.21.15.50:0/2623194085 >> v2:172.21.15.118:3300/0 conn(0x7fa19c06a8e0 0x7fa19c0a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient: _finish_hunting 0 2023-10-30T12:01:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 monclient: found mon.noname-a 2023-10-30T12:01:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.080+0000 7fa1a28c2640 1 -- 172.21.15.50:0/2623194085 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c09e190 con 0x7fa19c071400 2023-10-30T12:01:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a28c2640 1 --2- 172.21.15.50:0/2623194085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c071400 0x7fa19c0a5140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa19c069e30 tx=0x7fa19802bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3608e300aabbdc04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 1 -- 172.21.15.50:0/2623194085 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19802c980 con 0x7fa19c071400 2023-10-30T12:01:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: dump: 2023-10-30T12:01:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:03.104 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:03.104 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:03.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: _finish_auth 0 2023-10-30T12:01:03.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: _check_auth_tickets 2023-10-30T12:01:03.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:03.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 1 -- 172.21.15.50:0/2623194085 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19802cae0 con 0x7fa19c071400 2023-10-30T12:01:03.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a4325640 5 monclient: authenticate success, global_id 4419 2023-10-30T12:01:03.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a4325640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:03.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:03.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a4325640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:03.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a4325640 4 set_mon_vals no callback set 2023-10-30T12:01:03.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 1 -- 172.21.15.50:0/2623194085 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19802ce70 con 0x7fa19c071400 2023-10-30T12:01:03.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:03.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:03.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.081+0000 7fa1a10bf640 10 monclient: dump: 2023-10-30T12:01:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:03.108 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:03.108 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:03.108 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:03.108 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 10 monclient: shutdown 2023-10-30T12:01:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 -- 172.21.15.50:0/2623194085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c071400 msgr2=0x7fa19c0a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 --2- 172.21.15.50:0/2623194085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c071400 0x7fa19c0a5140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa19c069e30 tx=0x7fa19802bde0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 -- 172.21.15.50:0/2623194085 shutdown_connections 2023-10-30T12:01:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 --2- 172.21.15.50:0/2623194085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c071400 0x7fa19c0a5140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 --2- 172.21.15.50:0/2623194085 >> v2:172.21.15.118:3300/0 conn(0x7fa19c06a8e0 0x7fa19c0a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 --2- 172.21.15.50:0/2623194085 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c0a2dd0 0x7fa19c0a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 -- 172.21.15.50:0/2623194085 >> 172.21.15.50:0/2623194085 conn(0x7fa19c0a22b0 msgr2=0x7fa19c0a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 -- 172.21.15.50:0/2623194085 shutdown_connections 2023-10-30T12:01:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 1 -- 172.21.15.50:0/2623194085 wait complete. 2023-10-30T12:01:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 10 monclient: build_initial_monmap 2023-10-30T12:01:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.082+0000 7fa1a4325640 10 monclient: monmap: 2023-10-30T12:01:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:03.113 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 librados: starting msgr at 2023-10-30T12:01:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 librados: starting objecter 2023-10-30T12:01:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 Processor -- start 2023-10-30T12:01:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 -- start start 2023-10-30T12:01:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 librados: setting wanted keys 2023-10-30T12:01:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 librados: calling monclient init 2023-10-30T12:01:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient: init 2023-10-30T12:01:03.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:03.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:01:03.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c071400 0x7fa19c1ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:03.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c071400 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:03.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c0a2dd0 0x7fa19c139150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c0a2dd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa19c123580 0x7fa19c120a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c123580 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c1ca840 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c124250 con 0x7fa19c123580 2023-10-30T12:01:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c123190 con 0x7fa19c071400 2023-10-30T12:01:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a4325640 10 monclient(hunting): authenticate will time out at 908.266796s 2023-10-30T12:01:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a20c1640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c071400 msgr2=0x7fa19c1ca300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a20c1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c071400 0x7fa19c1ca300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a18c0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c0a2dd0 0x7fa19c139150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a18c0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c0a2dd0 0x7fa19c139150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42372/0 (socket says 172.21.15.50:42372) 2023-10-30T12:01:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a18c0640 1 -- 172.21.15.50:0/521620444 learned_addr learned my addr 172.21.15.50:0/521620444 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a18c0640 10 monclient(hunting): get_auth_request con 0x7fa19c0a2dd0 auth_method 0 2023-10-30T12:01:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a18c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a18c0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a18c0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a28c2640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa19c123580 0x7fa19c120a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a28c2640 10 monclient(hunting): get_auth_request con 0x7fa19c123580 auth_method 0 2023-10-30T12:01:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a28c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a28c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.083+0000 7fa1a28c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a28c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a28c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a28c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2023-10-30T12:01:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 1 -- 172.21.15.50:0/521620444 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c071400 msgr2=0x7fa19c1ca300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c071400 0x7fa19c1ca300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 1 -- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa19c123580 msgr2=0x7fa19c120a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa19c123580 0x7fa19c120a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 10 monclient: _finish_hunting 0 2023-10-30T12:01:03.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 1 monclient: found mon.noname-a 2023-10-30T12:01:03.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:03.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:03.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 1 -- 172.21.15.50:0/521620444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa198009b30 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a28c2640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa19c123580 0x7fa19c120a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:03.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a18c0640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c0a2dd0 0x7fa19c139150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00e8a0 tx=0x7fa18c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:03.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 client.?.objecter ms_handle_connect 0x7fa19c0a2dd0 2023-10-30T12:01:03.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:03.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 1 -- 172.21.15.50:0/521620444 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0094f0 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:03.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:03.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: dump: 2023-10-30T12:01:03.125 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:03.125 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:03.125 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.126 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.126 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:03.126 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:03.126 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:03.126 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: _finish_auth 0 2023-10-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:03.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 1 -- 172.21.15.50:0/521620444 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c00c940 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 5 monclient: authenticate success, global_id 4422 2023-10-30T12:01:03.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 10 monclient: _renew_subs 2023-10-30T12:01:03.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:03.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19c06b0a0 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:03.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 1 -- 172.21.15.50:0/521620444 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c00ccf0 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:03.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:03.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa192ffd640 10 monclient: dump: 2023-10-30T12:01:03.129 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:03.130 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:03.130 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.130 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:03.130 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:03.130 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:03.131 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:03.131 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:03.131 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:03.131 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:03.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:03.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 10 monclient: _renew_subs 2023-10-30T12:01:03.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a08be640 4 set_mon_vals no callback set 2023-10-30T12:01:03.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:03.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19c122720 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.084+0000 7fa1a4325640 1 librados: init done 2023-10-30T12:01:03.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa192ffd640 1 -- 172.21.15.50:0/521620444 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa18c013470 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa1a4325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:03.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa1a4325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:03.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa1a4325640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:03.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19c04e160 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa192ffd640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1800392a0 0x7fa18003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:03.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa192ffd640 1 -- 172.21.15.50:0/521620444 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fa18c04a650 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa192ffd640 10 client.4422.objecter ms_dispatch 0x7fa19c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:03.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa192ffd640 3 client.4422.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:03.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa192ffd640 3 client.4422.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:03.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa192ffd640 20 client.4422.objecter dump_active .. 0 homeless 2023-10-30T12:01:03.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.085+0000 7fa1a20c1640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1800392a0 0x7fa18003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:03.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.086+0000 7fa1a20c1640 10 monclient: get_auth_request con 0x7fa1800392a0 auth_method 0 2023-10-30T12:01:03.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.086+0000 7fa1a20c1640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1800392a0 0x7fa18003b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa194006fd0 tx=0x7fa194006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:03.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.086+0000 7fa192ffd640 10 client.4422.objecter ms_handle_connect 0x7fa1800392a0 2023-10-30T12:01:03.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.090+0000 7fa192ffd640 1 -- 172.21.15.50:0/521620444 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa18c00ce50 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.090+0000 7fa192ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:03.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.090+0000 7fa192ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:03.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.332+0000 7fa1a4325640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:03.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.332+0000 7fa1a4325640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.332+0000 7fa1a4325640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.332+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa19c06e050 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.332+0000 7fa192ffd640 1 -- 172.21.15.50:0/521620444 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fa18c01bb10 con 0x7fa19c0a2dd0 2023-10-30T12:01:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.332+0000 7fa192ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.332+0000 7fa192ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:03.336 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:03.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 10 librados: watch_flush enter 2023-10-30T12:01:03.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 10 librados: watch_flush exit 2023-10-30T12:01:03.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 20 client.4422.objecter shutdown clearing up homeless session... 2023-10-30T12:01:03.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 10 client.4422.objecter successfully canceled tick 2023-10-30T12:01:03.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1800392a0 msgr2=0x7fa18003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:03.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1800392a0 0x7fa18003b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa194006fd0 tx=0x7fa194006d80 comp rx=0 tx=0).stop 2023-10-30T12:01:03.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 10 monclient: shutdown 2023-10-30T12:01:03.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:03.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c0a2dd0 msgr2=0x7fa19c139150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:03.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c0a2dd0 0x7fa19c139150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00e8a0 tx=0x7fa18c00ed60 comp rx=0 tx=0).stop 2023-10-30T12:01:03.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 shutdown_connections 2023-10-30T12:01:03.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1800392a0 0x7fa18003b760 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa19c123580 0x7fa19c120a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa19c0a2dd0 0x7fa19c139150 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 --2- 172.21.15.50:0/521620444 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa19c071400 0x7fa19c1ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:03.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 >> 172.21.15.50:0/521620444 conn(0x7fa19c0a22b0 msgr2=0x7fa19c0a0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:03.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 shutdown_connections 2023-10-30T12:01:03.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 -- 172.21.15.50:0/521620444 wait complete. 2023-10-30T12:01:03.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:03.334+0000 7fa1a4325640 1 librados: shutdown 2023-10-30T12:01:06.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:06.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.462+0000 7f6d09ccf640 10 monclient: get_monmap_and_config 2023-10-30T12:01:06.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.462+0000 7f6d09ccf640 10 monclient: build_initial_monmap 2023-10-30T12:01:06.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.462+0000 7f6d09ccf640 10 monclient: monmap: 2023-10-30T12:01:06.464 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:06.465 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:06.465 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:06.463294+0000 2023-10-30T12:01:06.465 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:06.463294+0000 2023-10-30T12:01:06.465 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:06.465 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:06.466 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:06.466 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:06.466 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:06.466 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:06.466 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:06.467 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:06.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 1 Processor -- start 2023-10-30T12:01:06.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 1 -- start start 2023-10-30T12:01:06.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient: init 2023-10-30T12:01:06.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:01:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d040a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6d04099a40 0x7f6d0413aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient(hunting): picked mon.noname-c con 0x7f6d04099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d04132db0 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040a4f40 con 0x7f6d04132db0 2023-10-30T12:01:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040a4150 con 0x7f6d040a4b60 2023-10-30T12:01:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d040a3d70 con 0x7f6d04099a40 2023-10-30T12:01:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d09ccf640 10 monclient(hunting): authenticate will time out at 911.646682s 2023-10-30T12:01:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d03fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6d04132db0 0x7f6d0413cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35308/0 (socket says 172.21.15.50:35308) 2023-10-30T12:01:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d03fff640 1 -- 172.21.15.50:0/3021951161 learned_addr learned my addr 172.21.15.50:0/3021951161 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d02ffd640 1 --2- 172.21.15.50:0/3021951161 >> v2:172.21.15.118:3300/0 conn(0x7f6d04099a40 0x7f6d0413aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request con 0x7f6d04099a40 auth_method 0 2023-10-30T12:01:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d02ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.463+0000 7f6d02ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_done global_id 4423 payload 274 2023-10-30T12:01:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 1 -- 172.21.15.50:0/3021951161 >> v1:172.21.15.118:6789/0 conn(0x7f6d040a4b60 legacy=0x7f6d041326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 1 -- 172.21.15.50:0/3021951161 >> v1:172.21.15.50:6789/0 conn(0x7f6d04132db0 legacy=0x7f6d0413cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:01:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 1 monclient: found mon.noname-c 2023-10-30T12:01:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:01:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 1 -- 172.21.15.50:0/3021951161 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0409a440 con 0x7f6d04099a40 2023-10-30T12:01:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.464+0000 7f6d02ffd640 1 --2- 172.21.15.50:0/3021951161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d04099a40 0x7f6d0413aa80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097a0 tx=0x7f6cf802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b33ec2cb6dd53660 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 1 -- 172.21.15.50:0/3021951161 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf802e070 con 0x7f6d04099a40 2023-10-30T12:01:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: dump: 2023-10-30T12:01:06.483 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:06.483 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:06.483 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.483 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.483 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:06.484 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:06.484 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:06.484 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:06.484 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:06.484 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:06.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: _finish_auth 0 2023-10-30T12:01:06.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:06.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:06.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 1 -- 172.21.15.50:0/3021951161 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf80315a0 con 0x7f6d04099a40 2023-10-30T12:01:06.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 5 monclient: authenticate success, global_id 4423 2023-10-30T12:01:06.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:06.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:06.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 1 -- 172.21.15.50:0/3021951161 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8031930 con 0x7f6d04099a40 2023-10-30T12:01:06.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:06.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 4 set_mon_vals no callback set 2023-10-30T12:01:06.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:06.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:06.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d027fc640 10 monclient: dump: 2023-10-30T12:01:06.487 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:06.487 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:06.488 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.488 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.488 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:06.488 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:06.488 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:06.489 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:06.489 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:06.489 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:06.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 10 monclient: shutdown 2023-10-30T12:01:06.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:06.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3021951161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d04099a40 msgr2=0x7f6d0413aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:06.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3021951161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d04099a40 0x7f6d0413aa80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097a0 tx=0x7f6cf802b840 comp rx=0 tx=0).stop 2023-10-30T12:01:06.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3021951161 shutdown_connections 2023-10-30T12:01:06.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3021951161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d04099a40 0x7f6d0413aa80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3021951161 >> 172.21.15.50:0/3021951161 conn(0x7f6d040a8060 msgr2=0x7f6d040a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:06.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3021951161 shutdown_connections 2023-10-30T12:01:06.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.465+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3021951161 wait complete. 2023-10-30T12:01:06.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 10 monclient: build_initial_monmap 2023-10-30T12:01:06.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 10 monclient: monmap: 2023-10-30T12:01:06.491 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:06.493 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:06.493 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:06.493 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:06.493 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:06.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 1 librados: starting msgr at 2023-10-30T12:01:06.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 1 librados: starting objecter 2023-10-30T12:01:06.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 1 Processor -- start 2023-10-30T12:01:06.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 1 -- start start 2023-10-30T12:01:06.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 1 librados: setting wanted keys 2023-10-30T12:01:06.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 1 librados: calling monclient init 2023-10-30T12:01:06.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.466+0000 7f6d09ccf640 10 monclient: init 2023-10-30T12:01:06.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:06.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:01:06.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d040a4b60 0x7f6d04136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:06.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): picked mon.noname-b con 0x7f6d040a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:06.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d04132db0 0x7f6d04136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:06.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): picked mon.noname-a con 0x7f6d04132db0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:06.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d041372d0 0x7f6d041ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:06.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): picked mon.noname-c con 0x7f6d041372d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:06.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:06.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d04139630 con 0x7f6d04132db0 2023-10-30T12:01:06.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:06.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d041384c0 con 0x7f6d040a4b60 2023-10-30T12:01:06.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:06.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040a18d0 con 0x7f6d041372d0 2023-10-30T12:01:06.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:06.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d09ccf640 10 monclient(hunting): authenticate will time out at 911.650361s 2023-10-30T12:01:06.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d03fff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d041372d0 msgr2=0x7f6d041ca7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:06.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d041372d0 0x7f6d041ca7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:06.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d04132db0 0x7f6d04136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:06.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d04132db0 0x7f6d04136d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42388/0 (socket says 172.21.15.50:42388) 2023-10-30T12:01:06.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 -- 172.21.15.50:0/3479389854 learned_addr learned my addr 172.21.15.50:0/3479389854 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:06.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request con 0x7f6d04132db0 auth_method 0 2023-10-30T12:01:06.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:06.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:06.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:06.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d040a4b60 0x7f6d04136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:06.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 10 monclient(hunting): get_auth_request con 0x7f6d040a4b60 auth_method 0 2023-10-30T12:01:06.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:06.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:06.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:06.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:06.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:06.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:06.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:06.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:06.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:06.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2023-10-30T12:01:06.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 -- 172.21.15.50:0/3479389854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d041372d0 msgr2=0x7f6d041ca7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:06.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d041372d0 0x7f6d041ca7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 -- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d040a4b60 msgr2=0x7f6d04136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:06.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d040a4b60 0x7f6d04136850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:01:06.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 monclient: found mon.noname-a 2023-10-30T12:01:06.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:06.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:06.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 -- 172.21.15.50:0/3479389854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf8009400 con 0x7f6d04132db0 2023-10-30T12:01:06.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6d02ffd640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d04132db0 0x7f6d04136d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8034040 tx=0x7f6cf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:06.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6cebfff640 10 client.?.objecter ms_handle_connect 0x7f6d04132db0 2023-10-30T12:01:06.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.467+0000 7f6cebfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:06.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 1 -- 172.21.15.50:0/3479389854 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8032900 con 0x7f6d04132db0 2023-10-30T12:01:06.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:06.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:06.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: dump: 2023-10-30T12:01:06.505 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:06.505 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:06.505 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.506 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.506 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:06.506 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:06.506 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:06.507 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:06.507 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:06.507 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:06.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: _finish_auth 0 2023-10-30T12:01:06.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:06.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:06.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 1 -- 172.21.15.50:0/3479389854 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf8032a60 con 0x7f6d04132db0 2023-10-30T12:01:06.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 5 monclient: authenticate success, global_id 4425 2023-10-30T12:01:06.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 10 monclient: _renew_subs 2023-10-30T12:01:06.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:06.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0413db50 con 0x7f6d04132db0 2023-10-30T12:01:06.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:06.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 1 -- 172.21.15.50:0/3479389854 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8032e10 con 0x7f6d04132db0 2023-10-30T12:01:06.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:06.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:06.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 10 monclient: dump: 2023-10-30T12:01:06.510 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:06.510 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:06.510 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.510 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:06.511 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:06.511 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:06.511 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:06.511 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:06.512 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:06.512 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:06.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:06.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 10 monclient: _renew_subs 2023-10-30T12:01:06.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d01ffb640 4 set_mon_vals no callback set 2023-10-30T12:01:06.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:06.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d04137b00 con 0x7f6d04132db0 2023-10-30T12:01:06.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6d09ccf640 1 librados: init done 2023-10-30T12:01:06.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.468+0000 7f6cebfff640 1 -- 172.21.15.50:0/3479389854 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6cf804e430 con 0x7f6d04132db0 2023-10-30T12:01:06.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6d09ccf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:06.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6d09ccf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:06.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6d09ccf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:06.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d04137c90 con 0x7f6d04132db0 2023-10-30T12:01:06.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6cebfff640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6cf00392a0 0x7f6cf003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:06.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6cebfff640 1 -- 172.21.15.50:0/3479389854 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f6cf806c5a0 con 0x7f6d04132db0 2023-10-30T12:01:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6cebfff640 10 client.4425.objecter ms_dispatch 0x7f6d04000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6cebfff640 3 client.4425.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6cebfff640 3 client.4425.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6cebfff640 20 client.4425.objecter dump_active .. 0 homeless 2023-10-30T12:01:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6d037fe640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6cf00392a0 0x7f6cf003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6d037fe640 10 monclient: get_auth_request con 0x7f6cf00392a0 auth_method 0 2023-10-30T12:01:06.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6cebfff640 10 client.4425.objecter ms_handle_connect 0x7f6cf00392a0 2023-10-30T12:01:06.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.469+0000 7f6d037fe640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6cf00392a0 0x7f6cf003b760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4000f60 tx=0x7f6cf4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:06.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.473+0000 7f6cebfff640 1 -- 172.21.15.50:0/3479389854 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6cf80396d0 con 0x7f6d04132db0 2023-10-30T12:01:06.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.473+0000 7f6cebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:06.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.473+0000 7f6cebfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:06.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.718+0000 7f6d09ccf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:06.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.718+0000 7f6d09ccf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:06.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.718+0000 7f6d09ccf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:06.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.718+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6d0404e160 con 0x7f6d04132db0 2023-10-30T12:01:06.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.719+0000 7f6cebfff640 1 -- 172.21.15.50:0/3479389854 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f6cf8039310 con 0x7f6d04132db0 2023-10-30T12:01:06.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.719+0000 7f6cebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:06.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.719+0000 7f6cebfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:06.722 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:06.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 10 librados: watch_flush enter 2023-10-30T12:01:06.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 10 librados: watch_flush exit 2023-10-30T12:01:06.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 20 client.4425.objecter shutdown clearing up homeless session... 2023-10-30T12:01:06.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 10 client.4425.objecter successfully canceled tick 2023-10-30T12:01:06.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6cf00392a0 msgr2=0x7f6cf003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:06.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6cf00392a0 0x7f6cf003b760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4000f60 tx=0x7f6cf4009040 comp rx=0 tx=0).stop 2023-10-30T12:01:06.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 10 monclient: shutdown 2023-10-30T12:01:06.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:06.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d04132db0 msgr2=0x7f6d04136d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:06.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d04132db0 0x7f6d04136d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8034040 tx=0x7f6cf80046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 shutdown_connections 2023-10-30T12:01:06.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6cf00392a0 0x7f6cf003b760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d041372d0 0x7f6d041ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d04132db0 0x7f6d04136d90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 --2- 172.21.15.50:0/3479389854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d040a4b60 0x7f6d04136850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:06.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 >> 172.21.15.50:0/3479389854 conn(0x7f6d040a8060 msgr2=0x7f6d040a5750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:06.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 shutdown_connections 2023-10-30T12:01:06.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 -- 172.21.15.50:0/3479389854 wait complete. 2023-10-30T12:01:06.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:06.721+0000 7f6d09ccf640 1 librados: shutdown 2023-10-30T12:01:09.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:09.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.850+0000 7fc0967e9640 10 monclient: get_monmap_and_config 2023-10-30T12:01:09.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.850+0000 7fc0967e9640 10 monclient: build_initial_monmap 2023-10-30T12:01:09.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.850+0000 7fc0967e9640 10 monclient: monmap: 2023-10-30T12:01:09.853 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:09.853 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:09.853 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:09.851627+0000 2023-10-30T12:01:09.853 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:09.851627+0000 2023-10-30T12:01:09.853 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:09.854 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:09.854 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:09.854 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:09.854 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:09.855 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:09.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 1 Processor -- start 2023-10-30T12:01:09.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 1 -- start start 2023-10-30T12:01:09.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient: init 2023-10-30T12:01:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T12:01:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0900a0080 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient(hunting): picked mon.noname-a con 0x7fc0900720b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 0x7fc09013d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient(hunting): picked mon.noname-b con 0x7fc090073a30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09009ccc0 con 0x7fc0900a0080 2023-10-30T12:01:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc090073620 con 0x7fc0900720b0 2023-10-30T12:01:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc090071e90 con 0x7fc090073a30 2023-10-30T12:01:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc0967e9640 10 monclient(hunting): authenticate will time out at 915.034846s 2023-10-30T12:01:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc094d86640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 msgr2=0x7fc09013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc094d86640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 0x7fc09013d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.851+0000 7fc08f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60432/0 (socket says 172.21.15.50:60432) 2023-10-30T12:01:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 learned_addr learned my addr 172.21.15.50:0/3181597773 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request con 0x7fc0900720b0 auth_method 0 2023-10-30T12:01:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:09.852+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.052+0000 7fc094d86640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 msgr2=0x7fc09013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.052+0000 7fc094d86640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 0x7fc09013d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request con 0x7fc0900720b0 auth_method 0 2023-10-30T12:01:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.053+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:10.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.453+0000 7fc094d86640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 msgr2=0x7fc09013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.453+0000 7fc094d86640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 0x7fc09013d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:01:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request con 0x7fc0900720b0 auth_method 0 2023-10-30T12:01:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:10.454+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:01:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.254+0000 7fc094d86640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 msgr2=0x7fc09013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:11.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.254+0000 7fc094d86640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 0x7fc09013d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:01:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request con 0x7fc0900720b0 auth_method 0 2023-10-30T12:01:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:11.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:11.255+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:01:12.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient(hunting): tick 2023-10-30T12:01:12.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:01:12.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:01:12.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:01:12.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 msgr2=0x7fc09013d280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:12.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc090073a30 0x7fc09013d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 msgr2=0x7fc09013ad30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.50:3300/0 conn(0x7fc0900720b0 0x7fc09013ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 -- 172.21.15.50:0/3181597773 >> v1:172.21.15.118:6789/0 conn(0x7fc0900a0080 legacy=0x7fc0900a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient: _add_conns ranks=[4,0] 2023-10-30T12:01:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient(hunting): picked mon.noname-c con 0x7fc07c0036e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc07c006250 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 -- 172.21.15.50:0/3181597773 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09009c2b0 con 0x7fc07c006250 2023-10-30T12:01:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 1 -- 172.21.15.50:0/3181597773 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc090073620 con 0x7fc07c0036e0 2023-10-30T12:01:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08dffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:12.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:12.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 10 monclient(hunting): get_auth_request con 0x7fc07c0036e0 auth_method 0 2023-10-30T12:01:12.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:12.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:12.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:12.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 msgr2=0x7fc07c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:12.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 msgr2=0x7fc07c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:12.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:12.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:12.852+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:13.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:13.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 10 monclient(hunting): get_auth_request con 0x7fc07c0036e0 auth_method 0 2023-10-30T12:01:13.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:13.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:13.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:13.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 msgr2=0x7fc07c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:13.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 msgr2=0x7fc07c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:13.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:13.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.053+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 10 monclient(hunting): get_auth_request con 0x7fc07c0036e0 auth_method 0 2023-10-30T12:01:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 msgr2=0x7fc07c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 msgr2=0x7fc07c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.454+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:01:13.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.620+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869339806 0 0) 0x7fc09009c2b0 con 0x7fc07c006250 2023-10-30T12:01:13.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.620+0000 7fc08effd640 10 monclient(hunting): _negotiate 2023-10-30T12:01:13.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.620+0000 7fc08effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:13.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.620+0000 7fc08effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:13.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.620+0000 7fc08effd640 10 monclient(hunting): my global_id is 4428 2023-10-30T12:01:13.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.620+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc078003520 con 0x7fc07c006250 2023-10-30T12:01:13.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.620+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2718044079 0 0) 0x7fc078003520 con 0x7fc07c006250 2023-10-30T12:01:13.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 msgr2=0x7fc07c005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:13.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 --2- 172.21.15.50:0/3181597773 >> v2:172.21.15.118:3300/0 conn(0x7fc07c0036e0 0x7fc07c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: _finish_hunting 0 2023-10-30T12:01:13.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:01:13.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:13.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:01:13.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c00afe0 con 0x7fc07c006250 2023-10-30T12:01:13.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: _finish_auth 0 2023-10-30T12:01:13.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: _check_auth_tickets 2023-10-30T12:01:13.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:13.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fc08400c570 con 0x7fc07c006250 2023-10-30T12:01:13.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:13.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:01:13.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: dump: 2023-10-30T12:01:13.629 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:13.629 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:13.630 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.630 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.630 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:13.630 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:13.630 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:13.631 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:13.631 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:13.631 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:13.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:01:13.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:13.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 >> v1:172.21.15.50:6789/0 conn(0x7fc07c006250 legacy=0x7fc07c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:13.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 -- 172.21.15.50:0/3181597773 reap_dead start 2023-10-30T12:01:13.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:01:13.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc078003970 0x7fc078003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:13.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): picked mon.c con 0x7fc078003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:13.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780047c0 0x7fc078004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:13.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): picked mon.b con 0x7fc0780047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:13.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780051d0 0x7fc07800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:13.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): picked mon.a con 0x7fc0780051d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:13.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:13.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090073620 con 0x7fc0780051d0 2023-10-30T12:01:13.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:13.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07800c100 con 0x7fc0780047c0 2023-10-30T12:01:13.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:13.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07800c2c0 con 0x7fc078003970 2023-10-30T12:01:13.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:13.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08effd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:01:13.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08ffff640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc078003970 msgr2=0x7fc078003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:13.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08ffff640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc078003970 0x7fc078003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:13.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780051d0 0x7fc07800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:13.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780047c0 0x7fc078004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:13.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): get_auth_request con 0x7fc0780051d0 auth_method 0 2023-10-30T12:01:13.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:13.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:13.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:13.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request con 0x7fc0780047c0 auth_method 0 2023-10-30T12:01:13.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:13.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:13.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:13.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:13.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:13.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:01:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc08f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:01:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient(hunting): handle_auth_done global_id 4428 payload 293 2023-10-30T12:01:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc078003970 msgr2=0x7fc078003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc078003970 0x7fc078003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780047c0 msgr2=0x7fc078004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780047c0 0x7fc078004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient: _finish_hunting 0 2023-10-30T12:01:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 monclient: found mon.a 2023-10-30T12:01:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:13.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:13.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 1 -- 172.21.15.50:0/3181597773 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc078003520 con 0x7fc0780051d0 2023-10-30T12:01:13.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient: _finish_auth 0 2023-10-30T12:01:13.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 10 monclient: _check_auth_tickets 2023-10-30T12:01:13.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc094d86640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:13.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc0967e9640 5 monclient: authenticate success, global_id 4428 2023-10-30T12:01:13.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.621+0000 7fc0967e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:13.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.622+0000 7fc094d86640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780051d0 0x7fc07800b980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009f30 tx=0x7fc0780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3259b14873a60f67 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:13.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08802cc40 con 0x7fc0780051d0 2023-10-30T12:01:13.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:13.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:13.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 10 monclient: dump: 2023-10-30T12:01:13.642 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:13.642 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:13.642 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.642 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.643 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:13.643 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:13.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:13.643 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:13.643 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:13.643 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:13.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc0967e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:13.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08802cda0 con 0x7fc0780051d0 2023-10-30T12:01:13.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:13.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 1 -- 172.21.15.50:0/3181597773 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088034650 con 0x7fc0780051d0 2023-10-30T12:01:13.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc0967e9640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:13.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc0967e9640 4 set_mon_vals no callback set 2023-10-30T12:01:13.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:13.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:13.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.623+0000 7fc08effd640 10 monclient: dump: 2023-10-30T12:01:13.645 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:13.645 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:13.646 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.646 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.646 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:13.646 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:13.646 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:13.647 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:13.647 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:13.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 10 monclient: shutdown 2023-10-30T12:01:13.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:13.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 -- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780051d0 msgr2=0x7fc07800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:13.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780051d0 0x7fc07800b980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009f30 tx=0x7fc0780057f0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 -- 172.21.15.50:0/3181597773 shutdown_connections 2023-10-30T12:01:13.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0780051d0 0x7fc07800b980 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780047c0 0x7fc078004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 --2- 172.21.15.50:0/3181597773 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc078003970 0x7fc078003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 -- 172.21.15.50:0/3181597773 >> 172.21.15.50:0/3181597773 conn(0x7fc0900a4870 msgr2=0x7fc0900a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:13.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 -- 172.21.15.50:0/3181597773 shutdown_connections 2023-10-30T12:01:13.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.624+0000 7fc0967e9640 1 -- 172.21.15.50:0/3181597773 wait complete. 2023-10-30T12:01:13.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 10 monclient: build_initial_monmap 2023-10-30T12:01:13.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 10 monclient: monmap: 2023-10-30T12:01:13.655 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:13.655 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:13.655 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:13.655 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:13.655 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:13.656 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:13.656 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 1 librados: starting msgr at 2023-10-30T12:01:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 1 librados: starting objecter 2023-10-30T12:01:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 1 Processor -- start 2023-10-30T12:01:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 1 -- start start 2023-10-30T12:01:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 1 librados: setting wanted keys 2023-10-30T12:01:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 1 librados: calling monclient init 2023-10-30T12:01:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.625+0000 7fc0967e9640 10 monclient: init 2023-10-30T12:01:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:01:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc078003970 0x7fc0901c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): picked mon.noname-a con 0x7fc078003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0780047c0 0x7fc0901ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): picked mon.noname-c con 0x7fc0780047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780051d0 0x7fc0901bae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): picked mon.noname-b con 0x7fc0780051d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:13.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09009dcb0 con 0x7fc078003970 2023-10-30T12:01:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901b3500 con 0x7fc0780051d0 2023-10-30T12:01:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901b4080 con 0x7fc0780047c0 2023-10-30T12:01:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc0967e9640 10 monclient(hunting): authenticate will time out at 918.809465s 2023-10-30T12:01:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08f7fe640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0780047c0 msgr2=0x7fc0901ba8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0780047c0 0x7fc0901ba8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc078003970 0x7fc0901c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc078003970 0x7fc0901c9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60466/0 (socket says 172.21.15.50:60466) 2023-10-30T12:01:13.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780051d0 0x7fc0901bae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 1 -- 172.21.15.50:0/1828484712 learned_addr learned my addr 172.21.15.50:0/1828484712 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): get_auth_request con 0x7fc078003970 auth_method 0 2023-10-30T12:01:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:13.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 10 monclient(hunting): get_auth_request con 0x7fc0780051d0 auth_method 0 2023-10-30T12:01:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:13.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc094d86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:13.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 10 monclient(hunting): handle_auth_done global_id 4431 payload 824 2023-10-30T12:01:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 1 -- 172.21.15.50:0/1828484712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0780047c0 msgr2=0x7fc0901ba8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0780047c0 0x7fc0901ba8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 1 -- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780051d0 msgr2=0x7fc0901bae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.626+0000 7fc08ffff640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780051d0 0x7fc0901bae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:01:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08ffff640 1 monclient: found mon.noname-a 2023-10-30T12:01:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08ffff640 1 -- 172.21.15.50:0/1828484712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc090073e90 con 0x7fc078003970 2023-10-30T12:01:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc094d86640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780051d0 0x7fc0901bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08ffff640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc078003970 0x7fc0901c9dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc0800115c0 tx=0x7fc080011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 client.?.objecter ms_handle_connect 0x7fc078003970 2023-10-30T12:01:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 1 -- 172.21.15.50:0/1828484712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08001a070 con 0x7fc078003970 2023-10-30T12:01:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:13.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:13.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: dump: 2023-10-30T12:01:13.668 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:13.668 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:13.668 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.668 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.669 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:13.669 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:13.669 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:13.669 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:13.669 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:13.669 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: _finish_auth 0 2023-10-30T12:01:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:01:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 1 -- 172.21.15.50:0/1828484712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0800053e0 con 0x7fc078003970 2023-10-30T12:01:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 5 monclient: authenticate success, global_id 4431 2023-10-30T12:01:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 1 -- 172.21.15.50:0/1828484712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080006820 con 0x7fc078003970 2023-10-30T12:01:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 10 monclient: _renew_subs 2023-10-30T12:01:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0901b5070 con 0x7fc078003970 2023-10-30T12:01:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:13.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:13.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08cff9640 10 monclient: dump: 2023-10-30T12:01:13.672 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:13.672 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:13.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.673 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:13.673 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:13.673 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:13.673 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:13.673 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:13.673 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:13.674 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:13.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc08e7fc640 4 set_mon_vals no callback set 2023-10-30T12:01:13.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:13.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 10 monclient: _renew_subs 2023-10-30T12:01:13.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:13.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0901ca300 con 0x7fc078003970 2023-10-30T12:01:13.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.627+0000 7fc0967e9640 1 librados: init done 2023-10-30T12:01:13.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc08cff9640 1 -- 172.21.15.50:0/1828484712 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc080006980 con 0x7fc078003970 2023-10-30T12:01:13.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc0967e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:13.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc0967e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:13.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc0967e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:13.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc08cff9640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc07c039b70 0x7fc07c03c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:13.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0900a3cf0 con 0x7fc078003970 2023-10-30T12:01:13.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc08cff9640 1 -- 172.21.15.50:0/1828484712 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fc08004a360 con 0x7fc078003970 2023-10-30T12:01:13.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc08cff9640 10 client.4431.objecter ms_dispatch 0x7fc090000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc08cff9640 3 client.4431.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc08cff9640 3 client.4431.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.628+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc07c039b70 0x7fc07c03c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.629+0000 7fc08f7fe640 10 monclient: get_auth_request con 0x7fc07c039b70 auth_method 0 2023-10-30T12:01:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.629+0000 7fc08cff9640 20 client.4431.objecter dump_active .. 0 homeless 2023-10-30T12:01:13.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.629+0000 7fc08cff9640 10 client.4431.objecter ms_handle_connect 0x7fc07c039b70 2023-10-30T12:01:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.629+0000 7fc08f7fe640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc07c039b70 0x7fc07c03c030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc084001eb0 tx=0x7fc084008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.633+0000 7fc08cff9640 1 -- 172.21.15.50:0/1828484712 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc080006c30 con 0x7fc078003970 2023-10-30T12:01:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.634+0000 7fc08cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:13.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.634+0000 7fc08cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.879+0000 7fc0967e9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.879+0000 7fc0967e9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.879+0000 7fc0967e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.879+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc09004e160 con 0x7fc078003970 2023-10-30T12:01:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.879+0000 7fc08cff9640 1 -- 172.21.15.50:0/1828484712 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7fc0800143d0 con 0x7fc078003970 2023-10-30T12:01:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.879+0000 7fc08cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.879+0000 7fc08cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:13.883 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 10 librados: watch_flush enter 2023-10-30T12:01:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 10 librados: watch_flush exit 2023-10-30T12:01:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 20 client.4431.objecter shutdown clearing up homeless session... 2023-10-30T12:01:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 10 client.4431.objecter successfully canceled tick 2023-10-30T12:01:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc07c039b70 msgr2=0x7fc07c03c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc07c039b70 0x7fc07c03c030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc084001eb0 tx=0x7fc084008040 comp rx=0 tx=0).stop 2023-10-30T12:01:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 10 monclient: shutdown 2023-10-30T12:01:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc078003970 msgr2=0x7fc0901c9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc078003970 0x7fc0901c9dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc0800115c0 tx=0x7fc080011a80 comp rx=0 tx=0).stop 2023-10-30T12:01:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 shutdown_connections 2023-10-30T12:01:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc07c039b70 0x7fc07c03c030 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0780051d0 0x7fc0901bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0780047c0 0x7fc0901ba8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 --2- 172.21.15.50:0/1828484712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc078003970 0x7fc0901c9dc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 >> 172.21.15.50:0/1828484712 conn(0x7fc0900a4870 msgr2=0x7fc090098e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 shutdown_connections 2023-10-30T12:01:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 -- 172.21.15.50:0/1828484712 wait complete. 2023-10-30T12:01:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.881+0000 7fc0967e9640 1 librados: shutdown 2023-10-30T12:01:13.893 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:01:13.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:01:13.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:01:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.998+0000 7f027db8a640 10 monclient: get_monmap_and_config 2023-10-30T12:01:14.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.998+0000 7f027db8a640 10 monclient: build_initial_monmap 2023-10-30T12:01:14.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.998+0000 7f027db8a640 10 monclient: monmap: 2023-10-30T12:01:14.001 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:14.001 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.001 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:14.000017+0000 2023-10-30T12:01:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:14.000017+0000 2023-10-30T12:01:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:14.003 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.999+0000 7f027db8a640 1 Processor -- start 2023-10-30T12:01:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.999+0000 7f027db8a640 1 -- start start 2023-10-30T12:01:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:13.999+0000 7f027db8a640 10 monclient: init 2023-10-30T12:01:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-10-30T12:01:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f027806afc0 0x7f02780a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient(hunting): picked mon.noname-a con 0x7f027806afc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f027806b640 0x7f02780a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient(hunting): picked mon.noname-b con 0x7f027806b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02780a2710 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027806bde0 con 0x7f02780a2710 2023-10-30T12:01:14.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02780a2b40 con 0x7f027806afc0 2023-10-30T12:01:14.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027806b3a0 con 0x7f027806b640 2023-10-30T12:01:14.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:14.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f027db8a640 10 monclient(hunting): authenticate will time out at 919.183232s 2023-10-30T12:01:14.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f0276ffd640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f027806b640 msgr2=0x7f02780a21d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:14.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f0276ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f027806b640 0x7f02780a21d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:14.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f027806afc0 0x7f02780a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f027806afc0 0x7f02780a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60468/0 (socket says 172.21.15.50:60468) 2023-10-30T12:01:14.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 -- 172.21.15.50:0/3024714551 learned_addr learned my addr 172.21.15.50:0/3024714551 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:14.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): get_auth_request con 0x7f027806afc0 auth_method 0 2023-10-30T12:01:14.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient(hunting): handle_auth_done global_id 4434 payload 274 2023-10-30T12:01:14.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 -- 172.21.15.50:0/3024714551 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f027806b640 msgr2=0x7f02780a21d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:14.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 --2- 172.21.15.50:0/3024714551 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f027806b640 0x7f02780a21d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 -- 172.21.15.50:0/3024714551 >> v1:172.21.15.118:6789/0 conn(0x7f02780a2710 legacy=0x7f027813c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient: _finish_hunting 0 2023-10-30T12:01:14.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 monclient: found mon.noname-a 2023-10-30T12:01:14.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:14.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:14.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.000+0000 7f02777fe640 1 -- 172.21.15.50:0/3024714551 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027809e540 con 0x7f027806afc0 2023-10-30T12:01:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02777fe640 1 --2- 172.21.15.50:0/3024714551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806afc0 0x7f02780a1c90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0268009870 tx=0x7f026802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5228c7e31e144b7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 1 -- 172.21.15.50:0/3024714551 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02680317e0 con 0x7f027806afc0 2023-10-30T12:01:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: dump: 2023-10-30T12:01:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.023 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: _finish_auth 0 2023-10-30T12:01:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 1 -- 172.21.15.50:0/3024714551 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0268031940 con 0x7f027806afc0 2023-10-30T12:01:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 5 monclient: authenticate success, global_id 4434 2023-10-30T12:01:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 1 -- 172.21.15.50:0/3024714551 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268031cd0 con 0x7f027806afc0 2023-10-30T12:01:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 4 set_mon_vals no callback set 2023-10-30T12:01:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f02767fc640 10 monclient: dump: 2023-10-30T12:01:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.028 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 10 monclient: shutdown 2023-10-30T12:01:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 1 -- 172.21.15.50:0/3024714551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806afc0 msgr2=0x7f02780a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.001+0000 7f027db8a640 1 --2- 172.21.15.50:0/3024714551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806afc0 0x7f02780a1c90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0268009870 tx=0x7f026802b880 comp rx=0 tx=0).stop 2023-10-30T12:01:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 -- 172.21.15.50:0/3024714551 shutdown_connections 2023-10-30T12:01:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 --2- 172.21.15.50:0/3024714551 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f027806b640 0x7f02780a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 --2- 172.21.15.50:0/3024714551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806afc0 0x7f02780a1c90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 -- 172.21.15.50:0/3024714551 >> 172.21.15.50:0/3024714551 conn(0x7f02780a6200 msgr2=0x7f02780a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 -- 172.21.15.50:0/3024714551 shutdown_connections 2023-10-30T12:01:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 -- 172.21.15.50:0/3024714551 wait complete. 2023-10-30T12:01:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 10 monclient: build_initial_monmap 2023-10-30T12:01:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 10 monclient: monmap: 2023-10-30T12:01:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:14.032 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 librados: starting msgr at 2023-10-30T12:01:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 librados: starting objecter 2023-10-30T12:01:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.002+0000 7f027db8a640 1 Processor -- start 2023-10-30T12:01:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 -- start start 2023-10-30T12:01:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 librados: setting wanted keys 2023-10-30T12:01:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 librados: calling monclient init 2023-10-30T12:01:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient: init 2023-10-30T12:01:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:01:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f027806afc0 0x7f02781bde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): picked mon.noname-b con 0x7f027806afc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806b640 0x7f02781a7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): picked mon.noname-a con 0x7f027806b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02780a2710 0x7f02781a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): picked mon.noname-c con 0x7f02780a2710 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02781a9cc0 con 0x7f027806b640 2023-10-30T12:01:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02781caf00 con 0x7f027806afc0 2023-10-30T12:01:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02781ab150 con 0x7f02780a2710 2023-10-30T12:01:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f027db8a640 10 monclient(hunting): authenticate will time out at 919.186581s 2023-10-30T12:01:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0277fff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02780a2710 msgr2=0x7f02781a7870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0277fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02780a2710 0x7f02781a7870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806b640 0x7f02781a7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806b640 0x7f02781a7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60476/0 (socket says 172.21.15.50:60476) 2023-10-30T12:01:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 -- 172.21.15.50:0/2802791008 learned_addr learned my addr 172.21.15.50:0/2802791008 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): get_auth_request con 0x7f027806b640 auth_method 0 2023-10-30T12:01:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f027806afc0 0x7f02781bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 10 monclient(hunting): get_auth_request con 0x7f027806afc0 auth_method 0 2023-10-30T12:01:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2023-10-30T12:01:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02780a2710 msgr2=0x7f02781a7870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02780a2710 0x7f02781a7870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f027806afc0 msgr2=0x7f02781bde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f027806afc0 0x7f02781bde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:01:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 monclient: found mon.noname-a 2023-10-30T12:01:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.003+0000 7f0276ffd640 1 -- 172.21.15.50:0/2802791008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680094d0 con 0x7f027806b640 2023-10-30T12:01:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f0276ffd640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806b640 0x7f02781a7330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f026c00b550 tx=0x7f026c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 client.?.objecter ms_handle_connect 0x7f027806b640 2023-10-30T12:01:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c003dc0 con 0x7f027806b640 2023-10-30T12:01:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: dump: 2023-10-30T12:01:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.045 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: _finish_auth 0 2023-10-30T12:01:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c005070 con 0x7f027806b640 2023-10-30T12:01:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 5 monclient: authenticate success, global_id 4437 2023-10-30T12:01:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 10 monclient: _renew_subs 2023-10-30T12:01:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02781ab3a0 con 0x7f027806b640 2023-10-30T12:01:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c005420 con 0x7f027806b640 2023-10-30T12:01:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f025ffff640 10 monclient: dump: 2023-10-30T12:01:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.050 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 10 monclient: _renew_subs 2023-10-30T12:01:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f0275ffb640 4 set_mon_vals no callback set 2023-10-30T12:01:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02781a9990 con 0x7f027806b640 2023-10-30T12:01:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.004+0000 7f027db8a640 1 librados: init done 2023-10-30T12:01:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f026c005580 con 0x7f027806b640 2023-10-30T12:01:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f027db8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f027db8a640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a7db0 0x7f02781a8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f027db8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0278070d60 con 0x7f02781a7db0 2023-10-30T12:01:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a7db0 0x7f02781a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient: get_auth_request con 0x7f02781a7db0 auth_method 0 2023-10-30T12:01:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02640392a0 0x7f026403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f026c04a380 con 0x7f027806b640 2023-10-30T12:01:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 10 client.4437.objecter ms_dispatch 0x7f0278000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 3 client.4437.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 3 client.4437.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 20 client.4437.objecter dump_active .. 0 homeless 2023-10-30T12:01:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f0277fff640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02640392a0 0x7f026403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f0277fff640 10 monclient: get_auth_request con 0x7f02640392a0 auth_method 0 2023-10-30T12:01:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2023-10-30T12:01:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 1 -- 172.21.15.50:0/2802791008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f027809df80 con 0x7f02781a7db0 2023-10-30T12:01:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f02777fe640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a7db0 0x7f02781a8190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f02780712a0 tx=0x7f0268004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 10 client.4437.objecter ms_handle_connect 0x7f02781a7db0 2023-10-30T12:01:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 10 client.4437.objecter resend_mon_ops 2023-10-30T12:01:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268032a40 con 0x7f02781a7db0 2023-10-30T12:01:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.005+0000 7f025ffff640 10 monclient: dump: 2023-10-30T12:01:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.059 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 10 client.4437.objecter ms_handle_connect 0x7f02640392a0 2023-10-30T12:01:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f0277fff640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02640392a0 0x7f026403b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f02700013b0 tx=0x7f0270007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0268032ba0 con 0x7f02781a7db0 2023-10-30T12:01:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a7db0 msgr2=0x7f02781a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a7db0 0x7f02781a8190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f02780712a0 tx=0x7f0268004730 comp rx=0 tx=0).stop 2023-10-30T12:01:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a7db0 msgr2=0x7f02781a8190 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.006+0000 7f025ffff640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a7db0 0x7f02781a8190 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f02780712a0 tx=0x7f0268004730 comp rx=0 tx=0).stop 2023-10-30T12:01:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f027db8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f027db8a640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a3bd0 0x7f02781a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f027db8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02781ab530 con 0x7f02781a3bd0 2023-10-30T12:01:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a3bd0 0x7f02781a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient: get_auth_request con 0x7f02781a3bd0 auth_method 0 2023-10-30T12:01:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2023-10-30T12:01:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.012+0000 7f02777fe640 1 -- 172.21.15.50:0/2802791008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f027806a6f0 con 0x7f02781a3bd0 2023-10-30T12:01:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f02777fe640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a3bd0 0x7f02781a3fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f026802b8c0 tx=0x7f02680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 10 client.4437.objecter ms_handle_connect 0x7f02781a3bd0 2023-10-30T12:01:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 10 client.4437.objecter resend_mon_ops 2023-10-30T12:01:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268050660 con 0x7f02781a3bd0 2023-10-30T12:01:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 10 monclient: dump: 2023-10-30T12:01:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.068 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4621 (secure 0 0 0) 0x7f02680507c0 con 0x7f02781a3bd0 2023-10-30T12:01:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a3bd0 msgr2=0x7f02781a3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a3bd0 0x7f02781a3fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f026802b8c0 tx=0x7f02680046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a3bd0 msgr2=0x7f02781a3fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f025ffff640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02781a3bd0 0x7f02781a3fe0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f026802b8c0 tx=0x7f02680046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.070 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:14.070 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:14.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:14.070 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:01:14.070 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 8, 2023-10-30T12:01:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:01:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:01:14.071 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:14.072 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:14.072 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:14.072 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:14.072 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:14.072 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:14.073 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:14.073 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:14.073 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:14.073 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:14.073 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:14.074 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:14.074 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:14.074 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.074 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:14.074 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:14.075 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:14.075 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:14.075 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:14.075 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:14.075 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.077 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:14.077 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:14.077 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:14.077 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:14.078 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:14.078 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:14.078 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:14.078 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:14.078 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:14.079 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:14.079 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:14.079 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:14.079 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:14.079 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:14.080 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:14.080 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:14.080 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:14.080 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:14.080 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:14.081 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:14.081 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:14.081 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:14.081 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:14.081 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:14.082 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:14.082 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:14.082 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:14.082 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.082 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:14.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.083 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:14.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:14.083 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:14.083 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:14.084 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:14.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.084 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:14.084 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:14.084 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.085 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:14.085 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:14.085 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:14.086 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:14.087 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:14.087 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:14.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:14.088 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:14.088 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:14.088 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:14.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.088 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:14.088 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:14.089 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.089 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:14.089 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:14.090 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.090 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.090 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.090 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:14.091 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:14.091 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:14.091 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:14.091 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:14.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:14.092 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:14.092 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:14.092 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:14.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.093 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:14.093 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:14.093 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.094 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:14.094 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:14.094 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.095 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.095 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:14.095 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:14.095 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:14.096 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:14.096 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:14.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.096 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.096 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:14.097 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:14.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.097 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:14.097 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:14.097 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:01:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:01:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:14.101 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:14.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 10 librados: watch_flush enter 2023-10-30T12:01:14.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 10 librados: watch_flush exit 2023-10-30T12:01:14.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 20 client.4437.objecter shutdown clearing up homeless session... 2023-10-30T12:01:14.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 10 client.4437.objecter successfully canceled tick 2023-10-30T12:01:14.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02640392a0 msgr2=0x7f026403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02640392a0 0x7f026403b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f02700013b0 tx=0x7f0270007d30 comp rx=0 tx=0).stop 2023-10-30T12:01:14.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 10 monclient: shutdown 2023-10-30T12:01:14.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:14.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806b640 msgr2=0x7f02781a7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 --2- 172.21.15.50:0/2802791008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f027806b640 0x7f02781a7330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f026c00b550 tx=0x7f026c00ba10 comp rx=0 tx=0).stop 2023-10-30T12:01:14.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f0277fff640 1 -- 172.21.15.50:0/2802791008 reap_dead start 2023-10-30T12:01:14.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f0277fff640 1 -- 172.21.15.50:0/2802791008 reap_dead start 2023-10-30T12:01:14.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 shutdown_connections 2023-10-30T12:01:14.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 >> 172.21.15.50:0/2802791008 conn(0x7f02780a6200 msgr2=0x7f027806e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:14.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 shutdown_connections 2023-10-30T12:01:14.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 -- 172.21.15.50:0/2802791008 wait complete. 2023-10-30T12:01:14.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.013+0000 7f027db8a640 1 librados: shutdown 2023-10-30T12:01:14.107 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:01:14.107 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:01:14.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:01:14.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.212+0000 7fa90c9a0640 10 monclient: get_monmap_and_config 2023-10-30T12:01:14.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.212+0000 7fa90c9a0640 10 monclient: build_initial_monmap 2023-10-30T12:01:14.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.212+0000 7fa90c9a0640 10 monclient: monmap: 2023-10-30T12:01:14.215 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:14.215 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.215 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:14.213328+0000 2023-10-30T12:01:14.215 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:14.213328+0000 2023-10-30T12:01:14.215 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:14.215 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.216 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:14.216 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:14.216 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:14.216 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:14.216 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:14.217 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.212+0000 7fa90c9a0640 1 Processor -- start 2023-10-30T12:01:14.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 1 -- start start 2023-10-30T12:01:14.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient: init 2023-10-30T12:01:14.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:14.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T12:01:14.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa9080a31a0 0x7fa908099f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient(hunting): picked mon.noname-c con 0x7fa9080a31a0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:14.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9080a3580 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:14.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a39d0 0x7fa908135ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa9080a39d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:14.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90809c710 con 0x7fa9080a3580 2023-10-30T12:01:14.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90809c0c0 con 0x7fa9080a31a0 2023-10-30T12:01:14.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90809c3a0 con 0x7fa9080a39d0 2023-10-30T12:01:14.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:14.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa90c9a0640 10 monclient(hunting): authenticate will time out at 919.396573s 2023-10-30T12:01:14.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa907fff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a39d0 msgr2=0x7fa908135ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:14.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa907fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a39d0 0x7fa908135ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:14.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa906ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa9080a3580 0x7fa9081353c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54048/0 (socket says 172.21.15.50:54048) 2023-10-30T12:01:14.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa906ffd640 1 -- 172.21.15.50:0/1907115626 learned_addr learned my addr 172.21.15.50:0/1907115626 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:14.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 1 --2- 172.21.15.50:0/1907115626 >> v2:172.21.15.118:3300/0 conn(0x7fa9080a31a0 0x7fa908099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 10 monclient(hunting): get_auth_request con 0x7fa9080a31a0 auth_method 0 2023-10-30T12:01:14.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.213+0000 7fa9067fc640 1 -- 172.21.15.50:0/1907115626 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103354618 0 0) 0x7fa90809c710 con 0x7fa9080a3580 2023-10-30T12:01:14.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9067fc640 10 monclient(hunting): _negotiate 2023-10-30T12:01:14.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9067fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9067fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9067fc640 10 monclient(hunting): my global_id is 4446 2023-10-30T12:01:14.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9067fc640 1 -- 172.21.15.50:0/1907115626 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f0003520 con 0x7fa9080a3580 2023-10-30T12:01:14.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 10 monclient(hunting): handle_auth_done global_id 4438 payload 274 2023-10-30T12:01:14.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 1 -- 172.21.15.50:0/1907115626 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a39d0 msgr2=0x7fa908135ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:14.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 1 --2- 172.21.15.50:0/1907115626 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a39d0 0x7fa908135ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 1 -- 172.21.15.50:0/1907115626 >> v1:172.21.15.50:6789/0 conn(0x7fa9080a3580 legacy=0x7fa9081353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 10 monclient: _finish_hunting 0 2023-10-30T12:01:14.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 1 monclient: found mon.noname-c 2023-10-30T12:01:14.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:14.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:01:14.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 1 -- 172.21.15.50:0/1907115626 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9080a83f0 con 0x7fa9080a31a0 2023-10-30T12:01:14.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.214+0000 7fa9077fe640 1 --2- 172.21.15.50:0/1907115626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 0x7fa908099f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009790 tx=0x7fa8f802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe4916f66817a5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 1 -- 172.21.15.50:0/1907115626 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f802e070 con 0x7fa9080a31a0 2023-10-30T12:01:14.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:14.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: dump: 2023-10-30T12:01:14.235 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.235 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.235 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.236 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.236 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.236 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.236 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.236 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.236 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.237 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: _finish_auth 0 2023-10-30T12:01:14.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:14.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:14.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 1 -- 172.21.15.50:0/1907115626 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f80315a0 con 0x7fa9080a31a0 2023-10-30T12:01:14.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:14.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 5 monclient: authenticate success, global_id 4438 2023-10-30T12:01:14.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:14.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 4 set_mon_vals no callback set 2023-10-30T12:01:14.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 1 -- 172.21.15.50:0/1907115626 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8031910 con 0x7fa9080a31a0 2023-10-30T12:01:14.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:14.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa9067fc640 10 monclient: dump: 2023-10-30T12:01:14.239 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.239 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.240 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.240 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.240 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.240 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.240 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.241 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.241 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.241 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 10 monclient: shutdown 2023-10-30T12:01:14.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:14.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 -- 172.21.15.50:0/1907115626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 msgr2=0x7fa908099f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 --2- 172.21.15.50:0/1907115626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 0x7fa908099f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8009790 tx=0x7fa8f802b840 comp rx=0 tx=0).stop 2023-10-30T12:01:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 -- 172.21.15.50:0/1907115626 shutdown_connections 2023-10-30T12:01:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 --2- 172.21.15.50:0/1907115626 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a39d0 0x7fa908135ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 --2- 172.21.15.50:0/1907115626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 0x7fa908099f20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 -- 172.21.15.50:0/1907115626 >> 172.21.15.50:0/1907115626 conn(0x7fa9080a7d70 msgr2=0x7fa9080a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 -- 172.21.15.50:0/1907115626 shutdown_connections 2023-10-30T12:01:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.215+0000 7fa90c9a0640 1 -- 172.21.15.50:0/1907115626 wait complete. 2023-10-30T12:01:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 10 monclient: build_initial_monmap 2023-10-30T12:01:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 10 monclient: monmap: 2023-10-30T12:01:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:14.245 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 1 librados: starting msgr at 2023-10-30T12:01:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 1 librados: starting objecter 2023-10-30T12:01:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 1 Processor -- start 2023-10-30T12:01:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 1 -- start start 2023-10-30T12:01:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 1 librados: setting wanted keys 2023-10-30T12:01:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 1 librados: calling monclient init 2023-10-30T12:01:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.216+0000 7fa90c9a0640 10 monclient: init 2023-10-30T12:01:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:01:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 0x7fa9081ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa9080a31a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a3580 0x7fa908131410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): picked mon.noname-c con 0x7fa9080a3580 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9080a39d0 0x7fa908131950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): picked mon.noname-a con 0x7fa9080a39d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90811e550 con 0x7fa9080a39d0 2023-10-30T12:01:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908131e90 con 0x7fa9080a31a0 2023-10-30T12:01:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90809b3d0 con 0x7fa9080a3580 2023-10-30T12:01:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient(hunting): authenticate will time out at 919.400177s 2023-10-30T12:01:14.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa906ffd640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a3580 msgr2=0x7fa908131410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:01:14.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa906ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a3580 0x7fa908131410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:14.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9080a39d0 0x7fa908131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9080a39d0 0x7fa908131950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60512/0 (socket says 172.21.15.50:60512) 2023-10-30T12:01:14.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 0x7fa9081ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 -- 172.21.15.50:0/3498594712 learned_addr learned my addr 172.21.15.50:0/3498594712 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): get_auth_request con 0x7fa9080a39d0 auth_method 0 2023-10-30T12:01:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 10 monclient(hunting): get_auth_request con 0x7fa9080a31a0 auth_method 0 2023-10-30T12:01:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa9077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient(hunting): handle_auth_done global_id 4449 payload 824 2023-10-30T12:01:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a3580 msgr2=0x7fa908131410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9080a3580 0x7fa908131410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 msgr2=0x7fa9081ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a31a0 0x7fa9081ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient: _finish_hunting 0 2023-10-30T12:01:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 monclient: found mon.noname-a 2023-10-30T12:01:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 -- 172.21.15.50:0/3498594712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f80093a0 con 0x7fa9080a39d0 2023-10-30T12:01:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa907fff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9080a39d0 0x7fa908131950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa90000be50 tx=0x7fa90000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 client.?.objecter ms_handle_connect 0x7fa9080a39d0 2023-10-30T12:01:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90000ce90 con 0x7fa9080a39d0 2023-10-30T12:01:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 monclient: dump: 2023-10-30T12:01:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.258 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 monclient: _finish_auth 0 2023-10-30T12:01:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 5 monclient: authenticate success, global_id 4449 2023-10-30T12:01:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient: _renew_subs 2023-10-30T12:01:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa900016430 con 0x7fa9080a39d0 2023-10-30T12:01:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa908132250 con 0x7fa9080a39d0 2023-10-30T12:01:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.217+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000167e0 con 0x7fa9080a39d0 2023-10-30T12:01:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa8effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa8effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa8effff640 10 monclient: dump: 2023-10-30T12:01:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.262 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 10 monclient: _renew_subs 2023-10-30T12:01:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90811edf0 con 0x7fa9080a39d0 2023-10-30T12:01:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa905ffb640 4 set_mon_vals no callback set 2023-10-30T12:01:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 1 librados: init done 2023-10-30T12:01:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa900012020 con 0x7fa9080a39d0 2023-10-30T12:01:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa8effff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f40392d0 0x7fa8f403b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa90809ff20 0x7fa9080a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90811e8f0 con 0x7fa90809ff20 2023-10-30T12:01:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.218+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fa90004a720 con 0x7fa9080a39d0 2023-10-30T12:01:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 client.4449.objecter ms_dispatch 0x7fa908000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 3 client.4449.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 3 client.4449.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 20 client.4449.objecter dump_active .. 0 homeless 2023-10-30T12:01:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa90809ff20 0x7fa9080a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa9077fe640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f40392d0 0x7fa8f403b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa9077fe640 10 monclient: get_auth_request con 0x7fa8f40392d0 auth_method 0 2023-10-30T12:01:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient: get_auth_request con 0x7fa90809ff20 auth_method 0 2023-10-30T12:01:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 10 monclient(hunting): handle_auth_done global_id 4444 payload 824 2023-10-30T12:01:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 1 -- 172.21.15.50:0/3498594712 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa90809aca0 con 0x7fa90809ff20 2023-10-30T12:01:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa906ffd640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa90809ff20 0x7fa9080a0300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa90811dea0 tx=0x7fa8fc00c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 client.4449.objecter ms_handle_connect 0x7fa90809ff20 2023-10-30T12:01:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 client.4449.objecter resend_mon_ops 2023-10-30T12:01:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 client.4449.objecter ms_handle_connect 0x7fa8f40392d0 2023-10-30T12:01:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa9077fe640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f40392d0 0x7fa8f403b790 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8036000 tx=0x7fa8f803e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc005ce0 con 0x7fa90809ff20 2023-10-30T12:01:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 monclient: dump: 2023-10-30T12:01:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.272 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa8fc005e40 con 0x7fa90809ff20 2023-10-30T12:01:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa90809ff20 msgr2=0x7fa9080a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.219+0000 7fa8effff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa90809ff20 0x7fa9080a0300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa90811dea0 tx=0x7fa8fc00c8e0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.220+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa90809ff20 msgr2=0x7fa9080a0300 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.220+0000 7fa8effff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa90809ff20 0x7fa9080a0300 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa90811dea0 tx=0x7fa8fc00c8e0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa90c9a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:14.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa90c9a0640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a0840 0x7fa908134490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa90c9a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908073dc0 con 0x7fa9080a0840 2023-10-30T12:01:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a0840 0x7fa908134490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient: get_auth_request con 0x7fa9080a0840 auth_method 0 2023-10-30T12:01:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 10 monclient(hunting): handle_auth_done global_id 4447 payload 824 2023-10-30T12:01:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 1 -- 172.21.15.50:0/3498594712 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa90804e160 con 0x7fa9080a0840 2023-10-30T12:01:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa906ffd640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a0840 0x7fa908134490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0011e0 tx=0x7fa8fc000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa8effff640 10 client.4449.objecter ms_handle_connect 0x7fa9080a0840 2023-10-30T12:01:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa8effff640 10 client.4449.objecter resend_mon_ops 2023-10-30T12:01:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc012cc0 con 0x7fa9080a0840 2023-10-30T12:01:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa8effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa8effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.226+0000 7fa8effff640 10 monclient: dump: 2023-10-30T12:01:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:14.281 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4789 (secure 0 0 0) 0x7fa8fc01d450 con 0x7fa9080a0840 2023-10-30T12:01:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa8effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa8effff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a0840 msgr2=0x7fa908134490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa8effff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a0840 0x7fa908134490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0011e0 tx=0x7fa8fc000ed0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa8effff640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a0840 msgr2=0x7fa908134490 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa8effff640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9080a0840 0x7fa908134490 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0011e0 tx=0x7fa8fc000ed0 comp rx=0 tx=0).stop 2023-10-30T12:01:14.283 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:14.283 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:14.283 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:14.283 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:01:14.284 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 8, 2023-10-30T12:01:14.284 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:14.284 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:14.284 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:01:14.284 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.285 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:01:14.285 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:14.285 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:14.285 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:14.285 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:14.286 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:14.286 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:14.286 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:14.286 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:14.286 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:14.286 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:14.287 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:14.287 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:14.287 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:14.287 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.287 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:14.288 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:14.288 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:14.288 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:14.288 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:14.288 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:14.289 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:14.289 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:14.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:14.289 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:14.289 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:14.290 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:14.290 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.290 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:14.290 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:14.291 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:14.292 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:14.292 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:14.292 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:14.292 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:14.292 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:14.293 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:14.293 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:14.293 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:14.293 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:14.293 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:14.294 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:14.294 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:14.294 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:14.294 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:14.294 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:14.295 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:14.295 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:14.295 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:14.295 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:14.295 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.295 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:14.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.296 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:14.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.296 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:14.296 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:14.297 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:14.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:14.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.297 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:14.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:14.298 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.299 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.299 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.299 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:14.299 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:14.300 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:14.300 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:14.300 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:14.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.300 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:14.301 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:14.301 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:14.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:14.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:14.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:14.302 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:14.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:14.303 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.303 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.303 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:14.304 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:14.304 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:14.304 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:14.304 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:14.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.305 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:14.305 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:14.305 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:14.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:14.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.306 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:14.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:14.306 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:14.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:14.307 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:14.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:14.308 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:14.308 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:14.308 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:14.309 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:14.309 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.309 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.309 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:14.309 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:14.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.310 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:14.310 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:14.310 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:14.310 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.311 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.311 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:14.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.311 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:14.311 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:14.311 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:01:14.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.312 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.312 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:14.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.312 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:14.312 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:14.313 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:01:14.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.313 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:14.313 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:01:14.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:14.315 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 10 librados: watch_flush enter 2023-10-30T12:01:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 10 librados: watch_flush exit 2023-10-30T12:01:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 20 client.4449.objecter shutdown clearing up homeless session... 2023-10-30T12:01:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 10 client.4449.objecter successfully canceled tick 2023-10-30T12:01:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f40392d0 msgr2=0x7fa8f403b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f40392d0 0x7fa8f403b790 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8036000 tx=0x7fa8f803e040 comp rx=0 tx=0).stop 2023-10-30T12:01:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 10 monclient: shutdown 2023-10-30T12:01:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9080a39d0 msgr2=0x7fa908131950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 --2- 172.21.15.50:0/3498594712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9080a39d0 0x7fa908131950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa90000be50 tx=0x7fa90000be80 comp rx=0 tx=0).stop 2023-10-30T12:01:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa907fff640 1 -- 172.21.15.50:0/3498594712 reap_dead start 2023-10-30T12:01:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa907fff640 1 -- 172.21.15.50:0/3498594712 reap_dead start 2023-10-30T12:01:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 shutdown_connections 2023-10-30T12:01:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 >> 172.21.15.50:0/3498594712 conn(0x7fa9080a7d70 msgr2=0x7fa9081bd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 shutdown_connections 2023-10-30T12:01:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 -- 172.21.15.50:0/3498594712 wait complete. 2023-10-30T12:01:14.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:14.227+0000 7fa90c9a0640 1 librados: shutdown 2023-10-30T12:01:14.320 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:01:14.320 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:01:34.321 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:01:34.321 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-10-30T12:01:34.322 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:01:34.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:01:34.325 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:01:34.325 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:01:34.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:34.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.440+0000 7f4e5497c640 10 monclient: get_monmap_and_config 2023-10-30T12:01:34.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.440+0000 7f4e5497c640 10 monclient: build_initial_monmap 2023-10-30T12:01:34.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.440+0000 7f4e5497c640 10 monclient: monmap: 2023-10-30T12:01:34.442 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:34.443 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.443 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:34.441217+0000 2023-10-30T12:01:34.443 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:34.441217+0000 2023-10-30T12:01:34.443 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:34.443 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.444 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:34.444 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:34.444 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:34.444 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:34.444 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:34.444 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.440+0000 7f4e5497c640 1 Processor -- start 2023-10-30T12:01:34.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.440+0000 7f4e5497c640 1 -- start start 2023-10-30T12:01:34.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.440+0000 7f4e5497c640 10 monclient: init 2023-10-30T12:01:34.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:34.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-10-30T12:01:34.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 0x7f4e501326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): picked mon.noname-b con 0x7f4e500a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:34.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e50099a40 0x7f4e50132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): picked mon.noname-a con 0x7f4e50099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:34.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4e500a2350 0x7f4e500a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): picked mon.noname-c con 0x7f4e500a2350 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:34.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e500a3d70 con 0x7f4e50099a40 2023-10-30T12:01:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e50133120 con 0x7f4e500a2350 2023-10-30T12:01:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500a4f40 con 0x7f4e500a4b60 2023-10-30T12:01:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e5497c640 10 monclient(hunting): authenticate will time out at 939.624337s 2023-10-30T12:01:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 0x7f4e501326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e50099a40 0x7f4e50132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e50099a40 0x7f4e50132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58514/0 (socket says 172.21.15.50:58514) 2023-10-30T12:01:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 0x7f4e501326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46350/0 (socket says 172.21.15.50:46350) 2023-10-30T12:01:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 1 -- 172.21.15.50:0/3158780941 learned_addr learned my addr 172.21.15.50:0/3158780941 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 10 monclient(hunting): get_auth_request con 0x7f4e500a4b60 auth_method 0 2023-10-30T12:01:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 1 --2- 172.21.15.50:0/3158780941 >> v2:172.21.15.118:3300/0 conn(0x7f4e500a2350 0x7f4e500a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): get_auth_request con 0x7f4e50099a40 auth_method 0 2023-10-30T12:01:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 10 monclient(hunting): get_auth_request con 0x7f4e500a2350 auth_method 0 2023-10-30T12:01:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 1 -- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 msgr2=0x7f4e501326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 1 -- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 msgr2=0x7f4e501326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 1 --2- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 0x7f4e501326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4e59c640 1 --2- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 0x7f4e501326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient(hunting): handle_auth_done global_id 4452 payload 274 2023-10-30T12:01:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 -- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 msgr2=0x7f4e501326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 --2- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 0x7f4e501326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 -- 172.21.15.50:0/3158780941 >> v2:172.21.15.118:3300/0 conn(0x7f4e500a2350 msgr2=0x7f4e500a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 --2- 172.21.15.50:0/3158780941 >> v2:172.21.15.118:3300/0 conn(0x7f4e500a2350 0x7f4e500a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:01:34.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 monclient: found mon.noname-a 2023-10-30T12:01:34.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:34.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e45d9b640 1 -- 172.21.15.50:0/3158780941 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e500a4150 con 0x7f4e50099a40 2023-10-30T12:01:34.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.441+0000 7f4e4ed9d640 1 --2- 172.21.15.50:0/3158780941 >> v2:172.21.15.118:3300/0 conn(0x7f4e500a2350 0x7f4e500a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:34.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e45d9b640 1 --2- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e50099a40 0x7f4e50132be0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e500a53f0 tx=0x7f4e3802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7b1e6e1c2a6ad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 1 -- 172.21.15.50:0/3158780941 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38030710 con 0x7f4e50099a40 2023-10-30T12:01:34.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:34.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: dump: 2023-10-30T12:01:34.463 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.465 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.465 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.465 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: _finish_auth 0 2023-10-30T12:01:34.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:01:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 1 -- 172.21.15.50:0/3158780941 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3802e050 con 0x7f4e50099a40 2023-10-30T12:01:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 1 -- 172.21.15.50:0/3158780941 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38030a90 con 0x7f4e50099a40 2023-10-30T12:01:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e5497c640 5 monclient: authenticate success, global_id 4452 2023-10-30T12:01:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e5497c640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e5497c640 4 set_mon_vals no callback set 2023-10-30T12:01:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.442+0000 7f4e4dd9b640 10 monclient: dump: 2023-10-30T12:01:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.469 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 10 monclient: shutdown 2023-10-30T12:01:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 -- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e50099a40 msgr2=0x7f4e50132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 --2- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e50099a40 0x7f4e50132be0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e500a53f0 tx=0x7f4e3802b710 comp rx=0 tx=0).stop 2023-10-30T12:01:34.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 -- 172.21.15.50:0/3158780941 shutdown_connections 2023-10-30T12:01:34.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 --2- 172.21.15.50:0/3158780941 >> v2:172.21.15.118:3300/0 conn(0x7f4e500a2350 0x7f4e500a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 --2- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e50099a40 0x7f4e50132be0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 --2- 172.21.15.50:0/3158780941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e500a4b60 0x7f4e501326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 -- 172.21.15.50:0/3158780941 >> 172.21.15.50:0/3158780941 conn(0x7f4e500a8060 msgr2=0x7f4e500a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:34.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 -- 172.21.15.50:0/3158780941 shutdown_connections 2023-10-30T12:01:34.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.443+0000 7f4e5497c640 1 -- 172.21.15.50:0/3158780941 wait complete. 2023-10-30T12:01:34.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 10 monclient: build_initial_monmap 2023-10-30T12:01:34.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 10 monclient: monmap: 2023-10-30T12:01:34.471 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:34.472 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:34.472 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:34.472 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:34.472 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:34.472 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.472 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:34.473 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:34.473 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:34.473 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 1 librados: starting msgr at 2023-10-30T12:01:34.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 1 librados: starting objecter 2023-10-30T12:01:34.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 1 Processor -- start 2023-10-30T12:01:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 1 -- start start 2023-10-30T12:01:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 1 librados: setting wanted keys 2023-10-30T12:01:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 1 librados: calling monclient init 2023-10-30T12:01:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.444+0000 7f4e5497c640 10 monclient: init 2023-10-30T12:01:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:01:34.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500a2350 0x7f4e501ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): picked mon.noname-a con 0x7f4e500a2350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:34.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e500a4b60 0x7f4e501cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): picked mon.noname-b con 0x7f4e500a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:34.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 0x7f4e5011ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5011e870 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e50121020 con 0x7f4e500a2350 2023-10-30T12:01:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500a18d0 con 0x7f4e500a4b60 2023-10-30T12:01:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e50135c10 con 0x7f4e5011e870 2023-10-30T12:01:34.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:34.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e5497c640 10 monclient(hunting): authenticate will time out at 939.628192s 2023-10-30T12:01:34.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500a2350 0x7f4e501ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 0x7f4e5011ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500a2350 0x7f4e501ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58524/0 (socket says 172.21.15.50:58524) 2023-10-30T12:01:34.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 0x7f4e5011ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46360/0 (socket says 172.21.15.50:46360) 2023-10-30T12:01:34.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 -- 172.21.15.50:0/716734287 learned_addr learned my addr 172.21.15.50:0/716734287 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:34.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): get_auth_request con 0x7f4e500a2350 auth_method 0 2023-10-30T12:01:34.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e500a4b60 0x7f4e501cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 10 monclient(hunting): get_auth_request con 0x7f4e5011e870 auth_method 0 2023-10-30T12:01:34.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 10 monclient(hunting): get_auth_request con 0x7f4e500a4b60 auth_method 0 2023-10-30T12:01:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 1 -- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 msgr2=0x7f4e5011ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 1 -- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 msgr2=0x7f4e5011ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 0x7f4e5011ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4ed9d640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 0x7f4e5011ed20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2023-10-30T12:01:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 -- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 msgr2=0x7f4e5011ed20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 0x7f4e5011ed20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 -- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e500a4b60 msgr2=0x7f4e501cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e500a4b60 0x7f4e501cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:01:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 monclient: found mon.noname-a 2023-10-30T12:01:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 -- 172.21.15.50:0/716734287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e48009a70 con 0x7f4e500a2350 2023-10-30T12:01:34.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e45d9b640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e500a4b60 0x7f4e501cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:34.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e4e59c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500a2350 0x7f4e501ca8c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000e810 tx=0x7f4e4000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e477fe640 10 client.?.objecter ms_handle_connect 0x7f4e500a2350 2023-10-30T12:01:34.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.445+0000 7f4e477fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:34.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 1 -- 172.21.15.50:0/716734287 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e40004020 con 0x7f4e500a2350 2023-10-30T12:01:34.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:34.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: dump: 2023-10-30T12:01:34.485 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.485 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.485 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.486 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.486 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.486 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.486 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.486 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.486 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.487 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: _finish_auth 0 2023-10-30T12:01:34.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: _check_auth_tickets 2023-10-30T12:01:34.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:34.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 1 -- 172.21.15.50:0/716734287 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e40004180 con 0x7f4e500a2350 2023-10-30T12:01:34.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 5 monclient: authenticate success, global_id 4455 2023-10-30T12:01:34.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 10 monclient: _renew_subs 2023-10-30T12:01:34.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e38009400 con 0x7f4e500a2350 2023-10-30T12:01:34.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:34.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 1 -- 172.21.15.50:0/716734287 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000c6d0 con 0x7f4e500a2350 2023-10-30T12:01:34.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:34.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 10 monclient: dump: 2023-10-30T12:01:34.489 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.489 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.489 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.490 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.490 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.490 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.490 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.490 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.490 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.491 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:34.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 10 monclient: _renew_subs 2023-10-30T12:01:34.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e50121f10 con 0x7f4e500a2350 2023-10-30T12:01:34.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e5497c640 1 librados: init done 2023-10-30T12:01:34.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e4d59a640 4 set_mon_vals no callback set 2023-10-30T12:01:34.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.446+0000 7f4e477fe640 1 -- 172.21.15.50:0/716734287 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4e4000c830 con 0x7f4e500a2350 2023-10-30T12:01:34.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e5497c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:34.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e5497c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:34.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e5497c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e477fe640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e300392f0 0x7f4e3003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5004e160 con 0x7f4e500a2350 2023-10-30T12:01:34.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e477fe640 1 -- 172.21.15.50:0/716734287 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f4e4004a080 con 0x7f4e500a2350 2023-10-30T12:01:34.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e477fe640 10 client.4455.objecter ms_dispatch 0x7f4e50000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:34.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e477fe640 3 client.4455.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:34.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e477fe640 3 client.4455.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:34.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e45d9b640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e300392f0 0x7f4e3003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e477fe640 20 client.4455.objecter dump_active .. 0 homeless 2023-10-30T12:01:34.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e45d9b640 10 monclient: get_auth_request con 0x7f4e300392f0 auth_method 0 2023-10-30T12:01:34.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e45d9b640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e300392f0 0x7f4e3003b7b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e38034000 tx=0x7f4e3802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.447+0000 7f4e477fe640 10 client.4455.objecter ms_handle_connect 0x7f4e300392f0 2023-10-30T12:01:34.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.451+0000 7f4e477fe640 1 -- 172.21.15.50:0/716734287 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4e40003610 con 0x7f4e500a2350 2023-10-30T12:01:34.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.451+0000 7f4e477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:34.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.451+0000 7f4e477fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:34.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.698+0000 7f4e5497c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:34.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.698+0000 7f4e5497c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:34.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.698+0000 7f4e5497c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.698+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4e50122100 con 0x7f4e500a2350 2023-10-30T12:01:34.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.699+0000 7f4e477fe640 1 -- 172.21.15.50:0/716734287 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f4e50122100 con 0x7f4e500a2350 2023-10-30T12:01:34.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.699+0000 7f4e477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:34.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.699+0000 7f4e477fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:34.702 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:34.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 10 librados: watch_flush enter 2023-10-30T12:01:34.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 10 librados: watch_flush exit 2023-10-30T12:01:34.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 20 client.4455.objecter shutdown clearing up homeless session... 2023-10-30T12:01:34.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 10 client.4455.objecter successfully canceled tick 2023-10-30T12:01:34.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e300392f0 msgr2=0x7f4e3003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e300392f0 0x7f4e3003b7b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e38034000 tx=0x7f4e3802f000 comp rx=0 tx=0).stop 2023-10-30T12:01:34.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 10 monclient: shutdown 2023-10-30T12:01:34.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:34.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500a2350 msgr2=0x7f4e501ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500a2350 0x7f4e501ca8c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000e810 tx=0x7f4e4000ecd0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 shutdown_connections 2023-10-30T12:01:34.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e300392f0 0x7f4e3003b7b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5011e870 0x7f4e5011ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e500a4b60 0x7f4e501cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 --2- 172.21.15.50:0/716734287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500a2350 0x7f4e501ca8c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 >> 172.21.15.50:0/716734287 conn(0x7f4e500a8060 msgr2=0x7f4e50134540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:34.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 shutdown_connections 2023-10-30T12:01:34.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 -- 172.21.15.50:0/716734287 wait complete. 2023-10-30T12:01:34.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.701+0000 7f4e5497c640 1 librados: shutdown 2023-10-30T12:01:34.714 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:01:34.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:01:34.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:01:34.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.818+0000 7fd2d49d0640 10 monclient: get_monmap_and_config 2023-10-30T12:01:34.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.818+0000 7fd2d49d0640 10 monclient: build_initial_monmap 2023-10-30T12:01:34.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.818+0000 7fd2d49d0640 10 monclient: monmap: 2023-10-30T12:01:34.821 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:34.821 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.821 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:34.819648+0000 2023-10-30T12:01:34.821 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:34.819648+0000 2023-10-30T12:01:34.822 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:34.822 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.822 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:34.822 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:34.822 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:34.823 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:34.823 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:34.823 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 1 Processor -- start 2023-10-30T12:01:34.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 1 -- start start 2023-10-30T12:01:34.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient: init 2023-10-30T12:01:34.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:34.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-10-30T12:01:34.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2d0096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d0096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:34.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2d0096e60 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:34.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2d009b170 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:34.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d009bd30 con 0x7fd2d009b170 2023-10-30T12:01:34.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d009c400 con 0x7fd2d0096e60 2023-10-30T12:01:34.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d009c120 con 0x7fd2d0096540 2023-10-30T12:01:34.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:34.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2d49d0640 10 monclient(hunting): authenticate will time out at 940.002801s 2023-10-30T12:01:34.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2ced9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd2d009b170 0x7fd2d009b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49756/0 (socket says 172.21.15.50:49756) 2023-10-30T12:01:34.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2ce59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2d0096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2ced9d640 1 -- 172.21.15.50:0/3454057150 learned_addr learned my addr 172.21.15.50:0/3454057150 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:34.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request con 0x7fd2d0096540 auth_method 0 2023-10-30T12:01:34.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.819+0000 7fd2ce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 1 -- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 msgr2=0x7fd2d0096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 1 -- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 msgr2=0x7fd2d0096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2d0096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2d0096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:34.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262311690 0 0) 0x7fd2d009bd30 con 0x7fd2d009b170 2023-10-30T12:01:34.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _negotiate 2023-10-30T12:01:34.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): my global_id is 4458 2023-10-30T12:01:34.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc003520 con 0x7fd2d009b170 2023-10-30T12:01:34.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874328870 0 0) 0x7fd2d009c400 con 0x7fd2d0096e60 2023-10-30T12:01:34.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _negotiate 2023-10-30T12:01:34.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): my global_id is 4456 2023-10-30T12:01:34.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d009bd30 con 0x7fd2d0096e60 2023-10-30T12:01:34.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3081282965 0 0) 0x7fd2bc003520 con 0x7fd2d009b170 2023-10-30T12:01:34.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 msgr2=0x7fd2d0096920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2d0096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 >> v1:172.21.15.118:6789/0 conn(0x7fd2d0096e60 legacy=0x7fd2d009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:01:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:01:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:01:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d009f600 con 0x7fd2d009b170 2023-10-30T12:01:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: _finish_auth 0 2023-10-30T12:01:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:01:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2d49d0640 5 monclient: authenticate success, global_id 4458 2023-10-30T12:01:34.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fd2c80019f0 con 0x7fd2d009b170 2023-10-30T12:01:34.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2d49d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:34.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:01:34.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: dump: 2023-10-30T12:01:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.851 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:01:34.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:34.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 >> v1:172.21.15.50:6789/0 conn(0x7fd2d009b170 legacy=0x7fd2d009b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:01:34.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2bc003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): picked mon.c con 0x7fd2d0096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:34.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2bc004090 0x7fd2bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): picked mon.a con 0x7fd2bc004090 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:34.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2bc004d00 0x7fd2bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): picked mon.b con 0x7fd2bc004d00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:34.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d009c120 con 0x7fd2bc004090 2023-10-30T12:01:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00bb10 con 0x7fd2bc004d00 2023-10-30T12:01:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00bcd0 con 0x7fd2d0096540 2023-10-30T12:01:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2cdd9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:01:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2d49d0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2bc003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request con 0x7fd2d0096540 auth_method 0 2023-10-30T12:01:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2bc004090 0x7fd2bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 10 monclient(hunting): get_auth_request con 0x7fd2bc004090 auth_method 0 2023-10-30T12:01:34.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:34.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:01:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2bc004d00 0x7fd2bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:01:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 10 monclient(hunting): get_auth_request con 0x7fd2bc004d00 auth_method 0 2023-10-30T12:01:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:01:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.820+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_done global_id 4458 payload 293 2023-10-30T12:01:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 1 -- 172.21.15.50:0/3454057150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2bc004d00 msgr2=0x7fd2bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2bc004d00 0x7fd2bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 1 -- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2bc004090 msgr2=0x7fd2bc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:34.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2bc004090 0x7fd2bc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 10 monclient: _finish_hunting 0 2023-10-30T12:01:34.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 1 monclient: found mon.c 2023-10-30T12:01:34.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 1 -- 172.21.15.50:0/3454057150 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc003520 con 0x7fd2d0096540 2023-10-30T12:01:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 10 monclient: _finish_auth 0 2023-10-30T12:01:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 10 monclient: _check_auth_tickets 2023-10-30T12:01:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2c7fff640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2bc004090 0x7fd2bc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2bc003b30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000b9e0 tx=0x7fd2bc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5706604f6b5da57 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0037070 con 0x7fd2d0096540 2023-10-30T12:01:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:34.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 10 monclient: dump: 2023-10-30T12:01:34.864 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.864 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.864 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.864 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.865 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.865 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.865 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.865 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.865 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.866 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c0004020 con 0x7fd2d0096540 2023-10-30T12:01:34.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:34.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 1 -- 172.21.15.50:0/3454057150 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c00043d0 con 0x7fd2d0096540 2023-10-30T12:01:34.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2d49d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:34.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2d49d0640 4 set_mon_vals no callback set 2023-10-30T12:01:34.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:34.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.821+0000 7fd2cdd9b640 10 monclient: dump: 2023-10-30T12:01:34.867 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.867 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.869 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.869 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.869 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 10 monclient: shutdown 2023-10-30T12:01:34.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 1 -- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 msgr2=0x7fd2bc003b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 1 --2- 172.21.15.50:0/3454057150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0096540 0x7fd2bc003b30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000b9e0 tx=0x7fd2bc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2ced9d640 1 -- 172.21.15.50:0/3454057150 reap_dead start 2023-10-30T12:01:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 1 -- 172.21.15.50:0/3454057150 shutdown_connections 2023-10-30T12:01:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 1 -- 172.21.15.50:0/3454057150 >> 172.21.15.50:0/3454057150 conn(0x7fd2d00a3980 msgr2=0x7fd2d00a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 1 -- 172.21.15.50:0/3454057150 shutdown_connections 2023-10-30T12:01:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.822+0000 7fd2d49d0640 1 -- 172.21.15.50:0/3454057150 wait complete. 2023-10-30T12:01:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient: build_initial_monmap 2023-10-30T12:01:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient: monmap: 2023-10-30T12:01:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:34.872 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:34.872 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:34.872 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:34.872 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:34.872 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.873 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:34.873 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:34.873 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:34.873 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 librados: starting msgr at 2023-10-30T12:01:34.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 librados: starting objecter 2023-10-30T12:01:34.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 Processor -- start 2023-10-30T12:01:34.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 -- start start 2023-10-30T12:01:34.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 librados: setting wanted keys 2023-10-30T12:01:34.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 librados: calling monclient init 2023-10-30T12:01:34.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient: init 2023-10-30T12:01:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:01:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2d0096540 0x7fd2d0097560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d0096540 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01b6ee0 0x7fd2d0097aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d01b6ee0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 0x7fd2d01ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d0097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d0098af0 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01cf7c0 con 0x7fd2d0096540 2023-10-30T12:01:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01b8260 con 0x7fd2d0097fe0 2023-10-30T12:01:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.823+0000 7fd2d49d0640 10 monclient(hunting): authenticate will time out at 940.006998s 2023-10-30T12:01:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 0x7fd2d01ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01b6ee0 0x7fd2d0097aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2d0096540 0x7fd2d0097560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01b6ee0 0x7fd2d0097aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58554/0 (socket says 172.21.15.50:58554) 2023-10-30T12:01:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 -- 172.21.15.50:0/1155736289 learned_addr learned my addr 172.21.15.50:0/1155736289 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 0x7fd2d01ce6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46390/0 (socket says 172.21.15.50:46390) 2023-10-30T12:01:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): get_auth_request con 0x7fd2d01b6ee0 auth_method 0 2023-10-30T12:01:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request con 0x7fd2d0096540 auth_method 0 2023-10-30T12:01:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 10 monclient(hunting): get_auth_request con 0x7fd2d0097fe0 auth_method 0 2023-10-30T12:01:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 msgr2=0x7fd2d01ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 msgr2=0x7fd2d01ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 0x7fd2d01ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2ced9d640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 0x7fd2d01ce6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2023-10-30T12:01:34.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 msgr2=0x7fd2d01ce6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:34.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd2d0097fe0 0x7fd2d01ce6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2d0096540 msgr2=0x7fd2d0097560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd2d0096540 0x7fd2d0097560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:34.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:01:34.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 monclient: found mon.noname-a 2023-10-30T12:01:34.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:34.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 -- 172.21.15.50:0/1155736289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d01b6220 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c7fff640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01b6ee0 0x7fd2d0097aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8007850 tx=0x7fd2b800fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2d01b6ee0 2023-10-30T12:01:34.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:34.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b80052e0 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:34.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: dump: 2023-10-30T12:01:34.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.887 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.888 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.888 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:01:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:01:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 5 monclient: authenticate success, global_id 4461 2023-10-30T12:01:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b8005440 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 10 monclient: _renew_subs 2023-10-30T12:01:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d01b6500 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:34.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b800d930 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:34.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2c6ffd640 10 monclient: dump: 2023-10-30T12:01:34.891 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.891 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.892 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.892 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.892 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.892 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.892 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.893 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.893 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.893 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2cd59a640 4 set_mon_vals no callback set 2023-10-30T12:01:34.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:34.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 10 monclient: _renew_subs 2023-10-30T12:01:34.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:34.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d0098f00 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.824+0000 7fd2d49d0640 1 librados: init done 2023-10-30T12:01:34.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2d49d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:34.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd2b800da90 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2d49d0640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01cec30 0x7fd2d01cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2d49d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d0073ff0 con 0x7fd2d01cec30 2023-10-30T12:01:34.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01cec30 0x7fd2d01cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient: get_auth_request con 0x7fd2d01cec30 auth_method 0 2023-10-30T12:01:34.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2b0039320 0x7fd2b003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fd2b804a6d0 con 0x7fd2d01b6ee0 2023-10-30T12:01:34.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2c6ffd640 10 client.4461.objecter ms_dispatch 0x7fd2d0000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:34.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2c6ffd640 3 client.4461.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:34.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2c6ffd640 3 client.4461.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:34.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2023-10-30T12:01:34.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ce59c640 1 -- 172.21.15.50:0/1155736289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2d00999e0 con 0x7fd2d01cec30 2023-10-30T12:01:34.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2c6ffd640 20 client.4461.objecter dump_active .. 0 homeless 2023-10-30T12:01:34.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ced9d640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2b0039320 0x7fd2b003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.825+0000 7fd2ced9d640 10 monclient: get_auth_request con 0x7fd2b0039320 auth_method 0 2023-10-30T12:01:34.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 client.4461.objecter ms_handle_connect 0x7fd2d01cec30 2023-10-30T12:01:34.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 client.4461.objecter resend_mon_ops 2023-10-30T12:01:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01cec30 0x7fd2d01cf010 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006f70 tx=0x7fd2c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0037040 con 0x7fd2d01cec30 2023-10-30T12:01:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 monclient: dump: 2023-10-30T12:01:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.903 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2ced9d640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2b0039320 0x7fd2b003b7e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd2d009dc00 tx=0x7fd2c8002180 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 client.4461.objecter ms_handle_connect 0x7fd2b0039320 2023-10-30T12:01:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fd2c0041070 con 0x7fd2d01cec30 2023-10-30T12:01:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01cec30 msgr2=0x7fd2d01cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01cec30 0x7fd2d01cf010 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006f70 tx=0x7fd2c0004730 comp rx=0 tx=0).stop 2023-10-30T12:01:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01cec30 msgr2=0x7fd2d01cf010 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.826+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01cec30 0x7fd2d01cf010 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006f70 tx=0x7fd2c0004730 comp rx=0 tx=0).stop 2023-10-30T12:01:34.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2d49d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:34.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2d49d0640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d006dbb0 0x7fd2d006dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:34.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2d49d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:34.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01b63b0 con 0x7fd2d006dbb0 2023-10-30T12:01:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d006dbb0 0x7fd2d006dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 10 monclient: get_auth_request con 0x7fd2d006dbb0 auth_method 0 2023-10-30T12:01:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:34.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:34.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:34.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:34.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.832+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:34.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2ce59c640 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2023-10-30T12:01:34.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2ce59c640 1 -- 172.21.15.50:0/1155736289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd2d01b6740 con 0x7fd2d006dbb0 2023-10-30T12:01:34.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2ce59c640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d006dbb0 0x7fd2d006dfc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd2c002be50 tx=0x7fd2c0033000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:34.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 10 client.4461.objecter ms_handle_connect 0x7fd2d006dbb0 2023-10-30T12:01:34.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 10 client.4461.objecter resend_mon_ops 2023-10-30T12:01:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0033670 con 0x7fd2d006dbb0 2023-10-30T12:01:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 10 monclient: dump: 2023-10-30T12:01:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:34.911 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fd2c00337d0 con 0x7fd2d006dbb0 2023-10-30T12:01:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d006dbb0 msgr2=0x7fd2d006dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d006dbb0 0x7fd2d006dfc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd2c002be50 tx=0x7fd2c0033000 comp rx=0 tx=0).stop 2023-10-30T12:01:34.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d006dbb0 msgr2=0x7fd2d006dfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:34.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2c6ffd640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d006dbb0 0x7fd2d006dfc0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd2c002be50 tx=0x7fd2c0033000 comp rx=0 tx=0).stop 2023-10-30T12:01:34.913 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:34.913 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:34.914 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:34.914 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:01:34.914 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 10, 2023-10-30T12:01:34.914 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:34.914 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:34.915 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:34.915 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:34.915 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:34.915 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:01:34.915 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:34.916 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:34.916 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:34.916 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:34.916 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:34.916 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:34.917 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:34.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:34.917 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:34.917 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:34.918 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:34.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:34.918 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:34.918 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:34.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:34.919 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:34.919 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:34.919 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:34.919 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:34.919 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:34.919 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:34.920 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:34.920 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:34.920 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:34.920 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:34.920 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:34.921 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:34.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.921 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:34.921 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:34.922 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:34.922 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:34.922 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:34.922 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:34.922 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:34.923 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:34.923 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:34.923 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:34.923 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:34.924 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:34.924 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:34.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:34.924 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:34.924 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:34.925 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:34.925 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:34.925 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:34.925 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:34.925 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:34.926 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:34.927 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:34.927 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:34.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.927 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:34.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.928 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:34.928 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:34.928 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:34.928 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:34.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:34.929 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:34.929 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:34.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:34.930 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:34.930 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:34.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:34.931 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:34.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.931 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:34.931 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:34.932 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:34.932 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:34.932 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:34.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.933 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:34.933 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:34.933 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:34.933 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:34.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.934 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:34.934 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:34.934 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:34.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:34.936 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:34.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.936 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:34.936 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:34.936 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:34.937 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:34.937 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:34.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.938 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:34.938 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:34.938 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:34.938 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:34.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:34.939 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:34.939 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:34.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.940 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:34.940 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:34.940 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:34.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:34.941 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:34.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.941 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:34.941 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:34.941 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:34.942 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:34.942 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:34.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:34.942 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:34.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.943 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:34.943 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:34.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.943 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.945 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:34.945 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:34.945 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:01:34.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:34.945 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:34.946 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:34.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:34.946 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:34.946 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:34.947 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:01:34.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:34.947 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:34.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:34.947 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:34.947 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:34.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 10 librados: watch_flush enter 2023-10-30T12:01:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 10 librados: watch_flush exit 2023-10-30T12:01:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 20 client.4461.objecter shutdown clearing up homeless session... 2023-10-30T12:01:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 10 client.4461.objecter successfully canceled tick 2023-10-30T12:01:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2b0039320 msgr2=0x7fd2b003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd2b0039320 0x7fd2b003b7e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd2d009dc00 tx=0x7fd2c8002180 comp rx=0 tx=0).stop 2023-10-30T12:01:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 10 monclient: shutdown 2023-10-30T12:01:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01b6ee0 msgr2=0x7fd2d0097aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 1 --2- 172.21.15.50:0/1155736289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2d01b6ee0 0x7fd2d0097aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8007850 tx=0x7fd2b800fd20 comp rx=0 tx=0).stop 2023-10-30T12:01:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2ced9d640 1 -- 172.21.15.50:0/1155736289 reap_dead start 2023-10-30T12:01:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2ced9d640 1 -- 172.21.15.50:0/1155736289 reap_dead start 2023-10-30T12:01:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 shutdown_connections 2023-10-30T12:01:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.833+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 >> 172.21.15.50:0/1155736289 conn(0x7fd2d00a3980 msgr2=0x7fd2d01b9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.834+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 shutdown_connections 2023-10-30T12:01:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.834+0000 7fd2d49d0640 1 -- 172.21.15.50:0/1155736289 wait complete. 2023-10-30T12:01:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:34.834+0000 7fd2d49d0640 1 librados: shutdown 2023-10-30T12:01:34.953 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:01:34.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:01:34.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:01:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.058+0000 7efedd190640 10 monclient: get_monmap_and_config 2023-10-30T12:01:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.058+0000 7efedd190640 10 monclient: build_initial_monmap 2023-10-30T12:01:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.058+0000 7efedd190640 10 monclient: monmap: 2023-10-30T12:01:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:35.059519+0000 2023-10-30T12:01:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:35.059519+0000 2023-10-30T12:01:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:35.063 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 1 Processor -- start 2023-10-30T12:01:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 1 -- start start 2023-10-30T12:01:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient: init 2023-10-30T12:01:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T12:01:35.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efed80a2ae0 0x7efed80a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:35.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient(hunting): picked mon.noname-a con 0x7efed80a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:35.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efed80a2ec0 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:35.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efed8135df0 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:35.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed80a4190 con 0x7efed8135df0 2023-10-30T12:01:35.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed809a2e0 con 0x7efed80a2ec0 2023-10-30T12:01:35.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:35.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed80a2990 con 0x7efed80a2ae0 2023-10-30T12:01:35.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:35.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efedd190640 10 monclient(hunting): authenticate will time out at 940.242579s 2023-10-30T12:01:35.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efed80a2ae0 0x7efed80a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed759e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efed8135df0 0x7efed813da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49760/0 (socket says 172.21.15.50:49760) 2023-10-30T12:01:35.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed759e640 1 -- 172.21.15.50:0/4223083350 learned_addr learned my addr 172.21.15.50:0/4223083350 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:35.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): get_auth_request con 0x7efed80a2ae0 auth_method 0 2023-10-30T12:01:35.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:35.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:35.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 10 monclient(hunting): handle_auth_done global_id 4470 payload 274 2023-10-30T12:01:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 1 -- 172.21.15.50:0/4223083350 >> v1:172.21.15.118:6789/0 conn(0x7efed80a2ec0 legacy=0x7efed81356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.059+0000 7efed6d9d640 1 -- 172.21.15.50:0/4223083350 >> v1:172.21.15.50:6789/0 conn(0x7efed8135df0 legacy=0x7efed813da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:01:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed6d9d640 1 monclient: found mon.noname-a 2023-10-30T12:01:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed6d9d640 1 -- 172.21.15.50:0/4223083350 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed809a8a0 con 0x7efed80a2ae0 2023-10-30T12:01:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 1 -- 172.21.15.50:0/4223083350 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691090954 0 0) 0x7efed0001490 con 0x7efed8135df0 2023-10-30T12:01:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:01:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed6d9d640 1 --2- 172.21.15.50:0/4223083350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed80a2450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efed809a470 tx=0x7efec802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd06a8facc68bcc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 1 -- 172.21.15.50:0/4223083350 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec802e070 con 0x7efed80a2ae0 2023-10-30T12:01:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: dump: 2023-10-30T12:01:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.078 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:35.078 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:35.078 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:35.078 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:35.078 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:35.078 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:35.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:01:35.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:01:35.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:35.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 1 -- 172.21.15.50:0/4223083350 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec80315a0 con 0x7efed80a2ae0 2023-10-30T12:01:35.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efedd190640 5 monclient: authenticate success, global_id 4470 2023-10-30T12:01:35.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efedd190640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:35.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:35.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efedd190640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:35.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efedd190640 4 set_mon_vals no callback set 2023-10-30T12:01:35.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 1 -- 172.21.15.50:0/4223083350 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8031930 con 0x7efed80a2ae0 2023-10-30T12:01:35.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:35.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:35.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.060+0000 7efed5d9b640 10 monclient: dump: 2023-10-30T12:01:35.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:35.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:35.081 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.081 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:35.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:35.082 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:35.082 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:35.083 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:35.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:35.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 10 monclient: shutdown 2023-10-30T12:01:35.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:35.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 1 -- 172.21.15.50:0/4223083350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 msgr2=0x7efed80a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:35.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 1 --2- 172.21.15.50:0/4223083350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed80a2450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efed809a470 tx=0x7efec802b840 comp rx=0 tx=0).stop 2023-10-30T12:01:35.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 1 -- 172.21.15.50:0/4223083350 shutdown_connections 2023-10-30T12:01:35.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 1 --2- 172.21.15.50:0/4223083350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed80a2450 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:35.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 1 -- 172.21.15.50:0/4223083350 >> 172.21.15.50:0/4223083350 conn(0x7efed80a80b0 msgr2=0x7efed80a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:35.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 1 -- 172.21.15.50:0/4223083350 shutdown_connections 2023-10-30T12:01:35.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.061+0000 7efedd190640 1 -- 172.21.15.50:0/4223083350 wait complete. 2023-10-30T12:01:35.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.062+0000 7efedd190640 10 monclient: build_initial_monmap 2023-10-30T12:01:35.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.062+0000 7efedd190640 10 monclient: monmap: 2023-10-30T12:01:35.085 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:35.085 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:35.085 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:35.086 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:35.086 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:35.086 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:35.086 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:35.086 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:35.087 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:35.087 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:35.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.062+0000 7efedd190640 1 librados: starting msgr at 2023-10-30T12:01:35.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.062+0000 7efedd190640 1 librados: starting objecter 2023-10-30T12:01:35.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.062+0000 7efedd190640 1 Processor -- start 2023-10-30T12:01:35.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 -- start start 2023-10-30T12:01:35.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 librados: setting wanted keys 2023-10-30T12:01:35.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 librados: calling monclient init 2023-10-30T12:01:35.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient: init 2023-10-30T12:01:35.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:01:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed8132860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): picked mon.noname-a con 0x7efed80a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80a2ec0 0x7efed811e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): picked mon.noname-b con 0x7efed80a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 0x7efed811e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): picked mon.noname-c con 0x7efed8135df0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:35.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:35.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed81cad70 con 0x7efed80a2ae0 2023-10-30T12:01:35.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:35.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed81210b0 con 0x7efed80a2ec0 2023-10-30T12:01:35.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:35.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed80a1790 con 0x7efed8135df0 2023-10-30T12:01:35.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:35.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efedd190640 10 monclient(hunting): authenticate will time out at 940.246483s 2023-10-30T12:01:35.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed8132860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 0x7efed811e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed8132860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58594/0 (socket says 172.21.15.50:58594) 2023-10-30T12:01:35.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 1 -- 172.21.15.50:0/4134003980 learned_addr learned my addr 172.21.15.50:0/4134003980 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:35.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 10 monclient(hunting): get_auth_request con 0x7efed80a2ae0 auth_method 0 2023-10-30T12:01:35.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:35.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:35.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:35.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 10 monclient(hunting): get_auth_request con 0x7efed8135df0 auth_method 0 2023-10-30T12:01:35.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80a2ec0 0x7efed811e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:35.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:35.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:35.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 10 monclient(hunting): get_auth_request con 0x7efed80a2ec0 auth_method 0 2023-10-30T12:01:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 msgr2=0x7efed811e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 msgr2=0x7efed811e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 0x7efed811e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.063+0000 7efed759e640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 0x7efed811e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 10 monclient(hunting): handle_auth_done global_id 4476 payload 824 2023-10-30T12:01:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 msgr2=0x7efed811e5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:35.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8135df0 0x7efed811e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80a2ec0 msgr2=0x7efed811e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80a2ec0 0x7efed811e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:01:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 1 monclient: found mon.noname-a 2023-10-30T12:01:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:35.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec8009400 con 0x7efed80a2ae0 2023-10-30T12:01:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efed80a2ec0 0x7efed811e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed6d9d640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed8132860 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efec8034040 tx=0x7efec80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 client.?.objecter ms_handle_connect 0x7efed80a2ae0 2023-10-30T12:01:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8004230 con 0x7efed80a2ae0 2023-10-30T12:01:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: dump: 2023-10-30T12:01:35.099 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:35.100 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:35.101 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:35.101 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:35.101 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:35.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:01:35.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:01:35.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:35.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 5 monclient: authenticate success, global_id 4476 2023-10-30T12:01:35.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec8004390 con 0x7efed80a2ae0 2023-10-30T12:01:35.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:35.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 10 monclient: _renew_subs 2023-10-30T12:01:35.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8032920 con 0x7efed80a2ae0 2023-10-30T12:01:35.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:35.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed811f5a0 con 0x7efed80a2ae0 2023-10-30T12:01:35.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:35.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:35.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efebf7fe640 10 monclient: dump: 2023-10-30T12:01:35.103 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:35.104 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:35.104 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.104 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:35.104 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:35.104 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:35.105 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:35.105 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:35.105 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:35.105 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:35.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efed559a640 4 set_mon_vals no callback set 2023-10-30T12:01:35.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:35.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 10 monclient: _renew_subs 2023-10-30T12:01:35.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:35.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed80a09c0 con 0x7efed80a2ae0 2023-10-30T12:01:35.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.064+0000 7efedd190640 1 librados: init done 2023-10-30T12:01:35.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efedd190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:35.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efedd190640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:35.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efedd190640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:35.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed8073ff0 con 0x7efed8132da0 2023-10-30T12:01:35.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efec8032a80 con 0x7efed80a2ae0 2023-10-30T12:01:35.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 10 monclient: get_auth_request con 0x7efed8132da0 auth_method 0 2023-10-30T12:01:35.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:35.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:35.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:35.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:01:35.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:35.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:35.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efebf7fe640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efec40392f0 0x7efec403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:35.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:35.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7efec806c770 con 0x7efed80a2ae0 2023-10-30T12:01:35.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efebf7fe640 10 client.4476.objecter ms_dispatch 0x7efed8000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:35.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efebf7fe640 3 client.4476.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:35.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efebf7fe640 3 client.4476.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:35.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efebf7fe640 20 client.4476.objecter dump_active .. 0 homeless 2023-10-30T12:01:35.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed759e640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efec40392f0 0x7efec403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.065+0000 7efed759e640 10 monclient: get_auth_request con 0x7efec40392f0 auth_method 0 2023-10-30T12:01:35.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.066+0000 7efebf7fe640 10 client.4476.objecter ms_handle_connect 0x7efec40392f0 2023-10-30T12:01:35.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.066+0000 7efed759e640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efec40392f0 0x7efec403b7b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efed00012e0 tx=0x7efed0007d00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:35.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 10 monclient: get_auth_request con 0x7efed8132da0 auth_method 0 2023-10-30T12:01:35.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:35.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.266+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:35.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:35.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 10 monclient: get_auth_request con 0x7efed8132da0 auth_method 0 2023-10-30T12:01:35.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:35.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:35.667+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:01:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 10 monclient: get_auth_request con 0x7efed8132da0 auth_method 0 2023-10-30T12:01:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:36.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:36.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:36.468+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:01:38.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.063+0000 7efebe7fc640 10 monclient: tick 2023-10-30T12:01:38.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.063+0000 7efebe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:38.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.063+0000 7efebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:38.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.063+0000 7efebe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:38.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:38.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 10 monclient: get_auth_request con 0x7efed8132da0 auth_method 0 2023-10-30T12:01:38.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:38.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:38.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:38.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:38.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:38.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:38.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:38.070+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-30T12:01:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 10 monclient: tick 2023-10-30T12:01:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 5 monclient: _check_tell_commands timeout tell command 1 2023-10-30T12:01:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 msgr2=0x7efed811eb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:39.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed8132da0 0x7efed811eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efebe7fc640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb00060b0 con 0x7efeb0003540 2023-10-30T12:01:39.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 10 monclient: get_auth_request con 0x7efeb0003540 auth_method 0 2023-10-30T12:01:39.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 msgr2=0x7efeb0005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:39.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 msgr2=0x7efeb0005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:39.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.064+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:39.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 10 monclient: get_auth_request con 0x7efeb0003540 auth_method 0 2023-10-30T12:01:39.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:39.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 msgr2=0x7efeb0005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:01:39.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 msgr2=0x7efeb0005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:39.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:39.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.265+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient: get_auth_request con 0x7efeb0003540 auth_method 0 2023-10-30T12:01:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2023-10-30T12:01:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efeb0006220 con 0x7efeb0003540 2023-10-30T12:01:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efed80739b0 tx=0x7efecc003d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efebf7fe640 10 client.4476.objecter ms_handle_connect 0x7efeb0003540 2023-10-30T12:01:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efebf7fe640 10 client.4476.objecter resend_mon_ops 2023-10-30T12:01:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc01c070 con 0x7efeb0003540 2023-10-30T12:01:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.666+0000 7efebf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 10 monclient: dump: 2023-10-30T12:01:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:39.674 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7efecc017b90 con 0x7efeb0003540 2023-10-30T12:01:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 msgr2=0x7efeb0005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efed80739b0 tx=0x7efecc003d20 comp rx=0 tx=0).stop 2023-10-30T12:01:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 msgr2=0x7efeb0005a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.667+0000 7efebf7fe640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efeb0003540 0x7efeb0005a00 secure :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efed80739b0 tx=0x7efecc003d20 comp rx=0 tx=0).stop 2023-10-30T12:01:39.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efedd190640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:39.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efedd190640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed81ca310 0x7efed809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efedd190640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed8099b40 con 0x7efed81ca310 2023-10-30T12:01:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed81ca310 0x7efed809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient: get_auth_request con 0x7efed81ca310 auth_method 0 2023-10-30T12:01:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2023-10-30T12:01:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.674+0000 7efed659c640 1 -- 172.21.15.50:0/4134003980 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efed806b060 con 0x7efed81ca310 2023-10-30T12:01:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efed659c640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed81ca310 0x7efed809cd60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efecc01f000 tx=0x7efecc000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 10 client.4476.objecter ms_handle_connect 0x7efed81ca310 2023-10-30T12:01:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 10 client.4476.objecter resend_mon_ops 2023-10-30T12:01:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc017e90 con 0x7efed81ca310 2023-10-30T12:01:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 10 monclient: dump: 2023-10-30T12:01:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:39.683 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7efecc015a20 con 0x7efed81ca310 2023-10-30T12:01:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed81ca310 msgr2=0x7efed809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed81ca310 0x7efed809cd60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efecc01f000 tx=0x7efecc000ed0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed81ca310 msgr2=0x7efed809cd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efebf7fe640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efed81ca310 0x7efed809cd60 secure :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efecc01f000 tx=0x7efecc000ed0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.685 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:01:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 10, 2023-10-30T12:01:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:01:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.690 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:39.690 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:39.690 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:39.690 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:39.690 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:39.691 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:39.691 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:39.691 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:39.691 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:39.691 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:39.691 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:39.692 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:39.692 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.692 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:39.692 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:39.693 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:39.693 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:39.693 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:39.693 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:39.693 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:39.694 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:39.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:39.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:39.694 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:39.694 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:39.695 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:39.695 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:39.695 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:39.695 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:39.695 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:39.696 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:39.696 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:39.696 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:39.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:39.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:39.696 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:39.697 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:39.697 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:39.697 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:39.697 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:39.697 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.698 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:39.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.698 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:39.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.698 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:39.699 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:39.699 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:39.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:39.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:39.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:39.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:39.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:39.701 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.702 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:39.702 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:39.702 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:39.702 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:39.702 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:39.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.703 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:39.703 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:39.703 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:39.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:39.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:39.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:39.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:39.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:39.705 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:39.715 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:39.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efed759e640 1 -- 172.21.15.50:0/4134003980 reap_dead start 2023-10-30T12:01:39.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efedd190640 10 librados: watch_flush enter 2023-10-30T12:01:39.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efedd190640 10 librados: watch_flush exit 2023-10-30T12:01:39.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efedd190640 20 client.4476.objecter shutdown clearing up homeless session... 2023-10-30T12:01:39.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.675+0000 7efedd190640 10 client.4476.objecter successfully canceled tick 2023-10-30T12:01:39.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efec40392f0 msgr2=0x7efec403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efec40392f0 0x7efec403b7b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efed00012e0 tx=0x7efed0007d00 comp rx=0 tx=0).stop 2023-10-30T12:01:39.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 10 monclient: shutdown 2023-10-30T12:01:39.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:39.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 msgr2=0x7efed8132860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed8132860 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efec8034040 tx=0x7efec80046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 shutdown_connections 2023-10-30T12:01:39.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efec40392f0 0x7efec403b7b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 --2- 172.21.15.50:0/4134003980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed80a2ae0 0x7efed8132860 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 >> 172.21.15.50:0/4134003980 conn(0x7efed80a80b0 msgr2=0x7efed8135380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:39.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 shutdown_connections 2023-10-30T12:01:39.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 -- 172.21.15.50:0/4134003980 wait complete. 2023-10-30T12:01:39.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.676+0000 7efedd190640 1 librados: shutdown 2023-10-30T12:01:39.721 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:01:39.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:01:39.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:01:39.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.837+0000 7fe288e1f640 10 monclient: get_monmap_and_config 2023-10-30T12:01:39.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.837+0000 7fe288e1f640 10 monclient: build_initial_monmap 2023-10-30T12:01:39.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.837+0000 7fe288e1f640 10 monclient: monmap: 2023-10-30T12:01:39.839 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:39.839 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.840 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:39.838232+0000 2023-10-30T12:01:39.840 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:39.838232+0000 2023-10-30T12:01:39.840 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:39.840 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.840 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:39.841 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:39.841 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:39.841 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:39.841 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:39.841 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.837+0000 7fe288e1f640 1 Processor -- start 2023-10-30T12:01:39.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.837+0000 7fe288e1f640 1 -- start start 2023-10-30T12:01:39.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.837+0000 7fe288e1f640 10 monclient: init 2023-10-30T12:01:39.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:39.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T12:01:39.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe284133400 0x7fe2841337e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): picked mon.noname-c con 0x7fe284133400 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:39.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe284133d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe28409fcc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:39.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe28412ffe0 0x7fe284130470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): picked mon.noname-a con 0x7fe28412ffe0 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:39.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2840717f0 con 0x7fe28412ffe0 2023-10-30T12:01:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe284070090 con 0x7fe284133400 2023-10-30T12:01:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28406e6c0 con 0x7fe28409fcc0 2023-10-30T12:01:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe288e1f640 10 monclient(hunting): authenticate will time out at 945.021482s 2023-10-30T12:01:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe28412ffe0 0x7fe284130470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe284133d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe28412ffe0 0x7fe284130470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56524/0 (socket says 172.21.15.50:56524) 2023-10-30T12:01:39.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe284133d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41140/0 (socket says 172.21.15.50:41140) 2023-10-30T12:01:39.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 1 -- 172.21.15.50:0/3982682304 learned_addr learned my addr 172.21.15.50:0/3982682304 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:39.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 10 monclient(hunting): get_auth_request con 0x7fe28412ffe0 auth_method 0 2023-10-30T12:01:39.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): get_auth_request con 0x7fe28409fcc0 auth_method 0 2023-10-30T12:01:39.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 1 --2- 172.21.15.50:0/3982682304 >> v2:172.21.15.118:3300/0 conn(0x7fe284133400 0x7fe2841337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 10 monclient(hunting): get_auth_request con 0x7fe284133400 auth_method 0 2023-10-30T12:01:39.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient(hunting): handle_auth_done global_id 34106 payload 274 2023-10-30T12:01:39.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 -- 172.21.15.50:0/3982682304 >> v2:172.21.15.118:3300/0 conn(0x7fe284133400 msgr2=0x7fe2841337e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:39.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 --2- 172.21.15.50:0/3982682304 >> v2:172.21.15.118:3300/0 conn(0x7fe284133400 0x7fe2841337e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 -- 172.21.15.50:0/3982682304 >> v2:172.21.15.50:3300/0 conn(0x7fe28412ffe0 msgr2=0x7fe284130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:39.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 --2- 172.21.15.50:0/3982682304 >> v2:172.21.15.50:3300/0 conn(0x7fe28412ffe0 0x7fe284130470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:01:39.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 monclient: found mon.noname-b 2023-10-30T12:01:39.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:39.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:01:39.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe281d9b640 1 -- 172.21.15.50:0/3982682304 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe284134260 con 0x7fe28409fcc0 2023-10-30T12:01:39.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.838+0000 7fe282d9d640 1 --2- 172.21.15.50:0/3982682304 >> v2:172.21.15.50:3300/0 conn(0x7fe28412ffe0 0x7fe284130470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:39.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe281d9b640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe284133d20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe28406ead0 tx=0x7fe27802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6e687cfd6257aed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:39.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 1 -- 172.21.15.50:0/3982682304 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278030710 con 0x7fe28409fcc0 2023-10-30T12:01:39.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:39.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:39.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: dump: 2023-10-30T12:01:39.863 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:39.863 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.864 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.864 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.864 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:39.864 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.864 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:39.865 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: _finish_auth 0 2023-10-30T12:01:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: _check_auth_tickets 2023-10-30T12:01:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe288e1f640 5 monclient: authenticate success, global_id 34106 2023-10-30T12:01:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe288e1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 1 -- 172.21.15.50:0/3982682304 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27802e050 con 0x7fe28409fcc0 2023-10-30T12:01:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe281d9b640 1 -- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 msgr2=0x7fe284133d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2023-10-30T12:01:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe288e1f640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe281d9b640 1 -- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 msgr2=0x7fe284133d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:01:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe288e1f640 4 set_mon_vals no callback set 2023-10-30T12:01:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe281d9b640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe284133d20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe28406ead0 tx=0x7fe27802b710 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe281d9b640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe284133d20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe28406ead0 tx=0x7fe27802b710 comp rx=0 tx=0).stop 2023-10-30T12:01:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: ms_handle_reset current mon [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 1 -- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 msgr2=0x7fe284133d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.839+0000 7fe28159a640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe284133d20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 10 monclient: shutdown 2023-10-30T12:01:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:01:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe26c003750 0x7fe26c005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient(hunting): picked mon.b con 0x7fe26c003750 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0063c0 0x7fe26c008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient(hunting): picked mon.a con 0x7fe26c0063c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe26c0090e0 0x7fe26c00b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient(hunting): picked mon.c con 0x7fe26c0090e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 1 -- 172.21.15.50:0/3982682304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c00be60 con 0x7fe26c0063c0 2023-10-30T12:01:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 1 -- 172.21.15.50:0/3982682304 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c00bfd0 con 0x7fe26c003750 2023-10-30T12:01:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 1 -- 172.21.15.50:0/3982682304 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c00c190 con 0x7fe26c0090e0 2023-10-30T12:01:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 20 monclient(hunting): shutdown discarding 1 pending message(s) 2023-10-30T12:01:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 -- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe26c0090e0 msgr2=0x7fe26c00b570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe26c0090e0 0x7fe26c00b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe281d9b640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0063c0 0x7fe26c008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 -- 172.21.15.50:0/3982682304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe26c003750 msgr2=0x7fe26c005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:39.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28259c640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe26c003750 0x7fe26c005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe26c003750 0x7fe26c005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 -- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0063c0 msgr2=0x7fe26c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:39.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0063c0 0x7fe26c008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe282d9d640 1 -- 172.21.15.50:0/3982682304 reap_dead start 2023-10-30T12:01:39.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe281d9b640 10 monclient: get_auth_request con 0x7fe26c0063c0 auth_method 0 2023-10-30T12:01:39.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe281d9b640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0063c0 0x7fe26c008850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:01:39.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28259c640 10 monclient: get_auth_request con 0x7fe26c003750 auth_method 0 2023-10-30T12:01:39.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28259c640 1 --2- 172.21.15.50:0/3982682304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe26c003750 0x7fe26c005b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:01:39.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe28159a640 10 monclient: ms_handle_reset stray mon [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:39.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 -- 172.21.15.50:0/3982682304 shutdown_connections 2023-10-30T12:01:39.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 -- 172.21.15.50:0/3982682304 >> 172.21.15.50:0/3982682304 conn(0x7fe2840a6200 msgr2=0x7fe2840a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:39.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 -- 172.21.15.50:0/3982682304 shutdown_connections 2023-10-30T12:01:39.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.840+0000 7fe288e1f640 1 -- 172.21.15.50:0/3982682304 wait complete. 2023-10-30T12:01:39.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.841+0000 7fe288e1f640 10 monclient: build_initial_monmap 2023-10-30T12:01:39.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.841+0000 7fe288e1f640 10 monclient: monmap: 2023-10-30T12:01:39.875 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:39.875 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:39.876 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:39.876 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:39.876 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:39.876 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.876 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:39.876 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:39.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.841+0000 7fe288e1f640 1 librados: starting msgr at 2023-10-30T12:01:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.841+0000 7fe288e1f640 1 librados: starting objecter 2023-10-30T12:01:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 Processor -- start 2023-10-30T12:01:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 -- start start 2023-10-30T12:01:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 librados: setting wanted keys 2023-10-30T12:01:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 librados: calling monclient init 2023-10-30T12:01:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient: init 2023-10-30T12:01:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:39.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:01:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe28412f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): picked mon.noname-c con 0x7fe28409fcc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28409f000 0x7fe284069e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe28409f000 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:39.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406a350 0x7fe28406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): picked mon.noname-a con 0x7fe28406a350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:39.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c00be60 con 0x7fe28406a350 2023-10-30T12:01:39.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c00bfd0 con 0x7fe28409f000 2023-10-30T12:01:39.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c00c190 con 0x7fe28409fcc0 2023-10-30T12:01:39.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:39.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe288e1f640 10 monclient(hunting): authenticate will time out at 945.025594s 2023-10-30T12:01:39.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe28412f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406a350 0x7fe28406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406a350 0x7fe28406a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56540/0 (socket says 172.21.15.50:56540) 2023-10-30T12:01:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 1 -- 172.21.15.50:0/4000853196 learned_addr learned my addr 172.21.15.50:0/4000853196 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe28412f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41152/0 (socket says 172.21.15.50:41152) 2023-10-30T12:01:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): get_auth_request con 0x7fe28406a350 auth_method 0 2023-10-30T12:01:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28409f000 0x7fe284069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 10 monclient(hunting): get_auth_request con 0x7fe28409fcc0 auth_method 0 2023-10-30T12:01:39.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 10 monclient(hunting): get_auth_request con 0x7fe28409f000 auth_method 0 2023-10-30T12:01:39.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe281d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.842+0000 7fe282d9d640 10 monclient(hunting): handle_auth_done global_id 4482 payload 824 2023-10-30T12:01:39.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 msgr2=0x7fe28412f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe28409fcc0 0x7fe28412f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28409f000 msgr2=0x7fe284069e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28409f000 0x7fe284069e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:01:39.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 1 monclient: found mon.noname-a 2023-10-30T12:01:39.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:39.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:39.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 1 -- 172.21.15.50:0/4000853196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe278009400 con 0x7fe28406a350 2023-10-30T12:01:39.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe282d9d640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406a350 0x7fe28406a7e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe28406fab0 tx=0x7fe27c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:39.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 client.?.objecter ms_handle_connect 0x7fe28406a350 2023-10-30T12:01:39.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:39.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c010040 con 0x7fe28406a350 2023-10-30T12:01:39.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:39.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:39.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: dump: 2023-10-30T12:01:39.890 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:39.890 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.890 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.890 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.890 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:39.890 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.891 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:39.891 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:39.891 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:39.891 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: _finish_auth 0 2023-10-30T12:01:39.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:01:39.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:39.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c00bd30 con 0x7fe28406a350 2023-10-30T12:01:39.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:39.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c00bea0 con 0x7fe28406a350 2023-10-30T12:01:39.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 5 monclient: authenticate success, global_id 4482 2023-10-30T12:01:39.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 10 monclient: _renew_subs 2023-10-30T12:01:39.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:39.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c00c410 con 0x7fe28406a350 2023-10-30T12:01:39.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe280d99640 4 set_mon_vals no callback set 2023-10-30T12:01:39.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:39.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:39.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe272ffd640 10 monclient: dump: 2023-10-30T12:01:39.894 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:39.894 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.894 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.895 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.895 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:39.895 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.895 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:39.895 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:39.896 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:39.896 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:39.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 10 monclient: _renew_subs 2023-10-30T12:01:39.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:39.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28409c5f0 con 0x7fe28406a350 2023-10-30T12:01:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.843+0000 7fe288e1f640 1 librados: init done 2023-10-30T12:01:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe288e1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe288e1f640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9200 0x7fe2841b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe27c003bd0 con 0x7fe28406a350 2023-10-30T12:01:39.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe284073ff0 con 0x7fe2841b9200 2023-10-30T12:01:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe272ffd640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe264039320 0x7fe26403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9200 0x7fe2841b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient: get_auth_request con 0x7fe2841b9200 auth_method 0 2023-10-30T12:01:39.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fe27c04a5e0 con 0x7fe28406a350 2023-10-30T12:01:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe272ffd640 10 client.4482.objecter ms_dispatch 0x7fe284000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:39.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe272ffd640 3 client.4482.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe272ffd640 3 client.4482.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe272ffd640 20 client.4482.objecter dump_active .. 0 homeless 2023-10-30T12:01:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe281d9b640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe264039320 0x7fe26403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe281d9b640 10 monclient: get_auth_request con 0x7fe264039320 auth_method 0 2023-10-30T12:01:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 10 monclient(hunting): handle_auth_done global_id 4474 payload 824 2023-10-30T12:01:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.844+0000 7fe28259c640 1 -- 172.21.15.50:0/4000853196 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe28406fdb0 con 0x7fe2841b9200 2023-10-30T12:01:39.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 client.4482.objecter ms_handle_connect 0x7fe2841b9200 2023-10-30T12:01:39.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 client.4482.objecter resend_mon_ops 2023-10-30T12:01:39.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe28259c640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9200 0x7fe2841b95e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe28406e610 tx=0x7fe274013ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:39.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274004500 con 0x7fe2841b9200 2023-10-30T12:01:39.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:39.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe281d9b640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe264039320 0x7fe26403b7e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe284073340 tx=0x7fe27802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:39.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:39.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 monclient: dump: 2023-10-30T12:01:39.903 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:39.903 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.903 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.903 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.904 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:39.904 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.904 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:39.904 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:39.904 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:39.905 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 client.4482.objecter ms_handle_connect 0x7fe264039320 2023-10-30T12:01:39.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fe274003610 con 0x7fe2841b9200 2023-10-30T12:01:39.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:39.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:39.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9200 msgr2=0x7fe2841b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9200 0x7fe2841b95e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe28406e610 tx=0x7fe274013ae0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9200 msgr2=0x7fe2841b95e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:39.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.845+0000 7fe272ffd640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9200 0x7fe2841b95e0 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe28406e610 tx=0x7fe274013ae0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.851+0000 7fe288e1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:39.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.851+0000 7fe288e1f640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9b20 0x7fe2841d4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:39.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.851+0000 7fe288e1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:39.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.851+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2841bb220 con 0x7fe2841b9b20 2023-10-30T12:01:39.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.851+0000 7fe28259c640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9b20 0x7fe2841d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:39.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient: get_auth_request con 0x7fe2841b9b20 auth_method 0 2023-10-30T12:01:39.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:39.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:39.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:39.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:39.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:39.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:39.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2023-10-30T12:01:39.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 1 -- 172.21.15.50:0/4000853196 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe28409de70 con 0x7fe2841b9b20 2023-10-30T12:01:39.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe28259c640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9b20 0x7fe2841d4780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe28406e610 tx=0x7fe274004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:39.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 10 client.4482.objecter ms_handle_connect 0x7fe2841b9b20 2023-10-30T12:01:39.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 10 client.4482.objecter resend_mon_ops 2023-10-30T12:01:39.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274027070 con 0x7fe2841b9b20 2023-10-30T12:01:39.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:39.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:39.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 10 monclient: dump: 2023-10-30T12:01:39.912 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:39.912 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:39.912 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:39.913 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:39.913 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:39.913 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:39.913 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:39.914 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:39.914 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:39.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fe2740114b0 con 0x7fe2841b9b20 2023-10-30T12:01:39.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:39.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:39.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9b20 msgr2=0x7fe2841d4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9b20 0x7fe2841d4780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe28406e610 tx=0x7fe274004730 comp rx=0 tx=0).stop 2023-10-30T12:01:39.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9b20 msgr2=0x7fe2841d4780 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:39.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.852+0000 7fe272ffd640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe2841b9b20 0x7fe2841d4780 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe28406e610 tx=0x7fe274004730 comp rx=0 tx=0).stop 2023-10-30T12:01:39.915 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:39.916 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:39.916 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:39.916 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:01:39.916 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 10, 2023-10-30T12:01:39.916 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:39.917 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:39.917 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:39.917 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:39.917 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.917 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:01:39.918 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:39.918 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:39.918 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:39.918 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:39.918 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:39.919 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:39.919 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:39.919 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:39.919 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:39.919 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:39.920 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:39.920 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:39.920 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:39.920 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.920 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:39.920 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:39.921 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:39.921 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:39.921 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:39.921 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:39.921 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:39.922 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:39.922 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:39.922 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:39.922 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:39.922 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:39.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.923 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:39.923 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:39.923 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:39.923 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:39.924 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:39.924 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:39.924 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:39.924 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:39.924 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:39.925 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:39.925 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:39.925 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:39.925 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:39.925 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:39.926 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:39.926 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:39.926 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:39.926 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:39.926 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:39.927 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:39.927 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:39.927 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:39.927 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:39.927 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:39.927 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:39.928 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:39.928 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:39.928 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.928 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:39.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.929 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:39.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.929 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:39.929 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:39.929 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:39.930 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:39.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:39.930 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:39.930 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.931 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:39.931 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:39.931 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.932 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.932 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:39.932 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:39.932 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:39.933 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:39.933 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:39.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.933 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:39.934 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:39.934 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:39.934 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:39.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.934 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:39.935 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:39.935 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.935 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:39.935 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:39.936 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.936 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.936 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.936 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:39.937 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:39.937 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:39.937 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:39.937 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:39.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.938 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:39.938 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:39.938 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:39.938 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:39.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:39.939 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:39.939 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.940 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:39.940 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:39.940 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:39.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.940 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.941 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:39.941 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:39.941 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:39.941 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:39.941 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:39.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.942 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.942 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:39.942 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:39.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.943 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:39.943 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:39.943 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:39.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.944 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.944 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:39.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.944 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:39.944 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:39.944 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:01:39.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.945 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:39.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.945 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:39.946 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:39.946 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:01:39.946 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.946 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:39.946 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:01:39.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:39.947 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:39.947 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:39.947 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:39.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:39.948 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:39.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:39.948 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:39.948 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:39.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 10 librados: watch_flush enter 2023-10-30T12:01:39.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 10 librados: watch_flush exit 2023-10-30T12:01:39.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 20 client.4482.objecter shutdown clearing up homeless session... 2023-10-30T12:01:39.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 10 client.4482.objecter successfully canceled tick 2023-10-30T12:01:39.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe264039320 msgr2=0x7fe26403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe264039320 0x7fe26403b7e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe284073340 tx=0x7fe27802f000 comp rx=0 tx=0).stop 2023-10-30T12:01:39.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 10 monclient: shutdown 2023-10-30T12:01:39.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:39.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406a350 msgr2=0x7fe28406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:39.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe282d9d640 1 -- 172.21.15.50:0/4000853196 reap_dead start 2023-10-30T12:01:39.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406a350 0x7fe28406a7e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe28406fab0 tx=0x7fe27c00b480 comp rx=0 tx=0).stop 2023-10-30T12:01:39.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 shutdown_connections 2023-10-30T12:01:39.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 --2- 172.21.15.50:0/4000853196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406a350 0x7fe28406a7e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:39.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 >> 172.21.15.50:0/4000853196 conn(0x7fe2840a6200 msgr2=0x7fe2840a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:39.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 shutdown_connections 2023-10-30T12:01:39.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 -- 172.21.15.50:0/4000853196 wait complete. 2023-10-30T12:01:39.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:39.853+0000 7fe288e1f640 1 librados: shutdown 2023-10-30T12:01:39.953 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:01:39.954 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:01:39.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:01:40.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.062+0000 7fae8304f640 10 monclient: get_monmap_and_config 2023-10-30T12:01:40.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.062+0000 7fae8304f640 10 monclient: build_initial_monmap 2023-10-30T12:01:40.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.062+0000 7fae8304f640 10 monclient: monmap: 2023-10-30T12:01:40.064 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:40.064 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:40.065 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:40.063173+0000 2023-10-30T12:01:40.065 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:40.063173+0000 2023-10-30T12:01:40.065 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:40.065 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:40.065 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:40.065 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:40.066 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:40.066 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:40.066 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:40.066 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:40.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.062+0000 7fae8304f640 1 Processor -- start 2023-10-30T12:01:40.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.062+0000 7fae8304f640 1 -- start start 2023-10-30T12:01:40.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.062+0000 7fae8304f640 10 monclient: init 2023-10-30T12:01:40.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:40.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-10-30T12:01:40.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae7c0a6070 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:40.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fae7c0a57e0 0x7fae7c13a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:40.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c0a57e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:40.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c13d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:40.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c13adf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:40.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c09a270 con 0x7fae7c0a6070 2023-10-30T12:01:40.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:40.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c099f90 con 0x7fae7c0a57e0 2023-10-30T12:01:40.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:40.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c09a550 con 0x7fae7c13adf0 2023-10-30T12:01:40.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:40.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae8304f640 10 monclient(hunting): authenticate will time out at 945.246398s 2023-10-30T12:01:40.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c13d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:40.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae80deb640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fae7c0a6070 0x7fae7c0a6450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55560/0 (socket says 172.21.15.50:55560) 2023-10-30T12:01:40.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae80deb640 1 -- 172.21.15.50:0/2473442760 learned_addr learned my addr 172.21.15.50:0/2473442760 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:40.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c13d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41168/0 (socket says 172.21.15.50:41168) 2023-10-30T12:01:40.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): get_auth_request con 0x7fae7c13adf0 auth_method 0 2023-10-30T12:01:40.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:40.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:40.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:40.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 1 --2- 172.21.15.50:0/2473442760 >> v2:172.21.15.118:3300/0 conn(0x7fae7c0a57e0 0x7fae7c13a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:40.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 10 monclient(hunting): get_auth_request con 0x7fae7c0a57e0 auth_method 0 2023-10-30T12:01:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:40.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:40.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:40.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:40.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient(hunting): handle_auth_done global_id 34112 payload 274 2023-10-30T12:01:40.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 1 -- 172.21.15.50:0/2473442760 >> v2:172.21.15.118:3300/0 conn(0x7fae7c0a57e0 msgr2=0x7fae7c13a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 1 --2- 172.21.15.50:0/2473442760 >> v2:172.21.15.118:3300/0 conn(0x7fae7c0a57e0 0x7fae7c13a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7b7fe640 1 -- 172.21.15.50:0/2473442760 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860807398 0 0) 0x7fae7c09a270 con 0x7fae7c0a6070 2023-10-30T12:01:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 1 -- 172.21.15.50:0/2473442760 >> v1:172.21.15.50:6789/0 conn(0x7fae7c0a6070 legacy=0x7fae7c0a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient: _finish_hunting 0 2023-10-30T12:01:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 1 monclient: found mon.noname-b 2023-10-30T12:01:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:01:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae815ec640 1 -- 172.21.15.50:0/2473442760 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c09a830 con 0x7fae7c13adf0 2023-10-30T12:01:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.063+0000 7fae7b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:01:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae815ec640 1 --2- 172.21.15.50:0/2473442760 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c13d1e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fae740097a0 tx=0x7fae7402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=526d9a1e698d8377 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 1 -- 172.21.15.50:0/2473442760 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7402f050 con 0x7fae7c13adf0 2023-10-30T12:01:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: dump: 2023-10-30T12:01:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:40.086 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:40.086 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:40.086 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:40.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:01:40.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:01:40.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:40.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae8304f640 5 monclient: authenticate success, global_id 34112 2023-10-30T12:01:40.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae8304f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:40.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 1 -- 172.21.15.50:0/2473442760 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae74031700 con 0x7fae7c13adf0 2023-10-30T12:01:40.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:40.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae8304f640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:40.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae8304f640 4 set_mon_vals no callback set 2023-10-30T12:01:40.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 1 -- 172.21.15.50:0/2473442760 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74031a90 con 0x7fae7c13adf0 2023-10-30T12:01:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.064+0000 7fae7b7fe640 10 monclient: dump: 2023-10-30T12:01:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:40.091 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:40.091 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:40.091 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:40.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 10 monclient: shutdown 2023-10-30T12:01:40.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 -- 172.21.15.50:0/2473442760 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 msgr2=0x7fae7c13d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 --2- 172.21.15.50:0/2473442760 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c13d1e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fae740097a0 tx=0x7fae7402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 -- 172.21.15.50:0/2473442760 shutdown_connections 2023-10-30T12:01:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 --2- 172.21.15.50:0/2473442760 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c13d1e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 --2- 172.21.15.50:0/2473442760 >> v2:172.21.15.118:3300/0 conn(0x7fae7c0a57e0 0x7fae7c13a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 -- 172.21.15.50:0/2473442760 >> 172.21.15.50:0/2473442760 conn(0x7fae7c0a81d0 msgr2=0x7fae7c0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 -- 172.21.15.50:0/2473442760 shutdown_connections 2023-10-30T12:01:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 1 -- 172.21.15.50:0/2473442760 wait complete. 2023-10-30T12:01:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 10 monclient: build_initial_monmap 2023-10-30T12:01:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.065+0000 7fae8304f640 10 monclient: monmap: 2023-10-30T12:01:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:40.094 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:40.094 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:40.094 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:40.094 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:40.094 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:40.095 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:40.095 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:40.095 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:40.095 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:40.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 1 librados: starting msgr at 2023-10-30T12:01:40.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 1 librados: starting objecter 2023-10-30T12:01:40.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 1 Processor -- start 2023-10-30T12:01:40.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 1 -- start start 2023-10-30T12:01:40.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 1 librados: setting wanted keys 2023-10-30T12:01:40.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 1 librados: calling monclient init 2023-10-30T12:01:40.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 10 monclient: init 2023-10-30T12:01:40.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:40.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:01:40.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae7c0a57e0 0x7fae7c1ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:40.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.066+0000 7fae8304f640 10 monclient(hunting): picked mon.noname-a con 0x7fae7c0a57e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:40.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae7c0a6070 0x7fae7c121a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:40.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 10 monclient(hunting): picked mon.noname-b con 0x7fae7c0a6070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:40.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c121fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:40.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 10 monclient(hunting): picked mon.noname-c con 0x7fae7c13adf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:40.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:40.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c1251b0 con 0x7fae7c0a57e0 2023-10-30T12:01:40.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:40.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0a1690 con 0x7fae7c0a6070 2023-10-30T12:01:40.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:40.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c13a490 con 0x7fae7c13adf0 2023-10-30T12:01:40.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:40.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae8304f640 10 monclient(hunting): authenticate will time out at 945.250126s 2023-10-30T12:01:40.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae7c0a57e0 0x7fae7c1ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:40.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae7c0a57e0 0x7fae7c1ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56556/0 (socket says 172.21.15.50:56556) 2023-10-30T12:01:40.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 1 -- 172.21.15.50:0/2864649302 learned_addr learned my addr 172.21.15.50:0/2864649302 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:40.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 10 monclient(hunting): get_auth_request con 0x7fae7c0a57e0 auth_method 0 2023-10-30T12:01:40.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:40.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:40.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:40.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae7c0a6070 0x7fae7c121a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:40.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): get_auth_request con 0x7fae7c0a6070 auth_method 0 2023-10-30T12:01:40.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:40.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:40.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:40.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:40.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:40.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c121fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae80deb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 10 monclient(hunting): get_auth_request con 0x7fae7c13adf0 auth_method 0 2023-10-30T12:01:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:40.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae815ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient(hunting): handle_auth_done global_id 4483 payload 824 2023-10-30T12:01:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 -- 172.21.15.50:0/2864649302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 msgr2=0x7fae7c121fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c121fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 -- 172.21.15.50:0/2864649302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae7c0a57e0 msgr2=0x7fae7c1ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:40.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae7c0a57e0 0x7fae7c1ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:01:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 monclient: found mon.noname-b 2023-10-30T12:01:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:01:40.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 -- 172.21.15.50:0/2864649302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae74009400 con 0x7fae7c0a6070 2023-10-30T12:01:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae7bfff640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae7c0a6070 0x7fae7c121a60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00efc0 tx=0x7fae6c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae797fa640 10 client.?.objecter ms_handle_connect 0x7fae7c0a6070 2023-10-30T12:01:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.067+0000 7fae797fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 1 -- 172.21.15.50:0/2864649302 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c014480 con 0x7fae7c0a6070 2023-10-30T12:01:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:40.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: dump: 2023-10-30T12:01:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.108 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:40.109 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:40.110 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: _finish_auth 0 2023-10-30T12:01:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: _check_auth_tickets 2023-10-30T12:01:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:40.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 1 -- 172.21.15.50:0/2864649302 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c00f040 con 0x7fae7c0a6070 2023-10-30T12:01:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 5 monclient: authenticate success, global_id 4483 2023-10-30T12:01:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 1 -- 172.21.15.50:0/2864649302 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c014800 con 0x7fae7c0a6070 2023-10-30T12:01:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:40.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae797fa640 10 monclient: dump: 2023-10-30T12:01:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:40.113 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 10 monclient: _renew_subs 2023-10-30T12:01:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae7affd640 4 set_mon_vals no callback set 2023-10-30T12:01:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c125a50 con 0x7fae7c0a6070 2023-10-30T12:01:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 10 monclient: _renew_subs 2023-10-30T12:01:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c123610 con 0x7fae7c0a6070 2023-10-30T12:01:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.069+0000 7fae8304f640 1 librados: init done 2023-10-30T12:01:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae797fa640 1 -- 172.21.15.50:0/2864649302 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fae6c014960 con 0x7fae7c0a6070 2023-10-30T12:01:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae8304f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae8304f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae8304f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae797fa640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fae5c039320 0x7fae5c03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c04e160 con 0x7fae7c0a6070 2023-10-30T12:01:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae797fa640 1 -- 172.21.15.50:0/2864649302 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fae6c04b3b0 con 0x7fae7c0a6070 2023-10-30T12:01:40.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae797fa640 10 client.4483.objecter ms_dispatch 0x7fae7c000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:40.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae797fa640 3 client.4483.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:40.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae797fa640 3 client.4483.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:40.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae797fa640 20 client.4483.objecter dump_active .. 0 homeless 2023-10-30T12:01:40.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae80deb640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fae5c039320 0x7fae5c03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:40.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.070+0000 7fae80deb640 10 monclient: get_auth_request con 0x7fae5c039320 auth_method 0 2023-10-30T12:01:40.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.071+0000 7fae797fa640 10 client.4483.objecter ms_handle_connect 0x7fae5c039320 2023-10-30T12:01:40.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.071+0000 7fae80deb640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fae5c039320 0x7fae5c03b7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae64000f60 tx=0x7fae64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:40.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.079+0000 7fae797fa640 1 -- 172.21.15.50:0/2864649302 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae6c007d90 con 0x7fae7c0a6070 2023-10-30T12:01:40.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.079+0000 7fae797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:40.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.079+0000 7fae797fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:40.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.323+0000 7fae8304f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:01:40.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.323+0000 7fae8304f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:01:40.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.323+0000 7fae8304f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:40.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.323+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fae7c074580 con 0x7fae7c0a6070 2023-10-30T12:01:40.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.325+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.325+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3561883710,config=3140514009,health=2406479829,kv=2905431493,logm=1306321170}) 2023-10-30T12:01:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.325+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=8,logm=69} crc {auth=3561883710,config=3140514009,health=781701522,kv=2825148485,logm=1787789318}) 2023-10-30T12:01:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.325+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.325+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3561883710,config=3140514009,health=2406479829,kv=2905431493,logm=1306321170}) 2023-10-30T12:01:40.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.325+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=3536408595,config=3140514009,health=2406479829,kv=3201645878,logm=2567525331}) 2023-10-30T12:01:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.325+0000 7fae797fa640 1 -- 172.21.15.50:0/2864649302 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fae6c0393b0 con 0x7fae7c0a6070 2023-10-30T12:01:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.325+0000 7fae797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:01:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.325+0000 7fae797fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:40.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.326+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.326+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694727728}) 2023-10-30T12:01:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.326+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526005700}) 2023-10-30T12:01:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.326+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.326+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694727728}) 2023-10-30T12:01:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.326+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566505861}) 2023-10-30T12:01:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.327+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.327+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258574039}) 2023-10-30T12:01:40.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 10 librados: watch_flush enter 2023-10-30T12:01:40.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 10 librados: watch_flush exit 2023-10-30T12:01:40.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.327+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1353820724}) 2023-10-30T12:01:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 20 client.4483.objecter shutdown clearing up homeless session... 2023-10-30T12:01:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 10 client.4483.objecter successfully canceled tick 2023-10-30T12:01:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.327+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.327+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258574039}) 2023-10-30T12:01:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.327+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2555323525}) 2023-10-30T12:01:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fae5c039320 msgr2=0x7fae5c03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fae5c039320 0x7fae5c03b7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae64000f60 tx=0x7fae64009040 comp rx=0 tx=0).stop 2023-10-30T12:01:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 10 monclient: shutdown 2023-10-30T12:01:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae7c0a6070 msgr2=0x7fae7c121a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae7c0a6070 0x7fae7c121a60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00efc0 tx=0x7fae6c00c490 comp rx=0 tx=0).stop 2023-10-30T12:01:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 shutdown_connections 2023-10-30T12:01:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fae5c039320 0x7fae5c03b7e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae7c13adf0 0x7fae7c121fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fae7c0a6070 0x7fae7c121a60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 --2- 172.21.15.50:0/2864649302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae7c0a57e0 0x7fae7c1ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 >> 172.21.15.50:0/2864649302 conn(0x7fae7c0a81d0 msgr2=0x7fae7c0a3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 shutdown_connections 2023-10-30T12:01:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 -- 172.21.15.50:0/2864649302 wait complete. 2023-10-30T12:01:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:40.327+0000 7fae8304f640 1 librados: shutdown 2023-10-30T12:01:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.328+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.328+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3378049375,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3133902560,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=577945228,osdmap=3479436589}) 2023-10-30T12:01:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.328+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=21,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=2116055789,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3133902560,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=577945228,osdmap=2903036879}) 2023-10-30T12:01:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.328+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:01:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.328+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=3378049375,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3133902560,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=577945228,osdmap=3479436589}) 2023-10-30T12:01:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:01:40.328+0000 7f551d752640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=22,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=35} crc {logm=3378049375,mdsmap=2441318933,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3780290829,mon_config_key=715115367,monmap=3324057927,osd_metadata=2398621561,osd_pg_creating=577945228,osdmap=4007582904}) 2023-10-30T12:01:40.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:01:41.341 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:01:41.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.456+0000 7f102cfce640 10 monclient: get_monmap_and_config 2023-10-30T12:01:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.456+0000 7f102cfce640 10 monclient: build_initial_monmap 2023-10-30T12:01:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.456+0000 7f102cfce640 10 monclient: monmap: 2023-10-30T12:01:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:41.457133+0000 2023-10-30T12:01:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:41.457133+0000 2023-10-30T12:01:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:41.461 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.456+0000 7f102cfce640 1 Processor -- start 2023-10-30T12:01:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.456+0000 7f102cfce640 1 -- start start 2023-10-30T12:01:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.456+0000 7f102cfce640 10 monclient: init 2023-10-30T12:01:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:01:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f10280a6070 0x7f10280a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient(hunting): picked mon.noname-c con 0x7f10280a6070 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10280a6450 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1028135ca0 0x7f102813d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient(hunting): picked mon.noname-a con 0x7f1028135ca0 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102809a270 con 0x7f10280a6450 2023-10-30T12:01:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1028099f90 con 0x7f1028135ca0 2023-10-30T12:01:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102809a550 con 0x7f10280a6070 2023-10-30T12:01:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102cfce640 10 monclient(hunting): authenticate will time out at 946.640370s 2023-10-30T12:01:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1028135ca0 0x7f102813d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1025d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f10280a6450 0x7f1028135590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55572/0 (socket says 172.21.15.50:55572) 2023-10-30T12:01:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1025d9b640 1 -- 172.21.15.50:0/3140008523 learned_addr learned my addr 172.21.15.50:0/3140008523 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): get_auth_request con 0x7f1028135ca0 auth_method 0 2023-10-30T12:01:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 1 --2- 172.21.15.50:0/3140008523 >> v2:172.21.15.118:3300/0 conn(0x7f10280a6070 0x7f10280a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 10 monclient(hunting): get_auth_request con 0x7f10280a6070 auth_method 0 2023-10-30T12:01:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient(hunting): handle_auth_done global_id 4491 payload 274 2023-10-30T12:01:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 1 -- 172.21.15.50:0/3140008523 >> v2:172.21.15.118:3300/0 conn(0x7f10280a6070 msgr2=0x7f10280a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 1 --2- 172.21.15.50:0/3140008523 >> v2:172.21.15.118:3300/0 conn(0x7f10280a6070 0x7f10280a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 1 -- 172.21.15.50:0/3140008523 >> v1:172.21.15.50:6789/0 conn(0x7f10280a6450 legacy=0x7f1028135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:01:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 1 monclient: found mon.noname-a 2023-10-30T12:01:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f1026d9d640 1 -- 172.21.15.50:0/3140008523 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102809a830 con 0x7f1028135ca0 2023-10-30T12:01:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102559a640 1 -- 172.21.15.50:0/3140008523 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736727450 0 0) 0x7f102809a270 con 0x7f10280a6450 2023-10-30T12:01:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.457+0000 7f102559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:01:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f1026d9d640 1 --2- 172.21.15.50:0/3140008523 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f102813d8d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f10280a4ee0 tx=0x7f102002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f5110fc8943608 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 1 -- 172.21.15.50:0/3140008523 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020031710 con 0x7f1028135ca0 2023-10-30T12:01:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: dump: 2023-10-30T12:01:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.480 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: _finish_auth 0 2023-10-30T12:01:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: _check_auth_tickets 2023-10-30T12:01:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 1 -- 172.21.15.50:0/3140008523 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102002f050 con 0x7f1028135ca0 2023-10-30T12:01:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102cfce640 5 monclient: authenticate success, global_id 4491 2023-10-30T12:01:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102cfce640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102cfce640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 1 -- 172.21.15.50:0/3140008523 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020031a90 con 0x7f1028135ca0 2023-10-30T12:01:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102cfce640 4 set_mon_vals no callback set 2023-10-30T12:01:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.458+0000 7f102559a640 10 monclient: dump: 2023-10-30T12:01:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.485 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 10 monclient: shutdown 2023-10-30T12:01:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 -- 172.21.15.50:0/3140008523 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 msgr2=0x7f102813d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 --2- 172.21.15.50:0/3140008523 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f102813d8d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f10280a4ee0 tx=0x7f102002b7b0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 -- 172.21.15.50:0/3140008523 shutdown_connections 2023-10-30T12:01:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 --2- 172.21.15.50:0/3140008523 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f102813d8d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 --2- 172.21.15.50:0/3140008523 >> v2:172.21.15.118:3300/0 conn(0x7f10280a6070 0x7f10280a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 -- 172.21.15.50:0/3140008523 >> 172.21.15.50:0/3140008523 conn(0x7f10280a81d0 msgr2=0x7f10280a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 -- 172.21.15.50:0/3140008523 shutdown_connections 2023-10-30T12:01:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.459+0000 7f102cfce640 1 -- 172.21.15.50:0/3140008523 wait complete. 2023-10-30T12:01:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 10 monclient: build_initial_monmap 2023-10-30T12:01:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 10 monclient: monmap: 2023-10-30T12:01:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:41.489 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 1 librados: starting msgr at 2023-10-30T12:01:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 1 librados: starting objecter 2023-10-30T12:01:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 1 Processor -- start 2023-10-30T12:01:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 1 -- start start 2023-10-30T12:01:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 1 librados: setting wanted keys 2023-10-30T12:01:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 1 librados: calling monclient init 2023-10-30T12:01:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.460+0000 7f102cfce640 10 monclient: init 2023-10-30T12:01:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:01:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10280a6070 0x7f10281ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): picked mon.noname-c con 0x7f10280a6070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10280a6450 0x7f102811fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): picked mon.noname-b con 0x7f10280a6450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f1028120060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): picked mon.noname-a con 0x7f1028135ca0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10281235d0 con 0x7f1028135ca0 2023-10-30T12:01:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1028122470 con 0x7f10280a6450 2023-10-30T12:01:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10280a1690 con 0x7f10280a6070 2023-10-30T12:01:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102cfce640 10 monclient(hunting): authenticate will time out at 946.644105s 2023-10-30T12:01:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f1028120060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10280a6070 0x7f10281ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f1028120060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56574/0 (socket says 172.21.15.50:56574) 2023-10-30T12:01:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 -- 172.21.15.50:0/2588479644 learned_addr learned my addr 172.21.15.50:0/2588479644 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): get_auth_request con 0x7f1028135ca0 auth_method 0 2023-10-30T12:01:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 10 monclient(hunting): get_auth_request con 0x7f10280a6070 auth_method 0 2023-10-30T12:01:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10280a6450 0x7f102811fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 10 monclient(hunting): get_auth_request con 0x7f10280a6450 auth_method 0 2023-10-30T12:01:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1025d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient(hunting): handle_auth_done global_id 4497 payload 824 2023-10-30T12:01:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 -- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10280a6070 msgr2=0x7f10281ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10280a6070 0x7f10281ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 -- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10280a6450 msgr2=0x7f102811fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10280a6450 0x7f102811fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:01:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 monclient: found mon.noname-a 2023-10-30T12:01:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 -- 172.21.15.50:0/2588479644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1020009400 con 0x7f1028135ca0 2023-10-30T12:01:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f102659c640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10280a6070 0x7f10281ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f1026d9d640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f1028120060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1020009890 tx=0x7f10200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f101effd640 10 client.?.objecter ms_handle_connect 0x7f1028135ca0 2023-10-30T12:01:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f101effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f101effd640 1 -- 172.21.15.50:0/2588479644 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020004230 con 0x7f1028135ca0 2023-10-30T12:01:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f101effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f101effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.461+0000 7f101effd640 10 monclient: dump: 2023-10-30T12:01:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.503 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 10 monclient: _finish_auth 0 2023-10-30T12:01:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 10 monclient: _check_auth_tickets 2023-10-30T12:01:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 1 -- 172.21.15.50:0/2588479644 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1020004390 con 0x7f1028135ca0 2023-10-30T12:01:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 5 monclient: authenticate success, global_id 4497 2023-10-30T12:01:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 10 monclient: _renew_subs 2023-10-30T12:01:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10281353a0 con 0x7f1028135ca0 2023-10-30T12:01:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 1 -- 172.21.15.50:0/2588479644 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102003d820 con 0x7f1028135ca0 2023-10-30T12:01:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 10 monclient: dump: 2023-10-30T12:01:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.508 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f1024d99640 4 set_mon_vals no callback set 2023-10-30T12:01:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 10 monclient: _renew_subs 2023-10-30T12:01:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10281217c0 con 0x7f1028135ca0 2023-10-30T12:01:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f102cfce640 1 librados: init done 2023-10-30T12:01:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.462+0000 7f101effd640 1 -- 172.21.15.50:0/2588479644 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f102003a050 con 0x7f1028135ca0 2023-10-30T12:01:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f101effd640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f10080392f0 0x7f100803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f102cfce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f102cfce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f102cfce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1028123cd0 con 0x7f1028135ca0 2023-10-30T12:01:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f101effd640 1 -- 172.21.15.50:0/2588479644 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f102006bf80 con 0x7f1028135ca0 2023-10-30T12:01:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f101effd640 10 client.4497.objecter ms_dispatch 0x7f1028000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f101effd640 3 client.4497.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f101effd640 3 client.4497.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f101effd640 20 client.4497.objecter dump_active .. 0 homeless 2023-10-30T12:01:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f102659c640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f10080392f0 0x7f100803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f102659c640 10 monclient: get_auth_request con 0x7f10080392f0 auth_method 0 2023-10-30T12:01:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f102659c640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f10080392f0 0x7f100803b7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f101800e740 tx=0x7f101800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.463+0000 7f101effd640 10 client.4497.objecter ms_handle_connect 0x7f10080392f0 2023-10-30T12:01:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.467+0000 7f101effd640 1 -- 172.21.15.50:0/2588479644 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1028123cd0 con 0x7f1028135ca0 2023-10-30T12:01:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.468+0000 7f101effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.468+0000 7f101effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:41.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.712+0000 7f102cfce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:41.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.712+0000 7f102cfce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.712+0000 7f102cfce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.712+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1028123710 con 0x7f1028135ca0 2023-10-30T12:01:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.712+0000 7f101effd640 1 -- 172.21.15.50:0/2588479644 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f1028123710 con 0x7f1028135ca0 2023-10-30T12:01:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.712+0000 7f101effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.712+0000 7f101effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:41.716 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:41.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 10 librados: watch_flush enter 2023-10-30T12:01:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 10 librados: watch_flush exit 2023-10-30T12:01:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 20 client.4497.objecter shutdown clearing up homeless session... 2023-10-30T12:01:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 10 client.4497.objecter successfully canceled tick 2023-10-30T12:01:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f10080392f0 msgr2=0x7f100803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f10080392f0 0x7f100803b7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f101800e740 tx=0x7f101800b040 comp rx=0 tx=0).stop 2023-10-30T12:01:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 10 monclient: shutdown 2023-10-30T12:01:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 msgr2=0x7f1028120060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.714+0000 7f102cfce640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f1028120060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1020009890 tx=0x7f10200046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 shutdown_connections 2023-10-30T12:01:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f10080392f0 0x7f100803b7b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1028135ca0 0x7f1028120060 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f10280a6450 0x7f102811fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 --2- 172.21.15.50:0/2588479644 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10280a6070 0x7f10281ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 >> 172.21.15.50:0/2588479644 conn(0x7f10280a81d0 msgr2=0x7f10280a32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 shutdown_connections 2023-10-30T12:01:41.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 -- 172.21.15.50:0/2588479644 wait complete. 2023-10-30T12:01:41.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.715+0000 7f102cfce640 1 librados: shutdown 2023-10-30T12:01:41.727 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:01:41.727 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:01:41.727 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:01:41.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:01:41.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.832+0000 7fa4ab390640 10 monclient: get_monmap_and_config 2023-10-30T12:01:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.832+0000 7fa4ab390640 10 monclient: build_initial_monmap 2023-10-30T12:01:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.832+0000 7fa4ab390640 10 monclient: monmap: 2023-10-30T12:01:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:41.833141+0000 2023-10-30T12:01:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:41.833141+0000 2023-10-30T12:01:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:41.836 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.832+0000 7fa4ab390640 1 Processor -- start 2023-10-30T12:01:41.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.832+0000 7fa4ab390640 1 -- start start 2023-10-30T12:01:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.832+0000 7fa4ab390640 10 monclient: init 2023-10-30T12:01:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-10-30T12:01:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4a40a8010 0x7fa4a4132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a40a8010 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4a4099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa4a4132bd0 0x7fa4a4135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a4132bd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40a7190 con 0x7fa4a4099a40 2023-10-30T12:01:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40a83f0 con 0x7fa4a40a8010 2023-10-30T12:01:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40a7600 con 0x7fa4a4132bd0 2023-10-30T12:01:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4ab390640 10 monclient(hunting): authenticate will time out at 947.016319s 2023-10-30T12:01:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4a40a8010 0x7fa4a4132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4a40a8010 0x7fa4a4132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56580/0 (socket says 172.21.15.50:56580) 2023-10-30T12:01:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 -- 172.21.15.50:0/3420919791 learned_addr learned my addr 172.21.15.50:0/3420919791 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): get_auth_request con 0x7fa4a40a8010 auth_method 0 2023-10-30T12:01:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a992d640 1 --2- 172.21.15.50:0/3420919791 >> v2:172.21.15.118:3300/0 conn(0x7fa4a4132bd0 0x7fa4a4135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a992d640 10 monclient(hunting): get_auth_request con 0x7fa4a4132bd0 auth_method 0 2023-10-30T12:01:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a992d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a992d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a992d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient(hunting): handle_auth_done global_id 4500 payload 274 2023-10-30T12:01:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 -- 172.21.15.50:0/3420919791 >> v2:172.21.15.118:3300/0 conn(0x7fa4a4132bd0 msgr2=0x7fa4a4135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 --2- 172.21.15.50:0/3420919791 >> v2:172.21.15.118:3300/0 conn(0x7fa4a4132bd0 0x7fa4a4135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 -- 172.21.15.50:0/3420919791 >> v1:172.21.15.118:6789/0 conn(0x7fa4a4099a40 legacy=0x7fa4a41356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient: _finish_hunting 0 2023-10-30T12:01:41.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 monclient: found mon.noname-a 2023-10-30T12:01:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.833+0000 7fa4a912c640 1 -- 172.21.15.50:0/3420919791 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a409a440 con 0x7fa4a40a8010 2023-10-30T12:01:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a912c640 1 --2- 172.21.15.50:0/3420919791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a40a8010 0x7fa4a4132690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa4a409a210 tx=0x7fa49c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e936f89a8e63b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 1 -- 172.21.15.50:0/3420919791 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0317e0 con 0x7fa4a40a8010 2023-10-30T12:01:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: dump: 2023-10-30T12:01:41.853 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.854 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.855 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: _finish_auth 0 2023-10-30T12:01:41.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4ab390640 5 monclient: authenticate success, global_id 4500 2023-10-30T12:01:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4ab390640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 1 -- 172.21.15.50:0/3420919791 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c031940 con 0x7fa4a40a8010 2023-10-30T12:01:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4ab390640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:41.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4ab390640 4 set_mon_vals no callback set 2023-10-30T12:01:41.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 1 -- 172.21.15.50:0/3420919791 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c031cd0 con 0x7fa4a40a8010 2023-10-30T12:01:41.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:41.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.834+0000 7fa4a3fff640 10 monclient: dump: 2023-10-30T12:01:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.859 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 10 monclient: shutdown 2023-10-30T12:01:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 -- 172.21.15.50:0/3420919791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a40a8010 msgr2=0x7fa4a4132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 --2- 172.21.15.50:0/3420919791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a40a8010 0x7fa4a4132690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa4a409a210 tx=0x7fa49c02b880 comp rx=0 tx=0).stop 2023-10-30T12:01:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 -- 172.21.15.50:0/3420919791 shutdown_connections 2023-10-30T12:01:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 --2- 172.21.15.50:0/3420919791 >> v2:172.21.15.118:3300/0 conn(0x7fa4a4132bd0 0x7fa4a4135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 --2- 172.21.15.50:0/3420919791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a40a8010 0x7fa4a4132690 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 -- 172.21.15.50:0/3420919791 >> 172.21.15.50:0/3420919791 conn(0x7fa4a40a4a10 msgr2=0x7fa4a40a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 -- 172.21.15.50:0/3420919791 shutdown_connections 2023-10-30T12:01:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 1 -- 172.21.15.50:0/3420919791 wait complete. 2023-10-30T12:01:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 10 monclient: build_initial_monmap 2023-10-30T12:01:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.835+0000 7fa4ab390640 10 monclient: monmap: 2023-10-30T12:01:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:41.863 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:41.863 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.863 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:41.863 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:41.863 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:41.864 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 librados: starting msgr at 2023-10-30T12:01:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 librados: starting objecter 2023-10-30T12:01:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 Processor -- start 2023-10-30T12:01:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 -- start start 2023-10-30T12:01:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 librados: setting wanted keys 2023-10-30T12:01:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 librados: calling monclient init 2023-10-30T12:01:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient: init 2023-10-30T12:01:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:01:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4a40a8010 0x7fa4a41ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a40a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa4a4132bd0 0x7fa4a41cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a4132bd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411e470 0x7fa4a411e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a411e470 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4121c80 con 0x7fa4a411e470 2023-10-30T12:01:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41221b0 con 0x7fa4a40a8010 2023-10-30T12:01:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4135730 con 0x7fa4a4132bd0 2023-10-30T12:01:41.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:41.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4ab390640 10 monclient(hunting): authenticate will time out at 947.019725s 2023-10-30T12:01:41.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411e470 0x7fa4a411e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa4a4132bd0 0x7fa4a41cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411e470 0x7fa4a411e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56586/0 (socket says 172.21.15.50:56586) 2023-10-30T12:01:41.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 1 -- 172.21.15.50:0/2895428147 learned_addr learned my addr 172.21.15.50:0/2895428147 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:41.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 10 monclient(hunting): get_auth_request con 0x7fa4a411e470 auth_method 0 2023-10-30T12:01:41.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4a40a8010 0x7fa4a41ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 10 monclient(hunting): get_auth_request con 0x7fa4a4132bd0 auth_method 0 2023-10-30T12:01:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 10 monclient(hunting): get_auth_request con 0x7fa4a40a8010 auth_method 0 2023-10-30T12:01:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a992d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a892b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.836+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 10 monclient(hunting): handle_auth_done global_id 4503 payload 824 2023-10-30T12:01:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa4a4132bd0 msgr2=0x7fa4a41cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa4a4132bd0 0x7fa4a41cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4a40a8010 msgr2=0x7fa4a41ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4a40a8010 0x7fa4a41ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 10 monclient: _finish_hunting 0 2023-10-30T12:01:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 1 monclient: found mon.noname-a 2023-10-30T12:01:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 1 -- 172.21.15.50:0/2895428147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0094d0 con 0x7fa4a411e470 2023-10-30T12:01:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a992d640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411e470 0x7fa4a411e900 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa494007e40 tx=0x7fa49400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 client.?.objecter ms_handle_connect 0x7fa4a411e470 2023-10-30T12:01:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494014620 con 0x7fa4a411e470 2023-10-30T12:01:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: dump: 2023-10-30T12:01:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.878 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:01:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:01:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49400ee90 con 0x7fa4a411e470 2023-10-30T12:01:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 5 monclient: authenticate success, global_id 4503 2023-10-30T12:01:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 10 monclient: _renew_subs 2023-10-30T12:01:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a40a18d0 con 0x7fa4a411e470 2023-10-30T12:01:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940149e0 con 0x7fa4a411e470 2023-10-30T12:01:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a1ffb640 10 monclient: dump: 2023-10-30T12:01:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.883 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4a37fe640 4 set_mon_vals no callback set 2023-10-30T12:01:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 10 monclient: _renew_subs 2023-10-30T12:01:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:01:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a4131f20 con 0x7fa4a411e470 2023-10-30T12:01:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.837+0000 7fa4ab390640 1 librados: init done 2023-10-30T12:01:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4ab390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4ab390640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a4099a40 0x7fa4a402b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4ab390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4073fd0 con 0x7fa4a4099a40 2023-10-30T12:01:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa494014b40 con 0x7fa4a411e470 2023-10-30T12:01:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a4099a40 0x7fa4a402b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient: get_auth_request con 0x7fa4a4099a40 auth_method 0 2023-10-30T12:01:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a1ffb640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa4900392f0 0x7fa49003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fa49404af50 con 0x7fa4a411e470 2023-10-30T12:01:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a1ffb640 10 client.4503.objecter ms_dispatch 0x7fa4a4000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a1ffb640 3 client.4503.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a1ffb640 3 client.4503.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 10 monclient(hunting): handle_auth_done global_id 4506 payload 824 2023-10-30T12:01:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.838+0000 7fa4a912c640 1 -- 172.21.15.50:0/2895428147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4a409aa00 con 0x7fa4a4099a40 2023-10-30T12:01:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 20 client.4503.objecter dump_active .. 0 homeless 2023-10-30T12:01:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a892b640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa4900392f0 0x7fa49003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a892b640 10 monclient: get_auth_request con 0x7fa4900392f0 auth_method 0 2023-10-30T12:01:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 client.4503.objecter ms_handle_connect 0x7fa4a4099a40 2023-10-30T12:01:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a912c640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a4099a40 0x7fa4a402b430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40a18a0 tx=0x7fa49c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 client.4503.objecter resend_mon_ops 2023-10-30T12:01:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c032940 con 0x7fa4a4099a40 2023-10-30T12:01:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 monclient: dump: 2023-10-30T12:01:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.892 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a892b640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa4900392f0 0x7fa49003b7b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4a411fcc0 tx=0x7fa498009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 client.4503.objecter ms_handle_connect 0x7fa4900392f0 2023-10-30T12:01:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa49c032aa0 con 0x7fa4a4099a40 2023-10-30T12:01:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a4099a40 msgr2=0x7fa4a402b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a4099a40 0x7fa4a402b430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40a18a0 tx=0x7fa49c004730 comp rx=0 tx=0).stop 2023-10-30T12:01:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a4099a40 msgr2=0x7fa4a402b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.839+0000 7fa4a1ffb640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a4099a40 0x7fa4a402b430 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40a18a0 tx=0x7fa49c004730 comp rx=0 tx=0).stop 2023-10-30T12:01:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4ab390640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4ab390640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411f600 0x7fa4a402b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4ab390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a409a750 con 0x7fa4a411f600 2023-10-30T12:01:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4a912c640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411f600 0x7fa4a402b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4a912c640 10 monclient: get_auth_request con 0x7fa4a411f600 auth_method 0 2023-10-30T12:01:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4a912c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4a912c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.845+0000 7fa4a912c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a912c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a912c640 10 monclient(hunting): handle_auth_done global_id 4509 payload 824 2023-10-30T12:01:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a912c640 1 -- 172.21.15.50:0/2895428147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa4a4133080 con 0x7fa4a411f600 2023-10-30T12:01:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a912c640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411f600 0x7fa4a402b970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa49c02b8c0 tx=0x7fa49c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 10 client.4503.objecter ms_handle_connect 0x7fa4a411f600 2023-10-30T12:01:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 10 client.4503.objecter resend_mon_ops 2023-10-30T12:01:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c003ae0 con 0x7fa4a411f600 2023-10-30T12:01:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 10 monclient: dump: 2023-10-30T12:01:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:41.902 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fa49c054070 con 0x7fa4a411f600 2023-10-30T12:01:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411f600 msgr2=0x7fa4a402b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411f600 0x7fa4a402b970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa49c02b8c0 tx=0x7fa49c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411f600 msgr2=0x7fa4a402b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4a1ffb640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411f600 0x7fa4a402b970 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa49c02b8c0 tx=0x7fa49c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.904 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:41.904 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:41.904 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:41.904 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:01:41.904 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 10, 2023-10-30T12:01:41.905 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:41.905 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:41.905 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:41.905 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:41.905 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:41.906 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:01:41.906 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:41.906 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:41.906 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:41.906 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:41.907 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:41.907 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:41.907 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:41.907 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:41.907 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:41.908 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:41.908 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:41.908 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:41.908 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:41.908 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:41.909 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:41.909 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:41.909 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:41.909 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:41.909 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:41.910 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:41.910 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:41.910 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:41.910 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:41.910 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:41.911 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:41.911 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:41.911 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:41.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.911 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:41.912 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:41.912 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:41.912 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:41.912 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:41.912 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:41.913 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:41.913 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:41.913 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:41.913 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:41.913 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:41.913 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:41.914 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:41.914 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:41.914 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:41.914 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:41.914 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:41.915 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:41.915 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:41.915 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:41.915 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:41.915 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:41.916 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:41.916 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:41.916 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:41.916 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:41.916 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:41.917 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:41.917 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:41.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.917 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:41.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.917 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:41.918 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:41.918 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:41.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:41.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:41.919 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:41.919 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:41.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.919 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:41.920 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:41.920 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:41.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:41.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:41.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.920 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:41.921 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:41.921 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:41.921 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:41.921 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:41.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.922 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:41.922 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:41.922 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:41.922 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:41.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.923 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:41.923 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:41.923 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:41.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:41.924 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:41.924 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:41.924 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:41.925 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:41.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.925 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:41.925 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:41.925 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:41.926 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:41.926 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:41.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:01:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:01:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:41.936 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:41.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4ab390640 10 librados: watch_flush enter 2023-10-30T12:01:41.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4ab390640 10 librados: watch_flush exit 2023-10-30T12:01:41.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4ab390640 20 client.4503.objecter shutdown clearing up homeless session... 2023-10-30T12:01:41.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.846+0000 7fa4ab390640 10 client.4503.objecter successfully canceled tick 2023-10-30T12:01:41.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa4900392f0 msgr2=0x7fa49003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa4900392f0 0x7fa49003b7b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4a411fcc0 tx=0x7fa498009040 comp rx=0 tx=0).stop 2023-10-30T12:01:41.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 10 monclient: shutdown 2023-10-30T12:01:41.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:41.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411e470 msgr2=0x7fa4a411e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:41.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 --2- 172.21.15.50:0/2895428147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a411e470 0x7fa4a411e900 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa494007e40 tx=0x7fa49400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:01:41.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4a992d640 1 -- 172.21.15.50:0/2895428147 reap_dead start 2023-10-30T12:01:41.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4a992d640 1 -- 172.21.15.50:0/2895428147 reap_dead start 2023-10-30T12:01:41.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 shutdown_connections 2023-10-30T12:01:41.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 >> 172.21.15.50:0/2895428147 conn(0x7fa4a40a4a10 msgr2=0x7fa4a4133d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:41.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 shutdown_connections 2023-10-30T12:01:41.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 -- 172.21.15.50:0/2895428147 wait complete. 2023-10-30T12:01:41.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:41.847+0000 7fa4ab390640 1 librados: shutdown 2023-10-30T12:01:41.942 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:01:41.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:01:41.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:01:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.046+0000 7fa399ca2640 10 monclient: get_monmap_and_config 2023-10-30T12:01:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.046+0000 7fa399ca2640 10 monclient: build_initial_monmap 2023-10-30T12:01:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.046+0000 7fa399ca2640 10 monclient: monmap: 2023-10-30T12:01:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:42.047703+0000 2023-10-30T12:01:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:42.047703+0000 2023-10-30T12:01:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:42.053 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 1 Processor -- start 2023-10-30T12:01:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 1 -- start start 2023-10-30T12:01:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient: init 2023-10-30T12:01:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:01:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa3940a2ae0 0x7fa3940a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient(hunting): picked mon.noname-c con 0x7fa3940a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3940a2ec0 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa394135df0 0x7fa39413da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient(hunting): picked mon.noname-b con 0x7fa394135df0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3940a4190 con 0x7fa3940a2ec0 2023-10-30T12:01:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3940a2990 con 0x7fa3940a2ae0 2023-10-30T12:01:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3940a1630 con 0x7fa394135df0 2023-10-30T12:01:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.047+0000 7fa399ca2640 10 monclient(hunting): authenticate will time out at 947.230991s 2023-10-30T12:01:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa394135df0 0x7fa39413da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa394135df0 0x7fa39413da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41202/0 (socket says 172.21.15.50:41202) 2023-10-30T12:01:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 -- 172.21.15.50:0/1106914268 learned_addr learned my addr 172.21.15.50:0/1106914268 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): get_auth_request con 0x7fa394135df0 auth_method 0 2023-10-30T12:01:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 1 --2- 172.21.15.50:0/1106914268 >> v2:172.21.15.118:3300/0 conn(0x7fa3940a2ae0 0x7fa3940a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 10 monclient(hunting): get_auth_request con 0x7fa3940a2ae0 auth_method 0 2023-10-30T12:01:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient(hunting): handle_auth_done global_id 34124 payload 274 2023-10-30T12:01:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 -- 172.21.15.50:0/1106914268 >> v2:172.21.15.118:3300/0 conn(0x7fa3940a2ae0 msgr2=0x7fa3940a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 --2- 172.21.15.50:0/1106914268 >> v2:172.21.15.118:3300/0 conn(0x7fa3940a2ae0 0x7fa3940a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 -- 172.21.15.50:0/1106914268 >> v1:172.21.15.118:6789/0 conn(0x7fa3940a2ec0 legacy=0x7fa3941356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient: _finish_hunting 0 2023-10-30T12:01:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 monclient: found mon.noname-b 2023-10-30T12:01:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.048+0000 7fa393fff640 1 -- 172.21.15.50:0/1106914268 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39409a2e0 con 0x7fa394135df0 2023-10-30T12:01:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa393fff640 1 --2- 172.21.15.50:0/1106914268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa394135df0 0x7fa39413da20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009e50 tx=0x7fa38c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8420b210ece754 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 1 -- 172.21.15.50:0/1106914268 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c02c9e0 con 0x7fa394135df0 2023-10-30T12:01:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: dump: 2023-10-30T12:01:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: _finish_auth 0 2023-10-30T12:01:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 1 -- 172.21.15.50:0/1106914268 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c02cb40 con 0x7fa394135df0 2023-10-30T12:01:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 1 -- 172.21.15.50:0/1106914268 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0343f0 con 0x7fa394135df0 2023-10-30T12:01:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 5 monclient: authenticate success, global_id 34124 2023-10-30T12:01:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 4 set_mon_vals no callback set 2023-10-30T12:01:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa3927fc640 10 monclient: dump: 2023-10-30T12:01:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.078 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 10 monclient: shutdown 2023-10-30T12:01:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 -- 172.21.15.50:0/1106914268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa394135df0 msgr2=0x7fa39413da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 --2- 172.21.15.50:0/1106914268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa394135df0 0x7fa39413da20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009e50 tx=0x7fa38c02be40 comp rx=0 tx=0).stop 2023-10-30T12:01:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 -- 172.21.15.50:0/1106914268 shutdown_connections 2023-10-30T12:01:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 --2- 172.21.15.50:0/1106914268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa394135df0 0x7fa39413da20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 --2- 172.21.15.50:0/1106914268 >> v2:172.21.15.118:3300/0 conn(0x7fa3940a2ae0 0x7fa3940a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 -- 172.21.15.50:0/1106914268 >> 172.21.15.50:0/1106914268 conn(0x7fa3940a80b0 msgr2=0x7fa3940a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 -- 172.21.15.50:0/1106914268 shutdown_connections 2023-10-30T12:01:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.049+0000 7fa399ca2640 1 -- 172.21.15.50:0/1106914268 wait complete. 2023-10-30T12:01:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 10 monclient: build_initial_monmap 2023-10-30T12:01:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 10 monclient: monmap: 2023-10-30T12:01:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:42.082 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 1 librados: starting msgr at 2023-10-30T12:01:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 1 librados: starting objecter 2023-10-30T12:01:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 1 Processor -- start 2023-10-30T12:01:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 1 -- start start 2023-10-30T12:01:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 1 librados: setting wanted keys 2023-10-30T12:01:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 1 librados: calling monclient init 2023-10-30T12:01:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.050+0000 7fa399ca2640 10 monclient: init 2023-10-30T12:01:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:01:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3940a2ae0 0x7fa3941ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): picked mon.noname-c con 0x7fa3940a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3940a2ec0 0x7fa3941cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): picked mon.noname-b con 0x7fa3940a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa394135df0 0x7fa39409d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): picked mon.noname-a con 0x7fa394135df0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39411e150 con 0x7fa394135df0 2023-10-30T12:01:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39409fa90 con 0x7fa3940a2ec0 2023-10-30T12:01:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39411f780 con 0x7fa3940a2ae0 2023-10-30T12:01:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa399ca2640 10 monclient(hunting): authenticate will time out at 947.234418s 2023-10-30T12:01:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3940a2ae0 0x7fa3941ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3940a2ae0 0x7fa3941ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41216/0 (socket says 172.21.15.50:41216) 2023-10-30T12:01:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 -- 172.21.15.50:0/3346565796 learned_addr learned my addr 172.21.15.50:0/3346565796 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): get_auth_request con 0x7fa3940a2ae0 auth_method 0 2023-10-30T12:01:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa394135df0 0x7fa39409d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 10 monclient(hunting): get_auth_request con 0x7fa394135df0 auth_method 0 2023-10-30T12:01:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3940a2ec0 0x7fa3941cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 10 monclient(hunting): get_auth_request con 0x7fa3940a2ec0 auth_method 0 2023-10-30T12:01:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient(hunting): handle_auth_done global_id 34127 payload 824 2023-10-30T12:01:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3940a2ec0 msgr2=0x7fa3941cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3940a2ec0 0x7fa3941cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa394135df0 msgr2=0x7fa39409d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa394135df0 0x7fa39409d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient: _finish_hunting 0 2023-10-30T12:01:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 monclient: found mon.noname-c 2023-10-30T12:01:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 -- 172.21.15.50:0/3346565796 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c009b30 con 0x7fa3940a2ae0 2023-10-30T12:01:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa3937fe640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3940a2ae0 0x7fa3941ca6d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa38400e9b0 tx=0x7fa38400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa37bfff640 10 client.?.objecter ms_handle_connect 0x7fa3940a2ae0 2023-10-30T12:01:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.051+0000 7fa37bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384009670 con 0x7fa3940a2ae0 2023-10-30T12:01:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: dump: 2023-10-30T12:01:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.097 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: _finish_auth 0 2023-10-30T12:01:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38400c9a0 con 0x7fa3940a2ae0 2023-10-30T12:01:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 5 monclient: authenticate success, global_id 34127 2023-10-30T12:01:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 10 monclient: _renew_subs 2023-10-30T12:01:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3940a1790 con 0x7fa3940a2ae0 2023-10-30T12:01:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38400cd50 con 0x7fa3940a2ae0 2023-10-30T12:01:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 10 monclient: dump: 2023-10-30T12:01:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.101 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa391ffb640 4 set_mon_vals no callback set 2023-10-30T12:01:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 10 client.34127.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 10 monclient: _renew_subs 2023-10-30T12:01:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39411e4b0 con 0x7fa3940a2ae0 2023-10-30T12:01:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 1 librados: init done 2023-10-30T12:01:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa384013470 con 0x7fa3940a2ae0 2023-10-30T12:01:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409ddb0 0x7fa39409e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.052+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa394073ff0 con 0x7fa39409ddb0 2023-10-30T12:01:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409ddb0 0x7fa39409e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3800392f0 0x7fa38003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient: get_auth_request con 0x7fa39409ddb0 auth_method 0 2023-10-30T12:01:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fa38404a760 con 0x7fa3940a2ae0 2023-10-30T12:01:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 client.34127.objecter ms_dispatch 0x7fa394000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 3 client.34127.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 3 client.34127.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 20 client.34127.objecter dump_active .. 0 homeless 2023-10-30T12:01:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa393fff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3800392f0 0x7fa38003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa393fff640 10 monclient: get_auth_request con 0x7fa3800392f0 auth_method 0 2023-10-30T12:01:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 10 monclient(hunting): handle_auth_done global_id 34130 payload 824 2023-10-30T12:01:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 1 -- 172.21.15.50:0/3346565796 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa39409a5c0 con 0x7fa39409ddb0 2023-10-30T12:01:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 client.34127.objecter ms_handle_connect 0x7fa39409ddb0 2023-10-30T12:01:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 client.34127.objecter resend_mon_ops 2023-10-30T12:01:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa392ffd640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409ddb0 0x7fa39409e190 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3940a0230 tx=0x7fa38800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3880050d0 con 0x7fa39409ddb0 2023-10-30T12:01:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 monclient: dump: 2023-10-30T12:01:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.110 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 client.34127.objecter ms_handle_connect 0x7fa3800392f0 2023-10-30T12:01:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa393fff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3800392f0 0x7fa38003b7b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa38c035000 tx=0x7fa38c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa388005230 con 0x7fa39409ddb0 2023-10-30T12:01:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409ddb0 msgr2=0x7fa39409e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409ddb0 0x7fa39409e190 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3940a0230 tx=0x7fa38800fc00 comp rx=0 tx=0).stop 2023-10-30T12:01:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409ddb0 msgr2=0x7fa39409e190 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.053+0000 7fa37bfff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409ddb0 0x7fa39409e190 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3940a0230 tx=0x7fa38800fc00 comp rx=0 tx=0).stop 2023-10-30T12:01:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa399ca2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa399ca2640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409e6d0 0x7fa39409ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa399ca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa394099b40 con 0x7fa39409e6d0 2023-10-30T12:01:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409e6d0 0x7fa39409ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient: get_auth_request con 0x7fa39409e6d0 auth_method 0 2023-10-30T12:01:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 10 monclient(hunting): handle_auth_done global_id 34133 payload 824 2023-10-30T12:01:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 1 -- 172.21.15.50:0/3346565796 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa39406b060 con 0x7fa39409e6d0 2023-10-30T12:01:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 10 client.34127.objecter ms_handle_connect 0x7fa39409e6d0 2023-10-30T12:01:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 10 client.34127.objecter resend_mon_ops 2023-10-30T12:01:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa392ffd640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409e6d0 0x7fa39409ad80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa388017000 tx=0x7fa388001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388024070 con 0x7fa39409e6d0 2023-10-30T12:01:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 10 monclient: dump: 2023-10-30T12:01:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.117 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fa3880038c0 con 0x7fa39409e6d0 2023-10-30T12:01:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409e6d0 msgr2=0x7fa39409ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409e6d0 0x7fa39409ad80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa388017000 tx=0x7fa388001120 comp rx=0 tx=0).stop 2023-10-30T12:01:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409e6d0 msgr2=0x7fa39409ad80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.060+0000 7fa37bfff640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa39409e6d0 0x7fa39409ad80 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa388017000 tx=0x7fa388001120 comp rx=0 tx=0).stop 2023-10-30T12:01:42.119 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:42.119 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:42.120 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:42.120 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:01:42.120 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 10, 2023-10-30T12:01:42.120 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:42.120 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:42.121 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:42.121 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:42.121 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.121 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:01:42.121 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:42.122 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:42.122 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:42.122 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:42.122 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:42.122 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:42.123 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:42.123 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:42.123 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:42.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:42.123 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:42.123 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:42.124 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:42.124 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.124 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:42.124 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:42.124 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:42.125 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:42.125 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:42.125 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:42.125 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:42.125 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:42.125 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:42.126 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:42.126 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:42.126 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:42.126 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.127 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:42.127 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:42.127 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:42.127 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:42.127 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:42.128 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:42.128 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:42.128 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:42.128 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:42.128 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:42.129 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:42.129 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:42.129 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:42.129 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:42.129 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:42.130 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:42.130 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:42.130 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:42.130 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:42.130 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:42.131 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:42.131 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:42.131 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:42.131 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:42.131 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:42.131 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:42.132 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:42.132 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.132 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:42.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.132 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:42.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.133 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:42.133 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:42.133 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:42.133 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:42.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:42.134 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:42.134 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:42.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:42.135 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.135 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.136 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:42.136 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:42.136 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:42.136 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:01:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:42.150 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:42.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 10 librados: watch_flush enter 2023-10-30T12:01:42.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 10 librados: watch_flush exit 2023-10-30T12:01:42.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 20 client.34127.objecter shutdown clearing up homeless session... 2023-10-30T12:01:42.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 10 client.34127.objecter successfully canceled tick 2023-10-30T12:01:42.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3800392f0 msgr2=0x7fa38003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3800392f0 0x7fa38003b7b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa38c035000 tx=0x7fa38c030000 comp rx=0 tx=0).stop 2023-10-30T12:01:42.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 10 monclient: shutdown 2023-10-30T12:01:42.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:42.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3940a2ae0 msgr2=0x7fa3941ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3940a2ae0 0x7fa3941ca6d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa38400e9b0 tx=0x7fa38400ee70 comp rx=0 tx=0).stop 2023-10-30T12:01:42.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa393fff640 1 -- 172.21.15.50:0/3346565796 reap_dead start 2023-10-30T12:01:42.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 shutdown_connections 2023-10-30T12:01:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 --2- 172.21.15.50:0/3346565796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3940a2ae0 0x7fa3941ca6d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 >> 172.21.15.50:0/3346565796 conn(0x7fa3940a80b0 msgr2=0x7fa3940a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 shutdown_connections 2023-10-30T12:01:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 -- 172.21.15.50:0/3346565796 wait complete. 2023-10-30T12:01:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.061+0000 7fa399ca2640 1 librados: shutdown 2023-10-30T12:01:42.156 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:01:42.156 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:01:42.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:01:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.261+0000 7f8f061f5640 10 monclient: get_monmap_and_config 2023-10-30T12:01:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.261+0000 7f8f061f5640 10 monclient: build_initial_monmap 2023-10-30T12:01:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.261+0000 7f8f061f5640 10 monclient: monmap: 2023-10-30T12:01:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:42.262373+0000 2023-10-30T12:01:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:42.262373+0000 2023-10-30T12:01:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:42.265 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 1 Processor -- start 2023-10-30T12:01:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 1 -- start start 2023-10-30T12:01:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient: init 2023-10-30T12:01:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:01:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f00071c00 0x7f8f0009a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient(hunting): picked mon.noname-a con 0x7f8f00071c00 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f00097fe0 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f0009b170 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f00097600 con 0x7f8f00097fe0 2023-10-30T12:01:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f000978e0 con 0x7f8f0009b170 2023-10-30T12:01:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0009bf50 con 0x7f8f00071c00 2023-10-30T12:01:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8f061f5640 10 monclient(hunting): authenticate will time out at 947.445535s 2023-10-30T12:01:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f00071c00 0x7f8f0009a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8efeffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f00097fe0 0x7f8f0009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55588/0 (socket says 172.21.15.50:55588) 2023-10-30T12:01:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8efeffd640 1 -- 172.21.15.50:0/3714559532 learned_addr learned my addr 172.21.15.50:0/3714559532 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): get_auth_request con 0x7f8f00071c00 auth_method 0 2023-10-30T12:01:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_done global_id 4515 payload 274 2023-10-30T12:01:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 1 -- 172.21.15.50:0/3714559532 >> v1:172.21.15.118:6789/0 conn(0x7f8f0009b170 legacy=0x7f8f00139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 1 -- 172.21.15.50:0/3714559532 >> v1:172.21.15.50:6789/0 conn(0x7f8f00097fe0 legacy=0x7f8f0009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:01:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 1 monclient: found mon.noname-a 2023-10-30T12:01:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:01:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.262+0000 7f8eff7fe640 1 -- 172.21.15.50:0/3714559532 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0009d9b0 con 0x7f8f00071c00 2023-10-30T12:01:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8eff7fe640 1 --2- 172.21.15.50:0/3714559532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f00071c00 0x7f8f0009a150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009790 tx=0x7f8ef002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e474714fa224a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 1 -- 172.21.15.50:0/3714559532 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0031780 con 0x7f8f00071c00 2023-10-30T12:01:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:01:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: dump: 2023-10-30T12:01:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.282 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:01:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 1 -- 172.21.15.50:0/3714559532 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef00318e0 con 0x7f8f00071c00 2023-10-30T12:01:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8f061f5640 5 monclient: authenticate success, global_id 4515 2023-10-30T12:01:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8f061f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8f061f5640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 1 -- 172.21.15.50:0/3714559532 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0031c70 con 0x7f8f00071c00 2023-10-30T12:01:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8f061f5640 4 set_mon_vals no callback set 2023-10-30T12:01:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:01:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.263+0000 7f8efe7fc640 10 monclient: dump: 2023-10-30T12:01:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 10 monclient: shutdown 2023-10-30T12:01:42.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:42.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 1 -- 172.21.15.50:0/3714559532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f00071c00 msgr2=0x7f8f0009a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 1 --2- 172.21.15.50:0/3714559532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f00071c00 0x7f8f0009a150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009790 tx=0x7f8ef002b820 comp rx=0 tx=0).stop 2023-10-30T12:01:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 1 -- 172.21.15.50:0/3714559532 shutdown_connections 2023-10-30T12:01:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 1 --2- 172.21.15.50:0/3714559532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f00071c00 0x7f8f0009a150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 1 -- 172.21.15.50:0/3714559532 >> 172.21.15.50:0/3714559532 conn(0x7f8f000a3980 msgr2=0x7f8f000a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 1 -- 172.21.15.50:0/3714559532 shutdown_connections 2023-10-30T12:01:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 1 -- 172.21.15.50:0/3714559532 wait complete. 2023-10-30T12:01:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 10 monclient: build_initial_monmap 2023-10-30T12:01:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.264+0000 7f8f061f5640 10 monclient: monmap: 2023-10-30T12:01:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:42.291 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 librados: starting msgr at 2023-10-30T12:01:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 librados: starting objecter 2023-10-30T12:01:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 Processor -- start 2023-10-30T12:01:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 -- start start 2023-10-30T12:01:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 librados: setting wanted keys 2023-10-30T12:01:42.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 librados: calling monclient init 2023-10-30T12:01:42.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient: init 2023-10-30T12:01:42.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:42.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:01:42.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f00071c00 0x7f8f001b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): picked mon.noname-b con 0x7f8f00071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f00097fe0 0x7f8f001c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): picked mon.noname-c con 0x7f8f00097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0009b170 0x7f8f001b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): picked mon.noname-a con 0x7f8f0009b170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001b0c10 con 0x7f8f0009b170 2023-10-30T12:01:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001c5910 con 0x7f8f00071c00 2023-10-30T12:01:42.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001b27d0 con 0x7f8f00097fe0 2023-10-30T12:01:42.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:42.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.265+0000 7f8f061f5640 10 monclient(hunting): authenticate will time out at 947.448949s 2023-10-30T12:01:42.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f00097fe0 0x7f8f001c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0009b170 0x7f8f001b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f00071c00 0x7f8f001b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f00097fe0 0x7f8f001c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41256/0 (socket says 172.21.15.50:41256) 2023-10-30T12:01:42.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 -- 172.21.15.50:0/3699418277 learned_addr learned my addr 172.21.15.50:0/3699418277 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:42.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): get_auth_request con 0x7f8f00097fe0 auth_method 0 2023-10-30T12:01:42.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 10 monclient(hunting): get_auth_request con 0x7f8f00071c00 auth_method 0 2023-10-30T12:01:42.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 10 monclient(hunting): get_auth_request con 0x7f8f0009b170 auth_method 0 2023-10-30T12:01:42.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient(hunting): handle_auth_done global_id 34136 payload 824 2023-10-30T12:01:42.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f00071c00 msgr2=0x7f8f001b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f00071c00 0x7f8f001b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0009b170 msgr2=0x7f8f001b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0009b170 0x7f8f001b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:01:42.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 monclient: found mon.noname-c 2023-10-30T12:01:42.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:42.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 -- 172.21.15.50:0/3699418277 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00093a0 con 0x7f8f00097fe0 2023-10-30T12:01:42.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efffff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0009b170 0x7f8f001b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:42.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efeffd640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f00097fe0 0x7f8f001c53d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400c4a0 tx=0x7f8ef400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 client.?.objecter ms_handle_connect 0x7f8f00097fe0 2023-10-30T12:01:42.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4005ce0 con 0x7f8f00097fe0 2023-10-30T12:01:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: dump: 2023-10-30T12:01:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.305 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: _finish_auth 0 2023-10-30T12:01:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:42.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:42.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef4005e40 con 0x7f8f00097fe0 2023-10-30T12:01:42.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 5 monclient: authenticate success, global_id 34136 2023-10-30T12:01:42.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 10 monclient: _renew_subs 2023-10-30T12:01:42.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f001b2df0 con 0x7f8f00097fe0 2023-10-30T12:01:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4007270 con 0x7f8f00097fe0 2023-10-30T12:01:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8ee7fff640 10 monclient: dump: 2023-10-30T12:01:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.309 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8efdffb640 4 set_mon_vals no callback set 2023-10-30T12:01:42.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 10 client.34136.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:42.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 10 monclient: _renew_subs 2023-10-30T12:01:42.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f001b1280 con 0x7f8f00097fe0 2023-10-30T12:01:42.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.266+0000 7f8f061f5640 1 librados: init done 2023-10-30T12:01:42.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8f061f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:42.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8f061f5640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001b80d0 0x7f8f001c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8f061f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f001b3210 con 0x7f8f001b80d0 2023-10-30T12:01:42.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001b80d0 0x7f8f001c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 10 monclient: get_auth_request con 0x7f8f001b80d0 auth_method 0 2023-10-30T12:01:42.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8ef4007490 con 0x7f8f00097fe0 2023-10-30T12:01:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.267+0000 7f8ee7fff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8eec0392f0 0x7f8eec03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f8ef4049980 con 0x7f8f00097fe0 2023-10-30T12:01:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 client.34136.objecter ms_dispatch 0x7f8f00000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_done global_id 4513 payload 824 2023-10-30T12:01:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 3 client.34136.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 3 client.34136.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8eff7fe640 1 -- 172.21.15.50:0/3699418277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f00071fe0 con 0x7f8f001b80d0 2023-10-30T12:01:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 20 client.34136.objecter dump_active .. 0 homeless 2023-10-30T12:01:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8efffff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8eec0392f0 0x7f8eec03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8efffff640 10 monclient: get_auth_request con 0x7f8eec0392f0 auth_method 0 2023-10-30T12:01:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 client.34136.objecter ms_handle_connect 0x7f8f001b80d0 2023-10-30T12:01:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 client.34136.objecter resend_mon_ops 2023-10-30T12:01:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8eff7fe640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001b80d0 0x7f8f001c8410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f0006bb30 tx=0x7f8ef0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef00329e0 con 0x7f8f001b80d0 2023-10-30T12:01:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 monclient: dump: 2023-10-30T12:01:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.318 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 client.34136.objecter ms_handle_connect 0x7f8eec0392f0 2023-10-30T12:01:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8efffff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8eec0392f0 0x7f8eec03b7b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800efd0 tx=0x7f8ef800e5a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8ef0032b40 con 0x7f8f001b80d0 2023-10-30T12:01:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001b80d0 msgr2=0x7f8f001c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001b80d0 0x7f8f001c8410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f0006bb30 tx=0x7f8ef0004730 comp rx=0 tx=0).stop 2023-10-30T12:01:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001b80d0 msgr2=0x7f8f001c8410 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.268+0000 7f8ee7fff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001b80d0 0x7f8f001c8410 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f0006bb30 tx=0x7f8ef0004730 comp rx=0 tx=0).stop 2023-10-30T12:01:42.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8f061f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:42.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8f061f5640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001c8950 0x7f8f001c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8f061f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f00073ff0 con 0x7f8f001c8950 2023-10-30T12:01:42.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001c8950 0x7f8f001c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient: get_auth_request con 0x7f8f001c8950 auth_method 0 2023-10-30T12:01:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 10 monclient(hunting): handle_auth_done global_id 4516 payload 824 2023-10-30T12:01:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 1 -- 172.21.15.50:0/3699418277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f001c92a0 con 0x7f8f001c8950 2023-10-30T12:01:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8ee7fff640 10 client.34136.objecter ms_handle_connect 0x7f8f001c8950 2023-10-30T12:01:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8ee7fff640 10 client.34136.objecter resend_mon_ops 2023-10-30T12:01:42.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.275+0000 7f8eff7fe640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001c8950 0x7f8f001c8d60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ef002b860 tx=0x7f8ef00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef004f670 con 0x7f8f001c8950 2023-10-30T12:01:42.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:42.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 10 monclient: dump: 2023-10-30T12:01:42.326 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.326 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.326 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.326 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.326 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.326 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.327 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.327 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.327 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.327 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f8ef004f7d0 con 0x7f8f001c8950 2023-10-30T12:01:42.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:42.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:42.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001c8950 msgr2=0x7f8f001c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001c8950 0x7f8f001c8d60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ef002b860 tx=0x7f8ef00046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001c8950 msgr2=0x7f8f001c8d60 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:42.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8ee7fff640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f001c8950 0x7f8f001c8d60 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ef002b860 tx=0x7f8ef00046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.329 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:42.329 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:42.329 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:42.329 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:01:42.330 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 10, 2023-10-30T12:01:42.330 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:42.330 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:01:42.330 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:42.330 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:42.331 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.331 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:01:42.331 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:42.331 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:42.331 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:42.332 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:42.332 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:42.332 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:42.332 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:42.332 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:42.333 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:42.333 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:42.333 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:42.333 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:42.333 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:42.333 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.334 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:42.334 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:42.334 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:42.334 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:42.334 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:42.335 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:42.335 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:42.335 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:42.335 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:42.335 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:42.336 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:42.336 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:42.336 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.336 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:42.336 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:42.337 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:42.337 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:42.337 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:42.337 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:42.337 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:42.338 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:42.338 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:42.338 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:42.338 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:42.338 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:01:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:01:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:01:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:42.361 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:42.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 10 librados: watch_flush enter 2023-10-30T12:01:42.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 10 librados: watch_flush exit 2023-10-30T12:01:42.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 20 client.34136.objecter shutdown clearing up homeless session... 2023-10-30T12:01:42.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 10 client.34136.objecter successfully canceled tick 2023-10-30T12:01:42.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8eec0392f0 msgr2=0x7f8eec03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8eec0392f0 0x7f8eec03b7b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800efd0 tx=0x7f8ef800e5a0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 10 monclient: shutdown 2023-10-30T12:01:42.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:42.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f00097fe0 msgr2=0x7f8f001c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 --2- 172.21.15.50:0/3699418277 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f00097fe0 0x7f8f001c53d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400c4a0 tx=0x7f8ef400c960 comp rx=0 tx=0).stop 2023-10-30T12:01:42.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8efffff640 1 -- 172.21.15.50:0/3699418277 reap_dead start 2023-10-30T12:01:42.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8efffff640 1 -- 172.21.15.50:0/3699418277 reap_dead start 2023-10-30T12:01:42.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 shutdown_connections 2023-10-30T12:01:42.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 >> 172.21.15.50:0/3699418277 conn(0x7f8f000a3980 msgr2=0x7f8f001b8620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:42.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 shutdown_connections 2023-10-30T12:01:42.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 -- 172.21.15.50:0/3699418277 wait complete. 2023-10-30T12:01:42.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.276+0000 7f8f061f5640 1 librados: shutdown 2023-10-30T12:01:42.366 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:01:42.367 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:01:42.367 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:01:42.367 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:01:42.367 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:01:42.368 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:01:42.368 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:01:42.378 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:01:42.379 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:01:42.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.521+0000 7fd09a139640 10 monclient: get_monmap_and_config 2023-10-30T12:01:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.521+0000 7fd09a139640 10 monclient: build_initial_monmap 2023-10-30T12:01:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.521+0000 7fd09a139640 10 monclient: monmap: 2023-10-30T12:01:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:42.522112+0000 2023-10-30T12:01:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:42.522112+0000 2023-10-30T12:01:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:42.525 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.521+0000 7fd09a139640 1 Processor -- start 2023-10-30T12:01:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.521+0000 7fd09a139640 1 -- start start 2023-10-30T12:01:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.521+0000 7fd09a139640 10 monclient: init 2023-10-30T12:01:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:01:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd0940a6070 0x7fd0940a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient(hunting): picked mon.noname-a con 0x7fd0940a6070 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0940a6450 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd094135ca0 0x7fd09413d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient(hunting): picked mon.noname-c con 0x7fd094135ca0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09409a270 con 0x7fd0940a6450 2023-10-30T12:01:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd094099f90 con 0x7fd0940a6070 2023-10-30T12:01:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09409a550 con 0x7fd094135ca0 2023-10-30T12:01:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd09a139640 10 monclient(hunting): authenticate will time out at 947.705345s 2023-10-30T12:01:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd0937fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fd0940a6070 msgr2=0x7fd0940a57e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd0937fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd0940a6070 0x7fd0940a57e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd092ffd640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fd0940a6450 legacy=0x7fd094135590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd094135ca0 0x7fd09413d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd094135ca0 0x7fd09413d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:44366/0 (socket says 172.21.15.50:44366) 2023-10-30T12:01:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 -- 172.21.15.50:0/4114664930 learned_addr learned my addr 172.21.15.50:0/4114664930 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): get_auth_request con 0x7fd094135ca0 auth_method 0 2023-10-30T12:01:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient(hunting): handle_auth_done global_id 4519 payload 274 2023-10-30T12:01:42.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 -- 172.21.15.50:0/4114664930 >> v2:172.21.15.50:3300/0 conn(0x7fd0940a6070 msgr2=0x7fd0940a57e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:42.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 --2- 172.21.15.50:0/4114664930 >> v2:172.21.15.50:3300/0 conn(0x7fd0940a6070 0x7fd0940a57e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 -- 172.21.15.50:0/4114664930 >> v1:172.21.15.50:6789/0 conn(0x7fd0940a6450 legacy=0x7fd094135590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:42.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient: _finish_hunting 0 2023-10-30T12:01:42.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 monclient: found mon.noname-c 2023-10-30T12:01:42.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:42.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:01:42.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.522+0000 7fd093fff640 1 -- 172.21.15.50:0/4114664930 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09409a830 con 0x7fd094135ca0 2023-10-30T12:01:42.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd093fff640 1 --2- 172.21.15.50:0/4114664930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd094135ca0 0x7fd09413d8d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd0940a4ee0 tx=0x7fd08c02bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dee8b38c0327ff9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 1 -- 172.21.15.50:0/4114664930 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c02c980 con 0x7fd094135ca0 2023-10-30T12:01:42.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:42.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: dump: 2023-10-30T12:01:42.541 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.541 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.541 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.541 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.542 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.542 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.542 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.542 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.542 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.543 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: _finish_auth 0 2023-10-30T12:01:42.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: _check_auth_tickets 2023-10-30T12:01:42.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:42.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 1 -- 172.21.15.50:0/4114664930 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c02cae0 con 0x7fd094135ca0 2023-10-30T12:01:42.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 1 -- 172.21.15.50:0/4114664930 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c02ce70 con 0x7fd094135ca0 2023-10-30T12:01:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd09a139640 5 monclient: authenticate success, global_id 4519 2023-10-30T12:01:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd09a139640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd09a139640 4 set_mon_vals no callback set 2023-10-30T12:01:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.523+0000 7fd0927fc640 10 monclient: dump: 2023-10-30T12:01:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.546 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.546 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.546 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.546 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.546 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.547 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 10 monclient: shutdown 2023-10-30T12:01:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 -- 172.21.15.50:0/4114664930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd094135ca0 msgr2=0x7fd09413d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 --2- 172.21.15.50:0/4114664930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd094135ca0 0x7fd09413d8d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd0940a4ee0 tx=0x7fd08c02bde0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 -- 172.21.15.50:0/4114664930 shutdown_connections 2023-10-30T12:01:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 --2- 172.21.15.50:0/4114664930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd094135ca0 0x7fd09413d8d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 --2- 172.21.15.50:0/4114664930 >> v2:172.21.15.50:3300/0 conn(0x7fd0940a6070 0x7fd0940a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 -- 172.21.15.50:0/4114664930 >> 172.21.15.50:0/4114664930 conn(0x7fd0940a81d0 msgr2=0x7fd0940a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 -- 172.21.15.50:0/4114664930 shutdown_connections 2023-10-30T12:01:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.524+0000 7fd09a139640 1 -- 172.21.15.50:0/4114664930 wait complete. 2023-10-30T12:01:42.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 10 monclient: build_initial_monmap 2023-10-30T12:01:42.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 10 monclient: monmap: 2023-10-30T12:01:42.549 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:42.549 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:42.549 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:42.550 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:42.550 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:42.550 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.550 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:42.550 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:42.550 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:42.551 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 librados: starting msgr at 2023-10-30T12:01:42.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 librados: starting objecter 2023-10-30T12:01:42.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 Processor -- start 2023-10-30T12:01:42.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 -- start start 2023-10-30T12:01:42.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 librados: setting wanted keys 2023-10-30T12:01:42.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 librados: calling monclient init 2023-10-30T12:01:42.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 10 monclient: init 2023-10-30T12:01:42.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:42.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:01:42.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0940a6070 0x7fd0941ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 10 monclient(hunting): picked mon.noname-c con 0x7fd0940a6070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:42.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0940a6450 0x7fd0941328f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 10 monclient(hunting): picked mon.noname-b con 0x7fd0940a6450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:42.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.525+0000 7fd09a139640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd094135ca0 0x7fd094132e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient(hunting): picked mon.noname-a con 0x7fd094135ca0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:42.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0941cb1b0 con 0x7fd094135ca0 2023-10-30T12:01:42.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0941229c0 con 0x7fd0940a6450 2023-10-30T12:01:42.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:42.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd094123820 con 0x7fd0940a6070 2023-10-30T12:01:42.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:42.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient(hunting): authenticate will time out at 947.709037s 2023-10-30T12:01:42.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd093fff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd094135ca0 msgr2=0x7fd094132e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:42.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd093fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd094135ca0 0x7fd094132e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:42.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0940a6070 0x7fd0941ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0940a6070 0x7fd0941ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41278/0 (socket says 172.21.15.50:41278) 2023-10-30T12:01:42.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 -- 172.21.15.50:0/1912200783 learned_addr learned my addr 172.21.15.50:0/1912200783 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:42.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): get_auth_request con 0x7fd0940a6070 auth_method 0 2023-10-30T12:01:42.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0940a6450 0x7fd0941328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 10 monclient(hunting): get_auth_request con 0x7fd0940a6450 auth_method 0 2023-10-30T12:01:42.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:42.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:42.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:42.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:42.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:42.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:42.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient(hunting): handle_auth_done global_id 34139 payload 824 2023-10-30T12:01:42.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 -- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0940a6450 msgr2=0x7fd0941328f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0940a6450 0x7fd0941328f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 -- 172.21.15.50:0/1912200783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd094135ca0 msgr2=0x7fd094132e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:42.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd094135ca0 0x7fd094132e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient: _finish_hunting 0 2023-10-30T12:01:42.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 monclient: found mon.noname-c 2023-10-30T12:01:42.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:42.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 -- 172.21.15.50:0/1912200783 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c009b30 con 0x7fd0940a6070 2023-10-30T12:01:42.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd0937fe640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0940a6070 0x7fd0941ca470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd08400b550 tx=0x7fd08400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 client.?.objecter ms_handle_connect 0x7fd0940a6070 2023-10-30T12:01:42.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:42.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 1 -- 172.21.15.50:0/1912200783 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084003dc0 con 0x7fd0940a6070 2023-10-30T12:01:42.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:42.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: dump: 2023-10-30T12:01:42.562 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.562 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.563 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.563 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.563 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.563 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.563 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.563 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.564 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.564 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: _finish_auth 0 2023-10-30T12:01:42.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:42.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:42.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 5 monclient: authenticate success, global_id 34139 2023-10-30T12:01:42.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 1 -- 172.21.15.50:0/1912200783 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd084005070 con 0x7fd0940a6070 2023-10-30T12:01:42.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:42.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 1 -- 172.21.15.50:0/1912200783 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084005420 con 0x7fd0940a6070 2023-10-30T12:01:42.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:42.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:42.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd07bfff640 10 monclient: dump: 2023-10-30T12:01:42.566 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:42.566 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:42.566 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.566 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:42.567 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:42.567 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:42.567 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:42.567 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:42.567 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:42.568 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd091ffb640 4 set_mon_vals no callback set 2023-10-30T12:01:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient: _renew_subs 2023-10-30T12:01:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd094122270 con 0x7fd0940a6070 2023-10-30T12:01:42.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 client.34139.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient: _renew_subs 2023-10-30T12:01:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd094135600 con 0x7fd0940a6070 2023-10-30T12:01:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.526+0000 7fd09a139640 1 librados: init done 2023-10-30T12:01:42.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd07bfff640 1 -- 172.21.15.50:0/1912200783 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd084005580 con 0x7fd0940a6070 2023-10-30T12:01:42.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd09a139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:42.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd07bfff640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0800392a0 0x7fd08003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:42.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd09a139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:42.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd09a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0940a68e0 con 0x7fd0940a6070 2023-10-30T12:01:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd07bfff640 1 -- 172.21.15.50:0/1912200783 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fd08404a360 con 0x7fd0940a6070 2023-10-30T12:01:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd07bfff640 10 client.34139.objecter ms_dispatch 0x7fd094000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd07bfff640 3 client.34139.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd07bfff640 3 client.34139.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:42.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.527+0000 7fd07bfff640 20 client.34139.objecter dump_active .. 0 homeless 2023-10-30T12:01:42.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.528+0000 7fd092ffd640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0800392a0 0x7fd08003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:42.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.528+0000 7fd092ffd640 10 monclient: get_auth_request con 0x7fd0800392a0 auth_method 0 2023-10-30T12:01:42.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.528+0000 7fd07bfff640 10 client.34139.objecter ms_handle_connect 0x7fd0800392a0 2023-10-30T12:01:42.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.528+0000 7fd092ffd640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0800392a0 0x7fd08003b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd094123050 tx=0x7fd088003b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.532+0000 7fd07bfff640 1 -- 172.21.15.50:0/1912200783 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd0940a68e0 con 0x7fd0940a6070 2023-10-30T12:01:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.532+0000 7fd07bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:42.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.532+0000 7fd07bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:42.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.776+0000 7fd09a139640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.776+0000 7fd09a139640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.776+0000 7fd09a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.776+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd09404e160 con 0x7fd0940a6070 2023-10-30T12:01:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.777+0000 7fd07bfff640 1 -- 172.21.15.50:0/1912200783 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fd08403c070 con 0x7fd0940a6070 2023-10-30T12:01:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.777+0000 7fd07bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.777+0000 7fd07bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:42.780 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.778+0000 7fd09a139640 10 librados: watch_flush enter 2023-10-30T12:01:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.778+0000 7fd09a139640 10 librados: watch_flush exit 2023-10-30T12:01:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.778+0000 7fd09a139640 20 client.34139.objecter shutdown clearing up homeless session... 2023-10-30T12:01:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.778+0000 7fd09a139640 10 client.34139.objecter successfully canceled tick 2023-10-30T12:01:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0800392a0 msgr2=0x7fd08003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0800392a0 0x7fd08003b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd094123050 tx=0x7fd088003b90 comp rx=0 tx=0).stop 2023-10-30T12:01:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 10 monclient: shutdown 2023-10-30T12:01:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0940a6070 msgr2=0x7fd0941ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0940a6070 0x7fd0941ca470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd08400b550 tx=0x7fd08400ba10 comp rx=0 tx=0).stop 2023-10-30T12:01:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 shutdown_connections 2023-10-30T12:01:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0800392a0 0x7fd08003b760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd094135ca0 0x7fd094132e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0940a6450 0x7fd0941328f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 --2- 172.21.15.50:0/1912200783 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0940a6070 0x7fd0941ca470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 >> 172.21.15.50:0/1912200783 conn(0x7fd0940a81d0 msgr2=0x7fd0940a32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 shutdown_connections 2023-10-30T12:01:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 -- 172.21.15.50:0/1912200783 wait complete. 2023-10-30T12:01:42.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:42.779+0000 7fd09a139640 1 librados: shutdown 2023-10-30T12:01:45.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:45.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.907+0000 7fd6a751d640 10 monclient: get_monmap_and_config 2023-10-30T12:01:45.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.907+0000 7fd6a751d640 10 monclient: build_initial_monmap 2023-10-30T12:01:45.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.907+0000 7fd6a751d640 10 monclient: monmap: 2023-10-30T12:01:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:45.908350+0000 2023-10-30T12:01:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:45.908350+0000 2023-10-30T12:01:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:45.912 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:45.912 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:45.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.907+0000 7fd6a751d640 1 Processor -- start 2023-10-30T12:01:45.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 1 -- start start 2023-10-30T12:01:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient: init 2023-10-30T12:01:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T12:01:45.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6a00a2dd0 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a006a8e0 0x7fd6a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a006a8e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6a0071400 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a00a31b0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a006a6a0 con 0x7fd6a0071400 2023-10-30T12:01:45.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a006be90 con 0x7fd6a006a8e0 2023-10-30T12:01:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a751d640 10 monclient(hunting): authenticate will time out at 951.091619s 2023-10-30T12:01:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a52b9640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fd6a00a2dd0 legacy=0x7fd6a00a46c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a006a8e0 0x7fd6a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a006a8e0 0x7fd6a00a4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41294/0 (socket says 172.21.15.50:41294) 2023-10-30T12:01:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/626502714 learned_addr learned my addr 172.21.15.50:0/626502714 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 10 monclient(hunting): get_auth_request con 0x7fd6a006a8e0 auth_method 0 2023-10-30T12:01:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:45.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:45.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:45.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:45.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:45.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.908+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:45.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_done global_id 34142 payload 274 2023-10-30T12:01:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/626502714 >> v1:172.21.15.118:6789/0 conn(0x7fd6a0071400 legacy=0x7fd6a013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/626502714 >> v1:172.21.15.50:6789/0 conn(0x7fd6a00a2dd0 legacy=0x7fd6a00a46c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 10 monclient: _finish_hunting 0 2023-10-30T12:01:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 1 monclient: found mon.noname-b 2023-10-30T12:01:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:01:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/626502714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a009e190 con 0x7fd6a006a8e0 2023-10-30T12:01:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd6a4ab8640 1 --2- 172.21.15.50:0/626502714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a006a8e0 0x7fd6a00a4dd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6940098a0 tx=0x7fd69402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c36642104f60a18b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd69ffff640 1 -- 172.21.15.50:0/626502714 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6940317c0 con 0x7fd6a006a8e0 2023-10-30T12:01:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd69ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd69ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd69ffff640 10 monclient: dump: 2023-10-30T12:01:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:45.928 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd69ffff640 10 monclient: _finish_auth 0 2023-10-30T12:01:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.909+0000 7fd69ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd69ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 5 monclient: authenticate success, global_id 34142 2023-10-30T12:01:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd69ffff640 1 -- 172.21.15.50:0/626502714 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd694031920 con 0x7fd6a006a8e0 2023-10-30T12:01:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd69ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 4 set_mon_vals no callback set 2023-10-30T12:01:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd69ffff640 1 -- 172.21.15.50:0/626502714 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694031cb0 con 0x7fd6a006a8e0 2023-10-30T12:01:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd69ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd69ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd69ffff640 10 monclient: dump: 2023-10-30T12:01:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:45.931 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:45.931 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.931 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.931 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:45.931 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:45.932 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 10 monclient: shutdown 2023-10-30T12:01:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:45.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 1 -- 172.21.15.50:0/626502714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a006a8e0 msgr2=0x7fd6a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:45.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 1 --2- 172.21.15.50:0/626502714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a006a8e0 0x7fd6a00a4dd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6940098a0 tx=0x7fd69402b860 comp rx=0 tx=0).stop 2023-10-30T12:01:45.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 1 -- 172.21.15.50:0/626502714 shutdown_connections 2023-10-30T12:01:45.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 1 --2- 172.21.15.50:0/626502714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a006a8e0 0x7fd6a00a4dd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:45.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 1 -- 172.21.15.50:0/626502714 >> 172.21.15.50:0/626502714 conn(0x7fd6a00a22b0 msgr2=0x7fd6a00a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:45.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 1 -- 172.21.15.50:0/626502714 shutdown_connections 2023-10-30T12:01:45.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.910+0000 7fd6a751d640 1 -- 172.21.15.50:0/626502714 wait complete. 2023-10-30T12:01:45.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 10 monclient: build_initial_monmap 2023-10-30T12:01:45.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 10 monclient: monmap: 2023-10-30T12:01:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:45.937 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 1 librados: starting msgr at 2023-10-30T12:01:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 1 librados: starting objecter 2023-10-30T12:01:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 1 Processor -- start 2023-10-30T12:01:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 1 -- start start 2023-10-30T12:01:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 1 librados: setting wanted keys 2023-10-30T12:01:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 1 librados: calling monclient init 2023-10-30T12:01:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.911+0000 7fd6a751d640 10 monclient: init 2023-10-30T12:01:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:01:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a0071400 0x7fd6a01ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a0071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a00a2dd0 0x7fd6a01cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a00a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a011e3c0 0x7fd6a011e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a52b9640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a0071400 msgr2=0x7fd6a01ca780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a011e3c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0120a20 con 0x7fd6a0071400 2023-10-30T12:01:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a52b9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a0071400 0x7fd6a01ca780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a006b0a0 con 0x7fd6a011e3c0 2023-10-30T12:01:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a00708a0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient(hunting): authenticate will time out at 951.095164s 2023-10-30T12:01:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a00a2dd0 0x7fd6a01cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a00a2dd0 0x7fd6a01cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41310/0 (socket says 172.21.15.50:41310) 2023-10-30T12:01:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/2818416460 learned_addr learned my addr 172.21.15.50:0/2818416460 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): get_auth_request con 0x7fd6a00a2dd0 auth_method 0 2023-10-30T12:01:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:45.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:45.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a011e3c0 0x7fd6a011e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:45.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 10 monclient(hunting): get_auth_request con 0x7fd6a011e3c0 auth_method 0 2023-10-30T12:01:45.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:45.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:45.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:45.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a5aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:45.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient(hunting): handle_auth_done global_id 34145 payload 824 2023-10-30T12:01:45.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a011e3c0 msgr2=0x7fd6a011e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:45.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a011e3c0 0x7fd6a011e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:45.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/2818416460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a0071400 msgr2=0x7fd6a01ca780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a0071400 0x7fd6a01ca780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient: _finish_hunting 0 2023-10-30T12:01:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 monclient: found mon.noname-c 2023-10-30T12:01:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:45.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 -- 172.21.15.50:0/2818416460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940094d0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a4ab8640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a00a2dd0 0x7fd6a01cacc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd694000c00 tx=0x7fd6940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:45.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 client.?.objecter ms_handle_connect 0x7fd6a00a2dd0 2023-10-30T12:01:45.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:45.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 1 -- 172.21.15.50:0/2818416460 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6940329e0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: dump: 2023-10-30T12:01:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:45.950 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:45.950 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:45.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: _finish_auth 0 2023-10-30T12:01:45.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:01:45.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:45.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 1 -- 172.21.15.50:0/2818416460 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd694032b40 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 5 monclient: authenticate success, global_id 34145 2023-10-30T12:01:45.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:45.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 1 -- 172.21.15.50:0/2818416460 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6940453f0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient: _renew_subs 2023-10-30T12:01:45.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:45.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a0122430 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:45.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:45.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69dffb640 10 monclient: dump: 2023-10-30T12:01:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:45.954 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.912+0000 7fd69f7fe640 4 set_mon_vals no callback set 2023-10-30T12:01:45.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.913+0000 7fd6a751d640 10 client.34145.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:45.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.913+0000 7fd6a751d640 10 monclient: _renew_subs 2023-10-30T12:01:45.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.913+0000 7fd6a751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:45.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.913+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a0120060 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.913+0000 7fd6a751d640 1 librados: init done 2023-10-30T12:01:45.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.913+0000 7fd69dffb640 1 -- 172.21.15.50:0/2818416460 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd69403b3f0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd6a751d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:45.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd6a751d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:45.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd6a751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:45.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a004e160 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd69dffb640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd68c0392a0 0x7fd68c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd69dffb640 1 -- 172.21.15.50:0/2818416460 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fd69406cac0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd69dffb640 10 client.34145.objecter ms_dispatch 0x7fd6a0000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd69dffb640 3 client.34145.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd69dffb640 3 client.34145.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd69dffb640 20 client.34145.objecter dump_active .. 0 homeless 2023-10-30T12:01:45.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd6a52b9640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd68c0392a0 0x7fd68c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:45.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd6a52b9640 10 monclient: get_auth_request con 0x7fd68c0392a0 auth_method 0 2023-10-30T12:01:45.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd69dffb640 10 client.34145.objecter ms_handle_connect 0x7fd68c0392a0 2023-10-30T12:01:45.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.914+0000 7fd6a52b9640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd68c0392a0 0x7fd68c03b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd698006fd0 tx=0x7fd698006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.918+0000 7fd69dffb640 1 -- 172.21.15.50:0/2818416460 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd6940396d0 con 0x7fd6a00a2dd0 2023-10-30T12:01:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.918+0000 7fd69dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:45.918+0000 7fd69dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:46.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.164+0000 7fd6a751d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.164+0000 7fd6a751d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.164+0000 7fd6a751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.164+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd6a00745c0 con 0x7fd6a00a2dd0 2023-10-30T12:01:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.165+0000 7fd69dffb640 1 -- 172.21.15.50:0/2818416460 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fd69402d8b0 con 0x7fd6a00a2dd0 2023-10-30T12:01:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.165+0000 7fd69dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:46.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.165+0000 7fd69dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:46.168 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 10 librados: watch_flush enter 2023-10-30T12:01:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 10 librados: watch_flush exit 2023-10-30T12:01:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 20 client.34145.objecter shutdown clearing up homeless session... 2023-10-30T12:01:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 10 client.34145.objecter successfully canceled tick 2023-10-30T12:01:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd68c0392a0 msgr2=0x7fd68c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd68c0392a0 0x7fd68c03b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd698006fd0 tx=0x7fd698006d80 comp rx=0 tx=0).stop 2023-10-30T12:01:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 10 monclient: shutdown 2023-10-30T12:01:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a00a2dd0 msgr2=0x7fd6a01cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a00a2dd0 0x7fd6a01cacc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd694000c00 tx=0x7fd6940046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 shutdown_connections 2023-10-30T12:01:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd68c0392a0 0x7fd68c03b760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a011e3c0 0x7fd6a011e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a00a2dd0 0x7fd6a01cacc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 --2- 172.21.15.50:0/2818416460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a0071400 0x7fd6a01ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 >> 172.21.15.50:0/2818416460 conn(0x7fd6a00a22b0 msgr2=0x7fd6a00a0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 shutdown_connections 2023-10-30T12:01:46.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 -- 172.21.15.50:0/2818416460 wait complete. 2023-10-30T12:01:46.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:46.167+0000 7fd6a751d640 1 librados: shutdown 2023-10-30T12:01:49.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.295+0000 7ffbf38d4640 10 monclient: get_monmap_and_config 2023-10-30T12:01:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.295+0000 7ffbf38d4640 10 monclient: build_initial_monmap 2023-10-30T12:01:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.295+0000 7ffbf38d4640 10 monclient: monmap: 2023-10-30T12:01:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:49.296183+0000 2023-10-30T12:01:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:49.296183+0000 2023-10-30T12:01:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:49.300 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.295+0000 7ffbf38d4640 1 Processor -- start 2023-10-30T12:01:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.295+0000 7ffbf38d4640 1 -- start start 2023-10-30T12:01:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.295+0000 7ffbf38d4640 10 monclient: init 2023-10-30T12:01:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:01:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbec0a6070 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbec0a57e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffbec13bfd0 0x7ffbec13e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient(hunting): picked mon.noname-c con 0x7ffbec13bfd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec09a270 con 0x7ffbec0a57e0 2023-10-30T12:01:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec099f90 con 0x7ffbec0a6070 2023-10-30T12:01:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec09a550 con 0x7ffbec13bfd0 2023-10-30T12:01:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:49.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf38d4640 10 monclient(hunting): authenticate will time out at 954.479480s 2023-10-30T12:01:49.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf0e6f640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7ffbec0a57e0 legacy=0x7ffbec13a8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:49.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffbec13bfd0 0x7ffbec13e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:49.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1670640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7ffbec0a6070 0x7ffbec0a6450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:41758/0 (socket says 172.21.15.50:41758) 2023-10-30T12:01:49.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1670640 1 -- 172.21.15.50:0/369634873 learned_addr learned my addr 172.21.15.50:0/369634873 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:49.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 10 monclient(hunting): get_auth_request con 0x7ffbec13bfd0 auth_method 0 2023-10-30T12:01:49.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:49.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:49.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:49.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.296+0000 7ffbf1e71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 10 monclient(hunting): handle_auth_done global_id 4531 payload 274 2023-10-30T12:01:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 1 -- 172.21.15.50:0/369634873 >> v1:172.21.15.118:6789/0 conn(0x7ffbec0a6070 legacy=0x7ffbec0a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 1 -- 172.21.15.50:0/369634873 >> v1:172.21.15.50:6789/0 conn(0x7ffbec0a57e0 legacy=0x7ffbec13a8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 10 monclient: _finish_hunting 0 2023-10-30T12:01:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 1 monclient: found mon.noname-c 2023-10-30T12:01:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:01:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 1 -- 172.21.15.50:0/369634873 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec09a830 con 0x7ffbec13bfd0 2023-10-30T12:01:49.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.297+0000 7ffbf1e71640 1 --2- 172.21.15.50:0/369634873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 0x7ffbec13e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffbec0a4ee0 tx=0x7ffbe402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48f5191882b21c04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:49.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 1 -- 172.21.15.50:0/369634873 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe402cb30 con 0x7ffbec13bfd0 2023-10-30T12:01:49.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:49.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:49.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: dump: 2023-10-30T12:01:49.314 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:49.315 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:49.315 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.315 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.315 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:49.316 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:49.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: _finish_auth 0 2023-10-30T12:01:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 1 -- 172.21.15.50:0/369634873 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe402cc90 con 0x7ffbec13bfd0 2023-10-30T12:01:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 5 monclient: authenticate success, global_id 4531 2023-10-30T12:01:49.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 1 -- 172.21.15.50:0/369634873 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4034440 con 0x7ffbec13bfd0 2023-10-30T12:01:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 4 set_mon_vals no callback set 2023-10-30T12:01:49.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:49.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:49.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbe3fff640 10 monclient: dump: 2023-10-30T12:01:49.319 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:49.319 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:49.319 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.319 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.320 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:49.320 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:49.320 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:49.320 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:49.320 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:49.320 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:49.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 10 monclient: shutdown 2023-10-30T12:01:49.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:49.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 1 -- 172.21.15.50:0/369634873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 msgr2=0x7ffbec13e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:49.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/369634873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 0x7ffbec13e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffbec0a4ee0 tx=0x7ffbe402bd80 comp rx=0 tx=0).stop 2023-10-30T12:01:49.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 1 -- 172.21.15.50:0/369634873 shutdown_connections 2023-10-30T12:01:49.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/369634873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 0x7ffbec13e3c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:49.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 1 -- 172.21.15.50:0/369634873 >> 172.21.15.50:0/369634873 conn(0x7ffbec0a81d0 msgr2=0x7ffbec0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:49.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 1 -- 172.21.15.50:0/369634873 shutdown_connections 2023-10-30T12:01:49.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.298+0000 7ffbf38d4640 1 -- 172.21.15.50:0/369634873 wait complete. 2023-10-30T12:01:49.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 10 monclient: build_initial_monmap 2023-10-30T12:01:49.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 10 monclient: monmap: 2023-10-30T12:01:49.323 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:49.323 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:49.323 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:49.324 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:49.324 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:49.324 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:49.324 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:49.324 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:49.325 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:49.325 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:49.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 1 librados: starting msgr at 2023-10-30T12:01:49.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 1 librados: starting objecter 2023-10-30T12:01:49.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 1 Processor -- start 2023-10-30T12:01:49.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 1 -- start start 2023-10-30T12:01:49.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 1 librados: setting wanted keys 2023-10-30T12:01:49.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 1 librados: calling monclient init 2023-10-30T12:01:49.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 10 monclient: init 2023-10-30T12:01:49.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.299+0000 7ffbf38d4640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:49.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:01:49.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbec0a57e0 0x7ffbec1ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:49.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): picked mon.noname-c con 0x7ffbec0a57e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:49.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbec0a6070 0x7ffbec121a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:49.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): picked mon.noname-a con 0x7ffbec0a6070 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:49.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 0x7ffbec121fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:49.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): picked mon.noname-b con 0x7ffbec13bfd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:49.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:49.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1258a0 con 0x7ffbec0a6070 2023-10-30T12:01:49.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:49.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec125af0 con 0x7ffbec13bfd0 2023-10-30T12:01:49.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:49.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec13b6a0 con 0x7ffbec0a57e0 2023-10-30T12:01:49.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:49.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient(hunting): authenticate will time out at 954.483068s 2023-10-30T12:01:49.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf0e6f640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbec0a6070 msgr2=0x7ffbec121a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:49.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf0e6f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbec0a6070 0x7ffbec121a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:49.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbec0a57e0 0x7ffbec1ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:49.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbec0a57e0 0x7ffbec1ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56686/0 (socket says 172.21.15.50:56686) 2023-10-30T12:01:49.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 -- 172.21.15.50:0/2826891156 learned_addr learned my addr 172.21.15.50:0/2826891156 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:49.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): get_auth_request con 0x7ffbec0a57e0 auth_method 0 2023-10-30T12:01:49.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 0x7ffbec121fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:49.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:49.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:49.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:49.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 10 monclient(hunting): get_auth_request con 0x7ffbec13bfd0 auth_method 0 2023-10-30T12:01:49.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:49.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:49.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:49.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:49.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:49.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:49.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1e71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient(hunting): handle_auth_done global_id 34148 payload 824 2023-10-30T12:01:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 -- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 msgr2=0x7ffbec121fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 0x7ffbec121fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 -- 172.21.15.50:0/2826891156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbec0a6070 msgr2=0x7ffbec121a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbec0a6070 0x7ffbec121a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient: _finish_hunting 0 2023-10-30T12:01:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 monclient: found mon.noname-c 2023-10-30T12:01:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:49.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 -- 172.21.15.50:0/2826891156 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe4009a70 con 0x7ffbec0a57e0 2023-10-30T12:01:49.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf1670640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbec0a57e0 0x7ffbec1ca740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800c460 tx=0x7ffbe800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:49.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 client.?.objecter ms_handle_connect 0x7ffbec0a57e0 2023-10-30T12:01:49.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:49.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 1 -- 172.21.15.50:0/2826891156 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe8005ce0 con 0x7ffbec0a57e0 2023-10-30T12:01:49.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:49.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:49.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: dump: 2023-10-30T12:01:49.336 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:49.336 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:49.336 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.336 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.337 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:49.337 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:49.337 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:49.337 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:49.337 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:49.338 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:49.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:01:49.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:01:49.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:49.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 5 monclient: authenticate success, global_id 34148 2023-10-30T12:01:49.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 1 -- 172.21.15.50:0/2826891156 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe8005e40 con 0x7ffbec0a57e0 2023-10-30T12:01:49.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient: _renew_subs 2023-10-30T12:01:49.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:49.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec0a1690 con 0x7ffbec0a57e0 2023-10-30T12:01:49.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:49.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 1 -- 172.21.15.50:0/2826891156 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe8007290 con 0x7ffbec0a57e0 2023-10-30T12:01:49.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:49.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:49.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe1ffb640 10 monclient: dump: 2023-10-30T12:01:49.340 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:49.340 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:49.341 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.341 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:49.341 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:49.341 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:49.341 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:49.342 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:49.342 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:49.342 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:49.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.300+0000 7ffbe37fe640 4 set_mon_vals no callback set 2023-10-30T12:01:49.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 10 client.34148.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:49.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 10 monclient: _renew_subs 2023-10-30T12:01:49.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:49.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec123720 con 0x7ffbec0a57e0 2023-10-30T12:01:49.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 1 librados: init done 2023-10-30T12:01:49.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbe1ffb640 1 -- 172.21.15.50:0/2826891156 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ffbe80073f0 con 0x7ffbec0a57e0 2023-10-30T12:01:49.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:49.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:49.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:49.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbe1ffb640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd00392a0 0x7ffbd003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:49.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.301+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec04e160 con 0x7ffbec0a57e0 2023-10-30T12:01:49.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbe1ffb640 1 -- 172.21.15.50:0/2826891156 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7ffbe8049930 con 0x7ffbec0a57e0 2023-10-30T12:01:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbe1ffb640 10 client.34148.objecter ms_dispatch 0x7ffbec000b90 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbe1ffb640 3 client.34148.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbe1ffb640 3 client.34148.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbf0e6f640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd00392a0 0x7ffbd003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbe1ffb640 20 client.34148.objecter dump_active .. 0 homeless 2023-10-30T12:01:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbf0e6f640 10 monclient: get_auth_request con 0x7ffbd00392a0 auth_method 0 2023-10-30T12:01:49.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbf0e6f640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd00392a0 0x7ffbd003b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc006fd0 tx=0x7ffbdc006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:49.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.302+0000 7ffbe1ffb640 10 client.34148.objecter ms_handle_connect 0x7ffbd00392a0 2023-10-30T12:01:49.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.306+0000 7ffbe1ffb640 1 -- 172.21.15.50:0/2826891156 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffbe8017080 con 0x7ffbec0a57e0 2023-10-30T12:01:49.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.306+0000 7ffbe1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:49.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.306+0000 7ffbe1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:49.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.551+0000 7ffbf38d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.551+0000 7ffbf38d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.551+0000 7ffbf38d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.551+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffbec125c30 con 0x7ffbec0a57e0 2023-10-30T12:01:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.552+0000 7ffbe1ffb640 1 -- 172.21.15.50:0/2826891156 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7ffbec125c30 con 0x7ffbec0a57e0 2023-10-30T12:01:49.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.552+0000 7ffbe1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:49.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.552+0000 7ffbe1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:49.555 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.553+0000 7ffbf38d4640 10 librados: watch_flush enter 2023-10-30T12:01:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.553+0000 7ffbf38d4640 10 librados: watch_flush exit 2023-10-30T12:01:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.553+0000 7ffbf38d4640 20 client.34148.objecter shutdown clearing up homeless session... 2023-10-30T12:01:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.553+0000 7ffbf38d4640 10 client.34148.objecter successfully canceled tick 2023-10-30T12:01:49.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd00392a0 msgr2=0x7ffbd003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:49.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd00392a0 0x7ffbd003b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc006fd0 tx=0x7ffbdc006d80 comp rx=0 tx=0).stop 2023-10-30T12:01:49.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 10 monclient: shutdown 2023-10-30T12:01:49.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:49.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbec0a57e0 msgr2=0x7ffbec1ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:49.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbec0a57e0 0x7ffbec1ca740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800c460 tx=0x7ffbe800c920 comp rx=0 tx=0).stop 2023-10-30T12:01:49.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 shutdown_connections 2023-10-30T12:01:49.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd00392a0 0x7ffbd003b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:49.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbec13bfd0 0x7ffbec121fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:49.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbec0a6070 0x7ffbec121a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 --2- 172.21.15.50:0/2826891156 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbec0a57e0 0x7ffbec1ca740 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 >> 172.21.15.50:0/2826891156 conn(0x7ffbec0a81d0 msgr2=0x7ffbec0a32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 shutdown_connections 2023-10-30T12:01:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 -- 172.21.15.50:0/2826891156 wait complete. 2023-10-30T12:01:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:49.554+0000 7ffbf38d4640 1 librados: shutdown 2023-10-30T12:01:52.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:01:52.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.682+0000 7f0a6eca2640 10 monclient: get_monmap_and_config 2023-10-30T12:01:52.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.682+0000 7f0a6eca2640 10 monclient: build_initial_monmap 2023-10-30T12:01:52.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.682+0000 7f0a6eca2640 10 monclient: monmap: 2023-10-30T12:01:52.684 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:52.684 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:52.685 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:52.683153+0000 2023-10-30T12:01:52.685 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:52.683153+0000 2023-10-30T12:01:52.685 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:52.685 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:52.685 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:52.685 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:52.686 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:52.686 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:52.686 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:52.686 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:52.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.682+0000 7f0a6eca2640 1 Processor -- start 2023-10-30T12:01:52.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.682+0000 7f0a6eca2640 1 -- start start 2023-10-30T12:01:52.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.682+0000 7f0a6eca2640 10 monclient: init 2023-10-30T12:01:52.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:52.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-10-30T12:01:52.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a680a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:52.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 0x7f0a6813aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:52.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient(hunting): picked mon.noname-a con 0x7f0a68099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:52.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a68132db0 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:52.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680a4f40 con 0x7f0a68132db0 2023-10-30T12:01:52.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680a4150 con 0x7f0a680a4b60 2023-10-30T12:01:52.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:52.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a680a3d70 con 0x7f0a68099a40 2023-10-30T12:01:52.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:52.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6eca2640 10 monclient(hunting): authenticate will time out at 957.866505s 2023-10-30T12:01:52.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a67fff640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 msgr2=0x7f0a6813aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:52.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6d23f640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f0a68132db0 legacy=0x7f0a6813cfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:52.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a67fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 0x7f0a6813aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:52.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6ca3e640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f0a680a4b60 0x7f0a681326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:41774/0 (socket says 172.21.15.50:41774) 2023-10-30T12:01:52.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.683+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 learned_addr learned my addr 172.21.15.50:0/3360253947 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:52.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.884+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 msgr2=0x7f0a6813aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:52.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.884+0000 7f0a6d23f640 1 -- 172.21.15.50:0/3360253947 >> v1:172.21.15.50:6789/0 conn(0x7f0a68132db0 legacy=0x7f0a6813cfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:52.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:52.884+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 0x7f0a6813aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:52.929 INFO:tasks.workunit.client.0.smithi118.stderr:Note: switching to '397e85a5f8ee6b984077e94184404370497237a9'. 2023-10-30T12:01:52.929 INFO:tasks.workunit.client.0.smithi118.stderr: 2023-10-30T12:01:52.929 INFO:tasks.workunit.client.0.smithi118.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T12:01:52.929 INFO:tasks.workunit.client.0.smithi118.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T12:01:52.930 INFO:tasks.workunit.client.0.smithi118.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T12:01:52.930 INFO:tasks.workunit.client.0.smithi118.stderr: 2023-10-30T12:01:52.930 INFO:tasks.workunit.client.0.smithi118.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T12:01:52.930 INFO:tasks.workunit.client.0.smithi118.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T12:01:52.930 INFO:tasks.workunit.client.0.smithi118.stderr: 2023-10-30T12:01:52.931 INFO:tasks.workunit.client.0.smithi118.stderr: git switch -c 2023-10-30T12:01:52.931 INFO:tasks.workunit.client.0.smithi118.stderr: 2023-10-30T12:01:52.931 INFO:tasks.workunit.client.0.smithi118.stderr:Or undo this operation with: 2023-10-30T12:01:52.931 INFO:tasks.workunit.client.0.smithi118.stderr: 2023-10-30T12:01:52.931 INFO:tasks.workunit.client.0.smithi118.stderr: git switch - 2023-10-30T12:01:52.932 INFO:tasks.workunit.client.0.smithi118.stderr: 2023-10-30T12:01:52.932 INFO:tasks.workunit.client.0.smithi118.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T12:01:52.932 INFO:tasks.workunit.client.0.smithi118.stderr: 2023-10-30T12:01:52.932 INFO:tasks.workunit.client.0.smithi118.stderr:HEAD is now at 397e85a5f8e qa: rewrite "valgrind_post" to use ValgrindScanner 2023-10-30T12:01:52.933 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T12:01:53.239 INFO:tasks.workunit.client.0.smithi118.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T12:01:53.241 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:01:53.241 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T12:01:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:53.285+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 msgr2=0x7f0a6813aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:53.285+0000 7f0a6d23f640 1 -- 172.21.15.50:0/3360253947 >> v1:172.21.15.50:6789/0 conn(0x7f0a68132db0 legacy=0x7f0a6813cfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:53.285+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 0x7f0a6813aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:01:53.914 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T12:01:53.996 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T12:01:54.038 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:01:54.039 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:01:54.040 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T12:01:54.078 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:01:54.081 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-30T12:01:54.081 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T12:01:54.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:54.086+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 msgr2=0x7f0a6813aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:54.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:54.086+0000 7f0a6d23f640 1 -- 172.21.15.50:0/3360253947 >> v1:172.21.15.50:6789/0 conn(0x7f0a68132db0 legacy=0x7f0a6813cfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:54.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:54.086+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 0x7f0a6813aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:01:54.097 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-10-30T12:01:54.098 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-10-30T12:01:54.098 DEBUG:teuthology.orchestra.run.smithi118:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=397e85a5f8ee6b984077e94184404370497237a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-10-30T12:01:54.155 INFO:tasks.workunit.client.0.smithi118.stderr:+ parallel=1 2023-10-30T12:01:54.156 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' '' = --serial ']' 2023-10-30T12:01:54.156 INFO:tasks.workunit.client.0.smithi118.stderr:+ color= 2023-10-30T12:01:54.156 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -t 1 ']' 2023-10-30T12:01:54.156 INFO:tasks.workunit.client.0.smithi118.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-10-30T12:01:54.156 INFO:tasks.workunit.client.0.smithi118.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2023-10-30T12:01:54.157 INFO:tasks.workunit.client.0.smithi118.stderr:+ declare -A pids 2023-10-30T12:01:54.157 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.157 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.157 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_aio 2023-10-30T12:01:54.157 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_aio' 2023-10-30T12:01:54.158 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_aio 2023-10-30T12:01:54.158 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.158 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_aio 2023-10-30T12:01:54.159 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87193 2023-10-30T12:01:54.159 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_aio on pid 87193' 2023-10-30T12:01:54.159 INFO:tasks.workunit.client.0.smithi118.stdout:test api_aio on pid 87193 2023-10-30T12:01:54.162 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87193 2023-10-30T12:01:54.162 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.163 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-10-30T12:01:54.163 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.163 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_aio_pp 2023-10-30T12:01:54.163 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_aio_pp' 2023-10-30T12:01:54.163 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_aio_pp 2023-10-30T12:01:54.164 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.164 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.164 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.164 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.164 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.164 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.165 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.165 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.165 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.165 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.165 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.166 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.166 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.166 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.166 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.166 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.166 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.167 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.167 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.167 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.167 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.167 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.168 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.168 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.168 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.168 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.168 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_aio_pp 2023-10-30T12:01:54.168 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87200 2023-10-30T12:01:54.169 INFO:tasks.workunit.client.0.smithi118.stdout:test api_aio_pp on pid 87200 2023-10-30T12:01:54.172 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_aio_pp on pid 87200' 2023-10-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87200 2023-10-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-10-30T12:01:54.174 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_io 2023-10-30T12:01:54.174 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_io' 2023-10-30T12:01:54.174 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_io 2023-10-30T12:01:54.174 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.175 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.175 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.175 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.176 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.176 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.176 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.176 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.176 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.177 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.177 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.177 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.177 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.177 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.177 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.178 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.178 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.178 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.178 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.178 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.179 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.179 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.179 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.179 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.179 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.179 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.180 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.180 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.180 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.180 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.180 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.181 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.181 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.181 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.181 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.181 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.181 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.182 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.182 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.182 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.182 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.182 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.183 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.183 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.183 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.183 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.183 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_io 2023-10-30T12:01:54.184 INFO:tasks.workunit.client.0.smithi118.stdout:test api_io on pid 87209 2023-10-30T12:01:54.189 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87209 2023-10-30T12:01:54.189 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_io on pid 87209' 2023-10-30T12:01:54.189 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87209 2023-10-30T12:01:54.190 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-10-30T12:01:54.190 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.190 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.190 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_io_pp 2023-10-30T12:01:54.190 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_io_pp' 2023-10-30T12:01:54.191 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_io_pp 2023-10-30T12:01:54.191 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.191 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.191 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.191 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.192 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.192 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.192 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.192 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.192 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.192 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.193 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.193 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.193 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.193 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.193 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.193 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.194 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.194 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.194 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.194 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.194 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.195 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.195 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.195 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.195 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.195 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.196 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.196 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.196 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.196 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.196 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.196 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.197 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.197 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.197 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.197 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.197 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.197 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.198 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.198 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.198 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.198 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.198 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.199 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.199 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.199 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.199 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.199 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.200 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.200 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.200 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.200 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.200 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.200 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.201 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.201 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_io_pp 2023-10-30T12:01:54.201 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87221 2023-10-30T12:01:54.201 INFO:tasks.workunit.client.0.smithi118.stdout:test api_io_pp on pid 87221 2023-10-30T12:01:54.207 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_io_pp on pid 87221' 2023-10-30T12:01:54.207 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87221 2023-10-30T12:01:54.207 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.208 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.208 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-10-30T12:01:54.208 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_asio 2023-10-30T12:01:54.208 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_asio' 2023-10-30T12:01:54.208 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_asio 2023-10-30T12:01:54.209 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.209 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.209 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.209 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.209 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.209 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.210 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.210 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.210 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.210 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.210 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.211 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.211 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.211 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.211 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.211 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.211 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.212 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.212 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.212 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.212 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.212 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.213 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.213 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.213 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.213 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.213 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.213 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.214 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.214 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.214 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.214 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.214 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.215 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.215 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.215 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.215 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.215 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.215 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.216 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.216 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.216 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.216 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.216 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.217 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.217 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.217 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.217 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.217 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.217 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.218 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.218 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.218 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.218 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.218 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.219 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.219 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.219 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.219 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.219 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.219 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.220 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.220 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.220 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.220 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.220 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.221 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.221 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.221 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.221 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.221 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.222 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.222 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.222 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_asio 2023-10-30T12:01:54.222 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87234 2023-10-30T12:01:54.222 INFO:tasks.workunit.client.0.smithi118.stdout:test api_asio on pid 87234 2023-10-30T12:01:54.228 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_asio on pid 87234' 2023-10-30T12:01:54.228 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.229 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87234 2023-10-30T12:01:54.229 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.229 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.229 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.229 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-10-30T12:01:54.230 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.230 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.230 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.230 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_list 2023-10-30T12:01:54.230 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.231 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_list' 2023-10-30T12:01:54.231 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_list 2023-10-30T12:01:54.231 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.231 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.231 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.232 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.232 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.232 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.232 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.232 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.232 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.233 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.233 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.233 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.233 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.233 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.233 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.234 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.234 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.234 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.234 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.234 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.235 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.235 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.235 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.235 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.235 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.236 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.236 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.236 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.236 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.236 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.236 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.237 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.237 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.237 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.237 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.237 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.238 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.238 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.238 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.238 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.239 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.240 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.241 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.241 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.241 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.241 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.241 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.242 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.242 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.242 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.242 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.242 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.242 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.243 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.243 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.243 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.243 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.243 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.244 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.244 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.244 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.244 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.244 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.244 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.245 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.245 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.245 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.245 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.245 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.246 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.246 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.246 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.246 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.246 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.246 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.247 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.247 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.247 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.247 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.247 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.248 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.248 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.248 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.248 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.248 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87193/exe 2023-10-30T12:01:54.248 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_list 2023-10-30T12:01:54.249 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87251 2023-10-30T12:01:54.249 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_list on pid 87251' 2023-10-30T12:01:54.249 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-10-30T12:01:54.249 INFO:tasks.workunit.client.0.smithi118.stdout:test api_list on pid 87251 2023-10-30T12:01:54.258 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87251 2023-10-30T12:01:54.258 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.258 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.259 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.259 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.259 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.259 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.259 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.259 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.260 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_lock 2023-10-30T12:01:54.260 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.260 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_lock' 2023-10-30T12:01:54.260 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.260 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.261 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.261 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_lock 2023-10-30T12:01:54.261 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.261 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.261 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.262 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.262 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.262 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.262 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.262 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.263 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.263 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.263 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.263 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.263 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.263 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.264 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.264 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.264 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.264 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.264 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.265 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.265 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.265 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.265 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.265 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.266 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.266 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.266 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.266 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.266 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.267 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.267 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.267 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.267 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.267 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.267 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.268 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.268 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.268 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.268 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2023-10-30T12:01:54.268 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.269 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.269 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.269 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.269 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_aio: /' 2023-10-30T12:01:54.269 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.270 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.270 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_aio.log 2023-10-30T12:01:54.270 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.270 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.270 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.270 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.271 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.271 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.271 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.271 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.271 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.272 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.272 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.272 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.272 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.272 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.272 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.273 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.273 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.273 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.273 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.273 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.274 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.274 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.274 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.274 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.274 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.274 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.275 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.275 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.275 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.275 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.275 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.276 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.276 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.276 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.276 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.276 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.276 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.277 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.277 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.277 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.277 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.277 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.278 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.278 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.278 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.278 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.278 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.278 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.279 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.279 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.279 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.279 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.281 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.281 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.281 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.281 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.282 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.282 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87200/exe 2023-10-30T12:01:54.282 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.282 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.282 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_lock 2023-10-30T12:01:54.282 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.283 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.283 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.283 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87272 2023-10-30T12:01:54.283 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_lock on pid 87272' 2023-10-30T12:01:54.283 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-10-30T12:01:54.284 INFO:tasks.workunit.client.0.smithi118.stdout:test api_lock on pid 87272 2023-10-30T12:01:54.291 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87272 2023-10-30T12:01:54.291 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.292 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.292 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.292 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.292 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.292 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.293 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_lock_pp 2023-10-30T12:01:54.293 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.293 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.293 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.293 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.293 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.294 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.294 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_lock_pp' 2023-10-30T12:01:54.294 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.294 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.294 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.295 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.295 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.295 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.295 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.295 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.296 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_lock_pp 2023-10-30T12:01:54.296 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.296 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.296 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.296 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.296 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.297 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.297 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.297 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.297 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.297 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.297 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.298 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.298 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.298 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.298 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.298 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.299 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.299 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.299 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.299 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.299 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.300 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.300 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.300 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.300 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.300 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.300 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.301 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.301 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.301 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.301 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.301 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.301 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.302 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.302 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2023-10-30T12:01:54.302 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.302 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.302 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.303 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.303 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.303 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.303 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-10-30T12:01:54.303 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.303 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.304 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.305 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.305 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.305 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.305 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.305 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.305 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.306 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.306 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.306 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.306 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.306 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.307 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.308 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.308 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_aio_pp: /' 2023-10-30T12:01:54.308 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.308 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.308 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.309 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.309 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.309 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.309 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.309 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.309 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.310 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.310 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.310 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.310 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.310 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_lock_pp 2023-10-30T12:01:54.311 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.311 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.311 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87291 2023-10-30T12:01:54.311 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.311 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.311 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_lock_pp on pid 87291' 2023-10-30T12:01:54.312 INFO:tasks.workunit.client.0.smithi118.stdout:test api_lock_pp on pid 87291 2023-10-30T12:01:54.323 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-10-30T12:01:54.323 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87291 2023-10-30T12:01:54.323 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.323 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.324 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.324 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.324 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.324 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.324 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.324 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.325 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.325 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_misc 2023-10-30T12:01:54.325 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.325 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.325 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.326 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.326 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_misc' 2023-10-30T12:01:54.326 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.326 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87209/exe 2023-10-30T12:01:54.326 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.327 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.327 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.327 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.327 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.327 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.327 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.328 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.328 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.328 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.328 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.328 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.328 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.329 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.329 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.329 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.329 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.329 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.330 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.330 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.330 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.330 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.330 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.330 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.331 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.331 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.331 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.331 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.331 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.332 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.332 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.332 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.332 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_misc 2023-10-30T12:01:54.332 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.333 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.333 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.333 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.333 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.333 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.333 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.334 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.334 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.334 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.334 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.334 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.335 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.335 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.335 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.335 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.335 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.335 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.336 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.336 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.336 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.336 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.336 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.337 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.337 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.337 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.337 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.337 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.338 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.338 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.338 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.338 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.338 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.339 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.339 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.339 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.339 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.339 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.339 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.340 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.340 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.340 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.340 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.340 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.340 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.341 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.341 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.341 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.341 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.341 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.342 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.342 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.342 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.342 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.342 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.343 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.343 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.343 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.343 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.343 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.344 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.344 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.344 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_io.log 2023-10-30T12:01:54.344 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.344 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.344 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.345 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.345 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.345 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.345 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.345 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.346 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.346 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.346 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.346 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.346 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.346 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.347 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.347 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.347 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.347 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2023-10-30T12:01:54.347 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87221/exe 2023-10-30T12:01:54.347 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.348 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.348 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.348 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.348 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.348 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.349 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.349 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.349 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.349 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.349 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.350 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.350 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.350 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.350 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.350 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_misc 2023-10-30T12:01:54.351 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.351 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.351 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.351 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.351 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.351 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.352 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.352 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87314 2023-10-30T12:01:54.352 INFO:tasks.workunit.client.0.smithi118.stdout:test api_misc on pid 87314 2023-10-30T12:01:54.359 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_misc on pid 87314' 2023-10-30T12:01:54.359 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87314 2023-10-30T12:01:54.359 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.359 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.359 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.360 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.360 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.360 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.360 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_misc_pp 2023-10-30T12:01:54.360 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.361 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.361 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.361 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.361 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.361 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.361 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_misc_pp' 2023-10-30T12:01:54.362 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.362 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.362 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.362 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.362 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.363 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.363 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.363 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.363 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.363 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.364 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-10-30T12:01:54.364 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.364 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.364 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.364 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.364 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.365 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.365 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.365 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.365 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.365 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.366 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.366 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.366 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.366 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.366 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.367 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_io: /' 2023-10-30T12:01:54.367 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.367 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.367 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.367 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.368 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.368 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.368 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.368 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_misc_pp 2023-10-30T12:01:54.368 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.369 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.369 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.369 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.369 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.369 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.369 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.370 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.370 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.370 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.370 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.370 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.371 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87234/exe 2023-10-30T12:01:54.371 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-10-30T12:01:54.371 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_io_pp: /' 2023-10-30T12:01:54.371 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.371 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.371 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.372 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.372 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.372 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.372 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.372 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.373 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.373 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.373 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.373 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.373 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.374 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.374 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.374 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.374 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.374 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.375 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.375 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_misc_pp 2023-10-30T12:01:54.375 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.375 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.375 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87331 2023-10-30T12:01:54.376 INFO:tasks.workunit.client.0.smithi118.stdout:test api_misc_pp on pid 87331 2023-10-30T12:01:54.382 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_misc_pp on pid 87331' 2023-10-30T12:01:54.382 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87331 2023-10-30T12:01:54.382 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.382 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.383 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.383 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2023-10-30T12:01:54.383 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.383 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.383 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.384 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.384 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.384 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.384 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.384 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.385 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_asio.log 2023-10-30T12:01:54.385 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.385 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.385 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.385 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.386 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.386 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.386 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.386 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.386 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.387 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.387 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.387 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.387 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.387 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.388 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.388 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.388 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.388 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.388 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.389 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.389 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.389 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.389 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.389 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.389 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.390 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_tier_pp 2023-10-30T12:01:54.390 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_tier_pp' 2023-10-30T12:01:54.390 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_asio: /' 2023-10-30T12:01:54.390 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2023-10-30T12:01:54.391 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.391 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.391 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.391 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.391 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_tier_pp 2023-10-30T12:01:54.391 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.392 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.392 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.392 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.392 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.392 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.393 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.393 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.393 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.393 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.393 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.394 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.394 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.394 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.394 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.394 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.395 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.395 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.395 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.395 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.396 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.396 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.396 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.396 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.396 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.396 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.397 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.397 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.397 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.397 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.397 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.397 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.398 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.398 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.398 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.398 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87251/exe 2023-10-30T12:01:54.398 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.399 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-10-30T12:01:54.399 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_tier_pp 2023-10-30T12:01:54.399 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87347 2023-10-30T12:01:54.399 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_tier_pp on pid 87347' 2023-10-30T12:01:54.400 INFO:tasks.workunit.client.0.smithi118.stdout:test api_tier_pp on pid 87347 2023-10-30T12:01:54.409 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87347 2023-10-30T12:01:54.409 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.409 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.409 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.410 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.410 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.410 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.410 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.410 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.411 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.411 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.411 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.411 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.411 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.412 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.412 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.412 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.412 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.412 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.413 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.414 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.414 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.414 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.414 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.414 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.415 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.415 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.415 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.415 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_pool 2023-10-30T12:01:54.415 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_list.log 2023-10-30T12:01:54.415 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_pool' 2023-10-30T12:01:54.416 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.416 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.416 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.416 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.416 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.416 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.417 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-10-30T12:01:54.417 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.417 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.417 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.417 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.418 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.418 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.418 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.418 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.418 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.419 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.419 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.419 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.419 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.419 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.419 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.419 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.420 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.420 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.420 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.420 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.420 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.421 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.421 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.421 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.421 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.421 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.421 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.422 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.422 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.422 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.422 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.422 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.422 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.423 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.423 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.423 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.423 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.423 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.423 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.424 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.424 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.424 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.424 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.424 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.424 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.425 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.425 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.425 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.425 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.425 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.425 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.426 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.426 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.426 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.426 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.426 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.427 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.427 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.427 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.427 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.427 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.427 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.428 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.428 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.428 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.428 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.428 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.428 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.429 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.429 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_pool 2023-10-30T12:01:54.429 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.429 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.429 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.429 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.430 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.430 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.430 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.430 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.430 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.431 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.431 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_pool 2023-10-30T12:01:54.431 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87367 2023-10-30T12:01:54.431 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_pool on pid 87367' 2023-10-30T12:01:54.431 INFO:tasks.workunit.client.0.smithi118.stdout:test api_pool on pid 87367 2023-10-30T12:01:54.439 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87367 2023-10-30T12:01:54.439 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.440 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.440 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.440 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.440 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.440 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_snapshots 2023-10-30T12:01:54.441 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.441 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.441 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_snapshots' 2023-10-30T12:01:54.441 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.441 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.441 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.442 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.442 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-10-30T12:01:54.442 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.442 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.442 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.443 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.443 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.443 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.443 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.443 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.443 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.444 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.444 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.444 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.444 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.444 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.444 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.445 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.445 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.445 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.445 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.445 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.445 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.446 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.446 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.446 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.446 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.446 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.446 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.447 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.447 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.447 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.447 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.447 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.447 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.448 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.448 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.448 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.448 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.448 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.448 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.449 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.449 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.449 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.449 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.449 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.449 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.450 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.450 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.450 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.450 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.450 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.451 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.451 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.451 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.451 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.451 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.451 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.452 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.452 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.452 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.452 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_snapshots 2023-10-30T12:01:54.452 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.452 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.453 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.453 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.453 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.453 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.453 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.453 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.454 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.454 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.454 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.454 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.454 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.454 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.455 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.455 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.455 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.455 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.455 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.455 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.456 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.456 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.456 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.456 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.456 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.456 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.457 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.457 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.457 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_snapshots 2023-10-30T12:01:54.457 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.457 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2023-10-30T12:01:54.457 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.458 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.458 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.458 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.458 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87382 2023-10-30T12:01:54.458 INFO:tasks.workunit.client.0.smithi118.stdout:test api_snapshots on pid 87382 2023-10-30T12:01:54.465 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_snapshots on pid 87382' 2023-10-30T12:01:54.465 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87382 2023-10-30T12:01:54.465 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.465 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.465 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.466 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-10-30T12:01:54.466 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.466 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.466 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.466 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.467 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_snapshots_pp 2023-10-30T12:01:54.467 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.467 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.467 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.467 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_snapshots_pp' 2023-10-30T12:01:54.467 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.468 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.468 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_list: /' 2023-10-30T12:01:54.468 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.468 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.468 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.469 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_snapshots_pp 2023-10-30T12:01:54.469 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.469 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.469 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.469 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.469 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.470 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.470 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.470 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.470 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.470 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.471 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87272/exe 2023-10-30T12:01:54.471 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.471 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.471 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.471 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.471 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.472 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.472 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.472 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.472 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.472 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.473 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.473 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.473 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.473 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.473 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.473 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.474 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.474 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.474 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.474 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.474 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.474 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.475 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.475 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.475 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.475 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.475 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.475 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.476 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.476 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.476 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.476 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.476 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.476 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87314/exe 2023-10-30T12:01:54.477 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.477 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.477 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.477 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.477 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.477 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.478 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.478 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_snapshots_pp 2023-10-30T12:01:54.478 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.478 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.478 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.478 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.479 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.479 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.479 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.479 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.479 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.479 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87407 2023-10-30T12:01:54.480 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_snapshots_pp on pid 87407' 2023-10-30T12:01:54.480 INFO:tasks.workunit.client.0.smithi118.stdout:test api_snapshots_pp on pid 87407 2023-10-30T12:01:54.489 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87407 2023-10-30T12:01:54.489 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.489 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.490 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.490 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.490 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.490 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_stat 2023-10-30T12:01:54.490 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_stat' 2023-10-30T12:01:54.490 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.491 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.491 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.491 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.491 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.491 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.491 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.492 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.492 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.492 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.493 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-10-30T12:01:54.493 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87291/exe 2023-10-30T12:01:54.493 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_lock.log 2023-10-30T12:01:54.493 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.494 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.494 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.494 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.494 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.494 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.494 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.495 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.495 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.495 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.495 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.495 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.496 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.496 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.496 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_stat 2023-10-30T12:01:54.496 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.496 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.496 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.497 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.497 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.497 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.497 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.497 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.497 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.498 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.498 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.498 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.498 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.498 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.499 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.499 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2023-10-30T12:01:54.499 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.499 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.499 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.500 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.500 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2023-10-30T12:01:54.500 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.500 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.500 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.500 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.501 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.501 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.501 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.501 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.501 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.502 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.502 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.502 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.502 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.502 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.502 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.503 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.503 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.503 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_misc.log 2023-10-30T12:01:54.503 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.503 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.504 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.505 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.506 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.506 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.506 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.506 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.506 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.507 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.507 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.507 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.507 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_lock: /' 2023-10-30T12:01:54.507 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.508 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.508 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.508 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.508 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-10-30T12:01:54.508 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.508 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.509 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.509 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.509 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.509 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.509 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.510 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.510 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.510 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2023-10-30T12:01:54.510 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_misc: /' 2023-10-30T12:01:54.510 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_lock_pp: /' 2023-10-30T12:01:54.511 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_stat 2023-10-30T12:01:54.511 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87441 2023-10-30T12:01:54.511 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_stat on pid 87441' 2023-10-30T12:01:54.512 INFO:tasks.workunit.client.0.smithi118.stdout:test api_stat on pid 87441 2023-10-30T12:01:54.523 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87441 2023-10-30T12:01:54.523 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.523 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.523 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_stat_pp 2023-10-30T12:01:54.524 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.524 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-10-30T12:01:54.524 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_stat_pp' 2023-10-30T12:01:54.524 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.524 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.525 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.525 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.525 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.525 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.525 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.525 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.526 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.526 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.526 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.526 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_stat_pp 2023-10-30T12:01:54.526 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.527 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.527 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.527 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.527 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.527 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.527 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.528 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.528 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.528 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.528 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.528 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.529 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.529 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.529 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.529 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.529 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.529 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.530 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.530 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.530 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.530 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.530 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.531 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.531 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.531 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.531 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.531 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.531 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.532 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.532 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.532 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.532 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.532 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.533 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.533 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.533 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.533 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.533 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.533 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.534 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.534 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.534 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.534 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.534 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.534 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.535 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.535 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.535 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.535 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.535 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.536 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.536 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.536 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.536 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.536 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.537 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.537 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.537 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.537 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.537 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.537 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.538 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.538 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.538 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.538 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.538 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.539 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.539 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.539 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.539 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.539 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.540 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.540 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.540 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.540 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.540 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.540 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.541 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.541 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.541 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.541 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.541 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.542 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.542 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.542 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.542 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.542 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.542 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.543 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.543 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.543 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.543 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.543 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.544 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.544 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.544 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.544 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87347/exe 2023-10-30T12:01:54.544 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.545 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.545 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.545 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.545 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.545 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.545 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.546 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.546 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.546 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.546 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.546 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.547 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_stat_pp 2023-10-30T12:01:54.547 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.547 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.547 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.547 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87467 2023-10-30T12:01:54.547 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_stat_pp on pid 87467' 2023-10-30T12:01:54.548 INFO:tasks.workunit.client.0.smithi118.stdout:test api_stat_pp on pid 87467 2023-10-30T12:01:54.559 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87467 2023-10-30T12:01:54.559 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.559 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.559 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.559 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.560 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.560 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.560 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.560 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.560 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.561 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2023-10-30T12:01:54.561 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-10-30T12:01:54.561 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-10-30T12:01:54.561 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_tier_pp: /' 2023-10-30T12:01:54.561 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.562 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.562 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.562 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.562 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.562 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.562 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.563 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.563 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.563 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.563 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.563 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.564 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.564 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.564 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.564 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.564 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.564 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.565 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.565 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.565 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.565 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.565 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.566 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.566 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.566 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.566 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.566 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.566 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.567 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.567 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.567 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.567 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.567 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.568 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.568 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.568 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.568 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.568 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.569 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.569 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.569 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.569 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.569 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.570 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.570 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.570 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.570 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.570 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.571 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.571 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.571 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.571 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.571 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.571 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.572 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.572 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.572 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.572 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.572 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.572 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.573 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.573 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.573 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.573 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.573 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.574 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.574 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.574 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.574 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.574 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.575 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.575 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.575 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.575 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.575 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.575 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_watch_notify 2023-10-30T12:01:54.576 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.576 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_watch_notify' 2023-10-30T12:01:54.576 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.576 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.576 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.577 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.577 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.577 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_watch_notify 2023-10-30T12:01:54.577 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.577 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.577 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.578 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.578 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.578 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.578 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.578 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.579 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.579 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.579 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.579 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.579 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.579 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.580 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.580 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.580 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.580 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.580 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.581 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.581 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.581 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.581 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.581 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.582 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.582 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.582 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.582 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.582 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.582 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.583 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.583 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.583 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.583 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.583 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.584 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.584 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.584 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.584 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.584 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.584 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.585 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.585 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.585 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.585 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_watch_notify 2023-10-30T12:01:54.585 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.586 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.586 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.586 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87497 2023-10-30T12:01:54.586 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_watch_notify on pid 87497' 2023-10-30T12:01:54.586 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.587 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.587 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.587 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.587 INFO:tasks.workunit.client.0.smithi118.stdout:test api_watch_notify on pid 87497 2023-10-30T12:01:54.617 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87497 2023-10-30T12:01:54.617 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.617 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.617 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.617 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.618 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-10-30T12:01:54.618 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.618 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.618 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.618 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.619 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.619 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.619 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.619 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.619 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.619 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.620 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.620 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.620 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.620 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.620 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.621 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.621 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.621 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.621 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.621 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.621 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.622 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.622 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.622 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.622 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.622 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.623 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87382/exe 2023-10-30T12:01:54.623 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.623 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.623 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.623 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.624 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.624 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.624 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.624 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.624 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.624 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.625 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.625 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.625 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.625 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.625 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.626 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.626 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.626 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.626 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.626 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.626 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.627 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.627 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.627 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.627 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.627 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.628 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.628 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.628 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.628 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.628 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.629 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.629 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.629 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.629 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.629 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.630 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.630 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.630 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.630 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.630 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.631 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-10-30T12:01:54.631 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.631 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.631 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.632 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.632 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.632 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.632 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.632 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.633 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.633 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.633 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.633 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.633 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.634 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.634 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.634 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.634 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.634 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.635 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_snapshots: /' 2023-10-30T12:01:54.635 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.635 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.635 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.636 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.636 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.636 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.636 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.636 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.636 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.637 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.637 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.637 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.637 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.637 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.638 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.638 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.638 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.638 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.638 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.638 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.639 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.639 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.639 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.639 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.639 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.640 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.640 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.640 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.640 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.641 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.641 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.641 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.641 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.641 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.642 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.643 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.643 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.643 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.643 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.643 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.644 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.644 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.644 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.644 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.644 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.644 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.645 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.645 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.645 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.645 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.645 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.645 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.646 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.646 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.646 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.646 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.646 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.647 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.647 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.647 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.647 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_watch_notify_pp 2023-10-30T12:01:54.647 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.647 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.648 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.648 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_watch_notify_pp' 2023-10-30T12:01:54.648 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.648 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.648 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.649 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.650 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.650 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.650 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.650 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.650 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87367/exe 2023-10-30T12:01:54.650 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.651 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.651 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.651 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.651 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.651 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.652 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.652 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.652 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.652 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.652 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.653 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.653 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.653 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.653 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.653 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.654 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.654 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.654 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.654 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.654 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.654 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.655 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.655 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.655 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.655 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.655 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.655 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.656 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.656 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.656 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.656 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.656 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.656 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.657 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.658 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.658 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.658 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.658 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.658 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.659 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.659 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.659 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87441/exe 2023-10-30T12:01:54.659 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.659 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.660 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_pool.log 2023-10-30T12:01:54.660 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.660 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.660 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.660 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.661 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.661 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.661 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.661 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.662 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.662 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.662 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.662 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.662 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.663 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.663 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.663 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.663 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.663 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.663 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.664 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.664 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.664 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.664 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.664 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.664 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_stat: /' 2023-10-30T12:01:54.665 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.665 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.665 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.665 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.665 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.665 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.666 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.666 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2023-10-30T12:01:54.666 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.666 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.666 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.667 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.668 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.668 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.668 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.668 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.668 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2023-10-30T12:01:54.669 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.669 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.669 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.669 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.669 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.670 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.670 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.670 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.670 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.670 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.671 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.671 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.671 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.671 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.671 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2023-10-30T12:01:54.672 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.672 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_stat.log 2023-10-30T12:01:54.672 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.672 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.672 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.673 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.673 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.673 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.673 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.673 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.673 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.674 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.674 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.674 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.674 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.674 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_pool: /' 2023-10-30T12:01:54.675 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.675 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.675 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.675 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.675 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.676 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.677 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.677 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.677 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.677 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.677 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.677 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.678 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.678 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.678 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.678 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.678 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.679 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.679 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.679 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.679 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.679 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.679 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.680 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.680 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.680 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.680 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.680 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.681 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.681 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.681 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.681 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.681 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.681 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.682 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.682 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.682 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87331/exe 2023-10-30T12:01:54.682 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.682 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.683 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.683 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.683 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.683 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.683 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.683 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.684 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.684 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.684 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.684 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87407/exe 2023-10-30T12:01:54.684 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87467/exe 2023-10-30T12:01:54.684 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.685 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.685 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.685 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_watch_notify_pp 2023-10-30T12:01:54.685 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.685 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.686 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.686 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.686 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.686 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.686 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.686 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.687 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.687 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.687 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.687 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.687 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.687 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.688 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.688 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.688 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.688 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.688 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_watch_notify_pp 2023-10-30T12:01:54.688 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.689 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.689 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.689 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.689 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87575 2023-10-30T12:01:54.689 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.690 INFO:tasks.workunit.client.0.smithi118.stdout:test api_watch_notify_pp on pid 87575 2023-10-30T12:01:54.694 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_watch_notify_pp on pid 87575' 2023-10-30T12:01:54.695 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87575 2023-10-30T12:01:54.695 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.695 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.695 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.696 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.696 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.696 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.696 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.696 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.696 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.697 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_cmd 2023-10-30T12:01:54.697 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_cmd' 2023-10-30T12:01:54.697 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_stat_pp: /' 2023-10-30T12:01:54.697 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2023-10-30T12:01:54.697 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2023-10-30T12:01:54.698 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_cmd 2023-10-30T12:01:54.698 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-10-30T12:01:54.698 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.698 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.698 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.698 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.699 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.699 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.699 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.699 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-10-30T12:01:54.699 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.699 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.700 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.700 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.700 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.700 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.700 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.701 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.701 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.701 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.701 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.701 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.701 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.702 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.702 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.702 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.702 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.702 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.703 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.703 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.703 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.703 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.703 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.703 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.704 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.704 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.704 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.704 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.704 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_cmd 2023-10-30T12:01:54.705 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.705 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.705 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2023-10-30T12:01:54.705 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.705 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87598 2023-10-30T12:01:54.705 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_cmd on pid 87598' 2023-10-30T12:01:54.706 INFO:tasks.workunit.client.0.smithi118.stdout:test api_cmd on pid 87598 2023-10-30T12:01:54.716 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87598 2023-10-30T12:01:54.716 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.716 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.716 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.717 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.717 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.717 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.717 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.717 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_cmd_pp 2023-10-30T12:01:54.717 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_cmd_pp' 2023-10-30T12:01:54.718 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_cmd_pp 2023-10-30T12:01:54.718 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.718 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.718 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.718 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.719 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.719 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.719 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.719 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.719 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.719 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.720 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.720 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.720 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.720 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.720 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.720 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.721 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.721 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.721 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.721 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.721 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.722 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.722 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.722 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-10-30T12:01:54.722 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.722 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.723 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.723 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.723 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-10-30T12:01:54.723 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.723 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.724 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.724 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.724 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.724 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.724 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.724 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.725 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.725 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.725 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.725 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.725 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.725 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.726 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.726 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.726 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.726 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.726 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.727 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.727 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.727 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.727 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.727 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.727 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.728 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.728 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.728 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87497/exe 2023-10-30T12:01:54.728 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.728 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.728 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.729 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.729 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.729 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.729 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-10-30T12:01:54.729 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.730 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.730 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.730 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.730 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.730 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.730 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.731 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.731 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.731 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.731 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.731 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.732 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.732 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.732 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.732 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.732 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.732 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.733 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.733 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.733 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.733 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.733 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.734 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.734 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.734 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.734 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.734 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.734 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.735 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.735 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.735 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.735 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.735 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-10-30T12:01:54.736 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.736 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.736 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2023-10-30T12:01:54.736 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.736 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.736 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.737 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.737 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.737 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.737 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.737 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.737 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.738 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.738 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.738 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.738 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.738 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.738 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.739 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.739 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_misc_pp: /' 2023-10-30T12:01:54.739 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_cmd_pp 2023-10-30T12:01:54.739 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87619 2023-10-30T12:01:54.739 INFO:tasks.workunit.client.0.smithi118.stdout:test api_cmd_pp on pid 87619 2023-10-30T12:01:54.743 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_cmd_pp on pid 87619' 2023-10-30T12:01:54.743 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87619 2023-10-30T12:01:54.743 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.743 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.744 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-10-30T12:01:54.744 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_service 2023-10-30T12:01:54.744 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_service' 2023-10-30T12:01:54.744 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_service 2023-10-30T12:01:54.744 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-10-30T12:01:54.744 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.745 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.745 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.745 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.745 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.745 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.745 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.746 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.746 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_watch_notify: /' 2023-10-30T12:01:54.746 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.746 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.746 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.747 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.747 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.747 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.747 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.747 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.748 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.748 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.748 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.748 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.748 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.748 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.749 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.749 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.749 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.749 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.749 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.749 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.750 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_service 2023-10-30T12:01:54.750 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87647 2023-10-30T12:01:54.750 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_service on pid 87647' 2023-10-30T12:01:54.750 INFO:tasks.workunit.client.0.smithi118.stdout:test api_service on pid 87647 2023-10-30T12:01:54.762 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87647 2023-10-30T12:01:54.762 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.762 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.762 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.763 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.763 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.763 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.763 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.763 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.764 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.764 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-10-30T12:01:54.764 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.764 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.764 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.764 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.765 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.765 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.765 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.765 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.765 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.766 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.766 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.766 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.766 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.766 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.766 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.767 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.767 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.767 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.767 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.767 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.768 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.768 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.768 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.768 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.768 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.768 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.769 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.769 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.769 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.769 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.769 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.769 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.770 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.770 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.770 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.770 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.770 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.771 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.771 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.771 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.771 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.771 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.771 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.772 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.772 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.772 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.772 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.773 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.773 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.773 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.773 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.773 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.774 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.774 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.774 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.774 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.774 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.774 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.775 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.776 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.776 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.776 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.776 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.776 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.777 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.777 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.777 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.777 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.777 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.778 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.778 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.778 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.778 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.778 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.778 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.779 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.779 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.779 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.779 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.779 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.780 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87575/exe 2023-10-30T12:01:54.780 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.780 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.780 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.780 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.781 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.781 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.781 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.781 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.781 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.781 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.782 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.782 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.782 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.782 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.782 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.783 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.783 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.783 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.783 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.783 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.783 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.784 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.784 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.784 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.784 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.784 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.784 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.785 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.785 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.785 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.785 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.785 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2023-10-30T12:01:54.786 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-10-30T12:01:54.786 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.786 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.786 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.786 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.786 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.787 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.787 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.787 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.787 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.787 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_service_pp 2023-10-30T12:01:54.788 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_service_pp' 2023-10-30T12:01:54.788 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_service_pp 2023-10-30T12:01:54.788 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.788 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.788 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.788 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.789 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.789 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.789 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.789 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.789 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.790 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.790 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.790 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.790 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.790 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.790 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.791 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.791 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.791 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.791 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.791 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.792 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.792 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.792 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.792 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.792 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.792 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.793 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.793 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_service_pp 2023-10-30T12:01:54.793 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87690 2023-10-30T12:01:54.793 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_service_pp on pid 87690' 2023-10-30T12:01:54.794 INFO:tasks.workunit.client.0.smithi118.stdout:test api_service_pp on pid 87690 2023-10-30T12:01:54.802 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87690 2023-10-30T12:01:54.802 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.802 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.802 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_c_write_operations 2023-10-30T12:01:54.802 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_c_write_operations' 2023-10-30T12:01:54.803 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.803 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.803 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.803 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.803 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.804 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.804 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.804 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.804 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.804 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.804 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.805 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.805 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.805 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.805 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.805 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.805 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.806 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.806 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.806 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.806 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.806 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.807 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.808 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.808 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.808 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.808 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.808 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.808 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.809 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.809 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.809 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.809 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.809 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.810 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.810 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.810 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.810 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.810 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.811 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.811 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.811 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.811 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.811 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.811 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.812 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.812 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.812 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.812 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.812 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.813 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87619/exe 2023-10-30T12:01:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.814 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.815 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.815 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.815 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.815 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.815 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.816 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.816 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_c_write_operations 2023-10-30T12:01:54.816 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.816 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.816 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.817 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.817 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.817 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.817 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.817 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.818 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.818 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.818 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.818 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.818 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.819 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.819 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.819 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.819 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.819 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.819 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.820 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-10-30T12:01:54.820 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.820 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.820 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.820 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_c_write_operations 2023-10-30T12:01:54.821 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.821 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.821 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.821 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.821 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87711 2023-10-30T12:01:54.822 INFO:tasks.workunit.client.0.smithi118.stdout:test api_c_write_operations on pid 87711 2023-10-30T12:01:54.828 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_c_write_operations on pid 87711' 2023-10-30T12:01:54.828 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87711 2023-10-30T12:01:54.829 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.829 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.829 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.829 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-10-30T12:01:54.829 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.830 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.830 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.830 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.830 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.830 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.831 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.832 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-10-30T12:01:54.832 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-10-30T12:01:54.832 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-10-30T12:01:54.832 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.832 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.832 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.833 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.833 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.833 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.833 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_c_read_operations 2023-10-30T12:01:54.833 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_c_read_operations' 2023-10-30T12:01:54.833 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.834 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2023-10-30T12:01:54.834 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.834 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.834 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_c_read_operations 2023-10-30T12:01:54.834 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.835 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.835 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.835 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.835 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.835 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87647/exe 2023-10-30T12:01:54.835 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.836 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.836 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.836 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.836 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.836 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.837 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.837 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.837 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.837 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.837 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.837 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.838 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.838 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.838 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.838 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.838 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.838 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.839 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.839 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.839 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.839 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.839 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.839 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.840 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.840 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.840 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.840 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.840 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.841 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.841 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.841 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.841 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.841 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.842 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.842 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.842 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.842 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.842 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.843 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.843 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.843 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.843 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.843 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.844 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_c_read_operations 2023-10-30T12:01:54.844 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.844 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.844 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.844 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.845 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87744 2023-10-30T12:01:54.846 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.846 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-10-30T12:01:54.846 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.846 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_c_read_operations on pid 87744' 2023-10-30T12:01:54.847 INFO:tasks.workunit.client.0.smithi118.stdout:test api_c_read_operations on pid 87744 2023-10-30T12:01:54.853 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87744 2023-10-30T12:01:54.853 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.853 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2023-10-30T12:01:54.854 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.854 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_service: /' 2023-10-30T12:01:54.854 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s api_cls_remote_reads 2023-10-30T12:01:54.854 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' api_cls_remote_reads' 2023-10-30T12:01:54.854 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.855 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.855 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.855 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.855 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.855 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.856 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.856 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.856 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo api_cls_remote_reads 2023-10-30T12:01:54.856 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.856 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.856 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.857 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.857 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.857 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.857 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.857 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.858 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.858 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.858 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.858 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.858 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.859 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.859 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.859 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.859 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.859 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.859 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.860 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.860 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.860 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.860 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.860 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.861 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.861 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.861 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.861 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.861 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.861 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.862 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.862 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.862 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.862 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.862 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.863 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.863 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.863 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.863 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.863 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.863 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.864 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.864 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.864 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.864 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.864 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.864 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.865 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.865 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.865 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_service.log 2023-10-30T12:01:54.865 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.865 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.866 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.866 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.866 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.866 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.866 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87598/exe 2023-10-30T12:01:54.867 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.867 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.867 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.867 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.867 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.867 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.868 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=api_cls_remote_reads 2023-10-30T12:01:54.868 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.868 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.868 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87764 2023-10-30T12:01:54.868 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test api_cls_remote_reads on pid 87764' 2023-10-30T12:01:54.869 INFO:tasks.workunit.client.0.smithi118.stdout:test api_cls_remote_reads on pid 87764 2023-10-30T12:01:54.878 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.878 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87764 2023-10-30T12:01:54.878 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.879 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.879 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.879 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.879 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.879 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.880 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.880 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.880 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.880 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.880 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.881 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.881 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.881 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.881 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.881 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.882 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.882 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.882 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.882 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.882 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.883 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-10-30T12:01:54.883 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.883 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.883 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.883 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.884 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.884 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.884 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.884 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.884 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.884 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.885 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.885 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.885 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.885 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.885 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.886 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.886 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s list_parallel 2023-10-30T12:01:54.886 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_cmd.log 2023-10-30T12:01:54.886 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' list_parallel' 2023-10-30T12:01:54.886 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2023-10-30T12:01:54.887 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.887 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.887 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.887 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.887 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.887 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.888 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.888 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.888 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.888 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.888 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.889 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.889 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.889 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.889 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_cmd: /' 2023-10-30T12:01:54.889 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.890 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.890 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.890 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.890 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.890 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.890 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.891 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.891 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.891 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.891 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.891 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.892 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.892 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.892 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.892 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.892 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.893 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.893 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.893 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.893 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.893 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.893 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.894 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.894 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.894 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.894 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.894 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.895 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.895 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.895 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.895 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.895 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.896 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.897 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.897 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.898 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.898 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.898 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.898 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.898 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.899 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.899 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.899 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.899 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.899 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.900 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.900 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.900 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.900 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.900 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.900 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.901 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.902 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.902 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo list_parallel 2023-10-30T12:01:54.902 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.902 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=list_parallel 2023-10-30T12:01:54.902 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87788 2023-10-30T12:01:54.903 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test list_parallel on pid 87788' 2023-10-30T12:01:54.903 INFO:tasks.workunit.client.0.smithi118.stdout:test list_parallel on pid 87788 2023-10-30T12:01:54.911 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.912 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.912 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.912 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87788 2023-10-30T12:01:54.912 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.912 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.913 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.913 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.913 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.913 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.913 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.914 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.914 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.914 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.914 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.914 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.914 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.915 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.915 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.915 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.915 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.915 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.916 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.916 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.916 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.916 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.916 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s open_pools_parallel 2023-10-30T12:01:54.916 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' open_pools_parallel' 2023-10-30T12:01:54.917 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.917 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.917 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.917 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.917 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.918 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.918 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.918 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.918 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.918 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.919 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.919 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo open_pools_parallel 2023-10-30T12:01:54.919 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.919 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.919 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.919 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.920 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.920 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.920 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.920 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.920 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.921 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.921 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.921 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.921 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.921 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.922 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.922 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.922 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.922 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.922 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.923 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.923 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.923 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.923 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.923 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.923 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.924 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.924 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.924 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.924 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.924 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.924 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.925 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.925 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.925 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.925 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-10-30T12:01:54.925 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.925 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.926 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.926 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87690/exe 2023-10-30T12:01:54.926 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.926 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.926 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.927 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.927 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.927 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.927 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.927 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.927 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.928 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.928 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.928 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.928 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.928 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.928 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.929 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.929 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.929 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.929 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.929 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.929 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.930 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.930 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.930 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.930 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.930 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.930 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.931 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=open_pools_parallel 2023-10-30T12:01:54.931 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.931 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.931 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.931 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.932 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.932 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.932 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.932 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.932 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.932 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.933 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.933 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.933 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.933 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.933 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87803 2023-10-30T12:01:54.934 INFO:tasks.workunit.client.0.smithi118.stdout:test open_pools_parallel on pid 87803 2023-10-30T12:01:54.946 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test open_pools_parallel on pid 87803' 2023-10-30T12:01:54.946 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87803 2023-10-30T12:01:54.947 INFO:tasks.workunit.client.0.smithi118.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-10-30T12:01:54.947 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.947 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.947 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.948 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.948 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.948 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:54.948 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.948 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.948 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.949 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.949 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.949 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.949 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.949 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.950 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-10-30T12:01:54.950 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.950 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.950 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.950 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.951 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.951 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.951 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.951 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.951 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.952 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.952 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.952 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.952 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.952 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2023-10-30T12:01:54.952 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-10-30T12:01:54.953 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.953 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.953 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.953 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.953 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.954 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.954 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.954 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.954 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.954 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.955 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.955 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.955 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.955 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:54.955 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.955 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:54.956 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:54.956 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:54.956 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:54.956 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:54.956 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:54.956 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:54.957 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.957 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:54.957 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87711/exe 2023-10-30T12:01:54.957 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.957 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:54.958 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.958 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:54.958 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:54.958 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:54.958 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:54.959 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:54.959 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.959 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.959 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.959 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.959 INFO:tasks.workunit.client.0.smithi118.stderr:++ printf %25s delete_pools_parallel 2023-10-30T12:01:54.960 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.960 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:54.960 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:54.960 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.960 INFO:tasks.workunit.client.0.smithi118.stderr:+ r=' delete_pools_parallel' 2023-10-30T12:01:54.961 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.961 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:54.961 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:54.961 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.961 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:54.961 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.962 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:54.962 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.962 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:54.962 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.962 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:54.963 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.963 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:54.963 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:54.963 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.963 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:54.964 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:54.964 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.964 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:54.964 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.964 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.965 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.965 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:54.965 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.965 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.965 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.965 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.966 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.966 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.966 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.966 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:54.966 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.967 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.967 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.967 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:54.967 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:54.967 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.967 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:54.968 INFO:tasks.workunit.client.0.smithi118.stderr:++ echo delete_pools_parallel 2023-10-30T12:01:54.968 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:54.968 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:54.968 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.968 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.969 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.969 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:54.969 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:54.969 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:54.969 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:54.970 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:54.970 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:54.970 INFO:tasks.workunit.client.0.smithi118.stderr:++ awk '{print $1}' 2023-10-30T12:01:54.970 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:54.970 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:54.971 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:54.971 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:54.971 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:54.971 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.971 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:54.971 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:54.972 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:54.972 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:54.972 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:54.972 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.972 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:54.973 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.973 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:54.973 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:54.973 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.973 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:54.974 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.974 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:54.974 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:54.974 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-10-30T12:01:54.974 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:54.974 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:54.975 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:54.975 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2023-10-30T12:01:54.975 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.975 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:54.975 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.976 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:54.976 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:54.976 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:54.976 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.976 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:54.977 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.977 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:54.977 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:54.977 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:54.977 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.977 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:54.978 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.978 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:54.978 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:54.978 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:54.978 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:54.979 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:54.979 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.979 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:54.979 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.979 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:54.979 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:54.980 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.980 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:54.980 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:54.980 INFO:tasks.workunit.client.0.smithi118.stderr:+ ff=delete_pools_parallel 2023-10-30T12:01:54.980 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:54.981 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87836 2023-10-30T12:01:54.981 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:54.981 INFO:tasks.workunit.client.0.smithi118.stdout:test delete_pools_parallel on pid 87836 2023-10-30T12:01:55.005 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'test delete_pools_parallel on pid 87836' 2023-10-30T12:01:55.005 INFO:tasks.workunit.client.0.smithi118.stderr:+ pids[$f]=87836 2023-10-30T12:01:55.005 INFO:tasks.workunit.client.0.smithi118.stderr:+ ret=0 2023-10-30T12:01:55.005 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' 1 -eq 1 ']' 2023-10-30T12:01:55.006 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:55.006 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.006 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:55.006 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:55.006 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:01:55.007 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87347 2023-10-30T12:01:55.007 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87347 2023-10-30T12:01:55.007 INFO:tasks.workunit.client.0.smithi118.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-10-30T12:01:55.007 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:55.007 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87744/exe 2023-10-30T12:01:55.008 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:55.008 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -f /etc/bashrc ']' 2023-10-30T12:01:55.008 INFO:tasks.workunit.client.0.smithi118.stderr:+ . /etc/bashrc 2023-10-30T12:01:55.008 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -z '' ']' 2023-10-30T12:01:55.008 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.008 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:55.009 INFO:tasks.workunit.client.0.smithi118.stderr:++ BASHRCSOURCED=Y 2023-10-30T12:01:55.009 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.009 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:55.009 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:55.009 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.010 INFO:tasks.workunit.client.0.smithi118.stderr:++ shopt -q login_shell 2023-10-30T12:01:55.010 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:55.010 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:55.010 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:55.010 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:55.011 INFO:tasks.workunit.client.0.smithi118.stderr:+++ umask 2023-10-30T12:01:55.011 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:55.011 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:55.011 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:55.011 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' 0022 -eq 0 ']' 2023-10-30T12:01:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:++ SHELL=/bin/bash 2023-10-30T12:01:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-10-30T12:01:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.012 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/bash_completion.sh 2023-10-30T12:01:55.013 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-10-30T12:01:55.013 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.013 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-10-30T12:01:55.013 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.013 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/cobbler.sh 2023-10-30T12:01:55.014 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:55.014 INFO:tasks.workunit.client.0.smithi118.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-10-30T12:01:55.014 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.014 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-10-30T12:01:55.014 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.014 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorgrep.sh 2023-10-30T12:01:55.015 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:55.015 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:55.015 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:55.015 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:55.015 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:55.016 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:55.016 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:55.016 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.016 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:55.016 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.017 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:55.017 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:55.017 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:55.017 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:55.017 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:55.017 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:55.018 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.018 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:55.018 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.018 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:55.018 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:55.019 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.019 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.019 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:55.019 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-10-30T12:01:55.019 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2023-10-30T12:01:55.020 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.020 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:55.020 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:55.020 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:55.020 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:55.020 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.021 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:55.021 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.021 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:55.021 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:55.021 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:55.022 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.022 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:55.022 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:55.022 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.022 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:55.023 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.023 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:55.023 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:55.023 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-10-30T12:01:55.023 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-10-30T12:01:55.023 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_service_pp: /' 2023-10-30T12:01:55.024 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87764/exe 2023-10-30T12:01:55.024 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'grep=grep --color=auto' 2023-10-30T12:01:55.024 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'egrep=egrep --color=auto' 2023-10-30T12:01:55.024 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-10-30T12:01:55.024 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.025 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-10-30T12:01:55.025 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.025 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:55.025 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorls.sh 2023-10-30T12:01:55.025 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:55.026 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:55.026 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.026 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:55.026 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.026 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:55.027 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' '!' -t 0 ']' 2023-10-30T12:01:55.027 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:55.027 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.027 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:55.027 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.028 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:55.028 INFO:tasks.workunit.client.0.smithi118.stderr:+++ return 2023-10-30T12:01:55.028 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.028 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-10-30T12:01:55.028 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.028 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:55.029 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.029 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:55.029 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:55.029 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.029 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-10-30T12:01:55.030 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export S_COLORS=auto 2023-10-30T12:01:55.030 INFO:tasks.workunit.client.0.smithi118.stderr:+++ S_COLORS=auto 2023-10-30T12:01:55.030 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.030 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-10-30T12:01:55.030 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.031 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-10-30T12:01:55.031 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:55.031 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:55.031 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:55.031 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:55.031 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:55.032 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:55.032 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:55.032 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:55.032 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:55.032 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:55.033 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:55.033 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:55.033 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:55.033 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:55.033 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:55.034 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.034 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:55.034 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.034 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:55.034 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:55.035 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2023-10-30T12:01:55.035 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.035 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:55.035 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.035 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:55.036 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.036 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:55.036 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.036 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:55.036 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:55.037 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-10-30T12:01:55.037 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-10-30T12:01:55.037 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-10-30T12:01:55.037 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-10-30T12:01:55.037 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.037 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:55.038 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-10-30T12:01:55.038 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.038 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-10-30T12:01:55.038 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-10-30T12:01:55.038 INFO:tasks.workunit.client.0.smithi118.stderr:+++ /usr/libexec/grepconf.sh -c 2023-10-30T12:01:55.039 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:55.039 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.039 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:55.039 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.039 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:55.039 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:55.040 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:55.040 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-10-30T12:01:55.040 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-10-30T12:01:55.040 INFO:tasks.workunit.client.0.smithi118.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-10-30T12:01:55.040 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.041 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-10-30T12:01:55.041 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.041 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/debuginfod.sh 2023-10-30T12:01:55.041 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:55.041 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-10-30T12:01:55.042 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-10-30T12:01:55.042 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.042 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-10-30T12:01:55.042 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.042 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/gawk.sh 2023-10-30T12:01:55.043 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:55.043 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:55.043 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.043 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.043 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-10-30T12:01:55.044 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.044 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/lang.sh 2023-10-30T12:01:55.044 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:55.044 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup 2023-10-30T12:01:55.044 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.044 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:55.045 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:55.045 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.045 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.045 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:55.045 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.046 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:55.046 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.046 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:55.046 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:55.046 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.047 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:55.047 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.047 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/locale 2023-10-30T12:01:55.047 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:55.047 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:55.048 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87803/exe 2023-10-30T12:01:55.048 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.048 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:55.048 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.048 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:55.048 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:55.049 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:55.049 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:55.049 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:55.049 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:55.050 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:55.050 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:55.050 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:55.050 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:55.050 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:55.051 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:55.051 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:55.051 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:55.051 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.051 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG_backup=en_US.UTF-8 2023-10-30T12:01:55.052 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.052 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /etc/locale.conf ']' 2023-10-30T12:01:55.052 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -x /usr/bin/sed ']' 2023-10-30T12:01:55.052 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2023-10-30T12:01:55.052 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-10-30T12:01:55.052 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-10-30T12:01:55.053 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:55.053 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:55.053 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:55.053 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.053 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:55.054 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.054 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:55.054 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:55.054 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.054 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.055 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:55.055 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.055 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:55.055 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.055 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:55.055 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:55.056 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.056 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:55.056 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.056 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:55.056 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:55.057 INFO:tasks.workunit.client.0.smithi118.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-10-30T12:01:55.057 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87836/exe 2023-10-30T12:01:55.057 INFO:tasks.workunit.client.0.smithi118.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-10-30T12:01:55.057 INFO:tasks.workunit.client.0.smithi118.stderr:++++ export LANG=en_US.UTF-8 2023-10-30T12:01:55.057 INFO:tasks.workunit.client.0.smithi118.stderr:++++ LANG=en_US.UTF-8 2023-10-30T12:01:55.057 INFO:tasks.workunit.client.0.smithi118.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-10-30T12:01:55.058 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-10-30T12:01:55.058 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.058 INFO:tasks.workunit.client.0.smithi118.stderr:+++ LANG=en_US.UTF-8 2023-10-30T12:01:55.058 INFO:tasks.workunit.client.0.smithi118.stderr:+++ unset LANG_backup config 2023-10-30T12:01:55.058 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:55.059 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n '' ']' 2023-10-30T12:01:55.059 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-10-30T12:01:55.059 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' dumb = linux ']' 2023-10-30T12:01:55.059 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.059 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-10-30T12:01:55.060 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.060 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/less.sh 2023-10-30T12:01:55.060 INFO:tasks.workunit.client.0.smithi118.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-10-30T12:01:55.060 INFO:tasks.workunit.client.0.smithi118.stderr:++ for i in /etc/profile.d/*.sh 2023-10-30T12:01:55.060 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-10-30T12:01:55.061 INFO:tasks.workunit.client.0.smithi118.stderr:++ '[' '' ']' 2023-10-30T12:01:55.061 INFO:tasks.workunit.client.0.smithi118.stderr:++ . /etc/profile.d/which2.sh 2023-10-30T12:01:55.061 INFO:tasks.workunit.client.0.smithi118.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-10-30T12:01:55.061 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:55.061 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:55.061 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:55.062 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:55.062 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:55.062 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:55.062 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:55.062 INFO:tasks.workunit.client.0.smithi118.stderr:+++++ readlink /proc/87788/exe 2023-10-30T12:01:55.063 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:55.063 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:55.063 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:55.063 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2023-10-30T12:01:55.063 INFO:tasks.workunit.client.0.smithi118.stderr:++++ basename /usr/bin/bash 2023-10-30T12:01:55.064 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_declare='declare -f' 2023-10-30T12:01:55.064 INFO:tasks.workunit.client.0.smithi118.stderr:+++ which_opt=-f 2023-10-30T12:01:55.064 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export which_declare 2023-10-30T12:01:55.064 INFO:tasks.workunit.client.0.smithi118.stderr:+++ export -f which 2023-10-30T12:01:55.064 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset i 2023-10-30T12:01:55.064 INFO:tasks.workunit.client.0.smithi118.stderr:++ unset -f pathmunge 2023-10-30T12:01:55.065 INFO:tasks.workunit.client.0.smithi118.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-10-30T12:01:55.065 INFO:tasks.workunit.client.0.smithi118.stderr:+ export PATH 2023-10-30T12:01:55.065 INFO:tasks.workunit.client.0.smithi118.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-10-30T12:01:55.065 INFO:tasks.workunit.client.0.smithi118.stderr:+ unset rc 2023-10-30T12:01:55.065 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-10-30T12:01:55.065 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-10-30T12:01:55.066 INFO:tasks.workunit.client.0.smithi118.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2023-10-30T12:01:55.066 INFO:tasks.workunit.client.0.smithi118.stderr:+ sed 's/^/ list_parallel: /' 2023-10-30T12:01:55.066 INFO:tasks.workunit.client.0.smithi118.stderr:+ tee ceph_test_rados_list_parallel.log 2023-10-30T12:01:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): tick 2023-10-30T12:01:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:01:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 monclient(hunting): continuing hunt 2023-10-30T12:01:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:01:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 msgr2=0x7f0a6813aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a68099a40 0x7f0a6813aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 >> v1:172.21.15.118:6789/0 conn(0x7f0a680a4b60 legacy=0x7f0a681326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 >> v1:172.21.15.50:6789/0 conn(0x7f0a68132db0 legacy=0x7f0a6813cfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient: _add_conns ranks=[2,4] 2023-10-30T12:01:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): picked mon.noname-b con 0x7f0a480036c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): picked mon.noname-c con 0x7f0a48006240 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a48008d90 con 0x7f0a48006240 2023-10-30T12:01:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a48008f00 con 0x7f0a480036c0 2023-10-30T12:01:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a667fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request con 0x7f0a480036c0 auth_method 0 2023-10-30T12:01:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.683+0000 7f0a67fff640 10 monclient(hunting): get_auth_request con 0x7f0a48006240 auth_method 0 2023-10-30T12:01:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a67fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.684+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:55.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:55.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request con 0x7f0a480036c0 auth_method 0 2023-10-30T12:01:55.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:55.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:55.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:55.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:55.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:55.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:55.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:55.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:55.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 10 monclient(hunting): get_auth_request con 0x7f0a48006240 auth_method 0 2023-10-30T12:01:55.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:55.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:55.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:55.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:55.885+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:01:56.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:56.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request con 0x7f0a480036c0 auth_method 0 2023-10-30T12:01:56.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:56.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:56.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:56.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:56.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:56.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:56.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:56.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 10 monclient(hunting): get_auth_request con 0x7f0a48006240 auth_method 0 2023-10-30T12:01:56.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:56.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:56.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:56.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:01:56.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:56.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:56.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:56.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:56.286+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:01:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request con 0x7f0a480036c0 auth_method 0 2023-10-30T12:01:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:01:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:01:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 10 monclient(hunting): get_auth_request con 0x7f0a48006240 auth_method 0 2023-10-30T12:01:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:01:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:01:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:01:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.087+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:01:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:57.088+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:01:57.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-10-30T12:01:57.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [----------] Global test environment set-up. 2023-10-30T12:01:57.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [----------] 12 tests from AsioRados 2023-10-30T12:01:57.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.149+0000 7f5b32851780 10 monclient: get_monmap_and_config 2023-10-30T12:01:57.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.149+0000 7f5b32851780 10 monclient: build_initial_monmap 2023-10-30T12:01:57.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.149+0000 7f5b32851780 10 monclient: monmap: 2023-10-30T12:01:57.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: epoch 0 2023-10-30T12:01:57.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:57.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: last_changed 0.000000 2023-10-30T12:01:57.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: created 0.000000 2023-10-30T12:01:57.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T12:01:57.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: election_strategy: 1 2023-10-30T12:01:57.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:57.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:57.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:57.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.150+0000 7f5b32851780 1 Processor -- start 2023-10-30T12:01:57.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.151+0000 7f5b32851780 1 -- start start 2023-10-30T12:01:57.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.151+0000 7f5b32851780 10 monclient: init 2023-10-30T12:01:57.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient: _reopen_session rank -1 2023-10-30T12:01:57.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:01:57.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b18002260 0x55c10f2a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:57.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): picked mon.noname-b con 0x7f5b18002260 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:57.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c10f3cf4f0 0x55c10f3cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:57.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): picked mon.noname-a con 0x55c10f3cf4f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:57.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f32ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:57.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): picked mon.noname-c con 0x55c10f3cfe10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:57.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:57.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10f234150 con 0x55c10f3cf4f0 2023-10-30T12:01:57.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:57.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10f1cf320 con 0x7f5b18002260 2023-10-30T12:01:57.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:57.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c10f1cefc0 con 0x55c10f3cfe10 2023-10-30T12:01:57.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): _renew_subs 2023-10-30T12:01:57.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b32851780 10 monclient(hunting): authenticate will time out at 948.489390s 2023-10-30T12:01:57.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.152+0000 7f5b3041c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f32ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:57.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f32ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.118:42352/0 (socket says 172.21.15.118:42352) 2023-10-30T12:01:57.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 -- 172.21.15.118:0/339988924 learned_addr learned my addr 172.21.15.118:0/339988924 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:01:57.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): get_auth_request con 0x55c10f3cfe10 auth_method 0 2023-10-30T12:01:57.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:57.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:57.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:57.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:57.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:57.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:57.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient(hunting): handle_auth_done global_id 34172 payload 274 2023-10-30T12:01:57.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 -- 172.21.15.118:0/339988924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b18002260 msgr2=0x55c10f2a9af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:57.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 --2- 172.21.15.118:0/339988924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b18002260 0x55c10f2a9af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 -- 172.21.15.118:0/339988924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c10f3cf4f0 msgr2=0x55c10f3cf8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:57.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 --2- 172.21.15.118:0/339988924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c10f3cf4f0 0x55c10f3cf8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.165 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient: _finish_hunting 0 2023-10-30T12:01:57.165 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 monclient: found mon.noname-c 2023-10-30T12:01:57.165 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:57.165 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:57.165 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 -- 172.21.15.118:0/339988924 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c10f1ce920 con 0x55c10f3cfe10 2023-10-30T12:01:57.166 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.153+0000 7f5b3041c640 1 --2- 172.21.15.118:0/339988924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f32ced0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b200021a0 tx=0x7f5b2000a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a6537be6cce165 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:57.166 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 1 -- 172.21.15.118:0/339988924 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20005370 con 0x55c10f3cfe10 2023-10-30T12:01:57.166 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:57.167 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:57.167 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: dump: 2023-10-30T12:01:57.167 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: epoch 1 2023-10-30T12:01:57.167 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:57.167 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.167 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.168 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:01:57.168 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: election_strategy: 1 2023-10-30T12:01:57.168 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:57.168 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:57.168 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:57.169 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.169 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: _finish_auth 0 2023-10-30T12:01:57.169 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: _check_auth_tickets 2023-10-30T12:01:57.169 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:57.169 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 1 -- 172.21.15.118:0/339988924 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b20002500 con 0x55c10f3cfe10 2023-10-30T12:01:57.169 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b32851780 5 monclient: authenticate success, global_id 34172 2023-10-30T12:01:57.170 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b32851780 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:57.170 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:57.170 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b32851780 10 monclient: get_monmap_and_config success 2023-10-30T12:01:57.170 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b32851780 4 set_mon_vals no callback set 2023-10-30T12:01:57.170 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 1 -- 172.21.15.118:0/339988924 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200028b0 con 0x55c10f3cfe10 2023-10-30T12:01:57.170 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:57.171 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:57.171 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.159+0000 7f5b2e418640 10 monclient: dump: 2023-10-30T12:01:57.171 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: epoch 1 2023-10-30T12:01:57.171 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:57.171 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.172 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.172 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:01:57.172 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: election_strategy: 1 2023-10-30T12:01:57.172 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:57.172 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:57.173 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:57.173 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.173 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.160+0000 7f5b32851780 10 monclient: shutdown 2023-10-30T12:01:57.173 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.160+0000 7f5b32851780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:57.173 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.160+0000 7f5b32851780 1 -- 172.21.15.118:0/339988924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 msgr2=0x55c10f32ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:57.174 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.160+0000 7f5b32851780 1 --2- 172.21.15.118:0/339988924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f32ced0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b200021a0 tx=0x7f5b2000a4b0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.174 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 1 -- 172.21.15.118:0/339988924 shutdown_connections 2023-10-30T12:01:57.174 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 1 --2- 172.21.15.118:0/339988924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f32ced0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.174 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 1 --2- 172.21.15.118:0/339988924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x55c10f3cf4f0 0x55c10f3cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.174 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 1 --2- 172.21.15.118:0/339988924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b18002260 0x55c10f2a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.175 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 1 -- 172.21.15.118:0/339988924 >> 172.21.15.118:0/339988924 conn(0x55c10f2aaf30 msgr2=0x55c10f27e210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:57.175 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 1 -- 172.21.15.118:0/339988924 shutdown_connections 2023-10-30T12:01:57.175 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 1 -- 172.21.15.118:0/339988924 wait complete. 2023-10-30T12:01:57.175 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 10 monclient: build_initial_monmap 2023-10-30T12:01:57.175 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.161+0000 7f5b32851780 10 monclient: monmap: 2023-10-30T12:01:57.175 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: epoch 0 2023-10-30T12:01:57.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:57.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: last_changed 0.000000 2023-10-30T12:01:57.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: created 0.000000 2023-10-30T12:01:57.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: min_mon_release 0 (unknown) 2023-10-30T12:01:57.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: election_strategy: 1 2023-10-30T12:01:57.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:57.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:57.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:57.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.162+0000 7f5b32851780 1 librados: starting msgr at 2023-10-30T12:01:57.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.162+0000 7f5b32851780 1 librados: starting objecter 2023-10-30T12:01:57.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.162+0000 7f5b32851780 1 Processor -- start 2023-10-30T12:01:57.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.162+0000 7f5b32851780 1 -- start start 2023-10-30T12:01:57.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.162+0000 7f5b32851780 1 librados: setting wanted keys 2023-10-30T12:01:57.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.162+0000 7f5b32851780 1 librados: calling monclient init 2023-10-30T12:01:57.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.162+0000 7f5b32851780 10 monclient: init 2023-10-30T12:01:57.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient: _reopen_session rank -1 2023-10-30T12:01:57.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:01:57.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c10f3cf4f0 0x55c10f330510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:57.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): picked mon.noname-b con 0x55c10f3cf4f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:57.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f330a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:57.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): picked mon.noname-c con 0x55c10f3cfe10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:57.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18002260 0x55c10f330f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:57.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): picked mon.noname-a con 0x7f5b18002260 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:57.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:57.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10f1af4e0 con 0x7f5b18002260 2023-10-30T12:01:57.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:57.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10f333830 con 0x55c10f3cf4f0 2023-10-30T12:01:57.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:57.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c10f2a5420 con 0x55c10f3cfe10 2023-10-30T12:01:57.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): _renew_subs 2023-10-30T12:01:57.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b32851780 10 monclient(hunting): authenticate will time out at 948.500360s 2023-10-30T12:01:57.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c10f3cf4f0 0x55c10f330510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:57.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b3041c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18002260 msgr2=0x55c10f330f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:57.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b3041c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18002260 0x55c10f330f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:57.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c10f3cf4f0 0x55c10f330510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45108/0 (socket says 172.21.15.118:45108) 2023-10-30T12:01:57.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 -- 172.21.15.118:0/1944996220 learned_addr learned my addr 172.21.15.118:0/1944996220 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-30T12:01:57.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): get_auth_request con 0x55c10f3cf4f0 auth_method 0 2023-10-30T12:01:57.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:57.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:57.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:57.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:57.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:57.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:57.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2f41a640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f330a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:57.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2f41a640 10 monclient(hunting): get_auth_request con 0x55c10f3cfe10 auth_method 0 2023-10-30T12:01:57.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2f41a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:57.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2f41a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:57.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2f41a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:57.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient(hunting): handle_auth_done global_id 4654 payload 824 2023-10-30T12:01:57.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 -- 172.21.15.118:0/1944996220 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 msgr2=0x55c10f330a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:57.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f330a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 -- 172.21.15.118:0/1944996220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18002260 msgr2=0x55c10f330f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:57.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b18002260 0x55c10f330f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:57.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient: _finish_hunting 0 2023-10-30T12:01:57.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 monclient: found mon.noname-b 2023-10-30T12:01:57.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:57.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:01:57.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2fc1b640 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b20005be0 con 0x55c10f3cf4f0 2023-10-30T12:01:57.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.163+0000 7f5b2f41a640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x55c10f3cfe10 0x55c10f330a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:01:57.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.164+0000 7f5b2fc1b640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c10f3cf4f0 0x55c10f330510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5b280499d0 tx=0x7f5b28003330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:57.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.164+0000 7f5b2d416640 10 client.?.objecter ms_handle_connect 0x55c10f3cf4f0 2023-10-30T12:01:57.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.164+0000 7f5b2d416640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:57.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28059400 con 0x55c10f3cf4f0 2023-10-30T12:01:57.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b2d416640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:57.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b2d416640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:57.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b2d416640 10 monclient: dump: 2023-10-30T12:01:57.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: epoch 1 2023-10-30T12:01:57.200 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:57.277 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.277 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.277 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:01:57.278 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: election_strategy: 1 2023-10-30T12:01:57.278 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:57.278 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:57.278 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:57.279 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.279 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b2d416640 10 monclient: _finish_auth 0 2023-10-30T12:01:57.279 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b2d416640 10 monclient: _check_auth_tickets 2023-10-30T12:01:57.280 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b2d416640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:57.280 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 5 monclient: authenticate success, global_id 4654 2023-10-30T12:01:57.280 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 10 monclient: _renew_subs 2023-10-30T12:01:57.280 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:57.280 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c10f1ce380 con 0x55c10f3cf4f0 2023-10-30T12:01:57.281 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:57.281 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 10 monclient: _renew_subs 2023-10-30T12:01:57.281 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:57.281 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c10f333550 con 0x55c10f3cf4f0 2023-10-30T12:01:57.281 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 1 librados: init done 2023-10-30T12:01:57.282 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.193+0000 7f5b32851780 10 librados: wait_for_osdmap waiting 2023-10-30T12:01:57.282 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.250+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b28005840 con 0x55c10f3cf4f0 2023-10-30T12:01:57.282 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.250+0000 7f5b2d416640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:57.282 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.250+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28059bc0 con 0x55c10f3cf4f0 2023-10-30T12:01:57.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.250+0000 7f5b2d416640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:57.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.250+0000 7f5b2d416640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:57.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.250+0000 7f5b2d416640 10 monclient: dump: 2023-10-30T12:01:57.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: epoch 1 2023-10-30T12:01:57.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:57.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:57.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: min_mon_release 18 (reef) 2023-10-30T12:01:57.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: election_strategy: 1 2023-10-30T12:01:57.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:57.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:57.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:57.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.250+0000 7f5b2e418640 4 set_mon_vals no callback set 2023-10-30T12:01:57.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.271+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5b28003040 con 0x55c10f3cf4f0 2023-10-30T12:01:58.132 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.275+0000 7f5b2d416640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5b1c037490 0x7f5b1c039950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.275+0000 7f5b2f41a640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5b1c037490 0x7f5b1c039950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:58.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.275+0000 7f5b2f41a640 10 monclient: get_auth_request con 0x7f5b1c037490 auth_method 0 2023-10-30T12:01:58.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.276+0000 7f5b2d416640 10 client.4654.objecter ms_handle_connect 0x7f5b1c037490 2023-10-30T12:01:58.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.276+0000 7f5b2f41a640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5b1c037490 0x7f5b1c039950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55c10f271b80 tx=0x7f5b2400a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:58.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.291+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f5b2800c490 con 0x55c10f3cf4f0 2023-10-30T12:01:58.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.291+0000 7f5b2d416640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_map(16..16 src has 1..16) v4 2023-10-30T12:01:58.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.291+0000 7f5b2d416640 3 client.4654.objecter handle_osd_map got epochs [16,16] > 0 2023-10-30T12:01:58.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.291+0000 7f5b2d416640 3 client.4654.objecter handle_osd_map decoding full epoch 16 2023-10-30T12:01:58.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.292+0000 7f5b2d416640 20 client.4654.objecter dump_active .. 0 homeless 2023-10-30T12:01:58.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.292+0000 7f5b32851780 10 librados: wait_for_osdmap done waiting 2023-10-30T12:01:58.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.292+0000 7f5b32851780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:58.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.292+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5b28053070 con 0x55c10f3cf4f0 2023-10-30T12:01:58.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.364+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b28062980 con 0x55c10f3cf4f0 2023-10-30T12:01:58.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.364+0000 7f5b2d416640 10 monclient: handle_get_version_reply finishing 1 version 16 2023-10-30T12:01:58.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.364+0000 7f5b2dc17640 10 client.4654.objecter _get_latest_version latest 16, have it 2023-10-30T12:01:58.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.364+0000 7f5b32851780 10 client.4654.objecter create_pool name=ceph_test_rados_api_asio 2023-10-30T12:01:58.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.364+0000 7f5b32851780 10 client.4654.objecter pool_op_submit 1 2023-10-30T12:01:58.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.364+0000 7f5b32851780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:58.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:57.364+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55c10f23e0d0 con 0x55c10f3cf4f0 2023-10-30T12:01:58.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v17) v1 ==== 43+0+0 (secure 0 0 0) 0x7f5b28068030 con 0x55c10f3cf4f0 2023-10-30T12:01:58.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 pool_op_reply(tid 1 (0) Success v17) v1 2023-10-30T12:01:58.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 10 client.4654.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v17) v1 2023-10-30T12:01:58.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 10 client.4654.objecter have request 1 at 0x55c10f3397e0 Op: create 2023-10-30T12:01:58.138 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 20 client.4654.objecter waiting for client to reach epoch 17 before calling back 2023-10-30T12:01:58.138 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:58.138 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 10 monclient: _renew_subs 2023-10-30T12:01:58.138 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.130+0000 7f5b2d416640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:01:58.138 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.131+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=17}) v3 -- 0x7f5b1c03cc40 con 0x55c10f3cf4f0 2023-10-30T12:01:58.139 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.131+0000 7f5b2d416640 10 client.4654.objecter done 2023-10-30T12:01:58.139 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.131+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== mon.1 v2:172.21.15.118:3300/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 9487+0+0 (secure 0 0 0) 0x7f5b2806c7d0 con 0x55c10f3cf4f0 2023-10-30T12:01:58.139 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.131+0000 7f5b2d416640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_map(17..17 src has 1..17) v4 2023-10-30T12:01:58.139 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.131+0000 7f5b2d416640 3 client.4654.objecter handle_osd_map got epochs [17,17] > 16 2023-10-30T12:01:58.139 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.131+0000 7f5b2d416640 3 client.4654.objecter handle_osd_map decoding incremental epoch 17 2023-10-30T12:01:58.140 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b2d416640 20 client.4654.objecter dump_active .. 0 homeless 2023-10-30T12:01:58.140 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 10 librados: set snap read head -> 1 2023-10-30T12:01:58.140 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 10 librados: writefull oid=exist nspace= 2023-10-30T12:01:58.140 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3e17a0 2023-10-30T12:01:58.140 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 17 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:58.140 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:58.141 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:58.141 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x55c10f3e2150 0x55c10f3e4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.141 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:58.141 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 2 osd.3 2023-10-30T12:01:58.141 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:58.141 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 2 2023-10-30T12:01:58.142 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 15 client.4654.objecter _send_op 2 to 15.1 on osd.3 2023-10-30T12:01:58.142 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:2 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) v8 -- 0x55c10f3e4a80 con 0x55c10f3e2150 2023-10-30T12:01:58.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): tick 2023-10-30T12:01:58.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:01:58.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 monclient(hunting): continuing hunt 2023-10-30T12:01:58.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:01:58.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:58.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:58.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T12:01:58.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 -- 172.21.15.50:0/3360253947 reap_dead start 2023-10-30T12:01:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): picked mon.noname-c con 0x7f0a48006240 addr v2:172.21.15.118:3300/0 2023-10-30T12:01:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a480092c0 0x7f0a4800b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): picked mon.noname-a con 0x7f0a480092c0 addr v2:172.21.15.50:3300/0 2023-10-30T12:01:58.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): picked mon.noname-b con 0x7f0a480036c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:58.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:58.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a48008d90 con 0x7f0a480092c0 2023-10-30T12:01:58.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:58.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a48008f00 con 0x7f0a48006240 2023-10-30T12:01:58.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:58.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 1 -- 172.21.15.50:0/3360253947 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4800bc20 con 0x7f0a480036c0 2023-10-30T12:01:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a667fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a67fff640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a480092c0 msgr2=0x7f0a4800b6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:58.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a480092c0 0x7f0a4800b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): get_auth_request con 0x7f0a480036c0 auth_method 0 2023-10-30T12:01:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:58.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:58.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:58.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:58.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request con 0x7f0a48006240 auth_method 0 2023-10-30T12:01:58.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:58.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:58.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:58.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:58.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:58.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:58.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:58.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:58.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6ca3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:58.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_done global_id 34262 payload 274 2023-10-30T12:01:58.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 msgr2=0x7f0a48008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:58.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 -- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a480092c0 msgr2=0x7f0a4800b6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:58.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a480092c0 0x7f0a4800b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient: _finish_hunting 0 2023-10-30T12:01:58.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 monclient: found mon.noname-b 2023-10-30T12:01:58.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:58.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:01:58.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.684+0000 7f0a6d23f640 1 -- 172.21.15.50:0/3360253947 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6809a440 con 0x7f0a480036c0 2023-10-30T12:01:58.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a6d23f640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a680a3820 tx=0x7f0a5002b600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=307291eaa1c9e247 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:58.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 1 -- 172.21.15.50:0/3360253947 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50003b60 con 0x7f0a480036c0 2023-10-30T12:01:58.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:58.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:58.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 10 monclient: dump: 2023-10-30T12:01:58.703 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:58.703 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:58.703 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.703 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.704 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:58.704 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:58.704 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:58.704 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:58.704 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:58.705 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:58.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 10 monclient: _finish_auth 0 2023-10-30T12:01:58.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 10 monclient: _check_auth_tickets 2023-10-30T12:01:58.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:58.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a6eca2640 5 monclient: authenticate success, global_id 34262 2023-10-30T12:01:58.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a6eca2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:01:58.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.685+0000 7f0a677fe640 1 -- 172.21.15.50:0/3360253947 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5002e070 con 0x7f0a480036c0 2023-10-30T12:01:58.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a677fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:58.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:58.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 4 set_mon_vals no callback set 2023-10-30T12:01:58.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a677fe640 1 -- 172.21.15.50:0/3360253947 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50005070 con 0x7f0a480036c0 2023-10-30T12:01:58.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:58.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:58.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a677fe640 10 monclient: dump: 2023-10-30T12:01:58.708 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:58.708 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:58.708 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.708 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.709 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:58.709 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:58.709 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:58.709 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:58.709 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:58.710 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:58.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 10 monclient: shutdown 2023-10-30T12:01:58.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:58.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 -- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 msgr2=0x7f0a48005b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:58.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a680a3820 tx=0x7f0a5002b600 comp rx=0 tx=0).stop 2023-10-30T12:01:58.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 -- 172.21.15.50:0/3360253947 shutdown_connections 2023-10-30T12:01:58.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/3360253947 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480036c0 0x7f0a48005b10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.50:3300/0 conn(0x7f0a480092c0 0x7f0a4800b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/3360253947 >> v2:172.21.15.118:3300/0 conn(0x7f0a48006240 0x7f0a48008680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 -- 172.21.15.50:0/3360253947 >> 172.21.15.50:0/3360253947 conn(0x7f0a680a8060 msgr2=0x7f0a680a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:58.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 -- 172.21.15.50:0/3360253947 shutdown_connections 2023-10-30T12:01:58.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.686+0000 7f0a6eca2640 1 -- 172.21.15.50:0/3360253947 wait complete. 2023-10-30T12:01:58.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.687+0000 7f0a6eca2640 10 monclient: build_initial_monmap 2023-10-30T12:01:58.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.687+0000 7f0a6eca2640 10 monclient: monmap: 2023-10-30T12:01:58.719 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:58.720 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:58.720 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:58.720 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:58.720 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:58.721 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:58.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.687+0000 7f0a6eca2640 1 librados: starting msgr at 2023-10-30T12:01:58.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.687+0000 7f0a6eca2640 1 librados: starting objecter 2023-10-30T12:01:58.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 1 Processor -- start 2023-10-30T12:01:58.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 1 -- start start 2023-10-30T12:01:58.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 1 librados: setting wanted keys 2023-10-30T12:01:58.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 1 librados: calling monclient init 2023-10-30T12:01:58.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 10 monclient: init 2023-10-30T12:01:58.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:58.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:01:58.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a480036c0 0x7f0a681c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 10 monclient(hunting): picked mon.noname-a con 0x7f0a480036c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:58.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.688+0000 7f0a6eca2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a48006240 0x7f0a68131700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 10 monclient(hunting): picked mon.noname-b con 0x7f0a48006240 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:58.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480092c0 0x7f0a68131c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 10 monclient(hunting): picked mon.noname-c con 0x7f0a480092c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:58.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:58.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a680a18d0 con 0x7f0a480036c0 2023-10-30T12:01:58.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:58.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a681ca580 con 0x7f0a48006240 2023-10-30T12:01:58.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6811f600 con 0x7f0a480092c0 2023-10-30T12:01:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6eca2640 10 monclient(hunting): authenticate will time out at 963.872050s 2023-10-30T12:01:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6ca3e640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a480036c0 msgr2=0x7f0a681c9bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6ca3e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a480036c0 0x7f0a681c9bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480092c0 0x7f0a68131c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480092c0 0x7f0a68131c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57882/0 (socket says 172.21.15.50:57882) 2023-10-30T12:01:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 -- 172.21.15.50:0/553290184 learned_addr learned my addr 172.21.15.50:0/553290184 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): get_auth_request con 0x7f0a480092c0 auth_method 0 2023-10-30T12:01:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:58.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:58.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:58.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a48006240 0x7f0a68131700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:58.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 10 monclient(hunting): get_auth_request con 0x7f0a48006240 auth_method 0 2023-10-30T12:01:58.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:58.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:58.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:58.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:58.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:58.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:58.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:58.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:58.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:58.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient(hunting): handle_auth_done global_id 34265 payload 824 2023-10-30T12:01:58.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 -- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a48006240 msgr2=0x7f0a68131700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:58.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a48006240 0x7f0a68131700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 -- 172.21.15.50:0/553290184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a480036c0 msgr2=0x7f0a681c9bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:58.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a480036c0 0x7f0a681c9bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient: _finish_hunting 0 2023-10-30T12:01:58.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 monclient: found mon.noname-c 2023-10-30T12:01:58.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:58.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:58.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 -- 172.21.15.50:0/553290184 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50009460 con 0x7f0a480092c0 2023-10-30T12:01:58.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a6d23f640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480092c0 0x7f0a68131c40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a68133190 tx=0x7f0a50005980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:58.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a657fa640 10 client.?.objecter ms_handle_connect 0x7f0a480092c0 2023-10-30T12:01:58.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.689+0000 7f0a657fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 1 -- 172.21.15.50:0/553290184 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50005280 con 0x7f0a480092c0 2023-10-30T12:01:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: dump: 2023-10-30T12:01:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:58.737 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:58.737 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.737 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.737 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:58.737 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:58.738 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:58.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: _finish_auth 0 2023-10-30T12:01:58.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: _check_auth_tickets 2023-10-30T12:01:58.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:58.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 1 -- 172.21.15.50:0/553290184 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a500053e0 con 0x7f0a480092c0 2023-10-30T12:01:58.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 5 monclient: authenticate success, global_id 34265 2023-10-30T12:01:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 10 monclient: _renew_subs 2023-10-30T12:01:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6811f8e0 con 0x7f0a480092c0 2023-10-30T12:01:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 1 -- 172.21.15.50:0/553290184 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50005790 con 0x7f0a480092c0 2023-10-30T12:01:58.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:58.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:58.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a657fa640 10 monclient: dump: 2023-10-30T12:01:58.741 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:58.741 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:58.742 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.742 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:58.742 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:58.742 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:58.742 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:58.743 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:58.743 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:58.743 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:58.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a66ffd640 4 set_mon_vals no callback set 2023-10-30T12:01:58.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 10 client.34265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:58.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 10 monclient: _renew_subs 2023-10-30T12:01:58.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:58.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6809ffa0 con 0x7f0a480092c0 2023-10-30T12:01:58.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.690+0000 7f0a6eca2640 1 librados: init done 2023-10-30T12:01:58.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a657fa640 1 -- 172.21.15.50:0/553290184 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0a50030030 con 0x7f0a480092c0 2023-10-30T12:01:58.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a657fa640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a4803a790 0x7f0a4803cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:58.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a657fa640 1 -- 172.21.15.50:0/553290184 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 13545+0+0 (secure 0 0 0) 0x7f0a5006dcb0 con 0x7f0a480092c0 2023-10-30T12:01:58.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a6eca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:58.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a657fa640 10 client.34265.objecter ms_dispatch 0x7f0a68000b90 osd_map(17..17 src has 1..17) v4 2023-10-30T12:01:58.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a657fa640 3 client.34265.objecter handle_osd_map got epochs [17,17] > 0 2023-10-30T12:01:58.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a6eca2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:58.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a657fa640 3 client.34265.objecter handle_osd_map decoding full epoch 17 2023-10-30T12:01:58.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a6eca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:58.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6811fda0 con 0x7f0a480092c0 2023-10-30T12:01:58.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a657fa640 20 client.34265.objecter dump_active .. 0 homeless 2023-10-30T12:01:58.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a4803a790 0x7f0a4803cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:58.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.691+0000 7f0a6ca3e640 10 monclient: get_auth_request con 0x7f0a4803a790 auth_method 0 2023-10-30T12:01:58.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.692+0000 7f0a6ca3e640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a4803a790 0x7f0a4803cc50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a60001a10 tx=0x7f0a60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:58.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.692+0000 7f0a657fa640 10 client.34265.objecter ms_handle_connect 0x7f0a4803a790 2023-10-30T12:01:58.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.696+0000 7f0a657fa640 1 -- 172.21.15.50:0/553290184 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0a50040070 con 0x7f0a480092c0 2023-10-30T12:01:58.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.696+0000 7f0a657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:58.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.696+0000 7f0a657fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:58.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.942+0000 7f0a6eca2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:01:58.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.942+0000 7f0a6eca2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:58.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.942+0000 7f0a6eca2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:58.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.942+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0a6804e160 con 0x7f0a480092c0 2023-10-30T12:01:58.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.943+0000 7f0a657fa640 1 -- 172.21.15.50:0/553290184 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f0a5003b360 con 0x7f0a480092c0 2023-10-30T12:01:58.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.943+0000 7f0a657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:01:58.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.943+0000 7f0a657fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:58.946 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:01:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 10 librados: watch_flush enter 2023-10-30T12:01:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 10 librados: watch_flush exit 2023-10-30T12:01:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 20 client.34265.objecter shutdown clearing up homeless session... 2023-10-30T12:01:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 10 client.34265.objecter successfully canceled tick 2023-10-30T12:01:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a4803a790 msgr2=0x7f0a4803cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a4803a790 0x7f0a4803cc50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a60001a10 tx=0x7f0a60009040 comp rx=0 tx=0).stop 2023-10-30T12:01:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 10 monclient: shutdown 2023-10-30T12:01:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480092c0 msgr2=0x7f0a68131c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:58.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480092c0 0x7f0a68131c40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a68133190 tx=0x7f0a50005980 comp rx=0 tx=0).stop 2023-10-30T12:01:58.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 shutdown_connections 2023-10-30T12:01:58.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a4803a790 0x7f0a4803cc50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a480092c0 0x7f0a68131c40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a48006240 0x7f0a68131700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 --2- 172.21.15.50:0/553290184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a480036c0 0x7f0a681c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:58.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 >> 172.21.15.50:0/553290184 conn(0x7f0a680a8060 msgr2=0x7f0a681bb640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:58.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.944+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 shutdown_connections 2023-10-30T12:01:58.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.945+0000 7f0a6eca2640 1 -- 172.21.15.50:0/553290184 wait complete. 2023-10-30T12:01:58.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:58.945+0000 7f0a6eca2640 1 librados: shutdown 2023-10-30T12:01:58.957 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:01:58.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:01:58.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:01:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.061+0000 7f1ec02c2640 10 monclient: get_monmap_and_config 2023-10-30T12:01:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.061+0000 7f1ec02c2640 10 monclient: build_initial_monmap 2023-10-30T12:01:59.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.061+0000 7f1ec02c2640 10 monclient: monmap: 2023-10-30T12:01:59.064 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:59.064 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.064 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:59.062931+0000 2023-10-30T12:01:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:59.062931+0000 2023-10-30T12:01:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:59.067 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.062+0000 7f1ec02c2640 1 Processor -- start 2023-10-30T12:01:59.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.062+0000 7f1ec02c2640 1 -- start start 2023-10-30T12:01:59.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.062+0000 7f1ec02c2640 10 monclient: init 2023-10-30T12:01:59.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:59.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-10-30T12:01:59.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb806afc0 0x7f1eb80a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb806afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:59.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1eb806b640 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:59.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1eb813b070 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:59.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb806bde0 con 0x7f1eb813b070 2023-10-30T12:01:59.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb806b3a0 con 0x7f1eb806b640 2023-10-30T12:01:59.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb809e540 con 0x7f1eb806afc0 2023-10-30T12:01:59.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:59.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ec02c2640 10 monclient(hunting): authenticate will time out at 964.246110s 2023-10-30T12:01:59.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe85f640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f1eb813b070 legacy=0x7f1eb813d460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:59.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb806afc0 0x7f1eb80a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb806afc0 0x7f1eb80a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57886/0 (socket says 172.21.15.50:57886) 2023-10-30T12:01:59.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 1 -- 172.21.15.50:0/961403011 learned_addr learned my addr 172.21.15.50:0/961403011 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:59.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): get_auth_request con 0x7f1eb806afc0 auth_method 0 2023-10-30T12:01:59.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_done global_id 34268 payload 274 2023-10-30T12:01:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 1 -- 172.21.15.50:0/961403011 >> v1:172.21.15.118:6789/0 conn(0x7f1eb806b640 legacy=0x7f1eb80a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:01:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 1 -- 172.21.15.50:0/961403011 >> v1:172.21.15.50:6789/0 conn(0x7f1eb813b070 legacy=0x7f1eb813d460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient: _finish_hunting 0 2023-10-30T12:01:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 1 monclient: found mon.noname-b 2023-10-30T12:01:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.063+0000 7f1ebe05e640 1 -- 172.21.15.50:0/961403011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb809fa10 con 0x7f1eb806afc0 2023-10-30T12:01:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebe05e640 1 --2- 172.21.15.50:0/961403011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb806afc0 0x7f1eb80a1c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1eb806b530 tx=0x7f1eb402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1cc41864a963c7b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 1 -- 172.21.15.50:0/961403011 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4031710 con 0x7f1eb806afc0 2023-10-30T12:01:59.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:59.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: dump: 2023-10-30T12:01:59.087 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.088 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.088 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.088 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.088 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.088 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.089 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.089 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.089 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.089 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: _finish_auth 0 2023-10-30T12:01:59.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: _check_auth_tickets 2023-10-30T12:01:59.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:59.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 1 -- 172.21.15.50:0/961403011 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb402f050 con 0x7f1eb806afc0 2023-10-30T12:01:59.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:59.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 1 -- 172.21.15.50:0/961403011 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4031a90 con 0x7f1eb806afc0 2023-10-30T12:01:59.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ec02c2640 5 monclient: authenticate success, global_id 34268 2023-10-30T12:01:59.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ec02c2640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:59.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ec02c2640 4 set_mon_vals no callback set 2023-10-30T12:01:59.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:59.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ebd05c640 10 monclient: dump: 2023-10-30T12:01:59.092 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.092 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.092 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.092 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.093 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.093 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.093 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.093 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.094 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.094 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ec02c2640 10 monclient: shutdown 2023-10-30T12:01:59.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ec02c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:59.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ec02c2640 1 -- 172.21.15.50:0/961403011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb806afc0 msgr2=0x7f1eb80a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.064+0000 7f1ec02c2640 1 --2- 172.21.15.50:0/961403011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb806afc0 0x7f1eb80a1c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1eb806b530 tx=0x7f1eb402b7b0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 -- 172.21.15.50:0/961403011 shutdown_connections 2023-10-30T12:01:59.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 --2- 172.21.15.50:0/961403011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb806afc0 0x7f1eb80a1c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 -- 172.21.15.50:0/961403011 >> 172.21.15.50:0/961403011 conn(0x7f1eb80a6200 msgr2=0x7f1eb80a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:59.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 -- 172.21.15.50:0/961403011 shutdown_connections 2023-10-30T12:01:59.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 -- 172.21.15.50:0/961403011 wait complete. 2023-10-30T12:01:59.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 10 monclient: build_initial_monmap 2023-10-30T12:01:59.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 10 monclient: monmap: 2023-10-30T12:01:59.097 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:59.097 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:59.097 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:59.097 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:59.098 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:59.098 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.098 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:59.099 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:59.099 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:59.099 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 librados: starting msgr at 2023-10-30T12:01:59.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 librados: starting objecter 2023-10-30T12:01:59.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 Processor -- start 2023-10-30T12:01:59.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 -- start start 2023-10-30T12:01:59.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 librados: setting wanted keys 2023-10-30T12:01:59.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 1 librados: calling monclient init 2023-10-30T12:01:59.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.065+0000 7f1ec02c2640 10 monclient: init 2023-10-30T12:01:59.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:59.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:01:59.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb806afc0 0x7f1eb8138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb806afc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:59.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb806b640 0x7f1eb81218b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb806b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:59.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb813b070 0x7f1eb8121df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb813b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:59.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8125910 con 0x7f1eb806b640 2023-10-30T12:01:59.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb81c0160 con 0x7f1eb806afc0 2023-10-30T12:01:59.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80704b0 con 0x7f1eb813b070 2023-10-30T12:01:59.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:59.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ec02c2640 10 monclient(hunting): authenticate will time out at 964.249460s 2023-10-30T12:01:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebd85d640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb806b640 msgr2=0x7f1eb81218b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebd85d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb806b640 0x7f1eb81218b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb813b070 0x7f1eb8121df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb813b070 0x7f1eb8121df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57898/0 (socket says 172.21.15.50:57898) 2023-10-30T12:01:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 -- 172.21.15.50:0/804703862 learned_addr learned my addr 172.21.15.50:0/804703862 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): get_auth_request con 0x7f1eb813b070 auth_method 0 2023-10-30T12:01:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient(hunting): handle_auth_done global_id 34271 payload 824 2023-10-30T12:01:59.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb806afc0 msgr2=0x7f1eb8138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe05e640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb806afc0 0x7f1eb8138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb806afc0 0x7f1eb8138ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb806b640 msgr2=0x7f1eb81218b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:59.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb806b640 0x7f1eb81218b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient: _finish_hunting 0 2023-10-30T12:01:59.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 monclient: found mon.noname-c 2023-10-30T12:01:59.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:59.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 -- 172.21.15.50:0/804703862 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb40094d0 con 0x7f1eb813b070 2023-10-30T12:01:59.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1ebe85f640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb813b070 0x7f1eb8121df0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000be80 tx=0x7f1eb000beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.066+0000 7f1eaeffd640 10 client.?.objecter ms_handle_connect 0x7f1eb813b070 2023-10-30T12:01:59.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:59.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0016430 con 0x7f1eb813b070 2023-10-30T12:01:59.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:59.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: dump: 2023-10-30T12:01:59.112 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.112 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.112 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.113 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.113 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.113 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.113 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.113 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.114 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.114 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: _finish_auth 0 2023-10-30T12:01:59.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:01:59.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:59.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb0016590 con 0x7f1eb813b070 2023-10-30T12:01:59.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 5 monclient: authenticate success, global_id 34271 2023-10-30T12:01:59.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 10 monclient: _renew_subs 2023-10-30T12:01:59.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb8070de0 con 0x7f1eb813b070 2023-10-30T12:01:59.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:59.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0016940 con 0x7f1eb813b070 2023-10-30T12:01:59.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:59.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 10 monclient: dump: 2023-10-30T12:01:59.117 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.117 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.117 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.117 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.117 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.118 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.118 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.118 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.118 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.118 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ebc85b640 4 set_mon_vals no callback set 2023-10-30T12:01:59.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 10 client.34271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:59.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 10 monclient: _renew_subs 2023-10-30T12:01:59.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb8124fe0 con 0x7f1eb813b070 2023-10-30T12:01:59.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 1 librados: init done 2023-10-30T12:01:59.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:59.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1eb0012020 con 0x7f1eb813b070 2023-10-30T12:01:59.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb8122330 0x7f1eb8122710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.067+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8073ff0 con 0x7f1eb8122330 2023-10-30T12:01:59.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb8122330 0x7f1eb8122710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient: get_auth_request con 0x7f1eb8122330 auth_method 0 2023-10-30T12:01:59.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e9c0392d0 0x7f1e9c03b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 13545+0+0 (secure 0 0 0) 0x7f1eb004cb90 con 0x7f1eb813b070 2023-10-30T12:01:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 client.34271.objecter ms_dispatch 0x7f1eb8000b90 osd_map(17..17 src has 1..17) v4 2023-10-30T12:01:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 3 client.34271.objecter handle_osd_map got epochs [17,17] > 0 2023-10-30T12:01:59.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 3 client.34271.objecter handle_osd_map decoding full epoch 17 2023-10-30T12:01:59.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_done global_id 34274 payload 824 2023-10-30T12:01:59.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 1 -- 172.21.15.50:0/804703862 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1eb809f0b0 con 0x7f1eb8122330 2023-10-30T12:01:59.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebd85d640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e9c0392d0 0x7f1e9c03b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebd85d640 10 monclient: get_auth_request con 0x7f1e9c0392d0 auth_method 0 2023-10-30T12:01:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 20 client.34271.objecter dump_active .. 0 homeless 2023-10-30T12:01:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 client.34271.objecter ms_handle_connect 0x7f1eb8122330 2023-10-30T12:01:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 client.34271.objecter resend_mon_ops 2023-10-30T12:01:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebe05e640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb8122330 0x7f1eb8122710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80712a0 tx=0x7f1eb40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4033890 con 0x7f1eb8122330 2023-10-30T12:01:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 monclient: dump: 2023-10-30T12:01:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.129 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1eb40339f0 con 0x7f1eb8122330 2023-10-30T12:01:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:59.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:59.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1ebd85d640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e9c0392d0 0x7f1e9c03b790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8121850 tx=0x7f1ea8007a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb8122330 msgr2=0x7f1eb8122710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb8122330 0x7f1eb8122710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80712a0 tx=0x7f1eb40046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb8122330 msgr2=0x7f1eb8122710 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:59.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.068+0000 7f1eaeffd640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb8122330 0x7f1eb8122710 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.069+0000 7f1eaeffd640 10 client.34271.objecter ms_handle_connect 0x7f1e9c0392d0 2023-10-30T12:01:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ec02c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ec02c2640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb811deb0 0x7f1eb811e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ec02c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb809df80 con 0x7f1eb811deb0 2023-10-30T12:01:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb811deb0 0x7f1eb811e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient: get_auth_request con 0x7f1eb811deb0 auth_method 0 2023-10-30T12:01:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 10 monclient(hunting): handle_auth_done global_id 34277 payload 824 2023-10-30T12:01:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 1 -- 172.21.15.50:0/804703862 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1eb806aa80 con 0x7f1eb811deb0 2023-10-30T12:01:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1eaeffd640 10 client.34271.objecter ms_handle_connect 0x7f1eb811deb0 2023-10-30T12:01:59.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1eaeffd640 10 client.34271.objecter resend_mon_ops 2023-10-30T12:01:59.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1ebe05e640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb811deb0 0x7f1eb811e2c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40097f0 tx=0x7f1eb4031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4052860 con 0x7f1eb811deb0 2023-10-30T12:01:59.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1eaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1eaeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:59.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.075+0000 7f1eaeffd640 10 monclient: dump: 2023-10-30T12:01:59.137 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.137 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.139 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4621 (secure 0 0 0) 0x7f1eb40529c0 con 0x7f1eb811deb0 2023-10-30T12:01:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1eaeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1eaeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb811deb0 msgr2=0x7f1eb811e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1eaeffd640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb811deb0 0x7f1eb811e2c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40097f0 tx=0x7f1eb4031000 comp rx=0 tx=0).stop 2023-10-30T12:01:59.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1eaeffd640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb811deb0 msgr2=0x7f1eb811e2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:59.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1eaeffd640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb811deb0 0x7f1eb811e2c0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40097f0 tx=0x7f1eb4031000 comp rx=0 tx=0).stop 2023-10-30T12:01:59.141 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:59.142 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:59.142 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:59.142 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:01:59.142 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 12, 2023-10-30T12:01:59.143 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:59.143 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:59.143 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:59.143 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.143 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:01:59.144 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:59.144 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:59.144 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:59.144 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:59.145 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:59.145 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:59.145 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:59.145 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:59.145 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:59.146 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:59.146 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:59.146 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:59.146 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:59.146 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.147 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:59.147 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:59.147 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:59.147 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:59.147 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:59.148 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:59.148 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:59.148 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:59.148 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:59.149 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:59.149 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:59.149 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:59.149 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.150 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:59.150 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:59.150 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:59.150 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:59.150 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:59.151 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:59.151 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:59.151 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:59.151 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:59.151 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:59.152 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:59.152 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:59.152 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:59.153 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:59.153 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:59.153 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:59.153 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:59.153 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:59.154 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:59.154 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:59.154 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:59.154 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:59.155 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:59.155 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:59.155 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:59.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:59.156 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:59.156 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.156 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:59.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.157 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:59.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.157 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:59.157 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:59.158 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:59.158 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:59.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.158 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:59.159 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:59.159 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.160 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:59.160 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:59.160 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.161 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.161 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.161 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:59.161 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:59.162 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:59.162 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:59.162 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:59.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.163 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:59.163 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:59.163 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:59.163 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:59.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.164 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:59.164 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:59.164 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:59.165 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:59.166 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.166 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:59.167 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:59.168 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:59.168 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:59.168 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:59.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.169 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:59.169 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:59.169 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:59.169 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:59.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.170 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:59.170 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:59.170 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.171 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:59.171 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:59.171 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.171 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.172 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:59.172 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:59.172 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:59.172 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:59.173 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:59.173 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.173 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.173 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:59.174 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:59.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.174 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:59.174 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:59.175 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:59.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.175 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.175 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:59.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.176 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:59.176 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:59.176 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:01:59.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.176 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.177 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:59.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.177 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:59.177 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:59.178 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 10 2023-10-30T12:01:59.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.178 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:59.179 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:59.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 10 librados: watch_flush enter 2023-10-30T12:01:59.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 10 librados: watch_flush exit 2023-10-30T12:01:59.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 20 client.34271.objecter shutdown clearing up homeless session... 2023-10-30T12:01:59.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 10 client.34271.objecter successfully canceled tick 2023-10-30T12:01:59.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e9c0392d0 msgr2=0x7f1e9c03b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e9c0392d0 0x7f1e9c03b790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8121850 tx=0x7f1ea8007a10 comp rx=0 tx=0).stop 2023-10-30T12:01:59.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 10 monclient: shutdown 2023-10-30T12:01:59.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:59.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb813b070 msgr2=0x7f1eb8121df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 --2- 172.21.15.50:0/804703862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb813b070 0x7f1eb8121df0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000be80 tx=0x7f1eb000beb0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ebe85f640 1 -- 172.21.15.50:0/804703862 reap_dead start 2023-10-30T12:01:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ebe85f640 1 -- 172.21.15.50:0/804703862 reap_dead start 2023-10-30T12:01:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 shutdown_connections 2023-10-30T12:01:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 >> 172.21.15.50:0/804703862 conn(0x7f1eb80a6200 msgr2=0x7f1eb80a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 shutdown_connections 2023-10-30T12:01:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 -- 172.21.15.50:0/804703862 wait complete. 2023-10-30T12:01:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.076+0000 7f1ec02c2640 1 librados: shutdown 2023-10-30T12:01:59.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:01:59.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:01:59.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:01:59.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b3041c640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x55c10f3e2150 0x55c10f3e4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.132+0000 7f5b3041c640 10 monclient: get_auth_request con 0x55c10f3e2150 auth_method 0 2023-10-30T12:01:59.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.134+0000 7f5b2d416640 10 client.4654.objecter ms_handle_connect 0x55c10f3e2150 2023-10-30T12:01:59.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:58.134+0000 7f5b3041c640 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x55c10f3e2150 0x55c10f3e4540 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.278+0000 7f5b2d416640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 1 ==== osd_map(18..18 src has 1..18) v4 ==== 8121+0+0 (crc 0 0 0) 0x7f5b20005170 con 0x55c10f3e2150 2023-10-30T12:01:59.312 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.278+0000 7f5b2d416640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_map(18..18 src has 1..18) v4 2023-10-30T12:01:59.312 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.278+0000 7f5b2d416640 3 client.4654.objecter handle_osd_map got epochs [18,18] > 17 2023-10-30T12:01:59.312 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.278+0000 7f5b2d416640 3 client.4654.objecter handle_osd_map decoding incremental epoch 18 2023-10-30T12:01:59.313 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.279+0000 7f5b2d416640 10 client.4654.objecter checking op 2 2023-10-30T12:01:59.313 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.279+0000 7f5b2d416640 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:59.313 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.279+0000 7f5b2d416640 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.314 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.279+0000 7f5b2d416640 20 client.4654.objecter dump_active .. 0 homeless 2023-10-30T12:01:59.314 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.279+0000 7f5b2d416640 20 client.4654.objecter 2 15.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2023-10-30T12:01:59.314 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-10-30T12:01:59.314 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 2 ==== osd_op_reply(2 exist [writefull 0~5] v18'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.315 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(2 exist [writefull 0~5] v18'1 uv1 ondisk = 0) v8 2023-10-30T12:01:59.315 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.315 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 2 ondisk uv 1 in 15.1 attempt 0 2023-10-30T12:01:59.315 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.316 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 2 2023-10-30T12:01:59.316 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 15 client.4654.objecter _finish_op 2 2023-10-30T12:01:59.316 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.316 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 2 2023-10-30T12:01:59.316 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.317 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 10 librados: Objecter returned from writefull r=0 2023-10-30T12:01:59.317 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3e17a0 2023-10-30T12:01:59.318 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.318 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.318 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.318 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.319 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [read 0~256] tid 3 osd.3 2023-10-30T12:01:59.319 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.319 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 3 2023-10-30T12:01:59.319 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 15 client.4654.objecter _send_op 3 to 15.1 on osd.3 2023-10-30T12:01:59.319 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:3 15.1 15:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f3da470 con 0x55c10f3e2150 2023-10-30T12:01:59.320 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.320 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.320 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3da820 2023-10-30T12:01:59.320 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.320 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-10-30T12:01:59.321 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.321 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.321 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 10 client.4654.objecter _op_submit oid noexist '@15' '@15' [read 0~256] tid 4 osd.3 2023-10-30T12:01:59.321 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.321 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 4 2023-10-30T12:01:59.322 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 15 client.4654.objecter _send_op 4 to 15.1 on osd.3 2023-10-30T12:01:59.322 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:4 15.1 15:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f3dacc0 con 0x55c10f3e2150 2023-10-30T12:01:59.322 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.322 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.308+0000 7f5b32851780 5 client.4654.objecter 2 in flight 2023-10-30T12:01:59.322 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2023-10-30T12:01:59.323 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-10-30T12:01:59.323 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.309+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.323 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.309+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:01:59.323 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.309+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.324 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.309+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 3 ondisk uv 1 in 15.1 attempt 0 2023-10-30T12:01:59.324 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.309+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 5 2023-10-30T12:01:59.324 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 3 2023-10-30T12:01:59.324 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 15 client.4654.objecter _finish_op 3 2023-10-30T12:01:59.325 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.325 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 3 2023-10-30T12:01:59.325 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.325 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.325 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:01:59.326 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.326 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 4 ondisk uv 0 in 15.1 attempt 0 2023-10-30T12:01:59.326 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 10 client.4654.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:01:59.326 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.327 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 4 2023-10-30T12:01:59.327 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 15 client.4654.objecter _finish_op 4 2023-10-30T12:01:59.327 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.327 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 4 2023-10-30T12:01:59.328 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.328 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3da820 2023-10-30T12:01:59.328 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.328 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.328 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.329 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.329 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [read 0~256] tid 5 osd.3 2023-10-30T12:01:59.329 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.329 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 5 2023-10-30T12:01:59.330 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 15 client.4654.objecter _send_op 5 to 15.1 on osd.3 2023-10-30T12:01:59.331 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:5 15.1 15:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f3e17a0 con 0x55c10f3e2150 2023-10-30T12:01:59.331 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.331 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.331 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.332 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.332 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-10-30T12:01:59.332 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.332 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.333 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 10 client.4654.objecter _op_submit oid noexist '@15' '@15' [read 0~256] tid 6 osd.3 2023-10-30T12:01:59.333 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.333 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 6 2023-10-30T12:01:59.333 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 15 client.4654.objecter _send_op 6 to 15.1 on osd.3 2023-10-30T12:01:59.333 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:6 15.1 15:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f3246c0 con 0x55c10f3e2150 2023-10-30T12:01:59.334 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.334 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.310+0000 7f5b32851780 5 client.4654.objecter 2 in flight 2023-10-30T12:01:59.334 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.335 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:01:59.335 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.335 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 5 ondisk uv 1 in 15.1 attempt 0 2023-10-30T12:01:59.335 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 5 2023-10-30T12:01:59.335 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (9 ms) 2023-10-30T12:01:59.336 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 5 2023-10-30T12:01:59.336 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-10-30T12:01:59.336 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 15 client.4654.objecter _finish_op 5 2023-10-30T12:01:59.337 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.337 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 5 2023-10-30T12:01:59.337 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.337 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.338 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:01:59.338 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.338 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 6 ondisk uv 0 in 15.1 attempt 0 2023-10-30T12:01:59.338 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 10 client.4654.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:01:59.339 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.339 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 6 2023-10-30T12:01:59.339 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 15 client.4654.objecter _finish_op 6 2023-10-30T12:01:59.339 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.340 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 6 2023-10-30T12:01:59.340 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.317+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.340 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.341 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.341 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.341 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.342 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.342 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [read 0~256] tid 7 osd.3 2023-10-30T12:01:59.342 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.342 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 7 2023-10-30T12:01:59.343 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 15 client.4654.objecter _send_op 7 to 15.1 on osd.3 2023-10-30T12:01:59.343 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:7 15.1 15:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f3da820 con 0x55c10f3e2150 2023-10-30T12:01:59.343 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.343 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.343 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f325110 2023-10-30T12:01:59.344 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.344 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-10-30T12:01:59.344 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.344 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.344 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 10 client.4654.objecter _op_submit oid noexist '@15' '@15' [read 0~256] tid 8 osd.3 2023-10-30T12:01:59.345 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.345 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 8 2023-10-30T12:01:59.345 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 15 client.4654.objecter _send_op 8 to 15.1 on osd.3 2023-10-30T12:01:59.345 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:8 15.1 15:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f3255b0 con 0x55c10f3e2150 2023-10-30T12:01:59.346 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.346 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.318+0000 7f5b32851780 5 client.4654.objecter 2 in flight 2023-10-30T12:01:59.346 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.346 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-10-30T12:01:59.346 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.347 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 7 ondisk uv 1 in 15.1 attempt 0 2023-10-30T12:01:59.347 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 5 2023-10-30T12:01:59.347 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 7 2023-10-30T12:01:59.347 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 15 client.4654.objecter _finish_op 7 2023-10-30T12:01:59.348 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.348 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 7 2023-10-30T12:01:59.348 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.348 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2023-10-30T12:01:59.349 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-10-30T12:01:59.349 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.349 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:01:59.349 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.349 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 8 ondisk uv 0 in 15.1 attempt 0 2023-10-30T12:01:59.350 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 10 client.4654.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:01:59.350 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.350 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 8 2023-10-30T12:01:59.350 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 15 client.4654.objecter _finish_op 8 2023-10-30T12:01:59.350 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.351 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 8 2023-10-30T12:01:59.351 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.352 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:01:59.352 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 librados: queue_aio_write 0x55c10f22d130 completion 0x55c10f339b40 write_seq 1 2023-10-30T12:01:59.352 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.352 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:59.353 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.353 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.353 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.353 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [write 0~5 in=5b] tid 9 osd.3 2023-10-30T12:01:59.353 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.354 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 9 2023-10-30T12:01:59.354 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 15 client.4654.objecter _send_op 9 to 15.1 on osd.3 2023-10-30T12:01:59.354 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:9 15.1 15:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f324aa0 con 0x55c10f3e2150 2023-10-30T12:01:59.354 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.354 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.355 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.320+0000 7f5b32851780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:01:59.355 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 9 ==== osd_op_reply(9 exist [write 0~5] v18'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.355 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(9 exist [write 0~5] v18'2 uv2 ondisk = 0) v8 2023-10-30T12:01:59.355 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.355 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 9 ondisk uv 2 in 15.1 attempt 0 2023-10-30T12:01:59.356 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.356 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 9 2023-10-30T12:01:59.356 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 15 client.4654.objecter _finish_op 9 2023-10-30T12:01:59.356 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.357 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 9 2023-10-30T12:01:59.357 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.357 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.345+0000 7f5b3041c640 20 librados: complete_aio_write 0x55c10f339b40 2023-10-30T12:01:59.357 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (26 ms) 2023-10-30T12:01:59.357 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-10-30T12:01:59.358 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:01:59.358 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 20 librados: queue_aio_write 0x55c10f22d130 completion 0x55c10f324e00 write_seq 2 2023-10-30T12:01:59.358 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.358 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:59.358 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.359 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.359 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.359 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [write 0~5 in=5b] tid 10 osd.3 2023-10-30T12:01:59.359 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.360 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 10 2023-10-30T12:01:59.360 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 15 client.4654.objecter _send_op 10 to 15.1 on osd.3 2023-10-30T12:01:59.360 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.346+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:10 15.1 15:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f324aa0 con 0x55c10f3e2150 2023-10-30T12:01:59.360 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.347+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.361 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.347+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.361 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.347+0000 7f5b32851780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:01:59.361 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 10 ==== osd_op_reply(10 exist [write 0~5] v18'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.362 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(10 exist [write 0~5] v18'3 uv3 ondisk = 0) v8 2023-10-30T12:01:59.362 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.362 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 10 ondisk uv 3 in 15.1 attempt 0 2023-10-30T12:01:59.362 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.363 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 10 2023-10-30T12:01:59.363 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 15 client.4654.objecter _finish_op 10 2023-10-30T12:01:59.363 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.363 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 10 2023-10-30T12:01:59.364 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.364 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b3041c640 20 librados: complete_aio_write 0x55c10f324e00 2023-10-30T12:01:59.364 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3 ms) 2023-10-30T12:01:59.364 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-10-30T12:01:59.364 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-10-30T12:01:59.365 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 20 librados: queue_aio_write 0x55c10f22d130 completion 0x55c10f324c80 write_seq 3 2023-10-30T12:01:59.367 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.368 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:59.368 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.368 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.368 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.369 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [write 0~5 in=5b] tid 11 osd.3 2023-10-30T12:01:59.369 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.369 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 11 2023-10-30T12:01:59.370 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.349+0000 7f5b32851780 15 client.4654.objecter _send_op 11 to 15.1 on osd.3 2023-10-30T12:01:59.370 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.350+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:11 15.1 15:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f328730 con 0x55c10f3e2150 2023-10-30T12:01:59.370 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.350+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.370 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.350+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.371 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.350+0000 7f5b32851780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-10-30T12:01:59.371 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (7 ms) 2023-10-30T12:01:59.371 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-10-30T12:01:59.372 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.355+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 11 ==== osd_op_reply(11 exist [write 0~5] v18'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.372 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.355+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(11 exist [write 0~5] v18'4 uv4 ondisk = 0) v8 2023-10-30T12:01:59.372 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.372 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 11 ondisk uv 4 in 15.1 attempt 0 2023-10-30T12:01:59.373 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.373 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 11 2023-10-30T12:01:59.373 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 15 client.4654.objecter _finish_op 11 2023-10-30T12:01:59.373 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.373 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 11 2023-10-30T12:01:59.374 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.374 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b3041c640 20 librados: complete_aio_write 0x55c10f324c80 2023-10-30T12:01:59.374 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.374 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.375 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.375 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.375 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.375 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [read 0~0] tid 12 osd.3 2023-10-30T12:01:59.376 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.376 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 12 2023-10-30T12:01:59.376 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 15 client.4654.objecter _send_op 12 to 15.1 on osd.3 2023-10-30T12:01:59.376 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:12 15.1 15:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f3280c0 con 0x55c10f3e2150 2023-10-30T12:01:59.376 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.377 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.377 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f328530 2023-10-30T12:01:59.377 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.377 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-10-30T12:01:59.377 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.378 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.378 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 10 client.4654.objecter _op_submit oid noexist '@15' '@15' [read 0~0] tid 13 osd.3 2023-10-30T12:01:59.378 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.378 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 13 2023-10-30T12:01:59.379 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 15 client.4654.objecter _send_op 13 to 15.1 on osd.3 2023-10-30T12:01:59.379 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:13 15.1 15:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f325990 con 0x55c10f3e2150 2023-10-30T12:01:59.379 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.379 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.356+0000 7f5b32851780 5 client.4654.objecter 2 in flight 2023-10-30T12:01:59.379 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.380 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:01:59.380 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.380 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 12 ondisk uv 4 in 15.1 attempt 0 2023-10-30T12:01:59.380 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 5 2023-10-30T12:01:59.380 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 12 2023-10-30T12:01:59.381 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 15 client.4654.objecter _finish_op 12 2023-10-30T12:01:59.381 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.381 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 12 2023-10-30T12:01:59.382 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.382 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5b20002ad0 con 0x55c10f3e2150 2023-10-30T12:01:59.382 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:01:59.382 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.382 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 13 ondisk uv 0 in 15.1 attempt 0 2023-10-30T12:01:59.383 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 10 client.4654.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:01:59.383 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.383 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 13 2023-10-30T12:01:59.383 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 15 client.4654.objecter _finish_op 13 2023-10-30T12:01:59.384 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.384 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 13 2023-10-30T12:01:59.384 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.357+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.384 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (2 ms) 2023-10-30T12:01:59.385 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-10-30T12:01:59.385 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.358+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.385 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.385 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.385 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.385 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.386 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [read 0~0] tid 14 osd.3 2023-10-30T12:01:59.386 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.386 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 14 2023-10-30T12:01:59.386 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 15 client.4654.objecter _send_op 14 to 15.1 on osd.3 2023-10-30T12:01:59.387 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:14 15.1 15:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f325990 con 0x55c10f3e2150 2023-10-30T12:01:59.387 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.387 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.387 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f328530 2023-10-30T12:01:59.388 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.388 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-10-30T12:01:59.388 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.388 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.388 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 10 client.4654.objecter _op_submit oid noexist '@15' '@15' [read 0~0] tid 15 osd.3 2023-10-30T12:01:59.389 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.389 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 15 2023-10-30T12:01:59.389 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 15 client.4654.objecter _send_op 15 to 15.1 on osd.3 2023-10-30T12:01:59.389 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:15 15.1 15:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f325d40 con 0x55c10f3e2150 2023-10-30T12:01:59.389 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.390 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.359+0000 7f5b32851780 5 client.4654.objecter 2 in flight 2023-10-30T12:01:59.390 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5b20029040 con 0x55c10f3e2150 2023-10-30T12:01:59.390 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:01:59.390 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.391 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 14 ondisk uv 4 in 15.1 attempt 0 2023-10-30T12:01:59.391 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 5 2023-10-30T12:01:59.391 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 14 2023-10-30T12:01:59.391 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 15 client.4654.objecter _finish_op 14 2023-10-30T12:01:59.391 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.391 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 14 2023-10-30T12:01:59.392 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.392 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5b20029040 con 0x55c10f3e2150 2023-10-30T12:01:59.392 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:01:59.392 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.393 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 15 ondisk uv 0 in 15.1 attempt 0 2023-10-30T12:01:59.393 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 10 client.4654.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:01:59.393 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.393 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 15 2023-10-30T12:01:59.393 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 15 client.4654.objecter _finish_op 15 2023-10-30T12:01:59.394 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.394 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 15 2023-10-30T12:01:59.394 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.394 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (2 ms) 2023-10-30T12:01:59.395 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-10-30T12:01:59.395 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.395 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.395 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.395 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.396 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.396 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [read 0~0] tid 16 osd.3 2023-10-30T12:01:59.396 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.396 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 16 2023-10-30T12:01:59.397 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 15 client.4654.objecter _send_op 16 to 15.1 on osd.3 2023-10-30T12:01:59.397 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:16 15.1 15:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f325990 con 0x55c10f3e2150 2023-10-30T12:01:59.397 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.397 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.398 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3263b0 2023-10-30T12:01:59.398 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base noexist @15 precalc_pgid 0 pgid 0.0 is_read 2023-10-30T12:01:59.398 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter _calc_target target noexist @15 -> pgid 15.a7bbaa1 2023-10-30T12:01:59.398 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.a7bbaa1 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.399 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.399 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 10 client.4654.objecter _op_submit oid noexist '@15' '@15' [read 0~0] tid 17 osd.3 2023-10-30T12:01:59.399 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.399 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 17 2023-10-30T12:01:59.400 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 15 client.4654.objecter _send_op 17 to 15.1 on osd.3 2023-10-30T12:01:59.400 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:17 15.1 15:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f328660 con 0x55c10f3e2150 2023-10-30T12:01:59.400 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.400 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.360+0000 7f5b32851780 5 client.4654.objecter 2 in flight 2023-10-30T12:01:59.401 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f5b20029040 con 0x55c10f3e2150 2023-10-30T12:01:59.401 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-10-30T12:01:59.401 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.401 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 16 ondisk uv 4 in 15.1 attempt 0 2023-10-30T12:01:59.401 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 5 2023-10-30T12:01:59.402 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 16 2023-10-30T12:01:59.402 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 15 client.4654.objecter _finish_op 16 2023-10-30T12:01:59.402 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 5 2023-10-30T12:01:59.402 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 16 2023-10-30T12:01:59.402 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.403 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5b20029040 con 0x55c10f3e2150 2023-10-30T12:01:59.403 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-10-30T12:01:59.403 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.403 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 17 ondisk uv 0 in 15.1 attempt 0 2023-10-30T12:01:59.403 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 10 client.4654.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-10-30T12:01:59.404 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.404 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 17 2023-10-30T12:01:59.404 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 15 client.4654.objecter _finish_op 17 2023-10-30T12:01:59.404 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.404 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 17 2023-10-30T12:01:59.404 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.405 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2023-10-30T12:01:59.405 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-10-30T12:01:59.405 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 20 librados: queue_aio_write 0x55c10f22d130 completion 0x55c10f328530 write_seq 4 2023-10-30T12:01:59.405 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.405 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:59.406 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.406 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.406 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.406 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 18 osd.3 2023-10-30T12:01:59.406 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.407 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 18 2023-10-30T12:01:59.407 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 15 client.4654.objecter _send_op 18 to 15.1 on osd.3 2023-10-30T12:01:59.407 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:18 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f325990 con 0x55c10f3e2150 2023-10-30T12:01:59.407 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.407 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.361+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.408 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 18 ==== osd_op_reply(18 exist [writefull 0~5] v18'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5b20029040 con 0x55c10f3e2150 2023-10-30T12:01:59.408 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(18 exist [writefull 0~5] v18'5 uv5 ondisk = 0) v8 2023-10-30T12:01:59.408 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.408 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 18 ondisk uv 5 in 15.1 attempt 0 2023-10-30T12:01:59.408 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.409 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 18 2023-10-30T12:01:59.409 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 15 client.4654.objecter _finish_op 18 2023-10-30T12:01:59.409 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.409 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 18 2023-10-30T12:01:59.409 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.409 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b3041c640 20 librados: complete_aio_write 0x55c10f328530 2023-10-30T12:01:59.410 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2023-10-30T12:01:59.410 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-10-30T12:01:59.410 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 20 librados: queue_aio_write 0x55c10f22d130 completion 0x55c10f328660 write_seq 5 2023-10-30T12:01:59.410 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.410 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:59.411 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.411 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.413 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.413 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 19 osd.3 2023-10-30T12:01:59.413 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.413 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 19 2023-10-30T12:01:59.414 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 15 client.4654.objecter _send_op 19 to 15.1 on osd.3 2023-10-30T12:01:59.414 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:19 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f328c80 con 0x55c10f3e2150 2023-10-30T12:01:59.414 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.414 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.364+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.414 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 19 ==== osd_op_reply(19 exist [writefull 0~5] v18'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5b20029040 con 0x55c10f3e2150 2023-10-30T12:01:59.415 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(19 exist [writefull 0~5] v18'6 uv6 ondisk = 0) v8 2023-10-30T12:01:59.415 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.415 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 19 ondisk uv 6 in 15.1 attempt 0 2023-10-30T12:01:59.415 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.415 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 19 2023-10-30T12:01:59.416 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 15 client.4654.objecter _finish_op 19 2023-10-30T12:01:59.416 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.416 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 19 2023-10-30T12:01:59.416 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.417 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.366+0000 7f5b3041c640 20 librados: complete_aio_write 0x55c10f328660 2023-10-30T12:01:59.417 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (4 ms) 2023-10-30T12:01:59.417 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-10-30T12:01:59.417 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 20 librados: queue_aio_write 0x55c10f22d130 completion 0x55c10f329060 write_seq 6 2023-10-30T12:01:59.417 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 10 client.4654.objecter _op_submit op 0x55c10f3db070 2023-10-30T12:01:59.417 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 20 client.4654.objecter _calc_target epoch 18 base exist @15 precalc_pgid 0 pgid 0.0 is_write 2023-10-30T12:01:59.418 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 20 client.4654.objecter _calc_target target exist @15 -> pgid 15.d3a59201 2023-10-30T12:01:59.418 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 10 client.4654.objecter _calc_target raw pgid 15.d3a59201 -> actual 15.1 acting [3,2] primary 3 2023-10-30T12:01:59.418 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 20 client.4654.objecter _get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.419 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 10 client.4654.objecter _op_submit oid exist '@15' '@15' [writefull 0~5 in=5b] tid 20 osd.3 2023-10-30T12:01:59.419 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 20 client.4654.objecter get_session s=0x55c10f1cedb0 osd=3 3 2023-10-30T12:01:59.419 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 15 client.4654.objecter _session_op_assign 3 20 2023-10-30T12:01:59.419 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 15 client.4654.objecter _send_op 20 to 15.1 on osd.3 2023-10-30T12:01:59.419 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- osd_op(unknown.0.0:20 15.1 15:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x55c10f328c80 con 0x55c10f3e2150 2023-10-30T12:01:59.419 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.420 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.367+0000 7f5b32851780 5 client.4654.objecter 1 in flight 2023-10-30T12:01:59.420 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 <== osd.3 v2:172.21.15.118:6808/1009584072 20 ==== osd_op_reply(20 exist [writefull 0~5] v18'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f5b20029040 con 0x55c10f3e2150 2023-10-30T12:01:59.420 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 10 client.4654.objecter ms_dispatch 0x55c10f205d50 osd_op_reply(20 exist [writefull 0~5] v18'7 uv7 ondisk = 0) v8 2023-10-30T12:01:59.420 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 10 client.4654.objecter in handle_osd_op_reply 2023-10-30T12:01:59.421 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 7 client.4654.objecter handle_osd_op_reply 20 ondisk uv 7 in 15.1 attempt 0 2023-10-30T12:01:59.421 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 10 client.4654.objecter op 0 rval 0 len 0 2023-10-30T12:01:59.421 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 15 client.4654.objecter handle_osd_op_reply completed tid 20 2023-10-30T12:01:59.421 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 15 client.4654.objecter _finish_op 20 2023-10-30T12:01:59.422 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 4 2023-10-30T12:01:59.422 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 15 client.4654.objecter _session_op_remove 3 20 2023-10-30T12:01:59.422 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2023-10-30T12:01:59.422 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 5 client.4654.objecter 0 in flight 2023-10-30T12:01:59.422 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 20 librados: complete_aio_write 0x55c10f329060 2023-10-30T12:01:59.423 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 10 librados: watch_flush enter 2023-10-30T12:01:59.423 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 10 librados: watch_flush exit 2023-10-30T12:01:59.423 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 10 client.4654.objecter close_session for osd.3 2023-10-30T12:01:59.423 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x55c10f3e2150 msgr2=0x55c10f3e4540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.423 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x55c10f3e2150 0x55c10f3e4540 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.424 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 20 client.4654.objecter put_session s=0x55c10f1cedb0 osd=3 1 2023-10-30T12:01:59.424 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 20 client.4654.objecter shutdown clearing up homeless session... 2023-10-30T12:01:59.424 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 10 client.4654.objecter successfully canceled tick 2023-10-30T12:01:59.424 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5b1c037490 msgr2=0x7f5b1c039950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.425 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5b1c037490 0x7f5b1c039950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55c10f271b80 tx=0x7f5b2400a040 comp rx=0 tx=0).stop 2023-10-30T12:01:59.425 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 10 monclient: shutdown 2023-10-30T12:01:59.425 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:59.425 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c10f3cf4f0 msgr2=0x55c10f330510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.426 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 --2- 172.21.15.118:0/1944996220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c10f3cf4f0 0x55c10f330510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5b280499d0 tx=0x7f5b28003330 comp rx=0 tx=0).stop 2023-10-30T12:01:59.426 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b3041c640 1 -- 172.21.15.118:0/1944996220 reap_dead start 2023-10-30T12:01:59.426 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 shutdown_connections 2023-10-30T12:01:59.426 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.370+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 >> 172.21.15.118:0/1944996220 conn(0x55c10f2aaf30 msgr2=0x55c10f27e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:59.427 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.371+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 shutdown_connections 2023-10-30T12:01:59.427 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.371+0000 7f5b32851780 1 -- 172.21.15.118:0/1944996220 wait complete. 2023-10-30T12:01:59.427 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.371+0000 7f5b32851780 1 librados: shutdown 2023-10-30T12:01:59.427 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [----------] 12 tests from AsioRados (62 ms total) 2023-10-30T12:01:59.428 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: 2023-10-30T12:01:59.428 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [----------] Global test environment tear-down 2023-10-30T12:01:59.428 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2222 ms total) 2023-10-30T12:01:59.428 INFO:tasks.workunit.client.0.smithi118.stdout: api_asio: [ PASSED ] 12 tests. 2023-10-30T12:01:59.431 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-10-30T12:01:59.431 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-10-30T12:01:59.432 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-10-30T12:01:59.432 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-10-30T12:01:59.432 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-10-30T12:01:59.432 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T12:01:59.433 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:01:59.433 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-10-30T12:01:59.433 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: seed 87578 2023-10-30T12:01:59.433 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-10-30T12:01:59.433 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2023-10-30T12:01:59.434 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-10-30T12:01:59.434 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/misc_cxx.cc:45: Failure 2023-10-30T12:01:59.434 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Expected equality of these values: 2023-10-30T12:01:59.434 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 10 2023-10-30T12:01:59.435 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: ioctx.write(string(maxlen/2, 'a').c_str(), bl, bl.length(), 0) 2023-10-30T12:01:59.435 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Which is: 0 2023-10-30T12:01:59.435 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.LongNamePP (140 ms) 2023-10-30T12:01:59.435 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-10-30T12:01:59.435 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (61 ms) 2023-10-30T12:01:59.435 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-10-30T12:01:59.436 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (11 ms) 2023-10-30T12:01:59.436 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-10-30T12:01:59.436 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/misc_cxx.cc:138: Failure 2023-10-30T12:01:59.436 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Expected equality of these values: 2023-10-30T12:01:59.436 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 10 2023-10-30T12:01:59.437 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: ioctx.setxattr("bigattrobj", string(maxlen/2, 'a').c_str(), bl) 2023-10-30T12:01:59.437 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Which is: 0 2023-10-30T12:01:59.437 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.LongAttrNamePP (2 ms) 2023-10-30T12:01:59.437 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-10-30T12:01:59.438 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2023-10-30T12:01:59.438 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-10-30T12:01:59.438 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2023-10-30T12:01:59.438 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-10-30T12:01:59.438 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2023-10-30T12:01:59.439 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-10-30T12:01:59.439 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2023-10-30T12:01:59.439 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-10-30T12:01:59.439 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (13 ms) 2023-10-30T12:01:59.439 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-10-30T12:01:59.439 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/misc_cxx.cc:292: Failure 2023-10-30T12:01:59.440 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Expected equality of these values: 2023-10-30T12:01:59.440 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 10 2023-10-30T12:01:59.440 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: ioctx.aio_operate("foo", my_completion, &o) 2023-10-30T12:01:59.440 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Which is: 0 2023-10-30T12:01:59.440 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.AioOperatePP (0 ms) 2023-10-30T12:01:59.440 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-10-30T12:01:59.441 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/misc_cxx.cc:313: Failure 2023-10-30T12:01:59.441 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Expected equality of these values: 2023-10-30T12:01:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.359+0000 7fb76bfff640 10 monclient: get_monmap_and_config 2023-10-30T12:01:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.359+0000 7fb76bfff640 10 monclient: build_initial_monmap 2023-10-30T12:01:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.359+0000 7fb76bfff640 10 monclient: monmap: 2023-10-30T12:01:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:01:59.360391+0000 2023-10-30T12:01:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:01:59.360391+0000 2023-10-30T12:01:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:01:59.451 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:01:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:01:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:01:59.452 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:01:59.452 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 1 Processor -- start 2023-10-30T12:01:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 1 -- start start 2023-10-30T12:01:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient: init 2023-10-30T12:01:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:59.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-10-30T12:01:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb764027900 addr v1:172.21.15.118:6789/0 2023-10-30T12:01:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640c1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fb764027ce0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7640280c0 addr v1:172.21.15.50:6789/0 2023-10-30T12:01:59.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7640bcbb0 con 0x7fb7640280c0 2023-10-30T12:01:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb764023ca0 con 0x7fb764027900 2023-10-30T12:01:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640245d0 con 0x7fb764027ce0 2023-10-30T12:01:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76bfff640 10 monclient(hunting): authenticate will time out at 964.543823s 2023-10-30T12:01:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76b7fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fb7640280c0 legacy=0x7fb7640c3570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:01:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.360+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640c1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76affd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb764027900 0x7fb7640bc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:42420/0 (socket says 172.21.15.50:42420) 2023-10-30T12:01:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640c1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57926/0 (socket says 172.21.15.50:57926) 2023-10-30T12:01:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76affd640 1 -- 172.21.15.50:0/201810014 learned_addr learned my addr 172.21.15.50:0/201810014 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:01:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): get_auth_request con 0x7fb764027ce0 auth_method 0 2023-10-30T12:01:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_done global_id 34286 payload 274 2023-10-30T12:01:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 1 -- 172.21.15.50:0/201810014 >> v1:172.21.15.118:6789/0 conn(0x7fb764027900 legacy=0x7fb7640bc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 1 -- 172.21.15.50:0/201810014 >> v1:172.21.15.50:6789/0 conn(0x7fb7640280c0 legacy=0x7fb7640c3570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:01:59.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:01:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.361+0000 7fb76a7fc640 1 monclient: found mon.noname-b 2023-10-30T12:01:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.362+0000 7fb76a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.362+0000 7fb76a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.362+0000 7fb76a7fc640 1 -- 172.21.15.50:0/201810014 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640248b0 con 0x7fb764027ce0 2023-10-30T12:01:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb76a7fc640 1 --2- 172.21.15.50:0/201810014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640c1020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb760009750 tx=0x7fb76002b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2938ce03fbd846bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 1 -- 172.21.15.50:0/201810014 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760030710 con 0x7fb764027ce0 2023-10-30T12:01:59.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:01:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 10 monclient: dump: 2023-10-30T12:01:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.462 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.463 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.464 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 10 monclient: _finish_auth 0 2023-10-30T12:01:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:01:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 1 -- 172.21.15.50:0/201810014 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76002e050 con 0x7fb764027ce0 2023-10-30T12:01:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 1 -- 172.21.15.50:0/201810014 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760030710 con 0x7fb764027ce0 2023-10-30T12:01:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.363+0000 7fb769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb769ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb769ffb640 10 monclient: dump: 2023-10-30T12:01:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.468 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb76bfff640 5 monclient: authenticate success, global_id 34286 2023-10-30T12:01:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb76bfff640 10 monclient: get_monmap_and_config success 2023-10-30T12:01:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb76bfff640 4 set_mon_vals no callback set 2023-10-30T12:01:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb76bfff640 10 monclient: shutdown 2023-10-30T12:01:59.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb76bfff640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb76bfff640 1 -- 172.21.15.50:0/201810014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 msgr2=0x7fb7640c1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.364+0000 7fb76bfff640 1 --2- 172.21.15.50:0/201810014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640c1020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb760009750 tx=0x7fb76002b540 comp rx=0 tx=0).stop 2023-10-30T12:01:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.365+0000 7fb76bfff640 1 -- 172.21.15.50:0/201810014 shutdown_connections 2023-10-30T12:01:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.365+0000 7fb76bfff640 1 --2- 172.21.15.50:0/201810014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640c1020 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.365+0000 7fb76bfff640 1 -- 172.21.15.50:0/201810014 >> 172.21.15.50:0/201810014 conn(0x7fb764006670 msgr2=0x7fb764006a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:59.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.365+0000 7fb76bfff640 1 -- 172.21.15.50:0/201810014 shutdown_connections 2023-10-30T12:01:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.365+0000 7fb76bfff640 1 -- 172.21.15.50:0/201810014 wait complete. 2023-10-30T12:01:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.366+0000 7fb76bfff640 10 monclient: build_initial_monmap 2023-10-30T12:01:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.366+0000 7fb76bfff640 10 monclient: monmap: 2023-10-30T12:01:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:01:59.470 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:01:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:01:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:01:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:01:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.471 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:01:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:01:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:01:59.472 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.366+0000 7fb76bfff640 1 librados: starting msgr at 2023-10-30T12:01:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.366+0000 7fb76bfff640 1 librados: starting objecter 2023-10-30T12:01:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.366+0000 7fb76bfff640 1 Processor -- start 2023-10-30T12:01:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 -- start start 2023-10-30T12:01:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 librados: setting wanted keys 2023-10-30T12:01:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 librados: calling monclient init 2023-10-30T12:01:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient: init 2023-10-30T12:01:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient: _reopen_session rank -1 2023-10-30T12:01:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:01:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb764027900 0x7fb7640b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76affd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb764027900 msgr2=0x7fb7640b8a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:01:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb764027900 0x7fb7640b8a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:01:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fb764027900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:01:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fb764027ce0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:01:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7640280c0 0x7fb7640be3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fb7640280c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:01:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640c4520 con 0x7fb764027900 2023-10-30T12:01:59.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640c2160 con 0x7fb7640280c0 2023-10-30T12:01:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb764023970 con 0x7fb764027ce0 2023-10-30T12:01:59.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): _renew_subs 2023-10-30T12:01:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76bfff640 10 monclient(hunting): authenticate will time out at 964.550819s 2023-10-30T12:01:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640b8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57940/0 (socket says 172.21.15.50:57940) 2023-10-30T12:01:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76a7fc640 1 -- 172.21.15.50:0/2350637631 learned_addr learned my addr 172.21.15.50:0/2350637631 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:01:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76a7fc640 10 monclient(hunting): get_auth_request con 0x7fb764027ce0 auth_method 0 2023-10-30T12:01:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.367+0000 7fb76a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7640280c0 0x7fb7640be3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 10 monclient(hunting): get_auth_request con 0x7fb7640280c0 auth_method 0 2023-10-30T12:01:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 10 monclient(hunting): handle_auth_done global_id 34289 payload 824 2023-10-30T12:01:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7640280c0 msgr2=0x7fb7640be3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7640280c0 0x7fb7640be3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb764027900 msgr2=0x7fb7640b8a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:01:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb764027900 0x7fb7640b8a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:01:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 1 monclient: found mon.noname-c 2023-10-30T12:01:59.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:01:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 1 -- 172.21.15.50:0/2350637631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb760009400 con 0x7fb764027ce0 2023-10-30T12:01:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76b7fe640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7640280c0 0x7fb7640be3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:01:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb76a7fc640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640b8f70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb76002bb50 tx=0x7fb7600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb753fff640 10 client.?.objecter ms_handle_connect 0x7fb764027ce0 2023-10-30T12:01:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.368+0000 7fb753fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:01:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760003f50 con 0x7fb764027ce0 2023-10-30T12:01:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:01:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: dump: 2023-10-30T12:01:59.484 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.485 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.486 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.486 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: _finish_auth 0 2023-10-30T12:01:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: _check_auth_tickets 2023-10-30T12:01:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:01:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb760030510 con 0x7fb764027ce0 2023-10-30T12:01:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:01:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7600308a0 con 0x7fb764027ce0 2023-10-30T12:01:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:01:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb753fff640 10 monclient: dump: 2023-10-30T12:01:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.489 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.490 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.490 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.490 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.491 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.491 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.491 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 5 monclient: authenticate success, global_id 34289 2023-10-30T12:01:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 10 monclient: _renew_subs 2023-10-30T12:01:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7640c1b10 con 0x7fb764027ce0 2023-10-30T12:01:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 10 client.34289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:01:59.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 10 monclient: _renew_subs 2023-10-30T12:01:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:01:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7640c1800 con 0x7fb764027ce0 2023-10-30T12:01:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb76bfff640 1 librados: init done 2023-10-30T12:01:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.369+0000 7fb7697fa640 4 set_mon_vals no callback set 2023-10-30T12:01:59.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.370+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb760030a00 con 0x7fb764027ce0 2023-10-30T12:01:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.370+0000 7fb76bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.370+0000 7fb76bfff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c002840 0x7fb74c004ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.370+0000 7fb76bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.370+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c007c70 con 0x7fb74c002840 2023-10-30T12:01:59.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb753fff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb7580392a0 0x7fb75803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c002840 0x7fb74c004ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 13926+0+0 (secure 0 0 0) 0x7fb76006a350 con 0x7fb764027ce0 2023-10-30T12:01:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb753fff640 10 client.34289.objecter ms_dispatch 0x7fb76c000b90 osd_map(18..18 src has 1..18) v4 2023-10-30T12:01:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb753fff640 3 client.34289.objecter handle_osd_map got epochs [18,18] > 0 2023-10-30T12:01:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb753fff640 3 client.34289.objecter handle_osd_map decoding full epoch 18 2023-10-30T12:01:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 10 monclient: get_auth_request con 0x7fb74c002840 auth_method 0 2023-10-30T12:01:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb753fff640 20 client.34289.objecter dump_active .. 0 homeless 2023-10-30T12:01:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.371+0000 7fb76b7fe640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb7580392a0 0x7fb75803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb76b7fe640 10 monclient: get_auth_request con 0x7fb7580392a0 auth_method 0 2023-10-30T12:01:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb76affd640 10 monclient(hunting): handle_auth_done global_id 4747 payload 824 2023-10-30T12:01:59.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb76affd640 1 -- 172.21.15.50:0/2350637631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb74c007e30 con 0x7fb74c002840 2023-10-30T12:01:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb76affd640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c002840 0x7fb74c004ce0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb75c004810 tx=0x7fb75c00cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb753fff640 10 client.34289.objecter ms_handle_connect 0x7fb74c002840 2023-10-30T12:01:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb753fff640 10 client.34289.objecter resend_mon_ops 2023-10-30T12:01:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c004aa0 con 0x7fb74c002840 2023-10-30T12:01:59.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb753fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:59.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb753fff640 10 monclient: dump: 2023-10-30T12:01:59.499 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.499 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.499 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.499 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.500 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.500 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.500 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.500 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.500 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.500 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb753fff640 10 client.34289.objecter ms_handle_connect 0x7fb7580392a0 2023-10-30T12:01:59.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.372+0000 7fb76b7fe640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb7580392a0 0x7fb75803b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb76c069af0 tx=0x7fb76c06c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.373+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb75c004c00 con 0x7fb74c002840 2023-10-30T12:01:59.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.373+0000 7fb753fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:01:59.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.373+0000 7fb753fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:01:59.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.373+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c002840 msgr2=0x7fb74c004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.373+0000 7fb753fff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c002840 0x7fb74c004ce0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb75c004810 tx=0x7fb75c00cb10 comp rx=0 tx=0).stop 2023-10-30T12:01:59.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.373+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c002840 msgr2=0x7fb74c004ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:59.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.373+0000 7fb753fff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c002840 0x7fb74c004ce0 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb75c004810 tx=0x7fb75c00cb10 comp rx=0 tx=0).stop 2023-10-30T12:01:59.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.383+0000 7fb76bfff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:01:59.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.383+0000 7fb76bfff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c006cb0 0x7fb74c007120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:01:59.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.383+0000 7fb76bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:01:59.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.383+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c007660 con 0x7fb74c006cb0 2023-10-30T12:01:59.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c006cb0 0x7fb74c007120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:01:59.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient: get_auth_request con 0x7fb74c006cb0 auth_method 0 2023-10-30T12:01:59.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:01:59.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:01:59.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:01:59.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:01:59.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:01:59.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:01:59.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 10 monclient(hunting): handle_auth_done global_id 4753 payload 824 2023-10-30T12:01:59.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.384+0000 7fb76affd640 1 -- 172.21.15.50:0/2350637631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb74c007820 con 0x7fb74c006cb0 2023-10-30T12:01:59.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb76affd640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c006cb0 0x7fb74c007120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb75c007fd0 tx=0x7fb75c001300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:01:59.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 10 client.34289.objecter ms_handle_connect 0x7fb74c006cb0 2023-10-30T12:01:59.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 10 client.34289.objecter resend_mon_ops 2023-10-30T12:01:59.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00aa90 con 0x7fb74c006cb0 2023-10-30T12:01:59.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:01:59.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:01:59.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 10 monclient: dump: 2023-10-30T12:01:59.507 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:01:59.507 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:01:59.507 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.507 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:01:59.507 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:01:59.508 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:01:59.508 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:01:59.508 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:01:59.508 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:01:59.508 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:01:59.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4793 (secure 0 0 0) 0x7fb75c00abf0 con 0x7fb74c006cb0 2023-10-30T12:01:59.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:01:59.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:01:59.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c006cb0 msgr2=0x7fb74c007120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c006cb0 0x7fb74c007120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb75c007fd0 tx=0x7fb75c001300 comp rx=0 tx=0).stop 2023-10-30T12:01:59.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c006cb0 msgr2=0x7fb74c007120 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:01:59.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.385+0000 7fb753fff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb74c006cb0 0x7fb74c007120 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb75c007fd0 tx=0x7fb75c001300 comp rx=0 tx=0).stop 2023-10-30T12:01:59.510 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:01:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:01:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 12, 2023-10-30T12:01:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:01:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:01:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:01:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:01:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:01:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:01:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:01:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:01:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:01:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:01:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:01:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:01:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:01:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:01:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:01:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:01:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:01:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:01:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:01:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:01:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:01:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:01:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:01:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:01:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:01:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:01:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:01:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:01:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:01:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:01:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:01:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:01:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:01:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:01:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:01:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:01:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:01:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:01:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:01:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:01:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:01:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:01:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:01:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:01:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:01:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.532 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:59.532 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:01:59.532 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:01:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:01:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:01:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:01:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:01:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:01:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:01:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:01:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:01:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:01:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:01:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:01:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:01:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:01:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:01:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:01:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:01:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:01:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:01:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:01:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:01:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 28 2023-10-30T12:01:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:01:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:01:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:01:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:01:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:01:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:01:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:01:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:01:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:01:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:01:59.546 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:01:59.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 10 librados: watch_flush enter 2023-10-30T12:01:59.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 10 librados: watch_flush exit 2023-10-30T12:01:59.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 20 client.34289.objecter shutdown clearing up homeless session... 2023-10-30T12:01:59.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 10 client.34289.objecter successfully canceled tick 2023-10-30T12:01:59.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb7580392a0 msgr2=0x7fb75803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb7580392a0 0x7fb75803b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb76c069af0 tx=0x7fb76c06c040 comp rx=0 tx=0).stop 2023-10-30T12:01:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 10 monclient: shutdown 2023-10-30T12:01:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:01:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 msgr2=0x7fb7640b8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:01:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 --2- 172.21.15.50:0/2350637631 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb764027ce0 0x7fb7640b8f70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb76002bb50 tx=0x7fb7600046c0 comp rx=0 tx=0).stop 2023-10-30T12:01:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76b7fe640 1 -- 172.21.15.50:0/2350637631 reap_dead start 2023-10-30T12:01:59.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76b7fe640 1 -- 172.21.15.50:0/2350637631 reap_dead start 2023-10-30T12:01:59.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 shutdown_connections 2023-10-30T12:01:59.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 >> 172.21.15.50:0/2350637631 conn(0x7fb764006670 msgr2=0x7fb764027140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:01:59.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 shutdown_connections 2023-10-30T12:01:59.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 -- 172.21.15.50:0/2350637631 wait complete. 2023-10-30T12:01:59.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:01:59.386+0000 7fb76bfff640 1 librados: shutdown 2023-10-30T12:01:59.552 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:01:59.552 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:01:59.669 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-10-30T12:01:59.670 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] Global test environment set-up. 2023-10-30T12:01:59.670 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] 7 tests from LibRadosList 2023-10-30T12:01:59.670 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-10-30T12:01:59.670 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosList.ListObjects (228 ms) 2023-10-30T12:01:59.670 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-10-30T12:01:59.671 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (35 ms) 2023-10-30T12:01:59.671 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-10-30T12:01:59.671 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T12:01:59.671 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo1 2023-10-30T12:01:59.671 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo2 2023-10-30T12:01:59.671 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo3 2023-10-30T12:01:59.672 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T12:01:59.672 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo4 2023-10-30T12:01:59.672 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo5 2023-10-30T12:01:59.672 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo1 2023-10-30T12:01:59.672 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset foo6,foo7 2023-10-30T12:01:59.673 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo7 2023-10-30T12:01:59.673 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo6 2023-10-30T12:01:59.673 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T12:01:59.673 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns1:foo4 2023-10-30T12:01:59.673 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns1:foo5 2023-10-30T12:01:59.674 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns2:foo7 2023-10-30T12:01:59.674 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns2:foo6 2023-10-30T12:01:59.674 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns1:foo1 2023-10-30T12:01:59.674 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: :foo1 2023-10-30T12:01:59.674 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: :foo2 2023-10-30T12:01:59.675 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: :foo3 2023-10-30T12:01:59.675 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (84 ms) 2023-10-30T12:01:59.675 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-10-30T12:01:59.675 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 1 0 2023-10-30T12:01:59.675 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 10 0 2023-10-30T12:01:59.675 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 13 0 2023-10-30T12:01:59.676 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 7 0 2023-10-30T12:01:59.676 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 14 0 2023-10-30T12:01:59.676 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 0 0 2023-10-30T12:01:59.676 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 15 0 2023-10-30T12:01:59.676 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 11 0 2023-10-30T12:01:59.676 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 5 0 2023-10-30T12:01:59.677 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 8 0 2023-10-30T12:01:59.677 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 6 0 2023-10-30T12:01:59.677 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 3 0 2023-10-30T12:01:59.677 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 4 0 2023-10-30T12:01:59.677 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 12 0 2023-10-30T12:01:59.677 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 9 0 2023-10-30T12:01:59.678 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 2 0 2023-10-30T12:01:59.678 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T12:01:59.678 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (71 ms) 2023-10-30T12:01:59.678 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-10-30T12:01:59.678 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: x cursor=MIN 2023-10-30T12:01:59.679 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=1 cursor=24:02547ec2:::1:head 2023-10-30T12:01:59.679 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=10 cursor=24:52ea6a34:::10:head 2023-10-30T12:01:59.679 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=13 cursor=24:566253c9:::13:head 2023-10-30T12:01:59.679 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=7 cursor=24:5c6b0b28:::7:head 2023-10-30T12:01:59.679 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=14 cursor=24:62a1935d:::14:head 2023-10-30T12:01:59.679 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=0 cursor=24:6cac518f:::0:head 2023-10-30T12:01:59.680 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=15 cursor=24:863748b0:::15:head 2023-10-30T12:01:59.680 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=11 cursor=24:89d3ae78:::11:head 2023-10-30T12:01:59.680 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=5 cursor=24:b29083e3:::5:head 2023-10-30T12:01:59.680 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=8 cursor=24:bd63b0f1:::8:head 2023-10-30T12:01:59.681 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=6 cursor=24:c4fdafeb:::6:head 2023-10-30T12:01:59.681 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=3 cursor=24:cfc208b3:::3:head 2023-10-30T12:01:59.681 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=4 cursor=24:d83876eb:::4:head 2023-10-30T12:01:59.681 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=12 cursor=24:de5d7c5f:::12:head 2023-10-30T12:01:59.681 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=9 cursor=24:e960b815:::9:head 2023-10-30T12:01:59.681 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > oid=2 cursor=24:f905c69b:::2:head 2023-10-30T12:01:59.682 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: FIRST> seek to MIN oid=1 2023-10-30T12:01:59.682 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=1 cursor=24:02547ec2:::1:head 2023-10-30T12:01:59.682 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:02547ec2:::1:head 2023-10-30T12:01:59.682 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:02547ec2:::1:head -> 1 2023-10-30T12:01:59.682 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=10 cursor=24:52ea6a34:::10:head 2023-10-30T12:01:59.683 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:52ea6a34:::10:head 2023-10-30T12:01:59.683 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:52ea6a34:::10:head -> 10 2023-10-30T12:01:59.683 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=13 cursor=24:566253c9:::13:head 2023-10-30T12:01:59.683 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:566253c9:::13:head 2023-10-30T12:01:59.683 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:566253c9:::13:head -> 13 2023-10-30T12:01:59.683 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=7 cursor=24:5c6b0b28:::7:head 2023-10-30T12:01:59.684 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:5c6b0b28:::7:head 2023-10-30T12:01:59.684 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:5c6b0b28:::7:head -> 7 2023-10-30T12:01:59.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=14 cursor=24:62a1935d:::14:head 2023-10-30T12:01:59.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:62a1935d:::14:head 2023-10-30T12:01:59.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:62a1935d:::14:head -> 14 2023-10-30T12:01:59.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=0 cursor=24:6cac518f:::0:head 2023-10-30T12:01:59.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:6cac518f:::0:head 2023-10-30T12:01:59.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:6cac518f:::0:head -> 0 2023-10-30T12:01:59.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=15 cursor=24:863748b0:::15:head 2023-10-30T12:01:59.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:863748b0:::15:head 2023-10-30T12:01:59.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:863748b0:::15:head -> 15 2023-10-30T12:01:59.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=11 cursor=24:89d3ae78:::11:head 2023-10-30T12:01:59.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:89d3ae78:::11:head 2023-10-30T12:01:59.738 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:89d3ae78:::11:head -> 11 2023-10-30T12:01:59.738 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=5 cursor=24:b29083e3:::5:head 2023-10-30T12:01:59.738 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:b29083e3:::5:head 2023-10-30T12:01:59.738 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:b29083e3:::5:head -> 5 2023-10-30T12:01:59.738 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=8 cursor=24:bd63b0f1:::8:head 2023-10-30T12:01:59.739 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:bd63b0f1:::8:head 2023-10-30T12:01:59.739 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:bd63b0f1:::8:head -> 8 2023-10-30T12:01:59.739 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=6 cursor=24:c4fdafeb:::6:head 2023-10-30T12:01:59.739 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:c4fdafeb:::6:head 2023-10-30T12:01:59.740 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:c4fdafeb:::6:head -> 6 2023-10-30T12:01:59.740 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=3 cursor=24:cfc208b3:::3:head 2023-10-30T12:01:59.740 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:cfc208b3:::3:head 2023-10-30T12:01:59.740 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:cfc208b3:::3:head -> 3 2023-10-30T12:01:59.740 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=4 cursor=24:d83876eb:::4:head 2023-10-30T12:01:59.740 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:d83876eb:::4:head 2023-10-30T12:01:59.741 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:d83876eb:::4:head -> 4 2023-10-30T12:01:59.741 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=12 cursor=24:de5d7c5f:::12:head 2023-10-30T12:01:59.741 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:de5d7c5f:::12:head 2023-10-30T12:01:59.741 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:de5d7c5f:::12:head -> 12 2023-10-30T12:01:59.741 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=9 cursor=24:e960b815:::9:head 2023-10-30T12:01:59.741 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:e960b815:::9:head 2023-10-30T12:01:59.742 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:e960b815:::9:head -> 9 2023-10-30T12:01:59.742 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : oid=2 cursor=24:f905c69b:::2:head 2023-10-30T12:01:59.742 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:f905c69b:::2:head 2023-10-30T12:01:59.742 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:f905c69b:::2:head -> 2 2023-10-30T12:01:59.743 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:52ea6a34:::10:head 2023-10-30T12:01:59.743 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:52ea6a34:::10:head expected=24:52ea6a34:::10:head 2023-10-30T12:01:59.743 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:52ea6a34:::10:head -> 10 2023-10-30T12:01:59.743 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=10 expected=10 2023-10-30T12:01:59.743 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:e960b815:::9:head 2023-10-30T12:01:59.744 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:e960b815:::9:head expected=24:e960b815:::9:head 2023-10-30T12:01:59.744 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:e960b815:::9:head -> 9 2023-10-30T12:01:59.744 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=9 expected=9 2023-10-30T12:01:59.745 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:c4fdafeb:::6:head 2023-10-30T12:01:59.745 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:c4fdafeb:::6:head expected=24:c4fdafeb:::6:head 2023-10-30T12:01:59.745 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:c4fdafeb:::6:head -> 6 2023-10-30T12:01:59.745 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=6 expected=6 2023-10-30T12:01:59.745 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:f905c69b:::2:head 2023-10-30T12:01:59.746 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:f905c69b:::2:head expected=24:f905c69b:::2:head 2023-10-30T12:01:59.746 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:f905c69b:::2:head -> 2 2023-10-30T12:01:59.746 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=2 expected=2 2023-10-30T12:01:59.746 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:bd63b0f1:::8:head 2023-10-30T12:01:59.746 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:bd63b0f1:::8:head expected=24:bd63b0f1:::8:head 2023-10-30T12:01:59.746 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:bd63b0f1:::8:head -> 8 2023-10-30T12:01:59.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=8 expected=8 2023-10-30T12:01:59.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:b29083e3:::5:head 2023-10-30T12:01:59.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:b29083e3:::5:head expected=24:b29083e3:::5:head 2023-10-30T12:01:59.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:b29083e3:::5:head -> 5 2023-10-30T12:01:59.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=5 expected=5 2023-10-30T12:01:59.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:89d3ae78:::11:head 2023-10-30T12:01:59.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:89d3ae78:::11:head expected=24:89d3ae78:::11:head 2023-10-30T12:01:59.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:89d3ae78:::11:head -> 11 2023-10-30T12:01:59.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=11 expected=11 2023-10-30T12:01:59.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:863748b0:::15:head 2023-10-30T12:01:59.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:863748b0:::15:head expected=24:863748b0:::15:head 2023-10-30T12:01:59.749 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:863748b0:::15:head -> 15 2023-10-30T12:02:00.049 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : ent list_parallel: process_1_[87929]: starting. 2023-10-30T12:02:00.050 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_1_[87929]: creating pool ceph_test_rados_list_parallel.smithi118-87902 2023-10-30T12:02:00.050 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_1_[87929]: created object 0... 2023-10-30T12:02:00.050 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_1_[87929]: created object 25... 2023-10-30T12:02:00.051 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_1_[87929]: created object 49... 2023-10-30T12:02:00.051 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_1_[87929]: finishing. 2023-10-30T12:02:00.051 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_1_[87929]: shutting down. 2023-10-30T12:02:00.052 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_2_[87930]: starting. 2023-10-30T12:02:00.052 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_2_[87930]: listing objects. 2023-10-30T12:02:00.052 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_2_[87930]: listed object 0... 2023-10-30T12:02:00.052 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_2_[87930]: listed object 25... 2023-10-30T12:02:00.053 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_2_[87930]: saw 50 objects 2023-10-30T12:02:00.053 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_2_[87930]: shutting down. 2023-10-30T12:02:00.053 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.053 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_3_[88471]: starting. 2023-10-30T12:02:00.053 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_3_[88471]: creating pool ceph_test_rados_list_parallel.smithi118-87902 2023-10-30T12:02:00.053 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_3_[88471]: created object 0... 2023-10-30T12:02:00.054 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_3_[88471]: created object 25... 2023-10-30T12:02:00.054 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_3_[88471]: created object 49... 2023-10-30T12:02:00.054 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_3_[88471]: finishing. 2023-10-30T12:02:00.054 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_3_[88471]: shutting down. 2023-10-30T12:02:00.055 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.055 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_4_[88472]: starting. 2023-10-30T12:02:00.055 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_4_[88472]: listing objects. 2023-10-30T12:02:00.055 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_4_[88472]: listed object 0... 2023-10-30T12:02:00.055 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_4_[88472]: listed object 25... 2023-10-30T12:02:00.056 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_4_[88472]: saw 44 objects 2023-10-30T12:02:00.056 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_4_[88472]: shutting down. 2023-10-30T12:02:00.056 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.057 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_5_[88473]: starting. 2023-10-30T12:02:00.057 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_5_[88473]: removed 25 objects... 2023-10-30T12:02:00.057 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_5_[88473]: removed half of the objects 2023-10-30T12:02:00.058 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_5_[88473]: removed 50 objects... 2023-10-30T12:02:00.058 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_5_[88473]: removed 50 objects 2023-10-30T12:02:00.058 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_5_[88473]: shutting down. 2023-10-30T12:02:00.058 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.059 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.059 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_6_[88522]: starting. 2023-10-30T12:02:00.059 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_6_[88522]: creating pool ceph_test_rados_list_parallel.smithi118-87902 2023-10-30T12:02:00.060 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_6_[88522]: created object 0... 2023-10-30T12:02:00.060 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_6_[88522]: created object 25... 2023-10-30T12:02:00.060 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_6_[88522]: created object 49... 2023-10-30T12:02:00.060 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_6_[88522]: finishing. 2023-10-30T12:02:00.061 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_6_[88522]: shutting down. 2023-10-30T12:02:00.061 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.061 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.061 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_7_[88523]: starting. 2023-10-30T12:02:00.061 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_7_[88523]: listing objects. 2023-10-30T12:02:00.061 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_7_[88523]: listed object 0... 2023-10-30T12:02:00.062 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_7_[88523]: listed object 25... 2023-10-30T12:02:00.062 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_7_[88523]: listed object 50... 2023-10-30T12:02:00.062 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_7_[88523]: saw 57 objects 2023-10-30T12:02:00.062 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_7_[88523]: shutting down. 2023-10-30T12:02:00.063 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.063 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.063 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_8_[88524]: starting. 2023-10-30T12:02:00.063 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_8_[88524]: added 25 objects... 2023-10-30T12:02:00.063 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_8_[88524]: added half of the objects 2023-10-30T12:02:00.063 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_8_[88524]: added 50 objects... 2023-10-30T12:02:00.064 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_8_[88524]: added 50 objects 2023-10-30T12:02:00.064 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_8_[88524]: shutting down. 2023-10-30T12:02:00.064 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.064 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.176 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ********************** api_c_read_operations: Running main() from gmock_main.cc 2023-10-30T12:02:00.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-10-30T12:02:00.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-10-30T12:02:00.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-10-30T12:02:00.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-10-30T12:02:00.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-10-30T12:02:00.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-10-30T12:02:00.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (173 ms) 2023-10-30T12:02:00.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-10-30T12:02:00.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (19 ms) 2023-10-30T12:02:00.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-10-30T12:02:00.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (15 ms) 2023-10-30T12:02:00.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-10-30T12:02:00.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (26 ms) 2023-10-30T12:02:00.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-10-30T12:02:00.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2023-10-30T12:02:00.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-10-30T12:02:00.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (7 ms) 2023-10-30T12:02:00.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-10-30T12:02:00.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4 ms) 2023-10-30T12:02:00.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-10-30T12:02:00.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2023-10-30T12:02:00.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-10-30T12:02:00.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3 ms) 2023-10-30T12:02:00.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-10-30T12:02:00.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2023-10-30T12:02:00.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-10-30T12:02:00.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (8 ms) 2023-10-30T12:02:00.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-10-30T12:02:00.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2023-10-30T12:02:00.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-10-30T12:02:00.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (18 ms) 2023-10-30T12:02:00.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-10-30T12:02:00.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (10 ms) 2023-10-30T12:02:00.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-10-30T12:02:00.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (7 ms) 2023-10-30T12:02:00.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-10-30T12:02:00.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2023-10-30T12:02:00.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (321 ms total) 2023-10-30T12:02:00.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: 2023-10-30T12:02:00.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-10-30T12:02:00.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (5789 ms total) 2023-10-30T12:02:00.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-10-30T12:02:00.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-10-30T12:02:00.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-10-30T12:02:00.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-10-30T12:02:00.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-10-30T12:02:00.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (5745 ms) 2023-10-30T12:02:00.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (5745 ms total) 2023-10-30T12:02:00.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: 2023-10-30T12:02:00.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-10-30T12:02:00.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (5745 ms total) 2023-10-30T12:02:00.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-10-30T12:02:00.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-10-30T12:02:00.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-10-30T12:02:00.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-10-30T12:02:00.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-10-30T12:02:00.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-10-30T12:02:00.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (2820 ms) 2023-10-30T12:02:00.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-10-30T12:02:00.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (21 ms) 2023-10-30T12:02:00.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-10-30T12:02:00.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2990 ms) 2023-10-30T12:02:00.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (5831 ms total) 2023-10-30T12:02:00.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: 2023-10-30T12:02:00.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-10-30T12:02:00.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (5831 ms total) 2023-10-30T12:02:00.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-10-30T12:02:00.460 INFO:tasks.workunit.client.0.smithi118.stdout:********* 2023-10-30T12:02:00.460 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_9_[88573]: starting. 2023-10-30T12:02:00.461 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_9_[88573]: creating pool ceph_test_rados_list_parallel.smithi118-87902 2023-10-30T12:02:00.461 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_9_[88573]: created object 0... 2023-10-30T12:02:00.461 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_9_[88573]: created object 25... 2023-10-30T12:02:00.462 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_9_[88573]: created object 49... 2023-10-30T12:02:00.462 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_9_[88573]: finishing. 2023-10-30T12:02:00.462 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_9_[88573]: shutting down. 2023-10-30T12:02:00.463 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.463 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.464 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.464 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: starting. 2023-10-30T12:02:00.464 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: listing objects. 2023-10-30T12:02:00.465 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: listed object 0... 2023-10-30T12:02:00.465 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: listed object 25... 2023-10-30T12:02:00.465 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: listed object 50... 2023-10-30T12:02:00.466 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: listed object 75... 2023-10-30T12:02:00.466 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: listed object 100... 2023-10-30T12:02:00.466 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: saw 102 objects 2023-10-30T12:02:00.466 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_10_[88574]: shutting down. 2023-10-30T12:02:00.467 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.467 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.467 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.467 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_13_[88577]: starting. 2023-10-30T12:02:00.467 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_13_[88577]: removed 25 objects... 2023-10-30T12:02:00.467 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_13_[88577]: removed half of the objects 2023-10-30T12:02:00.468 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_13_[88577]: removed 50 objects... 2023-10-30T12:02:00.468 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_13_[88577]: removed 50 objects 2023-10-30T12:02:00.468 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_13_[88577]: shutting down. 2023-10-30T12:02:00.468 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.469 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.469 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.469 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_11_[88575]: starting. 2023-10-30T12:02:00.470 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_11_[88575]: added 25 objects... 2023-10-30T12:02:00.470 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_11_[88575]: added half of the objects 2023-10-30T12:02:00.470 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_11_[88575]: added 50 objects... 2023-10-30T12:02:00.471 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_11_[88575]: added 50 objects 2023-10-30T12:02:00.471 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_11_[88575]: shutting down. 2023-10-30T12:02:00.471 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.472 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.472 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.472 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_12_[88576]: starting. 2023-10-30T12:02:00.473 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_12_[88576]: added 25 objects... 2023-10-30T12:02:00.473 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_12_[88576]: added half of the objects 2023-10-30T12:02:00.473 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_12_[88576]: added 50 objects... 2023-10-30T12:02:00.474 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_12_[88576]: added 50 objects 2023-10-30T12:02:00.474 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_12_[88576]: shutting down. 2023-10-30T12:02:00.474 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.475 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.475 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.475 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.476 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_14_[88658]: starting. 2023-10-30T12:02:00.476 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_14_[88658]: creating pool ceph_test_rados_list_parallel.smithi118-87902 2023-10-30T12:02:00.476 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_14_[88658]: created object 0... 2023-10-30T12:02:00.477 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_14_[88658]: created object 25... 2023-10-30T12:02:00.477 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_14_[88658]: created object 49... 2023-10-30T12:02:00.477 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_14_[88658]: finishing. 2023-10-30T12:02:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_14_[88658]: shutting down. 2023-10-30T12:02:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.478 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:00.479 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: starting. 2023-10-30T12:02:00.480 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: listing objects. 2023-10-30T12:02:01.151 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: listed object 0... 2023-10-30T12:02:01.152 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: listed object 25... 2023-10-30T12:02:01.152 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: listed object 50... 2023-10-30T12:02:01.152 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: listed object 75... 2023-10-30T12:02:01.152 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: listed object 100... 2023-10-30T12:02:01.152 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: listed object 125... 2023-10-30T12:02:01.152 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: saw 150 objects 2023-10-30T12:02:01.153 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_15_[88659]: shutting down. 2023-10-30T12:02:01.153 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.153 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.153 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.154 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.154 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_16_[88660]: starting. 2023-10-30T12:02:01.154 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_16_[88660]: added 25 objects... 2023-10-30T12:02:01.154 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_16_[88660]: added half of the objects 2023-10-30T12:02:01.154 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_16_[88660]: added 50 objects... 2023-10-30T12:02:01.154 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_16_[88660]: added 50 objects 2023-10-30T12:02:01.155 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: process_16_[88660]: shutting down. 2023-10-30T12:02:01.155 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.155 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.155 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.155 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.155 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******************************* 2023-10-30T12:02:01.156 INFO:tasks.workunit.client.0.smithi118.stdout: list_parallel: ******* SUCCESS ********** 2023-10-30T12:02:01.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: Running main() from gmock_main.cc 2023-10-30T12:02:01.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:02:01.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [----------] Global test environment set-up. 2023-10-30T12:02:01.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-10-30T12:02:01.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-10-30T12:02:01.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (2816 ms) 2023-10-30T12:02:01.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-10-30T12:02:01.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (85 ms) 2023-10-30T12:02:01.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-10-30T12:02:01.214 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2905 ms) 2023-10-30T12:02:01.214 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-10-30T12:02:01.214 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:01:59.193207+0000 mon.b [WRN] Health check failed: 24 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-30T12:02:01.214 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.136497+0000 mon.b [INF] from='client.? 172.21.15.118:0/1439910775' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi118-87673-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi118-87673-1"}]': finished 2023-10-30T12:02:01.214 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.176163+0000 mon.b [INF] from='client.? 172.21.15.118:0/1226115549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87473-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:01.215 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.247216+0000 mon.c [INF] from='client.? 172.21.15.118:0/2852298563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi118-87542-7"}]: dispatch 2023-10-30T12:02:01.215 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.248172+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi118-87542-7"}]: dispatch 2023-10-30T12:02:01.215 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.250344+0000 mon.c [INF] from='client.? 172.21.15.118:0/2852298563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi118-87542-7"}]: dispatch 2023-10-30T12:02:01.215 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.250904+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi118-87542-7"}]: dispatch 2023-10-30T12:02:01.215 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.262323+0000 mon.c [INF] from='client.? 172.21.15.118:0/2852298563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi118-87542-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:01.215 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.262881+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi118-87542-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:01.216 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.332677+0000 mon.c [INF] from='client.? 172.21.15.118:0/1450205678' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T12:02:01.216 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.332884+0000 client.admin [INF] onexx 2023-10-30T12:02:01.216 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.348210+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-10-30T12:02:01.216 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.356575+0000 mon.c [INF] from='client.? 172.21.15.118:0/1120130767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi118-87326-22"}]: dispatch 2023-10-30T12:02:01.216 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.372901+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi118-87326-22"}]: dispatch 2023-10-30T12:02:01.217 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.373170+0000 mon.b [INF] from='client.? 172.21.15.118:0/179047066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi118-87307-16"}]: dispatch 2023-10-30T12:02:01.217 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.374648+0000 mon.c [INF] from='client.? 172.21.15.118:0/1120130767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi118-87326-22"}]: dispatch 2023-10-30T12:02:01.217 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.394735+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi118-87326-22"}]: dispatch 2023-10-30T12:02:01.217 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.395002+0000 mon.b [INF] from='client.? 172.21.15.118:0/179047066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi118-87307-16"}]: dispatch 2023-10-30T12:02:01.217 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.395516+0000 mon.c [INF] from='client.? 172.21.15.118:0/1120130767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi118-87326-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:01.217 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.415923+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi118-87326-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:01.218 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.417258+0000 mon.b [INF] from='client.? 172.21.15.118:0/179047066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi118-87307-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:01.218 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.417510+0000 mon.b [INF] from='client.? 172.21.15.118:0/1133054911' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi118-87577-7"}]: dispatch 2023-10-30T12:02:01.218 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.417949+0000 mon.b [INF] from='client.? 172.21.15.118:0/1133054911' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi118-87577-7"}]: dispatch 2023-10-30T12:02:01.218 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:00.418244+0000 mon.b [INF] from='client.? 172.21.15.118:0/1133054911' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi118-87577-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:01.218 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.140386+0000 mon.b [INF] from='client.? 172.21.15.118:0/1226115549' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87473-4","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:02:01.219 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.140452+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi118-87542-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:02:01.219 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.140500+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi118-87326-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:02:01.219 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.140556+0000 mon.b [INF] from='client.? 172.21.15.118:0/179047066' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi118-87307-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:02:01.219 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.140610+0000 mon.b [INF] from='client.? 172.21.15.118:0/1133054911' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi118-87577-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:02:03.175 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.1440 delete_pools_parallel: process_1_[87914]: starting. 2023-10-30T12:02:03.175 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_1_[87914]: creating pool ceph_test_rados_delete_pools_parallel.smithi118-87899 2023-10-30T12:02:03.176 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_1_[87914]: created object 0... 2023-10-30T12:02:03.176 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_1_[87914]: created object 25... 2023-10-30T12:02:03.176 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_1_[87914]: created object 49... 2023-10-30T12:02:03.176 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_1_[87914]: finishing. 2023-10-30T12:02:03.176 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_1_[87914]: shutting down. 2023-10-30T12:02:03.177 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_2_[87917]: starting. 2023-10-30T12:02:03.177 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_2_[87917]: deleting pool ceph_test_rados_delete_pools_parallel.smithi118-87899 2023-10-30T12:02:03.177 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_2_[87917]: shutting down. 2023-10-30T12:02:03.177 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:02:03.178 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_3_[88683]: starting. 2023-10-30T12:02:03.178 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_3_[88683]: creating pool ceph_test_rados_delete_pools_parallel.smithi118-87899 2023-10-30T12:02:03.178 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_3_[88683]: created object 0... 2023-10-30T12:02:03.178 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_3_[88683]: created object 25... 2023-10-30T12:02:03.178 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_3_[88683]: created object 49... 2023-10-30T12:02:03.179 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_3_[88683]: finishing. 2023-10-30T12:02:03.179 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_3_[88683]: shutting down. 2023-10-30T12:02:03.179 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:02:03.179 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_5_[88685]: starting. 2023-10-30T12:02:03.179 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_5_[88685]: listing objects. 2023-10-30T12:02:03.179 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_5_[88685]: listed object 0... 2023-10-30T12:02:03.180 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_5_[88685]: listed object 25... 2023-10-30T12:02:03.180 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_5_[88685]: saw 50 objects 2023-10-30T12:02:03.180 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_5_[88685]: shutting down. 2023-10-30T12:02:03.180 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:02:03.180 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_4_[88684]: starting. 2023-10-30T12:02:03.180 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_4_[88684]: deleting pool ceph_test_rados_delete_pools_parallel.smithi118-87899 2023-10-30T12:02:03.181 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: process_4_[88684]: shutting down. 2023-10-30T12:02:03.181 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:02:03.181 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: ******************************* 2023-10-30T12:02:03.181 INFO:tasks.workunit.client.0.smithi118.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-10-30T12:02:03.182 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_1_[87908]: starting. 2023-10-30T12:02:03.182 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_1_[87908]: creating pool ceph_test_rados_open_pools_parallel.smithi118-87888 2023-10-30T12:02:03.182 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_1_[87908]: created object 0... 2023-10-30T12:02:03.182 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_1_[87908]: created object 25... 2023-10-30T12:02:03.182 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_1_[87908]: created object 49... 2023-10-30T12:02:03.183 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_1_[87908]: finishing. 2023-10-30T12:02:03.183 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_1_[87908]: shutting down. 2023-10-30T12:02:03.183 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_2_[87909]: starting. 2023-10-30T12:02:03.183 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_2_[87909]: rados_pool_create. 2023-10-30T12:02:03.183 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_2_[87909]: rados_ioctx_create. 2023-10-30T12:02:03.184 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_2_[87909]: shutting down. 2023-10-30T12:02:03.184 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: ******************************* 2023-10-30T12:02:03.184 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_3_[88680]: starting. 2023-10-30T12:02:03.185 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_3_[88680]: creating pool ceph_test_rados_open_pools_parallel.smithi118-87888 2023-10-30T12:02:03.185 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_3_[88680]: created object 0... 2023-10-30T12:02:03.185 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_3_[88680]: created object 25... 2023-10-30T12:02:03.185 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_3_[88680]: created object 49... 2023-10-30T12:02:03.185 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_3_[88680]: finishing. 2023-10-30T12:02:03.186 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_3_[88680]: shutting down. 2023-10-30T12:02:03.186 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: ******************************* 2023-10-30T12:02:03.186 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_4_[88682]: starting. 2023-10-30T12:02:03.186 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_4_[88682]: rados_pool_create. 2023-10-30T12:02:03.186 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_4_[88682]: rados_ioctx_create. 2023-10-30T12:02:03.187 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: process_4_[88682]: shutting down. 2023-10-30T12:02:03.187 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: ******************************* 2023-10-30T12:02:03.187 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: ******************************* 2023-10-30T12:02:03.187 INFO:tasks.workunit.client.0.smithi118.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-10-30T12:02:04.244 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: Running main() from gmock_main.cc 2023-10-30T12:02:04.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-10-30T12:02:04.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [----------] Global test environment set-up. 2023-10-30T12:02:04.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-10-30T12:02:04.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: seed 87326 2023-10-30T12:02:04.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-10-30T12:02:04.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-10-30T12:02:04.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-10-30T12:02:04.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (159 ms) 2023-10-30T12:02:04.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-10-30T12:02:04.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (36 ms) 2023-10-30T12:02:04.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-10-30T12:02:04.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (3 ms) 2023-10-30T12:02:04.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-10-30T12:02:04.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2023-10-30T12:02:04.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-10-30T12:02:04.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (5 ms) 2023-10-30T12:02:04.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-10-30T12:02:04.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (4 ms) 2023-10-30T12:02:04.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-10-30T12:02:04.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2023-10-30T12:02:04.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-10-30T12:02:04.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2023-10-30T12:02:04.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-10-30T12:02:04.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2023-10-30T12:02:04.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-10-30T12:02:04.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2023-10-30T12:02:04.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-10-30T12:02:04.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4 ms) 2023-10-30T12:02:04.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-10-30T12:02:04.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (6 ms) 2023-10-30T12:02:04.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-10-30T12:02:04.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2023-10-30T12:02:04.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-10-30T12:02:04.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2023-10-30T12:02:04.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-10-30T12:02:04.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (14 ms) 2023-10-30T12:02:04.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T12:02:04.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2023-10-30T12:02:04.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-10-30T12:02:04.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2023-10-30T12:02:04.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-10-30T12:02:04.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2023-10-30T12:02:04.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-10-30T12:02:04.253 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2023-10-30T12:02:04.253 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (283 ms total) 2023-10-30T12:02:04.253 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: 2023-10-30T12:02:04.253 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-10-30T12:02:04.253 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-10-30T12:02:04.254 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1045 ms) 2023-10-30T12:02:04.254 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-10-30T12:02:04.254 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (34 ms) 2023-10-30T12:02:04.254 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-10-30T12:02:04.254 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2023-10-30T12:02:04.254 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-10-30T12:02:04.255 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2023-10-30T12:02:04.255 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-10-30T12:02:04.255 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2023-10-30T12:02:04.271 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ]19+0000 mon.b [INF] from='client.? 172.21.15.118:0/179047066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi118-87307-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi118-87307-16"}]: dispatch 2023-10-30T12:02:04.272 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.144397+0000 mon.b [INF] from='client.? 172.21.15.118:0/1133054911' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi118-87577-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi118-87577-7"}]: dispatch 2023-10-30T12:02:04.272 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.145294+0000 mon.c [INF] from='client.? 172.21.15.118:0/1120130767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi118-87326-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi118-87326-22"}]: dispatch 2023-10-30T12:02:04.272 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.145442+0000 mon.b [INF] from='client.? 172.21.15.118:0/1876531945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi118-87261-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.272 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.145591+0000 mon.c [INF] from='client.? 172.21.15.118:0/2852298563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi118-87542-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi118-87542-7"}]: dispatch 2023-10-30T12:02:04.272 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.147200+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi118-87326-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi118-87326-22"}]: dispatch 2023-10-30T12:02:04.273 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.147953+0000 mon.c [INF] from='client.? 172.21.15.118:0/2377691253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.273 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.148567+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi118-87542-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi118-87542-7"}]: dispatch 2023-10-30T12:02:04.273 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.150095+0000 mon.c [INF] from='client.? 172.21.15.118:0/1107984921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi118-87282-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.273 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.150791+0000 mon.c [INF] from='client.? 172.21.15.118:0/1079022999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87828-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.273 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.151178+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.274 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.177170+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi118-87282-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.274 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:01.177622+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87828-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.274 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.143493+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87537-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:02:04.274 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.143547+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi118-87282-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:02:04.274 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.143612+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87828-2","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:02:04.275 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.267720+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-10-30T12:02:04.275 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.271731+0000 mon.b [INF] from='client.? 172.21.15.118:0/1439910775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi118-87673-1"}]: dispatch 2023-10-30T12:02:04.275 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.271976+0000 mon.b [INF] from='client.? 172.21.15.118:0/1226115549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87473-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.275 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.288141+0000 mon.b [INF] from='client.? 172.21.15.118:0/4266279680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi118-87426-10"}]: dispatch 2023-10-30T12:02:04.275 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.289030+0000 mon.b [INF] from='client.? 172.21.15.118:0/1067822418' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi118-87397-10"}]: dispatch 2023-10-30T12:02:04.276 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.290612+0000 mon.b [INF] from='client.? 172.21.15.118:0/1067822418' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi118-87397-10"}]: dispatch 2023-10-30T12:02:04.276 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.294240+0000 mon.b [INF] from='client.? 172.21.15.118:0/1067822418' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi118-87397-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:04.276 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.294874+0000 mon.b [INF] from='client.? 172.21.15.118:0/4266279680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi118-87426-10"}]: dispatch 2023-10-30T12:02:04.276 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:02.296861+0000 mon.b [INF] from='client.? 172.21.15.118:0/4266279680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi118-87426-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-10-30T12:02:04.276 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.145855+0000 mon.b [INF] from='client.? 172.21.15.118:0/179047066' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi118-87307-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi118-87307-16"}]': finished 2023-10-30T12:02:04.277 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.145907+0000 mon.b [INF] from='client.? 172.21.15.118:0/1133054911' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi118-87577-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi118-87577-7"}]': finished 2023-10-30T12:02:04.277 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.145945+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi118-87326-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi118-87326-22"}]': finished 2023-10-30T12:02:04.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.145996+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi118-87542-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi118-87542-7"}]': finished 2023-10-30T12:02:04.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.146066+0000 mon.b [INF] from='client.? 172.21.15.118:0/1439910775' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi118-87673-1"}]': finished 2023-10-30T12:02:04.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.146128+0000 mon.b [INF] from='client.? 172.21.15.118:0/1226115549' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87473-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:02:04.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.146173+0000 mon.b [INF] from='client.? 172.21.15.118:0/1067822418' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi118-87397-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:02:04.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.146238+0000 mon.b [INF] from='client.? 172.21.15.118:0/4266279680' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi118-87426-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-10-30T12:02:04.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.151384+0000 mon.c [INF] from='client.? 172.21.15.118:0/1107984921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi118-87282-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.152578+0000 mon.b [INF] from='client.? 172.21.15.118:0/4266279680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi118-87426-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi118-87426-10"}]: dispatch 2023-10-30T12:02:04.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.152947+0000 mon.b [INF] from='client.? 172.21.15.118:0/1067822418' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi118-87397-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi118-87397-10"}]: dispatch 2023-10-30T12:02:04.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.153327+0000 mon.b [INF] from='client.? 172.21.15.118:0/1439910775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi118-87673-1"}]: dispatch 2023-10-30T12:02:04.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.153846+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi118-87282-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:04.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:03.158030+0000 mon.b [INF] from='client.? 172.21.15.118:0/1226115549' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi118-87473-4", "tierpool": "test-rados-api-smithi118-87473-4-cache"}]: dispatch 2023-10-30T12:02:04.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.148929+0000 mon.b [INF] from='client.? 172.21.15.118:0/1439910775' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi118-87673-1"}]': finished 2023-10-30T12:02:04.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.148981+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi118-87282-3","app": "rados","yes_i_really_mean_it": true}]': finished 2023-10-30T12:02:04.286 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.149076+0000 mon.b [INF] from='client.? 172.21.15.118:0/1226115549' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi118-87473-4", "tierpool": "test-rados-api-smithi118-87473-4-cache"}]': finished 2023-10-30T12:02:06.641 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.157260+0000 mon.c [INF] from='client.? 172.21.15.118:0/1107984921' entity='client.admin' cmd=[{" api_service: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:02:06.641 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [----------] Global test environment set-up. 2023-10-30T12:02:06.641 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [----------] 4 tests from LibRadosService 2023-10-30T12:02:06.642 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-10-30T12:02:06.642 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (7821 ms) 2023-10-30T12:02:06.642 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-10-30T12:02:06.642 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ OK ] LibRadosService.RegisterLate (94 ms) 2023-10-30T12:02:06.642 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-10-30T12:02:06.642 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: cluster: 2023-10-30T12:02:06.643 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: id: 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:06.643 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:02:06.643 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 5 pool(s) do not have an application enabled 2023-10-30T12:02:06.643 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.643 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: services: 2023-10-30T12:02:06.644 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: mon: 3 daemons, quorum b,c (age 5s), out of quorum: a 2023-10-30T12:02:06.644 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: mgr: x(active, since 78s) 2023-10-30T12:02:06.644 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: osd: 6 osds: 6 up (since 74s), 6 in (since 80s) 2023-10-30T12:02:06.644 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-10-30T12:02:06.644 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.645 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: data: 2023-10-30T12:02:06.645 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: pools: 15 pools, 401 pgs 2023-10-30T12:02:06.645 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: objects: 50 objects, 580 KiB 2023-10-30T12:02:06.645 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: usage: 161 MiB used, 540 GiB / 540 GiB avail 2023-10-30T12:02:06.645 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: pgs: 17.955% pgs unknown 2023-10-30T12:02:06.646 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 329 active+clean 2023-10-30T12:02:06.646 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 72 unknown 2023-10-30T12:02:06.646 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.646 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: io: 2023-10-30T12:02:06.646 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: client: 1.2 KiB/s rd, 7.7 KiB/s wr, 2 op/s rd, 10 op/s wr 2023-10-30T12:02:06.647 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.647 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.647 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: cluster: 2023-10-30T12:02:06.647 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: id: 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:06.647 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:02:06.647 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 6 pool(s) do not have an application enabled 2023-10-30T12:02:06.648 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.648 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: services: 2023-10-30T12:02:06.648 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: mon: 3 daemons, quorum b,c (age 7s), out of quorum: a 2023-10-30T12:02:06.648 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: mgr: x(active, since 80s) 2023-10-30T12:02:06.648 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: osd: 6 osds: 6 up (since 76s), 6 in (since 82s) 2023-10-30T12:02:06.649 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: foo: 2 portals active (1 hosts, 2 zones) 2023-10-30T12:02:06.649 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-10-30T12:02:06.649 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.649 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: data: 2023-10-30T12:02:06.649 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: pools: 19 pools, 553 pgs 2023-10-30T12:02:06.650 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: objects: 22 objects, 578 KiB 2023-10-30T12:02:06.650 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: usage: 161 MiB used, 540 GiB / 540 GiB avail 2023-10-30T12:02:06.650 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: pgs: 57.866% pgs unknown 2023-10-30T12:02:06.650 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 320 unknown 2023-10-30T12:02:06.650 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 233 active+clean 2023-10-30T12:02:06.650 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.651 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: io: 2023-10-30T12:02:06.651 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: client: 1.7 KiB/s rd, 6.0 KiB/s wr, 2 op/s rd, 8 op/s wr 2023-10-30T12:02:06.651 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.651 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.651 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: cluster: 2023-10-30T12:02:06.652 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: id: 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:06.652 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: health: HEALTH_WARN 2023-10-30T12:02:06.652 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 1 cache pools are missing hit_sets 2023-10-30T12:02:06.652 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 9 pool(s) do not have an application enabled 2023-10-30T12:02:06.652 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.653 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: services: 2023-10-30T12:02:06.653 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: mon: 3 daemons, quorum b,c (age 9s), out of quorum: a 2023-10-30T12:02:06.653 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: mgr: x(active, since 82s) 2023-10-30T12:02:06.653 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: osd: 6 osds: 6 up (since 78s), 6 in (since 84s) 2023-10-30T12:02:06.653 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-10-30T12:02:06.654 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-10-30T12:02:06.654 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:06.654 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: data: 2023-10-30T12:02:06.654 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: pools: 21 pools, 521 pgs 2023-10-30T12:02:07.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: objects: api_stat_pp: Running main() from gmock_main.cc 2023-10-30T12:02:07.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-10-30T12:02:07.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-10-30T12:02:07.176 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-10-30T12:02:07.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: seed 87577 2023-10-30T12:02:07.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-10-30T12:02:07.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (165 ms) 2023-10-30T12:02:07.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-10-30T12:02:07.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/stat_cxx.cc:49: Failure 2023-10-30T12:02:07.177 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: Expected equality of these values: 2023-10-30T12:02:07.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 10 2023-10-30T12:02:07.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: ioctx.stat2("foo", &size, &ts2) 2023-10-30T12:02:07.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: Which is: 0 2023-10-30T12:02:07.178 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ FAILED ] LibRadosStatPP.Stat2Mtime2PP (12 ms) 2023-10-30T12:02:07.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-10-30T12:02:07.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2023-10-30T12:02:07.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-10-30T12:02:07.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (14 ms) 2023-10-30T12:02:07.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-10-30T12:02:07.179 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/stat_cxx.cc:116: Failure 2023-10-30T12:02:07.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: Expected equality of these values: 2023-10-30T12:02:07.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 10 2023-10-30T12:02:07.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: ioctx.write("foo", bl, sizeof(buf), 0) 2023-10-30T12:02:07.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: Which is: 0 2023-10-30T12:02:07.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ FAILED ] LibRadosStatPP.StatPPNS (4 ms) 2023-10-30T12:02:07.180 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (196 ms total) 2023-10-30T12:02:07.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 2023-10-30T12:02:07.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-10-30T12:02:07.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-10-30T12:02:07.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/stat_cxx.cc:81: Failure 2023-10-30T12:02:07.181 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: Expected equality of these values: 2023-10-30T12:02:07.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 0 2023-10-30T12:02:07.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 1 2023-10-30T12:02:07.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ FAILED ] LibRadosStatECPP.StatPP (0 ms) 2023-10-30T12:02:07.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-10-30T12:02:07.182 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2023-10-30T12:02:07.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-10-30T12:02:07.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1048 ms) 2023-10-30T12:02:07.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-10-30T12:02:07.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (25 ms) 2023-10-30T12:02:07.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1074 ms total) 2023-10-30T12:02:07.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 2023-10-30T12:02:07.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-10-30T12:02:07.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (12879 ms total) 2023-10-30T12:02:07.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ PASSED ] 6 tests. 2023-10-30T12:02:07.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ FAILED ] 3 tests, listed below: 2023-10-30T12:02:07.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ FAILED ] LibRadosStatPP.Stat2Mtime2PP 2023-10-30T12:02:07.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ FAILED ] LibRadosStatPP.StatPPNS 2023-10-30T12:02:07.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: [ FAILED ] LibRadosStatECPP.StatPP 2023-10-30T12:02:07.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 2023-10-30T12:02:07.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat_pp: 3 FAILED TESTS 2023-10-30T12:02:07.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: Running main() from gmock_main.cc 2023-10-30T12:02:07.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-10-30T12:02:07.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [----------] Global test environment set-up. 2023-10-30T12:02:07.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-10-30T12:02:07.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-10-30T12:02:07.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (179 ms) 2023-10-30T12:02:07.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-10-30T12:02:07.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-10-30T12:02:07.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-10-30T12:02:07.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: no timeout :/ 2023-10-30T12:02:07.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: no timeout :/ 2023-10-30T12:02:07.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: no timeout :/ 2023-10-30T12:02:07.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: no timeout :/ 2023-10-30T12:02:07.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: no timeout :/ 2023-10-30T12:02:07.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (83 ms) 2023-10-30T12:02:07.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-10-30T12:02:07.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (6 ms) 2023-10-30T12:02:07.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-10-30T12:02:07.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2023-10-30T12:02:07.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-10-30T12:02:07.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (3 ms) 2023-10-30T12:02:07.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-10-30T12:02:07.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (6 ms) 2023-10-30T12:02:07.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-10-30T12:02:07.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4 ms) 2023-10-30T12:02:07.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-10-30T12:02:07.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2023-10-30T12:02:07.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-10-30T12:02:07.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2023-10-30T12:02:07.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-10-30T12:02:07.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2023-10-30T12:02:07.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-10-30T12:02:07.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2023-10-30T12:02:07.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-10-30T12:02:07.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.RmXattr (10 ms) 2023-10-30T12:02:07.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-10-30T12:02:07.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIo.XattrIter (5 ms) 2023-10-30T12:02:07.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [----------] 14 tests from LibRadosIo (315 ms total) 2023-10-30T12:02:07.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: 2023-10-30T12:02:07.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-10-30T12:02:07.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-10-30T12:02:07.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1045 ms) 2023-10-30T12:02:07.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-10-30T12:02:07.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (19 ms) 2023-10-30T12:02:07.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-10-30T12:02:07.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (6 ms) 2023-10-30T12:02:07.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-10-30T12:02:07.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (8 ms) 2023-10-30T12:02:07.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-10-30T12:02:07.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2023-10-30T12:02:07.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-10-30T12:02:07.195 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (5 ms) 2023-10-30T12:02:07.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-10-30T12:02:07.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2023-10-30T12:02:07.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-10-30T12:02:07.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (5 ms) 2023-10-30T12:02:07.196 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-10-30T12:02:07.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (11 ms) 2023-10-30T12:02:07.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-10-30T12:02:07.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2023-10-30T12:02:07.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1114 ms total) 2023-10-30T12:02:07.197 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: 2023-10-30T12:02:07.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [----- api_stat: Running main() from gmock_main.cc 2023-10-30T12:02:07.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-10-30T12:02:07.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [----------] Global test environment set-up. 2023-10-30T12:02:07.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-10-30T12:02:07.198 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-10-30T12:02:07.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStat.Stat (191 ms) 2023-10-30T12:02:07.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-10-30T12:02:07.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (30 ms) 2023-10-30T12:02:07.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-10-30T12:02:07.199 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStat.StatNS (14 ms) 2023-10-30T12:02:07.200 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-10-30T12:02:07.200 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2023-10-30T12:02:07.200 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-10-30T12:02:07.200 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2023-10-30T12:02:07.200 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [----------] 5 tests from LibRadosStat (240 ms total) 2023-10-30T12:02:07.200 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: 2023-10-30T12:02:07.201 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-10-30T12:02:07.201 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-10-30T12:02:07.201 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1041 ms) 2023-10-30T12:02:07.201 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-10-30T12:02:07.201 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (27 ms) 2023-10-30T12:02:07.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-10-30T12:02:07.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2023-10-30T12:02:07.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-10-30T12:02:07.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2023-10-30T12:02:07.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1070 ms total) 2023-10-30T12:02:07.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: 2023-10-30T12:02:07.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [----------] Global test environment tear-down 2023-10-30T12:02:07.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (12902 ms total) 2023-10-30T12:02:07.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_stat: [ PASSED ] 9 tests. 2023-10-30T12:02:07.203 INFO:tasks.workunit.client.0.smithi118.stdout: LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-10-30T12:02:07.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4 ms) 2023-10-30T12:02:07.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-10-30T12:02:07.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2023-10-30T12:02:07.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-10-30T12:02:07.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2023-10-30T12:02:07.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-10-30T12:02:07.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2023-10-30T12:02:07.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-10-30T12:02:07.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2023-10-30T12:02:07.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-10-30T12:02:07.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (4 ms) 2023-10-30T12:02:07.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-10-30T12:02:07.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2023-10-30T12:02:07.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-10-30T12:02:07.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2023-10-30T12:02:07.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-10-30T12:02:07.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (5 ms) 2023-10-30T12:02:07.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-10-30T12:02:07.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (4 ms) 2023-10-30T12:02:07.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-10-30T12:02:07.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2023-10-30T12:02:07.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-10-30T12:02:07.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2023-10-30T12:02:07.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1142 ms total) 2023-10-30T12:02:07.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: 2023-10-30T12:02:07.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [----------] Global test environment tear-down 2023-10-30T12:02:07.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (12974 ms total) 2023-10-30T12:02:07.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-10-30T12:02:07.209 INFO:tasks.workunit.client.0.smithi118.stdout:-----] Global test environment tear-down 2023-10-30T12:02:07.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [==========] 24 tests from 2 test suites ran. (12979 ms total) 2023-10-30T12:02:07.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_io: [ PASSED ] 24 tests. 2023-10-30T12:02:07.292 INFO:tasks.workunit.client.0.smithi118.stdout:prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi118-87282-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-10-30T12:02:07.292 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.204651+0000 mon.c [INF] from='client.? 172.21.15.118:0/1186838312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:07.292 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.204840+0000 mon.c [INF] from='client.? 172.21.15.118:0/565119959' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi118-87261-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:07.292 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.206295+0000 mon.b [INF] from='client.? 172.21.15.118:0/1226115549' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi118-87473-4", "overlaypool": "test-rados-api-smithi118-87473-4-cache"}]: dispatch 2023-10-30T12:02:07.293 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.207267+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi118-87282-3", "field": "max_bytes", "val": "4096"}]: dispatch 2023-10-30T12:02:07.293 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.207720+0000 mon.b [INF] from='client.? 172.21.15.118:0/1504836257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87828-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:07.293 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.212273+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi118-87537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:07.293 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.212567+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi118-87261-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-10-30T12:02:07.293 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.270899+0000 mon.c [INF] from='client.? 172.21.15.118:0/1450205678' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T12:02:07.294 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.271093+0000 client.admin [INF] threexx 2023-10-30T12:02:07.294 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: got: 2023-10-30T12:02:04.271396+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-10-30T12:02:07.294 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7126 ms) 2023-10-30T12:02:07.294 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (12932 ms total) 2023-10-30T12:02:07.294 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: 2023-10-30T12:02:07.295 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [----------] Global test environment tear-down 2023-10-30T12:02:07.295 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (12933 ms total) 2023-10-30T12:02:07.295 INFO:tasks.workunit.client.0.smithi118.stdout: api_cmd: [ PASSED ] 4 tests. 2023-10-30T12:02:10.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2023-10-30T12:02:10.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:02:10.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-10-30T12:02:10.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-10-30T12:02:10.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-10-30T12:02:10.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: notify 2023-10-30T12:02:10.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1034 ms) 2023-10-30T12:02:10.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-10-30T12:02:10.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2023-10-30T12:02:10.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1039 ms total) 2023-10-30T12:02:10.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: 2023-10-30T12:02:10.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-10-30T12:02:10.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-10-30T12:02:10.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: notify 2023-10-30T12:02:10.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (50 ms) 2023-10-30T12:02:10.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-10-30T12:02:10.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: notify 2023-10-30T12:02:10.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3916 ms) 2023-10-30T12:02:10.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-10-30T12:02:10.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (6 ms) 2023-10-30T12:02:10.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-10-30T12:02:10.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2023-10-30T12:02:10.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-10-30T12:02:10.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976085211648 notify_id 124554051585 notifier_gid 34505 2023-10-30T12:02:10.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (6 ms) 2023-10-30T12:02:10.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-10-30T12:02:10.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976086894608 notify_id 124554051587 notifier_gid 34505 2023-10-30T12:02:10.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2023-10-30T12:02:10.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-10-30T12:02:10.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976086894608 notify_id 124554051587 notifier_gid 34505 2023-10-30T12:02:10.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976086894608 notify_id 124554051588 notifier_gid 34505 2023-10-30T12:02:10.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (6 ms) 2023-10-30T12:02:10.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-10-30T12:02:10.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976087569664 notify_id 124554051588 notifier_gid 34505 2023-10-30T12:02:10.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2023-10-30T12:02:10.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-10-30T12:02:10.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976086108496 notify_id 124554051589 notifier_gid 34505 2023-10-30T12:02:10.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (5 ms) 2023-10-30T12:02:10.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-10-30T12:02:10.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976086108496 notify_id 124554051586 notifier_gid 34505 2023-10-30T12:02:10.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2023-10-30T12:02:10.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-10-30T12:02:10.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: trying... 2023-10-30T12:02:11.108 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976086108496 notify_id 124554051587 noti api_lock: Running main() from gmock_main.cc 2023-10-30T12:02:11.108 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:02:11.109 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [----------] Global test environment set-up. 2023-10-30T12:02:11.109 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-10-30T12:02:11.109 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-10-30T12:02:11.109 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (220 ms) 2023-10-30T12:02:11.109 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-10-30T12:02:11.110 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.LockShared (7 ms) 2023-10-30T12:02:11.110 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-10-30T12:02:11.110 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1017 ms) 2023-10-30T12:02:11.110 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-10-30T12:02:11.111 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1008 ms) 2023-10-30T12:02:11.111 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-10-30T12:02:11.111 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2023-10-30T12:02:11.111 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-10-30T12:02:11.111 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5 ms) 2023-10-30T12:02:11.112 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-10-30T12:02:11.112 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2023-10-30T12:02:11.112 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-10-30T12:02:11.112 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2023-10-30T12:02:11.112 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [----------] 8 tests from LibRadosLock (2268 ms total) 2023-10-30T12:02:11.112 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: 2023-10-30T12:02:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-10-30T12:02:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-10-30T12:02:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1057 ms) 2023-10-30T12:02:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-10-30T12:02:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (5 ms) 2023-10-30T12:02:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-10-30T12:02:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1006 ms) 2023-10-30T12:02:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-10-30T12:02:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1006 ms) 2023-10-30T12:02:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-10-30T12:02:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (5 ms) 2023-10-30T12:02:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-10-30T12:02:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (6 ms) 2023-10-30T12:02:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-10-30T12:02:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (7 ms) 2023-10-30T12:02:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-10-30T12:02:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2023-10-30T12:02:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3096 ms total) 2023-10-30T12:02:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: 2023-10-30T12:02:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [----------] Global test environment tear-down 2023-10-30T12:02:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (16866 ms total) 2023-10-30T12:02:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock: [ PASSED ] 16 tests. 2023-10-30T12:02:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-10-30T12:02:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-10-30T12:02:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-10-30T12:02:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-10-30T12:02:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: seed 87426 2023-10-30T12:02:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-10-30T12:02:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (207 ms) 2023-10-30T12:02:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-10-30T12:02:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (9 ms) 2023-10-30T12:02:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-10-30T12:02:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1014 ms) 2023-10-30T12:02:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-10-30T12:02:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1004 ms) 2023-10-30T12:02:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-10-30T12:02:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4 ms) 2023-10-30T12:02:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-10-30T12:02:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2023-10-30T12:02:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-10-30T12:02:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2023-10-30T12:02:11.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-10-30T12:02:11.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2023-10-30T12:02:11.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2249 ms total) 2023-10-30T12:02:11.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: 2023-10-30T12:02:11.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-10-30T12:02:11.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-10-30T12:02:11.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1058 ms) 2023-10-30T12:02:11.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-10-30T12:02:11.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (6 ms) 2023-10-30T12:02:11.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-10-30T12:02:11.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1006 ms) 2023-10-30T12:02:11.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-10-30T12:02:11.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1006 ms) 2023-10-30T12:02:11.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-10-30T12:02:11.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (5 ms) 2023-10-30T12:02:11.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-10-30T12:02:11.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (6 ms) 2023-10-30T12:02:11.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-10-30T12:02:11.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (7 ms) 2023-10-30T12:02:11.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-10-30T12:02:11.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2023-10-30T12:02:11.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3098 ms total) 2023-10-30T12:02:11.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: 2023-10-30T12:02:11.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-10-30T12:02:11.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (16862 ms total) 2023-10-30T12:02:11.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-10-30T12:02:18.149 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-10-30T12:02:18.149 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-10-30T12:02:18.149 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-10-30T12:02:18.149 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-10-30T12:02:18.150 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-10-30T12:02:18.150 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T12:02:18.150 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (210 ms) 2023-10-30T12:02:18.150 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-10-30T12:02:18.150 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T12:02:18.151 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93985205910016 err -107 2023-10-30T12:02:18.151 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1015 ms) 2023-10-30T12:02:18.151 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-10-30T12:02:18.151 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-10-30T12:02:18.151 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93985205910016 err -107 2023-10-30T12:02:18.152 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1006 ms) 2023-10-30T12:02:18.152 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-10-30T12:02:18.152 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 85899345920 cookie 93985208162224 2023-10-30T12:02:18.152 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (5 ms) 2023-10-30T12:02:18.152 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-10-30T12:02:18.152 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 85899345920 cookie 93985208162224 2023-10-30T12:02:18.153 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (5 ms) 2023-10-30T12:02:18.153 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-10-30T12:02:18.153 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 85899345921 cookie 93985208243920 2023-10-30T12:02:18.153 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (6 ms) 2023-10-30T12:02:18.153 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-10-30T12:02:18.154 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 85899345921 cookie 93985208243920 2023-10-30T12:02:18.154 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 85899345921 cookie 93985208265504 2023-10-30T12:02:18.154 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (7 ms) 2023-10-30T12:02:18.154 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-10-30T12:02:18.154 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 85899345922 cookie 93985208265504 2023-10-30T12:02:18.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 90194313219 cookie 93985208265504 2023-10-30T12:02:18.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2023-10-30T12:02:18.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-10-30T12:02:18.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-10-30T12:02:18.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93985208265504 err -107 2023-10-30T12:02:18.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_cb from 34163 notify_id 120259084290 cookie 93985208265504 2023-10-30T12:02:18.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5009 ms) 2023-10-30T12:02:18.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-10-30T12:02:18.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-10-30T12:02:18.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93985208265504 err -107 2023-10-30T12:02:18.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2023-10-30T12:02:18.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11274 ms total) 2023-10-30T12:02:18.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: 2023-10-30T12:02:18.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-10-30T12:02:18.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-10-30T12:02:18.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: watch_notify_test_cb 2023-10-30T12:02:18.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1071 ms) 2023-10-30T12:02:18.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1071 ms total) 2023-10-30T12:02:18.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: 2023-10-30T12:02:18.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-10-30T12:02:18.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (23839 ms total) 2023-10-30T12:02:18.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-10-30T12:02:19.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-10-30T12:02:19.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-10-30T12:02:19.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-10-30T12:02:19.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-10-30T12:02:19.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-10-30T12:02:19.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-10-30T12:02:19.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-10-30T12:02:19.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: 2023-10-30T12:02:19.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-10-30T12:02:19.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-10-30T12:02:19.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (5811 ms) 2023-10-30T12:02:19.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-10-30T12:02:19.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (2988 ms) 2023-10-30T12:02:19.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-10-30T12:02:19.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2983 ms) 2023-10-30T12:02:19.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-10-30T12:02:19.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3008 ms) 2023-10-30T12:02:19.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-10-30T12:02:19.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2933 ms) 2023-10-30T12:02:19.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-10-30T12:02:19.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3019 ms) 2023-10-30T12:02:19.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-10-30T12:02:19.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (4032 ms) 2023-10-30T12:02:19.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (24774 ms total) 2023-10-30T12:02:19.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: 2023-10-30T12:02:19.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-10-30T12:02:19.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (24774 ms total) 2023-10-30T12:02:19.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-10-30T12:02:19.553 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:02:19.554 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:02:19.554 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:02:19.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:02:19.557 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:02:19.557 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:02:19.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:02:19.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.666+0000 7f60b5d41640 10 monclient: get_monmap_and_config 2023-10-30T12:02:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.666+0000 7f60b5d41640 10 monclient: build_initial_monmap 2023-10-30T12:02:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.666+0000 7f60b5d41640 10 monclient: monmap: 2023-10-30T12:02:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:19.667436+0000 2023-10-30T12:02:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:19.667436+0000 2023-10-30T12:02:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:19.670 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:19.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 1 Processor -- start 2023-10-30T12:02:19.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 1 -- start start 2023-10-30T12:02:19.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient: init 2023-10-30T12:02:19.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:19.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-10-30T12:02:19.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f60b00a5f20 0x7f60b00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:19.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient(hunting): picked mon.noname-c con 0x7f60b00a5f20 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:19.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60b00a7140 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:19.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60b0135d80 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:19.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b00a1600 con 0x7f60b00a7140 2023-10-30T12:02:19.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b0099a40 con 0x7f60b0135d80 2023-10-30T12:02:19.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:19.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b00a76a0 con 0x7f60b00a5f20 2023-10-30T12:02:19.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:19.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60b5d41640 10 monclient(hunting): authenticate will time out at 984.850581s 2023-10-30T12:02:19.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60aeffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f60b00a7140 0x7f60b0135670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47638/0 (socket says 172.21.15.50:47638) 2023-10-30T12:02:19.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60aeffd640 1 -- 172.21.15.50:0/4136606168 learned_addr learned my addr 172.21.15.50:0/4136606168 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:19.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 1 --2- 172.21.15.50:0/4136606168 >> v2:172.21.15.118:3300/0 conn(0x7f60b00a5f20 0x7f60b00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:19.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 10 monclient(hunting): get_auth_request con 0x7f60b00a5f20 auth_method 0 2023-10-30T12:02:19.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:19.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:19.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:19.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:19.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:19.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.667+0000 7f60af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:19.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 10 monclient(hunting): handle_auth_done global_id 13525 payload 274 2023-10-30T12:02:19.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 1 -- 172.21.15.50:0/4136606168 >> v1:172.21.15.118:6789/0 conn(0x7f60b0135d80 legacy=0x7f60b013d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:19.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 1 -- 172.21.15.50:0/4136606168 >> v1:172.21.15.50:6789/0 conn(0x7f60b00a7140 legacy=0x7f60b0135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:19.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:02:19.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 1 monclient: found mon.noname-c 2023-10-30T12:02:19.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:19.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:02:19.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 1 -- 172.21.15.50:0/4136606168 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b009a600 con 0x7f60b00a5f20 2023-10-30T12:02:19.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.668+0000 7f60af7fe640 1 --2- 172.21.15.50:0/4136606168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 0x7f60b00a6300 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4ef3e611a166169 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:19.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 1 -- 172.21.15.50:0/4136606168 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a402e070 con 0x7f60b00a5f20 2023-10-30T12:02:19.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:19.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:19.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: dump: 2023-10-30T12:02:19.684 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:19.684 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:19.684 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.684 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.684 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:19.684 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:19.685 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:19.685 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:19.685 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:19.685 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:19.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: _finish_auth 0 2023-10-30T12:02:19.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:02:19.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:19.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 5 monclient: authenticate success, global_id 13525 2023-10-30T12:02:19.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:19.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 1 -- 172.21.15.50:0/4136606168 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a40315a0 con 0x7f60b00a5f20 2023-10-30T12:02:19.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:19.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 1 -- 172.21.15.50:0/4136606168 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4031930 con 0x7f60b00a5f20 2023-10-30T12:02:19.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:19.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 4 set_mon_vals no callback set 2023-10-30T12:02:19.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:19.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:19.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60ae7fc640 10 monclient: dump: 2023-10-30T12:02:19.687 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:19.688 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:19.688 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.688 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.688 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:19.688 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:19.688 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:19.689 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:19.689 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:19.689 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:19.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 10 monclient: shutdown 2023-10-30T12:02:19.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:19.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 1 -- 172.21.15.50:0/4136606168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 msgr2=0x7f60b00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:19.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 1 --2- 172.21.15.50:0/4136606168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 0x7f60b00a6300 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b840 comp rx=0 tx=0).stop 2023-10-30T12:02:19.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 1 -- 172.21.15.50:0/4136606168 shutdown_connections 2023-10-30T12:02:19.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 1 --2- 172.21.15.50:0/4136606168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 0x7f60b00a6300 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:19.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 1 -- 172.21.15.50:0/4136606168 >> 172.21.15.50:0/4136606168 conn(0x7f60b00a8240 msgr2=0x7f60b00a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:19.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 1 -- 172.21.15.50:0/4136606168 shutdown_connections 2023-10-30T12:02:19.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.669+0000 7f60b5d41640 1 -- 172.21.15.50:0/4136606168 wait complete. 2023-10-30T12:02:19.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 10 monclient: build_initial_monmap 2023-10-30T12:02:19.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 10 monclient: monmap: 2023-10-30T12:02:19.691 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:19.691 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:19.691 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:19.692 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:19.692 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:19.692 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:19.692 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:19.692 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:19.692 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:19.693 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:19.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 1 librados: starting msgr at 2023-10-30T12:02:19.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 1 librados: starting objecter 2023-10-30T12:02:19.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 1 Processor -- start 2023-10-30T12:02:19.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 1 -- start start 2023-10-30T12:02:19.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 1 librados: setting wanted keys 2023-10-30T12:02:19.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 1 librados: calling monclient init 2023-10-30T12:02:19.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.670+0000 7f60b5d41640 10 monclient: init 2023-10-30T12:02:19.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:19.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:02:19.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 0x7f60b01316e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:19.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): picked mon.noname-b con 0x7f60b00a5f20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:19.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 0x7f60b0131c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:19.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): picked mon.noname-a con 0x7f60b00a7140 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:19.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f60b0135d80 0x7f60b009db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:19.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): picked mon.noname-c con 0x7f60b0135d80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:19.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:19.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b011ee30 con 0x7f60b00a7140 2023-10-30T12:02:19.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:19.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b009ff40 con 0x7f60b00a5f20 2023-10-30T12:02:19.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:19.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01337a0 con 0x7f60b0135d80 2023-10-30T12:02:19.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:19.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60b5d41640 10 monclient(hunting): authenticate will time out at 984.854287s 2023-10-30T12:02:19.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 0x7f60b0131c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:19.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f60b0135d80 0x7f60b009db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:19.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f60b0135d80 0x7f60b009db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46504/0 (socket says 172.21.15.50:46504) 2023-10-30T12:02:19.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 -- 172.21.15.50:0/115842159 learned_addr learned my addr 172.21.15.50:0/115842159 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:19.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): get_auth_request con 0x7f60b0135d80 auth_method 0 2023-10-30T12:02:19.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:19.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:19.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:19.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:19.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:19.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 0x7f60b01316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:19.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:19.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 10 monclient(hunting): get_auth_request con 0x7f60b00a5f20 auth_method 0 2023-10-30T12:02:19.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:19.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:19.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:19.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 10 monclient(hunting): get_auth_request con 0x7f60b00a7140 auth_method 0 2023-10-30T12:02:19.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:19.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:19.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:19.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:19.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:19.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:19.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 1 -- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 msgr2=0x7f60b0131c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:02:19.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 1 -- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 msgr2=0x7f60b0131c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:02:19.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 0x7f60b0131c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:19.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient(hunting): handle_auth_done global_id 40322 payload 824 2023-10-30T12:02:19.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60aeffd640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 0x7f60b0131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:19.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 -- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 msgr2=0x7f60b01316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:19.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 0x7f60b01316e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 -- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 msgr2=0x7f60b0131c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:02:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 0x7f60b0131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 monclient: found mon.noname-c 2023-10-30T12:02:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 -- 172.21.15.50:0/115842159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a4009400 con 0x7f60b0135d80 2023-10-30T12:02:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f60affff640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f60b0135d80 0x7f60b009db10 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f60a800c800 tx=0x7f60a800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f6097fff640 10 client.?.objecter ms_handle_connect 0x7f60b0135d80 2023-10-30T12:02:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.671+0000 7f6097fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8003b70 con 0x7f60b0135d80 2023-10-30T12:02:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: dump: 2023-10-30T12:02:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.705 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:19.706 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:19.706 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:19.706 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:19.706 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:19.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: _finish_auth 0 2023-10-30T12:02:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: _check_auth_tickets 2023-10-30T12:02:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 5 monclient: authenticate success, global_id 40322 2023-10-30T12:02:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8003cf0 con 0x7f60b0135d80 2023-10-30T12:02:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 10 monclient: _renew_subs 2023-10-30T12:02:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b009fc90 con 0x7f60b0135d80 2023-10-30T12:02:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a800aa90 con 0x7f60b0135d80 2023-10-30T12:02:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f6097fff640 10 monclient: dump: 2023-10-30T12:02:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:19.710 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60adffb640 4 set_mon_vals no callback set 2023-10-30T12:02:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 10 client.40322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 10 monclient: _renew_subs 2023-10-30T12:02:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b00a1800 con 0x7f60b0135d80 2023-10-30T12:02:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.672+0000 7f60b5d41640 1 librados: init done 2023-10-30T12:02:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f60a800abf0 con 0x7f60b0135d80 2023-10-30T12:02:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f60b5d41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f60b5d41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f60b5d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b011ef70 con 0x7f60b0135d80 2023-10-30T12:02:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f609c0392f0 0x7f609c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 10024+0+0 (secure 0 0 0) 0x7f60a804b990 con 0x7f60b0135d80 2023-10-30T12:02:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 10 client.40322.objecter ms_dispatch 0x7f60b0000b90 osd_map(38..38 src has 1..38) v4 2023-10-30T12:02:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 3 client.40322.objecter handle_osd_map got epochs [38,38] > 0 2023-10-30T12:02:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 3 client.40322.objecter handle_osd_map decoding full epoch 38 2023-10-30T12:02:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 10 client.40322.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 10 monclient: _renew_subs 2023-10-30T12:02:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=39}) v3 -- 0x7f609c04a3c0 con 0x7f60b0135d80 2023-10-30T12:02:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f60af7fe640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f609c0392f0 0x7f609c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f6097fff640 20 client.40322.objecter dump_active .. 0 homeless 2023-10-30T12:02:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.673+0000 7f60af7fe640 10 monclient: get_auth_request con 0x7f609c0392f0 auth_method 0 2023-10-30T12:02:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.674+0000 7f6097fff640 10 client.40322.objecter ms_handle_connect 0x7f609c0392f0 2023-10-30T12:02:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.674+0000 7f60af7fe640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f609c0392f0 0x7f609c03b7b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f60a4036000 tx=0x7f60a403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.678+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f60b011ef70 con 0x7f60b0135d80 2023-10-30T12:02:19.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.678+0000 7f6097fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:19.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.678+0000 7f6097fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.943+0000 7f60b5d41640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:02:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.943+0000 7f60b5d41640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.943+0000 7f60b5d41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:19.943+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f60b011ef70 con 0x7f60b0135d80 2023-10-30T12:02:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:22.671+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:22.671+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:22.671+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:22.671+0000 7f6096ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:23.671+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:23.671+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:23.671+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:24.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:24.672+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:24.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:24.672+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:24.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:24.672+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:24.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:24.673+0000 7f60acff9640 10 client.40322.objecter tick 2023-10-30T12:02:25.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:25.671+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:25.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:25.671+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:25.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:25.671+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:26.671+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:26.671+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:26.671+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:26.746 INFO:tasks.workunit.client.0.smithi118.stdout:22 objects, 578 KiB 2023-10-30T12:02:26.746 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: usage: 161 MiB used, 540 GiB / 540 GiB avail 2023-10-30T12:02:26.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: pgs: 55.278% pgs unknown 2023-10-30T12:02:26.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 288 unknown 2023-10-30T12:02:26.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 233 active+clean 2023-10-30T12:02:26.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:26.747 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:26.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4369 ms) 2023-10-30T12:02:26.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ RUN ] LibRadosService.Status 2023-10-30T12:02:26.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ OK ] LibRadosService.Status (20039 ms) 2023-10-30T12:02:26.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [----------] 4 tests from LibRadosService (32324 ms total) 2023-10-30T12:02:26.748 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: 2023-10-30T12:02:26.749 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [----------] Global test environment tear-down 2023-10-30T12:02:26.749 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32324 ms total) 2023-10-30T12:02:26.749 INFO:tasks.workunit.client.0.smithi118.stdout: api_service: [ PASSED ] 4 tests. 2023-10-30T12:02:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:27.671+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:27.671+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:27.671+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:28.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:28.672+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:28.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:28.672+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:28.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:28.672+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:29.672+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:29.672+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:29.672+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:29.674+0000 7f60acff9640 10 client.40322.objecter tick 2023-10-30T12:02:30.308 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-10-30T12:02:30.309 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [----------] Global test environment set-up. 2023-10-30T12:02:30.309 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-10-30T12:02:30.309 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-10-30T12:02:30.309 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (7811 ms) 2023-10-30T12:02:30.309 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-10-30T12:02:30.309 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (100 ms) 2023-10-30T12:02:30.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-10-30T12:02:30.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20037 ms) 2023-10-30T12:02:30.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-10-30T12:02:30.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: attempt 0 of 20 2023-10-30T12:02:30.310 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (7969 ms) 2023-10-30T12:02:30.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (35917 ms total) 2023-10-30T12:02:30.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: 2023-10-30T12:02:30.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [----------] Global test environment tear-down 2023-10-30T12:02:30.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (35917 ms total) 2023-10-30T12:02:30.311 INFO:tasks.workunit.client.0.smithi118.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-10-30T12:02:30.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:30.672+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:30.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:30.672+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:30.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:30.672+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.672+0000 7f6096ffd640 10 monclient: tick 2023-10-30T12:02:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.672+0000 7f6096ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:31.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.672+0000 7f6096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:31.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.884+0000 7f6097fff640 1 -- 172.21.15.50:0/115842159 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f60a803d070 con 0x7f60b0135d80 2023-10-30T12:02:31.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.884+0000 7f6097fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:31.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.884+0000 7f6097fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:31.886 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:02:31.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 10 librados: watch_flush enter 2023-10-30T12:02:31.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 10 librados: watch_flush exit 2023-10-30T12:02:31.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 20 client.40322.objecter shutdown clearing up homeless session... 2023-10-30T12:02:31.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 10 client.40322.objecter successfully canceled tick 2023-10-30T12:02:31.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f609c0392f0 msgr2=0x7f609c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:31.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f609c0392f0 0x7f609c03b7b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f60a4036000 tx=0x7f60a403d040 comp rx=0 tx=0).stop 2023-10-30T12:02:31.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 10 monclient: shutdown 2023-10-30T12:02:31.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:31.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f60b0135d80 msgr2=0x7f60b009db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:31.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f60b0135d80 0x7f60b009db10 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f60a800c800 tx=0x7f60a800ccc0 comp rx=0 tx=0).stop 2023-10-30T12:02:31.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 shutdown_connections 2023-10-30T12:02:31.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f609c0392f0 0x7f609c03b7b0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:31.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f60b0135d80 0x7f60b009db10 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:31.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b00a7140 0x7f60b0131c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:31.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 --2- 172.21.15.50:0/115842159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60b00a5f20 0x7f60b01316e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:31.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.886+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 >> 172.21.15.50:0/115842159 conn(0x7f60b00a8240 msgr2=0x7f60b00a32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:31.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.887+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 shutdown_connections 2023-10-30T12:02:31.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.887+0000 7f60b5d41640 1 -- 172.21.15.50:0/115842159 wait complete. 2023-10-30T12:02:31.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:31.887+0000 7f60b5d41640 1 librados: shutdown 2023-10-30T12:02:31.904 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:02:31.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:02:31.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:02:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.016+0000 7f42d28c2640 10 monclient: get_monmap_and_config 2023-10-30T12:02:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.016+0000 7f42d28c2640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.016+0000 7f42d28c2640 10 monclient: monmap: 2023-10-30T12:02:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:32.017645+0000 2023-10-30T12:02:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:32.017645+0000 2023-10-30T12:02:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:32.020 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 1 Processor -- start 2023-10-30T12:02:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 1 -- start start 2023-10-30T12:02:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient: init 2023-10-30T12:02:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-10-30T12:02:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f42cc06afc0 0x7f42cc0a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient(hunting): picked mon.noname-c con 0x7f42cc06afc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f42cc06b640 0x7f42cc0a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient(hunting): picked mon.noname-a con 0x7f42cc06b640 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42cc0a2710 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc06bde0 con 0x7f42cc0a2710 2023-10-30T12:02:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc06b3a0 con 0x7f42cc06b640 2023-10-30T12:02:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc09e540 con 0x7f42cc06afc0 2023-10-30T12:02:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d28c2640 10 monclient(hunting): authenticate will time out at 997.200850s 2023-10-30T12:02:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42cb7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f42cc06b640 0x7f42cc0a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d0e5f640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f42cc0a2710 0x7f42cc13c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34374/0 (socket says 172.21.15.50:34374) 2023-10-30T12:02:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42cb7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f42cc06b640 0x7f42cc0a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40198/0 (socket says 172.21.15.50:40198) 2023-10-30T12:02:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42d0e5f640 1 -- 172.21.15.50:0/4052186454 learned_addr learned my addr 172.21.15.50:0/4052186454 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42cb7fe640 10 monclient(hunting): get_auth_request con 0x7f42cc06b640 auth_method 0 2023-10-30T12:02:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42cb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42cb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.017+0000 7f42cbfff640 1 --2- 172.21.15.50:0/4052186454 >> v2:172.21.15.118:3300/0 conn(0x7f42cc06afc0 0x7f42cc0a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cbfff640 10 monclient(hunting): get_auth_request con 0x7f42cc06afc0 auth_method 0 2023-10-30T12:02:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_done global_id 52194 payload 274 2023-10-30T12:02:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 1 -- 172.21.15.50:0/4052186454 >> v2:172.21.15.118:3300/0 conn(0x7f42cc06afc0 msgr2=0x7f42cc0a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 1 --2- 172.21.15.50:0/4052186454 >> v2:172.21.15.118:3300/0 conn(0x7f42cc06afc0 0x7f42cc0a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 1 -- 172.21.15.50:0/4052186454 >> v1:172.21.15.50:6789/0 conn(0x7f42cc0a2710 legacy=0x7f42cc13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 1 monclient: found mon.noname-a 2023-10-30T12:02:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42caffd640 1 -- 172.21.15.50:0/4052186454 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233983475 0 0) 0x7f42cc06bde0 con 0x7f42cc0a2710 2023-10-30T12:02:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42cb7fe640 1 -- 172.21.15.50:0/4052186454 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc09fa10 con 0x7f42cc06b640 2023-10-30T12:02:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.018+0000 7f42caffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:02:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42cb7fe640 1 --2- 172.21.15.50:0/4052186454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 0x7f42cc0a21d0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f42c00097a0 tx=0x7f42c002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4193861672434e38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 1 -- 172.21.15.50:0/4052186454 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c002f050 con 0x7f42cc06b640 2023-10-30T12:02:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: dump: 2023-10-30T12:02:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.039 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 1 -- 172.21.15.50:0/4052186454 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c0031700 con 0x7f42cc06b640 2023-10-30T12:02:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 5 monclient: authenticate success, global_id 52194 2023-10-30T12:02:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 4 set_mon_vals no callback set 2023-10-30T12:02:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 1 -- 172.21.15.50:0/4052186454 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0031a90 con 0x7f42cc06b640 2023-10-30T12:02:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42caffd640 10 monclient: dump: 2023-10-30T12:02:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.043 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 10 monclient: shutdown 2023-10-30T12:02:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 -- 172.21.15.50:0/4052186454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 msgr2=0x7f42cc0a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4052186454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 0x7f42cc0a21d0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f42c00097a0 tx=0x7f42c002b7a0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 -- 172.21.15.50:0/4052186454 shutdown_connections 2023-10-30T12:02:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4052186454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 0x7f42cc0a21d0 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4052186454 >> v2:172.21.15.118:3300/0 conn(0x7f42cc06afc0 0x7f42cc0a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 -- 172.21.15.50:0/4052186454 >> 172.21.15.50:0/4052186454 conn(0x7f42cc0a6200 msgr2=0x7f42cc0a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 -- 172.21.15.50:0/4052186454 shutdown_connections 2023-10-30T12:02:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.019+0000 7f42d28c2640 1 -- 172.21.15.50:0/4052186454 wait complete. 2023-10-30T12:02:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.020+0000 7f42d28c2640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.020+0000 7f42d28c2640 10 monclient: monmap: 2023-10-30T12:02:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:32.047 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.020+0000 7f42d28c2640 1 librados: starting msgr at 2023-10-30T12:02:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.020+0000 7f42d28c2640 1 librados: starting objecter 2023-10-30T12:02:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.020+0000 7f42d28c2640 1 Processor -- start 2023-10-30T12:02:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 -- start start 2023-10-30T12:02:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 librados: setting wanted keys 2023-10-30T12:02:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 librados: calling monclient init 2023-10-30T12:02:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient: init 2023-10-30T12:02:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:02:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42cc06afc0 0x7f42cc1c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): picked mon.noname-c con 0x7f42cc06afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 0x7f42cc1c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): picked mon.noname-a con 0x7f42cc06b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42cc0a2710 0x7f42cc1c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): picked mon.noname-b con 0x7f42cc0a2710 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc1c7260 con 0x7f42cc06b640 2023-10-30T12:02:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc1a8970 con 0x7f42cc0a2710 2023-10-30T12:02:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc070750 con 0x7f42cc06afc0 2023-10-30T12:02:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d28c2640 10 monclient(hunting): authenticate will time out at 997.204513s 2023-10-30T12:02:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42cc06afc0 0x7f42cc1c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 0x7f42cc1c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42cc06afc0 0x7f42cc1c3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38270/0 (socket says 172.21.15.50:38270) 2023-10-30T12:02:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 -- 172.21.15.50:0/4084950325 learned_addr learned my addr 172.21.15.50:0/4084950325 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): get_auth_request con 0x7f42cc06afc0 auth_method 0 2023-10-30T12:02:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 10 monclient(hunting): get_auth_request con 0x7f42cc06b640 auth_method 0 2023-10-30T12:02:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d0e5f640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42cc0a2710 0x7f42cc1c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d0e5f640 10 monclient(hunting): get_auth_request con 0x7f42cc0a2710 auth_method 0 2023-10-30T12:02:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d0e5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d0e5f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d0e5f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient(hunting): handle_auth_done global_id 40610 payload 824 2023-10-30T12:02:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42cc0a2710 msgr2=0x7f42cc1c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42cc0a2710 0x7f42cc1c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 msgr2=0x7f42cc1c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 0x7f42cc1c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 monclient: found mon.noname-c 2023-10-30T12:02:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cbfff640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c0009400 con 0x7f42cc06afc0 2023-10-30T12:02:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42cb7fe640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc06b640 0x7f42cc1c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:02:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.021+0000 7f42d0e5f640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42cc0a2710 0x7f42cc1c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:02:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42cbfff640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42cc06afc0 0x7f42cc1c3b60 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00e9b0 tx=0x7f42bc00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 client.?.objecter ms_handle_connect 0x7f42cc06afc0 2023-10-30T12:02:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc009670 con 0x7f42cc06afc0 2023-10-30T12:02:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: dump: 2023-10-30T12:02:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.060 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc00c9a0 con 0x7f42cc06afc0 2023-10-30T12:02:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00cd50 con 0x7f42cc06afc0 2023-10-30T12:02:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 5 monclient: authenticate success, global_id 40610 2023-10-30T12:02:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 10 monclient: _renew_subs 2023-10-30T12:02:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc1aac20 con 0x7f42cc06afc0 2023-10-30T12:02:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42ca7fc640 4 set_mon_vals no callback set 2023-10-30T12:02:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42c8ff9640 10 monclient: dump: 2023-10-30T12:02:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.064 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 10 client.40610.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 10 monclient: _renew_subs 2023-10-30T12:02:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc1ab1e0 con 0x7f42cc06afc0 2023-10-30T12:02:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.022+0000 7f42d28c2640 1 librados: init done 2023-10-30T12:02:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f42bc013470 con 0x7f42cc06afc0 2023-10-30T12:02:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42d28c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1c9d20 0x7f42cc1ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42d28c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc073ff0 con 0x7f42cc1c9d20 2023-10-30T12:02:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1c9d20 0x7f42cc1ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient: get_auth_request con 0x7f42cc1c9d20 auth_method 0 2023-10-30T12:02:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42b80392f0 0x7f42b803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 10425+0+0 (secure 0 0 0) 0x7f42bc04bfe0 con 0x7f42cc06afc0 2023-10-30T12:02:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 client.40610.objecter ms_dispatch 0x7f42cc000b90 osd_map(46..46 src has 1..46) v4 2023-10-30T12:02:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 3 client.40610.objecter handle_osd_map got epochs [46,46] > 0 2023-10-30T12:02:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 3 client.40610.objecter handle_osd_map decoding full epoch 46 2023-10-30T12:02:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_done global_id 52206 payload 824 2023-10-30T12:02:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42d0e5f640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42b80392f0 0x7f42b803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 client.40610.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42cc09df80 con 0x7f42cc1c9d20 2023-10-30T12:02:32.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42d0e5f640 10 monclient: get_auth_request con 0x7f42b80392f0 auth_method 0 2023-10-30T12:02:32.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 monclient: _renew_subs 2023-10-30T12:02:32.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7f42b804a1f0 con 0x7f42cc06afc0 2023-10-30T12:02:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 20 client.40610.objecter dump_active .. 0 homeless 2023-10-30T12:02:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 client.40610.objecter ms_handle_connect 0x7f42cc1c9d20 2023-10-30T12:02:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42cb7fe640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1c9d20 0x7f42cc1ca100 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f42cc069f50 tx=0x7f42c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 client.40610.objecter resend_mon_ops 2023-10-30T12:02:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c003d640 con 0x7f42cc1c9d20 2023-10-30T12:02:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.023+0000 7f42c8ff9640 10 monclient: dump: 2023-10-30T12:02:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.073 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 10 client.40610.objecter ms_handle_connect 0x7f42b80392f0 2023-10-30T12:02:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42d0e5f640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42b80392f0 0x7f42b803b7b0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f42c40012e0 tx=0x7f42c4007f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f42c0041070 con 0x7f42cc1c9d20 2023-10-30T12:02:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:32.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1c9d20 msgr2=0x7f42cc1ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1c9d20 0x7f42cc1ca100 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f42cc069f50 tx=0x7f42c00046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1c9d20 msgr2=0x7f42cc1ca100 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:32.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.024+0000 7f42c8ff9640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1c9d20 0x7f42cc1ca100 secure :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f42cc069f50 tx=0x7f42c00046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42d28c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1ca640 0x7f42cc1a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42d28c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc06a6f0 con 0x7f42cc1ca640 2023-10-30T12:02:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1ca640 0x7f42cc1a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient: get_auth_request con 0x7f42cc1ca640 auth_method 0 2023-10-30T12:02:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 10 monclient(hunting): handle_auth_done global_id 52224 payload 824 2023-10-30T12:02:32.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 1 -- 172.21.15.50:0/4084950325 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f42cc09f0b0 con 0x7f42cc1ca640 2023-10-30T12:02:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42c8ff9640 10 client.40610.objecter ms_handle_connect 0x7f42cc1ca640 2023-10-30T12:02:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42cb7fe640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1ca640 0x7f42cc1a3bd0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f42c00097a0 tx=0x7f42c0031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42c8ff9640 10 client.40610.objecter resend_mon_ops 2023-10-30T12:02:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0053070 con 0x7f42cc1ca640 2023-10-30T12:02:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.031+0000 7f42c8ff9640 10 monclient: dump: 2023-10-30T12:02:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.081 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f42c0046c00 con 0x7f42cc1ca640 2023-10-30T12:02:32.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42c8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:32.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42c8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:32.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1ca640 msgr2=0x7f42cc1a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42c8ff9640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1ca640 0x7f42cc1a3bd0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f42c00097a0 tx=0x7f42c0031000 comp rx=0 tx=0).stop 2023-10-30T12:02:32.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42c8ff9640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1ca640 msgr2=0x7f42cc1a3bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:32.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42c8ff9640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42cc1ca640 0x7f42cc1a3bd0 secure :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f42c00097a0 tx=0x7f42c0031000 comp rx=0 tx=0).stop 2023-10-30T12:02:32.083 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:32.083 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:32.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:32.084 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:02:32.084 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 20, 2023-10-30T12:02:32.084 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:32.084 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:02:32.084 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:32.085 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:32.085 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.085 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:02:32.085 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:32.085 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:32.085 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:32.086 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.086 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.086 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.086 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.086 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.086 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.087 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.087 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.087 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.087 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.087 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.087 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:32.088 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:32.088 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.088 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.088 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.088 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.088 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.089 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.089 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.089 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.089 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.089 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.089 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.090 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:32.090 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:32.090 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:32.090 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:32.091 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:32.091 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:32.091 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:32.091 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:32.091 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:32.091 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:32.092 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:32.093 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:32.093 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.093 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.093 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.093 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.093 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.094 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.094 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.094 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.094 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.094 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.095 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.095 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:32.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.095 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:32.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:32.096 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:32.096 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.096 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.096 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.096 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:32.097 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.097 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.097 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:32.097 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.098 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.098 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:32.098 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:32.098 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.099 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.099 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.099 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:32.099 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:32.100 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.100 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.100 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.100 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:32.100 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.101 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.101 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:32.101 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.102 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.102 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.102 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:32.102 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:32.103 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.103 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.103 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.103 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:32.104 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:32.104 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.104 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.104 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.104 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:32.105 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.105 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.105 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:32.106 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.106 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.106 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.106 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:32.106 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:32.107 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.107 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.107 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.107 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.107 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.108 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:32.108 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:32.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.108 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.108 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.108 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.110 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:02:32.110 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.110 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.110 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:32.110 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 10 librados: watch_flush enter 2023-10-30T12:02:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 10 librados: watch_flush exit 2023-10-30T12:02:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 20 client.40610.objecter shutdown clearing up homeless session... 2023-10-30T12:02:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 10 client.40610.objecter successfully canceled tick 2023-10-30T12:02:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42b80392f0 msgr2=0x7f42b803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42b80392f0 0x7f42b803b7b0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f42c40012e0 tx=0x7f42c4007f10 comp rx=0 tx=0).stop 2023-10-30T12:02:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 10 monclient: shutdown 2023-10-30T12:02:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42cc06afc0 msgr2=0x7f42cc1c3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42cc06afc0 0x7f42cc1c3b60 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00e9b0 tx=0x7f42bc00ee70 comp rx=0 tx=0).stop 2023-10-30T12:02:32.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d0e5f640 1 -- 172.21.15.50:0/4084950325 reap_dead start 2023-10-30T12:02:32.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 shutdown_connections 2023-10-30T12:02:32.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 --2- 172.21.15.50:0/4084950325 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42cc06afc0 0x7f42cc1c3b60 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 >> 172.21.15.50:0/4084950325 conn(0x7f42cc0a6200 msgr2=0x7f42cc06e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 shutdown_connections 2023-10-30T12:02:32.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 -- 172.21.15.50:0/4084950325 wait complete. 2023-10-30T12:02:32.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.032+0000 7f42d28c2640 1 librados: shutdown 2023-10-30T12:02:32.115 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:02:32.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:02:32.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:02:32.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.223+0000 7fbf6d8e7640 10 monclient: get_monmap_and_config 2023-10-30T12:02:32.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.223+0000 7fbf6d8e7640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.223+0000 7fbf6d8e7640 10 monclient: monmap: 2023-10-30T12:02:32.225 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.225 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.225 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:32.224651+0000 2023-10-30T12:02:32.226 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:32.224651+0000 2023-10-30T12:02:32.226 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.226 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.226 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:32.226 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:32.227 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:32.227 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:32.227 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:32.227 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 1 Processor -- start 2023-10-30T12:02:32.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 1 -- start start 2023-10-30T12:02:32.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient: init 2023-10-30T12:02:32.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-10-30T12:02:32.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf68073620 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:32.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf68074110 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:32.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf6813c400 0x7fbf6813e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient(hunting): picked mon.noname-a con 0x7fbf6813c400 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:32.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6809bdc0 con 0x7fbf68073620 2023-10-30T12:02:32.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680a01e0 con 0x7fbf68074110 2023-10-30T12:02:32.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf680744f0 con 0x7fbf6813c400 2023-10-30T12:02:32.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf6d8e7640 10 monclient(hunting): authenticate will time out at 997.407921s 2023-10-30T12:02:32.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf66ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbf68073620 0x7fbf68073a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34376/0 (socket says 172.21.15.50:34376) 2023-10-30T12:02:32.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.224+0000 7fbf66ffd640 1 -- 172.21.15.50:0/650382204 learned_addr learned my addr 172.21.15.50:0/650382204 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:32.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 1 --2- 172.21.15.50:0/650382204 >> v2:172.21.15.50:3300/0 conn(0x7fbf6813c400 0x7fbf6813e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): get_auth_request con 0x7fbf6813c400 auth_method 0 2023-10-30T12:02:32.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient(hunting): handle_auth_done global_id 52350 payload 274 2023-10-30T12:02:32.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 1 -- 172.21.15.50:0/650382204 >> v1:172.21.15.118:6789/0 conn(0x7fbf68074110 legacy=0x7fbf6813ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:32.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 1 -- 172.21.15.50:0/650382204 >> v1:172.21.15.50:6789/0 conn(0x7fbf68073620 legacy=0x7fbf68073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 1 monclient: found mon.noname-a 2023-10-30T12:02:32.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.225+0000 7fbf677fe640 1 -- 172.21.15.50:0/650382204 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680a0de0 con 0x7fbf6813c400 2023-10-30T12:02:32.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf677fe640 1 --2- 172.21.15.50:0/650382204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf6813c400 0x7fbf6813e7f0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009d90 tx=0x7fbf6002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27494324d889c1a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 1 -- 172.21.15.50:0/650382204 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6002cbd0 con 0x7fbf6813c400 2023-10-30T12:02:32.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:32.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: dump: 2023-10-30T12:02:32.242 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.242 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.243 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.243 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.243 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.243 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.243 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.244 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.244 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.244 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 5 monclient: authenticate success, global_id 52350 2023-10-30T12:02:32.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 1 -- 172.21.15.50:0/650382204 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6002cd30 con 0x7fbf6813c400 2023-10-30T12:02:32.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:32.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 4 set_mon_vals no callback set 2023-10-30T12:02:32.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 1 -- 172.21.15.50:0/650382204 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6002cea0 con 0x7fbf6813c400 2023-10-30T12:02:32.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf667fc640 10 monclient: dump: 2023-10-30T12:02:32.246 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.246 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.247 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.247 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.247 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.247 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.247 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.247 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.248 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 10 monclient: shutdown 2023-10-30T12:02:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/650382204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf6813c400 msgr2=0x7fbf6813e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 1 --2- 172.21.15.50:0/650382204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf6813c400 0x7fbf6813e7f0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009d90 tx=0x7fbf6002be20 comp rx=0 tx=0).stop 2023-10-30T12:02:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/650382204 shutdown_connections 2023-10-30T12:02:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 1 --2- 172.21.15.50:0/650382204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf6813c400 0x7fbf6813e7f0 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.226+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/650382204 >> 172.21.15.50:0/650382204 conn(0x7fbf680a12a0 msgr2=0x7fbf680a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.227+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/650382204 shutdown_connections 2023-10-30T12:02:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.227+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/650382204 wait complete. 2023-10-30T12:02:32.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.227+0000 7fbf6d8e7640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.227+0000 7fbf6d8e7640 10 monclient: monmap: 2023-10-30T12:02:32.250 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.250 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:32.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:32.252 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.227+0000 7fbf6d8e7640 1 librados: starting msgr at 2023-10-30T12:02:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.227+0000 7fbf6d8e7640 1 librados: starting objecter 2023-10-30T12:02:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.227+0000 7fbf6d8e7640 1 Processor -- start 2023-10-30T12:02:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 -- start start 2023-10-30T12:02:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 librados: setting wanted keys 2023-10-30T12:02:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 librados: calling monclient init 2023-10-30T12:02:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient: init 2023-10-30T12:02:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:02:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf68073620 0x7fbf6808db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): picked mon.noname-a con 0x7fbf68073620 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68074110 0x7fbf6808e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): picked mon.noname-c con 0x7fbf68074110 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf6813c400 0x7fbf6808e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): picked mon.noname-b con 0x7fbf6813c400 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68090e60 con 0x7fbf68073620 2023-10-30T12:02:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68091650 con 0x7fbf6813c400 2023-10-30T12:02:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68091370 con 0x7fbf68074110 2023-10-30T12:02:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf6d8e7640 10 monclient(hunting): authenticate will time out at 997.411511s 2023-10-30T12:02:32.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf68073620 0x7fbf6808db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68074110 0x7fbf6808e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf68073620 0x7fbf6808db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40242/0 (socket says 172.21.15.50:40242) 2023-10-30T12:02:32.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 1 -- 172.21.15.50:0/54351131 learned_addr learned my addr 172.21.15.50:0/54351131 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:32.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): get_auth_request con 0x7fbf68074110 auth_method 0 2023-10-30T12:02:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 10 monclient(hunting): get_auth_request con 0x7fbf68073620 auth_method 0 2023-10-30T12:02:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf6813c400 0x7fbf6808e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 10 monclient(hunting): get_auth_request con 0x7fbf6813c400 auth_method 0 2023-10-30T12:02:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient(hunting): handle_auth_done global_id 40751 payload 824 2023-10-30T12:02:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf6813c400 msgr2=0x7fbf6808e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf6813c400 0x7fbf6808e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf68073620 msgr2=0x7fbf6808db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf68073620 0x7fbf6808db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 1 monclient: found mon.noname-c 2023-10-30T12:02:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf5ffff640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60009a70 con 0x7fbf68074110 2023-10-30T12:02:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.228+0000 7fbf66ffd640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf68073620 0x7fbf6808db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:02:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf5ffff640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68074110 0x7fbf6808e0b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c410 tx=0x7fbf5000c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 client.?.objecter ms_handle_connect 0x7fbf68074110 2023-10-30T12:02:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf500035f0 con 0x7fbf68074110 2023-10-30T12:02:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: dump: 2023-10-30T12:02:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.265 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.265 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.265 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50003770 con 0x7fbf68074110 2023-10-30T12:02:32.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50003b60 con 0x7fbf68074110 2023-10-30T12:02:32.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf64ff9640 10 monclient: dump: 2023-10-30T12:02:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.268 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 5 monclient: authenticate success, global_id 40751 2023-10-30T12:02:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 10 monclient: _renew_subs 2023-10-30T12:02:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf68091090 con 0x7fbf68074110 2023-10-30T12:02:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf65ffb640 4 set_mon_vals no callback set 2023-10-30T12:02:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 10 client.40751.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 10 monclient: _renew_subs 2023-10-30T12:02:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6808f3c0 con 0x7fbf68074110 2023-10-30T12:02:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.229+0000 7fbf6d8e7640 1 librados: init done 2023-10-30T12:02:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbf50003cc0 con 0x7fbf68074110 2023-10-30T12:02:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf6d8e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf6d8e7640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68091a70 0x7fbf681ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf6d8e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6808ff10 con 0x7fbf68091a70 2023-10-30T12:02:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf480392f0 0x7fbf4803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68091a70 0x7fbf681ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient: get_auth_request con 0x7fbf68091a70 auth_method 0 2023-10-30T12:02:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 10425+0+0 (secure 0 0 0) 0x7fbf5004afb0 con 0x7fbf68074110 2023-10-30T12:02:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 10 client.40751.objecter ms_dispatch 0x7fbf68000b90 osd_map(46..46 src has 1..46) v4 2023-10-30T12:02:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 3 client.40751.objecter handle_osd_map got epochs [46,46] > 0 2023-10-30T12:02:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 3 client.40751.objecter handle_osd_map decoding full epoch 46 2023-10-30T12:02:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf677fe640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf480392f0 0x7fbf4803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 10 client.40751.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 10 monclient: _renew_subs 2023-10-30T12:02:32.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:32.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7fbf4804a270 con 0x7fbf68074110 2023-10-30T12:02:32.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf677fe640 10 monclient: get_auth_request con 0x7fbf480392f0 auth_method 0 2023-10-30T12:02:32.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf64ff9640 20 client.40751.objecter dump_active .. 0 homeless 2023-10-30T12:02:32.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_done global_id 40754 payload 824 2023-10-30T12:02:32.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.230+0000 7fbf66ffd640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf680722f0 con 0x7fbf68091a70 2023-10-30T12:02:32.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 client.40751.objecter ms_handle_connect 0x7fbf68091a70 2023-10-30T12:02:32.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 client.40751.objecter resend_mon_ops 2023-10-30T12:02:32.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf66ffd640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68091a70 0x7fbf681ca580 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fbf58000f60 tx=0x7fbf58001e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58005ac0 con 0x7fbf68091a70 2023-10-30T12:02:32.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:32.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 monclient: dump: 2023-10-30T12:02:32.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.276 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.276 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.277 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.277 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.277 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.277 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.277 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.278 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.278 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 client.40751.objecter ms_handle_connect 0x7fbf480392f0 2023-10-30T12:02:32.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf677fe640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf480392f0 0x7fbf4803b7b0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fbf60036000 tx=0x7fbf60030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fbf58005c20 con 0x7fbf68091a70 2023-10-30T12:02:32.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68091a70 msgr2=0x7fbf681ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68091a70 0x7fbf681ca580 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fbf58000f60 tx=0x7fbf58001e60 comp rx=0 tx=0).stop 2023-10-30T12:02:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68091a70 msgr2=0x7fbf681ca580 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.231+0000 7fbf64ff9640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68091a70 0x7fbf681ca580 secure :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fbf58000f60 tx=0x7fbf58001e60 comp rx=0 tx=0).stop 2023-10-30T12:02:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf6d8e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf6d8e7640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf681caac0 0x7fbf681caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf6d8e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6806a580 con 0x7fbf681caac0 2023-10-30T12:02:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf681caac0 0x7fbf681caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 10 monclient: get_auth_request con 0x7fbf681caac0 auth_method 0 2023-10-30T12:02:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.238+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf66ffd640 10 monclient(hunting): handle_auth_done global_id 40766 payload 824 2023-10-30T12:02:32.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf66ffd640 1 -- 172.21.15.50:0/54351131 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbf681cb410 con 0x7fbf681caac0 2023-10-30T12:02:32.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 10 client.40751.objecter ms_handle_connect 0x7fbf681caac0 2023-10-30T12:02:32.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 10 client.40751.objecter resend_mon_ops 2023-10-30T12:02:32.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf66ffd640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf681caac0 0x7fbf681caed0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fbf58000f60 tx=0x7fbf580057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58024070 con 0x7fbf681caac0 2023-10-30T12:02:32.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:32.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 10 monclient: dump: 2023-10-30T12:02:32.285 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.285 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.285 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.285 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.286 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.286 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.286 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.286 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7fbf58005d90 con 0x7fbf681caac0 2023-10-30T12:02:32.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:32.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:32.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf681caac0 msgr2=0x7fbf681caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf681caac0 0x7fbf681caed0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fbf58000f60 tx=0x7fbf580057f0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf681caac0 msgr2=0x7fbf681caed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:32.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.239+0000 7fbf64ff9640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf681caac0 0x7fbf681caed0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.288 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:32.288 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:32.289 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:32.289 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:02:32.289 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 20, 2023-10-30T12:02:32.289 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:32.289 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:02:32.289 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:32.290 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:32.290 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.290 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:02:32.290 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:32.290 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:32.291 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:32.291 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.291 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.291 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.291 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.291 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:02:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:02:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 12 2023-10-30T12:02:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:32.318 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 10 librados: watch_flush enter 2023-10-30T12:02:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 10 librados: watch_flush exit 2023-10-30T12:02:32.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 20 client.40751.objecter shutdown clearing up homeless session... 2023-10-30T12:02:32.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 10 client.40751.objecter successfully canceled tick 2023-10-30T12:02:32.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf480392f0 msgr2=0x7fbf4803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbf480392f0 0x7fbf4803b7b0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fbf60036000 tx=0x7fbf60030000 comp rx=0 tx=0).stop 2023-10-30T12:02:32.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 10 monclient: shutdown 2023-10-30T12:02:32.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68074110 msgr2=0x7fbf6808e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 --2- 172.21.15.50:0/54351131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbf68074110 0x7fbf6808e0b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c410 tx=0x7fbf5000c810 comp rx=0 tx=0).stop 2023-10-30T12:02:32.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf677fe640 1 -- 172.21.15.50:0/54351131 reap_dead start 2023-10-30T12:02:32.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf677fe640 1 -- 172.21.15.50:0/54351131 reap_dead start 2023-10-30T12:02:32.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 shutdown_connections 2023-10-30T12:02:32.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 >> 172.21.15.50:0/54351131 conn(0x7fbf680a12a0 msgr2=0x7fbf68138a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 shutdown_connections 2023-10-30T12:02:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 -- 172.21.15.50:0/54351131 wait complete. 2023-10-30T12:02:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.240+0000 7fbf6d8e7640 1 librados: shutdown 2023-10-30T12:02:32.323 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:02:32.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:02:32.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:02:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.434+0000 7f45b4d4e640 10 monclient: get_monmap_and_config 2023-10-30T12:02:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.434+0000 7f45b4d4e640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.434+0000 7f45b4d4e640 10 monclient: monmap: 2023-10-30T12:02:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:32.435677+0000 2023-10-30T12:02:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:32.435677+0000 2023-10-30T12:02:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:32.438 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:32.438 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:32.438 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:32.438 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 1 Processor -- start 2023-10-30T12:02:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 1 -- start start 2023-10-30T12:02:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient: init 2023-10-30T12:02:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-10-30T12:02:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45b009fa30 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45b006b720 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:32.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f45b013c080 0x7f45b013e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient(hunting): picked mon.noname-a con 0x7f45b013c080 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:32.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b009e010 con 0x7f45b006b720 2023-10-30T12:02:32.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b009fe10 con 0x7f45b009fa30 2023-10-30T12:02:32.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b006bde0 con 0x7f45b013c080 2023-10-30T12:02:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45b4d4e640 10 monclient(hunting): authenticate will time out at 997.618801s 2023-10-30T12:02:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45a5d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f45b006b720 0x7f45b00a2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34382/0 (socket says 172.21.15.50:34382) 2023-10-30T12:02:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45a5d9b640 1 -- 172.21.15.50:0/905162305 learned_addr learned my addr 172.21.15.50:0/905162305 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:32.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45aed9d640 1 --2- 172.21.15.50:0/905162305 >> v2:172.21.15.50:3300/0 conn(0x7f45b013c080 0x7f45b013e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45aed9d640 10 monclient(hunting): get_auth_request con 0x7f45b013c080 auth_method 0 2023-10-30T12:02:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45aed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45aed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.435+0000 7f45aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45add9b640 1 -- 172.21.15.50:0/905162305 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194289899 0 0) 0x7f45b009e010 con 0x7f45b006b720 2023-10-30T12:02:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45add9b640 10 monclient(hunting): _negotiate 2023-10-30T12:02:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45add9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45add9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45add9b640 10 monclient(hunting): my global_id is 52575 2023-10-30T12:02:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45add9b640 1 -- 172.21.15.50:0/905162305 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c003520 con 0x7f45b006b720 2023-10-30T12:02:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 10 monclient(hunting): handle_auth_done global_id 52572 payload 274 2023-10-30T12:02:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 1 -- 172.21.15.50:0/905162305 >> v1:172.21.15.118:6789/0 conn(0x7f45b009fa30 legacy=0x7f45b00a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 1 -- 172.21.15.50:0/905162305 >> v1:172.21.15.50:6789/0 conn(0x7f45b006b720 legacy=0x7f45b00a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 1 monclient: found mon.noname-a 2023-10-30T12:02:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 1 -- 172.21.15.50:0/905162305 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b009e540 con 0x7f45b013c080 2023-10-30T12:02:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.436+0000 7f45aed9d640 1 --2- 172.21.15.50:0/905162305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b013c080 0x7f45b013e470 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009dc0 tx=0x7f45a802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7492364c8797d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 1 -- 172.21.15.50:0/905162305 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a802cbb0 con 0x7f45b013c080 2023-10-30T12:02:32.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:32.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: dump: 2023-10-30T12:02:32.454 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.454 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.454 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.455 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.455 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.455 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.455 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.455 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.455 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.456 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 1 -- 172.21.15.50:0/905162305 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a802cd10 con 0x7f45b013c080 2023-10-30T12:02:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 1 -- 172.21.15.50:0/905162305 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8034660 con 0x7f45b013c080 2023-10-30T12:02:32.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 5 monclient: authenticate success, global_id 52572 2023-10-30T12:02:32.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:32.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 4 set_mon_vals no callback set 2023-10-30T12:02:32.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45add9b640 10 monclient: dump: 2023-10-30T12:02:32.458 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.458 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.458 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.458 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.459 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 10 monclient: shutdown 2023-10-30T12:02:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 1 -- 172.21.15.50:0/905162305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b013c080 msgr2=0x7f45b013e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 1 --2- 172.21.15.50:0/905162305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b013c080 0x7f45b013e470 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009dc0 tx=0x7f45a802be00 comp rx=0 tx=0).stop 2023-10-30T12:02:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 1 -- 172.21.15.50:0/905162305 shutdown_connections 2023-10-30T12:02:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 1 --2- 172.21.15.50:0/905162305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b013c080 0x7f45b013e470 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 1 -- 172.21.15.50:0/905162305 >> 172.21.15.50:0/905162305 conn(0x7f45b00a6200 msgr2=0x7f45b00a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 1 -- 172.21.15.50:0/905162305 shutdown_connections 2023-10-30T12:02:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.437+0000 7f45b4d4e640 1 -- 172.21.15.50:0/905162305 wait complete. 2023-10-30T12:02:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 10 monclient: monmap: 2023-10-30T12:02:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:32.463 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 1 librados: starting msgr at 2023-10-30T12:02:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 1 librados: starting objecter 2023-10-30T12:02:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 1 Processor -- start 2023-10-30T12:02:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 1 -- start start 2023-10-30T12:02:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 1 librados: setting wanted keys 2023-10-30T12:02:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 1 librados: calling monclient init 2023-10-30T12:02:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.438+0000 7f45b4d4e640 10 monclient: init 2023-10-30T12:02:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:02:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b009fa30 0x7f45b0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): picked mon.noname-a con 0x7f45b009fa30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45b013c080 0x7f45b0099f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): picked mon.noname-c con 0x7f45b013c080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:32.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b009a4c0 0x7f45b0133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): picked mon.noname-b con 0x7f45b009a4c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b009cbc0 con 0x7f45b009fa30 2023-10-30T12:02:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b009c8e0 con 0x7f45b009a4c0 2023-10-30T12:02:32.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b009b770 con 0x7f45b013c080 2023-10-30T12:02:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45b4d4e640 10 monclient(hunting): authenticate will time out at 997.622476s 2023-10-30T12:02:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b009fa30 0x7f45b0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45b013c080 0x7f45b0099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b009fa30 0x7f45b0099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40266/0 (socket says 172.21.15.50:40266) 2023-10-30T12:02:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45b013c080 0x7f45b0099f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38308/0 (socket says 172.21.15.50:38308) 2023-10-30T12:02:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 -- 172.21.15.50:0/1185992333 learned_addr learned my addr 172.21.15.50:0/1185992333 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): get_auth_request con 0x7f45b009fa30 auth_method 0 2023-10-30T12:02:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 10 monclient(hunting): get_auth_request con 0x7f45b013c080 auth_method 0 2023-10-30T12:02:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b009a4c0 0x7f45b0133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 10 monclient(hunting): get_auth_request con 0x7f45b009a4c0 auth_method 0 2023-10-30T12:02:32.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient(hunting): handle_auth_done global_id 52578 payload 824 2023-10-30T12:02:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45b013c080 msgr2=0x7f45b0099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45b013c080 0x7f45b0099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b009a4c0 msgr2=0x7f45b0133d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b009a4c0 0x7f45b0133d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 monclient: found mon.noname-a 2023-10-30T12:02:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45ae59c640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a8009a70 con 0x7f45b009fa30 2023-10-30T12:02:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.439+0000 7f45a5d9b640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45b013c080 0x7f45b0099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:02:32.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45ae59c640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b009fa30 0x7f45b0099a40 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f45a000c480 tx=0x7f45a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 client.?.objecter ms_handle_connect 0x7f45b009fa30 2023-10-30T12:02:32.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:32.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a00052d0 con 0x7f45b009fa30 2023-10-30T12:02:32.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:32.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: dump: 2023-10-30T12:02:32.475 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.475 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.475 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.475 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.476 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a0005430 con 0x7f45b009fa30 2023-10-30T12:02:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0012870 con 0x7f45b009fa30 2023-10-30T12:02:32.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45a77fe640 10 monclient: dump: 2023-10-30T12:02:32.478 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.478 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.479 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.479 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.479 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.479 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.479 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.480 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.480 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.480 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 5 monclient: authenticate success, global_id 52578 2023-10-30T12:02:32.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 10 monclient: _renew_subs 2023-10-30T12:02:32.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b0070e50 con 0x7f45b009fa30 2023-10-30T12:02:32.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45ad59a640 4 set_mon_vals no callback set 2023-10-30T12:02:32.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 10 client.52578.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 10 monclient: _renew_subs 2023-10-30T12:02:32.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b01386a0 con 0x7f45b009fa30 2023-10-30T12:02:32.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.440+0000 7f45b4d4e640 1 librados: init done 2023-10-30T12:02:32.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45b4d4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45b4d4e640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b006b720 0x7f45b01342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45b4d4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0073ff0 con 0x7f45b006b720 2023-10-30T12:02:32.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f45a00129d0 con 0x7f45b009fa30 2023-10-30T12:02:32.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b006b720 0x7f45b01342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45900392f0 0x7f459003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient: get_auth_request con 0x7f45b006b720 auth_method 0 2023-10-30T12:02:32.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 10425+0+0 (secure 0 0 0) 0x7f45a004b890 con 0x7f45b009fa30 2023-10-30T12:02:32.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 10 client.52578.objecter ms_dispatch 0x7f45b0000b90 osd_map(46..46 src has 1..46) v4 2023-10-30T12:02:32.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 3 client.52578.objecter handle_osd_map got epochs [46,46] > 0 2023-10-30T12:02:32.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 3 client.52578.objecter handle_osd_map decoding full epoch 46 2023-10-30T12:02:32.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 10 client.52578.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 10 monclient: _renew_subs 2023-10-30T12:02:32.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7f459004a1f0 con 0x7f45b009fa30 2023-10-30T12:02:32.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 20 client.52578.objecter dump_active .. 0 homeless 2023-10-30T12:02:32.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45aed9d640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45900392f0 0x7f459003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45aed9d640 10 monclient: get_auth_request con 0x7f45900392f0 auth_method 0 2023-10-30T12:02:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_done global_id 22894 payload 824 2023-10-30T12:02:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45b006ab30 con 0x7f45b006b720 2023-10-30T12:02:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 10 client.52578.objecter ms_handle_connect 0x7f45b006b720 2023-10-30T12:02:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a77fe640 10 client.52578.objecter resend_mon_ops 2023-10-30T12:02:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.441+0000 7f45a5d9b640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b006b720 0x7f45b01342c0 secure :-1 s=READY pgs=5612 cs=0 l=1 rev1=1 crypto rx=0x7f45b013b7e0 tx=0x7f4598001ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 10 client.52578.objecter ms_handle_connect 0x7f45900392f0 2023-10-30T12:02:32.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45aed9d640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45900392f0 0x7f459003b7b0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f45a8036000 tx=0x7f45a8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800f9b0 con 0x7f45b006b720 2023-10-30T12:02:32.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:32.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 10 monclient: dump: 2023-10-30T12:02:32.488 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.489 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.489 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.489 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.489 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.489 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.489 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.490 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.490 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.490 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f459800fb10 con 0x7f45b006b720 2023-10-30T12:02:32.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:32.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b006b720 msgr2=0x7f45b01342c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b006b720 0x7f45b01342c0 secure :-1 s=READY pgs=5612 cs=0 l=1 rev1=1 crypto rx=0x7f45b013b7e0 tx=0x7f4598001ca0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b006b720 msgr2=0x7f45b01342c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:32.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.442+0000 7f45a77fe640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b006b720 0x7f45b01342c0 unknown :-1 s=CLOSED pgs=5612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45b4d4e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:32.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45b4d4e640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b0134800 0x7f45b0136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45b4d4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b009f120 con 0x7f45b0134800 2023-10-30T12:02:32.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b0134800 0x7f45b0136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient: get_auth_request con 0x7f45b0134800 auth_method 0 2023-10-30T12:02:32.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 10 monclient(hunting): handle_auth_done global_id 22897 payload 824 2023-10-30T12:02:32.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.449+0000 7f45a5d9b640 1 -- 172.21.15.50:0/1185992333 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f45b006a690 con 0x7f45b0134800 2023-10-30T12:02:32.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a5d9b640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b0134800 0x7f45b0136300 secure :-1 s=READY pgs=5613 cs=0 l=1 rev1=1 crypto rx=0x7f4598018000 tx=0x7f4598001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 10 client.52578.objecter ms_handle_connect 0x7f45b0134800 2023-10-30T12:02:32.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 10 client.52578.objecter resend_mon_ops 2023-10-30T12:02:32.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598023070 con 0x7f45b0134800 2023-10-30T12:02:32.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:32.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 10 monclient: dump: 2023-10-30T12:02:32.497 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.497 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.498 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.498 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.498 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.498 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.498 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f459801c410 con 0x7f45b0134800 2023-10-30T12:02:32.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:32.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:32.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b0134800 msgr2=0x7f45b0136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b0134800 0x7f45b0136300 secure :-1 s=READY pgs=5613 cs=0 l=1 rev1=1 crypto rx=0x7f4598018000 tx=0x7f4598001150 comp rx=0 tx=0).stop 2023-10-30T12:02:32.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b0134800 msgr2=0x7f45b0136300 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:32.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45a77fe640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45b0134800 0x7f45b0136300 secure :-1 s=CLOSED pgs=5613 cs=0 l=1 rev1=1 crypto rx=0x7f4598018000 tx=0x7f4598001150 comp rx=0 tx=0).stop 2023-10-30T12:02:32.500 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:32.500 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:32.500 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:32.501 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:02:32.501 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 20, 2023-10-30T12:02:32.501 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:32.501 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:02:32.501 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:32.502 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:32.502 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.502 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:02:32.502 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:32.502 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:32.503 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:32.503 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.503 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.503 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.503 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.503 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.504 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.504 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.504 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.504 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.504 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.504 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.505 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:32.505 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:32.505 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.505 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.505 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.505 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.506 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.506 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.506 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.506 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.506 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.507 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.507 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.507 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:32.507 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:32.508 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:32.508 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:32.508 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:32.508 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:32.508 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:32.509 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:32.509 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:32.509 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:32.509 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:32.509 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:32.509 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:32.510 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:32.510 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:32.510 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:32.510 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:32.510 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:32.511 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:32.511 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:32.511 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:32.511 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:32.511 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:32.511 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:32.512 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:32.512 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:32.512 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:32.512 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.512 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:32.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.513 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:32.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.513 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:32.513 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:32.513 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.514 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:32.514 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.515 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.515 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:32.515 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.515 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.515 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.516 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.517 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:32.517 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:32.517 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.517 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.518 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.518 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:32.518 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.519 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.519 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:32.519 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.520 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.520 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:32.520 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:32.520 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.520 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.521 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.521 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:32.521 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:32.521 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:32.522 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:32.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.522 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:32.522 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:32.522 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.523 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:32.523 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:32.523 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:32.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.524 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.524 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:32.524 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:32.524 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:32.525 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:32.525 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:32.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.525 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.525 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:32.526 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:32.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.526 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.526 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.526 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:32.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.527 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.527 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:02:32.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.527 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.527 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.527 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:02:32.528 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.528 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.528 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:32.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.528 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.528 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.529 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 14 2023-10-30T12:02:32.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.529 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:32.529 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:02:32.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:32.529 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:32.530 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:32.530 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:32.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:32.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:32.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:32.531 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:32.531 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:32.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.450+0000 7f45b4d4e640 10 librados: watch_flush enter 2023-10-30T12:02:32.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 10 librados: watch_flush exit 2023-10-30T12:02:32.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 20 client.52578.objecter shutdown clearing up homeless session... 2023-10-30T12:02:32.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 10 client.52578.objecter successfully canceled tick 2023-10-30T12:02:32.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45900392f0 msgr2=0x7f459003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45900392f0 0x7f459003b7b0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f45a8036000 tx=0x7f45a8030000 comp rx=0 tx=0).stop 2023-10-30T12:02:32.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 10 monclient: shutdown 2023-10-30T12:02:32.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b009fa30 msgr2=0x7f45b0099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 --2- 172.21.15.50:0/1185992333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b009fa30 0x7f45b0099a40 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f45a000c480 tx=0x7f45a000c940 comp rx=0 tx=0).stop 2023-10-30T12:02:32.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45aed9d640 1 -- 172.21.15.50:0/1185992333 reap_dead start 2023-10-30T12:02:32.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45aed9d640 1 -- 172.21.15.50:0/1185992333 reap_dead start 2023-10-30T12:02:32.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 shutdown_connections 2023-10-30T12:02:32.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 >> 172.21.15.50:0/1185992333 conn(0x7f45b00a6200 msgr2=0x7f45b00a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 shutdown_connections 2023-10-30T12:02:32.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 -- 172.21.15.50:0/1185992333 wait complete. 2023-10-30T12:02:32.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.451+0000 7f45b4d4e640 1 librados: shutdown 2023-10-30T12:02:32.536 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:02:32.536 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:02:32.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:02:32.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.647+0000 7f7fb5664640 10 monclient: get_monmap_and_config 2023-10-30T12:02:32.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.647+0000 7f7fb5664640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.648+0000 7f7fb5664640 10 monclient: monmap: 2023-10-30T12:02:32.650 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.650 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:32.649042+0000 2023-10-30T12:02:32.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:32.649042+0000 2023-10-30T12:02:32.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.651 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.651 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:32.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:32.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:32.651 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:32.651 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:32.652 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.648+0000 7f7fb5664640 1 Processor -- start 2023-10-30T12:02:32.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.648+0000 7f7fb5664640 1 -- start start 2023-10-30T12:02:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.648+0000 7f7fb5664640 10 monclient: init 2023-10-30T12:02:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-10-30T12:02:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7fb00a6070 0x7f7fb00a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb00a6070 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7fb00a6450 0x7f7fb0135590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb00a6450 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb0135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb0135ad0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb0136460 con 0x7f7fb00a6070 2023-10-30T12:02:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb009a270 con 0x7f7fb00a6450 2023-10-30T12:02:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0099f90 con 0x7f7fb0135ad0 2023-10-30T12:02:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fb5664640 10 monclient(hunting): authenticate will time out at 997.832083s 2023-10-30T12:02:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb0135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7fb00a6070 0x7f7fb00a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb0135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38318/0 (socket says 172.21.15.50:38318) 2023-10-30T12:02:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 1 -- 172.21.15.50:0/3956859862 learned_addr learned my addr 172.21.15.50:0/3956859862 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): get_auth_request con 0x7f7fb00a6070 auth_method 0 2023-10-30T12:02:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 1 --2- 172.21.15.50:0/3956859862 >> v2:172.21.15.118:3300/0 conn(0x7f7fb00a6450 0x7f7fb0135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 10 monclient(hunting): get_auth_request con 0x7f7fb0135ad0 auth_method 0 2023-10-30T12:02:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 10 monclient(hunting): get_auth_request con 0x7f7fb00a6450 auth_method 0 2023-10-30T12:02:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7fa67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient(hunting): handle_auth_done global_id 52800 payload 274 2023-10-30T12:02:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 1 -- 172.21.15.50:0/3956859862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 msgr2=0x7f7fb0135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 1 --2- 172.21.15.50:0/3956859862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb0135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 1 -- 172.21.15.50:0/3956859862 >> v2:172.21.15.118:3300/0 conn(0x7f7fb00a6450 msgr2=0x7f7fb0135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 1 --2- 172.21.15.50:0/3956859862 >> v2:172.21.15.118:3300/0 conn(0x7f7fb00a6450 0x7f7fb0135590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 1 monclient: found mon.noname-a 2023-10-30T12:02:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.649+0000 7f7faeffd640 1 -- 172.21.15.50:0/3956859862 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb009a550 con 0x7f7fb00a6070 2023-10-30T12:02:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7faeffd640 1 --2- 172.21.15.50:0/3956859862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6070 0x7f7fb00a57e0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00a2110 tx=0x7f7fa002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8b535f17f3792e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 1 -- 172.21.15.50:0/3956859862 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0030710 con 0x7f7fb00a6070 2023-10-30T12:02:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: dump: 2023-10-30T12:02:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.671 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fb5664640 5 monclient: authenticate success, global_id 52800 2023-10-30T12:02:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fb5664640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 1 -- 172.21.15.50:0/3956859862 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa002e050 con 0x7f7fb00a6070 2023-10-30T12:02:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fb5664640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fb5664640 4 set_mon_vals no callback set 2023-10-30T12:02:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 1 -- 172.21.15.50:0/3956859862 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0030a90 con 0x7f7fb00a6070 2023-10-30T12:02:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.650+0000 7f7fae7fc640 10 monclient: dump: 2023-10-30T12:02:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.675 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 10 monclient: shutdown 2023-10-30T12:02:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 -- 172.21.15.50:0/3956859862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6070 msgr2=0x7f7fb00a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 --2- 172.21.15.50:0/3956859862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6070 0x7f7fb00a57e0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00a2110 tx=0x7f7fa002b710 comp rx=0 tx=0).stop 2023-10-30T12:02:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 -- 172.21.15.50:0/3956859862 shutdown_connections 2023-10-30T12:02:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 --2- 172.21.15.50:0/3956859862 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb0135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 --2- 172.21.15.50:0/3956859862 >> v2:172.21.15.118:3300/0 conn(0x7f7fb00a6450 0x7f7fb0135590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 --2- 172.21.15.50:0/3956859862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6070 0x7f7fb00a57e0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 -- 172.21.15.50:0/3956859862 >> 172.21.15.50:0/3956859862 conn(0x7f7fb00a81d0 msgr2=0x7f7fb00a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 -- 172.21.15.50:0/3956859862 shutdown_connections 2023-10-30T12:02:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 -- 172.21.15.50:0/3956859862 wait complete. 2023-10-30T12:02:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 10 monclient: build_initial_monmap 2023-10-30T12:02:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 10 monclient: monmap: 2023-10-30T12:02:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:32.679 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 librados: starting msgr at 2023-10-30T12:02:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.651+0000 7f7fb5664640 1 librados: starting objecter 2023-10-30T12:02:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 Processor -- start 2023-10-30T12:02:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 -- start start 2023-10-30T12:02:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 librados: setting wanted keys 2023-10-30T12:02:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 librados: calling monclient init 2023-10-30T12:02:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient: init 2023-10-30T12:02:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:02:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fb00a6070 0x7f7fb009e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb00a6070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6450 0x7f7fb009e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb00a6450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb009eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb0135ad0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb011f030 con 0x7f7fb00a6450 2023-10-30T12:02:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb00a1700 con 0x7f7fb00a6070 2023-10-30T12:02:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb011ed50 con 0x7f7fb0135ad0 2023-10-30T12:02:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fb5664640 10 monclient(hunting): authenticate will time out at 997.835861s 2023-10-30T12:02:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fa67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6450 0x7f7fb009e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7faf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb009eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fa67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6450 0x7f7fb009e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40280/0 (socket says 172.21.15.50:40280) 2023-10-30T12:02:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fa67fc640 1 -- 172.21.15.50:0/1899843191 learned_addr learned my addr 172.21.15.50:0/1899843191 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fa67fc640 10 monclient(hunting): get_auth_request con 0x7f7fb00a6450 auth_method 0 2023-10-30T12:02:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.652+0000 7f7fa67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faf7fe640 10 monclient(hunting): get_auth_request con 0x7f7fb0135ad0 auth_method 0 2023-10-30T12:02:32.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:32.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:32.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:32.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faeffd640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fb00a6070 0x7f7fb009e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faeffd640 10 monclient(hunting): get_auth_request con 0x7f7fb00a6070 auth_method 0 2023-10-30T12:02:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7faeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient(hunting): handle_auth_done global_id 52809 payload 824 2023-10-30T12:02:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 1 -- 172.21.15.50:0/1899843191 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 msgr2=0x7f7fb009eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb009eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 1 -- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fb00a6070 msgr2=0x7f7fb009e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fb00a6070 0x7f7fb009e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient: _finish_hunting 0 2023-10-30T12:02:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 1 monclient: found mon.noname-a 2023-10-30T12:02:32.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 1 -- 172.21.15.50:0/1899843191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa0009400 con 0x7f7fb00a6450 2023-10-30T12:02:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa67fc640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6450 0x7f7fb009e580 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800e9e0 tx=0x7f7f9800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 client.?.objecter ms_handle_connect 0x7f7fb00a6450 2023-10-30T12:02:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98009670 con 0x7f7fb00a6450 2023-10-30T12:02:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: dump: 2023-10-30T12:02:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.692 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: _finish_auth 0 2023-10-30T12:02:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:02:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9800c9d0 con 0x7f7fb00a6450 2023-10-30T12:02:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 5 monclient: authenticate success, global_id 52809 2023-10-30T12:02:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9800cd80 con 0x7f7fb00a6450 2023-10-30T12:02:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fa7fff640 10 monclient: dump: 2023-10-30T12:02:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:32.695 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fadffb640 4 set_mon_vals no callback set 2023-10-30T12:02:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 10 monclient: _renew_subs 2023-10-30T12:02:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb01324c0 con 0x7f7fb00a6450 2023-10-30T12:02:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 10 client.52809.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 10 monclient: _renew_subs 2023-10-30T12:02:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb011e030 con 0x7f7fb00a6450 2023-10-30T12:02:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.653+0000 7f7fb5664640 1 librados: init done 2023-10-30T12:02:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.654+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7f98013470 con 0x7f7fb00a6450 2023-10-30T12:02:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.654+0000 7f7fb5664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.654+0000 7f7fb5664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.654+0000 7f7fb5664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.654+0000 7f7fa7fff640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7f90039320 0x7f7f9003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.654+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb004e160 con 0x7f7fb00a6450 2023-10-30T12:02:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 10425+0+0 (secure 0 0 0) 0x7f7f9804bf30 con 0x7f7fb00a6450 2023-10-30T12:02:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 10 client.52809.objecter ms_dispatch 0x7f7fb0000b90 osd_map(46..46 src has 1..46) v4 2023-10-30T12:02:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 3 client.52809.objecter handle_osd_map got epochs [46,46] > 0 2023-10-30T12:02:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 3 client.52809.objecter handle_osd_map decoding full epoch 46 2023-10-30T12:02:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7faeffd640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7f90039320 0x7f7f9003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7faeffd640 10 monclient: get_auth_request con 0x7f7f90039320 auth_method 0 2023-10-30T12:02:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 10 client.52809.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 10 monclient: _renew_subs 2023-10-30T12:02:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=47}) v3 -- 0x7f7f9004a2a0 con 0x7f7fb00a6450 2023-10-30T12:02:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 20 client.52809.objecter dump_active .. 0 homeless 2023-10-30T12:02:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7fa7fff640 10 client.52809.objecter ms_handle_connect 0x7f7f90039320 2023-10-30T12:02:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.655+0000 7f7faeffd640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7f90039320 0x7f7f9003b7e0 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0034000 tx=0x7f7fa002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.660+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f980232f0 con 0x7f7fb00a6450 2023-10-30T12:02:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.660+0000 7f7fa7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.660+0000 7f7fa7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.662+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 7 ==== osd_map(47..47 src has 1..47) v4 ==== 2089+0+0 (secure 0 0 0) 0x7f7f98011e20 con 0x7f7fb00a6450 2023-10-30T12:02:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.662+0000 7f7fa7fff640 10 client.52809.objecter ms_dispatch 0x7f7fb0000b90 osd_map(47..47 src has 1..47) v4 2023-10-30T12:02:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.662+0000 7f7fa7fff640 3 client.52809.objecter handle_osd_map got epochs [47,47] > 46 2023-10-30T12:02:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.662+0000 7f7fa7fff640 3 client.52809.objecter handle_osd_map decoding incremental epoch 47 2023-10-30T12:02:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.663+0000 7f7fa7fff640 10 client.52809.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.663+0000 7f7fa7fff640 10 monclient: _renew_subs 2023-10-30T12:02:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.663+0000 7f7fa7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.663+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=48}) v3 -- 0x7f7f900453b0 con 0x7f7fb00a6450 2023-10-30T12:02:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.663+0000 7f7fa7fff640 20 client.52809.objecter dump_active .. 0 homeless 2023-10-30T12:02:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.962+0000 7f7fb5664640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:02:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.962+0000 7f7fb5664640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:02:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.962+0000 7f7fb5664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.962+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7fb009fd40 con 0x7f7fb00a6450 2023-10-30T12:02:32.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.963+0000 7f7fa7fff640 1 -- 172.21.15.50:0/1899843191 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7f9801bba0 con 0x7f7fb00a6450 2023-10-30T12:02:32.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.963+0000 7f7fa7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:02:32.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.963+0000 7f7fa7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:32.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=7,logm=68} crc {auth=3961774502,config=3140514009,health=1113179269,kv=2334449644,logm=2141448690}) 2023-10-30T12:02:32.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=9,logm=66} crc {auth=3417758096,config=3140514009,health=2536777970,kv=2905431493,logm=678971192}) 2023-10-30T12:02:32.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=7,logm=68} crc {auth=3961774502,config=3140514009,health=1113179269,kv=2334449644,logm=2141448690}) 2023-10-30T12:02:32.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=12,kv=9,logm=66} crc {auth=1416741582,config=3140514009,health=2536777970,kv=631706467,logm=3610048566}) 2023-10-30T12:02:32.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554319762}) 2023-10-30T12:02:32.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711182875}) 2023-10-30T12:02:32.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554319762}) 2023-10-30T12:02:32.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.964+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2843425479}) 2023-10-30T12:02:32.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 10 librados: watch_flush enter 2023-10-30T12:02:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 10 librados: watch_flush exit 2023-10-30T12:02:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 20 client.52809.objecter shutdown clearing up homeless session... 2023-10-30T12:02:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 10 client.52809.objecter successfully canceled tick 2023-10-30T12:02:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7f90039320 msgr2=0x7f7f9003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7f90039320 0x7f7f9003b7e0 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0034000 tx=0x7f7fa002f000 comp rx=0 tx=0).stop 2023-10-30T12:02:32.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 10 monclient: shutdown 2023-10-30T12:02:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6450 msgr2=0x7f7fb009e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6450 0x7f7fb009e580 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800e9e0 tx=0x7f7f9800eea0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 shutdown_connections 2023-10-30T12:02:32.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7f90039320 0x7f7f9003b7e0 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7fb0135ad0 0x7f7fb009eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fb00a6450 0x7f7fb009e580 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 --2- 172.21.15.50:0/1899843191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fb00a6070 0x7f7fb009e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 >> 172.21.15.50:0/1899843191 conn(0x7f7fb00a81d0 msgr2=0x7f7fb0134220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 shutdown_connections 2023-10-30T12:02:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 -- 172.21.15.50:0/1899843191 wait complete. 2023-10-30T12:02:32.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.965+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.965+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1454624671}) 2023-10-30T12:02:32.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.965+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1171389435}) 2023-10-30T12:02:32.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.965+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:32.965+0000 7f7fb5664640 1 librados: shutdown 2023-10-30T12:02:32.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.965+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1454624671}) 2023-10-30T12:02:32.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.965+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252481878}) 2023-10-30T12:02:32.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.966+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.966+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762864342}) 2023-10-30T12:02:32.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.966+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2472596440}) 2023-10-30T12:02:32.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.966+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.966+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762864342}) 2023-10-30T12:02:32.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.966+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4152036710}) 2023-10-30T12:02:32.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.967+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.967+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181148851}) 2023-10-30T12:02:32.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.967+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985918700}) 2023-10-30T12:02:32.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.967+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.967+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181148851}) 2023-10-30T12:02:32.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.967+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264781645}) 2023-10-30T12:02:32.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.968+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.968+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900565091}) 2023-10-30T12:02:32.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.968+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328920115}) 2023-10-30T12:02:32.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.968+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.968+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900565091}) 2023-10-30T12:02:32.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.968+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4134634569}) 2023-10-30T12:02:32.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.969+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.969+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246133462}) 2023-10-30T12:02:32.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.969+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762173428}) 2023-10-30T12:02:32.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.969+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.969+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246133462}) 2023-10-30T12:02:32.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.969+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542204379}) 2023-10-30T12:02:32.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.970+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.970+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967653881}) 2023-10-30T12:02:32.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.970+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2731331024}) 2023-10-30T12:02:32.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.970+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.970+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967653881}) 2023-10-30T12:02:32.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.970+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=127153021}) 2023-10-30T12:02:32.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.972+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.972+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=880458545,mdsmap=3984425268,mgr=2578319740,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=725214738,mon_config_key=2725764394}) 2023-10-30T12:02:32.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.972+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3} crc {logm=880432619,mdsmap=2441318933,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3298444937,mon_config_key=772658687}) 2023-10-30T12:02:32.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.972+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.972+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=880458545,mdsmap=3984425268,mgr=2578319740,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=725214738,mon_config_key=2725764394}) 2023-10-30T12:02:32.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.972+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=3807437118,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=725214738,mon_config_key=1505500295}) 2023-10-30T12:02:32.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.973+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.973+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=8,osdmap=80} crc {mon_config_key=443934650,monmap=4204254258,osd_metadata=2479346626,osd_pg_creating=2326176211,osd_snap=948112930,osdmap=1589936191}) 2023-10-30T12:02:32.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.973+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8,osdmap=79} crc {mon_config_key=443934650,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=2326176211,osd_snap=264305055,osdmap=3458822588}) 2023-10-30T12:02:32.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.973+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:02:32.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.973+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=8,osdmap=80} crc {mon_config_key=443934650,monmap=4204254258,osd_metadata=2479346626,osd_pg_creating=2326176211,osd_snap=948112930,osdmap=1589936191}) 2023-10-30T12:02:32.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:02:32.973+0000 7fd43e0a5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7,osdmap=81} crc {mon_config_key=3477291362,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=2326176211,osd_snap=279022559,osdmap=1250986593}) 2023-10-30T12:02:32.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:02:33.986 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:02:33.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:02:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.096+0000 7fbbc5c40640 10 monclient: get_monmap_and_config 2023-10-30T12:02:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.096+0000 7fbbc5c40640 10 monclient: build_initial_monmap 2023-10-30T12:02:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.096+0000 7fbbc5c40640 10 monclient: monmap: 2023-10-30T12:02:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:34.097400+0000 2023-10-30T12:02:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:34.097400+0000 2023-10-30T12:02:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:34.100 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 1 Processor -- start 2023-10-30T12:02:34.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 1 -- start start 2023-10-30T12:02:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient: init 2023-10-30T12:02:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:02:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbbc00a4b60 0x7fbbc01326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc00a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbc0099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbbc0132be0 0x7fbbc00a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc0132be0 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00a4f40 con 0x7fbbc0099a40 2023-10-30T12:02:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc00a3d70 con 0x7fbbc0132be0 2023-10-30T12:02:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc00a4150 con 0x7fbbc00a4b60 2023-10-30T12:02:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbc5c40640 10 monclient(hunting): authenticate will time out at 999.280831s 2023-10-30T12:02:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbbffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbbc0132be0 0x7fbbc00a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbbeffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbbc0099a40 0x7fbbc00a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34384/0 (socket says 172.21.15.50:34384) 2023-10-30T12:02:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbbeffd640 1 -- 172.21.15.50:0/3771957779 learned_addr learned my addr 172.21.15.50:0/3771957779 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbbffff640 10 monclient(hunting): get_auth_request con 0x7fbbc0132be0 auth_method 0 2023-10-30T12:02:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbbffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.097+0000 7fbbbffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbf7fe640 1 --2- 172.21.15.50:0/3771957779 >> v2:172.21.15.118:3300/0 conn(0x7fbbc00a4b60 0x7fbbc01326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbf7fe640 10 monclient(hunting): get_auth_request con 0x7fbbc00a4b60 auth_method 0 2023-10-30T12:02:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 10 monclient(hunting): handle_auth_done global_id 54252 payload 274 2023-10-30T12:02:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 1 -- 172.21.15.50:0/3771957779 >> v2:172.21.15.118:3300/0 conn(0x7fbbc00a4b60 msgr2=0x7fbbc01326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 1 --2- 172.21.15.50:0/3771957779 >> v2:172.21.15.118:3300/0 conn(0x7fbbc00a4b60 0x7fbbc01326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 1 -- 172.21.15.50:0/3771957779 >> v1:172.21.15.50:6789/0 conn(0x7fbbc0099a40 legacy=0x7fbbc00a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 1 monclient: found mon.noname-a 2023-10-30T12:02:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbffff640 1 -- 172.21.15.50:0/3771957779 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc009a440 con 0x7fbbc0132be0 2023-10-30T12:02:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbe7fc640 1 -- 172.21.15.50:0/3771957779 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626398792 0 0) 0x7fbbc00a4f40 con 0x7fbbc0099a40 2023-10-30T12:02:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbf7fe640 1 --2- 172.21.15.50:0/3771957779 >> v2:172.21.15.118:3300/0 conn(0x7fbbc00a4b60 0x7fbbc01326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:02:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.098+0000 7fbbbe7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:02:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbffff640 1 --2- 172.21.15.50:0/3771957779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc0132be0 0x7fbbc00a2a60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00a3820 tx=0x7fbbb802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770a7c30469679df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 1 -- 172.21.15.50:0/3771957779 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb802cb60 con 0x7fbbc0132be0 2023-10-30T12:02:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: dump: 2023-10-30T12:02:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.116 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:02:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:02:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbc5c40640 5 monclient: authenticate success, global_id 54252 2023-10-30T12:02:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbc5c40640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 1 -- 172.21.15.50:0/3771957779 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb802ccc0 con 0x7fbbc0132be0 2023-10-30T12:02:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 1 -- 172.21.15.50:0/3771957779 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8034660 con 0x7fbbc0132be0 2023-10-30T12:02:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbbe7fc640 10 monclient: dump: 2023-10-30T12:02:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.120 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbc5c40640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.099+0000 7fbbc5c40640 4 set_mon_vals no callback set 2023-10-30T12:02:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 10 monclient: shutdown 2023-10-30T12:02:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 -- 172.21.15.50:0/3771957779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc0132be0 msgr2=0x7fbbc00a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/3771957779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc0132be0 0x7fbbc00a2a60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00a3820 tx=0x7fbbb802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 -- 172.21.15.50:0/3771957779 shutdown_connections 2023-10-30T12:02:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/3771957779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc0132be0 0x7fbbc00a2a60 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/3771957779 >> v2:172.21.15.118:3300/0 conn(0x7fbbc00a4b60 0x7fbbc01326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 -- 172.21.15.50:0/3771957779 >> 172.21.15.50:0/3771957779 conn(0x7fbbc00a8060 msgr2=0x7fbbc00a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 -- 172.21.15.50:0/3771957779 shutdown_connections 2023-10-30T12:02:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 1 -- 172.21.15.50:0/3771957779 wait complete. 2023-10-30T12:02:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 10 monclient: build_initial_monmap 2023-10-30T12:02:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.100+0000 7fbbc5c40640 10 monclient: monmap: 2023-10-30T12:02:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:34.124 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 librados: starting msgr at 2023-10-30T12:02:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 librados: starting objecter 2023-10-30T12:02:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 Processor -- start 2023-10-30T12:02:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 -- start start 2023-10-30T12:02:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 librados: setting wanted keys 2023-10-30T12:02:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 librados: calling monclient init 2023-10-30T12:02:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient: init 2023-10-30T12:02:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:02:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbbc00a4b60 0x7fbbc01c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc00a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbc0132be0 0x7fbbc012f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc0132be0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc011ec70 0x7fbbc01c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc011ec70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc011fd00 con 0x7fbbc011ec70 2023-10-30T12:02:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00a0760 con 0x7fbbc0132be0 2023-10-30T12:02:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0131380 con 0x7fbbc00a4b60 2023-10-30T12:02:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbc5c40640 10 monclient(hunting): authenticate will time out at 999.284914s 2023-10-30T12:02:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.101+0000 7fbbbf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbbc00a4b60 0x7fbbc01c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc011ec70 0x7fbbc01c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc011ec70 0x7fbbc01c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40300/0 (socket says 172.21.15.50:40300) 2023-10-30T12:02:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 -- 172.21.15.50:0/1375551272 learned_addr learned my addr 172.21.15.50:0/1375551272 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): get_auth_request con 0x7fbbc011ec70 auth_method 0 2023-10-30T12:02:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbf7fe640 10 monclient(hunting): get_auth_request con 0x7fbbc00a4b60 auth_method 0 2023-10-30T12:02:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbc0132be0 0x7fbbc012f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 10 monclient(hunting): get_auth_request con 0x7fbbc0132be0 auth_method 0 2023-10-30T12:02:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient(hunting): handle_auth_done global_id 54270 payload 824 2023-10-30T12:02:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 -- 172.21.15.50:0/1375551272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbbc00a4b60 msgr2=0x7fbbc01c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbbc00a4b60 0x7fbbc01c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 -- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbc0132be0 msgr2=0x7fbbc012f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbc0132be0 0x7fbbc012f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 monclient: found mon.noname-a 2023-10-30T12:02:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 -- 172.21.15.50:0/1375551272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8009a70 con 0x7fbbc011ec70 2023-10-30T12:02:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbffff640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc011ec70 0x7fbbc01c9e20 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7fbbb802c3c0 tx=0x7fbbb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 client.?.objecter ms_handle_connect 0x7fbbc011ec70 2023-10-30T12:02:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 1 -- 172.21.15.50:0/1375551272 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb803e610 con 0x7fbbc011ec70 2023-10-30T12:02:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: dump: 2023-10-30T12:02:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.137 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: _finish_auth 0 2023-10-30T12:02:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:02:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 1 -- 172.21.15.50:0/1375551272 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb803e770 con 0x7fbbc011ec70 2023-10-30T12:02:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 1 -- 172.21.15.50:0/1375551272 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb803eb20 con 0x7fbbc011ec70 2023-10-30T12:02:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbba7fff640 10 monclient: dump: 2023-10-30T12:02:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.140 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbc5c40640 5 monclient: authenticate success, global_id 54270 2023-10-30T12:02:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbc5c40640 10 monclient: _renew_subs 2023-10-30T12:02:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbc5c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbbdffb640 4 set_mon_vals no callback set 2023-10-30T12:02:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.102+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc011f460 con 0x7fbbc011ec70 2023-10-30T12:02:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.103+0000 7fbbc5c40640 10 client.54270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.103+0000 7fbbc5c40640 10 monclient: _renew_subs 2023-10-30T12:02:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.103+0000 7fbbc5c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.103+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc00a18d0 con 0x7fbbc011ec70 2023-10-30T12:02:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.103+0000 7fbbc5c40640 1 librados: init done 2023-10-30T12:02:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 1 -- 172.21.15.50:0/1375551272 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbbb803b050 con 0x7fbbc011ec70 2023-10-30T12:02:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbbc5c40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbbc5c40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbbc5c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc004e160 con 0x7fbbc011ec70 2023-10-30T12:02:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbbac039320 0x7fbbac03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 1 -- 172.21.15.50:0/1375551272 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 9383+0+0 (secure 0 0 0) 0x7fbbb806ddb0 con 0x7fbbc011ec70 2023-10-30T12:02:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 10 client.54270.objecter ms_dispatch 0x7fbbc0000b90 osd_map(48..48 src has 1..48) v4 2023-10-30T12:02:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 3 client.54270.objecter handle_osd_map got epochs [48,48] > 0 2023-10-30T12:02:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 3 client.54270.objecter handle_osd_map decoding full epoch 48 2023-10-30T12:02:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbbbf7fe640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbbac039320 0x7fbbac03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbbbf7fe640 10 monclient: get_auth_request con 0x7fbbac039320 auth_method 0 2023-10-30T12:02:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 20 client.54270.objecter dump_active .. 0 homeless 2023-10-30T12:02:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbbbf7fe640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbbac039320 0x7fbbac03b7e0 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0006e80 tx=0x7fbbb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.104+0000 7fbba7fff640 10 client.54270.objecter ms_handle_connect 0x7fbbac039320 2023-10-30T12:02:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.109+0000 7fbba7fff640 1 -- 172.21.15.50:0/1375551272 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbbb8072070 con 0x7fbbc011ec70 2023-10-30T12:02:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.109+0000 7fbba7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.109+0000 7fbba7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.376+0000 7fbbc5c40640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:02:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.376+0000 7fbbc5c40640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.376+0000 7fbbc5c40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.376+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbbc011ff60 con 0x7fbbc011ec70 2023-10-30T12:02:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.376+0000 7fbba7fff640 1 -- 172.21.15.50:0/1375551272 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fbbb804d070 con 0x7fbbc011ec70 2023-10-30T12:02:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.376+0000 7fbba7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.376+0000 7fbba7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:34.380 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:02:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 10 librados: watch_flush enter 2023-10-30T12:02:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 10 librados: watch_flush exit 2023-10-30T12:02:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 20 client.54270.objecter shutdown clearing up homeless session... 2023-10-30T12:02:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 10 client.54270.objecter successfully canceled tick 2023-10-30T12:02:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbbac039320 msgr2=0x7fbbac03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbbac039320 0x7fbbac03b7e0 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0006e80 tx=0x7fbbb0008040 comp rx=0 tx=0).stop 2023-10-30T12:02:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 10 monclient: shutdown 2023-10-30T12:02:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc011ec70 msgr2=0x7fbbc01c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc011ec70 0x7fbbc01c9e20 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7fbbb802c3c0 tx=0x7fbbb8004a40 comp rx=0 tx=0).stop 2023-10-30T12:02:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 shutdown_connections 2023-10-30T12:02:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbbac039320 0x7fbbac03b7e0 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbc011ec70 0x7fbbc01c9e20 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbc0132be0 0x7fbbc012f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 --2- 172.21.15.50:0/1375551272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbbc00a4b60 0x7fbbc01c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 >> 172.21.15.50:0/1375551272 conn(0x7fbbc00a8060 msgr2=0x7fbbc0130a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 shutdown_connections 2023-10-30T12:02:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 -- 172.21.15.50:0/1375551272 wait complete. 2023-10-30T12:02:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.379+0000 7fbbc5c40640 1 librados: shutdown 2023-10-30T12:02:34.392 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:02:34.392 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:02:34.393 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:02:34.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:02:34.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.502+0000 7facd82b4640 10 monclient: get_monmap_and_config 2023-10-30T12:02:34.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.502+0000 7facd82b4640 10 monclient: build_initial_monmap 2023-10-30T12:02:34.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.502+0000 7facd82b4640 10 monclient: monmap: 2023-10-30T12:02:34.504 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:34.504 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.505 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:34.503483+0000 2023-10-30T12:02:34.505 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:34.503483+0000 2023-10-30T12:02:34.505 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:34.505 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.505 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:34.506 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:34.506 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:34.506 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:34.506 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:34.506 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.502+0000 7facd82b4640 1 Processor -- start 2023-10-30T12:02:34.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 1 -- start start 2023-10-30T12:02:34.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient: init 2023-10-30T12:02:34.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:34.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-10-30T12:02:34.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7facd00a2ae0 0x7facd00a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient(hunting): picked mon.noname-a con 0x7facd00a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7facd00a2ec0 0x7facd01356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient(hunting): picked mon.noname-c con 0x7facd00a2ec0 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facd0135c20 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd00a4190 con 0x7facd0135c20 2023-10-30T12:02:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd00a2990 con 0x7facd00a2ae0 2023-10-30T12:02:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd00a1630 con 0x7facd00a2ec0 2023-10-30T12:02:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd82b4640 10 monclient(hunting): authenticate will time out at 999.686696s 2023-10-30T12:02:34.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7facd00a2ae0 0x7facd00a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7facd00a2ae0 0x7facd00a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40316/0 (socket says 172.21.15.50:40316) 2023-10-30T12:02:34.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 1 -- 172.21.15.50:0/3421043995 learned_addr learned my addr 172.21.15.50:0/3421043995 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:34.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 10 monclient(hunting): get_auth_request con 0x7facd00a2ae0 auth_method 0 2023-10-30T12:02:34.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6851640 1 --1- 172.21.15.50:0/3421043995 >> v1:172.21.15.50:6789/0 conn(0x7facd0135c20 0x7facd013ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34398/0 (socket says 172.21.15.50:34398) 2023-10-30T12:02:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd584f640 1 --2- 172.21.15.50:0/3421043995 >> v2:172.21.15.118:3300/0 conn(0x7facd00a2ec0 0x7facd01356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd584f640 10 monclient(hunting): get_auth_request con 0x7facd00a2ec0 auth_method 0 2023-10-30T12:02:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd584f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd584f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd584f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd584f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.503+0000 7facd584f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd584f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 10 monclient(hunting): handle_auth_done global_id 54801 payload 274 2023-10-30T12:02:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 1 -- 172.21.15.50:0/3421043995 >> v2:172.21.15.118:3300/0 conn(0x7facd00a2ec0 msgr2=0x7facd01356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 1 --2- 172.21.15.50:0/3421043995 >> v2:172.21.15.118:3300/0 conn(0x7facd00a2ec0 0x7facd01356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 1 -- 172.21.15.50:0/3421043995 >> v1:172.21.15.50:6789/0 conn(0x7facd0135c20 legacy=0x7facd013ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 10 monclient: _finish_hunting 0 2023-10-30T12:02:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 1 monclient: found mon.noname-a 2023-10-30T12:02:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 1 -- 172.21.15.50:0/3421043995 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd009a2e0 con 0x7facd00a2ae0 2023-10-30T12:02:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd6050640 1 --2- 172.21.15.50:0/3421043995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ae0 0x7facd00a2450 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7facd00a4c80 tx=0x7faccc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a459464eb0cb33f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd504e640 1 -- 172.21.15.50:0/3421043995 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0317e0 con 0x7facd00a2ae0 2023-10-30T12:02:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd504e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd504e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd504e640 10 monclient: dump: 2023-10-30T12:02:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.524 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.524 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.524 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.524 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.524 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.525 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.525 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.525 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.504+0000 7facd504e640 10 monclient: _finish_auth 0 2023-10-30T12:02:34.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 10 monclient: _check_auth_tickets 2023-10-30T12:02:34.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:34.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 5 monclient: authenticate success, global_id 54801 2023-10-30T12:02:34.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:34.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 1 -- 172.21.15.50:0/3421043995 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc031940 con 0x7facd00a2ae0 2023-10-30T12:02:34.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:34.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:34.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 4 set_mon_vals no callback set 2023-10-30T12:02:34.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 1 -- 172.21.15.50:0/3421043995 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc031cd0 con 0x7facd00a2ae0 2023-10-30T12:02:34.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:34.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd504e640 10 monclient: dump: 2023-10-30T12:02:34.527 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.528 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.528 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.528 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.528 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.528 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.528 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.529 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.529 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.529 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 10 monclient: shutdown 2023-10-30T12:02:34.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:34.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 -- 172.21.15.50:0/3421043995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ae0 msgr2=0x7facd00a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 --2- 172.21.15.50:0/3421043995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ae0 0x7facd00a2450 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7facd00a4c80 tx=0x7faccc02b880 comp rx=0 tx=0).stop 2023-10-30T12:02:34.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 -- 172.21.15.50:0/3421043995 shutdown_connections 2023-10-30T12:02:34.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 --2- 172.21.15.50:0/3421043995 >> v2:172.21.15.118:3300/0 conn(0x7facd00a2ec0 0x7facd01356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 --2- 172.21.15.50:0/3421043995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ae0 0x7facd00a2450 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 -- 172.21.15.50:0/3421043995 >> 172.21.15.50:0/3421043995 conn(0x7facd00a80b0 msgr2=0x7facd00a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:34.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 -- 172.21.15.50:0/3421043995 shutdown_connections 2023-10-30T12:02:34.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.505+0000 7facd82b4640 1 -- 172.21.15.50:0/3421043995 wait complete. 2023-10-30T12:02:34.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 10 monclient: build_initial_monmap 2023-10-30T12:02:34.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 10 monclient: monmap: 2023-10-30T12:02:34.531 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:34.531 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:34.532 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:34.532 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:34.532 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:34.532 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.532 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:34.532 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:34.533 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:34.533 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 1 librados: starting msgr at 2023-10-30T12:02:34.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 1 librados: starting objecter 2023-10-30T12:02:34.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 1 Processor -- start 2023-10-30T12:02:34.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 1 -- start start 2023-10-30T12:02:34.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 1 librados: setting wanted keys 2023-10-30T12:02:34.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 1 librados: calling monclient init 2023-10-30T12:02:34.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.506+0000 7facd82b4640 10 monclient: init 2023-10-30T12:02:34.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:34.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:02:34.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd00a2ae0 0x7facd01ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): picked mon.noname-b con 0x7facd00a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:34.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ec0 0x7facd01ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): picked mon.noname-a con 0x7facd00a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:34.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7facd0135c20 0x7facd011f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): picked mon.noname-c con 0x7facd0135c20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:34.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd0122220 con 0x7facd00a2ec0 2023-10-30T12:02:34.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd00a1790 con 0x7facd00a2ae0 2023-10-30T12:02:34.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd0135530 con 0x7facd0135c20 2023-10-30T12:02:34.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:34.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient(hunting): authenticate will time out at 999.690136s 2023-10-30T12:02:34.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ec0 0x7facd01ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ec0 0x7facd01ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40324/0 (socket says 172.21.15.50:40324) 2023-10-30T12:02:34.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7facd0135c20 0x7facd011f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 1 -- 172.21.15.50:0/3332307737 learned_addr learned my addr 172.21.15.50:0/3332307737 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:34.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 10 monclient(hunting): get_auth_request con 0x7facd00a2ec0 auth_method 0 2023-10-30T12:02:34.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): get_auth_request con 0x7facd0135c20 auth_method 0 2023-10-30T12:02:34.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd00a2ae0 0x7facd01ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 10 monclient(hunting): get_auth_request con 0x7facd00a2ae0 auth_method 0 2023-10-30T12:02:34.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd584f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient(hunting): handle_auth_done global_id 42530 payload 824 2023-10-30T12:02:34.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd00a2ae0 msgr2=0x7facd01ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7facd00a2ae0 0x7facd01ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ec0 msgr2=0x7facd01ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd00a2ec0 0x7facd01ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient: _finish_hunting 0 2023-10-30T12:02:34.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 monclient: found mon.noname-c 2023-10-30T12:02:34.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:34.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:34.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 -- 172.21.15.50:0/3332307737 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0094d0 con 0x7facd0135c20 2023-10-30T12:02:34.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd6851640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7facd0135c20 0x7facd011f9c0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7facc800c6c0 tx=0x7facc800cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 client.?.objecter ms_handle_connect 0x7facd0135c20 2023-10-30T12:02:34.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:34.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8004d60 con 0x7facd0135c20 2023-10-30T12:02:34.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:34.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: dump: 2023-10-30T12:02:34.545 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.545 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.545 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.545 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.545 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.546 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.546 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.546 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.546 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.546 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:02:34.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:34.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:34.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc800a640 con 0x7facd0135c20 2023-10-30T12:02:34.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 5 monclient: authenticate success, global_id 42530 2023-10-30T12:02:34.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient: _renew_subs 2023-10-30T12:02:34.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:34.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd01239b0 con 0x7facd0135c20 2023-10-30T12:02:34.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:34.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc800a9f0 con 0x7facd0135c20 2023-10-30T12:02:34.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:34.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facc6ffd640 10 monclient: dump: 2023-10-30T12:02:34.549 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.549 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.549 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.549 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.549 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.550 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.550 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.550 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.550 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.550 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd484d640 4 set_mon_vals no callback set 2023-10-30T12:02:34.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 client.42530.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:34.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient: _renew_subs 2023-10-30T12:02:34.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.507+0000 7facd82b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:34.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.508+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd01243c0 con 0x7facd0135c20 2023-10-30T12:02:34.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.508+0000 7facd82b4640 1 librados: init done 2023-10-30T12:02:34.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.508+0000 7facd82b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:34.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.508+0000 7facd82b4640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd011ff00 0x7facd01202e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.508+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7facc800ab50 con 0x7facd0135c20 2023-10-30T12:02:34.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.508+0000 7facd82b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.508+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd0073ff0 con 0x7facd011ff00 2023-10-30T12:02:34.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd011ff00 0x7facd01202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient: get_auth_request con 0x7facd011ff00 auth_method 0 2023-10-30T12:02:34.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facb40392f0 0x7facb403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 9383+0+0 (secure 0 0 0) 0x7facc804aeb0 con 0x7facd0135c20 2023-10-30T12:02:34.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 10 client.42530.objecter ms_dispatch 0x7facd0000b90 osd_map(48..48 src has 1..48) v4 2023-10-30T12:02:34.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 3 client.42530.objecter handle_osd_map got epochs [48,48] > 0 2023-10-30T12:02:34.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 3 client.42530.objecter handle_osd_map decoding full epoch 48 2023-10-30T12:02:34.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 10 monclient(hunting): handle_auth_done global_id 54828 payload 824 2023-10-30T12:02:34.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 1 -- 172.21.15.50:0/3332307737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7facd009a5c0 con 0x7facd011ff00 2023-10-30T12:02:34.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 20 client.42530.objecter dump_active .. 0 homeless 2023-10-30T12:02:34.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd584f640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facb40392f0 0x7facb403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd584f640 10 monclient: get_auth_request con 0x7facb40392f0 auth_method 0 2023-10-30T12:02:34.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 10 client.42530.objecter ms_handle_connect 0x7facd011ff00 2023-10-30T12:02:34.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 10 client.42530.objecter resend_mon_ops 2023-10-30T12:02:34.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facd6050640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd011ff00 0x7facd01202e0 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7facd00a54d0 tx=0x7faccc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc049070 con 0x7facd011ff00 2023-10-30T12:02:34.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:34.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.509+0000 7facc6ffd640 10 monclient: dump: 2023-10-30T12:02:34.557 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.557 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.557 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.558 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.558 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.558 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.558 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.558 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.559 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.559 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 10 client.42530.objecter ms_handle_connect 0x7facb40392f0 2023-10-30T12:02:34.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facd584f640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facb40392f0 0x7facb403b7b0 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7facd01211e0 tx=0x7facc0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7faccc032740 con 0x7facd011ff00 2023-10-30T12:02:34.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:34.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:34.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd011ff00 msgr2=0x7facd01202e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd011ff00 0x7facd01202e0 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7facd00a54d0 tx=0x7faccc004730 comp rx=0 tx=0).stop 2023-10-30T12:02:34.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd011ff00 msgr2=0x7facd01202e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:34.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.510+0000 7facc6ffd640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd011ff00 0x7facd01202e0 secure :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7facd00a54d0 tx=0x7faccc004730 comp rx=0 tx=0).stop 2023-10-30T12:02:34.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.516+0000 7facd82b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:34.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.516+0000 7facd82b4640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd0120820 0x7facd009dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.516+0000 7facd82b4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.516+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd0073a20 con 0x7facd0120820 2023-10-30T12:02:34.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.516+0000 7facd6050640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd0120820 0x7facd009dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient: get_auth_request con 0x7facd0120820 auth_method 0 2023-10-30T12:02:34.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 10 monclient(hunting): handle_auth_done global_id 54846 payload 824 2023-10-30T12:02:34.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 1 -- 172.21.15.50:0/3332307737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7facd006b060 con 0x7facd0120820 2023-10-30T12:02:34.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 10 client.42530.objecter ms_handle_connect 0x7facd0120820 2023-10-30T12:02:34.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facd6050640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd0120820 0x7facd009dde0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7faccc02b8c0 tx=0x7faccc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 10 client.42530.objecter resend_mon_ops 2023-10-30T12:02:34.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc04e460 con 0x7facd0120820 2023-10-30T12:02:34.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:34.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 10 monclient: dump: 2023-10-30T12:02:34.566 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.566 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.566 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.566 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.567 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.567 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.567 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.567 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.567 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.568 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7faccc04e5c0 con 0x7facd0120820 2023-10-30T12:02:34.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:34.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:34.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd0120820 msgr2=0x7facd009dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd0120820 0x7facd009dde0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7faccc02b8c0 tx=0x7faccc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd0120820 msgr2=0x7facd009dde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:34.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.517+0000 7facc6ffd640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7facd0120820 0x7facd009dde0 secure :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7faccc02b8c0 tx=0x7faccc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.569 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:02:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 20, 2023-10-30T12:02:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:02:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:02:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:34.595 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.595 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.595 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:34.595 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:34.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.596 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:34.596 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:34.596 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:34.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.596 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:34.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.597 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:34.597 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:34.597 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:02:34.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.598 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.598 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:34.598 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:34.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 10 librados: watch_flush enter 2023-10-30T12:02:34.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 10 librados: watch_flush exit 2023-10-30T12:02:34.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 20 client.42530.objecter shutdown clearing up homeless session... 2023-10-30T12:02:34.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 10 client.42530.objecter successfully canceled tick 2023-10-30T12:02:34.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facb40392f0 msgr2=0x7facb403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facb40392f0 0x7facb403b7b0 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7facd01211e0 tx=0x7facc0009190 comp rx=0 tx=0).stop 2023-10-30T12:02:34.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 10 monclient: shutdown 2023-10-30T12:02:34.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:34.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7facd0135c20 msgr2=0x7facd011f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7facd0135c20 0x7facd011f9c0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7facc800c6c0 tx=0x7facc800cb80 comp rx=0 tx=0).stop 2023-10-30T12:02:34.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd6851640 1 -- 172.21.15.50:0/3332307737 reap_dead start 2023-10-30T12:02:34.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 shutdown_connections 2023-10-30T12:02:34.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 --2- 172.21.15.50:0/3332307737 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7facd0135c20 0x7facd011f9c0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 >> 172.21.15.50:0/3332307737 conn(0x7facd00a80b0 msgr2=0x7facd00a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:34.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 shutdown_connections 2023-10-30T12:02:34.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 -- 172.21.15.50:0/3332307737 wait complete. 2023-10-30T12:02:34.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.518+0000 7facd82b4640 1 librados: shutdown 2023-10-30T12:02:34.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:02:34.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:02:34.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:02:34.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.728+0000 7f33d84c9640 10 monclient: get_monmap_and_config 2023-10-30T12:02:34.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.728+0000 7f33d84c9640 10 monclient: build_initial_monmap 2023-10-30T12:02:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.728+0000 7f33d84c9640 10 monclient: monmap: 2023-10-30T12:02:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:34.729935+0000 2023-10-30T12:02:34.731 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:34.729935+0000 2023-10-30T12:02:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:34.733 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:34.733 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:34.733 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.729+0000 7f33d84c9640 1 Processor -- start 2023-10-30T12:02:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.729+0000 7f33d84c9640 1 -- start start 2023-10-30T12:02:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.729+0000 7f33d84c9640 10 monclient: init 2023-10-30T12:02:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-10-30T12:02:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33d00a8010 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d013aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient(hunting): picked mon.noname-b con 0x7f33d0099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f33d0132da0 0x7f33d013cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient(hunting): picked mon.noname-a con 0x7f33d0132da0 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d009a210 con 0x7f33d00a8010 2023-10-30T12:02:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d00a7190 con 0x7f33d0132da0 2023-10-30T12:02:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d00a83f0 con 0x7f33d0099a40 2023-10-30T12:02:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d84c9640 10 monclient(hunting): authenticate will time out at 999.913344s 2023-10-30T12:02:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d013aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6265640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f33d00a8010 0x7f33d0132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:36934/0 (socket says 172.21.15.50:36934) 2023-10-30T12:02:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d013aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38354/0 (socket says 172.21.15.50:38354) 2023-10-30T12:02:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6265640 1 -- 172.21.15.50:0/2459149345 learned_addr learned my addr 172.21.15.50:0/2459149345 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): get_auth_request con 0x7f33d0099a40 auth_method 0 2023-10-30T12:02:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 1 --2- 172.21.15.50:0/2459149345 >> v2:172.21.15.50:3300/0 conn(0x7f33d0132da0 0x7f33d013cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 10 monclient(hunting): get_auth_request con 0x7f33d0132da0 auth_method 0 2023-10-30T12:02:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d6a66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient(hunting): handle_auth_done global_id 42707 payload 274 2023-10-30T12:02:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 1 -- 172.21.15.50:0/2459149345 >> v2:172.21.15.50:3300/0 conn(0x7f33d0132da0 msgr2=0x7f33d013cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 1 --2- 172.21.15.50:0/2459149345 >> v2:172.21.15.50:3300/0 conn(0x7f33d0132da0 0x7f33d013cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 1 -- 172.21.15.50:0/2459149345 >> v1:172.21.15.118:6789/0 conn(0x7f33d00a8010 legacy=0x7f33d0132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient: _finish_hunting 0 2023-10-30T12:02:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 1 monclient: found mon.noname-b 2023-10-30T12:02:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:02:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.730+0000 7f33d5a64640 1 -- 172.21.15.50:0/2459149345 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d00a7600 con 0x7f33d0099a40 2023-10-30T12:02:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5a64640 1 --2- 172.21.15.50:0/2459149345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d013aa10 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f33c00097a0 tx=0x7f33c002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e5c0cde4133afbd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 1 -- 172.21.15.50:0/2459149345 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c002f050 con 0x7f33d0099a40 2023-10-30T12:02:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:02:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: dump: 2023-10-30T12:02:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.753 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: _finish_auth 0 2023-10-30T12:02:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: _check_auth_tickets 2023-10-30T12:02:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 1 -- 172.21.15.50:0/2459149345 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c0031700 con 0x7f33d0099a40 2023-10-30T12:02:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 1 -- 172.21.15.50:0/2459149345 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0031a90 con 0x7f33d0099a40 2023-10-30T12:02:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d84c9640 5 monclient: authenticate success, global_id 42707 2023-10-30T12:02:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d84c9640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d84c9640 4 set_mon_vals no callback set 2023-10-30T12:02:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.731+0000 7f33d5263640 10 monclient: dump: 2023-10-30T12:02:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.757 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 10 monclient: shutdown 2023-10-30T12:02:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 -- 172.21.15.50:0/2459149345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 msgr2=0x7f33d013aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 --2- 172.21.15.50:0/2459149345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d013aa10 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f33c00097a0 tx=0x7f33c002b7a0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 -- 172.21.15.50:0/2459149345 shutdown_connections 2023-10-30T12:02:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 --2- 172.21.15.50:0/2459149345 >> v2:172.21.15.50:3300/0 conn(0x7f33d0132da0 0x7f33d013cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 --2- 172.21.15.50:0/2459149345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d013aa10 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 -- 172.21.15.50:0/2459149345 >> 172.21.15.50:0/2459149345 conn(0x7f33d00a4a10 msgr2=0x7f33d00a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 -- 172.21.15.50:0/2459149345 shutdown_connections 2023-10-30T12:02:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.732+0000 7f33d84c9640 1 -- 172.21.15.50:0/2459149345 wait complete. 2023-10-30T12:02:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 10 monclient: build_initial_monmap 2023-10-30T12:02:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 10 monclient: monmap: 2023-10-30T12:02:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:34.761 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 1 librados: starting msgr at 2023-10-30T12:02:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 1 librados: starting objecter 2023-10-30T12:02:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 1 Processor -- start 2023-10-30T12:02:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 1 -- start start 2023-10-30T12:02:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 1 librados: setting wanted keys 2023-10-30T12:02:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 1 librados: calling monclient init 2023-10-30T12:02:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.733+0000 7f33d84c9640 10 monclient: init 2023-10-30T12:02:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:02:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33d00a8010 0x7f33d01ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): picked mon.noname-a con 0x7f33d00a8010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0132da0 0x7f33d01caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): picked mon.noname-c con 0x7f33d0132da0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33d0135d00 0x7f33d01361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): picked mon.noname-b con 0x7f33d0135d00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d0139d70 con 0x7f33d00a8010 2023-10-30T12:02:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d01385e0 con 0x7f33d0135d00 2023-10-30T12:02:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d0138360 con 0x7f33d0132da0 2023-10-30T12:02:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient(hunting): authenticate will time out at 999.917151s 2023-10-30T12:02:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33d00a8010 0x7f33d01ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33d00a8010 0x7f33d01ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40378/0 (socket says 172.21.15.50:40378) 2023-10-30T12:02:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 -- 172.21.15.50:0/3384338480 learned_addr learned my addr 172.21.15.50:0/3384338480 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): get_auth_request con 0x7f33d00a8010 auth_method 0 2023-10-30T12:02:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0132da0 0x7f33d01caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 10 monclient(hunting): get_auth_request con 0x7f33d0132da0 auth_method 0 2023-10-30T12:02:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient(hunting): handle_auth_done global_id 55044 payload 824 2023-10-30T12:02:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0132da0 msgr2=0x7f33d01caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0132da0 0x7f33d01caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33d0135d00 msgr2=0x7f33d01361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33d0135d00 0x7f33d01361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient: _finish_hunting 0 2023-10-30T12:02:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 monclient: found mon.noname-a 2023-10-30T12:02:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 -- 172.21.15.50:0/3384338480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009400 con 0x7f33d00a8010 2023-10-30T12:02:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d6265640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33d00a8010 0x7f33d01ca990 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00cba0 tx=0x7f33cc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 client.?.objecter ms_handle_connect 0x7f33d00a8010 2023-10-30T12:02:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00ad40 con 0x7f33d00a8010 2023-10-30T12:02:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: dump: 2023-10-30T12:02:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.775 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:02:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 5 monclient: authenticate success, global_id 55044 2023-10-30T12:02:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc012400 con 0x7f33d00a8010 2023-10-30T12:02:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient: _renew_subs 2023-10-30T12:02:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d00a18d0 con 0x7f33d00a8010 2023-10-30T12:02:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00aea0 con 0x7f33d00a8010 2023-10-30T12:02:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33c6ffd640 10 monclient: dump: 2023-10-30T12:02:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.780 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d4a62640 4 set_mon_vals no callback set 2023-10-30T12:02:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 client.55044.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient: _renew_subs 2023-10-30T12:02:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d0139a90 con 0x7f33d00a8010 2023-10-30T12:02:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.734+0000 7f33d84c9640 1 librados: init done 2023-10-30T12:02:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d84c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d84c9640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d84c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d0073fd0 con 0x7f33d0099a40 2023-10-30T12:02:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f33cc007070 con 0x7f33d00a8010 2023-10-30T12:02:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d5a64640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d5a64640 10 monclient: get_auth_request con 0x7f33d0099a40 auth_method 0 2023-10-30T12:02:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d5a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d5a64640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d5a64640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.735+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f33b40392f0 0x7f33b403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 9366+0+0 (secure 0 0 0) 0x7f33cc04b860 con 0x7f33d00a8010 2023-10-30T12:02:34.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 client.55044.objecter ms_dispatch 0x7f33d0000b90 osd_map(49..49 src has 1..49) v4 2023-10-30T12:02:34.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33d5a64640 10 monclient(hunting): handle_auth_done global_id 42719 payload 824 2023-10-30T12:02:34.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 3 client.55044.objecter handle_osd_map got epochs [49,49] > 0 2023-10-30T12:02:34.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 3 client.55044.objecter handle_osd_map decoding full epoch 49 2023-10-30T12:02:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33d5a64640 1 -- 172.21.15.50:0/3384338480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33d009aa00 con 0x7f33d0099a40 2023-10-30T12:02:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33d6a66640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f33b40392f0 0x7f33b403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33d6a66640 10 monclient: get_auth_request con 0x7f33b40392f0 auth_method 0 2023-10-30T12:02:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 20 client.55044.objecter dump_active .. 0 homeless 2023-10-30T12:02:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 client.55044.objecter ms_handle_connect 0x7f33d0099a40 2023-10-30T12:02:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33d5a64640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d002b430 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f33d00a8640 tx=0x7f33c00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 client.55044.objecter resend_mon_ops 2023-10-30T12:02:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0033b30 con 0x7f33d0099a40 2023-10-30T12:02:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 monclient: dump: 2023-10-30T12:02:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.788 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 client.55044.objecter ms_handle_connect 0x7f33b40392f0 2023-10-30T12:02:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33d6a66640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f33b40392f0 0x7f33b403b7b0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f33c8006a90 tx=0x7f33c8006a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f33c0041070 con 0x7f33d0099a40 2023-10-30T12:02:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 msgr2=0x7f33d002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d002b430 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f33d00a8640 tx=0x7f33c00046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 msgr2=0x7f33d002b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.736+0000 7f33c6ffd640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0099a40 0x7f33d002b430 secure :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f33d00a8640 tx=0x7f33c00046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d84c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d84c9640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0136eb0 0x7f33d002b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d84c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d009a720 con 0x7f33d0136eb0 2023-10-30T12:02:34.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0136eb0 0x7f33d002b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient: get_auth_request con 0x7f33d0136eb0 auth_method 0 2023-10-30T12:02:34.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 10 monclient(hunting): handle_auth_done global_id 42722 payload 824 2023-10-30T12:02:34.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 1 -- 172.21.15.50:0/3384338480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f33d013d4a0 con 0x7f33d0136eb0 2023-10-30T12:02:34.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33d5a64640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0136eb0 0x7f33d002b970 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f33d00a25f0 tx=0x7f33c0031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 10 client.55044.objecter ms_handle_connect 0x7f33d0136eb0 2023-10-30T12:02:34.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 10 client.55044.objecter resend_mon_ops 2023-10-30T12:02:34.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0033ea0 con 0x7f33d0136eb0 2023-10-30T12:02:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 10 monclient: dump: 2023-10-30T12:02:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.797 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.797 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.797 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.797 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.797 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.797 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.798 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.798 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f33c0052670 con 0x7f33d0136eb0 2023-10-30T12:02:34.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:34.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:34.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0136eb0 msgr2=0x7f33d002b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0136eb0 0x7f33d002b970 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f33d00a25f0 tx=0x7f33c0031000 comp rx=0 tx=0).stop 2023-10-30T12:02:34.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0136eb0 msgr2=0x7f33d002b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:34.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.744+0000 7f33c6ffd640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33d0136eb0 0x7f33d002b970 secure :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f33d00a25f0 tx=0x7f33c0031000 comp rx=0 tx=0).stop 2023-10-30T12:02:34.799 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:02:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 20, 2023-10-30T12:02:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:02:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:02:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:34.824 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:34.824 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:34.825 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:34.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.825 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:34.826 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:34.826 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.827 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:34.827 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:34.828 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.828 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.828 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.829 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:34.829 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:34.830 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:34.830 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:34.831 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:34.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.832 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:34.832 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:34.832 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:34.833 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:34.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.834 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:34.834 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:34.834 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.835 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:34.836 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:34.836 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.836 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.837 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.838 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:34.838 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:34.838 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:34.839 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:34.839 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:34.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.840 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:34.840 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:34.841 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:34.841 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:34.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.842 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:34.842 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:34.843 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.844 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:34.844 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:34.844 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:34.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.845 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.846 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:34.846 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:34.847 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:34.847 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:34.847 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:34.848 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.848 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.849 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:34.849 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:34.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.850 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:34.850 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:34.850 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:02:34.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.852 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:34.852 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:34.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:34.854 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:34.854 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:34.855 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:02:34.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:34.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:34.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:34.856 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:34.856 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 10 librados: watch_flush enter 2023-10-30T12:02:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 10 librados: watch_flush exit 2023-10-30T12:02:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 20 client.55044.objecter shutdown clearing up homeless session... 2023-10-30T12:02:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 10 client.55044.objecter successfully canceled tick 2023-10-30T12:02:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f33b40392f0 msgr2=0x7f33b403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f33b40392f0 0x7f33b403b7b0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f33c8006a90 tx=0x7f33c8006a20 comp rx=0 tx=0).stop 2023-10-30T12:02:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 10 monclient: shutdown 2023-10-30T12:02:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33d00a8010 msgr2=0x7f33d01ca990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 --2- 172.21.15.50:0/3384338480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33d00a8010 0x7f33d01ca990 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00cba0 tx=0x7f33cc00a430 comp rx=0 tx=0).stop 2023-10-30T12:02:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d6a66640 1 -- 172.21.15.50:0/3384338480 reap_dead start 2023-10-30T12:02:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d6a66640 1 -- 172.21.15.50:0/3384338480 reap_dead start 2023-10-30T12:02:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 shutdown_connections 2023-10-30T12:02:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 >> 172.21.15.50:0/3384338480 conn(0x7f33d00a4a10 msgr2=0x7f33d0066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 shutdown_connections 2023-10-30T12:02:34.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 -- 172.21.15.50:0/3384338480 wait complete. 2023-10-30T12:02:34.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.745+0000 7f33d84c9640 1 librados: shutdown 2023-10-30T12:02:34.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:02:34.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:02:34.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:02:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.970+0000 7f13a587c640 10 monclient: get_monmap_and_config 2023-10-30T12:02:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.970+0000 7f13a587c640 10 monclient: build_initial_monmap 2023-10-30T12:02:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.970+0000 7f13a587c640 10 monclient: monmap: 2023-10-30T12:02:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:34.971727+0000 2023-10-30T12:02:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:34.971727+0000 2023-10-30T12:02:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:34.975 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 1 Processor -- start 2023-10-30T12:02:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 1 -- start start 2023-10-30T12:02:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient: init 2023-10-30T12:02:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T12:02:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f13a00a8010 0x7f13a0132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient(hunting): picked mon.noname-c con 0x7f13a00a8010 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f13a0099a40 0x7f13a0132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient(hunting): picked mon.noname-a con 0x7f13a0099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13a01356d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a00a7190 con 0x7f13a01356d0 2023-10-30T12:02:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a0133110 con 0x7f13a0099a40 2023-10-30T12:02:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a00a83f0 con 0x7f13a00a8010 2023-10-30T12:02:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.971+0000 7f13a587c640 10 monclient(hunting): authenticate will time out at 1000.154976s 2023-10-30T12:02:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f13a0099a40 0x7f13a0132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f13a01356d0 0x7f13a0135b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34408/0 (socket says 172.21.15.50:34408) 2023-10-30T12:02:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139f7fe640 1 -- 172.21.15.50:0/3343291152 learned_addr learned my addr 172.21.15.50:0/3343291152 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): get_auth_request con 0x7f13a0099a40 auth_method 0 2023-10-30T12:02:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 1 --2- 172.21.15.50:0/3343291152 >> v2:172.21.15.118:3300/0 conn(0x7f13a00a8010 0x7f13a0132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 10 monclient(hunting): get_auth_request con 0x7f13a00a8010 auth_method 0 2023-10-30T12:02:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient(hunting): handle_auth_done global_id 55272 payload 274 2023-10-30T12:02:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 1 -- 172.21.15.50:0/3343291152 >> v2:172.21.15.118:3300/0 conn(0x7f13a00a8010 msgr2=0x7f13a0132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 1 --2- 172.21.15.50:0/3343291152 >> v2:172.21.15.118:3300/0 conn(0x7f13a00a8010 0x7f13a0132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 1 -- 172.21.15.50:0/3343291152 >> v1:172.21.15.50:6789/0 conn(0x7f13a01356d0 legacy=0x7f13a0135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 1 monclient: found mon.noname-a 2023-10-30T12:02:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:34.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.972+0000 7f1397fff640 1 -- 172.21.15.50:0/3343291152 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00a7600 con 0x7f13a0099a40 2023-10-30T12:02:34.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f1397fff640 1 --2- 172.21.15.50:0/3343291152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a0099a40 0x7f13a0132bd0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f13a00a8640 tx=0x7f138802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3b950fa480b8ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 1 -- 172.21.15.50:0/3343291152 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388031710 con 0x7f13a0099a40 2023-10-30T12:02:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: dump: 2023-10-30T12:02:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:34.996 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:02:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:02:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 1 -- 172.21.15.50:0/3343291152 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138802f050 con 0x7f13a0099a40 2023-10-30T12:02:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 1 -- 172.21.15.50:0/3343291152 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388031a90 con 0x7f13a0099a40 2023-10-30T12:02:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 5 monclient: authenticate success, global_id 55272 2023-10-30T12:02:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 4 set_mon_vals no callback set 2023-10-30T12:02:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f139e7fc640 10 monclient: dump: 2023-10-30T12:02:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.000 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 10 monclient: shutdown 2023-10-30T12:02:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 -- 172.21.15.50:0/3343291152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a0099a40 msgr2=0x7f13a0132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 --2- 172.21.15.50:0/3343291152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a0099a40 0x7f13a0132bd0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f13a00a8640 tx=0x7f138802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 -- 172.21.15.50:0/3343291152 shutdown_connections 2023-10-30T12:02:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 --2- 172.21.15.50:0/3343291152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a0099a40 0x7f13a0132bd0 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 --2- 172.21.15.50:0/3343291152 >> v2:172.21.15.118:3300/0 conn(0x7f13a00a8010 0x7f13a0132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 -- 172.21.15.50:0/3343291152 >> 172.21.15.50:0/3343291152 conn(0x7f13a00a4a10 msgr2=0x7f13a00a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 -- 172.21.15.50:0/3343291152 shutdown_connections 2023-10-30T12:02:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.973+0000 7f13a587c640 1 -- 172.21.15.50:0/3343291152 wait complete. 2023-10-30T12:02:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 10 monclient: build_initial_monmap 2023-10-30T12:02:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 10 monclient: monmap: 2023-10-30T12:02:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:35.005 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 1 librados: starting msgr at 2023-10-30T12:02:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 1 librados: starting objecter 2023-10-30T12:02:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 1 Processor -- start 2023-10-30T12:02:35.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 1 -- start start 2023-10-30T12:02:35.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 1 librados: setting wanted keys 2023-10-30T12:02:35.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 1 librados: calling monclient init 2023-10-30T12:02:35.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.974+0000 7f13a587c640 10 monclient: init 2023-10-30T12:02:35.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:35.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:02:35.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a00a8010 0x7f13a011dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): picked mon.noname-b con 0x7f13a00a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:35.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a01356d0 0x7f13a011e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): picked mon.noname-a con 0x7f13a01356d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:35.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13a011ea30 0x7f13a01ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): picked mon.noname-c con 0x7f13a011ea30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0120fe0 con 0x7f13a01356d0 2023-10-30T12:02:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a01200f0 con 0x7f13a00a8010 2023-10-30T12:02:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0131400 con 0x7f13a011ea30 2023-10-30T12:02:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f13a587c640 10 monclient(hunting): authenticate will time out at 1000.158341s 2023-10-30T12:02:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13a011ea30 0x7f13a01ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a01356d0 0x7f13a011e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a01356d0 0x7f13a011e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40392/0 (socket says 172.21.15.50:40392) 2023-10-30T12:02:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13a011ea30 0x7f13a01ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38382/0 (socket says 172.21.15.50:38382) 2023-10-30T12:02:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 -- 172.21.15.50:0/2166979910 learned_addr learned my addr 172.21.15.50:0/2166979910 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): get_auth_request con 0x7f13a01356d0 auth_method 0 2023-10-30T12:02:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a00a8010 0x7f13a011dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 10 monclient(hunting): get_auth_request con 0x7f13a011ea30 auth_method 0 2023-10-30T12:02:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 10 monclient(hunting): get_auth_request con 0x7f13a00a8010 auth_method 0 2023-10-30T12:02:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient(hunting): handle_auth_done global_id 55278 payload 824 2023-10-30T12:02:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13a011ea30 msgr2=0x7f13a01ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13a011ea30 0x7f13a01ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a00a8010 msgr2=0x7f13a011dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a00a8010 0x7f13a011dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 monclient: found mon.noname-a 2023-10-30T12:02:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:02:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 -- 172.21.15.50:0/2166979910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1388009400 con 0x7f13a01356d0 2023-10-30T12:02:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f1397fff640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a01356d0 0x7f13a011e4f0 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f1388000c00 tx=0x7f13880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139cff9640 10 client.?.objecter ms_handle_connect 0x7f13a01356d0 2023-10-30T12:02:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.975+0000 7f139cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:35.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138803c810 con 0x7f13a01356d0 2023-10-30T12:02:35.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:02:35.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: dump: 2023-10-30T12:02:35.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.018 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.018 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.018 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: _finish_auth 0 2023-10-30T12:02:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:02:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138803c990 con 0x7f13a01356d0 2023-10-30T12:02:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 5 monclient: authenticate success, global_id 55278 2023-10-30T12:02:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 10 monclient: _renew_subs 2023-10-30T12:02:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a00a18d0 con 0x7f13a01356d0 2023-10-30T12:02:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138803cd40 con 0x7f13a01356d0 2023-10-30T12:02:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:02:35.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139cff9640 10 monclient: dump: 2023-10-30T12:02:35.022 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.022 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.022 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.022 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.023 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.023 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.023 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.023 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.023 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.023 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f139dffb640 4 set_mon_vals no callback set 2023-10-30T12:02:35.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 10 client.55278.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:35.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 10 monclient: _renew_subs 2023-10-30T12:02:35.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:02:35.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a011f4b0 con 0x7f13a01356d0 2023-10-30T12:02:35.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.976+0000 7f13a587c640 1 librados: init done 2023-10-30T12:02:35.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f138803a040 con 0x7f13a01356d0 2023-10-30T12:02:35.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f13a587c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:35.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f13a587c640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0099a40 0x7f13a01caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f13a587c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0073fd0 con 0x7f13a0099a40 2023-10-30T12:02:35.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1380039320 0x7f138003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 9366+0+0 (secure 0 0 0) 0x7f138806ce30 con 0x7f13a01356d0 2023-10-30T12:02:35.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 10 client.55278.objecter ms_dispatch 0x7f13a0000b90 osd_map(49..49 src has 1..49) v4 2023-10-30T12:02:35.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 3 client.55278.objecter handle_osd_map got epochs [49,49] > 0 2023-10-30T12:02:35.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 3 client.55278.objecter handle_osd_map decoding full epoch 49 2023-10-30T12:02:35.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0099a40 0x7f13a01caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient: get_auth_request con 0x7f13a0099a40 auth_method 0 2023-10-30T12:02:35.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 20 client.55278.objecter dump_active .. 0 homeless 2023-10-30T12:02:35.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139f7fe640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1380039320 0x7f138003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139f7fe640 10 monclient: get_auth_request con 0x7f1380039320 auth_method 0 2023-10-30T12:02:35.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 10 monclient(hunting): handle_auth_done global_id 25915 payload 824 2023-10-30T12:02:35.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 1 -- 172.21.15.50:0/2166979910 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13a009a720 con 0x7f13a0099a40 2023-10-30T12:02:35.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 10 client.55278.objecter ms_handle_connect 0x7f13a0099a40 2023-10-30T12:02:35.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139effd640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0099a40 0x7f13a01caad0 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 crypto rx=0x7f13900118e0 tx=0x7f1390011da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:35.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 10 client.55278.objecter resend_mon_ops 2023-10-30T12:02:35.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139cff9640 10 client.55278.objecter ms_handle_connect 0x7f1380039320 2023-10-30T12:02:35.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.977+0000 7f139f7fe640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1380039320 0x7f138003b7e0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f1398005c00 tx=0x7f1398009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:35.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1390004300 con 0x7f13a0099a40 2023-10-30T12:02:35.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:35.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 10 monclient: dump: 2023-10-30T12:02:35.030 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.030 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.031 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.031 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.031 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.031 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.031 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.032 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.032 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.032 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1390004460 con 0x7f13a0099a40 2023-10-30T12:02:35.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:35.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:35.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0099a40 msgr2=0x7f13a01caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0099a40 0x7f13a01caad0 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 crypto rx=0x7f13900118e0 tx=0x7f1390011da0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0099a40 msgr2=0x7f13a01caad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:35.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.978+0000 7f139cff9640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0099a40 0x7f13a01caad0 secure :-1 s=CLOSED pgs=6411 cs=0 l=1 rev1=1 crypto rx=0x7f13900118e0 tx=0x7f1390011da0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f13a587c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:35.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f13a587c640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0121200 0x7f13a01cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f13a587c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a009a500 con 0x7f13a0121200 2023-10-30T12:02:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f139effd640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0121200 0x7f13a01cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f139effd640 10 monclient: get_auth_request con 0x7f13a0121200 auth_method 0 2023-10-30T12:02:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f139effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f139effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.984+0000 7f139effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139effd640 10 monclient(hunting): handle_auth_done global_id 25918 payload 824 2023-10-30T12:02:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139effd640 1 -- 172.21.15.50:0/2166979910 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f13a009aa00 con 0x7f13a0121200 2023-10-30T12:02:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139effd640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0121200 0x7f13a01cb010 secure :-1 s=READY pgs=6412 cs=0 l=1 rev1=1 crypto rx=0x7f139001b000 tx=0x7f13900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139cff9640 10 client.55278.objecter ms_handle_connect 0x7f13a0121200 2023-10-30T12:02:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139cff9640 10 client.55278.objecter resend_mon_ops 2023-10-30T12:02:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139002c070 con 0x7f13a0121200 2023-10-30T12:02:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.985+0000 7f139cff9640 10 monclient: dump: 2023-10-30T12:02:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.040 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f1390024040 con 0x7f13a0121200 2023-10-30T12:02:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0121200 msgr2=0x7f13a01cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139cff9640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0121200 0x7f13a01cb010 secure :-1 s=READY pgs=6412 cs=0 l=1 rev1=1 crypto rx=0x7f139001b000 tx=0x7f13900046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139cff9640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0121200 msgr2=0x7f13a01cb010 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139cff9640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13a0121200 0x7f13a01cb010 secure :-1 s=CLOSED pgs=6412 cs=0 l=1 rev1=1 crypto rx=0x7f139001b000 tx=0x7f13900046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.042 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:35.042 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:35.042 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:35.042 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:02:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 20, 2023-10-30T12:02:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:02:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:02:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:35.055 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:35.055 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:35.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:35.055 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:35.056 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:35.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:35.056 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:35.056 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:35.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.057 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:35.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.057 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:35.057 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:35.057 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.058 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:35.059 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:35.059 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:35.059 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:35.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.059 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:35.059 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:35.060 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:35.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.060 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:35.060 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:35.060 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:35.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.061 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:35.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.061 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:35.061 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:35.061 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:35.062 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:35.062 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:35.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:35.063 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:35.063 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:35.063 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:35.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.063 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:35.063 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:35.064 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:35.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:35.064 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:35.065 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:35.065 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.065 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:35.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.065 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:35.065 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:35.066 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:35.066 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:35.066 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:35.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.066 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:35.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.067 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:35.067 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:35.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.067 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:35.068 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:35.068 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:35.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.068 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:35.068 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:02:35.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.069 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:35.069 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:35.069 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:02:35.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.069 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:35.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:35.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.070 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:35.070 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:35.070 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 30 2023-10-30T12:02:35.070 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.071 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:35.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:02:35.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:35.071 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:35.071 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:35.071 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:35.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:35.072 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:35.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:35.072 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:35.072 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 10 librados: watch_flush enter 2023-10-30T12:02:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 10 librados: watch_flush exit 2023-10-30T12:02:35.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 20 client.55278.objecter shutdown clearing up homeless session... 2023-10-30T12:02:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 10 client.55278.objecter successfully canceled tick 2023-10-30T12:02:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1380039320 msgr2=0x7f138003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1380039320 0x7f138003b7e0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f1398005c00 tx=0x7f1398009040 comp rx=0 tx=0).stop 2023-10-30T12:02:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 10 monclient: shutdown 2023-10-30T12:02:35.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a01356d0 msgr2=0x7f13a011e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 --2- 172.21.15.50:0/2166979910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13a01356d0 0x7f13a011e4f0 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f1388000c00 tx=0x7f13880046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139f7fe640 1 -- 172.21.15.50:0/2166979910 reap_dead start 2023-10-30T12:02:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f139f7fe640 1 -- 172.21.15.50:0/2166979910 reap_dead start 2023-10-30T12:02:35.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 shutdown_connections 2023-10-30T12:02:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 >> 172.21.15.50:0/2166979910 conn(0x7f13a00a4a10 msgr2=0x7f13a0069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 shutdown_connections 2023-10-30T12:02:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 -- 172.21.15.50:0/2166979910 wait complete. 2023-10-30T12:02:35.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:34.986+0000 7f13a587c640 1 librados: shutdown 2023-10-30T12:02:35.077 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:02:35.078 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:02:35.078 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:02:35.078 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:02:35.078 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:02:35.079 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:02:35.079 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:02:35.088 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:02:35.089 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:02:35.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:02:35.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.237+0000 7fece7010640 10 monclient: get_monmap_and_config 2023-10-30T12:02:35.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.237+0000 7fece7010640 10 monclient: build_initial_monmap 2023-10-30T12:02:35.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.237+0000 7fece7010640 10 monclient: monmap: 2023-10-30T12:02:35.240 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:35.240 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.240 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:35.238541+0000 2023-10-30T12:02:35.240 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:35.238541+0000 2023-10-30T12:02:35.240 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:35.241 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.241 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:35.241 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:35.241 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:35.242 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:35.242 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:35.242 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 1 Processor -- start 2023-10-30T12:02:35.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 1 -- start start 2023-10-30T12:02:35.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient: init 2023-10-30T12:02:35.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:35.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-10-30T12:02:35.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fece00a2dd0 0x7fece00a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient(hunting): picked mon.noname-c con 0x7fece00a2dd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:35.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fece006a8e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:35.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fece0071400 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:35.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece00a31b0 con 0x7fece0071400 2023-10-30T12:02:35.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece006a6a0 con 0x7fece006a8e0 2023-10-30T12:02:35.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece006be90 con 0x7fece00a2dd0 2023-10-30T12:02:35.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:35.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece7010640 10 monclient(hunting): authenticate will time out at 1000.421710s 2023-10-30T12:02:35.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece55ad640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fece0071400 legacy=0x7fece013cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:35.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece4dac640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fece00a2dd0 0x7fece00a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece4dac640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fece00a2dd0 0x7fece00a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:55500/0 (socket says 172.21.15.50:55500) 2023-10-30T12:02:35.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece4dac640 1 -- 172.21.15.50:0/2834193783 learned_addr learned my addr 172.21.15.50:0/2834193783 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:35.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece4dac640 10 monclient(hunting): get_auth_request con 0x7fece00a2dd0 auth_method 0 2023-10-30T12:02:35.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece4dac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece4dac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.238+0000 7fece4dac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 10 monclient(hunting): handle_auth_done global_id 26263 payload 274 2023-10-30T12:02:35.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 1 -- 172.21.15.50:0/2834193783 >> v1:172.21.15.118:6789/0 conn(0x7fece006a8e0 legacy=0x7fece00a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 1 -- 172.21.15.50:0/2834193783 >> v1:172.21.15.50:6789/0 conn(0x7fece0071400 legacy=0x7fece013cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:35.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 10 monclient: _finish_hunting 0 2023-10-30T12:02:35.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 1 monclient: found mon.noname-c 2023-10-30T12:02:35.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:35.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:02:35.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.239+0000 7fece4dac640 1 -- 172.21.15.50:0/2834193783 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece009e190 con 0x7fece00a2dd0 2023-10-30T12:02:35.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece4dac640 1 --2- 172.21.15.50:0/2834193783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece00a2dd0 0x7fece00a46c0 secure :-1 s=READY pgs=6508 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009870 tx=0x7fecd002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef418de3b491b29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:35.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 1 -- 172.21.15.50:0/2834193783 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00317e0 con 0x7fece00a2dd0 2023-10-30T12:02:35.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:35.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: dump: 2023-10-30T12:02:35.257 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.257 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.257 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.258 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.258 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.258 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.258 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.259 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:02:35.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:02:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 5 monclient: authenticate success, global_id 26263 2023-10-30T12:02:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 1 -- 172.21.15.50:0/2834193783 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0031940 con 0x7fece00a2dd0 2023-10-30T12:02:35.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 1 -- 172.21.15.50:0/2834193783 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0031cd0 con 0x7fece00a2dd0 2023-10-30T12:02:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 4 set_mon_vals no callback set 2023-10-30T12:02:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:35.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fecdf7fe640 10 monclient: dump: 2023-10-30T12:02:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.262 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.263 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.263 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.263 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.263 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.263 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 10 monclient: shutdown 2023-10-30T12:02:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 1 -- 172.21.15.50:0/2834193783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece00a2dd0 msgr2=0x7fece00a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 1 --2- 172.21.15.50:0/2834193783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece00a2dd0 0x7fece00a46c0 secure :-1 s=READY pgs=6508 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009870 tx=0x7fecd002b880 comp rx=0 tx=0).stop 2023-10-30T12:02:35.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 1 -- 172.21.15.50:0/2834193783 shutdown_connections 2023-10-30T12:02:35.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 1 --2- 172.21.15.50:0/2834193783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece00a2dd0 0x7fece00a46c0 unknown :-1 s=CLOSED pgs=6508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 1 -- 172.21.15.50:0/2834193783 >> 172.21.15.50:0/2834193783 conn(0x7fece00a22b0 msgr2=0x7fece00a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:35.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 1 -- 172.21.15.50:0/2834193783 shutdown_connections 2023-10-30T12:02:35.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.240+0000 7fece7010640 1 -- 172.21.15.50:0/2834193783 wait complete. 2023-10-30T12:02:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 10 monclient: build_initial_monmap 2023-10-30T12:02:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 10 monclient: monmap: 2023-10-30T12:02:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:35.267 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:35.267 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:35.267 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:35.267 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.268 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:35.268 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:35.268 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:35.268 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 1 librados: starting msgr at 2023-10-30T12:02:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 1 librados: starting objecter 2023-10-30T12:02:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 1 Processor -- start 2023-10-30T12:02:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 1 -- start start 2023-10-30T12:02:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 1 librados: setting wanted keys 2023-10-30T12:02:35.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 1 librados: calling monclient init 2023-10-30T12:02:35.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.241+0000 7fece7010640 10 monclient: init 2023-10-30T12:02:35.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:35.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:02:35.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fece0071400 0x7fece01bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): picked mon.noname-c con 0x7fece0071400 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:35.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fece00a2dd0 0x7fece01be2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): picked mon.noname-a con 0x7fece00a2dd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:35.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece01a7470 0x7fece01a7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): picked mon.noname-b con 0x7fece01a7470 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:35.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece01aaf20 con 0x7fece00a2dd0 2023-10-30T12:02:35.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece01a9b80 con 0x7fece01a7470 2023-10-30T12:02:35.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:35.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece00a1de0 con 0x7fece0071400 2023-10-30T12:02:35.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:35.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece7010640 10 monclient(hunting): authenticate will time out at 1000.425467s 2023-10-30T12:02:35.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fecdffff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fece00a2dd0 msgr2=0x7fece01be2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:35.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fecdffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fece00a2dd0 0x7fece01be2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:35.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fece0071400 0x7fece01bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fece0071400 0x7fece01bdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38384/0 (socket says 172.21.15.50:38384) 2023-10-30T12:02:35.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 -- 172.21.15.50:0/2170723430 learned_addr learned my addr 172.21.15.50:0/2170723430 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:35.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): get_auth_request con 0x7fece0071400 auth_method 0 2023-10-30T12:02:35.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece01a7470 0x7fece01a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 10 monclient(hunting): get_auth_request con 0x7fece01a7470 auth_method 0 2023-10-30T12:02:35.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:35.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:35.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:35.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece55ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient(hunting): handle_auth_done global_id 43124 payload 824 2023-10-30T12:02:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 -- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece01a7470 msgr2=0x7fece01a7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece01a7470 0x7fece01a7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 -- 172.21.15.50:0/2170723430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fece00a2dd0 msgr2=0x7fece01be2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:02:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fece00a2dd0 0x7fece01be2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient: _finish_hunting 0 2023-10-30T12:02:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 monclient: found mon.noname-c 2023-10-30T12:02:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:35.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.242+0000 7fece4dac640 1 -- 172.21.15.50:0/2170723430 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd00094d0 con 0x7fece0071400 2023-10-30T12:02:35.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece4dac640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fece0071400 0x7fece01bdd90 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000c00 tx=0x7fecd00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:35.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 client.?.objecter ms_handle_connect 0x7fece0071400 2023-10-30T12:02:35.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:35.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 1 -- 172.21.15.50:0/2170723430 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0032a00 con 0x7fece0071400 2023-10-30T12:02:35.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:35.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: dump: 2023-10-30T12:02:35.281 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.281 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.281 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.282 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.282 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.282 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.282 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.282 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.283 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.283 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: _finish_auth 0 2023-10-30T12:02:35.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:02:35.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:35.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 1 -- 172.21.15.50:0/2170723430 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0032b80 con 0x7fece0071400 2023-10-30T12:02:35.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 5 monclient: authenticate success, global_id 43124 2023-10-30T12:02:35.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 10 monclient: _renew_subs 2023-10-30T12:02:35.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:35.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece006b0a0 con 0x7fece0071400 2023-10-30T12:02:35.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:35.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 1 -- 172.21.15.50:0/2170723430 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00453f0 con 0x7fece0071400 2023-10-30T12:02:35.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:35.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:35.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdd7fa640 10 monclient: dump: 2023-10-30T12:02:35.285 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:35.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:35.285 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.285 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:35.286 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:35.286 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:35.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:35.286 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:35.286 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:35.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:35.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 10 client.43124.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:35.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fecdeffd640 4 set_mon_vals no callback set 2023-10-30T12:02:35.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 10 monclient: _renew_subs 2023-10-30T12:02:35.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece01a9110 con 0x7fece0071400 2023-10-30T12:02:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.243+0000 7fece7010640 1 librados: init done 2023-10-30T12:02:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 1 -- 172.21.15.50:0/2170723430 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fecd004d3f0 con 0x7fece0071400 2023-10-30T12:02:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feccc0392a0 0x7feccc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fece7010640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 1 -- 172.21.15.50:0/2170723430 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 9366+0+0 (secure 0 0 0) 0x7fecd006d7f0 con 0x7fece0071400 2023-10-30T12:02:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 10 client.43124.objecter ms_dispatch 0x7fece0000b90 osd_map(49..49 src has 1..49) v4 2023-10-30T12:02:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 3 client.43124.objecter handle_osd_map got epochs [49,49] > 0 2023-10-30T12:02:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 3 client.43124.objecter handle_osd_map decoding full epoch 49 2023-10-30T12:02:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fece7010640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fece7010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece01ab060 con 0x7fece0071400 2023-10-30T12:02:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 20 client.43124.objecter dump_active .. 0 homeless 2023-10-30T12:02:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdffff640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feccc0392a0 0x7feccc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdffff640 10 monclient: get_auth_request con 0x7feccc0392a0 auth_method 0 2023-10-30T12:02:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdd7fa640 10 client.43124.objecter ms_handle_connect 0x7feccc0392a0 2023-10-30T12:02:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.244+0000 7fecdffff640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feccc0392a0 0x7feccc03b760 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fecd4001030 tx=0x7fecd4007a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.249+0000 7fecdd7fa640 1 -- 172.21.15.50:0/2170723430 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fece01ab060 con 0x7fece0071400 2023-10-30T12:02:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.249+0000 7fecdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.249+0000 7fecdd7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:35.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.505+0000 7fece7010640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:02:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.506+0000 7fece7010640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.506+0000 7fece7010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.506+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fece01ab060 con 0x7fece0071400 2023-10-30T12:02:35.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.506+0000 7fecdd7fa640 1 -- 172.21.15.50:0/2170723430 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fece01ab060 con 0x7fece0071400 2023-10-30T12:02:35.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.506+0000 7fecdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:35.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.506+0000 7fecdd7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:35.509 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:02:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 10 librados: watch_flush enter 2023-10-30T12:02:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 10 librados: watch_flush exit 2023-10-30T12:02:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 20 client.43124.objecter shutdown clearing up homeless session... 2023-10-30T12:02:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 10 client.43124.objecter successfully canceled tick 2023-10-30T12:02:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feccc0392a0 msgr2=0x7feccc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feccc0392a0 0x7feccc03b760 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fecd4001030 tx=0x7fecd4007a20 comp rx=0 tx=0).stop 2023-10-30T12:02:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 10 monclient: shutdown 2023-10-30T12:02:35.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fece0071400 msgr2=0x7fece01bdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fece0071400 0x7fece01bdd90 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000c00 tx=0x7fecd00046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 shutdown_connections 2023-10-30T12:02:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feccc0392a0 0x7feccc03b760 unknown :-1 s=CLOSED pgs=3613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fece01a7470 0x7fece01a7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fece00a2dd0 0x7fece01be2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 --2- 172.21.15.50:0/2170723430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fece0071400 0x7fece01bdd90 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:35.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 >> 172.21.15.50:0/2170723430 conn(0x7fece00a22b0 msgr2=0x7fece00a0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:35.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 shutdown_connections 2023-10-30T12:02:35.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 -- 172.21.15.50:0/2170723430 wait complete. 2023-10-30T12:02:35.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:35.508+0000 7fece7010640 1 librados: shutdown 2023-10-30T12:02:38.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:02:38.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.637+0000 7fb100a4b640 10 monclient: get_monmap_and_config 2023-10-30T12:02:38.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.637+0000 7fb100a4b640 10 monclient: build_initial_monmap 2023-10-30T12:02:38.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.637+0000 7fb100a4b640 10 monclient: monmap: 2023-10-30T12:02:38.639 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:38.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:38.640 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:38.638152+0000 2023-10-30T12:02:38.640 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:38.638152+0000 2023-10-30T12:02:38.640 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:38.642 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.637+0000 7fb100a4b640 1 Processor -- start 2023-10-30T12:02:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.637+0000 7fb100a4b640 1 -- start start 2023-10-30T12:02:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.637+0000 7fb100a4b640 10 monclient: init 2023-10-30T12:02:38.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:38.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:02:38.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0fc0a6070 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:38.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc13a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:38.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0fc0a57e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb0fc13adf0 0x7fb0fc13d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0fc13adf0 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0fc09a270 con 0x7fb0fc0a6070 2023-10-30T12:02:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:38.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc0a5c60 con 0x7fb0fc13adf0 2023-10-30T12:02:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc099f90 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb100a4b640 10 monclient(hunting): authenticate will time out at 1003.821354s 2023-10-30T12:02:38.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0fa59c640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fb0fc0a6070 legacy=0x7fb0fc0a6450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc13a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc13a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50530/0 (socket says 172.21.15.50:50530) 2023-10-30T12:02:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 -- 172.21.15.50:0/4203641921 learned_addr learned my addr 172.21.15.50:0/4203641921 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:38.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): get_auth_request con 0x7fb0fc0a57e0 auth_method 0 2023-10-30T12:02:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:38.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0fad9d640 1 --2- 172.21.15.50:0/4203641921 >> v2:172.21.15.118:3300/0 conn(0x7fb0fc13adf0 0x7fb0fc13d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient(hunting): handle_auth_done global_id 44126 payload 274 2023-10-30T12:02:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 -- 172.21.15.50:0/4203641921 >> v2:172.21.15.118:3300/0 conn(0x7fb0fc13adf0 msgr2=0x7fb0fc13d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 --2- 172.21.15.50:0/4203641921 >> v2:172.21.15.118:3300/0 conn(0x7fb0fc13adf0 0x7fb0fc13d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 -- 172.21.15.50:0/4203641921 >> v1:172.21.15.50:6789/0 conn(0x7fb0fc0a6070 legacy=0x7fb0fc0a6450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:38.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:38.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 monclient: found mon.noname-b 2023-10-30T12:02:38.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:38.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:02:38.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0f3fff640 1 -- 172.21.15.50:0/4203641921 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc09a550 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0fad9d640 10 monclient: get_auth_request con 0x7fb0fc13adf0 auth_method 0 2023-10-30T12:02:38.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.638+0000 7fb0fad9d640 1 --2- 172.21.15.50:0/4203641921 >> v2:172.21.15.118:3300/0 conn(0x7fb0fc13adf0 0x7fb0fc13d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:02:38.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f3fff640 1 --2- 172.21.15.50:0/4203641921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc13a8b0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc136630 tx=0x7fb0e402b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=823726cedccf344d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:38.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 1 -- 172.21.15.50:0/4203641921 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4031520 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:38.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:02:38.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: dump: 2023-10-30T12:02:38.658 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:38.658 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:38.658 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.658 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.659 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:38.659 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:38.659 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:38.659 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:38.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:38.660 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:38.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: _finish_auth 0 2023-10-30T12:02:38.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:02:38.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:38.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 1 -- 172.21.15.50:0/4203641921 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4031680 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb100a4b640 5 monclient: authenticate success, global_id 44126 2023-10-30T12:02:38.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb100a4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:38.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:38.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb100a4b640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:38.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb100a4b640 4 set_mon_vals no callback set 2023-10-30T12:02:38.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 1 -- 172.21.15.50:0/4203641921 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4031a10 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:38.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:38.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.639+0000 7fb0f9d9b640 10 monclient: dump: 2023-10-30T12:02:38.662 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:38.662 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:38.662 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.663 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.663 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:38.663 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:38.663 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:38.663 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:38.664 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:38.664 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:38.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 10 monclient: shutdown 2023-10-30T12:02:38.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:38.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 -- 172.21.15.50:0/4203641921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 msgr2=0x7fb0fc13a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 --2- 172.21.15.50:0/4203641921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc13a8b0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc136630 tx=0x7fb0e402b7c0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 -- 172.21.15.50:0/4203641921 shutdown_connections 2023-10-30T12:02:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 --2- 172.21.15.50:0/4203641921 >> v2:172.21.15.118:3300/0 conn(0x7fb0fc13adf0 0x7fb0fc13d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 --2- 172.21.15.50:0/4203641921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc13a8b0 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 -- 172.21.15.50:0/4203641921 >> 172.21.15.50:0/4203641921 conn(0x7fb0fc0a81d0 msgr2=0x7fb0fc0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 -- 172.21.15.50:0/4203641921 shutdown_connections 2023-10-30T12:02:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.640+0000 7fb100a4b640 1 -- 172.21.15.50:0/4203641921 wait complete. 2023-10-30T12:02:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient: build_initial_monmap 2023-10-30T12:02:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient: monmap: 2023-10-30T12:02:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:38.667 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:38.667 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:38.670 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:38.670 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:38.670 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:38.670 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:38.670 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:38.671 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 librados: starting msgr at 2023-10-30T12:02:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 librados: starting objecter 2023-10-30T12:02:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 Processor -- start 2023-10-30T12:02:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 -- start start 2023-10-30T12:02:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 librados: setting wanted keys 2023-10-30T12:02:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 librados: calling monclient init 2023-10-30T12:02:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient: init 2023-10-30T12:02:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:38.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:02:38.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:38.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0fc0a57e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:38.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0fc0a6070 0x7fb0fc131f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:38.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0fc0a6070 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:38.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0fc13adf0 0x7fb0fc09d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0fc13adf0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.641+0000 7fb100a4b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc133640 con 0x7fb0fc0a6070 2023-10-30T12:02:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0a1690 con 0x7fb0fc13adf0 2023-10-30T12:02:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc11e840 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient(hunting): authenticate will time out at 1003.825021s 2023-10-30T12:02:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f3fff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0fc0a6070 msgr2=0x7fb0fc131f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0fc0a6070 0x7fb0fc131f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc131a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50542/0 (socket says 172.21.15.50:50542) 2023-10-30T12:02:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 -- 172.21.15.50:0/3813229987 learned_addr learned my addr 172.21.15.50:0/3813229987 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): get_auth_request con 0x7fb0fc0a57e0 auth_method 0 2023-10-30T12:02:38.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:38.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:38.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:38.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:38.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:38.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0fc13adf0 0x7fb0fc09d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:38.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 10 monclient(hunting): get_auth_request con 0x7fb0fc13adf0 auth_method 0 2023-10-30T12:02:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:38.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient(hunting): handle_auth_done global_id 44129 payload 824 2023-10-30T12:02:38.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 -- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0fc13adf0 msgr2=0x7fb0fc09d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:38.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0fc13adf0 0x7fb0fc09d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 -- 172.21.15.50:0/3813229987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0fc0a6070 msgr2=0x7fb0fc131f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:02:38.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0fc0a6070 0x7fb0fc131f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient: _finish_hunting 0 2023-10-30T12:02:38.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 monclient: found mon.noname-c 2023-10-30T12:02:38.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:38.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:38.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 -- 172.21.15.50:0/3813229987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e40094d0 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0fa59c640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc131a00 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00b550 tx=0x7fb0ec00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:38.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 client.?.objecter ms_handle_connect 0x7fb0fc0a57e0 2023-10-30T12:02:38.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:38.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 1 -- 172.21.15.50:0/3813229987 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec003dc0 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:38.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:38.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: dump: 2023-10-30T12:02:38.688 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:38.688 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:38.688 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.689 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.689 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:38.689 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:38.689 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:38.689 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:38.689 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:38.690 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:38.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:02:38.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:38.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:38.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 1 -- 172.21.15.50:0/3813229987 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ec005070 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 5 monclient: authenticate success, global_id 44129 2023-10-30T12:02:38.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:38.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient: _renew_subs 2023-10-30T12:02:38.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 1 -- 172.21.15.50:0/3813229987 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec005420 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:38.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc0a0350 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:38.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:38.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f2ffd640 10 monclient: dump: 2023-10-30T12:02:38.692 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:38.693 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:38.693 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.693 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:38.693 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:38.693 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:38.693 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:38.694 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:38.694 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:38.694 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:38.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb0f959a640 4 set_mon_vals no callback set 2023-10-30T12:02:38.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 client.44129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:38.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient: _renew_subs 2023-10-30T12:02:38.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:38.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc132480 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.642+0000 7fb100a4b640 1 librados: init done 2023-10-30T12:02:38.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb0f2ffd640 1 -- 172.21.15.50:0/3813229987 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb0ec005580 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb100a4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:38.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb100a4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:38.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb100a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:38.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc04e160 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb0f2ffd640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0dc0392d0 0x7fb0dc03b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:38.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb0f2ffd640 1 -- 172.21.15.50:0/3813229987 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 9366+0+0 (secure 0 0 0) 0x7fb0ec04b7b0 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.643+0000 7fb0f2ffd640 10 client.44129.objecter ms_dispatch 0x7fb0fc000b90 osd_map(49..49 src has 1..49) v4 2023-10-30T12:02:38.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.644+0000 7fb0f2ffd640 3 client.44129.objecter handle_osd_map got epochs [49,49] > 0 2023-10-30T12:02:38.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.644+0000 7fb0f2ffd640 3 client.44129.objecter handle_osd_map decoding full epoch 49 2023-10-30T12:02:38.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.644+0000 7fb0f2ffd640 20 client.44129.objecter dump_active .. 0 homeless 2023-10-30T12:02:38.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.644+0000 7fb0f3fff640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0dc0392d0 0x7fb0dc03b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:38.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.644+0000 7fb0f3fff640 10 monclient: get_auth_request con 0x7fb0dc0392d0 auth_method 0 2023-10-30T12:02:38.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.644+0000 7fb0f2ffd640 10 client.44129.objecter ms_handle_connect 0x7fb0dc0392d0 2023-10-30T12:02:38.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.644+0000 7fb0f3fff640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0dc0392d0 0x7fb0dc03b790 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4035000 tx=0x7fb0e4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:38.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.648+0000 7fb0f2ffd640 1 -- 172.21.15.50:0/3813229987 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0ec020350 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.648+0000 7fb0f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:38.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.648+0000 7fb0f2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.894+0000 7fb100a4b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:02:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.894+0000 7fb100a4b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.894+0000 7fb100a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.894+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb0fc074580 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.894+0000 7fb0f2ffd640 1 -- 172.21.15.50:0/3813229987 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fb0ec013030 con 0x7fb0fc0a57e0 2023-10-30T12:02:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.894+0000 7fb0f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.894+0000 7fb0f2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:38.899 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:02:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 10 librados: watch_flush enter 2023-10-30T12:02:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 10 librados: watch_flush exit 2023-10-30T12:02:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 20 client.44129.objecter shutdown clearing up homeless session... 2023-10-30T12:02:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 10 client.44129.objecter successfully canceled tick 2023-10-30T12:02:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0dc0392d0 msgr2=0x7fb0dc03b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0dc0392d0 0x7fb0dc03b790 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4035000 tx=0x7fb0e4030000 comp rx=0 tx=0).stop 2023-10-30T12:02:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 10 monclient: shutdown 2023-10-30T12:02:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.896+0000 7fb100a4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 msgr2=0x7fb0fc131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc131a00 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00b550 tx=0x7fb0ec00ba10 comp rx=0 tx=0).stop 2023-10-30T12:02:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 shutdown_connections 2023-10-30T12:02:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0dc0392d0 0x7fb0dc03b790 unknown :-1 s=CLOSED pgs=3832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0fc13adf0 0x7fb0fc09d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0fc0a6070 0x7fb0fc131f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 --2- 172.21.15.50:0/3813229987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb0fc0a57e0 0x7fb0fc131a00 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 >> 172.21.15.50:0/3813229987 conn(0x7fb0fc0a81d0 msgr2=0x7fb0fc134f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 shutdown_connections 2023-10-30T12:02:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 -- 172.21.15.50:0/3813229987 wait complete. 2023-10-30T12:02:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:38.897+0000 7fb100a4b640 1 librados: shutdown 2023-10-30T12:02:41.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:02:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.025+0000 7fbee29d0640 10 monclient: get_monmap_and_config 2023-10-30T12:02:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.025+0000 7fbee29d0640 10 monclient: build_initial_monmap 2023-10-30T12:02:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.025+0000 7fbee29d0640 10 monclient: monmap: 2023-10-30T12:02:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:42.026677+0000 2023-10-30T12:02:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:42.026677+0000 2023-10-30T12:02:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:42.029 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:42.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 1 Processor -- start 2023-10-30T12:02:42.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 1 -- start start 2023-10-30T12:02:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient: init 2023-10-30T12:02:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:02:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbedc0a7000 0x7fbedc135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient(hunting): picked mon.noname-a con 0x7fbedc0a7000 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbedc06abe0 0x7fbedc1360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient(hunting): picked mon.noname-c con 0x7fbedc06abe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbedc0a4460 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbedc0a73e0 con 0x7fbedc0a4460 2023-10-30T12:02:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbedc136600 con 0x7fbedc0a7000 2023-10-30T12:02:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbedc09a000 con 0x7fbedc06abe0 2023-10-30T12:02:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbee29d0640 10 monclient(hunting): authenticate will time out at 1007.209960s 2023-10-30T12:02:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.026+0000 7fbedbfff640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fbedc0a7000 msgr2=0x7fbedc135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedbfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbedc0a7000 0x7fbedc135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbedc06abe0 0x7fbedc1360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbee0f6d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fbedc0a4460 0x7fbedc0a48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:38008/0 (socket says 172.21.15.50:38008) 2023-10-30T12:02:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbedc06abe0 0x7fbedc1360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:53778/0 (socket says 172.21.15.50:53778) 2023-10-30T12:02:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbee0f6d640 1 -- 172.21.15.50:0/3922576767 learned_addr learned my addr 172.21.15.50:0/3922576767 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:02:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): get_auth_request con 0x7fbedc06abe0 auth_method 0 2023-10-30T12:02:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient(hunting): handle_auth_done global_id 27679 payload 274 2023-10-30T12:02:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 1 -- 172.21.15.50:0/3922576767 >> v2:172.21.15.50:3300/0 conn(0x7fbedc0a7000 msgr2=0x7fbedc135b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 1 --2- 172.21.15.50:0/3922576767 >> v2:172.21.15.50:3300/0 conn(0x7fbedc0a7000 0x7fbedc135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 1 -- 172.21.15.50:0/3922576767 >> v1:172.21.15.118:6789/0 conn(0x7fbedc0a4460 legacy=0x7fbedc0a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:02:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 1 monclient: found mon.noname-c 2023-10-30T12:02:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:02:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.027+0000 7fbedb7fe640 1 -- 172.21.15.50:0/3922576767 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc09a2e0 con 0x7fbedc06abe0 2023-10-30T12:02:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedb7fe640 1 --2- 172.21.15.50:0/3922576767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc06abe0 0x7fbedc1360c0 secure :-1 s=READY pgs=6930 cs=0 l=1 rev1=1 crypto rx=0x7fbedc06a960 tx=0x7fbecc02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5dc39cbc33fe74c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 1 -- 172.21.15.50:0/3922576767 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0317c0 con 0x7fbedc06abe0 2023-10-30T12:02:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: dump: 2023-10-30T12:02:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:42.047 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: _finish_auth 0 2023-10-30T12:02:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 1 -- 172.21.15.50:0/3922576767 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc031920 con 0x7fbedc06abe0 2023-10-30T12:02:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbee29d0640 5 monclient: authenticate success, global_id 27679 2023-10-30T12:02:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbee29d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 1 -- 172.21.15.50:0/3922576767 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc031cb0 con 0x7fbedc06abe0 2023-10-30T12:02:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbee29d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbee29d0640 4 set_mon_vals no callback set 2023-10-30T12:02:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.028+0000 7fbedaffd640 10 monclient: dump: 2023-10-30T12:02:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:42.051 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 10 monclient: shutdown 2023-10-30T12:02:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 -- 172.21.15.50:0/3922576767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc06abe0 msgr2=0x7fbedc1360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3922576767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc06abe0 0x7fbedc1360c0 secure :-1 s=READY pgs=6930 cs=0 l=1 rev1=1 crypto rx=0x7fbedc06a960 tx=0x7fbecc02b860 comp rx=0 tx=0).stop 2023-10-30T12:02:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 -- 172.21.15.50:0/3922576767 shutdown_connections 2023-10-30T12:02:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3922576767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc06abe0 0x7fbedc1360c0 unknown :-1 s=CLOSED pgs=6930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3922576767 >> v2:172.21.15.50:3300/0 conn(0x7fbedc0a7000 0x7fbedc135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 -- 172.21.15.50:0/3922576767 >> 172.21.15.50:0/3922576767 conn(0x7fbedc066000 msgr2=0x7fbedc0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 -- 172.21.15.50:0/3922576767 shutdown_connections 2023-10-30T12:02:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 1 -- 172.21.15.50:0/3922576767 wait complete. 2023-10-30T12:02:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 10 monclient: build_initial_monmap 2023-10-30T12:02:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.029+0000 7fbee29d0640 10 monclient: monmap: 2023-10-30T12:02:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:42.056 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 librados: starting msgr at 2023-10-30T12:02:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 librados: starting objecter 2023-10-30T12:02:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 Processor -- start 2023-10-30T12:02:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 -- start start 2023-10-30T12:02:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 librados: setting wanted keys 2023-10-30T12:02:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 librados: calling monclient init 2023-10-30T12:02:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 10 monclient: init 2023-10-30T12:02:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:02:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbedc06abe0 0x7fbedc09e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 10 monclient(hunting): picked mon.noname-a con 0x7fbedc06abe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.030+0000 7fbee29d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc0a4460 0x7fbedc09ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient(hunting): picked mon.noname-b con 0x7fbedc0a4460 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbedc0a7000 0x7fbedc09efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient(hunting): picked mon.noname-c con 0x7fbedc0a7000 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedbfff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbedc06abe0 msgr2=0x7fbedc09e530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc11e060 con 0x7fbedc06abe0 2023-10-30T12:02:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc1324a0 con 0x7fbedc0a4460 2023-10-30T12:02:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc11ec80 con 0x7fbedc0a7000 2023-10-30T12:02:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedbfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbedc06abe0 0x7fbedc09e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient(hunting): authenticate will time out at 1007.214052s 2023-10-30T12:02:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbedc0a7000 0x7fbedc09efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbedc0a7000 0x7fbedc09efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50546/0 (socket says 172.21.15.50:50546) 2023-10-30T12:02:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 -- 172.21.15.50:0/3013797668 learned_addr learned my addr 172.21.15.50:0/3013797668 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): get_auth_request con 0x7fbedc0a7000 auth_method 0 2023-10-30T12:02:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc0a4460 0x7fbedc09ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 10 monclient(hunting): get_auth_request con 0x7fbedc0a4460 auth_method 0 2023-10-30T12:02:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbedb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient(hunting): handle_auth_done global_id 44132 payload 824 2023-10-30T12:02:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 -- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc0a4460 msgr2=0x7fbedc09ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc0a4460 0x7fbedc09ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 -- 172.21.15.50:0/3013797668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbedc06abe0 msgr2=0x7fbedc09e530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:02:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbedc06abe0 0x7fbedc09e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient: _finish_hunting 0 2023-10-30T12:02:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 monclient: found mon.noname-c 2023-10-30T12:02:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 -- 172.21.15.50:0/3013797668 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc009400 con 0x7fbedc0a7000 2023-10-30T12:02:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee0f6d640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbedc0a7000 0x7fbedc09efb0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fbed400cae0 tx=0x7fbed400a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 10 client.?.objecter ms_handle_connect 0x7fbedc0a7000 2023-10-30T12:02:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 1 -- 172.21.15.50:0/3013797668 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed400ac40 con 0x7fbedc0a7000 2023-10-30T12:02:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 10 monclient: dump: 2023-10-30T12:02:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:42.070 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 10 monclient: _finish_auth 0 2023-10-30T12:02:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:02:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 5 monclient: authenticate success, global_id 44132 2023-10-30T12:02:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbed8ff9640 1 -- 172.21.15.50:0/3013797668 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed400ada0 con 0x7fbedc0a7000 2023-10-30T12:02:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient: _renew_subs 2023-10-30T12:02:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.031+0000 7fbee29d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc11f1c0 con 0x7fbedc0a7000 2023-10-30T12:02:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbed8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbed8ff9640 1 -- 172.21.15.50:0/3013797668 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed4012620 con 0x7fbedc0a7000 2023-10-30T12:02:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbed8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbed8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbed8ff9640 10 monclient: dump: 2023-10-30T12:02:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:42.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbeda7fc640 4 set_mon_vals no callback set 2023-10-30T12:02:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 10 client.44132.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 10 monclient: _renew_subs 2023-10-30T12:02:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc11e7d0 con 0x7fbedc0a7000 2023-10-30T12:02:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 1 librados: init done 2023-10-30T12:02:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbed8ff9640 1 -- 172.21.15.50:0/3013797668 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbed4007070 con 0x7fbedc0a7000 2023-10-30T12:02:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:42.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.032+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc04e160 con 0x7fbedc0a7000 2023-10-30T12:02:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbed8ff9640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbec80392a0 0x7fbec803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbed8ff9640 1 -- 172.21.15.50:0/3013797668 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 9366+0+0 (secure 0 0 0) 0x7fbed404b7f0 con 0x7fbedc0a7000 2023-10-30T12:02:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbed8ff9640 10 client.44132.objecter ms_dispatch 0x7fbedc000b90 osd_map(49..49 src has 1..49) v4 2023-10-30T12:02:42.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbed8ff9640 3 client.44132.objecter handle_osd_map got epochs [49,49] > 0 2023-10-30T12:02:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbed8ff9640 3 client.44132.objecter handle_osd_map decoding full epoch 49 2023-10-30T12:02:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbed8ff9640 20 client.44132.objecter dump_active .. 0 homeless 2023-10-30T12:02:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbedbfff640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbec80392a0 0x7fbec803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbedbfff640 10 monclient: get_auth_request con 0x7fbec80392a0 auth_method 0 2023-10-30T12:02:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbed8ff9640 10 client.44132.objecter ms_handle_connect 0x7fbec80392a0 2023-10-30T12:02:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.033+0000 7fbedbfff640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbec80392a0 0x7fbec803b760 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7fbed0006fd0 tx=0x7fbed0006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.037+0000 7fbed8ff9640 1 -- 172.21.15.50:0/3013797668 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbed40196d0 con 0x7fbedc0a7000 2023-10-30T12:02:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.038+0000 7fbed8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.038+0000 7fbed8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.282+0000 7fbee29d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:02:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.282+0000 7fbee29d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.282+0000 7fbee29d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.282+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbedc09a650 con 0x7fbedc0a7000 2023-10-30T12:02:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.282+0000 7fbed8ff9640 1 -- 172.21.15.50:0/3013797668 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fbed4017620 con 0x7fbedc0a7000 2023-10-30T12:02:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.282+0000 7fbed8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.282+0000 7fbed8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:42.286 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:02:42.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 10 librados: watch_flush enter 2023-10-30T12:02:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 10 librados: watch_flush exit 2023-10-30T12:02:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 20 client.44132.objecter shutdown clearing up homeless session... 2023-10-30T12:02:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 10 client.44132.objecter successfully canceled tick 2023-10-30T12:02:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbec80392a0 msgr2=0x7fbec803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:42.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbec80392a0 0x7fbec803b760 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7fbed0006fd0 tx=0x7fbed0006d80 comp rx=0 tx=0).stop 2023-10-30T12:02:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 10 monclient: shutdown 2023-10-30T12:02:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbedc0a7000 msgr2=0x7fbedc09efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbedc0a7000 0x7fbedc09efb0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fbed400cae0 tx=0x7fbed400a060 comp rx=0 tx=0).stop 2023-10-30T12:02:42.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 shutdown_connections 2023-10-30T12:02:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbec80392a0 0x7fbec803b760 unknown :-1 s=CLOSED pgs=3833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbedc0a7000 0x7fbedc09efb0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbedc0a4460 0x7fbedc09ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 --2- 172.21.15.50:0/3013797668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbedc06abe0 0x7fbedc09e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:42.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.284+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 >> 172.21.15.50:0/3013797668 conn(0x7fbedc066000 msgr2=0x7fbedc131b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.285+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 shutdown_connections 2023-10-30T12:02:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.285+0000 7fbee29d0640 1 -- 172.21.15.50:0/3013797668 wait complete. 2023-10-30T12:02:42.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:42.285+0000 7fbee29d0640 1 librados: shutdown 2023-10-30T12:02:45.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:02:45.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.412+0000 7f22a2e39640 10 monclient: get_monmap_and_config 2023-10-30T12:02:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.412+0000 7f22a2e39640 10 monclient: build_initial_monmap 2023-10-30T12:02:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.412+0000 7f22a2e39640 10 monclient: monmap: 2023-10-30T12:02:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:45.413236+0000 2023-10-30T12:02:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:45.413236+0000 2023-10-30T12:02:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:45.416 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:45.416 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:45.416 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:45.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:45.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.412+0000 7f22a2e39640 1 Processor -- start 2023-10-30T12:02:45.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.412+0000 7f22a2e39640 1 -- start start 2023-10-30T12:02:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.412+0000 7f22a2e39640 10 monclient: init 2023-10-30T12:02:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T12:02:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient(hunting): picked mon.noname-c con 0x7f229c0a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f229c099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient(hunting): picked mon.noname-b con 0x7f229c132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0a4f40 con 0x7f229c099a40 2023-10-30T12:02:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c0a3d70 con 0x7f229c0a4b60 2023-10-30T12:02:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0a4150 con 0x7f229c132be0 2023-10-30T12:02:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a2e39640 10 monclient(hunting): authenticate will time out at 1010.596550s 2023-10-30T12:02:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f229bfff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f229c099a40 legacy=0x7f229c0a2350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50548/0 (socket says 172.21.15.50:50548) 2023-10-30T12:02:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 learned_addr learned my addr 172.21.15.50:0/1647472675 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 10 monclient(hunting): get_auth_request con 0x7f229c132be0 auth_method 0 2023-10-30T12:02:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request con 0x7f229c0a4b60 auth_method 0 2023-10-30T12:02:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:02:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:02:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.413+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:45.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> v1:172.21.15.50:6789/0 conn(0x7f229c099a40 legacy=0x7f229c0a2350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:45.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 10 monclient(hunting): get_auth_request con 0x7f229c132be0 auth_method 0 2023-10-30T12:02:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:02:45.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:45.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:45.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:02:45.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:45.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request con 0x7f229c0a4b60 auth_method 0 2023-10-30T12:02:45.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:45.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:45.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:45.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:02:45.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:45.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:45.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:45.614+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:02:46.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> v1:172.21.15.50:6789/0 conn(0x7f229c099a40 legacy=0x7f229c0a2350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:46.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:46.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 10 monclient(hunting): get_auth_request con 0x7f229c132be0 auth_method 0 2023-10-30T12:02:46.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:46.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:46.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:46.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:02:46.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:46.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:46.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:02:46.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:46.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request con 0x7f229c0a4b60 auth_method 0 2023-10-30T12:02:46.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:46.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:46.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:46.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:02:46.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:46.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:46.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.015+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:02:46.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> v1:172.21.15.50:6789/0 conn(0x7f229c099a40 legacy=0x7f229c0a2350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:46.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 10 monclient(hunting): get_auth_request con 0x7f229c132be0 auth_method 0 2023-10-30T12:02:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:46.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:02:46.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:46.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:46.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:02:46.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:46.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request con 0x7f229c0a4b60 auth_method 0 2023-10-30T12:02:46.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:46.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a0bd5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.816+0000 7f22a0bd5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.817+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:02:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.817+0000 7f22a0bd5640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:02:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.817+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:02:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:46.817+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:02:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient(hunting): tick 2023-10-30T12:02:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:02:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 monclient(hunting): continuing hunt 2023-10-30T12:02:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:02:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 msgr2=0x7f229c0a2a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f229c132be0 0x7f229c0a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 msgr2=0x7f229c1326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.118:3300/0 conn(0x7f229c0a4b60 0x7f229c1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 -- 172.21.15.50:0/1647472675 >> v1:172.21.15.50:6789/0 conn(0x7f229c099a40 legacy=0x7f229c0a2350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient: _add_conns ranks=[1,3] 2023-10-30T12:02:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22800036c0 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 0x7f2280009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f22800073f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:02:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 -- 172.21.15.50:0/1647472675 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c09a440 con 0x7f22800036c0 2023-10-30T12:02:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 1 -- 172.21.15.50:0/1647472675 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c0a4150 con 0x7f22800073f0 2023-10-30T12:02:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229a7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 msgr2=0x7f2280009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.413+0000 7f229bfff640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 0x7f2280009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.614+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 msgr2=0x7f2280009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:48.614+0000 7f229bfff640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 0x7f2280009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:02:49.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.015+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 msgr2=0x7f2280009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:49.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.015+0000 7f229bfff640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 0x7f2280009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:02:49.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.734+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769751583 0 0) 0x7f229c09a440 con 0x7f22800036c0 2023-10-30T12:02:49.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.734+0000 7f229b7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:02:49.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.734+0000 7f229b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:49.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.734+0000 7f229b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:49.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.734+0000 7f229b7fe640 10 monclient(hunting): my global_id is 27685 2023-10-30T12:02:49.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.734+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c003520 con 0x7f22800036c0 2023-10-30T12:02:49.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1280574184 0 0) 0x7f228c003520 con 0x7f22800036c0 2023-10-30T12:02:49.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 msgr2=0x7f2280009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:49.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 1 --2- 172.21.15.50:0/1647472675 >> v2:172.21.15.50:3300/0 conn(0x7f22800073f0 0x7f2280009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:02:49.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:02:49.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:49.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:02:49.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c0a4f40 con 0x7f22800036c0 2023-10-30T12:02:49.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:02:49.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:02:49.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:49.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f229000d580 con 0x7f22800036c0 2023-10-30T12:02:49.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:49.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:02:49.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.755+0000 7f229b7fe640 10 monclient: dump: 2023-10-30T12:02:49.762 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:49.762 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:49.762 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.762 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.763 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:49.763 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:49.763 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:49.763 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:49.763 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:49.764 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:02:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 >> v1:172.21.15.118:6789/0 conn(0x7f22800036c0 legacy=0x7f2280005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 reap_dead start 2023-10-30T12:02:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:02:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c003970 0x7f228c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): picked mon.b con 0x7f228c003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f228c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): picked mon.c con 0x7f228c0047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:49.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c0052a0 0x7f228c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:49.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): picked mon.a con 0x7f228c0052a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:49.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0a4150 con 0x7f228c0052a0 2023-10-30T12:02:49.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:49.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c00c1f0 con 0x7f228c003970 2023-10-30T12:02:49.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:49.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c00c3b0 con 0x7f228c0047c0 2023-10-30T12:02:49.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:49.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229b7fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:02:49.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a13d6640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c0052a0 msgr2=0x7f228c00b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:49.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a13d6640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c0052a0 0x7f228c00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:49.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f228c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:49.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): get_auth_request con 0x7f228c0047c0 auth_method 0 2023-10-30T12:02:49.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:49.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:49.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:49.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c003970 0x7f228c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:49.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request con 0x7f228c003970 auth_method 0 2023-10-30T12:02:49.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:49.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:49.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:02:49.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:49.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:49.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:02:49.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:49.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:49.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a0bd5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:02:49.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient(hunting): handle_auth_done global_id 27685 payload 293 2023-10-30T12:02:49.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c003970 msgr2=0x7f228c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:49.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c003970 0x7f228c003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c0052a0 msgr2=0x7f228c00b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c0052a0 0x7f228c00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 1 monclient: found mon.c 2023-10-30T12:02:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:49.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:49.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 1 -- 172.21.15.50:0/1647472675 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c003520 con 0x7f228c0047c0 2023-10-30T12:02:49.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient: _finish_auth 0 2023-10-30T12:02:49.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:02:49.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f229bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a2e39640 5 monclient: authenticate success, global_id 27685 2023-10-30T12:02:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.756+0000 7f22a2e39640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.757+0000 7f229bfff640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f228c004c40 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f229c0a5820 tx=0x7f228c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2684df2f20dac07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:49.803 INFO:tasks.workunit.client.0.smithi118.stdout:fier_gid 34505 2023-10-30T12:02:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: timed out 2023-10-30T12:02:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: flushing 2023-10-30T12:02:49.803 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: flushed 2023-10-30T12:02:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3005 ms) 2023-10-30T12:02:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-10-30T12:02:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: trying... 2023-10-30T12:02:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976087587472 notify_id 137438953478 notifier_gid 34505 2023-10-30T12:02:49.804 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: timed out 2023-10-30T12:02:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: flushing 2023-10-30T12:02:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: flushed 2023-10-30T12:02:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3006 ms) 2023-10-30T12:02:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-10-30T12:02:49.805 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: List watches 2023-10-30T12:02:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: notify2 2023-10-30T12:02:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976087588512 notify_id 150323855365 notifier_gid 34505 2023-10-30T12:02:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: notify2 done 2023-10-30T12:02:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: watch_check 2023-10-30T12:02:49.806 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: unwatch2 2023-10-30T12:02:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: flushing 2023-10-30T12:02:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: done 2023-10-30T12:02:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3008 ms) 2023-10-30T12:02:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-10-30T12:02:49.807 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: List watches 2023-10-30T12:02:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: notify2 2023-10-30T12:02:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: handle_notify cookie 93976087588512 notify_id 163208757254 notifier_gid 34505 2023-10-30T12:02:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: notify2 done 2023-10-30T12:02:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: watch_check 2023-10-30T12:02:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: unwatch2 2023-10-30T12:02:49.808 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: flushing 2023-10-30T12:02:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: done 2023-10-30T12:02:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3013 ms) 2023-10-30T12:02:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16041 ms total) 2023-10-30T12:02:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: 2023-10-30T12:02:49.809 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-10-30T12:02:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (55464 ms total) 2023-10-30T12:02:49.810 INFO:tasks.workunit.client.0.smithi118.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-10-30T12:02:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228802ccc0 con 0x7f228c0047c0 2023-10-30T12:02:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 10 monclient: dump: 2023-10-30T12:02:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:49.815 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:49.815 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:49.815 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:49.815 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:49.815 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228802ce20 con 0x7f228c0047c0 2023-10-30T12:02:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 1 -- 172.21.15.50:0/1647472675 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22880356b0 con 0x7f228c0047c0 2023-10-30T12:02:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f22a2e39640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f22a2e39640 4 set_mon_vals no callback set 2023-10-30T12:02:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.758+0000 7f229b7fe640 10 monclient: dump: 2023-10-30T12:02:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:49.819 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:49.819 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:49.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 10 monclient: shutdown 2023-10-30T12:02:49.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:49.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 -- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 msgr2=0x7f228c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f228c004c40 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f229c0a5820 tx=0x7f228c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 -- 172.21.15.50:0/1647472675 shutdown_connections 2023-10-30T12:02:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c0052a0 0x7f228c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f228c004c40 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 --2- 172.21.15.50:0/1647472675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c003970 0x7f228c003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 -- 172.21.15.50:0/1647472675 >> 172.21.15.50:0/1647472675 conn(0x7f229c0a8060 msgr2=0x7f229c0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 -- 172.21.15.50:0/1647472675 shutdown_connections 2023-10-30T12:02:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.759+0000 7f22a2e39640 1 -- 172.21.15.50:0/1647472675 wait complete. 2023-10-30T12:02:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 10 monclient: build_initial_monmap 2023-10-30T12:02:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 10 monclient: monmap: 2023-10-30T12:02:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:49.823 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 1 librados: starting msgr at 2023-10-30T12:02:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 1 librados: starting objecter 2023-10-30T12:02:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 1 Processor -- start 2023-10-30T12:02:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 1 -- start start 2023-10-30T12:02:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 1 librados: setting wanted keys 2023-10-30T12:02:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 1 librados: calling monclient init 2023-10-30T12:02:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.760+0000 7f22a2e39640 10 monclient: init 2023-10-30T12:02:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:02:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c003970 0x7f229c1c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): picked mon.noname-a con 0x7f228c003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f229c131480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): picked mon.noname-c con 0x7f228c0047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0052a0 0x7f229c1319c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): picked mon.noname-b con 0x7f228c0052a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c1cac60 con 0x7f228c003970 2023-10-30T12:02:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c11e4f0 con 0x7f228c0052a0 2023-10-30T12:02:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0a18d0 con 0x7f228c0047c0 2023-10-30T12:02:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a2e39640 10 monclient(hunting): authenticate will time out at 1014.944492s 2023-10-30T12:02:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a0bd5640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c003970 msgr2=0x7f229c1c9f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a0bd5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c003970 0x7f229c1c9f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f229c131480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f229c131480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60118/0 (socket says 172.21.15.50:60118) 2023-10-30T12:02:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 1 -- 172.21.15.50:0/4009902443 learned_addr learned my addr 172.21.15.50:0/4009902443 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 10 monclient(hunting): get_auth_request con 0x7f228c0047c0 auth_method 0 2023-10-30T12:02:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0052a0 0x7f229c1319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 10 monclient(hunting): get_auth_request con 0x7f228c0052a0 auth_method 0 2023-10-30T12:02:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f229bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.761+0000 7f22a13d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 10 monclient(hunting): handle_auth_done global_id 44135 payload 824 2023-10-30T12:02:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 1 -- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0052a0 msgr2=0x7f229c1319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0052a0 0x7f229c1319c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 1 -- 172.21.15.50:0/4009902443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c003970 msgr2=0x7f229c1c9f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:02:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c003970 0x7f229c1c9f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:02:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 1 monclient: found mon.noname-c 2023-10-30T12:02:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 1 -- 172.21.15.50:0/4009902443 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c11f5b0 con 0x7f228c0047c0 2023-10-30T12:02:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a13d6640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0052a0 0x7f229c1319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:02:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229bfff640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f229c131480 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f2288003980 tx=0x7f2288005f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 client.?.objecter ms_handle_connect 0x7f228c0047c0 2023-10-30T12:02:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 1 -- 172.21.15.50:0/4009902443 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288039070 con 0x7f228c0047c0 2023-10-30T12:02:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: dump: 2023-10-30T12:02:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:49.836 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:49.836 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:49.836 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:49.836 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:49.836 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: _finish_auth 0 2023-10-30T12:02:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: _check_auth_tickets 2023-10-30T12:02:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 1 -- 172.21.15.50:0/4009902443 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2288040420 con 0x7f228c0047c0 2023-10-30T12:02:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 1 -- 172.21.15.50:0/4009902443 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228802ccc0 con 0x7f228c0047c0 2023-10-30T12:02:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 5 monclient: authenticate success, global_id 44135 2023-10-30T12:02:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 10 monclient: _renew_subs 2023-10-30T12:02:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c11f890 con 0x7f228c0047c0 2023-10-30T12:02:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22997fa640 10 monclient: dump: 2023-10-30T12:02:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:49.841 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f229a7fc640 4 set_mon_vals no callback set 2023-10-30T12:02:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 10 client.44135.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 10 monclient: _renew_subs 2023-10-30T12:02:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c131f00 con 0x7f228c0047c0 2023-10-30T12:02:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.762+0000 7f22a2e39640 1 librados: init done 2023-10-30T12:02:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22997fa640 1 -- 172.21.15.50:0/4009902443 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2288032070 con 0x7f228c0047c0 2023-10-30T12:02:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22997fa640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2280039840 0x7f228003bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22997fa640 1 -- 172.21.15.50:0/4009902443 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 9366+0+0 (secure 0 0 0) 0x7f228803c040 con 0x7f228c0047c0 2023-10-30T12:02:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22997fa640 10 client.44135.objecter ms_dispatch 0x7f229c000b90 osd_map(49..49 src has 1..49) v4 2023-10-30T12:02:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22997fa640 3 client.44135.objecter handle_osd_map got epochs [49,49] > 0 2023-10-30T12:02:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22997fa640 3 client.44135.objecter handle_osd_map decoding full epoch 49 2023-10-30T12:02:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22997fa640 20 client.44135.objecter dump_active .. 0 homeless 2023-10-30T12:02:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.763+0000 7f22a2e39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.764+0000 7f22a2e39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.764+0000 7f22a2e39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.764+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c04e160 con 0x7f228c0047c0 2023-10-30T12:02:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.764+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2280039840 0x7f228003bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.764+0000 7f22a0bd5640 10 monclient: get_auth_request con 0x7f2280039840 auth_method 0 2023-10-30T12:02:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.764+0000 7f22a0bd5640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2280039840 0x7f228003bd00 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7f2290006e50 tx=0x7f2290008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:49.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.764+0000 7f22997fa640 10 client.44135.objecter ms_handle_connect 0x7f2280039840 2023-10-30T12:02:49.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.769+0000 7f22997fa640 1 -- 172.21.15.50:0/4009902443 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2288045070 con 0x7f228c0047c0 2023-10-30T12:02:49.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.769+0000 7f22997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:49.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:49.769+0000 7f22997fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:50.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.043+0000 7f22a2e39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:02:50.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.043+0000 7f22a2e39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.043+0000 7f22a2e39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.043+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f229c0745c0 con 0x7f228c0047c0 2023-10-30T12:02:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.044+0000 7f22997fa640 1 -- 172.21.15.50:0/4009902443 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f228803e320 con 0x7f228c0047c0 2023-10-30T12:02:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.044+0000 7f22997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:02:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.044+0000 7f22997fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:50.047 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:02:50.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 10 librados: watch_flush enter 2023-10-30T12:02:50.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 10 librados: watch_flush exit 2023-10-30T12:02:50.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 20 client.44135.objecter shutdown clearing up homeless session... 2023-10-30T12:02:50.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 10 client.44135.objecter successfully canceled tick 2023-10-30T12:02:50.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2280039840 msgr2=0x7f228003bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2280039840 0x7f228003bd00 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7f2290006e50 tx=0x7f2290008040 comp rx=0 tx=0).stop 2023-10-30T12:02:50.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 10 monclient: shutdown 2023-10-30T12:02:50.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:50.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 msgr2=0x7f229c131480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f229c131480 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f2288003980 tx=0x7f2288005f20 comp rx=0 tx=0).stop 2023-10-30T12:02:50.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 shutdown_connections 2023-10-30T12:02:50.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2280039840 0x7f228003bd00 unknown :-1 s=CLOSED pgs=3834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f228c0052a0 0x7f229c1319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f228c0047c0 0x7f229c131480 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 --2- 172.21.15.50:0/4009902443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f228c003970 0x7f229c1c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 >> 172.21.15.50:0/4009902443 conn(0x7f229c0a8060 msgr2=0x7f229c135c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:50.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 shutdown_connections 2023-10-30T12:02:50.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 -- 172.21.15.50:0/4009902443 wait complete. 2023-10-30T12:02:50.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.046+0000 7f22a2e39640 1 librados: shutdown 2023-10-30T12:02:50.058 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:02:50.059 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:02:50.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:02:50.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.183+0000 7f66e4300640 10 monclient: get_monmap_and_config 2023-10-30T12:02:50.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.183+0000 7f66e4300640 10 monclient: build_initial_monmap 2023-10-30T12:02:50.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.183+0000 7f66e4300640 10 monclient: monmap: 2023-10-30T12:02:50.186 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:50.186 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.186 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:50.184293+0000 2023-10-30T12:02:50.186 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:50.184293+0000 2023-10-30T12:02:50.186 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:50.187 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.187 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:50.187 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:50.187 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:50.188 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:50.188 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:50.188 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.183+0000 7f66e4300640 1 Processor -- start 2023-10-30T12:02:50.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.183+0000 7f66e4300640 1 -- start start 2023-10-30T12:02:50.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.183+0000 7f66e4300640 10 monclient: init 2023-10-30T12:02:50.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:50.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:02:50.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66dc0a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:50.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66dc0a2450 addr v1:172.21.15.118:6789/0 2023-10-30T12:02:50.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc13e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient(hunting): picked mon.noname-b con 0x7f66dc13c120 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:50.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc0a4190 con 0x7f66dc0a2ae0 2023-10-30T12:02:50.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc09a2e0 con 0x7f66dc0a2450 2023-10-30T12:02:50.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc0a1630 con 0x7f66dc13c120 2023-10-30T12:02:50.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:50.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e209c640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f66dc0a2ae0 legacy=0x7f66dc0a2ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e4300640 10 monclient(hunting): authenticate will time out at 1015.367523s 2023-10-30T12:02:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc13e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60122/0 (socket says 172.21.15.50:60122) 2023-10-30T12:02:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 1 -- 172.21.15.50:0/835674127 learned_addr learned my addr 172.21.15.50:0/835674127 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): get_auth_request con 0x7f66dc13c120 auth_method 0 2023-10-30T12:02:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient(hunting): handle_auth_done global_id 44144 payload 274 2023-10-30T12:02:50.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 1 -- 172.21.15.50:0/835674127 >> v1:172.21.15.118:6789/0 conn(0x7f66dc0a2450 legacy=0x7f66dc13aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 1 -- 172.21.15.50:0/835674127 >> v1:172.21.15.50:6789/0 conn(0x7f66dc0a2ae0 legacy=0x7f66dc0a2ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:50.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient: _finish_hunting 0 2023-10-30T12:02:50.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 1 monclient: found mon.noname-b 2023-10-30T12:02:50.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:50.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.184+0000 7f66e289d640 1 -- 172.21.15.50:0/835674127 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc09a8a0 con 0x7f66dc13c120 2023-10-30T12:02:50.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e289d640 1 --2- 172.21.15.50:0/835674127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc13e510 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009e80 tx=0x7f66d402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=167b8c8bb8a29d2f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 1 -- 172.21.15.50:0/835674127 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d402c9c0 con 0x7f66dc13c120 2023-10-30T12:02:50.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:02:50.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: dump: 2023-10-30T12:02:50.207 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.207 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.207 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.207 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.207 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.208 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.208 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.208 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.208 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.208 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: _finish_auth 0 2023-10-30T12:02:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: _check_auth_tickets 2023-10-30T12:02:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e4300640 5 monclient: authenticate success, global_id 44144 2023-10-30T12:02:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e4300640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:50.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 1 -- 172.21.15.50:0/835674127 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d402cb20 con 0x7f66dc13c120 2023-10-30T12:02:50.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:50.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 1 -- 172.21.15.50:0/835674127 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d40343f0 con 0x7f66dc13c120 2023-10-30T12:02:50.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e4300640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:50.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e4300640 4 set_mon_vals no callback set 2023-10-30T12:02:50.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:50.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.185+0000 7f66e109a640 10 monclient: dump: 2023-10-30T12:02:50.211 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.211 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.212 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.212 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.212 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.212 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.212 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.213 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.213 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.213 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 10 monclient: shutdown 2023-10-30T12:02:50.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 1 -- 172.21.15.50:0/835674127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 msgr2=0x7f66dc13e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 1 --2- 172.21.15.50:0/835674127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc13e510 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009e80 tx=0x7f66d402be20 comp rx=0 tx=0).stop 2023-10-30T12:02:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 1 -- 172.21.15.50:0/835674127 shutdown_connections 2023-10-30T12:02:50.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 1 --2- 172.21.15.50:0/835674127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc13e510 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 1 -- 172.21.15.50:0/835674127 >> 172.21.15.50:0/835674127 conn(0x7f66dc0a80b0 msgr2=0x7f66dc0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 1 -- 172.21.15.50:0/835674127 shutdown_connections 2023-10-30T12:02:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.186+0000 7f66e4300640 1 -- 172.21.15.50:0/835674127 wait complete. 2023-10-30T12:02:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient: build_initial_monmap 2023-10-30T12:02:50.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient: monmap: 2023-10-30T12:02:50.216 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:50.216 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:50.216 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:50.216 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:50.216 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:50.217 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 librados: starting msgr at 2023-10-30T12:02:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 librados: starting objecter 2023-10-30T12:02:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 Processor -- start 2023-10-30T12:02:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 -- start start 2023-10-30T12:02:50.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 librados: setting wanted keys 2023-10-30T12:02:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 librados: calling monclient init 2023-10-30T12:02:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient: init 2023-10-30T12:02:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:02:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66dc0a2450 0x7f66dc1ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): picked mon.noname-a con 0x7f66dc0a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:50.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66dc0a2ae0 0x7f66dc1cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): picked mon.noname-b con 0x7f66dc0a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:50.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc09c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): picked mon.noname-c con 0x7f66dc13c120 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:50.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc09ee70 con 0x7f66dc0a2450 2023-10-30T12:02:50.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc13b940 con 0x7f66dc0a2ae0 2023-10-30T12:02:50.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc0a1790 con 0x7f66dc13c120 2023-10-30T12:02:50.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:50.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e209c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66dc0a2450 msgr2=0x7f66dc1ca740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:50.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e4300640 10 monclient(hunting): authenticate will time out at 1015.370963s 2023-10-30T12:02:50.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.187+0000 7f66e209c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66dc0a2450 0x7f66dc1ca740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:50.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc09c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc09c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60132/0 (socket says 172.21.15.50:60132) 2023-10-30T12:02:50.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 -- 172.21.15.50:0/3110106480 learned_addr learned my addr 172.21.15.50:0/3110106480 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:50.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): get_auth_request con 0x7f66dc13c120 auth_method 0 2023-10-30T12:02:50.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e189b640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66dc0a2ae0 0x7f66dc1cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e189b640 10 monclient(hunting): get_auth_request con 0x7f66dc0a2ae0 auth_method 0 2023-10-30T12:02:50.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e189b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e189b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e189b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient(hunting): handle_auth_done global_id 44147 payload 824 2023-10-30T12:02:50.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66dc0a2ae0 msgr2=0x7f66dc1cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66dc0a2ae0 0x7f66dc1cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66dc0a2450 msgr2=0x7f66dc1ca740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:02:50.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66dc0a2450 0x7f66dc1ca740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient: _finish_hunting 0 2023-10-30T12:02:50.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 monclient: found mon.noname-c 2023-10-30T12:02:50.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:50.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 -- 172.21.15.50:0/3110106480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4009b30 con 0x7f66dc13c120 2023-10-30T12:02:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e189b640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66dc0a2ae0 0x7f66dc1cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:02:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e289d640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc09c820 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f66d4036040 tx=0x7f66d4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 client.?.objecter ms_handle_connect 0x7f66dc13c120 2023-10-30T12:02:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d403e450 con 0x7f66dc13c120 2023-10-30T12:02:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: dump: 2023-10-30T12:02:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.231 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.231 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.231 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.231 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:02:50.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:02:50.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:50.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d403e5d0 con 0x7f66dc13c120 2023-10-30T12:02:50.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:50.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d403e980 con 0x7f66dc13c120 2023-10-30T12:02:50.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 5 monclient: authenticate success, global_id 44147 2023-10-30T12:02:50.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 10 monclient: _renew_subs 2023-10-30T12:02:50.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e0899640 4 set_mon_vals no callback set 2023-10-30T12:02:50.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc099bc0 con 0x7f66dc13c120 2023-10-30T12:02:50.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:50.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66d2ffd640 10 monclient: dump: 2023-10-30T12:02:50.234 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.235 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.235 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.235 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.235 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.235 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.236 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.236 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.236 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.236 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 10 client.44147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:50.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 10 monclient: _renew_subs 2023-10-30T12:02:50.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc0a07d0 con 0x7f66dc13c120 2023-10-30T12:02:50.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.188+0000 7f66e4300640 1 librados: init done 2023-10-30T12:02:50.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e4300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:50.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f66d403eae0 con 0x7f66dc13c120 2023-10-30T12:02:50.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e4300640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d520 0x7f66dc02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e4300640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc0a0960 con 0x7f66dc09d520 2023-10-30T12:02:50.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d520 0x7f66dc02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient: get_auth_request con 0x7f66dc09d520 auth_method 0 2023-10-30T12:02:50.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66d2ffd640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66c00392a0 0x7f66c003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8594+0+0 (secure 0 0 0) 0x7f66d406e190 con 0x7f66dc13c120 2023-10-30T12:02:50.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66d2ffd640 10 client.44147.objecter ms_dispatch 0x7f66dc000b90 osd_map(51..51 src has 1..51) v4 2023-10-30T12:02:50.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66d2ffd640 3 client.44147.objecter handle_osd_map got epochs [51,51] > 0 2023-10-30T12:02:50.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66d2ffd640 3 client.44147.objecter handle_osd_map decoding full epoch 51 2023-10-30T12:02:50.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 10 monclient(hunting): handle_auth_done global_id 44150 payload 824 2023-10-30T12:02:50.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e209c640 1 -- 172.21.15.50:0/3110106480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66dc09a5c0 con 0x7f66dc09d520 2023-10-30T12:02:50.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e189b640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66c00392a0 0x7f66c003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66d2ffd640 20 client.44147.objecter dump_active .. 0 homeless 2023-10-30T12:02:50.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.189+0000 7f66e189b640 10 monclient: get_auth_request con 0x7f66c00392a0 auth_method 0 2023-10-30T12:02:50.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66e209c640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d520 0x7f66dc02b430 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f66d8003980 tx=0x7f66d800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 client.44147.objecter ms_handle_connect 0x7f66dc09d520 2023-10-30T12:02:50.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 client.44147.objecter resend_mon_ops 2023-10-30T12:02:50.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8003e10 con 0x7f66dc09d520 2023-10-30T12:02:50.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 monclient: dump: 2023-10-30T12:02:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.246 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 client.44147.objecter ms_handle_connect 0x7f66c00392a0 2023-10-30T12:02:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66e189b640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66c00392a0 0x7f66c003b760 secure :-1 s=READY pgs=3837 cs=0 l=1 rev1=1 crypto rx=0x7f66cc007fd0 tx=0x7f66cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f66d8005070 con 0x7f66dc09d520 2023-10-30T12:02:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d520 msgr2=0x7f66dc02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d520 0x7f66dc02b430 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f66d8003980 tx=0x7f66d800b8c0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d520 msgr2=0x7f66dc02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.190+0000 7f66d2ffd640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d520 0x7f66dc02b430 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e4300640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:50.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e4300640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d900 0x7f66dc02b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e4300640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc073ff0 con 0x7f66dc09d900 2023-10-30T12:02:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d900 0x7f66dc02b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 10 monclient: get_auth_request con 0x7f66dc09d900 auth_method 0 2023-10-30T12:02:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.196+0000 7f66e209c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e209c640 10 monclient(hunting): handle_auth_done global_id 44153 payload 824 2023-10-30T12:02:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e209c640 1 -- 172.21.15.50:0/3110106480 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f66dc0a28a0 con 0x7f66dc09d900 2023-10-30T12:02:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 10 client.44147.objecter ms_handle_connect 0x7f66dc09d900 2023-10-30T12:02:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 10 client.44147.objecter resend_mon_ops 2023-10-30T12:02:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e209c640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d900 0x7f66dc02b970 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f66d8012040 tx=0x7f66d8005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8020070 con 0x7f66dc09d900 2023-10-30T12:02:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 10 monclient: dump: 2023-10-30T12:02:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.254 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.254 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f66d8010400 con 0x7f66dc09d900 2023-10-30T12:02:50.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:50.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d900 msgr2=0x7f66dc02b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d900 0x7f66dc02b970 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f66d8012040 tx=0x7f66d8005a20 comp rx=0 tx=0).stop 2023-10-30T12:02:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d900 msgr2=0x7f66dc02b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:50.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66d2ffd640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc09d900 0x7f66dc02b970 secure :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f66d8012040 tx=0x7f66d8005a20 comp rx=0 tx=0).stop 2023-10-30T12:02:50.256 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:02:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 22, 2023-10-30T12:02:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:02:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:50.262 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:50.262 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:50.262 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:50.262 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:50.262 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:50.262 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:50.263 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.263 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:50.263 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:50.263 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:50.265 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:50.265 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:50.265 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:50.265 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:50.265 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:50.266 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:50.266 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:50.266 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:50.266 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:50.266 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:50.266 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:50.267 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:50.267 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:50.267 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:50.267 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:50.267 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:50.267 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:50.268 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:50.268 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.268 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:50.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.268 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:50.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.269 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:50.269 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:50.269 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:50.269 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:50.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.270 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:50.270 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:50.270 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.271 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:50.271 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:50.271 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.272 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:50.272 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:50.272 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:50.272 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:50.272 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:50.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.273 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:50.273 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:50.273 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:50.273 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.275 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:50.275 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:50.275 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.275 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.275 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.276 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:50.276 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:50.276 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:50.276 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:50.276 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:50.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.277 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:50.277 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:50.277 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:50.278 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:50.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.278 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:50.278 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:50.278 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.279 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:50.279 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:50.279 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.279 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.280 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.280 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:50.280 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:50.280 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:50.280 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:50.281 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:50.281 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.281 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.281 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:50.281 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:50.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.282 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:50.282 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:50.282 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:50.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.283 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.283 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:02:50.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.283 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:50.283 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:50.283 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:02:50.284 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.284 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:50.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.284 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:50.285 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:50.285 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:02:50.285 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.285 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.285 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:50.286 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:50.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 10 librados: watch_flush enter 2023-10-30T12:02:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 10 librados: watch_flush exit 2023-10-30T12:02:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 20 client.44147.objecter shutdown clearing up homeless session... 2023-10-30T12:02:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 10 client.44147.objecter successfully canceled tick 2023-10-30T12:02:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66c00392a0 msgr2=0x7f66c003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66c00392a0 0x7f66c003b760 secure :-1 s=READY pgs=3837 cs=0 l=1 rev1=1 crypto rx=0x7f66cc007fd0 tx=0x7f66cc009040 comp rx=0 tx=0).stop 2023-10-30T12:02:50.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 10 monclient: shutdown 2023-10-30T12:02:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 msgr2=0x7f66dc09c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e289d640 1 -- 172.21.15.50:0/3110106480 reap_dead start 2023-10-30T12:02:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc09c820 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f66d4036040 tx=0x7f66d4004a40 comp rx=0 tx=0).stop 2023-10-30T12:02:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 shutdown_connections 2023-10-30T12:02:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 1 --2- 172.21.15.50:0/3110106480 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66dc13c120 0x7f66dc09c820 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.197+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 >> 172.21.15.50:0/3110106480 conn(0x7f66dc0a80b0 msgr2=0x7f66dc133c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.198+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 shutdown_connections 2023-10-30T12:02:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.198+0000 7f66e4300640 1 -- 172.21.15.50:0/3110106480 wait complete. 2023-10-30T12:02:50.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.198+0000 7f66e4300640 1 librados: shutdown 2023-10-30T12:02:50.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:02:50.291 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:02:50.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:02:50.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.396+0000 7f0513e4b640 10 monclient: get_monmap_and_config 2023-10-30T12:02:50.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.396+0000 7f0513e4b640 10 monclient: build_initial_monmap 2023-10-30T12:02:50.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.396+0000 7f0513e4b640 10 monclient: monmap: 2023-10-30T12:02:50.398 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:50.398 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.399 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:02:50.397525+0000 2023-10-30T12:02:50.399 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:02:50.397525+0000 2023-10-30T12:02:50.399 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:50.399 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.399 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:02:50.400 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:02:50.400 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:02:50.400 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:02:50.400 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:02:50.400 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 1 Processor -- start 2023-10-30T12:02:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 1 -- start start 2023-10-30T12:02:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient: init 2023-10-30T12:02:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:02:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f050c06afc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:02:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c0a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient(hunting): picked mon.noname-b con 0x7f050c06b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f050c13b070 0x7f050c13d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient(hunting): picked mon.noname-c con 0x7f050c13b070 addr v2:172.21.15.118:3300/0 2023-10-30T12:02:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c06bde0 con 0x7f050c06afc0 2023-10-30T12:02:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050c06b3a0 con 0x7f050c13b070 2023-10-30T12:02:50.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c09e540 con 0x7f050c06b640 2023-10-30T12:02:50.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:50.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0513e4b640 10 monclient(hunting): authenticate will time out at 1015.580987s 2023-10-30T12:02:50.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.397+0000 7f0511be7640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f050c06afc0 legacy=0x7f050c0a1c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:02:50.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c0a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c0a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60144/0 (socket says 172.21.15.50:60144) 2023-10-30T12:02:50.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 -- 172.21.15.50:0/3078247270 learned_addr learned my addr 172.21.15.50:0/3078247270 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:50.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): get_auth_request con 0x7f050c06b640 auth_method 0 2023-10-30T12:02:50.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 1 --2- 172.21.15.50:0/3078247270 >> v2:172.21.15.118:3300/0 conn(0x7f050c13b070 0x7f050c13d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 10 monclient(hunting): get_auth_request con 0x7f050c13b070 auth_method 0 2023-10-30T12:02:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05123e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient(hunting): handle_auth_done global_id 44156 payload 274 2023-10-30T12:02:50.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 -- 172.21.15.50:0/3078247270 >> v2:172.21.15.118:3300/0 conn(0x7f050c13b070 msgr2=0x7f050c13d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:02:50.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 --2- 172.21.15.50:0/3078247270 >> v2:172.21.15.118:3300/0 conn(0x7f050c13b070 0x7f050c13d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 -- 172.21.15.50:0/3078247270 >> v1:172.21.15.50:6789/0 conn(0x7f050c06afc0 legacy=0x7f050c0a1c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:02:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient: _finish_hunting 0 2023-10-30T12:02:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 monclient: found mon.noname-b 2023-10-30T12:02:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.398+0000 7f05113e6640 1 -- 172.21.15.50:0/3078247270 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c09fa10 con 0x7f050c06b640 2023-10-30T12:02:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f05113e6640 1 --2- 172.21.15.50:0/3078247270 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c0a23a0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009870 tx=0x7f04fc02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4427b96a0788318 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 1 -- 172.21.15.50:0/3078247270 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc031740 con 0x7f050c06b640 2023-10-30T12:02:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:02:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: dump: 2023-10-30T12:02:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.419 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: _finish_auth 0 2023-10-30T12:02:50.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: _check_auth_tickets 2023-10-30T12:02:50.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:50.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 5 monclient: authenticate success, global_id 44156 2023-10-30T12:02:50.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 1 -- 172.21.15.50:0/3078247270 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc0318a0 con 0x7f050c06b640 2023-10-30T12:02:50.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:02:50.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:50.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 1 -- 172.21.15.50:0/3078247270 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc031c30 con 0x7f050c06b640 2023-10-30T12:02:50.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 10 monclient: get_monmap_and_config success 2023-10-30T12:02:50.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 4 set_mon_vals no callback set 2023-10-30T12:02:50.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:50.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0510be5640 10 monclient: dump: 2023-10-30T12:02:50.422 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.422 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.422 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.422 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.423 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 10 monclient: shutdown 2023-10-30T12:02:50.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:50.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 1 -- 172.21.15.50:0/3078247270 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 msgr2=0x7f050c0a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 1 --2- 172.21.15.50:0/3078247270 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c0a23a0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009870 tx=0x7f04fc02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 1 -- 172.21.15.50:0/3078247270 shutdown_connections 2023-10-30T12:02:50.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 1 --2- 172.21.15.50:0/3078247270 >> v2:172.21.15.118:3300/0 conn(0x7f050c13b070 0x7f050c13d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 1 --2- 172.21.15.50:0/3078247270 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c0a23a0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.399+0000 7f0513e4b640 1 -- 172.21.15.50:0/3078247270 >> 172.21.15.50:0/3078247270 conn(0x7f050c0a6200 msgr2=0x7f050c0a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:50.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 -- 172.21.15.50:0/3078247270 shutdown_connections 2023-10-30T12:02:50.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 -- 172.21.15.50:0/3078247270 wait complete. 2023-10-30T12:02:50.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 10 monclient: build_initial_monmap 2023-10-30T12:02:50.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 10 monclient: monmap: 2023-10-30T12:02:50.426 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:02:50.426 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:02:50.426 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:02:50.426 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:02:50.427 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:02:50.427 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.427 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:02:50.427 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:02:50.427 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:02:50.428 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 librados: starting msgr at 2023-10-30T12:02:50.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 librados: starting objecter 2023-10-30T12:02:50.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 Processor -- start 2023-10-30T12:02:50.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 -- start start 2023-10-30T12:02:50.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 librados: setting wanted keys 2023-10-30T12:02:50.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 1 librados: calling monclient init 2023-10-30T12:02:50.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.400+0000 7f0513e4b640 10 monclient: init 2023-10-30T12:02:50.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient: _reopen_session rank -1 2023-10-30T12:02:50.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:02:50.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f050c06afc0 0x7f050c1ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): picked mon.noname-a con 0x7f050c06afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:02:50.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c1ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): picked mon.noname-c con 0x7f050c06b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:02:50.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c13b070 0x7f050c11c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): picked mon.noname-b con 0x7f050c13b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:02:50.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c138590 con 0x7f050c06afc0 2023-10-30T12:02:50.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c070de0 con 0x7f050c13b070 2023-10-30T12:02:50.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c1204d0 con 0x7f050c06b640 2023-10-30T12:02:50.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): _renew_subs 2023-10-30T12:02:50.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0513e4b640 10 monclient(hunting): authenticate will time out at 1015.584394s 2023-10-30T12:02:50.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0511be7640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f050c06afc0 msgr2=0x7f050c1ca470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:02:50.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f0511be7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f050c06afc0 0x7f050c1ca470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:02:50.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c1ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c1ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60160/0 (socket says 172.21.15.50:60160) 2023-10-30T12:02:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 -- 172.21.15.50:0/1429916875 learned_addr learned my addr 172.21.15.50:0/1429916875 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:02:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): get_auth_request con 0x7f050c06b640 auth_method 0 2023-10-30T12:02:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient(hunting): handle_auth_done global_id 44159 payload 824 2023-10-30T12:02:50.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c13b070 msgr2=0x7f050c11c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c13b070 0x7f050c11c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f050c06afc0 msgr2=0x7f050c1ca470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:02:50.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f050c06afc0 0x7f050c1ca470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient: _finish_hunting 0 2023-10-30T12:02:50.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 monclient: found mon.noname-c 2023-10-30T12:02:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:02:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 -- 172.21.15.50:0/1429916875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0094d0 con 0x7f050c06b640 2023-10-30T12:02:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05113e6640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c1ca9b0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009870 tx=0x7f04fc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05027fc640 10 client.?.objecter ms_handle_connect 0x7f050c06b640 2023-10-30T12:02:50.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.401+0000 7f05027fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:02:50.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc0339c0 con 0x7f050c06b640 2023-10-30T12:02:50.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:02:50.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: dump: 2023-10-30T12:02:50.437 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.438 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.438 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.438 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.438 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.438 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.439 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.439 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.439 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.439 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: _finish_auth 0 2023-10-30T12:02:50.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: _check_auth_tickets 2023-10-30T12:02:50.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:02:50.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 5 monclient: authenticate success, global_id 44159 2023-10-30T12:02:50.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc033b20 con 0x7f050c06b640 2023-10-30T12:02:50.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:02:50.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 10 monclient: _renew_subs 2023-10-30T12:02:50.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc03c400 con 0x7f050c06b640 2023-10-30T12:02:50.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c11eac0 con 0x7f050c06b640 2023-10-30T12:02:50.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0503fff640 4 set_mon_vals no callback set 2023-10-30T12:02:50.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:02:50.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 10 monclient: dump: 2023-10-30T12:02:50.442 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.442 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.442 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.443 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.443 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.443 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.443 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.443 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.444 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.444 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 10 client.44159.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:02:50.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 10 monclient: _renew_subs 2023-10-30T12:02:50.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:02:50.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c11e100 con 0x7f050c06b640 2023-10-30T12:02:50.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 1 librados: init done 2023-10-30T12:02:50.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:50.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f04fc03a040 con 0x7f050c06b640 2023-10-30T12:02:50.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11c9f0 0x7f050c11cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.402+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c073ff0 con 0x7f050c11c9f0 2023-10-30T12:02:50.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11c9f0 0x7f050c11cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient: get_auth_request con 0x7f050c11c9f0 auth_method 0 2023-10-30T12:02:50.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f00392a0 0x7f04f003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8594+0+0 (secure 0 0 0) 0x7f04fc06d050 con 0x7f050c06b640 2023-10-30T12:02:50.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 10 client.44159.objecter ms_dispatch 0x7f050c000b90 osd_map(51..51 src has 1..51) v4 2023-10-30T12:02:50.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 3 client.44159.objecter handle_osd_map got epochs [51,51] > 0 2023-10-30T12:02:50.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 3 client.44159.objecter handle_osd_map decoding full epoch 51 2023-10-30T12:02:50.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05123e8640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f00392a0 0x7f04f003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05123e8640 10 monclient: get_auth_request con 0x7f04f00392a0 auth_method 0 2023-10-30T12:02:50.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 20 client.44159.objecter dump_active .. 0 homeless 2023-10-30T12:02:50.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 10 monclient(hunting): handle_auth_done global_id 27709 payload 824 2023-10-30T12:02:50.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 1 -- 172.21.15.50:0/1429916875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c120640 con 0x7f050c11c9f0 2023-10-30T12:02:50.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05123e8640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f00392a0 0x7f04f003b760 secure :-1 s=READY pgs=3838 cs=0 l=1 rev1=1 crypto rx=0x7f050c0a4b00 tx=0x7f050400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 10 client.44159.objecter ms_handle_connect 0x7f04f00392a0 2023-10-30T12:02:50.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 10 client.44159.objecter ms_handle_connect 0x7f050c11c9f0 2023-10-30T12:02:50.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 10 client.44159.objecter resend_mon_ops 2023-10-30T12:02:50.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f0511be7640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11c9f0 0x7f050c11cdd0 secure :-1 s=READY pgs=6937 cs=0 l=1 rev1=1 crypto rx=0x7f0508003980 tx=0x7f050800b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0508003e10 con 0x7f050c11c9f0 2023-10-30T12:02:50.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.403+0000 7f05027fc640 10 monclient: dump: 2023-10-30T12:02:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.452 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.452 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.452 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.452 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.452 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.404+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0508005070 con 0x7f050c11c9f0 2023-10-30T12:02:50.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.404+0000 7f05027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:02:50.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.404+0000 7f05027fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:02:50.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.404+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11c9f0 msgr2=0x7f050c11cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.404+0000 7f05027fc640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11c9f0 0x7f050c11cdd0 secure :-1 s=READY pgs=6937 cs=0 l=1 rev1=1 crypto rx=0x7f0508003980 tx=0x7f050800b8c0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.404+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11c9f0 msgr2=0x7f050c11cdd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:50.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.404+0000 7f05027fc640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11c9f0 0x7f050c11cdd0 secure :-1 s=CLOSED pgs=6937 cs=0 l=1 rev1=1 crypto rx=0x7f0508003980 tx=0x7f050800b8c0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0513e4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:02:50.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0513e4b640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11d310 0x7f050c135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:02:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0513e4b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:02:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c11fff0 con 0x7f050c11d310 2023-10-30T12:02:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11d310 0x7f050c135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:02:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 10 monclient: get_auth_request con 0x7f050c11d310 auth_method 0 2023-10-30T12:02:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:02:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:02:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:02:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:02:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:02:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.410+0000 7f0511be7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:02:50.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f0511be7640 10 monclient(hunting): handle_auth_done global_id 27712 payload 824 2023-10-30T12:02:50.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f0511be7640 1 -- 172.21.15.50:0/1429916875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f050c09f0b0 con 0x7f050c11d310 2023-10-30T12:02:50.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 10 client.44159.objecter ms_handle_connect 0x7f050c11d310 2023-10-30T12:02:50.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 10 client.44159.objecter resend_mon_ops 2023-10-30T12:02:50.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f0511be7640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11d310 0x7f050c135260 secure :-1 s=READY pgs=6938 cs=0 l=1 rev1=1 crypto rx=0x7f0508003980 tx=0x7f0508005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:02:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0508020070 con 0x7f050c11d310 2023-10-30T12:02:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:02:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:02:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 10 monclient: dump: 2023-10-30T12:02:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:02:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:02:50.461 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.461 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:02:50.461 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:02:50.461 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:02:50.461 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:02:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:02:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:02:50.462 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:02:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4793 (secure 0 0 0) 0x7f050801bcb0 con 0x7f050c11d310 2023-10-30T12:02:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:02:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:02:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11d310 msgr2=0x7f050c135260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11d310 0x7f050c135260 secure :-1 s=READY pgs=6938 cs=0 l=1 rev1=1 crypto rx=0x7f0508003980 tx=0x7f0508005a20 comp rx=0 tx=0).stop 2023-10-30T12:02:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11d310 msgr2=0x7f050c135260 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:02:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f05027fc640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f050c11d310 0x7f050c135260 secure :-1 s=CLOSED pgs=6938 cs=0 l=1 rev1=1 crypto rx=0x7f0508003980 tx=0x7f0508005a20 comp rx=0 tx=0).stop 2023-10-30T12:02:50.464 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:02:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:02:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 22, 2023-10-30T12:02:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:02:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:02:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:02:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:02:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:02:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:02:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:02:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:02:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:02:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:02:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:02:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:02:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:02:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:02:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:02:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:02:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:02:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:02:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:02:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:02:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:02:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:02:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:02:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:02:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:02:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:02:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:02:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:02:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:02:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:02:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:02:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:02:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:02:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:02:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:02:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:02:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:02:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:02:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:02:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:02:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:02:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:02:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:02:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:02:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:02:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:02:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:02:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:02:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:02:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:02:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:02:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:02:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:02:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:02:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:02:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:02:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:02:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:02:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:02:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:02:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:02:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:02:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:02:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:02:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:02:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:02:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:02:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 11 2023-10-30T12:02:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:02:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:02:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:02:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:02:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:02:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:02:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:02:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:02:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:02:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:02:50.495 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:02:50.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f0513e4b640 10 librados: watch_flush enter 2023-10-30T12:02:50.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f0513e4b640 10 librados: watch_flush exit 2023-10-30T12:02:50.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f0513e4b640 20 client.44159.objecter shutdown clearing up homeless session... 2023-10-30T12:02:50.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.411+0000 7f0513e4b640 10 client.44159.objecter successfully canceled tick 2023-10-30T12:02:50.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f00392a0 msgr2=0x7f04f003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f00392a0 0x7f04f003b760 secure :-1 s=READY pgs=3838 cs=0 l=1 rev1=1 crypto rx=0x7f050c0a4b00 tx=0x7f050400b040 comp rx=0 tx=0).stop 2023-10-30T12:02:50.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 10 monclient: shutdown 2023-10-30T12:02:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:02:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 msgr2=0x7f050c1ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:02:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 --2- 172.21.15.50:0/1429916875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f050c06b640 0x7f050c1ca9b0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009870 tx=0x7f04fc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:02:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f05123e8640 1 -- 172.21.15.50:0/1429916875 reap_dead start 2023-10-30T12:02:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f05123e8640 1 -- 172.21.15.50:0/1429916875 reap_dead start 2023-10-30T12:02:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 shutdown_connections 2023-10-30T12:02:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 >> 172.21.15.50:0/1429916875 conn(0x7f050c0a6200 msgr2=0x7f050c137460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:02:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 shutdown_connections 2023-10-30T12:02:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 -- 172.21.15.50:0/1429916875 wait complete. 2023-10-30T12:02:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:02:50.412+0000 7f0513e4b640 1 librados: shutdown 2023-10-30T12:02:50.501 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:02:50.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:03:01.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: Running main() from gmock_main.cc 2023-10-30T12:03:01.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-10-30T12:03:01.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [----------] Global test environment set-up. 2023-10-30T12:03:01.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-10-30T12:03:01.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-10-30T12:03:01.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5877 ms) 2023-10-30T12:03:01.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-10-30T12:03:01.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3026 ms) 2023-10-30T12:03:01.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-10-30T12:03:01.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2990 ms) 2023-10-30T12:03:01.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-10-30T12:03:01.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3007 ms) 2023-10-30T12:03:01.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-10-30T12:03:01.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2953 ms) 2023-10-30T12:03:01.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-10-30T12:03:01.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5010 ms) 2023-10-30T12:03:01.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-10-30T12:03:01.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (17519 ms) 2023-10-30T12:03:01.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-10-30T12:03:01.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (18448 ms) 2023-10-30T12:03:01.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-10-30T12:03:01.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8012 ms) 2023-10-30T12:03:01.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [----------] 9 tests from LibRadosPools (66843 ms total) 2023-10-30T12:03:01.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: 2023-10-30T12:03:01.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [----------] Global test environment tear-down 2023-10-30T12:03:01.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (66843 ms total) 2023-10-30T12:03:01.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_pool: [ PASSED ] 9 tests. 2023-10-30T12:03:10.502 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:03:10.503 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:03:10.503 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:03:10.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:03:10.506 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:03:10.507 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:03:10.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.621+0000 7eff353c5640 10 monclient: get_monmap_and_config 2023-10-30T12:03:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.621+0000 7eff353c5640 10 monclient: build_initial_monmap 2023-10-30T12:03:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.621+0000 7eff353c5640 10 monclient: monmap: 2023-10-30T12:03:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:10.622214+0000 2023-10-30T12:03:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:10.622214+0000 2023-10-30T12:03:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:10.625 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:10.625 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:10.625 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:10.625 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:10.625 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:10.625 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:10.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.621+0000 7eff353c5640 1 Processor -- start 2023-10-30T12:03:10.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.621+0000 7eff353c5640 1 -- start start 2023-10-30T12:03:10.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.621+0000 7eff353c5640 10 monclient: init 2023-10-30T12:03:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T12:03:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7eff300a6070 0x7eff300a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient(hunting): picked mon.noname-c con 0x7eff300a6070 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 0x7eff30135590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient(hunting): picked mon.noname-a con 0x7eff300a6450 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff30135ad0 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3009a270 con 0x7eff30135ad0 2023-10-30T12:03:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff30099f90 con 0x7eff300a6450 2023-10-30T12:03:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3009a550 con 0x7eff300a6070 2023-10-30T12:03:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff353c5640 10 monclient(hunting): authenticate will time out at 1035.805321s 2023-10-30T12:03:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 0x7eff30135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7eff30135ad0 0x7eff3013c8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39280/0 (socket says 172.21.15.50:39280) 2023-10-30T12:03:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2f7fe640 1 -- 172.21.15.50:0/655676024 learned_addr learned my addr 172.21.15.50:0/655676024 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 10 monclient(hunting): get_auth_request con 0x7eff300a6450 auth_method 0 2023-10-30T12:03:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 1 -- 172.21.15.50:0/655676024 >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 msgr2=0x7eff30135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 1 -- 172.21.15.50:0/655676024 >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 msgr2=0x7eff30135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 1 --2- 172.21.15.50:0/655676024 >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 0x7eff30135590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2e7fc640 1 --2- 172.21.15.50:0/655676024 >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 0x7eff30135590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 1 --2- 172.21.15.50:0/655676024 >> v2:172.21.15.118:3300/0 conn(0x7eff300a6070 0x7eff300a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 10 monclient(hunting): get_auth_request con 0x7eff300a6070 auth_method 0 2023-10-30T12:03:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.622+0000 7eff2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 10 monclient(hunting): handle_auth_done global_id 27805 payload 274 2023-10-30T12:03:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 1 -- 172.21.15.50:0/655676024 >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 msgr2=0x7eff30135590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 1 --2- 172.21.15.50:0/655676024 >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 0x7eff30135590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 1 -- 172.21.15.50:0/655676024 >> v1:172.21.15.50:6789/0 conn(0x7eff30135ad0 legacy=0x7eff3013c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 10 monclient: _finish_hunting 0 2023-10-30T12:03:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 1 monclient: found mon.noname-c 2023-10-30T12:03:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:03:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 1 -- 172.21.15.50:0/655676024 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3009a830 con 0x7eff300a6070 2023-10-30T12:03:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.623+0000 7eff2effd640 1 --2- 172.21.15.50:0/655676024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff300a6070 0x7eff300a57e0 secure :-1 s=READY pgs=6976 cs=0 l=1 rev1=1 crypto rx=0x7eff300a4ee0 tx=0x7eff1802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a4c8280810aa3e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:10.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 1 -- 172.21.15.50:0/655676024 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18031710 con 0x7eff300a6070 2023-10-30T12:03:10.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: dump: 2023-10-30T12:03:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:10.645 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: _finish_auth 0 2023-10-30T12:03:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:03:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 5 monclient: authenticate success, global_id 27805 2023-10-30T12:03:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 1 -- 172.21.15.50:0/655676024 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1802f050 con 0x7eff300a6070 2023-10-30T12:03:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 4 set_mon_vals no callback set 2023-10-30T12:03:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 1 -- 172.21.15.50:0/655676024 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18031a90 con 0x7eff300a6070 2023-10-30T12:03:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff2dffb640 10 monclient: dump: 2023-10-30T12:03:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:10.650 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 10 monclient: shutdown 2023-10-30T12:03:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 -- 172.21.15.50:0/655676024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff300a6070 msgr2=0x7eff300a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 --2- 172.21.15.50:0/655676024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff300a6070 0x7eff300a57e0 secure :-1 s=READY pgs=6976 cs=0 l=1 rev1=1 crypto rx=0x7eff300a4ee0 tx=0x7eff1802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 -- 172.21.15.50:0/655676024 shutdown_connections 2023-10-30T12:03:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 --2- 172.21.15.50:0/655676024 >> v2:172.21.15.50:3300/0 conn(0x7eff300a6450 0x7eff30135590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 --2- 172.21.15.50:0/655676024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff300a6070 0x7eff300a57e0 unknown :-1 s=CLOSED pgs=6976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 -- 172.21.15.50:0/655676024 >> 172.21.15.50:0/655676024 conn(0x7eff300a81d0 msgr2=0x7eff300a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 -- 172.21.15.50:0/655676024 shutdown_connections 2023-10-30T12:03:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.624+0000 7eff353c5640 1 -- 172.21.15.50:0/655676024 wait complete. 2023-10-30T12:03:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 10 monclient: build_initial_monmap 2023-10-30T12:03:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 10 monclient: monmap: 2023-10-30T12:03:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:10.654 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 1 librados: starting msgr at 2023-10-30T12:03:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 1 librados: starting objecter 2023-10-30T12:03:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 1 Processor -- start 2023-10-30T12:03:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 1 -- start start 2023-10-30T12:03:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 1 librados: setting wanted keys 2023-10-30T12:03:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 1 librados: calling monclient init 2023-10-30T12:03:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.625+0000 7eff353c5640 10 monclient: init 2023-10-30T12:03:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:03:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 0x7eff300a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): picked mon.noname-a con 0x7eff300a6070 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff300a6450 0x7eff301c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): picked mon.noname-c con 0x7eff300a6450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff30135ad0 0x7eff30132cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): picked mon.noname-b con 0x7eff30135ad0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff30121720 con 0x7eff300a6070 2023-10-30T12:03:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff30120df0 con 0x7eff30135ad0 2023-10-30T12:03:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff301210d0 con 0x7eff300a6450 2023-10-30T12:03:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff353c5640 10 monclient(hunting): authenticate will time out at 1035.809363s 2023-10-30T12:03:10.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 0x7eff300a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:10.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff300a6450 0x7eff301c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:10.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 0x7eff300a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36956/0 (socket says 172.21.15.50:36956) 2023-10-30T12:03:10.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 1 -- 172.21.15.50:0/1699956278 learned_addr learned my addr 172.21.15.50:0/1699956278 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:10.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 10 monclient(hunting): get_auth_request con 0x7eff300a6070 auth_method 0 2023-10-30T12:03:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): get_auth_request con 0x7eff300a6450 auth_method 0 2023-10-30T12:03:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:10.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:10.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:10.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 1 -- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 msgr2=0x7eff300a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:03:10.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 1 -- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 msgr2=0x7eff300a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:03:10.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 0x7eff300a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2effd640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 0x7eff300a0500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2f7fe640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff30135ad0 0x7eff30132cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2f7fe640 10 monclient(hunting): get_auth_request con 0x7eff30135ad0 auth_method 0 2023-10-30T12:03:10.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:10.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:10.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:10.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient(hunting): handle_auth_done global_id 44225 payload 824 2023-10-30T12:03:10.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 -- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff30135ad0 msgr2=0x7eff30132cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:10.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff30135ad0 0x7eff30132cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 -- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 msgr2=0x7eff300a0500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:10.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 0x7eff300a0500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:03:10.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 monclient: found mon.noname-c 2023-10-30T12:03:10.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:10.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:10.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 -- 172.21.15.50:0/1699956278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18009400 con 0x7eff300a6450 2023-10-30T12:03:10.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2f7fe640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff30135ad0 0x7eff30132cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:03:10.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff2e7fc640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff300a6450 0x7eff301c6d40 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7eff2000d5a0 tx=0x7eff2000da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:10.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.626+0000 7eff277fe640 10 client.?.objecter ms_handle_connect 0x7eff300a6450 2023-10-30T12:03:10.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:10.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 1 -- 172.21.15.50:0/1699956278 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff200125f0 con 0x7eff300a6450 2023-10-30T12:03:10.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:10.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:10.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: dump: 2023-10-30T12:03:10.667 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:10.667 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:10.667 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.667 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.667 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:10.667 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:10.668 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:10.668 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:10.668 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:10.668 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:10.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:10.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:10.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:10.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 1 -- 172.21.15.50:0/1699956278 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff20012750 con 0x7eff300a6450 2023-10-30T12:03:10.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:10.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 1 -- 172.21.15.50:0/1699956278 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20012b00 con 0x7eff300a6450 2023-10-30T12:03:10.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:10.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:10.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff277fe640 10 monclient: dump: 2023-10-30T12:03:10.670 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:10.670 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:10.670 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.671 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:10.671 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:10.671 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:10.671 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:10.671 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:10.671 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:10.672 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:10.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 5 monclient: authenticate success, global_id 44225 2023-10-30T12:03:10.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff2d7fa640 4 set_mon_vals no callback set 2023-10-30T12:03:10.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 10 monclient: _renew_subs 2023-10-30T12:03:10.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff301213b0 con 0x7eff300a6450 2023-10-30T12:03:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 10 client.44225.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 10 monclient: _renew_subs 2023-10-30T12:03:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:10.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff30133200 con 0x7eff300a6450 2023-10-30T12:03:10.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.627+0000 7eff353c5640 1 librados: init done 2023-10-30T12:03:10.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 1 -- 172.21.15.50:0/1699956278 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7eff20012c60 con 0x7eff300a6450 2023-10-30T12:03:10.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff353c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:10.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff353c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:10.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff353c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:10.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7eff100392f0 0x7eff1003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:10.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3004e160 con 0x7eff300a6450 2023-10-30T12:03:10.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 1 -- 172.21.15.50:0/1699956278 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8748+0+0 (secure 0 0 0) 0x7eff2004bb10 con 0x7eff300a6450 2023-10-30T12:03:10.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 10 client.44225.objecter ms_dispatch 0x7eff30000b90 osd_map(71..71 src has 1..71) v4 2023-10-30T12:03:10.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 3 client.44225.objecter handle_osd_map got epochs [71,71] > 0 2023-10-30T12:03:10.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 3 client.44225.objecter handle_osd_map decoding full epoch 71 2023-10-30T12:03:10.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff2effd640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7eff100392f0 0x7eff1003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:10.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 20 client.44225.objecter dump_active .. 0 homeless 2023-10-30T12:03:10.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff2effd640 10 monclient: get_auth_request con 0x7eff100392f0 auth_method 0 2023-10-30T12:03:10.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff2effd640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7eff100392f0 0x7eff1003b7b0 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7eff1802b7f0 tx=0x7eff18008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:10.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.628+0000 7eff277fe640 10 client.44225.objecter ms_handle_connect 0x7eff100392f0 2023-10-30T12:03:10.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.633+0000 7eff277fe640 1 -- 172.21.15.50:0/1699956278 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7eff20014430 con 0x7eff300a6450 2023-10-30T12:03:10.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.633+0000 7eff277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:10.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.633+0000 7eff277fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:10.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.879+0000 7eff353c5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:10.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.879+0000 7eff353c5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:10.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.879+0000 7eff353c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:10.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.879+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7eff30074580 con 0x7eff300a6450 2023-10-30T12:03:10.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.879+0000 7eff277fe640 1 -- 172.21.15.50:0/1699956278 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7eff20011340 con 0x7eff300a6450 2023-10-30T12:03:10.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.879+0000 7eff277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:10.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.879+0000 7eff277fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:10.883 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:10.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 10 librados: watch_flush enter 2023-10-30T12:03:10.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 10 librados: watch_flush exit 2023-10-30T12:03:10.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 20 client.44225.objecter shutdown clearing up homeless session... 2023-10-30T12:03:10.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 10 client.44225.objecter successfully canceled tick 2023-10-30T12:03:10.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7eff100392f0 msgr2=0x7eff1003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:10.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7eff100392f0 0x7eff1003b7b0 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7eff1802b7f0 tx=0x7eff18008030 comp rx=0 tx=0).stop 2023-10-30T12:03:10.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 10 monclient: shutdown 2023-10-30T12:03:10.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:10.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff300a6450 msgr2=0x7eff301c6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:10.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff300a6450 0x7eff301c6d40 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7eff2000d5a0 tx=0x7eff2000da60 comp rx=0 tx=0).stop 2023-10-30T12:03:10.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 shutdown_connections 2023-10-30T12:03:10.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7eff100392f0 0x7eff1003b7b0 unknown :-1 s=CLOSED pgs=3855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff30135ad0 0x7eff30132cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff300a6450 0x7eff301c6d40 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 --2- 172.21.15.50:0/1699956278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff300a6070 0x7eff300a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:10.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.881+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 >> 172.21.15.50:0/1699956278 conn(0x7eff300a81d0 msgr2=0x7eff300a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:10.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.882+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 shutdown_connections 2023-10-30T12:03:10.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.882+0000 7eff353c5640 1 -- 172.21.15.50:0/1699956278 wait complete. 2023-10-30T12:03:10.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:10.882+0000 7eff353c5640 1 librados: shutdown 2023-10-30T12:03:11.112 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: Running main() from gmock_main.cc 2023-10-30T12:03:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-10-30T12:03:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] Global test environment set-up. 2023-10-30T12:03:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-10-30T12:03:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-10-30T12:03:11.113 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (1982 ms) 2023-10-30T12:03:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-10-30T12:03:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2014 ms) 2023-10-30T12:03:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-10-30T12:03:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1995 ms) 2023-10-30T12:03:11.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-10-30T12:03:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2007 ms) 2023-10-30T12:03:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (7998 ms total) 2023-10-30T12:03:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: 2023-10-30T12:03:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-10-30T12:03:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-10-30T12:03:11.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4048 ms) 2023-10-30T12:03:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-10-30T12:03:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3987 ms) 2023-10-30T12:03:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-10-30T12:03:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (9850 ms) 2023-10-30T12:03:11.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (17885 ms total) 2023-10-30T12:03:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: 2023-10-30T12:03:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-10-30T12:03:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-10-30T12:03:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3008 ms) 2023-10-30T12:03:11.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-10-30T12:03:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (15422 ms) 2023-10-30T12:03:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-10-30T12:03:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2007 ms) 2023-10-30T12:03:11.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-10-30T12:03:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2004 ms) 2023-10-30T12:03:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (22441 ms total) 2023-10-30T12:03:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: 2023-10-30T12:03:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-10-30T12:03:11.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-10-30T12:03:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4038 ms) 2023-10-30T12:03:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-10-30T12:03:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3990 ms) 2023-10-30T12:03:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8028 ms total) 2023-10-30T12:03:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: 2023-10-30T12:03:11.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [----------] Global test environment tear-down 2023-10-30T12:03:11.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (76810 ms total) 2023-10-30T12:03:11.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-10-30T12:03:13.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:14.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.009+0000 7f81095b0640 10 monclient: get_monmap_and_config 2023-10-30T12:03:14.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.009+0000 7f81095b0640 10 monclient: build_initial_monmap 2023-10-30T12:03:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.009+0000 7f81095b0640 10 monclient: monmap: 2023-10-30T12:03:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:14.010716+0000 2023-10-30T12:03:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:14.010716+0000 2023-10-30T12:03:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:14.014 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 1 Processor -- start 2023-10-30T12:03:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 1 -- start start 2023-10-30T12:03:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient: init 2023-10-30T12:03:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-10-30T12:03:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81040a2dd0 0x7f81040a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient(hunting): picked mon.noname-b con 0x7f81040a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f810406a8e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8104071400 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81040a31b0 con 0x7f8104071400 2023-10-30T12:03:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810406a6a0 con 0x7f810406a8e0 2023-10-30T12:03:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810406be90 con 0x7f81040a2dd0 2023-10-30T12:03:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81095b0640 10 monclient(hunting): authenticate will time out at 1039.193770s 2023-10-30T12:03:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f8102ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81040a2dd0 0x7f81040a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f8102ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81040a2dd0 0x7f81040a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57966/0 (socket says 172.21.15.50:57966) 2023-10-30T12:03:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f81037fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8104071400 0x7f810413cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39284/0 (socket says 172.21.15.50:39284) 2023-10-30T12:03:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f8102ffd640 1 -- 172.21.15.50:0/294627927 learned_addr learned my addr 172.21.15.50:0/294627927 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f8102ffd640 10 monclient(hunting): get_auth_request con 0x7f81040a2dd0 auth_method 0 2023-10-30T12:03:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f8102ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f8102ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.010+0000 7f8102ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 10 monclient(hunting): handle_auth_done global_id 44240 payload 274 2023-10-30T12:03:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 1 -- 172.21.15.50:0/294627927 >> v1:172.21.15.118:6789/0 conn(0x7f810406a8e0 legacy=0x7f81040a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 1 -- 172.21.15.50:0/294627927 >> v1:172.21.15.50:6789/0 conn(0x7f8104071400 legacy=0x7f810413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:03:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 1 monclient: found mon.noname-b 2023-10-30T12:03:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:03:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 1 -- 172.21.15.50:0/294627927 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810409e190 con 0x7f81040a2dd0 2023-10-30T12:03:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.011+0000 7f8102ffd640 1 --2- 172.21.15.50:0/294627927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81040a2dd0 0x7f81040a46c0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f80f40098d0 tx=0x7f80f402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8e3487a3fcc6dc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 1 -- 172.21.15.50:0/294627927 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4031830 con 0x7f81040a2dd0 2023-10-30T12:03:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:03:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: dump: 2023-10-30T12:03:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:14.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: _finish_auth 0 2023-10-30T12:03:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:03:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 1 -- 172.21.15.50:0/294627927 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f4031990 con 0x7f81040a2dd0 2023-10-30T12:03:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 5 monclient: authenticate success, global_id 44240 2023-10-30T12:03:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 1 -- 172.21.15.50:0/294627927 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4031d20 con 0x7f81040a2dd0 2023-10-30T12:03:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 4 set_mon_vals no callback set 2023-10-30T12:03:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f8101ffb640 10 monclient: dump: 2023-10-30T12:03:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:14.035 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 10 monclient: shutdown 2023-10-30T12:03:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 1 -- 172.21.15.50:0/294627927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81040a2dd0 msgr2=0x7f81040a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 1 --2- 172.21.15.50:0/294627927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81040a2dd0 0x7f81040a46c0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f80f40098d0 tx=0x7f80f402b8d0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 1 -- 172.21.15.50:0/294627927 shutdown_connections 2023-10-30T12:03:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 1 --2- 172.21.15.50:0/294627927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81040a2dd0 0x7f81040a46c0 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 1 -- 172.21.15.50:0/294627927 >> 172.21.15.50:0/294627927 conn(0x7f81040a22b0 msgr2=0x7f81040a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 1 -- 172.21.15.50:0/294627927 shutdown_connections 2023-10-30T12:03:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.012+0000 7f81095b0640 1 -- 172.21.15.50:0/294627927 wait complete. 2023-10-30T12:03:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 10 monclient: build_initial_monmap 2023-10-30T12:03:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 10 monclient: monmap: 2023-10-30T12:03:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:14.037 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:14.038 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:14.038 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 1 librados: starting msgr at 2023-10-30T12:03:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 1 librados: starting objecter 2023-10-30T12:03:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 1 Processor -- start 2023-10-30T12:03:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 1 -- start start 2023-10-30T12:03:14.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 1 librados: setting wanted keys 2023-10-30T12:03:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 1 librados: calling monclient init 2023-10-30T12:03:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.013+0000 7f81095b0640 10 monclient: init 2023-10-30T12:03:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:03:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8104071400 0x7f81041ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): picked mon.noname-b con 0x7f8104071400 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 0x7f81041cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): picked mon.noname-a con 0x7f81040a2dd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810411e340 0x7f810411e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): picked mon.noname-c con 0x7f810411e340 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8104122160 con 0x7f81040a2dd0 2023-10-30T12:03:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81041223b0 con 0x7f8104071400 2023-10-30T12:03:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810406b0a0 con 0x7f810411e340 2023-10-30T12:03:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81095b0640 10 monclient(hunting): authenticate will time out at 1039.197348s 2023-10-30T12:03:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810411e340 0x7f810411e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 0x7f81041cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810411e340 0x7f810411e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57968/0 (socket says 172.21.15.50:57968) 2023-10-30T12:03:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 0x7f81041cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36966/0 (socket says 172.21.15.50:36966) 2023-10-30T12:03:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 -- 172.21.15.50:0/4097488634 learned_addr learned my addr 172.21.15.50:0/4097488634 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): get_auth_request con 0x7f810411e340 auth_method 0 2023-10-30T12:03:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 10 monclient(hunting): get_auth_request con 0x7f81040a2dd0 auth_method 0 2023-10-30T12:03:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 1 -- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 msgr2=0x7f81041cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:03:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 1 -- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 msgr2=0x7f81041cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:03:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 0x7f81041cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81027fc640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 0x7f81041cacc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f8102ffd640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8104071400 0x7f81041ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f8102ffd640 10 monclient(hunting): get_auth_request con 0x7f8104071400 auth_method 0 2023-10-30T12:03:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f8102ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f8102ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f8102ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient(hunting): handle_auth_done global_id 44243 payload 824 2023-10-30T12:03:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 -- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8104071400 msgr2=0x7f81041ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8104071400 0x7f81041ca780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 -- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 msgr2=0x7f81041cacc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 0x7f81041cacc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 monclient: found mon.noname-c 2023-10-30T12:03:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.014+0000 7f81037fe640 1 -- 172.21.15.50:0/4097488634 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f40094d0 con 0x7f810411e340 2023-10-30T12:03:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81037fe640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810411e340 0x7f810411e7c0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c760 tx=0x7f80fc00cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 client.?.objecter ms_handle_connect 0x7f810411e340 2023-10-30T12:03:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 1 -- 172.21.15.50:0/4097488634 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc00ce70 con 0x7f810411e340 2023-10-30T12:03:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: dump: 2023-10-30T12:03:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:14.053 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 5 monclient: authenticate success, global_id 44243 2023-10-30T12:03:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 1 -- 172.21.15.50:0/4097488634 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc00a640 con 0x7f810411e340 2023-10-30T12:03:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 1 -- 172.21.15.50:0/4097488634 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc00a9b0 con 0x7f810411e340 2023-10-30T12:03:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f80eb7fe640 10 monclient: dump: 2023-10-30T12:03:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:14.056 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81017fa640 4 set_mon_vals no callback set 2023-10-30T12:03:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 10 monclient: _renew_subs 2023-10-30T12:03:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81040708e0 con 0x7f810411e340 2023-10-30T12:03:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 10 client.44243.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 10 monclient: _renew_subs 2023-10-30T12:03:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810411ffe0 con 0x7f810411e340 2023-10-30T12:03:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.015+0000 7f81095b0640 1 librados: init done 2023-10-30T12:03:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f80eb7fe640 1 -- 172.21.15.50:0/4097488634 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f80fc00ab10 con 0x7f810411e340 2023-10-30T12:03:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f81095b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f81095b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f81095b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8104120c90 con 0x7f810411e340 2023-10-30T12:03:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f80eb7fe640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f80f00392f0 0x7f80f003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f80eb7fe640 1 -- 172.21.15.50:0/4097488634 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f80fc04af10 con 0x7f810411e340 2023-10-30T12:03:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f80eb7fe640 10 client.44243.objecter ms_dispatch 0x7f8104000b90 osd_map(74..74 src has 1..74) v4 2023-10-30T12:03:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f80eb7fe640 3 client.44243.objecter handle_osd_map got epochs [74,74] > 0 2023-10-30T12:03:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f80eb7fe640 3 client.44243.objecter handle_osd_map decoding full epoch 74 2023-10-30T12:03:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f80eb7fe640 20 client.44243.objecter dump_active .. 0 homeless 2023-10-30T12:03:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f8102ffd640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f80f00392f0 0x7f80f003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.016+0000 7f8102ffd640 10 monclient: get_auth_request con 0x7f80f00392f0 auth_method 0 2023-10-30T12:03:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.017+0000 7f80eb7fe640 10 client.44243.objecter ms_handle_connect 0x7f80f00392f0 2023-10-30T12:03:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.017+0000 7f8102ffd640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f80f00392f0 0x7f80f003b7b0 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7f80f4036000 tx=0x7f80f403c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.021+0000 7f80eb7fe640 1 -- 172.21.15.50:0/4097488634 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f80fc007070 con 0x7f810411e340 2023-10-30T12:03:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.021+0000 7f80eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.021+0000 7f80eb7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.277+0000 7f81095b0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.277+0000 7f81095b0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.277+0000 7f81095b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.277+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8104121c30 con 0x7f810411e340 2023-10-30T12:03:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.278+0000 7f80eb7fe640 1 -- 172.21.15.50:0/4097488634 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f80fc04c030 con 0x7f810411e340 2023-10-30T12:03:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.278+0000 7f80eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.278+0000 7f80eb7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:14.281 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 10 librados: watch_flush enter 2023-10-30T12:03:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 10 librados: watch_flush exit 2023-10-30T12:03:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 20 client.44243.objecter shutdown clearing up homeless session... 2023-10-30T12:03:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 10 client.44243.objecter successfully canceled tick 2023-10-30T12:03:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f80f00392f0 msgr2=0x7f80f003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f80f00392f0 0x7f80f003b7b0 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7f80f4036000 tx=0x7f80f403c040 comp rx=0 tx=0).stop 2023-10-30T12:03:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 10 monclient: shutdown 2023-10-30T12:03:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810411e340 msgr2=0x7f810411e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810411e340 0x7f810411e7c0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c760 tx=0x7f80fc00cc20 comp rx=0 tx=0).stop 2023-10-30T12:03:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 shutdown_connections 2023-10-30T12:03:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f80f00392f0 0x7f80f003b7b0 unknown :-1 s=CLOSED pgs=3858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810411e340 0x7f810411e7c0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81040a2dd0 0x7f81041cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 --2- 172.21.15.50:0/4097488634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8104071400 0x7f81041ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 >> 172.21.15.50:0/4097488634 conn(0x7f81040a22b0 msgr2=0x7f81040a0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 shutdown_connections 2023-10-30T12:03:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 -- 172.21.15.50:0/4097488634 wait complete. 2023-10-30T12:03:14.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:14.280+0000 7f81095b0640 1 librados: shutdown 2023-10-30T12:03:16.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2023-10-30T12:03:16.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] Global test environment set-up. 2023-10-30T12:03:16.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-10-30T12:03:16.183 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-10-30T12:03:16.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-10-30T12:03:16.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-10-30T12:03:16.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: 2023-10-30T12:03:16.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-10-30T12:03:16.184 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-10-30T12:03:16.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-10-30T12:03:16.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: 2023-10-30T12:01:57.169+0000 7f73b0669780 -1 failed for service _ceph-mon._tcp 2023-10-30T12:03:16.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: 2023-10-30T12:01:57.169+0000 7f73b0669780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-10-30T12:03:16.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (2913 ms) 2023-10-30T12:03:16.185 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-10-30T12:03:16.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2008 ms) 2023-10-30T12:03:16.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (4922 ms total) 2023-10-30T12:03:16.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: 2023-10-30T12:03:16.186 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-10-30T12:03:16.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-10-30T12:03:16.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: started 0x55bed341d830 2023-10-30T12:03:16.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: started 0x55bed280fca0 2023-10-30T12:03:16.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: started 2 aios 2023-10-30T12:03:16.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: waiting 0x55bed341d830 2023-10-30T12:03:16.187 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: waiting 0x55bed280fca0 2023-10-30T12:03:16.188 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: done. 2023-10-30T12:03:16.189 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5992 ms) 2023-10-30T12:03:16.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5992 ms total) 2023-10-30T12:03:16.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: 2023-10-30T12:03:16.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-10-30T12:03:16.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-10-30T12:03:16.190 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-10-30T12:03:16.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-10-30T12:03:16.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.Exec (32 ms) 2023-10-30T12:03:16.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-10-30T12:03:16.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (6 ms) 2023-10-30T12:03:16.191 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-10-30T12:03:16.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2 ms) 2023-10-30T12:03:16.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-10-30T12:03:16.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4891 ms) 2023-10-30T12:03:16.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-10-30T12:03:16.192 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-10-30T12:03:16.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-10-30T12:03:16.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-10-30T12:03:16.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-10-30T12:03:16.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (63498 ms) 2023-10-30T12:03:16.193 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] 8 tests from LibRadosMisc (68429 ms total) 2023-10-30T12:03:16.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: 2023-10-30T12:03:16.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [----------] Global test environment tear-down 2023-10-30T12:03:16.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (81870 ms total) 2023-10-30T12:03:16.194 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc: [ PASSED ] 12 tests. 2023-10-30T12:03:17.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:17.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.407+0000 7f9b355f5640 10 monclient: get_monmap_and_config 2023-10-30T12:03:17.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.407+0000 7f9b355f5640 10 monclient: build_initial_monmap 2023-10-30T12:03:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.407+0000 7f9b355f5640 10 monclient: monmap: 2023-10-30T12:03:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:17.408891+0000 2023-10-30T12:03:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:17.408891+0000 2023-10-30T12:03:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:17.414 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 1 Processor -- start 2023-10-30T12:03:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 1 -- start start 2023-10-30T12:03:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient: init 2023-10-30T12:03:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:03:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3009f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b30096080 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b30096b70 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3009f990 con 0x7f9b30096080 2023-10-30T12:03:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b30098810 con 0x7f9b30096b70 2023-10-30T12:03:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3006af10 con 0x7f9b3009f5b0 2023-10-30T12:03:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.408+0000 7f9b355f5640 10 monclient(hunting): authenticate will time out at 1042.591948s 2023-10-30T12:03:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2e7fc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9b30096080 0x7f9b30096460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39296/0 (socket says 172.21.15.50:39296) 2023-10-30T12:03:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 learned_addr learned my addr 172.21.15.50:0/2894907735 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b3009f5b0 auth_method 0 2023-10-30T12:03:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.409+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:17.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:17.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b3009f5b0 auth_method 0 2023-10-30T12:03:17.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:17.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:17.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:17.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:17.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:17.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:17.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:17.610+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:03:18.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:18.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b3009f5b0 auth_method 0 2023-10-30T12:03:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:18.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:18.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.011+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:03:18.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b3009f5b0 auth_method 0 2023-10-30T12:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:18.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:18.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:18.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:18.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:18.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:18.812+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:03:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): tick 2023-10-30T12:03:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:03:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:03:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:03:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 msgr2=0x7f9b30099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3009f5b0 0x7f9b30099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 >> v1:172.21.15.118:6789/0 conn(0x7f9b30096b70 legacy=0x7f9b3013ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 >> v1:172.21.15.50:6789/0 conn(0x7f9b30096080 legacy=0x7f9b30096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient: _add_conns ranks=[4,3] 2023-10-30T12:03:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1c003720 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1c006290 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3006af10 con 0x7f9b1c006290 2023-10-30T12:03:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1c008e10 con 0x7f9b1c003720 2023-10-30T12:03:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2cff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1c006290 auth_method 0 2023-10-30T12:03:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b1c003720 auth_method 0 2023-10-30T12:03:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.409+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1c006290 auth_method 0 2023-10-30T12:03:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:03:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b1c003720 auth_method 0 2023-10-30T12:03:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:20.610+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:03:21.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:21.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1c006290 auth_method 0 2023-10-30T12:03:21.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:21.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:21.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:21.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:21.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:21.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:21.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:21.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b1c003720 auth_method 0 2023-10-30T12:03:21.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:21.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:21.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:21.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:03:21.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:21.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:21.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:21.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.011+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:03:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1c006290 auth_method 0 2023-10-30T12:03:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:03:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b1c003720 auth_method 0 2023-10-30T12:03:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:21.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:21.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:21.812+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:03:23.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): tick 2023-10-30T12:03:23.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:03:23.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:03:23.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:03:23.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:23.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:23.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:03:23.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2894907735 reap_dead start 2023-10-30T12:03:23.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b30096080 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:23.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1c003720 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:23.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1c006290 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:23.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3009a1a0 con 0x7f9b30096080 2023-10-30T12:03:23.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3006af10 con 0x7f9b1c006290 2023-10-30T12:03:23.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2894907735 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1c008e10 con 0x7f9b1c003720 2023-10-30T12:03:23.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2cff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:23.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.50:3300/0 conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b1c006290 auth_method 0 2023-10-30T12:03:23.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1c003720 auth_method 0 2023-10-30T12:03:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_done global_id 64113 payload 274 2023-10-30T12:03:23.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 msgr2=0x7f9b1c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:23.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2894907735 >> v1:172.21.15.50:6789/0 conn(0x7f9b30096080 legacy=0x7f9b1c00b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:23.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:03:23.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:23.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:23.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2894907735 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c00d950 con 0x7f9b1c006290 2023-10-30T12:03:23.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b1c006290 0x7f9b1c008750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9b30096f50 tx=0x7f9b2802b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a0a5e335541892 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 1 -- 172.21.15.50:0/2894907735 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28005d40 con 0x7f9b1c006290 2023-10-30T12:03:23.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:23.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: dump: 2023-10-30T12:03:23.422 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.422 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.422 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.423 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.423 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.423 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.423 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.423 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.424 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.424 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: _finish_auth 0 2023-10-30T12:03:23.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:03:23.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:23.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 1 -- 172.21.15.50:0/2894907735 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2802e070 con 0x7f9b1c006290 2023-10-30T12:03:23.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b355f5640 5 monclient: authenticate success, global_id 64113 2023-10-30T12:03:23.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b355f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:23.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:23.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b355f5640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:23.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b355f5640 4 set_mon_vals no callback set 2023-10-30T12:03:23.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 1 -- 172.21.15.50:0/2894907735 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28007270 con 0x7f9b1c006290 2023-10-30T12:03:23.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:23.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.410+0000 7f9b2dffb640 10 monclient: dump: 2023-10-30T12:03:23.426 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.427 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.427 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.427 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.427 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.427 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.428 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.428 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.428 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.428 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 10 monclient: shutdown 2023-10-30T12:03:23.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:23.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 -- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b1c006290 msgr2=0x7f9b1c008750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b1c006290 0x7f9b1c008750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9b30096f50 tx=0x7f9b2802b730 comp rx=0 tx=0).stop 2023-10-30T12:03:23.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 -- 172.21.15.50:0/2894907735 shutdown_connections 2023-10-30T12:03:23.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 --2- 172.21.15.50:0/2894907735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b1c006290 0x7f9b1c008750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 --2- 172.21.15.50:0/2894907735 >> v2:172.21.15.118:3300/0 conn(0x7f9b1c003720 0x7f9b1c005ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 -- 172.21.15.50:0/2894907735 >> 172.21.15.50:0/2894907735 conn(0x7f9b300a05c0 msgr2=0x7f9b3009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:23.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 -- 172.21.15.50:0/2894907735 shutdown_connections 2023-10-30T12:03:23.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.411+0000 7f9b355f5640 1 -- 172.21.15.50:0/2894907735 wait complete. 2023-10-30T12:03:23.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 10 monclient: build_initial_monmap 2023-10-30T12:03:23.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 10 monclient: monmap: 2023-10-30T12:03:23.435 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:23.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:23.436 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:23.436 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:23.436 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:23.436 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.436 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:23.437 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:23.437 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:23.437 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 1 librados: starting msgr at 2023-10-30T12:03:23.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 1 librados: starting objecter 2023-10-30T12:03:23.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 1 Processor -- start 2023-10-30T12:03:23.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 1 -- start start 2023-10-30T12:03:23.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 1 librados: setting wanted keys 2023-10-30T12:03:23.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 1 librados: calling monclient init 2023-10-30T12:03:23.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.412+0000 7f9b355f5640 10 monclient: init 2023-10-30T12:03:23.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:23.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:03:23.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b1c003720 0x7f9b30095310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1c003720 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:23.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b1c006290 0x7f9b30095850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): picked mon.noname-b con 0x7f9b1c006290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:23.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30096080 0x7f9b3008afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): picked mon.noname-a con 0x7f9b30096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:23.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3008c270 con 0x7f9b30096080 2023-10-30T12:03:23.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30072080 con 0x7f9b1c006290 2023-10-30T12:03:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301bba90 con 0x7f9b1c003720 2023-10-30T12:03:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b355f5640 10 monclient(hunting): authenticate will time out at 1048.596456s 2023-10-30T12:03:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30096080 0x7f9b3008afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b1c003720 0x7f9b30095310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30096080 0x7f9b3008afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60752/0 (socket says 172.21.15.50:60752) 2023-10-30T12:03:23.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b1c003720 0x7f9b30095310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40660/0 (socket says 172.21.15.50:40660) 2023-10-30T12:03:23.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/3355986312 learned_addr learned my addr 172.21.15.50:0/3355986312 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:23.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b30096080 auth_method 0 2023-10-30T12:03:23.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b1c003720 auth_method 0 2023-10-30T12:03:23.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b1c006290 0x7f9b30095850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f9b1c006290 auth_method 0 2023-10-30T12:03:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_done global_id 64119 payload 824 2023-10-30T12:03:23.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b1c003720 msgr2=0x7f9b30095310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b1c003720 0x7f9b30095310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b1c006290 msgr2=0x7f9b30095850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b1c006290 0x7f9b30095850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:23.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:03:23.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:23.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:23.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/3355986312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b30098810 con 0x7f9b30096080 2023-10-30T12:03:23.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.413+0000 7f9b2effd640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b1c003720 0x7f9b30095310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:23.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30096080 0x7f9b3008afd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009a5d0 tx=0x7f9b280014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 client.?.objecter ms_handle_connect 0x7f9b30096080 2023-10-30T12:03:23.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:23.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 1 -- 172.21.15.50:0/3355986312 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28007630 con 0x7f9b30096080 2023-10-30T12:03:23.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:23.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: dump: 2023-10-30T12:03:23.450 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.450 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.450 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.450 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.450 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.451 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.451 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.451 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.451 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.451 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 5 monclient: authenticate success, global_id 64119 2023-10-30T12:03:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 10 monclient: _renew_subs 2023-10-30T12:03:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:23.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3008daa0 con 0x7f9b30096080 2023-10-30T12:03:23.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 1 -- 172.21.15.50:0/3355986312 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b28007790 con 0x7f9b30096080 2023-10-30T12:03:23.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:23.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 1 -- 172.21.15.50:0/3355986312 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28033430 con 0x7f9b30096080 2023-10-30T12:03:23.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:23.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b177fe640 10 monclient: dump: 2023-10-30T12:03:23.454 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.454 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.454 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.455 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.455 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.455 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.455 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.455 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.455 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.456 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 10 client.64119.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:23.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 10 monclient: _renew_subs 2023-10-30T12:03:23.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:23.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b2d7fa640 4 set_mon_vals no callback set 2023-10-30T12:03:23.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b300712d0 con 0x7f9b30096080 2023-10-30T12:03:23.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.414+0000 7f9b355f5640 1 librados: init done 2023-10-30T12:03:23.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b177fe640 1 -- 172.21.15.50:0/3355986312 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9b28030030 con 0x7f9b30096080 2023-10-30T12:03:23.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b355f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:23.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b355f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:23.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b355f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:23.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3008d5f0 con 0x7f9b30096080 2023-10-30T12:03:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b177fe640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c03a150 0x7f9b1c03c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b177fe640 1 -- 172.21.15.50:0/3355986312 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7733+0+0 (secure 0 0 0) 0x7f9b2806cf80 con 0x7f9b30096080 2023-10-30T12:03:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b177fe640 10 client.64119.objecter ms_dispatch 0x7f9b30000b90 osd_map(79..79 src has 1..79) v4 2023-10-30T12:03:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b177fe640 3 client.64119.objecter handle_osd_map got epochs [79,79] > 0 2023-10-30T12:03:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b177fe640 3 client.64119.objecter handle_osd_map decoding full epoch 79 2023-10-30T12:03:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b177fe640 20 client.64119.objecter dump_active .. 0 homeless 2023-10-30T12:03:23.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b2effd640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c03a150 0x7f9b1c03c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.415+0000 7f9b2effd640 10 monclient: get_auth_request con 0x7f9b1c03a150 auth_method 0 2023-10-30T12:03:23.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.416+0000 7f9b2effd640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c03a150 0x7f9b1c03c610 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f9b20006ab0 tx=0x7f9b20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.416+0000 7f9b177fe640 10 client.64119.objecter ms_handle_connect 0x7f9b1c03a150 2023-10-30T12:03:23.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.420+0000 7f9b177fe640 1 -- 172.21.15.50:0/3355986312 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b2803e420 con 0x7f9b30096080 2023-10-30T12:03:23.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.420+0000 7f9b177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:23.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.420+0000 7f9b177fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:23.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.662+0000 7f9b355f5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:23.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.662+0000 7f9b355f5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:23.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.662+0000 7f9b355f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:23.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.662+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9b3004e160 con 0x7f9b30096080 2023-10-30T12:03:23.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.663+0000 7f9b177fe640 1 -- 172.21.15.50:0/3355986312 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9b280323d0 con 0x7f9b30096080 2023-10-30T12:03:23.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.663+0000 7f9b177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:23.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.663+0000 7f9b177fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:23.667 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:23.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.664+0000 7f9b355f5640 10 librados: watch_flush enter 2023-10-30T12:03:23.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.664+0000 7f9b355f5640 10 librados: watch_flush exit 2023-10-30T12:03:23.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.664+0000 7f9b355f5640 20 client.64119.objecter shutdown clearing up homeless session... 2023-10-30T12:03:23.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.664+0000 7f9b355f5640 10 client.64119.objecter successfully canceled tick 2023-10-30T12:03:23.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c03a150 msgr2=0x7f9b1c03c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c03a150 0x7f9b1c03c610 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f9b20006ab0 tx=0x7f9b20008040 comp rx=0 tx=0).stop 2023-10-30T12:03:23.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 10 monclient: shutdown 2023-10-30T12:03:23.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:23.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30096080 msgr2=0x7f9b3008afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30096080 0x7f9b3008afd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009a5d0 tx=0x7f9b280014d0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 shutdown_connections 2023-10-30T12:03:23.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c03a150 0x7f9b1c03c610 unknown :-1 s=CLOSED pgs=3864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30096080 0x7f9b3008afd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b1c006290 0x7f9b30095850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 --2- 172.21.15.50:0/3355986312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b1c003720 0x7f9b30095310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 >> 172.21.15.50:0/3355986312 conn(0x7f9b300a05c0 msgr2=0x7f9b301ba8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:23.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 shutdown_connections 2023-10-30T12:03:23.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 -- 172.21.15.50:0/3355986312 wait complete. 2023-10-30T12:03:23.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.665+0000 7f9b355f5640 1 librados: shutdown 2023-10-30T12:03:23.676 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:03:23.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:03:23.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:03:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.781+0000 7fdc2b59e640 10 monclient: get_monmap_and_config 2023-10-30T12:03:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.781+0000 7fdc2b59e640 10 monclient: build_initial_monmap 2023-10-30T12:03:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.781+0000 7fdc2b59e640 10 monclient: monmap: 2023-10-30T12:03:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:23.782768+0000 2023-10-30T12:03:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:23.782768+0000 2023-10-30T12:03:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:23.786 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 1 Processor -- start 2023-10-30T12:03:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 1 -- start start 2023-10-30T12:03:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient: init 2023-10-30T12:03:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T12:03:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc2c09aa50 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc2c071510 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc2c071910 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c09e010 con 0x7fdc2c071910 2023-10-30T12:03:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c09ae30 con 0x7fdc2c09aa50 2023-10-30T12:03:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c09fee0 con 0x7fdc2c071510 2023-10-30T12:03:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.782+0000 7fdc2b59e640 10 monclient(hunting): authenticate will time out at 1048.965956s 2023-10-30T12:03:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc2ad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdc2c071910 0x7fdc2c13ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40256/0 (socket says 172.21.15.50:40256) 2023-10-30T12:03:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc2ad9d640 1 -- 172.21.15.50:0/2350770738 learned_addr learned my addr 172.21.15.50:0/2350770738 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c1385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40676/0 (socket says 172.21.15.50:40676) 2023-10-30T12:03:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): get_auth_request con 0x7fdc2c071510 auth_method 0 2023-10-30T12:03:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient(hunting): handle_auth_done global_id 44294 payload 274 2023-10-30T12:03:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc29d9b640 1 -- 172.21.15.50:0/2350770738 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458361050 0 0) 0x7fdc2c09e010 con 0x7fdc2c071910 2023-10-30T12:03:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 1 -- 172.21.15.50:0/2350770738 >> v1:172.21.15.118:6789/0 conn(0x7fdc2c09aa50 legacy=0x7fdc2c133300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 1 -- 172.21.15.50:0/2350770738 >> v1:172.21.15.50:6789/0 conn(0x7fdc2c071910 legacy=0x7fdc2c13ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient: _finish_hunting 0 2023-10-30T12:03:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 1 monclient: found mon.noname-b 2023-10-30T12:03:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:03:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc23fff640 1 -- 172.21.15.50:0/2350770738 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c133a10 con 0x7fdc2c071510 2023-10-30T12:03:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.783+0000 7fdc29d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:03:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2350770738 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c1385f0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097d0 tx=0x7fdc1402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e59448433dfe90 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 1 -- 172.21.15.50:0/2350770738 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14031770 con 0x7fdc2c071510 2023-10-30T12:03:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:03:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: dump: 2023-10-30T12:03:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.804 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: _finish_auth 0 2023-10-30T12:03:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:03:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 1 -- 172.21.15.50:0/2350770738 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc140318d0 con 0x7fdc2c071510 2023-10-30T12:03:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 5 monclient: authenticate success, global_id 44294 2023-10-30T12:03:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:23.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:23.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:23.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 1 -- 172.21.15.50:0/2350770738 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14031c60 con 0x7fdc2c071510 2023-10-30T12:03:23.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 4 set_mon_vals no callback set 2023-10-30T12:03:23.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:23.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc29d9b640 10 monclient: dump: 2023-10-30T12:03:23.807 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.807 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.807 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.809 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 10 monclient: shutdown 2023-10-30T12:03:23.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:23.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2350770738 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 msgr2=0x7fdc2c1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 1 --2- 172.21.15.50:0/2350770738 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c1385f0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097d0 tx=0x7fdc1402b810 comp rx=0 tx=0).stop 2023-10-30T12:03:23.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2350770738 shutdown_connections 2023-10-30T12:03:23.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 1 --2- 172.21.15.50:0/2350770738 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c1385f0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2350770738 >> 172.21.15.50:0/2350770738 conn(0x7fdc2c0a09b0 msgr2=0x7fdc2c0a0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:23.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2350770738 shutdown_connections 2023-10-30T12:03:23.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.784+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2350770738 wait complete. 2023-10-30T12:03:23.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.785+0000 7fdc2b59e640 10 monclient: build_initial_monmap 2023-10-30T12:03:23.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.785+0000 7fdc2b59e640 10 monclient: monmap: 2023-10-30T12:03:23.811 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:23.811 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:23.811 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:23.811 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:23.812 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:23.812 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.812 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:23.812 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:23.812 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:23.813 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.785+0000 7fdc2b59e640 1 librados: starting msgr at 2023-10-30T12:03:23.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.785+0000 7fdc2b59e640 1 librados: starting objecter 2023-10-30T12:03:23.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.785+0000 7fdc2b59e640 1 Processor -- start 2023-10-30T12:03:23.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 -- start start 2023-10-30T12:03:23.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 librados: setting wanted keys 2023-10-30T12:03:23.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 librados: calling monclient init 2023-10-30T12:03:23.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient: init 2023-10-30T12:03:23.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:23.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:03:23.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c09b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdc2c071510 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:23.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c071910 0x7fdc2c09b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdc2c071910 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:23.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c09aa50 0x7fdc2c09bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdc2c09aa50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:23.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c12c950 con 0x7fdc2c09aa50 2023-10-30T12:03:23.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c12a760 con 0x7fdc2c071910 2023-10-30T12:03:23.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c12a4b0 con 0x7fdc2c071510 2023-10-30T12:03:23.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:23.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2b59e640 10 monclient(hunting): authenticate will time out at 1048.969531s 2023-10-30T12:03:23.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c09b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c09aa50 0x7fdc2c09bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c09b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40678/0 (socket says 172.21.15.50:40678) 2023-10-30T12:03:23.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 -- 172.21.15.50:0/2919534326 learned_addr learned my addr 172.21.15.50:0/2919534326 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:23.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): get_auth_request con 0x7fdc2c071510 auth_method 0 2023-10-30T12:03:23.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 10 monclient(hunting): get_auth_request con 0x7fdc2c09aa50 auth_method 0 2023-10-30T12:03:23.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c071910 0x7fdc2c09b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient(hunting): handle_auth_done global_id 44297 payload 824 2023-10-30T12:03:23.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c071910 msgr2=0x7fdc2c09b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c071910 0x7fdc2c09b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c09aa50 msgr2=0x7fdc2c09bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c09aa50 0x7fdc2c09bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:03:23.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 monclient: found mon.noname-c 2023-10-30T12:03:23.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:23.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:23.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc2a59c640 1 -- 172.21.15.50:0/2919534326 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc14009400 con 0x7fdc2c071510 2023-10-30T12:03:23.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc23fff640 10 monclient: get_auth_request con 0x7fdc2c071910 auth_method 0 2023-10-30T12:03:23.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.786+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c071910 0x7fdc2c09b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:03:23.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2a59c640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c09b1d0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00c450 tx=0x7fdc1c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 client.?.objecter ms_handle_connect 0x7fdc2c071510 2023-10-30T12:03:23.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:23.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c0052d0 con 0x7fdc2c071510 2023-10-30T12:03:23.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:23.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: dump: 2023-10-30T12:03:23.824 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.824 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.825 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.825 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.825 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.825 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.825 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.826 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.826 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.826 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: _finish_auth 0 2023-10-30T12:03:23.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:03:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c005430 con 0x7fdc2c071510 2023-10-30T12:03:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 5 monclient: authenticate success, global_id 44297 2023-10-30T12:03:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 10 monclient: _renew_subs 2023-10-30T12:03:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c1308d0 con 0x7fdc2c071510 2023-10-30T12:03:23.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:23.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c003830 con 0x7fdc2c071510 2023-10-30T12:03:23.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:23.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc22ffd640 10 monclient: dump: 2023-10-30T12:03:23.829 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.829 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.829 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.829 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.829 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.829 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.830 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.830 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.830 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.830 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2959a640 4 set_mon_vals no callback set 2023-10-30T12:03:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 10 client.44297.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 10 monclient: _renew_subs 2023-10-30T12:03:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c130110 con 0x7fdc2c071510 2023-10-30T12:03:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.787+0000 7fdc2b59e640 1 librados: init done 2023-10-30T12:03:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc2b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:23.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdc1c003990 con 0x7fdc2c071510 2023-10-30T12:03:23.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc2b59e640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c12b100 0x7fdc2c1ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc2b59e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c066a20 con 0x7fdc2c12b100 2023-10-30T12:03:23.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c12b100 0x7fdc2c1ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient: get_auth_request con 0x7fdc2c12b100 auth_method 0 2023-10-30T12:03:23.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc0c0392f0 0x7fdc0c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7733+0+0 (secure 0 0 0) 0x7fdc1c04a7b0 con 0x7fdc2c071510 2023-10-30T12:03:23.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 10 client.44297.objecter ms_dispatch 0x7fdc2c000b90 osd_map(79..79 src has 1..79) v4 2023-10-30T12:03:23.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 3 client.44297.objecter handle_osd_map got epochs [79,79] > 0 2023-10-30T12:03:23.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 3 client.44297.objecter handle_osd_map decoding full epoch 79 2023-10-30T12:03:23.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 10 monclient(hunting): handle_auth_done global_id 64128 payload 824 2023-10-30T12:03:23.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 1 -- 172.21.15.50:0/2919534326 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc2c0a01c0 con 0x7fdc2c12b100 2023-10-30T12:03:23.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 20 client.44297.objecter dump_active .. 0 homeless 2023-10-30T12:03:23.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc2ad9d640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc0c0392f0 0x7fdc0c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc2ad9d640 10 monclient: get_auth_request con 0x7fdc0c0392f0 auth_method 0 2023-10-30T12:03:23.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 10 client.44297.objecter ms_handle_connect 0x7fdc2c12b100 2023-10-30T12:03:23.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c12b100 0x7fdc2c1ca2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0668f0 tx=0x7fdc14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 10 client.44297.objecter resend_mon_ops 2023-10-30T12:03:23.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14032690 con 0x7fdc2c12b100 2023-10-30T12:03:23.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:23.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.788+0000 7fdc22ffd640 10 monclient: dump: 2023-10-30T12:03:23.837 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.837 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.837 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.838 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.838 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.838 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.838 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.838 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.839 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.839 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 10 client.44297.objecter ms_handle_connect 0x7fdc0c0392f0 2023-10-30T12:03:23.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc2ad9d640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc0c0392f0 0x7fdc0c03b7b0 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c09d120 tx=0x7fdc240016b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fdc140327f0 con 0x7fdc2c12b100 2023-10-30T12:03:23.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c12b100 msgr2=0x7fdc2c1ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c12b100 0x7fdc2c1ca2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0668f0 tx=0x7fdc14004730 comp rx=0 tx=0).stop 2023-10-30T12:03:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c12b100 msgr2=0x7fdc2c1ca2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.789+0000 7fdc22ffd640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c12b100 0x7fdc2c1ca2a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc2b59e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:23.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc2b59e640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c1ca7e0 0x7fdc2c1cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc2b59e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c09fc00 con 0x7fdc2c1ca7e0 2023-10-30T12:03:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c1ca7e0 0x7fdc2c1cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient: get_auth_request con 0x7fdc2c1ca7e0 auth_method 0 2023-10-30T12:03:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 10 monclient(hunting): handle_auth_done global_id 64131 payload 824 2023-10-30T12:03:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.795+0000 7fdc23fff640 1 -- 172.21.15.50:0/2919534326 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdc2c13c260 con 0x7fdc2c1ca7e0 2023-10-30T12:03:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c1ca7e0 0x7fdc2c1cabc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000c00 tx=0x7fdc140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 10 client.44297.objecter ms_handle_connect 0x7fdc2c1ca7e0 2023-10-30T12:03:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 10 client.44297.objecter resend_mon_ops 2023-10-30T12:03:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14003d50 con 0x7fdc2c1ca7e0 2023-10-30T12:03:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 10 monclient: dump: 2023-10-30T12:03:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:23.848 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7fdc14003eb0 con 0x7fdc2c1ca7e0 2023-10-30T12:03:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c1ca7e0 msgr2=0x7fdc2c1cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c1ca7e0 0x7fdc2c1cabc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000c00 tx=0x7fdc140046c0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c1ca7e0 msgr2=0x7fdc2c1cabc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc22ffd640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc2c1ca7e0 0x7fdc2c1cabc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.850 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:03:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 30, 2023-10-30T12:03:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:03:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:03:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:23.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.875 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:23.875 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:23.875 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:23.875 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:23.875 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:23.879 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:23.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 10 librados: watch_flush enter 2023-10-30T12:03:23.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 10 librados: watch_flush exit 2023-10-30T12:03:23.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 20 client.44297.objecter shutdown clearing up homeless session... 2023-10-30T12:03:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 10 client.44297.objecter successfully canceled tick 2023-10-30T12:03:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc0c0392f0 msgr2=0x7fdc0c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc0c0392f0 0x7fdc0c03b7b0 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c09d120 tx=0x7fdc240016b0 comp rx=0 tx=0).stop 2023-10-30T12:03:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 10 monclient: shutdown 2023-10-30T12:03:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 msgr2=0x7fdc2c09b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 --2- 172.21.15.50:0/2919534326 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc2c071510 0x7fdc2c09b1d0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00c450 tx=0x7fdc1c00c910 comp rx=0 tx=0).stop 2023-10-30T12:03:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2ad9d640 1 -- 172.21.15.50:0/2919534326 reap_dead start 2023-10-30T12:03:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2ad9d640 1 -- 172.21.15.50:0/2919534326 reap_dead start 2023-10-30T12:03:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 shutdown_connections 2023-10-30T12:03:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 >> 172.21.15.50:0/2919534326 conn(0x7fdc2c0a09b0 msgr2=0x7fdc2c09e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 shutdown_connections 2023-10-30T12:03:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 -- 172.21.15.50:0/2919534326 wait complete. 2023-10-30T12:03:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.796+0000 7fdc2b59e640 1 librados: shutdown 2023-10-30T12:03:23.884 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:03:23.885 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:03:23.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:03:23.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.989+0000 7faf67d90640 10 monclient: get_monmap_and_config 2023-10-30T12:03:23.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.989+0000 7faf67d90640 10 monclient: build_initial_monmap 2023-10-30T12:03:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.989+0000 7faf67d90640 10 monclient: monmap: 2023-10-30T12:03:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:23.990964+0000 2023-10-30T12:03:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:23.990964+0000 2023-10-30T12:03:23.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:23.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:23.993 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:23.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:23.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:23.994 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.990+0000 7faf67d90640 1 Processor -- start 2023-10-30T12:03:24.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.990+0000 7faf67d90640 1 -- start start 2023-10-30T12:03:24.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.990+0000 7faf67d90640 10 monclient: init 2023-10-30T12:03:24.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:24.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-10-30T12:03:24.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf60097780 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:24.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7faf60073b10 0x7faf6013aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient(hunting): picked mon.noname-c con 0x7faf60073b10 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:24.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf6013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient(hunting): picked mon.noname-b con 0x7faf6009ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:24.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf60071ec0 con 0x7faf60097780 2023-10-30T12:03:24.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6009bd10 con 0x7faf60073b10 2023-10-30T12:03:24.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6009bff0 con 0x7faf6009ccd0 2023-10-30T12:03:24.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:24.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf67d90640 10 monclient(hunting): authenticate will time out at 1049.174270s 2023-10-30T12:03:24.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf65b2c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7faf60097780 0x7faf60097b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40268/0 (socket says 172.21.15.50:40268) 2023-10-30T12:03:24.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf65b2c640 1 -- 172.21.15.50:0/3742678902 learned_addr learned my addr 172.21.15.50:0/3742678902 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:24.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 1 --2- 172.21.15.50:0/3742678902 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf6013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): get_auth_request con 0x7faf6009ccd0 auth_method 0 2023-10-30T12:03:24.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 1 --2- 172.21.15.50:0/3742678902 >> v2:172.21.15.118:3300/0 conn(0x7faf60073b10 0x7faf6013aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 10 monclient(hunting): get_auth_request con 0x7faf60073b10 auth_method 0 2023-10-30T12:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf64b2a640 1 -- 172.21.15.50:0/3742678902 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290132372 0 0) 0x7faf60071ec0 con 0x7faf60097780 2023-10-30T12:03:24.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6532b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf64b2a640 10 monclient(hunting): _negotiate 2023-10-30T12:03:24.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf64b2a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf64b2a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf64b2a640 10 monclient(hunting): my global_id is 64134 2023-10-30T12:03:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf64b2a640 1 -- 172.21.15.50:0/3742678902 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4c003520 con 0x7faf60097780 2023-10-30T12:03:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient(hunting): handle_auth_done global_id 44300 payload 274 2023-10-30T12:03:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 1 -- 172.21.15.50:0/3742678902 >> v2:172.21.15.118:3300/0 conn(0x7faf60073b10 msgr2=0x7faf6013aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 1 --2- 172.21.15.50:0/3742678902 >> v2:172.21.15.118:3300/0 conn(0x7faf60073b10 0x7faf6013aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 1 -- 172.21.15.50:0/3742678902 >> v1:172.21.15.50:6789/0 conn(0x7faf60097780 legacy=0x7faf60097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient: _finish_hunting 0 2023-10-30T12:03:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 1 monclient: found mon.noname-b 2023-10-30T12:03:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:24.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:03:24.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.991+0000 7faf6632d640 1 -- 172.21.15.50:0/3742678902 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf60071b20 con 0x7faf6009ccd0 2023-10-30T12:03:24.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf6632d640 1 --2- 172.21.15.50:0/3742678902 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf6013cfe0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7faf58009d90 tx=0x7faf5802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8527b36ece4dbd91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 1 -- 172.21.15.50:0/3742678902 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5802cb30 con 0x7faf6009ccd0 2023-10-30T12:03:24.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:03:24.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: dump: 2023-10-30T12:03:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.014 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.014 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.014 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.014 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: _finish_auth 0 2023-10-30T12:03:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: _check_auth_tickets 2023-10-30T12:03:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 1 -- 172.21.15.50:0/3742678902 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5802cc90 con 0x7faf6009ccd0 2023-10-30T12:03:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf67d90640 5 monclient: authenticate success, global_id 44300 2023-10-30T12:03:24.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf67d90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:24.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:24.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf67d90640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:24.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 1 -- 172.21.15.50:0/3742678902 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58034440 con 0x7faf6009ccd0 2023-10-30T12:03:24.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf67d90640 4 set_mon_vals no callback set 2023-10-30T12:03:24.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:24.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.992+0000 7faf64b2a640 10 monclient: dump: 2023-10-30T12:03:24.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.017 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.017 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.018 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.018 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.018 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 10 monclient: shutdown 2023-10-30T12:03:24.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:24.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 -- 172.21.15.50:0/3742678902 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 msgr2=0x7faf6013cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 --2- 172.21.15.50:0/3742678902 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf6013cfe0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7faf58009d90 tx=0x7faf5802bd80 comp rx=0 tx=0).stop 2023-10-30T12:03:24.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 -- 172.21.15.50:0/3742678902 shutdown_connections 2023-10-30T12:03:24.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 --2- 172.21.15.50:0/3742678902 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf6013cfe0 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 --2- 172.21.15.50:0/3742678902 >> v2:172.21.15.118:3300/0 conn(0x7faf60073b10 0x7faf6013aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 -- 172.21.15.50:0/3742678902 >> 172.21.15.50:0/3742678902 conn(0x7faf600a1170 msgr2=0x7faf600a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:24.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 -- 172.21.15.50:0/3742678902 shutdown_connections 2023-10-30T12:03:24.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 -- 172.21.15.50:0/3742678902 wait complete. 2023-10-30T12:03:24.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 10 monclient: build_initial_monmap 2023-10-30T12:03:24.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 10 monclient: monmap: 2023-10-30T12:03:24.021 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:24.021 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:24.021 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:24.021 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:24.022 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:24.022 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.022 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:24.022 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:24.022 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:24.023 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 librados: starting msgr at 2023-10-30T12:03:24.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.993+0000 7faf67d90640 1 librados: starting objecter 2023-10-30T12:03:24.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 Processor -- start 2023-10-30T12:03:24.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 -- start start 2023-10-30T12:03:24.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 librados: setting wanted keys 2023-10-30T12:03:24.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 librados: calling monclient init 2023-10-30T12:03:24.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient: init 2023-10-30T12:03:24.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:24.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:03:24.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf60073b10 0x7faf60094740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): picked mon.noname-a con 0x7faf60073b10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:24.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf60097780 0x7faf601c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): picked mon.noname-b con 0x7faf60097780 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:24.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf601c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): picked mon.noname-c con 0x7faf6009ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:24.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6013c950 con 0x7faf60073b10 2023-10-30T12:03:24.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6008d220 con 0x7faf60097780 2023-10-30T12:03:24.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6013c3c0 con 0x7faf6009ccd0 2023-10-30T12:03:24.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:24.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf67d90640 10 monclient(hunting): authenticate will time out at 1049.177569s 2023-10-30T12:03:24.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf60073b10 0x7faf60094740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf601c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf601c6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40706/0 (socket says 172.21.15.50:40706) 2023-10-30T12:03:24.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 -- 172.21.15.50:0/1279560749 learned_addr learned my addr 172.21.15.50:0/1279560749 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:24.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf60073b10 0x7faf60094740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60792/0 (socket says 172.21.15.50:60792) 2023-10-30T12:03:24.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): get_auth_request con 0x7faf6009ccd0 auth_method 0 2023-10-30T12:03:24.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 10 monclient(hunting): get_auth_request con 0x7faf60073b10 auth_method 0 2023-10-30T12:03:24.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf60097780 0x7faf601c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 10 monclient(hunting): get_auth_request con 0x7faf60097780 auth_method 0 2023-10-30T12:03:24.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6532b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient(hunting): handle_auth_done global_id 44303 payload 824 2023-10-30T12:03:24.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf60097780 msgr2=0x7faf601c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf60097780 0x7faf601c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf60073b10 msgr2=0x7faf60094740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf60073b10 0x7faf60094740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient: _finish_hunting 0 2023-10-30T12:03:24.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 monclient: found mon.noname-c 2023-10-30T12:03:24.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:24.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:24.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf6632d640 1 -- 172.21.15.50:0/1279560749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf58009a70 con 0x7faf6009ccd0 2023-10-30T12:03:24.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.994+0000 7faf65b2c640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf60073b10 0x7faf60094740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:24.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf6632d640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf601c6eb0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7faf58009d90 tx=0x7faf58004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 client.?.objecter ms_handle_connect 0x7faf6009ccd0 2023-10-30T12:03:24.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:24.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58034e50 con 0x7faf6009ccd0 2023-10-30T12:03:24.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:24.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: dump: 2023-10-30T12:03:24.035 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.035 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.036 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.036 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.036 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.036 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.036 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.036 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.037 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.037 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: _finish_auth 0 2023-10-30T12:03:24.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: _check_auth_tickets 2023-10-30T12:03:24.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:24.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf58040450 con 0x7faf6009ccd0 2023-10-30T12:03:24.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 5 monclient: authenticate success, global_id 44303 2023-10-30T12:03:24.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 10 monclient: _renew_subs 2023-10-30T12:03:24.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:24.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6008c970 con 0x7faf6009ccd0 2023-10-30T12:03:24.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58040800 con 0x7faf6009ccd0 2023-10-30T12:03:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf567fc640 10 monclient: dump: 2023-10-30T12:03:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.040 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.040 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.040 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.040 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.041 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf57fff640 4 set_mon_vals no callback set 2023-10-30T12:03:24.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 10 client.44303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:24.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 10 monclient: _renew_subs 2023-10-30T12:03:24.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:24.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf601c73f0 con 0x7faf6009ccd0 2023-10-30T12:03:24.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.995+0000 7faf67d90640 1 librados: init done 2023-10-30T12:03:24.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf67d90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:24.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf67d90640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf60094c80 0x7faf601ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf67d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6006a5a0 con 0x7faf60094c80 2023-10-30T12:03:24.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7faf58040960 con 0x7faf6009ccd0 2023-10-30T12:03:24.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf60094c80 0x7faf601ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient: get_auth_request con 0x7faf60094c80 auth_method 0 2023-10-30T12:03:24.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf440392f0 0x7faf4403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7733+0+0 (secure 0 0 0) 0x7faf5806dd10 con 0x7faf6009ccd0 2023-10-30T12:03:24.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 10 client.44303.objecter ms_dispatch 0x7faf60000b90 osd_map(79..79 src has 1..79) v4 2023-10-30T12:03:24.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 3 client.44303.objecter handle_osd_map got epochs [79,79] > 0 2023-10-30T12:03:24.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 3 client.44303.objecter handle_osd_map decoding full epoch 79 2023-10-30T12:03:24.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 10 monclient(hunting): handle_auth_done global_id 44306 payload 824 2023-10-30T12:03:24.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 1 -- 172.21.15.50:0/1279560749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf6008afd0 con 0x7faf60094c80 2023-10-30T12:03:24.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 20 client.44303.objecter dump_active .. 0 homeless 2023-10-30T12:03:24.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 10 client.44303.objecter ms_handle_connect 0x7faf60094c80 2023-10-30T12:03:24.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 10 client.44303.objecter resend_mon_ops 2023-10-30T12:03:24.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf65b2c640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf60094c80 0x7faf601ca690 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf6532b640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf440392f0 0x7faf4403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf6532b640 10 monclient: get_auth_request con 0x7faf440392f0 auth_method 0 2023-10-30T12:03:24.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c00faa0 con 0x7faf60094c80 2023-10-30T12:03:24.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:24.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.996+0000 7faf567fc640 10 monclient: dump: 2023-10-30T12:03:24.048 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.048 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.048 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.048 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.048 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.048 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.049 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.049 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.049 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.049 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7faf5c00fc00 con 0x7faf60094c80 2023-10-30T12:03:24.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:24.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:24.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf60094c80 msgr2=0x7faf601ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf60094c80 0x7faf601ca690 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c007700 comp rx=0 tx=0).stop 2023-10-30T12:03:24.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf60094c80 msgr2=0x7faf601ca690 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:24.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf60094c80 0x7faf601ca690 secure :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c007700 comp rx=0 tx=0).stop 2023-10-30T12:03:24.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf567fc640 10 client.44303.objecter ms_handle_connect 0x7faf440392f0 2023-10-30T12:03:24.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:23.997+0000 7faf6532b640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf440392f0 0x7faf4403b7b0 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7faf6013d4f0 tx=0x7faf50009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf67d90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:24.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf67d90640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6013d520 0x7faf6002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf67d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6008cd30 con 0x7faf6013d520 2023-10-30T12:03:24.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6013d520 0x7faf6002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient: get_auth_request con 0x7faf6013d520 auth_method 0 2023-10-30T12:03:24.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 10 monclient(hunting): handle_auth_done global_id 44309 payload 824 2023-10-30T12:03:24.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 1 -- 172.21.15.50:0/1279560749 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faf601c7580 con 0x7faf6013d520 2023-10-30T12:03:24.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf567fc640 10 client.44303.objecter ms_handle_connect 0x7faf6013d520 2023-10-30T12:03:24.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf65b2c640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6013d520 0x7faf6002b430 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c001150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.003+0000 7faf567fc640 10 client.44303.objecter resend_mon_ops 2023-10-30T12:03:24.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c023070 con 0x7faf6013d520 2023-10-30T12:03:24.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:24.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 10 monclient: dump: 2023-10-30T12:03:24.056 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.057 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.057 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.057 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.057 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.057 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.057 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.058 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.058 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.058 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7faf5c01e500 con 0x7faf6013d520 2023-10-30T12:03:24.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:24.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:24.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6013d520 msgr2=0x7faf6002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6013d520 0x7faf6002b430 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c001150 comp rx=0 tx=0).stop 2023-10-30T12:03:24.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6013d520 msgr2=0x7faf6002b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:24.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf567fc640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6013d520 0x7faf6002b430 secure :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f60 tx=0x7faf5c001150 comp rx=0 tx=0).stop 2023-10-30T12:03:24.060 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:03:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 30, 2023-10-30T12:03:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:03:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:03:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:24.072 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:24.072 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:24.072 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:24.072 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.072 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:24.074 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:24.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:24.074 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:24.074 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.075 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:24.075 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:24.075 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.076 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.076 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.076 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:24.076 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:24.076 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:24.077 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:24.077 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:24.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:24.078 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:24.078 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:24.078 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:24.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.078 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:24.078 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:24.079 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.079 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:24.079 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:24.080 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.080 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.080 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.080 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:24.081 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:24.081 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:24.081 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:24.081 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:24.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:24.082 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:24.082 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:24.082 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:24.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:24.083 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:24.083 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.084 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:24.084 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:24.084 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.084 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.085 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:24.085 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:24.085 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:24.085 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:24.085 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:24.086 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.086 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.086 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:24.086 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:24.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.087 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:24.087 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:24.087 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:24.087 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.087 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.088 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:03:24.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.088 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:24.088 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:24.088 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:24.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.089 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.089 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:24.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.089 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:24.089 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:24.090 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:24.090 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.090 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.090 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:24.091 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:24.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 10 librados: watch_flush enter 2023-10-30T12:03:24.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 10 librados: watch_flush exit 2023-10-30T12:03:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 20 client.44303.objecter shutdown clearing up homeless session... 2023-10-30T12:03:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 10 client.44303.objecter successfully canceled tick 2023-10-30T12:03:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf440392f0 msgr2=0x7faf4403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf440392f0 0x7faf4403b7b0 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7faf6013d4f0 tx=0x7faf50009190 comp rx=0 tx=0).stop 2023-10-30T12:03:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 10 monclient: shutdown 2023-10-30T12:03:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:24.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 msgr2=0x7faf601c6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf601c6eb0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7faf58009d90 tx=0x7faf58004a40 comp rx=0 tx=0).stop 2023-10-30T12:03:24.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf6632d640 1 -- 172.21.15.50:0/1279560749 reap_dead start 2023-10-30T12:03:24.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 shutdown_connections 2023-10-30T12:03:24.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 --2- 172.21.15.50:0/1279560749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf6009ccd0 0x7faf601c6eb0 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 >> 172.21.15.50:0/1279560749 conn(0x7faf600a1170 msgr2=0x7faf60098570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:24.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 shutdown_connections 2023-10-30T12:03:24.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 -- 172.21.15.50:0/1279560749 wait complete. 2023-10-30T12:03:24.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.004+0000 7faf67d90640 1 librados: shutdown 2023-10-30T12:03:24.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:03:24.096 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:03:24.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:03:24.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.201+0000 7f4c76965640 10 monclient: get_monmap_and_config 2023-10-30T12:03:24.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.201+0000 7f4c76965640 10 monclient: build_initial_monmap 2023-10-30T12:03:24.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.201+0000 7f4c76965640 10 monclient: monmap: 2023-10-30T12:03:24.203 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:24.204 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.204 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:24.202782+0000 2023-10-30T12:03:24.204 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:24.202782+0000 2023-10-30T12:03:24.204 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:24.204 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.205 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:24.205 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:24.205 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:24.205 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:24.205 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:24.206 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 1 Processor -- start 2023-10-30T12:03:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 1 -- start start 2023-10-30T12:03:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient: init 2023-10-30T12:03:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:03:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c7009bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient(hunting): picked mon.noname-b con 0x7f4c7009d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4c7009c660 0x7f4c70139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient(hunting): picked mon.noname-c con 0x7f4c7009c660 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:24.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c70098170 con 0x7f4c7009bb70 2023-10-30T12:03:24.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7009cf60 con 0x7f4c7009c660 2023-10-30T12:03:24.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70097b20 con 0x7f4c7009d3f0 2023-10-30T12:03:24.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:24.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.202+0000 7f4c76965640 10 monclient(hunting): authenticate will time out at 1049.385915s 2023-10-30T12:03:24.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70137960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40728/0 (socket says 172.21.15.50:40728) 2023-10-30T12:03:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/3992017114 learned_addr learned my addr 172.21.15.50:0/3992017114 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4c7009d3f0 auth_method 0 2023-10-30T12:03:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 1 --2- 172.21.15.50:0/3992017114 >> v2:172.21.15.118:3300/0 conn(0x7f4c7009c660 0x7f4c70139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 10 monclient(hunting): get_auth_request con 0x7f4c7009c660 auth_method 0 2023-10-30T12:03:24.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c74f02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_done global_id 44312 payload 274 2023-10-30T12:03:24.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/3992017114 >> v2:172.21.15.118:3300/0 conn(0x7f4c7009c660 msgr2=0x7f4c70139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:24.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 --2- 172.21.15.50:0/3992017114 >> v2:172.21.15.118:3300/0 conn(0x7f4c7009c660 0x7f4c70139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/3992017114 >> v1:172.21.15.118:6789/0 conn(0x7f4c7009bb70 legacy=0x7f4c7009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:24.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 monclient: found mon.noname-b 2023-10-30T12:03:24.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:24.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:03:24.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.203+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/3992017114 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c70097e00 con 0x7f4c7009d3f0 2023-10-30T12:03:24.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6f7fe640 1 --2- 172.21.15.50:0/3992017114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70137960 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009870 tx=0x7f4c6002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb3c6958a96a947 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 1 -- 172.21.15.50:0/3992017114 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c600317c0 con 0x7f4c7009d3f0 2023-10-30T12:03:24.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:03:24.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: dump: 2023-10-30T12:03:24.225 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.225 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.225 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.226 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.226 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.226 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.226 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.226 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.227 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.227 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: _finish_auth 0 2023-10-30T12:03:24.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: _check_auth_tickets 2023-10-30T12:03:24.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:24.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 1 -- 172.21.15.50:0/3992017114 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c60031920 con 0x7f4c7009d3f0 2023-10-30T12:03:24.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 5 monclient: authenticate success, global_id 44312 2023-10-30T12:03:24.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:24.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:24.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:24.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 1 -- 172.21.15.50:0/3992017114 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60031cb0 con 0x7f4c7009d3f0 2023-10-30T12:03:24.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 4 set_mon_vals no callback set 2023-10-30T12:03:24.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:24.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c6effd640 10 monclient: dump: 2023-10-30T12:03:24.229 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.229 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.230 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.230 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.230 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.230 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.230 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.231 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.231 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 10 monclient: shutdown 2023-10-30T12:03:24.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:24.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 -- 172.21.15.50:0/3992017114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 msgr2=0x7f4c70137960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 --2- 172.21.15.50:0/3992017114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70137960 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009870 tx=0x7f4c6002b860 comp rx=0 tx=0).stop 2023-10-30T12:03:24.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 -- 172.21.15.50:0/3992017114 shutdown_connections 2023-10-30T12:03:24.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 --2- 172.21.15.50:0/3992017114 >> v2:172.21.15.118:3300/0 conn(0x7f4c7009c660 0x7f4c70139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 --2- 172.21.15.50:0/3992017114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70137960 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 -- 172.21.15.50:0/3992017114 >> 172.21.15.50:0/3992017114 conn(0x7f4c700a3980 msgr2=0x7f4c700a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:24.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 -- 172.21.15.50:0/3992017114 shutdown_connections 2023-10-30T12:03:24.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.204+0000 7f4c76965640 1 -- 172.21.15.50:0/3992017114 wait complete. 2023-10-30T12:03:24.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.205+0000 7f4c76965640 10 monclient: build_initial_monmap 2023-10-30T12:03:24.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.205+0000 7f4c76965640 10 monclient: monmap: 2023-10-30T12:03:24.234 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:24.234 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:24.234 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:24.234 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:24.234 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:24.235 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.235 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:24.235 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:24.235 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:24.235 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.205+0000 7f4c76965640 1 librados: starting msgr at 2023-10-30T12:03:24.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.205+0000 7f4c76965640 1 librados: starting objecter 2023-10-30T12:03:24.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.205+0000 7f4c76965640 1 Processor -- start 2023-10-30T12:03:24.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 -- start start 2023-10-30T12:03:24.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 librados: setting wanted keys 2023-10-30T12:03:24.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 librados: calling monclient init 2023-10-30T12:03:24.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient: init 2023-10-30T12:03:24.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:24.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:03:24.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7009bb70 0x7f4c701ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): picked mon.noname-b con 0x7f4c7009bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:24.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c7009c660 0x7f4c70132780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): picked mon.noname-a con 0x7f4c7009c660 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:24.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70132cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): picked mon.noname-c con 0x7f4c7009d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:24.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70097190 con 0x7f4c7009c660 2023-10-30T12:03:24.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70133200 con 0x7f4c7009bb70 2023-10-30T12:03:24.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70134960 con 0x7f4c7009d3f0 2023-10-30T12:03:24.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:24.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c76965640 10 monclient(hunting): authenticate will time out at 1049.389485s 2023-10-30T12:03:24.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c7009c660 0x7f4c70132780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70132cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c7009c660 0x7f4c70132780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60796/0 (socket says 172.21.15.50:60796) 2023-10-30T12:03:24.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/2835533430 learned_addr learned my addr 172.21.15.50:0/2835533430 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:24.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4c7009c660 auth_method 0 2023-10-30T12:03:24.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 10 monclient(hunting): get_auth_request con 0x7f4c7009d3f0 auth_method 0 2023-10-30T12:03:24.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7009bb70 0x7f4c701ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request con 0x7f4c7009bb70 auth_method 0 2023-10-30T12:03:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_done global_id 64140 payload 824 2023-10-30T12:03:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 msgr2=0x7f4c70132cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70132cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7009bb70 msgr2=0x7f4c701ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7009bb70 0x7f4c701ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:03:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c6f7fe640 1 -- 172.21.15.50:0/2835533430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c600094d0 con 0x7f4c7009c660 2023-10-30T12:03:24.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.206+0000 7f4c74f02640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c7009d3f0 0x7f4c70132cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:24.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6f7fe640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c7009c660 0x7f4c70132780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c6002b8a0 tx=0x7f4c600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 client.?.objecter ms_handle_connect 0x7f4c7009c660 2023-10-30T12:03:24.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:24.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60032860 con 0x7f4c7009c660 2023-10-30T12:03:24.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:24.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: dump: 2023-10-30T12:03:24.248 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.248 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.249 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.249 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.249 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.249 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.249 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.250 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.250 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.250 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: _finish_auth 0 2023-10-30T12:03:24.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:03:24.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:24.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c600329c0 con 0x7f4c7009c660 2023-10-30T12:03:24.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 5 monclient: authenticate success, global_id 64140 2023-10-30T12:03:24.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 10 monclient: _renew_subs 2023-10-30T12:03:24.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c700976e0 con 0x7f4c7009c660 2023-10-30T12:03:24.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:24.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60032d70 con 0x7f4c7009c660 2023-10-30T12:03:24.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:24.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6cff9640 10 monclient: dump: 2023-10-30T12:03:24.253 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.253 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.253 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.253 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.254 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.254 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.254 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.254 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.254 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.255 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 10 client.64140.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c6e7fc640 4 set_mon_vals no callback set 2023-10-30T12:03:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 10 monclient: _renew_subs 2023-10-30T12:03:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c70129c20 con 0x7f4c7009c660 2023-10-30T12:03:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.207+0000 7f4c76965640 1 librados: init done 2023-10-30T12:03:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c76965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c76965640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013a3f0 0x7f4c7013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c76965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4c6004e450 con 0x7f4c7009c660 2023-10-30T12:03:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70129db0 con 0x7f4c7013a3f0 2023-10-30T12:03:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c5c0392f0 0x7f4c5c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013a3f0 0x7f4c7013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient: get_auth_request con 0x7f4c7013a3f0 auth_method 0 2023-10-30T12:03:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7733+0+0 (secure 0 0 0) 0x7f4c6006d2c0 con 0x7f4c7009c660 2023-10-30T12:03:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 10 client.64140.objecter ms_dispatch 0x7f4c70000b90 osd_map(79..79 src has 1..79) v4 2023-10-30T12:03:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 3 client.64140.objecter handle_osd_map got epochs [79,79] > 0 2023-10-30T12:03:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 3 client.64140.objecter handle_osd_map decoding full epoch 79 2023-10-30T12:03:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c74f02640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c5c0392f0 0x7f4c5c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c74f02640 10 monclient: get_auth_request con 0x7f4c5c0392f0 auth_method 0 2023-10-30T12:03:24.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_done global_id 27925 payload 824 2023-10-30T12:03:24.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 20 client.64140.objecter dump_active .. 0 homeless 2023-10-30T12:03:24.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 1 -- 172.21.15.50:0/2835533430 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c70097560 con 0x7f4c7013a3f0 2023-10-30T12:03:24.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 10 client.64140.objecter ms_handle_connect 0x7f4c7013a3f0 2023-10-30T12:03:24.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 10 client.64140.objecter resend_mon_ops 2023-10-30T12:03:24.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6ffff640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013a3f0 0x7f4c7013a7d0 secure :-1 s=READY pgs=7017 cs=0 l=1 rev1=1 crypto rx=0x7f4c64000fb0 tx=0x7f4c6400f9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c640052e0 con 0x7f4c7013a3f0 2023-10-30T12:03:24.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:24.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.208+0000 7f4c6cff9640 10 monclient: dump: 2023-10-30T12:03:24.261 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.262 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.262 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.262 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.262 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.262 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.262 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.263 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.263 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.263 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c74f02640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c5c0392f0 0x7f4c5c03b7b0 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7f4c70128850 tx=0x7f4c6800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 10 client.64140.objecter ms_handle_connect 0x7f4c5c0392f0 2023-10-30T12:03:24.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f4c64005440 con 0x7f4c7013a3f0 2023-10-30T12:03:24.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:24.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:24.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013a3f0 msgr2=0x7f4c7013a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013a3f0 0x7f4c7013a7d0 secure :-1 s=READY pgs=7017 cs=0 l=1 rev1=1 crypto rx=0x7f4c64000fb0 tx=0x7f4c6400f9d0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013a3f0 msgr2=0x7f4c7013a7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:24.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.209+0000 7f4c6cff9640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013a3f0 0x7f4c7013a7d0 unknown :-1 s=CLOSED pgs=7017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c76965640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:24.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c76965640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013ad10 0x7f4c70137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c76965640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70073ff0 con 0x7f4c7013ad10 2023-10-30T12:03:24.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c6ffff640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013ad10 0x7f4c70137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c6ffff640 10 monclient: get_auth_request con 0x7f4c7013ad10 auth_method 0 2023-10-30T12:03:24.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c6ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.215+0000 7f4c6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6ffff640 10 monclient(hunting): handle_auth_done global_id 27928 payload 824 2023-10-30T12:03:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6ffff640 1 -- 172.21.15.50:0/2835533430 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c7009f600 con 0x7f4c7013ad10 2023-10-30T12:03:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6ffff640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013ad10 0x7f4c70137b90 secure :-1 s=READY pgs=7018 cs=0 l=1 rev1=1 crypto rx=0x7f4c64017000 tx=0x7f4c640010d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 10 client.64140.objecter ms_handle_connect 0x7f4c7013ad10 2023-10-30T12:03:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 10 client.64140.objecter resend_mon_ops 2023-10-30T12:03:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c64024070 con 0x7f4c7013ad10 2023-10-30T12:03:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 10 monclient: dump: 2023-10-30T12:03:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.272 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f4c6401f2e0 con 0x7f4c7013ad10 2023-10-30T12:03:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013ad10 msgr2=0x7f4c70137b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013ad10 0x7f4c70137b90 secure :-1 s=READY pgs=7018 cs=0 l=1 rev1=1 crypto rx=0x7f4c64017000 tx=0x7f4c640010d0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013ad10 msgr2=0x7f4c70137b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.216+0000 7f4c6cff9640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c7013ad10 0x7f4c70137b90 unknown :-1 s=CLOSED pgs=7018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.274 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:24.274 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:24.274 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:24.274 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:03:24.274 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 30, 2023-10-30T12:03:24.275 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:24.275 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:03:24.275 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:24.275 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:24.275 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.276 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:03:24.276 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:24.276 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:24.276 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:24.276 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:24.277 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:24.277 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:24.277 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:24.277 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:24.277 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:24.278 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:24.278 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:24.278 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:24.278 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:24.278 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.279 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:24.279 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:24.279 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:24.279 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:24.279 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:24.280 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:24.280 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:24.280 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:24.280 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:24.280 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:24.280 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:24.281 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:24.281 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.281 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:24.281 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:24.282 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:24.282 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:24.282 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:24.282 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:24.282 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:24.283 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:24.283 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:24.283 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:24.283 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:24.283 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:24.283 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:24.284 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:24.284 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:24.284 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:24.284 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:24.284 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:24.284 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:24.285 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:24.285 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:24.285 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:24.285 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:24.286 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:24.286 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:24.286 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:24.286 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:24.286 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.287 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:24.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.287 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:24.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.287 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:24.287 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:24.288 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:24.288 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:24.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.288 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:24.288 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:24.288 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.289 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:24.289 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:24.289 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.290 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.290 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.290 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:24.290 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:24.291 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:24.291 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:24.291 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:24.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.291 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:24.292 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:24.292 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:24.292 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:24.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.292 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:24.292 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:24.293 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.293 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:24.293 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:24.293 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.294 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.294 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:24.294 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.295 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:24.296 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:24.296 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:24.296 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:24.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.296 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:24.296 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:24.297 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.297 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:24.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:24.297 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:24.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.298 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.298 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:24.298 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:24.299 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:24.299 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:24.299 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:24.299 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.299 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.300 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:24.300 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:24.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.300 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:24.300 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:24.301 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:24.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.301 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.301 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:03:24.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.302 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:24.302 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:24.302 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:24.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.302 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.303 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:24.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.303 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:24.303 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:24.303 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 10 2023-10-30T12:03:24.304 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.304 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:24.304 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:03:24.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:24.304 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:24.305 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:24.305 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:24.305 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:24.305 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:24.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:24.305 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:24.306 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:24.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 10 librados: watch_flush enter 2023-10-30T12:03:24.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 10 librados: watch_flush exit 2023-10-30T12:03:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 20 client.64140.objecter shutdown clearing up homeless session... 2023-10-30T12:03:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 10 client.64140.objecter successfully canceled tick 2023-10-30T12:03:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c5c0392f0 msgr2=0x7f4c5c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c5c0392f0 0x7f4c5c03b7b0 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7f4c70128850 tx=0x7f4c6800b040 comp rx=0 tx=0).stop 2023-10-30T12:03:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 10 monclient: shutdown 2023-10-30T12:03:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c7009c660 msgr2=0x7f4c70132780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c7009c660 0x7f4c70132780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c6002b8a0 tx=0x7f4c600046c0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c74f02640 1 -- 172.21.15.50:0/2835533430 reap_dead start 2023-10-30T12:03:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 shutdown_connections 2023-10-30T12:03:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 --2- 172.21.15.50:0/2835533430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c7009c660 0x7f4c70132780 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 >> 172.21.15.50:0/2835533430 conn(0x7f4c700a3980 msgr2=0x7f4c70133830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 shutdown_connections 2023-10-30T12:03:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 -- 172.21.15.50:0/2835533430 wait complete. 2023-10-30T12:03:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.217+0000 7f4c76965640 1 librados: shutdown 2023-10-30T12:03:24.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:03:24.311 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:03:24.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:03:24.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.435+0000 7fa69e5cc640 10 monclient: get_monmap_and_config 2023-10-30T12:03:24.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.435+0000 7fa69e5cc640 10 monclient: build_initial_monmap 2023-10-30T12:03:24.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.435+0000 7fa69e5cc640 10 monclient: monmap: 2023-10-30T12:03:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:24.436650+0000 2023-10-30T12:03:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:24.436650+0000 2023-10-30T12:03:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:24.439 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.439 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:24.439 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:24.439 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:24.439 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:24.440 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:24.440 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 Processor -- start 2023-10-30T12:03:24.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 -- start start 2023-10-30T12:03:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient: init 2023-10-30T12:03:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T12:03:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa6980a2dd0 0x7fa6980a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): picked mon.noname-c con 0x7fa6980a2dd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:24.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa69806a8e0 0x7fa6980a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): picked mon.noname-a con 0x7fa69806a8e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:24.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6980a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): picked mon.noname-b con 0x7fa698071400 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6980a31b0 con 0x7fa69806a8e0 2023-10-30T12:03:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa69806a6a0 con 0x7fa6980a2dd0 2023-10-30T12:03:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69806be90 con 0x7fa698071400 2023-10-30T12:03:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69e5cc640 10 monclient(hunting): authenticate will time out at 1049.619793s 2023-10-30T12:03:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69cb69640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6980a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa6977fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa69806a8e0 0x7fa6980a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69cb69640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6980a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40740/0 (socket says 172.21.15.50:40740) 2023-10-30T12:03:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69cb69640 1 -- 172.21.15.50:0/574720913 learned_addr learned my addr 172.21.15.50:0/574720913 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69cb69640 10 monclient(hunting): get_auth_request con 0x7fa698071400 auth_method 0 2023-10-30T12:03:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69cb69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69cb69640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa69cb69640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa6977fe640 10 monclient(hunting): get_auth_request con 0x7fa69806a8e0 auth_method 0 2023-10-30T12:03:24.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa6977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa6977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.436+0000 7fa6977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa69cb69640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa69cb69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa69cb69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa697fff640 1 --2- 172.21.15.50:0/574720913 >> v2:172.21.15.118:3300/0 conn(0x7fa6980a2dd0 0x7fa6980a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa697fff640 10 monclient(hunting): get_auth_request con 0x7fa6980a2dd0 auth_method 0 2023-10-30T12:03:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa697fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa697fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 10 monclient(hunting): handle_auth_done global_id 64146 payload 274 2023-10-30T12:03:24.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 1 -- 172.21.15.50:0/574720913 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 msgr2=0x7fa6980a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:24.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 1 --2- 172.21.15.50:0/574720913 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6980a5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 1 -- 172.21.15.50:0/574720913 >> v2:172.21.15.118:3300/0 conn(0x7fa6980a2dd0 msgr2=0x7fa6980a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:24.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 1 --2- 172.21.15.50:0/574720913 >> v2:172.21.15.118:3300/0 conn(0x7fa6980a2dd0 0x7fa6980a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:24.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 1 monclient: found mon.noname-a 2023-10-30T12:03:24.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:24.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa6977fe640 1 -- 172.21.15.50:0/574720913 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69809e190 con 0x7fa69806a8e0 2023-10-30T12:03:24.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.437+0000 7fa697fff640 1 --2- 172.21.15.50:0/574720913 >> v2:172.21.15.118:3300/0 conn(0x7fa6980a2dd0 0x7fa6980a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:03:24.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa6977fe640 1 --2- 172.21.15.50:0/574720913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa69806a8e0 0x7fa6980a4c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa69806af70 tx=0x7fa68c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88d8327c30932a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 1 -- 172.21.15.50:0/574720913 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c030710 con 0x7fa69806a8e0 2023-10-30T12:03:24.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:24.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: dump: 2023-10-30T12:03:24.460 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.460 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.461 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.461 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.461 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.461 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.461 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.462 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.462 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.462 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: _finish_auth 0 2023-10-30T12:03:24.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:03:24.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:24.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 1 -- 172.21.15.50:0/574720913 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c02e050 con 0x7fa69806a8e0 2023-10-30T12:03:24.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 5 monclient: authenticate success, global_id 64146 2023-10-30T12:03:24.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:24.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:24.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:24.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 1 -- 172.21.15.50:0/574720913 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c030a90 con 0x7fa69806a8e0 2023-10-30T12:03:24.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 4 set_mon_vals no callback set 2023-10-30T12:03:24.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:24.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa696ffd640 10 monclient: dump: 2023-10-30T12:03:24.465 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.465 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.465 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.465 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.466 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.466 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.466 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.466 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.466 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.467 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 10 monclient: shutdown 2023-10-30T12:03:24.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:24.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 -- 172.21.15.50:0/574720913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa69806a8e0 msgr2=0x7fa6980a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/574720913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa69806a8e0 0x7fa6980a4c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa69806af70 tx=0x7fa68c02b710 comp rx=0 tx=0).stop 2023-10-30T12:03:24.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 -- 172.21.15.50:0/574720913 shutdown_connections 2023-10-30T12:03:24.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/574720913 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6980a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/574720913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa69806a8e0 0x7fa6980a4c00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/574720913 >> v2:172.21.15.118:3300/0 conn(0x7fa6980a2dd0 0x7fa6980a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 -- 172.21.15.50:0/574720913 >> 172.21.15.50:0/574720913 conn(0x7fa6980a22b0 msgr2=0x7fa6980a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:24.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 -- 172.21.15.50:0/574720913 shutdown_connections 2023-10-30T12:03:24.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.438+0000 7fa69e5cc640 1 -- 172.21.15.50:0/574720913 wait complete. 2023-10-30T12:03:24.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 10 monclient: build_initial_monmap 2023-10-30T12:03:24.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 10 monclient: monmap: 2023-10-30T12:03:24.470 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:24.470 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:24.470 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:24.470 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:24.470 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:24.471 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.471 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:24.471 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:24.471 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:24.471 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 1 librados: starting msgr at 2023-10-30T12:03:24.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 1 librados: starting objecter 2023-10-30T12:03:24.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 1 Processor -- start 2023-10-30T12:03:24.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 1 -- start start 2023-10-30T12:03:24.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 1 librados: setting wanted keys 2023-10-30T12:03:24.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 1 librados: calling monclient init 2023-10-30T12:03:24.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.439+0000 7fa69e5cc640 10 monclient: init 2023-10-30T12:03:24.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:24.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:03:24.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6981ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): picked mon.noname-c con 0x7fa698071400 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:24.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6980a2dd0 0x7fa698122510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): picked mon.noname-b con 0x7fa6980a2dd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:24.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa698122a50 0x7fa698122f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): picked mon.noname-a con 0x7fa698122a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:24.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa698124e30 con 0x7fa698122a50 2023-10-30T12:03:24.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6981262e0 con 0x7fa6980a2dd0 2023-10-30T12:03:24.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:24.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa698126530 con 0x7fa698071400 2023-10-30T12:03:24.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:24.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient(hunting): authenticate will time out at 1049.623258s 2023-10-30T12:03:24.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6981ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa698122a50 0x7fa698122f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6981ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40752/0 (socket says 172.21.15.50:40752) 2023-10-30T12:03:24.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 1 -- 172.21.15.50:0/275689696 learned_addr learned my addr 172.21.15.50:0/275689696 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:24.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): get_auth_request con 0x7fa698122a50 auth_method 0 2023-10-30T12:03:24.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 10 monclient(hunting): get_auth_request con 0x7fa698071400 auth_method 0 2023-10-30T12:03:24.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6980a2dd0 0x7fa698122510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa697fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 10 monclient(hunting): get_auth_request con 0x7fa6980a2dd0 auth_method 0 2023-10-30T12:03:24.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:24.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:24.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:24.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:24.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:24.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa6977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:24.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient(hunting): handle_auth_done global_id 64149 payload 824 2023-10-30T12:03:24.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 -- 172.21.15.50:0/275689696 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 msgr2=0x7fa6981ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6981ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 -- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6980a2dd0 msgr2=0x7fa698122510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6980a2dd0 0x7fa698122510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient: _finish_hunting 0 2023-10-30T12:03:24.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 monclient: found mon.noname-a 2023-10-30T12:03:24.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:24.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 -- 172.21.15.50:0/275689696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c009400 con 0x7fa698122a50 2023-10-30T12:03:24.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69cb69640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa698122a50 0x7fa698122f00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa690009d90 tx=0x7fa69000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 client.?.objecter ms_handle_connect 0x7fa698122a50 2023-10-30T12:03:24.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:24.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 1 -- 172.21.15.50:0/275689696 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69000cd20 con 0x7fa698122a50 2023-10-30T12:03:24.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:24.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 monclient: dump: 2023-10-30T12:03:24.484 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.484 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.485 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.485 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.485 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.485 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.485 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.485 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.486 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.486 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 monclient: _finish_auth 0 2023-10-30T12:03:24.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:03:24.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:24.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 1 -- 172.21.15.50:0/275689696 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69000f040 con 0x7fa698122a50 2023-10-30T12:03:24.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 5 monclient: authenticate success, global_id 64149 2023-10-30T12:03:24.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient: _renew_subs 2023-10-30T12:03:24.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69806b100 con 0x7fa698122a50 2023-10-30T12:03:24.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.440+0000 7fa694ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:24.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa694ff9640 1 -- 172.21.15.50:0/275689696 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690003bb0 con 0x7fa698122a50 2023-10-30T12:03:24.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:24.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa694ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:24.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa694ff9640 10 monclient: dump: 2023-10-30T12:03:24.488 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:24.489 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:24.489 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.489 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:24.489 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:24.489 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:24.489 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:24.490 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:24.490 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:24.490 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:24.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa6967fc640 4 set_mon_vals no callback set 2023-10-30T12:03:24.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 10 client.64149.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:24.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 10 monclient: _renew_subs 2023-10-30T12:03:24.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa698125c90 con 0x7fa698122a50 2023-10-30T12:03:24.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 1 librados: init done 2023-10-30T12:03:24.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa694ff9640 1 -- 172.21.15.50:0/275689696 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa690003d10 con 0x7fa698122a50 2023-10-30T12:03:24.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:24.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:24.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69804e160 con 0x7fa698122a50 2023-10-30T12:03:24.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.441+0000 7fa694ff9640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6840392f0 0x7fa68403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:24.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa694ff9640 1 -- 172.21.15.50:0/275689696 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fa69004b710 con 0x7fa698122a50 2023-10-30T12:03:24.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa694ff9640 10 client.64149.objecter ms_dispatch 0x7fa698000b90 osd_map(80..80 src has 1..80) v4 2023-10-30T12:03:24.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa694ff9640 3 client.64149.objecter handle_osd_map got epochs [80,80] > 0 2023-10-30T12:03:24.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa694ff9640 3 client.64149.objecter handle_osd_map decoding full epoch 80 2023-10-30T12:03:24.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa694ff9640 20 client.64149.objecter dump_active .. 0 homeless 2023-10-30T12:03:24.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa697fff640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6840392f0 0x7fa68403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:24.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa697fff640 10 monclient: get_auth_request con 0x7fa6840392f0 auth_method 0 2023-10-30T12:03:24.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa694ff9640 10 client.64149.objecter ms_handle_connect 0x7fa6840392f0 2023-10-30T12:03:24.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.442+0000 7fa697fff640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6840392f0 0x7fa68403b7b0 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7fa688006d40 tx=0x7fa688008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:24.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.446+0000 7fa694ff9640 1 -- 172.21.15.50:0/275689696 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa690004830 con 0x7fa698122a50 2023-10-30T12:03:24.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.446+0000 7fa694ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:24.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.446+0000 7fa694ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.691+0000 7fa69e5cc640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:03:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.691+0000 7fa69e5cc640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:03:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.691+0000 7fa69e5cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.691+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa698126670 con 0x7fa698122a50 2023-10-30T12:03:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.692+0000 7fa694ff9640 1 -- 172.21.15.50:0/275689696 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fa698126670 con 0x7fa698122a50 2023-10-30T12:03:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.692+0000 7fa694ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:03:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.692+0000 7fa694ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:24.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.692+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.692+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=1,health=12,kv=9,logm=65} crc {auth=3656679921,config=1228700967,health=1401171712,kv=2905431493,logm=644428195}) 2023-10-30T12:03:24.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.692+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=2479037165,config=3140514009,health=3725358826,kv=109951669,logm=3706655748}) 2023-10-30T12:03:24.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.692+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.697 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.692+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=1,health=12,kv=9,logm=65} crc {auth=3656679921,config=1228700967,health=1401171712,kv=2905431493,logm=644428195}) 2023-10-30T12:03:24.697 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.692+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=7,logm=66} crc {auth=3656679921,config=3140514009,health=3725358826,kv=3514218911,logm=597702772}) 2023-10-30T12:03:24.697 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.693+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.697 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.693+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305076365}) 2023-10-30T12:03:24.697 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.693+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542428415}) 2023-10-30T12:03:24.698 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.693+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.698 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.693+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305076365}) 2023-10-30T12:03:24.698 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.693+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171326758}) 2023-10-30T12:03:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 10 librados: watch_flush enter 2023-10-30T12:03:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 10 librados: watch_flush exit 2023-10-30T12:03:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 20 client.64149.objecter shutdown clearing up homeless session... 2023-10-30T12:03:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 10 client.64149.objecter successfully canceled tick 2023-10-30T12:03:24.700 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.694+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.700 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.694+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70830222}) 2023-10-30T12:03:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6840392f0 msgr2=0x7fa68403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6840392f0 0x7fa68403b7b0 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7fa688006d40 tx=0x7fa688008040 comp rx=0 tx=0).stop 2023-10-30T12:03:24.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 10 monclient: shutdown 2023-10-30T12:03:24.701 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.694+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307102896}) 2023-10-30T12:03:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa698122a50 msgr2=0x7fa698122f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa698122a50 0x7fa698122f00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa690009d90 tx=0x7fa69000c490 comp rx=0 tx=0).stop 2023-10-30T12:03:24.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.694+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.694+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70830222}) 2023-10-30T12:03:24.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.694+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3312012708}) 2023-10-30T12:03:24.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 shutdown_connections 2023-10-30T12:03:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6840392f0 0x7fa68403b7b0 unknown :-1 s=CLOSED pgs=3869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa698122a50 0x7fa698122f00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6980a2dd0 0x7fa698122510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 --2- 172.21.15.50:0/275689696 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa698071400 0x7fa6981ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 >> 172.21.15.50:0/275689696 conn(0x7fa6980a22b0 msgr2=0x7fa69806e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 shutdown_connections 2023-10-30T12:03:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 -- 172.21.15.50:0/275689696 wait complete. 2023-10-30T12:03:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:24.694+0000 7fa69e5cc640 1 librados: shutdown 2023-10-30T12:03:24.710 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.695+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.695+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892724554}) 2023-10-30T12:03:24.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.695+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=446660158}) 2023-10-30T12:03:24.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.695+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.695+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892724554}) 2023-10-30T12:03:24.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.695+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1508851226}) 2023-10-30T12:03:24.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.696+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.696+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277723379}) 2023-10-30T12:03:24.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.696+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3756925945}) 2023-10-30T12:03:24.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.696+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.696+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277723379}) 2023-10-30T12:03:24.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.696+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753502101}) 2023-10-30T12:03:24.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.697+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.697+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751684282}) 2023-10-30T12:03:24.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.697+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1056590725}) 2023-10-30T12:03:24.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.697+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.697+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751684282}) 2023-10-30T12:03:24.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.697+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1869299868}) 2023-10-30T12:03:24.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3077299651}) 2023-10-30T12:03:24.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1659649647}) 2023-10-30T12:03:24.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3077299651}) 2023-10-30T12:03:24.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3825317033}) 2023-10-30T12:03:24.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365676416}) 2023-10-30T12:03:24.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077177104}) 2023-10-30T12:03:24.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365676416}) 2023-10-30T12:03:24.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.698+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3029870053}) 2023-10-30T12:03:24.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.699+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.699+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3926733898}) 2023-10-30T12:03:24.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.699+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1277636314}) 2023-10-30T12:03:24.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.699+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.699+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3926733898}) 2023-10-30T12:03:24.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.699+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424333399}) 2023-10-30T12:03:24.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571059531}) 2023-10-30T12:03:24.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962402772}) 2023-10-30T12:03:24.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571059531}) 2023-10-30T12:03:24.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421164911}) 2023-10-30T12:03:24.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032362298}) 2023-10-30T12:03:24.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452110133}) 2023-10-30T12:03:24.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032362298}) 2023-10-30T12:03:24.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.700+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718639974}) 2023-10-30T12:03:24.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.701+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.701+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710118704}) 2023-10-30T12:03:24.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.701+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3103177255}) 2023-10-30T12:03:24.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.701+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.701+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710118704}) 2023-10-30T12:03:24.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.701+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150903083}) 2023-10-30T12:03:24.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.702+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.702+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131396184}) 2023-10-30T12:03:24.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.702+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=424546914}) 2023-10-30T12:03:24.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.702+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.702+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131396184}) 2023-10-30T12:03:24.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.702+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3902669819}) 2023-10-30T12:03:24.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.703+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.703+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=20,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=31} crc {logm=2520773491,mdsmap=3140514009,mgr=2119566901,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2668073844,mon_config_key=715115367,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1715520769,osd_snap=1117521552,osdmap=2519756587}) 2023-10-30T12:03:24.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.703+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=25} crc {logm=949476325,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3188020611,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1715520769,osd_snap=3935772952,osdmap=1408647881}) 2023-10-30T12:03:24.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.703+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.703+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=20,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=31} crc {logm=2520773491,mdsmap=3140514009,mgr=2119566901,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2668073844,mon_config_key=715115367,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1715520769,osd_snap=1117521552,osdmap=2519756587}) 2023-10-30T12:03:24.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.703+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=30} crc {logm=2042153968,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2004330281,mon_config_key=2646296515,monmap=3140514009,osd_metadata=2479346626,osd_pg_creating=1715520769,osd_snap=1941441584,osdmap=1156983844}) 2023-10-30T12:03:24.724 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:03:24.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.707+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.707+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2399024553}) 2023-10-30T12:03:24.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.707+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1595290445}) 2023-10-30T12:03:24.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.707+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:03:24.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.707+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2399024553}) 2023-10-30T12:03:24.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:03:24.707+0000 7f9787a85640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=3357149557}) 2023-10-30T12:03:25.725 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:03:25.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.831+0000 7f183d739640 10 monclient: get_monmap_and_config 2023-10-30T12:03:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.831+0000 7f183d739640 10 monclient: build_initial_monmap 2023-10-30T12:03:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.831+0000 7f183d739640 10 monclient: monmap: 2023-10-30T12:03:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:25.832305+0000 2023-10-30T12:03:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:25.832305+0000 2023-10-30T12:03:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:25.835 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:25.835 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:25.835 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:25.835 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:25.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.831+0000 7f183d739640 1 Processor -- start 2023-10-30T12:03:25.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.831+0000 7f183d739640 1 -- start start 2023-10-30T12:03:25.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.831+0000 7f183d739640 10 monclient: init 2023-10-30T12:03:25.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T12:03:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18380a6070 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18380a57e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f183813bfd0 0x7f183813e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient(hunting): picked mon.noname-c con 0x7f183813bfd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:25.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183809a270 con 0x7f18380a6070 2023-10-30T12:03:25.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1838099f90 con 0x7f18380a57e0 2023-10-30T12:03:25.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:25.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183809a550 con 0x7f183813bfd0 2023-10-30T12:03:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 10 monclient(hunting): authenticate will time out at 1051.015376s 2023-10-30T12:03:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1836ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f18380a6070 0x7f18380a6450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40284/0 (socket says 172.21.15.50:40284) 2023-10-30T12:03:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1836ffd640 1 -- 172.21.15.50:0/1750141834 learned_addr learned my addr 172.21.15.50:0/1750141834 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 1 --2- 172.21.15.50:0/1750141834 >> v2:172.21.15.118:3300/0 conn(0x7f183813bfd0 0x7f183813e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 10 monclient(hunting): get_auth_request con 0x7f183813bfd0 auth_method 0 2023-10-30T12:03:25.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:25.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:25.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:25.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233087886 0 0) 0x7f183809a270 con 0x7f18380a6070 2023-10-30T12:03:25.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:03:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient(hunting): my global_id is 64170 2023-10-30T12:03:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1820003520 con 0x7f18380a6070 2023-10-30T12:03:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f18377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145229982 0 0) 0x7f1820003520 con 0x7f18380a6070 2023-10-30T12:03:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 >> v2:172.21.15.118:3300/0 conn(0x7f183813bfd0 msgr2=0x7f183813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 --2- 172.21.15.50:0/1750141834 >> v2:172.21.15.118:3300/0 conn(0x7f183813bfd0 0x7f183813e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 >> v1:172.21.15.118:6789/0 conn(0x7f18380a57e0 legacy=0x7f183813a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient: _finish_hunting 0 2023-10-30T12:03:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:03:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:03:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183809a830 con 0x7f18380a6070 2023-10-30T12:03:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient: _finish_auth 0 2023-10-30T12:03:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:03:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f1828001a10 con 0x7f18380a6070 2023-10-30T12:03:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 5 monclient: authenticate success, global_id 64170 2023-10-30T12:03:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.832+0000 7f183d739640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:03:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient: dump: 2023-10-30T12:03:25.855 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:25.855 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:25.855 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.855 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.855 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:25.855 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:25.856 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:25.856 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:25.856 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:25.856 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:25.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:03:25.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:25.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 >> v1:172.21.15.50:6789/0 conn(0x7f18380a6070 legacy=0x7f18380a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:25.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:03:25.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820003970 0x7f1820003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): picked mon.b con 0x7f1820003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:25.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18200047c0 0x7f1820004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): picked mon.a con 0x7f18200047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:25.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200052a0 0x7f182000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): picked mon.c con 0x7f18200052a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:25.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:25.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183809a550 con 0x7f18200047c0 2023-10-30T12:03:25.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:25.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182000c170 con 0x7f1820003970 2023-10-30T12:03:25.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:25.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182000c330 con 0x7f18200052a0 2023-10-30T12:03:25.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:25.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1835ffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:03:25.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f183d739640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:25.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18200047c0 0x7f1820004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200052a0 0x7f182000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): get_auth_request con 0x7f18200047c0 auth_method 0 2023-10-30T12:03:25.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:25.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:25.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:25.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 10 monclient(hunting): get_auth_request con 0x7f18200052a0 auth_method 0 2023-10-30T12:03:25.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:25.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:25.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:25.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820003970 0x7f1820003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 10 monclient(hunting): get_auth_request con 0x7f1820003970 auth_method 0 2023-10-30T12:03:25.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:25.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:25.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:25.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:25.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:25.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:25.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:25.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:25.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:25.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:25.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:25.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f1836ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:25.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient(hunting): handle_auth_done global_id 64170 payload 293 2023-10-30T12:03:25.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 1 -- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200052a0 msgr2=0x7f182000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:25.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200052a0 0x7f182000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:25.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 1 -- 172.21.15.50:0/1750141834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820003970 msgr2=0x7f1820003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:25.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820003970 0x7f1820003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:25.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient: _finish_hunting 0 2023-10-30T12:03:25.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 1 monclient: found mon.a 2023-10-30T12:03:25.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:25.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:25.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 1 -- 172.21.15.50:0/1750141834 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820003520 con 0x7f18200047c0 2023-10-30T12:03:25.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient: _finish_auth 0 2023-10-30T12:03:25.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 10 monclient: _check_auth_tickets 2023-10-30T12:03:25.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:25.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.833+0000 7f18377fe640 1 -- 172.21.15.50:0/1750141834 reap_dead start 2023-10-30T12:03:25.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f18367fc640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18200047c0 0x7f1820004c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f182c001720 tx=0x7f18200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8b9f7fbcfc6406 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:25.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c0052d0 con 0x7f18200047c0 2023-10-30T12:03:25.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:25.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:25.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 10 monclient: dump: 2023-10-30T12:03:25.870 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:25.872 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c02e050 con 0x7f18200047c0 2023-10-30T12:03:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:25.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 1 -- 172.21.15.50:0/1750141834 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c003830 con 0x7f18200047c0 2023-10-30T12:03:25.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:25.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 4 set_mon_vals no callback set 2023-10-30T12:03:25.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:25.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f1835ffb640 10 monclient: dump: 2023-10-30T12:03:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:25.876 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 10 monclient: shutdown 2023-10-30T12:03:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 1 -- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18200047c0 msgr2=0x7f1820004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18200047c0 0x7f1820004c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f182c001720 tx=0x7f18200057f0 comp rx=0 tx=0).stop 2023-10-30T12:03:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 1 -- 172.21.15.50:0/1750141834 shutdown_connections 2023-10-30T12:03:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 1 --2- 172.21.15.50:0/1750141834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18200047c0 0x7f1820004c40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 1 -- 172.21.15.50:0/1750141834 >> 172.21.15.50:0/1750141834 conn(0x7f18380a81d0 msgr2=0x7f18380a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 1 -- 172.21.15.50:0/1750141834 shutdown_connections 2023-10-30T12:03:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.834+0000 7f183d739640 1 -- 172.21.15.50:0/1750141834 wait complete. 2023-10-30T12:03:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.835+0000 7f183d739640 10 monclient: build_initial_monmap 2023-10-30T12:03:25.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.835+0000 7f183d739640 10 monclient: monmap: 2023-10-30T12:03:25.878 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:25.878 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:25.878 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:25.878 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:25.879 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:25.879 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:25.879 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:25.879 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:25.879 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:25.879 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:25.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.835+0000 7f183d739640 1 librados: starting msgr at 2023-10-30T12:03:25.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.835+0000 7f183d739640 1 librados: starting objecter 2023-10-30T12:03:25.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.835+0000 7f183d739640 1 Processor -- start 2023-10-30T12:03:25.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 -- start start 2023-10-30T12:03:25.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 librados: setting wanted keys 2023-10-30T12:03:25.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 librados: calling monclient init 2023-10-30T12:03:25.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient: init 2023-10-30T12:03:25.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:25.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:03:25.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200047c0 0x7f18381ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): picked mon.noname-c con 0x7f18200047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:25.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18381ca9e0 0x7f183809c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): picked mon.noname-a con 0x7f18381ca9e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:25.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18381cae20 0x7f183809cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): picked mon.noname-b con 0x7f18381cae20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:25.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183809ffe0 con 0x7f18381ca9e0 2023-10-30T12:03:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183809ef20 con 0x7f18381cae20 2023-10-30T12:03:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18380a1690 con 0x7f18200047c0 2023-10-30T12:03:25.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f183d739640 10 monclient(hunting): authenticate will time out at 1051.019440s 2023-10-30T12:03:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200047c0 0x7f18381ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18381ca9e0 0x7f183809c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200047c0 0x7f18381ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40776/0 (socket says 172.21.15.50:40776) 2023-10-30T12:03:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 1 -- 172.21.15.50:0/537464898 learned_addr learned my addr 172.21.15.50:0/537464898 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): get_auth_request con 0x7f18381ca9e0 auth_method 0 2023-10-30T12:03:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 10 monclient(hunting): get_auth_request con 0x7f18200047c0 auth_method 0 2023-10-30T12:03:25.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:25.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f1836ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18377fe640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18381cae20 0x7f183809cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient(hunting): handle_auth_done global_id 64173 payload 824 2023-10-30T12:03:25.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 1 -- 172.21.15.50:0/537464898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200047c0 msgr2=0x7f18381ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:25.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200047c0 0x7f18381ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:25.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 1 -- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18381cae20 msgr2=0x7f183809cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:25.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18381cae20 0x7f183809cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:25.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient: _finish_hunting 0 2023-10-30T12:03:25.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 1 monclient: found mon.noname-a 2023-10-30T12:03:25.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:25.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:25.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.836+0000 7f18367fc640 1 -- 172.21.15.50:0/537464898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183813b6b0 con 0x7f18381ca9e0 2023-10-30T12:03:25.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f18367fc640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18381ca9e0 0x7f183809c8b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f182000c140 tx=0x7f182c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:25.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 client.?.objecter ms_handle_connect 0x7f18381ca9e0 2023-10-30T12:03:25.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:25.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 1 -- 172.21.15.50:0/537464898 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c007a30 con 0x7f18381ca9e0 2023-10-30T12:03:25.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:25.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:25.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: dump: 2023-10-30T12:03:25.891 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:25.891 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:25.891 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.891 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.891 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:25.892 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:25.892 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:25.892 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:25.892 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:25.892 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:25.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:25.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:25.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:25.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 5 monclient: authenticate success, global_id 64173 2023-10-30T12:03:25.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 10 monclient: _renew_subs 2023-10-30T12:03:25.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 1 -- 172.21.15.50:0/537464898 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c0069e0 con 0x7f18381ca9e0 2023-10-30T12:03:25.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:25.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18380a0880 con 0x7f18381ca9e0 2023-10-30T12:03:25.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:25.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 1 -- 172.21.15.50:0/537464898 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c006d90 con 0x7f18381ca9e0 2023-10-30T12:03:25.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:25.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:25.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f181f7fe640 10 monclient: dump: 2023-10-30T12:03:25.895 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:25.895 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:25.895 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.896 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:25.896 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:25.896 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:25.896 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:25.896 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:25.896 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:25.897 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:25.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f18357fa640 4 set_mon_vals no callback set 2023-10-30T12:03:25.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 10 client.64173.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:25.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 10 monclient: _renew_subs 2023-10-30T12:03:25.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:25.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18380a0ca0 con 0x7f18381ca9e0 2023-10-30T12:03:25.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.837+0000 7f183d739640 1 librados: init done 2023-10-30T12:03:25.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f181f7fe640 1 -- 172.21.15.50:0/537464898 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f182c0035f0 con 0x7f18381ca9e0 2023-10-30T12:03:25.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f183d739640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:25.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f183d739640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:25.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f183d739640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:25.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183804e160 con 0x7f18381ca9e0 2023-10-30T12:03:25.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f181f7fe640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18240392f0 0x7f182403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:25.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f181f7fe640 1 -- 172.21.15.50:0/537464898 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f182c06bcb0 con 0x7f18381ca9e0 2023-10-30T12:03:25.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f181f7fe640 10 client.64173.objecter ms_dispatch 0x7f1838000b90 osd_map(81..81 src has 1..81) v4 2023-10-30T12:03:25.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f181f7fe640 3 client.64173.objecter handle_osd_map got epochs [81,81] > 0 2023-10-30T12:03:25.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f181f7fe640 3 client.64173.objecter handle_osd_map decoding full epoch 81 2023-10-30T12:03:25.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f181f7fe640 20 client.64173.objecter dump_active .. 0 homeless 2023-10-30T12:03:25.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f1836ffd640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18240392f0 0x7f182403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:25.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.838+0000 7f1836ffd640 10 monclient: get_auth_request con 0x7f18240392f0 auth_method 0 2023-10-30T12:03:25.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.839+0000 7f181f7fe640 10 client.64173.objecter ms_handle_connect 0x7f18240392f0 2023-10-30T12:03:25.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.839+0000 7f1836ffd640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18240392f0 0x7f182403b7b0 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7f1828004c40 tx=0x7f1828006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:25.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.843+0000 7f181f7fe640 1 -- 172.21.15.50:0/537464898 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f182c047770 con 0x7f18381ca9e0 2023-10-30T12:03:25.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.843+0000 7f181f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:25.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:25.843+0000 7f181f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:26.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.089+0000 7f183d739640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:26.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.089+0000 7f183d739640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:26.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.089+0000 7f183d739640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.089+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1838074580 con 0x7f18381ca9e0 2023-10-30T12:03:26.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.090+0000 7f181f7fe640 1 -- 172.21.15.50:0/537464898 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f182c044030 con 0x7f18381ca9e0 2023-10-30T12:03:26.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.090+0000 7f181f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:26.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.090+0000 7f181f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:26.093 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:26.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 10 librados: watch_flush enter 2023-10-30T12:03:26.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 10 librados: watch_flush exit 2023-10-30T12:03:26.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 20 client.64173.objecter shutdown clearing up homeless session... 2023-10-30T12:03:26.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 10 client.64173.objecter successfully canceled tick 2023-10-30T12:03:26.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18240392f0 msgr2=0x7f182403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18240392f0 0x7f182403b7b0 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7f1828004c40 tx=0x7f1828006040 comp rx=0 tx=0).stop 2023-10-30T12:03:26.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 10 monclient: shutdown 2023-10-30T12:03:26.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18381ca9e0 msgr2=0x7f183809c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18381ca9e0 0x7f183809c8b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f182000c140 tx=0x7f182c001120 comp rx=0 tx=0).stop 2023-10-30T12:03:26.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 shutdown_connections 2023-10-30T12:03:26.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18240392f0 0x7f182403b7b0 unknown :-1 s=CLOSED pgs=3873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18381cae20 0x7f183809cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18381ca9e0 0x7f183809c8b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.091+0000 7f183d739640 1 --2- 172.21.15.50:0/537464898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18200047c0 0x7f18381ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.092+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 >> 172.21.15.50:0/537464898 conn(0x7f18380a81d0 msgr2=0x7f18381348b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.092+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 shutdown_connections 2023-10-30T12:03:26.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.092+0000 7f183d739640 1 -- 172.21.15.50:0/537464898 wait complete. 2023-10-30T12:03:26.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.092+0000 7f183d739640 1 librados: shutdown 2023-10-30T12:03:26.103 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:03:26.103 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:03:26.104 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:03:26.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:03:26.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.212+0000 7f8f14ce1640 10 monclient: get_monmap_and_config 2023-10-30T12:03:26.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.212+0000 7f8f14ce1640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.212+0000 7f8f14ce1640 10 monclient: monmap: 2023-10-30T12:03:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:26.213337+0000 2023-10-30T12:03:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:26.213337+0000 2023-10-30T12:03:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:26.216 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.212+0000 7f8f14ce1640 1 Processor -- start 2023-10-30T12:03:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.212+0000 7f8f14ce1640 1 -- start start 2023-10-30T12:03:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.212+0000 7f8f14ce1640 10 monclient: init 2023-10-30T12:03:26.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-10-30T12:03:26.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f100a8010 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:26.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f10099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:26.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f10132da0 0x7f8f1013e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient(hunting): picked mon.noname-a con 0x7f8f10132da0 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:26.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f100a7190 con 0x7f8f100a8010 2023-10-30T12:03:26.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f100a83f0 con 0x7f8f10099a40 2023-10-30T12:03:26.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f100a7600 con 0x7f8f10132da0 2023-10-30T12:03:26.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f14ce1640 10 monclient(hunting): authenticate will time out at 1051.396586s 2023-10-30T12:03:26.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0e59c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f100a8010 0x7f8f10132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40300/0 (socket says 172.21.15.50:40300) 2023-10-30T12:03:26.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0e59c640 1 -- 172.21.15.50:0/4125239787 learned_addr learned my addr 172.21.15.50:0/4125239787 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:26.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 1 --2- 172.21.15.50:0/4125239787 >> v2:172.21.15.50:3300/0 conn(0x7f8f10132da0 0x7f8f1013e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request con 0x7f8f10132da0 auth_method 0 2023-10-30T12:03:26.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.213+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_done global_id 64176 payload 274 2023-10-30T12:03:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/4125239787 >> v1:172.21.15.118:6789/0 conn(0x7f8f10099a40 legacy=0x7f8f1013aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/4125239787 >> v1:172.21.15.50:6789/0 conn(0x7f8f100a8010 legacy=0x7f8f10132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 1 monclient: found mon.noname-a 2023-10-30T12:03:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/4125239787 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1009a440 con 0x7f8f10132da0 2023-10-30T12:03:26.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 1 -- 172.21.15.50:0/4125239787 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153956893 0 0) 0x7f8f100a7190 con 0x7f8f100a8010 2023-10-30T12:03:26.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:03:26.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0ed9d640 1 --2- 172.21.15.50:0/4125239787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10132da0 0x7f8f1013e140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009d90 tx=0x7f8f0802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923c89203fada671 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 1 -- 172.21.15.50:0/4125239787 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0802cbd0 con 0x7f8f10132da0 2023-10-30T12:03:26.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:26.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: dump: 2023-10-30T12:03:26.237 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.237 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.238 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.238 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.238 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.238 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.238 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.239 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.239 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.239 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 1 -- 172.21.15.50:0/4125239787 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0802cd30 con 0x7f8f10132da0 2023-10-30T12:03:26.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f14ce1640 5 monclient: authenticate success, global_id 64176 2023-10-30T12:03:26.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f14ce1640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:26.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f14ce1640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:26.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 1 -- 172.21.15.50:0/4125239787 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0802cea0 con 0x7f8f10132da0 2023-10-30T12:03:26.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f14ce1640 4 set_mon_vals no callback set 2023-10-30T12:03:26.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:26.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.214+0000 7f8f0d59a640 10 monclient: dump: 2023-10-30T12:03:26.242 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.242 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.242 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.243 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.243 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.243 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.243 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.243 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.244 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.244 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 10 monclient: shutdown 2023-10-30T12:03:26.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 1 -- 172.21.15.50:0/4125239787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10132da0 msgr2=0x7f8f1013e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 1 --2- 172.21.15.50:0/4125239787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10132da0 0x7f8f1013e140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009d90 tx=0x7f8f0802be20 comp rx=0 tx=0).stop 2023-10-30T12:03:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 1 -- 172.21.15.50:0/4125239787 shutdown_connections 2023-10-30T12:03:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 1 --2- 172.21.15.50:0/4125239787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10132da0 0x7f8f1013e140 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 1 -- 172.21.15.50:0/4125239787 >> 172.21.15.50:0/4125239787 conn(0x7f8f100a4a10 msgr2=0x7f8f100a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 1 -- 172.21.15.50:0/4125239787 shutdown_connections 2023-10-30T12:03:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.215+0000 7f8f14ce1640 1 -- 172.21.15.50:0/4125239787 wait complete. 2023-10-30T12:03:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 10 monclient: monmap: 2023-10-30T12:03:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:26.248 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 1 librados: starting msgr at 2023-10-30T12:03:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 1 librados: starting objecter 2023-10-30T12:03:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 1 Processor -- start 2023-10-30T12:03:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 1 -- start start 2023-10-30T12:03:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 1 librados: setting wanted keys 2023-10-30T12:03:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 1 librados: calling monclient init 2023-10-30T12:03:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.216+0000 7f8f14ce1640 10 monclient: init 2023-10-30T12:03:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:03:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f100a8010 0x7f8f100a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): picked mon.noname-c con 0x7f8f100a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f10132da0 0x7f8f101367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): picked mon.noname-b con 0x7f8f10132da0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10136d00 0x7f8f10137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): picked mon.noname-a con 0x7f8f10136d00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1013a1c0 con 0x7f8f10136d00 2023-10-30T12:03:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100a18d0 con 0x7f8f10132da0 2023-10-30T12:03:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1013dac0 con 0x7f8f100a8010 2023-10-30T12:03:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient(hunting): authenticate will time out at 1051.400133s 2023-10-30T12:03:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10136d00 0x7f8f10137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f100a8010 0x7f8f100a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10136d00 0x7f8f10137190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60854/0 (socket says 172.21.15.50:60854) 2023-10-30T12:03:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/260037255 learned_addr learned my addr 172.21.15.50:0/260037255 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request con 0x7f8f10136d00 auth_method 0 2023-10-30T12:03:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request con 0x7f8f100a8010 auth_method 0 2023-10-30T12:03:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0dd9b640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f10132da0 0x7f8f101367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0dd9b640 10 monclient(hunting): get_auth_request con 0x7f8f10132da0 auth_method 0 2023-10-30T12:03:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_done global_id 64182 payload 824 2023-10-30T12:03:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f100a8010 msgr2=0x7f8f100a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f100a8010 0x7f8f100a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f10132da0 msgr2=0x7f8f101367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f10132da0 0x7f8f101367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 monclient: found mon.noname-a 2023-10-30T12:03:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/260037255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08009a70 con 0x7f8f10136d00 2023-10-30T12:03:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0ed9d640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10136d00 0x7f8f10137190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f08037040 tx=0x7f8f08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 client.?.objecter ms_handle_connect 0x7f8f10136d00 2023-10-30T12:03:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0802cea0 con 0x7f8f10136d00 2023-10-30T12:03:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: dump: 2023-10-30T12:03:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.264 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0803e610 con 0x7f8f10136d00 2023-10-30T12:03:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 5 monclient: authenticate success, global_id 64182 2023-10-30T12:03:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0803e9c0 con 0x7f8f10136d00 2023-10-30T12:03:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient: _renew_subs 2023-10-30T12:03:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f10139da0 con 0x7f8f10136d00 2023-10-30T12:03:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f0cd99640 4 set_mon_vals no callback set 2023-10-30T12:03:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8efeffd640 10 monclient: dump: 2023-10-30T12:03:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.269 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 client.64182.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient: _renew_subs 2023-10-30T12:03:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f101397e0 con 0x7f8f10136d00 2023-10-30T12:03:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.217+0000 7f8f14ce1640 1 librados: init done 2023-10-30T12:03:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8f0803b050 con 0x7f8f10136d00 2023-10-30T12:03:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f14ce1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f14ce1640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10099a40 0x7f8f1009cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f14ce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10073fd0 con 0x7f8f10099a40 2023-10-30T12:03:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f0e59c640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10099a40 0x7f8f1009cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f0e59c640 10 monclient: get_auth_request con 0x7f8f10099a40 auth_method 0 2023-10-30T12:03:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8efeffd640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef00392f0 0x7f8ef003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.218+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f8f0806d6c0 con 0x7f8f10136d00 2023-10-30T12:03:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 client.64182.objecter ms_dispatch 0x7f8f10000b90 osd_map(81..81 src has 1..81) v4 2023-10-30T12:03:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 3 client.64182.objecter handle_osd_map got epochs [81,81] > 0 2023-10-30T12:03:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 3 client.64182.objecter handle_osd_map decoding full epoch 81 2023-10-30T12:03:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0dd9b640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef00392f0 0x7f8ef003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 20 client.64182.objecter dump_active .. 0 homeless 2023-10-30T12:03:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0dd9b640 10 monclient: get_auth_request con 0x7f8ef00392f0 auth_method 0 2023-10-30T12:03:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_done global_id 64185 payload 824 2023-10-30T12:03:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0e59c640 1 -- 172.21.15.50:0/260037255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f1009aa00 con 0x7f8f10099a40 2023-10-30T12:03:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 client.64182.objecter ms_handle_connect 0x7f8f10099a40 2023-10-30T12:03:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0e59c640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10099a40 0x7f8f1009cec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000c00 tx=0x7f8f00001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 client.64182.objecter resend_mon_ops 2023-10-30T12:03:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0000f630 con 0x7f8f10099a40 2023-10-30T12:03:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 monclient: dump: 2023-10-30T12:03:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.277 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 client.64182.objecter ms_handle_connect 0x7f8ef00392f0 2023-10-30T12:03:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8f0dd9b640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef00392f0 0x7f8ef003b7b0 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7f8f100a0ed0 tx=0x7f8f04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8f0000f790 con 0x7f8f10099a40 2023-10-30T12:03:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10099a40 msgr2=0x7f8f1009cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10099a40 0x7f8f1009cec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000c00 tx=0x7f8f00001970 comp rx=0 tx=0).stop 2023-10-30T12:03:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10099a40 msgr2=0x7f8f1009cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.219+0000 7f8efeffd640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10099a40 0x7f8f1009cec0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000c00 tx=0x7f8f00001970 comp rx=0 tx=0).stop 2023-10-30T12:03:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f14ce1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f14ce1640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1009d400 0x7f8f1009d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f14ce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1009a720 con 0x7f8f1009d400 2023-10-30T12:03:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1009d400 0x7f8f1009d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient: get_auth_request con 0x7f8f1009d400 auth_method 0 2023-10-30T12:03:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_done global_id 64188 payload 824 2023-10-30T12:03:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 1 -- 172.21.15.50:0/260037255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f1013e680 con 0x7f8f1009d400 2023-10-30T12:03:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8efeffd640 10 client.64182.objecter ms_handle_connect 0x7f8f1009d400 2023-10-30T12:03:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8efeffd640 10 client.64182.objecter resend_mon_ops 2023-10-30T12:03:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8f0e59c640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1009d400 0x7f8f1009d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000c00 tx=0x7f8f00000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00024070 con 0x7f8f1009d400 2023-10-30T12:03:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.226+0000 7f8efeffd640 10 monclient: dump: 2023-10-30T12:03:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.285 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f8f0001faf0 con 0x7f8f1009d400 2023-10-30T12:03:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8efeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8efeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1009d400 msgr2=0x7f8f1009d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8efeffd640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1009d400 0x7f8f1009d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000c00 tx=0x7f8f00000ed0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8efeffd640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1009d400 msgr2=0x7f8f1009d810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8efeffd640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1009d400 0x7f8f1009d810 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f00000c00 tx=0x7f8f00000ed0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.287 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:26.287 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:26.287 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:26.287 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:03:26.288 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 30, 2023-10-30T12:03:26.288 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:26.288 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:03:26.288 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:26.288 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:26.289 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.289 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:03:26.289 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:26.289 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:26.289 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:26.290 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.290 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.290 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.290 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.290 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.291 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.291 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.291 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.291 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.292 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.292 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.292 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:26.292 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:26.292 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.293 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.293 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.293 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.293 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.293 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.293 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.294 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.294 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.294 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.294 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.295 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:26.295 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:26.295 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:26.295 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:26.295 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:26.296 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:26.296 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:26.296 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:26.296 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:26.296 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:26.297 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:26.297 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:26.297 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:26.297 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:26.297 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:26.297 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:26.298 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:26.298 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.298 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.298 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.298 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.299 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.299 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.299 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.299 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.299 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.300 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.300 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.300 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:26.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.300 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:26.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.301 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:26.301 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:26.301 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:26.302 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:26.303 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.303 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.303 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:26.304 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:26.304 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.304 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.305 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.305 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:26.305 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:26.305 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.306 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:26.306 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:26.307 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:26.308 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:26.309 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.309 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.309 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.309 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:26.310 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:26.310 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.310 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.311 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:26.311 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.311 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.312 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:26.312 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.312 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.313 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:26.313 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:26.313 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.313 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.313 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.314 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.314 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:26.314 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:26.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.315 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.315 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.315 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:26.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.316 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.316 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:26.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.316 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.316 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.316 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:26.317 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.317 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.317 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:26.317 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:26.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 10 librados: watch_flush enter 2023-10-30T12:03:26.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 10 librados: watch_flush exit 2023-10-30T12:03:26.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 20 client.64182.objecter shutdown clearing up homeless session... 2023-10-30T12:03:26.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 10 client.64182.objecter successfully canceled tick 2023-10-30T12:03:26.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef00392f0 msgr2=0x7f8ef003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef00392f0 0x7f8ef003b7b0 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7f8f100a0ed0 tx=0x7f8f04009040 comp rx=0 tx=0).stop 2023-10-30T12:03:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 10 monclient: shutdown 2023-10-30T12:03:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10136d00 msgr2=0x7f8f10137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 --2- 172.21.15.50:0/260037255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10136d00 0x7f8f10137190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f08037040 tx=0x7f8f08004a40 comp rx=0 tx=0).stop 2023-10-30T12:03:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/260037255 reap_dead start 2023-10-30T12:03:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f0ed9d640 1 -- 172.21.15.50:0/260037255 reap_dead start 2023-10-30T12:03:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 shutdown_connections 2023-10-30T12:03:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 >> 172.21.15.50:0/260037255 conn(0x7f8f100a4a10 msgr2=0x7f8f100a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 shutdown_connections 2023-10-30T12:03:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 -- 172.21.15.50:0/260037255 wait complete. 2023-10-30T12:03:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.227+0000 7f8f14ce1640 1 librados: shutdown 2023-10-30T12:03:26.323 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:03:26.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:03:26.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:03:26.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.450+0000 7f47fd697640 10 monclient: get_monmap_and_config 2023-10-30T12:03:26.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.450+0000 7f47fd697640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.450+0000 7f47fd697640 10 monclient: monmap: 2023-10-30T12:03:26.452 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.452 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.452 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:26.451197+0000 2023-10-30T12:03:26.452 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:26.451197+0000 2023-10-30T12:03:26.453 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.453 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.453 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:26.453 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:26.453 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:26.454 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:26.454 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:26.454 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.450+0000 7f47fd697640 1 Processor -- start 2023-10-30T12:03:26.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.450+0000 7f47fd697640 1 -- start start 2023-10-30T12:03:26.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.450+0000 7f47fd697640 10 monclient: init 2023-10-30T12:03:26.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:03:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47f80a8010 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47f8099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f47f8132da0 0x7f47f813e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient(hunting): picked mon.noname-c con 0x7f47f8132da0 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f80a7190 con 0x7f47f80a8010 2023-10-30T12:03:26.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f80a83f0 con 0x7f47f8099a40 2023-10-30T12:03:26.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80a7600 con 0x7f47f8132da0 2023-10-30T12:03:26.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47fd697640 10 monclient(hunting): authenticate will time out at 1051.634342s 2023-10-30T12:03:26.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f6ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f47f80a8010 0x7f47f8132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40304/0 (socket says 172.21.15.50:40304) 2023-10-30T12:03:26.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f6ffd640 1 -- 172.21.15.50:0/3868554948 learned_addr learned my addr 172.21.15.50:0/3868554948 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:26.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 1 --2- 172.21.15.50:0/3868554948 >> v2:172.21.15.118:3300/0 conn(0x7f47f8132da0 0x7f47f813e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): get_auth_request con 0x7f47f8132da0 auth_method 0 2023-10-30T12:03:26.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f5ffb640 1 -- 172.21.15.50:0/3868554948 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285824394 0 0) 0x7f47f80a7190 con 0x7f47f80a8010 2023-10-30T12:03:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f5ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:03:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f5ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f5ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f5ffb640 10 monclient(hunting): my global_id is 64191 2023-10-30T12:03:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f5ffb640 1 -- 172.21.15.50:0/3868554948 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e4003520 con 0x7f47f80a8010 2023-10-30T12:03:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient(hunting): handle_auth_done global_id 27973 payload 274 2023-10-30T12:03:26.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 1 -- 172.21.15.50:0/3868554948 >> v1:172.21.15.118:6789/0 conn(0x7f47f8099a40 legacy=0x7f47f813aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 1 -- 172.21.15.50:0/3868554948 >> v1:172.21.15.50:6789/0 conn(0x7f47f80a8010 legacy=0x7f47f8132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 1 monclient: found mon.noname-c 2023-10-30T12:03:26.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:03:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.451+0000 7f47f77fe640 1 -- 172.21.15.50:0/3868554948 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f809a440 con 0x7f47f8132da0 2023-10-30T12:03:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f77fe640 1 --2- 172.21.15.50:0/3868554948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 0x7f47f813e140 secure :-1 s=READY pgs=7029 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009d90 tx=0x7f47e802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e1128ea78e633c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 1 -- 172.21.15.50:0/3868554948 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e802cbd0 con 0x7f47f8132da0 2023-10-30T12:03:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: dump: 2023-10-30T12:03:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.472 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 1 -- 172.21.15.50:0/3868554948 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e802cd30 con 0x7f47f8132da0 2023-10-30T12:03:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47fd697640 5 monclient: authenticate success, global_id 27973 2023-10-30T12:03:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47fd697640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47fd697640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 1 -- 172.21.15.50:0/3868554948 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e802cea0 con 0x7f47f8132da0 2023-10-30T12:03:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47fd697640 4 set_mon_vals no callback set 2023-10-30T12:03:26.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:26.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.452+0000 7f47f5ffb640 10 monclient: dump: 2023-10-30T12:03:26.474 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.474 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.474 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.475 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.475 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.475 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.475 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.475 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.476 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.476 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 10 monclient: shutdown 2023-10-30T12:03:26.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 1 -- 172.21.15.50:0/3868554948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 msgr2=0x7f47f813e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 1 --2- 172.21.15.50:0/3868554948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 0x7f47f813e140 secure :-1 s=READY pgs=7029 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009d90 tx=0x7f47e802be20 comp rx=0 tx=0).stop 2023-10-30T12:03:26.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 1 -- 172.21.15.50:0/3868554948 shutdown_connections 2023-10-30T12:03:26.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 1 --2- 172.21.15.50:0/3868554948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 0x7f47f813e140 unknown :-1 s=CLOSED pgs=7029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 1 -- 172.21.15.50:0/3868554948 >> 172.21.15.50:0/3868554948 conn(0x7f47f80a4a10 msgr2=0x7f47f80a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 1 -- 172.21.15.50:0/3868554948 shutdown_connections 2023-10-30T12:03:26.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 1 -- 172.21.15.50:0/3868554948 wait complete. 2023-10-30T12:03:26.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.453+0000 7f47fd697640 10 monclient: monmap: 2023-10-30T12:03:26.478 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.478 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:26.478 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:26.478 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:26.479 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.479 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.479 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:26.479 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:26.479 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:26.480 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 1 librados: starting msgr at 2023-10-30T12:03:26.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 1 librados: starting objecter 2023-10-30T12:03:26.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 1 Processor -- start 2023-10-30T12:03:26.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 1 -- start start 2023-10-30T12:03:26.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 1 librados: setting wanted keys 2023-10-30T12:03:26.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 1 librados: calling monclient init 2023-10-30T12:03:26.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 10 monclient: init 2023-10-30T12:03:26.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:03:26.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f80a8010 0x7f47f81367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.454+0000 7f47fd697640 10 monclient(hunting): picked mon.noname-c con 0x7f47f80a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:26.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 0x7f47f8136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient(hunting): picked mon.noname-b con 0x7f47f8132da0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:26.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47f8137240 0x7f47f81d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient(hunting): picked mon.noname-a con 0x7f47f8137240 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:26.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8139c50 con 0x7f47f8137240 2023-10-30T12:03:26.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80a18d0 con 0x7f47f8132da0 2023-10-30T12:03:26.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f813d9c0 con 0x7f47f80a8010 2023-10-30T12:03:26.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient(hunting): authenticate will time out at 1051.638065s 2023-10-30T12:03:26.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47f8137240 0x7f47f81d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f80a8010 0x7f47f81367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47f8137240 0x7f47f81d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60870/0 (socket says 172.21.15.50:60870) 2023-10-30T12:03:26.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 10 monclient(hunting): get_auth_request con 0x7f47f80a8010 auth_method 0 2023-10-30T12:03:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 -- 172.21.15.50:0/3534573509 learned_addr learned my addr 172.21.15.50:0/3534573509 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): get_auth_request con 0x7f47f8137240 auth_method 0 2023-10-30T12:03:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f67fc640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 0x7f47f8136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f67fc640 10 monclient(hunting): get_auth_request con 0x7f47f8132da0 auth_method 0 2023-10-30T12:03:26.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient(hunting): handle_auth_done global_id 64194 payload 824 2023-10-30T12:03:26.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f80a8010 msgr2=0x7f47f81367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f80a8010 0x7f47f81367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 msgr2=0x7f47f8136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47f8132da0 0x7f47f8136d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 monclient: found mon.noname-a 2023-10-30T12:03:26.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 -- 172.21.15.50:0/3534573509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009a70 con 0x7f47f8137240 2023-10-30T12:03:26.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f6ffd640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f80a8010 0x7f47f81367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:26.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f77fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47f8137240 0x7f47f81d06d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009eb0 tx=0x7f47e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 client.?.objecter ms_handle_connect 0x7f47f8137240 2023-10-30T12:03:26.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:26.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e802cea0 con 0x7f47f8137240 2023-10-30T12:03:26.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:03:26.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: dump: 2023-10-30T12:03:26.492 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.492 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.492 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.492 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.493 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.493 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.493 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.493 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.493 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.494 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 5 monclient: authenticate success, global_id 64194 2023-10-30T12:03:26.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e803e630 con 0x7f47f8137240 2023-10-30T12:03:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient: _renew_subs 2023-10-30T12:03:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f8137860 con 0x7f47f8137240 2023-10-30T12:03:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e803e9e0 con 0x7f47f8137240 2023-10-30T12:03:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47f57fa640 4 set_mon_vals no callback set 2023-10-30T12:03:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47ef7fe640 10 monclient: dump: 2023-10-30T12:03:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.497 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.497 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.497 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.497 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.498 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 client.64194.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient: _renew_subs 2023-10-30T12:03:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f8139600 con 0x7f47f8137240 2023-10-30T12:03:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.455+0000 7f47fd697640 1 librados: init done 2023-10-30T12:03:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47fd697640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47fd697640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f8099a40 0x7f47f809cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47fd697640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8073fd0 con 0x7f47f8099a40 2023-10-30T12:03:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f47e803b050 con 0x7f47f8137240 2023-10-30T12:03:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f8099a40 0x7f47f809cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 10 monclient: get_auth_request con 0x7f47f8099a40 auth_method 0 2023-10-30T12:03:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47ef7fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47d80392f0 0x7f47d803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7720+0+0 (secure 0 0 0) 0x7f47e806d490 con 0x7f47f8137240 2023-10-30T12:03:26.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47ef7fe640 10 client.64194.objecter ms_dispatch 0x7f47f8000b90 osd_map(82..82 src has 1..82) v4 2023-10-30T12:03:26.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47ef7fe640 3 client.64194.objecter handle_osd_map got epochs [82,82] > 0 2023-10-30T12:03:26.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.456+0000 7f47ef7fe640 3 client.64194.objecter handle_osd_map decoding full epoch 82 2023-10-30T12:03:26.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_done global_id 44354 payload 824 2023-10-30T12:03:26.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47f6ffd640 1 -- 172.21.15.50:0/3534573509 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47f809aa00 con 0x7f47f8099a40 2023-10-30T12:03:26.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 20 client.64194.objecter dump_active .. 0 homeless 2023-10-30T12:03:26.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47f67fc640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47d80392f0 0x7f47d803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47f67fc640 10 monclient: get_auth_request con 0x7f47d80392f0 auth_method 0 2023-10-30T12:03:26.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 client.64194.objecter ms_handle_connect 0x7f47f8099a40 2023-10-30T12:03:26.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 client.64194.objecter resend_mon_ops 2023-10-30T12:03:26.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47f6ffd640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f8099a40 0x7f47f809cec0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f47f0000f60 tx=0x7f47f0007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f000faa0 con 0x7f47f8099a40 2023-10-30T12:03:26.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:26.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 monclient: dump: 2023-10-30T12:03:26.505 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.505 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.505 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.505 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.505 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.506 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.506 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.506 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.506 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.506 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 client.64194.objecter ms_handle_connect 0x7f47d80392f0 2023-10-30T12:03:26.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47f67fc640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47d80392f0 0x7f47d803b7b0 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f47f80a0cc0 tx=0x7f47e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f47f000fc00 con 0x7f47f8099a40 2023-10-30T12:03:26.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:26.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f8099a40 msgr2=0x7f47f809cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f8099a40 0x7f47f809cec0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f47f0000f60 tx=0x7f47f0007700 comp rx=0 tx=0).stop 2023-10-30T12:03:26.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f8099a40 msgr2=0x7f47f809cec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:26.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.457+0000 7f47ef7fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f8099a40 0x7f47f809cec0 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.463+0000 7f47fd697640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:26.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.463+0000 7f47fd697640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f809d400 0x7f47f809d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.463+0000 7f47fd697640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.463+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f809a720 con 0x7f47f809d400 2023-10-30T12:03:26.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f809d400 0x7f47f809d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient: get_auth_request con 0x7f47f809d400 auth_method 0 2023-10-30T12:03:26.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 10 monclient(hunting): handle_auth_done global_id 44357 payload 824 2023-10-30T12:03:26.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 1 -- 172.21.15.50:0/3534573509 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f47f813e680 con 0x7f47f809d400 2023-10-30T12:03:26.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 10 client.64194.objecter ms_handle_connect 0x7f47f809d400 2023-10-30T12:03:26.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 10 client.64194.objecter resend_mon_ops 2023-10-30T12:03:26.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47f6ffd640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f809d400 0x7f47f809d810 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f47f0018000 tx=0x7f47f0001150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0023070 con 0x7f47f809d400 2023-10-30T12:03:26.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:26.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 10 monclient: dump: 2023-10-30T12:03:26.514 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.514 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.514 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.514 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.514 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.515 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.515 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.515 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.515 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.515 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f47f0006670 con 0x7f47f809d400 2023-10-30T12:03:26.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:26.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:26.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f809d400 msgr2=0x7f47f809d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f809d400 0x7f47f809d810 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f47f0018000 tx=0x7f47f0001150 comp rx=0 tx=0).stop 2023-10-30T12:03:26.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f809d400 msgr2=0x7f47f809d810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:26.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.464+0000 7f47ef7fe640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47f809d400 0x7f47f809d810 secure :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f47f0018000 tx=0x7f47f0001150 comp rx=0 tx=0).stop 2023-10-30T12:03:26.517 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:26.517 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:26.517 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:26.517 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:03:26.518 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 30, 2023-10-30T12:03:26.518 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:26.518 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:03:26.518 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:26.518 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:26.518 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.519 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:03:26.519 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:26.519 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:26.519 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:26.519 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.520 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.521 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.521 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.521 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.521 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.521 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:26.522 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:26.522 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.522 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.522 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.522 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.522 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.523 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.523 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.523 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.523 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.523 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.523 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:26.525 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:26.525 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:26.525 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:26.525 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:26.525 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:26.525 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:26.526 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:26.526 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:26.526 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:26.526 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:26.526 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:26.527 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:26.527 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:26.527 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.527 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.527 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.527 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.528 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.528 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.528 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.528 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.528 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.528 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.529 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.529 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:26.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.529 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:26.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.529 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:26.530 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:26.530 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.530 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.530 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.530 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:26.531 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.531 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:26.532 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.532 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:26.532 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:26.533 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.533 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.533 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.533 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:26.534 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:26.534 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.534 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.534 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.535 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:26.535 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.535 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:26.536 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.536 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.536 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:26.536 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:26.537 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.537 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.537 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.537 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:26.538 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:26.538 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.538 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.538 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.539 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:26.539 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.539 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:26.540 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.540 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:26.540 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:26.540 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.541 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.541 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.541 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.541 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:26.542 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:26.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.542 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.542 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.542 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:26.543 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.543 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.543 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:03:26.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.543 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:26.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.545 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.545 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.545 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:26.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.545 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.546 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:26.546 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 10 librados: watch_flush enter 2023-10-30T12:03:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 10 librados: watch_flush exit 2023-10-30T12:03:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 20 client.64194.objecter shutdown clearing up homeless session... 2023-10-30T12:03:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 10 client.64194.objecter successfully canceled tick 2023-10-30T12:03:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47d80392f0 msgr2=0x7f47d803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47d80392f0 0x7f47d803b7b0 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f47f80a0cc0 tx=0x7f47e0009040 comp rx=0 tx=0).stop 2023-10-30T12:03:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 10 monclient: shutdown 2023-10-30T12:03:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47f8137240 msgr2=0x7f47f81d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 --2- 172.21.15.50:0/3534573509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47f8137240 0x7f47f81d06d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009eb0 tx=0x7f47e8004a40 comp rx=0 tx=0).stop 2023-10-30T12:03:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47f77fe640 1 -- 172.21.15.50:0/3534573509 reap_dead start 2023-10-30T12:03:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47f77fe640 1 -- 172.21.15.50:0/3534573509 reap_dead start 2023-10-30T12:03:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 shutdown_connections 2023-10-30T12:03:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 >> 172.21.15.50:0/3534573509 conn(0x7f47f80a4a10 msgr2=0x7f47f80a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 shutdown_connections 2023-10-30T12:03:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 -- 172.21.15.50:0/3534573509 wait complete. 2023-10-30T12:03:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.465+0000 7f47fd697640 1 librados: shutdown 2023-10-30T12:03:26.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:03:26.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:03:26.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:03:26.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.658+0000 7f48431f5640 10 monclient: get_monmap_and_config 2023-10-30T12:03:26.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.658+0000 7f48431f5640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.658+0000 7f48431f5640 10 monclient: monmap: 2023-10-30T12:03:26.661 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.661 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.661 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:26.659772+0000 2023-10-30T12:03:26.661 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:26.659772+0000 2023-10-30T12:03:26.662 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.662 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.662 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:26.662 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:26.662 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:26.663 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:26.663 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:26.663 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 1 Processor -- start 2023-10-30T12:03:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 1 -- start start 2023-10-30T12:03:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient: init 2023-10-30T12:03:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T12:03:26.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f483c06afc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:26.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f483c06b640 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:26.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f483c13c080 0x7f483c13e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient(hunting): picked mon.noname-c con 0x7f483c13c080 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:26.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c06b3a0 con 0x7f483c06afc0 2023-10-30T12:03:26.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c09e540 con 0x7f483c06b640 2023-10-30T12:03:26.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c0a2ab0 con 0x7f483c13c080 2023-10-30T12:03:26.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.659+0000 7f48431f5640 10 monclient(hunting): authenticate will time out at 1051.842930s 2023-10-30T12:03:26.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f483c06afc0 0x7f483c0a1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40306/0 (socket says 172.21.15.50:40306) 2023-10-30T12:03:26.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 1 -- 172.21.15.50:0/2525579874 learned_addr learned my addr 172.21.15.50:0/2525579874 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:03:26.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 1 --2- 172.21.15.50:0/2525579874 >> v2:172.21.15.118:3300/0 conn(0x7f483c13c080 0x7f483c13e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): get_auth_request con 0x7f483c13c080 auth_method 0 2023-10-30T12:03:26.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111027678 0 0) 0x7f483c06b3a0 con 0x7f483c06afc0 2023-10-30T12:03:26.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:03:26.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): my global_id is 64197 2023-10-30T12:03:26.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482c003520 con 0x7f483c06afc0 2023-10-30T12:03:26.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3475989 0 0) 0x7f482c003520 con 0x7f483c06afc0 2023-10-30T12:03:26.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 >> v2:172.21.15.118:3300/0 conn(0x7f483c13c080 msgr2=0x7f483c13e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:26.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 --2- 172.21.15.50:0/2525579874 >> v2:172.21.15.118:3300/0 conn(0x7f483c13c080 0x7f483c13e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 >> v1:172.21.15.118:6789/0 conn(0x7f483c06b640 legacy=0x7f483c0a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:03:26.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:03:26.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c09fa10 con 0x7f483c06afc0 2023-10-30T12:03:26.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f4830001690 con 0x7f483c06afc0 2023-10-30T12:03:26.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f48431f5640 5 monclient: authenticate success, global_id 64197 2023-10-30T12:03:26.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 1 --2- 172.21.15.50:0/2525579874 >> v2:172.21.15.118:3300/0 conn(0x7f483c13c080 0x7f483c13e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:26.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f48431f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:26.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:03:26.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: dump: 2023-10-30T12:03:26.682 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.683 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.683 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.683 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.684 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:03:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 >> v1:172.21.15.50:6789/0 conn(0x7f483c06afc0 legacy=0x7f483c0a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:03:26.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f482c0039a0 0x7f482c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): picked mon.c con 0x7f482c0039a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:26.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): picked mon.b con 0x7f482c0047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:26.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0052a0 0x7f482c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): picked mon.a con 0x7f482c0052a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:26.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0a2ab0 con 0x7f482c0052a0 2023-10-30T12:03:26.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c00c1a0 con 0x7f482c0047c0 2023-10-30T12:03:26.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c00c360 con 0x7f482c0039a0 2023-10-30T12:03:26.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483b7fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:03:26.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f48431f5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:26.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f482c0039a0 0x7f482c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0052a0 0x7f482c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): get_auth_request con 0x7f482c0052a0 auth_method 0 2023-10-30T12:03:26.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:26.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 10 monclient(hunting): get_auth_request con 0x7f482c0039a0 auth_method 0 2023-10-30T12:03:26.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:26.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483bfff640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483bfff640 10 monclient(hunting): get_auth_request con 0x7f482c0047c0 auth_method 0 2023-10-30T12:03:26.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:26.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:26.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:26.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.660+0000 7f483bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f483bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f483bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:26.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 10 monclient(hunting): handle_auth_done global_id 64197 payload 293 2023-10-30T12:03:26.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 -- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f482c0039a0 msgr2=0x7f482c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:26.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f482c0039a0 0x7f482c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 -- 172.21.15.50:0/2525579874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0047c0 msgr2=0x7f482c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:26.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 monclient: found mon.a 2023-10-30T12:03:26.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:03:26.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 -- 172.21.15.50:0/2525579874 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c003520 con 0x7f482c0052a0 2023-10-30T12:03:26.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4840f91640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f482c0039a0 0x7f482c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:26.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f483bfff640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0047c0 0x7f482c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:26.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 -- 172.21.15.50:0/2525579874 reap_dead start 2023-10-30T12:03:26.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.661+0000 7f4841792640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0052a0 0x7f482c00b980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f483c066c80 tx=0x7f482c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34734bab290d1987 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483402cd90 con 0x7f482c0052a0 2023-10-30T12:03:26.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:26.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 10 monclient: dump: 2023-10-30T12:03:26.696 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.696 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.696 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.696 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.696 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.697 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.697 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.697 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.697 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.697 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483400f040 con 0x7f482c0052a0 2023-10-30T12:03:26.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:26.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:26.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 1 -- 172.21.15.50:0/2525579874 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834003b90 con 0x7f482c0052a0 2023-10-30T12:03:26.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 4 set_mon_vals no callback set 2023-10-30T12:03:26.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:03:26.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f483b7fe640 10 monclient: dump: 2023-10-30T12:03:26.699 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.699 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.700 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.700 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.700 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.700 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.700 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.701 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.701 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.701 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 10 monclient: shutdown 2023-10-30T12:03:26.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 1 -- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0052a0 msgr2=0x7f482c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0052a0 0x7f482c00b980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f483c066c80 tx=0x7f482c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 1 -- 172.21.15.50:0/2525579874 shutdown_connections 2023-10-30T12:03:26.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 1 --2- 172.21.15.50:0/2525579874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0052a0 0x7f482c00b980 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 1 -- 172.21.15.50:0/2525579874 >> 172.21.15.50:0/2525579874 conn(0x7f483c0a6200 msgr2=0x7f483c0a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 1 -- 172.21.15.50:0/2525579874 shutdown_connections 2023-10-30T12:03:26.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.662+0000 7f48431f5640 1 -- 172.21.15.50:0/2525579874 wait complete. 2023-10-30T12:03:26.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 10 monclient: monmap: 2023-10-30T12:03:26.703 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.703 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:26.704 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:26.704 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:26.704 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.704 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.705 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:26.705 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:26.705 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:26.705 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 1 librados: starting msgr at 2023-10-30T12:03:26.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 1 librados: starting objecter 2023-10-30T12:03:26.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 1 Processor -- start 2023-10-30T12:03:26.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 1 -- start start 2023-10-30T12:03:26.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 1 librados: setting wanted keys 2023-10-30T12:03:26.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 1 librados: calling monclient init 2023-10-30T12:03:26.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.663+0000 7f48431f5640 10 monclient: init 2023-10-30T12:03:26.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:03:26.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0052a0 0x7f483c11bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): picked mon.noname-b con 0x7f482c0052a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:26.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f483c11c3e0 0x7f483c11c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): picked mon.noname-a con 0x7f483c11c3e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:26.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f483c1ca200 0x7f483c1ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): picked mon.noname-c con 0x7f483c1ca200 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:26.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c11cd60 con 0x7f483c11c3e0 2023-10-30T12:03:26.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c138520 con 0x7f482c0052a0 2023-10-30T12:03:26.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c11f070 con 0x7f483c1ca200 2023-10-30T12:03:26.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48431f5640 10 monclient(hunting): authenticate will time out at 1051.847216s 2023-10-30T12:03:26.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f483c1ca200 0x7f483c1ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f483c11c3e0 0x7f483c11c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f483c1ca200 0x7f483c1ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40832/0 (socket says 172.21.15.50:40832) 2023-10-30T12:03:26.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 -- 172.21.15.50:0/4057518943 learned_addr learned my addr 172.21.15.50:0/4057518943 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:26.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): get_auth_request con 0x7f483c1ca200 auth_method 0 2023-10-30T12:03:26.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 10 monclient(hunting): get_auth_request con 0x7f483c11c3e0 auth_method 0 2023-10-30T12:03:26.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0052a0 0x7f483c11bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 10 monclient(hunting): get_auth_request con 0x7f482c0052a0 auth_method 0 2023-10-30T12:03:26.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient(hunting): handle_auth_done global_id 44360 payload 824 2023-10-30T12:03:26.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0052a0 msgr2=0x7f483c11bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f482c0052a0 0x7f483c11bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f483c11c3e0 msgr2=0x7f483c11c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f483c11c3e0 0x7f483c11c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 monclient: found mon.noname-c 2023-10-30T12:03:26.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 -- 172.21.15.50:0/4057518943 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c11dbc0 con 0x7f483c1ca200 2023-10-30T12:03:26.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f483bfff640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f483c11c3e0 0x7f483c11c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:03:26.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f4841792640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f483c1ca200 0x7f483c1ca680 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f483402c800 tx=0x7f4834009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48397fa640 10 client.?.objecter ms_handle_connect 0x7f483c1ca200 2023-10-30T12:03:26.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48397fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:26.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.664+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834045070 con 0x7f483c1ca200 2023-10-30T12:03:26.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:26.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: dump: 2023-10-30T12:03:26.719 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.720 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.720 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.720 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.720 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.721 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 5 monclient: authenticate success, global_id 44360 2023-10-30T12:03:26.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4834035d20 con 0x7f483c1ca200 2023-10-30T12:03:26.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 10 monclient: _renew_subs 2023-10-30T12:03:26.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c11f350 con 0x7f483c1ca200 2023-10-30T12:03:26.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834003bb0 con 0x7f483c1ca200 2023-10-30T12:03:26.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:26.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 10 monclient: dump: 2023-10-30T12:03:26.724 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.725 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.725 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.725 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.725 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.725 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.726 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.726 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.726 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.726 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f483affd640 4 set_mon_vals no callback set 2023-10-30T12:03:26.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 10 client.44360.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:26.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 10 monclient: _renew_subs 2023-10-30T12:03:26.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c11ed90 con 0x7f483c1ca200 2023-10-30T12:03:26.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 1 librados: init done 2023-10-30T12:03:26.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c02c5b0 0x7f483c1cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4834040470 con 0x7f483c1ca200 2023-10-30T12:03:26.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.665+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c073ff0 con 0x7f483c02c5b0 2023-10-30T12:03:26.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c02c5b0 0x7f483c1cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient: get_auth_request con 0x7f483c02c5b0 auth_method 0 2023-10-30T12:03:26.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48200392f0 0x7f482003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7720+0+0 (secure 0 0 0) 0x7f483406d650 con 0x7f483c1ca200 2023-10-30T12:03:26.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 10 client.44360.objecter ms_dispatch 0x7f483c000b90 osd_map(82..82 src has 1..82) v4 2023-10-30T12:03:26.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 3 client.44360.objecter handle_osd_map got epochs [82,82] > 0 2023-10-30T12:03:26.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 3 client.44360.objecter handle_osd_map decoding full epoch 82 2023-10-30T12:03:26.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 20 client.44360.objecter dump_active .. 0 homeless 2023-10-30T12:03:26.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f483bfff640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48200392f0 0x7f482003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f483bfff640 10 monclient: get_auth_request con 0x7f48200392f0 auth_method 0 2023-10-30T12:03:26.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 10 monclient(hunting): handle_auth_done global_id 27991 payload 824 2023-10-30T12:03:26.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 1 -- 172.21.15.50:0/4057518943 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f483c09df80 con 0x7f483c02c5b0 2023-10-30T12:03:26.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f4840f91640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c02c5b0 0x7f483c1cabc0 secure :-1 s=READY pgs=7031 cs=0 l=1 rev1=1 crypto rx=0x7f483c06b530 tx=0x7f4830007780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 10 client.44360.objecter ms_handle_connect 0x7f483c02c5b0 2023-10-30T12:03:26.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 10 client.44360.objecter resend_mon_ops 2023-10-30T12:03:26.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300129a0 con 0x7f483c02c5b0 2023-10-30T12:03:26.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:26.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 10 monclient: dump: 2023-10-30T12:03:26.733 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.734 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.734 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.734 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.734 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.734 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.735 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.735 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.735 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.735 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f48397fa640 10 client.44360.objecter ms_handle_connect 0x7f48200392f0 2023-10-30T12:03:26.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.666+0000 7f483bfff640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48200392f0 0x7f482003b7b0 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7f483c11f280 tx=0x7f4828009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.667+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f4830012b00 con 0x7f483c02c5b0 2023-10-30T12:03:26.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.667+0000 7f48397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.667+0000 7f48397fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:26.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.667+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c02c5b0 msgr2=0x7f483c1cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.667+0000 7f48397fa640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c02c5b0 0x7f483c1cabc0 secure :-1 s=READY pgs=7031 cs=0 l=1 rev1=1 crypto rx=0x7f483c06b530 tx=0x7f4830007780 comp rx=0 tx=0).stop 2023-10-30T12:03:26.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.667+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c02c5b0 msgr2=0x7f483c1cabc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:26.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.667+0000 7f48397fa640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c02c5b0 0x7f483c1cabc0 secure :-1 s=CLOSED pgs=7031 cs=0 l=1 rev1=1 crypto rx=0x7f483c06b530 tx=0x7f4830007780 comp rx=0 tx=0).stop 2023-10-30T12:03:26.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f48431f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:26.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f48431f5640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c060fc0 0x7f483c0613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f48431f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c06a6f0 con 0x7f483c060fc0 2023-10-30T12:03:26.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c060fc0 0x7f483c0613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient: get_auth_request con 0x7f483c060fc0 auth_method 0 2023-10-30T12:03:26.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 10 monclient(hunting): handle_auth_done global_id 27994 payload 824 2023-10-30T12:03:26.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.673+0000 7f4840f91640 1 -- 172.21.15.50:0/4057518943 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f483c09f0b0 con 0x7f483c060fc0 2023-10-30T12:03:26.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f4840f91640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c060fc0 0x7f483c0613d0 secure :-1 s=READY pgs=7032 cs=0 l=1 rev1=1 crypto rx=0x7f483c06b530 tx=0x7f48300078f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 10 client.44360.objecter ms_handle_connect 0x7f483c060fc0 2023-10-30T12:03:26.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 10 client.44360.objecter resend_mon_ops 2023-10-30T12:03:26.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830026070 con 0x7f483c060fc0 2023-10-30T12:03:26.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:26.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 10 monclient: dump: 2023-10-30T12:03:26.743 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.743 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.743 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.744 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.744 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.744 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.744 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.744 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.745 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.745 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f483001f460 con 0x7f483c060fc0 2023-10-30T12:03:26.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:26.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:26.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c060fc0 msgr2=0x7f483c0613d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c060fc0 0x7f483c0613d0 secure :-1 s=READY pgs=7032 cs=0 l=1 rev1=1 crypto rx=0x7f483c06b530 tx=0x7f48300078f0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c060fc0 msgr2=0x7f483c0613d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:26.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48397fa640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f483c060fc0 0x7f483c0613d0 secure :-1 s=CLOSED pgs=7032 cs=0 l=1 rev1=1 crypto rx=0x7f483c06b530 tx=0x7f48300078f0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.747 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:26.747 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:26.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:26.748 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:03:26.748 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 30, 2023-10-30T12:03:26.748 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:26.748 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:03:26.748 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:26.749 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:26.749 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.749 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:03:26.749 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:26.749 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:26.750 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:26.750 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.750 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.750 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.750 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.751 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.751 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.751 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.751 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.751 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.751 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.752 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.752 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:26.752 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:26.752 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.752 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.752 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.753 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.753 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.753 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.753 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.753 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.754 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.754 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.754 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.754 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:26.755 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:26.755 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:26.755 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:26.755 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:26.755 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:26.755 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:26.756 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:26.756 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:26.756 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:26.756 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:26.756 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:26.757 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:26.757 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:26.757 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:26.757 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:26.757 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:26.758 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:26.758 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:26.758 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:26.758 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:26.758 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:26.759 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:26.759 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:26.759 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:26.759 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:26.759 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:26.760 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.760 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:26.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.760 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:26.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.760 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:26.761 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:26.761 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.761 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.761 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.762 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:26.762 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.762 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.762 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:26.763 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.763 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.763 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.763 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:26.764 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:26.764 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.764 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.764 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.765 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:26.765 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:26.765 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.765 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.766 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.766 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:26.766 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.766 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.767 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:26.767 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.767 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.767 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.768 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:26.768 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:26.768 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.768 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.768 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:26.769 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:26.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.770 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:26.770 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:26.770 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.771 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:26.771 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:26.771 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:26.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.771 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.772 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:26.772 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:26.772 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:26.773 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:26.773 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:26.773 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.773 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.774 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:26.774 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:26.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.774 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.774 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.775 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:26.775 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.775 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.775 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:03:26.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.776 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.776 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.776 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:26.776 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.776 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.777 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:26.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.777 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.777 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.778 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 10 2023-10-30T12:03:26.778 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.778 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:26.779 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:03:26.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:26.779 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:26.779 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:26.779 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:26.780 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:26.780 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:26.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:26.780 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:26.780 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 10 librados: watch_flush enter 2023-10-30T12:03:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 10 librados: watch_flush exit 2023-10-30T12:03:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 20 client.44360.objecter shutdown clearing up homeless session... 2023-10-30T12:03:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 10 client.44360.objecter successfully canceled tick 2023-10-30T12:03:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48200392f0 msgr2=0x7f482003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48200392f0 0x7f482003b7b0 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7f483c11f280 tx=0x7f4828009040 comp rx=0 tx=0).stop 2023-10-30T12:03:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 10 monclient: shutdown 2023-10-30T12:03:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f483c1ca200 msgr2=0x7f483c1ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 1 --2- 172.21.15.50:0/4057518943 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f483c1ca200 0x7f483c1ca680 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f483402c800 tx=0x7f4834009750 comp rx=0 tx=0).stop 2023-10-30T12:03:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f4841792640 1 -- 172.21.15.50:0/4057518943 reap_dead start 2023-10-30T12:03:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f4841792640 1 -- 172.21.15.50:0/4057518943 reap_dead start 2023-10-30T12:03:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 shutdown_connections 2023-10-30T12:03:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 >> 172.21.15.50:0/4057518943 conn(0x7f483c0a6200 msgr2=0x7f483c0705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.674+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 shutdown_connections 2023-10-30T12:03:26.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.675+0000 7f48431f5640 1 -- 172.21.15.50:0/4057518943 wait complete. 2023-10-30T12:03:26.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.675+0000 7f48431f5640 1 librados: shutdown 2023-10-30T12:03:26.786 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:03:26.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:03:26.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:03:26.787 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:03:26.787 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:03:26.788 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:03:26.788 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:03:26.801 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:03:26.801 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:03:26.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:26.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.947+0000 7f47e9461640 10 monclient: get_monmap_and_config 2023-10-30T12:03:26.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.947+0000 7f47e9461640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.947+0000 7f47e9461640 10 monclient: monmap: 2023-10-30T12:03:26.950 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.950 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.950 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:26.948879+0000 2023-10-30T12:03:26.950 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:26.948879+0000 2023-10-30T12:03:26.950 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.951 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.951 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:26.951 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:26.951 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:26.951 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:26.951 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:26.952 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.948+0000 7f47e9461640 1 Processor -- start 2023-10-30T12:03:26.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.948+0000 7f47e9461640 1 -- start start 2023-10-30T12:03:26.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.948+0000 7f47e9461640 10 monclient: init 2023-10-30T12:03:26.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.948+0000 7f47e9461640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.948+0000 7f47e9461640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T12:03:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4131b20 0x7f47e4132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 10 monclient(hunting): picked mon.noname-b con 0x7f47e4131b20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f47e4071bc0 0x7f47e4071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 10 monclient(hunting): picked mon.noname-c con 0x7f47e4071bc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47e4097410 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e409a9b0 con 0x7f47e4097410 2023-10-30T12:03:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e4131f00 con 0x7f47e4071bc0 2023-10-30T12:03:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e409d920 con 0x7f47e4131b20 2023-10-30T12:03:26.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e9461640 10 monclient(hunting): authenticate will time out at 1052.132084s 2023-10-30T12:03:26.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e37fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f47e4097410 legacy=0x7f47e40978a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:03:26.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4131b20 0x7f47e4132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4131b20 0x7f47e4132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40848/0 (socket says 172.21.15.50:40848) 2023-10-30T12:03:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 -- 172.21.15.50:0/548314716 learned_addr learned my addr 172.21.15.50:0/548314716 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): get_auth_request con 0x7f47e4131b20 auth_method 0 2023-10-30T12:03:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 1 --2- 172.21.15.50:0/548314716 >> v2:172.21.15.118:3300/0 conn(0x7f47e4071bc0 0x7f47e4071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 10 monclient(hunting): get_auth_request con 0x7f47e4071bc0 auth_method 0 2023-10-30T12:03:26.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient(hunting): handle_auth_done global_id 44363 payload 274 2023-10-30T12:03:26.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 -- 172.21.15.50:0/548314716 >> v2:172.21.15.118:3300/0 conn(0x7f47e4071bc0 msgr2=0x7f47e4071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:26.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 --2- 172.21.15.50:0/548314716 >> v2:172.21.15.118:3300/0 conn(0x7f47e4071bc0 0x7f47e4071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 -- 172.21.15.50:0/548314716 >> v1:172.21.15.50:6789/0 conn(0x7f47e4097410 legacy=0x7f47e40978a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:26.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 monclient: found mon.noname-b 2023-10-30T12:03:26.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.949+0000 7f47e2ffd640 1 -- 172.21.15.50:0/548314716 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e409a0b0 con 0x7f47e4131b20 2023-10-30T12:03:26.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e2ffd640 1 --2- 172.21.15.50:0/548314716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4131b20 0x7f47e4132270 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009850 tx=0x7f47d402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3112b81bc3a041bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 1 -- 172.21.15.50:0/548314716 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4004230 con 0x7f47e4131b20 2023-10-30T12:03:26.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:03:26.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: dump: 2023-10-30T12:03:26.968 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.968 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.968 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.969 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.969 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.969 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.969 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.969 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.970 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.970 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 1 -- 172.21.15.50:0/548314716 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d402f050 con 0x7f47e4131b20 2023-10-30T12:03:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 5 monclient: authenticate success, global_id 44363 2023-10-30T12:03:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 1 -- 172.21.15.50:0/548314716 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4003610 con 0x7f47e4131b20 2023-10-30T12:03:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 4 set_mon_vals no callback set 2023-10-30T12:03:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e27fc640 10 monclient: dump: 2023-10-30T12:03:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.973 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.973 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.973 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.973 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.973 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.973 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.974 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.974 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.974 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 10 monclient: shutdown 2023-10-30T12:03:26.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:26.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 1 -- 172.21.15.50:0/548314716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4131b20 msgr2=0x7f47e4132270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 1 --2- 172.21.15.50:0/548314716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4131b20 0x7f47e4132270 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009850 tx=0x7f47d402b7b0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 1 -- 172.21.15.50:0/548314716 shutdown_connections 2023-10-30T12:03:26.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 1 --2- 172.21.15.50:0/548314716 >> v2:172.21.15.118:3300/0 conn(0x7f47e4071bc0 0x7f47e4071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 1 --2- 172.21.15.50:0/548314716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4131b20 0x7f47e4132270 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 1 -- 172.21.15.50:0/548314716 >> 172.21.15.50:0/548314716 conn(0x7f47e40a05c0 msgr2=0x7f47e409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.950+0000 7f47e9461640 1 -- 172.21.15.50:0/548314716 shutdown_connections 2023-10-30T12:03:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 1 -- 172.21.15.50:0/548314716 wait complete. 2023-10-30T12:03:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 10 monclient: build_initial_monmap 2023-10-30T12:03:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 10 monclient: monmap: 2023-10-30T12:03:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:26.977 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:26.977 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:26.977 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:26.977 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:26.977 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.977 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:26.978 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:26.978 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:26.978 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 1 librados: starting msgr at 2023-10-30T12:03:26.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 1 librados: starting objecter 2023-10-30T12:03:26.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 1 Processor -- start 2023-10-30T12:03:26.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 1 -- start start 2023-10-30T12:03:26.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 1 librados: setting wanted keys 2023-10-30T12:03:26.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 1 librados: calling monclient init 2023-10-30T12:03:26.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.951+0000 7f47e9461640 10 monclient: init 2023-10-30T12:03:26.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:26.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:03:26.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47e4071bc0 0x7f47e413a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): picked mon.noname-a con 0x7f47e4071bc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:26.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4097410 0x7f47e413bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): picked mon.noname-c con 0x7f47e4097410 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:26.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47e4131b20 0x7f47e413c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): picked mon.noname-b con 0x7f47e4131b20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:26.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4138f60 con 0x7f47e4071bc0 2023-10-30T12:03:26.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41386b0 con 0x7f47e4131b20 2023-10-30T12:03:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e40993c0 con 0x7f47e4097410 2023-10-30T12:03:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e2ffd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47e4071bc0 msgr2=0x7f47e413a430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e9461640 10 monclient(hunting): authenticate will time out at 1052.135310s 2023-10-30T12:03:26.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47e4071bc0 0x7f47e413a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:26.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4097410 0x7f47e413bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4097410 0x7f47e413bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40850/0 (socket says 172.21.15.50:40850) 2023-10-30T12:03:26.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 -- 172.21.15.50:0/292589272 learned_addr learned my addr 172.21.15.50:0/292589272 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:26.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): get_auth_request con 0x7f47e4097410 auth_method 0 2023-10-30T12:03:26.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:26.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:26.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:26.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e37fe640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47e4131b20 0x7f47e413c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e37fe640 10 monclient(hunting): get_auth_request con 0x7f47e4131b20 auth_method 0 2023-10-30T12:03:26.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:26.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:26.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:26.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient(hunting): handle_auth_done global_id 44366 payload 824 2023-10-30T12:03:26.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 -- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47e4131b20 msgr2=0x7f47e413c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:26.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47e4131b20 0x7f47e413c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 -- 172.21.15.50:0/292589272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47e4071bc0 msgr2=0x7f47e413a430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:26.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47e4071bc0 0x7f47e413a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:26.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:03:26.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 monclient: found mon.noname-c 2023-10-30T12:03:26.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:26.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 -- 172.21.15.50:0/292589272 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4009460 con 0x7f47e4097410 2023-10-30T12:03:26.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47dbfff640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4097410 0x7f47e413bf80 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00e9b0 tx=0x7f47cc00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47db7fe640 10 client.?.objecter ms_handle_connect 0x7f47e4097410 2023-10-30T12:03:26.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.952+0000 7f47db7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:26.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 1 -- 172.21.15.50:0/292589272 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc009670 con 0x7f47e4097410 2023-10-30T12:03:26.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:26.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: dump: 2023-10-30T12:03:26.989 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.989 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.989 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.989 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.990 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.990 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.990 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.990 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.990 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 1 -- 172.21.15.50:0/292589272 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc00c9a0 con 0x7f47e4097410 2023-10-30T12:03:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 5 monclient: authenticate success, global_id 44366 2023-10-30T12:03:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 monclient: _renew_subs 2023-10-30T12:03:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 1 -- 172.21.15.50:0/292589272 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc00cd50 con 0x7f47e4097410 2023-10-30T12:03:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e409bc10 con 0x7f47e4097410 2023-10-30T12:03:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e1ffb640 4 set_mon_vals no callback set 2023-10-30T12:03:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 10 monclient: dump: 2023-10-30T12:03:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:26.995 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 client.44366.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 monclient: _renew_subs 2023-10-30T12:03:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e413da50 con 0x7f47e4097410 2023-10-30T12:03:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 1 librados: init done 2023-10-30T12:03:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47db7fe640 1 -- 172.21.15.50:0/292589272 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f47cc013470 con 0x7f47e4097410 2023-10-30T12:03:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.953+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e4139900 con 0x7f47e4097410 2023-10-30T12:03:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47db7fe640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47c40392a0 0x7f47c403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47db7fe640 1 -- 172.21.15.50:0/292589272 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7720+0+0 (secure 0 0 0) 0x7f47cc04b3e0 con 0x7f47e4097410 2023-10-30T12:03:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47db7fe640 10 client.44366.objecter ms_dispatch 0x7f47e4000b90 osd_map(82..82 src has 1..82) v4 2023-10-30T12:03:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47db7fe640 3 client.44366.objecter handle_osd_map got epochs [82,82] > 0 2023-10-30T12:03:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47db7fe640 3 client.44366.objecter handle_osd_map decoding full epoch 82 2023-10-30T12:03:26.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47db7fe640 20 client.44366.objecter dump_active .. 0 homeless 2023-10-30T12:03:26.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47e2ffd640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47c40392a0 0x7f47c403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:26.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47e2ffd640 10 monclient: get_auth_request con 0x7f47c40392a0 auth_method 0 2023-10-30T12:03:26.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47e2ffd640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47c40392a0 0x7f47c403b760 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7f47d4035000 tx=0x7f47d4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:26.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.954+0000 7f47db7fe640 10 client.44366.objecter ms_handle_connect 0x7f47c40392a0 2023-10-30T12:03:26.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.958+0000 7f47db7fe640 1 -- 172.21.15.50:0/292589272 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f47cc01bd80 con 0x7f47e4097410 2023-10-30T12:03:26.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.958+0000 7f47db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:27.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:26.958+0000 7f47db7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.202+0000 7f47e9461640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.202+0000 7f47e9461640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.202+0000 7f47e9461640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.202+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f47e409b8a0 con 0x7f47e4097410 2023-10-30T12:03:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.202+0000 7f47db7fe640 1 -- 172.21.15.50:0/292589272 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f47cc01bba0 con 0x7f47e4097410 2023-10-30T12:03:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.202+0000 7f47db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.202+0000 7f47db7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:27.205 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 10 librados: watch_flush enter 2023-10-30T12:03:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 10 librados: watch_flush exit 2023-10-30T12:03:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 20 client.44366.objecter shutdown clearing up homeless session... 2023-10-30T12:03:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 10 client.44366.objecter successfully canceled tick 2023-10-30T12:03:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47c40392a0 msgr2=0x7f47c403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47c40392a0 0x7f47c403b760 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7f47d4035000 tx=0x7f47d4008030 comp rx=0 tx=0).stop 2023-10-30T12:03:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 10 monclient: shutdown 2023-10-30T12:03:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4097410 msgr2=0x7f47e413bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4097410 0x7f47e413bf80 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00e9b0 tx=0x7f47cc00ee70 comp rx=0 tx=0).stop 2023-10-30T12:03:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 shutdown_connections 2023-10-30T12:03:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f47c40392a0 0x7f47c403b760 unknown :-1 s=CLOSED pgs=3877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f47e4131b20 0x7f47e413c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f47e4097410 0x7f47e413bf80 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 --2- 172.21.15.50:0/292589272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47e4071bc0 0x7f47e413a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 >> 172.21.15.50:0/292589272 conn(0x7f47e40a05c0 msgr2=0x7f47e409ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 shutdown_connections 2023-10-30T12:03:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 -- 172.21.15.50:0/292589272 wait complete. 2023-10-30T12:03:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:27.204+0000 7f47e9461640 1 librados: shutdown 2023-10-30T12:03:30.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.333+0000 7f4db88f2640 10 monclient: get_monmap_and_config 2023-10-30T12:03:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.333+0000 7f4db88f2640 10 monclient: build_initial_monmap 2023-10-30T12:03:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.333+0000 7f4db88f2640 10 monclient: monmap: 2023-10-30T12:03:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:30.334624+0000 2023-10-30T12:03:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:30.334624+0000 2023-10-30T12:03:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:30.337 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:30.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 1 Processor -- start 2023-10-30T12:03:30.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 1 -- start start 2023-10-30T12:03:30.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient: init 2023-10-30T12:03:30.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:30.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:03:30.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4db40a2dd0 0x7f4db40a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:30.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient(hunting): picked mon.noname-a con 0x7f4db40a2dd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:30.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4db406a8e0 0x7f4db40a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:30.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient(hunting): picked mon.noname-c con 0x7f4db406a8e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:30.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4db4071400 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:30.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db40a31b0 con 0x7f4db4071400 2023-10-30T12:03:30.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:30.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db406a6a0 con 0x7f4db40a2dd0 2023-10-30T12:03:30.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:30.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db406be90 con 0x7f4db406a8e0 2023-10-30T12:03:30.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:30.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db88f2640 10 monclient(hunting): authenticate will time out at 1055.517919s 2023-10-30T12:03:30.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db259c640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f4db40a2dd0 msgr2=0x7f4db40a46c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:30.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.334+0000 7f4db259c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4db40a2dd0 0x7f4db40a46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:30.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4db406a8e0 0x7f4db40a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:30.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4db406a8e0 0x7f4db40a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:43156/0 (socket says 172.21.15.50:43156) 2023-10-30T12:03:30.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 -- 172.21.15.50:0/3402448613 learned_addr learned my addr 172.21.15.50:0/3402448613 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:30.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): get_auth_request con 0x7f4db406a8e0 auth_method 0 2023-10-30T12:03:30.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:30.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:30.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:30.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:30.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:30.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:30.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient(hunting): handle_auth_done global_id 28003 payload 274 2023-10-30T12:03:30.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 -- 172.21.15.50:0/3402448613 >> v2:172.21.15.50:3300/0 conn(0x7f4db40a2dd0 msgr2=0x7f4db40a46c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:30.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 --2- 172.21.15.50:0/3402448613 >> v2:172.21.15.50:3300/0 conn(0x7f4db40a2dd0 0x7f4db40a46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 -- 172.21.15.50:0/3402448613 >> v1:172.21.15.118:6789/0 conn(0x7f4db4071400 legacy=0x7f4db413bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:30.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient: _finish_hunting 0 2023-10-30T12:03:30.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 monclient: found mon.noname-c 2023-10-30T12:03:30.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:30.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:03:30.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4dabfff640 1 -- 172.21.15.50:0/3402448613 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db409e190 con 0x7f4db406a8e0 2023-10-30T12:03:30.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4db1d9b640 1 -- 172.21.15.50:0/3402448613 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815368735 0 0) 0x7f4db40a31b0 con 0x7f4db4071400 2023-10-30T12:03:30.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.335+0000 7f4db1d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:03:30.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4dabfff640 1 --2- 172.21.15.50:0/3402448613 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db406a8e0 0x7f4db40a4c00 secure :-1 s=READY pgs=7040 cs=0 l=1 rev1=1 crypto rx=0x7f4db4073f20 tx=0x7f4d9c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad7491bd4a48d864 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:30.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 1 -- 172.21.15.50:0/3402448613 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c031800 con 0x7f4db406a8e0 2023-10-30T12:03:30.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:30.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:30.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: dump: 2023-10-30T12:03:30.354 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:30.354 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:30.354 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.354 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.355 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:30.355 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:30.355 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:30.355 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:30.355 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:30.356 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:30.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: _finish_auth 0 2023-10-30T12:03:30.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:03:30.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:30.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 1 -- 172.21.15.50:0/3402448613 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c031960 con 0x7f4db406a8e0 2023-10-30T12:03:30.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db88f2640 5 monclient: authenticate success, global_id 28003 2023-10-30T12:03:30.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db88f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:30.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:30.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db88f2640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:30.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db88f2640 4 set_mon_vals no callback set 2023-10-30T12:03:30.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 1 -- 172.21.15.50:0/3402448613 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c031cf0 con 0x7f4db406a8e0 2023-10-30T12:03:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.336+0000 7f4db1d9b640 10 monclient: dump: 2023-10-30T12:03:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:30.360 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:30.360 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:30.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 10 monclient: shutdown 2023-10-30T12:03:30.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:30.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 -- 172.21.15.50:0/3402448613 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db406a8e0 msgr2=0x7f4db40a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:30.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 --2- 172.21.15.50:0/3402448613 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db406a8e0 0x7f4db40a4c00 secure :-1 s=READY pgs=7040 cs=0 l=1 rev1=1 crypto rx=0x7f4db4073f20 tx=0x7f4d9c02b8a0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 -- 172.21.15.50:0/3402448613 shutdown_connections 2023-10-30T12:03:30.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 --2- 172.21.15.50:0/3402448613 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db406a8e0 0x7f4db40a4c00 unknown :-1 s=CLOSED pgs=7040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 --2- 172.21.15.50:0/3402448613 >> v2:172.21.15.50:3300/0 conn(0x7f4db40a2dd0 0x7f4db40a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 -- 172.21.15.50:0/3402448613 >> 172.21.15.50:0/3402448613 conn(0x7f4db40a22b0 msgr2=0x7f4db40a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 -- 172.21.15.50:0/3402448613 shutdown_connections 2023-10-30T12:03:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.337+0000 7f4db88f2640 1 -- 172.21.15.50:0/3402448613 wait complete. 2023-10-30T12:03:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 10 monclient: build_initial_monmap 2023-10-30T12:03:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 10 monclient: monmap: 2023-10-30T12:03:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:30.363 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:30.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:30.363 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:30.363 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:30.363 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:30.363 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:30.364 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 librados: starting msgr at 2023-10-30T12:03:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 librados: starting objecter 2023-10-30T12:03:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 Processor -- start 2023-10-30T12:03:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 -- start start 2023-10-30T12:03:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 librados: setting wanted keys 2023-10-30T12:03:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 librados: calling monclient init 2023-10-30T12:03:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 10 monclient: init 2023-10-30T12:03:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:03:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4db4071400 0x7f4db41c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 10 monclient(hunting): picked mon.noname-a con 0x7f4db4071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db40a2dd0 0x7f4db41c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 10 monclient(hunting): picked mon.noname-b con 0x7f4db40a2dd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.338+0000 7f4db88f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4db41c6d80 0x7f4db41c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient(hunting): picked mon.noname-c con 0x7f4db41c6d80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db406b130 con 0x7f4db4071400 2023-10-30T12:03:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:30.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41a8980 con 0x7f4db40a2dd0 2023-10-30T12:03:30.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:30.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40705b0 con 0x7f4db41c6d80 2023-10-30T12:03:30.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:30.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient(hunting): authenticate will time out at 1055.522034s 2023-10-30T12:03:30.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db259c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4db4071400 msgr2=0x7f4db41c3bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4db4071400 0x7f4db41c3bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4db41c6d80 0x7f4db41c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4db41c6d80 0x7f4db41c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53654/0 (socket says 172.21.15.50:53654) 2023-10-30T12:03:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 -- 172.21.15.50:0/2504503612 learned_addr learned my addr 172.21.15.50:0/2504503612 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): get_auth_request con 0x7f4db41c6d80 auth_method 0 2023-10-30T12:03:30.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:30.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:30.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:30.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:30.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db40a2dd0 0x7f4db41c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 10 monclient(hunting): get_auth_request con 0x7f4db40a2dd0 auth_method 0 2023-10-30T12:03:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:30.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:30.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:30.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4dabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:30.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient(hunting): handle_auth_done global_id 44369 payload 824 2023-10-30T12:03:30.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 -- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db40a2dd0 msgr2=0x7f4db41c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db40a2dd0 0x7f4db41c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 -- 172.21.15.50:0/2504503612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4db4071400 msgr2=0x7f4db41c3bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4db4071400 0x7f4db41c3bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:03:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 monclient: found mon.noname-c 2023-10-30T12:03:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 -- 172.21.15.50:0/2504503612 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c0094d0 con 0x7f4db41c6d80 2023-10-30T12:03:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db2d9d640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4db41c6d80 0x7f4db41c9d80 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c7f0 tx=0x7f4dac00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 client.?.objecter ms_handle_connect 0x7f4db41c6d80 2023-10-30T12:03:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 1 -- 172.21.15.50:0/2504503612 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac003da0 con 0x7f4db41c6d80 2023-10-30T12:03:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: dump: 2023-10-30T12:03:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:30.377 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: _finish_auth 0 2023-10-30T12:03:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:03:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 5 monclient: authenticate success, global_id 44369 2023-10-30T12:03:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 1 -- 172.21.15.50:0/2504503612 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac005070 con 0x7f4db41c6d80 2023-10-30T12:03:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 1 -- 172.21.15.50:0/2504503612 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac005420 con 0x7f4db41c6d80 2023-10-30T12:03:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient: _renew_subs 2023-10-30T12:03:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db159a640 4 set_mon_vals no callback set 2023-10-30T12:03:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db41aaf60 con 0x7f4db41c6d80 2023-10-30T12:03:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4daaffd640 10 monclient: dump: 2023-10-30T12:03:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:30.381 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 client.44369.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient: _renew_subs 2023-10-30T12:03:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db41a9200 con 0x7f4db41c6d80 2023-10-30T12:03:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.339+0000 7f4db88f2640 1 librados: init done 2023-10-30T12:03:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.340+0000 7f4daaffd640 1 -- 172.21.15.50:0/2504503612 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4dac005580 con 0x7f4db41c6d80 2023-10-30T12:03:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.340+0000 7f4db88f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.340+0000 7f4db88f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.340+0000 7f4db88f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.340+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db404e160 con 0x7f4db41c6d80 2023-10-30T12:03:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4daaffd640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4d940392a0 0x7f4d9403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4daaffd640 1 -- 172.21.15.50:0/2504503612 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7720+0+0 (secure 0 0 0) 0x7f4dac04aa00 con 0x7f4db41c6d80 2023-10-30T12:03:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4daaffd640 10 client.44369.objecter ms_dispatch 0x7f4db4000b90 osd_map(82..82 src has 1..82) v4 2023-10-30T12:03:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4daaffd640 3 client.44369.objecter handle_osd_map got epochs [82,82] > 0 2023-10-30T12:03:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4daaffd640 3 client.44369.objecter handle_osd_map decoding full epoch 82 2023-10-30T12:03:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4daaffd640 20 client.44369.objecter dump_active .. 0 homeless 2023-10-30T12:03:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4db259c640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4d940392a0 0x7f4d9403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4db259c640 10 monclient: get_auth_request con 0x7f4d940392a0 auth_method 0 2023-10-30T12:03:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4daaffd640 10 client.44369.objecter ms_handle_connect 0x7f4d940392a0 2023-10-30T12:03:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.341+0000 7f4db259c640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4d940392a0 0x7f4d9403b760 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f4db41a8c50 tx=0x7f4da4006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.345+0000 7f4daaffd640 1 -- 172.21.15.50:0/2504503612 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4dac01b3b0 con 0x7f4db41c6d80 2023-10-30T12:03:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.345+0000 7f4daaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.345+0000 7f4daaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:30.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.588+0000 7f4db88f2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:30.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.588+0000 7f4db88f2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:30.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.588+0000 7f4db88f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.588+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4db41ab5b0 con 0x7f4db41c6d80 2023-10-30T12:03:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.588+0000 7f4daaffd640 1 -- 172.21.15.50:0/2504503612 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f4dac017030 con 0x7f4db41c6d80 2023-10-30T12:03:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.589+0000 7f4daaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.589+0000 7f4daaffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:30.592 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:30.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 10 librados: watch_flush enter 2023-10-30T12:03:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 10 librados: watch_flush exit 2023-10-30T12:03:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 20 client.44369.objecter shutdown clearing up homeless session... 2023-10-30T12:03:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 10 client.44369.objecter successfully canceled tick 2023-10-30T12:03:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4d940392a0 msgr2=0x7f4d9403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4d940392a0 0x7f4d9403b760 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f4db41a8c50 tx=0x7f4da4006d30 comp rx=0 tx=0).stop 2023-10-30T12:03:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 10 monclient: shutdown 2023-10-30T12:03:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4db41c6d80 msgr2=0x7f4db41c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4db41c6d80 0x7f4db41c9d80 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c7f0 tx=0x7f4dac00ccb0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 shutdown_connections 2023-10-30T12:03:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4d940392a0 0x7f4d9403b760 unknown :-1 s=CLOSED pgs=3878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4db41c6d80 0x7f4db41c9d80 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4db40a2dd0 0x7f4db41c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 --2- 172.21.15.50:0/2504503612 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4db4071400 0x7f4db41c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.590+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 >> 172.21.15.50:0/2504503612 conn(0x7f4db40a22b0 msgr2=0x7f4db406e790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.591+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 shutdown_connections 2023-10-30T12:03:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.591+0000 7f4db88f2640 1 -- 172.21.15.50:0/2504503612 wait complete. 2023-10-30T12:03:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:30.591+0000 7f4db88f2640 1 librados: shutdown 2023-10-30T12:03:33.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.719+0000 7feb2c4c2640 10 monclient: get_monmap_and_config 2023-10-30T12:03:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.719+0000 7feb2c4c2640 10 monclient: build_initial_monmap 2023-10-30T12:03:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.719+0000 7feb2c4c2640 10 monclient: monmap: 2023-10-30T12:03:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:33.720217+0000 2023-10-30T12:03:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:33.720217+0000 2023-10-30T12:03:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:33.723 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.719+0000 7feb2c4c2640 1 Processor -- start 2023-10-30T12:03:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.719+0000 7feb2c4c2640 1 -- start start 2023-10-30T12:03:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.719+0000 7feb2c4c2640 10 monclient: init 2023-10-30T12:03:33.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-10-30T12:03:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb2409cd60 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb2409fd50 0x7feb24133e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient(hunting): picked mon.noname-b con 0x7feb2409fd50 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb241386c0 0x7feb2413aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient(hunting): picked mon.noname-a con 0x7feb241386c0 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2409d140 con 0x7feb2409cd60 2023-10-30T12:03:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2409dac0 con 0x7feb241386c0 2023-10-30T12:03:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2409dda0 con 0x7feb2409fd50 2023-10-30T12:03:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2c4c2640 10 monclient(hunting): authenticate will time out at 1058.903509s 2023-10-30T12:03:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2aa5f640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7feb241386c0 msgr2=0x7feb2413aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb2aa5f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb241386c0 0x7feb2413aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb2409fd50 0x7feb24133e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb2409fd50 0x7feb24133e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53670/0 (socket says 172.21.15.50:53670) 2023-10-30T12:03:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 1 -- 172.21.15.50:0/1293047959 learned_addr learned my addr 172.21.15.50:0/1293047959 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): get_auth_request con 0x7feb2409fd50 auth_method 0 2023-10-30T12:03:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:33.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:33.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:33.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:33.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:33.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient(hunting): handle_auth_done global_id 44372 payload 274 2023-10-30T12:03:33.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 1 -- 172.21.15.50:0/1293047959 >> v2:172.21.15.50:3300/0 conn(0x7feb241386c0 msgr2=0x7feb2413aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:33.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 1 --2- 172.21.15.50:0/1293047959 >> v2:172.21.15.50:3300/0 conn(0x7feb241386c0 0x7feb2413aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 1 -- 172.21.15.50:0/1293047959 >> v1:172.21.15.118:6789/0 conn(0x7feb2409cd60 legacy=0x7feb24133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:33.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 10 monclient: _finish_hunting 0 2023-10-30T12:03:33.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.720+0000 7feb29a5d640 1 monclient: found mon.noname-b 2023-10-30T12:03:33.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb29a5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:33.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb29a5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:03:33.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb29a5d640 1 -- 172.21.15.50:0/1293047959 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2409d470 con 0x7feb2409fd50 2023-10-30T12:03:33.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb29a5d640 1 --2- 172.21.15.50:0/1293047959 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb2409fd50 0x7feb24133e20 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7feb18009870 tx=0x7feb1802b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8563159042b18ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:33.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 1 -- 172.21.15.50:0/1293047959 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb180317e0 con 0x7feb2409fd50 2023-10-30T12:03:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:03:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: dump: 2023-10-30T12:03:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:33.740 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:33.740 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:33.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: _finish_auth 0 2023-10-30T12:03:33.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: _check_auth_tickets 2023-10-30T12:03:33.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:33.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 1 -- 172.21.15.50:0/1293047959 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18031940 con 0x7feb2409fd50 2023-10-30T12:03:33.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2c4c2640 5 monclient: authenticate success, global_id 44372 2023-10-30T12:03:33.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2c4c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:33.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:33.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2c4c2640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:33.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 1 -- 172.21.15.50:0/1293047959 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18031cd0 con 0x7feb2409fd50 2023-10-30T12:03:33.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2c4c2640 4 set_mon_vals no callback set 2023-10-30T12:03:33.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:33.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:33.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.721+0000 7feb2925c640 10 monclient: dump: 2023-10-30T12:03:33.742 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:33.742 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:33.743 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.743 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.743 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:33.743 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:33.743 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:33.744 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:33.744 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:33.744 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:33.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 10 monclient: shutdown 2023-10-30T12:03:33.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:33.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 -- 172.21.15.50:0/1293047959 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb2409fd50 msgr2=0x7feb24133e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:33.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/1293047959 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb2409fd50 0x7feb24133e20 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7feb18009870 tx=0x7feb1802b880 comp rx=0 tx=0).stop 2023-10-30T12:03:33.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 -- 172.21.15.50:0/1293047959 shutdown_connections 2023-10-30T12:03:33.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/1293047959 >> v2:172.21.15.50:3300/0 conn(0x7feb241386c0 0x7feb2413aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/1293047959 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb2409fd50 0x7feb24133e20 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 -- 172.21.15.50:0/1293047959 >> 172.21.15.50:0/1293047959 conn(0x7feb240a4670 msgr2=0x7feb240a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:33.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 -- 172.21.15.50:0/1293047959 shutdown_connections 2023-10-30T12:03:33.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.722+0000 7feb2c4c2640 1 -- 172.21.15.50:0/1293047959 wait complete. 2023-10-30T12:03:33.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 10 monclient: build_initial_monmap 2023-10-30T12:03:33.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 10 monclient: monmap: 2023-10-30T12:03:33.746 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:33.747 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:33.747 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:33.747 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:33.747 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:33.747 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:33.747 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:33.748 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:33.748 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:33.748 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:33.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 1 librados: starting msgr at 2023-10-30T12:03:33.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 1 librados: starting objecter 2023-10-30T12:03:33.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 1 Processor -- start 2023-10-30T12:03:33.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 1 -- start start 2023-10-30T12:03:33.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 1 librados: setting wanted keys 2023-10-30T12:03:33.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 1 librados: calling monclient init 2023-10-30T12:03:33.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.723+0000 7feb2c4c2640 10 monclient: init 2023-10-30T12:03:33.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:33.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:03:33.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2409cd60 0x7feb241c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:33.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): picked mon.noname-a con 0x7feb2409cd60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:33.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb241386c0 0x7feb24136f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:33.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): picked mon.noname-c con 0x7feb241386c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:33.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb24137450 0x7feb241378e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:33.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): picked mon.noname-b con 0x7feb24137450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:33.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:33.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2411c740 con 0x7feb2409cd60 2023-10-30T12:03:33.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:33.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2a25e640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2409cd60 msgr2=0x7feb241c8730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:33.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2411c990 con 0x7feb24137450 2023-10-30T12:03:33.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:33.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb24071010 con 0x7feb241386c0 2023-10-30T12:03:33.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:33.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient(hunting): authenticate will time out at 1058.907142s 2023-10-30T12:03:33.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2a25e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2409cd60 0x7feb241c8730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:33.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb241386c0 0x7feb24136f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:33.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb241386c0 0x7feb24136f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53682/0 (socket says 172.21.15.50:53682) 2023-10-30T12:03:33.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 -- 172.21.15.50:0/2448546222 learned_addr learned my addr 172.21.15.50:0/2448546222 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:33.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): get_auth_request con 0x7feb241386c0 auth_method 0 2023-10-30T12:03:33.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:33.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:33.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:33.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:33.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:33.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:33.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2aa5f640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb24137450 0x7feb241378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:33.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2aa5f640 10 monclient(hunting): get_auth_request con 0x7feb24137450 auth_method 0 2023-10-30T12:03:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2aa5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2aa5f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2aa5f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient(hunting): handle_auth_done global_id 44375 payload 824 2023-10-30T12:03:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 -- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb24137450 msgr2=0x7feb241378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb24137450 0x7feb241378e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 -- 172.21.15.50:0/2448546222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2409cd60 msgr2=0x7feb241c8730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2409cd60 0x7feb241c8730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient: _finish_hunting 0 2023-10-30T12:03:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 monclient: found mon.noname-c 2023-10-30T12:03:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 -- 172.21.15.50:0/2448546222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb180094d0 con 0x7feb241386c0 2023-10-30T12:03:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb29a5d640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb241386c0 0x7feb24136f10 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7feb1802b8c0 tx=0x7feb180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 client.?.objecter ms_handle_connect 0x7feb241386c0 2023-10-30T12:03:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 1 -- 172.21.15.50:0/2448546222 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb180329c0 con 0x7feb241386c0 2023-10-30T12:03:33.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:33.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:33.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: dump: 2023-10-30T12:03:33.758 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:33.758 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:33.759 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.759 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.759 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:33.759 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:33.759 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:33.759 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:33.760 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:33.760 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:33.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: _finish_auth 0 2023-10-30T12:03:33.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:03:33.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 1 -- 172.21.15.50:0/2448546222 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18032b40 con 0x7feb241386c0 2023-10-30T12:03:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 5 monclient: authenticate success, global_id 44375 2023-10-30T12:03:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 1 -- 172.21.15.50:0/2448546222 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb180453f0 con 0x7feb241386c0 2023-10-30T12:03:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient: _renew_subs 2023-10-30T12:03:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:33.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb241334e0 con 0x7feb241386c0 2023-10-30T12:03:33.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:33.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:33.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb28a5b640 4 set_mon_vals no callback set 2023-10-30T12:03:33.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb22ffd640 10 monclient: dump: 2023-10-30T12:03:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:33.764 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 client.44375.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient: _renew_subs 2023-10-30T12:03:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.724+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2411c0f0 con 0x7feb241386c0 2023-10-30T12:03:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.725+0000 7feb2c4c2640 1 librados: init done 2023-10-30T12:03:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.725+0000 7feb22ffd640 1 -- 172.21.15.50:0/2448546222 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7feb1804d3f0 con 0x7feb241386c0 2023-10-30T12:03:33.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.725+0000 7feb22ffd640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb100392d0 0x7feb1003b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:33.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.725+0000 7feb22ffd640 1 -- 172.21.15.50:0/2448546222 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7720+0+0 (secure 0 0 0) 0x7feb1806d080 con 0x7feb241386c0 2023-10-30T12:03:33.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.725+0000 7feb22ffd640 10 client.44375.objecter ms_dispatch 0x7feb24000b90 osd_map(82..82 src has 1..82) v4 2023-10-30T12:03:33.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.725+0000 7feb22ffd640 3 client.44375.objecter handle_osd_map got epochs [82,82] > 0 2023-10-30T12:03:33.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.725+0000 7feb22ffd640 3 client.44375.objecter handle_osd_map decoding full epoch 82 2023-10-30T12:03:33.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb2c4c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:33.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb2c4c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:33.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb2c4c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:33.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb22ffd640 20 client.44375.objecter dump_active .. 0 homeless 2023-10-30T12:03:33.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb241332f0 con 0x7feb241386c0 2023-10-30T12:03:33.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb2a25e640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb100392d0 0x7feb1003b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:33.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb2a25e640 10 monclient: get_auth_request con 0x7feb100392d0 auth_method 0 2023-10-30T12:03:33.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb2a25e640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb100392d0 0x7feb1003b790 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7feb1c001030 tx=0x7feb1c007a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:33.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.726+0000 7feb22ffd640 10 client.44375.objecter ms_handle_connect 0x7feb100392d0 2023-10-30T12:03:33.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.730+0000 7feb22ffd640 1 -- 172.21.15.50:0/2448546222 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb1803b3f0 con 0x7feb241386c0 2023-10-30T12:03:33.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.730+0000 7feb22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:33.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.730+0000 7feb22ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:33.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.972+0000 7feb2c4c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:33.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.972+0000 7feb2c4c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:33.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.972+0000 7feb2c4c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:33.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.972+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feb2404e160 con 0x7feb241386c0 2023-10-30T12:03:33.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.973+0000 7feb22ffd640 1 -- 172.21.15.50:0/2448546222 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7feb18039310 con 0x7feb241386c0 2023-10-30T12:03:33.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.973+0000 7feb22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:33.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.973+0000 7feb22ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:33.976 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:33.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.974+0000 7feb2c4c2640 10 librados: watch_flush enter 2023-10-30T12:03:33.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.974+0000 7feb2c4c2640 10 librados: watch_flush exit 2023-10-30T12:03:33.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.974+0000 7feb2c4c2640 20 client.44375.objecter shutdown clearing up homeless session... 2023-10-30T12:03:33.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.974+0000 7feb2c4c2640 10 client.44375.objecter successfully canceled tick 2023-10-30T12:03:33.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.974+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb100392d0 msgr2=0x7feb1003b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:33.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.974+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb100392d0 0x7feb1003b790 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7feb1c001030 tx=0x7feb1c007a20 comp rx=0 tx=0).stop 2023-10-30T12:03:33.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 10 monclient: shutdown 2023-10-30T12:03:33.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:33.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb241386c0 msgr2=0x7feb24136f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:33.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb241386c0 0x7feb24136f10 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7feb1802b8c0 tx=0x7feb180046c0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 shutdown_connections 2023-10-30T12:03:33.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb100392d0 0x7feb1003b790 unknown :-1 s=CLOSED pgs=3879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb24137450 0x7feb241378e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb241386c0 0x7feb24136f10 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 --2- 172.21.15.50:0/2448546222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2409cd60 0x7feb241c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:33.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 >> 172.21.15.50:0/2448546222 conn(0x7feb240a4670 msgr2=0x7feb241395f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:33.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 shutdown_connections 2023-10-30T12:03:33.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 -- 172.21.15.50:0/2448546222 wait complete. 2023-10-30T12:03:33.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:33.975+0000 7feb2c4c2640 1 librados: shutdown 2023-10-30T12:03:36.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:03:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.108+0000 7f7a36f25640 10 monclient: get_monmap_and_config 2023-10-30T12:03:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.108+0000 7f7a36f25640 10 monclient: build_initial_monmap 2023-10-30T12:03:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.108+0000 7f7a36f25640 10 monclient: monmap: 2023-10-30T12:03:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:37.109228+0000 2023-10-30T12:03:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:37.109228+0000 2023-10-30T12:03:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:37.112 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:37.112 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:37.112 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.108+0000 7f7a36f25640 1 Processor -- start 2023-10-30T12:03:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.108+0000 7f7a36f25640 1 -- start start 2023-10-30T12:03:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.108+0000 7f7a36f25640 10 monclient: init 2023-10-30T12:03:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:03:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient(hunting): picked mon.noname-c con 0x7f7a30131b20 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a30071bc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 0x7f7a30097b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient(hunting): picked mon.noname-a con 0x7f7a30071fc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3009a9b0 con 0x7f7a30071bc0 2023-10-30T12:03:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a30131f00 con 0x7f7a30071fc0 2023-10-30T12:03:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3009d920 con 0x7f7a30131b20 2023-10-30T12:03:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a2ffff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f7a30071bc0 legacy=0x7f7a30097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:03:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a36f25640 10 monclient(hunting): authenticate will time out at 1062.292508s 2023-10-30T12:03:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a354c2640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 msgr2=0x7f7a30097b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a354c2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 0x7f7a30097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a34cc1640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a34cc1640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:43182/0 (socket says 172.21.15.50:43182) 2023-10-30T12:03:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 learned_addr learned my addr 172.21.15.50:0/4240104763 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request con 0x7f7a30131b20 auth_method 0 2023-10-30T12:03:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a34cc1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.109+0000 7f7a34cc1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.110+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.110+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.110+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.110+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.310+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> v1:172.21.15.50:6789/0 conn(0x7f7a30071bc0 legacy=0x7f7a30097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:03:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.310+0000 7f7a354c2640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 msgr2=0x7f7a30097b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:37.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.310+0000 7f7a354c2640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 0x7f7a30097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:03:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request con 0x7f7a30131b20 auth_method 0 2023-10-30T12:03:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.311+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:03:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.711+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> v1:172.21.15.50:6789/0 conn(0x7f7a30071bc0 legacy=0x7f7a30097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:03:37.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.711+0000 7f7a354c2640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 msgr2=0x7f7a30097b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:37.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.711+0000 7f7a354c2640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 0x7f7a30097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:03:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request con 0x7f7a30131b20 auth_method 0 2023-10-30T12:03:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:37.712+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:03:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.512+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> v1:172.21.15.50:6789/0 conn(0x7f7a30071bc0 legacy=0x7f7a30097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:03:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.512+0000 7f7a354c2640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 msgr2=0x7f7a30097b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.512+0000 7f7a354c2640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 0x7f7a30097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:03:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request con 0x7f7a30131b20 auth_method 0 2023-10-30T12:03:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:03:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:38.514+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:03:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient(hunting): tick 2023-10-30T12:03:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:03:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 monclient(hunting): continuing hunt 2023-10-30T12:03:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:03:40.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 msgr2=0x7f7a30132270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.118:3300/0 conn(0x7f7a30131b20 0x7f7a30132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 -- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 msgr2=0x7f7a30097b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 --2- 172.21.15.50:0/4240104763 >> v2:172.21.15.50:3300/0 conn(0x7f7a30071fc0 0x7f7a30097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 -- 172.21.15.50:0/4240104763 >> v1:172.21.15.50:6789/0 conn(0x7f7a30071bc0 legacy=0x7f7a30097410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:40.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient: _add_conns ranks=[1,2] 2023-10-30T12:03:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a14003690 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f7a14007360 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 -- 172.21.15.50:0/4240104763 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3009a0b0 con 0x7f7a14003690 2023-10-30T12:03:40.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 1 -- 172.21.15.50:0/4240104763 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30131f00 con 0x7f7a14007360 2023-10-30T12:03:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2e7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.109+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request con 0x7f7a14007360 auth_method 0 2023-10-30T12:03:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:40.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 msgr2=0x7f7a14009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 msgr2=0x7f7a14009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:40.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.110+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:40.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request con 0x7f7a14007360 auth_method 0 2023-10-30T12:03:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 msgr2=0x7f7a14009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:40.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 msgr2=0x7f7a14009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:40.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:40.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.311+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:03:40.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:40.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request con 0x7f7a14007360 auth_method 0 2023-10-30T12:03:40.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:40.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:40.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:40.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 msgr2=0x7f7a14009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:03:40.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 msgr2=0x7f7a14009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:03:40.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:03:40.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:40.712+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:03:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.412+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162383006 0 0) 0x7f7a3009a0b0 con 0x7f7a14003690 2023-10-30T12:03:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.412+0000 7f7a2f7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:03:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.412+0000 7f7a2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.412+0000 7f7a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.412+0000 7f7a2f7fe640 10 monclient(hunting): my global_id is 28015 2023-10-30T12:03:41.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.412+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a20003520 con 0x7f7a14003690 2023-10-30T12:03:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420860778 0 0) 0x7f7a20003520 con 0x7f7a14003690 2023-10-30T12:03:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 msgr2=0x7f7a14009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a14007360 0x7f7a14009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:03:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:03:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:03:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3009a9b0 con 0x7f7a14003690 2023-10-30T12:03:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:03:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:03:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f7a2400e7d0 con 0x7f7a14003690 2023-10-30T12:03:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a36f25640 5 monclient: authenticate success, global_id 28015 2023-10-30T12:03:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a36f25640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:03:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: dump: 2023-10-30T12:03:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.443 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:03:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 >> v1:172.21.15.118:6789/0 conn(0x7f7a14003690 legacy=0x7f7a14005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a354c2640 1 -- 172.21.15.50:0/4240104763 reap_dead start 2023-10-30T12:03:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:03:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a20003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): picked mon.a con 0x7f7a20003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a20004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): picked mon.c con 0x7f7a200047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a2000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): picked mon.b con 0x7f7a200051d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30131f00 con 0x7f7a20003970 2023-10-30T12:03:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2000c100 con 0x7f7a200051d0 2023-10-30T12:03:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2000c2c0 con 0x7f7a200047c0 2023-10-30T12:03:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a2f7fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:03:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a34cc1640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 msgr2=0x7f7a20003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.434+0000 7f7a36f25640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a20003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a20004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request con 0x7f7a200047c0 auth_method 0 2023-10-30T12:03:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a2000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 10 monclient(hunting): get_auth_request con 0x7f7a200051d0 auth_method 0 2023-10-30T12:03:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:03:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a354c2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:03:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_done global_id 28015 payload 293 2023-10-30T12:03:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 msgr2=0x7f7a2000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a2000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 msgr2=0x7f7a20003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a20003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:03:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 1 monclient: found mon.c 2023-10-30T12:03:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 1 -- 172.21.15.50:0/4240104763 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a20003520 con 0x7f7a200047c0 2023-10-30T12:03:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient: _finish_auth 0 2023-10-30T12:03:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:03:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.435+0000 7f7a2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.436+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a20004c40 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c003980 tx=0x7f7a200057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd8cf9d40e9b169d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c033650 con 0x7f7a200047c0 2023-10-30T12:03:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 10 monclient: dump: 2023-10-30T12:03:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.462 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a36f25640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c006e90 con 0x7f7a200047c0 2023-10-30T12:03:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a36f25640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 1 -- 172.21.15.50:0/4240104763 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0339f0 con 0x7f7a200047c0 2023-10-30T12:03:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a36f25640 4 set_mon_vals no callback set 2023-10-30T12:03:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a2f7fe640 10 monclient: dump: 2023-10-30T12:03:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.465 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a36f25640 10 monclient: shutdown 2023-10-30T12:03:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a36f25640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a36f25640 1 -- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 msgr2=0x7f7a20004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.437+0000 7f7a36f25640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a20004c40 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c003980 tx=0x7f7a200057f0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 1 -- 172.21.15.50:0/4240104763 shutdown_connections 2023-10-30T12:03:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a2000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a20004c40 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 1 --2- 172.21.15.50:0/4240104763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a20003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 1 -- 172.21.15.50:0/4240104763 >> 172.21.15.50:0/4240104763 conn(0x7f7a300a05c0 msgr2=0x7f7a3009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 1 -- 172.21.15.50:0/4240104763 shutdown_connections 2023-10-30T12:03:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 1 -- 172.21.15.50:0/4240104763 wait complete. 2023-10-30T12:03:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 10 monclient: build_initial_monmap 2023-10-30T12:03:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.438+0000 7f7a36f25640 10 monclient: monmap: 2023-10-30T12:03:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:41.470 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.439+0000 7f7a36f25640 1 librados: starting msgr at 2023-10-30T12:03:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.439+0000 7f7a36f25640 1 librados: starting objecter 2023-10-30T12:03:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.439+0000 7f7a36f25640 1 Processor -- start 2023-10-30T12:03:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.439+0000 7f7a36f25640 1 -- start start 2023-10-30T12:03:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.439+0000 7f7a36f25640 1 librados: setting wanted keys 2023-10-30T12:03:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.439+0000 7f7a36f25640 1 librados: calling monclient init 2023-10-30T12:03:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.439+0000 7f7a36f25640 10 monclient: init 2023-10-30T12:03:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:03:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a301c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): picked mon.noname-a con 0x7f7a20003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a3012e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): picked mon.noname-c con 0x7f7a200047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a3012ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): picked mon.noname-b con 0x7f7a200051d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301c5b50 con 0x7f7a20003970 2023-10-30T12:03:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30070ff0 con 0x7f7a200051d0 2023-10-30T12:03:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300712d0 con 0x7f7a200047c0 2023-10-30T12:03:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a36f25640 10 monclient(hunting): authenticate will time out at 1066.623246s 2023-10-30T12:03:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a34cc1640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 msgr2=0x7f7a301c5610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a34cc1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a301c5610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a3012e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a3012e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57210/0 (socket says 172.21.15.50:57210) 2023-10-30T12:03:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 -- 172.21.15.50:0/640106362 learned_addr learned my addr 172.21.15.50:0/640106362 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request con 0x7f7a200047c0 auth_method 0 2023-10-30T12:03:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a3012ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 10 monclient(hunting): get_auth_request con 0x7f7a200051d0 auth_method 0 2023-10-30T12:03:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a354c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_done global_id 44378 payload 824 2023-10-30T12:03:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 -- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 msgr2=0x7f7a3012ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a3012ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 -- 172.21.15.50:0/640106362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 msgr2=0x7f7a301c5610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a301c5610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:03:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 monclient: found mon.noname-c 2023-10-30T12:03:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.440+0000 7f7a2ffff640 1 -- 172.21.15.50:0/640106362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300715b0 con 0x7f7a200047c0 2023-10-30T12:03:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2ffff640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a3012e8d0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c035040 tx=0x7f7a1c005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 client.?.objecter ms_handle_connect 0x7f7a200047c0 2023-10-30T12:03:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 1 -- 172.21.15.50:0/640106362 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c033da0 con 0x7f7a200047c0 2023-10-30T12:03:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: dump: 2023-10-30T12:03:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.484 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:03:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:03:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 1 -- 172.21.15.50:0/640106362 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c045410 con 0x7f7a200047c0 2023-10-30T12:03:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 5 monclient: authenticate success, global_id 44378 2023-10-30T12:03:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 10 monclient: _renew_subs 2023-10-30T12:03:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a30070320 con 0x7f7a200047c0 2023-10-30T12:03:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 1 -- 172.21.15.50:0/640106362 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0457c0 con 0x7f7a200047c0 2023-10-30T12:03:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2d7fa640 10 monclient: dump: 2023-10-30T12:03:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.488 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a2effd640 4 set_mon_vals no callback set 2023-10-30T12:03:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 10 client.44378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 10 monclient: _renew_subs 2023-10-30T12:03:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3006f4c0 con 0x7f7a200047c0 2023-10-30T12:03:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.441+0000 7f7a36f25640 1 librados: init done 2023-10-30T12:03:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a2d7fa640 1 -- 172.21.15.50:0/640106362 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7a1c03c3f0 con 0x7f7a200047c0 2023-10-30T12:03:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a2d7fa640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a14039a60 0x7f7a1403bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a2d7fa640 1 -- 172.21.15.50:0/640106362 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7720+0+0 (secure 0 0 0) 0x7f7a1c06d880 con 0x7f7a200047c0 2023-10-30T12:03:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a2d7fa640 10 client.44378.objecter ms_dispatch 0x7f7a30000b90 osd_map(82..82 src has 1..82) v4 2023-10-30T12:03:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a2d7fa640 3 client.44378.objecter handle_osd_map got epochs [82,82] > 0 2023-10-30T12:03:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a2d7fa640 3 client.44378.objecter handle_osd_map decoding full epoch 82 2023-10-30T12:03:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a2d7fa640 20 client.44378.objecter dump_active .. 0 homeless 2023-10-30T12:03:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a36f25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a36f25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a36f25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a30071740 con 0x7f7a200047c0 2023-10-30T12:03:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.442+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a14039a60 0x7f7a1403bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.443+0000 7f7a34cc1640 10 monclient: get_auth_request con 0x7f7a14039a60 auth_method 0 2023-10-30T12:03:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.443+0000 7f7a2d7fa640 10 client.44378.objecter ms_handle_connect 0x7f7a14039a60 2023-10-30T12:03:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.443+0000 7f7a34cc1640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a14039a60 0x7f7a1403bf20 secure :-1 s=READY pgs=3880 cs=0 l=1 rev1=1 crypto rx=0x7f7a24006d40 tx=0x7f7a24006b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.448+0000 7f7a2d7fa640 1 -- 172.21.15.50:0/640106362 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a1c04a080 con 0x7f7a200047c0 2023-10-30T12:03:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.448+0000 7f7a2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.448+0000 7f7a2d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:41.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.737+0000 7f7a36f25640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:03:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.737+0000 7f7a36f25640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.737+0000 7f7a36f25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.737+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7a3004e160 con 0x7f7a200047c0 2023-10-30T12:03:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.738+0000 7f7a2d7fa640 1 -- 172.21.15.50:0/640106362 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f7a1c043310 con 0x7f7a200047c0 2023-10-30T12:03:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.738+0000 7f7a2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:03:41.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.738+0000 7f7a2d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:41.741 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:03:41.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 10 librados: watch_flush enter 2023-10-30T12:03:41.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 10 librados: watch_flush exit 2023-10-30T12:03:41.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 20 client.44378.objecter shutdown clearing up homeless session... 2023-10-30T12:03:41.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 10 client.44378.objecter successfully canceled tick 2023-10-30T12:03:41.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a14039a60 msgr2=0x7f7a1403bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a14039a60 0x7f7a1403bf20 secure :-1 s=READY pgs=3880 cs=0 l=1 rev1=1 crypto rx=0x7f7a24006d40 tx=0x7f7a24006b90 comp rx=0 tx=0).stop 2023-10-30T12:03:41.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 10 monclient: shutdown 2023-10-30T12:03:41.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:41.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 msgr2=0x7f7a3012e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.739+0000 7f7a36f25640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a3012e8d0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c035040 tx=0x7f7a1c005db0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 shutdown_connections 2023-10-30T12:03:41.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a14039a60 0x7f7a1403bf20 unknown :-1 s=CLOSED pgs=3880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a200051d0 0x7f7a3012ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a200047c0 0x7f7a3012e8d0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 --2- 172.21.15.50:0/640106362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a20003970 0x7f7a301c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 >> 172.21.15.50:0/640106362 conn(0x7f7a300a05c0 msgr2=0x7f7a3009b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:41.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 shutdown_connections 2023-10-30T12:03:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 -- 172.21.15.50:0/640106362 wait complete. 2023-10-30T12:03:41.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.740+0000 7f7a36f25640 1 librados: shutdown 2023-10-30T12:03:41.751 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:03:41.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:03:41.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:03:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.857+0000 7fceb58f2640 10 monclient: get_monmap_and_config 2023-10-30T12:03:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.857+0000 7fceb58f2640 10 monclient: build_initial_monmap 2023-10-30T12:03:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.857+0000 7fceb58f2640 10 monclient: monmap: 2023-10-30T12:03:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:41.859 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:41.858226+0000 2023-10-30T12:03:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:41.858226+0000 2023-10-30T12:03:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:41.861 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.857+0000 7fceb58f2640 1 Processor -- start 2023-10-30T12:03:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.857+0000 7fceb58f2640 1 -- start start 2023-10-30T12:03:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.857+0000 7fceb58f2640 10 monclient: init 2023-10-30T12:03:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-10-30T12:03:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb0132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient(hunting): picked mon.noname-b con 0x7fceb00a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fceb0099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:03:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fceb0132bd0 0x7fceb0135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient(hunting): picked mon.noname-c con 0x7fceb0132bd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb00a7190 con 0x7fceb0099a40 2023-10-30T12:03:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb00a83f0 con 0x7fceb0132bd0 2023-10-30T12:03:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00a7600 con 0x7fceb00a8010 2023-10-30T12:03:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceb58f2640 10 monclient(hunting): authenticate will time out at 1067.041269s 2023-10-30T12:03:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb0132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb0132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57222/0 (socket says 172.21.15.50:57222) 2023-10-30T12:03:41.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 -- 172.21.15.50:0/4015876719 learned_addr learned my addr 172.21.15.50:0/4015876719 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): get_auth_request con 0x7fceb00a8010 auth_method 0 2023-10-30T12:03:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient(hunting): handle_auth_done global_id 44381 payload 274 2023-10-30T12:03:41.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaf7fe640 1 --2- 172.21.15.50:0/4015876719 >> v2:172.21.15.118:3300/0 conn(0x7fceb0132bd0 0x7fceb0135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 -- 172.21.15.50:0/4015876719 >> v2:172.21.15.118:3300/0 conn(0x7fceb0132bd0 msgr2=0x7fceb0135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 --2- 172.21.15.50:0/4015876719 >> v2:172.21.15.118:3300/0 conn(0x7fceb0132bd0 0x7fceb0135de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 -- 172.21.15.50:0/4015876719 >> v1:172.21.15.118:6789/0 conn(0x7fceb0099a40 legacy=0x7fceb01356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:03:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:03:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 monclient: found mon.noname-b 2023-10-30T12:03:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.858+0000 7fceaeffd640 1 -- 172.21.15.50:0/4015876719 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb009a440 con 0x7fceb00a8010 2023-10-30T12:03:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceaeffd640 1 --2- 172.21.15.50:0/4015876719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb0132690 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009870 tx=0x7fcea002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2650b87261b8518d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 1 -- 172.21.15.50:0/4015876719 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0031740 con 0x7fceb00a8010 2023-10-30T12:03:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:03:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: dump: 2023-10-30T12:03:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.879 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.880 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: _finish_auth 0 2023-10-30T12:03:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:03:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 1 -- 172.21.15.50:0/4015876719 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea00318a0 con 0x7fceb00a8010 2023-10-30T12:03:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceb58f2640 5 monclient: authenticate success, global_id 44381 2023-10-30T12:03:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceb58f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceb58f2640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 1 -- 172.21.15.50:0/4015876719 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0031c30 con 0x7fceb00a8010 2023-10-30T12:03:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceb58f2640 4 set_mon_vals no callback set 2023-10-30T12:03:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.859+0000 7fceae7fc640 10 monclient: dump: 2023-10-30T12:03:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.885 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 10 monclient: shutdown 2023-10-30T12:03:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 -- 172.21.15.50:0/4015876719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 msgr2=0x7fceb0132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 --2- 172.21.15.50:0/4015876719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb0132690 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009870 tx=0x7fcea002b7e0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 -- 172.21.15.50:0/4015876719 shutdown_connections 2023-10-30T12:03:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 --2- 172.21.15.50:0/4015876719 >> v2:172.21.15.118:3300/0 conn(0x7fceb0132bd0 0x7fceb0135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 --2- 172.21.15.50:0/4015876719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb0132690 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 -- 172.21.15.50:0/4015876719 >> 172.21.15.50:0/4015876719 conn(0x7fceb00a4a10 msgr2=0x7fceb00a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 -- 172.21.15.50:0/4015876719 shutdown_connections 2023-10-30T12:03:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 -- 172.21.15.50:0/4015876719 wait complete. 2023-10-30T12:03:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 10 monclient: build_initial_monmap 2023-10-30T12:03:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 10 monclient: monmap: 2023-10-30T12:03:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:41.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 librados: starting msgr at 2023-10-30T12:03:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 librados: starting objecter 2023-10-30T12:03:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.860+0000 7fceb58f2640 1 Processor -- start 2023-10-30T12:03:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 -- start start 2023-10-30T12:03:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 librados: setting wanted keys 2023-10-30T12:03:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 librados: calling monclient init 2023-10-30T12:03:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient: init 2023-10-30T12:03:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:03:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb01ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): picked mon.noname-c con 0x7fceb00a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceb0132bd0 0x7fceb01cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): picked mon.noname-a con 0x7fceb0132bd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceb011e740 0x7fceb011ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): picked mon.noname-b con 0x7fceb011e740 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01209d0 con 0x7fceb0132bd0 2023-10-30T12:03:41.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0121f10 con 0x7fceb011e740 2023-10-30T12:03:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0123190 con 0x7fceb00a8010 2023-10-30T12:03:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:41.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceb58f2640 10 monclient(hunting): authenticate will time out at 1067.044642s 2023-10-30T12:03:41.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fcea67fc640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceb0132bd0 msgr2=0x7fceb01cab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:41.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fcea67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceb0132bd0 0x7fceb01cab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:41.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb01ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb01ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57232/0 (socket says 172.21.15.50:57232) 2023-10-30T12:03:41.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 -- 172.21.15.50:0/1656084569 learned_addr learned my addr 172.21.15.50:0/1656084569 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:41.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): get_auth_request con 0x7fceb00a8010 auth_method 0 2023-10-30T12:03:41.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaf7fe640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceb011e740 0x7fceb011ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaf7fe640 10 monclient(hunting): get_auth_request con 0x7fceb011e740 auth_method 0 2023-10-30T12:03:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient(hunting): handle_auth_done global_id 44384 payload 824 2023-10-30T12:03:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceb011e740 msgr2=0x7fceb011ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceb011e740 0x7fceb011ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceb0132bd0 msgr2=0x7fceb01cab70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceb0132bd0 0x7fceb01cab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:03:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 monclient: found mon.noname-c 2023-10-30T12:03:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:41.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.861+0000 7fceaeffd640 1 -- 172.21.15.50:0/1656084569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00094d0 con 0x7fceb00a8010 2023-10-30T12:03:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceaeffd640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb01ca630 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fcea0000c00 tx=0x7fcea00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 client.?.objecter ms_handle_connect 0x7fceb00a8010 2023-10-30T12:03:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0033d60 con 0x7fceb00a8010 2023-10-30T12:03:41.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: dump: 2023-10-30T12:03:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.900 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.901 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.901 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: _finish_auth 0 2023-10-30T12:03:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: _check_auth_tickets 2023-10-30T12:03:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea003c400 con 0x7fceb00a8010 2023-10-30T12:03:41.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 5 monclient: authenticate success, global_id 44384 2023-10-30T12:03:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 10 monclient: _renew_subs 2023-10-30T12:03:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb01359e0 con 0x7fceb00a8010 2023-10-30T12:03:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea003c7b0 con 0x7fceb00a8010 2023-10-30T12:03:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:41.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 10 client.44384.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:41.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceadffb640 4 set_mon_vals no callback set 2023-10-30T12:03:41.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceacff9640 10 monclient: dump: 2023-10-30T12:03:41.904 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.905 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.905 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.905 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.905 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.905 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.905 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.906 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.906 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.906 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 10 monclient: _renew_subs 2023-10-30T12:03:41.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:41.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb00a18d0 con 0x7fceb00a8010 2023-10-30T12:03:41.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.862+0000 7fceb58f2640 1 librados: init done 2023-10-30T12:03:41.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcea003a040 con 0x7fceb00a8010 2023-10-30T12:03:41.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceb58f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:41.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceb58f2640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb0099a40 0x7fceb002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceb58f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0073fd0 con 0x7fceb0099a40 2023-10-30T12:03:41.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb0099a40 0x7fceb002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient: get_auth_request con 0x7fceb0099a40 auth_method 0 2023-10-30T12:03:41.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce900392a0 0x7fce9003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:41.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8302+0+0 (secure 0 0 0) 0x7fcea006d230 con 0x7fceb00a8010 2023-10-30T12:03:41.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 10 client.44384.objecter ms_dispatch 0x7fceb0000b90 osd_map(83..83 src has 1..83) v4 2023-10-30T12:03:41.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 3 client.44384.objecter handle_osd_map got epochs [83,83] > 0 2023-10-30T12:03:41.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 3 client.44384.objecter handle_osd_map decoding full epoch 83 2023-10-30T12:03:41.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 10 monclient(hunting): handle_auth_done global_id 44387 payload 824 2023-10-30T12:03:41.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 1 -- 172.21.15.50:0/1656084569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fceb009aa00 con 0x7fceb0099a40 2023-10-30T12:03:41.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 20 client.44384.objecter dump_active .. 0 homeless 2023-10-30T12:03:41.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceaf7fe640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce900392a0 0x7fce9003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceaf7fe640 10 monclient: get_auth_request con 0x7fce900392a0 auth_method 0 2023-10-30T12:03:41.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 10 client.44384.objecter ms_handle_connect 0x7fceb0099a40 2023-10-30T12:03:41.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 10 client.44384.objecter resend_mon_ops 2023-10-30T12:03:41.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fcea67fc640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb0099a40 0x7fceb002b430 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fceb011ffe0 tx=0x7fce9800c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980074e0 con 0x7fceb0099a40 2023-10-30T12:03:41.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:41.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.863+0000 7fceacff9640 10 monclient: dump: 2023-10-30T12:03:41.913 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.913 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.913 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.914 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.914 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.914 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.914 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.914 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.915 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 10 client.44384.objecter ms_handle_connect 0x7fce900392a0 2023-10-30T12:03:41.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceaf7fe640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce900392a0 0x7fce9003b760 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 crypto rx=0x7fcea8007f20 tx=0x7fcea80074b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fce98007640 con 0x7fceb0099a40 2023-10-30T12:03:41.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:41.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:41.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb0099a40 msgr2=0x7fceb002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb0099a40 0x7fceb002b430 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fceb011ffe0 tx=0x7fce9800c7a0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb0099a40 msgr2=0x7fceb002b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:41.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.864+0000 7fceacff9640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb0099a40 0x7fceb002b430 secure :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fceb011ffe0 tx=0x7fce9800c7a0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fceb58f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:41.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fceb58f2640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb011f8d0 0x7fceb002b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:41.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fceb58f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:41.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb009a720 con 0x7fceb011f8d0 2023-10-30T12:03:41.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb011f8d0 0x7fceb002b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:41.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient: get_auth_request con 0x7fceb011f8d0 auth_method 0 2023-10-30T12:03:41.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:41.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:41.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:41.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:41.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:41.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:41.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 10 monclient(hunting): handle_auth_done global_id 44390 payload 824 2023-10-30T12:03:41.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 1 -- 172.21.15.50:0/1656084569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fceb0133040 con 0x7fceb011f8d0 2023-10-30T12:03:41.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 10 client.44384.objecter ms_handle_connect 0x7fceb011f8d0 2023-10-30T12:03:41.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.870+0000 7fcea67fc640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb011f8d0 0x7fceb002b970 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fce9800cfd0 tx=0x7fce98000fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:41.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 10 client.44384.objecter resend_mon_ops 2023-10-30T12:03:41.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9801c460 con 0x7fceb011f8d0 2023-10-30T12:03:41.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:41.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:41.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 10 monclient: dump: 2023-10-30T12:03:41.923 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:41.923 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:41.923 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.923 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:41.923 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:41.923 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:41.924 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:41.924 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:41.924 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:41.924 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:41.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7fce98023070 con 0x7fceb011f8d0 2023-10-30T12:03:41.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:41.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:41.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb011f8d0 msgr2=0x7fceb002b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb011f8d0 0x7fceb002b970 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fce9800cfd0 tx=0x7fce98000fd0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb011f8d0 msgr2=0x7fceb002b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:41.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceacff9640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb011f8d0 0x7fceb002b970 secure :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fce9800cfd0 tx=0x7fce98000fd0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.926 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:41.926 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:41.926 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:03:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 32, 2023-10-30T12:03:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:41.927 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:03:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:41.928 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:41.929 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:41.930 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:41.931 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:41.932 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.933 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:41.934 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:41.935 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:41.936 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:41.937 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:41.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:41.937 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:41.937 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:41.937 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:41.938 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:41.938 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:41.938 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:41.938 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:41.938 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:41.938 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:41.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.939 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:41.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.939 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:41.939 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:41.939 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:41.940 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:41.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.940 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:41.940 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:41.940 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:41.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.941 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:41.941 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:41.941 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:41.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:41.942 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:41.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.942 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:41.942 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:41.942 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:41.943 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:41.943 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:41.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.943 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:41.943 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:41.944 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:41.944 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:41.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:41.944 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:41.944 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:41.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.945 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:41.945 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:41.945 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:41.946 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:41.946 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:41.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.946 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:41.946 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:41.947 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:41.947 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:41.947 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:41.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.947 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:41.948 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:41.948 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:41.948 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:41.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.948 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:41.949 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:41.949 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:41.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.949 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:41.950 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:41.950 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:41.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:41.950 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:41.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.950 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:41.951 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:41.951 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:41.951 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:41.951 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:41.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:41.952 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:41.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.952 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:41.952 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:41.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.953 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:41.953 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:41.953 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:41.953 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:41.953 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:41.954 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:03:41.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.954 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:41.954 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:41.955 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:41.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:41.955 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:41.955 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:41.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:41.956 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:41.956 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:41.956 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:03:41.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:41.956 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:41.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:41.957 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:41.957 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 10 librados: watch_flush enter 2023-10-30T12:03:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 10 librados: watch_flush exit 2023-10-30T12:03:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 20 client.44384.objecter shutdown clearing up homeless session... 2023-10-30T12:03:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 10 client.44384.objecter successfully canceled tick 2023-10-30T12:03:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce900392a0 msgr2=0x7fce9003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce900392a0 0x7fce9003b760 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 crypto rx=0x7fcea8007f20 tx=0x7fcea80074b0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 10 monclient: shutdown 2023-10-30T12:03:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 msgr2=0x7fceb01ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 --2- 172.21.15.50:0/1656084569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceb00a8010 0x7fceb01ca630 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fcea0000c00 tx=0x7fcea00046c0 comp rx=0 tx=0).stop 2023-10-30T12:03:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceaf7fe640 1 -- 172.21.15.50:0/1656084569 reap_dead start 2023-10-30T12:03:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceaf7fe640 1 -- 172.21.15.50:0/1656084569 reap_dead start 2023-10-30T12:03:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 shutdown_connections 2023-10-30T12:03:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 >> 172.21.15.50:0/1656084569 conn(0x7fceb00a4a10 msgr2=0x7fceb006a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 shutdown_connections 2023-10-30T12:03:41.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 -- 172.21.15.50:0/1656084569 wait complete. 2023-10-30T12:03:41.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:41.871+0000 7fceb58f2640 1 librados: shutdown 2023-10-30T12:03:41.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:03:41.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:03:41.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:03:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.067+0000 7f1cf1139640 10 monclient: get_monmap_and_config 2023-10-30T12:03:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.067+0000 7f1cf1139640 10 monclient: build_initial_monmap 2023-10-30T12:03:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.067+0000 7f1cf1139640 10 monclient: monmap: 2023-10-30T12:03:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:03:42.068924+0000 2023-10-30T12:03:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:03:42.068924+0000 2023-10-30T12:03:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:03:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:03:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:03:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:03:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:03:42.073 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 1 Processor -- start 2023-10-30T12:03:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 1 -- start start 2023-10-30T12:03:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient: init 2023-10-30T12:03:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:03:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1cec0a8010 0x7f1cec132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec0a8010 addr v2:172.21.15.50:3300/0 2023-10-30T12:03:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1cec099a40 0x7f1cec132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:03:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cec1356d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:03:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0a7190 con 0x7f1cec1356d0 2023-10-30T12:03:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec133110 con 0x7f1cec0a8010 2023-10-30T12:03:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec0a83f0 con 0x7f1cec099a40 2023-10-30T12:03:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.068+0000 7f1cf1139640 10 monclient(hunting): authenticate will time out at 1067.252004s 2023-10-30T12:03:42.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1cead9d640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f1cec0a8010 msgr2=0x7f1cec132690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ceb59e640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f1cec1356d0 legacy=0x7f1cec135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:03:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1cead9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1cec0a8010 0x7f1cec132690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1cec099a40 0x7f1cec132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1cec099a40 0x7f1cec132bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:52178/0 (socket says 172.21.15.50:52178) 2023-10-30T12:03:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 -- 172.21.15.50:0/3272786396 learned_addr learned my addr 172.21.15.50:0/3272786396 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): get_auth_request con 0x7f1cec099a40 auth_method 0 2023-10-30T12:03:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient(hunting): handle_auth_done global_id 28024 payload 274 2023-10-30T12:03:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 -- 172.21.15.50:0/3272786396 >> v2:172.21.15.50:3300/0 conn(0x7f1cec0a8010 msgr2=0x7f1cec132690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:42.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 --2- 172.21.15.50:0/3272786396 >> v2:172.21.15.50:3300/0 conn(0x7f1cec0a8010 0x7f1cec132690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 -- 172.21.15.50:0/3272786396 >> v1:172.21.15.50:6789/0 conn(0x7f1cec1356d0 legacy=0x7f1cec135b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:03:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient: _finish_hunting 0 2023-10-30T12:03:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 monclient: found mon.noname-c 2023-10-30T12:03:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:03:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.069+0000 7f1ce259c640 1 -- 172.21.15.50:0/3272786396 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0a7600 con 0x7f1cec099a40 2023-10-30T12:03:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1ce259c640 1 --2- 172.21.15.50:0/3272786396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec132bd0 secure :-1 s=READY pgs=7045 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0a18a0 tx=0x7f1cd402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3d4144101ade370 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 1 -- 172.21.15.50:0/3272786396 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4031770 con 0x7f1cec099a40 2023-10-30T12:03:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: dump: 2023-10-30T12:03:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:42.091 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:42.091 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: _finish_auth 0 2023-10-30T12:03:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: _check_auth_tickets 2023-10-30T12:03:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 1 -- 172.21.15.50:0/3272786396 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd40318d0 con 0x7f1cec099a40 2023-10-30T12:03:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 5 monclient: authenticate success, global_id 28024 2023-10-30T12:03:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:03:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 10 monclient: get_monmap_and_config success 2023-10-30T12:03:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 4 set_mon_vals no callback set 2023-10-30T12:03:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 1 -- 172.21.15.50:0/3272786396 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4031c60 con 0x7f1cec099a40 2023-10-30T12:03:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cea59c640 10 monclient: dump: 2023-10-30T12:03:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:42.096 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 10 monclient: shutdown 2023-10-30T12:03:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 1 -- 172.21.15.50:0/3272786396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 msgr2=0x7f1cec132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.070+0000 7f1cf1139640 1 --2- 172.21.15.50:0/3272786396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec132bd0 secure :-1 s=READY pgs=7045 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0a18a0 tx=0x7f1cd402b810 comp rx=0 tx=0).stop 2023-10-30T12:03:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 -- 172.21.15.50:0/3272786396 shutdown_connections 2023-10-30T12:03:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 --2- 172.21.15.50:0/3272786396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec132bd0 unknown :-1 s=CLOSED pgs=7045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 --2- 172.21.15.50:0/3272786396 >> v2:172.21.15.50:3300/0 conn(0x7f1cec0a8010 0x7f1cec132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 -- 172.21.15.50:0/3272786396 >> 172.21.15.50:0/3272786396 conn(0x7f1cec0a4a10 msgr2=0x7f1cec0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 -- 172.21.15.50:0/3272786396 shutdown_connections 2023-10-30T12:03:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 -- 172.21.15.50:0/3272786396 wait complete. 2023-10-30T12:03:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 10 monclient: build_initial_monmap 2023-10-30T12:03:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 10 monclient: monmap: 2023-10-30T12:03:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:03:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:03:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:03:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:03:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:03:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:03:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:03:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:03:42.100 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 librados: starting msgr at 2023-10-30T12:03:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.071+0000 7f1cf1139640 1 librados: starting objecter 2023-10-30T12:03:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 Processor -- start 2023-10-30T12:03:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 -- start start 2023-10-30T12:03:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 librados: setting wanted keys 2023-10-30T12:03:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 librados: calling monclient init 2023-10-30T12:03:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient: init 2023-10-30T12:03:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient: _reopen_session rank -1 2023-10-30T12:03:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:03:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0a8010 0x7f1cec1ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec0a8010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:03:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec1356d0 0x7f1cec1cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec1356d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:03:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cec11e5c0 0x7f1cec11ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec11e5c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:03:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec122410 con 0x7f1cec0a8010 2023-10-30T12:03:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec121d50 con 0x7f1cec1356d0 2023-10-30T12:03:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0a18d0 con 0x7f1cec11e5c0 2023-10-30T12:03:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): _renew_subs 2023-10-30T12:03:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cf1139640 10 monclient(hunting): authenticate will time out at 1067.255744s 2023-10-30T12:03:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cead9d640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0a8010 msgr2=0x7f1cec1ca800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:03:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1cead9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0a8010 0x7f1cec1ca800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:03:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cec11e5c0 0x7f1cec11ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cec11e5c0 0x7f1cec11ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57270/0 (socket says 172.21.15.50:57270) 2023-10-30T12:03:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 1 -- 172.21.15.50:0/4060938338 learned_addr learned my addr 172.21.15.50:0/4060938338 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:03:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 10 monclient(hunting): get_auth_request con 0x7f1cec11e5c0 auth_method 0 2023-10-30T12:03:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.072+0000 7f1ceb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce259c640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec1356d0 0x7f1cec1cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce259c640 10 monclient(hunting): get_auth_request con 0x7f1cec1356d0 auth_method 0 2023-10-30T12:03:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:42.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 10 monclient(hunting): handle_auth_done global_id 44393 payload 824 2023-10-30T12:03:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec1356d0 msgr2=0x7f1cec1cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec1356d0 0x7f1cec1cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0a8010 msgr2=0x7f1cec1ca800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:03:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cec0a8010 0x7f1cec1ca800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 10 monclient: _finish_hunting 0 2023-10-30T12:03:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 1 monclient: found mon.noname-c 2023-10-30T12:03:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:03:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:03:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 1 -- 172.21.15.50:0/4060938338 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40094d0 con 0x7f1cec11e5c0 2023-10-30T12:03:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ceb59e640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cec11e5c0 0x7f1cec11ea70 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400be90 tx=0x7f1ce400bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 client.?.objecter ms_handle_connect 0x7f1cec11e5c0 2023-10-30T12:03:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:03:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4016430 con 0x7f1cec11e5c0 2023-10-30T12:03:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:03:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: dump: 2023-10-30T12:03:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:42.112 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: _finish_auth 0 2023-10-30T12:03:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:03:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:03:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 5 monclient: authenticate success, global_id 44393 2023-10-30T12:03:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 10 monclient: _renew_subs 2023-10-30T12:03:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec120ec0 con 0x7f1cec11e5c0 2023-10-30T12:03:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce4016590 con 0x7f1cec11e5c0 2023-10-30T12:03:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:03:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4016940 con 0x7f1cec11e5c0 2023-10-30T12:03:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:03:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce3fff640 10 monclient: dump: 2023-10-30T12:03:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:42.115 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:42.116 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:42.117 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1ce9d9b640 4 set_mon_vals no callback set 2023-10-30T12:03:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 10 client.44393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:03:42.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 10 monclient: _renew_subs 2023-10-30T12:03:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:03:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec120220 con 0x7f1cec11e5c0 2023-10-30T12:03:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.073+0000 7f1cf1139640 1 librados: init done 2023-10-30T12:03:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cf1139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cf1139640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cf1139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec073fd0 con 0x7f1cec099a40 2023-10-30T12:03:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1ce4012020 con 0x7f1cec11e5c0 2023-10-30T12:03:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient: get_auth_request con 0x7f1cec099a40 auth_method 0 2023-10-30T12:03:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce3fff640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ccc0392d0 0x7f1ccc03b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:42.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:42.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:42.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8302+0+0 (secure 0 0 0) 0x7f1ce404b5d0 con 0x7f1cec11e5c0 2023-10-30T12:03:42.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce3fff640 10 client.44393.objecter ms_dispatch 0x7f1cec000b90 osd_map(83..83 src has 1..83) v4 2023-10-30T12:03:42.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce3fff640 3 client.44393.objecter handle_osd_map got epochs [83,83] > 0 2023-10-30T12:03:42.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce3fff640 3 client.44393.objecter handle_osd_map decoding full epoch 83 2023-10-30T12:03:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 10 monclient(hunting): handle_auth_done global_id 28030 payload 824 2023-10-30T12:03:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce259c640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ccc0392d0 0x7f1ccc03b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce3fff640 20 client.44393.objecter dump_active .. 0 homeless 2023-10-30T12:03:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1cead9d640 1 -- 172.21.15.50:0/4060938338 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cec09a720 con 0x7f1cec099a40 2023-10-30T12:03:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.074+0000 7f1ce259c640 10 monclient: get_auth_request con 0x7f1ccc0392d0 auth_method 0 2023-10-30T12:03:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 client.44393.objecter ms_handle_connect 0x7f1cec099a40 2023-10-30T12:03:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1cead9d640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec02b430 secure :-1 s=READY pgs=7046 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc003980 tx=0x7f1cdc00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 client.44393.objecter resend_mon_ops 2023-10-30T12:03:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc003e10 con 0x7f1cec099a40 2023-10-30T12:03:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 monclient: dump: 2023-10-30T12:03:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:42.125 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 client.44393.objecter ms_handle_connect 0x7f1ccc0392d0 2023-10-30T12:03:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce259c640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ccc0392d0 0x7f1ccc03b790 secure :-1 s=READY pgs=3882 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4036000 tx=0x7f1cd403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1cdc005070 con 0x7f1cec099a40 2023-10-30T12:03:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:03:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:03:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 msgr2=0x7f1cec02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec02b430 secure :-1 s=READY pgs=7046 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc003980 tx=0x7f1cdc00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 msgr2=0x7f1cec02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.075+0000 7f1ce3fff640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec099a40 0x7f1cec02b430 secure :-1 s=CLOSED pgs=7046 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc003980 tx=0x7f1cdc00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.081+0000 7f1cf1139640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:03:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.081+0000 7f1cf1139640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec11f770 0x7f1cec02b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:03:42.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.081+0000 7f1cf1139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:03:42.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.081+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec09a500 con 0x7f1cec11f770 2023-10-30T12:03:42.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec11f770 0x7f1cec02b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:03:42.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient: get_auth_request con 0x7f1cec11f770 auth_method 0 2023-10-30T12:03:42.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:03:42.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:03:42.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:03:42.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:03:42.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:03:42.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:03:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 10 monclient(hunting): handle_auth_done global_id 28033 payload 824 2023-10-30T12:03:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 1 -- 172.21.15.50:0/4060938338 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1cec09aa00 con 0x7f1cec11f770 2023-10-30T12:03:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1cead9d640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec11f770 0x7f1cec02b970 secure :-1 s=READY pgs=7047 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc000c00 tx=0x7f1cdc005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:03:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 10 client.44393.objecter ms_handle_connect 0x7f1cec11f770 2023-10-30T12:03:42.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 10 client.44393.objecter resend_mon_ops 2023-10-30T12:03:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc020070 con 0x7f1cec11f770 2023-10-30T12:03:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:03:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:03:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 10 monclient: dump: 2023-10-30T12:03:42.133 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:03:42.134 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:03:42.134 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.134 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:03:42.134 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:03:42.134 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:03:42.135 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:03:42.135 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:03:42.135 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:03:42.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:03:42.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4793 (secure 0 0 0) 0x7f1cdc01bcb0 con 0x7f1cec11f770 2023-10-30T12:03:42.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:03:42.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:03:42.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec11f770 msgr2=0x7f1cec02b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:42.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.082+0000 7f1ce3fff640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec11f770 0x7f1cec02b970 secure :-1 s=READY pgs=7047 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc000c00 tx=0x7f1cdc005a20 comp rx=0 tx=0).stop 2023-10-30T12:03:42.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1ce3fff640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec11f770 msgr2=0x7f1cec02b970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:03:42.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1ce3fff640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cec11f770 0x7f1cec02b970 unknown :-1 s=CLOSED pgs=7047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:03:42.137 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:03:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:42.137 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:03:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 32, 2023-10-30T12:03:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:03:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:03:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:03:42.138 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:03:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:03:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:03:42.139 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:42.140 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:42.141 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:03:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:03:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:42.142 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:42.143 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:03:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:03:42.144 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:03:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:03:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:03:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:03:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:42.145 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:03:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:03:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:03:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:03:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:03:42.146 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:03:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:03:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:03:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:03:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:03:42.147 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:03:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:03:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:03:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:03:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:03:42.148 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:03:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:03:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:03:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:03:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:03:42.149 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:03:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:03:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:03:42.150 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:03:42.151 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:42.151 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:42.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.151 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:42.151 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:03:42.152 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:42.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.152 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:42.152 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:03:42.153 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:42.153 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.153 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:42.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.153 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:03:42.154 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:03:42.154 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:42.154 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:42.154 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:42.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.155 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:03:42.155 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:03:42.155 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:42.155 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:42.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.156 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:42.156 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:03:42.156 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:42.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.157 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:42.157 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:03:42.157 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:42.157 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.158 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:42.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.158 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:03:42.158 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:03:42.158 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:42.159 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:42.159 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:42.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.159 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:03:42.160 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:03:42.160 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:03:42.160 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:03:42.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.160 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:03:42.160 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:03:42.161 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:42.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.161 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:03:42.161 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:03:42.162 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:03:42.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.162 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:42.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.162 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:03:42.163 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:03:42.163 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:03:42.163 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:03:42.163 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:03:42.163 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.164 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:42.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.164 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:03:42.164 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:03:42.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.164 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:42.165 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:42.165 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:42.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.165 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:42.165 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:03:42.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.166 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:42.166 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:42.166 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:03:42.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.167 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:42.167 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:03:42.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.167 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:42.167 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:42.167 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 11 2023-10-30T12:03:42.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.168 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:03:42.168 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:03:42.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:03:42.168 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:03:42.169 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:03:42.169 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:03:42.169 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:03:42.169 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:03:42.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:03:42.170 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:03:42.170 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:03:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 10 librados: watch_flush enter 2023-10-30T12:03:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 10 librados: watch_flush exit 2023-10-30T12:03:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 20 client.44393.objecter shutdown clearing up homeless session... 2023-10-30T12:03:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 10 client.44393.objecter successfully canceled tick 2023-10-30T12:03:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ccc0392d0 msgr2=0x7f1ccc03b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ccc0392d0 0x7f1ccc03b790 secure :-1 s=READY pgs=3882 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4036000 tx=0x7f1cd403d040 comp rx=0 tx=0).stop 2023-10-30T12:03:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 10 monclient: shutdown 2023-10-30T12:03:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:03:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cec11e5c0 msgr2=0x7f1cec11ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:03:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 --2- 172.21.15.50:0/4060938338 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cec11e5c0 0x7f1cec11ea70 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400be90 tx=0x7f1ce400bf80 comp rx=0 tx=0).stop 2023-10-30T12:03:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1ceb59e640 1 -- 172.21.15.50:0/4060938338 reap_dead start 2023-10-30T12:03:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1ceb59e640 1 -- 172.21.15.50:0/4060938338 reap_dead start 2023-10-30T12:03:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 shutdown_connections 2023-10-30T12:03:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 >> 172.21.15.50:0/4060938338 conn(0x7f1cec0a4a10 msgr2=0x7f1cec1338c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:03:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 shutdown_connections 2023-10-30T12:03:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 -- 172.21.15.50:0/4060938338 wait complete. 2023-10-30T12:03:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:03:42.083+0000 7f1cf1139640 1 librados: shutdown 2023-10-30T12:03:42.175 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:03:42.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:03:45.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 10 2023-10-30T12:03:45.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: ioctx.create("asdffoo", true) 2023-10-30T12:03:45.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: Which is: 0 2023-10-30T12:03:45.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.AssertExistsPP (4 ms) 2023-10-30T12:03:45.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-10-30T12:03:45.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (11 ms) 2023-10-30T12:03:45.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-10-30T12:03:45.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: osd_max_attr_size = 0 2023-10-30T12:03:45.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-10-30T12:03:45.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1588 ms) 2023-10-30T12:03:45.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-10-30T12:03:45.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (251 ms) 2023-10-30T12:03:45.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-10-30T12:03:45.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-10-30T12:03:45.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: done waiting, doing copies 2023-10-30T12:03:45.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: waiting for final deep scrubs... 2023-10-30T12:03:45.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: done waiting 2023-10-30T12:03:45.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60350 ms) 2023-10-30T12:03:45.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-10-30T12:03:45.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2023-10-30T12:03:45.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-10-30T12:03:45.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (2 ms) 2023-10-30T12:03:45.124 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-10-30T12:03:45.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4483 ms) 2023-10-30T12:03:45.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-10-30T12:03:45.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-10-30T12:03:45.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-10-30T12:03:45.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-10-30T12:03:45.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-10-30T12:03:45.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-10-30T12:03:45.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (66934 ms total) 2023-10-30T12:03:45.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:03:45.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-10-30T12:03:45.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-10-30T12:03:45.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (93 ms) 2023-10-30T12:03:45.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (93 ms total) 2023-10-30T12:03:45.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:03:45.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-10-30T12:03:45.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-10-30T12:03:45.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (39 ms) 2023-10-30T12:03:45.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-10-30T12:03:45.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2023-10-30T12:03:45.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (43 ms total) 2023-10-30T12:03:45.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:03:45.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-10-30T12:03:45.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-10-30T12:03:45.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (41 ms) 2023-10-30T12:03:45.130 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-10-30T12:03:45.130 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (8 ms) 2023-10-30T12:03:45.130 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (49 ms total) 2023-10-30T12:03:45.130 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:03:45.130 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-10-30T12:03:45.131 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-10-30T12:03:45.131 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (42 ms) 2023-10-30T12:03:45.131 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-10-30T12:03:45.131 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (8 ms) 2023-10-30T12:03:52.125 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (50 ms total) 2023-10-30T12:03:52.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:03:52.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-10-30T12:03:52.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-10-30T12:03:52.126 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1030 ms) 2023-10-30T12:03:52.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1030 ms total) 2023-10-30T12:03:52.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:03:52.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-10-30T12:03:52.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (117811 ms total) 2023-10-30T12:03:52.127 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ PASSED ] 27 tests. 2023-10-30T12:03:52.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] 4 tests, listed below: 2023-10-30T12:03:52.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.LongNamePP 2023-10-30T12:03:52.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.LongAttrNamePP 2023-10-30T12:03:52.128 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.AioOperatePP 2023-10-30T12:03:52.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: [ FAILED ] LibRadosMiscPP.AssertExistsPP 2023-10-30T12:03:52.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 2023-10-30T12:03:52.129 INFO:tasks.workunit.client.0.smithi118.stdout: api_misc_pp: 4 FAILED TESTS 2023-10-30T12:03:56.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-10-30T12:03:56.114 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-10-30T12:03:56.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-10-30T12:03:56.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-10-30T12:03:56.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: seed 87580 2023-10-30T12:03:56.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-10-30T12:03:56.115 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (1980 ms) 2023-10-30T12:03:56.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-10-30T12:03:56.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2016 ms) 2023-10-30T12:03:56.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-10-30T12:03:56.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1990 ms) 2023-10-30T12:03:56.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-10-30T12:03:56.116 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2004 ms) 2023-10-30T12:03:56.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-10-30T12:03:56.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3933 ms) 2023-10-30T12:03:56.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11923 ms total) 2023-10-30T12:03:56.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: 2023-10-30T12:03:56.117 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-10-30T12:03:56.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-10-30T12:03:56.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4014 ms) 2023-10-30T12:03:56.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-10-30T12:03:56.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (16529 ms) 2023-10-30T12:03:56.118 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-10-30T12:03:56.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (19451 ms) 2023-10-30T12:03:56.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-10-30T12:03:56.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4015 ms) 2023-10-30T12:03:56.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-10-30T12:03:56.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2011 ms) 2023-10-30T12:03:56.119 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T12:03:56.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0.0-6851-g003372b1/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-10-30T12:03:56.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: 2023-10-30T12:03:56.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-10-30T12:03:56.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-10-30T12:03:56.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi118-87580-7 2023-10-30T12:03:56.120 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: waiting for snaps to purge 2023-10-30T12:03:56.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17984 ms) 2023-10-30T12:03:56.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (64004 ms total) 2023-10-30T12:03:56.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: 2023-10-30T12:03:56.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-10-30T12:03:56.121 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-10-30T12:03:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3006 ms) 2023-10-30T12:03:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-10-30T12:03:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (1629 ms) 2023-10-30T12:03:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-10-30T12:03:56.122 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2005 ms) 2023-10-30T12:03:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-10-30T12:03:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1995 ms) 2023-10-30T12:03:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8635 ms total) 2023-10-30T12:03:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: 2023-10-30T12:03:56.123 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-10-30T12:04:00.154 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] Li api_aio: Running main() from gmock_main.cc 2023-10-30T12:04:00.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-10-30T12:04:00.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [----------] Global test environment set-up. 2023-10-30T12:04:00.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-10-30T12:04:00.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-10-30T12:04:00.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5996 ms) 2023-10-30T12:04:00.155 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-10-30T12:04:00.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (2998 ms) 2023-10-30T12:04:00.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-10-30T12:04:00.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2986 ms) 2023-10-30T12:04:00.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-10-30T12:04:00.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3000 ms) 2023-10-30T12:04:00.156 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-10-30T12:04:00.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2932 ms) 2023-10-30T12:04:00.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-10-30T12:04:00.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3013 ms) 2023-10-30T12:04:00.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-10-30T12:04:00.157 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3021 ms) 2023-10-30T12:04:00.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-10-30T12:04:00.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (15611 ms) 2023-10-30T12:04:00.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-10-30T12:04:00.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (16354 ms) 2023-10-30T12:04:00.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-10-30T12:04:00.158 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3007 ms) 2023-10-30T12:04:00.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-10-30T12:04:00.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3005 ms) 2023-10-30T12:04:00.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-10-30T12:04:00.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3033 ms) 2023-10-30T12:04:00.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-10-30T12:04:00.159 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2973 ms) 2023-10-30T12:04:00.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-10-30T12:04:00.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2987 ms) 2023-10-30T12:04:00.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-10-30T12:04:00.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.Flush (3008 ms) 2023-10-30T12:04:00.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-10-30T12:04:00.160 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2997 ms) 2023-10-30T12:04:00.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-10-30T12:04:00.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3004 ms) 2023-10-30T12:04:00.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-10-30T12:04:00.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (7819 ms) 2023-10-30T12:04:00.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-10-30T12:04:00.161 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3429 ms) 2023-10-30T12:04:00.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T12:04:00.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (17120 ms) 2023-10-30T12:04:00.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-10-30T12:04:00.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2629 ms) 2023-10-30T12:04:00.162 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-10-30T12:04:00.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3006 ms) 2023-10-30T12:04:00.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-10-30T12:04:00.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2997 ms) 2023-10-30T12:04:00.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-10-30T12:04:00.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3007 ms) 2023-10-30T12:04:00.163 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-10-30T12:04:00.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2995 ms) 2023-10-30T12:04:00.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-10-30T12:04:00.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3038 ms) 2023-10-30T12:04:00.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [----------] 26 tests from LibRadosAio (125966 ms total) 2023-10-30T12:04:00.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: 2023-10-30T12:04:00.164 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-10-30T12:04:02.177 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:04:02.177 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:04:02.177 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:04:02.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:04:02.180 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:04:02.181 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:04:02.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:04:02.225 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWriry=15 expected=15 2023-10-30T12:04:02.225 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:6cac518f:::0:head 2023-10-30T12:04:02.225 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:6cac518f:::0:head expected=24:6cac518f:::0:head 2023-10-30T12:04:02.226 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:6cac518f:::0:head -> 0 2023-10-30T12:04:02.226 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=0 expected=0 2023-10-30T12:04:02.226 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:62a1935d:::14:head 2023-10-30T12:04:02.226 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:62a1935d:::14:head expected=24:62a1935d:::14:head 2023-10-30T12:04:02.226 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:62a1935d:::14:head -> 14 2023-10-30T12:04:02.226 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=14 expected=14 2023-10-30T12:04:02.227 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:5c6b0b28:::7:head 2023-10-30T12:04:02.227 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:5c6b0b28:::7:head expected=24:5c6b0b28:::7:head 2023-10-30T12:04:02.227 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:5c6b0b28:::7:head -> 7 2023-10-30T12:04:02.227 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=7 expected=7 2023-10-30T12:04:02.227 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:566253c9:::13:head 2023-10-30T12:04:02.228 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:566253c9:::13:head expected=24:566253c9:::13:head 2023-10-30T12:04:02.228 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:566253c9:::13:head -> 13 2023-10-30T12:04:02.228 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=13 expected=13 2023-10-30T12:04:02.228 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:de5d7c5f:::12:head 2023-10-30T12:04:02.228 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:de5d7c5f:::12:head expected=24:de5d7c5f:::12:head 2023-10-30T12:04:02.228 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:de5d7c5f:::12:head -> 12 2023-10-30T12:04:02.229 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=12 expected=12 2023-10-30T12:04:02.229 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:d83876eb:::4:head 2023-10-30T12:04:02.229 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:d83876eb:::4:head expected=24:d83876eb:::4:head 2023-10-30T12:04:02.229 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:d83876eb:::4:head -> 4 2023-10-30T12:04:02.229 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=4 expected=4 2023-10-30T12:04:02.229 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:cfc208b3:::3:head 2023-10-30T12:04:02.230 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:cfc208b3:::3:head expected=24:cfc208b3:::3:head 2023-10-30T12:04:02.230 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:cfc208b3:::3:head -> 3 2023-10-30T12:04:02.230 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=3 expected=3 2023-10-30T12:04:02.230 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : seek to 24:02547ec2:::1:head 2023-10-30T12:04:02.230 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : cursor()=24:02547ec2:::1:head expected=24:02547ec2:::1:head 2023-10-30T12:04:02.231 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: > 24:02547ec2:::1:head -> 1 2023-10-30T12:04:02.231 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: : entry=1 expected=1 2023-10-30T12:04:02.231 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (210 ms) 2023-10-30T12:04:02.231 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-10-30T12:04:02.231 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (71376 ms) 2023-10-30T12:04:02.231 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-10-30T12:04:02.232 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: split 0/5 -> MIN 24:33333333::::head 2023-10-30T12:04:02.232 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: split 1/5 -> 24:33333333::::head 24:66666666::::head 2023-10-30T12:04:02.232 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: split 2/5 -> 24:66666666::::head 24:99999999::::head 2023-10-30T12:04:02.232 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: split 3/5 -> 24:99999999::::head 24:cccccccc::::head 2023-10-30T12:04:02.232 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: split 4/5 -> 24:cccccccc::::head MAX 2023-10-30T12:04:02.232 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (46023 ms) 2023-10-30T12:04:02.233 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] 7 tests from LibRadosList (118027 ms total) 2023-10-30T12:04:02.233 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 2023-10-30T12:04:02.233 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-10-30T12:04:02.233 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-10-30T12:04:02.233 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1048 ms) 2023-10-30T12:04:02.233 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-10-30T12:04:02.234 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset foo1,foo2,foo3 2023-10-30T12:04:02.234 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo1 2023-10-30T12:04:02.234 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo2 2023-10-30T12:04:02.234 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo3 2023-10-30T12:04:02.234 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset foo1,foo4,foo5 2023-10-30T12:04:02.234 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo4 2023-10-30T12:04:02.235 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo5 2023-10-30T12:04:02.235 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo1 2023-10-30T12:04:02.235 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset foo6,foo7 2023-10-30T12:04:02.235 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo7 2023-10-30T12:04:02.235 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: foo6 2023-10-30T12:04:02.236 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-10-30T12:04:02.236 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns1:foo4 2023-10-30T12:04:02.236 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns1:foo5 2023-10-30T12:04:02.236 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns2:foo7 2023-10-30T12:04:02.236 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns2:foo6 2023-10-30T12:04:02.236 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: ns1:foo1 2023-10-30T12:04:02.237 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: :foo1 2023-10-30T12:04:02.237 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: :foo2 2023-10-30T12:04:02.237 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: :foo3 2023-10-30T12:04:02.237 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (43 ms) 2023-10-30T12:04:02.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.308+0000 7f0876c26640 10 monclient: get_monmap_and_config 2023-10-30T12:04:02.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.308+0000 7f0876c26640 10 monclient: build_initial_monmap 2023-10-30T12:04:02.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.308+0000 7f0876c26640 10 monclient: monmap: 2023-10-30T12:04:02.310 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:02.309647+0000 2023-10-30T12:04:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:02.309647+0000 2023-10-30T12:04:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:02.312 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:02.312 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:02.312 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:02.312 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:02.312 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:02.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 1 Processor -- start 2023-10-30T12:04:02.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 1 -- start start 2023-10-30T12:04:02.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient: init 2023-10-30T12:04:02.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:02.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-10-30T12:04:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08700a6070 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08700a57e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f087013e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient(hunting): picked mon.noname-b con 0x7f087013bfd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087009a270 con 0x7f08700a6070 2023-10-30T12:04:02.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0870099f90 con 0x7f08700a57e0 2023-10-30T12:04:02.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:02.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087009a550 con 0x7f087013bfd0 2023-10-30T12:04:02.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:02.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f0876c26640 10 monclient(hunting): authenticate will time out at 1087.492885s 2023-10-30T12:04:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.309+0000 7f08751c3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f087013e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08749c2640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f08700a6070 0x7f08700a6450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35456/0 (socket says 172.21.15.50:35456) 2023-10-30T12:04:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08749c2640 1 -- 172.21.15.50:0/2600496018 learned_addr learned my addr 172.21.15.50:0/2600496018 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:04:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): get_auth_request con 0x7f087013bfd0 auth_method 0 2023-10-30T12:04:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient(hunting): handle_auth_done global_id 44477 payload 274 2023-10-30T12:04:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 1 -- 172.21.15.50:0/2600496018 >> v1:172.21.15.118:6789/0 conn(0x7f08700a57e0 legacy=0x7f087013a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:02.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 1 -- 172.21.15.50:0/2600496018 >> v1:172.21.15.50:6789/0 conn(0x7f08700a6070 legacy=0x7f08700a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:02.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient: _finish_hunting 0 2023-10-30T12:04:02.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 1 monclient: found mon.noname-b 2023-10-30T12:04:02.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:02.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:02.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.310+0000 7f08751c3640 1 -- 172.21.15.50:0/2600496018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087009a830 con 0x7f087013bfd0 2023-10-30T12:04:02.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f08751c3640 1 --2- 172.21.15.50:0/2600496018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f087013e3c0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f0868009dc0 tx=0x7f086802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a104243729bd4f40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:02.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 1 -- 172.21.15.50:0/2600496018 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086802cbb0 con 0x7f087013bfd0 2023-10-30T12:04:02.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:02.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:02.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: dump: 2023-10-30T12:04:02.327 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:02.327 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:02.327 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.327 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.327 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:02.327 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:02.328 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:02.328 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:02.328 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:02.328 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:02.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:04:02.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:04:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 5 monclient: authenticate success, global_id 44477 2023-10-30T12:04:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 1 -- 172.21.15.50:0/2600496018 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086802cd10 con 0x7f087013bfd0 2023-10-30T12:04:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:02.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 1 -- 172.21.15.50:0/2600496018 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868034660 con 0x7f087013bfd0 2023-10-30T12:04:02.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 4 set_mon_vals no callback set 2023-10-30T12:04:02.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:02.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:02.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f086f7fe640 10 monclient: dump: 2023-10-30T12:04:02.330 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:02.331 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:02.331 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.331 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.331 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:02.331 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:02.332 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 10 monclient: shutdown 2023-10-30T12:04:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 1 -- 172.21.15.50:0/2600496018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 msgr2=0x7f087013e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 1 --2- 172.21.15.50:0/2600496018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f087013e3c0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f0868009dc0 tx=0x7f086802be00 comp rx=0 tx=0).stop 2023-10-30T12:04:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 1 -- 172.21.15.50:0/2600496018 shutdown_connections 2023-10-30T12:04:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 1 --2- 172.21.15.50:0/2600496018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f087013e3c0 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 1 -- 172.21.15.50:0/2600496018 >> 172.21.15.50:0/2600496018 conn(0x7f08700a81d0 msgr2=0x7f08700a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 1 -- 172.21.15.50:0/2600496018 shutdown_connections 2023-10-30T12:04:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.311+0000 7f0876c26640 1 -- 172.21.15.50:0/2600496018 wait complete. 2023-10-30T12:04:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 10 monclient: build_initial_monmap 2023-10-30T12:04:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 10 monclient: monmap: 2023-10-30T12:04:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:02.336 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:02.336 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:02.336 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:02.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 1 librados: starting msgr at 2023-10-30T12:04:02.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 1 librados: starting objecter 2023-10-30T12:04:02.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 1 Processor -- start 2023-10-30T12:04:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 1 -- start start 2023-10-30T12:04:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 1 librados: setting wanted keys 2023-10-30T12:04:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 1 librados: calling monclient init 2023-10-30T12:04:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.312+0000 7f0876c26640 10 monclient: init 2023-10-30T12:04:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:02.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:04:02.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 0x7f08701ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:02.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): picked mon.noname-a con 0x7f08700a57e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:02.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08700a6070 0x7f087013a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:02.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): picked mon.noname-b con 0x7f08700a6070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:02.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f0870121d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:02.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): picked mon.noname-c con 0x7f087013bfd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:02.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:02.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08700a1700 con 0x7f08700a57e0 2023-10-30T12:04:02.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:02.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087013b730 con 0x7f08700a6070 2023-10-30T12:04:02.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08701254d0 con 0x7f087013bfd0 2023-10-30T12:04:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f0876c26640 10 monclient(hunting): authenticate will time out at 1087.496347s 2023-10-30T12:04:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 0x7f08701ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f0870121d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f0870121d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47900/0 (socket says 172.21.15.50:47900) 2023-10-30T12:04:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 0x7f08701ca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45552/0 (socket says 172.21.15.50:45552) 2023-10-30T12:04:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 -- 172.21.15.50:0/85942678 learned_addr learned my addr 172.21.15.50:0/85942678 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): get_auth_request con 0x7f087013bfd0 auth_method 0 2023-10-30T12:04:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 10 monclient(hunting): get_auth_request con 0x7f08700a57e0 auth_method 0 2023-10-30T12:04:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 1 -- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 msgr2=0x7f08701ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:04:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 1 -- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 msgr2=0x7f08701ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:04:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 0x7f08701ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08749c2640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 0x7f08701ca1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f086ffff640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08700a6070 0x7f087013a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f086ffff640 10 monclient(hunting): get_auth_request con 0x7f08700a6070 auth_method 0 2023-10-30T12:04:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f086ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f086ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f086ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient(hunting): handle_auth_done global_id 44480 payload 824 2023-10-30T12:04:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 -- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08700a6070 msgr2=0x7f087013a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08700a6070 0x7f087013a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 -- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 msgr2=0x7f08701ca1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:04:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 0x7f08701ca1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient: _finish_hunting 0 2023-10-30T12:04:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 monclient: found mon.noname-c 2023-10-30T12:04:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 -- 172.21.15.50:0/85942678 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0868009a70 con 0x7f087013bfd0 2023-10-30T12:04:02.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f08751c3640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f0870121d00 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f0868037040 tx=0x7f0868004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:02.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f086d7fa640 10 client.?.objecter ms_handle_connect 0x7f087013bfd0 2023-10-30T12:04:02.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.313+0000 7f086d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:02.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 1 -- 172.21.15.50:0/85942678 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868040650 con 0x7f087013bfd0 2023-10-30T12:04:02.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:02.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: dump: 2023-10-30T12:04:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:02.349 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:02.349 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:02.349 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:02.349 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:02.349 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:02.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:04:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:04:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 1 -- 172.21.15.50:0/85942678 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08680407d0 con 0x7f087013bfd0 2023-10-30T12:04:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 5 monclient: authenticate success, global_id 44480 2023-10-30T12:04:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 1 -- 172.21.15.50:0/85942678 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868040b80 con 0x7f087013bfd0 2023-10-30T12:04:02.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 10 monclient: _renew_subs 2023-10-30T12:04:02.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:02.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08701257b0 con 0x7f087013bfd0 2023-10-30T12:04:02.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:02.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086effd640 4 set_mon_vals no callback set 2023-10-30T12:04:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 10 monclient: dump: 2023-10-30T12:04:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:02.353 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:02.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 10 client.44480.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:02.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 10 monclient: _renew_subs 2023-10-30T12:04:02.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:02.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087013a910 con 0x7f087013bfd0 2023-10-30T12:04:02.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f0876c26640 1 librados: init done 2023-10-30T12:04:02.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.314+0000 7f086d7fa640 1 -- 172.21.15.50:0/85942678 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0868040ce0 con 0x7f087013bfd0 2023-10-30T12:04:02.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f0876c26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:02.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f0876c26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:02.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f0876c26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:02.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087013ab00 con 0x7f087013bfd0 2023-10-30T12:04:02.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f086d7fa640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08540392f0 0x7f085403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:02.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f086d7fa640 1 -- 172.21.15.50:0/85942678 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8407+0+0 (secure 0 0 0) 0x7f086806e4c0 con 0x7f087013bfd0 2023-10-30T12:04:02.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f086d7fa640 10 client.44480.objecter ms_dispatch 0x7f0870000b90 osd_map(104..104 src has 1..104) v4 2023-10-30T12:04:02.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f086d7fa640 3 client.44480.objecter handle_osd_map got epochs [104,104] > 0 2023-10-30T12:04:02.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f086d7fa640 3 client.44480.objecter handle_osd_map decoding full epoch 104 2023-10-30T12:04:02.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f086d7fa640 20 client.44480.objecter dump_active .. 0 homeless 2023-10-30T12:04:02.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f08749c2640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08540392f0 0x7f085403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:02.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.315+0000 7f08749c2640 10 monclient: get_auth_request con 0x7f08540392f0 auth_method 0 2023-10-30T12:04:02.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.316+0000 7f08749c2640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08540392f0 0x7f085403b7b0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 crypto rx=0x7f0864000f60 tx=0x7f0864009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:02.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.316+0000 7f086d7fa640 10 client.44480.objecter ms_handle_connect 0x7f08540392f0 2023-10-30T12:04:02.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.319+0000 7f086d7fa640 1 -- 172.21.15.50:0/85942678 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0868033bd0 con 0x7f087013bfd0 2023-10-30T12:04:02.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.319+0000 7f086d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:02.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.319+0000 7f086d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.563+0000 7f0876c26640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:04:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.563+0000 7f0876c26640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.563+0000 7f0876c26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.563+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f08700a1170 con 0x7f087013bfd0 2023-10-30T12:04:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.563+0000 7f086d7fa640 1 -- 172.21.15.50:0/85942678 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f0868044020 con 0x7f087013bfd0 2023-10-30T12:04:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.563+0000 7f086d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.563+0000 7f086d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:02.566 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:04:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.564+0000 7f0876c26640 10 librados: watch_flush enter 2023-10-30T12:04:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 10 librados: watch_flush exit 2023-10-30T12:04:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 20 client.44480.objecter shutdown clearing up homeless session... 2023-10-30T12:04:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 10 client.44480.objecter successfully canceled tick 2023-10-30T12:04:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08540392f0 msgr2=0x7f085403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08540392f0 0x7f085403b7b0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 crypto rx=0x7f0864000f60 tx=0x7f0864009040 comp rx=0 tx=0).stop 2023-10-30T12:04:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 10 monclient: shutdown 2023-10-30T12:04:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 msgr2=0x7f0870121d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f0870121d00 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f0868037040 tx=0x7f0868004a40 comp rx=0 tx=0).stop 2023-10-30T12:04:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 shutdown_connections 2023-10-30T12:04:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08540392f0 0x7f085403b7b0 unknown :-1 s=CLOSED pgs=3902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f087013bfd0 0x7f0870121d00 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08700a6070 0x7f087013a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 --2- 172.21.15.50:0/85942678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08700a57e0 0x7f08701ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 >> 172.21.15.50:0/85942678 conn(0x7f08700a81d0 msgr2=0x7f087009e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 shutdown_connections 2023-10-30T12:04:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 -- 172.21.15.50:0/85942678 wait complete. 2023-10-30T12:04:02.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:02.565+0000 7f0876c26640 1 librados: shutdown 2023-10-30T12:04:05.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:04:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.693+0000 7f6a02e39640 10 monclient: get_monmap_and_config 2023-10-30T12:04:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.693+0000 7f6a02e39640 10 monclient: build_initial_monmap 2023-10-30T12:04:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.693+0000 7f6a02e39640 10 monclient: monmap: 2023-10-30T12:04:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:05.694131+0000 2023-10-30T12:04:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:05.694131+0000 2023-10-30T12:04:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:05.697 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:05.697 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:05.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.693+0000 7f6a02e39640 1 Processor -- start 2023-10-30T12:04:05.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.693+0000 7f6a02e39640 1 -- start start 2023-10-30T12:04:05.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.693+0000 7f6a02e39640 10 monclient: init 2023-10-30T12:04:05.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:05.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-10-30T12:04:05.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 0x7f69fc1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:05.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient(hunting): picked mon.noname-a con 0x7f69fc0a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:05.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69fc099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:05.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc132be0 0x7f69fc0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:05.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient(hunting): picked mon.noname-b con 0x7f69fc132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:05.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc0a4f40 con 0x7f69fc099a40 2023-10-30T12:04:05.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:05.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69fc0a3d70 con 0x7f69fc0a4b60 2023-10-30T12:04:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0a4150 con 0x7f69fc132be0 2023-10-30T12:04:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a02e39640 10 monclient(hunting): authenticate will time out at 1090.877554s 2023-10-30T12:04:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc132be0 0x7f69fc0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 0x7f69fc1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:05.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc132be0 0x7f69fc0a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47970/0 (socket says 172.21.15.50:47970) 2023-10-30T12:04:05.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 1 -- 172.21.15.50:0/905603000 learned_addr learned my addr 172.21.15.50:0/905603000 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:05.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 0x7f69fc1326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45564/0 (socket says 172.21.15.50:45564) 2023-10-30T12:04:05.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): get_auth_request con 0x7f69fc132be0 auth_method 0 2023-10-30T12:04:05.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f69fbfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f69fc099a40 0x7f69fc0a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35460/0 (socket says 172.21.15.50:35460) 2023-10-30T12:04:05.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:05.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:05.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:05.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 10 monclient(hunting): get_auth_request con 0x7f69fc0a4b60 auth_method 0 2023-10-30T12:04:05.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:05.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:05.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:05.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:05.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 1 -- 172.21.15.50:0/905603000 >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 msgr2=0x7f69fc1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:05.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:05.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 1 -- 172.21.15.50:0/905603000 >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 msgr2=0x7f69fc1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:05.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 1 --2- 172.21.15.50:0/905603000 >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 0x7f69fc1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:05.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:05.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a00bd5640 1 --2- 172.21.15.50:0/905603000 >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 0x7f69fc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:05.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.694+0000 7f6a013d6640 10 monclient(hunting): handle_auth_done global_id 44489 payload 274 2023-10-30T12:04:05.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 1 -- 172.21.15.50:0/905603000 >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 msgr2=0x7f69fc1326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:05.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 1 --2- 172.21.15.50:0/905603000 >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 0x7f69fc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 1 -- 172.21.15.50:0/905603000 >> v1:172.21.15.50:6789/0 conn(0x7f69fc099a40 legacy=0x7f69fc0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:05.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 10 monclient: _finish_hunting 0 2023-10-30T12:04:05.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 1 monclient: found mon.noname-b 2023-10-30T12:04:05.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:05.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:05.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 1 -- 172.21.15.50:0/905603000 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc09a440 con 0x7f69fc132be0 2023-10-30T12:04:05.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a013d6640 1 --2- 172.21.15.50:0/905603000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc132be0 0x7f69fc0a2a60 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009870 tx=0x7f69ec02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6580bd1c383af87 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:05.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 1 -- 172.21.15.50:0/905603000 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0317c0 con 0x7f69fc132be0 2023-10-30T12:04:05.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:05.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:05.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: dump: 2023-10-30T12:04:05.714 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:05.714 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:05.714 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.715 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.715 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:05.715 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:05.715 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:05.715 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:05.716 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:05.716 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:05.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:04:05.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:04:05.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:05.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 1 -- 172.21.15.50:0/905603000 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec031920 con 0x7f69fc132be0 2023-10-30T12:04:05.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:05.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 1 -- 172.21.15.50:0/905603000 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec031cb0 con 0x7f69fc132be0 2023-10-30T12:04:05.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:05.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:05.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f69fb7fe640 10 monclient: dump: 2023-10-30T12:04:05.718 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:05.718 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:05.718 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.718 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.718 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:05.719 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:05.719 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:05.719 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:05.719 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:05.719 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:05.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a02e39640 5 monclient: authenticate success, global_id 44489 2023-10-30T12:04:05.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a02e39640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:05.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.695+0000 7f6a02e39640 4 set_mon_vals no callback set 2023-10-30T12:04:05.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 10 monclient: shutdown 2023-10-30T12:04:05.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:05.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 -- 172.21.15.50:0/905603000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc132be0 msgr2=0x7f69fc0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:05.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 --2- 172.21.15.50:0/905603000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc132be0 0x7f69fc0a2a60 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009870 tx=0x7f69ec02b860 comp rx=0 tx=0).stop 2023-10-30T12:04:05.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 -- 172.21.15.50:0/905603000 shutdown_connections 2023-10-30T12:04:05.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 --2- 172.21.15.50:0/905603000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc132be0 0x7f69fc0a2a60 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 --2- 172.21.15.50:0/905603000 >> v2:172.21.15.50:3300/0 conn(0x7f69fc0a4b60 0x7f69fc1326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 -- 172.21.15.50:0/905603000 >> 172.21.15.50:0/905603000 conn(0x7f69fc0a8060 msgr2=0x7f69fc0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:05.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 -- 172.21.15.50:0/905603000 shutdown_connections 2023-10-30T12:04:05.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.696+0000 7f6a02e39640 1 -- 172.21.15.50:0/905603000 wait complete. 2023-10-30T12:04:05.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 10 monclient: build_initial_monmap 2023-10-30T12:04:05.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 10 monclient: monmap: 2023-10-30T12:04:05.722 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:05.722 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:05.723 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:05.723 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:05.723 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:05.723 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:05.723 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:05.723 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:05.724 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:05.724 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:05.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 1 librados: starting msgr at 2023-10-30T12:04:05.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 1 librados: starting objecter 2023-10-30T12:04:05.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 1 Processor -- start 2023-10-30T12:04:05.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 1 -- start start 2023-10-30T12:04:05.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 1 librados: setting wanted keys 2023-10-30T12:04:05.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 1 librados: calling monclient init 2023-10-30T12:04:05.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.697+0000 7f6a02e39640 10 monclient: init 2023-10-30T12:04:05.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:05.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:04:05.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 0x7f69fc1ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:05.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): picked mon.noname-a con 0x7f69fc0a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:05.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f69fc132be0 0x7f69fc1cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:05.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): picked mon.noname-b con 0x7f69fc132be0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:05.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 0x7f69fc11e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:05.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): picked mon.noname-c con 0x7f69fc11e480 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:05.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:05.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0a18d0 con 0x7f69fc0a4b60 2023-10-30T12:04:05.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:05.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc135bf0 con 0x7f69fc132be0 2023-10-30T12:04:05.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:05.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc121b70 con 0x7f69fc11e480 2023-10-30T12:04:05.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:05.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a02e39640 10 monclient(hunting): authenticate will time out at 1090.881372s 2023-10-30T12:04:05.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 0x7f69fc1ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:05.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 0x7f69fc1ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45572/0 (socket says 172.21.15.50:45572) 2023-10-30T12:04:05.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a013d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 0x7f69fc11e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:05.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 1 -- 172.21.15.50:0/1973672030 learned_addr learned my addr 172.21.15.50:0/1973672030 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:05.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 10 monclient(hunting): get_auth_request con 0x7f69fc0a4b60 auth_method 0 2023-10-30T12:04:05.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:05.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:05.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:05.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a013d6640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 msgr2=0x7f69fc11e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:04:05.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a013d6640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 msgr2=0x7f69fc11e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:04:05.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a013d6640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 0x7f69fc11e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:05.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a013d6640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 0x7f69fc11e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:05.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 msgr2=0x7f69fc1ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:04:05.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 msgr2=0x7f69fc1ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:04:05.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 0x7f69fc1ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:05.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f6a00bd5640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 0x7f69fc1ca830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:05.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f69fc132be0 0x7f69fc1cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:05.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 10 monclient(hunting): get_auth_request con 0x7f69fc132be0 auth_method 0 2023-10-30T12:04:05.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:05.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:05.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:05.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.698+0000 7f69fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 10 monclient(hunting): handle_auth_done global_id 28150 payload 824 2023-10-30T12:04:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 msgr2=0x7f69fc11e8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:04:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 0x7f69fc11e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 msgr2=0x7f69fc1ca830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:04:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 0x7f69fc1ca830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:04:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 1 monclient: found mon.noname-b 2023-10-30T12:04:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:04:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 1 -- 172.21.15.50:0/1973672030 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec0094d0 con 0x7f69fc132be0 2023-10-30T12:04:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69fbfff640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f69fc132be0 0x7f69fc1cad70 secure :-1 s=READY pgs=7091 cs=0 l=1 rev1=1 crypto rx=0x7f69fc070610 tx=0x7f69e400c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 client.?.objecter ms_handle_connect 0x7f69fc132be0 2023-10-30T12:04:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 1 -- 172.21.15.50:0/1973672030 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e40074e0 con 0x7f69fc132be0 2023-10-30T12:04:05.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:05.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:05.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: dump: 2023-10-30T12:04:05.736 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:05.736 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:05.737 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.737 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.737 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:05.737 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:05.737 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:05.737 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:05.738 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:05.738 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:05.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: _finish_auth 0 2023-10-30T12:04:05.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: _check_auth_tickets 2023-10-30T12:04:05.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:05.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 1 -- 172.21.15.50:0/1973672030 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e4007640 con 0x7f69fc132be0 2023-10-30T12:04:05.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:05.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 1 -- 172.21.15.50:0/1973672030 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4004a00 con 0x7f69fc132be0 2023-10-30T12:04:05.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 5 monclient: authenticate success, global_id 28150 2023-10-30T12:04:05.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 10 monclient: _renew_subs 2023-10-30T12:04:05.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:04:05.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69faffd640 4 set_mon_vals no callback set 2023-10-30T12:04:05.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc121e50 con 0x7f69fc132be0 2023-10-30T12:04:05.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:05.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:05.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f69f97fa640 10 monclient: dump: 2023-10-30T12:04:05.741 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:05.741 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:05.741 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.741 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:05.741 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:05.742 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:05.742 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:05.742 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:05.742 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:05.742 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:05.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 10 client.28150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:05.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 10 monclient: _renew_subs 2023-10-30T12:04:05.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:04:05.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc120060 con 0x7f69fc132be0 2023-10-30T12:04:05.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.699+0000 7f6a02e39640 1 librados: init done 2023-10-30T12:04:05.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f69f97fa640 1 -- 172.21.15.50:0/1973672030 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f69e4004b60 con 0x7f69fc132be0 2023-10-30T12:04:05.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f6a02e39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:05.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f6a02e39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:05.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f6a02e39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:04:05.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69fc122180 con 0x7f69fc132be0 2023-10-30T12:04:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f69f97fa640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f69dc0392f0 0x7f69dc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f69f97fa640 1 -- 172.21.15.50:0/1973672030 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 7280+0+0 (secure 0 0 0) 0x7f69e404a9f0 con 0x7f69fc132be0 2023-10-30T12:04:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f69f97fa640 10 client.28150.objecter ms_dispatch 0x7f69fc000b90 osd_map(107..107 src has 1..107) v4 2023-10-30T12:04:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f69f97fa640 3 client.28150.objecter handle_osd_map got epochs [107,107] > 0 2023-10-30T12:04:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f69f97fa640 3 client.28150.objecter handle_osd_map decoding full epoch 107 2023-10-30T12:04:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f69f97fa640 20 client.28150.objecter dump_active .. 0 homeless 2023-10-30T12:04:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f6a00bd5640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f69dc0392f0 0x7f69dc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.700+0000 7f6a00bd5640 10 monclient: get_auth_request con 0x7f69dc0392f0 auth_method 0 2023-10-30T12:04:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.701+0000 7f6a00bd5640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f69dc0392f0 0x7f69dc03b7b0 secure :-1 s=READY pgs=3905 cs=0 l=1 rev1=1 crypto rx=0x7f69f40094c0 tx=0x7f69f400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.701+0000 7f69f97fa640 10 client.28150.objecter ms_handle_connect 0x7f69dc0392f0 2023-10-30T12:04:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.705+0000 7f69f97fa640 1 -- 172.21.15.50:0/1973672030 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f69e40192c0 con 0x7f69fc132be0 2023-10-30T12:04:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.705+0000 7f69f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.705+0000 7f69f97fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:05.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.948+0000 7f6a02e39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:04:05.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.948+0000 7f6a02e39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:05.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.948+0000 7f6a02e39640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:04:05.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.948+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f69fc04e160 con 0x7f69fc132be0 2023-10-30T12:04:05.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.949+0000 7f69f97fa640 1 -- 172.21.15.50:0/1973672030 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f69e401aab0 con 0x7f69fc132be0 2023-10-30T12:04:05.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.949+0000 7f69f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:05.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.949+0000 7f69f97fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:05.952 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:04:05.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 10 librados: watch_flush enter 2023-10-30T12:04:05.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 10 librados: watch_flush exit 2023-10-30T12:04:05.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 20 client.28150.objecter shutdown clearing up homeless session... 2023-10-30T12:04:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 10 client.28150.objecter successfully canceled tick 2023-10-30T12:04:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f69dc0392f0 msgr2=0x7f69dc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f69dc0392f0 0x7f69dc03b7b0 secure :-1 s=READY pgs=3905 cs=0 l=1 rev1=1 crypto rx=0x7f69f40094c0 tx=0x7f69f400b040 comp rx=0 tx=0).stop 2023-10-30T12:04:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 10 monclient: shutdown 2023-10-30T12:04:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:05.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f69fc132be0 msgr2=0x7f69fc1cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:05.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f69fc132be0 0x7f69fc1cad70 secure :-1 s=READY pgs=7091 cs=0 l=1 rev1=1 crypto rx=0x7f69fc070610 tx=0x7f69e400c820 comp rx=0 tx=0).stop 2023-10-30T12:04:05.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 shutdown_connections 2023-10-30T12:04:05.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f69dc0392f0 0x7f69dc03b7b0 unknown :-1 s=CLOSED pgs=3905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f69fc11e480 0x7f69fc11e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f69fc132be0 0x7f69fc1cad70 unknown :-1 s=CLOSED pgs=7091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 --2- 172.21.15.50:0/1973672030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69fc0a4b60 0x7f69fc1ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 >> 172.21.15.50:0/1973672030 conn(0x7f69fc0a8060 msgr2=0x7f69fc134320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 shutdown_connections 2023-10-30T12:04:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 -- 172.21.15.50:0/1973672030 wait complete. 2023-10-30T12:04:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:05.951+0000 7f6a02e39640 1 librados: shutdown 2023-10-30T12:04:08.132 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-10-30T12:04:08.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 1 0 2023-10-30T12:04:08.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 10 0 2023-10-30T12:04:08.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 13 0 2023-10-30T12:04:08.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 7 0 2023-10-30T12:04:08.133 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 14 0 2023-10-30T12:04:08.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 0 0 2023-10-30T12:04:08.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 15 0 2023-10-30T12:04:08.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 11 0 2023-10-30T12:04:08.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 5 0 2023-10-30T12:04:08.134 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 8 0 2023-10-30T12:04:08.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 6 0 2023-10-30T12:04:08.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 3 0 2023-10-30T12:04:08.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 4 0 2023-10-30T12:04:08.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 12 0 2023-10-30T12:04:08.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 9 0 2023-10-30T12:04:08.135 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 2 0 2023-10-30T12:04:08.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-10-30T12:04:08.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (56 ms) 2023-10-30T12:04:08.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] 3 tests from LibRadosListEC (1147 ms total) 2023-10-30T12:04:08.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 2023-10-30T12:04:08.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-10-30T12:04:08.136 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-10-30T12:04:08.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3007 ms) 2023-10-30T12:04:08.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] 1 test from LibRadosListNP (3007 ms total) 2023-10-30T12:04:08.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: 2023-10-30T12:04:08.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [----------] Global test environment tear-down 2023-10-30T12:04:08.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [==========] 11 tests from 3 test suites ran. (133879 ms total) 2023-10-30T12:04:08.137 INFO:tasks.workunit.client.0.smithi118.stdout: api_list: [ PASSED ] 11 tests. 2023-10-30T12:04:08.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:04:09.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.078+0000 7fce15e52640 10 monclient: get_monmap_and_config 2023-10-30T12:04:09.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.078+0000 7fce15e52640 10 monclient: build_initial_monmap 2023-10-30T12:04:09.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.078+0000 7fce15e52640 10 monclient: monmap: 2023-10-30T12:04:09.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:09.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:09.081 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:09.079817+0000 2023-10-30T12:04:09.081 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:09.079817+0000 2023-10-30T12:04:09.081 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:09.081 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:09.082 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:09.082 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:09.082 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:09.082 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:09.082 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:09.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:09.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.079+0000 7fce15e52640 1 Processor -- start 2023-10-30T12:04:09.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.079+0000 7fce15e52640 1 -- start start 2023-10-30T12:04:09.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.079+0000 7fce15e52640 10 monclient: init 2023-10-30T12:04:09.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:09.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T12:04:09.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:09.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient(hunting): picked mon.noname-b con 0x7fce1009f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:09.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:09.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient(hunting): picked mon.noname-c con 0x7fce10096080 addr v2:172.21.15.118:3300/0 2023-10-30T12:04:09.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce100969a0 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:09.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce10098810 con 0x7fce100969a0 2023-10-30T12:04:09.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:09.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1006aef0 con 0x7fce10096080 2023-10-30T12:04:09.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:09.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1009a1a0 con 0x7fce1009f5b0 2023-10-30T12:04:09.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:09.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce15e52640 10 monclient(hunting): authenticate will time out at 1094.263249s 2023-10-30T12:04:09.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:09.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48746/0 (socket says 172.21.15.50:48746) 2023-10-30T12:04:09.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 learned_addr learned my addr 172.21.15.50:0/982858282 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:09.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request con 0x7fce1009f5b0 auth_method 0 2023-10-30T12:04:09.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:09.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:09.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:09.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:04:09.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:09.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:09.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:09.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:09.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fce10096080 auth_method 0 2023-10-30T12:04:09.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:09.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:09.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:09.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:09.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:09.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:09.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.080+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:09.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:09.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request con 0x7fce1009f5b0 auth_method 0 2023-10-30T12:04:09.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:09.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:04:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fce10096080 auth_method 0 2023-10-30T12:04:09.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:09.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:09.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:09.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:09.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:04:09.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:09.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:09.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:09.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.281+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:04:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request con 0x7fce1009f5b0 auth_method 0 2023-10-30T12:04:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fce10096080 auth_method 0 2023-10-30T12:04:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:04:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:04:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:09.682+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:04:10.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request con 0x7fce1009f5b0 auth_method 0 2023-10-30T12:04:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:04:10.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fce10096080 auth_method 0 2023-10-30T12:04:10.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:10.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:10.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:10.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:04:10.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:10.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:10.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:10.484+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:04:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient(hunting): tick 2023-10-30T12:04:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:04:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:04:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:04:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 msgr2=0x7fce10099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce1009f5b0 0x7fce10099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fce10096460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fce10096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 >> v1:172.21.15.118:6789/0 conn(0x7fce100969a0 legacy=0x7fce1009cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient: _add_conns ranks=[0,3] 2023-10-30T12:04:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcdfc003710 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcdfc007440 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce10098af0 con 0x7fcdfc003710 2023-10-30T12:04:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1006aef0 con 0x7fcdfc007440 2023-10-30T12:04:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0d7fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fcdfc007440 auth_method 0 2023-10-30T12:04:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.080+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:12.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:12.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fcdfc007440 auth_method 0 2023-10-30T12:04:12.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:12.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:12.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:12.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:12.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:12.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:12.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.281+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:04:12.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.295+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> v1:172.21.15.50:6789/0 conn(0x7fcdfc003710 legacy=0x7fcdfc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-10-30T12:04:12.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.295+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> v1:172.21.15.50:6789/0 conn(0x7fcdfc003710 legacy=0x7fcdfc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:04:12.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.295+0000 7fce0f7fe640 1 --1- 172.21.15.50:0/982858282 >> v1:172.21.15.50:6789/0 conn(0x7fcdfc003710 0x7fcdfc005b90 :-1 s=OPENED pgs=28 cs=1 l=1).handle_message read tag failed 2023-10-30T12:04:12.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.295+0000 7fce0f7fe640 1 --1- 172.21.15.50:0/982858282 >> v1:172.21.15.50:6789/0 conn(0x7fcdfc003710 0x7fcdfc005b90 :-1 s=OPENED pgs=28 cs=1 l=1).fault on lossy channel, failing 2023-10-30T12:04:12.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.295+0000 7fce0e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.50:6789/0 2023-10-30T12:04:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fcdfc007440 auth_method 0 2023-10-30T12:04:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:04:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:12.682+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:04:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fcdfc007440 auth_method 0 2023-10-30T12:04:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:04:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:13.483+0000 7fce0effd640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:04:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): tick 2023-10-30T12:04:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:04:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:04:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:04:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 msgr2=0x7fcdfc0098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc007440 0x7fcdfc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 >> v1:172.21.15.50:6789/0 conn(0x7fcdfc003710 legacy=0x7fcdfc005b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-10-30T12:04:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc009ff0 0x7fcdfc00a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0ffff640 1 -- 172.21.15.50:0/982858282 reap_dead start 2023-10-30T12:04:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcdfc009ff0 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce100969a0 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fcdfc00a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fce10096080 addr v2:172.21.15.118:3300/0 2023-10-30T12:04:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc00afb0 con 0x7fce100969a0 2023-10-30T12:04:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1006aef0 con 0x7fcdfc009ff0 2023-10-30T12:04:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 1 -- 172.21.15.50:0/982858282 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1009a1a0 con 0x7fce10096080 2023-10-30T12:04:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0d7fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.50:3300/0 conn(0x7fcdfc009ff0 0x7fcdfc00a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request con 0x7fcdfc009ff0 auth_method 0 2023-10-30T12:04:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0ffff640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fcdfc00a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.080+0000 7fce0ffff640 10 monclient(hunting): get_auth_request con 0x7fce10096080 auth_method 0 2023-10-30T12:04:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_done global_id 74103 payload 274 2023-10-30T12:04:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 msgr2=0x7fcdfc00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fcdfc00a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 >> v1:172.21.15.118:6789/0 conn(0x7fce100969a0 legacy=0x7fcdfc00d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:04:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.081+0000 7fce0f7fe640 1 -- 172.21.15.50:0/982858282 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc009a40 con 0x7fcdfc009ff0 2023-10-30T12:04:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdfc009ff0 0x7fcdfc00a3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009930 tx=0x7fce0002c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20956fc2e3637a71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 1 -- 172.21.15.50:0/982858282 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00034670 con 0x7fcdfc009ff0 2023-10-30T12:04:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: dump: 2023-10-30T12:04:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.096 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 1 -- 172.21.15.50:0/982858282 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0000c040 con 0x7fcdfc009ff0 2023-10-30T12:04:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 5 monclient: authenticate success, global_id 74103 2023-10-30T12:04:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 1 -- 172.21.15.50:0/982858282 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce000349d0 con 0x7fcdfc009ff0 2023-10-30T12:04:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 4 set_mon_vals no callback set 2023-10-30T12:04:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce0e7fc640 10 monclient: dump: 2023-10-30T12:04:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.100 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 10 monclient: shutdown 2023-10-30T12:04:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 1 -- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdfc009ff0 msgr2=0x7fcdfc00a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdfc009ff0 0x7fcdfc00a3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009930 tx=0x7fce0002c910 comp rx=0 tx=0).stop 2023-10-30T12:04:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 1 -- 172.21.15.50:0/982858282 shutdown_connections 2023-10-30T12:04:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 1 --2- 172.21.15.50:0/982858282 >> v2:172.21.15.118:3300/0 conn(0x7fce10096080 0x7fcdfc00a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 1 --2- 172.21.15.50:0/982858282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdfc009ff0 0x7fcdfc00a3d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.082+0000 7fce15e52640 1 -- 172.21.15.50:0/982858282 >> 172.21.15.50:0/982858282 conn(0x7fce100a05c0 msgr2=0x7fce1009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.083+0000 7fce15e52640 1 -- 172.21.15.50:0/982858282 shutdown_connections 2023-10-30T12:04:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.083+0000 7fce15e52640 1 -- 172.21.15.50:0/982858282 wait complete. 2023-10-30T12:04:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.083+0000 7fce15e52640 10 monclient: build_initial_monmap 2023-10-30T12:04:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.083+0000 7fce15e52640 10 monclient: monmap: 2023-10-30T12:04:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:15.108 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.083+0000 7fce15e52640 1 librados: starting msgr at 2023-10-30T12:04:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.083+0000 7fce15e52640 1 librados: starting objecter 2023-10-30T12:04:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 Processor -- start 2023-10-30T12:04:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 -- start start 2023-10-30T12:04:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 librados: setting wanted keys 2023-10-30T12:04:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 librados: calling monclient init 2023-10-30T12:04:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient: init 2023-10-30T12:04:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:04:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdfc009ff0 0x7fce10070990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): picked mon.noname-c con 0x7fcdfc009ff0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce10096080 0x7fce101b6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): picked mon.noname-b con 0x7fce10096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100969a0 0x7fce101c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): picked mon.noname-a con 0x7fce100969a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10071520 con 0x7fce100969a0 2023-10-30T12:04:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce101af930 con 0x7fce10096080 2023-10-30T12:04:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce101afc10 con 0x7fcdfc009ff0 2023-10-30T12:04:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce15e52640 10 monclient(hunting): authenticate will time out at 1100.267857s 2023-10-30T12:04:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce0f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdfc009ff0 0x7fce10070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.084+0000 7fce0ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100969a0 0x7fce101c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdfc009ff0 0x7fce10070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48792/0 (socket says 172.21.15.50:48792) 2023-10-30T12:04:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 -- 172.21.15.50:0/3883447494 learned_addr learned my addr 172.21.15.50:0/3883447494 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request con 0x7fcdfc009ff0 auth_method 0 2023-10-30T12:04:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 10 monclient(hunting): get_auth_request con 0x7fce100969a0 auth_method 0 2023-10-30T12:04:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce10096080 0x7fce101b6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 10 monclient(hunting): get_auth_request con 0x7fce10096080 auth_method 0 2023-10-30T12:04:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient(hunting): handle_auth_done global_id 44507 payload 824 2023-10-30T12:04:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 -- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce10096080 msgr2=0x7fce101b6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce10096080 0x7fce101b6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 -- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100969a0 msgr2=0x7fce101c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100969a0 0x7fce101c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 monclient: found mon.noname-c 2023-10-30T12:04:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 -- 172.21.15.50:0/3883447494 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce10098af0 con 0x7fcdfc009ff0 2023-10-30T12:04:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0ffff640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100969a0 0x7fce101c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0effd640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce10096080 0x7fce101b6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0f7fe640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdfc009ff0 0x7fce10070990 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fce00000c00 tx=0x7fce000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 client.?.objecter ms_handle_connect 0x7fcdfc009ff0 2023-10-30T12:04:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 1 -- 172.21.15.50:0/3883447494 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00004020 con 0x7fcdfc009ff0 2023-10-30T12:04:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: dump: 2023-10-30T12:04:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.122 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 1 -- 172.21.15.50:0/3883447494 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00004180 con 0x7fcdfc009ff0 2023-10-30T12:04:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 1 -- 172.21.15.50:0/3883447494 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00003610 con 0x7fcdfc009ff0 2023-10-30T12:04:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 5 monclient: authenticate success, global_id 44507 2023-10-30T12:04:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 10 monclient: _renew_subs 2023-10-30T12:04:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce101afef0 con 0x7fcdfc009ff0 2023-10-30T12:04:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce0dffb640 4 set_mon_vals no callback set 2023-10-30T12:04:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fcdf7fff640 10 monclient: dump: 2023-10-30T12:04:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.126 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 10 client.44507.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 10 monclient: _renew_subs 2023-10-30T12:04:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce101ae9d0 con 0x7fcdfc009ff0 2023-10-30T12:04:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.085+0000 7fce15e52640 1 librados: init done 2023-10-30T12:04:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.086+0000 7fcdf7fff640 1 -- 172.21.15.50:0/3883447494 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fce00003770 con 0x7fcdfc009ff0 2023-10-30T12:04:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.086+0000 7fce15e52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.086+0000 7fce15e52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.086+0000 7fce15e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.086+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce101aebc0 con 0x7fcdfc009ff0 2023-10-30T12:04:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fcdf7fff640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdfc03a7b0 0x7fcdfc03cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fcdf7fff640 1 -- 172.21.15.50:0/3883447494 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6804+0+0 (secure 0 0 0) 0x7fce0002e040 con 0x7fcdfc009ff0 2023-10-30T12:04:15.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fcdf7fff640 10 client.44507.objecter ms_dispatch 0x7fce10000b90 osd_map(111..111 src has 1..111) v4 2023-10-30T12:04:15.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fcdf7fff640 3 client.44507.objecter handle_osd_map got epochs [111,111] > 0 2023-10-30T12:04:15.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fcdf7fff640 3 client.44507.objecter handle_osd_map decoding full epoch 111 2023-10-30T12:04:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fcdf7fff640 20 client.44507.objecter dump_active .. 0 homeless 2023-10-30T12:04:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fce0effd640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdfc03a7b0 0x7fcdfc03cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fce0effd640 10 monclient: get_auth_request con 0x7fcdfc03a7b0 auth_method 0 2023-10-30T12:04:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fcdf7fff640 10 client.44507.objecter ms_handle_connect 0x7fcdfc03a7b0 2023-10-30T12:04:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.087+0000 7fce0effd640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdfc03a7b0 0x7fcdfc03cc70 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 crypto rx=0x7fce101c5c50 tx=0x7fce04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.091+0000 7fcdf7fff640 1 -- 172.21.15.50:0/3883447494 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fce00050440 con 0x7fcdfc009ff0 2023-10-30T12:04:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.091+0000 7fcdf7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.091+0000 7fcdf7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:15.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.343+0000 7fce15e52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:04:15.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.343+0000 7fce15e52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:15.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.343+0000 7fce15e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.343+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fce1004e160 con 0x7fcdfc009ff0 2023-10-30T12:04:15.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.343+0000 7fcdf7fff640 1 -- 172.21.15.50:0/3883447494 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fce00031b90 con 0x7fcdfc009ff0 2023-10-30T12:04:15.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.343+0000 7fcdf7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:15.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.343+0000 7fcdf7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:15.347 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:04:15.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 10 librados: watch_flush enter 2023-10-30T12:04:15.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 10 librados: watch_flush exit 2023-10-30T12:04:15.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 20 client.44507.objecter shutdown clearing up homeless session... 2023-10-30T12:04:15.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 10 client.44507.objecter successfully canceled tick 2023-10-30T12:04:15.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdfc03a7b0 msgr2=0x7fcdfc03cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdfc03a7b0 0x7fcdfc03cc70 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 crypto rx=0x7fce101c5c50 tx=0x7fce04008040 comp rx=0 tx=0).stop 2023-10-30T12:04:15.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 10 monclient: shutdown 2023-10-30T12:04:15.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdfc009ff0 msgr2=0x7fce10070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdfc009ff0 0x7fce10070990 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fce00000c00 tx=0x7fce000046c0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 shutdown_connections 2023-10-30T12:04:15.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdfc03a7b0 0x7fcdfc03cc70 unknown :-1 s=CLOSED pgs=3909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100969a0 0x7fce101c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce10096080 0x7fce101b6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 --2- 172.21.15.50:0/3883447494 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdfc009ff0 0x7fce10070990 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 >> 172.21.15.50:0/3883447494 conn(0x7fce100a05c0 msgr2=0x7fce101b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 shutdown_connections 2023-10-30T12:04:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 -- 172.21.15.50:0/3883447494 wait complete. 2023-10-30T12:04:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.345+0000 7fce15e52640 1 librados: shutdown 2023-10-30T12:04:15.357 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:04:15.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:04:15.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:04:15.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.462+0000 7f2a602c2640 10 monclient: get_monmap_and_config 2023-10-30T12:04:15.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.462+0000 7f2a602c2640 10 monclient: build_initial_monmap 2023-10-30T12:04:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.462+0000 7f2a602c2640 10 monclient: monmap: 2023-10-30T12:04:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:15.463291+0000 2023-10-30T12:04:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:15.463291+0000 2023-10-30T12:04:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:15.466 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.462+0000 7f2a602c2640 1 Processor -- start 2023-10-30T12:04:15.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.462+0000 7f2a602c2640 1 -- start start 2023-10-30T12:04:15.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.462+0000 7f2a602c2640 10 monclient: init 2023-10-30T12:04:15.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:15.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:04:15.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a5809e7e0 0x7f2a5806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5809e7e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2a5809ebc0 0x7f2a5809f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5809ebc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:04:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a58133650 0x7f2a58133a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): picked mon.noname-a con 0x7f2a58133650 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5809d5f0 con 0x7f2a58133650 2023-10-30T12:04:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5809fbf0 con 0x7f2a5809ebc0 2023-10-30T12:04:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5809fed0 con 0x7f2a5809e7e0 2023-10-30T12:04:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a602c2640 10 monclient(hunting): authenticate will time out at 1100.646507s 2023-10-30T12:04:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a58133650 0x7f2a58133a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a5809e7e0 0x7f2a5806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a5809e7e0 0x7f2a5806c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48798/0 (socket says 172.21.15.50:48798) 2023-10-30T12:04:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a58133650 0x7f2a58133a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59498/0 (socket says 172.21.15.50:59498) 2023-10-30T12:04:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 -- 172.21.15.50:0/1861688669 learned_addr learned my addr 172.21.15.50:0/1861688669 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): get_auth_request con 0x7f2a5809e7e0 auth_method 0 2023-10-30T12:04:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 10 monclient(hunting): get_auth_request con 0x7f2a58133650 auth_method 0 2023-10-30T12:04:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e85f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5d85d640 1 --2- 172.21.15.50:0/1861688669 >> v2:172.21.15.118:3300/0 conn(0x7f2a5809ebc0 0x7f2a5809f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5d85d640 10 monclient(hunting): get_auth_request con 0x7f2a5809ebc0 auth_method 0 2023-10-30T12:04:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5d85d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5d85d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5d85d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_done global_id 44510 payload 274 2023-10-30T12:04:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 -- 172.21.15.50:0/1861688669 >> v2:172.21.15.118:3300/0 conn(0x7f2a5809ebc0 msgr2=0x7f2a5809f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/1861688669 >> v2:172.21.15.118:3300/0 conn(0x7f2a5809ebc0 0x7f2a5809f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 -- 172.21.15.50:0/1861688669 >> v2:172.21.15.50:3300/0 conn(0x7f2a58133650 msgr2=0x7f2a58133a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/1861688669 >> v2:172.21.15.50:3300/0 conn(0x7f2a58133650 0x7f2a58133a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 monclient: found mon.noname-b 2023-10-30T12:04:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.463+0000 7f2a5e05e640 1 -- 172.21.15.50:0/1861688669 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a580a01b0 con 0x7f2a5809e7e0 2023-10-30T12:04:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5e85f640 1 --2- 172.21.15.50:0/1861688669 >> v2:172.21.15.50:3300/0 conn(0x7f2a58133650 0x7f2a58133a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/1861688669 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a5809e7e0 0x7f2a5806c3c0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f2a5809afa0 tx=0x7f2a5402b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=907d945a07394fc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 1 -- 172.21.15.50:0/1861688669 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54031750 con 0x7f2a5809e7e0 2023-10-30T12:04:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: dump: 2023-10-30T12:04:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.487 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 1 -- 172.21.15.50:0/1861688669 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a540318b0 con 0x7f2a5809e7e0 2023-10-30T12:04:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a602c2640 5 monclient: authenticate success, global_id 44510 2023-10-30T12:04:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a602c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a602c2640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a602c2640 4 set_mon_vals no callback set 2023-10-30T12:04:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 1 -- 172.21.15.50:0/1861688669 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54031c40 con 0x7f2a5809e7e0 2023-10-30T12:04:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.464+0000 7f2a5d05c640 10 monclient: dump: 2023-10-30T12:04:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.491 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 10 monclient: shutdown 2023-10-30T12:04:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 -- 172.21.15.50:0/1861688669 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a5809e7e0 msgr2=0x7f2a5806c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 --2- 172.21.15.50:0/1861688669 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a5809e7e0 0x7f2a5806c3c0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f2a5809afa0 tx=0x7f2a5402b7f0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 -- 172.21.15.50:0/1861688669 shutdown_connections 2023-10-30T12:04:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 --2- 172.21.15.50:0/1861688669 >> v2:172.21.15.50:3300/0 conn(0x7f2a58133650 0x7f2a58133a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 --2- 172.21.15.50:0/1861688669 >> v2:172.21.15.118:3300/0 conn(0x7f2a5809ebc0 0x7f2a5809f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 --2- 172.21.15.50:0/1861688669 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a5809e7e0 0x7f2a5806c3c0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 -- 172.21.15.50:0/1861688669 >> 172.21.15.50:0/1861688669 conn(0x7f2a580a4800 msgr2=0x7f2a580a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 -- 172.21.15.50:0/1861688669 shutdown_connections 2023-10-30T12:04:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.465+0000 7f2a602c2640 1 -- 172.21.15.50:0/1861688669 wait complete. 2023-10-30T12:04:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient: build_initial_monmap 2023-10-30T12:04:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient: monmap: 2023-10-30T12:04:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:15.495 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 librados: starting msgr at 2023-10-30T12:04:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 librados: starting objecter 2023-10-30T12:04:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 Processor -- start 2023-10-30T12:04:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 -- start start 2023-10-30T12:04:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 librados: setting wanted keys 2023-10-30T12:04:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 librados: calling monclient init 2023-10-30T12:04:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient: init 2023-10-30T12:04:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:04:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a5809e7e0 0x7f2a581c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5809e7e0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5809ebc0 0x7f2a58132580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): picked mon.noname-a con 0x7f2a5809ebc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a58133650 0x7f2a58132ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): picked mon.noname-c con 0x7f2a58133650 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a581c8590 con 0x7f2a5809ebc0 2023-10-30T12:04:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5811f980 con 0x7f2a5809e7e0 2023-10-30T12:04:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a581bb7b0 con 0x7f2a58133650 2023-10-30T12:04:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a602c2640 10 monclient(hunting): authenticate will time out at 1100.649867s 2023-10-30T12:04:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a5d85d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5809ebc0 0x7f2a58132580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a5e85f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a58133650 0x7f2a58132ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a5e85f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a58133650 0x7f2a58132ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48806/0 (socket says 172.21.15.50:48806) 2023-10-30T12:04:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a5d85d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5809ebc0 0x7f2a58132580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59502/0 (socket says 172.21.15.50:59502) 2023-10-30T12:04:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a5e85f640 1 -- 172.21.15.50:0/499306567 learned_addr learned my addr 172.21.15.50:0/499306567 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.466+0000 7f2a5e85f640 10 monclient(hunting): get_auth_request con 0x7f2a58133650 auth_method 0 2023-10-30T12:04:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e85f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e85f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e85f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): get_auth_request con 0x7f2a5809ebc0 auth_method 0 2023-10-30T12:04:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a5809e7e0 0x7f2a581c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 10 monclient(hunting): get_auth_request con 0x7f2a5809e7e0 auth_method 0 2023-10-30T12:04:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e85f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e85f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e85f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient(hunting): handle_auth_done global_id 74112 payload 824 2023-10-30T12:04:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a58133650 msgr2=0x7f2a58132ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a58133650 0x7f2a58132ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a5809e7e0 msgr2=0x7f2a581c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a5809e7e0 0x7f2a581c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 1 monclient: found mon.noname-a 2023-10-30T12:04:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 1 -- 172.21.15.50:0/499306567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50009a70 con 0x7f2a5809ebc0 2023-10-30T12:04:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5e85f640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a58133650 0x7f2a58132ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5d85d640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5809ebc0 0x7f2a58132580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800d5f0 tx=0x7f2a4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 client.?.objecter ms_handle_connect 0x7f2a5809ebc0 2023-10-30T12:04:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a480125f0 con 0x7f2a5809ebc0 2023-10-30T12:04:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: dump: 2023-10-30T12:04:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.509 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a48012750 con 0x7f2a5809ebc0 2023-10-30T12:04:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48012b00 con 0x7f2a5809ebc0 2023-10-30T12:04:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 5 monclient: authenticate success, global_id 74112 2023-10-30T12:04:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 10 monclient: _renew_subs 2023-10-30T12:04:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a540094d0 con 0x7f2a5809ebc0 2023-10-30T12:04:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a5c85b640 4 set_mon_vals no callback set 2023-10-30T12:04:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a4effd640 10 monclient: dump: 2023-10-30T12:04:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.514 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 10 client.74112.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 10 monclient: _renew_subs 2023-10-30T12:04:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a5811ff40 con 0x7f2a5809ebc0 2023-10-30T12:04:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.467+0000 7f2a602c2640 1 librados: init done 2023-10-30T12:04:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a602c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a602c2640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c030 0x7f2a5811c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a580a1280 con 0x7f2a5811c030 2023-10-30T12:04:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2a48012c60 con 0x7f2a5809ebc0 2023-10-30T12:04:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c030 0x7f2a5811c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient: get_auth_request con 0x7f2a5811c030 auth_method 0 2023-10-30T12:04:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a4effd640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a3c03d700 0x7f2a3c03fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_done global_id 74115 payload 824 2023-10-30T12:04:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a5e05e640 1 -- 172.21.15.50:0/499306567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a5809f560 con 0x7f2a5811c030 2023-10-30T12:04:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f2a4804b280 con 0x7f2a5809ebc0 2023-10-30T12:04:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a4effd640 10 client.74112.objecter ms_dispatch 0x7f2a58000b90 osd_map(111..111 src has 1..111) v4 2023-10-30T12:04:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a4effd640 3 client.74112.objecter handle_osd_map got epochs [111,111] > 0 2023-10-30T12:04:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.468+0000 7f2a4effd640 3 client.74112.objecter handle_osd_map decoding full epoch 111 2023-10-30T12:04:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 20 client.74112.objecter dump_active .. 0 homeless 2023-10-30T12:04:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a5e85f640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a3c03d700 0x7f2a3c03fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a5e85f640 10 monclient: get_auth_request con 0x7f2a3c03d700 auth_method 0 2023-10-30T12:04:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 client.74112.objecter ms_handle_connect 0x7f2a5811c030 2023-10-30T12:04:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 client.74112.objecter resend_mon_ops 2023-10-30T12:04:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c030 0x7f2a5811c410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a58071290 tx=0x7f2a540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54033bd0 con 0x7f2a5811c030 2023-10-30T12:04:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 monclient: dump: 2023-10-30T12:04:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.522 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f2a54041070 con 0x7f2a5811c030 2023-10-30T12:04:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a5e85f640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a3c03d700 0x7f2a3c03fbc0 secure :-1 s=READY pgs=3910 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009fd0 tx=0x7f2a5000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c030 msgr2=0x7f2a5811c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c030 0x7f2a5811c410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a58071290 tx=0x7f2a540046c0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c030 msgr2=0x7f2a5811c410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:15.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c030 0x7f2a5811c410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.469+0000 7f2a4effd640 10 client.74112.objecter ms_handle_connect 0x7f2a3c03d700 2023-10-30T12:04:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.475+0000 7f2a602c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.475+0000 7f2a602c2640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c950 0x7f2a5806e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.475+0000 7f2a602c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.475+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a58074800 con 0x7f2a5811c950 2023-10-30T12:04:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.475+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c950 0x7f2a5806e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient: get_auth_request con 0x7f2a5811c950 auth_method 0 2023-10-30T12:04:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 10 monclient(hunting): handle_auth_done global_id 74118 payload 824 2023-10-30T12:04:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 1 -- 172.21.15.50:0/499306567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a5809d2f0 con 0x7f2a5811c950 2023-10-30T12:04:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 10 client.74112.objecter ms_handle_connect 0x7f2a5811c950 2023-10-30T12:04:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 10 client.74112.objecter resend_mon_ops 2023-10-30T12:04:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a5e05e640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c950 0x7f2a5806e5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a5402b830 tx=0x7f2a54031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54031ee0 con 0x7f2a5811c950 2023-10-30T12:04:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 10 monclient: dump: 2023-10-30T12:04:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.531 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f2a54052660 con 0x7f2a5811c950 2023-10-30T12:04:15.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:15.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:15.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c950 msgr2=0x7f2a5806e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c950 0x7f2a5806e5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a5402b830 tx=0x7f2a54031000 comp rx=0 tx=0).stop 2023-10-30T12:04:15.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c950 msgr2=0x7f2a5806e5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:15.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a4effd640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5811c950 0x7f2a5806e5a0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a5402b830 tx=0x7f2a54031000 comp rx=0 tx=0).stop 2023-10-30T12:04:15.533 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:15.533 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:15.533 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:04:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 40, 2023-10-30T12:04:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:04:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:04:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:04:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:15.563 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:15.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a602c2640 10 librados: watch_flush enter 2023-10-30T12:04:15.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a602c2640 10 librados: watch_flush exit 2023-10-30T12:04:15.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a602c2640 20 client.74112.objecter shutdown clearing up homeless session... 2023-10-30T12:04:15.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.476+0000 7f2a602c2640 10 client.74112.objecter successfully canceled tick 2023-10-30T12:04:15.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a3c03d700 msgr2=0x7f2a3c03fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a3c03d700 0x7f2a3c03fbc0 secure :-1 s=READY pgs=3910 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009fd0 tx=0x7f2a5000b040 comp rx=0 tx=0).stop 2023-10-30T12:04:15.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 10 monclient: shutdown 2023-10-30T12:04:15.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5809ebc0 msgr2=0x7f2a58132580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 --2- 172.21.15.50:0/499306567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a5809ebc0 0x7f2a58132580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800d5f0 tx=0x7f2a4800dab0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a5e85f640 1 -- 172.21.15.50:0/499306567 reap_dead start 2023-10-30T12:04:15.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a5e85f640 1 -- 172.21.15.50:0/499306567 reap_dead start 2023-10-30T12:04:15.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 shutdown_connections 2023-10-30T12:04:15.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 >> 172.21.15.50:0/499306567 conn(0x7f2a580a4800 msgr2=0x7f2a580a1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 shutdown_connections 2023-10-30T12:04:15.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 -- 172.21.15.50:0/499306567 wait complete. 2023-10-30T12:04:15.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.477+0000 7f2a602c2640 1 librados: shutdown 2023-10-30T12:04:15.568 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:04:15.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:04:15.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:04:15.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.674+0000 7f221227c640 10 monclient: get_monmap_and_config 2023-10-30T12:04:15.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.674+0000 7f221227c640 10 monclient: build_initial_monmap 2023-10-30T12:04:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.674+0000 7f221227c640 10 monclient: monmap: 2023-10-30T12:04:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:15.675113+0000 2023-10-30T12:04:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:15.675113+0000 2023-10-30T12:04:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:15.678 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.674+0000 7f221227c640 1 Processor -- start 2023-10-30T12:04:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.674+0000 7f221227c640 1 -- start start 2023-10-30T12:04:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.674+0000 7f221227c640 10 monclient: init 2023-10-30T12:04:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-10-30T12:04:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f220c0a8010 0x7f220c132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient(hunting): picked mon.noname-a con 0x7f220c0a8010 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient(hunting): picked mon.noname-b con 0x7f220c099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f220c1356d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220c0a7190 con 0x7f220c1356d0 2023-10-30T12:04:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220c133110 con 0x7f220c0a8010 2023-10-30T12:04:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0a83f0 con 0x7f220c099a40 2023-10-30T12:04:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f221227c640 10 monclient(hunting): authenticate will time out at 1100.858289s 2023-10-30T12:04:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f220c0a8010 0x7f220c132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f220c0a8010 0x7f220c132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59522/0 (socket says 172.21.15.50:59522) 2023-10-30T12:04:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f2210819640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f220c1356d0 0x7f220c135b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34628/0 (socket says 172.21.15.50:34628) 2023-10-30T12:04:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 1 -- 172.21.15.50:0/1864281673 learned_addr learned my addr 172.21.15.50:0/1864281673 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 10 monclient(hunting): get_auth_request con 0x7f220c0a8010 auth_method 0 2023-10-30T12:04:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): get_auth_request con 0x7f220c099a40 auth_method 0 2023-10-30T12:04:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient(hunting): handle_auth_done global_id 44516 payload 274 2023-10-30T12:04:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 1 -- 172.21.15.50:0/1864281673 >> v2:172.21.15.50:3300/0 conn(0x7f220c0a8010 msgr2=0x7f220c132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 1 --2- 172.21.15.50:0/1864281673 >> v2:172.21.15.50:3300/0 conn(0x7f220c0a8010 0x7f220c132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 1 -- 172.21.15.50:0/1864281673 >> v1:172.21.15.50:6789/0 conn(0x7f220c1356d0 legacy=0x7f220c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 1 monclient: found mon.noname-b 2023-10-30T12:04:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220b7fe640 1 -- 172.21.15.50:0/1864281673 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c0a7600 con 0x7f220c099a40 2023-10-30T12:04:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.675+0000 7f220bfff640 1 --2- 172.21.15.50:0/1864281673 >> v2:172.21.15.50:3300/0 conn(0x7f220c0a8010 0x7f220c132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220b7fe640 1 --2- 172.21.15.50:0/1864281673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c132bd0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0097a0 tx=0x7f21fc02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2977204b71f6a628 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 1 -- 172.21.15.50:0/1864281673 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc0316f0 con 0x7f220c099a40 2023-10-30T12:04:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: dump: 2023-10-30T12:04:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.698 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 1 -- 172.21.15.50:0/1864281673 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc031850 con 0x7f220c099a40 2023-10-30T12:04:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f221227c640 5 monclient: authenticate success, global_id 44516 2023-10-30T12:04:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f221227c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 1 -- 172.21.15.50:0/1864281673 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc031be0 con 0x7f220c099a40 2023-10-30T12:04:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f221227c640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f221227c640 4 set_mon_vals no callback set 2023-10-30T12:04:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.676+0000 7f220affd640 10 monclient: dump: 2023-10-30T12:04:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 10 monclient: shutdown 2023-10-30T12:04:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 -- 172.21.15.50:0/1864281673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 msgr2=0x7f220c132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 --2- 172.21.15.50:0/1864281673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c132bd0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0097a0 tx=0x7f21fc02b790 comp rx=0 tx=0).stop 2023-10-30T12:04:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 -- 172.21.15.50:0/1864281673 shutdown_connections 2023-10-30T12:04:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 --2- 172.21.15.50:0/1864281673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c132bd0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 --2- 172.21.15.50:0/1864281673 >> v2:172.21.15.50:3300/0 conn(0x7f220c0a8010 0x7f220c132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 -- 172.21.15.50:0/1864281673 >> 172.21.15.50:0/1864281673 conn(0x7f220c0a4a10 msgr2=0x7f220c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 -- 172.21.15.50:0/1864281673 shutdown_connections 2023-10-30T12:04:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 -- 172.21.15.50:0/1864281673 wait complete. 2023-10-30T12:04:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 10 monclient: build_initial_monmap 2023-10-30T12:04:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 10 monclient: monmap: 2023-10-30T12:04:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:15.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 librados: starting msgr at 2023-10-30T12:04:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.677+0000 7f221227c640 1 librados: starting objecter 2023-10-30T12:04:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 Processor -- start 2023-10-30T12:04:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 -- start start 2023-10-30T12:04:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 librados: setting wanted keys 2023-10-30T12:04:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 librados: calling monclient init 2023-10-30T12:04:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient: init 2023-10-30T12:04:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:04:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f220c0a8010 0x7f220c1ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): picked mon.noname-b con 0x7f220c0a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f220c1356d0 0x7f220c1cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): picked mon.noname-a con 0x7f220c1356d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11e6c0 0x7f220c11eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): picked mon.noname-c con 0x7f220c11e6c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c121d60 con 0x7f220c1356d0 2023-10-30T12:04:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c120e70 con 0x7f220c0a8010 2023-10-30T12:04:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c134b10 con 0x7f220c11e6c0 2023-10-30T12:04:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f221227c640 10 monclient(hunting): authenticate will time out at 1100.861724s 2023-10-30T12:04:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11e6c0 0x7f220c11eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f220c1356d0 0x7f220c1cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11e6c0 0x7f220c11eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48810/0 (socket says 172.21.15.50:48810) 2023-10-30T12:04:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f220c1356d0 0x7f220c1cac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59534/0 (socket says 172.21.15.50:59534) 2023-10-30T12:04:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 1 -- 172.21.15.50:0/3984399819 learned_addr learned my addr 172.21.15.50:0/3984399819 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 10 monclient(hunting): get_auth_request con 0x7f220c11e6c0 auth_method 0 2023-10-30T12:04:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 10 monclient(hunting): get_auth_request con 0x7f220c1356d0 auth_method 0 2023-10-30T12:04:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f220b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.678+0000 7f2210819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220bfff640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f220c0a8010 0x7f220c1ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 10 monclient(hunting): handle_auth_done global_id 74127 payload 824 2023-10-30T12:04:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11e6c0 msgr2=0x7f220c11eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11e6c0 0x7f220c11eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f220c0a8010 msgr2=0x7f220c1ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f220c0a8010 0x7f220c1ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:04:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 1 -- 172.21.15.50:0/3984399819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000094d0 con 0x7f220c1356d0 2023-10-30T12:04:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220bfff640 10 monclient: get_auth_request con 0x7f220c0a8010 auth_method 0 2023-10-30T12:04:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220bfff640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f220c0a8010 0x7f220c1ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:04:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2210819640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11e6c0 0x7f220c11eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220b7fe640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f220c1356d0 0x7f220c1cac60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21fc033000 tx=0x7f21fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 client.?.objecter ms_handle_connect 0x7f220c1356d0 2023-10-30T12:04:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc034680 con 0x7f220c1356d0 2023-10-30T12:04:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: dump: 2023-10-30T12:04:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.720 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc034800 con 0x7f220c1356d0 2023-10-30T12:04:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc034bb0 con 0x7f220c1356d0 2023-10-30T12:04:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 5 monclient: authenticate success, global_id 74127 2023-10-30T12:04:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 10 monclient: _renew_subs 2023-10-30T12:04:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc009400 con 0x7f220c1356d0 2023-10-30T12:04:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f220a7fc640 4 set_mon_vals no callback set 2023-10-30T12:04:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f2208ff9640 10 monclient: dump: 2023-10-30T12:04:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.725 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 10 client.74127.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 10 monclient: _renew_subs 2023-10-30T12:04:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c122600 con 0x7f220c1356d0 2023-10-30T12:04:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.679+0000 7f221227c640 1 librados: init done 2023-10-30T12:04:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f21fc03b040 con 0x7f220c1356d0 2023-10-30T12:04:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f221227c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f221227c640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f221227c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c073fd0 con 0x7f220c099a40 2023-10-30T12:04:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient: get_auth_request con 0x7f220c099a40 auth_method 0 2023-10-30T12:04:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21f80392f0 0x7f21f803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6804+0+0 (secure 0 0 0) 0x7f21fc06c780 con 0x7f220c1356d0 2023-10-30T12:04:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 10 client.74127.objecter ms_dispatch 0x7f220c000b90 osd_map(111..111 src has 1..111) v4 2023-10-30T12:04:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 3 client.74127.objecter handle_osd_map got epochs [111,111] > 0 2023-10-30T12:04:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 3 client.74127.objecter handle_osd_map decoding full epoch 111 2023-10-30T12:04:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 20 client.74127.objecter dump_active .. 0 homeless 2023-10-30T12:04:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 10 monclient(hunting): handle_auth_done global_id 44522 payload 824 2023-10-30T12:04:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 1 -- 172.21.15.50:0/3984399819 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f220c1220d0 con 0x7f220c099a40 2023-10-30T12:04:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 10 client.74127.objecter ms_handle_connect 0x7f220c099a40 2023-10-30T12:04:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2208ff9640 10 client.74127.objecter resend_mon_ops 2023-10-30T12:04:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f220bfff640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c02b430 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f22000108b0 tx=0x7f2200010d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2210819640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21f80392f0 0x7f21f803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.680+0000 7f2210819640 10 monclient: get_auth_request con 0x7f21f80392f0 auth_method 0 2023-10-30T12:04:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220000ebd0 con 0x7f220c099a40 2023-10-30T12:04:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 10 monclient: dump: 2023-10-30T12:04:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.733 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.733 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.733 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.733 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f220000ed30 con 0x7f220c099a40 2023-10-30T12:04:15.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:15.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 msgr2=0x7f220c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c02b430 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f22000108b0 tx=0x7f2200010d70 comp rx=0 tx=0).stop 2023-10-30T12:04:15.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 msgr2=0x7f220c02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:15.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c099a40 0x7f220c02b430 secure :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f22000108b0 tx=0x7f2200010d70 comp rx=0 tx=0).stop 2023-10-30T12:04:15.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2208ff9640 10 client.74127.objecter ms_handle_connect 0x7f21f80392f0 2023-10-30T12:04:15.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.681+0000 7f2210819640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21f80392f0 0x7f21f803b7b0 secure :-1 s=READY pgs=3911 cs=0 l=1 rev1=1 crypto rx=0x7f22040012e0 tx=0x7f2204001710 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f221227c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:15.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f221227c640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11f850 0x7f220c02b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f221227c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c122240 con 0x7f220c11f850 2023-10-30T12:04:15.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11f850 0x7f220c02b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient: get_auth_request con 0x7f220c11f850 auth_method 0 2023-10-30T12:04:15.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 10 monclient(hunting): handle_auth_done global_id 44525 payload 824 2023-10-30T12:04:15.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.687+0000 7f220bfff640 1 -- 172.21.15.50:0/3984399819 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f220c09aa00 con 0x7f220c11f850 2023-10-30T12:04:15.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f220bfff640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11f850 0x7f220c02b970 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f22000108b0 tx=0x7f2200004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 10 client.74127.objecter ms_handle_connect 0x7f220c11f850 2023-10-30T12:04:15.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 10 client.74127.objecter resend_mon_ops 2023-10-30T12:04:15.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200025610 con 0x7f220c11f850 2023-10-30T12:04:15.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:15.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 10 monclient: dump: 2023-10-30T12:04:15.741 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.741 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.741 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.741 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.742 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.742 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.742 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.742 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.742 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.743 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f2200025770 con 0x7f220c11f850 2023-10-30T12:04:15.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:15.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:15.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11f850 msgr2=0x7f220c02b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11f850 0x7f220c02b970 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f22000108b0 tx=0x7f2200004730 comp rx=0 tx=0).stop 2023-10-30T12:04:15.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11f850 msgr2=0x7f220c02b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:15.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2208ff9640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f220c11f850 0x7f220c02b970 secure :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f22000108b0 tx=0x7f2200004730 comp rx=0 tx=0).stop 2023-10-30T12:04:15.744 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:15.744 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:15.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:15.745 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:04:15.745 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 40, 2023-10-30T12:04:15.745 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:15.745 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:04:15.746 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:15.746 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:15.746 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.746 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:04:15.746 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:15.747 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:15.747 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:15.747 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.747 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.747 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.748 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.748 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.748 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.748 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.748 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.749 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.749 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.749 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.749 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:15.749 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:15.750 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.750 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.750 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.750 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.750 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.751 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.751 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.751 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.751 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.751 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.752 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.752 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:15.752 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:15.752 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:15.752 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:15.753 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:15.753 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:15.753 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:15.753 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:15.753 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:15.754 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:15.754 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:15.754 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:15.754 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:15.754 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:15.755 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:15.755 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:15.755 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:15.755 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.755 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.755 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.756 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.756 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.756 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.756 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.756 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.757 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.757 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.757 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.757 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:15.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.758 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:15.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.758 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:15.758 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:15.758 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.759 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.759 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.759 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:15.759 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.760 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.760 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:15.760 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.760 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.761 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.761 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:15.761 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:15.761 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.762 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.762 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.762 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:15.762 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:15.763 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.763 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.763 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.763 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:15.764 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.764 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.764 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:15.766 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.766 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.766 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.767 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:15.767 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:15.767 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.767 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.768 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.768 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:15.768 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.769 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.769 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:15.769 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.769 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.770 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:15.770 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:15.770 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.770 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.770 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.771 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.771 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:15.771 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:15.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.772 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.772 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.772 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:15.772 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.772 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.773 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:04:15.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.773 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.773 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.773 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:04:15.774 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.774 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.774 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:15.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.774 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.775 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.775 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:15.775 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.775 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.776 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:15.776 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:15.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 10 librados: watch_flush enter 2023-10-30T12:04:15.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 10 librados: watch_flush exit 2023-10-30T12:04:15.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 20 client.74127.objecter shutdown clearing up homeless session... 2023-10-30T12:04:15.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 10 client.74127.objecter successfully canceled tick 2023-10-30T12:04:15.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21f80392f0 msgr2=0x7f21f803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21f80392f0 0x7f21f803b7b0 secure :-1 s=READY pgs=3911 cs=0 l=1 rev1=1 crypto rx=0x7f22040012e0 tx=0x7f2204001710 comp rx=0 tx=0).stop 2023-10-30T12:04:15.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 10 monclient: shutdown 2023-10-30T12:04:15.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f220c1356d0 msgr2=0x7f220c1cac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 1 --2- 172.21.15.50:0/3984399819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f220c1356d0 0x7f220c1cac60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21fc033000 tx=0x7f21fc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2210819640 1 -- 172.21.15.50:0/3984399819 reap_dead start 2023-10-30T12:04:15.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f2210819640 1 -- 172.21.15.50:0/3984399819 reap_dead start 2023-10-30T12:04:15.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 shutdown_connections 2023-10-30T12:04:15.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.688+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 >> 172.21.15.50:0/3984399819 conn(0x7f220c0a4a10 msgr2=0x7f220c0a3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.689+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 shutdown_connections 2023-10-30T12:04:15.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.689+0000 7f221227c640 1 -- 172.21.15.50:0/3984399819 wait complete. 2023-10-30T12:04:15.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.689+0000 7f221227c640 1 librados: shutdown 2023-10-30T12:04:15.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:04:15.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:04:15.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:04:15.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.886+0000 7fc2ce864640 10 monclient: get_monmap_and_config 2023-10-30T12:04:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.886+0000 7fc2ce864640 10 monclient: build_initial_monmap 2023-10-30T12:04:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.886+0000 7fc2ce864640 10 monclient: monmap: 2023-10-30T12:04:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:15.887432+0000 2023-10-30T12:04:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:15.887432+0000 2023-10-30T12:04:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:15.890 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 1 Processor -- start 2023-10-30T12:04:15.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 1 -- start start 2023-10-30T12:04:15.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient: init 2023-10-30T12:04:15.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:15.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-10-30T12:04:15.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2c809fa30 0x7fc2c80a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient(hunting): picked mon.noname-a con 0x7fc2c809fa30 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:15.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2c806b720 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:15.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c813b070 0x7fc2c813d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c813b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c809e010 con 0x7fc2c806b720 2023-10-30T12:04:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c809fe10 con 0x7fc2c809fa30 2023-10-30T12:04:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c806bde0 con 0x7fc2c813b070 2023-10-30T12:04:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2ce864640 10 monclient(hunting): authenticate will time out at 1101.070604s 2023-10-30T12:04:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2c809fa30 0x7fc2c80a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2c809fa30 0x7fc2c80a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59548/0 (socket says 172.21.15.50:59548) 2023-10-30T12:04:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1756807820 learned_addr learned my addr 172.21.15.50:0/1756807820 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 10 monclient(hunting): get_auth_request con 0x7fc2c809fa30 auth_method 0 2023-10-30T12:04:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2cce01640 1 --2- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c813b070 0x7fc2c813d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2cce01640 10 monclient(hunting): get_auth_request con 0x7fc2c813b070 auth_method 0 2023-10-30T12:04:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2cce01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2cce01640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.887+0000 7fc2cce01640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2cce01640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2cce01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2cce01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_done global_id 74130 payload 274 2023-10-30T12:04:15.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c813b070 msgr2=0x7fc2c813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:15.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 1 --2- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c813b070 0x7fc2c813d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1756807820 >> v1:172.21.15.118:6789/0 conn(0x7fc2c806b720 legacy=0x7fc2c80a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 1 monclient: found mon.noname-a 2023-10-30T12:04:15.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:15.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1756807820 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c809e540 con 0x7fc2c809fa30 2023-10-30T12:04:15.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c7fff640 1 --2- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c809fa30 0x7fc2c80a1d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80098a0 tx=0x7fc2b802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10775f3f583f5966 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c6ffd640 1 -- 172.21.15.50:0/1756807820 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b80317c0 con 0x7fc2c809fa30 2023-10-30T12:04:15.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:15.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.888+0000 7fc2c6ffd640 10 monclient: dump: 2023-10-30T12:04:15.907 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.908 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.908 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.908 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.908 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.908 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.909 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 5 monclient: authenticate success, global_id 74130 2023-10-30T12:04:15.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 1 -- 172.21.15.50:0/1756807820 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b8031920 con 0x7fc2c809fa30 2023-10-30T12:04:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 4 set_mon_vals no callback set 2023-10-30T12:04:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 1 -- 172.21.15.50:0/1756807820 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8031cb0 con 0x7fc2c809fa30 2023-10-30T12:04:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2c6ffd640 10 monclient: dump: 2023-10-30T12:04:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.914 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 10 monclient: shutdown 2023-10-30T12:04:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 -- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c809fa30 msgr2=0x7fc2c80a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c809fa30 0x7fc2c80a1d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80098a0 tx=0x7fc2b802b860 comp rx=0 tx=0).stop 2023-10-30T12:04:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 -- 172.21.15.50:0/1756807820 shutdown_connections 2023-10-30T12:04:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c813b070 0x7fc2c813d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1756807820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c809fa30 0x7fc2c80a1d30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 -- 172.21.15.50:0/1756807820 >> 172.21.15.50:0/1756807820 conn(0x7fc2c80a6200 msgr2=0x7fc2c80a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 -- 172.21.15.50:0/1756807820 shutdown_connections 2023-10-30T12:04:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.889+0000 7fc2ce864640 1 -- 172.21.15.50:0/1756807820 wait complete. 2023-10-30T12:04:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 10 monclient: build_initial_monmap 2023-10-30T12:04:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 10 monclient: monmap: 2023-10-30T12:04:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:15.918 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 1 librados: starting msgr at 2023-10-30T12:04:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 1 librados: starting objecter 2023-10-30T12:04:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 1 Processor -- start 2023-10-30T12:04:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 1 -- start start 2023-10-30T12:04:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 1 librados: setting wanted keys 2023-10-30T12:04:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 1 librados: calling monclient init 2023-10-30T12:04:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.890+0000 7fc2ce864640 10 monclient: init 2023-10-30T12:04:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:15.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:04:15.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c809fa30 0x7fc2c8139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): picked mon.noname-c con 0x7fc2c809fa30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:15.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c813b070 0x7fc2c8122610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c813b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c8122b50 0x7fc2c8122fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): picked mon.noname-a con 0x7fc2c8122b50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c81cb040 con 0x7fc2c8122b50 2023-10-30T12:04:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8070e50 con 0x7fc2c813b070 2023-10-30T12:04:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8070520 con 0x7fc2c809fa30 2023-10-30T12:04:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2ce864640 10 monclient(hunting): authenticate will time out at 1101.074315s 2023-10-30T12:04:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c809fa30 0x7fc2c8139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c8122b50 0x7fc2c8122fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c809fa30 0x7fc2c8139cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48844/0 (socket says 172.21.15.50:48844) 2023-10-30T12:04:15.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1190125562 learned_addr learned my addr 172.21.15.50:0/1190125562 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:15.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c8122b50 0x7fc2c8122fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59556/0 (socket says 172.21.15.50:59556) 2023-10-30T12:04:15.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): get_auth_request con 0x7fc2c809fa30 auth_method 0 2023-10-30T12:04:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 10 monclient(hunting): get_auth_request con 0x7fc2c8122b50 auth_method 0 2023-10-30T12:04:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2cce01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c813b070 0x7fc2c8122610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 10 monclient(hunting): get_auth_request con 0x7fc2c813b070 auth_method 0 2023-10-30T12:04:15.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient(hunting): handle_auth_done global_id 44531 payload 824 2023-10-30T12:04:15.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c813b070 msgr2=0x7fc2c8122610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c813b070 0x7fc2c8122610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c8122b50 msgr2=0x7fc2c8122fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2c8122b50 0x7fc2c8122fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:04:15.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 monclient: found mon.noname-c 2023-10-30T12:04:15.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:15.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 -- 172.21.15.50:0/1190125562 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b80094d0 con 0x7fc2c809fa30 2023-10-30T12:04:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c7fff640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c809fa30 0x7fc2c8139cd0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8000c00 tx=0x7fc2b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c4ff9640 10 client.?.objecter ms_handle_connect 0x7fc2c809fa30 2023-10-30T12:04:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.891+0000 7fc2c4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8032860 con 0x7fc2c809fa30 2023-10-30T12:04:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:15.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: dump: 2023-10-30T12:04:15.931 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.931 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.931 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.931 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.932 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.932 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.932 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.932 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.932 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.933 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:04:15.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:04:15.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:15.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b80329c0 con 0x7fc2c809fa30 2023-10-30T12:04:15.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 5 monclient: authenticate success, global_id 44531 2023-10-30T12:04:15.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 10 monclient: _renew_subs 2023-10-30T12:04:15.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c809f1b0 con 0x7fc2c809fa30 2023-10-30T12:04:15.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:15.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8032d70 con 0x7fc2c809fa30 2023-10-30T12:04:15.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:15.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 10 monclient: dump: 2023-10-30T12:04:15.935 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.935 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.936 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.936 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.936 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.936 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.936 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.937 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.937 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.937 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c67fc640 4 set_mon_vals no callback set 2023-10-30T12:04:15.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 10 client.44531.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:15.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 10 monclient: _renew_subs 2023-10-30T12:04:15.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:15.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c8125de0 con 0x7fc2c809fa30 2023-10-30T12:04:15.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 1 librados: init done 2023-10-30T12:04:15.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc2b804e450 con 0x7fc2c809fa30 2023-10-30T12:04:15.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c806b720 0x7fc2c8123500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.892+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8073ff0 con 0x7fc2c806b720 2023-10-30T12:04:15.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc2a80392f0 0x7fc2a803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6804+0+0 (secure 0 0 0) 0x7fc2b806cf90 con 0x7fc2c809fa30 2023-10-30T12:04:15.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c806b720 0x7fc2c8123500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 10 client.44531.objecter ms_dispatch 0x7fc2c8000b90 osd_map(111..111 src has 1..111) v4 2023-10-30T12:04:15.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 3 client.44531.objecter handle_osd_map got epochs [111,111] > 0 2023-10-30T12:04:15.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 3 client.44531.objecter handle_osd_map decoding full epoch 111 2023-10-30T12:04:15.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient: get_auth_request con 0x7fc2c806b720 auth_method 0 2023-10-30T12:04:15.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 20 client.44531.objecter dump_active .. 0 homeless 2023-10-30T12:04:15.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2cce01640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc2a80392f0 0x7fc2a803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2cce01640 10 monclient: get_auth_request con 0x7fc2a80392f0 auth_method 0 2023-10-30T12:04:15.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_done global_id 28189 payload 824 2023-10-30T12:04:15.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 1 -- 172.21.15.50:0/1190125562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2c806ab30 con 0x7fc2c806b720 2023-10-30T12:04:15.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 10 client.44531.objecter ms_handle_connect 0x7fc2a80392f0 2023-10-30T12:04:15.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2cce01640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc2a80392f0 0x7fc2a803b7b0 secure :-1 s=READY pgs=3912 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8071860 tx=0x7fc2c000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 10 client.44531.objecter ms_handle_connect 0x7fc2c806b720 2023-10-30T12:04:15.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 10 client.44531.objecter resend_mon_ops 2023-10-30T12:04:15.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c77fe640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c806b720 0x7fc2c8123500 secure :-1 s=READY pgs=7105 cs=0 l=1 rev1=1 crypto rx=0x7fc2c81225b0 tx=0x7fc2b000fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b00050d0 con 0x7fc2c806b720 2023-10-30T12:04:15.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:15.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.893+0000 7fc2c4ff9640 10 monclient: dump: 2023-10-30T12:04:15.944 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.944 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.945 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.945 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.945 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.945 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.945 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.946 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.946 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.946 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.894+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc2b0005230 con 0x7fc2c806b720 2023-10-30T12:04:15.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.894+0000 7fc2c4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:15.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.894+0000 7fc2c4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:15.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.894+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c806b720 msgr2=0x7fc2c8123500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.894+0000 7fc2c4ff9640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c806b720 0x7fc2c8123500 secure :-1 s=READY pgs=7105 cs=0 l=1 rev1=1 crypto rx=0x7fc2c81225b0 tx=0x7fc2b000fc00 comp rx=0 tx=0).stop 2023-10-30T12:04:15.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.894+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c806b720 msgr2=0x7fc2c8123500 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:15.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.894+0000 7fc2c4ff9640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c806b720 0x7fc2c8123500 secure :-1 s=CLOSED pgs=7105 cs=0 l=1 rev1=1 crypto rx=0x7fc2c81225b0 tx=0x7fc2b000fc00 comp rx=0 tx=0).stop 2023-10-30T12:04:15.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2ce864640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c811eec0 0x7fc2c811f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2ce864640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c806a690 con 0x7fc2c811eec0 2023-10-30T12:04:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c811eec0 0x7fc2c811f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 10 monclient: get_auth_request con 0x7fc2c811eec0 auth_method 0 2023-10-30T12:04:15.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:15.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:15.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:15.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:15.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:15.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.900+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:15.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c77fe640 10 monclient(hunting): handle_auth_done global_id 28192 payload 824 2023-10-30T12:04:15.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c77fe640 1 -- 172.21.15.50:0/1190125562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc2c806afc0 con 0x7fc2c811eec0 2023-10-30T12:04:15.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c77fe640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c811eec0 0x7fc2c811f2d0 secure :-1 s=READY pgs=7106 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0017000 tx=0x7fc2b0001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:15.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 10 client.44531.objecter ms_handle_connect 0x7fc2c811eec0 2023-10-30T12:04:15.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 10 client.44531.objecter resend_mon_ops 2023-10-30T12:04:15.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0024070 con 0x7fc2c811eec0 2023-10-30T12:04:15.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:15.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:15.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 10 monclient: dump: 2023-10-30T12:04:15.953 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:15.954 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:15.954 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.954 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:15.954 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:15.954 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:15.955 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fc2b0007300 con 0x7fc2c811eec0 2023-10-30T12:04:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:15.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:15.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c811eec0 msgr2=0x7fc2c811f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c811eec0 0x7fc2c811f2d0 secure :-1 s=READY pgs=7106 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0017000 tx=0x7fc2b0001120 comp rx=0 tx=0).stop 2023-10-30T12:04:15.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c811eec0 msgr2=0x7fc2c811f2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:15.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2c4ff9640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2c811eec0 0x7fc2c811f2d0 unknown :-1 s=CLOSED pgs=7106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.957 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:15.957 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:15.957 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:15.957 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:04:15.958 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 40, 2023-10-30T12:04:15.958 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:15.958 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:04:15.958 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:15.958 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:04:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:15.959 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.960 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.961 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.962 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.963 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.963 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.963 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.963 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.963 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.964 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.964 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.964 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.964 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:15.965 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:15.965 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:15.965 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:15.965 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:15.965 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:15.965 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:15.966 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:15.966 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:15.966 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:15.966 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:15.966 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:15.967 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:15.967 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:15.967 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:15.967 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:15.967 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:15.968 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:15.968 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:15.968 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:15.968 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:15.968 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:15.968 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:15.969 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:15.969 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:15.969 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:15.969 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:15.969 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.970 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:15.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.970 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:15.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.970 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:15.970 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:15.971 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.971 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.971 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.971 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:15.972 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.972 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.972 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:15.973 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.973 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:15.974 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:15.974 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.974 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.974 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.975 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:15.975 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:15.975 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.975 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.976 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.976 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:15.976 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.977 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.977 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:15.977 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.977 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.977 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.979 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.979 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:15.979 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:15.980 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:15.980 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:15.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.980 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:15.980 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:15.982 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:15.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.982 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.982 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:15.982 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:15.983 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:15.983 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:15.983 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:15.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.983 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.984 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:15.984 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:15.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.984 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.985 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.985 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:15.985 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.985 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.985 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:04:15.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.986 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.986 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.986 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:04:15.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 9 2023-10-30T12:04:15.988 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.988 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:15.988 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:04:15.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:15.988 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:15.989 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:15.989 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:15.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:15.989 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:15.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:15.990 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:15.990 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:15.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2ce864640 10 librados: watch_flush enter 2023-10-30T12:04:15.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2ce864640 10 librados: watch_flush exit 2023-10-30T12:04:15.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2ce864640 20 client.44531.objecter shutdown clearing up homeless session... 2023-10-30T12:04:15.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.901+0000 7fc2ce864640 10 client.44531.objecter successfully canceled tick 2023-10-30T12:04:15.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc2a80392f0 msgr2=0x7fc2a803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc2a80392f0 0x7fc2a803b7b0 secure :-1 s=READY pgs=3912 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8071860 tx=0x7fc2c000b040 comp rx=0 tx=0).stop 2023-10-30T12:04:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 10 monclient: shutdown 2023-10-30T12:04:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c809fa30 msgr2=0x7fc2c8139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c809fa30 0x7fc2c8139cd0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8000c00 tx=0x7fc2b80046c0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2cce01640 1 -- 172.21.15.50:0/1190125562 reap_dead start 2023-10-30T12:04:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 shutdown_connections 2023-10-30T12:04:15.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 --2- 172.21.15.50:0/1190125562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc2c809fa30 0x7fc2c8139cd0 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:15.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 >> 172.21.15.50:0/1190125562 conn(0x7fc2c80a6200 msgr2=0x7fc2c806e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:15.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 shutdown_connections 2023-10-30T12:04:15.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 -- 172.21.15.50:0/1190125562 wait complete. 2023-10-30T12:04:15.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:15.902+0000 7fc2ce864640 1 librados: shutdown 2023-10-30T12:04:15.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:04:15.995 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:04:15.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:04:16.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.102+0000 7fee193d0640 10 monclient: get_monmap_and_config 2023-10-30T12:04:16.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.102+0000 7fee193d0640 10 monclient: build_initial_monmap 2023-10-30T12:04:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.102+0000 7fee193d0640 10 monclient: monmap: 2023-10-30T12:04:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:16.104025+0000 2023-10-30T12:04:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:16.104025+0000 2023-10-30T12:04:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:16.107 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:16.107 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:16.107 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:16.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.103+0000 7fee193d0640 1 Processor -- start 2023-10-30T12:04:16.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.103+0000 7fee193d0640 1 -- start start 2023-10-30T12:04:16.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.103+0000 7fee193d0640 10 monclient: init 2023-10-30T12:04:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T12:04:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fee14130290 0x7fee14070fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): picked mon.noname-c con 0x7fee14130290 addr v2:172.21.15.118:3300/0 2023-10-30T12:04:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fee14130670 0x7fee140a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): picked mon.noname-a con 0x7fee14130670 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130ab0 0x7fee140a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): picked mon.noname-b con 0x7fee14130ab0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:16.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee140714e0 con 0x7fee14130670 2023-10-30T12:04:16.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:16.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee140702e0 con 0x7fee14130290 2023-10-30T12:04:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee14070000 con 0x7fee14130ab0 2023-10-30T12:04:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee193d0640 10 monclient(hunting): authenticate will time out at 1101.287176s 2023-10-30T12:04:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fee14130670 0x7fee140a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130ab0 0x7fee140a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130ab0 0x7fee140a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48852/0 (socket says 172.21.15.50:48852) 2023-10-30T12:04:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fee14130670 0x7fee140a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59570/0 (socket says 172.21.15.50:59570) 2023-10-30T12:04:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 1 -- 172.21.15.50:0/1647569337 learned_addr learned my addr 172.21.15.50:0/1647569337 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 10 monclient(hunting): get_auth_request con 0x7fee14130ab0 auth_method 0 2023-10-30T12:04:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): get_auth_request con 0x7fee14130670 auth_method 0 2023-10-30T12:04:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:16.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:16.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:16.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee12ffd640 1 --2- 172.21.15.50:0/1647569337 >> v2:172.21.15.118:3300/0 conn(0x7fee14130290 0x7fee14070fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee12ffd640 10 monclient(hunting): get_auth_request con 0x7fee14130290 auth_method 0 2023-10-30T12:04:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee12ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee12ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient(hunting): handle_auth_done global_id 74136 payload 274 2023-10-30T12:04:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 -- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130ab0 msgr2=0x7fee140a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 --2- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130ab0 0x7fee140a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 -- 172.21.15.50:0/1647569337 >> v2:172.21.15.118:3300/0 conn(0x7fee14130290 msgr2=0x7fee14070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 --2- 172.21.15.50:0/1647569337 >> v2:172.21.15.118:3300/0 conn(0x7fee14130290 0x7fee14070fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient: _finish_hunting 0 2023-10-30T12:04:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 monclient: found mon.noname-a 2023-10-30T12:04:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:16.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee127fc640 1 -- 172.21.15.50:0/1647569337 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1406f2d0 con 0x7fee14130670 2023-10-30T12:04:16.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee137fe640 1 --2- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130ab0 0x7fee140a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:16.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.104+0000 7fee12ffd640 1 --2- 172.21.15.50:0/1647569337 >> v2:172.21.15.118:3300/0 conn(0x7fee14130290 0x7fee14070fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:04:16.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee127fc640 1 --2- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130670 0x7fee140a46c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fee1406ea40 tx=0x7fee0802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee280dc3d6a30d15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 1 -- 172.21.15.50:0/1647569337 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08030710 con 0x7fee14130670 2023-10-30T12:04:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: dump: 2023-10-30T12:04:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:16.130 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:16.130 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:16.130 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:16.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: _finish_auth 0 2023-10-30T12:04:16.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:04:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee193d0640 5 monclient: authenticate success, global_id 74136 2023-10-30T12:04:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee193d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 1 -- 172.21.15.50:0/1647569337 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0802e050 con 0x7fee14130670 2023-10-30T12:04:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee193d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee193d0640 4 set_mon_vals no callback set 2023-10-30T12:04:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 1 -- 172.21.15.50:0/1647569337 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08030a90 con 0x7fee14130670 2023-10-30T12:04:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.105+0000 7fee11ffb640 10 monclient: dump: 2023-10-30T12:04:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:16.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 10 monclient: shutdown 2023-10-30T12:04:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 -- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130670 msgr2=0x7fee140a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 --2- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130670 0x7fee140a46c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fee1406ea40 tx=0x7fee0802b710 comp rx=0 tx=0).stop 2023-10-30T12:04:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 -- 172.21.15.50:0/1647569337 shutdown_connections 2023-10-30T12:04:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 --2- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130ab0 0x7fee140a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 --2- 172.21.15.50:0/1647569337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130670 0x7fee140a46c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 --2- 172.21.15.50:0/1647569337 >> v2:172.21.15.118:3300/0 conn(0x7fee14130290 0x7fee14070fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 -- 172.21.15.50:0/1647569337 >> 172.21.15.50:0/1647569337 conn(0x7fee140a22b0 msgr2=0x7fee140a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 -- 172.21.15.50:0/1647569337 shutdown_connections 2023-10-30T12:04:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 -- 172.21.15.50:0/1647569337 wait complete. 2023-10-30T12:04:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 10 monclient: build_initial_monmap 2023-10-30T12:04:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 10 monclient: monmap: 2023-10-30T12:04:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:16.140 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 librados: starting msgr at 2023-10-30T12:04:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.106+0000 7fee193d0640 1 librados: starting objecter 2023-10-30T12:04:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 Processor -- start 2023-10-30T12:04:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 -- start start 2023-10-30T12:04:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 librados: setting wanted keys 2023-10-30T12:04:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 librados: calling monclient init 2023-10-30T12:04:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient: init 2023-10-30T12:04:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:04:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee14130290 0x7fee141ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): picked mon.noname-b con 0x7fee14130290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130670 0x7fee141ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): picked mon.noname-c con 0x7fee14130670 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130ab0 0x7fee141cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): picked mon.noname-a con 0x7fee14130ab0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1412e8f0 con 0x7fee14130ab0 2023-10-30T12:04:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1406bd50 con 0x7fee14130290 2023-10-30T12:04:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1406efa0 con 0x7fee14130670 2023-10-30T12:04:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee193d0640 10 monclient(hunting): authenticate will time out at 1101.290651s 2023-10-30T12:04:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130670 0x7fee141ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130670 0x7fee141ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48858/0 (socket says 172.21.15.50:48858) 2023-10-30T12:04:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130ab0 0x7fee141cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 1 -- 172.21.15.50:0/4127218364 learned_addr learned my addr 172.21.15.50:0/4127218364 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 10 monclient(hunting): get_auth_request con 0x7fee14130670 auth_method 0 2023-10-30T12:04:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 10 monclient(hunting): get_auth_request con 0x7fee14130ab0 auth_method 0 2023-10-30T12:04:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.107+0000 7fee12ffd640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee14130290 0x7fee141ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee12ffd640 10 monclient(hunting): get_auth_request con 0x7fee14130290 auth_method 0 2023-10-30T12:04:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee12ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee12ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 10 monclient(hunting): handle_auth_done global_id 44537 payload 824 2023-10-30T12:04:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 1 -- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee14130290 msgr2=0x7fee141ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee14130290 0x7fee141ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 1 -- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130ab0 msgr2=0x7fee141cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130ab0 0x7fee141cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 10 monclient: _finish_hunting 0 2023-10-30T12:04:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 1 monclient: found mon.noname-c 2023-10-30T12:04:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 1 -- 172.21.15.50:0/4127218364 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04009b30 con 0x7fee14130670 2023-10-30T12:04:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee137fe640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130ab0 0x7fee141cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee127fc640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130670 0x7fee141ca7b0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c00 tx=0x7fee080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 client.?.objecter ms_handle_connect 0x7fee14130670 2023-10-30T12:04:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 1 -- 172.21.15.50:0/4127218364 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08032690 con 0x7fee14130670 2023-10-30T12:04:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: dump: 2023-10-30T12:04:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:16.153 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:16.153 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:16.155 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:16.155 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:16.155 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:16.156 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:16.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:04:16.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:04:16.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:16.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 5 monclient: authenticate success, global_id 44537 2023-10-30T12:04:16.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 10 monclient: _renew_subs 2023-10-30T12:04:16.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 1 -- 172.21.15.50:0/4127218364 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee08032830 con 0x7fee14130670 2023-10-30T12:04:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee08009400 con 0x7fee14130670 2023-10-30T12:04:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 1 -- 172.21.15.50:0/4127218364 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08032be0 con 0x7fee14130670 2023-10-30T12:04:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fedfb7fe640 10 monclient: dump: 2023-10-30T12:04:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:16.161 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 10 client.44537.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee117fa640 4 set_mon_vals no callback set 2023-10-30T12:04:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 10 monclient: _renew_subs 2023-10-30T12:04:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1406ba60 con 0x7fee14130670 2023-10-30T12:04:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.108+0000 7fee193d0640 1 librados: init done 2023-10-30T12:04:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fedfb7fe640 1 -- 172.21.15.50:0/4127218364 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fee08039070 con 0x7fee14130670 2023-10-30T12:04:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fee193d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fee193d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fee193d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1404e160 con 0x7fee14130670 2023-10-30T12:04:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fedfb7fe640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee000392f0 0x7fee0003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fedfb7fe640 1 -- 172.21.15.50:0/4127218364 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 7320+0+0 (secure 0 0 0) 0x7fee0806c8e0 con 0x7fee14130670 2023-10-30T12:04:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fedfb7fe640 10 client.44537.objecter ms_dispatch 0x7fee14000b90 osd_map(112..112 src has 1..112) v4 2023-10-30T12:04:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fedfb7fe640 3 client.44537.objecter handle_osd_map got epochs [112,112] > 0 2023-10-30T12:04:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fedfb7fe640 3 client.44537.objecter handle_osd_map decoding full epoch 112 2023-10-30T12:04:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fedfb7fe640 20 client.44537.objecter dump_active .. 0 homeless 2023-10-30T12:04:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fee12ffd640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee000392f0 0x7fee0003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.109+0000 7fee12ffd640 10 monclient: get_auth_request con 0x7fee000392f0 auth_method 0 2023-10-30T12:04:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.110+0000 7fedfb7fe640 10 client.44537.objecter ms_handle_connect 0x7fee000392f0 2023-10-30T12:04:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.110+0000 7fee12ffd640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee000392f0 0x7fee0003b7b0 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 crypto rx=0x7fee0c006fd0 tx=0x7fee0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.113+0000 7fedfb7fe640 1 -- 172.21.15.50:0/4127218364 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fee08032d40 con 0x7fee14130670 2023-10-30T12:04:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.113+0000 7fedfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.113+0000 7fedfb7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.358+0000 7fee193d0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:04:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.358+0000 7fee193d0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:04:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.358+0000 7fee193d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.358+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fee140745c0 con 0x7fee14130670 2023-10-30T12:04:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fedfb7fe640 1 -- 172.21.15.50:0/4127218364 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fee08037030 con 0x7fee14130670 2023-10-30T12:04:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fedfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:04:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fedfb7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:16.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=3061272641,config=3140514009,health=3737324895,kv=2905431493,logm=2164482144}) 2023-10-30T12:04:16.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=2464499460,config=3140514009,health=3737324895,kv=2905431493,logm=1517867058}) 2023-10-30T12:04:16.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=3061272641,config=3140514009,health=3737324895,kv=2905431493,logm=2164482144}) 2023-10-30T12:04:16.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.360+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=2464499460,config=3140514009,health=2542184495,kv=2905431493,logm=3662309619}) 2023-10-30T12:04:16.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.361+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.361+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555783239}) 2023-10-30T12:04:16.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.361+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2721943677}) 2023-10-30T12:04:16.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.361+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.361+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555783239}) 2023-10-30T12:04:16.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.361+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844747794}) 2023-10-30T12:04:16.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961034917}) 2023-10-30T12:04:16.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4216608026}) 2023-10-30T12:04:16.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 10 librados: watch_flush enter 2023-10-30T12:04:16.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961034917}) 2023-10-30T12:04:16.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850936784}) 2023-10-30T12:04:16.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 10 librados: watch_flush exit 2023-10-30T12:04:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 20 client.44537.objecter shutdown clearing up homeless session... 2023-10-30T12:04:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 10 client.44537.objecter successfully canceled tick 2023-10-30T12:04:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee000392f0 msgr2=0x7fee0003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee000392f0 0x7fee0003b7b0 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 crypto rx=0x7fee0c006fd0 tx=0x7fee0c008040 comp rx=0 tx=0).stop 2023-10-30T12:04:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 10 monclient: shutdown 2023-10-30T12:04:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130670 msgr2=0x7fee141ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130670 0x7fee141ca7b0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c00 tx=0x7fee080046c0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 shutdown_connections 2023-10-30T12:04:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee000392f0 0x7fee0003b7b0 unknown :-1 s=CLOSED pgs=3913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee14130ab0 0x7fee141cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee14130670 0x7fee141ca7b0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 --2- 172.21.15.50:0/4127218364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee14130290 0x7fee141ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 >> 172.21.15.50:0/4127218364 conn(0x7fee140a22b0 msgr2=0x7fee140a0120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 shutdown_connections 2023-10-30T12:04:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 -- 172.21.15.50:0/4127218364 wait complete. 2023-10-30T12:04:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:16.362+0000 7fee193d0640 1 librados: shutdown 2023-10-30T12:04:16.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.363+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.363+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205453818}) 2023-10-30T12:04:16.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.363+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10224821}) 2023-10-30T12:04:16.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.363+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.363+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205453818}) 2023-10-30T12:04:16.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.363+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=129645992}) 2023-10-30T12:04:16.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.364+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.364+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496832658}) 2023-10-30T12:04:16.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.364+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794393622}) 2023-10-30T12:04:16.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.364+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.364+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496832658}) 2023-10-30T12:04:16.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.364+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567586686}) 2023-10-30T12:04:16.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.365+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.365+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108512157}) 2023-10-30T12:04:16.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.365+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855575994}) 2023-10-30T12:04:16.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.365+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.365+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108512157}) 2023-10-30T12:04:16.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.365+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2034289362}) 2023-10-30T12:04:16.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.366+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.366+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693280208}) 2023-10-30T12:04:16.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.366+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1852647335}) 2023-10-30T12:04:16.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.366+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.366+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693280208}) 2023-10-30T12:04:16.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.366+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1451394956}) 2023-10-30T12:04:16.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651816347}) 2023-10-30T12:04:16.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193212964}) 2023-10-30T12:04:16.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651816347}) 2023-10-30T12:04:16.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578415229}) 2023-10-30T12:04:16.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350959023}) 2023-10-30T12:04:16.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1069878313}) 2023-10-30T12:04:16.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350959023}) 2023-10-30T12:04:16.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.367+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780529814}) 2023-10-30T12:04:16.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.368+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.368+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067759786}) 2023-10-30T12:04:16.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.368+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1352791371}) 2023-10-30T12:04:16.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.368+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.368+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067759786}) 2023-10-30T12:04:16.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.368+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339413932}) 2023-10-30T12:04:16.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.369+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.369+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84014754}) 2023-10-30T12:04:16.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.369+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123011892}) 2023-10-30T12:04:16.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.369+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.369+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84014754}) 2023-10-30T12:04:16.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.369+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3533602643}) 2023-10-30T12:04:16.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358472924}) 2023-10-30T12:04:16.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412513314}) 2023-10-30T12:04:16.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358472924}) 2023-10-30T12:04:16.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3701453124}) 2023-10-30T12:04:16.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814082629}) 2023-10-30T12:04:16.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=440625113}) 2023-10-30T12:04:16.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.370+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814082629}) 2023-10-30T12:04:16.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4242030338}) 2023-10-30T12:04:16.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076060079}) 2023-10-30T12:04:16.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2913568874}) 2023-10-30T12:04:16.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076060079}) 2023-10-30T12:04:16.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.371+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=101554751}) 2023-10-30T12:04:16.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.372+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.372+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2164128183}) 2023-10-30T12:04:16.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.372+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=329653177}) 2023-10-30T12:04:16.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.372+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.372+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2164128183}) 2023-10-30T12:04:16.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.372+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434767042}) 2023-10-30T12:04:16.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.373+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.373+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338506531}) 2023-10-30T12:04:16.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.373+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488518905}) 2023-10-30T12:04:16.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.373+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.373+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338506531}) 2023-10-30T12:04:16.394 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:04:16.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.373+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098181128}) 2023-10-30T12:04:16.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.374+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.374+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469596689}) 2023-10-30T12:04:16.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.374+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987989928}) 2023-10-30T12:04:16.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.374+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.374+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469596689}) 2023-10-30T12:04:16.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.374+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1883583880}) 2023-10-30T12:04:16.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.375+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.375+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1} crc {logm=4048466778,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=701405386,mon_config_key=715115367,monmap=4204254258,osd_metadata=2479346626,osd_pg_creating=195125472}) 2023-10-30T12:04:16.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.375+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=3610403944,mdsmap=3984425268,mgr=2906314846,mgrstat=3975542098,mon_config_key=3965234311,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=195125472,osd_snap=137038486}) 2023-10-30T12:04:16.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.375+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.375+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1} crc {logm=4048466778,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=701405386,mon_config_key=715115367,monmap=4204254258,osd_metadata=2479346626,osd_pg_creating=195125472}) 2023-10-30T12:04:16.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.375+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1} crc {logm=4179808800,mdsmap=3984425268,mgr=1715906815,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1063860357,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=195125472}) 2023-10-30T12:04:16.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.376+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.376+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=1149614667,osdmap=3602018502}) 2023-10-30T12:04:16.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.376+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=2221919638,osdmap=2886150084}) 2023-10-30T12:04:16.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.376+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.376+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=1149614667,osdmap=3602018502}) 2023-10-30T12:04:16.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.376+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=2816929348,osdmap=4103887676}) 2023-10-30T12:04:16.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.381+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.381+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=97} crc {osdmap=1081487062}) 2023-10-30T12:04:16.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.381+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=93} crc {osdmap=2113133281}) 2023-10-30T12:04:16.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.381+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:04:16.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.381+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=97} crc {osdmap=1081487062}) 2023-10-30T12:04:16.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:04:16.381+0000 7fda8c1e0640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=3097812930}) 2023-10-30T12:04:17.006 INFO:tasks.workunit.client.0.smithi118.stdout:bRadosSnapshotsSelfManagedECPP.SnapPP 2023-10-30T12:04:17.006 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4049 ms) 2023-10-30T12:04:17.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-10-30T12:04:17.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3977 ms) 2023-10-30T12:04:17.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-10-30T12:04:17.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3983 ms) 2023-10-30T12:04:17.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12009 ms total) 2023-10-30T12:04:17.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: 2023-10-30T12:04:17.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-10-30T12:04:17.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (142690 ms total) 2023-10-30T12:04:17.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-10-30T12:04:17.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-10-30T12:04:17.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-10-30T12:04:17.395 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:04:17.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:04:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.507+0000 7f45fd0d7640 10 monclient: get_monmap_and_config 2023-10-30T12:04:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.507+0000 7f45fd0d7640 10 monclient: build_initial_monmap 2023-10-30T12:04:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.507+0000 7f45fd0d7640 10 monclient: monmap: 2023-10-30T12:04:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:17.508132+0000 2023-10-30T12:04:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:17.508132+0000 2023-10-30T12:04:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:17.511 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.507+0000 7f45fd0d7640 1 Processor -- start 2023-10-30T12:04:17.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.507+0000 7f45fd0d7640 1 -- start start 2023-10-30T12:04:17.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.507+0000 7f45fd0d7640 10 monclient: init 2023-10-30T12:04:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-10-30T12:04:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45f80a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8099a40 0x7f45f813aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient(hunting): picked mon.noname-b con 0x7f45f8099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45f8132db0 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f80a4f40 con 0x7f45f80a4b60 2023-10-30T12:04:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f80a4150 con 0x7f45f8132db0 2023-10-30T12:04:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80a3d70 con 0x7f45f8099a40 2023-10-30T12:04:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45fd0d7640 10 monclient(hunting): authenticate will time out at 1102.691284s 2023-10-30T12:04:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8099a40 0x7f45f813aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f6d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f45f80a4b60 0x7f45f81326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34632/0 (socket says 172.21.15.50:34632) 2023-10-30T12:04:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f6d9d640 1 -- 172.21.15.50:0/1783336607 learned_addr learned my addr 172.21.15.50:0/1783336607 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:04:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): get_auth_request con 0x7f45f8099a40 auth_method 0 2023-10-30T12:04:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient(hunting): handle_auth_done global_id 44540 payload 274 2023-10-30T12:04:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 1 -- 172.21.15.50:0/1783336607 >> v1:172.21.15.118:6789/0 conn(0x7f45f8132db0 legacy=0x7f45f813cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 1 -- 172.21.15.50:0/1783336607 >> v1:172.21.15.50:6789/0 conn(0x7f45f80a4b60 legacy=0x7f45f81326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient: _finish_hunting 0 2023-10-30T12:04:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f5d9b640 1 -- 172.21.15.50:0/1783336607 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126660228 0 0) 0x7f45f80a4f40 con 0x7f45f80a4b60 2023-10-30T12:04:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 1 monclient: found mon.noname-b 2023-10-30T12:04:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f659c640 1 -- 172.21.15.50:0/1783336607 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f809a440 con 0x7f45f8099a40 2023-10-30T12:04:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.508+0000 7f45f5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:04:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f659c640 1 --2- 172.21.15.50:0/1783336607 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8099a40 0x7f45f813aa80 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f45e00097d0 tx=0x7f45e002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=160d6be03e1568ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 1 -- 172.21.15.50:0/1783336607 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e00316f0 con 0x7f45f8099a40 2023-10-30T12:04:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: dump: 2023-10-30T12:04:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.529 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:04:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:04:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 1 -- 172.21.15.50:0/1783336607 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e0031850 con 0x7f45f8099a40 2023-10-30T12:04:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45fd0d7640 5 monclient: authenticate success, global_id 44540 2023-10-30T12:04:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45fd0d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45fd0d7640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 1 -- 172.21.15.50:0/1783336607 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0031be0 con 0x7f45f8099a40 2023-10-30T12:04:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45fd0d7640 4 set_mon_vals no callback set 2023-10-30T12:04:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.509+0000 7f45f5d9b640 10 monclient: dump: 2023-10-30T12:04:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.533 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 10 monclient: shutdown 2023-10-30T12:04:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 1 -- 172.21.15.50:0/1783336607 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8099a40 msgr2=0x7f45f813aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/1783336607 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8099a40 0x7f45f813aa80 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f45e00097d0 tx=0x7f45e002b790 comp rx=0 tx=0).stop 2023-10-30T12:04:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 1 -- 172.21.15.50:0/1783336607 shutdown_connections 2023-10-30T12:04:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/1783336607 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8099a40 0x7f45f813aa80 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 1 -- 172.21.15.50:0/1783336607 >> 172.21.15.50:0/1783336607 conn(0x7f45f80a8060 msgr2=0x7f45f80a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 1 -- 172.21.15.50:0/1783336607 shutdown_connections 2023-10-30T12:04:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 1 -- 172.21.15.50:0/1783336607 wait complete. 2023-10-30T12:04:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 10 monclient: build_initial_monmap 2023-10-30T12:04:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.510+0000 7f45fd0d7640 10 monclient: monmap: 2023-10-30T12:04:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:17.537 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 1 librados: starting msgr at 2023-10-30T12:04:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 1 librados: starting objecter 2023-10-30T12:04:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 1 Processor -- start 2023-10-30T12:04:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 1 -- start start 2023-10-30T12:04:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 1 librados: setting wanted keys 2023-10-30T12:04:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 1 librados: calling monclient init 2023-10-30T12:04:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 10 monclient: init 2023-10-30T12:04:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:04:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.511+0000 7f45fd0d7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f80a4b60 0x7f45f8136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): picked mon.noname-b con 0x7f45f80a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8132db0 0x7f45f8136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): picked mon.noname-c con 0x7f45f8132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45f81372d0 0x7f45f81d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): picked mon.noname-a con 0x7f45f81372d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80a18d0 con 0x7f45f81372d0 2023-10-30T12:04:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f813d950 con 0x7f45f80a4b60 2023-10-30T12:04:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f81389d0 con 0x7f45f8132db0 2023-10-30T12:04:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient(hunting): authenticate will time out at 1102.695088s 2023-10-30T12:04:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8132db0 0x7f45f8136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45f81372d0 0x7f45f81d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45f81372d0 0x7f45f81d09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59578/0 (socket says 172.21.15.50:59578) 2023-10-30T12:04:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8132db0 0x7f45f8136d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48866/0 (socket says 172.21.15.50:48866) 2023-10-30T12:04:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 -- 172.21.15.50:0/387304081 learned_addr learned my addr 172.21.15.50:0/387304081 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): get_auth_request con 0x7f45f81372d0 auth_method 0 2023-10-30T12:04:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 10 monclient(hunting): get_auth_request con 0x7f45f8132db0 auth_method 0 2023-10-30T12:04:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f6d9d640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f80a4b60 0x7f45f8136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient(hunting): handle_auth_done global_id 74145 payload 824 2023-10-30T12:04:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 -- 172.21.15.50:0/387304081 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8132db0 msgr2=0x7f45f8136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8132db0 0x7f45f8136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 -- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f80a4b60 msgr2=0x7f45f8136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f80a4b60 0x7f45f8136850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient: _finish_hunting 0 2023-10-30T12:04:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 monclient: found mon.noname-a 2023-10-30T12:04:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 -- 172.21.15.50:0/387304081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0009400 con 0x7f45f81372d0 2023-10-30T12:04:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f6d9d640 10 monclient: get_auth_request con 0x7f45f80a4b60 auth_method 0 2023-10-30T12:04:17.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f6d9d640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f80a4b60 0x7f45f8136850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:04:17.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f759e640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45f81372d0 0x7f45f81d09f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45f000c410 tx=0x7f45f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 client.?.objecter ms_handle_connect 0x7f45f81372d0 2023-10-30T12:04:17.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:17.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 1 -- 172.21.15.50:0/387304081 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0011790 con 0x7f45f81372d0 2023-10-30T12:04:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: dump: 2023-10-30T12:04:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.551 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: _finish_auth 0 2023-10-30T12:04:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:04:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 1 -- 172.21.15.50:0/387304081 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f00118f0 con 0x7f45f81372d0 2023-10-30T12:04:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 5 monclient: authenticate success, global_id 74145 2023-10-30T12:04:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient: _renew_subs 2023-10-30T12:04:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f809a7d0 con 0x7f45f81372d0 2023-10-30T12:04:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 1 -- 172.21.15.50:0/387304081 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f0011ce0 con 0x7f45f81372d0 2023-10-30T12:04:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45ef7fe640 10 monclient: dump: 2023-10-30T12:04:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.555 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45f559a640 4 set_mon_vals no callback set 2023-10-30T12:04:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 client.74145.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient: _renew_subs 2023-10-30T12:04:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f813a030 con 0x7f45f81372d0 2023-10-30T12:04:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.512+0000 7f45fd0d7640 1 librados: init done 2023-10-30T12:04:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.513+0000 7f45ef7fe640 1 -- 172.21.15.50:0/387304081 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f45f001c460 con 0x7f45f81372d0 2023-10-30T12:04:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.513+0000 7f45ef7fe640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45d80392f0 0x7f45d803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.513+0000 7f45fd0d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.513+0000 7f45fd0d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:17.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.513+0000 7f45fd0d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:17.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.513+0000 7f45ef7fe640 1 -- 172.21.15.50:0/387304081 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 7206+0+0 (secure 0 0 0) 0x7f45f004abe0 con 0x7f45f81372d0 2023-10-30T12:04:17.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45ef7fe640 10 client.74145.objecter ms_dispatch 0x7f45f8000b90 osd_map(113..113 src has 1..113) v4 2023-10-30T12:04:17.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45ef7fe640 3 client.74145.objecter handle_osd_map got epochs [113,113] > 0 2023-10-30T12:04:17.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45ef7fe640 3 client.74145.objecter handle_osd_map decoding full epoch 113 2023-10-30T12:04:17.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.513+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f804e160 con 0x7f45f81372d0 2023-10-30T12:04:17.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45ef7fe640 20 client.74145.objecter dump_active .. 0 homeless 2023-10-30T12:04:17.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45f6d9d640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45d80392f0 0x7f45d803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45f6d9d640 10 monclient: get_auth_request con 0x7f45d80392f0 auth_method 0 2023-10-30T12:04:17.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45ef7fe640 10 client.74145.objecter ms_handle_connect 0x7f45d80392f0 2023-10-30T12:04:17.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.514+0000 7f45f6d9d640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45d80392f0 0x7f45d803b7b0 secure :-1 s=READY pgs=3914 cs=0 l=1 rev1=1 crypto rx=0x7f45e80012e0 tx=0x7f45e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.518+0000 7f45ef7fe640 1 -- 172.21.15.50:0/387304081 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f45f0011e40 con 0x7f45f81372d0 2023-10-30T12:04:17.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.518+0000 7f45ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:17.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.518+0000 7f45ef7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.766+0000 7f45fd0d7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:04:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.766+0000 7f45fd0d7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.766+0000 7f45fd0d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.766+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f45f80745c0 con 0x7f45f81372d0 2023-10-30T12:04:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.767+0000 7f45ef7fe640 1 -- 172.21.15.50:0/387304081 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f45f001a360 con 0x7f45f81372d0 2023-10-30T12:04:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.767+0000 7f45ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.767+0000 7f45ef7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:17.770 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:04:17.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 10 librados: watch_flush enter 2023-10-30T12:04:17.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 10 librados: watch_flush exit 2023-10-30T12:04:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 20 client.74145.objecter shutdown clearing up homeless session... 2023-10-30T12:04:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 10 client.74145.objecter successfully canceled tick 2023-10-30T12:04:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45d80392f0 msgr2=0x7f45d803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45d80392f0 0x7f45d803b7b0 secure :-1 s=READY pgs=3914 cs=0 l=1 rev1=1 crypto rx=0x7f45e80012e0 tx=0x7f45e8009040 comp rx=0 tx=0).stop 2023-10-30T12:04:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 10 monclient: shutdown 2023-10-30T12:04:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45f81372d0 msgr2=0x7f45f81d09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45f81372d0 0x7f45f81d09f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45f000c410 tx=0x7f45f000c8d0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 shutdown_connections 2023-10-30T12:04:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45d80392f0 0x7f45d803b7b0 unknown :-1 s=CLOSED pgs=3914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45f81372d0 0x7f45f81d09f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45f8132db0 0x7f45f8136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 --2- 172.21.15.50:0/387304081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f80a4b60 0x7f45f8136850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 >> 172.21.15.50:0/387304081 conn(0x7f45f80a8060 msgr2=0x7f45f8135d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 shutdown_connections 2023-10-30T12:04:17.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 -- 172.21.15.50:0/387304081 wait complete. 2023-10-30T12:04:17.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.769+0000 7f45fd0d7640 1 librados: shutdown 2023-10-30T12:04:17.781 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:04:17.781 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:04:17.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:04:17.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:04:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.885+0000 7f4078fd0640 10 monclient: get_monmap_and_config 2023-10-30T12:04:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.885+0000 7f4078fd0640 10 monclient: build_initial_monmap 2023-10-30T12:04:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.885+0000 7f4078fd0640 10 monclient: monmap: 2023-10-30T12:04:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:17.886889+0000 2023-10-30T12:04:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:17.886889+0000 2023-10-30T12:04:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:17.890 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.886+0000 7f4078fd0640 1 Processor -- start 2023-10-30T12:04:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.886+0000 7f4078fd0640 1 -- start start 2023-10-30T12:04:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.886+0000 7f4078fd0640 10 monclient: init 2023-10-30T12:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40740a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f407406afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f407406b640 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f407413b070 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407406bde0 con 0x7f407406b640 2023-10-30T12:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407406b3a0 con 0x7f407413b070 2023-10-30T12:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407409e540 con 0x7f407406afc0 2023-10-30T12:04:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4078fd0640 10 monclient(hunting): authenticate will time out at 1103.070195s 2023-10-30T12:04:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40740a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4071d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f407406b640 0x7f40740a21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39954/0 (socket says 172.21.15.50:39954) 2023-10-30T12:04:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f4071d9b640 1 -- 172.21.15.50:0/2443715273 learned_addr learned my addr 172.21.15.50:0/2443715273 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:04:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): get_auth_request con 0x7f407406afc0 auth_method 0 2023-10-30T12:04:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient(hunting): handle_auth_done global_id 44546 payload 274 2023-10-30T12:04:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 1 -- 172.21.15.50:0/2443715273 >> v1:172.21.15.118:6789/0 conn(0x7f407413b070 legacy=0x7f407413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 1 -- 172.21.15.50:0/2443715273 >> v1:172.21.15.50:6789/0 conn(0x7f407406b640 legacy=0x7f40740a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient: _finish_hunting 0 2023-10-30T12:04:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 1 monclient: found mon.noname-b 2023-10-30T12:04:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407259c640 1 -- 172.21.15.50:0/2443715273 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407409fa10 con 0x7f407406afc0 2023-10-30T12:04:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407159a640 1 -- 172.21.15.50:0/2443715273 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551200935 0 0) 0x7f407406bde0 con 0x7f407406b640 2023-10-30T12:04:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.887+0000 7f407159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:04:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407259c640 1 --2- 172.21.15.50:0/2443715273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40740a1c90 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f407406b530 tx=0x7f406c02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd75009623230520 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 1 -- 172.21.15.50:0/2443715273 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c02cb20 con 0x7f407406afc0 2023-10-30T12:04:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: dump: 2023-10-30T12:04:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.906 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: _finish_auth 0 2023-10-30T12:04:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: _check_auth_tickets 2023-10-30T12:04:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 5 monclient: authenticate success, global_id 44546 2023-10-30T12:04:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 1 -- 172.21.15.50:0/2443715273 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c02cc80 con 0x7f407406afc0 2023-10-30T12:04:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 1 -- 172.21.15.50:0/2443715273 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c034440 con 0x7f407406afc0 2023-10-30T12:04:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 4 set_mon_vals no callback set 2023-10-30T12:04:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f407159a640 10 monclient: dump: 2023-10-30T12:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.911 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 10 monclient: shutdown 2023-10-30T12:04:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 1 -- 172.21.15.50:0/2443715273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 msgr2=0x7f40740a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.888+0000 7f4078fd0640 1 --2- 172.21.15.50:0/2443715273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40740a1c90 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f407406b530 tx=0x7f406c02bd70 comp rx=0 tx=0).stop 2023-10-30T12:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 1 -- 172.21.15.50:0/2443715273 shutdown_connections 2023-10-30T12:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 1 --2- 172.21.15.50:0/2443715273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40740a1c90 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 1 -- 172.21.15.50:0/2443715273 >> 172.21.15.50:0/2443715273 conn(0x7f40740a6200 msgr2=0x7f40740a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 1 -- 172.21.15.50:0/2443715273 shutdown_connections 2023-10-30T12:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 1 -- 172.21.15.50:0/2443715273 wait complete. 2023-10-30T12:04:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 10 monclient: build_initial_monmap 2023-10-30T12:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 10 monclient: monmap: 2023-10-30T12:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:17.914 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 1 librados: starting msgr at 2023-10-30T12:04:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.889+0000 7f4078fd0640 1 librados: starting objecter 2023-10-30T12:04:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 Processor -- start 2023-10-30T12:04:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 -- start start 2023-10-30T12:04:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 librados: setting wanted keys 2023-10-30T12:04:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 librados: calling monclient init 2023-10-30T12:04:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient: init 2023-10-30T12:04:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:04:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40741ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f407406afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407406b640 0x7f40741ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f407406b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f407413b070 0x7f407411c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f407413b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407411ea20 con 0x7f407406b640 2023-10-30T12:04:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40741200f0 con 0x7f407413b070 2023-10-30T12:04:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4074120340 con 0x7f407406afc0 2023-10-30T12:04:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4078fd0640 10 monclient(hunting): authenticate will time out at 1103.073568s 2023-10-30T12:04:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40741ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407406b640 0x7f40741ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40741ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:54242/0 (socket says 172.21.15.50:54242) 2023-10-30T12:04:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 -- 172.21.15.50:0/620116537 learned_addr learned my addr 172.21.15.50:0/620116537 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): get_auth_request con 0x7f407406afc0 auth_method 0 2023-10-30T12:04:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407406b640 0x7f40741ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59994/0 (socket says 172.21.15.50:59994) 2023-10-30T12:04:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 10 monclient(hunting): get_auth_request con 0x7f407406b640 auth_method 0 2023-10-30T12:04:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f4072d9d640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f407413b070 0x7f407411c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient(hunting): handle_auth_done global_id 44549 payload 824 2023-10-30T12:04:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f407413b070 msgr2=0x7f407411c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f407413b070 0x7f407411c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407406b640 msgr2=0x7f40741ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407406b640 0x7f40741ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient: _finish_hunting 0 2023-10-30T12:04:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 monclient: found mon.noname-c 2023-10-30T12:04:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.890+0000 7f407259c640 1 -- 172.21.15.50:0/620116537 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c009a70 con 0x7f407406afc0 2023-10-30T12:04:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4071d9b640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407406b640 0x7f40741ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4072d9d640 10 monclient: get_auth_request con 0x7f407413b070 auth_method 0 2023-10-30T12:04:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4072d9d640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f407413b070 0x7f407411c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:04:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f407259c640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40741ca470 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f406c009e80 tx=0x7f406c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 client.?.objecter ms_handle_connect 0x7f407406afc0 2023-10-30T12:04:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c003990 con 0x7f407406afc0 2023-10-30T12:04:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: dump: 2023-10-30T12:04:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.928 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: _finish_auth 0 2023-10-30T12:04:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: _check_auth_tickets 2023-10-30T12:04:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c003af0 con 0x7f407406afc0 2023-10-30T12:04:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 5 monclient: authenticate success, global_id 44549 2023-10-30T12:04:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 10 monclient: _renew_subs 2023-10-30T12:04:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4074134e20 con 0x7f407406afc0 2023-10-30T12:04:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c003e80 con 0x7f407406afc0 2023-10-30T12:04:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4070d99640 4 set_mon_vals no callback set 2023-10-30T12:04:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f406affd640 10 monclient: dump: 2023-10-30T12:04:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.932 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 10 client.44549.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 10 monclient: _renew_subs 2023-10-30T12:04:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407411df70 con 0x7f407406afc0 2023-10-30T12:04:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.891+0000 7f4078fd0640 1 librados: init done 2023-10-30T12:04:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4078fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4078fd0640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411c860 0x7f407411cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4078fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4074073ff0 con 0x7f407411c860 2023-10-30T12:04:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f406c003fe0 con 0x7f407406afc0 2023-10-30T12:04:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411c860 0x7f407411cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient: get_auth_request con 0x7f407411c860 auth_method 0 2023-10-30T12:04:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40540392f0 0x7f405403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 10 monclient(hunting): handle_auth_done global_id 74154 payload 824 2023-10-30T12:04:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 7206+0+0 (secure 0 0 0) 0x7f406c06dbe0 con 0x7f407406afc0 2023-10-30T12:04:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 10 client.44549.objecter ms_dispatch 0x7f4074000b90 osd_map(113..113 src has 1..113) v4 2023-10-30T12:04:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 3 client.44549.objecter handle_osd_map got epochs [113,113] > 0 2023-10-30T12:04:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 1 -- 172.21.15.50:0/620116537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f407409f0b0 con 0x7f407411c860 2023-10-30T12:04:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 3 client.44549.objecter handle_osd_map decoding full epoch 113 2023-10-30T12:04:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4072d9d640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40540392f0 0x7f405403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 20 client.44549.objecter dump_active .. 0 homeless 2023-10-30T12:04:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4072d9d640 10 monclient: get_auth_request con 0x7f40540392f0 auth_method 0 2023-10-30T12:04:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 10 client.44549.objecter ms_handle_connect 0x7f407411c860 2023-10-30T12:04:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 10 client.44549.objecter resend_mon_ops 2023-10-30T12:04:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f4071d9b640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411c860 0x7f407411cc40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f407411db70 tx=0x7f405c001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c00fa70 con 0x7f407411c860 2023-10-30T12:04:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.892+0000 7f406affd640 10 monclient: dump: 2023-10-30T12:04:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.940 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 10 client.44549.objecter ms_handle_connect 0x7f40540392f0 2023-10-30T12:04:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f4072d9d640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40540392f0 0x7f405403b7b0 secure :-1 s=READY pgs=3915 cs=0 l=1 rev1=1 crypto rx=0x7f407411f670 tx=0x7f4064009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f405c00fbd0 con 0x7f407411c860 2023-10-30T12:04:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411c860 msgr2=0x7f407411cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411c860 0x7f407411cc40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f407411db70 tx=0x7f405c001d40 comp rx=0 tx=0).stop 2023-10-30T12:04:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411c860 msgr2=0x7f407411cc40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.893+0000 7f406affd640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411c860 0x7f407411cc40 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f407411db70 tx=0x7f405c001d40 comp rx=0 tx=0).stop 2023-10-30T12:04:17.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4078fd0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:17.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4078fd0640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411d180 0x7f407406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:17.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4078fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:17.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407409df80 con 0x7f407411d180 2023-10-30T12:04:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411d180 0x7f407406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient: get_auth_request con 0x7f407411d180 auth_method 0 2023-10-30T12:04:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 10 monclient(hunting): handle_auth_done global_id 74157 payload 824 2023-10-30T12:04:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.899+0000 7f4071d9b640 1 -- 172.21.15.50:0/620116537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f407406aa80 con 0x7f407411d180 2023-10-30T12:04:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 10 client.44549.objecter ms_handle_connect 0x7f407411d180 2023-10-30T12:04:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 10 client.44549.objecter resend_mon_ops 2023-10-30T12:04:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4071d9b640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411d180 0x7f407406e2e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f407411db70 tx=0x7f405c001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c023070 con 0x7f407411d180 2023-10-30T12:04:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 10 monclient: dump: 2023-10-30T12:04:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:17.949 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:17.949 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:17.949 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:17.949 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:17.949 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:17.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f405c0136e0 con 0x7f407411d180 2023-10-30T12:04:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411d180 msgr2=0x7f407406e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411d180 0x7f407406e2e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f407411db70 tx=0x7f405c001080 comp rx=0 tx=0).stop 2023-10-30T12:04:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411d180 msgr2=0x7f407406e2e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:17.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f406affd640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f407411d180 0x7f407406e2e0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f407411db70 tx=0x7f405c001080 comp rx=0 tx=0).stop 2023-10-30T12:04:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:17.951 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:17.951 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:17.952 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:04:17.952 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 40, 2023-10-30T12:04:17.952 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:17.952 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:04:17.952 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:17.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:17.953 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:17.953 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:04:17.953 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:17.953 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:17.953 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:17.954 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:17.954 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:17.954 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:17.954 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:17.954 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:17.955 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:17.955 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:17.955 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:17.955 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:17.955 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:17.955 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:17.956 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:17.956 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:17.956 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:17.956 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:17.956 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:17.956 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:17.957 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:17.957 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:17.957 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:17.957 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:17.957 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:17.958 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:17.958 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:17.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.958 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:17.958 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:17.959 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:17.959 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:17.959 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:17.959 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:17.959 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:17.959 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:17.960 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:17.960 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:17.960 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:17.960 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:17.960 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:17.961 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:17.961 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:17.961 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:17.961 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:17.961 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:17.961 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:17.962 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:17.962 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:17.962 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:17.962 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:17.962 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:17.963 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:17.963 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:17.963 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:17.963 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:17.963 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:17.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.964 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:17.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.964 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:17.964 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:17.964 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:17.964 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:17.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.965 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:17.965 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:17.965 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:17.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.966 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:17.966 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:17.966 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:17.966 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:17.966 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:17.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.967 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:17.967 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:17.967 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:17.967 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:17.968 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:17.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.968 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:17.968 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:17.968 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:17.969 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:17.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.969 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:17.969 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:17.969 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:17.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.970 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:17.970 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:17.970 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:17.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:17.971 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:17.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.971 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:17.971 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:17.971 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:17.980 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:17.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 10 librados: watch_flush enter 2023-10-30T12:04:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 10 librados: watch_flush exit 2023-10-30T12:04:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 20 client.44549.objecter shutdown clearing up homeless session... 2023-10-30T12:04:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 10 client.44549.objecter successfully canceled tick 2023-10-30T12:04:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40540392f0 msgr2=0x7f405403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40540392f0 0x7f405403b7b0 secure :-1 s=READY pgs=3915 cs=0 l=1 rev1=1 crypto rx=0x7f407411f670 tx=0x7f4064009040 comp rx=0 tx=0).stop 2023-10-30T12:04:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 10 monclient: shutdown 2023-10-30T12:04:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 msgr2=0x7f40741ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 --2- 172.21.15.50:0/620116537 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f407406afc0 0x7f40741ca470 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f406c009e80 tx=0x7f406c004a40 comp rx=0 tx=0).stop 2023-10-30T12:04:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4072d9d640 1 -- 172.21.15.50:0/620116537 reap_dead start 2023-10-30T12:04:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4072d9d640 1 -- 172.21.15.50:0/620116537 reap_dead start 2023-10-30T12:04:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 shutdown_connections 2023-10-30T12:04:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 >> 172.21.15.50:0/620116537 conn(0x7f40740a6200 msgr2=0x7f40740a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 shutdown_connections 2023-10-30T12:04:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 -- 172.21.15.50:0/620116537 wait complete. 2023-10-30T12:04:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:17.900+0000 7f4078fd0640 1 librados: shutdown 2023-10-30T12:04:17.985 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:04:17.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:04:17.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:04:18.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.100+0000 7fcabd290640 10 monclient: get_monmap_and_config 2023-10-30T12:04:18.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.100+0000 7fcabd290640 10 monclient: build_initial_monmap 2023-10-30T12:04:18.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.100+0000 7fcabd290640 10 monclient: monmap: 2023-10-30T12:04:18.103 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:18.103 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.103 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:18.101379+0000 2023-10-30T12:04:18.103 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:18.101379+0000 2023-10-30T12:04:18.103 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:18.104 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.104 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:18.104 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:18.104 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:18.104 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:18.105 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.100+0000 7fcabd290640 1 Processor -- start 2023-10-30T12:04:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.100+0000 7fcabd290640 1 -- start start 2023-10-30T12:04:18.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.100+0000 7fcabd290640 10 monclient: init 2023-10-30T12:04:18.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:18.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-10-30T12:04:18.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcab80a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcab80a2450 0x7fcab813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient(hunting): picked mon.noname-a con 0x7fcab80a2450 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab813d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient(hunting): picked mon.noname-b con 0x7fcab813af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab80a4190 con 0x7fcab80a2ae0 2023-10-30T12:04:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab80a1630 con 0x7fcab80a2450 2023-10-30T12:04:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab809a2e0 con 0x7fcab813af40 2023-10-30T12:04:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcabd290640 10 monclient(hunting): authenticate will time out at 1103.284403s 2023-10-30T12:04:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab813d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcab80a2450 0x7fcab813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab6ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcab80a2ae0 0x7fcab80a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39966/0 (socket says 172.21.15.50:39966) 2023-10-30T12:04:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab6ffd640 1 -- 172.21.15.50:0/1862210453 learned_addr learned my addr 172.21.15.50:0/1862210453 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:04:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): get_auth_request con 0x7fcab813af40 auth_method 0 2023-10-30T12:04:18.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 10 monclient(hunting): get_auth_request con 0x7fcab80a2450 auth_method 0 2023-10-30T12:04:18.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient(hunting): handle_auth_done global_id 44555 payload 274 2023-10-30T12:04:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 1 -- 172.21.15.50:0/1862210453 >> v2:172.21.15.50:3300/0 conn(0x7fcab80a2450 msgr2=0x7fcab813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 1 --2- 172.21.15.50:0/1862210453 >> v2:172.21.15.50:3300/0 conn(0x7fcab80a2450 0x7fcab813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 1 -- 172.21.15.50:0/1862210453 >> v1:172.21.15.50:6789/0 conn(0x7fcab80a2ae0 legacy=0x7fcab80a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:04:18.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 1 monclient: found mon.noname-b 2023-10-30T12:04:18.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:18.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab5ffb640 1 -- 172.21.15.50:0/1862210453 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837128419 0 0) 0x7fcab0001110 con 0x7fcab80a2ae0 2023-10-30T12:04:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab77fe640 1 -- 172.21.15.50:0/1862210453 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab809a8a0 con 0x7fcab813af40 2023-10-30T12:04:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab67fc640 1 --2- 172.21.15.50:0/1862210453 >> v2:172.21.15.50:3300/0 conn(0x7fcab80a2450 0x7fcab813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.101+0000 7fcab5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:04:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab77fe640 1 --2- 172.21.15.50:0/1862210453 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab813d330 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009d90 tx=0x7fcaa802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2718f8eb14d3583 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 1 -- 172.21.15.50:0/1862210453 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa802cb30 con 0x7fcab813af40 2023-10-30T12:04:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: dump: 2023-10-30T12:04:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.125 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.125 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.125 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:04:18.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:04:18.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:18.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 1 -- 172.21.15.50:0/1862210453 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa802cc90 con 0x7fcab813af40 2023-10-30T12:04:18.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:18.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 1 -- 172.21.15.50:0/1862210453 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8034440 con 0x7fcab813af40 2023-10-30T12:04:18.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcabd290640 5 monclient: authenticate success, global_id 44555 2023-10-30T12:04:18.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcabd290640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:18.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcabd290640 4 set_mon_vals no callback set 2023-10-30T12:04:18.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:18.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.102+0000 7fcab5ffb640 10 monclient: dump: 2023-10-30T12:04:18.128 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.128 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.128 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.128 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.128 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.128 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.129 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 10 monclient: shutdown 2023-10-30T12:04:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 -- 172.21.15.50:0/1862210453 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 msgr2=0x7fcab813d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 --2- 172.21.15.50:0/1862210453 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab813d330 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009d90 tx=0x7fcaa802bd80 comp rx=0 tx=0).stop 2023-10-30T12:04:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 -- 172.21.15.50:0/1862210453 shutdown_connections 2023-10-30T12:04:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 --2- 172.21.15.50:0/1862210453 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab813d330 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 --2- 172.21.15.50:0/1862210453 >> v2:172.21.15.50:3300/0 conn(0x7fcab80a2450 0x7fcab813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 -- 172.21.15.50:0/1862210453 >> 172.21.15.50:0/1862210453 conn(0x7fcab80a80b0 msgr2=0x7fcab80a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:18.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 -- 172.21.15.50:0/1862210453 shutdown_connections 2023-10-30T12:04:18.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.103+0000 7fcabd290640 1 -- 172.21.15.50:0/1862210453 wait complete. 2023-10-30T12:04:18.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient: build_initial_monmap 2023-10-30T12:04:18.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient: monmap: 2023-10-30T12:04:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:18.133 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 librados: starting msgr at 2023-10-30T12:04:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 librados: starting objecter 2023-10-30T12:04:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 Processor -- start 2023-10-30T12:04:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 -- start start 2023-10-30T12:04:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 librados: setting wanted keys 2023-10-30T12:04:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 librados: calling monclient init 2023-10-30T12:04:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient: init 2023-10-30T12:04:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:04:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcab80a2450 0x7fcab8134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): picked mon.noname-b con 0x7fcab80a2450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcab80a2ae0 0x7fcab8135300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): picked mon.noname-a con 0x7fcab80a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab8135840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): picked mon.noname-c con 0x7fcab813af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8138bd0 con 0x7fcab80a2ae0 2023-10-30T12:04:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8138510 con 0x7fcab80a2450 2023-10-30T12:04:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab813d540 con 0x7fcab813af40 2023-10-30T12:04:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.104+0000 7fcabd290640 10 monclient(hunting): authenticate will time out at 1103.287987s 2023-10-30T12:04:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcab80a2ae0 0x7fcab8135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcab80a2ae0 0x7fcab8135300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60042/0 (socket says 172.21.15.50:60042) 2023-10-30T12:04:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 -- 172.21.15.50:0/3245179782 learned_addr learned my addr 172.21.15.50:0/3245179782 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): get_auth_request con 0x7fcab80a2ae0 auth_method 0 2023-10-30T12:04:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab8135840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 10 monclient(hunting): get_auth_request con 0x7fcab813af40 auth_method 0 2023-10-30T12:04:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab6ffd640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcab80a2450 0x7fcab8134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient(hunting): handle_auth_done global_id 74172 payload 824 2023-10-30T12:04:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 msgr2=0x7fcab8135840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab8135840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcab80a2450 msgr2=0x7fcab8134dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcab80a2450 0x7fcab8134dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient: _finish_hunting 0 2023-10-30T12:04:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 monclient: found mon.noname-a 2023-10-30T12:04:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 -- 172.21.15.50:0/3245179782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac009400 con 0x7fcab80a2ae0 2023-10-30T12:04:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab6ffd640 10 monclient: get_auth_request con 0x7fcab80a2450 auth_method 0 2023-10-30T12:04:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab6ffd640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcab80a2450 0x7fcab8134dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:04:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab77fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab813af40 0x7fcab8135840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab67fc640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcab80a2ae0 0x7fcab8135300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00ebe0 tx=0x7fcaac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 client.?.objecter ms_handle_connect 0x7fcab80a2ae0 2023-10-30T12:04:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac009760 con 0x7fcab80a2ae0 2023-10-30T12:04:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: dump: 2023-10-30T12:04:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.146 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:04:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:04:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac00cc30 con 0x7fcab80a2ae0 2023-10-30T12:04:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 5 monclient: authenticate success, global_id 74172 2023-10-30T12:04:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 10 monclient: _renew_subs 2023-10-30T12:04:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa8009a70 con 0x7fcab80a2ae0 2023-10-30T12:04:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac00eee0 con 0x7fcab80a2ae0 2023-10-30T12:04:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fca9f7fe640 10 monclient: dump: 2023-10-30T12:04:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.151 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 10 client.74172.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 10 monclient: _renew_subs 2023-10-30T12:04:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcab57fa640 4 set_mon_vals no callback set 2023-10-30T12:04:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab8099bc0 con 0x7fcab80a2ae0 2023-10-30T12:04:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.105+0000 7fcabd290640 1 librados: init done 2023-10-30T12:04:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcabd290640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcabd290640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8139540 0x7fcab81321e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcabd290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8073ff0 con 0x7fcab8139540 2023-10-30T12:04:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcaac013470 con 0x7fcab80a2ae0 2023-10-30T12:04:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8139540 0x7fcab81321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 10 monclient: get_auth_request con 0x7fcab8139540 auth_method 0 2023-10-30T12:04:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.106+0000 7fca9f7fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcaa40392f0 0x7fcaa403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_done global_id 44561 payload 824 2023-10-30T12:04:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6394+0+0 (secure 0 0 0) 0x7fcaac04af80 con 0x7fcab80a2ae0 2023-10-30T12:04:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 client.74172.objecter ms_dispatch 0x7fcab8000b90 osd_map(114..114 src has 1..114) v4 2023-10-30T12:04:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 3 client.74172.objecter handle_osd_map got epochs [114,114] > 0 2023-10-30T12:04:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 3 client.74172.objecter handle_osd_map decoding full epoch 114 2023-10-30T12:04:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fcab6ffd640 1 -- 172.21.15.50:0/3245179782 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcab809a5c0 con 0x7fcab8139540 2023-10-30T12:04:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 20 client.74172.objecter dump_active .. 0 homeless 2023-10-30T12:04:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 client.74172.objecter ms_handle_connect 0x7fcab8139540 2023-10-30T12:04:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 client.74172.objecter resend_mon_ops 2023-10-30T12:04:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fcab6ffd640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8139540 0x7fcab81321e0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000f60 tx=0x7fcab000db80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fcab77fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcaa40392f0 0x7fcaa403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fcab77fe640 10 monclient: get_auth_request con 0x7fcaa40392f0 auth_method 0 2023-10-30T12:04:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab00072d0 con 0x7fcab8139540 2023-10-30T12:04:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 monclient: dump: 2023-10-30T12:04:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.159 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fcab0007430 con 0x7fcab8139540 2023-10-30T12:04:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8139540 msgr2=0x7fcab81321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8139540 0x7fcab81321e0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000f60 tx=0x7fcab000db80 comp rx=0 tx=0).stop 2023-10-30T12:04:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8139540 msgr2=0x7fcab81321e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8139540 0x7fcab81321e0 secure :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000f60 tx=0x7fcab000db80 comp rx=0 tx=0).stop 2023-10-30T12:04:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fcab77fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcaa40392f0 0x7fcaa403b7b0 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8036000 tx=0x7fcaa8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.107+0000 7fca9f7fe640 10 client.74172.objecter ms_handle_connect 0x7fcaa40392f0 2023-10-30T12:04:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.113+0000 7fcabd290640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcabd290640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8132720 0x7fcab8132b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcabd290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80a28a0 con 0x7fcab8132720 2023-10-30T12:04:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8132720 0x7fcab8132b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient: get_auth_request con 0x7fcab8132720 auth_method 0 2023-10-30T12:04:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 10 monclient(hunting): handle_auth_done global_id 44564 payload 824 2023-10-30T12:04:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 1 -- 172.21.15.50:0/3245179782 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcab806b060 con 0x7fcab8132720 2023-10-30T12:04:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fcab6ffd640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8132720 0x7fcab8132b30 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000f60 tx=0x7fcab0001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 10 client.74172.objecter ms_handle_connect 0x7fcab8132720 2023-10-30T12:04:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 10 client.74172.objecter resend_mon_ops 2023-10-30T12:04:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0023070 con 0x7fcab8132720 2023-10-30T12:04:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 10 monclient: dump: 2023-10-30T12:04:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.168 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fcab000bc30 con 0x7fcab8132720 2023-10-30T12:04:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8132720 msgr2=0x7fcab8132b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8132720 0x7fcab8132b30 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000f60 tx=0x7fcab0001080 comp rx=0 tx=0).stop 2023-10-30T12:04:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8132720 msgr2=0x7fcab8132b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.114+0000 7fca9f7fe640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcab8132720 0x7fcab8132b30 secure :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000f60 tx=0x7fcab0001080 comp rx=0 tx=0).stop 2023-10-30T12:04:18.170 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:04:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 40, 2023-10-30T12:04:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:04:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:04:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:04:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:04:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:18.199 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:18.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 10 librados: watch_flush enter 2023-10-30T12:04:18.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 10 librados: watch_flush exit 2023-10-30T12:04:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 20 client.74172.objecter shutdown clearing up homeless session... 2023-10-30T12:04:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 10 client.74172.objecter successfully canceled tick 2023-10-30T12:04:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcaa40392f0 msgr2=0x7fcaa403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcaa40392f0 0x7fcaa403b7b0 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8036000 tx=0x7fcaa8030000 comp rx=0 tx=0).stop 2023-10-30T12:04:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 10 monclient: shutdown 2023-10-30T12:04:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcab80a2ae0 msgr2=0x7fcab8135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcab80a2ae0 0x7fcab8135300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00ebe0 tx=0x7fcaac00c520 comp rx=0 tx=0).stop 2023-10-30T12:04:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcab77fe640 1 -- 172.21.15.50:0/3245179782 reap_dead start 2023-10-30T12:04:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 shutdown_connections 2023-10-30T12:04:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 --2- 172.21.15.50:0/3245179782 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcab80a2ae0 0x7fcab8135300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 >> 172.21.15.50:0/3245179782 conn(0x7fcab80a80b0 msgr2=0x7fcab80a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 shutdown_connections 2023-10-30T12:04:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 -- 172.21.15.50:0/3245179782 wait complete. 2023-10-30T12:04:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.115+0000 7fcabd290640 1 librados: shutdown 2023-10-30T12:04:18.205 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:04:18.205 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:04:18.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:04:18.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.310+0000 7f8837f73640 10 monclient: get_monmap_and_config 2023-10-30T12:04:18.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.310+0000 7f8837f73640 10 monclient: build_initial_monmap 2023-10-30T12:04:18.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.310+0000 7f8837f73640 10 monclient: monmap: 2023-10-30T12:04:18.312 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:18.312 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.312 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:18.311158+0000 2023-10-30T12:04:18.312 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:18.311158+0000 2023-10-30T12:04:18.313 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:18.313 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.313 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:18.313 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:18.313 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:18.314 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:18.314 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:18.314 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.310+0000 7f8837f73640 1 Processor -- start 2023-10-30T12:04:18.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.310+0000 7f8837f73640 1 -- start start 2023-10-30T12:04:18.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.310+0000 7f8837f73640 10 monclient: init 2023-10-30T12:04:18.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:18.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-10-30T12:04:18.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88300a2ae0 0x7f88300a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): picked mon.noname-b con 0x7f88300a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:18.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f88300a2ec0 0x7f88301356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): picked mon.noname-a con 0x7f88300a2ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:18.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f88300a32e0 0x7f8830135c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): picked mon.noname-c con 0x7f88300a32e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:04:18.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88300a2990 con 0x7f88300a2ec0 2023-10-30T12:04:18.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88300a1630 con 0x7f88300a32e0 2023-10-30T12:04:18.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88300a4190 con 0x7f88300a2ae0 2023-10-30T12:04:18.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:18.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8837f73640 10 monclient(hunting): authenticate will time out at 1103.494440s 2023-10-30T12:04:18.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f88300a2ec0 0x7f88301356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88300a2ae0 0x7f88300a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f88300a2ec0 0x7f88301356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60054/0 (socket says 172.21.15.50:60054) 2023-10-30T12:04:18.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88300a2ae0 0x7f88300a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:54286/0 (socket says 172.21.15.50:54286) 2023-10-30T12:04:18.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 1 -- 172.21.15.50:0/2874830448 learned_addr learned my addr 172.21.15.50:0/2874830448 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:18.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 10 monclient(hunting): get_auth_request con 0x7f88300a2ec0 auth_method 0 2023-10-30T12:04:18.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): get_auth_request con 0x7f88300a2ae0 auth_method 0 2023-10-30T12:04:18.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8836510640 1 --2- 172.21.15.50:0/2874830448 >> v2:172.21.15.118:3300/0 conn(0x7f88300a32e0 0x7f8830135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8836510640 10 monclient(hunting): get_auth_request con 0x7f88300a32e0 auth_method 0 2023-10-30T12:04:18.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8836510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8836510640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8836510640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient(hunting): handle_auth_done global_id 44567 payload 274 2023-10-30T12:04:18.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 -- 172.21.15.50:0/2874830448 >> v2:172.21.15.118:3300/0 conn(0x7f88300a32e0 msgr2=0x7f8830135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:18.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 --2- 172.21.15.50:0/2874830448 >> v2:172.21.15.118:3300/0 conn(0x7f88300a32e0 0x7f8830135c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 -- 172.21.15.50:0/2874830448 >> v2:172.21.15.50:3300/0 conn(0x7f88300a2ec0 msgr2=0x7f88301356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:18.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 --2- 172.21.15.50:0/2874830448 >> v2:172.21.15.50:3300/0 conn(0x7f88300a2ec0 0x7f88301356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient: _finish_hunting 0 2023-10-30T12:04:18.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 monclient: found mon.noname-b 2023-10-30T12:04:18.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:18.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:18.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f8835d0f640 1 -- 172.21.15.50:0/2874830448 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883009a2e0 con 0x7f88300a2ae0 2023-10-30T12:04:18.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.311+0000 7f883550e640 1 --2- 172.21.15.50:0/2874830448 >> v2:172.21.15.50:3300/0 conn(0x7f88300a2ec0 0x7f88301356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:18.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8835d0f640 1 --2- 172.21.15.50:0/2874830448 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88300a2ae0 0x7f88300a2450 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f88300a5500 tx=0x7f882002bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10684cc01d6cc1fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 1 -- 172.21.15.50:0/2874830448 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882002cab0 con 0x7f88300a2ae0 2023-10-30T12:04:18.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:18.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: dump: 2023-10-30T12:04:18.335 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.335 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.337 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: _finish_auth 0 2023-10-30T12:04:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: _check_auth_tickets 2023-10-30T12:04:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8837f73640 5 monclient: authenticate success, global_id 44567 2023-10-30T12:04:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 1 -- 172.21.15.50:0/2874830448 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882002cc10 con 0x7f88300a2ae0 2023-10-30T12:04:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8837f73640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8837f73640 4 set_mon_vals no callback set 2023-10-30T12:04:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 1 -- 172.21.15.50:0/2874830448 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820034440 con 0x7f88300a2ae0 2023-10-30T12:04:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.312+0000 7f8834d0d640 10 monclient: dump: 2023-10-30T12:04:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.341 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 10 monclient: shutdown 2023-10-30T12:04:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 -- 172.21.15.50:0/2874830448 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88300a2ae0 msgr2=0x7f88300a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874830448 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88300a2ae0 0x7f88300a2450 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f88300a5500 tx=0x7f882002bd00 comp rx=0 tx=0).stop 2023-10-30T12:04:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 -- 172.21.15.50:0/2874830448 shutdown_connections 2023-10-30T12:04:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874830448 >> v2:172.21.15.118:3300/0 conn(0x7f88300a32e0 0x7f8830135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874830448 >> v2:172.21.15.50:3300/0 conn(0x7f88300a2ec0 0x7f88301356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874830448 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88300a2ae0 0x7f88300a2450 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 -- 172.21.15.50:0/2874830448 >> 172.21.15.50:0/2874830448 conn(0x7f88300a80b0 msgr2=0x7f88300a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 -- 172.21.15.50:0/2874830448 shutdown_connections 2023-10-30T12:04:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.313+0000 7f8837f73640 1 -- 172.21.15.50:0/2874830448 wait complete. 2023-10-30T12:04:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient: build_initial_monmap 2023-10-30T12:04:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient: monmap: 2023-10-30T12:04:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:18.345 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 librados: starting msgr at 2023-10-30T12:04:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 librados: starting objecter 2023-10-30T12:04:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 Processor -- start 2023-10-30T12:04:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 -- start start 2023-10-30T12:04:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 librados: setting wanted keys 2023-10-30T12:04:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 librados: calling monclient init 2023-10-30T12:04:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient: init 2023-10-30T12:04:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:04:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88300a2ae0 0x7f88301ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): picked mon.noname-a con 0x7f88300a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a2ec0 0x7f8830132570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): picked mon.noname-b con 0x7f88300a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8830132ab0 0x7f883011e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): picked mon.noname-c con 0x7f8830132ab0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88301ca7d0 con 0x7f88300a2ae0 2023-10-30T12:04:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8830120730 con 0x7f88300a2ec0 2023-10-30T12:04:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8830120c60 con 0x7f8830132ab0 2023-10-30T12:04:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.314+0000 7f8837f73640 10 monclient(hunting): authenticate will time out at 1103.497987s 2023-10-30T12:04:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88300a2ae0 0x7f88301ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88300a2ae0 0x7f88301ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60064/0 (socket says 172.21.15.50:60064) 2023-10-30T12:04:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 -- 172.21.15.50:0/2874744471 learned_addr learned my addr 172.21.15.50:0/2874744471 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): get_auth_request con 0x7f88300a2ae0 auth_method 0 2023-10-30T12:04:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8830132ab0 0x7f883011e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 10 monclient(hunting): get_auth_request con 0x7f8830132ab0 auth_method 0 2023-10-30T12:04:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f883550e640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a2ec0 0x7f8830132570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f883550e640 10 monclient(hunting): get_auth_request con 0x7f88300a2ec0 auth_method 0 2023-10-30T12:04:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f883550e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f883550e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f883550e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient(hunting): handle_auth_done global_id 74178 payload 824 2023-10-30T12:04:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8830132ab0 msgr2=0x7f883011e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8830132ab0 0x7f883011e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a2ec0 msgr2=0x7f8830132570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a2ec0 0x7f8830132570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient: _finish_hunting 0 2023-10-30T12:04:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 monclient: found mon.noname-a 2023-10-30T12:04:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:04:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 -- 172.21.15.50:0/2874744471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88240094d0 con 0x7f88300a2ae0 2023-10-30T12:04:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8836510640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8830132ab0 0x7f883011e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:04:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f883550e640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a2ec0 0x7f8830132570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:04:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8835d0f640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88300a2ae0 0x7f88301ca290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8820009e80 tx=0x7f8820004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 client.?.objecter ms_handle_connect 0x7f88300a2ae0 2023-10-30T12:04:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820040650 con 0x7f88300a2ae0 2023-10-30T12:04:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:04:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: dump: 2023-10-30T12:04:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.360 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:04:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:04:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88200407b0 con 0x7f88300a2ae0 2023-10-30T12:04:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 5 monclient: authenticate success, global_id 74178 2023-10-30T12:04:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820040b60 con 0x7f88300a2ae0 2023-10-30T12:04:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 10 monclient: _renew_subs 2023-10-30T12:04:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8820009a70 con 0x7f88300a2ae0 2023-10-30T12:04:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882ffff640 4 set_mon_vals no callback set 2023-10-30T12:04:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:04:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f882e7fc640 10 monclient: dump: 2023-10-30T12:04:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.364 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 10 client.74178.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 10 monclient: _renew_subs 2023-10-30T12:04:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:04:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883011f840 con 0x7f88300a2ae0 2023-10-30T12:04:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.315+0000 7f8837f73640 1 librados: init done 2023-10-30T12:04:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8820040cc0 con 0x7f88300a2ae0 2023-10-30T12:04:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f8837f73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a32e0 0x7f883011e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8830073ff0 con 0x7f88300a32e0 2023-10-30T12:04:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f882e7fc640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f881c0392f0 0x7f881c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f882006dc00 con 0x7f88300a2ae0 2023-10-30T12:04:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f882e7fc640 10 client.74178.objecter ms_dispatch 0x7f8830000b90 osd_map(114..114 src has 1..114) v4 2023-10-30T12:04:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f882e7fc640 3 client.74178.objecter handle_osd_map got epochs [114,114] > 0 2023-10-30T12:04:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.316+0000 7f882e7fc640 3 client.74178.objecter handle_osd_map decoding full epoch 114 2023-10-30T12:04:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 20 client.74178.objecter dump_active .. 0 homeless 2023-10-30T12:04:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a32e0 0x7f883011e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient: get_auth_request con 0x7f88300a32e0 auth_method 0 2023-10-30T12:04:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f8836510640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f881c0392f0 0x7f881c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f8836510640 10 monclient: get_auth_request con 0x7f881c0392f0 auth_method 0 2023-10-30T12:04:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 10 monclient(hunting): handle_auth_done global_id 28216 payload 824 2023-10-30T12:04:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 1 -- 172.21.15.50:0/2874744471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883009fef0 con 0x7f88300a32e0 2023-10-30T12:04:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 10 client.74178.objecter ms_handle_connect 0x7f881c0392f0 2023-10-30T12:04:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f8836510640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f881c0392f0 0x7f881c03b7b0 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7f8828006fd0 tx=0x7f8828008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 10 client.74178.objecter ms_handle_connect 0x7f88300a32e0 2023-10-30T12:04:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 10 client.74178.objecter resend_mon_ops 2023-10-30T12:04:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f883550e640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a32e0 0x7f883011e5e0 secure :-1 s=READY pgs=7111 cs=0 l=1 rev1=1 crypto rx=0x7f883009f940 tx=0x7f8824011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824011ee0 con 0x7f88300a32e0 2023-10-30T12:04:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.317+0000 7f882e7fc640 10 monclient: dump: 2023-10-30T12:04:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.318+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8824004220 con 0x7f88300a32e0 2023-10-30T12:04:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.318+0000 7f882e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.318+0000 7f882e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.318+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a32e0 msgr2=0x7f883011e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.318+0000 7f882e7fc640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a32e0 0x7f883011e5e0 secure :-1 s=READY pgs=7111 cs=0 l=1 rev1=1 crypto rx=0x7f883009f940 tx=0x7f8824011c90 comp rx=0 tx=0).stop 2023-10-30T12:04:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.318+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a32e0 msgr2=0x7f883011e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.318+0000 7f882e7fc640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88300a32e0 0x7f883011e5e0 secure :-1 s=CLOSED pgs=7111 cs=0 l=1 rev1=1 crypto rx=0x7f883009f940 tx=0x7f8824011c90 comp rx=0 tx=0).stop 2023-10-30T12:04:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f8837f73640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f883011eb20 0x7f883009c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f8837f73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883009a5c0 con 0x7f883011eb20 2023-10-30T12:04:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f883011eb20 0x7f883009c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 10 monclient: get_auth_request con 0x7f883011eb20 auth_method 0 2023-10-30T12:04:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.324+0000 7f883550e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f883550e640 10 monclient(hunting): handle_auth_done global_id 28219 payload 824 2023-10-30T12:04:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f883550e640 1 -- 172.21.15.50:0/2874744471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8830099b40 con 0x7f883011eb20 2023-10-30T12:04:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f883550e640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f883011eb20 0x7f883009c3b0 secure :-1 s=READY pgs=7112 cs=0 l=1 rev1=1 crypto rx=0x7f883009f940 tx=0x7f88240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 10 client.74178.objecter ms_handle_connect 0x7f883011eb20 2023-10-30T12:04:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 10 client.74178.objecter resend_mon_ops 2023-10-30T12:04:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824004500 con 0x7f883011eb20 2023-10-30T12:04:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 10 monclient: dump: 2023-10-30T12:04:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:18.382 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f8824015660 con 0x7f883011eb20 2023-10-30T12:04:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f883011eb20 msgr2=0x7f883009c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f883011eb20 0x7f883009c3b0 secure :-1 s=READY pgs=7112 cs=0 l=1 rev1=1 crypto rx=0x7f883009f940 tx=0x7f88240046c0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f883011eb20 msgr2=0x7f883009c3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f882e7fc640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f883011eb20 0x7f883009c3b0 unknown :-1 s=CLOSED pgs=7112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:18.384 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:04:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 40, 2023-10-30T12:04:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:04:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.386 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:04:18.386 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:18.386 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:18.386 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:18.386 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:18.387 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:18.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:18.387 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:18.387 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:18.387 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:18.388 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:18.388 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:18.388 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:18.388 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:18.388 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.389 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:18.389 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:18.389 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:18.389 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:18.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:18.390 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:18.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:18.390 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:18.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:18.390 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:18.390 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:18.391 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:18.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.391 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:18.391 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:18.392 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:18.392 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:18.392 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:18.392 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:18.392 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:18.393 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:18.393 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:18.393 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:18.393 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:18.393 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:18.394 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:18.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:18.394 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:18.394 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:18.394 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:18.395 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:18.395 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:18.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:18.395 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:18.395 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:18.396 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:18.396 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:18.396 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:18.396 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:18.396 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:18.397 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.397 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:18.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:18.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.397 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:18.398 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:18.398 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:18.398 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:18.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.398 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:18.399 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:18.399 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.399 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:18.400 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:18.400 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.400 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.400 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:18.401 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:18.401 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:18.401 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:18.401 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:18.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.402 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:18.402 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:18.402 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:18.402 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:18.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.403 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:18.403 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:18.403 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.404 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:18.404 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:18.404 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.404 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.404 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.405 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:18.405 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:18.405 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:18.405 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:18.405 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:18.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.406 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:18.406 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:18.406 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:18.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:18.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.407 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:18.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:18.407 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:18.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:18.408 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:18.408 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.409 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.409 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:18.409 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:18.409 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:18.409 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:18.410 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:18.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.410 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:18.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:18.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.411 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:18.411 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:18.411 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:18.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.412 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.412 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:04:18.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.412 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:18.413 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:18.413 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:04:18.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.413 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.413 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:18.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.414 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:18.414 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:18.414 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 8 2023-10-30T12:04:18.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.414 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:18.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:04:18.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:18.415 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:18.415 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:18.415 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:18.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:18.416 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:18.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:18.416 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:18.416 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f8837f73640 10 librados: watch_flush enter 2023-10-30T12:04:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f8837f73640 10 librados: watch_flush exit 2023-10-30T12:04:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f8837f73640 20 client.74178.objecter shutdown clearing up homeless session... 2023-10-30T12:04:18.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f8837f73640 10 client.74178.objecter successfully canceled tick 2023-10-30T12:04:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.325+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f881c0392f0 msgr2=0x7f881c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f881c0392f0 0x7f881c03b7b0 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7f8828006fd0 tx=0x7f8828008040 comp rx=0 tx=0).stop 2023-10-30T12:04:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 10 monclient: shutdown 2023-10-30T12:04:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88300a2ae0 msgr2=0x7f88301ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:18.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 --2- 172.21.15.50:0/2874744471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88300a2ae0 0x7f88301ca290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8820009e80 tx=0x7f8820004a40 comp rx=0 tx=0).stop 2023-10-30T12:04:18.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8836510640 1 -- 172.21.15.50:0/2874744471 reap_dead start 2023-10-30T12:04:18.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8836510640 1 -- 172.21.15.50:0/2874744471 reap_dead start 2023-10-30T12:04:18.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 shutdown_connections 2023-10-30T12:04:18.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 >> 172.21.15.50:0/2874744471 conn(0x7f88300a80b0 msgr2=0x7f88301b96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:18.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 shutdown_connections 2023-10-30T12:04:18.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 -- 172.21.15.50:0/2874744471 wait complete. 2023-10-30T12:04:18.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:18.326+0000 7f8837f73640 1 librados: shutdown 2023-10-30T12:04:18.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:04:18.422 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:04:18.422 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2023-10-30T12:04:18.422 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:04:18.422 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:04:18.423 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:04:18.423 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:04:18.432 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:04:18.432 INFO:tasks.mon_thrash:Sending STOP to mon b 2023-10-30T12:04:18.433 INFO:tasks.ceph.mon.b:Sent signal 19 2023-10-30T12:04:18.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-10-30T12:04:33.434 INFO:tasks.mon_thrash:Sending CONT to mon b 2023-10-30T12:04:33.435 INFO:tasks.ceph.mon.b:Sent signal 18 2023-10-30T12:04:33.435 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:04:33.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.550+0000 7f812571d640 10 monclient: get_monmap_and_config 2023-10-30T12:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.550+0000 7f812571d640 10 monclient: build_initial_monmap 2023-10-30T12:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.550+0000 7f812571d640 10 monclient: monmap: 2023-10-30T12:04:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:33.551871+0000 2023-10-30T12:04:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:33.551871+0000 2023-10-30T12:04:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:33.554 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:33.554 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:33.554 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:33.554 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:33.554 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:33.555 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 1 Processor -- start 2023-10-30T12:04:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 1 -- start start 2023-10-30T12:04:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient: init 2023-10-30T12:04:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:04:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 0x7f812009e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:33.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient(hunting): picked mon.noname-a con 0x7f81200707d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:33.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8120070bb0 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:33.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:33.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient(hunting): picked mon.noname-c con 0x7f812013aba0 addr v2:172.21.15.118:3300/0 2023-10-30T12:04:33.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812009ee40 con 0x7f8120070bb0 2023-10-30T12:04:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812009f4f0 con 0x7f81200707d0 2023-10-30T12:04:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812009f7d0 con 0x7f812013aba0 2023-10-30T12:04:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.551+0000 7f812571d640 10 monclient(hunting): authenticate will time out at 1118.735002s 2023-10-30T12:04:33.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811e7fc640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8120070bb0 legacy=0x7f8120071020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:04:33.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811effd640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 msgr2=0x7f812009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:33.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 0x7f812009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:33.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:33.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:52494/0 (socket says 172.21.15.50:52494) 2023-10-30T12:04:33.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 learned_addr learned my addr 172.21.15.50:0/638211819 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 10 monclient(hunting): get_auth_request con 0x7f812013aba0 auth_method 0 2023-10-30T12:04:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:04:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:33.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:33.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.552+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> v1:172.21.15.50:6789/0 conn(0x7f8120070bb0 legacy=0x7f8120071020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:04:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811effd640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 msgr2=0x7f812009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811effd640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 0x7f812009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:04:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 10 monclient(hunting): get_auth_request con 0x7f812013aba0 auth_method 0 2023-10-30T12:04:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:33.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:33.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:33.753+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:04:34.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> v1:172.21.15.50:6789/0 conn(0x7f8120070bb0 legacy=0x7f8120071020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:04:34.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811effd640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 msgr2=0x7f812009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:34.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811effd640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 0x7f812009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:04:34.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:34.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 10 monclient(hunting): get_auth_request con 0x7f812013aba0 auth_method 0 2023-10-30T12:04:34.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:34.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:34.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:34.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:04:34.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:34.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:34.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.154+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:04:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> v1:172.21.15.50:6789/0 conn(0x7f8120070bb0 legacy=0x7f8120071020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:04:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811effd640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 msgr2=0x7f812009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811effd640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 0x7f812009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:04:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 10 monclient(hunting): get_auth_request con 0x7f812013aba0 auth_method 0 2023-10-30T12:04:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:04:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:34.955+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:04:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient(hunting): tick 2023-10-30T12:04:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:04:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:04:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:04:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 msgr2=0x7f812013cf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.118:3300/0 conn(0x7f812013aba0 0x7f812013cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 -- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 msgr2=0x7f812009e7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 --2- 172.21.15.50:0/638211819 >> v2:172.21.15.50:3300/0 conn(0x7f81200707d0 0x7f812009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 -- 172.21.15.50:0/638211819 >> v1:172.21.15.50:6789/0 conn(0x7f8120070bb0 legacy=0x7f8120071020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient: _add_conns ranks=[1,2] 2023-10-30T12:04:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f810c003670 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f810c007360 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 -- 172.21.15.50:0/638211819 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812009fab0 con 0x7f810c003670 2023-10-30T12:04:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 1 -- 172.21.15.50:0/638211819 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812009f4f0 con 0x7f810c007360 2023-10-30T12:04:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811cff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 10 monclient(hunting): get_auth_request con 0x7f810c007360 auth_method 0 2023-10-30T12:04:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:36.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:36.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:36.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.552+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 10 monclient(hunting): get_auth_request con 0x7f810c007360 auth_method 0 2023-10-30T12:04:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:36.753+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:04:37.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:37.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 10 monclient(hunting): get_auth_request con 0x7f810c007360 auth_method 0 2023-10-30T12:04:37.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:37.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:37.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:37.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:37.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:37.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.154+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:04:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 10 monclient(hunting): get_auth_request con 0x7f810c007360 auth_method 0 2023-10-30T12:04:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:37.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:04:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:04:37.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:04:37.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:37.955+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:04:38.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.467+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613084191 0 0) 0x7f812009fab0 con 0x7f810c003670 2023-10-30T12:04:38.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.467+0000 7f811dffb640 10 monclient(hunting): _negotiate 2023-10-30T12:04:38.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.467+0000 7f811dffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.467+0000 7f811dffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:38.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.467+0000 7f811dffb640 10 monclient(hunting): my global_id is 28222 2023-10-30T12:04:38.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.467+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8108003520 con 0x7f810c003670 2023-10-30T12:04:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2545110134 0 0) 0x7f8108003520 con 0x7f810c003670 2023-10-30T12:04:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 msgr2=0x7f810c009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f810c007360 0x7f810c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: _finish_hunting 0 2023-10-30T12:04:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:04:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:04:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812009ee40 con 0x7f810c003670 2023-10-30T12:04:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: _finish_auth 0 2023-10-30T12:04:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:04:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f8110004060 con 0x7f810c003670 2023-10-30T12:04:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f812571d640 5 monclient: authenticate success, global_id 28222 2023-10-30T12:04:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f812571d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:04:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: dump: 2023-10-30T12:04:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.498 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:04:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 >> v1:172.21.15.118:6789/0 conn(0x7f810c003670 legacy=0x7f810c005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 reap_dead start 2023-10-30T12:04:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:04:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8108003970 0x7f8108003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): picked mon.b con 0x7f8108003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81080047c0 0x7f8108004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): picked mon.c con 0x7f81080047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080051d0 0x7f810800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): picked mon.a con 0x7f81080051d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812009f4f0 con 0x7f81080051d0 2023-10-30T12:04:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810800c100 con 0x7f8108003970 2023-10-30T12:04:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811f7fe640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080051d0 msgr2=0x7f810800b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810800c2c0 con 0x7f81080047c0 2023-10-30T12:04:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811f7fe640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080051d0 0x7f810800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811dffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:04:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f812571d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8108003970 0x7f8108003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81080047c0 0x7f8108004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): get_auth_request con 0x7f8108003970 auth_method 0 2023-10-30T12:04:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 10 monclient(hunting): get_auth_request con 0x7f81080047c0 auth_method 0 2023-10-30T12:04:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:04:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:04:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:04:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient(hunting): handle_auth_done global_id 28222 payload 293 2023-10-30T12:04:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81080047c0 msgr2=0x7f8108004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:04:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81080047c0 0x7f8108004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080051d0 msgr2=0x7f810800b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080051d0 0x7f810800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient: _finish_hunting 0 2023-10-30T12:04:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 1 monclient: found mon.b 2023-10-30T12:04:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:04:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 1 -- 172.21.15.50:0/638211819 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108003520 con 0x7f8108003970 2023-10-30T12:04:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient: _finish_auth 0 2023-10-30T12:04:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 10 monclient: _check_auth_tickets 2023-10-30T12:04:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.490+0000 7f811effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.491+0000 7f811effd640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8108003970 0x7f8108003d70 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 crypto rx=0x7f8110003980 tx=0x7f81080057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90fad1ac41ce419f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8110033620 con 0x7f8108003970 2023-10-30T12:04:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 10 monclient: dump: 2023-10-30T12:04:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.510 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811002f070 con 0x7f8108003970 2023-10-30T12:04:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 1 -- 172.21.15.50:0/638211819 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100339e0 con 0x7f8108003970 2023-10-30T12:04:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 4 set_mon_vals no callback set 2023-10-30T12:04:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f811dffb640 10 monclient: dump: 2023-10-30T12:04:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.513 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 10 monclient: shutdown 2023-10-30T12:04:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 1 -- 172.21.15.50:0/638211819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8108003970 msgr2=0x7f8108003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8108003970 0x7f8108003d70 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 crypto rx=0x7f8110003980 tx=0x7f81080057f0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 1 -- 172.21.15.50:0/638211819 shutdown_connections 2023-10-30T12:04:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080051d0 0x7f810800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f81080047c0 0x7f8108004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 1 --2- 172.21.15.50:0/638211819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8108003970 0x7f8108003d70 unknown :-1 s=CLOSED pgs=7115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.492+0000 7f812571d640 1 -- 172.21.15.50:0/638211819 >> 172.21.15.50:0/638211819 conn(0x7f81200a47f0 msgr2=0x7f81200a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.493+0000 7f812571d640 1 -- 172.21.15.50:0/638211819 shutdown_connections 2023-10-30T12:04:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.493+0000 7f812571d640 1 -- 172.21.15.50:0/638211819 wait complete. 2023-10-30T12:04:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.493+0000 7f812571d640 10 monclient: build_initial_monmap 2023-10-30T12:04:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.493+0000 7f812571d640 10 monclient: monmap: 2023-10-30T12:04:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:38.522 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.494+0000 7f812571d640 1 librados: starting msgr at 2023-10-30T12:04:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.494+0000 7f812571d640 1 librados: starting objecter 2023-10-30T12:04:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.494+0000 7f812571d640 1 Processor -- start 2023-10-30T12:04:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.494+0000 7f812571d640 1 -- start start 2023-10-30T12:04:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.494+0000 7f812571d640 1 librados: setting wanted keys 2023-10-30T12:04:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.494+0000 7f812571d640 1 librados: calling monclient init 2023-10-30T12:04:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.494+0000 7f812571d640 10 monclient: init 2023-10-30T12:04:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:04:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8108003970 0x7f81201c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): picked mon.noname-c con 0x7f8108003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080047c0 0x7f81201ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): picked mon.noname-a con 0x7f81080047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81080051d0 0x7f812011d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): picked mon.noname-b con 0x7f81080051d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81201200b0 con 0x7f81080047c0 2023-10-30T12:04:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8120121e30 con 0x7f81080051d0 2023-10-30T12:04:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8120121a10 con 0x7f8108003970 2023-10-30T12:04:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f812571d640 10 monclient(hunting): authenticate will time out at 1123.678191s 2023-10-30T12:04:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811e7fc640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080047c0 msgr2=0x7f81201ca4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811e7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080047c0 0x7f81201ca4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8108003970 0x7f81201c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8108003970 0x7f81201c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36248/0 (socket says 172.21.15.50:36248) 2023-10-30T12:04:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 -- 172.21.15.50:0/1744862001 learned_addr learned my addr 172.21.15.50:0/1744862001 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): get_auth_request con 0x7f8108003970 auth_method 0 2023-10-30T12:04:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811f7fe640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81080051d0 0x7f812011d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811f7fe640 10 monclient(hunting): get_auth_request con 0x7f81080051d0 auth_method 0 2023-10-30T12:04:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient(hunting): handle_auth_done global_id 44573 payload 824 2023-10-30T12:04:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 -- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81080051d0 msgr2=0x7f812011d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81080051d0 0x7f812011d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 -- 172.21.15.50:0/1744862001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080047c0 msgr2=0x7f81201ca4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:04:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080047c0 0x7f81201ca4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient: _finish_hunting 0 2023-10-30T12:04:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 monclient: found mon.noname-c 2023-10-30T12:04:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 -- 172.21.15.50:0/1744862001 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812009d6a0 con 0x7f8108003970 2023-10-30T12:04:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f811effd640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8108003970 0x7f81201c9fa0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f8110003df0 tx=0x7f81100062a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f81077fe640 10 client.?.objecter ms_handle_connect 0x7f8108003970 2023-10-30T12:04:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.495+0000 7f81077fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 1 -- 172.21.15.50:0/1744862001 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8110033830 con 0x7f8108003970 2023-10-30T12:04:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: dump: 2023-10-30T12:04:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.534 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: _finish_auth 0 2023-10-30T12:04:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: _check_auth_tickets 2023-10-30T12:04:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 5 monclient: authenticate success, global_id 44573 2023-10-30T12:04:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 1 -- 172.21.15.50:0/1744862001 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81100339d0 con 0x7f8108003970 2023-10-30T12:04:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 1 -- 172.21.15.50:0/1744862001 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8110033d80 con 0x7f8108003970 2023-10-30T12:04:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 10 monclient: _renew_subs 2023-10-30T12:04:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812009cb80 con 0x7f8108003970 2023-10-30T12:04:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f81077fe640 10 monclient: dump: 2023-10-30T12:04:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.538 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f811d7fa640 4 set_mon_vals no callback set 2023-10-30T12:04:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 10 client.44573.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 10 monclient: _renew_subs 2023-10-30T12:04:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8120121730 con 0x7f8108003970 2023-10-30T12:04:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.496+0000 7f812571d640 1 librados: init done 2023-10-30T12:04:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.497+0000 7f81077fe640 1 -- 172.21.15.50:0/1744862001 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f811003a030 con 0x7f8108003970 2023-10-30T12:04:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.497+0000 7f81077fe640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f810c039a40 0x7f810c03bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.497+0000 7f812571d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.497+0000 7f812571d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.497+0000 7f812571d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f81077fe640 1 -- 172.21.15.50:0/1744862001 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f811006dab0 con 0x7f8108003970 2023-10-30T12:04:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812004e160 con 0x7f8108003970 2023-10-30T12:04:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f81077fe640 10 client.44573.objecter ms_dispatch 0x7f8120000b90 osd_map(114..114 src has 1..114) v4 2023-10-30T12:04:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f81077fe640 3 client.44573.objecter handle_osd_map got epochs [114,114] > 0 2023-10-30T12:04:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f81077fe640 3 client.44573.objecter handle_osd_map decoding full epoch 114 2023-10-30T12:04:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f81077fe640 20 client.44573.objecter dump_active .. 0 homeless 2023-10-30T12:04:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f811e7fc640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f810c039a40 0x7f810c03bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f811e7fc640 10 monclient: get_auth_request con 0x7f810c039a40 auth_method 0 2023-10-30T12:04:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f81077fe640 10 client.44573.objecter ms_handle_connect 0x7f810c039a40 2023-10-30T12:04:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.498+0000 7f811e7fc640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f810c039a40 0x7f810c03bf00 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f812011d940 tx=0x7f811400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.503+0000 7f81077fe640 1 -- 172.21.15.50:0/1744862001 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8110038030 con 0x7f8108003970 2023-10-30T12:04:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.503+0000 7f81077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:38.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.503+0000 7f81077fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:38.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.760+0000 7f812571d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:04:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.760+0000 7f812571d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.760+0000 7f812571d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.760+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8120121920 con 0x7f8108003970 2023-10-30T12:04:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.760+0000 7f81077fe640 1 -- 172.21.15.50:0/1744862001 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f8110038070 con 0x7f8108003970 2023-10-30T12:04:38.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.760+0000 7f81077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:04:38.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.760+0000 7f81077fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:38.763 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":42,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:04:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 10 librados: watch_flush enter 2023-10-30T12:04:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 10 librados: watch_flush exit 2023-10-30T12:04:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 20 client.44573.objecter shutdown clearing up homeless session... 2023-10-30T12:04:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 10 client.44573.objecter successfully canceled tick 2023-10-30T12:04:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f810c039a40 msgr2=0x7f810c03bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f810c039a40 0x7f810c03bf00 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f812011d940 tx=0x7f811400d040 comp rx=0 tx=0).stop 2023-10-30T12:04:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 10 monclient: shutdown 2023-10-30T12:04:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8108003970 msgr2=0x7f81201c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8108003970 0x7f81201c9fa0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f8110003df0 tx=0x7f81100062a0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 shutdown_connections 2023-10-30T12:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f810c039a40 0x7f810c03bf00 unknown :-1 s=CLOSED pgs=3919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81080051d0 0x7f812011d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81080047c0 0x7f81201ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 --2- 172.21.15.50:0/1744862001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8108003970 0x7f81201c9fa0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 >> 172.21.15.50:0/1744862001 conn(0x7f81200a47f0 msgr2=0x7f812009c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 shutdown_connections 2023-10-30T12:04:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 -- 172.21.15.50:0/1744862001 wait complete. 2023-10-30T12:04:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.762+0000 7f812571d640 1 librados: shutdown 2023-10-30T12:04:38.774 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:04:38.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:04:38.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:04:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.879+0000 7f1f50a7c640 10 monclient: get_monmap_and_config 2023-10-30T12:04:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.879+0000 7f1f50a7c640 10 monclient: build_initial_monmap 2023-10-30T12:04:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.879+0000 7f1f50a7c640 10 monclient: monmap: 2023-10-30T12:04:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:38.880546+0000 2023-10-30T12:04:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:38.880546+0000 2023-10-30T12:04:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:38.883 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 1 Processor -- start 2023-10-30T12:04:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 1 -- start start 2023-10-30T12:04:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient: init 2023-10-30T12:04:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:04:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f4c132970 addr v1:172.21.15.50:6789/0 2023-10-30T12:04:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1f4c0a03d0 0x7f1f4c13aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c0a03d0 addr v2:172.21.15.118:3300/0 2023-10-30T12:04:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f4c13b020 0x7f1f4c13d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4c13b020 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c0a17d0 con 0x7f1f4c132970 2023-10-30T12:04:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c133460 con 0x7f1f4c13b020 2023-10-30T12:04:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c0a1510 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f50a7c640 10 monclient(hunting): authenticate will time out at 1124.063586s 2023-10-30T12:04:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f4a59c640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f1f4c132970 legacy=0x7f1f4c132d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:04:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f4ad9d640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f1f4c13b020 msgr2=0x7f1f4c13d410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.880+0000 7f1f4ad9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f4c13b020 0x7f1f4c13d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1f4c0a03d0 0x7f1f4c13aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1f4c0a03d0 0x7f1f4c13aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:36872/0 (socket says 172.21.15.50:36872) 2023-10-30T12:04:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 -- 172.21.15.50:0/2790006958 learned_addr learned my addr 172.21.15.50:0/2790006958 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): get_auth_request con 0x7f1f4c0a03d0 auth_method 0 2023-10-30T12:04:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient(hunting): handle_auth_done global_id 28228 payload 274 2023-10-30T12:04:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 -- 172.21.15.50:0/2790006958 >> v2:172.21.15.50:3300/0 conn(0x7f1f4c13b020 msgr2=0x7f1f4c13d410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 --2- 172.21.15.50:0/2790006958 >> v2:172.21.15.50:3300/0 conn(0x7f1f4c13b020 0x7f1f4c13d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 -- 172.21.15.50:0/2790006958 >> v1:172.21.15.50:6789/0 conn(0x7f1f4c132970 legacy=0x7f1f4c132d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:04:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 monclient: found mon.noname-c 2023-10-30T12:04:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:04:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.881+0000 7f1f41d9b640 1 -- 172.21.15.50:0/2790006958 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c09f680 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f41d9b640 1 --2- 172.21.15.50:0/2790006958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f4c0a03d0 0x7f1f4c13aae0 secure :-1 s=READY pgs=7116 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c1357c0 tx=0x7f1f3402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19cde0df4f95bb3a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 1 -- 172.21.15.50:0/2790006958 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34031770 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: dump: 2023-10-30T12:04:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.901 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: _finish_auth 0 2023-10-30T12:04:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:04:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 1 -- 172.21.15.50:0/2790006958 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f340318d0 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 5 monclient: authenticate success, global_id 28228 2023-10-30T12:04:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 4 set_mon_vals no callback set 2023-10-30T12:04:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 1 -- 172.21.15.50:0/2790006958 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34031c60 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f49d9b640 10 monclient: dump: 2023-10-30T12:04:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.905 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 10 monclient: shutdown 2023-10-30T12:04:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2790006958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f4c0a03d0 msgr2=0x7f1f4c13aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 1 --2- 172.21.15.50:0/2790006958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f4c0a03d0 0x7f1f4c13aae0 secure :-1 s=READY pgs=7116 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c1357c0 tx=0x7f1f3402b810 comp rx=0 tx=0).stop 2023-10-30T12:04:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2790006958 shutdown_connections 2023-10-30T12:04:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 1 --2- 172.21.15.50:0/2790006958 >> v2:172.21.15.50:3300/0 conn(0x7f1f4c13b020 0x7f1f4c13d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 1 --2- 172.21.15.50:0/2790006958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f4c0a03d0 0x7f1f4c13aae0 unknown :-1 s=CLOSED pgs=7116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2790006958 >> 172.21.15.50:0/2790006958 conn(0x7f1f4c0a4800 msgr2=0x7f1f4c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.882+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2790006958 shutdown_connections 2023-10-30T12:04:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.883+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2790006958 wait complete. 2023-10-30T12:04:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.883+0000 7f1f50a7c640 10 monclient: build_initial_monmap 2023-10-30T12:04:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.883+0000 7f1f50a7c640 10 monclient: monmap: 2023-10-30T12:04:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:38.909 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.883+0000 7f1f50a7c640 1 librados: starting msgr at 2023-10-30T12:04:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.883+0000 7f1f50a7c640 1 librados: starting objecter 2023-10-30T12:04:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 Processor -- start 2023-10-30T12:04:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 -- start start 2023-10-30T12:04:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 librados: setting wanted keys 2023-10-30T12:04:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 librados: calling monclient init 2023-10-30T12:04:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient: init 2023-10-30T12:04:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:04:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4c132970 0x7f1f4c12d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4c132970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f4c13b020 0x7f1f4c12dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4c13b020 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1c7c60 0x7f1f4c1ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c1c7c60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c138e00 con 0x7f1f4c132970 2023-10-30T12:04:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c09d7c0 con 0x7f1f4c13b020 2023-10-30T12:04:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c0a2df0 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f50a7c640 10 monclient(hunting): authenticate will time out at 1124.067596s 2023-10-30T12:04:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4a59c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4c132970 msgr2=0x7f1f4c12d960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4c132970 0x7f1f4c12d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1c7c60 0x7f1f4c1ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1c7c60 0x7f1f4c1ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36264/0 (socket says 172.21.15.50:36264) 2023-10-30T12:04:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 -- 172.21.15.50:0/2869659610 learned_addr learned my addr 172.21.15.50:0/2869659610 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): get_auth_request con 0x7f1f4c1c7c60 auth_method 0 2023-10-30T12:04:38.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:38.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:38.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient(hunting): handle_auth_done global_id 44576 payload 824 2023-10-30T12:04:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f4c13b020 msgr2=0x7f1f4c12dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f4c13b020 0x7f1f4c12dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4c132970 msgr2=0x7f1f4c12d960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:04:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f4c132970 0x7f1f4c12d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:04:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 1 monclient: found mon.noname-c 2023-10-30T12:04:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.884+0000 7f1f4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f4ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f4ad9d640 1 -- 172.21.15.50:0/2869659610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f340094d0 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f4ad9d640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1c7c60 0x7f1f4c1ca6f0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400be90 tx=0x7f1f4400bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 client.?.objecter ms_handle_connect 0x7f1f4c1c7c60 2023-10-30T12:04:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44016430 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: dump: 2023-10-30T12:04:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.921 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: _finish_auth 0 2023-10-30T12:04:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: _check_auth_tickets 2023-10-30T12:04:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 5 monclient: authenticate success, global_id 44576 2023-10-30T12:04:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f44016590 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 10 monclient: _renew_subs 2023-10-30T12:04:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c139ec0 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44016980 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f437fe640 10 monclient: dump: 2023-10-30T12:04:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.925 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 10 client.44576.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 10 monclient: _renew_subs 2023-10-30T12:04:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f4959a640 4 set_mon_vals no callback set 2023-10-30T12:04:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c1c7460 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.885+0000 7f1f50a7c640 1 librados: init done 2023-10-30T12:04:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f50a7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f50a7c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c0a03d0 0x7f1f4c1cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f50a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c071a70 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1f44012020 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c0a03d0 0x7f1f4c1cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient: get_auth_request con 0x7f1f4c0a03d0 auth_method 0 2023-10-30T12:04:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:38.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:38.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:38.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:38.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f2c0392a0 0x7f1f2c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1f4404aef0 con 0x7f1f4c1c7c60 2023-10-30T12:04:38.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 10 client.44576.objecter ms_dispatch 0x7f1f4c000b90 osd_map(115..115 src has 1..115) v4 2023-10-30T12:04:38.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 3 client.44576.objecter handle_osd_map got epochs [115,115] > 0 2023-10-30T12:04:38.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 3 client.44576.objecter handle_osd_map decoding full epoch 115 2023-10-30T12:04:38.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_done global_id 44579 payload 824 2023-10-30T12:04:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 1 -- 172.21.15.50:0/2869659610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f4c09f960 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 20 client.44576.objecter dump_active .. 0 homeless 2023-10-30T12:04:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 10 client.44576.objecter ms_handle_connect 0x7f1f4c0a03d0 2023-10-30T12:04:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 10 client.44576.objecter resend_mon_ops 2023-10-30T12:04:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f4a59c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c0a03d0 0x7f1f4c1cac30 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c003980 tx=0x7f1f3c00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f41d9b640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f2c0392a0 0x7f1f2c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c003e10 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f437fe640 10 monclient: dump: 2023-10-30T12:04:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.933 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.933 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.933 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.886+0000 7f1f41d9b640 10 monclient: get_auth_request con 0x7f1f2c0392a0 auth_method 0 2023-10-30T12:04:38.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1f3c005070 con 0x7f1f4c0a03d0 2023-10-30T12:04:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c0a03d0 msgr2=0x7f1f4c1cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c0a03d0 0x7f1f4c1cac30 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c003980 tx=0x7f1f3c00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c0a03d0 msgr2=0x7f1f4c1cac30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c0a03d0 0x7f1f4c1cac30 secure :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c003980 tx=0x7f1f3c00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:04:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f437fe640 10 client.44576.objecter ms_handle_connect 0x7f1f2c0392a0 2023-10-30T12:04:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.887+0000 7f1f41d9b640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f2c0392a0 0x7f1f2c03b760 secure :-1 s=READY pgs=3920 cs=0 l=1 rev1=1 crypto rx=0x7f1f34036000 tx=0x7f1f34030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f50a7c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f50a7c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1cb170 0x7f1f4c12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f50a7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c074800 con 0x7f1f4c1cb170 2023-10-30T12:04:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1cb170 0x7f1f4c12e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient: get_auth_request con 0x7f1f4c1cb170 auth_method 0 2023-10-30T12:04:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 10 monclient(hunting): handle_auth_done global_id 44582 payload 824 2023-10-30T12:04:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.893+0000 7f1f4a59c640 1 -- 172.21.15.50:0/2869659610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1f4c1cb580 con 0x7f1f4c1cb170 2023-10-30T12:04:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 10 client.44576.objecter ms_handle_connect 0x7f1f4c1cb170 2023-10-30T12:04:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f4a59c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1cb170 0x7f1f4c12e5f0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c012040 tx=0x7f1f3c005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 10 client.44576.objecter resend_mon_ops 2023-10-30T12:04:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c020070 con 0x7f1f4c1cb170 2023-10-30T12:04:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 10 monclient: dump: 2023-10-30T12:04:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:38.942 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f1f3c010400 con 0x7f1f4c1cb170 2023-10-30T12:04:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1cb170 msgr2=0x7f1f4c12e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1cb170 0x7f1f4c12e5f0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c012040 tx=0x7f1f3c005a20 comp rx=0 tx=0).stop 2023-10-30T12:04:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1cb170 msgr2=0x7f1f4c12e5f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f437fe640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1cb170 0x7f1f4c12e5f0 secure :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c012040 tx=0x7f1f3c005a20 comp rx=0 tx=0).stop 2023-10-30T12:04:38.944 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:38.944 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:38.944 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:38.944 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:04:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 42, 2023-10-30T12:04:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:04:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:38.955 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:38.955 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:38.955 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:38.955 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:38.955 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:38.956 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:38.956 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:38.956 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:38.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.956 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:38.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.957 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:38.957 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:38.957 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:38.957 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:38.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:38.958 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:38.958 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:38.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.958 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:38.959 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:38.959 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:38.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:38.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:38.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:38.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:38.961 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:38.963 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:38.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.964 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:38.964 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:38.964 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:38.964 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:38.964 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:38.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.965 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:38.965 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:38.965 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:38.966 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:38.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.966 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:38.966 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:38.966 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:38.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.967 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:38.967 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:38.967 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:38.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:38.968 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:38.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.968 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:38.968 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:38.968 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:38.969 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:38.969 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:38.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:38.969 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:38.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.970 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:38.970 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:38.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.970 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:38.970 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:38.970 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:38.971 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:38.971 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:38.971 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:04:38.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.971 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:38.971 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:04:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:04:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:38.974 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 10 librados: watch_flush enter 2023-10-30T12:04:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 10 librados: watch_flush exit 2023-10-30T12:04:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 20 client.44576.objecter shutdown clearing up homeless session... 2023-10-30T12:04:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 10 client.44576.objecter successfully canceled tick 2023-10-30T12:04:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f2c0392a0 msgr2=0x7f1f2c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f2c0392a0 0x7f1f2c03b760 secure :-1 s=READY pgs=3920 cs=0 l=1 rev1=1 crypto rx=0x7f1f34036000 tx=0x7f1f34030000 comp rx=0 tx=0).stop 2023-10-30T12:04:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 10 monclient: shutdown 2023-10-30T12:04:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1c7c60 msgr2=0x7f1f4c1ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 --2- 172.21.15.50:0/2869659610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f4c1c7c60 0x7f1f4c1ca6f0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400be90 tx=0x7f1f4400bf80 comp rx=0 tx=0).stop 2023-10-30T12:04:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f4ad9d640 1 -- 172.21.15.50:0/2869659610 reap_dead start 2023-10-30T12:04:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f4ad9d640 1 -- 172.21.15.50:0/2869659610 reap_dead start 2023-10-30T12:04:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 shutdown_connections 2023-10-30T12:04:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 >> 172.21.15.50:0/2869659610 conn(0x7f1f4c0a4800 msgr2=0x7f1f4c09c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 shutdown_connections 2023-10-30T12:04:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 -- 172.21.15.50:0/2869659610 wait complete. 2023-10-30T12:04:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:38.894+0000 7f1f50a7c640 1 librados: shutdown 2023-10-30T12:04:38.979 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:04:38.979 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:04:38.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:04:39.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.085+0000 7efdb5a61640 10 monclient: get_monmap_and_config 2023-10-30T12:04:39.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.085+0000 7efdb5a61640 10 monclient: build_initial_monmap 2023-10-30T12:04:39.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.085+0000 7efdb5a61640 10 monclient: monmap: 2023-10-30T12:04:39.087 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:39.087 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:39.087 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:04:39.086341+0000 2023-10-30T12:04:39.088 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:04:39.086341+0000 2023-10-30T12:04:39.088 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:39.088 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.088 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:04:39.088 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:04:39.089 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:04:39.089 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:04:39.089 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:04:39.089 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.085+0000 7efdb5a61640 1 Processor -- start 2023-10-30T12:04:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.085+0000 7efdb5a61640 1 -- start start 2023-10-30T12:04:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.085+0000 7efdb5a61640 10 monclient: init 2023-10-30T12:04:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T12:04:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efdb006afc0 0x7efdb00a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient(hunting): picked mon.noname-a con 0x7efdb006afc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:04:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdb006b640 addr v1:172.21.15.118:6789/0 2023-10-30T12:04:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb013d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient(hunting): picked mon.noname-b con 0x7efdb013b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb006b3a0 con 0x7efdb006b640 2023-10-30T12:04:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb009e540 con 0x7efdb006afc0 2023-10-30T12:04:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb009fa10 con 0x7efdb013b070 2023-10-30T12:04:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdb5a61640 10 monclient(hunting): authenticate will time out at 1124.269367s 2023-10-30T12:04:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaeffd640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7efdb006afc0 msgr2=0x7efdb00a1c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaeffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efdb006afc0 0x7efdb00a1c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb013d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb013d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36294/0 (socket says 172.21.15.50:36294) 2023-10-30T12:04:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 -- 172.21.15.50:0/3769240803 learned_addr learned my addr 172.21.15.50:0/3769240803 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): get_auth_request con 0x7efdb013b070 auth_method 0 2023-10-30T12:04:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_done global_id 44585 payload 274 2023-10-30T12:04:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 -- 172.21.15.50:0/3769240803 >> v2:172.21.15.50:3300/0 conn(0x7efdb006afc0 msgr2=0x7efdb00a1c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:04:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 --2- 172.21.15.50:0/3769240803 >> v2:172.21.15.50:3300/0 conn(0x7efdb006afc0 0x7efdb00a1c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 -- 172.21.15.50:0/3769240803 >> v1:172.21.15.118:6789/0 conn(0x7efdb006b640 legacy=0x7efdb00a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:04:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 monclient: found mon.noname-b 2023-10-30T12:04:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:04:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.086+0000 7efdaf7fe640 1 -- 172.21.15.50:0/3769240803 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb006a6f0 con 0x7efdb013b070 2023-10-30T12:04:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdaf7fe640 1 --2- 172.21.15.50:0/3769240803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb013d460 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7efda0009e50 tx=0x7efda002be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=844508d40f7b5c80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 1 -- 172.21.15.50:0/3769240803 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda002ca10 con 0x7efdb013b070 2023-10-30T12:04:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:04:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: dump: 2023-10-30T12:04:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:39.107 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: _finish_auth 0 2023-10-30T12:04:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: _check_auth_tickets 2023-10-30T12:04:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdb5a61640 5 monclient: authenticate success, global_id 44585 2023-10-30T12:04:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdb5a61640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:04:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 1 -- 172.21.15.50:0/3769240803 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda002cb70 con 0x7efdb013b070 2023-10-30T12:04:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdb5a61640 10 monclient: get_monmap_and_config success 2023-10-30T12:04:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdb5a61640 4 set_mon_vals no callback set 2023-10-30T12:04:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 1 -- 172.21.15.50:0/3769240803 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda00343f0 con 0x7efdb013b070 2023-10-30T12:04:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.087+0000 7efdadffb640 10 monclient: dump: 2023-10-30T12:04:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:39.111 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 10 monclient: shutdown 2023-10-30T12:04:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 -- 172.21.15.50:0/3769240803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 msgr2=0x7efdb013d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 --2- 172.21.15.50:0/3769240803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb013d460 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7efda0009e50 tx=0x7efda002be70 comp rx=0 tx=0).stop 2023-10-30T12:04:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 -- 172.21.15.50:0/3769240803 shutdown_connections 2023-10-30T12:04:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 --2- 172.21.15.50:0/3769240803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb013d460 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 --2- 172.21.15.50:0/3769240803 >> v2:172.21.15.50:3300/0 conn(0x7efdb006afc0 0x7efdb00a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 -- 172.21.15.50:0/3769240803 >> 172.21.15.50:0/3769240803 conn(0x7efdb00a6200 msgr2=0x7efdb00a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 -- 172.21.15.50:0/3769240803 shutdown_connections 2023-10-30T12:04:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 1 -- 172.21.15.50:0/3769240803 wait complete. 2023-10-30T12:04:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 10 monclient: build_initial_monmap 2023-10-30T12:04:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.088+0000 7efdb5a61640 10 monclient: monmap: 2023-10-30T12:04:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:04:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:04:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:04:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:04:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:04:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:04:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:04:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:04:39.115 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 librados: starting msgr at 2023-10-30T12:04:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 librados: starting objecter 2023-10-30T12:04:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 Processor -- start 2023-10-30T12:04:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 -- start start 2023-10-30T12:04:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 librados: setting wanted keys 2023-10-30T12:04:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 librados: calling monclient init 2023-10-30T12:04:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient: init 2023-10-30T12:04:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient: _reopen_session rank -1 2023-10-30T12:04:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:04:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb006afc0 0x7efdb011cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): picked mon.noname-b con 0x7efdb006afc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:04:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdb006b640 0x7efdb011d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): picked mon.noname-a con 0x7efdb006b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:04:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb011d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): picked mon.noname-c con 0x7efdb013b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:04:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb01204c0 con 0x7efdb006b640 2023-10-30T12:04:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0070de0 con 0x7efdb006afc0 2023-10-30T12:04:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0138440 con 0x7efdb013b070 2023-10-30T12:04:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): _renew_subs 2023-10-30T12:04:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdb5a61640 10 monclient(hunting): authenticate will time out at 1124.272803s 2023-10-30T12:04:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdae7fc640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdb006b640 msgr2=0x7efdb011d430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:04:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdae7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdb006b640 0x7efdb011d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:04:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb011d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb011d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36304/0 (socket says 172.21.15.50:36304) 2023-10-30T12:04:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 1 -- 172.21.15.50:0/2870408655 learned_addr learned my addr 172.21.15.50:0/2870408655 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:04:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 10 monclient(hunting): get_auth_request con 0x7efdb013b070 auth_method 0 2023-10-30T12:04:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.089+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 10 monclient(hunting): handle_auth_done global_id 44588 payload 824 2023-10-30T12:04:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb006afc0 msgr2=0x7efdb011cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaeffd640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb006afc0 0x7efdb011cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb006afc0 0x7efdb011cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdb006b640 msgr2=0x7efdb011d430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:04:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdb006b640 0x7efdb011d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:04:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 1 monclient: found mon.noname-c 2023-10-30T12:04:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:04:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:04:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 1 -- 172.21.15.50:0/2870408655 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0009b30 con 0x7efdb013b070 2023-10-30T12:04:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaeffd640 10 monclient: get_auth_request con 0x7efdb006afc0 auth_method 0 2023-10-30T12:04:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaeffd640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb006afc0 0x7efdb011cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:04:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdaf7fe640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb011d970 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7efda0009e50 tx=0x7efda0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 client.?.objecter ms_handle_connect 0x7efdb013b070 2023-10-30T12:04:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:04:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda003e450 con 0x7efdb013b070 2023-10-30T12:04:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:04:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: dump: 2023-10-30T12:04:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:39.127 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: _finish_auth 0 2023-10-30T12:04:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:04:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:04:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda003e5d0 con 0x7efdb013b070 2023-10-30T12:04:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 5 monclient: authenticate success, global_id 44588 2023-10-30T12:04:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 10 monclient: _renew_subs 2023-10-30T12:04:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb011fc20 con 0x7efdb013b070 2023-10-30T12:04:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:04:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda003e980 con 0x7efdb013b070 2023-10-30T12:04:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:04:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efda7fff640 10 monclient: dump: 2023-10-30T12:04:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:39.132 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdad7fa640 4 set_mon_vals no callback set 2023-10-30T12:04:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 10 client.44588.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:04:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 10 monclient: _renew_subs 2023-10-30T12:04:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:04:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb0136d20 con 0x7efdb013b070 2023-10-30T12:04:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.090+0000 7efdb5a61640 1 librados: init done 2023-10-30T12:04:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdb5a61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdb5a61640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb011e130 0x7efdb01ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdb5a61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0120ac0 con 0x7efdb011e130 2023-10-30T12:04:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efda003eae0 con 0x7efdb013b070 2023-10-30T12:04:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb011e130 0x7efdb01ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient: get_auth_request con 0x7efdb011e130 auth_method 0 2023-10-30T12:04:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efda7fff640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd900392a0 0x7efd9003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6115+0+0 (secure 0 0 0) 0x7efda006da00 con 0x7efdb013b070 2023-10-30T12:04:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efda7fff640 10 client.44588.objecter ms_dispatch 0x7efdb0000b90 osd_map(115..115 src has 1..115) v4 2023-10-30T12:04:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efda7fff640 3 client.44588.objecter handle_osd_map got epochs [115,115] > 0 2023-10-30T12:04:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efda7fff640 3 client.44588.objecter handle_osd_map decoding full epoch 115 2023-10-30T12:04:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efda7fff640 20 client.44588.objecter dump_active .. 0 homeless 2023-10-30T12:04:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdae7fc640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd900392a0 0x7efd9003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 10 monclient(hunting): handle_auth_done global_id 28231 payload 824 2023-10-30T12:04:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdaeffd640 1 -- 172.21.15.50:0/2870408655 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdb009f0b0 con 0x7efdb011e130 2023-10-30T12:04:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.091+0000 7efdae7fc640 10 monclient: get_auth_request con 0x7efd900392a0 auth_method 0 2023-10-30T12:04:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efdaeffd640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb011e130 0x7efdb01ca9c0 secure :-1 s=READY pgs=7118 cs=0 l=1 rev1=1 crypto rx=0x7efda800c6f0 tx=0x7efda800cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 client.44588.objecter ms_handle_connect 0x7efdb011e130 2023-10-30T12:04:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 client.44588.objecter resend_mon_ops 2023-10-30T12:04:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8003bb0 con 0x7efdb011e130 2023-10-30T12:04:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 monclient: dump: 2023-10-30T12:04:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:39.140 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 client.44588.objecter ms_handle_connect 0x7efd900392a0 2023-10-30T12:04:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efdae7fc640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd900392a0 0x7efd9003b760 secure :-1 s=READY pgs=3921 cs=0 l=1 rev1=1 crypto rx=0x7efdb0073770 tx=0x7efd980079f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7efda8003d10 con 0x7efdb011e130 2023-10-30T12:04:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:04:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:04:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb011e130 msgr2=0x7efdb01ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb011e130 0x7efdb01ca9c0 secure :-1 s=READY pgs=7118 cs=0 l=1 rev1=1 crypto rx=0x7efda800c6f0 tx=0x7efda800cbb0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb011e130 msgr2=0x7efdb01ca9c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.092+0000 7efda7fff640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb011e130 0x7efdb01ca9c0 secure :-1 s=CLOSED pgs=7118 cs=0 l=1 rev1=1 crypto rx=0x7efda800c6f0 tx=0x7efda800cbb0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.098+0000 7efdb5a61640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:04:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.098+0000 7efdb5a61640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb002b430 0x7efdb002b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:04:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.098+0000 7efdb5a61640 10 monclient(hunting): start opening mon connection 2023-10-30T12:04:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.098+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0073ff0 con 0x7efdb002b430 2023-10-30T12:04:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb002b430 0x7efdb002b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:04:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient: get_auth_request con 0x7efdb002b430 auth_method 0 2023-10-30T12:04:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:04:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:04:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:04:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:04:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:04:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:04:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 10 monclient(hunting): handle_auth_done global_id 28234 payload 824 2023-10-30T12:04:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 1 -- 172.21.15.50:0/2870408655 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efdb009df80 con 0x7efdb002b430 2023-10-30T12:04:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efdaeffd640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb002b430 0x7efdb002b840 secure :-1 s=READY pgs=7119 cs=0 l=1 rev1=1 crypto rx=0x7efda800afd0 tx=0x7efda80059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:04:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efda7fff640 10 client.44588.objecter ms_handle_connect 0x7efdb002b430 2023-10-30T12:04:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.099+0000 7efda7fff640 10 client.44588.objecter resend_mon_ops 2023-10-30T12:04:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8022070 con 0x7efdb002b430 2023-10-30T12:04:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:04:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:04:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 10 monclient: dump: 2023-10-30T12:04:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:04:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:04:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:04:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:04:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:04:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:04:39.149 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:04:39.149 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:04:39.149 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:04:39.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7efda8007ba0 con 0x7efdb002b430 2023-10-30T12:04:39.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:04:39.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:04:39.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb002b430 msgr2=0x7efdb002b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:39.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb002b430 0x7efdb002b840 secure :-1 s=READY pgs=7119 cs=0 l=1 rev1=1 crypto rx=0x7efda800afd0 tx=0x7efda80059d0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb002b430 msgr2=0x7efdb002b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:04:39.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efda7fff640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdb002b430 0x7efdb002b840 secure :-1 s=CLOSED pgs=7119 cs=0 l=1 rev1=1 crypto rx=0x7efda800afd0 tx=0x7efda80059d0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.151 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:04:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:04:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 42, 2023-10-30T12:04:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:04:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:04:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:04:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:04:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:04:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:04:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:04:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:04:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:04:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:04:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:04:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:04:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:04:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:04:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:04:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:04:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:04:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:04:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:04:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:04:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:04:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:04:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:04:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:04:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:04:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:04:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:04:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:04:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:04:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:04:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:04:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:04:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:04:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:04:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:04:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:04:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:04:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:04:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:04:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:04:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:04:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:04:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:04:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:04:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:04:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:04:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:04:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:04:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:04:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:04:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:04:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:04:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:04:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:04:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:04:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:04:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:04:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:04:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:04:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:04:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:04:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:04:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:04:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:04:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:04:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:04:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:04:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 8 2023-10-30T12:04:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:04:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:04:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:04:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:04:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:04:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:04:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:04:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:04:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:04:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:04:39.183 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:04:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 10 librados: watch_flush enter 2023-10-30T12:04:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 10 librados: watch_flush exit 2023-10-30T12:04:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 20 client.44588.objecter shutdown clearing up homeless session... 2023-10-30T12:04:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 10 client.44588.objecter successfully canceled tick 2023-10-30T12:04:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd900392a0 msgr2=0x7efd9003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efd900392a0 0x7efd9003b760 secure :-1 s=READY pgs=3921 cs=0 l=1 rev1=1 crypto rx=0x7efdb0073770 tx=0x7efd980079f0 comp rx=0 tx=0).stop 2023-10-30T12:04:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 10 monclient: shutdown 2023-10-30T12:04:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:04:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 msgr2=0x7efdb011d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:04:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 --2- 172.21.15.50:0/2870408655 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdb013b070 0x7efdb011d970 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7efda0009e50 tx=0x7efda0004a40 comp rx=0 tx=0).stop 2023-10-30T12:04:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdaf7fe640 1 -- 172.21.15.50:0/2870408655 reap_dead start 2023-10-30T12:04:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdaf7fe640 1 -- 172.21.15.50:0/2870408655 reap_dead start 2023-10-30T12:04:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 shutdown_connections 2023-10-30T12:04:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 >> 172.21.15.50:0/2870408655 conn(0x7efdb00a6200 msgr2=0x7efdb00705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:04:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 shutdown_connections 2023-10-30T12:04:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 -- 172.21.15.50:0/2870408655 wait complete. 2023-10-30T12:04:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:04:39.100+0000 7efdb5a61640 1 librados: shutdown 2023-10-30T12:04:39.188 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:04:39.188 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:04:49.447 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: Running main() from gmock_main.cc 2023-10-30T12:04:49.447 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-10-30T12:04:49.447 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-10-30T12:04:49.448 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-10-30T12:04:49.448 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-10-30T12:04:49.448 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (5961 ms) 2023-10-30T12:04:49.448 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-10-30T12:04:49.448 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (33590 ms) 2023-10-30T12:04:49.449 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-10-30T12:04:49.449 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (19364 ms) 2023-10-30T12:04:49.449 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-10-30T12:04:49.449 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3005 ms) 2023-10-30T12:04:49.449 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-10-30T12:04:49.450 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3031 ms) 2023-10-30T12:04:49.450 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-10-30T12:04:49.450 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2969 ms) 2023-10-30T12:04:49.450 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-10-30T12:04:49.450 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2991 ms) 2023-10-30T12:04:49.451 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-10-30T12:04:49.451 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3010 ms) 2023-10-30T12:04:49.451 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-10-30T12:04:49.451 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2996 ms) 2023-10-30T12:04:49.451 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-10-30T12:04:49.452 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3005 ms) 2023-10-30T12:04:49.452 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-10-30T12:04:49.452 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (7821 ms) 2023-10-30T12:04:49.452 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-10-30T12:04:49.453 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3427 ms) 2023-10-30T12:04:49.453 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-10-30T12:04:49.453 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (17116 ms) 2023-10-30T12:04:49.453 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-10-30T12:04:49.453 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2630 ms) 2023-10-30T12:04:49.454 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-10-30T12:04:49.454 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3009 ms) 2023-10-30T12:04:49.454 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-10-30T12:04:49.454 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2997 ms) 2023-10-30T12:04:49.454 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-10-30T12:04:49.455 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3005 ms) 2023-10-30T12:04:49.455 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-10-30T12:04:49.455 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2997 ms) 2023-10-30T12:04:49.455 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-10-30T12:04:49.455 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3038 ms) 2023-10-30T12:04:49.456 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-10-30T12:04:49.456 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2985 ms) 2023-10-30T12:04:49.456 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-10-30T12:04:49.456 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2983 ms) 2023-10-30T12:04:49.456 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-10-30T12:04:49.457 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (8901 ms) 2023-10-30T12:04:49.457 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-10-30T12:04:49.457 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2982 ms) 2023-10-30T12:04:49.457 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-10-30T12:04:49.457 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (22537 ms) 2023-10-30T12:04:49.458 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-10-30T12:04:49.458 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2997 ms) 2023-10-30T12:04:49.458 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-10-30T12:04:49.458 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3015 ms) 2023-10-30T12:04:49.458 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-10-30T12:04:59.189 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:04:59.190 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:04:59.193 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:04:59.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:04:59.197 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:04:59.197 INFO:tasks.mon_thrash:Sending STOP to mon b 2023-10-30T12:04:59.197 INFO:tasks.ceph.mon.b:Sent signal 19 2023-10-30T12:04:59.198 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-10-30T12:05:14.199 INFO:tasks.mon_thrash:Sending CONT to mon b 2023-10-30T12:05:14.199 INFO:tasks.ceph.mon.b:Sent signal 18 2023-10-30T12:05:14.200 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:05:14.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:14.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.314+0000 7f7755610640 10 monclient: get_monmap_and_config 2023-10-30T12:05:14.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.314+0000 7f7755610640 10 monclient: build_initial_monmap 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.314+0000 7f7755610640 10 monclient: monmap: 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:14.316007+0000 2023-10-30T12:05:14.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:14.316007+0000 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:14.319 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:14.319 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:14.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.315+0000 7f7755610640 1 Processor -- start 2023-10-30T12:05:14.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.315+0000 7f7755610640 1 -- start start 2023-10-30T12:05:14.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.315+0000 7f7755610640 10 monclient: init 2023-10-30T12:05:14.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.315+0000 7f7755610640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:14.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.315+0000 7f7755610640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-10-30T12:05:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.315+0000 7f7755610640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 10 monclient(hunting): picked mon.noname-a con 0x7f7750098520 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 10 monclient(hunting): picked mon.noname-c con 0x7f7750098900 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7750132740 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775009ad30 con 0x7f7750132740 2023-10-30T12:05:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775009c460 con 0x7f7750098520 2023-10-30T12:05:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775009b260 con 0x7f7750098900 2023-10-30T12:05:14.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:14.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f7755610640 10 monclient(hunting): authenticate will time out at 1159.499060s 2023-10-30T12:05:14.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37784/0 (socket says 172.21.15.50:37784) 2023-10-30T12:05:14.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 1 -- 172.21.15.50:0/3476503544 learned_addr learned my addr 172.21.15.50:0/3476503544 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 10 monclient(hunting): get_auth_request con 0x7f7750098520 auth_method 0 2023-10-30T12:05:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 msgr2=0x7f775009bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:05:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 msgr2=0x7f775009bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774effd640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 10 monclient(hunting): get_auth_request con 0x7f7750098900 auth_method 0 2023-10-30T12:05:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 msgr2=0x7f7750098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:05:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 msgr2=0x7f7750098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.316+0000 7f774e7fc640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:14.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 10 monclient(hunting): get_auth_request con 0x7f7750098520 auth_method 0 2023-10-30T12:05:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 msgr2=0x7f775009bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:05:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 msgr2=0x7f775009bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774effd640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:05:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 10 monclient(hunting): get_auth_request con 0x7f7750098900 auth_method 0 2023-10-30T12:05:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 msgr2=0x7f7750098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:05:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 msgr2=0x7f7750098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.517+0000 7f774e7fc640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:05:14.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.623+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196797259 0 0) 0x7f775009ad30 con 0x7f7750132740 2023-10-30T12:05:14.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.624+0000 7f774dffb640 10 monclient(hunting): _negotiate 2023-10-30T12:05:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.624+0000 7f774dffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.624+0000 7f774dffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.624+0000 7f774dffb640 10 monclient(hunting): my global_id is 28300 2023-10-30T12:05:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.624+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c005730 con 0x7f7750132740 2023-10-30T12:05:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2438808508 0 0) 0x7f773c005730 con 0x7f7750132740 2023-10-30T12:05:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 msgr2=0x7f7750098d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.118:3300/0 conn(0x7f7750098900 0x7f7750098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 msgr2=0x7f775009bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 --2- 172.21.15.50:0/3476503544 >> v2:172.21.15.50:3300/0 conn(0x7f7750098520 0x7f775009bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: _finish_hunting 0 2023-10-30T12:05:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:05:14.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:14.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:05:14.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750072120 con 0x7f7750132740 2023-10-30T12:05:14.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: _finish_auth 0 2023-10-30T12:05:14.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:05:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f7748001670 con 0x7f7750132740 2023-10-30T12:05:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:05:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: dump: 2023-10-30T12:05:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:14.665 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:05:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 >> v1:172.21.15.118:6789/0 conn(0x7f7750132740 legacy=0x7f7750132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:05:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c005bd0 0x7f773c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): picked mon.b con 0x7f773c005bd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f773c0069d0 0x7f773c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): picked mon.a con 0x7f773c0069d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f773c0074b0 0x7f773c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): picked mon.c con 0x7f773c0074b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775009c460 con 0x7f773c0069d0 2023-10-30T12:05:14.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775009b260 con 0x7f773c005bd0 2023-10-30T12:05:14.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c00e400 con 0x7f773c0074b0 2023-10-30T12:05:14.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:14.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774dffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:05:14.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774e7fc640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f773c0069d0 0x7f773c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.655+0000 7f774f7fe640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f773c0074b0 0x7f773c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): get_auth_request con 0x7f773c0074b0 auth_method 0 2023-10-30T12:05:14.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:14.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774e7fc640 10 monclient(hunting): get_auth_request con 0x7f773c0069d0 auth_method 0 2023-10-30T12:05:14.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:14.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c005bd0 0x7f773c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 10 monclient(hunting): get_auth_request con 0x7f773c005bd0 auth_method 0 2023-10-30T12:05:14.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:05:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:05:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:05:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient(hunting): handle_auth_done global_id 28300 payload 293 2023-10-30T12:05:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 1 -- 172.21.15.50:0/3476503544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c005bd0 msgr2=0x7f773c005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c005bd0 0x7f773c005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 1 -- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f773c0069d0 msgr2=0x7f773c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f773c0069d0 0x7f773c006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:05:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 1 monclient: found mon.c 2023-10-30T12:05:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 1 -- 172.21.15.50:0/3476503544 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c005730 con 0x7f773c0074b0 2023-10-30T12:05:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:05:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:05:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f774f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f7755610640 5 monclient: authenticate success, global_id 28300 2023-10-30T12:05:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.656+0000 7f7755610640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774f7fe640 1 -- 172.21.15.50:0/3476503544 reap_dead start 2023-10-30T12:05:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774f7fe640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f773c0074b0 0x7f773c00db90 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7748007f20 tx=0x7f773c007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ac74440f588c3f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774802c940 con 0x7f773c0074b0 2023-10-30T12:05:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 10 monclient: dump: 2023-10-30T12:05:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:14.679 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:14.679 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:14.679 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:14.679 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:14.679 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f7755610640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774802caa0 con 0x7f773c0074b0 2023-10-30T12:05:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f7755610640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 1 -- 172.21.15.50:0/3476503544 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774802ce50 con 0x7f773c0074b0 2023-10-30T12:05:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:14.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f7755610640 4 set_mon_vals no callback set 2023-10-30T12:05:14.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:14.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.657+0000 7f774dffb640 10 monclient: dump: 2023-10-30T12:05:14.681 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:14.681 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:14.681 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.682 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.682 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:14.682 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:14.682 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:14.682 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:14.683 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:14.683 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:14.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 10 monclient: shutdown 2023-10-30T12:05:14.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:14.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 1 -- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f773c0074b0 msgr2=0x7f773c00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:14.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f773c0074b0 0x7f773c00db90 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7748007f20 tx=0x7f773c007a00 comp rx=0 tx=0).stop 2023-10-30T12:05:14.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 1 -- 172.21.15.50:0/3476503544 shutdown_connections 2023-10-30T12:05:14.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 1 --2- 172.21.15.50:0/3476503544 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f773c0074b0 0x7f773c00db90 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 1 -- 172.21.15.50:0/3476503544 >> 172.21.15.50:0/3476503544 conn(0x7f77500a05c0 msgr2=0x7f775009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:14.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 1 -- 172.21.15.50:0/3476503544 shutdown_connections 2023-10-30T12:05:14.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.658+0000 7f7755610640 1 -- 172.21.15.50:0/3476503544 wait complete. 2023-10-30T12:05:14.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.659+0000 7f7755610640 10 monclient: build_initial_monmap 2023-10-30T12:05:14.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.659+0000 7f7755610640 10 monclient: monmap: 2023-10-30T12:05:14.685 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:14.685 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:14.687 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:14.687 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:14.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.659+0000 7f7755610640 1 librados: starting msgr at 2023-10-30T12:05:14.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.659+0000 7f7755610640 1 librados: starting objecter 2023-10-30T12:05:14.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 Processor -- start 2023-10-30T12:05:14.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 -- start start 2023-10-30T12:05:14.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 librados: setting wanted keys 2023-10-30T12:05:14.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 librados: calling monclient init 2023-10-30T12:05:14.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient: init 2023-10-30T12:05:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:05:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c0074b0 0x7f77501c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): picked mon.noname-b con 0x7f773c0074b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f77501c6460 0x7f775012a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): picked mon.noname-c con 0x7f77501c6460 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:14.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f775012a570 0x7f775012aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): picked mon.noname-a con 0x7f775012a570 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:14.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77501c6880 con 0x7f775012a570 2023-10-30T12:05:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775012c830 con 0x7f773c0074b0 2023-10-30T12:05:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77500a2da0 con 0x7f77501c6460 2023-10-30T12:05:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:14.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f7755610640 10 monclient(hunting): authenticate will time out at 1159.843617s 2023-10-30T12:05:14.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f77501c6460 0x7f775012a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f775012a570 0x7f775012aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f775012a570 0x7f775012aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37820/0 (socket says 172.21.15.50:37820) 2023-10-30T12:05:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 1 -- 172.21.15.50:0/2977721706 learned_addr learned my addr 172.21.15.50:0/2977721706 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 10 monclient(hunting): get_auth_request con 0x7f77501c6460 auth_method 0 2023-10-30T12:05:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 10 monclient(hunting): get_auth_request con 0x7f775012a570 auth_method 0 2023-10-30T12:05:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774effd640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c0074b0 0x7f77501c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:14.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:14.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:14.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774effd640 10 monclient(hunting): get_auth_request con 0x7f773c0074b0 auth_method 0 2023-10-30T12:05:14.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:14.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.660+0000 7f774f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:14.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:14.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:14.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 10 monclient(hunting): handle_auth_done global_id 44630 payload 824 2023-10-30T12:05:14.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 1 -- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c0074b0 msgr2=0x7f77501c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:14.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c0074b0 0x7f77501c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 1 -- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f775012a570 msgr2=0x7f775012aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:14.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f775012a570 0x7f775012aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:05:14.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 1 monclient: found mon.noname-c 2023-10-30T12:05:14.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:14.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:05:14.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 1 -- 172.21.15.50:0/2977721706 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750071df0 con 0x7f77501c6460 2023-10-30T12:05:14.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774f7fe640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f775012a570 0x7f775012aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:14.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774e7fc640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f77501c6460 0x7f775012a030 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f7738003ec0 tx=0x7f773800d920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:14.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 client.?.objecter ms_handle_connect 0x7f77501c6460 2023-10-30T12:05:14.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 1 -- 172.21.15.50:0/2977721706 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7738014740 con 0x7f77501c6460 2023-10-30T12:05:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: dump: 2023-10-30T12:05:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:14.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.705 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.705 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:14.705 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:14.705 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:14.705 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:14.706 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:14.706 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:14.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: _finish_auth 0 2023-10-30T12:05:14.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: _check_auth_tickets 2023-10-30T12:05:14.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:14.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 1 -- 172.21.15.50:0/2977721706 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77380148a0 con 0x7f77501c6460 2023-10-30T12:05:14.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 5 monclient: authenticate success, global_id 44630 2023-10-30T12:05:14.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:14.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 10 monclient: _renew_subs 2023-10-30T12:05:14.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 1 -- 172.21.15.50:0/2977721706 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7738014a30 con 0x7f77501c6460 2023-10-30T12:05:14.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:14.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775012dce0 con 0x7f77501c6460 2023-10-30T12:05:14.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:14.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:14.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f77477fe640 10 monclient: dump: 2023-10-30T12:05:14.708 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:14.709 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:14.709 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.709 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:14.709 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:14.709 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:14.710 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:14.710 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:14.710 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:14.710 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:14.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f774d7fa640 4 set_mon_vals no callback set 2023-10-30T12:05:14.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 10 client.44630.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:14.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 10 monclient: _renew_subs 2023-10-30T12:05:14.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:14.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775012bbf0 con 0x7f77501c6460 2023-10-30T12:05:14.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.661+0000 7f7755610640 1 librados: init done 2023-10-30T12:05:14.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f77477fe640 1 -- 172.21.15.50:0/2977721706 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7738014b90 con 0x7f77501c6460 2023-10-30T12:05:14.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f7755610640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:14.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f7755610640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:14.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f7755610640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:14.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f77477fe640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f77300392f0 0x7f773003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775004e160 con 0x7f77501c6460 2023-10-30T12:05:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f77477fe640 1 -- 172.21.15.50:0/2977721706 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6450+0+0 (secure 0 0 0) 0x7f773804af50 con 0x7f77501c6460 2023-10-30T12:05:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f77477fe640 10 client.44630.objecter ms_dispatch 0x7f7750000b90 osd_map(135..135 src has 1..135) v4 2023-10-30T12:05:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f77477fe640 3 client.44630.objecter handle_osd_map got epochs [135,135] > 0 2023-10-30T12:05:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f77477fe640 3 client.44630.objecter handle_osd_map decoding full epoch 135 2023-10-30T12:05:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.662+0000 7f77477fe640 20 client.44630.objecter dump_active .. 0 homeless 2023-10-30T12:05:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.663+0000 7f774effd640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f77300392f0 0x7f773003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.663+0000 7f774effd640 10 monclient: get_auth_request con 0x7f77300392f0 auth_method 0 2023-10-30T12:05:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.663+0000 7f77477fe640 10 client.44630.objecter ms_handle_connect 0x7f77300392f0 2023-10-30T12:05:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.663+0000 7f774effd640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f77300392f0 0x7f773003b7b0 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7f77400072e0 tx=0x7f7740009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.667+0000 7f77477fe640 1 -- 172.21.15.50:0/2977721706 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7738023020 con 0x7f77501c6460 2023-10-30T12:05:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.667+0000 7f77477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.667+0000 7f77477fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:14.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.917+0000 7f7755610640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:14.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.917+0000 7f7755610640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:14.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.917+0000 7f7755610640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:14.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.917+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7750070980 con 0x7f77501c6460 2023-10-30T12:05:14.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.918+0000 7f77477fe640 1 -- 172.21.15.50:0/2977721706 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f7738015450 con 0x7f77501c6460 2023-10-30T12:05:14.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.918+0000 7f77477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:14.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.918+0000 7f77477fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:14.921 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:14.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 10 librados: watch_flush enter 2023-10-30T12:05:14.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 10 librados: watch_flush exit 2023-10-30T12:05:14.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 20 client.44630.objecter shutdown clearing up homeless session... 2023-10-30T12:05:14.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 10 client.44630.objecter successfully canceled tick 2023-10-30T12:05:14.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f77300392f0 msgr2=0x7f773003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:14.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f77300392f0 0x7f773003b7b0 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7f77400072e0 tx=0x7f7740009040 comp rx=0 tx=0).stop 2023-10-30T12:05:14.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 10 monclient: shutdown 2023-10-30T12:05:14.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:14.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f77501c6460 msgr2=0x7f775012a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:14.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f77501c6460 0x7f775012a030 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f7738003ec0 tx=0x7f773800d920 comp rx=0 tx=0).stop 2023-10-30T12:05:14.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 shutdown_connections 2023-10-30T12:05:14.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f77300392f0 0x7f773003b7b0 unknown :-1 s=CLOSED pgs=3933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f775012a570 0x7f775012aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f77501c6460 0x7f775012a030 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 --2- 172.21.15.50:0/2977721706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f773c0074b0 0x7f77501c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 >> 172.21.15.50:0/2977721706 conn(0x7f77500a05c0 msgr2=0x7f775006db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 shutdown_connections 2023-10-30T12:05:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 -- 172.21.15.50:0/2977721706 wait complete. 2023-10-30T12:05:14.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:14.920+0000 7f7755610640 1 librados: shutdown 2023-10-30T12:05:14.932 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:05:14.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:05:14.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:05:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.037+0000 7feb5bc75640 10 monclient: get_monmap_and_config 2023-10-30T12:05:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.037+0000 7feb5bc75640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.037+0000 7feb5bc75640 10 monclient: monmap: 2023-10-30T12:05:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:15.038094+0000 2023-10-30T12:05:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:15.038094+0000 2023-10-30T12:05:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:15.041 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:15.041 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:15.041 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:15.041 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:15.042 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.037+0000 7feb5bc75640 1 Processor -- start 2023-10-30T12:05:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.037+0000 7feb5bc75640 1 -- start start 2023-10-30T12:05:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.037+0000 7feb5bc75640 10 monclient: init 2023-10-30T12:05:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-10-30T12:05:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb540a8010 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7feb54099a40 0x7feb5413aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient(hunting): picked mon.noname-c con 0x7feb54099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb5413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient(hunting): picked mon.noname-b con 0x7feb54132da0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb540a7190 con 0x7feb540a8010 2023-10-30T12:05:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb540a83f0 con 0x7feb54099a40 2023-10-30T12:05:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb540a7600 con 0x7feb54132da0 2023-10-30T12:05:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5bc75640 10 monclient(hunting): authenticate will time out at 1160.221331s 2023-10-30T12:05:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb5413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59a11640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7feb540a8010 0x7feb54132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43268/0 (socket says 172.21.15.50:43268) 2023-10-30T12:05:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59a11640 1 -- 172.21.15.50:0/3098639588 learned_addr learned my addr 172.21.15.50:0/3098639588 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): get_auth_request con 0x7feb54132da0 auth_method 0 2023-10-30T12:05:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 1 --2- 172.21.15.50:0/3098639588 >> v2:172.21.15.118:3300/0 conn(0x7feb54099a40 0x7feb5413aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 10 monclient(hunting): get_auth_request con 0x7feb54099a40 auth_method 0 2023-10-30T12:05:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb59210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient(hunting): handle_auth_done global_id 44633 payload 274 2023-10-30T12:05:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb58a0f640 1 -- 172.21.15.50:0/3098639588 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336957896 0 0) 0x7feb540a7190 con 0x7feb540a8010 2023-10-30T12:05:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 1 -- 172.21.15.50:0/3098639588 >> v2:172.21.15.118:3300/0 conn(0x7feb54099a40 msgr2=0x7feb5413aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 1 --2- 172.21.15.50:0/3098639588 >> v2:172.21.15.118:3300/0 conn(0x7feb54099a40 0x7feb5413aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 1 -- 172.21.15.50:0/3098639588 >> v1:172.21.15.50:6789/0 conn(0x7feb540a8010 legacy=0x7feb54132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 1 monclient: found mon.noname-b 2023-10-30T12:05:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb5a212640 1 -- 172.21.15.50:0/3098639588 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5409a440 con 0x7feb54132da0 2023-10-30T12:05:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.038+0000 7feb58a0f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:05:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb5a212640 1 --2- 172.21.15.50:0/3098639588 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb5413cf60 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7feb44009d90 tx=0x7feb4402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6819a1be352eadf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 1 -- 172.21.15.50:0/3098639588 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4402cb30 con 0x7feb54132da0 2023-10-30T12:05:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:05:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: dump: 2023-10-30T12:05:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.061 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.061 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.061 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.061 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.061 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb5bc75640 5 monclient: authenticate success, global_id 44633 2023-10-30T12:05:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb5bc75640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 1 -- 172.21.15.50:0/3098639588 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4402cc90 con 0x7feb54132da0 2023-10-30T12:05:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb5bc75640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 1 -- 172.21.15.50:0/3098639588 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44034440 con 0x7feb54132da0 2023-10-30T12:05:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb5bc75640 4 set_mon_vals no callback set 2023-10-30T12:05:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.039+0000 7feb58a0f640 10 monclient: dump: 2023-10-30T12:05:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.065 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 10 monclient: shutdown 2023-10-30T12:05:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 -- 172.21.15.50:0/3098639588 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 msgr2=0x7feb5413cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 --2- 172.21.15.50:0/3098639588 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb5413cf60 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7feb44009d90 tx=0x7feb4402bd80 comp rx=0 tx=0).stop 2023-10-30T12:05:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 -- 172.21.15.50:0/3098639588 shutdown_connections 2023-10-30T12:05:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 --2- 172.21.15.50:0/3098639588 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb5413cf60 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 --2- 172.21.15.50:0/3098639588 >> v2:172.21.15.118:3300/0 conn(0x7feb54099a40 0x7feb5413aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 -- 172.21.15.50:0/3098639588 >> 172.21.15.50:0/3098639588 conn(0x7feb540a4a10 msgr2=0x7feb540a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 -- 172.21.15.50:0/3098639588 shutdown_connections 2023-10-30T12:05:15.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 1 -- 172.21.15.50:0/3098639588 wait complete. 2023-10-30T12:05:15.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.040+0000 7feb5bc75640 10 monclient: monmap: 2023-10-30T12:05:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:15.069 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 librados: starting msgr at 2023-10-30T12:05:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 librados: starting objecter 2023-10-30T12:05:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 Processor -- start 2023-10-30T12:05:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 -- start start 2023-10-30T12:05:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 librados: setting wanted keys 2023-10-30T12:05:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 librados: calling monclient init 2023-10-30T12:05:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient: init 2023-10-30T12:05:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:05:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb540a8010 0x7feb541367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): picked mon.noname-a con 0x7feb540a8010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb54136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): picked mon.noname-c con 0x7feb54132da0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb54138700 0x7feb54137240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): picked mon.noname-b con 0x7feb54138700 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb541396b0 con 0x7feb540a8010 2023-10-30T12:05:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5413b900 con 0x7feb54138700 2023-10-30T12:05:15.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5413a090 con 0x7feb54132da0 2023-10-30T12:05:15.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5bc75640 10 monclient(hunting): authenticate will time out at 1160.224773s 2023-10-30T12:05:15.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb540a8010 0x7feb541367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb540a8010 0x7feb541367c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37836/0 (socket says 172.21.15.50:37836) 2023-10-30T12:05:15.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 1 -- 172.21.15.50:0/92843576 learned_addr learned my addr 172.21.15.50:0/92843576 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:15.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 10 monclient(hunting): get_auth_request con 0x7feb540a8010 auth_method 0 2023-10-30T12:05:15.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb54136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 10 monclient(hunting): get_auth_request con 0x7feb54132da0 auth_method 0 2023-10-30T12:05:15.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5a212640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb54138700 0x7feb54137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5a212640 10 monclient(hunting): get_auth_request con 0x7feb54138700 auth_method 0 2023-10-30T12:05:15.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5a212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5a212640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb5a212640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.041+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5a212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5a212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5a212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 10 monclient(hunting): handle_auth_done global_id 44636 payload 824 2023-10-30T12:05:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb54138700 msgr2=0x7feb54137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb54138700 0x7feb54137240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb540a8010 msgr2=0x7feb541367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb540a8010 0x7feb541367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 1 monclient: found mon.noname-c 2023-10-30T12:05:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 1 -- 172.21.15.50:0/92843576 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44009a70 con 0x7feb54132da0 2023-10-30T12:05:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb59210640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb54136d00 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7feb540a6f10 tx=0x7feb4800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 client.?.objecter ms_handle_connect 0x7feb54132da0 2023-10-30T12:05:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4800c730 con 0x7feb54132da0 2023-10-30T12:05:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: dump: 2023-10-30T12:05:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4800c890 con 0x7feb54132da0 2023-10-30T12:05:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4800cc40 con 0x7feb54132da0 2023-10-30T12:05:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 5 monclient: authenticate success, global_id 44636 2023-10-30T12:05:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 10 monclient: _renew_subs 2023-10-30T12:05:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb54138260 con 0x7feb54132da0 2023-10-30T12:05:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb53fff640 4 set_mon_vals no callback set 2023-10-30T12:05:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb527fc640 10 monclient: dump: 2023-10-30T12:05:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.087 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 10 client.44636.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 10 monclient: _renew_subs 2023-10-30T12:05:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb541378a0 con 0x7feb54132da0 2023-10-30T12:05:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.042+0000 7feb5bc75640 1 librados: init done 2023-10-30T12:05:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb5bc75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb5bc75640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb54099a40 0x7feb5409cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb5bc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7feb48013470 con 0x7feb54132da0 2023-10-30T12:05:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb54073fd0 con 0x7feb54099a40 2023-10-30T12:05:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb54099a40 0x7feb5409cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient: get_auth_request con 0x7feb54099a40 auth_method 0 2023-10-30T12:05:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb400392f0 0x7feb4003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 10 monclient(hunting): handle_auth_done global_id 84108 payload 824 2023-10-30T12:05:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6533+0+0 (secure 0 0 0) 0x7feb4804acc0 con 0x7feb54132da0 2023-10-30T12:05:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 10 client.44636.objecter ms_dispatch 0x7feb54000b90 osd_map(136..136 src has 1..136) v4 2023-10-30T12:05:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 3 client.44636.objecter handle_osd_map got epochs [136,136] > 0 2023-10-30T12:05:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 3 client.44636.objecter handle_osd_map decoding full epoch 136 2023-10-30T12:05:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 1 -- 172.21.15.50:0/92843576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb5409aa00 con 0x7feb54099a40 2023-10-30T12:05:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 20 client.44636.objecter dump_active .. 0 homeless 2023-10-30T12:05:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 10 client.44636.objecter ms_handle_connect 0x7feb54099a40 2023-10-30T12:05:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb527fc640 10 client.44636.objecter resend_mon_ops 2023-10-30T12:05:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.043+0000 7feb59a11640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb54099a40 0x7feb5409cec0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0011e0 tx=0x7feb4c001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb5a212640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb400392f0 0x7feb4003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c00f820 con 0x7feb54099a40 2023-10-30T12:05:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 10 monclient: dump: 2023-10-30T12:05:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.095 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb5a212640 10 monclient: get_auth_request con 0x7feb400392f0 auth_method 0 2023-10-30T12:05:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7feb4c00f980 con 0x7feb54099a40 2023-10-30T12:05:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb54099a40 msgr2=0x7feb5409cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb54099a40 0x7feb5409cec0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0011e0 tx=0x7feb4c001a50 comp rx=0 tx=0).stop 2023-10-30T12:05:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb54099a40 msgr2=0x7feb5409cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb54099a40 0x7feb5409cec0 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0011e0 tx=0x7feb4c001a50 comp rx=0 tx=0).stop 2023-10-30T12:05:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb527fc640 10 client.44636.objecter ms_handle_connect 0x7feb400392f0 2023-10-30T12:05:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.044+0000 7feb5a212640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb400392f0 0x7feb4003b7b0 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 crypto rx=0x7feb44036000 tx=0x7feb44030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb5bc75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb5bc75640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5409d400 0x7feb5409d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb5bc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5409a720 con 0x7feb5409d400 2023-10-30T12:05:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5409d400 0x7feb5409d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient: get_auth_request con 0x7feb5409d400 auth_method 0 2023-10-30T12:05:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.050+0000 7feb59a11640 10 monclient(hunting): handle_auth_done global_id 84111 payload 824 2023-10-30T12:05:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb59a11640 1 -- 172.21.15.50:0/92843576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb5413d4a0 con 0x7feb5409d400 2023-10-30T12:05:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 10 client.44636.objecter ms_handle_connect 0x7feb5409d400 2023-10-30T12:05:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 10 client.44636.objecter resend_mon_ops 2023-10-30T12:05:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb59a11640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5409d400 0x7feb5409d810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0011e0 tx=0x7feb4c001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c024070 con 0x7feb5409d400 2023-10-30T12:05:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 10 monclient: dump: 2023-10-30T12:05:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.105 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7feb4c01fb80 con 0x7feb5409d400 2023-10-30T12:05:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5409d400 msgr2=0x7feb5409d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5409d400 0x7feb5409d810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0011e0 tx=0x7feb4c001080 comp rx=0 tx=0).stop 2023-10-30T12:05:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5409d400 msgr2=0x7feb5409d810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb527fc640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5409d400 0x7feb5409d810 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0011e0 tx=0x7feb4c001080 comp rx=0 tx=0).stop 2023-10-30T12:05:15.106 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:15.107 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:15.107 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:15.107 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:05:15.107 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 50, 2023-10-30T12:05:15.107 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:15.108 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:15.108 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:05:15.108 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:15.108 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.108 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:05:15.108 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:15.109 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:15.109 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:15.109 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.109 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.109 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.110 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.110 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.110 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.110 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.110 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.111 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.111 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.111 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.111 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:15.111 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:15.111 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.112 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.112 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.112 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.112 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.112 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.113 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.113 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.113 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.113 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.113 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.114 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:15.114 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:15.114 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:15.114 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:15.115 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:15.115 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:15.115 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:15.115 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:15.115 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:15.115 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:15.116 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:15.116 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:15.116 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:15.116 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:15.116 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:15.117 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:15.117 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:15.117 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.117 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.117 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.118 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.118 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.118 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.118 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.118 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.118 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.119 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.119 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.119 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:15.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.119 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:15.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.120 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:15.120 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:15.120 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.120 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.121 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.121 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:15.121 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.122 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:15.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.122 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.123 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:15.123 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:15.123 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.123 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.124 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.124 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:15.124 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:15.124 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.125 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.125 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.125 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:15.125 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.126 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.126 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:15.126 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.127 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.127 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:15.127 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:15.127 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.127 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.128 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.128 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:15.128 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:15.128 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.129 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.129 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:15.129 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.130 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:15.130 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.131 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.131 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:15.131 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:15.132 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.132 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.132 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.132 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.133 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:15.133 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:15.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.133 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.133 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.134 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:15.134 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.134 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.134 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:15.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.135 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.135 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.135 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:15.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.135 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.136 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:15.136 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:15.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 10 librados: watch_flush enter 2023-10-30T12:05:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 10 librados: watch_flush exit 2023-10-30T12:05:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 20 client.44636.objecter shutdown clearing up homeless session... 2023-10-30T12:05:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 10 client.44636.objecter successfully canceled tick 2023-10-30T12:05:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb400392f0 msgr2=0x7feb4003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb400392f0 0x7feb4003b7b0 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 crypto rx=0x7feb44036000 tx=0x7feb44030000 comp rx=0 tx=0).stop 2023-10-30T12:05:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 10 monclient: shutdown 2023-10-30T12:05:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 msgr2=0x7feb54136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 --2- 172.21.15.50:0/92843576 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb54132da0 0x7feb54136d00 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7feb540a6f10 tx=0x7feb4800ecd0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5a212640 1 -- 172.21.15.50:0/92843576 reap_dead start 2023-10-30T12:05:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5a212640 1 -- 172.21.15.50:0/92843576 reap_dead start 2023-10-30T12:05:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 shutdown_connections 2023-10-30T12:05:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 >> 172.21.15.50:0/92843576 conn(0x7feb540a4a10 msgr2=0x7feb54069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 shutdown_connections 2023-10-30T12:05:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.051+0000 7feb5bc75640 1 -- 172.21.15.50:0/92843576 wait complete. 2023-10-30T12:05:15.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.052+0000 7feb5bc75640 1 librados: shutdown 2023-10-30T12:05:15.141 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:05:15.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:05:15.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:05:15.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.246+0000 7ff79e261640 10 monclient: get_monmap_and_config 2023-10-30T12:05:15.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.246+0000 7ff79e261640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.246+0000 7ff79e261640 10 monclient: monmap: 2023-10-30T12:05:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:15.248015+0000 2023-10-30T12:05:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:15.248015+0000 2023-10-30T12:05:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:15.251 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.247+0000 7ff79e261640 1 Processor -- start 2023-10-30T12:05:15.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.247+0000 7ff79e261640 1 -- start start 2023-10-30T12:05:15.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.247+0000 7ff79e261640 10 monclient: init 2023-10-30T12:05:15.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:05:15.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff798097090 0x7ff798097470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient(hunting): picked mon.noname-c con 0x7ff798097090 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:15.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7980979b0 0x7ff79809bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient(hunting): picked mon.noname-b con 0x7ff7980979b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff79809c0b0 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980a0ca0 con 0x7ff79809c0b0 2023-10-30T12:05:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79809e1a0 con 0x7ff798097090 2023-10-30T12:05:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980a09f0 con 0x7ff7980979b0 2023-10-30T12:05:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff79e261640 10 monclient(hunting): authenticate will time out at 1160.431287s 2023-10-30T12:05:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7980979b0 0x7ff79809bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7980979b0 0x7ff79809bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42972/0 (socket says 172.21.15.50:42972) 2023-10-30T12:05:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 -- 172.21.15.50:0/1995255208 learned_addr learned my addr 172.21.15.50:0/1995255208 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): get_auth_request con 0x7ff7980979b0 auth_method 0 2023-10-30T12:05:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 1 --2- 172.21.15.50:0/1995255208 >> v2:172.21.15.118:3300/0 conn(0x7ff798097090 0x7ff798097470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 10 monclient(hunting): get_auth_request con 0x7ff798097090 auth_method 0 2023-10-30T12:05:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient(hunting): handle_auth_done global_id 44639 payload 274 2023-10-30T12:05:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 -- 172.21.15.50:0/1995255208 >> v2:172.21.15.118:3300/0 conn(0x7ff798097090 msgr2=0x7ff798097470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 --2- 172.21.15.50:0/1995255208 >> v2:172.21.15.118:3300/0 conn(0x7ff798097090 0x7ff798097470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 -- 172.21.15.50:0/1995255208 >> v1:172.21.15.118:6789/0 conn(0x7ff79809c0b0 legacy=0x7ff79809c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 monclient: found mon.noname-b 2023-10-30T12:05:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.248+0000 7ff796ffd640 1 -- 172.21.15.50:0/1995255208 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79809d900 con 0x7ff7980979b0 2023-10-30T12:05:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff796ffd640 1 --2- 172.21.15.50:0/1995255208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7980979b0 0x7ff79809bb70 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009870 tx=0x7ff78c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292caba3c2bef54b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 1 -- 172.21.15.50:0/1995255208 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c0317c0 con 0x7ff7980979b0 2023-10-30T12:05:15.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:05:15.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: dump: 2023-10-30T12:05:15.269 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.269 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.270 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.270 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.270 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.270 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.270 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.271 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff79e261640 5 monclient: authenticate success, global_id 44639 2023-10-30T12:05:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 1 -- 172.21.15.50:0/1995255208 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c031920 con 0x7ff7980979b0 2023-10-30T12:05:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff79e261640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff79e261640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff79e261640 4 set_mon_vals no callback set 2023-10-30T12:05:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 1 -- 172.21.15.50:0/1995255208 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c031cb0 con 0x7ff7980979b0 2023-10-30T12:05:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.249+0000 7ff7967fc640 10 monclient: dump: 2023-10-30T12:05:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.275 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 10 monclient: shutdown 2023-10-30T12:05:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 -- 172.21.15.50:0/1995255208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7980979b0 msgr2=0x7ff79809bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 --2- 172.21.15.50:0/1995255208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7980979b0 0x7ff79809bb70 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009870 tx=0x7ff78c02b860 comp rx=0 tx=0).stop 2023-10-30T12:05:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 -- 172.21.15.50:0/1995255208 shutdown_connections 2023-10-30T12:05:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 --2- 172.21.15.50:0/1995255208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7980979b0 0x7ff79809bb70 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 --2- 172.21.15.50:0/1995255208 >> v2:172.21.15.118:3300/0 conn(0x7ff798097090 0x7ff798097470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 -- 172.21.15.50:0/1995255208 >> 172.21.15.50:0/1995255208 conn(0x7ff7980a1170 msgr2=0x7ff7980a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 -- 172.21.15.50:0/1995255208 shutdown_connections 2023-10-30T12:05:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 1 -- 172.21.15.50:0/1995255208 wait complete. 2023-10-30T12:05:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.250+0000 7ff79e261640 10 monclient: monmap: 2023-10-30T12:05:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:15.279 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 librados: starting msgr at 2023-10-30T12:05:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 librados: starting objecter 2023-10-30T12:05:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 Processor -- start 2023-10-30T12:05:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 -- start start 2023-10-30T12:05:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 librados: setting wanted keys 2023-10-30T12:05:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 librados: calling monclient init 2023-10-30T12:05:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient: init 2023-10-30T12:05:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:05:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff798097090 0x7ff7981ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): picked mon.noname-c con 0x7ff798097090 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7980979b0 0x7ff7981b86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): picked mon.noname-a con 0x7ff7980979b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79809c0b0 0x7ff7981b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): picked mon.noname-b con 0x7ff79809c0b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7981b3e80 con 0x7ff7980979b0 2023-10-30T12:05:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff798071760 con 0x7ff79809c0b0 2023-10-30T12:05:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7981b2a40 con 0x7ff798097090 2023-10-30T12:05:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff79e261640 10 monclient(hunting): authenticate will time out at 1160.434840s 2023-10-30T12:05:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff7977fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff798097090 0x7ff7981ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff796ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7980979b0 0x7ff7981b86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff7977fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff798097090 0x7ff7981ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42974/0 (socket says 172.21.15.50:42974) 2023-10-30T12:05:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff7977fe640 1 -- 172.21.15.50:0/2078932121 learned_addr learned my addr 172.21.15.50:0/2078932121 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff7977fe640 10 monclient(hunting): get_auth_request con 0x7ff798097090 auth_method 0 2023-10-30T12:05:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff7977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff7977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff7977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff796ffd640 10 monclient(hunting): get_auth_request con 0x7ff7980979b0 auth_method 0 2023-10-30T12:05:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.251+0000 7ff796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79809c0b0 0x7ff7981b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 10 monclient(hunting): get_auth_request con 0x7ff79809c0b0 auth_method 0 2023-10-30T12:05:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff797fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 10 monclient(hunting): handle_auth_done global_id 44642 payload 824 2023-10-30T12:05:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79809c0b0 msgr2=0x7ff7981b8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff79809c0b0 0x7ff7981b8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7980979b0 msgr2=0x7ff7981b86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7980979b0 0x7ff7981b86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 1 monclient: found mon.noname-c 2023-10-30T12:05:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 1 -- 172.21.15.50:0/2078932121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c0094d0 con 0x7ff798097090 2023-10-30T12:05:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff7977fe640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff798097090 0x7ff7981ca090 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7ff78800e870 tx=0x7ff78800ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 client.?.objecter ms_handle_connect 0x7ff798097090 2023-10-30T12:05:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788009510 con 0x7ff798097090 2023-10-30T12:05:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: dump: 2023-10-30T12:05:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.293 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78800eec0 con 0x7ff798097090 2023-10-30T12:05:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78800cb80 con 0x7ff798097090 2023-10-30T12:05:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 5 monclient: authenticate success, global_id 44642 2023-10-30T12:05:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff77ffff640 10 monclient: dump: 2023-10-30T12:05:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.297 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 10 monclient: _renew_subs 2023-10-30T12:05:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff795ffb640 4 set_mon_vals no callback set 2023-10-30T12:05:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff798097f10 con 0x7ff798097090 2023-10-30T12:05:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 10 client.44642.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 10 monclient: _renew_subs 2023-10-30T12:05:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7981b9140 con 0x7ff798097090 2023-10-30T12:05:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.252+0000 7ff79e261640 1 librados: init done 2023-10-30T12:05:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff78800cd20 con 0x7ff798097090 2023-10-30T12:05:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff79e261640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff79e261640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806d4a0 0x7ff79806d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff79e261640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79806a5a0 con 0x7ff79806d4a0 2023-10-30T12:05:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806d4a0 0x7ff79806d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient: get_auth_request con 0x7ff79806d4a0 auth_method 0 2023-10-30T12:05:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff77ffff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7840392f0 0x7ff78403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6533+0+0 (secure 0 0 0) 0x7ff78804aeb0 con 0x7ff798097090 2023-10-30T12:05:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff77ffff640 10 client.44642.objecter ms_dispatch 0x7ff798000b90 osd_map(136..136 src has 1..136) v4 2023-10-30T12:05:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff77ffff640 3 client.44642.objecter handle_osd_map got epochs [136,136] > 0 2023-10-30T12:05:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff77ffff640 3 client.44642.objecter handle_osd_map decoding full epoch 136 2023-10-30T12:05:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff77ffff640 20 client.44642.objecter dump_active .. 0 homeless 2023-10-30T12:05:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 10 monclient(hunting): handle_auth_done global_id 44645 payload 824 2023-10-30T12:05:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff796ffd640 1 -- 172.21.15.50:0/2078932121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7981b3ff0 con 0x7ff79806d4a0 2023-10-30T12:05:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff797fff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7840392f0 0x7ff78403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.253+0000 7ff797fff640 10 monclient: get_auth_request con 0x7ff7840392f0 auth_method 0 2023-10-30T12:05:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 client.44642.objecter ms_handle_connect 0x7ff79806d4a0 2023-10-30T12:05:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff796ffd640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806d4a0 0x7ff79806d880 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7ff79809b810 tx=0x7ff78c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 client.44642.objecter resend_mon_ops 2023-10-30T12:05:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c032920 con 0x7ff79806d4a0 2023-10-30T12:05:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 monclient: dump: 2023-10-30T12:05:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.307 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 client.44642.objecter ms_handle_connect 0x7ff7840392f0 2023-10-30T12:05:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff797fff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7840392f0 0x7ff78403b7b0 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7ff7900012e0 tx=0x7ff790009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ff78c032a80 con 0x7ff79806d4a0 2023-10-30T12:05:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806d4a0 msgr2=0x7ff79806d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806d4a0 0x7ff79806d880 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7ff79809b810 tx=0x7ff78c004730 comp rx=0 tx=0).stop 2023-10-30T12:05:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806d4a0 msgr2=0x7ff79806d880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.254+0000 7ff77ffff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806d4a0 0x7ff79806d880 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff79e261640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff79e261640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806ddc0 0x7ff7981b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff79e261640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff798097d90 con 0x7ff79806ddc0 2023-10-30T12:05:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806ddc0 0x7ff7981b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 10 monclient: get_auth_request con 0x7ff79806ddc0 auth_method 0 2023-10-30T12:05:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.260+0000 7ff796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff796ffd640 10 monclient(hunting): handle_auth_done global_id 44648 payload 824 2023-10-30T12:05:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff796ffd640 1 -- 172.21.15.50:0/2078932121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff79809e930 con 0x7ff79806ddc0 2023-10-30T12:05:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 10 client.44642.objecter ms_handle_connect 0x7ff79806ddc0 2023-10-30T12:05:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff796ffd640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806ddc0 0x7ff7981b98b0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7ff78c046000 tx=0x7ff78c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 10 client.44642.objecter resend_mon_ops 2023-10-30T12:05:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c04f660 con 0x7ff79806ddc0 2023-10-30T12:05:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 10 monclient: dump: 2023-10-30T12:05:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.316 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7ff78c053070 con 0x7ff79806ddc0 2023-10-30T12:05:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806ddc0 msgr2=0x7ff7981b98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806ddc0 0x7ff7981b98b0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7ff78c046000 tx=0x7ff78c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806ddc0 msgr2=0x7ff7981b98b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff77ffff640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff79806ddc0 0x7ff7981b98b0 secure :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7ff78c046000 tx=0x7ff78c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.317 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:15.318 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:15.318 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:15.318 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:05:15.318 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 50, 2023-10-30T12:05:15.318 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:15.319 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:15.319 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:05:15.319 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:15.319 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.319 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:05:15.320 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:15.320 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:15.320 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:15.320 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.320 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.321 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.321 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.321 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.321 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.321 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.321 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.322 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.322 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.322 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.322 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:15.322 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:15.323 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.323 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.323 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.323 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.323 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.324 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.324 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.324 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.324 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.324 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.324 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.325 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:15.325 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:15.325 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:15.325 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:15.326 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:15.326 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:15.326 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:15.326 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:15.326 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:15.327 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:15.327 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:15.327 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:15.327 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:15.327 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:15.328 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:15.328 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:15.328 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:15.328 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.328 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.329 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.329 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.329 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.329 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.329 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.330 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.330 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.330 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.330 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.330 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:15.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.331 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:15.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.331 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:15.332 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:15.332 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.332 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.333 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:15.333 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.333 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.333 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:15.334 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.334 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.334 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:15.335 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:15.335 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.335 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.335 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.336 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:15.336 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:15.336 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.336 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.337 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.337 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:15.337 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.337 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.338 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:15.338 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.338 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.339 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:15.339 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:15.339 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.339 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.339 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.340 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:15.340 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:15.340 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.340 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.341 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.341 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:15.341 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.342 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.342 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:15.342 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.342 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.343 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:15.343 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:15.343 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.343 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.344 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.344 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.344 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.344 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:15.345 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:15.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.345 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.345 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.345 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:15.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.346 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.346 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:05:15.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.346 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.347 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.347 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:05:15.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.347 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.347 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:15.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.348 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.348 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.348 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:05:15.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.349 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:15.349 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 10 librados: watch_flush enter 2023-10-30T12:05:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 10 librados: watch_flush exit 2023-10-30T12:05:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 20 client.44642.objecter shutdown clearing up homeless session... 2023-10-30T12:05:15.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 10 client.44642.objecter successfully canceled tick 2023-10-30T12:05:15.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7840392f0 msgr2=0x7ff78403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7840392f0 0x7ff78403b7b0 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7ff7900012e0 tx=0x7ff790009040 comp rx=0 tx=0).stop 2023-10-30T12:05:15.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 10 monclient: shutdown 2023-10-30T12:05:15.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff798097090 msgr2=0x7ff7981ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 1 --2- 172.21.15.50:0/2078932121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff798097090 0x7ff7981ca090 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7ff78800e870 tx=0x7ff78800ec70 comp rx=0 tx=0).stop 2023-10-30T12:05:15.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff797fff640 1 -- 172.21.15.50:0/2078932121 reap_dead start 2023-10-30T12:05:15.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff797fff640 1 -- 172.21.15.50:0/2078932121 reap_dead start 2023-10-30T12:05:15.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 shutdown_connections 2023-10-30T12:05:15.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.261+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 >> 172.21.15.50:0/2078932121 conn(0x7ff7980a1170 msgr2=0x7ff798073620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.262+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 shutdown_connections 2023-10-30T12:05:15.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.262+0000 7ff79e261640 1 -- 172.21.15.50:0/2078932121 wait complete. 2023-10-30T12:05:15.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.262+0000 7ff79e261640 1 librados: shutdown 2023-10-30T12:05:15.355 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:05:15.355 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:05:15.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:05:15.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.460+0000 7f712e409640 10 monclient: get_monmap_and_config 2023-10-30T12:05:15.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.460+0000 7f712e409640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.460+0000 7f712e409640 10 monclient: monmap: 2023-10-30T12:05:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:15.461885+0000 2023-10-30T12:05:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:15.461885+0000 2023-10-30T12:05:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:15.466 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.461+0000 7f712e409640 1 Processor -- start 2023-10-30T12:05:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.461+0000 7f712e409640 1 -- start start 2023-10-30T12:05:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.461+0000 7f712e409640 10 monclient: init 2023-10-30T12:05:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-10-30T12:05:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f712806afc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:15.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f712806b640 0x7f71280a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient(hunting): picked mon.noname-c con 0x7f712806b640 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f712813b070 0x7f712813d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient(hunting): picked mon.noname-a con 0x7f712813b070 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f712806b3a0 con 0x7f712806afc0 2023-10-30T12:05:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712809e540 con 0x7f712813b070 2023-10-30T12:05:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712809fa10 con 0x7f712806b640 2023-10-30T12:05:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712e409640 10 monclient(hunting): authenticate will time out at 1160.645143s 2023-10-30T12:05:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f7127fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f712806afc0 0x7f71280a1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43272/0 (socket says 172.21.15.50:43272) 2023-10-30T12:05:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f712813b070 0x7f712813d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f7127fff640 1 -- 172.21.15.50:0/1047108237 learned_addr learned my addr 172.21.15.50:0/1047108237 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): get_auth_request con 0x7f712813b070 auth_method 0 2023-10-30T12:05:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f71277fe640 1 --2- 172.21.15.50:0/1047108237 >> v2:172.21.15.118:3300/0 conn(0x7f712806b640 0x7f71280a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f71277fe640 10 monclient(hunting): get_auth_request con 0x7f712806b640 auth_method 0 2023-10-30T12:05:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f71277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f71277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f71277fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient(hunting): handle_auth_done global_id 84117 payload 274 2023-10-30T12:05:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 1 -- 172.21.15.50:0/1047108237 >> v2:172.21.15.118:3300/0 conn(0x7f712806b640 msgr2=0x7f71280a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 1 --2- 172.21.15.50:0/1047108237 >> v2:172.21.15.118:3300/0 conn(0x7f712806b640 0x7f71280a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 1 -- 172.21.15.50:0/1047108237 >> v1:172.21.15.50:6789/0 conn(0x7f712806afc0 legacy=0x7f71280a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 1 monclient: found mon.noname-a 2023-10-30T12:05:15.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f712c9a6640 1 -- 172.21.15.50:0/1047108237 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712806a6f0 con 0x7f712813b070 2023-10-30T12:05:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f7126ffd640 1 -- 172.21.15.50:0/1047108237 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163187623 0 0) 0x7f712806b3a0 con 0x7f712806afc0 2023-10-30T12:05:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.462+0000 7f7126ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:05:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712c9a6640 1 --2- 172.21.15.50:0/1047108237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712813b070 0x7f712813d460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7120009d90 tx=0x7f712002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb69b19f227d6bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 1 -- 172.21.15.50:0/1047108237 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712002cb40 con 0x7f712813b070 2023-10-30T12:05:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: dump: 2023-10-30T12:05:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.488 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 1 -- 172.21.15.50:0/1047108237 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712002cca0 con 0x7f712813b070 2023-10-30T12:05:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 5 monclient: authenticate success, global_id 84117 2023-10-30T12:05:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 4 set_mon_vals no callback set 2023-10-30T12:05:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 1 -- 172.21.15.50:0/1047108237 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120034660 con 0x7f712813b070 2023-10-30T12:05:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f7126ffd640 10 monclient: dump: 2023-10-30T12:05:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.493 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 10 monclient: shutdown 2023-10-30T12:05:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 1 -- 172.21.15.50:0/1047108237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712813b070 msgr2=0x7f712813d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.463+0000 7f712e409640 1 --2- 172.21.15.50:0/1047108237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712813b070 0x7f712813d460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7120009d90 tx=0x7f712002bd90 comp rx=0 tx=0).stop 2023-10-30T12:05:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 -- 172.21.15.50:0/1047108237 shutdown_connections 2023-10-30T12:05:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 --2- 172.21.15.50:0/1047108237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712813b070 0x7f712813d460 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 --2- 172.21.15.50:0/1047108237 >> v2:172.21.15.118:3300/0 conn(0x7f712806b640 0x7f71280a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 -- 172.21.15.50:0/1047108237 >> 172.21.15.50:0/1047108237 conn(0x7f71280a6200 msgr2=0x7f71280a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 -- 172.21.15.50:0/1047108237 shutdown_connections 2023-10-30T12:05:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 -- 172.21.15.50:0/1047108237 wait complete. 2023-10-30T12:05:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 10 monclient: monmap: 2023-10-30T12:05:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:15.497 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 librados: starting msgr at 2023-10-30T12:05:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 librados: starting objecter 2023-10-30T12:05:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.464+0000 7f712e409640 1 Processor -- start 2023-10-30T12:05:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 -- start start 2023-10-30T12:05:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 librados: setting wanted keys 2023-10-30T12:05:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 librados: calling monclient init 2023-10-30T12:05:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient: init 2023-10-30T12:05:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:05:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712806afc0 0x7f71281ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): picked mon.noname-a con 0x7f712806afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f712806b640 0x7f71281caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): picked mon.noname-c con 0x7f712806b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712813b070 0x7f712811c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): picked mon.noname-b con 0x7f712813b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712811fc60 con 0x7f712806afc0 2023-10-30T12:05:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7128070de0 con 0x7f712813b070 2023-10-30T12:05:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7128138420 con 0x7f712806b640 2023-10-30T12:05:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712e409640 10 monclient(hunting): authenticate will time out at 1160.648670s 2023-10-30T12:05:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712806afc0 0x7f71281ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712806afc0 0x7f71281ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37884/0 (socket says 172.21.15.50:37884) 2023-10-30T12:05:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 1 -- 172.21.15.50:0/2256152659 learned_addr learned my addr 172.21.15.50:0/2256152659 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 10 monclient(hunting): get_auth_request con 0x7f712806afc0 auth_method 0 2023-10-30T12:05:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f712806b640 0x7f71281caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 10 monclient(hunting): get_auth_request con 0x7f712806b640 auth_method 0 2023-10-30T12:05:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712c9a6640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712813b070 0x7f712811c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712c9a6640 10 monclient(hunting): get_auth_request con 0x7f712813b070 auth_method 0 2023-10-30T12:05:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712c9a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712c9a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712c9a6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712c9a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.465+0000 7f712c9a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712c9a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 10 monclient(hunting): handle_auth_done global_id 84123 payload 824 2023-10-30T12:05:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f712806b640 msgr2=0x7f71281caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f712806b640 0x7f71281caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712813b070 msgr2=0x7f712811c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712813b070 0x7f712811c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 1 monclient: found mon.noname-a 2023-10-30T12:05:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 1 -- 172.21.15.50:0/2256152659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120009a70 con 0x7f712806afc0 2023-10-30T12:05:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f71277fe640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f712806b640 0x7f71281caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7127fff640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712806afc0 0x7f71281ca4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f711800c4a0 tx=0x7f711800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 client.?.objecter ms_handle_connect 0x7f712806afc0 2023-10-30T12:05:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118005ce0 con 0x7f712806afc0 2023-10-30T12:05:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: dump: 2023-10-30T12:05:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.510 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7118005e40 con 0x7f712806afc0 2023-10-30T12:05:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71180072b0 con 0x7f712806afc0 2023-10-30T12:05:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 5 monclient: authenticate success, global_id 84123 2023-10-30T12:05:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f7124ff9640 10 monclient: dump: 2023-10-30T12:05:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.514 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f71267fc640 4 set_mon_vals no callback set 2023-10-30T12:05:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 10 monclient: _renew_subs 2023-10-30T12:05:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7128120500 con 0x7f712806afc0 2023-10-30T12:05:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 10 client.84123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 10 monclient: _renew_subs 2023-10-30T12:05:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712811e130 con 0x7f712806afc0 2023-10-30T12:05:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.466+0000 7f712e409640 1 librados: init done 2023-10-30T12:05:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f712e409640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f712e409640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d190 0x7f712802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f712e409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7128073ff0 con 0x7f712811d190 2023-10-30T12:05:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7118007410 con 0x7f712806afc0 2023-10-30T12:05:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d190 0x7f712802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f7124ff9640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f71080392f0 0x7f710803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient: get_auth_request con 0x7f712811d190 auth_method 0 2023-10-30T12:05:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6533+0+0 (secure 0 0 0) 0x7f711804a2d0 con 0x7f712806afc0 2023-10-30T12:05:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f7124ff9640 10 client.84123.objecter ms_dispatch 0x7f7128000b90 osd_map(136..136 src has 1..136) v4 2023-10-30T12:05:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f7124ff9640 3 client.84123.objecter handle_osd_map got epochs [136,136] > 0 2023-10-30T12:05:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f7124ff9640 3 client.84123.objecter handle_osd_map decoding full epoch 136 2023-10-30T12:05:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f7124ff9640 20 client.84123.objecter dump_active .. 0 homeless 2023-10-30T12:05:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f712c9a6640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f71080392f0 0x7f710803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f712c9a6640 10 monclient: get_auth_request con 0x7f71080392f0 auth_method 0 2023-10-30T12:05:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 10 monclient(hunting): handle_auth_done global_id 28327 payload 824 2023-10-30T12:05:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.467+0000 7f71277fe640 1 -- 172.21.15.50:0/2256152659 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f712809f0b0 con 0x7f712811d190 2023-10-30T12:05:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 client.84123.objecter ms_handle_connect 0x7f712811d190 2023-10-30T12:05:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 client.84123.objecter resend_mon_ops 2023-10-30T12:05:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f71277fe640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d190 0x7f712802b430 secure :-1 s=READY pgs=7151 cs=0 l=1 rev1=1 crypto rx=0x7f7110006fd0 tx=0x7f711000e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 client.84123.objecter ms_handle_connect 0x7f71080392f0 2023-10-30T12:05:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f712c9a6640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f71080392f0 0x7f710803b7b0 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f7120036000 tx=0x7f7120030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7110010c90 con 0x7f712811d190 2023-10-30T12:05:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:05:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 monclient: dump: 2023-10-30T12:05:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.524 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.524 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f7110010df0 con 0x7f712811d190 2023-10-30T12:05:15.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d190 msgr2=0x7f712802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d190 0x7f712802b430 secure :-1 s=READY pgs=7151 cs=0 l=1 rev1=1 crypto rx=0x7f7110006fd0 tx=0x7f711000e5a0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d190 msgr2=0x7f712802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.468+0000 7f7124ff9640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d190 0x7f712802b430 secure :-1 s=CLOSED pgs=7151 cs=0 l=1 rev1=1 crypto rx=0x7f7110006fd0 tx=0x7f711000e5a0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.474+0000 7f712e409640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.474+0000 7f712e409640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d570 0x7f712802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.474+0000 7f712e409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.474+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7128120140 con 0x7f712811d570 2023-10-30T12:05:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d570 0x7f712802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient: get_auth_request con 0x7f712811d570 auth_method 0 2023-10-30T12:05:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 10 monclient(hunting): handle_auth_done global_id 28330 payload 824 2023-10-30T12:05:15.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 1 -- 172.21.15.50:0/2256152659 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f712809df80 con 0x7f712811d570 2023-10-30T12:05:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f71277fe640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d570 0x7f712802b970 secure :-1 s=READY pgs=7152 cs=0 l=1 rev1=1 crypto rx=0x7f7110006fd0 tx=0x7f71100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f7124ff9640 10 client.84123.objecter ms_handle_connect 0x7f712811d570 2023-10-30T12:05:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f7124ff9640 10 client.84123.objecter resend_mon_ops 2023-10-30T12:05:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711002b070 con 0x7f712811d570 2023-10-30T12:05:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f7124ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f7124ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:05:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.475+0000 7f7124ff9640 10 monclient: dump: 2023-10-30T12:05:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.530 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.531 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f7110023040 con 0x7f712811d570 2023-10-30T12:05:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f7124ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f7124ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d570 msgr2=0x7f712802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f7124ff9640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d570 0x7f712802b970 secure :-1 s=READY pgs=7152 cs=0 l=1 rev1=1 crypto rx=0x7f7110006fd0 tx=0x7f71100046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f7124ff9640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d570 msgr2=0x7f712802b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:15.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f7124ff9640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f712811d570 0x7f712802b970 unknown :-1 s=CLOSED pgs=7152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.532 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:15.533 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:15.533 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:15.533 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:05:15.533 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 50, 2023-10-30T12:05:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:05:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:15.534 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:05:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:15.535 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.536 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.537 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.538 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.539 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:15.541 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:15.542 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:15.543 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:15.544 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:15.545 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:15.546 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.547 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.550 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.551 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:15.552 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:15.553 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:15.554 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:15.555 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:15.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.559 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:15.560 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.561 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.562 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:05:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.563 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.564 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:05:15.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.564 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.564 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:15.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.564 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.565 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.565 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 8 2023-10-30T12:05:15.565 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.565 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:15.565 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:05:15.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:15.566 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:15.566 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:15.566 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:15.566 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:15.567 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:15.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:15.567 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:15.567 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:15.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 10 librados: watch_flush enter 2023-10-30T12:05:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 10 librados: watch_flush exit 2023-10-30T12:05:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 20 client.84123.objecter shutdown clearing up homeless session... 2023-10-30T12:05:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 10 client.84123.objecter successfully canceled tick 2023-10-30T12:05:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f71080392f0 msgr2=0x7f710803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f71080392f0 0x7f710803b7b0 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f7120036000 tx=0x7f7120030000 comp rx=0 tx=0).stop 2023-10-30T12:05:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 10 monclient: shutdown 2023-10-30T12:05:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712806afc0 msgr2=0x7f71281ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 --2- 172.21.15.50:0/2256152659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712806afc0 0x7f71281ca4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f711800c4a0 tx=0x7f711800c960 comp rx=0 tx=0).stop 2023-10-30T12:05:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712c9a6640 1 -- 172.21.15.50:0/2256152659 reap_dead start 2023-10-30T12:05:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712c9a6640 1 -- 172.21.15.50:0/2256152659 reap_dead start 2023-10-30T12:05:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 shutdown_connections 2023-10-30T12:05:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 >> 172.21.15.50:0/2256152659 conn(0x7f71280a6200 msgr2=0x7f71280705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 shutdown_connections 2023-10-30T12:05:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 -- 172.21.15.50:0/2256152659 wait complete. 2023-10-30T12:05:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.476+0000 7f712e409640 1 librados: shutdown 2023-10-30T12:05:15.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:05:15.575 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:05:15.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:05:15.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.679+0000 7f968cb16640 10 monclient: get_monmap_and_config 2023-10-30T12:05:15.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.679+0000 7f968cb16640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.680+0000 7f968cb16640 10 monclient: monmap: 2023-10-30T12:05:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:15.681039+0000 2023-10-30T12:05:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:15.681039+0000 2023-10-30T12:05:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:15.684 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:15.684 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:15.684 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:15.684 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.680+0000 7f968cb16640 1 Processor -- start 2023-10-30T12:05:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.680+0000 7f968cb16640 1 -- start start 2023-10-30T12:05:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.680+0000 7f968cb16640 10 monclient: init 2023-10-30T12:05:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-10-30T12:05:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f968809f5b0 0x7f9688099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient(hunting): picked mon.noname-b con 0x7f968809f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9688096080 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9688096b70 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968809f990 con 0x7f9688096b70 2023-10-30T12:05:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9688098810 con 0x7f9688096080 2023-10-30T12:05:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968806af10 con 0x7f968809f5b0 2023-10-30T12:05:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968cb16640 10 monclient(hunting): authenticate will time out at 1160.864233s 2023-10-30T12:05:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f968809f5b0 0x7f9688099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f9686d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9688096b70 0x7f968813ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43284/0 (socket says 172.21.15.50:43284) 2023-10-30T12:05:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f9686d9d640 1 -- 172.21.15.50:0/1602749295 learned_addr learned my addr 172.21.15.50:0/1602749295 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): get_auth_request con 0x7f968809f5b0 auth_method 0 2023-10-30T12:05:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient(hunting): handle_auth_done global_id 44654 payload 274 2023-10-30T12:05:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 1 -- 172.21.15.50:0/1602749295 >> v1:172.21.15.118:6789/0 conn(0x7f9688096080 legacy=0x7f9688096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 1 -- 172.21.15.50:0/1602749295 >> v1:172.21.15.50:6789/0 conn(0x7f9688096b70 legacy=0x7f968813ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 1 monclient: found mon.noname-b 2023-10-30T12:05:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:05:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968659c640 1 -- 172.21.15.50:0/1602749295 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968809a1a0 con 0x7f968809f5b0 2023-10-30T12:05:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968559a640 1 -- 172.21.15.50:0/1602749295 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992923489 0 0) 0x7f968809f990 con 0x7f9688096b70 2023-10-30T12:05:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.681+0000 7f968559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:05:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968659c640 1 --2- 172.21.15.50:0/1602749295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f968809f5b0 0x7f9688099b80 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f967c0097c0 tx=0x7f967c02b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cd7959527dd1cf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 1 -- 172.21.15.50:0/1602749295 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c030710 con 0x7f968809f5b0 2023-10-30T12:05:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:05:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: dump: 2023-10-30T12:05:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.701 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968cb16640 5 monclient: authenticate success, global_id 44654 2023-10-30T12:05:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968cb16640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 1 -- 172.21.15.50:0/1602749295 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c02e050 con 0x7f968809f5b0 2023-10-30T12:05:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968cb16640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 1 -- 172.21.15.50:0/1602749295 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c030a90 con 0x7f968809f5b0 2023-10-30T12:05:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968cb16640 4 set_mon_vals no callback set 2023-10-30T12:05:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968559a640 10 monclient: dump: 2023-10-30T12:05:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.705 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.682+0000 7f968cb16640 10 monclient: shutdown 2023-10-30T12:05:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 1 -- 172.21.15.50:0/1602749295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f968809f5b0 msgr2=0x7f9688099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 1 --2- 172.21.15.50:0/1602749295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f968809f5b0 0x7f9688099b80 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f967c0097c0 tx=0x7f967c02b730 comp rx=0 tx=0).stop 2023-10-30T12:05:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 1 -- 172.21.15.50:0/1602749295 shutdown_connections 2023-10-30T12:05:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 1 --2- 172.21.15.50:0/1602749295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f968809f5b0 0x7f9688099b80 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 1 -- 172.21.15.50:0/1602749295 >> 172.21.15.50:0/1602749295 conn(0x7f96880a05c0 msgr2=0x7f968809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 1 -- 172.21.15.50:0/1602749295 shutdown_connections 2023-10-30T12:05:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 1 -- 172.21.15.50:0/1602749295 wait complete. 2023-10-30T12:05:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 10 monclient: build_initial_monmap 2023-10-30T12:05:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.683+0000 7f968cb16640 10 monclient: monmap: 2023-10-30T12:05:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:15.709 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 librados: starting msgr at 2023-10-30T12:05:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 librados: starting objecter 2023-10-30T12:05:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 Processor -- start 2023-10-30T12:05:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 -- start start 2023-10-30T12:05:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 librados: setting wanted keys 2023-10-30T12:05:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 librados: calling monclient init 2023-10-30T12:05:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient: init 2023-10-30T12:05:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:05:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9688096080 0x7f96881ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): picked mon.noname-a con 0x7f9688096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9688096b70 0x7f9688095040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): picked mon.noname-c con 0x7f9688096b70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968809f5b0 0x7f9688095580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): picked mon.noname-b con 0x7f968809f5b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968808f8d0 con 0x7f9688096080 2023-10-30T12:05:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968808dca0 con 0x7f968809f5b0 2023-10-30T12:05:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9688072010 con 0x7f9688096b70 2023-10-30T12:05:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968cb16640 10 monclient(hunting): authenticate will time out at 1160.867684s 2023-10-30T12:05:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9688096080 0x7f96881ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9688096b70 0x7f9688095040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9688096080 0x7f96881ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37900/0 (socket says 172.21.15.50:37900) 2023-10-30T12:05:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 1 -- 172.21.15.50:0/1960019976 learned_addr learned my addr 172.21.15.50:0/1960019976 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:15.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 10 monclient(hunting): get_auth_request con 0x7f9688096080 auth_method 0 2023-10-30T12:05:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 10 monclient(hunting): get_auth_request con 0x7f9688096b70 auth_method 0 2023-10-30T12:05:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9686d9d640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968809f5b0 0x7f9688095580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9686d9d640 10 monclient(hunting): get_auth_request con 0x7f968809f5b0 auth_method 0 2023-10-30T12:05:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9686d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9686d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.684+0000 7f9686d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 10 monclient(hunting): handle_auth_done global_id 84129 payload 824 2023-10-30T12:05:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 1 -- 172.21.15.50:0/1960019976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9688096b70 msgr2=0x7f9688095040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9688096b70 0x7f9688095040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 1 -- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968809f5b0 msgr2=0x7f9688095580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968809f5b0 0x7f9688095580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 10 monclient: _finish_hunting 0 2023-10-30T12:05:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 1 monclient: found mon.noname-a 2023-10-30T12:05:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 1 -- 172.21.15.50:0/1960019976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0093a0 con 0x7f9688096080 2023-10-30T12:05:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9686d9d640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968809f5b0 0x7f9688095580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:05:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968659c640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9688096080 0x7f96881ca670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f967c000c00 tx=0x7f967c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 client.?.objecter ms_handle_connect 0x7f9688096080 2023-10-30T12:05:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 1 -- 172.21.15.50:0/1960019976 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c005e60 con 0x7f9688096080 2023-10-30T12:05:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: dump: 2023-10-30T12:05:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.722 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.723 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: _finish_auth 0 2023-10-30T12:05:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:05:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 1 -- 172.21.15.50:0/1960019976 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c004030 con 0x7f9688096080 2023-10-30T12:05:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 5 monclient: authenticate success, global_id 84129 2023-10-30T12:05:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 10 monclient: _renew_subs 2023-10-30T12:05:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9688095ac0 con 0x7f9688096080 2023-10-30T12:05:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 1 -- 172.21.15.50:0/1960019976 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c0043e0 con 0x7f9688096080 2023-10-30T12:05:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9676ffd640 10 monclient: dump: 2023-10-30T12:05:15.725 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:15.726 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:15.727 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f9684d99640 4 set_mon_vals no callback set 2023-10-30T12:05:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 10 client.84129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 10 monclient: _renew_subs 2023-10-30T12:05:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968808fad0 con 0x7f9688096080 2023-10-30T12:05:15.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.685+0000 7f968cb16640 1 librados: init done 2023-10-30T12:05:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9676ffd640 1 -- 172.21.15.50:0/1960019976 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f967c039070 con 0x7f9688096080 2023-10-30T12:05:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9676ffd640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f96680392f0 0x7f966803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f968cb16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f968cb16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f968cb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9676ffd640 1 -- 172.21.15.50:0/1960019976 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6533+0+0 (secure 0 0 0) 0x7f967c06cd40 con 0x7f9688096080 2023-10-30T12:05:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9676ffd640 10 client.84129.objecter ms_dispatch 0x7f9688000b90 osd_map(136..136 src has 1..136) v4 2023-10-30T12:05:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9676ffd640 3 client.84129.objecter handle_osd_map got epochs [136,136] > 0 2023-10-30T12:05:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9676ffd640 3 client.84129.objecter handle_osd_map decoding full epoch 136 2023-10-30T12:05:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968804e160 con 0x7f9688096080 2023-10-30T12:05:15.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9676ffd640 20 client.84129.objecter dump_active .. 0 homeless 2023-10-30T12:05:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9685d9b640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f96680392f0 0x7f966803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.686+0000 7f9685d9b640 10 monclient: get_auth_request con 0x7f96680392f0 auth_method 0 2023-10-30T12:05:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.687+0000 7f9685d9b640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f96680392f0 0x7f966803b7b0 secure :-1 s=READY pgs=3937 cs=0 l=1 rev1=1 crypto rx=0x7f968813ba00 tx=0x7f9678009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.687+0000 7f9676ffd640 10 client.84129.objecter ms_handle_connect 0x7f96680392f0 2023-10-30T12:05:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.691+0000 7f9676ffd640 1 -- 172.21.15.50:0/1960019976 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f967c037070 con 0x7f9688096080 2023-10-30T12:05:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.691+0000 7f9676ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.691+0000 7f9676ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:15.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.947+0000 7f968cb16640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:05:15.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.947+0000 7f968cb16640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:05:15.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.947+0000 7f968cb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:15.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.947+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9688137890 con 0x7f9688096080 2023-10-30T12:05:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.948+0000 7f9676ffd640 1 -- 172.21.15.50:0/1960019976 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f967c053080 con 0x7f9688096080 2023-10-30T12:05:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.948+0000 7f9676ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:05:15.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.948+0000 7f9676ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:15.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.948+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.948+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=9,logm=64} crc {auth=986848999,config=3140514009,health=3359852149,kv=2905431493,logm=1976551601}) 2023-10-30T12:05:15.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.948+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=1,health=11,kv=7,logm=68} crc {auth=2413975747,config=1228700967,health=4180235331,kv=3824953070,logm=3118090651}) 2023-10-30T12:05:15.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.948+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.948+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=9,logm=64} crc {auth=986848999,config=3140514009,health=3359852149,kv=2905431493,logm=1976551601}) 2023-10-30T12:05:15.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.948+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=2631455432,config=3140514009,health=3748659849,kv=2905431493,logm=1144728174}) 2023-10-30T12:05:15.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.949+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.949+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728966709}) 2023-10-30T12:05:15.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.949+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634931417}) 2023-10-30T12:05:15.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.949+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.949+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728966709}) 2023-10-30T12:05:15.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.949+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532922781}) 2023-10-30T12:05:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 10 librados: watch_flush enter 2023-10-30T12:05:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 10 librados: watch_flush exit 2023-10-30T12:05:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 20 client.84129.objecter shutdown clearing up homeless session... 2023-10-30T12:05:15.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 10 client.84129.objecter successfully canceled tick 2023-10-30T12:05:15.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.950+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.950+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385666085}) 2023-10-30T12:05:15.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.950+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902371185}) 2023-10-30T12:05:15.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f96680392f0 msgr2=0x7f966803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.950+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.950+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385666085}) 2023-10-30T12:05:15.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.950+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027614113}) 2023-10-30T12:05:15.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f96680392f0 0x7f966803b7b0 secure :-1 s=READY pgs=3937 cs=0 l=1 rev1=1 crypto rx=0x7f968813ba00 tx=0x7f9678009060 comp rx=0 tx=0).stop 2023-10-30T12:05:15.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 10 monclient: shutdown 2023-10-30T12:05:15.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:15.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9688096080 msgr2=0x7f96881ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:15.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9688096080 0x7f96881ca670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f967c000c00 tx=0x7f967c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 shutdown_connections 2023-10-30T12:05:15.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f96680392f0 0x7f966803b7b0 unknown :-1 s=CLOSED pgs=3937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968809f5b0 0x7f9688095580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9688096b70 0x7f9688095040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 --2- 172.21.15.50:0/1960019976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9688096080 0x7f96881ca670 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:15.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 >> 172.21.15.50:0/1960019976 conn(0x7f96880a05c0 msgr2=0x7f96880a2120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:15.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 shutdown_connections 2023-10-30T12:05:15.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 -- 172.21.15.50:0/1960019976 wait complete. 2023-10-30T12:05:15.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:15.950+0000 7f968cb16640 1 librados: shutdown 2023-10-30T12:05:15.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.951+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.951+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672414449}) 2023-10-30T12:05:15.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.951+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863921662}) 2023-10-30T12:05:15.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.951+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.951+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672414449}) 2023-10-30T12:05:15.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.951+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943316347}) 2023-10-30T12:05:15.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882260644}) 2023-10-30T12:05:15.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1470175660}) 2023-10-30T12:05:15.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882260644}) 2023-10-30T12:05:15.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=362350124}) 2023-10-30T12:05:15.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274645452}) 2023-10-30T12:05:15.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2596804705}) 2023-10-30T12:05:15.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.952+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274645452}) 2023-10-30T12:05:15.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.953+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822359314}) 2023-10-30T12:05:15.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.953+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.953+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589023026}) 2023-10-30T12:05:15.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.953+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299771900}) 2023-10-30T12:05:15.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.953+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.953+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589023026}) 2023-10-30T12:05:15.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.953+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3761153748}) 2023-10-30T12:05:15.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.954+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.954+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213744833}) 2023-10-30T12:05:15.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.954+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=247373146}) 2023-10-30T12:05:15.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.954+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.954+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213744833}) 2023-10-30T12:05:15.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.954+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2509680067}) 2023-10-30T12:05:15.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.955+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.955+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845471820}) 2023-10-30T12:05:15.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.955+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2541573925}) 2023-10-30T12:05:15.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.955+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.955+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845471820}) 2023-10-30T12:05:15.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.955+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648059415}) 2023-10-30T12:05:15.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.956+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.956+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273919159}) 2023-10-30T12:05:15.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.956+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1634817798}) 2023-10-30T12:05:15.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.956+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.956+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273919159}) 2023-10-30T12:05:15.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.956+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226214802}) 2023-10-30T12:05:15.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.957+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.957+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076199468}) 2023-10-30T12:05:15.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.957+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896573099}) 2023-10-30T12:05:15.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.957+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.957+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076199468}) 2023-10-30T12:05:15.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.957+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1529267872}) 2023-10-30T12:05:15.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256829684}) 2023-10-30T12:05:15.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190612535}) 2023-10-30T12:05:15.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256829684}) 2023-10-30T12:05:15.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937812542}) 2023-10-30T12:05:15.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177835738}) 2023-10-30T12:05:15.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556094406}) 2023-10-30T12:05:15.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177835738}) 2023-10-30T12:05:15.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.958+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592058778}) 2023-10-30T12:05:15.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.959+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.959+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683569667}) 2023-10-30T12:05:15.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.959+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536024426}) 2023-10-30T12:05:15.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.959+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.959+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683569667}) 2023-10-30T12:05:15.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.959+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3480312525}) 2023-10-30T12:05:15.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.960+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.960+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369441436}) 2023-10-30T12:05:15.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.960+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4020909227}) 2023-10-30T12:05:15.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.960+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.960+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369441436}) 2023-10-30T12:05:15.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.960+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188204679}) 2023-10-30T12:05:15.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.961+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.961+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3595619242}) 2023-10-30T12:05:15.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.961+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076302028}) 2023-10-30T12:05:15.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.961+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.961+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3595619242}) 2023-10-30T12:05:15.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.961+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237856809}) 2023-10-30T12:05:15.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:05:15.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.962+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.962+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2066801187}) 2023-10-30T12:05:15.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.962+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899283423}) 2023-10-30T12:05:15.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.962+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.962+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2066801187}) 2023-10-30T12:05:15.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.962+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3552141131}) 2023-10-30T12:05:15.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.963+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.963+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699668227}) 2023-10-30T12:05:15.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.963+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1763724233}) 2023-10-30T12:05:15.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.963+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.963+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699668227}) 2023-10-30T12:05:15.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.963+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015860681}) 2023-10-30T12:05:15.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192197125}) 2023-10-30T12:05:15.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1329525416}) 2023-10-30T12:05:15.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192197125}) 2023-10-30T12:05:15.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556721288}) 2023-10-30T12:05:15.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.964+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475587494}) 2023-10-30T12:05:15.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.965+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073744280}) 2023-10-30T12:05:15.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.965+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.965+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475587494}) 2023-10-30T12:05:15.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.965+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3892246420}) 2023-10-30T12:05:15.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.965+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.966+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=1302336247,mdsmap=3984425268,mgr=1598784600,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=741697843}) 2023-10-30T12:05:15.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.966+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1756345018,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1041968964}) 2023-10-30T12:05:15.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.966+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.966+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=1302336247,mdsmap=3984425268,mgr=1598784600,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=741697843}) 2023-10-30T12:05:15.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.966+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=82,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2208234267,mdsmap=2441318933,mgr=1597429152,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3167432521}) 2023-10-30T12:05:15.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.967+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.967+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=4,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=60} crc {mgrstat=2713223816,mon_config_key=1821124913,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1856407482,osd_snap=3556931772,osdmap=1354925408}) 2023-10-30T12:05:15.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.967+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20,osdmap=63} crc {mgrstat=2713223816,mon_config_key=3217606506,monmap=4204254258,osd_metadata=1549492346,osd_pg_creating=1856407482,osd_snap=487555120,osdmap=48569048}) 2023-10-30T12:05:15.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.967+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.967+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=4,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=60} crc {mgrstat=2713223816,mon_config_key=1821124913,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1856407482,osd_snap=3556931772,osdmap=1354925408}) 2023-10-30T12:05:15.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.967+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=4,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=57} crc {mgrstat=2713223816,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1856407482,osd_snap=3387285674,osdmap=2387059094}) 2023-10-30T12:05:15.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.968+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.968+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3653711662}) 2023-10-30T12:05:15.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.968+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3553222302}) 2023-10-30T12:05:15.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.968+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.968+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3653711662}) 2023-10-30T12:05:15.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.968+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4271551656}) 2023-10-30T12:05:15.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.973+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.973+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1027488383}) 2023-10-30T12:05:15.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.974+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=1565793852}) 2023-10-30T12:05:15.998 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.974+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:05:15.998 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.974+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1027488383}) 2023-10-30T12:05:15.998 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:05:15.974+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=690646835}) 2023-10-30T12:05:16.986 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:05:16.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:17.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.102+0000 7f8cbe44f640 10 monclient: get_monmap_and_config 2023-10-30T12:05:17.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.102+0000 7f8cbe44f640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.102+0000 7f8cbe44f640 10 monclient: monmap: 2023-10-30T12:05:17.104 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.104 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.104 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:17.103327+0000 2023-10-30T12:05:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:17.103327+0000 2023-10-30T12:05:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:17.106 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 1 Processor -- start 2023-10-30T12:05:17.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 1 -- start start 2023-10-30T12:05:17.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient: init 2023-10-30T12:05:17.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T12:05:17.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8cb80a4b60 0x7f8cb81326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb80a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:17.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8cb8099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:17.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8cb8132be0 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:17.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80a4f40 con 0x7f8cb8132be0 2023-10-30T12:05:17.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80a4150 con 0x7f8cb8099a40 2023-10-30T12:05:17.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80a3d70 con 0x7f8cb80a4b60 2023-10-30T12:05:17.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbe44f640 10 monclient(hunting): authenticate will time out at 1162.286792s 2023-10-30T12:05:17.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbc9ec640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8cb8132be0 0x7f8cb80a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43294/0 (socket says 172.21.15.50:43294) 2023-10-30T12:05:17.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.103+0000 7f8cbc9ec640 1 -- 172.21.15.50:0/1318754718 learned_addr learned my addr 172.21.15.50:0/1318754718 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:17.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 1 --2- 172.21.15.50:0/1318754718 >> v2:172.21.15.118:3300/0 conn(0x7f8cb80a4b60 0x7f8cb81326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): get_auth_request con 0x7f8cb80a4b60 auth_method 0 2023-10-30T12:05:17.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 1 -- 172.21.15.50:0/1318754718 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512776561 0 0) 0x7f8cb80a4f40 con 0x7f8cb8132be0 2023-10-30T12:05:17.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:05:17.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): my global_id is 84141 2023-10-30T12:05:17.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 1 -- 172.21.15.50:0/1318754718 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca0003520 con 0x7f8cb8132be0 2023-10-30T12:05:17.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 1 -- 172.21.15.50:0/1318754718 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012920663 0 0) 0x7f8cb80a4150 con 0x7f8cb8099a40 2023-10-30T12:05:17.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:05:17.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 10 monclient(hunting): my global_id is 28342 2023-10-30T12:05:17.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb6ffd640 1 -- 172.21.15.50:0/1318754718 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb80a4f40 con 0x7f8cb8099a40 2023-10-30T12:05:17.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_done global_id 28339 payload 274 2023-10-30T12:05:17.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 1 -- 172.21.15.50:0/1318754718 >> v1:172.21.15.118:6789/0 conn(0x7f8cb8099a40 legacy=0x7f8cb80a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 1 -- 172.21.15.50:0/1318754718 >> v1:172.21.15.50:6789/0 conn(0x7f8cb8132be0 legacy=0x7f8cb80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 1 monclient: found mon.noname-c 2023-10-30T12:05:17.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:05:17.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.104+0000 7f8cb7fff640 1 -- 172.21.15.50:0/1318754718 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80a3170 con 0x7f8cb80a4b60 2023-10-30T12:05:17.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb7fff640 1 --2- 172.21.15.50:0/1318754718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80a4b60 0x7f8cb81326a0 secure :-1 s=READY pgs=7155 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0097a0 tx=0x7f8cac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a08427f406a81a5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 1 -- 172.21.15.50:0/1318754718 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac02e070 con 0x7f8cb80a4b60 2023-10-30T12:05:17.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:17.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: dump: 2023-10-30T12:05:17.124 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.124 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.124 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.124 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.124 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.124 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.125 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.125 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.125 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.125 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 1 -- 172.21.15.50:0/1318754718 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac0315a0 con 0x7f8cb80a4b60 2023-10-30T12:05:17.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 1 -- 172.21.15.50:0/1318754718 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac031930 con 0x7f8cb80a4b60 2023-10-30T12:05:17.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 5 monclient: authenticate success, global_id 28339 2023-10-30T12:05:17.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:17.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 4 set_mon_vals no callback set 2023-10-30T12:05:17.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:05:17.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cb6ffd640 10 monclient: dump: 2023-10-30T12:05:17.128 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.128 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.128 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.128 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.128 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.129 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.129 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.129 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.129 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.129 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 10 monclient: shutdown 2023-10-30T12:05:17.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:17.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 1 -- 172.21.15.50:0/1318754718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80a4b60 msgr2=0x7f8cb81326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/1318754718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80a4b60 0x7f8cb81326a0 secure :-1 s=READY pgs=7155 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0097a0 tx=0x7f8cac02b840 comp rx=0 tx=0).stop 2023-10-30T12:05:17.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 1 -- 172.21.15.50:0/1318754718 shutdown_connections 2023-10-30T12:05:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.105+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/1318754718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb80a4b60 0x7f8cb81326a0 unknown :-1 s=CLOSED pgs=7155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 1 -- 172.21.15.50:0/1318754718 >> 172.21.15.50:0/1318754718 conn(0x7f8cb80a8060 msgr2=0x7f8cb80a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 1 -- 172.21.15.50:0/1318754718 shutdown_connections 2023-10-30T12:05:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 1 -- 172.21.15.50:0/1318754718 wait complete. 2023-10-30T12:05:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 10 monclient: monmap: 2023-10-30T12:05:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:17.133 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 1 librados: starting msgr at 2023-10-30T12:05:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 1 librados: starting objecter 2023-10-30T12:05:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.106+0000 7f8cbe44f640 1 Processor -- start 2023-10-30T12:05:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 -- start start 2023-10-30T12:05:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 librados: setting wanted keys 2023-10-30T12:05:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 librados: calling monclient init 2023-10-30T12:05:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient: init 2023-10-30T12:05:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:05:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8cb80a4b60 0x7f8cb809d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb80a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cb8132be0 0x7f8cb809dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb8132be0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb809e190 0x7f8cb81d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb809e190 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb812e8b0 con 0x7f8cb8132be0 2023-10-30T12:05:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb812eb20 con 0x7f8cb809e190 2023-10-30T12:05:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80a18d0 con 0x7f8cb80a4b60 2023-10-30T12:05:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbe44f640 10 monclient(hunting): authenticate will time out at 1162.290700s 2023-10-30T12:05:17.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cb8132be0 0x7f8cb809dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8cb80a4b60 0x7f8cb809d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8cb80a4b60 0x7f8cb809d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43016/0 (socket says 172.21.15.50:43016) 2023-10-30T12:05:17.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 10 monclient(hunting): get_auth_request con 0x7f8cb8132be0 auth_method 0 2023-10-30T12:05:17.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 1 -- 172.21.15.50:0/2269843512 learned_addr learned my addr 172.21.15.50:0/2269843512 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:17.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 10 monclient(hunting): get_auth_request con 0x7f8cb80a4b60 auth_method 0 2023-10-30T12:05:17.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbc9ec640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb809e190 0x7f8cb81d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbc9ec640 10 monclient(hunting): get_auth_request con 0x7f8cb809e190 auth_method 0 2023-10-30T12:05:17.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbc9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbc9ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.107+0000 7f8cbc9ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbc9ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbc9ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbc9ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 10 monclient(hunting): handle_auth_done global_id 44666 payload 824 2023-10-30T12:05:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 1 -- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb809e190 msgr2=0x7f8cb81d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb809e190 0x7f8cb81d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 1 -- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cb8132be0 msgr2=0x7f8cb809dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cb8132be0 0x7f8cb809dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 1 monclient: found mon.noname-c 2023-10-30T12:05:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 1 -- 172.21.15.50:0/2269843512 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac009400 con 0x7f8cb80a4b60 2023-10-30T12:05:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb77fe640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cb8132be0 0x7f8cb809dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb7fff640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8cb80a4b60 0x7f8cb809d710 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f8cac02be50 tx=0x7f8cac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 client.?.objecter ms_handle_connect 0x7f8cb80a4b60 2023-10-30T12:05:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 1 -- 172.21.15.50:0/2269843512 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac032840 con 0x7f8cb80a4b60 2023-10-30T12:05:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: dump: 2023-10-30T12:05:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.147 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 1 -- 172.21.15.50:0/2269843512 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac031f00 con 0x7f8cb80a4b60 2023-10-30T12:05:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 5 monclient: authenticate success, global_id 44666 2023-10-30T12:05:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 10 monclient: _renew_subs 2023-10-30T12:05:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb80a0400 con 0x7f8cb80a4b60 2023-10-30T12:05:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 1 -- 172.21.15.50:0/2269843512 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac032c00 con 0x7f8cb80a4b60 2023-10-30T12:05:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb4ff9640 10 monclient: dump: 2023-10-30T12:05:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.152 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cb67fc640 4 set_mon_vals no callback set 2023-10-30T12:05:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 10 client.44666.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 10 monclient: _renew_subs 2023-10-30T12:05:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb809eea0 con 0x7f8cb80a4b60 2023-10-30T12:05:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.108+0000 7f8cbe44f640 1 librados: init done 2023-10-30T12:05:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb4ff9640 1 -- 172.21.15.50:0/2269843512 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8cac0453f0 con 0x7f8cb80a4b60 2023-10-30T12:05:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb4ff9640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ca40392f0 0x7f8ca403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb4ff9640 1 -- 172.21.15.50:0/2269843512 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6697+0+0 (secure 0 0 0) 0x7f8cac06cbc0 con 0x7f8cb80a4b60 2023-10-30T12:05:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb4ff9640 10 client.44666.objecter ms_dispatch 0x7f8cb8000b90 osd_map(138..138 src has 1..138) v4 2023-10-30T12:05:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb4ff9640 3 client.44666.objecter handle_osd_map got epochs [138,138] > 0 2023-10-30T12:05:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb4ff9640 3 client.44666.objecter handle_osd_map decoding full epoch 138 2023-10-30T12:05:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cbe44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cbe44f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cbe44f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb812ec60 con 0x7f8cb80a4b60 2023-10-30T12:05:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb4ff9640 20 client.44666.objecter dump_active .. 0 homeless 2023-10-30T12:05:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb77fe640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ca40392f0 0x7f8ca403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.109+0000 7f8cb77fe640 10 monclient: get_auth_request con 0x7f8ca40392f0 auth_method 0 2023-10-30T12:05:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.110+0000 7f8cb4ff9640 10 client.44666.objecter ms_handle_connect 0x7f8ca40392f0 2023-10-30T12:05:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.110+0000 7f8cb77fe640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ca40392f0 0x7f8ca403b7b0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8007fd0 tx=0x7f8ca8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.114+0000 7f8cb4ff9640 1 -- 172.21.15.50:0/2269843512 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8cb812ec60 con 0x7f8cb80a4b60 2023-10-30T12:05:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.114+0000 7f8cb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.114+0000 7f8cb4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:17.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.362+0000 7f8cbe44f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:17.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.362+0000 7f8cbe44f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:17.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.362+0000 7f8cbe44f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.362+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8cb812ec60 con 0x7f8cb80a4b60 2023-10-30T12:05:17.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.362+0000 7f8cb4ff9640 1 -- 172.21.15.50:0/2269843512 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f8cb812ec60 con 0x7f8cb80a4b60 2023-10-30T12:05:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.362+0000 7f8cb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.362+0000 7f8cb4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:17.365 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 10 librados: watch_flush enter 2023-10-30T12:05:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 10 librados: watch_flush exit 2023-10-30T12:05:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 20 client.44666.objecter shutdown clearing up homeless session... 2023-10-30T12:05:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 10 client.44666.objecter successfully canceled tick 2023-10-30T12:05:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ca40392f0 msgr2=0x7f8ca403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ca40392f0 0x7f8ca403b7b0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8007fd0 tx=0x7f8ca8009040 comp rx=0 tx=0).stop 2023-10-30T12:05:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 10 monclient: shutdown 2023-10-30T12:05:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8cb80a4b60 msgr2=0x7f8cb809d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8cb80a4b60 0x7f8cb809d710 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f8cac02be50 tx=0x7f8cac0046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 shutdown_connections 2023-10-30T12:05:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ca40392f0 0x7f8ca403b7b0 unknown :-1 s=CLOSED pgs=3939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8cb809e190 0x7f8cb81d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cb8132be0 0x7f8cb809dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 --2- 172.21.15.50:0/2269843512 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8cb80a4b60 0x7f8cb809d710 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 >> 172.21.15.50:0/2269843512 conn(0x7f8cb80a8060 msgr2=0x7f8cb812c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 shutdown_connections 2023-10-30T12:05:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 -- 172.21.15.50:0/2269843512 wait complete. 2023-10-30T12:05:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.364+0000 7f8cbe44f640 1 librados: shutdown 2023-10-30T12:05:17.376 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:05:17.376 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:05:17.376 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:05:17.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:05:17.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.482+0000 7efe29b2c640 10 monclient: get_monmap_and_config 2023-10-30T12:05:17.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.482+0000 7efe29b2c640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.482+0000 7efe29b2c640 10 monclient: monmap: 2023-10-30T12:05:17.484 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.485 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.485 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:17.483573+0000 2023-10-30T12:05:17.485 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:17.483573+0000 2023-10-30T12:05:17.485 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.485 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.486 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:17.486 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:17.486 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:17.486 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:17.487 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:17.487 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 1 Processor -- start 2023-10-30T12:05:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 1 -- start start 2023-10-30T12:05:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient: init 2023-10-30T12:05:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-10-30T12:05:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe2406afc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2406b640 0x7efe240a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient(hunting): picked mon.noname-b con 0x7efe2406b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efe2413b070 0x7efe2413d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient(hunting): picked mon.noname-c con 0x7efe2413b070 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2406bde0 con 0x7efe2406afc0 2023-10-30T12:05:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2406b3a0 con 0x7efe2413b070 2023-10-30T12:05:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2409e540 con 0x7efe2406b640 2023-10-30T12:05:17.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe29b2c640 10 monclient(hunting): authenticate will time out at 1162.666750s 2023-10-30T12:05:17.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe22ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2406b640 0x7efe240a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe22ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2406b640 0x7efe240a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43018/0 (socket says 172.21.15.50:43018) 2023-10-30T12:05:17.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe22ffd640 1 -- 172.21.15.50:0/1462312070 learned_addr learned my addr 172.21.15.50:0/1462312070 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:17.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe22ffd640 10 monclient(hunting): get_auth_request con 0x7efe2406b640 auth_method 0 2023-10-30T12:05:17.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe22ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.483+0000 7efe22ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe23fff640 1 --2- 172.21.15.50:0/1462312070 >> v2:172.21.15.118:3300/0 conn(0x7efe2413b070 0x7efe2413d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe23fff640 10 monclient(hunting): get_auth_request con 0x7efe2413b070 auth_method 0 2023-10-30T12:05:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 10 monclient(hunting): handle_auth_done global_id 44669 payload 274 2023-10-30T12:05:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 1 -- 172.21.15.50:0/1462312070 >> v2:172.21.15.118:3300/0 conn(0x7efe2413b070 msgr2=0x7efe2413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 1 --2- 172.21.15.50:0/1462312070 >> v2:172.21.15.118:3300/0 conn(0x7efe2413b070 0x7efe2413d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 1 -- 172.21.15.50:0/1462312070 >> v1:172.21.15.118:6789/0 conn(0x7efe2406afc0 legacy=0x7efe240a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 1 monclient: found mon.noname-b 2023-10-30T12:05:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 1 -- 172.21.15.50:0/1462312070 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2409fa10 con 0x7efe2406b640 2023-10-30T12:05:17.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.484+0000 7efe22ffd640 1 --2- 172.21.15.50:0/1462312070 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2406b640 0x7efe240a23a0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7efe18009870 tx=0x7efe1802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4c65d64e324c6e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 1 -- 172.21.15.50:0/1462312070 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18031740 con 0x7efe2406b640 2023-10-30T12:05:17.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:05:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: dump: 2023-10-30T12:05:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.507 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.507 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.507 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.507 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.507 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.507 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.508 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 5 monclient: authenticate success, global_id 44669 2023-10-30T12:05:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 1 -- 172.21.15.50:0/1462312070 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe180318a0 con 0x7efe2406b640 2023-10-30T12:05:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 4 set_mon_vals no callback set 2023-10-30T12:05:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 1 -- 172.21.15.50:0/1462312070 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18031c30 con 0x7efe2406b640 2023-10-30T12:05:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe227fc640 10 monclient: dump: 2023-10-30T12:05:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.512 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.512 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.512 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.512 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 10 monclient: shutdown 2023-10-30T12:05:17.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:17.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 -- 172.21.15.50:0/1462312070 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2406b640 msgr2=0x7efe240a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1462312070 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2406b640 0x7efe240a23a0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7efe18009870 tx=0x7efe1802b7e0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 -- 172.21.15.50:0/1462312070 shutdown_connections 2023-10-30T12:05:17.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1462312070 >> v2:172.21.15.118:3300/0 conn(0x7efe2413b070 0x7efe2413d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1462312070 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2406b640 0x7efe240a23a0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 -- 172.21.15.50:0/1462312070 >> 172.21.15.50:0/1462312070 conn(0x7efe240a6200 msgr2=0x7efe240a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:17.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 -- 172.21.15.50:0/1462312070 shutdown_connections 2023-10-30T12:05:17.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.485+0000 7efe29b2c640 1 -- 172.21.15.50:0/1462312070 wait complete. 2023-10-30T12:05:17.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 10 monclient: monmap: 2023-10-30T12:05:17.515 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.515 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:17.515 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:17.515 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:17.516 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.516 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.516 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:17.516 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:17.516 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:17.517 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 1 librados: starting msgr at 2023-10-30T12:05:17.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 1 librados: starting objecter 2023-10-30T12:05:17.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 1 Processor -- start 2023-10-30T12:05:17.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 1 -- start start 2023-10-30T12:05:17.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 1 librados: setting wanted keys 2023-10-30T12:05:17.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 1 librados: calling monclient init 2023-10-30T12:05:17.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.486+0000 7efe29b2c640 10 monclient: init 2023-10-30T12:05:17.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:05:17.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2406afc0 0x7efe24133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): picked mon.noname-a con 0x7efe2406afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:17.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe2406b640 0x7efe2411c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): picked mon.noname-b con 0x7efe2406b640 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2413b070 0x7efe2411cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): picked mon.noname-c con 0x7efe2413b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe24070de0 con 0x7efe2406afc0 2023-10-30T12:05:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe241385b0 con 0x7efe2406b640 2023-10-30T12:05:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe241cafd0 con 0x7efe2413b070 2023-10-30T12:05:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe29b2c640 10 monclient(hunting): authenticate will time out at 1162.670156s 2023-10-30T12:05:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2413b070 0x7efe2411cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2406afc0 0x7efe24133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2413b070 0x7efe2411cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43032/0 (socket says 172.21.15.50:43032) 2023-10-30T12:05:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 -- 172.21.15.50:0/1004928187 learned_addr learned my addr 172.21.15.50:0/1004928187 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2406afc0 0x7efe24133d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37924/0 (socket says 172.21.15.50:37924) 2023-10-30T12:05:17.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 10 monclient(hunting): get_auth_request con 0x7efe2406afc0 auth_method 0 2023-10-30T12:05:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): get_auth_request con 0x7efe2413b070 auth_method 0 2023-10-30T12:05:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe22ffd640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe2406b640 0x7efe2411c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe22ffd640 10 monclient(hunting): get_auth_request con 0x7efe2406b640 auth_method 0 2023-10-30T12:05:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe22ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe22ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient(hunting): handle_auth_done global_id 44672 payload 824 2023-10-30T12:05:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe2406b640 msgr2=0x7efe2411c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe2406b640 0x7efe2411c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2406afc0 msgr2=0x7efe24133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2406afc0 0x7efe24133d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 monclient: found mon.noname-c 2023-10-30T12:05:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 -- 172.21.15.50:0/1004928187 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe180094d0 con 0x7efe2413b070 2023-10-30T12:05:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe22ffd640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe2406b640 0x7efe2411c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:05:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe237fe640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2406afc0 0x7efe24133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe23fff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2413b070 0x7efe2411cb80 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7efe1c007e40 tx=0x7efe1c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 10 client.?.objecter ms_handle_connect 0x7efe2413b070 2023-10-30T12:05:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c014620 con 0x7efe2413b070 2023-10-30T12:05:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 10 monclient: dump: 2023-10-30T12:05:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.530 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.531 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.532 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.487+0000 7efe0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 5 monclient: authenticate success, global_id 44672 2023-10-30T12:05:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c00ee90 con 0x7efe2413b070 2023-10-30T12:05:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 10 monclient: _renew_subs 2023-10-30T12:05:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe241206b0 con 0x7efe2413b070 2023-10-30T12:05:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0149e0 con 0x7efe2413b070 2023-10-30T12:05:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:17.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 10 monclient: dump: 2023-10-30T12:05:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.535 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.536 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.536 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 10 client.44672.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 10 monclient: _renew_subs 2023-10-30T12:05:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2411fe10 con 0x7efe2413b070 2023-10-30T12:05:17.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 1 librados: init done 2023-10-30T12:05:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe21ffb640 4 set_mon_vals no callback set 2023-10-30T12:05:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2411d0c0 0x7efe2411d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe241201b0 con 0x7efe2411d0c0 2023-10-30T12:05:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efe1c014b40 con 0x7efe2413b070 2023-10-30T12:05:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2411d0c0 0x7efe2411d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 10 monclient: get_auth_request con 0x7efe2411d0c0 auth_method 0 2023-10-30T12:05:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.488+0000 7efe0bfff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe100392f0 0x7efe1003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6697+0+0 (secure 0 0 0) 0x7efe1c04b720 con 0x7efe2413b070 2023-10-30T12:05:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 client.44672.objecter ms_dispatch 0x7efe24000b90 osd_map(138..138 src has 1..138) v4 2023-10-30T12:05:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 3 client.44672.objecter handle_osd_map got epochs [138,138] > 0 2023-10-30T12:05:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 3 client.44672.objecter handle_osd_map decoding full epoch 138 2023-10-30T12:05:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe237fe640 10 monclient(hunting): handle_auth_done global_id 84150 payload 824 2023-10-30T12:05:17.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe237fe640 1 -- 172.21.15.50:0/1004928187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe2409f0b0 con 0x7efe2411d0c0 2023-10-30T12:05:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 20 client.44672.objecter dump_active .. 0 homeless 2023-10-30T12:05:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe22ffd640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe100392f0 0x7efe1003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe22ffd640 10 monclient: get_auth_request con 0x7efe100392f0 auth_method 0 2023-10-30T12:05:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe237fe640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2411d0c0 0x7efe2411d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe1400fb10 tx=0x7efe140044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 client.44672.objecter ms_handle_connect 0x7efe2411d0c0 2023-10-30T12:05:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 client.44672.objecter resend_mon_ops 2023-10-30T12:05:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400daa0 con 0x7efe2411d0c0 2023-10-30T12:05:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 monclient: dump: 2023-10-30T12:05:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.544 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.545 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.546 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 client.44672.objecter ms_handle_connect 0x7efe100392f0 2023-10-30T12:05:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe22ffd640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe100392f0 0x7efe1003b7b0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7efe18035000 tx=0x7efe18008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7efe1400dc00 con 0x7efe2411d0c0 2023-10-30T12:05:17.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2411d0c0 msgr2=0x7efe2411d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2411d0c0 0x7efe2411d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe1400fb10 tx=0x7efe140044c0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2411d0c0 msgr2=0x7efe2411d4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:17.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.489+0000 7efe0bfff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe2411d0c0 0x7efe2411d4a0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe1400fb10 tx=0x7efe140044c0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe29b2c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe24135260 0x7efe24135670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe29b2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2411ffa0 con 0x7efe24135260 2023-10-30T12:05:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe24135260 0x7efe24135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient: get_auth_request con 0x7efe24135260 auth_method 0 2023-10-30T12:05:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 10 monclient(hunting): handle_auth_done global_id 84153 payload 824 2023-10-30T12:05:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 1 -- 172.21.15.50:0/1004928187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efe2409df80 con 0x7efe24135260 2023-10-30T12:05:17.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe237fe640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe24135260 0x7efe24135670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efe14019000 tx=0x7efe14003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 10 client.44672.objecter ms_handle_connect 0x7efe24135260 2023-10-30T12:05:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 10 client.44672.objecter resend_mon_ops 2023-10-30T12:05:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14024070 con 0x7efe24135260 2023-10-30T12:05:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 10 monclient: dump: 2023-10-30T12:05:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.554 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.555 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.556 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7efe14014790 con 0x7efe24135260 2023-10-30T12:05:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:17.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe24135260 msgr2=0x7efe24135670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe24135260 0x7efe24135670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efe14019000 tx=0x7efe14003760 comp rx=0 tx=0).stop 2023-10-30T12:05:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe24135260 msgr2=0x7efe24135670 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.496+0000 7efe0bfff640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe24135260 0x7efe24135670 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efe14019000 tx=0x7efe14003760 comp rx=0 tx=0).stop 2023-10-30T12:05:17.558 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:05:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 50, 2023-10-30T12:05:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:05:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:05:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:17.568 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:17.568 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:17.568 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:17.568 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:17.568 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:17.569 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:17.569 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:17.569 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:17.569 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:17.569 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:17.570 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:17.570 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:17.570 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:17.570 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:17.570 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:17.571 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:17.571 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:17.571 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:17.571 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.571 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:17.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.572 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:17.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.572 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:17.572 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:17.573 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:17.573 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:17.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.573 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:17.573 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:17.574 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.574 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:17.575 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:17.575 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.575 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.576 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:17.576 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:17.576 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:17.576 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:17.577 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:17.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.577 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:17.578 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:17.578 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:17.578 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:17.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.578 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:17.579 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:17.579 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.579 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:17.579 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:17.580 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.580 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.580 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.580 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:17.581 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:17.581 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:17.581 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:17.581 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:17.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.582 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:17.582 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:17.582 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:17.582 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:17.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.583 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:17.583 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:17.583 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.584 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:17.584 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:17.584 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.584 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:17.585 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:17.586 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:17.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.586 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:17.587 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:17.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.587 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:17.587 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:17.588 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:17.588 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.588 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.588 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:17.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.589 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:17.589 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:17.589 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:17.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.589 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.590 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:17.590 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 10 librados: watch_flush enter 2023-10-30T12:05:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 10 librados: watch_flush exit 2023-10-30T12:05:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 20 client.44672.objecter shutdown clearing up homeless session... 2023-10-30T12:05:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 10 client.44672.objecter successfully canceled tick 2023-10-30T12:05:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe100392f0 msgr2=0x7efe1003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe100392f0 0x7efe1003b7b0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7efe18035000 tx=0x7efe18008030 comp rx=0 tx=0).stop 2023-10-30T12:05:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 10 monclient: shutdown 2023-10-30T12:05:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:17.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2413b070 msgr2=0x7efe2411cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2413b070 0x7efe2411cb80 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7efe1c007e40 tx=0x7efe1c00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe23fff640 1 -- 172.21.15.50:0/1004928187 reap_dead start 2023-10-30T12:05:17.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 shutdown_connections 2023-10-30T12:05:17.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 --2- 172.21.15.50:0/1004928187 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe2413b070 0x7efe2411cb80 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 >> 172.21.15.50:0/1004928187 conn(0x7efe240a6200 msgr2=0x7efe24137480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 shutdown_connections 2023-10-30T12:05:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 -- 172.21.15.50:0/1004928187 wait complete. 2023-10-30T12:05:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.497+0000 7efe29b2c640 1 librados: shutdown 2023-10-30T12:05:17.596 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:05:17.596 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:05:17.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:05:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.701+0000 7f1241597640 10 monclient: get_monmap_and_config 2023-10-30T12:05:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.701+0000 7f1241597640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.701+0000 7f1241597640 10 monclient: monmap: 2023-10-30T12:05:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:17.702362+0000 2023-10-30T12:05:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:17.702362+0000 2023-10-30T12:05:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:17.706 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.701+0000 7f1241597640 1 Processor -- start 2023-10-30T12:05:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.701+0000 7f1241597640 1 -- start start 2023-10-30T12:05:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.701+0000 7f1241597640 10 monclient: init 2023-10-30T12:05:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T12:05:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a2ae0 0x7f123c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient(hunting): picked mon.noname-b con 0x7f123c0a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f123c0a2ec0 0x7f123c1356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient(hunting): picked mon.noname-a con 0x7f123c0a2ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:17.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f123c0a32e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:17.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c0a4190 con 0x7f123c0a32e0 2023-10-30T12:05:17.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123c0a2990 con 0x7f123c0a2ec0 2023-10-30T12:05:17.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c0a1630 con 0x7f123c0a2ae0 2023-10-30T12:05:17.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f1241597640 10 monclient(hunting): authenticate will time out at 1162.885407s 2023-10-30T12:05:17.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a2ae0 0x7f123c0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f123c0a2ec0 0x7f123c1356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f123c0a2ec0 0x7f123c1356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53792/0 (socket says 172.21.15.50:53792) 2023-10-30T12:05:17.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a2ae0 0x7f123c0a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47042/0 (socket says 172.21.15.50:47042) 2023-10-30T12:05:17.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 -- 172.21.15.50:0/3553253114 learned_addr learned my addr 172.21.15.50:0/3553253114 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:17.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): get_auth_request con 0x7f123c0a2ec0 auth_method 0 2023-10-30T12:05:17.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 10 monclient(hunting): get_auth_request con 0x7f123c0a2ae0 auth_method 0 2023-10-30T12:05:17.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient(hunting): handle_auth_done global_id 84156 payload 274 2023-10-30T12:05:17.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 -- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a2ae0 msgr2=0x7f123c0a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 --2- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a2ae0 0x7f123c0a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 -- 172.21.15.50:0/3553253114 >> v1:172.21.15.118:6789/0 conn(0x7f123c0a32e0 legacy=0x7f123c135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 monclient: found mon.noname-a 2023-10-30T12:05:17.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123a7fc640 1 -- 172.21.15.50:0/3553253114 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c09a2e0 con 0x7f123c0a2ec0 2023-10-30T12:05:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.702+0000 7f123affd640 1 --2- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a2ae0 0x7f123c0a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f123a7fc640 1 --2- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ec0 0x7f123c1356e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1224009870 tx=0x7f122402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4deae82cbab738e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 1 -- 172.21.15.50:0/3553253114 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224031740 con 0x7f123c0a2ec0 2023-10-30T12:05:17.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:17.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: dump: 2023-10-30T12:05:17.724 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.724 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.724 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.725 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.725 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.725 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.725 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.725 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.726 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 1 -- 172.21.15.50:0/3553253114 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12240318a0 con 0x7f123c0a2ec0 2023-10-30T12:05:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1241597640 5 monclient: authenticate success, global_id 84156 2023-10-30T12:05:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1241597640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1241597640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:17.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 1 -- 172.21.15.50:0/3553253114 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224031c30 con 0x7f123c0a2ec0 2023-10-30T12:05:17.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1241597640 4 set_mon_vals no callback set 2023-10-30T12:05:17.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:17.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.703+0000 7f1239ffb640 10 monclient: dump: 2023-10-30T12:05:17.729 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.730 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.730 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.730 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.730 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.731 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.731 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.732 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 10 monclient: shutdown 2023-10-30T12:05:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:17.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 -- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ec0 msgr2=0x7f123c1356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 --2- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ec0 0x7f123c1356e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1224009870 tx=0x7f122402b7e0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 -- 172.21.15.50:0/3553253114 shutdown_connections 2023-10-30T12:05:17.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 --2- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ec0 0x7f123c1356e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 --2- 172.21.15.50:0/3553253114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a2ae0 0x7f123c0a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 -- 172.21.15.50:0/3553253114 >> 172.21.15.50:0/3553253114 conn(0x7f123c0a80b0 msgr2=0x7f123c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:17.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 -- 172.21.15.50:0/3553253114 shutdown_connections 2023-10-30T12:05:17.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 1 -- 172.21.15.50:0/3553253114 wait complete. 2023-10-30T12:05:17.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.704+0000 7f1241597640 10 monclient: monmap: 2023-10-30T12:05:17.736 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.736 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:17.736 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:17.736 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:17.739 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.739 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.739 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:17.739 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:17.740 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:17.740 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 librados: starting msgr at 2023-10-30T12:05:17.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 librados: starting objecter 2023-10-30T12:05:17.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 Processor -- start 2023-10-30T12:05:17.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 -- start start 2023-10-30T12:05:17.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 librados: setting wanted keys 2023-10-30T12:05:17.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 librados: calling monclient init 2023-10-30T12:05:17.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient: init 2023-10-30T12:05:17.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:05:17.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ae0 0x7f123c11e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): picked mon.noname-a con 0x7f123c0a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:17.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f123c0a2ec0 0x7f123c132760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): picked mon.noname-b con 0x7f123c0a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:17.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c132ca0 0x7f123c1c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): picked mon.noname-c con 0x7f123c132ca0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:17.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c120d20 con 0x7f123c0a2ae0 2023-10-30T12:05:17.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c11eaa0 con 0x7f123c0a2ec0 2023-10-30T12:05:17.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c0a1790 con 0x7f123c132ca0 2023-10-30T12:05:17.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f1241597640 10 monclient(hunting): authenticate will time out at 1162.888816s 2023-10-30T12:05:17.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ae0 0x7f123c11e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ae0 0x7f123c11e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53796/0 (socket says 172.21.15.50:53796) 2023-10-30T12:05:17.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123affd640 1 -- 172.21.15.50:0/3721195284 learned_addr learned my addr 172.21.15.50:0/3721195284 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:17.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c132ca0 0x7f123c1c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123affd640 10 monclient(hunting): get_auth_request con 0x7f123c0a2ae0 auth_method 0 2023-10-30T12:05:17.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123b7fe640 10 monclient(hunting): get_auth_request con 0x7f123c132ca0 auth_method 0 2023-10-30T12:05:17.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.705+0000 7f123a7fc640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f123c0a2ec0 0x7f123c132760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123a7fc640 10 monclient(hunting): get_auth_request con 0x7f123c0a2ec0 auth_method 0 2023-10-30T12:05:17.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 10 monclient(hunting): handle_auth_done global_id 84159 payload 824 2023-10-30T12:05:17.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c132ca0 msgr2=0x7f123c1c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c132ca0 0x7f123c1c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f123c0a2ec0 msgr2=0x7f123c132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f123c0a2ec0 0x7f123c132760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 1 monclient: found mon.noname-a 2023-10-30T12:05:17.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 1 -- 172.21.15.50:0/3721195284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1234009400 con 0x7f123c0a2ae0 2023-10-30T12:05:17.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123b7fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c132ca0 0x7f123c1c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:17.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f123affd640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ae0 0x7f123c11e560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f123400ebe0 tx=0x7f123400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 client.?.objecter ms_handle_connect 0x7f123c0a2ae0 2023-10-30T12:05:17.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:17.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234009760 con 0x7f123c0a2ae0 2023-10-30T12:05:17.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:17.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: dump: 2023-10-30T12:05:17.756 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.756 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.756 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.757 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.757 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.757 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.757 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.757 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.758 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.758 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123400cc30 con 0x7f123c0a2ae0 2023-10-30T12:05:17.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123400eee0 con 0x7f123c0a2ae0 2023-10-30T12:05:17.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:17.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12337fe640 10 monclient: dump: 2023-10-30T12:05:17.760 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.761 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.761 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.761 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.761 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.761 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.762 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.762 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 5 monclient: authenticate success, global_id 84159 2023-10-30T12:05:17.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 10 monclient: _renew_subs 2023-10-30T12:05:17.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12240094d0 con 0x7f123c0a2ae0 2023-10-30T12:05:17.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f12397fa640 4 set_mon_vals no callback set 2023-10-30T12:05:17.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 10 client.84159.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:17.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 10 monclient: _renew_subs 2023-10-30T12:05:17.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c11f240 con 0x7f123c0a2ae0 2023-10-30T12:05:17.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.706+0000 7f1241597640 1 librados: init done 2023-10-30T12:05:17.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1234013470 con 0x7f123c0a2ae0 2023-10-30T12:05:17.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f1241597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f1241597640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a32e0 0x7f123c1206c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f1241597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c073ff0 con 0x7f123c0a32e0 2023-10-30T12:05:17.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a32e0 0x7f123c1206c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f12337fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f121c0392f0 0x7f121c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient: get_auth_request con 0x7f123c0a32e0 auth_method 0 2023-10-30T12:05:17.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6697+0+0 (secure 0 0 0) 0x7f123404b480 con 0x7f123c0a2ae0 2023-10-30T12:05:17.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f12337fe640 10 client.84159.objecter ms_dispatch 0x7f123c000b90 osd_map(138..138 src has 1..138) v4 2023-10-30T12:05:17.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f12337fe640 3 client.84159.objecter handle_osd_map got epochs [138,138] > 0 2023-10-30T12:05:17.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f12337fe640 3 client.84159.objecter handle_osd_map decoding full epoch 138 2023-10-30T12:05:17.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f12337fe640 20 client.84159.objecter dump_active .. 0 homeless 2023-10-30T12:05:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123b7fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f121c0392f0 0x7f121c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123b7fe640 10 monclient: get_auth_request con 0x7f121c0392f0 auth_method 0 2023-10-30T12:05:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 10 monclient(hunting): handle_auth_done global_id 44681 payload 824 2023-10-30T12:05:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.707+0000 7f123a7fc640 1 -- 172.21.15.50:0/3721195284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f123c09a5c0 con 0x7f123c0a32e0 2023-10-30T12:05:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 client.84159.objecter ms_handle_connect 0x7f123c0a32e0 2023-10-30T12:05:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 client.84159.objecter resend_mon_ops 2023-10-30T12:05:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f123a7fc640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a32e0 0x7f123c1206c0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f123c0a5500 tx=0x7f12240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224033bc0 con 0x7f123c0a32e0 2023-10-30T12:05:17.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:17.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 monclient: dump: 2023-10-30T12:05:17.770 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.770 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.771 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.771 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.771 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.771 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.772 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 client.84159.objecter ms_handle_connect 0x7f121c0392f0 2023-10-30T12:05:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f123b7fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f121c0392f0 0x7f121c03b7b0 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 crypto rx=0x7f123c072030 tx=0x7f122c003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1224041070 con 0x7f123c0a32e0 2023-10-30T12:05:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:17.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a32e0 msgr2=0x7f123c1206c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a32e0 0x7f123c1206c0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f123c0a5500 tx=0x7f12240046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a32e0 msgr2=0x7f123c1206c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:17.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.708+0000 7f12337fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c0a32e0 0x7f123c1206c0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f1241597640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:17.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f1241597640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c1ca410 0x7f123c1ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f1241597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c073a20 con 0x7f123c1ca410 2023-10-30T12:05:17.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f123a7fc640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c1ca410 0x7f123c1ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f123a7fc640 10 monclient: get_auth_request con 0x7f123c1ca410 auth_method 0 2023-10-30T12:05:17.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f123a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f123a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.714+0000 7f123a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f123a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f123a7fc640 10 monclient(hunting): handle_auth_done global_id 44684 payload 824 2023-10-30T12:05:17.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f123a7fc640 1 -- 172.21.15.50:0/3721195284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f123c099b40 con 0x7f123c1ca410 2023-10-30T12:05:17.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f123a7fc640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c1ca410 0x7f123c1ca7f0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f122402b820 tx=0x7f1224031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 10 client.84159.objecter ms_handle_connect 0x7f123c1ca410 2023-10-30T12:05:17.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 10 client.84159.objecter resend_mon_ops 2023-10-30T12:05:17.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224052470 con 0x7f123c1ca410 2023-10-30T12:05:17.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:17.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 10 monclient: dump: 2023-10-30T12:05:17.781 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.781 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.781 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.781 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.781 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.782 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.782 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.782 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.782 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.783 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f12240525d0 con 0x7f123c1ca410 2023-10-30T12:05:17.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:17.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:17.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c1ca410 msgr2=0x7f123c1ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c1ca410 0x7f123c1ca7f0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f122402b820 tx=0x7f1224031000 comp rx=0 tx=0).stop 2023-10-30T12:05:17.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c1ca410 msgr2=0x7f123c1ca7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:17.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f12337fe640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f123c1ca410 0x7f123c1ca7f0 secure :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f122402b820 tx=0x7f1224031000 comp rx=0 tx=0).stop 2023-10-30T12:05:17.784 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:17.785 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:17.785 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:17.785 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:05:17.785 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 50, 2023-10-30T12:05:17.786 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:17.786 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:17.786 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:05:17.786 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:17.786 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.787 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:05:17.787 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:17.787 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:17.787 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:17.787 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:17.788 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:17.788 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:17.788 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:17.788 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:17.788 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:17.789 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:17.789 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:17.789 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:17.789 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:17.790 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.790 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:17.790 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:17.790 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:17.790 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:17.791 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:17.791 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:17.791 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:17.791 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:17.791 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:17.792 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:17.792 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:17.792 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:17.792 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.793 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:17.793 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:17.793 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:17.793 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:17.794 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:17.794 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:17.794 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:17.794 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:17.794 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:17.795 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:17.795 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:17.795 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:17.795 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:17.795 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:17.796 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:17.796 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:17.796 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:17.796 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:17.796 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:17.797 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:17.797 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:17.797 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:17.797 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:17.797 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:17.797 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:17.798 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:17.798 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:17.798 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.798 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:17.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.799 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:17.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.799 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:17.799 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:17.799 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:17.800 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:17.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.800 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:17.800 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:17.800 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.801 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:17.801 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:17.801 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.801 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.802 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.802 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:17.802 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:17.802 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:17.803 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:17.803 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:17.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.803 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:17.803 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:17.804 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:17.804 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:17.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.804 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:17.804 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:17.805 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.805 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:17.805 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:17.806 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.806 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.806 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.806 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:17.807 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:17.807 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:17.807 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:17.807 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:17.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.808 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:17.808 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:17.808 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:17.808 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:17.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.809 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:17.809 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:17.809 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.810 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:17.810 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:17.810 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:17.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.811 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.811 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:17.811 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:17.811 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:17.812 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:17.812 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:17.812 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.812 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.812 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:17.813 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:17.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.813 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:17.813 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:17.813 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:17.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.814 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.814 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:05:17.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.814 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:17.815 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:17.815 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:05:17.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.815 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:17.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:17.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:17.816 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:17.816 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:17.816 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:05:17.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:17.816 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:17.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:17.817 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:17.817 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f1241597640 10 librados: watch_flush enter 2023-10-30T12:05:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f1241597640 10 librados: watch_flush exit 2023-10-30T12:05:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f1241597640 20 client.84159.objecter shutdown clearing up homeless session... 2023-10-30T12:05:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.715+0000 7f1241597640 10 client.84159.objecter successfully canceled tick 2023-10-30T12:05:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f121c0392f0 msgr2=0x7f121c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f121c0392f0 0x7f121c03b7b0 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 crypto rx=0x7f123c072030 tx=0x7f122c003d10 comp rx=0 tx=0).stop 2023-10-30T12:05:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 10 monclient: shutdown 2023-10-30T12:05:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ae0 msgr2=0x7f123c11e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 --2- 172.21.15.50:0/3721195284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f123c0a2ae0 0x7f123c11e560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f123400ebe0 tx=0x7f123400c520 comp rx=0 tx=0).stop 2023-10-30T12:05:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f123b7fe640 1 -- 172.21.15.50:0/3721195284 reap_dead start 2023-10-30T12:05:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f123b7fe640 1 -- 172.21.15.50:0/3721195284 reap_dead start 2023-10-30T12:05:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 shutdown_connections 2023-10-30T12:05:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 >> 172.21.15.50:0/3721195284 conn(0x7f123c0a80b0 msgr2=0x7f123c0a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 shutdown_connections 2023-10-30T12:05:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 -- 172.21.15.50:0/3721195284 wait complete. 2023-10-30T12:05:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.716+0000 7f1241597640 1 librados: shutdown 2023-10-30T12:05:17.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:05:17.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:05:17.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:05:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.936+0000 7f58acaa2640 10 monclient: get_monmap_and_config 2023-10-30T12:05:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.936+0000 7f58acaa2640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.936+0000 7f58acaa2640 10 monclient: monmap: 2023-10-30T12:05:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:17.937811+0000 2023-10-30T12:05:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:17.937811+0000 2023-10-30T12:05:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:17.941 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 1 Processor -- start 2023-10-30T12:05:17.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 1 -- start start 2023-10-30T12:05:17.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient: init 2023-10-30T12:05:17.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:05:17.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f58a80a2ae0 0x7f58a80a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient(hunting): picked mon.noname-a con 0x7f58a80a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:17.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f58a80a2ec0 0x7f58a81356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient(hunting): picked mon.noname-c con 0x7f58a80a2ec0 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58a8135c20 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a80a4190 con 0x7f58a8135c20 2023-10-30T12:05:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a80a2990 con 0x7f58a80a2ae0 2023-10-30T12:05:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a80a1630 con 0x7f58a80a2ec0 2023-10-30T12:05:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.937+0000 7f58acaa2640 10 monclient(hunting): authenticate will time out at 1163.120976s 2023-10-30T12:05:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f58a80a2ae0 0x7f58a80a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a6d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f58a8135c20 0x7f58a813ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53844/0 (socket says 172.21.15.50:53844) 2023-10-30T12:05:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a6d9d640 1 -- 172.21.15.50:0/1900273543 learned_addr learned my addr 172.21.15.50:0/1900273543 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): get_auth_request con 0x7f58a80a2ae0 auth_method 0 2023-10-30T12:05:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 1 --2- 172.21.15.50:0/1900273543 >> v2:172.21.15.118:3300/0 conn(0x7f58a80a2ec0 0x7f58a81356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 10 monclient(hunting): get_auth_request con 0x7f58a80a2ec0 auth_method 0 2023-10-30T12:05:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f589ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient(hunting): handle_auth_done global_id 84162 payload 274 2023-10-30T12:05:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 1 -- 172.21.15.50:0/1900273543 >> v2:172.21.15.118:3300/0 conn(0x7f58a80a2ec0 msgr2=0x7f58a81356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 1 --2- 172.21.15.50:0/1900273543 >> v2:172.21.15.118:3300/0 conn(0x7f58a80a2ec0 0x7f58a81356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 1 -- 172.21.15.50:0/1900273543 >> v1:172.21.15.50:6789/0 conn(0x7f58a8135c20 legacy=0x7f58a813ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 1 monclient: found mon.noname-a 2023-10-30T12:05:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a5d9b640 1 -- 172.21.15.50:0/1900273543 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070024985 0 0) 0x7f58a0001490 con 0x7f58a8135c20 2023-10-30T12:05:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a659c640 1 -- 172.21.15.50:0/1900273543 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a809a2e0 con 0x7f58a80a2ae0 2023-10-30T12:05:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.938+0000 7f58a5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:05:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a659c640 1 --2- 172.21.15.50:0/1900273543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a80a2ae0 0x7f58a80a2450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58a80a4c80 tx=0x7f589802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d51a528a127853 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 1 -- 172.21.15.50:0/1900273543 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898031710 con 0x7f58a80a2ae0 2023-10-30T12:05:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: dump: 2023-10-30T12:05:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.962 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 1 -- 172.21.15.50:0/1900273543 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589802f050 con 0x7f58a80a2ae0 2023-10-30T12:05:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 5 monclient: authenticate success, global_id 84162 2023-10-30T12:05:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 1 -- 172.21.15.50:0/1900273543 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898031a90 con 0x7f58a80a2ae0 2023-10-30T12:05:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 4 set_mon_vals no callback set 2023-10-30T12:05:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58a5d9b640 10 monclient: dump: 2023-10-30T12:05:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.967 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 10 monclient: shutdown 2023-10-30T12:05:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 1 -- 172.21.15.50:0/1900273543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a80a2ae0 msgr2=0x7f58a80a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 1 --2- 172.21.15.50:0/1900273543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a80a2ae0 0x7f58a80a2450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58a80a4c80 tx=0x7f589802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 1 -- 172.21.15.50:0/1900273543 shutdown_connections 2023-10-30T12:05:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 1 --2- 172.21.15.50:0/1900273543 >> v2:172.21.15.118:3300/0 conn(0x7f58a80a2ec0 0x7f58a81356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 1 --2- 172.21.15.50:0/1900273543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a80a2ae0 0x7f58a80a2450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.939+0000 7f58acaa2640 1 -- 172.21.15.50:0/1900273543 >> 172.21.15.50:0/1900273543 conn(0x7f58a80a80b0 msgr2=0x7f58a80a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:17.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.940+0000 7f58acaa2640 1 -- 172.21.15.50:0/1900273543 shutdown_connections 2023-10-30T12:05:17.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.940+0000 7f58acaa2640 1 -- 172.21.15.50:0/1900273543 wait complete. 2023-10-30T12:05:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.940+0000 7f58acaa2640 10 monclient: build_initial_monmap 2023-10-30T12:05:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.940+0000 7f58acaa2640 10 monclient: monmap: 2023-10-30T12:05:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:17.971 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:17.971 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:17.971 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:17.972 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:17.972 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.972 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:17.972 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:17.972 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:17.973 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.940+0000 7f58acaa2640 1 librados: starting msgr at 2023-10-30T12:05:17.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.940+0000 7f58acaa2640 1 librados: starting objecter 2023-10-30T12:05:17.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 Processor -- start 2023-10-30T12:05:17.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 -- start start 2023-10-30T12:05:17.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 librados: setting wanted keys 2023-10-30T12:05:17.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 librados: calling monclient init 2023-10-30T12:05:17.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient: init 2023-10-30T12:05:17.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:17.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:05:17.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a80a2ae0 0x7f58a81ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): picked mon.noname-b con 0x7f58a80a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:17.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58a80a2ec0 0x7f58a81322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): picked mon.noname-c con 0x7f58a80a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:17.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a8135c20 0x7f58a8132800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): picked mon.noname-a con 0x7f58a8135c20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:17.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81cae20 con 0x7f58a8135c20 2023-10-30T12:05:17.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a8122520 con 0x7f58a80a2ae0 2023-10-30T12:05:17.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a80a1790 con 0x7f58a80a2ec0 2023-10-30T12:05:17.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:17.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58acaa2640 10 monclient(hunting): authenticate will time out at 1163.124530s 2023-10-30T12:05:17.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a8135c20 0x7f58a8132800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58a80a2ec0 0x7f58a81322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a8135c20 0x7f58a8132800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53812/0 (socket says 172.21.15.50:53812) 2023-10-30T12:05:17.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 -- 172.21.15.50:0/3114702475 learned_addr learned my addr 172.21.15.50:0/3114702475 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:17.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): get_auth_request con 0x7f58a8135c20 auth_method 0 2023-10-30T12:05:17.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58a80a2ec0 0x7f58a81322c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47078/0 (socket says 172.21.15.50:47078) 2023-10-30T12:05:17.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 10 monclient(hunting): get_auth_request con 0x7f58a80a2ec0 auth_method 0 2023-10-30T12:05:17.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a80a2ae0 0x7f58a81ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 10 monclient(hunting): get_auth_request con 0x7f58a80a2ae0 auth_method 0 2023-10-30T12:05:17.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f589ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient(hunting): handle_auth_done global_id 84168 payload 824 2023-10-30T12:05:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58a80a2ec0 msgr2=0x7f58a81322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58a80a2ec0 0x7f58a81322c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a80a2ae0 msgr2=0x7f58a81ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a80a2ae0 0x7f58a81ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:05:17.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 1 monclient: found mon.noname-a 2023-10-30T12:05:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.941+0000 7f58a6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58a6d9d640 1 -- 172.21.15.50:0/3114702475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5898009400 con 0x7f58a8135c20 2023-10-30T12:05:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589ffff640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f58a80a2ec0 0x7f58a81322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58a6d9d640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a8135c20 0x7f58a8132800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f58a000c7d0 tx=0x7f58a000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:17.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 client.?.objecter ms_handle_connect 0x7f58a8135c20 2023-10-30T12:05:17.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:17.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a000a640 con 0x7f58a8135c20 2023-10-30T12:05:17.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:17.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: dump: 2023-10-30T12:05:17.986 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.986 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.987 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.987 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.987 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.987 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.988 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.988 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.988 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: _finish_auth 0 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: _check_auth_tickets 2023-10-30T12:05:17.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:17.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 5 monclient: authenticate success, global_id 84168 2023-10-30T12:05:17.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a000a7c0 con 0x7f58a8135c20 2023-10-30T12:05:17.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:17.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 10 monclient: _renew_subs 2023-10-30T12:05:17.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a000abb0 con 0x7f58a8135c20 2023-10-30T12:05:17.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a8135510 con 0x7f58a8135c20 2023-10-30T12:05:17.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58a559a640 4 set_mon_vals no callback set 2023-10-30T12:05:17.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:17.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:17.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f589effd640 10 monclient: dump: 2023-10-30T12:05:17.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:17.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:17.993 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.993 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:17.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:17.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:17.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:17.994 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:17.994 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:17.994 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:17.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 10 client.84168.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:17.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 10 monclient: _renew_subs 2023-10-30T12:05:17.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:17.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a81231f0 con 0x7f58a8135c20 2023-10-30T12:05:17.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.942+0000 7f58acaa2640 1 librados: init done 2023-10-30T12:05:17.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58acaa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:17.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58acaa2640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a811fa20 0x7f58a811fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58acaa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:17.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a8073ff0 con 0x7f58a811fa20 2023-10-30T12:05:17.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f58a000ad10 con 0x7f58a8135c20 2023-10-30T12:05:17.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a811fa20 0x7f58a811fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:17.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient: get_auth_request con 0x7f58a811fa20 auth_method 0 2023-10-30T12:05:17.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:17.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:17.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:17.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:17.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:17.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:17.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589effd640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5888039320 0x7f588803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:17.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f58a0049f60 con 0x7f58a8135c20 2023-10-30T12:05:17.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 10 monclient(hunting): handle_auth_done global_id 28366 payload 824 2023-10-30T12:05:17.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 1 -- 172.21.15.50:0/3114702475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58a809a5c0 con 0x7f58a811fa20 2023-10-30T12:05:17.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589effd640 10 client.84168.objecter ms_dispatch 0x7f58a8000b90 osd_map(139..139 src has 1..139) v4 2023-10-30T12:05:18.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589effd640 3 client.84168.objecter handle_osd_map got epochs [139,139] > 0 2023-10-30T12:05:18.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589effd640 3 client.84168.objecter handle_osd_map decoding full epoch 139 2023-10-30T12:05:18.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589ffff640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5888039320 0x7f588803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:18.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589ffff640 10 monclient: get_auth_request con 0x7f5888039320 auth_method 0 2023-10-30T12:05:18.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f589effd640 20 client.84168.objecter dump_active .. 0 homeless 2023-10-30T12:05:18.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.943+0000 7f58a659c640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a811fa20 0x7f58a811fe00 secure :-1 s=READY pgs=7159 cs=0 l=1 rev1=1 crypto rx=0x7f58a80a54d0 tx=0x7f58980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:18.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 client.84168.objecter ms_handle_connect 0x7f58a811fa20 2023-10-30T12:05:18.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 client.84168.objecter resend_mon_ops 2023-10-30T12:05:18.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589803d910 con 0x7f58a811fa20 2023-10-30T12:05:18.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:18.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:05:18.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 monclient: dump: 2023-10-30T12:05:18.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:18.003 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.003 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:18.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:18.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 client.84168.objecter ms_handle_connect 0x7f5888039320 2023-10-30T12:05:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589ffff640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5888039320 0x7f588803b7e0 secure :-1 s=READY pgs=3942 cs=0 l=1 rev1=1 crypto rx=0x7f58a8121b20 tx=0x7f5890009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5898041070 con 0x7f58a811fa20 2023-10-30T12:05:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a811fa20 msgr2=0x7f58a811fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a811fa20 0x7f58a811fe00 secure :-1 s=READY pgs=7159 cs=0 l=1 rev1=1 crypto rx=0x7f58a80a54d0 tx=0x7f58980046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a811fa20 msgr2=0x7f58a811fe00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.944+0000 7f589effd640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a811fa20 0x7f58a811fe00 unknown :-1 s=CLOSED pgs=7159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.950+0000 7f58acaa2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:18.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.950+0000 7f58acaa2640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a8120340 0x7f58a809dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.950+0000 7f58acaa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.950+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a8073a20 con 0x7f58a8120340 2023-10-30T12:05:18.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a8120340 0x7f58a809dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:18.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient: get_auth_request con 0x7f58a8120340 auth_method 0 2023-10-30T12:05:18.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:18.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:18.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:18.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:18.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:18.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:18.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 10 monclient(hunting): handle_auth_done global_id 28372 payload 824 2023-10-30T12:05:18.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 1 -- 172.21.15.50:0/3114702475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f58a806b060 con 0x7f58a8120340 2023-10-30T12:05:18.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f58a659c640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a8120340 0x7f58a809dd70 secure :-1 s=READY pgs=7161 cs=0 l=1 rev1=1 crypto rx=0x7f589802b7f0 tx=0x7f5898009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:18.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f589effd640 10 client.84168.objecter ms_handle_connect 0x7f58a8120340 2023-10-30T12:05:18.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f589effd640 10 client.84168.objecter resend_mon_ops 2023-10-30T12:05:18.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898046800 con 0x7f58a8120340 2023-10-30T12:05:18.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f589effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:18.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f589effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:05:18.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.951+0000 7f589effd640 10 monclient: dump: 2023-10-30T12:05:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.013 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:18.014 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.014 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:18.014 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:18.014 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:18.014 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f5898051480 con 0x7f58a8120340 2023-10-30T12:05:18.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f589effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:18.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f589effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:18.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a8120340 msgr2=0x7f58a809dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f589effd640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a8120340 0x7f58a809dd70 secure :-1 s=READY pgs=7161 cs=0 l=1 rev1=1 crypto rx=0x7f589802b7f0 tx=0x7f5898009190 comp rx=0 tx=0).stop 2023-10-30T12:05:18.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f589effd640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a8120340 msgr2=0x7f58a809dd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:18.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f589effd640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f58a8120340 0x7f58a809dd70 unknown :-1 s=CLOSED pgs=7161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.016 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:05:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 50, 2023-10-30T12:05:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:05:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:05:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:05:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:05:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:05:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:05:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:18.051 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:18.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 10 librados: watch_flush enter 2023-10-30T12:05:18.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 10 librados: watch_flush exit 2023-10-30T12:05:18.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 20 client.84168.objecter shutdown clearing up homeless session... 2023-10-30T12:05:18.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 10 client.84168.objecter successfully canceled tick 2023-10-30T12:05:18.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5888039320 msgr2=0x7f588803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5888039320 0x7f588803b7e0 secure :-1 s=READY pgs=3942 cs=0 l=1 rev1=1 crypto rx=0x7f58a8121b20 tx=0x7f5890009190 comp rx=0 tx=0).stop 2023-10-30T12:05:18.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 10 monclient: shutdown 2023-10-30T12:05:18.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:18.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a8135c20 msgr2=0x7f58a8132800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 --2- 172.21.15.50:0/3114702475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a8135c20 0x7f58a8132800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f58a000c7d0 tx=0x7f58a000cc90 comp rx=0 tx=0).stop 2023-10-30T12:05:18.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58a6d9d640 1 -- 172.21.15.50:0/3114702475 reap_dead start 2023-10-30T12:05:18.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58a6d9d640 1 -- 172.21.15.50:0/3114702475 reap_dead start 2023-10-30T12:05:18.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 shutdown_connections 2023-10-30T12:05:18.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 >> 172.21.15.50:0/3114702475 conn(0x7f58a80a80b0 msgr2=0x7f58a80a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:18.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 shutdown_connections 2023-10-30T12:05:18.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 -- 172.21.15.50:0/3114702475 wait complete. 2023-10-30T12:05:18.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:17.952+0000 7f58acaa2640 1 librados: shutdown 2023-10-30T12:05:18.057 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:05:18.057 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-10-30T12:05:18.057 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:05:18.057 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-10-30T12:05:18.058 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:05:18.058 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:05:18.058 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:05:18.075 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:05:18.075 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:05:18.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.179+0000 7f8ef1026640 10 monclient: get_monmap_and_config 2023-10-30T12:05:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.179+0000 7f8ef1026640 10 monclient: build_initial_monmap 2023-10-30T12:05:18.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.179+0000 7f8ef1026640 10 monclient: monmap: 2023-10-30T12:05:18.181 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:18.181 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.182 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:18.180233+0000 2023-10-30T12:05:18.182 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:18.180233+0000 2023-10-30T12:05:18.182 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:18.182 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:18.184 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.179+0000 7f8ef1026640 1 Processor -- start 2023-10-30T12:05:18.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.179+0000 7f8ef1026640 1 -- start start 2023-10-30T12:05:18.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.179+0000 7f8ef1026640 10 monclient: init 2023-10-30T12:05:18.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:18.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:05:18.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8eec0a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:18.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8eec099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:18.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8eec132db0 0x7f8eec13e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient(hunting): picked mon.noname-c con 0x7f8eec132db0 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:18.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eec0a4f40 con 0x7f8eec0a4b60 2023-10-30T12:05:18.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eec0a4150 con 0x7f8eec099a40 2023-10-30T12:05:18.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eec0a3d70 con 0x7f8eec132db0 2023-10-30T12:05:18.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:18.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ef1026640 10 monclient(hunting): authenticate will time out at 1163.363415s 2023-10-30T12:05:18.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8eead9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8eec0a4b60 0x7f8eec1326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53852/0 (socket says 172.21.15.50:53852) 2023-10-30T12:05:18.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8eead9d640 1 -- 172.21.15.50:0/268961535 learned_addr learned my addr 172.21.15.50:0/268961535 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:18.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8eea59c640 1 -- 172.21.15.50:0/268961535 >> v1:172.21.15.118:6789/0 conn(0x7f8eec099a40 legacy=0x7f8eec13aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:18.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8eeb59e640 1 -- 172.21.15.50:0/268961535 >> v2:172.21.15.118:3300/0 conn(0x7f8eec132db0 msgr2=0x7f8eec13e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:18.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8eeb59e640 1 --2- 172.21.15.50:0/268961535 >> v2:172.21.15.118:3300/0 conn(0x7f8eec132db0 0x7f8eec13e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:18.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050213326 0 0) 0x7f8eec0a4f40 con 0x7f8eec0a4b60 2023-10-30T12:05:18.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:05:18.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:18.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:18.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient(hunting): my global_id is 84177 2023-10-30T12:05:18.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed8003520 con 0x7f8eec0a4b60 2023-10-30T12:05:18.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2882700135 0 0) 0x7f8ed8003520 con 0x7f8eec0a4b60 2023-10-30T12:05:18.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 >> v2:172.21.15.118:3300/0 conn(0x7f8eec132db0 msgr2=0x7f8eec13e1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:18.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 --2- 172.21.15.50:0/268961535 >> v2:172.21.15.118:3300/0 conn(0x7f8eec132db0 0x7f8eec13e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 >> v1:172.21.15.118:6789/0 conn(0x7f8eec099a40 legacy=0x7f8eec13aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:18.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:05:18.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:05:18.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:18.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:05:18.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec09a440 con 0x7f8eec0a4b60 2023-10-30T12:05:18.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient: _finish_auth 0 2023-10-30T12:05:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:05:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f8edc0019f0 con 0x7f8eec0a4b60 2023-10-30T12:05:18.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.180+0000 7f8ee9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:05:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient: dump: 2023-10-30T12:05:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:18.205 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:05:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 >> v1:172.21.15.50:6789/0 conn(0x7f8eec0a4b60 legacy=0x7f8eec1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:05:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ed8003970 0x7f8ed8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): picked mon.c con 0x7f8ed8003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed80047c0 0x7f8ed8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): picked mon.a con 0x7f8ed80047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ed8005330 0x7f8ed800b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): picked mon.b con 0x7f8ed8005330 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed800c250 con 0x7f8ed80047c0 2023-10-30T12:05:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed800c3c0 con 0x7f8ed8005330 2023-10-30T12:05:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed800c580 con 0x7f8ed8003970 2023-10-30T12:05:18.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:18.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ee9d9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:05:18.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ed8003970 0x7f8ed8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:18.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): get_auth_request con 0x7f8ed8003970 auth_method 0 2023-10-30T12:05:18.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:18.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:18.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:18.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed80047c0 0x7f8ed8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:18.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eeb59e640 1 -- 172.21.15.50:0/268961535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ed8005330 msgr2=0x7f8ed800b9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:18.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eeb59e640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ed8005330 0x7f8ed800b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:18.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 10 monclient(hunting): get_auth_request con 0x7f8ed80047c0 auth_method 0 2023-10-30T12:05:18.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:18.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:18.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:18.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:18.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:18.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:05:18.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:18.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:18.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:05:18.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient(hunting): handle_auth_done global_id 84177 payload 293 2023-10-30T12:05:18.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 1 -- 172.21.15.50:0/268961535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ed8005330 msgr2=0x7f8ed800b9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:18.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ed8005330 0x7f8ed800b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 1 -- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed80047c0 msgr2=0x7f8ed8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:18.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed80047c0 0x7f8ed8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient: _finish_hunting 0 2023-10-30T12:05:18.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 1 monclient: found mon.c 2023-10-30T12:05:18.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:18.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:18.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eeb59e640 1 -- 172.21.15.50:0/268961535 reap_dead start 2023-10-30T12:05:18.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 1 -- 172.21.15.50:0/268961535 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed8003520 con 0x7f8ed8003970 2023-10-30T12:05:18.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient: _finish_auth 0 2023-10-30T12:05:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 10 monclient: _check_auth_tickets 2023-10-30T12:05:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eead9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ef1026640 5 monclient: authenticate success, global_id 84177 2023-10-30T12:05:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8ef1026640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.181+0000 7f8eea59c640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed80047c0 0x7f8ed8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:18.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8eead9d640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ed8003970 0x7f8ed8003d70 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00a820 tx=0x7f8ed80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be4ecf5bd6e6d4fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:18.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0072d0 con 0x7f8ed8003970 2023-10-30T12:05:18.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:18.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:18.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 10 monclient: dump: 2023-10-30T12:05:18.217 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:18.218 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.218 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.218 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.218 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:18.218 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.219 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:18.219 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:18.219 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:18.219 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc02f070 con 0x7f8ed8003970 2023-10-30T12:05:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 1 -- 172.21.15.50:0/268961535 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0076b0 con 0x7f8ed8003970 2023-10-30T12:05:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:18.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 4 set_mon_vals no callback set 2023-10-30T12:05:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ee9d9b640 10 monclient: dump: 2023-10-30T12:05:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.221 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.222 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.222 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:18.222 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.222 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:18.223 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:18.223 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:18.223 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 10 monclient: shutdown 2023-10-30T12:05:18.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 1 -- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ed8003970 msgr2=0x7f8ed8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ed8003970 0x7f8ed8003d70 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00a820 tx=0x7f8ed80057f0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 1 -- 172.21.15.50:0/268961535 shutdown_connections 2023-10-30T12:05:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 1 --2- 172.21.15.50:0/268961535 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ed8003970 0x7f8ed8003d70 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 1 -- 172.21.15.50:0/268961535 >> 172.21.15.50:0/268961535 conn(0x7f8eec0a8060 msgr2=0x7f8eec0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 1 -- 172.21.15.50:0/268961535 shutdown_connections 2023-10-30T12:05:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.182+0000 7f8ef1026640 1 -- 172.21.15.50:0/268961535 wait complete. 2023-10-30T12:05:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.183+0000 7f8ef1026640 10 monclient: build_initial_monmap 2023-10-30T12:05:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.183+0000 7f8ef1026640 10 monclient: monmap: 2023-10-30T12:05:18.225 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:18.226 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.227 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:18.227 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:18.227 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:18.227 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.183+0000 7f8ef1026640 1 librados: starting msgr at 2023-10-30T12:05:18.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.183+0000 7f8ef1026640 1 librados: starting objecter 2023-10-30T12:05:18.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 Processor -- start 2023-10-30T12:05:18.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 -- start start 2023-10-30T12:05:18.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 librados: setting wanted keys 2023-10-30T12:05:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 librados: calling monclient init 2023-10-30T12:05:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient: init 2023-10-30T12:05:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:05:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed8003970 0x7f8eec1ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed8003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8eec131540 0x7f8eec1319c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): picked mon.noname-c con 0x7f8eec131540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8eec09e820 0x7f8eec09eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): picked mon.noname-b con 0x7f8eec09e820 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec09fd80 con 0x7f8ed8003970 2023-10-30T12:05:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec11ffc0 con 0x7f8eec09e820 2023-10-30T12:05:18.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:18.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec0a18d0 con 0x7f8eec131540 2023-10-30T12:05:18.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:18.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8ef1026640 10 monclient(hunting): authenticate will time out at 1163.367740s 2023-10-30T12:05:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8eec131540 0x7f8eec1319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed8003970 0x7f8eec1ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed8003970 0x7f8eec1ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53828/0 (socket says 172.21.15.50:53828) 2023-10-30T12:05:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8eec131540 0x7f8eec1319c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47094/0 (socket says 172.21.15.50:47094) 2023-10-30T12:05:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 1 -- 172.21.15.50:0/3818470262 learned_addr learned my addr 172.21.15.50:0/3818470262 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 10 monclient(hunting): get_auth_request con 0x7f8ed8003970 auth_method 0 2023-10-30T12:05:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eeb59e640 1 -- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8eec09e820 msgr2=0x7f8eec09eca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 10 monclient(hunting): get_auth_request con 0x7f8eec131540 auth_method 0 2023-10-30T12:05:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eeb59e640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8eec09e820 0x7f8eec09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.184+0000 7f8eea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:18.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 10 monclient(hunting): handle_auth_done global_id 84180 payload 824 2023-10-30T12:05:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 1 -- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8eec131540 msgr2=0x7f8eec1319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8eec131540 0x7f8eec1319c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 1 -- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8eec09e820 msgr2=0x7f8eec09eca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:05:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8eec09e820 0x7f8eec09eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 10 monclient: _finish_hunting 0 2023-10-30T12:05:18.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 1 monclient: found mon.noname-a 2023-10-30T12:05:18.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:18.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:18.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 1 -- 172.21.15.50:0/3818470262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec11e520 con 0x7f8ed8003970 2023-10-30T12:05:18.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eea59c640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8eec131540 0x7f8eec1319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:18.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8eead9d640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed8003970 0x7f8eec1ca120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8edc001c70 tx=0x7f8edc001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:18.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 client.?.objecter ms_handle_connect 0x7f8ed8003970 2023-10-30T12:05:18.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:18.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 1 -- 172.21.15.50:0/3818470262 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0072e0 con 0x7f8ed8003970 2023-10-30T12:05:18.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: dump: 2023-10-30T12:05:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:18.244 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: _finish_auth 0 2023-10-30T12:05:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:05:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 1 -- 172.21.15.50:0/3818470262 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc02ce90 con 0x7f8ed8003970 2023-10-30T12:05:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 5 monclient: authenticate success, global_id 84180 2023-10-30T12:05:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 10 monclient: _renew_subs 2023-10-30T12:05:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eec11f690 con 0x7f8ed8003970 2023-10-30T12:05:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 1 -- 172.21.15.50:0/3818470262 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc007640 con 0x7f8ed8003970 2023-10-30T12:05:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee37fe640 10 monclient: dump: 2023-10-30T12:05:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:18.248 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:18.249 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ee959a640 4 set_mon_vals no callback set 2023-10-30T12:05:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 10 client.84180.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 10 monclient: _renew_subs 2023-10-30T12:05:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eec0a04c0 con 0x7f8ed8003970 2023-10-30T12:05:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.185+0000 7f8ef1026640 1 librados: init done 2023-10-30T12:05:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ef1026640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ef1026640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ef1026640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ee37fe640 1 -- 172.21.15.50:0/3818470262 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8edc031030 con 0x7f8ed8003970 2023-10-30T12:05:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eec11fa90 con 0x7f8ed8003970 2023-10-30T12:05:18.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ee37fe640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ecc0392a0 0x7f8ecc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:18.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ee37fe640 1 -- 172.21.15.50:0/3818470262 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8edc06cd20 con 0x7f8ed8003970 2023-10-30T12:05:18.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ee37fe640 10 client.84180.objecter ms_dispatch 0x7f8eec000b90 osd_map(139..139 src has 1..139) v4 2023-10-30T12:05:18.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ee37fe640 3 client.84180.objecter handle_osd_map got epochs [139,139] > 0 2023-10-30T12:05:18.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ee37fe640 3 client.84180.objecter handle_osd_map decoding full epoch 139 2023-10-30T12:05:18.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.186+0000 7f8ee37fe640 20 client.84180.objecter dump_active .. 0 homeless 2023-10-30T12:05:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.187+0000 7f8eea59c640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ecc0392a0 0x7f8ecc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.187+0000 7f8eea59c640 10 monclient: get_auth_request con 0x7f8ecc0392a0 auth_method 0 2023-10-30T12:05:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.187+0000 7f8eea59c640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ecc0392a0 0x7f8ecc03b760 secure :-1 s=READY pgs=3944 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40094f0 tx=0x7f8ed40090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.187+0000 7f8ee37fe640 10 client.84180.objecter ms_handle_connect 0x7f8ecc0392a0 2023-10-30T12:05:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.191+0000 7f8ee37fe640 1 -- 172.21.15.50:0/3818470262 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8edc0479a0 con 0x7f8ed8003970 2023-10-30T12:05:18.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.191+0000 7f8ee37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:18.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.191+0000 7f8ee37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:18.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.437+0000 7f8ef1026640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:18.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.437+0000 7f8ef1026640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:18.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.437+0000 7f8ef1026640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:18.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.437+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8eec04e160 con 0x7f8ed8003970 2023-10-30T12:05:18.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.438+0000 7f8ee37fe640 1 -- 172.21.15.50:0/3818470262 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f8edc06c070 con 0x7f8ed8003970 2023-10-30T12:05:18.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.438+0000 7f8ee37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:18.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.438+0000 7f8ee37fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:18.441 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:18.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 10 librados: watch_flush enter 2023-10-30T12:05:18.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 10 librados: watch_flush exit 2023-10-30T12:05:18.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 20 client.84180.objecter shutdown clearing up homeless session... 2023-10-30T12:05:18.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 10 client.84180.objecter successfully canceled tick 2023-10-30T12:05:18.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ecc0392a0 msgr2=0x7f8ecc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ecc0392a0 0x7f8ecc03b760 secure :-1 s=READY pgs=3944 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40094f0 tx=0x7f8ed40090e0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 10 monclient: shutdown 2023-10-30T12:05:18.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:18.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed8003970 msgr2=0x7f8eec1ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:18.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed8003970 0x7f8eec1ca120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8edc001c70 tx=0x7f8edc001d40 comp rx=0 tx=0).stop 2023-10-30T12:05:18.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 shutdown_connections 2023-10-30T12:05:18.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ecc0392a0 0x7f8ecc03b760 unknown :-1 s=CLOSED pgs=3944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8eec09e820 0x7f8eec09eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8eec131540 0x7f8eec1319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 --2- 172.21.15.50:0/3818470262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ed8003970 0x7f8eec1ca120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:18.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 >> 172.21.15.50:0/3818470262 conn(0x7f8eec0a8060 msgr2=0x7f8eec1b9920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:18.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 shutdown_connections 2023-10-30T12:05:18.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 -- 172.21.15.50:0/3818470262 wait complete. 2023-10-30T12:05:18.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:18.440+0000 7f8ef1026640 1 librados: shutdown 2023-10-30T12:05:21.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:21.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.567+0000 7f26eec1f640 10 monclient: get_monmap_and_config 2023-10-30T12:05:21.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.567+0000 7f26eec1f640 10 monclient: build_initial_monmap 2023-10-30T12:05:21.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.567+0000 7f26eec1f640 10 monclient: monmap: 2023-10-30T12:05:21.569 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:21.569 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:21.570 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:21.568525+0000 2023-10-30T12:05:21.570 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:21.568525+0000 2023-10-30T12:05:21.570 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:21.570 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:21.570 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:21.571 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:21.571 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:21.571 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:21.571 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:21.571 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:21.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 1 Processor -- start 2023-10-30T12:05:21.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 1 -- start start 2023-10-30T12:05:21.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient: init 2023-10-30T12:05:21.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:21.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-10-30T12:05:21.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26e8098520 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:21.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8099010 0x7f26e809bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:21.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient(hunting): picked mon.noname-b con 0x7f26e8099010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:21.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f26e81379f0 0x7f26e8139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:21.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient(hunting): picked mon.noname-c con 0x7f26e81379f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:21.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e809ad30 con 0x7f26e8098520 2023-10-30T12:05:21.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:21.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80993f0 con 0x7f26e81379f0 2023-10-30T12:05:21.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:21.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e809c460 con 0x7f26e8099010 2023-10-30T12:05:21.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:21.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26eec1f640 10 monclient(hunting): authenticate will time out at 1166.751746s 2023-10-30T12:05:21.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26e7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8099010 0x7f26e809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:21.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26ec9bb640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f26e8098520 0x7f26e8098900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53860/0 (socket says 172.21.15.50:53860) 2023-10-30T12:05:21.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26ec9bb640 1 -- 172.21.15.50:0/1785048092 learned_addr learned my addr 172.21.15.50:0/1785048092 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:21.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26ed1bc640 1 -- 172.21.15.50:0/1785048092 >> v2:172.21.15.118:3300/0 conn(0x7f26e81379f0 msgr2=0x7f26e8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:21.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26e7fff640 10 monclient(hunting): get_auth_request con 0x7f26e8099010 auth_method 0 2023-10-30T12:05:21.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:21.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26e7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:21.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26e7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:21.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.568+0000 7f26ed1bc640 1 --2- 172.21.15.50:0/1785048092 >> v2:172.21.15.118:3300/0 conn(0x7f26e81379f0 0x7f26e8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:21.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:21.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:21.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:21.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 10 monclient(hunting): handle_auth_done global_id 44696 payload 274 2023-10-30T12:05:21.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e77fe640 1 -- 172.21.15.50:0/1785048092 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644655362 0 0) 0x7f26e809ad30 con 0x7f26e8098520 2023-10-30T12:05:21.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 1 -- 172.21.15.50:0/1785048092 >> v2:172.21.15.118:3300/0 conn(0x7f26e81379f0 msgr2=0x7f26e8139de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:21.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 1 --2- 172.21.15.50:0/1785048092 >> v2:172.21.15.118:3300/0 conn(0x7f26e81379f0 0x7f26e8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 1 -- 172.21.15.50:0/1785048092 >> v1:172.21.15.50:6789/0 conn(0x7f26e8098520 legacy=0x7f26e8098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:21.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:21.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 1 monclient: found mon.noname-b 2023-10-30T12:05:21.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:21.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:05:21.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 1 -- 172.21.15.50:0/1785048092 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e809b260 con 0x7f26e8099010 2023-10-30T12:05:21.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:05:21.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.569+0000 7f26e7fff640 1 --2- 172.21.15.50:0/1785048092 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8099010 0x7f26e809bf20 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f26d40097a0 tx=0x7f26d402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fecf2aa5aab91d7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:21.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 1 -- 172.21.15.50:0/1785048092 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4031710 con 0x7f26e8099010 2023-10-30T12:05:21.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:21.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:05:21.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: dump: 2023-10-30T12:05:21.587 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:21.587 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:21.587 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.588 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.588 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:21.588 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:21.588 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:21.588 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:21.589 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:21.589 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:21.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: _finish_auth 0 2023-10-30T12:05:21.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:05:21.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:21.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 1 -- 172.21.15.50:0/1785048092 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d402f050 con 0x7f26e8099010 2023-10-30T12:05:21.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 5 monclient: authenticate success, global_id 44696 2023-10-30T12:05:21.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:21.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:21.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:21.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 1 -- 172.21.15.50:0/1785048092 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4031a90 con 0x7f26e8099010 2023-10-30T12:05:21.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:21.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 4 set_mon_vals no callback set 2023-10-30T12:05:21.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:21.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26e77fe640 10 monclient: dump: 2023-10-30T12:05:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:21.593 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:21.593 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:21.593 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:21.593 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:21.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 10 monclient: shutdown 2023-10-30T12:05:21.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:21.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 -- 172.21.15.50:0/1785048092 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8099010 msgr2=0x7f26e809bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:21.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 --2- 172.21.15.50:0/1785048092 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8099010 0x7f26e809bf20 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f26d40097a0 tx=0x7f26d402b7b0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 -- 172.21.15.50:0/1785048092 shutdown_connections 2023-10-30T12:05:21.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 --2- 172.21.15.50:0/1785048092 >> v2:172.21.15.118:3300/0 conn(0x7f26e81379f0 0x7f26e8139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 --2- 172.21.15.50:0/1785048092 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8099010 0x7f26e809bf20 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 -- 172.21.15.50:0/1785048092 >> 172.21.15.50:0/1785048092 conn(0x7f26e80a05c0 msgr2=0x7f26e809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:21.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 -- 172.21.15.50:0/1785048092 shutdown_connections 2023-10-30T12:05:21.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.570+0000 7f26eec1f640 1 -- 172.21.15.50:0/1785048092 wait complete. 2023-10-30T12:05:21.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 10 monclient: build_initial_monmap 2023-10-30T12:05:21.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 10 monclient: monmap: 2023-10-30T12:05:21.596 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:21.596 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:21.596 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:21.596 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:21.596 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:21.597 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:21.597 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:21.597 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:21.597 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:21.597 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:21.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 1 librados: starting msgr at 2023-10-30T12:05:21.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 1 librados: starting objecter 2023-10-30T12:05:21.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 1 Processor -- start 2023-10-30T12:05:21.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 1 -- start start 2023-10-30T12:05:21.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 1 librados: setting wanted keys 2023-10-30T12:05:21.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 1 librados: calling monclient init 2023-10-30T12:05:21.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.571+0000 7f26eec1f640 10 monclient: init 2023-10-30T12:05:21.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:21.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:05:21.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8098520 0x7f26e80713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:21.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): picked mon.noname-c con 0x7f26e8098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:21.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26e8099010 0x7f26e808afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:21.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): picked mon.noname-a con 0x7f26e8099010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:21.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26e81379f0 0x7f26e808b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:21.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): picked mon.noname-b con 0x7f26e81379f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:21.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:21.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80a32c0 con 0x7f26e8099010 2023-10-30T12:05:21.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:21.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e808e740 con 0x7f26e81379f0 2023-10-30T12:05:21.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:21.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8071df0 con 0x7f26e8098520 2023-10-30T12:05:21.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:21.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26eec1f640 10 monclient(hunting): authenticate will time out at 1166.755246s 2023-10-30T12:05:21.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8098520 0x7f26e80713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:21.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8098520 0x7f26e80713e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47118/0 (socket says 172.21.15.50:47118) 2023-10-30T12:05:21.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26e8099010 0x7f26e808afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:21.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 -- 172.21.15.50:0/2433605991 learned_addr learned my addr 172.21.15.50:0/2433605991 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:21.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): get_auth_request con 0x7f26e8098520 auth_method 0 2023-10-30T12:05:21.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:21.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:21.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:21.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 10 monclient(hunting): get_auth_request con 0x7f26e8099010 auth_method 0 2023-10-30T12:05:21.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:21.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:21.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:21.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ed1bc640 1 -- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26e81379f0 msgr2=0x7f26e808b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:21.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ed1bc640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26e81379f0 0x7f26e808b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:21.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:21.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:21.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:21.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:21.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:21.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:21.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient(hunting): handle_auth_done global_id 44699 payload 824 2023-10-30T12:05:21.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 -- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26e81379f0 msgr2=0x7f26e808b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:05:21.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26e81379f0 0x7f26e808b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 -- 172.21.15.50:0/2433605991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26e8099010 msgr2=0x7f26e808afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:21.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26e8099010 0x7f26e808afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient: _finish_hunting 0 2023-10-30T12:05:21.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 monclient: found mon.noname-c 2023-10-30T12:05:21.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:21.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:05:21.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 -- 172.21.15.50:0/2433605991 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d4009400 con 0x7f26e8098520 2023-10-30T12:05:21.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26ec9bb640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8098520 0x7f26e80713e0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0010e0 tx=0x7f26dc00c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:21.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e57fa640 10 client.?.objecter ms_handle_connect 0x7f26e8098520 2023-10-30T12:05:21.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e57fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:21.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.572+0000 7f26e57fa640 1 -- 172.21.15.50:0/2433605991 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc007480 con 0x7f26e8098520 2023-10-30T12:05:21.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:21.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:21.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: dump: 2023-10-30T12:05:21.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:21.610 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:21.610 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.610 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.610 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:21.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:21.611 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:21.611 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:21.611 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:21.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:21.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: _finish_auth 0 2023-10-30T12:05:21.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: _check_auth_tickets 2023-10-30T12:05:21.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:21.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 1 -- 172.21.15.50:0/2433605991 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc0075e0 con 0x7f26e8098520 2023-10-30T12:05:21.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 5 monclient: authenticate success, global_id 44699 2023-10-30T12:05:21.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 10 monclient: _renew_subs 2023-10-30T12:05:21.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e808e150 con 0x7f26e8098520 2023-10-30T12:05:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 1 -- 172.21.15.50:0/2433605991 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0059c0 con 0x7f26e8098520 2023-10-30T12:05:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 10 monclient: dump: 2023-10-30T12:05:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:21.615 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:21.615 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:21.615 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:21.615 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:21.615 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:21.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e6ffd640 4 set_mon_vals no callback set 2023-10-30T12:05:21.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 10 client.44699.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:21.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 10 monclient: _renew_subs 2023-10-30T12:05:21.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:21.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e8096e30 con 0x7f26e8098520 2023-10-30T12:05:21.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26eec1f640 1 librados: init done 2023-10-30T12:05:21.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.573+0000 7f26e57fa640 1 -- 172.21.15.50:0/2433605991 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f26dc005b20 con 0x7f26e8098520 2023-10-30T12:05:21.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e57fa640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f26cc0392a0 0x7f26cc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:21.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e57fa640 1 -- 172.21.15.50:0/2433605991 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f26dc04a1f0 con 0x7f26e8098520 2023-10-30T12:05:21.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e57fa640 10 client.44699.objecter ms_dispatch 0x7f26e8000b90 osd_map(139..139 src has 1..139) v4 2023-10-30T12:05:21.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e57fa640 3 client.44699.objecter handle_osd_map got epochs [139,139] > 0 2023-10-30T12:05:21.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e57fa640 3 client.44699.objecter handle_osd_map decoding full epoch 139 2023-10-30T12:05:21.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26eec1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:21.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26eec1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:21.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26eec1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:21.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e804e160 con 0x7f26e8098520 2023-10-30T12:05:21.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e57fa640 20 client.44699.objecter dump_active .. 0 homeless 2023-10-30T12:05:21.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e7fff640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f26cc0392a0 0x7f26cc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:21.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.574+0000 7f26e7fff640 10 monclient: get_auth_request con 0x7f26cc0392a0 auth_method 0 2023-10-30T12:05:21.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.575+0000 7f26e57fa640 10 client.44699.objecter ms_handle_connect 0x7f26cc0392a0 2023-10-30T12:05:21.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.575+0000 7f26e7fff640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f26cc0392a0 0x7f26cc03b760 secure :-1 s=READY pgs=3945 cs=0 l=1 rev1=1 crypto rx=0x7f26d4035000 tx=0x7f26d4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:21.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.579+0000 7f26e57fa640 1 -- 172.21.15.50:0/2433605991 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f26dc019b40 con 0x7f26e8098520 2023-10-30T12:05:21.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.579+0000 7f26e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:21.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.579+0000 7f26e57fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.825+0000 7f26eec1f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.825+0000 7f26eec1f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.825+0000 7f26eec1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.825+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f26e80745c0 con 0x7f26e8098520 2023-10-30T12:05:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.825+0000 7f26e57fa640 1 -- 172.21.15.50:0/2433605991 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f26dc017070 con 0x7f26e8098520 2023-10-30T12:05:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.825+0000 7f26e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.825+0000 7f26e57fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:21.828 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 10 librados: watch_flush enter 2023-10-30T12:05:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 10 librados: watch_flush exit 2023-10-30T12:05:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 20 client.44699.objecter shutdown clearing up homeless session... 2023-10-30T12:05:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 10 client.44699.objecter successfully canceled tick 2023-10-30T12:05:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f26cc0392a0 msgr2=0x7f26cc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f26cc0392a0 0x7f26cc03b760 secure :-1 s=READY pgs=3945 cs=0 l=1 rev1=1 crypto rx=0x7f26d4035000 tx=0x7f26d4008030 comp rx=0 tx=0).stop 2023-10-30T12:05:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 10 monclient: shutdown 2023-10-30T12:05:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8098520 msgr2=0x7f26e80713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8098520 0x7f26e80713e0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0010e0 tx=0x7f26dc00c9b0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 shutdown_connections 2023-10-30T12:05:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f26cc0392a0 0x7f26cc03b760 unknown :-1 s=CLOSED pgs=3945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26e81379f0 0x7f26e808b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26e8099010 0x7f26e808afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 --2- 172.21.15.50:0/2433605991 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26e8098520 0x7f26e80713e0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 >> 172.21.15.50:0/2433605991 conn(0x7f26e80a05c0 msgr2=0x7f26e8138020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:21.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 shutdown_connections 2023-10-30T12:05:21.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 -- 172.21.15.50:0/2433605991 wait complete. 2023-10-30T12:05:21.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:21.827+0000 7f26eec1f640 1 librados: shutdown 2023-10-30T12:05:24.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:24.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.955+0000 7fa36d083640 10 monclient: get_monmap_and_config 2023-10-30T12:05:24.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.955+0000 7fa36d083640 10 monclient: build_initial_monmap 2023-10-30T12:05:24.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.955+0000 7fa36d083640 10 monclient: monmap: 2023-10-30T12:05:24.957 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:24.957 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:24.957 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:24.956120+0000 2023-10-30T12:05:24.957 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:24.956120+0000 2023-10-30T12:05:24.958 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:24.958 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:24.958 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:24.958 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:24.959 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:24.959 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:24.959 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:24.959 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.955+0000 7fa36d083640 1 Processor -- start 2023-10-30T12:05:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.955+0000 7fa36d083640 1 -- start start 2023-10-30T12:05:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.955+0000 7fa36d083640 10 monclient: init 2023-10-30T12:05:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:24.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T12:05:24.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3680a4b60 0x7fa3681326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:24.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient(hunting): picked mon.noname-b con 0x7fa3680a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:24.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa368099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:24.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa368132be0 0x7fa3680a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient(hunting): picked mon.noname-a con 0x7fa368132be0 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680a4f40 con 0x7fa368099a40 2023-10-30T12:05:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3680a3d70 con 0x7fa368132be0 2023-10-30T12:05:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:24.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680a4150 con 0x7fa3680a4b60 2023-10-30T12:05:24.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:24.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36d083640 10 monclient(hunting): authenticate will time out at 1170.139191s 2023-10-30T12:05:24.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3680a4b60 0x7fa3681326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:24.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa368132be0 0x7fa3680a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3680a4b60 0x7fa3681326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47120/0 (socket says 172.21.15.50:47120) 2023-10-30T12:05:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 1 -- 172.21.15.50:0/2383667897 learned_addr learned my addr 172.21.15.50:0/2383667897 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 10 monclient(hunting): get_auth_request con 0x7fa3680a4b60 auth_method 0 2023-10-30T12:05:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:24.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:24.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36659c640 1 -- 172.21.15.50:0/2383667897 >> v1:172.21.15.118:6789/0 conn(0x7fa368099a40 legacy=0x7fa3680a2350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:24.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): get_auth_request con 0x7fa368132be0 auth_method 0 2023-10-30T12:05:24.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:24.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient(hunting): handle_auth_done global_id 84189 payload 274 2023-10-30T12:05:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 1 -- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3680a4b60 msgr2=0x7fa3681326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 1 --2- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3680a4b60 0x7fa3681326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 1 -- 172.21.15.50:0/2383667897 >> v1:172.21.15.118:6789/0 conn(0x7fa368099a40 legacy=0x7fa3680a2350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:24.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient: _finish_hunting 0 2023-10-30T12:05:24.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 1 monclient: found mon.noname-a 2023-10-30T12:05:24.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:24.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:24.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa36759e640 1 -- 172.21.15.50:0/2383667897 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36809a440 con 0x7fa368132be0 2023-10-30T12:05:24.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.956+0000 7fa366d9d640 1 --2- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3680a4b60 0x7fa3681326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:24.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa36759e640 1 --2- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa368132be0 0x7fa3680a2a60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa3680a53f0 tx=0x7fa35802bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14326cfba48bd35 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:24.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 1 -- 172.21.15.50:0/2383667897 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35802ca90 con 0x7fa368132be0 2023-10-30T12:05:24.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:24.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:24.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: dump: 2023-10-30T12:05:24.977 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:24.977 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:24.977 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:24.977 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:24.977 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:24.978 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:24.978 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:24.978 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:24.978 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:24.978 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:24.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: _finish_auth 0 2023-10-30T12:05:24.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:05:24.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:24.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 1 -- 172.21.15.50:0/2383667897 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35802cbf0 con 0x7fa368132be0 2023-10-30T12:05:24.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa36d083640 5 monclient: authenticate success, global_id 84189 2023-10-30T12:05:24.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa36d083640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 1 -- 172.21.15.50:0/2383667897 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358034440 con 0x7fa368132be0 2023-10-30T12:05:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa36d083640 4 set_mon_vals no callback set 2023-10-30T12:05:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.957+0000 7fa365d9b640 10 monclient: dump: 2023-10-30T12:05:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:24.982 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 10 monclient: shutdown 2023-10-30T12:05:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 -- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa368132be0 msgr2=0x7fa3680a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 --2- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa368132be0 0x7fa3680a2a60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa3680a53f0 tx=0x7fa35802bce0 comp rx=0 tx=0).stop 2023-10-30T12:05:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 -- 172.21.15.50:0/2383667897 shutdown_connections 2023-10-30T12:05:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 --2- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa368132be0 0x7fa3680a2a60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:24.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 --2- 172.21.15.50:0/2383667897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3680a4b60 0x7fa3681326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:24.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 -- 172.21.15.50:0/2383667897 >> 172.21.15.50:0/2383667897 conn(0x7fa3680a8060 msgr2=0x7fa3680a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:24.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 -- 172.21.15.50:0/2383667897 shutdown_connections 2023-10-30T12:05:24.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 1 -- 172.21.15.50:0/2383667897 wait complete. 2023-10-30T12:05:24.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 10 monclient: build_initial_monmap 2023-10-30T12:05:24.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.958+0000 7fa36d083640 10 monclient: monmap: 2023-10-30T12:05:24.985 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:24.985 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:24.985 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:24.985 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:24.985 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:24.985 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:24.986 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:24.986 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:24.986 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:24.986 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:24.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 1 librados: starting msgr at 2023-10-30T12:05:24.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 1 librados: starting objecter 2023-10-30T12:05:24.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 1 Processor -- start 2023-10-30T12:05:24.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 1 -- start start 2023-10-30T12:05:24.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 1 librados: setting wanted keys 2023-10-30T12:05:24.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 1 librados: calling monclient init 2023-10-30T12:05:24.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 10 monclient: init 2023-10-30T12:05:24.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:24.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.959+0000 7fa36d083640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:05:24.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3680a4b60 0x7fa36811f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:24.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): picked mon.noname-b con 0x7fa3680a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:24.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa368132be0 0x7fa36811f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:24.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): picked mon.noname-c con 0x7fa368132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:24.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa36811fa90 0x7fa3681ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:24.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): picked mon.noname-a con 0x7fa36811fa90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:24.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:24.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa368120ef0 con 0x7fa36811fa90 2023-10-30T12:05:24.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:24.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680a18d0 con 0x7fa3680a4b60 2023-10-30T12:05:24.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:24.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa368135c10 con 0x7fa368132be0 2023-10-30T12:05:24.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:24.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient(hunting): authenticate will time out at 1170.143089s 2023-10-30T12:05:24.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa368132be0 0x7fa36811f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:24.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa36811fa90 0x7fa3681ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:24.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa366d9d640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3680a4b60 msgr2=0x7fa36811f010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:24.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa366d9d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3680a4b60 0x7fa36811f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:24.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa36811fa90 0x7fa3681ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53862/0 (socket says 172.21.15.50:53862) 2023-10-30T12:05:24.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 -- 172.21.15.50:0/111296158 learned_addr learned my addr 172.21.15.50:0/111296158 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:24.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 10 monclient(hunting): get_auth_request con 0x7fa368132be0 auth_method 0 2023-10-30T12:05:24.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:24.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:24.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:24.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): get_auth_request con 0x7fa36811fa90 auth_method 0 2023-10-30T12:05:24.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:24.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:24.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:24.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:24.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:24.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:24.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:24.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:24.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:24.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient(hunting): handle_auth_done global_id 84192 payload 824 2023-10-30T12:05:24.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 -- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa368132be0 msgr2=0x7fa36811f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:24.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa368132be0 0x7fa36811f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:24.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 -- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3680a4b60 msgr2=0x7fa36811f010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:05:24.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3680a4b60 0x7fa36811f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:24.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient: _finish_hunting 0 2023-10-30T12:05:24.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 monclient: found mon.noname-a 2023-10-30T12:05:24.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:24.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:24.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 -- 172.21.15.50:0/111296158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600094d0 con 0x7fa36811fa90 2023-10-30T12:05:24.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36659c640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa368132be0 0x7fa36811f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:24.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36759e640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa36811fa90 0x7fa3681ca6c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa358037040 tx=0x7fa358004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:24.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 client.?.objecter ms_handle_connect 0x7fa36811fa90 2023-10-30T12:05:24.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:24.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 1 -- 172.21.15.50:0/111296158 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358040650 con 0x7fa36811fa90 2023-10-30T12:05:24.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:24.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:24.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: dump: 2023-10-30T12:05:24.998 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:24.998 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:24.998 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:24.998 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:24.998 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:24.999 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:24.999 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:24.999 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:24.999 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:24.999 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:24.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:05:25.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:05:25.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:25.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 1 -- 172.21.15.50:0/111296158 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3580407b0 con 0x7fa36811fa90 2023-10-30T12:05:25.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 5 monclient: authenticate success, global_id 84192 2023-10-30T12:05:25.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient: _renew_subs 2023-10-30T12:05:25.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:25.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa358009a70 con 0x7fa36811fa90 2023-10-30T12:05:25.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:25.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 1 -- 172.21.15.50:0/111296158 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358040b40 con 0x7fa36811fa90 2023-10-30T12:05:25.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:25.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:25.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa34f7fe640 10 monclient: dump: 2023-10-30T12:05:25.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:25.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:25.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:25.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:25.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:25.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:25.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:25.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:25.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:25.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:25.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.961+0000 7fa36d083640 10 client.84192.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:25.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.961+0000 7fa36d083640 10 monclient: _renew_subs 2023-10-30T12:05:25.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.961+0000 7fa36d083640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:25.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.961+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa368121d70 con 0x7fa36811fa90 2023-10-30T12:05:25.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.961+0000 7fa36d083640 1 librados: init done 2023-10-30T12:05:25.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.960+0000 7fa36559a640 4 set_mon_vals no callback set 2023-10-30T12:05:25.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.961+0000 7fa34f7fe640 1 -- 172.21.15.50:0/111296158 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa358040ca0 con 0x7fa36811fa90 2023-10-30T12:05:25.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa36d083640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:25.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa36d083640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:25.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa36d083640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:25.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa34f7fe640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3540392a0 0x7fa35403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:25.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36804e160 con 0x7fa36811fa90 2023-10-30T12:05:25.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa34f7fe640 1 -- 172.21.15.50:0/111296158 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa35806d900 con 0x7fa36811fa90 2023-10-30T12:05:25.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa34f7fe640 10 client.84192.objecter ms_dispatch 0x7fa368000b90 osd_map(139..139 src has 1..139) v4 2023-10-30T12:05:25.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa34f7fe640 3 client.84192.objecter handle_osd_map got epochs [139,139] > 0 2023-10-30T12:05:25.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa34f7fe640 3 client.84192.objecter handle_osd_map decoding full epoch 139 2023-10-30T12:05:25.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa34f7fe640 20 client.84192.objecter dump_active .. 0 homeless 2023-10-30T12:05:25.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa366d9d640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3540392a0 0x7fa35403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:25.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa366d9d640 10 monclient: get_auth_request con 0x7fa3540392a0 auth_method 0 2023-10-30T12:05:25.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa366d9d640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3540392a0 0x7fa35403b760 secure :-1 s=READY pgs=3946 cs=0 l=1 rev1=1 crypto rx=0x7fa360006c20 tx=0x7fa360009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:25.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.962+0000 7fa34f7fe640 10 client.84192.objecter ms_handle_connect 0x7fa3540392a0 2023-10-30T12:05:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.967+0000 7fa34f7fe640 1 -- 172.21.15.50:0/111296158 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa358044020 con 0x7fa36811fa90 2023-10-30T12:05:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.967+0000 7fa34f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:24.967+0000 7fa34f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.210+0000 7fa36d083640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.210+0000 7fa36d083640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.210+0000 7fa36d083640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.210+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa368073620 con 0x7fa36811fa90 2023-10-30T12:05:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.210+0000 7fa34f7fe640 1 -- 172.21.15.50:0/111296158 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fa358039070 con 0x7fa36811fa90 2023-10-30T12:05:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.210+0000 7fa34f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.210+0000 7fa34f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:25.213 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 10 librados: watch_flush enter 2023-10-30T12:05:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 10 librados: watch_flush exit 2023-10-30T12:05:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 20 client.84192.objecter shutdown clearing up homeless session... 2023-10-30T12:05:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 10 client.84192.objecter successfully canceled tick 2023-10-30T12:05:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3540392a0 msgr2=0x7fa35403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3540392a0 0x7fa35403b760 secure :-1 s=READY pgs=3946 cs=0 l=1 rev1=1 crypto rx=0x7fa360006c20 tx=0x7fa360009240 comp rx=0 tx=0).stop 2023-10-30T12:05:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 10 monclient: shutdown 2023-10-30T12:05:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa36811fa90 msgr2=0x7fa3681ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa36811fa90 0x7fa3681ca6c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa358037040 tx=0x7fa358004a40 comp rx=0 tx=0).stop 2023-10-30T12:05:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 shutdown_connections 2023-10-30T12:05:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3540392a0 0x7fa35403b760 unknown :-1 s=CLOSED pgs=3946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa36811fa90 0x7fa3681ca6c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa368132be0 0x7fa36811f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 --2- 172.21.15.50:0/111296158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3680a4b60 0x7fa36811f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 >> 172.21.15.50:0/111296158 conn(0x7fa3680a8060 msgr2=0x7fa3681341d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 shutdown_connections 2023-10-30T12:05:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 -- 172.21.15.50:0/111296158 wait complete. 2023-10-30T12:05:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:25.212+0000 7fa36d083640 1 librados: shutdown 2023-10-30T12:05:28.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:28.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient: get_monmap_and_config 2023-10-30T12:05:28.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient: build_initial_monmap 2023-10-30T12:05:28.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient: monmap: 2023-10-30T12:05:28.343 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:28.343 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:28.343 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:28.341669+0000 2023-10-30T12:05:28.343 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:28.341669+0000 2023-10-30T12:05:28.343 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:28.345 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:28.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 1 Processor -- start 2023-10-30T12:05:28.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 1 -- start start 2023-10-30T12:05:28.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient: init 2023-10-30T12:05:28.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:28.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:05:28.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 0x7f8f10132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:28.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient(hunting): picked mon.noname-c con 0x7f8f10131b20 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f10071bc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient(hunting): picked mon.noname-a con 0x7f8f10071fc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1009a9b0 con 0x7f8f10071bc0 2023-10-30T12:05:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f10131f00 con 0x7f8f10071fc0 2023-10-30T12:05:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1009d920 con 0x7f8f10131b20 2023-10-30T12:05:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f16e09640 10 monclient(hunting): authenticate will time out at 1173.524851s 2023-10-30T12:05:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f153a6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.341+0000 7f8f14ba5640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 msgr2=0x7f8f10132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f0ffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f10071bc0 0x7f8f10097410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46404/0 (socket says 172.21.15.50:46404) 2023-10-30T12:05:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56364/0 (socket says 172.21.15.50:56364) 2023-10-30T12:05:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 learned_addr learned my addr 172.21.15.50:0/812495756 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 10 monclient(hunting): get_auth_request con 0x7f8f10071fc0 auth_method 0 2023-10-30T12:05:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f14ba5640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 0x7f8f10132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:05:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.342+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 msgr2=0x7f8f10132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f14ba5640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 0x7f8f10132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:05:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 10 monclient(hunting): get_auth_request con 0x7f8f10071fc0 auth_method 0 2023-10-30T12:05:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:05:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.543+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:05:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 msgr2=0x7f8f10132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f14ba5640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 0x7f8f10132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:05:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 10 monclient(hunting): get_auth_request con 0x7f8f10071fc0 auth_method 0 2023-10-30T12:05:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:05:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:28.944+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:05:29.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 msgr2=0x7f8f10132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f14ba5640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 0x7f8f10132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:05:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 10 monclient(hunting): get_auth_request con 0x7f8f10071fc0 auth_method 0 2023-10-30T12:05:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:05:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:29.745+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:05:31.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.341+0000 7f8f0e7fc640 10 monclient(hunting): tick 2023-10-30T12:05:31.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.341+0000 7f8f0e7fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:05:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.341+0000 7f8f0e7fc640 1 monclient(hunting): continuing hunt 2023-10-30T12:05:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.341+0000 7f8f0e7fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:05:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.341+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 msgr2=0x7f8f10132270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8f10131b20 0x7f8f10132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 msgr2=0x7f8f10097b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.50:3300/0 conn(0x7f8f10071fc0 0x7f8f10097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.50:6789/0 conn(0x7f8f10071bc0 legacy=0x7f8f10097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 10 monclient: _add_conns ranks=[1,2] 2023-10-30T12:05:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ef4003690 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef4007380 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1009a0b0 con 0x7f8ef4003690 2023-10-30T12:05:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10131f00 con 0x7f8ef4007380 2023-10-30T12:05:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0e7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.118:6789/0 conn(0x7f8ef4003690 legacy=0x7f8ef4005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request con 0x7f8ef4007380 auth_method 0 2023-10-30T12:05:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:05:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.342+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:31.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.118:6789/0 conn(0x7f8ef4003690 legacy=0x7f8ef4005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:31.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request con 0x7f8ef4007380 auth_method 0 2023-10-30T12:05:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:05:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.543+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:05:31.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:31.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.118:6789/0 conn(0x7f8ef4003690 legacy=0x7f8ef4005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:31.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request con 0x7f8ef4007380 auth_method 0 2023-10-30T12:05:31.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:31.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:31.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:31.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:05:31.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:31.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:31.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:31.944+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:05:32.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.118:6789/0 conn(0x7f8ef4003690 legacy=0x7f8ef4005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:32.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:32.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request con 0x7f8ef4007380 auth_method 0 2023-10-30T12:05:32.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:32.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:32.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:05:32.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:05:32.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:05:32.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:32.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:32.745+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:05:34.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): tick 2023-10-30T12:05:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:05:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 monclient(hunting): continuing hunt 2023-10-30T12:05:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:05:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 msgr2=0x7f8ef4009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4007380 0x7f8ef4009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.118:6789/0 conn(0x7f8ef4003690 legacy=0x7f8ef4005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:05:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 reap_dead start 2023-10-30T12:05:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ef4009f70 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:34.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef400ab00 0x7f8ef4009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef400ab00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:34.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8ef4005c00 0x7f8ef400d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef4005c00 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:34.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef400b050 con 0x7f8ef4009f70 2023-10-30T12:05:34.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f10131f00 con 0x7f8ef4005c00 2023-10-30T12:05:34.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 1 -- 172.21.15.50:0/812495756 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1009d920 con 0x7f8ef400ab00 2023-10-30T12:05:34.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0e7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:34.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f14ba5640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.118:6789/0 conn(0x7f8ef4009f70 legacy=0x7f8ef400a350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:34.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f153a6640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8ef4005c00 msgr2=0x7f8ef400d450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:34.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f153a6640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8ef4005c00 0x7f8ef400d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:34.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef400ab00 0x7f8ef4009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request con 0x7f8ef400ab00 auth_method 0 2023-10-30T12:05:34.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.342+0000 7f8f0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_done global_id 44708 payload 274 2023-10-30T12:05:34.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8ef4005c00 msgr2=0x7f8ef400d450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:34.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8ef4005c00 0x7f8ef400d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 >> v1:172.21.15.118:6789/0 conn(0x7f8ef4009f70 legacy=0x7f8ef400a350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:34.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:34.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 1 monclient: found mon.noname-b 2023-10-30T12:05:34.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:34.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:05:34.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.343+0000 7f8f0ffff640 1 -- 172.21.15.50:0/812495756 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1009a0b0 con 0x7f8ef400ab00 2023-10-30T12:05:34.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef400ab00 0x7f8ef4009820 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f8efc007d20 tx=0x7f8efc02c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed8a56a6f4584711 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 1 -- 172.21.15.50:0/812495756 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc036400 con 0x7f8ef400ab00 2023-10-30T12:05:34.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:05:34.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: dump: 2023-10-30T12:05:34.355 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.356 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.356 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.356 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.356 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.356 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.357 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.357 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.357 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.357 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:05:34.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:05:34.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:34.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 1 -- 172.21.15.50:0/812495756 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc030070 con 0x7f8ef400ab00 2023-10-30T12:05:34.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f16e09640 5 monclient: authenticate success, global_id 44708 2023-10-30T12:05:34.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f16e09640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:34.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:34.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f16e09640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:34.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f16e09640 4 set_mon_vals no callback set 2023-10-30T12:05:34.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 1 -- 172.21.15.50:0/812495756 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0367c0 con 0x7f8ef400ab00 2023-10-30T12:05:34.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:34.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.344+0000 7f8f0f7fe640 10 monclient: dump: 2023-10-30T12:05:34.360 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.360 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.361 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.361 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.361 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.361 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.361 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.362 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.362 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.362 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 10 monclient: shutdown 2023-10-30T12:05:34.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:34.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 -- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef400ab00 msgr2=0x7f8ef4009820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef400ab00 0x7f8ef4009820 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f8efc007d20 tx=0x7f8efc02c310 comp rx=0 tx=0).stop 2023-10-30T12:05:34.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 -- 172.21.15.50:0/812495756 shutdown_connections 2023-10-30T12:05:34.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 --2- 172.21.15.50:0/812495756 >> v2:172.21.15.118:3300/0 conn(0x7f8ef4005c00 0x7f8ef400d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 --2- 172.21.15.50:0/812495756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef400ab00 0x7f8ef4009820 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 -- 172.21.15.50:0/812495756 >> 172.21.15.50:0/812495756 conn(0x7f8f100a05c0 msgr2=0x7f8f1009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:34.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 -- 172.21.15.50:0/812495756 shutdown_connections 2023-10-30T12:05:34.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.345+0000 7f8f16e09640 1 -- 172.21.15.50:0/812495756 wait complete. 2023-10-30T12:05:34.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 10 monclient: build_initial_monmap 2023-10-30T12:05:34.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 10 monclient: monmap: 2023-10-30T12:05:34.369 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:34.369 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:34.370 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:34.370 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:34.370 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:34.370 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.370 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:34.371 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:34.371 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:34.371 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 1 librados: starting msgr at 2023-10-30T12:05:34.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 1 librados: starting objecter 2023-10-30T12:05:34.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 1 Processor -- start 2023-10-30T12:05:34.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 1 -- start start 2023-10-30T12:05:34.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 1 librados: setting wanted keys 2023-10-30T12:05:34.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 1 librados: calling monclient init 2023-10-30T12:05:34.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.346+0000 7f8f16e09640 10 monclient: init 2023-10-30T12:05:34.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:34.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:05:34.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ef4005c00 0x7f8f101ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef4005c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:34.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4009f70 0x7f8f101ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef4009f70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:34.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ef400ab00 0x7f8f1012c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef400ab00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:34.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100993c0 con 0x7f8ef400ab00 2023-10-30T12:05:34.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1009c6a0 con 0x7f8ef4005c00 2023-10-30T12:05:34.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1008b630 con 0x7f8ef4009f70 2023-10-30T12:05:34.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:34.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f16e09640 10 monclient(hunting): authenticate will time out at 1179.530537s 2023-10-30T12:05:34.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f14ba5640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ef4005c00 msgr2=0x7f8f101ca320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:34.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4009f70 0x7f8f101ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ef400ab00 0x7f8f1012c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f14ba5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ef4005c00 0x7f8f101ca320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:34.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4009f70 0x7f8f101ca860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51982/0 (socket says 172.21.15.50:51982) 2023-10-30T12:05:34.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 1 -- 172.21.15.50:0/1490416376 learned_addr learned my addr 172.21.15.50:0/1490416376 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:34.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request con 0x7f8ef4009f70 auth_method 0 2023-10-30T12:05:34.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 10 monclient(hunting): get_auth_request con 0x7f8ef400ab00 auth_method 0 2023-10-30T12:05:34.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.347+0000 7f8f153a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 10 monclient(hunting): handle_auth_done global_id 44711 payload 824 2023-10-30T12:05:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 1 -- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ef4005c00 msgr2=0x7f8f101ca320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:05:34.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ef4005c00 0x7f8f101ca320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 1 -- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ef400ab00 msgr2=0x7f8f1012c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ef400ab00 0x7f8f1012c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 1 monclient: found mon.noname-c 2023-10-30T12:05:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:34.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:05:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 1 -- 172.21.15.50:0/1490416376 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc00f040 con 0x7f8ef4009f70 2023-10-30T12:05:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f153a6640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ef400ab00 0x7f8f1012c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0ffff640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4009f70 0x7f8f101ca860 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f8efc02c350 tx=0x7f8efc001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 client.?.objecter ms_handle_connect 0x7f8ef4009f70 2023-10-30T12:05:34.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 1 -- 172.21.15.50:0/1490416376 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc041470 con 0x7f8ef4009f70 2023-10-30T12:05:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:05:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: dump: 2023-10-30T12:05:34.382 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.383 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.384 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:05:34.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:05:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 5 monclient: authenticate success, global_id 44711 2023-10-30T12:05:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 10 monclient: _renew_subs 2023-10-30T12:05:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 1 -- 172.21.15.50:0/1490416376 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc0415d0 con 0x7f8ef4009f70 2023-10-30T12:05:34.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:34.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1012e6b0 con 0x7f8ef4009f70 2023-10-30T12:05:34.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:34.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 1 -- 172.21.15.50:0/1490416376 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc041980 con 0x7f8ef4009f70 2023-10-30T12:05:34.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:34.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0d7fa640 10 monclient: dump: 2023-10-30T12:05:34.387 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.387 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.387 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.387 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.388 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.388 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.388 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.388 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.388 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.389 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f0e7fc640 4 set_mon_vals no callback set 2023-10-30T12:05:34.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 10 client.44711.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:34.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 10 monclient: _renew_subs 2023-10-30T12:05:34.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:34.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f101cada0 con 0x7f8ef4009f70 2023-10-30T12:05:34.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.348+0000 7f8f16e09640 1 librados: init done 2023-10-30T12:05:34.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.349+0000 7f8f0d7fa640 1 -- 172.21.15.50:0/1490416376 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8efc041ae0 con 0x7f8ef4009f70 2023-10-30T12:05:34.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.349+0000 7f8f0d7fa640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef403b000 0x7f8ef403d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f16e09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:34.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f16e09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:34.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f16e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:34.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f0d7fa640 1 -- 172.21.15.50:0/1490416376 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 7202+0+0 (secure 0 0 0) 0x7f8efc06e1c0 con 0x7f8ef4009f70 2023-10-30T12:05:34.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f0d7fa640 10 client.44711.objecter ms_dispatch 0x7f8f10000b90 osd_map(141..141 src has 1..141) v4 2023-10-30T12:05:34.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f0d7fa640 3 client.44711.objecter handle_osd_map got epochs [141,141] > 0 2023-10-30T12:05:34.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f0d7fa640 3 client.44711.objecter handle_osd_map decoding full epoch 141 2023-10-30T12:05:34.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f101caf30 con 0x7f8ef4009f70 2023-10-30T12:05:34.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f0d7fa640 20 client.44711.objecter dump_active .. 0 homeless 2023-10-30T12:05:34.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f14ba5640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef403b000 0x7f8ef403d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f14ba5640 10 monclient: get_auth_request con 0x7f8ef403b000 auth_method 0 2023-10-30T12:05:34.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f0d7fa640 10 client.44711.objecter ms_handle_connect 0x7f8ef403b000 2023-10-30T12:05:34.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.350+0000 7f8f14ba5640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef403b000 0x7f8ef403d4c0 secure :-1 s=READY pgs=3947 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40072f0 tx=0x7f8f040090f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.355+0000 7f8f0d7fa640 1 -- 172.21.15.50:0/1490416376 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8efc045070 con 0x7f8ef4009f70 2023-10-30T12:05:34.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.355+0000 7f8f0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:34.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.355+0000 7f8f0d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:34.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.601+0000 7f8f16e09640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:34.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.601+0000 7f8f16e09640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:34.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.601+0000 7f8f16e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:05:34.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.601+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8f100988b0 con 0x7f8ef4009f70 2023-10-30T12:05:34.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.601+0000 7f8f0d7fa640 1 -- 172.21.15.50:0/1490416376 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f8efc058620 con 0x7f8ef4009f70 2023-10-30T12:05:34.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.601+0000 7f8f0d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:34.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.601+0000 7f8f0d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:34.604 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:34.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 10 librados: watch_flush enter 2023-10-30T12:05:34.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 10 librados: watch_flush exit 2023-10-30T12:05:34.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 20 client.44711.objecter shutdown clearing up homeless session... 2023-10-30T12:05:34.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 10 client.44711.objecter successfully canceled tick 2023-10-30T12:05:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef403b000 msgr2=0x7f8ef403d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef403b000 0x7f8ef403d4c0 secure :-1 s=READY pgs=3947 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40072f0 tx=0x7f8f040090f0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 10 monclient: shutdown 2023-10-30T12:05:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4009f70 msgr2=0x7f8f101ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4009f70 0x7f8f101ca860 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f8efc02c350 tx=0x7f8efc001120 comp rx=0 tx=0).stop 2023-10-30T12:05:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 shutdown_connections 2023-10-30T12:05:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ef403b000 0x7f8ef403d4c0 unknown :-1 s=CLOSED pgs=3947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ef400ab00 0x7f8f1012c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ef4009f70 0x7f8f101ca860 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 --2- 172.21.15.50:0/1490416376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ef4005c00 0x7f8f101ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 >> 172.21.15.50:0/1490416376 conn(0x7f8f100a05c0 msgr2=0x7f8f1006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:34.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 shutdown_connections 2023-10-30T12:05:34.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 -- 172.21.15.50:0/1490416376 wait complete. 2023-10-30T12:05:34.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.603+0000 7f8f16e09640 1 librados: shutdown 2023-10-30T12:05:34.615 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:05:34.615 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:05:34.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:05:34.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.720+0000 7f85c475e640 10 monclient: get_monmap_and_config 2023-10-30T12:05:34.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.720+0000 7f85c475e640 10 monclient: build_initial_monmap 2023-10-30T12:05:34.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.720+0000 7f85c475e640 10 monclient: monmap: 2023-10-30T12:05:34.722 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:34.721297+0000 2023-10-30T12:05:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:34.721297+0000 2023-10-30T12:05:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:34.725 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.720+0000 7f85c475e640 1 Processor -- start 2023-10-30T12:05:34.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.720+0000 7f85c475e640 1 -- start start 2023-10-30T12:05:34.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.720+0000 7f85c475e640 10 monclient: init 2023-10-30T12:05:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:05:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85bc0a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85bc0a2450 0x7f85bc13aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient(hunting): picked mon.noname-a con 0x7f85bc0a2450 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f85bc13af40 0x7f85bc13d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient(hunting): picked mon.noname-c con 0x7f85bc13af40 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc0a4190 con 0x7f85bc0a2ae0 2023-10-30T12:05:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc0a1630 con 0x7f85bc0a2450 2023-10-30T12:05:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc09a2e0 con 0x7f85bc13af40 2023-10-30T12:05:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c475e640 10 monclient(hunting): authenticate will time out at 1179.904500s 2023-10-30T12:05:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85bc0a2450 0x7f85bc13aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85bc0a2450 0x7f85bc13aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56406/0 (socket says 172.21.15.50:56406) 2023-10-30T12:05:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c2cfb640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f85bc13af40 msgr2=0x7f85bc13d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 -- 172.21.15.50:0/4023019260 learned_addr learned my addr 172.21.15.50:0/4023019260 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): get_auth_request con 0x7f85bc0a2450 auth_method 0 2023-10-30T12:05:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c24fa640 1 --1- 172.21.15.50:0/4023019260 >> v1:172.21.15.50:6789/0 conn(0x7f85bc0a2ae0 0x7f85bc0a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46410/0 (socket says 172.21.15.50:46410) 2023-10-30T12:05:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c2cfb640 1 --2- 172.21.15.50:0/4023019260 >> v2:172.21.15.118:3300/0 conn(0x7f85bc13af40 0x7f85bc13d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_done global_id 84198 payload 274 2023-10-30T12:05:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 -- 172.21.15.50:0/4023019260 >> v2:172.21.15.118:3300/0 conn(0x7f85bc13af40 msgr2=0x7f85bc13d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 --2- 172.21.15.50:0/4023019260 >> v2:172.21.15.118:3300/0 conn(0x7f85bc13af40 0x7f85bc13d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 -- 172.21.15.50:0/4023019260 >> v1:172.21.15.50:6789/0 conn(0x7f85bc0a2ae0 legacy=0x7f85bc0a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient: _finish_hunting 0 2023-10-30T12:05:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 monclient: found mon.noname-a 2023-10-30T12:05:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c1cf9640 1 -- 172.21.15.50:0/4023019260 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc09a8a0 con 0x7f85bc0a2450 2023-10-30T12:05:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c14f8640 1 -- 172.21.15.50:0/4023019260 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274511579 0 0) 0x7f85b4001110 con 0x7f85bc0a2ae0 2023-10-30T12:05:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.721+0000 7f85c14f8640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:05:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c1cf9640 1 --2- 172.21.15.50:0/4023019260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc13aa00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0705b0 tx=0x7f85b002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba009b25139393ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 1 -- 172.21.15.50:0/4023019260 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0031800 con 0x7f85bc0a2450 2023-10-30T12:05:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: dump: 2023-10-30T12:05:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.745 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: _finish_auth 0 2023-10-30T12:05:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: _check_auth_tickets 2023-10-30T12:05:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 1 -- 172.21.15.50:0/4023019260 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b0031960 con 0x7f85bc0a2450 2023-10-30T12:05:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c475e640 5 monclient: authenticate success, global_id 84198 2023-10-30T12:05:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c475e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 1 -- 172.21.15.50:0/4023019260 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0031cf0 con 0x7f85bc0a2450 2023-10-30T12:05:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c14f8640 10 monclient: dump: 2023-10-30T12:05:34.747 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.748 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.749 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c475e640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.722+0000 7f85c475e640 4 set_mon_vals no callback set 2023-10-30T12:05:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 10 monclient: shutdown 2023-10-30T12:05:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 -- 172.21.15.50:0/4023019260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 msgr2=0x7f85bc13aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 --2- 172.21.15.50:0/4023019260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc13aa00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0705b0 tx=0x7f85b002b8a0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 -- 172.21.15.50:0/4023019260 shutdown_connections 2023-10-30T12:05:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 --2- 172.21.15.50:0/4023019260 >> v2:172.21.15.118:3300/0 conn(0x7f85bc13af40 0x7f85bc13d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 --2- 172.21.15.50:0/4023019260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc13aa00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 -- 172.21.15.50:0/4023019260 >> 172.21.15.50:0/4023019260 conn(0x7f85bc0a80b0 msgr2=0x7f85bc0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 -- 172.21.15.50:0/4023019260 shutdown_connections 2023-10-30T12:05:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.723+0000 7f85c475e640 1 -- 172.21.15.50:0/4023019260 wait complete. 2023-10-30T12:05:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient: build_initial_monmap 2023-10-30T12:05:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient: monmap: 2023-10-30T12:05:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:34.754 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 librados: starting msgr at 2023-10-30T12:05:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 librados: starting objecter 2023-10-30T12:05:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 Processor -- start 2023-10-30T12:05:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 -- start start 2023-10-30T12:05:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 librados: setting wanted keys 2023-10-30T12:05:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 librados: calling monclient init 2023-10-30T12:05:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient: init 2023-10-30T12:05:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:05:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): picked mon.noname-a con 0x7f85bc0a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f85bc0a2ae0 0x7f85bc132720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): picked mon.noname-c con 0x7f85bc0a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85bc13af40 0x7f85bc09d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): picked mon.noname-b con 0x7f85bc13af40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc13a810 con 0x7f85bc0a2450 2023-10-30T12:05:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc11e6b0 con 0x7f85bc13af40 2023-10-30T12:05:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc1cb180 con 0x7f85bc0a2ae0 2023-10-30T12:05:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c475e640 10 monclient(hunting): authenticate will time out at 1179.907920s 2023-10-30T12:05:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.724+0000 7f85c24fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f85bc0a2ae0 0x7f85bc132720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc1321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56410/0 (socket says 172.21.15.50:56410) 2023-10-30T12:05:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 -- 172.21.15.50:0/4174732767 learned_addr learned my addr 172.21.15.50:0/4174732767 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): get_auth_request con 0x7f85bc0a2450 auth_method 0 2023-10-30T12:05:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 10 monclient(hunting): get_auth_request con 0x7f85bc0a2ae0 auth_method 0 2023-10-30T12:05:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c2cfb640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85bc13af40 msgr2=0x7f85bc09d9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c2cfb640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85bc13af40 0x7f85bc09d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient(hunting): handle_auth_done global_id 84204 payload 824 2023-10-30T12:05:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f85bc0a2ae0 msgr2=0x7f85bc132720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f85bc0a2ae0 0x7f85bc132720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85bc13af40 msgr2=0x7f85bc09d9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:05:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85bc13af40 0x7f85bc09d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient: _finish_hunting 0 2023-10-30T12:05:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 monclient: found mon.noname-a 2023-10-30T12:05:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 -- 172.21.15.50:0/4174732767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b00094d0 con 0x7f85bc0a2450 2023-10-30T12:05:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c24fa640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc1321e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85b400c780 tx=0x7f85b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 client.?.objecter ms_handle_connect 0x7f85bc0a2450 2023-10-30T12:05:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40072d0 con 0x7f85bc0a2450 2023-10-30T12:05:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: dump: 2023-10-30T12:05:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.767 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: _finish_auth 0 2023-10-30T12:05:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: _check_auth_tickets 2023-10-30T12:05:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b4007450 con 0x7f85bc0a2450 2023-10-30T12:05:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 5 monclient: authenticate success, global_id 84204 2023-10-30T12:05:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40047e0 con 0x7f85bc0a2450 2023-10-30T12:05:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85baffd640 10 monclient: dump: 2023-10-30T12:05:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.771 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 10 monclient: _renew_subs 2023-10-30T12:05:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c0cf7640 4 set_mon_vals no callback set 2023-10-30T12:05:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc0a1790 con 0x7f85bc0a2450 2023-10-30T12:05:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 10 client.84204.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 10 monclient: _renew_subs 2023-10-30T12:05:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc09f5f0 con 0x7f85bc0a2450 2023-10-30T12:05:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.725+0000 7f85c475e640 1 librados: init done 2023-10-30T12:05:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c475e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f85b4004960 con 0x7f85bc0a2450 2023-10-30T12:05:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c475e640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc09fde0 0x7f85bc02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c475e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc073ff0 con 0x7f85bc09fde0 2023-10-30T12:05:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc09fde0 0x7f85bc02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient: get_auth_request con 0x7f85bc09fde0 auth_method 0 2023-10-30T12:05:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85baffd640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85a80392a0 0x7f85a803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 7202+0+0 (secure 0 0 0) 0x7f85b404a0c0 con 0x7f85bc0a2450 2023-10-30T12:05:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85baffd640 10 client.84204.objecter ms_dispatch 0x7f85bc000b90 osd_map(141..141 src has 1..141) v4 2023-10-30T12:05:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85baffd640 3 client.84204.objecter handle_osd_map got epochs [141,141] > 0 2023-10-30T12:05:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85baffd640 3 client.84204.objecter handle_osd_map decoding full epoch 141 2023-10-30T12:05:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_done global_id 84207 payload 824 2023-10-30T12:05:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85baffd640 20 client.84204.objecter dump_active .. 0 homeless 2023-10-30T12:05:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.726+0000 7f85c1cf9640 1 -- 172.21.15.50:0/4174732767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85bc09a5c0 con 0x7f85bc09fde0 2023-10-30T12:05:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85c2cfb640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85a80392a0 0x7f85a803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85c2cfb640 10 monclient: get_auth_request con 0x7f85a80392a0 auth_method 0 2023-10-30T12:05:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 client.84204.objecter ms_handle_connect 0x7f85bc09fde0 2023-10-30T12:05:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 client.84204.objecter resend_mon_ops 2023-10-30T12:05:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85c1cf9640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc09fde0 0x7f85bc02b430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0a3c80 tx=0x7f85b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0049070 con 0x7f85bc09fde0 2023-10-30T12:05:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 monclient: dump: 2023-10-30T12:05:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.780 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.781 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.781 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 client.84204.objecter ms_handle_connect 0x7f85a80392a0 2023-10-30T12:05:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85c2cfb640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85a80392a0 0x7f85a803b760 secure :-1 s=READY pgs=3948 cs=0 l=1 rev1=1 crypto rx=0x7f85bc11e500 tx=0x7f85ac009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f85b003ab70 con 0x7f85bc09fde0 2023-10-30T12:05:34.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc09fde0 msgr2=0x7f85bc02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc09fde0 0x7f85bc02b430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0a3c80 tx=0x7f85b0004730 comp rx=0 tx=0).stop 2023-10-30T12:05:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc09fde0 msgr2=0x7f85bc02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:34.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.727+0000 7f85baffd640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc09fde0 0x7f85bc02b430 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0a3c80 tx=0x7f85b0004730 comp rx=0 tx=0).stop 2023-10-30T12:05:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.733+0000 7f85c475e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:34.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.733+0000 7f85c475e640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc02b970 0x7f85bc02bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.733+0000 7f85c475e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.733+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc11e2f0 con 0x7f85bc02b970 2023-10-30T12:05:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.733+0000 7f85c1cf9640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc02b970 0x7f85bc02bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient: get_auth_request con 0x7f85bc02b970 auth_method 0 2023-10-30T12:05:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 10 monclient(hunting): handle_auth_done global_id 84210 payload 824 2023-10-30T12:05:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 1 -- 172.21.15.50:0/4174732767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f85bc0a28a0 con 0x7f85bc02b970 2023-10-30T12:05:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 10 client.84204.objecter ms_handle_connect 0x7f85bc02b970 2023-10-30T12:05:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 10 client.84204.objecter resend_mon_ops 2023-10-30T12:05:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c1cf9640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc02b970 0x7f85bc02bd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85b002b8e0 tx=0x7f85b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b0003e60 con 0x7f85bc02b970 2023-10-30T12:05:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 10 monclient: dump: 2023-10-30T12:05:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.789 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.791 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.791 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.791 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.791 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7f85b00443d0 con 0x7f85bc02b970 2023-10-30T12:05:34.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc02b970 msgr2=0x7f85bc02bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc02b970 0x7f85bc02bd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85b002b8e0 tx=0x7f85b00046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc02b970 msgr2=0x7f85bc02bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85baffd640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc02b970 0x7f85bc02bd50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.793 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:34.793 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:34.793 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:34.794 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:05:34.794 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 54, 2023-10-30T12:05:34.794 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:34.794 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:34.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:34.794 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:34.795 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:05:34.795 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:34.795 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:34.795 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:34.795 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:34.796 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:34.796 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:34.796 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:34.796 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:34.796 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:34.797 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:34.797 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:34.797 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:34.797 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:34.797 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:34.798 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:34.798 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:34.798 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:34.798 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:34.798 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:34.801 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:34.802 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:34.803 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:34.804 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:34.805 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:34.806 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:34.807 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:34.808 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.809 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:34.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.812 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.813 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:34.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:34.815 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:34.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:34.817 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:34.818 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:34.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:34.820 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.821 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:05:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:34.822 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:05:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:34.823 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:34.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:34.824 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:34.824 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:34.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 10 librados: watch_flush enter 2023-10-30T12:05:34.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 10 librados: watch_flush exit 2023-10-30T12:05:34.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 20 client.84204.objecter shutdown clearing up homeless session... 2023-10-30T12:05:34.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 10 client.84204.objecter successfully canceled tick 2023-10-30T12:05:34.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85a80392a0 msgr2=0x7f85a803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85a80392a0 0x7f85a803b760 secure :-1 s=READY pgs=3948 cs=0 l=1 rev1=1 crypto rx=0x7f85bc11e500 tx=0x7f85ac009270 comp rx=0 tx=0).stop 2023-10-30T12:05:34.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 10 monclient: shutdown 2023-10-30T12:05:34.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:34.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 msgr2=0x7f85bc1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.734+0000 7f85c475e640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc1321e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85b400c780 tx=0x7f85b400cc40 comp rx=0 tx=0).stop 2023-10-30T12:05:34.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.735+0000 7f85c2cfb640 1 -- 172.21.15.50:0/4174732767 reap_dead start 2023-10-30T12:05:34.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.735+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 shutdown_connections 2023-10-30T12:05:34.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.735+0000 7f85c475e640 1 --2- 172.21.15.50:0/4174732767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85bc0a2450 0x7f85bc1321e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.735+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 >> 172.21.15.50:0/4174732767 conn(0x7f85bc0a80b0 msgr2=0x7f85bc134da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:34.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.735+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 shutdown_connections 2023-10-30T12:05:34.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.735+0000 7f85c475e640 1 -- 172.21.15.50:0/4174732767 wait complete. 2023-10-30T12:05:34.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.735+0000 7f85c475e640 1 librados: shutdown 2023-10-30T12:05:34.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:05:34.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:05:34.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:05:34.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.934+0000 7f4678c56640 10 monclient: get_monmap_and_config 2023-10-30T12:05:34.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.934+0000 7f4678c56640 10 monclient: build_initial_monmap 2023-10-30T12:05:34.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.934+0000 7f4678c56640 10 monclient: monmap: 2023-10-30T12:05:34.937 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:34.937 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.937 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:34.935866+0000 2023-10-30T12:05:34.937 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:34.935866+0000 2023-10-30T12:05:34.937 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:34.938 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.938 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:34.938 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:34.938 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:34.938 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:34.939 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:34.939 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 1 Processor -- start 2023-10-30T12:05:34.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 1 -- start start 2023-10-30T12:05:34.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 10 monclient: init 2023-10-30T12:05:34.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:34.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T12:05:34.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467406afc0 0x7f46740a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 10 monclient(hunting): picked mon.noname-b con 0x7f467406afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:34.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f467406b640 0x7f46740a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.935+0000 7f4678c56640 10 monclient(hunting): picked mon.noname-a con 0x7f467406b640 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:34.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46740a2710 addr v1:172.21.15.118:6789/0 2023-10-30T12:05:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467406bde0 con 0x7f46740a2710 2023-10-30T12:05:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46740a2b40 con 0x7f467406b640 2023-10-30T12:05:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467406b3a0 con 0x7f467406afc0 2023-10-30T12:05:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4678c56640 10 monclient(hunting): authenticate will time out at 1180.119055s 2023-10-30T12:05:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467406afc0 0x7f46740a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f467406b640 0x7f46740a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467406afc0 0x7f46740a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51996/0 (socket says 172.21.15.50:51996) 2023-10-30T12:05:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 1 -- 172.21.15.50:0/448938546 learned_addr learned my addr 172.21.15.50:0/448938546 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 10 monclient(hunting): get_auth_request con 0x7f467406afc0 auth_method 0 2023-10-30T12:05:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f4672d9d640 1 -- 172.21.15.50:0/448938546 >> v1:172.21.15.118:6789/0 conn(0x7f46740a2710 legacy=0x7f467413c070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:05:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): get_auth_request con 0x7f467406b640 auth_method 0 2023-10-30T12:05:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient(hunting): handle_auth_done global_id 84213 payload 274 2023-10-30T12:05:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 1 -- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467406afc0 msgr2=0x7f46740a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 1 --2- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467406afc0 0x7f46740a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 1 -- 172.21.15.50:0/448938546 >> v1:172.21.15.118:6789/0 conn(0x7f46740a2710 legacy=0x7f467413c070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:05:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 1 monclient: found mon.noname-a 2023-10-30T12:05:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f466bfff640 1 -- 172.21.15.50:0/448938546 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467409e540 con 0x7f467406b640 2023-10-30T12:05:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.936+0000 7f467259c640 1 --2- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467406afc0 0x7f46740a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f466bfff640 1 --2- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406b640 0x7f46740a21d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f465c0097a0 tx=0x7f465c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddd989a67278939 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 1 -- 172.21.15.50:0/448938546 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c030710 con 0x7f467406b640 2023-10-30T12:05:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: dump: 2023-10-30T12:05:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.958 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.959 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: _finish_auth 0 2023-10-30T12:05:34.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:05:34.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:34.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 1 -- 172.21.15.50:0/448938546 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c02e050 con 0x7f467406b640 2023-10-30T12:05:34.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 5 monclient: authenticate success, global_id 84213 2023-10-30T12:05:34.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:34.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:34.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:34.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 1 -- 172.21.15.50:0/448938546 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c030a90 con 0x7f467406b640 2023-10-30T12:05:34.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 4 set_mon_vals no callback set 2023-10-30T12:05:34.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:34.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4671d9b640 10 monclient: dump: 2023-10-30T12:05:34.962 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.962 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.962 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.962 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.963 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.963 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.963 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.963 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.963 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.963 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 10 monclient: shutdown 2023-10-30T12:05:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 -- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406b640 msgr2=0x7f46740a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 --2- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406b640 0x7f46740a21d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f465c0097a0 tx=0x7f465c02b710 comp rx=0 tx=0).stop 2023-10-30T12:05:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 -- 172.21.15.50:0/448938546 shutdown_connections 2023-10-30T12:05:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 --2- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406b640 0x7f46740a21d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 --2- 172.21.15.50:0/448938546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467406afc0 0x7f46740a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 -- 172.21.15.50:0/448938546 >> 172.21.15.50:0/448938546 conn(0x7f46740a6200 msgr2=0x7f46740a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 -- 172.21.15.50:0/448938546 shutdown_connections 2023-10-30T12:05:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.937+0000 7f4678c56640 1 -- 172.21.15.50:0/448938546 wait complete. 2023-10-30T12:05:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.938+0000 7f4678c56640 10 monclient: build_initial_monmap 2023-10-30T12:05:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.938+0000 7f4678c56640 10 monclient: monmap: 2023-10-30T12:05:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:34.968 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.938+0000 7f4678c56640 1 librados: starting msgr at 2023-10-30T12:05:34.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.938+0000 7f4678c56640 1 librados: starting objecter 2023-10-30T12:05:34.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.938+0000 7f4678c56640 1 Processor -- start 2023-10-30T12:05:34.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 -- start start 2023-10-30T12:05:34.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 librados: setting wanted keys 2023-10-30T12:05:34.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 librados: calling monclient init 2023-10-30T12:05:34.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient: init 2023-10-30T12:05:34.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:34.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:05:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406afc0 0x7f46741bdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): picked mon.noname-a con 0x7f467406afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f467406b640 0x7f46741a7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): picked mon.noname-b con 0x7f467406b640 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46740a2710 0x7f46741a7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): picked mon.noname-c con 0x7f46740a2710 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:34.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46741caf00 con 0x7f467406afc0 2023-10-30T12:05:34.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46741ab470 con 0x7f467406b640 2023-10-30T12:05:34.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4674070770 con 0x7f46740a2710 2023-10-30T12:05:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4678c56640 10 monclient(hunting): authenticate will time out at 1180.122455s 2023-10-30T12:05:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406afc0 0x7f46741bdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46740a2710 0x7f46741a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f466bfff640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f467406b640 msgr2=0x7f46741a7400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:05:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406afc0 0x7f46741bdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56444/0 (socket says 172.21.15.50:56444) 2023-10-30T12:05:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 -- 172.21.15.50:0/397046332 learned_addr learned my addr 172.21.15.50:0/397046332 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 10 monclient(hunting): get_auth_request con 0x7f46740a2710 auth_method 0 2023-10-30T12:05:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f466bfff640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f467406b640 0x7f46741a7400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): get_auth_request con 0x7f467406afc0 auth_method 0 2023-10-30T12:05:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient(hunting): handle_auth_done global_id 84216 payload 824 2023-10-30T12:05:34.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46740a2710 msgr2=0x7f46741a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:34.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46740a2710 0x7f46741a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f467406b640 msgr2=0x7f46741a7400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:05:34.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f467406b640 0x7f46741a7400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:34.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient: _finish_hunting 0 2023-10-30T12:05:34.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 monclient: found mon.noname-a 2023-10-30T12:05:34.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:34.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f467259c640 1 -- 172.21.15.50:0/397046332 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640094d0 con 0x7f467406afc0 2023-10-30T12:05:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.939+0000 7f4672d9d640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46740a2710 0x7f46741a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f467259c640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406afc0 0x7f46741bdf50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f466400ea00 tx=0x7f466400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 client.?.objecter ms_handle_connect 0x7f467406afc0 2023-10-30T12:05:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466400ca60 con 0x7f467406afc0 2023-10-30T12:05:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:34.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: dump: 2023-10-30T12:05:34.980 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.981 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.981 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.981 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.982 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.982 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.982 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.984 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: _finish_auth 0 2023-10-30T12:05:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: _check_auth_tickets 2023-10-30T12:05:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466400cbc0 con 0x7f467406afc0 2023-10-30T12:05:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 5 monclient: authenticate success, global_id 84216 2023-10-30T12:05:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664013470 con 0x7f467406afc0 2023-10-30T12:05:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f466affd640 10 monclient: dump: 2023-10-30T12:05:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:34.988 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:34.990 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:34.991 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:34.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 10 monclient: _renew_subs 2023-10-30T12:05:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f467159a640 4 set_mon_vals no callback set 2023-10-30T12:05:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c009400 con 0x7f467406afc0 2023-10-30T12:05:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 10 client.84216.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 10 monclient: _renew_subs 2023-10-30T12:05:34.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:34.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46741ab220 con 0x7f467406afc0 2023-10-30T12:05:34.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.940+0000 7f4678c56640 1 librados: init done 2023-10-30T12:05:34.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f4678c56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:34.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f4678c56640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402b430 0x7f467402b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f4678c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46741ab600 con 0x7f467402b430 2023-10-30T12:05:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f46640135d0 con 0x7f467406afc0 2023-10-30T12:05:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402b430 0x7f467402b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient: get_auth_request con 0x7f467402b430 auth_method 0 2023-10-30T12:05:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:34.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:34.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46540392d0 0x7f465403b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 7202+0+0 (secure 0 0 0) 0x7f466404b360 con 0x7f467406afc0 2023-10-30T12:05:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 10 client.84216.objecter ms_dispatch 0x7f4674000b90 osd_map(141..141 src has 1..141) v4 2023-10-30T12:05:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 3 client.84216.objecter handle_osd_map got epochs [141,141] > 0 2023-10-30T12:05:34.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 3 client.84216.objecter handle_osd_map decoding full epoch 141 2023-10-30T12:05:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 10 monclient(hunting): handle_auth_done global_id 44723 payload 824 2023-10-30T12:05:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 1 -- 172.21.15.50:0/397046332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f467409df80 con 0x7f467402b430 2023-10-30T12:05:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 20 client.84216.objecter dump_active .. 0 homeless 2023-10-30T12:05:34.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 10 client.84216.objecter ms_handle_connect 0x7f467402b430 2023-10-30T12:05:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 10 client.84216.objecter resend_mon_ops 2023-10-30T12:05:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466bfff640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402b430 0x7f467402b810 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f46740712a0 tx=0x7f465c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f4672d9d640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46540392d0 0x7f465403b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:34.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f4672d9d640 10 monclient: get_auth_request con 0x7f46540392d0 auth_method 0 2023-10-30T12:05:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c004220 con 0x7f467402b430 2023-10-30T12:05:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:35.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.941+0000 7f466affd640 10 monclient: dump: 2023-10-30T12:05:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:35.001 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:35.002 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:35.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:35.003 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 10 client.84216.objecter ms_handle_connect 0x7f46540392d0 2023-10-30T12:05:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f4672d9d640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46540392d0 0x7f465403b790 secure :-1 s=READY pgs=3949 cs=0 l=1 rev1=1 crypto rx=0x7f466c003bc0 tx=0x7f466c00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f465c004380 con 0x7f467402b430 2023-10-30T12:05:35.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402b430 msgr2=0x7f467402b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402b430 0x7f467402b810 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f46740712a0 tx=0x7f465c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:35.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402b430 msgr2=0x7f467402b810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:35.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.942+0000 7f466affd640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402b430 0x7f467402b810 secure :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f46740712a0 tx=0x7f465c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:05:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f4678c56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:05:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f4678c56640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402bd50 0x7f467402c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f4678c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:35.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4674073ff0 con 0x7f467402bd50 2023-10-30T12:05:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402bd50 0x7f467402c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient: get_auth_request con 0x7f467402bd50 auth_method 0 2023-10-30T12:05:35.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 10 monclient(hunting): handle_auth_done global_id 44726 payload 824 2023-10-30T12:05:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.948+0000 7f466bfff640 1 -- 172.21.15.50:0/397046332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f467406a6f0 con 0x7f467402bd50 2023-10-30T12:05:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.949+0000 7f466affd640 10 client.84216.objecter ms_handle_connect 0x7f467402bd50 2023-10-30T12:05:35.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.949+0000 7f466affd640 10 client.84216.objecter resend_mon_ops 2023-10-30T12:05:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.949+0000 7f466bfff640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402bd50 0x7f467402c160 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f465c0097a0 tx=0x7f465c009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c04c070 con 0x7f467402bd50 2023-10-30T12:05:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:35.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:05:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 10 monclient: dump: 2023-10-30T12:05:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:35.014 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:35.015 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f465c003b70 con 0x7f467402bd50 2023-10-30T12:05:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:05:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402bd50 msgr2=0x7f467402c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402bd50 0x7f467402c160 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f465c0097a0 tx=0x7f465c009190 comp rx=0 tx=0).stop 2023-10-30T12:05:35.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402bd50 msgr2=0x7f467402c160 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:05:35.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f466affd640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f467402bd50 0x7f467402c160 secure :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f465c0097a0 tx=0x7f465c009190 comp rx=0 tx=0).stop 2023-10-30T12:05:35.017 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:05:35.018 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:35.018 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:35.018 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:05:35.018 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 54, 2023-10-30T12:05:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:05:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:05:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:05:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:05:35.020 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:35.020 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:05:35.020 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:05:35.020 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:35.020 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:35.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:35.021 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:35.021 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:35.021 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:35.021 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:35.022 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:35.022 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:35.022 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:35.022 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:35.023 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:05:35.023 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:05:35.023 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:35.023 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:35.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:35.024 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:35.024 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:35.024 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:35.025 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:35.025 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:35.025 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:35.025 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:35.025 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:35.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.026 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:05:35.026 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:05:35.026 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:05:35.026 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:05:35.027 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:05:35.027 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:05:35.027 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:35.027 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:05:35.028 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:05:35.028 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:05:35.028 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:05:35.028 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:05:35.028 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:05:35.029 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:05:35.029 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:05:35.029 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:05:35.029 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:05:35.029 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:05:35.030 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:05:35.030 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:05:35.030 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:05:35.030 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:05:35.030 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:05:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:05:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:05:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:05:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:05:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:35.032 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:05:35.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.032 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:05:35.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.033 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:05:35.033 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:05:35.033 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:35.033 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:35.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:35.034 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:05:35.034 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:35.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.035 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:35.035 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:05:35.035 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:35.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:35.035 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:35.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.036 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:05:35.036 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:05:35.036 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:35.036 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:35.037 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:35.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.037 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:05:35.037 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:05:35.038 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:35.038 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:35.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:35.039 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:05:35.039 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:35.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.039 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:35.042 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:05:35.042 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:35.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:35.042 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:05:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:05:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:35.043 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:05:35.044 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:05:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:05:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:05:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:05:35.045 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:05:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:05:35.046 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:05:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:05:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:35.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:05:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:05:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:05:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:05:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:05:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:05:35.049 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:05:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:05:35.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:05:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:35.051 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:05:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:05:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:05:35.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:05:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:05:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:05:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:05:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:05:35.053 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:05:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:05:35.054 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:05:35.054 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:05:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 10 librados: watch_flush enter 2023-10-30T12:05:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 10 librados: watch_flush exit 2023-10-30T12:05:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 20 client.84216.objecter shutdown clearing up homeless session... 2023-10-30T12:05:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 10 client.84216.objecter successfully canceled tick 2023-10-30T12:05:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46540392d0 msgr2=0x7f465403b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46540392d0 0x7f465403b790 secure :-1 s=READY pgs=3949 cs=0 l=1 rev1=1 crypto rx=0x7f466c003bc0 tx=0x7f466c00d040 comp rx=0 tx=0).stop 2023-10-30T12:05:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 10 monclient: shutdown 2023-10-30T12:05:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406afc0 msgr2=0x7f46741bdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 --2- 172.21.15.50:0/397046332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f467406afc0 0x7f46741bdf50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f466400ea00 tx=0x7f466400eec0 comp rx=0 tx=0).stop 2023-10-30T12:05:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4672d9d640 1 -- 172.21.15.50:0/397046332 reap_dead start 2023-10-30T12:05:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4672d9d640 1 -- 172.21.15.50:0/397046332 reap_dead start 2023-10-30T12:05:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 shutdown_connections 2023-10-30T12:05:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 >> 172.21.15.50:0/397046332 conn(0x7f46740a6200 msgr2=0x7f467406e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 shutdown_connections 2023-10-30T12:05:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 -- 172.21.15.50:0/397046332 wait complete. 2023-10-30T12:05:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:34.951+0000 7f4678c56640 1 librados: shutdown 2023-10-30T12:05:35.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:05:35.060 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:05:55.062 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:05:55.062 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-10-30T12:05:55.062 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:05:55.063 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:05:55.066 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:05:55.067 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:05:55.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:55.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.187+0000 7f7aa6897640 10 monclient: get_monmap_and_config 2023-10-30T12:05:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.187+0000 7f7aa6897640 10 monclient: build_initial_monmap 2023-10-30T12:05:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.187+0000 7f7aa6897640 10 monclient: monmap: 2023-10-30T12:05:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:55.189 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:55.188135+0000 2023-10-30T12:05:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:55.188135+0000 2023-10-30T12:05:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:55.190 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:55.191 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.187+0000 7f7aa6897640 1 Processor -- start 2023-10-30T12:05:55.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.187+0000 7f7aa6897640 1 -- start start 2023-10-30T12:05:55.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.187+0000 7f7aa6897640 10 monclient: init 2023-10-30T12:05:55.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:55.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:05:55.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7aa0131b20 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7aa0132270 0x7f7aa0071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa0132270 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7aa0072100 0x7f7aa0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa0072100 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa009a9b0 con 0x7f7aa0131b20 2023-10-30T12:05:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa0131f00 con 0x7f7aa0072100 2023-10-30T12:05:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa009d920 con 0x7f7aa0132270 2023-10-30T12:05:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa6897640 10 monclient(hunting): authenticate will time out at 1200.371326s 2023-10-30T12:05:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7aa0072100 0x7f7aa0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7aa0072100 0x7f7aa0139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36006/0 (socket says 172.21.15.50:36006) 2023-10-30T12:05:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9ffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7aa0131b20 0x7f7aa009f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42118/0 (socket says 172.21.15.50:42118) 2023-10-30T12:05:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 -- 172.21.15.50:0/4127687450 learned_addr learned my addr 172.21.15.50:0/4127687450 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): get_auth_request con 0x7f7aa0072100 auth_method 0 2023-10-30T12:05:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9f7fe640 1 --2- 172.21.15.50:0/4127687450 >> v2:172.21.15.118:3300/0 conn(0x7f7aa0132270 0x7f7aa0071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9f7fe640 10 monclient(hunting): get_auth_request con 0x7f7aa0132270 auth_method 0 2023-10-30T12:05:55.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient(hunting): handle_auth_done global_id 84297 payload 274 2023-10-30T12:05:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 -- 172.21.15.50:0/4127687450 >> v2:172.21.15.118:3300/0 conn(0x7f7aa0132270 msgr2=0x7f7aa0071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 --2- 172.21.15.50:0/4127687450 >> v2:172.21.15.118:3300/0 conn(0x7f7aa0132270 0x7f7aa0071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 -- 172.21.15.50:0/4127687450 >> v1:172.21.15.50:6789/0 conn(0x7f7aa0131b20 legacy=0x7f7aa009f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:55.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient: _finish_hunting 0 2023-10-30T12:05:55.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 monclient: found mon.noname-a 2023-10-30T12:05:55.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:55.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9effd640 1 -- 172.21.15.50:0/4127687450 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343515539 0 0) 0x7f7aa009a9b0 con 0x7f7aa0131b20 2023-10-30T12:05:55.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7aa4e34640 1 -- 172.21.15.50:0/4127687450 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa009a0b0 con 0x7f7aa0072100 2023-10-30T12:05:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.188+0000 7f7a9effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:05:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7aa4e34640 1 --2- 172.21.15.50:0/4127687450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa0139f60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0098b70 tx=0x7f7a9802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37528c9b3c0a232 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 1 -- 172.21.15.50:0/4127687450 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9802ea00 con 0x7f7aa0072100 2023-10-30T12:05:55.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: dump: 2023-10-30T12:05:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:55.209 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:55.210 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: _finish_auth 0 2023-10-30T12:05:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: _check_auth_tickets 2023-10-30T12:05:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 1 -- 172.21.15.50:0/4127687450 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9802eb60 con 0x7f7aa0072100 2023-10-30T12:05:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7aa6897640 5 monclient: authenticate success, global_id 84297 2023-10-30T12:05:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7aa6897640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7aa6897640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 1 -- 172.21.15.50:0/4127687450 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98009600 con 0x7f7aa0072100 2023-10-30T12:05:55.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:55.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7aa6897640 4 set_mon_vals no callback set 2023-10-30T12:05:55.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:55.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.189+0000 7f7a9effd640 10 monclient: dump: 2023-10-30T12:05:55.212 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:55.213 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:55.213 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.213 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.213 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:55.213 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:55.214 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 10 monclient: shutdown 2023-10-30T12:05:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:55.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 -- 172.21.15.50:0/4127687450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 msgr2=0x7f7aa0139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:55.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 --2- 172.21.15.50:0/4127687450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa0139f60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0098b70 tx=0x7f7a9802e060 comp rx=0 tx=0).stop 2023-10-30T12:05:55.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 -- 172.21.15.50:0/4127687450 shutdown_connections 2023-10-30T12:05:55.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 --2- 172.21.15.50:0/4127687450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa0139f60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 --2- 172.21.15.50:0/4127687450 >> v2:172.21.15.118:3300/0 conn(0x7f7aa0132270 0x7f7aa0071bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 -- 172.21.15.50:0/4127687450 >> 172.21.15.50:0/4127687450 conn(0x7f7aa00a05c0 msgr2=0x7f7aa009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:55.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 -- 172.21.15.50:0/4127687450 shutdown_connections 2023-10-30T12:05:55.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 1 -- 172.21.15.50:0/4127687450 wait complete. 2023-10-30T12:05:55.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.190+0000 7f7aa6897640 10 monclient: build_initial_monmap 2023-10-30T12:05:55.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 10 monclient: monmap: 2023-10-30T12:05:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:55.218 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 1 librados: starting msgr at 2023-10-30T12:05:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 1 librados: starting objecter 2023-10-30T12:05:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 1 Processor -- start 2023-10-30T12:05:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 1 -- start start 2023-10-30T12:05:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 1 librados: setting wanted keys 2023-10-30T12:05:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 1 librados: calling monclient init 2023-10-30T12:05:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 10 monclient: init 2023-10-30T12:05:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.191+0000 7f7aa6897640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:05:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa01c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa0072100 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7aa0131b20 0x7f7aa0130d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:55.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa0131b20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:55.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0132270 0x7f7aa0131240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:55.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa0132270 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:55.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:55.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa00a2db0 con 0x7f7aa0072100 2023-10-30T12:05:55.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:55.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa01c6c10 con 0x7f7aa0132270 2023-10-30T12:05:55.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:55.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa012c690 con 0x7f7aa0131b20 2023-10-30T12:05:55.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient(hunting): authenticate will time out at 1200.375108s 2023-10-30T12:05:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa01c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7aa0131b20 0x7f7aa0130d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa01c5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36020/0 (socket says 172.21.15.50:36020) 2023-10-30T12:05:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 -- 172.21.15.50:0/242892372 learned_addr learned my addr 172.21.15.50:0/242892372 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): get_auth_request con 0x7f7aa0072100 auth_method 0 2023-10-30T12:05:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:55.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 10 monclient(hunting): get_auth_request con 0x7f7aa0131b20 auth_method 0 2023-10-30T12:05:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient(hunting): handle_auth_done global_id 84303 payload 824 2023-10-30T12:05:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 -- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7aa0131b20 msgr2=0x7f7aa0130d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7aa0131b20 0x7f7aa0130d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 -- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0132270 msgr2=0x7f7aa0131240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0132270 0x7f7aa0131240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:05:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 monclient: found mon.noname-a 2023-10-30T12:05:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 -- 172.21.15.50:0/242892372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9800bd40 con 0x7f7aa0072100 2023-10-30T12:05:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9f7fe640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7aa0131b20 0x7f7aa0130d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9ffff640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa01c5f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000c550 tx=0x7f7a9000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 client.?.objecter ms_handle_connect 0x7f7aa0072100 2023-10-30T12:05:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 1 -- 172.21.15.50:0/242892372 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a90007500 con 0x7f7aa0072100 2023-10-30T12:05:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: dump: 2023-10-30T12:05:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:55.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: _finish_auth 0 2023-10-30T12:05:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:05:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 1 -- 172.21.15.50:0/242892372 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a900057a0 con 0x7f7aa0072100 2023-10-30T12:05:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:55.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 1 -- 172.21.15.50:0/242892372 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a90005b50 con 0x7f7aa0072100 2023-10-30T12:05:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 5 monclient: authenticate success, global_id 84303 2023-10-30T12:05:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient: _renew_subs 2023-10-30T12:05:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa012d1f0 con 0x7f7aa0072100 2023-10-30T12:05:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:55.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9e7fc640 4 set_mon_vals no callback set 2023-10-30T12:05:55.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:55.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.192+0000 7f7a9cff9640 10 monclient: dump: 2023-10-30T12:05:55.234 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:55.234 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:55.235 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.235 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:55.235 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:55.235 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:55.235 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:55.235 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:55.236 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:55.236 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:55.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.193+0000 7f7aa6897640 10 client.84303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:55.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.193+0000 7f7aa6897640 10 monclient: _renew_subs 2023-10-30T12:05:55.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.193+0000 7f7aa6897640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:55.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.193+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa012b470 con 0x7f7aa0072100 2023-10-30T12:05:55.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.193+0000 7f7aa6897640 1 librados: init done 2023-10-30T12:05:55.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.193+0000 7f7a9cff9640 1 -- 172.21.15.50:0/242892372 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7a90005cb0 con 0x7f7aa0072100 2023-10-30T12:05:55.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7aa6897640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:55.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7aa6897640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:55.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7aa6897640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:55.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa009d4d0 con 0x7f7aa0072100 2023-10-30T12:05:55.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9cff9640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a8c0392f0 0x7f7a8c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:55.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9cff9640 1 -- 172.21.15.50:0/242892372 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 8413+0+0 (secure 0 0 0) 0x7f7a9004abe0 con 0x7f7aa0072100 2023-10-30T12:05:55.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9cff9640 10 client.84303.objecter ms_dispatch 0x7f7aa0000b90 osd_map(162..162 src has 1..162) v4 2023-10-30T12:05:55.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9cff9640 3 client.84303.objecter handle_osd_map got epochs [162,162] > 0 2023-10-30T12:05:55.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9cff9640 3 client.84303.objecter handle_osd_map decoding full epoch 162 2023-10-30T12:05:55.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9cff9640 20 client.84303.objecter dump_active .. 0 homeless 2023-10-30T12:05:55.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9f7fe640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a8c0392f0 0x7f7a8c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:55.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9f7fe640 10 monclient: get_auth_request con 0x7f7a8c0392f0 auth_method 0 2023-10-30T12:05:55.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9f7fe640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a8c0392f0 0x7f7a8c03b7b0 secure :-1 s=READY pgs=3963 cs=0 l=1 rev1=1 crypto rx=0x7f7aa012af50 tx=0x7f7a94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:55.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.194+0000 7f7a9cff9640 10 client.84303.objecter ms_handle_connect 0x7f7a8c0392f0 2023-10-30T12:05:55.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.199+0000 7f7a9cff9640 1 -- 172.21.15.50:0/242892372 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a9000e280 con 0x7f7aa0072100 2023-10-30T12:05:55.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.199+0000 7f7a9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:55.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.199+0000 7f7a9cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:55.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.452+0000 7f7aa6897640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:55.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.452+0000 7f7aa6897640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:55.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.452+0000 7f7aa6897640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:55.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.452+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7aa00a14b0 con 0x7f7aa0072100 2023-10-30T12:05:55.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.453+0000 7f7a9cff9640 1 -- 172.21.15.50:0/242892372 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f7a90017070 con 0x7f7aa0072100 2023-10-30T12:05:55.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.453+0000 7f7a9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:55.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.453+0000 7f7a9cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:55.456 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:55.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 10 librados: watch_flush enter 2023-10-30T12:05:55.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 10 librados: watch_flush exit 2023-10-30T12:05:55.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 20 client.84303.objecter shutdown clearing up homeless session... 2023-10-30T12:05:55.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 10 client.84303.objecter successfully canceled tick 2023-10-30T12:05:55.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a8c0392f0 msgr2=0x7f7a8c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:55.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a8c0392f0 0x7f7a8c03b7b0 secure :-1 s=READY pgs=3963 cs=0 l=1 rev1=1 crypto rx=0x7f7aa012af50 tx=0x7f7a94008040 comp rx=0 tx=0).stop 2023-10-30T12:05:55.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 10 monclient: shutdown 2023-10-30T12:05:55.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:55.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 msgr2=0x7f7aa01c5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:55.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.454+0000 7f7aa6897640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa01c5f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000c550 tx=0x7f7a9000ca10 comp rx=0 tx=0).stop 2023-10-30T12:05:55.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 shutdown_connections 2023-10-30T12:05:55.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a8c0392f0 0x7f7a8c03b7b0 unknown :-1 s=CLOSED pgs=3963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0132270 0x7f7aa0131240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7aa0131b20 0x7f7aa0130d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 --2- 172.21.15.50:0/242892372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa0072100 0x7f7aa01c5f30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 >> 172.21.15.50:0/242892372 conn(0x7f7aa00a05c0 msgr2=0x7f7aa009b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 shutdown_connections 2023-10-30T12:05:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 -- 172.21.15.50:0/242892372 wait complete. 2023-10-30T12:05:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:55.455+0000 7f7aa6897640 1 librados: shutdown 2023-10-30T12:05:58.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:05:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.580+0000 7f7897939640 10 monclient: get_monmap_and_config 2023-10-30T12:05:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.580+0000 7f7897939640 10 monclient: build_initial_monmap 2023-10-30T12:05:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.580+0000 7f7897939640 10 monclient: monmap: 2023-10-30T12:05:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:05:58.581847+0000 2023-10-30T12:05:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:05:58.581847+0000 2023-10-30T12:05:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:05:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:05:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:05:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:05:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:05:58.585 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.581+0000 7f7897939640 1 Processor -- start 2023-10-30T12:05:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.581+0000 7f7897939640 1 -- start start 2023-10-30T12:05:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.581+0000 7f7897939640 10 monclient: init 2023-10-30T12:05:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.581+0000 7f7897939640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.581+0000 7f7897939640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-10-30T12:05:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7890131b20 0x7f7890132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 10 monclient(hunting): picked mon.noname-c con 0x7f7890131b20 addr v2:172.21.15.118:3300/0 2023-10-30T12:05:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7890071bc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:05:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7890071fc0 0x7f7890097b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 10 monclient(hunting): picked mon.noname-a con 0x7f7890071fc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:05:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789009a9b0 con 0x7f7890071bc0 2023-10-30T12:05:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7890131f00 con 0x7f7890071fc0 2023-10-30T12:05:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789009d920 con 0x7f7890131b20 2023-10-30T12:05:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7897939640 10 monclient(hunting): authenticate will time out at 1203.765078s 2023-10-30T12:05:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7890071fc0 0x7f7890097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7894ed4640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7890071bc0 0x7f7890097410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55958/0 (socket says 172.21.15.50:55958) 2023-10-30T12:05:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7894ed4640 1 -- 172.21.15.50:0/1859191163 learned_addr learned my addr 172.21.15.50:0/1859191163 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:05:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): get_auth_request con 0x7f7890071fc0 auth_method 0 2023-10-30T12:05:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f78956d5640 1 --2- 172.21.15.50:0/1859191163 >> v2:172.21.15.118:3300/0 conn(0x7f7890131b20 0x7f7890132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f78956d5640 10 monclient(hunting): get_auth_request con 0x7f7890131b20 auth_method 0 2023-10-30T12:05:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f78956d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f78956d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f78956d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient(hunting): handle_auth_done global_id 84312 payload 274 2023-10-30T12:05:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 1 -- 172.21.15.50:0/1859191163 >> v2:172.21.15.118:3300/0 conn(0x7f7890131b20 msgr2=0x7f7890132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:05:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 1 --2- 172.21.15.50:0/1859191163 >> v2:172.21.15.118:3300/0 conn(0x7f7890131b20 0x7f7890132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 1 -- 172.21.15.50:0/1859191163 >> v1:172.21.15.50:6789/0 conn(0x7f7890071bc0 legacy=0x7f7890097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient: _finish_hunting 0 2023-10-30T12:05:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 1 monclient: found mon.noname-a 2023-10-30T12:05:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.582+0000 7f7895ed6640 1 -- 172.21.15.50:0/1859191163 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789009a0b0 con 0x7f7890071fc0 2023-10-30T12:05:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7895ed6640 1 --2- 172.21.15.50:0/1859191163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890071fc0 0x7f7890097b20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f789009a5d0 tx=0x7f788802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f8c4eceeb0ef0c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 1 -- 172.21.15.50:0/1859191163 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788802cb60 con 0x7f7890071fc0 2023-10-30T12:05:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: dump: 2023-10-30T12:05:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:58.605 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: _finish_auth 0 2023-10-30T12:05:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: _check_auth_tickets 2023-10-30T12:05:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 1 -- 172.21.15.50:0/1859191163 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788802ccc0 con 0x7f7890071fc0 2023-10-30T12:05:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 5 monclient: authenticate success, global_id 84312 2023-10-30T12:05:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:05:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 10 monclient: get_monmap_and_config success 2023-10-30T12:05:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 4 set_mon_vals no callback set 2023-10-30T12:05:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 1 -- 172.21.15.50:0/1859191163 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888034660 con 0x7f7890071fc0 2023-10-30T12:05:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7887fff640 10 monclient: dump: 2023-10-30T12:05:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:58.610 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 10 monclient: shutdown 2023-10-30T12:05:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 1 -- 172.21.15.50:0/1859191163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890071fc0 msgr2=0x7f7890097b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.583+0000 7f7897939640 1 --2- 172.21.15.50:0/1859191163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890071fc0 0x7f7890097b20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f789009a5d0 tx=0x7f788802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 -- 172.21.15.50:0/1859191163 shutdown_connections 2023-10-30T12:05:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 --2- 172.21.15.50:0/1859191163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890071fc0 0x7f7890097b20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 --2- 172.21.15.50:0/1859191163 >> v2:172.21.15.118:3300/0 conn(0x7f7890131b20 0x7f7890132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 -- 172.21.15.50:0/1859191163 >> 172.21.15.50:0/1859191163 conn(0x7f78900a05c0 msgr2=0x7f789009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 -- 172.21.15.50:0/1859191163 shutdown_connections 2023-10-30T12:05:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 -- 172.21.15.50:0/1859191163 wait complete. 2023-10-30T12:05:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 10 monclient: build_initial_monmap 2023-10-30T12:05:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 10 monclient: monmap: 2023-10-30T12:05:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:05:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:05:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:05:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:05:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:05:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:05:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:05:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:05:58.614 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 librados: starting msgr at 2023-10-30T12:05:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 librados: starting objecter 2023-10-30T12:05:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.584+0000 7f7897939640 1 Processor -- start 2023-10-30T12:05:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 -- start start 2023-10-30T12:05:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 librados: setting wanted keys 2023-10-30T12:05:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 librados: calling monclient init 2023-10-30T12:05:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient: init 2023-10-30T12:05:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient: _reopen_session rank -1 2023-10-30T12:05:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:05:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 0x7f789013a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): picked mon.noname-b con 0x7f7890071bc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:05:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7890071fc0 0x7f789013aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): picked mon.noname-c con 0x7f7890071fc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:05:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890131b20 0x7f78901ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): picked mon.noname-a con 0x7f7890131b20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:05:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789006d810 con 0x7f7890131b20 2023-10-30T12:05:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789006da60 con 0x7f7890071bc0 2023-10-30T12:05:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:05:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789006c620 con 0x7f7890071fc0 2023-10-30T12:05:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): _renew_subs 2023-10-30T12:05:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7897939640 10 monclient(hunting): authenticate will time out at 1203.768500s 2023-10-30T12:05:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890131b20 0x7f78901ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7890071fc0 0x7f789013aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890131b20 0x7f78901ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38592/0 (socket says 172.21.15.50:38592) 2023-10-30T12:05:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 -- 172.21.15.50:0/1017816136 learned_addr learned my addr 172.21.15.50:0/1017816136 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:05:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): get_auth_request con 0x7f7890131b20 auth_method 0 2023-10-30T12:05:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 10 monclient(hunting): get_auth_request con 0x7f7890071fc0 auth_method 0 2023-10-30T12:05:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:58.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:58.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:58.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:58.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:58.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:58.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:05:58.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:05:58.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:05:58.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 0x7f789013a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:58.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 10 monclient(hunting): get_auth_request con 0x7f7890071bc0 auth_method 0 2023-10-30T12:05:58.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:05:58.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:05:58.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:05:58.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient(hunting): handle_auth_done global_id 84318 payload 824 2023-10-30T12:05:58.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 1 -- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 msgr2=0x7f789013a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:05:58.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 -- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7890071fc0 msgr2=0x7f789013aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:58.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 1 -- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 msgr2=0x7f789013a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:05:58.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7890071fc0 0x7f789013aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 0x7f789013a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:05:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 -- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 msgr2=0x7f789013a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f78956d5640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 0x7f789013a9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:05:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 0x7f789013a9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient: _finish_hunting 0 2023-10-30T12:05:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 monclient: found mon.noname-a 2023-10-30T12:05:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:05:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:05:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7895ed6640 1 -- 172.21.15.50:0/1017816136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888009a70 con 0x7f7890131b20 2023-10-30T12:05:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.585+0000 7f7894ed4640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7890071fc0 0x7f789013aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:05:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7895ed6640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890131b20 0x7f78901ca010 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f788802c3c0 tx=0x7f7888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 client.?.objecter ms_handle_connect 0x7f7890131b20 2023-10-30T12:05:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:05:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 1 -- 172.21.15.50:0/1017816136 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888040650 con 0x7f7890131b20 2023-10-30T12:05:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:05:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: dump: 2023-10-30T12:05:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:58.632 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: _finish_auth 0 2023-10-30T12:05:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:05:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:05:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 1 -- 172.21.15.50:0/1017816136 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78880407d0 con 0x7f7890131b20 2023-10-30T12:05:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 5 monclient: authenticate success, global_id 84318 2023-10-30T12:05:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 10 monclient: _renew_subs 2023-10-30T12:05:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789006bef0 con 0x7f7890131b20 2023-10-30T12:05:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:05:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 1 -- 172.21.15.50:0/1017816136 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888040b80 con 0x7f7890131b20 2023-10-30T12:05:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:05:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:05:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7885ffb640 10 monclient: dump: 2023-10-30T12:05:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:05:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:05:58.635 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.635 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:05:58.635 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:05:58.635 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:05:58.635 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:05:58.636 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:05:58.636 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:05:58.636 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:05:58.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f78877fe640 4 set_mon_vals no callback set 2023-10-30T12:05:58.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 10 client.84318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:05:58.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 10 monclient: _renew_subs 2023-10-30T12:05:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78900a1b90 con 0x7f7890131b20 2023-10-30T12:05:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.586+0000 7f7897939640 1 librados: init done 2023-10-30T12:05:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7885ffb640 1 -- 172.21.15.50:0/1017816136 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7888040ce0 con 0x7f7890131b20 2023-10-30T12:05:58.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7897939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7897939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7897939640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789004e160 con 0x7f7890131b20 2023-10-30T12:05:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7885ffb640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78740392f0 0x7f787403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:05:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7885ffb640 1 -- 172.21.15.50:0/1017816136 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 7531+0+0 (secure 0 0 0) 0x7f788806e4c0 con 0x7f7890131b20 2023-10-30T12:05:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7885ffb640 10 client.84318.objecter ms_dispatch 0x7f7890000b90 osd_map(166..166 src has 1..166) v4 2023-10-30T12:05:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7885ffb640 3 client.84318.objecter handle_osd_map got epochs [166,166] > 0 2023-10-30T12:05:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7885ffb640 3 client.84318.objecter handle_osd_map decoding full epoch 166 2023-10-30T12:05:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f7885ffb640 20 client.84318.objecter dump_active .. 0 homeless 2023-10-30T12:05:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f78956d5640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78740392f0 0x7f787403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:05:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.587+0000 7f78956d5640 10 monclient: get_auth_request con 0x7f78740392f0 auth_method 0 2023-10-30T12:05:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.588+0000 7f7885ffb640 10 client.84318.objecter ms_handle_connect 0x7f78740392f0 2023-10-30T12:05:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.588+0000 7f78956d5640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78740392f0 0x7f787403b7b0 secure :-1 s=READY pgs=3965 cs=0 l=1 rev1=1 crypto rx=0x7f788c006a00 tx=0x7f788c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:05:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.592+0000 7f7885ffb640 1 -- 172.21.15.50:0/1017816136 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f78880517c0 con 0x7f7890131b20 2023-10-30T12:05:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.592+0000 7f7885ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:05:58.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.592+0000 7f7885ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:05:58.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.835+0000 7f7897939640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:05:58.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.835+0000 7f7897939640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:58.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.835+0000 7f7897939640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:05:58.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.835+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f789006dba0 con 0x7f7890131b20 2023-10-30T12:05:58.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.835+0000 7f7885ffb640 1 -- 172.21.15.50:0/1017816136 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f7888044030 con 0x7f7890131b20 2023-10-30T12:05:58.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.835+0000 7f7885ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:05:58.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.835+0000 7f7885ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:05:58.838 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:05:58.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 10 librados: watch_flush enter 2023-10-30T12:05:58.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 10 librados: watch_flush exit 2023-10-30T12:05:58.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 20 client.84318.objecter shutdown clearing up homeless session... 2023-10-30T12:05:58.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 10 client.84318.objecter successfully canceled tick 2023-10-30T12:05:58.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78740392f0 msgr2=0x7f787403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:58.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78740392f0 0x7f787403b7b0 secure :-1 s=READY pgs=3965 cs=0 l=1 rev1=1 crypto rx=0x7f788c006a00 tx=0x7f788c008040 comp rx=0 tx=0).stop 2023-10-30T12:05:58.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 10 monclient: shutdown 2023-10-30T12:05:58.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:05:58.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890131b20 msgr2=0x7f78901ca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:05:58.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890131b20 0x7f78901ca010 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f788802c3c0 tx=0x7f7888004a40 comp rx=0 tx=0).stop 2023-10-30T12:05:58.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 shutdown_connections 2023-10-30T12:05:58.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78740392f0 0x7f787403b7b0 unknown :-1 s=CLOSED pgs=3965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7890131b20 0x7f78901ca010 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7890071fc0 0x7f789013aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 --2- 172.21.15.50:0/1017816136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7890071bc0 0x7f789013a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:05:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 >> 172.21.15.50:0/1017816136 conn(0x7f78900a05c0 msgr2=0x7f789009ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:05:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 shutdown_connections 2023-10-30T12:05:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 -- 172.21.15.50:0/1017816136 wait complete. 2023-10-30T12:05:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:05:58.837+0000 7f7897939640 1 librados: shutdown 2023-10-30T12:06:01.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:01.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.964+0000 7fb913a97640 10 monclient: get_monmap_and_config 2023-10-30T12:06:01.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.964+0000 7fb913a97640 10 monclient: build_initial_monmap 2023-10-30T12:06:01.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.964+0000 7fb913a97640 10 monclient: monmap: 2023-10-30T12:06:01.967 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:01.967 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:01.967 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:01.965781+0000 2023-10-30T12:06:01.967 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:01.965781+0000 2023-10-30T12:06:01.968 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:01.968 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:01.968 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:01.968 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:01.968 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:01.969 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:01.969 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:01.969 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:01.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 1 Processor -- start 2023-10-30T12:06:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 1 -- start start 2023-10-30T12:06:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 10 monclient: init 2023-10-30T12:06:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:01.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-10-30T12:06:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb90c09f5b0 0x7fb90c099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 10 monclient(hunting): picked mon.noname-a con 0x7fb90c09f5b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb90c096080 0x7fb90c096490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.965+0000 7fb913a97640 10 monclient(hunting): picked mon.noname-c con 0x7fb90c096080 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:01.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c0969d0 0x7fb90c09cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 10 monclient(hunting): picked mon.noname-b con 0x7fb90c0969d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c098810 con 0x7fb90c09f5b0 2023-10-30T12:06:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:01.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c06aef0 con 0x7fb90c096080 2023-10-30T12:06:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c09a1a0 con 0x7fb90c0969d0 2023-10-30T12:06:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb913a97640 10 monclient(hunting): authenticate will time out at 1207.149036s 2023-10-30T12:06:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb90c09f5b0 0x7fb90c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:01.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c0969d0 0x7fb90c09cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb90c09f5b0 0x7fb90c099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38598/0 (socket says 172.21.15.50:38598) 2023-10-30T12:06:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c0969d0 0x7fb90c09cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53210/0 (socket says 172.21.15.50:53210) 2023-10-30T12:06:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 -- 172.21.15.50:0/1391271374 learned_addr learned my addr 172.21.15.50:0/1391271374 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): get_auth_request con 0x7fb90c09f5b0 auth_method 0 2023-10-30T12:06:01.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 10 monclient(hunting): get_auth_request con 0x7fb90c0969d0 auth_method 0 2023-10-30T12:06:01.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:01.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:01.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:01.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:01.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:01.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:01.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 1 --2- 172.21.15.50:0/1391271374 >> v2:172.21.15.118:3300/0 conn(0x7fb90c096080 0x7fb90c096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:01.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:01.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:01.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:01.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 10 monclient(hunting): get_auth_request con 0x7fb90c096080 auth_method 0 2023-10-30T12:06:01.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:01.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:01.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:01.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:01.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:01.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911032640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:01.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient(hunting): handle_auth_done global_id 84330 payload 274 2023-10-30T12:06:01.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 -- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c0969d0 msgr2=0x7fb90c09cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:01.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 --2- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c0969d0 0x7fb90c09cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:01.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 -- 172.21.15.50:0/1391271374 >> v2:172.21.15.118:3300/0 conn(0x7fb90c096080 msgr2=0x7fb90c096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:01.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 --2- 172.21.15.50:0/1391271374 >> v2:172.21.15.118:3300/0 conn(0x7fb90c096080 0x7fb90c096490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:01.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient: _finish_hunting 0 2023-10-30T12:06:01.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 monclient: found mon.noname-a 2023-10-30T12:06:01.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:01.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:01.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb911833640 1 -- 172.21.15.50:0/1391271374 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c096e60 con 0x7fb90c09f5b0 2023-10-30T12:06:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.966+0000 7fb912034640 1 --2- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c0969d0 0x7fb90c09cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb911833640 1 --2- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c09f5b0 0x7fb90c099b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb908009850 tx=0x7fb90802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b00c6c713f6353 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 1 -- 172.21.15.50:0/1391271374 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908004230 con 0x7fb90c09f5b0 2023-10-30T12:06:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: dump: 2023-10-30T12:06:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:01.990 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:01.990 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:01.990 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:01.990 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:01.990 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:01.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: _finish_auth 0 2023-10-30T12:06:01.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: _check_auth_tickets 2023-10-30T12:06:01.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:01.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 1 -- 172.21.15.50:0/1391271374 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90802f050 con 0x7fb90c09f5b0 2023-10-30T12:06:01.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 5 monclient: authenticate success, global_id 84330 2023-10-30T12:06:01.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:01.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 1 -- 172.21.15.50:0/1391271374 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908003610 con 0x7fb90c09f5b0 2023-10-30T12:06:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 4 set_mon_vals no callback set 2023-10-30T12:06:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb910831640 10 monclient: dump: 2023-10-30T12:06:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:01.993 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:01.994 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:01.994 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:01.994 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:01.994 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:01.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:01.995 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:01.995 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:01.995 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:01.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 10 monclient: shutdown 2023-10-30T12:06:01.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 1 -- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c09f5b0 msgr2=0x7fb90c099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.967+0000 7fb913a97640 1 --2- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c09f5b0 0x7fb90c099b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb908009850 tx=0x7fb90802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:06:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 -- 172.21.15.50:0/1391271374 shutdown_connections 2023-10-30T12:06:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 --2- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c0969d0 0x7fb90c09cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 --2- 172.21.15.50:0/1391271374 >> v2:172.21.15.118:3300/0 conn(0x7fb90c096080 0x7fb90c096490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 --2- 172.21.15.50:0/1391271374 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c09f5b0 0x7fb90c099b80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 -- 172.21.15.50:0/1391271374 >> 172.21.15.50:0/1391271374 conn(0x7fb90c0a05c0 msgr2=0x7fb90c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 -- 172.21.15.50:0/1391271374 shutdown_connections 2023-10-30T12:06:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 -- 172.21.15.50:0/1391271374 wait complete. 2023-10-30T12:06:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 10 monclient: build_initial_monmap 2023-10-30T12:06:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 10 monclient: monmap: 2023-10-30T12:06:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:01.998 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:01.998 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:01.998 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:01.998 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:01.999 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:01.999 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:01.999 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:01.999 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:01.999 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 librados: starting msgr at 2023-10-30T12:06:02.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.968+0000 7fb913a97640 1 librados: starting objecter 2023-10-30T12:06:02.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 Processor -- start 2023-10-30T12:06:02.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 -- start start 2023-10-30T12:06:02.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 librados: setting wanted keys 2023-10-30T12:06:02.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 librados: calling monclient init 2023-10-30T12:06:02.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient: init 2023-10-30T12:06:02.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:06:02.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb90c096080 0x7fb90c1cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): picked mon.noname-b con 0x7fb90c096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:02.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c0969d0 0x7fb90c1cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): picked mon.noname-a con 0x7fb90c0969d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:02.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c09f5b0 0x7fb90c06fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): picked mon.noname-c con 0x7fb90c09f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:02.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c09a910 con 0x7fb90c0969d0 2023-10-30T12:06:02.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c09b4d0 con 0x7fb90c096080 2023-10-30T12:06:02.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c09c910 con 0x7fb90c09f5b0 2023-10-30T12:06:02.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb913a97640 10 monclient(hunting): authenticate will time out at 1207.152568s 2023-10-30T12:06:02.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c09f5b0 0x7fb90c06fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c0969d0 0x7fb90c1cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c0969d0 0x7fb90c1cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38608/0 (socket says 172.21.15.50:38608) 2023-10-30T12:06:02.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c09f5b0 0x7fb90c06fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53218/0 (socket says 172.21.15.50:53218) 2023-10-30T12:06:02.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 1 -- 172.21.15.50:0/3318250101 learned_addr learned my addr 172.21.15.50:0/3318250101 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:02.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 10 monclient(hunting): get_auth_request con 0x7fb90c0969d0 auth_method 0 2023-10-30T12:06:02.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): get_auth_request con 0x7fb90c09f5b0 auth_method 0 2023-10-30T12:06:02.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911032640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911833640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb90c096080 0x7fb90c1cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911833640 10 monclient(hunting): get_auth_request con 0x7fb90c096080 auth_method 0 2023-10-30T12:06:02.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911833640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911833640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb911833640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient(hunting): handle_auth_done global_id 44807 payload 824 2023-10-30T12:06:02.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 -- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb90c096080 msgr2=0x7fb90c1cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb90c096080 0x7fb90c1cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 -- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c0969d0 msgr2=0x7fb90c1cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c0969d0 0x7fb90c1cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient: _finish_hunting 0 2023-10-30T12:06:02.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 monclient: found mon.noname-c 2023-10-30T12:06:02.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:02.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:02.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.969+0000 7fb912034640 1 -- 172.21.15.50:0/3318250101 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904009a70 con 0x7fb90c09f5b0 2023-10-30T12:06:02.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb911032640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c0969d0 0x7fb90c1cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:02.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb912034640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c09f5b0 0x7fb90c06fbb0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fb90400efc0 tx=0x7fb90400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 client.?.objecter ms_handle_connect 0x7fb90c09f5b0 2023-10-30T12:06:02.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:02.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 1 -- 172.21.15.50:0/3318250101 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90400cd30 con 0x7fb90c09f5b0 2023-10-30T12:06:02.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:02.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: dump: 2023-10-30T12:06:02.013 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.013 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.013 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.014 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.014 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.014 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.014 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.014 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.015 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.015 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:02.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:02.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:02.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 1 -- 172.21.15.50:0/3318250101 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90400f040 con 0x7fb90c09f5b0 2023-10-30T12:06:02.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 5 monclient: authenticate success, global_id 44807 2023-10-30T12:06:02.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 10 monclient: _renew_subs 2023-10-30T12:06:02.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:02.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb908009460 con 0x7fb90c09f5b0 2023-10-30T12:06:02.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:02.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 1 -- 172.21.15.50:0/3318250101 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904003bd0 con 0x7fb90c09f5b0 2023-10-30T12:06:02.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:02.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb9027fc640 10 monclient: dump: 2023-10-30T12:06:02.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.018 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.018 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.018 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.019 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.019 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb903fff640 4 set_mon_vals no callback set 2023-10-30T12:06:02.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 10 client.44807.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:02.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 10 monclient: _renew_subs 2023-10-30T12:06:02.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:02.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c1b6260 con 0x7fb90c09f5b0 2023-10-30T12:06:02.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.970+0000 7fb913a97640 1 librados: init done 2023-10-30T12:06:02.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb9027fc640 1 -- 172.21.15.50:0/3318250101 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb904014480 con 0x7fb90c09f5b0 2023-10-30T12:06:02.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb913a97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb913a97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb913a97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:02.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c04e160 con 0x7fb90c09f5b0 2023-10-30T12:06:02.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb9027fc640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb8f00392f0 0x7fb8f003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb9027fc640 1 -- 172.21.15.50:0/3318250101 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6885+0+0 (secure 0 0 0) 0x7fb90404b970 con 0x7fb90c09f5b0 2023-10-30T12:06:02.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb9027fc640 10 client.44807.objecter ms_dispatch 0x7fb90c000b90 osd_map(169..169 src has 1..169) v4 2023-10-30T12:06:02.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb9027fc640 3 client.44807.objecter handle_osd_map got epochs [169,169] > 0 2023-10-30T12:06:02.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb9027fc640 3 client.44807.objecter handle_osd_map decoding full epoch 169 2023-10-30T12:06:02.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb9027fc640 20 client.44807.objecter dump_active .. 0 homeless 2023-10-30T12:06:02.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb911833640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb8f00392f0 0x7fb8f003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.971+0000 7fb911833640 10 monclient: get_auth_request con 0x7fb8f00392f0 auth_method 0 2023-10-30T12:06:02.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.972+0000 7fb9027fc640 10 client.44807.objecter ms_handle_connect 0x7fb8f00392f0 2023-10-30T12:06:02.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.972+0000 7fb911833640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb8f00392f0 0x7fb8f003b7b0 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7fb908035000 tx=0x7fb908008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.976+0000 7fb9027fc640 1 -- 172.21.15.50:0/3318250101 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb90401b950 con 0x7fb90c09f5b0 2023-10-30T12:06:02.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.976+0000 7fb9027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:01.976+0000 7fb9027fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:02.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.224+0000 7fb913a97640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:02.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.224+0000 7fb913a97640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:02.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.224+0000 7fb913a97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:02.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.224+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb90c099ee0 con 0x7fb90c09f5b0 2023-10-30T12:06:02.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.224+0000 7fb9027fc640 1 -- 172.21.15.50:0/3318250101 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fb90401b360 con 0x7fb90c09f5b0 2023-10-30T12:06:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.224+0000 7fb9027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.224+0000 7fb9027fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:02.227 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:02.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 10 librados: watch_flush enter 2023-10-30T12:06:02.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 10 librados: watch_flush exit 2023-10-30T12:06:02.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 20 client.44807.objecter shutdown clearing up homeless session... 2023-10-30T12:06:02.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 10 client.44807.objecter successfully canceled tick 2023-10-30T12:06:02.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb8f00392f0 msgr2=0x7fb8f003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb8f00392f0 0x7fb8f003b7b0 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7fb908035000 tx=0x7fb908008030 comp rx=0 tx=0).stop 2023-10-30T12:06:02.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 10 monclient: shutdown 2023-10-30T12:06:02.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:02.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c09f5b0 msgr2=0x7fb90c06fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c09f5b0 0x7fb90c06fbb0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fb90400efc0 tx=0x7fb90400c490 comp rx=0 tx=0).stop 2023-10-30T12:06:02.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 shutdown_connections 2023-10-30T12:06:02.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb8f00392f0 0x7fb8f003b7b0 unknown :-1 s=CLOSED pgs=3967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb90c09f5b0 0x7fb90c06fbb0 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb90c0969d0 0x7fb90c1cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 --2- 172.21.15.50:0/3318250101 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb90c096080 0x7fb90c1cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 >> 172.21.15.50:0/3318250101 conn(0x7fb90c0a05c0 msgr2=0x7fb90c1b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:02.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 shutdown_connections 2023-10-30T12:06:02.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 -- 172.21.15.50:0/3318250101 wait complete. 2023-10-30T12:06:02.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.226+0000 7fb913a97640 1 librados: shutdown 2023-10-30T12:06:02.238 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:06:02.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:06:02.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:06:02.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.350+0000 7fa7aa7a0640 10 monclient: get_monmap_and_config 2023-10-30T12:06:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.350+0000 7fa7aa7a0640 10 monclient: build_initial_monmap 2023-10-30T12:06:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.350+0000 7fa7aa7a0640 10 monclient: monmap: 2023-10-30T12:06:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:02.351300+0000 2023-10-30T12:06:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:02.351300+0000 2023-10-30T12:06:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:02.353 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:02.354 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:02.354 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:02.354 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.350+0000 7fa7aa7a0640 1 Processor -- start 2023-10-30T12:06:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 1 -- start start 2023-10-30T12:06:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient: init 2023-10-30T12:06:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T12:06:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7a4135950 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa7a4135460 0x7fa7a413a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a4135460 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a413ad20 0x7fa7a413d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a413ad20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a409a830 con 0x7fa7a4135950 2023-10-30T12:06:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a40a15e0 con 0x7fa7a4135460 2023-10-30T12:06:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4136440 con 0x7fa7a413ad20 2023-10-30T12:06:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7aa7a0640 10 monclient(hunting): authenticate will time out at 1207.534600s 2023-10-30T12:06:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a413ad20 0x7fa7a413d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa7a4135460 0x7fa7a413a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa7a4135460 0x7fa7a413a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38622/0 (socket says 172.21.15.50:38622) 2023-10-30T12:06:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a413ad20 0x7fa7a413d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53226/0 (socket says 172.21.15.50:53226) 2023-10-30T12:06:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 1 -- 172.21.15.50:0/3062518587 learned_addr learned my addr 172.21.15.50:0/3062518587 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): get_auth_request con 0x7fa7a4135460 auth_method 0 2023-10-30T12:06:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 10 monclient(hunting): get_auth_request con 0x7fa7a413ad20 auth_method 0 2023-10-30T12:06:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a9f9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_done global_id 84336 payload 274 2023-10-30T12:06:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 1 -- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a413ad20 msgr2=0x7fa7a413d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.351+0000 7fa7a8f9d640 1 --2- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a413ad20 0x7fa7a413d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a8f9d640 1 -- 172.21.15.50:0/3062518587 >> v1:172.21.15.118:6789/0 conn(0x7fa7a4135950 legacy=0x7fa7a4135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a8f9d640 10 monclient: _finish_hunting 0 2023-10-30T12:06:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a8f9d640 1 monclient: found mon.noname-a 2023-10-30T12:06:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a8f9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:02.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a8f9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a8f9d640 1 -- 172.21.15.50:0/3062518587 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a409a270 con 0x7fa7a4135460 2023-10-30T12:06:02.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a9f9f640 1 --2- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a413ad20 0x7fa7a413d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:02.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7a8f9d640 1 --2- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135460 0x7fa7a413a7e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa798009870 tx=0x7fa79802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7720400df01b707c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 1 -- 172.21.15.50:0/3062518587 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798030840 con 0x7fa7a4135460 2023-10-30T12:06:02.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:02.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: dump: 2023-10-30T12:06:02.373 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.373 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.373 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.375 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: _finish_auth 0 2023-10-30T12:06:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 1 -- 172.21.15.50:0/3062518587 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7980309a0 con 0x7fa7a4135460 2023-10-30T12:06:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7aa7a0640 5 monclient: authenticate success, global_id 84336 2023-10-30T12:06:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7aa7a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7aa7a0640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 1 -- 172.21.15.50:0/3062518587 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798030d30 con 0x7fa7a4135460 2023-10-30T12:06:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa7aa7a0640 4 set_mon_vals no callback set 2023-10-30T12:06:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.352+0000 7fa797fff640 10 monclient: dump: 2023-10-30T12:06:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.378 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.378 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.378 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.378 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.378 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.379 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.379 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.379 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.379 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 10 monclient: shutdown 2023-10-30T12:06:02.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:02.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135460 msgr2=0x7fa7a413a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135460 0x7fa7a413a7e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa798009870 tx=0x7fa79802b980 comp rx=0 tx=0).stop 2023-10-30T12:06:02.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/3062518587 shutdown_connections 2023-10-30T12:06:02.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a413ad20 0x7fa7a413d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/3062518587 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135460 0x7fa7a413a7e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/3062518587 >> 172.21.15.50:0/3062518587 conn(0x7fa7a40a7e70 msgr2=0x7fa7a40a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/3062518587 shutdown_connections 2023-10-30T12:06:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.353+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/3062518587 wait complete. 2023-10-30T12:06:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient: build_initial_monmap 2023-10-30T12:06:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient: monmap: 2023-10-30T12:06:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:02.383 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 librados: starting msgr at 2023-10-30T12:06:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 librados: starting objecter 2023-10-30T12:06:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 Processor -- start 2023-10-30T12:06:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 -- start start 2023-10-30T12:06:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 librados: setting wanted keys 2023-10-30T12:06:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 librados: calling monclient init 2023-10-30T12:06:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient: init 2023-10-30T12:06:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:06:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a4135460 0x7fa7a41ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a4135460 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135950 0x7fa7a4132310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a4135950 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 0x7fa7a409dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a413ad20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a411ed70 con 0x7fa7a4135950 2023-10-30T12:06:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a40a1720 con 0x7fa7a413ad20 2023-10-30T12:06:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a40a6970 con 0x7fa7a4135460 2023-10-30T12:06:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7aa7a0640 10 monclient(hunting): authenticate will time out at 1207.537926s 2023-10-30T12:06:02.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.354+0000 7fa7a979e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a4135460 0x7fa7a41ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135950 0x7fa7a4132310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a4135460 0x7fa7a41ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53230/0 (socket says 172.21.15.50:53230) 2023-10-30T12:06:02.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 1 -- 172.21.15.50:0/2988431663 learned_addr learned my addr 172.21.15.50:0/2988431663 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 10 monclient(hunting): get_auth_request con 0x7fa7a4135460 auth_method 0 2023-10-30T12:06:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): get_auth_request con 0x7fa7a4135950 auth_method 0 2023-10-30T12:06:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a9f9f640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 0x7fa7a409dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a9f9f640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 msgr2=0x7fa7a409dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:06:02.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a9f9f640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 msgr2=0x7fa7a409dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:06:02.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a9f9f640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 0x7fa7a409dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:02.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a9f9f640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 0x7fa7a409dc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:02.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient(hunting): handle_auth_done global_id 84339 payload 824 2023-10-30T12:06:02.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a4135460 msgr2=0x7fa7a41ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a4135460 0x7fa7a41ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 msgr2=0x7fa7a409dc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:02.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7a413ad20 0x7fa7a409dc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient: _finish_hunting 0 2023-10-30T12:06:02.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 monclient: found mon.noname-a 2023-10-30T12:06:02.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:02.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 -- 172.21.15.50:0/2988431663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c009a70 con 0x7fa7a4135950 2023-10-30T12:06:02.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a979e640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7a4135460 0x7fa7a41ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:02.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7a8f9d640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135950 0x7fa7a4132310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa798033000 tx=0x7fa798004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 client.?.objecter ms_handle_connect 0x7fa7a4135950 2023-10-30T12:06:02.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:02.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798039c10 con 0x7fa7a4135950 2023-10-30T12:06:02.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:02.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: dump: 2023-10-30T12:06:02.396 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.396 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.396 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.396 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.396 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.397 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.397 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.397 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.397 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.397 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: _finish_auth 0 2023-10-30T12:06:02.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:06:02.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:02.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 5 monclient: authenticate success, global_id 84339 2023-10-30T12:06:02.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa798039d70 con 0x7fa7a4135950 2023-10-30T12:06:02.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 10 monclient: _renew_subs 2023-10-30T12:06:02.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7980094d0 con 0x7fa7a4135950 2023-10-30T12:06:02.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:02.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798043640 con 0x7fa7a4135950 2023-10-30T12:06:02.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa795ffb640 10 monclient: dump: 2023-10-30T12:06:02.400 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.400 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.400 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.400 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.401 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.401 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.401 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.401 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.401 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.402 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7977fe640 4 set_mon_vals no callback set 2023-10-30T12:06:02.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 10 client.84339.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:02.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 10 monclient: _renew_subs 2023-10-30T12:06:02.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a41caeb0 con 0x7fa7a4135950 2023-10-30T12:06:02.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.355+0000 7fa7aa7a0640 1 librados: init done 2023-10-30T12:06:02.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.356+0000 7fa7aa7a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.356+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409e920 0x7fa7a402b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.356+0000 7fa7aa7a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.356+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4066190 con 0x7fa7a409e920 2023-10-30T12:06:02.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.356+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa7980437a0 con 0x7fa7a4135950 2023-10-30T12:06:02.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409e920 0x7fa7a402b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient: get_auth_request con 0x7fa7a409e920 auth_method 0 2023-10-30T12:06:02.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa7880392f0 0x7fa78803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6846+0+0 (secure 0 0 0) 0x7fa79803e070 con 0x7fa7a4135950 2023-10-30T12:06:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 client.84339.objecter ms_dispatch 0x7fa7a4000b90 osd_map(170..170 src has 1..170) v4 2023-10-30T12:06:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 3 client.84339.objecter handle_osd_map got epochs [170,170] > 0 2023-10-30T12:06:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 3 client.84339.objecter handle_osd_map decoding full epoch 170 2023-10-30T12:06:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 10 monclient(hunting): handle_auth_done global_id 84342 payload 824 2023-10-30T12:06:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 1 -- 172.21.15.50:0/2988431663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7a4099f90 con 0x7fa7a409e920 2023-10-30T12:06:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 20 client.84339.objecter dump_active .. 0 homeless 2023-10-30T12:06:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a9f9f640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa7880392f0 0x7fa78803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a9f9f640 10 monclient: get_auth_request con 0x7fa7880392f0 auth_method 0 2023-10-30T12:06:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 client.84339.objecter ms_handle_connect 0x7fa7a409e920 2023-10-30T12:06:02.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 client.84339.objecter resend_mon_ops 2023-10-30T12:06:02.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a979e640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409e920 0x7fa7a402b430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000fa30 tx=0x7fa7a000fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a00052e0 con 0x7fa7a409e920 2023-10-30T12:06:02.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 monclient: dump: 2023-10-30T12:06:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 client.84339.objecter ms_handle_connect 0x7fa7880392f0 2023-10-30T12:06:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa7a9f9f640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa7880392f0 0x7fa78803b7b0 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 crypto rx=0x7fa7a409ffe0 tx=0x7fa79c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa7a0005440 con 0x7fa7a409e920 2023-10-30T12:06:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409e920 msgr2=0x7fa7a402b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409e920 0x7fa7a402b430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000fa30 tx=0x7fa7a000fef0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409e920 msgr2=0x7fa7a402b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.357+0000 7fa795ffb640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409e920 0x7fa7a402b430 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000fa30 tx=0x7fa7a000fef0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7aa7a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:02.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409ed00 0x7fa7a402b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7aa7a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a411e840 con 0x7fa7a409ed00 2023-10-30T12:06:02.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409ed00 0x7fa7a402b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient: get_auth_request con 0x7fa7a409ed00 auth_method 0 2023-10-30T12:06:02.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 10 monclient(hunting): handle_auth_done global_id 84345 payload 824 2023-10-30T12:06:02.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 1 -- 172.21.15.50:0/2988431663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa7a4139cb0 con 0x7fa7a409ed00 2023-10-30T12:06:02.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa795ffb640 10 client.84339.objecter ms_handle_connect 0x7fa7a409ed00 2023-10-30T12:06:02.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa795ffb640 10 client.84339.objecter resend_mon_ops 2023-10-30T12:06:02.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa7a979e640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409ed00 0x7fa7a402b970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0000fb0 tx=0x7fa7a0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0006be0 con 0x7fa7a409ed00 2023-10-30T12:06:02.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa795ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.364+0000 7fa795ffb640 10 monclient: dump: 2023-10-30T12:06:02.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.418 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.420 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fa7a0006d40 con 0x7fa7a409ed00 2023-10-30T12:06:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa795ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa795ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409ed00 msgr2=0x7fa7a402b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa795ffb640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409ed00 0x7fa7a402b970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0000fb0 tx=0x7fa7a0001120 comp rx=0 tx=0).stop 2023-10-30T12:06:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa795ffb640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409ed00 msgr2=0x7fa7a402b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa795ffb640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a409ed00 0x7fa7a402b970 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0000fb0 tx=0x7fa7a0001120 comp rx=0 tx=0).stop 2023-10-30T12:06:02.421 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:02.422 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:02.422 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:02.422 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:06:02.422 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 56, 2023-10-30T12:06:02.422 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:02.423 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:02.423 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:02.423 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:02.423 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.423 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:06:02.424 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:02.424 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:02.424 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:02.424 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.424 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.424 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.425 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.425 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.425 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.425 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.426 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.426 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.426 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.426 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:02.426 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:02.427 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.427 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.427 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.427 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.428 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.428 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.428 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.428 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.428 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:02.430 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:02.430 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:02.430 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:02.430 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:02.430 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:02.431 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:02.431 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:02.431 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:02.431 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:02.431 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:02.431 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:02.432 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:02.432 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:02.432 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.432 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.432 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.433 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.433 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.433 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.433 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.433 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.434 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.434 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.434 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:02.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.434 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:02.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.435 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:02.435 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:02.435 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.436 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:02.436 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:02.438 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:02.438 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.438 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.439 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.439 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:02.439 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:02.439 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:02.440 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.441 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:02.441 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:02.442 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:02.443 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.443 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.443 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.443 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:02.444 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:02.444 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.444 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:02.445 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:02.446 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.446 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.446 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:02.447 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:02.447 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.447 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.447 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.448 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:02.448 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:02.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.448 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.449 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.449 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:02.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.449 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.449 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:02.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.450 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.450 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.450 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:06:02.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.451 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.451 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:06:02.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.451 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.451 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.451 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:02.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.452 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:02.452 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:02.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 10 librados: watch_flush enter 2023-10-30T12:06:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 10 librados: watch_flush exit 2023-10-30T12:06:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 20 client.84339.objecter shutdown clearing up homeless session... 2023-10-30T12:06:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 10 client.84339.objecter successfully canceled tick 2023-10-30T12:06:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa7880392f0 msgr2=0x7fa78803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa7880392f0 0x7fa78803b7b0 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 crypto rx=0x7fa7a409ffe0 tx=0x7fa79c00b040 comp rx=0 tx=0).stop 2023-10-30T12:06:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 10 monclient: shutdown 2023-10-30T12:06:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135950 msgr2=0x7fa7a4132310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135950 0x7fa7a4132310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa798033000 tx=0x7fa798004730 comp rx=0 tx=0).stop 2023-10-30T12:06:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7a9f9f640 1 -- 172.21.15.50:0/2988431663 reap_dead start 2023-10-30T12:06:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 shutdown_connections 2023-10-30T12:06:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 --2- 172.21.15.50:0/2988431663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7a4135950 0x7fa7a4132310 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 >> 172.21.15.50:0/2988431663 conn(0x7fa7a40a7e70 msgr2=0x7fa7a4133f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 shutdown_connections 2023-10-30T12:06:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 -- 172.21.15.50:0/2988431663 wait complete. 2023-10-30T12:06:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.365+0000 7fa7aa7a0640 1 librados: shutdown 2023-10-30T12:06:02.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:02.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:06:02.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:06:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.562+0000 7f0a6e0e0640 10 monclient: get_monmap_and_config 2023-10-30T12:06:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.562+0000 7f0a6e0e0640 10 monclient: build_initial_monmap 2023-10-30T12:06:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.562+0000 7f0a6e0e0640 10 monclient: monmap: 2023-10-30T12:06:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:02.564000+0000 2023-10-30T12:06:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:02.564000+0000 2023-10-30T12:06:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:02.567 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.563+0000 7f0a6e0e0640 1 Processor -- start 2023-10-30T12:06:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.563+0000 7f0a6e0e0640 1 -- start start 2023-10-30T12:06:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.563+0000 7f0a6e0e0640 10 monclient: init 2023-10-30T12:06:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-10-30T12:06:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a6806afc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a6806b640 0x7f0a680a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient(hunting): picked mon.noname-a con 0x7f0a6806b640 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0a6813b070 0x7f0a6813d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6813b070 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6806b3a0 con 0x7f0a6806afc0 2023-10-30T12:06:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6809e540 con 0x7f0a6806b640 2023-10-30T12:06:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6809fa10 con 0x7f0a6813b070 2023-10-30T12:06:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a6e0e0640 10 monclient(hunting): authenticate will time out at 1207.747206s 2023-10-30T12:06:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a6806b640 0x7f0a680a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a6806b640 0x7f0a680a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38664/0 (socket says 172.21.15.50:38664) 2023-10-30T12:06:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 -- 172.21.15.50:0/3588745177 learned_addr learned my addr 172.21.15.50:0/3588745177 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): get_auth_request con 0x7f0a6806b640 auth_method 0 2023-10-30T12:06:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a67fff640 1 --2- 172.21.15.50:0/3588745177 >> v2:172.21.15.118:3300/0 conn(0x7f0a6813b070 0x7f0a6813d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_done global_id 84348 payload 274 2023-10-30T12:06:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 -- 172.21.15.50:0/3588745177 >> v2:172.21.15.118:3300/0 conn(0x7f0a6813b070 msgr2=0x7f0a6813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 --2- 172.21.15.50:0/3588745177 >> v2:172.21.15.118:3300/0 conn(0x7f0a6813b070 0x7f0a6813d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 -- 172.21.15.50:0/3588745177 >> v1:172.21.15.118:6789/0 conn(0x7f0a6806afc0 legacy=0x7f0a680a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:02.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:02.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 monclient: found mon.noname-a 2023-10-30T12:06:02.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:02.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.564+0000 7f0a66ffd640 1 -- 172.21.15.50:0/3588745177 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6806a6f0 con 0x7f0a6806b640 2023-10-30T12:06:02.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a66ffd640 1 --2- 172.21.15.50:0/3588745177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a680a23a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a6806a180 tx=0x7f0a5c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c12e31247bcbb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 1 -- 172.21.15.50:0/3588745177 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c031740 con 0x7f0a6806b640 2023-10-30T12:06:02.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:02.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: dump: 2023-10-30T12:06:02.583 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.583 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.583 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.583 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.584 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 1 -- 172.21.15.50:0/3588745177 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c0318a0 con 0x7f0a6806b640 2023-10-30T12:06:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 5 monclient: authenticate success, global_id 84348 2023-10-30T12:06:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 4 set_mon_vals no callback set 2023-10-30T12:06:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 1 -- 172.21.15.50:0/3588745177 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c031c30 con 0x7f0a6806b640 2023-10-30T12:06:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a667fc640 10 monclient: dump: 2023-10-30T12:06:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.589 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 10 monclient: shutdown 2023-10-30T12:06:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/3588745177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 msgr2=0x7f0a680a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.565+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/3588745177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a680a23a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a6806a180 tx=0x7f0a5c02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/3588745177 shutdown_connections 2023-10-30T12:06:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/3588745177 >> v2:172.21.15.118:3300/0 conn(0x7f0a6813b070 0x7f0a6813d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/3588745177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a680a23a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/3588745177 >> 172.21.15.50:0/3588745177 conn(0x7f0a680a6200 msgr2=0x7f0a680a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/3588745177 shutdown_connections 2023-10-30T12:06:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/3588745177 wait complete. 2023-10-30T12:06:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 10 monclient: build_initial_monmap 2023-10-30T12:06:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 10 monclient: monmap: 2023-10-30T12:06:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:02.593 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 librados: starting msgr at 2023-10-30T12:06:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 librados: starting objecter 2023-10-30T12:06:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.566+0000 7f0a6e0e0640 1 Processor -- start 2023-10-30T12:06:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 -- start start 2023-10-30T12:06:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 librados: setting wanted keys 2023-10-30T12:06:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 librados: calling monclient init 2023-10-30T12:06:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient: init 2023-10-30T12:06:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:06:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6806afc0 0x7f0a681ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6806afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a6811c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): picked mon.noname-a con 0x7f0a6806b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a6813b070 0x7f0a6811cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): picked mon.noname-b con 0x7f0a6813b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a681c0160 con 0x7f0a6806b640 2023-10-30T12:06:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a681200b0 con 0x7f0a6813b070 2023-10-30T12:06:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a681385e0 con 0x7f0a6806afc0 2023-10-30T12:06:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a6e0e0640 10 monclient(hunting): authenticate will time out at 1207.750556s 2023-10-30T12:06:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a6811c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6806afc0 0x7f0a681ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a6811c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38666/0 (socket says 172.21.15.50:38666) 2023-10-30T12:06:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 -- 172.21.15.50:0/974354766 learned_addr learned my addr 172.21.15.50:0/974354766 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): get_auth_request con 0x7f0a6806b640 auth_method 0 2023-10-30T12:06:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 10 monclient(hunting): get_auth_request con 0x7f0a6806afc0 auth_method 0 2023-10-30T12:06:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a67fff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a6813b070 0x7f0a6811cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a67fff640 10 monclient(hunting): get_auth_request con 0x7f0a6813b070 auth_method 0 2023-10-30T12:06:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a67fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a67fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient(hunting): handle_auth_done global_id 84351 payload 824 2023-10-30T12:06:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6806afc0 msgr2=0x7f0a681ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6806afc0 0x7f0a681ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a6813b070 msgr2=0x7f0a6811cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a6813b070 0x7f0a6811cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 monclient: found mon.noname-a 2023-10-30T12:06:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a66ffd640 1 -- 172.21.15.50:0/974354766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c0094d0 con 0x7f0a6806b640 2023-10-30T12:06:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.567+0000 7f0a677fe640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6806afc0 0x7f0a681ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a66ffd640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a6811c600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c000c00 tx=0x7f0a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 client.?.objecter ms_handle_connect 0x7f0a6806b640 2023-10-30T12:06:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c033d60 con 0x7f0a6806b640 2023-10-30T12:06:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: dump: 2023-10-30T12:06:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.607 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: _finish_auth 0 2023-10-30T12:06:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:02.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:02.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c03c400 con 0x7f0a6806b640 2023-10-30T12:06:02.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 5 monclient: authenticate success, global_id 84351 2023-10-30T12:06:02.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:02.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c03c7b0 con 0x7f0a6806b640 2023-10-30T12:06:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a4ffff640 10 monclient: dump: 2023-10-30T12:06:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.611 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 10 monclient: _renew_subs 2023-10-30T12:06:02.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a65ffb640 4 set_mon_vals no callback set 2023-10-30T12:06:02.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a68120670 con 0x7f0a6806b640 2023-10-30T12:06:02.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 10 client.84351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:02.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 10 monclient: _renew_subs 2023-10-30T12:06:02.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6811fdd0 con 0x7f0a6806b640 2023-10-30T12:06:02.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.568+0000 7f0a6e0e0640 1 librados: init done 2023-10-30T12:06:02.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a6e0e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0a5c03a040 con 0x7f0a6806b640 2023-10-30T12:06:02.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a681cadb0 0x7f0a6811d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a6e0e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a68073ff0 con 0x7f0a681cadb0 2023-10-30T12:06:02.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a681cadb0 0x7f0a6811d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient: get_auth_request con 0x7f0a681cadb0 auth_method 0 2023-10-30T12:06:02.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a540392f0 0x7f0a5403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6846+0+0 (secure 0 0 0) 0x7f0a5c06cc20 con 0x7f0a6806b640 2023-10-30T12:06:02.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 10 client.84351.objecter ms_dispatch 0x7f0a68000b90 osd_map(170..170 src has 1..170) v4 2023-10-30T12:06:02.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 3 client.84351.objecter handle_osd_map got epochs [170,170] > 0 2023-10-30T12:06:02.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 3 client.84351.objecter handle_osd_map decoding full epoch 170 2023-10-30T12:06:02.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 10 monclient(hunting): handle_auth_done global_id 44819 payload 824 2023-10-30T12:06:02.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 20 client.84351.objecter dump_active .. 0 homeless 2023-10-30T12:06:02.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 1 -- 172.21.15.50:0/974354766 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a6809f0b0 con 0x7f0a681cadb0 2023-10-30T12:06:02.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a67fff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a540392f0 0x7f0a5403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a67fff640 10 monclient: get_auth_request con 0x7f0a540392f0 auth_method 0 2023-10-30T12:06:02.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 10 client.84351.objecter ms_handle_connect 0x7f0a681cadb0 2023-10-30T12:06:02.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a677fe640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a681cadb0 0x7f0a6811d080 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f0a5800fb10 tx=0x7f0a58001cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 10 client.84351.objecter resend_mon_ops 2023-10-30T12:06:02.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a580052e0 con 0x7f0a681cadb0 2023-10-30T12:06:02.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:02.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.569+0000 7f0a4ffff640 10 monclient: dump: 2023-10-30T12:06:02.618 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.619 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.619 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.619 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.619 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.619 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.620 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.620 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.620 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.620 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 10 client.84351.objecter ms_handle_connect 0x7f0a540392f0 2023-10-30T12:06:02.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a67fff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a540392f0 0x7f0a5403b7b0 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7f0a6811fb10 tx=0x7f0a600074b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0a58005440 con 0x7f0a681cadb0 2023-10-30T12:06:02.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:02.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a681cadb0 msgr2=0x7f0a6811d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a681cadb0 0x7f0a6811d080 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f0a5800fb10 tx=0x7f0a58001cf0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a681cadb0 msgr2=0x7f0a6811d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:02.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.570+0000 7f0a4ffff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a681cadb0 0x7f0a6811d080 secure :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f0a5800fb10 tx=0x7f0a58001cf0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a6e0e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:02.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6811d5c0 0x7f0a681352d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a6e0e0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a68120800 con 0x7f0a6811d5c0 2023-10-30T12:06:02.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6811d5c0 0x7f0a681352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient: get_auth_request con 0x7f0a6811d5c0 auth_method 0 2023-10-30T12:06:02.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 10 monclient(hunting): handle_auth_done global_id 44822 payload 824 2023-10-30T12:06:02.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 1 -- 172.21.15.50:0/974354766 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a6809df80 con 0x7f0a6811d5c0 2023-10-30T12:06:02.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a4ffff640 10 client.84351.objecter ms_handle_connect 0x7f0a6811d5c0 2023-10-30T12:06:02.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a4ffff640 10 client.84351.objecter resend_mon_ops 2023-10-30T12:06:02.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.576+0000 7f0a677fe640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6811d5c0 0x7f0a681352d0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f0a58017000 tx=0x7f0a58001020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58024070 con 0x7f0a6811d5c0 2023-10-30T12:06:02.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:02.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 10 monclient: dump: 2023-10-30T12:06:02.628 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.628 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.628 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.628 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.628 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.629 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.629 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.629 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.629 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f0a58003e50 con 0x7f0a6811d5c0 2023-10-30T12:06:02.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:02.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:02.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6811d5c0 msgr2=0x7f0a681352d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6811d5c0 0x7f0a681352d0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f0a58017000 tx=0x7f0a58001020 comp rx=0 tx=0).stop 2023-10-30T12:06:02.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6811d5c0 msgr2=0x7f0a681352d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:02.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a4ffff640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a6811d5c0 0x7f0a681352d0 secure :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f0a58017000 tx=0x7f0a58001020 comp rx=0 tx=0).stop 2023-10-30T12:06:02.631 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:02.631 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:02.632 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:02.632 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:02.632 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 56, 2023-10-30T12:06:02.632 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:02.632 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:02.632 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:02.633 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:02.633 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.633 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:06:02.633 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:02.633 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:02.634 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:02.634 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.634 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.634 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.634 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.635 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.635 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.635 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.635 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.635 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.636 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.636 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.636 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:02.636 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:02.636 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.637 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.637 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.637 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.637 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.638 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.638 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.638 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.638 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.638 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.639 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:02.639 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:02.639 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:02.639 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:02.639 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:02.640 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:02.640 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:02.640 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:02.640 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:02.640 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:02.641 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:02.641 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:02.641 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:02.641 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:02.641 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:02.642 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:02.642 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:02.642 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.642 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.642 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.643 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.643 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.643 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.643 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.643 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.644 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.644 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.644 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:02.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.644 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.646 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.646 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:02.646 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.647 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:02.647 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.647 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.648 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:02.648 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:02.648 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.648 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.648 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.649 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:02.649 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:02.649 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.650 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.650 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.650 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:02.650 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.651 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.651 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:02.651 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.651 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.652 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.652 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:02.652 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:02.652 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.653 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.653 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.653 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:02.654 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:02.654 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.654 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.654 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.655 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:02.655 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.655 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.655 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:02.656 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.656 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.656 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:06:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:06:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:06:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:02.662 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 10 librados: watch_flush enter 2023-10-30T12:06:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 10 librados: watch_flush exit 2023-10-30T12:06:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 20 client.84351.objecter shutdown clearing up homeless session... 2023-10-30T12:06:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 10 client.84351.objecter successfully canceled tick 2023-10-30T12:06:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a540392f0 msgr2=0x7f0a5403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a540392f0 0x7f0a5403b7b0 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7f0a6811fb10 tx=0x7f0a600074b0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 10 monclient: shutdown 2023-10-30T12:06:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 msgr2=0x7f0a6811c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a6811c600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c000c00 tx=0x7f0a5c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a67fff640 1 -- 172.21.15.50:0/974354766 reap_dead start 2023-10-30T12:06:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 shutdown_connections 2023-10-30T12:06:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 --2- 172.21.15.50:0/974354766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a6806b640 0x7f0a6811c600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 >> 172.21.15.50:0/974354766 conn(0x7f0a680a6200 msgr2=0x7f0a680a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 shutdown_connections 2023-10-30T12:06:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 -- 172.21.15.50:0/974354766 wait complete. 2023-10-30T12:06:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.577+0000 7f0a6e0e0640 1 librados: shutdown 2023-10-30T12:06:02.667 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:02.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:06:02.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:06:02.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.773+0000 7f5c117e2640 10 monclient: get_monmap_and_config 2023-10-30T12:06:02.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.773+0000 7f5c117e2640 10 monclient: build_initial_monmap 2023-10-30T12:06:02.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.773+0000 7f5c117e2640 10 monclient: monmap: 2023-10-30T12:06:02.775 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:02.775 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.775 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:02.774174+0000 2023-10-30T12:06:02.775 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:02.774174+0000 2023-10-30T12:06:02.776 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:02.776 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.776 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:02.776 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:02.776 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:02.776 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:02.777 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:02.777 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.773+0000 7f5c117e2640 1 Processor -- start 2023-10-30T12:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.773+0000 7f5c117e2640 1 -- start start 2023-10-30T12:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.773+0000 7f5c117e2640 10 monclient: init 2023-10-30T12:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-10-30T12:06:02.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5c0c0a2ae0 0x7f5c0c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0c0a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:02.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c0c0a2ec0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:02.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5c0c135df0 0x7f5c0c13da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0c135df0 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:02.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0c0a4190 con 0x7f5c0c0a2ec0 2023-10-30T12:06:02.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0c0a2990 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0c0a1630 con 0x7f5c0c135df0 2023-10-30T12:06:02.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c117e2640 10 monclient(hunting): authenticate will time out at 1207.957274s 2023-10-30T12:06:02.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5c0c0a2ae0 0x7f5c0c0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5c0c0a2ae0 0x7f5c0c0a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38672/0 (socket says 172.21.15.50:38672) 2023-10-30T12:06:02.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 -- 172.21.15.50:0/929294987 learned_addr learned my addr 172.21.15.50:0/929294987 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:02.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): get_auth_request con 0x7f5c0c0a2ae0 auth_method 0 2023-10-30T12:06:02.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 1 --2- 172.21.15.50:0/929294987 >> v2:172.21.15.118:3300/0 conn(0x7f5c0c135df0 0x7f5c0c13da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 10 monclient(hunting): get_auth_request con 0x7f5c0c135df0 auth_method 0 2023-10-30T12:06:02.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient(hunting): handle_auth_done global_id 84354 payload 274 2023-10-30T12:06:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 -- 172.21.15.50:0/929294987 >> v2:172.21.15.118:3300/0 conn(0x7f5c0c135df0 msgr2=0x7f5c0c13da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 --2- 172.21.15.50:0/929294987 >> v2:172.21.15.118:3300/0 conn(0x7f5c0c135df0 0x7f5c0c13da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 -- 172.21.15.50:0/929294987 >> v1:172.21.15.118:6789/0 conn(0x7f5c0c0a2ec0 legacy=0x7f5c0c1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 monclient: found mon.noname-a 2023-10-30T12:06:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.774+0000 7f5c0affd640 1 -- 172.21.15.50:0/929294987 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c09a2e0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c0affd640 1 --2- 172.21.15.50:0/929294987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c0a2450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0a54d0 tx=0x7f5bfc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b738150aac4d4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 1 -- 172.21.15.50:0/929294987 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc0317e0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: dump: 2023-10-30T12:06:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.797 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: _finish_auth 0 2023-10-30T12:06:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:06:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c117e2640 5 monclient: authenticate success, global_id 84354 2023-10-30T12:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 1 -- 172.21.15.50:0/929294987 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc031940 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c117e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c117e2640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c117e2640 4 set_mon_vals no callback set 2023-10-30T12:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 1 -- 172.21.15.50:0/929294987 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc031cd0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.775+0000 7f5c09ffb640 10 monclient: dump: 2023-10-30T12:06:02.799 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.801 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 10 monclient: shutdown 2023-10-30T12:06:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 -- 172.21.15.50:0/929294987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 msgr2=0x7f5c0c0a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 --2- 172.21.15.50:0/929294987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c0a2450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0a54d0 tx=0x7f5bfc02b880 comp rx=0 tx=0).stop 2023-10-30T12:06:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 -- 172.21.15.50:0/929294987 shutdown_connections 2023-10-30T12:06:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 --2- 172.21.15.50:0/929294987 >> v2:172.21.15.118:3300/0 conn(0x7f5c0c135df0 0x7f5c0c13da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 --2- 172.21.15.50:0/929294987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c0a2450 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 -- 172.21.15.50:0/929294987 >> 172.21.15.50:0/929294987 conn(0x7f5c0c0a80b0 msgr2=0x7f5c0c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 -- 172.21.15.50:0/929294987 shutdown_connections 2023-10-30T12:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 1 -- 172.21.15.50:0/929294987 wait complete. 2023-10-30T12:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 10 monclient: build_initial_monmap 2023-10-30T12:06:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.776+0000 7f5c117e2640 10 monclient: monmap: 2023-10-30T12:06:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.805 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:02.805 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:02.805 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:02.805 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 librados: starting msgr at 2023-10-30T12:06:02.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 librados: starting objecter 2023-10-30T12:06:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 Processor -- start 2023-10-30T12:06:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 -- start start 2023-10-30T12:06:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 librados: setting wanted keys 2023-10-30T12:06:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 librados: calling monclient init 2023-10-30T12:06:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient: init 2023-10-30T12:06:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:06:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c1ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0c0a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c0a2ec0 0x7f5c0c1ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5c0c0a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c0c135df0 0x7f5c0c11f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0c135df0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c122fb0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c135630 con 0x7f5c0c0a2ec0 2023-10-30T12:06:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c121a30 con 0x7f5c0c135df0 2023-10-30T12:06:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c117e2640 10 monclient(hunting): authenticate will time out at 1207.960902s 2023-10-30T12:06:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.777+0000 7f5c0affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c1ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c1ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38686/0 (socket says 172.21.15.50:38686) 2023-10-30T12:06:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 -- 172.21.15.50:0/1620718088 learned_addr learned my addr 172.21.15.50:0/1620718088 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): get_auth_request con 0x7f5c0c0a2ae0 auth_method 0 2023-10-30T12:06:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c0c135df0 0x7f5c0c11f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c0a2ec0 0x7f5c0c1ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 10 monclient(hunting): get_auth_request con 0x7f5c0c0a2ec0 auth_method 0 2023-10-30T12:06:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 10 monclient(hunting): get_auth_request con 0x7f5c0c135df0 auth_method 0 2023-10-30T12:06:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient(hunting): handle_auth_done global_id 84357 payload 824 2023-10-30T12:06:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c0c135df0 msgr2=0x7f5c0c11f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c0c135df0 0x7f5c0c11f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c0a2ec0 msgr2=0x7f5c0c1ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c0a2ec0 0x7f5c0c1ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 monclient: found mon.noname-a 2023-10-30T12:06:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 -- 172.21.15.50:0/1620718088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0094d0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c0affd640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c1ca2b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc000c00 tx=0x7f5bfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 client.?.objecter ms_handle_connect 0x7f5c0c0a2ae0 2023-10-30T12:06:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc032880 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: dump: 2023-10-30T12:06:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.819 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.819 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.819 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.819 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.819 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: _finish_auth 0 2023-10-30T12:06:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:06:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc0329e0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc032d90 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5bf37fe640 10 monclient: dump: 2023-10-30T12:06:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.823 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 5 monclient: authenticate success, global_id 84357 2023-10-30T12:06:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 10 monclient: _renew_subs 2023-10-30T12:06:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0c1220c0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c097fa640 4 set_mon_vals no callback set 2023-10-30T12:06:02.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 10 client.84357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:02.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 10 monclient: _renew_subs 2023-10-30T12:06:02.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:02.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0c0a17f0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.778+0000 7f5c117e2640 1 librados: init done 2023-10-30T12:06:02.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5c117e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5c117e2640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c11fd60 0x7f5c0c120140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5c117e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5bfc0453f0 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c073ff0 con 0x7f5c0c11fd60 2023-10-30T12:06:02.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5bf37fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5bf80392f0 0x7f5bf803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6846+0+0 (secure 0 0 0) 0x7f5bfc06d020 con 0x7f5c0c0a2ae0 2023-10-30T12:06:02.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5bf37fe640 10 client.84357.objecter ms_dispatch 0x7f5c0c000b90 osd_map(170..170 src has 1..170) v4 2023-10-30T12:06:02.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5bf37fe640 3 client.84357.objecter handle_osd_map got epochs [170,170] > 0 2023-10-30T12:06:02.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5bf37fe640 3 client.84357.objecter handle_osd_map decoding full epoch 170 2023-10-30T12:06:02.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5bf37fe640 20 client.84357.objecter dump_active .. 0 homeless 2023-10-30T12:06:02.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.779+0000 7f5c0a7fc640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c11fd60 0x7f5c0c120140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient: get_auth_request con 0x7f5c0c11fd60 auth_method 0 2023-10-30T12:06:02.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0b7fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5bf80392f0 0x7f5bf803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0b7fe640 10 monclient: get_auth_request con 0x7f5bf80392f0 auth_method 0 2023-10-30T12:06:02.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_done global_id 94123 payload 824 2023-10-30T12:06:02.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 1 -- 172.21.15.50:0/1620718088 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c0c09a5c0 con 0x7f5c0c11fd60 2023-10-30T12:06:02.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5bf37fe640 10 client.84357.objecter ms_handle_connect 0x7f5bf80392f0 2023-10-30T12:06:02.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0b7fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5bf80392f0 0x7f5bf803b7b0 secure :-1 s=READY pgs=3970 cs=0 l=1 rev1=1 crypto rx=0x7f5c040113e0 tx=0x7f5c0400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5bf37fe640 10 client.84357.objecter ms_handle_connect 0x7f5c0c11fd60 2023-10-30T12:06:02.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5bf37fe640 10 client.84357.objecter resend_mon_ops 2023-10-30T12:06:02.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5c0a7fc640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c11fd60 0x7f5c0c120140 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000f60 tx=0x7f5c0000fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c000050d0 con 0x7f5c0c11fd60 2023-10-30T12:06:02.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5bf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:02.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.780+0000 7f5bf37fe640 10 monclient: dump: 2023-10-30T12:06:02.831 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.831 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.831 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.832 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.832 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.832 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.832 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.832 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.833 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.833 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.781+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5c00005230 con 0x7f5c0c11fd60 2023-10-30T12:06:02.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.781+0000 7f5bf37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:02.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.781+0000 7f5bf37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:02.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.781+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c11fd60 msgr2=0x7f5c0c120140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.781+0000 7f5bf37fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c11fd60 0x7f5c0c120140 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000f60 tx=0x7f5c0000fc00 comp rx=0 tx=0).stop 2023-10-30T12:06:02.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.781+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c11fd60 msgr2=0x7f5c0c120140 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:02.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.781+0000 7f5bf37fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c11fd60 0x7f5c0c120140 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000f60 tx=0x7f5c0000fc00 comp rx=0 tx=0).stop 2023-10-30T12:06:02.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c117e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:02.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c117e2640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c120680 0x7f5c0c09dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c117e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c099b40 con 0x7f5c0c120680 2023-10-30T12:06:02.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c120680 0x7f5c0c09dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient: get_auth_request con 0x7f5c0c120680 auth_method 0 2023-10-30T12:06:02.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:02.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:02.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:02.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:02.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.787+0000 7f5c0a7fc640 10 monclient(hunting): handle_auth_done global_id 94126 payload 824 2023-10-30T12:06:02.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5c0a7fc640 1 -- 172.21.15.50:0/1620718088 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5c0c06b060 con 0x7f5c0c120680 2023-10-30T12:06:02.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5c0a7fc640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c120680 0x7f5c0c09dde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000fb0 tx=0x7f5c00001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:02.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 10 client.84357.objecter ms_handle_connect 0x7f5c0c120680 2023-10-30T12:06:02.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 10 client.84357.objecter resend_mon_ops 2023-10-30T12:06:02.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00024070 con 0x7f5c0c120680 2023-10-30T12:06:02.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:02.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:02.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 10 monclient: dump: 2023-10-30T12:06:02.840 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:02.840 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.840 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.841 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:02.841 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:02.841 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.841 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:02.841 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:02.842 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:02.842 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7f5c00007300 con 0x7f5c0c120680 2023-10-30T12:06:02.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:02.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:02.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c120680 msgr2=0x7f5c0c09dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c120680 0x7f5c0c09dde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000fb0 tx=0x7f5c00001120 comp rx=0 tx=0).stop 2023-10-30T12:06:02.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c120680 msgr2=0x7f5c0c09dde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:02.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.788+0000 7f5bf37fe640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c0c120680 0x7f5c0c09dde0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.843 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:02.844 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:02.844 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:02.844 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:02.844 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 56, 2023-10-30T12:06:02.845 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:02.845 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:02.845 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:02.845 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:02.845 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.845 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:06:02.846 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:02.846 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:02.846 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:02.846 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.846 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.847 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.847 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.847 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.847 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.847 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.848 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.848 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.848 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.848 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.848 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:02.849 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:02.849 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.849 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.849 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.849 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.849 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.850 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.850 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.850 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.850 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.850 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.851 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.851 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:02.851 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:02.851 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:02.852 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:02.852 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:02.852 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:02.852 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:02.852 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:02.853 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:02.853 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:02.853 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:02.853 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:02.853 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:02.853 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:02.854 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:02.854 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:02.854 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:02.854 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:02.854 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:02.855 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:02.855 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:02.855 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:02.855 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:02.855 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:02.856 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:02.856 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:02.856 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:02.856 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.856 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:02.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.857 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:02.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.857 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:02.857 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:02.857 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.858 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.858 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.858 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:02.858 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.859 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.859 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:02.859 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.860 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.860 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:02.860 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:02.860 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.861 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.861 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.861 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:02.862 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:02.862 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.862 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.862 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.863 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:02.863 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.863 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.863 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:02.865 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:02.865 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.865 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.865 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.866 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:02.866 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:02.866 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:02.866 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:02.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.867 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:02.867 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:02.867 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.867 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:02.868 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:02.868 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:02.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.868 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:02.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.870 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.870 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:02.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:02.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.871 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.871 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.871 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:02.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.871 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:02.871 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:02.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:02.872 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:02.872 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:02.872 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:06:02.872 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:02.873 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:02.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:02.873 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:02.873 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:02.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 10 librados: watch_flush enter 2023-10-30T12:06:02.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 10 librados: watch_flush exit 2023-10-30T12:06:02.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 20 client.84357.objecter shutdown clearing up homeless session... 2023-10-30T12:06:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 10 client.84357.objecter successfully canceled tick 2023-10-30T12:06:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5bf80392f0 msgr2=0x7f5bf803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5bf80392f0 0x7f5bf803b7b0 secure :-1 s=READY pgs=3970 cs=0 l=1 rev1=1 crypto rx=0x7f5c040113e0 tx=0x7f5c0400b040 comp rx=0 tx=0).stop 2023-10-30T12:06:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 10 monclient: shutdown 2023-10-30T12:06:02.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 msgr2=0x7f5c0c1ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 --2- 172.21.15.50:0/1620718088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c0c0a2ae0 0x7f5c0c1ca2b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc000c00 tx=0x7f5bfc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c0b7fe640 1 -- 172.21.15.50:0/1620718088 reap_dead start 2023-10-30T12:06:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c0b7fe640 1 -- 172.21.15.50:0/1620718088 reap_dead start 2023-10-30T12:06:02.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 shutdown_connections 2023-10-30T12:06:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 >> 172.21.15.50:0/1620718088 conn(0x7f5c0c0a80b0 msgr2=0x7f5c0c0a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 shutdown_connections 2023-10-30T12:06:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 -- 172.21.15.50:0/1620718088 wait complete. 2023-10-30T12:06:02.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.789+0000 7f5c117e2640 1 librados: shutdown 2023-10-30T12:06:02.879 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:02.879 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:06:02.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:06:02.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.982+0000 7f6476ab0640 10 monclient: get_monmap_and_config 2023-10-30T12:06:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.982+0000 7f6476ab0640 10 monclient: build_initial_monmap 2023-10-30T12:06:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.982+0000 7f6476ab0640 10 monclient: monmap: 2023-10-30T12:06:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:02.983963+0000 2023-10-30T12:06:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:02.983963+0000 2023-10-30T12:06:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:02.987 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.983+0000 7f6476ab0640 1 Processor -- start 2023-10-30T12:06:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.983+0000 7f6476ab0640 1 -- start start 2023-10-30T12:06:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.983+0000 7f6476ab0640 10 monclient: init 2023-10-30T12:06:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-10-30T12:06:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6470130290 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f6470138680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient(hunting): picked mon.noname-b con 0x7f6470070fa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6470138bc0 0x7f647013afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient(hunting): picked mon.noname-c con 0x7f6470138bc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6470130d80 con 0x7f6470130290 2023-10-30T12:06:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64700702e0 con 0x7f6470138bc0 2023-10-30T12:06:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6470070000 con 0x7f6470070fa0 2023-10-30T12:06:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f6476ab0640 10 monclient(hunting): authenticate will time out at 1208.167258s 2023-10-30T12:06:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f6470138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f647484c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6470130290 0x7f6470130670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55972/0 (socket says 172.21.15.50:55972) 2023-10-30T12:06:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f647484c640 1 -- 172.21.15.50:0/2643276468 learned_addr learned my addr 172.21.15.50:0/2643276468 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:06:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): get_auth_request con 0x7f6470070fa0 auth_method 0 2023-10-30T12:06:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient(hunting): handle_auth_done global_id 44828 payload 274 2023-10-30T12:06:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 1 -- 172.21.15.50:0/2643276468 >> v2:172.21.15.118:3300/0 conn(0x7f6470138bc0 msgr2=0x7f647013afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 1 --2- 172.21.15.50:0/2643276468 >> v2:172.21.15.118:3300/0 conn(0x7f6470138bc0 0x7f647013afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 1 -- 172.21.15.50:0/2643276468 >> v1:172.21.15.50:6789/0 conn(0x7f6470130290 legacy=0x7f6470130670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 1 monclient: found mon.noname-b 2023-10-30T12:06:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:03.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:03.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646f7fe640 1 -- 172.21.15.50:0/2643276468 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038911633 0 0) 0x7f6470130d80 con 0x7f6470130290 2023-10-30T12:06:03.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646ffff640 1 -- 172.21.15.50:0/2643276468 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647006f2d0 con 0x7f6470070fa0 2023-10-30T12:06:03.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.984+0000 7f646f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:06:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646ffff640 1 --2- 172.21.15.50:0/2643276468 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f6470138680 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f64600097a0 tx=0x7f646002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=443b08acee8213f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 1 -- 172.21.15.50:0/2643276468 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646002f050 con 0x7f6470070fa0 2023-10-30T12:06:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: dump: 2023-10-30T12:06:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:03.005 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:03.005 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:03.005 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:03.005 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:03.006 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:03.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:06:03.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:06:03.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:03.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 1 -- 172.21.15.50:0/2643276468 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6460031700 con 0x7f6470070fa0 2023-10-30T12:06:03.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f6476ab0640 5 monclient: authenticate success, global_id 44828 2023-10-30T12:06:03.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f6476ab0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:03.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:03.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f6476ab0640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:03.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f6476ab0640 4 set_mon_vals no callback set 2023-10-30T12:06:03.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 1 -- 172.21.15.50:0/2643276468 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460031a90 con 0x7f6470070fa0 2023-10-30T12:06:03.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:03.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:03.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.985+0000 7f646f7fe640 10 monclient: dump: 2023-10-30T12:06:03.009 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:03.009 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:03.009 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.010 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.010 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:03.010 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:03.010 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:03.010 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:03.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:03.011 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:03.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 10 monclient: shutdown 2023-10-30T12:06:03.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:03.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 -- 172.21.15.50:0/2643276468 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 msgr2=0x7f6470138680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 --2- 172.21.15.50:0/2643276468 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f6470138680 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f64600097a0 tx=0x7f646002b7a0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 -- 172.21.15.50:0/2643276468 shutdown_connections 2023-10-30T12:06:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 --2- 172.21.15.50:0/2643276468 >> v2:172.21.15.118:3300/0 conn(0x7f6470138bc0 0x7f647013afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 --2- 172.21.15.50:0/2643276468 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f6470138680 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 -- 172.21.15.50:0/2643276468 >> 172.21.15.50:0/2643276468 conn(0x7f64700a22b0 msgr2=0x7f64700a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 -- 172.21.15.50:0/2643276468 shutdown_connections 2023-10-30T12:06:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 -- 172.21.15.50:0/2643276468 wait complete. 2023-10-30T12:06:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 10 monclient: build_initial_monmap 2023-10-30T12:06:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 10 monclient: monmap: 2023-10-30T12:06:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:03.015 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:03.015 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:03.015 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:03.015 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:03.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 librados: starting msgr at 2023-10-30T12:06:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 librados: starting objecter 2023-10-30T12:06:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.986+0000 7f6476ab0640 1 Processor -- start 2023-10-30T12:06:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 -- start start 2023-10-30T12:06:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 librados: setting wanted keys 2023-10-30T12:06:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 librados: calling monclient init 2023-10-30T12:06:03.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient: init 2023-10-30T12:06:03.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:03.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:06:03.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f647012f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:03.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): picked mon.noname-c con 0x7f6470070fa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:03.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6470130290 0x7f6470069de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:03.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): picked mon.noname-a con 0x7f6470130290 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:03.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6470138bc0 0x7f647006a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:03.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): picked mon.noname-b con 0x7f6470138bc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:03.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:03.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647006f010 con 0x7f6470130290 2023-10-30T12:06:03.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:03.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647006b640 con 0x7f6470138bc0 2023-10-30T12:06:03.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700a4520 con 0x7f6470070fa0 2023-10-30T12:06:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f6476ab0640 10 monclient(hunting): authenticate will time out at 1208.170596s 2023-10-30T12:06:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f647012f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f647012f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53282/0 (socket says 172.21.15.50:53282) 2023-10-30T12:06:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6470130290 0x7f6470069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 1 -- 172.21.15.50:0/3693795362 learned_addr learned my addr 172.21.15.50:0/3693795362 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): get_auth_request con 0x7f6470070fa0 auth_method 0 2023-10-30T12:06:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 10 monclient(hunting): get_auth_request con 0x7f6470130290 auth_method 0 2023-10-30T12:06:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:03.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:03.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f646ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:03.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 10 monclient(hunting): handle_auth_done global_id 44831 payload 824 2023-10-30T12:06:03.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647484c640 1 -- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6470138bc0 msgr2=0x7f647006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:03.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.987+0000 7f647504d640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6470138bc0 0x7f647006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:03.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6470138bc0 0x7f647006a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 1 -- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6470130290 msgr2=0x7f6470069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:03.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6470130290 0x7f6470069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 10 monclient: _finish_hunting 0 2023-10-30T12:06:03.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 1 monclient: found mon.noname-c 2023-10-30T12:06:03.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:03.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:03.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 1 -- 172.21.15.50:0/3693795362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6460009400 con 0x7f6470070fa0 2023-10-30T12:06:03.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647504d640 10 monclient: get_auth_request con 0x7f6470138bc0 auth_method 0 2023-10-30T12:06:03.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646ffff640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6470130290 0x7f6470069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:03.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647504d640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6470138bc0 0x7f647006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:06:03.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f647484c640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f647012f1f0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f646800c890 tx=0x7f646800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:03.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 client.?.objecter ms_handle_connect 0x7f6470070fa0 2023-10-30T12:06:03.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:03.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 1 -- 172.21.15.50:0/3693795362 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800a640 con 0x7f6470070fa0 2023-10-30T12:06:03.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:03.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:03.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: dump: 2023-10-30T12:06:03.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:03.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:03.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:03.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:03.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:03.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:03.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:03.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:03.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:06:03.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:06:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 1 -- 172.21.15.50:0/3693795362 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646800a7c0 con 0x7f6470070fa0 2023-10-30T12:06:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 5 monclient: authenticate success, global_id 44831 2023-10-30T12:06:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 10 monclient: _renew_subs 2023-10-30T12:06:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647011ce80 con 0x7f6470070fa0 2023-10-30T12:06:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 1 -- 172.21.15.50:0/3693795362 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800ab70 con 0x7f6470070fa0 2023-10-30T12:06:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:03.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646d7fa640 10 monclient: dump: 2023-10-30T12:06:03.033 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:03.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:03.033 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.033 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:03.035 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f646effd640 4 set_mon_vals no callback set 2023-10-30T12:06:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 10 client.44831.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 10 monclient: _renew_subs 2023-10-30T12:06:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647011bf90 con 0x7f6470070fa0 2023-10-30T12:06:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.988+0000 7f6476ab0640 1 librados: init done 2023-10-30T12:06:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646d7fa640 1 -- 172.21.15.50:0/3693795362 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f646800acd0 con 0x7f6470070fa0 2023-10-30T12:06:03.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f6476ab0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:03.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f6476ab0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:03.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f6476ab0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:03.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647009fe10 con 0x7f6470070fa0 2023-10-30T12:06:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646d7fa640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64500392f0 0x7f645003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646d7fa640 1 -- 172.21.15.50:0/3693795362 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6846+0+0 (secure 0 0 0) 0x7f646804a3c0 con 0x7f6470070fa0 2023-10-30T12:06:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646d7fa640 10 client.44831.objecter ms_dispatch 0x7f6470000b90 osd_map(170..170 src has 1..170) v4 2023-10-30T12:06:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646d7fa640 3 client.44831.objecter handle_osd_map got epochs [170,170] > 0 2023-10-30T12:06:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646d7fa640 3 client.44831.objecter handle_osd_map decoding full epoch 170 2023-10-30T12:06:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646d7fa640 20 client.44831.objecter dump_active .. 0 homeless 2023-10-30T12:06:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646ffff640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64500392f0 0x7f645003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.989+0000 7f646ffff640 10 monclient: get_auth_request con 0x7f64500392f0 auth_method 0 2023-10-30T12:06:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.990+0000 7f646d7fa640 10 client.44831.objecter ms_handle_connect 0x7f64500392f0 2023-10-30T12:06:03.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.990+0000 7f646ffff640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64500392f0 0x7f645003b7b0 secure :-1 s=READY pgs=3971 cs=0 l=1 rev1=1 crypto rx=0x7f6460035000 tx=0x7f6460008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:03.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.993+0000 7f646d7fa640 1 -- 172.21.15.50:0/3693795362 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f647009fe10 con 0x7f6470070fa0 2023-10-30T12:06:03.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.993+0000 7f646d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:03.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:02.993+0000 7f646d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:03.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.237+0000 7f6476ab0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:06:03.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.237+0000 7f6476ab0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:06:03.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.237+0000 7f6476ab0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:03.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.237+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f647009fe10 con 0x7f6470070fa0 2023-10-30T12:06:03.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.238+0000 7f646d7fa640 1 -- 172.21.15.50:0/3693795362 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f647009fe10 con 0x7f6470070fa0 2023-10-30T12:06:03.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.238+0000 7f646d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:06:03.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.238+0000 7f646d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:03.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.238+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.238+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=8,logm=63} crc {auth=931480671,config=3140514009,health=2948274808,kv=784550477,logm=3556650797}) 2023-10-30T12:06:03.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.238+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=9,logm=64} crc {auth=3773371734,config=3140514009,health=1320491610,kv=2905431493,logm=2047175851}) 2023-10-30T12:06:03.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.238+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.238+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=8,logm=63} crc {auth=931480671,config=3140514009,health=2948274808,kv=784550477,logm=3556650797}) 2023-10-30T12:06:03.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.238+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=10,kv=9,logm=64} crc {auth=1266076718,config=3140514009,health=1357266946,kv=2905431493,logm=1865960474}) 2023-10-30T12:06:03.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.240+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.240+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163146264}) 2023-10-30T12:06:03.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.240+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912743842}) 2023-10-30T12:06:03.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.240+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.240+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163146264}) 2023-10-30T12:06:03.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.240+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137623844}) 2023-10-30T12:06:03.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.241+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.241+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156713666}) 2023-10-30T12:06:03.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.241+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234832750}) 2023-10-30T12:06:03.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.241+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.241+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156713666}) 2023-10-30T12:06:03.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.241+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365187364}) 2023-10-30T12:06:03.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 10 librados: watch_flush enter 2023-10-30T12:06:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 10 librados: watch_flush exit 2023-10-30T12:06:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 20 client.44831.objecter shutdown clearing up homeless session... 2023-10-30T12:06:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 10 client.44831.objecter successfully canceled tick 2023-10-30T12:06:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64500392f0 msgr2=0x7f645003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64500392f0 0x7f645003b7b0 secure :-1 s=READY pgs=3971 cs=0 l=1 rev1=1 crypto rx=0x7f6460035000 tx=0x7f6460008030 comp rx=0 tx=0).stop 2023-10-30T12:06:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 10 monclient: shutdown 2023-10-30T12:06:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 msgr2=0x7f647012f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.241+0000 7f6476ab0640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f647012f1f0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f646800c890 tx=0x7f646800cd50 comp rx=0 tx=0).stop 2023-10-30T12:06:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 shutdown_connections 2023-10-30T12:06:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64500392f0 0x7f645003b7b0 unknown :-1 s=CLOSED pgs=3971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6470138bc0 0x7f647006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6470130290 0x7f6470069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 --2- 172.21.15.50:0/3693795362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6470070fa0 0x7f647012f1f0 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 >> 172.21.15.50:0/3693795362 conn(0x7f64700a22b0 msgr2=0x7f64700a10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:03.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 shutdown_connections 2023-10-30T12:06:03.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 -- 172.21.15.50:0/3693795362 wait complete. 2023-10-30T12:06:03.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.242+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.242+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702050222}) 2023-10-30T12:06:03.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.242+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044375665}) 2023-10-30T12:06:03.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.242+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.242+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702050222}) 2023-10-30T12:06:03.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.242+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292109925}) 2023-10-30T12:06:03.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:03.242+0000 7f6476ab0640 1 librados: shutdown 2023-10-30T12:06:03.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.243+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.243+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456274532}) 2023-10-30T12:06:03.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.243+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692434150}) 2023-10-30T12:06:03.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.243+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.243+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456274532}) 2023-10-30T12:06:03.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.243+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006880427}) 2023-10-30T12:06:03.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194202899}) 2023-10-30T12:06:03.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3393978332}) 2023-10-30T12:06:03.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194202899}) 2023-10-30T12:06:03.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2977945284}) 2023-10-30T12:06:03.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504916416}) 2023-10-30T12:06:03.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2061718309}) 2023-10-30T12:06:03.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504916416}) 2023-10-30T12:06:03.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.244+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318008876}) 2023-10-30T12:06:03.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.245+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.245+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301774907}) 2023-10-30T12:06:03.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.245+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930032565}) 2023-10-30T12:06:03.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.245+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.245+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301774907}) 2023-10-30T12:06:03.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.245+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1289060211}) 2023-10-30T12:06:03.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603028461}) 2023-10-30T12:06:03.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190971078}) 2023-10-30T12:06:03.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603028461}) 2023-10-30T12:06:03.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2180880495}) 2023-10-30T12:06:03.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487744200}) 2023-10-30T12:06:03.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.246+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3365368036}) 2023-10-30T12:06:03.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487744200}) 2023-10-30T12:06:03.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918622451}) 2023-10-30T12:06:03.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742441554}) 2023-10-30T12:06:03.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2834243159}) 2023-10-30T12:06:03.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742441554}) 2023-10-30T12:06:03.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.247+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808654304}) 2023-10-30T12:06:03.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.248+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.248+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110193283}) 2023-10-30T12:06:03.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.248+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2928953396}) 2023-10-30T12:06:03.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.248+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.248+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110193283}) 2023-10-30T12:06:03.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.248+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149199509}) 2023-10-30T12:06:03.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.249+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.249+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1812009381}) 2023-10-30T12:06:03.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.249+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076457431}) 2023-10-30T12:06:03.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.249+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.249+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1812009381}) 2023-10-30T12:06:03.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.249+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3399237465}) 2023-10-30T12:06:03.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105189300}) 2023-10-30T12:06:03.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1288317771}) 2023-10-30T12:06:03.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105189300}) 2023-10-30T12:06:03.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862006898}) 2023-10-30T12:06:03.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444768130}) 2023-10-30T12:06:03.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409287621}) 2023-10-30T12:06:03.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444768130}) 2023-10-30T12:06:03.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.250+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443688022}) 2023-10-30T12:06:03.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.251+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.251+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491111788}) 2023-10-30T12:06:03.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.251+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=650610037}) 2023-10-30T12:06:03.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.251+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.251+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491111788}) 2023-10-30T12:06:03.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.251+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069964487}) 2023-10-30T12:06:03.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814285621}) 2023-10-30T12:06:03.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1236760398}) 2023-10-30T12:06:03.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814285621}) 2023-10-30T12:06:03.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271974719}) 2023-10-30T12:06:03.272 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:06:03.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2295270981}) 2023-10-30T12:06:03.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661098078}) 2023-10-30T12:06:03.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.252+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2295270981}) 2023-10-30T12:06:03.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.253+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971512099}) 2023-10-30T12:06:03.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.253+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.253+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658487655}) 2023-10-30T12:06:03.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.253+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1150345190}) 2023-10-30T12:06:03.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.253+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.253+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658487655}) 2023-10-30T12:06:03.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.253+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=292273420}) 2023-10-30T12:06:03.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131673804}) 2023-10-30T12:06:03.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467604929}) 2023-10-30T12:06:03.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131673804}) 2023-10-30T12:06:03.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3685967596}) 2023-10-30T12:06:03.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765247442}) 2023-10-30T12:06:03.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4239637070}) 2023-10-30T12:06:03.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765247442}) 2023-10-30T12:06:03.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.254+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753192550}) 2023-10-30T12:06:03.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.255+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.255+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101921237}) 2023-10-30T12:06:03.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.255+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2458979544}) 2023-10-30T12:06:03.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.255+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.255+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101921237}) 2023-10-30T12:06:03.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.255+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=175940098}) 2023-10-30T12:06:03.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212634174}) 2023-10-30T12:06:03.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2798790018}) 2023-10-30T12:06:03.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212634174}) 2023-10-30T12:06:03.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890656733}) 2023-10-30T12:06:03.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080247471}) 2023-10-30T12:06:03.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1729581309}) 2023-10-30T12:06:03.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080247471}) 2023-10-30T12:06:03.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.256+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285057688}) 2023-10-30T12:06:03.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.257+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.257+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002801912}) 2023-10-30T12:06:03.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.257+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1913288420}) 2023-10-30T12:06:03.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.257+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.257+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002801912}) 2023-10-30T12:06:03.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.257+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869270607}) 2023-10-30T12:06:03.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.258+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.258+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=27,osdmap=15} crc {logm=444507157,mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1321077719,mon_config_key=2148863383,monmap=3324057927,osd_metadata=1261359280,osd_pg_creating=3260485431,osd_snap=2119047375,osdmap=2715190535}) 2023-10-30T12:06:03.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.258+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=15} crc {logm=377916389,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2001212215,mon_config_key=3217606506,monmap=3324057927,osd_metadata=451695420,osd_pg_creating=3260485431,osd_snap=3163795322,osdmap=3607562020}) 2023-10-30T12:06:03.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.258+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.258+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=27,osdmap=15} crc {logm=444507157,mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1321077719,mon_config_key=2148863383,monmap=3324057927,osd_metadata=1261359280,osd_pg_creating=3260485431,osd_snap=2119047375,osdmap=2715190535}) 2023-10-30T12:06:03.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.258+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=24,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=16} crc {logm=2600971060,mdsmap=3984425268,mgr=4291836917,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=747874102,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=3260485431,osd_snap=3462770239,osdmap=3748134397}) 2023-10-30T12:06:03.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.259+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.259+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3283436644}) 2023-10-30T12:06:03.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.259+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3576053090}) 2023-10-30T12:06:03.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.259+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.259+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3283436644}) 2023-10-30T12:06:03.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.259+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3555661553}) 2023-10-30T12:06:03.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.260+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.260+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=560540882}) 2023-10-30T12:06:03.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.260+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=215299050}) 2023-10-30T12:06:03.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.260+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.288 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.260+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=560540882}) 2023-10-30T12:06:03.288 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.260+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1112642183}) 2023-10-30T12:06:03.288 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.266+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.288 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.266+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=1629487259}) 2023-10-30T12:06:03.288 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.266+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=45} crc {osdmap=3327281816}) 2023-10-30T12:06:03.288 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.266+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:03.289 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.266+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=1629487259}) 2023-10-30T12:06:03.289 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:03.266+0000 7fb4917ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=40796128}) 2023-10-30T12:06:04.274 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:06:04.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.396+0000 7f598a553640 10 monclient: get_monmap_and_config 2023-10-30T12:06:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.396+0000 7f598a553640 10 monclient: build_initial_monmap 2023-10-30T12:06:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.396+0000 7f598a553640 10 monclient: monmap: 2023-10-30T12:06:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:04.397135+0000 2023-10-30T12:06:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:04.397135+0000 2023-10-30T12:06:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:04.400 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.396+0000 7f598a553640 1 Processor -- start 2023-10-30T12:06:04.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.396+0000 7f598a553640 1 -- start start 2023-10-30T12:06:04.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.396+0000 7f598a553640 10 monclient: init 2023-10-30T12:06:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T12:06:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59840a4b60 0x7f59841326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient(hunting): picked mon.noname-b con 0x7f59840a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:04.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5984099a40 0x7f5984132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient(hunting): picked mon.noname-c con 0x7f5984099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:04.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59840a2350 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:04.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5984133120 con 0x7f59840a2350 2023-10-30T12:06:04.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59840a3d70 con 0x7f5984099a40 2023-10-30T12:06:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840a4f40 con 0x7f59840a4b60 2023-10-30T12:06:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f598a553640 10 monclient(hunting): authenticate will time out at 1209.580469s 2023-10-30T12:06:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59840a4b60 0x7f59841326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59840a4b60 0x7f59841326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53298/0 (socket says 172.21.15.50:53298) 2023-10-30T12:06:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 -- 172.21.15.50:0/206729334 learned_addr learned my addr 172.21.15.50:0/206729334 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): get_auth_request con 0x7f59840a4b60 auth_method 0 2023-10-30T12:06:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 1 --2- 172.21.15.50:0/206729334 >> v2:172.21.15.118:3300/0 conn(0x7f5984099a40 0x7f5984132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 10 monclient(hunting): get_auth_request con 0x7f5984099a40 auth_method 0 2023-10-30T12:06:04.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f59837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient(hunting): handle_auth_done global_id 44840 payload 274 2023-10-30T12:06:04.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5982ffd640 1 -- 172.21.15.50:0/206729334 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30854775 0 0) 0x7f5984133120 con 0x7f59840a2350 2023-10-30T12:06:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 -- 172.21.15.50:0/206729334 >> v2:172.21.15.118:3300/0 conn(0x7f5984099a40 msgr2=0x7f5984132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 --2- 172.21.15.50:0/206729334 >> v2:172.21.15.118:3300/0 conn(0x7f5984099a40 0x7f5984132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 -- 172.21.15.50:0/206729334 >> v1:172.21.15.50:6789/0 conn(0x7f59840a2350 legacy=0x7f59840a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 monclient: found mon.noname-b 2023-10-30T12:06:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5983fff640 1 -- 172.21.15.50:0/206729334 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840a4150 con 0x7f59840a4b60 2023-10-30T12:06:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.397+0000 7f5982ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:06:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5983fff640 1 --2- 172.21.15.50:0/206729334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59840a4b60 0x7f59841326a0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f5974009870 tx=0x7f597402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4e082368b8b4429 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 1 -- 172.21.15.50:0/206729334 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740317c0 con 0x7f59840a4b60 2023-10-30T12:06:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: dump: 2023-10-30T12:06:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.420 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: _finish_auth 0 2023-10-30T12:06:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:04.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:04.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 1 -- 172.21.15.50:0/206729334 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974031920 con 0x7f59840a4b60 2023-10-30T12:06:04.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f598a553640 5 monclient: authenticate success, global_id 44840 2023-10-30T12:06:04.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f598a553640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:04.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:04.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f598a553640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:04.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 1 -- 172.21.15.50:0/206729334 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974031cb0 con 0x7f59840a4b60 2023-10-30T12:06:04.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f598a553640 4 set_mon_vals no callback set 2023-10-30T12:06:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.398+0000 7f5982ffd640 10 monclient: dump: 2023-10-30T12:06:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.424 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.424 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.424 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.424 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.424 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 10 monclient: shutdown 2023-10-30T12:06:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 -- 172.21.15.50:0/206729334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59840a4b60 msgr2=0x7f59841326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 --2- 172.21.15.50:0/206729334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59840a4b60 0x7f59841326a0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f5974009870 tx=0x7f597402b860 comp rx=0 tx=0).stop 2023-10-30T12:06:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 -- 172.21.15.50:0/206729334 shutdown_connections 2023-10-30T12:06:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 --2- 172.21.15.50:0/206729334 >> v2:172.21.15.118:3300/0 conn(0x7f5984099a40 0x7f5984132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 --2- 172.21.15.50:0/206729334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59840a4b60 0x7f59841326a0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 -- 172.21.15.50:0/206729334 >> 172.21.15.50:0/206729334 conn(0x7f59840a8060 msgr2=0x7f59840a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:04.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 -- 172.21.15.50:0/206729334 shutdown_connections 2023-10-30T12:06:04.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.399+0000 7f598a553640 1 -- 172.21.15.50:0/206729334 wait complete. 2023-10-30T12:06:04.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 10 monclient: build_initial_monmap 2023-10-30T12:06:04.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 10 monclient: monmap: 2023-10-30T12:06:04.428 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:04.428 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:04.428 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:04.428 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:04.429 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:04.429 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.429 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:04.429 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:04.429 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:04.430 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 1 librados: starting msgr at 2023-10-30T12:06:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 1 librados: starting objecter 2023-10-30T12:06:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 1 Processor -- start 2023-10-30T12:06:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 1 -- start start 2023-10-30T12:06:04.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 1 librados: setting wanted keys 2023-10-30T12:06:04.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 1 librados: calling monclient init 2023-10-30T12:06:04.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.400+0000 7f598a553640 10 monclient: init 2023-10-30T12:06:04.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:04.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:06:04.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59840a2350 0x7f59841ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): picked mon.noname-a con 0x7f59840a2350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:04.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59840a4b60 0x7f59841caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): picked mon.noname-b con 0x7f59840a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:04.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5984125b10 0x7f5984121680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): picked mon.noname-c con 0x7f5984125b10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:04.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840a18d0 con 0x7f59840a2350 2023-10-30T12:06:04.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59841256f0 con 0x7f59840a4b60 2023-10-30T12:06:04.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840a0880 con 0x7f5984125b10 2023-10-30T12:06:04.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:04.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient(hunting): authenticate will time out at 1209.584186s 2023-10-30T12:06:04.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59840a2350 0x7f59841ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5984125b10 0x7f5984121680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59840a2350 0x7f59841ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38700/0 (socket says 172.21.15.50:38700) 2023-10-30T12:06:04.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 1 -- 172.21.15.50:0/3201044935 learned_addr learned my addr 172.21.15.50:0/3201044935 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:04.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 10 monclient(hunting): get_auth_request con 0x7f59840a2350 auth_method 0 2023-10-30T12:06:04.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): get_auth_request con 0x7f5984125b10 auth_method 0 2023-10-30T12:06:04.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5983fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f59837fe640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59840a4b60 0x7f59841caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f59837fe640 10 monclient(hunting): get_auth_request con 0x7f59840a4b60 auth_method 0 2023-10-30T12:06:04.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f59837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f59837fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f59837fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient(hunting): handle_auth_done global_id 44843 payload 824 2023-10-30T12:06:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 -- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59840a4b60 msgr2=0x7f59841caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59840a4b60 0x7f59841caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 -- 172.21.15.50:0/3201044935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59840a2350 msgr2=0x7f59841ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59840a2350 0x7f59841ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient: _finish_hunting 0 2023-10-30T12:06:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 monclient: found mon.noname-c 2023-10-30T12:06:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 -- 172.21.15.50:0/3201044935 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740094d0 con 0x7f5984125b10 2023-10-30T12:06:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5988af0640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5984125b10 0x7f5984121680 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f597c001560 tx=0x7f597c00cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 client.?.objecter ms_handle_connect 0x7f5984125b10 2023-10-30T12:06:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 1 -- 172.21.15.50:0/3201044935 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c003da0 con 0x7f5984125b10 2023-10-30T12:06:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 monclient: dump: 2023-10-30T12:06:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.444 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.444 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.444 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:04.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:04.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:04.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 5 monclient: authenticate success, global_id 44843 2023-10-30T12:06:04.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 1 -- 172.21.15.50:0/3201044935 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c00ce30 con 0x7f5984125b10 2023-10-30T12:06:04.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:04.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f5980ff9640 1 -- 172.21.15.50:0/3201044935 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c005290 con 0x7f5984125b10 2023-10-30T12:06:04.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f598a553640 10 monclient: _renew_subs 2023-10-30T12:06:04.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f598a553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5984124e50 con 0x7f5984125b10 2023-10-30T12:06:04.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.401+0000 7f59827fc640 4 set_mon_vals no callback set 2023-10-30T12:06:04.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f5980ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f5980ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f5980ff9640 10 monclient: dump: 2023-10-30T12:06:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.448 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f598a553640 10 client.44843.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f598a553640 10 monclient: _renew_subs 2023-10-30T12:06:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f598a553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59841251e0 con 0x7f5984125b10 2023-10-30T12:06:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.402+0000 7f598a553640 1 librados: init done 2023-10-30T12:06:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f598a553640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:04.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f598a553640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:04.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f598a553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5980ff9640 1 -- 172.21.15.50:0/3201044935 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f597c0053f0 con 0x7f5984125b10 2023-10-30T12:06:04.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984125370 con 0x7f5984125b10 2023-10-30T12:06:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5980ff9640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59700392f0 0x7f597003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5980ff9640 1 -- 172.21.15.50:0/3201044935 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f597c04a1f0 con 0x7f5984125b10 2023-10-30T12:06:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5980ff9640 10 client.44843.objecter ms_dispatch 0x7f5984000b90 osd_map(172..172 src has 1..172) v4 2023-10-30T12:06:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5980ff9640 3 client.44843.objecter handle_osd_map got epochs [172,172] > 0 2023-10-30T12:06:04.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5980ff9640 3 client.44843.objecter handle_osd_map decoding full epoch 172 2023-10-30T12:06:04.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5980ff9640 20 client.44843.objecter dump_active .. 0 homeless 2023-10-30T12:06:04.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5983fff640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59700392f0 0x7f597003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.403+0000 7f5983fff640 10 monclient: get_auth_request con 0x7f59700392f0 auth_method 0 2023-10-30T12:06:04.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.404+0000 7f5980ff9640 10 client.44843.objecter ms_handle_connect 0x7f59700392f0 2023-10-30T12:06:04.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.404+0000 7f5983fff640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59700392f0 0x7f597003b7b0 secure :-1 s=READY pgs=3974 cs=0 l=1 rev1=1 crypto rx=0x7f59740090b0 tx=0x7f597403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.408+0000 7f5980ff9640 1 -- 172.21.15.50:0/3201044935 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5984125370 con 0x7f5984125b10 2023-10-30T12:06:04.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.408+0000 7f5980ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:04.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.408+0000 7f5980ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:04.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.660+0000 7f598a553640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.660+0000 7f598a553640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.660+0000 7f598a553640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.660+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5984125370 con 0x7f5984125b10 2023-10-30T12:06:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.661+0000 7f5980ff9640 1 -- 172.21.15.50:0/3201044935 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f5984125370 con 0x7f5984125b10 2023-10-30T12:06:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.661+0000 7f5980ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.661+0000 7f5980ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:04.664 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.663+0000 7f598a553640 10 librados: watch_flush enter 2023-10-30T12:06:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 10 librados: watch_flush exit 2023-10-30T12:06:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 20 client.44843.objecter shutdown clearing up homeless session... 2023-10-30T12:06:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 10 client.44843.objecter successfully canceled tick 2023-10-30T12:06:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59700392f0 msgr2=0x7f597003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59700392f0 0x7f597003b7b0 secure :-1 s=READY pgs=3974 cs=0 l=1 rev1=1 crypto rx=0x7f59740090b0 tx=0x7f597403d040 comp rx=0 tx=0).stop 2023-10-30T12:06:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 10 monclient: shutdown 2023-10-30T12:06:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5984125b10 msgr2=0x7f5984121680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5984125b10 0x7f5984121680 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f597c001560 tx=0x7f597c00cbe0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 shutdown_connections 2023-10-30T12:06:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59700392f0 0x7f597003b7b0 unknown :-1 s=CLOSED pgs=3974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5984125b10 0x7f5984121680 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59840a4b60 0x7f59841caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 --2- 172.21.15.50:0/3201044935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59840a2350 0x7f59841ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 >> 172.21.15.50:0/3201044935 conn(0x7f59840a8060 msgr2=0x7f598409d900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.664+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 shutdown_connections 2023-10-30T12:06:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.665+0000 7f598a553640 1 -- 172.21.15.50:0/3201044935 wait complete. 2023-10-30T12:06:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.665+0000 7f598a553640 1 librados: shutdown 2023-10-30T12:06:04.677 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:06:04.677 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:06:04.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:06:04.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:06:04.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.782+0000 7f2a1ddc5640 10 monclient: get_monmap_and_config 2023-10-30T12:06:04.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.782+0000 7f2a1ddc5640 10 monclient: build_initial_monmap 2023-10-30T12:06:04.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.782+0000 7f2a1ddc5640 10 monclient: monmap: 2023-10-30T12:06:04.784 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:04.784 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.784 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:04.783432+0000 2023-10-30T12:06:04.785 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:04.783432+0000 2023-10-30T12:06:04.785 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:04.785 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.785 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:04.785 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:04.786 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:04.786 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:04.786 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:04.786 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 Processor -- start 2023-10-30T12:06:04.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 -- start start 2023-10-30T12:06:04.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient: init 2023-10-30T12:06:04.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:04.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T12:06:04.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a1806afc0 0x7f2a180a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1806afc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:04.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2a1806b640 0x7f2a180a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a1806b640 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:04.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a1813c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a180a2710 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:04.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1806bde0 con 0x7f2a1806afc0 2023-10-30T12:06:04.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1806b3a0 con 0x7f2a1806b640 2023-10-30T12:06:04.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1809e540 con 0x7f2a180a2710 2023-10-30T12:06:04.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:04.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a1ddc5640 10 monclient(hunting): authenticate will time out at 1209.966785s 2023-10-30T12:06:04.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a17fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a1813c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a177fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a1806afc0 0x7f2a180a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a17fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a1813c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53322/0 (socket says 172.21.15.50:53322) 2023-10-30T12:06:04.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a177fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a1806afc0 0x7f2a180a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38704/0 (socket says 172.21.15.50:38704) 2023-10-30T12:06:04.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a17fff640 1 -- 172.21.15.50:0/3125281246 learned_addr learned my addr 172.21.15.50:0/3125281246 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a17fff640 10 monclient(hunting): get_auth_request con 0x7f2a180a2710 auth_method 0 2023-10-30T12:06:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a17fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a17fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a177fe640 10 monclient(hunting): get_auth_request con 0x7f2a1806afc0 auth_method 0 2023-10-30T12:06:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.783+0000 7f2a177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 1 --2- 172.21.15.50:0/3125281246 >> v2:172.21.15.118:3300/0 conn(0x7f2a1806b640 0x7f2a180a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 10 monclient(hunting): get_auth_request con 0x7f2a1806b640 auth_method 0 2023-10-30T12:06:04.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 10 monclient(hunting): handle_auth_done global_id 44846 payload 274 2023-10-30T12:06:04.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 1 -- 172.21.15.50:0/3125281246 >> v2:172.21.15.118:3300/0 conn(0x7f2a1806b640 msgr2=0x7f2a180a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:04.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 1 --2- 172.21.15.50:0/3125281246 >> v2:172.21.15.118:3300/0 conn(0x7f2a1806b640 0x7f2a180a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 1 -- 172.21.15.50:0/3125281246 >> v2:172.21.15.50:3300/0 conn(0x7f2a1806afc0 msgr2=0x7f2a180a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:04.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 1 --2- 172.21.15.50:0/3125281246 >> v2:172.21.15.50:3300/0 conn(0x7f2a1806afc0 0x7f2a180a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:04.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 1 monclient: found mon.noname-b 2023-10-30T12:06:04.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:04.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 1 -- 172.21.15.50:0/3125281246 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1809fa10 con 0x7f2a180a2710 2023-10-30T12:06:04.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a177fe640 1 --2- 172.21.15.50:0/3125281246 >> v2:172.21.15.50:3300/0 conn(0x7f2a1806afc0 0x7f2a180a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:04.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.784+0000 7f2a17fff640 1 --2- 172.21.15.50:0/3125281246 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a1813c070 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f2a1806a180 tx=0x7f2a1002bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10fed4dad095eb64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 1 -- 172.21.15.50:0/3125281246 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1002c960 con 0x7f2a180a2710 2023-10-30T12:06:04.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: dump: 2023-10-30T12:06:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.807 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.807 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.807 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.807 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.807 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.808 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.808 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: _finish_auth 0 2023-10-30T12:06:04.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:04.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:04.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 5 monclient: authenticate success, global_id 44846 2023-10-30T12:06:04.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:04.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 1 -- 172.21.15.50:0/3125281246 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1002cac0 con 0x7f2a180a2710 2023-10-30T12:06:04.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:04.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:04.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 4 set_mon_vals no callback set 2023-10-30T12:06:04.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 1 -- 172.21.15.50:0/3125281246 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1002ce50 con 0x7f2a180a2710 2023-10-30T12:06:04.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:04.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a16ffd640 10 monclient: dump: 2023-10-30T12:06:04.810 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.811 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.811 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.811 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.811 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.811 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.812 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.812 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.812 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.812 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 10 monclient: shutdown 2023-10-30T12:06:04.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:04.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/3125281246 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 msgr2=0x7f2a1813c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/3125281246 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a1813c070 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f2a1806a180 tx=0x7f2a1002bdc0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/3125281246 shutdown_connections 2023-10-30T12:06:04.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/3125281246 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a1813c070 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/3125281246 >> v2:172.21.15.118:3300/0 conn(0x7f2a1806b640 0x7f2a180a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/3125281246 >> v2:172.21.15.50:3300/0 conn(0x7f2a1806afc0 0x7f2a180a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/3125281246 >> 172.21.15.50:0/3125281246 conn(0x7f2a180a6200 msgr2=0x7f2a180a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:04.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/3125281246 shutdown_connections 2023-10-30T12:06:04.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.785+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/3125281246 wait complete. 2023-10-30T12:06:04.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 10 monclient: build_initial_monmap 2023-10-30T12:06:04.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 10 monclient: monmap: 2023-10-30T12:06:04.816 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:04.816 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:04.816 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:04.816 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:04.816 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:04.816 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.817 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:04.817 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:04.817 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:04.817 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 1 librados: starting msgr at 2023-10-30T12:06:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 1 librados: starting objecter 2023-10-30T12:06:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 1 Processor -- start 2023-10-30T12:06:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 1 -- start start 2023-10-30T12:06:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 1 librados: setting wanted keys 2023-10-30T12:06:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 1 librados: calling monclient init 2023-10-30T12:06:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.786+0000 7f2a1ddc5640 10 monclient: init 2023-10-30T12:06:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:06:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a1806afc0 0x7f2a181c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1806afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a1806b640 0x7f2a181c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a1806b640 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a181c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a180a2710 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a181c7400 con 0x7f2a1806afc0 2023-10-30T12:06:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a181a8ac0 con 0x7f2a1806b640 2023-10-30T12:06:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a18070510 con 0x7f2a180a2710 2023-10-30T12:06:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient(hunting): authenticate will time out at 1209.970192s 2023-10-30T12:06:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a1806afc0 0x7f2a181c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a181c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a1806afc0 0x7f2a181c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38718/0 (socket says 172.21.15.50:38718) 2023-10-30T12:06:04.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 1 -- 172.21.15.50:0/2840507247 learned_addr learned my addr 172.21.15.50:0/2840507247 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:04.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 10 monclient(hunting): get_auth_request con 0x7f2a1806afc0 auth_method 0 2023-10-30T12:06:04.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): get_auth_request con 0x7f2a180a2710 auth_method 0 2023-10-30T12:06:04.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient(hunting): handle_auth_done global_id 44849 payload 824 2023-10-30T12:06:04.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a0ffff640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a1806b640 0x7f2a181c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a1806b640 msgr2=0x7f2a181c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a1806b640 0x7f2a181c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a1806afc0 msgr2=0x7f2a181c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a1806afc0 0x7f2a181c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 monclient: found mon.noname-c 2023-10-30T12:06:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 -- 172.21.15.50:0/2840507247 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08009400 con 0x7f2a180a2710 2023-10-30T12:06:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a177fe640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a1806afc0 0x7f2a181c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a17fff640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a181c6ec0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f2a10032000 tx=0x7f2a10004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 client.?.objecter ms_handle_connect 0x7f2a180a2710 2023-10-30T12:06:04.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:04.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10009510 con 0x7f2a180a2710 2023-10-30T12:06:04.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 monclient: dump: 2023-10-30T12:06:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.832 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1003e450 con 0x7f2a180a2710 2023-10-30T12:06:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 5 monclient: authenticate success, global_id 44849 2023-10-30T12:06:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient: _renew_subs 2023-10-30T12:06:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1003e800 con 0x7f2a180a2710 2023-10-30T12:06:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a10009b30 con 0x7f2a180a2710 2023-10-30T12:06:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a14ff9640 10 monclient: dump: 2023-10-30T12:06:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.835 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.835 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.835 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.835 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.835 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.836 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.836 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.836 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.837 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.787+0000 7f2a167fc640 4 set_mon_vals no callback set 2023-10-30T12:06:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 10 client.44849.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:04.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 10 monclient: _renew_subs 2023-10-30T12:06:04.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:04.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a181aae70 con 0x7f2a180a2710 2023-10-30T12:06:04.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 1 librados: init done 2023-10-30T12:06:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2a1003e960 con 0x7f2a180a2710 2023-10-30T12:06:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181ca680 0x7f2a1802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a18073ff0 con 0x7f2a181ca680 2023-10-30T12:06:04.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181ca680 0x7f2a1802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 10 monclient: get_auth_request con 0x7f2a181ca680 auth_method 0 2023-10-30T12:06:04.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a14ff9640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f29f80392f0 0x7f29f803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.788+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f2a1006d690 con 0x7f2a180a2710 2023-10-30T12:06:04.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 client.44849.objecter ms_dispatch 0x7f2a18000b90 osd_map(172..172 src has 1..172) v4 2023-10-30T12:06:04.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 3 client.44849.objecter handle_osd_map got epochs [172,172] > 0 2023-10-30T12:06:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 3 client.44849.objecter handle_osd_map decoding full epoch 172 2023-10-30T12:06:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a177fe640 10 monclient(hunting): handle_auth_done global_id 84387 payload 824 2023-10-30T12:06:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 20 client.44849.objecter dump_active .. 0 homeless 2023-10-30T12:06:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a177fe640 1 -- 172.21.15.50:0/2840507247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a1809df80 con 0x7f2a181ca680 2023-10-30T12:06:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a0ffff640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f29f80392f0 0x7f29f803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a0ffff640 10 monclient: get_auth_request con 0x7f29f80392f0 auth_method 0 2023-10-30T12:06:04.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 client.44849.objecter ms_handle_connect 0x7f2a181ca680 2023-10-30T12:06:04.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 client.44849.objecter resend_mon_ops 2023-10-30T12:06:04.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a177fe640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181ca680 0x7f2a1802b430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800f470 tx=0x7f2a0800f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08004240 con 0x7f2a181ca680 2023-10-30T12:06:04.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:04.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 monclient: dump: 2023-10-30T12:06:04.845 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.845 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.845 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.846 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.846 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.846 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.846 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.846 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.847 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.847 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 client.44849.objecter ms_handle_connect 0x7f29f80392f0 2023-10-30T12:06:04.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a0ffff640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f29f80392f0 0x7f29f803b7b0 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 crypto rx=0x7f2a181caf80 tx=0x7f2a00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f2a080043a0 con 0x7f2a181ca680 2023-10-30T12:06:04.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:04.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:04.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181ca680 msgr2=0x7f2a1802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181ca680 0x7f2a1802b430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800f470 tx=0x7f2a0800f930 comp rx=0 tx=0).stop 2023-10-30T12:06:04.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181ca680 msgr2=0x7f2a1802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:04.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.789+0000 7f2a14ff9640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181ca680 0x7f2a1802b430 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800f470 tx=0x7f2a0800f930 comp rx=0 tx=0).stop 2023-10-30T12:06:04.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.795+0000 7f2a1ddc5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:04.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.795+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181caa60 0x7f2a1802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:04.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.795+0000 7f2a1ddc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:04.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.795+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1806a6f0 con 0x7f2a181caa60 2023-10-30T12:06:04.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181caa60 0x7f2a1802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:04.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient: get_auth_request con 0x7f2a181caa60 auth_method 0 2023-10-30T12:06:04.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:04.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:04.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:04.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:04.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:04.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:04.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 10 monclient(hunting): handle_auth_done global_id 84390 payload 824 2023-10-30T12:06:04.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 1 -- 172.21.15.50:0/2840507247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a1809f0b0 con 0x7f2a181caa60 2023-10-30T12:06:04.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 10 client.44849.objecter ms_handle_connect 0x7f2a181caa60 2023-10-30T12:06:04.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 10 client.44849.objecter resend_mon_ops 2023-10-30T12:06:04.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a177fe640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181caa60 0x7f2a1802b970 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800f470 tx=0x7f2a080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:04.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0802c070 con 0x7f2a181caa60 2023-10-30T12:06:04.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:04.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:04.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 10 monclient: dump: 2023-10-30T12:06:04.854 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:04.854 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:04.854 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.855 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:04.855 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:04.855 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:04.855 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:04.855 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:04.856 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:04.856 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:04.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7f2a08024040 con 0x7f2a181caa60 2023-10-30T12:06:04.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:04.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:04.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181caa60 msgr2=0x7f2a1802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181caa60 0x7f2a1802b970 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800f470 tx=0x7f2a080046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181caa60 msgr2=0x7f2a1802b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:04.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.796+0000 7f2a14ff9640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a181caa60 0x7f2a1802b970 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800f470 tx=0x7f2a080046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:04.857 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:04.858 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:04.858 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:04.858 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:06:04.858 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 56, 2023-10-30T12:06:04.858 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:04.859 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:04.859 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:04.859 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:04.859 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:04.859 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:06:04.860 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:04.860 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:04.860 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:04.860 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:04.860 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:04.861 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:04.861 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:04.861 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:04.861 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:04.862 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:04.862 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:04.862 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:04.862 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:04.863 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:04.863 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:04.863 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:04.863 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:04.863 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:04.864 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:04.864 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:04.864 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:04.864 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:04.865 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:04.865 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:04.865 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:04.865 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:04.865 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:04.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.866 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:04.866 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:04.866 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:04.866 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:04.866 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.884 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:04.884 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:04.884 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:04.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.885 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:04.885 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:04.885 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:04.886 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:04.886 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:04.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.886 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:04.886 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:04.887 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:04.887 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:04.887 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:04.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:04.887 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:04.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.888 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:04.888 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:04.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.888 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:04.888 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:04.889 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:04.889 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:04.889 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:04.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:04.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.889 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:04.890 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:04.890 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:06:04.890 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:04.890 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:04.890 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:06:04.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:04.891 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:04.891 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:04.891 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:04.891 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:04.891 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:04.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:04.892 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:04.892 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:04.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 10 librados: watch_flush enter 2023-10-30T12:06:04.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 10 librados: watch_flush exit 2023-10-30T12:06:04.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 20 client.44849.objecter shutdown clearing up homeless session... 2023-10-30T12:06:04.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 10 client.44849.objecter successfully canceled tick 2023-10-30T12:06:04.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f29f80392f0 msgr2=0x7f29f803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f29f80392f0 0x7f29f803b7b0 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 crypto rx=0x7f2a181caf80 tx=0x7f2a00008040 comp rx=0 tx=0).stop 2023-10-30T12:06:04.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 10 monclient: shutdown 2023-10-30T12:06:04.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:04.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 msgr2=0x7f2a181c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:04.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 --2- 172.21.15.50:0/2840507247 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a180a2710 0x7f2a181c6ec0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f2a10032000 tx=0x7f2a10004a40 comp rx=0 tx=0).stop 2023-10-30T12:06:04.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a17fff640 1 -- 172.21.15.50:0/2840507247 reap_dead start 2023-10-30T12:06:04.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a17fff640 1 -- 172.21.15.50:0/2840507247 reap_dead start 2023-10-30T12:06:04.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 shutdown_connections 2023-10-30T12:06:04.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 >> 172.21.15.50:0/2840507247 conn(0x7f2a180a6200 msgr2=0x7f2a1806e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:04.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 shutdown_connections 2023-10-30T12:06:04.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 -- 172.21.15.50:0/2840507247 wait complete. 2023-10-30T12:06:04.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:04.797+0000 7f2a1ddc5640 1 librados: shutdown 2023-10-30T12:06:04.897 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:04.897 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:06:04.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:06:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.003+0000 7f83dd90c640 10 monclient: get_monmap_and_config 2023-10-30T12:06:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.003+0000 7f83dd90c640 10 monclient: build_initial_monmap 2023-10-30T12:06:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.003+0000 7f83dd90c640 10 monclient: monmap: 2023-10-30T12:06:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:05.004537+0000 2023-10-30T12:06:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:05.004537+0000 2023-10-30T12:06:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:05.007 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 1 Processor -- start 2023-10-30T12:06:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 1 -- start start 2023-10-30T12:06:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient: init 2023-10-30T12:06:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-10-30T12:06:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83d809bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83d809d3f0 0x7f83d8137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient(hunting): picked mon.noname-a con 0x7f83d809d3f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f83d809c660 0x7f83d8139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient(hunting): picked mon.noname-c con 0x7f83d809c660 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d8098170 con 0x7f83d809bb70 2023-10-30T12:06:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d809cf60 con 0x7f83d809d3f0 2023-10-30T12:06:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d8097b20 con 0x7f83d809c660 2023-10-30T12:06:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83dd90c640 10 monclient(hunting): authenticate will time out at 1210.187711s 2023-10-30T12:06:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83d809d3f0 0x7f83d8137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83d809d3f0 0x7f83d8137960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38748/0 (socket says 172.21.15.50:38748) 2023-10-30T12:06:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 1 -- 172.21.15.50:0/656068569 learned_addr learned my addr 172.21.15.50:0/656068569 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 10 monclient(hunting): get_auth_request con 0x7f83d809d3f0 auth_method 0 2023-10-30T12:06:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.004+0000 7f83cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83d77fe640 1 --2- 172.21.15.50:0/656068569 >> v2:172.21.15.118:3300/0 conn(0x7f83d809c660 0x7f83d8139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83d77fe640 10 monclient(hunting): get_auth_request con 0x7f83d809c660 auth_method 0 2023-10-30T12:06:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83d77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83d77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 10 monclient(hunting): handle_auth_done global_id 84393 payload 274 2023-10-30T12:06:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 1 -- 172.21.15.50:0/656068569 >> v2:172.21.15.118:3300/0 conn(0x7f83d809c660 msgr2=0x7f83d8139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 1 --2- 172.21.15.50:0/656068569 >> v2:172.21.15.118:3300/0 conn(0x7f83d809c660 0x7f83d8139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 1 -- 172.21.15.50:0/656068569 >> v1:172.21.15.118:6789/0 conn(0x7f83d809bb70 legacy=0x7f83d809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 1 monclient: found mon.noname-a 2023-10-30T12:06:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 1 -- 172.21.15.50:0/656068569 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8097e00 con 0x7f83d809d3f0 2023-10-30T12:06:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.005+0000 7f83cffff640 1 --2- 172.21.15.50:0/656068569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8137960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f83d809fe70 tx=0x7f83c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5462b7d5e89d0304 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 1 -- 172.21.15.50:0/656068569 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c00317e0 con 0x7f83d809d3f0 2023-10-30T12:06:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: dump: 2023-10-30T12:06:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.026 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 1 -- 172.21.15.50:0/656068569 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0031940 con 0x7f83d809d3f0 2023-10-30T12:06:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 5 monclient: authenticate success, global_id 84393 2023-10-30T12:06:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 1 -- 172.21.15.50:0/656068569 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0031cd0 con 0x7f83d809d3f0 2023-10-30T12:06:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 4 set_mon_vals no callback set 2023-10-30T12:06:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83d67fc640 10 monclient: dump: 2023-10-30T12:06:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.032 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 10 monclient: shutdown 2023-10-30T12:06:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 -- 172.21.15.50:0/656068569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 msgr2=0x7f83d8137960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 --2- 172.21.15.50:0/656068569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8137960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f83d809fe70 tx=0x7f83c002b880 comp rx=0 tx=0).stop 2023-10-30T12:06:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 -- 172.21.15.50:0/656068569 shutdown_connections 2023-10-30T12:06:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 --2- 172.21.15.50:0/656068569 >> v2:172.21.15.118:3300/0 conn(0x7f83d809c660 0x7f83d8139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 --2- 172.21.15.50:0/656068569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8137960 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 -- 172.21.15.50:0/656068569 >> 172.21.15.50:0/656068569 conn(0x7f83d80a3980 msgr2=0x7f83d80a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 -- 172.21.15.50:0/656068569 shutdown_connections 2023-10-30T12:06:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.006+0000 7f83dd90c640 1 -- 172.21.15.50:0/656068569 wait complete. 2023-10-30T12:06:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 10 monclient: build_initial_monmap 2023-10-30T12:06:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 10 monclient: monmap: 2023-10-30T12:06:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:05.040 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 librados: starting msgr at 2023-10-30T12:06:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 librados: starting objecter 2023-10-30T12:06:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 Processor -- start 2023-10-30T12:06:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 -- start start 2023-10-30T12:06:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 librados: setting wanted keys 2023-10-30T12:06:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 librados: calling monclient init 2023-10-30T12:06:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 10 monclient: init 2023-10-30T12:06:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:06:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d809bb70 0x7f83d8096080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 10 monclient(hunting): picked mon.noname-c con 0x7f83d809bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83d809c660 0x7f83d80965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.007+0000 7f83dd90c640 10 monclient(hunting): picked mon.noname-b con 0x7f83d809c660 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8096b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient(hunting): picked mon.noname-a con 0x7f83d809d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d813a540 con 0x7f83d809d3f0 2023-10-30T12:06:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d8139ed0 con 0x7f83d809c660 2023-10-30T12:06:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d8097150 con 0x7f83d809bb70 2023-10-30T12:06:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient(hunting): authenticate will time out at 1210.191049s 2023-10-30T12:06:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8096b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d809bb70 0x7f83d8096080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8096b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38762/0 (socket says 172.21.15.50:38762) 2023-10-30T12:06:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 -- 172.21.15.50:0/4091442051 learned_addr learned my addr 172.21.15.50:0/4091442051 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): get_auth_request con 0x7f83d809d3f0 auth_method 0 2023-10-30T12:06:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d809bb70 0x7f83d8096080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53334/0 (socket says 172.21.15.50:53334) 2023-10-30T12:06:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 10 monclient(hunting): get_auth_request con 0x7f83d809bb70 auth_method 0 2023-10-30T12:06:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83d809c660 0x7f83d80965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 10 monclient(hunting): get_auth_request con 0x7f83d809c660 auth_method 0 2023-10-30T12:06:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient(hunting): handle_auth_done global_id 84396 payload 824 2023-10-30T12:06:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d809bb70 msgr2=0x7f83d8096080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d809bb70 0x7f83d8096080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83d809c660 msgr2=0x7f83d80965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83d809c660 0x7f83d80965c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 monclient: found mon.noname-a 2023-10-30T12:06:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 -- 172.21.15.50:0/4091442051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c00094d0 con 0x7f83d809d3f0 2023-10-30T12:06:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d77fe640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8096b00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f83d0007e40 tx=0x7f83d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 client.?.objecter ms_handle_connect 0x7f83d809d3f0 2023-10-30T12:06:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0014620 con 0x7f83d809d3f0 2023-10-30T12:06:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: dump: 2023-10-30T12:06:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.057 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d000ee90 con 0x7f83d809d3f0 2023-10-30T12:06:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d00149e0 con 0x7f83d809d3f0 2023-10-30T12:06:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 5 monclient: authenticate success, global_id 84396 2023-10-30T12:06:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient: _renew_subs 2023-10-30T12:06:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d80976e0 con 0x7f83d809d3f0 2023-10-30T12:06:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d5ffb640 4 set_mon_vals no callback set 2023-10-30T12:06:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83d4ff9640 10 monclient: dump: 2023-10-30T12:06:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.063 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 client.84396.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient: _renew_subs 2023-10-30T12:06:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d8137d50 con 0x7f83d809d3f0 2023-10-30T12:06:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.008+0000 7f83dd90c640 1 librados: init done 2023-10-30T12:06:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83dd90c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f83d0014b40 con 0x7f83d809d3f0 2023-10-30T12:06:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83dd90c640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d81372c0 0x7f83d8060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83dd90c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d8073ff0 con 0x7f83d81372c0 2023-10-30T12:06:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d81372c0 0x7f83d8060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 10 monclient: get_auth_request con 0x7f83d81372c0 auth_method 0 2023-10-30T12:06:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d4ff9640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f83b80392f0 0x7f83b803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f83d004b350 con 0x7f83d809d3f0 2023-10-30T12:06:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d4ff9640 10 client.84396.objecter ms_dispatch 0x7f83d8000b90 osd_map(172..172 src has 1..172) v4 2023-10-30T12:06:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d4ff9640 3 client.84396.objecter handle_osd_map got epochs [172,172] > 0 2023-10-30T12:06:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.009+0000 7f83d4ff9640 3 client.84396.objecter handle_osd_map decoding full epoch 172 2023-10-30T12:06:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 20 client.84396.objecter dump_active .. 0 homeless 2023-10-30T12:06:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_done global_id 44855 payload 824 2023-10-30T12:06:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d6ffd640 1 -- 172.21.15.50:0/4091442051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83d8097560 con 0x7f83d81372c0 2023-10-30T12:06:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83cffff640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f83b80392f0 0x7f83b803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83cffff640 10 monclient: get_auth_request con 0x7f83b80392f0 auth_method 0 2023-10-30T12:06:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 client.84396.objecter ms_handle_connect 0x7f83d81372c0 2023-10-30T12:06:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 client.84396.objecter resend_mon_ops 2023-10-30T12:06:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d6ffd640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d81372c0 0x7f83d8060fc0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000fb0 tx=0x7f83c800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c80050d0 con 0x7f83d81372c0 2023-10-30T12:06:05.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 monclient: dump: 2023-10-30T12:06:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.072 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.072 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.072 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.073 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.073 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 client.84396.objecter ms_handle_connect 0x7f83b80392f0 2023-10-30T12:06:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83cffff640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f83b80392f0 0x7f83b803b7b0 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f83c8005230 con 0x7f83d81372c0 2023-10-30T12:06:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d81372c0 msgr2=0x7f83d8060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d81372c0 0x7f83d8060fc0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000fb0 tx=0x7f83c800fc00 comp rx=0 tx=0).stop 2023-10-30T12:06:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d81372c0 msgr2=0x7f83d8060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.010+0000 7f83d4ff9640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d81372c0 0x7f83d8060fc0 secure :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000fb0 tx=0x7f83c800fc00 comp rx=0 tx=0).stop 2023-10-30T12:06:05.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.016+0000 7f83dd90c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:05.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.016+0000 7f83dd90c640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d8061500 0x7f83d802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.016+0000 7f83dd90c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.016+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d809f600 con 0x7f83d8061500 2023-10-30T12:06:05.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d8061500 0x7f83d802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient: get_auth_request con 0x7f83d8061500 auth_method 0 2023-10-30T12:06:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 10 monclient(hunting): handle_auth_done global_id 44858 payload 824 2023-10-30T12:06:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 1 -- 172.21.15.50:0/4091442051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f83d804e160 con 0x7f83d8061500 2023-10-30T12:06:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 10 client.84396.objecter ms_handle_connect 0x7f83d8061500 2023-10-30T12:06:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d6ffd640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d8061500 0x7f83d802b430 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000fb0 tx=0x7f83c8001170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 10 client.84396.objecter resend_mon_ops 2023-10-30T12:06:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c8024070 con 0x7f83d8061500 2023-10-30T12:06:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 10 monclient: dump: 2023-10-30T12:06:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.083 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.083 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.083 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f83c8003960 con 0x7f83d8061500 2023-10-30T12:06:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d8061500 msgr2=0x7f83d802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d8061500 0x7f83d802b430 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000fb0 tx=0x7f83c8001170 comp rx=0 tx=0).stop 2023-10-30T12:06:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d8061500 msgr2=0x7f83d802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.017+0000 7f83d4ff9640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83d8061500 0x7f83d802b430 secure :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000fb0 tx=0x7f83c8001170 comp rx=0 tx=0).stop 2023-10-30T12:06:05.085 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:05.085 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:05.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:05.086 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:05.086 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 56, 2023-10-30T12:06:05.086 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:05.086 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:05.086 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:05.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:05.087 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.087 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:06:05.087 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:05.087 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:05.087 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:05.088 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:05.088 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:05.088 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:05.088 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:05.088 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:05.089 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:05.089 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:05.089 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:05.089 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:05.089 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:05.090 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.090 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:05.090 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:05.090 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:05.090 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:05.091 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:05.091 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:05.091 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:05.091 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:05.091 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:05.091 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:05.092 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:05.092 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:05.092 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.092 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:05.093 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:05.093 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:05.093 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:05.093 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:05.093 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:05.094 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:05.094 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:05.094 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:05.094 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:05.094 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:05.094 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:05.095 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:05.095 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:05.095 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:05.095 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:05.095 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:05.096 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:05.096 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:05.096 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:05.096 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:05.096 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:05.097 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:05.097 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:05.097 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:05.097 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:05.097 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:05.097 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.098 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:05.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.098 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:05.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.098 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:05.099 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:05.099 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:05.099 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:05.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.099 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:05.100 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:05.100 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.100 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:05.100 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:05.101 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.101 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.101 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:05.102 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:05.102 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:05.102 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:05.102 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:05.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.103 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:05.103 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:05.103 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:05.103 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:05.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.104 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:05.104 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:05.104 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.105 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:05.105 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:05.105 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.105 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.106 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:05.106 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:05.106 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:05.106 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:05.107 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:05.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.107 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:05.107 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:05.107 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:05.108 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:05.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.108 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:05.108 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:05.108 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.109 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:05.109 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:05.109 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.109 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.110 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.110 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:05.110 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:05.110 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:05.111 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:05.111 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:05.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.111 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.112 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:05.112 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:05.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.112 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:05.112 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:05.113 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:05.113 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.113 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.113 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:06:05.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.113 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:05.114 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:05.114 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:06:05.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.114 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.114 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:05.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.115 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:05.115 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:05.115 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:06:05.115 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.115 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.116 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:05.116 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:05.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 10 librados: watch_flush enter 2023-10-30T12:06:05.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 10 librados: watch_flush exit 2023-10-30T12:06:05.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 20 client.84396.objecter shutdown clearing up homeless session... 2023-10-30T12:06:05.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 10 client.84396.objecter successfully canceled tick 2023-10-30T12:06:05.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f83b80392f0 msgr2=0x7f83b803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f83b80392f0 0x7f83b803b7b0 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c003d040 comp rx=0 tx=0).stop 2023-10-30T12:06:05.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 10 monclient: shutdown 2023-10-30T12:06:05.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:05.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 msgr2=0x7f83d8096b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8096b00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f83d0007e40 tx=0x7f83d000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83d77fe640 1 -- 172.21.15.50:0/4091442051 reap_dead start 2023-10-30T12:06:05.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 shutdown_connections 2023-10-30T12:06:05.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 --2- 172.21.15.50:0/4091442051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d809d3f0 0x7f83d8096b00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 >> 172.21.15.50:0/4091442051 conn(0x7f83d80a3980 msgr2=0x7f83d806dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:05.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 shutdown_connections 2023-10-30T12:06:05.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 -- 172.21.15.50:0/4091442051 wait complete. 2023-10-30T12:06:05.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.018+0000 7f83dd90c640 1 librados: shutdown 2023-10-30T12:06:05.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:05.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:06:05.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:06:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.226+0000 7fdf7dce1640 10 monclient: get_monmap_and_config 2023-10-30T12:06:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.226+0000 7fdf7dce1640 10 monclient: build_initial_monmap 2023-10-30T12:06:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.226+0000 7fdf7dce1640 10 monclient: monmap: 2023-10-30T12:06:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:05.227701+0000 2023-10-30T12:06:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:05.227701+0000 2023-10-30T12:06:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:05.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 1 Processor -- start 2023-10-30T12:06:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 1 -- start start 2023-10-30T12:06:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient: init 2023-10-30T12:06:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:06:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf78132970 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf780a03d0 0x7fdf7813aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient(hunting): picked mon.noname-b con 0x7fdf780a03d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdf7813b020 0x7fdf7813d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient(hunting): picked mon.noname-c con 0x7fdf7813b020 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:05.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf780a17d0 con 0x7fdf78132970 2023-10-30T12:06:05.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf78133460 con 0x7fdf7813b020 2023-10-30T12:06:05.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf780a1510 con 0x7fdf780a03d0 2023-10-30T12:06:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.227+0000 7fdf7dce1640 10 monclient(hunting): authenticate will time out at 1210.410934s 2023-10-30T12:06:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf780a03d0 0x7fdf7813aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf780a03d0 0x7fdf7813aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53346/0 (socket says 172.21.15.50:53346) 2023-10-30T12:06:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 -- 172.21.15.50:0/1033224926 learned_addr learned my addr 172.21.15.50:0/1033224926 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): get_auth_request con 0x7fdf780a03d0 auth_method 0 2023-10-30T12:06:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 1 --2- 172.21.15.50:0/1033224926 >> v2:172.21.15.118:3300/0 conn(0x7fdf7813b020 0x7fdf7813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 10 monclient(hunting): get_auth_request con 0x7fdf7813b020 auth_method 0 2023-10-30T12:06:05.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient(hunting): handle_auth_done global_id 44861 payload 274 2023-10-30T12:06:05.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 -- 172.21.15.50:0/1033224926 >> v2:172.21.15.118:3300/0 conn(0x7fdf7813b020 msgr2=0x7fdf7813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:05.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 --2- 172.21.15.50:0/1033224926 >> v2:172.21.15.118:3300/0 conn(0x7fdf7813b020 0x7fdf7813d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 -- 172.21.15.50:0/1033224926 >> v1:172.21.15.118:6789/0 conn(0x7fdf78132970 legacy=0x7fdf78132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:05.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 monclient: found mon.noname-b 2023-10-30T12:06:05.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:05.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.228+0000 7fdf76ffd640 1 -- 172.21.15.50:0/1033224926 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7809f680 con 0x7fdf780a03d0 2023-10-30T12:06:05.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf76ffd640 1 --2- 172.21.15.50:0/1033224926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf780a03d0 0x7fdf7813aae0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009870 tx=0x7fdf6002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2d9973c0b4c2ffd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 1 -- 172.21.15.50:0/1033224926 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf600317c0 con 0x7fdf780a03d0 2023-10-30T12:06:05.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:05.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: dump: 2023-10-30T12:06:05.248 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.248 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.248 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.248 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.248 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.248 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.249 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.249 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.249 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.249 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:05.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:05.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:05.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 1 -- 172.21.15.50:0/1033224926 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf60031920 con 0x7fdf780a03d0 2023-10-30T12:06:05.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 5 monclient: authenticate success, global_id 44861 2023-10-30T12:06:05.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:05.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:05.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:05.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 1 -- 172.21.15.50:0/1033224926 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60031cb0 con 0x7fdf780a03d0 2023-10-30T12:06:05.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 4 set_mon_vals no callback set 2023-10-30T12:06:05.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:05.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf767fc640 10 monclient: dump: 2023-10-30T12:06:05.252 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.252 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.252 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.252 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.252 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.253 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.253 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.253 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.253 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.253 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 10 monclient: shutdown 2023-10-30T12:06:05.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:05.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 -- 172.21.15.50:0/1033224926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf780a03d0 msgr2=0x7fdf7813aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 --2- 172.21.15.50:0/1033224926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf780a03d0 0x7fdf7813aae0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009870 tx=0x7fdf6002b860 comp rx=0 tx=0).stop 2023-10-30T12:06:05.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 -- 172.21.15.50:0/1033224926 shutdown_connections 2023-10-30T12:06:05.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 --2- 172.21.15.50:0/1033224926 >> v2:172.21.15.118:3300/0 conn(0x7fdf7813b020 0x7fdf7813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 --2- 172.21.15.50:0/1033224926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf780a03d0 0x7fdf7813aae0 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 -- 172.21.15.50:0/1033224926 >> 172.21.15.50:0/1033224926 conn(0x7fdf780a4800 msgr2=0x7fdf780a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:05.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 -- 172.21.15.50:0/1033224926 shutdown_connections 2023-10-30T12:06:05.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.229+0000 7fdf7dce1640 1 -- 172.21.15.50:0/1033224926 wait complete. 2023-10-30T12:06:05.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 10 monclient: build_initial_monmap 2023-10-30T12:06:05.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 10 monclient: monmap: 2023-10-30T12:06:05.256 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:05.256 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:05.256 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:05.256 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:05.256 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:05.256 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.257 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:05.257 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:05.257 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:05.257 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 1 librados: starting msgr at 2023-10-30T12:06:05.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 1 librados: starting objecter 2023-10-30T12:06:05.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 1 Processor -- start 2023-10-30T12:06:05.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 1 -- start start 2023-10-30T12:06:05.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 1 librados: setting wanted keys 2023-10-30T12:06:05.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 1 librados: calling monclient init 2023-10-30T12:06:05.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.230+0000 7fdf7dce1640 10 monclient: init 2023-10-30T12:06:05.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:05.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:06:05.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf78132970 0x7fdf78131570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): picked mon.noname-b con 0x7fdf78132970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:05.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf7813b020 0x7fdf78131ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): picked mon.noname-a con 0x7fdf7813b020 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:05.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf781ca590 0x7fdf7811a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): picked mon.noname-c con 0x7fdf781ca590 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:05.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf781cb2d0 con 0x7fdf7813b020 2023-10-30T12:06:05.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7811cad0 con 0x7fdf78132970 2023-10-30T12:06:05.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7811d9f0 con 0x7fdf781ca590 2023-10-30T12:06:05.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:05.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf7dce1640 10 monclient(hunting): authenticate will time out at 1210.414339s 2023-10-30T12:06:05.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf781ca590 0x7fdf7811a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf7813b020 0x7fdf78131ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf781ca590 0x7fdf7811a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53350/0 (socket says 172.21.15.50:53350) 2023-10-30T12:06:05.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 -- 172.21.15.50:0/3337439011 learned_addr learned my addr 172.21.15.50:0/3337439011 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:05.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): get_auth_request con 0x7fdf781ca590 auth_method 0 2023-10-30T12:06:05.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 10 monclient(hunting): get_auth_request con 0x7fdf7813b020 auth_method 0 2023-10-30T12:06:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf777fe640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf78132970 0x7fdf78131570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf777fe640 10 monclient(hunting): get_auth_request con 0x7fdf78132970 auth_method 0 2023-10-30T12:06:05.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient(hunting): handle_auth_done global_id 44864 payload 824 2023-10-30T12:06:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf78132970 msgr2=0x7fdf78131570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf78132970 0x7fdf78131570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf7813b020 msgr2=0x7fdf78131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf7813b020 0x7fdf78131ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 monclient: found mon.noname-c 2023-10-30T12:06:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 -- 172.21.15.50:0/3337439011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf600094d0 con 0x7fdf781ca590 2023-10-30T12:06:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf777fe640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf78132970 0x7fdf78131570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:06:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf77fff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf781ca590 0x7fdf7811a430 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009dd0 tx=0x7fdf7000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf6ffff640 10 client.?.objecter ms_handle_connect 0x7fdf781ca590 2023-10-30T12:06:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.231+0000 7fdf6ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70014480 con 0x7fdf781ca590 2023-10-30T12:06:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: dump: 2023-10-30T12:06:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.271 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: _finish_auth 0 2023-10-30T12:06:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 5 monclient: authenticate success, global_id 44864 2023-10-30T12:06:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7000f040 con 0x7fdf781ca590 2023-10-30T12:06:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70014840 con 0x7fdf781ca590 2023-10-30T12:06:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 10 monclient: _renew_subs 2023-10-30T12:06:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf75ffb640 4 set_mon_vals no callback set 2023-10-30T12:06:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7811df20 con 0x7fdf781ca590 2023-10-30T12:06:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 10 monclient: dump: 2023-10-30T12:06:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.275 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 10 client.44864.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 10 monclient: _renew_subs 2023-10-30T12:06:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7809c3e0 con 0x7fdf781ca590 2023-10-30T12:06:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf7dce1640 1 librados: init done 2023-10-30T12:06:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.232+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdf700149a0 con 0x7fdf781ca590 2023-10-30T12:06:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf7dce1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf7dce1640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf780a03d0 0x7fdf7802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf7dce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78071a70 con 0x7fdf780a03d0 2023-10-30T12:06:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf6ffff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf580392f0 0x7fdf5803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5725+0+0 (secure 0 0 0) 0x7fdf7004b4c0 con 0x7fdf781ca590 2023-10-30T12:06:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf6ffff640 10 client.44864.objecter ms_dispatch 0x7fdf78000b90 osd_map(172..172 src has 1..172) v4 2023-10-30T12:06:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf6ffff640 3 client.44864.objecter handle_osd_map got epochs [172,172] > 0 2023-10-30T12:06:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf6ffff640 3 client.44864.objecter handle_osd_map decoding full epoch 172 2023-10-30T12:06:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf6ffff640 20 client.44864.objecter dump_active .. 0 homeless 2023-10-30T12:06:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf780a03d0 0x7fdf7802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient: get_auth_request con 0x7fdf780a03d0 auth_method 0 2023-10-30T12:06:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf76ffd640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf580392f0 0x7fdf5803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf76ffd640 10 monclient: get_auth_request con 0x7fdf580392f0 auth_method 0 2023-10-30T12:06:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 10 monclient(hunting): handle_auth_done global_id 94168 payload 824 2023-10-30T12:06:05.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf777fe640 1 -- 172.21.15.50:0/3337439011 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf7809f960 con 0x7fdf780a03d0 2023-10-30T12:06:05.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf6ffff640 10 client.44864.objecter ms_handle_connect 0x7fdf580392f0 2023-10-30T12:06:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.233+0000 7fdf76ffd640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf580392f0 0x7fdf5803b7b0 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7fdf60036000 tx=0x7fdf6003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 10 client.44864.objecter ms_handle_connect 0x7fdf780a03d0 2023-10-30T12:06:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 10 client.44864.objecter resend_mon_ops 2023-10-30T12:06:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf777fe640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf780a03d0 0x7fdf7802b430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf7813a660 tx=0x7fdf6800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf680050d0 con 0x7fdf780a03d0 2023-10-30T12:06:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:05.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 10 monclient: dump: 2023-10-30T12:06:05.282 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.282 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.282 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.283 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.283 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.283 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.283 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.283 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.283 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.284 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fdf68005230 con 0x7fdf780a03d0 2023-10-30T12:06:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf780a03d0 msgr2=0x7fdf7802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf780a03d0 0x7fdf7802b430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf7813a660 tx=0x7fdf6800eb90 comp rx=0 tx=0).stop 2023-10-30T12:06:05.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf780a03d0 msgr2=0x7fdf7802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:05.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.234+0000 7fdf6ffff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf780a03d0 0x7fdf7802b430 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf7813a660 tx=0x7fdf6800eb90 comp rx=0 tx=0).stop 2023-10-30T12:06:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.240+0000 7fdf7dce1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.240+0000 7fdf7dce1640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf7802b970 0x7fdf7802bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.240+0000 7fdf7dce1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.240+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78074800 con 0x7fdf7802b970 2023-10-30T12:06:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf7802b970 0x7fdf7802bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient: get_auth_request con 0x7fdf7802b970 auth_method 0 2023-10-30T12:06:05.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 10 monclient(hunting): handle_auth_done global_id 94171 payload 824 2023-10-30T12:06:05.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 1 -- 172.21.15.50:0/3337439011 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdf7811db60 con 0x7fdf7802b970 2023-10-30T12:06:05.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf777fe640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf7802b970 0x7fdf7802bd80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf7813a660 tx=0x7fdf68001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf6ffff640 10 client.44864.objecter ms_handle_connect 0x7fdf7802b970 2023-10-30T12:06:05.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf6ffff640 10 client.44864.objecter resend_mon_ops 2023-10-30T12:06:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68023070 con 0x7fdf7802b970 2023-10-30T12:06:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.241+0000 7fdf6ffff640 10 monclient: dump: 2023-10-30T12:06:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.292 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fdf68007680 con 0x7fdf7802b970 2023-10-30T12:06:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf6ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf6ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf7802b970 msgr2=0x7fdf7802bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf6ffff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf7802b970 0x7fdf7802bd80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf7813a660 tx=0x7fdf68001120 comp rx=0 tx=0).stop 2023-10-30T12:06:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf6ffff640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf7802b970 msgr2=0x7fdf7802bd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf6ffff640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdf7802b970 0x7fdf7802bd80 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf7813a660 tx=0x7fdf68001120 comp rx=0 tx=0).stop 2023-10-30T12:06:05.294 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 56, 2023-10-30T12:06:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 6, 2023-10-30T12:06:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:05.297 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:05.297 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:05.297 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:05.297 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:05.297 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:05.298 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:05.298 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:05.298 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:05.298 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:05.298 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.299 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:05.299 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:05.299 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:05.299 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:05.299 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:05.299 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:05.300 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:05.300 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:05.300 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:05.300 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:05.300 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:05.301 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:05.301 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.301 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:05.301 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:05.302 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:05.302 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:05.302 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:05.302 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:05.302 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:05.303 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:05.303 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:05.303 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:05.303 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:05.303 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:05.303 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:05.304 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:05.304 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:05.304 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:05.304 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:05.304 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:05.305 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:05.305 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:05.305 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:05.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:05.305 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:05.305 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:05.312 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:05.312 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:05.312 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:05.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.312 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:05.313 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:05.313 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.313 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:05.315 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:05.315 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:05.315 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:05.315 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:05.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.316 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:05.316 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:05.316 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:05.316 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:05.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.317 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:05.317 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:05.317 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.318 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:05.318 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:05.318 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:05.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.318 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.319 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:05.319 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:05.319 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:05.319 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:05.320 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:05.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.320 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.320 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:05.320 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:05.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.321 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:05.321 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:05.321 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:05.321 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.322 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:05.322 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:05.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:05.322 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:05.322 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:05.323 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:06:05.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:05.323 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:05.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:05.323 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:05.323 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 10 librados: watch_flush enter 2023-10-30T12:06:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 10 librados: watch_flush exit 2023-10-30T12:06:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 20 client.44864.objecter shutdown clearing up homeless session... 2023-10-30T12:06:05.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 10 client.44864.objecter successfully canceled tick 2023-10-30T12:06:05.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf580392f0 msgr2=0x7fdf5803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdf580392f0 0x7fdf5803b7b0 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7fdf60036000 tx=0x7fdf6003d040 comp rx=0 tx=0).stop 2023-10-30T12:06:05.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 10 monclient: shutdown 2023-10-30T12:06:05.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf781ca590 msgr2=0x7fdf7811a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 --2- 172.21.15.50:0/3337439011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf781ca590 0x7fdf7811a430 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009dd0 tx=0x7fdf7000c490 comp rx=0 tx=0).stop 2023-10-30T12:06:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf77fff640 1 -- 172.21.15.50:0/3337439011 reap_dead start 2023-10-30T12:06:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf77fff640 1 -- 172.21.15.50:0/3337439011 reap_dead start 2023-10-30T12:06:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 shutdown_connections 2023-10-30T12:06:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 >> 172.21.15.50:0/3337439011 conn(0x7fdf780a4800 msgr2=0x7fdf780a27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 shutdown_connections 2023-10-30T12:06:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 -- 172.21.15.50:0/3337439011 wait complete. 2023-10-30T12:06:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.242+0000 7fdf7dce1640 1 librados: shutdown 2023-10-30T12:06:05.329 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:05.329 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:06:05.329 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:06:05.329 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:06:05.329 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:06:05.330 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:06:05.330 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:06:05.341 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:06:05.341 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:06:05.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.484+0000 7f361d56e640 10 monclient: get_monmap_and_config 2023-10-30T12:06:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.484+0000 7f361d56e640 10 monclient: build_initial_monmap 2023-10-30T12:06:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.484+0000 7f361d56e640 10 monclient: monmap: 2023-10-30T12:06:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:05.487 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.487 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:05.485757+0000 2023-10-30T12:06:05.487 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:05.485757+0000 2023-10-30T12:06:05.487 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:05.487 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.488 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:05.488 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:05.488 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:05.488 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:05.488 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:05.489 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 1 Processor -- start 2023-10-30T12:06:05.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 1 -- start start 2023-10-30T12:06:05.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient: init 2023-10-30T12:06:05.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:05.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:06:05.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f36180707d0 0x7f3618070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient(hunting): picked mon.noname-c con 0x7f36180707d0 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:05.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36180a0170 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:05.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f36180710f0 0x7f3618134d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient(hunting): picked mon.noname-a con 0x7f36180710f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:05.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36180a11f0 con 0x7f36180a0170 2023-10-30T12:06:05.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361809f4b0 con 0x7f36180710f0 2023-10-30T12:06:05.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361809d9d0 con 0x7f36180707d0 2023-10-30T12:06:05.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:05.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f361d56e640 10 monclient(hunting): authenticate will time out at 1210.668908s 2023-10-30T12:06:05.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f36177fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f36180710f0 msgr2=0x7f3618134d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:05.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.485+0000 7f36177fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f36180710f0 0x7f3618134d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:05.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f36180707d0 0x7f3618070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f36180707d0 0x7f3618070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:40380/0 (socket says 172.21.15.50:40380) 2023-10-30T12:06:05.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 -- 172.21.15.50:0/4040866278 learned_addr learned my addr 172.21.15.50:0/4040866278 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:05.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): get_auth_request con 0x7f36180707d0 auth_method 0 2023-10-30T12:06:05.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient(hunting): handle_auth_done global_id 94174 payload 274 2023-10-30T12:06:05.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 -- 172.21.15.50:0/4040866278 >> v2:172.21.15.50:3300/0 conn(0x7f36180710f0 msgr2=0x7f3618134d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:05.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 --2- 172.21.15.50:0/4040866278 >> v2:172.21.15.50:3300/0 conn(0x7f36180710f0 0x7f3618134d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 -- 172.21.15.50:0/4040866278 >> v1:172.21.15.118:6789/0 conn(0x7f36180a0170 legacy=0x7f36181345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:05.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 monclient: found mon.noname-c 2023-10-30T12:06:05.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:05.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:06:05.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.486+0000 7f3616ffd640 1 -- 172.21.15.50:0/4040866278 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36180a13f0 con 0x7f36180707d0 2023-10-30T12:06:05.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f3616ffd640 1 --2- 172.21.15.50:0/4040866278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180707d0 0x7f3618070bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f36180a1180 tx=0x7f360802b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cffd4296b86488c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 1 -- 172.21.15.50:0/4040866278 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608031800 con 0x7f36180707d0 2023-10-30T12:06:05.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:05.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: dump: 2023-10-30T12:06:05.504 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.504 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.504 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.505 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.505 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.505 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.505 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.505 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.505 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.506 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:05.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:05.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:05.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 1 -- 172.21.15.50:0/4040866278 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608031960 con 0x7f36180707d0 2023-10-30T12:06:05.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:05.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f361d56e640 5 monclient: authenticate success, global_id 94174 2023-10-30T12:06:05.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 1 -- 172.21.15.50:0/4040866278 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608031cf0 con 0x7f36180707d0 2023-10-30T12:06:05.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:05.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f36167fc640 10 monclient: dump: 2023-10-30T12:06:05.508 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.508 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.508 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.508 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.508 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.509 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.509 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.509 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.509 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.509 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f361d56e640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:05.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.487+0000 7f361d56e640 4 set_mon_vals no callback set 2023-10-30T12:06:05.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 10 monclient: shutdown 2023-10-30T12:06:05.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:05.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 -- 172.21.15.50:0/4040866278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180707d0 msgr2=0x7f3618070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 --2- 172.21.15.50:0/4040866278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180707d0 0x7f3618070bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f36180a1180 tx=0x7f360802b8a0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 -- 172.21.15.50:0/4040866278 shutdown_connections 2023-10-30T12:06:05.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 --2- 172.21.15.50:0/4040866278 >> v2:172.21.15.50:3300/0 conn(0x7f36180710f0 0x7f3618134d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 --2- 172.21.15.50:0/4040866278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180707d0 0x7f3618070bb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 -- 172.21.15.50:0/4040866278 >> 172.21.15.50:0/4040866278 conn(0x7f36180a47f0 msgr2=0x7f36180a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:05.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 -- 172.21.15.50:0/4040866278 shutdown_connections 2023-10-30T12:06:05.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 -- 172.21.15.50:0/4040866278 wait complete. 2023-10-30T12:06:05.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 10 monclient: build_initial_monmap 2023-10-30T12:06:05.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 10 monclient: monmap: 2023-10-30T12:06:05.512 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:05.512 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:05.513 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:05.513 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:05.513 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:05.513 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.513 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:05.513 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:05.514 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:05.514 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 librados: starting msgr at 2023-10-30T12:06:05.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.488+0000 7f361d56e640 1 librados: starting objecter 2023-10-30T12:06:05.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 Processor -- start 2023-10-30T12:06:05.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 -- start start 2023-10-30T12:06:05.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 librados: setting wanted keys 2023-10-30T12:06:05.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 librados: calling monclient init 2023-10-30T12:06:05.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient: init 2023-10-30T12:06:05.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:05.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:06:05.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f36180707d0 0x7f36181c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): picked mon.noname-c con 0x7f36180707d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:05.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36180710f0 0x7f36181b7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): picked mon.noname-a con 0x7f36180710f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:05.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180a0170 0x7f36181ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): picked mon.noname-b con 0x7f36180a0170 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:05.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36181a49f0 con 0x7f36180710f0 2023-10-30T12:06:05.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36181c7780 con 0x7f36180a0170 2023-10-30T12:06:05.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:05.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36181a4080 con 0x7f36180707d0 2023-10-30T12:06:05.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:05.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f361d56e640 10 monclient(hunting): authenticate will time out at 1210.672579s 2023-10-30T12:06:05.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f360ffff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36180710f0 msgr2=0x7f36181b7980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:05.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f360ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36180710f0 0x7f36181b7980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:05.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f36180707d0 0x7f36181c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f36180707d0 0x7f36181c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53354/0 (socket says 172.21.15.50:53354) 2023-10-30T12:06:05.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 -- 172.21.15.50:0/1290970268 learned_addr learned my addr 172.21.15.50:0/1290970268 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:05.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): get_auth_request con 0x7f36180707d0 auth_method 0 2023-10-30T12:06:05.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:05.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f36177fe640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180a0170 0x7f36181ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient(hunting): handle_auth_done global_id 44867 payload 824 2023-10-30T12:06:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 -- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180a0170 msgr2=0x7f36181ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180a0170 0x7f36181ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 -- 172.21.15.50:0/1290970268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36180710f0 msgr2=0x7f36181b7980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36180710f0 0x7f36181b7980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 monclient: found mon.noname-c 2023-10-30T12:06:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f3616ffd640 1 -- 172.21.15.50:0/1290970268 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36080094d0 con 0x7f36180707d0 2023-10-30T12:06:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f36177fe640 10 monclient: get_auth_request con 0x7f36180a0170 auth_method 0 2023-10-30T12:06:05.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.489+0000 7f36177fe640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180a0170 0x7f36181ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:06:05.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f3616ffd640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f36180707d0 0x7f36181c6c30 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f3608000c00 tx=0x7f36080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 client.?.objecter ms_handle_connect 0x7f36180707d0 2023-10-30T12:06:05.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:05.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 1 -- 172.21.15.50:0/1290970268 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360803ad30 con 0x7f36180707d0 2023-10-30T12:06:05.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:05.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: dump: 2023-10-30T12:06:05.524 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.524 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.524 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.525 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.525 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.525 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.525 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.525 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.526 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.526 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:06:05.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:06:05.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:05.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 1 -- 172.21.15.50:0/1290970268 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360803ae90 con 0x7f36180707d0 2023-10-30T12:06:05.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 5 monclient: authenticate success, global_id 44867 2023-10-30T12:06:05.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 10 monclient: _renew_subs 2023-10-30T12:06:05.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361809d6a0 con 0x7f36180707d0 2023-10-30T12:06:05.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:05.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 1 -- 172.21.15.50:0/1290970268 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080445f0 con 0x7f36180707d0 2023-10-30T12:06:05.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:05.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:05.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f360f7fe640 10 monclient: dump: 2023-10-30T12:06:05.528 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:05.528 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:05.529 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.529 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:05.529 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:05.529 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:05.529 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:05.529 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:05.530 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:05.530 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:05.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f3615ffb640 4 set_mon_vals no callback set 2023-10-30T12:06:05.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 10 client.44867.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:05.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 10 monclient: _renew_subs 2023-10-30T12:06:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36181a5d90 con 0x7f36180707d0 2023-10-30T12:06:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.490+0000 7f361d56e640 1 librados: init done 2023-10-30T12:06:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360f7fe640 1 -- 172.21.15.50:0/1290970268 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f360804d450 con 0x7f36180707d0 2023-10-30T12:06:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f361d56e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f361d56e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f361d56e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36181a6170 con 0x7f36180707d0 2023-10-30T12:06:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360f7fe640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35f80392a0 0x7f35f803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360f7fe640 1 -- 172.21.15.50:0/1290970268 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f360802e070 con 0x7f36180707d0 2023-10-30T12:06:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360f7fe640 10 client.44867.objecter ms_dispatch 0x7f3618000b90 osd_map(173..173 src has 1..173) v4 2023-10-30T12:06:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360f7fe640 3 client.44867.objecter handle_osd_map got epochs [173,173] > 0 2023-10-30T12:06:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360f7fe640 3 client.44867.objecter handle_osd_map decoding full epoch 173 2023-10-30T12:06:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360f7fe640 20 client.44867.objecter dump_active .. 0 homeless 2023-10-30T12:06:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360ffff640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35f80392a0 0x7f35f803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.491+0000 7f360ffff640 10 monclient: get_auth_request con 0x7f35f80392a0 auth_method 0 2023-10-30T12:06:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.492+0000 7f360ffff640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35f80392a0 0x7f35f803b760 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f3600001030 tx=0x7f3600007a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.492+0000 7f360f7fe640 10 client.44867.objecter ms_handle_connect 0x7f35f80392a0 2023-10-30T12:06:05.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.495+0000 7f360f7fe640 1 -- 172.21.15.50:0/1290970268 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f360804b940 con 0x7f36180707d0 2023-10-30T12:06:05.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.495+0000 7f360f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:05.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.495+0000 7f360f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.743+0000 7f361d56e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.743+0000 7f361d56e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.743+0000 7f361d56e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.743+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f361804e160 con 0x7f36180707d0 2023-10-30T12:06:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.743+0000 7f360f7fe640 1 -- 172.21.15.50:0/1290970268 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f360804b350 con 0x7f36180707d0 2023-10-30T12:06:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.743+0000 7f360f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.743+0000 7f360f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:05.984 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 10 librados: watch_flush enter 2023-10-30T12:06:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 10 librados: watch_flush exit 2023-10-30T12:06:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 20 client.44867.objecter shutdown clearing up homeless session... 2023-10-30T12:06:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 10 client.44867.objecter successfully canceled tick 2023-10-30T12:06:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35f80392a0 msgr2=0x7f35f803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35f80392a0 0x7f35f803b760 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f3600001030 tx=0x7f3600007a30 comp rx=0 tx=0).stop 2023-10-30T12:06:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 10 monclient: shutdown 2023-10-30T12:06:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f36180707d0 msgr2=0x7f36181c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.745+0000 7f361d56e640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f36180707d0 0x7f36181c6c30 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f3608000c00 tx=0x7f36080046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 shutdown_connections 2023-10-30T12:06:05.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35f80392a0 0x7f35f803b760 unknown :-1 s=CLOSED pgs=3978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36180a0170 0x7f36181ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36180710f0 0x7f36181b7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 --2- 172.21.15.50:0/1290970268 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f36180707d0 0x7f36181c6c30 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 >> 172.21.15.50:0/1290970268 conn(0x7f36180a47f0 msgr2=0x7f36181b9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 shutdown_connections 2023-10-30T12:06:05.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 -- 172.21.15.50:0/1290970268 wait complete. 2023-10-30T12:06:05.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:05.746+0000 7f361d56e640 1 librados: shutdown 2023-10-30T12:06:08.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:09.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.107+0000 7f8543456640 10 monclient: get_monmap_and_config 2023-10-30T12:06:09.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.107+0000 7f8543456640 10 monclient: build_initial_monmap 2023-10-30T12:06:09.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.107+0000 7f8543456640 10 monclient: monmap: 2023-10-30T12:06:09.109 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:09.109 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:09.109 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:09.108176+0000 2023-10-30T12:06:09.110 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:09.108176+0000 2023-10-30T12:06:09.110 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:09.110 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:09.110 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:09.110 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:09.111 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:09.111 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:09.111 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:09.111 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:09.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.107+0000 7f8543456640 1 Processor -- start 2023-10-30T12:06:09.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 1 -- start start 2023-10-30T12:06:09.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient: init 2023-10-30T12:06:09.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:09.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-10-30T12:06:09.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f853c0a4b60 0x7f853c1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:09.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient(hunting): picked mon.noname-a con 0x7f853c0a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:09.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c099a40 0x7f853c132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:09.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient(hunting): picked mon.noname-b con 0x7f853c099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:09.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f853c0a2350 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:09.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c133120 con 0x7f853c0a2350 2023-10-30T12:06:09.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:09.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853c0a3d70 con 0x7f853c0a4b60 2023-10-30T12:06:09.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:09.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c0a4f40 con 0x7f853c099a40 2023-10-30T12:06:09.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:09.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f8543456640 10 monclient(hunting): authenticate will time out at 1214.291521s 2023-10-30T12:06:09.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85411f2640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f853c0a4b60 msgr2=0x7f853c1326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:09.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85411f2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f853c0a4b60 0x7f853c1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:09.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c099a40 0x7f853c132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:09.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c099a40 0x7f853c132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59370/0 (socket says 172.21.15.50:59370) 2023-10-30T12:06:09.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 1 -- 172.21.15.50:0/3183704069 learned_addr learned my addr 172.21.15.50:0/3183704069 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:09.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): get_auth_request con 0x7f853c099a40 auth_method 0 2023-10-30T12:06:09.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:09.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:09.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:09.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:09.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:09.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:09.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 10 monclient(hunting): handle_auth_done global_id 44870 payload 274 2023-10-30T12:06:09.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 1 -- 172.21.15.50:0/3183704069 >> v2:172.21.15.50:3300/0 conn(0x7f853c0a4b60 msgr2=0x7f853c1326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:09.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 1 --2- 172.21.15.50:0/3183704069 >> v2:172.21.15.50:3300/0 conn(0x7f853c0a4b60 0x7f853c1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.108+0000 7f85409f1640 1 -- 172.21.15.50:0/3183704069 >> v1:172.21.15.118:6789/0 conn(0x7f853c0a2350 legacy=0x7f853c0a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:09.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f85409f1640 10 monclient: _finish_hunting 0 2023-10-30T12:06:09.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f85409f1640 1 monclient: found mon.noname-b 2023-10-30T12:06:09.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f85409f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:09.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f85409f1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:09.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f85409f1640 1 -- 172.21.15.50:0/3183704069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0a4150 con 0x7f853c099a40 2023-10-30T12:06:09.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f85409f1640 1 --2- 172.21.15.50:0/3183704069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c099a40 0x7f853c132be0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f852c009870 tx=0x7f852c02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac2133c8bf2e8ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:09.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 1 -- 172.21.15.50:0/3183704069 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0317e0 con 0x7f853c099a40 2023-10-30T12:06:09.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:09.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:09.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 10 monclient: dump: 2023-10-30T12:06:09.127 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:09.127 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:09.127 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.127 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.128 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:09.128 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:09.128 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:09.128 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:09.128 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:09.129 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:09.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 10 monclient: _finish_auth 0 2023-10-30T12:06:09.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:09.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:09.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8533fff640 1 -- 172.21.15.50:0/3183704069 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c031940 con 0x7f853c099a40 2023-10-30T12:06:09.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.109+0000 7f8543456640 5 monclient: authenticate success, global_id 44870 2023-10-30T12:06:09.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:09.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8533fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:09.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:09.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 4 set_mon_vals no callback set 2023-10-30T12:06:09.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8533fff640 1 -- 172.21.15.50:0/3183704069 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c031cd0 con 0x7f853c099a40 2023-10-30T12:06:09.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:09.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8533fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:09.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8533fff640 10 monclient: dump: 2023-10-30T12:06:09.131 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:09.132 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:09.132 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.132 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.132 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:09.132 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:09.132 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:09.133 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:09.133 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:09.133 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:09.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 10 monclient: shutdown 2023-10-30T12:06:09.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:09.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 -- 172.21.15.50:0/3183704069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c099a40 msgr2=0x7f853c132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:09.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 --2- 172.21.15.50:0/3183704069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c099a40 0x7f853c132be0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f852c009870 tx=0x7f852c02b880 comp rx=0 tx=0).stop 2023-10-30T12:06:09.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 -- 172.21.15.50:0/3183704069 shutdown_connections 2023-10-30T12:06:09.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 --2- 172.21.15.50:0/3183704069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c099a40 0x7f853c132be0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 --2- 172.21.15.50:0/3183704069 >> v2:172.21.15.50:3300/0 conn(0x7f853c0a4b60 0x7f853c1326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 -- 172.21.15.50:0/3183704069 >> 172.21.15.50:0/3183704069 conn(0x7f853c0a8060 msgr2=0x7f853c0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:09.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 -- 172.21.15.50:0/3183704069 shutdown_connections 2023-10-30T12:06:09.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.110+0000 7f8543456640 1 -- 172.21.15.50:0/3183704069 wait complete. 2023-10-30T12:06:09.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 10 monclient: build_initial_monmap 2023-10-30T12:06:09.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 10 monclient: monmap: 2023-10-30T12:06:09.136 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:09.136 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:09.136 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:09.136 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:09.136 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:09.137 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:09.137 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:09.137 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:09.137 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:09.137 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:09.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 1 librados: starting msgr at 2023-10-30T12:06:09.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 1 librados: starting objecter 2023-10-30T12:06:09.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 1 Processor -- start 2023-10-30T12:06:09.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 1 -- start start 2023-10-30T12:06:09.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 1 librados: setting wanted keys 2023-10-30T12:06:09.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 1 librados: calling monclient init 2023-10-30T12:06:09.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.111+0000 7f8543456640 10 monclient: init 2023-10-30T12:06:09.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:09.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:06:09.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f853c0a2350 0x7f853c1ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:09.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): picked mon.noname-b con 0x7f853c0a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:09.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f853c0a4b60 0x7f853c1cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:09.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): picked mon.noname-a con 0x7f853c0a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:09.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c11e440 0x7f853c11e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:09.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): picked mon.noname-c con 0x7f853c11e440 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:09.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c131230 con 0x7f853c0a4b60 2023-10-30T12:06:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c1206f0 con 0x7f853c0a2350 2023-10-30T12:06:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853c122260 con 0x7f853c11e440 2023-10-30T12:06:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient(hunting): authenticate will time out at 1214.295111s 2023-10-30T12:06:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85409f1640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f853c0a4b60 msgr2=0x7f853c1cae00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85409f1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f853c0a4b60 0x7f853c1cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c11e440 0x7f853c11e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c11e440 0x7f853c11e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59372/0 (socket says 172.21.15.50:59372) 2023-10-30T12:06:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 -- 172.21.15.50:0/637268715 learned_addr learned my addr 172.21.15.50:0/637268715 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): get_auth_request con 0x7f853c11e440 auth_method 0 2023-10-30T12:06:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f853c0a2350 0x7f853c1ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 10 monclient(hunting): get_auth_request con 0x7f853c0a2350 auth_method 0 2023-10-30T12:06:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85411f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient(hunting): handle_auth_done global_id 44873 payload 824 2023-10-30T12:06:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 -- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f853c0a2350 msgr2=0x7f853c1ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f853c0a2350 0x7f853c1ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 -- 172.21.15.50:0/637268715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f853c0a4b60 msgr2=0x7f853c1cae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f853c0a4b60 0x7f853c1cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient: _finish_hunting 0 2023-10-30T12:06:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 monclient: found mon.noname-c 2023-10-30T12:06:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 -- 172.21.15.50:0/637268715 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0094d0 con 0x7f853c11e440 2023-10-30T12:06:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85419f3640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c11e440 0x7f853c11e8d0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f8534001eb0 tx=0x7f853400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 client.?.objecter ms_handle_connect 0x7f853c11e440 2023-10-30T12:06:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 1 -- 172.21.15.50:0/637268715 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85340052d0 con 0x7f853c11e440 2023-10-30T12:06:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 monclient: dump: 2023-10-30T12:06:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:09.151 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:09.151 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:09.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 monclient: _finish_auth 0 2023-10-30T12:06:09.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:06:09.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 1 -- 172.21.15.50:0/637268715 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534005430 con 0x7f853c11e440 2023-10-30T12:06:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 5 monclient: authenticate success, global_id 44873 2023-10-30T12:06:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient: _renew_subs 2023-10-30T12:06:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c0a18d0 con 0x7f853c11e440 2023-10-30T12:06:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 1 -- 172.21.15.50:0/637268715 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534003810 con 0x7f853c11e440 2023-10-30T12:06:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f8531ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8531ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8531ffb640 10 monclient: dump: 2023-10-30T12:06:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:09.155 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8543456640 10 client.44873.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.112+0000 7f85337fe640 4 set_mon_vals no callback set 2023-10-30T12:06:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8543456640 10 monclient: _renew_subs 2023-10-30T12:06:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8543456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c1200e0 con 0x7f853c11e440 2023-10-30T12:06:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8543456640 1 librados: init done 2023-10-30T12:06:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.113+0000 7f8531ffb640 1 -- 172.21.15.50:0/637268715 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8534003970 con 0x7f853c11e440 2023-10-30T12:06:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8531ffb640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85200392a0 0x7f852003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8531ffb640 1 -- 172.21.15.50:0/637268715 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f853404a340 con 0x7f853c11e440 2023-10-30T12:06:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8531ffb640 10 client.44873.objecter ms_dispatch 0x7f853c000b90 osd_map(173..173 src has 1..173) v4 2023-10-30T12:06:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8531ffb640 3 client.44873.objecter handle_osd_map got epochs [173,173] > 0 2023-10-30T12:06:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8531ffb640 3 client.44873.objecter handle_osd_map decoding full epoch 173 2023-10-30T12:06:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8543456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8543456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8543456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8531ffb640 20 client.44873.objecter dump_active .. 0 homeless 2023-10-30T12:06:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c1220c0 con 0x7f853c11e440 2023-10-30T12:06:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f85411f2640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85200392a0 0x7f852003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.114+0000 7f85411f2640 10 monclient: get_auth_request con 0x7f85200392a0 auth_method 0 2023-10-30T12:06:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.115+0000 7f8531ffb640 10 client.44873.objecter ms_handle_connect 0x7f85200392a0 2023-10-30T12:06:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.115+0000 7f85411f2640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85200392a0 0x7f852003b760 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f8538009420 tx=0x7f853800d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.118+0000 7f8531ffb640 1 -- 172.21.15.50:0/637268715 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f853c1220c0 con 0x7f853c11e440 2023-10-30T12:06:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.118+0000 7f8531ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.118+0000 7f8531ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:09.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.361+0000 7f8543456640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:09.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.361+0000 7f8543456640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:09.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.361+0000 7f8543456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.361+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f853c1220c0 con 0x7f853c11e440 2023-10-30T12:06:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.361+0000 7f8531ffb640 1 -- 172.21.15.50:0/637268715 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f853c1220c0 con 0x7f853c11e440 2023-10-30T12:06:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.361+0000 7f8531ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.361+0000 7f8531ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:09.365 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 10 librados: watch_flush enter 2023-10-30T12:06:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 10 librados: watch_flush exit 2023-10-30T12:06:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 20 client.44873.objecter shutdown clearing up homeless session... 2023-10-30T12:06:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 10 client.44873.objecter successfully canceled tick 2023-10-30T12:06:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85200392a0 msgr2=0x7f852003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85200392a0 0x7f852003b760 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f8538009420 tx=0x7f853800d040 comp rx=0 tx=0).stop 2023-10-30T12:06:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 10 monclient: shutdown 2023-10-30T12:06:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:09.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c11e440 msgr2=0x7f853c11e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:09.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c11e440 0x7f853c11e8d0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f8534001eb0 tx=0x7f853400c890 comp rx=0 tx=0).stop 2023-10-30T12:06:09.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.363+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 shutdown_connections 2023-10-30T12:06:09.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85200392a0 0x7f852003b760 unknown :-1 s=CLOSED pgs=3979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f853c11e440 0x7f853c11e8d0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f853c0a4b60 0x7f853c1cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 --2- 172.21.15.50:0/637268715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f853c0a2350 0x7f853c1ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:09.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 >> 172.21.15.50:0/637268715 conn(0x7f853c0a8060 msgr2=0x7f853c134610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:09.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 shutdown_connections 2023-10-30T12:06:09.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 -- 172.21.15.50:0/637268715 wait complete. 2023-10-30T12:06:09.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:09.364+0000 7f8543456640 1 librados: shutdown 2023-10-30T12:06:12.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.492+0000 7f894bd25640 10 monclient: get_monmap_and_config 2023-10-30T12:06:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.492+0000 7f894bd25640 10 monclient: build_initial_monmap 2023-10-30T12:06:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.492+0000 7f894bd25640 10 monclient: monmap: 2023-10-30T12:06:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:12.493848+0000 2023-10-30T12:06:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:12.493848+0000 2023-10-30T12:06:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:12.497 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:12.497 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:12.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.493+0000 7f894bd25640 1 Processor -- start 2023-10-30T12:06:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.493+0000 7f894bd25640 1 -- start start 2023-10-30T12:06:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.493+0000 7f894bd25640 10 monclient: init 2023-10-30T12:06:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-10-30T12:06:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f89440707d0 0x7f894409e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient(hunting): picked mon.noname-a con 0x7f89440707d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8944070bb0 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f894413aba0 0x7f894413cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient(hunting): picked mon.noname-c con 0x7f894413aba0 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894409ee40 con 0x7f8944070bb0 2023-10-30T12:06:12.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:12.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894409f4f0 con 0x7f89440707d0 2023-10-30T12:06:12.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:12.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894409f7d0 con 0x7f894413aba0 2023-10-30T12:06:12.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f8949ac1640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f89440707d0 msgr2=0x7f894409e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f89492c0640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8944070bb0 legacy=0x7f8944071020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894bd25640 10 monclient(hunting): authenticate will time out at 1217.677238s 2023-10-30T12:06:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f8949ac1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f89440707d0 0x7f894409e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f894413aba0 0x7f894413cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f894413aba0 0x7f894413cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:49292/0 (socket says 172.21.15.50:49292) 2023-10-30T12:06:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 -- 172.21.15.50:0/1381023159 learned_addr learned my addr 172.21.15.50:0/1381023159 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): get_auth_request con 0x7f894413aba0 auth_method 0 2023-10-30T12:06:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient(hunting): handle_auth_done global_id 94183 payload 274 2023-10-30T12:06:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 -- 172.21.15.50:0/1381023159 >> v2:172.21.15.50:3300/0 conn(0x7f89440707d0 msgr2=0x7f894409e7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 --2- 172.21.15.50:0/1381023159 >> v2:172.21.15.50:3300/0 conn(0x7f89440707d0 0x7f894409e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 -- 172.21.15.50:0/1381023159 >> v1:172.21.15.50:6789/0 conn(0x7f8944070bb0 legacy=0x7f8944071020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient: _finish_hunting 0 2023-10-30T12:06:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 monclient: found mon.noname-c 2023-10-30T12:06:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:06:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.494+0000 7f894a2c2640 1 -- 172.21.15.50:0/1381023159 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894409fab0 con 0x7f894413aba0 2023-10-30T12:06:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f894a2c2640 1 --2- 172.21.15.50:0/1381023159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894413cf90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f89440a1500 tx=0x7f893c02bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5616623656bba628 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 1 -- 172.21.15.50:0/1381023159 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c02c980 con 0x7f894413aba0 2023-10-30T12:06:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: dump: 2023-10-30T12:06:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:12.514 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.514 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.514 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:12.514 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:12.515 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:12.515 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:12.515 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:12.515 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:12.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: _finish_auth 0 2023-10-30T12:06:12.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: _check_auth_tickets 2023-10-30T12:06:12.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:12.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 1 -- 172.21.15.50:0/1381023159 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c02cae0 con 0x7f894413aba0 2023-10-30T12:06:12.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:12.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f894bd25640 5 monclient: authenticate success, global_id 94183 2023-10-30T12:06:12.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f894bd25640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:12.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f894bd25640 4 set_mon_vals no callback set 2023-10-30T12:06:12.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 1 -- 172.21.15.50:0/1381023159 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c02ce70 con 0x7f894413aba0 2023-10-30T12:06:12.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:12.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:12.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.495+0000 7f8948abf640 10 monclient: dump: 2023-10-30T12:06:12.518 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:12.518 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:12.519 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.519 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.519 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:12.519 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:12.519 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:12.519 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:12.520 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 10 monclient: shutdown 2023-10-30T12:06:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 -- 172.21.15.50:0/1381023159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 msgr2=0x7f894413cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 --2- 172.21.15.50:0/1381023159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894413cf90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f89440a1500 tx=0x7f893c02bde0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 -- 172.21.15.50:0/1381023159 shutdown_connections 2023-10-30T12:06:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 --2- 172.21.15.50:0/1381023159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894413cf90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 --2- 172.21.15.50:0/1381023159 >> v2:172.21.15.50:3300/0 conn(0x7f89440707d0 0x7f894409e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 -- 172.21.15.50:0/1381023159 >> 172.21.15.50:0/1381023159 conn(0x7f89440a47f0 msgr2=0x7f89440a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 -- 172.21.15.50:0/1381023159 shutdown_connections 2023-10-30T12:06:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.496+0000 7f894bd25640 1 -- 172.21.15.50:0/1381023159 wait complete. 2023-10-30T12:06:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 10 monclient: build_initial_monmap 2023-10-30T12:06:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 10 monclient: monmap: 2023-10-30T12:06:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:12.524 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 1 librados: starting msgr at 2023-10-30T12:06:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 1 librados: starting objecter 2023-10-30T12:06:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 1 Processor -- start 2023-10-30T12:06:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 1 -- start start 2023-10-30T12:06:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 1 librados: setting wanted keys 2023-10-30T12:06:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 1 librados: calling monclient init 2023-10-30T12:06:12.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.497+0000 7f894bd25640 10 monclient: init 2023-10-30T12:06:12.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:12.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:06:12.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f89440707d0 0x7f89441ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:12.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): picked mon.noname-c con 0x7f89440707d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8944070bb0 0x7f89441cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): picked mon.noname-a con 0x7f8944070bb0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894411cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): picked mon.noname-b con 0x7f894413aba0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894411f1c0 con 0x7f8944070bb0 2023-10-30T12:06:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894409d6a0 con 0x7f894413aba0 2023-10-30T12:06:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894409b490 con 0x7f89440707d0 2023-10-30T12:06:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f89492c0640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8944070bb0 msgr2=0x7f89441cab10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient(hunting): authenticate will time out at 1217.681193s 2023-10-30T12:06:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f89492c0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8944070bb0 0x7f89441cab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f89440707d0 0x7f89441ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f89440707d0 0x7f89441ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59388/0 (socket says 172.21.15.50:59388) 2023-10-30T12:06:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 -- 172.21.15.50:0/1081874254 learned_addr learned my addr 172.21.15.50:0/1081874254 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): get_auth_request con 0x7f89440707d0 auth_method 0 2023-10-30T12:06:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894a2c2640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894411cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient(hunting): handle_auth_done global_id 44876 payload 824 2023-10-30T12:06:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 -- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 msgr2=0x7f894411cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894411cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 -- 172.21.15.50:0/1081874254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8944070bb0 msgr2=0x7f89441cab10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8944070bb0 0x7f89441cab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient: _finish_hunting 0 2023-10-30T12:06:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 monclient: found mon.noname-c 2023-10-30T12:06:12.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:12.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:12.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 -- 172.21.15.50:0/1081874254 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c009b30 con 0x7f89440707d0 2023-10-30T12:06:12.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894a2c2640 10 monclient: get_auth_request con 0x7f894413aba0 auth_method 0 2023-10-30T12:06:12.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894a2c2640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894411cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:06:12.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f8949ac1640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f89440707d0 0x7f89441ca5d0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f894000b550 tx=0x7f894000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:12.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 client.?.objecter ms_handle_connect 0x7f89440707d0 2023-10-30T12:06:12.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:12.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 1 -- 172.21.15.50:0/1081874254 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940003dc0 con 0x7f89440707d0 2023-10-30T12:06:12.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:12.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:12.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: dump: 2023-10-30T12:06:12.535 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:12.535 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:12.535 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.536 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.536 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:12.536 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:12.536 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:12.537 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:12.538 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:12.538 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:12.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:12.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:12.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:12.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 1 -- 172.21.15.50:0/1081874254 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8940005070 con 0x7f89440707d0 2023-10-30T12:06:12.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 5 monclient: authenticate success, global_id 44876 2023-10-30T12:06:12.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient: _renew_subs 2023-10-30T12:06:12.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:12.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8944120b20 con 0x7f89440707d0 2023-10-30T12:06:12.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:12.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 1 -- 172.21.15.50:0/1081874254 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940005420 con 0x7f89440707d0 2023-10-30T12:06:12.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:12.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:12.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893a7fc640 10 monclient: dump: 2023-10-30T12:06:12.540 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:12.541 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:12.541 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.541 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:12.541 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:12.541 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:12.542 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:12.542 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:12.542 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:12.542 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:12.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.498+0000 7f893bfff640 4 set_mon_vals no callback set 2023-10-30T12:06:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 10 client.44876.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 10 monclient: _renew_subs 2023-10-30T12:06:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894411e750 con 0x7f89440707d0 2023-10-30T12:06:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 1 librados: init done 2023-10-30T12:06:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f893a7fc640 1 -- 172.21.15.50:0/1081874254 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8940005580 con 0x7f89440707d0 2023-10-30T12:06:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894404e160 con 0x7f89440707d0 2023-10-30T12:06:12.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.499+0000 7f893a7fc640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89280392a0 0x7f892803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:12.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f893a7fc640 1 -- 172.21.15.50:0/1081874254 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f894004ad30 con 0x7f89440707d0 2023-10-30T12:06:12.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f893a7fc640 10 client.44876.objecter ms_dispatch 0x7f8944000b90 osd_map(173..173 src has 1..173) v4 2023-10-30T12:06:12.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f893a7fc640 3 client.44876.objecter handle_osd_map got epochs [173,173] > 0 2023-10-30T12:06:12.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f893a7fc640 3 client.44876.objecter handle_osd_map decoding full epoch 173 2023-10-30T12:06:12.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f893a7fc640 20 client.44876.objecter dump_active .. 0 homeless 2023-10-30T12:06:12.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f89492c0640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89280392a0 0x7f892803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:12.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f89492c0640 10 monclient: get_auth_request con 0x7f89280392a0 auth_method 0 2023-10-30T12:06:12.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f89492c0640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89280392a0 0x7f892803b760 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7f894411e350 tx=0x7f8934006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:12.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.500+0000 7f893a7fc640 10 client.44876.objecter ms_handle_connect 0x7f89280392a0 2023-10-30T12:06:12.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.504+0000 7f893a7fc640 1 -- 172.21.15.50:0/1081874254 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8940050020 con 0x7f89440707d0 2023-10-30T12:06:12.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.504+0000 7f893a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:12.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.504+0000 7f893a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:12.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.750+0000 7f894bd25640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:12.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.750+0000 7f894bd25640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:12.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.750+0000 7f894bd25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:12.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.750+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8944120620 con 0x7f89440707d0 2023-10-30T12:06:12.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.751+0000 7f893a7fc640 1 -- 172.21.15.50:0/1081874254 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f8940005830 con 0x7f89440707d0 2023-10-30T12:06:12.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.751+0000 7f893a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:12.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.751+0000 7f893a7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:12.753 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:12.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 10 librados: watch_flush enter 2023-10-30T12:06:12.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 10 librados: watch_flush exit 2023-10-30T12:06:12.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 20 client.44876.objecter shutdown clearing up homeless session... 2023-10-30T12:06:12.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 10 client.44876.objecter successfully canceled tick 2023-10-30T12:06:12.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89280392a0 msgr2=0x7f892803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:12.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89280392a0 0x7f892803b760 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7f894411e350 tx=0x7f8934006d30 comp rx=0 tx=0).stop 2023-10-30T12:06:12.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 10 monclient: shutdown 2023-10-30T12:06:12.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:12.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f89440707d0 msgr2=0x7f89441ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:12.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f89440707d0 0x7f89441ca5d0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f894000b550 tx=0x7f894000ba10 comp rx=0 tx=0).stop 2023-10-30T12:06:12.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 shutdown_connections 2023-10-30T12:06:12.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89280392a0 0x7f892803b760 unknown :-1 s=CLOSED pgs=3980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f894413aba0 0x7f894411cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8944070bb0 0x7f89441cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 --2- 172.21.15.50:0/1081874254 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f89440707d0 0x7f89441ca5d0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:12.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 >> 172.21.15.50:0/1081874254 conn(0x7f89440a47f0 msgr2=0x7f89440a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:12.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 shutdown_connections 2023-10-30T12:06:12.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 -- 172.21.15.50:0/1081874254 wait complete. 2023-10-30T12:06:12.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:12.753+0000 7f894bd25640 1 librados: shutdown 2023-10-30T12:06:15.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.881+0000 7fd755bf0640 10 monclient: get_monmap_and_config 2023-10-30T12:06:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.881+0000 7fd755bf0640 10 monclient: build_initial_monmap 2023-10-30T12:06:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.881+0000 7fd755bf0640 10 monclient: monmap: 2023-10-30T12:06:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:15.882822+0000 2023-10-30T12:06:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:15.882822+0000 2023-10-30T12:06:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:15.886 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.882+0000 7fd755bf0640 1 Processor -- start 2023-10-30T12:06:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.882+0000 7fd755bf0640 1 -- start start 2023-10-30T12:06:15.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.882+0000 7fd755bf0640 10 monclient: init 2023-10-30T12:06:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.882+0000 7fd755bf0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-10-30T12:06:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 0x7fd75009e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 10 monclient(hunting): picked mon.noname-a con 0x7fd7500707d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:15.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd750070bb0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd75013aba0 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75009ee40 con 0x7fd75013aba0 2023-10-30T12:06:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75009f4f0 con 0x7fd750070bb0 2023-10-30T12:06:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:15.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75009f7d0 con 0x7fd7500707d0 2023-10-30T12:06:15.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:15.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd755bf0640 10 monclient(hunting): authenticate will time out at 1221.066097s 2023-10-30T12:06:15.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd74ffff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fd75013aba0 legacy=0x7fd75013cf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:15.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd74f7fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 msgr2=0x7fd75009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:15.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd74f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 0x7fd75009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:15.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd74effd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fd750070bb0 0x7fd750071020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:38318/0 (socket says 172.21.15.50:38318) 2023-10-30T12:06:15.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:15.883+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 learned_addr learned my addr 172.21.15.50:0/1215720750 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:06:16.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:16.084+0000 7fd74ffff640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.50:6789/0 conn(0x7fd75013aba0 legacy=0x7fd75013cf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:16.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:16.084+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 msgr2=0x7fd75009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:16.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:16.084+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 0x7fd75009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:06:16.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:16.485+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 msgr2=0x7fd75009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:16.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:16.485+0000 7fd74ffff640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.50:6789/0 conn(0x7fd75013aba0 legacy=0x7fd75013cf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:16.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:16.485+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 0x7fd75009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:06:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:17.286+0000 7fd74ffff640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.50:6789/0 conn(0x7fd75013aba0 legacy=0x7fd75013cf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:17.286+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 msgr2=0x7fd75009e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:17.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:17.286+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 0x7fd75009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:06:18.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): tick 2023-10-30T12:06:18.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:06:18.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:06:18.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:06:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 msgr2=0x7fd75009e7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.50:3300/0 conn(0x7fd7500707d0 0x7fd75009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.118:6789/0 conn(0x7fd750070bb0 legacy=0x7fd750071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.50:6789/0 conn(0x7fd75013aba0 legacy=0x7fd75013cf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:18.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient: _add_conns ranks=[2,4] 2023-10-30T12:06:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fd73c0036c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fd73c006240 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:18.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75009f7d0 con 0x7fd73c006240 2023-10-30T12:06:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c008dc0 con 0x7fd73c0036c0 2023-10-30T12:06:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74d7fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request con 0x7fd73c0036c0 auth_method 0 2023-10-30T12:06:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 10 monclient(hunting): get_auth_request con 0x7fd73c006240 auth_method 0 2023-10-30T12:06:18.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:18.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:18.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:18.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:18.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:18.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:18.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:18.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:18.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:18.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:18.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:18.883+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request con 0x7fd73c0036c0 auth_method 0 2023-10-30T12:06:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:19.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:19.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:06:19.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 10 monclient(hunting): get_auth_request con 0x7fd73c006240 auth_method 0 2023-10-30T12:06:19.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:19.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.084+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:06:19.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:19.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request con 0x7fd73c0036c0 auth_method 0 2023-10-30T12:06:19.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:19.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:19.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:19.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:19.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:19.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:19.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:06:19.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:19.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 10 monclient(hunting): get_auth_request con 0x7fd73c006240 auth_method 0 2023-10-30T12:06:19.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:19.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:19.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:19.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:19.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:19.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:19.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:19.485+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:06:20.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request con 0x7fd73c0036c0 auth_method 0 2023-10-30T12:06:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.286+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:06:20.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:20.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 10 monclient(hunting): get_auth_request con 0x7fd73c006240 auth_method 0 2023-10-30T12:06:20.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:20.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:20.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:20.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:20.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:20.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:20.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:20.287+0000 7fd74effd640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:06:21.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): tick 2023-10-30T12:06:21.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:06:21.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:06:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:06:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 msgr2=0x7fd73c005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c0036c0 0x7fd73c005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:21.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 msgr2=0x7fd73c008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:21.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 --2- 172.21.15.50:0/1215720750 >> v2:172.21.15.118:3300/0 conn(0x7fd73c006240 0x7fd73c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:21.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:06:21.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd75013aba0 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:21.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 1 -- 172.21.15.50:0/1215720750 reap_dead start 2023-10-30T12:06:21.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd750070bb0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:21.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c003b00 0x7fd73c0066f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:21.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fd73c003b00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:21.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75009fab0 con 0x7fd75013aba0 2023-10-30T12:06:21.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74f7fe640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.50:6789/0 conn(0x7fd75013aba0 legacy=0x7fd73c00b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:21.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75009ee40 con 0x7fd750070bb0 2023-10-30T12:06:21.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:21.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 1 -- 172.21.15.50:0/1215720750 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75009f7d0 con 0x7fd73c003b00 2023-10-30T12:06:21.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74d7fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:21.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c003b00 0x7fd73c0066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:21.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 10 monclient(hunting): get_auth_request con 0x7fd73c003b00 auth_method 0 2023-10-30T12:06:21.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:21.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:21.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:21.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:21.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:21.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.883+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:21.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 10 monclient(hunting): handle_auth_done global_id 44882 payload 274 2023-10-30T12:06:21.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.118:6789/0 conn(0x7fd750070bb0 legacy=0x7fd73c00e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:21.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 1 -- 172.21.15.50:0/1215720750 >> v1:172.21.15.50:6789/0 conn(0x7fd75013aba0 legacy=0x7fd73c00b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:21.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 1 monclient: found mon.noname-b 2023-10-30T12:06:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.884+0000 7fd74ffff640 1 -- 172.21.15.50:0/1215720750 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c007ce0 con 0x7fd73c003b00 2023-10-30T12:06:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74ffff640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c003b00 0x7fd73c0066f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fd75009aef0 tx=0x7fd74802bdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94c4be48ae40c03f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 1 -- 172.21.15.50:0/1215720750 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74802cb80 con 0x7fd73c003b00 2023-10-30T12:06:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: dump: 2023-10-30T12:06:21.895 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:21.896 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:21.897 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:21.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 1 -- 172.21.15.50:0/1215720750 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74802cce0 con 0x7fd73c003b00 2023-10-30T12:06:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd755bf0640 5 monclient: authenticate success, global_id 44882 2023-10-30T12:06:21.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd755bf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd755bf0640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd755bf0640 4 set_mon_vals no callback set 2023-10-30T12:06:21.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 1 -- 172.21.15.50:0/1215720750 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480356b0 con 0x7fd73c003b00 2023-10-30T12:06:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.885+0000 7fd74e7fc640 10 monclient: dump: 2023-10-30T12:06:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:21.900 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:21.901 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:21.902 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:21.902 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:21.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 10 monclient: shutdown 2023-10-30T12:06:21.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:21.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 1 -- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c003b00 msgr2=0x7fd73c0066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:21.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c003b00 0x7fd73c0066f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fd75009aef0 tx=0x7fd74802bdd0 comp rx=0 tx=0).stop 2023-10-30T12:06:21.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 1 -- 172.21.15.50:0/1215720750 shutdown_connections 2023-10-30T12:06:21.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 1 --2- 172.21.15.50:0/1215720750 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd73c003b00 0x7fd73c0066f0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:21.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 1 -- 172.21.15.50:0/1215720750 >> 172.21.15.50:0/1215720750 conn(0x7fd7500a47f0 msgr2=0x7fd7500a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:21.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 1 -- 172.21.15.50:0/1215720750 shutdown_connections 2023-10-30T12:06:21.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.886+0000 7fd755bf0640 1 -- 172.21.15.50:0/1215720750 wait complete. 2023-10-30T12:06:21.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 10 monclient: build_initial_monmap 2023-10-30T12:06:21.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 10 monclient: monmap: 2023-10-30T12:06:21.909 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:21.909 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:21.909 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:21.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:21.910 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:21.910 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:21.910 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:21.911 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:21.912 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:21.912 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:21.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 1 librados: starting msgr at 2023-10-30T12:06:21.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 1 librados: starting objecter 2023-10-30T12:06:21.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 1 Processor -- start 2023-10-30T12:06:21.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 1 -- start start 2023-10-30T12:06:21.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 1 librados: setting wanted keys 2023-10-30T12:06:21.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 1 librados: calling monclient init 2023-10-30T12:06:21.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.887+0000 7fd755bf0640 10 monclient: init 2023-10-30T12:06:21.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:21.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:06:21.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd73c003b00 0x7fd7501ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:21.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): picked mon.noname-a con 0x7fd73c003b00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:21.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd750070bb0 0x7fd7501318f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:21.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): picked mon.noname-b con 0x7fd750070bb0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:21.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd75013aba0 0x7fd750131e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:21.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): picked mon.noname-c con 0x7fd75013aba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:21.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:21.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75011e3a0 con 0x7fd73c003b00 2023-10-30T12:06:21.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:21.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75011dce0 con 0x7fd750070bb0 2023-10-30T12:06:21.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:21.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75009d6a0 con 0x7fd75013aba0 2023-10-30T12:06:21.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:21.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74f7fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd73c003b00 msgr2=0x7fd7501ca190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:21.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd755bf0640 10 monclient(hunting): authenticate will time out at 1227.071404s 2023-10-30T12:06:21.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd73c003b00 0x7fd7501ca190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:21.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd75013aba0 0x7fd750131e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:21.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd75013aba0 0x7fd750131e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43260/0 (socket says 172.21.15.50:43260) 2023-10-30T12:06:21.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 1 -- 172.21.15.50:0/3203458917 learned_addr learned my addr 172.21.15.50:0/3203458917 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:21.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): get_auth_request con 0x7fd75013aba0 auth_method 0 2023-10-30T12:06:21.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:21.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:21.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:21.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:21.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:21.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd750070bb0 0x7fd7501318f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:21.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:21.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 10 monclient(hunting): get_auth_request con 0x7fd750070bb0 auth_method 0 2023-10-30T12:06:21.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:21.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:21.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:21.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:21.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:21.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:21.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 10 monclient(hunting): handle_auth_done global_id 44885 payload 824 2023-10-30T12:06:21.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.888+0000 7fd74ffff640 1 -- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd750070bb0 msgr2=0x7fd7501318f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:21.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd750070bb0 0x7fd7501318f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:21.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 1 -- 172.21.15.50:0/3203458917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd73c003b00 msgr2=0x7fd7501ca190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:21.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd73c003b00 0x7fd7501ca190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:21.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:21.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 1 monclient: found mon.noname-c 2023-10-30T12:06:21.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:21.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:21.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 1 -- 172.21.15.50:0/3203458917 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd748007920 con 0x7fd75013aba0 2023-10-30T12:06:21.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74ffff640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd75013aba0 0x7fd750131e30 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7fd748001220 tx=0x7fd748005f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:21.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 client.?.objecter ms_handle_connect 0x7fd75013aba0 2023-10-30T12:06:21.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:21.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 1 -- 172.21.15.50:0/3203458917 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748041610 con 0x7fd75013aba0 2023-10-30T12:06:21.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:21.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:21.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: dump: 2023-10-30T12:06:21.925 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:21.925 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:21.925 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.925 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.925 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:21.926 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:21.926 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:21.926 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:21.926 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:21.926 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:21.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: _finish_auth 0 2023-10-30T12:06:21.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:21.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:21.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 1 -- 172.21.15.50:0/3203458917 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd748041770 con 0x7fd75013aba0 2023-10-30T12:06:21.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 5 monclient: authenticate success, global_id 44885 2023-10-30T12:06:21.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 10 monclient: _renew_subs 2023-10-30T12:06:21.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:21.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75011bef0 con 0x7fd75013aba0 2023-10-30T12:06:21.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:21.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 1 -- 172.21.15.50:0/3203458917 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748041ae0 con 0x7fd75013aba0 2023-10-30T12:06:21.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:21.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:21.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd737fff640 10 monclient: dump: 2023-10-30T12:06:21.929 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:21.930 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:21.930 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.930 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:21.930 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:21.930 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:21.931 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:21.931 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:21.931 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:21.931 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:21.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 10 client.44885.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:21.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 10 monclient: _renew_subs 2023-10-30T12:06:21.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:21.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75009b6f0 con 0x7fd75013aba0 2023-10-30T12:06:21.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd755bf0640 1 librados: init done 2023-10-30T12:06:21.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.889+0000 7fd74dffb640 4 set_mon_vals no callback set 2023-10-30T12:06:21.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.890+0000 7fd737fff640 1 -- 172.21.15.50:0/3203458917 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd748032070 con 0x7fd75013aba0 2023-10-30T12:06:21.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.890+0000 7fd755bf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:21.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.890+0000 7fd755bf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:21.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.890+0000 7fd755bf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:21.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.890+0000 7fd737fff640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd73c03a590 0x7fd73c03ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:21.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.890+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7501ca790 con 0x7fd75013aba0 2023-10-30T12:06:21.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd737fff640 1 -- 172.21.15.50:0/3203458917 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fd74806d540 con 0x7fd75013aba0 2023-10-30T12:06:21.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd737fff640 10 client.44885.objecter ms_dispatch 0x7fd750000b90 osd_map(175..175 src has 1..175) v4 2023-10-30T12:06:21.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd737fff640 3 client.44885.objecter handle_osd_map got epochs [175,175] > 0 2023-10-30T12:06:21.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd737fff640 3 client.44885.objecter handle_osd_map decoding full epoch 175 2023-10-30T12:06:21.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd737fff640 20 client.44885.objecter dump_active .. 0 homeless 2023-10-30T12:06:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd73c03a590 0x7fd73c03ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd74f7fe640 10 monclient: get_auth_request con 0x7fd73c03a590 auth_method 0 2023-10-30T12:06:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd737fff640 10 client.44885.objecter ms_handle_connect 0x7fd73c03a590 2023-10-30T12:06:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.891+0000 7fd74f7fe640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd73c03a590 0x7fd73c03ca50 secure :-1 s=READY pgs=3982 cs=0 l=1 rev1=1 crypto rx=0x7fd740006fd0 tx=0x7fd740008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.896+0000 7fd737fff640 1 -- 172.21.15.50:0/3203458917 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd748033370 con 0x7fd75013aba0 2023-10-30T12:06:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.896+0000 7fd737fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:21.896+0000 7fd737fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:22.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.144+0000 7fd755bf0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.144+0000 7fd755bf0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.144+0000 7fd755bf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.144+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd75004e160 con 0x7fd75013aba0 2023-10-30T12:06:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.144+0000 7fd737fff640 1 -- 172.21.15.50:0/3203458917 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fd74803f360 con 0x7fd75013aba0 2023-10-30T12:06:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.144+0000 7fd737fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.144+0000 7fd737fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:22.149 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:22.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 10 librados: watch_flush enter 2023-10-30T12:06:22.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 10 librados: watch_flush exit 2023-10-30T12:06:22.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 20 client.44885.objecter shutdown clearing up homeless session... 2023-10-30T12:06:22.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 10 client.44885.objecter successfully canceled tick 2023-10-30T12:06:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd73c03a590 msgr2=0x7fd73c03ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd73c03a590 0x7fd73c03ca50 secure :-1 s=READY pgs=3982 cs=0 l=1 rev1=1 crypto rx=0x7fd740006fd0 tx=0x7fd740008040 comp rx=0 tx=0).stop 2023-10-30T12:06:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 10 monclient: shutdown 2023-10-30T12:06:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd75013aba0 msgr2=0x7fd750131e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd75013aba0 0x7fd750131e30 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7fd748001220 tx=0x7fd748005f20 comp rx=0 tx=0).stop 2023-10-30T12:06:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 shutdown_connections 2023-10-30T12:06:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd73c03a590 0x7fd73c03ca50 unknown :-1 s=CLOSED pgs=3982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd75013aba0 0x7fd750131e30 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd750070bb0 0x7fd7501318f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 --2- 172.21.15.50:0/3203458917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd73c003b00 0x7fd7501ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 >> 172.21.15.50:0/3203458917 conn(0x7fd7500a47f0 msgr2=0x7fd750135b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 shutdown_connections 2023-10-30T12:06:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.146+0000 7fd755bf0640 1 -- 172.21.15.50:0/3203458917 wait complete. 2023-10-30T12:06:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.147+0000 7fd755bf0640 1 librados: shutdown 2023-10-30T12:06:22.158 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:06:22.159 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:06:22.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:06:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.263+0000 7f9c6afa9640 10 monclient: get_monmap_and_config 2023-10-30T12:06:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.263+0000 7f9c6afa9640 10 monclient: build_initial_monmap 2023-10-30T12:06:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.263+0000 7f9c6afa9640 10 monclient: monmap: 2023-10-30T12:06:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:22.264407+0000 2023-10-30T12:06:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:22.264407+0000 2023-10-30T12:06:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:22.267 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 1 Processor -- start 2023-10-30T12:06:22.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 1 -- start start 2023-10-30T12:06:22.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient: init 2023-10-30T12:06:22.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:22.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:06:22.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c6409bb70 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:22.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c6409d3f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:22.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6413b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient(hunting): picked mon.noname-b con 0x7f9c6409c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:22.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c64098170 con 0x7f9c6409bb70 2023-10-30T12:06:22.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6409cf60 con 0x7f9c6409d3f0 2023-10-30T12:06:22.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64097b20 con 0x7f9c6409c660 2023-10-30T12:06:22.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:22.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c6afa9640 10 monclient(hunting): authenticate will time out at 1227.447550s 2023-10-30T12:06:22.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c68d45640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f9c6409bb70 legacy=0x7f9c6409bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:22.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6413b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6413b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43270/0 (socket says 172.21.15.50:43270) 2023-10-30T12:06:22.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 1 -- 172.21.15.50:0/3445227625 learned_addr learned my addr 172.21.15.50:0/3445227625 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:22.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): get_auth_request con 0x7f9c6409c660 auth_method 0 2023-10-30T12:06:22.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient(hunting): handle_auth_done global_id 44888 payload 274 2023-10-30T12:06:22.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 1 -- 172.21.15.50:0/3445227625 >> v1:172.21.15.118:6789/0 conn(0x7f9c6409d3f0 legacy=0x7f9c64137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:22.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 1 -- 172.21.15.50:0/3445227625 >> v1:172.21.15.50:6789/0 conn(0x7f9c6409bb70 legacy=0x7f9c6409bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:22.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient: _finish_hunting 0 2023-10-30T12:06:22.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 1 monclient: found mon.noname-b 2023-10-30T12:06:22.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:22.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.264+0000 7f9c69546640 1 -- 172.21.15.50:0/3445227625 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c64097e00 con 0x7f9c6409c660 2023-10-30T12:06:22.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c69546640 1 --2- 172.21.15.50:0/3445227625 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6413b090 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009e80 tx=0x7f9c5c02bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7416d4753d8aade server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 1 -- 172.21.15.50:0/3445227625 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c02c930 con 0x7f9c6409c660 2023-10-30T12:06:22.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:22.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: dump: 2023-10-30T12:06:22.283 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.284 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.284 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.284 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.284 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.284 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.285 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.285 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.285 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.285 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: _finish_auth 0 2023-10-30T12:06:22.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: _check_auth_tickets 2023-10-30T12:06:22.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:22.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 1 -- 172.21.15.50:0/3445227625 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c02ca90 con 0x7f9c6409c660 2023-10-30T12:06:22.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c6afa9640 5 monclient: authenticate success, global_id 44888 2023-10-30T12:06:22.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c6afa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:22.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:22.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c6afa9640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:22.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 1 -- 172.21.15.50:0/3445227625 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c02ce20 con 0x7f9c6409c660 2023-10-30T12:06:22.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c6afa9640 4 set_mon_vals no callback set 2023-10-30T12:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.265+0000 7f9c637fe640 10 monclient: dump: 2023-10-30T12:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.290 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.290 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 10 monclient: shutdown 2023-10-30T12:06:22.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:22.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 1 -- 172.21.15.50:0/3445227625 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 msgr2=0x7f9c6413b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 1 --2- 172.21.15.50:0/3445227625 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6413b090 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009e80 tx=0x7f9c5c02bd90 comp rx=0 tx=0).stop 2023-10-30T12:06:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 1 -- 172.21.15.50:0/3445227625 shutdown_connections 2023-10-30T12:06:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 1 --2- 172.21.15.50:0/3445227625 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6413b090 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 1 -- 172.21.15.50:0/3445227625 >> 172.21.15.50:0/3445227625 conn(0x7f9c640a3980 msgr2=0x7f9c640a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 1 -- 172.21.15.50:0/3445227625 shutdown_connections 2023-10-30T12:06:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 1 -- 172.21.15.50:0/3445227625 wait complete. 2023-10-30T12:06:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 10 monclient: build_initial_monmap 2023-10-30T12:06:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.266+0000 7f9c6afa9640 10 monclient: monmap: 2023-10-30T12:06:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:22.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 librados: starting msgr at 2023-10-30T12:06:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 librados: starting objecter 2023-10-30T12:06:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 Processor -- start 2023-10-30T12:06:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 -- start start 2023-10-30T12:06:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 librados: setting wanted keys 2023-10-30T12:06:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 librados: calling monclient init 2023-10-30T12:06:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient: init 2023-10-30T12:06:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:06:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c6409bb70 0x7f9c6406bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): picked mon.noname-b con 0x7f9c6409bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6406c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6409c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:22.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c6409d3f0 0x7f9c6406c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): picked mon.noname-a con 0x7f9c6409d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:22.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64092090 con 0x7f9c6409d3f0 2023-10-30T12:06:22.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64097120 con 0x7f9c6409bb70 2023-10-30T12:06:22.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64092790 con 0x7f9c6409c660 2023-10-30T12:06:22.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:22.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c6afa9640 10 monclient(hunting): authenticate will time out at 1227.450853s 2023-10-30T12:06:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c69546640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c6409d3f0 msgr2=0x7f9c6406c620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c69546640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c6409d3f0 0x7f9c6406c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c63fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6406c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c63fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6406c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43272/0 (socket says 172.21.15.50:43272) 2023-10-30T12:06:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c63fff640 1 -- 172.21.15.50:0/1242344067 learned_addr learned my addr 172.21.15.50:0/1242344067 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c63fff640 10 monclient(hunting): get_auth_request con 0x7f9c6409c660 auth_method 0 2023-10-30T12:06:22.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.267+0000 7f9c63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c68d45640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c6409bb70 0x7f9c6406bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c68d45640 10 monclient(hunting): get_auth_request con 0x7f9c6409bb70 auth_method 0 2023-10-30T12:06:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c68d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c68d45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c68d45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 10 monclient(hunting): handle_auth_done global_id 44891 payload 824 2023-10-30T12:06:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c6409bb70 msgr2=0x7f9c6406bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c6409bb70 0x7f9c6406bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c6409d3f0 msgr2=0x7f9c6406c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c6409d3f0 0x7f9c6406c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 1 monclient: found mon.noname-c 2023-10-30T12:06:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 1 -- 172.21.15.50:0/1242344067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c009b30 con 0x7f9c6409c660 2023-10-30T12:06:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c68d45640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c6409bb70 0x7f9c6406bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:06:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c63fff640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6406c0e0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f9c5000c410 tx=0x7f9c5000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 client.?.objecter ms_handle_connect 0x7f9c6409c660 2023-10-30T12:06:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c500035f0 con 0x7f9c6409c660 2023-10-30T12:06:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: dump: 2023-10-30T12:06:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.306 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.306 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.306 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.306 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.306 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.307 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.307 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.307 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.307 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: _finish_auth 0 2023-10-30T12:06:22.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: _check_auth_tickets 2023-10-30T12:06:22.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:22.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c50003750 con 0x7f9c6409c660 2023-10-30T12:06:22.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 5 monclient: authenticate success, global_id 44891 2023-10-30T12:06:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c50003b40 con 0x7f9c6409c660 2023-10-30T12:06:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c617fa640 10 monclient: dump: 2023-10-30T12:06:22.310 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.310 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.310 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.310 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.310 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.311 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.311 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.311 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.311 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.311 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 10 monclient: _renew_subs 2023-10-30T12:06:22.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c62ffd640 4 set_mon_vals no callback set 2023-10-30T12:06:22.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c640976e0 con 0x7f9c6409c660 2023-10-30T12:06:22.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 10 client.44891.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:22.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 10 monclient: _renew_subs 2023-10-30T12:06:22.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6406d6f0 con 0x7f9c6409c660 2023-10-30T12:06:22.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.268+0000 7f9c6afa9640 1 librados: init done 2023-10-30T12:06:22.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9c50003ca0 con 0x7f9c6409c660 2023-10-30T12:06:22.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c6afa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c6afa9640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64094d30 0x7f9c6406fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c6afa9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6406d8b0 con 0x7f9c64094d30 2023-10-30T12:06:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c480392a0 0x7f9c4803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f9c5004a050 con 0x7f9c6409c660 2023-10-30T12:06:22.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 10 client.44891.objecter ms_dispatch 0x7f9c64000b90 osd_map(175..175 src has 1..175) v4 2023-10-30T12:06:22.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 3 client.44891.objecter handle_osd_map got epochs [175,175] > 0 2023-10-30T12:06:22.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 3 client.44891.objecter handle_osd_map decoding full epoch 175 2023-10-30T12:06:22.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64094d30 0x7f9c6406fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient: get_auth_request con 0x7f9c64094d30 auth_method 0 2023-10-30T12:06:22.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 20 client.44891.objecter dump_active .. 0 homeless 2023-10-30T12:06:22.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c69546640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c480392a0 0x7f9c4803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c69546640 10 monclient: get_auth_request con 0x7f9c480392a0 auth_method 0 2023-10-30T12:06:22.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 10 monclient(hunting): handle_auth_done global_id 44894 payload 824 2023-10-30T12:06:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 1 -- 172.21.15.50:0/1242344067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c64097560 con 0x7f9c64094d30 2023-10-30T12:06:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c68d45640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64094d30 0x7f9c6406fdc0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800d700 tx=0x7f9c5800dbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 10 client.44891.objecter ms_handle_connect 0x7f9c64094d30 2023-10-30T12:06:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.269+0000 7f9c617fa640 10 client.44891.objecter resend_mon_ops 2023-10-30T12:06:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58005a90 con 0x7f9c64094d30 2023-10-30T12:06:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 10 monclient: dump: 2023-10-30T12:06:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.320 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.320 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.320 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.320 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.320 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.321 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.321 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.321 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 10 client.44891.objecter ms_handle_connect 0x7f9c480392a0 2023-10-30T12:06:22.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c69546640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c480392a0 0x7f9c4803b760 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c035000 tx=0x7f9c5c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9c58005bf0 con 0x7f9c64094d30 2023-10-30T12:06:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64094d30 msgr2=0x7f9c6406fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64094d30 0x7f9c6406fdc0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800d700 tx=0x7f9c5800dbc0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64094d30 msgr2=0x7f9c6406fdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:22.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.270+0000 7f9c617fa640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64094d30 0x7f9c6406fdc0 secure :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800d700 tx=0x7f9c5800dbc0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c6afa9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:22.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c6afa9640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64070300 0x7f9c64070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c6afa9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64073ff0 con 0x7f9c64070300 2023-10-30T12:06:22.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64070300 0x7f9c64070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient: get_auth_request con 0x7f9c64070300 auth_method 0 2023-10-30T12:06:22.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 10 monclient(hunting): handle_auth_done global_id 44897 payload 824 2023-10-30T12:06:22.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.276+0000 7f9c68d45640 1 -- 172.21.15.50:0/1242344067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9c6409f600 con 0x7f9c64070300 2023-10-30T12:06:22.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c68d45640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64070300 0x7f9c64070710 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800d700 tx=0x7f9c58005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 10 client.44891.objecter ms_handle_connect 0x7f9c64070300 2023-10-30T12:06:22.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 10 client.44891.objecter resend_mon_ops 2023-10-30T12:06:22.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58022070 con 0x7f9c64070300 2023-10-30T12:06:22.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:22.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 10 monclient: dump: 2023-10-30T12:06:22.329 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.329 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.329 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.329 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.330 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.330 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.330 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.330 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.331 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.331 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f9c5800ba80 con 0x7f9c64070300 2023-10-30T12:06:22.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:22.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:22.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64070300 msgr2=0x7f9c64070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64070300 0x7f9c64070710 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800d700 tx=0x7f9c58005a20 comp rx=0 tx=0).stop 2023-10-30T12:06:22.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64070300 msgr2=0x7f9c64070710 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:22.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c617fa640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c64070300 0x7f9c64070710 secure :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800d700 tx=0x7f9c58005a20 comp rx=0 tx=0).stop 2023-10-30T12:06:22.332 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:22.333 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:22.333 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:22.333 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:22.333 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 58, 2023-10-30T12:06:22.333 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:22.334 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:22.334 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:22.334 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.334 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:06:22.334 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:22.335 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:22.335 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:22.335 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:22.335 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:22.335 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:22.336 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:22.336 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:22.336 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:22.336 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:22.336 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:22.337 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:22.337 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:22.337 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.337 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:22.337 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:22.339 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:22.339 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:22.339 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:22.339 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:22.339 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:22.339 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.340 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:22.340 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:22.340 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:22.340 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:22.341 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:22.341 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:22.341 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:22.341 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:22.341 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:22.342 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:22.342 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:22.342 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:22.342 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:22.342 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:22.343 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:22.343 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:22.343 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:22.343 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:22.343 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:22.343 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:22.344 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:22.344 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:22.344 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:22.344 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:22.345 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:22.345 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:22.345 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:22.345 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.345 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:22.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.346 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:22.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.346 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:22.346 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:22.347 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:22.347 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:22.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:22.347 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:22.347 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.348 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:22.348 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:22.348 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.349 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.349 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.349 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:22.349 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:22.350 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:22.350 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:22.350 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:22.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.351 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:22.351 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:22.351 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:22.351 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:22.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:22.352 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:22.352 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:22.353 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:22.353 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.353 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.354 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:22.354 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:22.354 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:22.354 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:22.354 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:22.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.355 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:22.355 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:22.355 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:22.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:22.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:22.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:22.357 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.357 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:22.357 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:22.357 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.358 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.358 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:22.358 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:22.359 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:22.359 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:22.359 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:22.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.359 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.360 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:22.360 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:22.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.360 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:22.360 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:22.361 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:22.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.361 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.361 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:06:22.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.361 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:22.362 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:22.362 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:06:22.362 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.362 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.362 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:22.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.363 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:22.363 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:22.363 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:06:22.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.364 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.364 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:22.364 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:22.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 10 librados: watch_flush enter 2023-10-30T12:06:22.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 10 librados: watch_flush exit 2023-10-30T12:06:22.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 20 client.44891.objecter shutdown clearing up homeless session... 2023-10-30T12:06:22.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 10 client.44891.objecter successfully canceled tick 2023-10-30T12:06:22.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c480392a0 msgr2=0x7f9c4803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c480392a0 0x7f9c4803b760 secure :-1 s=READY pgs=3983 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c035000 tx=0x7f9c5c030000 comp rx=0 tx=0).stop 2023-10-30T12:06:22.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 10 monclient: shutdown 2023-10-30T12:06:22.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:22.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 msgr2=0x7f9c6406c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 --2- 172.21.15.50:0/1242344067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6409c660 0x7f9c6406c0e0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f9c5000c410 tx=0x7f9c5000c8d0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c69546640 1 -- 172.21.15.50:0/1242344067 reap_dead start 2023-10-30T12:06:22.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c69546640 1 -- 172.21.15.50:0/1242344067 reap_dead start 2023-10-30T12:06:22.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 shutdown_connections 2023-10-30T12:06:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 >> 172.21.15.50:0/1242344067 conn(0x7f9c640a3980 msgr2=0x7f9c640965b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 shutdown_connections 2023-10-30T12:06:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.277+0000 7f9c6afa9640 1 -- 172.21.15.50:0/1242344067 wait complete. 2023-10-30T12:06:22.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.278+0000 7f9c6afa9640 1 librados: shutdown 2023-10-30T12:06:22.370 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:06:22.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:06:22.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:06:22.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.480+0000 7ff5a99d0640 10 monclient: get_monmap_and_config 2023-10-30T12:06:22.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.480+0000 7ff5a99d0640 10 monclient: build_initial_monmap 2023-10-30T12:06:22.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.480+0000 7ff5a99d0640 10 monclient: monmap: 2023-10-30T12:06:22.482 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:22.481739+0000 2023-10-30T12:06:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:22.481739+0000 2023-10-30T12:06:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:22.485 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 1 Processor -- start 2023-10-30T12:06:22.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 1 -- start start 2023-10-30T12:06:22.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient: init 2023-10-30T12:06:22.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:22.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:06:22.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5a4132970 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a40a03d0 0x7ff5a413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a40a03d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff5a413b020 0x7ff5a413d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a413b020 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a40a17d0 con 0x7ff5a4132970 2023-10-30T12:06:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a4133460 con 0x7ff5a413b020 2023-10-30T12:06:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40a1510 con 0x7ff5a40a03d0 2023-10-30T12:06:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a99d0640 10 monclient(hunting): authenticate will time out at 1227.664926s 2023-10-30T12:06:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a2ffd640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7ff5a4132970 legacy=0x7ff5a4132d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:06:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.481+0000 7ff5a37fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7ff5a413b020 msgr2=0x7ff5a413d410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff5a37fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff5a413b020 0x7ff5a413d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a40a03d0 0x7ff5a413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a40a03d0 0x7ff5a413aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43296/0 (socket says 172.21.15.50:43296) 2023-10-30T12:06:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 -- 172.21.15.50:0/2477996864 learned_addr learned my addr 172.21.15.50:0/2477996864 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): get_auth_request con 0x7ff5a40a03d0 auth_method 0 2023-10-30T12:06:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient(hunting): handle_auth_done global_id 44900 payload 274 2023-10-30T12:06:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 -- 172.21.15.50:0/2477996864 >> v2:172.21.15.50:3300/0 conn(0x7ff5a413b020 msgr2=0x7ff5a413d410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 --2- 172.21.15.50:0/2477996864 >> v2:172.21.15.50:3300/0 conn(0x7ff5a413b020 0x7ff5a413d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 -- 172.21.15.50:0/2477996864 >> v1:172.21.15.50:6789/0 conn(0x7ff5a4132970 legacy=0x7ff5a4132d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 monclient: found mon.noname-b 2023-10-30T12:06:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.482+0000 7ff59bfff640 1 -- 172.21.15.50:0/2477996864 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a409f680 con 0x7ff5a40a03d0 2023-10-30T12:06:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff59bfff640 1 --2- 172.21.15.50:0/2477996864 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a40a03d0 0x7ff5a413aae0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009870 tx=0x7ff58c02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bc769e192d4a612 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 1 -- 172.21.15.50:0/2477996864 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c031750 con 0x7ff5a40a03d0 2023-10-30T12:06:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: dump: 2023-10-30T12:06:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.503 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:22.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:22.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:22.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 1 -- 172.21.15.50:0/2477996864 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c0318b0 con 0x7ff5a40a03d0 2023-10-30T12:06:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 5 monclient: authenticate success, global_id 44900 2023-10-30T12:06:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 1 -- 172.21.15.50:0/2477996864 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c031c40 con 0x7ff5a40a03d0 2023-10-30T12:06:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 4 set_mon_vals no callback set 2023-10-30T12:06:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a27fc640 10 monclient: dump: 2023-10-30T12:06:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.506 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.506 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.506 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.506 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.506 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.507 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 10 monclient: shutdown 2023-10-30T12:06:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 -- 172.21.15.50:0/2477996864 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a40a03d0 msgr2=0x7ff5a413aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/2477996864 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a40a03d0 0x7ff5a413aae0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009870 tx=0x7ff58c02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 -- 172.21.15.50:0/2477996864 shutdown_connections 2023-10-30T12:06:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/2477996864 >> v2:172.21.15.50:3300/0 conn(0x7ff5a413b020 0x7ff5a413d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/2477996864 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a40a03d0 0x7ff5a413aae0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 -- 172.21.15.50:0/2477996864 >> 172.21.15.50:0/2477996864 conn(0x7ff5a40a4800 msgr2=0x7ff5a40a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 -- 172.21.15.50:0/2477996864 shutdown_connections 2023-10-30T12:06:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.483+0000 7ff5a99d0640 1 -- 172.21.15.50:0/2477996864 wait complete. 2023-10-30T12:06:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 10 monclient: build_initial_monmap 2023-10-30T12:06:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 10 monclient: monmap: 2023-10-30T12:06:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:22.511 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 1 librados: starting msgr at 2023-10-30T12:06:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 1 librados: starting objecter 2023-10-30T12:06:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 1 Processor -- start 2023-10-30T12:06:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 1 -- start start 2023-10-30T12:06:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 1 librados: setting wanted keys 2023-10-30T12:06:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 1 librados: calling monclient init 2023-10-30T12:06:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.484+0000 7ff5a99d0640 10 monclient: init 2023-10-30T12:06:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:06:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a4132970 0x7ff5a41ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a4132970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a413b020 0x7ff5a41caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a413b020 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a41cafa0 0x7ff5a41381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a41cafa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a409d880 con 0x7ff5a41cafa0 2023-10-30T12:06:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a409c540 con 0x7ff5a413b020 2023-10-30T12:06:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4139f70 con 0x7ff5a4132970 2023-10-30T12:06:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 10 monclient(hunting): authenticate will time out at 1227.668335s 2023-10-30T12:06:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a37fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a41cafa0 msgr2=0x7ff5a41381f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:06:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a41cafa0 0x7ff5a41381f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:22.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a4132970 0x7ff5a41ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a4132970 0x7ff5a41ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43298/0 (socket says 172.21.15.50:43298) 2023-10-30T12:06:22.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 -- 172.21.15.50:0/4144835416 learned_addr learned my addr 172.21.15.50:0/4144835416 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:22.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): get_auth_request con 0x7ff5a4132970 auth_method 0 2023-10-30T12:06:22.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient(hunting): handle_auth_done global_id 44903 payload 824 2023-10-30T12:06:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a413b020 msgr2=0x7ff5a41caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a413b020 0x7ff5a41caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a41cafa0 msgr2=0x7ff5a41381f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5a41cafa0 0x7ff5a41381f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:22.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 monclient: found mon.noname-c 2023-10-30T12:06:22.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:22.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 -- 172.21.15.50:0/4144835416 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c0094d0 con 0x7ff5a4132970 2023-10-30T12:06:22.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a2ffd640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a4132970 0x7ff5a41ca290 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7ff59400b550 tx=0x7ff59400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 10 client.?.objecter ms_handle_connect 0x7ff5a4132970 2023-10-30T12:06:22.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:22.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594003dc0 con 0x7ff5a4132970 2023-10-30T12:06:22.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 10 monclient: dump: 2023-10-30T12:06:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.522 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff594005070 con 0x7ff5a4132970 2023-10-30T12:06:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.485+0000 7ff5a99d0640 5 monclient: authenticate success, global_id 44903 2023-10-30T12:06:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a0ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594005420 con 0x7ff5a4132970 2023-10-30T12:06:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 10 monclient: _renew_subs 2023-10-30T12:06:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a411b600 con 0x7ff5a4132970 2023-10-30T12:06:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a1ffb640 4 set_mon_vals no callback set 2023-10-30T12:06:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a0ff9640 10 monclient: dump: 2023-10-30T12:06:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.527 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 10 client.44903.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 10 monclient: _renew_subs 2023-10-30T12:06:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a411a6b0 con 0x7ff5a4132970 2023-10-30T12:06:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 1 librados: init done 2023-10-30T12:06:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a40a03d0 0x7ff5a4138730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4071a70 con 0x7ff5a40a03d0 2023-10-30T12:06:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.486+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff594005580 con 0x7ff5a4132970 2023-10-30T12:06:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff5840392a0 0x7ff58403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a40a03d0 0x7ff5a4138730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 7173+0+0 (secure 0 0 0) 0x7ff59404aef0 con 0x7ff5a4132970 2023-10-30T12:06:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 10 client.44903.objecter ms_dispatch 0x7ff5a4000b90 osd_map(176..176 src has 1..176) v4 2023-10-30T12:06:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 3 client.44903.objecter handle_osd_map got epochs [176,176] > 0 2023-10-30T12:06:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 3 client.44903.objecter handle_osd_map decoding full epoch 176 2023-10-30T12:06:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient: get_auth_request con 0x7ff5a40a03d0 auth_method 0 2023-10-30T12:06:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 20 client.44903.objecter dump_active .. 0 homeless 2023-10-30T12:06:22.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a37fe640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff5840392a0 0x7ff58403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a37fe640 10 monclient: get_auth_request con 0x7ff5840392a0 auth_method 0 2023-10-30T12:06:22.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 10 monclient(hunting): handle_auth_done global_id 94201 payload 824 2023-10-30T12:06:22.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 1 -- 172.21.15.50:0/4144835416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5a409f960 con 0x7ff5a40a03d0 2023-10-30T12:06:22.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff59bfff640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a40a03d0 0x7ff5a4138730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5a409aef0 tx=0x7ff58c033670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 10 client.44903.objecter ms_handle_connect 0x7ff5a40a03d0 2023-10-30T12:06:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 10 client.44903.objecter resend_mon_ops 2023-10-30T12:06:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a0ff9640 10 client.44903.objecter ms_handle_connect 0x7ff5840392a0 2023-10-30T12:06:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.487+0000 7ff5a37fe640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff5840392a0 0x7ff58403b760 secure :-1 s=READY pgs=3984 cs=0 l=1 rev1=1 crypto rx=0x7ff5a413a500 tx=0x7ff59c009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c033c40 con 0x7ff5a40a03d0 2023-10-30T12:06:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 10 monclient: dump: 2023-10-30T12:06:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.543 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.543 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.543 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ff58c033da0 con 0x7ff5a40a03d0 2023-10-30T12:06:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a40a03d0 msgr2=0x7ff5a4138730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a40a03d0 0x7ff5a4138730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5a409aef0 tx=0x7ff58c033670 comp rx=0 tx=0).stop 2023-10-30T12:06:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a40a03d0 msgr2=0x7ff5a4138730 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.488+0000 7ff5a0ff9640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a40a03d0 0x7ff5a4138730 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5a409aef0 tx=0x7ff58c033670 comp rx=0 tx=0).stop 2023-10-30T12:06:22.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff5a99d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:22.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a4138c70 0x7ff5a4135830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:22.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff5a99d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:22.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4074800 con 0x7ff5a4138c70 2023-10-30T12:06:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff59bfff640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a4138c70 0x7ff5a4135830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff59bfff640 10 monclient: get_auth_request con 0x7ff5a4138c70 auth_method 0 2023-10-30T12:06:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff59bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff59bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.494+0000 7ff59bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff59bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff59bfff640 10 monclient(hunting): handle_auth_done global_id 94204 payload 824 2023-10-30T12:06:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff59bfff640 1 -- 172.21.15.50:0/4144835416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff5a4139080 con 0x7ff5a4138c70 2023-10-30T12:06:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff59bfff640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a4138c70 0x7ff5a4135830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff58c02b830 tx=0x7ff58c008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 10 client.44903.objecter ms_handle_connect 0x7ff5a4138c70 2023-10-30T12:06:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 10 client.44903.objecter resend_mon_ops 2023-10-30T12:06:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c048810 con 0x7ff5a4138c70 2023-10-30T12:06:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 10 monclient: dump: 2023-10-30T12:06:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:22.554 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:22.554 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:22.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7ff58c0323e0 con 0x7ff5a4138c70 2023-10-30T12:06:22.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:22.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a4138c70 msgr2=0x7ff5a4135830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a4138c70 0x7ff5a4135830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff58c02b830 tx=0x7ff58c008030 comp rx=0 tx=0).stop 2023-10-30T12:06:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a4138c70 msgr2=0x7ff5a4135830 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.495+0000 7ff5a0ff9640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff5a4138c70 0x7ff5a4135830 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff58c02b830 tx=0x7ff58c008030 comp rx=0 tx=0).stop 2023-10-30T12:06:22.556 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:22.556 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:22.556 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:22.556 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:06:22.556 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 58, 2023-10-30T12:06:22.557 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:22.557 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:22.557 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:22.557 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.557 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:06:22.558 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:22.558 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:22.558 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:22.558 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:22.558 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:22.559 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:22.559 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:22.559 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:22.559 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:22.560 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:22.560 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:22.560 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:22.560 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:22.560 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.561 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:22.561 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:22.561 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:22.561 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:22.562 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:22.562 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:22.562 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:22.562 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:22.562 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:22.562 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:22.563 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:22.563 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:22.563 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.563 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:22.564 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:22.565 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:22.565 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:22.565 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:22.565 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:22.565 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:22.566 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:22.566 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:22.566 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:22.566 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:22.566 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:22.567 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:22.567 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:22.567 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:22.567 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:22.567 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:22.567 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:22.568 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:22.568 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:22.568 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:22.568 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:22.568 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.569 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:22.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.569 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:22.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.569 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:22.569 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:22.570 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:22.570 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:22.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.570 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:22.570 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:22.570 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.571 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:22.571 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:22.571 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.572 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.572 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.572 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:22.572 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:22.572 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:22.573 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:22.573 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:22.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.573 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:22.574 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:22.574 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:22.574 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:22.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.574 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:22.575 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:22.575 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.575 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:22.575 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:22.576 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.576 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.576 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.576 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:22.577 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:22.577 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:22.577 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:22.577 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:22.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.578 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:22.578 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:22.578 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:22.578 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:22.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.579 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:22.579 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:22.579 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.579 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:22.580 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:22.580 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:22.580 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.580 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.581 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:22.581 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:22.581 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:22.581 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:22.581 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:22.581 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.582 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.582 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:22.582 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:22.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.583 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:22.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.584 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:22.584 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:22.584 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:06:22.584 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.585 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:22.585 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:06:22.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:22.585 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:22.585 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:22.585 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:22.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:22.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:22.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:22.586 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:22.586 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 10 librados: watch_flush enter 2023-10-30T12:06:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 10 librados: watch_flush exit 2023-10-30T12:06:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 20 client.44903.objecter shutdown clearing up homeless session... 2023-10-30T12:06:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 10 client.44903.objecter successfully canceled tick 2023-10-30T12:06:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff5840392a0 msgr2=0x7ff58403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff5840392a0 0x7ff58403b760 secure :-1 s=READY pgs=3984 cs=0 l=1 rev1=1 crypto rx=0x7ff5a413a500 tx=0x7ff59c009270 comp rx=0 tx=0).stop 2023-10-30T12:06:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 10 monclient: shutdown 2023-10-30T12:06:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a4132970 msgr2=0x7ff5a41ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a4132970 0x7ff5a41ca290 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7ff59400b550 tx=0x7ff59400ba10 comp rx=0 tx=0).stop 2023-10-30T12:06:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a37fe640 1 -- 172.21.15.50:0/4144835416 reap_dead start 2023-10-30T12:06:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 shutdown_connections 2023-10-30T12:06:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 --2- 172.21.15.50:0/4144835416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5a4132970 0x7ff5a41ca290 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 >> 172.21.15.50:0/4144835416 conn(0x7ff5a40a4800 msgr2=0x7ff5a40a2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 shutdown_connections 2023-10-30T12:06:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 -- 172.21.15.50:0/4144835416 wait complete. 2023-10-30T12:06:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:22.496+0000 7ff5a99d0640 1 librados: shutdown 2023-10-30T12:06:22.592 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:06:22.592 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:06:42.593 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:06:42.594 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:06:42.594 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:06:42.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:06:42.597 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:06:42.598 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:06:42.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.713+0000 7f48727a9640 10 monclient: get_monmap_and_config 2023-10-30T12:06:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.713+0000 7f48727a9640 10 monclient: build_initial_monmap 2023-10-30T12:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.713+0000 7f48727a9640 10 monclient: monmap: 2023-10-30T12:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:42.714792+0000 2023-10-30T12:06:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:42.714792+0000 2023-10-30T12:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:42.718 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 1 Processor -- start 2023-10-30T12:06:42.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 1 -- start start 2023-10-30T12:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient: init 2023-10-30T12:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T12:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f486c098520 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c099010 0x7f486c09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient(hunting): picked mon.noname-b con 0x7f486c099010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f486c1379f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486c09ad30 con 0x7f486c1379f0 2023-10-30T12:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486c09c460 con 0x7f486c098520 2023-10-30T12:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:42.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c0993f0 con 0x7f486c099010 2023-10-30T12:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.714+0000 7f48727a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f48727a9640 10 monclient(hunting): authenticate will time out at 1247.898016s 2023-10-30T12:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c099010 0x7f486c09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f4870d46640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f486c1379f0 0x7f486c139de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51734/0 (socket says 172.21.15.50:51734) 2023-10-30T12:06:42.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f4870d46640 1 -- 172.21.15.50:0/2677682098 learned_addr learned my addr 172.21.15.50:0/2677682098 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): get_auth_request con 0x7f486c099010 auth_method 0 2023-10-30T12:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:42.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient(hunting): handle_auth_done global_id 44930 payload 274 2023-10-30T12:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 1 -- 172.21.15.50:0/2677682098 >> v1:172.21.15.118:6789/0 conn(0x7f486c098520 legacy=0x7f486c098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 1 -- 172.21.15.50:0/2677682098 >> v1:172.21.15.50:6789/0 conn(0x7f486c1379f0 legacy=0x7f486c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 1 monclient: found mon.noname-b 2023-10-30T12:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.715+0000 7f486b7fe640 1 -- 172.21.15.50:0/2677682098 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c09b260 con 0x7f486c099010 2023-10-30T12:06:42.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486b7fe640 1 --2- 172.21.15.50:0/2677682098 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c099010 0x7f486c09bf20 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f485c0097d0 tx=0x7f485c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5041ac60ce35608 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 1 -- 172.21.15.50:0/2677682098 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c0316f0 con 0x7f486c099010 2023-10-30T12:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: dump: 2023-10-30T12:06:42.732 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:42.733 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: _finish_auth 0 2023-10-30T12:06:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 5 monclient: authenticate success, global_id 44930 2023-10-30T12:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 1 -- 172.21.15.50:0/2677682098 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c031850 con 0x7f486c099010 2023-10-30T12:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 4 set_mon_vals no callback set 2023-10-30T12:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 1 -- 172.21.15.50:0/2677682098 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c031be0 con 0x7f486c099010 2023-10-30T12:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f486affd640 10 monclient: dump: 2023-10-30T12:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 10 monclient: shutdown 2023-10-30T12:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 1 -- 172.21.15.50:0/2677682098 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c099010 msgr2=0x7f486c09bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 1 --2- 172.21.15.50:0/2677682098 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c099010 0x7f486c09bf20 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f485c0097d0 tx=0x7f485c02b790 comp rx=0 tx=0).stop 2023-10-30T12:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 1 -- 172.21.15.50:0/2677682098 shutdown_connections 2023-10-30T12:06:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 1 --2- 172.21.15.50:0/2677682098 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c099010 0x7f486c09bf20 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 1 -- 172.21.15.50:0/2677682098 >> 172.21.15.50:0/2677682098 conn(0x7f486c0a05c0 msgr2=0x7f486c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 1 -- 172.21.15.50:0/2677682098 shutdown_connections 2023-10-30T12:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.716+0000 7f48727a9640 1 -- 172.21.15.50:0/2677682098 wait complete. 2023-10-30T12:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.717+0000 7f48727a9640 10 monclient: build_initial_monmap 2023-10-30T12:06:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.717+0000 7f48727a9640 10 monclient: monmap: 2023-10-30T12:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.717+0000 7f48727a9640 1 librados: starting msgr at 2023-10-30T12:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.717+0000 7f48727a9640 1 librados: starting objecter 2023-10-30T12:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.717+0000 7f48727a9640 1 Processor -- start 2023-10-30T12:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 -- start start 2023-10-30T12:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 librados: setting wanted keys 2023-10-30T12:06:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 librados: calling monclient init 2023-10-30T12:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient: init 2023-10-30T12:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c098520 0x7f486c1ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): picked mon.noname-c con 0x7f486c098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 0x7f486c0965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): picked mon.noname-a con 0x7f486c099010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f486c1379f0 0x7f486c096b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): picked mon.noname-b con 0x7f486c1379f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c097070 con 0x7f486c099010 2023-10-30T12:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c094030 con 0x7f486c1379f0 2023-10-30T12:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c094820 con 0x7f486c098520 2023-10-30T12:06:42.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f48727a9640 10 monclient(hunting): authenticate will time out at 1247.901513s 2023-10-30T12:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 0x7f486c0965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c098520 0x7f486c1ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 0x7f486c0965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47350/0 (socket says 172.21.15.50:47350) 2023-10-30T12:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c098520 0x7f486c1ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60888/0 (socket says 172.21.15.50:60888) 2023-10-30T12:06:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 1 -- 172.21.15.50:0/1907851149 learned_addr learned my addr 172.21.15.50:0/1907851149 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 10 monclient(hunting): get_auth_request con 0x7f486c099010 auth_method 0 2023-10-30T12:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): get_auth_request con 0x7f486c098520 auth_method 0 2023-10-30T12:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 1 -- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 msgr2=0x7f486c0965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 1 -- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 msgr2=0x7f486c0965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 0x7f486c0965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486b7fe640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 0x7f486c0965f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:42.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f4870d46640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f486c1379f0 0x7f486c096b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient(hunting): handle_auth_done global_id 44933 payload 824 2023-10-30T12:06:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 1 -- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f486c1379f0 msgr2=0x7f486c096b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f486c1379f0 0x7f486c096b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 1 -- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 msgr2=0x7f486c0965f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 0x7f486c0965f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:42.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:42.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 1 monclient: found mon.noname-c 2023-10-30T12:06:42.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:42.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.718+0000 7f486bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:42.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f486bfff640 1 -- 172.21.15.50:0/1907851149 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c009400 con 0x7f486c098520 2023-10-30T12:06:42.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4870d46640 10 monclient: get_auth_request con 0x7f486c1379f0 auth_method 0 2023-10-30T12:06:42.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4870d46640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f486c1379f0 0x7f486c096b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:06:42.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f486bfff640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c098520 0x7f486c1ca260 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f486000c360 tx=0x7f486000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:42.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 client.?.objecter ms_handle_connect 0x7f486c098520 2023-10-30T12:06:42.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:42.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 1 -- 172.21.15.50:0/1907851149 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860011790 con 0x7f486c098520 2023-10-30T12:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: dump: 2023-10-30T12:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 1 -- 172.21.15.50:0/1907851149 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48600118f0 con 0x7f486c098520 2023-10-30T12:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 5 monclient: authenticate success, global_id 44933 2023-10-30T12:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 10 monclient: _renew_subs 2023-10-30T12:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486c094de0 con 0x7f486c098520 2023-10-30T12:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 1 -- 172.21.15.50:0/1907851149 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860011ca0 con 0x7f486c098520 2023-10-30T12:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f4868ff9640 10 monclient: dump: 2023-10-30T12:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:42.759 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f486a7fc640 4 set_mon_vals no callback set 2023-10-30T12:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 10 client.44933.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 10 monclient: _renew_subs 2023-10-30T12:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486c1ca7a0 con 0x7f486c098520 2023-10-30T12:06:42.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.719+0000 7f48727a9640 1 librados: init done 2023-10-30T12:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f4868ff9640 1 -- 172.21.15.50:0/1907851149 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4860012420 con 0x7f486c098520 2023-10-30T12:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f48727a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f48727a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f48727a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486c04e160 con 0x7f486c098520 2023-10-30T12:06:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f4868ff9640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48580392f0 0x7f485803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f4868ff9640 1 -- 172.21.15.50:0/1907851149 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6941+0+0 (secure 0 0 0) 0x7f486004ab70 con 0x7f486c098520 2023-10-30T12:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f4868ff9640 10 client.44933.objecter ms_dispatch 0x7f486c000b90 osd_map(196..196 src has 1..196) v4 2023-10-30T12:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f4868ff9640 3 client.44933.objecter handle_osd_map got epochs [196,196] > 0 2023-10-30T12:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f4868ff9640 3 client.44933.objecter handle_osd_map decoding full epoch 196 2023-10-30T12:06:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f4868ff9640 20 client.44933.objecter dump_active .. 0 homeless 2023-10-30T12:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.720+0000 7f486b7fe640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48580392f0 0x7f485803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.721+0000 7f486b7fe640 10 monclient: get_auth_request con 0x7f48580392f0 auth_method 0 2023-10-30T12:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.721+0000 7f486b7fe640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48580392f0 0x7f485803b7b0 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7f485c02b7d0 tx=0x7f485c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:42.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.721+0000 7f4868ff9640 10 client.44933.objecter ms_handle_connect 0x7f48580392f0 2023-10-30T12:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.725+0000 7f4868ff9640 1 -- 172.21.15.50:0/1907851149 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f486000e870 con 0x7f486c098520 2023-10-30T12:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.725+0000 7f4868ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:42.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.725+0000 7f4868ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.971+0000 7f48727a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.971+0000 7f48727a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.971+0000 7f48727a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.971+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f486c0745c0 con 0x7f486c098520 2023-10-30T12:06:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.972+0000 7f4868ff9640 1 -- 172.21.15.50:0/1907851149 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f486000e280 con 0x7f486c098520 2023-10-30T12:06:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.972+0000 7f4868ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.972+0000 7f4868ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:42.975 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 10 librados: watch_flush enter 2023-10-30T12:06:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 10 librados: watch_flush exit 2023-10-30T12:06:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 20 client.44933.objecter shutdown clearing up homeless session... 2023-10-30T12:06:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 10 client.44933.objecter successfully canceled tick 2023-10-30T12:06:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48580392f0 msgr2=0x7f485803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48580392f0 0x7f485803b7b0 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7f485c02b7d0 tx=0x7f485c008030 comp rx=0 tx=0).stop 2023-10-30T12:06:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 10 monclient: shutdown 2023-10-30T12:06:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c098520 msgr2=0x7f486c1ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c098520 0x7f486c1ca260 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f486000c360 tx=0x7f486000c820 comp rx=0 tx=0).stop 2023-10-30T12:06:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.973+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 shutdown_connections 2023-10-30T12:06:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48580392f0 0x7f485803b7b0 unknown :-1 s=CLOSED pgs=3992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f486c1379f0 0x7f486c096b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f486c099010 0x7f486c0965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 --2- 172.21.15.50:0/1907851149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f486c098520 0x7f486c1ca260 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 >> 172.21.15.50:0/1907851149 conn(0x7f486c0a05c0 msgr2=0x7f486c070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 shutdown_connections 2023-10-30T12:06:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 -- 172.21.15.50:0/1907851149 wait complete. 2023-10-30T12:06:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:42.974+0000 7f48727a9640 1 librados: shutdown 2023-10-30T12:06:45.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:46.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient: get_monmap_and_config 2023-10-30T12:06:46.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient: build_initial_monmap 2023-10-30T12:06:46.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient: monmap: 2023-10-30T12:06:46.106 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:46.106 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:46.106 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:46.104590+0000 2023-10-30T12:06:46.106 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:46.104590+0000 2023-10-30T12:06:46.106 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:46.107 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:46.107 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:46.107 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:46.107 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:46.107 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:46.108 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:46.108 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:46.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 1 Processor -- start 2023-10-30T12:06:46.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 1 -- start start 2023-10-30T12:06:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient: init 2023-10-30T12:06:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:06:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 0x7f75a409bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient(hunting): picked mon.noname-a con 0x7f75a4098520 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:46.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75a4098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:46.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75a4137980 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:46.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a4099930 con 0x7f75a4098900 2023-10-30T12:06:46.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a409aca0 con 0x7f75a4137980 2023-10-30T12:06:46.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a409af80 con 0x7f75a4098520 2023-10-30T12:06:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.104+0000 7f75aace7640 10 monclient(hunting): authenticate will time out at 1251.287860s 2023-10-30T12:06:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 0x7f75a409bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a3fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f75a4098900 0x7f75a4098d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51748/0 (socket says 172.21.15.50:51748) 2023-10-30T12:06:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a3fff640 1 -- 172.21.15.50:0/2078965525 learned_addr learned my addr 172.21.15.50:0/2078965525 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:06:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 1 --2- 172.21.15.50:0/2078965525 >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 0x7f75a409bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47362/0 (socket says 172.21.15.50:47362) 2023-10-30T12:06:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): get_auth_request con 0x7f75a4098520 auth_method 0 2023-10-30T12:06:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 1 -- 172.21.15.50:0/2078965525 >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 msgr2=0x7f75a409bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 1 -- 172.21.15.50:0/2078965525 >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 msgr2=0x7f75a409bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 1 --2- 172.21.15.50:0/2078965525 >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 0x7f75a409bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 1 --2- 172.21.15.50:0/2078965525 >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 0x7f75a409bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833959793 0 0) 0x7f75a409aca0 con 0x7f75a4137980 2023-10-30T12:06:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): _negotiate 2023-10-30T12:06:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): my global_id is 94246 2023-10-30T12:06:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7594003520 con 0x7f75a4137980 2023-10-30T12:06:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991392489 0 0) 0x7f7594003520 con 0x7f75a4137980 2023-10-30T12:06:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 msgr2=0x7f75a409bf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 --2- 172.21.15.50:0/2078965525 >> v2:172.21.15.50:3300/0 conn(0x7f75a4098520 0x7f75a409bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 >> v1:172.21.15.50:6789/0 conn(0x7f75a4098900 legacy=0x7f75a4098d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:06:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:06:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a409b260 con 0x7f75a4137980 2023-10-30T12:06:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: _finish_auth 0 2023-10-30T12:06:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:06:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f759c0019f0 con 0x7f75a4137980 2023-10-30T12:06:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75aace7640 5 monclient: authenticate success, global_id 94246 2023-10-30T12:06:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75aace7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:06:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: dump: 2023-10-30T12:06:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:46.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:46.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:06:46.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:46.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 >> v1:172.21.15.118:6789/0 conn(0x7f75a4137980 legacy=0x7f75a4139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:46.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:06:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7594003970 0x7f7594003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): picked mon.a con 0x7f7594003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f7594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): picked mon.c con 0x7f75940047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7594005380 0x7f759400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): picked mon.b con 0x7f7594005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a409af80 con 0x7f7594003970 2023-10-30T12:06:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759400c210 con 0x7f7594005380 2023-10-30T12:06:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759400c3d0 con 0x7f75940047c0 2023-10-30T12:06:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a37fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:06:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75aace7640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7594003970 0x7f7594003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): get_auth_request con 0x7f7594003970 auth_method 0 2023-10-30T12:06:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a8a83640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.105+0000 7f75a3fff640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f7594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): get_auth_request con 0x7f75940047c0 auth_method 0 2023-10-30T12:06:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:46.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7594005380 0x7f759400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a8a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:46.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a8a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:46.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a8a83640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:06:46.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 10 monclient(hunting): get_auth_request con 0x7f7594005380 auth_method 0 2023-10-30T12:06:46.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:46.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:46.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:46.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:46.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:06:46.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:46.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:46.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:06:46.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient(hunting): handle_auth_done global_id 94246 payload 293 2023-10-30T12:06:46.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 1 -- 172.21.15.50:0/2078965525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7594005380 msgr2=0x7f759400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:46.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7594005380 0x7f759400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 1 -- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7594003970 msgr2=0x7f7594003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:46.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7594003970 0x7f7594003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 1 monclient: found mon.c 2023-10-30T12:06:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 1 -- 172.21.15.50:0/2078965525 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7594003520 con 0x7f75940047c0 2023-10-30T12:06:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient: _finish_auth 0 2023-10-30T12:06:46.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:46.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:46.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a9284640 1 -- 172.21.15.50:0/2078965525 reap_dead start 2023-10-30T12:06:46.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.106+0000 7f75a8a83640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7594003970 0x7f7594003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:46.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a3fff640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f7594004c40 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f7590001a40 tx=0x7f75940057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9ed6ae29c558877 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:46.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590008cf0 con 0x7f75940047c0 2023-10-30T12:06:46.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:46.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:46.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 10 monclient: dump: 2023-10-30T12:06:46.147 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:46.148 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:46.148 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.148 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.148 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:46.148 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:46.149 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:46.149 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:46.149 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:46.149 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:46.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:46.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590008e70 con 0x7f75940047c0 2023-10-30T12:06:46.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:46.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:46.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 4 set_mon_vals no callback set 2023-10-30T12:06:46.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 1 -- 172.21.15.50:0/2078965525 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590004ac0 con 0x7f75940047c0 2023-10-30T12:06:46.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:46.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:46.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75a37fe640 10 monclient: dump: 2023-10-30T12:06:46.151 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:46.151 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:46.151 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.152 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.152 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:46.152 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:46.152 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:46.152 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:46.153 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:46.153 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:46.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 10 monclient: shutdown 2023-10-30T12:06:46.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:46.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 1 -- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 msgr2=0x7f7594004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:46.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f7594004c40 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f7590001a40 tx=0x7f75940057f0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 1 -- 172.21.15.50:0/2078965525 shutdown_connections 2023-10-30T12:06:46.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 1 --2- 172.21.15.50:0/2078965525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f7594004c40 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 1 -- 172.21.15.50:0/2078965525 >> 172.21.15.50:0/2078965525 conn(0x7f75a40a05c0 msgr2=0x7f75a409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:46.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 1 -- 172.21.15.50:0/2078965525 shutdown_connections 2023-10-30T12:06:46.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.107+0000 7f75aace7640 1 -- 172.21.15.50:0/2078965525 wait complete. 2023-10-30T12:06:46.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 10 monclient: build_initial_monmap 2023-10-30T12:06:46.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 10 monclient: monmap: 2023-10-30T12:06:46.158 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:46.158 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:46.159 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:46.160 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:46.160 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:46.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 1 librados: starting msgr at 2023-10-30T12:06:46.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 1 librados: starting objecter 2023-10-30T12:06:46.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 1 Processor -- start 2023-10-30T12:06:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 1 -- start start 2023-10-30T12:06:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 1 librados: setting wanted keys 2023-10-30T12:06:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 1 librados: calling monclient init 2023-10-30T12:06:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.108+0000 7f75aace7640 10 monclient: init 2023-10-30T12:06:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:06:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f75a41ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): picked mon.noname-c con 0x7f75940047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 0x7f75a40965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): picked mon.noname-a con 0x7f75a4096170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a408f3e0 0x7f75a408f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): picked mon.noname-b con 0x7f75a408f3e0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:46.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:46.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4071d50 con 0x7f75a4096170 2023-10-30T12:06:46.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:46.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40a30b0 con 0x7f75a408f3e0 2023-10-30T12:06:46.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:46.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4092c20 con 0x7f75940047c0 2023-10-30T12:06:46.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:46.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75aace7640 10 monclient(hunting): authenticate will time out at 1251.292519s 2023-10-30T12:06:46.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f75a41ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f75a41ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60954/0 (socket says 172.21.15.50:60954) 2023-10-30T12:06:46.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 -- 172.21.15.50:0/34169843 learned_addr learned my addr 172.21.15.50:0/34169843 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:46.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): get_auth_request con 0x7f75940047c0 auth_method 0 2023-10-30T12:06:46.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 0x7f75a40965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:46.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:46.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 10 monclient(hunting): get_auth_request con 0x7f75a4096170 auth_method 0 2023-10-30T12:06:46.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:46.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 1 -- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 msgr2=0x7f75a40965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:06:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 1 -- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 msgr2=0x7f75a40965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:06:46.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 0x7f75a40965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:46.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:46.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a3fff640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 0x7f75a40965f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:46.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a408f3e0 0x7f75a408f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 10 monclient(hunting): get_auth_request con 0x7f75a408f3e0 auth_method 0 2023-10-30T12:06:46.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:46.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:46.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:46.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:46.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:46.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a9284640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:46.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient(hunting): handle_auth_done global_id 44936 payload 824 2023-10-30T12:06:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 -- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a408f3e0 msgr2=0x7f75a408f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a408f3e0 0x7f75a408f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 -- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 msgr2=0x7f75a40965f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 0x7f75a40965f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient: _finish_hunting 0 2023-10-30T12:06:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 monclient: found mon.noname-c 2023-10-30T12:06:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.109+0000 7f75a8a83640 1 -- 172.21.15.50:0/34169843 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a4090dd0 con 0x7f75940047c0 2023-10-30T12:06:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a8a83640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f75a41ca030 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f7598006320 tx=0x7f7598006720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 client.?.objecter ms_handle_connect 0x7f75940047c0 2023-10-30T12:06:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 1 -- 172.21.15.50:0/34169843 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598010960 con 0x7f75940047c0 2023-10-30T12:06:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:46.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: dump: 2023-10-30T12:06:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:46.174 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:46.174 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:46.174 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:46.174 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:46.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: _finish_auth 0 2023-10-30T12:06:46.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: _check_auth_tickets 2023-10-30T12:06:46.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:46.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 1 -- 172.21.15.50:0/34169843 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598010ac0 con 0x7f75940047c0 2023-10-30T12:06:46.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 5 monclient: authenticate success, global_id 44936 2023-10-30T12:06:46.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 10 monclient: _renew_subs 2023-10-30T12:06:46.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:46.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a4092940 con 0x7f75940047c0 2023-10-30T12:06:46.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:46.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 1 -- 172.21.15.50:0/34169843 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598010e70 con 0x7f75940047c0 2023-10-30T12:06:46.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:46.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:46.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a17fa640 10 monclient: dump: 2023-10-30T12:06:46.177 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:46.177 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:46.177 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.178 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:46.178 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:46.178 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:46.178 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:46.178 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:46.178 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:46.179 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:46.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75a2ffd640 4 set_mon_vals no callback set 2023-10-30T12:06:46.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 10 client.44936.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:46.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 10 monclient: _renew_subs 2023-10-30T12:06:46.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:46.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a40908b0 con 0x7f75940047c0 2023-10-30T12:06:46.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.110+0000 7f75aace7640 1 librados: init done 2023-10-30T12:06:46.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a17fa640 1 -- 172.21.15.50:0/34169843 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7598014430 con 0x7f75940047c0 2023-10-30T12:06:46.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75aace7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:46.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75aace7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:46.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75aace7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:46.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a404e160 con 0x7f75940047c0 2023-10-30T12:06:46.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a17fa640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75880392f0 0x7f758803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:46.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a17fa640 1 -- 172.21.15.50:0/34169843 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 7050+0+0 (secure 0 0 0) 0x7f759804b5a0 con 0x7f75940047c0 2023-10-30T12:06:46.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a17fa640 10 client.44936.objecter ms_dispatch 0x7f75a4000b90 osd_map(199..199 src has 1..199) v4 2023-10-30T12:06:46.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a17fa640 3 client.44936.objecter handle_osd_map got epochs [199,199] > 0 2023-10-30T12:06:46.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a17fa640 3 client.44936.objecter handle_osd_map decoding full epoch 199 2023-10-30T12:06:46.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a17fa640 20 client.44936.objecter dump_active .. 0 homeless 2023-10-30T12:06:46.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a3fff640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75880392f0 0x7f758803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:46.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.111+0000 7f75a3fff640 10 monclient: get_auth_request con 0x7f75880392f0 auth_method 0 2023-10-30T12:06:46.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.112+0000 7f75a17fa640 10 client.44936.objecter ms_handle_connect 0x7f75880392f0 2023-10-30T12:06:46.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.112+0000 7f75a3fff640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75880392f0 0x7f758803b7b0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f7590035000 tx=0x7f759000b030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:46.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.116+0000 7f75a17fa640 1 -- 172.21.15.50:0/34169843 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f759800b5e0 con 0x7f75940047c0 2023-10-30T12:06:46.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.116+0000 7f75a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:46.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.116+0000 7f75a17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.367+0000 7f75aace7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.367+0000 7f75aace7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.367+0000 7f75aace7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.367+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f75a406e890 con 0x7f75940047c0 2023-10-30T12:06:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.368+0000 7f75a17fa640 1 -- 172.21.15.50:0/34169843 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f759801c3b0 con 0x7f75940047c0 2023-10-30T12:06:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.368+0000 7f75a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.368+0000 7f75a17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:46.371 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":26,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 10 librados: watch_flush enter 2023-10-30T12:06:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 10 librados: watch_flush exit 2023-10-30T12:06:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 20 client.44936.objecter shutdown clearing up homeless session... 2023-10-30T12:06:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 10 client.44936.objecter successfully canceled tick 2023-10-30T12:06:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75880392f0 msgr2=0x7f758803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75880392f0 0x7f758803b7b0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f7590035000 tx=0x7f759000b030 comp rx=0 tx=0).stop 2023-10-30T12:06:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 10 monclient: shutdown 2023-10-30T12:06:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:46.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 msgr2=0x7f75a41ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:46.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.369+0000 7f75aace7640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f75a41ca030 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f7598006320 tx=0x7f7598006720 comp rx=0 tx=0).stop 2023-10-30T12:06:46.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 shutdown_connections 2023-10-30T12:06:46.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75880392f0 0x7f758803b7b0 unknown :-1 s=CLOSED pgs=3993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a408f3e0 0x7f75a408f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a4096170 0x7f75a40965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 --2- 172.21.15.50:0/34169843 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75940047c0 0x7f75a41ca030 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:46.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 >> 172.21.15.50:0/34169843 conn(0x7f75a40a05c0 msgr2=0x7f75a406ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:46.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 shutdown_connections 2023-10-30T12:06:46.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 -- 172.21.15.50:0/34169843 wait complete. 2023-10-30T12:06:46.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:46.370+0000 7f75aace7640 1 librados: shutdown 2023-10-30T12:06:49.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:49.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.497+0000 7fe52f61a640 10 monclient: get_monmap_and_config 2023-10-30T12:06:49.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.497+0000 7fe52f61a640 10 monclient: build_initial_monmap 2023-10-30T12:06:49.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.497+0000 7fe52f61a640 10 monclient: monmap: 2023-10-30T12:06:49.500 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:49.500 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:49.500 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:49.498677+0000 2023-10-30T12:06:49.500 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:49.498677+0000 2023-10-30T12:06:49.500 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:49.501 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:49.501 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:49.501 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:49.501 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:49.501 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:49.502 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:49.502 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:49.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 1 Processor -- start 2023-10-30T12:06:49.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 1 -- start start 2023-10-30T12:06:49.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient: init 2023-10-30T12:06:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T12:06:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient(hunting): picked mon.noname-a con 0x7fe528131b20 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe528132270 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient(hunting): picked mon.noname-b con 0x7fe528132670 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52809a9b0 con 0x7fe528132270 2023-10-30T12:06:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe528131f00 con 0x7fe528131b20 2023-10-30T12:06:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52809d920 con 0x7fe528132670 2023-10-30T12:06:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.498+0000 7fe52f61a640 10 monclient(hunting): authenticate will time out at 1254.681932s 2023-10-30T12:06:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53032/0 (socket says 172.21.15.50:53032) 2023-10-30T12:06:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 learned_addr learned my addr 172.21.15.50:0/3325289102 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request con 0x7fe528131b20 auth_method 0 2023-10-30T12:06:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request con 0x7fe528132670 auth_method 0 2023-10-30T12:06:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:49.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:06:49.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:49.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:49.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:49.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.499+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request con 0x7fe528132670 auth_method 0 2023-10-30T12:06:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request con 0x7fe528131b20 auth_method 0 2023-10-30T12:06:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:06:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:06:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:49.700+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:06:50.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:50.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request con 0x7fe528132670 auth_method 0 2023-10-30T12:06:50.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:50.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:50.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:50.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:50.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request con 0x7fe528131b20 auth_method 0 2023-10-30T12:06:50.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:50.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:50.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:50.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:50.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:50.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:50.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:06:50.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:50.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:50.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:06:50.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.101+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:06:50.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:50.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:50.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request con 0x7fe528131b20 auth_method 0 2023-10-30T12:06:50.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:50.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:50.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:50.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request con 0x7fe528132670 auth_method 0 2023-10-30T12:06:50.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:50.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:50.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:50.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:50.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:50.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:50.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:06:50.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:06:50.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:50.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:50.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:50.902+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:06:52.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient(hunting): tick 2023-10-30T12:06:52.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:06:52.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 monclient(hunting): continuing hunt 2023-10-30T12:06:52.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:06:52.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 msgr2=0x7fe528139f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:52.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe528132670 0x7fe528139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:52.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 msgr2=0x7fe52809f5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:52.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.50:3300/0 conn(0x7fe528131b20 0x7fe52809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:52.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 >> v1:172.21.15.118:6789/0 conn(0x7fe528132270 legacy=0x7fe528071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:52.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient: _add_conns ranks=[0,4] 2023-10-30T12:06:52.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe50c0036c0 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:52.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:52.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient(hunting): picked mon.noname-c con 0x7fe50c007350 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:52.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52809a0b0 con 0x7fe50c0036c0 2023-10-30T12:06:52.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:52.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe528131f00 con 0x7fe50c007350 2023-10-30T12:06:52.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe51effd640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:52.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:52.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request con 0x7fe50c007350 auth_method 0 2023-10-30T12:06:52.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:52.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:52.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:52.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:06:52.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:52.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:52.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.499+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:52.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.699+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v1:172.21.15.50:6789/0 conn(0x7fe50c0036c0 legacy=0x7fe50c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-10-30T12:06:52.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.699+0000 7fe52d3b6640 1 -- 172.21.15.50:0/3325289102 >> v1:172.21.15.50:6789/0 conn(0x7fe50c0036c0 legacy=0x7fe50c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:06:52.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.699+0000 7fe52d3b6640 1 --1- 172.21.15.50:0/3325289102 >> v1:172.21.15.50:6789/0 conn(0x7fe50c0036c0 0x7fe50c005b20 :-1 s=OPENED pgs=26 cs=1 l=1).handle_message read tag failed 2023-10-30T12:06:52.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.699+0000 7fe52d3b6640 1 --1- 172.21.15.50:0/3325289102 >> v1:172.21.15.50:6789/0 conn(0x7fe50c0036c0 0x7fe50c005b20 :-1 s=OPENED pgs=26 cs=1 l=1).fault on lossy channel, failing 2023-10-30T12:06:52.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.699+0000 7fe51ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.50:6789/0 2023-10-30T12:06:52.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:52.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request con 0x7fe50c007350 auth_method 0 2023-10-30T12:06:52.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:52.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:52.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:52.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:52.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:52.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:52.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:52.700+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:06:53.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:53.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request con 0x7fe50c007350 auth_method 0 2023-10-30T12:06:53.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:53.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:53.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:53.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:53.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:53.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:53.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.101+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:06:53.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request con 0x7fe50c007350 auth_method 0 2023-10-30T12:06:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:53.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:06:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:06:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:06:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:53.903+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:06:55.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): tick 2023-10-30T12:06:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:06:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 monclient(hunting): continuing hunt 2023-10-30T12:06:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:06:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 msgr2=0x7fe50c0097f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:06:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 --2- 172.21.15.50:0/3325289102 >> v2:172.21.15.118:3300/0 conn(0x7fe50c007350 0x7fe50c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 >> v1:172.21.15.50:6789/0 conn(0x7fe50c0036c0 legacy=0x7fe50c005b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T12:06:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe52dbb7640 1 -- 172.21.15.50:0/3325289102 reap_dead start 2023-10-30T12:06:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe50c0036c0 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe50c009740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): picked mon.noname-b con 0x7fe50c009f00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe50c00a3d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50c00af60 con 0x7fe50c0036c0 2023-10-30T12:06:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50c00aac0 con 0x7fe50c00a3d0 2023-10-30T12:06:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 1 -- 172.21.15.50:0/3325289102 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe528131f00 con 0x7fe50c009f00 2023-10-30T12:06:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe51effd640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe50c009740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request con 0x7fe50c009f00 auth_method 0 2023-10-30T12:06:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe52cbb5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.499+0000 7fe52cbb5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe51ffff640 1 -- 172.21.15.50:0/3325289102 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132692677 0 0) 0x7fe50c00af60 con 0x7fe50c0036c0 2023-10-30T12:06:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe51ffff640 10 monclient(hunting): _negotiate 2023-10-30T12:06:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe51ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe51ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe51ffff640 10 monclient(hunting): my global_id is 104106 2023-10-30T12:06:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe51ffff640 1 -- 172.21.15.50:0/3325289102 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe514003520 con 0x7fe50c0036c0 2023-10-30T12:06:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_done global_id 44948 payload 274 2023-10-30T12:06:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v1:172.21.15.118:6789/0 conn(0x7fe50c00a3d0 legacy=0x7fe50c00d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 >> v1:172.21.15.50:6789/0 conn(0x7fe50c0036c0 legacy=0x7fe50c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 10 monclient: _finish_hunting 0 2023-10-30T12:06:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 1 monclient: found mon.noname-b 2023-10-30T12:06:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.500+0000 7fe52cbb5640 1 -- 172.21.15.50:0/3325289102 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c0109a0 con 0x7fe50c009f00 2023-10-30T12:06:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe50c009740 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fe518005290 tx=0x7fe51802c110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d75aaf39ab700b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 1 -- 172.21.15.50:0/3325289102 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51802cd80 con 0x7fe50c009f00 2023-10-30T12:06:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: dump: 2023-10-30T12:06:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.513 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: _finish_auth 0 2023-10-30T12:06:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 1 -- 172.21.15.50:0/3325289102 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518037460 con 0x7fe50c009f00 2023-10-30T12:06:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe52f61a640 5 monclient: authenticate success, global_id 44948 2023-10-30T12:06:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe52f61a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe52f61a640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 1 -- 172.21.15.50:0/3325289102 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518037810 con 0x7fe50c009f00 2023-10-30T12:06:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe52f61a640 4 set_mon_vals no callback set 2023-10-30T12:06:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.501+0000 7fe51ffff640 10 monclient: dump: 2023-10-30T12:06:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.517 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 10 monclient: shutdown 2023-10-30T12:06:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 1 -- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 msgr2=0x7fe50c009740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe50c009740 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fe518005290 tx=0x7fe51802c110 comp rx=0 tx=0).stop 2023-10-30T12:06:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 1 -- 172.21.15.50:0/3325289102 shutdown_connections 2023-10-30T12:06:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 1 --2- 172.21.15.50:0/3325289102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe50c009740 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 1 -- 172.21.15.50:0/3325289102 >> 172.21.15.50:0/3325289102 conn(0x7fe5280a05c0 msgr2=0x7fe52809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 1 -- 172.21.15.50:0/3325289102 shutdown_connections 2023-10-30T12:06:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.502+0000 7fe52f61a640 1 -- 172.21.15.50:0/3325289102 wait complete. 2023-10-30T12:06:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 10 monclient: build_initial_monmap 2023-10-30T12:06:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 10 monclient: monmap: 2023-10-30T12:06:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:55.525 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 1 librados: starting msgr at 2023-10-30T12:06:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 1 librados: starting objecter 2023-10-30T12:06:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 1 Processor -- start 2023-10-30T12:06:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 1 -- start start 2023-10-30T12:06:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 1 librados: setting wanted keys 2023-10-30T12:06:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 1 librados: calling monclient init 2023-10-30T12:06:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.503+0000 7fe52f61a640 10 monclient: init 2023-10-30T12:06:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:06:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe50c0036c0 0x7fe5281c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): picked mon.noname-b con 0x7fe50c0036c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe528130c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): picked mon.noname-c con 0x7fe50c009f00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50c00a3d0 0x7fe5281311d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): picked mon.noname-a con 0x7fe50c00a3d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528071ca0 con 0x7fe50c00a3d0 2023-10-30T12:06:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52812dcf0 con 0x7fe50c0036c0 2023-10-30T12:06:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52812df60 con 0x7fe50c009f00 2023-10-30T12:06:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52f61a640 10 monclient(hunting): authenticate will time out at 1260.687346s 2023-10-30T12:06:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe528130c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe528130c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35484/0 (socket says 172.21.15.50:35484) 2023-10-30T12:06:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 -- 172.21.15.50:0/1567380023 learned_addr learned my addr 172.21.15.50:0/1567380023 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request con 0x7fe50c009f00 auth_method 0 2023-10-30T12:06:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50c00a3d0 0x7fe5281311d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request con 0x7fe50c00a3d0 auth_method 0 2023-10-30T12:06:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe50c0036c0 0x7fe5281c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request con 0x7fe50c0036c0 auth_method 0 2023-10-30T12:06:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52d3b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52d3b6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52d3b6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient(hunting): handle_auth_done global_id 44951 payload 824 2023-10-30T12:06:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 -- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe50c0036c0 msgr2=0x7fe5281c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe50c0036c0 0x7fe5281c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 -- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50c00a3d0 msgr2=0x7fe5281311d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50c00a3d0 0x7fe5281311d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient: _finish_hunting 0 2023-10-30T12:06:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 monclient: found mon.noname-c 2023-10-30T12:06:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52cbb5640 1 -- 172.21.15.50:0/1567380023 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51800f040 con 0x7fe50c009f00 2023-10-30T12:06:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.504+0000 7fe52dbb7640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50c00a3d0 0x7fe5281311d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52cbb5640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe528130c90 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fe51802c720 tx=0x7fe518001190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 client.?.objecter ms_handle_connect 0x7fe50c009f00 2023-10-30T12:06:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 1 -- 172.21.15.50:0/1567380023 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518043630 con 0x7fe50c009f00 2023-10-30T12:06:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: dump: 2023-10-30T12:06:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.540 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: _finish_auth 0 2023-10-30T12:06:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:06:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 1 -- 172.21.15.50:0/1567380023 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518043790 con 0x7fe50c009f00 2023-10-30T12:06:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 5 monclient: authenticate success, global_id 44951 2023-10-30T12:06:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 1 -- 172.21.15.50:0/1567380023 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518043b40 con 0x7fe50c009f00 2023-10-30T12:06:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 10 monclient: _renew_subs 2023-10-30T12:06:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52812e240 con 0x7fe50c009f00 2023-10-30T12:06:55.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51f7fe640 4 set_mon_vals no callback set 2023-10-30T12:06:55.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:55.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe51dffb640 10 monclient: dump: 2023-10-30T12:06:55.542 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.543 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.543 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.543 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.543 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.543 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.544 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.544 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.544 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.544 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 10 client.44951.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:55.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 10 monclient: _renew_subs 2023-10-30T12:06:55.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52812c150 con 0x7fe50c009f00 2023-10-30T12:06:55.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.505+0000 7fe52f61a640 1 librados: init done 2023-10-30T12:06:55.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe51dffb640 1 -- 172.21.15.50:0/1567380023 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe518040030 con 0x7fe50c009f00 2023-10-30T12:06:55.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe51dffb640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe50c03ac30 0x7fe50c03d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe52f61a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:55.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe52f61a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:55.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe52f61a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe51dffb640 1 -- 172.21.15.50:0/1567380023 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fe51806d300 con 0x7fe50c009f00 2023-10-30T12:06:55.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52804e160 con 0x7fe50c009f00 2023-10-30T12:06:55.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe51dffb640 10 client.44951.objecter ms_dispatch 0x7fe528000b90 osd_map(203..203 src has 1..203) v4 2023-10-30T12:06:55.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe51dffb640 3 client.44951.objecter handle_osd_map got epochs [203,203] > 0 2023-10-30T12:06:55.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe51dffb640 3 client.44951.objecter handle_osd_map decoding full epoch 203 2023-10-30T12:06:55.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe51dffb640 20 client.44951.objecter dump_active .. 0 homeless 2023-10-30T12:06:55.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.506+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe50c03ac30 0x7fe50c03d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.507+0000 7fe52d3b6640 10 monclient: get_auth_request con 0x7fe50c03ac30 auth_method 0 2023-10-30T12:06:55.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.507+0000 7fe52d3b6640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe50c03ac30 0x7fe50c03d0f0 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7fe524001eb0 tx=0x7fe524008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.507+0000 7fe51dffb640 10 client.44951.objecter ms_handle_connect 0x7fe50c03ac30 2023-10-30T12:06:55.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.511+0000 7fe51dffb640 1 -- 172.21.15.50:0/1567380023 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe518035dc0 con 0x7fe50c009f00 2023-10-30T12:06:55.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.511+0000 7fe51dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:55.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.511+0000 7fe51dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:55.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.758+0000 7fe52f61a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.758+0000 7fe52f61a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.758+0000 7fe52f61a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.758+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe52804e160 con 0x7fe50c009f00 2023-10-30T12:06:55.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.758+0000 7fe51dffb640 1 -- 172.21.15.50:0/1567380023 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fe518035360 con 0x7fe50c009f00 2023-10-30T12:06:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.758+0000 7fe51dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.758+0000 7fe51dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:55.761 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 10 librados: watch_flush enter 2023-10-30T12:06:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 10 librados: watch_flush exit 2023-10-30T12:06:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 20 client.44951.objecter shutdown clearing up homeless session... 2023-10-30T12:06:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 10 client.44951.objecter successfully canceled tick 2023-10-30T12:06:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe50c03ac30 msgr2=0x7fe50c03d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe50c03ac30 0x7fe50c03d0f0 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7fe524001eb0 tx=0x7fe524008040 comp rx=0 tx=0).stop 2023-10-30T12:06:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 10 monclient: shutdown 2023-10-30T12:06:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 msgr2=0x7fe528130c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe528130c90 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fe51802c720 tx=0x7fe518001190 comp rx=0 tx=0).stop 2023-10-30T12:06:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 shutdown_connections 2023-10-30T12:06:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe50c03ac30 0x7fe50c03d0f0 unknown :-1 s=CLOSED pgs=3996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50c00a3d0 0x7fe5281311d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50c009f00 0x7fe528130c90 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 --2- 172.21.15.50:0/1567380023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe50c0036c0 0x7fe5281c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 >> 172.21.15.50:0/1567380023 conn(0x7fe5280a05c0 msgr2=0x7fe52806db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 shutdown_connections 2023-10-30T12:06:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 -- 172.21.15.50:0/1567380023 wait complete. 2023-10-30T12:06:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.760+0000 7fe52f61a640 1 librados: shutdown 2023-10-30T12:06:55.772 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:06:55.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:06:55.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:06:55.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.877+0000 7f3832875640 10 monclient: get_monmap_and_config 2023-10-30T12:06:55.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.877+0000 7f3832875640 10 monclient: build_initial_monmap 2023-10-30T12:06:55.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.877+0000 7f3832875640 10 monclient: monmap: 2023-10-30T12:06:55.880 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:55.880 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.880 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:55.878892+0000 2023-10-30T12:06:55.880 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:55.878892+0000 2023-10-30T12:06:55.880 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:55.882 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 1 Processor -- start 2023-10-30T12:06:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 1 -- start start 2023-10-30T12:06:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 10 monclient: init 2023-10-30T12:06:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:55.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-10-30T12:06:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f382c096540 0x7f382c096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 10 monclient(hunting): picked mon.noname-c con 0x7f382c096540 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f382c096e60 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c09b170 0x7f382c09b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 10 monclient(hunting): picked mon.noname-b con 0x7f382c09b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.878+0000 7f3832875640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382c09bb40 con 0x7f382c096e60 2023-10-30T12:06:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3832875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3832875640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382c09bd30 con 0x7f382c096540 2023-10-30T12:06:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3832875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3832875640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c09c400 con 0x7f382c09b170 2023-10-30T12:06:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3832875640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3832875640 10 monclient(hunting): authenticate will time out at 1261.062032s 2023-10-30T12:06:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c09b170 0x7f382c09b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382b7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f382c096e60 0x7f382c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37888/0 (socket says 172.21.15.50:37888) 2023-10-30T12:06:55.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c09b170 0x7f382c09b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35490/0 (socket says 172.21.15.50:35490) 2023-10-30T12:06:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382b7fe640 1 -- 172.21.15.50:0/1178087252 learned_addr learned my addr 172.21.15.50:0/1178087252 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:06:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): get_auth_request con 0x7f382c09b170 auth_method 0 2023-10-30T12:06:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 1 --2- 172.21.15.50:0/1178087252 >> v2:172.21.15.118:3300/0 conn(0x7f382c096540 0x7f382c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 10 monclient(hunting): get_auth_request con 0x7f382c096540 auth_method 0 2023-10-30T12:06:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient(hunting): handle_auth_done global_id 44954 payload 274 2023-10-30T12:06:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 1 -- 172.21.15.50:0/1178087252 >> v2:172.21.15.118:3300/0 conn(0x7f382c096540 msgr2=0x7f382c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382affd640 1 -- 172.21.15.50:0/1178087252 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526690439 0 0) 0x7f382c09bb40 con 0x7f382c096e60 2023-10-30T12:06:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 1 --2- 172.21.15.50:0/1178087252 >> v2:172.21.15.118:3300/0 conn(0x7f382c096540 0x7f382c096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 1 -- 172.21.15.50:0/1178087252 >> v1:172.21.15.50:6789/0 conn(0x7f382c096e60 legacy=0x7f382c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient: _finish_hunting 0 2023-10-30T12:06:55.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 1 monclient: found mon.noname-b 2023-10-30T12:06:55.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:55.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f3830e12640 1 -- 172.21.15.50:0/1178087252 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c09c120 con 0x7f382c09b170 2023-10-30T12:06:55.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.879+0000 7f382affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:06:55.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3830e12640 1 --2- 172.21.15.50:0/1178087252 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c09b170 0x7f382c09b600 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f3824009d90 tx=0x7f382402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2d87f271bdd5ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 1 -- 172.21.15.50:0/1178087252 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382402cc50 con 0x7f382c09b170 2023-10-30T12:06:55.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: dump: 2023-10-30T12:06:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.902 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: _finish_auth 0 2023-10-30T12:06:55.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:55.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:55.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 5 monclient: authenticate success, global_id 44954 2023-10-30T12:06:55.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:55.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 1 -- 172.21.15.50:0/1178087252 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382402cdb0 con 0x7f382c09b170 2023-10-30T12:06:55.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:55.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:55.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 4 set_mon_vals no callback set 2023-10-30T12:06:55.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 1 -- 172.21.15.50:0/1178087252 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824034660 con 0x7f382c09b170 2023-10-30T12:06:55.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:55.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f382affd640 10 monclient: dump: 2023-10-30T12:06:55.905 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.905 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.906 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.906 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.906 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.906 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.906 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.907 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.907 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.907 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 10 monclient: shutdown 2023-10-30T12:06:55.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:55.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 1 -- 172.21.15.50:0/1178087252 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c09b170 msgr2=0x7f382c09b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.880+0000 7f3832875640 1 --2- 172.21.15.50:0/1178087252 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c09b170 0x7f382c09b600 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f3824009d90 tx=0x7f382402bea0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 -- 172.21.15.50:0/1178087252 shutdown_connections 2023-10-30T12:06:55.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 --2- 172.21.15.50:0/1178087252 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c09b170 0x7f382c09b600 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 --2- 172.21.15.50:0/1178087252 >> v2:172.21.15.118:3300/0 conn(0x7f382c096540 0x7f382c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 -- 172.21.15.50:0/1178087252 >> 172.21.15.50:0/1178087252 conn(0x7f382c0a3980 msgr2=0x7f382c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 -- 172.21.15.50:0/1178087252 shutdown_connections 2023-10-30T12:06:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 -- 172.21.15.50:0/1178087252 wait complete. 2023-10-30T12:06:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 10 monclient: build_initial_monmap 2023-10-30T12:06:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 10 monclient: monmap: 2023-10-30T12:06:55.910 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:55.910 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:55.910 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:55.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:55.910 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:55.911 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 librados: starting msgr at 2023-10-30T12:06:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 librados: starting objecter 2023-10-30T12:06:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.881+0000 7f3832875640 1 Processor -- start 2023-10-30T12:06:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 -- start start 2023-10-30T12:06:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 librados: setting wanted keys 2023-10-30T12:06:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 librados: calling monclient init 2023-10-30T12:06:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient: init 2023-10-30T12:06:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:06:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c096540 0x7f382c1ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): picked mon.noname-a con 0x7f382c096540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c096e60 0x7f382c1cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): picked mon.noname-c con 0x7f382c096e60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f382c09b170 0x7f382c097590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): picked mon.noname-b con 0x7f382c09b170 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c09a080 con 0x7f382c096540 2023-10-30T12:06:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c090c00 con 0x7f382c09b170 2023-10-30T12:06:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c090ee0 con 0x7f382c096e60 2023-10-30T12:06:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:55.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3832875640 10 monclient(hunting): authenticate will time out at 1261.065631s 2023-10-30T12:06:55.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c096540 0x7f382c1ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c096e60 0x7f382c1cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c096e60 0x7f382c1cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35504/0 (socket says 172.21.15.50:35504) 2023-10-30T12:06:55.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 1 -- 172.21.15.50:0/236460839 learned_addr learned my addr 172.21.15.50:0/236460839 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:55.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 10 monclient(hunting): get_auth_request con 0x7f382c096e60 auth_method 0 2023-10-30T12:06:55.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c096540 0x7f382c1ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53080/0 (socket says 172.21.15.50:53080) 2023-10-30T12:06:55.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 10 monclient(hunting): get_auth_request con 0x7f382c096540 auth_method 0 2023-10-30T12:06:55.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f382c09b170 0x7f382c097590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 10 monclient(hunting): get_auth_request con 0x7f382c09b170 auth_method 0 2023-10-30T12:06:55.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.882+0000 7f3830e12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 10 monclient(hunting): handle_auth_done global_id 44957 payload 824 2023-10-30T12:06:55.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f382c09b170 msgr2=0x7f382c097590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f382c09b170 0x7f382c097590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c096540 msgr2=0x7f382c1ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c096540 0x7f382c1ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:55.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 1 monclient: found mon.noname-c 2023-10-30T12:06:55.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:55.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 1 -- 172.21.15.50:0/236460839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824009a70 con 0x7f382c096e60 2023-10-30T12:06:55.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382bfff640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c096540 0x7f382c1ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:55.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382b7fe640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c096e60 0x7f382c1cad10 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f381c00c4b0 tx=0x7f381c00c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 client.?.objecter ms_handle_connect 0x7f382c096e60 2023-10-30T12:06:55.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:55.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c0074e0 con 0x7f382c096e60 2023-10-30T12:06:55.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:55.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: dump: 2023-10-30T12:06:55.924 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.924 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.925 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.925 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.925 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.925 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.925 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.926 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.926 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.926 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:55.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:55.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:55.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381c007660 con 0x7f382c096e60 2023-10-30T12:06:55.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 5 monclient: authenticate success, global_id 44957 2023-10-30T12:06:55.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:55.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c004a20 con 0x7f382c096e60 2023-10-30T12:06:55.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:55.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3828ff9640 10 monclient: dump: 2023-10-30T12:06:55.928 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.928 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.929 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.929 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.929 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.929 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.929 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.929 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.930 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.930 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 10 monclient: _renew_subs 2023-10-30T12:06:55.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f382a7fc640 4 set_mon_vals no callback set 2023-10-30T12:06:55.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c0911c0 con 0x7f382c096e60 2023-10-30T12:06:55.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 10 client.44957.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:55.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 10 monclient: _renew_subs 2023-10-30T12:06:55.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:55.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c1cb250 con 0x7f382c096e60 2023-10-30T12:06:55.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.883+0000 7f3832875640 1 librados: init done 2023-10-30T12:06:55.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f381c004b80 con 0x7f382c096e60 2023-10-30T12:06:55.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3832875640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:55.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3832875640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c094e70 0x7f382c097ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3832875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c073ff0 con 0x7f382c094e70 2023-10-30T12:06:55.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c094e70 0x7f382c097ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient: get_auth_request con 0x7f382c094e70 auth_method 0 2023-10-30T12:06:55.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f38180392f0 0x7f381803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f381c04a130 con 0x7f382c096e60 2023-10-30T12:06:55.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 10 client.44957.objecter ms_dispatch 0x7f382c000b90 osd_map(203..203 src has 1..203) v4 2023-10-30T12:06:55.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 3 client.44957.objecter handle_osd_map got epochs [203,203] > 0 2023-10-30T12:06:55.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 3 client.44957.objecter handle_osd_map decoding full epoch 203 2023-10-30T12:06:55.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 10 monclient(hunting): handle_auth_done global_id 104118 payload 824 2023-10-30T12:06:55.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 1 -- 172.21.15.50:0/236460839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f382c09d3a0 con 0x7f382c094e70 2023-10-30T12:06:55.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 20 client.44957.objecter dump_active .. 0 homeless 2023-10-30T12:06:55.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3830e12640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f38180392f0 0x7f381803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3830e12640 10 monclient: get_auth_request con 0x7f38180392f0 auth_method 0 2023-10-30T12:06:55.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 10 client.44957.objecter ms_handle_connect 0x7f382c094e70 2023-10-30T12:06:55.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f3828ff9640 10 client.44957.objecter resend_mon_ops 2023-10-30T12:06:55.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.884+0000 7f382bfff640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c094e70 0x7f382c097ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3820011d20 tx=0x7f382000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820004020 con 0x7f382c094e70 2023-10-30T12:06:55.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:55.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 10 monclient: dump: 2023-10-30T12:06:55.938 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.938 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.938 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.938 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.939 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.939 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.939 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.939 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.939 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.939 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f3820004180 con 0x7f382c094e70 2023-10-30T12:06:55.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:55.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:55.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3830e12640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f38180392f0 0x7f381803b7b0 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f3824036000 tx=0x7f3824030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c094e70 msgr2=0x7f382c097ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c094e70 0x7f382c097ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3820011d20 tx=0x7f382000f610 comp rx=0 tx=0).stop 2023-10-30T12:06:55.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c094e70 msgr2=0x7f382c097ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:55.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c094e70 0x7f382c097ad0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.885+0000 7f3828ff9640 10 client.44957.objecter ms_handle_connect 0x7f38180392f0 2023-10-30T12:06:55.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f3832875640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:55.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f3832875640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c098010 0x7f382c098420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:55.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f3832875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:55.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c098960 con 0x7f382c098010 2023-10-30T12:06:55.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c098010 0x7f382c098420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:55.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient: get_auth_request con 0x7f382c098010 auth_method 0 2023-10-30T12:06:55.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:55.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:55.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:55.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:55.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:55.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:55.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 10 monclient(hunting): handle_auth_done global_id 104121 payload 824 2023-10-30T12:06:55.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.891+0000 7f382bfff640 1 -- 172.21.15.50:0/236460839 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f382c04e160 con 0x7f382c098010 2023-10-30T12:06:55.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f382bfff640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c098010 0x7f382c098420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f382001b000 tx=0x7f38200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:55.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 10 client.44957.objecter ms_handle_connect 0x7f382c098010 2023-10-30T12:06:55.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 10 client.44957.objecter resend_mon_ops 2023-10-30T12:06:55.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820014ab0 con 0x7f382c098010 2023-10-30T12:06:55.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:55.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:55.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 10 monclient: dump: 2023-10-30T12:06:55.947 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:55.947 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:55.948 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.948 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:55.948 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:55.948 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:55.948 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:55.949 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:55.949 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:55.949 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:55.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f38200273a0 con 0x7f382c098010 2023-10-30T12:06:55.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:55.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:55.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c098010 msgr2=0x7f382c098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c098010 0x7f382c098420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f382001b000 tx=0x7f38200046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c098010 msgr2=0x7f382c098420 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:55.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3828ff9640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382c098010 0x7f382c098420 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f382001b000 tx=0x7f38200046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.951 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:55.951 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:55.951 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:55.951 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:06:55.951 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 66, 2023-10-30T12:06:55.952 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:55.952 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:55.952 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:55.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:55.952 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:55.953 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:06:55.953 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:55.953 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:55.953 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:55.953 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:55.953 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:55.954 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:55.954 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:55.954 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:55.954 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:55.955 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:55.955 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:55.955 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:55.955 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:55.955 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:55.955 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:55.956 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:55.956 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:55.956 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:55.956 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:55.956 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:55.957 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:55.957 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:55.957 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:55.957 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:55.957 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:55.957 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:55.958 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:55.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.958 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:55.958 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:55.958 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:55.959 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:55.959 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:55.959 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:55.959 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:55.959 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:55.960 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:55.960 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:55.960 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:55.960 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:55.960 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:55.960 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:55.961 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:55.961 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:55.961 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:55.961 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:55.962 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:55.962 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:55.962 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:55.962 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:55.962 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:55.963 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:55.963 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:55.963 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:55.963 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:55.963 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:55.963 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:55.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.964 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:55.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.966 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:55.966 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:55.966 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:55.966 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:55.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.966 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:55.967 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:55.967 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:55.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.968 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:55.968 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:55.968 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:55.968 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:55.969 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:55.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.969 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:55.969 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:55.969 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:55.969 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:55.970 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:55.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.970 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:55.970 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:55.971 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:55.971 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:55.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.971 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:55.971 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:55.972 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:55.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.972 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:55.972 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:55.973 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:55.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:55.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:55.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.973 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:55.982 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:55.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:55.982 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:55.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:55.982 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:55.983 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:55.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 10 librados: watch_flush enter 2023-10-30T12:06:55.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 10 librados: watch_flush exit 2023-10-30T12:06:55.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 20 client.44957.objecter shutdown clearing up homeless session... 2023-10-30T12:06:55.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 10 client.44957.objecter successfully canceled tick 2023-10-30T12:06:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f38180392f0 msgr2=0x7f381803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f38180392f0 0x7f381803b7b0 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f3824036000 tx=0x7f3824030000 comp rx=0 tx=0).stop 2023-10-30T12:06:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 10 monclient: shutdown 2023-10-30T12:06:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c096e60 msgr2=0x7f382c1cad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c096e60 0x7f382c1cad10 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f381c00c4b0 tx=0x7f381c00c970 comp rx=0 tx=0).stop 2023-10-30T12:06:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3830e12640 1 -- 172.21.15.50:0/236460839 reap_dead start 2023-10-30T12:06:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 shutdown_connections 2023-10-30T12:06:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 --2- 172.21.15.50:0/236460839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f382c096e60 0x7f382c1cad10 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 >> 172.21.15.50:0/236460839 conn(0x7f382c0a3980 msgr2=0x7f382c1ba900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.892+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 shutdown_connections 2023-10-30T12:06:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.893+0000 7f3832875640 1 -- 172.21.15.50:0/236460839 wait complete. 2023-10-30T12:06:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:55.893+0000 7f3832875640 1 librados: shutdown 2023-10-30T12:06:55.988 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:55.989 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:06:55.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:06:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.094+0000 7f0fb5875640 10 monclient: get_monmap_and_config 2023-10-30T12:06:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.094+0000 7f0fb5875640 10 monclient: build_initial_monmap 2023-10-30T12:06:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.094+0000 7f0fb5875640 10 monclient: monmap: 2023-10-30T12:06:56.096 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:56.095760+0000 2023-10-30T12:06:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:56.095760+0000 2023-10-30T12:06:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:56.098 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:56.099 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 1 Processor -- start 2023-10-30T12:06:56.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 1 -- start start 2023-10-30T12:06:56.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient: init 2023-10-30T12:06:56.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:56.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T12:06:56.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009bb70 0x7f0fb009bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient(hunting): picked mon.noname-b con 0x7f0fb009bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:56.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fb009d3f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:56.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0fb009c490 0x7f0fb0096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient(hunting): picked mon.noname-a con 0x7f0fb009c490 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:56.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb0098170 con 0x7f0fb009d3f0 2023-10-30T12:06:56.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb009cf60 con 0x7f0fb009c490 2023-10-30T12:06:56.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0097b20 con 0x7f0fb009bb70 2023-10-30T12:06:56.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:56.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.095+0000 7f0fb5875640 10 monclient(hunting): authenticate will time out at 1261.278901s 2023-10-30T12:06:56.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009bb70 0x7f0fb009bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0fb009c490 0x7f0fb0096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009bb70 0x7f0fb009bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35510/0 (socket says 172.21.15.50:35510) 2023-10-30T12:06:56.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 1 -- 172.21.15.50:0/1886041415 learned_addr learned my addr 172.21.15.50:0/1886041415 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:56.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 10 monclient(hunting): get_auth_request con 0x7f0fb009bb70 auth_method 0 2023-10-30T12:06:56.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): get_auth_request con 0x7f0fb009c490 auth_method 0 2023-10-30T12:06:56.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_done global_id 104124 payload 274 2023-10-30T12:06:56.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009bb70 msgr2=0x7f0fb009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:56.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 1 --2- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009bb70 0x7f0fb009bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1886041415 >> v1:172.21.15.118:6789/0 conn(0x7f0fb009d3f0 legacy=0x7f0fb0096080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:56.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:56.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 1 monclient: found mon.noname-a 2023-10-30T12:06:56.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:56.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:56.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1886041415 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0097e00 con 0x7f0fb009c490 2023-10-30T12:06:56.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.096+0000 7f0faeffd640 1 --2- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009bb70 0x7f0fb009bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:56.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0faf7fe640 1 --2- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009c490 0x7f0fb0096790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009d90 tx=0x7f0fa802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bacde7cd5e4469 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 1 -- 172.21.15.50:0/1886041415 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa802cb30 con 0x7f0fb009c490 2023-10-30T12:06:56.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:56.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: dump: 2023-10-30T12:06:56.124 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.124 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.124 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.125 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.125 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.125 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.125 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.125 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.126 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.126 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: _finish_auth 0 2023-10-30T12:06:56.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:06:56.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:56.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 1 -- 172.21.15.50:0/1886041415 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa802cc90 con 0x7f0fb009c490 2023-10-30T12:06:56.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 5 monclient: authenticate success, global_id 104124 2023-10-30T12:06:56.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:56.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:56.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:56.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 1 -- 172.21.15.50:0/1886041415 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8034440 con 0x7f0fb009c490 2023-10-30T12:06:56.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 4 set_mon_vals no callback set 2023-10-30T12:06:56.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:56.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fae7fc640 10 monclient: dump: 2023-10-30T12:06:56.129 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.129 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.129 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.129 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.129 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.130 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.130 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.130 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.130 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.130 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 10 monclient: shutdown 2023-10-30T12:06:56.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:56.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 -- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009c490 msgr2=0x7f0fb0096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 --2- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009c490 0x7f0fb0096790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009d90 tx=0x7f0fa802bd80 comp rx=0 tx=0).stop 2023-10-30T12:06:56.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 -- 172.21.15.50:0/1886041415 shutdown_connections 2023-10-30T12:06:56.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 --2- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009c490 0x7f0fb0096790 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 --2- 172.21.15.50:0/1886041415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009bb70 0x7f0fb009bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 -- 172.21.15.50:0/1886041415 >> 172.21.15.50:0/1886041415 conn(0x7f0fb00a3980 msgr2=0x7f0fb00a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:56.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 -- 172.21.15.50:0/1886041415 shutdown_connections 2023-10-30T12:06:56.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.097+0000 7f0fb5875640 1 -- 172.21.15.50:0/1886041415 wait complete. 2023-10-30T12:06:56.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 10 monclient: build_initial_monmap 2023-10-30T12:06:56.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 10 monclient: monmap: 2023-10-30T12:06:56.133 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:56.133 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:56.133 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:56.133 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:56.134 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:56.134 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.134 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:56.134 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:56.134 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:56.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 1 librados: starting msgr at 2023-10-30T12:06:56.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 1 librados: starting objecter 2023-10-30T12:06:56.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 1 Processor -- start 2023-10-30T12:06:56.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 1 -- start start 2023-10-30T12:06:56.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 1 librados: setting wanted keys 2023-10-30T12:06:56.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 1 librados: calling monclient init 2023-10-30T12:06:56.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.098+0000 7f0fb5875640 10 monclient: init 2023-10-30T12:06:56.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:56.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:06:56.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb009bb70 0x7f0fb006e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): picked mon.noname-b con 0x7f0fb009bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:56.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009c490 0x7f0fb01c5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): picked mon.noname-c con 0x7f0fb009c490 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:56.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009d3f0 0x7f0fb0071220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): picked mon.noname-a con 0x7f0fb009d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:56.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb006d2b0 con 0x7f0fb009d3f0 2023-10-30T12:06:56.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb006da60 con 0x7f0fb009bb70 2023-10-30T12:06:56.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0097120 con 0x7f0fb009c490 2023-10-30T12:06:56.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:56.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fb5875640 10 monclient(hunting): authenticate will time out at 1261.282303s 2023-10-30T12:06:56.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009d3f0 0x7f0fb0071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009c490 0x7f0fb01c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009d3f0 0x7f0fb0071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53126/0 (socket says 172.21.15.50:53126) 2023-10-30T12:06:56.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1594705963 learned_addr learned my addr 172.21.15.50:0/1594705963 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:56.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): get_auth_request con 0x7f0fb009d3f0 auth_method 0 2023-10-30T12:06:56.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 10 monclient(hunting): get_auth_request con 0x7f0fb009c490 auth_method 0 2023-10-30T12:06:56.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faeffd640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb009bb70 0x7f0fb006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0fa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faeffd640 10 monclient(hunting): get_auth_request con 0x7f0fb009bb70 auth_method 0 2023-10-30T12:06:56.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient(hunting): handle_auth_done global_id 104127 payload 824 2023-10-30T12:06:56.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009c490 msgr2=0x7f0fb01c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb009c490 0x7f0fb01c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb009bb70 msgr2=0x7f0fb006e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb009bb70 0x7f0fb006e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 monclient: found mon.noname-a 2023-10-30T12:06:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1594705963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa0009460 con 0x7f0fb009d3f0 2023-10-30T12:06:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faeffd640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fb009bb70 0x7f0fb006e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:06:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0faf7fe640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009d3f0 0x7f0fb0071220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009d90 tx=0x7f0fa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0facff9640 10 client.?.objecter ms_handle_connect 0x7f0fb009d3f0 2023-10-30T12:06:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.099+0000 7f0facff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8040450 con 0x7f0fb009d3f0 2023-10-30T12:06:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: dump: 2023-10-30T12:06:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.149 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa80405d0 con 0x7f0fb009d3f0 2023-10-30T12:06:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 5 monclient: authenticate success, global_id 104127 2023-10-30T12:06:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 10 monclient: _renew_subs 2023-10-30T12:06:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:56.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa8009a70 con 0x7f0fb009d3f0 2023-10-30T12:06:56.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:56.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8040980 con 0x7f0fb009d3f0 2023-10-30T12:06:56.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 10 monclient: dump: 2023-10-30T12:06:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.154 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 10 client.104127.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 10 monclient: _renew_subs 2023-10-30T12:06:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fadffb640 4 set_mon_vals no callback set 2023-10-30T12:06:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb00976e0 con 0x7f0fb009d3f0 2023-10-30T12:06:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 1 librados: init done 2023-10-30T12:06:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0fa8040ae0 con 0x7f0fb009d3f0 2023-10-30T12:06:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb0071760 0x7f0fb01c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.100+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0073ff0 con 0x7f0fb0071760 2023-10-30T12:06:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb0071760 0x7f0fb01c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient: get_auth_request con 0x7f0fb0071760 auth_method 0 2023-10-30T12:06:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0f900392f0 0x7f0f9003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f0fa806dc00 con 0x7f0fb009d3f0 2023-10-30T12:06:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 client.104127.objecter ms_dispatch 0x7f0fb0000b90 osd_map(203..203 src has 1..203) v4 2023-10-30T12:06:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 3 client.104127.objecter handle_osd_map got epochs [203,203] > 0 2023-10-30T12:06:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 3 client.104127.objecter handle_osd_map decoding full epoch 203 2023-10-30T12:06:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 20 client.104127.objecter dump_active .. 0 homeless 2023-10-30T12:06:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 10 monclient(hunting): handle_auth_done global_id 44966 payload 824 2023-10-30T12:06:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 1 -- 172.21.15.50:0/1594705963 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb0097560 con 0x7f0fb0071760 2023-10-30T12:06:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0fa7fff640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0f900392f0 0x7f0f9003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0fa7fff640 10 monclient: get_auth_request con 0x7f0f900392f0 auth_method 0 2023-10-30T12:06:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 client.104127.objecter ms_handle_connect 0x7f0fb0071760 2023-10-30T12:06:56.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 client.104127.objecter resend_mon_ops 2023-10-30T12:06:56.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0faeffd640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb0071760 0x7f0fb01c8810 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0011830 tx=0x7f0fa0011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0004220 con 0x7f0fb0071760 2023-10-30T12:06:56.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:56.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 monclient: dump: 2023-10-30T12:06:56.161 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.161 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.161 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.161 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.161 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.162 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 client.104127.objecter ms_handle_connect 0x7f0f900392f0 2023-10-30T12:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0fa7fff640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0f900392f0 0x7f0f9003b7b0 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7f0fb006d560 tx=0x7f0f98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0fa0004380 con 0x7f0fb0071760 2023-10-30T12:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb0071760 msgr2=0x7f0fb01c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.101+0000 7f0facff9640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb0071760 0x7f0fb01c8810 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0011830 tx=0x7f0fa0011cf0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.102+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb0071760 msgr2=0x7f0fb01c8810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.102+0000 7f0facff9640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb0071760 0x7f0fb01c8810 secure :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0011830 tx=0x7f0fa0011cf0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0fb5875640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0fb5875640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb006f510 0x7f0fb002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0fb5875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb009c900 con 0x7f0fb006f510 2023-10-30T12:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb006f510 0x7f0fb002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient: get_auth_request con 0x7f0fb006f510 auth_method 0 2023-10-30T12:06:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 10 monclient(hunting): handle_auth_done global_id 44969 payload 824 2023-10-30T12:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 1 -- 172.21.15.50:0/1594705963 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0fb006fad0 con 0x7f0fb006f510 2023-10-30T12:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0faeffd640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb006f510 0x7f0fb002b430 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0000c00 tx=0x7f0fa00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0facff9640 10 client.104127.objecter ms_handle_connect 0x7f0fb006f510 2023-10-30T12:06:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.108+0000 7f0facff9640 10 client.104127.objecter resend_mon_ops 2023-10-30T12:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0015660 con 0x7f0fb006f510 2023-10-30T12:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 10 monclient: dump: 2023-10-30T12:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.170 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f0fa00157c0 con 0x7f0fb006f510 2023-10-30T12:06:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb006f510 msgr2=0x7f0fb002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb006f510 0x7f0fb002b430 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0000c00 tx=0x7f0fa00046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb006f510 msgr2=0x7f0fb002b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0facff9640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0fb006f510 0x7f0fb002b430 secure :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0000c00 tx=0x7f0fa00046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.172 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:56.172 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:56.172 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:56.172 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:56.173 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 66, 2023-10-30T12:06:56.173 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:56.173 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:56.173 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:56.173 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:56.173 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.174 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:06:56.174 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:56.174 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:56.174 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:56.174 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:56.175 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:56.175 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:56.175 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:56.175 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:56.175 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:56.176 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:56.176 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:56.176 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:56.176 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:56.176 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.177 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:56.177 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:56.177 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:56.177 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:56.177 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:56.177 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:56.178 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:56.178 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:56.178 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:56.178 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:56.178 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:56.179 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:56.179 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.179 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:56.179 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:56.180 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:56.180 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:56.180 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:56.180 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:56.180 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:56.180 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:56.181 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:56.181 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:56.181 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:56.181 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:56.181 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:56.181 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:56.182 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:56.182 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:56.182 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:56.182 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:56.182 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:56.183 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:56.183 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:56.183 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:56.183 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:56.183 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:56.183 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:56.184 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:56.184 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:56.184 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.184 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:56.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.184 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:56.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.185 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:56.185 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:56.185 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:56.185 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:56.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.186 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:56.186 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:56.186 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.187 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:56.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:56.187 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.188 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:56.188 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:56.188 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:56.188 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:56.188 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:56.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.189 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:56.189 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:56.190 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:56.190 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:56.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.190 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:56.190 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:56.191 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:56.191 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:56.192 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.192 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.192 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.192 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:56.192 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:56.193 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:56.193 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:56.193 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:56.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.194 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:56.194 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:56.194 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:56.194 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:56.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.194 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:56.195 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:56.195 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.195 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:56.196 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:56.196 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.196 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.196 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.197 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:56.197 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:56.197 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:56.197 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:56.197 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:56.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.198 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.198 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:56.198 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:56.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.199 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:56.199 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:56.199 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:56.199 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.199 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.200 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:06:56.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.200 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:56.200 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:56.200 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:06:56.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.201 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:56.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.201 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:56.201 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:56.201 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:06:56.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.202 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.202 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:56.202 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 10 librados: watch_flush enter 2023-10-30T12:06:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 10 librados: watch_flush exit 2023-10-30T12:06:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 20 client.104127.objecter shutdown clearing up homeless session... 2023-10-30T12:06:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 10 client.104127.objecter successfully canceled tick 2023-10-30T12:06:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0f900392f0 msgr2=0x7f0f9003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0f900392f0 0x7f0f9003b7b0 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7f0fb006d560 tx=0x7f0f98009040 comp rx=0 tx=0).stop 2023-10-30T12:06:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 10 monclient: shutdown 2023-10-30T12:06:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009d3f0 msgr2=0x7f0fb0071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 --2- 172.21.15.50:0/1594705963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0fb009d3f0 0x7f0fb0071220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009d90 tx=0x7f0fa8004a40 comp rx=0 tx=0).stop 2023-10-30T12:06:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1594705963 reap_dead start 2023-10-30T12:06:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0faf7fe640 1 -- 172.21.15.50:0/1594705963 reap_dead start 2023-10-30T12:06:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 shutdown_connections 2023-10-30T12:06:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 >> 172.21.15.50:0/1594705963 conn(0x7f0fb00a3980 msgr2=0x7f0fb013aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 shutdown_connections 2023-10-30T12:06:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 -- 172.21.15.50:0/1594705963 wait complete. 2023-10-30T12:06:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.109+0000 7f0fb5875640 1 librados: shutdown 2023-10-30T12:06:56.208 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:56.208 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:06:56.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:06:56.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.314+0000 7fa15f1a0640 10 monclient: get_monmap_and_config 2023-10-30T12:06:56.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.314+0000 7fa15f1a0640 10 monclient: build_initial_monmap 2023-10-30T12:06:56.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.314+0000 7fa15f1a0640 10 monclient: monmap: 2023-10-30T12:06:56.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:56.317 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:56.315844+0000 2023-10-30T12:06:56.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:56.315844+0000 2023-10-30T12:06:56.317 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:56.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:56.319 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.315+0000 7fa15f1a0640 1 Processor -- start 2023-10-30T12:06:56.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.315+0000 7fa15f1a0640 1 -- start start 2023-10-30T12:06:56.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.315+0000 7fa15f1a0640 10 monclient: init 2023-10-30T12:06:56.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:56.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:06:56.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158132070 0x7fa158132450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa158132070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:56.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa158132990 0x7fa158135570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient(hunting): picked mon.noname-c con 0x7fa158132990 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:56.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa158135ab0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:56.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1580a4780 con 0x7fa158135ab0 2023-10-30T12:06:56.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580a1650 con 0x7fa158132990 2023-10-30T12:06:56.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa158099c20 con 0x7fa158132070 2023-10-30T12:06:56.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:56.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15f1a0640 10 monclient(hunting): authenticate will time out at 1261.499220s 2023-10-30T12:06:56.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158132070 0x7fa158132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158132070 0x7fa158132450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35544/0 (socket says 172.21.15.50:35544) 2023-10-30T12:06:56.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 -- 172.21.15.50:0/339125638 learned_addr learned my addr 172.21.15.50:0/339125638 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:56.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): get_auth_request con 0x7fa158132070 auth_method 0 2023-10-30T12:06:56.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15d99d640 1 --2- 172.21.15.50:0/339125638 >> v2:172.21.15.118:3300/0 conn(0x7fa158132990 0x7fa158135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15d99d640 10 monclient(hunting): get_auth_request con 0x7fa158132990 auth_method 0 2023-10-30T12:06:56.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15d99d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15d99d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15d99d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient(hunting): handle_auth_done global_id 44972 payload 274 2023-10-30T12:06:56.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 -- 172.21.15.50:0/339125638 >> v2:172.21.15.118:3300/0 conn(0x7fa158132990 msgr2=0x7fa158135570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:56.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 --2- 172.21.15.50:0/339125638 >> v2:172.21.15.118:3300/0 conn(0x7fa158132990 0x7fa158135570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 -- 172.21.15.50:0/339125638 >> v1:172.21.15.118:6789/0 conn(0x7fa158135ab0 legacy=0x7fa158135f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:56.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient: _finish_hunting 0 2023-10-30T12:06:56.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 monclient: found mon.noname-b 2023-10-30T12:06:56.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:56.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.316+0000 7fa15e19e640 1 -- 172.21.15.50:0/339125638 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15809a040 con 0x7fa158132070 2023-10-30T12:06:56.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15e19e640 1 --2- 172.21.15.50:0/339125638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158132070 0x7fa158132450 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fa150009e50 tx=0x7fa15002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b8af5f41240821 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 1 -- 172.21.15.50:0/339125638 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15002c9e0 con 0x7fa158132070 2023-10-30T12:06:56.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:56.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: dump: 2023-10-30T12:06:56.336 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.336 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.337 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.337 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.337 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.337 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.337 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.338 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.338 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.338 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: _finish_auth 0 2023-10-30T12:06:56.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: _check_auth_tickets 2023-10-30T12:06:56.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:56.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 1 -- 172.21.15.50:0/339125638 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15002cb40 con 0x7fa158132070 2023-10-30T12:06:56.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:56.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15f1a0640 5 monclient: authenticate success, global_id 44972 2023-10-30T12:06:56.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15f1a0640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:56.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 1 -- 172.21.15.50:0/339125638 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1500343f0 con 0x7fa158132070 2023-10-30T12:06:56.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15f1a0640 4 set_mon_vals no callback set 2023-10-30T12:06:56.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:56.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.317+0000 7fa15d19c640 10 monclient: dump: 2023-10-30T12:06:56.340 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.341 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.341 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.341 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.341 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.341 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.342 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.342 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.342 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.342 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 10 monclient: shutdown 2023-10-30T12:06:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 -- 172.21.15.50:0/339125638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158132070 msgr2=0x7fa158132450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/339125638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158132070 0x7fa158132450 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fa150009e50 tx=0x7fa15002be40 comp rx=0 tx=0).stop 2023-10-30T12:06:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 -- 172.21.15.50:0/339125638 shutdown_connections 2023-10-30T12:06:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/339125638 >> v2:172.21.15.118:3300/0 conn(0x7fa158132990 0x7fa158135570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/339125638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158132070 0x7fa158132450 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 -- 172.21.15.50:0/339125638 >> 172.21.15.50:0/339125638 conn(0x7fa1580a7f80 msgr2=0x7fa158069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 -- 172.21.15.50:0/339125638 shutdown_connections 2023-10-30T12:06:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 -- 172.21.15.50:0/339125638 wait complete. 2023-10-30T12:06:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 10 monclient: build_initial_monmap 2023-10-30T12:06:56.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 10 monclient: monmap: 2023-10-30T12:06:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:56.345 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:56.346 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 librados: starting msgr at 2023-10-30T12:06:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.318+0000 7fa15f1a0640 1 librados: starting objecter 2023-10-30T12:06:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 Processor -- start 2023-10-30T12:06:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 -- start start 2023-10-30T12:06:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 librados: setting wanted keys 2023-10-30T12:06:56.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 librados: calling monclient init 2023-10-30T12:06:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient: init 2023-10-30T12:06:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:06:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158132070 0x7fa158131720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa158132070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa158132990 0x7fa1581c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): picked mon.noname-a con 0x7fa158132990 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158135ab0 0x7fa1581c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): picked mon.noname-c con 0x7fa158135ab0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa158122210 con 0x7fa158132990 2023-10-30T12:06:56.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15811f5f0 con 0x7fa158132070 2023-10-30T12:06:56.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580a1840 con 0x7fa158135ab0 2023-10-30T12:06:56.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:56.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15f1a0640 10 monclient(hunting): authenticate will time out at 1261.502601s 2023-10-30T12:06:56.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158135ab0 0x7fa1581c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa158132990 0x7fa1581c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158135ab0 0x7fa1581c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35556/0 (socket says 172.21.15.50:35556) 2023-10-30T12:06:56.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 1 -- 172.21.15.50:0/402781159 learned_addr learned my addr 172.21.15.50:0/402781159 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 10 monclient(hunting): get_auth_request con 0x7fa158135ab0 auth_method 0 2023-10-30T12:06:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 10 monclient(hunting): get_auth_request con 0x7fa158132990 auth_method 0 2023-10-30T12:06:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e99f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15d99d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.319+0000 7fa15e19e640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158132070 0x7fa158131720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 10 monclient(hunting): handle_auth_done global_id 44975 payload 824 2023-10-30T12:06:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158132070 msgr2=0x7fa158131720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158132070 0x7fa158131720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa158132990 msgr2=0x7fa1581c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa158132990 0x7fa1581c6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 10 monclient: _finish_hunting 0 2023-10-30T12:06:56.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 1 monclient: found mon.noname-c 2023-10-30T12:06:56.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:56.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 1 -- 172.21.15.50:0/402781159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150009b30 con 0x7fa158135ab0 2023-10-30T12:06:56.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e19e640 10 monclient: get_auth_request con 0x7fa158132070 auth_method 0 2023-10-30T12:06:56.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e19e640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158132070 0x7fa158131720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:06:56.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15d99d640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa158132990 0x7fa1581c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:56.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15e99f640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158135ab0 0x7fa1581c6cb0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fa15400c440 tx=0x7fa15400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 client.?.objecter ms_handle_connect 0x7fa158135ab0 2023-10-30T12:06:56.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:56.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154011790 con 0x7fa158135ab0 2023-10-30T12:06:56.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:56.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: dump: 2023-10-30T12:06:56.358 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.358 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.359 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.359 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.359 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.359 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.359 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.360 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.360 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.360 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: _finish_auth 0 2023-10-30T12:06:56.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1540118f0 con 0x7fa158135ab0 2023-10-30T12:06:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 5 monclient: authenticate success, global_id 44975 2023-10-30T12:06:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 10 monclient: _renew_subs 2023-10-30T12:06:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa158121b50 con 0x7fa158135ab0 2023-10-30T12:06:56.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:56.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154011ca0 con 0x7fa158135ab0 2023-10-30T12:06:56.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:56.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15c99b640 4 set_mon_vals no callback set 2023-10-30T12:06:56.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa14effd640 10 monclient: dump: 2023-10-30T12:06:56.363 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.363 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.363 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.365 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 10 client.44975.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 10 monclient: _renew_subs 2023-10-30T12:06:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15811fd40 con 0x7fa158135ab0 2023-10-30T12:06:56.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.320+0000 7fa15f1a0640 1 librados: init done 2023-10-30T12:06:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15f1a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158131c60 0x7fa1581c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15f1a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa158073de0 con 0x7fa158131c60 2023-10-30T12:06:56.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa15401c460 con 0x7fa158135ab0 2023-10-30T12:06:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa14effd640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa13c0392f0 0x7fa13c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6913+0+0 (secure 0 0 0) 0x7fa15404acd0 con 0x7fa158135ab0 2023-10-30T12:06:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa14effd640 10 client.44975.objecter ms_dispatch 0x7fa158000b90 osd_map(203..203 src has 1..203) v4 2023-10-30T12:06:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa14effd640 3 client.44975.objecter handle_osd_map got epochs [203,203] > 0 2023-10-30T12:06:56.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa14effd640 3 client.44975.objecter handle_osd_map decoding full epoch 203 2023-10-30T12:06:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158131c60 0x7fa1581c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient: get_auth_request con 0x7fa158131c60 auth_method 0 2023-10-30T12:06:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa14effd640 20 client.44975.objecter dump_active .. 0 homeless 2023-10-30T12:06:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15d99d640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa13c0392f0 0x7fa13c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15d99d640 10 monclient: get_auth_request con 0x7fa13c0392f0 auth_method 0 2023-10-30T12:06:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.321+0000 7fa15e19e640 10 monclient(hunting): handle_auth_done global_id 94285 payload 824 2023-10-30T12:06:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa15e19e640 1 -- 172.21.15.50:0/402781159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa158132dd0 con 0x7fa158131c60 2023-10-30T12:06:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 client.44975.objecter ms_handle_connect 0x7fa13c0392f0 2023-10-30T12:06:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa15d99d640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa13c0392f0 0x7fa13c03b7b0 secure :-1 s=READY pgs=3999 cs=0 l=1 rev1=1 crypto rx=0x7fa15811f940 tx=0x7fa148008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 client.44975.objecter ms_handle_connect 0x7fa158131c60 2023-10-30T12:06:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 client.44975.objecter resend_mon_ops 2023-10-30T12:06:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa15e19e640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158131c60 0x7fa1581c9c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa1580a44f0 tx=0x7fa150004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150040670 con 0x7fa158131c60 2023-10-30T12:06:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 monclient: dump: 2023-10-30T12:06:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.372 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.373 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa1500407d0 con 0x7fa158131c60 2023-10-30T12:06:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158131c60 msgr2=0x7fa1581c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158131c60 0x7fa1581c9c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa1580a44f0 tx=0x7fa150004a40 comp rx=0 tx=0).stop 2023-10-30T12:06:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158131c60 msgr2=0x7fa1581c9c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:56.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.322+0000 7fa14effd640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa158131c60 0x7fa1581c9c90 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa1580a44f0 tx=0x7fa150004a40 comp rx=0 tx=0).stop 2023-10-30T12:06:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.328+0000 7fa15f1a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.328+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1581ca1d0 0x7fa1581ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.328+0000 7fa15f1a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.328+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15809a600 con 0x7fa1581ca1d0 2023-10-30T12:06:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1581ca1d0 0x7fa1581ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient: get_auth_request con 0x7fa1581ca1d0 auth_method 0 2023-10-30T12:06:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 10 monclient(hunting): handle_auth_done global_id 94288 payload 824 2023-10-30T12:06:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 1 -- 172.21.15.50:0/402781159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa1580a4c10 con 0x7fa1581ca1d0 2023-10-30T12:06:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa15e19e640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1581ca1d0 0x7fa1581ca5e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa150000f80 tx=0x7fa150009000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa14effd640 10 client.44975.objecter ms_handle_connect 0x7fa1581ca1d0 2023-10-30T12:06:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa14effd640 10 client.44975.objecter resend_mon_ops 2023-10-30T12:06:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150053640 con 0x7fa1581ca1d0 2023-10-30T12:06:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa14effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa14effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.329+0000 7fa14effd640 10 monclient: dump: 2023-10-30T12:06:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.381 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.382 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.383 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fa1500537a0 con 0x7fa1581ca1d0 2023-10-30T12:06:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa14effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa14effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1581ca1d0 msgr2=0x7fa1581ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa14effd640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1581ca1d0 0x7fa1581ca5e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa150000f80 tx=0x7fa150009000 comp rx=0 tx=0).stop 2023-10-30T12:06:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa14effd640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1581ca1d0 msgr2=0x7fa1581ca5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:56.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa14effd640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1581ca1d0 0x7fa1581ca5e0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa150000f80 tx=0x7fa150009000 comp rx=0 tx=0).stop 2023-10-30T12:06:56.384 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:56.385 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:56.385 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:56.385 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:56.385 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 66, 2023-10-30T12:06:56.385 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:56.386 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:56.386 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:56.386 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:56.386 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.386 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:06:56.387 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:56.387 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:56.387 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:56.387 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:56.387 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:56.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:56.388 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:56.388 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:56.388 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:56.388 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:56.388 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:56.389 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:56.389 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:56.389 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.389 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:56.389 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:56.390 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:56.390 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:56.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:56.390 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:56.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:56.391 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:56.391 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:56.391 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:56.391 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:56.391 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:56.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.392 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:56.392 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:56.392 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:56.392 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:56.393 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:56.393 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:56.393 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:56.393 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:56.393 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:56.394 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:56.394 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:56.394 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:56.394 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:56.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:56.395 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:56.395 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:56.395 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:56.395 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:56.395 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:56.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:56.396 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:56.396 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:56.396 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:56.396 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:56.396 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:56.397 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:56.397 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:56.397 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.397 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:56.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:56.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.398 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:56.398 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:56.398 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:56.399 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:56.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.399 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:56.399 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:56.399 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:56.402 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:56.402 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:56.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.402 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:56.403 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:56.403 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:56.403 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:56.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.403 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:56.405 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:56.405 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:56.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:56.406 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.406 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:56.407 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:56.407 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:56.407 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:56.408 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:56.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.408 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:56.408 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:56.409 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:56.409 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:56.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.409 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:56.409 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:56.410 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.410 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:56.410 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:56.411 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:56.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.411 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:56.411 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:56.412 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:56.412 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:56.412 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:56.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.412 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.413 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:56.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:56.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.413 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:56.414 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:56.414 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:56.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.414 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.414 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:56.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.415 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:56.415 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:56.415 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:06:56.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.415 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:56.416 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:06:56.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:56.416 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:56.416 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:56.416 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:56.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:56.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:56.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:56.417 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:56.417 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 10 librados: watch_flush enter 2023-10-30T12:06:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 10 librados: watch_flush exit 2023-10-30T12:06:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 20 client.44975.objecter shutdown clearing up homeless session... 2023-10-30T12:06:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 10 client.44975.objecter successfully canceled tick 2023-10-30T12:06:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa13c0392f0 msgr2=0x7fa13c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa13c0392f0 0x7fa13c03b7b0 secure :-1 s=READY pgs=3999 cs=0 l=1 rev1=1 crypto rx=0x7fa15811f940 tx=0x7fa148008040 comp rx=0 tx=0).stop 2023-10-30T12:06:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 10 monclient: shutdown 2023-10-30T12:06:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158135ab0 msgr2=0x7fa1581c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158135ab0 0x7fa1581c6cb0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fa15400c440 tx=0x7fa15400c900 comp rx=0 tx=0).stop 2023-10-30T12:06:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15e99f640 1 -- 172.21.15.50:0/402781159 reap_dead start 2023-10-30T12:06:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 shutdown_connections 2023-10-30T12:06:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 --2- 172.21.15.50:0/402781159 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa158135ab0 0x7fa1581c6cb0 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 >> 172.21.15.50:0/402781159 conn(0x7fa1580a7f80 msgr2=0x7fa1580a68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 shutdown_connections 2023-10-30T12:06:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 -- 172.21.15.50:0/402781159 wait complete. 2023-10-30T12:06:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.330+0000 7fa15f1a0640 1 librados: shutdown 2023-10-30T12:06:56.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:56.423 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:06:56.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:06:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.527+0000 7f0d128e1640 10 monclient: get_monmap_and_config 2023-10-30T12:06:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.527+0000 7f0d128e1640 10 monclient: build_initial_monmap 2023-10-30T12:06:56.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.527+0000 7f0d128e1640 10 monclient: monmap: 2023-10-30T12:06:56.529 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:56.529 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.529 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:56.528180+0000 2023-10-30T12:06:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:56.528180+0000 2023-10-30T12:06:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:56.531 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.527+0000 7f0d128e1640 1 Processor -- start 2023-10-30T12:06:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.527+0000 7f0d128e1640 1 -- start start 2023-10-30T12:06:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.527+0000 7f0d128e1640 10 monclient: init 2023-10-30T12:06:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:06:56.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0d0c0a4b60 0x7f0d0c1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c0a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d0c099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:56.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c0a4f40 con 0x7f0d0c099a40 2023-10-30T12:06:56.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c0a3d70 con 0x7f0d0c0a4b60 2023-10-30T12:06:56.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0a4150 con 0x7f0d0c132be0 2023-10-30T12:06:56.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:56.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d128e1640 10 monclient(hunting): authenticate will time out at 1261.711333s 2023-10-30T12:06:56.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c0a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35560/0 (socket says 172.21.15.50:35560) 2023-10-30T12:06:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 -- 172.21.15.50:0/2230763423 learned_addr learned my addr 172.21.15.50:0/2230763423 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): get_auth_request con 0x7f0d0c132be0 auth_method 0 2023-10-30T12:06:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 1 --2- 172.21.15.50:0/2230763423 >> v2:172.21.15.118:3300/0 conn(0x7f0d0c0a4b60 0x7f0d0c1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 10 monclient(hunting): get_auth_request con 0x7f0d0c0a4b60 auth_method 0 2023-10-30T12:06:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient(hunting): handle_auth_done global_id 44978 payload 274 2023-10-30T12:06:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 -- 172.21.15.50:0/2230763423 >> v2:172.21.15.118:3300/0 conn(0x7f0d0c0a4b60 msgr2=0x7f0d0c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:56.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 --2- 172.21.15.50:0/2230763423 >> v2:172.21.15.118:3300/0 conn(0x7f0d0c0a4b60 0x7f0d0c1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 -- 172.21.15.50:0/2230763423 >> v1:172.21.15.118:6789/0 conn(0x7f0d0c099a40 legacy=0x7f0d0c0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient: _finish_hunting 0 2023-10-30T12:06:56.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 monclient: found mon.noname-b 2023-10-30T12:06:56.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:56.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.528+0000 7f0d10e7e640 1 -- 172.21.15.50:0/2230763423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c09a440 con 0x7f0d0c132be0 2023-10-30T12:06:56.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d10e7e640 1 --2- 172.21.15.50:0/2230763423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c0a2a60 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009e50 tx=0x7f0d0402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2c76d33fa8a95ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 1 -- 172.21.15.50:0/2230763423 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0402c9e0 con 0x7f0d0c132be0 2023-10-30T12:06:56.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:56.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: dump: 2023-10-30T12:06:56.551 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.551 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.553 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: _finish_auth 0 2023-10-30T12:06:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 1 -- 172.21.15.50:0/2230763423 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0402cb40 con 0x7f0d0c132be0 2023-10-30T12:06:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d128e1640 5 monclient: authenticate success, global_id 44978 2023-10-30T12:06:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d128e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d128e1640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 1 -- 172.21.15.50:0/2230763423 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d040343f0 con 0x7f0d0c132be0 2023-10-30T12:06:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d128e1640 4 set_mon_vals no callback set 2023-10-30T12:06:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.529+0000 7f0d0affd640 10 monclient: dump: 2023-10-30T12:06:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.558 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 10 monclient: shutdown 2023-10-30T12:06:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 -- 172.21.15.50:0/2230763423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 msgr2=0x7f0d0c0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 --2- 172.21.15.50:0/2230763423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c0a2a60 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009e50 tx=0x7f0d0402be40 comp rx=0 tx=0).stop 2023-10-30T12:06:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 -- 172.21.15.50:0/2230763423 shutdown_connections 2023-10-30T12:06:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 --2- 172.21.15.50:0/2230763423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c0a2a60 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 --2- 172.21.15.50:0/2230763423 >> v2:172.21.15.118:3300/0 conn(0x7f0d0c0a4b60 0x7f0d0c1326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 -- 172.21.15.50:0/2230763423 >> 172.21.15.50:0/2230763423 conn(0x7f0d0c0a8060 msgr2=0x7f0d0c0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 -- 172.21.15.50:0/2230763423 shutdown_connections 2023-10-30T12:06:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 1 -- 172.21.15.50:0/2230763423 wait complete. 2023-10-30T12:06:56.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 10 monclient: build_initial_monmap 2023-10-30T12:06:56.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.530+0000 7f0d128e1640 10 monclient: monmap: 2023-10-30T12:06:56.561 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:56.561 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:56.561 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:56.563 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 librados: starting msgr at 2023-10-30T12:06:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 librados: starting objecter 2023-10-30T12:06:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 Processor -- start 2023-10-30T12:06:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 -- start start 2023-10-30T12:06:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 librados: setting wanted keys 2023-10-30T12:06:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 librados: calling monclient init 2023-10-30T12:06:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient: init 2023-10-30T12:06:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:06:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c0a4b60 0x7f0d0c11e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c0a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c11ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d0c11f170 0x7f0d0c1d0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0c11f170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c122830 con 0x7f0d0c11f170 2023-10-30T12:06:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c120cf0 con 0x7f0d0c0a4b60 2023-10-30T12:06:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0a18d0 con 0x7f0d0c132be0 2023-10-30T12:06:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d128e1640 10 monclient(hunting): authenticate will time out at 1261.714858s 2023-10-30T12:06:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c11ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c11ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35574/0 (socket says 172.21.15.50:35574) 2023-10-30T12:06:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d10e7e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d0c11f170 0x7f0d0c1d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d0b7fe640 1 -- 172.21.15.50:0/855419372 learned_addr learned my addr 172.21.15.50:0/855419372 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d0b7fe640 10 monclient(hunting): get_auth_request con 0x7f0d0c132be0 auth_method 0 2023-10-30T12:06:56.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d0b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.531+0000 7f0d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 10 monclient(hunting): get_auth_request con 0x7f0d0c11f170 auth_method 0 2023-10-30T12:06:56.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c0a4b60 0x7f0d0c11e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 10 monclient(hunting): get_auth_request con 0x7f0d0c0a4b60 auth_method 0 2023-10-30T12:06:56.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:56.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:56.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:56.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:56.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:56.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:56.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 10 monclient(hunting): handle_auth_done global_id 44981 payload 824 2023-10-30T12:06:56.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 1 -- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c0a4b60 msgr2=0x7f0d0c11e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c0a4b60 0x7f0d0c11e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 1 -- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d0c11f170 msgr2=0x7f0d0c1d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d0c11f170 0x7f0d0c1d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:56.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 1 monclient: found mon.noname-c 2023-10-30T12:06:56.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 1 -- 172.21.15.50:0/855419372 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d04009b30 con 0x7f0d0c132be0 2023-10-30T12:06:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d10e7e640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d0c11f170 0x7f0d0c1d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0b7fe640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c11ec30 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00c3f0 tx=0x7f0cfc00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 client.?.objecter ms_handle_connect 0x7f0d0c132be0 2023-10-30T12:06:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 1 -- 172.21.15.50:0/855419372 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc0052d0 con 0x7f0d0c132be0 2023-10-30T12:06:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: dump: 2023-10-30T12:06:56.578 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.578 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.578 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.578 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.579 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.579 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.579 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.579 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.579 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.580 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 5 monclient: authenticate success, global_id 44981 2023-10-30T12:06:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 10 monclient: _renew_subs 2023-10-30T12:06:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 1 -- 172.21.15.50:0/855419372 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc005430 con 0x7f0d0c132be0 2023-10-30T12:06:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c135cf0 con 0x7f0d0c132be0 2023-10-30T12:06:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 1 -- 172.21.15.50:0/855419372 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc003830 con 0x7f0d0c132be0 2023-10-30T12:06:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d08ff9640 10 monclient: dump: 2023-10-30T12:06:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:56.585 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:56.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 10 client.44981.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:56.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d0a7fc640 4 set_mon_vals no callback set 2023-10-30T12:06:56.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 10 monclient: _renew_subs 2023-10-30T12:06:56.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c120060 con 0x7f0d0c132be0 2023-10-30T12:06:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.532+0000 7f0d128e1640 1 librados: init done 2023-10-30T12:06:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d08ff9640 1 -- 172.21.15.50:0/855419372 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0cfc003990 con 0x7f0d0c132be0 2023-10-30T12:06:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d128e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d128e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d128e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c122550 con 0x7f0d0c132be0 2023-10-30T12:06:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d08ff9640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0cf80392f0 0x7f0cf803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d08ff9640 1 -- 172.21.15.50:0/855419372 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 7176+0+0 (secure 0 0 0) 0x7f0cfc04a490 con 0x7f0d0c132be0 2023-10-30T12:06:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d08ff9640 10 client.44981.objecter ms_dispatch 0x7f0d0c000b90 osd_map(204..204 src has 1..204) v4 2023-10-30T12:06:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d08ff9640 3 client.44981.objecter handle_osd_map got epochs [204,204] > 0 2023-10-30T12:06:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d08ff9640 3 client.44981.objecter handle_osd_map decoding full epoch 204 2023-10-30T12:06:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d08ff9640 20 client.44981.objecter dump_active .. 0 homeless 2023-10-30T12:06:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.533+0000 7f0d0bfff640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0cf80392f0 0x7f0cf803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.534+0000 7f0d0bfff640 10 monclient: get_auth_request con 0x7f0cf80392f0 auth_method 0 2023-10-30T12:06:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.534+0000 7f0d08ff9640 10 client.44981.objecter ms_handle_connect 0x7f0cf80392f0 2023-10-30T12:06:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.534+0000 7f0d0bfff640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0cf80392f0 0x7f0cf803b7b0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009b40 tx=0x7f0d00009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.538+0000 7f0d08ff9640 1 -- 172.21.15.50:0/855419372 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0d0c122550 con 0x7f0d0c132be0 2023-10-30T12:06:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.538+0000 7f0d08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:56.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.538+0000 7f0d08ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:56.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.786+0000 7f0d128e1640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:06:56.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.786+0000 7f0d128e1640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:06:56.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.786+0000 7f0d128e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:56.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.786+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0d0c122550 con 0x7f0d0c132be0 2023-10-30T12:06:56.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.787+0000 7f0d08ff9640 1 -- 172.21.15.50:0/855419372 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0d0c122550 con 0x7f0d0c132be0 2023-10-30T12:06:56.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.787+0000 7f0d08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:06:56.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.787+0000 7f0d08ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:56.790 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.788+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.790 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.788+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=8,logm=61} crc {auth=1372879262,config=3140514009,health=4268702962,kv=474600764,logm=4004509937}) 2023-10-30T12:06:56.790 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.788+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=4054490097,config=3140514009,health=4268702962,kv=2905431493,logm=2649516748}) 2023-10-30T12:06:56.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.788+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.788+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=8,logm=61} crc {auth=1372879262,config=3140514009,health=4268702962,kv=474600764,logm=4004509937}) 2023-10-30T12:06:56.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.788+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=2121187763,config=3140514009,health=1424623705,kv=474600764,logm=1589366972}) 2023-10-30T12:06:56.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 10 librados: watch_flush enter 2023-10-30T12:06:56.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 10 librados: watch_flush exit 2023-10-30T12:06:56.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 20 client.44981.objecter shutdown clearing up homeless session... 2023-10-30T12:06:56.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 10 client.44981.objecter successfully canceled tick 2023-10-30T12:06:56.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0cf80392f0 msgr2=0x7f0cf803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0cf80392f0 0x7f0cf803b7b0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009b40 tx=0x7f0d00009190 comp rx=0 tx=0).stop 2023-10-30T12:06:56.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.789+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 10 monclient: shutdown 2023-10-30T12:06:56.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:56.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 msgr2=0x7f0d0c11ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:56.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c11ec30 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00c3f0 tx=0x7f0cfc00c8b0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.789+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750101475}) 2023-10-30T12:06:56.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.789+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259235182}) 2023-10-30T12:06:56.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.789+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 shutdown_connections 2023-10-30T12:06:56.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0cf80392f0 0x7f0cf803b7b0 unknown :-1 s=CLOSED pgs=4000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d0c11f170 0x7f0d0c1d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d0c132be0 0x7f0d0c11ec30 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.789+0000 7f0d128e1640 1 --2- 172.21.15.50:0/855419372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c0a4b60 0x7f0d0c11e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:56.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.790+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750101475}) 2023-10-30T12:06:56.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.790+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159397138}) 2023-10-30T12:06:56.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.790+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 >> 172.21.15.50:0/855419372 conn(0x7f0d0c0a8060 msgr2=0x7f0d0c134260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:56.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.790+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 shutdown_connections 2023-10-30T12:06:56.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.790+0000 7f0d128e1640 1 -- 172.21.15.50:0/855419372 wait complete. 2023-10-30T12:06:56.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:56.790+0000 7f0d128e1640 1 librados: shutdown 2023-10-30T12:06:56.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.791+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.791+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563504045}) 2023-10-30T12:06:56.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.791+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214189482}) 2023-10-30T12:06:56.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.791+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.806 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.791+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563504045}) 2023-10-30T12:06:56.806 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.791+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320076523}) 2023-10-30T12:06:56.806 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.806 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535091221}) 2023-10-30T12:06:56.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1957141241}) 2023-10-30T12:06:56.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535091221}) 2023-10-30T12:06:56.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306451716}) 2023-10-30T12:06:56.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390097692}) 2023-10-30T12:06:56.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409271533}) 2023-10-30T12:06:56.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390097692}) 2023-10-30T12:06:56.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.792+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472275426}) 2023-10-30T12:06:56.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.793+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.793+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151173109}) 2023-10-30T12:06:56.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.793+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763669023}) 2023-10-30T12:06:56.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.793+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.793+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151173109}) 2023-10-30T12:06:56.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.793+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3554993640}) 2023-10-30T12:06:56.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.794+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.794+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2033987183}) 2023-10-30T12:06:56.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.794+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=923280583}) 2023-10-30T12:06:56.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.794+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.794+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2033987183}) 2023-10-30T12:06:56.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.794+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251586347}) 2023-10-30T12:06:56.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.795+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.795+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831674913}) 2023-10-30T12:06:56.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.795+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3524884021}) 2023-10-30T12:06:56.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.795+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.795+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831674913}) 2023-10-30T12:06:56.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.795+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055476282}) 2023-10-30T12:06:56.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.796+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.796+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135929868}) 2023-10-30T12:06:56.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.796+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982061347}) 2023-10-30T12:06:56.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.796+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.796+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135929868}) 2023-10-30T12:06:56.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.796+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1806848280}) 2023-10-30T12:06:56.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.797+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.797+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626996550}) 2023-10-30T12:06:56.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.797+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272365495}) 2023-10-30T12:06:56.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.797+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.797+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626996550}) 2023-10-30T12:06:56.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.797+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047634510}) 2023-10-30T12:06:56.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.798+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.798+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534554846}) 2023-10-30T12:06:56.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.798+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533305685}) 2023-10-30T12:06:56.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.798+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.798+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534554846}) 2023-10-30T12:06:56.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.798+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381393863}) 2023-10-30T12:06:56.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.799+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.799+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296558168}) 2023-10-30T12:06:56.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.799+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713965644}) 2023-10-30T12:06:56.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.799+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.799+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296558168}) 2023-10-30T12:06:56.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.799+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326015628}) 2023-10-30T12:06:56.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.800+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.800+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920623885}) 2023-10-30T12:06:56.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.800+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=783550247}) 2023-10-30T12:06:56.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.800+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.800+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920623885}) 2023-10-30T12:06:56.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.800+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2652726214}) 2023-10-30T12:06:56.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.801+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.801+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662244441}) 2023-10-30T12:06:56.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.801+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3235782712}) 2023-10-30T12:06:56.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.801+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.801+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662244441}) 2023-10-30T12:06:56.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.801+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064984729}) 2023-10-30T12:06:56.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.802+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.802+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821658080}) 2023-10-30T12:06:56.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.802+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025645276}) 2023-10-30T12:06:56.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.802+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.802+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821658080}) 2023-10-30T12:06:56.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.802+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99434529}) 2023-10-30T12:06:56.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.803+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.803+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877815451}) 2023-10-30T12:06:56.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.803+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850772006}) 2023-10-30T12:06:56.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.803+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.803+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877815451}) 2023-10-30T12:06:56.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.803+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549174423}) 2023-10-30T12:06:56.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:06:56.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.804+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.804+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215551186}) 2023-10-30T12:06:56.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.804+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205228740}) 2023-10-30T12:06:56.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.804+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.804+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215551186}) 2023-10-30T12:06:56.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.804+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1448024023}) 2023-10-30T12:06:56.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.805+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.805+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663395917}) 2023-10-30T12:06:56.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.805+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550949622}) 2023-10-30T12:06:56.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.805+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.805+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663395917}) 2023-10-30T12:06:56.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.805+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2379035911}) 2023-10-30T12:06:56.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.806+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.806+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134573629}) 2023-10-30T12:06:56.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.806+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838482521}) 2023-10-30T12:06:56.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.806+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.806+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134573629}) 2023-10-30T12:06:56.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.806+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435337783}) 2023-10-30T12:06:56.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.807+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.807+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587931439}) 2023-10-30T12:06:56.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.807+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2318511622}) 2023-10-30T12:06:56.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.807+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.807+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587931439}) 2023-10-30T12:06:56.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.807+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459809256}) 2023-10-30T12:06:56.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.808+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.808+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419315649}) 2023-10-30T12:06:56.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.808+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919481180}) 2023-10-30T12:06:56.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.808+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.808+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419315649}) 2023-10-30T12:06:56.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.808+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2869673440}) 2023-10-30T12:06:56.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.809+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.809+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676699014}) 2023-10-30T12:06:56.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.809+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3427491318}) 2023-10-30T12:06:56.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.809+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.809+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676699014}) 2023-10-30T12:06:56.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.809+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3249749932}) 2023-10-30T12:06:56.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.810+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.810+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139228172}) 2023-10-30T12:06:56.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.810+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387912774}) 2023-10-30T12:06:56.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.810+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.810+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139228172}) 2023-10-30T12:06:56.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.810+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517612799}) 2023-10-30T12:06:56.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.811+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.811+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373118551}) 2023-10-30T12:06:56.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.811+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=708026583}) 2023-10-30T12:06:56.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.811+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.811+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373118551}) 2023-10-30T12:06:56.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.811+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562311664}) 2023-10-30T12:06:56.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.812+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.812+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194786354}) 2023-10-30T12:06:56.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.812+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989732915}) 2023-10-30T12:06:56.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.812+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.812+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194786354}) 2023-10-30T12:06:56.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.812+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2659999031}) 2023-10-30T12:06:56.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908605937}) 2023-10-30T12:06:56.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37057862}) 2023-10-30T12:06:56.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908605937}) 2023-10-30T12:06:56.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426822803}) 2023-10-30T12:06:56.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3869248331}) 2023-10-30T12:06:56.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2012591813}) 2023-10-30T12:06:56.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.813+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3869248331}) 2023-10-30T12:06:56.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3124389679}) 2023-10-30T12:06:56.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299681598}) 2023-10-30T12:06:56.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=159162616}) 2023-10-30T12:06:56.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299681598}) 2023-10-30T12:06:56.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.814+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178653488}) 2023-10-30T12:06:56.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.815+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.815+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879971318}) 2023-10-30T12:06:56.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.815+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072687565}) 2023-10-30T12:06:56.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.815+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.815+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879971318}) 2023-10-30T12:06:56.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.815+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370407373}) 2023-10-30T12:06:56.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.816+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.816+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007593046}) 2023-10-30T12:06:56.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.816+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578454938}) 2023-10-30T12:06:56.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.816+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.816+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007593046}) 2023-10-30T12:06:56.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.816+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=681301853}) 2023-10-30T12:06:56.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.817+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.817+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=29} crc {logm=3055443284,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=753921037,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=4085558033,osd_snap=2630283292,osdmap=2806403440}) 2023-10-30T12:06:56.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.817+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=30} crc {logm=3055443284,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2244507811,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=4085558033,osd_snap=2175200121,osdmap=3217851825}) 2023-10-30T12:06:56.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.817+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.817+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=29} crc {logm=3055443284,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=753921037,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=4085558033,osd_snap=2630283292,osdmap=2806403440}) 2023-10-30T12:06:56.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.817+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=32} crc {logm=3055443284,mdsmap=3984425268,mgr=3871354117,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=753921036,mon_config_key=3965234311,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=4085558033,osd_snap=3344623510,osdmap=553747158}) 2023-10-30T12:06:56.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.818+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.819+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1376446179}) 2023-10-30T12:06:56.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.819+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3192312545}) 2023-10-30T12:06:56.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.819+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.819+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1376446179}) 2023-10-30T12:06:56.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.819+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2901301404}) 2023-10-30T12:06:56.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.820+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.820+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1176724207}) 2023-10-30T12:06:56.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.820+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2170254546}) 2023-10-30T12:06:56.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.820+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.820+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1176724207}) 2023-10-30T12:06:56.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.820+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=656807600}) 2023-10-30T12:06:56.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.828+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.828+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=1875726174}) 2023-10-30T12:06:56.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.828+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=82} crc {osdmap=3927654518}) 2023-10-30T12:06:56.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.828+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:06:56.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.828+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=1875726174}) 2023-10-30T12:06:56.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:06:56.828+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=81} crc {osdmap=2628298008}) 2023-10-30T12:06:57.823 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:06:57.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:57.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.938+0000 7febb297c640 10 monclient: get_monmap_and_config 2023-10-30T12:06:57.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.938+0000 7febb297c640 10 monclient: build_initial_monmap 2023-10-30T12:06:57.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.939+0000 7febb297c640 10 monclient: monmap: 2023-10-30T12:06:57.941 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:57.941 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:57.941 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:57.940038+0000 2023-10-30T12:06:57.941 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:57.940038+0000 2023-10-30T12:06:57.942 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:57.942 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:57.942 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:57.942 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:57.942 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:57.943 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:57.943 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:57.943 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.939+0000 7febb297c640 1 Processor -- start 2023-10-30T12:06:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.939+0000 7febb297c640 1 -- start start 2023-10-30T12:06:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.939+0000 7febb297c640 10 monclient: init 2023-10-30T12:06:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:06:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febac09f5b0 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febac099b80 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7febac139b40 0x7febac13bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient(hunting): picked mon.noname-c con 0x7febac139b40 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac098810 con 0x7febac099b80 2023-10-30T12:06:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac096ea0 con 0x7febac09f5b0 2023-10-30T12:06:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac06aef0 con 0x7febac139b40 2023-10-30T12:06:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:57.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb297c640 10 monclient(hunting): authenticate will time out at 1263.123250s 2023-10-30T12:06:57.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febab7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7febac099b80 0x7febac096790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50866/0 (socket says 172.21.15.50:50866) 2023-10-30T12:06:57.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febab7fe640 1 -- 172.21.15.50:0/730436781 learned_addr learned my addr 172.21.15.50:0/730436781 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:06:57.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 1 --2- 172.21.15.50:0/730436781 >> v2:172.21.15.118:3300/0 conn(0x7febac139b40 0x7febac13bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:57.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505071834 0 0) 0x7febac098810 con 0x7febac099b80 2023-10-30T12:06:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient(hunting): _negotiate 2023-10-30T12:06:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient(hunting): my global_id is 104136 2023-10-30T12:06:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb98003520 con 0x7febac099b80 2023-10-30T12:06:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 10 monclient(hunting): get_auth_request con 0x7febac139b40 auth_method 0 2023-10-30T12:06:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 369203644 0 0) 0x7feb98003520 con 0x7febac099b80 2023-10-30T12:06:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 >> v2:172.21.15.118:3300/0 conn(0x7febac139b40 msgr2=0x7febac13bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 --2- 172.21.15.50:0/730436781 >> v2:172.21.15.118:3300/0 conn(0x7febac139b40 0x7febac13bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 >> v1:172.21.15.118:6789/0 conn(0x7febac09f5b0 legacy=0x7febac096080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:06:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:06:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac09a1a0 con 0x7febac099b80 2023-10-30T12:06:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: _finish_auth 0 2023-10-30T12:06:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7feb9c001690 con 0x7febac099b80 2023-10-30T12:06:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:06:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: dump: 2023-10-30T12:06:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:57.963 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:06:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.940+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 >> v1:172.21.15.50:6789/0 conn(0x7febac099b80 legacy=0x7febac096790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:06:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb980039a0 0x7feb98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): picked mon.a con 0x7feb980039a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980047c0 0x7feb98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): picked mon.b con 0x7feb980047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb980052a0 0x7feb9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): picked mon.c con 0x7feb980052a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac06aef0 con 0x7feb980039a0 2023-10-30T12:06:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9800c1a0 con 0x7feb980047c0 2023-10-30T12:06:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9800c360 con 0x7feb980052a0 2023-10-30T12:06:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febaaffd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:06:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb980039a0 0x7feb98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb980052a0 0x7feb9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980047c0 0x7feb98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): get_auth_request con 0x7feb980039a0 auth_method 0 2023-10-30T12:06:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 10 monclient(hunting): get_auth_request con 0x7feb980047c0 auth_method 0 2023-10-30T12:06:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 10 monclient(hunting): get_auth_request con 0x7feb980052a0 auth_method 0 2023-10-30T12:06:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:06:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:06:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:06:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:06:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient(hunting): handle_auth_done global_id 104136 payload 293 2023-10-30T12:06:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 1 -- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb980052a0 msgr2=0x7feb9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb980052a0 0x7feb9800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 1 -- 172.21.15.50:0/730436781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980047c0 msgr2=0x7feb98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:57.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980047c0 0x7feb98004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:57.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:57.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 1 monclient: found mon.a 2023-10-30T12:06:57.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:57.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:57.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 1 -- 172.21.15.50:0/730436781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98003520 con 0x7feb980039a0 2023-10-30T12:06:57.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient: _finish_auth 0 2023-10-30T12:06:57.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 10 monclient: _check_auth_tickets 2023-10-30T12:06:57.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:57.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb0f19640 1 -- 172.21.15.50:0/730436781 reap_dead start 2023-10-30T12:06:57.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb297c640 5 monclient: authenticate success, global_id 104136 2023-10-30T12:06:57.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febb297c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:57.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.941+0000 7febab7fe640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980047c0 0x7feb98004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:57.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febabfff640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb980039a0 0x7feb98003dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb94007fd0 tx=0x7feb980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3565ecdd17ac48f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:57.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940052d0 con 0x7feb980039a0 2023-10-30T12:06:57.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:57.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:57.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 10 monclient: dump: 2023-10-30T12:06:57.975 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:57.975 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:57.976 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:57.977 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9402e050 con 0x7feb980039a0 2023-10-30T12:06:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febb297c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febb297c640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 1 -- 172.21.15.50:0/730436781 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94003640 con 0x7feb980039a0 2023-10-30T12:06:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:57.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febb297c640 4 set_mon_vals no callback set 2023-10-30T12:06:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febaaffd640 10 monclient: dump: 2023-10-30T12:06:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:57.980 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:57.981 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febb297c640 10 monclient: shutdown 2023-10-30T12:06:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febb297c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:57.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febb297c640 1 -- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb980039a0 msgr2=0x7feb98003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.942+0000 7febb297c640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb980039a0 0x7feb98003dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb94007fd0 tx=0x7feb980057f0 comp rx=0 tx=0).stop 2023-10-30T12:06:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 -- 172.21.15.50:0/730436781 shutdown_connections 2023-10-30T12:06:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 --2- 172.21.15.50:0/730436781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb980039a0 0x7feb98003dc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 -- 172.21.15.50:0/730436781 >> 172.21.15.50:0/730436781 conn(0x7febac0a05c0 msgr2=0x7febac09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:57.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 -- 172.21.15.50:0/730436781 shutdown_connections 2023-10-30T12:06:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 -- 172.21.15.50:0/730436781 wait complete. 2023-10-30T12:06:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 10 monclient: build_initial_monmap 2023-10-30T12:06:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 10 monclient: monmap: 2023-10-30T12:06:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:57.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:57.985 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:57.985 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:57.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 librados: starting msgr at 2023-10-30T12:06:57.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 librados: starting objecter 2023-10-30T12:06:57.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.943+0000 7febb297c640 1 Processor -- start 2023-10-30T12:06:57.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 -- start start 2023-10-30T12:06:57.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 librados: setting wanted keys 2023-10-30T12:06:57.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 librados: calling monclient init 2023-10-30T12:06:57.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient: init 2023-10-30T12:06:57.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:57.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:06:57.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980039a0 0x7febac070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:57.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): picked mon.noname-b con 0x7feb980039a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:57.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac09af90 0x7febac095520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:57.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): picked mon.noname-a con 0x7febac09af90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:57.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febac095a60 0x7febac1ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:57.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): picked mon.noname-c con 0x7febac095a60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:57.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:57.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac095e70 con 0x7febac09af90 2023-10-30T12:06:57.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:57.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac072010 con 0x7feb980039a0 2023-10-30T12:06:57.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:57.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac136710 con 0x7febac095a60 2023-10-30T12:06:57.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:57.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb297c640 10 monclient(hunting): authenticate will time out at 1263.127650s 2023-10-30T12:06:57.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac09af90 0x7febac095520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:57.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febac095a60 0x7febac1ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:57.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac09af90 0x7febac095520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40864/0 (socket says 172.21.15.50:40864) 2023-10-30T12:06:57.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febac095a60 0x7febac1ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51168/0 (socket says 172.21.15.50:51168) 2023-10-30T12:06:57.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 1 -- 172.21.15.50:0/2833187950 learned_addr learned my addr 172.21.15.50:0/2833187950 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:57.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 10 monclient(hunting): get_auth_request con 0x7febac09af90 auth_method 0 2023-10-30T12:06:57.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:57.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:57.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 10 monclient(hunting): get_auth_request con 0x7febac095a60 auth_method 0 2023-10-30T12:06:57.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:57.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febb0f19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febabfff640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980039a0 0x7febac070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febabfff640 10 monclient(hunting): get_auth_request con 0x7feb980039a0 auth_method 0 2023-10-30T12:06:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.944+0000 7febabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:57.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 10 monclient(hunting): handle_auth_done global_id 104139 payload 824 2023-10-30T12:06:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 1 -- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febac095a60 msgr2=0x7febac1ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febac095a60 0x7febac1ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 1 -- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980039a0 msgr2=0x7febac070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:57.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980039a0 0x7febac070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 1 monclient: found mon.noname-a 2023-10-30T12:06:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 1 -- 172.21.15.50:0/2833187950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac09be80 con 0x7febac09af90 2023-10-30T12:06:57.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb0f19640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febac095a60 0x7febac1ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febab7fe640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac09af90 0x7febac095520 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0072c0 tx=0x7feb9c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 client.?.objecter ms_handle_connect 0x7febac09af90 2023-10-30T12:06:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 1 -- 172.21.15.50:0/2833187950 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c00f940 con 0x7febac09af90 2023-10-30T12:06:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:57.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: dump: 2023-10-30T12:06:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.998 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.000 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: _finish_auth 0 2023-10-30T12:06:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:06:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:58.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 1 -- 172.21.15.50:0/2833187950 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c00faa0 con 0x7febac09af90 2023-10-30T12:06:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 1 -- 172.21.15.50:0/2833187950 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c00fe10 con 0x7febac09af90 2023-10-30T12:06:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:58.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7feba8ff9640 10 monclient: dump: 2023-10-30T12:06:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.002 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.003 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 5 monclient: authenticate success, global_id 104139 2023-10-30T12:06:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 10 monclient: _renew_subs 2023-10-30T12:06:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac070d00 con 0x7febac09af90 2023-10-30T12:06:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febaa7fc640 4 set_mon_vals no callback set 2023-10-30T12:06:58.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 10 client.104139.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 10 monclient: _renew_subs 2023-10-30T12:06:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac1ca7f0 con 0x7febac09af90 2023-10-30T12:06:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.945+0000 7febb297c640 1 librados: init done 2023-10-30T12:06:58.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7feba8ff9640 1 -- 172.21.15.50:0/2833187950 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7feb9c01d460 con 0x7febac09af90 2023-10-30T12:06:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7febb297c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7feba8ff9640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb8c0392f0 0x7feb8c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7febb297c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7febb297c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac09c2f0 con 0x7febac09af90 2023-10-30T12:06:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7feba8ff9640 1 -- 172.21.15.50:0/2833187950 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6870+0+0 (secure 0 0 0) 0x7feb9c04aeb0 con 0x7febac09af90 2023-10-30T12:06:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7feba8ff9640 10 client.104139.objecter ms_dispatch 0x7febac000b90 osd_map(205..205 src has 1..205) v4 2023-10-30T12:06:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7feba8ff9640 3 client.104139.objecter handle_osd_map got epochs [205,205] > 0 2023-10-30T12:06:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7feba8ff9640 3 client.104139.objecter handle_osd_map decoding full epoch 205 2023-10-30T12:06:58.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7feba8ff9640 20 client.104139.objecter dump_active .. 0 homeless 2023-10-30T12:06:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7febabfff640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb8c0392f0 0x7feb8c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.946+0000 7febabfff640 10 monclient: get_auth_request con 0x7feb8c0392f0 auth_method 0 2023-10-30T12:06:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.947+0000 7feba8ff9640 10 client.104139.objecter ms_handle_connect 0x7feb8c0392f0 2023-10-30T12:06:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.947+0000 7febabfff640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb8c0392f0 0x7feb8c03b7b0 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7feb94033000 tx=0x7feb94009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.951+0000 7feba8ff9640 1 -- 172.21.15.50:0/2833187950 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb9c007d10 con 0x7febac09af90 2023-10-30T12:06:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.951+0000 7feba8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:57.951+0000 7feba8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:58.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.195+0000 7febb297c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.195+0000 7febb297c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.195+0000 7febb297c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.195+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7febac04e160 con 0x7febac09af90 2023-10-30T12:06:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.195+0000 7feba8ff9640 1 -- 172.21.15.50:0/2833187950 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7feb9c03c030 con 0x7febac09af90 2023-10-30T12:06:58.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.195+0000 7feba8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:58.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.195+0000 7feba8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:58.198 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 10 librados: watch_flush enter 2023-10-30T12:06:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 10 librados: watch_flush exit 2023-10-30T12:06:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 20 client.104139.objecter shutdown clearing up homeless session... 2023-10-30T12:06:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 10 client.104139.objecter successfully canceled tick 2023-10-30T12:06:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb8c0392f0 msgr2=0x7feb8c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb8c0392f0 0x7feb8c03b7b0 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7feb94033000 tx=0x7feb94009030 comp rx=0 tx=0).stop 2023-10-30T12:06:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 10 monclient: shutdown 2023-10-30T12:06:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac09af90 msgr2=0x7febac095520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac09af90 0x7febac095520 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0072c0 tx=0x7feb9c007780 comp rx=0 tx=0).stop 2023-10-30T12:06:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 shutdown_connections 2023-10-30T12:06:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feb8c0392f0 0x7feb8c03b7b0 unknown :-1 s=CLOSED pgs=4001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febac095a60 0x7febac1ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac09af90 0x7febac095520 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 --2- 172.21.15.50:0/2833187950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb980039a0 0x7febac070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 >> 172.21.15.50:0/2833187950 conn(0x7febac0a05c0 msgr2=0x7febac138930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:58.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 shutdown_connections 2023-10-30T12:06:58.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 -- 172.21.15.50:0/2833187950 wait complete. 2023-10-30T12:06:58.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.197+0000 7febb297c640 1 librados: shutdown 2023-10-30T12:06:58.209 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:06:58.209 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:06:58.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:06:58.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:06:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.313+0000 7ffab4583640 10 monclient: get_monmap_and_config 2023-10-30T12:06:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.313+0000 7ffab4583640 10 monclient: build_initial_monmap 2023-10-30T12:06:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.313+0000 7ffab4583640 10 monclient: monmap: 2023-10-30T12:06:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:58.314814+0000 2023-10-30T12:06:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:58.314814+0000 2023-10-30T12:06:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:58.317 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:58.318 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 1 Processor -- start 2023-10-30T12:06:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 1 -- start start 2023-10-30T12:06:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 10 monclient: init 2023-10-30T12:06:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:06:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffaac096540 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffaac136960 0x7ffaac138d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 10 monclient(hunting): picked mon.noname-c con 0x7ffaac136960 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffaac139290 0x7ffaac13b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.314+0000 7ffab4583640 10 monclient(hunting): picked mon.noname-a con 0x7ffaac139290 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab4583640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaac09bd30 con 0x7ffaac096540 2023-10-30T12:06:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab4583640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab4583640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaac09c400 con 0x7ffaac139290 2023-10-30T12:06:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab4583640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab4583640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaac09c120 con 0x7ffaac136960 2023-10-30T12:06:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab4583640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab4583640 10 monclient(hunting): authenticate will time out at 1263.498041s 2023-10-30T12:06:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffaac139290 0x7ffaac13b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffaac139290 0x7ffaac13b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40880/0 (socket says 172.21.15.50:40880) 2023-10-30T12:06:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 -- 172.21.15.50:0/939003853 learned_addr learned my addr 172.21.15.50:0/939003853 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): get_auth_request con 0x7ffaac139290 auth_method 0 2023-10-30T12:06:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab1b1e640 1 --2- 172.21.15.50:0/939003853 >> v2:172.21.15.118:3300/0 conn(0x7ffaac136960 0x7ffaac138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab1b1e640 10 monclient(hunting): get_auth_request con 0x7ffaac136960 auth_method 0 2023-10-30T12:06:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab1b1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab1b1e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab1b1e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient(hunting): handle_auth_done global_id 104142 payload 274 2023-10-30T12:06:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 -- 172.21.15.50:0/939003853 >> v2:172.21.15.118:3300/0 conn(0x7ffaac136960 msgr2=0x7ffaac138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 --2- 172.21.15.50:0/939003853 >> v2:172.21.15.118:3300/0 conn(0x7ffaac136960 0x7ffaac138d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 -- 172.21.15.50:0/939003853 >> v1:172.21.15.118:6789/0 conn(0x7ffaac096540 legacy=0x7ffaac096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient: _finish_hunting 0 2023-10-30T12:06:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 monclient: found mon.noname-a 2023-10-30T12:06:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab2b20640 1 -- 172.21.15.50:0/939003853 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac097030 con 0x7ffaac139290 2023-10-30T12:06:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.315+0000 7ffab1b1e640 1 --2- 172.21.15.50:0/939003853 >> v2:172.21.15.118:3300/0 conn(0x7ffaac136960 0x7ffaac138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:06:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab2b20640 1 --2- 172.21.15.50:0/939003853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac13b680 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffaac074290 tx=0x7ffaa402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cab662d9ed582c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 1 -- 172.21.15.50:0/939003853 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa402ca80 con 0x7ffaac139290 2023-10-30T12:06:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: dump: 2023-10-30T12:06:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.337 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: _finish_auth 0 2023-10-30T12:06:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: _check_auth_tickets 2023-10-30T12:06:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab4583640 5 monclient: authenticate success, global_id 104142 2023-10-30T12:06:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab4583640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 1 -- 172.21.15.50:0/939003853 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa402cbe0 con 0x7ffaac139290 2023-10-30T12:06:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab4583640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab4583640 4 set_mon_vals no callback set 2023-10-30T12:06:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 1 -- 172.21.15.50:0/939003853 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40343f0 con 0x7ffaac139290 2023-10-30T12:06:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.316+0000 7ffab131d640 10 monclient: dump: 2023-10-30T12:06:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.342 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 10 monclient: shutdown 2023-10-30T12:06:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 -- 172.21.15.50:0/939003853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 msgr2=0x7ffaac13b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 --2- 172.21.15.50:0/939003853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac13b680 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffaac074290 tx=0x7ffaa402bee0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 -- 172.21.15.50:0/939003853 shutdown_connections 2023-10-30T12:06:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 --2- 172.21.15.50:0/939003853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac13b680 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 --2- 172.21.15.50:0/939003853 >> v2:172.21.15.118:3300/0 conn(0x7ffaac136960 0x7ffaac138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 -- 172.21.15.50:0/939003853 >> 172.21.15.50:0/939003853 conn(0x7ffaac0a3980 msgr2=0x7ffaac0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 -- 172.21.15.50:0/939003853 shutdown_connections 2023-10-30T12:06:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 -- 172.21.15.50:0/939003853 wait complete. 2023-10-30T12:06:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 10 monclient: build_initial_monmap 2023-10-30T12:06:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 10 monclient: monmap: 2023-10-30T12:06:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:58.346 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 librados: starting msgr at 2023-10-30T12:06:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 librados: starting objecter 2023-10-30T12:06:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.317+0000 7ffab4583640 1 Processor -- start 2023-10-30T12:06:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 -- start start 2023-10-30T12:06:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 librados: setting wanted keys 2023-10-30T12:06:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 librados: calling monclient init 2023-10-30T12:06:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient: init 2023-10-30T12:06:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:06:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffaac096540 0x7ffaac08b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): picked mon.noname-c con 0x7ffaac096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac136960 0x7ffaac08bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): picked mon.noname-b con 0x7ffaac136960 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac08c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): picked mon.noname-a con 0x7ffaac139290 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac08d2b0 con 0x7ffaac139290 2023-10-30T12:06:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac09a080 con 0x7ffaac136960 2023-10-30T12:06:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac0a2850 con 0x7ffaac096540 2023-10-30T12:06:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab4583640 10 monclient(hunting): authenticate will time out at 1263.501476s 2023-10-30T12:06:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac08c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffaac096540 0x7ffaac08b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac08c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40894/0 (socket says 172.21.15.50:40894) 2023-10-30T12:06:58.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 -- 172.21.15.50:0/2336450879 learned_addr learned my addr 172.21.15.50:0/2336450879 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:58.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 10 monclient(hunting): get_auth_request con 0x7ffaac096540 auth_method 0 2023-10-30T12:06:58.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac136960 0x7ffaac08bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): get_auth_request con 0x7ffaac139290 auth_method 0 2023-10-30T12:06:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 10 monclient(hunting): get_auth_request con 0x7ffaac136960 auth_method 0 2023-10-30T12:06:58.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab1b1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient(hunting): handle_auth_done global_id 104145 payload 824 2023-10-30T12:06:58.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffaac096540 msgr2=0x7ffaac08b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffaac096540 0x7ffaac08b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac136960 msgr2=0x7ffaac08bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffaac136960 0x7ffaac08bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient: _finish_hunting 0 2023-10-30T12:06:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 monclient: found mon.noname-a 2023-10-30T12:06:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.318+0000 7ffab2b20640 1 -- 172.21.15.50:0/2336450879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4009b30 con 0x7ffaac139290 2023-10-30T12:06:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab2b20640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac08c0e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009f40 tx=0x7ffaa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 client.?.objecter ms_handle_connect 0x7ffaac139290 2023-10-30T12:06:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4009510 con 0x7ffaac139290 2023-10-30T12:06:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: dump: 2023-10-30T12:06:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.366 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:06:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa403e450 con 0x7ffaac139290 2023-10-30T12:06:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 5 monclient: authenticate success, global_id 104145 2023-10-30T12:06:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 10 monclient: _renew_subs 2023-10-30T12:06:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaac08dde0 con 0x7ffaac139290 2023-10-30T12:06:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa403e7e0 con 0x7ffaac139290 2023-10-30T12:06:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffaa2ffd640 10 monclient: dump: 2023-10-30T12:06:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.370 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 10 client.104145.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 10 monclient: _renew_subs 2023-10-30T12:06:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac08ed30 con 0x7ffaac139290 2023-10-30T12:06:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 1 librados: init done 2023-10-30T12:06:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab0b1c640 4 set_mon_vals no callback set 2023-10-30T12:06:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac08c620 0x7ffaac097560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.319+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac073ff0 con 0x7ffaac08c620 2023-10-30T12:06:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ffaa402cd50 con 0x7ffaac139290 2023-10-30T12:06:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac08c620 0x7ffaac097560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient: get_auth_request con 0x7ffaac08c620 auth_method 0 2023-10-30T12:06:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa900392f0 0x7ffa9003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6870+0+0 (secure 0 0 0) 0x7ffaa406d8f0 con 0x7ffaac139290 2023-10-30T12:06:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 client.104145.objecter ms_dispatch 0x7ffaac000b90 osd_map(205..205 src has 1..205) v4 2023-10-30T12:06:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 3 client.104145.objecter handle_osd_map got epochs [205,205] > 0 2023-10-30T12:06:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 3 client.104145.objecter handle_osd_map decoding full epoch 205 2023-10-30T12:06:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 10 monclient(hunting): handle_auth_done global_id 104148 payload 824 2023-10-30T12:06:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 1 -- 172.21.15.50:0/2336450879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac09ba70 con 0x7ffaac08c620 2023-10-30T12:06:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 20 client.104145.objecter dump_active .. 0 homeless 2023-10-30T12:06:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab1b1e640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa900392f0 0x7ffa9003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 client.104145.objecter ms_handle_connect 0x7ffaac08c620 2023-10-30T12:06:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 client.104145.objecter resend_mon_ops 2023-10-30T12:06:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab231f640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac08c620 0x7ffaac097560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800f520 tx=0x7ffaa800f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab1b1e640 10 monclient: get_auth_request con 0x7ffa900392f0 auth_method 0 2023-10-30T12:06:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80052e0 con 0x7ffaac08c620 2023-10-30T12:06:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 monclient: dump: 2023-10-30T12:06:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.380 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ffaa8005440 con 0x7ffaac08c620 2023-10-30T12:06:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac08c620 msgr2=0x7ffaac097560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac08c620 0x7ffaac097560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800f520 tx=0x7ffaa800f9e0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac08c620 msgr2=0x7ffaac097560 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac08c620 0x7ffaac097560 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800f520 tx=0x7ffaa800f9e0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffaa2ffd640 10 client.104145.objecter ms_handle_connect 0x7ffa900392f0 2023-10-30T12:06:58.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.320+0000 7ffab1b1e640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa900392f0 0x7ffa9003b7b0 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 crypto rx=0x7ffaac08cde0 tx=0x7ffa9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab4583640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab4583640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac097aa0 0x7ffaac097eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab4583640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac094010 con 0x7ffaac097aa0 2023-10-30T12:06:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac097aa0 0x7ffaac097eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient: get_auth_request con 0x7ffaac097aa0 auth_method 0 2023-10-30T12:06:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 10 monclient(hunting): handle_auth_done global_id 104151 payload 824 2023-10-30T12:06:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 1 -- 172.21.15.50:0/2336450879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffaac04e160 con 0x7ffaac097aa0 2023-10-30T12:06:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffab231f640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac097aa0 0x7ffaac097eb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8001000 tx=0x7ffaa8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 10 client.104145.objecter ms_handle_connect 0x7ffaac097aa0 2023-10-30T12:06:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 10 client.104145.objecter resend_mon_ops 2023-10-30T12:06:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8024070 con 0x7ffaac097aa0 2023-10-30T12:06:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 10 monclient: dump: 2023-10-30T12:06:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.390 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7ffaa801f2e0 con 0x7ffaac097aa0 2023-10-30T12:06:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac097aa0 msgr2=0x7ffaac097eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac097aa0 0x7ffaac097eb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8001000 tx=0x7ffaa8000f40 comp rx=0 tx=0).stop 2023-10-30T12:06:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac097aa0 msgr2=0x7ffaac097eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.327+0000 7ffaa2ffd640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac097aa0 0x7ffaac097eb0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8001000 tx=0x7ffaa8000f40 comp rx=0 tx=0).stop 2023-10-30T12:06:58.392 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:58.392 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:58.393 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:58.393 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:06:58.393 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 66, 2023-10-30T12:06:58.393 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:58.393 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:58.393 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:58.394 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:58.394 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.394 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:06:58.394 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:58.395 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:58.395 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:58.395 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.395 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.396 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.396 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.396 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.396 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.396 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.397 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.397 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.397 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.398 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:58.398 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:58.398 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.398 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.399 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.399 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.399 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.400 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.400 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.400 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.401 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.401 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.402 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:58.402 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:58.403 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:58.403 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:58.403 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:58.404 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:58.404 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:58.404 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:58.405 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:58.405 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:58.405 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:58.406 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:58.406 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:58.406 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:58.407 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:58.407 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:58.407 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:58.408 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.408 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.408 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.408 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.409 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.409 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.409 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.409 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.409 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.410 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.410 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:58.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:58.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.411 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:58.411 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:58.411 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:58.412 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:58.414 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:58.414 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.414 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.414 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.415 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:58.415 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:58.415 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.415 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.416 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:58.416 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.417 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.417 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:58.417 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:58.418 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:58.418 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.418 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.419 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.419 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:58.419 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:58.419 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.420 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.420 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.420 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:58.420 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.423 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.423 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.423 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.423 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.424 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:58.424 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:58.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.424 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.425 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.425 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:58.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.425 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.426 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:58.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.426 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.426 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:06:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:58.428 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:58.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 10 librados: watch_flush enter 2023-10-30T12:06:58.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 10 librados: watch_flush exit 2023-10-30T12:06:58.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 20 client.104145.objecter shutdown clearing up homeless session... 2023-10-30T12:06:58.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 10 client.104145.objecter successfully canceled tick 2023-10-30T12:06:58.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa900392f0 msgr2=0x7ffa9003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa900392f0 0x7ffa9003b7b0 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 crypto rx=0x7ffaac08cde0 tx=0x7ffa9c008040 comp rx=0 tx=0).stop 2023-10-30T12:06:58.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 10 monclient: shutdown 2023-10-30T12:06:58.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:58.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 msgr2=0x7ffaac08c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 --2- 172.21.15.50:0/2336450879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaac139290 0x7ffaac08c0e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009f40 tx=0x7ffaa4004a40 comp rx=0 tx=0).stop 2023-10-30T12:06:58.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab2b20640 1 -- 172.21.15.50:0/2336450879 reap_dead start 2023-10-30T12:06:58.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab2b20640 1 -- 172.21.15.50:0/2336450879 reap_dead start 2023-10-30T12:06:58.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 shutdown_connections 2023-10-30T12:06:58.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 >> 172.21.15.50:0/2336450879 conn(0x7ffaac0a3980 msgr2=0x7ffaac071500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:58.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 shutdown_connections 2023-10-30T12:06:58.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 -- 172.21.15.50:0/2336450879 wait complete. 2023-10-30T12:06:58.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.328+0000 7ffab4583640 1 librados: shutdown 2023-10-30T12:06:58.432 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:58.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:06:58.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:06:58.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.542+0000 7fe99d2bb640 10 monclient: get_monmap_and_config 2023-10-30T12:06:58.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.542+0000 7fe99d2bb640 10 monclient: build_initial_monmap 2023-10-30T12:06:58.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.542+0000 7fe99d2bb640 10 monclient: monmap: 2023-10-30T12:06:58.544 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:58.545 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.545 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:58.543128+0000 2023-10-30T12:06:58.546 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:58.543128+0000 2023-10-30T12:06:58.546 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:58.546 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.547 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:58.547 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:58.547 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:58.548 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:58.548 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:58.548 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.542+0000 7fe99d2bb640 1 Processor -- start 2023-10-30T12:06:58.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.542+0000 7fe99d2bb640 1 -- start start 2023-10-30T12:06:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.542+0000 7fe99d2bb640 10 monclient: init 2023-10-30T12:06:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T12:06:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe9980a8010 0x7fe998132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): picked mon.noname-c con 0x7fe9980a8010 addr v2:172.21.15.118:3300/0 2023-10-30T12:06:58.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe998132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): picked mon.noname-b con 0x7fe998099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:58.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe9981356d0 0x7fe998135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): picked mon.noname-a con 0x7fe9981356d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:58.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9980a7190 con 0x7fe9981356d0 2023-10-30T12:06:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe998133110 con 0x7fe9980a8010 2023-10-30T12:06:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980a83f0 con 0x7fe998099a40 2023-10-30T12:06:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe99d2bb640 10 monclient(hunting): authenticate will time out at 1263.726254s 2023-10-30T12:06:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe998132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe9981356d0 0x7fe998135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe998132bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51194/0 (socket says 172.21.15.50:51194) 2023-10-30T12:06:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 1 -- 172.21.15.50:0/2798805191 learned_addr learned my addr 172.21.15.50:0/2798805191 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 10 monclient(hunting): get_auth_request con 0x7fe998099a40 auth_method 0 2023-10-30T12:06:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): get_auth_request con 0x7fe9981356d0 auth_method 0 2023-10-30T12:06:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 1 --2- 172.21.15.50:0/2798805191 >> v2:172.21.15.118:3300/0 conn(0x7fe9980a8010 0x7fe998132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 10 monclient(hunting): get_auth_request con 0x7fe9980a8010 auth_method 0 2023-10-30T12:06:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient(hunting): handle_auth_done global_id 104154 payload 274 2023-10-30T12:06:58.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 1 -- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 msgr2=0x7fe998132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:58.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 1 --2- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe998132bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 1 -- 172.21.15.50:0/2798805191 >> v2:172.21.15.118:3300/0 conn(0x7fe9980a8010 msgr2=0x7fe998132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:58.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 1 --2- 172.21.15.50:0/2798805191 >> v2:172.21.15.118:3300/0 conn(0x7fe9980a8010 0x7fe998132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient: _finish_hunting 0 2023-10-30T12:06:58.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 1 monclient: found mon.noname-a 2023-10-30T12:06:58.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:58.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:58.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9977fe640 1 -- 172.21.15.50:0/2798805191 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9980a7600 con 0x7fe9981356d0 2023-10-30T12:06:58.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.543+0000 7fe9967fc640 1 --2- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe998132bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:58.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe9977fe640 1 --2- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9981356d0 0x7fe998135b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe9980a8640 tx=0x7fe99002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8800c1f33379c04a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 1 -- 172.21.15.50:0/2798805191 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe990030710 con 0x7fe9981356d0 2023-10-30T12:06:58.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:58.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: dump: 2023-10-30T12:06:58.579 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.579 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.579 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.579 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.579 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.580 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.580 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.580 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.580 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.580 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: _finish_auth 0 2023-10-30T12:06:58.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:06:58.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:58.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 1 -- 172.21.15.50:0/2798805191 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99002e050 con 0x7fe9981356d0 2023-10-30T12:06:58.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:58.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 1 -- 172.21.15.50:0/2798805191 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe990030a90 con 0x7fe9981356d0 2023-10-30T12:06:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe99d2bb640 5 monclient: authenticate success, global_id 104154 2023-10-30T12:06:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe99d2bb640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe99d2bb640 4 set_mon_vals no callback set 2023-10-30T12:06:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.544+0000 7fe995ffb640 10 monclient: dump: 2023-10-30T12:06:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.584 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.584 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 10 monclient: shutdown 2023-10-30T12:06:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9981356d0 msgr2=0x7fe998135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9981356d0 0x7fe998135b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe9980a8640 tx=0x7fe99002b710 comp rx=0 tx=0).stop 2023-10-30T12:06:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2798805191 shutdown_connections 2023-10-30T12:06:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9981356d0 0x7fe998135b40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2798805191 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe998132bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2798805191 >> v2:172.21.15.118:3300/0 conn(0x7fe9980a8010 0x7fe998132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2798805191 >> 172.21.15.50:0/2798805191 conn(0x7fe9980a4a10 msgr2=0x7fe9980a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:58.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2798805191 shutdown_connections 2023-10-30T12:06:58.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2798805191 wait complete. 2023-10-30T12:06:58.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 10 monclient: build_initial_monmap 2023-10-30T12:06:58.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 10 monclient: monmap: 2023-10-30T12:06:58.587 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:58.587 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:58.587 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:58.587 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:58.587 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:58.588 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 librados: starting msgr at 2023-10-30T12:06:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.545+0000 7fe99d2bb640 1 librados: starting objecter 2023-10-30T12:06:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 Processor -- start 2023-10-30T12:06:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 -- start start 2023-10-30T12:06:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 librados: setting wanted keys 2023-10-30T12:06:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 librados: calling monclient init 2023-10-30T12:06:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient: init 2023-10-30T12:06:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:06:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9980a8010 0x7fe9981ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): picked mon.noname-b con 0x7fe9980a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9981356d0 0x7fe9981cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): picked mon.noname-c con 0x7fe9981356d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe99811e590 0x7fe99811ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): picked mon.noname-a con 0x7fe99811e590 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe998122420 con 0x7fe99811e590 2023-10-30T12:06:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe998121d60 con 0x7fe9980a8010 2023-10-30T12:06:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe998120ee0 con 0x7fe9981356d0 2023-10-30T12:06:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe99d2bb640 10 monclient(hunting): authenticate will time out at 1263.729730s 2023-10-30T12:06:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9981356d0 0x7fe9981cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe99811e590 0x7fe99811ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9981356d0 0x7fe9981cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51206/0 (socket says 172.21.15.50:51206) 2023-10-30T12:06:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 1 -- 172.21.15.50:0/2869545495 learned_addr learned my addr 172.21.15.50:0/2869545495 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 10 monclient(hunting): get_auth_request con 0x7fe9981356d0 auth_method 0 2023-10-30T12:06:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 10 monclient(hunting): get_auth_request con 0x7fe99811e590 auth_method 0 2023-10-30T12:06:58.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.546+0000 7fe9977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe996ffd640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9980a8010 0x7fe9981ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe996ffd640 10 monclient(hunting): get_auth_request con 0x7fe9980a8010 auth_method 0 2023-10-30T12:06:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe996ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe996ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe996ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 10 monclient(hunting): handle_auth_done global_id 44993 payload 824 2023-10-30T12:06:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9980a8010 msgr2=0x7fe9981ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9980a8010 0x7fe9981ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe99811e590 msgr2=0x7fe99811ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe99811e590 0x7fe99811ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 10 monclient: _finish_hunting 0 2023-10-30T12:06:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 1 monclient: found mon.noname-c 2023-10-30T12:06:58.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 1 -- 172.21.15.50:0/2869545495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c0094d0 con 0x7fe9981356d0 2023-10-30T12:06:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9967fc640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9981356d0 0x7fe9981cab70 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00e930 tx=0x7fe98c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 client.?.objecter ms_handle_connect 0x7fe9981356d0 2023-10-30T12:06:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:58.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c00ca00 con 0x7fe9981356d0 2023-10-30T12:06:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: dump: 2023-10-30T12:06:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.600 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.601 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.602 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:06:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:06:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c00cb60 con 0x7fe9981356d0 2023-10-30T12:06:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 5 monclient: authenticate success, global_id 44993 2023-10-30T12:06:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 10 monclient: _renew_subs 2023-10-30T12:06:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe990009400 con 0x7fe9981356d0 2023-10-30T12:06:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c013470 con 0x7fe9981356d0 2023-10-30T12:06:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe97f7fe640 10 monclient: dump: 2023-10-30T12:06:58.604 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.606 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe9957fa640 4 set_mon_vals no callback set 2023-10-30T12:06:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 10 client.44993.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 10 monclient: _renew_subs 2023-10-30T12:06:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe998122600 con 0x7fe9981356d0 2023-10-30T12:06:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.547+0000 7fe99d2bb640 1 librados: init done 2023-10-30T12:06:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe98c0135d0 con 0x7fe9981356d0 2023-10-30T12:06:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe99d2bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe99802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe998073fd0 con 0x7fe998099a40 2023-10-30T12:06:58.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe99802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient: get_auth_request con 0x7fe998099a40 auth_method 0 2023-10-30T12:06:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe984039320 0x7fe98403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7fe98c04b110 con 0x7fe9981356d0 2023-10-30T12:06:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 10 client.44993.objecter ms_dispatch 0x7fe998000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:06:58.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 3 client.44993.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:06:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 3 client.44993.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:06:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 10 monclient(hunting): handle_auth_done global_id 44996 payload 824 2023-10-30T12:06:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 1 -- 172.21.15.50:0/2869545495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe99809aa00 con 0x7fe998099a40 2023-10-30T12:06:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 20 client.44993.objecter dump_active .. 0 homeless 2023-10-30T12:06:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe9977fe640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe984039320 0x7fe98403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe9977fe640 10 monclient: get_auth_request con 0x7fe984039320 auth_method 0 2023-10-30T12:06:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 10 client.44993.objecter ms_handle_connect 0x7fe998099a40 2023-10-30T12:06:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe996ffd640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe99802b430 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fe988007f00 tx=0x7fe9880115a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.548+0000 7fe97f7fe640 10 client.44993.objecter resend_mon_ops 2023-10-30T12:06:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98800e030 con 0x7fe998099a40 2023-10-30T12:06:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 10 monclient: dump: 2023-10-30T12:06:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.613 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.614 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe9977fe640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe984039320 0x7fe98403b7e0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7fe990034000 tx=0x7fe99002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 10 client.44993.objecter ms_handle_connect 0x7fe984039320 2023-10-30T12:06:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fe988003990 con 0x7fe998099a40 2023-10-30T12:06:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:58.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 msgr2=0x7fe99802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe99802b430 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fe988007f00 tx=0x7fe9880115a0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 msgr2=0x7fe99802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.549+0000 7fe97f7fe640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe998099a40 0x7fe99802b430 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe99d2bb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe99811f740 0x7fe99802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe99d2bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99809a720 con 0x7fe99811f740 2023-10-30T12:06:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe99811f740 0x7fe99802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient: get_auth_request con 0x7fe99811f740 auth_method 0 2023-10-30T12:06:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 10 monclient(hunting): handle_auth_done global_id 44999 payload 824 2023-10-30T12:06:58.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.555+0000 7fe996ffd640 1 -- 172.21.15.50:0/2869545495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe998136080 con 0x7fe99811f740 2023-10-30T12:06:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 10 client.44993.objecter ms_handle_connect 0x7fe99811f740 2023-10-30T12:06:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 10 client.44993.objecter resend_mon_ops 2023-10-30T12:06:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe996ffd640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe99811f740 0x7fe99802b970 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fe988000f80 tx=0x7fe988004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988025070 con 0x7fe99811f740 2023-10-30T12:06:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 10 monclient: dump: 2023-10-30T12:06:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.620 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.621 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fe988016470 con 0x7fe99811f740 2023-10-30T12:06:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe99811f740 msgr2=0x7fe99802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe99811f740 0x7fe99802b970 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fe988000f80 tx=0x7fe988004a40 comp rx=0 tx=0).stop 2023-10-30T12:06:58.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe99811f740 msgr2=0x7fe99802b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:58.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe97f7fe640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe99811f740 0x7fe99802b970 secure :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fe988000f80 tx=0x7fe988004a40 comp rx=0 tx=0).stop 2023-10-30T12:06:58.623 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 66, 2023-10-30T12:06:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:06:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.648 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.648 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.648 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:58.648 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:58.648 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.649 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.649 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.649 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:58.650 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:58.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.650 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.650 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.650 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:58.650 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.651 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.651 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:06:58.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.651 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.651 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.652 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:06:58.652 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.652 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.652 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:58.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.652 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.653 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.653 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:06:58.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.653 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.653 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:58.654 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:58.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 10 librados: watch_flush enter 2023-10-30T12:06:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 10 librados: watch_flush exit 2023-10-30T12:06:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 20 client.44993.objecter shutdown clearing up homeless session... 2023-10-30T12:06:58.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 10 client.44993.objecter successfully canceled tick 2023-10-30T12:06:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe984039320 msgr2=0x7fe98403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe984039320 0x7fe98403b7e0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7fe990034000 tx=0x7fe99002f000 comp rx=0 tx=0).stop 2023-10-30T12:06:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 10 monclient: shutdown 2023-10-30T12:06:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:58.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9981356d0 msgr2=0x7fe9981cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 --2- 172.21.15.50:0/2869545495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9981356d0 0x7fe9981cab70 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00e930 tx=0x7fe98c00edf0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe9977fe640 1 -- 172.21.15.50:0/2869545495 reap_dead start 2023-10-30T12:06:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe9977fe640 1 -- 172.21.15.50:0/2869545495 reap_dead start 2023-10-30T12:06:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 shutdown_connections 2023-10-30T12:06:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 >> 172.21.15.50:0/2869545495 conn(0x7fe9980a4a10 msgr2=0x7fe998134f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:58.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 shutdown_connections 2023-10-30T12:06:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.556+0000 7fe99d2bb640 1 -- 172.21.15.50:0/2869545495 wait complete. 2023-10-30T12:06:58.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.557+0000 7fe99d2bb640 1 librados: shutdown 2023-10-30T12:06:58.660 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:58.660 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:06:58.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:06:58.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.765+0000 7f04f0896640 10 monclient: get_monmap_and_config 2023-10-30T12:06:58.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.765+0000 7f04f0896640 10 monclient: build_initial_monmap 2023-10-30T12:06:58.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.765+0000 7f04f0896640 10 monclient: monmap: 2023-10-30T12:06:58.767 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:58.767 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:58.766680+0000 2023-10-30T12:06:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:58.766680+0000 2023-10-30T12:06:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:58.769 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:58.769 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:58.769 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:58.769 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 1 Processor -- start 2023-10-30T12:06:58.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 1 -- start start 2023-10-30T12:06:58.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient: init 2023-10-30T12:06:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T12:06:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient(hunting): picked mon.noname-b con 0x7f04ec097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f04ec071c00 0x7f04ec09f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec071c00 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:58.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04ec09aa60 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:58.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec097600 con 0x7f04ec09aa60 2023-10-30T12:06:58.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec0978e0 con 0x7f04ec071c00 2023-10-30T12:06:58.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec09bf50 con 0x7f04ec097fe0 2023-10-30T12:06:58.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:58.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.766+0000 7f04f0896640 10 monclient(hunting): authenticate will time out at 1263.949956s 2023-10-30T12:06:58.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f04ec071c00 0x7f04ec09f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec09a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51240/0 (socket says 172.21.15.50:51240) 2023-10-30T12:06:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 -- 172.21.15.50:0/1576715275 learned_addr learned my addr 172.21.15.50:0/1576715275 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): get_auth_request con 0x7f04ec097fe0 auth_method 0 2023-10-30T12:06:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 10 monclient(hunting): get_auth_request con 0x7f04ec071c00 auth_method 0 2023-10-30T12:06:58.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient(hunting): handle_auth_done global_id 45002 payload 274 2023-10-30T12:06:58.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 -- 172.21.15.50:0/1576715275 >> v2:172.21.15.50:3300/0 conn(0x7f04ec071c00 msgr2=0x7f04ec09f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:06:58.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 --2- 172.21.15.50:0/1576715275 >> v2:172.21.15.50:3300/0 conn(0x7f04ec071c00 0x7f04ec09f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 -- 172.21.15.50:0/1576715275 >> v1:172.21.15.118:6789/0 conn(0x7f04ec09aa60 legacy=0x7f04ec09af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient: _finish_hunting 0 2023-10-30T12:06:58.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 monclient: found mon.noname-b 2023-10-30T12:06:58.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:58.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:06:58.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04ea59c640 1 -- 172.21.15.50:0/1576715275 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec09d9b0 con 0x7f04ec097fe0 2023-10-30T12:06:58.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.767+0000 7f04e3fff640 1 --2- 172.21.15.50:0/1576715275 >> v2:172.21.15.50:3300/0 conn(0x7f04ec071c00 0x7f04ec09f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:58.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04ea59c640 1 --2- 172.21.15.50:0/1576715275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec09a150 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009850 tx=0x7f04dc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d0851978633f58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 1 -- 172.21.15.50:0/1576715275 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc02e070 con 0x7f04ec097fe0 2023-10-30T12:06:58.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:06:58.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: dump: 2023-10-30T12:06:58.786 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.786 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.786 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.786 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.787 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.787 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.787 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.787 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.787 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.787 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: _finish_auth 0 2023-10-30T12:06:58.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:06:58.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:58.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 5 monclient: authenticate success, global_id 45002 2023-10-30T12:06:58.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:06:58.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 1 -- 172.21.15.50:0/1576715275 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc004030 con 0x7f04ec097fe0 2023-10-30T12:06:58.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:58.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:58.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 4 set_mon_vals no callback set 2023-10-30T12:06:58.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 1 -- 172.21.15.50:0/1576715275 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc004420 con 0x7f04ec097fe0 2023-10-30T12:06:58.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:58.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04e9d9b640 10 monclient: dump: 2023-10-30T12:06:58.790 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.790 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.790 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.790 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.791 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.791 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.791 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.791 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.791 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.791 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 10 monclient: shutdown 2023-10-30T12:06:58.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:58.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 -- 172.21.15.50:0/1576715275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 msgr2=0x7f04ec09a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 --2- 172.21.15.50:0/1576715275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec09a150 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009850 tx=0x7f04dc02b840 comp rx=0 tx=0).stop 2023-10-30T12:06:58.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 -- 172.21.15.50:0/1576715275 shutdown_connections 2023-10-30T12:06:58.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 --2- 172.21.15.50:0/1576715275 >> v2:172.21.15.50:3300/0 conn(0x7f04ec071c00 0x7f04ec09f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 --2- 172.21.15.50:0/1576715275 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec09a150 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 -- 172.21.15.50:0/1576715275 >> 172.21.15.50:0/1576715275 conn(0x7f04ec0a3980 msgr2=0x7f04ec0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:58.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 -- 172.21.15.50:0/1576715275 shutdown_connections 2023-10-30T12:06:58.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.768+0000 7f04f0896640 1 -- 172.21.15.50:0/1576715275 wait complete. 2023-10-30T12:06:58.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 10 monclient: build_initial_monmap 2023-10-30T12:06:58.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 10 monclient: monmap: 2023-10-30T12:06:58.794 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:58.794 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:58.794 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:58.794 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:58.794 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:58.795 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 1 librados: starting msgr at 2023-10-30T12:06:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 1 librados: starting objecter 2023-10-30T12:06:58.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 1 Processor -- start 2023-10-30T12:06:58.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 1 -- start start 2023-10-30T12:06:58.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 1 librados: setting wanted keys 2023-10-30T12:06:58.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 1 librados: calling monclient init 2023-10-30T12:06:58.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.769+0000 7f04f0896640 10 monclient: init 2023-10-30T12:06:58.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:58.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:06:58.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec071c00 0x7f04ec128950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): picked mon.noname-b con 0x7f04ec071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:58.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec128e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): picked mon.noname-c con 0x7f04ec097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:58.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ec09aa60 0x7f04ec1293d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec09aa60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:58.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0963a0 con 0x7f04ec09aa60 2023-10-30T12:06:58.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec130520 con 0x7f04ec071c00 2023-10-30T12:06:58.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec130a50 con 0x7f04ec097fe0 2023-10-30T12:06:58.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:58.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04f0896640 10 monclient(hunting): authenticate will time out at 1263.953441s 2023-10-30T12:06:58.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec128e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec128e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51244/0 (socket says 172.21.15.50:51244) 2023-10-30T12:06:58.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ec09aa60 0x7f04ec1293d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 -- 172.21.15.50:0/2334246362 learned_addr learned my addr 172.21.15.50:0/2334246362 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:58.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): get_auth_request con 0x7f04ec097fe0 auth_method 0 2023-10-30T12:06:58.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec071c00 0x7f04ec128950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 10 monclient(hunting): get_auth_request con 0x7f04ec09aa60 auth_method 0 2023-10-30T12:06:58.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 10 monclient(hunting): get_auth_request con 0x7f04ec071c00 auth_method 0 2023-10-30T12:06:58.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient(hunting): handle_auth_done global_id 45005 payload 824 2023-10-30T12:06:58.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec071c00 msgr2=0x7f04ec128950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec071c00 0x7f04ec128950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ec09aa60 msgr2=0x7f04ec1293d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ec09aa60 0x7f04ec1293d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:06:58.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 monclient: found mon.noname-c 2023-10-30T12:06:58.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:58.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:06:58.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04e3fff640 1 -- 172.21.15.50:0/2334246362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d4009400 con 0x7f04ec097fe0 2023-10-30T12:06:58.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.770+0000 7f04ead9d640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ec09aa60 0x7f04ec1293d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:06:58.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e3fff640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec128e90 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f04d400ec10 tx=0x7f04d400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 client.?.objecter ms_handle_connect 0x7f04ec097fe0 2023-10-30T12:06:58.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:58.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4009760 con 0x7f04ec097fe0 2023-10-30T12:06:58.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:06:58.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: dump: 2023-10-30T12:06:58.807 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.807 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.809 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.809 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.809 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:06:58.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:06:58.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:58.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d400cc30 con 0x7f04ec097fe0 2023-10-30T12:06:58.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:58.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4013470 con 0x7f04ec097fe0 2023-10-30T12:06:58.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 5 monclient: authenticate success, global_id 45005 2023-10-30T12:06:58.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 10 monclient: _renew_subs 2023-10-30T12:06:58.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:58.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc009460 con 0x7f04ec097fe0 2023-10-30T12:06:58.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e959a640 4 set_mon_vals no callback set 2023-10-30T12:06:58.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:06:58.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04e2ffd640 10 monclient: dump: 2023-10-30T12:06:58.812 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.812 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.812 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.812 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.812 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.812 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.813 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.813 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.813 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.813 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 10 client.45005.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:58.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 10 monclient: _renew_subs 2023-10-30T12:06:58.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:06:58.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec12a550 con 0x7f04ec097fe0 2023-10-30T12:06:58.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.771+0000 7f04f0896640 1 librados: init done 2023-10-30T12:06:58.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f04d40135d0 con 0x7f04ec097fe0 2023-10-30T12:06:58.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04f0896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04f0896640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec131150 0x7f04ec1cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04f0896640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec1301d0 con 0x7f04ec131150 2023-10-30T12:06:58.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04cc0392f0 0x7f04cc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f04d404b240 con 0x7f04ec097fe0 2023-10-30T12:06:58.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 10 client.45005.objecter ms_dispatch 0x7f04ec000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:06:58.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 3 client.45005.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:06:58.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 3 client.45005.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:06:58.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec131150 0x7f04ec1cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient: get_auth_request con 0x7f04ec131150 auth_method 0 2023-10-30T12:06:58.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 20 client.45005.objecter dump_active .. 0 homeless 2023-10-30T12:06:58.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ead9d640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04cc0392f0 0x7f04cc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ead9d640 10 monclient: get_auth_request con 0x7f04cc0392f0 auth_method 0 2023-10-30T12:06:58.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 10 monclient(hunting): handle_auth_done global_id 94327 payload 824 2023-10-30T12:06:58.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 1 -- 172.21.15.50:0/2334246362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04ec072020 con 0x7f04ec131150 2023-10-30T12:06:58.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ead9d640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04cc0392f0 0x7f04cc03b7b0 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7f04ec12a910 tx=0x7f04e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 10 client.45005.objecter ms_handle_connect 0x7f04cc0392f0 2023-10-30T12:06:58.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 10 client.45005.objecter ms_handle_connect 0x7f04ec131150 2023-10-30T12:06:58.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04e2ffd640 10 client.45005.objecter resend_mon_ops 2023-10-30T12:06:58.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.772+0000 7f04ea59c640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec131150 0x7f04ec1cae20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04ec09cdd0 tx=0x7f04dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc003810 con 0x7f04ec131150 2023-10-30T12:06:58.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:58.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 10 monclient: dump: 2023-10-30T12:06:58.820 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.820 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.820 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.820 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.821 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.821 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.821 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.821 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.821 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.822 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f04dc041070 con 0x7f04ec131150 2023-10-30T12:06:58.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:58.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:58.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec131150 msgr2=0x7f04ec1cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec131150 0x7f04ec1cae20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04ec09cdd0 tx=0x7f04dc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec131150 msgr2=0x7f04ec1cae20 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:58.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.773+0000 7f04e2ffd640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec131150 0x7f04ec1cae20 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04ec09cdd0 tx=0x7f04dc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.779+0000 7f04f0896640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:06:58.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.779+0000 7f04f0896640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec02b430 0x7f04ec02b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:58.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.779+0000 7f04f0896640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:58.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.779+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec073ff0 con 0x7f04ec02b430 2023-10-30T12:06:58.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.779+0000 7f04ea59c640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec02b430 0x7f04ec02b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:58.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient: get_auth_request con 0x7f04ec02b430 auth_method 0 2023-10-30T12:06:58.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:58.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:58.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:58.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:58.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:58.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 10 monclient(hunting): handle_auth_done global_id 94330 payload 824 2023-10-30T12:06:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 1 -- 172.21.15.50:0/2334246362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04ec04e160 con 0x7f04ec02b430 2023-10-30T12:06:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04ea59c640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec02b430 0x7f04ec02b840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04dc000c00 tx=0x7f04dc031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 10 client.45005.objecter ms_handle_connect 0x7f04ec02b430 2023-10-30T12:06:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 10 client.45005.objecter resend_mon_ops 2023-10-30T12:06:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc006700 con 0x7f04ec02b430 2023-10-30T12:06:58.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:58.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:06:58.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 10 monclient: dump: 2023-10-30T12:06:58.828 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:58.828 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:58.828 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.829 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:58.829 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:58.829 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:58.829 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:58.829 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:58.829 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:58.830 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:58.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f04dc006860 con 0x7f04ec02b430 2023-10-30T12:06:58.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:06:58.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:58.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec02b430 msgr2=0x7f04ec02b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec02b430 0x7f04ec02b840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04dc000c00 tx=0x7f04dc031000 comp rx=0 tx=0).stop 2023-10-30T12:06:58.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec02b430 msgr2=0x7f04ec02b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:06:58.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.780+0000 7f04e2ffd640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ec02b430 0x7f04ec02b840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.831 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:06:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:06:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 66, 2023-10-30T12:06:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:06:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:06:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:06:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:06:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:06:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:06:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:06:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:06:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:06:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:06:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:06:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:06:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:06:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:06:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:06:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:06:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:06:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:06:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:06:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:06:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:06:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:06:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:06:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:06:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:06:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:06:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:06:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:06:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:06:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:06:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:06:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:06:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:06:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:06:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:06:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:06:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:06:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:06:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:06:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.845 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:06:58.845 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.846 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.846 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:06:58.846 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.847 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.847 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:06:58.847 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:06:58.847 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.847 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.848 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.848 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:06:58.848 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:06:58.848 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:06:58.849 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.850 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.850 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:06:58.850 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.850 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.851 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:06:58.851 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:06:58.851 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.851 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.851 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.852 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:06:58.852 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:06:58.852 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:06:58.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:06:58.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:06:58.853 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:06:58.853 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:06:58.854 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:06:58.854 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:06:58.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.854 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.855 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:06:58.855 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:06:58.855 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:06:58.855 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:06:58.855 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:06:58.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.856 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.856 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:06:58.856 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:06:58.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.856 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.857 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.857 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:58.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.857 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.857 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:06:58.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.858 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.858 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.858 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:06:58.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.858 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:06:58.858 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:06:58.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:06:58.859 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:06:58.859 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:06:58.859 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:06:58.859 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:06:58.859 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:06:58.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:06:58.860 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:06:58.860 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:06:58.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 10 librados: watch_flush enter 2023-10-30T12:06:58.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 10 librados: watch_flush exit 2023-10-30T12:06:58.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 20 client.45005.objecter shutdown clearing up homeless session... 2023-10-30T12:06:58.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 10 client.45005.objecter successfully canceled tick 2023-10-30T12:06:58.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04cc0392f0 msgr2=0x7f04cc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04cc0392f0 0x7f04cc03b7b0 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7f04ec12a910 tx=0x7f04e4009040 comp rx=0 tx=0).stop 2023-10-30T12:06:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 10 monclient: shutdown 2023-10-30T12:06:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 msgr2=0x7f04ec128e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec128e90 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f04d400ec10 tx=0x7f04d400c520 comp rx=0 tx=0).stop 2023-10-30T12:06:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04ead9d640 1 -- 172.21.15.50:0/2334246362 reap_dead start 2023-10-30T12:06:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 shutdown_connections 2023-10-30T12:06:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 --2- 172.21.15.50:0/2334246362 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ec097fe0 0x7f04ec128e90 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 >> 172.21.15.50:0/2334246362 conn(0x7f04ec0a3980 msgr2=0x7f04ec095a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 shutdown_connections 2023-10-30T12:06:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 -- 172.21.15.50:0/2334246362 wait complete. 2023-10-30T12:06:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:58.781+0000 7f04f0896640 1 librados: shutdown 2023-10-30T12:06:58.865 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:06:58.865 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-10-30T12:06:58.865 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:06:58.865 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-10-30T12:06:58.866 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:06:58.866 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:06:58.866 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:06:58.880 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:06:58.880 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:06:58.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:06:59.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.023+0000 7f7a65610640 10 monclient: get_monmap_and_config 2023-10-30T12:06:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.023+0000 7f7a65610640 10 monclient: build_initial_monmap 2023-10-30T12:06:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.023+0000 7f7a65610640 10 monclient: monmap: 2023-10-30T12:06:59.025 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:06:59.024664+0000 2023-10-30T12:06:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:06:59.024664+0000 2023-10-30T12:06:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:06:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:06:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:06:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:06:59.027 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:06:59.027 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 1 Processor -- start 2023-10-30T12:06:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 1 -- start start 2023-10-30T12:06:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient: init 2023-10-30T12:06:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:59.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-10-30T12:06:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a600a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:06:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a60099a40 0x7f7a6013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient(hunting): picked mon.noname-a con 0x7f7a60099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:06:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a60132db0 addr v1:172.21.15.50:6789/0 2023-10-30T12:06:59.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a600a4f40 con 0x7f7a60132db0 2023-10-30T12:06:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a600a4150 con 0x7f7a600a4b60 2023-10-30T12:06:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a600a3d70 con 0x7f7a60099a40 2023-10-30T12:06:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a65610640 10 monclient(hunting): authenticate will time out at 1264.207711s 2023-10-30T12:06:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a60099a40 0x7f7a6013aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a60099a40 0x7f7a6013aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40982/0 (socket says 172.21.15.50:40982) 2023-10-30T12:06:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 1 -- 172.21.15.50:0/2751248395 learned_addr learned my addr 172.21.15.50:0/2751248395 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 10 monclient(hunting): get_auth_request con 0x7f7a60099a40 auth_method 0 2023-10-30T12:06:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.024+0000 7f7a5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 10 monclient(hunting): handle_auth_done global_id 104166 payload 274 2023-10-30T12:06:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 1 -- 172.21.15.50:0/2751248395 >> v1:172.21.15.118:6789/0 conn(0x7f7a600a4b60 legacy=0x7f7a601326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 1 -- 172.21.15.50:0/2751248395 >> v1:172.21.15.50:6789/0 conn(0x7f7a60132db0 legacy=0x7f7a6013cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:06:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 1 monclient: found mon.noname-a 2023-10-30T12:06:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 1 -- 172.21.15.50:0/2751248395 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6009a440 con 0x7f7a60099a40 2023-10-30T12:06:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.025+0000 7f7a5e7fc640 1 --2- 172.21.15.50:0/2751248395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a60099a40 0x7f7a6013aa80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009870 tx=0x7f7a5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fe95a6a22ddb07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 1 -- 172.21.15.50:0/2751248395 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54031870 con 0x7f7a60099a40 2023-10-30T12:06:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: dump: 2023-10-30T12:06:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:59.043 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: _finish_auth 0 2023-10-30T12:06:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:06:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 5 monclient: authenticate success, global_id 104166 2023-10-30T12:06:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 1 -- 172.21.15.50:0/2751248395 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a540319d0 con 0x7f7a60099a40 2023-10-30T12:06:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 1 -- 172.21.15.50:0/2751248395 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54031d60 con 0x7f7a60099a40 2023-10-30T12:06:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a5dffb640 10 monclient: dump: 2023-10-30T12:06:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:59.047 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 10 monclient: get_monmap_and_config success 2023-10-30T12:06:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 4 set_mon_vals no callback set 2023-10-30T12:06:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 10 monclient: shutdown 2023-10-30T12:06:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 1 -- 172.21.15.50:0/2751248395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a60099a40 msgr2=0x7f7a6013aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 1 --2- 172.21.15.50:0/2751248395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a60099a40 0x7f7a6013aa80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009870 tx=0x7f7a5402b910 comp rx=0 tx=0).stop 2023-10-30T12:06:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 1 -- 172.21.15.50:0/2751248395 shutdown_connections 2023-10-30T12:06:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 1 --2- 172.21.15.50:0/2751248395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a60099a40 0x7f7a6013aa80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 1 -- 172.21.15.50:0/2751248395 >> 172.21.15.50:0/2751248395 conn(0x7f7a600a8060 msgr2=0x7f7a600a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 1 -- 172.21.15.50:0/2751248395 shutdown_connections 2023-10-30T12:06:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.026+0000 7f7a65610640 1 -- 172.21.15.50:0/2751248395 wait complete. 2023-10-30T12:06:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 10 monclient: build_initial_monmap 2023-10-30T12:06:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 10 monclient: monmap: 2023-10-30T12:06:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:06:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:06:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:06:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:06:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:06:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:06:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:06:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:06:59.051 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 1 librados: starting msgr at 2023-10-30T12:06:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 1 librados: starting objecter 2023-10-30T12:06:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 1 Processor -- start 2023-10-30T12:06:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 1 -- start start 2023-10-30T12:06:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 1 librados: setting wanted keys 2023-10-30T12:06:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 1 librados: calling monclient init 2023-10-30T12:06:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.027+0000 7f7a65610640 10 monclient: init 2023-10-30T12:06:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient: _reopen_session rank -1 2023-10-30T12:06:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:06:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a600a4b60 0x7f7a60136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): picked mon.noname-a con 0x7f7a600a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:06:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a60132db0 0x7f7a60136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): picked mon.noname-c con 0x7f7a60132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:06:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a601372d0 0x7f7a601ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): picked mon.noname-b con 0x7f7a601372d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:06:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:59.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60139630 con 0x7f7a600a4b60 2023-10-30T12:06:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a601384c0 con 0x7f7a601372d0 2023-10-30T12:06:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): start opening mon connection 2023-10-30T12:06:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a600a18d0 con 0x7f7a60132db0 2023-10-30T12:06:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): _renew_subs 2023-10-30T12:06:59.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient(hunting): authenticate will time out at 1264.211147s 2023-10-30T12:06:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5e7fc640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a60132db0 msgr2=0x7f7a60136d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:06:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5e7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a60132db0 0x7f7a60136d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:06:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a600a4b60 0x7f7a60136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a600a4b60 0x7f7a60136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40988/0 (socket says 172.21.15.50:40988) 2023-10-30T12:06:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 -- 172.21.15.50:0/3526621487 learned_addr learned my addr 172.21.15.50:0/3526621487 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:06:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): get_auth_request con 0x7f7a600a4b60 auth_method 0 2023-10-30T12:06:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:06:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:06:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:06:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:06:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:06:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:06:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5f7fe640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a601372d0 0x7f7a601ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient(hunting): handle_auth_done global_id 104172 payload 824 2023-10-30T12:06:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 -- 172.21.15.50:0/3526621487 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a60132db0 msgr2=0x7f7a60136d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:06:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a60132db0 0x7f7a60136d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 -- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a601372d0 msgr2=0x7f7a601ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a601372d0 0x7f7a601ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient: _finish_hunting 0 2023-10-30T12:06:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 monclient: found mon.noname-a 2023-10-30T12:06:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:06:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:06:59.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 -- 172.21.15.50:0/3526621487 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a540094d0 con 0x7f7a600a4b60 2023-10-30T12:06:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5f7fe640 10 monclient: get_auth_request con 0x7f7a601372d0 auth_method 0 2023-10-30T12:06:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5f7fe640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a601372d0 0x7f7a601ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:06:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5effd640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a600a4b60 0x7f7a60136850 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000c450 tx=0x7f7a5000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 client.?.objecter ms_handle_connect 0x7f7a600a4b60 2023-10-30T12:06:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:06:59.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 1 -- 172.21.15.50:0/3526621487 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a500052d0 con 0x7f7a600a4b60 2023-10-30T12:06:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:06:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: dump: 2023-10-30T12:06:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:59.061 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:59.063 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: _finish_auth 0 2023-10-30T12:06:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: _check_auth_tickets 2023-10-30T12:06:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:06:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 5 monclient: authenticate success, global_id 104172 2023-10-30T12:06:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 1 -- 172.21.15.50:0/3526621487 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a50005430 con 0x7f7a600a4b60 2023-10-30T12:06:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:06:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 1 -- 172.21.15.50:0/3526621487 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50012870 con 0x7f7a600a4b60 2023-10-30T12:06:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient: _renew_subs 2023-10-30T12:06:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6013db50 con 0x7f7a600a4b60 2023-10-30T12:06:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a5d7fa640 4 set_mon_vals no callback set 2023-10-30T12:06:59.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:06:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:06:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a477fe640 10 monclient: dump: 2023-10-30T12:06:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:06:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:06:59.067 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:06:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:06:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:06:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:06:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:06:59.068 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:06:59.069 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:06:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 client.104172.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:06:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient: _renew_subs 2023-10-30T12:06:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a60137b00 con 0x7f7a600a4b60 2023-10-30T12:06:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.028+0000 7f7a65610640 1 librados: init done 2023-10-30T12:06:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a477fe640 1 -- 172.21.15.50:0/3526621487 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7a500129d0 con 0x7f7a600a4b60 2023-10-30T12:06:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a477fe640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a4c0392a0 0x7f7a4c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:06:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a65610640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:59.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a65610640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a65610640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a477fe640 1 -- 172.21.15.50:0/3526621487 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f7a5004aba0 con 0x7f7a600a4b60 2023-10-30T12:06:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a477fe640 10 client.104172.objecter ms_dispatch 0x7f7a60000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:06:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a477fe640 3 client.104172.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:06:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a477fe640 3 client.104172.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:06:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.029+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6004e160 con 0x7f7a600a4b60 2023-10-30T12:06:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.030+0000 7f7a477fe640 20 client.104172.objecter dump_active .. 0 homeless 2023-10-30T12:06:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.030+0000 7f7a5e7fc640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a4c0392a0 0x7f7a4c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:06:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.030+0000 7f7a5e7fc640 10 monclient: get_auth_request con 0x7f7a4c0392a0 auth_method 0 2023-10-30T12:06:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.030+0000 7f7a5e7fc640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a4c0392a0 0x7f7a4c03b760 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7f7a54036000 tx=0x7f7a54008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:06:59.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.030+0000 7f7a477fe640 10 client.104172.objecter ms_handle_connect 0x7f7a4c0392a0 2023-10-30T12:06:59.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.034+0000 7f7a477fe640 1 -- 172.21.15.50:0/3526621487 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a50003b10 con 0x7f7a600a4b60 2023-10-30T12:06:59.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.034+0000 7f7a477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:06:59.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.034+0000 7f7a477fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:06:59.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.277+0000 7f7a65610640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:06:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.277+0000 7f7a65610640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.277+0000 7f7a65610640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:06:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.277+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7a601376e0 con 0x7f7a600a4b60 2023-10-30T12:06:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.277+0000 7f7a477fe640 1 -- 172.21.15.50:0/3526621487 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f7a50019af0 con 0x7f7a600a4b60 2023-10-30T12:06:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.277+0000 7f7a477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:06:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.277+0000 7f7a477fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:06:59.280 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:06:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 10 librados: watch_flush enter 2023-10-30T12:06:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 10 librados: watch_flush exit 2023-10-30T12:06:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 20 client.104172.objecter shutdown clearing up homeless session... 2023-10-30T12:06:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 10 client.104172.objecter successfully canceled tick 2023-10-30T12:06:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a4c0392a0 msgr2=0x7f7a4c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:59.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a4c0392a0 0x7f7a4c03b760 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7f7a54036000 tx=0x7f7a54008030 comp rx=0 tx=0).stop 2023-10-30T12:06:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 10 monclient: shutdown 2023-10-30T12:06:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:06:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a600a4b60 msgr2=0x7f7a60136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:06:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a600a4b60 0x7f7a60136850 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000c450 tx=0x7f7a5000c910 comp rx=0 tx=0).stop 2023-10-30T12:06:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 shutdown_connections 2023-10-30T12:06:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7a4c0392a0 0x7f7a4c03b760 unknown :-1 s=CLOSED pgs=4005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a601372d0 0x7f7a601ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a60132db0 0x7f7a60136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 --2- 172.21.15.50:0/3526621487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a600a4b60 0x7f7a60136850 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:06:59.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 >> 172.21.15.50:0/3526621487 conn(0x7f7a600a8060 msgr2=0x7f7a600a5750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:06:59.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 shutdown_connections 2023-10-30T12:06:59.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 -- 172.21.15.50:0/3526621487 wait complete. 2023-10-30T12:06:59.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:06:59.279+0000 7f7a65610640 1 librados: shutdown 2023-10-30T12:07:02.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:02.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.407+0000 7f5963390640 10 monclient: get_monmap_and_config 2023-10-30T12:07:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.407+0000 7f5963390640 10 monclient: build_initial_monmap 2023-10-30T12:07:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.407+0000 7f5963390640 10 monclient: monmap: 2023-10-30T12:07:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:02.408470+0000 2023-10-30T12:07:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:02.408470+0000 2023-10-30T12:07:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:02.412 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:02.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.407+0000 7f5963390640 1 Processor -- start 2023-10-30T12:07:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 1 -- start start 2023-10-30T12:07:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient: init 2023-10-30T12:07:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-10-30T12:07:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f595c0707d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f595c0712c0 0x7f595c09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient(hunting): picked mon.noname-c con 0x7f595c0712c0 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c13cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:02.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient(hunting): picked mon.noname-b con 0x7f595c13aba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595c09ee40 con 0x7f595c0707d0 2023-10-30T12:07:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595c09f4f0 con 0x7f595c0712c0 2023-10-30T12:07:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c09f7d0 con 0x7f595c13aba0 2023-10-30T12:07:02.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:02.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f5963390640 10 monclient(hunting): authenticate will time out at 1267.591720s 2023-10-30T12:07:02.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f596192d640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 msgr2=0x7f595c13cf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:02.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.408+0000 7f596192d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c13cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:02.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f595c0712c0 0x7f595c09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:02.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596112c640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f595c0707d0 0x7f595c070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:34536/0 (socket says 172.21.15.50:34536) 2023-10-30T12:07:02.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596112c640 1 -- 172.21.15.50:0/1624366131 learned_addr learned my addr 172.21.15.50:0/1624366131 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:02.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): get_auth_request con 0x7f595c0712c0 auth_method 0 2023-10-30T12:07:02.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:02.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:02.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:02.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:02.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:02.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:02.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient(hunting): handle_auth_done global_id 94336 payload 274 2023-10-30T12:07:02.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 1 -- 172.21.15.50:0/1624366131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 msgr2=0x7f595c13cf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:02.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 1 --2- 172.21.15.50:0/1624366131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c13cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 1 -- 172.21.15.50:0/1624366131 >> v1:172.21.15.118:6789/0 conn(0x7f595c0707d0 legacy=0x7f595c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:02.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient: _finish_hunting 0 2023-10-30T12:07:02.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 1 monclient: found mon.noname-c 2023-10-30T12:07:02.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:02.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:07:02.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.409+0000 7f596092b640 1 -- 172.21.15.50:0/1624366131 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c09fab0 con 0x7f595c0712c0 2023-10-30T12:07:02.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f596092b640 1 --2- 172.21.15.50:0/1624366131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c09e7e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f594c0097a0 tx=0x7f594c02b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9dd52679b2827f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:02.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 1 -- 172.21.15.50:0/1624366131 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c031710 con 0x7f595c0712c0 2023-10-30T12:07:02.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:02.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:02.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: dump: 2023-10-30T12:07:02.427 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:02.427 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:02.428 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.428 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.428 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:02.428 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:02.428 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:02.429 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:02.429 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:02.429 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:02.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: _finish_auth 0 2023-10-30T12:07:02.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:02.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:02.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 1 -- 172.21.15.50:0/1624366131 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594c02f050 con 0x7f595c0712c0 2023-10-30T12:07:02.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:02.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5963390640 5 monclient: authenticate success, global_id 94336 2023-10-30T12:07:02.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5963390640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:02.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5963390640 4 set_mon_vals no callback set 2023-10-30T12:07:02.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 1 -- 172.21.15.50:0/1624366131 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c031a90 con 0x7f595c0712c0 2023-10-30T12:07:02.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:02.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:02.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.410+0000 7f5953fff640 10 monclient: dump: 2023-10-30T12:07:02.431 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:02.432 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:02.432 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.432 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.432 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:02.432 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:02.432 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:02.433 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:02.433 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:02.433 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:02.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 10 monclient: shutdown 2023-10-30T12:07:02.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:02.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 -- 172.21.15.50:0/1624366131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 msgr2=0x7f595c09e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:02.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 --2- 172.21.15.50:0/1624366131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c09e7e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f594c0097a0 tx=0x7f594c02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 -- 172.21.15.50:0/1624366131 shutdown_connections 2023-10-30T12:07:02.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 --2- 172.21.15.50:0/1624366131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c13cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 --2- 172.21.15.50:0/1624366131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c09e7e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 -- 172.21.15.50:0/1624366131 >> 172.21.15.50:0/1624366131 conn(0x7f595c0a47f0 msgr2=0x7f595c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:02.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 -- 172.21.15.50:0/1624366131 shutdown_connections 2023-10-30T12:07:02.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 1 -- 172.21.15.50:0/1624366131 wait complete. 2023-10-30T12:07:02.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 10 monclient: build_initial_monmap 2023-10-30T12:07:02.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.411+0000 7f5963390640 10 monclient: monmap: 2023-10-30T12:07:02.435 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:02.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:02.436 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:02.436 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:02.436 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:02.436 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:02.437 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:02.437 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:02.437 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:02.437 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:02.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 librados: starting msgr at 2023-10-30T12:07:02.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 librados: starting objecter 2023-10-30T12:07:02.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 Processor -- start 2023-10-30T12:07:02.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 -- start start 2023-10-30T12:07:02.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 librados: setting wanted keys 2023-10-30T12:07:02.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 librados: calling monclient init 2023-10-30T12:07:02.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient: init 2023-10-30T12:07:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0707d0 0x7f595c1328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): picked mon.noname-a con 0x7f595c0707d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c132de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): picked mon.noname-b con 0x7f595c0712c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c11b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): picked mon.noname-c con 0x7f595c13aba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c11ec90 con 0x7f595c0707d0 2023-10-30T12:07:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c1cb060 con 0x7f595c0712c0 2023-10-30T12:07:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c09d6a0 con 0x7f595c13aba0 2023-10-30T12:07:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f5963390640 10 monclient(hunting): authenticate will time out at 1267.595683s 2023-10-30T12:07:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596192d640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 msgr2=0x7f595c11b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596192d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c11b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0707d0 0x7f595c1328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0707d0 0x7f595c1328a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41004/0 (socket says 172.21.15.50:41004) 2023-10-30T12:07:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 1 -- 172.21.15.50:0/2048882728 learned_addr learned my addr 172.21.15.50:0/2048882728 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 10 monclient(hunting): get_auth_request con 0x7f595c0707d0 auth_method 0 2023-10-30T12:07:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596092b640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596092b640 10 monclient(hunting): get_auth_request con 0x7f595c0712c0 auth_method 0 2023-10-30T12:07:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596092b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596092b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596092b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:02.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596112c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:02.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596092b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:02.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.412+0000 7f596092b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:02.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:02.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 10 monclient(hunting): handle_auth_done global_id 94339 payload 824 2023-10-30T12:07:02.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 1 -- 172.21.15.50:0/2048882728 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 msgr2=0x7f595c11b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:02.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c11b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 1 -- 172.21.15.50:0/2048882728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0707d0 msgr2=0x7f595c1328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:02.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0707d0 0x7f595c1328a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 10 monclient: _finish_hunting 0 2023-10-30T12:07:02.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 1 monclient: found mon.noname-b 2023-10-30T12:07:02.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:02.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:07:02.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 1 -- 172.21.15.50:0/2048882728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c009400 con 0x7f595c0712c0 2023-10-30T12:07:02.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f596092b640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c132de0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f595c0661a0 tx=0x7f594c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:02.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 client.?.objecter ms_handle_connect 0x7f595c0712c0 2023-10-30T12:07:02.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:02.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 1 -- 172.21.15.50:0/2048882728 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c033a10 con 0x7f595c0712c0 2023-10-30T12:07:02.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:02.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:07:02.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: dump: 2023-10-30T12:07:02.449 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:02.449 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:02.449 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.449 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.449 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:02.450 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:02.450 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:02.450 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:02.450 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:02.450 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:02.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: _finish_auth 0 2023-10-30T12:07:02.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:07:02.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:02.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 1 -- 172.21.15.50:0/2048882728 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594c033b70 con 0x7f595c0712c0 2023-10-30T12:07:02.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 5 monclient: authenticate success, global_id 94339 2023-10-30T12:07:02.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 10 monclient: _renew_subs 2023-10-30T12:07:02.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:07:02.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c1345f0 con 0x7f595c0712c0 2023-10-30T12:07:02.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:02.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 1 -- 172.21.15.50:0/2048882728 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c03d470 con 0x7f595c0712c0 2023-10-30T12:07:02.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:02.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:02.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5951ffb640 10 monclient: dump: 2023-10-30T12:07:02.453 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:02.453 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:02.453 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.454 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:02.454 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:02.454 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:02.454 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:02.454 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:02.455 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 10 client.94339.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f59537fe640 4 set_mon_vals no callback set 2023-10-30T12:07:02.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 10 monclient: _renew_subs 2023-10-30T12:07:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:07:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595c11ce80 con 0x7f595c0712c0 2023-10-30T12:07:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.413+0000 7f5963390640 1 librados: init done 2023-10-30T12:07:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.414+0000 7f5963390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:02.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.414+0000 7f5963390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.414+0000 7f5963390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:07:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.414+0000 7f5951ffb640 1 -- 172.21.15.50:0/2048882728 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f594c03a040 con 0x7f595c0712c0 2023-10-30T12:07:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.414+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595c11efc0 con 0x7f595c0712c0 2023-10-30T12:07:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f5951ffb640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59400392a0 0x7f594003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f5951ffb640 1 -- 172.21.15.50:0/2048882728 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f594c06cf80 con 0x7f595c0712c0 2023-10-30T12:07:02.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f5951ffb640 10 client.94339.objecter ms_dispatch 0x7f595c000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:07:02.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f5951ffb640 3 client.94339.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:07:02.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f5951ffb640 3 client.94339.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:07:02.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f5951ffb640 20 client.94339.objecter dump_active .. 0 homeless 2023-10-30T12:07:02.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f596112c640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59400392a0 0x7f594003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:02.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f596112c640 10 monclient: get_auth_request con 0x7f59400392a0 auth_method 0 2023-10-30T12:07:02.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f5951ffb640 10 client.94339.objecter ms_handle_connect 0x7f59400392a0 2023-10-30T12:07:02.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.415+0000 7f596112c640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59400392a0 0x7f594003b760 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7f5958004e40 tx=0x7f5958007f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:02.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.420+0000 7f5951ffb640 1 -- 172.21.15.50:0/2048882728 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f594c03d5d0 con 0x7f595c0712c0 2023-10-30T12:07:02.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.420+0000 7f5951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:02.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.420+0000 7f5951ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.667+0000 7f5963390640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:02.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.667+0000 7f5963390640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:02.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.667+0000 7f5963390640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:07:02.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.667+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f595c04e160 con 0x7f595c0712c0 2023-10-30T12:07:02.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.668+0000 7f5951ffb640 1 -- 172.21.15.50:0/2048882728 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f594c038030 con 0x7f595c0712c0 2023-10-30T12:07:02.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.668+0000 7f5951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:02.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.668+0000 7f5951ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:02.671 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:02.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 10 librados: watch_flush enter 2023-10-30T12:07:02.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 10 librados: watch_flush exit 2023-10-30T12:07:02.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 20 client.94339.objecter shutdown clearing up homeless session... 2023-10-30T12:07:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 10 client.94339.objecter successfully canceled tick 2023-10-30T12:07:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59400392a0 msgr2=0x7f594003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59400392a0 0x7f594003b760 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7f5958004e40 tx=0x7f5958007f10 comp rx=0 tx=0).stop 2023-10-30T12:07:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 10 monclient: shutdown 2023-10-30T12:07:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 msgr2=0x7f595c132de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:02.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c132de0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f595c0661a0 tx=0x7f594c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 shutdown_connections 2023-10-30T12:07:02.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59400392a0 0x7f594003b760 unknown :-1 s=CLOSED pgs=4006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f595c13aba0 0x7f595c11b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f595c0712c0 0x7f595c132de0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 --2- 172.21.15.50:0/2048882728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f595c0707d0 0x7f595c1328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 >> 172.21.15.50:0/2048882728 conn(0x7f595c0a47f0 msgr2=0x7f595c133880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 shutdown_connections 2023-10-30T12:07:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 -- 172.21.15.50:0/2048882728 wait complete. 2023-10-30T12:07:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:02.670+0000 7f5963390640 1 librados: shutdown 2023-10-30T12:07:05.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.798+0000 7f1175c32640 10 monclient: get_monmap_and_config 2023-10-30T12:07:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.798+0000 7f1175c32640 10 monclient: build_initial_monmap 2023-10-30T12:07:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.798+0000 7f1175c32640 10 monclient: monmap: 2023-10-30T12:07:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:05.799337+0000 2023-10-30T12:07:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:05.799337+0000 2023-10-30T12:07:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:05.802 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 1 Processor -- start 2023-10-30T12:07:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 1 -- start start 2023-10-30T12:07:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient: init 2023-10-30T12:07:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:07:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11700a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1170099a40 0x7f117013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient(hunting): picked mon.noname-b con 0x7f1170099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1170132db0 0x7f117013cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient(hunting): picked mon.noname-a con 0x7f1170132db0 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11700a4f40 con 0x7f11700a4b60 2023-10-30T12:07:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11700a3d70 con 0x7f1170132db0 2023-10-30T12:07:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700a4150 con 0x7f1170099a40 2023-10-30T12:07:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f1175c32640 10 monclient(hunting): authenticate will time out at 1270.982616s 2023-10-30T12:07:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116effd640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1170099a40 msgr2=0x7f117013aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1170099a40 0x7f117013aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1170132db0 0x7f117013cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f11700a4b60 0x7f11701326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50874/0 (socket says 172.21.15.50:50874) 2023-10-30T12:07:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1170132db0 0x7f117013cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41006/0 (socket says 172.21.15.50:41006) 2023-10-30T12:07:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116f7fe640 1 -- 172.21.15.50:0/3040766858 learned_addr learned my addr 172.21.15.50:0/3040766858 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 10 monclient(hunting): get_auth_request con 0x7f1170132db0 auth_method 0 2023-10-30T12:07:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.799+0000 7f116ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 10 monclient(hunting): handle_auth_done global_id 104178 payload 274 2023-10-30T12:07:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 1 -- 172.21.15.50:0/3040766858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1170099a40 msgr2=0x7f117013aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 1 --2- 172.21.15.50:0/3040766858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1170099a40 0x7f117013aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 1 -- 172.21.15.50:0/3040766858 >> v1:172.21.15.50:6789/0 conn(0x7f11700a4b60 legacy=0x7f11701326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:07:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 1 monclient: found mon.noname-a 2023-10-30T12:07:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 1 -- 172.21.15.50:0/3040766858 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117009a440 con 0x7f1170132db0 2023-10-30T12:07:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116ffff640 1 --2- 172.21.15.50:0/3040766858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1170132db0 0x7f117013cfd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1168009d90 tx=0x7f116802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850a0146938dc5e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 1 -- 172.21.15.50:0/3040766858 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116802cb60 con 0x7f1170132db0 2023-10-30T12:07:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 10 monclient: dump: 2023-10-30T12:07:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:05.819 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:07:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:07:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f1175c32640 5 monclient: authenticate success, global_id 104178 2023-10-30T12:07:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f1175c32640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 1 -- 172.21.15.50:0/3040766858 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116802ccc0 con 0x7f1170132db0 2023-10-30T12:07:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f1175c32640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f1175c32640 4 set_mon_vals no callback set 2023-10-30T12:07:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 1 -- 172.21.15.50:0/3040766858 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168034660 con 0x7f1170132db0 2023-10-30T12:07:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.800+0000 7f116e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f116e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f116e7fc640 10 monclient: dump: 2023-10-30T12:07:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:05.824 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 10 monclient: shutdown 2023-10-30T12:07:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 -- 172.21.15.50:0/3040766858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1170132db0 msgr2=0x7f117013cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 --2- 172.21.15.50:0/3040766858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1170132db0 0x7f117013cfd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1168009d90 tx=0x7f116802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:07:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 -- 172.21.15.50:0/3040766858 shutdown_connections 2023-10-30T12:07:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 --2- 172.21.15.50:0/3040766858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1170132db0 0x7f117013cfd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 --2- 172.21.15.50:0/3040766858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1170099a40 0x7f117013aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 -- 172.21.15.50:0/3040766858 >> 172.21.15.50:0/3040766858 conn(0x7f11700a8060 msgr2=0x7f11700a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 -- 172.21.15.50:0/3040766858 shutdown_connections 2023-10-30T12:07:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.801+0000 7f1175c32640 1 -- 172.21.15.50:0/3040766858 wait complete. 2023-10-30T12:07:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 10 monclient: build_initial_monmap 2023-10-30T12:07:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 10 monclient: monmap: 2023-10-30T12:07:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:05.830 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 1 librados: starting msgr at 2023-10-30T12:07:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 1 librados: starting objecter 2023-10-30T12:07:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 1 Processor -- start 2023-10-30T12:07:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 1 -- start start 2023-10-30T12:07:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 1 librados: setting wanted keys 2023-10-30T12:07:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 1 librados: calling monclient init 2023-10-30T12:07:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.802+0000 7f1175c32640 10 monclient: init 2023-10-30T12:07:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11700a4b60 0x7f1170136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): picked mon.noname-a con 0x7f11700a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1170132db0 0x7f1170136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): picked mon.noname-b con 0x7f1170132db0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11701372d0 0x7f11701ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): picked mon.noname-c con 0x7f11701372d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170139600 con 0x7f11700a4b60 2023-10-30T12:07:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170138540 con 0x7f1170132db0 2023-10-30T12:07:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700a18d0 con 0x7f11701372d0 2023-10-30T12:07:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1175c32640 10 monclient(hunting): authenticate will time out at 1270.986290s 2023-10-30T12:07:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116ffff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11701372d0 msgr2=0x7f11701ca620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11701372d0 0x7f11701ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11700a4b60 0x7f1170136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11700a4b60 0x7f1170136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41020/0 (socket says 172.21.15.50:41020) 2023-10-30T12:07:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 -- 172.21.15.50:0/4038545593 learned_addr learned my addr 172.21.15.50:0/4038545593 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): get_auth_request con 0x7f11700a4b60 auth_method 0 2023-10-30T12:07:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116effd640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1170132db0 0x7f1170136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116effd640 10 monclient(hunting): get_auth_request con 0x7f1170132db0 auth_method 0 2023-10-30T12:07:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient(hunting): handle_auth_done global_id 104181 payload 824 2023-10-30T12:07:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 -- 172.21.15.50:0/4038545593 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11701372d0 msgr2=0x7f11701ca620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11701372d0 0x7f11701ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 -- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1170132db0 msgr2=0x7f1170136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1170132db0 0x7f1170136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:07:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:07:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 -- 172.21.15.50:0/4038545593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168009a70 con 0x7f11700a4b60 2023-10-30T12:07:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f116f7fe640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11700a4b60 0x7f1170136850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f116000c4a0 tx=0x7f116000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1157fff640 10 client.?.objecter ms_handle_connect 0x7f11700a4b60 2023-10-30T12:07:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.803+0000 7f1157fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 1 -- 172.21.15.50:0/4038545593 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160005ce0 con 0x7f11700a4b60 2023-10-30T12:07:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: dump: 2023-10-30T12:07:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:05.843 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: _finish_auth 0 2023-10-30T12:07:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 5 monclient: authenticate success, global_id 104181 2023-10-30T12:07:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 1 -- 172.21.15.50:0/4038545593 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1160005e40 con 0x7f11700a4b60 2023-10-30T12:07:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 1 -- 172.21.15.50:0/4038545593 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160007270 con 0x7f11700a4b60 2023-10-30T12:07:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 10 monclient: _renew_subs 2023-10-30T12:07:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117013c8d0 con 0x7f11700a4b60 2023-10-30T12:07:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f116dffb640 4 set_mon_vals no callback set 2023-10-30T12:07:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 10 monclient: dump: 2023-10-30T12:07:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:05.848 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 10 client.104181.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 10 monclient: _renew_subs 2023-10-30T12:07:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1170138930 con 0x7f11700a4b60 2023-10-30T12:07:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1175c32640 1 librados: init done 2023-10-30T12:07:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.804+0000 7f1157fff640 1 -- 172.21.15.50:0/4038545593 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f11600073d0 con 0x7f11700a4b60 2023-10-30T12:07:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1175c32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1175c32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1175c32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117004e160 con 0x7f11700a4b60 2023-10-30T12:07:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1157fff640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f115c0392a0 0x7f115c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1157fff640 1 -- 172.21.15.50:0/4038545593 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f116004a400 con 0x7f11700a4b60 2023-10-30T12:07:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1157fff640 10 client.104181.objecter ms_dispatch 0x7f1170000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:07:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1157fff640 3 client.104181.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:07:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1157fff640 3 client.104181.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:07:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f1157fff640 20 client.104181.objecter dump_active .. 0 homeless 2023-10-30T12:07:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f116effd640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f115c0392a0 0x7f115c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.805+0000 7f116effd640 10 monclient: get_auth_request con 0x7f115c0392a0 auth_method 0 2023-10-30T12:07:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.806+0000 7f1157fff640 10 client.104181.objecter ms_handle_connect 0x7f115c0392a0 2023-10-30T12:07:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.806+0000 7f116effd640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f115c0392a0 0x7f115c03b760 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7f11700a0db0 tx=0x7f1164008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.809+0000 7f1157fff640 1 -- 172.21.15.50:0/4038545593 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f116001b3b0 con 0x7f11700a4b60 2023-10-30T12:07:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.809+0000 7f1157fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:05.809+0000 7f1157fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.052+0000 7f1175c32640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.052+0000 7f1175c32640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.052+0000 7f1175c32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.052+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1170139800 con 0x7f11700a4b60 2023-10-30T12:07:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.052+0000 7f1157fff640 1 -- 172.21.15.50:0/4038545593 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f1160017080 con 0x7f11700a4b60 2023-10-30T12:07:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.052+0000 7f1157fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.052+0000 7f1157fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:06.055 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 10 librados: watch_flush enter 2023-10-30T12:07:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 10 librados: watch_flush exit 2023-10-30T12:07:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 20 client.104181.objecter shutdown clearing up homeless session... 2023-10-30T12:07:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 10 client.104181.objecter successfully canceled tick 2023-10-30T12:07:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f115c0392a0 msgr2=0x7f115c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f115c0392a0 0x7f115c03b760 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7f11700a0db0 tx=0x7f1164008040 comp rx=0 tx=0).stop 2023-10-30T12:07:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 10 monclient: shutdown 2023-10-30T12:07:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11700a4b60 msgr2=0x7f1170136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11700a4b60 0x7f1170136850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f116000c4a0 tx=0x7f116000c960 comp rx=0 tx=0).stop 2023-10-30T12:07:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 shutdown_connections 2023-10-30T12:07:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f115c0392a0 0x7f115c03b760 unknown :-1 s=CLOSED pgs=4007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11701372d0 0x7f11701ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1170132db0 0x7f1170136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 --2- 172.21.15.50:0/4038545593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11700a4b60 0x7f1170136850 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.054+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 >> 172.21.15.50:0/4038545593 conn(0x7f11700a8060 msgr2=0x7f11700a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.055+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 shutdown_connections 2023-10-30T12:07:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.055+0000 7f1175c32640 1 -- 172.21.15.50:0/4038545593 wait complete. 2023-10-30T12:07:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:06.055+0000 7f1175c32640 1 librados: shutdown 2023-10-30T12:07:09.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:09.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.184+0000 7f8e95bb7640 10 monclient: get_monmap_and_config 2023-10-30T12:07:09.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.184+0000 7f8e95bb7640 10 monclient: build_initial_monmap 2023-10-30T12:07:09.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.184+0000 7f8e95bb7640 10 monclient: monmap: 2023-10-30T12:07:09.186 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:09.187 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:09.187 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:09.185628+0000 2023-10-30T12:07:09.187 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:09.185628+0000 2023-10-30T12:07:09.187 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:09.187 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:09.188 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:09.188 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:09.188 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:09.188 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:09.189 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:09.189 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:09.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 Processor -- start 2023-10-30T12:07:09.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 -- start start 2023-10-30T12:07:09.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient: init 2023-10-30T12:07:09.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:09.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:07:09.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:09.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): picked mon.noname-a con 0x7f8e900707d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:09.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e90070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:09.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): picked mon.noname-b con 0x7f8e90070bb0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:09.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:09.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): picked mon.noname-c con 0x7f8e901345f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:09.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:09.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9009ee40 con 0x7f8e900707d0 2023-10-30T12:07:09.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:09.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9009f4f0 con 0x7f8e901345f0 2023-10-30T12:07:09.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:09.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9009f7d0 con 0x7f8e90070bb0 2023-10-30T12:07:09.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:09.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e95bb7640 10 monclient(hunting): authenticate will time out at 1274.368924s 2023-10-30T12:07:09.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e8effd640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 msgr2=0x7f8e90070ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:09.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.185+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e90070ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:09.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:09.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55778/0 (socket says 172.21.15.50:55778) 2023-10-30T12:07:09.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 learned_addr learned my addr 172.21.15.50:0/1491602558 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:09.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f8e900707d0 auth_method 0 2023-10-30T12:07:09.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:09.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:09.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:09.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:07:09.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:09.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:09.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:09.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:09.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request con 0x7f8e901345f0 auth_method 0 2023-10-30T12:07:09.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:09.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:09.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:09.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:07:09.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:09.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:09.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.186+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:09.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.386+0000 7f8e8effd640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 msgr2=0x7f8e90070ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.386+0000 7f8e8effd640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e90070ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:07:09.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:09.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f8e900707d0 auth_method 0 2023-10-30T12:07:09.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request con 0x7f8e901345f0 auth_method 0 2023-10-30T12:07:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:07:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:07:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:07:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.387+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:07:09.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.787+0000 7f8e8effd640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 msgr2=0x7f8e90070ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:09.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.787+0000 7f8e8effd640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e90070ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:07:09.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:09.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f8e900707d0 auth_method 0 2023-10-30T12:07:09.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:07:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:07:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request con 0x7f8e901345f0 auth_method 0 2023-10-30T12:07:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:07:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:09.788+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:07:10.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.588+0000 7f8e8effd640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 msgr2=0x7f8e90070ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:10.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.588+0000 7f8e8effd640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e90070ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:07:10.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:10.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f8e900707d0 auth_method 0 2023-10-30T12:07:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:07:10.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:07:10.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request con 0x7f8e901345f0 auth_method 0 2023-10-30T12:07:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.589+0000 7f8e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.590+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:07:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.590+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.590+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:10.590+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:07:12.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 10 monclient(hunting): tick 2023-10-30T12:07:12.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:07:12.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:07:12.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:07:12.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 msgr2=0x7f8e90070ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:12.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e90070ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:12.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 msgr2=0x7f8e90134a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:12.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.118:3300/0 conn(0x7f8e901345f0 0x7f8e90134a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:12.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 -- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 msgr2=0x7f8e9009e7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:12.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 --2- 172.21.15.50:0/1491602558 >> v2:172.21.15.50:3300/0 conn(0x7f8e900707d0 0x7f8e9009e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:12.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 10 monclient: _add_conns ranks=[1,0] 2023-10-30T12:07:12.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e7c0036b0 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:12.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e7c0073e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:12.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 -- 172.21.15.50:0/1491602558 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9009fab0 con 0x7f8e7c0073e0 2023-10-30T12:07:12.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 1 -- 172.21.15.50:0/1491602558 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c00c160 con 0x7f8e7c0036b0 2023-10-30T12:07:12.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:12.186+0000 7f8e8d7fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.441+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248125479 0 0) 0x7f8e9009fab0 con 0x7f8e7c0073e0 2023-10-30T12:07:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.441+0000 7f8e8e7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:07:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.441+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.441+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.441+0000 7f8e8e7fc640 10 monclient(hunting): my global_id is 104184 2023-10-30T12:07:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.441+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e78003520 con 0x7f8e7c0073e0 2023-10-30T12:07:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486032761 0 0) 0x7f8e7c00c160 con 0x7f8e7c0036b0 2023-10-30T12:07:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:07:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): my global_id is 94345 2023-10-30T12:07:13.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9009fab0 con 0x7f8e7c0036b0 2023-10-30T12:07:13.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1559363255 0 0) 0x7f8e78003520 con 0x7f8e7c0073e0 2023-10-30T12:07:13.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 >> v1:172.21.15.118:6789/0 conn(0x7f8e7c0036b0 legacy=0x7f8e7c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:07:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:07:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:07:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c00d340 con 0x7f8e7c0073e0 2023-10-30T12:07:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:07:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:07:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f8e80004340 con 0x7f8e7c0073e0 2023-10-30T12:07:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:07:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: dump: 2023-10-30T12:07:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.453 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.453 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.453 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.453 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.453 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:07:13.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:13.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 >> v1:172.21.15.50:6789/0 conn(0x7f8e7c0073e0 legacy=0x7f8e7c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:07:13.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78003b30 0x7f8e78003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): picked mon.a con 0x7f8e78003b30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 reap_dead start 2023-10-30T12:07:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): picked mon.c con 0x7f8e90070bb0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78004e20 0x7f8e7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): picked mon.b con 0x7f8e78004e20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e780055b0 con 0x7f8e78003b30 2023-10-30T12:07:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7800bec0 con 0x7f8e78004e20 2023-10-30T12:07:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7800c000 con 0x7f8e90070bb0 2023-10-30T12:07:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.442+0000 7f8e8e7fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:07:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8effd640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 msgr2=0x7f8e780047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8effd640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e780047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78004e20 0x7f8e7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request con 0x7f8e78004e20 auth_method 0 2023-10-30T12:07:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:07:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient(hunting): handle_auth_done global_id 104184 payload 293 2023-10-30T12:07:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 msgr2=0x7f8e780047c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e780047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78003b30 msgr2=0x7f8e78003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78003b30 0x7f8e78003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:07:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 1 monclient: found mon.b 2023-10-30T12:07:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:07:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 1 -- 172.21.15.50:0/1491602558 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78003520 con 0x7f8e78004e20 2023-10-30T12:07:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient: _finish_auth 0 2023-10-30T12:07:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e95bb7640 5 monclient: authenticate success, global_id 104184 2023-10-30T12:07:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.443+0000 7f8e95bb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78004e20 0x7f8e7800b980 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8e901349d0 tx=0x7f8e780057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0efc1cc757cbf5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88035410 con 0x7f8e78004e20 2023-10-30T12:07:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 10 monclient: dump: 2023-10-30T12:07:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.465 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e95bb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8802f030 con 0x7f8e78004e20 2023-10-30T12:07:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 1 -- 172.21.15.50:0/1491602558 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88035790 con 0x7f8e78004e20 2023-10-30T12:07:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e95bb7640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e95bb7640 4 set_mon_vals no callback set 2023-10-30T12:07:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.444+0000 7f8e8e7fc640 10 monclient: dump: 2023-10-30T12:07:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.468 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 10 monclient: shutdown 2023-10-30T12:07:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1491602558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78004e20 msgr2=0x7f8e7800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78004e20 0x7f8e7800b980 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8e901349d0 tx=0x7f8e780057f0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1491602558 shutdown_connections 2023-10-30T12:07:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78004e20 0x7f8e7800b980 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e90070bb0 0x7f8e780047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1491602558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78003b30 0x7f8e78003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1491602558 >> 172.21.15.50:0/1491602558 conn(0x7f8e900a47f0 msgr2=0x7f8e900a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1491602558 shutdown_connections 2023-10-30T12:07:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.445+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1491602558 wait complete. 2023-10-30T12:07:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.446+0000 7f8e95bb7640 10 monclient: build_initial_monmap 2023-10-30T12:07:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.446+0000 7f8e95bb7640 10 monclient: monmap: 2023-10-30T12:07:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:13.473 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.446+0000 7f8e95bb7640 1 librados: starting msgr at 2023-10-30T12:07:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.446+0000 7f8e95bb7640 1 librados: starting objecter 2023-10-30T12:07:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.446+0000 7f8e95bb7640 1 Processor -- start 2023-10-30T12:07:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 -- start start 2023-10-30T12:07:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 librados: setting wanted keys 2023-10-30T12:07:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 librados: calling monclient init 2023-10-30T12:07:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient: init 2023-10-30T12:07:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:07:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e78003b30 0x7f8e901319d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): picked mon.noname-c con 0x7f8e78003b30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78004e20 0x7f8e90131f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): picked mon.noname-a con 0x7f8e78004e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e90070bb0 0x7f8e9011b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): picked mon.noname-b con 0x7f8e90070bb0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9011ee40 con 0x7f8e78004e20 2023-10-30T12:07:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9011f120 con 0x7f8e90070bb0 2023-10-30T12:07:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9009cde0 con 0x7f8e78003b30 2023-10-30T12:07:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e95bb7640 10 monclient(hunting): authenticate will time out at 1278.630990s 2023-10-30T12:07:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.447+0000 7f8e8f7fe640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e78003b30 msgr2=0x7f8e901319d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e78003b30 0x7f8e901319d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78004e20 0x7f8e90131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78004e20 0x7f8e90131f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55826/0 (socket says 172.21.15.50:55826) 2023-10-30T12:07:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 -- 172.21.15.50:0/1961329226 learned_addr learned my addr 172.21.15.50:0/1961329226 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): get_auth_request con 0x7f8e78004e20 auth_method 0 2023-10-30T12:07:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e90070bb0 0x7f8e9011b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request con 0x7f8e90070bb0 auth_method 0 2023-10-30T12:07:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient(hunting): handle_auth_done global_id 104187 payload 824 2023-10-30T12:07:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 -- 172.21.15.50:0/1961329226 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e78003b30 msgr2=0x7f8e901319d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e78003b30 0x7f8e901319d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 -- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e90070bb0 msgr2=0x7f8e9011b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e90070bb0 0x7f8e9011b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient: _finish_hunting 0 2023-10-30T12:07:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 monclient: found mon.noname-a 2023-10-30T12:07:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 -- 172.21.15.50:0/1961329226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9009d6a0 con 0x7f8e78004e20 2023-10-30T12:07:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e8effd640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78004e20 0x7f8e90131f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000afd0 tx=0x7f8e8000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e77fff640 10 client.?.objecter ms_handle_connect 0x7f8e78004e20 2023-10-30T12:07:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.448+0000 7f8e77fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 1 -- 172.21.15.50:0/1961329226 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80016030 con 0x7f8e78004e20 2023-10-30T12:07:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: dump: 2023-10-30T12:07:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.491 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: _finish_auth 0 2023-10-30T12:07:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 5 monclient: authenticate success, global_id 104187 2023-10-30T12:07:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 10 monclient: _renew_subs 2023-10-30T12:07:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9011d710 con 0x7f8e78004e20 2023-10-30T12:07:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 1 -- 172.21.15.50:0/1961329226 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e80007aa0 con 0x7f8e78004e20 2023-10-30T12:07:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 1 -- 172.21.15.50:0/1961329226 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80007e90 con 0x7f8e78004e20 2023-10-30T12:07:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e77fff640 10 monclient: dump: 2023-10-30T12:07:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.496 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.496 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 10 client.104187.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:13.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 10 monclient: _renew_subs 2023-10-30T12:07:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e901bee10 con 0x7f8e78004e20 2023-10-30T12:07:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e8dffb640 4 set_mon_vals no callback set 2023-10-30T12:07:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.449+0000 7f8e95bb7640 1 librados: init done 2023-10-30T12:07:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e77fff640 1 -- 172.21.15.50:0/1961329226 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8e80019070 con 0x7f8e78004e20 2023-10-30T12:07:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e77fff640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e7c039b00 0x7f8e7c03bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e95bb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e95bb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e95bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e77fff640 1 -- 172.21.15.50:0/1961329226 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f8e80017070 con 0x7f8e78004e20 2023-10-30T12:07:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e77fff640 10 client.104187.objecter ms_dispatch 0x7f8e90000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:07:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e77fff640 3 client.104187.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:07:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e77fff640 3 client.104187.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:07:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9004e160 con 0x7f8e78004e20 2023-10-30T12:07:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e77fff640 20 client.104187.objecter dump_active .. 0 homeless 2023-10-30T12:07:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e7c039b00 0x7f8e7c03bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.450+0000 7f8e8f7fe640 10 monclient: get_auth_request con 0x7f8e7c039b00 auth_method 0 2023-10-30T12:07:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.451+0000 7f8e8f7fe640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e7c039b00 0x7f8e7c03bfc0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7f8e84006b10 tx=0x7f8e84006a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.451+0000 7f8e77fff640 10 client.104187.objecter ms_handle_connect 0x7f8e7c039b00 2023-10-30T12:07:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.456+0000 7f8e77fff640 1 -- 172.21.15.50:0/1961329226 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8e8000f320 con 0x7f8e78004e20 2023-10-30T12:07:13.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.456+0000 7f8e77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:13.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.456+0000 7f8e77fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.701+0000 7f8e95bb7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.701+0000 7f8e95bb7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.701+0000 7f8e95bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.701+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8e9011eac0 con 0x7f8e78004e20 2023-10-30T12:07:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.702+0000 7f8e77fff640 1 -- 172.21.15.50:0/1961329226 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f8e8001abb0 con 0x7f8e78004e20 2023-10-30T12:07:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.702+0000 7f8e77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.702+0000 7f8e77fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:13.705 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 10 librados: watch_flush enter 2023-10-30T12:07:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 10 librados: watch_flush exit 2023-10-30T12:07:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 20 client.104187.objecter shutdown clearing up homeless session... 2023-10-30T12:07:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 10 client.104187.objecter successfully canceled tick 2023-10-30T12:07:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e7c039b00 msgr2=0x7f8e7c03bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e7c039b00 0x7f8e7c03bfc0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7f8e84006b10 tx=0x7f8e84006a40 comp rx=0 tx=0).stop 2023-10-30T12:07:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 10 monclient: shutdown 2023-10-30T12:07:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78004e20 msgr2=0x7f8e90131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78004e20 0x7f8e90131f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000afd0 tx=0x7f8e8000cbe0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 shutdown_connections 2023-10-30T12:07:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e7c039b00 0x7f8e7c03bfc0 unknown :-1 s=CLOSED pgs=4008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e90070bb0 0x7f8e9011b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78004e20 0x7f8e90131f10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 --2- 172.21.15.50:0/1961329226 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e78003b30 0x7f8e901319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.703+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 >> 172.21.15.50:0/1961329226 conn(0x7f8e900a47f0 msgr2=0x7f8e90132df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.704+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 shutdown_connections 2023-10-30T12:07:13.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.704+0000 7f8e95bb7640 1 -- 172.21.15.50:0/1961329226 wait complete. 2023-10-30T12:07:13.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.704+0000 7f8e95bb7640 1 librados: shutdown 2023-10-30T12:07:13.716 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:07:13.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:07:13.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:07:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.820+0000 7f70d47a9640 10 monclient: get_monmap_and_config 2023-10-30T12:07:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.820+0000 7f70d47a9640 10 monclient: build_initial_monmap 2023-10-30T12:07:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.820+0000 7f70d47a9640 10 monclient: monmap: 2023-10-30T12:07:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:13.823 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.823 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:13.821731+0000 2023-10-30T12:07:13.823 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:13.821731+0000 2023-10-30T12:07:13.823 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:13.823 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.824 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:13.824 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:13.824 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:13.824 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:13.824 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:13.825 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 Processor -- start 2023-10-30T12:07:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 -- start start 2023-10-30T12:07:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient: init 2023-10-30T12:07:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-10-30T12:07:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f70cc09bb70 0x7f70cc09bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): picked mon.noname-a con 0x7f70cc09bb70 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f70cc09d3f0 0x7f70cc09c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): picked mon.noname-c con 0x7f70cc09d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc096080 0x7f70cc0964e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): picked mon.noname-b con 0x7f70cc096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70cc098170 con 0x7f70cc09bb70 2023-10-30T12:07:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70cc09c9d0 con 0x7f70cc09d3f0 2023-10-30T12:07:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc09cf60 con 0x7f70cc096080 2023-10-30T12:07:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d47a9640 10 monclient(hunting): authenticate will time out at 1279.004894s 2023-10-30T12:07:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d2d46640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc096080 msgr2=0x7f70cc0964e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d2d46640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc096080 0x7f70cc0964e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.821+0000 7f70d2545640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f70cc09bb70 0x7f70cc09bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f70cc09bb70 0x7f70cc09bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55840/0 (socket says 172.21.15.50:55840) 2023-10-30T12:07:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 -- 172.21.15.50:0/1528168641 learned_addr learned my addr 172.21.15.50:0/1528168641 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): get_auth_request con 0x7f70cc09bb70 auth_method 0 2023-10-30T12:07:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d1d44640 1 --2- 172.21.15.50:0/1528168641 >> v2:172.21.15.118:3300/0 conn(0x7f70cc09d3f0 0x7f70cc09c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient(hunting): handle_auth_done global_id 104190 payload 274 2023-10-30T12:07:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 -- 172.21.15.50:0/1528168641 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc096080 msgr2=0x7f70cc0964e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 --2- 172.21.15.50:0/1528168641 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc096080 0x7f70cc0964e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 -- 172.21.15.50:0/1528168641 >> v2:172.21.15.118:3300/0 conn(0x7f70cc09d3f0 msgr2=0x7f70cc09c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 --2- 172.21.15.50:0/1528168641 >> v2:172.21.15.118:3300/0 conn(0x7f70cc09d3f0 0x7f70cc09c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient: _finish_hunting 0 2023-10-30T12:07:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 monclient: found mon.noname-a 2023-10-30T12:07:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 -- 172.21.15.50:0/1528168641 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70cc097b20 con 0x7f70cc09bb70 2023-10-30T12:07:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d1d44640 10 monclient: get_auth_request con 0x7f70cc09d3f0 auth_method 0 2023-10-30T12:07:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d1d44640 1 --2- 172.21.15.50:0/1528168641 >> v2:172.21.15.118:3300/0 conn(0x7f70cc09d3f0 0x7f70cc09c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:07:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.822+0000 7f70d2545640 1 --2- 172.21.15.50:0/1528168641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc09bb70 0x7f70cc09bf50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f70cc09d840 tx=0x7f70c802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96105457bbe3cee9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 1 -- 172.21.15.50:0/1528168641 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8004230 con 0x7f70cc09bb70 2023-10-30T12:07:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: dump: 2023-10-30T12:07:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.845 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: _finish_auth 0 2023-10-30T12:07:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: _check_auth_tickets 2023-10-30T12:07:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 1 -- 172.21.15.50:0/1528168641 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c8004390 con 0x7f70cc09bb70 2023-10-30T12:07:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 1 -- 172.21.15.50:0/1528168641 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8003810 con 0x7f70cc09bb70 2023-10-30T12:07:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d1543640 10 monclient: dump: 2023-10-30T12:07:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.849 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 5 monclient: authenticate success, global_id 104190 2023-10-30T12:07:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 4 set_mon_vals no callback set 2023-10-30T12:07:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 10 monclient: shutdown 2023-10-30T12:07:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 -- 172.21.15.50:0/1528168641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc09bb70 msgr2=0x7f70cc09bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 --2- 172.21.15.50:0/1528168641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc09bb70 0x7f70cc09bf50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f70cc09d840 tx=0x7f70c802b7f0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 -- 172.21.15.50:0/1528168641 shutdown_connections 2023-10-30T12:07:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 --2- 172.21.15.50:0/1528168641 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc096080 0x7f70cc0964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 --2- 172.21.15.50:0/1528168641 >> v2:172.21.15.118:3300/0 conn(0x7f70cc09d3f0 0x7f70cc09c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 --2- 172.21.15.50:0/1528168641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc09bb70 0x7f70cc09bf50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 -- 172.21.15.50:0/1528168641 >> 172.21.15.50:0/1528168641 conn(0x7f70cc0a3980 msgr2=0x7f70cc0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 -- 172.21.15.50:0/1528168641 shutdown_connections 2023-10-30T12:07:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.823+0000 7f70d47a9640 1 -- 172.21.15.50:0/1528168641 wait complete. 2023-10-30T12:07:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 10 monclient: build_initial_monmap 2023-10-30T12:07:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 10 monclient: monmap: 2023-10-30T12:07:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:13.854 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 1 librados: starting msgr at 2023-10-30T12:07:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 1 librados: starting objecter 2023-10-30T12:07:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 1 Processor -- start 2023-10-30T12:07:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 1 -- start start 2023-10-30T12:07:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 1 librados: setting wanted keys 2023-10-30T12:07:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 1 librados: calling monclient init 2023-10-30T12:07:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.824+0000 7f70d47a9640 10 monclient: init 2023-10-30T12:07:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc096080 0x7f70cc08afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): picked mon.noname-a con 0x7f70cc096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70cc09bb70 0x7f70cc08b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): picked mon.noname-b con 0x7f70cc09bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc09d3f0 0x7f70cc08ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): picked mon.noname-c con 0x7f70cc09d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc1bb7d0 con 0x7f70cc096080 2023-10-30T12:07:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc08d1c0 con 0x7f70cc09bb70 2023-10-30T12:07:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc08d4a0 con 0x7f70cc09d3f0 2023-10-30T12:07:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 10 monclient(hunting): authenticate will time out at 1279.008165s 2023-10-30T12:07:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2d46640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc09d3f0 msgr2=0x7f70cc08ba50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2d46640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc09d3f0 0x7f70cc08ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc096080 0x7f70cc08afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc096080 0x7f70cc08afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55848/0 (socket says 172.21.15.50:55848) 2023-10-30T12:07:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 -- 172.21.15.50:0/3951241977 learned_addr learned my addr 172.21.15.50:0/3951241977 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): get_auth_request con 0x7f70cc096080 auth_method 0 2023-10-30T12:07:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d1d44640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70cc09bb70 0x7f70cc08b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d1d44640 10 monclient(hunting): get_auth_request con 0x7f70cc09bb70 auth_method 0 2023-10-30T12:07:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d1d44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d1d44640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d1d44640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient(hunting): handle_auth_done global_id 104193 payload 824 2023-10-30T12:07:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc09d3f0 msgr2=0x7f70cc08ba50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70cc09d3f0 0x7f70cc08ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70cc09bb70 msgr2=0x7f70cc08b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70cc09bb70 0x7f70cc08b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient: _finish_hunting 0 2023-10-30T12:07:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 monclient: found mon.noname-a 2023-10-30T12:07:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 -- 172.21.15.50:0/3951241977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8009460 con 0x7f70cc096080 2023-10-30T12:07:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d1d44640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70cc09bb70 0x7f70cc08b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:07:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d2545640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc096080 0x7f70cc08afd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70c8036040 tx=0x7f70c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 client.?.objecter ms_handle_connect 0x7f70cc096080 2023-10-30T12:07:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8005da0 con 0x7f70cc096080 2023-10-30T12:07:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: dump: 2023-10-30T12:07:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.866 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:07:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:07:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c8003e60 con 0x7f70cc096080 2023-10-30T12:07:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8006250 con 0x7f70cc096080 2023-10-30T12:07:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d47a9640 5 monclient: authenticate success, global_id 104193 2023-10-30T12:07:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70c2ffd640 10 monclient: dump: 2023-10-30T12:07:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.870 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.825+0000 7f70d0d42640 4 set_mon_vals no callback set 2023-10-30T12:07:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 10 monclient: _renew_subs 2023-10-30T12:07:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70cc0976e0 con 0x7f70cc096080 2023-10-30T12:07:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 10 client.104193.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 10 monclient: _renew_subs 2023-10-30T12:07:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc071490 con 0x7f70cc096080 2023-10-30T12:07:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 1 librados: init done 2023-10-30T12:07:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f70c80063b0 con 0x7f70cc096080 2023-10-30T12:07:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7430 0x7f70cc1b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc073ff0 con 0x7f70cc1b7430 2023-10-30T12:07:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70c2ffd640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f70b00392a0 0x7f70b003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f70c806d4b0 con 0x7f70cc096080 2023-10-30T12:07:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.826+0000 7f70c2ffd640 10 client.104193.objecter ms_dispatch 0x7f70cc000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:07:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 3 client.104193.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:07:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 3 client.104193.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:07:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7430 0x7f70cc1b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient: get_auth_request con 0x7f70cc1b7430 auth_method 0 2023-10-30T12:07:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 20 client.104193.objecter dump_active .. 0 homeless 2023-10-30T12:07:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d2d46640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f70b00392a0 0x7f70b003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d2d46640 10 monclient: get_auth_request con 0x7f70b00392a0 auth_method 0 2023-10-30T12:07:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 10 monclient(hunting): handle_auth_done global_id 104196 payload 824 2023-10-30T12:07:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 1 -- 172.21.15.50:0/3951241977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70cc097560 con 0x7f70cc1b7430 2023-10-30T12:07:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d1d44640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7430 0x7f70cc1b7810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70cc071090 tx=0x7f70bc00f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 client.104193.objecter ms_handle_connect 0x7f70cc1b7430 2023-10-30T12:07:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 client.104193.objecter resend_mon_ops 2023-10-30T12:07:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc004230 con 0x7f70cc1b7430 2023-10-30T12:07:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 monclient: dump: 2023-10-30T12:07:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.879 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 client.104193.objecter ms_handle_connect 0x7f70b00392a0 2023-10-30T12:07:13.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70d2d46640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f70b00392a0 0x7f70b003b760 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 crypto rx=0x7f70c4003d00 tx=0x7f70c4009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f70bc00c030 con 0x7f70cc1b7430 2023-10-30T12:07:13.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:13.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:13.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7430 msgr2=0x7f70cc1b7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7430 0x7f70cc1b7810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70cc071090 tx=0x7f70bc00f970 comp rx=0 tx=0).stop 2023-10-30T12:07:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7430 msgr2=0x7f70cc1b7810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.827+0000 7f70c2ffd640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7430 0x7f70cc1b7810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d47a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d47a9640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7d50 0x7f70cc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d47a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc097e00 con 0x7f70cc1b7d50 2023-10-30T12:07:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7d50 0x7f70cc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient: get_auth_request con 0x7f70cc1b7d50 auth_method 0 2023-10-30T12:07:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 10 monclient(hunting): handle_auth_done global_id 104199 payload 824 2023-10-30T12:07:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 1 -- 172.21.15.50:0/3951241977 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f70cc08d8d0 con 0x7f70cc1b7d50 2023-10-30T12:07:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70c2ffd640 10 client.104193.objecter ms_handle_connect 0x7f70cc1b7d50 2023-10-30T12:07:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70c2ffd640 10 client.104193.objecter resend_mon_ops 2023-10-30T12:07:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70d1d44640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7d50 0x7f70cc06ddb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70bc019000 tx=0x7f70bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc023070 con 0x7f70cc1b7d50 2023-10-30T12:07:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.834+0000 7f70c2ffd640 10 monclient: dump: 2023-10-30T12:07:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:13.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:13.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:13.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:13.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:13.888 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:13.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:13.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7f70bc003ca0 con 0x7f70cc1b7d50 2023-10-30T12:07:13.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70c2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:13.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70c2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:13.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7d50 msgr2=0x7f70cc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70c2ffd640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7d50 0x7f70cc06ddb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70bc019000 tx=0x7f70bc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70c2ffd640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7d50 msgr2=0x7f70cc06ddb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:13.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70c2ffd640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc1b7d50 0x7f70cc06ddb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.890 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:07:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 70, 2023-10-30T12:07:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:07:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:07:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:07:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:07:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:07:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:13.921 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 10 librados: watch_flush enter 2023-10-30T12:07:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 10 librados: watch_flush exit 2023-10-30T12:07:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 20 client.104193.objecter shutdown clearing up homeless session... 2023-10-30T12:07:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 10 client.104193.objecter successfully canceled tick 2023-10-30T12:07:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f70b00392a0 msgr2=0x7f70b003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f70b00392a0 0x7f70b003b760 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 crypto rx=0x7f70c4003d00 tx=0x7f70c4009270 comp rx=0 tx=0).stop 2023-10-30T12:07:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 10 monclient: shutdown 2023-10-30T12:07:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc096080 msgr2=0x7f70cc08afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 --2- 172.21.15.50:0/3951241977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70cc096080 0x7f70cc08afd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70c8036040 tx=0x7f70c80046c0 comp rx=0 tx=0).stop 2023-10-30T12:07:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d2d46640 1 -- 172.21.15.50:0/3951241977 reap_dead start 2023-10-30T12:07:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d2d46640 1 -- 172.21.15.50:0/3951241977 reap_dead start 2023-10-30T12:07:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 shutdown_connections 2023-10-30T12:07:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 >> 172.21.15.50:0/3951241977 conn(0x7f70cc0a3980 msgr2=0x7f70cc1ba6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 shutdown_connections 2023-10-30T12:07:13.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 -- 172.21.15.50:0/3951241977 wait complete. 2023-10-30T12:07:13.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:13.835+0000 7f70d47a9640 1 librados: shutdown 2023-10-30T12:07:13.926 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:07:13.926 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:07:13.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:07:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.031+0000 7f822ec4f640 10 monclient: get_monmap_and_config 2023-10-30T12:07:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.031+0000 7f822ec4f640 10 monclient: build_initial_monmap 2023-10-30T12:07:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.032+0000 7f822ec4f640 10 monclient: monmap: 2023-10-30T12:07:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:14.033054+0000 2023-10-30T12:07:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:14.033054+0000 2023-10-30T12:07:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:14.035 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:14.036 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.032+0000 7f822ec4f640 1 Processor -- start 2023-10-30T12:07:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.032+0000 7f822ec4f640 1 -- start start 2023-10-30T12:07:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.032+0000 7f822ec4f640 10 monclient: init 2023-10-30T12:07:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-10-30T12:07:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8228097780 0x7f8228097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient(hunting): picked mon.noname-c con 0x7f8228097780 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8228073b10 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f822809ccd0 0x7f82280a0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient(hunting): picked mon.noname-a con 0x7f822809ccd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228071ec0 con 0x7f8228073b10 2023-10-30T12:07:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822809bd10 con 0x7f822809ccd0 2023-10-30T12:07:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822809bff0 con 0x7f8228097780 2023-10-30T12:07:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822ec4f640 10 monclient(hunting): authenticate will time out at 1279.216229s 2023-10-30T12:07:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f822809ccd0 0x7f82280a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f8227fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8228073b10 0x7f82280a0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48300/0 (socket says 172.21.15.50:48300) 2023-10-30T12:07:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f8227fff640 1 -- 172.21.15.50:0/4221758506 learned_addr learned my addr 172.21.15.50:0/4221758506 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): get_auth_request con 0x7f822809ccd0 auth_method 0 2023-10-30T12:07:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 1 --2- 172.21.15.50:0/4221758506 >> v2:172.21.15.118:3300/0 conn(0x7f8228097780 0x7f8228097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 10 monclient(hunting): get_auth_request con 0x7f8228097780 auth_method 0 2023-10-30T12:07:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient(hunting): handle_auth_done global_id 104202 payload 274 2023-10-30T12:07:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 1 -- 172.21.15.50:0/4221758506 >> v2:172.21.15.118:3300/0 conn(0x7f8228097780 msgr2=0x7f8228097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 1 --2- 172.21.15.50:0/4221758506 >> v2:172.21.15.118:3300/0 conn(0x7f8228097780 0x7f8228097b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 1 -- 172.21.15.50:0/4221758506 >> v1:172.21.15.50:6789/0 conn(0x7f8228073b10 legacy=0x7f82280a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient: _finish_hunting 0 2023-10-30T12:07:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 1 monclient: found mon.noname-a 2023-10-30T12:07:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.033+0000 7f822d1ec640 1 -- 172.21.15.50:0/4221758506 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82280980a0 con 0x7f822809ccd0 2023-10-30T12:07:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f822d1ec640 1 --2- 172.21.15.50:0/4221758506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f82280a0790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f822806a280 tx=0x7f822002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7427dbb478f49f11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 1 -- 172.21.15.50:0/4221758506 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822002cb60 con 0x7f822809ccd0 2023-10-30T12:07:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: dump: 2023-10-30T12:07:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:14.055 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: _finish_auth 0 2023-10-30T12:07:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: _check_auth_tickets 2023-10-30T12:07:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 1 -- 172.21.15.50:0/4221758506 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822002ccc0 con 0x7f822809ccd0 2023-10-30T12:07:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f822ec4f640 5 monclient: authenticate success, global_id 104202 2023-10-30T12:07:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f822ec4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f822ec4f640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 1 -- 172.21.15.50:0/4221758506 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8220034660 con 0x7f822809ccd0 2023-10-30T12:07:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f822ec4f640 4 set_mon_vals no callback set 2023-10-30T12:07:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.034+0000 7f82277fe640 10 monclient: dump: 2023-10-30T12:07:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:14.059 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 10 monclient: shutdown 2023-10-30T12:07:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 -- 172.21.15.50:0/4221758506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 msgr2=0x7f82280a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 --2- 172.21.15.50:0/4221758506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f82280a0790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f822806a280 tx=0x7f822002bdb0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 -- 172.21.15.50:0/4221758506 shutdown_connections 2023-10-30T12:07:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 --2- 172.21.15.50:0/4221758506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f82280a0790 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 --2- 172.21.15.50:0/4221758506 >> v2:172.21.15.118:3300/0 conn(0x7f8228097780 0x7f8228097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 -- 172.21.15.50:0/4221758506 >> 172.21.15.50:0/4221758506 conn(0x7f82280a1170 msgr2=0x7f82280a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 -- 172.21.15.50:0/4221758506 shutdown_connections 2023-10-30T12:07:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 -- 172.21.15.50:0/4221758506 wait complete. 2023-10-30T12:07:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 10 monclient: build_initial_monmap 2023-10-30T12:07:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 10 monclient: monmap: 2023-10-30T12:07:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:14.063 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 librados: starting msgr at 2023-10-30T12:07:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.035+0000 7f822ec4f640 1 librados: starting objecter 2023-10-30T12:07:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 Processor -- start 2023-10-30T12:07:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 -- start start 2023-10-30T12:07:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 librados: setting wanted keys 2023-10-30T12:07:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 librados: calling monclient init 2023-10-30T12:07:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient: init 2023-10-30T12:07:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:07:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228073b10 0x7f82281ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): picked mon.noname-b con 0x7f8228073b10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8228097780 0x7f82281cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): picked mon.noname-c con 0x7f8228097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f822808f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): picked mon.noname-a con 0x7f822809ccd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228092d20 con 0x7f822809ccd0 2023-10-30T12:07:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228073f70 con 0x7f8228073b10 2023-10-30T12:07:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82280a3b40 con 0x7f8228097780 2023-10-30T12:07:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822ec4f640 10 monclient(hunting): authenticate will time out at 1279.219714s 2023-10-30T12:07:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f8227fff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8228097780 msgr2=0x7f82281cae80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:07:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f8227fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8228097780 0x7f82281cae80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f822808f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f822808f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55882/0 (socket says 172.21.15.50:55882) 2023-10-30T12:07:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 1 -- 172.21.15.50:0/1764527806 learned_addr learned my addr 172.21.15.50:0/1764527806 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 10 monclient(hunting): get_auth_request con 0x7f822809ccd0 auth_method 0 2023-10-30T12:07:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822c9eb640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228073b10 0x7f82281ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822d1ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822c9eb640 10 monclient(hunting): get_auth_request con 0x7f8228073b10 auth_method 0 2023-10-30T12:07:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822c9eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822c9eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.036+0000 7f822c9eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 10 monclient(hunting): handle_auth_done global_id 104208 payload 824 2023-10-30T12:07:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8228097780 msgr2=0x7f82281cae80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8228097780 0x7f82281cae80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228073b10 msgr2=0x7f82281ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228073b10 0x7f82281ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 10 monclient: _finish_hunting 0 2023-10-30T12:07:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 1 monclient: found mon.noname-a 2023-10-30T12:07:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 1 -- 172.21.15.50:0/1764527806 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8220009a70 con 0x7f822809ccd0 2023-10-30T12:07:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822d1ec640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f822808f550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8220009e80 tx=0x7f8220004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 client.?.objecter ms_handle_connect 0x7f822809ccd0 2023-10-30T12:07:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8220040650 con 0x7f822809ccd0 2023-10-30T12:07:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: dump: 2023-10-30T12:07:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:14.076 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: _finish_auth 0 2023-10-30T12:07:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: _check_auth_tickets 2023-10-30T12:07:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82200407b0 con 0x7f822809ccd0 2023-10-30T12:07:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 5 monclient: authenticate success, global_id 104208 2023-10-30T12:07:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8220040b60 con 0x7f822809ccd0 2023-10-30T12:07:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f82257fa640 10 monclient: dump: 2023-10-30T12:07:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:14.080 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 10 monclient: _renew_subs 2023-10-30T12:07:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f8226ffd640 4 set_mon_vals no callback set 2023-10-30T12:07:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82280935c0 con 0x7f822809ccd0 2023-10-30T12:07:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 10 client.104208.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 10 monclient: _renew_subs 2023-10-30T12:07:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82280911f0 con 0x7f822809ccd0 2023-10-30T12:07:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.037+0000 7f822ec4f640 1 librados: init done 2023-10-30T12:07:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8220040cc0 con 0x7f822809ccd0 2023-10-30T12:07:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822ec4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822ec4f640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090250 0x7f822802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822ec4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228091380 con 0x7f8228090250 2023-10-30T12:07:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f820c0392a0 0x7f820c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f822006dce0 con 0x7f822809ccd0 2023-10-30T12:07:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 10 client.104208.objecter ms_dispatch 0x7f8228000b90 osd_map(206..206 src has 1..206) v4 2023-10-30T12:07:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 3 client.104208.objecter handle_osd_map got epochs [206,206] > 0 2023-10-30T12:07:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 3 client.104208.objecter handle_osd_map decoding full epoch 206 2023-10-30T12:07:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090250 0x7f822802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient: get_auth_request con 0x7f8228090250 auth_method 0 2023-10-30T12:07:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 20 client.104208.objecter dump_active .. 0 homeless 2023-10-30T12:07:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f8227fff640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f820c0392a0 0x7f820c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f8227fff640 10 monclient: get_auth_request con 0x7f820c0392a0 auth_method 0 2023-10-30T12:07:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 10 monclient(hunting): handle_auth_done global_id 94360 payload 824 2023-10-30T12:07:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 1 -- 172.21.15.50:0/1764527806 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f822804e160 con 0x7f8228090250 2023-10-30T12:07:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 10 client.104208.objecter ms_handle_connect 0x7f8228090250 2023-10-30T12:07:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f82257fa640 10 client.104208.objecter resend_mon_ops 2023-10-30T12:07:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.038+0000 7f822c9eb640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090250 0x7f822802b430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f821c0118e0 tx=0x7f821c011da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 10 client.104208.objecter ms_handle_connect 0x7f820c0392a0 2023-10-30T12:07:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f8227fff640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f820c0392a0 0x7f820c03b760 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7f8228090e90 tx=0x7f8214007ae0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c004300 con 0x7f8228090250 2023-10-30T12:07:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 10 monclient: dump: 2023-10-30T12:07:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:14.089 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.089 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:14.089 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:14.089 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:14.089 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f821c004460 con 0x7f8228090250 2023-10-30T12:07:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090250 msgr2=0x7f822802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090250 0x7f822802b430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f821c0118e0 tx=0x7f821c011da0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090250 msgr2=0x7f822802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.039+0000 7f82257fa640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090250 0x7f822802b430 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.045+0000 7f822ec4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.045+0000 7f822ec4f640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090630 0x7f822802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.045+0000 7f822ec4f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.045+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822806a5a0 con 0x7f8228090630 2023-10-30T12:07:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.045+0000 7f822c9eb640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090630 0x7f822802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient: get_auth_request con 0x7f8228090630 auth_method 0 2023-10-30T12:07:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 10 monclient(hunting): handle_auth_done global_id 94363 payload 824 2023-10-30T12:07:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 1 -- 172.21.15.50:0/1764527806 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f822808c6e0 con 0x7f8228090630 2023-10-30T12:07:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f822c9eb640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090630 0x7f822802b970 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f821c01b000 tx=0x7f821c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 10 client.104208.objecter ms_handle_connect 0x7f8228090630 2023-10-30T12:07:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 10 client.104208.objecter resend_mon_ops 2023-10-30T12:07:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c02c070 con 0x7f8228090630 2023-10-30T12:07:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 10 monclient: dump: 2023-10-30T12:07:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:14.098 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:14.098 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:14.098 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:14.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7f821c024040 con 0x7f8228090630 2023-10-30T12:07:14.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:14.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:14.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090630 msgr2=0x7f822802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:14.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090630 0x7f822802b970 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f821c01b000 tx=0x7f821c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.046+0000 7f82257fa640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090630 msgr2=0x7f822802b970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:14.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f82257fa640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8228090630 0x7f822802b970 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.100 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:07:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 70, 2023-10-30T12:07:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:07:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:07:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:07:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:07:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 8 2023-10-30T12:07:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:07:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:14.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:14.131 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:14.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:14.131 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:14.131 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 10 librados: watch_flush enter 2023-10-30T12:07:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 10 librados: watch_flush exit 2023-10-30T12:07:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 20 client.104208.objecter shutdown clearing up homeless session... 2023-10-30T12:07:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 10 client.104208.objecter successfully canceled tick 2023-10-30T12:07:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f820c0392a0 msgr2=0x7f820c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f820c0392a0 0x7f820c03b760 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7f8228090e90 tx=0x7f8214007ae0 comp rx=0 tx=0).stop 2023-10-30T12:07:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 10 monclient: shutdown 2023-10-30T12:07:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 msgr2=0x7f822808f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 --2- 172.21.15.50:0/1764527806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f822809ccd0 0x7f822808f550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8220009e80 tx=0x7f8220004a40 comp rx=0 tx=0).stop 2023-10-30T12:07:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822d1ec640 1 -- 172.21.15.50:0/1764527806 reap_dead start 2023-10-30T12:07:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822d1ec640 1 -- 172.21.15.50:0/1764527806 reap_dead start 2023-10-30T12:07:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 shutdown_connections 2023-10-30T12:07:14.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 >> 172.21.15.50:0/1764527806 conn(0x7f82280a1170 msgr2=0x7f822809a580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:14.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 shutdown_connections 2023-10-30T12:07:14.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 -- 172.21.15.50:0/1764527806 wait complete. 2023-10-30T12:07:14.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:14.047+0000 7f822ec4f640 1 librados: shutdown 2023-10-30T12:07:14.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:07:14.137 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:07:16.480 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRados api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-10-30T12:07:16.480 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-10-30T12:07:16.481 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-10-30T12:07:16.481 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: seed 87473 2023-10-30T12:07:16.481 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-10-30T12:07:16.481 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (185 ms) 2023-10-30T12:07:16.481 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-10-30T12:07:16.482 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10745 ms) 2023-10-30T12:07:16.482 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-10-30T12:07:16.482 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (13 ms) 2023-10-30T12:07:16.482 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10944 ms total) 2023-10-30T12:07:16.483 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: 2023-10-30T12:07:16.483 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-10-30T12:07:16.483 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-10-30T12:07:16.483 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (11834 ms) 2023-10-30T12:07:16.483 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-10-30T12:07:16.484 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (24823 ms) 2023-10-30T12:07:16.484 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-10-30T12:07:16.484 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9349 ms) 2023-10-30T12:07:16.484 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-10-30T12:07:16.484 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: my_snaps [4] 2023-10-30T12:07:16.485 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: my_snaps [5,4] 2023-10-30T12:07:16.485 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: my_snaps [6,5,4] 2023-10-30T12:07:16.485 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-10-30T12:07:16.485 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: promoting some heads 2023-10-30T12:07:16.485 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: promoting from clones for snap 7 2023-10-30T12:07:16.486 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: promoting from clones for snap 6 2023-10-30T12:07:16.486 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: promoting from clones for snap 5 2023-10-30T12:07:16.486 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: promoting from clones for snap 4 2023-10-30T12:07:16.486 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T12:07:16.486 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: done waiting 2023-10-30T12:07:16.487 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47994 ms) 2023-10-30T12:07:16.487 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-10-30T12:07:16.487 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9978 ms) 2023-10-30T12:07:16.487 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-10-30T12:07:16.487 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8001 ms) 2023-10-30T12:07:16.487 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-10-30T12:07:16.488 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (33410 ms) 2023-10-30T12:07:16.488 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-10-30T12:07:16.488 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8024 ms) 2023-10-30T12:07:16.488 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-10-30T12:07:16.488 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9916 ms) 2023-10-30T12:07:16.489 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-10-30T12:07:16.489 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (38508 ms) 2023-10-30T12:07:16.489 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-10-30T12:07:16.489 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9345 ms) 2023-10-30T12:07:16.489 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-10-30T12:07:16.490 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12014 ms) 2023-10-30T12:07:16.490 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-10-30T12:07:16.490 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7988 ms) 2023-10-30T12:07:16.490 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-10-30T12:07:16.490 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (21860 ms) 2023-10-30T12:07:16.491 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-10-30T12:07:16.491 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13004 ms) 2023-10-30T12:07:16.491 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-10-30T12:07:16.491 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7997 ms) 2023-10-30T12:07:16.491 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-10-30T12:07:16.491 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (29299 ms) 2023-10-30T12:07:20.730 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Hite 2023-10-30T12:07:20.731 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6971 ms) 2023-10-30T12:07:20.731 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-10-30T12:07:20.731 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (32407 ms) 2023-10-30T12:07:20.731 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-10-30T12:07:20.732 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7020 ms) 2023-10-30T12:07:20.732 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-10-30T12:07:20.732 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6903 ms) 2023-10-30T12:07:20.732 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-10-30T12:07:20.732 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (22444 ms) 2023-10-30T12:07:20.733 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-10-30T12:07:20.733 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (21084 ms) 2023-10-30T12:07:20.733 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-10-30T12:07:20.733 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6342 ms) 2023-10-30T12:07:20.733 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-10-30T12:07:20.734 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6989 ms) 2023-10-30T12:07:20.734 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-10-30T12:07:20.734 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7027 ms) 2023-10-30T12:07:20.734 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-10-30T12:07:20.734 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6978 ms) 2023-10-30T12:07:20.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-10-30T12:07:20.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (20861 ms) 2023-10-30T12:07:20.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-10-30T12:07:20.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7021 ms) 2023-10-30T12:07:20.735 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-10-30T12:07:20.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6985 ms) 2023-10-30T12:07:20.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-10-30T12:07:20.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6995 ms) 2023-10-30T12:07:20.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-10-30T12:07:20.736 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (28258 ms) 2023-10-30T12:07:20.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-10-30T12:07:20.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6287 ms) 2023-10-30T12:07:20.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (200574 ms total) 2023-10-30T12:07:20.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: 2023-10-30T12:07:20.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [----------] Global test environment tear-down 2023-10-30T12:07:20.737 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (326540 ms total) 2023-10-30T12:07:20.738 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio: [ PASSED ] 42 tests. 2023-10-30T12:07:30.998 INFO:tasks.workunit.client.0.smithi118.stdout:tSetRead 2023-10-30T12:07:30.999 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T12:07:30.999 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: ok, hit_set contains 184:602f83fe:::foo:head 2023-10-30T12:07:30.999 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8252 ms) 2023-10-30T12:07:30.999 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-10-30T12:07:30.999 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg_num = 32 2023-10-30T12:07:31.000 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 0 ls 1698667651,0 2023-10-30T12:07:31.000 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 1 ls 1698667651,0 2023-10-30T12:07:31.000 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 2 ls 1698667651,0 2023-10-30T12:07:31.000 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 3 ls 1698667651,0 2023-10-30T12:07:31.000 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 4 ls 1698667651,0 2023-10-30T12:07:31.000 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 5 ls 1698667651,0 2023-10-30T12:07:31.001 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 6 ls 1698667651,0 2023-10-30T12:07:31.001 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 7 ls 1698667651,0 2023-10-30T12:07:31.001 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 8 ls 1698667651,0 2023-10-30T12:07:31.001 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 9 ls 1698667651,0 2023-10-30T12:07:31.001 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 10 ls 1698667651,0 2023-10-30T12:07:31.002 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 11 ls 1698667651,0 2023-10-30T12:07:31.002 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 12 ls 1698667651,0 2023-10-30T12:07:31.002 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 13 ls 1698667651,0 2023-10-30T12:07:31.002 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 14 ls 1698667651,0 2023-10-30T12:07:31.002 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 15 ls 1698667651,0 2023-10-30T12:07:31.003 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 16 ls 1698667651,0 2023-10-30T12:07:31.003 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 17 ls 1698667651,0 2023-10-30T12:07:31.003 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 18 ls 1698667651,0 2023-10-30T12:07:31.003 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 19 ls 1698667651,0 2023-10-30T12:07:31.003 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 20 ls 1698667651,0 2023-10-30T12:07:31.004 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 21 ls 1698667651,0 2023-10-30T12:07:31.004 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 22 ls 1698667651,0 2023-10-30T12:07:31.004 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 23 ls 1698667651,0 2023-10-30T12:07:31.004 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 24 ls 1698667651,0 2023-10-30T12:07:31.004 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 25 ls 1698667651,0 2023-10-30T12:07:31.005 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 26 ls 1698667651,0 2023-10-30T12:07:31.005 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 27 ls 1698667651,0 2023-10-30T12:07:31.005 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 28 ls 1698667651,0 2023-10-30T12:07:31.005 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 29 ls 1698667651,0 2023-10-30T12:07:31.005 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 30 ls 1698667651,0 2023-10-30T12:07:31.005 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg 31 ls 1698667651,0 2023-10-30T12:07:31.006 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: pg_num = 32 2023-10-30T12:07:31.006 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6cac518f:::0:head 2023-10-30T12:07:31.006 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:02547ec2:::1:head 2023-10-30T12:07:31.006 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f905c69b:::2:head 2023-10-30T12:07:31.006 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:cfc208b3:::3:head 2023-10-30T12:07:31.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d83876eb:::4:head 2023-10-30T12:07:31.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b29083e3:::5:head 2023-10-30T12:07:31.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c4fdafeb:::6:head 2023-10-30T12:07:31.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:5c6b0b28:::7:head 2023-10-30T12:07:31.007 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:bd63b0f1:::8:head 2023-10-30T12:07:31.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:e960b815:::9:head 2023-10-30T12:07:31.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:52ea6a34:::10:head 2023-10-30T12:07:31.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:89d3ae78:::11:head 2023-10-30T12:07:31.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:de5d7c5f:::12:head 2023-10-30T12:07:31.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:566253c9:::13:head 2023-10-30T12:07:31.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:62a1935d:::14:head 2023-10-30T12:07:31.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:863748b0:::15:head 2023-10-30T12:07:31.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:3958e169:::16:head 2023-10-30T12:07:31.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:4d4dabf9:::17:head 2023-10-30T12:07:31.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:8391935d:::18:head 2023-10-30T12:07:31.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:28883081:::19:head 2023-10-30T12:07:31.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:69259c59:::20:head 2023-10-30T12:07:31.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:4bdb80b7:::21:head 2023-10-30T12:07:31.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a11c5d71:::22:head 2023-10-30T12:07:31.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:271af37b:::23:head 2023-10-30T12:07:31.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:95b121be:::24:head 2023-10-30T12:07:31.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:58d1031b:::25:head 2023-10-30T12:07:31.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:0a050783:::26:head 2023-10-30T12:07:31.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c709704c:::27:head 2023-10-30T12:07:31.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:cbe56eaf:::28:head 2023-10-30T12:07:31.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:86b4b162:::29:head 2023-10-30T12:07:31.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:70d89383:::30:head 2023-10-30T12:07:31.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:dd450c7c:::31:head 2023-10-30T12:07:31.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6d5729b1:::32:head 2023-10-30T12:07:31.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c388f3fb:::33:head 2023-10-30T12:07:31.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:56cfea31:::34:head 2023-10-30T12:07:31.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:9dbc1bf7:::35:head 2023-10-30T12:07:31.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:40b74ccd:::36:head 2023-10-30T12:07:31.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:4d5aaf42:::37:head 2023-10-30T12:07:31.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:920f362c:::38:head 2023-10-30T12:07:31.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6cc53222:::39:head 2023-10-30T12:07:31.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:9cad833f:::40:head 2023-10-30T12:07:31.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:1ea84d41:::41:head 2023-10-30T12:07:31.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c4480ef6:::42:head 2023-10-30T12:07:31.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a694361e:::43:head 2023-10-30T12:07:31.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d1bd33e9:::44:head 2023-10-30T12:07:31.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:ddc2cd5d:::45:head 2023-10-30T12:07:31.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:2b782207:::46:head 2023-10-30T12:07:31.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:7b187fca:::47:head 2023-10-30T12:07:31.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:90ecdf6f:::48:head 2023-10-30T12:07:31.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a5ed95fe:::49:head 2023-10-30T12:07:31.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:ea0eaa55:::50:head 2023-10-30T12:07:31.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f33ef17b:::51:head 2023-10-30T12:07:31.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a0d1b2f6:::52:head 2023-10-30T12:07:31.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:60c5229e:::53:head 2023-10-30T12:07:31.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:edcbc575:::54:head 2023-10-30T12:07:31.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:102cf253:::55:head 2023-10-30T12:07:31.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:efb7fb0b:::56:head 2023-10-30T12:07:31.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:50d0a326:::57:head 2023-10-30T12:07:31.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d4dc5daf:::58:head 2023-10-30T12:07:31.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:3a130462:::59:head 2023-10-30T12:07:31.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:ec87ed71:::60:head 2023-10-30T12:07:31.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d5bc9454:::61:head 2023-10-30T12:07:31.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:3ddfe313:::62:head 2023-10-30T12:07:31.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:7c2816b9:::63:head 2023-10-30T12:07:31.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:47e00e4d:::64:head 2023-10-30T12:07:31.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c6410c18:::65:head 2023-10-30T12:07:31.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b48ed237:::66:head 2023-10-30T12:07:31.019 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:cd63ad31:::67:head 2023-10-30T12:07:31.019 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b179e92b:::68:head 2023-10-30T12:07:31.019 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:0d9f741a:::69:head 2023-10-30T12:07:31.019 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6d3352ae:::70:head 2023-10-30T12:07:31.019 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c6d5c19e:::71:head 2023-10-30T12:07:31.020 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:bc4729c3:::72:head 2023-10-30T12:07:31.020 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:77e930b9:::73:head 2023-10-30T12:07:31.020 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:0abeecfd:::74:head 2023-10-30T12:07:31.020 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b7c37e15:::75:head 2023-10-30T12:07:31.020 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b6378398:::76:head 2023-10-30T12:07:31.021 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:02bd68de:::77:head 2023-10-30T12:07:31.021 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:cc795d2d:::78:head 2023-10-30T12:07:31.021 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:630d4fea:::79:head 2023-10-30T12:07:31.021 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:e0d29ef5:::80:head 2023-10-30T12:07:31.021 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:fd6f13d2:::81:head 2023-10-30T12:07:31.021 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:606461d5:::82:head 2023-10-30T12:07:31.022 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:eadbdc43:::83:head 2023-10-30T12:07:31.022 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:8761d0bb:::84:head 2023-10-30T12:07:31.022 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:9ef0186f:::85:head 2023-10-30T12:07:31.022 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:e0d41294:::86:head 2023-10-30T12:07:31.022 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:961de695:::87:head 2023-10-30T12:07:31.023 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:1423148f:::88:head 2023-10-30T12:07:31.023 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:633a8fa2:::89:head 2023-10-30T12:07:31.023 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a8653809:::90:head 2023-10-30T12:07:31.023 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:3dac8b33:::91:head 2023-10-30T12:07:31.023 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:35aad435:::92:head 2023-10-30T12:07:31.024 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f6dcc343:::93:head 2023-10-30T12:07:31.024 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:dbbdad87:::94:head 2023-10-30T12:07:31.024 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:1cb48ce0:::95:head 2023-10-30T12:07:31.024 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:03cd461c:::96:head 2023-10-30T12:07:31.024 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:17a4ea99:::97:head 2023-10-30T12:07:31.024 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:9993c9a7:::98:head 2023-10-30T12:07:31.025 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6394211c:::99:head 2023-10-30T12:07:31.025 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:94c7ae57:::100:head 2023-10-30T12:07:31.025 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6fdee5bb:::101:head 2023-10-30T12:07:31.025 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:9a477fd1:::102:head 2023-10-30T12:07:31.025 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:eb850916:::103:head 2023-10-30T12:07:31.026 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:affc56b9:::104:head 2023-10-30T12:07:31.026 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b42dc814:::105:head 2023-10-30T12:07:31.026 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f319f8f0:::106:head 2023-10-30T12:07:31.026 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:9a40b9de:::107:head 2023-10-30T12:07:31.026 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:8b524f28:::108:head 2023-10-30T12:07:31.027 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:e3de589f:::109:head 2023-10-30T12:07:31.027 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:90f90a5b:::110:head 2023-10-30T12:07:31.027 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a7b4f1d7:::111:head 2023-10-30T12:07:31.027 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:af51766e:::112:head 2023-10-30T12:07:31.029 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b6f90bd1:::113:head 2023-10-30T12:07:31.029 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:e0261208:::114:head 2023-10-30T12:07:31.029 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c9569ef7:::115:head 2023-10-30T12:07:31.030 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:61bebe50:::116:head 2023-10-30T12:07:31.030 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:fe93412b:::117:head 2023-10-30T12:07:31.030 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d3d38bee:::118:head 2023-10-30T12:07:31.030 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:3100ba0c:::119:head 2023-10-30T12:07:31.030 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d0560ada:::120:head 2023-10-30T12:07:31.031 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f0ea8b35:::121:head 2023-10-30T12:07:31.031 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:766f231a:::122:head 2023-10-30T12:07:31.031 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a07a2582:::123:head 2023-10-30T12:07:31.031 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:bd7c6b3a:::124:head 2023-10-30T12:07:31.031 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:fb2ddaff:::125:head 2023-10-30T12:07:31.032 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:4408e1fe:::126:head 2023-10-30T12:07:31.032 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:ee1df7a7:::127:head 2023-10-30T12:07:31.032 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c3002909:::128:head 2023-10-30T12:07:31.032 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:4f48ffa9:::129:head 2023-10-30T12:07:31.032 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:edf38733:::130:head 2023-10-30T12:07:31.033 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c08425c0:::131:head 2023-10-30T12:07:31.033 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:5f902d98:::132:head 2023-10-30T12:07:31.033 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:41ea2c93:::133:head 2023-10-30T12:07:31.033 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:813cee13:::134:head 2023-10-30T12:07:31.033 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:0131818d:::135:head 2023-10-30T12:07:31.033 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:26ba5a85:::136:head 2023-10-30T12:07:31.034 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:381b8a5a:::137:head 2023-10-30T12:07:31.034 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:28797e47:::138:head 2023-10-30T12:07:31.034 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:bfca7f22:::139:head 2023-10-30T12:07:31.034 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:36807075:::140:head 2023-10-30T12:07:31.034 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:80b03975:::141:head 2023-10-30T12:07:31.035 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:5c15709b:::142:head 2023-10-30T12:07:31.035 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f39ea15e:::143:head 2023-10-30T12:07:31.035 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:ea992956:::144:head 2023-10-30T12:07:31.035 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:48887b1c:::145:head 2023-10-30T12:07:31.035 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:9f24a9dd:::146:head 2023-10-30T12:07:31.036 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:987f100b:::147:head 2023-10-30T12:07:31.036 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d2dd3581:::148:head 2023-10-30T12:07:31.036 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:7fed1808:::149:head 2023-10-30T12:07:31.036 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c80b70e9:::150:head 2023-10-30T12:07:31.036 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:85ed90f9:::151:head 2023-10-30T12:07:31.037 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:36428b24:::152:head 2023-10-30T12:07:31.037 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d044c34a:::153:head 2023-10-30T12:07:31.037 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:7c18bf58:::154:head 2023-10-30T12:07:31.037 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d1c21232:::155:head 2023-10-30T12:07:31.037 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a7a3c575:::156:head 2023-10-30T12:07:31.038 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:87da0633:::157:head 2023-10-30T12:07:31.038 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d5ac3822:::158:head 2023-10-30T12:07:31.038 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:3f20522d:::159:head 2023-10-30T12:07:31.038 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6ca26563:::160:head 2023-10-30T12:07:31.038 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:532ce135:::161:head 2023-10-30T12:07:31.038 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:c78863e6:::162:head 2023-10-30T12:07:34.138 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:07:34.139 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-10-30T12:07:34.139 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:07:34.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:07:34.142 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:07:34.143 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:07:34.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:34.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.257+0000 7f8946ca9640 10 monclient: get_monmap_and_config 2023-10-30T12:07:34.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.257+0000 7f8946ca9640 10 monclient: build_initial_monmap 2023-10-30T12:07:34.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.257+0000 7f8946ca9640 10 monclient: monmap: 2023-10-30T12:07:34.259 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:34.260 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:34.260 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:34.258494+0000 2023-10-30T12:07:34.260 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:34.258494+0000 2023-10-30T12:07:34.260 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:34.260 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:34.261 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:34.261 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:34.261 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:34.261 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:34.261 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:34.262 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:34.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 1 Processor -- start 2023-10-30T12:07:34.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 1 -- start start 2023-10-30T12:07:34.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient: init 2023-10-30T12:07:34.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:34.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:07:34.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f89400a4b60 0x7f89401326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:34.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f89400a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:34.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8940099a40 0x7f8940132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f8940099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89400a2350 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8940133120 con 0x7f89400a2350 2023-10-30T12:07:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89400a3d70 con 0x7f89400a4b60 2023-10-30T12:07:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89400a4f40 con 0x7f8940099a40 2023-10-30T12:07:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8946ca9640 10 monclient(hunting): authenticate will time out at 1299.441726s 2023-10-30T12:07:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8944a45640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f89400a4b60 0x7f89401326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8945246640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f89400a2350 0x7f89400a27e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57066/0 (socket says 172.21.15.50:57066) 2023-10-30T12:07:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8945246640 1 -- 172.21.15.50:0/3529087067 learned_addr learned my addr 172.21.15.50:0/3529087067 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8944a45640 10 monclient(hunting): get_auth_request con 0x7f89400a4b60 auth_method 0 2023-10-30T12:07:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8944a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8944a45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8944a45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8944a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.258+0000 7f8944a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f893ffff640 1 --2- 172.21.15.50:0/3529087067 >> v2:172.21.15.118:3300/0 conn(0x7f8940099a40 0x7f8940132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f893ffff640 10 monclient(hunting): get_auth_request con 0x7f8940099a40 auth_method 0 2023-10-30T12:07:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f893ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f893ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f893ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 10 monclient(hunting): handle_auth_done global_id 104229 payload 274 2023-10-30T12:07:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 1 -- 172.21.15.50:0/3529087067 >> v2:172.21.15.118:3300/0 conn(0x7f8940099a40 msgr2=0x7f8940132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 1 --2- 172.21.15.50:0/3529087067 >> v2:172.21.15.118:3300/0 conn(0x7f8940099a40 0x7f8940132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 1 -- 172.21.15.50:0/3529087067 >> v1:172.21.15.50:6789/0 conn(0x7f89400a2350 legacy=0x7f89400a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 10 monclient: _finish_hunting 0 2023-10-30T12:07:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 1 monclient: found mon.noname-a 2023-10-30T12:07:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 1 -- 172.21.15.50:0/3529087067 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89400a4150 con 0x7f89400a4b60 2023-10-30T12:07:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.259+0000 7f8944a45640 1 --2- 172.21.15.50:0/3529087067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401326a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8940099ed0 tx=0x7f893402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb28159cdcdd580 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 1 -- 172.21.15.50:0/3529087067 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934031710 con 0x7f89400a4b60 2023-10-30T12:07:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: dump: 2023-10-30T12:07:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:34.279 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:07:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:07:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 1 -- 172.21.15.50:0/3529087067 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893402f050 con 0x7f89400a4b60 2023-10-30T12:07:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 5 monclient: authenticate success, global_id 104229 2023-10-30T12:07:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 1 -- 172.21.15.50:0/3529087067 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934031a90 con 0x7f89400a4b60 2023-10-30T12:07:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 4 set_mon_vals no callback set 2023-10-30T12:07:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f893f7fe640 10 monclient: dump: 2023-10-30T12:07:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:34.284 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 10 monclient: shutdown 2023-10-30T12:07:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 -- 172.21.15.50:0/3529087067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 msgr2=0x7f89401326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3529087067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401326a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8940099ed0 tx=0x7f893402b7b0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 -- 172.21.15.50:0/3529087067 shutdown_connections 2023-10-30T12:07:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3529087067 >> v2:172.21.15.118:3300/0 conn(0x7f8940099a40 0x7f8940132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3529087067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401326a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 -- 172.21.15.50:0/3529087067 >> 172.21.15.50:0/3529087067 conn(0x7f89400a8060 msgr2=0x7f89400a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 -- 172.21.15.50:0/3529087067 shutdown_connections 2023-10-30T12:07:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.260+0000 7f8946ca9640 1 -- 172.21.15.50:0/3529087067 wait complete. 2023-10-30T12:07:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 10 monclient: build_initial_monmap 2023-10-30T12:07:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 10 monclient: monmap: 2023-10-30T12:07:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:34.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 1 librados: starting msgr at 2023-10-30T12:07:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 1 librados: starting objecter 2023-10-30T12:07:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 1 Processor -- start 2023-10-30T12:07:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 1 -- start start 2023-10-30T12:07:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 1 librados: setting wanted keys 2023-10-30T12:07:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 1 librados: calling monclient init 2023-10-30T12:07:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.261+0000 7f8946ca9640 10 monclient: init 2023-10-30T12:07:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:07:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89400a2350 0x7f89401ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): picked mon.noname-b con 0x7f89400a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f89400a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 0x7f8940121d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f8940121860 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89400a18d0 con 0x7f89400a4b60 2023-10-30T12:07:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89400a0a30 con 0x7f89400a2350 2023-10-30T12:07:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89401258d0 con 0x7f8940121860 2023-10-30T12:07:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8946ca9640 10 monclient(hunting): authenticate will time out at 1299.445367s 2023-10-30T12:07:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 0x7f8940121d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36710/0 (socket says 172.21.15.50:36710) 2023-10-30T12:07:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 0x7f8940121d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45166/0 (socket says 172.21.15.50:45166) 2023-10-30T12:07:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 -- 172.21.15.50:0/3873307111 learned_addr learned my addr 172.21.15.50:0/3873307111 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): get_auth_request con 0x7f89400a4b60 auth_method 0 2023-10-30T12:07:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 10 monclient(hunting): get_auth_request con 0x7f8940121860 auth_method 0 2023-10-30T12:07:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 1 -- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 msgr2=0x7f8940121d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:07:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 1 -- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 msgr2=0x7f8940121d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 0x7f8940121d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8945246640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 0x7f8940121d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8944a45640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89400a2350 0x7f89401ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8944a45640 10 monclient(hunting): get_auth_request con 0x7f89400a2350 auth_method 0 2023-10-30T12:07:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8944a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8944a45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f8944a45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient(hunting): handle_auth_done global_id 104235 payload 824 2023-10-30T12:07:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 -- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 msgr2=0x7f8940121d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 0x7f8940121d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 -- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89400a2350 msgr2=0x7f89401ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89400a2350 0x7f89401ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:07:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 monclient: found mon.noname-a 2023-10-30T12:07:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.262+0000 7f893ffff640 1 -- 172.21.15.50:0/3873307111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934009400 con 0x7f89400a4b60 2023-10-30T12:07:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893ffff640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401caac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f892c00d5d0 tx=0x7f892c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 client.?.objecter ms_handle_connect 0x7f89400a4b60 2023-10-30T12:07:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 1 -- 172.21.15.50:0/3873307111 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c0125f0 con 0x7f89400a4b60 2023-10-30T12:07:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: dump: 2023-10-30T12:07:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:34.301 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:07:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:07:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 1 -- 172.21.15.50:0/3873307111 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c012750 con 0x7f89400a4b60 2023-10-30T12:07:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 1 -- 172.21.15.50:0/3873307111 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c012ae0 con 0x7f89400a4b60 2023-10-30T12:07:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 5 monclient: authenticate success, global_id 104235 2023-10-30T12:07:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 10 monclient: _renew_subs 2023-10-30T12:07:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893effd640 4 set_mon_vals no callback set 2023-10-30T12:07:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8940123ec0 con 0x7f89400a4b60 2023-10-30T12:07:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f893d7fa640 10 monclient: dump: 2023-10-30T12:07:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:34.306 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:34.306 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:34.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:34.306 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:34.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 10 client.104235.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:34.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 10 monclient: _renew_subs 2023-10-30T12:07:34.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:34.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8940123500 con 0x7f89400a4b60 2023-10-30T12:07:34.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.263+0000 7f8946ca9640 1 librados: init done 2023-10-30T12:07:34.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f893d7fa640 1 -- 172.21.15.50:0/3873307111 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f892c012c40 con 0x7f89400a4b60 2023-10-30T12:07:34.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f8946ca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:34.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f8946ca9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:34.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f8946ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:34.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f893d7fa640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89240392f0 0x7f892403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:34.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894004e160 con 0x7f89400a4b60 2023-10-30T12:07:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f893d7fa640 1 -- 172.21.15.50:0/3873307111 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f892c04afb0 con 0x7f89400a4b60 2023-10-30T12:07:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f893d7fa640 10 client.104235.objecter ms_dispatch 0x7f8940000b90 osd_map(227..227 src has 1..227) v4 2023-10-30T12:07:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f893d7fa640 3 client.104235.objecter handle_osd_map got epochs [227,227] > 0 2023-10-30T12:07:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f893d7fa640 3 client.104235.objecter handle_osd_map decoding full epoch 227 2023-10-30T12:07:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f893d7fa640 20 client.104235.objecter dump_active .. 0 homeless 2023-10-30T12:07:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f8944a45640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89240392f0 0x7f892403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.264+0000 7f8944a45640 10 monclient: get_auth_request con 0x7f89240392f0 auth_method 0 2023-10-30T12:07:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.265+0000 7f893d7fa640 10 client.104235.objecter ms_handle_connect 0x7f89240392f0 2023-10-30T12:07:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.265+0000 7f8944a45640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89240392f0 0x7f892403b7b0 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7f8934035000 tx=0x7f8934008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:34.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.269+0000 7f893d7fa640 1 -- 172.21.15.50:0/3873307111 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f892c014430 con 0x7f89400a4b60 2023-10-30T12:07:34.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.269+0000 7f893d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:34.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.269+0000 7f893d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.514+0000 7f8946ca9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.514+0000 7f8946ca9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:34.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.514+0000 7f8946ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.514+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8940125270 con 0x7f89400a4b60 2023-10-30T12:07:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.515+0000 7f893d7fa640 1 -- 172.21.15.50:0/3873307111 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f892c01d3c0 con 0x7f89400a4b60 2023-10-30T12:07:34.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.515+0000 7f893d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:34.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.515+0000 7f893d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:34.518 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.516+0000 7f8946ca9640 10 librados: watch_flush enter 2023-10-30T12:07:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.516+0000 7f8946ca9640 10 librados: watch_flush exit 2023-10-30T12:07:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.516+0000 7f8946ca9640 20 client.104235.objecter shutdown clearing up homeless session... 2023-10-30T12:07:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.516+0000 7f8946ca9640 10 client.104235.objecter successfully canceled tick 2023-10-30T12:07:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89240392f0 msgr2=0x7f892403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89240392f0 0x7f892403b7b0 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7f8934035000 tx=0x7f8934008030 comp rx=0 tx=0).stop 2023-10-30T12:07:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 10 monclient: shutdown 2023-10-30T12:07:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 msgr2=0x7f89401caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401caac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f892c00d5d0 tx=0x7f892c00da90 comp rx=0 tx=0).stop 2023-10-30T12:07:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 shutdown_connections 2023-10-30T12:07:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f89240392f0 0x7f892403b7b0 unknown :-1 s=CLOSED pgs=4017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8940121860 0x7f8940121d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89400a4b60 0x7f89401caac0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 --2- 172.21.15.50:0/3873307111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89400a2350 0x7f89401ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 >> 172.21.15.50:0/3873307111 conn(0x7f89400a8060 msgr2=0x7f894009dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 shutdown_connections 2023-10-30T12:07:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 -- 172.21.15.50:0/3873307111 wait complete. 2023-10-30T12:07:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:34.517+0000 7f8946ca9640 1 librados: shutdown 2023-10-30T12:07:37.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.647+0000 7fda36ef2640 10 monclient: get_monmap_and_config 2023-10-30T12:07:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.647+0000 7fda36ef2640 10 monclient: build_initial_monmap 2023-10-30T12:07:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.647+0000 7fda36ef2640 10 monclient: monmap: 2023-10-30T12:07:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:37.648088+0000 2023-10-30T12:07:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:37.648088+0000 2023-10-30T12:07:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:37.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.647+0000 7fda36ef2640 1 Processor -- start 2023-10-30T12:07:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.647+0000 7fda36ef2640 1 -- start start 2023-10-30T12:07:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.647+0000 7fda36ef2640 10 monclient: init 2023-10-30T12:07:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-10-30T12:07:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fda30098520 0x7fda3009bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient(hunting): picked mon.noname-c con 0x7fda30098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda30098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda301379f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3009ad30 con 0x7fda30098900 2023-10-30T12:07:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3009c460 con 0x7fda301379f0 2023-10-30T12:07:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3009b260 con 0x7fda30098520 2023-10-30T12:07:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda36ef2640 10 monclient(hunting): authenticate will time out at 1302.831423s 2023-10-30T12:07:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2ffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fda30098900 0x7fda30098d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39822/0 (socket says 172.21.15.50:39822) 2023-10-30T12:07:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2ffff640 1 -- 172.21.15.50:0/3878787526 learned_addr learned my addr 172.21.15.50:0/3878787526 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 1 --2- 172.21.15.50:0/3878787526 >> v2:172.21.15.118:3300/0 conn(0x7fda30098520 0x7fda3009bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836742632 0 0) 0x7fda3009ad30 con 0x7fda30098900 2023-10-30T12:07:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 10 monclient(hunting): get_auth_request con 0x7fda30098520 auth_method 0 2023-10-30T12:07:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2f7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:07:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2f7fe640 10 monclient(hunting): my global_id is 104238 2023-10-30T12:07:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda18003520 con 0x7fda30098900 2023-10-30T12:07:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.648+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189500696 0 0) 0x7fda3009c460 con 0x7fda301379f0 2023-10-30T12:07:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:07:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): my global_id is 94414 2023-10-30T12:07:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3009ad30 con 0x7fda301379f0 2023-10-30T12:07:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859944144 0 0) 0x7fda18003520 con 0x7fda30098900 2023-10-30T12:07:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 >> v2:172.21.15.118:3300/0 conn(0x7fda30098520 msgr2=0x7fda3009bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 --2- 172.21.15.50:0/3878787526 >> v2:172.21.15.118:3300/0 conn(0x7fda30098520 0x7fda3009bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 >> v1:172.21.15.118:6789/0 conn(0x7fda301379f0 legacy=0x7fda30139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:07:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:07:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:07:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30072120 con 0x7fda30098900 2023-10-30T12:07:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:07:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:07:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fda24001670 con 0x7fda30098900 2023-10-30T12:07:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:07:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: dump: 2023-10-30T12:07:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:37.674 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:07:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 >> v1:172.21.15.50:6789/0 conn(0x7fda30098900 legacy=0x7fda30098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:07:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda18003b00 0x7fda18003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): picked mon.b con 0x7fda18003b00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda180047c0 0x7fda18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): picked mon.c con 0x7fda180047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda180051f0 0x7fda1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): picked mon.a con 0x7fda180051f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3009b260 con 0x7fda180051f0 2023-10-30T12:07:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1800c150 con 0x7fda18003b00 2023-10-30T12:07:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1800c310 con 0x7fda180047c0 2023-10-30T12:07:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2f7fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:07:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda180047c0 0x7fda18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda180051f0 0x7fda1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda18003b00 0x7fda18003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): get_auth_request con 0x7fda180047c0 auth_method 0 2023-10-30T12:07:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 10 monclient(hunting): get_auth_request con 0x7fda18003b00 auth_method 0 2023-10-30T12:07:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 10 monclient(hunting): get_auth_request con 0x7fda180051f0 auth_method 0 2023-10-30T12:07:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:07:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:07:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:07:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient(hunting): handle_auth_done global_id 104238 payload 293 2023-10-30T12:07:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 1 -- 172.21.15.50:0/3878787526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda18003b00 msgr2=0x7fda18003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda18003b00 0x7fda18003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 1 -- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda180051f0 msgr2=0x7fda1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda180051f0 0x7fda1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:07:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 1 monclient: found mon.c 2023-10-30T12:07:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 1 -- 172.21.15.50:0/3878787526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18003520 con 0x7fda180047c0 2023-10-30T12:07:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient: _finish_auth 0 2023-10-30T12:07:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda36ef2640 5 monclient: authenticate success, global_id 104238 2023-10-30T12:07:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda36ef2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda180051f0 0x7fda1800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.649+0000 7fda3548f640 1 -- 172.21.15.50:0/3878787526 reap_dead start 2023-10-30T12:07:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2ffff640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda180047c0 0x7fda18004c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c160 tx=0x7fda180057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37aec784089ef492 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24005ad0 con 0x7fda180047c0 2023-10-30T12:07:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 10 monclient: dump: 2023-10-30T12:07:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:37.688 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda24005c50 con 0x7fda180047c0 2023-10-30T12:07:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda36ef2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda36ef2640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 1 -- 172.21.15.50:0/3878787526 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2402c830 con 0x7fda180047c0 2023-10-30T12:07:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda36ef2640 4 set_mon_vals no callback set 2023-10-30T12:07:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.650+0000 7fda2f7fe640 10 monclient: dump: 2023-10-30T12:07:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:37.691 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 10 monclient: shutdown 2023-10-30T12:07:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 1 -- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda180047c0 msgr2=0x7fda18004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda180047c0 0x7fda18004c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c160 tx=0x7fda180057f0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 1 -- 172.21.15.50:0/3878787526 shutdown_connections 2023-10-30T12:07:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3878787526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda180047c0 0x7fda18004c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 1 -- 172.21.15.50:0/3878787526 >> 172.21.15.50:0/3878787526 conn(0x7fda300a05c0 msgr2=0x7fda3009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 1 -- 172.21.15.50:0/3878787526 shutdown_connections 2023-10-30T12:07:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 1 -- 172.21.15.50:0/3878787526 wait complete. 2023-10-30T12:07:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 10 monclient: build_initial_monmap 2023-10-30T12:07:37.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.651+0000 7fda36ef2640 10 monclient: monmap: 2023-10-30T12:07:37.694 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:37.694 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:37.694 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:37.694 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:37.694 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:37.695 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:37.695 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:37.695 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:37.695 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:37.695 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 librados: starting msgr at 2023-10-30T12:07:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 librados: starting objecter 2023-10-30T12:07:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 Processor -- start 2023-10-30T12:07:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 -- start start 2023-10-30T12:07:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 librados: setting wanted keys 2023-10-30T12:07:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 librados: calling monclient init 2023-10-30T12:07:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient: init 2023-10-30T12:07:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:07:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda180047c0 0x7fda30096230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): picked mon.noname-b con 0x7fda180047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 0x7fda3008f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): picked mon.noname-c con 0x7fda30096770 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30096b90 0x7fda3008f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): picked mon.noname-a con 0x7fda30096b90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300a2f80 con 0x7fda30096b90 2023-10-30T12:07:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30071df0 con 0x7fda180047c0 2023-10-30T12:07:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda30092d00 con 0x7fda30096770 2023-10-30T12:07:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.652+0000 7fda36ef2640 10 monclient(hunting): authenticate will time out at 1302.835957s 2023-10-30T12:07:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30096b90 0x7fda3008f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 0x7fda3008f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30096b90 0x7fda3008f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38844/0 (socket says 172.21.15.50:38844) 2023-10-30T12:07:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 0x7fda3008f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44338/0 (socket says 172.21.15.50:44338) 2023-10-30T12:07:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 -- 172.21.15.50:0/3446139057 learned_addr learned my addr 172.21.15.50:0/3446139057 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): get_auth_request con 0x7fda30096b90 auth_method 0 2023-10-30T12:07:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 10 monclient(hunting): get_auth_request con 0x7fda30096770 auth_method 0 2023-10-30T12:07:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda180047c0 0x7fda30096230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 10 monclient(hunting): get_auth_request con 0x7fda180047c0 auth_method 0 2023-10-30T12:07:37.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:37.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:37.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:37.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:37.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:37.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:37.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 1 -- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 msgr2=0x7fda3008f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:07:37.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 1 -- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 msgr2=0x7fda3008f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:37.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 0x7fda3008f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:37.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:37.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:37.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda34c8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:37.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2ffff640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 0x7fda3008f3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:37.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient(hunting): handle_auth_done global_id 104241 payload 824 2023-10-30T12:07:37.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 -- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 msgr2=0x7fda3008f3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:37.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 0x7fda3008f3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 -- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda180047c0 msgr2=0x7fda30096230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:37.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda180047c0 0x7fda30096230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient: _finish_hunting 0 2023-10-30T12:07:37.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 monclient: found mon.noname-a 2023-10-30T12:07:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 -- 172.21.15.50:0/3446139057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30092460 con 0x7fda30096b90 2023-10-30T12:07:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda3548f640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30096b90 0x7fda3008f920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda300a0e10 tx=0x7fda2800feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 client.?.objecter ms_handle_connect 0x7fda30096b90 2023-10-30T12:07:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 1 -- 172.21.15.50:0/3446139057 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28003ba0 con 0x7fda30096b90 2023-10-30T12:07:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: dump: 2023-10-30T12:07:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:37.711 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:07:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:07:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 1 -- 172.21.15.50:0/3446139057 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28003d00 con 0x7fda30096b90 2023-10-30T12:07:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 5 monclient: authenticate success, global_id 104241 2023-10-30T12:07:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 10 monclient: _renew_subs 2023-10-30T12:07:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda30092740 con 0x7fda30096b90 2023-10-30T12:07:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 1 -- 172.21.15.50:0/3446139057 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280052d0 con 0x7fda30096b90 2023-10-30T12:07:37.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:37.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:37.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2d7fa640 10 monclient: dump: 2023-10-30T12:07:37.713 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:37.713 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:37.715 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda2effd640 4 set_mon_vals no callback set 2023-10-30T12:07:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 10 client.104241.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 10 monclient: _renew_subs 2023-10-30T12:07:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda300716e0 con 0x7fda30096b90 2023-10-30T12:07:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.653+0000 7fda36ef2640 1 librados: init done 2023-10-30T12:07:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda2d7fa640 1 -- 172.21.15.50:0/3446139057 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fda28005430 con 0x7fda30096b90 2023-10-30T12:07:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda36ef2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda36ef2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda36ef2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda30092930 con 0x7fda30096b90 2023-10-30T12:07:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda2d7fa640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda1c0392f0 0x7fda1c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda2d7fa640 1 -- 172.21.15.50:0/3446139057 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fda2804ae90 con 0x7fda30096b90 2023-10-30T12:07:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda2d7fa640 10 client.104241.objecter ms_dispatch 0x7fda30000b90 osd_map(230..230 src has 1..230) v4 2023-10-30T12:07:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda2d7fa640 3 client.104241.objecter handle_osd_map got epochs [230,230] > 0 2023-10-30T12:07:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.654+0000 7fda2d7fa640 3 client.104241.objecter handle_osd_map decoding full epoch 230 2023-10-30T12:07:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.655+0000 7fda2d7fa640 20 client.104241.objecter dump_active .. 0 homeless 2023-10-30T12:07:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.655+0000 7fda34c8e640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda1c0392f0 0x7fda1c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.655+0000 7fda34c8e640 10 monclient: get_auth_request con 0x7fda1c0392f0 auth_method 0 2023-10-30T12:07:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.655+0000 7fda2d7fa640 10 client.104241.objecter ms_handle_connect 0x7fda1c0392f0 2023-10-30T12:07:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.655+0000 7fda34c8e640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda1c0392f0 0x7fda1c03b7b0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7fda20009b50 tx=0x7fda20009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.659+0000 7fda2d7fa640 1 -- 172.21.15.50:0/3446139057 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fda2804e030 con 0x7fda30096b90 2023-10-30T12:07:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.659+0000 7fda2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.659+0000 7fda2d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:37.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.909+0000 7fda36ef2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:37.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.909+0000 7fda36ef2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:37.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.909+0000 7fda36ef2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:37.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.909+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fda3004e160 con 0x7fda30096b90 2023-10-30T12:07:37.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.909+0000 7fda2d7fa640 1 -- 172.21.15.50:0/3446139057 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fda2803c620 con 0x7fda30096b90 2023-10-30T12:07:37.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.909+0000 7fda2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:37.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.909+0000 7fda2d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:37.912 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:37.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 10 librados: watch_flush enter 2023-10-30T12:07:37.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 10 librados: watch_flush exit 2023-10-30T12:07:37.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 20 client.104241.objecter shutdown clearing up homeless session... 2023-10-30T12:07:37.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 10 client.104241.objecter successfully canceled tick 2023-10-30T12:07:37.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda1c0392f0 msgr2=0x7fda1c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:37.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda1c0392f0 0x7fda1c03b7b0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7fda20009b50 tx=0x7fda20009110 comp rx=0 tx=0).stop 2023-10-30T12:07:37.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 10 monclient: shutdown 2023-10-30T12:07:37.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:37.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30096b90 msgr2=0x7fda3008f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:37.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30096b90 0x7fda3008f920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda300a0e10 tx=0x7fda2800feb0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 shutdown_connections 2023-10-30T12:07:37.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda1c0392f0 0x7fda1c03b7b0 unknown :-1 s=CLOSED pgs=4018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda30096b90 0x7fda3008f920 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda30096770 0x7fda3008f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 --2- 172.21.15.50:0/3446139057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda180047c0 0x7fda30096230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:37.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 >> 172.21.15.50:0/3446139057 conn(0x7fda300a05c0 msgr2=0x7fda3006bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:37.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 shutdown_connections 2023-10-30T12:07:37.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 -- 172.21.15.50:0/3446139057 wait complete. 2023-10-30T12:07:37.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:37.911+0000 7fda36ef2640 1 librados: shutdown 2023-10-30T12:07:40.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:41.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.039+0000 7f862771d640 10 monclient: get_monmap_and_config 2023-10-30T12:07:41.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.039+0000 7f862771d640 10 monclient: build_initial_monmap 2023-10-30T12:07:41.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.039+0000 7f862771d640 10 monclient: monmap: 2023-10-30T12:07:41.041 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:41.042 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.042 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:41.040452+0000 2023-10-30T12:07:41.042 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:41.040452+0000 2023-10-30T12:07:41.042 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:41.042 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.043 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:41.043 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:41.043 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:41.043 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:41.043 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:41.044 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.039+0000 7f862771d640 1 Processor -- start 2023-10-30T12:07:41.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 1 -- start start 2023-10-30T12:07:41.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient: init 2023-10-30T12:07:41.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:41.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:07:41.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f8620136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): picked mon.noname-b con 0x7f8620135d20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:41.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8620135670 0x7f8620070390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): picked mon.noname-c con 0x7f8620135670 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:41.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f86200708d0 0x7f8620070d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): picked mon.noname-a con 0x7f86200708d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:41.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862009d4e0 con 0x7f86200708d0 2023-10-30T12:07:41.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8620071620 con 0x7f8620135670 2023-10-30T12:07:41.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86200a01a0 con 0x7f8620135d20 2023-10-30T12:07:41.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:41.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f862771d640 10 monclient(hunting): authenticate will time out at 1306.223716s 2023-10-30T12:07:41.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f8625cba640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f86200708d0 0x7f8620070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f86254b9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f8620136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f8625cba640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f86200708d0 0x7f8620070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38846/0 (socket says 172.21.15.50:38846) 2023-10-30T12:07:41.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f8625cba640 1 -- 172.21.15.50:0/206035486 learned_addr learned my addr 172.21.15.50:0/206035486 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:41.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f86254b9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f8620136100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44364/0 (socket says 172.21.15.50:44364) 2023-10-30T12:07:41.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f8625cba640 10 monclient(hunting): get_auth_request con 0x7f86200708d0 auth_method 0 2023-10-30T12:07:41.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f8625cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f8625cba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f8625cba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f86254b9640 10 monclient(hunting): get_auth_request con 0x7f8620135d20 auth_method 0 2023-10-30T12:07:41.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f86254b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f86254b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.040+0000 7f86254b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 1 --2- 172.21.15.50:0/206035486 >> v2:172.21.15.118:3300/0 conn(0x7f8620135670 0x7f8620070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 10 monclient(hunting): get_auth_request con 0x7f8620135670 auth_method 0 2023-10-30T12:07:41.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f86254b9640 1 -- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 msgr2=0x7f8620136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:07:41.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f86254b9640 1 -- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 msgr2=0x7f8620136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:41.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f86254b9640 1 --2- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f8620136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f86254b9640 1 --2- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f8620136100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:41.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8624cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 10 monclient(hunting): handle_auth_done global_id 104250 payload 274 2023-10-30T12:07:41.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 1 -- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 msgr2=0x7f8620136100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:41.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 1 --2- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f8620136100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 1 -- 172.21.15.50:0/206035486 >> v2:172.21.15.118:3300/0 conn(0x7f8620135670 msgr2=0x7f8620070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:41.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 1 --2- 172.21.15.50:0/206035486 >> v2:172.21.15.118:3300/0 conn(0x7f8620135670 0x7f8620070390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 10 monclient: _finish_hunting 0 2023-10-30T12:07:41.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 1 monclient: found mon.noname-a 2023-10-30T12:07:41.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:41.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 1 -- 172.21.15.50:0/206035486 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8620099f60 con 0x7f86200708d0 2023-10-30T12:07:41.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.041+0000 7f8625cba640 1 --2- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f8620070d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8620067040 tx=0x7f861802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28aa7b4afe92fb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 1 -- 172.21.15.50:0/206035486 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861802c950 con 0x7f86200708d0 2023-10-30T12:07:41.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:41.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: dump: 2023-10-30T12:07:41.064 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.064 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.065 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.065 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.065 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.065 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.066 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: _finish_auth 0 2023-10-30T12:07:41.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:41.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:41.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 5 monclient: authenticate success, global_id 104250 2023-10-30T12:07:41.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 1 -- 172.21.15.50:0/206035486 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861802cab0 con 0x7f86200708d0 2023-10-30T12:07:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 4 set_mon_vals no callback set 2023-10-30T12:07:41.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 1 -- 172.21.15.50:0/206035486 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861802ce40 con 0x7f86200708d0 2023-10-30T12:07:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f8617fff640 10 monclient: dump: 2023-10-30T12:07:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.069 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.071 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 10 monclient: shutdown 2023-10-30T12:07:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 -- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 msgr2=0x7f8620070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 --2- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f8620070d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8620067040 tx=0x7f861802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 -- 172.21.15.50:0/206035486 shutdown_connections 2023-10-30T12:07:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 --2- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f8620070d40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 --2- 172.21.15.50:0/206035486 >> v2:172.21.15.118:3300/0 conn(0x7f8620135670 0x7f8620070390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 --2- 172.21.15.50:0/206035486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f8620136100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 -- 172.21.15.50:0/206035486 >> 172.21.15.50:0/206035486 conn(0x7f86200a4670 msgr2=0x7f86200a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:41.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 -- 172.21.15.50:0/206035486 shutdown_connections 2023-10-30T12:07:41.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.042+0000 7f862771d640 1 -- 172.21.15.50:0/206035486 wait complete. 2023-10-30T12:07:41.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 10 monclient: build_initial_monmap 2023-10-30T12:07:41.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 10 monclient: monmap: 2023-10-30T12:07:41.074 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:41.074 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:41.074 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:41.075 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:41.075 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:41.075 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.075 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:41.075 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:41.075 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:41.076 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 1 librados: starting msgr at 2023-10-30T12:07:41.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 1 librados: starting objecter 2023-10-30T12:07:41.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 1 Processor -- start 2023-10-30T12:07:41.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 1 -- start start 2023-10-30T12:07:41.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 1 librados: setting wanted keys 2023-10-30T12:07:41.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 1 librados: calling monclient init 2023-10-30T12:07:41.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.043+0000 7f862771d640 10 monclient: init 2023-10-30T12:07:41.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:41.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:41.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f86201ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): picked mon.noname-a con 0x7f86200708d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:41.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8620135670 0x7f86201ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): picked mon.noname-b con 0x7f8620135670 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:41.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f862011f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): picked mon.noname-c con 0x7f8620135d20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:41.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8620123650 con 0x7f86200708d0 2023-10-30T12:07:41.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86201238a0 con 0x7f8620135670 2023-10-30T12:07:41.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8620122110 con 0x7f8620135d20 2023-10-30T12:07:41.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:41.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f862771d640 10 monclient(hunting): authenticate will time out at 1306.227276s 2023-10-30T12:07:41.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f86201ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f86201ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38852/0 (socket says 172.21.15.50:38852) 2023-10-30T12:07:41.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 -- 172.21.15.50:0/534741319 learned_addr learned my addr 172.21.15.50:0/534741319 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:41.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): get_auth_request con 0x7f86200708d0 auth_method 0 2023-10-30T12:07:41.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f862011f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 10 monclient(hunting): get_auth_request con 0x7f8620135d20 auth_method 0 2023-10-30T12:07:41.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 1 -- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 msgr2=0x7f862011f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:07:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 1 -- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 msgr2=0x7f862011f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:41.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f862011f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8625cba640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f862011f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:41.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8624cb8640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8620135670 0x7f86201ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient(hunting): handle_auth_done global_id 104253 payload 824 2023-10-30T12:07:41.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 -- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 msgr2=0x7f862011f830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:41.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f862011f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 -- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8620135670 msgr2=0x7f86201ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8620135670 0x7f86201ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient: _finish_hunting 0 2023-10-30T12:07:41.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 monclient: found mon.noname-a 2023-10-30T12:07:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 -- 172.21.15.50:0/534741319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8618009b30 con 0x7f86200708d0 2023-10-30T12:07:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8624cb8640 10 monclient: get_auth_request con 0x7f8620135670 auth_method 0 2023-10-30T12:07:41.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8624cb8640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8620135670 0x7f86201ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:07:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f86254b9640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f86201ca170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f861c00d7d0 tx=0x7f861c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8615ffb640 10 client.?.objecter ms_handle_connect 0x7f86200708d0 2023-10-30T12:07:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.044+0000 7f8615ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 1 -- 172.21.15.50:0/534741319 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c00b6c0 con 0x7f86200708d0 2023-10-30T12:07:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: dump: 2023-10-30T12:07:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.092 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: _finish_auth 0 2023-10-30T12:07:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:07:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 5 monclient: authenticate success, global_id 104253 2023-10-30T12:07:41.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 10 monclient: _renew_subs 2023-10-30T12:07:41.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 1 -- 172.21.15.50:0/534741319 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c00b840 con 0x7f86200708d0 2023-10-30T12:07:41.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8620134d80 con 0x7f86200708d0 2023-10-30T12:07:41.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:41.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 1 -- 172.21.15.50:0/534741319 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c00bbf0 con 0x7f86200708d0 2023-10-30T12:07:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f8615ffb640 10 monclient: dump: 2023-10-30T12:07:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.095 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.095 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.095 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.095 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.095 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.096 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.096 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f86177fe640 4 set_mon_vals no callback set 2023-10-30T12:07:41.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 10 client.104253.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:41.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 10 monclient: _renew_subs 2023-10-30T12:07:41.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86201214d0 con 0x7f86200708d0 2023-10-30T12:07:41.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.045+0000 7f862771d640 1 librados: init done 2023-10-30T12:07:41.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8615ffb640 1 -- 172.21.15.50:0/534741319 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f861c00bd50 con 0x7f86200708d0 2023-10-30T12:07:41.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8615ffb640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86040392f0 0x7f860403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f862771d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:41.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8615ffb640 1 -- 172.21.15.50:0/534741319 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f861c04b0c0 con 0x7f86200708d0 2023-10-30T12:07:41.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f862771d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:41.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f862771d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8615ffb640 10 client.104253.objecter ms_dispatch 0x7f8620000b90 osd_map(234..234 src has 1..234) v4 2023-10-30T12:07:41.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8615ffb640 3 client.104253.objecter handle_osd_map got epochs [234,234] > 0 2023-10-30T12:07:41.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8615ffb640 3 client.104253.objecter handle_osd_map decoding full epoch 234 2023-10-30T12:07:41.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862004e160 con 0x7f86200708d0 2023-10-30T12:07:41.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8615ffb640 20 client.104253.objecter dump_active .. 0 homeless 2023-10-30T12:07:41.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8624cb8640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86040392f0 0x7f860403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.046+0000 7f8624cb8640 10 monclient: get_auth_request con 0x7f86040392f0 auth_method 0 2023-10-30T12:07:41.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.047+0000 7f8624cb8640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86040392f0 0x7f860403b7b0 secure :-1 s=READY pgs=4020 cs=0 l=1 rev1=1 crypto rx=0x7f8620070490 tx=0x7f8610008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.047+0000 7f8615ffb640 10 client.104253.objecter ms_handle_connect 0x7f86040392f0 2023-10-30T12:07:41.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.051+0000 7f8615ffb640 1 -- 172.21.15.50:0/534741319 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f861c01c770 con 0x7f86200708d0 2023-10-30T12:07:41.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.051+0000 7f8615ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:41.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.051+0000 7f8615ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:41.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.300+0000 7f862771d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:41.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.301+0000 7f862771d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:41.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.301+0000 7f862771d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.301+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8620069de0 con 0x7f86200708d0 2023-10-30T12:07:41.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.301+0000 7f8615ffb640 1 -- 172.21.15.50:0/534741319 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f861c03c070 con 0x7f86200708d0 2023-10-30T12:07:41.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.301+0000 7f8615ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:41.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.301+0000 7f8615ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:41.304 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:41.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 10 librados: watch_flush enter 2023-10-30T12:07:41.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 10 librados: watch_flush exit 2023-10-30T12:07:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 20 client.104253.objecter shutdown clearing up homeless session... 2023-10-30T12:07:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 10 client.104253.objecter successfully canceled tick 2023-10-30T12:07:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86040392f0 msgr2=0x7f860403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86040392f0 0x7f860403b7b0 secure :-1 s=READY pgs=4020 cs=0 l=1 rev1=1 crypto rx=0x7f8620070490 tx=0x7f8610008040 comp rx=0 tx=0).stop 2023-10-30T12:07:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 10 monclient: shutdown 2023-10-30T12:07:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 msgr2=0x7f86201ca170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f86201ca170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f861c00d7d0 tx=0x7f861c00dc90 comp rx=0 tx=0).stop 2023-10-30T12:07:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 shutdown_connections 2023-10-30T12:07:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86040392f0 0x7f860403b7b0 unknown :-1 s=CLOSED pgs=4020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8620135d20 0x7f862011f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8620135670 0x7f86201ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 --2- 172.21.15.50:0/534741319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86200708d0 0x7f86201ca170 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 >> 172.21.15.50:0/534741319 conn(0x7f86200a4670 msgr2=0x7f86200a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 shutdown_connections 2023-10-30T12:07:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 -- 172.21.15.50:0/534741319 wait complete. 2023-10-30T12:07:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.303+0000 7f862771d640 1 librados: shutdown 2023-10-30T12:07:41.315 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:07:41.315 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:07:41.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:07:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.420+0000 7fc75c490640 10 monclient: get_monmap_and_config 2023-10-30T12:07:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.420+0000 7fc75c490640 10 monclient: build_initial_monmap 2023-10-30T12:07:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.420+0000 7fc75c490640 10 monclient: monmap: 2023-10-30T12:07:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:41.421869+0000 2023-10-30T12:07:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:41.421869+0000 2023-10-30T12:07:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:41.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 1 Processor -- start 2023-10-30T12:07:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 1 -- start start 2023-10-30T12:07:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient: init 2023-10-30T12:07:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-10-30T12:07:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc75409a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient(hunting): picked mon.noname-b con 0x7fc754071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc754097fe0 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc75409b170 0x7fc754139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient(hunting): picked mon.noname-c con 0x7fc75409b170 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc754097600 con 0x7fc754097fe0 2023-10-30T12:07:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7540978e0 con 0x7fc75409b170 2023-10-30T12:07:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75409bf50 con 0x7fc754071c00 2023-10-30T12:07:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.421+0000 7fc75c490640 10 monclient(hunting): authenticate will time out at 1306.605003s 2023-10-30T12:07:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc75409a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc759a2b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc754097fe0 0x7fc75409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39826/0 (socket says 172.21.15.50:39826) 2023-10-30T12:07:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc759a2b640 1 -- 172.21.15.50:0/3375792610 learned_addr learned my addr 172.21.15.50:0/3375792610 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 10 monclient(hunting): get_auth_request con 0x7fc754071c00 auth_method 0 2023-10-30T12:07:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 1 -- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 msgr2=0x7fc75409a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:07:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 1 -- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 msgr2=0x7fc75409a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc75409a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc75409a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 1 --2- 172.21.15.50:0/3375792610 >> v2:172.21.15.118:3300/0 conn(0x7fc75409b170 0x7fc754139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979008094 0 0) 0x7fc754097600 con 0x7fc754097fe0 2023-10-30T12:07:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): _negotiate 2023-10-30T12:07:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): my global_id is 104256 2023-10-30T12:07:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc740003440 con 0x7fc754097fe0 2023-10-30T12:07:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): get_auth_request con 0x7fc75409b170 auth_method 0 2023-10-30T12:07:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3983183455 0 0) 0x7fc740003440 con 0x7fc754097fe0 2023-10-30T12:07:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 msgr2=0x7fc75409a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc75409a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 >> v2:172.21.15.118:3300/0 conn(0x7fc75409b170 msgr2=0x7fc754139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 --2- 172.21.15.50:0/3375792610 >> v2:172.21.15.118:3300/0 conn(0x7fc75409b170 0x7fc754139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: _finish_hunting 0 2023-10-30T12:07:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:07:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:07:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75409d9b0 con 0x7fc754097fe0 2023-10-30T12:07:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: _finish_auth 0 2023-10-30T12:07:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: _check_auth_tickets 2023-10-30T12:07:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fc750001670 con 0x7fc754097fe0 2023-10-30T12:07:41.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75c490640 5 monclient: authenticate success, global_id 104256 2023-10-30T12:07:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75c490640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:07:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: dump: 2023-10-30T12:07:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.450 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.451 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:07:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 >> v1:172.21.15.50:6789/0 conn(0x7fc754097fe0 legacy=0x7fc75409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:07:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc740003a60 0x7fc740003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): picked mon.b con 0x7fc740003a60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc74000ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): picked mon.a con 0x7fc740008850 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc74000d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): picked mon.c con 0x7fc754071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540978e0 con 0x7fc740008850 2023-10-30T12:07:41.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75409bf50 con 0x7fc740003a60 2023-10-30T12:07:41.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74000dd20 con 0x7fc754071c00 2023-10-30T12:07:41.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:41.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75922a640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:07:41.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75c490640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:41.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc759a2b640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc74000ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc74000d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc759a2b640 10 monclient(hunting): get_auth_request con 0x7fc740008850 auth_method 0 2023-10-30T12:07:41.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc759a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc759a2b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc759a2b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:41.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75a22c640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc740003a60 0x7fc740003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.422+0000 7fc75aa2d640 10 monclient(hunting): get_auth_request con 0x7fc754071c00 auth_method 0 2023-10-30T12:07:41.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75aa2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75aa2d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75aa2d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:41.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75a22c640 10 monclient(hunting): get_auth_request con 0x7fc740003a60 auth_method 0 2023-10-30T12:07:41.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75a22c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75a22c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75a22c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:07:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75aa2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75aa2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75aa2d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:07:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75a22c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75a22c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc75a22c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:07:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient(hunting): handle_auth_done global_id 104256 payload 293 2023-10-30T12:07:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 1 -- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 msgr2=0x7fc74000d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc754071c00 0x7fc74000d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 1 -- 172.21.15.50:0/3375792610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc740003a60 msgr2=0x7fc740003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc740003a60 0x7fc740003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient: _finish_hunting 0 2023-10-30T12:07:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 1 monclient: found mon.a 2023-10-30T12:07:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:41.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 1 -- 172.21.15.50:0/3375792610 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc740003440 con 0x7fc740008850 2023-10-30T12:07:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient: _finish_auth 0 2023-10-30T12:07:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 10 monclient: _check_auth_tickets 2023-10-30T12:07:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:41.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.423+0000 7fc759a2b640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc74000ace0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c290 tx=0x7fc7400036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc270116d8d8ea6e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750005ad0 con 0x7fc740008850 2023-10-30T12:07:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 10 monclient: dump: 2023-10-30T12:07:41.463 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.464 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.465 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750005c30 con 0x7fc740008850 2023-10-30T12:07:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 1 -- 172.21.15.50:0/3375792610 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75002c8d0 con 0x7fc740008850 2023-10-30T12:07:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 4 set_mon_vals no callback set 2023-10-30T12:07:41.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75922a640 10 monclient: dump: 2023-10-30T12:07:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.467 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.469 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 10 monclient: shutdown 2023-10-30T12:07:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 1 -- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 msgr2=0x7fc74000ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 1 --2- 172.21.15.50:0/3375792610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc74000ace0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c290 tx=0x7fc7400036f0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75aa2d640 1 -- 172.21.15.50:0/3375792610 reap_dead start 2023-10-30T12:07:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 1 -- 172.21.15.50:0/3375792610 shutdown_connections 2023-10-30T12:07:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 1 -- 172.21.15.50:0/3375792610 >> 172.21.15.50:0/3375792610 conn(0x7fc7540a3980 msgr2=0x7fc7540a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:41.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 1 -- 172.21.15.50:0/3375792610 shutdown_connections 2023-10-30T12:07:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.424+0000 7fc75c490640 1 -- 172.21.15.50:0/3375792610 wait complete. 2023-10-30T12:07:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 10 monclient: build_initial_monmap 2023-10-30T12:07:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 10 monclient: monmap: 2023-10-30T12:07:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:41.471 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:41.472 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:41.473 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 1 librados: starting msgr at 2023-10-30T12:07:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 1 librados: starting objecter 2023-10-30T12:07:41.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 1 Processor -- start 2023-10-30T12:07:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 1 -- start start 2023-10-30T12:07:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 1 librados: setting wanted keys 2023-10-30T12:07:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 1 librados: calling monclient init 2023-10-30T12:07:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 10 monclient: init 2023-10-30T12:07:41.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.425+0000 7fc75c490640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc75408f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): picked mon.noname-a con 0x7fc740008850 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7540910f0 0x7fc75408f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): picked mon.noname-b con 0x7fc7540910f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:41.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 0x7fc7541ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): picked mon.noname-c con 0x7fc75408fe60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754090f60 con 0x7fc740008850 2023-10-30T12:07:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754091de0 con 0x7fc7540910f0 2023-10-30T12:07:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540905f0 con 0x7fc75408fe60 2023-10-30T12:07:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient(hunting): authenticate will time out at 1306.609068s 2023-10-30T12:07:41.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc75408f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc75408f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38868/0 (socket says 172.21.15.50:38868) 2023-10-30T12:07:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 -- 172.21.15.50:0/1450406766 learned_addr learned my addr 172.21.15.50:0/1450406766 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): get_auth_request con 0x7fc740008850 auth_method 0 2023-10-30T12:07:41.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 0x7fc7541ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 10 monclient(hunting): get_auth_request con 0x7fc75408fe60 auth_method 0 2023-10-30T12:07:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc759a2b640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7540910f0 0x7fc75408f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 msgr2=0x7fc7541ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:07:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 msgr2=0x7fc7541ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 0x7fc7541ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc759a2b640 10 monclient(hunting): get_auth_request con 0x7fc7540910f0 auth_method 0 2023-10-30T12:07:41.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc759a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc759a2b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc759a2b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75aa2d640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 0x7fc7541ca890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient(hunting): handle_auth_done global_id 104259 payload 824 2023-10-30T12:07:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 msgr2=0x7fc7541ca890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc75408fe60 0x7fc7541ca890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7540910f0 msgr2=0x7fc75408f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7540910f0 0x7fc75408f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient: _finish_hunting 0 2023-10-30T12:07:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 monclient: found mon.noname-a 2023-10-30T12:07:41.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 -- 172.21.15.50:0/1450406766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75409cbc0 con 0x7fc740008850 2023-10-30T12:07:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75a22c640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc75408f3e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc7480060c0 tx=0x7fc748006580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 client.?.objecter ms_handle_connect 0x7fc740008850 2023-10-30T12:07:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7480107d0 con 0x7fc740008850 2023-10-30T12:07:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: dump: 2023-10-30T12:07:41.485 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.486 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.487 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: _finish_auth 0 2023-10-30T12:07:41.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:07:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 5 monclient: authenticate success, global_id 104259 2023-10-30T12:07:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc748010930 con 0x7fc740008850 2023-10-30T12:07:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:41.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient: _renew_subs 2023-10-30T12:07:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748010ce0 con 0x7fc740008850 2023-10-30T12:07:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc754090c40 con 0x7fc740008850 2023-10-30T12:07:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc758a29640 4 set_mon_vals no callback set 2023-10-30T12:07:41.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc746ffd640 10 monclient: dump: 2023-10-30T12:07:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.490 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.491 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 client.104259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient: _renew_subs 2023-10-30T12:07:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7540711a0 con 0x7fc740008850 2023-10-30T12:07:41.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.426+0000 7fc75c490640 1 librados: init done 2023-10-30T12:07:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc75c490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc75c490640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7541cadd0 0x7fc7541cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc75c490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc748014430 con 0x7fc740008850 2023-10-30T12:07:41.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754073ff0 con 0x7fc7541cadd0 2023-10-30T12:07:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7541cadd0 0x7fc7541cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient: get_auth_request con 0x7fc7541cadd0 auth_method 0 2023-10-30T12:07:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc746ffd640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc738039320 0x7fc73803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fc74804b080 con 0x7fc740008850 2023-10-30T12:07:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc746ffd640 10 client.104259.objecter ms_dispatch 0x7fc754000b90 osd_map(234..234 src has 1..234) v4 2023-10-30T12:07:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc746ffd640 3 client.104259.objecter handle_osd_map got epochs [234,234] > 0 2023-10-30T12:07:41.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc746ffd640 3 client.104259.objecter handle_osd_map decoding full epoch 234 2023-10-30T12:07:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc746ffd640 20 client.104259.objecter dump_active .. 0 homeless 2023-10-30T12:07:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc75aa2d640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc738039320 0x7fc73803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.427+0000 7fc759a2b640 10 monclient(hunting): handle_auth_done global_id 104262 payload 824 2023-10-30T12:07:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc759a2b640 1 -- 172.21.15.50:0/1450406766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc754071fe0 con 0x7fc7541cadd0 2023-10-30T12:07:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc75aa2d640 10 monclient: get_auth_request con 0x7fc738039320 auth_method 0 2023-10-30T12:07:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 client.104259.objecter ms_handle_connect 0x7fc7541cadd0 2023-10-30T12:07:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 client.104259.objecter resend_mon_ops 2023-10-30T12:07:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc759a2b640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7541cadd0 0x7fc7541cb1b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc750001050 tx=0x7fc750000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75002cd00 con 0x7fc7541cadd0 2023-10-30T12:07:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 monclient: dump: 2023-10-30T12:07:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.500 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.501 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.501 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 client.104259.objecter ms_handle_connect 0x7fc738039320 2023-10-30T12:07:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc75aa2d640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc738039320 0x7fc73803b7e0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7fc75409fec0 tx=0x7fc74c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc75002ce60 con 0x7fc7541cadd0 2023-10-30T12:07:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7541cadd0 msgr2=0x7fc7541cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7541cadd0 0x7fc7541cb1b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc750001050 tx=0x7fc750000fe0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7541cadd0 msgr2=0x7fc7541cb1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.428+0000 7fc746ffd640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7541cadd0 0x7fc7541cb1b0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc750001050 tx=0x7fc750000fe0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.434+0000 7fc75c490640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.434+0000 7fc75c490640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc75406dbb0 0x7fc75406dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.434+0000 7fc75c490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.434+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754071330 con 0x7fc75406dbb0 2023-10-30T12:07:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc75406dbb0 0x7fc75406dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient: get_auth_request con 0x7fc75406dbb0 auth_method 0 2023-10-30T12:07:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 10 monclient(hunting): handle_auth_done global_id 104265 payload 824 2023-10-30T12:07:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 1 -- 172.21.15.50:0/1450406766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc754092ff0 con 0x7fc75406dbb0 2023-10-30T12:07:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc759a2b640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc75406dbb0 0x7fc75406dfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c290 tx=0x7fc750009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 10 client.104259.objecter ms_handle_connect 0x7fc75406dbb0 2023-10-30T12:07:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 10 client.104259.objecter resend_mon_ops 2023-10-30T12:07:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750050070 con 0x7fc75406dbb0 2023-10-30T12:07:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 10 monclient: dump: 2023-10-30T12:07:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.509 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fc750035980 con 0x7fc75406dbb0 2023-10-30T12:07:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc75406dbb0 msgr2=0x7fc75406dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc75406dbb0 0x7fc75406dfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c290 tx=0x7fc750009030 comp rx=0 tx=0).stop 2023-10-30T12:07:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc75406dbb0 msgr2=0x7fc75406dfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.435+0000 7fc746ffd640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc75406dbb0 0x7fc75406dfc0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c290 tx=0x7fc750009030 comp rx=0 tx=0).stop 2023-10-30T12:07:41.511 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:41.511 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:41.512 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:41.512 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:07:41.512 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 72, 2023-10-30T12:07:41.512 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:41.512 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:41.512 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:07:41.513 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:07:41.513 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:41.513 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:07:41.513 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:41.513 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:41.514 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:41.514 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:41.514 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:41.514 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:41.514 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:41.515 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:41.515 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:41.515 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:41.515 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:41.515 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:41.516 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:41.516 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:41.516 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:41.516 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:41.516 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:41.517 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:41.517 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:41.517 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:41.517 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:41.517 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:41.517 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:41.518 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:41.518 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:41.518 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:41.518 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:41.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:41.520 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:41.520 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:41.520 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:41.520 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:41.520 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:41.521 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:41.521 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:41.521 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:41.521 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:41.521 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:41.522 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:41.522 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:41.522 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:41.522 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:41.523 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:41.523 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:41.523 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:41.523 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:41.523 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:41.523 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:41.524 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:41.524 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:41.524 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:41.524 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:41.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.525 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:41.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.525 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:41.525 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:41.525 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:41.526 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:41.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.526 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:41.526 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:41.526 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:41.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.527 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:41.527 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:41.527 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:41.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:41.528 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:41.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.528 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:41.528 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:41.528 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:41.529 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:41.529 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:41.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.529 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:41.531 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:41.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:41.531 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:41.532 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:41.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:41.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:41.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.532 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:41.533 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:41.533 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:41.533 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:41.533 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:41.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.534 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:41.534 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:41.534 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:41.534 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:41.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:41.535 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:41.535 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:41.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.536 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:41.536 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:41.536 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:41.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:41.536 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:41.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.537 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:41.537 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:41.537 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:41.537 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:41.538 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:41.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:41.538 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:41.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.538 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:41.539 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:41.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.539 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:41.539 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:41.540 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:41.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:41.540 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:41.540 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:07:41.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.540 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:41.541 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:41.541 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:07:41.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:41.541 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:41.541 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:41.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:41.542 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:41.542 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:41.542 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:07:41.542 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:41.543 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:41.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:41.543 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:41.543 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:41.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 10 librados: watch_flush enter 2023-10-30T12:07:41.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 10 librados: watch_flush exit 2023-10-30T12:07:41.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 20 client.104259.objecter shutdown clearing up homeless session... 2023-10-30T12:07:41.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 10 client.104259.objecter successfully canceled tick 2023-10-30T12:07:41.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc738039320 msgr2=0x7fc73803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc738039320 0x7fc73803b7e0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7fc75409fec0 tx=0x7fc74c00b040 comp rx=0 tx=0).stop 2023-10-30T12:07:41.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 10 monclient: shutdown 2023-10-30T12:07:41.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:41.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 msgr2=0x7fc75408f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc75408f3e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc7480060c0 tx=0x7fc748006580 comp rx=0 tx=0).stop 2023-10-30T12:07:41.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75aa2d640 1 -- 172.21.15.50:0/1450406766 reap_dead start 2023-10-30T12:07:41.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 shutdown_connections 2023-10-30T12:07:41.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 --2- 172.21.15.50:0/1450406766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc740008850 0x7fc75408f3e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 >> 172.21.15.50:0/1450406766 conn(0x7fc7540a3980 msgr2=0x7fc754096540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:41.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 shutdown_connections 2023-10-30T12:07:41.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 -- 172.21.15.50:0/1450406766 wait complete. 2023-10-30T12:07:41.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.436+0000 7fc75c490640 1 librados: shutdown 2023-10-30T12:07:41.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:07:41.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:07:41.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:07:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.654+0000 7fc4ed54e640 10 monclient: get_monmap_and_config 2023-10-30T12:07:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.654+0000 7fc4ed54e640 10 monclient: build_initial_monmap 2023-10-30T12:07:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.654+0000 7fc4ed54e640 10 monclient: monmap: 2023-10-30T12:07:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:41.655696+0000 2023-10-30T12:07:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:41.655696+0000 2023-10-30T12:07:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:41.659 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 1 Processor -- start 2023-10-30T12:07:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 1 -- start start 2023-10-30T12:07:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient: init 2023-10-30T12:07:41.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-10-30T12:07:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4e8132970 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc4e80a03d0 0x7fc4e813aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4e80a03d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e813d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4e813b020 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:41.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e80a17d0 con 0x7fc4e8132970 2023-10-30T12:07:41.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e8133460 con 0x7fc4e80a03d0 2023-10-30T12:07:41.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e80a1510 con 0x7fc4e813b020 2023-10-30T12:07:41.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:41.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4ed54e640 10 monclient(hunting): authenticate will time out at 1306.838703s 2023-10-30T12:07:41.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e67fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc4e80a03d0 0x7fc4e813aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e813d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44398/0 (socket says 172.21.15.50:44398) 2023-10-30T12:07:41.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 1 -- 172.21.15.50:0/3372660049 learned_addr learned my addr 172.21.15.50:0/3372660049 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:41.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e67fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc4e80a03d0 0x7fc4e813aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38902/0 (socket says 172.21.15.50:38902) 2023-10-30T12:07:41.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 10 monclient(hunting): get_auth_request con 0x7fc4e813b020 auth_method 0 2023-10-30T12:07:41.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e6ffd640 1 --1- 172.21.15.50:0/3372660049 >> v1:172.21.15.50:6789/0 conn(0x7fc4e8132970 0x7fc4e8132d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39834/0 (socket says 172.21.15.50:39834) 2023-10-30T12:07:41.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e67fc640 10 monclient(hunting): get_auth_request con 0x7fc4e80a03d0 auth_method 0 2023-10-30T12:07:41.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 1 -- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 msgr2=0x7fc4e813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:07:41.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.655+0000 7fc4e77fe640 1 -- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 msgr2=0x7fc4e813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:41.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e77fe640 1 --2- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e813d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e77fe640 1 --2- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e813d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:41.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 10 monclient(hunting): handle_auth_done global_id 104268 payload 274 2023-10-30T12:07:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 1 -- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 msgr2=0x7fc4e813d410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 1 --2- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e813d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 1 -- 172.21.15.50:0/3372660049 >> v1:172.21.15.50:6789/0 conn(0x7fc4e8132970 legacy=0x7fc4e8132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 10 monclient: _finish_hunting 0 2023-10-30T12:07:41.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 1 monclient: found mon.noname-a 2023-10-30T12:07:41.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:41.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 1 -- 172.21.15.50:0/3372660049 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e809f680 con 0x7fc4e80a03d0 2023-10-30T12:07:41.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e5ffb640 1 -- 172.21.15.50:0/3372660049 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063212322 0 0) 0x7fc4e80a17d0 con 0x7fc4e8132970 2023-10-30T12:07:41.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:07:41.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.656+0000 7fc4e67fc640 1 --2- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e80a03d0 0x7fc4e813aae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0097a0 tx=0x7fc4dc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c00716ff22ed167 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 1 -- 172.21.15.50:0/3372660049 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc02f050 con 0x7fc4e80a03d0 2023-10-30T12:07:41.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:41.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: dump: 2023-10-30T12:07:41.689 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.690 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.691 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.691 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.691 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.692 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.692 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.693 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.693 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:07:41.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:07:41.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:41.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 5 monclient: authenticate success, global_id 104268 2023-10-30T12:07:41.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 1 -- 172.21.15.50:0/3372660049 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc031700 con 0x7fc4e80a03d0 2023-10-30T12:07:41.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:41.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:41.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 4 set_mon_vals no callback set 2023-10-30T12:07:41.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 1 -- 172.21.15.50:0/3372660049 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc031a90 con 0x7fc4e80a03d0 2023-10-30T12:07:41.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4e5ffb640 10 monclient: dump: 2023-10-30T12:07:41.698 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.698 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.699 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.699 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.700 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.700 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.700 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.701 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.701 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.701 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 10 monclient: shutdown 2023-10-30T12:07:41.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:41.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 -- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e813aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 --2- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e80a03d0 0x7fc4e813aae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0097a0 tx=0x7fc4dc02b7a0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 -- 172.21.15.50:0/3372660049 shutdown_connections 2023-10-30T12:07:41.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 --2- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 --2- 172.21.15.50:0/3372660049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e80a03d0 0x7fc4e813aae0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 -- 172.21.15.50:0/3372660049 >> 172.21.15.50:0/3372660049 conn(0x7fc4e80a4800 msgr2=0x7fc4e80a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:41.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 -- 172.21.15.50:0/3372660049 shutdown_connections 2023-10-30T12:07:41.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.657+0000 7fc4ed54e640 1 -- 172.21.15.50:0/3372660049 wait complete. 2023-10-30T12:07:41.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 10 monclient: build_initial_monmap 2023-10-30T12:07:41.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 10 monclient: monmap: 2023-10-30T12:07:41.706 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:41.707 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:41.707 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:41.707 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:41.708 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:41.708 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.709 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:41.709 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:41.709 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:41.710 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 1 librados: starting msgr at 2023-10-30T12:07:41.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 1 librados: starting objecter 2023-10-30T12:07:41.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 1 Processor -- start 2023-10-30T12:07:41.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 1 -- start start 2023-10-30T12:07:41.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 1 librados: setting wanted keys 2023-10-30T12:07:41.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 1 librados: calling monclient init 2023-10-30T12:07:41.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.658+0000 7fc4ed54e640 10 monclient: init 2023-10-30T12:07:41.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:41.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:07:41.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4e8132970 0x7fc4e8139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4e8132970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e81ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4e813b020 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:41.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e811a430 0x7fc4e811a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4e811a430 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:41.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e813a730 con 0x7fc4e811a430 2023-10-30T12:07:41.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e809d7c0 con 0x7fc4e8132970 2023-10-30T12:07:41.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e811c2c0 con 0x7fc4e813b020 2023-10-30T12:07:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient(hunting): authenticate will time out at 1306.842172s 2023-10-30T12:07:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e81ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e811a430 0x7fc4e811a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e81ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44414/0 (socket says 172.21.15.50:44414) 2023-10-30T12:07:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 1 -- 172.21.15.50:0/1957855885 learned_addr learned my addr 172.21.15.50:0/1957855885 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:41.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 10 monclient(hunting): get_auth_request con 0x7fc4e813b020 auth_method 0 2023-10-30T12:07:41.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): get_auth_request con 0x7fc4e811a430 auth_method 0 2023-10-30T12:07:41.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 msgr2=0x7fc4e81ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:07:41.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 msgr2=0x7fc4e81ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:41.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e81ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4e8132970 0x7fc4e8139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e67fc640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e81ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:41.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request con 0x7fc4e8132970 auth_method 0 2023-10-30T12:07:41.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:41.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:41.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:41.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_done global_id 104274 payload 824 2023-10-30T12:07:41.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 msgr2=0x7fc4e81ca8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:41.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e813b020 0x7fc4e81ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4e8132970 msgr2=0x7fc4e8139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4e8132970 0x7fc4e8139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:07:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 monclient: found mon.noname-a 2023-10-30T12:07:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 -- 172.21.15.50:0/1957855885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc009400 con 0x7fc4e811a430 2023-10-30T12:07:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e77fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e811a430 0x7fc4e811a8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0007a60 tx=0x7fc4e0007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 client.?.objecter ms_handle_connect 0x7fc4e811a430 2023-10-30T12:07:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:41.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e000db80 con 0x7fc4e811a430 2023-10-30T12:07:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 monclient: dump: 2023-10-30T12:07:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.731 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.732 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:07:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:07:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e000f040 con 0x7fc4e811a430 2023-10-30T12:07:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0015400 con 0x7fc4e811a430 2023-10-30T12:07:41.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 5 monclient: authenticate success, global_id 104274 2023-10-30T12:07:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient: _renew_subs 2023-10-30T12:07:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e811d430 con 0x7fc4e811a430 2023-10-30T12:07:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.659+0000 7fc4e57fa640 4 set_mon_vals no callback set 2023-10-30T12:07:41.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4cf7fe640 10 monclient: dump: 2023-10-30T12:07:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.735 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:41.736 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:41.737 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:41.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 10 client.104274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:41.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 10 monclient: _renew_subs 2023-10-30T12:07:41.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:41.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e811b900 con 0x7fc4e811a430 2023-10-30T12:07:41.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 1 librados: init done 2023-10-30T12:07:41.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:41.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:41.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e8071a70 con 0x7fc4e80a03d0 2023-10-30T12:07:41.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.660+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc4e001e3b0 con 0x7fc4e811a430 2023-10-30T12:07:41.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 10 monclient: get_auth_request con 0x7fc4e80a03d0 auth_method 0 2023-10-30T12:07:41.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:07:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:41.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4cf7fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4d40392f0 0x7fc4d403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:41.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fc4e004aae0 con 0x7fc4e811a430 2023-10-30T12:07:41.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4cf7fe640 10 client.104274.objecter ms_dispatch 0x7fc4e8000b90 osd_map(234..234 src has 1..234) v4 2023-10-30T12:07:41.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4cf7fe640 3 client.104274.objecter handle_osd_map got epochs [234,234] > 0 2023-10-30T12:07:41.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4cf7fe640 3 client.104274.objecter handle_osd_map decoding full epoch 234 2023-10-30T12:07:41.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4cf7fe640 20 client.104274.objecter dump_active .. 0 homeless 2023-10-30T12:07:41.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e67fc640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4d40392f0 0x7fc4d403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e67fc640 10 monclient: get_auth_request con 0x7fc4d40392f0 auth_method 0 2023-10-30T12:07:41.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4cf7fe640 10 client.104274.objecter ms_handle_connect 0x7fc4d40392f0 2023-10-30T12:07:41.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.661+0000 7fc4e67fc640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4d40392f0 0x7fc4d403b7b0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8139ee0 tx=0x7fc4dc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:41.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 10 monclient: get_auth_request con 0x7fc4e80a03d0 auth_method 0 2023-10-30T12:07:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:07:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:41.862+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:07:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 10 monclient: get_auth_request con 0x7fc4e80a03d0 auth_method 0 2023-10-30T12:07:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:07:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:42.263+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:07:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 10 monclient: get_auth_request con 0x7fc4e80a03d0 auth_method 0 2023-10-30T12:07:43.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:07:43.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:07:43.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:43.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:43.064+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:07:44.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.659+0000 7fc4ce7fc640 10 monclient: tick 2023-10-30T12:07:44.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.659+0000 7fc4ce7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:07:44.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.659+0000 7fc4ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:44.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.659+0000 7fc4ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:44.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient: get_auth_request con 0x7fc4e80a03d0 auth_method 0 2023-10-30T12:07:44.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:44.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_done global_id 114101 payload 824 2023-10-30T12:07:44.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.666+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4e809f960 con 0x7fc4e80a03d0 2023-10-30T12:07:44.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80011e0 tx=0x7fc4d8001990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:44.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 10 client.104274.objecter ms_handle_connect 0x7fc4e80a03d0 2023-10-30T12:07:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 10 client.104274.objecter resend_mon_ops 2023-10-30T12:07:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d80177e0 con 0x7fc4e80a03d0 2023-10-30T12:07:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 10 monclient: dump: 2023-10-30T12:07:44.672 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.672 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.672 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.672 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.673 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.673 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.673 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.673 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.673 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc4d8017940 con 0x7fc4e80a03d0 2023-10-30T12:07:44.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:44.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:44.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80011e0 tx=0x7fc4d8001990 comp rx=0 tx=0).stop 2023-10-30T12:07:44.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 msgr2=0x7fc4e811ae20 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:44.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.667+0000 7fc4cf7fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e80a03d0 0x7fc4e811ae20 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80011e0 tx=0x7fc4d8001990 comp rx=0 tx=0).stop 2023-10-30T12:07:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.674+0000 7fc4ed54e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.674+0000 7fc4ed54e640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e81365c0 0x7fc4e81369d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.674+0000 7fc4ed54e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.674+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e8074800 con 0x7fc4e81365c0 2023-10-30T12:07:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e81365c0 0x7fc4e81369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient: get_auth_request con 0x7fc4e81365c0 auth_method 0 2023-10-30T12:07:44.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 10 monclient(hunting): handle_auth_done global_id 114104 payload 824 2023-10-30T12:07:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 1 -- 172.21.15.50:0/1957855885 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc4e811b360 con 0x7fc4e81365c0 2023-10-30T12:07:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4e6ffd640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e81365c0 0x7fc4e81369d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc4d801d000 tx=0x7fc4d8001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4cf7fe640 10 client.104274.objecter ms_handle_connect 0x7fc4e81365c0 2023-10-30T12:07:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4cf7fe640 10 client.104274.objecter resend_mon_ops 2023-10-30T12:07:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8029070 con 0x7fc4e81365c0 2023-10-30T12:07:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.675+0000 7fc4cf7fe640 10 monclient: dump: 2023-10-30T12:07:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.683 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fc4d8018440 con 0x7fc4e81365c0 2023-10-30T12:07:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4cf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4cf7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e81365c0 msgr2=0x7fc4e81369d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4cf7fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e81365c0 0x7fc4e81369d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc4d801d000 tx=0x7fc4d8001080 comp rx=0 tx=0).stop 2023-10-30T12:07:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4cf7fe640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e81365c0 msgr2=0x7fc4e81369d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:44.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4cf7fe640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4e81365c0 0x7fc4e81369d0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc4d801d000 tx=0x7fc4d8001080 comp rx=0 tx=0).stop 2023-10-30T12:07:44.685 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:44.685 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:44.685 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:44.686 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:07:44.686 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 72, 2023-10-30T12:07:44.686 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:44.686 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:44.686 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:07:44.687 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:07:44.687 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.687 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:07:44.687 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:44.687 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:44.688 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:44.688 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:44.688 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:44.688 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:44.688 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:44.689 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:44.689 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:44.689 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:44.689 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:44.689 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:44.689 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:44.690 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.690 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:44.690 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:44.690 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:44.690 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:44.690 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:44.691 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:44.691 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:44.691 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:44.691 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:44.691 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:44.692 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:44.692 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:44.692 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.692 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:44.692 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:44.693 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:44.693 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:44.693 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:44.693 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:44.693 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:44.694 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:44.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:44.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:44.694 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:44.694 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:44.694 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:44.695 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:44.695 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:44.695 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:44.695 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:44.695 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:44.695 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:44.697 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:44.697 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:44.697 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:44.697 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.697 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:44.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.698 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:44.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.698 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:44.698 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:44.698 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:44.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:44.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:44.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:44.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:44.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:44.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:44.701 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:44.701 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:44.701 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:44.701 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:44.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.702 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:44.702 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:44.702 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:44.703 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:44.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.703 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:44.703 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:44.703 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:44.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:44.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.704 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:44.705 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:44.705 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:44.706 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:44.706 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:44.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.706 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:44.707 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:44.707 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:44.707 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:44.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.707 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:44.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:44.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:44.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:44.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.709 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:44.710 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:44.710 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:44.710 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:44.710 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:44.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.711 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.711 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:44.711 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:44.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.712 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:44.712 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:44.712 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:44.712 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.712 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.712 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:44.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.713 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:44.713 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:44.713 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:44.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.714 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:44.714 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:44.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 10 librados: watch_flush enter 2023-10-30T12:07:44.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 10 librados: watch_flush exit 2023-10-30T12:07:44.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 20 client.104274.objecter shutdown clearing up homeless session... 2023-10-30T12:07:44.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 10 client.104274.objecter successfully canceled tick 2023-10-30T12:07:44.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4d40392f0 msgr2=0x7fc4d403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4d40392f0 0x7fc4d403b7b0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8139ee0 tx=0x7fc4dc008030 comp rx=0 tx=0).stop 2023-10-30T12:07:44.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 10 monclient: shutdown 2023-10-30T12:07:44.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:44.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e811a430 msgr2=0x7fc4e811a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 --2- 172.21.15.50:0/1957855885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4e811a430 0x7fc4e811a8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0007a60 tx=0x7fc4e0007e60 comp rx=0 tx=0).stop 2023-10-30T12:07:44.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4e77fe640 1 -- 172.21.15.50:0/1957855885 reap_dead start 2023-10-30T12:07:44.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4e77fe640 1 -- 172.21.15.50:0/1957855885 reap_dead start 2023-10-30T12:07:44.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 shutdown_connections 2023-10-30T12:07:44.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 >> 172.21.15.50:0/1957855885 conn(0x7fc4e80a4800 msgr2=0x7fc4e80a2050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:44.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 shutdown_connections 2023-10-30T12:07:44.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 -- 172.21.15.50:0/1957855885 wait complete. 2023-10-30T12:07:44.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.676+0000 7fc4ed54e640 1 librados: shutdown 2023-10-30T12:07:44.719 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:07:44.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:07:44.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:07:44.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.841+0000 7f3c718bb640 10 monclient: get_monmap_and_config 2023-10-30T12:07:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.841+0000 7f3c718bb640 10 monclient: build_initial_monmap 2023-10-30T12:07:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.841+0000 7f3c718bb640 10 monclient: monmap: 2023-10-30T12:07:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:44.842387+0000 2023-10-30T12:07:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:44.842387+0000 2023-10-30T12:07:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:44.845 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:44.845 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:44.845 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:44.845 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.841+0000 7f3c718bb640 1 Processor -- start 2023-10-30T12:07:44.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.841+0000 7f3c718bb640 1 -- start start 2023-10-30T12:07:44.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.841+0000 7f3c718bb640 10 monclient: init 2023-10-30T12:07:44.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:44.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T12:07:44.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3c6c0a8010 0x7f3c6c132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6c0a8010 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:44.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c6c099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:44.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3c6c132bd0 0x7f3c6c135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6c132bd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:44.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c0a7190 con 0x7f3c6c099a40 2023-10-30T12:07:44.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c0a83f0 con 0x7f3c6c132bd0 2023-10-30T12:07:44.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c0a7600 con 0x7f3c6c0a8010 2023-10-30T12:07:44.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:44.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c718bb640 10 monclient(hunting): authenticate will time out at 1310.025463s 2023-10-30T12:07:44.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3c6c132bd0 0x7f3c6c135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3c6c132bd0 0x7f3c6c135de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38944/0 (socket says 172.21.15.50:38944) 2023-10-30T12:07:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 -- 172.21.15.50:0/23736180 learned_addr learned my addr 172.21.15.50:0/23736180 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f3c6c132bd0 auth_method 0 2023-10-30T12:07:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 1 --2- 172.21.15.50:0/23736180 >> v2:172.21.15.118:3300/0 conn(0x7f3c6c0a8010 0x7f3c6c132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 10 monclient(hunting): get_auth_request con 0x7f3c6c0a8010 auth_method 0 2023-10-30T12:07:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient(hunting): handle_auth_done global_id 104277 payload 274 2023-10-30T12:07:44.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 -- 172.21.15.50:0/23736180 >> v2:172.21.15.118:3300/0 conn(0x7f3c6c0a8010 msgr2=0x7f3c6c132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:44.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 --2- 172.21.15.50:0/23736180 >> v2:172.21.15.118:3300/0 conn(0x7f3c6c0a8010 0x7f3c6c132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 -- 172.21.15.50:0/23736180 >> v1:172.21.15.118:6789/0 conn(0x7f3c6c099a40 legacy=0x7f3c6c1356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:07:44.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:07:44.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:44.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:44.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.842+0000 7f3c6b7fe640 1 -- 172.21.15.50:0/23736180 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c09a440 con 0x7f3c6c132bd0 2023-10-30T12:07:44.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6affd640 1 --2- 172.21.15.50:0/23736180 >> v2:172.21.15.118:3300/0 conn(0x7f3c6c0a8010 0x7f3c6c132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:44.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6b7fe640 1 --2- 172.21.15.50:0/23736180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c135de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0a8640 tx=0x7f3c6402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a624f87ef21cad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:44.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 1 -- 172.21.15.50:0/23736180 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6402c9f0 con 0x7f3c6c132bd0 2023-10-30T12:07:44.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:44.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: dump: 2023-10-30T12:07:44.864 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.864 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.864 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.864 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.864 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.865 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.865 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.865 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.865 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.865 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:07:44.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:07:44.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:44.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 1 -- 172.21.15.50:0/23736180 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6402cb50 con 0x7f3c6c132bd0 2023-10-30T12:07:44.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c718bb640 5 monclient: authenticate success, global_id 104277 2023-10-30T12:07:44.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:44.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c718bb640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:44.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c718bb640 4 set_mon_vals no callback set 2023-10-30T12:07:44.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 1 -- 172.21.15.50:0/23736180 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c640343f0 con 0x7f3c6c132bd0 2023-10-30T12:07:44.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:44.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.843+0000 7f3c6a7fc640 10 monclient: dump: 2023-10-30T12:07:44.868 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.868 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.868 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.868 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.869 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.869 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.869 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.869 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.869 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.870 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 10 monclient: shutdown 2023-10-30T12:07:44.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:44.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 -- 172.21.15.50:0/23736180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 msgr2=0x7f3c6c135de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 --2- 172.21.15.50:0/23736180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c135de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0a8640 tx=0x7f3c6402be50 comp rx=0 tx=0).stop 2023-10-30T12:07:44.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 -- 172.21.15.50:0/23736180 shutdown_connections 2023-10-30T12:07:44.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 --2- 172.21.15.50:0/23736180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c135de0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 --2- 172.21.15.50:0/23736180 >> v2:172.21.15.118:3300/0 conn(0x7f3c6c0a8010 0x7f3c6c132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 -- 172.21.15.50:0/23736180 >> 172.21.15.50:0/23736180 conn(0x7f3c6c0a4a10 msgr2=0x7f3c6c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:44.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 -- 172.21.15.50:0/23736180 shutdown_connections 2023-10-30T12:07:44.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.844+0000 7f3c718bb640 1 -- 172.21.15.50:0/23736180 wait complete. 2023-10-30T12:07:44.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 10 monclient: build_initial_monmap 2023-10-30T12:07:44.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 10 monclient: monmap: 2023-10-30T12:07:44.872 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:44.872 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:44.873 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:44.873 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:44.873 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:44.873 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.873 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:44.874 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:44.874 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:44.874 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 1 librados: starting msgr at 2023-10-30T12:07:44.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 1 librados: starting objecter 2023-10-30T12:07:44.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 1 Processor -- start 2023-10-30T12:07:44.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 1 -- start start 2023-10-30T12:07:44.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 1 librados: setting wanted keys 2023-10-30T12:07:44.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 1 librados: calling monclient init 2023-10-30T12:07:44.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 10 monclient: init 2023-10-30T12:07:44.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.845+0000 7f3c718bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:44.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:07:44.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c0a8010 0x7f3c6c11e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): picked mon.noname-b con 0x7f3c6c0a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:44.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c11ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6c132bd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:44.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3c6c11f160 0x7f3c6c1d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6c11f160 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:44.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c122810 con 0x7f3c6c132bd0 2023-10-30T12:07:44.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c120cd0 con 0x7f3c6c0a8010 2023-10-30T12:07:44.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1357f0 con 0x7f3c6c11f160 2023-10-30T12:07:44.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:44.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient(hunting): authenticate will time out at 1310.029072s 2023-10-30T12:07:44.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c11ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3c6c11f160 0x7f3c6c1d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c11ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38956/0 (socket says 172.21.15.50:38956) 2023-10-30T12:07:44.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 -- 172.21.15.50:0/769024188 learned_addr learned my addr 172.21.15.50:0/769024188 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:44.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): get_auth_request con 0x7f3c6c132bd0 auth_method 0 2023-10-30T12:07:44.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f3c6c11f160 auth_method 0 2023-10-30T12:07:44.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c0a8010 0x7f3c6c11e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 10 monclient(hunting): get_auth_request con 0x7f3c6c0a8010 auth_method 0 2023-10-30T12:07:44.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient(hunting): handle_auth_done global_id 104280 payload 824 2023-10-30T12:07:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3c6c11f160 msgr2=0x7f3c6c1d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3c6c11f160 0x7f3c6c1d05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c0a8010 msgr2=0x7f3c6c11e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c0a8010 0x7f3c6c11e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient: _finish_hunting 0 2023-10-30T12:07:44.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 monclient: found mon.noname-a 2023-10-30T12:07:44.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:44.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:44.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 -- 172.21.15.50:0/769024188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c009400 con 0x7f3c6c132bd0 2023-10-30T12:07:44.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6b7fe640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3c6c11f160 0x7f3c6c1d05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:44.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c6affd640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c0a8010 0x7f3c6c11e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:44.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c627fc640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c11ec20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c3f0 tx=0x7f3c5400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:44.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 client.?.objecter ms_handle_connect 0x7f3c6c132bd0 2023-10-30T12:07:44.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:44.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c540052d0 con 0x7f3c6c132bd0 2023-10-30T12:07:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: dump: 2023-10-30T12:07:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: _finish_auth 0 2023-10-30T12:07:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:07:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c54005430 con 0x7f3c6c132bd0 2023-10-30T12:07:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54003830 con 0x7f3c6c132bd0 2023-10-30T12:07:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 5 monclient: authenticate success, global_id 104280 2023-10-30T12:07:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c68ff9640 10 monclient: dump: 2023-10-30T12:07:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.892 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.892 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.892 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.892 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.892 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.893 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c69ffb640 4 set_mon_vals no callback set 2023-10-30T12:07:44.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient: _renew_subs 2023-10-30T12:07:44.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:44.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.846+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c64009b30 con 0x7f3c6c132bd0 2023-10-30T12:07:44.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 10 client.104280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:44.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 10 monclient: _renew_subs 2023-10-30T12:07:44.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:44.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c1223f0 con 0x7f3c6c132bd0 2023-10-30T12:07:44.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 1 librados: init done 2023-10-30T12:07:44.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:44.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c099a40 0x7f3c6c02c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c073fd0 con 0x7f3c6c099a40 2023-10-30T12:07:44.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.847+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3c54003990 con 0x7f3c6c132bd0 2023-10-30T12:07:44.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c099a40 0x7f3c6c02c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3c4c0392f0 0x7f3c4c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient: get_auth_request con 0x7f3c6c099a40 auth_method 0 2023-10-30T12:07:44.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6931+0+0 (secure 0 0 0) 0x7f3c5404a3d0 con 0x7f3c6c132bd0 2023-10-30T12:07:44.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 10 client.104280.objecter ms_dispatch 0x7f3c6c000b90 osd_map(238..238 src has 1..238) v4 2023-10-30T12:07:44.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 3 client.104280.objecter handle_osd_map got epochs [238,238] > 0 2023-10-30T12:07:44.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 3 client.104280.objecter handle_osd_map decoding full epoch 238 2023-10-30T12:07:44.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 20 client.104280.objecter dump_active .. 0 homeless 2023-10-30T12:07:44.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6b7fe640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3c4c0392f0 0x7f3c4c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6b7fe640 10 monclient: get_auth_request con 0x7f3c4c0392f0 auth_method 0 2023-10-30T12:07:44.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 10 monclient(hunting): handle_auth_done global_id 94447 payload 824 2023-10-30T12:07:44.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 1 -- 172.21.15.50:0/769024188 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c6c09aa00 con 0x7f3c6c099a40 2023-10-30T12:07:44.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6affd640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c099a40 0x7f3c6c02c5b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009f60 tx=0x7f3c5c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:44.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 10 client.104280.objecter ms_handle_connect 0x7f3c6c099a40 2023-10-30T12:07:44.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 10 client.104280.objecter resend_mon_ops 2023-10-30T12:07:44.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c004240 con 0x7f3c6c099a40 2023-10-30T12:07:44.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:44.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 10 monclient: dump: 2023-10-30T12:07:44.900 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.900 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.901 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.901 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.901 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.901 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.901 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.901 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.902 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.902 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c68ff9640 10 client.104280.objecter ms_handle_connect 0x7f3c4c0392f0 2023-10-30T12:07:44.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.848+0000 7f3c6b7fe640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3c4c0392f0 0x7f3c4c03b7b0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7f3c64035000 tx=0x7f3c64030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:44.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.849+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f3c5c0043a0 con 0x7f3c6c099a40 2023-10-30T12:07:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.849+0000 7f3c68ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.849+0000 7f3c68ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.849+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c099a40 msgr2=0x7f3c6c02c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.849+0000 7f3c68ff9640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c099a40 0x7f3c6c02c5b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009f60 tx=0x7f3c5c009f90 comp rx=0 tx=0).stop 2023-10-30T12:07:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.849+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c099a40 msgr2=0x7f3c6c02c5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:44.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.849+0000 7f3c68ff9640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c099a40 0x7f3c6c02c5b0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009f60 tx=0x7f3c5c009f90 comp rx=0 tx=0).stop 2023-10-30T12:07:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c718bb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c718bb640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c11f570 0x7f3c6c060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c718bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c09a720 con 0x7f3c6c11f570 2023-10-30T12:07:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c11f570 0x7f3c6c060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient: get_auth_request con 0x7f3c6c11f570 auth_method 0 2023-10-30T12:07:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 10 monclient(hunting): handle_auth_done global_id 94450 payload 824 2023-10-30T12:07:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.855+0000 7f3c6affd640 1 -- 172.21.15.50:0/769024188 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3c6c133080 con 0x7f3c6c11f570 2023-10-30T12:07:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 10 client.104280.objecter ms_handle_connect 0x7f3c6c11f570 2023-10-30T12:07:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c6affd640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c11f570 0x7f3c6c060fc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c01b000 tx=0x7f3c5c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 10 client.104280.objecter resend_mon_ops 2023-10-30T12:07:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c01ec30 con 0x7f3c6c11f570 2023-10-30T12:07:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 10 monclient: dump: 2023-10-30T12:07:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:44.911 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f3c5c004510 con 0x7f3c6c11f570 2023-10-30T12:07:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c11f570 msgr2=0x7f3c6c060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c11f570 0x7f3c6c060fc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c01b000 tx=0x7f3c5c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c11f570 msgr2=0x7f3c6c060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:44.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c68ff9640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c11f570 0x7f3c6c060fc0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c01b000 tx=0x7f3c5c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.913 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:44.913 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:44.914 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:44.914 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:07:44.914 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 72, 2023-10-30T12:07:44.914 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:44.914 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:44.915 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:07:44.915 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:07:44.915 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.915 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:07:44.915 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:44.916 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:44.916 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:44.916 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:44.916 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:44.916 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:44.917 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:44.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:44.917 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:44.917 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:44.917 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:44.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:44.918 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:44.918 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:44.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:44.919 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:44.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:44.921 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:44.922 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:44.922 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:44.922 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:44.922 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:44.922 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:44.923 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:44.923 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:44.923 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:44.923 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:44.923 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:44.923 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:44.924 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:44.924 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:44.924 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:44.924 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:44.925 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:44.925 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:44.925 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:44.925 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:44.925 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:44.926 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:44.926 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:44.926 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:44.926 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.926 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:44.927 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:44.928 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:44.928 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:44.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.928 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:44.928 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:44.929 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:44.929 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:44.930 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.930 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:44.931 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:44.931 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:44.931 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:44.931 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:44.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.932 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:44.932 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:44.932 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:44.932 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:44.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.933 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:44.933 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:44.933 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.934 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:44.934 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:44.934 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.934 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:44.936 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:44.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.936 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:44.936 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:44.937 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:44.937 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:44.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.937 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:44.937 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:44.938 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:44.938 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:44.938 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:44.939 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.939 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.939 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:44.939 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:44.940 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:44.940 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:44.940 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:44.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.940 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.941 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:44.941 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:44.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.941 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:44.941 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:44.942 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:44.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.942 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.942 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:07:44.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.943 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:44.943 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:44.943 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:07:44.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.943 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.944 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:44.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.944 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:44.944 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:44.944 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:07:44.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.945 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:44.945 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:07:44.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:44.945 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:44.945 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:44.946 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:44.946 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:44.946 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:44.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:44.946 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:44.947 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:44.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 10 librados: watch_flush enter 2023-10-30T12:07:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 10 librados: watch_flush exit 2023-10-30T12:07:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 20 client.104280.objecter shutdown clearing up homeless session... 2023-10-30T12:07:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 10 client.104280.objecter successfully canceled tick 2023-10-30T12:07:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3c4c0392f0 msgr2=0x7f3c4c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3c4c0392f0 0x7f3c4c03b7b0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7f3c64035000 tx=0x7f3c64030000 comp rx=0 tx=0).stop 2023-10-30T12:07:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 10 monclient: shutdown 2023-10-30T12:07:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 msgr2=0x7f3c6c11ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c718bb640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c11ec20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c3f0 tx=0x7f3c5400c8b0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.856+0000 7f3c6b7fe640 1 -- 172.21.15.50:0/769024188 reap_dead start 2023-10-30T12:07:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.857+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 shutdown_connections 2023-10-30T12:07:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.857+0000 7f3c718bb640 1 --2- 172.21.15.50:0/769024188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c6c132bd0 0x7f3c6c11ec20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.857+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 >> 172.21.15.50:0/769024188 conn(0x7f3c6c0a4a10 msgr2=0x7f3c6c133fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.857+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 shutdown_connections 2023-10-30T12:07:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.857+0000 7f3c718bb640 1 -- 172.21.15.50:0/769024188 wait complete. 2023-10-30T12:07:44.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:44.857+0000 7f3c718bb640 1 librados: shutdown 2023-10-30T12:07:44.952 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:07:44.952 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:07:44.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:07:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.057+0000 7f3860339640 10 monclient: get_monmap_and_config 2023-10-30T12:07:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.057+0000 7f3860339640 10 monclient: build_initial_monmap 2023-10-30T12:07:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.057+0000 7f3860339640 10 monclient: monmap: 2023-10-30T12:07:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:45.058828+0000 2023-10-30T12:07:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:45.058828+0000 2023-10-30T12:07:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:45.062 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 Processor -- start 2023-10-30T12:07:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 -- start start 2023-10-30T12:07:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient: init 2023-10-30T12:07:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:45.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:07:45.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38580a2dd0 0x7f38580a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:45.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): picked mon.noname-b con 0x7f38580a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:45.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f385806a8e0 0x7f38580a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:45.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): picked mon.noname-c con 0x7f385806a8e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:45.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3858071400 0x7f38580a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:45.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): picked mon.noname-a con 0x7f3858071400 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:45.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:45.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38580a31b0 con 0x7f3858071400 2023-10-30T12:07:45.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:45.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385806a6a0 con 0x7f385806a8e0 2023-10-30T12:07:45.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:45.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385806be90 con 0x7f38580a2dd0 2023-10-30T12:07:45.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:45.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.058+0000 7f3860339640 10 monclient(hunting): authenticate will time out at 1310.241978s 2023-10-30T12:07:45.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38580a2dd0 0x7f38580a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:45.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3858071400 0x7f38580a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:45.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3858071400 0x7f38580a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38968/0 (socket says 172.21.15.50:38968) 2023-10-30T12:07:45.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 -- 172.21.15.50:0/4263729876 learned_addr learned my addr 172.21.15.50:0/4263729876 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:45.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): get_auth_request con 0x7f3858071400 auth_method 0 2023-10-30T12:07:45.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:45.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:45.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:45.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38580a2dd0 0x7f38580a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44482/0 (socket says 172.21.15.50:44482) 2023-10-30T12:07:45.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 10 monclient(hunting): get_auth_request con 0x7f38580a2dd0 auth_method 0 2023-10-30T12:07:45.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:45.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:45.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:45.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:45.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:45.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 1 --2- 172.21.15.50:0/4263729876 >> v2:172.21.15.118:3300/0 conn(0x7f385806a8e0 0x7f38580a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:45.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:45.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:45.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:45.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:45.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 10 monclient(hunting): get_auth_request con 0x7f385806a8e0 auth_method 0 2023-10-30T12:07:45.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:45.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:45.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:45.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:45.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:45.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385d8d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:45.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient(hunting): handle_auth_done global_id 104283 payload 274 2023-10-30T12:07:45.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 -- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38580a2dd0 msgr2=0x7f38580a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:45.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 --2- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38580a2dd0 0x7f38580a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 -- 172.21.15.50:0/4263729876 >> v2:172.21.15.118:3300/0 conn(0x7f385806a8e0 msgr2=0x7f38580a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:45.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 --2- 172.21.15.50:0/4263729876 >> v2:172.21.15.118:3300/0 conn(0x7f385806a8e0 0x7f38580a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient: _finish_hunting 0 2023-10-30T12:07:45.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 monclient: found mon.noname-a 2023-10-30T12:07:45.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:45.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:45.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e8d6640 1 -- 172.21.15.50:0/4263729876 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385809e190 con 0x7f3858071400 2023-10-30T12:07:45.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.059+0000 7f385e0d5640 1 --2- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38580a2dd0 0x7f38580a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:45.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385e8d6640 1 --2- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f38580a5140 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3858069e30 tx=0x7f385002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa25dfbab0732fc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:45.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 1 -- 172.21.15.50:0/4263729876 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385002c960 con 0x7f3858071400 2023-10-30T12:07:45.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:45.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:45.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: dump: 2023-10-30T12:07:45.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:45.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:45.081 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.082 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:45.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:45.082 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:45.082 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:45.082 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:45.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:45.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: _finish_auth 0 2023-10-30T12:07:45.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: _check_auth_tickets 2023-10-30T12:07:45.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:45.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 1 -- 172.21.15.50:0/4263729876 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385002cac0 con 0x7f3858071400 2023-10-30T12:07:45.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 5 monclient: authenticate success, global_id 104283 2023-10-30T12:07:45.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:45.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:45.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:45.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 1 -- 172.21.15.50:0/4263729876 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385002ce50 con 0x7f3858071400 2023-10-30T12:07:45.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:45.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 4 set_mon_vals no callback set 2023-10-30T12:07:45.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:45.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f385d0d3640 10 monclient: dump: 2023-10-30T12:07:45.085 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:45.086 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:45.086 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.086 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.086 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:45.086 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:45.087 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:45.087 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:45.087 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:45.087 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:45.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 10 monclient: shutdown 2023-10-30T12:07:45.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:45.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 1 -- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 msgr2=0x7f38580a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:45.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 1 --2- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f38580a5140 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3858069e30 tx=0x7f385002bdc0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 1 -- 172.21.15.50:0/4263729876 shutdown_connections 2023-10-30T12:07:45.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 1 --2- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f38580a5140 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 1 --2- 172.21.15.50:0/4263729876 >> v2:172.21.15.118:3300/0 conn(0x7f385806a8e0 0x7f38580a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 1 --2- 172.21.15.50:0/4263729876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38580a2dd0 0x7f38580a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.060+0000 7f3860339640 1 -- 172.21.15.50:0/4263729876 >> 172.21.15.50:0/4263729876 conn(0x7f38580a22b0 msgr2=0x7f38580a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:45.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 -- 172.21.15.50:0/4263729876 shutdown_connections 2023-10-30T12:07:45.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 -- 172.21.15.50:0/4263729876 wait complete. 2023-10-30T12:07:45.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 10 monclient: build_initial_monmap 2023-10-30T12:07:45.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 10 monclient: monmap: 2023-10-30T12:07:45.090 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:45.090 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:45.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:45.090 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:45.091 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:45.091 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:45.091 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:45.091 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:45.091 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:45.092 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:45.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 librados: starting msgr at 2023-10-30T12:07:45.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 librados: starting objecter 2023-10-30T12:07:45.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 Processor -- start 2023-10-30T12:07:45.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 -- start start 2023-10-30T12:07:45.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 librados: setting wanted keys 2023-10-30T12:07:45.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 1 librados: calling monclient init 2023-10-30T12:07:45.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.061+0000 7f3860339640 10 monclient: init 2023-10-30T12:07:45.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:45.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:45.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f385813ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:45.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): picked mon.noname-a con 0x7f3858071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:45.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38580a2dd0 0x7f3858122710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:45.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): picked mon.noname-b con 0x7f38580a2dd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:45.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3858122c50 0x7f38581230e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:45.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): picked mon.noname-c con 0x7f3858122c50 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:45.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:45.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38581cafa0 con 0x7f3858071400 2023-10-30T12:07:45.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:45.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3858126780 con 0x7f38580a2dd0 2023-10-30T12:07:45.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:45.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385806b0a0 con 0x7f3858122c50 2023-10-30T12:07:45.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:45.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f3860339640 10 monclient(hunting): authenticate will time out at 1310.245351s 2023-10-30T12:07:45.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f385813ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:45.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3858122c50 0x7f38581230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:45.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3858122c50 0x7f38581230e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44486/0 (socket says 172.21.15.50:44486) 2023-10-30T12:07:45.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 -- 172.21.15.50:0/4245965767 learned_addr learned my addr 172.21.15.50:0/4245965767 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:45.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f385813ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38974/0 (socket says 172.21.15.50:38974) 2023-10-30T12:07:45.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): get_auth_request con 0x7f3858122c50 auth_method 0 2023-10-30T12:07:45.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:45.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:45.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:45.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 10 monclient(hunting): get_auth_request con 0x7f3858071400 auth_method 0 2023-10-30T12:07:45.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:45.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:45.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:45.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:45.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:45.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:45.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:45.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:45.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38580a2dd0 0x7f3858122710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:45.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e0d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:45.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 10 monclient(hunting): get_auth_request con 0x7f38580a2dd0 auth_method 0 2023-10-30T12:07:45.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:45.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:45.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:45.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:45.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:45.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385d8d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:45.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient(hunting): handle_auth_done global_id 114113 payload 824 2023-10-30T12:07:45.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 -- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38580a2dd0 msgr2=0x7f3858122710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:45.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38580a2dd0 0x7f3858122710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 -- 172.21.15.50:0/4245965767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 msgr2=0x7f385813ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:45.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f385813ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient: _finish_hunting 0 2023-10-30T12:07:45.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 monclient: found mon.noname-c 2023-10-30T12:07:45.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:45.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:07:45.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 -- 172.21.15.50:0/4245965767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3854009400 con 0x7f3858122c50 2023-10-30T12:07:45.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f385e8d6640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3858122c50 0x7f38581230e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3850036040 tx=0x7f3850004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:45.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f384effd640 10 client.?.objecter ms_handle_connect 0x7f3858122c50 2023-10-30T12:07:45.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.062+0000 7f384effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:45.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 1 -- 172.21.15.50:0/4245965767 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850009510 con 0x7f3858122c50 2023-10-30T12:07:45.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:45.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:45.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: dump: 2023-10-30T12:07:45.105 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:45.105 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:45.105 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.105 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.106 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:45.106 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:45.106 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:45.106 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:45.106 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:45.107 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:45.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: _finish_auth 0 2023-10-30T12:07:45.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: _check_auth_tickets 2023-10-30T12:07:45.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:45.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 5 monclient: authenticate success, global_id 114113 2023-10-30T12:07:45.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 1 -- 172.21.15.50:0/4245965767 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385003e450 con 0x7f3858122c50 2023-10-30T12:07:45.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:45.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 1 -- 172.21.15.50:0/4245965767 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385003e7e0 con 0x7f3858122c50 2023-10-30T12:07:45.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:45.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:45.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 10 monclient: dump: 2023-10-30T12:07:45.108 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:45.109 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:45.109 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.109 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:45.109 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:45.109 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:45.110 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:45.110 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:45.110 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:45.110 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:45.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f385c8d2640 4 set_mon_vals no callback set 2023-10-30T12:07:45.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 10 monclient: _renew_subs 2023-10-30T12:07:45.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:45.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3850009b30 con 0x7f3858122c50 2023-10-30T12:07:45.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 10 client.114113.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:45.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 10 monclient: _renew_subs 2023-10-30T12:07:45.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:45.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3858125ee0 con 0x7f3858122c50 2023-10-30T12:07:45.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f3860339640 1 librados: init done 2023-10-30T12:07:45.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.063+0000 7f384effd640 1 -- 172.21.15.50:0/4245965767 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f385003e940 con 0x7f3858122c50 2023-10-30T12:07:45.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f384effd640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f383c0392f0 0x7f383c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:45.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f3860339640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:45.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f384effd640 1 -- 172.21.15.50:0/4245965767 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6931+0+0 (secure 0 0 0) 0x7f385006daa0 con 0x7f3858122c50 2023-10-30T12:07:45.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f3860339640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:45.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f384effd640 10 client.114113.objecter ms_dispatch 0x7f3858000b90 osd_map(238..238 src has 1..238) v4 2023-10-30T12:07:45.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f3860339640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:45.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f384effd640 3 client.114113.objecter handle_osd_map got epochs [238,238] > 0 2023-10-30T12:07:45.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f384effd640 3 client.114113.objecter handle_osd_map decoding full epoch 238 2023-10-30T12:07:45.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385804e160 con 0x7f3858122c50 2023-10-30T12:07:45.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f385e0d5640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f383c0392f0 0x7f383c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:45.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f385e0d5640 10 monclient: get_auth_request con 0x7f383c0392f0 auth_method 0 2023-10-30T12:07:45.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f384effd640 20 client.114113.objecter dump_active .. 0 homeless 2023-10-30T12:07:45.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f384effd640 10 client.114113.objecter ms_handle_connect 0x7f383c0392f0 2023-10-30T12:07:45.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.064+0000 7f385e0d5640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f383c0392f0 0x7f383c03b7b0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f3854009c50 tx=0x7f3854009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:45.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.069+0000 7f384effd640 1 -- 172.21.15.50:0/4245965767 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3850043070 con 0x7f3858122c50 2023-10-30T12:07:45.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.069+0000 7f384effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:45.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.069+0000 7f384effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:45.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.317+0000 7f3860339640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:07:45.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.317+0000 7f3860339640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:07:45.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.317+0000 7f3860339640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:45.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.317+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f385811f520 con 0x7f3858122c50 2023-10-30T12:07:45.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.318+0000 7f384effd640 1 -- 172.21.15.50:0/4245965767 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f385003c350 con 0x7f3858122c50 2023-10-30T12:07:45.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.318+0000 7f384effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:07:45.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.318+0000 7f384effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:45.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.318+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.318+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=9,logm=61} crc {auth=842426543,config=3140514009,health=1827187281,kv=2905431493,logm=3813407977}) 2023-10-30T12:07:45.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.318+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=1668918021,config=3140514009,health=1638073345,kv=2905431493,logm=4090392985}) 2023-10-30T12:07:45.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.318+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.318+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=9,logm=61} crc {auth=842426543,config=3140514009,health=1827187281,kv=2905431493,logm=3813407977}) 2023-10-30T12:07:45.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.318+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=9,logm=59} crc {auth=1730462629,config=3140514009,health=1638073345,kv=2849054840,logm=293878470}) 2023-10-30T12:07:45.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.319+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.319+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564082696}) 2023-10-30T12:07:45.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.319+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2487057137}) 2023-10-30T12:07:45.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.319+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.319+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564082696}) 2023-10-30T12:07:45.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.319+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3906388541}) 2023-10-30T12:07:45.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 10 librados: watch_flush enter 2023-10-30T12:07:45.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 10 librados: watch_flush exit 2023-10-30T12:07:45.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 20 client.114113.objecter shutdown clearing up homeless session... 2023-10-30T12:07:45.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 10 client.114113.objecter successfully canceled tick 2023-10-30T12:07:45.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.320+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.320+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210610182}) 2023-10-30T12:07:45.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.320+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879512594}) 2023-10-30T12:07:45.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f383c0392f0 msgr2=0x7f383c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:45.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f383c0392f0 0x7f383c03b7b0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f3854009c50 tx=0x7f3854009190 comp rx=0 tx=0).stop 2023-10-30T12:07:45.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 10 monclient: shutdown 2023-10-30T12:07:45.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:45.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3858122c50 msgr2=0x7f38581230e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:45.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.320+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.320+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210610182}) 2023-10-30T12:07:45.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.320+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902866079}) 2023-10-30T12:07:45.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3858122c50 0x7f38581230e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3850036040 tx=0x7f3850004a40 comp rx=0 tx=0).stop 2023-10-30T12:07:45.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 shutdown_connections 2023-10-30T12:07:45.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f383c0392f0 0x7f383c03b7b0 unknown :-1 s=CLOSED pgs=4024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3858122c50 0x7f38581230e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38580a2dd0 0x7f3858122710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 --2- 172.21.15.50:0/4245965767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3858071400 0x7f385813ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:45.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 >> 172.21.15.50:0/4245965767 conn(0x7f38580a22b0 msgr2=0x7f385806e570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:45.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 shutdown_connections 2023-10-30T12:07:45.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 -- 172.21.15.50:0/4245965767 wait complete. 2023-10-30T12:07:45.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:45.320+0000 7f3860339640 1 librados: shutdown 2023-10-30T12:07:45.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.321+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.321+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583732807}) 2023-10-30T12:07:45.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.321+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646043159}) 2023-10-30T12:07:45.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.321+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.321+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583732807}) 2023-10-30T12:07:45.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.321+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194856550}) 2023-10-30T12:07:45.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912131677}) 2023-10-30T12:07:45.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122111290}) 2023-10-30T12:07:45.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912131677}) 2023-10-30T12:07:45.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2588391140}) 2023-10-30T12:07:45.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422443187}) 2023-10-30T12:07:45.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=202273746}) 2023-10-30T12:07:45.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422443187}) 2023-10-30T12:07:45.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.322+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912218670}) 2023-10-30T12:07:45.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.323+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.323+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977499030}) 2023-10-30T12:07:45.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.323+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181623733}) 2023-10-30T12:07:45.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.323+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.323+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977499030}) 2023-10-30T12:07:45.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.323+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=333787896}) 2023-10-30T12:07:45.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.324+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.324+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124005290}) 2023-10-30T12:07:45.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.324+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2099749965}) 2023-10-30T12:07:45.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.324+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.324+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124005290}) 2023-10-30T12:07:45.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.324+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719231357}) 2023-10-30T12:07:45.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.325+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.325+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165556077}) 2023-10-30T12:07:45.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.325+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2156691260}) 2023-10-30T12:07:45.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.325+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.325+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165556077}) 2023-10-30T12:07:45.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.325+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767289980}) 2023-10-30T12:07:45.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.326+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.326+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466334764}) 2023-10-30T12:07:45.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.326+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2825925377}) 2023-10-30T12:07:45.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.326+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.326+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466334764}) 2023-10-30T12:07:45.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.326+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477921290}) 2023-10-30T12:07:45.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167024280}) 2023-10-30T12:07:45.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128087885}) 2023-10-30T12:07:45.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167024280}) 2023-10-30T12:07:45.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=917653510}) 2023-10-30T12:07:45.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239729212}) 2023-10-30T12:07:45.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997233162}) 2023-10-30T12:07:45.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239729212}) 2023-10-30T12:07:45.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.327+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196151322}) 2023-10-30T12:07:45.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.328+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.328+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214205554}) 2023-10-30T12:07:45.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.328+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3889440181}) 2023-10-30T12:07:45.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.328+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.328+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214205554}) 2023-10-30T12:07:45.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.328+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983523140}) 2023-10-30T12:07:45.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.329+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.329+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553543719}) 2023-10-30T12:07:45.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.329+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966436555}) 2023-10-30T12:07:45.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.329+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.329+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553543719}) 2023-10-30T12:07:45.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.329+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363190595}) 2023-10-30T12:07:45.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.330+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.330+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687836088}) 2023-10-30T12:07:45.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.330+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094366238}) 2023-10-30T12:07:45.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.330+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.330+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687836088}) 2023-10-30T12:07:45.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.330+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=837628237}) 2023-10-30T12:07:45.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.331+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.331+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=672761369}) 2023-10-30T12:07:45.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.331+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068985306}) 2023-10-30T12:07:45.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.331+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:07:45.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.331+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=672761369}) 2023-10-30T12:07:45.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.331+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288789578}) 2023-10-30T12:07:45.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.332+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.332+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453833878}) 2023-10-30T12:07:45.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.332+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394183601}) 2023-10-30T12:07:45.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.332+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.332+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453833878}) 2023-10-30T12:07:45.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.332+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337203424}) 2023-10-30T12:07:45.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.333+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.333+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988504455}) 2023-10-30T12:07:45.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.333+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2307682513}) 2023-10-30T12:07:45.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.333+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.333+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988504455}) 2023-10-30T12:07:45.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.333+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015467359}) 2023-10-30T12:07:45.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137411948}) 2023-10-30T12:07:45.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494055658}) 2023-10-30T12:07:45.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137411948}) 2023-10-30T12:07:45.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647888298}) 2023-10-30T12:07:45.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263442610}) 2023-10-30T12:07:45.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2790617502}) 2023-10-30T12:07:45.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.334+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263442610}) 2023-10-30T12:07:45.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.335+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024549936}) 2023-10-30T12:07:45.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.335+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.335+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531207586}) 2023-10-30T12:07:45.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.335+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=379094220}) 2023-10-30T12:07:45.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.335+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.335+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531207586}) 2023-10-30T12:07:45.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.335+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688134612}) 2023-10-30T12:07:45.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.336+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.336+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175207284}) 2023-10-30T12:07:45.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.336+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3213229545}) 2023-10-30T12:07:45.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.336+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.336+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175207284}) 2023-10-30T12:07:45.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.336+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334229760}) 2023-10-30T12:07:45.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.336+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293211660}) 2023-10-30T12:07:45.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2959347508}) 2023-10-30T12:07:45.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293211660}) 2023-10-30T12:07:45.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3955050967}) 2023-10-30T12:07:45.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994098749}) 2023-10-30T12:07:45.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3732029545}) 2023-10-30T12:07:45.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994098749}) 2023-10-30T12:07:45.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.337+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=692987731}) 2023-10-30T12:07:45.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.338+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.338+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806615971}) 2023-10-30T12:07:45.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.338+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321376217}) 2023-10-30T12:07:45.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.338+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.338+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806615971}) 2023-10-30T12:07:45.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.338+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556964698}) 2023-10-30T12:07:45.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.339+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.339+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111474801}) 2023-10-30T12:07:45.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.339+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2758859679}) 2023-10-30T12:07:45.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.339+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.339+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111474801}) 2023-10-30T12:07:45.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.339+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2165901319}) 2023-10-30T12:07:45.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.340+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.340+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280120798}) 2023-10-30T12:07:45.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.340+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1707220848}) 2023-10-30T12:07:45.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.340+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.340+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280120798}) 2023-10-30T12:07:45.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.340+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804943070}) 2023-10-30T12:07:45.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.340+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692158386}) 2023-10-30T12:07:45.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744107095}) 2023-10-30T12:07:45.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692158386}) 2023-10-30T12:07:45.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2093442165}) 2023-10-30T12:07:45.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248599610}) 2023-10-30T12:07:45.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3089496103}) 2023-10-30T12:07:45.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248599610}) 2023-10-30T12:07:45.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.341+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=642055205}) 2023-10-30T12:07:45.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.342+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.342+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3882691576}) 2023-10-30T12:07:45.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.342+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3229195587}) 2023-10-30T12:07:45.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.342+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.342+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3882691576}) 2023-10-30T12:07:45.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.342+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3863263298}) 2023-10-30T12:07:45.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996903523}) 2023-10-30T12:07:45.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854279082}) 2023-10-30T12:07:45.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996903523}) 2023-10-30T12:07:45.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315243376}) 2023-10-30T12:07:45.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973957906}) 2023-10-30T12:07:45.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4041998626}) 2023-10-30T12:07:45.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973957906}) 2023-10-30T12:07:45.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.343+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=298380021}) 2023-10-30T12:07:45.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.344+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.344+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368079857}) 2023-10-30T12:07:45.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.344+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640181823}) 2023-10-30T12:07:45.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.344+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.344+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368079857}) 2023-10-30T12:07:45.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.344+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3151098688}) 2023-10-30T12:07:45.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.345+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.345+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467158782}) 2023-10-30T12:07:45.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.345+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=84695100}) 2023-10-30T12:07:45.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.345+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.345+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467158782}) 2023-10-30T12:07:45.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.345+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845462419}) 2023-10-30T12:07:45.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.346+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.346+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=12} crc {logm=1605345087,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1647937721,mon_config_key=3003229359,monmap=3324057927,osd_metadata=1549492346,osd_pg_creating=2274049588,osd_snap=2733206726}) 2023-10-30T12:07:45.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.346+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=50,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=12} crc {logm=1127136363,mdsmap=3984425268,mgr=2119566901,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1647937722,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_snap=3092174847}) 2023-10-30T12:07:45.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.346+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.346+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=12} crc {logm=1605345087,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1647937721,mon_config_key=3003229359,monmap=3324057927,osd_metadata=1549492346,osd_pg_creating=2274049588,osd_snap=2733206726}) 2023-10-30T12:07:45.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.346+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=375391362,mdsmap=3984425268,mgr=1597429152,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1647937721,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=2274049588,osd_snap=1480326626}) 2023-10-30T12:07:45.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.347+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.347+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=2437870242,osdmap=899799454}) 2023-10-30T12:07:45.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.347+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=614890509,osdmap=3953065577}) 2023-10-30T12:07:45.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.347+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.347+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=2437870242,osdmap=899799454}) 2023-10-30T12:07:45.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.347+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=2424712286,osdmap=1553570474}) 2023-10-30T12:07:45.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.348+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.348+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3879724286}) 2023-10-30T12:07:45.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.348+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1239595617}) 2023-10-30T12:07:45.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.348+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.348+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3879724286}) 2023-10-30T12:07:45.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.348+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=998808620}) 2023-10-30T12:07:45.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.349+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.349+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=474529540}) 2023-10-30T12:07:45.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.349+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=386101591}) 2023-10-30T12:07:45.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.349+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.349+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=474529540}) 2023-10-30T12:07:45.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.349+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4110992390}) 2023-10-30T12:07:45.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.356+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.356+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=4012580832}) 2023-10-30T12:07:45.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.356+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=68} crc {osdmap=762724624}) 2023-10-30T12:07:45.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.356+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:07:45.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.356+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=4012580832}) 2023-10-30T12:07:45.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:07:45.356+0000 7fba881ab640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=65} crc {osdmap=1410798788}) 2023-10-30T12:07:46.354 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:07:46.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:46.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.470+0000 7f852ca0e640 10 monclient: get_monmap_and_config 2023-10-30T12:07:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.470+0000 7f852ca0e640 10 monclient: build_initial_monmap 2023-10-30T12:07:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.470+0000 7f852ca0e640 10 monclient: monmap: 2023-10-30T12:07:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:46.471527+0000 2023-10-30T12:07:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:46.471527+0000 2023-10-30T12:07:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:46.474 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:46.474 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:46.474 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:46.474 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:46.474 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 1 Processor -- start 2023-10-30T12:07:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 1 -- start start 2023-10-30T12:07:46.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient: init 2023-10-30T12:07:46.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:46.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-10-30T12:07:46.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8528131b20 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:46.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8528071bc0 0x7f8528071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient(hunting): picked mon.noname-c con 0x7f8528071bc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:46.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8528136e40 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:46.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852809a9b0 con 0x7f8528136e40 2023-10-30T12:07:46.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8528131f00 con 0x7f8528131b20 2023-10-30T12:07:46.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8528132980 con 0x7f8528071bc0 2023-10-30T12:07:46.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:46.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f852ca0e640 10 monclient(hunting): authenticate will time out at 1311.654787s 2023-10-30T12:07:46.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f8526d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8528136e40 0x7f8528139230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39842/0 (socket says 172.21.15.50:39842) 2023-10-30T12:07:46.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.471+0000 7f8526d9d640 1 -- 172.21.15.50:0/2077881862 learned_addr learned my addr 172.21.15.50:0/2077881862 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:46.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 1 --2- 172.21.15.50:0/2077881862 >> v2:172.21.15.118:3300/0 conn(0x7f8528071bc0 0x7f8528071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): get_auth_request con 0x7f8528071bc0 auth_method 0 2023-10-30T12:07:46.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 1 -- 172.21.15.50:0/2077881862 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215008651 0 0) 0x7f852809a9b0 con 0x7f8528136e40 2023-10-30T12:07:46.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): _negotiate 2023-10-30T12:07:46.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): my global_id is 104289 2023-10-30T12:07:46.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 1 -- 172.21.15.50:0/2077881862 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8510003440 con 0x7f8528136e40 2023-10-30T12:07:46.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 1 -- 172.21.15.50:0/2077881862 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641286891 0 0) 0x7f8528131f00 con 0x7f8528131b20 2023-10-30T12:07:46.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): _negotiate 2023-10-30T12:07:46.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 10 monclient(hunting): my global_id is 94462 2023-10-30T12:07:46.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f852559a640 1 -- 172.21.15.50:0/2077881862 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852809a9b0 con 0x7f8528131b20 2023-10-30T12:07:46.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient(hunting): handle_auth_done global_id 94459 payload 274 2023-10-30T12:07:46.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 1 -- 172.21.15.50:0/2077881862 >> v1:172.21.15.118:6789/0 conn(0x7f8528131b20 legacy=0x7f8528132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 1 -- 172.21.15.50:0/2077881862 >> v1:172.21.15.50:6789/0 conn(0x7f8528136e40 legacy=0x7f8528139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:07:46.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 1 monclient: found mon.noname-c 2023-10-30T12:07:46.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:46.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:07:46.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.472+0000 7f8525d9b640 1 -- 172.21.15.50:0/2077881862 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852809d920 con 0x7f8528071bc0 2023-10-30T12:07:46.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f8525d9b640 1 --2- 172.21.15.50:0/2077881862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528071bc0 0x7f8528071fc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f852809db10 tx=0x7f851c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bf19181914aa86e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 1 -- 172.21.15.50:0/2077881862 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c02e070 con 0x7f8528071bc0 2023-10-30T12:07:46.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:46.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: dump: 2023-10-30T12:07:46.493 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.493 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.493 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.493 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.493 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.494 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.494 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.494 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.494 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.494 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: _finish_auth 0 2023-10-30T12:07:46.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: _check_auth_tickets 2023-10-30T12:07:46.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:46.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 1 -- 172.21.15.50:0/2077881862 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c0315a0 con 0x7f8528071bc0 2023-10-30T12:07:46.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:46.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 1 -- 172.21.15.50:0/2077881862 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c031910 con 0x7f8528071bc0 2023-10-30T12:07:46.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852ca0e640 5 monclient: authenticate success, global_id 94459 2023-10-30T12:07:46.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852ca0e640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:46.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852ca0e640 4 set_mon_vals no callback set 2023-10-30T12:07:46.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:46.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852559a640 10 monclient: dump: 2023-10-30T12:07:46.497 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.497 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.498 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.498 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.498 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.498 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.499 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852ca0e640 10 monclient: shutdown 2023-10-30T12:07:46.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852ca0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:46.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852ca0e640 1 -- 172.21.15.50:0/2077881862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528071bc0 msgr2=0x7f8528071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.473+0000 7f852ca0e640 1 --2- 172.21.15.50:0/2077881862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528071bc0 0x7f8528071fc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f852809db10 tx=0x7f851c02b840 comp rx=0 tx=0).stop 2023-10-30T12:07:46.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.474+0000 7f852ca0e640 1 -- 172.21.15.50:0/2077881862 shutdown_connections 2023-10-30T12:07:46.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.474+0000 7f852ca0e640 1 --2- 172.21.15.50:0/2077881862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528071bc0 0x7f8528071fc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.474+0000 7f852ca0e640 1 -- 172.21.15.50:0/2077881862 >> 172.21.15.50:0/2077881862 conn(0x7f85280a05c0 msgr2=0x7f852809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:46.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.474+0000 7f852ca0e640 1 -- 172.21.15.50:0/2077881862 shutdown_connections 2023-10-30T12:07:46.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.474+0000 7f852ca0e640 1 -- 172.21.15.50:0/2077881862 wait complete. 2023-10-30T12:07:46.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.474+0000 7f852ca0e640 10 monclient: build_initial_monmap 2023-10-30T12:07:46.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.474+0000 7f852ca0e640 10 monclient: monmap: 2023-10-30T12:07:46.501 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:46.501 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:46.501 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:46.502 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:46.502 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:46.502 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.502 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:46.503 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:46.503 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:46.503 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 librados: starting msgr at 2023-10-30T12:07:46.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 librados: starting objecter 2023-10-30T12:07:46.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 Processor -- start 2023-10-30T12:07:46.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 -- start start 2023-10-30T12:07:46.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 librados: setting wanted keys 2023-10-30T12:07:46.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 librados: calling monclient init 2023-10-30T12:07:46.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient: init 2023-10-30T12:07:46.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:46.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:07:46.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8528071bc0 0x7f8528134640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): picked mon.noname-a con 0x7f8528071bc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:46.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8528131b20 0x7f8528134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): picked mon.noname-c con 0x7f8528131b20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:46.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528136e40 0x7f85281350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): picked mon.noname-b con 0x7f8528136e40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:46.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85280a26b0 con 0x7f8528071bc0 2023-10-30T12:07:46.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85280993c0 con 0x7f8528136e40 2023-10-30T12:07:46.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852806d290 con 0x7f8528131b20 2023-10-30T12:07:46.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:46.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852ca0e640 10 monclient(hunting): authenticate will time out at 1311.658852s 2023-10-30T12:07:46.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8528071bc0 0x7f8528134640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f8525d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8528131b20 0x7f8528134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8528071bc0 0x7f8528134640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38978/0 (socket says 172.21.15.50:38978) 2023-10-30T12:07:46.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852659c640 1 -- 172.21.15.50:0/66921132 learned_addr learned my addr 172.21.15.50:0/66921132 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:46.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852659c640 10 monclient(hunting): get_auth_request con 0x7f8528071bc0 auth_method 0 2023-10-30T12:07:46.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.475+0000 7f852659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 10 monclient(hunting): get_auth_request con 0x7f8528131b20 auth_method 0 2023-10-30T12:07:46.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528136e40 0x7f85281350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 10 monclient(hunting): get_auth_request con 0x7f8528136e40 auth_method 0 2023-10-30T12:07:46.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8526d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 10 monclient(hunting): handle_auth_done global_id 104292 payload 824 2023-10-30T12:07:46.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 1 -- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8528131b20 msgr2=0x7f8528134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8528131b20 0x7f8528134b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 1 -- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528136e40 msgr2=0x7f85281350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528136e40 0x7f85281350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 10 monclient: _finish_hunting 0 2023-10-30T12:07:46.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 1 monclient: found mon.noname-a 2023-10-30T12:07:46.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:46.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 1 -- 172.21.15.50:0/66921132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c009400 con 0x7f8528071bc0 2023-10-30T12:07:46.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8525d9b640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8528131b20 0x7f8528134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:46.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852659c640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8528071bc0 0x7f8528134640 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f851800c460 tx=0x7f851800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 client.?.objecter ms_handle_connect 0x7f8528071bc0 2023-10-30T12:07:46.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:46.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 1 -- 172.21.15.50:0/66921132 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85180052d0 con 0x7f8528071bc0 2023-10-30T12:07:46.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:46.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: dump: 2023-10-30T12:07:46.517 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.517 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.517 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.517 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.518 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.518 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.518 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.518 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.518 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.519 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: _finish_auth 0 2023-10-30T12:07:46.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:07:46.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:46.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 5 monclient: authenticate success, global_id 104292 2023-10-30T12:07:46.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 10 monclient: _renew_subs 2023-10-30T12:07:46.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852806c9f0 con 0x7f8528071bc0 2023-10-30T12:07:46.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 1 -- 172.21.15.50:0/66921132 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8518005450 con 0x7f8528071bc0 2023-10-30T12:07:46.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:46.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 1 -- 172.21.15.50:0/66921132 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8518003880 con 0x7f8528071bc0 2023-10-30T12:07:46.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 10 client.104292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:46.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:46.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8516ffd640 10 monclient: dump: 2023-10-30T12:07:46.521 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.522 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.522 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.522 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.522 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.522 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.523 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.523 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.523 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.523 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 10 monclient: _renew_subs 2023-10-30T12:07:46.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8528136070 con 0x7f8528071bc0 2023-10-30T12:07:46.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f852ca0e640 1 librados: init done 2023-10-30T12:07:46.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.476+0000 7f8524d99640 4 set_mon_vals no callback set 2023-10-30T12:07:46.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f8516ffd640 1 -- 172.21.15.50:0/66921132 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f85180039e0 con 0x7f8528071bc0 2023-10-30T12:07:46.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f852ca0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:46.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f852ca0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:46.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f852ca0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852806cf10 con 0x7f8528071bc0 2023-10-30T12:07:46.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f8516ffd640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85080392f0 0x7f850803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f8516ffd640 1 -- 172.21.15.50:0/66921132 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f851804a950 con 0x7f8528071bc0 2023-10-30T12:07:46.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f8516ffd640 10 client.104292.objecter ms_dispatch 0x7f8528000b90 osd_map(239..239 src has 1..239) v4 2023-10-30T12:07:46.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f8516ffd640 3 client.104292.objecter handle_osd_map got epochs [239,239] > 0 2023-10-30T12:07:46.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.477+0000 7f8516ffd640 3 client.104292.objecter handle_osd_map decoding full epoch 239 2023-10-30T12:07:46.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.478+0000 7f8516ffd640 20 client.104292.objecter dump_active .. 0 homeless 2023-10-30T12:07:46.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.478+0000 7f8525d9b640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85080392f0 0x7f850803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.478+0000 7f8525d9b640 10 monclient: get_auth_request con 0x7f85080392f0 auth_method 0 2023-10-30T12:07:46.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.478+0000 7f8516ffd640 10 client.104292.objecter ms_handle_connect 0x7f85080392f0 2023-10-30T12:07:46.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.478+0000 7f8525d9b640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85080392f0 0x7f850803b7b0 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f851c036000 tx=0x7f851c03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.482+0000 7f8516ffd640 1 -- 172.21.15.50:0/66921132 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f852806cf10 con 0x7f8528071bc0 2023-10-30T12:07:46.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.482+0000 7f8516ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:46.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.482+0000 7f8516ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.723+0000 7f852ca0e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.723+0000 7f852ca0e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.723+0000 7f852ca0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.723+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f852806cf10 con 0x7f8528071bc0 2023-10-30T12:07:46.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.723+0000 7f8516ffd640 1 -- 172.21.15.50:0/66921132 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f852806cf10 con 0x7f8528071bc0 2023-10-30T12:07:46.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.723+0000 7f8516ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:46.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.723+0000 7f8516ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:46.727 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 10 librados: watch_flush enter 2023-10-30T12:07:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 10 librados: watch_flush exit 2023-10-30T12:07:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 20 client.104292.objecter shutdown clearing up homeless session... 2023-10-30T12:07:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 10 client.104292.objecter successfully canceled tick 2023-10-30T12:07:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85080392f0 msgr2=0x7f850803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85080392f0 0x7f850803b7b0 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f851c036000 tx=0x7f851c03d040 comp rx=0 tx=0).stop 2023-10-30T12:07:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 10 monclient: shutdown 2023-10-30T12:07:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8528071bc0 msgr2=0x7f8528134640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8528071bc0 0x7f8528134640 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f851800c460 tx=0x7f851800c920 comp rx=0 tx=0).stop 2023-10-30T12:07:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 shutdown_connections 2023-10-30T12:07:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85080392f0 0x7f850803b7b0 unknown :-1 s=CLOSED pgs=4025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8528136e40 0x7f85281350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8528131b20 0x7f8528134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 --2- 172.21.15.50:0/66921132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8528071bc0 0x7f8528134640 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 >> 172.21.15.50:0/66921132 conn(0x7f85280a05c0 msgr2=0x7f8528094500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 shutdown_connections 2023-10-30T12:07:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.725+0000 7f852ca0e640 1 -- 172.21.15.50:0/66921132 wait complete. 2023-10-30T12:07:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.726+0000 7f852ca0e640 1 librados: shutdown 2023-10-30T12:07:46.737 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:07:46.737 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:07:46.738 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:07:46.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:07:46.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.843+0000 7feb125b0640 10 monclient: get_monmap_and_config 2023-10-30T12:07:46.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.843+0000 7feb125b0640 10 monclient: build_initial_monmap 2023-10-30T12:07:46.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.844+0000 7feb125b0640 10 monclient: monmap: 2023-10-30T12:07:46.846 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:46.846 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.846 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:46.845052+0000 2023-10-30T12:07:46.846 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:46.845052+0000 2023-10-30T12:07:46.847 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:46.847 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.847 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:46.847 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:46.847 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:46.848 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:46.848 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:46.848 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.844+0000 7feb125b0640 1 Processor -- start 2023-10-30T12:07:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.844+0000 7feb125b0640 1 -- start start 2023-10-30T12:07:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.844+0000 7feb125b0640 10 monclient: init 2023-10-30T12:07:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:46.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-10-30T12:07:46.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb0c0a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:46.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7feb0c0a2450 0x7feb0c13aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient(hunting): picked mon.noname-c con 0x7feb0c0a2450 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:46.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb0c13af40 0x7feb0c13d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient(hunting): picked mon.noname-a con 0x7feb0c13af40 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c0a4190 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c0a1630 con 0x7feb0c13af40 2023-10-30T12:07:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c09a2e0 con 0x7feb0c0a2450 2023-10-30T12:07:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb125b0640 10 monclient(hunting): authenticate will time out at 1312.028259s 2023-10-30T12:07:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb0c13af40 0x7feb0c13d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb0c13af40 0x7feb0c13d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38988/0 (socket says 172.21.15.50:38988) 2023-10-30T12:07:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 -- 172.21.15.50:0/2446107784 learned_addr learned my addr 172.21.15.50:0/2446107784 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): get_auth_request con 0x7feb0c13af40 auth_method 0 2023-10-30T12:07:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb0b7fe640 1 --2- 172.21.15.50:0/2446107784 >> v2:172.21.15.118:3300/0 conn(0x7feb0c0a2450 0x7feb0c13aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient(hunting): handle_auth_done global_id 104295 payload 274 2023-10-30T12:07:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 -- 172.21.15.50:0/2446107784 >> v2:172.21.15.118:3300/0 conn(0x7feb0c0a2450 msgr2=0x7feb0c13aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 --2- 172.21.15.50:0/2446107784 >> v2:172.21.15.118:3300/0 conn(0x7feb0c0a2450 0x7feb0c13aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 -- 172.21.15.50:0/2446107784 >> v1:172.21.15.118:6789/0 conn(0x7feb0c0a2ae0 legacy=0x7feb0c0a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient: _finish_hunting 0 2023-10-30T12:07:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 monclient: found mon.noname-a 2023-10-30T12:07:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb10b4d640 1 -- 172.21.15.50:0/2446107784 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c09a8a0 con 0x7feb0c13af40 2023-10-30T12:07:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb0b7fe640 10 monclient: get_auth_request con 0x7feb0c0a2450 auth_method 0 2023-10-30T12:07:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.845+0000 7feb0b7fe640 1 --2- 172.21.15.50:0/2446107784 >> v2:172.21.15.118:3300/0 conn(0x7feb0c0a2450 0x7feb0c13aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:07:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb10b4d640 1 --2- 172.21.15.50:0/2446107784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c13af40 0x7feb0c13d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0a4c80 tx=0x7feb0402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c64d523ce602d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 1 -- 172.21.15.50:0/2446107784 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0402c960 con 0x7feb0c13af40 2023-10-30T12:07:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:46.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: dump: 2023-10-30T12:07:46.867 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.868 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.868 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.868 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.868 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.868 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.869 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.869 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.869 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.869 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: _finish_auth 0 2023-10-30T12:07:46.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: _check_auth_tickets 2023-10-30T12:07:46.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:46.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb125b0640 5 monclient: authenticate success, global_id 104295 2023-10-30T12:07:46.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb125b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 1 -- 172.21.15.50:0/2446107784 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0402cac0 con 0x7feb0c13af40 2023-10-30T12:07:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb125b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 1 -- 172.21.15.50:0/2446107784 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0402ce50 con 0x7feb0c13af40 2023-10-30T12:07:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb125b0640 4 set_mon_vals no callback set 2023-10-30T12:07:46.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:46.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.846+0000 7feb0affd640 10 monclient: dump: 2023-10-30T12:07:46.873 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.875 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.875 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.876 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.876 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.876 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 10 monclient: shutdown 2023-10-30T12:07:46.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:46.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 -- 172.21.15.50:0/2446107784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c13af40 msgr2=0x7feb0c13d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 --2- 172.21.15.50:0/2446107784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c13af40 0x7feb0c13d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0a4c80 tx=0x7feb0402bdc0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 -- 172.21.15.50:0/2446107784 shutdown_connections 2023-10-30T12:07:46.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 --2- 172.21.15.50:0/2446107784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c13af40 0x7feb0c13d330 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 --2- 172.21.15.50:0/2446107784 >> v2:172.21.15.118:3300/0 conn(0x7feb0c0a2450 0x7feb0c13aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 -- 172.21.15.50:0/2446107784 >> 172.21.15.50:0/2446107784 conn(0x7feb0c0a80b0 msgr2=0x7feb0c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 -- 172.21.15.50:0/2446107784 shutdown_connections 2023-10-30T12:07:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 1 -- 172.21.15.50:0/2446107784 wait complete. 2023-10-30T12:07:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 10 monclient: build_initial_monmap 2023-10-30T12:07:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.847+0000 7feb125b0640 10 monclient: monmap: 2023-10-30T12:07:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:46.882 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 librados: starting msgr at 2023-10-30T12:07:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 librados: starting objecter 2023-10-30T12:07:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 Processor -- start 2023-10-30T12:07:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 -- start start 2023-10-30T12:07:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 librados: setting wanted keys 2023-10-30T12:07:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 librados: calling monclient init 2023-10-30T12:07:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient: init 2023-10-30T12:07:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:07:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0a2450 0x7feb0c1ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): picked mon.noname-b con 0x7feb0c0a2450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c0a2ae0 0x7feb0c136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): picked mon.noname-a con 0x7feb0c0a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb0c13af40 0x7feb0c1371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): picked mon.noname-c con 0x7feb0c13af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c139560 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0a1790 con 0x7feb0c0a2450 2023-10-30T12:07:46.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c11e4b0 con 0x7feb0c13af40 2023-10-30T12:07:46.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:46.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb125b0640 10 monclient(hunting): authenticate will time out at 1312.031852s 2023-10-30T12:07:46.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c0a2ae0 0x7feb0c136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c0a2ae0 0x7feb0c136c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38994/0 (socket says 172.21.15.50:38994) 2023-10-30T12:07:46.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0b7fe640 1 -- 172.21.15.50:0/1448744510 learned_addr learned my addr 172.21.15.50:0/1448744510 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:46.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0b7fe640 10 monclient(hunting): get_auth_request con 0x7feb0c0a2ae0 auth_method 0 2023-10-30T12:07:46.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb10b4d640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb0c13af40 0x7feb0c1371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.848+0000 7feb0bfff640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0a2450 0x7feb0c1ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb10b4d640 10 monclient(hunting): get_auth_request con 0x7feb0c13af40 auth_method 0 2023-10-30T12:07:46.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb10b4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb10b4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb10b4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0bfff640 10 monclient(hunting): get_auth_request con 0x7feb0c0a2450 auth_method 0 2023-10-30T12:07:46.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb10b4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb10b4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb10b4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 10 monclient(hunting): handle_auth_done global_id 104298 payload 824 2023-10-30T12:07:46.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb0c13af40 msgr2=0x7feb0c1371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7feb0c13af40 0x7feb0c1371c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0a2450 msgr2=0x7feb0c1ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb0c0a2450 0x7feb0c1ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:07:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:07:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 1 -- 172.21.15.50:0/1448744510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb04009b30 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0b7fe640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c0a2ae0 0x7feb0c136c80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feb0000c7f0 tx=0x7feb0000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 client.?.objecter ms_handle_connect 0x7feb0c0a2ae0 2023-10-30T12:07:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00007a30 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:46.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: dump: 2023-10-30T12:07:46.897 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.897 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.898 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.898 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.898 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.898 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.899 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: _finish_auth 0 2023-10-30T12:07:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:07:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb00007b90 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 5 monclient: authenticate success, global_id 104298 2023-10-30T12:07:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 10 monclient: _renew_subs 2023-10-30T12:07:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c11e790 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0000ce40 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb08ff9640 10 monclient: dump: 2023-10-30T12:07:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.905 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb0a7fc640 4 set_mon_vals no callback set 2023-10-30T12:07:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 10 client.104298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 10 monclient: _renew_subs 2023-10-30T12:07:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c138e50 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.849+0000 7feb125b0640 1 librados: init done 2023-10-30T12:07:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7feb000113f0 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb125b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb125b0640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c137700 0x7feb0c137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb125b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c073ff0 con 0x7feb0c137700 2023-10-30T12:07:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c137700 0x7feb0c137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient: get_auth_request con 0x7feb0c137700 auth_method 0 2023-10-30T12:07:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb08ff9640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feaf8039320 0x7feaf803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6275+0+0 (secure 0 0 0) 0x7feb0004ae20 con 0x7feb0c0a2ae0 2023-10-30T12:07:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb08ff9640 10 client.104298.objecter ms_dispatch 0x7feb0c000b90 osd_map(240..240 src has 1..240) v4 2023-10-30T12:07:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb08ff9640 3 client.104298.objecter handle_osd_map got epochs [240,240] > 0 2023-10-30T12:07:46.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb08ff9640 3 client.104298.objecter handle_osd_map decoding full epoch 240 2023-10-30T12:07:46.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb08ff9640 20 client.104298.objecter dump_active .. 0 homeless 2023-10-30T12:07:46.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 10 monclient(hunting): handle_auth_done global_id 104301 payload 824 2023-10-30T12:07:46.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb0bfff640 1 -- 172.21.15.50:0/1448744510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb0c09a5c0 con 0x7feb0c137700 2023-10-30T12:07:46.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.850+0000 7feb10b4d640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feaf8039320 0x7feaf803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb10b4d640 10 monclient: get_auth_request con 0x7feaf8039320 auth_method 0 2023-10-30T12:07:46.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 client.104298.objecter ms_handle_connect 0x7feb0c137700 2023-10-30T12:07:46.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 client.104298.objecter resend_mon_ops 2023-10-30T12:07:46.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb0bfff640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c137700 0x7feb0c137ae0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feafc00f7a0 tx=0x7feafc00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc004a40 con 0x7feb0c137700 2023-10-30T12:07:46.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:46.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 monclient: dump: 2023-10-30T12:07:46.913 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.913 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.915 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.915 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.915 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.915 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 client.104298.objecter ms_handle_connect 0x7feaf8039320 2023-10-30T12:07:46.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb10b4d640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feaf8039320 0x7feaf803b7e0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7feb04035000 tx=0x7feb04030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7feafc003940 con 0x7feb0c137700 2023-10-30T12:07:46.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:46.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:46.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c137700 msgr2=0x7feb0c137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c137700 0x7feb0c137ae0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feafc00f7a0 tx=0x7feafc00fc60 comp rx=0 tx=0).stop 2023-10-30T12:07:46.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c137700 msgr2=0x7feb0c137ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:46.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.851+0000 7feb08ff9640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c137700 0x7feb0c137ae0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feafc00f7a0 tx=0x7feafc00fc60 comp rx=0 tx=0).stop 2023-10-30T12:07:46.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.857+0000 7feb125b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:46.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.857+0000 7feb125b0640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c1332d0 0x7feb0c1336e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:46.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.857+0000 7feb125b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:46.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.857+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c11e920 con 0x7feb0c1332d0 2023-10-30T12:07:46.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c1332d0 0x7feb0c1336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:46.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient: get_auth_request con 0x7feb0c1332d0 auth_method 0 2023-10-30T12:07:46.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:46.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:46.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:46.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 10 monclient(hunting): handle_auth_done global_id 104304 payload 824 2023-10-30T12:07:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 1 -- 172.21.15.50:0/1448744510 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb0c11e620 con 0x7feb0c1332d0 2023-10-30T12:07:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb0bfff640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c1332d0 0x7feb0c1336e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feafc018000 tx=0x7feafc003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 10 client.104298.objecter ms_handle_connect 0x7feb0c1332d0 2023-10-30T12:07:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 10 client.104298.objecter resend_mon_ops 2023-10-30T12:07:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc023070 con 0x7feb0c1332d0 2023-10-30T12:07:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 10 monclient: dump: 2023-10-30T12:07:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:46.925 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7feafc0043c0 con 0x7feb0c1332d0 2023-10-30T12:07:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c1332d0 msgr2=0x7feb0c1336e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c1332d0 0x7feb0c1336e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feafc018000 tx=0x7feafc003760 comp rx=0 tx=0).stop 2023-10-30T12:07:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c1332d0 msgr2=0x7feb0c1336e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb08ff9640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c1332d0 0x7feb0c1336e0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feafc018000 tx=0x7feafc003760 comp rx=0 tx=0).stop 2023-10-30T12:07:46.926 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:46.926 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:46.927 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:46.927 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:07:46.927 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 72, 2023-10-30T12:07:46.927 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:46.927 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:46.928 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:07:46.928 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:07:46.928 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:46.928 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:07:46.928 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:46.929 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:46.929 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:46.929 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:46.929 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:46.929 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:46.929 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:46.930 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:46.930 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:46.930 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:46.930 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:46.930 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:46.931 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:46.931 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:46.931 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:46.931 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:46.931 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:46.931 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:46.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:46.932 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:46.932 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:46.932 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:46.932 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:46.933 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:46.933 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:46.933 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:46.933 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:46.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.933 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:46.934 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:46.934 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:46.934 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:46.934 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:46.935 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:46.935 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:46.935 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:46.935 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:46.935 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:46.936 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:46.936 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:46.936 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:46.936 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:46.936 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:46.937 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:46.937 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:46.937 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:46.937 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:46.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:46.937 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:46.938 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:46.938 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:46.938 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:46.938 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:46.938 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:46.938 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:46.939 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:46.939 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:46.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.939 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:46.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.940 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:46.940 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:46.940 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:46.940 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:46.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.941 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:46.941 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:46.941 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:46.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.941 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:46.942 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:46.942 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:46.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:46.942 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:46.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.943 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:46.943 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:46.943 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:46.943 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:46.943 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:46.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.944 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:46.944 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:46.944 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:46.944 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:46.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.945 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:46.945 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:46.945 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:46.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.946 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:46.946 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:46.946 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:46.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:46.947 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:46.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.947 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:46.947 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:46.947 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:46.948 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:46.948 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:46.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.948 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:46.949 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:46.949 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:46.949 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:46.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.949 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:46.949 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:46.950 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:46.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.950 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:46.950 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:46.951 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:46.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:46.951 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:46.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.951 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:46.952 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:46.952 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:46.952 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:46.952 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:46.952 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:46.952 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:46.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.953 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:46.953 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:46.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.953 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:46.953 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:46.954 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:46.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:46.954 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:46.954 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:07:46.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.955 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:46.955 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:46.955 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:07:46.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:46.956 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:46.956 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:46.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:46.956 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:46.956 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:46.957 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:07:46.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:46.957 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:46.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:46.957 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:46.958 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:46.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb125b0640 10 librados: watch_flush enter 2023-10-30T12:07:46.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb125b0640 10 librados: watch_flush exit 2023-10-30T12:07:46.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.858+0000 7feb125b0640 20 client.104298.objecter shutdown clearing up homeless session... 2023-10-30T12:07:46.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 10 client.104298.objecter successfully canceled tick 2023-10-30T12:07:46.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feaf8039320 msgr2=0x7feaf803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7feaf8039320 0x7feaf803b7e0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7feb04035000 tx=0x7feb04030000 comp rx=0 tx=0).stop 2023-10-30T12:07:46.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 10 monclient: shutdown 2023-10-30T12:07:46.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:46.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c0a2ae0 msgr2=0x7feb0c136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:46.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 --2- 172.21.15.50:0/1448744510 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb0c0a2ae0 0x7feb0c136c80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feb0000c7f0 tx=0x7feb0000cbf0 comp rx=0 tx=0).stop 2023-10-30T12:07:46.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb10b4d640 1 -- 172.21.15.50:0/1448744510 reap_dead start 2023-10-30T12:07:46.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb10b4d640 1 -- 172.21.15.50:0/1448744510 reap_dead start 2023-10-30T12:07:46.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 shutdown_connections 2023-10-30T12:07:46.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 >> 172.21.15.50:0/1448744510 conn(0x7feb0c0a80b0 msgr2=0x7feb0c0a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:46.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 shutdown_connections 2023-10-30T12:07:46.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 -- 172.21.15.50:0/1448744510 wait complete. 2023-10-30T12:07:46.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:46.859+0000 7feb125b0640 1 librados: shutdown 2023-10-30T12:07:46.963 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:07:46.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:07:46.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:07:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.069+0000 7f24570c2640 10 monclient: get_monmap_and_config 2023-10-30T12:07:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.069+0000 7f24570c2640 10 monclient: build_initial_monmap 2023-10-30T12:07:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.069+0000 7f24570c2640 10 monclient: monmap: 2023-10-30T12:07:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:47.070159+0000 2023-10-30T12:07:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:47.070159+0000 2023-10-30T12:07:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:47.073 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.069+0000 7f24570c2640 1 Processor -- start 2023-10-30T12:07:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.069+0000 7f24570c2640 1 -- start start 2023-10-30T12:07:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.069+0000 7f24570c2640 10 monclient: init 2023-10-30T12:07:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-10-30T12:07:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24500a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24500a2450 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f245013c120 0x7f245013e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient(hunting): picked mon.noname-b con 0x7f245013c120 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24500a4190 con 0x7f24500a2450 2023-10-30T12:07:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245009a2e0 con 0x7f24500a2ae0 2023-10-30T12:07:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24500a1630 con 0x7f245013c120 2023-10-30T12:07:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:47.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f24570c2640 10 monclient(hunting): authenticate will time out at 1312.253442s 2023-10-30T12:07:47.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f244ffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f24500a2450 0x7f245013aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39854/0 (socket says 172.21.15.50:39854) 2023-10-30T12:07:47.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f244ffff640 1 -- 172.21.15.50:0/226406562 learned_addr learned my addr 172.21.15.50:0/226406562 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:47.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f245013c120 0x7f245013e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): get_auth_request con 0x7f245013c120 auth_method 0 2023-10-30T12:07:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient(hunting): handle_auth_done global_id 114122 payload 274 2023-10-30T12:07:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f244f7fe640 1 -- 172.21.15.50:0/226406562 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848492757 0 0) 0x7f2444001110 con 0x7f24500a2450 2023-10-30T12:07:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 1 -- 172.21.15.50:0/226406562 >> v1:172.21.15.118:6789/0 conn(0x7f24500a2ae0 legacy=0x7f24500a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 1 -- 172.21.15.50:0/226406562 >> v1:172.21.15.50:6789/0 conn(0x7f24500a2450 legacy=0x7f245013aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient: _finish_hunting 0 2023-10-30T12:07:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 1 monclient: found mon.noname-b 2023-10-30T12:07:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:07:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f245565f640 1 -- 172.21.15.50:0/226406562 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245009a8a0 con 0x7f245013c120 2023-10-30T12:07:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.070+0000 7f244f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:07:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f245565f640 1 --2- 172.21.15.50:0/226406562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f245013c120 0x7f245013e510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f24400097d0 tx=0x7f244002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c355caed5daa5c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 1 -- 172.21.15.50:0/226406562 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440031770 con 0x7f245013c120 2023-10-30T12:07:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:07:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: dump: 2023-10-30T12:07:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.097 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:07:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:07:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 1 -- 172.21.15.50:0/226406562 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24400318d0 con 0x7f245013c120 2023-10-30T12:07:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f24570c2640 5 monclient: authenticate success, global_id 114122 2023-10-30T12:07:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f24570c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f24570c2640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f24570c2640 4 set_mon_vals no callback set 2023-10-30T12:07:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 1 -- 172.21.15.50:0/226406562 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440031c60 con 0x7f245013c120 2023-10-30T12:07:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.071+0000 7f244f7fe640 10 monclient: dump: 2023-10-30T12:07:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.102 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 10 monclient: shutdown 2023-10-30T12:07:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 1 -- 172.21.15.50:0/226406562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f245013c120 msgr2=0x7f245013e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 1 --2- 172.21.15.50:0/226406562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f245013c120 0x7f245013e510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f24400097d0 tx=0x7f244002b810 comp rx=0 tx=0).stop 2023-10-30T12:07:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 1 -- 172.21.15.50:0/226406562 shutdown_connections 2023-10-30T12:07:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 1 --2- 172.21.15.50:0/226406562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f245013c120 0x7f245013e510 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 1 -- 172.21.15.50:0/226406562 >> 172.21.15.50:0/226406562 conn(0x7f24500a80b0 msgr2=0x7f24500a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 1 -- 172.21.15.50:0/226406562 shutdown_connections 2023-10-30T12:07:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 1 -- 172.21.15.50:0/226406562 wait complete. 2023-10-30T12:07:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 10 monclient: build_initial_monmap 2023-10-30T12:07:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.072+0000 7f24570c2640 10 monclient: monmap: 2023-10-30T12:07:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:47.106 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 librados: starting msgr at 2023-10-30T12:07:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 librados: starting objecter 2023-10-30T12:07:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 Processor -- start 2023-10-30T12:07:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 -- start start 2023-10-30T12:07:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 librados: setting wanted keys 2023-10-30T12:07:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 librados: calling monclient init 2023-10-30T12:07:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient: init 2023-10-30T12:07:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:07:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24500a2450 0x7f24501caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): picked mon.noname-c con 0x7f24500a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24500a2ae0 0x7f2450136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): picked mon.noname-a con 0x7f24500a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f245013c120 0x7f24501374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): picked mon.noname-b con 0x7f245013c120 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2450132810 con 0x7f24500a2ae0 2023-10-30T12:07:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24500a1790 con 0x7f245013c120 2023-10-30T12:07:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245013b7f0 con 0x7f24500a2450 2023-10-30T12:07:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.073+0000 7f24570c2640 10 monclient(hunting): authenticate will time out at 1312.256958s 2023-10-30T12:07:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24500a2450 0x7f24501caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24500a2ae0 0x7f2450136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24500a2ae0 0x7f2450136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39022/0 (socket says 172.21.15.50:39022) 2023-10-30T12:07:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24500a2450 0x7f24501caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44512/0 (socket says 172.21.15.50:44512) 2023-10-30T12:07:47.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 -- 172.21.15.50:0/1392869185 learned_addr learned my addr 172.21.15.50:0/1392869185 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:47.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): get_auth_request con 0x7f24500a2ae0 auth_method 0 2023-10-30T12:07:47.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 10 monclient(hunting): get_auth_request con 0x7f24500a2450 auth_method 0 2023-10-30T12:07:47.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f245013c120 0x7f24501374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 10 monclient(hunting): get_auth_request con 0x7f245013c120 auth_method 0 2023-10-30T12:07:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f245565f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient(hunting): handle_auth_done global_id 104310 payload 824 2023-10-30T12:07:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24500a2450 msgr2=0x7f24501caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24500a2450 0x7f24501caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f245013c120 msgr2=0x7f24501374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f245013c120 0x7f24501374c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:07:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 monclient: found mon.noname-a 2023-10-30T12:07:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 -- 172.21.15.50:0/1392869185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440009400 con 0x7f24500a2ae0 2023-10-30T12:07:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f2454e5e640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24500a2450 0x7f24501caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244ffff640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24500a2ae0 0x7f2450136f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f244400c780 tx=0x7f244400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 client.?.objecter ms_handle_connect 0x7f24500a2ae0 2023-10-30T12:07:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24440072d0 con 0x7f24500a2ae0 2023-10-30T12:07:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: dump: 2023-10-30T12:07:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.129 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:07:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:07:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2444007450 con 0x7f24500a2ae0 2023-10-30T12:07:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 5 monclient: authenticate success, global_id 104310 2023-10-30T12:07:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 10 monclient: _renew_subs 2023-10-30T12:07:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2450099bc0 con 0x7f24500a2ae0 2023-10-30T12:07:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444001d10 con 0x7f24500a2ae0 2023-10-30T12:07:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244d7fa640 10 monclient: dump: 2023-10-30T12:07:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.133 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 10 client.104310.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 10 monclient: _renew_subs 2023-10-30T12:07:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f245013a750 con 0x7f24500a2ae0 2023-10-30T12:07:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f24570c2640 1 librados: init done 2023-10-30T12:07:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.074+0000 7f244effd640 4 set_mon_vals no callback set 2023-10-30T12:07:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f24570c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f24570c2640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2450137a00 0x7f2450137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f24570c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2450073ff0 con 0x7f2450137a00 2023-10-30T12:07:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f24440047e0 con 0x7f24500a2ae0 2023-10-30T12:07:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2450137a00 0x7f2450137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient: get_auth_request con 0x7f2450137a00 auth_method 0 2023-10-30T12:07:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f244d7fa640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f243c0392f0 0x7f243c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6275+0+0 (secure 0 0 0) 0x7f2444049c40 con 0x7f24500a2ae0 2023-10-30T12:07:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f244d7fa640 10 client.104310.objecter ms_dispatch 0x7f2450000b90 osd_map(240..240 src has 1..240) v4 2023-10-30T12:07:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f244d7fa640 3 client.104310.objecter handle_osd_map got epochs [240,240] > 0 2023-10-30T12:07:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f244d7fa640 3 client.104310.objecter handle_osd_map decoding full epoch 240 2023-10-30T12:07:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 10 monclient(hunting): handle_auth_done global_id 114128 payload 824 2023-10-30T12:07:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.075+0000 7f2454e5e640 1 -- 172.21.15.50:0/1392869185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f245009a5c0 con 0x7f2450137a00 2023-10-30T12:07:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f245565f640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f243c0392f0 0x7f243c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 20 client.104310.objecter dump_active .. 0 homeless 2023-10-30T12:07:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f245565f640 10 monclient: get_auth_request con 0x7f243c0392f0 auth_method 0 2023-10-30T12:07:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 client.104310.objecter ms_handle_connect 0x7f2450137a00 2023-10-30T12:07:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f2454e5e640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2450137a00 0x7f2450137de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f244800ffd0 tx=0x7f244800d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 client.104310.objecter resend_mon_ops 2023-10-30T12:07:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448001e20 con 0x7f2450137a00 2023-10-30T12:07:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 monclient: dump: 2023-10-30T12:07:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.144 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 client.104310.objecter ms_handle_connect 0x7f243c0392f0 2023-10-30T12:07:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f245565f640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f243c0392f0 0x7f243c03b7b0 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f2440036000 tx=0x7f244003e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f244800de60 con 0x7f2450137a00 2023-10-30T12:07:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2450137a00 msgr2=0x7f2450137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2450137a00 0x7f2450137de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f244800ffd0 tx=0x7f244800d520 comp rx=0 tx=0).stop 2023-10-30T12:07:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2450137a00 msgr2=0x7f2450137de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.076+0000 7f244d7fa640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2450137a00 0x7f2450137de0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.082+0000 7f24570c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.082+0000 7f24570c2640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24501332d0 0x7f24501336e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.082+0000 7f24570c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.082+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24500a28a0 con 0x7f24501332d0 2023-10-30T12:07:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24501332d0 0x7f24501336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient: get_auth_request con 0x7f24501332d0 auth_method 0 2023-10-30T12:07:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 10 monclient(hunting): handle_auth_done global_id 114131 payload 824 2023-10-30T12:07:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 1 -- 172.21.15.50:0/1392869185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f245006b060 con 0x7f24501332d0 2023-10-30T12:07:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f2454e5e640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24501332d0 0x7f24501336e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f244800dfd0 tx=0x7f24480014a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 10 client.104310.objecter ms_handle_connect 0x7f24501332d0 2023-10-30T12:07:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 10 client.104310.objecter resend_mon_ops 2023-10-30T12:07:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448007030 con 0x7f24501332d0 2023-10-30T12:07:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 10 monclient: dump: 2023-10-30T12:07:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.152 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7f2448007190 con 0x7f24501332d0 2023-10-30T12:07:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24501332d0 msgr2=0x7f24501336e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24501332d0 0x7f24501336e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f244800dfd0 tx=0x7f24480014a0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24501332d0 msgr2=0x7f24501336e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f244d7fa640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24501332d0 0x7f24501336e0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f244800dfd0 tx=0x7f24480014a0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.154 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:47.154 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:47.154 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:47.154 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:07:47.154 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 72, 2023-10-30T12:07:47.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:47.155 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:47.155 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:07:47.155 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:07:47.155 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.156 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:07:47.156 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:47.156 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:47.156 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:47.156 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:47.156 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:47.157 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:47.157 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:47.157 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:47.157 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:47.157 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:47.158 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:47.158 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:47.158 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:47.158 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.158 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:47.159 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:47.159 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:47.159 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:47.160 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:47.160 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:47.161 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:47.161 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:47.161 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:47.162 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:47.162 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:47.162 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:47.163 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.163 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:47.163 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:47.164 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:47.164 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:47.165 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:47.165 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:47.165 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:47.188 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:47.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.083+0000 7f24570c2640 10 librados: watch_flush enter 2023-10-30T12:07:47.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 10 librados: watch_flush exit 2023-10-30T12:07:47.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 20 client.104310.objecter shutdown clearing up homeless session... 2023-10-30T12:07:47.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 10 client.104310.objecter successfully canceled tick 2023-10-30T12:07:47.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f243c0392f0 msgr2=0x7f243c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f243c0392f0 0x7f243c03b7b0 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f2440036000 tx=0x7f244003e040 comp rx=0 tx=0).stop 2023-10-30T12:07:47.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 10 monclient: shutdown 2023-10-30T12:07:47.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:47.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24500a2ae0 msgr2=0x7f2450136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 --2- 172.21.15.50:0/1392869185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24500a2ae0 0x7f2450136f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f244400c780 tx=0x7f244400cc40 comp rx=0 tx=0).stop 2023-10-30T12:07:47.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f245565f640 1 -- 172.21.15.50:0/1392869185 reap_dead start 2023-10-30T12:07:47.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f245565f640 1 -- 172.21.15.50:0/1392869185 reap_dead start 2023-10-30T12:07:47.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 shutdown_connections 2023-10-30T12:07:47.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 >> 172.21.15.50:0/1392869185 conn(0x7f24500a80b0 msgr2=0x7f245009e260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:47.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 shutdown_connections 2023-10-30T12:07:47.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 -- 172.21.15.50:0/1392869185 wait complete. 2023-10-30T12:07:47.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.084+0000 7f24570c2640 1 librados: shutdown 2023-10-30T12:07:47.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:07:47.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:07:47.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:07:47.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.298+0000 7fc73d646640 10 monclient: get_monmap_and_config 2023-10-30T12:07:47.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.298+0000 7fc73d646640 10 monclient: build_initial_monmap 2023-10-30T12:07:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.299+0000 7fc73d646640 10 monclient: monmap: 2023-10-30T12:07:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:47.300059+0000 2023-10-30T12:07:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:47.300059+0000 2023-10-30T12:07:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:47.303 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:47.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.299+0000 7fc73d646640 1 Processor -- start 2023-10-30T12:07:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.299+0000 7fc73d646640 1 -- start start 2023-10-30T12:07:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.299+0000 7fc73d646640 10 monclient: init 2023-10-30T12:07:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:47.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-10-30T12:07:47.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc73809fa30 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:47.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc73806b720 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:47.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc73813c080 0x7fc73813e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient(hunting): picked mon.noname-a con 0x7fc73813c080 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:47.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73809e010 con 0x7fc73806b720 2023-10-30T12:07:47.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73809fe10 con 0x7fc73809fa30 2023-10-30T12:07:47.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73806bde0 con 0x7fc73813c080 2023-10-30T12:07:47.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:47.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc73d646640 10 monclient(hunting): authenticate will time out at 1312.483097s 2023-10-30T12:07:47.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc73813c080 0x7fc73813e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7367fc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc73806b720 0x7fc7380a2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39856/0 (socket says 172.21.15.50:39856) 2023-10-30T12:07:47.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7367fc640 1 -- 172.21.15.50:0/3226603902 learned_addr learned my addr 172.21.15.50:0/3226603902 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:47.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 1 --2- 172.21.15.50:0/3226603902 >> v2:172.21.15.50:3300/0 conn(0x7fc73813c080 0x7fc73813e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39026/0 (socket says 172.21.15.50:39026) 2023-10-30T12:07:47.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): get_auth_request con 0x7fc73813c080 auth_method 0 2023-10-30T12:07:47.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient(hunting): handle_auth_done global_id 104313 payload 274 2023-10-30T12:07:47.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 1 -- 172.21.15.50:0/3226603902 >> v1:172.21.15.118:6789/0 conn(0x7fc73809fa30 legacy=0x7fc7380a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 1 -- 172.21.15.50:0/3226603902 >> v1:172.21.15.50:6789/0 conn(0x7fc73806b720 legacy=0x7fc7380a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient: _finish_hunting 0 2023-10-30T12:07:47.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 1 monclient: found mon.noname-a 2023-10-30T12:07:47.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:47.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:47.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc735ffb640 1 -- 172.21.15.50:0/3226603902 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734222164 0 0) 0x7fc73809e010 con 0x7fc73806b720 2023-10-30T12:07:47.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc7377fe640 1 -- 172.21.15.50:0/3226603902 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73809e540 con 0x7fc73813c080 2023-10-30T12:07:47.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.300+0000 7fc735ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:07:47.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc7377fe640 1 --2- 172.21.15.50:0/3226603902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73813c080 0x7fc73813e470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc7300097d0 tx=0x7fc73002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664f7b113ef4f71a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 1 -- 172.21.15.50:0/3226603902 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73002e070 con 0x7fc73813c080 2023-10-30T12:07:47.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:47.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: dump: 2023-10-30T12:07:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.320 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.320 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.320 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.320 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: _finish_auth 0 2023-10-30T12:07:47.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:07:47.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:47.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 1 -- 172.21.15.50:0/3226603902 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7300315a0 con 0x7fc73813c080 2023-10-30T12:07:47.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:47.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 1 -- 172.21.15.50:0/3226603902 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc730031930 con 0x7fc73813c080 2023-10-30T12:07:47.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc73d646640 5 monclient: authenticate success, global_id 104313 2023-10-30T12:07:47.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc73d646640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:47.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc73d646640 4 set_mon_vals no callback set 2023-10-30T12:07:47.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:47.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.301+0000 7fc735ffb640 10 monclient: dump: 2023-10-30T12:07:47.322 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.323 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.323 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.323 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.323 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.323 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.324 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.324 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.324 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.324 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 10 monclient: shutdown 2023-10-30T12:07:47.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 -- 172.21.15.50:0/3226603902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73813c080 msgr2=0x7fc73813e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 --2- 172.21.15.50:0/3226603902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73813c080 0x7fc73813e470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc7300097d0 tx=0x7fc73002b840 comp rx=0 tx=0).stop 2023-10-30T12:07:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 -- 172.21.15.50:0/3226603902 shutdown_connections 2023-10-30T12:07:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 --2- 172.21.15.50:0/3226603902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73813c080 0x7fc73813e470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 -- 172.21.15.50:0/3226603902 >> 172.21.15.50:0/3226603902 conn(0x7fc7380a6200 msgr2=0x7fc7380a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 -- 172.21.15.50:0/3226603902 shutdown_connections 2023-10-30T12:07:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 -- 172.21.15.50:0/3226603902 wait complete. 2023-10-30T12:07:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 10 monclient: build_initial_monmap 2023-10-30T12:07:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 10 monclient: monmap: 2023-10-30T12:07:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:47.328 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 librados: starting msgr at 2023-10-30T12:07:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.302+0000 7fc73d646640 1 librados: starting objecter 2023-10-30T12:07:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 Processor -- start 2023-10-30T12:07:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 -- start start 2023-10-30T12:07:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 librados: setting wanted keys 2023-10-30T12:07:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 librados: calling monclient init 2023-10-30T12:07:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient: init 2023-10-30T12:07:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73809fa30 0x7fc73811bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): picked mon.noname-a con 0x7fc73809fa30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73813c080 0x7fc73811c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): picked mon.noname-b con 0x7fc73813c080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc73811c920 0x7fc7381ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): picked mon.noname-c con 0x7fc73811c920 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73811ed80 con 0x7fc73809fa30 2023-10-30T12:07:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73811f520 con 0x7fc73813c080 2023-10-30T12:07:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc738070e50 con 0x7fc73811c920 2023-10-30T12:07:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc73d646640 10 monclient(hunting): authenticate will time out at 1312.486593s 2023-10-30T12:07:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73809fa30 0x7fc73811bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc73811c920 0x7fc7381ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73809fa30 0x7fc73811bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39032/0 (socket says 172.21.15.50:39032) 2023-10-30T12:07:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 1 -- 172.21.15.50:0/1979161394 learned_addr learned my addr 172.21.15.50:0/1979161394 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 10 monclient(hunting): get_auth_request con 0x7fc73809fa30 auth_method 0 2023-10-30T12:07:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 10 monclient(hunting): get_auth_request con 0x7fc73811c920 auth_method 0 2023-10-30T12:07:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73813c080 0x7fc73811c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc736ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 10 monclient(hunting): get_auth_request con 0x7fc73813c080 auth_method 0 2023-10-30T12:07:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.303+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 10 monclient(hunting): handle_auth_done global_id 104319 payload 824 2023-10-30T12:07:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc73811c920 msgr2=0x7fc7381ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc73811c920 0x7fc7381ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73813c080 msgr2=0x7fc73811c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73813c080 0x7fc73811c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:07:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 1 monclient: found mon.noname-a 2023-10-30T12:07:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:07:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 1 -- 172.21.15.50:0/1979161394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc730009400 con 0x7fc73809fa30 2023-10-30T12:07:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc7377fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc73811c920 0x7fc7381ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:07:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc736ffd640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73809fa30 0x7fc73811bea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc72800c870 tx=0x7fc72800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 client.?.objecter ms_handle_connect 0x7fc73809fa30 2023-10-30T12:07:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728003b70 con 0x7fc73809fa30 2023-10-30T12:07:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:07:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: dump: 2023-10-30T12:07:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.342 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:07:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:07:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc728003cf0 con 0x7fc73809fa30 2023-10-30T12:07:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 5 monclient: authenticate success, global_id 104319 2023-10-30T12:07:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 10 monclient: _renew_subs 2023-10-30T12:07:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72800aa90 con 0x7fc73809fa30 2023-10-30T12:07:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc738138640 con 0x7fc73809fa30 2023-10-30T12:07:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:07:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc71f7fe640 10 monclient: dump: 2023-10-30T12:07:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.346 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc7357fa640 4 set_mon_vals no callback set 2023-10-30T12:07:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 10 client.104319.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 10 monclient: _renew_subs 2023-10-30T12:07:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:07:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73811db10 con 0x7fc73809fa30 2023-10-30T12:07:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.304+0000 7fc73d646640 1 librados: init done 2023-10-30T12:07:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc72800abf0 con 0x7fc73809fa30 2023-10-30T12:07:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc73d646640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc73d646640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73806b720 0x7fc7381ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc73d646640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc738073ff0 con 0x7fc73806b720 2023-10-30T12:07:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc71f7fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7240392f0 0x7fc72403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6275+0+0 (secure 0 0 0) 0x7fc72804aa70 con 0x7fc73809fa30 2023-10-30T12:07:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc71f7fe640 10 client.104319.objecter ms_dispatch 0x7fc738000b90 osd_map(240..240 src has 1..240) v4 2023-10-30T12:07:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc71f7fe640 3 client.104319.objecter handle_osd_map got epochs [240,240] > 0 2023-10-30T12:07:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc71f7fe640 3 client.104319.objecter handle_osd_map decoding full epoch 240 2023-10-30T12:07:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73806b720 0x7fc7381ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient: get_auth_request con 0x7fc73806b720 auth_method 0 2023-10-30T12:07:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc71f7fe640 20 client.104319.objecter dump_active .. 0 homeless 2023-10-30T12:07:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7377fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7240392f0 0x7fc72403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7377fe640 10 monclient: get_auth_request con 0x7fc7240392f0 auth_method 0 2023-10-30T12:07:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 10 monclient(hunting): handle_auth_done global_id 94483 payload 824 2023-10-30T12:07:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.305+0000 7fc7367fc640 1 -- 172.21.15.50:0/1979161394 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73806ab30 con 0x7fc73806b720 2023-10-30T12:07:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 client.104319.objecter ms_handle_connect 0x7fc73806b720 2023-10-30T12:07:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc7367fc640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73806b720 0x7fc7381ca740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc72c000f60 tx=0x7fc72c0018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 client.104319.objecter resend_mon_ops 2023-10-30T12:07:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 client.104319.objecter ms_handle_connect 0x7fc7240392f0 2023-10-30T12:07:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc7377fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7240392f0 0x7fc72403b7b0 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7fc730036000 tx=0x7fc73003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c011580 con 0x7fc73806b720 2023-10-30T12:07:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 monclient: dump: 2023-10-30T12:07:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.355 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc72c0116e0 con 0x7fc73806b720 2023-10-30T12:07:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73806b720 msgr2=0x7fc7381ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73806b720 0x7fc7381ca740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc72c000f60 tx=0x7fc72c0018c0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73806b720 msgr2=0x7fc7381ca740 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.306+0000 7fc71f7fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc73806b720 0x7fc7381ca740 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.312+0000 7fc73d646640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:07:47.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.312+0000 7fc73d646640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7381cac80 0x7fc738136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.312+0000 7fc73d646640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.312+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73809f120 con 0x7fc7381cac80 2023-10-30T12:07:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7381cac80 0x7fc738136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient: get_auth_request con 0x7fc7381cac80 auth_method 0 2023-10-30T12:07:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 10 monclient(hunting): handle_auth_done global_id 94486 payload 824 2023-10-30T12:07:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 1 -- 172.21.15.50:0/1979161394 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc73806a690 con 0x7fc7381cac80 2023-10-30T12:07:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc7367fc640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7381cac80 0x7fc738136300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc72c017000 tx=0x7fc72c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc71f7fe640 10 client.104319.objecter ms_handle_connect 0x7fc7381cac80 2023-10-30T12:07:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc71f7fe640 10 client.104319.objecter resend_mon_ops 2023-10-30T12:07:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c024070 con 0x7fc7381cac80 2023-10-30T12:07:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc71f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc71f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.313+0000 7fc71f7fe640 10 monclient: dump: 2023-10-30T12:07:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.364 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fc72c01f0d0 con 0x7fc7381cac80 2023-10-30T12:07:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc71f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:07:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc71f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7381cac80 msgr2=0x7fc738136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc71f7fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7381cac80 0x7fc738136300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc72c017000 tx=0x7fc72c007f90 comp rx=0 tx=0).stop 2023-10-30T12:07:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc71f7fe640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7381cac80 msgr2=0x7fc738136300 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:07:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc71f7fe640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc7381cac80 0x7fc738136300 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc72c017000 tx=0x7fc72c007f90 comp rx=0 tx=0).stop 2023-10-30T12:07:47.365 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:07:47.366 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:47.366 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:47.366 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:07:47.366 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 72, 2023-10-30T12:07:47.366 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:07:47.367 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:07:47.367 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:07:47.367 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:07:47.367 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.367 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:07:47.368 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:47.368 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:07:47.368 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:07:47.368 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:47.368 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:47.368 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:47.369 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:47.369 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:47.369 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:47.369 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:47.369 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:07:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:07:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:47.372 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:47.372 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:47.372 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:47.372 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:47.372 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.373 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:07:47.373 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:07:47.373 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:07:47.373 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:07:47.373 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:07:47.374 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:07:47.374 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:47.374 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:07:47.374 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:07:47.374 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:07:47.374 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:07:47.375 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:07:47.375 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:07:47.375 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:07:47.375 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:07:47.375 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:07:47.376 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:07:47.376 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:07:47.376 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:07:47.376 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:07:47.376 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:07:47.377 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:07:47.377 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:07:47.377 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:07:47.377 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:07:47.377 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:07:47.378 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:07:47.378 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.378 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:07:47.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.378 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:07:47.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.379 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:07:47.379 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:07:47.379 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:47.379 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:47.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:47.380 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:07:47.380 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.380 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:47.381 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:07:47.381 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.381 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.381 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:07:47.382 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:07:47.382 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:47.382 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:47.382 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:47.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.383 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:07:47.383 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:07:47.383 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:47.383 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:47.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:47.384 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:07:47.384 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:07:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:07:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:07:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:07:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:07:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:07:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:07:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:07:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:07:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:07:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:07:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:07:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:07:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:07:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:07:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:07:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:07:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:07:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:07:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:07:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:07:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:07:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:07:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:07:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:07:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:07:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:07:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:07:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:07:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:07:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:07:47.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:07:47.397 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:07:47.397 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:07:47.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 10 librados: watch_flush enter 2023-10-30T12:07:47.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 10 librados: watch_flush exit 2023-10-30T12:07:47.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 20 client.104319.objecter shutdown clearing up homeless session... 2023-10-30T12:07:47.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 10 client.104319.objecter successfully canceled tick 2023-10-30T12:07:47.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7240392f0 msgr2=0x7fc72403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7240392f0 0x7fc72403b7b0 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7fc730036000 tx=0x7fc73003d040 comp rx=0 tx=0).stop 2023-10-30T12:07:47.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 10 monclient: shutdown 2023-10-30T12:07:47.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:47.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73809fa30 msgr2=0x7fc73811bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 --2- 172.21.15.50:0/1979161394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73809fa30 0x7fc73811bea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc72800c870 tx=0x7fc72800cd30 comp rx=0 tx=0).stop 2023-10-30T12:07:47.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc7377fe640 1 -- 172.21.15.50:0/1979161394 reap_dead start 2023-10-30T12:07:47.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc7377fe640 1 -- 172.21.15.50:0/1979161394 reap_dead start 2023-10-30T12:07:47.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 shutdown_connections 2023-10-30T12:07:47.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 >> 172.21.15.50:0/1979161394 conn(0x7fc7380a6200 msgr2=0x7fc7380a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:47.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 shutdown_connections 2023-10-30T12:07:47.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 -- 172.21.15.50:0/1979161394 wait complete. 2023-10-30T12:07:47.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.314+0000 7fc73d646640 1 librados: shutdown 2023-10-30T12:07:47.402 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:07:47.402 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:07:47.403 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:07:47.403 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:07:47.403 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:07:47.403 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:07:47.404 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:07:47.416 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:07:47.416 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:07:47.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.559+0000 7f9eab89b640 10 monclient: get_monmap_and_config 2023-10-30T12:07:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.559+0000 7f9eab89b640 10 monclient: build_initial_monmap 2023-10-30T12:07:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.559+0000 7f9eab89b640 10 monclient: monmap: 2023-10-30T12:07:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:47.560319+0000 2023-10-30T12:07:47.562 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:47.560319+0000 2023-10-30T12:07:47.562 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:47.562 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.562 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:47.562 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:47.563 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:47.563 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:47.563 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:47.563 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.559+0000 7f9eab89b640 1 Processor -- start 2023-10-30T12:07:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 1 -- start start 2023-10-30T12:07:47.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient: init 2023-10-30T12:07:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-10-30T12:07:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ea40707d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9ea40712c0 0x7f9ea413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient(hunting): picked mon.noname-c con 0x7f9ea40712c0 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:47.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ea40a0170 0x7f9ea413d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient(hunting): picked mon.noname-a con 0x7f9ea40a0170 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:47.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea40a11f0 con 0x7f9ea40707d0 2023-10-30T12:07:47.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea409f4b0 con 0x7f9ea40a0170 2023-10-30T12:07:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea409d9d0 con 0x7f9ea40712c0 2023-10-30T12:07:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9eab89b640 10 monclient(hunting): authenticate will time out at 1312.743609s 2023-10-30T12:07:47.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea9e38640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f9ea40a0170 msgr2=0x7f9ea413d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:07:47.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea9e38640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ea40a0170 0x7f9ea413d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:47.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea9637640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f9ea40707d0 0x7f9ea4070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:51666/0 (socket says 172.21.15.50:51666) 2023-10-30T12:07:47.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea8e36640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9ea40712c0 0x7f9ea413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea9637640 1 -- 172.21.15.50:0/1521314686 learned_addr learned my addr 172.21.15.50:0/1521314686 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:07:47.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea8e36640 10 monclient(hunting): get_auth_request con 0x7f9ea40712c0 auth_method 0 2023-10-30T12:07:47.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea8e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea8e36640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.560+0000 7f9ea8e36640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9e9bfff640 1 -- 172.21.15.50:0/1521314686 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374455153 0 0) 0x7f9ea40a11f0 con 0x7f9ea40707d0 2023-10-30T12:07:47.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9e9bfff640 10 monclient(hunting): _negotiate 2023-10-30T12:07:47.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9e9bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9e9bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9e9bfff640 10 monclient(hunting): my global_id is 94492 2023-10-30T12:07:47.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9e9bfff640 1 -- 172.21.15.50:0/1521314686 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e90003520 con 0x7f9ea40707d0 2023-10-30T12:07:47.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 10 monclient(hunting): handle_auth_done global_id 94489 payload 274 2023-10-30T12:07:47.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 1 -- 172.21.15.50:0/1521314686 >> v2:172.21.15.50:3300/0 conn(0x7f9ea40a0170 msgr2=0x7f9ea413d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:47.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 1 --2- 172.21.15.50:0/1521314686 >> v2:172.21.15.50:3300/0 conn(0x7f9ea40a0170 0x7f9ea413d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 1 -- 172.21.15.50:0/1521314686 >> v1:172.21.15.118:6789/0 conn(0x7f9ea40707d0 legacy=0x7f9ea4070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 10 monclient: _finish_hunting 0 2023-10-30T12:07:47.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 1 monclient: found mon.noname-c 2023-10-30T12:07:47.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:47.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:07:47.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.561+0000 7f9ea8e36640 1 -- 172.21.15.50:0/1521314686 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea40a13f0 con 0x7f9ea40712c0 2023-10-30T12:07:47.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9ea8e36640 1 --2- 172.21.15.50:0/1521314686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 0x7f9ea413aba0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4066030 tx=0x7f9e9402b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adbc748aa81f4eef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 1 -- 172.21.15.50:0/1521314686 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94031740 con 0x7f9ea40712c0 2023-10-30T12:07:47.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:47.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: dump: 2023-10-30T12:07:47.580 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.580 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.581 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.581 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.581 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.581 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.581 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.581 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.582 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.582 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: _finish_auth 0 2023-10-30T12:07:47.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:47.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:47.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 5 monclient: authenticate success, global_id 94489 2023-10-30T12:07:47.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:47.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 1 -- 172.21.15.50:0/1521314686 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e940318a0 con 0x7f9ea40712c0 2023-10-30T12:07:47.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:47.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:47.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 4 set_mon_vals no callback set 2023-10-30T12:07:47.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 1 -- 172.21.15.50:0/1521314686 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94031c30 con 0x7f9ea40712c0 2023-10-30T12:07:47.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:47.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9e9bfff640 10 monclient: dump: 2023-10-30T12:07:47.584 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.585 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.585 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.585 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.585 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.586 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.586 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.586 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.586 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 10 monclient: shutdown 2023-10-30T12:07:47.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:47.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 -- 172.21.15.50:0/1521314686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 msgr2=0x7f9ea413aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 --2- 172.21.15.50:0/1521314686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 0x7f9ea413aba0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4066030 tx=0x7f9e9402b7e0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 -- 172.21.15.50:0/1521314686 shutdown_connections 2023-10-30T12:07:47.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 --2- 172.21.15.50:0/1521314686 >> v2:172.21.15.50:3300/0 conn(0x7f9ea40a0170 0x7f9ea413d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 --2- 172.21.15.50:0/1521314686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 0x7f9ea413aba0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 -- 172.21.15.50:0/1521314686 >> 172.21.15.50:0/1521314686 conn(0x7f9ea40a47f0 msgr2=0x7f9ea40a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:47.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 -- 172.21.15.50:0/1521314686 shutdown_connections 2023-10-30T12:07:47.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.562+0000 7f9eab89b640 1 -- 172.21.15.50:0/1521314686 wait complete. 2023-10-30T12:07:47.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 10 monclient: build_initial_monmap 2023-10-30T12:07:47.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 10 monclient: monmap: 2023-10-30T12:07:47.588 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:47.589 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:47.589 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:47.589 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:47.589 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:47.589 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.589 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:47.590 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:47.590 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:47.590 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 1 librados: starting msgr at 2023-10-30T12:07:47.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 1 librados: starting objecter 2023-10-30T12:07:47.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 1 Processor -- start 2023-10-30T12:07:47.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 1 -- start start 2023-10-30T12:07:47.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 1 librados: setting wanted keys 2023-10-30T12:07:47.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 1 librados: calling monclient init 2023-10-30T12:07:47.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.563+0000 7f9eab89b640 10 monclient: init 2023-10-30T12:07:47.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:47.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:07:47.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ea40707d0 0x7f9ea41c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): picked mon.noname-a con 0x7f9ea40707d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:47.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 0x7f9ea41bb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): picked mon.noname-b con 0x7f9ea40712c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:47.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ea40a0170 0x7f9ea41bbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): picked mon.noname-c con 0x7f9ea40a0170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:47.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea41a7500 con 0x7f9ea40707d0 2023-10-30T12:07:47.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea41a8ed0 con 0x7f9ea40712c0 2023-10-30T12:07:47.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:47.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea41cac70 con 0x7f9ea40a0170 2023-10-30T12:07:47.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:47.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9eab89b640 10 monclient(hunting): authenticate will time out at 1312.747222s 2023-10-30T12:07:47.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9637640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ea40707d0 msgr2=0x7f9ea41c9fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:07:47.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9637640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ea40707d0 0x7f9ea41c9fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:47.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ea40a0170 0x7f9ea41bbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ea40a0170 0x7f9ea41bbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44598/0 (socket says 172.21.15.50:44598) 2023-10-30T12:07:47.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 -- 172.21.15.50:0/4217346846 learned_addr learned my addr 172.21.15.50:0/4217346846 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:47.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): get_auth_request con 0x7f9ea40a0170 auth_method 0 2023-10-30T12:07:47.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 0x7f9ea41bb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 10 monclient(hunting): get_auth_request con 0x7f9ea40712c0 auth_method 0 2023-10-30T12:07:47.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:47.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:47.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:47.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:47.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:47.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea8e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:47.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient(hunting): handle_auth_done global_id 114137 payload 824 2023-10-30T12:07:47.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 -- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 msgr2=0x7f9ea41bb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 0x7f9ea41bb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 -- 172.21.15.50:0/4217346846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ea40707d0 msgr2=0x7f9ea41c9fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:47.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ea40707d0 0x7f9ea41c9fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient: _finish_hunting 0 2023-10-30T12:07:47.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 monclient: found mon.noname-c 2023-10-30T12:07:47.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:47.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:07:47.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 -- 172.21.15.50:0/4217346846 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e94009400 con 0x7f9ea40a0170 2023-10-30T12:07:47.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9ea9e38640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ea40a0170 0x7f9ea41bbbf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00be50 tx=0x7f9e9c00be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9e99ffb640 10 client.?.objecter ms_handle_connect 0x7f9ea40a0170 2023-10-30T12:07:47.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.564+0000 7f9e99ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:47.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 1 -- 172.21.15.50:0/4217346846 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c016430 con 0x7f9ea40a0170 2023-10-30T12:07:47.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:47.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: dump: 2023-10-30T12:07:47.601 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.601 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.601 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.602 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.602 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.602 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.602 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.602 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.603 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.603 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: _finish_auth 0 2023-10-30T12:07:47.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:07:47.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:47.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 1 -- 172.21.15.50:0/4217346846 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9c016590 con 0x7f9ea40a0170 2023-10-30T12:07:47.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:47.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 1 -- 172.21.15.50:0/4217346846 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c016940 con 0x7f9ea40a0170 2023-10-30T12:07:47.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 5 monclient: authenticate success, global_id 114137 2023-10-30T12:07:47.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 10 monclient: _renew_subs 2023-10-30T12:07:47.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:47.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e9b7fe640 4 set_mon_vals no callback set 2023-10-30T12:07:47.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea41bd590 con 0x7f9ea40a0170 2023-10-30T12:07:47.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:47.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:47.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 10 monclient: dump: 2023-10-30T12:07:47.605 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:47.606 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:47.606 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.606 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:47.606 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:47.606 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:47.607 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:47.607 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:47.607 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:47.607 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:47.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 10 client.114137.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:47.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 10 monclient: _renew_subs 2023-10-30T12:07:47.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:47.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea41a8880 con 0x7f9ea40a0170 2023-10-30T12:07:47.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9eab89b640 1 librados: init done 2023-10-30T12:07:47.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.565+0000 7f9e99ffb640 1 -- 172.21.15.50:0/4217346846 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9e9c012020 con 0x7f9ea40a0170 2023-10-30T12:07:47.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9e99ffb640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e880392a0 0x7f9e8803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:47.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9eab89b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:47.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9eab89b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:47.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9eab89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:47.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9e99ffb640 1 -- 172.21.15.50:0/4217346846 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6275+0+0 (secure 0 0 0) 0x7f9e9c04aeb0 con 0x7f9ea40a0170 2023-10-30T12:07:47.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9e99ffb640 10 client.114137.objecter ms_dispatch 0x7f9ea4000b90 osd_map(240..240 src has 1..240) v4 2023-10-30T12:07:47.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9e99ffb640 3 client.114137.objecter handle_osd_map got epochs [240,240] > 0 2023-10-30T12:07:47.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9e99ffb640 3 client.114137.objecter handle_osd_map decoding full epoch 240 2023-10-30T12:07:47.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea404e160 con 0x7f9ea40a0170 2023-10-30T12:07:47.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9e99ffb640 20 client.114137.objecter dump_active .. 0 homeless 2023-10-30T12:07:47.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9ea9637640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e880392a0 0x7f9e8803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:47.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9ea9637640 10 monclient: get_auth_request con 0x7f9e880392a0 auth_method 0 2023-10-30T12:07:47.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9ea9637640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e880392a0 0x7f9e8803b760 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000c00 tx=0x7f9ea0007970 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:47.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.566+0000 7f9e99ffb640 10 client.114137.objecter ms_handle_connect 0x7f9e880392a0 2023-10-30T12:07:47.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.571+0000 7f9e99ffb640 1 -- 172.21.15.50:0/4217346846 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9e9c0072e0 con 0x7f9ea40a0170 2023-10-30T12:07:47.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.571+0000 7f9e99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:47.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.571+0000 7f9e99ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:47.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.813+0000 7f9eab89b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:47.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.813+0000 7f9eab89b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:47.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.813+0000 7f9eab89b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:47.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.813+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9ea406a1b0 con 0x7f9ea40a0170 2023-10-30T12:07:47.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.813+0000 7f9e99ffb640 1 -- 172.21.15.50:0/4217346846 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9e9c013ad0 con 0x7f9ea40a0170 2023-10-30T12:07:47.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.813+0000 7f9e99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:47.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.813+0000 7f9e99ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:47.816 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:47.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 10 librados: watch_flush enter 2023-10-30T12:07:47.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 10 librados: watch_flush exit 2023-10-30T12:07:47.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 20 client.114137.objecter shutdown clearing up homeless session... 2023-10-30T12:07:47.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 10 client.114137.objecter successfully canceled tick 2023-10-30T12:07:47.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e880392a0 msgr2=0x7f9e8803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e880392a0 0x7f9e8803b760 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000c00 tx=0x7f9ea0007970 comp rx=0 tx=0).stop 2023-10-30T12:07:47.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 10 monclient: shutdown 2023-10-30T12:07:47.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ea40a0170 msgr2=0x7f9ea41bbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ea40a0170 0x7f9ea41bbbf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00be50 tx=0x7f9e9c00be80 comp rx=0 tx=0).stop 2023-10-30T12:07:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 shutdown_connections 2023-10-30T12:07:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e880392a0 0x7f9e8803b760 unknown :-1 s=CLOSED pgs=4029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ea40a0170 0x7f9ea41bbbf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea40712c0 0x7f9ea41bb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 --2- 172.21.15.50:0/4217346846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ea40707d0 0x7f9ea41c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 >> 172.21.15.50:0/4217346846 conn(0x7f9ea40a47f0 msgr2=0x7f9ea409bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 shutdown_connections 2023-10-30T12:07:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 -- 172.21.15.50:0/4217346846 wait complete. 2023-10-30T12:07:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:47.815+0000 7f9eab89b640 1 librados: shutdown 2023-10-30T12:07:50.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.943+0000 7faf0de10640 10 monclient: get_monmap_and_config 2023-10-30T12:07:50.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.943+0000 7faf0de10640 10 monclient: build_initial_monmap 2023-10-30T12:07:50.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.943+0000 7faf0de10640 10 monclient: monmap: 2023-10-30T12:07:50.945 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:50.945 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:50.946 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:50.944451+0000 2023-10-30T12:07:50.946 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:50.944451+0000 2023-10-30T12:07:50.946 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:50.946 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:50.946 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:50.946 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:50.947 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:50.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 1 Processor -- start 2023-10-30T12:07:50.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 1 -- start start 2023-10-30T12:07:50.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient: init 2023-10-30T12:07:50.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:50.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:07:50.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7faf080a0080 0x7faf080a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:50.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient(hunting): picked mon.noname-c con 0x7faf080a0080 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:50.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf08073620 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:50.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a09a0 0x7faf08133f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:50.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient(hunting): picked mon.noname-b con 0x7faf080a09a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:50.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0809e580 con 0x7faf08073620 2023-10-30T12:07:50.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:50.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf08073af0 con 0x7faf080a0080 2023-10-30T12:07:50.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:50.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0809e780 con 0x7faf080a09a0 2023-10-30T12:07:50.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:50.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf0de10640 10 monclient(hunting): authenticate will time out at 1316.127842s 2023-10-30T12:07:50.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.944+0000 7faf07fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a09a0 0x7faf08133f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:50.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a09a0 0x7faf08133f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47036/0 (socket says 172.21.15.50:47036) 2023-10-30T12:07:50.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 1 -- 172.21.15.50:0/620119182 learned_addr learned my addr 172.21.15.50:0/620119182 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:50.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): get_auth_request con 0x7faf080a09a0 auth_method 0 2023-10-30T12:07:50.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:50.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:50.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:50.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:50.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:50.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:50.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient(hunting): handle_auth_done global_id 114140 payload 274 2023-10-30T12:07:50.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 1 -- 172.21.15.50:0/620119182 >> v2:172.21.15.118:3300/0 conn(0x7faf080a0080 msgr2=0x7faf080a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:50.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf077fe640 1 --2- 172.21.15.50:0/620119182 >> v2:172.21.15.118:3300/0 conn(0x7faf080a0080 0x7faf080a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:50.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 1 --2- 172.21.15.50:0/620119182 >> v2:172.21.15.118:3300/0 conn(0x7faf080a0080 0x7faf080a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:50.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 1 -- 172.21.15.50:0/620119182 >> v1:172.21.15.118:6789/0 conn(0x7faf08073620 legacy=0x7faf08133850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:07:50.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient: _finish_hunting 0 2023-10-30T12:07:50.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 1 monclient: found mon.noname-b 2023-10-30T12:07:50.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:50.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:07:50.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf07fff640 1 -- 172.21.15.50:0/620119182 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08074360 con 0x7faf080a09a0 2023-10-30T12:07:50.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf077fe640 10 monclient: get_auth_request con 0x7faf080a0080 auth_method 0 2023-10-30T12:07:50.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.945+0000 7faf077fe640 1 --2- 172.21.15.50:0/620119182 >> v2:172.21.15.118:3300/0 conn(0x7faf080a0080 0x7faf080a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:07:50.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf07fff640 1 --2- 172.21.15.50:0/620119182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a09a0 0x7faf08133f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faf00009e50 tx=0x7faf0002be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34585ecfda10911e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:50.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 1 -- 172.21.15.50:0/620119182 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0002c9f0 con 0x7faf080a09a0 2023-10-30T12:07:50.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:50.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:07:50.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: dump: 2023-10-30T12:07:50.963 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:50.963 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:50.964 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.964 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.964 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:50.964 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:50.964 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:50.965 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:50.965 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:50.965 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:50.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: _finish_auth 0 2023-10-30T12:07:50.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: _check_auth_tickets 2023-10-30T12:07:50.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:50.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 1 -- 172.21.15.50:0/620119182 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0002cb50 con 0x7faf080a09a0 2023-10-30T12:07:50.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 5 monclient: authenticate success, global_id 114140 2023-10-30T12:07:50.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:07:50.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:50.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:50.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 1 -- 172.21.15.50:0/620119182 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf000343f0 con 0x7faf080a09a0 2023-10-30T12:07:50.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:50.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 4 set_mon_vals no callback set 2023-10-30T12:07:50.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:50.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf067fc640 10 monclient: dump: 2023-10-30T12:07:50.968 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:50.968 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:50.968 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.968 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.968 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:50.969 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:50.969 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:50.969 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:50.969 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:50.969 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:50.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 10 monclient: shutdown 2023-10-30T12:07:50.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:50.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 -- 172.21.15.50:0/620119182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a09a0 msgr2=0x7faf08133f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:50.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 --2- 172.21.15.50:0/620119182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a09a0 0x7faf08133f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faf00009e50 tx=0x7faf0002be50 comp rx=0 tx=0).stop 2023-10-30T12:07:50.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 -- 172.21.15.50:0/620119182 shutdown_connections 2023-10-30T12:07:50.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 --2- 172.21.15.50:0/620119182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a09a0 0x7faf08133f60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:50.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 --2- 172.21.15.50:0/620119182 >> v2:172.21.15.118:3300/0 conn(0x7faf080a0080 0x7faf080a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:50.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 -- 172.21.15.50:0/620119182 >> 172.21.15.50:0/620119182 conn(0x7faf080a4870 msgr2=0x7faf080a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:50.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 -- 172.21.15.50:0/620119182 shutdown_connections 2023-10-30T12:07:50.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.946+0000 7faf0de10640 1 -- 172.21.15.50:0/620119182 wait complete. 2023-10-30T12:07:50.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.947+0000 7faf0de10640 10 monclient: build_initial_monmap 2023-10-30T12:07:50.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.947+0000 7faf0de10640 10 monclient: monmap: 2023-10-30T12:07:50.972 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:50.972 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:50.972 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:50.972 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:50.973 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:50.973 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:50.973 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:50.973 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:50.973 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:50.973 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:50.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.947+0000 7faf0de10640 1 librados: starting msgr at 2023-10-30T12:07:50.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.947+0000 7faf0de10640 1 librados: starting objecter 2023-10-30T12:07:50.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.947+0000 7faf0de10640 1 Processor -- start 2023-10-30T12:07:50.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 -- start start 2023-10-30T12:07:50.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 librados: setting wanted keys 2023-10-30T12:07:50.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 librados: calling monclient init 2023-10-30T12:07:50.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient: init 2023-10-30T12:07:50.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:50.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:07:50.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf08073620 0x7faf081c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:50.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): picked mon.noname-b con 0x7faf08073620 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:50.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a0080 0x7faf08095220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:50.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): picked mon.noname-c con 0x7faf080a0080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:50.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf080a09a0 0x7faf08095760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:50.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): picked mon.noname-a con 0x7faf080a09a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:50.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:50.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08090420 con 0x7faf080a09a0 2023-10-30T12:07:50.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:50.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08073d20 con 0x7faf08073620 2023-10-30T12:07:50.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:50.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0808f530 con 0x7faf080a0080 2023-10-30T12:07:50.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:50.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf0de10640 10 monclient(hunting): authenticate will time out at 1316.131519s 2023-10-30T12:07:50.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf07fff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf080a09a0 msgr2=0x7faf08095760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:07:50.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf07fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf080a09a0 0x7faf08095760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:50.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a0080 0x7faf08095220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:50.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a0080 0x7faf08095220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47038/0 (socket says 172.21.15.50:47038) 2023-10-30T12:07:50.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 -- 172.21.15.50:0/1710681152 learned_addr learned my addr 172.21.15.50:0/1710681152 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:50.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): get_auth_request con 0x7faf080a0080 auth_method 0 2023-10-30T12:07:50.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:50.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:50.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:50.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:50.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:50.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:50.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf077fe640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf08073620 0x7faf081c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:50.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf077fe640 10 monclient(hunting): get_auth_request con 0x7faf08073620 auth_method 0 2023-10-30T12:07:50.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:50.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf077fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:50.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf077fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:50.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient(hunting): handle_auth_done global_id 114143 payload 824 2023-10-30T12:07:50.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 -- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf08073620 msgr2=0x7faf081c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:50.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf08073620 0x7faf081c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:50.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 -- 172.21.15.50:0/1710681152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf080a09a0 msgr2=0x7faf08095760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:50.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf080a09a0 0x7faf08095760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:50.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:07:50.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 monclient: found mon.noname-c 2023-10-30T12:07:50.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:50.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:07:50.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.948+0000 7faf06ffd640 1 -- 172.21.15.50:0/1710681152 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf00009b30 con 0x7faf080a0080 2023-10-30T12:07:50.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf06ffd640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a0080 0x7faf08095220 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faef800c400 tx=0x7faef800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:50.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 client.?.objecter ms_handle_connect 0x7faf080a0080 2023-10-30T12:07:50.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:50.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 1 -- 172.21.15.50:0/1710681152 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8011790 con 0x7faf080a0080 2023-10-30T12:07:50.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:50.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:50.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: dump: 2023-10-30T12:07:50.984 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:50.984 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:50.984 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.985 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.985 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:50.985 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:50.985 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:50.985 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:50.985 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:50.986 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:50.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: _finish_auth 0 2023-10-30T12:07:50.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: _check_auth_tickets 2023-10-30T12:07:50.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:50.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 1 -- 172.21.15.50:0/1710681152 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef8011930 con 0x7faf080a0080 2023-10-30T12:07:50.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 5 monclient: authenticate success, global_id 114143 2023-10-30T12:07:50.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 10 monclient: _renew_subs 2023-10-30T12:07:50.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:50.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0808f9f0 con 0x7faf080a0080 2023-10-30T12:07:50.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:50.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 1 -- 172.21.15.50:0/1710681152 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8011ce0 con 0x7faf080a0080 2023-10-30T12:07:50.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:50.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:50.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faeeffff640 10 monclient: dump: 2023-10-30T12:07:50.988 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:50.988 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:50.989 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:50.989 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:50.989 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:50.989 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:50.989 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:50.990 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:50.990 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:50.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf05ffb640 4 set_mon_vals no callback set 2023-10-30T12:07:50.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 10 client.114143.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:50.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 10 monclient: _renew_subs 2023-10-30T12:07:50.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:50.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0808e8f0 con 0x7faf080a0080 2023-10-30T12:07:50.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.949+0000 7faf0de10640 1 librados: init done 2023-10-30T12:07:50.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faeeffff640 1 -- 172.21.15.50:0/1710681152 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7faef801c460 con 0x7faf080a0080 2023-10-30T12:07:50.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faf0de10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:50.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faf0de10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:50.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faf0de10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:50.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faeeffff640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faef40392f0 0x7faef403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:50.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0804e160 con 0x7faf080a0080 2023-10-30T12:07:50.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faeeffff640 1 -- 172.21.15.50:0/1710681152 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6275+0+0 (secure 0 0 0) 0x7faef804ab40 con 0x7faf080a0080 2023-10-30T12:07:50.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faeeffff640 10 client.114143.objecter ms_dispatch 0x7faf08000b90 osd_map(240..240 src has 1..240) v4 2023-10-30T12:07:50.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faeeffff640 3 client.114143.objecter handle_osd_map got epochs [240,240] > 0 2023-10-30T12:07:50.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faeeffff640 3 client.114143.objecter handle_osd_map decoding full epoch 240 2023-10-30T12:07:50.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faeeffff640 20 client.114143.objecter dump_active .. 0 homeless 2023-10-30T12:07:50.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faf077fe640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faef40392f0 0x7faef403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:50.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.950+0000 7faf077fe640 10 monclient: get_auth_request con 0x7faef40392f0 auth_method 0 2023-10-30T12:07:50.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.951+0000 7faeeffff640 10 client.114143.objecter ms_handle_connect 0x7faef40392f0 2023-10-30T12:07:50.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.951+0000 7faf077fe640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faef40392f0 0x7faef403b7b0 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7faefc006fd0 tx=0x7faefc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:50.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.955+0000 7faeeffff640 1 -- 172.21.15.50:0/1710681152 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faef8011e40 con 0x7faf080a0080 2023-10-30T12:07:50.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.955+0000 7faeeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:50.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:50.955+0000 7faeeffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:51.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.199+0000 7faf0de10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:51.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.199+0000 7faf0de10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:51.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.199+0000 7faf0de10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:51.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.199+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faf0806ab70 con 0x7faf080a0080 2023-10-30T12:07:51.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.200+0000 7faeeffff640 1 -- 172.21.15.50:0/1710681152 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7faef80102e0 con 0x7faf080a0080 2023-10-30T12:07:51.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.200+0000 7faeeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:51.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.200+0000 7faeeffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:51.203 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:51.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 10 librados: watch_flush enter 2023-10-30T12:07:51.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 10 librados: watch_flush exit 2023-10-30T12:07:51.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 20 client.114143.objecter shutdown clearing up homeless session... 2023-10-30T12:07:51.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 10 client.114143.objecter successfully canceled tick 2023-10-30T12:07:51.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faef40392f0 msgr2=0x7faef403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:51.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faef40392f0 0x7faef403b7b0 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7faefc006fd0 tx=0x7faefc008040 comp rx=0 tx=0).stop 2023-10-30T12:07:51.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 10 monclient: shutdown 2023-10-30T12:07:51.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:51.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a0080 msgr2=0x7faf08095220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:51.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a0080 0x7faf08095220 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faef800c400 tx=0x7faef800c8c0 comp rx=0 tx=0).stop 2023-10-30T12:07:51.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 shutdown_connections 2023-10-30T12:07:51.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faef40392f0 0x7faef403b7b0 unknown :-1 s=CLOSED pgs=4030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:51.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf080a09a0 0x7faf08095760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:51.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf080a0080 0x7faf08095220 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:51.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 --2- 172.21.15.50:0/1710681152 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf08073620 0x7faf081c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:51.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 >> 172.21.15.50:0/1710681152 conn(0x7faf080a4870 msgr2=0x7faf08097530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:51.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 shutdown_connections 2023-10-30T12:07:51.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 -- 172.21.15.50:0/1710681152 wait complete. 2023-10-30T12:07:51.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:51.202+0000 7faf0de10640 1 librados: shutdown 2023-10-30T12:07:54.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:54.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.332+0000 7f6b0ee56640 10 monclient: get_monmap_and_config 2023-10-30T12:07:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.332+0000 7f6b0ee56640 10 monclient: build_initial_monmap 2023-10-30T12:07:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.332+0000 7f6b0ee56640 10 monclient: monmap: 2023-10-30T12:07:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:54.334 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:54.333230+0000 2023-10-30T12:07:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:54.333230+0000 2023-10-30T12:07:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:54.335 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:54.336 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:54.336 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.332+0000 7f6b0ee56640 1 Processor -- start 2023-10-30T12:07:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 1 -- start start 2023-10-30T12:07:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient: init 2023-10-30T12:07:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-10-30T12:07:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6b080a4b60 0x7f6b081326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient(hunting): picked mon.noname-a con 0x7f6b080a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:07:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6b08099a40 0x7f6b08132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient(hunting): picked mon.noname-c con 0x7f6b08099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b080a2350 addr v1:172.21.15.50:6789/0 2023-10-30T12:07:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b08133120 con 0x7f6b080a2350 2023-10-30T12:07:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b080a3d70 con 0x7f6b080a4b60 2023-10-30T12:07:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b080a4f40 con 0x7f6b08099a40 2023-10-30T12:07:54.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0ee56640 10 monclient(hunting): authenticate will time out at 1319.516533s 2023-10-30T12:07:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0cbf2640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f6b080a4b60 msgr2=0x7f6b081326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:07:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0d3f3640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f6b080a2350 legacy=0x7f6b080a27e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:07:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b0cbf2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6b080a4b60 0x7f6b081326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b07fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6b08099a40 0x7f6b08132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b07fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6b08099a40 0x7f6b08132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:39738/0 (socket says 172.21.15.50:39738) 2023-10-30T12:07:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b07fff640 1 -- 172.21.15.50:0/1813713832 learned_addr learned my addr 172.21.15.50:0/1813713832 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b07fff640 10 monclient(hunting): get_auth_request con 0x7f6b08099a40 auth_method 0 2023-10-30T12:07:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b07fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.333+0000 7f6b07fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 10 monclient(hunting): handle_auth_done global_id 94501 payload 274 2023-10-30T12:07:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 1 -- 172.21.15.50:0/1813713832 >> v2:172.21.15.50:3300/0 conn(0x7f6b080a4b60 msgr2=0x7f6b081326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 1 --2- 172.21.15.50:0/1813713832 >> v2:172.21.15.50:3300/0 conn(0x7f6b080a4b60 0x7f6b081326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 1 -- 172.21.15.50:0/1813713832 >> v1:172.21.15.50:6789/0 conn(0x7f6b080a2350 legacy=0x7f6b080a27e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:07:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 10 monclient: _finish_hunting 0 2023-10-30T12:07:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 1 monclient: found mon.noname-c 2023-10-30T12:07:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:07:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.334+0000 7f6b07fff640 1 -- 172.21.15.50:0/1813713832 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080a4150 con 0x7f6b08099a40 2023-10-30T12:07:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b07fff640 1 --2- 172.21.15.50:0/1813713832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b08099a40 0x7f6b08132be0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6b08099ed0 tx=0x7f6af402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44b5ad595439f3cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 1 -- 172.21.15.50:0/1813713832 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4031770 con 0x7f6b08099a40 2023-10-30T12:07:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: dump: 2023-10-30T12:07:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:54.354 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: _finish_auth 0 2023-10-30T12:07:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: _check_auth_tickets 2023-10-30T12:07:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 1 -- 172.21.15.50:0/1813713832 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af40318d0 con 0x7f6b08099a40 2023-10-30T12:07:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 1 -- 172.21.15.50:0/1813713832 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4031c60 con 0x7f6b08099a40 2023-10-30T12:07:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:07:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b077fe640 10 monclient: dump: 2023-10-30T12:07:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:54.357 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:54.358 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 5 monclient: authenticate success, global_id 94501 2023-10-30T12:07:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 10 monclient: get_monmap_and_config success 2023-10-30T12:07:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 4 set_mon_vals no callback set 2023-10-30T12:07:54.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 10 monclient: shutdown 2023-10-30T12:07:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1813713832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b08099a40 msgr2=0x7f6b08132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1813713832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b08099a40 0x7f6b08132be0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6b08099ed0 tx=0x7f6af402b810 comp rx=0 tx=0).stop 2023-10-30T12:07:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1813713832 shutdown_connections 2023-10-30T12:07:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1813713832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b08099a40 0x7f6b08132be0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1813713832 >> v2:172.21.15.50:3300/0 conn(0x7f6b080a4b60 0x7f6b081326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1813713832 >> 172.21.15.50:0/1813713832 conn(0x7f6b080a8060 msgr2=0x7f6b080a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1813713832 shutdown_connections 2023-10-30T12:07:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.335+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1813713832 wait complete. 2023-10-30T12:07:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 10 monclient: build_initial_monmap 2023-10-30T12:07:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 10 monclient: monmap: 2023-10-30T12:07:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:07:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:07:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:07:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:07:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:07:54.362 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:07:54.362 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 1 librados: starting msgr at 2023-10-30T12:07:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 1 librados: starting objecter 2023-10-30T12:07:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 1 Processor -- start 2023-10-30T12:07:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 1 -- start start 2023-10-30T12:07:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 1 librados: setting wanted keys 2023-10-30T12:07:54.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 1 librados: calling monclient init 2023-10-30T12:07:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.336+0000 7f6b0ee56640 10 monclient: init 2023-10-30T12:07:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:07:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b080a2350 0x7f6b081ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:54.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): picked mon.noname-c con 0x7f6b080a2350 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b080a4b60 0x7f6b081cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): picked mon.noname-a con 0x7f6b080a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:07:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b0811e540 0x7f6b0811e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): picked mon.noname-b con 0x7f6b0811e540 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:07:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:54.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08121d10 con 0x7f6b080a4b60 2023-10-30T12:07:54.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:54.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08120e20 con 0x7f6b0811e540 2023-10-30T12:07:54.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:54.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08135ce0 con 0x7f6b080a2350 2023-10-30T12:07:54.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0ee56640 10 monclient(hunting): authenticate will time out at 1319.520458s 2023-10-30T12:07:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b07fff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b080a4b60 msgr2=0x7f6b081cae00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:07:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b07fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b080a4b60 0x7f6b081cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b080a2350 0x7f6b081ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b080a2350 0x7f6b081ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47044/0 (socket says 172.21.15.50:47044) 2023-10-30T12:07:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 -- 172.21.15.50:0/1807956069 learned_addr learned my addr 172.21.15.50:0/1807956069 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): get_auth_request con 0x7f6b080a2350 auth_method 0 2023-10-30T12:07:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:54.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b0811e540 0x7f6b0811e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 10 monclient(hunting): get_auth_request con 0x7f6b0811e540 auth_method 0 2023-10-30T12:07:54.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:07:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:07:54.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0d3f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:07:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient(hunting): handle_auth_done global_id 114146 payload 824 2023-10-30T12:07:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 -- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b0811e540 msgr2=0x7f6b0811e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b0811e540 0x7f6b0811e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 -- 172.21.15.50:0/1807956069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b080a4b60 msgr2=0x7f6b081cae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:07:54.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b080a4b60 0x7f6b081cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient: _finish_hunting 0 2023-10-30T12:07:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 monclient: found mon.noname-c 2023-10-30T12:07:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:07:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:07:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.337+0000 7f6b0cbf2640 1 -- 172.21.15.50:0/1807956069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af40094d0 con 0x7f6b080a2350 2023-10-30T12:07:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0cbf2640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b080a2350 0x7f6b081ca8c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00b550 tx=0x7f6afc00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 client.?.objecter ms_handle_connect 0x7f6b080a2350 2023-10-30T12:07:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:07:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 1 -- 172.21.15.50:0/1807956069 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc003dc0 con 0x7f6b080a2350 2023-10-30T12:07:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:54.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:07:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: dump: 2023-10-30T12:07:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.374 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:54.375 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: _finish_auth 0 2023-10-30T12:07:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: _check_auth_tickets 2023-10-30T12:07:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:07:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 5 monclient: authenticate success, global_id 114146 2023-10-30T12:07:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 10 monclient: _renew_subs 2023-10-30T12:07:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b080a18d0 con 0x7f6b080a2350 2023-10-30T12:07:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 1 -- 172.21.15.50:0/1807956069 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc005070 con 0x7f6b080a2350 2023-10-30T12:07:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:07:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 1 -- 172.21.15.50:0/1807956069 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc005420 con 0x7f6b080a2350 2023-10-30T12:07:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:07:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:07:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b057fa640 10 monclient: dump: 2023-10-30T12:07:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:07:54.378 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:07:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:07:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:07:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:07:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:07:54.380 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b06ffd640 4 set_mon_vals no callback set 2023-10-30T12:07:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 10 client.114146.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:07:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 10 monclient: _renew_subs 2023-10-30T12:07:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b081201e0 con 0x7f6b080a2350 2023-10-30T12:07:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.338+0000 7f6b0ee56640 1 librados: init done 2023-10-30T12:07:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b057fa640 1 -- 172.21.15.50:0/1807956069 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6afc005580 con 0x7f6b080a2350 2023-10-30T12:07:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b057fa640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6aec0392d0 0x7f6aec03b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b057fa640 1 -- 172.21.15.50:0/1807956069 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6275+0+0 (secure 0 0 0) 0x7f6afc04aaf0 con 0x7f6b080a2350 2023-10-30T12:07:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b057fa640 10 client.114146.objecter ms_dispatch 0x7f6b08000b90 osd_map(240..240 src has 1..240) v4 2023-10-30T12:07:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b057fa640 3 client.114146.objecter handle_osd_map got epochs [240,240] > 0 2023-10-30T12:07:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b057fa640 3 client.114146.objecter handle_osd_map decoding full epoch 240 2023-10-30T12:07:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b0ee56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b0ee56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b0ee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b057fa640 20 client.114146.objecter dump_active .. 0 homeless 2023-10-30T12:07:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0804e160 con 0x7f6b080a2350 2023-10-30T12:07:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b07fff640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6aec0392d0 0x7f6aec03b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.339+0000 7f6b07fff640 10 monclient: get_auth_request con 0x7f6aec0392d0 auth_method 0 2023-10-30T12:07:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.340+0000 7f6b057fa640 10 client.114146.objecter ms_handle_connect 0x7f6aec0392d0 2023-10-30T12:07:54.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.340+0000 7f6b07fff640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6aec0392d0 0x7f6aec03b790 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7f6af4036000 tx=0x7f6af4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:07:54.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.344+0000 7f6b057fa640 1 -- 172.21.15.50:0/1807956069 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6afc013020 con 0x7f6b080a2350 2023-10-30T12:07:54.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.344+0000 7f6b057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:07:54.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.344+0000 7f6b057fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:07:54.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.587+0000 7f6b0ee56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:07:54.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.587+0000 7f6b0ee56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:54.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.587+0000 7f6b0ee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:07:54.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.587+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6b08121f30 con 0x7f6b080a2350 2023-10-30T12:07:54.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.587+0000 7f6b057fa640 1 -- 172.21.15.50:0/1807956069 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f6afc005830 con 0x7f6b080a2350 2023-10-30T12:07:54.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.587+0000 7f6b057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:07:54.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.587+0000 7f6b057fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:07:54.590 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:07:54.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 10 librados: watch_flush enter 2023-10-30T12:07:54.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 10 librados: watch_flush exit 2023-10-30T12:07:54.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 20 client.114146.objecter shutdown clearing up homeless session... 2023-10-30T12:07:54.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 10 client.114146.objecter successfully canceled tick 2023-10-30T12:07:54.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6aec0392d0 msgr2=0x7f6aec03b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:54.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6aec0392d0 0x7f6aec03b790 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7f6af4036000 tx=0x7f6af4008030 comp rx=0 tx=0).stop 2023-10-30T12:07:54.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 10 monclient: shutdown 2023-10-30T12:07:54.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:07:54.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b080a2350 msgr2=0x7f6b081ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:07:54.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.589+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b080a2350 0x7f6b081ca8c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00b550 tx=0x7f6afc00ba10 comp rx=0 tx=0).stop 2023-10-30T12:07:54.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 shutdown_connections 2023-10-30T12:07:54.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6aec0392d0 0x7f6aec03b790 unknown :-1 s=CLOSED pgs=4031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b0811e540 0x7f6b0811e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b080a4b60 0x7f6b081cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 --2- 172.21.15.50:0/1807956069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b080a2350 0x7f6b081ca8c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:07:54.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 >> 172.21.15.50:0/1807956069 conn(0x7f6b080a8060 msgr2=0x7f6b081332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:07:54.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 shutdown_connections 2023-10-30T12:07:54.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 -- 172.21.15.50:0/1807956069 wait complete. 2023-10-30T12:07:54.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:54.590+0000 7f6b0ee56640 1 librados: shutdown 2023-10-30T12:07:57.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:07:57.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.718+0000 7f59afce5640 10 monclient: get_monmap_and_config 2023-10-30T12:07:57.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.718+0000 7f59afce5640 10 monclient: build_initial_monmap 2023-10-30T12:07:57.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.718+0000 7f59afce5640 10 monclient: monmap: 2023-10-30T12:07:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:07:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:07:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:07:57.719873+0000 2023-10-30T12:07:57.721 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:07:57.719873+0000 2023-10-30T12:07:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:07:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:07:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:07:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:07:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:07:57.722 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:07:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:07:57.723 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:07:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.719+0000 7f59afce5640 1 Processor -- start 2023-10-30T12:07:57.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.719+0000 7f59afce5640 1 -- start start 2023-10-30T12:07:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.719+0000 7f59afce5640 10 monclient: init 2023-10-30T12:07:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient: _reopen_session rank -1 2023-10-30T12:07:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-10-30T12:07:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59a8135ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:07:57.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient(hunting): picked mon.noname-c con 0x7f59a8135ec0 addr v2:172.21.15.118:3300/0 2023-10-30T12:07:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:07:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient(hunting): picked mon.noname-b con 0x7f59a81362d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:07:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a80974b0 con 0x7f59a8135ae0 2023-10-30T12:07:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a8071cf0 con 0x7f59a8135ec0 2023-10-30T12:07:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:07:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8072380 con 0x7f59a81362d0 2023-10-30T12:07:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient(hunting): _renew_subs 2023-10-30T12:07:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59afce5640 10 monclient(hunting): authenticate will time out at 1322.903280s 2023-10-30T12:07:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49248/0 (socket says 172.21.15.50:49248) 2023-10-30T12:07:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 learned_addr learned my addr 172.21.15.50:0/3026001860 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:07:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 10 monclient(hunting): get_auth_request con 0x7f59a81362d0 auth_method 0 2023-10-30T12:07:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:07:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:57.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:57.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:57.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 10 monclient(hunting): get_auth_request con 0x7f59a8135ec0 auth_method 0 2023-10-30T12:07:57.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:57.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:57.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:07:57.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:07:57.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:57.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:57.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.720+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:07:57.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:57.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 10 monclient(hunting): get_auth_request con 0x7f59a81362d0 auth_method 0 2023-10-30T12:07:57.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:57.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:57.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:57.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:57.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:07:57.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:57.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 10 monclient(hunting): get_auth_request con 0x7f59a8135ec0 auth_method 0 2023-10-30T12:07:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:07:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:07:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:57.921+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:07:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 10 monclient(hunting): get_auth_request con 0x7f59a81362d0 auth_method 0 2023-10-30T12:07:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:07:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:07:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 10 monclient(hunting): get_auth_request con 0x7f59a8135ec0 auth_method 0 2023-10-30T12:07:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:07:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:58.322+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:07:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 10 monclient(hunting): get_auth_request con 0x7f59a81362d0 auth_method 0 2023-10-30T12:07:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:07:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:07:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:07:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 10 monclient(hunting): get_auth_request con 0x7f59a8135ec0 auth_method 0 2023-10-30T12:07:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:07:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:07:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:07:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:07:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:07:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:07:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:07:59.124+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:08:00.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient(hunting): tick 2023-10-30T12:08:00.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:08:00.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:08:00.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:08:00.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 msgr2=0x7f59a813cf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:00.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59a81362d0 0x7f59a813cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:00.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 msgr2=0x7f59a813aa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:00.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.118:3300/0 conn(0x7f59a8135ec0 0x7f59a813aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:00.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.118:6789/0 conn(0x7f59a8135ae0 legacy=0x7f59a8097e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:00.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient: _add_conns ranks=[0,3] 2023-10-30T12:08:00.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f598c0036e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:00.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 0x7f598c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:00.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f598c007370 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:00.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a8097790 con 0x7f598c0036e0 2023-10-30T12:08:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a8071cf0 con 0x7f598c007370 2023-10-30T12:08:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f599f7fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 msgr2=0x7f598c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 0x7f598c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:00.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.721+0000 7f59ada81640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.50:6789/0 conn(0x7f598c0036e0 legacy=0x7f598c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:00.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.922+0000 7f59ada81640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.50:6789/0 conn(0x7f598c0036e0 legacy=0x7f598c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:00.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.922+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 msgr2=0x7f598c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:00.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:00.922+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 0x7f598c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:08:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:01.323+0000 7f59ada81640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.50:6789/0 conn(0x7f598c0036e0 legacy=0x7f598c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:01.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:01.323+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 msgr2=0x7f598c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:01.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:01.323+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 0x7f598c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:08:02.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:02.124+0000 7f59ada81640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.50:6789/0 conn(0x7f598c0036e0 legacy=0x7f598c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:02.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:02.124+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 msgr2=0x7f598c009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:02.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:02.124+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 0x7f598c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:08:03.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): tick 2023-10-30T12:08:03.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:08:03.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:08:03.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:08:03.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 msgr2=0x7f598c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:03.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c007370 0x7f598c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:03.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.50:6789/0 conn(0x7f598c0036e0 legacy=0x7f598c005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:03.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:08:03.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 reap_dead start 2023-10-30T12:08:03.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f598c009ec0 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:03.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c00aad0 0x7f598c0097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:03.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f598c00aad0 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:03.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f598c0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:03.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f598c003bf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:03.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c00af20 con 0x7f598c009ec0 2023-10-30T12:08:03.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:03.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a8071cf0 con 0x7f598c00aad0 2023-10-30T12:08:03.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:03.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 1 -- 172.21.15.50:0/3026001860 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8072380 con 0x7f598c003bf0 2023-10-30T12:08:03.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f599f7fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ada81640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.50:6789/0 conn(0x7f598c009ec0 legacy=0x7f598c00a300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ad280640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c00aad0 msgr2=0x7f598c0097c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ad280640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c00aad0 0x7f598c0097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f598c0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): get_auth_request con 0x7f598c003bf0 auth_method 0 2023-10-30T12:08:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:03.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:03.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient(hunting): handle_auth_done global_id 114152 payload 274 2023-10-30T12:08:03.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c00aad0 msgr2=0x7f598c0097c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:03.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c00aad0 0x7f598c0097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:03.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 >> v1:172.21.15.50:6789/0 conn(0x7f598c009ec0 legacy=0x7f598c00a300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:03.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient: _finish_hunting 0 2023-10-30T12:08:03.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 1 monclient: found mon.noname-b 2023-10-30T12:08:03.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:03.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:03.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.721+0000 7f59ae282640 1 -- 172.21.15.50:0/3026001860 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a8097790 con 0x7f598c003bf0 2023-10-30T12:08:03.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59ae282640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f598c0060b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59a80a13e0 tx=0x7f59a002c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb60e9e321b68831 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:03.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 1 -- 172.21.15.50:0/3026001860 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a0032540 con 0x7f598c003bf0 2023-10-30T12:08:03.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:03.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:03.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: dump: 2023-10-30T12:08:03.732 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:03.733 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:03.733 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.733 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:03.733 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:03.734 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:03.734 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:03.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:03.734 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:03.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: _finish_auth 0 2023-10-30T12:08:03.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: _check_auth_tickets 2023-10-30T12:08:03.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:03.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 1 -- 172.21.15.50:0/3026001860 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a002f030 con 0x7f598c003bf0 2023-10-30T12:08:03.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 1 -- 172.21.15.50:0/3026001860 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a00326c0 con 0x7f598c003bf0 2023-10-30T12:08:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59afce5640 5 monclient: authenticate success, global_id 114152 2023-10-30T12:08:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59afce5640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59afce5640 4 set_mon_vals no callback set 2023-10-30T12:08:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:03.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.722+0000 7f59aca7f640 10 monclient: dump: 2023-10-30T12:08:03.737 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:03.737 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:03.737 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.737 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.738 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:03.738 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:03.738 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:03.738 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:03.738 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:03.739 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:03.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 10 monclient: shutdown 2023-10-30T12:08:03.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:03.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 -- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 msgr2=0x7f598c0060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:03.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f598c0060b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59a80a13e0 tx=0x7f59a002c630 comp rx=0 tx=0).stop 2023-10-30T12:08:03.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 -- 172.21.15.50:0/3026001860 shutdown_connections 2023-10-30T12:08:03.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 --2- 172.21.15.50:0/3026001860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f598c0060b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:03.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 --2- 172.21.15.50:0/3026001860 >> v2:172.21.15.50:3300/0 conn(0x7f598c00aad0 0x7f598c0097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:03.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 -- 172.21.15.50:0/3026001860 >> 172.21.15.50:0/3026001860 conn(0x7f59a80a05c0 msgr2=0x7f59a809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:03.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 -- 172.21.15.50:0/3026001860 shutdown_connections 2023-10-30T12:08:03.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.723+0000 7f59afce5640 1 -- 172.21.15.50:0/3026001860 wait complete. 2023-10-30T12:08:03.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.724+0000 7f59afce5640 10 monclient: build_initial_monmap 2023-10-30T12:08:03.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.724+0000 7f59afce5640 10 monclient: monmap: 2023-10-30T12:08:03.746 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:03.746 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:03.747 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:03.747 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:03.747 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:03.747 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:03.747 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:03.748 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:03.748 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:03.748 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:03.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.724+0000 7f59afce5640 1 librados: starting msgr at 2023-10-30T12:08:03.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.724+0000 7f59afce5640 1 librados: starting objecter 2023-10-30T12:08:03.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.724+0000 7f59afce5640 1 Processor -- start 2023-10-30T12:08:03.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 -- start start 2023-10-30T12:08:03.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 librados: setting wanted keys 2023-10-30T12:08:03.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 librados: calling monclient init 2023-10-30T12:08:03.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient: init 2023-10-30T12:08:03.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:03.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:08:03.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f59a8090bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:03.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): picked mon.noname-c con 0x7f598c003bf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:03.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f598c009ec0 0x7f59a8091100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:03.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): picked mon.noname-a con 0x7f598c009ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:03.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f598c00aad0 0x7f59a8091640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:03.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): picked mon.noname-b con 0x7f598c00aad0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:03.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:03.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8094710 con 0x7f598c009ec0 2023-10-30T12:08:03.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:03.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a809d140 con 0x7f598c00aad0 2023-10-30T12:08:03.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:03.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8094430 con 0x7f598c003bf0 2023-10-30T12:08:03.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:03.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59afce5640 10 monclient(hunting): authenticate will time out at 1328.908601s 2023-10-30T12:08:03.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ad280640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f598c009ec0 msgr2=0x7f59a8091100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:03.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ad280640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f598c009ec0 0x7f59a8091100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:03.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f59a8090bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:03.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f59a8090bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49290/0 (socket says 172.21.15.50:49290) 2023-10-30T12:08:03.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 1 -- 172.21.15.50:0/2320662828 learned_addr learned my addr 172.21.15.50:0/2320662828 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:03.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 10 monclient(hunting): get_auth_request con 0x7f598c003bf0 auth_method 0 2023-10-30T12:08:03.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:03.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:03.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:03.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:03.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ada81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ae282640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f598c00aad0 0x7f59a8091640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ae282640 10 monclient(hunting): get_auth_request con 0x7f598c00aad0 auth_method 0 2023-10-30T12:08:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ae282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ae282640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.725+0000 7f59ae282640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ae282640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ae282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ae282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 10 monclient(hunting): handle_auth_done global_id 114155 payload 824 2023-10-30T12:08:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 1 -- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f598c00aad0 msgr2=0x7f59a8091640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f598c00aad0 0x7f59a8091640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 1 -- 172.21.15.50:0/2320662828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f598c009ec0 msgr2=0x7f59a8091100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f598c009ec0 0x7f59a8091100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 10 monclient: _finish_hunting 0 2023-10-30T12:08:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 1 monclient: found mon.noname-c 2023-10-30T12:08:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 1 -- 172.21.15.50:0/2320662828 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00097a0 con 0x7f598c003bf0 2023-10-30T12:08:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59ada81640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f59a8090bc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f59a400c750 tx=0x7f59a400cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 client.?.objecter ms_handle_connect 0x7f598c003bf0 2023-10-30T12:08:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 1 -- 172.21.15.50:0/2320662828 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4007270 con 0x7f598c003bf0 2023-10-30T12:08:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:03.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: dump: 2023-10-30T12:08:03.760 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:03.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:03.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.760 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:03.761 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:08:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:08:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 1 -- 172.21.15.50:0/2320662828 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a40073d0 con 0x7f598c003bf0 2023-10-30T12:08:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 1 -- 172.21.15.50:0/2320662828 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4001da0 con 0x7f598c003bf0 2023-10-30T12:08:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599e7fc640 10 monclient: dump: 2023-10-30T12:08:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:03.765 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:03.765 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:03.765 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:03.765 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:03.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 5 monclient: authenticate success, global_id 114155 2023-10-30T12:08:03.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 10 monclient: _renew_subs 2023-10-30T12:08:03.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:03.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a80930d0 con 0x7f598c003bf0 2023-10-30T12:08:03.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f599ffff640 4 set_mon_vals no callback set 2023-10-30T12:08:03.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 10 client.114155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 10 monclient: _renew_subs 2023-10-30T12:08:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a8092940 con 0x7f598c003bf0 2023-10-30T12:08:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.726+0000 7f59afce5640 1 librados: init done 2023-10-30T12:08:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.727+0000 7f599e7fc640 1 -- 172.21.15.50:0/2320662828 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f59a4005810 con 0x7f598c003bf0 2023-10-30T12:08:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f599e7fc640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f598c03acf0 0x7f598c03d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f599e7fc640 1 -- 172.21.15.50:0/2320662828 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6009+0+0 (secure 0 0 0) 0x7f59a404a050 con 0x7f598c003bf0 2023-10-30T12:08:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f59afce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f599e7fc640 10 client.114155.objecter ms_dispatch 0x7f59a8000b90 osd_map(242..242 src has 1..242) v4 2023-10-30T12:08:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f599e7fc640 3 client.114155.objecter handle_osd_map got epochs [242,242] > 0 2023-10-30T12:08:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f599e7fc640 3 client.114155.objecter handle_osd_map decoding full epoch 242 2023-10-30T12:08:03.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f59afce5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:03.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f59afce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:03.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a8096a20 con 0x7f598c003bf0 2023-10-30T12:08:03.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f599e7fc640 20 client.114155.objecter dump_active .. 0 homeless 2023-10-30T12:08:03.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f59ad280640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f598c03acf0 0x7f598c03d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:03.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.728+0000 7f59ad280640 10 monclient: get_auth_request con 0x7f598c03acf0 auth_method 0 2023-10-30T12:08:03.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.729+0000 7f599e7fc640 10 client.114155.objecter ms_handle_connect 0x7f598c03acf0 2023-10-30T12:08:03.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.729+0000 7f59ad280640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f598c03acf0 0x7f598c03d1b0 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f5998006fd0 tx=0x7f5998008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:03.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.733+0000 7f599e7fc640 1 -- 172.21.15.50:0/2320662828 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f59a4017020 con 0x7f598c003bf0 2023-10-30T12:08:03.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.733+0000 7f599e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:03.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.733+0000 7f599e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.988+0000 7f59afce5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.988+0000 7f59afce5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.988+0000 7f59afce5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.988+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f59a804e160 con 0x7f598c003bf0 2023-10-30T12:08:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.988+0000 7f599e7fc640 1 -- 172.21.15.50:0/2320662828 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f59a400a640 con 0x7f598c003bf0 2023-10-30T12:08:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.988+0000 7f599e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.988+0000 7f599e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:03.993 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":74,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:03.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 10 librados: watch_flush enter 2023-10-30T12:08:03.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 10 librados: watch_flush exit 2023-10-30T12:08:03.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 20 client.114155.objecter shutdown clearing up homeless session... 2023-10-30T12:08:03.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 10 client.114155.objecter successfully canceled tick 2023-10-30T12:08:03.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f598c03acf0 msgr2=0x7f598c03d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:03.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f598c03acf0 0x7f598c03d1b0 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f5998006fd0 tx=0x7f5998008040 comp rx=0 tx=0).stop 2023-10-30T12:08:03.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 10 monclient: shutdown 2023-10-30T12:08:03.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:03.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 msgr2=0x7f59a8090bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:03.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f59a8090bc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f59a400c750 tx=0x7f59a400cc10 comp rx=0 tx=0).stop 2023-10-30T12:08:03.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 shutdown_connections 2023-10-30T12:08:04.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f598c03acf0 0x7f598c03d1b0 unknown :-1 s=CLOSED pgs=4033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f598c00aad0 0x7f59a8091640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f598c009ec0 0x7f59a8091100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 --2- 172.21.15.50:0/2320662828 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f598c003bf0 0x7f59a8090bc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 >> 172.21.15.50:0/2320662828 conn(0x7f59a80a05c0 msgr2=0x7f59a808afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:04.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 shutdown_connections 2023-10-30T12:08:04.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 -- 172.21.15.50:0/2320662828 wait complete. 2023-10-30T12:08:04.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:03.990+0000 7f59afce5640 1 librados: shutdown 2023-10-30T12:08:04.004 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:08:04.005 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:08:04.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:08:04.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient: get_monmap_and_config 2023-10-30T12:08:04.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient: build_initial_monmap 2023-10-30T12:08:04.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient: monmap: 2023-10-30T12:08:04.114 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:04.114 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.114 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:04.112435+0000 2023-10-30T12:08:04.115 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:04.112435+0000 2023-10-30T12:08:04.115 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:04.116 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.116 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:04.116 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:04.116 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:04.117 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:04.117 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:04.117 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 Processor -- start 2023-10-30T12:08:04.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 -- start start 2023-10-30T12:08:04.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient: init 2023-10-30T12:08:04.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:04.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T12:08:04.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc3809bb70 0x7fcc3809bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc3809bb70 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:04.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcc3809d3f0 0x7fcc3809c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc3809d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:04.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc380964e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc38096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:04.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc38098170 con 0x7fcc3809bb70 2023-10-30T12:08:04.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3809c9d0 con 0x7fcc3809d3f0 2023-10-30T12:08:04.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3809cf60 con 0x7fcc38096080 2023-10-30T12:08:04.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:04.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3f1b6640 10 monclient(hunting): authenticate will time out at 1329.295676s 2023-10-30T12:08:04.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3cf52640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fcc3809bb70 msgr2=0x7fcc3809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:04.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3cf52640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc3809bb70 0x7fcc3809bf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:04.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc380964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc380964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49292/0 (socket says 172.21.15.50:49292) 2023-10-30T12:08:04.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 1 -- 172.21.15.50:0/967651673 learned_addr learned my addr 172.21.15.50:0/967651673 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:04.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 10 monclient(hunting): get_auth_request con 0x7fcc38096080 auth_method 0 2023-10-30T12:08:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.112+0000 7fcc3d753640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 10 monclient(hunting): handle_auth_done global_id 114158 payload 274 2023-10-30T12:08:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 1 -- 172.21.15.50:0/967651673 >> v2:172.21.15.118:3300/0 conn(0x7fcc3809d3f0 msgr2=0x7fcc3809c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc37fff640 1 --2- 172.21.15.50:0/967651673 >> v2:172.21.15.118:3300/0 conn(0x7fcc3809d3f0 0x7fcc3809c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 1 --2- 172.21.15.50:0/967651673 >> v2:172.21.15.118:3300/0 conn(0x7fcc3809d3f0 0x7fcc3809c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 1 -- 172.21.15.50:0/967651673 >> v2:172.21.15.50:3300/0 conn(0x7fcc3809bb70 msgr2=0x7fcc3809bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 1 --2- 172.21.15.50:0/967651673 >> v2:172.21.15.50:3300/0 conn(0x7fcc3809bb70 0x7fcc3809bf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 10 monclient: _finish_hunting 0 2023-10-30T12:08:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 1 monclient: found mon.noname-b 2023-10-30T12:08:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 1 -- 172.21.15.50:0/967651673 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38097b20 con 0x7fcc38096080 2023-10-30T12:08:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3d753640 1 --2- 172.21.15.50:0/967651673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc380964e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc3809d840 tx=0x7fcc3002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b4cbe2ac9806e17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 1 -- 172.21.15.50:0/967651673 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3002c980 con 0x7fcc38096080 2023-10-30T12:08:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 10 monclient: dump: 2023-10-30T12:08:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.141 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.141 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 10 monclient: _finish_auth 0 2023-10-30T12:08:04.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 10 monclient: _check_auth_tickets 2023-10-30T12:08:04.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:04.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 1 -- 172.21.15.50:0/967651673 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3002cae0 con 0x7fcc38096080 2023-10-30T12:08:04.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:04.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc377fe640 1 -- 172.21.15.50:0/967651673 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3002ce70 con 0x7fcc38096080 2023-10-30T12:08:04.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.113+0000 7fcc3f1b6640 5 monclient: authenticate success, global_id 114158 2023-10-30T12:08:04.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:04.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 4 set_mon_vals no callback set 2023-10-30T12:08:04.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:04.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc377fe640 10 monclient: dump: 2023-10-30T12:08:04.144 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.144 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.145 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.145 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.145 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.146 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.146 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.146 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.146 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.146 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 10 monclient: shutdown 2023-10-30T12:08:04.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:04.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/967651673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 msgr2=0x7fcc380964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/967651673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc380964e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc3809d840 tx=0x7fcc3002bde0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/967651673 shutdown_connections 2023-10-30T12:08:04.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/967651673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc380964e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/967651673 >> v2:172.21.15.118:3300/0 conn(0x7fcc3809d3f0 0x7fcc3809c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/967651673 >> v2:172.21.15.50:3300/0 conn(0x7fcc3809bb70 0x7fcc3809bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/967651673 >> 172.21.15.50:0/967651673 conn(0x7fcc380a3980 msgr2=0x7fcc380a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:04.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/967651673 shutdown_connections 2023-10-30T12:08:04.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.114+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/967651673 wait complete. 2023-10-30T12:08:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 10 monclient: build_initial_monmap 2023-10-30T12:08:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 10 monclient: monmap: 2023-10-30T12:08:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:04.150 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:04.150 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:04.150 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.150 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:04.150 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:04.151 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:04.151 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 1 librados: starting msgr at 2023-10-30T12:08:04.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 1 librados: starting objecter 2023-10-30T12:08:04.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 1 Processor -- start 2023-10-30T12:08:04.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 1 -- start start 2023-10-30T12:08:04.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 1 librados: setting wanted keys 2023-10-30T12:08:04.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 1 librados: calling monclient init 2023-10-30T12:08:04.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.115+0000 7fcc3f1b6640 10 monclient: init 2023-10-30T12:08:04.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:04.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:08:04.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc38139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc38096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:04.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc3809bb70 0x7fcc38139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc3809bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:04.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc3809d3f0 0x7fcc3813a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc3809d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:04.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38097190 con 0x7fcc3809d3f0 2023-10-30T12:08:04.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3806f4c0 con 0x7fcc3809bb70 2023-10-30T12:08:04.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3813b3b0 con 0x7fcc38096080 2023-10-30T12:08:04.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:04.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient(hunting): authenticate will time out at 1329.299206s 2023-10-30T12:08:04.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3d753640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc3809d3f0 msgr2=0x7fcc3813a3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:04.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3d753640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc3809d3f0 0x7fcc3813a3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:04.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc38139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc38139970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49298/0 (socket says 172.21.15.50:49298) 2023-10-30T12:08:04.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 -- 172.21.15.50:0/1323056099 learned_addr learned my addr 172.21.15.50:0/1323056099 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:04.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): get_auth_request con 0x7fcc38096080 auth_method 0 2023-10-30T12:08:04.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc37fff640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc3809bb70 0x7fcc38139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc37fff640 10 monclient(hunting): get_auth_request con 0x7fcc3809bb70 auth_method 0 2023-10-30T12:08:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient(hunting): handle_auth_done global_id 114161 payload 824 2023-10-30T12:08:04.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc3809bb70 msgr2=0x7fcc38139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc3809bb70 0x7fcc38139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc3809d3f0 msgr2=0x7fcc3813a3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:04.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc3809d3f0 0x7fcc3813a3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient: _finish_hunting 0 2023-10-30T12:08:04.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 monclient: found mon.noname-c 2023-10-30T12:08:04.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:04.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:04.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 -- 172.21.15.50:0/1323056099 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30009b30 con 0x7fcc38096080 2023-10-30T12:08:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3cf52640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc38139970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00b500 tx=0x7fcc2c00b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 client.?.objecter ms_handle_connect 0x7fcc38096080 2023-10-30T12:08:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c003e10 con 0x7fcc38096080 2023-10-30T12:08:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: dump: 2023-10-30T12:08:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.169 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.169 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.169 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.170 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.170 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.170 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.171 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: _finish_auth 0 2023-10-30T12:08:04.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: _check_auth_tickets 2023-10-30T12:08:04.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:04.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2c005070 con 0x7fcc38096080 2023-10-30T12:08:04.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:04.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c005420 con 0x7fcc38096080 2023-10-30T12:08:04.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:04.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc357fa640 10 monclient: dump: 2023-10-30T12:08:04.173 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.173 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.175 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.175 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.175 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 5 monclient: authenticate success, global_id 114161 2023-10-30T12:08:04.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient: _renew_subs 2023-10-30T12:08:04.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:04.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc36ffd640 4 set_mon_vals no callback set 2023-10-30T12:08:04.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc380976e0 con 0x7fcc38096080 2023-10-30T12:08:04.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 client.114161.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:04.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient: _renew_subs 2023-10-30T12:08:04.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:04.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc38095c30 con 0x7fcc38096080 2023-10-30T12:08:04.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.116+0000 7fcc3f1b6640 1 librados: init done 2023-10-30T12:08:04.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc3f1b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:04.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc3802c5b0 0x7fcc3813a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38073ff0 con 0x7fcc3802c5b0 2023-10-30T12:08:04.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcc2c005580 con 0x7fcc38096080 2023-10-30T12:08:04.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc37fff640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc3802c5b0 0x7fcc3813a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc37fff640 10 monclient: get_auth_request con 0x7fcc3802c5b0 auth_method 0 2023-10-30T12:08:04.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.117+0000 7fcc37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc1c0392a0 0x7fcc1c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fcc2c04ac70 con 0x7fcc38096080 2023-10-30T12:08:04.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 client.114161.objecter ms_dispatch 0x7fcc38000b90 osd_map(243..243 src has 1..243) v4 2023-10-30T12:08:04.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 3 client.114161.objecter handle_osd_map got epochs [243,243] > 0 2023-10-30T12:08:04.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 3 client.114161.objecter handle_osd_map decoding full epoch 243 2023-10-30T12:08:04.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc37fff640 10 monclient(hunting): handle_auth_done global_id 114164 payload 824 2023-10-30T12:08:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc37fff640 1 -- 172.21.15.50:0/1323056099 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc38097560 con 0x7fcc3802c5b0 2023-10-30T12:08:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 20 client.114161.objecter dump_active .. 0 homeless 2023-10-30T12:08:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc3d753640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc1c0392a0 0x7fcc1c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc3d753640 10 monclient: get_auth_request con 0x7fcc1c0392a0 auth_method 0 2023-10-30T12:08:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 client.114161.objecter ms_handle_connect 0x7fcc3802c5b0 2023-10-30T12:08:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 client.114161.objecter resend_mon_ops 2023-10-30T12:08:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc37fff640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc3802c5b0 0x7fcc3813a930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e7a0 tx=0x7fcc2400ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2400ee50 con 0x7fcc3802c5b0 2023-10-30T12:08:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 monclient: dump: 2023-10-30T12:08:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.184 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.184 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.184 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.184 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.184 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.185 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fcc2400a990 con 0x7fcc3802c5b0 2023-10-30T12:08:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc3802c5b0 msgr2=0x7fcc3813a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc3802c5b0 0x7fcc3813a930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e7a0 tx=0x7fcc2400ec60 comp rx=0 tx=0).stop 2023-10-30T12:08:04.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc3802c5b0 msgr2=0x7fcc3813a930 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:04.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc3802c5b0 0x7fcc3813a930 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e7a0 tx=0x7fcc2400ec60 comp rx=0 tx=0).stop 2023-10-30T12:08:04.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc357fa640 10 client.114161.objecter ms_handle_connect 0x7fcc1c0392a0 2023-10-30T12:08:04.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.118+0000 7fcc3d753640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc1c0392a0 0x7fcc1c03b760 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7fcc3809d840 tx=0x7fcc30030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc3f1b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38060fc0 0x7fcc380613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc3f1b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38097e00 con 0x7fcc38060fc0 2023-10-30T12:08:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38060fc0 0x7fcc380613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient: get_auth_request con 0x7fcc38060fc0 auth_method 0 2023-10-30T12:08:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 10 monclient(hunting): handle_auth_done global_id 114167 payload 824 2023-10-30T12:08:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 1 -- 172.21.15.50:0/1323056099 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcc3802c990 con 0x7fcc38060fc0 2023-10-30T12:08:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc37fff640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38060fc0 0x7fcc380613d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e7a0 tx=0x7fcc240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 10 client.114161.objecter ms_handle_connect 0x7fcc38060fc0 2023-10-30T12:08:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 10 client.114161.objecter resend_mon_ops 2023-10-30T12:08:04.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24020500 con 0x7fcc38060fc0 2023-10-30T12:08:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 10 monclient: dump: 2023-10-30T12:08:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.192 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7fcc24003eb0 con 0x7fcc38060fc0 2023-10-30T12:08:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:04.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38060fc0 msgr2=0x7fcc380613d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38060fc0 0x7fcc380613d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e7a0 tx=0x7fcc240046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38060fc0 msgr2=0x7fcc380613d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:04.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.125+0000 7fcc357fa640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38060fc0 0x7fcc380613d0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e7a0 tx=0x7fcc240046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.194 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:04.194 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:04.194 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:04.194 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:08:04.195 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 74, 2023-10-30T12:08:04.195 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:04.195 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:04.195 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:04.195 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.196 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:08:04.196 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:04.196 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:04.196 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:04.196 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:04.196 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:04.197 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:04.197 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:04.197 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:04.197 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:04.197 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:04.198 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:04.198 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:04.198 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:04.198 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.198 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:04.199 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:04.199 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:04.199 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:04.199 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:04.199 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:04.200 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:04.200 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:04.200 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:04.200 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:04.200 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:04.200 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:04.202 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:04.202 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:04.202 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:04.202 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:04.202 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:04.204 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:04.204 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:04.204 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:04.204 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:04.204 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:04.205 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:04.205 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:04.205 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:04.205 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:04.205 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:04.205 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:04.206 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:04.206 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:04.206 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.206 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:04.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:04.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.207 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:04.207 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:04.207 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:04.207 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:04.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.208 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:04.208 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:04.208 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.209 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:04.209 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:04.209 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.210 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.210 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:04.210 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:04.210 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:04.210 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:04.211 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:04.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.211 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:04.211 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:04.212 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:04.212 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:04.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.212 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:04.212 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:04.212 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.213 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:04.213 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:04.213 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.214 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.214 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:04.214 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:04.215 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:04.215 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:04.215 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:04.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.215 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:04.216 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:04.216 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:04.216 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:04.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.216 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:04.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:04.217 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:04.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:04.218 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.218 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.218 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.218 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:04.219 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:04.219 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:04.219 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:04.219 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:04.219 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.220 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.220 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:04.220 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:04.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.220 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:04.221 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:04.221 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:04.221 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.221 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.221 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:04.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.222 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:04.222 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:04.222 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:04.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.223 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:04.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.223 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:04.223 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:04.224 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:08:04.224 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.224 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.224 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:04.225 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:04.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 10 librados: watch_flush enter 2023-10-30T12:08:04.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 10 librados: watch_flush exit 2023-10-30T12:08:04.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 20 client.114161.objecter shutdown clearing up homeless session... 2023-10-30T12:08:04.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 10 client.114161.objecter successfully canceled tick 2023-10-30T12:08:04.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc1c0392a0 msgr2=0x7fcc1c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc1c0392a0 0x7fcc1c03b760 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7fcc3809d840 tx=0x7fcc30030000 comp rx=0 tx=0).stop 2023-10-30T12:08:04.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 10 monclient: shutdown 2023-10-30T12:08:04.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:04.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 msgr2=0x7fcc38139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 --2- 172.21.15.50:0/1323056099 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc38096080 0x7fcc38139970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00b500 tx=0x7fcc2c00b9c0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3d753640 1 -- 172.21.15.50:0/1323056099 reap_dead start 2023-10-30T12:08:04.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3d753640 1 -- 172.21.15.50:0/1323056099 reap_dead start 2023-10-30T12:08:04.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 shutdown_connections 2023-10-30T12:08:04.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 >> 172.21.15.50:0/1323056099 conn(0x7fcc380a3980 msgr2=0x7fcc38138d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:04.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 shutdown_connections 2023-10-30T12:08:04.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 -- 172.21.15.50:0/1323056099 wait complete. 2023-10-30T12:08:04.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.126+0000 7fcc3f1b6640 1 librados: shutdown 2023-10-30T12:08:04.230 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:08:04.230 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:08:04.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:08:04.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.337+0000 7f9bb3b1d640 10 monclient: get_monmap_and_config 2023-10-30T12:08:04.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.337+0000 7f9bb3b1d640 10 monclient: build_initial_monmap 2023-10-30T12:08:04.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.337+0000 7f9bb3b1d640 10 monclient: monmap: 2023-10-30T12:08:04.338 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:04.338 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.339 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:04.337559+0000 2023-10-30T12:08:04.339 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:04.337559+0000 2023-10-30T12:08:04.339 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:04.339 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.339 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:04.340 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:04.340 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:04.340 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:04.340 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:04.340 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 1 Processor -- start 2023-10-30T12:08:04.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 1 -- start start 2023-10-30T12:08:04.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient: init 2023-10-30T12:08:04.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-10-30T12:08:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9bac135ba0 0x7f9bac135f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9bac135ba0 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9bac135670 0x7f9bac0703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9bac135670 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:04.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9bac0708f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:08:04.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac09a870 con 0x7f9bac0708f0 2023-10-30T12:08:04.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac09a040 con 0x7f9bac135ba0 2023-10-30T12:08:04.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac09a2b0 con 0x7f9bac135670 2023-10-30T12:08:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb3b1d640 10 monclient(hunting): authenticate will time out at 1329.520887s 2023-10-30T12:08:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb18b9640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f9bac135ba0 msgr2=0x7f9bac135f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb18b9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9bac135ba0 0x7f9bac135f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9bac135670 0x7f9bac0703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9bac135670 0x7f9bac0703b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:34578/0 (socket says 172.21.15.50:34578) 2023-10-30T12:08:04.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb20ba640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f9bac0708f0 0x7f9bac13c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:57814/0 (socket says 172.21.15.50:57814) 2023-10-30T12:08:04.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 -- 172.21.15.50:0/1832693831 learned_addr learned my addr 172.21.15.50:0/1832693831 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:04.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): get_auth_request con 0x7f9bac135670 auth_method 0 2023-10-30T12:08:04.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_done global_id 94519 payload 274 2023-10-30T12:08:04.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 -- 172.21.15.50:0/1832693831 >> v2:172.21.15.50:3300/0 conn(0x7f9bac135ba0 msgr2=0x7f9bac135f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:04.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 --2- 172.21.15.50:0/1832693831 >> v2:172.21.15.50:3300/0 conn(0x7f9bac135ba0 0x7f9bac135f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 -- 172.21.15.50:0/1832693831 >> v1:172.21.15.118:6789/0 conn(0x7f9bac0708f0 legacy=0x7f9bac13c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient: _finish_hunting 0 2023-10-30T12:08:04.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 monclient: found mon.noname-c 2023-10-30T12:08:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:08:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb08b7640 1 -- 172.21.15.50:0/1832693831 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038305134 0 0) 0x7f9bac09a870 con 0x7f9bac0708f0 2023-10-30T12:08:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb10b8640 1 -- 172.21.15.50:0/1832693831 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac09a590 con 0x7f9bac135670 2023-10-30T12:08:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.338+0000 7f9bb08b7640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:08:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb10b8640 1 --2- 172.21.15.50:0/1832693831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135670 0x7f9bac0703b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9bac0a39c0 tx=0x7f9b9c02b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be7978a5b7dca28e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 1 -- 172.21.15.50:0/1832693831 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c031890 con 0x7f9bac135670 2023-10-30T12:08:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: dump: 2023-10-30T12:08:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.360 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: _finish_auth 0 2023-10-30T12:08:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: _check_auth_tickets 2023-10-30T12:08:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 1 -- 172.21.15.50:0/1832693831 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c0319f0 con 0x7f9bac135670 2023-10-30T12:08:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb3b1d640 5 monclient: authenticate success, global_id 94519 2023-10-30T12:08:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb3b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:04.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:04.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 1 -- 172.21.15.50:0/1832693831 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c031d80 con 0x7f9bac135670 2023-10-30T12:08:04.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:04.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb08b7640 10 monclient: dump: 2023-10-30T12:08:04.363 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.363 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.363 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.363 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.364 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.364 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.364 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.364 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.364 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb3b1d640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb3b1d640 4 set_mon_vals no callback set 2023-10-30T12:08:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb3b1d640 10 monclient: shutdown 2023-10-30T12:08:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb3b1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.339+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/1832693831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135670 msgr2=0x7f9bac0703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 --2- 172.21.15.50:0/1832693831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135670 0x7f9bac0703b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9bac0a39c0 tx=0x7f9b9c02b930 comp rx=0 tx=0).stop 2023-10-30T12:08:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/1832693831 shutdown_connections 2023-10-30T12:08:04.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 --2- 172.21.15.50:0/1832693831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135670 0x7f9bac0703b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 --2- 172.21.15.50:0/1832693831 >> v2:172.21.15.50:3300/0 conn(0x7f9bac135ba0 0x7f9bac135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/1832693831 >> 172.21.15.50:0/1832693831 conn(0x7f9bac0a4500 msgr2=0x7f9bac0a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:04.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/1832693831 shutdown_connections 2023-10-30T12:08:04.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/1832693831 wait complete. 2023-10-30T12:08:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 10 monclient: build_initial_monmap 2023-10-30T12:08:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 10 monclient: monmap: 2023-10-30T12:08:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:04.369 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 librados: starting msgr at 2023-10-30T12:08:04.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 librados: starting objecter 2023-10-30T12:08:04.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.340+0000 7f9bb3b1d640 1 Processor -- start 2023-10-30T12:08:04.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 -- start start 2023-10-30T12:08:04.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 librados: setting wanted keys 2023-10-30T12:08:04.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 librados: calling monclient init 2023-10-30T12:08:04.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient: init 2023-10-30T12:08:04.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:04.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:08:04.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bac0708f0 0x7f9bac11b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9bac0708f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:04.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bac135670 0x7f9bac11b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9bac135670 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:04.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135ba0 0x7f9bac11be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f9bac135ba0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:04.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac11eac0 con 0x7f9bac0708f0 2023-10-30T12:08:04.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac11e620 con 0x7f9bac135ba0 2023-10-30T12:08:04.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1325c0 con 0x7f9bac135670 2023-10-30T12:08:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb3b1d640 10 monclient(hunting): authenticate will time out at 1329.524503s 2023-10-30T12:08:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb18b9640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bac0708f0 msgr2=0x7f9bac11b3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb18b9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bac0708f0 0x7f9bac11b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bac135670 0x7f9bac11b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bac135670 0x7f9bac11b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49316/0 (socket says 172.21.15.50:49316) 2023-10-30T12:08:04.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 -- 172.21.15.50:0/742913882 learned_addr learned my addr 172.21.15.50:0/742913882 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:04.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): get_auth_request con 0x7f9bac135670 auth_method 0 2023-10-30T12:08:04.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb20ba640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135ba0 0x7f9bac11be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb20ba640 10 monclient(hunting): get_auth_request con 0x7f9bac135ba0 auth_method 0 2023-10-30T12:08:04.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb20ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb20ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb20ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient(hunting): handle_auth_done global_id 114170 payload 824 2023-10-30T12:08:04.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135ba0 msgr2=0x7f9bac11be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac135ba0 0x7f9bac11be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bac0708f0 msgr2=0x7f9bac11b3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:04.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bac0708f0 0x7f9bac11b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient: _finish_hunting 0 2023-10-30T12:08:04.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 monclient: found mon.noname-c 2023-10-30T12:08:04.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:04.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:04.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.341+0000 7f9bb10b8640 1 -- 172.21.15.50:0/742913882 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c0094d0 con 0x7f9bac135670 2023-10-30T12:08:04.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb10b8640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bac135670 0x7f9bac11b910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0090b0 tx=0x7f9b9c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 client.?.objecter ms_handle_connect 0x7f9bac135670 2023-10-30T12:08:04.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:04.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c004230 con 0x7f9bac135670 2023-10-30T12:08:04.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:04.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: dump: 2023-10-30T12:08:04.380 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.380 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.380 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.380 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.380 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.381 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.381 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.381 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.381 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.381 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: _finish_auth 0 2023-10-30T12:08:04.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:08:04.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:04.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c004390 con 0x7f9bac135670 2023-10-30T12:08:04.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 5 monclient: authenticate success, global_id 114170 2023-10-30T12:08:04.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 10 monclient: _renew_subs 2023-10-30T12:08:04.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:04.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac09d690 con 0x7f9bac135670 2023-10-30T12:08:04.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:04.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c03bbb0 con 0x7f9bac135670 2023-10-30T12:08:04.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba27fc640 10 monclient: dump: 2023-10-30T12:08:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.385 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.385 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.385 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.385 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.385 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.385 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9ba37fe640 4 set_mon_vals no callback set 2023-10-30T12:08:04.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 10 client.114170.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:04.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 10 monclient: _renew_subs 2023-10-30T12:08:04.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:04.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac11cf90 con 0x7f9bac135670 2023-10-30T12:08:04.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.342+0000 7f9bb3b1d640 1 librados: init done 2023-10-30T12:08:04.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb3b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:04.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb3b1d640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac133260 0x7f9bac133640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb3b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac066730 con 0x7f9bac133260 2023-10-30T12:08:04.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9b9c043070 con 0x7f9bac135670 2023-10-30T12:08:04.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac133260 0x7f9bac133640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient: get_auth_request con 0x7f9bac133260 auth_method 0 2023-10-30T12:08:04.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b900392a0 0x7f9b9003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f9b9c06d140 con 0x7f9bac135670 2023-10-30T12:08:04.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 10 client.114170.objecter ms_dispatch 0x7f9bac000b90 osd_map(243..243 src has 1..243) v4 2023-10-30T12:08:04.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 3 client.114170.objecter handle_osd_map got epochs [243,243] > 0 2023-10-30T12:08:04.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 3 client.114170.objecter handle_osd_map decoding full epoch 243 2023-10-30T12:08:04.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 20 client.114170.objecter dump_active .. 0 homeless 2023-10-30T12:08:04.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb20ba640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b900392a0 0x7f9b9003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb20ba640 10 monclient: get_auth_request con 0x7f9b900392a0 auth_method 0 2023-10-30T12:08:04.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_done global_id 94528 payload 824 2023-10-30T12:08:04.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 1 -- 172.21.15.50:0/742913882 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bac1364c0 con 0x7f9bac133260 2023-10-30T12:08:04.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 10 client.114170.objecter ms_handle_connect 0x7f9bac133260 2023-10-30T12:08:04.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9ba27fc640 10 client.114170.objecter resend_mon_ops 2023-10-30T12:08:04.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.343+0000 7f9bb18b9640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac133260 0x7f9bac133640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800bcc0 tx=0x7f9ba8009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 10 client.114170.objecter ms_handle_connect 0x7f9b900392a0 2023-10-30T12:08:04.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9bb20ba640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b900392a0 0x7f9b9003b760 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4000f60 tx=0x7f9ba4007c70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8006ea0 con 0x7f9bac133260 2023-10-30T12:08:04.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 10 monclient: dump: 2023-10-30T12:08:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.394 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.394 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.394 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.394 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.394 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9ba800f070 con 0x7f9bac133260 2023-10-30T12:08:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac133260 msgr2=0x7f9bac133640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac133260 0x7f9bac133640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800bcc0 tx=0x7f9ba8009f30 comp rx=0 tx=0).stop 2023-10-30T12:08:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac133260 msgr2=0x7f9bac133640 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:04.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.344+0000 7f9ba27fc640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac133260 0x7f9bac133640 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.351+0000 7f9bb3b1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.351+0000 7f9bb3b1d640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac11ece0 0x7f9bac133b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.351+0000 7f9bb3b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.351+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac04e160 con 0x7f9bac11ece0 2023-10-30T12:08:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac11ece0 0x7f9bac133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient: get_auth_request con 0x7f9bac11ece0 auth_method 0 2023-10-30T12:08:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 10 monclient(hunting): handle_auth_done global_id 94531 payload 824 2023-10-30T12:08:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 1 -- 172.21.15.50:0/742913882 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9bac069de0 con 0x7f9bac11ece0 2023-10-30T12:08:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9ba27fc640 10 client.114170.objecter ms_handle_connect 0x7f9bac11ece0 2023-10-30T12:08:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9ba27fc640 10 client.114170.objecter resend_mon_ops 2023-10-30T12:08:04.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9bb18b9640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac11ece0 0x7f9bac133b80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8000f80 tx=0x7f9ba8005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:04.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8013c10 con 0x7f9bac11ece0 2023-10-30T12:08:04.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:04.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9ba27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:04.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9ba27fc640 10 monclient: dump: 2023-10-30T12:08:04.402 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:04.402 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:04.402 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:04.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:04.402 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:04.403 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:04.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:04.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:04.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:04.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.352+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7f9ba8013d70 con 0x7f9bac11ece0 2023-10-30T12:08:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9ba27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9ba27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac11ece0 msgr2=0x7f9bac133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9ba27fc640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac11ece0 0x7f9bac133b80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8000f80 tx=0x7f9ba8005d50 comp rx=0 tx=0).stop 2023-10-30T12:08:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9ba27fc640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac11ece0 msgr2=0x7f9bac133b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9ba27fc640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bac11ece0 0x7f9bac133b80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:04.405 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:04.405 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:04.406 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:08:04.406 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 74, 2023-10-30T12:08:04.406 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:04.406 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:04.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:04.406 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.407 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:08:04.407 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:04.407 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:04.407 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:04.407 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:04.408 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:04.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:04.408 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:04.408 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:04.408 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:04.409 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:04.409 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:04.409 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:04.409 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:04.409 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.409 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:04.410 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:04.410 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:04.410 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:04.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:04.410 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:04.411 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:04.411 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:04.411 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:04.411 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:04.411 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:04.412 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:04.413 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:04.413 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:04.413 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:04.413 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:04.413 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:04.414 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:04.414 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:04.414 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:04.414 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:04.414 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:04.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:04.415 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:04.415 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:04.415 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:04.415 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:04.415 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:04.416 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:04.416 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:04.416 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:04.416 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:04.416 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:04.417 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:04.417 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:04.417 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:04.417 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.417 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:04.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.418 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:04.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.418 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:04.418 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:04.418 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:04.419 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:04.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.419 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:04.419 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:04.419 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.421 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:04.421 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:04.421 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:04.422 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:04.422 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:04.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.422 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:04.423 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:04.423 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:04.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:04.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.423 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:04.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:04.424 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.424 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:04.424 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:04.425 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.425 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.425 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:04.425 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:04.426 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:04.426 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:04.426 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:04.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.427 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:04.427 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:04.427 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:04.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:04.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:04.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:04.428 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:04.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:04.429 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:04.429 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.429 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.430 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:04.430 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:04.430 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:04.430 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:04.430 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:04.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.431 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:04.431 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:04.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.432 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:04.432 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:04.432 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:04.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.432 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:04.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.433 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:04.433 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:04.433 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:04.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.434 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.434 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:04.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.434 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:04.434 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:04.435 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:08:04.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.435 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:04.435 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:08:04.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:04.436 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:04.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:04.437 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:04.437 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 10 librados: watch_flush enter 2023-10-30T12:08:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 10 librados: watch_flush exit 2023-10-30T12:08:04.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 20 client.114170.objecter shutdown clearing up homeless session... 2023-10-30T12:08:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 10 client.114170.objecter successfully canceled tick 2023-10-30T12:08:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b900392a0 msgr2=0x7f9b9003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b900392a0 0x7f9b9003b760 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4000f60 tx=0x7f9ba4007c70 comp rx=0 tx=0).stop 2023-10-30T12:08:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 10 monclient: shutdown 2023-10-30T12:08:04.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bac135670 msgr2=0x7f9bac11b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 --2- 172.21.15.50:0/742913882 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bac135670 0x7f9bac11b910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0090b0 tx=0x7f9b9c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb20ba640 1 -- 172.21.15.50:0/742913882 reap_dead start 2023-10-30T12:08:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb20ba640 1 -- 172.21.15.50:0/742913882 reap_dead start 2023-10-30T12:08:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 shutdown_connections 2023-10-30T12:08:04.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 >> 172.21.15.50:0/742913882 conn(0x7f9bac0a4500 msgr2=0x7f9bac0a1910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 shutdown_connections 2023-10-30T12:08:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 -- 172.21.15.50:0/742913882 wait complete. 2023-10-30T12:08:04.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:04.353+0000 7f9bb3b1d640 1 librados: shutdown 2023-10-30T12:08:04.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:08:04.443 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:08:24.444 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:08:24.444 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:08:24.445 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:08:24.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:08:24.449 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:08:24.449 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:08:24.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:24.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.563+0000 7ff6ea0a9640 10 monclient: get_monmap_and_config 2023-10-30T12:08:24.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.563+0000 7ff6ea0a9640 10 monclient: build_initial_monmap 2023-10-30T12:08:24.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.563+0000 7ff6ea0a9640 10 monclient: monmap: 2023-10-30T12:08:24.565 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:24.566 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:24.566 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:24.564795+0000 2023-10-30T12:08:24.566 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:24.564795+0000 2023-10-30T12:08:24.566 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:24.566 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:24.567 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:24.567 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:24.567 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:24.567 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:24.567 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:24.568 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.564+0000 7ff6ea0a9640 1 Processor -- start 2023-10-30T12:08:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.564+0000 7ff6ea0a9640 1 -- start start 2023-10-30T12:08:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.564+0000 7ff6ea0a9640 10 monclient: init 2023-10-30T12:08:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:08:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e40a6070 0x7ff6e40a6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e40a6070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff6e4099a40 0x7ff6e41355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e4099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 0x7ff6e4135f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e4135b10 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40a6de0 con 0x7ff6e4135b10 2023-10-30T12:08:24.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:24.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e409a850 con 0x7ff6e4099a40 2023-10-30T12:08:24.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:24.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40a5c90 con 0x7ff6e40a6070 2023-10-30T12:08:24.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6ea0a9640 10 monclient(hunting): authenticate will time out at 1349.748193s 2023-10-30T12:08:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 0x7ff6e4135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e40a6070 0x7ff6e40a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 0x7ff6e4135f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59134/0 (socket says 172.21.15.50:59134) 2023-10-30T12:08:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 1 -- 172.21.15.50:0/4191151746 learned_addr learned my addr 172.21.15.50:0/4191151746 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 10 monclient(hunting): get_auth_request con 0x7ff6e4135b10 auth_method 0 2023-10-30T12:08:24.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:24.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:24.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:24.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): get_auth_request con 0x7ff6e40a6070 auth_method 0 2023-10-30T12:08:24.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 1 -- 172.21.15.50:0/4191151746 >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 msgr2=0x7ff6e4135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:08:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 1 -- 172.21.15.50:0/4191151746 >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 msgr2=0x7ff6e4135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 0x7ff6e4135f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e3fff640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 0x7ff6e4135f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e2ffd640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.118:3300/0 conn(0x7ff6e4099a40 0x7ff6e41355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e2ffd640 10 monclient(hunting): get_auth_request con 0x7ff6e4099a40 auth_method 0 2023-10-30T12:08:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient(hunting): handle_auth_done global_id 114182 payload 274 2023-10-30T12:08:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 1 -- 172.21.15.50:0/4191151746 >> v2:172.21.15.118:3300/0 conn(0x7ff6e4099a40 msgr2=0x7ff6e41355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.118:3300/0 conn(0x7ff6e4099a40 0x7ff6e41355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 1 -- 172.21.15.50:0/4191151746 >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 msgr2=0x7ff6e4135f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 0x7ff6e4135f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:08:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 1 monclient: found mon.noname-b 2023-10-30T12:08:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:24.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:24.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e37fe640 1 -- 172.21.15.50:0/4191151746 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e409a570 con 0x7ff6e40a6070 2023-10-30T12:08:24.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.565+0000 7ff6e2ffd640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.118:3300/0 conn(0x7ff6e4099a40 0x7ff6e41355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:08:24.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e37fe640 1 --2- 172.21.15.50:0/4191151746 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e40a6070 0x7ff6e40a6450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40a8a10 tx=0x7ff6d402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b67639bfbdbcb9da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:24.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 1 -- 172.21.15.50:0/4191151746 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4030710 con 0x7ff6e40a6070 2023-10-30T12:08:24.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:24.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:24.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: dump: 2023-10-30T12:08:24.586 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:24.586 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:24.587 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.587 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.587 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:24.587 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:24.587 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:24.587 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:24.588 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: _finish_auth 0 2023-10-30T12:08:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:08:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 1 -- 172.21.15.50:0/4191151746 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d402e050 con 0x7ff6e40a6070 2023-10-30T12:08:24.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6ea0a9640 5 monclient: authenticate success, global_id 114182 2023-10-30T12:08:24.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6ea0a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:24.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:24.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6ea0a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:24.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6ea0a9640 4 set_mon_vals no callback set 2023-10-30T12:08:24.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 1 -- 172.21.15.50:0/4191151746 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4030a90 con 0x7ff6e40a6070 2023-10-30T12:08:24.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:24.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:24.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.566+0000 7ff6e27fc640 10 monclient: dump: 2023-10-30T12:08:24.590 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:24.590 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:24.591 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.591 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.591 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:24.591 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:24.591 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:24.592 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:24.592 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:24.592 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:24.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 10 monclient: shutdown 2023-10-30T12:08:24.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:24.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/4191151746 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e40a6070 msgr2=0x7ff6e40a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:24.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/4191151746 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e40a6070 0x7ff6e40a6450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40a8a10 tx=0x7ff6d402b710 comp rx=0 tx=0).stop 2023-10-30T12:08:24.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/4191151746 shutdown_connections 2023-10-30T12:08:24.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.50:3300/0 conn(0x7ff6e4135b10 0x7ff6e4135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/4191151746 >> v2:172.21.15.118:3300/0 conn(0x7ff6e4099a40 0x7ff6e41355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/4191151746 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e40a6070 0x7ff6e40a6450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/4191151746 >> 172.21.15.50:0/4191151746 conn(0x7ff6e40a8200 msgr2=0x7ff6e40a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:24.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/4191151746 shutdown_connections 2023-10-30T12:08:24.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/4191151746 wait complete. 2023-10-30T12:08:24.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 10 monclient: build_initial_monmap 2023-10-30T12:08:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.567+0000 7ff6ea0a9640 10 monclient: monmap: 2023-10-30T12:08:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:24.596 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 librados: starting msgr at 2023-10-30T12:08:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 librados: starting objecter 2023-10-30T12:08:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 Processor -- start 2023-10-30T12:08:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 -- start start 2023-10-30T12:08:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 librados: setting wanted keys 2023-10-30T12:08:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 librados: calling monclient init 2023-10-30T12:08:24.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient: init 2023-10-30T12:08:24.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:24.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:08:24.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6e4099a40 0x7ff6e41ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:24.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e4099a40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:24.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 0x7ff6e41cea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:24.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e40a6070 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:24.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e4135b10 0x7ff6e41cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:24.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e4135b10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:24.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:24.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e406d9a0 con 0x7ff6e40a6070 2023-10-30T12:08:24.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:24.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e406ea60 con 0x7ff6e4099a40 2023-10-30T12:08:24.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:24.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e406db70 con 0x7ff6e4135b10 2023-10-30T12:08:24.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:24.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6ea0a9640 10 monclient(hunting): authenticate will time out at 1349.751914s 2023-10-30T12:08:24.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.568+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 0x7ff6e41cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:24.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e4135b10 0x7ff6e41cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:24.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 0x7ff6e41cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59148/0 (socket says 172.21.15.50:59148) 2023-10-30T12:08:24.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e4135b10 0x7ff6e41cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52138/0 (socket says 172.21.15.50:52138) 2023-10-30T12:08:24.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 1 -- 172.21.15.50:0/3475845894 learned_addr learned my addr 172.21.15.50:0/3475845894 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:24.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 10 monclient(hunting): get_auth_request con 0x7ff6e40a6070 auth_method 0 2023-10-30T12:08:24.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:24.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:24.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:24.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): get_auth_request con 0x7ff6e4135b10 auth_method 0 2023-10-30T12:08:24.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:24.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:24.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:24.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 1 -- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 msgr2=0x7ff6e41cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:08:24.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6e4099a40 0x7ff6e41ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:24.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:24.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 1 -- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 msgr2=0x7ff6e41cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:08:24.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:24.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 0x7ff6e41cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:24.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:24.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 10 monclient(hunting): get_auth_request con 0x7ff6e4099a40 auth_method 0 2023-10-30T12:08:24.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e2ffd640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 0x7ff6e41cea60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:24.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:24.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:24.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:24.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:24.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:24.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:24.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient(hunting): handle_auth_done global_id 114185 payload 824 2023-10-30T12:08:24.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 -- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6e4099a40 msgr2=0x7ff6e41ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:24.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6e4099a40 0x7ff6e41ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 -- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 msgr2=0x7ff6e41cea60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:24.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 0x7ff6e41cea60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:08:24.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 monclient: found mon.noname-c 2023-10-30T12:08:24.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:24.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:24.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 -- 172.21.15.50:0/3475845894 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009400 con 0x7ff6e4135b10 2023-10-30T12:08:24.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e3fff640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e4135b10 0x7ff6e41cefa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc007a60 tx=0x7ff6dc007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:24.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 client.?.objecter ms_handle_connect 0x7ff6e4135b10 2023-10-30T12:08:24.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:24.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 1 -- 172.21.15.50:0/3475845894 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00dbe0 con 0x7ff6e4135b10 2023-10-30T12:08:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: dump: 2023-10-30T12:08:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:24.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: _finish_auth 0 2023-10-30T12:08:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:08:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 5 monclient: authenticate success, global_id 114185 2023-10-30T12:08:24.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 1 -- 172.21.15.50:0/3475845894 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc00f040 con 0x7ff6e4135b10 2023-10-30T12:08:24.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 10 monclient: _renew_subs 2023-10-30T12:08:24.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:24.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e406f360 con 0x7ff6e4135b10 2023-10-30T12:08:24.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 1 -- 172.21.15.50:0/3475845894 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc015400 con 0x7ff6e4135b10 2023-10-30T12:08:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6cbfff640 10 monclient: dump: 2023-10-30T12:08:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:24.614 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.614 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:24.614 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:24.614 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:24.614 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:24.615 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:24.615 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:24.615 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:24.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 10 client.114185.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:24.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 10 monclient: _renew_subs 2023-10-30T12:08:24.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6e1ffb640 4 set_mon_vals no callback set 2023-10-30T12:08:24.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:24.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e406f020 con 0x7ff6e4135b10 2023-10-30T12:08:24.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.569+0000 7ff6ea0a9640 1 librados: init done 2023-10-30T12:08:24.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6cbfff640 1 -- 172.21.15.50:0/3475845894 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff6dc01e3b0 con 0x7ff6e4135b10 2023-10-30T12:08:24.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6ea0a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:24.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6ea0a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:24.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6ea0a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:24.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e40a7500 con 0x7ff6e4135b10 2023-10-30T12:08:24.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6cbfff640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6d00392f0 0x7ff6d003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:24.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6cbfff640 1 -- 172.21.15.50:0/3475845894 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6524+0+0 (secure 0 0 0) 0x7ff6dc04abd0 con 0x7ff6e4135b10 2023-10-30T12:08:24.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6cbfff640 10 client.114185.objecter ms_dispatch 0x7ff6e4000b90 osd_map(264..264 src has 1..264) v4 2023-10-30T12:08:24.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6cbfff640 3 client.114185.objecter handle_osd_map got epochs [264,264] > 0 2023-10-30T12:08:24.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.570+0000 7ff6cbfff640 3 client.114185.objecter handle_osd_map decoding full epoch 264 2023-10-30T12:08:24.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.571+0000 7ff6e37fe640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6d00392f0 0x7ff6d003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:24.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.571+0000 7ff6cbfff640 20 client.114185.objecter dump_active .. 0 homeless 2023-10-30T12:08:24.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.571+0000 7ff6e37fe640 10 monclient: get_auth_request con 0x7ff6d00392f0 auth_method 0 2023-10-30T12:08:24.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.571+0000 7ff6cbfff640 10 client.114185.objecter ms_handle_connect 0x7ff6d00392f0 2023-10-30T12:08:24.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.571+0000 7ff6e37fe640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6d00392f0 0x7ff6d003b7b0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4034000 tx=0x7ff6d402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:24.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.575+0000 7ff6cbfff640 1 -- 172.21.15.50:0/3475845894 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6e40a7500 con 0x7ff6e4135b10 2023-10-30T12:08:24.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.575+0000 7ff6cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:24.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.575+0000 7ff6cbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.819+0000 7ff6ea0a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.819+0000 7ff6ea0a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.819+0000 7ff6ea0a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:24.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.819+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff6e40a7500 con 0x7ff6e4135b10 2023-10-30T12:08:24.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.820+0000 7ff6cbfff640 1 -- 172.21.15.50:0/3475845894 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7ff6e40a7500 con 0x7ff6e4135b10 2023-10-30T12:08:24.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.820+0000 7ff6cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:24.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.820+0000 7ff6cbfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:24.823 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":74,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:24.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 10 librados: watch_flush enter 2023-10-30T12:08:24.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 10 librados: watch_flush exit 2023-10-30T12:08:24.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 20 client.114185.objecter shutdown clearing up homeless session... 2023-10-30T12:08:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 10 client.114185.objecter successfully canceled tick 2023-10-30T12:08:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6d00392f0 msgr2=0x7ff6d003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6d00392f0 0x7ff6d003b7b0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4034000 tx=0x7ff6d402f000 comp rx=0 tx=0).stop 2023-10-30T12:08:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 10 monclient: shutdown 2023-10-30T12:08:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e4135b10 msgr2=0x7ff6e41cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e4135b10 0x7ff6e41cefa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc007a60 tx=0x7ff6dc007f20 comp rx=0 tx=0).stop 2023-10-30T12:08:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 shutdown_connections 2023-10-30T12:08:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6d00392f0 0x7ff6d003b7b0 unknown :-1 s=CLOSED pgs=4040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6e4135b10 0x7ff6e41cefa0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e40a6070 0x7ff6e41cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 --2- 172.21.15.50:0/3475845894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff6e4099a40 0x7ff6e41ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.821+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 >> 172.21.15.50:0/3475845894 conn(0x7ff6e40a8200 msgr2=0x7ff6e4131a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.822+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 shutdown_connections 2023-10-30T12:08:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.822+0000 7ff6ea0a9640 1 -- 172.21.15.50:0/3475845894 wait complete. 2023-10-30T12:08:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:24.822+0000 7ff6ea0a9640 1 librados: shutdown 2023-10-30T12:08:27.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:27.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.949+0000 7f653a661640 10 monclient: get_monmap_and_config 2023-10-30T12:08:27.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.949+0000 7f653a661640 10 monclient: build_initial_monmap 2023-10-30T12:08:27.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.949+0000 7f653a661640 10 monclient: monmap: 2023-10-30T12:08:27.951 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:27.952 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:27.952 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:27.950906+0000 2023-10-30T12:08:27.952 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:27.950906+0000 2023-10-30T12:08:27.952 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:27.952 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:27.953 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:27.953 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:27.953 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:27.953 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:27.953 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:27.953 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:27.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.950+0000 7f653a661640 1 Processor -- start 2023-10-30T12:08:27.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.950+0000 7f653a661640 1 -- start start 2023-10-30T12:08:27.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.950+0000 7f653a661640 10 monclient: init 2023-10-30T12:08:27.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:27.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T12:08:27.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6534135ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:27.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6534135ec0 0x7f65341362a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:27.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient(hunting): picked mon.noname-b con 0x7f6534135ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:27.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f653413aa20 addr v1:172.21.15.118:6789/0 2023-10-30T12:08:27.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6534071cf0 con 0x7f6534135ae0 2023-10-30T12:08:27.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6534072380 con 0x7f653413aa20 2023-10-30T12:08:27.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:27.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6534097790 con 0x7f6534135ec0 2023-10-30T12:08:27.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:27.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f653a661640 10 monclient(hunting): authenticate will time out at 1353.134141s 2023-10-30T12:08:27.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6534135ec0 0x7f65341362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:27.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f6533fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6534135ae0 0x7f6534097e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53332/0 (socket says 172.21.15.50:53332) 2023-10-30T12:08:27.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6534135ec0 0x7f65341362a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37828/0 (socket says 172.21.15.50:37828) 2023-10-30T12:08:27.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f6533fff640 1 -- 172.21.15.50:0/2933561896 learned_addr learned my addr 172.21.15.50:0/2933561896 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:08:27.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): get_auth_request con 0x7f6534135ec0 auth_method 0 2023-10-30T12:08:27.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:27.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:27.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:27.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:27.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:27.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:27.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient(hunting): handle_auth_done global_id 114188 payload 274 2023-10-30T12:08:27.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 1 -- 172.21.15.50:0/2933561896 >> v1:172.21.15.118:6789/0 conn(0x7f653413aa20 legacy=0x7f653413ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:27.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 1 -- 172.21.15.50:0/2933561896 >> v1:172.21.15.50:6789/0 conn(0x7f6534135ae0 legacy=0x7f6534097e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:27.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient: _finish_hunting 0 2023-10-30T12:08:27.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 1 monclient: found mon.noname-b 2023-10-30T12:08:27.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:27.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:27.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.951+0000 7f65337fe640 1 -- 172.21.15.50:0/2933561896 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6534097a70 con 0x7f6534135ec0 2023-10-30T12:08:27.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f65337fe640 1 --2- 172.21.15.50:0/2933561896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6534135ec0 0x7f65341362a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65240097d0 tx=0x7f652402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb7201d885e91f8b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:27.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 1 -- 172.21.15.50:0/2933561896 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524031800 con 0x7f6534135ec0 2023-10-30T12:08:27.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:27.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:27.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: dump: 2023-10-30T12:08:27.967 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:27.968 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:27.968 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.968 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.968 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:27.968 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:27.969 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:27.969 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:27.969 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:27.969 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:27.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:27.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:27.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:27.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 1 -- 172.21.15.50:0/2933561896 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6524031960 con 0x7f6534135ec0 2023-10-30T12:08:27.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f653a661640 5 monclient: authenticate success, global_id 114188 2023-10-30T12:08:27.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f653a661640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:27.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:27.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 1 -- 172.21.15.50:0/2933561896 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524031cf0 con 0x7f6534135ec0 2023-10-30T12:08:27.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f653a661640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:27.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f653a661640 4 set_mon_vals no callback set 2023-10-30T12:08:27.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:27.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:27.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.952+0000 7f6532ffd640 10 monclient: dump: 2023-10-30T12:08:27.972 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:27.972 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:27.972 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.972 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.972 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:27.973 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:27.973 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:27.973 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:27.973 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:27.973 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:27.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 10 monclient: shutdown 2023-10-30T12:08:27.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:27.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 1 -- 172.21.15.50:0/2933561896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6534135ec0 msgr2=0x7f65341362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:27.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 1 --2- 172.21.15.50:0/2933561896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6534135ec0 0x7f65341362a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65240097d0 tx=0x7f652402b8a0 comp rx=0 tx=0).stop 2023-10-30T12:08:27.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 1 -- 172.21.15.50:0/2933561896 shutdown_connections 2023-10-30T12:08:27.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 1 --2- 172.21.15.50:0/2933561896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6534135ec0 0x7f65341362a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:27.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 1 -- 172.21.15.50:0/2933561896 >> 172.21.15.50:0/2933561896 conn(0x7f65340a05c0 msgr2=0x7f653409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:27.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 1 -- 172.21.15.50:0/2933561896 shutdown_connections 2023-10-30T12:08:27.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 1 -- 172.21.15.50:0/2933561896 wait complete. 2023-10-30T12:08:27.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 10 monclient: build_initial_monmap 2023-10-30T12:08:27.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.953+0000 7f653a661640 10 monclient: monmap: 2023-10-30T12:08:27.976 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:27.976 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:27.976 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:27.976 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:27.976 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:27.976 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:27.977 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:27.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:27.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:27.977 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:27.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 librados: starting msgr at 2023-10-30T12:08:27.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 librados: starting objecter 2023-10-30T12:08:27.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 Processor -- start 2023-10-30T12:08:27.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 -- start start 2023-10-30T12:08:27.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 librados: setting wanted keys 2023-10-30T12:08:27.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 librados: calling monclient init 2023-10-30T12:08:27.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient: init 2023-10-30T12:08:27.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:27.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:08:27.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 0x7f6534096200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:27.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): picked mon.noname-a con 0x7f6534135ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:27.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6534135ec0 0x7f6534096740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:27.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): picked mon.noname-b con 0x7f6534135ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:27.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f653413aa20 0x7f6534137cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:27.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): picked mon.noname-c con 0x7f653413aa20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:27.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:27.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653408c420 con 0x7f6534135ae0 2023-10-30T12:08:27.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:27.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65340a1780 con 0x7f6534135ec0 2023-10-30T12:08:27.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:27.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6534096fd0 con 0x7f653413aa20 2023-10-30T12:08:27.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:27.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.954+0000 7f653a661640 10 monclient(hunting): authenticate will time out at 1353.137926s 2023-10-30T12:08:27.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 0x7f6534096200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:27.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f653413aa20 0x7f6534137cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:27.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 0x7f6534096200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60246/0 (socket says 172.21.15.50:60246) 2023-10-30T12:08:27.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 1 -- 172.21.15.50:0/1109105311 learned_addr learned my addr 172.21.15.50:0/1109105311 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:27.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 10 monclient(hunting): get_auth_request con 0x7f6534135ae0 auth_method 0 2023-10-30T12:08:27.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:27.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:27.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:27.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): get_auth_request con 0x7f653413aa20 auth_method 0 2023-10-30T12:08:27.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:27.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:27.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:27.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6534135ec0 0x7f6534096740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:27.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 1 -- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 msgr2=0x7f6534096200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:08:27.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 1 -- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 msgr2=0x7f6534096200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:08:27.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 0x7f6534096200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:27.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:27.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:27.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:27.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 10 monclient(hunting): get_auth_request con 0x7f6534135ec0 auth_method 0 2023-10-30T12:08:27.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:27.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:27.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:27.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6533fff640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 0x7f6534096200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:27.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:27.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:27.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f65337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:27.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient(hunting): handle_auth_done global_id 114191 payload 824 2023-10-30T12:08:27.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 -- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6534135ec0 msgr2=0x7f6534096740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:27.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6534135ec0 0x7f6534096740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:27.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 -- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 msgr2=0x7f6534096200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:27.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 0x7f6534096200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:27.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient: _finish_hunting 0 2023-10-30T12:08:27.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 monclient: found mon.noname-c 2023-10-30T12:08:27.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:27.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:27.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 -- 172.21.15.50:0/1109105311 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524009400 con 0x7f653413aa20 2023-10-30T12:08:27.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6538bfe640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f653413aa20 0x7f6534137cf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f652c00c690 tx=0x7f652c00cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:27.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 client.?.objecter ms_handle_connect 0x7f653413aa20 2023-10-30T12:08:27.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:27.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 1 -- 172.21.15.50:0/1109105311 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c003b70 con 0x7f653413aa20 2023-10-30T12:08:27.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:27.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:27.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: dump: 2023-10-30T12:08:27.990 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:27.990 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:27.990 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.990 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.991 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:27.991 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:27.991 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:27.991 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:27.991 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:27.991 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:27.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: _finish_auth 0 2023-10-30T12:08:27.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:08:27.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:27.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 1 -- 172.21.15.50:0/1109105311 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652c003cd0 con 0x7f653413aa20 2023-10-30T12:08:27.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 5 monclient: authenticate success, global_id 114191 2023-10-30T12:08:27.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 10 monclient: _renew_subs 2023-10-30T12:08:27.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:27.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653413a090 con 0x7f653413aa20 2023-10-30T12:08:27.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:27.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 1 -- 172.21.15.50:0/1109105311 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c005310 con 0x7f653413aa20 2023-10-30T12:08:27.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:27.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:27.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6530ff9640 10 monclient: dump: 2023-10-30T12:08:27.994 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:27.994 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:27.994 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.995 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:27.995 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:27.995 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:27.995 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:27.995 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:27.996 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:27.996 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:27.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f6531ffb640 4 set_mon_vals no callback set 2023-10-30T12:08:27.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 10 client.114191.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:27.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 10 monclient: _renew_subs 2023-10-30T12:08:27.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:27.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65341398d0 con 0x7f653413aa20 2023-10-30T12:08:27.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.955+0000 7f653a661640 1 librados: init done 2023-10-30T12:08:27.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f6530ff9640 1 -- 172.21.15.50:0/1109105311 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f652c005470 con 0x7f653413aa20 2023-10-30T12:08:27.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f653a661640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:27.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f653a661640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:27.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f653a661640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:27.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65340996f0 con 0x7f653413aa20 2023-10-30T12:08:27.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f6530ff9640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65140392f0 0x7f651403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:27.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f6530ff9640 1 -- 172.21.15.50:0/1109105311 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6524+0+0 (secure 0 0 0) 0x7f652c04a9f0 con 0x7f653413aa20 2023-10-30T12:08:27.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f6530ff9640 10 client.114191.objecter ms_dispatch 0x7f6534000b90 osd_map(267..267 src has 1..267) v4 2023-10-30T12:08:27.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f6530ff9640 3 client.114191.objecter handle_osd_map got epochs [267,267] > 0 2023-10-30T12:08:27.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.956+0000 7f6530ff9640 3 client.114191.objecter handle_osd_map decoding full epoch 267 2023-10-30T12:08:27.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.957+0000 7f6530ff9640 20 client.114191.objecter dump_active .. 0 homeless 2023-10-30T12:08:27.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.957+0000 7f6533fff640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65140392f0 0x7f651403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:27.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.957+0000 7f6533fff640 10 monclient: get_auth_request con 0x7f65140392f0 auth_method 0 2023-10-30T12:08:27.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.957+0000 7f6530ff9640 10 client.114191.objecter ms_handle_connect 0x7f65140392f0 2023-10-30T12:08:28.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.957+0000 7f6533fff640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65140392f0 0x7f651403b7b0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7f651c0010e0 tx=0x7f651c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:28.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.961+0000 7f6530ff9640 1 -- 172.21.15.50:0/1109105311 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f652c003e40 con 0x7f653413aa20 2023-10-30T12:08:28.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.961+0000 7f6530ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:28.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:27.961+0000 7f6530ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:28.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.206+0000 7f653a661640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:28.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.206+0000 7f653a661640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:28.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.206+0000 7f653a661640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:28.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.206+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f653404e160 con 0x7f653413aa20 2023-10-30T12:08:28.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.206+0000 7f6530ff9640 1 -- 172.21.15.50:0/1109105311 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f652c011380 con 0x7f653413aa20 2023-10-30T12:08:28.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.206+0000 7f6530ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:28.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.206+0000 7f6530ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:28.209 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":74,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":26,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:28.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 10 librados: watch_flush enter 2023-10-30T12:08:28.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 10 librados: watch_flush exit 2023-10-30T12:08:28.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 20 client.114191.objecter shutdown clearing up homeless session... 2023-10-30T12:08:28.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 10 client.114191.objecter successfully canceled tick 2023-10-30T12:08:28.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65140392f0 msgr2=0x7f651403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:28.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65140392f0 0x7f651403b7b0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7f651c0010e0 tx=0x7f651c009040 comp rx=0 tx=0).stop 2023-10-30T12:08:28.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 10 monclient: shutdown 2023-10-30T12:08:28.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:28.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f653413aa20 msgr2=0x7f6534137cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:28.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f653413aa20 0x7f6534137cf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f652c00c690 tx=0x7f652c00cb50 comp rx=0 tx=0).stop 2023-10-30T12:08:28.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 shutdown_connections 2023-10-30T12:08:28.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.208+0000 7f653a661640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65140392f0 0x7f651403b7b0 unknown :-1 s=CLOSED pgs=4041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:28.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.209+0000 7f653a661640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f653413aa20 0x7f6534137cf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:28.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.209+0000 7f653a661640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6534135ec0 0x7f6534096740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:28.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.209+0000 7f653a661640 1 --2- 172.21.15.50:0/1109105311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6534135ae0 0x7f6534096200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:28.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.209+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 >> 172.21.15.50:0/1109105311 conn(0x7f65340a05c0 msgr2=0x7f653409dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:28.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.209+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 shutdown_connections 2023-10-30T12:08:28.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.209+0000 7f653a661640 1 -- 172.21.15.50:0/1109105311 wait complete. 2023-10-30T12:08:28.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:28.209+0000 7f653a661640 1 librados: shutdown 2023-10-30T12:08:29.201 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:7462ddf6Aio.RoundTripAppendPP (2891 ms) 2023-10-30T12:08:29.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-10-30T12:08:29.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3008 ms) 2023-10-30T12:08:29.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-10-30T12:08:29.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3027 ms) 2023-10-30T12:08:29.202 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-10-30T12:08:29.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2967 ms) 2023-10-30T12:08:29.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-10-30T12:08:29.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: setting pool EIO 2023-10-30T12:08:29.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: max_success 99, min_failed 100 2023-10-30T12:08:29.203 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (19482 ms) 2023-10-30T12:08:29.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-10-30T12:08:29.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (17043 ms) 2023-10-30T12:08:29.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (220782 ms total) 2023-10-30T12:08:29.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: 2023-10-30T12:08:29.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-10-30T12:08:29.204 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-10-30T12:08:29.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3010 ms) 2023-10-30T12:08:29.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-10-30T12:08:29.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8346 ms) 2023-10-30T12:08:29.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-10-30T12:08:29.205 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14986 ms) 2023-10-30T12:08:29.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-10-30T12:08:29.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3003 ms) 2023-10-30T12:08:29.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29346 ms total) 2023-10-30T12:08:29.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: 2023-10-30T12:08:29.206 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-10-30T12:08:29.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-10-30T12:08:29.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (17110 ms) 2023-10-30T12:08:29.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (17110 ms total) 2023-10-30T12:08:29.207 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: 2023-10-30T12:08:29.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-10-30T12:08:29.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-10-30T12:08:29.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13748 ms) 2023-10-30T12:08:29.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-10-30T12:08:29.208 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7023 ms) 2023-10-30T12:08:29.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-10-30T12:08:29.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (13194 ms) 2023-10-30T12:08:29.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-10-30T12:08:29.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (22064 ms) 2023-10-30T12:08:29.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-10-30T12:08:29.209 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2262 ms) 2023-10-30T12:08:29.210 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-10-30T12:08:29.210 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7025 ms) 2023-10-30T12:08:29.210 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-10-30T12:08:29.210 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6999 ms) 2023-10-30T12:08:29.210 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-10-30T12:08:29.211 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7061 ms) 2023-10-30T12:08:29.211 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-10-30T12:08:29.211 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (21131 ms) 2023-10-30T12:08:29.211 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-10-30T12:08:29.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6257 ms) 2023-10-30T12:08:29.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-10-30T12:08:29.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6996 ms) 2023-10-30T12:08:29.212 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-10-30T12:08:29.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7033 ms) 2023-10-30T12:08:29.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-10-30T12:08:29.213 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6979 ms) 2023-10-30T12:08:31.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.339+0000 7fde3ea39640 10 monclient: get_monmap_and_config 2023-10-30T12:08:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.339+0000 7fde3ea39640 10 monclient: build_initial_monmap 2023-10-30T12:08:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.339+0000 7fde3ea39640 10 monclient: monmap: 2023-10-30T12:08:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:31.340151+0000 2023-10-30T12:08:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:31.340151+0000 2023-10-30T12:08:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:31.343 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:31.343 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.339+0000 7fde3ea39640 1 Processor -- start 2023-10-30T12:08:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.339+0000 7fde3ea39640 1 -- start start 2023-10-30T12:08:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.339+0000 7fde3ea39640 10 monclient: init 2023-10-30T12:08:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:08:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 0x7fde381326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): picked mon.noname-a con 0x7fde380a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38099a40 0x7fde38132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): picked mon.noname-b con 0x7fde38099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fde380a2350 0x7fde380a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): picked mon.noname-c con 0x7fde380a2350 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde380a3d70 con 0x7fde380a4b60 2023-10-30T12:08:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde38133120 con 0x7fde380a2350 2023-10-30T12:08:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde380a4f40 con 0x7fde38099a40 2023-10-30T12:08:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3ea39640 10 monclient(hunting): authenticate will time out at 1356.523458s 2023-10-30T12:08:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 0x7fde381326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38099a40 0x7fde38132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38099a40 0x7fde38132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37872/0 (socket says 172.21.15.50:37872) 2023-10-30T12:08:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 0x7fde381326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60250/0 (socket says 172.21.15.50:60250) 2023-10-30T12:08:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 -- 172.21.15.50:0/1456768569 learned_addr learned my addr 172.21.15.50:0/1456768569 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): get_auth_request con 0x7fde38099a40 auth_method 0 2023-10-30T12:08:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 10 monclient(hunting): get_auth_request con 0x7fde380a4b60 auth_method 0 2023-10-30T12:08:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 1 -- 172.21.15.50:0/1456768569 >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 msgr2=0x7fde381326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 1 -- 172.21.15.50:0/1456768569 >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 msgr2=0x7fde381326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 1 --2- 172.21.15.50:0/1456768569 >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 0x7fde381326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3cfd6640 1 --2- 172.21.15.50:0/1456768569 >> v2:172.21.15.118:3300/0 conn(0x7fde380a2350 0x7fde380a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde37fff640 1 --2- 172.21.15.50:0/1456768569 >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 0x7fde381326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3cfd6640 10 monclient(hunting): get_auth_request con 0x7fde380a2350 auth_method 0 2023-10-30T12:08:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3cfd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3cfd6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde3cfd6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient(hunting): handle_auth_done global_id 114200 payload 274 2023-10-30T12:08:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 -- 172.21.15.50:0/1456768569 >> v2:172.21.15.118:3300/0 conn(0x7fde380a2350 msgr2=0x7fde380a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 --2- 172.21.15.50:0/1456768569 >> v2:172.21.15.118:3300/0 conn(0x7fde380a2350 0x7fde380a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 -- 172.21.15.50:0/1456768569 >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 msgr2=0x7fde381326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 --2- 172.21.15.50:0/1456768569 >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 0x7fde381326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient: _finish_hunting 0 2023-10-30T12:08:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 monclient: found mon.noname-b 2023-10-30T12:08:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.340+0000 7fde377fe640 1 -- 172.21.15.50:0/1456768569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde380a4150 con 0x7fde38099a40 2023-10-30T12:08:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde377fe640 1 --2- 172.21.15.50:0/1456768569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38099a40 0x7fde38132be0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde380a53f0 tx=0x7fde2c02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=665c8da2733cc9b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 1 -- 172.21.15.50:0/1456768569 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c031750 con 0x7fde38099a40 2023-10-30T12:08:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: dump: 2023-10-30T12:08:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:31.366 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde3ea39640 5 monclient: authenticate success, global_id 114200 2023-10-30T12:08:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 1 -- 172.21.15.50:0/1456768569 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2c0318b0 con 0x7fde38099a40 2023-10-30T12:08:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde3ea39640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde3ea39640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde3ea39640 4 set_mon_vals no callback set 2023-10-30T12:08:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 1 -- 172.21.15.50:0/1456768569 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c031c40 con 0x7fde38099a40 2023-10-30T12:08:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.341+0000 7fde36ffd640 10 monclient: dump: 2023-10-30T12:08:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:31.371 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 10 monclient: shutdown 2023-10-30T12:08:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 -- 172.21.15.50:0/1456768569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38099a40 msgr2=0x7fde38132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 --2- 172.21.15.50:0/1456768569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38099a40 0x7fde38132be0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde380a53f0 tx=0x7fde2c02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 -- 172.21.15.50:0/1456768569 shutdown_connections 2023-10-30T12:08:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 --2- 172.21.15.50:0/1456768569 >> v2:172.21.15.118:3300/0 conn(0x7fde380a2350 0x7fde380a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 --2- 172.21.15.50:0/1456768569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38099a40 0x7fde38132be0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 --2- 172.21.15.50:0/1456768569 >> v2:172.21.15.50:3300/0 conn(0x7fde380a4b60 0x7fde381326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 -- 172.21.15.50:0/1456768569 >> 172.21.15.50:0/1456768569 conn(0x7fde380a8060 msgr2=0x7fde380a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 -- 172.21.15.50:0/1456768569 shutdown_connections 2023-10-30T12:08:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.342+0000 7fde3ea39640 1 -- 172.21.15.50:0/1456768569 wait complete. 2023-10-30T12:08:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 10 monclient: build_initial_monmap 2023-10-30T12:08:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 10 monclient: monmap: 2023-10-30T12:08:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:31.374 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:31.374 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:31.374 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:31.376 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:31.376 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:31.376 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:31.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 1 librados: starting msgr at 2023-10-30T12:08:31.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 1 librados: starting objecter 2023-10-30T12:08:31.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 1 Processor -- start 2023-10-30T12:08:31.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 1 -- start start 2023-10-30T12:08:31.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 1 librados: setting wanted keys 2023-10-30T12:08:31.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 1 librados: calling monclient init 2023-10-30T12:08:31.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.343+0000 7fde3ea39640 10 monclient: init 2023-10-30T12:08:31.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:31.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:08:31.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde380a2350 0x7fde381ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:31.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): picked mon.noname-b con 0x7fde380a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:31.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 0x7fde381caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:31.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): picked mon.noname-a con 0x7fde380a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:31.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38121770 0x7fde38121c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:31.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): picked mon.noname-c con 0x7fde38121770 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:31.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:31.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde38124e20 con 0x7fde380a4b60 2023-10-30T12:08:31.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:31.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde38123fa0 con 0x7fde380a2350 2023-10-30T12:08:31.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:31.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde38125ae0 con 0x7fde38121770 2023-10-30T12:08:31.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:31.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient(hunting): authenticate will time out at 1356.527087s 2023-10-30T12:08:31.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 0x7fde381caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:31.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38121770 0x7fde38121c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:31.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 0x7fde381caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60264/0 (socket says 172.21.15.50:60264) 2023-10-30T12:08:31.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38121770 0x7fde38121c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37884/0 (socket says 172.21.15.50:37884) 2023-10-30T12:08:31.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 1 -- 172.21.15.50:0/3026256639 learned_addr learned my addr 172.21.15.50:0/3026256639 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:31.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 10 monclient(hunting): get_auth_request con 0x7fde380a4b60 auth_method 0 2023-10-30T12:08:31.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:31.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:31.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:31.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): get_auth_request con 0x7fde38121770 auth_method 0 2023-10-30T12:08:31.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:31.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:31.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:31.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde380a2350 0x7fde381ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:31.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 10 monclient(hunting): get_auth_request con 0x7fde380a2350 auth_method 0 2023-10-30T12:08:31.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:31.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:31.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:31.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:31.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:31.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 1 -- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 msgr2=0x7fde381caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:08:31.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 1 -- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 msgr2=0x7fde381caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:08:31.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:31.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 0x7fde381caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:31.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde377fe640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 0x7fde381caac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:31.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:31.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:31.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient(hunting): handle_auth_done global_id 114203 payload 824 2023-10-30T12:08:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 -- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde380a2350 msgr2=0x7fde381ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde380a2350 0x7fde381ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 -- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 msgr2=0x7fde381caac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 0x7fde381caac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient: _finish_hunting 0 2023-10-30T12:08:31.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 monclient: found mon.noname-c 2023-10-30T12:08:31.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:31.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:31.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 -- 172.21.15.50:0/3026256639 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0094d0 con 0x7fde38121770 2023-10-30T12:08:31.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3cfd6640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38121770 0x7fde38121c20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde30007d90 tx=0x7fde3000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:31.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 client.?.objecter ms_handle_connect 0x7fde38121770 2023-10-30T12:08:31.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:31.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 1 -- 172.21.15.50:0/3026256639 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30018070 con 0x7fde38121770 2023-10-30T12:08:31.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:31.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:31.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: dump: 2023-10-30T12:08:31.401 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:31.401 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:31.401 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:31.402 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:31.402 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:31.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:31.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:31.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:31.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: _finish_auth 0 2023-10-30T12:08:31.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:08:31.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:31.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 1 -- 172.21.15.50:0/3026256639 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3000edd0 con 0x7fde38121770 2023-10-30T12:08:31.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 5 monclient: authenticate success, global_id 114203 2023-10-30T12:08:31.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient: _renew_subs 2023-10-30T12:08:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde380a18d0 con 0x7fde38121770 2023-10-30T12:08:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 1 -- 172.21.15.50:0/3026256639 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30003bb0 con 0x7fde38121770 2023-10-30T12:08:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:31.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:31.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde34ff9640 10 monclient: dump: 2023-10-30T12:08:31.406 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:31.406 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:31.407 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.407 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:31.407 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:31.407 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:31.407 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:31.408 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:31.408 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:31.408 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:31.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde367fc640 4 set_mon_vals no callback set 2023-10-30T12:08:31.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 client.114203.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:31.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.344+0000 7fde3ea39640 10 monclient: _renew_subs 2023-10-30T12:08:31.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde3ea39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:31.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde381256c0 con 0x7fde38121770 2023-10-30T12:08:31.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde3ea39640 1 librados: init done 2023-10-30T12:08:31.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde34ff9640 1 -- 172.21.15.50:0/3026256639 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fde30014420 con 0x7fde38121770 2023-10-30T12:08:31.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde3ea39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:31.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde3ea39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:31.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde3ea39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:31.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.345+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde38124f60 con 0x7fde38121770 2023-10-30T12:08:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde34ff9640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde240392f0 0x7fde2403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde34ff9640 1 -- 172.21.15.50:0/3026256639 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 6538+0+0 (secure 0 0 0) 0x7fde3004afc0 con 0x7fde38121770 2023-10-30T12:08:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde34ff9640 10 client.114203.objecter ms_dispatch 0x7fde38000b90 osd_map(271..271 src has 1..271) v4 2023-10-30T12:08:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde34ff9640 3 client.114203.objecter handle_osd_map got epochs [271,271] > 0 2023-10-30T12:08:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde34ff9640 3 client.114203.objecter handle_osd_map decoding full epoch 271 2023-10-30T12:08:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde34ff9640 20 client.114203.objecter dump_active .. 0 homeless 2023-10-30T12:08:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde37fff640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde240392f0 0x7fde2403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde37fff640 10 monclient: get_auth_request con 0x7fde240392f0 auth_method 0 2023-10-30T12:08:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde34ff9640 10 client.114203.objecter ms_handle_connect 0x7fde240392f0 2023-10-30T12:08:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.346+0000 7fde37fff640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde240392f0 0x7fde2403b7b0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7fde28006fd0 tx=0x7fde28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:31.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.350+0000 7fde34ff9640 1 -- 172.21.15.50:0/3026256639 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde3007dca0 con 0x7fde38121770 2023-10-30T12:08:31.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.350+0000 7fde34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:31.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.350+0000 7fde34ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.591+0000 7fde3ea39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.591+0000 7fde3ea39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.591+0000 7fde3ea39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.591+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fde3804e160 con 0x7fde38121770 2023-10-30T12:08:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.592+0000 7fde34ff9640 1 -- 172.21.15.50:0/3026256639 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fde38124f60 con 0x7fde38121770 2023-10-30T12:08:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.592+0000 7fde34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.592+0000 7fde34ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:31.595 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":74,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":29,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:31.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 10 librados: watch_flush enter 2023-10-30T12:08:31.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 10 librados: watch_flush exit 2023-10-30T12:08:31.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 20 client.114203.objecter shutdown clearing up homeless session... 2023-10-30T12:08:31.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 10 client.114203.objecter successfully canceled tick 2023-10-30T12:08:31.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde240392f0 msgr2=0x7fde2403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:31.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde240392f0 0x7fde2403b7b0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7fde28006fd0 tx=0x7fde28008040 comp rx=0 tx=0).stop 2023-10-30T12:08:31.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 10 monclient: shutdown 2023-10-30T12:08:31.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:31.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38121770 msgr2=0x7fde38121c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:31.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38121770 0x7fde38121c20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde30007d90 tx=0x7fde3000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 shutdown_connections 2023-10-30T12:08:31.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fde240392f0 0x7fde2403b7b0 unknown :-1 s=CLOSED pgs=4043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38121770 0x7fde38121c20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde380a4b60 0x7fde381caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 --2- 172.21.15.50:0/3026256639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde380a2350 0x7fde381ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 >> 172.21.15.50:0/3026256639 conn(0x7fde380a8060 msgr2=0x7fde380a5750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 shutdown_connections 2023-10-30T12:08:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 -- 172.21.15.50:0/3026256639 wait complete. 2023-10-30T12:08:31.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:31.594+0000 7fde3ea39640 1 librados: shutdown 2023-10-30T12:08:34.241 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWri:::163:head 2023-10-30T12:08:34.241 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:5d165639:::164:head 2023-10-30T12:08:34.242 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f43765fc:::165:head 2023-10-30T12:08:34.242 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b4c720e9:::166:head 2023-10-30T12:08:34.242 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:e694b040:::167:head 2023-10-30T12:08:34.242 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:afa38db2:::168:head 2023-10-30T12:08:34.243 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:77ba9f53:::169:head 2023-10-30T12:08:34.243 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:87495034:::170:head 2023-10-30T12:08:34.243 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:7c96bf0e:::171:head 2023-10-30T12:08:34.243 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:dbe346cc:::172:head 2023-10-30T12:08:34.243 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:e943ec24:::173:head 2023-10-30T12:08:34.243 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f97a9c0c:::174:head 2023-10-30T12:08:34.244 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:6f26e74d:::175:head 2023-10-30T12:08:34.244 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:4f95e106:::176:head 2023-10-30T12:08:34.244 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:0e6f2f8f:::177:head 2023-10-30T12:08:34.244 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:05db05f1:::178:head 2023-10-30T12:08:34.244 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:38a78d66:::179:head 2023-10-30T12:08:34.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:d095610b:::180:head 2023-10-30T12:08:34.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:a1a9d709:::181:head 2023-10-30T12:08:34.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:1e5d39db:::182:head 2023-10-30T12:08:34.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:f7df4fb9:::183:head 2023-10-30T12:08:34.245 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:03a7f161:::184:head 2023-10-30T12:08:34.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:ba70721e:::185:head 2023-10-30T12:08:34.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:28e5662d:::186:head 2023-10-30T12:08:34.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:973d52de:::187:head 2023-10-30T12:08:34.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:4303eb1c:::188:head 2023-10-30T12:08:34.246 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:b990b48e:::189:head 2023-10-30T12:08:34.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:29b8165b:::190:head 2023-10-30T12:08:34.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:3547f197:::191:head 2023-10-30T12:08:34.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:7e260936:::192:head 2023-10-30T12:08:34.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:1abec7b1:::193:head 2023-10-30T12:08:34.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:10fdda93:::194:head 2023-10-30T12:08:34.247 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:15817eea:::195:head 2023-10-30T12:08:34.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:770bab57:::196:head 2023-10-30T12:08:34.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:ed9e13e7:::197:head 2023-10-30T12:08:34.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:71471a8f:::198:head 2023-10-30T12:08:34.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: checking for 187:10fb1d02:::199:head 2023-10-30T12:08:34.248 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8003 ms) 2023-10-30T12:08:34.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-10-30T12:08:34.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,0 2023-10-30T12:08:34.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: first is 1698667660 2023-10-30T12:08:34.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,0 2023-10-30T12:08:34.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,0 2023-10-30T12:08:34.249 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,0 2023-10-30T12:08:34.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,0 2023-10-30T12:08:34.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,0 2023-10-30T12:08:34.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,1698667665,1698667666,0 2023-10-30T12:08:34.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,1698667665,1698667666,0 2023-10-30T12:08:34.250 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,1698667665,1698667666,0 2023-10-30T12:08:34.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,1698667665,1698667666,1698667668,1698667669,0 2023-10-30T12:08:34.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,1698667665,1698667666,1698667668,1698667669,0 2023-10-30T12:08:34.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667660,1698667662,1698667663,1698667665,1698667666,1698667668,1698667669,0 2023-10-30T12:08:34.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698667663,1698667665,1698667666,1698667668,1698667669,1698667671,1698667672,0 2023-10-30T12:08:34.251 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: first now 1698667663, trimmed 2023-10-30T12:08:34.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (30206 ms) 2023-10-30T12:08:34.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-10-30T12:08:34.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: foo0 2023-10-30T12:08:34.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T12:08:34.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13271 ms) 2023-10-30T12:08:34.252 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-10-30T12:08:34.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:34.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.721+0000 7fee53a39640 10 monclient: get_monmap_and_config 2023-10-30T12:08:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.721+0000 7fee53a39640 10 monclient: build_initial_monmap 2023-10-30T12:08:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.721+0000 7fee53a39640 10 monclient: monmap: 2023-10-30T12:08:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:34.722320+0000 2023-10-30T12:08:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:34.722320+0000 2023-10-30T12:08:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:34.725 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:34.725 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:34.725 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:34.725 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:34.725 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:34.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.721+0000 7fee53a39640 1 Processor -- start 2023-10-30T12:08:34.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 1 -- start start 2023-10-30T12:08:34.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient: init 2023-10-30T12:08:34.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:34.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-10-30T12:08:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135d20 0x7fee4c136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient(hunting): picked mon.noname-b con 0x7fee4c135d20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee4c135670 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 0x7fee4c13d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:34.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient(hunting): picked mon.noname-a con 0x7fee4c070aa0 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c09d4e0 con 0x7fee4c135670 2023-10-30T12:08:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c071620 con 0x7fee4c070aa0 2023-10-30T12:08:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0a01a0 con 0x7fee4c135d20 2023-10-30T12:08:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee53a39640 10 monclient(hunting): authenticate will time out at 1359.905724s 2023-10-30T12:08:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee51fd6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 0x7fee4c13d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee517d5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135d20 0x7fee4c136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee51fd6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 0x7fee4c13d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60278/0 (socket says 172.21.15.50:60278) 2023-10-30T12:08:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee517d5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135d20 0x7fee4c136100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37888/0 (socket says 172.21.15.50:37888) 2023-10-30T12:08:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee51fd6640 1 -- 172.21.15.50:0/2783502266 learned_addr learned my addr 172.21.15.50:0/2783502266 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee50fd4640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fee4c135670 0x7fee4c070390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53334/0 (socket says 172.21.15.50:53334) 2023-10-30T12:08:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee51fd6640 10 monclient(hunting): get_auth_request con 0x7fee4c070aa0 auth_method 0 2023-10-30T12:08:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee51fd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee51fd6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee51fd6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee517d5640 10 monclient(hunting): get_auth_request con 0x7fee4c135d20 auth_method 0 2023-10-30T12:08:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee517d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee517d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.722+0000 7fee517d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee51fd6640 1 -- 172.21.15.50:0/2783502266 >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 msgr2=0x7fee4c13d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:08:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee51fd6640 1 -- 172.21.15.50:0/2783502266 >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 msgr2=0x7fee4c13d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee51fd6640 1 --2- 172.21.15.50:0/2783502266 >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 0x7fee4c13d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee51fd6640 1 --2- 172.21.15.50:0/2783502266 >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 0x7fee4c13d9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 10 monclient(hunting): handle_auth_done global_id 114212 payload 274 2023-10-30T12:08:34.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 1 -- 172.21.15.50:0/2783502266 >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 msgr2=0x7fee4c13d9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 1 --2- 172.21.15.50:0/2783502266 >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 0x7fee4c13d9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 1 -- 172.21.15.50:0/2783502266 >> v1:172.21.15.50:6789/0 conn(0x7fee4c135670 legacy=0x7fee4c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 10 monclient: _finish_hunting 0 2023-10-30T12:08:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 1 monclient: found mon.noname-b 2023-10-30T12:08:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:34.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 1 -- 172.21.15.50:0/2783502266 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c099f60 con 0x7fee4c135d20 2023-10-30T12:08:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.723+0000 7fee517d5640 1 --2- 172.21.15.50:0/2783502266 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135d20 0x7fee4c136100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee480097a0 tx=0x7fee4802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad17be992f6821f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 1 -- 172.21.15.50:0/2783502266 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4802f050 con 0x7fee4c135d20 2023-10-30T12:08:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:34.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: dump: 2023-10-30T12:08:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.743 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:34.744 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:34.744 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: _finish_auth 0 2023-10-30T12:08:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: _check_auth_tickets 2023-10-30T12:08:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 1 -- 172.21.15.50:0/2783502266 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee48031700 con 0x7fee4c135d20 2023-10-30T12:08:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 5 monclient: authenticate success, global_id 114212 2023-10-30T12:08:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 1 -- 172.21.15.50:0/2783502266 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48031a90 con 0x7fee4c135d20 2023-10-30T12:08:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 4 set_mon_vals no callback set 2023-10-30T12:08:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:34.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee43fff640 10 monclient: dump: 2023-10-30T12:08:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.751 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:34.752 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:34.752 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 10 monclient: shutdown 2023-10-30T12:08:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 -- 172.21.15.50:0/2783502266 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135d20 msgr2=0x7fee4c136100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 --2- 172.21.15.50:0/2783502266 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135d20 0x7fee4c136100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee480097a0 tx=0x7fee4802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 -- 172.21.15.50:0/2783502266 shutdown_connections 2023-10-30T12:08:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 --2- 172.21.15.50:0/2783502266 >> v2:172.21.15.50:3300/0 conn(0x7fee4c070aa0 0x7fee4c13d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 --2- 172.21.15.50:0/2783502266 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135d20 0x7fee4c136100 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 -- 172.21.15.50:0/2783502266 >> 172.21.15.50:0/2783502266 conn(0x7fee4c0a4670 msgr2=0x7fee4c0a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 -- 172.21.15.50:0/2783502266 shutdown_connections 2023-10-30T12:08:34.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.724+0000 7fee53a39640 1 -- 172.21.15.50:0/2783502266 wait complete. 2023-10-30T12:08:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 10 monclient: build_initial_monmap 2023-10-30T12:08:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 10 monclient: monmap: 2023-10-30T12:08:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:34.756 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:34.757 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 1 librados: starting msgr at 2023-10-30T12:08:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 1 librados: starting objecter 2023-10-30T12:08:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 1 Processor -- start 2023-10-30T12:08:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 1 -- start start 2023-10-30T12:08:34.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 1 librados: setting wanted keys 2023-10-30T12:08:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 1 librados: calling monclient init 2023-10-30T12:08:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.725+0000 7fee53a39640 10 monclient: init 2023-10-30T12:08:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:08:34.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee4c070aa0 0x7fee4c1ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): picked mon.noname-b con 0x7fee4c070aa0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135670 0x7fee4c1320e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): picked mon.noname-c con 0x7fee4c135670 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 0x7fee4c132620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): picked mon.noname-a con 0x7fee4c135d20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c09d720 con 0x7fee4c135d20 2023-10-30T12:08:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c135080 con 0x7fee4c070aa0 2023-10-30T12:08:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c123240 con 0x7fee4c135670 2023-10-30T12:08:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee53a39640 10 monclient(hunting): authenticate will time out at 1359.909363s 2023-10-30T12:08:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135670 0x7fee4c1320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 0x7fee4c132620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:34.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135670 0x7fee4c1320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37892/0 (socket says 172.21.15.50:37892) 2023-10-30T12:08:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 -- 172.21.15.50:0/2094831344 learned_addr learned my addr 172.21.15.50:0/2094831344 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): get_auth_request con 0x7fee4c135670 auth_method 0 2023-10-30T12:08:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 10 monclient(hunting): get_auth_request con 0x7fee4c135d20 auth_method 0 2023-10-30T12:08:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:34.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 1 -- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 msgr2=0x7fee4c132620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:08:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 1 -- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 msgr2=0x7fee4c132620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:08:34.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 0x7fee4c132620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee51fd6640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 0x7fee4c132620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee517d5640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee4c070aa0 0x7fee4c1ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient(hunting): handle_auth_done global_id 114215 payload 824 2023-10-30T12:08:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 -- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee4c070aa0 msgr2=0x7fee4c1ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:34.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee4c070aa0 0x7fee4c1ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 -- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 msgr2=0x7fee4c132620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 0x7fee4c132620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient: _finish_hunting 0 2023-10-30T12:08:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 monclient: found mon.noname-c 2023-10-30T12:08:34.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 -- 172.21.15.50:0/2094831344 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee48009400 con 0x7fee4c135670 2023-10-30T12:08:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee50fd4640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135670 0x7fee4c1320e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00c4d0 tx=0x7fee3c00c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.726+0000 7fee41ffb640 10 client.?.objecter ms_handle_connect 0x7fee4c135670 2023-10-30T12:08:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:34.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 1 -- 172.21.15.50:0/2094831344 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c007480 con 0x7fee4c135670 2023-10-30T12:08:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: dump: 2023-10-30T12:08:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:34.769 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:34.770 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: _finish_auth 0 2023-10-30T12:08:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:08:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 1 -- 172.21.15.50:0/2094831344 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3c0075e0 con 0x7fee4c135670 2023-10-30T12:08:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 5 monclient: authenticate success, global_id 114215 2023-10-30T12:08:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 10 monclient: _renew_subs 2023-10-30T12:08:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:34.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c123520 con 0x7fee4c135670 2023-10-30T12:08:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 1 -- 172.21.15.50:0/2094831344 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c0049e0 con 0x7fee4c135670 2023-10-30T12:08:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee41ffb640 10 monclient: dump: 2023-10-30T12:08:34.772 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:34.773 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:34.774 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:34.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 10 client.114215.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee437fe640 4 set_mon_vals no callback set 2023-10-30T12:08:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 10 monclient: _renew_subs 2023-10-30T12:08:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c1238b0 con 0x7fee4c135670 2023-10-30T12:08:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.727+0000 7fee53a39640 1 librados: init done 2023-10-30T12:08:34.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee41ffb640 1 -- 172.21.15.50:0/2094831344 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fee3c004b40 con 0x7fee4c135670 2023-10-30T12:08:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee41ffb640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee300392f0 0x7fee3003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee53a39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee53a39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee53a39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:34.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c123a40 con 0x7fee4c135670 2023-10-30T12:08:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee41ffb640 1 -- 172.21.15.50:0/2094831344 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fee3c04a1f0 con 0x7fee4c135670 2023-10-30T12:08:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee41ffb640 10 client.114215.objecter ms_dispatch 0x7fee4c000b90 osd_map(274..274 src has 1..274) v4 2023-10-30T12:08:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee41ffb640 3 client.114215.objecter handle_osd_map got epochs [274,274] > 0 2023-10-30T12:08:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee41ffb640 3 client.114215.objecter handle_osd_map decoding full epoch 274 2023-10-30T12:08:34.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee41ffb640 20 client.114215.objecter dump_active .. 0 homeless 2023-10-30T12:08:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee517d5640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee300392f0 0x7fee3003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.728+0000 7fee517d5640 10 monclient: get_auth_request con 0x7fee300392f0 auth_method 0 2023-10-30T12:08:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.729+0000 7fee517d5640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee300392f0 0x7fee3003b7b0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7fee48035000 tx=0x7fee48008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:34.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.729+0000 7fee41ffb640 10 client.114215.objecter ms_handle_connect 0x7fee300392f0 2023-10-30T12:08:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.733+0000 7fee41ffb640 1 -- 172.21.15.50:0/2094831344 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fee4c123a40 con 0x7fee4c135670 2023-10-30T12:08:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.733+0000 7fee41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:34.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.733+0000 7fee41ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:34.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.978+0000 7fee53a39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:34.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.978+0000 7fee53a39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:34.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.978+0000 7fee53a39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:34.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.978+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fee4c123a40 con 0x7fee4c135670 2023-10-30T12:08:34.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.978+0000 7fee41ffb640 1 -- 172.21.15.50:0/2094831344 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fee4c123a40 con 0x7fee4c135670 2023-10-30T12:08:34.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.978+0000 7fee41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:34.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.978+0000 7fee41ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:34.982 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":74,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":33,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 10 librados: watch_flush enter 2023-10-30T12:08:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 10 librados: watch_flush exit 2023-10-30T12:08:34.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 20 client.114215.objecter shutdown clearing up homeless session... 2023-10-30T12:08:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 10 client.114215.objecter successfully canceled tick 2023-10-30T12:08:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee300392f0 msgr2=0x7fee3003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee300392f0 0x7fee3003b7b0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7fee48035000 tx=0x7fee48008030 comp rx=0 tx=0).stop 2023-10-30T12:08:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 10 monclient: shutdown 2023-10-30T12:08:34.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135670 msgr2=0x7fee4c1320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135670 0x7fee4c1320e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00c4d0 tx=0x7fee3c00c990 comp rx=0 tx=0).stop 2023-10-30T12:08:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 shutdown_connections 2023-10-30T12:08:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fee300392f0 0x7fee3003b7b0 unknown :-1 s=CLOSED pgs=4045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee4c135d20 0x7fee4c132620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee4c135670 0x7fee4c1320e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 --2- 172.21.15.50:0/2094831344 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee4c070aa0 0x7fee4c1ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.980+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 >> 172.21.15.50:0/2094831344 conn(0x7fee4c0a4670 msgr2=0x7fee4c133830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.981+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 shutdown_connections 2023-10-30T12:08:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.981+0000 7fee53a39640 1 -- 172.21.15.50:0/2094831344 wait complete. 2023-10-30T12:08:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:34.981+0000 7fee53a39640 1 librados: shutdown 2023-10-30T12:08:37.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:38.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.108+0000 7f4e5bc1a640 10 monclient: get_monmap_and_config 2023-10-30T12:08:38.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.108+0000 7f4e5bc1a640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.108+0000 7f4e5bc1a640 10 monclient: monmap: 2023-10-30T12:08:38.111 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.111 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.111 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:38.109949+0000 2023-10-30T12:08:38.111 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:38.109949+0000 2023-10-30T12:08:38.112 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.112 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.112 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:38.112 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:38.112 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:38.112 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:38.113 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:38.113 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.109+0000 7f4e5bc1a640 1 Processor -- start 2023-10-30T12:08:38.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.109+0000 7f4e5bc1a640 1 -- start start 2023-10-30T12:08:38.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.109+0000 7f4e5bc1a640 10 monclient: init 2023-10-30T12:08:38.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T12:08:38.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4e54098520 0x7f4e5409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): picked mon.noname-c con 0x7f4e54098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:38.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e54098900 0x7f4e54098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): picked mon.noname-a con 0x7f4e54098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:38.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e54132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): picked mon.noname-b con 0x7f4e54132740 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5409ad30 con 0x7f4e54098900 2023-10-30T12:08:38.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5409c460 con 0x7f4e54098520 2023-10-30T12:08:38.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5409b260 con 0x7f4e54132740 2023-10-30T12:08:38.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5bc1a640 10 monclient(hunting): authenticate will time out at 1363.293305s 2023-10-30T12:08:38.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e54098900 0x7f4e54098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e54132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e54098900 0x7f4e54098d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50132/0 (socket says 172.21.15.50:50132) 2023-10-30T12:08:38.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 1 -- 172.21.15.50:0/2543375591 learned_addr learned my addr 172.21.15.50:0/2543375591 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): get_auth_request con 0x7f4e54132740 auth_method 0 2023-10-30T12:08:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 10 monclient(hunting): get_auth_request con 0x7f4e54098900 auth_method 0 2023-10-30T12:08:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e599b6640 1 --2- 172.21.15.50:0/2543375591 >> v2:172.21.15.118:3300/0 conn(0x7f4e54098520 0x7f4e5409bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e599b6640 10 monclient(hunting): get_auth_request con 0x7f4e54098520 auth_method 0 2023-10-30T12:08:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e599b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e599b6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e599b6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_done global_id 114218 payload 274 2023-10-30T12:08:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 1 -- 172.21.15.50:0/2543375591 >> v2:172.21.15.118:3300/0 conn(0x7f4e54098520 msgr2=0x7f4e5409bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 1 --2- 172.21.15.50:0/2543375591 >> v2:172.21.15.118:3300/0 conn(0x7f4e54098520 0x7f4e5409bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 1 -- 172.21.15.50:0/2543375591 >> v2:172.21.15.50:3300/0 conn(0x7f4e54098900 msgr2=0x7f4e54098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 1 --2- 172.21.15.50:0/2543375591 >> v2:172.21.15.50:3300/0 conn(0x7f4e54098900 0x7f4e54098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 1 monclient: found mon.noname-b 2023-10-30T12:08:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:38.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e5a1b7640 1 -- 172.21.15.50:0/2543375591 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e54072120 con 0x7f4e54132740 2023-10-30T12:08:38.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.110+0000 7f4e591b5640 1 --2- 172.21.15.50:0/2543375591 >> v2:172.21.15.50:3300/0 conn(0x7f4e54098900 0x7f4e54098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:38.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e5a1b7640 1 --2- 172.21.15.50:0/2543375591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e54132bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e54073e60 tx=0x7f4e4c02bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1edbb571595be89d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 1 -- 172.21.15.50:0/2543375591 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c02cab0 con 0x7f4e54132740 2023-10-30T12:08:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: dump: 2023-10-30T12:08:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.134 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e5bc1a640 5 monclient: authenticate success, global_id 114218 2023-10-30T12:08:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 1 -- 172.21.15.50:0/2543375591 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c02cc10 con 0x7f4e54132740 2023-10-30T12:08:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e5bc1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e5bc1a640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 1 -- 172.21.15.50:0/2543375591 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c034440 con 0x7f4e54132740 2023-10-30T12:08:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e5bc1a640 4 set_mon_vals no callback set 2023-10-30T12:08:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.111+0000 7f4e589b4640 10 monclient: dump: 2023-10-30T12:08:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.138 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.138 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.138 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.138 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 10 monclient: shutdown 2023-10-30T12:08:38.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/2543375591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 msgr2=0x7f4e54132bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/2543375591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e54132bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e54073e60 tx=0x7f4e4c02bd00 comp rx=0 tx=0).stop 2023-10-30T12:08:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/2543375591 shutdown_connections 2023-10-30T12:08:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/2543375591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e54132bb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/2543375591 >> v2:172.21.15.50:3300/0 conn(0x7f4e54098900 0x7f4e54098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/2543375591 >> v2:172.21.15.118:3300/0 conn(0x7f4e54098520 0x7f4e5409bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/2543375591 >> 172.21.15.50:0/2543375591 conn(0x7f4e540a05c0 msgr2=0x7f4e5409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:38.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/2543375591 shutdown_connections 2023-10-30T12:08:38.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.112+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/2543375591 wait complete. 2023-10-30T12:08:38.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient: monmap: 2023-10-30T12:08:38.141 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.141 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:38.141 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:38.142 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:38.143 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 librados: starting msgr at 2023-10-30T12:08:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 librados: starting objecter 2023-10-30T12:08:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 Processor -- start 2023-10-30T12:08:38.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 -- start start 2023-10-30T12:08:38.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 librados: setting wanted keys 2023-10-30T12:08:38.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 librados: calling monclient init 2023-10-30T12:08:38.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient: init 2023-10-30T12:08:38.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:08:38.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54098520 0x7f4e541318e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): picked mon.noname-b con 0x7f4e54098520 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:38.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e54098900 0x7f4e54131e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): picked mon.noname-a con 0x7f4e54098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:38.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e5406bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): picked mon.noname-c con 0x7f4e54132740 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e541cad80 con 0x7f4e54098900 2023-10-30T12:08:38.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54071df0 con 0x7f4e54098520 2023-10-30T12:08:38.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540a3070 con 0x7f4e54132740 2023-10-30T12:08:38.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5bc1a640 10 monclient(hunting): authenticate will time out at 1363.296881s 2023-10-30T12:08:38.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e5a1b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e5406bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.113+0000 7f4e591b5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e54098900 0x7f4e54131e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e5406bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36170/0 (socket says 172.21.15.50:36170) 2023-10-30T12:08:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 -- 172.21.15.50:0/1216459764 learned_addr learned my addr 172.21.15.50:0/1216459764 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): get_auth_request con 0x7f4e54132740 auth_method 0 2023-10-30T12:08:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 10 monclient(hunting): get_auth_request con 0x7f4e54098900 auth_method 0 2023-10-30T12:08:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54098520 0x7f4e541318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 10 monclient(hunting): get_auth_request con 0x7f4e54098520 auth_method 0 2023-10-30T12:08:38.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e599b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient(hunting): handle_auth_done global_id 114221 payload 824 2023-10-30T12:08:38.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 -- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54098520 msgr2=0x7f4e541318e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54098520 0x7f4e541318e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 -- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e54098900 msgr2=0x7f4e54131e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e54098900 0x7f4e54131e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 monclient: found mon.noname-c 2023-10-30T12:08:38.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:38.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 -- 172.21.15.50:0/1216459764 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480094d0 con 0x7f4e54132740 2023-10-30T12:08:38.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e591b5640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e54098900 0x7f4e54131e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:38.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5a1b7640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e5406bb20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009e50 tx=0x7f4e4c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 client.?.objecter ms_handle_connect 0x7f4e54132740 2023-10-30T12:08:38.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:38.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 1 -- 172.21.15.50:0/1216459764 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c040650 con 0x7f4e54132740 2023-10-30T12:08:38.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:38.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: dump: 2023-10-30T12:08:38.156 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.156 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.156 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.156 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.156 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.156 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.157 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.157 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.157 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.157 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 1 -- 172.21.15.50:0/1216459764 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c0407b0 con 0x7f4e54132740 2023-10-30T12:08:38.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 5 monclient: authenticate success, global_id 114221 2023-10-30T12:08:38.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 10 monclient: _renew_subs 2023-10-30T12:08:38.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:38.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c009a70 con 0x7f4e54132740 2023-10-30T12:08:38.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 1 -- 172.21.15.50:0/1216459764 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c040b60 con 0x7f4e54132740 2023-10-30T12:08:38.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:38.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e467fc640 10 monclient: dump: 2023-10-30T12:08:38.160 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.160 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.160 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.160 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.161 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.161 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.161 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.161 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.161 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.161 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 10 client.114221.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:38.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 10 monclient: _renew_subs 2023-10-30T12:08:38.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e47fff640 4 set_mon_vals no callback set 2023-10-30T12:08:38.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:38.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5412d0a0 con 0x7f4e54132740 2023-10-30T12:08:38.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.114+0000 7f4e5bc1a640 1 librados: init done 2023-10-30T12:08:38.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.115+0000 7f4e467fc640 1 -- 172.21.15.50:0/1216459764 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4e4c040cc0 con 0x7f4e54132740 2023-10-30T12:08:38.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.115+0000 7f4e5bc1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.115+0000 7f4e5bc1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.115+0000 7f4e5bc1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:38.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.115+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5404e160 con 0x7f4e54132740 2023-10-30T12:08:38.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.115+0000 7f4e467fc640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e380392f0 0x7f4e3803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e467fc640 1 -- 172.21.15.50:0/1216459764 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 6508+0+0 (secure 0 0 0) 0x7f4e4c06db60 con 0x7f4e54132740 2023-10-30T12:08:38.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e467fc640 10 client.114221.objecter ms_dispatch 0x7f4e54000b90 osd_map(277..277 src has 1..277) v4 2023-10-30T12:08:38.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e467fc640 3 client.114221.objecter handle_osd_map got epochs [277,277] > 0 2023-10-30T12:08:38.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e467fc640 3 client.114221.objecter handle_osd_map decoding full epoch 277 2023-10-30T12:08:38.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e467fc640 20 client.114221.objecter dump_active .. 0 homeless 2023-10-30T12:08:38.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e599b6640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e380392f0 0x7f4e3803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e599b6640 10 monclient: get_auth_request con 0x7f4e380392f0 auth_method 0 2023-10-30T12:08:38.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e467fc640 10 client.114221.objecter ms_handle_connect 0x7f4e380392f0 2023-10-30T12:08:38.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.116+0000 7f4e599b6640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e380392f0 0x7f4e3803b7b0 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 crypto rx=0x7f4e50005f80 tx=0x7f4e50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.120+0000 7f4e467fc640 1 -- 172.21.15.50:0/1216459764 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4e4c048360 con 0x7f4e54132740 2023-10-30T12:08:38.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.120+0000 7f4e467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.120+0000 7f4e467fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:38.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.368+0000 7f4e5bc1a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:38.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.368+0000 7f4e5bc1a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:38.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.368+0000 7f4e5bc1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:38.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.368+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4e5412cb30 con 0x7f4e54132740 2023-10-30T12:08:38.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.368+0000 7f4e467fc640 1 -- 172.21.15.50:0/1216459764 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f4e4c06be60 con 0x7f4e54132740 2023-10-30T12:08:38.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.369+0000 7f4e467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:38.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.369+0000 7f4e467fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:38.372 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 10 librados: watch_flush enter 2023-10-30T12:08:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 10 librados: watch_flush exit 2023-10-30T12:08:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 20 client.114221.objecter shutdown clearing up homeless session... 2023-10-30T12:08:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 10 client.114221.objecter successfully canceled tick 2023-10-30T12:08:38.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e380392f0 msgr2=0x7f4e3803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e380392f0 0x7f4e3803b7b0 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 crypto rx=0x7f4e50005f80 tx=0x7f4e50008040 comp rx=0 tx=0).stop 2023-10-30T12:08:38.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 10 monclient: shutdown 2023-10-30T12:08:38.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:38.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 msgr2=0x7f4e5406bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e5406bb20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009e50 tx=0x7f4e4c004a40 comp rx=0 tx=0).stop 2023-10-30T12:08:38.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 shutdown_connections 2023-10-30T12:08:38.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e380392f0 0x7f4e3803b7b0 unknown :-1 s=CLOSED pgs=4046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e54132740 0x7f4e5406bb20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e54098900 0x7f4e54131e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 --2- 172.21.15.50:0/1216459764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54098520 0x7f4e541318e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 >> 172.21.15.50:0/1216459764 conn(0x7f4e540a05c0 msgr2=0x7f4e54097410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:38.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 shutdown_connections 2023-10-30T12:08:38.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.370+0000 7f4e5bc1a640 1 -- 172.21.15.50:0/1216459764 wait complete. 2023-10-30T12:08:38.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.371+0000 7f4e5bc1a640 1 librados: shutdown 2023-10-30T12:08:38.382 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:08:38.382 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:08:38.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:08:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.489+0000 7f33bb810640 10 monclient: get_monmap_and_config 2023-10-30T12:08:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.489+0000 7f33bb810640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.489+0000 7f33bb810640 10 monclient: monmap: 2023-10-30T12:08:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:38.490178+0000 2023-10-30T12:08:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:38.490178+0000 2023-10-30T12:08:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:38.493 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.489+0000 7f33bb810640 1 Processor -- start 2023-10-30T12:08:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.489+0000 7f33bb810640 1 -- start start 2023-10-30T12:08:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.489+0000 7f33bb810640 10 monclient: init 2023-10-30T12:08:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-10-30T12:08:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f33b40a2ae0 0x7f33b40a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): picked mon.noname-a con 0x7f33b40a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f33b40a2ec0 0x7f33b41356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): picked mon.noname-c con 0x7f33b40a2ec0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b4135c20 0x7f33b4136070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): picked mon.noname-b con 0x7f33b4135c20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b41365b0 con 0x7f33b40a2ae0 2023-10-30T12:08:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b40a2990 con 0x7f33b40a2ec0 2023-10-30T12:08:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b40a1630 con 0x7f33b4135c20 2023-10-30T12:08:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33bb810640 10 monclient(hunting): authenticate will time out at 1363.673374s 2023-10-30T12:08:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f33b40a2ae0 0x7f33b40a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b4135c20 0x7f33b4136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f33b40a2ae0 0x7f33b40a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50146/0 (socket says 172.21.15.50:50146) 2023-10-30T12:08:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 -- 172.21.15.50:0/3849653736 learned_addr learned my addr 172.21.15.50:0/3849653736 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): get_auth_request con 0x7f33b40a2ae0 auth_method 0 2023-10-30T12:08:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 10 monclient(hunting): get_auth_request con 0x7f33b4135c20 auth_method 0 2023-10-30T12:08:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b8dab640 1 --2- 172.21.15.50:0/3849653736 >> v2:172.21.15.118:3300/0 conn(0x7f33b40a2ec0 0x7f33b41356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b8dab640 10 monclient(hunting): get_auth_request con 0x7f33b40a2ec0 auth_method 0 2023-10-30T12:08:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b8dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b8dab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b8dab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient(hunting): handle_auth_done global_id 124104 payload 274 2023-10-30T12:08:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 -- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b4135c20 msgr2=0x7f33b4136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 --2- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b4135c20 0x7f33b4136070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 -- 172.21.15.50:0/3849653736 >> v2:172.21.15.118:3300/0 conn(0x7f33b40a2ec0 msgr2=0x7f33b41356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 --2- 172.21.15.50:0/3849653736 >> v2:172.21.15.118:3300/0 conn(0x7f33b40a2ec0 0x7f33b41356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 monclient: found mon.noname-a 2023-10-30T12:08:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b95ac640 1 -- 172.21.15.50:0/3849653736 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b40a4190 con 0x7f33b40a2ae0 2023-10-30T12:08:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.490+0000 7f33b9dad640 1 --2- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b4135c20 0x7f33b4136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33b95ac640 1 --2- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ae0 0x7f33b40a2450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f33b40a4c80 tx=0x7f33b002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362bc3738dab54c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 1 -- 172.21.15.50:0/3849653736 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b0031750 con 0x7f33b40a2ae0 2023-10-30T12:08:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: dump: 2023-10-30T12:08:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.514 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 1 -- 172.21.15.50:0/3849653736 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b00318b0 con 0x7f33b40a2ae0 2023-10-30T12:08:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33bb810640 5 monclient: authenticate success, global_id 124104 2023-10-30T12:08:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33bb810640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 1 -- 172.21.15.50:0/3849653736 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b0031c40 con 0x7f33b40a2ae0 2023-10-30T12:08:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33bb810640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33bb810640 4 set_mon_vals no callback set 2023-10-30T12:08:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.491+0000 7f33abfff640 10 monclient: dump: 2023-10-30T12:08:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.518 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 10 monclient: shutdown 2023-10-30T12:08:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 -- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ae0 msgr2=0x7f33b40a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 --2- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ae0 0x7f33b40a2450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f33b40a4c80 tx=0x7f33b002b7f0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 -- 172.21.15.50:0/3849653736 shutdown_connections 2023-10-30T12:08:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 --2- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b4135c20 0x7f33b4136070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 --2- 172.21.15.50:0/3849653736 >> v2:172.21.15.118:3300/0 conn(0x7f33b40a2ec0 0x7f33b41356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 --2- 172.21.15.50:0/3849653736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ae0 0x7f33b40a2450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 -- 172.21.15.50:0/3849653736 >> 172.21.15.50:0/3849653736 conn(0x7f33b40a80b0 msgr2=0x7f33b40a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 -- 172.21.15.50:0/3849653736 shutdown_connections 2023-10-30T12:08:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.492+0000 7f33bb810640 1 -- 172.21.15.50:0/3849653736 wait complete. 2023-10-30T12:08:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient: monmap: 2023-10-30T12:08:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:38.523 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 librados: starting msgr at 2023-10-30T12:08:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 librados: starting objecter 2023-10-30T12:08:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 Processor -- start 2023-10-30T12:08:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 -- start start 2023-10-30T12:08:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 librados: setting wanted keys 2023-10-30T12:08:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 librados: calling monclient init 2023-10-30T12:08:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient: init 2023-10-30T12:08:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:08:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b40a2ae0 0x7f33b411e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): picked mon.noname-c con 0x7f33b40a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ec0 0x7f33b411e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): picked mon.noname-a con 0x7f33b40a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4135c20 0x7f33b411eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): picked mon.noname-b con 0x7f33b4135c20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b4120cb0 con 0x7f33b40a2ec0 2023-10-30T12:08:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b4121350 con 0x7f33b4135c20 2023-10-30T12:08:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b411fe30 con 0x7f33b40a2ae0 2023-10-30T12:08:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33b95ac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b40a2ae0 0x7f33b411e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33bb810640 10 monclient(hunting): authenticate will time out at 1363.676970s 2023-10-30T12:08:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33b95ac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b40a2ae0 0x7f33b411e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36178/0 (socket says 172.21.15.50:36178) 2023-10-30T12:08:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.493+0000 7f33b95ac640 1 -- 172.21.15.50:0/944695741 learned_addr learned my addr 172.21.15.50:0/944695741 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 10 monclient(hunting): get_auth_request con 0x7f33b40a2ae0 auth_method 0 2023-10-30T12:08:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ec0 0x7f33b411e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): get_auth_request con 0x7f33b40a2ec0 auth_method 0 2023-10-30T12:08:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4135c20 0x7f33b411eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 10 monclient(hunting): get_auth_request con 0x7f33b4135c20 auth_method 0 2023-10-30T12:08:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient(hunting): handle_auth_done global_id 124107 payload 824 2023-10-30T12:08:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b40a2ae0 msgr2=0x7f33b411e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b40a2ae0 0x7f33b411e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4135c20 msgr2=0x7f33b411eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33b4135c20 0x7f33b411eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 monclient: found mon.noname-a 2023-10-30T12:08:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 -- 172.21.15.50:0/944695741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac009a70 con 0x7f33b40a2ec0 2023-10-30T12:08:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b95ac640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f33b40a2ae0 0x7f33b411e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33b8dab640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ec0 0x7f33b411e5e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f33a400d620 tx=0x7f33a400da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 client.?.objecter ms_handle_connect 0x7f33b40a2ec0 2023-10-30T12:08:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a40125f0 con 0x7f33b40a2ec0 2023-10-30T12:08:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: dump: 2023-10-30T12:08:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.536 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.537 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.538 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a4012750 con 0x7f33b40a2ec0 2023-10-30T12:08:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4012ae0 con 0x7f33b40a2ec0 2023-10-30T12:08:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 5 monclient: authenticate success, global_id 124107 2023-10-30T12:08:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 10 monclient: _renew_subs 2023-10-30T12:08:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b00094d0 con 0x7f33b40a2ec0 2023-10-30T12:08:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33ab7fe640 4 set_mon_vals no callback set 2023-10-30T12:08:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33a9ffb640 10 monclient: dump: 2023-10-30T12:08:38.540 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.541 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.542 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 10 client.124107.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:38.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 10 monclient: _renew_subs 2023-10-30T12:08:38.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b4121550 con 0x7f33b40a2ec0 2023-10-30T12:08:38.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.494+0000 7f33bb810640 1 librados: init done 2023-10-30T12:08:38.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f33a4012c40 con 0x7f33b40a2ec0 2023-10-30T12:08:38.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33a9ffb640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 0x7f339803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33bb810640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f33a404af10 con 0x7f33b40a2ec0 2023-10-30T12:08:38.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33a9ffb640 10 client.124107.objecter ms_dispatch 0x7f33b4000b90 osd_map(278..278 src has 1..278) v4 2023-10-30T12:08:38.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33a9ffb640 3 client.124107.objecter handle_osd_map got epochs [278,278] > 0 2023-10-30T12:08:38.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33a9ffb640 3 client.124107.objecter handle_osd_map decoding full epoch 278 2023-10-30T12:08:38.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33bb810640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134270 0x7f33b4134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b4073ff0 con 0x7f33b4134270 2023-10-30T12:08:38.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.495+0000 7f33a9ffb640 20 client.124107.objecter dump_active .. 0 homeless 2023-10-30T12:08:38.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134270 0x7f33b4134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient: get_auth_request con 0x7f33b4134270 auth_method 0 2023-10-30T12:08:38.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b95ac640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 0x7f339803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b95ac640 10 monclient: get_auth_request con 0x7f3398039320 auth_method 0 2023-10-30T12:08:38.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 10 monclient(hunting): handle_auth_done global_id 124110 payload 824 2023-10-30T12:08:38.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 1 -- 172.21.15.50:0/944695741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33b409a8a0 con 0x7f33b4134270 2023-10-30T12:08:38.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 10 client.124107.objecter ms_handle_connect 0x7f33b4134270 2023-10-30T12:08:38.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 10 client.124107.objecter resend_mon_ops 2023-10-30T12:08:38.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b9dad640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134270 0x7f33b4134650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f33b41204e0 tx=0x7f33ac00f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b95ac640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 msgr2=0x7f339803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:08:38.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b95ac640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 msgr2=0x7f339803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:08:38.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b95ac640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 0x7f339803b7e0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f33b002b830 tx=0x7f33b0008030 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:38.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33b95ac640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 0x7f339803b7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:38.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac00e040 con 0x7f33b4134270 2023-10-30T12:08:38.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 10 monclient: dump: 2023-10-30T12:08:38.550 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.550 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.550 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.550 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.551 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.551 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.551 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.551 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.551 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.551 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f33ac00f040 con 0x7f33b4134270 2023-10-30T12:08:38.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:38.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134270 msgr2=0x7f33b4134650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134270 0x7f33b4134650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f33b41204e0 tx=0x7f33ac00f870 comp rx=0 tx=0).stop 2023-10-30T12:08:38.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134270 msgr2=0x7f33b4134650 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:38.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.496+0000 7f33a9ffb640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134270 0x7f33b4134650 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f33b41204e0 tx=0x7f33ac00f870 comp rx=0 tx=0).stop 2023-10-30T12:08:38.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33bb810640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:38.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33bb810640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134b90 0x7f33b41d4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33bb810640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b409a5c0 con 0x7f33b4134b90 2023-10-30T12:08:38.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134b90 0x7f33b41d4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient: get_auth_request con 0x7f33b4134b90 auth_method 0 2023-10-30T12:08:38.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 10 monclient(hunting): handle_auth_done global_id 124113 payload 824 2023-10-30T12:08:38.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 1 -- 172.21.15.50:0/944695741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f33b4099b20 con 0x7f33b4134b90 2023-10-30T12:08:38.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 10 client.124107.objecter ms_handle_connect 0x7f33b4134b90 2023-10-30T12:08:38.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33b9dad640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134b90 0x7f33b41d4540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33ac01b000 tx=0x7f33ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 10 client.124107.objecter resend_mon_ops 2023-10-30T12:08:38.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac02b070 con 0x7f33b4134b90 2023-10-30T12:08:38.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 10 monclient: dump: 2023-10-30T12:08:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.560 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.560 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.560 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.560 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.560 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.561 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.561 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f33ac023040 con 0x7f33b4134b90 2023-10-30T12:08:38.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:38.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134b90 msgr2=0x7f33b41d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134b90 0x7f33b41d4540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33ac01b000 tx=0x7f33ac004a40 comp rx=0 tx=0).stop 2023-10-30T12:08:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134b90 msgr2=0x7f33b41d4540 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.503+0000 7f33a9ffb640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b4134b90 0x7f33b41d4540 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33ac01b000 tx=0x7f33ac004a40 comp rx=0 tx=0).stop 2023-10-30T12:08:38.563 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:38.563 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:38.563 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:38.563 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:08:38.563 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 78, 2023-10-30T12:08:38.564 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:38.564 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:08:38.564 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:38.564 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:38.564 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.564 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:08:38.565 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:38.565 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:38.565 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:38.565 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:38.565 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:38.566 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:38.566 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:38.566 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:38.566 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:38.566 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:38.567 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:38.567 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:38.567 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:38.567 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.567 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:38.568 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:38.568 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:38.568 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:38.568 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:38.568 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:38.569 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:38.569 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:38.569 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:38.569 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:38.569 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:38.569 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:38.570 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.570 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:38.570 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:38.570 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:38.571 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:38.571 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:38.571 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:38.571 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:38.571 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:38.572 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:38.572 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:38.572 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:38.572 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:38.572 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:38.572 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:38.573 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:38.573 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:38.573 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:38.573 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:38.573 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:38.574 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:38.574 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:38.574 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:38.574 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:38.574 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:38.575 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:38.575 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:38.575 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:38.575 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.575 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:38.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.576 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:38.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.576 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:38.576 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:38.577 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:38.577 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:38.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.577 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:38.577 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:38.577 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.578 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:38.578 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:38.578 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.579 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.579 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.579 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:38.579 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.580 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:38.581 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:38.581 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:38.581 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:38.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.581 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:38.582 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:38.582 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.582 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:38.583 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:38.583 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.583 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.583 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.584 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:38.584 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:38.584 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:38.584 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:38.584 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:38.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.585 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:38.585 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:38.585 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:38.585 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:38.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.586 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:38.586 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:38.586 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:08:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:38.593 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:38.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 10 librados: watch_flush enter 2023-10-30T12:08:38.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 10 librados: watch_flush exit 2023-10-30T12:08:38.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 20 client.124107.objecter shutdown clearing up homeless session... 2023-10-30T12:08:38.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 10 client.124107.objecter successfully canceled tick 2023-10-30T12:08:38.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 msgr2=0x7f339803b7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:38.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3398039320 0x7f339803b7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 10 monclient: shutdown 2023-10-30T12:08:38.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:38.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ec0 msgr2=0x7f33b411e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33b9dad640 1 -- 172.21.15.50:0/944695741 reap_dead start 2023-10-30T12:08:38.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ec0 0x7f33b411e5e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f33a400d620 tx=0x7f33a400da20 comp rx=0 tx=0).stop 2023-10-30T12:08:38.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 shutdown_connections 2023-10-30T12:08:38.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 --2- 172.21.15.50:0/944695741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33b40a2ec0 0x7f33b411e5e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 >> 172.21.15.50:0/944695741 conn(0x7f33b40a80b0 msgr2=0x7f33b4131ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:38.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 shutdown_connections 2023-10-30T12:08:38.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 -- 172.21.15.50:0/944695741 wait complete. 2023-10-30T12:08:38.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.504+0000 7f33bb810640 1 librados: shutdown 2023-10-30T12:08:38.598 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:08:38.598 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:08:38.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:08:38.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.703+0000 7f46f47b0640 10 monclient: get_monmap_and_config 2023-10-30T12:08:38.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.703+0000 7f46f47b0640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.703+0000 7f46f47b0640 10 monclient: monmap: 2023-10-30T12:08:38.705 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.705 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:38.704442+0000 2023-10-30T12:08:38.706 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:38.704442+0000 2023-10-30T12:08:38.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.706 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.706 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:38.706 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:38.707 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:38.707 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:38.707 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:38.709 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.703+0000 7f46f47b0640 1 Processor -- start 2023-10-30T12:08:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 1 -- start start 2023-10-30T12:08:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient: init 2023-10-30T12:08:38.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-10-30T12:08:38.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46ec096540 0x7f46ec096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): picked mon.noname-a con 0x7f46ec096540 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:38.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f46ec096e60 0x7f46ec09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): picked mon.noname-c con 0x7f46ec096e60 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec09afa0 0x7f46ec09b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): picked mon.noname-b con 0x7f46ec09afa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec09bd30 con 0x7f46ec096540 2023-10-30T12:08:38.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec09c400 con 0x7f46ec096e60 2023-10-30T12:08:38.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec09c120 con 0x7f46ec09afa0 2023-10-30T12:08:38.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f47b0640 10 monclient(hunting): authenticate will time out at 1363.887596s 2023-10-30T12:08:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46ec096540 0x7f46ec096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec09afa0 0x7f46ec09b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46ec096540 0x7f46ec096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50180/0 (socket says 172.21.15.50:50180) 2023-10-30T12:08:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec09afa0 0x7f46ec09b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36182/0 (socket says 172.21.15.50:36182) 2023-10-30T12:08:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 1 -- 172.21.15.50:0/479649689 learned_addr learned my addr 172.21.15.50:0/479649689 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 10 monclient(hunting): get_auth_request con 0x7f46ec096540 auth_method 0 2023-10-30T12:08:38.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 10 monclient(hunting): get_auth_request con 0x7f46ec09afa0 auth_method 0 2023-10-30T12:08:38.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f2d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f1d4b640 1 --2- 172.21.15.50:0/479649689 >> v2:172.21.15.118:3300/0 conn(0x7f46ec096e60 0x7f46ec09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f1d4b640 10 monclient(hunting): get_auth_request con 0x7f46ec096e60 auth_method 0 2023-10-30T12:08:38.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f1d4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f1d4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.704+0000 7f46f1d4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 10 monclient(hunting): handle_auth_done global_id 124116 payload 274 2023-10-30T12:08:38.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 1 -- 172.21.15.50:0/479649689 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec09afa0 msgr2=0x7f46ec09b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:38.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 1 --2- 172.21.15.50:0/479649689 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec09afa0 0x7f46ec09b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 1 -- 172.21.15.50:0/479649689 >> v2:172.21.15.118:3300/0 conn(0x7f46ec096e60 msgr2=0x7f46ec09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:38.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 1 --2- 172.21.15.50:0/479649689 >> v2:172.21.15.118:3300/0 conn(0x7f46ec096e60 0x7f46ec09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 1 monclient: found mon.noname-a 2023-10-30T12:08:38.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 1 -- 172.21.15.50:0/479649689 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec09f600 con 0x7f46ec096540 2023-10-30T12:08:38.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f1d4b640 1 --2- 172.21.15.50:0/479649689 >> v2:172.21.15.118:3300/0 conn(0x7f46ec096e60 0x7f46ec09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:08:38.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f254c640 1 --2- 172.21.15.50:0/479649689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec096540 0x7f46ec096920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46ec074290 tx=0x7f46e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4393fb8a42b3f02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 1 -- 172.21.15.50:0/479649689 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8004230 con 0x7f46ec096540 2023-10-30T12:08:38.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:38.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: dump: 2023-10-30T12:08:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.734 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 1 -- 172.21.15.50:0/479649689 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e8004390 con 0x7f46ec096540 2023-10-30T12:08:38.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 1 -- 172.21.15.50:0/479649689 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8003810 con 0x7f46ec096540 2023-10-30T12:08:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f47b0640 5 monclient: authenticate success, global_id 124116 2023-10-30T12:08:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f47b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f47b0640 4 set_mon_vals no callback set 2023-10-30T12:08:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.705+0000 7f46f154a640 10 monclient: dump: 2023-10-30T12:08:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.738 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 10 monclient: shutdown 2023-10-30T12:08:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 -- 172.21.15.50:0/479649689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec096540 msgr2=0x7f46ec096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 --2- 172.21.15.50:0/479649689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec096540 0x7f46ec096920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46ec074290 tx=0x7f46e802b8f0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 -- 172.21.15.50:0/479649689 shutdown_connections 2023-10-30T12:08:38.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 --2- 172.21.15.50:0/479649689 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec09afa0 0x7f46ec09b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 --2- 172.21.15.50:0/479649689 >> v2:172.21.15.118:3300/0 conn(0x7f46ec096e60 0x7f46ec09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 --2- 172.21.15.50:0/479649689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec096540 0x7f46ec096920 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 -- 172.21.15.50:0/479649689 >> 172.21.15.50:0/479649689 conn(0x7f46ec0a3980 msgr2=0x7f46ec0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:38.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 -- 172.21.15.50:0/479649689 shutdown_connections 2023-10-30T12:08:38.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.706+0000 7f46f47b0640 1 -- 172.21.15.50:0/479649689 wait complete. 2023-10-30T12:08:38.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 10 monclient: monmap: 2023-10-30T12:08:38.741 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.741 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:38.741 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:38.741 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:38.742 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:38.743 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 1 librados: starting msgr at 2023-10-30T12:08:38.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 1 librados: starting objecter 2023-10-30T12:08:38.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 1 Processor -- start 2023-10-30T12:08:38.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 1 -- start start 2023-10-30T12:08:38.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 1 librados: setting wanted keys 2023-10-30T12:08:38.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 1 librados: calling monclient init 2023-10-30T12:08:38.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 10 monclient: init 2023-10-30T12:08:38.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.707+0000 7f46f47b0640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:08:38.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46ec096540 0x7f46ec070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): picked mon.noname-b con 0x7f46ec096540 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:38.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec096e60 0x7f46ec1ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): picked mon.noname-c con 0x7f46ec096e60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec09afa0 0x7f46ec093670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): picked mon.noname-a con 0x7f46ec09afa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec08f980 con 0x7f46ec09afa0 2023-10-30T12:08:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec08fb80 con 0x7f46ec096540 2023-10-30T12:08:38.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec08fe60 con 0x7f46ec096e60 2023-10-30T12:08:38.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient(hunting): authenticate will time out at 1363.891085s 2023-10-30T12:08:38.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec096e60 0x7f46ec1ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec09afa0 0x7f46ec093670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec096e60 0x7f46ec1ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36188/0 (socket says 172.21.15.50:36188) 2023-10-30T12:08:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 1 -- 172.21.15.50:0/2237650399 learned_addr learned my addr 172.21.15.50:0/2237650399 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): get_auth_request con 0x7f46ec09afa0 auth_method 0 2023-10-30T12:08:38.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 10 monclient(hunting): get_auth_request con 0x7f46ec096e60 auth_method 0 2023-10-30T12:08:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f254c640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46ec096540 0x7f46ec070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f254c640 10 monclient(hunting): get_auth_request con 0x7f46ec096540 auth_method 0 2023-10-30T12:08:38.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f254c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f254c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f254c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient(hunting): handle_auth_done global_id 124119 payload 824 2023-10-30T12:08:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec096e60 msgr2=0x7f46ec1ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec096e60 0x7f46ec1ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46ec096540 msgr2=0x7f46ec070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46ec096540 0x7f46ec070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 monclient: found mon.noname-a 2023-10-30T12:08:38.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 -- 172.21.15.50:0/2237650399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e8009460 con 0x7f46ec09afa0 2023-10-30T12:08:38.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f1d4b640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec096e60 0x7f46ec1ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:38.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f2d4d640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec09afa0 0x7f46ec093670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46e400efc0 tx=0x7f46e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 client.?.objecter ms_handle_connect 0x7f46ec09afa0 2023-10-30T12:08:38.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:38.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4009c00 con 0x7f46ec09afa0 2023-10-30T12:08:38.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:38.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: dump: 2023-10-30T12:08:38.755 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.756 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.756 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.756 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.756 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.756 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.757 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.757 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.757 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.757 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 5 monclient: authenticate success, global_id 124119 2023-10-30T12:08:38.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e400f040 con 0x7f46ec09afa0 2023-10-30T12:08:38.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient: _renew_subs 2023-10-30T12:08:38.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec090140 con 0x7f46ec09afa0 2023-10-30T12:08:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e40146c0 con 0x7f46ec09afa0 2023-10-30T12:08:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46e2ffd640 10 monclient: dump: 2023-10-30T12:08:38.760 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.760 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.760 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.761 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.761 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.761 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.761 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.761 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 client.124119.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f0d49640 4 set_mon_vals no callback set 2023-10-30T12:08:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient: _renew_subs 2023-10-30T12:08:38.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec093bb0 con 0x7f46ec09afa0 2023-10-30T12:08:38.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.708+0000 7f46f47b0640 1 librados: init done 2023-10-30T12:08:38.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f47b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f47b0640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097560 0x7f46ec097940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec073ff0 con 0x7f46ec097560 2023-10-30T12:08:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f46e4014820 con 0x7f46ec09afa0 2023-10-30T12:08:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097560 0x7f46ec097940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient: get_auth_request con 0x7f46ec097560 auth_method 0 2023-10-30T12:08:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46e2ffd640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46d00392f0 0x7f46d003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 10 monclient(hunting): handle_auth_done global_id 114236 payload 824 2023-10-30T12:08:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46f254c640 1 -- 172.21.15.50:0/2237650399 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46ec09b970 con 0x7f46ec097560 2023-10-30T12:08:38.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f46e404b990 con 0x7f46ec09afa0 2023-10-30T12:08:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46e2ffd640 10 client.124119.objecter ms_dispatch 0x7f46ec000b90 osd_map(278..278 src has 1..278) v4 2023-10-30T12:08:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46e2ffd640 3 client.124119.objecter handle_osd_map got epochs [278,278] > 0 2023-10-30T12:08:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.709+0000 7f46e2ffd640 3 client.124119.objecter handle_osd_map decoding full epoch 278 2023-10-30T12:08:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46f254c640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097560 0x7f46ec097940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46ec06afd0 tx=0x7f46e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 20 client.124119.objecter dump_active .. 0 homeless 2023-10-30T12:08:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 client.124119.objecter ms_handle_connect 0x7f46ec097560 2023-10-30T12:08:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 client.124119.objecter resend_mon_ops 2023-10-30T12:08:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46f1d4b640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46d00392f0 0x7f46d003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46f1d4b640 10 monclient: get_auth_request con 0x7f46d00392f0 auth_method 0 2023-10-30T12:08:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8004500 con 0x7f46ec097560 2023-10-30T12:08:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 monclient: dump: 2023-10-30T12:08:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.771 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f46e8003b70 con 0x7f46ec097560 2023-10-30T12:08:38.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:38.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097560 msgr2=0x7f46ec097940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097560 0x7f46ec097940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46ec06afd0 tx=0x7f46e80046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097560 msgr2=0x7f46ec097940 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:38.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097560 0x7f46ec097940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46e2ffd640 10 client.124119.objecter ms_handle_connect 0x7f46d00392f0 2023-10-30T12:08:38.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.710+0000 7f46f1d4b640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46d00392f0 0x7f46d003b7b0 secure :-1 s=READY pgs=4047 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0736d0 tx=0x7f46dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f47b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:38.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f47b0640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097e80 0x7f46ec06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f47b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec071910 con 0x7f46ec097e80 2023-10-30T12:08:38.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f254c640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097e80 0x7f46ec06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f254c640 10 monclient: get_auth_request con 0x7f46ec097e80 auth_method 0 2023-10-30T12:08:38.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f254c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f254c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f254c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.716+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f254c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f254c640 10 monclient(hunting): handle_auth_done global_id 114239 payload 824 2023-10-30T12:08:38.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f254c640 1 -- 172.21.15.50:0/2237650399 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46ec04e160 con 0x7f46ec097e80 2023-10-30T12:08:38.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f254c640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097e80 0x7f46ec06dbb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f46e8042000 tx=0x7f46e8031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 10 client.124119.objecter ms_handle_connect 0x7f46ec097e80 2023-10-30T12:08:38.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 10 client.124119.objecter resend_mon_ops 2023-10-30T12:08:38.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8031b80 con 0x7f46ec097e80 2023-10-30T12:08:38.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:38.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 10 monclient: dump: 2023-10-30T12:08:38.776 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.777 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.777 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.777 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.777 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.777 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.778 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.778 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.778 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.778 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f46e8006890 con 0x7f46ec097e80 2023-10-30T12:08:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097e80 msgr2=0x7f46ec06dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097e80 0x7f46ec06dbb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f46e8042000 tx=0x7f46e8031000 comp rx=0 tx=0).stop 2023-10-30T12:08:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097e80 msgr2=0x7f46ec06dbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46e2ffd640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f46ec097e80 0x7f46ec06dbb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.780 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:38.780 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:38.780 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:38.780 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:08:38.781 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 78, 2023-10-30T12:08:38.781 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:38.781 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:08:38.781 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:38.781 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:38.782 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.782 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:08:38.782 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:38.782 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:38.782 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:38.783 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:38.783 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:38.783 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:38.783 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:38.783 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:38.784 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:38.784 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:38.784 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:38.784 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:38.784 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:38.784 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.785 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:38.785 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:38.785 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:38.785 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:38.785 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:38.786 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:38.786 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:38.786 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:38.786 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:38.786 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:38.787 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:38.787 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:38.787 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.787 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:38.788 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:38.788 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:38.788 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:38.788 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:38.788 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:38.789 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:38.789 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:38.789 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:38.789 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:38.789 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:38.790 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:38.790 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:38.790 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:38.790 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:38.790 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:38.790 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:38.791 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:38.791 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:38.791 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:38.791 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:38.791 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:38.792 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:38.792 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:38.792 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:38.792 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:38.792 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:38.793 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.793 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:38.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.793 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:38.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:38.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:38.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:38.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.799 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:38.799 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:38.799 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.800 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:38.800 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:38.800 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.800 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.801 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:38.801 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:38.801 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:38.801 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:38.801 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:38.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.802 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:38.802 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:38.802 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:38.802 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:38.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.803 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:38.803 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:38.803 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.803 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:38.804 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:38.804 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:38.804 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.804 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.805 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:38.805 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:38.805 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:38.805 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:38.805 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:38.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.806 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:38.807 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:38.807 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:38.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.807 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.807 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:38.810 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:38.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:38.810 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:38.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:38.811 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:38.811 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f47b0640 10 librados: watch_flush enter 2023-10-30T12:08:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f47b0640 10 librados: watch_flush exit 2023-10-30T12:08:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f47b0640 20 client.124119.objecter shutdown clearing up homeless session... 2023-10-30T12:08:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f47b0640 10 client.124119.objecter successfully canceled tick 2023-10-30T12:08:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46d00392f0 msgr2=0x7f46d003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.717+0000 7f46f47b0640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46d00392f0 0x7f46d003b7b0 secure :-1 s=READY pgs=4047 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0736d0 tx=0x7f46dc008040 comp rx=0 tx=0).stop 2023-10-30T12:08:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 10 monclient: shutdown 2023-10-30T12:08:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec09afa0 msgr2=0x7f46ec093670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f2d4d640 1 -- 172.21.15.50:0/2237650399 reap_dead start 2023-10-30T12:08:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec09afa0 0x7f46ec093670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46e400efc0 tx=0x7f46e400c490 comp rx=0 tx=0).stop 2023-10-30T12:08:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 shutdown_connections 2023-10-30T12:08:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 --2- 172.21.15.50:0/2237650399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46ec09afa0 0x7f46ec093670 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 >> 172.21.15.50:0/2237650399 conn(0x7f46ec0a3980 msgr2=0x7f46ec1b9840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 shutdown_connections 2023-10-30T12:08:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 -- 172.21.15.50:0/2237650399 wait complete. 2023-10-30T12:08:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.718+0000 7f46f47b0640 1 librados: shutdown 2023-10-30T12:08:38.816 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:08:38.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:08:38.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:08:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.921+0000 7f8a4b1f0640 10 monclient: get_monmap_and_config 2023-10-30T12:08:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.921+0000 7f8a4b1f0640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.921+0000 7f8a4b1f0640 10 monclient: monmap: 2023-10-30T12:08:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:38.922466+0000 2023-10-30T12:08:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:38.922466+0000 2023-10-30T12:08:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:38.925 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:38.926 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 1 Processor -- start 2023-10-30T12:08:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 1 -- start start 2023-10-30T12:08:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient: init 2023-10-30T12:08:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-10-30T12:08:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a4406afc0 0x7f8a440a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4406afc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a4406b640 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4413b070 0x7f8a4413d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4413b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4406b3a0 con 0x7f8a4406b640 2023-10-30T12:08:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4409e540 con 0x7f8a4406afc0 2023-10-30T12:08:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4409fa10 con 0x7f8a4413b070 2023-10-30T12:08:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4b1f0640 10 monclient(hunting): authenticate will time out at 1364.105645s 2023-10-30T12:08:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a4406afc0 0x7f8a440a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4413b070 0x7f8a4413d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4413b070 0x7f8a4413d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36222/0 (socket says 172.21.15.50:36222) 2023-10-30T12:08:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a4406afc0 0x7f8a440a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50196/0 (socket says 172.21.15.50:50196) 2023-10-30T12:08:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 1 -- 172.21.15.50:0/3501500210 learned_addr learned my addr 172.21.15.50:0/3501500210 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a43fff640 1 --1- 172.21.15.50:0/3501500210 >> v1:172.21.15.50:6789/0 conn(0x7f8a4406b640 0x7f8a440a21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40376/0 (socket says 172.21.15.50:40376) 2023-10-30T12:08:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 10 monclient(hunting): get_auth_request con 0x7f8a4413b070 auth_method 0 2023-10-30T12:08:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 10 monclient(hunting): get_auth_request con 0x7f8a4406afc0 auth_method 0 2023-10-30T12:08:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a4978d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.922+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_done global_id 124122 payload 274 2023-10-30T12:08:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 1 -- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4413b070 msgr2=0x7f8a4413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 1 --2- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4413b070 0x7f8a4413d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 1 -- 172.21.15.50:0/3501500210 >> v1:172.21.15.50:6789/0 conn(0x7f8a4406b640 legacy=0x7f8a440a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 1 monclient: found mon.noname-a 2023-10-30T12:08:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 1 -- 172.21.15.50:0/3501500210 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4406a6f0 con 0x7f8a4406afc0 2023-10-30T12:08:38.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a4978d640 1 --2- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4413b070 0x7f8a4413d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:38.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.923+0000 7f8a48f8c640 1 --2- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406afc0 0x7f8a440a1c90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009d90 tx=0x7f8a3c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a70c6bc7a8aa5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 1 -- 172.21.15.50:0/3501500210 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c02cb30 con 0x7f8a4406afc0 2023-10-30T12:08:38.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: dump: 2023-10-30T12:08:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.947 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.947 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.947 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.947 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.948 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.948 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.948 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 1 -- 172.21.15.50:0/3501500210 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c02cc90 con 0x7f8a4406afc0 2023-10-30T12:08:38.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 5 monclient: authenticate success, global_id 124122 2023-10-30T12:08:38.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:38.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:38.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 1 -- 172.21.15.50:0/3501500210 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c034440 con 0x7f8a4406afc0 2023-10-30T12:08:38.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 4 set_mon_vals no callback set 2023-10-30T12:08:38.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a437fe640 10 monclient: dump: 2023-10-30T12:08:38.951 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.951 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.951 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.951 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.951 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.952 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.952 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.952 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.952 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.952 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 10 monclient: shutdown 2023-10-30T12:08:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406afc0 msgr2=0x7f8a440a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406afc0 0x7f8a440a1c90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009d90 tx=0x7f8a3c02bd80 comp rx=0 tx=0).stop 2023-10-30T12:08:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/3501500210 shutdown_connections 2023-10-30T12:08:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4413b070 0x7f8a4413d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/3501500210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406afc0 0x7f8a440a1c90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/3501500210 >> 172.21.15.50:0/3501500210 conn(0x7f8a440a6200 msgr2=0x7f8a440a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/3501500210 shutdown_connections 2023-10-30T12:08:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.924+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/3501500210 wait complete. 2023-10-30T12:08:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 10 monclient: build_initial_monmap 2023-10-30T12:08:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 10 monclient: monmap: 2023-10-30T12:08:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:38.957 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 1 librados: starting msgr at 2023-10-30T12:08:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 1 librados: starting objecter 2023-10-30T12:08:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 1 Processor -- start 2023-10-30T12:08:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 1 -- start start 2023-10-30T12:08:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 1 librados: setting wanted keys 2023-10-30T12:08:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 1 librados: calling monclient init 2023-10-30T12:08:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.925+0000 7f8a4b1f0640 10 monclient: init 2023-10-30T12:08:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:08:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4406afc0 0x7f8a441ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): picked mon.noname-c con 0x7f8a4406afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406b640 0x7f8a4411c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4406b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4413b070 0x7f8a4411cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4413b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441385b0 con 0x7f8a4406b640 2023-10-30T12:08:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44070de0 con 0x7f8a4413b070 2023-10-30T12:08:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4411ece0 con 0x7f8a4406afc0 2023-10-30T12:08:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient(hunting): authenticate will time out at 1364.109147s 2023-10-30T12:08:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406b640 0x7f8a4411c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4406afc0 0x7f8a441ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4406afc0 0x7f8a441ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36228/0 (socket says 172.21.15.50:36228) 2023-10-30T12:08:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 1 -- 172.21.15.50:0/1690012264 learned_addr learned my addr 172.21.15.50:0/1690012264 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406b640 0x7f8a4411c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50208/0 (socket says 172.21.15.50:50208) 2023-10-30T12:08:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 10 monclient(hunting): get_auth_request con 0x7f8a4406afc0 auth_method 0 2023-10-30T12:08:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): get_auth_request con 0x7f8a4406b640 auth_method 0 2023-10-30T12:08:38.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient(hunting): handle_auth_done global_id 124128 payload 824 2023-10-30T12:08:38.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4978d640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4413b070 0x7f8a4411cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4406afc0 msgr2=0x7f8a441ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4406afc0 0x7f8a441ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4413b070 msgr2=0x7f8a4411cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4413b070 0x7f8a4411cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:38.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient: _finish_hunting 0 2023-10-30T12:08:38.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 monclient: found mon.noname-a 2023-10-30T12:08:38.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:38.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 -- 172.21.15.50:0/1690012264 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a340094d0 con 0x7f8a4406b640 2023-10-30T12:08:38.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4978d640 10 monclient: get_auth_request con 0x7f8a4413b070 auth_method 0 2023-10-30T12:08:38.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a48f8c640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a4406afc0 0x7f8a441ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:38.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4978d640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4413b070 0x7f8a4411cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:08:38.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a43fff640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406b640 0x7f8a4411c680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800c400 tx=0x7f8a3800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 client.?.objecter ms_handle_connect 0x7f8a4406b640 2023-10-30T12:08:38.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:38.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380074e0 con 0x7f8a4406b640 2023-10-30T12:08:38.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:38.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: dump: 2023-10-30T12:08:38.969 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.969 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.970 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.970 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.970 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.970 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.970 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.971 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.971 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.971 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: _finish_auth 0 2023-10-30T12:08:38.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: _check_auth_tickets 2023-10-30T12:08:38.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:38.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a38007680 con 0x7f8a4406b640 2023-10-30T12:08:38.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 5 monclient: authenticate success, global_id 124128 2023-10-30T12:08:38.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient: _renew_subs 2023-10-30T12:08:38.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c009a70 con 0x7f8a4406b640 2023-10-30T12:08:38.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:38.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38004a20 con 0x7f8a4406b640 2023-10-30T12:08:38.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:38.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a417fa640 10 monclient: dump: 2023-10-30T12:08:38.974 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.974 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.974 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.974 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.974 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.975 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 client.124128.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient: _renew_subs 2023-10-30T12:08:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a42ffd640 4 set_mon_vals no callback set 2023-10-30T12:08:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.926+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4411ff00 con 0x7f8a4406b640 2023-10-30T12:08:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.927+0000 7f8a4b1f0640 1 librados: init done 2023-10-30T12:08:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.927+0000 7f8a4b1f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.927+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4411d100 0x7f8a4411d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.927+0000 7f8a4b1f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.927+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441201f0 con 0x7f8a4411d100 2023-10-30T12:08:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.927+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8a38004b80 con 0x7f8a4406b640 2023-10-30T12:08:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4411d100 0x7f8a4411d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a300392f0 0x7f8a3003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient: get_auth_request con 0x7f8a4411d100 auth_method 0 2023-10-30T12:08:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6253+0+0 (secure 0 0 0) 0x7f8a3804a610 con 0x7f8a4406b640 2023-10-30T12:08:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 10 client.124128.objecter ms_dispatch 0x7f8a44000b90 osd_map(278..278 src has 1..278) v4 2023-10-30T12:08:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 3 client.124128.objecter handle_osd_map got epochs [278,278] > 0 2023-10-30T12:08:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 3 client.124128.objecter handle_osd_map decoding full epoch 278 2023-10-30T12:08:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 20 client.124128.objecter dump_active .. 0 homeless 2023-10-30T12:08:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a4978d640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a300392f0 0x7f8a3003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a4978d640 10 monclient: get_auth_request con 0x7f8a300392f0 auth_method 0 2023-10-30T12:08:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_done global_id 94597 payload 824 2023-10-30T12:08:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 1 -- 172.21.15.50:0/1690012264 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a4409f0b0 con 0x7f8a4411d100 2023-10-30T12:08:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 10 client.124128.objecter ms_handle_connect 0x7f8a4411d100 2023-10-30T12:08:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 10 client.124128.objecter resend_mon_ops 2023-10-30T12:08:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a48f8c640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4411d100 0x7f8a4411d4e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c02bdc0 tx=0x7f8a3c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 10 client.124128.objecter ms_handle_connect 0x7f8a300392f0 2023-10-30T12:08:38.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a4978d640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a300392f0 0x7f8a3003b7b0 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 crypto rx=0x7f8a44134a20 tx=0x7f8a34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c032650 con 0x7f8a4411d100 2023-10-30T12:08:38.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.928+0000 7f8a417fa640 10 monclient: dump: 2023-10-30T12:08:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.984 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.929+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8a3c0327b0 con 0x7f8a4411d100 2023-10-30T12:08:38.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.929+0000 7f8a417fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:38.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.929+0000 7f8a417fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:38.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.929+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4411d100 msgr2=0x7f8a4411d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.929+0000 7f8a417fa640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4411d100 0x7f8a4411d4e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c02bdc0 tx=0x7f8a3c004a40 comp rx=0 tx=0).stop 2023-10-30T12:08:38.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.929+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4411d100 msgr2=0x7f8a4411d4e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:38.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.929+0000 7f8a417fa640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a4411d100 0x7f8a4411d4e0 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c02bdc0 tx=0x7f8a3c004a40 comp rx=0 tx=0).stop 2023-10-30T12:08:38.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a4b1f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:38.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a44135260 0x7f8a44135670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:38.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a4b1f0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:38.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a44073ff0 con 0x7f8a44135260 2023-10-30T12:08:38.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a44135260 0x7f8a44135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:38.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 10 monclient: get_auth_request con 0x7f8a44135260 auth_method 0 2023-10-30T12:08:38.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:38.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:38.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:38.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:38.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:38.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.935+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:38.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a48f8c640 10 monclient(hunting): handle_auth_done global_id 94600 payload 824 2023-10-30T12:08:38.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a48f8c640 1 -- 172.21.15.50:0/1690012264 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8a4409df80 con 0x7f8a44135260 2023-10-30T12:08:38.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a48f8c640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a44135260 0x7f8a44135670 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c045000 tx=0x7f8a3c031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:38.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 10 client.124128.objecter ms_handle_connect 0x7f8a44135260 2023-10-30T12:08:38.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 10 client.124128.objecter resend_mon_ops 2023-10-30T12:08:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c004240 con 0x7f8a44135260 2023-10-30T12:08:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 10 monclient: dump: 2023-10-30T12:08:38.991 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:38.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:38.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:38.992 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:38.992 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:38.992 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:38.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:38.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:38.993 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:38.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f8a3c0043a0 con 0x7f8a44135260 2023-10-30T12:08:38.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:38.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:38.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a44135260 msgr2=0x7f8a44135670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:38.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a44135260 0x7f8a44135670 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c045000 tx=0x7f8a3c031000 comp rx=0 tx=0).stop 2023-10-30T12:08:38.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a44135260 msgr2=0x7f8a44135670 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:38.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a417fa640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a44135260 0x7f8a44135670 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c045000 tx=0x7f8a3c031000 comp rx=0 tx=0).stop 2023-10-30T12:08:38.995 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:08:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 78, 2023-10-30T12:08:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:08:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:08:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.019 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:39.019 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:39.019 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:39.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.019 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:08:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:08:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:39.028 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:39.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 10 librados: watch_flush enter 2023-10-30T12:08:39.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 10 librados: watch_flush exit 2023-10-30T12:08:39.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 20 client.124128.objecter shutdown clearing up homeless session... 2023-10-30T12:08:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 10 client.124128.objecter successfully canceled tick 2023-10-30T12:08:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a300392f0 msgr2=0x7f8a3003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a300392f0 0x7f8a3003b7b0 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 crypto rx=0x7f8a44134a20 tx=0x7f8a34008040 comp rx=0 tx=0).stop 2023-10-30T12:08:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 10 monclient: shutdown 2023-10-30T12:08:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406b640 msgr2=0x7f8a4411c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406b640 0x7f8a4411c680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800c400 tx=0x7f8a3800c8c0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.936+0000 7f8a4978d640 1 -- 172.21.15.50:0/1690012264 reap_dead start 2023-10-30T12:08:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.937+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 shutdown_connections 2023-10-30T12:08:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.937+0000 7f8a4b1f0640 1 --2- 172.21.15.50:0/1690012264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a4406b640 0x7f8a4411c680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.937+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 >> 172.21.15.50:0/1690012264 conn(0x7f8a440a6200 msgr2=0x7f8a440a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.937+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 shutdown_connections 2023-10-30T12:08:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.937+0000 7f8a4b1f0640 1 -- 172.21.15.50:0/1690012264 wait complete. 2023-10-30T12:08:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:38.937+0000 7f8a4b1f0640 1 librados: shutdown 2023-10-30T12:08:39.034 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:08:39.034 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:08:39.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:08:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.137+0000 7fc8e38e7640 10 monclient: get_monmap_and_config 2023-10-30T12:08:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.137+0000 7fc8e38e7640 10 monclient: build_initial_monmap 2023-10-30T12:08:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.137+0000 7fc8e38e7640 10 monclient: monmap: 2023-10-30T12:08:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:39.138815+0000 2023-10-30T12:08:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:39.138815+0000 2023-10-30T12:08:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:39.141 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:39.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 Processor -- start 2023-10-30T12:08:39.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 -- start start 2023-10-30T12:08:39.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient: init 2023-10-30T12:08:39.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:39.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:08:39.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc8dc09f5b0 0x7fc8dc099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:39.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc09f5b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:39.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc096080 0x7fc8dc096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:39.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:39.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc8dc0969a0 0x7fc8dc09cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:39.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc0969a0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:39.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:39.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc098810 con 0x7fc8dc09f5b0 2023-10-30T12:08:39.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:39.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc06aef0 con 0x7fc8dc0969a0 2023-10-30T12:08:39.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:39.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc09a1a0 con 0x7fc8dc096080 2023-10-30T12:08:39.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:39.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.138+0000 7fc8e38e7640 10 monclient(hunting): authenticate will time out at 1364.321955s 2023-10-30T12:08:39.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc096080 0x7fc8dc096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:39.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc8dc09f5b0 0x7fc8dc099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:39.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc8dc09f5b0 0x7fc8dc099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50210/0 (socket says 172.21.15.50:50210) 2023-10-30T12:08:39.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc096080 0x7fc8dc096460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36236/0 (socket says 172.21.15.50:36236) 2023-10-30T12:08:39.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 -- 172.21.15.50:0/1276301526 learned_addr learned my addr 172.21.15.50:0/1276301526 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:39.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): get_auth_request con 0x7fc8dc09f5b0 auth_method 0 2023-10-30T12:08:39.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:39.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:39.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:39.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 10 monclient(hunting): get_auth_request con 0x7fc8dc096080 auth_method 0 2023-10-30T12:08:39.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:39.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:39.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:39.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:39.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:39.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:39.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:39.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:39.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:39.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1e84640 1 --2- 172.21.15.50:0/1276301526 >> v2:172.21.15.118:3300/0 conn(0x7fc8dc0969a0 0x7fc8dc09cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:39.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1e84640 10 monclient(hunting): get_auth_request con 0x7fc8dc0969a0 auth_method 0 2023-10-30T12:08:39.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:39.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1e84640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:39.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1e84640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:39.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient(hunting): handle_auth_done global_id 124131 payload 274 2023-10-30T12:08:39.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 -- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc096080 msgr2=0x7fc8dc096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 --2- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc096080 0x7fc8dc096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 -- 172.21.15.50:0/1276301526 >> v2:172.21.15.118:3300/0 conn(0x7fc8dc0969a0 msgr2=0x7fc8dc09cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 --2- 172.21.15.50:0/1276301526 >> v2:172.21.15.118:3300/0 conn(0x7fc8dc0969a0 0x7fc8dc09cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient: _finish_hunting 0 2023-10-30T12:08:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 monclient: found mon.noname-a 2023-10-30T12:08:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e1683640 1 -- 172.21.15.50:0/1276301526 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc098af0 con 0x7fc8dc09f5b0 2023-10-30T12:08:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.139+0000 7fc8e0e82640 1 --2- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc096080 0x7fc8dc096460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e1683640 1 --2- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc099b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009850 tx=0x7fc8d802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e881e3902d023841 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 1 -- 172.21.15.50:0/1276301526 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8004230 con 0x7fc8dc09f5b0 2023-10-30T12:08:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: dump: 2023-10-30T12:08:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:39.163 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: _finish_auth 0 2023-10-30T12:08:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:08:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 1 -- 172.21.15.50:0/1276301526 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d802f050 con 0x7fc8dc09f5b0 2023-10-30T12:08:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 1 -- 172.21.15.50:0/1276301526 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8003610 con 0x7fc8dc09f5b0 2023-10-30T12:08:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 5 monclient: authenticate success, global_id 124131 2023-10-30T12:08:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 4 set_mon_vals no callback set 2023-10-30T12:08:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8d3fff640 10 monclient: dump: 2023-10-30T12:08:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:39.167 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 10 monclient: shutdown 2023-10-30T12:08:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 1 -- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc09f5b0 msgr2=0x7fc8dc099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc099b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009850 tx=0x7fc8d802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 1 -- 172.21.15.50:0/1276301526 shutdown_connections 2023-10-30T12:08:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/1276301526 >> v2:172.21.15.118:3300/0 conn(0x7fc8dc0969a0 0x7fc8dc09cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc096080 0x7fc8dc096460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/1276301526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc099b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.140+0000 7fc8e38e7640 1 -- 172.21.15.50:0/1276301526 >> 172.21.15.50:0/1276301526 conn(0x7fc8dc0a05c0 msgr2=0x7fc8dc09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.141+0000 7fc8e38e7640 1 -- 172.21.15.50:0/1276301526 shutdown_connections 2023-10-30T12:08:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.141+0000 7fc8e38e7640 1 -- 172.21.15.50:0/1276301526 wait complete. 2023-10-30T12:08:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.141+0000 7fc8e38e7640 10 monclient: build_initial_monmap 2023-10-30T12:08:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.141+0000 7fc8e38e7640 10 monclient: monmap: 2023-10-30T12:08:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:39.172 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.141+0000 7fc8e38e7640 1 librados: starting msgr at 2023-10-30T12:08:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.141+0000 7fc8e38e7640 1 librados: starting objecter 2023-10-30T12:08:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.141+0000 7fc8e38e7640 1 Processor -- start 2023-10-30T12:08:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 -- start start 2023-10-30T12:08:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 librados: setting wanted keys 2023-10-30T12:08:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 librados: calling monclient init 2023-10-30T12:08:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient: init 2023-10-30T12:08:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:08:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc096080 0x7fc8dc08b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc0969a0 0x7fc8dc095350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc0969a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc095890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc09f5b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc08db90 con 0x7fc8dc096080 2023-10-30T12:08:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc08d2f0 con 0x7fc8dc09f5b0 2023-10-30T12:08:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc08d5d0 con 0x7fc8dc0969a0 2023-10-30T12:08:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e38e7640 10 monclient(hunting): authenticate will time out at 1364.325512s 2023-10-30T12:08:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc096080 0x7fc8dc08b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc0969a0 0x7fc8dc095350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc0969a0 0x7fc8dc095350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36252/0 (socket says 172.21.15.50:36252) 2023-10-30T12:08:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc096080 0x7fc8dc08b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50222/0 (socket says 172.21.15.50:50222) 2023-10-30T12:08:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 1 -- 172.21.15.50:0/2657748627 learned_addr learned my addr 172.21.15.50:0/2657748627 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 10 monclient(hunting): get_auth_request con 0x7fc8dc0969a0 auth_method 0 2023-10-30T12:08:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): get_auth_request con 0x7fc8dc096080 auth_method 0 2023-10-30T12:08:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e0e82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1e84640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc095890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1e84640 10 monclient(hunting): get_auth_request con 0x7fc8dc09f5b0 auth_method 0 2023-10-30T12:08:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1e84640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1e84640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient(hunting): handle_auth_done global_id 124134 payload 824 2023-10-30T12:08:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 1 -- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc0969a0 msgr2=0x7fc8dc095350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc0969a0 0x7fc8dc095350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 1 -- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc09f5b0 msgr2=0x7fc8dc095890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc095890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient: _finish_hunting 0 2023-10-30T12:08:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 1 monclient: found mon.noname-a 2023-10-30T12:08:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.142+0000 7fc8e1683640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e1683640 1 -- 172.21.15.50:0/2657748627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc009400 con 0x7fc8dc096080 2023-10-30T12:08:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e1e84640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc095890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:08:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e0e82640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc0969a0 0x7fc8dc095350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e1683640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc096080 0x7fc8dc08b490 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8004000 tx=0x7fc8d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 client.?.objecter ms_handle_connect 0x7fc8dc096080 2023-10-30T12:08:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 1 -- 172.21.15.50:0/2657748627 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8003a20 con 0x7fc8dc096080 2023-10-30T12:08:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: dump: 2023-10-30T12:08:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:39.186 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:08:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:08:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 5 monclient: authenticate success, global_id 124134 2023-10-30T12:08:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 1 -- 172.21.15.50:0/2657748627 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d8003b80 con 0x7fc8dc096080 2023-10-30T12:08:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 10 monclient: _renew_subs 2023-10-30T12:08:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d8009460 con 0x7fc8dc096080 2023-10-30T12:08:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 1 -- 172.21.15.50:0/2657748627 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8006040 con 0x7fc8dc096080 2023-10-30T12:08:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d1ffb640 10 monclient: dump: 2023-10-30T12:08:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:39.190 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8d37fe640 4 set_mon_vals no callback set 2023-10-30T12:08:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 10 client.124134.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 10 monclient: _renew_subs 2023-10-30T12:08:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc071050 con 0x7fc8dc096080 2023-10-30T12:08:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.143+0000 7fc8e38e7640 1 librados: init done 2023-10-30T12:08:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8d1ffb640 1 -- 172.21.15.50:0/2657748627 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc8d803a040 con 0x7fc8dc096080 2023-10-30T12:08:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8e38e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8e38e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8e38e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc08dcd0 con 0x7fc8dc096080 2023-10-30T12:08:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8d1ffb640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc8c00392f0 0x7fc8c003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8d1ffb640 1 -- 172.21.15.50:0/2657748627 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6253+0+0 (secure 0 0 0) 0x7fc8d806c930 con 0x7fc8dc096080 2023-10-30T12:08:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8d1ffb640 10 client.124134.objecter ms_dispatch 0x7fc8dc000b90 osd_map(278..278 src has 1..278) v4 2023-10-30T12:08:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8d1ffb640 3 client.124134.objecter handle_osd_map got epochs [278,278] > 0 2023-10-30T12:08:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8d1ffb640 3 client.124134.objecter handle_osd_map decoding full epoch 278 2023-10-30T12:08:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8d1ffb640 20 client.124134.objecter dump_active .. 0 homeless 2023-10-30T12:08:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8e0e82640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc8c00392f0 0x7fc8c003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.144+0000 7fc8e0e82640 10 monclient: get_auth_request con 0x7fc8c00392f0 auth_method 0 2023-10-30T12:08:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.145+0000 7fc8d1ffb640 10 client.124134.objecter ms_handle_connect 0x7fc8c00392f0 2023-10-30T12:08:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.145+0000 7fc8e0e82640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc8c00392f0 0x7fc8c003b7b0 secure :-1 s=READY pgs=4049 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc08c390 tx=0x7fc8cc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.149+0000 7fc8d1ffb640 1 -- 172.21.15.50:0/2657748627 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc8dc08dcd0 con 0x7fc8dc096080 2023-10-30T12:08:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.149+0000 7fc8d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.149+0000 7fc8d1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:39.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.399+0000 7fc8e38e7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:08:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.399+0000 7fc8e38e7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:08:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.399+0000 7fc8e38e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.399+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc8dc08dcd0 con 0x7fc8dc096080 2023-10-30T12:08:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.400+0000 7fc8d1ffb640 1 -- 172.21.15.50:0/2657748627 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc8dc08dcd0 con 0x7fc8dc096080 2023-10-30T12:08:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.400+0000 7fc8d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:08:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.400+0000 7fc8d1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:39.403 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.400+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.400+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=9,logm=61} crc {auth=3160425676,config=3140514009,health=3523097348,kv=2905431493,logm=863658897}) 2023-10-30T12:08:39.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.400+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=8,logm=59} crc {auth=3676713303,config=3140514009,health=1436182930,kv=1634934840,logm=965203342}) 2023-10-30T12:08:39.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.400+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.400+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=9,logm=61} crc {auth=3160425676,config=3140514009,health=3523097348,kv=2905431493,logm=863658897}) 2023-10-30T12:08:39.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.400+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=7,logm=61} crc {auth=2823661657,config=3140514009,health=1436182930,kv=1634934840,logm=1958456772}) 2023-10-30T12:08:39.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.402+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.402+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251770866}) 2023-10-30T12:08:39.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.402+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895464270}) 2023-10-30T12:08:39.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.402+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.402+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251770866}) 2023-10-30T12:08:39.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.402+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7739447}) 2023-10-30T12:08:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 10 librados: watch_flush enter 2023-10-30T12:08:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 10 librados: watch_flush exit 2023-10-30T12:08:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 20 client.124134.objecter shutdown clearing up homeless session... 2023-10-30T12:08:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 10 client.124134.objecter successfully canceled tick 2023-10-30T12:08:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc8c00392f0 msgr2=0x7fc8c003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc8c00392f0 0x7fc8c003b7b0 secure :-1 s=READY pgs=4049 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc08c390 tx=0x7fc8cc009190 comp rx=0 tx=0).stop 2023-10-30T12:08:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 10 monclient: shutdown 2023-10-30T12:08:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc096080 msgr2=0x7fc8dc08b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc096080 0x7fc8dc08b490 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8004000 tx=0x7fc8d80046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 shutdown_connections 2023-10-30T12:08:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc8c00392f0 0x7fc8c003b7b0 unknown :-1 s=CLOSED pgs=4049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc09f5b0 0x7fc8dc095890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc8dc0969a0 0x7fc8dc095350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 --2- 172.21.15.50:0/2657748627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8dc096080 0x7fc8dc08b490 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 >> 172.21.15.50:0/2657748627 conn(0x7fc8dc0a05c0 msgr2=0x7fc8dc1ba700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 shutdown_connections 2023-10-30T12:08:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 -- 172.21.15.50:0/2657748627 wait complete. 2023-10-30T12:08:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:39.403+0000 7fc8e38e7640 1 librados: shutdown 2023-10-30T12:08:39.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.403+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.403+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555962784}) 2023-10-30T12:08:39.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.403+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010328397}) 2023-10-30T12:08:39.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.403+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.403+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555962784}) 2023-10-30T12:08:39.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.403+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=123753741}) 2023-10-30T12:08:39.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.404+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.404+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000811773}) 2023-10-30T12:08:39.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.404+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859327474}) 2023-10-30T12:08:39.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.404+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.404+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000811773}) 2023-10-30T12:08:39.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.404+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529483462}) 2023-10-30T12:08:39.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.405+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.405+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805240778}) 2023-10-30T12:08:39.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.405+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445418100}) 2023-10-30T12:08:39.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.405+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.420 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.405+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805240778}) 2023-10-30T12:08:39.420 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.405+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718829350}) 2023-10-30T12:08:39.420 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.420 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680398262}) 2023-10-30T12:08:39.420 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930764540}) 2023-10-30T12:08:39.421 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.421 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680398262}) 2023-10-30T12:08:39.421 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2674047436}) 2023-10-30T12:08:39.421 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.421 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283073946}) 2023-10-30T12:08:39.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4250458578}) 2023-10-30T12:08:39.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283073946}) 2023-10-30T12:08:39.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.406+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4136415404}) 2023-10-30T12:08:39.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.407+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.407+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036712}) 2023-10-30T12:08:39.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.407+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2392201203}) 2023-10-30T12:08:39.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.407+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.407+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036712}) 2023-10-30T12:08:39.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.407+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652045904}) 2023-10-30T12:08:39.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988346061}) 2023-10-30T12:08:39.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521217685}) 2023-10-30T12:08:39.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988346061}) 2023-10-30T12:08:39.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461326206}) 2023-10-30T12:08:39.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1723689778}) 2023-10-30T12:08:39.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4150462830}) 2023-10-30T12:08:39.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1723689778}) 2023-10-30T12:08:39.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.408+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3957523323}) 2023-10-30T12:08:39.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.409+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.409+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028299682}) 2023-10-30T12:08:39.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.409+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1662826405}) 2023-10-30T12:08:39.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.409+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.409+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028299682}) 2023-10-30T12:08:39.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.409+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=283080640}) 2023-10-30T12:08:39.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918503941}) 2023-10-30T12:08:39.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381267392}) 2023-10-30T12:08:39.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918503941}) 2023-10-30T12:08:39.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704868587}) 2023-10-30T12:08:39.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988875943}) 2023-10-30T12:08:39.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1541740163}) 2023-10-30T12:08:39.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988875943}) 2023-10-30T12:08:39.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.410+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266896540}) 2023-10-30T12:08:39.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.411+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.411+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331272593}) 2023-10-30T12:08:39.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.411+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253723138}) 2023-10-30T12:08:39.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.411+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.411+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331272593}) 2023-10-30T12:08:39.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.411+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869811092}) 2023-10-30T12:08:39.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.412+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.412+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250553096}) 2023-10-30T12:08:39.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.412+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930301600}) 2023-10-30T12:08:39.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.412+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.412+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250553096}) 2023-10-30T12:08:39.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.412+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114561853}) 2023-10-30T12:08:39.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2913490740}) 2023-10-30T12:08:39.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848900125}) 2023-10-30T12:08:39.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2913490740}) 2023-10-30T12:08:39.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118759742}) 2023-10-30T12:08:39.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329547086}) 2023-10-30T12:08:39.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261568012}) 2023-10-30T12:08:39.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329547086}) 2023-10-30T12:08:39.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.413+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3230787223}) 2023-10-30T12:08:39.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.414+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.434 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.414+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484309138}) 2023-10-30T12:08:39.434 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.414+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607271214}) 2023-10-30T12:08:39.434 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.414+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.434 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.414+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484309138}) 2023-10-30T12:08:39.434 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.414+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428386976}) 2023-10-30T12:08:39.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:08:39.445 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.415+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.445 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.415+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961337897}) 2023-10-30T12:08:39.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.415+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1211472779}) 2023-10-30T12:08:39.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.415+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.415+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961337897}) 2023-10-30T12:08:39.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.415+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348453911}) 2023-10-30T12:08:39.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726716142}) 2023-10-30T12:08:39.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163999431}) 2023-10-30T12:08:39.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726716142}) 2023-10-30T12:08:39.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258143945}) 2023-10-30T12:08:39.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204716428}) 2023-10-30T12:08:39.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2928612423}) 2023-10-30T12:08:39.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204716428}) 2023-10-30T12:08:39.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.416+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791618181}) 2023-10-30T12:08:39.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.417+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.417+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406729780}) 2023-10-30T12:08:39.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.417+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603418901}) 2023-10-30T12:08:39.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.417+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.417+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406729780}) 2023-10-30T12:08:39.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.417+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752835753}) 2023-10-30T12:08:39.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428792545}) 2023-10-30T12:08:39.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=804397986}) 2023-10-30T12:08:39.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428792545}) 2023-10-30T12:08:39.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3603037316}) 2023-10-30T12:08:39.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177810390}) 2023-10-30T12:08:39.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911719061}) 2023-10-30T12:08:39.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177810390}) 2023-10-30T12:08:39.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.418+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3599606010}) 2023-10-30T12:08:39.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.419+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.419+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558624917}) 2023-10-30T12:08:39.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.419+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245363185}) 2023-10-30T12:08:39.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.419+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.419+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558624917}) 2023-10-30T12:08:39.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.419+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043016830}) 2023-10-30T12:08:39.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506307522}) 2023-10-30T12:08:39.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4140806535}) 2023-10-30T12:08:39.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506307522}) 2023-10-30T12:08:39.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1572599655}) 2023-10-30T12:08:39.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385834335}) 2023-10-30T12:08:39.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420333068}) 2023-10-30T12:08:39.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385834335}) 2023-10-30T12:08:39.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.420+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571983710}) 2023-10-30T12:08:39.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.421+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.421+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228938313}) 2023-10-30T12:08:39.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.421+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808420251}) 2023-10-30T12:08:39.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.421+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.421+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228938313}) 2023-10-30T12:08:39.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.421+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=200993200}) 2023-10-30T12:08:39.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902261622}) 2023-10-30T12:08:39.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125194244}) 2023-10-30T12:08:39.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902261622}) 2023-10-30T12:08:39.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525602259}) 2023-10-30T12:08:39.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870584798}) 2023-10-30T12:08:39.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032809757}) 2023-10-30T12:08:39.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870584798}) 2023-10-30T12:08:39.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.422+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=289193119}) 2023-10-30T12:08:39.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.423+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.423+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916817118}) 2023-10-30T12:08:39.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.423+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=433109059}) 2023-10-30T12:08:39.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.423+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.423+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916817118}) 2023-10-30T12:08:39.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.423+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003730993}) 2023-10-30T12:08:39.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870833722}) 2023-10-30T12:08:39.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=639415056}) 2023-10-30T12:08:39.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870833722}) 2023-10-30T12:08:39.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2142507959}) 2023-10-30T12:08:39.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761575166}) 2023-10-30T12:08:39.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140241855}) 2023-10-30T12:08:39.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761575166}) 2023-10-30T12:08:39.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.424+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2822545699}) 2023-10-30T12:08:39.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.425+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.425+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476271816}) 2023-10-30T12:08:39.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.425+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117640707}) 2023-10-30T12:08:39.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.425+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.425+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476271816}) 2023-10-30T12:08:39.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.425+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1489889349}) 2023-10-30T12:08:39.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426351536}) 2023-10-30T12:08:39.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2769299650}) 2023-10-30T12:08:39.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426351536}) 2023-10-30T12:08:39.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753850900}) 2023-10-30T12:08:39.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697875450}) 2023-10-30T12:08:39.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457116684}) 2023-10-30T12:08:39.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697875450}) 2023-10-30T12:08:39.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.426+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2393932057}) 2023-10-30T12:08:39.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.427+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.427+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600679237}) 2023-10-30T12:08:39.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.427+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162300015}) 2023-10-30T12:08:39.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.427+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.427+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600679237}) 2023-10-30T12:08:39.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.427+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726503878}) 2023-10-30T12:08:39.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.428+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.428+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339506062}) 2023-10-30T12:08:39.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.428+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409819610}) 2023-10-30T12:08:39.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.428+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.428+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339506062}) 2023-10-30T12:08:39.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.428+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3690641935}) 2023-10-30T12:08:39.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.429+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.429+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326634757}) 2023-10-30T12:08:39.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.429+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=709065788}) 2023-10-30T12:08:39.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.429+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.429+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326634757}) 2023-10-30T12:08:39.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.429+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202477976}) 2023-10-30T12:08:39.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.430+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.430+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=7} crc {logm=4167310100,mdsmap=3984425268,mgr=1711728957,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3119165962,mon_config_key=3217606506,monmap=3324057927,osd_metadata=965620590,osd_pg_creating=2454960632,osd_snap=4247523408}) 2023-10-30T12:08:39.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.430+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=4235362361,mdsmap=3984425268,mgr=2906314847,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1521509918,mon_config_key=3003229359,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=2454960632,osd_snap=4247523408}) 2023-10-30T12:08:39.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.430+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.430+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=7} crc {logm=4167310100,mdsmap=3984425268,mgr=1711728957,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3119165962,mon_config_key=3217606506,monmap=3324057927,osd_metadata=965620590,osd_pg_creating=2454960632,osd_snap=4247523408}) 2023-10-30T12:08:39.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.430+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=58,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=6} crc {logm=2142545171,mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1337354103,mon_config_key=3217606506,monmap=4204254258,osd_metadata=3473538980,osd_pg_creating=2454960632,osd_snap=2743564566}) 2023-10-30T12:08:39.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=3194679779,osdmap=4098741193}) 2023-10-30T12:08:39.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=1981785187,osdmap=2131218464}) 2023-10-30T12:08:39.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=3194679779,osdmap=4098741193}) 2023-10-30T12:08:39.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=21,osdmap=79} crc {osd_snap=3861798579,osdmap=3823153103}) 2023-10-30T12:08:39.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=480593373}) 2023-10-30T12:08:39.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=99947613}) 2023-10-30T12:08:39.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=480593373}) 2023-10-30T12:08:39.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.431+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3924996115}) 2023-10-30T12:08:39.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.432+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.432+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2526727201}) 2023-10-30T12:08:39.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.432+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2674114151}) 2023-10-30T12:08:39.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.432+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.432+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2526727201}) 2023-10-30T12:08:39.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.433+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1355135269}) 2023-10-30T12:08:39.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.434+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.434+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3679538032}) 2023-10-30T12:08:39.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.434+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2491913579}) 2023-10-30T12:08:39.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.434+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.434+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3679538032}) 2023-10-30T12:08:39.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.434+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=193937932}) 2023-10-30T12:08:39.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.439+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.439+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=39} crc {osdmap=3455097828}) 2023-10-30T12:08:39.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.439+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=2393943696}) 2023-10-30T12:08:39.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.439+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:08:39.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.439+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=39} crc {osdmap=3455097828}) 2023-10-30T12:08:39.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:08:39.439+0000 7f4803930640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=35} crc {osdmap=3496346644}) 2023-10-30T12:08:40.437 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:08:40.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:40.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.553+0000 7fe7788d7640 10 monclient: get_monmap_and_config 2023-10-30T12:08:40.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.553+0000 7fe7788d7640 10 monclient: build_initial_monmap 2023-10-30T12:08:40.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.553+0000 7fe7788d7640 10 monclient: monmap: 2023-10-30T12:08:40.555 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:40.556 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.557 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:40.554707+0000 2023-10-30T12:08:40.557 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:40.554707+0000 2023-10-30T12:08:40.558 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:40.558 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.558 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:40.558 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:40.558 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:40.559 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:40.559 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:40.559 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 1 Processor -- start 2023-10-30T12:08:40.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 1 -- start start 2023-10-30T12:08:40.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient: init 2023-10-30T12:08:40.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:40.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T12:08:40.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe7740a0080 0x7fe7740a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient(hunting): picked mon.noname-c con 0x7fe7740a0080 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:40.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740720b0 0x7fe7740a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient(hunting): picked mon.noname-b con 0x7fe7740720b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:40.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe774073a30 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:40.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77409ccc0 con 0x7fe774073a30 2023-10-30T12:08:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe774073620 con 0x7fe7740a0080 2023-10-30T12:08:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740a0ee0 con 0x7fe7740720b0 2023-10-30T12:08:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.554+0000 7fe7788d7640 10 monclient(hunting): authenticate will time out at 1365.737918s 2023-10-30T12:08:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740720b0 0x7fe7740a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe772d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe774073a30 0x7fe774097320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40378/0 (socket says 172.21.15.50:40378) 2023-10-30T12:08:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe772d9d640 1 -- 172.21.15.50:0/3362771211 learned_addr learned my addr 172.21.15.50:0/3362771211 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:08:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): get_auth_request con 0x7fe7740720b0 auth_method 0 2023-10-30T12:08:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 1 --2- 172.21.15.50:0/3362771211 >> v2:172.21.15.118:3300/0 conn(0x7fe7740a0080 0x7fe7740a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 10 monclient(hunting): get_auth_request con 0x7fe7740a0080 auth_method 0 2023-10-30T12:08:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77159a640 1 -- 172.21.15.50:0/3362771211 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051929370 0 0) 0x7fe77409ccc0 con 0x7fe774073a30 2023-10-30T12:08:40.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient(hunting): handle_auth_done global_id 114257 payload 274 2023-10-30T12:08:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 1 -- 172.21.15.50:0/3362771211 >> v2:172.21.15.118:3300/0 conn(0x7fe7740a0080 msgr2=0x7fe7740a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 1 --2- 172.21.15.50:0/3362771211 >> v2:172.21.15.118:3300/0 conn(0x7fe7740a0080 0x7fe7740a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 1 -- 172.21.15.50:0/3362771211 >> v1:172.21.15.50:6789/0 conn(0x7fe774073a30 legacy=0x7fe774097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:08:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 1 monclient: found mon.noname-b 2023-10-30T12:08:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:40.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:40.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe771d9b640 1 -- 172.21.15.50:0/3362771211 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe774071e90 con 0x7fe7740720b0 2023-10-30T12:08:40.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.555+0000 7fe77159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:08:40.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe771d9b640 1 --2- 172.21.15.50:0/3362771211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740720b0 0x7fe7740a09a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe7680097a0 tx=0x7fe76802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44af12597102b65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 1 -- 172.21.15.50:0/3362771211 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76802f050 con 0x7fe7740720b0 2023-10-30T12:08:40.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:40.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: dump: 2023-10-30T12:08:40.577 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.577 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.578 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.578 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.578 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.578 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.578 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.578 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.579 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.579 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: _finish_auth 0 2023-10-30T12:08:40.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: _check_auth_tickets 2023-10-30T12:08:40.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:40.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 1 -- 172.21.15.50:0/3362771211 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768031700 con 0x7fe7740720b0 2023-10-30T12:08:40.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 5 monclient: authenticate success, global_id 114257 2023-10-30T12:08:40.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:40.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:40.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:40.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 1 -- 172.21.15.50:0/3362771211 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768031a90 con 0x7fe7740720b0 2023-10-30T12:08:40.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 4 set_mon_vals no callback set 2023-10-30T12:08:40.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:40.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe77159a640 10 monclient: dump: 2023-10-30T12:08:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.582 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.583 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.583 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.583 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.583 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 10 monclient: shutdown 2023-10-30T12:08:40.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:40.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 1 -- 172.21.15.50:0/3362771211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740720b0 msgr2=0x7fe7740a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 1 --2- 172.21.15.50:0/3362771211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740720b0 0x7fe7740a09a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe7680097a0 tx=0x7fe76802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 1 -- 172.21.15.50:0/3362771211 shutdown_connections 2023-10-30T12:08:40.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 1 --2- 172.21.15.50:0/3362771211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740720b0 0x7fe7740a09a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 1 --2- 172.21.15.50:0/3362771211 >> v2:172.21.15.118:3300/0 conn(0x7fe7740a0080 0x7fe7740a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.556+0000 7fe7788d7640 1 -- 172.21.15.50:0/3362771211 >> 172.21.15.50:0/3362771211 conn(0x7fe7740a4870 msgr2=0x7fe7740a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:40.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.557+0000 7fe7788d7640 1 -- 172.21.15.50:0/3362771211 shutdown_connections 2023-10-30T12:08:40.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.557+0000 7fe7788d7640 1 -- 172.21.15.50:0/3362771211 wait complete. 2023-10-30T12:08:40.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.557+0000 7fe7788d7640 10 monclient: build_initial_monmap 2023-10-30T12:08:40.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.557+0000 7fe7788d7640 10 monclient: monmap: 2023-10-30T12:08:40.586 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:40.586 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:40.586 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:40.586 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:40.587 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:40.587 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.587 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:40.587 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:40.587 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:40.587 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 librados: starting msgr at 2023-10-30T12:08:40.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 librados: starting objecter 2023-10-30T12:08:40.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 Processor -- start 2023-10-30T12:08:40.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 -- start start 2023-10-30T12:08:40.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 librados: setting wanted keys 2023-10-30T12:08:40.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 librados: calling monclient init 2023-10-30T12:08:40.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient: init 2023-10-30T12:08:40.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:40.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:08:40.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe774073a30 0x7fe7741c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): picked mon.noname-a con 0x7fe774073a30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:40.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740a0080 0x7fe774094180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): picked mon.noname-c con 0x7fe7740a0080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:40.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7740946c0 0x7fe77408c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): picked mon.noname-b con 0x7fe7740946c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:40.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77408f9f0 con 0x7fe774073a30 2023-10-30T12:08:40.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe774073e90 con 0x7fe7740946c0 2023-10-30T12:08:40.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7741bd700 con 0x7fe7740a0080 2023-10-30T12:08:40.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:40.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe7788d7640 10 monclient(hunting): authenticate will time out at 1365.741733s 2023-10-30T12:08:40.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740a0080 0x7fe774094180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe774073a30 0x7fe7741c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe774073a30 0x7fe7741c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50230/0 (socket says 172.21.15.50:50230) 2023-10-30T12:08:40.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740a0080 0x7fe774094180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36268/0 (socket says 172.21.15.50:36268) 2023-10-30T12:08:40.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 1 -- 172.21.15.50:0/1590078385 learned_addr learned my addr 172.21.15.50:0/1590078385 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:40.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 10 monclient(hunting): get_auth_request con 0x7fe774073a30 auth_method 0 2023-10-30T12:08:40.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 10 monclient(hunting): get_auth_request con 0x7fe7740a0080 auth_method 0 2023-10-30T12:08:40.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe77259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe771d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.558+0000 7fe772d9d640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7740946c0 0x7fe77408c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe772d9d640 10 monclient(hunting): get_auth_request con 0x7fe7740946c0 auth_method 0 2023-10-30T12:08:40.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe772d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe772d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe772d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe772d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe772d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe772d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 10 monclient(hunting): handle_auth_done global_id 124149 payload 824 2023-10-30T12:08:40.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 1 -- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740a0080 msgr2=0x7fe774094180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740a0080 0x7fe774094180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 1 -- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7740946c0 msgr2=0x7fe77408c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7740946c0 0x7fe77408c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 10 monclient: _finish_hunting 0 2023-10-30T12:08:40.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 1 monclient: found mon.noname-a 2023-10-30T12:08:40.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:40.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 1 -- 172.21.15.50:0/1590078385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009400 con 0x7fe774073a30 2023-10-30T12:08:40.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe771d9b640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740a0080 0x7fe774094180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:40.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe77259c640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe774073a30 0x7fe7741c9fc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe76400e900 tx=0x7fe76400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 client.?.objecter ms_handle_connect 0x7fe774073a30 2023-10-30T12:08:40.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:40.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 1 -- 172.21.15.50:0/1590078385 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764009660 con 0x7fe774073a30 2023-10-30T12:08:40.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:40.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: dump: 2023-10-30T12:08:40.600 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.600 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.601 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.601 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.601 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.601 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.601 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.602 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.602 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.602 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:40.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:40.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:40.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 1 -- 172.21.15.50:0/1590078385 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76400ca00 con 0x7fe774073a30 2023-10-30T12:08:40.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 5 monclient: authenticate success, global_id 124149 2023-10-30T12:08:40.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 10 monclient: _renew_subs 2023-10-30T12:08:40.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77408fc40 con 0x7fe774073a30 2023-10-30T12:08:40.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:40.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 1 -- 172.21.15.50:0/1590078385 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76400cd70 con 0x7fe774073a30 2023-10-30T12:08:40.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:40.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe762ffd640 10 monclient: dump: 2023-10-30T12:08:40.605 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.605 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.605 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.605 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.605 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.605 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.606 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.606 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.606 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.606 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe763fff640 4 set_mon_vals no callback set 2023-10-30T12:08:40.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 10 client.124149.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:40.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 10 monclient: _renew_subs 2023-10-30T12:08:40.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77408f3a0 con 0x7fe774073a30 2023-10-30T12:08:40.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.559+0000 7fe7788d7640 1 librados: init done 2023-10-30T12:08:40.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe762ffd640 1 -- 172.21.15.50:0/1590078385 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe764013470 con 0x7fe774073a30 2023-10-30T12:08:40.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe762ffd640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe754039320 0x7fe75403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe762ffd640 1 -- 172.21.15.50:0/1590078385 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fe76404b1f0 con 0x7fe774073a30 2023-10-30T12:08:40.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe762ffd640 10 client.124149.objecter ms_dispatch 0x7fe774000b90 osd_map(280..280 src has 1..280) v4 2023-10-30T12:08:40.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe762ffd640 3 client.124149.objecter handle_osd_map got epochs [280,280] > 0 2023-10-30T12:08:40.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe762ffd640 3 client.124149.objecter handle_osd_map decoding full epoch 280 2023-10-30T12:08:40.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe7788d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:40.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe7788d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:40.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe762ffd640 20 client.124149.objecter dump_active .. 0 homeless 2023-10-30T12:08:40.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe7788d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77404e160 con 0x7fe774073a30 2023-10-30T12:08:40.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe771d9b640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe754039320 0x7fe75403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.560+0000 7fe771d9b640 10 monclient: get_auth_request con 0x7fe754039320 auth_method 0 2023-10-30T12:08:40.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.561+0000 7fe771d9b640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe754039320 0x7fe75403b7e0 secure :-1 s=READY pgs=4051 cs=0 l=1 rev1=1 crypto rx=0x7fe768035000 tx=0x7fe768008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.561+0000 7fe762ffd640 10 client.124149.objecter ms_handle_connect 0x7fe754039320 2023-10-30T12:08:40.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.565+0000 7fe762ffd640 1 -- 172.21.15.50:0/1590078385 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe764017080 con 0x7fe774073a30 2023-10-30T12:08:40.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.565+0000 7fe762ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:40.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.565+0000 7fe762ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:40.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.808+0000 7fe7788d7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:40.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.808+0000 7fe7788d7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:40.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.808+0000 7fe7788d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.808+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe77409c2e0 con 0x7fe774073a30 2023-10-30T12:08:40.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.808+0000 7fe762ffd640 1 -- 172.21.15.50:0/1590078385 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fe76401b3b0 con 0x7fe774073a30 2023-10-30T12:08:40.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.808+0000 7fe762ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:40.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.808+0000 7fe762ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:40.811 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 10 librados: watch_flush enter 2023-10-30T12:08:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 10 librados: watch_flush exit 2023-10-30T12:08:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 20 client.124149.objecter shutdown clearing up homeless session... 2023-10-30T12:08:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 10 client.124149.objecter successfully canceled tick 2023-10-30T12:08:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe754039320 msgr2=0x7fe75403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe754039320 0x7fe75403b7e0 secure :-1 s=READY pgs=4051 cs=0 l=1 rev1=1 crypto rx=0x7fe768035000 tx=0x7fe768008030 comp rx=0 tx=0).stop 2023-10-30T12:08:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 10 monclient: shutdown 2023-10-30T12:08:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe774073a30 msgr2=0x7fe7741c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe774073a30 0x7fe7741c9fc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe76400e900 tx=0x7fe76400edc0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 shutdown_connections 2023-10-30T12:08:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe754039320 0x7fe75403b7e0 unknown :-1 s=CLOSED pgs=4051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7740946c0 0x7fe77408c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7740a0080 0x7fe774094180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 --2- 172.21.15.50:0/1590078385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe774073a30 0x7fe7741c9fc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 >> 172.21.15.50:0/1590078385 conn(0x7fe7740a4870 msgr2=0x7fe7741bd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 shutdown_connections 2023-10-30T12:08:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 -- 172.21.15.50:0/1590078385 wait complete. 2023-10-30T12:08:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.810+0000 7fe7788d7640 1 librados: shutdown 2023-10-30T12:08:40.822 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:08:40.822 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:08:40.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:08:40.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:08:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.927+0000 7f8af4d18640 10 monclient: get_monmap_and_config 2023-10-30T12:08:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.927+0000 7f8af4d18640 10 monclient: build_initial_monmap 2023-10-30T12:08:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.927+0000 7f8af4d18640 10 monclient: monmap: 2023-10-30T12:08:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:40.928220+0000 2023-10-30T12:08:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:40.928220+0000 2023-10-30T12:08:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:40.931 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.927+0000 7f8af4d18640 1 Processor -- start 2023-10-30T12:08:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.927+0000 7f8af4d18640 1 -- start start 2023-10-30T12:08:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.927+0000 7f8af4d18640 10 monclient: init 2023-10-30T12:08:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:40.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-10-30T12:08:40.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ae0 0x7f8af00a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient(hunting): picked mon.noname-b con 0x7f8af00a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:40.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8af00a2ec0 0x7f8af01356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient(hunting): picked mon.noname-a con 0x7f8af00a2ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8af00a32e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af00a4190 con 0x7f8af00a32e0 2023-10-30T12:08:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af00a2990 con 0x7f8af00a2ec0 2023-10-30T12:08:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af00a1630 con 0x7f8af00a2ae0 2023-10-30T12:08:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8af4d18640 10 monclient(hunting): authenticate will time out at 1366.111408s 2023-10-30T12:08:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ae0 0x7f8af00a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8af00a2ec0 0x7f8af01356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8af00a2ec0 0x7f8af01356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50244/0 (socket says 172.21.15.50:50244) 2023-10-30T12:08:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ae0 0x7f8af00a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36282/0 (socket says 172.21.15.50:36282) 2023-10-30T12:08:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 -- 172.21.15.50:0/1245617568 learned_addr learned my addr 172.21.15.50:0/1245617568 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): get_auth_request con 0x7f8af00a2ec0 auth_method 0 2023-10-30T12:08:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 10 monclient(hunting): get_auth_request con 0x7f8af00a2ae0 auth_method 0 2023-10-30T12:08:40.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_done global_id 124152 payload 274 2023-10-30T12:08:40.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 -- 172.21.15.50:0/1245617568 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ae0 msgr2=0x7f8af00a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:40.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 --2- 172.21.15.50:0/1245617568 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ae0 0x7f8af00a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 -- 172.21.15.50:0/1245617568 >> v1:172.21.15.50:6789/0 conn(0x7f8af00a32e0 legacy=0x7f8af0135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:08:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 monclient: found mon.noname-a 2023-10-30T12:08:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.928+0000 7f8aedd9b640 1 -- 172.21.15.50:0/1245617568 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af009a2e0 con 0x7f8af00a2ec0 2023-10-30T12:08:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aedd9b640 1 --2- 172.21.15.50:0/1245617568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ec0 0x7f8af01356e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009870 tx=0x7f8ad802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cae72004ec3f730 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 1 -- 172.21.15.50:0/1245617568 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80317c0 con 0x7f8af00a2ec0 2023-10-30T12:08:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: dump: 2023-10-30T12:08:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.951 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: _finish_auth 0 2023-10-30T12:08:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: _check_auth_tickets 2023-10-30T12:08:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8af4d18640 5 monclient: authenticate success, global_id 124152 2023-10-30T12:08:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8af4d18640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 1 -- 172.21.15.50:0/1245617568 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad8031920 con 0x7f8af00a2ec0 2023-10-30T12:08:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8af4d18640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8af4d18640 4 set_mon_vals no callback set 2023-10-30T12:08:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 1 -- 172.21.15.50:0/1245617568 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8031cb0 con 0x7f8af00a2ec0 2023-10-30T12:08:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.929+0000 7f8aed59a640 10 monclient: dump: 2023-10-30T12:08:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.955 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 10 monclient: shutdown 2023-10-30T12:08:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 -- 172.21.15.50:0/1245617568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ec0 msgr2=0x7f8af01356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 --2- 172.21.15.50:0/1245617568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ec0 0x7f8af01356e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009870 tx=0x7f8ad802b860 comp rx=0 tx=0).stop 2023-10-30T12:08:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 -- 172.21.15.50:0/1245617568 shutdown_connections 2023-10-30T12:08:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 --2- 172.21.15.50:0/1245617568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ec0 0x7f8af01356e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 --2- 172.21.15.50:0/1245617568 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ae0 0x7f8af00a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 -- 172.21.15.50:0/1245617568 >> 172.21.15.50:0/1245617568 conn(0x7f8af00a80b0 msgr2=0x7f8af00a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 -- 172.21.15.50:0/1245617568 shutdown_connections 2023-10-30T12:08:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.930+0000 7f8af4d18640 1 -- 172.21.15.50:0/1245617568 wait complete. 2023-10-30T12:08:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient: build_initial_monmap 2023-10-30T12:08:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient: monmap: 2023-10-30T12:08:40.957 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:40.959 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 librados: starting msgr at 2023-10-30T12:08:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 librados: starting objecter 2023-10-30T12:08:40.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 Processor -- start 2023-10-30T12:08:40.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 -- start start 2023-10-30T12:08:40.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 librados: setting wanted keys 2023-10-30T12:08:40.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 librados: calling monclient init 2023-10-30T12:08:40.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient: init 2023-10-30T12:08:40.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:08:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ae0 0x7f8af01ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): picked mon.noname-a con 0x7f8af00a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ec0 0x7f8af0132550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): picked mon.noname-c con 0x7f8af00a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:40.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af0132a90 0x7f8af011e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): picked mon.noname-b con 0x7f8af0132a90 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:40.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01ca7d0 con 0x7f8af00a2ae0 2023-10-30T12:08:40.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af00a1790 con 0x7f8af0132a90 2023-10-30T12:08:40.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0120680 con 0x7f8af00a2ec0 2023-10-30T12:08:40.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:40.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.931+0000 7f8af4d18640 10 monclient(hunting): authenticate will time out at 1366.114954s 2023-10-30T12:08:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ae0 0x7f8af01ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ec0 0x7f8af0132550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ae0 0x7f8af01ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50248/0 (socket says 172.21.15.50:50248) 2023-10-30T12:08:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ec0 0x7f8af0132550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36296/0 (socket says 172.21.15.50:36296) 2023-10-30T12:08:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 -- 172.21.15.50:0/3682902743 learned_addr learned my addr 172.21.15.50:0/3682902743 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): get_auth_request con 0x7f8af00a2ae0 auth_method 0 2023-10-30T12:08:40.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 10 monclient(hunting): get_auth_request con 0x7f8af00a2ec0 auth_method 0 2023-10-30T12:08:40.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient(hunting): handle_auth_done global_id 124155 payload 824 2023-10-30T12:08:40.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aeed9d640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af0132a90 0x7f8af011e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ec0 msgr2=0x7f8af0132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ec0 0x7f8af0132550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af0132a90 msgr2=0x7f8af011e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af0132a90 0x7f8af011e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient: _finish_hunting 0 2023-10-30T12:08:40.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 monclient: found mon.noname-a 2023-10-30T12:08:40.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:40.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 -- 172.21.15.50:0/3682902743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad80094d0 con 0x7f8af00a2ae0 2023-10-30T12:08:40.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aedd9b640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8af00a2ec0 0x7f8af0132550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:08:40.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aee59c640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ae0 0x7f8af01ca290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800e9b0 tx=0x7f8ae800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 client.?.objecter ms_handle_connect 0x7f8af00a2ae0 2023-10-30T12:08:40.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:40.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8009670 con 0x7f8af00a2ae0 2023-10-30T12:08:40.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:40.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: dump: 2023-10-30T12:08:40.970 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.971 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.971 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.971 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.971 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.971 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.971 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.972 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.972 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.972 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:40.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae800c9a0 con 0x7f8af00a2ae0 2023-10-30T12:08:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 5 monclient: authenticate success, global_id 124155 2023-10-30T12:08:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 10 monclient: _renew_subs 2023-10-30T12:08:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0120960 con 0x7f8af00a2ae0 2023-10-30T12:08:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800cd50 con 0x7f8af00a2ae0 2023-10-30T12:08:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8ae6ffd640 10 monclient: dump: 2023-10-30T12:08:40.975 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.975 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.975 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.975 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.975 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.975 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.976 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.976 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.976 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.976 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8aecd99640 4 set_mon_vals no callback set 2023-10-30T12:08:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 10 client.124155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 10 monclient: _renew_subs 2023-10-30T12:08:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af0120cf0 con 0x7f8af00a2ae0 2023-10-30T12:08:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.932+0000 7f8af4d18640 1 librados: init done 2023-10-30T12:08:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8af4d18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8af4d18640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a32e0 0x7f8af011e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8af4d18640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0073ff0 con 0x7f8af00a32e0 2023-10-30T12:08:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8ae8013470 con 0x7f8af00a2ae0 2023-10-30T12:08:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a32e0 0x7f8af011e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient: get_auth_request con 0x7f8af00a32e0 auth_method 0 2023-10-30T12:08:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8ae6ffd640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ad00392f0 0x7f8ad003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_done global_id 124158 payload 824 2023-10-30T12:08:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8aedd9b640 1 -- 172.21.15.50:0/3682902743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8af009a5c0 con 0x7f8af00a32e0 2023-10-30T12:08:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f8ae804af80 con 0x7f8af00a2ae0 2023-10-30T12:08:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8ae6ffd640 10 client.124155.objecter ms_dispatch 0x7f8af0000b90 osd_map(280..280 src has 1..280) v4 2023-10-30T12:08:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8ae6ffd640 3 client.124155.objecter handle_osd_map got epochs [280,280] > 0 2023-10-30T12:08:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.933+0000 7f8ae6ffd640 3 client.124155.objecter handle_osd_map decoding full epoch 280 2023-10-30T12:08:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 20 client.124155.objecter dump_active .. 0 homeless 2023-10-30T12:08:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8aedd9b640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a32e0 0x7f8af011e5e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8af00a5500 tx=0x7f8ad8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8aeed9d640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ad00392f0 0x7f8ad003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8aeed9d640 10 monclient: get_auth_request con 0x7f8ad00392f0 auth_method 0 2023-10-30T12:08:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 client.124155.objecter ms_handle_connect 0x7f8af00a32e0 2023-10-30T12:08:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 client.124155.objecter resend_mon_ops 2023-10-30T12:08:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8049070 con 0x7f8af00a32e0 2023-10-30T12:08:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 monclient: dump: 2023-10-30T12:08:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.984 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.984 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.984 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.985 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.985 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8ad8032af0 con 0x7f8af00a32e0 2023-10-30T12:08:40.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:40.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:40.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a32e0 msgr2=0x7f8af011e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a32e0 0x7f8af011e5e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8af00a5500 tx=0x7f8ad8004730 comp rx=0 tx=0).stop 2023-10-30T12:08:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a32e0 msgr2=0x7f8af011e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a32e0 0x7f8af011e5e0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8af00a5500 tx=0x7f8ad8004730 comp rx=0 tx=0).stop 2023-10-30T12:08:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8ae6ffd640 10 client.124155.objecter ms_handle_connect 0x7f8ad00392f0 2023-10-30T12:08:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.934+0000 7f8aeed9d640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ad00392f0 0x7f8ad003b7b0 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0007fd0 tx=0x7f8ae0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8af4d18640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8af4d18640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af011eb20 0x7f8af009c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8af4d18640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01207f0 con 0x7f8af011eb20 2023-10-30T12:08:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af011eb20 0x7f8af009c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 10 monclient: get_auth_request con 0x7f8af011eb20 auth_method 0 2023-10-30T12:08:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:40.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:40.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:40.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:40.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:40.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.940+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:40.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8aedd9b640 10 monclient(hunting): handle_auth_done global_id 124161 payload 824 2023-10-30T12:08:40.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8aedd9b640 1 -- 172.21.15.50:0/3682902743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8af0099b40 con 0x7f8af011eb20 2023-10-30T12:08:40.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 10 client.124155.objecter ms_handle_connect 0x7f8af011eb20 2023-10-30T12:08:40.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 10 client.124155.objecter resend_mon_ops 2023-10-30T12:08:40.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8aedd9b640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af011eb20 0x7f8af009c400 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009870 tx=0x7f8ad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:40.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad804e840 con 0x7f8af011eb20 2023-10-30T12:08:40.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:40.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:40.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 10 monclient: dump: 2023-10-30T12:08:40.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:40.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:40.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.993 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:40.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:40.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:40.993 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:40.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:40.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:40.994 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f8ad8052070 con 0x7f8af011eb20 2023-10-30T12:08:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af011eb20 msgr2=0x7f8af009c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af011eb20 0x7f8af009c400 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009870 tx=0x7f8ad80046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af011eb20 msgr2=0x7f8af009c400 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:40.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8ae6ffd640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af011eb20 0x7f8af009c400 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009870 tx=0x7f8ad80046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:40.995 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:40.995 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:40.996 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:40.996 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:08:40.996 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 78, 2023-10-30T12:08:40.996 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:40.996 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:08:40.996 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:08:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:08:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:41.022 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:41.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 10 librados: watch_flush enter 2023-10-30T12:08:41.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 10 librados: watch_flush exit 2023-10-30T12:08:41.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 20 client.124155.objecter shutdown clearing up homeless session... 2023-10-30T12:08:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 10 client.124155.objecter successfully canceled tick 2023-10-30T12:08:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ad00392f0 msgr2=0x7f8ad003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ad00392f0 0x7f8ad003b7b0 secure :-1 s=READY pgs=4052 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0007fd0 tx=0x7f8ae0009040 comp rx=0 tx=0).stop 2023-10-30T12:08:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 10 monclient: shutdown 2023-10-30T12:08:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ae0 msgr2=0x7f8af01ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 1 --2- 172.21.15.50:0/3682902743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8af00a2ae0 0x7f8af01ca290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800e9b0 tx=0x7f8ae800ee70 comp rx=0 tx=0).stop 2023-10-30T12:08:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8aeed9d640 1 -- 172.21.15.50:0/3682902743 reap_dead start 2023-10-30T12:08:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8aeed9d640 1 -- 172.21.15.50:0/3682902743 reap_dead start 2023-10-30T12:08:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 shutdown_connections 2023-10-30T12:08:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.941+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 >> 172.21.15.50:0/3682902743 conn(0x7f8af00a80b0 msgr2=0x7f8af01b9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.942+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 shutdown_connections 2023-10-30T12:08:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.942+0000 7f8af4d18640 1 -- 172.21.15.50:0/3682902743 wait complete. 2023-10-30T12:08:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:40.942+0000 7f8af4d18640 1 librados: shutdown 2023-10-30T12:08:41.027 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:08:41.028 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:08:41.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:08:41.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.132+0000 7f54db173640 10 monclient: get_monmap_and_config 2023-10-30T12:08:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.132+0000 7f54db173640 10 monclient: build_initial_monmap 2023-10-30T12:08:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.132+0000 7f54db173640 10 monclient: monmap: 2023-10-30T12:08:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:41.133600+0000 2023-10-30T12:08:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:41.133600+0000 2023-10-30T12:08:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:41.136 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 1 Processor -- start 2023-10-30T12:08:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 1 -- start start 2023-10-30T12:08:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient: init 2023-10-30T12:08:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:08:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f54d409bb70 0x7f54d409bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient(hunting): picked mon.noname-a con 0x7f54d409bb70 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f54d409d3f0 0x7f54d409c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient(hunting): picked mon.noname-c con 0x7f54d409d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54d4096080 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4098170 con 0x7f54d4096080 2023-10-30T12:08:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d409c9d0 con 0x7f54d409bb70 2023-10-30T12:08:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d409cf60 con 0x7f54d409d3f0 2023-10-30T12:08:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54db173640 10 monclient(hunting): authenticate will time out at 1366.316767s 2023-10-30T12:08:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d8f0f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f54d409bb70 0x7f54d409bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d8f0f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f54d409bb70 0x7f54d409bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50270/0 (socket says 172.21.15.50:50270) 2023-10-30T12:08:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d8f0f640 1 -- 172.21.15.50:0/4243659426 learned_addr learned my addr 172.21.15.50:0/4243659426 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:41.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d9710640 1 --1- 172.21.15.50:0/4243659426 >> v1:172.21.15.50:6789/0 conn(0x7f54d4096080 0x7f54d4096510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40394/0 (socket says 172.21.15.50:40394) 2023-10-30T12:08:41.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d8f0f640 10 monclient(hunting): get_auth_request con 0x7f54d409bb70 auth_method 0 2023-10-30T12:08:41.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d8f0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d8f0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.133+0000 7f54d8f0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d3fff640 1 --2- 172.21.15.50:0/4243659426 >> v2:172.21.15.118:3300/0 conn(0x7f54d409d3f0 0x7f54d409c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d3fff640 10 monclient(hunting): get_auth_request con 0x7f54d409d3f0 auth_method 0 2023-10-30T12:08:41.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_done global_id 124164 payload 274 2023-10-30T12:08:41.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 1 -- 172.21.15.50:0/4243659426 >> v2:172.21.15.118:3300/0 conn(0x7f54d409d3f0 msgr2=0x7f54d409c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:41.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 1 --2- 172.21.15.50:0/4243659426 >> v2:172.21.15.118:3300/0 conn(0x7f54d409d3f0 0x7f54d409c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 1 -- 172.21.15.50:0/4243659426 >> v1:172.21.15.50:6789/0 conn(0x7f54d4096080 legacy=0x7f54d4096510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 10 monclient: _finish_hunting 0 2023-10-30T12:08:41.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 1 monclient: found mon.noname-a 2023-10-30T12:08:41.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:41.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:41.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 1 -- 172.21.15.50:0/4243659426 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4097b20 con 0x7f54d409bb70 2023-10-30T12:08:41.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d3fff640 1 --2- 172.21.15.50:0/4243659426 >> v2:172.21.15.118:3300/0 conn(0x7f54d409d3f0 0x7f54d409c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:08:41.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.134+0000 7f54d8f0f640 1 --2- 172.21.15.50:0/4243659426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d409bb70 0x7f54d409bf50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54d409d840 tx=0x7f54c802e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3963053dc24efc16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 1 -- 172.21.15.50:0/4243659426 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c802ede0 con 0x7f54d409bb70 2023-10-30T12:08:41.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:41.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: dump: 2023-10-30T12:08:41.153 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.153 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.153 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.153 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.153 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.154 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.154 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.154 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.154 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.154 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: _finish_auth 0 2023-10-30T12:08:41.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:08:41.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:41.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 1 -- 172.21.15.50:0/4243659426 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c8004030 con 0x7f54d409bb70 2023-10-30T12:08:41.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 5 monclient: authenticate success, global_id 124164 2023-10-30T12:08:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 1 -- 172.21.15.50:0/4243659426 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8004420 con 0x7f54d409bb70 2023-10-30T12:08:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 4 set_mon_vals no callback set 2023-10-30T12:08:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:41.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54d37fe640 10 monclient: dump: 2023-10-30T12:08:41.157 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.157 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.157 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.158 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.158 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.158 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.158 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.158 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.159 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.159 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 10 monclient: shutdown 2023-10-30T12:08:41.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:41.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 -- 172.21.15.50:0/4243659426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d409bb70 msgr2=0x7f54d409bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 --2- 172.21.15.50:0/4243659426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d409bb70 0x7f54d409bf50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54d409d840 tx=0x7f54c802e030 comp rx=0 tx=0).stop 2023-10-30T12:08:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 -- 172.21.15.50:0/4243659426 shutdown_connections 2023-10-30T12:08:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 --2- 172.21.15.50:0/4243659426 >> v2:172.21.15.118:3300/0 conn(0x7f54d409d3f0 0x7f54d409c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 --2- 172.21.15.50:0/4243659426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d409bb70 0x7f54d409bf50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 -- 172.21.15.50:0/4243659426 >> 172.21.15.50:0/4243659426 conn(0x7f54d40a3980 msgr2=0x7f54d40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 -- 172.21.15.50:0/4243659426 shutdown_connections 2023-10-30T12:08:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.135+0000 7f54db173640 1 -- 172.21.15.50:0/4243659426 wait complete. 2023-10-30T12:08:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 10 monclient: build_initial_monmap 2023-10-30T12:08:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 10 monclient: monmap: 2023-10-30T12:08:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:41.163 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 1 librados: starting msgr at 2023-10-30T12:08:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 1 librados: starting objecter 2023-10-30T12:08:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 1 Processor -- start 2023-10-30T12:08:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 1 -- start start 2023-10-30T12:08:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 1 librados: setting wanted keys 2023-10-30T12:08:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 1 librados: calling monclient init 2023-10-30T12:08:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.136+0000 7f54db173640 10 monclient: init 2023-10-30T12:08:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:08:41.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d4096080 0x7f54d41c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): picked mon.noname-a con 0x7f54d4096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:41.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54d409bb70 0x7f54d40954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): picked mon.noname-b con 0x7f54d409bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:41.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d409d3f0 0x7f54d40959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): picked mon.noname-c con 0x7f54d409d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:41.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d408d160 con 0x7f54d4096080 2023-10-30T12:08:41.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4097120 con 0x7f54d409bb70 2023-10-30T12:08:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d408e440 con 0x7f54d409d3f0 2023-10-30T12:08:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient(hunting): authenticate will time out at 1366.320139s 2023-10-30T12:08:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d4096080 0x7f54d41c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d4096080 0x7f54d41c4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50286/0 (socket says 172.21.15.50:50286) 2023-10-30T12:08:41.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 -- 172.21.15.50:0/1769262794 learned_addr learned my addr 172.21.15.50:0/1769262794 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): get_auth_request con 0x7f54d4096080 auth_method 0 2023-10-30T12:08:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d409d3f0 0x7f54d40959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 10 monclient(hunting): get_auth_request con 0x7f54d409d3f0 auth_method 0 2023-10-30T12:08:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d3fff640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54d409bb70 0x7f54d40954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d9710640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d3fff640 10 monclient(hunting): get_auth_request con 0x7f54d409bb70 auth_method 0 2023-10-30T12:08:41.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient(hunting): handle_auth_done global_id 124170 payload 824 2023-10-30T12:08:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d409d3f0 msgr2=0x7f54d40959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d409d3f0 0x7f54d40959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54d409bb70 msgr2=0x7f54d40954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54d409bb70 0x7f54d40954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient: _finish_hunting 0 2023-10-30T12:08:41.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 monclient: found mon.noname-a 2023-10-30T12:08:41.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:41.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:41.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 -- 172.21.15.50:0/1769262794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c800b700 con 0x7f54d4096080 2023-10-30T12:08:41.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d8f0f640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d4096080 0x7f54d41c4210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f54c8038040 tx=0x7f54c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 client.?.objecter ms_handle_connect 0x7f54d4096080 2023-10-30T12:08:41.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:41.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8003a20 con 0x7f54d4096080 2023-10-30T12:08:41.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:41.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: dump: 2023-10-30T12:08:41.176 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.176 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.176 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.177 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.177 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.177 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.177 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.177 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.177 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.178 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: _finish_auth 0 2023-10-30T12:08:41.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: _check_auth_tickets 2023-10-30T12:08:41.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:41.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c8003b80 con 0x7f54d4096080 2023-10-30T12:08:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8005e10 con 0x7f54d4096080 2023-10-30T12:08:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 5 monclient: authenticate success, global_id 124170 2023-10-30T12:08:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d17fa640 10 monclient: dump: 2023-10-30T12:08:41.180 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.180 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.180 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.180 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.180 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.181 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.181 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.181 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.181 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.181 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient: _renew_subs 2023-10-30T12:08:41.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54d2ffd640 4 set_mon_vals no callback set 2023-10-30T12:08:41.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:41.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.137+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d40976e0 con 0x7f54d4096080 2023-10-30T12:08:41.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 10 client.124170.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:41.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 10 monclient: _renew_subs 2023-10-30T12:08:41.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:41.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d408c8a0 con 0x7f54d4096080 2023-10-30T12:08:41.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 1 librados: init done 2023-10-30T12:08:41.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f54c8006040 con 0x7f54d4096080 2023-10-30T12:08:41.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:41.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406dbb0 0x7f54d406df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4073ff0 con 0x7f54d406dbb0 2023-10-30T12:08:41.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54d3fff640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406dbb0 0x7f54d406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54d3fff640 10 monclient: get_auth_request con 0x7f54d406dbb0 auth_method 0 2023-10-30T12:08:41.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.138+0000 7f54d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54b80392f0 0x7f54b803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f54c806fe90 con 0x7f54d4096080 2023-10-30T12:08:41.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 client.124170.objecter ms_dispatch 0x7f54d4000b90 osd_map(280..280 src has 1..280) v4 2023-10-30T12:08:41.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 3 client.124170.objecter handle_osd_map got epochs [280,280] > 0 2023-10-30T12:08:41.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 3 client.124170.objecter handle_osd_map decoding full epoch 280 2023-10-30T12:08:41.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 20 client.124170.objecter dump_active .. 0 homeless 2023-10-30T12:08:41.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d3fff640 10 monclient(hunting): handle_auth_done global_id 114272 payload 824 2023-10-30T12:08:41.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d3fff640 1 -- 172.21.15.50:0/1769262794 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d4097560 con 0x7f54d406dbb0 2023-10-30T12:08:41.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d9710640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54b80392f0 0x7f54b803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d9710640 10 monclient: get_auth_request con 0x7f54b80392f0 auth_method 0 2023-10-30T12:08:41.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 client.124170.objecter ms_handle_connect 0x7f54d406dbb0 2023-10-30T12:08:41.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d3fff640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406dbb0 0x7f54d406df90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c0010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 client.124170.objecter resend_mon_ops 2023-10-30T12:08:41.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c000e9c0 con 0x7f54d406dbb0 2023-10-30T12:08:41.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:41.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 monclient: dump: 2023-10-30T12:08:41.189 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.189 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.189 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.189 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.189 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.190 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.190 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.190 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.190 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.190 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 client.124170.objecter ms_handle_connect 0x7f54b80392f0 2023-10-30T12:08:41.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d9710640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54b80392f0 0x7f54b803b7b0 secure :-1 s=READY pgs=4053 cs=0 l=1 rev1=1 crypto rx=0x7f54d408edb0 tx=0x7f54cc001670 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f54c000c050 con 0x7f54d406dbb0 2023-10-30T12:08:41.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:41.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:41.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406dbb0 msgr2=0x7f54d406df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406dbb0 0x7f54d406df90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c0010c20 comp rx=0 tx=0).stop 2023-10-30T12:08:41.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406dbb0 msgr2=0x7f54d406df90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:41.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.139+0000 7f54d17fa640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406dbb0 0x7f54d406df90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54db173640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:41.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54db173640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406e4d0 0x7f54d408afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54db173640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4097e00 con 0x7f54d406e4d0 2023-10-30T12:08:41.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406e4d0 0x7f54d408afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient: get_auth_request con 0x7f54d406e4d0 auth_method 0 2023-10-30T12:08:41.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 10 monclient(hunting): handle_auth_done global_id 114275 payload 824 2023-10-30T12:08:41.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 1 -- 172.21.15.50:0/1769262794 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f54d404e160 con 0x7f54d406e4d0 2023-10-30T12:08:41.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d17fa640 10 client.124170.objecter ms_handle_connect 0x7f54d406e4d0 2023-10-30T12:08:41.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d17fa640 10 client.124170.objecter resend_mon_ops 2023-10-30T12:08:41.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d3fff640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406e4d0 0x7f54d408afd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0003cb0 con 0x7f54d406e4d0 2023-10-30T12:08:41.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:41.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.146+0000 7f54d17fa640 10 monclient: dump: 2023-10-30T12:08:41.198 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.198 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.198 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.198 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.199 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.199 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.199 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.199 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.199 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.200 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f54c0003e10 con 0x7f54d406e4d0 2023-10-30T12:08:41.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:41.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:41.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406e4d0 msgr2=0x7f54d408afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d17fa640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406e4d0 0x7f54d408afd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c00046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d17fa640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406e4d0 msgr2=0x7f54d408afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:41.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d17fa640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54d406e4d0 0x7f54d408afd0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c00046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.201 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:41.202 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:41.202 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:41.202 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:08:41.202 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 78, 2023-10-30T12:08:41.202 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:41.203 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:08:41.203 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:41.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:41.203 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.203 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:08:41.203 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:41.204 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:41.204 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:41.204 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.204 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.204 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.205 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.206 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.206 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.206 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:41.206 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:41.206 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.207 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.207 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.207 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.207 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.208 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.208 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.208 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.208 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.208 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.209 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:41.209 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:41.209 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:41.209 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:41.209 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:41.209 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:41.210 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:41.210 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:41.210 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:41.210 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:41.210 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:41.211 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:41.211 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:41.211 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:41.211 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:41.211 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:41.211 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.213 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.213 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.213 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.213 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.213 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.213 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:41.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.214 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:41.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.214 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:41.214 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:41.215 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.215 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.215 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.215 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:41.216 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.216 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.216 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:41.216 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.217 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:41.217 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:41.217 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.218 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.218 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.218 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:41.219 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:41.219 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.219 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.219 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.219 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:41.220 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.220 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.220 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:41.221 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.221 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.221 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.221 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:41.222 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:41.222 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.222 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.222 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.223 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:41.223 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:41.223 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.223 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.224 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.224 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:41.224 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.224 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.225 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:41.225 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.225 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.225 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.226 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:41.226 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:41.226 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.226 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.226 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.226 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.227 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.227 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:41.227 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:41.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.228 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.228 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.228 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:41.228 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.228 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.228 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:41.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.229 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.229 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.229 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:41.229 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.230 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.230 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:41.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.230 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.230 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.230 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:41.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.231 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.231 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:41.231 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:41.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 10 librados: watch_flush enter 2023-10-30T12:08:41.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 10 librados: watch_flush exit 2023-10-30T12:08:41.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 20 client.124170.objecter shutdown clearing up homeless session... 2023-10-30T12:08:41.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 10 client.124170.objecter successfully canceled tick 2023-10-30T12:08:41.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54b80392f0 msgr2=0x7f54b803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54b80392f0 0x7f54b803b7b0 secure :-1 s=READY pgs=4053 cs=0 l=1 rev1=1 crypto rx=0x7f54d408edb0 tx=0x7f54cc001670 comp rx=0 tx=0).stop 2023-10-30T12:08:41.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 10 monclient: shutdown 2023-10-30T12:08:41.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:41.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d4096080 msgr2=0x7f54d41c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d4096080 0x7f54d41c4210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f54c8038040 tx=0x7f54c80046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54d9710640 1 -- 172.21.15.50:0/1769262794 reap_dead start 2023-10-30T12:08:41.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 shutdown_connections 2023-10-30T12:08:41.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 --2- 172.21.15.50:0/1769262794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54d4096080 0x7f54d41c4210 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 >> 172.21.15.50:0/1769262794 conn(0x7f54d40a3980 msgr2=0x7f54d40a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:41.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 shutdown_connections 2023-10-30T12:08:41.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 -- 172.21.15.50:0/1769262794 wait complete. 2023-10-30T12:08:41.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.147+0000 7f54db173640 1 librados: shutdown 2023-10-30T12:08:41.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:08:41.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:08:41.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:08:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.345+0000 7f674c2da640 10 monclient: get_monmap_and_config 2023-10-30T12:08:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.345+0000 7f674c2da640 10 monclient: build_initial_monmap 2023-10-30T12:08:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.345+0000 7f674c2da640 10 monclient: monmap: 2023-10-30T12:08:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.348 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:41.346516+0000 2023-10-30T12:08:41.348 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:41.346516+0000 2023-10-30T12:08:41.348 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:41.348 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.348 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:41.348 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:41.349 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:41.349 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:41.349 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:41.349 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 1 Processor -- start 2023-10-30T12:08:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 1 -- start start 2023-10-30T12:08:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient: init 2023-10-30T12:08:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T12:08:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6744071c00 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6744097fe0 0x7f674409a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient(hunting): picked mon.noname-b con 0x7f6744097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f674409b170 addr v1:172.21.15.118:6789/0 2023-10-30T12:08:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6744097600 con 0x7f6744071c00 2023-10-30T12:08:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67440978e0 con 0x7f674409b170 2023-10-30T12:08:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674409bf50 con 0x7f6744097fe0 2023-10-30T12:08:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674c2da640 10 monclient(hunting): authenticate will time out at 1366.529706s 2023-10-30T12:08:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6744097fe0 0x7f674409a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674a076640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6744071c00 0x7f674409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40398/0 (socket says 172.21.15.50:40398) 2023-10-30T12:08:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f674a076640 1 -- 172.21.15.50:0/1624705806 learned_addr learned my addr 172.21.15.50:0/1624705806 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:08:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 10 monclient(hunting): get_auth_request con 0x7f6744097fe0 auth_method 0 2023-10-30T12:08:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.346+0000 7f6749875640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 10 monclient(hunting): handle_auth_done global_id 114278 payload 274 2023-10-30T12:08:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749074640 1 -- 172.21.15.50:0/1624705806 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948095143 0 0) 0x7f6744097600 con 0x7f6744071c00 2023-10-30T12:08:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 1 -- 172.21.15.50:0/1624705806 >> v1:172.21.15.118:6789/0 conn(0x7f674409b170 legacy=0x7f6744139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 1 -- 172.21.15.50:0/1624705806 >> v1:172.21.15.50:6789/0 conn(0x7f6744071c00 legacy=0x7f674409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 10 monclient: _finish_hunting 0 2023-10-30T12:08:41.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 1 monclient: found mon.noname-b 2023-10-30T12:08:41.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:41.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:41.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 1 -- 172.21.15.50:0/1624705806 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674409d9b0 con 0x7f6744097fe0 2023-10-30T12:08:41.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749074640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:08:41.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.347+0000 7f6749875640 1 --2- 172.21.15.50:0/1624705806 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6744097fe0 0x7f674409a150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67380097d0 tx=0x7f673802c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e7e0fdad28e073c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 1 -- 172.21.15.50:0/1624705806 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738032880 con 0x7f6744097fe0 2023-10-30T12:08:41.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:41.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: dump: 2023-10-30T12:08:41.364 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.364 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.364 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.364 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.365 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.365 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.365 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.365 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.365 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.365 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: _finish_auth 0 2023-10-30T12:08:41.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: _check_auth_tickets 2023-10-30T12:08:41.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:41.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 1 -- 172.21.15.50:0/1624705806 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67380329e0 con 0x7f6744097fe0 2023-10-30T12:08:41.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 5 monclient: authenticate success, global_id 114278 2023-10-30T12:08:41.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:41.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:41.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:41.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 1 -- 172.21.15.50:0/1624705806 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738032d70 con 0x7f6744097fe0 2023-10-30T12:08:41.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 4 set_mon_vals no callback set 2023-10-30T12:08:41.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:41.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f6749074640 10 monclient: dump: 2023-10-30T12:08:41.368 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.368 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.368 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.369 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.369 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.369 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.369 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.369 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.369 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.370 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 10 monclient: shutdown 2023-10-30T12:08:41.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:41.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 1 -- 172.21.15.50:0/1624705806 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6744097fe0 msgr2=0x7f674409a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 1 --2- 172.21.15.50:0/1624705806 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6744097fe0 0x7f674409a150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67380097d0 tx=0x7f673802c920 comp rx=0 tx=0).stop 2023-10-30T12:08:41.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 1 -- 172.21.15.50:0/1624705806 shutdown_connections 2023-10-30T12:08:41.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 1 --2- 172.21.15.50:0/1624705806 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6744097fe0 0x7f674409a150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 1 -- 172.21.15.50:0/1624705806 >> 172.21.15.50:0/1624705806 conn(0x7f67440a3980 msgr2=0x7f67440a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:41.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 1 -- 172.21.15.50:0/1624705806 shutdown_connections 2023-10-30T12:08:41.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.348+0000 7f674c2da640 1 -- 172.21.15.50:0/1624705806 wait complete. 2023-10-30T12:08:41.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient: build_initial_monmap 2023-10-30T12:08:41.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient: monmap: 2023-10-30T12:08:41.372 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:41.372 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:41.372 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:41.373 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:41.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:41.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:41.373 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:41.373 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:41.374 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 librados: starting msgr at 2023-10-30T12:08:41.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 librados: starting objecter 2023-10-30T12:08:41.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 Processor -- start 2023-10-30T12:08:41.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 -- start start 2023-10-30T12:08:41.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 librados: setting wanted keys 2023-10-30T12:08:41.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 librados: calling monclient init 2023-10-30T12:08:41.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient: init 2023-10-30T12:08:41.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:41.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:08:41.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6744071c00 0x7f67441c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient(hunting): picked mon.noname-b con 0x7f6744071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:41.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6744097fe0 0x7f67441b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient(hunting): picked mon.noname-a con 0x7f6744097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:41.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f674409b170 0x7f67441b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient(hunting): picked mon.noname-c con 0x7f674409b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:41.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674409cbc0 con 0x7f6744097fe0 2023-10-30T12:08:41.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.349+0000 7f674c2da640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6744070c60 con 0x7f6744071c00 2023-10-30T12:08:41.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6744071090 con 0x7f674409b170 2023-10-30T12:08:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 monclient(hunting): authenticate will time out at 1366.533021s 2023-10-30T12:08:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6744097fe0 0x7f67441b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6744097fe0 0x7f67441b54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50296/0 (socket says 172.21.15.50:50296) 2023-10-30T12:08:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 -- 172.21.15.50:0/508405025 learned_addr learned my addr 172.21.15.50:0/508405025 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): get_auth_request con 0x7f6744097fe0 auth_method 0 2023-10-30T12:08:41.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f674409b170 0x7f67441b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 10 monclient(hunting): get_auth_request con 0x7f674409b170 auth_method 0 2023-10-30T12:08:41.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a877640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a076640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6744071c00 0x7f67441c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a076640 10 monclient(hunting): get_auth_request con 0x7f6744071c00 auth_method 0 2023-10-30T12:08:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a076640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a076640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674a076640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient(hunting): handle_auth_done global_id 124176 payload 824 2023-10-30T12:08:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f674409b170 msgr2=0x7f67441b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f674409b170 0x7f67441b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6744071c00 msgr2=0x7f67441c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6744071c00 0x7f67441c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient: _finish_hunting 0 2023-10-30T12:08:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 monclient: found mon.noname-a 2023-10-30T12:08:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:08:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 -- 172.21.15.50:0/508405025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6738009400 con 0x7f6744097fe0 2023-10-30T12:08:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6749875640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6744097fe0 0x7f67441b54e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f67380098f0 tx=0x7f67380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 client.?.objecter ms_handle_connect 0x7f6744097fe0 2023-10-30T12:08:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738032800 con 0x7f6744097fe0 2023-10-30T12:08:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:08:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: dump: 2023-10-30T12:08:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.389 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.389 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.389 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:41.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738047430 con 0x7f6744097fe0 2023-10-30T12:08:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67380477e0 con 0x7f6744097fe0 2023-10-30T12:08:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 5 monclient: authenticate success, global_id 124176 2023-10-30T12:08:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 monclient: _renew_subs 2023-10-30T12:08:41.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:41.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6748873640 4 set_mon_vals no callback set 2023-10-30T12:08:41.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67441aec80 con 0x7f6744097fe0 2023-10-30T12:08:41.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:08:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f6736ffd640 10 monclient: dump: 2023-10-30T12:08:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.393 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 client.124176.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 monclient: _renew_subs 2023-10-30T12:08:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:08:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67440706e0 con 0x7f6744097fe0 2023-10-30T12:08:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.350+0000 7f674c2da640 1 librados: init done 2023-10-30T12:08:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674c2da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674c2da640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406ee00 0x7f674406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674c2da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6744073ff0 con 0x7f674406ee00 2023-10-30T12:08:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f673803c430 con 0x7f6744097fe0 2023-10-30T12:08:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406ee00 0x7f674406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 10 monclient: get_auth_request con 0x7f674406ee00 auth_method 0 2023-10-30T12:08:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f6736ffd640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67280392f0 0x7f672803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f6738042030 con 0x7f6744097fe0 2023-10-30T12:08:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f6736ffd640 10 client.124176.objecter ms_dispatch 0x7f6744000b90 osd_map(281..281 src has 1..281) v4 2023-10-30T12:08:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f6736ffd640 3 client.124176.objecter handle_osd_map got epochs [281,281] > 0 2023-10-30T12:08:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f6736ffd640 3 client.124176.objecter handle_osd_map decoding full epoch 281 2023-10-30T12:08:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.351+0000 7f674a076640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 20 client.124176.objecter dump_active .. 0 homeless 2023-10-30T12:08:41.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f674a877640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67280392f0 0x7f672803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f674a877640 10 monclient: get_auth_request con 0x7f67280392f0 auth_method 0 2023-10-30T12:08:41.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f674a076640 10 monclient(hunting): handle_auth_done global_id 94630 payload 824 2023-10-30T12:08:41.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f674a076640 1 -- 172.21.15.50:0/508405025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6744071fe0 con 0x7f674406ee00 2023-10-30T12:08:41.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f674a076640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406ee00 0x7f674406f1e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f674000eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 client.124176.objecter ms_handle_connect 0x7f674406ee00 2023-10-30T12:08:41.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 client.124176.objecter resend_mon_ops 2023-10-30T12:08:41.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67400070d0 con 0x7f674406ee00 2023-10-30T12:08:41.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f674a877640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67280392f0 0x7f672803b7b0 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 crypto rx=0x7f673c0012e0 tx=0x7f673c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 monclient: dump: 2023-10-30T12:08:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.401 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.401 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.401 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.401 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.401 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 client.124176.objecter ms_handle_connect 0x7f67280392f0 2023-10-30T12:08:41.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f6740007230 con 0x7f674406ee00 2023-10-30T12:08:41.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:41.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:41.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406ee00 msgr2=0x7f674406f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406ee00 0x7f674406f1e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f674000eb90 comp rx=0 tx=0).stop 2023-10-30T12:08:41.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406ee00 msgr2=0x7f674406f1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:41.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.352+0000 7f6736ffd640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406ee00 0x7f674406f1e0 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f674000eb90 comp rx=0 tx=0).stop 2023-10-30T12:08:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674c2da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674c2da640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406f720 0x7f67441b6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674c2da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67441aee10 con 0x7f674406f720 2023-10-30T12:08:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406f720 0x7f67441b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient: get_auth_request con 0x7f674406f720 auth_method 0 2023-10-30T12:08:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 10 monclient(hunting): handle_auth_done global_id 94633 payload 824 2023-10-30T12:08:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 1 -- 172.21.15.50:0/508405025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f674404e160 con 0x7f674406f720 2023-10-30T12:08:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f674a076640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406f720 0x7f67441b6520 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f6740001200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f6736ffd640 10 client.124176.objecter ms_handle_connect 0x7f674406f720 2023-10-30T12:08:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.359+0000 7f6736ffd640 10 client.124176.objecter resend_mon_ops 2023-10-30T12:08:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740023070 con 0x7f674406f720 2023-10-30T12:08:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 10 monclient: dump: 2023-10-30T12:08:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f6740005cb0 con 0x7f674406f720 2023-10-30T12:08:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406f720 msgr2=0x7f67441b6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406f720 0x7f67441b6520 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f6740001200 comp rx=0 tx=0).stop 2023-10-30T12:08:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406f720 msgr2=0x7f67441b6520 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f6736ffd640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f674406f720 0x7f67441b6520 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f6740001200 comp rx=0 tx=0).stop 2023-10-30T12:08:41.412 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:41.412 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:41.412 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:41.413 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:08:41.413 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 78, 2023-10-30T12:08:41.413 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:41.413 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:08:41.413 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:41.414 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:41.414 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.414 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:08:41.414 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:41.414 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:41.414 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:41.415 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.415 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.415 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.415 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.416 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.416 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.416 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.416 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.416 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.416 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.417 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:41.417 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:41.417 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.417 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.417 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.417 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.418 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.418 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.418 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.418 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.418 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.419 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.419 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.419 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:41.419 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:41.422 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:41.428 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:41.428 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.428 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.428 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.429 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:41.429 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:41.429 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.433 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:41.433 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:41.433 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:41.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:41.436 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:41.436 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:41.436 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:41.436 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:41.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.439 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.439 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.439 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:41.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.439 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.440 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:41.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.440 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.440 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.440 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:08:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:08:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:41.442 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 10 librados: watch_flush enter 2023-10-30T12:08:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 10 librados: watch_flush exit 2023-10-30T12:08:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 20 client.124176.objecter shutdown clearing up homeless session... 2023-10-30T12:08:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 10 client.124176.objecter successfully canceled tick 2023-10-30T12:08:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67280392f0 msgr2=0x7f672803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67280392f0 0x7f672803b7b0 secure :-1 s=READY pgs=4054 cs=0 l=1 rev1=1 crypto rx=0x7f673c0012e0 tx=0x7f673c009040 comp rx=0 tx=0).stop 2023-10-30T12:08:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 10 monclient: shutdown 2023-10-30T12:08:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6744097fe0 msgr2=0x7f67441b54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 --2- 172.21.15.50:0/508405025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6744097fe0 0x7f67441b54e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f67380098f0 tx=0x7f67380046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674a877640 1 -- 172.21.15.50:0/508405025 reap_dead start 2023-10-30T12:08:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674a877640 1 -- 172.21.15.50:0/508405025 reap_dead start 2023-10-30T12:08:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 shutdown_connections 2023-10-30T12:08:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 >> 172.21.15.50:0/508405025 conn(0x7f67440a3980 msgr2=0x7f67441bbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 shutdown_connections 2023-10-30T12:08:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 -- 172.21.15.50:0/508405025 wait complete. 2023-10-30T12:08:41.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.360+0000 7f674c2da640 1 librados: shutdown 2023-10-30T12:08:41.447 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:08:41.448 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:08:41.448 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:08:41.448 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:08:41.448 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:08:41.449 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:08:41.449 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:08:41.459 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:08:41.459 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:08:41.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:41.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.602+0000 7f34b8fb0640 10 monclient: get_monmap_and_config 2023-10-30T12:08:41.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.602+0000 7f34b8fb0640 10 monclient: build_initial_monmap 2023-10-30T12:08:41.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.602+0000 7f34b8fb0640 10 monclient: monmap: 2023-10-30T12:08:41.604 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:41.603826+0000 2023-10-30T12:08:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:41.603826+0000 2023-10-30T12:08:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:41.606 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:41.606 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:41.606 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:41.606 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:41.606 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 1 Processor -- start 2023-10-30T12:08:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 1 -- start start 2023-10-30T12:08:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 10 monclient: init 2023-10-30T12:08:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:08:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34b40659d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:08:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f34b4071410 0x7f34b4138720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.603+0000 7f34b8fb0640 10 monclient(hunting): picked mon.noname-c con 0x7f34b4071410 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b8fb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34b4071850 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b8fb0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b409fb30 con 0x7f34b4071850 2023-10-30T12:08:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b8fb0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b4065db0 con 0x7f34b40659d0 2023-10-30T12:08:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b8fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b8fb0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b409e090 con 0x7f34b4071410 2023-10-30T12:08:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b8fb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b8fb0640 10 monclient(hunting): authenticate will time out at 1366.787069s 2023-10-30T12:08:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b2d9d640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f34b4071850 legacy=0x7f34b413ac70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f34b4071410 0x7f34b4138720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b259c640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f34b40659d0 0x7f34b412fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:41242/0 (socket says 172.21.15.50:41242) 2023-10-30T12:08:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34b259c640 1 -- 172.21.15.50:0/3878503127 learned_addr learned my addr 172.21.15.50:0/3878503127 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:08:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): get_auth_request con 0x7f34b4071410 auth_method 0 2023-10-30T12:08:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient(hunting): handle_auth_done global_id 94636 payload 274 2023-10-30T12:08:41.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 1 -- 172.21.15.50:0/3878503127 >> v1:172.21.15.118:6789/0 conn(0x7f34b40659d0 legacy=0x7f34b412fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 1 -- 172.21.15.50:0/3878503127 >> v1:172.21.15.50:6789/0 conn(0x7f34b4071850 legacy=0x7f34b413ac70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:41.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient: _finish_hunting 0 2023-10-30T12:08:41.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 1 monclient: found mon.noname-c 2023-10-30T12:08:41.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:41.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:08:41.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.604+0000 7f34abfff640 1 -- 172.21.15.50:0/3878503127 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b4130570 con 0x7f34b4071410 2023-10-30T12:08:41.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34abfff640 1 --2- 172.21.15.50:0/3878503127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 0x7f34b4138720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f349c0097a0 tx=0x7f349c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed8f9e5c2990a068 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 1 -- 172.21.15.50:0/3878503127 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c02f050 con 0x7f34b4071410 2023-10-30T12:08:41.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:41.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: dump: 2023-10-30T12:08:41.620 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.620 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.620 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.622 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: _finish_auth 0 2023-10-30T12:08:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:08:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b8fb0640 5 monclient: authenticate success, global_id 94636 2023-10-30T12:08:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 1 -- 172.21.15.50:0/3878503127 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c031700 con 0x7f34b4071410 2023-10-30T12:08:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b8fb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b8fb0640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b8fb0640 4 set_mon_vals no callback set 2023-10-30T12:08:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 1 -- 172.21.15.50:0/3878503127 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c031a90 con 0x7f34b4071410 2023-10-30T12:08:41.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:41.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.605+0000 7f34b1d9b640 10 monclient: dump: 2023-10-30T12:08:41.624 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.624 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.625 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.626 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 10 monclient: shutdown 2023-10-30T12:08:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 1 -- 172.21.15.50:0/3878503127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 msgr2=0x7f34b4138720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/3878503127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 0x7f34b4138720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f349c0097a0 tx=0x7f349c02b7a0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 1 -- 172.21.15.50:0/3878503127 shutdown_connections 2023-10-30T12:08:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/3878503127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 0x7f34b4138720 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 1 -- 172.21.15.50:0/3878503127 >> 172.21.15.50:0/3878503127 conn(0x7f34b40a47f0 msgr2=0x7f34b40a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:41.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 1 -- 172.21.15.50:0/3878503127 shutdown_connections 2023-10-30T12:08:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 1 -- 172.21.15.50:0/3878503127 wait complete. 2023-10-30T12:08:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 10 monclient: build_initial_monmap 2023-10-30T12:08:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.606+0000 7f34b8fb0640 10 monclient: monmap: 2023-10-30T12:08:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:41.630 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 librados: starting msgr at 2023-10-30T12:08:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 librados: starting objecter 2023-10-30T12:08:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 Processor -- start 2023-10-30T12:08:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 -- start start 2023-10-30T12:08:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 librados: setting wanted keys 2023-10-30T12:08:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 librados: calling monclient init 2023-10-30T12:08:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient: init 2023-10-30T12:08:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:08:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34b40659d0 0x7f34b409b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): picked mon.noname-c con 0x7f34b40659d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 0x7f34b409b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): picked mon.noname-b con 0x7f34b4071410 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34b4071850 0x7f34b409bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): picked mon.noname-a con 0x7f34b4071850 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b409c9c0 con 0x7f34b4071850 2023-10-30T12:08:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4136390 con 0x7f34b4071410 2023-10-30T12:08:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4136670 con 0x7f34b40659d0 2023-10-30T12:08:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b8fb0640 10 monclient(hunting): authenticate will time out at 1366.790636s 2023-10-30T12:08:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b2d9d640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34b4071850 msgr2=0x7f34b409bd60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34b4071850 0x7f34b409bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34b40659d0 0x7f34b409b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34b40659d0 0x7f34b409b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36360/0 (socket says 172.21.15.50:36360) 2023-10-30T12:08:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 1 -- 172.21.15.50:0/1239525520 learned_addr learned my addr 172.21.15.50:0/1239525520 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 10 monclient(hunting): get_auth_request con 0x7f34b40659d0 auth_method 0 2023-10-30T12:08:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34abfff640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 0x7f34b409b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34abfff640 10 monclient(hunting): get_auth_request con 0x7f34b4071410 auth_method 0 2023-10-30T12:08:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34abfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.607+0000 7f34abfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 10 monclient(hunting): handle_auth_done global_id 114284 payload 824 2023-10-30T12:08:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 1 -- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 msgr2=0x7f34b409b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 0x7f34b409b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 1 -- 172.21.15.50:0/1239525520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34b4071850 msgr2=0x7f34b409bd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34b4071850 0x7f34b409bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 10 monclient: _finish_hunting 0 2023-10-30T12:08:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 1 monclient: found mon.noname-c 2023-10-30T12:08:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 1 -- 172.21.15.50:0/1239525520 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c009400 con 0x7f34b40659d0 2023-10-30T12:08:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b259c640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34b40659d0 0x7f34b409b2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c4a0 tx=0x7f34a400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 client.?.objecter ms_handle_connect 0x7f34b40659d0 2023-10-30T12:08:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 1 -- 172.21.15.50:0/1239525520 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4005ce0 con 0x7f34b40659d0 2023-10-30T12:08:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: dump: 2023-10-30T12:08:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.643 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 5 monclient: authenticate success, global_id 114284 2023-10-30T12:08:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 10 monclient: _renew_subs 2023-10-30T12:08:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b4136950 con 0x7f34b40659d0 2023-10-30T12:08:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 1 -- 172.21.15.50:0/1239525520 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a4005e40 con 0x7f34b40659d0 2023-10-30T12:08:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 1 -- 172.21.15.50:0/1239525520 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4007270 con 0x7f34b40659d0 2023-10-30T12:08:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34aaffd640 10 monclient: dump: 2023-10-30T12:08:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:41.647 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 10 client.114284.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 10 monclient: _renew_subs 2023-10-30T12:08:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b159a640 4 set_mon_vals no callback set 2023-10-30T12:08:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b413bc90 con 0x7f34b40659d0 2023-10-30T12:08:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.608+0000 7f34b8fb0640 1 librados: init done 2023-10-30T12:08:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34aaffd640 1 -- 172.21.15.50:0/1239525520 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f34a40073d0 con 0x7f34b40659d0 2023-10-30T12:08:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34b8fb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34b8fb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34b8fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b404e160 con 0x7f34b40659d0 2023-10-30T12:08:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34aaffd640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34aaffd640 1 -- 172.21.15.50:0/1239525520 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f34a404a1d0 con 0x7f34b40659d0 2023-10-30T12:08:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34aaffd640 10 client.114284.objecter ms_dispatch 0x7f34b4000b90 osd_map(281..281 src has 1..281) v4 2023-10-30T12:08:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34aaffd640 3 client.114284.objecter handle_osd_map got epochs [281,281] > 0 2023-10-30T12:08:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34aaffd640 3 client.114284.objecter handle_osd_map decoding full epoch 281 2023-10-30T12:08:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34aaffd640 20 client.114284.objecter dump_active .. 0 homeless 2023-10-30T12:08:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34abfff640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34abfff640 1 -- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 msgr2=0x7f349403b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:08:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34abfff640 1 -- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 msgr2=0x7f349403b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:08:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34abfff640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.609+0000 7f34abfff640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.614+0000 7f34aaffd640 1 -- 172.21.15.50:0/1239525520 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34a401b3b0 con 0x7f34b40659d0 2023-10-30T12:08:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.614+0000 7f34aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.614+0000 7f34aaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:41.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.810+0000 7f34abfff640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.810+0000 7f34abfff640 10 monclient: get_auth_request con 0x7f34940392a0 auth_method 0 2023-10-30T12:08:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.810+0000 7f34abfff640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 secure :-1 s=READY pgs=4055 cs=0 l=1 rev1=1 crypto rx=0x7f349c035000 tx=0x7f349c03f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.810+0000 7f34aaffd640 10 client.114284.objecter ms_handle_connect 0x7f34940392a0 2023-10-30T12:08:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.859+0000 7f34b8fb0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.859+0000 7f34b8fb0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.859+0000 7f34b8fb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:41.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.859+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f34b406a1b0 con 0x7f34b40659d0 2023-10-30T12:08:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.859+0000 7f34aaffd640 1 -- 172.21.15.50:0/1239525520 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f34a4017080 con 0x7f34b40659d0 2023-10-30T12:08:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.859+0000 7f34aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:41.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.859+0000 7f34aaffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:41.863 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 10 librados: watch_flush enter 2023-10-30T12:08:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 10 librados: watch_flush exit 2023-10-30T12:08:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 20 client.114284.objecter shutdown clearing up homeless session... 2023-10-30T12:08:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 10 client.114284.objecter successfully canceled tick 2023-10-30T12:08:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 msgr2=0x7f349403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 secure :-1 s=READY pgs=4055 cs=0 l=1 rev1=1 crypto rx=0x7f349c035000 tx=0x7f349c03f040 comp rx=0 tx=0).stop 2023-10-30T12:08:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 10 monclient: shutdown 2023-10-30T12:08:41.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34b40659d0 msgr2=0x7f34b409b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.861+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34b40659d0 0x7f34b409b2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c4a0 tx=0x7f34a400c960 comp rx=0 tx=0).stop 2023-10-30T12:08:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 shutdown_connections 2023-10-30T12:08:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f34940392a0 0x7f349403b760 unknown :-1 s=CLOSED pgs=4055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34b4071850 0x7f34b409bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b4071410 0x7f34b409b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 --2- 172.21.15.50:0/1239525520 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34b40659d0 0x7f34b409b2e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 >> 172.21.15.50:0/1239525520 conn(0x7f34b40a47f0 msgr2=0x7f34b412dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 shutdown_connections 2023-10-30T12:08:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 -- 172.21.15.50:0/1239525520 wait complete. 2023-10-30T12:08:41.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:41.862+0000 7f34b8fb0640 1 librados: shutdown 2023-10-30T12:08:44.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:44.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.989+0000 7f7e01725640 10 monclient: get_monmap_and_config 2023-10-30T12:08:44.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.989+0000 7f7e01725640 10 monclient: build_initial_monmap 2023-10-30T12:08:44.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.989+0000 7f7e01725640 10 monclient: monmap: 2023-10-30T12:08:44.991 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:44.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:44.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:44.990827+0000 2023-10-30T12:08:44.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:44.990827+0000 2023-10-30T12:08:44.992 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:44.992 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:44.992 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:44.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:44.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:44.993 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:44.993 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:44.993 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:45.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.990+0000 7f7e01725640 1 Processor -- start 2023-10-30T12:08:45.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.990+0000 7f7e01725640 1 -- start start 2023-10-30T12:08:45.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.990+0000 7f7e01725640 10 monclient: init 2023-10-30T12:08:45.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:45.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T12:08:45.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:45.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient(hunting): picked mon.noname-b con 0x7f7dfc098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:45.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7dfc098900 0x7f7dfc098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:45.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient(hunting): picked mon.noname-c con 0x7f7dfc098900 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:45.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7dfc132740 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:45.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dfc09ad30 con 0x7f7dfc132740 2023-10-30T12:08:45.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dfc09c460 con 0x7f7dfc098900 2023-10-30T12:08:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc09b260 con 0x7f7dfc098520 2023-10-30T12:08:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7e01725640 10 monclient(hunting): authenticate will time out at 1370.174168s 2023-10-30T12:08:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfb7fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f7dfc132740 legacy=0x7f7dfc132bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc09bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36362/0 (socket says 172.21.15.50:36362) 2023-10-30T12:08:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 -- 172.21.15.50:0/4057857563 learned_addr learned my addr 172.21.15.50:0/4057857563 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): get_auth_request con 0x7f7dfc098520 auth_method 0 2023-10-30T12:08:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7df3fff640 1 --2- 172.21.15.50:0/4057857563 >> v2:172.21.15.118:3300/0 conn(0x7f7dfc098900 0x7f7dfc098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_done global_id 114287 payload 274 2023-10-30T12:08:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 -- 172.21.15.50:0/4057857563 >> v2:172.21.15.118:3300/0 conn(0x7f7dfc098900 msgr2=0x7f7dfc098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 --2- 172.21.15.50:0/4057857563 >> v2:172.21.15.118:3300/0 conn(0x7f7dfc098900 0x7f7dfc098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 -- 172.21.15.50:0/4057857563 >> v1:172.21.15.50:6789/0 conn(0x7f7dfc132740 legacy=0x7f7dfc132bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:08:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 monclient: found mon.noname-b 2023-10-30T12:08:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.991+0000 7f7dfaffd640 1 -- 172.21.15.50:0/4057857563 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc072120 con 0x7f7dfc098520 2023-10-30T12:08:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfaffd640 1 --2- 172.21.15.50:0/4057857563 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc09bf20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009850 tx=0x7f7dec02b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6fcc30ffce4ffa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 1 -- 172.21.15.50:0/4057857563 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec004230 con 0x7f7dfc098520 2023-10-30T12:08:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: dump: 2023-10-30T12:08:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:45.011 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: _finish_auth 0 2023-10-30T12:08:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:08:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 1 -- 172.21.15.50:0/4057857563 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dec02f050 con 0x7f7dfc098520 2023-10-30T12:08:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7e01725640 5 monclient: authenticate success, global_id 114287 2023-10-30T12:08:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7e01725640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7e01725640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 1 -- 172.21.15.50:0/4057857563 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec003610 con 0x7f7dfc098520 2023-10-30T12:08:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7e01725640 4 set_mon_vals no callback set 2023-10-30T12:08:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.992+0000 7f7dfa7fc640 10 monclient: dump: 2023-10-30T12:08:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:45.016 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 10 monclient: shutdown 2023-10-30T12:08:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 -- 172.21.15.50:0/4057857563 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 msgr2=0x7f7dfc09bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 --2- 172.21.15.50:0/4057857563 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc09bf20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009850 tx=0x7f7dec02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 -- 172.21.15.50:0/4057857563 shutdown_connections 2023-10-30T12:08:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 --2- 172.21.15.50:0/4057857563 >> v2:172.21.15.118:3300/0 conn(0x7f7dfc098900 0x7f7dfc098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 --2- 172.21.15.50:0/4057857563 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc09bf20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 -- 172.21.15.50:0/4057857563 >> 172.21.15.50:0/4057857563 conn(0x7f7dfc0a05c0 msgr2=0x7f7dfc09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 -- 172.21.15.50:0/4057857563 shutdown_connections 2023-10-30T12:08:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 -- 172.21.15.50:0/4057857563 wait complete. 2023-10-30T12:08:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 10 monclient: build_initial_monmap 2023-10-30T12:08:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 10 monclient: monmap: 2023-10-30T12:08:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:45.020 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 librados: starting msgr at 2023-10-30T12:08:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.993+0000 7f7e01725640 1 librados: starting objecter 2023-10-30T12:08:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 Processor -- start 2023-10-30T12:08:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 -- start start 2023-10-30T12:08:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 librados: setting wanted keys 2023-10-30T12:08:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 librados: calling monclient init 2023-10-30T12:08:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient: init 2023-10-30T12:08:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:08:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc06e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): picked mon.noname-c con 0x7f7dfc098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7dfc098900 0x7f7dfc06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): picked mon.noname-b con 0x7f7dfc098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dfc132740 0x7f7dfc06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): picked mon.noname-a con 0x7f7dfc132740 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc071a60 con 0x7f7dfc132740 2023-10-30T12:08:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0706c0 con 0x7f7dfc098900 2023-10-30T12:08:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc071df0 con 0x7f7dfc098520 2023-10-30T12:08:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7e01725640 10 monclient(hunting): authenticate will time out at 1370.177587s 2023-10-30T12:08:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfb7fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dfc132740 msgr2=0x7f7dfc06ead0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dfc132740 0x7f7dfc06ead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc06e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36376/0 (socket says 172.21.15.50:36376) 2023-10-30T12:08:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 1 -- 172.21.15.50:0/3867525060 learned_addr learned my addr 172.21.15.50:0/3867525060 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): get_auth_request con 0x7f7dfc098520 auth_method 0 2023-10-30T12:08:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7df3fff640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7dfc098900 0x7f7dfc06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7df3fff640 10 monclient(hunting): get_auth_request con 0x7f7dfc098900 auth_method 0 2023-10-30T12:08:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7df3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7df3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.994+0000 7f7dfaffd640 10 monclient(hunting): handle_auth_done global_id 114290 payload 824 2023-10-30T12:08:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 1 -- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7dfc098900 msgr2=0x7f7dfc06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7dfc098900 0x7f7dfc06e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 1 -- 172.21.15.50:0/3867525060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dfc132740 msgr2=0x7f7dfc06ead0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dfc132740 0x7f7dfc06ead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:08:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 1 monclient: found mon.noname-c 2023-10-30T12:08:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 1 -- 172.21.15.50:0/3867525060 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec009460 con 0x7f7dfc098520 2023-10-30T12:08:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df3fff640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7dfc098900 0x7f7dfc06e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:08:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7dfaffd640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc06e050 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0090b0 tx=0x7f7dec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 client.?.objecter ms_handle_connect 0x7f7dfc098520 2023-10-30T12:08:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 1 -- 172.21.15.50:0/3867525060 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec003b60 con 0x7f7dfc098520 2023-10-30T12:08:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: dump: 2023-10-30T12:08:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:45.032 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: _finish_auth 0 2023-10-30T12:08:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:08:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 1 -- 172.21.15.50:0/3867525060 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dec003cc0 con 0x7f7dfc098520 2023-10-30T12:08:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 5 monclient: authenticate success, global_id 114290 2023-10-30T12:08:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 10 monclient: _renew_subs 2023-10-30T12:08:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dfc0a30e0 con 0x7f7dfc098520 2023-10-30T12:08:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 1 -- 172.21.15.50:0/3867525060 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec006240 con 0x7f7dfc098520 2023-10-30T12:08:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df37fe640 10 monclient: dump: 2023-10-30T12:08:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:45.036 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7df9ffb640 4 set_mon_vals no callback set 2023-10-30T12:08:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 10 client.114290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 10 monclient: _renew_subs 2023-10-30T12:08:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dfc06fc50 con 0x7f7dfc098520 2023-10-30T12:08:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.995+0000 7f7e01725640 1 librados: init done 2023-10-30T12:08:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df37fe640 1 -- 172.21.15.50:0/3867525060 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7dec03a040 con 0x7f7dfc098520 2023-10-30T12:08:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df37fe640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7ddc0392a0 0x7f7ddc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df37fe640 1 -- 172.21.15.50:0/3867525060 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f7dec06ca90 con 0x7f7dfc098520 2023-10-30T12:08:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df37fe640 10 client.114290.objecter ms_dispatch 0x7f7dfc000b90 osd_map(281..281 src has 1..281) v4 2023-10-30T12:08:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df37fe640 3 client.114290.objecter handle_osd_map got epochs [281,281] > 0 2023-10-30T12:08:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df37fe640 3 client.114290.objecter handle_osd_map decoding full epoch 281 2023-10-30T12:08:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7e01725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7e01725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7e01725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dfc071ba0 con 0x7f7dfc098520 2023-10-30T12:08:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df37fe640 20 client.114290.objecter dump_active .. 0 homeless 2023-10-30T12:08:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df3fff640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7ddc0392a0 0x7f7ddc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.996+0000 7f7df3fff640 10 monclient: get_auth_request con 0x7f7ddc0392a0 auth_method 0 2023-10-30T12:08:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.997+0000 7f7df37fe640 10 client.114290.objecter ms_handle_connect 0x7f7ddc0392a0 2023-10-30T12:08:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:44.997+0000 7f7df3fff640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7ddc0392a0 0x7f7ddc03b760 secure :-1 s=READY pgs=4056 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc06f850 tx=0x7f7de4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.001+0000 7f7df37fe640 1 -- 172.21.15.50:0/3867525060 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7dec038070 con 0x7f7dfc098520 2023-10-30T12:08:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.001+0000 7f7df37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.001+0000 7f7df37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:45.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.246+0000 7f7e01725640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:45.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.246+0000 7f7e01725640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:45.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.246+0000 7f7e01725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:45.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.246+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7dfc08b210 con 0x7f7dfc098520 2023-10-30T12:08:45.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.246+0000 7f7df37fe640 1 -- 172.21.15.50:0/3867525060 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f7dec03b390 con 0x7f7dfc098520 2023-10-30T12:08:45.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.246+0000 7f7df37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:45.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.246+0000 7f7df37fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:45.249 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:45.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 10 librados: watch_flush enter 2023-10-30T12:08:45.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 10 librados: watch_flush exit 2023-10-30T12:08:45.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 20 client.114290.objecter shutdown clearing up homeless session... 2023-10-30T12:08:45.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 10 client.114290.objecter successfully canceled tick 2023-10-30T12:08:45.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7ddc0392a0 msgr2=0x7f7ddc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:45.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7ddc0392a0 0x7f7ddc03b760 secure :-1 s=READY pgs=4056 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc06f850 tx=0x7f7de4008040 comp rx=0 tx=0).stop 2023-10-30T12:08:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 10 monclient: shutdown 2023-10-30T12:08:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 msgr2=0x7f7dfc06e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc06e050 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0090b0 tx=0x7f7dec0046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 shutdown_connections 2023-10-30T12:08:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7ddc0392a0 0x7f7ddc03b760 unknown :-1 s=CLOSED pgs=4056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dfc132740 0x7f7dfc06ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7dfc098900 0x7f7dfc06e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 --2- 172.21.15.50:0/3867525060 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7dfc098520 0x7f7dfc06e050 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 >> 172.21.15.50:0/3867525060 conn(0x7f7dfc0a05c0 msgr2=0x7f7dfc131fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:45.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 shutdown_connections 2023-10-30T12:08:45.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 -- 172.21.15.50:0/3867525060 wait complete. 2023-10-30T12:08:45.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:45.248+0000 7f7e01725640 1 librados: shutdown 2023-10-30T12:08:48.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.377+0000 7fe425064640 10 monclient: get_monmap_and_config 2023-10-30T12:08:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.377+0000 7fe425064640 10 monclient: build_initial_monmap 2023-10-30T12:08:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.377+0000 7fe425064640 10 monclient: monmap: 2023-10-30T12:08:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:48.378766+0000 2023-10-30T12:08:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:48.378766+0000 2023-10-30T12:08:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:48.382 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:48.382 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:48.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.378+0000 7fe425064640 1 Processor -- start 2023-10-30T12:08:48.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.378+0000 7fe425064640 1 -- start start 2023-10-30T12:08:48.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.378+0000 7fe425064640 10 monclient: init 2023-10-30T12:08:48.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.378+0000 7fe425064640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:48.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.378+0000 7fe425064640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-10-30T12:08:48.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.378+0000 7fe425064640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe420098520 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:48.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.378+0000 7fe425064640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe420099010 0x7fe42009bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:48.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 10 monclient(hunting): picked mon.noname-c con 0x7fe420099010 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4201379f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:08:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42009ad30 con 0x7fe420098520 2023-10-30T12:08:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42009c460 con 0x7fe4201379f0 2023-10-30T12:08:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe41ed9d640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fe420098520 legacy=0x7fe420098900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42009b260 con 0x7fe420099010 2023-10-30T12:08:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe425064640 10 monclient(hunting): authenticate will time out at 1373.562095s 2023-10-30T12:08:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe420099010 0x7fe42009bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe420099010 0x7fe42009bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:36878/0 (socket says 172.21.15.50:36878) 2023-10-30T12:08:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 1 -- 172.21.15.50:0/1211522717 learned_addr learned my addr 172.21.15.50:0/1211522717 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): get_auth_request con 0x7fe420099010 auth_method 0 2023-10-30T12:08:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe41f59e640 1 --1- 172.21.15.50:0/1211522717 >> v1:172.21.15.118:6789/0 conn(0x7fe4201379f0 0x7fe420139de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:50666/0 (socket says 172.21.15.50:50666) 2023-10-30T12:08:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient(hunting): handle_auth_done global_id 94648 payload 274 2023-10-30T12:08:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 1 -- 172.21.15.50:0/1211522717 >> v1:172.21.15.118:6789/0 conn(0x7fe4201379f0 legacy=0x7fe420139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 1 -- 172.21.15.50:0/1211522717 >> v1:172.21.15.50:6789/0 conn(0x7fe420098520 legacy=0x7fe420098900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:48.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient: _finish_hunting 0 2023-10-30T12:08:48.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 1 monclient: found mon.noname-c 2023-10-30T12:08:48.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:48.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:08:48.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.379+0000 7fe417fff640 1 -- 172.21.15.50:0/1211522717 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe420072120 con 0x7fe420099010 2023-10-30T12:08:48.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe417fff640 1 --2- 172.21.15.50:0/1211522717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe420099010 0x7fe42009bf20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe40800bb10 tx=0x7fe40802e050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f430e5579b81de68 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:48.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 1 -- 172.21.15.50:0/1211522717 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40802ee00 con 0x7fe420099010 2023-10-30T12:08:48.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:48.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:48.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: dump: 2023-10-30T12:08:48.396 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:48.397 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:48.397 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.397 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.397 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:48.397 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:48.398 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:48.398 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:48.398 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:48.398 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:48.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: _finish_auth 0 2023-10-30T12:08:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: _check_auth_tickets 2023-10-30T12:08:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 1 -- 172.21.15.50:0/1211522717 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe408036440 con 0x7fe420099010 2023-10-30T12:08:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 1 -- 172.21.15.50:0/1211522717 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4080367d0 con 0x7fe420099010 2023-10-30T12:08:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe425064640 5 monclient: authenticate success, global_id 94648 2023-10-30T12:08:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe425064640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe425064640 4 set_mon_vals no callback set 2023-10-30T12:08:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.380+0000 7fe41e59c640 10 monclient: dump: 2023-10-30T12:08:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:48.402 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 10 monclient: shutdown 2023-10-30T12:08:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 1 -- 172.21.15.50:0/1211522717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe420099010 msgr2=0x7fe42009bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 1 --2- 172.21.15.50:0/1211522717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe420099010 0x7fe42009bf20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe40800bb10 tx=0x7fe40802e050 comp rx=0 tx=0).stop 2023-10-30T12:08:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 1 -- 172.21.15.50:0/1211522717 shutdown_connections 2023-10-30T12:08:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 1 --2- 172.21.15.50:0/1211522717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe420099010 0x7fe42009bf20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 1 -- 172.21.15.50:0/1211522717 >> 172.21.15.50:0/1211522717 conn(0x7fe4200a05c0 msgr2=0x7fe42009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 1 -- 172.21.15.50:0/1211522717 shutdown_connections 2023-10-30T12:08:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.381+0000 7fe425064640 1 -- 172.21.15.50:0/1211522717 wait complete. 2023-10-30T12:08:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient: build_initial_monmap 2023-10-30T12:08:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient: monmap: 2023-10-30T12:08:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:48.406 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 librados: starting msgr at 2023-10-30T12:08:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 librados: starting objecter 2023-10-30T12:08:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 Processor -- start 2023-10-30T12:08:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 -- start start 2023-10-30T12:08:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 librados: setting wanted keys 2023-10-30T12:08:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 librados: calling monclient init 2023-10-30T12:08:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient: init 2023-10-30T12:08:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:08:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe420098520 0x7fe4201c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): picked mon.noname-c con 0x7fe420098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe420099010 0x7fe4200961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): picked mon.noname-a con 0x7fe420099010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4201379f0 0x7fe4200966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): picked mon.noname-b con 0x7fe4201379f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42008e370 con 0x7fe420099010 2023-10-30T12:08:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42008cdf0 con 0x7fe4201379f0 2023-10-30T12:08:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe420071df0 con 0x7fe420098520 2023-10-30T12:08:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe425064640 10 monclient(hunting): authenticate will time out at 1373.565946s 2023-10-30T12:08:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe417fff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe420099010 msgr2=0x7fe4200961a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.382+0000 7fe417fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe420099010 0x7fe4200961a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe420098520 0x7fe4201c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe420098520 0x7fe4201c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53206/0 (socket says 172.21.15.50:53206) 2023-10-30T12:08:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 -- 172.21.15.50:0/793138385 learned_addr learned my addr 172.21.15.50:0/793138385 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): get_auth_request con 0x7fe420098520 auth_method 0 2023-10-30T12:08:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4201379f0 0x7fe4200966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 10 monclient(hunting): get_auth_request con 0x7fe4201379f0 auth_method 0 2023-10-30T12:08:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient(hunting): handle_auth_done global_id 114293 payload 824 2023-10-30T12:08:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 -- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4201379f0 msgr2=0x7fe4200966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4201379f0 0x7fe4200966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 -- 172.21.15.50:0/793138385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe420099010 msgr2=0x7fe4200961a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe420099010 0x7fe4200961a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:08:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 monclient: found mon.noname-c 2023-10-30T12:08:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 -- 172.21.15.50:0/793138385 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40800b770 con 0x7fe420098520 2023-10-30T12:08:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41ed9d640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe420098520 0x7fe4201c5fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe41000b530 tx=0x7fe41000b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 client.?.objecter ms_handle_connect 0x7fe420098520 2023-10-30T12:08:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 1 -- 172.21.15.50:0/793138385 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410003e10 con 0x7fe420098520 2023-10-30T12:08:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: dump: 2023-10-30T12:08:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:48.420 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: _finish_auth 0 2023-10-30T12:08:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: _check_auth_tickets 2023-10-30T12:08:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 1 -- 172.21.15.50:0/793138385 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe410005070 con 0x7fe420098520 2023-10-30T12:08:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 1 -- 172.21.15.50:0/793138385 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410005420 con 0x7fe420098520 2023-10-30T12:08:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 5 monclient: authenticate success, global_id 114293 2023-10-30T12:08:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 10 monclient: _renew_subs 2023-10-30T12:08:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4200a2fd0 con 0x7fe420098520 2023-10-30T12:08:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe41dd9b640 4 set_mon_vals no callback set 2023-10-30T12:08:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe4177fe640 10 monclient: dump: 2023-10-30T12:08:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:48.424 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 10 client.114293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 10 monclient: _renew_subs 2023-10-30T12:08:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42008c860 con 0x7fe420098520 2023-10-30T12:08:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.383+0000 7fe425064640 1 librados: init done 2023-10-30T12:08:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.384+0000 7fe4177fe640 1 -- 172.21.15.50:0/793138385 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe410005580 con 0x7fe420098520 2023-10-30T12:08:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe4177fe640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4000392a0 0x7fe40003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe425064640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe425064640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe425064640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42004e160 con 0x7fe420098520 2023-10-30T12:08:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe4177fe640 1 -- 172.21.15.50:0/793138385 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fe41004aac0 con 0x7fe420098520 2023-10-30T12:08:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe4177fe640 10 client.114293.objecter ms_dispatch 0x7fe420000b90 osd_map(281..281 src has 1..281) v4 2023-10-30T12:08:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe4177fe640 3 client.114293.objecter handle_osd_map got epochs [281,281] > 0 2023-10-30T12:08:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe4177fe640 3 client.114293.objecter handle_osd_map decoding full epoch 281 2023-10-30T12:08:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe4177fe640 20 client.114293.objecter dump_active .. 0 homeless 2023-10-30T12:08:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe417fff640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4000392a0 0x7fe40003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe417fff640 10 monclient: get_auth_request con 0x7fe4000392a0 auth_method 0 2023-10-30T12:08:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe4177fe640 10 client.114293.objecter ms_handle_connect 0x7fe4000392a0 2023-10-30T12:08:48.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.385+0000 7fe417fff640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4000392a0 0x7fe40003b760 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7fe408038000 tx=0x7fe408032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:48.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.389+0000 7fe4177fe640 1 -- 172.21.15.50:0/793138385 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe410020350 con 0x7fe420098520 2023-10-30T12:08:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.389+0000 7fe4177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.389+0000 7fe4177fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.633+0000 7fe425064640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.633+0000 7fe425064640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.633+0000 7fe425064640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.633+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe42008e590 con 0x7fe420098520 2023-10-30T12:08:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.633+0000 7fe4177fe640 1 -- 172.21.15.50:0/793138385 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fe410013070 con 0x7fe420098520 2023-10-30T12:08:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.633+0000 7fe4177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.633+0000 7fe4177fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:48.636 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 10 librados: watch_flush enter 2023-10-30T12:08:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 10 librados: watch_flush exit 2023-10-30T12:08:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 20 client.114293.objecter shutdown clearing up homeless session... 2023-10-30T12:08:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 10 client.114293.objecter successfully canceled tick 2023-10-30T12:08:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4000392a0 msgr2=0x7fe40003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4000392a0 0x7fe40003b760 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7fe408038000 tx=0x7fe408032000 comp rx=0 tx=0).stop 2023-10-30T12:08:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 10 monclient: shutdown 2023-10-30T12:08:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe420098520 msgr2=0x7fe4201c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe420098520 0x7fe4201c5fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe41000b530 tx=0x7fe41000b9f0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 shutdown_connections 2023-10-30T12:08:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4000392a0 0x7fe40003b760 unknown :-1 s=CLOSED pgs=4057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4201379f0 0x7fe4200966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe420099010 0x7fe4200961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 --2- 172.21.15.50:0/793138385 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe420098520 0x7fe4201c5fc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 >> 172.21.15.50:0/793138385 conn(0x7fe4200a05c0 msgr2=0x7fe42006bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 shutdown_connections 2023-10-30T12:08:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 -- 172.21.15.50:0/793138385 wait complete. 2023-10-30T12:08:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:48.635+0000 7fe425064640 1 librados: shutdown 2023-10-30T12:08:51.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:08:51.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.763+0000 7f1ce3690640 10 monclient: get_monmap_and_config 2023-10-30T12:08:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.763+0000 7f1ce3690640 10 monclient: build_initial_monmap 2023-10-30T12:08:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.763+0000 7f1ce3690640 10 monclient: monmap: 2023-10-30T12:08:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:51.764601+0000 2023-10-30T12:08:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:51.764601+0000 2023-10-30T12:08:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:51.768 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:51.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 1 Processor -- start 2023-10-30T12:08:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 1 -- start start 2023-10-30T12:08:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient: init 2023-10-30T12:08:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:08:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient(hunting): picked mon.noname-b con 0x7f1cdc098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cdc098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:08:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cdc1379f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:08:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cdc09ad30 con 0x7f1cdc098900 2023-10-30T12:08:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cdc09c460 con 0x7f1cdc1379f0 2023-10-30T12:08:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce0c2b640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f1cdc098900 legacy=0x7f1cdc098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc0993f0 con 0x7f1cdc098520 2023-10-30T12:08:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce3690640 10 monclient(hunting): authenticate will time out at 1376.947993s 2023-10-30T12:08:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.764+0000 7f1ce142c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53218/0 (socket says 172.21.15.50:53218) 2023-10-30T12:08:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 learned_addr learned my addr 172.21.15.50:0/2216801816 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cdc098520 auth_method 0 2023-10-30T12:08:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.765+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:51.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.965+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v1:172.21.15.50:6789/0 conn(0x7f1cdc098900 legacy=0x7f1cdc098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:51.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:51.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cdc098520 auth_method 0 2023-10-30T12:08:51.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:51.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:51.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:08:51.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:51.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:51.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:51.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:51.966+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:08:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.366+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v1:172.21.15.50:6789/0 conn(0x7f1cdc098900 legacy=0x7f1cdc098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cdc098520 auth_method 0 2023-10-30T12:08:52.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:52.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:52.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:08:52.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:52.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:52.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:52.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:52.367+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:08:53.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.167+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v1:172.21.15.50:6789/0 conn(0x7f1cdc098900 legacy=0x7f1cdc098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:08:53.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cdc098520 auth_method 0 2023-10-30T12:08:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:08:53.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:53.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:53.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:53.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:53.168+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:08:54.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): tick 2023-10-30T12:08:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:08:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 monclient(hunting): continuing hunt 2023-10-30T12:08:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:08:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc09bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cdc09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 >> v1:172.21.15.118:6789/0 conn(0x7f1cdc1379f0 legacy=0x7f1cdc139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:54.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 >> v1:172.21.15.50:6789/0 conn(0x7f1cdc098900 legacy=0x7f1cdc098ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:54.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient: _add_conns ranks=[4,3] 2023-10-30T12:08:54.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:54.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc00036f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:54.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:54.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc0006240 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:54.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:54.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cdc0993f0 con 0x7f1cc0006240 2023-10-30T12:08:54.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:54.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc0008da0 con 0x7f1cc00036f0 2023-10-30T12:08:54.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1cd2ffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:54.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 msgr2=0x7f1cc00086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:54.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:54.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:54.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cc00036f0 auth_method 0 2023-10-30T12:08:54.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:54.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:54.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:54.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:54.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:54.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:54.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.765+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 msgr2=0x7f1cc00086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:08:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cc00036f0 auth_method 0 2023-10-30T12:08:54.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:54.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:54.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:08:54.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:54.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:54.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:54.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:54.966+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:08:55.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 msgr2=0x7f1cc00086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:55.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:08:55.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:55.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cc00036f0 auth_method 0 2023-10-30T12:08:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:08:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:55.367+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:08:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 msgr2=0x7f1cc00086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:08:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cc00036f0 auth_method 0 2023-10-30T12:08:56.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:56.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:56.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce142c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:08:56.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:08:56.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.168+0000 7f1ce142c640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:08:56.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.169+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:08:56.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:56.169+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:08:57.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): tick 2023-10-30T12:08:57.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:08:57.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 monclient(hunting): continuing hunt 2023-10-30T12:08:57.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:08:57.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:57.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 msgr2=0x7f1cc00086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:57.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T12:08:57.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:57.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc00036f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:57.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 1 -- 172.21.15.50:0/2216801816 reap_dead start 2023-10-30T12:08:57.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:57.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc0006240 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:57.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cc000b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:57.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f1cdc098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:57.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:57.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cdc0993f0 con 0x7f1cc0006240 2023-10-30T12:08:57.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:57.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc0008da0 con 0x7f1cc00036f0 2023-10-30T12:08:57.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:57.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 1 -- 172.21.15.50:0/2216801816 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc000b6a0 con 0x7f1cdc098520 2023-10-30T12:08:57.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1cd2ffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:57.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 msgr2=0x7f1cc00086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:57.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:57.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cc000b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:57.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 10 monclient(hunting): get_auth_request con 0x7f1cdc098520 auth_method 0 2023-10-30T12:08:57.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:57.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:57.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:57.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:57.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:57.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce1c2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:57.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce142c640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:57.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cc00036f0 auth_method 0 2023-10-30T12:08:57.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:57.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:57.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.765+0000 7f1ce142c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:57.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce142c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:57.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce142c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:57.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce142c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:57.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 10 monclient(hunting): handle_auth_done global_id 114296 payload 274 2023-10-30T12:08:57.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 msgr2=0x7f1cc0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:57.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 1 -- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 msgr2=0x7f1cc00086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:57.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 10 monclient: _finish_hunting 0 2023-10-30T12:08:57.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 1 monclient: found mon.noname-b 2023-10-30T12:08:57.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:57.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:57.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 1 -- 172.21.15.50:0/2216801816 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc09b260 con 0x7f1cdc098520 2023-10-30T12:08:57.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.766+0000 7f1ce1c2d640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cc000b160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09d960 tx=0x7f1cd402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500bf246c06fb63b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:57.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 1 -- 172.21.15.50:0/2216801816 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4007070 con 0x7f1cdc098520 2023-10-30T12:08:57.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:57.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:57.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: dump: 2023-10-30T12:08:57.779 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:57.779 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:57.780 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.780 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.780 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:57.780 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:57.780 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:57.781 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:57.781 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:57.781 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:57.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: _finish_auth 0 2023-10-30T12:08:57.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:08:57.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:57.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 1 -- 172.21.15.50:0/2216801816 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd40071d0 con 0x7f1cdc098520 2023-10-30T12:08:57.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:57.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 1 -- 172.21.15.50:0/2216801816 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4007580 con 0x7f1cdc098520 2023-10-30T12:08:57.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1ce3690640 5 monclient: authenticate success, global_id 114296 2023-10-30T12:08:57.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1ce3690640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:57.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:57.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1ce3690640 4 set_mon_vals no callback set 2023-10-30T12:08:57.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:57.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.767+0000 7f1cd3fff640 10 monclient: dump: 2023-10-30T12:08:57.783 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:57.785 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:57.785 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:57.785 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:57.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 10 monclient: shutdown 2023-10-30T12:08:57.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:57.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 -- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 msgr2=0x7f1cc000b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cc000b160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc09d960 tx=0x7f1cd402b790 comp rx=0 tx=0).stop 2023-10-30T12:08:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 -- 172.21.15.50:0/2216801816 shutdown_connections 2023-10-30T12:08:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 --2- 172.21.15.50:0/2216801816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cdc098520 0x7f1cc000b160 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.50:3300/0 conn(0x7f1cc0006240 0x7f1cc00086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 --2- 172.21.15.50:0/2216801816 >> v2:172.21.15.118:3300/0 conn(0x7f1cc00036f0 0x7f1cc0005b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 -- 172.21.15.50:0/2216801816 >> 172.21.15.50:0/2216801816 conn(0x7f1cdc0a05c0 msgr2=0x7f1cdc09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:57.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 -- 172.21.15.50:0/2216801816 shutdown_connections 2023-10-30T12:08:57.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.768+0000 7f1ce3690640 1 -- 172.21.15.50:0/2216801816 wait complete. 2023-10-30T12:08:57.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 10 monclient: build_initial_monmap 2023-10-30T12:08:57.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 10 monclient: monmap: 2023-10-30T12:08:57.795 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:57.795 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:57.795 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:57.795 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:57.795 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:57.796 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:57.796 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:57.796 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:57.796 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:57.796 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:57.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 1 librados: starting msgr at 2023-10-30T12:08:57.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 1 librados: starting objecter 2023-10-30T12:08:57.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 1 Processor -- start 2023-10-30T12:08:57.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 1 -- start start 2023-10-30T12:08:57.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 1 librados: setting wanted keys 2023-10-30T12:08:57.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 1 librados: calling monclient init 2023-10-30T12:08:57.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.769+0000 7f1ce3690640 10 monclient: init 2023-10-30T12:08:57.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:57.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:08:57.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc00036f0 0x7f1cdc06e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:57.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc00036f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:57.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc0006240 0x7f1cdc06eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:57.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc0006240 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:57.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cdc098520 0x7f1cdc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:57.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): picked mon.noname-a con 0x7f1cdc098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:57.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:57.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc08b800 con 0x7f1cdc098520 2023-10-30T12:08:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc071df0 con 0x7f1cc00036f0 2023-10-30T12:08:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc0705b0 con 0x7f1cc0006240 2023-10-30T12:08:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce3690640 10 monclient(hunting): authenticate will time out at 1382.953357s 2023-10-30T12:08:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce1c2d640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce1c2d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cdc098520 0x7f1cdc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc0006240 0x7f1cdc06eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc0006240 0x7f1cdc06eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59372/0 (socket says 172.21.15.50:59372) 2023-10-30T12:08:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/3870405826 learned_addr learned my addr 172.21.15.50:0/3870405826 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:57.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): get_auth_request con 0x7f1cc0006240 auth_method 0 2023-10-30T12:08:57.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:57.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:57.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:57.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc00036f0 0x7f1cdc06e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:57.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:57.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:57.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:57.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 10 monclient(hunting): get_auth_request con 0x7f1cc00036f0 auth_method 0 2023-10-30T12:08:57.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:57.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:57.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:57.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:57.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:57.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce142c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:57.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient(hunting): handle_auth_done global_id 114299 payload 824 2023-10-30T12:08:57.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc00036f0 msgr2=0x7f1cdc06e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:57.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc00036f0 0x7f1cdc06e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/3870405826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cdc098520 msgr2=0x7f1cdc06f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:57.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cdc098520 0x7f1cdc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:57.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient: _finish_hunting 0 2023-10-30T12:08:57.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 monclient: found mon.noname-c 2023-10-30T12:08:57.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:57.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:57.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.770+0000 7f1ce0c2b640 1 -- 172.21.15.50:0/3870405826 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc09c460 con 0x7f1cc0006240 2023-10-30T12:08:57.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce0c2b640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc0006240 0x7f1cdc06eb40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00b760 tx=0x7f1ccc00bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:57.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 client.?.objecter ms_handle_connect 0x7f1cc0006240 2023-10-30T12:08:57.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:57.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 1 -- 172.21.15.50:0/3870405826 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00c930 con 0x7f1cc0006240 2023-10-30T12:08:57.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:57.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: dump: 2023-10-30T12:08:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:57.809 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:57.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:08:57.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:08:57.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:57.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 1 -- 172.21.15.50:0/3870405826 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc00ca90 con 0x7f1cc0006240 2023-10-30T12:08:57.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 5 monclient: authenticate success, global_id 114299 2023-10-30T12:08:57.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 10 monclient: _renew_subs 2023-10-30T12:08:57.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:57.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc0a3040 con 0x7f1cc0006240 2023-10-30T12:08:57.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:57.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 1 -- 172.21.15.50:0/3870405826 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00ce20 con 0x7f1cc0006240 2023-10-30T12:08:57.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:57.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:57.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd1ffb640 10 monclient: dump: 2023-10-30T12:08:57.812 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:57.812 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:57.812 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.813 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:57.813 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:57.813 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:57.813 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:57.813 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:57.814 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:57.814 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:57.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1cd37fe640 4 set_mon_vals no callback set 2023-10-30T12:08:57.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 10 client.114299.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:57.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 10 monclient: _renew_subs 2023-10-30T12:08:57.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:57.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc070030 con 0x7f1cc0006240 2023-10-30T12:08:57.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.771+0000 7f1ce3690640 1 librados: init done 2023-10-30T12:08:57.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.772+0000 7f1cd1ffb640 1 -- 172.21.15.50:0/3870405826 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1ccc012020 con 0x7f1cc0006240 2023-10-30T12:08:57.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1cd1ffb640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1cc003a110 0x7f1cc003c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1cd1ffb640 1 -- 172.21.15.50:0/3870405826 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 7100+0+0 (secure 0 0 0) 0x7f1ccc04a440 con 0x7f1cc0006240 2023-10-30T12:08:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1cd1ffb640 10 client.114299.objecter ms_dispatch 0x7f1cdc000b90 osd_map(283..283 src has 1..283) v4 2023-10-30T12:08:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1cd1ffb640 3 client.114299.objecter handle_osd_map got epochs [283,283] > 0 2023-10-30T12:08:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1cd1ffb640 3 client.114299.objecter handle_osd_map decoding full epoch 283 2023-10-30T12:08:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1ce3690640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1cd1ffb640 20 client.114299.objecter dump_active .. 0 homeless 2023-10-30T12:08:57.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1ce3690640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:57.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1ce3690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:57.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc08b2c0 con 0x7f1cc0006240 2023-10-30T12:08:57.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1ce142c640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1cc003a110 0x7f1cc003c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:57.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1ce142c640 10 monclient: get_auth_request con 0x7f1cc003a110 auth_method 0 2023-10-30T12:08:57.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1cd1ffb640 10 client.114299.objecter ms_handle_connect 0x7f1cc003a110 2023-10-30T12:08:57.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.773+0000 7f1ce142c640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1cc003a110 0x7f1cc003c5d0 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8006f60 tx=0x7f1cd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:57.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.778+0000 7f1cd1ffb640 1 -- 172.21.15.50:0/3870405826 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ccc011280 con 0x7f1cc0006240 2023-10-30T12:08:57.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.778+0000 7f1cd1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:57.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:57.778+0000 7f1cd1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:58.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.025+0000 7f1ce3690640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:08:58.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.025+0000 7f1ce3690640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.025+0000 7f1ce3690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.025+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1cdc04e160 con 0x7f1cc0006240 2023-10-30T12:08:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.025+0000 7f1cd1ffb640 1 -- 172.21.15.50:0/3870405826 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f1ccc0122d0 con 0x7f1cc0006240 2023-10-30T12:08:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.025+0000 7f1cd1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:08:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.025+0000 7f1cd1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:58.028 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":80,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:08:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 10 librados: watch_flush enter 2023-10-30T12:08:58.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 10 librados: watch_flush exit 2023-10-30T12:08:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 20 client.114299.objecter shutdown clearing up homeless session... 2023-10-30T12:08:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 10 client.114299.objecter successfully canceled tick 2023-10-30T12:08:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1cc003a110 msgr2=0x7f1cc003c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1cc003a110 0x7f1cc003c5d0 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8006f60 tx=0x7f1cd8008040 comp rx=0 tx=0).stop 2023-10-30T12:08:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 10 monclient: shutdown 2023-10-30T12:08:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc0006240 msgr2=0x7f1cdc06eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc0006240 0x7f1cdc06eb40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00b760 tx=0x7f1ccc00bc20 comp rx=0 tx=0).stop 2023-10-30T12:08:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 shutdown_connections 2023-10-30T12:08:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1cc003a110 0x7f1cc003c5d0 unknown :-1 s=CLOSED pgs=4058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cdc098520 0x7f1cdc06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc0006240 0x7f1cdc06eb40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 --2- 172.21.15.50:0/3870405826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc00036f0 0x7f1cdc06e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 >> 172.21.15.50:0/3870405826 conn(0x7f1cdc0a05c0 msgr2=0x7f1cdc139de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 shutdown_connections 2023-10-30T12:08:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 -- 172.21.15.50:0/3870405826 wait complete. 2023-10-30T12:08:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.027+0000 7f1ce3690640 1 librados: shutdown 2023-10-30T12:08:58.039 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:08:58.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:08:58.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:08:58.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.144+0000 7fd053ebb640 10 monclient: get_monmap_and_config 2023-10-30T12:08:58.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.144+0000 7fd053ebb640 10 monclient: build_initial_monmap 2023-10-30T12:08:58.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.144+0000 7fd053ebb640 10 monclient: monmap: 2023-10-30T12:08:58.146 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:58.146 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.146 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:58.145318+0000 2023-10-30T12:08:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:58.145318+0000 2023-10-30T12:08:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:58.147 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:58.148 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:58.148 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:58.148 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:58.148 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.144+0000 7fd053ebb640 1 Processor -- start 2023-10-30T12:08:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 1 -- start start 2023-10-30T12:08:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient: init 2023-10-30T12:08:58.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:58.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:08:58.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c09bb70 0x7fd04c09bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): picked mon.noname-b con 0x7fd04c09bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:58.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd04c09d3f0 0x7fd04c09c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): picked mon.noname-c con 0x7fd04c09d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:58.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd04c096080 0x7fd04c0964f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): picked mon.noname-a con 0x7fd04c096080 addr v2:172.21.15.50:3300/0 2023-10-30T12:08:58.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c098170 con 0x7fd04c096080 2023-10-30T12:08:58.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c09c9d0 con 0x7fd04c09d3f0 2023-10-30T12:08:58.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c09cf60 con 0x7fd04c09bb70 2023-10-30T12:08:58.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:58.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd053ebb640 10 monclient(hunting): authenticate will time out at 1383.328512s 2023-10-30T12:08:58.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd052458640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fd04c096080 msgr2=0x7fd04c0964f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:58.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd052458640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd04c096080 0x7fd04c0964f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:58.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c09bb70 0x7fd04c09bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c09bb70 0x7fd04c09bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59376/0 (socket says 172.21.15.50:59376) 2023-10-30T12:08:58.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 -- 172.21.15.50:0/340475425 learned_addr learned my addr 172.21.15.50:0/340475425 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:58.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): get_auth_request con 0x7fd04c09bb70 auth_method 0 2023-10-30T12:08:58.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient(hunting): handle_auth_done global_id 114302 payload 274 2023-10-30T12:08:58.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 -- 172.21.15.50:0/340475425 >> v2:172.21.15.118:3300/0 conn(0x7fd04c09d3f0 msgr2=0x7fd04c09c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:58.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051456640 1 --2- 172.21.15.50:0/340475425 >> v2:172.21.15.118:3300/0 conn(0x7fd04c09d3f0 0x7fd04c09c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 --2- 172.21.15.50:0/340475425 >> v2:172.21.15.118:3300/0 conn(0x7fd04c09d3f0 0x7fd04c09c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 -- 172.21.15.50:0/340475425 >> v2:172.21.15.50:3300/0 conn(0x7fd04c096080 msgr2=0x7fd04c0964f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:08:58.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 --2- 172.21.15.50:0/340475425 >> v2:172.21.15.50:3300/0 conn(0x7fd04c096080 0x7fd04c0964f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient: _finish_hunting 0 2023-10-30T12:08:58.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 monclient: found mon.noname-b 2023-10-30T12:08:58.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:58.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.145+0000 7fd051c57640 1 -- 172.21.15.50:0/340475425 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c097b20 con 0x7fd04c09bb70 2023-10-30T12:08:58.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd051c57640 1 --2- 172.21.15.50:0/340475425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c09bb70 0x7fd04c09bf50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd04c06b050 tx=0x7fd04802b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57750fb2aee0830d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 1 -- 172.21.15.50:0/340475425 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048004230 con 0x7fd04c09bb70 2023-10-30T12:08:58.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:58.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: dump: 2023-10-30T12:08:58.166 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.166 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.166 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.166 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.167 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.167 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.167 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.167 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.167 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.168 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: _finish_auth 0 2023-10-30T12:08:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: _check_auth_tickets 2023-10-30T12:08:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 1 -- 172.21.15.50:0/340475425 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd048004390 con 0x7fd04c09bb70 2023-10-30T12:08:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd053ebb640 5 monclient: authenticate success, global_id 114302 2023-10-30T12:08:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd053ebb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd053ebb640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 1 -- 172.21.15.50:0/340475425 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048003810 con 0x7fd04c09bb70 2023-10-30T12:08:58.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd053ebb640 4 set_mon_vals no callback set 2023-10-30T12:08:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.146+0000 7fd050c55640 10 monclient: dump: 2023-10-30T12:08:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.171 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.172 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 10 monclient: shutdown 2023-10-30T12:08:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 -- 172.21.15.50:0/340475425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c09bb70 msgr2=0x7fd04c09bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 --2- 172.21.15.50:0/340475425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c09bb70 0x7fd04c09bf50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd04c06b050 tx=0x7fd04802b7f0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 -- 172.21.15.50:0/340475425 shutdown_connections 2023-10-30T12:08:58.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 --2- 172.21.15.50:0/340475425 >> v2:172.21.15.50:3300/0 conn(0x7fd04c096080 0x7fd04c0964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 --2- 172.21.15.50:0/340475425 >> v2:172.21.15.118:3300/0 conn(0x7fd04c09d3f0 0x7fd04c09c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 --2- 172.21.15.50:0/340475425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c09bb70 0x7fd04c09bf50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 -- 172.21.15.50:0/340475425 >> 172.21.15.50:0/340475425 conn(0x7fd04c0a3980 msgr2=0x7fd04c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:58.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 -- 172.21.15.50:0/340475425 shutdown_connections 2023-10-30T12:08:58.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.147+0000 7fd053ebb640 1 -- 172.21.15.50:0/340475425 wait complete. 2023-10-30T12:08:58.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient: build_initial_monmap 2023-10-30T12:08:58.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient: monmap: 2023-10-30T12:08:58.175 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:58.175 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:58.175 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:58.175 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:58.175 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:58.176 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.176 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:58.176 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:58.176 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:58.176 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 librados: starting msgr at 2023-10-30T12:08:58.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 librados: starting objecter 2023-10-30T12:08:58.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 Processor -- start 2023-10-30T12:08:58.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 -- start start 2023-10-30T12:08:58.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 librados: setting wanted keys 2023-10-30T12:08:58.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 librados: calling monclient init 2023-10-30T12:08:58.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient: init 2023-10-30T12:08:58.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:58.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:08:58.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c096080 0x7fd04c1ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): picked mon.noname-c con 0x7fd04c096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:58.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd04c09bb70 0x7fd04c1cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): picked mon.noname-a con 0x7fd04c09bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:58.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd04c09d3f0 0x7fd04c1cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): picked mon.noname-b con 0x7fd04c09d3f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:58.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0971b0 con 0x7fd04c09bb70 2023-10-30T12:08:58.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c06e3c0 con 0x7fd04c09d3f0 2023-10-30T12:08:58.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c13b760 con 0x7fd04c096080 2023-10-30T12:08:58.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:58.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd051456640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd04c09bb70 msgr2=0x7fd04c1cae70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:58.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd053ebb640 10 monclient(hunting): authenticate will time out at 1383.331960s 2023-10-30T12:08:58.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd051456640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd04c09bb70 0x7fd04c1cae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:58.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.148+0000 7fd051c57640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c096080 0x7fd04c1ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c096080 0x7fd04c1ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59388/0 (socket says 172.21.15.50:59388) 2023-10-30T12:08:58.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 -- 172.21.15.50:0/3640855002 learned_addr learned my addr 172.21.15.50:0/3640855002 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:58.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): get_auth_request con 0x7fd04c096080 auth_method 0 2023-10-30T12:08:58.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient(hunting): handle_auth_done global_id 114305 payload 824 2023-10-30T12:08:58.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd052458640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd04c09d3f0 0x7fd04c1cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd04c09d3f0 msgr2=0x7fd04c1cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd04c09d3f0 0x7fd04c1cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd04c09bb70 msgr2=0x7fd04c1cae70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:58.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd04c09bb70 0x7fd04c1cae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient: _finish_hunting 0 2023-10-30T12:08:58.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 monclient: found mon.noname-c 2023-10-30T12:08:58.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:58.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 -- 172.21.15.50:0/3640855002 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd048009460 con 0x7fd04c096080 2023-10-30T12:08:58.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd051c57640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c096080 0x7fd04c1ca930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd048009940 tx=0x7fd0480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 client.?.objecter ms_handle_connect 0x7fd04c096080 2023-10-30T12:08:58.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:58.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048004500 con 0x7fd04c096080 2023-10-30T12:08:58.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:58.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: dump: 2023-10-30T12:08:58.187 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.187 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.187 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.187 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.187 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.187 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.188 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.188 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.188 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.188 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: _finish_auth 0 2023-10-30T12:08:58.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: _check_auth_tickets 2023-10-30T12:08:58.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:58.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 5 monclient: authenticate success, global_id 114305 2023-10-30T12:08:58.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd048003c50 con 0x7fd04c096080 2023-10-30T12:08:58.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 10 monclient: _renew_subs 2023-10-30T12:08:58.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c0976e0 con 0x7fd04c096080 2023-10-30T12:08:58.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:58.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048006040 con 0x7fd04c096080 2023-10-30T12:08:58.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd043fff640 4 set_mon_vals no callback set 2023-10-30T12:08:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd0427fc640 10 monclient: dump: 2023-10-30T12:08:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.193 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.193 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 10 client.114305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:58.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 10 monclient: _renew_subs 2023-10-30T12:08:58.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c06f590 con 0x7fd04c096080 2023-10-30T12:08:58.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.149+0000 7fd053ebb640 1 librados: init done 2023-10-30T12:08:58.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd053ebb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:58.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd053ebb640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c13ab20 0x7fd04c060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c073ff0 con 0x7fd04c13ab20 2023-10-30T12:08:58.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd04803a070 con 0x7fd04c096080 2023-10-30T12:08:58.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c13ab20 0x7fd04c060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient: get_auth_request con 0x7fd04c13ab20 auth_method 0 2023-10-30T12:08:58.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0300392a0 0x7fd03003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 10 monclient(hunting): handle_auth_done global_id 114308 payload 824 2023-10-30T12:08:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 1 -- 172.21.15.50:0/3640855002 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd04c097560 con 0x7fd04c13ab20 2023-10-30T12:08:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 7100+0+0 (secure 0 0 0) 0x7fd04806cd20 con 0x7fd04c096080 2023-10-30T12:08:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 10 client.114305.objecter ms_dispatch 0x7fd04c000b90 osd_map(283..283 src has 1..283) v4 2023-10-30T12:08:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 3 client.114305.objecter handle_osd_map got epochs [283,283] > 0 2023-10-30T12:08:58.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 3 client.114305.objecter handle_osd_map decoding full epoch 283 2023-10-30T12:08:58.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 20 client.114305.objecter dump_active .. 0 homeless 2023-10-30T12:08:58.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 10 client.114305.objecter ms_handle_connect 0x7fd04c13ab20 2023-10-30T12:08:58.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd0427fc640 10 client.114305.objecter resend_mon_ops 2023-10-30T12:08:58.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd051456640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c13ab20 0x7fd04c060fc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd04c06ed50 tx=0x7fd03c00cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd052458640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0300392a0 0x7fd03003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.150+0000 7fd052458640 10 monclient: get_auth_request con 0x7fd0300392a0 auth_method 0 2023-10-30T12:08:58.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c004220 con 0x7fd04c13ab20 2023-10-30T12:08:58.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:58.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 10 monclient: dump: 2023-10-30T12:08:58.199 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.200 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.201 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fd03c004380 con 0x7fd04c13ab20 2023-10-30T12:08:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c13ab20 msgr2=0x7fd04c060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c13ab20 0x7fd04c060fc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd04c06ed50 tx=0x7fd03c00cb80 comp rx=0 tx=0).stop 2023-10-30T12:08:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd052458640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0300392a0 0x7fd03003b760 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7fd044003cb0 tx=0x7fd0440071c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c13ab20 msgr2=0x7fd04c060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:58.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c13ab20 0x7fd04c060fc0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd04c06ed50 tx=0x7fd03c00cb80 comp rx=0 tx=0).stop 2023-10-30T12:08:58.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.151+0000 7fd0427fc640 10 client.114305.objecter ms_handle_connect 0x7fd0300392a0 2023-10-30T12:08:58.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd053ebb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:58.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd053ebb640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c061500 0x7fd04c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd053ebb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c097e00 con 0x7fd04c061500 2023-10-30T12:08:58.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c061500 0x7fd04c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient: get_auth_request con 0x7fd04c061500 auth_method 0 2023-10-30T12:08:58.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 10 monclient(hunting): handle_auth_done global_id 114311 payload 824 2023-10-30T12:08:58.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.157+0000 7fd051456640 1 -- 172.21.15.50:0/3640855002 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd04c04e160 con 0x7fd04c061500 2023-10-30T12:08:58.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 10 client.114305.objecter ms_handle_connect 0x7fd04c061500 2023-10-30T12:08:58.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 10 client.114305.objecter resend_mon_ops 2023-10-30T12:08:58.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd051456640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c061500 0x7fd04c02b430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd03c000c00 tx=0x7fd03c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c003df0 con 0x7fd04c061500 2023-10-30T12:08:58.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:58.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 10 monclient: dump: 2023-10-30T12:08:58.209 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.209 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.209 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.209 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.209 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.210 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.210 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.210 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.210 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.210 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7fd03c0113f0 con 0x7fd04c061500 2023-10-30T12:08:58.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:58.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:58.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c061500 msgr2=0x7fd04c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c061500 0x7fd04c02b430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd03c000c00 tx=0x7fd03c004730 comp rx=0 tx=0).stop 2023-10-30T12:08:58.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c061500 msgr2=0x7fd04c02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:58.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd0427fc640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c061500 0x7fd04c02b430 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd03c000c00 tx=0x7fd03c004730 comp rx=0 tx=0).stop 2023-10-30T12:08:58.212 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:58.212 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:58.212 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:58.213 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:08:58.213 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 80, 2023-10-30T12:08:58.213 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:58.213 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:58.213 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:58.214 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.214 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:08:58.214 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:58.214 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:58.214 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:58.215 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:58.215 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:58.215 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:58.215 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:58.215 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:58.215 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:58.216 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:58.216 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:58.216 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:58.216 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:58.216 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.217 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:58.217 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:58.217 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:58.217 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:58.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:58.217 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:58.218 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:58.218 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:58.218 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:58.218 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:58.218 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:58.219 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:58.219 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.219 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:58.219 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:58.220 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:58.220 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:58.220 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:58.220 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:58.220 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:58.220 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:58.221 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:58.221 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:58.221 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:58.221 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:58.221 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:58.222 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:58.222 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:58.222 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:58.222 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:58.222 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:58.223 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:58.223 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:58.223 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:58.223 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:58.223 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:58.223 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:58.224 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:58.224 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:58.224 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:58.224 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.224 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:58.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.225 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:58.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.225 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:58.225 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:58.225 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:58.226 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:58.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.226 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:58.226 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:58.226 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.227 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:58.227 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:58.227 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:58.228 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:58.229 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:58.229 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:58.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.229 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:58.230 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:58.230 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:58.230 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:58.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.230 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:58.231 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:58.231 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.231 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:58.232 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:58.232 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.232 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.233 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:58.233 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:58.233 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:58.233 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:58.233 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:58.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.234 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:58.234 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:58.234 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:58.234 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:58.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.235 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:58.235 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:58.235 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:58.236 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:58.236 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.236 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.237 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.237 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:58.237 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:58.237 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:58.238 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:58.238 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:58.238 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.238 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.239 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:58.239 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:58.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.239 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:58.239 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.241 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:58.241 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:58.241 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:58.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.241 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.242 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:58.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.242 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:58.242 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:58.242 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:08:58.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.243 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.243 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:58.243 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:58.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 10 librados: watch_flush enter 2023-10-30T12:08:58.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 10 librados: watch_flush exit 2023-10-30T12:08:58.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 20 client.114305.objecter shutdown clearing up homeless session... 2023-10-30T12:08:58.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 10 client.114305.objecter successfully canceled tick 2023-10-30T12:08:58.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0300392a0 msgr2=0x7fd03003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd0300392a0 0x7fd03003b760 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7fd044003cb0 tx=0x7fd0440071c0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 10 monclient: shutdown 2023-10-30T12:08:58.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:58.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c096080 msgr2=0x7fd04c1ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 --2- 172.21.15.50:0/3640855002 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd04c096080 0x7fd04c1ca930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd048009940 tx=0x7fd0480046c0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd052458640 1 -- 172.21.15.50:0/3640855002 reap_dead start 2023-10-30T12:08:58.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd052458640 1 -- 172.21.15.50:0/3640855002 reap_dead start 2023-10-30T12:08:58.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 shutdown_connections 2023-10-30T12:08:58.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 >> 172.21.15.50:0/3640855002 conn(0x7fd04c0a3980 msgr2=0x7fd04c138e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:58.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 shutdown_connections 2023-10-30T12:08:58.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 -- 172.21.15.50:0/3640855002 wait complete. 2023-10-30T12:08:58.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.158+0000 7fd053ebb640 1 librados: shutdown 2023-10-30T12:08:58.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:08:58.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:08:58.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:08:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.354+0000 7f025a70f640 10 monclient: get_monmap_and_config 2023-10-30T12:08:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.354+0000 7f025a70f640 10 monclient: build_initial_monmap 2023-10-30T12:08:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.354+0000 7f025a70f640 10 monclient: monmap: 2023-10-30T12:08:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:08:58.355743+0000 2023-10-30T12:08:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:08:58.355743+0000 2023-10-30T12:08:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:08:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:08:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:08:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:08:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:08:58.358 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 1 Processor -- start 2023-10-30T12:08:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 1 -- start start 2023-10-30T12:08:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 10 monclient: init 2023-10-30T12:08:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:08:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f02540a2ae0 0x7f02540a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 10 monclient(hunting): picked mon.noname-c con 0x7f02540a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:08:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02540a2ec0 addr v1:172.21.15.118:6789/0 2023-10-30T12:08:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0254135df0 0x7f025413da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.355+0000 7f025a70f640 10 monclient(hunting): picked mon.noname-b con 0x7f0254135df0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f025a70f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02540a4190 con 0x7f02540a2ec0 2023-10-30T12:08:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f025a70f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f025a70f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02540a2990 con 0x7f02540a2ae0 2023-10-30T12:08:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f025a70f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f025a70f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02540a1630 con 0x7f0254135df0 2023-10-30T12:08:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f025a70f640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f025a70f640 10 monclient(hunting): authenticate will time out at 1383.539036s 2023-10-30T12:08:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0254135df0 0x7f025413da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0254135df0 0x7f025413da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59410/0 (socket says 172.21.15.50:59410) 2023-10-30T12:08:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 -- 172.21.15.50:0/100842183 learned_addr learned my addr 172.21.15.50:0/100842183 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): get_auth_request con 0x7f0254135df0 auth_method 0 2023-10-30T12:08:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 1 --2- 172.21.15.50:0/100842183 >> v2:172.21.15.118:3300/0 conn(0x7f02540a2ae0 0x7f02540a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 10 monclient(hunting): get_auth_request con 0x7f02540a2ae0 auth_method 0 2023-10-30T12:08:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient(hunting): handle_auth_done global_id 114314 payload 274 2023-10-30T12:08:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 -- 172.21.15.50:0/100842183 >> v2:172.21.15.118:3300/0 conn(0x7f02540a2ae0 msgr2=0x7f02540a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:08:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 --2- 172.21.15.50:0/100842183 >> v2:172.21.15.118:3300/0 conn(0x7f02540a2ae0 0x7f02540a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 -- 172.21.15.50:0/100842183 >> v1:172.21.15.118:6789/0 conn(0x7f02540a2ec0 legacy=0x7f02541356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient: _finish_hunting 0 2023-10-30T12:08:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 monclient: found mon.noname-b 2023-10-30T12:08:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.356+0000 7f0258cac640 1 -- 172.21.15.50:0/100842183 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025409a2e0 con 0x7f0254135df0 2023-10-30T12:08:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0258cac640 1 --2- 172.21.15.50:0/100842183 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0254135df0 0x7f025413da20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f024c009e50 tx=0x7f024c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd224411b4c0fab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 1 -- 172.21.15.50:0/100842183 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c02c9e0 con 0x7f0254135df0 2023-10-30T12:08:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:08:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: dump: 2023-10-30T12:08:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.380 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: _finish_auth 0 2023-10-30T12:08:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:08:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 5 monclient: authenticate success, global_id 114314 2023-10-30T12:08:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:08:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 1 -- 172.21.15.50:0/100842183 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c02cb40 con 0x7f0254135df0 2023-10-30T12:08:58.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:58.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 10 monclient: get_monmap_and_config success 2023-10-30T12:08:58.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 4 set_mon_vals no callback set 2023-10-30T12:08:58.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 1 -- 172.21.15.50:0/100842183 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c0343f0 con 0x7f0254135df0 2023-10-30T12:08:58.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f0252ffd640 10 monclient: dump: 2023-10-30T12:08:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.385 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 10 monclient: shutdown 2023-10-30T12:08:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 1 -- 172.21.15.50:0/100842183 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0254135df0 msgr2=0x7f025413da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 1 --2- 172.21.15.50:0/100842183 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0254135df0 0x7f025413da20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f024c009e50 tx=0x7f024c02be40 comp rx=0 tx=0).stop 2023-10-30T12:08:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 1 -- 172.21.15.50:0/100842183 shutdown_connections 2023-10-30T12:08:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 1 --2- 172.21.15.50:0/100842183 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0254135df0 0x7f025413da20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 1 --2- 172.21.15.50:0/100842183 >> v2:172.21.15.118:3300/0 conn(0x7f02540a2ae0 0x7f02540a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.357+0000 7f025a70f640 1 -- 172.21.15.50:0/100842183 >> 172.21.15.50:0/100842183 conn(0x7f02540a80b0 msgr2=0x7f02540a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.358+0000 7f025a70f640 1 -- 172.21.15.50:0/100842183 shutdown_connections 2023-10-30T12:08:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.358+0000 7f025a70f640 1 -- 172.21.15.50:0/100842183 wait complete. 2023-10-30T12:08:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.358+0000 7f025a70f640 10 monclient: build_initial_monmap 2023-10-30T12:08:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.358+0000 7f025a70f640 10 monclient: monmap: 2023-10-30T12:08:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:08:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:08:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:08:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:08:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:08:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:08:58.388 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:08:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:08:58.389 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.358+0000 7f025a70f640 1 librados: starting msgr at 2023-10-30T12:08:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.358+0000 7f025a70f640 1 librados: starting objecter 2023-10-30T12:08:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.358+0000 7f025a70f640 1 Processor -- start 2023-10-30T12:08:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 -- start start 2023-10-30T12:08:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 librados: setting wanted keys 2023-10-30T12:08:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 librados: calling monclient init 2023-10-30T12:08:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient: init 2023-10-30T12:08:58.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient: _reopen_session rank -1 2023-10-30T12:08:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:08:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02540a2ae0 0x7f02541ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): picked mon.noname-c con 0x7f02540a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:08:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02540a2ec0 0x7f02541ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): picked mon.noname-b con 0x7f02540a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:08:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0254135df0 0x7f025411f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): picked mon.noname-a con 0x7f0254135df0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:08:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02541236a0 con 0x7f0254135df0 2023-10-30T12:08:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02541238f0 con 0x7f02540a2ec0 2023-10-30T12:08:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0254122160 con 0x7f02540a2ae0 2023-10-30T12:08:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): _renew_subs 2023-10-30T12:08:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f025a70f640 10 monclient(hunting): authenticate will time out at 1383.542605s 2023-10-30T12:08:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0258cac640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0254135df0 msgr2=0x7f025411f920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:08:58.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0258cac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0254135df0 0x7f025411f920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:08:58.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02540a2ae0 0x7f02541ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02540a2ae0 0x7f02541ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59418/0 (socket says 172.21.15.50:59418) 2023-10-30T12:08:58.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 -- 172.21.15.50:0/855855753 learned_addr learned my addr 172.21.15.50:0/855855753 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:08:58.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): get_auth_request con 0x7f02540a2ae0 auth_method 0 2023-10-30T12:08:58.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02540a2ec0 0x7f02541ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 10 monclient(hunting): get_auth_request con 0x7f02540a2ec0 auth_method 0 2023-10-30T12:08:58.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient(hunting): handle_auth_done global_id 114317 payload 824 2023-10-30T12:08:58.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02540a2ec0 msgr2=0x7f02541ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02540a2ec0 0x7f02541ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0254135df0 msgr2=0x7f025411f920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:08:58.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0254135df0 0x7f025411f920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient: _finish_hunting 0 2023-10-30T12:08:58.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 monclient: found mon.noname-c 2023-10-30T12:08:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:08:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.359+0000 7f0253fff640 1 -- 172.21.15.50:0/855855753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c009b30 con 0x7f02540a2ae0 2023-10-30T12:08:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0253fff640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02540a2ae0 0x7f02541ca270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f024800e9e0 tx=0x7f024800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 client.?.objecter ms_handle_connect 0x7f02540a2ae0 2023-10-30T12:08:58.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:08:58.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248009670 con 0x7f02540a2ae0 2023-10-30T12:08:58.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:08:58.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: dump: 2023-10-30T12:08:58.401 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.401 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.401 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.401 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.401 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.401 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.402 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.402 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.402 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.402 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: _finish_auth 0 2023-10-30T12:08:58.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:08:58.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:08:58.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 5 monclient: authenticate success, global_id 114317 2023-10-30T12:08:58.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024800c9d0 con 0x7f02540a2ae0 2023-10-30T12:08:58.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 10 monclient: _renew_subs 2023-10-30T12:08:58.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02540a1790 con 0x7f02540a2ae0 2023-10-30T12:08:58.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:08:58.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024800cd80 con 0x7f02540a2ae0 2023-10-30T12:08:58.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:08:58.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f0250ff9640 10 monclient: dump: 2023-10-30T12:08:58.405 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.405 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.405 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.405 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.406 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.406 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.406 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.406 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.406 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.406 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 10 client.114317.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:08:58.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 10 monclient: _renew_subs 2023-10-30T12:08:58.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:08:58.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f02527fc640 4 set_mon_vals no callback set 2023-10-30T12:08:58.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0254121520 con 0x7f02540a2ae0 2023-10-30T12:08:58.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.360+0000 7f025a70f640 1 librados: init done 2023-10-30T12:08:58.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0248013470 con 0x7f02540a2ae0 2023-10-30T12:08:58.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f025a70f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:58.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f025a70f640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f025411fe60 0x7f0254120240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f025a70f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0254123a80 con 0x7f025411fe60 2023-10-30T12:08:58.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0250ff9640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02400392a0 0x7f024003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 7100+0+0 (secure 0 0 0) 0x7f024804b4c0 con 0x7f02540a2ae0 2023-10-30T12:08:58.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0250ff9640 10 client.114317.objecter ms_dispatch 0x7f0254000b90 osd_map(283..283 src has 1..283) v4 2023-10-30T12:08:58.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0250ff9640 3 client.114317.objecter handle_osd_map got epochs [283,283] > 0 2023-10-30T12:08:58.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0250ff9640 3 client.114317.objecter handle_osd_map decoding full epoch 283 2023-10-30T12:08:58.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f025411fe60 0x7f0254120240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient: get_auth_request con 0x7f025411fe60 auth_method 0 2023-10-30T12:08:58.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0250ff9640 20 client.114317.objecter dump_active .. 0 homeless 2023-10-30T12:08:58.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0258cac640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02400392a0 0x7f024003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f0258cac640 10 monclient: get_auth_request con 0x7f02400392a0 auth_method 0 2023-10-30T12:08:58.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 10 monclient(hunting): handle_auth_done global_id 94669 payload 824 2023-10-30T12:08:58.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.361+0000 7f02537fe640 1 -- 172.21.15.50:0/855855753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f025409a5c0 con 0x7f025411fe60 2023-10-30T12:08:58.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 client.114317.objecter ms_handle_connect 0x7f025411fe60 2023-10-30T12:08:58.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 client.114317.objecter resend_mon_ops 2023-10-30T12:08:58.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f02537fe640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f025411fe60 0x7f0254120240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f024400fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 client.114317.objecter ms_handle_connect 0x7f02400392a0 2023-10-30T12:08:58.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0258cac640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02400392a0 0x7f024003b760 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7f024c035000 tx=0x7f024c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244017070 con 0x7f025411fe60 2023-10-30T12:08:58.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:58.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 monclient: dump: 2023-10-30T12:08:58.414 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.414 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.414 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.415 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.415 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.415 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.415 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.415 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.415 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f02440050d0 con 0x7f025411fe60 2023-10-30T12:08:58.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:08:58.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:08:58.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f025411fe60 msgr2=0x7f0254120240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f025411fe60 0x7f0254120240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f024400fc00 comp rx=0 tx=0).stop 2023-10-30T12:08:58.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f025411fe60 msgr2=0x7f0254120240 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:58.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.362+0000 7f0250ff9640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f025411fe60 0x7f0254120240 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f024400fc00 comp rx=0 tx=0).stop 2023-10-30T12:08:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.368+0000 7f025a70f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:08:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.368+0000 7f025a70f640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0254120780 0x7f025409dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:08:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.368+0000 7f025a70f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:08:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.368+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0254073ff0 con 0x7f0254120780 2023-10-30T12:08:58.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0254120780 0x7f025409dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:08:58.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient: get_auth_request con 0x7f0254120780 auth_method 0 2023-10-30T12:08:58.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:08:58.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:08:58.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:08:58.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:08:58.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:08:58.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:08:58.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 10 monclient(hunting): handle_auth_done global_id 94672 payload 824 2023-10-30T12:08:58.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 1 -- 172.21.15.50:0/855855753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0254099b40 con 0x7f0254120780 2023-10-30T12:08:58.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f02537fe640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0254120780 0x7f025409dde0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f0244001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:08:58.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f0250ff9640 10 client.114317.objecter ms_handle_connect 0x7f0254120780 2023-10-30T12:08:58.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f0250ff9640 10 client.114317.objecter resend_mon_ops 2023-10-30T12:08:58.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244006aa0 con 0x7f0254120780 2023-10-30T12:08:58.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f0250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:08:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f0250ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:08:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.369+0000 7f0250ff9640 10 monclient: dump: 2023-10-30T12:08:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:08:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:08:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:08:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:08:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:08:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:08:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:08:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:08:58.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:08:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7f0244006c00 con 0x7f0254120780 2023-10-30T12:08:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0250ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:08:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0250ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:08:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0254120780 msgr2=0x7f025409dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0250ff9640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0254120780 0x7f025409dde0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f0244001120 comp rx=0 tx=0).stop 2023-10-30T12:08:58.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0250ff9640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0254120780 msgr2=0x7f025409dde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:08:58.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0250ff9640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0254120780 0x7f025409dde0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0244000f60 tx=0x7f0244001120 comp rx=0 tx=0).stop 2023-10-30T12:08:58.426 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:08:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:08:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 80, 2023-10-30T12:08:58.427 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:08:58.428 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:08:58.428 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:08:58.428 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.428 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:08:58.428 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:58.429 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:08:58.429 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:08:58.429 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:58.429 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:58.429 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:58.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:58.430 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:58.430 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:58.430 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:58.430 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:58.430 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:58.431 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:58.431 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.431 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:08:58.431 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:08:58.431 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:58.432 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:58.432 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:58.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:58.432 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:58.432 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:58.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:58.433 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:58.433 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:58.433 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:58.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.434 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:08:58.434 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:08:58.434 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:08:58.434 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:08:58.434 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:08:58.435 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:08:58.435 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:58.435 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:08:58.435 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:08:58.435 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:08:58.436 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:08:58.436 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:08:58.436 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:08:58.436 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:08:58.436 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:08:58.436 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:08:58.437 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:08:58.437 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:08:58.437 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:08:58.437 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:08:58.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:08:58.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:08:58.438 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:08:58.438 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:08:58.438 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:08:58.438 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:08:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:08:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:08:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.439 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:08:58.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.440 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:08:58.440 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:08:58.440 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:58.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:58.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:58.441 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:08:58.441 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:58.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:08:58.442 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:08:58.443 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:08:58.443 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:58.443 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:58.444 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:58.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.444 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:08:58.444 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:08:58.445 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:58.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:58.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:58.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:08:58.446 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:58.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:08:58.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:08:58.447 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:08:58.448 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:58.448 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:58.448 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:58.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.449 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:08:58.449 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:08:58.449 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:08:58.449 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:08:58.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.449 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:08:58.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:08:58.450 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:08:58.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:08:58.451 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:08:58.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.451 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.451 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:08:58.451 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:08:58.452 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:08:58.452 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:08:58.452 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:08:58.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.453 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:08:58.453 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:08:58.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.453 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:58.453 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:58.454 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:58.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.454 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.454 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:08:58.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.455 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:58.455 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:58.455 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:08:58.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.455 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:08:58.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.456 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:58.456 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:58.456 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:08:58.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.457 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:08:58.457 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:08:58.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:08:58.457 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:08:58.457 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:08:58.458 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:08:58.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:08:58.458 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:08:58.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:08:58.458 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:08:58.459 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:08:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 10 librados: watch_flush enter 2023-10-30T12:08:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 10 librados: watch_flush exit 2023-10-30T12:08:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 20 client.114317.objecter shutdown clearing up homeless session... 2023-10-30T12:08:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 10 client.114317.objecter successfully canceled tick 2023-10-30T12:08:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02400392a0 msgr2=0x7f024003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02400392a0 0x7f024003b760 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7f024c035000 tx=0x7f024c030000 comp rx=0 tx=0).stop 2023-10-30T12:08:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 10 monclient: shutdown 2023-10-30T12:08:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:08:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02540a2ae0 msgr2=0x7f02541ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:08:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 --2- 172.21.15.50:0/855855753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02540a2ae0 0x7f02541ca270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f024800e9e0 tx=0x7f024800eea0 comp rx=0 tx=0).stop 2023-10-30T12:08:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0258cac640 1 -- 172.21.15.50:0/855855753 reap_dead start 2023-10-30T12:08:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f0258cac640 1 -- 172.21.15.50:0/855855753 reap_dead start 2023-10-30T12:08:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 shutdown_connections 2023-10-30T12:08:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 >> 172.21.15.50:0/855855753 conn(0x7f02540a80b0 msgr2=0x7f02540a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:08:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 shutdown_connections 2023-10-30T12:08:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 -- 172.21.15.50:0/855855753 wait complete. 2023-10-30T12:08:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:08:58.370+0000 7f025a70f640 1 librados: shutdown 2023-10-30T12:08:58.464 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:08:58.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:09:18.466 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:09:18.466 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:09:18.466 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:09:18.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:09:18.470 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:09:18.470 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:09:18.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:18.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.584+0000 7fd403f9e640 10 monclient: get_monmap_and_config 2023-10-30T12:09:18.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.584+0000 7fd403f9e640 10 monclient: build_initial_monmap 2023-10-30T12:09:18.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.584+0000 7fd403f9e640 10 monclient: monmap: 2023-10-30T12:09:18.587 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:18.587 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:18.587 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:18.585789+0000 2023-10-30T12:09:18.587 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:18.585789+0000 2023-10-30T12:09:18.587 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:18.588 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:18.588 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:18.588 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:18.588 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:18.588 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:18.589 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:18.589 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:18.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 1 Processor -- start 2023-10-30T12:09:18.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 1 -- start start 2023-10-30T12:09:18.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 10 monclient: init 2023-10-30T12:09:18.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:18.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-10-30T12:09:18.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 0x7fd3fc1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:18.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3fc0a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:18.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.585+0000 7fd403f9e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc099a40 0x7fd3fc132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:18.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3fc099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:18.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3fc0a2350 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:18.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc133120 con 0x7fd3fc0a2350 2023-10-30T12:09:18.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:18.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3fc0a3d70 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:18.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc0a4f40 con 0x7fd3fc099a40 2023-10-30T12:09:18.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:18.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd403f9e640 10 monclient(hunting): authenticate will time out at 1403.769054s 2023-10-30T12:09:18.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 0x7fd3fc1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd40253b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd3fc0a2350 0x7fd3fc0a27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41524/0 (socket says 172.21.15.50:41524) 2023-10-30T12:09:18.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc099a40 0x7fd3fc132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd40253b640 1 -- 172.21.15.50:0/2993198192 learned_addr learned my addr 172.21.15.50:0/2993198192 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): get_auth_request con 0x7fd3fc099a40 auth_method 0 2023-10-30T12:09:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 10 monclient(hunting): get_auth_request con 0x7fd3fc0a4b60 auth_method 0 2023-10-30T12:09:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 1 -- 172.21.15.50:0/2993198192 >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 msgr2=0x7fd3fc1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 1 -- 172.21.15.50:0/2993198192 >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 msgr2=0x7fd3fc1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 1 --2- 172.21.15.50:0/2993198192 >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 0x7fd3fc1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401d3a640 1 --2- 172.21.15.50:0/2993198192 >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 0x7fd3fc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient(hunting): handle_auth_done global_id 114329 payload 274 2023-10-30T12:09:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 1 -- 172.21.15.50:0/2993198192 >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 msgr2=0x7fd3fc1326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 1 --2- 172.21.15.50:0/2993198192 >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 0x7fd3fc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 1 -- 172.21.15.50:0/2993198192 >> v1:172.21.15.50:6789/0 conn(0x7fd3fc0a2350 legacy=0x7fd3fc0a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient: _finish_hunting 0 2023-10-30T12:09:18.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 1 monclient: found mon.noname-b 2023-10-30T12:09:18.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:18.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:09:18.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.586+0000 7fd401539640 1 -- 172.21.15.50:0/2993198192 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc0a4150 con 0x7fd3fc099a40 2023-10-30T12:09:18.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd401539640 1 --2- 172.21.15.50:0/2993198192 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc099a40 0x7fd3fc132be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0097a0 tx=0x7fd3ec02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a397495db448c285 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 1 -- 172.21.15.50:0/2993198192 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec02f050 con 0x7fd3fc099a40 2023-10-30T12:09:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: dump: 2023-10-30T12:09:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:18.606 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.606 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.606 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:18.606 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:18.606 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:18.607 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:18.607 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:18.607 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:18.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: _finish_auth 0 2023-10-30T12:09:18.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: _check_auth_tickets 2023-10-30T12:09:18.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:18.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 1 -- 172.21.15.50:0/2993198192 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec031700 con 0x7fd3fc099a40 2023-10-30T12:09:18.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:18.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 1 -- 172.21.15.50:0/2993198192 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec031a90 con 0x7fd3fc099a40 2023-10-30T12:09:18.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 5 monclient: authenticate success, global_id 114329 2023-10-30T12:09:18.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:18.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:18.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd400d38640 10 monclient: dump: 2023-10-30T12:09:18.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:18.609 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:18.609 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.609 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.610 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:18.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:18.610 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:18.610 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:18.610 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:18.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:18.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:18.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 4 set_mon_vals no callback set 2023-10-30T12:09:18.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 10 monclient: shutdown 2023-10-30T12:09:18.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:18.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 1 -- 172.21.15.50:0/2993198192 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc099a40 msgr2=0x7fd3fc132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:18.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 1 --2- 172.21.15.50:0/2993198192 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc099a40 0x7fd3fc132be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0097a0 tx=0x7fd3ec02b7a0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 1 -- 172.21.15.50:0/2993198192 shutdown_connections 2023-10-30T12:09:18.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 1 --2- 172.21.15.50:0/2993198192 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc099a40 0x7fd3fc132be0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 1 --2- 172.21.15.50:0/2993198192 >> v2:172.21.15.50:3300/0 conn(0x7fd3fc0a4b60 0x7fd3fc1326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.587+0000 7fd403f9e640 1 -- 172.21.15.50:0/2993198192 >> 172.21.15.50:0/2993198192 conn(0x7fd3fc0a8060 msgr2=0x7fd3fc0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:18.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 -- 172.21.15.50:0/2993198192 shutdown_connections 2023-10-30T12:09:18.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 -- 172.21.15.50:0/2993198192 wait complete. 2023-10-30T12:09:18.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 10 monclient: build_initial_monmap 2023-10-30T12:09:18.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 10 monclient: monmap: 2023-10-30T12:09:18.613 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:18.614 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:18.614 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:18.614 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:18.614 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:18.614 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:18.614 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:18.615 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:18.615 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:18.615 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:18.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 librados: starting msgr at 2023-10-30T12:09:18.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 librados: starting objecter 2023-10-30T12:09:18.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 Processor -- start 2023-10-30T12:09:18.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 -- start start 2023-10-30T12:09:18.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 librados: setting wanted keys 2023-10-30T12:09:18.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 1 librados: calling monclient init 2023-10-30T12:09:18.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.588+0000 7fd403f9e640 10 monclient: init 2023-10-30T12:09:18.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:18.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:09:18.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3fc0a2350 0x7fd3fc1ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:18.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3fc0a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:18.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc0a4b60 0x7fd3fc1cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3fc0a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 0x7fd3fc11eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3fc11e690 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc120960 con 0x7fd3fc11e690 2023-10-30T12:09:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc121e60 con 0x7fd3fc0a2350 2023-10-30T12:09:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc120f70 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd403f9e640 10 monclient(hunting): authenticate will time out at 1403.772414s 2023-10-30T12:09:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc0a4b60 0x7fd3fc1cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc0a4b60 0x7fd3fc1cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42682/0 (socket says 172.21.15.50:42682) 2023-10-30T12:09:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 -- 172.21.15.50:0/387917069 learned_addr learned my addr 172.21.15.50:0/387917069 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): get_auth_request con 0x7fd3fc0a4b60 auth_method 0 2023-10-30T12:09:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:18.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 0x7fd3fc11eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 10 monclient(hunting): get_auth_request con 0x7fd3fc11e690 auth_method 0 2023-10-30T12:09:18.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:18.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 1 -- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 msgr2=0x7fd3fc11eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:09:18.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 1 -- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 msgr2=0x7fd3fc11eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:09:18.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 0x7fd3fc11eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:18.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd40253b640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 0x7fd3fc11eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:18.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401d3a640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3fc0a2350 0x7fd3fc1ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401d3a640 10 monclient(hunting): get_auth_request con 0x7fd3fc0a2350 auth_method 0 2023-10-30T12:09:18.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:18.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401d3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:18.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401d3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:18.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient(hunting): handle_auth_done global_id 114332 payload 824 2023-10-30T12:09:18.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 -- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3fc0a2350 msgr2=0x7fd3fc1ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:18.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3fc0a2350 0x7fd3fc1ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 -- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 msgr2=0x7fd3fc11eb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:18.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 0x7fd3fc11eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient: _finish_hunting 0 2023-10-30T12:09:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 monclient: found mon.noname-c 2023-10-30T12:09:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.589+0000 7fd401539640 1 -- 172.21.15.50:0/387917069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec009400 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd401539640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc0a4b60 0x7fd3fc1cae00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec036040 tx=0x7fd3ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 client.?.objecter ms_handle_connect 0x7fd3fc0a4b60 2023-10-30T12:09:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 1 -- 172.21.15.50:0/387917069 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec033c30 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:18.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: dump: 2023-10-30T12:09:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:18.629 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 1 -- 172.21.15.50:0/387917069 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec033db0 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 5 monclient: authenticate success, global_id 114332 2023-10-30T12:09:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 10 monclient: _renew_subs 2023-10-30T12:09:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc0a18d0 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 1 -- 172.21.15.50:0/387917069 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec03c640 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:18.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f27fc640 10 monclient: dump: 2023-10-30T12:09:18.632 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:18.632 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:18.632 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.632 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:18.632 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:18.633 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:18.633 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:18.633 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:18.633 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:18.633 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:18.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd3f3fff640 4 set_mon_vals no callback set 2023-10-30T12:09:18.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 10 client.114332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:18.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 10 monclient: _renew_subs 2023-10-30T12:09:18.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:18.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc122700 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.590+0000 7fd403f9e640 1 librados: init done 2023-10-30T12:09:18.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd403f9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:18.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd403f9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:18.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd403f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:18.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd3f27fc640 1 -- 172.21.15.50:0/387917069 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd3ec03a040 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc04e160 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd3f27fc640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3e00392f0 0x7fd3e003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:18.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd3f27fc640 1 -- 172.21.15.50:0/387917069 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(304..304 src has 1..304) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fd3ec06c550 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd3f27fc640 10 client.114332.objecter ms_dispatch 0x7fd3fc000b90 osd_map(304..304 src has 1..304) v4 2023-10-30T12:09:18.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd3f27fc640 3 client.114332.objecter handle_osd_map got epochs [304,304] > 0 2023-10-30T12:09:18.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd3f27fc640 3 client.114332.objecter handle_osd_map decoding full epoch 304 2023-10-30T12:09:18.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd3f27fc640 20 client.114332.objecter dump_active .. 0 homeless 2023-10-30T12:09:18.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd401d3a640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3e00392f0 0x7fd3e003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:18.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.591+0000 7fd401d3a640 10 monclient: get_auth_request con 0x7fd3e00392f0 auth_method 0 2023-10-30T12:09:18.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.592+0000 7fd401d3a640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3e00392f0 0x7fd3e003b7b0 secure :-1 s=READY pgs=4065 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8006fd0 tx=0x7fd3f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:18.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.592+0000 7fd3f27fc640 10 client.114332.objecter ms_handle_connect 0x7fd3e00392f0 2023-10-30T12:09:18.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.596+0000 7fd3f27fc640 1 -- 172.21.15.50:0/387917069 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd3ec04ecd0 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.596+0000 7fd3f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:18.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.596+0000 7fd3f27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:18.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.838+0000 7fd403f9e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:18.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.838+0000 7fd403f9e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:18.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.838+0000 7fd403f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:18.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.838+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd3fc136010 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.838+0000 7fd3f27fc640 1 -- 172.21.15.50:0/387917069 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fd3ec038080 con 0x7fd3fc0a4b60 2023-10-30T12:09:18.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.839+0000 7fd3f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:18.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.839+0000 7fd3f27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:18.841 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":80,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:18.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 10 librados: watch_flush enter 2023-10-30T12:09:18.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 10 librados: watch_flush exit 2023-10-30T12:09:18.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 20 client.114332.objecter shutdown clearing up homeless session... 2023-10-30T12:09:18.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 10 client.114332.objecter successfully canceled tick 2023-10-30T12:09:18.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3e00392f0 msgr2=0x7fd3e003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:18.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3e00392f0 0x7fd3e003b7b0 secure :-1 s=READY pgs=4065 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8006fd0 tx=0x7fd3f8008040 comp rx=0 tx=0).stop 2023-10-30T12:09:18.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 10 monclient: shutdown 2023-10-30T12:09:18.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:18.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc0a4b60 msgr2=0x7fd3fc1cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:18.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc0a4b60 0x7fd3fc1cae00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec036040 tx=0x7fd3ec0046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 shutdown_connections 2023-10-30T12:09:18.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3e00392f0 0x7fd3e003b7b0 unknown :-1 s=CLOSED pgs=4065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3fc11e690 0x7fd3fc11eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3fc0a4b60 0x7fd3fc1cae00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 --2- 172.21.15.50:0/387917069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3fc0a2350 0x7fd3fc1ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:18.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 >> 172.21.15.50:0/387917069 conn(0x7fd3fc0a8060 msgr2=0x7fd3fc0a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:18.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 shutdown_connections 2023-10-30T12:09:18.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.840+0000 7fd403f9e640 1 -- 172.21.15.50:0/387917069 wait complete. 2023-10-30T12:09:18.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:18.841+0000 7fd403f9e640 1 librados: shutdown 2023-10-30T12:09:21.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:21.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.970+0000 7fbac8e87640 10 monclient: get_monmap_and_config 2023-10-30T12:09:21.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.970+0000 7fbac8e87640 10 monclient: build_initial_monmap 2023-10-30T12:09:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.970+0000 7fbac8e87640 10 monclient: monmap: 2023-10-30T12:09:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:21.970214+0000 2023-10-30T12:09:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:21.970214+0000 2023-10-30T12:09:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:21.973 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:21.973 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:21.973 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:21.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.970+0000 7fbac8e87640 1 Processor -- start 2023-10-30T12:09:21.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.970+0000 7fbac8e87640 1 -- start start 2023-10-30T12:09:21.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.970+0000 7fbac8e87640 10 monclient: init 2023-10-30T12:09:21.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:21.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:09:21.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbac4130290 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:21.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbac4070fa0 0x7fbac4138680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:21.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient(hunting): picked mon.noname-c con 0x7fbac4070fa0 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:21.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbac4138bc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:21.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac4130d80 con 0x7fbac4138bc0 2023-10-30T12:09:21.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac4070000 con 0x7fbac4130290 2023-10-30T12:09:21.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:21.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac40702e0 con 0x7fbac4070fa0 2023-10-30T12:09:21.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:21.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac8e87640 10 monclient(hunting): authenticate will time out at 1407.153498s 2023-10-30T12:09:21.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac2d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbac4138bc0 0x7fbac413afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41534/0 (socket says 172.21.15.50:41534) 2023-10-30T12:09:21.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbac2d9d640 1 -- 172.21.15.50:0/1207442163 learned_addr learned my addr 172.21.15.50:0/1207442163 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:21.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 1 --2- 172.21.15.50:0/1207442163 >> v2:172.21.15.118:3300/0 conn(0x7fbac4070fa0 0x7fbac4138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:21.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 10 monclient(hunting): get_auth_request con 0x7fbac4070fa0 auth_method 0 2023-10-30T12:09:21.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:21.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:21.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:21.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.971+0000 7fbabbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:21.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbac1d9b640 1 -- 172.21.15.50:0/1207442163 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922657692 0 0) 0x7fbac4070000 con 0x7fbac4130290 2023-10-30T12:09:21.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbac1d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:09:21.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbac1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:21.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbac1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:21.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbac1d9b640 10 monclient(hunting): my global_id is 94708 2023-10-30T12:09:21.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbac1d9b640 1 -- 172.21.15.50:0/1207442163 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab0003520 con 0x7fbac4130290 2023-10-30T12:09:21.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 10 monclient(hunting): handle_auth_done global_id 94705 payload 274 2023-10-30T12:09:21.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 1 -- 172.21.15.50:0/1207442163 >> v1:172.21.15.118:6789/0 conn(0x7fbac4130290 legacy=0x7fbac4130670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 1 -- 172.21.15.50:0/1207442163 >> v1:172.21.15.50:6789/0 conn(0x7fbac4138bc0 legacy=0x7fbac413afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:21.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 1 monclient: found mon.noname-c 2023-10-30T12:09:21.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:21.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:09:21.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 1 -- 172.21.15.50:0/1207442163 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac406f2d0 con 0x7fbac4070fa0 2023-10-30T12:09:21.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.972+0000 7fbabbfff640 1 --2- 172.21.15.50:0/1207442163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4070fa0 0x7fbac4138680 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097d0 tx=0x7fbaac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15bdd6b4edbf0746 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:21.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 1 -- 172.21.15.50:0/1207442163 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac02e070 con 0x7fbac4070fa0 2023-10-30T12:09:21.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:21.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: dump: 2023-10-30T12:09:21.991 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:21.991 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:21.991 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:21.991 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:21.991 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.992 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:21.992 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:21.992 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:21.992 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:21.992 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:21.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: _finish_auth 0 2023-10-30T12:09:21.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:09:21.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:21.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 1 -- 172.21.15.50:0/1207442163 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac0315a0 con 0x7fbac4070fa0 2023-10-30T12:09:21.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:21.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 5 monclient: authenticate success, global_id 94705 2023-10-30T12:09:21.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:21.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 4 set_mon_vals no callback set 2023-10-30T12:09:21.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 1 -- 172.21.15.50:0/1207442163 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac031910 con 0x7fbac4070fa0 2023-10-30T12:09:21.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:21.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:21.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac1d9b640 10 monclient: dump: 2023-10-30T12:09:21.995 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:21.995 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:21.995 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:21.997 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 10 monclient: shutdown 2023-10-30T12:09:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 1 -- 172.21.15.50:0/1207442163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4070fa0 msgr2=0x7fbac4138680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:21.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1207442163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4070fa0 0x7fbac4138680 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097d0 tx=0x7fbaac02b840 comp rx=0 tx=0).stop 2023-10-30T12:09:21.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 1 -- 172.21.15.50:0/1207442163 shutdown_connections 2023-10-30T12:09:21.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1207442163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4070fa0 0x7fbac4138680 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:21.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 1 -- 172.21.15.50:0/1207442163 >> 172.21.15.50:0/1207442163 conn(0x7fbac40a22b0 msgr2=0x7fbac40a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:21.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 1 -- 172.21.15.50:0/1207442163 shutdown_connections 2023-10-30T12:09:21.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.973+0000 7fbac8e87640 1 -- 172.21.15.50:0/1207442163 wait complete. 2023-10-30T12:09:21.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 10 monclient: build_initial_monmap 2023-10-30T12:09:21.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 10 monclient: monmap: 2023-10-30T12:09:21.999 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:21.999 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:22.000 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:22.000 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:22.000 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:22.000 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:22.001 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:22.001 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:22.001 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:22.001 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:22.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 1 librados: starting msgr at 2023-10-30T12:09:22.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 1 librados: starting objecter 2023-10-30T12:09:22.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 1 Processor -- start 2023-10-30T12:09:22.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 1 -- start start 2023-10-30T12:09:22.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 1 librados: setting wanted keys 2023-10-30T12:09:22.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 1 librados: calling monclient init 2023-10-30T12:09:22.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.974+0000 7fbac8e87640 10 monclient: init 2023-10-30T12:09:22.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:22.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:09:22.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbac4070fa0 0x7fbac41ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): picked mon.noname-c con 0x7fbac4070fa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:22.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4130290 0x7fbac412eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): picked mon.noname-b con 0x7fbac4130290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:22.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 0x7fbac412f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): picked mon.noname-a con 0x7fbac4138bc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:22.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41cb0d0 con 0x7fbac4138bc0 2023-10-30T12:09:22.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac406efa0 con 0x7fbac4130290 2023-10-30T12:09:22.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:22.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4138490 con 0x7fbac4070fa0 2023-10-30T12:09:22.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:22.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac8e87640 10 monclient(hunting): authenticate will time out at 1407.157457s 2023-10-30T12:09:22.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 0x7fbac412f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbac4070fa0 0x7fbac41ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbac4070fa0 0x7fbac41ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42726/0 (socket says 172.21.15.50:42726) 2023-10-30T12:09:22.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 -- 172.21.15.50:0/1728799345 learned_addr learned my addr 172.21.15.50:0/1728799345 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:22.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 0x7fbac412f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40304/0 (socket says 172.21.15.50:40304) 2023-10-30T12:09:22.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): get_auth_request con 0x7fbac4070fa0 auth_method 0 2023-10-30T12:09:22.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 10 monclient(hunting): get_auth_request con 0x7fbac4138bc0 auth_method 0 2023-10-30T12:09:22.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 1 -- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 msgr2=0x7fbac412f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:09:22.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 1 -- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 msgr2=0x7fbac412f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:09:22.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 0x7fbac412f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:22.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac2d9d640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 0x7fbac412f430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:22.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4130290 0x7fbac412eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 10 monclient(hunting): get_auth_request con 0x7fbac4130290 auth_method 0 2023-10-30T12:09:22.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:22.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:22.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:22.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:22.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:22.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbabbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:22.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient(hunting): handle_auth_done global_id 114338 payload 824 2023-10-30T12:09:22.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 -- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4130290 msgr2=0x7fbac412eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4130290 0x7fbac412eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 -- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 msgr2=0x7fbac412f430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:22.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 0x7fbac412f430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:22.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 monclient: found mon.noname-c 2023-10-30T12:09:22.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:22.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:22.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.975+0000 7fbac259c640 1 -- 172.21.15.50:0/1728799345 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac009400 con 0x7fbac4070fa0 2023-10-30T12:09:22.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac259c640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbac4070fa0 0x7fbac41ca3d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c4c0 tx=0x7fbab400c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:22.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 client.?.objecter ms_handle_connect 0x7fbac4070fa0 2023-10-30T12:09:22.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:22.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 1 -- 172.21.15.50:0/1728799345 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab40052d0 con 0x7fbac4070fa0 2023-10-30T12:09:22.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:22.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: dump: 2023-10-30T12:09:22.015 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:22.015 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:22.015 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:22.015 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:22.015 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.016 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:22.016 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:22.016 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:22.016 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:22.016 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:22.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:22.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:22.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:22.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 1 -- 172.21.15.50:0/1728799345 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4005430 con 0x7fbac4070fa0 2023-10-30T12:09:22.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 5 monclient: authenticate success, global_id 114338 2023-10-30T12:09:22.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 10 monclient: _renew_subs 2023-10-30T12:09:22.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:22.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac411c8b0 con 0x7fbac4070fa0 2023-10-30T12:09:22.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:22.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 1 -- 172.21.15.50:0/1728799345 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4012870 con 0x7fbac4070fa0 2023-10-30T12:09:22.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:22.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:22.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbabaffd640 10 monclient: dump: 2023-10-30T12:09:22.019 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:22.019 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:22.019 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:22.020 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:22.020 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:22.020 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:22.020 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:22.020 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:22.020 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:22.021 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:22.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac159a640 4 set_mon_vals no callback set 2023-10-30T12:09:22.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 10 client.114338.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:22.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 10 monclient: _renew_subs 2023-10-30T12:09:22.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:22.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac411c010 con 0x7fbac4070fa0 2023-10-30T12:09:22.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.976+0000 7fbac8e87640 1 librados: init done 2023-10-30T12:09:22.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabaffd640 1 -- 172.21.15.50:0/1728799345 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbab40129d0 con 0x7fbac4070fa0 2023-10-30T12:09:22.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbac8e87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbac8e87640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbac8e87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:22.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac406fd20 con 0x7fbac4070fa0 2023-10-30T12:09:22.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabaffd640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbaa40392f0 0x7fbaa403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:22.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabaffd640 1 -- 172.21.15.50:0/1728799345 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(307..307 src has 1..307) v4 ==== 6270+0+0 (secure 0 0 0) 0x7fbab404a8c0 con 0x7fbac4070fa0 2023-10-30T12:09:22.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabaffd640 10 client.114338.objecter ms_dispatch 0x7fbac4000b90 osd_map(307..307 src has 1..307) v4 2023-10-30T12:09:22.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabaffd640 3 client.114338.objecter handle_osd_map got epochs [307,307] > 0 2023-10-30T12:09:22.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabaffd640 3 client.114338.objecter handle_osd_map decoding full epoch 307 2023-10-30T12:09:22.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabaffd640 20 client.114338.objecter dump_active .. 0 homeless 2023-10-30T12:09:22.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabbfff640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbaa40392f0 0x7fbaa403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:22.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.977+0000 7fbabbfff640 10 monclient: get_auth_request con 0x7fbaa40392f0 auth_method 0 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.978+0000 7fbabaffd640 10 client.114338.objecter ms_handle_connect 0x7fbaa40392f0 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.978+0000 7fbabbfff640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbaa40392f0 0x7fbaa403b7b0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7fbaac036000 tx=0x7fbaac03e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.982+0000 7fbabaffd640 1 -- 172.21.15.50:0/1728799345 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbab4012c80 con 0x7fbac4070fa0 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.982+0000 7fbabaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:22.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:21.982+0000 7fbabaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.237+0000 7fbac8e87640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.237+0000 7fbac8e87640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.237+0000 7fbac8e87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.237+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbac404e160 con 0x7fbac4070fa0 2023-10-30T12:09:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.237+0000 7fbabaffd640 1 -- 172.21.15.50:0/1728799345 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fbab401a360 con 0x7fbac4070fa0 2023-10-30T12:09:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.237+0000 7fbabaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.237+0000 7fbabaffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:22.240 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":80,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 10 librados: watch_flush enter 2023-10-30T12:09:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 10 librados: watch_flush exit 2023-10-30T12:09:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 20 client.114338.objecter shutdown clearing up homeless session... 2023-10-30T12:09:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 10 client.114338.objecter successfully canceled tick 2023-10-30T12:09:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbaa40392f0 msgr2=0x7fbaa403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbaa40392f0 0x7fbaa403b7b0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7fbaac036000 tx=0x7fbaac03e040 comp rx=0 tx=0).stop 2023-10-30T12:09:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 10 monclient: shutdown 2023-10-30T12:09:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbac4070fa0 msgr2=0x7fbac41ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbac4070fa0 0x7fbac41ca3d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c4c0 tx=0x7fbab400c980 comp rx=0 tx=0).stop 2023-10-30T12:09:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 shutdown_connections 2023-10-30T12:09:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbaa40392f0 0x7fbaa403b7b0 unknown :-1 s=CLOSED pgs=4067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbac4138bc0 0x7fbac412f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbac4130290 0x7fbac412eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 --2- 172.21.15.50:0/1728799345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbac4070fa0 0x7fbac41ca3d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 >> 172.21.15.50:0/1728799345 conn(0x7fbac40a22b0 msgr2=0x7fbac409e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 shutdown_connections 2023-10-30T12:09:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 -- 172.21.15.50:0/1728799345 wait complete. 2023-10-30T12:09:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:22.240+0000 7fbac8e87640 1 librados: shutdown 2023-10-30T12:09:25.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:25.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.367+0000 7f231e887640 10 monclient: get_monmap_and_config 2023-10-30T12:09:25.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.367+0000 7f231e887640 10 monclient: build_initial_monmap 2023-10-30T12:09:25.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.367+0000 7f231e887640 10 monclient: monmap: 2023-10-30T12:09:25.369 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:25.369 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:25.369 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:25.368388+0000 2023-10-30T12:09:25.370 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:25.368388+0000 2023-10-30T12:09:25.370 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:25.370 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:25.370 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:25.370 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:25.371 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 Processor -- start 2023-10-30T12:09:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 -- start start 2023-10-30T12:09:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient: init 2023-10-30T12:09:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-10-30T12:09:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): picked mon.noname-b con 0x7f2318098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): picked mon.noname-c con 0x7f2318098900 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): picked mon.noname-a con 0x7f23181326d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2318099930 con 0x7f23181326d0 2023-10-30T12:09:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231809aca0 con 0x7f2318098900 2023-10-30T12:09:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231809af80 con 0x7f2318098520 2023-10-30T12:09:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231e887640 10 monclient(hunting): authenticate will time out at 1410.551603s 2023-10-30T12:09:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42752/0 (socket says 172.21.15.50:42752) 2023-10-30T12:09:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 learned_addr learned my addr 172.21.15.50:0/138865836 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 10 monclient(hunting): get_auth_request con 0x7f2318098520 auth_method 0 2023-10-30T12:09:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:25.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 10 monclient(hunting): get_auth_request con 0x7f23181326d0 auth_method 0 2023-10-30T12:09:25.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:25.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:25.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:25.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:25.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:25.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.368+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 10 monclient(hunting): get_auth_request con 0x7f2318098900 auth_method 0 2023-10-30T12:09:25.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:25.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:25.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.369+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 10 monclient(hunting): get_auth_request con 0x7f2318098520 auth_method 0 2023-10-30T12:09:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:25.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 10 monclient(hunting): get_auth_request con 0x7f23181326d0 auth_method 0 2023-10-30T12:09:25.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:25.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.569+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 10 monclient(hunting): get_auth_request con 0x7f2318098900 auth_method 0 2023-10-30T12:09:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.570+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:25.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 10 monclient(hunting): get_auth_request con 0x7f23181326d0 auth_method 0 2023-10-30T12:09:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 10 monclient(hunting): get_auth_request con 0x7f2318098520 auth_method 0 2023-10-30T12:09:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:25.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:25.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:25.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:25.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.970+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 10 monclient(hunting): get_auth_request con 0x7f2318098900 auth_method 0 2023-10-30T12:09:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:25.971+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 10 monclient(hunting): get_auth_request con 0x7f23181326d0 auth_method 0 2023-10-30T12:09:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 10 monclient(hunting): get_auth_request con 0x7f2318098520 auth_method 0 2023-10-30T12:09:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f231ce24640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.771+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 10 monclient(hunting): get_auth_request con 0x7f2318098900 auth_method 0 2023-10-30T12:09:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:09:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:26.772+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:28.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 10 monclient(hunting): tick 2023-10-30T12:09:28.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:28.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:28.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:28.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 msgr2=0x7f231809bf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:28.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2318098520 0x7f231809bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:28.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 msgr2=0x7f2318098d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.118:3300/0 conn(0x7f2318098900 0x7f2318098d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 -- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 msgr2=0x7f2318132b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 --2- 172.21.15.50:0/138865836 >> v2:172.21.15.50:3300/0 conn(0x7f23181326d0 0x7f2318132b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 10 monclient: _add_conns ranks=[0,1] 2023-10-30T12:09:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23000036b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23000073c0 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 -- 172.21.15.50:0/138865836 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231809b260 con 0x7f23000036b0 2023-10-30T12:09:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 1 -- 172.21.15.50:0/138865836 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230000c120 con 0x7f23000073c0 2023-10-30T12:09:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.368+0000 7f2315ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.572+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> v1:172.21.15.50:6789/0 conn(0x7f23000036b0 legacy=0x7f2300005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-10-30T12:09:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.572+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> v1:172.21.15.50:6789/0 conn(0x7f23000036b0 legacy=0x7f2300005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:09:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.572+0000 7f2317fff640 1 --1- 172.21.15.50:0/138865836 >> v1:172.21.15.50:6789/0 conn(0x7f23000036b0 0x7f2300005af0 :-1 s=OPENED pgs=27 cs=1 l=1).handle_message read tag failed 2023-10-30T12:09:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.572+0000 7f2317fff640 1 --1- 172.21.15.50:0/138865836 >> v1:172.21.15.50:6789/0 conn(0x7f23000036b0 0x7f2300005af0 :-1 s=OPENED pgs=27 cs=1 l=1).fault on lossy channel, failing 2023-10-30T12:09:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:28.572+0000 7f2316ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.50:6789/0 2023-10-30T12:09:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309706050 0 0) 0x7f230000c120 con 0x7f23000073c0 2023-10-30T12:09:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:09:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): my global_id is 94714 2023-10-30T12:09:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304003520 con 0x7f23000073c0 2023-10-30T12:09:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015035739 0 0) 0x7f2304003520 con 0x7f23000073c0 2023-10-30T12:09:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 >> v1:172.21.15.50:6789/0 conn(0x7f23000036b0 legacy=0x7f2300005af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:09:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:09:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230000d380 con 0x7f23000073c0 2023-10-30T12:09:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f230c00f790 con 0x7f23000073c0 2023-10-30T12:09:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:09:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: dump: 2023-10-30T12:09:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:29.847 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:09:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 >> v1:172.21.15.118:6789/0 conn(0x7f23000073c0 legacy=0x7f2300009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:09:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f231ce24640 1 -- 172.21.15.50:0/138865836 reap_dead start 2023-10-30T12:09:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f2304003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): picked mon.c con 0x7f2304003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23040047c0 0x7f2304004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): picked mon.b con 0x7f23040047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304005360 0x7f230400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): picked mon.a con 0x7f2304005360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230400c270 con 0x7f2304005360 2023-10-30T12:09:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230400c3e0 con 0x7f23040047c0 2023-10-30T12:09:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230400c5a0 con 0x7f2304003970 2023-10-30T12:09:29.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.837+0000 7f2316ffd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:09:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f2304003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): get_auth_request con 0x7f2304003970 auth_method 0 2023-10-30T12:09:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:09:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f23177fe640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23040047c0 0x7f2304004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f23177fe640 10 monclient(hunting): get_auth_request con 0x7f23040047c0 auth_method 0 2023-10-30T12:09:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f23177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f23177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f23177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient(hunting): handle_auth_done global_id 94714 payload 293 2023-10-30T12:09:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23040047c0 msgr2=0x7f2304004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23040047c0 0x7f2304004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304005360 msgr2=0x7f230400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304005360 0x7f230400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 1 monclient: found mon.c 2023-10-30T12:09:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 1 -- 172.21.15.50:0/138865836 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304003520 con 0x7f2304003970 2023-10-30T12:09:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient: _finish_auth 0 2023-10-30T12:09:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f2317fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f231e887640 5 monclient: authenticate success, global_id 94714 2023-10-30T12:09:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.838+0000 7f231e887640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2317fff640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f2304003d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2308012a70 tx=0x7f23040057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1d148cffec43fb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308030070 con 0x7f2304003970 2023-10-30T12:09:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 10 monclient: dump: 2023-10-30T12:09:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:29.860 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f231e887640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:29.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2308032050 con 0x7f2304003970 2023-10-30T12:09:29.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:29.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f231e887640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:29.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 1 -- 172.21.15.50:0/138865836 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308037630 con 0x7f2304003970 2023-10-30T12:09:29.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f231e887640 4 set_mon_vals no callback set 2023-10-30T12:09:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.839+0000 7f2316ffd640 10 monclient: dump: 2023-10-30T12:09:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:29.864 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:29.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 10 monclient: shutdown 2023-10-30T12:09:29.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:29.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 -- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 msgr2=0x7f2304003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:29.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f2304003d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2308012a70 tx=0x7f23040057f0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 -- 172.21.15.50:0/138865836 shutdown_connections 2023-10-30T12:09:29.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2304005360 0x7f230400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f23040047c0 0x7f2304004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 --2- 172.21.15.50:0/138865836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f2304003d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 -- 172.21.15.50:0/138865836 >> 172.21.15.50:0/138865836 conn(0x7f23180a05c0 msgr2=0x7f231809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:29.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 -- 172.21.15.50:0/138865836 shutdown_connections 2023-10-30T12:09:29.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.840+0000 7f231e887640 1 -- 172.21.15.50:0/138865836 wait complete. 2023-10-30T12:09:29.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.841+0000 7f231e887640 10 monclient: build_initial_monmap 2023-10-30T12:09:29.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.841+0000 7f231e887640 10 monclient: monmap: 2023-10-30T12:09:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:29.873 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.842+0000 7f231e887640 1 librados: starting msgr at 2023-10-30T12:09:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.842+0000 7f231e887640 1 librados: starting objecter 2023-10-30T12:09:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.842+0000 7f231e887640 1 Processor -- start 2023-10-30T12:09:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.842+0000 7f231e887640 1 -- start start 2023-10-30T12:09:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.842+0000 7f231e887640 1 librados: setting wanted keys 2023-10-30T12:09:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.842+0000 7f231e887640 1 librados: calling monclient init 2023-10-30T12:09:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.842+0000 7f231e887640 10 monclient: init 2023-10-30T12:09:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f23181c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): picked mon.noname-c con 0x7f2304003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23040047c0 0x7f23181c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): picked mon.noname-a con 0x7f23040047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2304005360 0x7f2318129f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): picked mon.noname-b con 0x7f2304005360 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231812da00 con 0x7f23040047c0 2023-10-30T12:09:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2318071d50 con 0x7f2304005360 2023-10-30T12:09:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23181324c0 con 0x7f2304003970 2023-10-30T12:09:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231e887640 10 monclient(hunting): authenticate will time out at 1415.026245s 2023-10-30T12:09:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f23181c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f23181c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52248/0 (socket says 172.21.15.50:52248) 2023-10-30T12:09:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 1 -- 172.21.15.50:0/2521549818 learned_addr learned my addr 172.21.15.50:0/2521549818 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 10 monclient(hunting): get_auth_request con 0x7f2304003970 auth_method 0 2023-10-30T12:09:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23040047c0 0x7f23181c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:29.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:29.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2304005360 0x7f2318129f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:29.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): get_auth_request con 0x7f23040047c0 auth_method 0 2023-10-30T12:09:29.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:29.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 10 monclient(hunting): get_auth_request con 0x7f2304005360 auth_method 0 2023-10-30T12:09:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 1 -- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 msgr2=0x7f23181c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:09:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 1 -- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 msgr2=0x7f23181c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:09:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f23181c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f2317fff640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f23181c5b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f231ce24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 10 monclient(hunting): handle_auth_done global_id 134097 payload 824 2023-10-30T12:09:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 1 -- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 msgr2=0x7f23181c5b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f23181c5b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 1 -- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2304005360 msgr2=0x7f2318129f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.843+0000 7f23177fe640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2304005360 0x7f2318129f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f23177fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f23177fe640 1 monclient: found mon.noname-a 2023-10-30T12:09:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f23177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f23177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f23177fe640 1 -- 172.21.15.50:0/2521549818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231812df30 con 0x7f23040047c0 2023-10-30T12:09:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f23177fe640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23040047c0 0x7f23181c60b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f230c006d00 tx=0x7f230c0107b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 10 client.?.objecter ms_handle_connect 0x7f23040047c0 2023-10-30T12:09:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 1 -- 172.21.15.50:0/2521549818 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c004230 con 0x7f23040047c0 2023-10-30T12:09:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 10 monclient: dump: 2023-10-30T12:09:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:29.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:29.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:29.888 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:29.888 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:29.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 10 monclient: _finish_auth 0 2023-10-30T12:09:29.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:09:29.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f2314ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:29.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 5 monclient: authenticate success, global_id 134097 2023-10-30T12:09:29.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 10 monclient: _renew_subs 2023-10-30T12:09:29.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231812d690 con 0x7f23040047c0 2023-10-30T12:09:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 10 client.134097.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 10 monclient: _renew_subs 2023-10-30T12:09:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23180a2b90 con 0x7f23040047c0 2023-10-30T12:09:29.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.844+0000 7f231e887640 1 librados: init done 2023-10-30T12:09:29.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.845+0000 7f2314ff9640 1 -- 172.21.15.50:0/2521549818 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c018040 con 0x7f23040047c0 2023-10-30T12:09:29.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.845+0000 7f2314ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:29.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.845+0000 7f2314ff9640 1 -- 172.21.15.50:0/2521549818 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c012e60 con 0x7f23040047c0 2023-10-30T12:09:29.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.845+0000 7f2314ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:29.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.845+0000 7f2314ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:29.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.845+0000 7f2314ff9640 10 monclient: dump: 2023-10-30T12:09:29.892 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:29.892 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:29.892 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.893 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:29.893 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:29.893 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:29.893 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:29.893 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:29.894 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:29.894 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:29.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.845+0000 7f23167fc640 4 set_mon_vals no callback set 2023-10-30T12:09:29.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2314ff9640 1 -- 172.21.15.50:0/2521549818 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f230c005990 con 0x7f23040047c0 2023-10-30T12:09:29.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f231e887640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:29.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f231e887640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:29.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f231e887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:29.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2314ff9640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2300039b00 0x7f230003bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:29.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f8005040 con 0x7f23040047c0 2023-10-30T12:09:29.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2314ff9640 1 -- 172.21.15.50:0/2521549818 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(310..310 src has 1..310) v4 ==== 7045+0+0 (secure 0 0 0) 0x7f230c04d870 con 0x7f23040047c0 2023-10-30T12:09:29.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2314ff9640 10 client.134097.objecter ms_dispatch 0x7f2318000b90 osd_map(310..310 src has 1..310) v4 2023-10-30T12:09:29.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2314ff9640 3 client.134097.objecter handle_osd_map got epochs [310,310] > 0 2023-10-30T12:09:29.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2314ff9640 3 client.134097.objecter handle_osd_map decoding full epoch 310 2023-10-30T12:09:29.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2314ff9640 20 client.134097.objecter dump_active .. 0 homeless 2023-10-30T12:09:29.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2317fff640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2300039b00 0x7f230003bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:29.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.846+0000 7f2317fff640 10 monclient: get_auth_request con 0x7f2300039b00 auth_method 0 2023-10-30T12:09:29.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.847+0000 7f2317fff640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2300039b00 0x7f230003bfc0 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 crypto rx=0x7f2308012a70 tx=0x7f2308030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:29.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.847+0000 7f2314ff9640 10 client.134097.objecter ms_handle_connect 0x7f2300039b00 2023-10-30T12:09:29.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.851+0000 7f2314ff9640 1 -- 172.21.15.50:0/2521549818 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f230c01b760 con 0x7f23040047c0 2023-10-30T12:09:29.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.851+0000 7f2314ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:29.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:29.851+0000 7f2314ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:30.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.100+0000 7f231e887640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:30.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.100+0000 7f231e887640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:30.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.100+0000 7f231e887640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:30.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.100+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f22f80055c0 con 0x7f23040047c0 2023-10-30T12:09:30.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.100+0000 7f2314ff9640 1 -- 172.21.15.50:0/2521549818 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f230c021080 con 0x7f23040047c0 2023-10-30T12:09:30.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.100+0000 7f2314ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:30.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.100+0000 7f2314ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:30.103 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":84,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:30.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 10 librados: watch_flush enter 2023-10-30T12:09:30.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 10 librados: watch_flush exit 2023-10-30T12:09:30.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 20 client.134097.objecter shutdown clearing up homeless session... 2023-10-30T12:09:30.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 10 client.134097.objecter successfully canceled tick 2023-10-30T12:09:30.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2300039b00 msgr2=0x7f230003bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:30.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2300039b00 0x7f230003bfc0 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 crypto rx=0x7f2308012a70 tx=0x7f2308030000 comp rx=0 tx=0).stop 2023-10-30T12:09:30.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 10 monclient: shutdown 2023-10-30T12:09:30.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:30.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23040047c0 msgr2=0x7f23181c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:30.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23040047c0 0x7f23181c60b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f230c006d00 tx=0x7f230c0107b0 comp rx=0 tx=0).stop 2023-10-30T12:09:30.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 shutdown_connections 2023-10-30T12:09:30.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2300039b00 0x7f230003bfc0 unknown :-1 s=CLOSED pgs=4068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:30.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2304005360 0x7f2318129f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:30.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23040047c0 0x7f23181c60b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:30.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 --2- 172.21.15.50:0/2521549818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2304003970 0x7f23181c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:30.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 >> 172.21.15.50:0/2521549818 conn(0x7f23180a05c0 msgr2=0x7f231806da70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:30.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 shutdown_connections 2023-10-30T12:09:30.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 -- 172.21.15.50:0/2521549818 wait complete. 2023-10-30T12:09:30.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:30.102+0000 7f231e887640 1 librados: shutdown 2023-10-30T12:09:33.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:33.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.230+0000 7f7630db6640 10 monclient: get_monmap_and_config 2023-10-30T12:09:33.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.230+0000 7f7630db6640 10 monclient: build_initial_monmap 2023-10-30T12:09:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.230+0000 7f7630db6640 10 monclient: monmap: 2023-10-30T12:09:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:33.231184+0000 2023-10-30T12:09:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:33.231184+0000 2023-10-30T12:09:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:33.234 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:33.234 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.230+0000 7f7630db6640 1 Processor -- start 2023-10-30T12:09:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.230+0000 7f7630db6640 1 -- start start 2023-10-30T12:09:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.230+0000 7f7630db6640 10 monclient: init 2023-10-30T12:09:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-10-30T12:09:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c0a4b60 0x7f762c1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient(hunting): picked mon.noname-b con 0x7f762c0a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f762c099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f762c132be0 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762c0a4f40 con 0x7f762c132be0 2023-10-30T12:09:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762c0a4150 con 0x7f762c099a40 2023-10-30T12:09:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0a3d70 con 0x7f762c0a4b60 2023-10-30T12:09:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7630db6640 10 monclient(hunting): authenticate will time out at 1418.414320s 2023-10-30T12:09:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c0a4b60 0x7f762c1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c0a4b60 0x7f762c1326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52262/0 (socket says 172.21.15.50:52262) 2023-10-30T12:09:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 1 -- 172.21.15.50:0/1319973569 learned_addr learned my addr 172.21.15.50:0/1319973569 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762ad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f762c132be0 0x7f762c0a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58994/0 (socket says 172.21.15.50:58994) 2023-10-30T12:09:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): get_auth_request con 0x7f762c0a4b60 auth_method 0 2023-10-30T12:09:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient(hunting): handle_auth_done global_id 114341 payload 274 2023-10-30T12:09:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 1 -- 172.21.15.50:0/1319973569 >> v1:172.21.15.118:6789/0 conn(0x7f762c099a40 legacy=0x7f762c0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 1 -- 172.21.15.50:0/1319973569 >> v1:172.21.15.50:6789/0 conn(0x7f762c132be0 legacy=0x7f762c0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:09:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 1 monclient: found mon.noname-b 2023-10-30T12:09:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f762a59c640 1 -- 172.21.15.50:0/1319973569 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0a3170 con 0x7f762c0a4b60 2023-10-30T12:09:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7629d9b640 1 -- 172.21.15.50:0/1319973569 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863697763 0 0) 0x7f762c0a4f40 con 0x7f762c132be0 2023-10-30T12:09:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.231+0000 7f7629d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:09:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f762a59c640 1 --2- 172.21.15.50:0/1319973569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c0a4b60 0x7f762c1326a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f761c0098a0 tx=0x7f761c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fc728359926aa74 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 1 -- 172.21.15.50:0/1319973569 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c031640 con 0x7f762c0a4b60 2023-10-30T12:09:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: dump: 2023-10-30T12:09:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.251 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: _finish_auth 0 2023-10-30T12:09:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:09:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 1 -- 172.21.15.50:0/1319973569 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c0317a0 con 0x7f762c0a4b60 2023-10-30T12:09:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7630db6640 5 monclient: authenticate success, global_id 114341 2023-10-30T12:09:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 1 -- 172.21.15.50:0/1319973569 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c031b30 con 0x7f762c0a4b60 2023-10-30T12:09:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7630db6640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7630db6640 4 set_mon_vals no callback set 2023-10-30T12:09:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.232+0000 7f7629d9b640 10 monclient: dump: 2023-10-30T12:09:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.255 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 10 monclient: shutdown 2023-10-30T12:09:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 1 -- 172.21.15.50:0/1319973569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c0a4b60 msgr2=0x7f762c1326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 1 --2- 172.21.15.50:0/1319973569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c0a4b60 0x7f762c1326a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f761c0098a0 tx=0x7f761c02b8f0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 1 -- 172.21.15.50:0/1319973569 shutdown_connections 2023-10-30T12:09:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 1 --2- 172.21.15.50:0/1319973569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c0a4b60 0x7f762c1326a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 1 -- 172.21.15.50:0/1319973569 >> 172.21.15.50:0/1319973569 conn(0x7f762c0a8060 msgr2=0x7f762c0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 1 -- 172.21.15.50:0/1319973569 shutdown_connections 2023-10-30T12:09:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.233+0000 7f7630db6640 1 -- 172.21.15.50:0/1319973569 wait complete. 2023-10-30T12:09:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 10 monclient: build_initial_monmap 2023-10-30T12:09:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 10 monclient: monmap: 2023-10-30T12:09:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:33.259 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 1 librados: starting msgr at 2023-10-30T12:09:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 1 librados: starting objecter 2023-10-30T12:09:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 1 Processor -- start 2023-10-30T12:09:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 1 -- start start 2023-10-30T12:09:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 1 librados: setting wanted keys 2023-10-30T12:09:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 1 librados: calling monclient init 2023-10-30T12:09:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.234+0000 7f7630db6640 10 monclient: init 2023-10-30T12:09:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:09:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f762c0a4b60 0x7f762c136be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): picked mon.noname-b con 0x7f762c0a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c132be0 0x7f762c137120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): picked mon.noname-c con 0x7f762c132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f762c137660 0x7f762c1d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): picked mon.noname-a con 0x7f762c137660 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0a0330 con 0x7f762c137660 2023-10-30T12:09:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c13d630 con 0x7f762c0a4b60 2023-10-30T12:09:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c13a9e0 con 0x7f762c132be0 2023-10-30T12:09:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7630db6640 10 monclient(hunting): authenticate will time out at 1418.418340s 2023-10-30T12:09:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c132be0 0x7f762c137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f762c137660 0x7f762c1d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c132be0 0x7f762c137120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52274/0 (socket says 172.21.15.50:52274) 2023-10-30T12:09:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 -- 172.21.15.50:0/2919592058 learned_addr learned my addr 172.21.15.50:0/2919592058 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): get_auth_request con 0x7f762c132be0 auth_method 0 2023-10-30T12:09:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 10 monclient(hunting): get_auth_request con 0x7f762c137660 auth_method 0 2023-10-30T12:09:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762a59c640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f762c0a4b60 0x7f762c136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762a59c640 10 monclient(hunting): get_auth_request con 0x7f762c0a4b60 auth_method 0 2023-10-30T12:09:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f762a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient(hunting): handle_auth_done global_id 114344 payload 824 2023-10-30T12:09:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 -- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f762c0a4b60 msgr2=0x7f762c136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f762c0a4b60 0x7f762c136be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 -- 172.21.15.50:0/2919592058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f762c137660 msgr2=0x7f762c1d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f762c137660 0x7f762c1d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:09:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 monclient: found mon.noname-c 2023-10-30T12:09:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.235+0000 7f7621d9b640 1 -- 172.21.15.50:0/2919592058 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c0094d0 con 0x7f762c132be0 2023-10-30T12:09:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7621d9b640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c132be0 0x7f762c137120 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f761400c430 tx=0x7f761400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 client.?.objecter ms_handle_connect 0x7f762c132be0 2023-10-30T12:09:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 1 -- 172.21.15.50:0/2919592058 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76140052d0 con 0x7f762c132be0 2023-10-30T12:09:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: dump: 2023-10-30T12:09:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.272 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: _finish_auth 0 2023-10-30T12:09:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: _check_auth_tickets 2023-10-30T12:09:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 5 monclient: authenticate success, global_id 114344 2023-10-30T12:09:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 10 monclient: _renew_subs 2023-10-30T12:09:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 1 -- 172.21.15.50:0/2919592058 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7614005430 con 0x7f762c132be0 2023-10-30T12:09:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c0a18d0 con 0x7f762c132be0 2023-10-30T12:09:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 1 -- 172.21.15.50:0/2919592058 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614003830 con 0x7f762c132be0 2023-10-30T12:09:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f762959a640 4 set_mon_vals no callback set 2023-10-30T12:09:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f76237fe640 10 monclient: dump: 2023-10-30T12:09:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.277 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 10 client.114344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 10 monclient: _renew_subs 2023-10-30T12:09:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c13a420 con 0x7f762c132be0 2023-10-30T12:09:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.236+0000 7f7630db6640 1 librados: init done 2023-10-30T12:09:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f7630db6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f7630db6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:33.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f7630db6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f76237fe640 1 -- 172.21.15.50:0/2919592058 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7614003990 con 0x7f762c132be0 2023-10-30T12:09:33.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c13a610 con 0x7f762c132be0 2023-10-30T12:09:33.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f76237fe640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f760c0392f0 0x7f760c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f76237fe640 1 -- 172.21.15.50:0/2919592058 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(313..313 src has 1..313) v4 ==== 6021+0+0 (secure 0 0 0) 0x7f761404a090 con 0x7f762c132be0 2023-10-30T12:09:33.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f76237fe640 10 client.114344.objecter ms_dispatch 0x7f762c000b90 osd_map(313..313 src has 1..313) v4 2023-10-30T12:09:33.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f76237fe640 3 client.114344.objecter handle_osd_map got epochs [313,313] > 0 2023-10-30T12:09:33.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f76237fe640 3 client.114344.objecter handle_osd_map decoding full epoch 313 2023-10-30T12:09:33.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f76237fe640 20 client.114344.objecter dump_active .. 0 homeless 2023-10-30T12:09:33.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f762a59c640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f760c0392f0 0x7f760c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.237+0000 7f762a59c640 10 monclient: get_auth_request con 0x7f760c0392f0 auth_method 0 2023-10-30T12:09:33.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.238+0000 7f76237fe640 10 client.114344.objecter ms_handle_connect 0x7f760c0392f0 2023-10-30T12:09:33.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.238+0000 7f762a59c640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f760c0392f0 0x7f760c03b7b0 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7f761c036000 tx=0x7f761c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.241+0000 7f76237fe640 1 -- 172.21.15.50:0/2919592058 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f761400e280 con 0x7f762c132be0 2023-10-30T12:09:33.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.242+0000 7f76237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:33.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.242+0000 7f76237fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:33.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ teFullPP2 2023-10-30T12:09:33.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3021 ms) 2023-10-30T12:09:33.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-10-30T12:09:33.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7072 ms) 2023-10-30T12:09:33.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-10-30T12:09:33.283 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (21171 ms) 2023-10-30T12:09:33.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-10-30T12:09:33.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6737 ms) 2023-10-30T12:09:33.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-10-30T12:09:33.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7041 ms) 2023-10-30T12:09:33.284 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-10-30T12:09:33.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6969 ms) 2023-10-30T12:09:33.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-10-30T12:09:33.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (12064 ms) 2023-10-30T12:09:33.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (191847 ms total) 2023-10-30T12:09:33.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: 2023-10-30T12:09:33.285 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-10-30T12:09:33.286 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (459085 ms total) 2023-10-30T12:09:33.286 INFO:tasks.workunit.client.0.smithi118.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-10-30T12:09:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.490+0000 7f7630db6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.490+0000 7f7630db6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.490+0000 7f7630db6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.490+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f762c04e160 con 0x7f762c132be0 2023-10-30T12:09:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.490+0000 7f76237fe640 1 -- 172.21.15.50:0/2919592058 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f7614019a70 con 0x7f762c132be0 2023-10-30T12:09:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.490+0000 7f76237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.490+0000 7f76237fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:33.494 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 10 librados: watch_flush enter 2023-10-30T12:09:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 10 librados: watch_flush exit 2023-10-30T12:09:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 20 client.114344.objecter shutdown clearing up homeless session... 2023-10-30T12:09:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 10 client.114344.objecter successfully canceled tick 2023-10-30T12:09:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f760c0392f0 msgr2=0x7f760c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f760c0392f0 0x7f760c03b7b0 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7f761c036000 tx=0x7f761c008030 comp rx=0 tx=0).stop 2023-10-30T12:09:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 10 monclient: shutdown 2023-10-30T12:09:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c132be0 msgr2=0x7f762c137120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c132be0 0x7f762c137120 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f761400c430 tx=0x7f761400c8f0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 shutdown_connections 2023-10-30T12:09:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f760c0392f0 0x7f760c03b7b0 unknown :-1 s=CLOSED pgs=4069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f762c137660 0x7f762c1d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f762c132be0 0x7f762c137120 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 --2- 172.21.15.50:0/2919592058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f762c0a4b60 0x7f762c136be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 >> 172.21.15.50:0/2919592058 conn(0x7f762c0a8060 msgr2=0x7f762c0a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 shutdown_connections 2023-10-30T12:09:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 -- 172.21.15.50:0/2919592058 wait complete. 2023-10-30T12:09:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.492+0000 7f7630db6640 1 librados: shutdown 2023-10-30T12:09:33.504 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:09:33.504 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:09:33.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:09:33.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.615+0000 7f8e32297640 10 monclient: get_monmap_and_config 2023-10-30T12:09:33.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.615+0000 7f8e32297640 10 monclient: build_initial_monmap 2023-10-30T12:09:33.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.615+0000 7f8e32297640 10 monclient: monmap: 2023-10-30T12:09:33.618 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:33.618 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.618 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:33.616939+0000 2023-10-30T12:09:33.618 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:33.616939+0000 2023-10-30T12:09:33.618 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:33.620 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:33.620 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.616+0000 7f8e32297640 1 Processor -- start 2023-10-30T12:09:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.616+0000 7f8e32297640 1 -- start start 2023-10-30T12:09:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.616+0000 7f8e32297640 10 monclient: init 2023-10-30T12:09:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.616+0000 7f8e32297640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:09:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e2c097780 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e2c073b10 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8e2c09ccd0 0x7f8e2c13e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 10 monclient(hunting): picked mon.noname-c con 0x7f8e2c09ccd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2c071ec0 con 0x7f8e2c073b10 2023-10-30T12:09:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2c09bd10 con 0x7f8e2c097780 2023-10-30T12:09:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2c09bff0 con 0x7f8e2c09ccd0 2023-10-30T12:09:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e32297640 10 monclient(hunting): authenticate will time out at 1418.800083s 2023-10-30T12:09:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2b7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8e2c073b10 0x7f8e2c13aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59004/0 (socket says 172.21.15.50:59004) 2023-10-30T12:09:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2b7fe640 1 -- 172.21.15.50:0/2541398155 learned_addr learned my addr 172.21.15.50:0/2541398155 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 1 --2- 172.21.15.50:0/2541398155 >> v2:172.21.15.118:3300/0 conn(0x7f8e2c09ccd0 0x7f8e2c13e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): get_auth_request con 0x7f8e2c09ccd0 auth_method 0 2023-10-30T12:09:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 1 -- 172.21.15.50:0/2541398155 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621163255 0 0) 0x7f8e2c071ec0 con 0x7f8e2c073b10 2023-10-30T12:09:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): _negotiate 2023-10-30T12:09:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): my global_id is 134106 2023-10-30T12:09:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 1 -- 172.21.15.50:0/2541398155 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e18003520 con 0x7f8e2c073b10 2023-10-30T12:09:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 1 -- 172.21.15.50:0/2541398155 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139417581 0 0) 0x7f8e2c09bd10 con 0x7f8e2c097780 2023-10-30T12:09:33.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): _negotiate 2023-10-30T12:09:33.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 10 monclient(hunting): my global_id is 94726 2023-10-30T12:09:33.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e2affd640 1 -- 172.21.15.50:0/2541398155 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c071ec0 con 0x7f8e2c097780 2023-10-30T12:09:33.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient(hunting): handle_auth_done global_id 94723 payload 274 2023-10-30T12:09:33.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 1 -- 172.21.15.50:0/2541398155 >> v1:172.21.15.118:6789/0 conn(0x7f8e2c097780 legacy=0x7f8e2c097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 1 -- 172.21.15.50:0/2541398155 >> v1:172.21.15.50:6789/0 conn(0x7f8e2c073b10 legacy=0x7f8e2c13aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient: _finish_hunting 0 2023-10-30T12:09:33.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 1 monclient: found mon.noname-c 2023-10-30T12:09:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:09:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.617+0000 7f8e30834640 1 -- 172.21.15.50:0/2541398155 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c071b20 con 0x7f8e2c09ccd0 2023-10-30T12:09:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e30834640 1 --2- 172.21.15.50:0/2541398155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 0x7f8e2c13e1c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009d90 tx=0x7f8e2402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5c46a4eed2fdf10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 1 -- 172.21.15.50:0/2541398155 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2402cbd0 con 0x7f8e2c09ccd0 2023-10-30T12:09:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: dump: 2023-10-30T12:09:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.639 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.639 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.639 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.639 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.639 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.640 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.640 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: _finish_auth 0 2023-10-30T12:09:33.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:33.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:33.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e32297640 5 monclient: authenticate success, global_id 94723 2023-10-30T12:09:33.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 1 -- 172.21.15.50:0/2541398155 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2402cd30 con 0x7f8e2c09ccd0 2023-10-30T12:09:33.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e32297640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:33.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:33.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e32297640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:33.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e32297640 4 set_mon_vals no callback set 2023-10-30T12:09:33.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 1 -- 172.21.15.50:0/2541398155 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2402cea0 con 0x7f8e2c09ccd0 2023-10-30T12:09:33.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:33.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.618+0000 7f8e2affd640 10 monclient: dump: 2023-10-30T12:09:33.642 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.642 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.643 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.643 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.643 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.643 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.644 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.644 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.644 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 10 monclient: shutdown 2023-10-30T12:09:33.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:33.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 -- 172.21.15.50:0/2541398155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 msgr2=0x7f8e2c13e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 --2- 172.21.15.50:0/2541398155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 0x7f8e2c13e1c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009d90 tx=0x7f8e2402be20 comp rx=0 tx=0).stop 2023-10-30T12:09:33.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 -- 172.21.15.50:0/2541398155 shutdown_connections 2023-10-30T12:09:33.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 --2- 172.21.15.50:0/2541398155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 0x7f8e2c13e1c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 -- 172.21.15.50:0/2541398155 >> 172.21.15.50:0/2541398155 conn(0x7f8e2c0a1170 msgr2=0x7f8e2c0a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:33.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 -- 172.21.15.50:0/2541398155 shutdown_connections 2023-10-30T12:09:33.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 -- 172.21.15.50:0/2541398155 wait complete. 2023-10-30T12:09:33.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 10 monclient: build_initial_monmap 2023-10-30T12:09:33.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 10 monclient: monmap: 2023-10-30T12:09:33.646 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:33.646 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:33.646 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:33.647 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:33.647 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:33.647 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.647 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:33.647 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:33.648 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:33.648 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 librados: starting msgr at 2023-10-30T12:09:33.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.619+0000 7f8e32297640 1 librados: starting objecter 2023-10-30T12:09:33.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 Processor -- start 2023-10-30T12:09:33.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 -- start start 2023-10-30T12:09:33.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 librados: setting wanted keys 2023-10-30T12:09:33.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 librados: calling monclient init 2023-10-30T12:09:33.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient: init 2023-10-30T12:09:33.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:33.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:33.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e2c073b10 0x7f8e2c1c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): picked mon.noname-c con 0x7f8e2c073b10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:33.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c097780 0x7f8e2c094850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): picked mon.noname-a con 0x7f8e2c097780 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:33.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 0x7f8e2c1ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): picked mon.noname-b con 0x7f8e2c09ccd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:33.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c08bcf0 con 0x7f8e2c097780 2023-10-30T12:09:33.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c08b0d0 con 0x7f8e2c09ccd0 2023-10-30T12:09:33.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c13cea0 con 0x7f8e2c073b10 2023-10-30T12:09:33.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:33.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e32297640 10 monclient(hunting): authenticate will time out at 1418.803737s 2023-10-30T12:09:33.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c097780 0x7f8e2c094850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e2c073b10 0x7f8e2c1c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c097780 0x7f8e2c094850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52270/0 (socket says 172.21.15.50:52270) 2023-10-30T12:09:33.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 1 -- 172.21.15.50:0/1018687163 learned_addr learned my addr 172.21.15.50:0/1018687163 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:33.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 10 monclient(hunting): get_auth_request con 0x7f8e2c097780 auth_method 0 2023-10-30T12:09:33.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 10 monclient(hunting): get_auth_request con 0x7f8e2c073b10 auth_method 0 2023-10-30T12:09:33.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.620+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e30834640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 0x7f8e2c1ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e30834640 10 monclient(hunting): get_auth_request con 0x7f8e2c09ccd0 auth_method 0 2023-10-30T12:09:33.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e30834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e30834640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e30834640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 10 monclient(hunting): handle_auth_done global_id 134109 payload 824 2023-10-30T12:09:33.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e2c073b10 msgr2=0x7f8e2c1c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e2c073b10 0x7f8e2c1c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 msgr2=0x7f8e2c1ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e2c09ccd0 0x7f8e2c1ca140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:33.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:09:33.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:33.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:33.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 1 -- 172.21.15.50:0/1018687163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24009a70 con 0x7f8e2c097780 2023-10-30T12:09:33.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2b7fe640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c097780 0x7f8e2c094850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c530 tx=0x7f8e1c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 client.?.objecter ms_handle_connect 0x7f8e2c097780 2023-10-30T12:09:33.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:33.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c005ce0 con 0x7f8e2c097780 2023-10-30T12:09:33.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:33.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: dump: 2023-10-30T12:09:33.660 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.660 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.660 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.660 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.660 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.661 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.661 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.661 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.661 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.661 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: _finish_auth 0 2023-10-30T12:09:33.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:09:33.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:33.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 5 monclient: authenticate success, global_id 134109 2023-10-30T12:09:33.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c005e60 con 0x7f8e2c097780 2023-10-30T12:09:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 10 monclient: _renew_subs 2023-10-30T12:09:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2c073f70 con 0x7f8e2c097780 2023-10-30T12:09:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0072b0 con 0x7f8e2c097780 2023-10-30T12:09:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:33.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e28ff9640 10 monclient: dump: 2023-10-30T12:09:33.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.665 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.665 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.665 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.665 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.666 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.666 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 10 client.134109.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:33.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 10 monclient: _renew_subs 2023-10-30T12:09:33.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e2a7fc640 4 set_mon_vals no callback set 2023-10-30T12:09:33.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:33.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2c13c9a0 con 0x7f8e2c097780 2023-10-30T12:09:33.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.621+0000 7f8e32297640 1 librados: init done 2023-10-30T12:09:33.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e32297640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:33.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e32297640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c13e700 0x7f8e2c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e32297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c06a5a0 con 0x7f8e2c13e700 2023-10-30T12:09:33.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8e1c007410 con 0x7f8e2c097780 2023-10-30T12:09:33.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c13e700 0x7f8e2c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient: get_auth_request con 0x7f8e2c13e700 auth_method 0 2023-10-30T12:09:33.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e0c0392f0 0x7f8e0c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f8e1c04a2f0 con 0x7f8e2c097780 2023-10-30T12:09:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 10 client.134109.objecter ms_dispatch 0x7f8e2c000b90 osd_map(315..315 src has 1..315) v4 2023-10-30T12:09:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 3 client.134109.objecter handle_osd_map got epochs [315,315] > 0 2023-10-30T12:09:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 3 client.134109.objecter handle_osd_map decoding full epoch 315 2023-10-30T12:09:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 20 client.134109.objecter dump_active .. 0 homeless 2023-10-30T12:09:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_done global_id 134112 payload 824 2023-10-30T12:09:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 1 -- 172.21.15.50:0/1018687163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e2c04e160 con 0x7f8e2c13e700 2023-10-30T12:09:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e30834640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e0c0392f0 0x7f8e0c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e30834640 10 monclient: get_auth_request con 0x7f8e0c0392f0 auth_method 0 2023-10-30T12:09:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 10 client.134109.objecter ms_handle_connect 0x7f8e2c13e700 2023-10-30T12:09:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e28ff9640 10 client.134109.objecter resend_mon_ops 2023-10-30T12:09:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.622+0000 7f8e2bfff640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c13e700 0x7f8e2c02b430 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400f770 tx=0x7f8e1400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e140050d0 con 0x7f8e2c13e700 2023-10-30T12:09:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 10 monclient: dump: 2023-10-30T12:09:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.674 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 10 client.134109.objecter ms_handle_connect 0x7f8e0c0392f0 2023-10-30T12:09:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e30834640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e0c0392f0 0x7f8e0c03b7b0 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7f8e24036000 tx=0x7f8e24030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8e14005230 con 0x7f8e2c13e700 2023-10-30T12:09:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c13e700 msgr2=0x7f8e2c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c13e700 0x7f8e2c02b430 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400f770 tx=0x7f8e1400fc30 comp rx=0 tx=0).stop 2023-10-30T12:09:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c13e700 msgr2=0x7f8e2c02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.623+0000 7f8e28ff9640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c13e700 0x7f8e2c02b430 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400f770 tx=0x7f8e1400fc30 comp rx=0 tx=0).stop 2023-10-30T12:09:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e32297640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e32297640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c094d90 0x7f8e2c02b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e32297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c09ac90 con 0x7f8e2c094d90 2023-10-30T12:09:33.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c094d90 0x7f8e2c02b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 10 monclient: get_auth_request con 0x7f8e2c094d90 auth_method 0 2023-10-30T12:09:33.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.629+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e2bfff640 10 monclient(hunting): handle_auth_done global_id 134115 payload 824 2023-10-30T12:09:33.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e2bfff640 1 -- 172.21.15.50:0/1018687163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8e2c06ab70 con 0x7f8e2c094d90 2023-10-30T12:09:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 10 client.134109.objecter ms_handle_connect 0x7f8e2c094d90 2023-10-30T12:09:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e2bfff640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c094d90 0x7f8e2c02b970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e14017000 tx=0x7f8e14001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 10 client.134109.objecter resend_mon_ops 2023-10-30T12:09:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14024070 con 0x7f8e2c094d90 2023-10-30T12:09:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 10 monclient: dump: 2023-10-30T12:09:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.684 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f8e14003b50 con 0x7f8e2c094d90 2023-10-30T12:09:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c094d90 msgr2=0x7f8e2c02b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c094d90 0x7f8e2c02b970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e14017000 tx=0x7f8e14001150 comp rx=0 tx=0).stop 2023-10-30T12:09:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c094d90 msgr2=0x7f8e2c02b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e28ff9640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c094d90 0x7f8e2c02b970 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8e14017000 tx=0x7f8e14001150 comp rx=0 tx=0).stop 2023-10-30T12:09:33.685 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:33.686 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:33.686 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:33.686 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:09:33.686 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 88, 2023-10-30T12:09:33.686 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:33.686 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:33.687 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:09:33.687 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:33.687 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.687 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:09:33.687 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:33.688 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:33.688 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:33.688 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:33.688 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:33.688 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:33.689 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:33.689 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:33.689 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:33.689 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:33.689 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:33.689 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:33.690 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:33.690 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.690 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:33.690 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:33.690 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:33.692 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:33.692 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:33.692 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:33.692 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:33.692 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.693 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:33.693 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:33.693 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:33.693 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:33.693 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:33.694 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:33.694 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:33.694 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:33.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:33.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:33.695 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:33.695 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:33.695 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:33.695 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:33.695 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:33.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:33.698 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.698 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:33.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.698 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:33.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.699 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:33.699 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:33.699 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:33.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:33.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:33.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:33.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:33.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:33.701 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:33.702 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:33.702 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:33.702 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:33.702 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:33.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.703 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:33.703 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:33.703 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:33.703 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:33.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:33.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:33.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:33.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:33.705 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:33.706 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:33.706 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:33.706 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:33.707 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:33.708 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:33.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:33.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:33.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:33.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.709 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:33.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:33.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.710 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:33.710 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:33.711 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:33.711 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:33.711 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:33.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.711 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.712 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:33.712 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:33.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.712 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:33.712 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:33.713 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:33.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.713 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.713 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:33.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.714 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:33.714 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:33.714 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:09:33.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.715 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:33.715 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:33.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 10 librados: watch_flush enter 2023-10-30T12:09:33.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 10 librados: watch_flush exit 2023-10-30T12:09:33.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 20 client.134109.objecter shutdown clearing up homeless session... 2023-10-30T12:09:33.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 10 client.134109.objecter successfully canceled tick 2023-10-30T12:09:33.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e0c0392f0 msgr2=0x7f8e0c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8e0c0392f0 0x7f8e0c03b7b0 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7f8e24036000 tx=0x7f8e24030000 comp rx=0 tx=0).stop 2023-10-30T12:09:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 10 monclient: shutdown 2023-10-30T12:09:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c097780 msgr2=0x7f8e2c094850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e30834640 1 -- 172.21.15.50:0/1018687163 reap_dead start 2023-10-30T12:09:33.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c097780 0x7f8e2c094850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c530 tx=0x7f8e1c00c9f0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 shutdown_connections 2023-10-30T12:09:33.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 1 --2- 172.21.15.50:0/1018687163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e2c097780 0x7f8e2c094850 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.630+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 >> 172.21.15.50:0/1018687163 conn(0x7f8e2c0a1170 msgr2=0x7f8e2c13dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:33.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.631+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 shutdown_connections 2023-10-30T12:09:33.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.631+0000 7f8e32297640 1 -- 172.21.15.50:0/1018687163 wait complete. 2023-10-30T12:09:33.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.631+0000 7f8e32297640 1 librados: shutdown 2023-10-30T12:09:33.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:09:33.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:09:33.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:09:33.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.826+0000 7f8a7a0cf640 10 monclient: get_monmap_and_config 2023-10-30T12:09:33.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.826+0000 7f8a7a0cf640 10 monclient: build_initial_monmap 2023-10-30T12:09:33.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.826+0000 7f8a7a0cf640 10 monclient: monmap: 2023-10-30T12:09:33.828 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:33.828 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.828 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:33.827075+0000 2023-10-30T12:09:33.828 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:33.827075+0000 2023-10-30T12:09:33.828 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:33.829 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:33.830 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:33.830 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.826+0000 7f8a7a0cf640 1 Processor -- start 2023-10-30T12:09:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.826+0000 7f8a7a0cf640 1 -- start start 2023-10-30T12:09:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.826+0000 7f8a7a0cf640 10 monclient: init 2023-10-30T12:09:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:09:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8a7406afc0 0x7f8a740a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7406afc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a7406b640 0x7f8a740a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient(hunting): picked mon.noname-b con 0x7f8a7406b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a740a2710 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7406bde0 con 0x7f8a740a2710 2023-10-30T12:09:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a740a2b40 con 0x7f8a7406afc0 2023-10-30T12:09:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7406b3a0 con 0x7f8a7406b640 2023-10-30T12:09:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a7a0cf640 10 monclient(hunting): authenticate will time out at 1419.010243s 2023-10-30T12:09:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a7406b640 0x7f8a740a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a7406b640 0x7f8a740a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52282/0 (socket says 172.21.15.50:52282) 2023-10-30T12:09:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 -- 172.21.15.50:0/3936922120 learned_addr learned my addr 172.21.15.50:0/3936922120 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): get_auth_request con 0x7f8a7406b640 auth_method 0 2023-10-30T12:09:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a737fe640 1 --2- 172.21.15.50:0/3936922120 >> v2:172.21.15.118:3300/0 conn(0x7f8a7406afc0 0x7f8a740a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a737fe640 10 monclient(hunting): get_auth_request con 0x7f8a7406afc0 auth_method 0 2023-10-30T12:09:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_done global_id 114350 payload 274 2023-10-30T12:09:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 -- 172.21.15.50:0/3936922120 >> v2:172.21.15.118:3300/0 conn(0x7f8a7406afc0 msgr2=0x7f8a740a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 --2- 172.21.15.50:0/3936922120 >> v2:172.21.15.118:3300/0 conn(0x7f8a7406afc0 0x7f8a740a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 -- 172.21.15.50:0/3936922120 >> v1:172.21.15.118:6789/0 conn(0x7f8a740a2710 legacy=0x7f8a7413c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:33.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:33.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 monclient: found mon.noname-b 2023-10-30T12:09:33.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:33.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.827+0000 7f8a72ffd640 1 -- 172.21.15.50:0/3936922120 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7409e540 con 0x7f8a7406b640 2023-10-30T12:09:33.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a72ffd640 1 --2- 172.21.15.50:0/3936922120 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a7406b640 0x7f8a740a21d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009870 tx=0x7f8a6802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1189a8bf8325197f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 1 -- 172.21.15.50:0/3936922120 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68031740 con 0x7f8a7406b640 2023-10-30T12:09:33.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: dump: 2023-10-30T12:09:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.848 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.848 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.848 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.848 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 5 monclient: authenticate success, global_id 114350 2023-10-30T12:09:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:33.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 1 -- 172.21.15.50:0/3936922120 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a680318a0 con 0x7f8a7406b640 2023-10-30T12:09:33.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:33.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:33.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 4 set_mon_vals no callback set 2023-10-30T12:09:33.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 1 -- 172.21.15.50:0/3936922120 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68031c30 con 0x7f8a7406b640 2023-10-30T12:09:33.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:33.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a727fc640 10 monclient: dump: 2023-10-30T12:09:33.851 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.851 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.852 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.852 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.852 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.852 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.852 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.853 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.853 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.853 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 10 monclient: shutdown 2023-10-30T12:09:33.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:33.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/3936922120 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a7406b640 msgr2=0x7f8a740a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.828+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/3936922120 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a7406b640 0x7f8a740a21d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009870 tx=0x7f8a6802b7e0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/3936922120 shutdown_connections 2023-10-30T12:09:33.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/3936922120 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a7406b640 0x7f8a740a21d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/3936922120 >> v2:172.21.15.118:3300/0 conn(0x7f8a7406afc0 0x7f8a740a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/3936922120 >> 172.21.15.50:0/3936922120 conn(0x7f8a740a6200 msgr2=0x7f8a740a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:33.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/3936922120 shutdown_connections 2023-10-30T12:09:33.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/3936922120 wait complete. 2023-10-30T12:09:33.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 10 monclient: build_initial_monmap 2023-10-30T12:09:33.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 10 monclient: monmap: 2023-10-30T12:09:33.855 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:33.856 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:33.856 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:33.856 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:33.856 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:33.856 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.857 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:33.857 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:33.857 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:33.857 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 librados: starting msgr at 2023-10-30T12:09:33.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 librados: starting objecter 2023-10-30T12:09:33.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.829+0000 7f8a7a0cf640 1 Processor -- start 2023-10-30T12:09:33.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 -- start start 2023-10-30T12:09:33.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 librados: setting wanted keys 2023-10-30T12:09:33.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 librados: calling monclient init 2023-10-30T12:09:33.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient: init 2023-10-30T12:09:33.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:33.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:33.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7406afc0 0x7f8a741c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7406afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:33.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a7406b640 0x7f8a741c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): picked mon.noname-b con 0x7f8a7406b640 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:33.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a740a2710 0x7f8a741c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8a740a2710 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:33.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a741c7440 con 0x7f8a7406afc0 2023-10-30T12:09:33.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a741a8b00 con 0x7f8a7406b640 2023-10-30T12:09:33.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a740706c0 con 0x7f8a740a2710 2023-10-30T12:09:33.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:33.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a7a0cf640 10 monclient(hunting): authenticate will time out at 1419.013592s 2023-10-30T12:09:33.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a740a2710 0x7f8a741c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7406afc0 0x7f8a741c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a740a2710 0x7f8a741c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52292/0 (socket says 172.21.15.50:52292) 2023-10-30T12:09:33.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 1 -- 172.21.15.50:0/4068430730 learned_addr learned my addr 172.21.15.50:0/4068430730 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:33.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): get_auth_request con 0x7f8a740a2710 auth_method 0 2023-10-30T12:09:33.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 10 monclient(hunting): get_auth_request con 0x7f8a7406afc0 auth_method 0 2023-10-30T12:09:33.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a72ffd640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a7406b640 0x7f8a741c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a72ffd640 10 monclient(hunting): get_auth_request con 0x7f8a7406b640 auth_method 0 2023-10-30T12:09:33.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.830+0000 7f8a73fff640 10 monclient(hunting): handle_auth_done global_id 114353 payload 824 2023-10-30T12:09:33.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a7406b640 msgr2=0x7f8a741c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a7406b640 0x7f8a741c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7406afc0 msgr2=0x7f8a741c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7406afc0 0x7f8a741c3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:33.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 1 monclient: found mon.noname-c 2023-10-30T12:09:33.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:33.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 1 -- 172.21.15.50:0/4068430730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a680094d0 con 0x7f8a740a2710 2023-10-30T12:09:33.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a72ffd640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a7406b640 0x7f8a741c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:09:33.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a73fff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a740a2710 0x7f8a741c6f00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c780 tx=0x7f8a6c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 client.?.objecter ms_handle_connect 0x7f8a740a2710 2023-10-30T12:09:33.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:33.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c004b60 con 0x7f8a740a2710 2023-10-30T12:09:33.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:33.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: dump: 2023-10-30T12:09:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.871 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.871 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.871 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.871 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.871 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: _finish_auth 0 2023-10-30T12:09:33.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:33.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:33.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 5 monclient: authenticate success, global_id 114353 2023-10-30T12:09:33.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c00ce90 con 0x7f8a740a2710 2023-10-30T12:09:33.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:33.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 10 monclient: _renew_subs 2023-10-30T12:09:33.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c00a850 con 0x7f8a740a2710 2023-10-30T12:09:33.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a741aae00 con 0x7f8a740a2710 2023-10-30T12:09:33.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a5bfff640 10 monclient: dump: 2023-10-30T12:09:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.876 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a71ffb640 4 set_mon_vals no callback set 2023-10-30T12:09:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 10 client.114353.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 10 monclient: _renew_subs 2023-10-30T12:09:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:33.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a741ab3c0 con 0x7f8a740a2710 2023-10-30T12:09:33.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.831+0000 7f8a7a0cf640 1 librados: init done 2023-10-30T12:09:33.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8a6c00a9b0 con 0x7f8a740a2710 2023-10-30T12:09:33.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a7a0cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:33.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741c9f00 0x7f8a741ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a7a0cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a74073ff0 con 0x7f8a741c9f00 2023-10-30T12:09:33.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741c9f00 0x7f8a741ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient: get_auth_request con 0x7f8a741c9f00 auth_method 0 2023-10-30T12:09:33.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f8a6c04ad60 con 0x7f8a740a2710 2023-10-30T12:09:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 10 client.114353.objecter ms_dispatch 0x7f8a74000b90 osd_map(315..315 src has 1..315) v4 2023-10-30T12:09:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 3 client.114353.objecter handle_osd_map got epochs [315,315] > 0 2023-10-30T12:09:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 3 client.114353.objecter handle_osd_map decoding full epoch 315 2023-10-30T12:09:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 10 monclient(hunting): handle_auth_done global_id 114356 payload 824 2023-10-30T12:09:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 1 -- 172.21.15.50:0/4068430730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a7409df80 con 0x7f8a741c9f00 2023-10-30T12:09:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 20 client.114353.objecter dump_active .. 0 homeless 2023-10-30T12:09:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a72ffd640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 10 client.114353.objecter ms_handle_connect 0x7f8a741c9f00 2023-10-30T12:09:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a72ffd640 10 monclient: get_auth_request con 0x7f8a600392f0 auth_method 0 2023-10-30T12:09:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 10 client.114353.objecter resend_mon_ops 2023-10-30T12:09:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a737fe640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741c9f00 0x7f8a741ca2e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a741a8dd0 tx=0x7f8a64010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a64010e60 con 0x7f8a741c9f00 2023-10-30T12:09:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.832+0000 7f8a5bfff640 10 monclient: dump: 2023-10-30T12:09:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.884 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8a64004240 con 0x7f8a741c9f00 2023-10-30T12:09:33.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:33.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:33.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a72ffd640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 secure :-1 s=READY pgs=4071 cs=0 l=1 rev1=1 crypto rx=0x7f8a68035000 tx=0x7f8a68008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741c9f00 msgr2=0x7f8a741ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741c9f00 0x7f8a741ca2e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a741a8dd0 tx=0x7f8a64010c10 comp rx=0 tx=0).stop 2023-10-30T12:09:33.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741c9f00 msgr2=0x7f8a741ca2e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:33.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741c9f00 0x7f8a741ca2e0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a741a8dd0 tx=0x7f8a64010c10 comp rx=0 tx=0).stop 2023-10-30T12:09:33.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.833+0000 7f8a5bfff640 10 client.114353.objecter ms_handle_connect 0x7f8a600392f0 2023-10-30T12:09:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a7a0cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741ca820 0x7f8a741a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:33.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a7a0cf640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:33.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7406a6f0 con 0x7f8a741ca820 2023-10-30T12:09:33.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741ca820 0x7f8a741a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:33.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient: get_auth_request con 0x7f8a741ca820 auth_method 0 2023-10-30T12:09:33.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 10 monclient(hunting): handle_auth_done global_id 114359 payload 824 2023-10-30T12:09:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.839+0000 7f8a737fe640 1 -- 172.21.15.50:0/4068430730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8a7409f0b0 con 0x7f8a741ca820 2023-10-30T12:09:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a737fe640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741ca820 0x7f8a741a3bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8a64000c00 tx=0x7f8a640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 10 client.114353.objecter ms_handle_connect 0x7f8a741ca820 2023-10-30T12:09:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 10 client.114353.objecter resend_mon_ops 2023-10-30T12:09:33.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a64025070 con 0x7f8a741ca820 2023-10-30T12:09:33.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:33.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:33.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 10 monclient: dump: 2023-10-30T12:09:33.892 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:33.893 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:33.893 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.893 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:33.893 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:33.893 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:33.894 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:33.894 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:33.894 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:33.894 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:33.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f8a64003b70 con 0x7f8a741ca820 2023-10-30T12:09:33.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:33.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:33.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741ca820 msgr2=0x7f8a741a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741ca820 0x7f8a741a3bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8a64000c00 tx=0x7f8a640046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741ca820 msgr2=0x7f8a741a3bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:33.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a5bfff640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a741ca820 0x7f8a741a3bd0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8a64000c00 tx=0x7f8a640046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.897 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:33.897 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:33.898 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:33.898 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:09:33.898 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 88, 2023-10-30T12:09:33.898 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:33.899 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:33.899 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:09:33.899 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:33.899 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.899 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:09:33.899 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:33.900 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:33.900 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:33.900 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:33.900 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:33.900 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:33.901 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:33.901 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:33.901 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:33.901 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:33.901 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:33.902 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:33.902 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:33.902 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.902 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:33.902 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:33.903 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:33.903 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:33.903 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:33.903 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:33.903 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:33.904 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:33.904 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:33.904 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:33.904 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:33.904 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:33.904 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.905 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:33.905 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:33.905 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:33.905 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:33.906 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:33.906 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:33.906 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:33.906 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:33.906 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:33.907 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:33.907 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:33.907 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:33.907 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:33.907 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:33.908 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:33.908 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:33.908 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:33.908 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:33.908 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:33.909 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:33.909 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:33.909 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:33.909 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:33.909 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:33.909 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:33.910 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:33.910 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:33.910 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.910 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:33.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.911 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:33.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.911 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:33.911 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:33.911 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:33.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:33.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:33.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:33.912 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:33.913 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:33.913 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.914 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.914 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:33.914 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:33.914 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:33.915 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:33.915 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:33.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.915 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:33.916 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:33.916 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:33.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:33.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.916 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:33.917 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:33.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:33.919 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:33.919 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:33.919 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:33.919 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:33.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.920 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:33.920 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:33.920 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:33.920 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:33.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:33.921 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:33.921 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.922 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:33.922 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:33.922 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:33.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.922 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.923 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:33.923 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:33.923 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:33.923 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:33.924 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:33.924 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.924 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.924 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:33.924 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:33.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.925 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:33.925 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:33.925 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:33.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.926 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.926 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:09:33.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.926 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:33.926 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:33.927 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:09:33.927 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.927 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:33.927 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:33.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:33.927 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:33.928 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:33.928 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:09:33.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:33.928 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:33.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:33.929 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:33.929 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:33.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 10 librados: watch_flush enter 2023-10-30T12:09:33.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 10 librados: watch_flush exit 2023-10-30T12:09:33.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 20 client.114353.objecter shutdown clearing up homeless session... 2023-10-30T12:09:33.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 10 client.114353.objecter successfully canceled tick 2023-10-30T12:09:33.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 msgr2=0x7f8a6003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8a600392f0 0x7f8a6003b7b0 secure :-1 s=READY pgs=4071 cs=0 l=1 rev1=1 crypto rx=0x7f8a68035000 tx=0x7f8a68008030 comp rx=0 tx=0).stop 2023-10-30T12:09:33.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 10 monclient: shutdown 2023-10-30T12:09:33.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:33.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a740a2710 msgr2=0x7f8a741c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:33.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a73fff640 1 -- 172.21.15.50:0/4068430730 reap_dead start 2023-10-30T12:09:33.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a740a2710 0x7f8a741c6f00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c780 tx=0x7f8a6c00cc40 comp rx=0 tx=0).stop 2023-10-30T12:09:33.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 shutdown_connections 2023-10-30T12:09:33.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 --2- 172.21.15.50:0/4068430730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8a740a2710 0x7f8a741c6f00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:33.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 >> 172.21.15.50:0/4068430730 conn(0x7f8a740a6200 msgr2=0x7f8a7406e810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:33.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 shutdown_connections 2023-10-30T12:09:33.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 -- 172.21.15.50:0/4068430730 wait complete. 2023-10-30T12:09:33.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:33.840+0000 7f8a7a0cf640 1 librados: shutdown 2023-10-30T12:09:33.934 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:09:33.935 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:09:33.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:09:34.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.040+0000 7fb1e13b6640 10 monclient: get_monmap_and_config 2023-10-30T12:09:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.040+0000 7fb1e13b6640 10 monclient: build_initial_monmap 2023-10-30T12:09:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.040+0000 7fb1e13b6640 10 monclient: monmap: 2023-10-30T12:09:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:34.041597+0000 2023-10-30T12:09:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:34.041597+0000 2023-10-30T12:09:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:34.044 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:34.044 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:34.044 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:34.044 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:34.044 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 1 Processor -- start 2023-10-30T12:09:34.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 1 -- start start 2023-10-30T12:09:34.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient: init 2023-10-30T12:09:34.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:34.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-10-30T12:09:34.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb1dc071c00 0x7fb1dc09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1dc071c00 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:34.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1dc097fe0 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:34.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb1dc09b170 0x7fb1dc139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1dc09b170 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:34.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc097600 con 0x7fb1dc097fe0 2023-10-30T12:09:34.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc0978e0 con 0x7fb1dc071c00 2023-10-30T12:09:34.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc09bf50 con 0x7fb1dc09b170 2023-10-30T12:09:34.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:34.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1e13b6640 10 monclient(hunting): authenticate will time out at 1419.224785s 2023-10-30T12:09:34.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1daffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb1dc071c00 0x7fb1dc09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1daffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb1dc071c00 0x7fb1dc09a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52294/0 (socket says 172.21.15.50:52294) 2023-10-30T12:09:34.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1daffd640 1 -- 172.21.15.50:0/3393501839 learned_addr learned my addr 172.21.15.50:0/3393501839 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:34.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1daffd640 10 monclient(hunting): get_auth_request con 0x7fb1dc071c00 auth_method 0 2023-10-30T12:09:34.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1daffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.041+0000 7fb1daffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1db7fe640 1 --2- 172.21.15.50:0/3393501839 >> v2:172.21.15.118:3300/0 conn(0x7fb1dc09b170 0x7fb1dc139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1db7fe640 10 monclient(hunting): get_auth_request con 0x7fb1dc09b170 auth_method 0 2023-10-30T12:09:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1db7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1db7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 10 monclient(hunting): handle_auth_done global_id 134121 payload 274 2023-10-30T12:09:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 1 -- 172.21.15.50:0/3393501839 >> v2:172.21.15.118:3300/0 conn(0x7fb1dc09b170 msgr2=0x7fb1dc139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 1 --2- 172.21.15.50:0/3393501839 >> v2:172.21.15.118:3300/0 conn(0x7fb1dc09b170 0x7fb1dc139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 1 -- 172.21.15.50:0/3393501839 >> v1:172.21.15.118:6789/0 conn(0x7fb1dc097fe0 legacy=0x7fb1dc09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 1 monclient: found mon.noname-a 2023-10-30T12:09:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 1 -- 172.21.15.50:0/3393501839 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc09d9b0 con 0x7fb1dc071c00 2023-10-30T12:09:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.042+0000 7fb1daffd640 1 --2- 172.21.15.50:0/3393501839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc071c00 0x7fb1dc09a150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc09a6c0 tx=0x7fb1cc02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bab74463c111482 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 1 -- 172.21.15.50:0/3393501839 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc02ede0 con 0x7fb1dc071c00 2023-10-30T12:09:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: dump: 2023-10-30T12:09:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.063 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.063 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.063 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.063 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.063 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:34.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:34.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:34.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 1 -- 172.21.15.50:0/3393501839 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc004030 con 0x7fb1dc071c00 2023-10-30T12:09:34.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 5 monclient: authenticate success, global_id 134121 2023-10-30T12:09:34.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:34.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:34.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:34.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 1 -- 172.21.15.50:0/3393501839 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc004420 con 0x7fb1dc071c00 2023-10-30T12:09:34.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 4 set_mon_vals no callback set 2023-10-30T12:09:34.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:34.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1da7fc640 10 monclient: dump: 2023-10-30T12:09:34.066 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.066 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.066 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.068 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.068 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 10 monclient: shutdown 2023-10-30T12:09:34.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:34.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 -- 172.21.15.50:0/3393501839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc071c00 msgr2=0x7fb1dc09a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 --2- 172.21.15.50:0/3393501839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc071c00 0x7fb1dc09a150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc09a6c0 tx=0x7fb1cc02e030 comp rx=0 tx=0).stop 2023-10-30T12:09:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 -- 172.21.15.50:0/3393501839 shutdown_connections 2023-10-30T12:09:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 --2- 172.21.15.50:0/3393501839 >> v2:172.21.15.118:3300/0 conn(0x7fb1dc09b170 0x7fb1dc139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 --2- 172.21.15.50:0/3393501839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc071c00 0x7fb1dc09a150 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 -- 172.21.15.50:0/3393501839 >> 172.21.15.50:0/3393501839 conn(0x7fb1dc0a3980 msgr2=0x7fb1dc0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 -- 172.21.15.50:0/3393501839 shutdown_connections 2023-10-30T12:09:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.043+0000 7fb1e13b6640 1 -- 172.21.15.50:0/3393501839 wait complete. 2023-10-30T12:09:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 10 monclient: build_initial_monmap 2023-10-30T12:09:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 10 monclient: monmap: 2023-10-30T12:09:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:34.071 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:34.071 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:34.071 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:34.071 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.071 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:34.071 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:34.072 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:34.072 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 1 librados: starting msgr at 2023-10-30T12:09:34.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 1 librados: starting objecter 2023-10-30T12:09:34.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 1 Processor -- start 2023-10-30T12:09:34.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 1 -- start start 2023-10-30T12:09:34.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 1 librados: setting wanted keys 2023-10-30T12:09:34.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 1 librados: calling monclient init 2023-10-30T12:09:34.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.044+0000 7fb1e13b6640 10 monclient: init 2023-10-30T12:09:34.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:34.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:09:34.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc071c00 0x7fb1dc1c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1dc071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:34.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc097fe0 0x7fb1dc0944d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1dc097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:34.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1dc09b170 0x7fb1dc094a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1dc09b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:34.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1c6800 con 0x7fb1dc097fe0 2023-10-30T12:09:34.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc08c750 con 0x7fb1dc071c00 2023-10-30T12:09:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc094f50 con 0x7fb1dc09b170 2023-10-30T12:09:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1e13b6640 10 monclient(hunting): authenticate will time out at 1419.228233s 2023-10-30T12:09:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc097fe0 0x7fb1dc0944d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc097fe0 0x7fb1dc0944d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52306/0 (socket says 172.21.15.50:52306) 2023-10-30T12:09:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1dc09b170 0x7fb1dc094a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 -- 172.21.15.50:0/2395247928 learned_addr learned my addr 172.21.15.50:0/2395247928 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): get_auth_request con 0x7fb1dc097fe0 auth_method 0 2023-10-30T12:09:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 10 monclient(hunting): get_auth_request con 0x7fb1dc09b170 auth_method 0 2023-10-30T12:09:34.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc071c00 0x7fb1dc1c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 10 monclient(hunting): get_auth_request con 0x7fb1dc071c00 auth_method 0 2023-10-30T12:09:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient(hunting): handle_auth_done global_id 134124 payload 824 2023-10-30T12:09:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1dc09b170 msgr2=0x7fb1dc094a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1dc09b170 0x7fb1dc094a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc071c00 msgr2=0x7fb1dc1c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc071c00 0x7fb1dc1c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 monclient: found mon.noname-a 2023-10-30T12:09:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 -- 172.21.15.50:0/2395247928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc00b700 con 0x7fb1dc097fe0 2023-10-30T12:09:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1db7fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1dc09b170 0x7fb1dc094a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d3fff640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc097fe0 0x7fb1dc0944d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c450 tx=0x7fb1c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d37fe640 10 client.?.objecter ms_handle_connect 0x7fb1dc097fe0 2023-10-30T12:09:34.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:34.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.045+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c40052d0 con 0x7fb1dc097fe0 2023-10-30T12:09:34.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: dump: 2023-10-30T12:09:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.086 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.086 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.086 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.086 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.086 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.087 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: _finish_auth 0 2023-10-30T12:09:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:09:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c4005430 con 0x7fb1dc097fe0 2023-10-30T12:09:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 5 monclient: authenticate success, global_id 134124 2023-10-30T12:09:34.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 10 monclient: _renew_subs 2023-10-30T12:09:34.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:34.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc08d6b0 con 0x7fb1dc097fe0 2023-10-30T12:09:34.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:34.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4012870 con 0x7fb1dc097fe0 2023-10-30T12:09:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 10 monclient: dump: 2023-10-30T12:09:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.090 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.090 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.090 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.090 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.091 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 10 client.134124.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 10 monclient: _renew_subs 2023-10-30T12:09:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc071720 con 0x7fb1dc097fe0 2023-10-30T12:09:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d9ffb640 4 set_mon_vals no callback set 2023-10-30T12:09:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 1 librados: init done 2023-10-30T12:09:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb1c40129d0 con 0x7fb1dc097fe0 2023-10-30T12:09:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:34.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08afd0 0x7fb1dc08b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.046+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc073ff0 con 0x7fb1dc08afd0 2023-10-30T12:09:34.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1bc0392f0 0x7fb1bc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fb1c404a810 con 0x7fb1dc097fe0 2023-10-30T12:09:34.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08afd0 0x7fb1dc08b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 10 client.134124.objecter ms_dispatch 0x7fb1dc000b90 osd_map(315..315 src has 1..315) v4 2023-10-30T12:09:34.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 3 client.134124.objecter handle_osd_map got epochs [315,315] > 0 2023-10-30T12:09:34.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 3 client.134124.objecter handle_osd_map decoding full epoch 315 2023-10-30T12:09:34.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient: get_auth_request con 0x7fb1dc08afd0 auth_method 0 2023-10-30T12:09:34.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 20 client.134124.objecter dump_active .. 0 homeless 2023-10-30T12:09:34.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1db7fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1bc0392f0 0x7fb1bc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1db7fe640 10 monclient: get_auth_request con 0x7fb1bc0392f0 auth_method 0 2023-10-30T12:09:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 10 monclient(hunting): handle_auth_done global_id 94747 payload 824 2023-10-30T12:09:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 1 -- 172.21.15.50:0/2395247928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1dc071fe0 con 0x7fb1dc08afd0 2023-10-30T12:09:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 10 client.134124.objecter ms_handle_connect 0x7fb1bc0392f0 2023-10-30T12:09:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1db7fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1bc0392f0 0x7fb1bc03b7b0 secure :-1 s=READY pgs=4072 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc08c6e0 tx=0x7fb1d4009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 10 client.134124.objecter ms_handle_connect 0x7fb1dc08afd0 2023-10-30T12:09:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 10 client.134124.objecter resend_mon_ops 2023-10-30T12:09:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1daffd640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08afd0 0x7fb1dc08b3b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc06bb30 tx=0x7fb1cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc003a20 con 0x7fb1dc08afd0 2023-10-30T12:09:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.047+0000 7fb1d37fe640 10 monclient: dump: 2023-10-30T12:09:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.099 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.100 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.048+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb1cc003b80 con 0x7fb1dc08afd0 2023-10-30T12:09:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.048+0000 7fb1d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.048+0000 7fb1d37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.048+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08afd0 msgr2=0x7fb1dc08b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.048+0000 7fb1d37fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08afd0 0x7fb1dc08b3b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc06bb30 tx=0x7fb1cc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.048+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08afd0 msgr2=0x7fb1dc08b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.048+0000 7fb1d37fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08afd0 0x7fb1dc08b3b0 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc06bb30 tx=0x7fb1cc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1e13b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:34.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1e13b6640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08b8f0 0x7fb1dc06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1e13b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc08dbd0 con 0x7fb1dc08b8f0 2023-10-30T12:09:34.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08b8f0 0x7fb1dc06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 10 monclient: get_auth_request con 0x7fb1dc08b8f0 auth_method 0 2023-10-30T12:09:34.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.054+0000 7fb1daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1daffd640 10 monclient(hunting): handle_auth_done global_id 94750 payload 824 2023-10-30T12:09:34.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1daffd640 1 -- 172.21.15.50:0/2395247928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb1dc04e160 con 0x7fb1dc08b8f0 2023-10-30T12:09:34.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1daffd640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08b8f0 0x7fb1dc06dbb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00b2c0 tx=0x7fb1cc033000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 10 client.134124.objecter ms_handle_connect 0x7fb1dc08b8f0 2023-10-30T12:09:34.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 10 client.134124.objecter resend_mon_ops 2023-10-30T12:09:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc050070 con 0x7fb1dc08b8f0 2023-10-30T12:09:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 10 monclient: dump: 2023-10-30T12:09:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.109 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fb1cc036700 con 0x7fb1dc08b8f0 2023-10-30T12:09:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08b8f0 msgr2=0x7fb1dc06dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08b8f0 0x7fb1dc06dbb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00b2c0 tx=0x7fb1cc033000 comp rx=0 tx=0).stop 2023-10-30T12:09:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08b8f0 msgr2=0x7fb1dc06dbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1d37fe640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1dc08b8f0 0x7fb1dc06dbb0 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00b2c0 tx=0x7fb1cc033000 comp rx=0 tx=0).stop 2023-10-30T12:09:34.111 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:34.111 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:09:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 88, 2023-10-30T12:09:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:09:34.112 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:09:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:34.113 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:34.114 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:34.115 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:34.116 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:34.117 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:34.118 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:34.119 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:34.120 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:34.121 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:34.122 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:34.123 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:34.124 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:34.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.124 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:34.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.124 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:34.124 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:34.125 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:34.125 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:34.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.125 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:34.125 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:34.126 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:34.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.126 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:34.126 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:34.127 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:34.127 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.127 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:34.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.127 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:34.127 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:34.128 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:34.128 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:34.128 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:34.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.129 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:34.129 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:34.129 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:34.129 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:34.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:34.130 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:34.130 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:34.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:34.131 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:34.131 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:34.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.131 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:34.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.132 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:34.132 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:34.132 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:34.132 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:34.132 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:34.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.133 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:34.133 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:34.133 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:34.133 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:34.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:34.134 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:34.134 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:34.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:34.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:34.135 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:34.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.136 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:34.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.136 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:34.136 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:34.136 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:34.136 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:34.137 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:34.137 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.137 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:34.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.137 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:34.138 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:34.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.138 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:34.138 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:34.138 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:34.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.139 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:34.139 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:09:34.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.139 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:34.139 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:34.140 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:09:34.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.140 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:34.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:34.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.141 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:34.141 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:34.141 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:09:34.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.141 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:34.142 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:09:34.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:34.142 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:34.142 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:34.142 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:34.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:34.143 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:34.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:34.143 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:34.143 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1e13b6640 10 librados: watch_flush enter 2023-10-30T12:09:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1e13b6640 10 librados: watch_flush exit 2023-10-30T12:09:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1e13b6640 20 client.134124.objecter shutdown clearing up homeless session... 2023-10-30T12:09:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.055+0000 7fb1e13b6640 10 client.134124.objecter successfully canceled tick 2023-10-30T12:09:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1bc0392f0 msgr2=0x7fb1bc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1bc0392f0 0x7fb1bc03b7b0 secure :-1 s=READY pgs=4072 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc08c6e0 tx=0x7fb1d4009820 comp rx=0 tx=0).stop 2023-10-30T12:09:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 10 monclient: shutdown 2023-10-30T12:09:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc097fe0 msgr2=0x7fb1dc0944d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 --2- 172.21.15.50:0/2395247928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1dc097fe0 0x7fb1dc0944d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c450 tx=0x7fb1c400c910 comp rx=0 tx=0).stop 2023-10-30T12:09:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1db7fe640 1 -- 172.21.15.50:0/2395247928 reap_dead start 2023-10-30T12:09:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1db7fe640 1 -- 172.21.15.50:0/2395247928 reap_dead start 2023-10-30T12:09:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 shutdown_connections 2023-10-30T12:09:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 >> 172.21.15.50:0/2395247928 conn(0x7fb1dc0a3980 msgr2=0x7fb1dc096590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 shutdown_connections 2023-10-30T12:09:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 -- 172.21.15.50:0/2395247928 wait complete. 2023-10-30T12:09:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.056+0000 7fb1e13b6640 1 librados: shutdown 2023-10-30T12:09:34.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:09:34.149 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:09:34.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:09:34.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.253+0000 7f6659b4c640 10 monclient: get_monmap_and_config 2023-10-30T12:09:34.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.253+0000 7f6659b4c640 10 monclient: build_initial_monmap 2023-10-30T12:09:34.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.253+0000 7f6659b4c640 10 monclient: monmap: 2023-10-30T12:09:34.255 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:34.255 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.255 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:34.254186+0000 2023-10-30T12:09:34.255 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:34.254186+0000 2023-10-30T12:09:34.256 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:34.256 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.256 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:34.256 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:34.256 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:34.256 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:34.257 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:34.257 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.253+0000 7f6659b4c640 1 Processor -- start 2023-10-30T12:09:34.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.253+0000 7f6659b4c640 1 -- start start 2023-10-30T12:09:34.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.253+0000 7f6659b4c640 10 monclient: init 2023-10-30T12:09:34.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:34.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-10-30T12:09:34.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66540a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:34.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6654099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:34.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6654132db0 0x7f665413e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient(hunting): picked mon.noname-c con 0x7f6654132db0 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:34.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540a4f40 con 0x7f6654099a40 2023-10-30T12:09:34.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540a4150 con 0x7f66540a4b60 2023-10-30T12:09:34.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66540a3d70 con 0x7f6654132db0 2023-10-30T12:09:34.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:34.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6659b4c640 10 monclient(hunting): authenticate will time out at 1419.437347s 2023-10-30T12:09:34.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6652ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6654099a40 0x7f665413aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59012/0 (socket says 172.21.15.50:59012) 2023-10-30T12:09:34.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6652ffd640 1 -- 172.21.15.50:0/3529643128 learned_addr learned my addr 172.21.15.50:0/3529643128 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:34.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 1 --2- 172.21.15.50:0/3529643128 >> v2:172.21.15.118:3300/0 conn(0x7f6654132db0 0x7f665413e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): get_auth_request con 0x7f6654132db0 auth_method 0 2023-10-30T12:09:34.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 1 -- 172.21.15.50:0/3529643128 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811786839 0 0) 0x7f66540a4f40 con 0x7f6654099a40 2023-10-30T12:09:34.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): _negotiate 2023-10-30T12:09:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): my global_id is 134127 2023-10-30T12:09:34.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 1 -- 172.21.15.50:0/3529643128 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c003520 con 0x7f6654099a40 2023-10-30T12:09:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 1 -- 172.21.15.50:0/3529643128 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777024880 0 0) 0x7f66540a4150 con 0x7f66540a4b60 2023-10-30T12:09:34.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): _negotiate 2023-10-30T12:09:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 10 monclient(hunting): my global_id is 94756 2023-10-30T12:09:34.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f66527fc640 1 -- 172.21.15.50:0/3529643128 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66540a4f40 con 0x7f66540a4b60 2023-10-30T12:09:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient(hunting): handle_auth_done global_id 94753 payload 274 2023-10-30T12:09:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 1 -- 172.21.15.50:0/3529643128 >> v1:172.21.15.118:6789/0 conn(0x7f66540a4b60 legacy=0x7f66541326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 1 -- 172.21.15.50:0/3529643128 >> v1:172.21.15.50:6789/0 conn(0x7f6654099a40 legacy=0x7f665413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 1 monclient: found mon.noname-c 2023-10-30T12:09:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:09:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.254+0000 7f6653fff640 1 -- 172.21.15.50:0/3529643128 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665409a440 con 0x7f6654132db0 2023-10-30T12:09:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f6653fff640 1 --2- 172.21.15.50:0/3529643128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654132db0 0x7f665413e1b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f664c009d90 tx=0x7f664c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=711c1d4d15209b86 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 1 -- 172.21.15.50:0/3529643128 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c02cbd0 con 0x7f6654132db0 2023-10-30T12:09:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: dump: 2023-10-30T12:09:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.279 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:34.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f6659b4c640 5 monclient: authenticate success, global_id 94753 2023-10-30T12:09:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 1 -- 172.21.15.50:0/3529643128 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c02cd30 con 0x7f6654132db0 2023-10-30T12:09:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 1 -- 172.21.15.50:0/3529643128 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c02cea0 con 0x7f6654132db0 2023-10-30T12:09:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f66527fc640 10 monclient: dump: 2023-10-30T12:09:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.283 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f6659b4c640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.255+0000 7f6659b4c640 4 set_mon_vals no callback set 2023-10-30T12:09:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 10 monclient: shutdown 2023-10-30T12:09:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 1 -- 172.21.15.50:0/3529643128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654132db0 msgr2=0x7f665413e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3529643128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654132db0 0x7f665413e1b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f664c009d90 tx=0x7f664c02be20 comp rx=0 tx=0).stop 2023-10-30T12:09:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 1 -- 172.21.15.50:0/3529643128 shutdown_connections 2023-10-30T12:09:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3529643128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654132db0 0x7f665413e1b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 1 -- 172.21.15.50:0/3529643128 >> 172.21.15.50:0/3529643128 conn(0x7f66540a8060 msgr2=0x7f66540a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 1 -- 172.21.15.50:0/3529643128 shutdown_connections 2023-10-30T12:09:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.256+0000 7f6659b4c640 1 -- 172.21.15.50:0/3529643128 wait complete. 2023-10-30T12:09:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 10 monclient: build_initial_monmap 2023-10-30T12:09:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 10 monclient: monmap: 2023-10-30T12:09:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:34.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 1 librados: starting msgr at 2023-10-30T12:09:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 1 librados: starting objecter 2023-10-30T12:09:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 1 Processor -- start 2023-10-30T12:09:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 1 -- start start 2023-10-30T12:09:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 1 librados: setting wanted keys 2023-10-30T12:09:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 1 librados: calling monclient init 2023-10-30T12:09:34.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.257+0000 7f6659b4c640 10 monclient: init 2023-10-30T12:09:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:09:34.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66540a4b60 0x7f6654130ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): picked mon.noname-a con 0x7f66540a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6654132db0 0x7f665412e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): picked mon.noname-c con 0x7f6654132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654130a70 0x7f665412eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): picked mon.noname-b con 0x7f6654130a70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665409b180 con 0x7f66540a4b60 2023-10-30T12:09:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66540a0d30 con 0x7f6654130a70 2023-10-30T12:09:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665409b7f0 con 0x7f6654132db0 2023-10-30T12:09:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient(hunting): authenticate will time out at 1419.441124s 2023-10-30T12:09:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66540a4b60 0x7f6654130ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6654132db0 0x7f665412e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66540a4b60 0x7f6654130ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52322/0 (socket says 172.21.15.50:52322) 2023-10-30T12:09:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 1 -- 172.21.15.50:0/3552509976 learned_addr learned my addr 172.21.15.50:0/3552509976 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 10 monclient(hunting): get_auth_request con 0x7f66540a4b60 auth_method 0 2023-10-30T12:09:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): get_auth_request con 0x7f6654132db0 auth_method 0 2023-10-30T12:09:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654130a70 0x7f665412eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 10 monclient(hunting): get_auth_request con 0x7f6654130a70 auth_method 0 2023-10-30T12:09:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:34.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient(hunting): handle_auth_done global_id 114365 payload 824 2023-10-30T12:09:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 -- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654130a70 msgr2=0x7f665412eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654130a70 0x7f665412eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 -- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66540a4b60 msgr2=0x7f6654130ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66540a4b60 0x7f6654130ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 monclient: found mon.noname-c 2023-10-30T12:09:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 -- 172.21.15.50:0/3552509976 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c009a70 con 0x7f6654132db0 2023-10-30T12:09:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f66537fe640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66540a4b60 0x7f6654130ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:34.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6652ffd640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6654132db0 0x7f665412e660 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f664800c390 tx=0x7f664800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 client.?.objecter ms_handle_connect 0x7f6654132db0 2023-10-30T12:09:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 1 -- 172.21.15.50:0/3552509976 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648005ce0 con 0x7f6654132db0 2023-10-30T12:09:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: dump: 2023-10-30T12:09:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.306 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: _finish_auth 0 2023-10-30T12:09:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 5 monclient: authenticate success, global_id 114365 2023-10-30T12:09:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 1 -- 172.21.15.50:0/3552509976 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648005e40 con 0x7f6654132db0 2023-10-30T12:09:34.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient: _renew_subs 2023-10-30T12:09:34.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:34.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66540a18d0 con 0x7f6654132db0 2023-10-30T12:09:34.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:34.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 1 -- 172.21.15.50:0/3552509976 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480072d0 con 0x7f6654132db0 2023-10-30T12:09:34.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:34.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:34.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f663bfff640 10 monclient: dump: 2023-10-30T12:09:34.312 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:34.313 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:34.313 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.313 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:34.313 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:34.313 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:34.314 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:34.314 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:34.314 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:34.314 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:34.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6651ffb640 4 set_mon_vals no callback set 2023-10-30T12:09:34.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 client.114365.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:34.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient: _renew_subs 2023-10-30T12:09:34.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:34.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.258+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665412f0e0 con 0x7f6654132db0 2023-10-30T12:09:34.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.259+0000 7f6659b4c640 1 librados: init done 2023-10-30T12:09:34.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.259+0000 7f663bfff640 1 -- 172.21.15.50:0/3552509976 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6648007430 con 0x7f6654132db0 2023-10-30T12:09:34.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.259+0000 7f6659b4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:34.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.259+0000 7f6659b4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:34.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.259+0000 7f6659b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:34.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.259+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665404e160 con 0x7f6654132db0 2023-10-30T12:09:34.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f663bfff640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66400392f0 0x7f664003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:34.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f663bfff640 1 -- 172.21.15.50:0/3552509976 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f664804a140 con 0x7f6654132db0 2023-10-30T12:09:34.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f663bfff640 10 client.114365.objecter ms_dispatch 0x7f6654000b90 osd_map(315..315 src has 1..315) v4 2023-10-30T12:09:34.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f663bfff640 3 client.114365.objecter handle_osd_map got epochs [315,315] > 0 2023-10-30T12:09:34.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f663bfff640 3 client.114365.objecter handle_osd_map decoding full epoch 315 2023-10-30T12:09:34.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f663bfff640 20 client.114365.objecter dump_active .. 0 homeless 2023-10-30T12:09:34.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f66537fe640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66400392f0 0x7f664003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:34.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f66537fe640 10 monclient: get_auth_request con 0x7f66400392f0 auth_method 0 2023-10-30T12:09:34.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f663bfff640 10 client.114365.objecter ms_handle_connect 0x7f66400392f0 2023-10-30T12:09:34.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.260+0000 7f66537fe640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66400392f0 0x7f664003b7b0 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 crypto rx=0x7f6644007fd0 tx=0x7f6644009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:34.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.264+0000 7f663bfff640 1 -- 172.21.15.50:0/3552509976 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f664801b730 con 0x7f6654132db0 2023-10-30T12:09:34.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.264+0000 7f663bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:34.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.264+0000 7f663bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:34.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.509+0000 7f6659b4c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:09:34.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.509+0000 7f6659b4c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:09:34.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.509+0000 7f6659b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:34.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.509+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f66540745c0 con 0x7f6654132db0 2023-10-30T12:09:34.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.510+0000 7f663bfff640 1 -- 172.21.15.50:0/3552509976 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f6648017070 con 0x7f6654132db0 2023-10-30T12:09:34.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.510+0000 7f663bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:09:34.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.510+0000 7f663bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:34.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.511+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.511+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=9,logm=56} crc {auth=1401246271,config=3140514009,health=2441606716,kv=1704049247,logm=3421403795}) 2023-10-30T12:09:34.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.511+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=3647945868,config=3140514009,health=2441606716,kv=3826327778,logm=4220273817}) 2023-10-30T12:09:34.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.511+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.511+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=9,logm=56} crc {auth=1401246271,config=3140514009,health=2441606716,kv=1704049247,logm=3421403795}) 2023-10-30T12:09:34.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.511+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=2181396855,config=3140514009,health=2441606716,kv=2905431493,logm=1991741355}) 2023-10-30T12:09:34.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959440992}) 2023-10-30T12:09:34.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=433423371}) 2023-10-30T12:09:34.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959440992}) 2023-10-30T12:09:34.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016478453}) 2023-10-30T12:09:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f6659b4c640 10 librados: watch_flush enter 2023-10-30T12:09:34.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f6659b4c640 10 librados: watch_flush exit 2023-10-30T12:09:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f6659b4c640 20 client.114365.objecter shutdown clearing up homeless session... 2023-10-30T12:09:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.512+0000 7f6659b4c640 10 client.114365.objecter successfully canceled tick 2023-10-30T12:09:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66400392f0 msgr2=0x7f664003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66400392f0 0x7f664003b7b0 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 crypto rx=0x7f6644007fd0 tx=0x7f6644009040 comp rx=0 tx=0).stop 2023-10-30T12:09:34.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 10 monclient: shutdown 2023-10-30T12:09:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6654132db0 msgr2=0x7f665412e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6654132db0 0x7f665412e660 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f664800c390 tx=0x7f664800c850 comp rx=0 tx=0).stop 2023-10-30T12:09:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 shutdown_connections 2023-10-30T12:09:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66400392f0 0x7f664003b7b0 unknown :-1 s=CLOSED pgs=4073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6654130a70 0x7f665412eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6654132db0 0x7f665412e660 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 --2- 172.21.15.50:0/3552509976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66540a4b60 0x7f6654130ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 >> 172.21.15.50:0/3552509976 conn(0x7f66540a8060 msgr2=0x7f66540a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 shutdown_connections 2023-10-30T12:09:34.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 -- 172.21.15.50:0/3552509976 wait complete. 2023-10-30T12:09:34.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f6659b4c640 1 librados: shutdown 2023-10-30T12:09:34.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796696918}) 2023-10-30T12:09:34.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=805634514}) 2023-10-30T12:09:34.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796696918}) 2023-10-30T12:09:34.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.513+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977953782}) 2023-10-30T12:09:34.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.514+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.514+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678633943}) 2023-10-30T12:09:34.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.514+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=965455458}) 2023-10-30T12:09:34.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.514+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.514+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678633943}) 2023-10-30T12:09:34.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.514+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=661409336}) 2023-10-30T12:09:34.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.515+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.515+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316972315}) 2023-10-30T12:09:34.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.515+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1054474760}) 2023-10-30T12:09:34.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.515+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.515+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316972315}) 2023-10-30T12:09:34.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.515+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120402977}) 2023-10-30T12:09:34.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.516+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.516+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955915458}) 2023-10-30T12:09:34.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.516+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513244978}) 2023-10-30T12:09:34.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.516+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.516+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955915458}) 2023-10-30T12:09:34.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.516+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960869657}) 2023-10-30T12:09:34.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704273391}) 2023-10-30T12:09:34.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3977708394}) 2023-10-30T12:09:34.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704273391}) 2023-10-30T12:09:34.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2659432397}) 2023-10-30T12:09:34.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025736080}) 2023-10-30T12:09:34.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2177476935}) 2023-10-30T12:09:34.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025736080}) 2023-10-30T12:09:34.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.517+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1290093623}) 2023-10-30T12:09:34.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.518+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.518+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222676206}) 2023-10-30T12:09:34.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.518+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399840668}) 2023-10-30T12:09:34.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.518+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.518+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222676206}) 2023-10-30T12:09:34.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.518+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3660403305}) 2023-10-30T12:09:34.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384565405}) 2023-10-30T12:09:34.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123126227}) 2023-10-30T12:09:34.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384565405}) 2023-10-30T12:09:34.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328016705}) 2023-10-30T12:09:34.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463812406}) 2023-10-30T12:09:34.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3725539863}) 2023-10-30T12:09:34.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463812406}) 2023-10-30T12:09:34.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.519+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453378611}) 2023-10-30T12:09:34.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.520+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.520+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634900830}) 2023-10-30T12:09:34.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.520+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1352141259}) 2023-10-30T12:09:34.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.520+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.520+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634900830}) 2023-10-30T12:09:34.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.520+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1735490891}) 2023-10-30T12:09:34.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673876755}) 2023-10-30T12:09:34.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198119658}) 2023-10-30T12:09:34.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673876755}) 2023-10-30T12:09:34.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360829480}) 2023-10-30T12:09:34.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894338}) 2023-10-30T12:09:34.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394266473}) 2023-10-30T12:09:34.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894338}) 2023-10-30T12:09:34.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.521+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261237287}) 2023-10-30T12:09:34.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.522+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.522+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85407211}) 2023-10-30T12:09:34.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.522+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1841314551}) 2023-10-30T12:09:34.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.522+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.522+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85407211}) 2023-10-30T12:09:34.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.522+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987893667}) 2023-10-30T12:09:34.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668008822}) 2023-10-30T12:09:34.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680191717}) 2023-10-30T12:09:34.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668008822}) 2023-10-30T12:09:34.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2167409725}) 2023-10-30T12:09:34.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617633846}) 2023-10-30T12:09:34.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.523+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247111179}) 2023-10-30T12:09:34.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617633846}) 2023-10-30T12:09:34.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1709441100}) 2023-10-30T12:09:34.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972076351}) 2023-10-30T12:09:34.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239386457}) 2023-10-30T12:09:34.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972076351}) 2023-10-30T12:09:34.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.524+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560272176}) 2023-10-30T12:09:34.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.525+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.525+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145413157}) 2023-10-30T12:09:34.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.525+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346025882}) 2023-10-30T12:09:34.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.525+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.525+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145413157}) 2023-10-30T12:09:34.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.525+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3709089701}) 2023-10-30T12:09:34.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:09:34.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.526+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.526+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600050528}) 2023-10-30T12:09:34.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.526+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782511029}) 2023-10-30T12:09:34.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.526+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.526+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600050528}) 2023-10-30T12:09:34.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.526+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1314913704}) 2023-10-30T12:09:34.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.527+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.527+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435108338}) 2023-10-30T12:09:34.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.527+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1760571787}) 2023-10-30T12:09:34.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.527+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.527+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435108338}) 2023-10-30T12:09:34.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.527+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298578933}) 2023-10-30T12:09:34.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.527+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485731386}) 2023-10-30T12:09:34.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740152755}) 2023-10-30T12:09:34.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485731386}) 2023-10-30T12:09:34.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715484513}) 2023-10-30T12:09:34.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438253844}) 2023-10-30T12:09:34.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1908346504}) 2023-10-30T12:09:34.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438253844}) 2023-10-30T12:09:34.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.528+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042207473}) 2023-10-30T12:09:34.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354068491}) 2023-10-30T12:09:34.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985147657}) 2023-10-30T12:09:34.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354068491}) 2023-10-30T12:09:34.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564585036}) 2023-10-30T12:09:34.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110662497}) 2023-10-30T12:09:34.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1784308505}) 2023-10-30T12:09:34.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110662497}) 2023-10-30T12:09:34.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.529+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534997225}) 2023-10-30T12:09:34.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.530+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.530+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117658502}) 2023-10-30T12:09:34.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.530+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406631239}) 2023-10-30T12:09:34.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.530+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.530+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117658502}) 2023-10-30T12:09:34.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.530+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988342828}) 2023-10-30T12:09:34.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148103774}) 2023-10-30T12:09:34.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438405371}) 2023-10-30T12:09:34.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148103774}) 2023-10-30T12:09:34.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722111117}) 2023-10-30T12:09:34.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290382193}) 2023-10-30T12:09:34.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2183395087}) 2023-10-30T12:09:34.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290382193}) 2023-10-30T12:09:34.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.531+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672692171}) 2023-10-30T12:09:34.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.532+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.532+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027807410}) 2023-10-30T12:09:34.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.532+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=806280864}) 2023-10-30T12:09:34.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.532+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.532+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027807410}) 2023-10-30T12:09:34.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.532+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037716964}) 2023-10-30T12:09:34.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976514156}) 2023-10-30T12:09:34.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3418574891}) 2023-10-30T12:09:34.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976514156}) 2023-10-30T12:09:34.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962268428}) 2023-10-30T12:09:34.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280351844}) 2023-10-30T12:09:34.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3252877639}) 2023-10-30T12:09:34.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280351844}) 2023-10-30T12:09:34.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.533+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118673310}) 2023-10-30T12:09:34.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.534+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.534+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858076419}) 2023-10-30T12:09:34.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.534+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3031415513}) 2023-10-30T12:09:34.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.534+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.534+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858076419}) 2023-10-30T12:09:34.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.534+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3778771407}) 2023-10-30T12:09:34.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029403859}) 2023-10-30T12:09:34.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1911412685}) 2023-10-30T12:09:34.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029403859}) 2023-10-30T12:09:34.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265198861}) 2023-10-30T12:09:34.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175033495}) 2023-10-30T12:09:34.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027373076}) 2023-10-30T12:09:34.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175033495}) 2023-10-30T12:09:34.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.535+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=735356062}) 2023-10-30T12:09:34.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.536+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.536+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505234241}) 2023-10-30T12:09:34.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.536+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717739203}) 2023-10-30T12:09:34.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.536+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.536+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505234241}) 2023-10-30T12:09:34.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.536+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210174743}) 2023-10-30T12:09:34.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.537+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.537+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296905021}) 2023-10-30T12:09:34.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.537+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2569147851}) 2023-10-30T12:09:34.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.537+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.537+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296905021}) 2023-10-30T12:09:34.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.537+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941100854}) 2023-10-30T12:09:34.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811700713}) 2023-10-30T12:09:34.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909740592}) 2023-10-30T12:09:34.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811700713}) 2023-10-30T12:09:34.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592283979}) 2023-10-30T12:09:34.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442169032}) 2023-10-30T12:09:34.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2012779502}) 2023-10-30T12:09:34.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442169032}) 2023-10-30T12:09:34.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.538+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3762154249}) 2023-10-30T12:09:34.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826370595}) 2023-10-30T12:09:34.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4175133352}) 2023-10-30T12:09:34.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826370595}) 2023-10-30T12:09:34.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2848322606}) 2023-10-30T12:09:34.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994090951}) 2023-10-30T12:09:34.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=267458464}) 2023-10-30T12:09:34.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994090951}) 2023-10-30T12:09:34.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.539+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452582002}) 2023-10-30T12:09:34.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.540+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.540+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3011434490}) 2023-10-30T12:09:34.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.540+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517776573}) 2023-10-30T12:09:34.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.540+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.540+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3011434490}) 2023-10-30T12:09:34.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.540+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079049223}) 2023-10-30T12:09:34.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952687540}) 2023-10-30T12:09:34.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4063692472}) 2023-10-30T12:09:34.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952687540}) 2023-10-30T12:09:34.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776425074}) 2023-10-30T12:09:34.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200104643}) 2023-10-30T12:09:34.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3251189383}) 2023-10-30T12:09:34.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200104643}) 2023-10-30T12:09:34.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.541+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866330581}) 2023-10-30T12:09:34.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.542+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.542+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976519062}) 2023-10-30T12:09:34.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.542+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513728519}) 2023-10-30T12:09:34.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.542+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.542+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976519062}) 2023-10-30T12:09:34.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.542+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139590177}) 2023-10-30T12:09:34.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.543+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.543+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1126558783,mdsmap=3984425268,mgr=1204984937}) 2023-10-30T12:09:34.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.543+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=2520334180,mdsmap=3984425268,mgr=3853015086}) 2023-10-30T12:09:34.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.543+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.543+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1126558783,mdsmap=3984425268,mgr=1204984937}) 2023-10-30T12:09:34.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.543+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=3774919802,mdsmap=3984425268,mgr=2906314846}) 2023-10-30T12:09:34.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.544+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.544+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27,osdmap=42} crc {mgr=349486855,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2169249085,mon_config_key=3217606506,monmap=3324057927,osd_metadata=3009730626,osd_pg_creating=905830555,osd_snap=3022830193,osdmap=3682164682}) 2023-10-30T12:09:34.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.544+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=44} crc {mgr=4291836917,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2714222533,mon_config_key=3217606506,monmap=3324057927,osd_metadata=3009730626,osd_pg_creating=905830555,osd_snap=2603374560,osdmap=3892431639}) 2023-10-30T12:09:34.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.544+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.544+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27,osdmap=42} crc {mgr=349486855,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2169249085,mon_config_key=3217606506,monmap=3324057927,osd_metadata=3009730626,osd_pg_creating=905830555,osd_snap=3022830193,osdmap=3682164682}) 2023-10-30T12:09:34.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.544+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=39} crc {mgr=613727388,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2169249085,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=905830555,osd_snap=3845386176,osdmap=1139887973}) 2023-10-30T12:09:34.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2410240030}) 2023-10-30T12:09:34.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=532408964}) 2023-10-30T12:09:34.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2410240030}) 2023-10-30T12:09:34.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2095410268}) 2023-10-30T12:09:34.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=706515117}) 2023-10-30T12:09:34.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=801139683}) 2023-10-30T12:09:34.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=706515117}) 2023-10-30T12:09:34.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.545+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1305326182}) 2023-10-30T12:09:34.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.547+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.547+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3071313570}) 2023-10-30T12:09:34.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.547+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3683841591}) 2023-10-30T12:09:34.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.547+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.547+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3071313570}) 2023-10-30T12:09:34.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.547+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3888950039}) 2023-10-30T12:09:34.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.549+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.549+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=76713406}) 2023-10-30T12:09:34.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.549+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2342607918}) 2023-10-30T12:09:34.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.549+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.549+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=76713406}) 2023-10-30T12:09:34.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.549+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3261192192}) 2023-10-30T12:09:34.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.554+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.554+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=3972978099}) 2023-10-30T12:09:34.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.554+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=1019818446}) 2023-10-30T12:09:34.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.554+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:09:34.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.554+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=3972978099}) 2023-10-30T12:09:34.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:09:34.554+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=1894620588}) 2023-10-30T12:09:35.550 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:09:35.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:35.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.668+0000 7f7c05b7c640 10 monclient: get_monmap_and_config 2023-10-30T12:09:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.668+0000 7f7c05b7c640 10 monclient: build_initial_monmap 2023-10-30T12:09:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.668+0000 7f7c05b7c640 10 monclient: monmap: 2023-10-30T12:09:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:35.671 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:35.671 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:35.669647+0000 2023-10-30T12:09:35.671 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:35.669647+0000 2023-10-30T12:09:35.671 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:35.671 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:35.672 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:35.672 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:35.672 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:35.672 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:35.672 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:35.672 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:35.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 1 Processor -- start 2023-10-30T12:09:35.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 1 -- start start 2023-10-30T12:09:35.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient: init 2023-10-30T12:09:35.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:35.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T12:09:35.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c00098520 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00099010 0x7f7c0009bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f7c00099010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c001379f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0009ad30 con 0x7f7c00098520 2023-10-30T12:09:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0009c460 con 0x7f7c001379f0 2023-10-30T12:09:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000993f0 con 0x7f7c00099010 2023-10-30T12:09:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.669+0000 7f7c05b7c640 10 monclient(hunting): authenticate will time out at 1420.852917s 2023-10-30T12:09:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00099010 0x7f7c0009bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bff7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7c00098520 0x7f7c00098900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59018/0 (socket says 172.21.15.50:59018) 2023-10-30T12:09:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bff7fe640 1 -- 172.21.15.50:0/927554202 learned_addr learned my addr 172.21.15.50:0/927554202 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): get_auth_request con 0x7f7c00099010 auth_method 0 2023-10-30T12:09:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient(hunting): handle_auth_done global_id 114368 payload 274 2023-10-30T12:09:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 1 -- 172.21.15.50:0/927554202 >> v1:172.21.15.118:6789/0 conn(0x7f7c001379f0 legacy=0x7f7c00139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:35.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfe7fc640 1 -- 172.21.15.50:0/927554202 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020947042 0 0) 0x7f7c0009ad30 con 0x7f7c00098520 2023-10-30T12:09:35.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 1 -- 172.21.15.50:0/927554202 >> v1:172.21.15.50:6789/0 conn(0x7f7c00098520 legacy=0x7f7c00098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:09:35.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 1 monclient: found mon.noname-b 2023-10-30T12:09:35.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:35.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:09:35.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfeffd640 1 -- 172.21.15.50:0/927554202 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0009b260 con 0x7f7c00099010 2023-10-30T12:09:35.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.670+0000 7f7bfe7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:09:35.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfeffd640 1 --2- 172.21.15.50:0/927554202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00099010 0x7f7c0009bf20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40097d0 tx=0x7f7bf402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aa94f65f892d48b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:35.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 1 -- 172.21.15.50:0/927554202 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf40316f0 con 0x7f7c00099010 2023-10-30T12:09:35.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:35.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:35.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: dump: 2023-10-30T12:09:35.688 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:35.688 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:35.688 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.689 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.689 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:35.689 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:35.689 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:35.689 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:35.689 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:35.690 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:35.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:35.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:35.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:35.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 5 monclient: authenticate success, global_id 114368 2023-10-30T12:09:35.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 1 -- 172.21.15.50:0/927554202 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf4031850 con 0x7f7c00099010 2023-10-30T12:09:35.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:35.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:35.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 4 set_mon_vals no callback set 2023-10-30T12:09:35.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 1 -- 172.21.15.50:0/927554202 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4031be0 con 0x7f7c00099010 2023-10-30T12:09:35.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7bfe7fc640 10 monclient: dump: 2023-10-30T12:09:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.693 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.693 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:35.693 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:35.693 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:35.693 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:35.693 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:35.694 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:35.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 10 monclient: shutdown 2023-10-30T12:09:35.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:35.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 1 -- 172.21.15.50:0/927554202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00099010 msgr2=0x7f7c0009bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/927554202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00099010 0x7f7c0009bf20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40097d0 tx=0x7f7bf402b790 comp rx=0 tx=0).stop 2023-10-30T12:09:35.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 1 -- 172.21.15.50:0/927554202 shutdown_connections 2023-10-30T12:09:35.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/927554202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00099010 0x7f7c0009bf20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 1 -- 172.21.15.50:0/927554202 >> 172.21.15.50:0/927554202 conn(0x7f7c000a05c0 msgr2=0x7f7c0009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:35.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 1 -- 172.21.15.50:0/927554202 shutdown_connections 2023-10-30T12:09:35.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.671+0000 7f7c05b7c640 1 -- 172.21.15.50:0/927554202 wait complete. 2023-10-30T12:09:35.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 10 monclient: build_initial_monmap 2023-10-30T12:09:35.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 10 monclient: monmap: 2023-10-30T12:09:35.696 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:35.696 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:35.696 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:35.697 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:35.697 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:35.697 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:35.697 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:35.697 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:35.698 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:35.698 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:35.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 1 librados: starting msgr at 2023-10-30T12:09:35.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 1 librados: starting objecter 2023-10-30T12:09:35.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 1 Processor -- start 2023-10-30T12:09:35.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 1 -- start start 2023-10-30T12:09:35.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 1 librados: setting wanted keys 2023-10-30T12:09:35.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 1 librados: calling monclient init 2023-10-30T12:09:35.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.672+0000 7f7c05b7c640 10 monclient: init 2023-10-30T12:09:35.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:35.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:09:35.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00098520 0x7f7c001ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:35.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f7c00098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:35.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c00099010 0x7f7c00095780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:35.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f7c00099010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:35.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c001379f0 0x7f7c00095cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:35.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f7c001379f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:35.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:35.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000939b0 con 0x7f7c001379f0 2023-10-30T12:09:35.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:35.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00092400 con 0x7f7c00099010 2023-10-30T12:09:35.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:35.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000a30c0 con 0x7f7c00098520 2023-10-30T12:09:35.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:35.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7c05b7c640 10 monclient(hunting): authenticate will time out at 1420.856444s 2023-10-30T12:09:35.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00098520 0x7f7c001ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:35.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00098520 0x7f7c001ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52338/0 (socket says 172.21.15.50:52338) 2023-10-30T12:09:35.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 -- 172.21.15.50:0/3053543063 learned_addr learned my addr 172.21.15.50:0/3053543063 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:35.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): get_auth_request con 0x7f7c00098520 auth_method 0 2023-10-30T12:09:35.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:35.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:35.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:35.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c001379f0 0x7f7c00095cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:35.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 10 monclient(hunting): get_auth_request con 0x7f7c001379f0 auth_method 0 2023-10-30T12:09:35.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:35.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:35.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:35.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:35.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:35.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:35.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:35.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:35.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:35.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfeffd640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c00099010 0x7f7c00095780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:35.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfeffd640 10 monclient(hunting): get_auth_request con 0x7f7c00099010 auth_method 0 2023-10-30T12:09:35.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:35.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:35.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:35.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient(hunting): handle_auth_done global_id 114371 payload 824 2023-10-30T12:09:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 -- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c00099010 msgr2=0x7f7c00095780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c00099010 0x7f7c00095780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 -- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c001379f0 msgr2=0x7f7c00095cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c001379f0 0x7f7c00095cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 monclient: found mon.noname-c 2023-10-30T12:09:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bff7fe640 1 -- 172.21.15.50:0/3053543063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4009400 con 0x7f7c00098520 2023-10-30T12:09:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.673+0000 7f7bfffff640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c001379f0 0x7f7c00095cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7bff7fe640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00098520 0x7f7c001ca280 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000c520 tx=0x7f7bf000c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 client.?.objecter ms_handle_connect 0x7f7c00098520 2023-10-30T12:09:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 1 -- 172.21.15.50:0/3053543063 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0007480 con 0x7f7c00098520 2023-10-30T12:09:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:35.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: dump: 2023-10-30T12:09:35.710 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:35.710 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:35.710 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.710 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.710 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:35.711 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:35.711 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:35.711 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:35.711 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:35.711 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: _finish_auth 0 2023-10-30T12:09:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 1 -- 172.21.15.50:0/3053543063 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf00075e0 con 0x7f7c00098520 2023-10-30T12:09:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 5 monclient: authenticate success, global_id 114371 2023-10-30T12:09:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 10 monclient: _renew_subs 2023-10-30T12:09:35.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:35.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00071df0 con 0x7f7c00098520 2023-10-30T12:09:35.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:35.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 1 -- 172.21.15.50:0/3053543063 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00059c0 con 0x7f7c00098520 2023-10-30T12:09:35.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:35.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:35.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7be7fff640 10 monclient: dump: 2023-10-30T12:09:35.714 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:35.714 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:35.714 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.714 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:35.715 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:35.715 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:35.715 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:35.715 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:35.715 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:35.716 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7bfdffb640 4 set_mon_vals no callback set 2023-10-30T12:09:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 10 client.114371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 10 monclient: _renew_subs 2023-10-30T12:09:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c00092b10 con 0x7f7c00098520 2023-10-30T12:09:35.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.674+0000 7f7c05b7c640 1 librados: init done 2023-10-30T12:09:35.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7be7fff640 1 -- 172.21.15.50:0/3053543063 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7bf0005b20 con 0x7f7c00098520 2023-10-30T12:09:35.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7c05b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:35.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7be7fff640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7bec0392f0 0x7f7bec03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7c05b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7c05b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0004e160 con 0x7f7c00098520 2023-10-30T12:09:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7be7fff640 1 -- 172.21.15.50:0/3053543063 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(317..317 src has 1..317) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f7bf004a1e0 con 0x7f7c00098520 2023-10-30T12:09:35.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7be7fff640 10 client.114371.objecter ms_dispatch 0x7f7c00000b90 osd_map(317..317 src has 1..317) v4 2023-10-30T12:09:35.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7be7fff640 3 client.114371.objecter handle_osd_map got epochs [317,317] > 0 2023-10-30T12:09:35.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7be7fff640 3 client.114371.objecter handle_osd_map decoding full epoch 317 2023-10-30T12:09:35.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7be7fff640 20 client.114371.objecter dump_active .. 0 homeless 2023-10-30T12:09:35.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7bfeffd640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7bec0392f0 0x7f7bec03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.675+0000 7f7bfeffd640 10 monclient: get_auth_request con 0x7f7bec0392f0 auth_method 0 2023-10-30T12:09:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.676+0000 7f7be7fff640 10 client.114371.objecter ms_handle_connect 0x7f7bec0392f0 2023-10-30T12:09:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.676+0000 7f7bfeffd640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7bec0392f0 0x7f7bec03b7b0 secure :-1 s=READY pgs=4074 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4036000 tx=0x7f7bf4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:35.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.680+0000 7f7be7fff640 1 -- 172.21.15.50:0/3053543063 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7bf0017020 con 0x7f7c00098520 2023-10-30T12:09:35.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.680+0000 7f7be7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:35.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.680+0000 7f7be7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:35.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.923+0000 7f7c05b7c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:35.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.923+0000 7f7c05b7c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:35.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.923+0000 7f7c05b7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:35.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.923+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7c00096260 con 0x7f7c00098520 2023-10-30T12:09:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.923+0000 7f7be7fff640 1 -- 172.21.15.50:0/3053543063 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f7bf0010d60 con 0x7f7c00098520 2023-10-30T12:09:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.924+0000 7f7be7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.924+0000 7f7be7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:35.927 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:35.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 10 librados: watch_flush enter 2023-10-30T12:09:35.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 10 librados: watch_flush exit 2023-10-30T12:09:35.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 20 client.114371.objecter shutdown clearing up homeless session... 2023-10-30T12:09:35.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 10 client.114371.objecter successfully canceled tick 2023-10-30T12:09:35.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7bec0392f0 msgr2=0x7f7bec03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7bec0392f0 0x7f7bec03b7b0 secure :-1 s=READY pgs=4074 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4036000 tx=0x7f7bf4008030 comp rx=0 tx=0).stop 2023-10-30T12:09:35.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 10 monclient: shutdown 2023-10-30T12:09:35.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:35.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00098520 msgr2=0x7f7c001ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:35.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00098520 0x7f7c001ca280 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000c520 tx=0x7f7bf000c9e0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 shutdown_connections 2023-10-30T12:09:35.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7bec0392f0 0x7f7bec03b7b0 unknown :-1 s=CLOSED pgs=4074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c001379f0 0x7f7c00095cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c00099010 0x7f7c00095780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 --2- 172.21.15.50:0/3053543063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c00098520 0x7f7c001ca280 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:35.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 >> 172.21.15.50:0/3053543063 conn(0x7f7c000a05c0 msgr2=0x7f7c000967f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:35.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.925+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 shutdown_connections 2023-10-30T12:09:35.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.926+0000 7f7c05b7c640 1 -- 172.21.15.50:0/3053543063 wait complete. 2023-10-30T12:09:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:35.926+0000 7f7c05b7c640 1 librados: shutdown 2023-10-30T12:09:35.937 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:09:35.937 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:09:35.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:09:35.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:09:36.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.043+0000 7f37d8c96640 10 monclient: get_monmap_and_config 2023-10-30T12:09:36.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.043+0000 7f37d8c96640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.043+0000 7f37d8c96640 10 monclient: monmap: 2023-10-30T12:09:36.045 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.045 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.045 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:36.044144+0000 2023-10-30T12:09:36.045 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:36.044144+0000 2023-10-30T12:09:36.046 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.046 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.046 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:36.046 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:36.047 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:36.047 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:36.047 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:36.047 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.043+0000 7f37d8c96640 1 Processor -- start 2023-10-30T12:09:36.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.043+0000 7f37d8c96640 1 -- start start 2023-10-30T12:09:36.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.043+0000 7f37d8c96640 10 monclient: init 2023-10-30T12:09:36.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-10-30T12:09:36.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37d40a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:36.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f37d40a2450 0x7f37d413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient(hunting): picked mon.noname-a con 0x7f37d40a2450 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:36.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f37d413af40 0x7f37d413d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient(hunting): picked mon.noname-c con 0x7f37d413af40 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d40a4190 con 0x7f37d40a2ae0 2023-10-30T12:09:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d40a1630 con 0x7f37d40a2450 2023-10-30T12:09:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d409a2e0 con 0x7f37d413af40 2023-10-30T12:09:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d8c96640 10 monclient(hunting): authenticate will time out at 1421.227395s 2023-10-30T12:09:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d259c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f37d40a2ae0 0x7f37d40a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59030/0 (socket says 172.21.15.50:59030) 2023-10-30T12:09:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d259c640 1 -- 172.21.15.50:0/3447537646 learned_addr learned my addr 172.21.15.50:0/3447537646 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 1 --2- 172.21.15.50:0/3447537646 >> v2:172.21.15.50:3300/0 conn(0x7f37d40a2450 0x7f37d413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): get_auth_request con 0x7f37d40a2450 auth_method 0 2023-10-30T12:09:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d2d9d640 1 --2- 172.21.15.50:0/3447537646 >> v2:172.21.15.118:3300/0 conn(0x7f37d413af40 0x7f37d413d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d2d9d640 10 monclient(hunting): get_auth_request con 0x7f37d413af40 auth_method 0 2023-10-30T12:09:36.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient(hunting): handle_auth_done global_id 134139 payload 274 2023-10-30T12:09:36.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 1 -- 172.21.15.50:0/3447537646 >> v2:172.21.15.118:3300/0 conn(0x7f37d413af40 msgr2=0x7f37d413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:36.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 1 --2- 172.21.15.50:0/3447537646 >> v2:172.21.15.118:3300/0 conn(0x7f37d413af40 0x7f37d413d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 1 -- 172.21.15.50:0/3447537646 >> v1:172.21.15.50:6789/0 conn(0x7f37d40a2ae0 legacy=0x7f37d40a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 1 monclient: found mon.noname-a 2023-10-30T12:09:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.044+0000 7f37d1d9b640 1 -- 172.21.15.50:0/3447537646 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d409a8a0 con 0x7f37d40a2450 2023-10-30T12:09:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d1d9b640 1 --2- 172.21.15.50:0/3447537646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2450 0x7f37d413aa00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37d40705b0 tx=0x7f37bc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7402450831fd0d33 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 1 -- 172.21.15.50:0/3447537646 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc031710 con 0x7f37d40a2450 2023-10-30T12:09:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: dump: 2023-10-30T12:09:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.069 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 1 -- 172.21.15.50:0/3447537646 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc02f050 con 0x7f37d40a2450 2023-10-30T12:09:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d8c96640 5 monclient: authenticate success, global_id 134139 2023-10-30T12:09:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d8c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 1 -- 172.21.15.50:0/3447537646 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc031a90 con 0x7f37d40a2450 2023-10-30T12:09:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d8c96640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d8c96640 4 set_mon_vals no callback set 2023-10-30T12:09:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.045+0000 7f37d159a640 10 monclient: dump: 2023-10-30T12:09:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 10 monclient: shutdown 2023-10-30T12:09:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 -- 172.21.15.50:0/3447537646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2450 msgr2=0x7f37d413aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 --2- 172.21.15.50:0/3447537646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2450 0x7f37d413aa00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37d40705b0 tx=0x7f37bc02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 -- 172.21.15.50:0/3447537646 shutdown_connections 2023-10-30T12:09:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 --2- 172.21.15.50:0/3447537646 >> v2:172.21.15.118:3300/0 conn(0x7f37d413af40 0x7f37d413d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 --2- 172.21.15.50:0/3447537646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2450 0x7f37d413aa00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 -- 172.21.15.50:0/3447537646 >> 172.21.15.50:0/3447537646 conn(0x7f37d40a80b0 msgr2=0x7f37d40a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 -- 172.21.15.50:0/3447537646 shutdown_connections 2023-10-30T12:09:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.046+0000 7f37d8c96640 1 -- 172.21.15.50:0/3447537646 wait complete. 2023-10-30T12:09:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient: monmap: 2023-10-30T12:09:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:36.079 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 librados: starting msgr at 2023-10-30T12:09:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 librados: starting objecter 2023-10-30T12:09:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 Processor -- start 2023-10-30T12:09:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 -- start start 2023-10-30T12:09:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 librados: setting wanted keys 2023-10-30T12:09:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 librados: calling monclient init 2023-10-30T12:09:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient: init 2023-10-30T12:09:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:09:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d40a2450 0x7f37d41320e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): picked mon.noname-b con 0x7f37d40a2450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2ae0 0x7f37d409d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): picked mon.noname-a con 0x7f37d40a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37d413af40 0x7f37d409dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): picked mon.noname-c con 0x7f37d413af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d411eb20 con 0x7f37d40a2ae0 2023-10-30T12:09:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d4136030 con 0x7f37d40a2450 2023-10-30T12:09:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40a1790 con 0x7f37d413af40 2023-10-30T12:09:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d8c96640 10 monclient(hunting): authenticate will time out at 1421.230917s 2023-10-30T12:09:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.047+0000 7f37d1d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2ae0 0x7f37d409d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2ae0 0x7f37d409d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52358/0 (socket says 172.21.15.50:52358) 2023-10-30T12:09:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 1 -- 172.21.15.50:0/444232540 learned_addr learned my addr 172.21.15.50:0/444232540 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 10 monclient(hunting): get_auth_request con 0x7f37d40a2ae0 auth_method 0 2023-10-30T12:09:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37d413af40 0x7f37d409dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): get_auth_request con 0x7f37d413af40 auth_method 0 2023-10-30T12:09:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d40a2450 0x7f37d41320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 10 monclient(hunting): get_auth_request con 0x7f37d40a2450 auth_method 0 2023-10-30T12:09:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient(hunting): handle_auth_done global_id 114374 payload 824 2023-10-30T12:09:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d40a2450 msgr2=0x7f37d41320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37d40a2450 0x7f37d41320e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2ae0 msgr2=0x7f37d409d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2ae0 0x7f37d409d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 monclient: found mon.noname-c 2023-10-30T12:09:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 -- 172.21.15.50:0/444232540 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc009400 con 0x7f37d413af40 2023-10-30T12:09:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d1d9b640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d40a2ae0 0x7f37d409d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d2d9d640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37d413af40 0x7f37d409dd50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f37c4007d60 tx=0x7f37c400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 client.?.objecter ms_handle_connect 0x7f37d413af40 2023-10-30T12:09:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4018070 con 0x7f37d413af40 2023-10-30T12:09:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: dump: 2023-10-30T12:09:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.097 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c400edd0 con 0x7f37d413af40 2023-10-30T12:09:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 5 monclient: authenticate success, global_id 114374 2023-10-30T12:09:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4003bb0 con 0x7f37d413af40 2023-10-30T12:09:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37caffd640 10 monclient: dump: 2023-10-30T12:09:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.101 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 10 monclient: _renew_subs 2023-10-30T12:09:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d0d99640 4 set_mon_vals no callback set 2023-10-30T12:09:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d411e1f0 con 0x7f37d413af40 2023-10-30T12:09:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 10 client.114374.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 10 monclient: _renew_subs 2023-10-30T12:09:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d409f4b0 con 0x7f37d413af40 2023-10-30T12:09:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.048+0000 7f37d8c96640 1 librados: init done 2023-10-30T12:09:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f37c4014420 con 0x7f37d413af40 2023-10-30T12:09:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d8c96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d8c96640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d41c0030 0x7f37d409e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d8c96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d4073ff0 con 0x7f37d41c0030 2023-10-30T12:09:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d41c0030 0x7f37d409e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient: get_auth_request con 0x7f37d41c0030 auth_method 0 2023-10-30T12:09:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37caffd640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37b40392f0 0x7f37b403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(317..317 src has 1..317) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f37c404add0 con 0x7f37d413af40 2023-10-30T12:09:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37caffd640 10 client.114374.objecter ms_dispatch 0x7f37d4000b90 osd_map(317..317 src has 1..317) v4 2023-10-30T12:09:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37caffd640 3 client.114374.objecter handle_osd_map got epochs [317,317] > 0 2023-10-30T12:09:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37caffd640 3 client.114374.objecter handle_osd_map decoding full epoch 317 2023-10-30T12:09:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37caffd640 20 client.114374.objecter dump_active .. 0 homeless 2023-10-30T12:09:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 10 monclient(hunting): handle_auth_done global_id 134148 payload 824 2023-10-30T12:09:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d259c640 1 -- 172.21.15.50:0/444232540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37d409a5c0 con 0x7f37d41c0030 2023-10-30T12:09:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.049+0000 7f37d1d9b640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37b40392f0 0x7f37b403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37d259c640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d41c0030 0x7f37d409e290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 client.114374.objecter ms_handle_connect 0x7f37d41c0030 2023-10-30T12:09:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 client.114374.objecter resend_mon_ops 2023-10-30T12:09:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37d1d9b640 10 monclient: get_auth_request con 0x7f37b40392f0 auth_method 0 2023-10-30T12:09:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc0058b0 con 0x7f37d41c0030 2023-10-30T12:09:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 monclient: dump: 2023-10-30T12:09:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.111 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f37cc005a10 con 0x7f37d41c0030 2023-10-30T12:09:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d41c0030 msgr2=0x7f37d409e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d41c0030 0x7f37d409e290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc00fc00 comp rx=0 tx=0).stop 2023-10-30T12:09:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d41c0030 msgr2=0x7f37d409e290 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:36.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d41c0030 0x7f37d409e290 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37d1d9b640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37b40392f0 0x7f37b403b7b0 secure :-1 s=READY pgs=4075 cs=0 l=1 rev1=1 crypto rx=0x7f37d409f0b0 tx=0x7f37bc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.050+0000 7f37caffd640 10 client.114374.objecter ms_handle_connect 0x7f37b40392f0 2023-10-30T12:09:36.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d8c96640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:36.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d8c96640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d409e7d0 0x7f37d409ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d8c96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d4099b40 con 0x7f37d409e7d0 2023-10-30T12:09:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d409e7d0 0x7f37d409ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient: get_auth_request con 0x7f37d409e7d0 auth_method 0 2023-10-30T12:09:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.056+0000 7f37d259c640 10 monclient(hunting): handle_auth_done global_id 134151 payload 824 2023-10-30T12:09:36.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d259c640 1 -- 172.21.15.50:0/444232540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f37d40a28a0 con 0x7f37d409e7d0 2023-10-30T12:09:36.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d259c640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d409e7d0 0x7f37d409ad80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 10 client.114374.objecter ms_handle_connect 0x7f37d409e7d0 2023-10-30T12:09:36.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 10 client.114374.objecter resend_mon_ops 2023-10-30T12:09:36.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc024070 con 0x7f37d409e7d0 2023-10-30T12:09:36.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:36.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 10 monclient: dump: 2023-10-30T12:09:36.120 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.120 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.120 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.120 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.120 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.121 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.121 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.121 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.121 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.121 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f37cc007760 con 0x7f37d409e7d0 2023-10-30T12:09:36.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:36.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:36.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d409e7d0 msgr2=0x7f37d409ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d409e7d0 0x7f37d409ad80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d409e7d0 msgr2=0x7f37d409ad80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:36.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37caffd640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37d409e7d0 0x7f37d409ad80 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.123 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:36.124 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:36.124 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:36.124 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:09:36.124 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 88, 2023-10-30T12:09:36.124 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:36.125 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:36.125 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:09:36.125 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:36.125 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.125 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:09:36.126 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:36.126 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:36.126 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:36.126 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.127 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.127 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.127 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.127 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.127 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.128 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.128 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.128 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.128 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.129 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.129 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:36.129 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:36.129 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.129 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.130 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.130 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.130 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.130 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.131 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.131 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.131 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.131 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.132 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.132 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:36.132 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:36.132 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:36.133 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:36.133 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:36.133 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:36.133 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:36.133 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:36.134 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:36.134 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:36.134 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:36.134 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:36.135 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:36.135 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:36.135 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:36.135 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:36.135 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:36.136 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.136 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.136 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.136 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.136 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.136 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.137 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.137 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.137 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.137 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.138 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.138 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:36.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.138 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:36.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.139 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:36.139 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:36.139 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.139 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.140 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.140 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:36.140 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.141 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.141 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:36.141 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.141 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.142 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:36.142 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:36.142 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.142 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.143 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.143 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:36.144 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:36.144 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.144 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.144 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:36.145 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.145 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.146 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:36.146 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.146 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.147 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:36.147 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:36.147 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.147 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.147 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.148 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:36.148 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:36.148 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.149 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.149 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.149 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:36.149 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.150 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.150 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:36.150 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.151 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.151 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.151 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:36.151 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:36.152 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.152 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.152 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.152 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.153 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:36.153 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:36.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.154 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.154 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.154 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:36.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.154 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:36.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.155 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.155 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.155 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:36.156 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.156 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.156 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:36.157 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:36.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 10 librados: watch_flush enter 2023-10-30T12:09:36.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 10 librados: watch_flush exit 2023-10-30T12:09:36.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 20 client.114374.objecter shutdown clearing up homeless session... 2023-10-30T12:09:36.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 10 client.114374.objecter successfully canceled tick 2023-10-30T12:09:36.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37b40392f0 msgr2=0x7f37b403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37b40392f0 0x7f37b403b7b0 secure :-1 s=READY pgs=4075 cs=0 l=1 rev1=1 crypto rx=0x7f37d409f0b0 tx=0x7f37bc008030 comp rx=0 tx=0).stop 2023-10-30T12:09:36.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 10 monclient: shutdown 2023-10-30T12:09:36.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37d413af40 msgr2=0x7f37d409dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d2d9d640 1 -- 172.21.15.50:0/444232540 reap_dead start 2023-10-30T12:09:36.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37d413af40 0x7f37d409dd50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f37c4007d60 tx=0x7f37c400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 shutdown_connections 2023-10-30T12:09:36.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 --2- 172.21.15.50:0/444232540 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37d413af40 0x7f37d409dd50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 >> 172.21.15.50:0/444232540 conn(0x7f37d40a80b0 msgr2=0x7f37d4134720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:36.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.057+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 shutdown_connections 2023-10-30T12:09:36.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.058+0000 7f37d8c96640 1 -- 172.21.15.50:0/444232540 wait complete. 2023-10-30T12:09:36.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.058+0000 7f37d8c96640 1 librados: shutdown 2023-10-30T12:09:36.165 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:09:36.165 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:09:36.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:09:36.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.270+0000 7efd1b305640 10 monclient: get_monmap_and_config 2023-10-30T12:09:36.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.270+0000 7efd1b305640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.270+0000 7efd1b305640 10 monclient: monmap: 2023-10-30T12:09:36.272 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.273 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.273 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:36.271777+0000 2023-10-30T12:09:36.273 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:36.271777+0000 2023-10-30T12:09:36.273 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:36.275 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:36.275 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 1 Processor -- start 2023-10-30T12:09:36.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 1 -- start start 2023-10-30T12:09:36.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient: init 2023-10-30T12:09:36.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:09:36.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd14097fe0 0x7efd1409a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient(hunting): picked mon.noname-a con 0x7efd14097fe0 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:36.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd14071c00 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:36.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efd14137850 0x7efd14139c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient(hunting): picked mon.noname-c con 0x7efd14137850 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:36.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd14097600 con 0x7efd14071c00 2023-10-30T12:09:36.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1409faa0 con 0x7efd14097fe0 2023-10-30T12:09:36.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd140978e0 con 0x7efd14137850 2023-10-30T12:09:36.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.271+0000 7efd1b305640 10 monclient(hunting): authenticate will time out at 1421.454963s 2023-10-30T12:09:36.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd14097fe0 0x7efd1409a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd188a0640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efd14071c00 0x7efd1409f390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59032/0 (socket says 172.21.15.50:59032) 2023-10-30T12:09:36.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd188a0640 1 -- 172.21.15.50:0/2119937369 learned_addr learned my addr 172.21.15.50:0/2119937369 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:36.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd14097fe0 0x7efd1409a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52388/0 (socket says 172.21.15.50:52388) 2023-10-30T12:09:36.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): get_auth_request con 0x7efd14097fe0 auth_method 0 2023-10-30T12:09:36.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd198a2640 1 --2- 172.21.15.50:0/2119937369 >> v2:172.21.15.118:3300/0 conn(0x7efd14137850 0x7efd14139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd198a2640 10 monclient(hunting): get_auth_request con 0x7efd14137850 auth_method 0 2023-10-30T12:09:36.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd198a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd198a2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd198a2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient(hunting): handle_auth_done global_id 134154 payload 274 2023-10-30T12:09:36.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 1 -- 172.21.15.50:0/2119937369 >> v2:172.21.15.118:3300/0 conn(0x7efd14137850 msgr2=0x7efd14139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:36.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 1 --2- 172.21.15.50:0/2119937369 >> v2:172.21.15.118:3300/0 conn(0x7efd14137850 0x7efd14139c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 1 -- 172.21.15.50:0/2119937369 >> v1:172.21.15.50:6789/0 conn(0x7efd14071c00 legacy=0x7efd1409f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 1 monclient: found mon.noname-a 2023-10-30T12:09:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.272+0000 7efd190a1640 1 -- 172.21.15.50:0/2119937369 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1409bf50 con 0x7efd14097fe0 2023-10-30T12:09:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd190a1640 1 --2- 172.21.15.50:0/2119937369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1409a150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd1409c680 tx=0x7efd0802df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed26327b190a06e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 1 -- 172.21.15.50:0/2119937369 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0802ed30 con 0x7efd14097fe0 2023-10-30T12:09:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: dump: 2023-10-30T12:09:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.294 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.295 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.296 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 1 -- 172.21.15.50:0/2119937369 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0802ee90 con 0x7efd14097fe0 2023-10-30T12:09:36.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 5 monclient: authenticate success, global_id 134154 2023-10-30T12:09:36.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:36.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:36.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 1 -- 172.21.15.50:0/2119937369 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08036660 con 0x7efd14097fe0 2023-10-30T12:09:36.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 4 set_mon_vals no callback set 2023-10-30T12:09:36.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:36.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd07fff640 10 monclient: dump: 2023-10-30T12:09:36.299 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.299 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.299 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.299 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.301 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 10 monclient: shutdown 2023-10-30T12:09:36.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 -- 172.21.15.50:0/2119937369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 msgr2=0x7efd1409a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 --2- 172.21.15.50:0/2119937369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1409a150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd1409c680 tx=0x7efd0802df80 comp rx=0 tx=0).stop 2023-10-30T12:09:36.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 -- 172.21.15.50:0/2119937369 shutdown_connections 2023-10-30T12:09:36.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 --2- 172.21.15.50:0/2119937369 >> v2:172.21.15.118:3300/0 conn(0x7efd14137850 0x7efd14139c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 --2- 172.21.15.50:0/2119937369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1409a150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 -- 172.21.15.50:0/2119937369 >> 172.21.15.50:0/2119937369 conn(0x7efd140a3980 msgr2=0x7efd140a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:36.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 -- 172.21.15.50:0/2119937369 shutdown_connections 2023-10-30T12:09:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.273+0000 7efd1b305640 1 -- 172.21.15.50:0/2119937369 wait complete. 2023-10-30T12:09:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 10 monclient: monmap: 2023-10-30T12:09:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:36.305 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 1 librados: starting msgr at 2023-10-30T12:09:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 1 librados: starting objecter 2023-10-30T12:09:36.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 1 Processor -- start 2023-10-30T12:09:36.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 1 -- start start 2023-10-30T12:09:36.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 1 librados: setting wanted keys 2023-10-30T12:09:36.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 1 librados: calling monclient init 2023-10-30T12:09:36.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.274+0000 7efd1b305640 10 monclient: init 2023-10-30T12:09:36.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:09:36.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14071c00 0x7efd1408afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): picked mon.noname-c con 0x7efd14071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:36.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1408b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): picked mon.noname-a con 0x7efd14097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:36.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd14137850 0x7efd1408ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): picked mon.noname-b con 0x7efd14137850 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:36.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd140718f0 con 0x7efd14097fe0 2023-10-30T12:09:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd14093020 con 0x7efd14137850 2023-10-30T12:09:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd14099100 con 0x7efd14071c00 2023-10-30T12:09:36.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd1b305640 10 monclient(hunting): authenticate will time out at 1421.458271s 2023-10-30T12:09:36.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1408b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14071c00 0x7efd1408afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1408b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52404/0 (socket says 172.21.15.50:52404) 2023-10-30T12:09:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 -- 172.21.15.50:0/912417498 learned_addr learned my addr 172.21.15.50:0/912417498 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): get_auth_request con 0x7efd14097fe0 auth_method 0 2023-10-30T12:09:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 10 monclient(hunting): get_auth_request con 0x7efd14071c00 auth_method 0 2023-10-30T12:09:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd198a2640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd14137850 0x7efd1408ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd198a2640 10 monclient(hunting): get_auth_request con 0x7efd14137850 auth_method 0 2023-10-30T12:09:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd198a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd198a2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd198a2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient(hunting): handle_auth_done global_id 134160 payload 824 2023-10-30T12:09:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14071c00 msgr2=0x7efd1408afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14071c00 0x7efd1408afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd14137850 msgr2=0x7efd1408ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd14137850 0x7efd1408ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 monclient: found mon.noname-a 2023-10-30T12:09:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 -- 172.21.15.50:0/912417498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0800b640 con 0x7efd14097fe0 2023-10-30T12:09:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd190a1640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14071c00 0x7efd1408afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd188a0640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1408b510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd1000c450 tx=0x7efd1000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd05ffb640 10 client.?.objecter ms_handle_connect 0x7efd14097fe0 2023-10-30T12:09:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd05ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10005ce0 con 0x7efd14097fe0 2023-10-30T12:09:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.275+0000 7efd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: dump: 2023-10-30T12:09:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.320 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 5 monclient: authenticate success, global_id 134160 2023-10-30T12:09:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd10005e40 con 0x7efd14097fe0 2023-10-30T12:09:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 10 monclient: _renew_subs 2023-10-30T12:09:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1408d3d0 con 0x7efd14097fe0 2023-10-30T12:09:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10007310 con 0x7efd14097fe0 2023-10-30T12:09:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 10 monclient: dump: 2023-10-30T12:09:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.325 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd077fe640 4 set_mon_vals no callback set 2023-10-30T12:09:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 10 client.134160.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 10 monclient: _renew_subs 2023-10-30T12:09:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1408d090 con 0x7efd14097fe0 2023-10-30T12:09:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 1 librados: init done 2023-10-30T12:09:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efd10007470 con 0x7efd14097fe0 2023-10-30T12:09:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095410 0x7efd140957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.276+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd14073ff0 con 0x7efd14095410 2023-10-30T12:09:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095410 0x7efd140957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient: get_auth_request con 0x7efd14095410 auth_method 0 2023-10-30T12:09:36.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efcf80392f0 0x7efcf803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(317..317 src has 1..317) v4 ==== 6299+0+0 (secure 0 0 0) 0x7efd1004a240 con 0x7efd14097fe0 2023-10-30T12:09:36.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 10 client.134160.objecter ms_dispatch 0x7efd14000b90 osd_map(317..317 src has 1..317) v4 2023-10-30T12:09:36.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 3 client.134160.objecter handle_osd_map got epochs [317,317] > 0 2023-10-30T12:09:36.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 3 client.134160.objecter handle_osd_map decoding full epoch 317 2023-10-30T12:09:36.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 20 client.134160.objecter dump_active .. 0 homeless 2023-10-30T12:09:36.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 10 monclient(hunting): handle_auth_done global_id 114380 payload 824 2023-10-30T12:09:36.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd198a2640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efcf80392f0 0x7efcf803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 1 -- 172.21.15.50:0/912417498 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd14072030 con 0x7efd14095410 2023-10-30T12:09:36.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd198a2640 10 monclient: get_auth_request con 0x7efcf80392f0 auth_method 0 2023-10-30T12:09:36.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 10 client.134160.objecter ms_handle_connect 0x7efd14095410 2023-10-30T12:09:36.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 10 client.134160.objecter resend_mon_ops 2023-10-30T12:09:36.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd190a1640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095410 0x7efd140957f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd1406bb30 tx=0x7efd08004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08005b90 con 0x7efd14095410 2023-10-30T12:09:36.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:36.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 10 monclient: dump: 2023-10-30T12:09:36.333 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.333 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.333 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.334 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.334 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.334 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.334 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.334 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.335 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.335 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd05ffb640 10 client.134160.objecter ms_handle_connect 0x7efcf80392f0 2023-10-30T12:09:36.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.277+0000 7efd198a2640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efcf80392f0 0x7efcf803b7b0 secure :-1 s=READY pgs=4076 cs=0 l=1 rev1=1 crypto rx=0x7efd1408c710 tx=0x7efd0c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.278+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7efd08005cf0 con 0x7efd14095410 2023-10-30T12:09:36.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.278+0000 7efd05ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.278+0000 7efd05ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:36.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.278+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095410 msgr2=0x7efd140957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.278+0000 7efd05ffb640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095410 0x7efd140957f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd1406bb30 tx=0x7efd08004730 comp rx=0 tx=0).stop 2023-10-30T12:09:36.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.278+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095410 msgr2=0x7efd140957f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:36.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.278+0000 7efd05ffb640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095410 0x7efd140957f0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd1406bb30 tx=0x7efd08004730 comp rx=0 tx=0).stop 2023-10-30T12:09:36.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd1b305640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:36.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd1b305640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095d30 0x7efd141d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd1b305640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1409cc20 con 0x7efd14095d30 2023-10-30T12:09:36.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095d30 0x7efd141d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient: get_auth_request con 0x7efd14095d30 auth_method 0 2023-10-30T12:09:36.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 10 monclient(hunting): handle_auth_done global_id 114383 payload 824 2023-10-30T12:09:36.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.284+0000 7efd190a1640 1 -- 172.21.15.50:0/912417498 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efd1409d9b0 con 0x7efd14095d30 2023-10-30T12:09:36.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd190a1640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095d30 0x7efd141d4a70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd1408cdf0 tx=0x7efd080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 10 client.134160.objecter ms_handle_connect 0x7efd14095d30 2023-10-30T12:09:36.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 10 client.134160.objecter resend_mon_ops 2023-10-30T12:09:36.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08051070 con 0x7efd14095d30 2023-10-30T12:09:36.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:36.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 10 monclient: dump: 2023-10-30T12:09:36.343 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.343 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.343 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.344 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.344 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.344 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.344 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.345 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.345 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.345 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7efd08006640 con 0x7efd14095d30 2023-10-30T12:09:36.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:36.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:36.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095d30 msgr2=0x7efd141d4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095d30 0x7efd141d4a70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd1408cdf0 tx=0x7efd080046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095d30 msgr2=0x7efd141d4a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:36.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd05ffb640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efd14095d30 0x7efd141d4a70 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd1408cdf0 tx=0x7efd080046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.347 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:36.347 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:36.347 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:36.348 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:09:36.348 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 88, 2023-10-30T12:09:36.348 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:36.348 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:36.348 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:09:36.349 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:36.349 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.349 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:09:36.349 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:36.349 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:36.350 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:36.350 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.350 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.350 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.351 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.351 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.351 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.351 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.351 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.352 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.352 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.352 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.352 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:36.352 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:36.353 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.353 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.353 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.353 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.353 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.354 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.354 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.354 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.354 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.354 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.355 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:36.355 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:36.355 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:36.356 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:36.356 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:36.356 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:36.356 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:36.357 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:36.357 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:36.357 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:36.357 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:36.357 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:36.358 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:36.358 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:36.358 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:36.358 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:36.358 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:36.359 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.359 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.359 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.359 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.359 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.360 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.360 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.360 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.360 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.360 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.361 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.361 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:36.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.361 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:36.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.362 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:36.362 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:36.362 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.363 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.363 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:36.363 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.365 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.365 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:36.365 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.366 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.366 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:36.367 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:36.367 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.367 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.367 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.368 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:36.368 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:36.368 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.369 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.369 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:36.369 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.370 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:36.370 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.371 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.371 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.371 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:36.372 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:36.372 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.372 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.372 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.373 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:36.373 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:36.373 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.374 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.374 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.374 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:36.374 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.375 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:36.376 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.376 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.376 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:36.377 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:36.377 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.377 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.377 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.378 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.378 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:36.378 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:09:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:09:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:36.383 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:36.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 10 librados: watch_flush enter 2023-10-30T12:09:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 10 librados: watch_flush exit 2023-10-30T12:09:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 20 client.134160.objecter shutdown clearing up homeless session... 2023-10-30T12:09:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 10 client.134160.objecter successfully canceled tick 2023-10-30T12:09:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efcf80392f0 msgr2=0x7efcf803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efcf80392f0 0x7efcf803b7b0 secure :-1 s=READY pgs=4076 cs=0 l=1 rev1=1 crypto rx=0x7efd1408c710 tx=0x7efd0c00b040 comp rx=0 tx=0).stop 2023-10-30T12:09:36.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 10 monclient: shutdown 2023-10-30T12:09:36.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 msgr2=0x7efd1408b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 --2- 172.21.15.50:0/912417498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd14097fe0 0x7efd1408b510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd1000c450 tx=0x7efd1000c910 comp rx=0 tx=0).stop 2023-10-30T12:09:36.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd198a2640 1 -- 172.21.15.50:0/912417498 reap_dead start 2023-10-30T12:09:36.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd198a2640 1 -- 172.21.15.50:0/912417498 reap_dead start 2023-10-30T12:09:36.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 shutdown_connections 2023-10-30T12:09:36.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 >> 172.21.15.50:0/912417498 conn(0x7efd140a3980 msgr2=0x7efd1409a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:36.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 shutdown_connections 2023-10-30T12:09:36.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.285+0000 7efd1b305640 1 -- 172.21.15.50:0/912417498 wait complete. 2023-10-30T12:09:36.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.286+0000 7efd1b305640 1 librados: shutdown 2023-10-30T12:09:36.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:09:36.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:09:36.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:09:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.494+0000 7f54c7e75640 10 monclient: get_monmap_and_config 2023-10-30T12:09:36.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.494+0000 7f54c7e75640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.494+0000 7f54c7e75640 10 monclient: monmap: 2023-10-30T12:09:36.496 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:36.495542+0000 2023-10-30T12:09:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:36.495542+0000 2023-10-30T12:09:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:36.500 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 1 Processor -- start 2023-10-30T12:09:36.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 1 -- start start 2023-10-30T12:09:36.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient: init 2023-10-30T12:09:36.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-10-30T12:09:36.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54c006afc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:36.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54c006b640 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:36.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c013e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient(hunting): picked mon.noname-b con 0x7f54c013c080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c006b3a0 con 0x7f54c006afc0 2023-10-30T12:09:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c009e540 con 0x7f54c006b640 2023-10-30T12:09:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00a2ab0 con 0x7f54c013c080 2023-10-30T12:09:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c7e75640 10 monclient(hunting): authenticate will time out at 1421.678754s 2023-10-30T12:09:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c5c11640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f54c006afc0 0x7f54c00a1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59036/0 (socket says 172.21.15.50:59036) 2023-10-30T12:09:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c5c11640 1 -- 172.21.15.50:0/987194777 learned_addr learned my addr 172.21.15.50:0/987194777 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c6412640 1 --2- 172.21.15.50:0/987194777 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c013e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c6412640 10 monclient(hunting): get_auth_request con 0x7f54c013c080 auth_method 0 2023-10-30T12:09:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c6412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c6412640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.495+0000 7f54c6412640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 10 monclient(hunting): handle_auth_done global_id 114386 payload 274 2023-10-30T12:09:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c4c0f640 1 -- 172.21.15.50:0/987194777 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672877489 0 0) 0x7f54c006b3a0 con 0x7f54c006afc0 2023-10-30T12:09:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 1 -- 172.21.15.50:0/987194777 >> v1:172.21.15.118:6789/0 conn(0x7f54c006b640 legacy=0x7f54c00a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 1 -- 172.21.15.50:0/987194777 >> v1:172.21.15.50:6789/0 conn(0x7f54c006afc0 legacy=0x7f54c00a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 1 monclient: found mon.noname-b 2023-10-30T12:09:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:09:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 1 -- 172.21.15.50:0/987194777 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c009fa10 con 0x7f54c013c080 2023-10-30T12:09:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c4c0f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:09:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c6412640 1 --2- 172.21.15.50:0/987194777 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c013e470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009dc0 tx=0x7f54b802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fdd5e473d3c9bfa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.496+0000 7f54c4c0f640 1 -- 172.21.15.50:0/987194777 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b802cbb0 con 0x7f54c013c080 2023-10-30T12:09:36.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:36.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: dump: 2023-10-30T12:09:36.520 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.520 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.520 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.520 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.520 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.521 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 5 monclient: authenticate success, global_id 114386 2023-10-30T12:09:36.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 1 -- 172.21.15.50:0/987194777 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b802cd10 con 0x7f54c013c080 2023-10-30T12:09:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 1 -- 172.21.15.50:0/987194777 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8034660 con 0x7f54c013c080 2023-10-30T12:09:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 4 set_mon_vals no callback set 2023-10-30T12:09:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c4c0f640 10 monclient: dump: 2023-10-30T12:09:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.525 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.525 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.525 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.525 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.525 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.526 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.526 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.526 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 10 monclient: shutdown 2023-10-30T12:09:36.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 1 -- 172.21.15.50:0/987194777 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 msgr2=0x7f54c013e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 1 --2- 172.21.15.50:0/987194777 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c013e470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009dc0 tx=0x7f54b802be00 comp rx=0 tx=0).stop 2023-10-30T12:09:36.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 1 -- 172.21.15.50:0/987194777 shutdown_connections 2023-10-30T12:09:36.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 1 --2- 172.21.15.50:0/987194777 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c013e470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 1 -- 172.21.15.50:0/987194777 >> 172.21.15.50:0/987194777 conn(0x7f54c00a6200 msgr2=0x7f54c00a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 1 -- 172.21.15.50:0/987194777 shutdown_connections 2023-10-30T12:09:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.497+0000 7f54c7e75640 1 -- 172.21.15.50:0/987194777 wait complete. 2023-10-30T12:09:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient: monmap: 2023-10-30T12:09:36.529 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.529 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:36.529 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:36.529 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:36.529 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.530 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.530 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:36.530 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:36.530 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:36.531 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 librados: starting msgr at 2023-10-30T12:09:36.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 librados: starting objecter 2023-10-30T12:09:36.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 Processor -- start 2023-10-30T12:09:36.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 -- start start 2023-10-30T12:09:36.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 librados: setting wanted keys 2023-10-30T12:09:36.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 librados: calling monclient init 2023-10-30T12:09:36.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient: init 2023-10-30T12:09:36.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:36.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54c006afc0 0x7f54c01ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): picked mon.noname-a con 0x7f54c006afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:36.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c006b640 0x7f54c01ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): picked mon.noname-b con 0x7f54c006b640 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:36.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c011c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): picked mon.noname-c con 0x7f54c013c080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:36.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c0070de0 con 0x7f54c006afc0 2023-10-30T12:09:36.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c0120770 con 0x7f54c006b640 2023-10-30T12:09:36.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c0120a50 con 0x7f54c013c080 2023-10-30T12:09:36.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.498+0000 7f54c7e75640 10 monclient(hunting): authenticate will time out at 1421.681946s 2023-10-30T12:09:36.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54c006afc0 0x7f54c01ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c011c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54c006afc0 0x7f54c01ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52410/0 (socket says 172.21.15.50:52410) 2023-10-30T12:09:36.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 1 -- 172.21.15.50:0/3841276289 learned_addr learned my addr 172.21.15.50:0/3841276289 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:36.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): get_auth_request con 0x7f54c013c080 auth_method 0 2023-10-30T12:09:36.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 10 monclient(hunting): get_auth_request con 0x7f54c006afc0 auth_method 0 2023-10-30T12:09:36.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c006b640 0x7f54c01ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 10 monclient(hunting): get_auth_request con 0x7f54c006b640 auth_method 0 2023-10-30T12:09:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5410640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient(hunting): handle_auth_done global_id 114389 payload 824 2023-10-30T12:09:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c006b640 msgr2=0x7f54c01ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c006b640 0x7f54c01ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54c006afc0 msgr2=0x7f54c01ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54c006afc0 0x7f54c01ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 monclient: found mon.noname-c 2023-10-30T12:09:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 -- 172.21.15.50:0/3841276289 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b8009a70 con 0x7f54c013c080 2023-10-30T12:09:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c5c11640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54c006afc0 0x7f54c01ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c6412640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c011c9e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f54b8033000 tx=0x7f54b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 client.?.objecter ms_handle_connect 0x7f54c013c080 2023-10-30T12:09:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8003b90 con 0x7f54c013c080 2023-10-30T12:09:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: dump: 2023-10-30T12:09:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.548 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8003cf0 con 0x7f54c013c080 2023-10-30T12:09:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 5 monclient: authenticate success, global_id 114389 2023-10-30T12:09:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 10 monclient: _renew_subs 2023-10-30T12:09:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c0138380 con 0x7f54c013c080 2023-10-30T12:09:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8004270 con 0x7f54c013c080 2023-10-30T12:09:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b27fc640 10 monclient: dump: 2023-10-30T12:09:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.553 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 10 client.114389.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 10 monclient: _renew_subs 2023-10-30T12:09:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54b3fff640 4 set_mon_vals no callback set 2023-10-30T12:09:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c01201b0 con 0x7f54c013c080 2023-10-30T12:09:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.499+0000 7f54c7e75640 1 librados: init done 2023-10-30T12:09:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c7e75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c7e75640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011cf20 0x7f54c011d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c7e75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c0073ff0 con 0x7f54c011cf20 2023-10-30T12:09:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f54b803b050 con 0x7f54c013c080 2023-10-30T12:09:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54b27fc640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a40392f0 0x7f54a403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011cf20 0x7f54c011d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 10 monclient: get_auth_request con 0x7f54c011cf20 auth_method 0 2023-10-30T12:09:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(317..317 src has 1..317) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f54b806d230 con 0x7f54c013c080 2023-10-30T12:09:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54b27fc640 10 client.114389.objecter ms_dispatch 0x7f54c0000b90 osd_map(317..317 src has 1..317) v4 2023-10-30T12:09:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54b27fc640 3 client.114389.objecter handle_osd_map got epochs [317,317] > 0 2023-10-30T12:09:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54b27fc640 3 client.114389.objecter handle_osd_map decoding full epoch 317 2023-10-30T12:09:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54b27fc640 20 client.114389.objecter dump_active .. 0 homeless 2023-10-30T12:09:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.500+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54c5410640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a40392f0 0x7f54a403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54c5410640 10 monclient: get_auth_request con 0x7f54a40392f0 auth_method 0 2023-10-30T12:09:36.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54c5c11640 10 monclient(hunting): handle_auth_done global_id 94783 payload 824 2023-10-30T12:09:36.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54c5c11640 1 -- 172.21.15.50:0/3841276289 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54c009df80 con 0x7f54c011cf20 2023-10-30T12:09:36.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54c5c11640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011cf20 0x7f54c011d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00fa70 tx=0x7f54bc00ff30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 client.114389.objecter ms_handle_connect 0x7f54c011cf20 2023-10-30T12:09:36.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 client.114389.objecter resend_mon_ops 2023-10-30T12:09:36.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc00dab0 con 0x7f54c011cf20 2023-10-30T12:09:36.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54c5410640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a40392f0 0x7f54a403b7b0 secure :-1 s=READY pgs=4077 cs=0 l=1 rev1=1 crypto rx=0x7f54c011e320 tx=0x7f54b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:36.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 monclient: dump: 2023-10-30T12:09:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.562 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 client.114389.objecter ms_handle_connect 0x7f54a40392f0 2023-10-30T12:09:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f54bc00dc10 con 0x7f54c011cf20 2023-10-30T12:09:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011cf20 msgr2=0x7f54c011d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011cf20 0x7f54c011d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00fa70 tx=0x7f54bc00ff30 comp rx=0 tx=0).stop 2023-10-30T12:09:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011cf20 msgr2=0x7f54c011d300 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.501+0000 7f54b27fc640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011cf20 0x7f54c011d300 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00fa70 tx=0x7f54bc00ff30 comp rx=0 tx=0).stop 2023-10-30T12:09:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c7e75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c7e75640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011d840 0x7f54c0138510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c7e75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c006a6f0 con 0x7f54c011d840 2023-10-30T12:09:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011d840 0x7f54c0138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient: get_auth_request con 0x7f54c011d840 auth_method 0 2023-10-30T12:09:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 10 monclient(hunting): handle_auth_done global_id 94786 payload 824 2023-10-30T12:09:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 1 -- 172.21.15.50:0/3841276289 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f54c009f0b0 con 0x7f54c011d840 2023-10-30T12:09:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54c5c11640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011d840 0x7f54c0138510 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00fa70 tx=0x7f54bc000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54b27fc640 10 client.114389.objecter ms_handle_connect 0x7f54c011d840 2023-10-30T12:09:36.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.508+0000 7f54b27fc640 10 client.114389.objecter resend_mon_ops 2023-10-30T12:09:36.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc023070 con 0x7f54c011d840 2023-10-30T12:09:36.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:09:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 10 monclient: dump: 2023-10-30T12:09:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.571 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f54bc007720 con 0x7f54c011d840 2023-10-30T12:09:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011d840 msgr2=0x7f54c0138510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011d840 0x7f54c0138510 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00fa70 tx=0x7f54bc000ed0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011d840 msgr2=0x7f54c0138510 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54b27fc640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54c011d840 0x7f54c0138510 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00fa70 tx=0x7f54bc000ed0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.573 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:36.573 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:36.573 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:36.573 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:09:36.574 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 88, 2023-10-30T12:09:36.574 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:36.574 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:36.574 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:09:36.574 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:36.575 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.575 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:09:36.575 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:36.575 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:36.576 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:36.576 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.576 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.576 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.576 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.577 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.577 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.577 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.577 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.577 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.578 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.578 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.578 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:36.578 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:36.579 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.579 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.579 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.579 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.579 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.580 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.580 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.580 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.580 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.581 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.581 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.581 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:36.581 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:36.582 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:36.582 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:36.582 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:36.582 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:36.582 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:36.583 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:36.583 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:36.583 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:36.583 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:36.584 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:36.584 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:36.584 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:36.584 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:36.584 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:36.585 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:36.585 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:36.585 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:36.585 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:36.585 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:36.586 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:36.586 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:36.586 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:36.586 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:36.586 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:36.587 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:36.587 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.587 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:36.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.587 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:36.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.588 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:36.588 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:36.588 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.588 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.589 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.589 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:36.589 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.590 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.590 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:36.590 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.591 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.591 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:36.591 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:36.592 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.592 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.592 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.593 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:36.593 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:36.593 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.593 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.594 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.594 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:36.594 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.595 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.595 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:36.595 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.595 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.596 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.596 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:36.596 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:36.596 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.597 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.597 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.597 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:36.598 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:36.598 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:36.598 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:36.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:36.599 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:36.599 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.599 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:36.600 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:36.600 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:36.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.600 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.601 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:36.601 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:36.601 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:36.601 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:36.602 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:36.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.602 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.602 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:36.603 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:36.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.603 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.603 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.603 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:36.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.604 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.604 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:09:36.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.605 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.605 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.605 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:09:36.605 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.605 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.606 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:36.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.606 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.606 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.606 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:09:36.607 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.607 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:36.607 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:09:36.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:36.608 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:36.608 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:36.608 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:36.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:36.608 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:36.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:36.609 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:36.609 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:36.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 10 librados: watch_flush enter 2023-10-30T12:09:36.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 10 librados: watch_flush exit 2023-10-30T12:09:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 20 client.114389.objecter shutdown clearing up homeless session... 2023-10-30T12:09:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 10 client.114389.objecter successfully canceled tick 2023-10-30T12:09:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a40392f0 msgr2=0x7f54a403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a40392f0 0x7f54a403b7b0 secure :-1 s=READY pgs=4077 cs=0 l=1 rev1=1 crypto rx=0x7f54c011e320 tx=0x7f54b4009040 comp rx=0 tx=0).stop 2023-10-30T12:09:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 10 monclient: shutdown 2023-10-30T12:09:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 msgr2=0x7f54c011c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 --2- 172.21.15.50:0/3841276289 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54c013c080 0x7f54c011c9e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f54b8033000 tx=0x7f54b8004a40 comp rx=0 tx=0).stop 2023-10-30T12:09:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c6412640 1 -- 172.21.15.50:0/3841276289 reap_dead start 2023-10-30T12:09:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c6412640 1 -- 172.21.15.50:0/3841276289 reap_dead start 2023-10-30T12:09:36.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 shutdown_connections 2023-10-30T12:09:36.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 >> 172.21.15.50:0/3841276289 conn(0x7f54c00a6200 msgr2=0x7f54c013b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:36.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 shutdown_connections 2023-10-30T12:09:36.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 -- 172.21.15.50:0/3841276289 wait complete. 2023-10-30T12:09:36.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.509+0000 7f54c7e75640 1 librados: shutdown 2023-10-30T12:09:36.615 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:09:36.615 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-10-30T12:09:36.616 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:09:36.616 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-10-30T12:09:36.616 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:09:36.616 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:09:36.617 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:09:36.633 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:09:36.634 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:09:36.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:36.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.737+0000 7f9009410640 10 monclient: get_monmap_and_config 2023-10-30T12:09:36.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.737+0000 7f9009410640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.737+0000 7f9009410640 10 monclient: monmap: 2023-10-30T12:09:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:36.738301+0000 2023-10-30T12:09:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:36.738301+0000 2023-10-30T12:09:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:36.742 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:36.742 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:36.742 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:36.743 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.737+0000 7f9009410640 1 Processor -- start 2023-10-30T12:09:36.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.737+0000 7f9009410640 1 -- start start 2023-10-30T12:09:36.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.737+0000 7f9009410640 10 monclient: init 2023-10-30T12:09:36.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-10-30T12:09:36.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f90040a4b60 0x7f90041326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient(hunting): picked mon.noname-c con 0x7f90040a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:36.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9004099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:36.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9004132be0 0x7f90040a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient(hunting): picked mon.noname-a con 0x7f9004132be0 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:36.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040a4f40 con 0x7f9004099a40 2023-10-30T12:09:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90040a3d70 con 0x7f9004132be0 2023-10-30T12:09:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90040a4150 con 0x7f90040a4b60 2023-10-30T12:09:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9009410640 10 monclient(hunting): authenticate will time out at 1421.921325s 2023-10-30T12:09:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9004132be0 0x7f90040a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90027fc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9004099a40 0x7f90040a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59052/0 (socket says 172.21.15.50:59052) 2023-10-30T12:09:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90027fc640 1 -- 172.21.15.50:0/1585812251 learned_addr learned my addr 172.21.15.50:0/1585812251 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:09:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): get_auth_request con 0x7f9004132be0 auth_method 0 2023-10-30T12:09:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9002ffd640 1 -- 172.21.15.50:0/1585812251 >> v2:172.21.15.118:3300/0 conn(0x7f90040a4b60 msgr2=0x7f90041326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9002ffd640 1 --2- 172.21.15.50:0/1585812251 >> v2:172.21.15.118:3300/0 conn(0x7f90040a4b60 0x7f90041326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient(hunting): handle_auth_done global_id 134169 payload 274 2023-10-30T12:09:36.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 1 -- 172.21.15.50:0/1585812251 >> v2:172.21.15.118:3300/0 conn(0x7f90040a4b60 msgr2=0x7f90041326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:36.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 1 --2- 172.21.15.50:0/1585812251 >> v2:172.21.15.118:3300/0 conn(0x7f90040a4b60 0x7f90041326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 1 -- 172.21.15.50:0/1585812251 >> v1:172.21.15.50:6789/0 conn(0x7f9004099a40 legacy=0x7f90040a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 1 monclient: found mon.noname-a 2023-10-30T12:09:36.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f90037fe640 1 -- 172.21.15.50:0/1585812251 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900409a440 con 0x7f9004132be0 2023-10-30T12:09:36.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9001ffb640 1 -- 172.21.15.50:0/1585812251 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978163945 0 0) 0x7f90040a4f40 con 0x7f9004099a40 2023-10-30T12:09:36.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.738+0000 7f9001ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:09:36.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f90037fe640 1 --2- 172.21.15.50:0/1585812251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f90040a2a60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90040a3820 tx=0x7f8ff402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f10ca8bf2876e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 1 -- 172.21.15.50:0/1585812251 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff402cb70 con 0x7f9004132be0 2023-10-30T12:09:36.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: dump: 2023-10-30T12:09:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.762 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.762 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 1 -- 172.21.15.50:0/1585812251 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff402ccd0 con 0x7f9004132be0 2023-10-30T12:09:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9009410640 5 monclient: authenticate success, global_id 134169 2023-10-30T12:09:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9009410640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 1 -- 172.21.15.50:0/1585812251 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4034660 con 0x7f9004132be0 2023-10-30T12:09:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9009410640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9009410640 4 set_mon_vals no callback set 2023-10-30T12:09:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:36.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.739+0000 7f9001ffb640 10 monclient: dump: 2023-10-30T12:09:36.765 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.765 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.765 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.766 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.767 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.767 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 10 monclient: shutdown 2023-10-30T12:09:36.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 -- 172.21.15.50:0/1585812251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 msgr2=0x7f90040a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 --2- 172.21.15.50:0/1585812251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f90040a2a60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f90040a3820 tx=0x7f8ff402bdc0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 -- 172.21.15.50:0/1585812251 shutdown_connections 2023-10-30T12:09:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 --2- 172.21.15.50:0/1585812251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f90040a2a60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 --2- 172.21.15.50:0/1585812251 >> v2:172.21.15.118:3300/0 conn(0x7f90040a4b60 0x7f90041326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 -- 172.21.15.50:0/1585812251 >> 172.21.15.50:0/1585812251 conn(0x7f90040a8060 msgr2=0x7f90040a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 -- 172.21.15.50:0/1585812251 shutdown_connections 2023-10-30T12:09:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 -- 172.21.15.50:0/1585812251 wait complete. 2023-10-30T12:09:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 10 monclient: build_initial_monmap 2023-10-30T12:09:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 10 monclient: monmap: 2023-10-30T12:09:36.770 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:36.770 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:36.770 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:36.770 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:36.771 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:36.771 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.771 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:36.771 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:36.772 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:36.772 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 librados: starting msgr at 2023-10-30T12:09:36.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.740+0000 7f9009410640 1 librados: starting objecter 2023-10-30T12:09:36.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 Processor -- start 2023-10-30T12:09:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 -- start start 2023-10-30T12:09:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 librados: setting wanted keys 2023-10-30T12:09:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 librados: calling monclient init 2023-10-30T12:09:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient: init 2023-10-30T12:09:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:09:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90040a4b60 0x7f900411dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): picked mon.noname-b con 0x7f90040a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:36.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f900411e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): picked mon.noname-a con 0x7f9004132be0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:36.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f900411ea40 0x7f90041ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): picked mon.noname-c con 0x7f900411ea40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:36.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004130280 con 0x7f9004132be0 2023-10-30T12:09:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900411f740 con 0x7f90040a4b60 2023-10-30T12:09:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90041314b0 con 0x7f900411ea40 2023-10-30T12:09:36.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:36.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9009410640 10 monclient(hunting): authenticate will time out at 1421.924707s 2023-10-30T12:09:36.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f900411e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9002ffd640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90040a4b60 msgr2=0x7f900411dfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:36.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f900411e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52432/0 (socket says 172.21.15.50:52432) 2023-10-30T12:09:36.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 1 -- 172.21.15.50:0/1220734189 learned_addr learned my addr 172.21.15.50:0/1220734189 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:36.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 10 monclient(hunting): get_auth_request con 0x7f9004132be0 auth_method 0 2023-10-30T12:09:36.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f900411ea40 0x7f90041ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f9002ffd640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90040a4b60 0x7f900411dfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:36.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 10 monclient(hunting): get_auth_request con 0x7f900411ea40 auth_method 0 2023-10-30T12:09:36.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:36.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.741+0000 7f90037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 10 monclient(hunting): handle_auth_done global_id 134175 payload 824 2023-10-30T12:09:36.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 1 -- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f900411ea40 msgr2=0x7f90041ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f900411ea40 0x7f90041ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 1 -- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90040a4b60 msgr2=0x7f900411dfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:36.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90040a4b60 0x7f900411dfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:36.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 10 monclient: _finish_hunting 0 2023-10-30T12:09:36.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 1 monclient: found mon.noname-a 2023-10-30T12:09:36.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:36.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 1 -- 172.21.15.50:0/1220734189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4009a70 con 0x7f9004132be0 2023-10-30T12:09:36.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90027fc640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f900411e500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00c440 tx=0x7f8fec00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 client.?.objecter ms_handle_connect 0x7f9004132be0 2023-10-30T12:09:36.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:36.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 1 -- 172.21.15.50:0/1220734189 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0074e0 con 0x7f9004132be0 2023-10-30T12:09:36.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:36.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: dump: 2023-10-30T12:09:36.788 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.788 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.788 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.788 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.789 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.789 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.789 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.789 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.790 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.790 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:09:36.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:09:36.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:36.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 1 -- 172.21.15.50:0/1220734189 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec007660 con 0x7f9004132be0 2023-10-30T12:09:36.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 5 monclient: authenticate success, global_id 134175 2023-10-30T12:09:36.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 10 monclient: _renew_subs 2023-10-30T12:09:36.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90040a18d0 con 0x7f9004132be0 2023-10-30T12:09:36.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:36.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 1 -- 172.21.15.50:0/1220734189 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec004a20 con 0x7f9004132be0 2023-10-30T12:09:36.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:36.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:36.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f8ffb7fe640 10 monclient: dump: 2023-10-30T12:09:36.793 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:36.793 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:36.793 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.793 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:36.794 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:36.794 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:36.794 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:36.794 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:36.795 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f90017fa640 4 set_mon_vals no callback set 2023-10-30T12:09:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 10 client.134175.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 10 monclient: _renew_subs 2023-10-30T12:09:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9004120f50 con 0x7f9004132be0 2023-10-30T12:09:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.742+0000 7f9009410640 1 librados: init done 2023-10-30T12:09:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f8ffb7fe640 1 -- 172.21.15.50:0/1220734189 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8fec004b80 con 0x7f9004132be0 2023-10-30T12:09:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f9009410640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f8ffb7fe640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fe40392a0 0x7f8fe403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:36.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f9009410640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f9009410640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f8ffb7fe640 1 -- 172.21.15.50:0/1220734189 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(318..318 src has 1..318) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f8fec04a950 con 0x7f9004132be0 2023-10-30T12:09:36.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f8ffb7fe640 10 client.134175.objecter ms_dispatch 0x7f9004000b90 osd_map(318..318 src has 1..318) v4 2023-10-30T12:09:36.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f8ffb7fe640 3 client.134175.objecter handle_osd_map got epochs [318,318] > 0 2023-10-30T12:09:36.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f8ffb7fe640 3 client.134175.objecter handle_osd_map decoding full epoch 318 2023-10-30T12:09:36.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004121170 con 0x7f9004132be0 2023-10-30T12:09:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f8ffb7fe640 20 client.134175.objecter dump_active .. 0 homeless 2023-10-30T12:09:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f9002ffd640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fe40392a0 0x7f8fe403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.743+0000 7f9002ffd640 10 monclient: get_auth_request con 0x7f8fe40392a0 auth_method 0 2023-10-30T12:09:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.744+0000 7f8ffb7fe640 10 client.134175.objecter ms_handle_connect 0x7f8fe40392a0 2023-10-30T12:09:36.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.744+0000 7f9002ffd640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fe40392a0 0x7f8fe403b760 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc006fd0 tx=0x7f8ffc006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:36.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.748+0000 7f8ffb7fe640 1 -- 172.21.15.50:0/1220734189 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8fec004e30 con 0x7f9004132be0 2023-10-30T12:09:36.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.748+0000 7f8ffb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:36.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.748+0000 7f8ffb7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:36.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.992+0000 7f9009410640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:36.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.992+0000 7f9009410640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:36.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.992+0000 7f9009410640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.992+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f900411ee50 con 0x7f9004132be0 2023-10-30T12:09:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.993+0000 7f8ffb7fe640 1 -- 172.21.15.50:0/1220734189 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f8fec011360 con 0x7f9004132be0 2023-10-30T12:09:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.993+0000 7f8ffb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:36.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.993+0000 7f8ffb7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:36.996 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:36.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 10 librados: watch_flush enter 2023-10-30T12:09:36.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 10 librados: watch_flush exit 2023-10-30T12:09:36.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 20 client.134175.objecter shutdown clearing up homeless session... 2023-10-30T12:09:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 10 client.134175.objecter successfully canceled tick 2023-10-30T12:09:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fe40392a0 msgr2=0x7f8fe403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fe40392a0 0x7f8fe403b760 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc006fd0 tx=0x7f8ffc006d80 comp rx=0 tx=0).stop 2023-10-30T12:09:36.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 10 monclient: shutdown 2023-10-30T12:09:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 msgr2=0x7f900411e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f900411e500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00c440 tx=0x7f8fec00c900 comp rx=0 tx=0).stop 2023-10-30T12:09:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 shutdown_connections 2023-10-30T12:09:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fe40392a0 0x7f8fe403b760 unknown :-1 s=CLOSED pgs=4078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f900411ea40 0x7f90041ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9004132be0 0x7f900411e500 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 --2- 172.21.15.50:0/1220734189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f90040a4b60 0x7f900411dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 >> 172.21.15.50:0/1220734189 conn(0x7f90040a8060 msgr2=0x7f90040a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:37.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 shutdown_connections 2023-10-30T12:09:37.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 -- 172.21.15.50:0/1220734189 wait complete. 2023-10-30T12:09:37.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:36.994+0000 7f9009410640 1 librados: shutdown 2023-10-30T12:09:40.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:40.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.121+0000 7f246a7a5640 10 monclient: get_monmap_and_config 2023-10-30T12:09:40.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.121+0000 7f246a7a5640 10 monclient: build_initial_monmap 2023-10-30T12:09:40.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.121+0000 7f246a7a5640 10 monclient: monmap: 2023-10-30T12:09:40.124 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:40.124 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:40.124 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:40.122889+0000 2023-10-30T12:09:40.124 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:40.122889+0000 2023-10-30T12:09:40.124 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:40.124 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:40.125 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:40.125 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:40.125 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:40.125 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:40.125 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:40.126 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:40.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.122+0000 7f246a7a5640 1 Processor -- start 2023-10-30T12:09:40.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.122+0000 7f246a7a5640 1 -- start start 2023-10-30T12:09:40.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.122+0000 7f246a7a5640 10 monclient: init 2023-10-30T12:09:40.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:40.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:09:40.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f24640a4b60 0x7f24641326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:40.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient(hunting): picked mon.noname-c con 0x7f24640a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:40.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2464099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:40.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2464132be0 0x7f24640a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:40.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient(hunting): picked mon.noname-a con 0x7f2464132be0 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:40.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640a4f40 con 0x7f2464099a40 2023-10-30T12:09:40.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:40.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24640a3d70 con 0x7f2464132be0 2023-10-30T12:09:40.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:40.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24640a4150 con 0x7f24640a4b60 2023-10-30T12:09:40.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f246a7a5640 10 monclient(hunting): authenticate will time out at 1425.306227s 2023-10-30T12:09:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2463fff640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f24640a4b60 msgr2=0x7f24641326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2464132be0 0x7f24640a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2463fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f24640a4b60 0x7f24641326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2464132be0 0x7f24640a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58686/0 (socket says 172.21.15.50:58686) 2023-10-30T12:09:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 -- 172.21.15.50:0/4117297770 learned_addr learned my addr 172.21.15.50:0/4117297770 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): get_auth_request con 0x7f2464132be0 auth_method 0 2023-10-30T12:09:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient(hunting): handle_auth_done global_id 134178 payload 274 2023-10-30T12:09:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 -- 172.21.15.50:0/4117297770 >> v2:172.21.15.118:3300/0 conn(0x7f24640a4b60 msgr2=0x7f24641326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:40.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 --2- 172.21.15.50:0/4117297770 >> v2:172.21.15.118:3300/0 conn(0x7f24640a4b60 0x7f24641326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 -- 172.21.15.50:0/4117297770 >> v1:172.21.15.50:6789/0 conn(0x7f2464099a40 legacy=0x7f24640a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:40.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient: _finish_hunting 0 2023-10-30T12:09:40.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 monclient: found mon.noname-a 2023-10-30T12:09:40.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:40.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:40.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.123+0000 7f2468d42640 1 -- 172.21.15.50:0/4117297770 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246409a440 con 0x7f2464132be0 2023-10-30T12:09:40.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2468d42640 1 --2- 172.21.15.50:0/4117297770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2464132be0 0x7f24640a2a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24640a3820 tx=0x7f245c02be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830d07208c6c02dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:40.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 1 -- 172.21.15.50:0/4117297770 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c02ca10 con 0x7f2464132be0 2023-10-30T12:09:40.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:40.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:40.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: dump: 2023-10-30T12:09:40.141 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:40.142 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:40.142 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.142 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.143 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:40.143 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:40.143 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:40.143 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:40.143 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:40.144 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:40.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:40.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:40.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:40.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 1 -- 172.21.15.50:0/4117297770 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c02cb70 con 0x7f2464132be0 2023-10-30T12:09:40.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f246a7a5640 5 monclient: authenticate success, global_id 134178 2023-10-30T12:09:40.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f246a7a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:40.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:40.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f246a7a5640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:40.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 1 -- 172.21.15.50:0/4117297770 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c0343f0 con 0x7f2464132be0 2023-10-30T12:09:40.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:40.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f246a7a5640 4 set_mon_vals no callback set 2023-10-30T12:09:40.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:40.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.124+0000 7f2462ffd640 10 monclient: dump: 2023-10-30T12:09:40.146 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:40.146 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:40.147 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.147 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.147 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:40.147 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:40.147 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:40.148 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:40.148 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:40.148 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:40.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 10 monclient: shutdown 2023-10-30T12:09:40.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:40.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 -- 172.21.15.50:0/4117297770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2464132be0 msgr2=0x7f24640a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:40.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 --2- 172.21.15.50:0/4117297770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2464132be0 0x7f24640a2a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24640a3820 tx=0x7f245c02be70 comp rx=0 tx=0).stop 2023-10-30T12:09:40.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 -- 172.21.15.50:0/4117297770 shutdown_connections 2023-10-30T12:09:40.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 --2- 172.21.15.50:0/4117297770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2464132be0 0x7f24640a2a60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 --2- 172.21.15.50:0/4117297770 >> v2:172.21.15.118:3300/0 conn(0x7f24640a4b60 0x7f24641326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 -- 172.21.15.50:0/4117297770 >> 172.21.15.50:0/4117297770 conn(0x7f24640a8060 msgr2=0x7f24640a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:40.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 -- 172.21.15.50:0/4117297770 shutdown_connections 2023-10-30T12:09:40.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 -- 172.21.15.50:0/4117297770 wait complete. 2023-10-30T12:09:40.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 10 monclient: build_initial_monmap 2023-10-30T12:09:40.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 10 monclient: monmap: 2023-10-30T12:09:40.151 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:40.151 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:40.151 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:40.151 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:40.152 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:40.152 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:40.152 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:40.152 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:40.153 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:40.153 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:40.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 librados: starting msgr at 2023-10-30T12:09:40.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 librados: starting objecter 2023-10-30T12:09:40.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.125+0000 7f246a7a5640 1 Processor -- start 2023-10-30T12:09:40.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 -- start start 2023-10-30T12:09:40.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 librados: setting wanted keys 2023-10-30T12:09:40.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 librados: calling monclient init 2023-10-30T12:09:40.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient: init 2023-10-30T12:09:40.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:40.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:40.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24640a4b60 0x7f24641ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:40.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): picked mon.noname-a con 0x7f24640a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:40.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2464132be0 0x7f24641cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:40.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): picked mon.noname-b con 0x7f2464132be0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:40.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f246411e4a0 0x7f246411e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:40.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): picked mon.noname-c con 0x7f246411e4a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:40.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:40.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2464131260 con 0x7f24640a4b60 2023-10-30T12:09:40.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:40.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2464120730 con 0x7f2464132be0 2023-10-30T12:09:40.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:40.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2464122930 con 0x7f246411e4a0 2023-10-30T12:09:40.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:40.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f246a7a5640 10 monclient(hunting): authenticate will time out at 1425.309700s 2023-10-30T12:09:40.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24640a4b60 0x7f24641ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:40.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24640a4b60 0x7f24641ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58694/0 (socket says 172.21.15.50:58694) 2023-10-30T12:09:40.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 1 -- 172.21.15.50:0/1684710708 learned_addr learned my addr 172.21.15.50:0/1684710708 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:40.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 10 monclient(hunting): get_auth_request con 0x7f24640a4b60 auth_method 0 2023-10-30T12:09:40.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:40.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:40.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:40.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f246411e4a0 0x7f246411e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:40.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f24637fe640 1 -- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2464132be0 msgr2=0x7f24641cae00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:40.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f24637fe640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2464132be0 0x7f24641cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:40.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 10 monclient(hunting): get_auth_request con 0x7f246411e4a0 auth_method 0 2023-10-30T12:09:40.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:40.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:40.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:40.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:40.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:40.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2463fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:40.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:40.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:40.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.126+0000 7f2468d42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:40.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 10 monclient(hunting): handle_auth_done global_id 134181 payload 824 2023-10-30T12:09:40.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 1 -- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f246411e4a0 msgr2=0x7f246411e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:40.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f246411e4a0 0x7f246411e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 1 -- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2464132be0 msgr2=0x7f24641cae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:40.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2464132be0 0x7f24641cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 10 monclient: _finish_hunting 0 2023-10-30T12:09:40.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 1 monclient: found mon.noname-a 2023-10-30T12:09:40.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:40.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:40.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 1 -- 172.21.15.50:0/1684710708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c009b30 con 0x7f24640a4b60 2023-10-30T12:09:40.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2463fff640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24640a4b60 0x7f24641ca8c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f245400b580 tx=0x7f245400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:40.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 client.?.objecter ms_handle_connect 0x7f24640a4b60 2023-10-30T12:09:40.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:40.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 1 -- 172.21.15.50:0/1684710708 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454003dc0 con 0x7f24640a4b60 2023-10-30T12:09:40.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:40.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:40.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: dump: 2023-10-30T12:09:40.166 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:40.166 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:40.167 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.167 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.167 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:40.167 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:40.167 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:40.168 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: _finish_auth 0 2023-10-30T12:09:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:09:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 1 -- 172.21.15.50:0/1684710708 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454005070 con 0x7f24640a4b60 2023-10-30T12:09:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 1 -- 172.21.15.50:0/1684710708 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454005420 con 0x7f24640a4b60 2023-10-30T12:09:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 5 monclient: authenticate success, global_id 134181 2023-10-30T12:09:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:40.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:40.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f2460ff9640 10 monclient: dump: 2023-10-30T12:09:40.170 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:40.170 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:40.170 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:40.172 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 10 monclient: _renew_subs 2023-10-30T12:09:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f24627fc640 4 set_mon_vals no callback set 2023-10-30T12:09:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:40.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2464120d80 con 0x7f24640a4b60 2023-10-30T12:09:40.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 10 client.134181.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:40.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 10 monclient: _renew_subs 2023-10-30T12:09:40.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:40.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2464122230 con 0x7f24640a4b60 2023-10-30T12:09:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.127+0000 7f246a7a5640 1 librados: init done 2023-10-30T12:09:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f2460ff9640 1 -- 172.21.15.50:0/1684710708 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2454005580 con 0x7f24640a4b60 2023-10-30T12:09:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f246a7a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f246a7a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f246a7a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2464121db0 con 0x7f24640a4b60 2023-10-30T12:09:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f2460ff9640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24500392a0 0x7f245003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f2460ff9640 1 -- 172.21.15.50:0/1684710708 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(318..318 src has 1..318) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f245404ac70 con 0x7f24640a4b60 2023-10-30T12:09:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f2460ff9640 10 client.134181.objecter ms_dispatch 0x7f2464000b90 osd_map(318..318 src has 1..318) v4 2023-10-30T12:09:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f2460ff9640 3 client.134181.objecter handle_osd_map got epochs [318,318] > 0 2023-10-30T12:09:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f2460ff9640 3 client.134181.objecter handle_osd_map decoding full epoch 318 2023-10-30T12:09:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.128+0000 7f2460ff9640 20 client.134181.objecter dump_active .. 0 homeless 2023-10-30T12:09:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.129+0000 7f24637fe640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24500392a0 0x7f245003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.129+0000 7f24637fe640 10 monclient: get_auth_request con 0x7f24500392a0 auth_method 0 2023-10-30T12:09:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.129+0000 7f2460ff9640 10 client.134181.objecter ms_handle_connect 0x7f24500392a0 2023-10-30T12:09:40.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.129+0000 7f24637fe640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24500392a0 0x7f245003b760 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 crypto rx=0x7f246411fd40 tx=0x7f24580079d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:40.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.133+0000 7f2460ff9640 1 -- 172.21.15.50:0/1684710708 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2464121db0 con 0x7f24640a4b60 2023-10-30T12:09:40.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.133+0000 7f2460ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:40.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.133+0000 7f2460ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.377+0000 7f246a7a5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.377+0000 7f246a7a5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.377+0000 7f246a7a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.377+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2464121db0 con 0x7f24640a4b60 2023-10-30T12:09:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.377+0000 7f2460ff9640 1 -- 172.21.15.50:0/1684710708 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f2464121db0 con 0x7f24640a4b60 2023-10-30T12:09:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.377+0000 7f2460ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.377+0000 7f2460ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:40.381 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 10 librados: watch_flush enter 2023-10-30T12:09:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 10 librados: watch_flush exit 2023-10-30T12:09:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 20 client.134181.objecter shutdown clearing up homeless session... 2023-10-30T12:09:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 10 client.134181.objecter successfully canceled tick 2023-10-30T12:09:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24500392a0 msgr2=0x7f245003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24500392a0 0x7f245003b760 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 crypto rx=0x7f246411fd40 tx=0x7f24580079d0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 10 monclient: shutdown 2023-10-30T12:09:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24640a4b60 msgr2=0x7f24641ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24640a4b60 0x7f24641ca8c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f245400b580 tx=0x7f245400ba40 comp rx=0 tx=0).stop 2023-10-30T12:09:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 shutdown_connections 2023-10-30T12:09:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24500392a0 0x7f245003b760 unknown :-1 s=CLOSED pgs=4079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f246411e4a0 0x7f246411e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2464132be0 0x7f24641cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 --2- 172.21.15.50:0/1684710708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24640a4b60 0x7f24641ca8c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 >> 172.21.15.50:0/1684710708 conn(0x7f24640a8060 msgr2=0x7f2464134340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 shutdown_connections 2023-10-30T12:09:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 -- 172.21.15.50:0/1684710708 wait complete. 2023-10-30T12:09:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:40.379+0000 7f246a7a5640 1 librados: shutdown 2023-10-30T12:09:43.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:43.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.506+0000 7f31a80a2640 10 monclient: get_monmap_and_config 2023-10-30T12:09:43.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.506+0000 7f31a80a2640 10 monclient: build_initial_monmap 2023-10-30T12:09:43.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.506+0000 7f31a80a2640 10 monclient: monmap: 2023-10-30T12:09:43.508 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:43.509 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:43.509 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:43.507904+0000 2023-10-30T12:09:43.509 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:43.507904+0000 2023-10-30T12:09:43.509 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:43.509 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:43.510 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:43.510 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:43.510 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:43.510 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:43.510 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:43.511 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:43.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.507+0000 7f31a80a2640 1 Processor -- start 2023-10-30T12:09:43.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.507+0000 7f31a80a2640 1 -- start start 2023-10-30T12:09:43.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.507+0000 7f31a80a2640 10 monclient: init 2023-10-30T12:09:43.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:43.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T12:09:43.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a009f5b0 0x7f31a0099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:43.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient(hunting): picked mon.noname-b con 0x7f31a009f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:43.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f31a0096080 0x7f31a0096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:43.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient(hunting): picked mon.noname-a con 0x7f31a0096080 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:43.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31a00969a0 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:43.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a0098810 con 0x7f31a00969a0 2023-10-30T12:09:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a006aef0 con 0x7f31a0096080 2023-10-30T12:09:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a009a1a0 con 0x7f31a009f5b0 2023-10-30T12:09:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a80a2640 10 monclient(hunting): authenticate will time out at 1428.691279s 2023-10-30T12:09:43.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f31a0096080 0x7f31a0096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:43.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a009f5b0 0x7f31a0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:43.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a663f640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f31a00969a0 legacy=0x7f31a009cb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:09:43.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f31a0096080 0x7f31a0096460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58704/0 (socket says 172.21.15.50:58704) 2023-10-30T12:09:43.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a009f5b0 0x7f31a0099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55366/0 (socket says 172.21.15.50:55366) 2023-10-30T12:09:43.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 -- 172.21.15.50:0/3549054367 learned_addr learned my addr 172.21.15.50:0/3549054367 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:43.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): get_auth_request con 0x7f31a0096080 auth_method 0 2023-10-30T12:09:43.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:43.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:43.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:43.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 10 monclient(hunting): get_auth_request con 0x7f31a009f5b0 auth_method 0 2023-10-30T12:09:43.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:43.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:43.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:43.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:43.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:43.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:43.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:43.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:43.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:43.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient(hunting): handle_auth_done global_id 134184 payload 274 2023-10-30T12:09:43.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 -- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a009f5b0 msgr2=0x7f31a0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:43.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 --2- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a009f5b0 0x7f31a0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 -- 172.21.15.50:0/3549054367 >> v1:172.21.15.118:6789/0 conn(0x7f31a00969a0 legacy=0x7f31a009cb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:43.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient: _finish_hunting 0 2023-10-30T12:09:43.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 monclient: found mon.noname-a 2023-10-30T12:09:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a563d640 1 -- 172.21.15.50:0/3549054367 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a0098af0 con 0x7f31a0096080 2023-10-30T12:09:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.508+0000 7f31a5e3e640 1 --2- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a009f5b0 0x7f31a0099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a563d640 1 --2- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a0096460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f31900098a0 tx=0x7f319002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59eae2f8b00c69cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 1 -- 172.21.15.50:0/3549054367 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190031740 con 0x7f31a0096080 2023-10-30T12:09:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: dump: 2023-10-30T12:09:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:43.528 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:43.529 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:43.529 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:43.529 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:43.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: _finish_auth 0 2023-10-30T12:09:43.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: _check_auth_tickets 2023-10-30T12:09:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a80a2640 5 monclient: authenticate success, global_id 134184 2023-10-30T12:09:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 1 -- 172.21.15.50:0/3549054367 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31900318a0 con 0x7f31a0096080 2023-10-30T12:09:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 1 -- 172.21.15.50:0/3549054367 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190031c30 con 0x7f31a0096080 2023-10-30T12:09:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a80a2640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a80a2640 4 set_mon_vals no callback set 2023-10-30T12:09:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.509+0000 7f31a4e3c640 10 monclient: dump: 2023-10-30T12:09:43.531 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:43.532 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:43.532 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.532 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.532 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:43.532 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:43.533 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:43.533 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:43.533 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:43.533 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:43.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 10 monclient: shutdown 2023-10-30T12:09:43.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:43.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 -- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 msgr2=0x7f31a0096460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:43.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 --2- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a0096460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f31900098a0 tx=0x7f319002b7e0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 -- 172.21.15.50:0/3549054367 shutdown_connections 2023-10-30T12:09:43.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 --2- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a0096460 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 --2- 172.21.15.50:0/3549054367 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a009f5b0 0x7f31a0099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 -- 172.21.15.50:0/3549054367 >> 172.21.15.50:0/3549054367 conn(0x7f31a00a05c0 msgr2=0x7f31a009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:43.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 -- 172.21.15.50:0/3549054367 shutdown_connections 2023-10-30T12:09:43.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 1 -- 172.21.15.50:0/3549054367 wait complete. 2023-10-30T12:09:43.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 10 monclient: build_initial_monmap 2023-10-30T12:09:43.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.510+0000 7f31a80a2640 10 monclient: monmap: 2023-10-30T12:09:43.538 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:43.538 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:43.538 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:43.538 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:43.538 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:43.539 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:43.539 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:43.539 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:43.539 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:43.539 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 librados: starting msgr at 2023-10-30T12:09:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 librados: starting objecter 2023-10-30T12:09:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 Processor -- start 2023-10-30T12:09:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 -- start start 2023-10-30T12:09:43.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 librados: setting wanted keys 2023-10-30T12:09:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 librados: calling monclient init 2023-10-30T12:09:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient: init 2023-10-30T12:09:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:09:43.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a01c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:43.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): picked mon.noname-a con 0x7f31a0096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:43.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a00969a0 0x7f31a01b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:43.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): picked mon.noname-c con 0x7f31a00969a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:43.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31a009f5b0 0x7f31a01b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:43.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): picked mon.noname-b con 0x7f31a009f5b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:43.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:43.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a01ae100 con 0x7f31a0096080 2023-10-30T12:09:43.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:43.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a01b61b0 con 0x7f31a009f5b0 2023-10-30T12:09:43.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0072010 con 0x7f31a00969a0 2023-10-30T12:09:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a80a2640 10 monclient(hunting): authenticate will time out at 1428.694773s 2023-10-30T12:09:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a01c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a563d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a00969a0 0x7f31a01b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:43.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a01c8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58708/0 (socket says 172.21.15.50:58708) 2023-10-30T12:09:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 1 -- 172.21.15.50:0/805391300 learned_addr learned my addr 172.21.15.50:0/805391300 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 10 monclient(hunting): get_auth_request con 0x7f31a0096080 auth_method 0 2023-10-30T12:09:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:43.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a563d640 10 monclient(hunting): get_auth_request con 0x7f31a00969a0 auth_method 0 2023-10-30T12:09:43.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a563d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:43.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a563d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:43.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a563d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:43.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a663f640 1 -- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31a009f5b0 msgr2=0x7f31a01b5c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a663f640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31a009f5b0 0x7f31a01b5c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.511+0000 7f31a5e3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:43.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a563d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:43.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a563d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:43.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a563d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:43.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 10 monclient(hunting): handle_auth_done global_id 134187 payload 824 2023-10-30T12:09:43.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 1 -- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a00969a0 msgr2=0x7f31a01b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:43.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a00969a0 0x7f31a01b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 1 -- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31a009f5b0 msgr2=0x7f31a01b5c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31a009f5b0 0x7f31a01b5c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 10 monclient: _finish_hunting 0 2023-10-30T12:09:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 1 monclient: found mon.noname-a 2023-10-30T12:09:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 1 -- 172.21.15.50:0/805391300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c0093a0 con 0x7f31a0096080 2023-10-30T12:09:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a563d640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a00969a0 0x7f31a01b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a5e3e640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a01c8060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f319c00ebe0 tx=0x7f319c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 client.?.objecter ms_handle_connect 0x7f31a0096080 2023-10-30T12:09:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 1 -- 172.21.15.50:0/805391300 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c009750 con 0x7f31a0096080 2023-10-30T12:09:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: dump: 2023-10-30T12:09:43.551 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:43.552 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:43.553 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 1 -- 172.21.15.50:0/805391300 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c00cc30 con 0x7f31a0096080 2023-10-30T12:09:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 5 monclient: authenticate success, global_id 134187 2023-10-30T12:09:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 10 monclient: _renew_subs 2023-10-30T12:09:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31900094d0 con 0x7f31a0096080 2023-10-30T12:09:43.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:43.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 1 -- 172.21.15.50:0/805391300 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c00eee0 con 0x7f31a0096080 2023-10-30T12:09:43.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:43.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:43.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31967fc640 10 monclient: dump: 2023-10-30T12:09:43.556 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:43.556 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:43.556 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.556 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:43.556 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:43.557 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:43.557 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:43.557 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:43.557 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:43.557 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:43.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f3197fff640 4 set_mon_vals no callback set 2023-10-30T12:09:43.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 10 client.134187.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:43.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 10 monclient: _renew_subs 2023-10-30T12:09:43.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:43.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a00713a0 con 0x7f31a0096080 2023-10-30T12:09:43.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.512+0000 7f31a80a2640 1 librados: init done 2023-10-30T12:09:43.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31967fc640 1 -- 172.21.15.50:0/805391300 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f319c005990 con 0x7f31a0096080 2023-10-30T12:09:43.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31a80a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:43.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31a80a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:43.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31a80a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:43.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31967fc640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f31840392a0 0x7f318403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:43.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a004e160 con 0x7f31a0096080 2023-10-30T12:09:43.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31967fc640 1 -- 172.21.15.50:0/805391300 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(318..318 src has 1..318) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f319c04b290 con 0x7f31a0096080 2023-10-30T12:09:43.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31967fc640 10 client.134187.objecter ms_dispatch 0x7f31a0000b90 osd_map(318..318 src has 1..318) v4 2023-10-30T12:09:43.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31967fc640 3 client.134187.objecter handle_osd_map got epochs [318,318] > 0 2023-10-30T12:09:43.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31967fc640 3 client.134187.objecter handle_osd_map decoding full epoch 318 2023-10-30T12:09:43.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31967fc640 20 client.134187.objecter dump_active .. 0 homeless 2023-10-30T12:09:43.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31a563d640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f31840392a0 0x7f318403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:43.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.513+0000 7f31a563d640 10 monclient: get_auth_request con 0x7f31840392a0 auth_method 0 2023-10-30T12:09:43.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.514+0000 7f31a563d640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f31840392a0 0x7f318403b760 secure :-1 s=READY pgs=4080 cs=0 l=1 rev1=1 crypto rx=0x7f3190035000 tx=0x7f3190008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:43.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.514+0000 7f31967fc640 10 client.134187.objecter ms_handle_connect 0x7f31840392a0 2023-10-30T12:09:43.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.518+0000 7f31967fc640 1 -- 172.21.15.50:0/805391300 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f319c022710 con 0x7f31a0096080 2023-10-30T12:09:43.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.518+0000 7f31967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:43.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.518+0000 7f31967fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:43.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.763+0000 7f31a80a2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:43.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.764+0000 7f31a80a2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:43.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.764+0000 7f31a80a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.764+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f31a00745c0 con 0x7f31a0096080 2023-10-30T12:09:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.764+0000 7f31967fc640 1 -- 172.21.15.50:0/805391300 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f319c011350 con 0x7f31a0096080 2023-10-30T12:09:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.764+0000 7f31967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.764+0000 7f31967fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:43.767 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:43.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.765+0000 7f31a80a2640 10 librados: watch_flush enter 2023-10-30T12:09:43.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 10 librados: watch_flush exit 2023-10-30T12:09:43.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 20 client.134187.objecter shutdown clearing up homeless session... 2023-10-30T12:09:43.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 10 client.134187.objecter successfully canceled tick 2023-10-30T12:09:43.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f31840392a0 msgr2=0x7f318403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:43.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f31840392a0 0x7f318403b760 secure :-1 s=READY pgs=4080 cs=0 l=1 rev1=1 crypto rx=0x7f3190035000 tx=0x7f3190008030 comp rx=0 tx=0).stop 2023-10-30T12:09:43.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 10 monclient: shutdown 2023-10-30T12:09:43.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:43.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 msgr2=0x7f31a01c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:43.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a01c8060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f319c00ebe0 tx=0x7f319c00c520 comp rx=0 tx=0).stop 2023-10-30T12:09:43.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 shutdown_connections 2023-10-30T12:09:43.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f31840392a0 0x7f318403b760 unknown :-1 s=CLOSED pgs=4080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31a009f5b0 0x7f31a01b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31a00969a0 0x7f31a01b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 --2- 172.21.15.50:0/805391300 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31a0096080 0x7f31a01c8060 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:43.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 >> 172.21.15.50:0/805391300 conn(0x7f31a00a05c0 msgr2=0x7f31a01b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:43.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 shutdown_connections 2023-10-30T12:09:43.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 -- 172.21.15.50:0/805391300 wait complete. 2023-10-30T12:09:43.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:43.766+0000 7f31a80a2640 1 librados: shutdown 2023-10-30T12:09:46.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:09:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.892+0000 7fa76c3e9640 10 monclient: get_monmap_and_config 2023-10-30T12:09:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.892+0000 7fa76c3e9640 10 monclient: build_initial_monmap 2023-10-30T12:09:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.892+0000 7fa76c3e9640 10 monclient: monmap: 2023-10-30T12:09:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:46.894010+0000 2023-10-30T12:09:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:46.894010+0000 2023-10-30T12:09:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:46.896 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:46.897 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:46.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.893+0000 7fa76c3e9640 1 Processor -- start 2023-10-30T12:09:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.893+0000 7fa76c3e9640 1 -- start start 2023-10-30T12:09:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.893+0000 7fa76c3e9640 10 monclient: init 2023-10-30T12:09:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:09:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 0x7fa764132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient(hunting): picked mon.noname-c con 0x7fa764131b20 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa764071bc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient(hunting): picked mon.noname-b con 0x7fa764071fc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:46.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76409a9b0 con 0x7fa764071bc0 2023-10-30T12:09:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa764131f00 con 0x7fa764131b20 2023-10-30T12:09:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa764072400 con 0x7fa764071fc0 2023-10-30T12:09:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76c3e9640 10 monclient(hunting): authenticate will time out at 1432.077264s 2023-10-30T12:09:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa769984640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7fa764071bc0 legacy=0x7fa764097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:09:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a185640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 msgr2=0x7fa764132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a185640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 0x7fa764132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55392/0 (socket says 172.21.15.50:55392) 2023-10-30T12:09:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 learned_addr learned my addr 172.21.15.50:0/1698410105 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 10 monclient(hunting): get_auth_request con 0x7fa764071fc0 auth_method 0 2023-10-30T12:09:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:46.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:46.894+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 10 monclient(hunting): get_auth_request con 0x7fa764071fc0 auth_method 0 2023-10-30T12:09:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a185640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 msgr2=0x7fa764132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v1:172.21.15.118:6789/0 conn(0x7fa764071bc0 legacy=0x7fa764097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:09:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a185640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 0x7fa764132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:09:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.095+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:47.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a185640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 msgr2=0x7fa764132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:47.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v1:172.21.15.118:6789/0 conn(0x7fa764071bc0 legacy=0x7fa764097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:09:47.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a185640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 0x7fa764132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:47.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:47.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 10 monclient(hunting): get_auth_request con 0x7fa764071fc0 auth_method 0 2023-10-30T12:09:47.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:47.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:47.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:47.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:47.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:47.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:47.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:47.496+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:48.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v1:172.21.15.118:6789/0 conn(0x7fa764071bc0 legacy=0x7fa764097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:09:48.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a185640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 msgr2=0x7fa764132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:48.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a185640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 0x7fa764132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:48.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:48.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 10 monclient(hunting): get_auth_request con 0x7fa764071fc0 auth_method 0 2023-10-30T12:09:48.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:48.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:48.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:48.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:48.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:48.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:48.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:48.297+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:49.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient(hunting): tick 2023-10-30T12:09:49.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:09:49.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:09:49.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:09:49.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 msgr2=0x7fa764097b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:49.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa764071fc0 0x7fa764097b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:49.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 msgr2=0x7fa764132270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:49.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.118:3300/0 conn(0x7fa764131b20 0x7fa764132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:49.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 -- 172.21.15.50:0/1698410105 >> v1:172.21.15.118:6789/0 conn(0x7fa764071bc0 legacy=0x7fa764097410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:49.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient: _add_conns ranks=[0,3] 2023-10-30T12:09:49.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7480058d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:49.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:49.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa748009600 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:49.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 -- 172.21.15.50:0/1698410105 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76409d920 con 0x7fa7480058d0 2023-10-30T12:09:49.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:49.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 1 -- 172.21.15.50:0/1698410105 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa764131f00 con 0x7fa748009600 2023-10-30T12:09:49.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa757fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:49.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:49.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 10 monclient(hunting): get_auth_request con 0x7fa748009600 auth_method 0 2023-10-30T12:09:49.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:49.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:49.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:49.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:49.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:49.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:49.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:49.894+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:50.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:50.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 10 monclient(hunting): get_auth_request con 0x7fa748009600 auth_method 0 2023-10-30T12:09:50.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:50.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:50.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:50.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:50.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:50.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:50.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.096+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:09:50.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 10 monclient(hunting): get_auth_request con 0x7fa748009600 auth_method 0 2023-10-30T12:09:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:50.497+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:09:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 10 monclient(hunting): get_auth_request con 0x7fa748009600 auth_method 0 2023-10-30T12:09:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:09:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:09:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:09:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.298+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:09:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865139731 0 0) 0x7fa76409d920 con 0x7fa7480058d0 2023-10-30T12:09:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): _negotiate 2023-10-30T12:09:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): my global_id is 134190 2023-10-30T12:09:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa750003520 con 0x7fa7480058d0 2023-10-30T12:09:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039601244 0 0) 0x7fa750003520 con 0x7fa7480058d0 2023-10-30T12:09:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 msgr2=0x7fa74800bac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 --2- 172.21.15.50:0/1698410105 >> v2:172.21.15.50:3300/0 conn(0x7fa748009600 0x7fa74800bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: _finish_hunting 0 2023-10-30T12:09:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:09:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:09:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76409a9b0 con 0x7fa7480058d0 2023-10-30T12:09:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: _finish_auth 0 2023-10-30T12:09:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: _check_auth_tickets 2023-10-30T12:09:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fa760004130 con 0x7fa7480058d0 2023-10-30T12:09:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa76c3e9640 5 monclient: authenticate success, global_id 134190 2023-10-30T12:09:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa76c3e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:09:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: dump: 2023-10-30T12:09:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:51.618 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:09:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 >> v1:172.21.15.50:6789/0 conn(0x7fa7480058d0 legacy=0x7fa748007d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa76a986640 1 -- 172.21.15.50:0/1698410105 reap_dead start 2023-10-30T12:09:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:09:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa750003970 0x7fa750003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): picked mon.c con 0x7fa750003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7500047c0 0x7fa750004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): picked mon.b con 0x7fa7500047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500051d0 0x7fa75000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): picked mon.a con 0x7fa7500051d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.608+0000 7fa769183640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa764131f00 con 0x7fa7500051d0 2023-10-30T12:09:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769183640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75000c100 con 0x7fa7500047c0 2023-10-30T12:09:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769183640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75000c2c0 con 0x7fa750003970 2023-10-30T12:09:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769183640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769183640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:09:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76c3e9640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769984640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7500047c0 msgr2=0x7fa750004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa750003970 0x7fa750003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa769984640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7500047c0 0x7fa750004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500051d0 0x7fa75000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): get_auth_request con 0x7fa750003970 auth_method 0 2023-10-30T12:09:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 10 monclient(hunting): get_auth_request con 0x7fa7500051d0 auth_method 0 2023-10-30T12:09:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:09:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:09:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:09:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient(hunting): handle_auth_done global_id 134190 payload 293 2023-10-30T12:09:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7500047c0 msgr2=0x7fa750004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7500047c0 0x7fa750004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500051d0 msgr2=0x7fa75000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500051d0 0x7fa75000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient: _finish_hunting 0 2023-10-30T12:09:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 1 monclient: found mon.c 2023-10-30T12:09:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 1 -- 172.21.15.50:0/1698410105 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa750003520 con 0x7fa750003970 2023-10-30T12:09:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient: _finish_auth 0 2023-10-30T12:09:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 10 monclient: _check_auth_tickets 2023-10-30T12:09:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a185640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.609+0000 7fa76a986640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500051d0 0x7fa75000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.610+0000 7fa76a185640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa750003970 0x7fa750003d70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa7600038b0 tx=0x7fa7500057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e25546e40ff0e434 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76002cd00 con 0x7fa750003970 2023-10-30T12:09:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 10 monclient: dump: 2023-10-30T12:09:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:51.631 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76002ce60 con 0x7fa750003970 2023-10-30T12:09:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa76c3e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa76c3e9640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 1 -- 172.21.15.50:0/1698410105 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600356b0 con 0x7fa750003970 2023-10-30T12:09:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa76c3e9640 4 set_mon_vals no callback set 2023-10-30T12:09:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.611+0000 7fa769183640 10 monclient: dump: 2023-10-30T12:09:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:51.635 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 10 monclient: shutdown 2023-10-30T12:09:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 -- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa750003970 msgr2=0x7fa750003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa750003970 0x7fa750003d70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa7600038b0 tx=0x7fa7500057f0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 -- 172.21.15.50:0/1698410105 shutdown_connections 2023-10-30T12:09:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500051d0 0x7fa75000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa7500047c0 0x7fa750004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/1698410105 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa750003970 0x7fa750003d70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 -- 172.21.15.50:0/1698410105 >> 172.21.15.50:0/1698410105 conn(0x7fa7640a05c0 msgr2=0x7fa76409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 -- 172.21.15.50:0/1698410105 shutdown_connections 2023-10-30T12:09:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.612+0000 7fa76c3e9640 1 -- 172.21.15.50:0/1698410105 wait complete. 2023-10-30T12:09:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 10 monclient: build_initial_monmap 2023-10-30T12:09:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 10 monclient: monmap: 2023-10-30T12:09:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:51.644 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 1 librados: starting msgr at 2023-10-30T12:09:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 1 librados: starting objecter 2023-10-30T12:09:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 1 Processor -- start 2023-10-30T12:09:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 1 -- start start 2023-10-30T12:09:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 1 librados: setting wanted keys 2023-10-30T12:09:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 1 librados: calling monclient init 2023-10-30T12:09:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 10 monclient: init 2023-10-30T12:09:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.613+0000 7fa76c3e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:09:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa750003970 0x7fa7641c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): picked mon.noname-b con 0x7fa750003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500047c0 0x7fa7641309a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): picked mon.noname-a con 0x7fa7500047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7500051d0 0x7fa764130ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): picked mon.noname-c con 0x7fa7500051d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7640993c0 con 0x7fa7500047c0 2023-10-30T12:09:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7641c63f0 con 0x7fa750003970 2023-10-30T12:09:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa764098210 con 0x7fa7500051d0 2023-10-30T12:09:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76c3e9640 10 monclient(hunting): authenticate will time out at 1436.797101s 2023-10-30T12:09:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500047c0 0x7fa7641309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7500051d0 0x7fa764130ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a185640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa750003970 msgr2=0x7fa7641c5710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7500051d0 0x7fa764130ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56254/0 (socket says 172.21.15.50:56254) 2023-10-30T12:09:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a185640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa750003970 0x7fa7641c5710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 -- 172.21.15.50:0/2210513805 learned_addr learned my addr 172.21.15.50:0/2210513805 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 10 monclient(hunting): get_auth_request con 0x7fa7500047c0 auth_method 0 2023-10-30T12:09:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): get_auth_request con 0x7fa7500051d0 auth_method 0 2023-10-30T12:09:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient(hunting): handle_auth_done global_id 114404 payload 824 2023-10-30T12:09:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 -- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa750003970 msgr2=0x7fa7641c5710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa750003970 0x7fa7641c5710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 -- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500047c0 msgr2=0x7fa7641309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:51.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500047c0 0x7fa7641309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient: _finish_hunting 0 2023-10-30T12:09:51.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 monclient: found mon.noname-c 2023-10-30T12:09:51.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:51.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 -- 172.21.15.50:0/2210513805 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640716f0 con 0x7fa7500051d0 2023-10-30T12:09:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa769984640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500047c0 0x7fa7641309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa76a986640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7500051d0 0x7fa764130ee0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa748009570 tx=0x7fa75c008ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa756ffd640 10 client.?.objecter ms_handle_connect 0x7fa7500051d0 2023-10-30T12:09:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.614+0000 7fa756ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 1 -- 172.21.15.50:0/2210513805 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c003e60 con 0x7fa7500051d0 2023-10-30T12:09:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: dump: 2023-10-30T12:09:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:51.658 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: _finish_auth 0 2023-10-30T12:09:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 1 -- 172.21.15.50:0/2210513805 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c010070 con 0x7fa7500051d0 2023-10-30T12:09:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 1 -- 172.21.15.50:0/2210513805 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c015b50 con 0x7fa7500051d0 2023-10-30T12:09:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa756ffd640 10 monclient: dump: 2023-10-30T12:09:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:51.661 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 5 monclient: authenticate success, global_id 114404 2023-10-30T12:09:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 10 monclient: _renew_subs 2023-10-30T12:09:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa757fff640 4 set_mon_vals no callback set 2023-10-30T12:09:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa764129b80 con 0x7fa7500051d0 2023-10-30T12:09:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 10 client.114404.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 10 monclient: _renew_subs 2023-10-30T12:09:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa764131420 con 0x7fa7500051d0 2023-10-30T12:09:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.615+0000 7fa76c3e9640 1 librados: init done 2023-10-30T12:09:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa756ffd640 1 -- 172.21.15.50:0/2210513805 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa75c015cb0 con 0x7fa7500051d0 2023-10-30T12:09:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa756ffd640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa748039a20 0x7fa74803bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:51.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa76c3e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:51.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa756ffd640 1 -- 172.21.15.50:0/2210513805 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(318..318 src has 1..318) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fa75c04bdf0 con 0x7fa7500051d0 2023-10-30T12:09:51.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa756ffd640 10 client.114404.objecter ms_dispatch 0x7fa764000b90 osd_map(318..318 src has 1..318) v4 2023-10-30T12:09:51.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa76c3e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:51.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa756ffd640 3 client.114404.objecter handle_osd_map got epochs [318,318] > 0 2023-10-30T12:09:51.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa756ffd640 3 client.114404.objecter handle_osd_map decoding full epoch 318 2023-10-30T12:09:51.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa76c3e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:51.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76404e160 con 0x7fa7500051d0 2023-10-30T12:09:51.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa756ffd640 20 client.114404.objecter dump_active .. 0 homeless 2023-10-30T12:09:51.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa76a185640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa748039a20 0x7fa74803bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:51.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.616+0000 7fa76a185640 10 monclient: get_auth_request con 0x7fa748039a20 auth_method 0 2023-10-30T12:09:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.617+0000 7fa76a185640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa748039a20 0x7fa74803bee0 secure :-1 s=READY pgs=4081 cs=0 l=1 rev1=1 crypto rx=0x7fa760034000 tx=0x7fa760007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.617+0000 7fa756ffd640 10 client.114404.objecter ms_handle_connect 0x7fa748039a20 2023-10-30T12:09:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.621+0000 7fa756ffd640 1 -- 172.21.15.50:0/2210513805 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa75c004550 con 0x7fa7500051d0 2023-10-30T12:09:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.621+0000 7fa756ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:51.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.621+0000 7fa756ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.869+0000 7fa76c3e9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:09:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.869+0000 7fa76c3e9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.869+0000 7fa76c3e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.869+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa7640745c0 con 0x7fa7500051d0 2023-10-30T12:09:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.870+0000 7fa756ffd640 1 -- 172.21.15.50:0/2210513805 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fa75c02f030 con 0x7fa7500051d0 2023-10-30T12:09:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.870+0000 7fa756ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:09:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.870+0000 7fa756ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:51.873 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:09:51.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 10 librados: watch_flush enter 2023-10-30T12:09:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 10 librados: watch_flush exit 2023-10-30T12:09:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 20 client.114404.objecter shutdown clearing up homeless session... 2023-10-30T12:09:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 10 client.114404.objecter successfully canceled tick 2023-10-30T12:09:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa748039a20 msgr2=0x7fa74803bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa748039a20 0x7fa74803bee0 secure :-1 s=READY pgs=4081 cs=0 l=1 rev1=1 crypto rx=0x7fa760034000 tx=0x7fa760007000 comp rx=0 tx=0).stop 2023-10-30T12:09:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 10 monclient: shutdown 2023-10-30T12:09:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:51.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7500051d0 msgr2=0x7fa764130ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:51.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7500051d0 0x7fa764130ee0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa748009570 tx=0x7fa75c008ec0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 shutdown_connections 2023-10-30T12:09:51.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa748039a20 0x7fa74803bee0 unknown :-1 s=CLOSED pgs=4081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7500051d0 0x7fa764130ee0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7500047c0 0x7fa7641309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 --2- 172.21.15.50:0/2210513805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa750003970 0x7fa7641c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:51.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 >> 172.21.15.50:0/2210513805 conn(0x7fa7640a05c0 msgr2=0x7fa7641b9080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:51.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 shutdown_connections 2023-10-30T12:09:51.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 -- 172.21.15.50:0/2210513805 wait complete. 2023-10-30T12:09:51.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.872+0000 7fa76c3e9640 1 librados: shutdown 2023-10-30T12:09:51.884 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:09:51.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:09:51.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:09:51.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.988+0000 7f7156439640 10 monclient: get_monmap_and_config 2023-10-30T12:09:51.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.988+0000 7f7156439640 10 monclient: build_initial_monmap 2023-10-30T12:09:51.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.988+0000 7f7156439640 10 monclient: monmap: 2023-10-30T12:09:51.991 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:51.991 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:51.991 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:51.989733+0000 2023-10-30T12:09:51.991 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:51.989733+0000 2023-10-30T12:09:51.992 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:51.992 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:51.992 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:51.992 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:51.992 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:51.992 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:51.993 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:51.993 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 1 Processor -- start 2023-10-30T12:09:52.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 1 -- start start 2023-10-30T12:09:52.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient: init 2023-10-30T12:09:52.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:52.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:09:52.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f715009bb70 addr v1:172.21.15.50:6789/0 2023-10-30T12:09:52.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f715009d3f0 0x7f7150137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient(hunting): picked mon.noname-a con 0x7f715009d3f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:09:52.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009c660 0x7f7150139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient(hunting): picked mon.noname-b con 0x7f715009c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:52.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7150098170 con 0x7f715009bb70 2023-10-30T12:09:52.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715009cf60 con 0x7f715009d3f0 2023-10-30T12:09:52.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7150097b20 con 0x7f715009c660 2023-10-30T12:09:52.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:52.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.989+0000 7f7156439640 10 monclient(hunting): authenticate will time out at 1437.172934s 2023-10-30T12:09:52.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009c660 0x7f7150139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f715009d3f0 0x7f7150137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009c660 0x7f7150139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56268/0 (socket says 172.21.15.50:56268) 2023-10-30T12:09:52.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): get_auth_request con 0x7f715009d3f0 auth_method 0 2023-10-30T12:09:52.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 1 -- 172.21.15.50:0/1720049854 learned_addr learned my addr 172.21.15.50:0/1720049854 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:52.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 10 monclient(hunting): get_auth_request con 0x7f715009c660 auth_method 0 2023-10-30T12:09:52.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f71549d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient(hunting): handle_auth_done global_id 134196 payload 274 2023-10-30T12:09:52.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 1 -- 172.21.15.50:0/1720049854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009c660 msgr2=0x7f7150139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:09:52.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 1 --2- 172.21.15.50:0/1720049854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009c660 0x7f7150139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 1 -- 172.21.15.50:0/1720049854 >> v1:172.21.15.50:6789/0 conn(0x7f715009bb70 legacy=0x7f715009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:52.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:09:52.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:52.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:52.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.990+0000 7f714f7fe640 1 -- 172.21.15.50:0/1720049854 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150097e00 con 0x7f715009d3f0 2023-10-30T12:09:52.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714f7fe640 1 --2- 172.21.15.50:0/1720049854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009d3f0 0x7f7150137960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f714000ba40 tx=0x7f714002df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85629cd404eb61e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 1 -- 172.21.15.50:0/1720049854 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714002ed30 con 0x7f715009d3f0 2023-10-30T12:09:52.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:52.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: dump: 2023-10-30T12:09:52.010 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.010 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.012 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.012 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: _finish_auth 0 2023-10-30T12:09:52.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: _check_auth_tickets 2023-10-30T12:09:52.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:52.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 5 monclient: authenticate success, global_id 134196 2023-10-30T12:09:52.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 1 -- 172.21.15.50:0/1720049854 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714002ee90 con 0x7f715009d3f0 2023-10-30T12:09:52.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:09:52.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:52.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:52.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 4 set_mon_vals no callback set 2023-10-30T12:09:52.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 1 -- 172.21.15.50:0/1720049854 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140036660 con 0x7f715009d3f0 2023-10-30T12:09:52.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:52.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f714effd640 10 monclient: dump: 2023-10-30T12:09:52.014 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.015 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.015 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.015 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.015 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.015 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.015 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.016 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.016 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.016 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 10 monclient: shutdown 2023-10-30T12:09:52.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:52.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 -- 172.21.15.50:0/1720049854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009d3f0 msgr2=0x7f7150137960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 --2- 172.21.15.50:0/1720049854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009d3f0 0x7f7150137960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f714000ba40 tx=0x7f714002df80 comp rx=0 tx=0).stop 2023-10-30T12:09:52.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 -- 172.21.15.50:0/1720049854 shutdown_connections 2023-10-30T12:09:52.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 --2- 172.21.15.50:0/1720049854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009c660 0x7f7150139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 --2- 172.21.15.50:0/1720049854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009d3f0 0x7f7150137960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 -- 172.21.15.50:0/1720049854 >> 172.21.15.50:0/1720049854 conn(0x7f71500a3980 msgr2=0x7f71500a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:52.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 -- 172.21.15.50:0/1720049854 shutdown_connections 2023-10-30T12:09:52.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.991+0000 7f7156439640 1 -- 172.21.15.50:0/1720049854 wait complete. 2023-10-30T12:09:52.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.992+0000 7f7156439640 10 monclient: build_initial_monmap 2023-10-30T12:09:52.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.992+0000 7f7156439640 10 monclient: monmap: 2023-10-30T12:09:52.018 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:52.019 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:52.019 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:52.019 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:52.019 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:52.019 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.020 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:52.020 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:52.020 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:52.020 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.992+0000 7f7156439640 1 librados: starting msgr at 2023-10-30T12:09:52.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.992+0000 7f7156439640 1 librados: starting objecter 2023-10-30T12:09:52.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.992+0000 7f7156439640 1 Processor -- start 2023-10-30T12:09:52.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 -- start start 2023-10-30T12:09:52.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 librados: setting wanted keys 2023-10-30T12:09:52.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 librados: calling monclient init 2023-10-30T12:09:52.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient: init 2023-10-30T12:09:52.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:52.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:09:52.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f715009bb70 0x7f715008f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): picked mon.noname-b con 0x7f715009bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:52.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009c660 0x7f715008f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): picked mon.noname-a con 0x7f715009c660 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:52.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009d3f0 0x7f715008fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): picked mon.noname-c con 0x7f715009d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:52.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150090f80 con 0x7f715009c660 2023-10-30T12:09:52.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150091ba0 con 0x7f715009bb70 2023-10-30T12:09:52.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7150091e80 con 0x7f715009d3f0 2023-10-30T12:09:52.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:52.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f7156439640 10 monclient(hunting): authenticate will time out at 1437.176605s 2023-10-30T12:09:52.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009c660 0x7f715008f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009d3f0 0x7f715008fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714ffff640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f715009bb70 msgr2=0x7f715008f3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:52.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009d3f0 0x7f715008fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56276/0 (socket says 172.21.15.50:56276) 2023-10-30T12:09:52.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009c660 0x7f715008f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43260/0 (socket says 172.21.15.50:43260) 2023-10-30T12:09:52.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 1 -- 172.21.15.50:0/1694600898 learned_addr learned my addr 172.21.15.50:0/1694600898 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:52.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): get_auth_request con 0x7f715009d3f0 auth_method 0 2023-10-30T12:09:52.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714ffff640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f715009bb70 0x7f715008f3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:52.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 10 monclient(hunting): get_auth_request con 0x7f715009c660 auth_method 0 2023-10-30T12:09:52.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f714f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.993+0000 7f71549d6640 10 monclient(hunting): handle_auth_done global_id 114410 payload 824 2023-10-30T12:09:52.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f715009bb70 msgr2=0x7f715008f3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:52.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f715009bb70 0x7f715008f3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009c660 msgr2=0x7f715008f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009c660 0x7f715008f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 10 monclient: _finish_hunting 0 2023-10-30T12:09:52.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 1 monclient: found mon.noname-c 2023-10-30T12:09:52.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:52.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:09:52.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 1 -- 172.21.15.50:0/1694600898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714000b6a0 con 0x7f715009d3f0 2023-10-30T12:09:52.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714f7fe640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f715009c660 0x7f715008f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:52.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f71549d6640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009d3f0 0x7f715008fe60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f714800efc0 tx=0x7f714800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 client.?.objecter ms_handle_connect 0x7f715009d3f0 2023-10-30T12:09:52.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:52.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714800cd20 con 0x7f715009d3f0 2023-10-30T12:09:52.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:09:52.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: dump: 2023-10-30T12:09:52.032 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.033 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.033 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.033 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.033 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.034 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.034 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.034 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.034 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: _finish_auth 0 2023-10-30T12:09:52.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:09:52.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:52.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714800f040 con 0x7f715009d3f0 2023-10-30T12:09:52.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:52.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148003bd0 con 0x7f715009d3f0 2023-10-30T12:09:52.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 5 monclient: authenticate success, global_id 114410 2023-10-30T12:09:52.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:52.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714cff9640 10 monclient: dump: 2023-10-30T12:09:52.036 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.036 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.037 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.037 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.037 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.037 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.037 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.037 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.038 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.038 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 10 monclient: _renew_subs 2023-10-30T12:09:52.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f714e7fc640 4 set_mon_vals no callback set 2023-10-30T12:09:52.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:52.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71500976e0 con 0x7f715009d3f0 2023-10-30T12:09:52.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 10 client.114410.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:52.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 10 monclient: _renew_subs 2023-10-30T12:09:52.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:09:52.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7150090c40 con 0x7f715009d3f0 2023-10-30T12:09:52.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.994+0000 7f7156439640 1 librados: init done 2023-10-30T12:09:52.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f7156439640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:52.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f7156439640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7150092040 0x7f71501ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f7156439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150073ff0 con 0x7f7150092040 2023-10-30T12:09:52.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7148003d30 con 0x7f715009d3f0 2023-10-30T12:09:52.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7150092040 0x7f71501ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient: get_auth_request con 0x7f7150092040 auth_method 0 2023-10-30T12:09:52.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f713c0392a0 0x7f713c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(318..318 src has 1..318) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f714804b1a0 con 0x7f715009d3f0 2023-10-30T12:09:52.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 10 client.114410.objecter ms_dispatch 0x7f7150000b90 osd_map(318..318 src has 1..318) v4 2023-10-30T12:09:52.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 3 client.114410.objecter handle_osd_map got epochs [318,318] > 0 2023-10-30T12:09:52.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 3 client.114410.objecter handle_osd_map decoding full epoch 318 2023-10-30T12:09:52.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 10 monclient(hunting): handle_auth_done global_id 134205 payload 824 2023-10-30T12:09:52.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 1 -- 172.21.15.50:0/1694600898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7150071870 con 0x7f7150092040 2023-10-30T12:09:52.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 20 client.114410.objecter dump_active .. 0 homeless 2023-10-30T12:09:52.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 10 client.114410.objecter ms_handle_connect 0x7f7150092040 2023-10-30T12:09:52.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714ffff640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7150092040 0x7f71501ca6b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7144001070 tx=0x7f714400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 10 client.114410.objecter resend_mon_ops 2023-10-30T12:09:52.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714f7fe640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f713c0392a0 0x7f713c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71440074e0 con 0x7f7150092040 2023-10-30T12:09:52.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:52.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714cff9640 10 monclient: dump: 2023-10-30T12:09:52.045 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.045 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.045 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.046 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.046 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.046 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.046 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.046 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.047 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.047 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.995+0000 7f714f7fe640 10 monclient: get_auth_request con 0x7f713c0392a0 auth_method 0 2023-10-30T12:09:52.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f7144007640 con 0x7f7150092040 2023-10-30T12:09:52.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:52.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:52.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7150092040 msgr2=0x7f71501ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7150092040 0x7f71501ca6b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7144001070 tx=0x7f714400c7f0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7150092040 msgr2=0x7f71501ca6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:52.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7150092040 0x7f71501ca6b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714cff9640 10 client.114410.objecter ms_handle_connect 0x7f713c0392a0 2023-10-30T12:09:52.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:51.996+0000 7f714f7fe640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f713c0392a0 0x7f713c03b760 secure :-1 s=READY pgs=4082 cs=0 l=1 rev1=1 crypto rx=0x7f7140038000 tx=0x7f7140008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f7156439640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:52.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f7156439640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71501cabf0 0x7f71501cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f7156439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150097560 con 0x7f71501cabf0 2023-10-30T12:09:52.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71501cabf0 0x7f71501cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient: get_auth_request con 0x7f71501cabf0 auth_method 0 2023-10-30T12:09:52.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 10 monclient(hunting): handle_auth_done global_id 134208 payload 824 2023-10-30T12:09:52.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.002+0000 7f714ffff640 1 -- 172.21.15.50:0/1694600898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f715009f600 con 0x7f71501cabf0 2023-10-30T12:09:52.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714ffff640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71501cabf0 0x7f71501cb000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f714400cfd0 tx=0x7f71440011c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 10 client.114410.objecter ms_handle_connect 0x7f71501cabf0 2023-10-30T12:09:52.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 10 client.114410.objecter resend_mon_ops 2023-10-30T12:09:52.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714401c660 con 0x7f71501cabf0 2023-10-30T12:09:52.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:52.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 10 monclient: dump: 2023-10-30T12:09:52.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.055 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.055 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.055 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.055 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.056 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.056 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.056 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.056 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7f7144023070 con 0x7f71501cabf0 2023-10-30T12:09:52.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:52.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:52.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71501cabf0 msgr2=0x7f71501cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71501cabf0 0x7f71501cb000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f714400cfd0 tx=0x7f71440011c0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71501cabf0 msgr2=0x7f71501cb000 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:52.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f714cff9640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71501cabf0 0x7f71501cb000 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f714400cfd0 tx=0x7f71440011c0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.058 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:52.058 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:52.058 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:52.059 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:09:52.059 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 92, 2023-10-30T12:09:52.059 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:52.059 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:52.059 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:52.059 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.060 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:09:52.060 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:52.060 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:52.060 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:52.060 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:52.061 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:52.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:52.061 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:52.061 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:52.061 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:52.062 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:52.062 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:52.062 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:52.062 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:52.062 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.062 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:52.063 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:52.063 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:52.063 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:52.063 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:52.063 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:52.064 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:52.064 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:52.064 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:52.064 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:52.064 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:52.064 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:52.065 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.065 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:52.065 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:52.065 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:52.066 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:52.066 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:52.066 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:52.066 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:52.066 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:52.067 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:52.067 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:52.067 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:52.067 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:52.067 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:52.067 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:52.068 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:52.068 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:52.068 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:52.068 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:52.068 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:52.069 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:52.069 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:52.069 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:52.069 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:52.069 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:52.070 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:52.070 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:52.070 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:52.070 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.070 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:52.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:52.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:52.071 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.073 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:52.073 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:52.073 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.074 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.074 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:52.074 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:52.074 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:52.075 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:52.075 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:52.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:52.076 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:52.076 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:52.076 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:52.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.076 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:52.076 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:52.077 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.077 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:52.077 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:52.078 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.078 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.078 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:52.079 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:52.079 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:52.079 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:52.079 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:52.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:52.080 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:52.080 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:52.080 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:52.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.082 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:52.082 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:52.082 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.082 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.082 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.083 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:52.083 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:52.083 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:52.083 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:52.083 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:52.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.084 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.084 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:52.084 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:52.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.085 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:52.085 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:52.085 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:52.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.085 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.086 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:09:52.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.086 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:52.086 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:52.086 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:09:52.087 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.087 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.087 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:52.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.087 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:52.088 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:52.088 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:09:52.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.088 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.088 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:09:52.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.090 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:52.090 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:52.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 10 librados: watch_flush enter 2023-10-30T12:09:52.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 10 librados: watch_flush exit 2023-10-30T12:09:52.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 20 client.114410.objecter shutdown clearing up homeless session... 2023-10-30T12:09:52.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 10 client.114410.objecter successfully canceled tick 2023-10-30T12:09:52.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f713c0392a0 msgr2=0x7f713c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f713c0392a0 0x7f713c03b760 secure :-1 s=READY pgs=4082 cs=0 l=1 rev1=1 crypto rx=0x7f7140038000 tx=0x7f7140008040 comp rx=0 tx=0).stop 2023-10-30T12:09:52.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 10 monclient: shutdown 2023-10-30T12:09:52.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:52.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009d3f0 msgr2=0x7f715008fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f71549d6640 1 -- 172.21.15.50:0/1694600898 reap_dead start 2023-10-30T12:09:52.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009d3f0 0x7f715008fe60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f714800efc0 tx=0x7f714800c490 comp rx=0 tx=0).stop 2023-10-30T12:09:52.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 shutdown_connections 2023-10-30T12:09:52.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 --2- 172.21.15.50:0/1694600898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f715009d3f0 0x7f715008fe60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 >> 172.21.15.50:0/1694600898 conn(0x7f71500a3980 msgr2=0x7f7150096520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:52.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 shutdown_connections 2023-10-30T12:09:52.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 -- 172.21.15.50:0/1694600898 wait complete. 2023-10-30T12:09:52.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.003+0000 7f7156439640 1 librados: shutdown 2023-10-30T12:09:52.096 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:09:52.096 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:09:52.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:09:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.201+0000 7f3b11cf2640 10 monclient: get_monmap_and_config 2023-10-30T12:09:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.201+0000 7f3b11cf2640 10 monclient: build_initial_monmap 2023-10-30T12:09:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.201+0000 7f3b11cf2640 10 monclient: monmap: 2023-10-30T12:09:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:09:52.202760+0000 2023-10-30T12:09:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:09:52.202760+0000 2023-10-30T12:09:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:09:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:09:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:09:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:09:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:09:52.206 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 1 Processor -- start 2023-10-30T12:09:52.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 1 -- start start 2023-10-30T12:09:52.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient: init 2023-10-30T12:09:52.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:52.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T12:09:52.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c071c00 0x7f3b0c09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient(hunting): picked mon.noname-b con 0x7f3b0c071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:52.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3b0c097fe0 0x7f3b0c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient(hunting): picked mon.noname-c con 0x7f3b0c097fe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:09:52.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b0c09afa0 addr v1:172.21.15.118:6789/0 2023-10-30T12:09:52.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0c097600 con 0x7f3b0c09afa0 2023-10-30T12:09:52.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0c0978e0 con 0x7f3b0c097fe0 2023-10-30T12:09:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c09bf50 con 0x7f3b0c071c00 2023-10-30T12:09:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.202+0000 7f3b11cf2640 10 monclient(hunting): authenticate will time out at 1437.385920s 2023-10-30T12:09:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c071c00 0x7f3b0c09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0bfff640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f3b0c09afa0 legacy=0x7f3b0c138750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:09:52.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0affd640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f3b0c097fe0 msgr2=0x7f3b0c09aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:52.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0affd640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3b0c097fe0 0x7f3b0c09aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:52.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c071c00 0x7f3b0c09a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56278/0 (socket says 172.21.15.50:56278) 2023-10-30T12:09:52.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/791751302 learned_addr learned my addr 172.21.15.50:0/791751302 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:52.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): get_auth_request con 0x7f3b0c071c00 auth_method 0 2023-10-30T12:09:52.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_done global_id 114413 payload 274 2023-10-30T12:09:52.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/791751302 >> v2:172.21.15.118:3300/0 conn(0x7f3b0c097fe0 msgr2=0x7f3b0c09aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:52.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 --2- 172.21.15.50:0/791751302 >> v2:172.21.15.118:3300/0 conn(0x7f3b0c097fe0 0x7f3b0c09aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/791751302 >> v1:172.21.15.118:6789/0 conn(0x7f3b0c09afa0 legacy=0x7f3b0c138750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:09:52.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:52.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 monclient: found mon.noname-b 2023-10-30T12:09:52.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:52.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:09:52.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.203+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/791751302 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c09d9b0 con 0x7f3b0c071c00 2023-10-30T12:09:52.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0b7fe640 1 --2- 172.21.15.50:0/791751302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c071c00 0x7f3b0c09a150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009850 tx=0x7f3afc02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a01e24e4359a0756 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 1 -- 172.21.15.50:0/791751302 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc004230 con 0x7f3b0c071c00 2023-10-30T12:09:52.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:09:52.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: dump: 2023-10-30T12:09:52.223 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.223 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.224 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.224 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.224 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.224 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.225 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.225 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.225 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.225 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:09:52.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:09:52.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:52.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 1 -- 172.21.15.50:0/791751302 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc004390 con 0x7f3b0c071c00 2023-10-30T12:09:52.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:52.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 1 -- 172.21.15.50:0/791751302 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc003810 con 0x7f3b0c071c00 2023-10-30T12:09:52.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 5 monclient: authenticate success, global_id 114413 2023-10-30T12:09:52.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 10 monclient: get_monmap_and_config success 2023-10-30T12:09:52.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 4 set_mon_vals no callback set 2023-10-30T12:09:52.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:52.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b0a7fc640 10 monclient: dump: 2023-10-30T12:09:52.228 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.228 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.229 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.229 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.229 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.229 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.230 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.230 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.230 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.230 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 10 monclient: shutdown 2023-10-30T12:09:52.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:52.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 -- 172.21.15.50:0/791751302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c071c00 msgr2=0x7f3b0c09a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 --2- 172.21.15.50:0/791751302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c071c00 0x7f3b0c09a150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009850 tx=0x7f3afc02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 -- 172.21.15.50:0/791751302 shutdown_connections 2023-10-30T12:09:52.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 --2- 172.21.15.50:0/791751302 >> v2:172.21.15.118:3300/0 conn(0x7f3b0c097fe0 0x7f3b0c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 --2- 172.21.15.50:0/791751302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c071c00 0x7f3b0c09a150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 -- 172.21.15.50:0/791751302 >> 172.21.15.50:0/791751302 conn(0x7f3b0c0a3980 msgr2=0x7f3b0c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:52.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 -- 172.21.15.50:0/791751302 shutdown_connections 2023-10-30T12:09:52.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.204+0000 7f3b11cf2640 1 -- 172.21.15.50:0/791751302 wait complete. 2023-10-30T12:09:52.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 10 monclient: build_initial_monmap 2023-10-30T12:09:52.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 10 monclient: monmap: 2023-10-30T12:09:52.233 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:09:52.233 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:09:52.233 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:09:52.234 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:09:52.234 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:09:52.234 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.234 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:09:52.234 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:09:52.235 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:09:52.235 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 1 librados: starting msgr at 2023-10-30T12:09:52.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 1 librados: starting objecter 2023-10-30T12:09:52.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 1 Processor -- start 2023-10-30T12:09:52.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 1 -- start start 2023-10-30T12:09:52.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 1 librados: setting wanted keys 2023-10-30T12:09:52.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 1 librados: calling monclient init 2023-10-30T12:09:52.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.205+0000 7f3b11cf2640 10 monclient: init 2023-10-30T12:09:52.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:09:52.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:09:52.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b0c071c00 0x7f3b0c1b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): picked mon.noname-a con 0x7f3b0c071c00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:09:52.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b0c097fe0 0x7f3b0c1c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): picked mon.noname-b con 0x7f3b0c097fe0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:09:52.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c09afa0 0x7f3b0c1c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): picked mon.noname-c con 0x7f3b0c09afa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:09:52.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1b1dc0 con 0x7f3b0c071c00 2023-10-30T12:09:52.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1c7890 con 0x7f3b0c097fe0 2023-10-30T12:09:52.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c0708d0 con 0x7f3b0c09afa0 2023-10-30T12:09:52.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:09:52.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b11cf2640 10 monclient(hunting): authenticate will time out at 1437.389414s 2023-10-30T12:09:52.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b0c071c00 0x7f3b0c1b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0affd640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b0c097fe0 msgr2=0x7f3b0c1c6e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:09:52.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b0c071c00 0x7f3b0c1b5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43288/0 (socket says 172.21.15.50:43288) 2023-10-30T12:09:52.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c09afa0 0x7f3b0c1c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/4204888971 learned_addr learned my addr 172.21.15.50:0/4204888971 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:09:52.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): get_auth_request con 0x7f3b0c071c00 auth_method 0 2023-10-30T12:09:52.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0affd640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b0c097fe0 0x7f3b0c1c6e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:09:52.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 10 monclient(hunting): get_auth_request con 0x7f3b0c09afa0 auth_method 0 2023-10-30T12:09:52.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient(hunting): handle_auth_done global_id 134211 payload 824 2023-10-30T12:09:52.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c09afa0 msgr2=0x7f3b0c1c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c09afa0 0x7f3b0c1c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b0c097fe0 msgr2=0x7f3b0c1c6e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:09:52.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b0c097fe0 0x7f3b0c1c6e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:09:52.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:09:52.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:09:52.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:09:52.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0b7fe640 1 -- 172.21.15.50:0/4204888971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc009460 con 0x7f3b0c071c00 2023-10-30T12:09:52.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.206+0000 7f3b0bfff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c09afa0 0x7f3b0c1c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:09:52.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b0b7fe640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b0c071c00 0x7f3b0c1b5b40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000c00 tx=0x7f3afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 client.?.objecter ms_handle_connect 0x7f3b0c071c00 2023-10-30T12:09:52.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:09:52.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc003e50 con 0x7f3b0c071c00 2023-10-30T12:09:52.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:09:52.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: dump: 2023-10-30T12:09:52.248 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.248 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.249 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.249 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.249 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.249 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.249 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.250 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.250 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.250 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: _finish_auth 0 2023-10-30T12:09:52.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:09:52.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:09:52.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 5 monclient: authenticate success, global_id 134211 2023-10-30T12:09:52.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc005e50 con 0x7f3b0c071c00 2023-10-30T12:09:52.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 10 monclient: _renew_subs 2023-10-30T12:09:52.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:52.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0c1b2030 con 0x7f3b0c071c00 2023-10-30T12:09:52.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:09:52.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc006240 con 0x7f3b0c071c00 2023-10-30T12:09:52.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:09:52.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3af3fff640 10 monclient: dump: 2023-10-30T12:09:52.253 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.253 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.254 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.254 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.254 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.254 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.255 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.255 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.255 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.255 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 10 client.134211.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:09:52.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 10 monclient: _renew_subs 2023-10-30T12:09:52.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b09ffb640 4 set_mon_vals no callback set 2023-10-30T12:09:52.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:09:52.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0c1b6080 con 0x7f3b0c071c00 2023-10-30T12:09:52.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 1 librados: init done 2023-10-30T12:09:52.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:52.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1ca370 0x7f3b0c1ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.207+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1b62d0 con 0x7f3b0c1ca370 2023-10-30T12:09:52.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3afc03a050 con 0x7f3b0c071c00 2023-10-30T12:09:52.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1ca370 0x7f3b0c1ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient: get_auth_request con 0x7f3b0c1ca370 auth_method 0 2023-10-30T12:09:52.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3af80392a0 0x7f3af803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 10 monclient(hunting): handle_auth_done global_id 114419 payload 824 2023-10-30T12:09:52.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 1 -- 172.21.15.50:0/4204888971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b0c071fe0 con 0x7f3b0c1ca370 2023-10-30T12:09:52.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(318..318 src has 1..318) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f3afc06c9f0 con 0x7f3b0c071c00 2023-10-30T12:09:52.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 client.134211.objecter ms_dispatch 0x7f3b0c000b90 osd_map(318..318 src has 1..318) v4 2023-10-30T12:09:52.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 3 client.134211.objecter handle_osd_map got epochs [318,318] > 0 2023-10-30T12:09:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 3 client.134211.objecter handle_osd_map decoding full epoch 318 2023-10-30T12:09:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 20 client.134211.objecter dump_active .. 0 homeless 2023-10-30T12:09:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 client.134211.objecter ms_handle_connect 0x7f3b0c1ca370 2023-10-30T12:09:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 client.134211.objecter resend_mon_ops 2023-10-30T12:09:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0affd640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1ca370 0x7f3b0c1ca750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c09c7f0 tx=0x7f3b0000b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00003d40 con 0x7f3b0c1ca370 2023-10-30T12:09:52.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:52.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 monclient: dump: 2023-10-30T12:09:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.264 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.264 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.264 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.264 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.264 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0bfff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3af80392a0 0x7f3af803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3b0bfff640 10 monclient: get_auth_request con 0x7f3af80392a0 auth_method 0 2023-10-30T12:09:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f3b00005070 con 0x7f3b0c1ca370 2023-10-30T12:09:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:09:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:09:52.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1ca370 msgr2=0x7f3b0c1ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1ca370 0x7f3b0c1ca750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c09c7f0 tx=0x7f3b0000b8e0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1ca370 msgr2=0x7f3b0c1ca750 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:52.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.208+0000 7f3af3fff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1ca370 0x7f3b0c1ca750 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c09c7f0 tx=0x7f3b0000b8e0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.209+0000 7f3af3fff640 10 client.134211.objecter ms_handle_connect 0x7f3af80392a0 2023-10-30T12:09:52.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.209+0000 7f3b0bfff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3af80392a0 0x7f3af803b760 secure :-1 s=READY pgs=4083 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c070b80 tx=0x7f3b0400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b11cf2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:09:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b11cf2640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1cac90 0x7f3b0c06bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:09:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b11cf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:09:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c073ff0 con 0x7f3b0c1cac90 2023-10-30T12:09:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1cac90 0x7f3b0c06bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:09:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient: get_auth_request con 0x7f3b0c1cac90 auth_method 0 2023-10-30T12:09:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:09:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:09:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:09:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:09:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:09:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:09:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 10 monclient(hunting): handle_auth_done global_id 114422 payload 824 2023-10-30T12:09:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 1 -- 172.21.15.50:0/4204888971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b0c070130 con 0x7f3b0c1cac90 2023-10-30T12:09:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 10 client.134211.objecter ms_handle_connect 0x7f3b0c1cac90 2023-10-30T12:09:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 10 client.134211.objecter resend_mon_ops 2023-10-30T12:09:52.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3b0affd640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1cac90 0x7f3b0c06bba0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b00013000 tx=0x7f3b00005990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:09:52.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0001da20 con 0x7f3b0c1cac90 2023-10-30T12:09:52.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:09:52.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:09:52.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 10 monclient: dump: 2023-10-30T12:09:52.274 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:09:52.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:09:52.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.274 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:09:52.275 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:09:52.275 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:09:52.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:09:52.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:09:52.275 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:09:52.276 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:09:52.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f3b0001db80 con 0x7f3b0c1cac90 2023-10-30T12:09:52.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:09:52.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:09:52.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1cac90 msgr2=0x7f3b0c06bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1cac90 0x7f3b0c06bba0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b00013000 tx=0x7f3b00005990 comp rx=0 tx=0).stop 2023-10-30T12:09:52.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1cac90 msgr2=0x7f3b0c06bba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:09:52.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.215+0000 7f3af3fff640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b0c1cac90 0x7f3b0c06bba0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b00013000 tx=0x7f3b00005990 comp rx=0 tx=0).stop 2023-10-30T12:09:52.278 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:09:52.278 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:52.278 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:52.278 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:09:52.278 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 92, 2023-10-30T12:09:52.279 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:09:52.279 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:09:52.279 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:09:52.279 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.280 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:09:52.280 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:52.280 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:09:52.280 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:09:52.280 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:52.281 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:52.281 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:52.281 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:52.281 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:52.282 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:52.282 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:52.282 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:52.282 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:52.283 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:52.283 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.283 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:09:52.283 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:09:52.283 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:52.284 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:52.284 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:52.284 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:52.284 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:52.284 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:52.285 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:52.285 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:52.285 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:52.285 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:52.285 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.286 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:09:52.286 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:09:52.286 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:09:52.287 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:09:52.287 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:09:52.287 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:09:52.287 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:52.287 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:09:52.288 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:09:52.288 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:09:52.288 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:09:52.288 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:09:52.289 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:09:52.289 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:09:52.289 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:09:52.289 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:09:52.289 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:09:52.290 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:09:52.290 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:09:52.290 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:09:52.290 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:09:52.291 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:09:52.291 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:09:52.291 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:09:52.291 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:09:52.291 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:09:52.292 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:09:52.292 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.292 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:09:52.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.292 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:09:52.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.293 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:09:52.293 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:09:52.293 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:52.294 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:52.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:52.294 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:09:52.295 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.295 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:52.296 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:09:52.296 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.296 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.296 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:09:52.297 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:09:52.297 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:52.298 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:52.298 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:52.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.298 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:09:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:09:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:09:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:09:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:09:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:09:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:09:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:09:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:09:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:09:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:09:52.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:09:52.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.306 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:09:52.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:09:52.306 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:09:52.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.306 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:09:52.307 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:09:52.307 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:09:52.307 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:09:52.308 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:09:52.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.309 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:09:52.309 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:09:52.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.309 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:52.309 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:52.310 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:09:52.310 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.310 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.310 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:09:52.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.311 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:52.311 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:52.311 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:09:52.311 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.312 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:09:52.312 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:09:52.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:09:52.312 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:09:52.313 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:09:52.313 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:09:52.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:09:52.313 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:09:52.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:09:52.314 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:09:52.314 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:09:52.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 10 librados: watch_flush enter 2023-10-30T12:09:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 10 librados: watch_flush exit 2023-10-30T12:09:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 20 client.134211.objecter shutdown clearing up homeless session... 2023-10-30T12:09:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 10 client.134211.objecter successfully canceled tick 2023-10-30T12:09:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3af80392a0 msgr2=0x7f3af803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3af80392a0 0x7f3af803b760 secure :-1 s=READY pgs=4083 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c070b80 tx=0x7f3b0400d040 comp rx=0 tx=0).stop 2023-10-30T12:09:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 10 monclient: shutdown 2023-10-30T12:09:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:09:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b0c071c00 msgr2=0x7f3b0c1b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:09:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 --2- 172.21.15.50:0/4204888971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b0c071c00 0x7f3b0c1b5b40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000c00 tx=0x7f3afc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:09:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b0bfff640 1 -- 172.21.15.50:0/4204888971 reap_dead start 2023-10-30T12:09:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b0bfff640 1 -- 172.21.15.50:0/4204888971 reap_dead start 2023-10-30T12:09:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 shutdown_connections 2023-10-30T12:09:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 >> 172.21.15.50:0/4204888971 conn(0x7f3b0c0a3980 msgr2=0x7f3b0c096540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:09:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 shutdown_connections 2023-10-30T12:09:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 -- 172.21.15.50:0/4204888971 wait complete. 2023-10-30T12:09:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:09:52.216+0000 7f3b11cf2640 1 librados: shutdown 2023-10-30T12:09:52.320 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:09:52.321 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:10:12.322 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:10:12.322 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-10-30T12:10:12.323 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:10:12.323 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:10:12.326 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:10:12.327 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:10:12.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.443+0000 7f3a1b640640 10 monclient: get_monmap_and_config 2023-10-30T12:10:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.443+0000 7f3a1b640640 10 monclient: build_initial_monmap 2023-10-30T12:10:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.443+0000 7f3a1b640640 10 monclient: monmap: 2023-10-30T12:10:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:12.444359+0000 2023-10-30T12:10:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:12.444359+0000 2023-10-30T12:10:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:12.447 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.443+0000 7f3a1b640640 1 Processor -- start 2023-10-30T12:10:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 1 -- start start 2023-10-30T12:10:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient: init 2023-10-30T12:10:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-10-30T12:10:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a14098520 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a14099010 0x7f3a1409bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient(hunting): picked mon.noname-b con 0x7f3a14099010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3a141378f0 0x7f3a14139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient(hunting): picked mon.noname-a con 0x7f3a141378f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a140999f0 con 0x7f3a14098520 2023-10-30T12:10:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a140993f0 con 0x7f3a141378f0 2023-10-30T12:10:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1409b380 con 0x7f3a14099010 2023-10-30T12:10:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a1b640640 10 monclient(hunting): authenticate will time out at 1457.627616s 2023-10-30T12:10:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3a141378f0 0x7f3a14139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a14099010 0x7f3a1409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3a141378f0 0x7f3a14139ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44836/0 (socket says 172.21.15.50:44836) 2023-10-30T12:10:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a14099010 0x7f3a1409bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41128/0 (socket says 172.21.15.50:41128) 2023-10-30T12:10:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 1 -- 172.21.15.50:0/320569942 learned_addr learned my addr 172.21.15.50:0/320569942 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 10 monclient(hunting): get_auth_request con 0x7f3a141378f0 auth_method 0 2023-10-30T12:10:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 10 monclient(hunting): get_auth_request con 0x7f3a14099010 auth_method 0 2023-10-30T12:10:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a19bdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.444+0000 7f3a18bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 10 monclient(hunting): handle_auth_done global_id 114425 payload 274 2023-10-30T12:10:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 1 -- 172.21.15.50:0/320569942 >> v2:172.21.15.50:3300/0 conn(0x7f3a141378f0 msgr2=0x7f3a14139ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 1 --2- 172.21.15.50:0/320569942 >> v2:172.21.15.50:3300/0 conn(0x7f3a141378f0 0x7f3a14139ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 1 -- 172.21.15.50:0/320569942 >> v1:172.21.15.118:6789/0 conn(0x7f3a14098520 legacy=0x7f3a14098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 10 monclient: _finish_hunting 0 2023-10-30T12:10:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 1 monclient: found mon.noname-b 2023-10-30T12:10:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:10:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 1 -- 172.21.15.50:0/320569942 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1409b5d0 con 0x7f3a14099010 2023-10-30T12:10:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a19bdd640 1 --2- 172.21.15.50:0/320569942 >> v2:172.21.15.50:3300/0 conn(0x7f3a141378f0 0x7f3a14139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.445+0000 7f3a18bdb640 1 --2- 172.21.15.50:0/320569942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a14099010 0x7f3a1409bfb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3a040097a0 tx=0x7f3a0402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=393dd5f6b05dfc6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 1 -- 172.21.15.50:0/320569942 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0402f050 con 0x7f3a14099010 2023-10-30T12:10:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: dump: 2023-10-30T12:10:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:12.466 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 1 -- 172.21.15.50:0/320569942 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04031700 con 0x7f3a14099010 2023-10-30T12:10:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 5 monclient: authenticate success, global_id 114425 2023-10-30T12:10:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:12.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 4 set_mon_vals no callback set 2023-10-30T12:10:12.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 1 -- 172.21.15.50:0/320569942 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04031a90 con 0x7f3a14099010 2023-10-30T12:10:12.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:12.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a0bfff640 10 monclient: dump: 2023-10-30T12:10:12.468 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:12.469 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:12.469 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.469 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.469 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.469 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:12.470 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:12.470 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:12.470 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:12.470 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:12.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 10 monclient: shutdown 2023-10-30T12:10:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 -- 172.21.15.50:0/320569942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a14099010 msgr2=0x7f3a1409bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 --2- 172.21.15.50:0/320569942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a14099010 0x7f3a1409bfb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3a040097a0 tx=0x7f3a0402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 -- 172.21.15.50:0/320569942 shutdown_connections 2023-10-30T12:10:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 --2- 172.21.15.50:0/320569942 >> v2:172.21.15.50:3300/0 conn(0x7f3a141378f0 0x7f3a14139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 --2- 172.21.15.50:0/320569942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a14099010 0x7f3a1409bfb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 -- 172.21.15.50:0/320569942 >> 172.21.15.50:0/320569942 conn(0x7f3a140a0390 msgr2=0x7f3a140a07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 -- 172.21.15.50:0/320569942 shutdown_connections 2023-10-30T12:10:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.446+0000 7f3a1b640640 1 -- 172.21.15.50:0/320569942 wait complete. 2023-10-30T12:10:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 10 monclient: build_initial_monmap 2023-10-30T12:10:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 10 monclient: monmap: 2023-10-30T12:10:12.473 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:12.473 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:12.473 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:12.473 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:12.473 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:12.474 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 1 librados: starting msgr at 2023-10-30T12:10:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 1 librados: starting objecter 2023-10-30T12:10:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 1 Processor -- start 2023-10-30T12:10:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 1 -- start start 2023-10-30T12:10:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 1 librados: setting wanted keys 2023-10-30T12:10:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 1 librados: calling monclient init 2023-10-30T12:10:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.447+0000 7f3a1b640640 10 monclient: init 2023-10-30T12:10:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 0x7f3a14136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): picked mon.noname-b con 0x7f3a14098520 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:12.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a14099010 0x7f3a1408efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): picked mon.noname-a con 0x7f3a14099010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:12.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a141378f0 0x7f3a1408f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): picked mon.noname-c con 0x7f3a141378f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:12.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14099e30 con 0x7f3a14099010 2023-10-30T12:10:12.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14090e10 con 0x7f3a14098520 2023-10-30T12:10:12.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:12.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14091ee0 con 0x7f3a141378f0 2023-10-30T12:10:12.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:12.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a1b640640 10 monclient(hunting): authenticate will time out at 1457.631209s 2023-10-30T12:10:12.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a14099010 0x7f3a1408efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a14099010 0x7f3a1408efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44842/0 (socket says 172.21.15.50:44842) 2023-10-30T12:10:12.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 1 -- 172.21.15.50:0/3195635904 learned_addr learned my addr 172.21.15.50:0/3195635904 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:12.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a141378f0 0x7f3a1408f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 10 monclient(hunting): get_auth_request con 0x7f3a14099010 auth_method 0 2023-10-30T12:10:12.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): get_auth_request con 0x7f3a141378f0 auth_method 0 2023-10-30T12:10:12.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:12.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:12.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:12.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 0x7f3a14136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 10 monclient(hunting): get_auth_request con 0x7f3a14098520 auth_method 0 2023-10-30T12:10:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient(hunting): handle_auth_done global_id 114428 payload 824 2023-10-30T12:10:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 -- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 msgr2=0x7f3a14136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 1 -- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 msgr2=0x7f3a14136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:10:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 1 -- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 msgr2=0x7f3a14136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:10:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 0x7f3a14136d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a193dc640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 0x7f3a14136d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 0x7f3a14136d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 -- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a14099010 msgr2=0x7f3a1408efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a14099010 0x7f3a1408efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 monclient: found mon.noname-c 2023-10-30T12:10:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:10:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 -- 172.21.15.50:0/3195635904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c00bd40 con 0x7f3a141378f0 2023-10-30T12:10:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a18bdb640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a14099010 0x7f3a1408efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a19bdd640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a141378f0 0x7f3a1408f500 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009420 tx=0x7f3a0c00e490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a09ffb640 10 client.?.objecter ms_handle_connect 0x7f3a141378f0 2023-10-30T12:10:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.448+0000 7f3a09ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 1 -- 172.21.15.50:0/3195635904 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c010040 con 0x7f3a141378f0 2023-10-30T12:10:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: dump: 2023-10-30T12:10:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:12.489 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:12.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:12.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 1 -- 172.21.15.50:0/3195635904 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c00b2f0 con 0x7f3a141378f0 2023-10-30T12:10:12.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 5 monclient: authenticate success, global_id 114428 2023-10-30T12:10:12.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 10 monclient: _renew_subs 2023-10-30T12:10:12.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:12.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a04009400 con 0x7f3a141378f0 2023-10-30T12:10:12.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:12.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 1 -- 172.21.15.50:0/3195635904 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c009840 con 0x7f3a141378f0 2023-10-30T12:10:12.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:12.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:12.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a09ffb640 10 monclient: dump: 2023-10-30T12:10:12.492 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:12.492 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:12.492 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.492 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:12.492 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:12.493 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:12.493 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:12.493 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:12.493 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:12.493 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a0b7fe640 4 set_mon_vals no callback set 2023-10-30T12:10:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 10 client.114428.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 10 monclient: _renew_subs 2023-10-30T12:10:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a14092740 con 0x7f3a141378f0 2023-10-30T12:10:12.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.449+0000 7f3a1b640640 1 librados: init done 2023-10-30T12:10:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 1 -- 172.21.15.50:0/3195635904 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3a0c0099a0 con 0x7f3a141378f0 2023-10-30T12:10:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f39f8039320 0x7f39f803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 1 -- 172.21.15.50:0/3195635904 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(323..323 src has 1..323) v4 ==== 6327+0+0 (secure 0 0 0) 0x7f3a0c04b090 con 0x7f3a141378f0 2023-10-30T12:10:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 10 client.114428.objecter ms_dispatch 0x7f3a14000b90 osd_map(323..323 src has 1..323) v4 2023-10-30T12:10:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 3 client.114428.objecter handle_osd_map got epochs [323,323] > 0 2023-10-30T12:10:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 3 client.114428.objecter handle_osd_map decoding full epoch 323 2023-10-30T12:10:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a1b640640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a1b640640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a1b640640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 20 client.114428.objecter dump_active .. 0 homeless 2023-10-30T12:10:12.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a14092b40 con 0x7f3a141378f0 2023-10-30T12:10:12.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a193dc640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f39f8039320 0x7f39f803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:12.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a193dc640 10 monclient: get_auth_request con 0x7f39f8039320 auth_method 0 2023-10-30T12:10:12.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a09ffb640 10 client.114428.objecter ms_handle_connect 0x7f39f8039320 2023-10-30T12:10:12.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.450+0000 7f3a193dc640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f39f8039320 0x7f39f803b7e0 secure :-1 s=READY pgs=4085 cs=0 l=1 rev1=1 crypto rx=0x7f3a10007c70 tx=0x7f3a10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:12.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.455+0000 7f3a09ffb640 1 -- 172.21.15.50:0/3195635904 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a0c01a020 con 0x7f3a141378f0 2023-10-30T12:10:12.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.455+0000 7f3a09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:12.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.455+0000 7f3a09ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:12.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.698+0000 7f3a1b640640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:12.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.698+0000 7f3a1b640640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:12.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.698+0000 7f3a1b640640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:12.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.698+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3a1404e200 con 0x7f3a141378f0 2023-10-30T12:10:12.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.699+0000 7f3a09ffb640 1 -- 172.21.15.50:0/3195635904 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f3a0c00b460 con 0x7f3a141378f0 2023-10-30T12:10:12.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.699+0000 7f3a09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:12.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.699+0000 7f3a09ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:12.702 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:12.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.700+0000 7f3a1b640640 10 librados: watch_flush enter 2023-10-30T12:10:12.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.700+0000 7f3a1b640640 10 librados: watch_flush exit 2023-10-30T12:10:12.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.700+0000 7f3a1b640640 20 client.114428.objecter shutdown clearing up homeless session... 2023-10-30T12:10:12.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.700+0000 7f3a1b640640 10 client.114428.objecter successfully canceled tick 2023-10-30T12:10:12.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f39f8039320 msgr2=0x7f39f803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f39f8039320 0x7f39f803b7e0 secure :-1 s=READY pgs=4085 cs=0 l=1 rev1=1 crypto rx=0x7f3a10007c70 tx=0x7f3a10009040 comp rx=0 tx=0).stop 2023-10-30T12:10:12.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 10 monclient: shutdown 2023-10-30T12:10:12.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:12.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a141378f0 msgr2=0x7f3a1408f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:12.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a141378f0 0x7f3a1408f500 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009420 tx=0x7f3a0c00e490 comp rx=0 tx=0).stop 2023-10-30T12:10:12.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 shutdown_connections 2023-10-30T12:10:12.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f39f8039320 0x7f39f803b7e0 unknown :-1 s=CLOSED pgs=4085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a141378f0 0x7f3a1408f500 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a14099010 0x7f3a1408efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 --2- 172.21.15.50:0/3195635904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a14098520 0x7f3a14136d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:12.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 >> 172.21.15.50:0/3195635904 conn(0x7f3a140a0390 msgr2=0x7f3a141bdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:12.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 shutdown_connections 2023-10-30T12:10:12.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 -- 172.21.15.50:0/3195635904 wait complete. 2023-10-30T12:10:12.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:12.701+0000 7f3a1b640640 1 librados: shutdown 2023-10-30T12:10:15.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.828+0000 7fddbc1d0640 10 monclient: get_monmap_and_config 2023-10-30T12:10:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.828+0000 7fddbc1d0640 10 monclient: build_initial_monmap 2023-10-30T12:10:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.828+0000 7fddbc1d0640 10 monclient: monmap: 2023-10-30T12:10:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:15.830023+0000 2023-10-30T12:10:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:15.830023+0000 2023-10-30T12:10:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:15.833 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.829+0000 7fddbc1d0640 1 Processor -- start 2023-10-30T12:10:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.829+0000 7fddbc1d0640 1 -- start start 2023-10-30T12:10:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.829+0000 7fddbc1d0640 10 monclient: init 2023-10-30T12:10:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-10-30T12:10:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb40a0080 0x7fddb40a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient(hunting): picked mon.noname-b con 0x7fddb40a0080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddb40720b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 0x7fddb40a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient(hunting): picked mon.noname-c con 0x7fddb4073a30 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb409ccc0 con 0x7fddb40720b0 2023-10-30T12:10:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb4073620 con 0x7fddb4073a30 2023-10-30T12:10:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb40a0ee0 con 0x7fddb40a0080 2023-10-30T12:10:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddbc1d0640 10 monclient(hunting): authenticate will time out at 1461.013278s 2023-10-30T12:10:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb976b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fddb40720b0 0x7fddb4097320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51156/0 (socket says 172.21.15.50:51156) 2023-10-30T12:10:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb976b640 1 -- 172.21.15.50:0/3575914985 learned_addr learned my addr 172.21.15.50:0/3575914985 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:10:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/3575914985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb40a0080 0x7fddb40a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): get_auth_request con 0x7fddb40a0080 auth_method 0 2023-10-30T12:10:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 1 --2- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 0x7fddb40a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 10 monclient(hunting): get_auth_request con 0x7fddb4073a30 auth_method 0 2023-10-30T12:10:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb8f6a640 1 -- 172.21.15.50:0/3575914985 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91727492 0 0) 0x7fddb409ccc0 con 0x7fddb40720b0 2023-10-30T12:10:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb8f6a640 10 monclient(hunting): _negotiate 2023-10-30T12:10:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb8f6a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb8f6a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb8f6a640 10 monclient(hunting): my global_id is 134220 2023-10-30T12:10:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb8f6a640 1 -- 172.21.15.50:0/3575914985 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda0003520 con 0x7fddb40720b0 2023-10-30T12:10:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient(hunting): handle_auth_done global_id 114431 payload 274 2023-10-30T12:10:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 1 -- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 msgr2=0x7fddb40a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 1 -- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 msgr2=0x7fddb40a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:10:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 1 -- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 msgr2=0x7fddb40a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 1 --2- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 0x7fddb40a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddba76d640 1 --2- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 0x7fddb40a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 0x7fddb40a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 1 -- 172.21.15.50:0/3575914985 >> v1:172.21.15.50:6789/0 conn(0x7fddb40720b0 legacy=0x7fddb4097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient: _finish_hunting 0 2023-10-30T12:10:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 1 monclient: found mon.noname-b 2023-10-30T12:10:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:10:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.830+0000 7fddb9f6c640 1 -- 172.21.15.50:0/3575914985 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb4071e90 con 0x7fddb40a0080 2023-10-30T12:10:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/3575914985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb40a0080 0x7fddb40a0460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009790 tx=0x7fddb002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3a7e36d97ec2fbf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 1 -- 172.21.15.50:0/3575914985 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb00316f0 con 0x7fddb40a0080 2023-10-30T12:10:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: dump: 2023-10-30T12:10:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:15.854 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: _finish_auth 0 2023-10-30T12:10:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 1 -- 172.21.15.50:0/3575914985 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb0031850 con 0x7fddb40a0080 2023-10-30T12:10:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 1 -- 172.21.15.50:0/3575914985 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0031be0 con 0x7fddb40a0080 2023-10-30T12:10:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddbc1d0640 5 monclient: authenticate success, global_id 114431 2023-10-30T12:10:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddbc1d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddbc1d0640 4 set_mon_vals no callback set 2023-10-30T12:10:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.831+0000 7fddb8f6a640 10 monclient: dump: 2023-10-30T12:10:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:15.858 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:15.858 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:15.858 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:15.858 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:15.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 10 monclient: shutdown 2023-10-30T12:10:15.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:15.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 -- 172.21.15.50:0/3575914985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb40a0080 msgr2=0x7fddb40a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/3575914985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb40a0080 0x7fddb40a0460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009790 tx=0x7fddb002b790 comp rx=0 tx=0).stop 2023-10-30T12:10:15.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 -- 172.21.15.50:0/3575914985 shutdown_connections 2023-10-30T12:10:15.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/3575914985 >> v2:172.21.15.118:3300/0 conn(0x7fddb4073a30 0x7fddb40a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/3575914985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb40a0080 0x7fddb40a0460 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 -- 172.21.15.50:0/3575914985 >> 172.21.15.50:0/3575914985 conn(0x7fddb40a4870 msgr2=0x7fddb40a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:15.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 -- 172.21.15.50:0/3575914985 shutdown_connections 2023-10-30T12:10:15.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.832+0000 7fddbc1d0640 1 -- 172.21.15.50:0/3575914985 wait complete. 2023-10-30T12:10:15.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 10 monclient: build_initial_monmap 2023-10-30T12:10:15.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 10 monclient: monmap: 2023-10-30T12:10:15.861 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:15.861 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:15.861 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:15.861 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:15.861 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:15.862 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:15.862 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:15.862 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:15.862 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:15.862 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:15.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 librados: starting msgr at 2023-10-30T12:10:15.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 librados: starting objecter 2023-10-30T12:10:15.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 Processor -- start 2023-10-30T12:10:15.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 -- start start 2023-10-30T12:10:15.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 librados: setting wanted keys 2023-10-30T12:10:15.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 librados: calling monclient init 2023-10-30T12:10:15.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 10 monclient: init 2023-10-30T12:10:15.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:15.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:15.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 0x7fddb41c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 10 monclient(hunting): picked mon.noname-b con 0x7fddb4073a30 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:15.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb40a0080 0x7fddb41c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.833+0000 7fddbc1d0640 10 monclient(hunting): picked mon.noname-a con 0x7fddb40a0080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:15.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb4094580 0x7fddb4094a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient(hunting): picked mon.noname-c con 0x7fddb4094580 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:15.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb408fd40 con 0x7fddb40a0080 2023-10-30T12:10:15.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb408eab0 con 0x7fddb4073a30 2023-10-30T12:10:15.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:15.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb4094f50 con 0x7fddb4094580 2023-10-30T12:10:15.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:15.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient(hunting): authenticate will time out at 1461.017045s 2023-10-30T12:10:15.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb4094580 0x7fddb4094a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb40a0080 0x7fddb41c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb4094580 0x7fddb4094a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41160/0 (socket says 172.21.15.50:41160) 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): get_auth_request con 0x7fddb40a0080 auth_method 0 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 1 -- 172.21.15.50:0/2348778372 learned_addr learned my addr 172.21.15.50:0/2348778372 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 10 monclient(hunting): get_auth_request con 0x7fddb4094580 auth_method 0 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:15.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:15.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 0x7fddb41c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 10 monclient(hunting): get_auth_request con 0x7fddb4073a30 auth_method 0 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:15.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient(hunting): handle_auth_done global_id 134223 payload 824 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 -- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb4094580 msgr2=0x7fddb4094a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb4094580 0x7fddb4094a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 1 -- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 msgr2=0x7fddb41c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 -- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 msgr2=0x7fddb41c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 1 -- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 msgr2=0x7fddb41c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:10:15.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 0x7fddb41c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 0x7fddb41c69a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 0x7fddb41c69a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 monclient: found mon.noname-a 2023-10-30T12:10:15.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 -- 172.21.15.50:0/2348778372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb00093a0 con 0x7fddb40a0080 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddba76d640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb4094580 0x7fddb4094a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:15.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddb976b640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb40a0080 0x7fddb41c7420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdda400c430 tx=0x7fdda400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 client.?.objecter ms_handle_connect 0x7fddb40a0080 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 1 -- 172.21.15.50:0/2348778372 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4005ce0 con 0x7fddb40a0080 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: dump: 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.876 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:15.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:15.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:15.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:15.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:15.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 1 -- 172.21.15.50:0/2348778372 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda4005e40 con 0x7fddb40a0080 2023-10-30T12:10:15.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 5 monclient: authenticate success, global_id 134223 2023-10-30T12:10:15.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient: _renew_subs 2023-10-30T12:10:15.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:15.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb408f780 con 0x7fddb40a0080 2023-10-30T12:10:15.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:15.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 1 -- 172.21.15.50:0/2348778372 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda40072d0 con 0x7fddb40a0080 2023-10-30T12:10:15.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:15.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:15.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddaa7fc640 10 monclient: dump: 2023-10-30T12:10:15.880 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:15.880 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:15.880 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.881 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:15.881 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:15.881 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:15.881 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:15.882 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:15.882 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:15.882 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:15.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddabfff640 4 set_mon_vals no callback set 2023-10-30T12:10:15.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 client.134223.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient: _renew_subs 2023-10-30T12:10:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb408dc50 con 0x7fddb40a0080 2023-10-30T12:10:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.834+0000 7fddbc1d0640 1 librados: init done 2023-10-30T12:10:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.835+0000 7fddaa7fc640 1 -- 172.21.15.50:0/2348778372 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdda4007430 con 0x7fddb40a0080 2023-10-30T12:10:15.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.835+0000 7fddbc1d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.835+0000 7fddbc1d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.835+0000 7fddbc1d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.835+0000 7fddaa7fc640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdd980392f0 0x7fdd9803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.835+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb408ff60 con 0x7fddb40a0080 2023-10-30T12:10:15.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddaa7fc640 1 -- 172.21.15.50:0/2348778372 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fdda404a220 con 0x7fddb40a0080 2023-10-30T12:10:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddaa7fc640 10 client.134223.objecter ms_dispatch 0x7fddb4000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddaa7fc640 3 client.134223.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddaa7fc640 3 client.134223.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddaa7fc640 20 client.134223.objecter dump_active .. 0 homeless 2023-10-30T12:10:15.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdd980392f0 0x7fdd9803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:15.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddb9f6c640 10 monclient: get_auth_request con 0x7fdd980392f0 auth_method 0 2023-10-30T12:10:15.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddaa7fc640 10 client.134223.objecter ms_handle_connect 0x7fdd980392f0 2023-10-30T12:10:15.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.836+0000 7fddb9f6c640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdd980392f0 0x7fdd9803b7b0 secure :-1 s=READY pgs=4086 cs=0 l=1 rev1=1 crypto rx=0x7fddb0036000 tx=0x7fddb0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:15.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.840+0000 7fddaa7fc640 1 -- 172.21.15.50:0/2348778372 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fddb408ff60 con 0x7fddb40a0080 2023-10-30T12:10:15.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.840+0000 7fddaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:15.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:15.840+0000 7fddaa7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:16.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.089+0000 7fddbc1d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:16.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.089+0000 7fddbc1d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:16.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.089+0000 7fddbc1d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:16.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.089+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fddb408ff60 con 0x7fddb40a0080 2023-10-30T12:10:16.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.089+0000 7fddaa7fc640 1 -- 172.21.15.50:0/2348778372 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fddb408ff60 con 0x7fddb40a0080 2023-10-30T12:10:16.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.089+0000 7fddaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:16.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.089+0000 7fddaa7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:16.093 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:16.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 10 librados: watch_flush enter 2023-10-30T12:10:16.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 10 librados: watch_flush exit 2023-10-30T12:10:16.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 20 client.134223.objecter shutdown clearing up homeless session... 2023-10-30T12:10:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 10 client.134223.objecter successfully canceled tick 2023-10-30T12:10:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdd980392f0 msgr2=0x7fdd9803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdd980392f0 0x7fdd9803b7b0 secure :-1 s=READY pgs=4086 cs=0 l=1 rev1=1 crypto rx=0x7fddb0036000 tx=0x7fddb0008030 comp rx=0 tx=0).stop 2023-10-30T12:10:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 10 monclient: shutdown 2023-10-30T12:10:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:16.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb40a0080 msgr2=0x7fddb41c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:16.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb40a0080 0x7fddb41c7420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdda400c430 tx=0x7fdda400c8f0 comp rx=0 tx=0).stop 2023-10-30T12:10:16.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 shutdown_connections 2023-10-30T12:10:16.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdd980392f0 0x7fdd9803b7b0 unknown :-1 s=CLOSED pgs=4086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:16.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fddb4094580 0x7fddb4094a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:16.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fddb40a0080 0x7fddb41c7420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:16.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 --2- 172.21.15.50:0/2348778372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fddb4073a30 0x7fddb41c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:16.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 >> 172.21.15.50:0/2348778372 conn(0x7fddb40a4870 msgr2=0x7fddb41bcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:16.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 shutdown_connections 2023-10-30T12:10:16.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.091+0000 7fddbc1d0640 1 -- 172.21.15.50:0/2348778372 wait complete. 2023-10-30T12:10:16.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:16.092+0000 7fddbc1d0640 1 librados: shutdown 2023-10-30T12:10:19.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:19.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.220+0000 7ff66caf2640 10 monclient: get_monmap_and_config 2023-10-30T12:10:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.220+0000 7ff66caf2640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.220+0000 7ff66caf2640 10 monclient: monmap: 2023-10-30T12:10:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:19.221407+0000 2023-10-30T12:10:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:19.221407+0000 2023-10-30T12:10:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:19.224 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.220+0000 7ff66caf2640 1 Processor -- start 2023-10-30T12:10:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.220+0000 7ff66caf2640 1 -- start start 2023-10-30T12:10:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.220+0000 7ff66caf2640 10 monclient: init 2023-10-30T12:10:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T12:10:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff66809f950 0x7ff668097410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient(hunting): picked mon.noname-a con 0x7ff66809f950 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff668099ad0 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff668139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient(hunting): picked mon.noname-b con 0x7ff6681376e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66809fd30 con 0x7ff668099ad0 2023-10-30T12:10:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6680998e0 con 0x7ff66809f950 2023-10-30T12:10:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff668071e40 con 0x7ff6681376e0 2023-10-30T12:10:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66caf2640 10 monclient(hunting): authenticate will time out at 1464.404627s 2023-10-30T12:10:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff66809f950 0x7ff668097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff668139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff66809f950 0x7ff668097410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33950/0 (socket says 172.21.15.50:33950) 2023-10-30T12:10:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 1 -- 172.21.15.50:0/834925975 learned_addr learned my addr 172.21.15.50:0/834925975 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 10 monclient(hunting): get_auth_request con 0x7ff66809f950 auth_method 0 2023-10-30T12:10:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 10 monclient(hunting): get_auth_request con 0x7ff6681376e0 auth_method 0 2023-10-30T12:10:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.221+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 10 monclient(hunting): handle_auth_done global_id 134226 payload 274 2023-10-30T12:10:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 1 -- 172.21.15.50:0/834925975 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 msgr2=0x7ff668139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 1 --2- 172.21.15.50:0/834925975 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff668139ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 1 -- 172.21.15.50:0/834925975 >> v1:172.21.15.118:6789/0 conn(0x7ff668099ad0 legacy=0x7ff668097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 1 monclient: found mon.noname-a 2023-10-30T12:10:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 1 -- 172.21.15.50:0/834925975 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff668098a70 con 0x7ff66809f950 2023-10-30T12:10:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66659c640 1 --2- 172.21.15.50:0/834925975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668097410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff668098e80 tx=0x7ff65802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b93e68bc7bcc41 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66559a640 1 -- 172.21.15.50:0/834925975 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65802c930 con 0x7ff66809f950 2023-10-30T12:10:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.222+0000 7ff66559a640 10 monclient: dump: 2023-10-30T12:10:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.247 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 5 monclient: authenticate success, global_id 134226 2023-10-30T12:10:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 1 -- 172.21.15.50:0/834925975 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65802ca90 con 0x7ff66809f950 2023-10-30T12:10:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 1 -- 172.21.15.50:0/834925975 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65802ce20 con 0x7ff66809f950 2023-10-30T12:10:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 4 set_mon_vals no callback set 2023-10-30T12:10:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66559a640 10 monclient: dump: 2023-10-30T12:10:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.251 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 10 monclient: shutdown 2023-10-30T12:10:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 -- 172.21.15.50:0/834925975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 msgr2=0x7ff668097410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 --2- 172.21.15.50:0/834925975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668097410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff668098e80 tx=0x7ff65802bd90 comp rx=0 tx=0).stop 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 -- 172.21.15.50:0/834925975 shutdown_connections 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 --2- 172.21.15.50:0/834925975 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff668139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 --2- 172.21.15.50:0/834925975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668097410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 -- 172.21.15.50:0/834925975 >> 172.21.15.50:0/834925975 conn(0x7ff6680a0390 msgr2=0x7ff6680a07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 -- 172.21.15.50:0/834925975 shutdown_connections 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.223+0000 7ff66caf2640 1 -- 172.21.15.50:0/834925975 wait complete. 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 10 monclient: monmap: 2023-10-30T12:10:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:19.255 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 1 librados: starting msgr at 2023-10-30T12:10:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 1 librados: starting objecter 2023-10-30T12:10:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 1 Processor -- start 2023-10-30T12:10:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 1 -- start start 2023-10-30T12:10:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 1 librados: setting wanted keys 2023-10-30T12:10:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 1 librados: calling monclient init 2023-10-30T12:10:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.224+0000 7ff66caf2640 10 monclient: init 2023-10-30T12:10:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff668099ad0 0x7ff668136900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): picked mon.noname-b con 0x7ff668099ad0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): picked mon.noname-a con 0x7ff66809f950 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff66806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): picked mon.noname-c con 0x7ff6681376e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff668096300 con 0x7ff66809f950 2023-10-30T12:10:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66808b9e0 con 0x7ff668099ad0 2023-10-30T12:10:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66813be00 con 0x7ff6681376e0 2023-10-30T12:10:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66caf2640 10 monclient(hunting): authenticate will time out at 1464.408357s 2023-10-30T12:10:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668136e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33958/0 (socket says 172.21.15.50:33958) 2023-10-30T12:10:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff66806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 -- 172.21.15.50:0/2323708930 learned_addr learned my addr 172.21.15.50:0/2323708930 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:19.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): get_auth_request con 0x7ff66809f950 auth_method 0 2023-10-30T12:10:19.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 10 monclient(hunting): get_auth_request con 0x7ff6681376e0 auth_method 0 2023-10-30T12:10:19.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66659c640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff668099ad0 0x7ff668136900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66659c640 10 monclient(hunting): get_auth_request con 0x7ff668099ad0 auth_method 0 2023-10-30T12:10:19.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient(hunting): handle_auth_done global_id 134229 payload 824 2023-10-30T12:10:19.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 -- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 msgr2=0x7ff66806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff66806f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 -- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff668099ad0 msgr2=0x7ff668136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff668099ad0 0x7ff668136900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 monclient: found mon.noname-a 2023-10-30T12:10:19.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff665d9b640 1 -- 172.21.15.50:0/2323708930 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658009b30 con 0x7ff66809f950 2023-10-30T12:10:19.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff666d9d640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff66806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:19.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.225+0000 7ff66659c640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff668099ad0 0x7ff668136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:10:19.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff665d9b640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668136e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c400 tx=0x7ff65c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 client.?.objecter ms_handle_connect 0x7ff66809f950 2023-10-30T12:10:19.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:19.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 1 -- 172.21.15.50:0/2323708930 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c011790 con 0x7ff66809f950 2023-10-30T12:10:19.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:19.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: dump: 2023-10-30T12:10:19.267 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.267 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.267 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.267 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.267 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.268 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.268 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.268 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.268 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.269 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 1 -- 172.21.15.50:0/2323708930 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c011930 con 0x7ff66809f950 2023-10-30T12:10:19.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 5 monclient: authenticate success, global_id 134229 2023-10-30T12:10:19.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 10 monclient: _renew_subs 2023-10-30T12:10:19.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 1 -- 172.21.15.50:0/2323708930 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c011ce0 con 0x7ff66809f950 2023-10-30T12:10:19.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66808c640 con 0x7ff66809f950 2023-10-30T12:10:19.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:19.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff656ffd640 10 monclient: dump: 2023-10-30T12:10:19.271 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.271 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.272 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.272 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.272 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.272 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.272 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.273 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.273 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.273 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff664d99640 4 set_mon_vals no callback set 2023-10-30T12:10:19.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 10 client.134229.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:19.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 10 monclient: _renew_subs 2023-10-30T12:10:19.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff668071ae0 con 0x7ff66809f950 2023-10-30T12:10:19.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.226+0000 7ff66caf2640 1 librados: init done 2023-10-30T12:10:19.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff656ffd640 1 -- 172.21.15.50:0/2323708930 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff65c01c460 con 0x7ff66809f950 2023-10-30T12:10:19.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff66caf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff66caf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff66caf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66804e200 con 0x7ff66809f950 2023-10-30T12:10:19.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff656ffd640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6480392f0 0x7ff64803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff656ffd640 1 -- 172.21.15.50:0/2323708930 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7ff65c04aba0 con 0x7ff66809f950 2023-10-30T12:10:19.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff656ffd640 10 client.134229.objecter ms_dispatch 0x7ff668000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:19.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff656ffd640 3 client.134229.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:19.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff656ffd640 3 client.134229.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:19.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff656ffd640 20 client.134229.objecter dump_active .. 0 homeless 2023-10-30T12:10:19.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff66659c640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6480392f0 0x7ff64803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.227+0000 7ff66659c640 10 monclient: get_auth_request con 0x7ff6480392f0 auth_method 0 2023-10-30T12:10:19.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.228+0000 7ff656ffd640 10 client.134229.objecter ms_handle_connect 0x7ff6480392f0 2023-10-30T12:10:19.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.228+0000 7ff66659c640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6480392f0 0x7ff64803b7b0 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 crypto rx=0x7ff658035000 tx=0x7ff658030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.232+0000 7ff656ffd640 1 -- 172.21.15.50:0/2323708930 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff65c011e40 con 0x7ff66809f950 2023-10-30T12:10:19.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.232+0000 7ff656ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.232+0000 7ff656ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:19.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.474+0000 7ff66caf2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:19.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.474+0000 7ff66caf2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:19.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.474+0000 7ff66caf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.474+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff668071760 con 0x7ff66809f950 2023-10-30T12:10:19.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.474+0000 7ff656ffd640 1 -- 172.21.15.50:0/2323708930 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7ff65c0102e0 con 0x7ff66809f950 2023-10-30T12:10:19.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.474+0000 7ff656ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:19.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.474+0000 7ff656ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:19.477 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 10 librados: watch_flush enter 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 10 librados: watch_flush exit 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 20 client.134229.objecter shutdown clearing up homeless session... 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 10 client.134229.objecter successfully canceled tick 2023-10-30T12:10:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6480392f0 msgr2=0x7ff64803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6480392f0 0x7ff64803b7b0 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 crypto rx=0x7ff658035000 tx=0x7ff658030000 comp rx=0 tx=0).stop 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 10 monclient: shutdown 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 msgr2=0x7ff668136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668136e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c400 tx=0x7ff65c00c8c0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 shutdown_connections 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff6480392f0 0x7ff64803b7b0 unknown :-1 s=CLOSED pgs=4087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff6681376e0 0x7ff66806f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66809f950 0x7ff668136e40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 --2- 172.21.15.50:0/2323708930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff668099ad0 0x7ff668136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 >> 172.21.15.50:0/2323708930 conn(0x7ff6680a0390 msgr2=0x7ff66809f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 shutdown_connections 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 -- 172.21.15.50:0/2323708930 wait complete. 2023-10-30T12:10:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.476+0000 7ff66caf2640 1 librados: shutdown 2023-10-30T12:10:19.488 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:10:19.488 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:10:19.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:10:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.593+0000 7fb5e2db6640 10 monclient: get_monmap_and_config 2023-10-30T12:10:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.593+0000 7fb5e2db6640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.593+0000 7fb5e2db6640 10 monclient: monmap: 2023-10-30T12:10:19.596 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:19.596 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.596 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:19.594440+0000 2023-10-30T12:10:19.596 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:19.594440+0000 2023-10-30T12:10:19.597 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.597 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.597 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:19.597 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:19.597 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:19.598 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 1 Processor -- start 2023-10-30T12:10:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 1 -- start start 2023-10-30T12:10:19.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient: init 2023-10-30T12:10:19.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-10-30T12:10:19.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5dc071c00 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:19.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc097fe0 0x7fb5dc09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5dc097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:19.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb5dc09b170 0x7fb5dc139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5dc09b170 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:19.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5dc097600 con 0x7fb5dc071c00 2023-10-30T12:10:19.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc0978e0 con 0x7fb5dc09b170 2023-10-30T12:10:19.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc09bf50 con 0x7fb5dc097fe0 2023-10-30T12:10:19.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e2db6640 10 monclient(hunting): authenticate will time out at 1464.777575s 2023-10-30T12:10:19.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc097fe0 0x7fb5dc09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb5dc09b170 0x7fb5dc139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc097fe0 0x7fb5dc09a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52790/0 (socket says 172.21.15.50:52790) 2023-10-30T12:10:19.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb5dc09b170 0x7fb5dc139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33970/0 (socket says 172.21.15.50:33970) 2023-10-30T12:10:19.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 1 -- 172.21.15.50:0/3062456663 learned_addr learned my addr 172.21.15.50:0/3062456663 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:19.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request con 0x7fb5dc097fe0 auth_method 0 2023-10-30T12:10:19.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 10 monclient(hunting): get_auth_request con 0x7fb5dc09b170 auth_method 0 2023-10-30T12:10:19.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.594+0000 7fb5e1353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_done global_id 114443 payload 274 2023-10-30T12:10:19.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 1 -- 172.21.15.50:0/3062456663 >> v2:172.21.15.50:3300/0 conn(0x7fb5dc09b170 msgr2=0x7fb5dc139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:19.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 1 --2- 172.21.15.50:0/3062456663 >> v2:172.21.15.50:3300/0 conn(0x7fb5dc09b170 0x7fb5dc139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 1 -- 172.21.15.50:0/3062456663 >> v1:172.21.15.118:6789/0 conn(0x7fb5dc071c00 legacy=0x7fb5dc09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 1 monclient: found mon.noname-b 2023-10-30T12:10:19.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:10:19.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 1 -- 172.21.15.50:0/3062456663 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc09d9b0 con 0x7fb5dc097fe0 2023-10-30T12:10:19.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5e1353640 1 --2- 172.21.15.50:0/3062456663 >> v2:172.21.15.50:3300/0 conn(0x7fb5dc09b170 0x7fb5dc139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:19.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5dbfff640 1 --2- 172.21.15.50:0/3062456663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc097fe0 0x7fb5dc09a150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009870 tx=0x7fb5cc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a20adb8ae57e08f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 1 -- 172.21.15.50:0/3062456663 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc0317c0 con 0x7fb5dc097fe0 2023-10-30T12:10:19.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:19.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: dump: 2023-10-30T12:10:19.618 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.618 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.619 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.619 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.619 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.619 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.619 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.620 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.620 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.620 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 1 -- 172.21.15.50:0/3062456663 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc031920 con 0x7fb5dc097fe0 2023-10-30T12:10:19.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5e2db6640 5 monclient: authenticate success, global_id 114443 2023-10-30T12:10:19.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5e2db6640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:19.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5e2db6640 4 set_mon_vals no callback set 2023-10-30T12:10:19.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 1 -- 172.21.15.50:0/3062456663 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc031cb0 con 0x7fb5dc097fe0 2023-10-30T12:10:19.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:19.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.595+0000 7fb5db7fe640 10 monclient: dump: 2023-10-30T12:10:19.623 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.623 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.623 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.623 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.624 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.624 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.624 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.624 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.624 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.625 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 10 monclient: shutdown 2023-10-30T12:10:19.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 -- 172.21.15.50:0/3062456663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc097fe0 msgr2=0x7fb5dc09a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/3062456663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc097fe0 0x7fb5dc09a150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009870 tx=0x7fb5cc02b860 comp rx=0 tx=0).stop 2023-10-30T12:10:19.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 -- 172.21.15.50:0/3062456663 shutdown_connections 2023-10-30T12:10:19.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/3062456663 >> v2:172.21.15.50:3300/0 conn(0x7fb5dc09b170 0x7fb5dc139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/3062456663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc097fe0 0x7fb5dc09a150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 -- 172.21.15.50:0/3062456663 >> 172.21.15.50:0/3062456663 conn(0x7fb5dc0a3980 msgr2=0x7fb5dc0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 -- 172.21.15.50:0/3062456663 shutdown_connections 2023-10-30T12:10:19.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.596+0000 7fb5e2db6640 1 -- 172.21.15.50:0/3062456663 wait complete. 2023-10-30T12:10:19.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 10 monclient: monmap: 2023-10-30T12:10:19.627 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:19.627 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:19.628 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:19.628 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:19.628 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.628 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.628 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:19.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:19.629 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:19.629 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 1 librados: starting msgr at 2023-10-30T12:10:19.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 1 librados: starting objecter 2023-10-30T12:10:19.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 1 Processor -- start 2023-10-30T12:10:19.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 1 -- start start 2023-10-30T12:10:19.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 1 librados: setting wanted keys 2023-10-30T12:10:19.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 1 librados: calling monclient init 2023-10-30T12:10:19.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.597+0000 7fb5e2db6640 10 monclient: init 2023-10-30T12:10:19.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:19.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc071c00 0x7fb5dc06d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5dc071c00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5dc097fe0 0x7fb5dc071260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5dc097fe0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc09b170 0x7fb5dc1c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): picked mon.noname-c con 0x7fb5dc09b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc06f770 con 0x7fb5dc071c00 2023-10-30T12:10:19.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc06da10 con 0x7fb5dc097fe0 2023-10-30T12:10:19.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc09cbc0 con 0x7fb5dc09b170 2023-10-30T12:10:19.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e2db6640 10 monclient(hunting): authenticate will time out at 1464.781226s 2023-10-30T12:10:19.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc071c00 0x7fb5dc06d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc09b170 0x7fb5dc1c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc071c00 0x7fb5dc06d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33984/0 (socket says 172.21.15.50:33984) 2023-10-30T12:10:19.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 10 monclient(hunting): get_auth_request con 0x7fb5dc09b170 auth_method 0 2023-10-30T12:10:19.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 -- 172.21.15.50:0/521371990 learned_addr learned my addr 172.21.15.50:0/521371990 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:19.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): get_auth_request con 0x7fb5dc071c00 auth_method 0 2023-10-30T12:10:19.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5dc097fe0 0x7fb5dc071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request con 0x7fb5dc097fe0 auth_method 0 2023-10-30T12:10:19.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e1353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient(hunting): handle_auth_done global_id 134235 payload 824 2023-10-30T12:10:19.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc09b170 msgr2=0x7fb5dc1c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5dc09b170 0x7fb5dc1c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5dc097fe0 msgr2=0x7fb5dc071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5dc097fe0 0x7fb5dc071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 monclient: found mon.noname-a 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 -- 172.21.15.50:0/521371990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d400ab80 con 0x7fb5dc071c00 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5e0b52640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc071c00 0x7fb5dc06d090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400c3f0 tx=0x7fb5c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5d97fa640 10 client.?.objecter ms_handle_connect 0x7fb5dc071c00 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.598+0000 7fb5d97fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c40052d0 con 0x7fb5dc071c00 2023-10-30T12:10:19.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: dump: 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.644 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.645 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.646 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c4005470 con 0x7fb5dc071c00 2023-10-30T12:10:19.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 5 monclient: authenticate success, global_id 134235 2023-10-30T12:10:19.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4006a60 con 0x7fb5dc071c00 2023-10-30T12:10:19.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:19.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5d97fa640 10 monclient: dump: 2023-10-30T12:10:19.648 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.648 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.648 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.648 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.650 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 10 monclient: _renew_subs 2023-10-30T12:10:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5daffd640 4 set_mon_vals no callback set 2023-10-30T12:10:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc0094d0 con 0x7fb5dc071c00 2023-10-30T12:10:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 10 client.134235.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 10 monclient: _renew_subs 2023-10-30T12:10:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc0717a0 con 0x7fb5dc071c00 2023-10-30T12:10:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.600+0000 7fb5e2db6640 1 librados: init done 2023-10-30T12:10:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb5c4006bc0 con 0x7fb5dc071c00 2023-10-30T12:10:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5e2db6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1c8b30 0x7fb5dc1c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5e2db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc073ff0 con 0x7fb5dc1c8b30 2023-10-30T12:10:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1c8b30 0x7fb5dc1c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb5bc0392f0 0x7fb5bc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient: get_auth_request con 0x7fb5dc1c8b30 auth_method 0 2023-10-30T12:10:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fb5c4049fc0 con 0x7fb5dc071c00 2023-10-30T12:10:19.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 10 client.134235.objecter ms_dispatch 0x7fb5dc000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 3 client.134235.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 3 client.134235.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 20 client.134235.objecter dump_active .. 0 homeless 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5e1353640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb5bc0392f0 0x7fb5bc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5e1353640 10 monclient: get_auth_request con 0x7fb5bc0392f0 auth_method 0 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_done global_id 134238 payload 824 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 1 -- 172.21.15.50:0/521371990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5dc071fe0 con 0x7fb5dc1c8b30 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 10 client.134235.objecter ms_handle_connect 0x7fb5dc1c8b30 2023-10-30T12:10:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5dbfff640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1c8b30 0x7fb5dc1c5640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc09fdf0 tx=0x7fb5cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 10 client.134235.objecter resend_mon_ops 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc032860 con 0x7fb5dc1c8b30 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.601+0000 7fb5d97fa640 10 monclient: dump: 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 10 client.134235.objecter ms_handle_connect 0x7fb5bc0392f0 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5e1353640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb5bc0392f0 0x7fb5bc03b7b0 secure :-1 s=READY pgs=4088 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc06d600 tx=0x7fb5d400a860 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb5cc0329c0 con 0x7fb5dc1c8b30 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:19.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1c8b30 msgr2=0x7fb5dc1c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1c8b30 0x7fb5dc1c5640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc09fdf0 tx=0x7fb5cc004730 comp rx=0 tx=0).stop 2023-10-30T12:10:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1c8b30 msgr2=0x7fb5dc1c5640 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.602+0000 7fb5d97fa640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc1c8b30 0x7fb5dc1c5640 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc09fdf0 tx=0x7fb5cc004730 comp rx=0 tx=0).stop 2023-10-30T12:10:19.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5e2db6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:19.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc02b430 0x7fb5dc02b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5e2db6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc06fbe0 con 0x7fb5dc02b430 2023-10-30T12:10:19.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc02b430 0x7fb5dc02b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient: get_auth_request con 0x7fb5dc02b430 auth_method 0 2023-10-30T12:10:19.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_done global_id 134241 payload 824 2023-10-30T12:10:19.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.608+0000 7fb5dbfff640 1 -- 172.21.15.50:0/521371990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb5dc0a1560 con 0x7fb5dc02b430 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5dbfff640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc02b430 0x7fb5dc02b840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009870 tx=0x7fb5cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 10 client.134235.objecter ms_handle_connect 0x7fb5dc02b430 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 10 client.134235.objecter resend_mon_ops 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc04fbc0 con 0x7fb5dc02b430 2023-10-30T12:10:19.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 10 monclient: dump: 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.669 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4803 (secure 0 0 0) 0x7fb5cc059410 con 0x7fb5dc02b430 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc02b430 msgr2=0x7fb5dc02b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc02b430 0x7fb5dc02b840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009870 tx=0x7fb5cc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc02b430 msgr2=0x7fb5dc02b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:19.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5d97fa640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc02b430 0x7fb5dc02b840 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009870 tx=0x7fb5cc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.672 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:19.672 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 94, 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:19.673 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:10:19.674 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:19.676 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:19.677 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:19.677 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:19.677 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.677 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:19.677 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:19.678 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:19.679 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:19.680 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:19.681 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:19.682 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:19.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:19.684 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:19.684 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:19.684 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:19.684 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:19.684 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:19.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:19.686 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:19.687 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:19.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.687 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:19.687 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:19.687 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:19.688 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.689 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.689 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:19.689 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:19.690 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:19.690 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:19.690 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:19.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.690 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:19.691 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:19.691 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:19.691 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:19.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.691 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:19.692 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:19.692 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.692 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:19.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:19.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.697 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.698 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:19.698 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:19.698 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:19.698 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:19.699 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:19.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.700 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:19.700 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:19.700 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:19.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:19.701 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:19.702 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:10:19.702 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.702 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.702 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:19.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.703 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:19.704 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.705 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:19.705 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 10 librados: watch_flush enter 2023-10-30T12:10:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 10 librados: watch_flush exit 2023-10-30T12:10:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 20 client.134235.objecter shutdown clearing up homeless session... 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 10 client.134235.objecter successfully canceled tick 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb5bc0392f0 msgr2=0x7fb5bc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb5bc0392f0 0x7fb5bc03b7b0 secure :-1 s=READY pgs=4088 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc06d600 tx=0x7fb5d400a860 comp rx=0 tx=0).stop 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 10 monclient: shutdown 2023-10-30T12:10:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc071c00 msgr2=0x7fb5dc06d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e1353640 1 -- 172.21.15.50:0/521371990 reap_dead start 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc071c00 0x7fb5dc06d090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400c3f0 tx=0x7fb5c400c8b0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 shutdown_connections 2023-10-30T12:10:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 --2- 172.21.15.50:0/521371990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5dc071c00 0x7fb5dc06d090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 >> 172.21.15.50:0/521371990 conn(0x7fb5dc0a3980 msgr2=0x7fb5dc139360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 shutdown_connections 2023-10-30T12:10:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 -- 172.21.15.50:0/521371990 wait complete. 2023-10-30T12:10:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.609+0000 7fb5e2db6640 1 librados: shutdown 2023-10-30T12:10:19.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:10:19.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:10:19.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:10:19.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.816+0000 7f031231d640 10 monclient: get_monmap_and_config 2023-10-30T12:10:19.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.816+0000 7f031231d640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.816+0000 7f031231d640 10 monclient: monmap: 2023-10-30T12:10:19.818 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:19.819 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.819 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:19.817718+0000 2023-10-30T12:10:19.819 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:19.817718+0000 2023-10-30T12:10:19.819 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.819 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.819 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:19.820 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:19.820 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:19.820 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:19.820 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:19.820 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 1 Processor -- start 2023-10-30T12:10:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 1 -- start start 2023-10-30T12:10:19.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient: init 2023-10-30T12:10:19.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-10-30T12:10:19.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f030c132970 0x7f030c132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient(hunting): picked mon.noname-c con 0x7f030c132970 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:19.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f030c0a03d0 0x7f030c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient(hunting): picked mon.noname-a con 0x7f030c0a03d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:19.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f030c070d10 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:19.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c0a17d0 con 0x7f030c070d10 2023-10-30T12:10:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c0a1510 con 0x7f030c0a03d0 2023-10-30T12:10:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c09f680 con 0x7f030c132970 2023-10-30T12:10:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.817+0000 7f031231d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f031231d640 10 monclient(hunting): authenticate will time out at 1465.001015s 2023-10-30T12:10:19.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f030c0a03d0 0x7f030c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f030c0a03d0 0x7f030c0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34002/0 (socket says 172.21.15.50:34002) 2023-10-30T12:10:19.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 -- 172.21.15.50:0/221565563 learned_addr learned my addr 172.21.15.50:0/221565563 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:19.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): get_auth_request con 0x7f030c0a03d0 auth_method 0 2023-10-30T12:10:19.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 1 --2- 172.21.15.50:0/221565563 >> v2:172.21.15.118:3300/0 conn(0x7f030c132970 0x7f030c132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 10 monclient(hunting): get_auth_request con 0x7f030c132970 auth_method 0 2023-10-30T12:10:19.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient(hunting): handle_auth_done global_id 134244 payload 274 2023-10-30T12:10:19.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 -- 172.21.15.50:0/221565563 >> v2:172.21.15.118:3300/0 conn(0x7f030c132970 msgr2=0x7f030c132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:19.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 --2- 172.21.15.50:0/221565563 >> v2:172.21.15.118:3300/0 conn(0x7f030c132970 0x7f030c132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 -- 172.21.15.50:0/221565563 >> v1:172.21.15.118:6789/0 conn(0x7f030c070d10 legacy=0x7f030c13caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:10:19.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.818+0000 7f030b7fe640 1 -- 172.21.15.50:0/221565563 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c09f960 con 0x7f030c0a03d0 2023-10-30T12:10:19.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030b7fe640 1 --2- 172.21.15.50:0/221565563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c0a03d0 0x7f030c0707d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f030c09aef0 tx=0x7f02fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271a01af1c61b4fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 1 -- 172.21.15.50:0/221565563 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0317e0 con 0x7f030c0a03d0 2023-10-30T12:10:19.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:19.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: dump: 2023-10-30T12:10:19.839 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.839 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.839 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.839 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.839 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.840 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.840 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.840 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.840 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.840 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 5 monclient: authenticate success, global_id 134244 2023-10-30T12:10:19.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 1 -- 172.21.15.50:0/221565563 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc031940 con 0x7f030c0a03d0 2023-10-30T12:10:19.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:19.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:19.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 4 set_mon_vals no callback set 2023-10-30T12:10:19.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 1 -- 172.21.15.50:0/221565563 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc031cd0 con 0x7f030c0a03d0 2023-10-30T12:10:19.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:19.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f030affd640 10 monclient: dump: 2023-10-30T12:10:19.843 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.843 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.843 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.843 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.844 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.844 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.844 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.844 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.844 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.844 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 10 monclient: shutdown 2023-10-30T12:10:19.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 -- 172.21.15.50:0/221565563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c0a03d0 msgr2=0x7f030c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 --2- 172.21.15.50:0/221565563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c0a03d0 0x7f030c0707d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f030c09aef0 tx=0x7f02fc02b880 comp rx=0 tx=0).stop 2023-10-30T12:10:19.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 -- 172.21.15.50:0/221565563 shutdown_connections 2023-10-30T12:10:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 --2- 172.21.15.50:0/221565563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c0a03d0 0x7f030c0707d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 --2- 172.21.15.50:0/221565563 >> v2:172.21.15.118:3300/0 conn(0x7f030c132970 0x7f030c132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 -- 172.21.15.50:0/221565563 >> 172.21.15.50:0/221565563 conn(0x7f030c0a4800 msgr2=0x7f030c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 -- 172.21.15.50:0/221565563 shutdown_connections 2023-10-30T12:10:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.819+0000 7f031231d640 1 -- 172.21.15.50:0/221565563 wait complete. 2023-10-30T12:10:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.820+0000 7f031231d640 10 monclient: build_initial_monmap 2023-10-30T12:10:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.820+0000 7f031231d640 10 monclient: monmap: 2023-10-30T12:10:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:19.849 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.820+0000 7f031231d640 1 librados: starting msgr at 2023-10-30T12:10:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.820+0000 7f031231d640 1 librados: starting objecter 2023-10-30T12:10:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.820+0000 7f031231d640 1 Processor -- start 2023-10-30T12:10:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 -- start start 2023-10-30T12:10:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 librados: setting wanted keys 2023-10-30T12:10:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 librados: calling monclient init 2023-10-30T12:10:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient: init 2023-10-30T12:10:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c070d10 0x7f030c1ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): picked mon.noname-c con 0x7f030c070d10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:19.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c132970 0x7f030c1cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): picked mon.noname-a con 0x7f030c132970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:19.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f030c11dcd0 0x7f030c11e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): picked mon.noname-b con 0x7f030c11dcd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:19.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c1211f0 con 0x7f030c132970 2023-10-30T12:10:19.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c11fc40 con 0x7f030c11dcd0 2023-10-30T12:10:19.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0a3050 con 0x7f030c070d10 2023-10-30T12:10:19.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:19.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f031231d640 10 monclient(hunting): authenticate will time out at 1465.004541s 2023-10-30T12:10:19.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c070d10 0x7f030c1ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c132970 0x7f030c1cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c070d10 0x7f030c1ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52800/0 (socket says 172.21.15.50:52800) 2023-10-30T12:10:19.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 1 -- 172.21.15.50:0/236817629 learned_addr learned my addr 172.21.15.50:0/236817629 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:19.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 10 monclient(hunting): get_auth_request con 0x7f030c070d10 auth_method 0 2023-10-30T12:10:19.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c132970 0x7f030c1cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34012/0 (socket says 172.21.15.50:34012) 2023-10-30T12:10:19.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): get_auth_request con 0x7f030c132970 auth_method 0 2023-10-30T12:10:19.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f03108ba640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f030c11dcd0 0x7f030c11e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f03108ba640 10 monclient(hunting): get_auth_request con 0x7f030c11dcd0 auth_method 0 2023-10-30T12:10:19.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f03108ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f03108ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f03108ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient(hunting): handle_auth_done global_id 134247 payload 824 2023-10-30T12:10:19.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c070d10 msgr2=0x7f030c1ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c070d10 0x7f030c1ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f030c11dcd0 msgr2=0x7f030c11e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f030c11dcd0 0x7f030c11e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:10:19.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:10:19.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:19.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030b7fe640 1 -- 172.21.15.50:0/236817629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc0094d0 con 0x7f030c132970 2023-10-30T12:10:19.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.821+0000 7f030bfff640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c070d10 0x7f030c1ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:19.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f030b7fe640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c132970 0x7f030c1cadf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02fc033000 tx=0x7f02fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 client.?.objecter ms_handle_connect 0x7f030c132970 2023-10-30T12:10:19.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:19.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc032880 con 0x7f030c132970 2023-10-30T12:10:19.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:19.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: dump: 2023-10-30T12:10:19.861 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.861 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.861 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.861 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.861 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.861 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.862 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.862 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.862 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.862 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:19.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:19.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:19.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc032a00 con 0x7f030c132970 2023-10-30T12:10:19.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 5 monclient: authenticate success, global_id 134247 2023-10-30T12:10:19.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 10 monclient: _renew_subs 2023-10-30T12:10:19.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c09d7c0 con 0x7f030c132970 2023-10-30T12:10:19.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:19.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc032db0 con 0x7f030c132970 2023-10-30T12:10:19.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:19.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f0308ff9640 10 monclient: dump: 2023-10-30T12:10:19.865 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.865 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.865 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.865 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.865 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.866 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.866 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.866 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.866 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.866 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 10 client.134247.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 10 monclient: _renew_subs 2023-10-30T12:10:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f030a7fc640 4 set_mon_vals no callback set 2023-10-30T12:10:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c11f6c0 con 0x7f030c132970 2023-10-30T12:10:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.822+0000 7f031231d640 1 librados: init done 2023-10-30T12:10:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f031231d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f02fc04e450 con 0x7f030c132970 2023-10-30T12:10:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f031231d640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c0a03d0 0x7f030c11e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f031231d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c071a70 con 0x7f030c0a03d0 2023-10-30T12:10:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c0a03d0 0x7f030c11e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient: get_auth_request con 0x7f030c0a03d0 auth_method 0 2023-10-30T12:10:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02f8039320 0x7f02f803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f02fc06cc20 con 0x7f030c132970 2023-10-30T12:10:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 10 client.134247.objecter ms_dispatch 0x7f030c000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 3 client.134247.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 3 client.134247.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 20 client.134247.objecter dump_active .. 0 homeless 2023-10-30T12:10:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 10 monclient(hunting): handle_auth_done global_id 114452 payload 824 2023-10-30T12:10:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 1 -- 172.21.15.50:0/236817629 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f030c133290 con 0x7f030c0a03d0 2023-10-30T12:10:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f03108ba640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02f8039320 0x7f02f803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f03108ba640 10 monclient: get_auth_request con 0x7f02f8039320 auth_method 0 2023-10-30T12:10:19.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 10 client.134247.objecter ms_handle_connect 0x7f030c0a03d0 2023-10-30T12:10:19.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 10 client.134247.objecter resend_mon_ops 2023-10-30T12:10:19.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f030bfff640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c0a03d0 0x7f030c11e6a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0300011db0 tx=0x7f030000f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300004030 con 0x7f030c0a03d0 2023-10-30T12:10:19.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:19.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.823+0000 7f0308ff9640 10 monclient: dump: 2023-10-30T12:10:19.873 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.873 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.873 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.873 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.875 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 10 client.134247.objecter ms_handle_connect 0x7f02f8039320 2023-10-30T12:10:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f03108ba640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02f8039320 0x7f02f803b7e0 secure :-1 s=READY pgs=4089 cs=0 l=1 rev1=1 crypto rx=0x7f0304007fd0 tx=0x7f0304009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0300004190 con 0x7f030c0a03d0 2023-10-30T12:10:19.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c0a03d0 msgr2=0x7f030c11e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c0a03d0 0x7f030c11e6a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0300011db0 tx=0x7f030000f610 comp rx=0 tx=0).stop 2023-10-30T12:10:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c0a03d0 msgr2=0x7f030c11e6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:19.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.824+0000 7f0308ff9640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c0a03d0 0x7f030c11e6a0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0300011db0 tx=0x7f030000f610 comp rx=0 tx=0).stop 2023-10-30T12:10:19.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f031231d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:19.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f031231d640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c11ebe0 0x7f030c11a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:19.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f031231d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:19.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c074800 con 0x7f030c11ebe0 2023-10-30T12:10:19.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c11ebe0 0x7f030c11a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:19.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient: get_auth_request con 0x7f030c11ebe0 auth_method 0 2023-10-30T12:10:19.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:19.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:19.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:19.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:19.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:19.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:19.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 10 monclient(hunting): handle_auth_done global_id 114455 payload 824 2023-10-30T12:10:19.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 1 -- 172.21.15.50:0/236817629 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f030c121710 con 0x7f030c11ebe0 2023-10-30T12:10:19.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f030bfff640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c11ebe0 0x7f030c11a430 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f030001b000 tx=0x7f03000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:19.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f0308ff9640 10 client.134247.objecter ms_handle_connect 0x7f030c11ebe0 2023-10-30T12:10:19.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f0308ff9640 10 client.134247.objecter resend_mon_ops 2023-10-30T12:10:19.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300014910 con 0x7f030c11ebe0 2023-10-30T12:10:19.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.830+0000 7f0308ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:19.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:19.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 10 monclient: dump: 2023-10-30T12:10:19.882 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:19.882 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:19.883 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.883 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:19.883 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:19.883 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:19.883 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:19.884 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:19.884 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:19.884 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:19.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f03000133a0 con 0x7f030c11ebe0 2023-10-30T12:10:19.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:19.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:19.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c11ebe0 msgr2=0x7f030c11a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c11ebe0 0x7f030c11a430 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f030001b000 tx=0x7f03000046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c11ebe0 msgr2=0x7f030c11a430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:19.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f0308ff9640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f030c11ebe0 0x7f030c11a430 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f030001b000 tx=0x7f03000046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.886 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:19.886 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:19.886 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:19.886 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:10:19.886 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 94, 2023-10-30T12:10:19.887 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:19.887 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:19.887 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:10:19.887 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:19.887 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.887 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:10:19.888 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:19.888 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:19.888 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:19.888 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:19.888 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:19.889 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:19.889 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:19.889 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:19.889 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:19.889 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:19.890 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:19.890 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:19.890 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:19.890 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.890 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:19.890 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:19.891 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:19.891 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:19.891 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:19.891 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:19.891 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:19.892 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:19.892 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:19.892 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:19.892 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:19.892 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:19.892 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.893 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:19.893 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:19.893 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:19.893 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:19.894 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:19.894 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:19.894 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:19.894 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:19.894 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:19.895 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:19.895 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:19.895 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:19.895 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:19.895 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:19.895 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:19.896 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:19.896 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:19.896 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:19.896 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:19.896 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:19.897 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:19.897 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:19.897 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:19.897 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:19.897 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:19.897 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:19.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.903 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:19.903 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:19.903 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:19.903 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:19.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:19.904 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:19.904 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.905 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:19.905 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:19.905 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.905 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.905 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.906 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:19.906 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:19.906 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:19.906 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:19.907 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:19.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.907 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:19.907 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:19.907 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:19.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:19.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:19.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:19.908 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:19.909 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.910 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.910 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:19.910 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:19.910 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:19.911 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:19.911 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:19.911 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.911 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.912 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:19.912 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:19.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.912 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:19.912 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:19.912 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:19.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.913 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.913 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:10:19.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.913 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:19.914 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:19.914 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:10:19.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.914 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:19.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:19.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:19.915 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:19.915 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:19.915 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:10:19.915 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:19.915 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:19.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:19.916 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:19.916 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 10 librados: watch_flush enter 2023-10-30T12:10:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 10 librados: watch_flush exit 2023-10-30T12:10:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 20 client.134247.objecter shutdown clearing up homeless session... 2023-10-30T12:10:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 10 client.134247.objecter successfully canceled tick 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02f8039320 msgr2=0x7f02f803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02f8039320 0x7f02f803b7e0 secure :-1 s=READY pgs=4089 cs=0 l=1 rev1=1 crypto rx=0x7f0304007fd0 tx=0x7f0304009040 comp rx=0 tx=0).stop 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 10 monclient: shutdown 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c132970 msgr2=0x7f030c1cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 --2- 172.21.15.50:0/236817629 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f030c132970 0x7f030c1cadf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f02fc033000 tx=0x7f02fc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:19.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f03108ba640 1 -- 172.21.15.50:0/236817629 reap_dead start 2023-10-30T12:10:19.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f03108ba640 1 -- 172.21.15.50:0/236817629 reap_dead start 2023-10-30T12:10:19.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 shutdown_connections 2023-10-30T12:10:19.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 >> 172.21.15.50:0/236817629 conn(0x7f030c0a4800 msgr2=0x7f030c1307a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:19.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 shutdown_connections 2023-10-30T12:10:19.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 -- 172.21.15.50:0/236817629 wait complete. 2023-10-30T12:10:19.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:19.831+0000 7f031231d640 1 librados: shutdown 2023-10-30T12:10:19.921 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:10:19.921 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:10:19.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:10:20.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.026+0000 7f648f324640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.026+0000 7f648f324640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.026+0000 7f648f324640 10 monclient: monmap: 2023-10-30T12:10:20.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:20.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.028 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:20.027379+0000 2023-10-30T12:10:20.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:20.027379+0000 2023-10-30T12:10:20.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.029 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:20.030 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:20.030 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:20.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.026+0000 7f648f324640 1 Processor -- start 2023-10-30T12:10:20.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 1 -- start start 2023-10-30T12:10:20.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient: init 2023-10-30T12:10:20.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-10-30T12:10:20.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488097780 0x7f6488097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient(hunting): picked mon.noname-b con 0x7f6488097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:20.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6488073b10 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f648809ccd0 0x7f64880a0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient(hunting): picked mon.noname-c con 0x7f648809ccd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6488071ec0 con 0x7f6488073b10 2023-10-30T12:10:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648809bd10 con 0x7f648809ccd0 2023-10-30T12:10:20.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648809bff0 con 0x7f6488097780 2023-10-30T12:10:20.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648f324640 10 monclient(hunting): authenticate will time out at 1465.210535s 2023-10-30T12:10:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488097780 0x7f6488097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488097780 0x7f6488097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52834/0 (socket says 172.21.15.50:52834) 2023-10-30T12:10:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 -- 172.21.15.50:0/2651749054 learned_addr learned my addr 172.21.15.50:0/2651749054 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): get_auth_request con 0x7f6488097780 auth_method 0 2023-10-30T12:10:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d8c1640 1 --2- 172.21.15.50:0/2651749054 >> v2:172.21.15.118:3300/0 conn(0x7f648809ccd0 0x7f64880a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d8c1640 10 monclient(hunting): get_auth_request con 0x7f648809ccd0 auth_method 0 2023-10-30T12:10:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d8c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d8c1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d8c1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient(hunting): handle_auth_done global_id 114458 payload 274 2023-10-30T12:10:20.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 -- 172.21.15.50:0/2651749054 >> v2:172.21.15.118:3300/0 conn(0x7f648809ccd0 msgr2=0x7f64880a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 --2- 172.21.15.50:0/2651749054 >> v2:172.21.15.118:3300/0 conn(0x7f648809ccd0 0x7f64880a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 -- 172.21.15.50:0/2651749054 >> v1:172.21.15.118:6789/0 conn(0x7f6488073b10 legacy=0x7f64880a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d0c0640 1 -- 172.21.15.50:0/2651749054 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64880980a0 con 0x7f6488097780 2023-10-30T12:10:20.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.027+0000 7f648d8c1640 1 --2- 172.21.15.50:0/2651749054 >> v2:172.21.15.118:3300/0 conn(0x7f648809ccd0 0x7f64880a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:10:20.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f648d0c0640 1 --2- 172.21.15.50:0/2651749054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488097780 0x7f6488097b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6484009870 tx=0x7f648402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0afe7811653b74f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 1 -- 172.21.15.50:0/2651749054 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64840317c0 con 0x7f6488097780 2023-10-30T12:10:20.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: dump: 2023-10-30T12:10:20.051 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.051 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.051 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.051 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.051 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.052 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f648f324640 5 monclient: authenticate success, global_id 114458 2023-10-30T12:10:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 1 -- 172.21.15.50:0/2651749054 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484031920 con 0x7f6488097780 2023-10-30T12:10:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f648f324640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f648f324640 4 set_mon_vals no callback set 2023-10-30T12:10:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 1 -- 172.21.15.50:0/2651749054 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484031cb0 con 0x7f6488097780 2023-10-30T12:10:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.028+0000 7f647bfff640 10 monclient: dump: 2023-10-30T12:10:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.057 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 10 monclient: shutdown 2023-10-30T12:10:20.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 -- 172.21.15.50:0/2651749054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488097780 msgr2=0x7f6488097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 --2- 172.21.15.50:0/2651749054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488097780 0x7f6488097b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6484009870 tx=0x7f648402b860 comp rx=0 tx=0).stop 2023-10-30T12:10:20.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 -- 172.21.15.50:0/2651749054 shutdown_connections 2023-10-30T12:10:20.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 --2- 172.21.15.50:0/2651749054 >> v2:172.21.15.118:3300/0 conn(0x7f648809ccd0 0x7f64880a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 --2- 172.21.15.50:0/2651749054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488097780 0x7f6488097b60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 -- 172.21.15.50:0/2651749054 >> 172.21.15.50:0/2651749054 conn(0x7f64880a1170 msgr2=0x7f64880a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 -- 172.21.15.50:0/2651749054 shutdown_connections 2023-10-30T12:10:20.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.029+0000 7f648f324640 1 -- 172.21.15.50:0/2651749054 wait complete. 2023-10-30T12:10:20.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient: monmap: 2023-10-30T12:10:20.059 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:20.060 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.060 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:20.060 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:20.060 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.060 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.061 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:20.061 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:20.061 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:20.061 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 librados: starting msgr at 2023-10-30T12:10:20.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 librados: starting objecter 2023-10-30T12:10:20.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 Processor -- start 2023-10-30T12:10:20.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 -- start start 2023-10-30T12:10:20.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 librados: setting wanted keys 2023-10-30T12:10:20.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 librados: calling monclient init 2023-10-30T12:10:20.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient: init 2023-10-30T12:10:20.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:20.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488073b10 0x7f648808e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): picked mon.noname-c con 0x7f6488073b10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:20.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6488097780 0x7f648808edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): picked mon.noname-a con 0x7f6488097780 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:20.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648809ccd0 0x7f648808f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): picked mon.noname-b con 0x7f648809ccd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:20.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6488090c80 con 0x7f6488097780 2023-10-30T12:10:20.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64880a3bb0 con 0x7f648809ccd0 2023-10-30T12:10:20.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6488073f70 con 0x7f6488073b10 2023-10-30T12:10:20.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648f324640 10 monclient(hunting): authenticate will time out at 1465.213841s 2023-10-30T12:10:20.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d0c0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488073b10 0x7f648808e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d0c0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488073b10 0x7f648808e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52836/0 (socket says 172.21.15.50:52836) 2023-10-30T12:10:20.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d0c0640 1 -- 172.21.15.50:0/3901621138 learned_addr learned my addr 172.21.15.50:0/3901621138 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:20.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d0c0640 10 monclient(hunting): get_auth_request con 0x7f6488073b10 auth_method 0 2023-10-30T12:10:20.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d0c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d0c0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d0c0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648c8bf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6488097780 0x7f648808edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648d8c1640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648809ccd0 0x7f648808f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.030+0000 7f648c8bf640 10 monclient(hunting): get_auth_request con 0x7f6488097780 auth_method 0 2023-10-30T12:10:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648c8bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648c8bf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648c8bf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d8c1640 10 monclient(hunting): get_auth_request con 0x7f648809ccd0 auth_method 0 2023-10-30T12:10:20.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d8c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d8c1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d8c1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d8c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d8c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d8c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 10 monclient(hunting): handle_auth_done global_id 114461 payload 824 2023-10-30T12:10:20.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648809ccd0 msgr2=0x7f648808f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648809ccd0 0x7f648808f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6488097780 msgr2=0x7f648808edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6488097780 0x7f648808edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 1 -- 172.21.15.50:0/3901621138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64840094d0 con 0x7f6488073b10 2023-10-30T12:10:20.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6488097780 0x7f648808edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:20.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648d0c0640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488073b10 0x7f648808e880 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f648402b8a0 tx=0x7f64840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 client.?.objecter ms_handle_connect 0x7f6488073b10 2023-10-30T12:10:20.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484032860 con 0x7f6488073b10 2023-10-30T12:10:20.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: dump: 2023-10-30T12:10:20.075 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.076 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.076 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.076 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.076 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.076 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.077 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.077 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.077 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.077 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 5 monclient: authenticate success, global_id 114461 2023-10-30T12:10:20.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484032a00 con 0x7f6488073b10 2023-10-30T12:10:20.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484032db0 con 0x7f6488073b10 2023-10-30T12:10:20.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 10 monclient: _renew_subs 2023-10-30T12:10:20.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64880925e0 con 0x7f6488073b10 2023-10-30T12:10:20.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f647b7fe640 4 set_mon_vals no callback set 2023-10-30T12:10:20.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f6479ffb640 10 monclient: dump: 2023-10-30T12:10:20.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.081 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.081 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.081 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.082 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.082 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.082 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.082 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 10 client.114461.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 10 monclient: _renew_subs 2023-10-30T12:10:20.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6488090210 con 0x7f6488073b10 2023-10-30T12:10:20.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.031+0000 7f648f324640 1 librados: init done 2023-10-30T12:10:20.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f648404e450 con 0x7f6488073b10 2023-10-30T12:10:20.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648f324640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648f324640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808afd0 0x7f648808b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648f324640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648806a5a0 con 0x7f648808afd0 2023-10-30T12:10:20.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f6479ffb640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f646c0392f0 0x7f646c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f648406d030 con 0x7f6488073b10 2023-10-30T12:10:20.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808afd0 0x7f648808b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f6479ffb640 10 client.114461.objecter ms_dispatch 0x7f6488000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:20.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f6479ffb640 3 client.114461.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:20.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f6479ffb640 3 client.114461.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:20.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648c8bf640 10 monclient: get_auth_request con 0x7f648808afd0 auth_method 0 2023-10-30T12:10:20.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648c8bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648c8bf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.032+0000 7f648c8bf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648d8c1640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f646c0392f0 0x7f646c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648d8c1640 10 monclient: get_auth_request con 0x7f646c0392f0 auth_method 0 2023-10-30T12:10:20.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 20 client.114461.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648c8bf640 10 monclient(hunting): handle_auth_done global_id 144127 payload 824 2023-10-30T12:10:20.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648c8bf640 1 -- 172.21.15.50:0/3901621138 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64880a0cd0 con 0x7f648808afd0 2023-10-30T12:10:20.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 client.114461.objecter ms_handle_connect 0x7f648808afd0 2023-10-30T12:10:20.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 client.114461.objecter resend_mon_ops 2023-10-30T12:10:20.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808afd0 0x7f648808b3b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f647c000f60 tx=0x7f647c00ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 client.114461.objecter ms_handle_connect 0x7f646c0392f0 2023-10-30T12:10:20.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f648d8c1640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f646c0392f0 0x7f646c03b7b0 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 crypto rx=0x7f64880920d0 tx=0x7f6480009850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647c0052e0 con 0x7f648808afd0 2023-10-30T12:10:20.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 monclient: dump: 2023-10-30T12:10:20.090 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.091 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.091 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.091 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.091 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.092 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.092 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.092 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.092 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.092 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f647c005440 con 0x7f648808afd0 2023-10-30T12:10:20.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:20.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808afd0 msgr2=0x7f648808b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808afd0 0x7f648808b3b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f647c000f60 tx=0x7f647c00ff70 comp rx=0 tx=0).stop 2023-10-30T12:10:20.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808afd0 msgr2=0x7f648808b3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:20.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.033+0000 7f6479ffb640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808afd0 0x7f648808b3b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648f324640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:20.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648f324640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648f324640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6488069fd0 con 0x7f648808b8f0 2023-10-30T12:10:20.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648c8bf640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 msgr2=0x7f64881d46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:10:20.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648c8bf640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 msgr2=0x7f64881d46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:10:20.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:20.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.040+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:20.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient: get_auth_request con 0x7f648808b8f0 auth_method 0 2023-10-30T12:10:20.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 10 monclient(hunting): handle_auth_done global_id 144130 payload 824 2023-10-30T12:10:20.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 1 -- 172.21.15.50:0/3901621138 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f648804e160 con 0x7f648808b8f0 2023-10-30T12:10:20.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f648c8bf640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f647c016000 tx=0x7f647c000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f6479ffb640 10 client.114461.objecter ms_handle_connect 0x7f648808b8f0 2023-10-30T12:10:20.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f6479ffb640 10 client.114461.objecter resend_mon_ops 2023-10-30T12:10:20.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647c006be0 con 0x7f648808b8f0 2023-10-30T12:10:20.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f6479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f6479ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:20.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.241+0000 7f6479ffb640 10 monclient: dump: 2023-10-30T12:10:20.248 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.248 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.249 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.249 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.249 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.249 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.249 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.249 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.250 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.250 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7f647c022050 con 0x7f648808b8f0 2023-10-30T12:10:20.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f6479ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:20.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f6479ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:20.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 msgr2=0x7f64881d46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f6479ffb640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f647c016000 tx=0x7f647c000ed0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f6479ffb640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 msgr2=0x7f64881d46b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:20.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f6479ffb640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f648808b8f0 0x7f64881d46b0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f647c016000 tx=0x7f647c000ed0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.253 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:20.253 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:20.254 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:20.254 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:10:20.254 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 94, 2023-10-30T12:10:20.254 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:20.254 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:20.255 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:10:20.255 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:20.255 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:20.255 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:10:20.256 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:20.256 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:20.256 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:20.256 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:20.256 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:20.257 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:20.257 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:20.257 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:20.257 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:20.257 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:20.258 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:20.258 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:20.258 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:20.258 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:20.259 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:20.259 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:20.259 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:20.259 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:20.259 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:20.260 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:20.260 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:20.260 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:20.260 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:20.260 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:20.261 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:20.261 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:20.261 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:20.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.261 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:20.261 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:20.262 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:20.262 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:20.262 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:20.262 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:20.263 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:20.263 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:20.263 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:20.263 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:20.263 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:20.263 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:20.264 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:20.264 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:20.264 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:20.264 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:20.264 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:20.265 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:20.265 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:20.265 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:20.265 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:20.265 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:20.266 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:20.266 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:20.266 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:20.266 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:20.267 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:20.267 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:20.267 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:20.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.267 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:20.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.269 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:20.269 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:20.269 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:20.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.270 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:20.270 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:20.270 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:20.270 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:20.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:20.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.271 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:20.271 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:20.272 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:20.272 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:20.272 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:20.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.273 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:20.273 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:20.273 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:20.273 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:20.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.273 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:20.274 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:20.274 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:20.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.274 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:20.275 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:20.275 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:20.275 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:20.275 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:20.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.276 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:20.276 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:20.276 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:20.276 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:20.276 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:20.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.277 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:20.277 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:20.277 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:20.277 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:20.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.278 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:20.278 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:20.278 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:20.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.279 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:20.279 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:20.279 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:20.279 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:20.280 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:20.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.280 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:20.280 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:20.280 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:20.281 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:20.281 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:20.281 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:20.281 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:20.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.282 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:20.282 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:20.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.282 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:20.282 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:20.283 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:20.283 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:20.283 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:20.283 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:20.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:20.284 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:20.284 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:20.284 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:10:20.284 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:20.284 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:20.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:20.285 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:20.285 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:20.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 10 librados: watch_flush enter 2023-10-30T12:10:20.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 10 librados: watch_flush exit 2023-10-30T12:10:20.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 20 client.114461.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 10 client.114461.objecter successfully canceled tick 2023-10-30T12:10:20.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f646c0392f0 msgr2=0x7f646c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f646c0392f0 0x7f646c03b7b0 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 crypto rx=0x7f64880920d0 tx=0x7f6480009850 comp rx=0 tx=0).stop 2023-10-30T12:10:20.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 10 monclient: shutdown 2023-10-30T12:10:20.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488073b10 msgr2=0x7f648808e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488073b10 0x7f648808e880 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f648402b8a0 tx=0x7f64840046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648d8c1640 1 -- 172.21.15.50:0/3901621138 reap_dead start 2023-10-30T12:10:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 shutdown_connections 2023-10-30T12:10:20.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 --2- 172.21.15.50:0/3901621138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6488073b10 0x7f648808e880 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 >> 172.21.15.50:0/3901621138 conn(0x7f64880a1170 msgr2=0x7f64880a3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 shutdown_connections 2023-10-30T12:10:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 -- 172.21.15.50:0/3901621138 wait complete. 2023-10-30T12:10:20.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.242+0000 7f648f324640 1 librados: shutdown 2023-10-30T12:10:20.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:10:20.291 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:10:20.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:10:20.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.396+0000 7f18f5826640 10 monclient: get_monmap_and_config 2023-10-30T12:10:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.396+0000 7f18f5826640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.396+0000 7f18f5826640 10 monclient: monmap: 2023-10-30T12:10:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:20.397399+0000 2023-10-30T12:10:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:20.397399+0000 2023-10-30T12:10:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:20.400 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.396+0000 7f18f5826640 1 Processor -- start 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.396+0000 7f18f5826640 1 -- start start 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.396+0000 7f18f5826640 10 monclient: init 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18f00a0080 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f00720b0 0x7f18f013ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient(hunting): picked mon.noname-b con 0x7f18f00720b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f18f0073a30 0x7f18f013d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient(hunting): picked mon.noname-c con 0x7f18f0073a30 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f009ccc0 con 0x7f18f00a0080 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f0073620 con 0x7f18f0073a30 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f0071e90 con 0x7f18f00720b0 2023-10-30T12:10:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18f5826640 10 monclient(hunting): authenticate will time out at 1465.580513s 2023-10-30T12:10:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f00720b0 0x7f18f013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f00720b0 0x7f18f013ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52838/0 (socket says 172.21.15.50:52838) 2023-10-30T12:10:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 -- 172.21.15.50:0/2925218447 learned_addr learned my addr 172.21.15.50:0/2925218447 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): get_auth_request con 0x7f18f00720b0 auth_method 0 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ef7fe640 1 --2- 172.21.15.50:0/2925218447 >> v2:172.21.15.118:3300/0 conn(0x7f18f0073a30 0x7f18f013d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ef7fe640 10 monclient(hunting): get_auth_request con 0x7f18f0073a30 auth_method 0 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ef7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient(hunting): handle_auth_done global_id 114464 payload 274 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 -- 172.21.15.50:0/2925218447 >> v2:172.21.15.118:3300/0 conn(0x7f18f0073a30 msgr2=0x7f18f013d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 --2- 172.21.15.50:0/2925218447 >> v2:172.21.15.118:3300/0 conn(0x7f18f0073a30 0x7f18f013d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 -- 172.21.15.50:0/2925218447 >> v1:172.21.15.118:6789/0 conn(0x7f18f00a0080 legacy=0x7f18f00a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 monclient: found mon.noname-b 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ee7fc640 1 -- 172.21.15.50:0/2925218447 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f009c2b0 con 0x7f18f00720b0 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.397+0000 7f18ef7fe640 1 --2- 172.21.15.50:0/2925218447 >> v2:172.21.15.118:3300/0 conn(0x7f18f0073a30 0x7f18f013d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18ee7fc640 1 --2- 172.21.15.50:0/2925218447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f00720b0 0x7f18f013ad30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009870 tx=0x7f18e402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b9acf7552eacf16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 1 -- 172.21.15.50:0/2925218447 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40317c0 con 0x7f18f00720b0 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: dump: 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18f5826640 5 monclient: authenticate success, global_id 114464 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18f5826640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 1 -- 172.21.15.50:0/2925218447 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4031920 con 0x7f18f00720b0 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18f5826640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18f5826640 4 set_mon_vals no callback set 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 1 -- 172.21.15.50:0/2925218447 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4031cb0 con 0x7f18f00720b0 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.398+0000 7f18edffb640 10 monclient: dump: 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 10 monclient: shutdown 2023-10-30T12:10:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 -- 172.21.15.50:0/2925218447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f00720b0 msgr2=0x7f18f013ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 --2- 172.21.15.50:0/2925218447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f00720b0 0x7f18f013ad30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009870 tx=0x7f18e402b860 comp rx=0 tx=0).stop 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 -- 172.21.15.50:0/2925218447 shutdown_connections 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 --2- 172.21.15.50:0/2925218447 >> v2:172.21.15.118:3300/0 conn(0x7f18f0073a30 0x7f18f013d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 --2- 172.21.15.50:0/2925218447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f00720b0 0x7f18f013ad30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 -- 172.21.15.50:0/2925218447 >> 172.21.15.50:0/2925218447 conn(0x7f18f00a4870 msgr2=0x7f18f00a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 -- 172.21.15.50:0/2925218447 shutdown_connections 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.399+0000 7f18f5826640 1 -- 172.21.15.50:0/2925218447 wait complete. 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 10 monclient: build_initial_monmap 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 10 monclient: monmap: 2023-10-30T12:10:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 librados: starting msgr at 2023-10-30T12:10:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 librados: starting objecter 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 Processor -- start 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 -- start start 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 librados: setting wanted keys 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 librados: calling monclient init 2023-10-30T12:10:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 10 monclient: init 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f0073a30 0x7f18f01ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 10 monclient(hunting): picked mon.noname-c con 0x7f18f0073a30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f00a0080 0x7f18f01b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 10 monclient(hunting): picked mon.noname-a con 0x7f18f00a0080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.400+0000 7f18f5826640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18f01b7e90 0x7f18f00995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient(hunting): picked mon.noname-b con 0x7f18f01b7e90 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f01b04c0 con 0x7f18f00a0080 2023-10-30T12:10:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f01afb20 con 0x7f18f01b7e90 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f01b00e0 con 0x7f18f0073a30 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient(hunting): authenticate will time out at 1465.584028s 2023-10-30T12:10:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f0073a30 0x7f18f01ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f00a0080 0x7f18f01b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f0073a30 0x7f18f01ca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52852/0 (socket says 172.21.15.50:52852) 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 -- 172.21.15.50:0/3441681420 learned_addr learned my addr 172.21.15.50:0/3441681420 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): get_auth_request con 0x7f18f0073a30 auth_method 0 2023-10-30T12:10:20.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 10 monclient(hunting): get_auth_request con 0x7f18f00a0080 auth_method 0 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18f01b7e90 0x7f18f00995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 10 monclient(hunting): get_auth_request con 0x7f18f01b7e90 auth_method 0 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient(hunting): handle_auth_done global_id 114467 payload 824 2023-10-30T12:10:20.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 -- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18f01b7e90 msgr2=0x7f18f00995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18f01b7e90 0x7f18f00995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 -- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f00a0080 msgr2=0x7f18f01b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f00a0080 0x7f18f01b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 monclient: found mon.noname-c 2023-10-30T12:10:20.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:20.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 -- 172.21.15.50:0/3441681420 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e40094d0 con 0x7f18f0073a30 2023-10-30T12:10:20.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ee7fc640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f00a0080 0x7f18f01b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:20.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18eeffd640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f0073a30 0x7f18f01ca1c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f18e000c480 tx=0x7f18e000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 client.?.objecter ms_handle_connect 0x7f18f0073a30 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 1 -- 172.21.15.50:0/3441681420 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00052d0 con 0x7f18f0073a30 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:20.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: dump: 2023-10-30T12:10:20.440 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.440 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.440 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.440 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.441 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.441 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.441 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.441 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.441 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: _finish_auth 0 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:20.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 1 -- 172.21.15.50:0/3441681420 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e0005450 con 0x7f18f0073a30 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 5 monclient: authenticate success, global_id 114467 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 1 -- 172.21.15.50:0/3441681420 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0003880 con 0x7f18f0073a30 2023-10-30T12:10:20.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18d7fff640 10 monclient: dump: 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.444 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:20.445 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient: _renew_subs 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f01afe00 con 0x7f18f0073a30 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18ed7fa640 4 set_mon_vals no callback set 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 client.114467.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient: _renew_subs 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f01b8310 con 0x7f18f0073a30 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.401+0000 7f18f5826640 1 librados: init done 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.402+0000 7f18d7fff640 1 -- 172.21.15.50:0/3441681420 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f18e00039e0 con 0x7f18f0073a30 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.402+0000 7f18f5826640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.402+0000 7f18f5826640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.402+0000 7f18f5826640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.402+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f009b270 con 0x7f18f0073a30 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.402+0000 7f18d7fff640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18dc0392f0 0x7f18dc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18d7fff640 1 -- 172.21.15.50:0/3441681420 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f18e004a920 con 0x7f18f0073a30 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18d7fff640 10 client.114467.objecter ms_dispatch 0x7f18f0000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18d7fff640 3 client.114467.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:20.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18d7fff640 3 client.114467.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:20.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18d7fff640 20 client.114467.objecter dump_active .. 0 homeless 2023-10-30T12:10:20.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18ee7fc640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18dc0392f0 0x7f18dc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:20.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18ee7fc640 10 monclient: get_auth_request con 0x7f18dc0392f0 auth_method 0 2023-10-30T12:10:20.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18d7fff640 10 client.114467.objecter ms_handle_connect 0x7f18dc0392f0 2023-10-30T12:10:20.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.403+0000 7f18ee7fc640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18dc0392f0 0x7f18dc03b7b0 secure :-1 s=READY pgs=4091 cs=0 l=1 rev1=1 crypto rx=0x7f18e4036000 tx=0x7f18e403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:20.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.407+0000 7f18d7fff640 1 -- 172.21.15.50:0/3441681420 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f18e0003c90 con 0x7f18f0073a30 2023-10-30T12:10:20.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.407+0000 7f18d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:20.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.407+0000 7f18d7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.658+0000 7f18f5826640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.658+0000 7f18f5826640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.658+0000 7f18f5826640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.658+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f18f004e160 con 0x7f18f0073a30 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f18d7fff640 1 -- 172.21.15.50:0/3441681420 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f18e001a360 con 0x7f18f0073a30 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f18d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:10:20.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f18d7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:20.662 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.662 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=7,logm=57} crc {auth=2294429720,config=3140514009,health=809127085,kv=3824953070,logm=2753329113}) 2023-10-30T12:10:20.663 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=215065583,config=3140514009,health=809127085,kv=784550477,logm=1548410138}) 2023-10-30T12:10:20.663 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.663 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=7,logm=57} crc {auth=2294429720,config=3140514009,health=809127085,kv=3824953070,logm=2753329113}) 2023-10-30T12:10:20.663 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.660+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=7,logm=57} crc {auth=2436057691,config=3140514009,health=809127085,kv=2362381073,logm=4186327897}) 2023-10-30T12:10:20.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 10 librados: watch_flush enter 2023-10-30T12:10:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 10 librados: watch_flush exit 2023-10-30T12:10:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 20 client.114467.objecter shutdown clearing up homeless session... 2023-10-30T12:10:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 10 client.114467.objecter successfully canceled tick 2023-10-30T12:10:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18dc0392f0 msgr2=0x7f18dc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18dc0392f0 0x7f18dc03b7b0 secure :-1 s=READY pgs=4091 cs=0 l=1 rev1=1 crypto rx=0x7f18e4036000 tx=0x7f18e403d040 comp rx=0 tx=0).stop 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 10 monclient: shutdown 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f0073a30 msgr2=0x7f18f01ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:20.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f0073a30 0x7f18f01ca1c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f18e000c480 tx=0x7f18e000c940 comp rx=0 tx=0).stop 2023-10-30T12:10:20.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 shutdown_connections 2023-10-30T12:10:20.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f18dc0392f0 0x7f18dc03b7b0 unknown :-1 s=CLOSED pgs=4091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.667 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.668 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527390180}) 2023-10-30T12:10:20.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18f01b7e90 0x7f18f00995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f00a0080 0x7f18f01b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 --2- 172.21.15.50:0/3441681420 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f18f0073a30 0x7f18f01ca1c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:20.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 >> 172.21.15.50:0/3441681420 conn(0x7f18f00a4870 msgr2=0x7f18f01bc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:20.670 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649452373}) 2023-10-30T12:10:20.670 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.670 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527390180}) 2023-10-30T12:10:20.670 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3578996756}) 2023-10-30T12:10:20.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 shutdown_connections 2023-10-30T12:10:20.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 -- 172.21.15.50:0/3441681420 wait complete. 2023-10-30T12:10:20.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:20.662+0000 7f18f5826640 1 librados: shutdown 2023-10-30T12:10:20.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.663+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.663+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147732396}) 2023-10-30T12:10:20.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.663+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=332784060}) 2023-10-30T12:10:20.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.663+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.663+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147732396}) 2023-10-30T12:10:20.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.663+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216076966}) 2023-10-30T12:10:20.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.665+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.665+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217259345}) 2023-10-30T12:10:20.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.665+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1978188080}) 2023-10-30T12:10:20.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.665+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.665+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217259345}) 2023-10-30T12:10:20.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.665+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137032923}) 2023-10-30T12:10:20.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.666+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.666+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753614692}) 2023-10-30T12:10:20.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.666+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=865391010}) 2023-10-30T12:10:20.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.666+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.666+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753614692}) 2023-10-30T12:10:20.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.666+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845002446}) 2023-10-30T12:10:20.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2426633764}) 2023-10-30T12:10:20.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=802787013}) 2023-10-30T12:10:20.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2426633764}) 2023-10-30T12:10:20.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438052664}) 2023-10-30T12:10:20.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116757384}) 2023-10-30T12:10:20.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727937852}) 2023-10-30T12:10:20.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116757384}) 2023-10-30T12:10:20.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.667+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116510929}) 2023-10-30T12:10:20.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.668+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.668+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3215408881}) 2023-10-30T12:10:20.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.668+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266738226}) 2023-10-30T12:10:20.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.668+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.668+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3215408881}) 2023-10-30T12:10:20.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.668+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2842395905}) 2023-10-30T12:10:20.686 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.669+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.686 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.669+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547359629}) 2023-10-30T12:10:20.686 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.669+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1613800703}) 2023-10-30T12:10:20.686 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.669+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.669+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547359629}) 2023-10-30T12:10:20.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.669+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923399422}) 2023-10-30T12:10:20.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.670+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.670+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=26043921}) 2023-10-30T12:10:20.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.670+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3107555790}) 2023-10-30T12:10:20.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.670+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.670+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=26043921}) 2023-10-30T12:10:20.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.670+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360933963}) 2023-10-30T12:10:20.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276868420}) 2023-10-30T12:10:20.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377998816}) 2023-10-30T12:10:20.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276868420}) 2023-10-30T12:10:20.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059845151}) 2023-10-30T12:10:20.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.690 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964226814}) 2023-10-30T12:10:20.690 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2764928382}) 2023-10-30T12:10:20.690 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.690 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964226814}) 2023-10-30T12:10:20.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.671+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088586193}) 2023-10-30T12:10:20.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.672+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.672+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487247689}) 2023-10-30T12:10:20.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.672+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1951834779}) 2023-10-30T12:10:20.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.672+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.672+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487247689}) 2023-10-30T12:10:20.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.672+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653135316}) 2023-10-30T12:10:20.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.673+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.673+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848698206}) 2023-10-30T12:10:20.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.673+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115077890}) 2023-10-30T12:10:20.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.673+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.673+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848698206}) 2023-10-30T12:10:20.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.673+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53974710}) 2023-10-30T12:10:20.694 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:10:20.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754349324}) 2023-10-30T12:10:20.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1910099017}) 2023-10-30T12:10:20.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754349324}) 2023-10-30T12:10:20.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2702361431}) 2023-10-30T12:10:20.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539684561}) 2023-10-30T12:10:20.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2096145404}) 2023-10-30T12:10:20.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539684561}) 2023-10-30T12:10:20.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.674+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098532394}) 2023-10-30T12:10:20.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.675+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.675+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092695018}) 2023-10-30T12:10:20.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.675+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1051088241}) 2023-10-30T12:10:20.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.675+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.675+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092695018}) 2023-10-30T12:10:20.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.675+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461017989}) 2023-10-30T12:10:20.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438589857}) 2023-10-30T12:10:20.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3300256408}) 2023-10-30T12:10:20.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438589857}) 2023-10-30T12:10:20.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=109411688}) 2023-10-30T12:10:20.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254556845}) 2023-10-30T12:10:20.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060862598}) 2023-10-30T12:10:20.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.676+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254556845}) 2023-10-30T12:10:20.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.677+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452323985}) 2023-10-30T12:10:20.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.677+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.677+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891975495}) 2023-10-30T12:10:20.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.677+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1688943046}) 2023-10-30T12:10:20.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.677+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.677+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891975495}) 2023-10-30T12:10:20.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.677+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2091053607}) 2023-10-30T12:10:20.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.678+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.678+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870405511}) 2023-10-30T12:10:20.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.678+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3248324395}) 2023-10-30T12:10:20.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.678+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.678+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870405511}) 2023-10-30T12:10:20.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.678+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2083002934}) 2023-10-30T12:10:20.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338775382}) 2023-10-30T12:10:20.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3926465848}) 2023-10-30T12:10:20.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338775382}) 2023-10-30T12:10:20.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699084394}) 2023-10-30T12:10:20.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790994242}) 2023-10-30T12:10:20.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1890384036}) 2023-10-30T12:10:20.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790994242}) 2023-10-30T12:10:20.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.679+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3969853167}) 2023-10-30T12:10:20.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.680+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.680+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146761327}) 2023-10-30T12:10:20.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.680+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1211662895}) 2023-10-30T12:10:20.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.680+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.680+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146761327}) 2023-10-30T12:10:20.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.680+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2901985123}) 2023-10-30T12:10:20.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.681+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.681+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640495999}) 2023-10-30T12:10:20.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.681+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806443312}) 2023-10-30T12:10:20.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.681+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.681+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640495999}) 2023-10-30T12:10:20.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.681+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152412485}) 2023-10-30T12:10:20.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796481185}) 2023-10-30T12:10:20.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2261257648}) 2023-10-30T12:10:20.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796481185}) 2023-10-30T12:10:20.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435719095}) 2023-10-30T12:10:20.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649385359}) 2023-10-30T12:10:20.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534296509}) 2023-10-30T12:10:20.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649385359}) 2023-10-30T12:10:20.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.682+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=344332947}) 2023-10-30T12:10:20.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.683+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.683+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862488783}) 2023-10-30T12:10:20.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.683+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3759984424}) 2023-10-30T12:10:20.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.683+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.683+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862488783}) 2023-10-30T12:10:20.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.683+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252651111}) 2023-10-30T12:10:20.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065350787}) 2023-10-30T12:10:20.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922548296}) 2023-10-30T12:10:20.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065350787}) 2023-10-30T12:10:20.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3902735448}) 2023-10-30T12:10:20.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312591409}) 2023-10-30T12:10:20.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1165304592}) 2023-10-30T12:10:20.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312591409}) 2023-10-30T12:10:20.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.684+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019036012}) 2023-10-30T12:10:20.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.685+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.685+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108961441}) 2023-10-30T12:10:20.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.685+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2158245648}) 2023-10-30T12:10:20.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.685+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.685+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108961441}) 2023-10-30T12:10:20.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.685+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=768911156}) 2023-10-30T12:10:20.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724291942}) 2023-10-30T12:10:20.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3297389873}) 2023-10-30T12:10:20.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724291942}) 2023-10-30T12:10:20.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=438988980}) 2023-10-30T12:10:20.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115142601}) 2023-10-30T12:10:20.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2809342933}) 2023-10-30T12:10:20.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115142601}) 2023-10-30T12:10:20.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.686+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977987235}) 2023-10-30T12:10:20.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.687+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.687+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865777970}) 2023-10-30T12:10:20.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.687+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=19818551}) 2023-10-30T12:10:20.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.687+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.687+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865777970}) 2023-10-30T12:10:20.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.687+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4090359122}) 2023-10-30T12:10:20.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.687+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4112783336}) 2023-10-30T12:10:20.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697027785}) 2023-10-30T12:10:20.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4112783336}) 2023-10-30T12:10:20.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1989970072}) 2023-10-30T12:10:20.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492691255}) 2023-10-30T12:10:20.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723300100}) 2023-10-30T12:10:20.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492691255}) 2023-10-30T12:10:20.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.688+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988790623}) 2023-10-30T12:10:20.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.689+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.689+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287995492}) 2023-10-30T12:10:20.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.689+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=150584905}) 2023-10-30T12:10:20.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.689+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.689+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287995492}) 2023-10-30T12:10:20.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.689+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3087143040}) 2023-10-30T12:10:20.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255661147}) 2023-10-30T12:10:20.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4213661517}) 2023-10-30T12:10:20.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255661147}) 2023-10-30T12:10:20.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1867429317}) 2023-10-30T12:10:20.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334154963}) 2023-10-30T12:10:20.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270288391}) 2023-10-30T12:10:20.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334154963}) 2023-10-30T12:10:20.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.690+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919136717}) 2023-10-30T12:10:20.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.691+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.691+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636145576}) 2023-10-30T12:10:20.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.691+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998163990}) 2023-10-30T12:10:20.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.691+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.691+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636145576}) 2023-10-30T12:10:20.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.691+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945351559}) 2023-10-30T12:10:20.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415467088}) 2023-10-30T12:10:20.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248601239}) 2023-10-30T12:10:20.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415467088}) 2023-10-30T12:10:20.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693064335}) 2023-10-30T12:10:20.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544209240}) 2023-10-30T12:10:20.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317763948}) 2023-10-30T12:10:20.748 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.748 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544209240}) 2023-10-30T12:10:20.748 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.692+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=973671518}) 2023-10-30T12:10:20.748 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.693+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.748 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.693+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901161280}) 2023-10-30T12:10:20.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.693+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596716230}) 2023-10-30T12:10:20.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.693+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.693+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901161280}) 2023-10-30T12:10:20.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.693+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2586462173}) 2023-10-30T12:10:20.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866742416}) 2023-10-30T12:10:20.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833147721}) 2023-10-30T12:10:20.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866742416}) 2023-10-30T12:10:20.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403095781}) 2023-10-30T12:10:20.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95165997}) 2023-10-30T12:10:20.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483425649}) 2023-10-30T12:10:20.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95165997}) 2023-10-30T12:10:20.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.694+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=749426544}) 2023-10-30T12:10:20.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.695+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.695+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048487732}) 2023-10-30T12:10:20.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.695+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3619048799}) 2023-10-30T12:10:20.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.695+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.695+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048487732}) 2023-10-30T12:10:20.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.695+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951472782}) 2023-10-30T12:10:20.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.696+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.696+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411448342}) 2023-10-30T12:10:20.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.696+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=770406883}) 2023-10-30T12:10:20.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.696+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.696+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411448342}) 2023-10-30T12:10:20.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.696+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825618185}) 2023-10-30T12:10:20.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.697+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.697+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825843474}) 2023-10-30T12:10:20.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.697+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2337253723}) 2023-10-30T12:10:20.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.697+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.697+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825843474}) 2023-10-30T12:10:20.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.697+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1299064733}) 2023-10-30T12:10:20.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.698+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.698+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=1654341821,mdsmap=3984425268,mgr=1142129444,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1477899396,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=914880057,osd_snap=1919623895}) 2023-10-30T12:10:20.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.698+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=53,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=11} crc {logm=3628730287,mdsmap=3984425268,mgr=1715906815,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3050262862,mon_config_key=3217606506,monmap=3324057927,osd_metadata=965620590,osd_pg_creating=914880057,osd_snap=1110661407}) 2023-10-30T12:10:20.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.698+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.698+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=1654341821,mdsmap=3984425268,mgr=1142129444,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1477899396,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=914880057,osd_snap=1919623895}) 2023-10-30T12:10:20.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.698+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=50,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=4166897054,mdsmap=3984425268,mgr=1715906815,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1477899396,mon_config_key=1821124913,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=914880057,osd_snap=3092174847}) 2023-10-30T12:10:20.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.699+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.699+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=660191237,osdmap=239561022}) 2023-10-30T12:10:20.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.699+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=3108140433,osdmap=1381663116}) 2023-10-30T12:10:20.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.699+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.699+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=660191237,osdmap=239561022}) 2023-10-30T12:10:20.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.699+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=3108140433,osdmap=4006666008}) 2023-10-30T12:10:20.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.700+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.700+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1562988270}) 2023-10-30T12:10:20.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.700+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2934904762}) 2023-10-30T12:10:20.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.700+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.700+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1562988270}) 2023-10-30T12:10:20.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.700+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2248958993}) 2023-10-30T12:10:20.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.701+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.701+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3466931252}) 2023-10-30T12:10:20.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.701+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2761967377}) 2023-10-30T12:10:20.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.701+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.701+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3466931252}) 2023-10-30T12:10:20.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.701+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=258704040}) 2023-10-30T12:10:20.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.703+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.703+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1716724173}) 2023-10-30T12:10:20.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.703+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3261727344}) 2023-10-30T12:10:20.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.703+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.703+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1716724173}) 2023-10-30T12:10:20.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.703+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4170192608}) 2023-10-30T12:10:20.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.709+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.709+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=991748967}) 2023-10-30T12:10:20.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.709+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=203303863}) 2023-10-30T12:10:20.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.709+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:10:20.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.709+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=991748967}) 2023-10-30T12:10:20.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:10:20.709+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2374487139}) 2023-10-30T12:10:21.695 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:10:21.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:21.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.811+0000 7fc73b3b0640 10 monclient: get_monmap_and_config 2023-10-30T12:10:21.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.811+0000 7fc73b3b0640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.811+0000 7fc73b3b0640 10 monclient: monmap: 2023-10-30T12:10:21.813 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:21.812802+0000 2023-10-30T12:10:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:21.812802+0000 2023-10-30T12:10:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:21.815 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.812+0000 7fc73b3b0640 1 Processor -- start 2023-10-30T12:10:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.812+0000 7fc73b3b0640 1 -- start start 2023-10-30T12:10:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.812+0000 7fc73b3b0640 10 monclient: init 2023-10-30T12:10:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-10-30T12:10:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc73409f5b0 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734099b80 0x7fc734096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient(hunting): picked mon.noname-b con 0x7fc734099b80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc734138b30 0x7fc73413af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient(hunting): picked mon.noname-a con 0x7fc734138b30 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73409f990 con 0x7fc73409f5b0 2023-10-30T12:10:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc734098810 con 0x7fc734138b30 2023-10-30T12:10:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73406aef0 con 0x7fc734099b80 2023-10-30T12:10:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73b3b0640 10 monclient(hunting): authenticate will time out at 1466.996187s 2023-10-30T12:10:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc734138b30 0x7fc73413af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734099b80 0x7fc734096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc734138b30 0x7fc73413af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34042/0 (socket says 172.21.15.50:34042) 2023-10-30T12:10:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 -- 172.21.15.50:0/724822811 learned_addr learned my addr 172.21.15.50:0/724822811 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 10 monclient(hunting): get_auth_request con 0x7fc734099b80 auth_method 0 2023-10-30T12:10:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): get_auth_request con 0x7fc734138b30 auth_method 0 2023-10-30T12:10:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73894b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient(hunting): handle_auth_done global_id 134256 payload 274 2023-10-30T12:10:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 -- 172.21.15.50:0/724822811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734099b80 msgr2=0x7fc734096790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 --2- 172.21.15.50:0/724822811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734099b80 0x7fc734096790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 -- 172.21.15.50:0/724822811 >> v1:172.21.15.118:6789/0 conn(0x7fc73409f5b0 legacy=0x7fc734096080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 monclient: found mon.noname-a 2023-10-30T12:10:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.813+0000 7fc73994d640 1 -- 172.21.15.50:0/724822811 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73409a1a0 con 0x7fc734138b30 2023-10-30T12:10:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc73994d640 1 --2- 172.21.15.50:0/724822811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc734138b30 0x7fc73413af20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc73409a390 tx=0x7fc72c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c640782e4c293a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 1 -- 172.21.15.50:0/724822811 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c02c930 con 0x7fc734138b30 2023-10-30T12:10:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: dump: 2023-10-30T12:10:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:21.834 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:21.834 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:21.834 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:21.834 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:21.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 1 -- 172.21.15.50:0/724822811 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c02ca90 con 0x7fc734138b30 2023-10-30T12:10:21.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc73b3b0640 5 monclient: authenticate success, global_id 134256 2023-10-30T12:10:21.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc73b3b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:21.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc73b3b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:21.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 1 -- 172.21.15.50:0/724822811 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c02ce20 con 0x7fc734138b30 2023-10-30T12:10:21.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc73b3b0640 4 set_mon_vals no callback set 2023-10-30T12:10:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.814+0000 7fc72bfff640 10 monclient: dump: 2023-10-30T12:10:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:21.838 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.838 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.838 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.838 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:21.838 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:21.839 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:21.839 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:21.839 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:21.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 10 monclient: shutdown 2023-10-30T12:10:21.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:21.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 -- 172.21.15.50:0/724822811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc734138b30 msgr2=0x7fc73413af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/724822811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc734138b30 0x7fc73413af20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc73409a390 tx=0x7fc72c02bd90 comp rx=0 tx=0).stop 2023-10-30T12:10:21.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 -- 172.21.15.50:0/724822811 shutdown_connections 2023-10-30T12:10:21.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/724822811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc734138b30 0x7fc73413af20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/724822811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734099b80 0x7fc734096790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 -- 172.21.15.50:0/724822811 >> 172.21.15.50:0/724822811 conn(0x7fc7340a05c0 msgr2=0x7fc73409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:21.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 -- 172.21.15.50:0/724822811 shutdown_connections 2023-10-30T12:10:21.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 1 -- 172.21.15.50:0/724822811 wait complete. 2023-10-30T12:10:21.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 10 monclient: build_initial_monmap 2023-10-30T12:10:21.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.815+0000 7fc73b3b0640 10 monclient: monmap: 2023-10-30T12:10:21.841 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:21.842 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:21.842 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:21.842 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:21.842 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:21.842 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:21.843 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:21.843 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:21.843 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:21.843 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:21.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 librados: starting msgr at 2023-10-30T12:10:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 librados: starting objecter 2023-10-30T12:10:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 Processor -- start 2023-10-30T12:10:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 -- start start 2023-10-30T12:10:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 librados: setting wanted keys 2023-10-30T12:10:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 librados: calling monclient init 2023-10-30T12:10:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient: init 2023-10-30T12:10:21.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:21.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:21.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc734099b80 0x7fc7341ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): picked mon.noname-b con 0x7fc734099b80 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:21.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73409f5b0 0x7fc734095650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): picked mon.noname-a con 0x7fc73409f5b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:21.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734138b30 0x7fc734095b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): picked mon.noname-c con 0x7fc734138b30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:21.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc734072080 con 0x7fc73409f5b0 2023-10-30T12:10:21.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7340960d0 con 0x7fc734099b80 2023-10-30T12:10:21.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:21.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73409c560 con 0x7fc734138b30 2023-10-30T12:10:21.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:21.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73b3b0640 10 monclient(hunting): authenticate will time out at 1466.999663s 2023-10-30T12:10:21.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734138b30 0x7fc734095b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73409f5b0 0x7fc734095650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734138b30 0x7fc734095b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52866/0 (socket says 172.21.15.50:52866) 2023-10-30T12:10:21.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 1 -- 172.21.15.50:0/2809714757 learned_addr learned my addr 172.21.15.50:0/2809714757 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:21.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 10 monclient(hunting): get_auth_request con 0x7fc734138b30 auth_method 0 2023-10-30T12:10:21.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 10 monclient(hunting): get_auth_request con 0x7fc73409f5b0 auth_method 0 2023-10-30T12:10:21.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73994d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73894b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73914c640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc734099b80 0x7fc7341ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73914c640 10 monclient(hunting): get_auth_request con 0x7fc734099b80 auth_method 0 2023-10-30T12:10:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73914c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:21.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73914c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.816+0000 7fc73914c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73914c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73914c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73914c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 10 monclient(hunting): handle_auth_done global_id 134259 payload 824 2023-10-30T12:10:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 1 -- 172.21.15.50:0/2809714757 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734138b30 msgr2=0x7fc734095b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734138b30 0x7fc734095b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 1 -- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc734099b80 msgr2=0x7fc7341ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc734099b80 0x7fc7341ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 1 monclient: found mon.noname-a 2023-10-30T12:10:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 1 -- 172.21.15.50:0/2809714757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c009b30 con 0x7fc73409f5b0 2023-10-30T12:10:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73894b640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73409f5b0 0x7fc734095650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc72400e870 tx=0x7fc72400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 client.?.objecter ms_handle_connect 0x7fc73409f5b0 2023-10-30T12:10:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 1 -- 172.21.15.50:0/2809714757 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72400ee60 con 0x7fc73409f5b0 2023-10-30T12:10:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: dump: 2023-10-30T12:10:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:21.857 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:21.857 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:21.857 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:21.857 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:21.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 1 -- 172.21.15.50:0/2809714757 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72400c940 con 0x7fc73409f5b0 2023-10-30T12:10:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 1 -- 172.21.15.50:0/2809714757 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72400ccd0 con 0x7fc73409f5b0 2023-10-30T12:10:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc729ffb640 10 monclient: dump: 2023-10-30T12:10:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:21.861 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 5 monclient: authenticate success, global_id 134259 2023-10-30T12:10:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc72b7fe640 4 set_mon_vals no callback set 2023-10-30T12:10:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 10 monclient: _renew_subs 2023-10-30T12:10:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7341367a0 con 0x7fc73409f5b0 2023-10-30T12:10:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 10 client.134259.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 10 monclient: _renew_subs 2023-10-30T12:10:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7341ca8c0 con 0x7fc73409f5b0 2023-10-30T12:10:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.817+0000 7fc73b3b0640 1 librados: init done 2023-10-30T12:10:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.818+0000 7fc729ffb640 1 -- 172.21.15.50:0/2809714757 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc724013470 con 0x7fc73409f5b0 2023-10-30T12:10:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.818+0000 7fc73b3b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.818+0000 7fc73b3b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.818+0000 7fc73b3b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.818+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73404e160 con 0x7fc73409f5b0 2023-10-30T12:10:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc729ffb640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7180392f0 0x7fc71803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc729ffb640 1 -- 172.21.15.50:0/2809714757 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fc72404b040 con 0x7fc73409f5b0 2023-10-30T12:10:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc729ffb640 10 client.134259.objecter ms_dispatch 0x7fc734000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc729ffb640 3 client.134259.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc729ffb640 3 client.134259.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc729ffb640 20 client.134259.objecter dump_active .. 0 homeless 2023-10-30T12:10:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc73914c640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7180392f0 0x7fc71803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.819+0000 7fc73914c640 10 monclient: get_auth_request con 0x7fc7180392f0 auth_method 0 2023-10-30T12:10:21.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.820+0000 7fc729ffb640 10 client.134259.objecter ms_handle_connect 0x7fc7180392f0 2023-10-30T12:10:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.820+0000 7fc73914c640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7180392f0 0x7fc71803b7b0 secure :-1 s=READY pgs=4092 cs=0 l=1 rev1=1 crypto rx=0x7fc730007fd0 tx=0x7fc730007e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.824+0000 7fc729ffb640 1 -- 172.21.15.50:0/2809714757 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc724013720 con 0x7fc73409f5b0 2023-10-30T12:10:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.824+0000 7fc729ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:21.824+0000 7fc729ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.067+0000 7fc73b3b0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:22.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.067+0000 7fc73b3b0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:22.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.067+0000 7fc73b3b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.067+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc7340745c0 con 0x7fc73409f5b0 2023-10-30T12:10:22.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.067+0000 7fc729ffb640 1 -- 172.21.15.50:0/2809714757 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fc724017020 con 0x7fc73409f5b0 2023-10-30T12:10:22.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.067+0000 7fc729ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:22.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.067+0000 7fc729ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.070 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:22.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 10 librados: watch_flush enter 2023-10-30T12:10:22.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 10 librados: watch_flush exit 2023-10-30T12:10:22.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 20 client.134259.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 10 client.134259.objecter successfully canceled tick 2023-10-30T12:10:22.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7180392f0 msgr2=0x7fc71803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7180392f0 0x7fc71803b7b0 secure :-1 s=READY pgs=4092 cs=0 l=1 rev1=1 crypto rx=0x7fc730007fd0 tx=0x7fc730007e80 comp rx=0 tx=0).stop 2023-10-30T12:10:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 10 monclient: shutdown 2023-10-30T12:10:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73409f5b0 msgr2=0x7fc734095650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73409f5b0 0x7fc734095650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc72400e870 tx=0x7fc72400ed30 comp rx=0 tx=0).stop 2023-10-30T12:10:22.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 shutdown_connections 2023-10-30T12:10:22.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc7180392f0 0x7fc71803b7b0 unknown :-1 s=CLOSED pgs=4092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc734138b30 0x7fc734095b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc73409f5b0 0x7fc734095650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 --2- 172.21.15.50:0/2809714757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc734099b80 0x7fc7341ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 >> 172.21.15.50:0/2809714757 conn(0x7fc7340a05c0 msgr2=0x7fc7340a28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 shutdown_connections 2023-10-30T12:10:22.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 -- 172.21.15.50:0/2809714757 wait complete. 2023-10-30T12:10:22.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.069+0000 7fc73b3b0640 1 librados: shutdown 2023-10-30T12:10:22.081 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:10:22.081 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:10:22.081 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:10:22.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:10:22.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.186+0000 7f91bb432640 10 monclient: get_monmap_and_config 2023-10-30T12:10:22.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.186+0000 7f91bb432640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.186+0000 7f91bb432640 10 monclient: monmap: 2023-10-30T12:10:22.188 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.188 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.188 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:22.187207+0000 2023-10-30T12:10:22.188 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:22.187207+0000 2023-10-30T12:10:22.189 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.189 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.189 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:22.189 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:22.189 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:22.189 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:22.190 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:22.190 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.186+0000 7f91bb432640 1 Processor -- start 2023-10-30T12:10:22.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.186+0000 7f91bb432640 1 -- start start 2023-10-30T12:10:22.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.186+0000 7f91bb432640 10 monclient: init 2023-10-30T12:10:22.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-10-30T12:10:22.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f91b40a2ae0 0x7f91b40a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient(hunting): picked mon.noname-c con 0x7f91b40a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:22.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91b40a2ec0 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:22.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91b4135df0 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:22.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b40a4190 con 0x7f91b40a2ec0 2023-10-30T12:10:22.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b409a2e0 con 0x7f91b4135df0 2023-10-30T12:10:22.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b40a2990 con 0x7f91b40a2ae0 2023-10-30T12:10:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91bb432640 10 monclient(hunting): authenticate will time out at 1467.370404s 2023-10-30T12:10:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b89cd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f91b40a2ec0 0x7f91b41356e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41162/0 (socket says 172.21.15.50:41162) 2023-10-30T12:10:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b89cd640 1 -- 172.21.15.50:0/13678733 learned_addr learned my addr 172.21.15.50:0/13678733 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:10:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 1 --2- 172.21.15.50:0/13678733 >> v2:172.21.15.118:3300/0 conn(0x7f91b40a2ae0 0x7f91b40a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071255469 0 0) 0x7f91a4001110 con 0x7f91b40a2ec0 2023-10-30T12:10:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91abfff640 10 monclient(hunting): _negotiate 2023-10-30T12:10:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91abfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91abfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91abfff640 10 monclient(hunting): my global_id is 134262 2023-10-30T12:10:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a0003520 con 0x7f91b40a2ec0 2023-10-30T12:10:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 10 monclient(hunting): get_auth_request con 0x7f91b40a2ae0 auth_method 0 2023-10-30T12:10:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91b91ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.187+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3139822437 0 0) 0x7f91a0003520 con 0x7f91b40a2ec0 2023-10-30T12:10:22.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 >> v2:172.21.15.118:3300/0 conn(0x7f91b40a2ae0 msgr2=0x7f91b40a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 --2- 172.21.15.50:0/13678733 >> v2:172.21.15.118:3300/0 conn(0x7f91b40a2ae0 0x7f91b40a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 >> v1:172.21.15.118:6789/0 conn(0x7f91b4135df0 legacy=0x7f91b413da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:10:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:10:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b409a8a0 con 0x7f91b40a2ec0 2023-10-30T12:10:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f91a4001800 con 0x7f91b40a2ec0 2023-10-30T12:10:22.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:10:22.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: dump: 2023-10-30T12:10:22.209 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.209 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.209 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.211 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:10:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 >> v1:172.21.15.50:6789/0 conn(0x7f91b40a2ec0 legacy=0x7f91b41356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:10:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a0003970 0x7f91a0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): picked mon.b con 0x7f91a0003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91a00047c0 0x7f91a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): picked mon.c con 0x7f91a00047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91a00052a0 0x7f91a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): picked mon.a con 0x7f91a00052a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a000c140 con 0x7f91a00052a0 2023-10-30T12:10:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a000c2b0 con 0x7f91a0003970 2023-10-30T12:10:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a000c470 con 0x7f91a00047c0 2023-10-30T12:10:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91abfff640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:10:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91a00047c0 0x7f91a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a0003970 0x7f91a0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91a00052a0 0x7f91a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 10 monclient(hunting): get_auth_request con 0x7f91a00047c0 auth_method 0 2023-10-30T12:10:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): get_auth_request con 0x7f91a00052a0 auth_method 0 2023-10-30T12:10:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 10 monclient(hunting): get_auth_request con 0x7f91a0003970 auth_method 0 2023-10-30T12:10:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient(hunting): handle_auth_done global_id 134262 payload 293 2023-10-30T12:10:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 1 -- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91a00047c0 msgr2=0x7f91a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91a00047c0 0x7f91a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 1 -- 172.21.15.50:0/13678733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a0003970 msgr2=0x7f91a0003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a0003970 0x7f91a0003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 1 monclient: found mon.a 2023-10-30T12:10:22.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 1 -- 172.21.15.50:0/13678733 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0003520 con 0x7f91a00052a0 2023-10-30T12:10:22.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b99cf640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91bb432640 5 monclient: authenticate success, global_id 134262 2023-10-30T12:10:22.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91bb432640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:22.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b89cd640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91a00047c0 0x7f91a0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:22.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.188+0000 7f91b91ce640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a0003970 0x7f91a0003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:22.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91b99cf640 1 -- 172.21.15.50:0/13678733 reap_dead start 2023-10-30T12:10:22.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91b99cf640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91a00052a0 0x7f91a000b980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f91ac001e50 tx=0x7f91a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed986571a9fa2d28 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac003b70 con 0x7f91a00052a0 2023-10-30T12:10:22.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 10 monclient: dump: 2023-10-30T12:10:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.224 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.224 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.224 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.224 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ac003cf0 con 0x7f91a00052a0 2023-10-30T12:10:22.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91bb432640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:22.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91bb432640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:22.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 1 -- 172.21.15.50:0/13678733 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac005310 con 0x7f91a00052a0 2023-10-30T12:10:22.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91bb432640 4 set_mon_vals no callback set 2023-10-30T12:10:22.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91abfff640 10 monclient: dump: 2023-10-30T12:10:22.226 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.226 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.226 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.227 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.227 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.227 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.227 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.227 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.227 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.228 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91bb432640 10 monclient: shutdown 2023-10-30T12:10:22.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91bb432640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91bb432640 1 -- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91a00052a0 msgr2=0x7f91a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.189+0000 7f91bb432640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91a00052a0 0x7f91a000b980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f91ac001e50 tx=0x7f91a00057f0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 1 -- 172.21.15.50:0/13678733 shutdown_connections 2023-10-30T12:10:22.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 1 --2- 172.21.15.50:0/13678733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91a00052a0 0x7f91a000b980 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 1 -- 172.21.15.50:0/13678733 >> 172.21.15.50:0/13678733 conn(0x7f91b40a80b0 msgr2=0x7f91b40a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 1 -- 172.21.15.50:0/13678733 shutdown_connections 2023-10-30T12:10:22.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 1 -- 172.21.15.50:0/13678733 wait complete. 2023-10-30T12:10:22.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 10 monclient: monmap: 2023-10-30T12:10:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:22.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 1 librados: starting msgr at 2023-10-30T12:10:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.190+0000 7f91bb432640 1 librados: starting objecter 2023-10-30T12:10:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 Processor -- start 2023-10-30T12:10:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 -- start start 2023-10-30T12:10:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 librados: setting wanted keys 2023-10-30T12:10:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 librados: calling monclient init 2023-10-30T12:10:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient: init 2023-10-30T12:10:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a00052a0 0x7f91b41ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): picked mon.noname-b con 0x7f91a00052a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b41ca780 0x7f91b411f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): picked mon.noname-a con 0x7f91b41ca780 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91b41caba0 0x7f91b411fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): picked mon.noname-c con 0x7f91b41caba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b4123000 con 0x7f91b41ca780 2023-10-30T12:10:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b4123530 con 0x7f91a00052a0 2023-10-30T12:10:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b4122080 con 0x7f91b41caba0 2023-10-30T12:10:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91bb432640 10 monclient(hunting): authenticate will time out at 1467.374527s 2023-10-30T12:10:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b41ca780 0x7f91b411f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91b41caba0 0x7f91b411fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b41ca780 0x7f91b411f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34088/0 (socket says 172.21.15.50:34088) 2023-10-30T12:10:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 10 monclient(hunting): get_auth_request con 0x7f91b41caba0 auth_method 0 2023-10-30T12:10:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 -- 172.21.15.50:0/4031278486 learned_addr learned my addr 172.21.15.50:0/4031278486 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): get_auth_request con 0x7f91b41ca780 auth_method 0 2023-10-30T12:10:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b91ce640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a00052a0 0x7f91b41ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b91ce640 10 monclient(hunting): get_auth_request con 0x7f91a00052a0 auth_method 0 2023-10-30T12:10:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b91ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b91ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b91ce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient(hunting): handle_auth_done global_id 134265 payload 824 2023-10-30T12:10:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91b41caba0 msgr2=0x7f91b411fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91b41caba0 0x7f91b411fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a00052a0 msgr2=0x7f91b41ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a00052a0 0x7f91b41ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 monclient: found mon.noname-a 2023-10-30T12:10:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b89cd640 1 -- 172.21.15.50:0/4031278486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b4135550 con 0x7f91b41ca780 2023-10-30T12:10:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.191+0000 7f91b99cf640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91b41caba0 0x7f91b411fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91b89cd640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b41ca780 0x7f91b411f7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f91a4007320 tx=0x7f91a40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 client.?.objecter ms_handle_connect 0x7f91b41ca780 2023-10-30T12:10:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a400f960 con 0x7f91b41ca780 2023-10-30T12:10:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: dump: 2023-10-30T12:10:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.244 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a400fac0 con 0x7f91b41ca780 2023-10-30T12:10:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 5 monclient: authenticate success, global_id 134265 2023-10-30T12:10:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a400fe30 con 0x7f91b41ca780 2023-10-30T12:10:22.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 10 monclient: _renew_subs 2023-10-30T12:10:22.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b40a1790 con 0x7f91b41ca780 2023-10-30T12:10:22.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91ab7fe640 4 set_mon_vals no callback set 2023-10-30T12:10:22.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91a9ffb640 10 monclient: dump: 2023-10-30T12:10:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.249 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 10 client.134265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 10 monclient: _renew_subs 2023-10-30T12:10:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b4121440 con 0x7f91b41ca780 2023-10-30T12:10:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.192+0000 7f91bb432640 1 librados: init done 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f91a401d460 con 0x7f91b41ca780 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91980392f0 0x7f919803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f91a404ad50 con 0x7f91b41ca780 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 10 client.134265.objecter ms_dispatch 0x7f91b4000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 3 client.134265.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 3 client.134265.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91bb432640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91bb432640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b4120220 0x7f91b4120600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91bb432640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b4073ff0 con 0x7f91b4120220 2023-10-30T12:10:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 20 client.134265.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b4120220 0x7f91b4120600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient: get_auth_request con 0x7f91b4120220 auth_method 0 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b91ce640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91980392f0 0x7f919803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b91ce640 10 monclient: get_auth_request con 0x7f91980392f0 auth_method 0 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 10 monclient(hunting): handle_auth_done global_id 134268 payload 824 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 1 -- 172.21.15.50:0/4031278486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91b409a5c0 con 0x7f91b4120220 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91b99cf640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b4120220 0x7f91b4120600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f91b406b400 tx=0x7f91ac0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 10 client.134265.objecter ms_handle_connect 0x7f91b4120220 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 10 client.134265.objecter resend_mon_ops 2023-10-30T12:10:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac03e070 con 0x7f91b4120220 2023-10-30T12:10:22.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.193+0000 7f91a9ffb640 10 monclient: dump: 2023-10-30T12:10:22.255 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.255 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 10 client.134265.objecter ms_handle_connect 0x7f91980392f0 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91b91ce640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91980392f0 0x7f919803b7b0 secure :-1 s=READY pgs=4093 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009c20 tx=0x7f91b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f91ac007db0 con 0x7f91b4120220 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b4120220 msgr2=0x7f91b4120600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b4120220 0x7f91b4120600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f91b406b400 tx=0x7f91ac0014a0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b4120220 msgr2=0x7f91b4120600 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.194+0000 7f91a9ffb640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b4120220 0x7f91b4120600 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f91b406b400 tx=0x7f91ac0014a0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91bb432640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91bb432640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b409dd70 0x7f91b409e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91bb432640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b4099b40 con 0x7f91b409dd70 2023-10-30T12:10:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b409dd70 0x7f91b409e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient: get_auth_request con 0x7f91b409dd70 auth_method 0 2023-10-30T12:10:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 10 monclient(hunting): handle_auth_done global_id 134271 payload 824 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.200+0000 7f91b99cf640 1 -- 172.21.15.50:0/4031278486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f91b406b060 con 0x7f91b409dd70 2023-10-30T12:10:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91b99cf640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b409dd70 0x7f91b409e180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f91ac040000 tx=0x7f91ac031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 10 client.134265.objecter ms_handle_connect 0x7f91b409dd70 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 10 client.134265.objecter resend_mon_ops 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac052070 con 0x7f91b409dd70 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 10 monclient: dump: 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.266 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4803 (secure 0 0 0) 0x7f91ac039ca0 con 0x7f91b409dd70 2023-10-30T12:10:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b409dd70 msgr2=0x7f91b409e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b409dd70 0x7f91b409e180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f91ac040000 tx=0x7f91ac031000 comp rx=0 tx=0).stop 2023-10-30T12:10:22.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b409dd70 msgr2=0x7f91b409e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:22.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91a9ffb640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b409dd70 0x7f91b409e180 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f91ac040000 tx=0x7f91ac031000 comp rx=0 tx=0).stop 2023-10-30T12:10:22.268 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:22.268 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 94, 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:22.269 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:10:22.270 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.271 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.272 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.273 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.274 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.275 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:22.276 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:22.277 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:22.278 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.279 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.280 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.281 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.281 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:22.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.281 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:22.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.281 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:22.282 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:22.282 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.282 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.282 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.283 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.284 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.286 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.288 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.289 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:22.290 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:22.291 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:22.292 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.293 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.293 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.293 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:22.293 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.295 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.296 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:22.296 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.296 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.296 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:10:22.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.297 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:10:22.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.299 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:22.300 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.300 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.300 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:22.300 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 10 librados: watch_flush enter 2023-10-30T12:10:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 10 librados: watch_flush exit 2023-10-30T12:10:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 20 client.134265.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 10 client.134265.objecter successfully canceled tick 2023-10-30T12:10:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91980392f0 msgr2=0x7f919803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91980392f0 0x7f919803b7b0 secure :-1 s=READY pgs=4093 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009c20 tx=0x7f91b0008040 comp rx=0 tx=0).stop 2023-10-30T12:10:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 10 monclient: shutdown 2023-10-30T12:10:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b41ca780 msgr2=0x7f91b411f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 --2- 172.21.15.50:0/4031278486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f91b41ca780 0x7f91b411f7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f91a4007320 tx=0x7f91a40077e0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91b99cf640 1 -- 172.21.15.50:0/4031278486 reap_dead start 2023-10-30T12:10:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91b99cf640 1 -- 172.21.15.50:0/4031278486 reap_dead start 2023-10-30T12:10:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 shutdown_connections 2023-10-30T12:10:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 >> 172.21.15.50:0/4031278486 conn(0x7f91b40a80b0 msgr2=0x7f91b40a5950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 shutdown_connections 2023-10-30T12:10:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 -- 172.21.15.50:0/4031278486 wait complete. 2023-10-30T12:10:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.201+0000 7f91bb432640 1 librados: shutdown 2023-10-30T12:10:22.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:10:22.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:10:22.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:10:22.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.410+0000 7ffa56373640 10 monclient: get_monmap_and_config 2023-10-30T12:10:22.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.410+0000 7ffa56373640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.410+0000 7ffa56373640 10 monclient: monmap: 2023-10-30T12:10:22.413 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.413 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.413 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:22.411842+0000 2023-10-30T12:10:22.413 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:22.411842+0000 2023-10-30T12:10:22.414 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.414 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.414 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:22.414 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:22.414 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:22.415 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:22.415 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:22.415 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.411+0000 7ffa56373640 1 Processor -- start 2023-10-30T12:10:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.411+0000 7ffa56373640 1 -- start start 2023-10-30T12:10:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.411+0000 7ffa56373640 10 monclient: init 2023-10-30T12:10:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.411+0000 7ffa56373640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.411+0000 7ffa56373640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T12:10:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5009bb70 0x7ffa5009bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 10 monclient(hunting): picked mon.noname-b con 0x7ffa5009bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffa5009d3f0 0x7ffa5009c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 10 monclient(hunting): picked mon.noname-c con 0x7ffa5009d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa50096080 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa50098170 con 0x7ffa50096080 2023-10-30T12:10:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa5009c9d0 con 0x7ffa5009d3f0 2023-10-30T12:10:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5009cf60 con 0x7ffa5009bb70 2023-10-30T12:10:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa56373640 10 monclient(hunting): authenticate will time out at 1467.595104s 2023-10-30T12:10:22.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5009bb70 0x7ffa5009bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5009bb70 0x7ffa5009bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52904/0 (socket says 172.21.15.50:52904) 2023-10-30T12:10:22.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2814182780 learned_addr learned my addr 172.21.15.50:0/2814182780 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:22.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): get_auth_request con 0x7ffa5009bb70 auth_method 0 2023-10-30T12:10:22.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4f7fe640 1 --2- 172.21.15.50:0/2814182780 >> v2:172.21.15.118:3300/0 conn(0x7ffa5009d3f0 0x7ffa5009c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_done global_id 114479 payload 274 2023-10-30T12:10:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2814182780 >> v2:172.21.15.118:3300/0 conn(0x7ffa5009d3f0 msgr2=0x7ffa5009c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 --2- 172.21.15.50:0/2814182780 >> v2:172.21.15.118:3300/0 conn(0x7ffa5009d3f0 0x7ffa5009c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2814182780 >> v1:172.21.15.50:6789/0 conn(0x7ffa50096080 legacy=0x7ffa500964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 monclient: found mon.noname-b 2023-10-30T12:10:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:10:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2814182780 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa50097b20 con 0x7ffa5009bb70 2023-10-30T12:10:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4effd640 1 -- 172.21.15.50:0/2814182780 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439586772 0 0) 0x7ffa50098170 con 0x7ffa50096080 2023-10-30T12:10:22.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:10:22.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4f7fe640 10 monclient: get_auth_request con 0x7ffa5009d3f0 auth_method 0 2023-10-30T12:10:22.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.412+0000 7ffa4f7fe640 1 --2- 172.21.15.50:0/2814182780 >> v2:172.21.15.118:3300/0 conn(0x7ffa5009d3f0 0x7ffa5009c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:10:22.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4ffff640 1 --2- 172.21.15.50:0/2814182780 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5009bb70 0x7ffa5009bf50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400baf0 tx=0x7ffa4402e010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d758766f8430b81c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 1 -- 172.21.15.50:0/2814182780 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4402edc0 con 0x7ffa5009bb70 2023-10-30T12:10:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: dump: 2023-10-30T12:10:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.433 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 5 monclient: authenticate success, global_id 114479 2023-10-30T12:10:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 1 -- 172.21.15.50:0/2814182780 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa44004030 con 0x7ffa5009bb70 2023-10-30T12:10:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 4 set_mon_vals no callback set 2023-10-30T12:10:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 1 -- 172.21.15.50:0/2814182780 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa440043e0 con 0x7ffa5009bb70 2023-10-30T12:10:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa4effd640 10 monclient: dump: 2023-10-30T12:10:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.437 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.437 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.437 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.437 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.437 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 10 monclient: shutdown 2023-10-30T12:10:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 1 -- 172.21.15.50:0/2814182780 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5009bb70 msgr2=0x7ffa5009bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 1 --2- 172.21.15.50:0/2814182780 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5009bb70 0x7ffa5009bf50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400baf0 tx=0x7ffa4402e010 comp rx=0 tx=0).stop 2023-10-30T12:10:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 1 -- 172.21.15.50:0/2814182780 shutdown_connections 2023-10-30T12:10:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 1 --2- 172.21.15.50:0/2814182780 >> v2:172.21.15.118:3300/0 conn(0x7ffa5009d3f0 0x7ffa5009c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 1 --2- 172.21.15.50:0/2814182780 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5009bb70 0x7ffa5009bf50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.413+0000 7ffa56373640 1 -- 172.21.15.50:0/2814182780 >> 172.21.15.50:0/2814182780 conn(0x7ffa500a3980 msgr2=0x7ffa500a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.414+0000 7ffa56373640 1 -- 172.21.15.50:0/2814182780 shutdown_connections 2023-10-30T12:10:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.414+0000 7ffa56373640 1 -- 172.21.15.50:0/2814182780 wait complete. 2023-10-30T12:10:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.414+0000 7ffa56373640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.414+0000 7ffa56373640 10 monclient: monmap: 2023-10-30T12:10:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:22.441 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.414+0000 7ffa56373640 1 librados: starting msgr at 2023-10-30T12:10:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.414+0000 7ffa56373640 1 librados: starting objecter 2023-10-30T12:10:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.414+0000 7ffa56373640 1 Processor -- start 2023-10-30T12:10:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 -- start start 2023-10-30T12:10:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 librados: setting wanted keys 2023-10-30T12:10:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 librados: calling monclient init 2023-10-30T12:10:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient: init 2023-10-30T12:10:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:10:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa50096080 0x7ffa501ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): picked mon.noname-c con 0x7ffa50096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa5009bb70 0x7ffa50095490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): picked mon.noname-b con 0x7ffa5009bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa5009d3f0 0x7ffa500959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): picked mon.noname-a con 0x7ffa5009d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa50095f10 con 0x7ffa5009d3f0 2023-10-30T12:10:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa50092a40 con 0x7ffa5009bb70 2023-10-30T12:10:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa50097120 con 0x7ffa50096080 2023-10-30T12:10:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa56373640 10 monclient(hunting): authenticate will time out at 1467.598506s 2023-10-30T12:10:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa50096080 0x7ffa501ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa5009d3f0 0x7ffa500959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa50096080 0x7ffa501ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52918/0 (socket says 172.21.15.50:52918) 2023-10-30T12:10:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa5009d3f0 0x7ffa500959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34118/0 (socket says 172.21.15.50:34118) 2023-10-30T12:10:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2427409096 learned_addr learned my addr 172.21.15.50:0/2427409096 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): get_auth_request con 0x7ffa50096080 auth_method 0 2023-10-30T12:10:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 10 monclient(hunting): get_auth_request con 0x7ffa5009d3f0 auth_method 0 2023-10-30T12:10:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient(hunting): handle_auth_done global_id 114482 payload 824 2023-10-30T12:10:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa5009bb70 msgr2=0x7ffa50095490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa5009bb70 0x7ffa50095490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa5009d3f0 msgr2=0x7ffa500959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa5009d3f0 0x7ffa500959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 monclient: found mon.noname-c 2023-10-30T12:10:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:10:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa4ffff640 1 -- 172.21.15.50:0/2427409096 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4400b700 con 0x7ffa50096080 2023-10-30T12:10:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.415+0000 7ffa54910640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa5009d3f0 0x7ffa500959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4ffff640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa50096080 0x7ffa501ca1b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffa4402e050 tx=0x7ffa440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 client.?.objecter ms_handle_connect 0x7ffa50096080 2023-10-30T12:10:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa44043070 con 0x7ffa50096080 2023-10-30T12:10:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:10:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: dump: 2023-10-30T12:10:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.454 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.454 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.454 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.454 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.454 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.455 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4402edd0 con 0x7ffa50096080 2023-10-30T12:10:22.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 5 monclient: authenticate success, global_id 114482 2023-10-30T12:10:22.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 10 monclient: _renew_subs 2023-10-30T12:10:22.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:22.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa500976e0 con 0x7ffa50096080 2023-10-30T12:10:22.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa44003830 con 0x7ffa50096080 2023-10-30T12:10:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4cff9640 10 monclient: dump: 2023-10-30T12:10:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.458 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.458 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.458 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.458 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.458 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.459 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.459 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa4e7fc640 4 set_mon_vals no callback set 2023-10-30T12:10:22.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 10 client.114482.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:22.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 10 monclient: _renew_subs 2023-10-30T12:10:22.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:10:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa50092760 con 0x7ffa50096080 2023-10-30T12:10:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.416+0000 7ffa56373640 1 librados: init done 2023-10-30T12:10:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa56373640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa56373640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008f3e0 0x7ffa5008f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa56373640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa50073ff0 con 0x7ffa5008f3e0 2023-10-30T12:10:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ffa44003990 con 0x7ffa50096080 2023-10-30T12:10:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008f3e0 0x7ffa5008f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient: get_auth_request con 0x7ffa5008f3e0 auth_method 0 2023-10-30T12:10:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa3c0392f0 0x7ffa3c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_done global_id 114485 payload 824 2023-10-30T12:10:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 1 -- 172.21.15.50:0/2427409096 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa50097560 con 0x7ffa5008f3e0 2023-10-30T12:10:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7ffa4406cb10 con 0x7ffa50096080 2023-10-30T12:10:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 10 client.114482.objecter ms_dispatch 0x7ffa50000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 3 client.114482.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 3 client.114482.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 20 client.114482.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 10 client.114482.objecter ms_handle_connect 0x7ffa5008f3e0 2023-10-30T12:10:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 10 client.114482.objecter resend_mon_ops 2023-10-30T12:10:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4f7fe640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008f3e0 0x7ffa5008f7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090550 tx=0x7ffa4000cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa54910640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa3c0392f0 0x7ffa3c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40007890 con 0x7ffa5008f3e0 2023-10-30T12:10:22.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:22.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa4cff9640 10 monclient: dump: 2023-10-30T12:10:22.465 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.467 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.467 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.467 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.467 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.417+0000 7ffa54910640 10 monclient: get_auth_request con 0x7ffa3c0392f0 auth_method 0 2023-10-30T12:10:22.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ffa400079f0 con 0x7ffa5008f3e0 2023-10-30T12:10:22.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008f3e0 msgr2=0x7ffa5008f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008f3e0 0x7ffa5008f7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090550 tx=0x7ffa4000cb80 comp rx=0 tx=0).stop 2023-10-30T12:10:22.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008f3e0 msgr2=0x7ffa5008f7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:22.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008f3e0 0x7ffa5008f7c0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090550 tx=0x7ffa4000cb80 comp rx=0 tx=0).stop 2023-10-30T12:10:22.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa4cff9640 10 client.114482.objecter ms_handle_connect 0x7ffa3c0392f0 2023-10-30T12:10:22.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.418+0000 7ffa54910640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa3c0392f0 0x7ffa3c03b7b0 secure :-1 s=READY pgs=4094 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090db0 tx=0x7ffa48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa56373640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa56373640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008fd00 0x7ffa5006dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa56373640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa50097e00 con 0x7ffa5008fd00 2023-10-30T12:10:22.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008fd00 0x7ffa5006dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient: get_auth_request con 0x7ffa5008fd00 auth_method 0 2023-10-30T12:10:22.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 10 monclient(hunting): handle_auth_done global_id 114488 payload 824 2023-10-30T12:10:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.424+0000 7ffa4f7fe640 1 -- 172.21.15.50:0/2427409096 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa5004e160 con 0x7ffa5008fd00 2023-10-30T12:10:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 10 client.114482.objecter ms_handle_connect 0x7ffa5008fd00 2023-10-30T12:10:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 10 client.114482.objecter resend_mon_ops 2023-10-30T12:10:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4f7fe640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008fd00 0x7ffa5006dbb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090550 tx=0x7ffa400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40028070 con 0x7ffa5008fd00 2023-10-30T12:10:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 10 monclient: dump: 2023-10-30T12:10:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.476 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.476 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.476 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.476 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.476 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.477 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7ffa40020040 con 0x7ffa5008fd00 2023-10-30T12:10:22.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:22.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008fd00 msgr2=0x7ffa5006dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008fd00 0x7ffa5006dbb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090550 tx=0x7ffa400046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008fd00 msgr2=0x7ffa5006dbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:22.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa4cff9640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa5008fd00 0x7ffa5006dbb0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090550 tx=0x7ffa400046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.478 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:22.478 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:22.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:22.479 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:10:22.479 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 94, 2023-10-30T12:10:22.479 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:22.479 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:22.480 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:10:22.480 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:22.480 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.480 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:10:22.480 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:22.480 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:22.481 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:22.481 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.481 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.481 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.481 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.481 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.482 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.482 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.482 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.482 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.482 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.484 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.484 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.484 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.484 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.484 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.484 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.485 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.485 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.485 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.485 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:22.486 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:22.486 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:22.486 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:22.486 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:22.486 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:22.487 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:22.487 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:22.487 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:22.487 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:22.487 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:22.487 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:22.488 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:22.488 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:22.488 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:22.488 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:22.488 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:22.489 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.489 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.489 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.489 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.489 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.489 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.490 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.490 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.490 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.490 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.490 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.491 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:22.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.491 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:22.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:22.491 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:22.492 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.492 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.492 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.492 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:22.492 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.493 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.493 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:22.493 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.493 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.494 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.494 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:22.494 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:22.494 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.495 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.495 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.495 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:22.495 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:22.496 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.496 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.496 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:22.497 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.497 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:22.498 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.498 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.498 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:22.499 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:22.499 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.499 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.499 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.500 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:22.500 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:22.500 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.500 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:22.501 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.502 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:22.502 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.502 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.502 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.503 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.504 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:22.504 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:22.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.505 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.505 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.505 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:22.505 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.505 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.506 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:10:22.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.506 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.506 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.506 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:10:22.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.507 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.507 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:22.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.507 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.507 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.508 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:10:22.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.508 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:22.509 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 10 librados: watch_flush enter 2023-10-30T12:10:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 10 librados: watch_flush exit 2023-10-30T12:10:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 20 client.114482.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 10 client.114482.objecter successfully canceled tick 2023-10-30T12:10:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa3c0392f0 msgr2=0x7ffa3c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa3c0392f0 0x7ffa3c03b7b0 secure :-1 s=READY pgs=4094 cs=0 l=1 rev1=1 crypto rx=0x7ffa50090db0 tx=0x7ffa48009040 comp rx=0 tx=0).stop 2023-10-30T12:10:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 10 monclient: shutdown 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa50096080 msgr2=0x7ffa501ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 --2- 172.21.15.50:0/2427409096 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa50096080 0x7ffa501ca1b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffa4402e050 tx=0x7ffa440046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa54910640 1 -- 172.21.15.50:0/2427409096 reap_dead start 2023-10-30T12:10:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa54910640 1 -- 172.21.15.50:0/2427409096 reap_dead start 2023-10-30T12:10:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 shutdown_connections 2023-10-30T12:10:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 >> 172.21.15.50:0/2427409096 conn(0x7ffa500a3980 msgr2=0x7ffa501b9660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 shutdown_connections 2023-10-30T12:10:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.425+0000 7ffa56373640 1 -- 172.21.15.50:0/2427409096 wait complete. 2023-10-30T12:10:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.426+0000 7ffa56373640 1 librados: shutdown 2023-10-30T12:10:22.514 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:10:22.514 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:10:22.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:10:22.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.620+0000 7fb63b01a640 10 monclient: get_monmap_and_config 2023-10-30T12:10:22.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.620+0000 7fb63b01a640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.620+0000 7fb63b01a640 10 monclient: monmap: 2023-10-30T12:10:22.622 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.622 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:22.621536+0000 2023-10-30T12:10:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:22.621536+0000 2023-10-30T12:10:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:22.624 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:22.624 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:22.624 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:22.624 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:22.624 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 Processor -- start 2023-10-30T12:10:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 -- start start 2023-10-30T12:10:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient: init 2023-10-30T12:10:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T12:10:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb634097780 0x7fb634097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): picked mon.noname-c con 0x7fb634097780 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb634073b10 0x7fb6340a0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): picked mon.noname-a con 0x7fb634073b10 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): picked mon.noname-b con 0x7fb63409ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb634071ec0 con 0x7fb634073b10 2023-10-30T12:10:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63409bd10 con 0x7fb634097780 2023-10-30T12:10:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63409bff0 con 0x7fb63409ccd0 2023-10-30T12:10:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb63b01a640 10 monclient(hunting): authenticate will time out at 1467.804742s 2023-10-30T12:10:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb6395b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb634073b10 0x7fb6340a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb6395b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52936/0 (socket says 172.21.15.50:52936) 2023-10-30T12:10:22.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb634073b10 0x7fb6340a0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34120/0 (socket says 172.21.15.50:34120) 2023-10-30T12:10:22.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb6395b7640 1 -- 172.21.15.50:0/1304052127 learned_addr learned my addr 172.21.15.50:0/1304052127 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:22.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb6395b7640 10 monclient(hunting): get_auth_request con 0x7fb63409ccd0 auth_method 0 2023-10-30T12:10:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb6395b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb6395b7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb6395b7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 10 monclient(hunting): get_auth_request con 0x7fb634073b10 auth_method 0 2023-10-30T12:10:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.621+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb6395b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb6395b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb6395b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb638db6640 1 --2- 172.21.15.50:0/1304052127 >> v2:172.21.15.118:3300/0 conn(0x7fb634097780 0x7fb634097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb638db6640 10 monclient(hunting): get_auth_request con 0x7fb634097780 auth_method 0 2023-10-30T12:10:22.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb638db6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb638db6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb638db6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 10 monclient(hunting): handle_auth_done global_id 134280 payload 274 2023-10-30T12:10:22.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 1 -- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 msgr2=0x7fb6340a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 1 --2- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 1 -- 172.21.15.50:0/1304052127 >> v2:172.21.15.118:3300/0 conn(0x7fb634097780 msgr2=0x7fb634097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 1 --2- 172.21.15.50:0/1304052127 >> v2:172.21.15.118:3300/0 conn(0x7fb634097780 0x7fb634097b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 1 monclient: found mon.noname-a 2023-10-30T12:10:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 1 -- 172.21.15.50:0/1304052127 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6340980a0 con 0x7fb634073b10 2023-10-30T12:10:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb638db6640 1 --2- 172.21.15.50:0/1304052127 >> v2:172.21.15.118:3300/0 conn(0x7fb634097780 0x7fb634097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:10:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb6395b7640 1 --2- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.622+0000 7fb633fff640 1 --2- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6340a0080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb63406a280 tx=0x7fb62802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340c41a65123e651 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 1 -- 172.21.15.50:0/1304052127 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628030710 con 0x7fb634073b10 2023-10-30T12:10:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: dump: 2023-10-30T12:10:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.647 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.647 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.647 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 5 monclient: authenticate success, global_id 134280 2023-10-30T12:10:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 1 -- 172.21.15.50:0/1304052127 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62802e050 con 0x7fb634073b10 2023-10-30T12:10:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 4 set_mon_vals no callback set 2023-10-30T12:10:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 1 -- 172.21.15.50:0/1304052127 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628030a90 con 0x7fb634073b10 2023-10-30T12:10:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb6337fe640 10 monclient: dump: 2023-10-30T12:10:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 10 monclient: shutdown 2023-10-30T12:10:22.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 -- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 msgr2=0x7fb6340a0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6340a0080 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb63406a280 tx=0x7fb62802b710 comp rx=0 tx=0).stop 2023-10-30T12:10:22.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 -- 172.21.15.50:0/1304052127 shutdown_connections 2023-10-30T12:10:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1304052127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6340a0080 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1304052127 >> v2:172.21.15.118:3300/0 conn(0x7fb634097780 0x7fb634097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 -- 172.21.15.50:0/1304052127 >> 172.21.15.50:0/1304052127 conn(0x7fb6340a1170 msgr2=0x7fb6340a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 -- 172.21.15.50:0/1304052127 shutdown_connections 2023-10-30T12:10:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.623+0000 7fb63b01a640 1 -- 172.21.15.50:0/1304052127 wait complete. 2023-10-30T12:10:22.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 10 monclient: monmap: 2023-10-30T12:10:22.654 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.654 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:22.654 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:22.656 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 1 librados: starting msgr at 2023-10-30T12:10:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 1 librados: starting objecter 2023-10-30T12:10:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 1 Processor -- start 2023-10-30T12:10:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 1 -- start start 2023-10-30T12:10:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 1 librados: setting wanted keys 2023-10-30T12:10:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 1 librados: calling monclient init 2023-10-30T12:10:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.624+0000 7fb63b01a640 10 monclient: init 2023-10-30T12:10:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6341ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): picked mon.noname-a con 0x7fb634073b10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb634097780 0x7fb634094460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): picked mon.noname-b con 0x7fb634097780 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340949a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): picked mon.noname-c con 0x7fb63409ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63408d0b0 con 0x7fb634073b10 2023-10-30T12:10:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb634073f70 con 0x7fb634097780 2023-10-30T12:10:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340a3ad0 con 0x7fb63409ccd0 2023-10-30T12:10:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb63b01a640 10 monclient(hunting): authenticate will time out at 1467.808279s 2023-10-30T12:10:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6341ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6341ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34122/0 (socket says 172.21.15.50:34122) 2023-10-30T12:10:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 -- 172.21.15.50:0/1724272131 learned_addr learned my addr 172.21.15.50:0/1724272131 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): get_auth_request con 0x7fb634073b10 auth_method 0 2023-10-30T12:10:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340949a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 10 monclient(hunting): get_auth_request con 0x7fb63409ccd0 auth_method 0 2023-10-30T12:10:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6395b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient(hunting): handle_auth_done global_id 134283 payload 824 2023-10-30T12:10:22.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 msgr2=0x7fb6340949a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb63409ccd0 0x7fb6340949a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb634097780 msgr2=0x7fb634094460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb633fff640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb634097780 0x7fb634094460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb634097780 0x7fb634094460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 monclient: found mon.noname-a 2023-10-30T12:10:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 -- 172.21.15.50:0/1724272131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c009b30 con 0x7fb634073b10 2023-10-30T12:10:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb638db6640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6341ca420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb62400e7e0 tx=0x7fb62400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6317fa640 10 client.?.objecter ms_handle_connect 0x7fb634073b10 2023-10-30T12:10:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6317fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624004030 con 0x7fb634073b10 2023-10-30T12:10:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.625+0000 7fb6317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: dump: 2023-10-30T12:10:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.669 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb624004190 con 0x7fb634073b10 2023-10-30T12:10:22.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 5 monclient: authenticate success, global_id 134283 2023-10-30T12:10:22.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 10 monclient: _renew_subs 2023-10-30T12:10:22.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb628009400 con 0x7fb634073b10 2023-10-30T12:10:22.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624004520 con 0x7fb634073b10 2023-10-30T12:10:22.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 10 monclient: dump: 2023-10-30T12:10:22.671 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.671 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.672 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.672 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.672 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.672 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.673 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 10 client.134283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 10 monclient: _renew_subs 2023-10-30T12:10:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb632ffd640 4 set_mon_vals no callback set 2023-10-30T12:10:22.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63408e650 con 0x7fb634073b10 2023-10-30T12:10:22.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 1 librados: init done 2023-10-30T12:10:22.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb62400c6d0 con 0x7fb634073b10 2023-10-30T12:10:22.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408afd0 0x7fb63408b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.626+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63408e980 con 0x7fb63408afd0 2023-10-30T12:10:22.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb6200392f0 0x7fb62003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fb62404a970 con 0x7fb634073b10 2023-10-30T12:10:22.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 10 client.134283.objecter ms_dispatch 0x7fb634000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:22.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 3 client.134283.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:22.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 3 client.134283.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:22.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408afd0 0x7fb63408b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient: get_auth_request con 0x7fb63408afd0 auth_method 0 2023-10-30T12:10:22.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 20 client.134283.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6395b7640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb6200392f0 0x7fb62003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6395b7640 10 monclient: get_auth_request con 0x7fb6200392f0 auth_method 0 2023-10-30T12:10:22.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 10 monclient(hunting): handle_auth_done global_id 144157 payload 824 2023-10-30T12:10:22.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 1 -- 172.21.15.50:0/1724272131 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6340a0b00 con 0x7fb63408afd0 2023-10-30T12:10:22.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 10 client.134283.objecter ms_handle_connect 0x7fb63408afd0 2023-10-30T12:10:22.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb633fff640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408afd0 0x7fb63408b3b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb63406a230 tx=0x7fb6280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 10 client.134283.objecter resend_mon_ops 2023-10-30T12:10:22.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 10 client.134283.objecter ms_handle_connect 0x7fb6200392f0 2023-10-30T12:10:22.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6395b7640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb6200392f0 0x7fb62003b7b0 secure :-1 s=READY pgs=4095 cs=0 l=1 rev1=1 crypto rx=0x7fb63408e440 tx=0x7fb62c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62803d890 con 0x7fb63408afd0 2023-10-30T12:10:22.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:22.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.627+0000 7fb6317fa640 10 monclient: dump: 2023-10-30T12:10:22.680 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.681 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.681 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.681 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.681 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.682 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.682 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.682 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.628+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb62803d9f0 con 0x7fb63408afd0 2023-10-30T12:10:22.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.628+0000 7fb6317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.628+0000 7fb6317fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:22.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.628+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408afd0 msgr2=0x7fb63408b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.628+0000 7fb6317fa640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408afd0 0x7fb63408b3b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb63406a230 tx=0x7fb6280046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.628+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408afd0 msgr2=0x7fb63408b3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:22.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.628+0000 7fb6317fa640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408afd0 0x7fb63408b3b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb63b01a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:22.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408b8f0 0x7fb634098570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb63b01a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63406a5a0 con 0x7fb63408b8f0 2023-10-30T12:10:22.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb633fff640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408b8f0 0x7fb634098570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb633fff640 10 monclient: get_auth_request con 0x7fb63408b8f0 auth_method 0 2023-10-30T12:10:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb633fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb633fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.634+0000 7fb633fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb633fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb633fff640 10 monclient(hunting): handle_auth_done global_id 144160 payload 824 2023-10-30T12:10:22.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb633fff640 1 -- 172.21.15.50:0/1724272131 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb63404e160 con 0x7fb63408b8f0 2023-10-30T12:10:22.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb633fff640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408b8f0 0x7fb634098570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb62802b750 tx=0x7fb628009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb6317fa640 10 client.134283.objecter ms_handle_connect 0x7fb63408b8f0 2023-10-30T12:10:22.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb6317fa640 10 client.134283.objecter resend_mon_ops 2023-10-30T12:10:22.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62804c040 con 0x7fb63408b8f0 2023-10-30T12:10:22.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb6317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb6317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:10:22.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.635+0000 7fb6317fa640 10 monclient: dump: 2023-10-30T12:10:22.689 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.690 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.690 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.690 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.690 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.691 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.691 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.691 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fb628051480 con 0x7fb63408b8f0 2023-10-30T12:10:22.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:22.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6317fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:22.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408b8f0 msgr2=0x7fb634098570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6317fa640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408b8f0 0x7fb634098570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb62802b750 tx=0x7fb628009190 comp rx=0 tx=0).stop 2023-10-30T12:10:22.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6317fa640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408b8f0 msgr2=0x7fb634098570 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:22.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6317fa640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb63408b8f0 0x7fb634098570 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb62802b750 tx=0x7fb628009190 comp rx=0 tx=0).stop 2023-10-30T12:10:22.693 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:22.693 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:22.693 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:22.693 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:10:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 94, 2023-10-30T12:10:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:10:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:10:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:22.703 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:22.703 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:22.703 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:22.703 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:22.703 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:22.704 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:22.704 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:22.704 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:22.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:22.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:22.704 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:22.705 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:22.705 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:22.705 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:22.705 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:22.705 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.706 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:22.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.706 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:22.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.706 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:22.707 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:22.707 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.707 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.707 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.707 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:22.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:22.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.709 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:22.710 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:22.710 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.710 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.710 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.711 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:22.711 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:22.711 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.711 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.712 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:22.712 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.713 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.713 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:22.713 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.713 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.714 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:22.714 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:22.714 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.714 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.715 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.715 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:22.715 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:22.715 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:22.716 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:22.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:22.716 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:22.716 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.717 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:22.717 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:22.717 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:22.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.718 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.718 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:22.718 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:22.718 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:22.719 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:22.719 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:22.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.719 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.721 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:22.721 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:22.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.721 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.722 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.722 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:22.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.722 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:22.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:22.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:22.723 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:22.723 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:22.723 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:10:22.723 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:22.724 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:22.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:22.724 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:22.724 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:22.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 10 librados: watch_flush enter 2023-10-30T12:10:22.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 10 librados: watch_flush exit 2023-10-30T12:10:22.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 20 client.134283.objecter shutdown clearing up homeless session... 2023-10-30T12:10:22.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 10 client.134283.objecter successfully canceled tick 2023-10-30T12:10:22.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb6200392f0 msgr2=0x7fb62003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb6200392f0 0x7fb62003b7b0 secure :-1 s=READY pgs=4095 cs=0 l=1 rev1=1 crypto rx=0x7fb63408e440 tx=0x7fb62c00b040 comp rx=0 tx=0).stop 2023-10-30T12:10:22.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 10 monclient: shutdown 2023-10-30T12:10:22.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 msgr2=0x7fb6341ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 --2- 172.21.15.50:0/1724272131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb634073b10 0x7fb6341ca420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb62400e7e0 tx=0x7fb62400eca0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6395b7640 1 -- 172.21.15.50:0/1724272131 reap_dead start 2023-10-30T12:10:22.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb6395b7640 1 -- 172.21.15.50:0/1724272131 reap_dead start 2023-10-30T12:10:22.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 shutdown_connections 2023-10-30T12:10:22.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 >> 172.21.15.50:0/1724272131 conn(0x7fb6340a1170 msgr2=0x7fb6341bd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 shutdown_connections 2023-10-30T12:10:22.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 -- 172.21.15.50:0/1724272131 wait complete. 2023-10-30T12:10:22.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.636+0000 7fb63b01a640 1 librados: shutdown 2023-10-30T12:10:22.729 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:10:22.730 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-10-30T12:10:22.730 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:10:22.730 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-10-30T12:10:22.730 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:10:22.731 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:10:22.731 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:10:22.743 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:10:22.743 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:10:22.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.846+0000 7f54a76f2640 10 monclient: get_monmap_and_config 2023-10-30T12:10:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.846+0000 7f54a76f2640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.846+0000 7f54a76f2640 10 monclient: monmap: 2023-10-30T12:10:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:22.848017+0000 2023-10-30T12:10:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:22.848017+0000 2023-10-30T12:10:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.850 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:22.850 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:22.850 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:22.850 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:22.850 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:22.851 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.847+0000 7f54a76f2640 1 Processor -- start 2023-10-30T12:10:22.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.847+0000 7f54a76f2640 1 -- start start 2023-10-30T12:10:22.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.847+0000 7f54a76f2640 10 monclient: init 2023-10-30T12:10:22.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-10-30T12:10:22.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54a00659d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:22.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54a0071410 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:22.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f54a0071850 0x7f54a013be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient(hunting): picked mon.noname-c con 0x7f54a0071850 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:22.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a009fb30 con 0x7f54a00659d0 2023-10-30T12:10:22.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a0065db0 con 0x7f54a0071410 2023-10-30T12:10:22.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a009e090 con 0x7f54a0071850 2023-10-30T12:10:22.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 10 monclient(hunting): authenticate will time out at 1468.031354s 2023-10-30T12:10:22.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a548e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f54a00659d0 0x7f54a012fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41170/0 (socket says 172.21.15.50:41170) 2023-10-30T12:10:22.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a548e640 1 -- 172.21.15.50:0/1355353313 learned_addr learned my addr 172.21.15.50:0/1355353313 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:10:22.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a4c8d640 1 -- 172.21.15.50:0/1355353313 >> v1:172.21.15.118:6789/0 conn(0x7f54a0071410 legacy=0x7f54a0138720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:22.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a5c8f640 1 -- 172.21.15.50:0/1355353313 >> v2:172.21.15.118:3300/0 conn(0x7f54a0071850 msgr2=0x7f54a013be50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:22.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a5c8f640 1 --2- 172.21.15.50:0/1355353313 >> v2:172.21.15.118:3300/0 conn(0x7f54a0071850 0x7f54a013be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:22.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762112013 0 0) 0x7f54a009fb30 con 0x7f54a00659d0 2023-10-30T12:10:22.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient(hunting): _negotiate 2023-10-30T12:10:22.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient(hunting): my global_id is 134286 2023-10-30T12:10:22.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5488003520 con 0x7f54a00659d0 2023-10-30T12:10:22.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959008991 0 0) 0x7f5488003520 con 0x7f54a00659d0 2023-10-30T12:10:22.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 >> v2:172.21.15.118:3300/0 conn(0x7f54a0071850 msgr2=0x7f54a013be50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:22.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 --2- 172.21.15.50:0/1355353313 >> v2:172.21.15.118:3300/0 conn(0x7f54a0071850 0x7f54a013be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 >> v1:172.21.15.118:6789/0 conn(0x7f54a0071410 legacy=0x7f54a0138720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:22.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:10:22.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:10:22.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a0130570 con 0x7f54a00659d0 2023-10-30T12:10:22.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f54a76f2640 5 monclient: authenticate success, global_id 134286 2023-10-30T12:10:22.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f5498001670 con 0x7f54a00659d0 2023-10-30T12:10:22.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:10:22.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: dump: 2023-10-30T12:10:22.867 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.867 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.868 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.868 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.868 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.868 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.868 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.869 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.869 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.869 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:10:22.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 >> v1:172.21.15.50:6789/0 conn(0x7f54a00659d0 legacy=0x7f54a012fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:22.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.848+0000 7f549ffff640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5488003970 0x7f5488003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): picked mon.c con 0x7f5488003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f5488004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): picked mon.a con 0x7f54880047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:22.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5488005380 0x7f548800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): picked mon.b con 0x7f5488005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:22.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548800c1f0 con 0x7f54880047c0 2023-10-30T12:10:22.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548800c360 con 0x7f5488005380 2023-10-30T12:10:22.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548800c520 con 0x7f5488003970 2023-10-30T12:10:22.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f549ffff640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:10:22.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a76f2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:22.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f5488004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5488003970 0x7f5488003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): get_auth_request con 0x7f54880047c0 auth_method 0 2023-10-30T12:10:22.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:22.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 10 monclient(hunting): get_auth_request con 0x7f5488003970 auth_method 0 2023-10-30T12:10:22.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:22.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a5c8f640 1 -- 172.21.15.50:0/1355353313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5488005380 msgr2=0x7f548800b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:22.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a5c8f640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5488005380 0x7f548800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:22.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:22.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a548e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:22.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient(hunting): handle_auth_done global_id 134286 payload 293 2023-10-30T12:10:22.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 1 -- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5488003970 msgr2=0x7f5488003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:22.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5488003970 0x7f5488003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 1 -- 172.21.15.50:0/1355353313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5488005380 msgr2=0x7f548800b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:22.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5488005380 0x7f548800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 1 monclient: found mon.a 2023-10-30T12:10:22.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 1 -- 172.21.15.50:0/1355353313 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5488003520 con 0x7f54880047c0 2023-10-30T12:10:22.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a5c8f640 1 -- 172.21.15.50:0/1355353313 reap_dead start 2023-10-30T12:10:22.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.849+0000 7f54a4c8d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a4c8d640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f5488004c40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5494009900 tx=0x7f54880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408a1dded536c69d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494003bb0 con 0x7f54880047c0 2023-10-30T12:10:22.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 10 monclient: dump: 2023-10-30T12:10:22.880 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.880 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.880 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.880 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.880 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.881 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.881 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.881 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.881 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.881 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:22.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5494003d10 con 0x7f54880047c0 2023-10-30T12:10:22.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:22.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 4 set_mon_vals no callback set 2023-10-30T12:10:22.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 1 -- 172.21.15.50:0/1355353313 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940052b0 con 0x7f54880047c0 2023-10-30T12:10:22.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f549ffff640 10 monclient: dump: 2023-10-30T12:10:22.883 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.883 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.884 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.884 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.884 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.884 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.884 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.885 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.885 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.885 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 10 monclient: shutdown 2023-10-30T12:10:22.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:22.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 1 -- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 msgr2=0x7f5488004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f5488004c40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5494009900 tx=0x7f54880057f0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 1 -- 172.21.15.50:0/1355353313 shutdown_connections 2023-10-30T12:10:22.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 1 --2- 172.21.15.50:0/1355353313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f5488004c40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 1 -- 172.21.15.50:0/1355353313 >> 172.21.15.50:0/1355353313 conn(0x7f54a00a47f0 msgr2=0x7f54a00a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:22.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 1 -- 172.21.15.50:0/1355353313 shutdown_connections 2023-10-30T12:10:22.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.850+0000 7f54a76f2640 1 -- 172.21.15.50:0/1355353313 wait complete. 2023-10-30T12:10:22.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 10 monclient: build_initial_monmap 2023-10-30T12:10:22.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 10 monclient: monmap: 2023-10-30T12:10:22.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:22.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:22.888 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:22.888 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:22.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:22.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:22.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:22.889 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:22.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 1 librados: starting msgr at 2023-10-30T12:10:22.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 1 librados: starting objecter 2023-10-30T12:10:22.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 1 Processor -- start 2023-10-30T12:10:22.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 1 -- start start 2023-10-30T12:10:22.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 1 librados: setting wanted keys 2023-10-30T12:10:22.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 1 librados: calling monclient init 2023-10-30T12:10:22.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.851+0000 7f54a76f2640 10 monclient: init 2023-10-30T12:10:22.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:22.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:10:22.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f54a01c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): picked mon.noname-a con 0x7f54880047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:22.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54a01c8330 0x7f54a00709f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): picked mon.noname-c con 0x7f54a01c8330 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:22.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a012d3c0 0x7f54a012aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): picked mon.noname-b con 0x7f54a012d3c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:22.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a01c8cf0 con 0x7f54880047c0 2023-10-30T12:10:22.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a006cc20 con 0x7f54a012d3c0 2023-10-30T12:10:22.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:22.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a006cf00 con 0x7f54a01c8330 2023-10-30T12:10:22.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:22.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a76f2640 10 monclient(hunting): authenticate will time out at 1468.035385s 2023-10-30T12:10:22.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f54a01c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54a01c8330 0x7f54a00709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f54a01c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34148/0 (socket says 172.21.15.50:34148) 2023-10-30T12:10:22.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 -- 172.21.15.50:0/432540781 learned_addr learned my addr 172.21.15.50:0/432540781 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:22.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 10 monclient(hunting): get_auth_request con 0x7f54a01c8330 auth_method 0 2023-10-30T12:10:22.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): get_auth_request con 0x7f54880047c0 auth_method 0 2023-10-30T12:10:22.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:22.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:22.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:22.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a5c8f640 1 -- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a012d3c0 msgr2=0x7f54a012aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:22.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a5c8f640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a012d3c0 0x7f54a012aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:22.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:22.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:22.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:22.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient(hunting): handle_auth_done global_id 134289 payload 824 2023-10-30T12:10:22.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 -- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54a01c8330 msgr2=0x7f54a00709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:22.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54a01c8330 0x7f54a00709f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 -- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a012d3c0 msgr2=0x7f54a012aba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:10:22.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a012d3c0 0x7f54a012aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:22.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient: _finish_hunting 0 2023-10-30T12:10:22.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 monclient: found mon.noname-a 2023-10-30T12:10:22.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:22.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 -- 172.21.15.50:0/432540781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a006d1e0 con 0x7f54880047c0 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a4c8d640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54a01c8330 0x7f54a00709f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f54a548e640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f54a01c7df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5498007290 tx=0x7f5498007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f549dffb640 10 client.?.objecter ms_handle_connect 0x7f54880047c0 2023-10-30T12:10:22.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.852+0000 7f549dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 1 -- 172.21.15.50:0/432540781 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498004ea0 con 0x7f54880047c0 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: dump: 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.901 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.902 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.902 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.902 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:22.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:22.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:22.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 5 monclient: authenticate success, global_id 134289 2023-10-30T12:10:22.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 1 -- 172.21.15.50:0/432540781 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549800f940 con 0x7f54880047c0 2023-10-30T12:10:22.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:22.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 1 -- 172.21.15.50:0/432540781 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549800fcf0 con 0x7f54880047c0 2023-10-30T12:10:22.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549dffb640 10 monclient: dump: 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:22.904 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:22.905 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f549f7fe640 4 set_mon_vals no callback set 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 10 monclient: _renew_subs 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a006d4c0 con 0x7f54880047c0 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 10 client.134289.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 10 monclient: _renew_subs 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a0071110 con 0x7f54880047c0 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.853+0000 7f54a76f2640 1 librados: init done 2023-10-30T12:10:22.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f549dffb640 1 -- 172.21.15.50:0/432540781 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f549801d460 con 0x7f54880047c0 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f549dffb640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f548c0392d0 0x7f548c03b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f549dffb640 1 -- 172.21.15.50:0/432540781 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f549804ab80 con 0x7f54880047c0 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f549dffb640 10 client.134289.objecter ms_dispatch 0x7f54a0000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f549dffb640 3 client.134289.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:22.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f549dffb640 3 client.134289.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f54a76f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f54a76f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f54a76f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a004e160 con 0x7f54880047c0 2023-10-30T12:10:22.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f549dffb640 20 client.134289.objecter dump_active .. 0 homeless 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f54a4c8d640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f548c0392d0 0x7f548c03b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.854+0000 7f54a4c8d640 10 monclient: get_auth_request con 0x7f548c0392d0 auth_method 0 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.855+0000 7f54a4c8d640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f548c0392d0 0x7f548c03b790 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 crypto rx=0x7f5494033000 tx=0x7f5494005000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:22.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.855+0000 7f549dffb640 10 client.134289.objecter ms_handle_connect 0x7f548c0392d0 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.859+0000 7f549dffb640 1 -- 172.21.15.50:0/432540781 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f549804ae60 con 0x7f54880047c0 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.859+0000 7f549dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:22.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:22.859+0000 7f549dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.101+0000 7f54a76f2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.101+0000 7f54a76f2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.101+0000 7f54a76f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.101+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f54a009e0e0 con 0x7f54880047c0 2023-10-30T12:10:23.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.101+0000 7f549dffb640 1 -- 172.21.15.50:0/432540781 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f5498012940 con 0x7f54880047c0 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.101+0000 7f549dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.101+0000 7f549dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:23.104 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 10 librados: watch_flush enter 2023-10-30T12:10:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 10 librados: watch_flush exit 2023-10-30T12:10:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 20 client.134289.objecter shutdown clearing up homeless session... 2023-10-30T12:10:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 10 client.134289.objecter successfully canceled tick 2023-10-30T12:10:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f548c0392d0 msgr2=0x7f548c03b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f548c0392d0 0x7f548c03b790 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 crypto rx=0x7f5494033000 tx=0x7f5494005000 comp rx=0 tx=0).stop 2023-10-30T12:10:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 10 monclient: shutdown 2023-10-30T12:10:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 msgr2=0x7f54a01c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f54a01c7df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5498007290 tx=0x7f5498007750 comp rx=0 tx=0).stop 2023-10-30T12:10:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 shutdown_connections 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f548c0392d0 0x7f548c03b790 unknown :-1 s=CLOSED pgs=4096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54a012d3c0 0x7f54a012aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54a01c8330 0x7f54a00709f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 --2- 172.21.15.50:0/432540781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54880047c0 0x7f54a01c7df0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 >> 172.21.15.50:0/432540781 conn(0x7f54a00a47f0 msgr2=0x7f54a01299e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 shutdown_connections 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 -- 172.21.15.50:0/432540781 wait complete. 2023-10-30T12:10:23.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:23.103+0000 7f54a76f2640 1 librados: shutdown 2023-10-30T12:10:26.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:26.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.231+0000 7f3a25421640 10 monclient: get_monmap_and_config 2023-10-30T12:10:26.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.231+0000 7f3a25421640 10 monclient: build_initial_monmap 2023-10-30T12:10:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.232+0000 7f3a25421640 10 monclient: monmap: 2023-10-30T12:10:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:26.233061+0000 2023-10-30T12:10:26.234 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:26.233061+0000 2023-10-30T12:10:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:26.235 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:26.236 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:26.236 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:26.236 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.232+0000 7f3a25421640 1 Processor -- start 2023-10-30T12:10:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.232+0000 7f3a25421640 1 -- start start 2023-10-30T12:10:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.232+0000 7f3a25421640 10 monclient: init 2023-10-30T12:10:26.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T12:10:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3a20098520 0x7f3a2009bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient(hunting): picked mon.noname-c con 0x7f3a20098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a20098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:26.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a20137980 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a20099930 con 0x7f3a20137980 2023-10-30T12:10:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2009aca0 con 0x7f3a20098900 2023-10-30T12:10:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a2009af80 con 0x7f3a20098520 2023-10-30T12:10:26.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a25421640 10 monclient(hunting): authenticate will time out at 1471.416176s 2023-10-30T12:10:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1e7fc640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f3a20098900 legacy=0x7f3a20098d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3a20137980 0x7f3a20139d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41186/0 (socket says 172.21.15.50:41186) 2023-10-30T12:10:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1f7fe640 1 -- 172.21.15.50:0/2170917364 learned_addr learned my addr 172.21.15.50:0/2170917364 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:10:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1effd640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f3a20098520 msgr2=0x7f3a2009bf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1effd640 1 --2- 172.21.15.50:0/2170917364 >> v2:172.21.15.118:3300/0 conn(0x7f3a20098520 0x7f3a2009bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347066577 0 0) 0x7f3a20099930 con 0x7f3a20137980 2023-10-30T12:10:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): _negotiate 2023-10-30T12:10:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:26.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): my global_id is 134292 2023-10-30T12:10:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a08005730 con 0x7f3a20137980 2023-10-30T12:10:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 318435480 0 0) 0x7f3a08005730 con 0x7f3a20137980 2023-10-30T12:10:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 >> v2:172.21.15.118:3300/0 conn(0x7f3a20098520 msgr2=0x7f3a2009bf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:26.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 --2- 172.21.15.50:0/2170917364 >> v2:172.21.15.118:3300/0 conn(0x7f3a20098520 0x7f3a2009bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 >> v1:172.21.15.118:6789/0 conn(0x7f3a20098900 legacy=0x7f3a20098d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: _finish_hunting 0 2023-10-30T12:10:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:10:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:10:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2009b260 con 0x7f3a20137980 2023-10-30T12:10:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f3a180019f0 con 0x7f3a20137980 2023-10-30T12:10:26.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:26.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:10:26.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: dump: 2023-10-30T12:10:26.253 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:26.253 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:26.253 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:26.255 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:10:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:26.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 >> v1:172.21.15.50:6789/0 conn(0x7f3a20137980 legacy=0x7f3a20139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:10:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a08005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): picked mon.a con 0x7f3a08005b80 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a080069d0 0x7f3a08006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): picked mon.c con 0x7f3a080069d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a08007590 0x7f3a0800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): picked mon.b con 0x7f3a08007590 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2009af80 con 0x7f3a08005b80 2023-10-30T12:10:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0800e430 con 0x7f3a08007590 2023-10-30T12:10:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0800e5f0 con 0x7f3a080069d0 2023-10-30T12:10:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.233+0000 7f3a1dffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:10:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a08005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): get_auth_request con 0x7f3a08005b80 auth_method 0 2023-10-30T12:10:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:26.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1f7fe640 1 -- 172.21.15.50:0/2170917364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a08007590 msgr2=0x7f3a0800db90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a080069d0 0x7f3a08006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1f7fe640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a08007590 0x7f3a0800db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 10 monclient(hunting): get_auth_request con 0x7f3a080069d0 auth_method 0 2023-10-30T12:10:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:26.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:26.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:10:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient(hunting): handle_auth_done global_id 134292 payload 293 2023-10-30T12:10:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 1 -- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a080069d0 msgr2=0x7f3a08006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:10:26.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a080069d0 0x7f3a08006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 1 -- 172.21.15.50:0/2170917364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a08007590 msgr2=0x7f3a0800db90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a08007590 0x7f3a0800db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 1 monclient: found mon.a 2023-10-30T12:10:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:26.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1f7fe640 1 -- 172.21.15.50:0/2170917364 reap_dead start 2023-10-30T12:10:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 1 -- 172.21.15.50:0/2170917364 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a08005730 con 0x7f3a08005b80 2023-10-30T12:10:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient: _finish_auth 0 2023-10-30T12:10:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:26.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a25421640 5 monclient: authenticate success, global_id 134292 2023-10-30T12:10:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a1e7fc640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a080069d0 0x7f3a08006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.234+0000 7f3a25421640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1effd640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a08005f80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3a100098f0 tx=0x7f3a08007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe335217be0f653 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10003c00 con 0x7f3a08005b80 2023-10-30T12:10:26.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 10 monclient: dump: 2023-10-30T12:10:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:26.266 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:26.267 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:26.268 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1002e070 con 0x7f3a08005b80 2023-10-30T12:10:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 4 set_mon_vals no callback set 2023-10-30T12:10:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 1 -- 172.21.15.50:0/2170917364 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10005070 con 0x7f3a08005b80 2023-10-30T12:10:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:26.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a1dffb640 10 monclient: dump: 2023-10-30T12:10:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:26.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:26.271 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 10 monclient: shutdown 2023-10-30T12:10:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 1 -- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 msgr2=0x7f3a08005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a08005f80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3a100098f0 tx=0x7f3a08007a00 comp rx=0 tx=0).stop 2023-10-30T12:10:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 1 -- 172.21.15.50:0/2170917364 shutdown_connections 2023-10-30T12:10:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 1 --2- 172.21.15.50:0/2170917364 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a08005f80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 1 -- 172.21.15.50:0/2170917364 >> 172.21.15.50:0/2170917364 conn(0x7f3a200a05c0 msgr2=0x7f3a2009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 1 -- 172.21.15.50:0/2170917364 shutdown_connections 2023-10-30T12:10:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.235+0000 7f3a25421640 1 -- 172.21.15.50:0/2170917364 wait complete. 2023-10-30T12:10:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.236+0000 7f3a25421640 10 monclient: build_initial_monmap 2023-10-30T12:10:26.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.236+0000 7f3a25421640 10 monclient: monmap: 2023-10-30T12:10:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:26.275 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.236+0000 7f3a25421640 1 librados: starting msgr at 2023-10-30T12:10:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.236+0000 7f3a25421640 1 librados: starting objecter 2023-10-30T12:10:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.236+0000 7f3a25421640 1 Processor -- start 2023-10-30T12:10:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 -- start start 2023-10-30T12:10:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 librados: setting wanted keys 2023-10-30T12:10:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 librados: calling monclient init 2023-10-30T12:10:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient: init 2023-10-30T12:10:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:10:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a2012ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): picked mon.noname-a con 0x7f3a08005b80 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a2012cf80 0x7f3a2012d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2012cf80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a201cab50 0x7f3a201cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): picked mon.noname-b con 0x7f3a201cab50 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2012f3f0 con 0x7f3a08005b80 2023-10-30T12:10:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200a3060 con 0x7f3a201cab50 2023-10-30T12:10:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2012d8e0 con 0x7f3a2012cf80 2023-10-30T12:10:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a25421640 10 monclient(hunting): authenticate will time out at 1471.420492s 2023-10-30T12:10:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a2012ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a2012ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34168/0 (socket says 172.21.15.50:34168) 2023-10-30T12:10:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a2012cf80 0x7f3a2012d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 -- 172.21.15.50:0/4283746636 learned_addr learned my addr 172.21.15.50:0/4283746636 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): get_auth_request con 0x7f3a08005b80 auth_method 0 2023-10-30T12:10:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 10 monclient(hunting): get_auth_request con 0x7f3a2012cf80 auth_method 0 2023-10-30T12:10:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1f7fe640 1 -- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a201cab50 msgr2=0x7f3a201cafd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1f7fe640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a201cab50 0x7f3a201cafd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient(hunting): handle_auth_done global_id 134295 payload 824 2023-10-30T12:10:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 -- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a2012cf80 msgr2=0x7f3a2012d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a2012cf80 0x7f3a2012d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 -- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a201cab50 msgr2=0x7f3a201cafd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:10:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a201cab50 0x7f3a201cafd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient: _finish_hunting 0 2023-10-30T12:10:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 monclient: found mon.noname-a 2023-10-30T12:10:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1effd640 1 -- 172.21.15.50:0/4283746636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2012ffb0 con 0x7f3a08005b80 2023-10-30T12:10:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.237+0000 7f3a1e7fc640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a2012cf80 0x7f3a2012d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a1effd640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a2012ca40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000ea0 tx=0x7f3a10005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 client.?.objecter ms_handle_connect 0x7f3a08005b80 2023-10-30T12:10:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 1 -- 172.21.15.50:0/4283746636 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10003c00 con 0x7f3a08005b80 2023-10-30T12:10:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: dump: 2023-10-30T12:10:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:26.289 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 1 -- 172.21.15.50:0/4283746636 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a10003d60 con 0x7f3a08005b80 2023-10-30T12:10:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 5 monclient: authenticate success, global_id 134295 2023-10-30T12:10:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 10 monclient: _renew_subs 2023-10-30T12:10:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a20071d50 con 0x7f3a08005b80 2023-10-30T12:10:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:26.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 1 -- 172.21.15.50:0/4283746636 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10005290 con 0x7f3a08005b80 2023-10-30T12:10:26.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:26.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:26.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a077fe640 10 monclient: dump: 2023-10-30T12:10:26.291 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:26.292 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:26.292 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.292 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:26.292 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:26.292 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:26.292 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:26.293 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:26.293 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:26.293 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:26.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 10 client.134295.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:26.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 10 monclient: _renew_subs 2023-10-30T12:10:26.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a1d7fa640 4 set_mon_vals no callback set 2023-10-30T12:10:26.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:26.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2012f5d0 con 0x7f3a08005b80 2023-10-30T12:10:26.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.238+0000 7f3a25421640 1 librados: init done 2023-10-30T12:10:26.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a077fe640 1 -- 172.21.15.50:0/4283746636 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3a10030030 con 0x7f3a08005b80 2023-10-30T12:10:26.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a25421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:26.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a25421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:26.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a25421640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:26.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2004e160 con 0x7f3a08005b80 2023-10-30T12:10:26.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a077fe640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a0c0392a0 0x7f3a0c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:26.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a077fe640 1 -- 172.21.15.50:0/4283746636 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f3a1006c030 con 0x7f3a08005b80 2023-10-30T12:10:26.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a077fe640 10 client.134295.objecter ms_dispatch 0x7f3a20000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:26.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a077fe640 3 client.134295.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:26.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a077fe640 3 client.134295.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:26.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a077fe640 20 client.134295.objecter dump_active .. 0 homeless 2023-10-30T12:10:26.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a1e7fc640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a0c0392a0 0x7f3a0c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.239+0000 7f3a1e7fc640 10 monclient: get_auth_request con 0x7f3a0c0392a0 auth_method 0 2023-10-30T12:10:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.240+0000 7f3a1e7fc640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a0c0392a0 0x7f3a0c03b760 secure :-1 s=READY pgs=4097 cs=0 l=1 rev1=1 crypto rx=0x7f3a140094d0 tx=0x7f3a140090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.240+0000 7f3a077fe640 10 client.134295.objecter ms_handle_connect 0x7f3a0c0392a0 2023-10-30T12:10:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.244+0000 7f3a077fe640 1 -- 172.21.15.50:0/4283746636 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a100327e0 con 0x7f3a08005b80 2023-10-30T12:10:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.244+0000 7f3a077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:26.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.244+0000 7f3a077fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.493+0000 7f3a25421640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.493+0000 7f3a25421640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:26.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.493+0000 7f3a25421640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.493+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3a2009bbc0 con 0x7f3a08005b80 2023-10-30T12:10:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.494+0000 7f3a077fe640 1 -- 172.21.15.50:0/4283746636 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f3a10041020 con 0x7f3a08005b80 2023-10-30T12:10:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.494+0000 7f3a077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:26.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.494+0000 7f3a077fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:26.497 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 10 librados: watch_flush enter 2023-10-30T12:10:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 10 librados: watch_flush exit 2023-10-30T12:10:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 20 client.134295.objecter shutdown clearing up homeless session... 2023-10-30T12:10:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 10 client.134295.objecter successfully canceled tick 2023-10-30T12:10:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a0c0392a0 msgr2=0x7f3a0c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a0c0392a0 0x7f3a0c03b760 secure :-1 s=READY pgs=4097 cs=0 l=1 rev1=1 crypto rx=0x7f3a140094d0 tx=0x7f3a140090e0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 10 monclient: shutdown 2023-10-30T12:10:26.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 msgr2=0x7f3a2012ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a2012ca40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000ea0 tx=0x7f3a10005a60 comp rx=0 tx=0).stop 2023-10-30T12:10:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 shutdown_connections 2023-10-30T12:10:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a0c0392a0 0x7f3a0c03b760 unknown :-1 s=CLOSED pgs=4097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a201cab50 0x7f3a201cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3a2012cf80 0x7f3a2012d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 --2- 172.21.15.50:0/4283746636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a08005b80 0x7f3a2012ca40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.495+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 >> 172.21.15.50:0/4283746636 conn(0x7f3a200a05c0 msgr2=0x7f3a20134760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:26.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.496+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 shutdown_connections 2023-10-30T12:10:26.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.496+0000 7f3a25421640 1 -- 172.21.15.50:0/4283746636 wait complete. 2023-10-30T12:10:26.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:26.496+0000 7f3a25421640 1 librados: shutdown 2023-10-30T12:10:29.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:29.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.624+0000 7f4b7b19e640 10 monclient: get_monmap_and_config 2023-10-30T12:10:29.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.624+0000 7f4b7b19e640 10 monclient: build_initial_monmap 2023-10-30T12:10:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.624+0000 7f4b7b19e640 10 monclient: monmap: 2023-10-30T12:10:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:29.625164+0000 2023-10-30T12:10:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:29.625164+0000 2023-10-30T12:10:29.627 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:29.627 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:29.627 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:29.627 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:29.627 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:29.628 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:29.628 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:29.628 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.624+0000 7f4b7b19e640 1 Processor -- start 2023-10-30T12:10:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.624+0000 7f4b7b19e640 1 -- start start 2023-10-30T12:10:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.624+0000 7f4b7b19e640 10 monclient: init 2023-10-30T12:10:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T12:10:29.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4b740a4b60 0x7f4b741326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:29.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b740a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:29.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b74099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:29.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b74132be0 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:29.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b740a4f40 con 0x7f4b74132be0 2023-10-30T12:10:29.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b740a4150 con 0x7f4b74099a40 2023-10-30T12:10:29.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:29.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b740a3d70 con 0x7f4b740a4b60 2023-10-30T12:10:29.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:29.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7b19e640 10 monclient(hunting): authenticate will time out at 1474.808403s 2023-10-30T12:10:29.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4b740a4b60 0x7f4b741326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:29.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b73fff640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f4b74099a40 legacy=0x7f4b740a2350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:29.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4b740a4b60 0x7f4b741326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50240/0 (socket says 172.21.15.50:50240) 2023-10-30T12:10:29.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 1 -- 172.21.15.50:0/2493075219 learned_addr learned my addr 172.21.15.50:0/2493075219 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:29.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b7973b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4b74132be0 0x7f4b740a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35878/0 (socket says 172.21.15.50:35878) 2023-10-30T12:10:29.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): get_auth_request con 0x7f4b740a4b60 auth_method 0 2023-10-30T12:10:29.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:29.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:29.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:29.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:29.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:29.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:29.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_done global_id 134298 payload 274 2023-10-30T12:10:29.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 1 -- 172.21.15.50:0/2493075219 >> v1:172.21.15.118:6789/0 conn(0x7f4b74099a40 legacy=0x7f4b740a2350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:29.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 1 -- 172.21.15.50:0/2493075219 >> v1:172.21.15.50:6789/0 conn(0x7f4b74132be0 legacy=0x7f4b740a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:29.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient: _finish_hunting 0 2023-10-30T12:10:29.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 1 monclient: found mon.noname-a 2023-10-30T12:10:29.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:29.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:29.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.625+0000 7f4b78f3a640 1 -- 172.21.15.50:0/2493075219 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b740a3170 con 0x7f4b740a4b60 2023-10-30T12:10:29.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b78f3a640 1 --2- 172.21.15.50:0/2493075219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741326a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009e50 tx=0x7f4b6c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976a409423c50f49 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:29.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 1 -- 172.21.15.50:0/2493075219 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c02c9f0 con 0x7f4b740a4b60 2023-10-30T12:10:29.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:29.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:29.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: dump: 2023-10-30T12:10:29.642 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:29.642 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:29.643 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.643 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.643 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:29.643 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:29.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:29.644 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:29.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:29.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 1 -- 172.21.15.50:0/2493075219 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c02cb50 con 0x7f4b740a4b60 2023-10-30T12:10:29.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b7b19e640 5 monclient: authenticate success, global_id 134298 2023-10-30T12:10:29.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b7b19e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:29.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:29.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b7b19e640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:29.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 1 -- 172.21.15.50:0/2493075219 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0343f0 con 0x7f4b740a4b60 2023-10-30T12:10:29.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:29.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b7b19e640 4 set_mon_vals no callback set 2023-10-30T12:10:29.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:29.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.626+0000 7f4b737fe640 10 monclient: dump: 2023-10-30T12:10:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:29.648 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 10 monclient: shutdown 2023-10-30T12:10:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 1 -- 172.21.15.50:0/2493075219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 msgr2=0x7f4b741326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/2493075219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741326a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009e50 tx=0x7f4b6c02be50 comp rx=0 tx=0).stop 2023-10-30T12:10:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 1 -- 172.21.15.50:0/2493075219 shutdown_connections 2023-10-30T12:10:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/2493075219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741326a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 1 -- 172.21.15.50:0/2493075219 >> 172.21.15.50:0/2493075219 conn(0x7f4b740a8060 msgr2=0x7f4b740a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 1 -- 172.21.15.50:0/2493075219 shutdown_connections 2023-10-30T12:10:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 1 -- 172.21.15.50:0/2493075219 wait complete. 2023-10-30T12:10:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 10 monclient: build_initial_monmap 2023-10-30T12:10:29.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.627+0000 7f4b7b19e640 10 monclient: monmap: 2023-10-30T12:10:29.651 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:29.651 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:29.651 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:29.651 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:29.651 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:29.652 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:29.652 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:29.652 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:29.652 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:29.652 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:29.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 librados: starting msgr at 2023-10-30T12:10:29.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 librados: starting objecter 2023-10-30T12:10:29.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 Processor -- start 2023-10-30T12:10:29.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 -- start start 2023-10-30T12:10:29.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 librados: setting wanted keys 2023-10-30T12:10:29.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 librados: calling monclient init 2023-10-30T12:10:29.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient: init 2023-10-30T12:10:29.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:29.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:10:29.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:29.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b740a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:29.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b74132be0 0x7f4b741caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:29.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): picked mon.noname-c con 0x7f4b74132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:29.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b741bb8a0 0x7f4b741bbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:29.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b741bb8a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:29.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:29.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b741be660 con 0x7f4b740a4b60 2023-10-30T12:10:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740a0980 con 0x7f4b741bb8a0 2023-10-30T12:10:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740a18d0 con 0x7f4b74132be0 2023-10-30T12:10:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.628+0000 7f4b7b19e640 10 monclient(hunting): authenticate will time out at 1474.811982s 2023-10-30T12:10:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7973b640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b741bb8a0 msgr2=0x7f4b741bbd10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50248/0 (socket says 172.21.15.50:50248) 2023-10-30T12:10:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b74132be0 0x7f4b741caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 -- 172.21.15.50:0/1139183201 learned_addr learned my addr 172.21.15.50:0/1139183201 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:29.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): get_auth_request con 0x7f4b740a4b60 auth_method 0 2023-10-30T12:10:29.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:29.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:29.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:29.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7973b640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b741bb8a0 0x7f4b741bbd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:29.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 10 monclient(hunting): get_auth_request con 0x7f4b74132be0 auth_method 0 2023-10-30T12:10:29.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:29.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:29.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:29.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:29.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:29.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:29.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:29.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:29.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:29.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient(hunting): handle_auth_done global_id 134301 payload 824 2023-10-30T12:10:29.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 -- 172.21.15.50:0/1139183201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b74132be0 msgr2=0x7f4b741caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:29.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b74132be0 0x7f4b741caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 -- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b741bb8a0 msgr2=0x7f4b741bbd10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:10:29.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b741bb8a0 0x7f4b741bbd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient: _finish_hunting 0 2023-10-30T12:10:29.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 monclient: found mon.noname-a 2023-10-30T12:10:29.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 -- 172.21.15.50:0/1139183201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c009b30 con 0x7f4b740a4b60 2023-10-30T12:10:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b78f3a640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741ca9e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009e50 tx=0x7f4b6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 client.?.objecter ms_handle_connect 0x7f4b740a4b60 2023-10-30T12:10:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 1 -- 172.21.15.50:0/1139183201 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c03e450 con 0x7f4b740a4b60 2023-10-30T12:10:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: dump: 2023-10-30T12:10:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:29.666 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: _finish_auth 0 2023-10-30T12:10:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: _check_auth_tickets 2023-10-30T12:10:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 5 monclient: authenticate success, global_id 134301 2023-10-30T12:10:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 1 -- 172.21.15.50:0/1139183201 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c03e5b0 con 0x7f4b740a4b60 2023-10-30T12:10:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 1 -- 172.21.15.50:0/1139183201 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c03e960 con 0x7f4b740a4b60 2023-10-30T12:10:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 10 monclient: _renew_subs 2023-10-30T12:10:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7411e340 con 0x7f4b740a4b60 2023-10-30T12:10:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b72ffd640 4 set_mon_vals no callback set 2023-10-30T12:10:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:29.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b717fa640 10 monclient: dump: 2023-10-30T12:10:29.669 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:29.669 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:29.669 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.669 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:29.670 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:29.670 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:29.670 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:29.670 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:29.670 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:29.670 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:29.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 10 client.134301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:29.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 10 monclient: _renew_subs 2023-10-30T12:10:29.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:29.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b741bdeb0 con 0x7f4b740a4b60 2023-10-30T12:10:29.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.629+0000 7f4b7b19e640 1 librados: init done 2023-10-30T12:10:29.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.630+0000 7f4b717fa640 1 -- 172.21.15.50:0/1139183201 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4b6c03eac0 con 0x7f4b740a4b60 2023-10-30T12:10:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.630+0000 7f4b7b19e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.630+0000 7f4b7b19e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.630+0000 7f4b7b19e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.630+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b741bf020 con 0x7f4b740a4b60 2023-10-30T12:10:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.630+0000 7f4b717fa640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4b600392d0 0x7f4b6003b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b717fa640 1 -- 172.21.15.50:0/1139183201 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(325..325 src has 1..325) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f4b6c06d890 con 0x7f4b740a4b60 2023-10-30T12:10:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b717fa640 10 client.134301.objecter ms_dispatch 0x7f4b74000b90 osd_map(325..325 src has 1..325) v4 2023-10-30T12:10:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b717fa640 3 client.134301.objecter handle_osd_map got epochs [325,325] > 0 2023-10-30T12:10:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b717fa640 3 client.134301.objecter handle_osd_map decoding full epoch 325 2023-10-30T12:10:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b717fa640 20 client.134301.objecter dump_active .. 0 homeless 2023-10-30T12:10:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b73fff640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4b600392d0 0x7f4b6003b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b73fff640 10 monclient: get_auth_request con 0x7f4b600392d0 auth_method 0 2023-10-30T12:10:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b73fff640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4b600392d0 0x7f4b6003b790 secure :-1 s=READY pgs=4098 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006b40 tx=0x7f4b68006ad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.631+0000 7f4b717fa640 10 client.134301.objecter ms_handle_connect 0x7f4b600392d0 2023-10-30T12:10:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.635+0000 7f4b717fa640 1 -- 172.21.15.50:0/1139183201 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4b6c0473b0 con 0x7f4b740a4b60 2023-10-30T12:10:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.635+0000 7f4b717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.635+0000 7f4b717fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.876+0000 7f4b7b19e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.876+0000 7f4b7b19e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.877+0000 7f4b7b19e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.877+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4b7404e160 con 0x7f4b740a4b60 2023-10-30T12:10:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.877+0000 7f4b717fa640 1 -- 172.21.15.50:0/1139183201 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f4b6c043020 con 0x7f4b740a4b60 2023-10-30T12:10:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.877+0000 7f4b717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.877+0000 7f4b717fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:29.880 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.878+0000 7f4b7b19e640 10 librados: watch_flush enter 2023-10-30T12:10:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 10 librados: watch_flush exit 2023-10-30T12:10:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 20 client.134301.objecter shutdown clearing up homeless session... 2023-10-30T12:10:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 10 client.134301.objecter successfully canceled tick 2023-10-30T12:10:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4b600392d0 msgr2=0x7f4b6003b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4b600392d0 0x7f4b6003b790 secure :-1 s=READY pgs=4098 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006b40 tx=0x7f4b68006ad0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 10 monclient: shutdown 2023-10-30T12:10:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 msgr2=0x7f4b741ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741ca9e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009e50 tx=0x7f4b6c004a40 comp rx=0 tx=0).stop 2023-10-30T12:10:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 shutdown_connections 2023-10-30T12:10:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4b600392d0 0x7f4b6003b790 unknown :-1 s=CLOSED pgs=4098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4b741bb8a0 0x7f4b741bbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b74132be0 0x7f4b741caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 --2- 172.21.15.50:0/1139183201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b740a4b60 0x7f4b741ca9e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 >> 172.21.15.50:0/1139183201 conn(0x7f4b740a8060 msgr2=0x7f4b740a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 shutdown_connections 2023-10-30T12:10:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 -- 172.21.15.50:0/1139183201 wait complete. 2023-10-30T12:10:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:29.879+0000 7f4b7b19e640 1 librados: shutdown 2023-10-30T12:10:32.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.006+0000 7f75ccdd7640 10 monclient: get_monmap_and_config 2023-10-30T12:10:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.006+0000 7f75ccdd7640 10 monclient: build_initial_monmap 2023-10-30T12:10:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.006+0000 7f75ccdd7640 10 monclient: monmap: 2023-10-30T12:10:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:33.007461+0000 2023-10-30T12:10:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:33.007461+0000 2023-10-30T12:10:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:33.010 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 1 Processor -- start 2023-10-30T12:10:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 1 -- start start 2023-10-30T12:10:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient: init 2023-10-30T12:10:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:10:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 0x7f75c809f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient(hunting): picked mon.noname-c con 0x7f75c8131b20 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75c8132270 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient(hunting): picked mon.noname-a con 0x7f75c8132670 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c8131f00 con 0x7f75c8132270 2023-10-30T12:10:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c809d920 con 0x7f75c8132670 2023-10-30T12:10:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c809a0b0 con 0x7f75c8131b20 2023-10-30T12:10:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75ccdd7640 10 monclient(hunting): authenticate will time out at 1478.190795s 2023-10-30T12:10:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c659c640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 msgr2=0x7f75c809f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c659c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 0x7f75c809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c6d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c6d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50258/0 (socket says 172.21.15.50:50258) 2023-10-30T12:10:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 learned_addr learned my addr 172.21.15.50:0/2721439310 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request con 0x7f75c8132670 auth_method 0 2023-10-30T12:10:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.007+0000 7f75c6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.008+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:10:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.008+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.008+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.008+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 msgr2=0x7f75c809f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 0x7f75c809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:10:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request con 0x7f75c8132670 auth_method 0 2023-10-30T12:10:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:10:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.209+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:10:33.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 msgr2=0x7f75c809f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:33.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:33.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 0x7f75c809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:10:33.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request con 0x7f75c8132670 auth_method 0 2023-10-30T12:10:33.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:33.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:33.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:33.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:10:33.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:33.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:33.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:33.610+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:10:34.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 msgr2=0x7f75c809f5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:34.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:34.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 0x7f75c809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:10:34.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request con 0x7f75c8132670 auth_method 0 2023-10-30T12:10:34.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:34.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:34.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:34.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:10:34.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:34.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:34.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:34.411+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:10:36.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient(hunting): tick 2023-10-30T12:10:36.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:10:36.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:10:36.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:10:36.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 msgr2=0x7f75c809f5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:36.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.118:3300/0 conn(0x7f75c8131b20 0x7f75c809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:36.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 msgr2=0x7f75c8139f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:36.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 --2- 172.21.15.50:0/2721439310 >> v2:172.21.15.50:3300/0 conn(0x7f75c8132670 0x7f75c8139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:36.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.50:6789/0 conn(0x7f75c8132270 legacy=0x7f75c8071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:36.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient: _add_conns ranks=[2,1] 2023-10-30T12:10:36.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:36.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f75a8007ac0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:36.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75a800a620 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:36.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c809a390 con 0x7f75a800a620 2023-10-30T12:10:36.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:36.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c809d920 con 0x7f75a8007ac0 2023-10-30T12:10:36.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75b7fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:36.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c5d9b640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.118:6789/0 conn(0x7f75a800a620 legacy=0x7f75a800cae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:36.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:36.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 10 monclient(hunting): get_auth_request con 0x7f75a8007ac0 auth_method 0 2023-10-30T12:10:36.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:36.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:36.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:36.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:10:36.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:36.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:36.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.008+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:36.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c5d9b640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.118:6789/0 conn(0x7f75a800a620 legacy=0x7f75a800cae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:36.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:36.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 10 monclient(hunting): get_auth_request con 0x7f75a8007ac0 auth_method 0 2023-10-30T12:10:36.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:36.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:36.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:36.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:10:36.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:36.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:36.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.209+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:10:36.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c5d9b640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.118:6789/0 conn(0x7f75a800a620 legacy=0x7f75a800cae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 10 monclient(hunting): get_auth_request con 0x7f75a8007ac0 auth_method 0 2023-10-30T12:10:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:36.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:10:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:36.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:36.610+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:10:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c5d9b640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.118:6789/0 conn(0x7f75a800a620 legacy=0x7f75a800cae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 10 monclient(hunting): get_auth_request con 0x7f75a8007ac0 auth_method 0 2023-10-30T12:10:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:10:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:10:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:37.411+0000 7f75c659c640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:10:39.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): tick 2023-10-30T12:10:39.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:10:39.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:10:39.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:10:39.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 msgr2=0x7f75a8009ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:39.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a8007ac0 0x7f75a8009ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.118:6789/0 conn(0x7f75a800a620 legacy=0x7f75a800cae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:39.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 reap_dead start 2023-10-30T12:10:39.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-10-30T12:10:39.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75a800e360 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:39.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800eef0 0x7f75a8009ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f75a800eef0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:39.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75a800cc50 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:39.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a800f460 con 0x7f75a800e360 2023-10-30T12:10:39.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c809a390 con 0x7f75a800cc50 2023-10-30T12:10:39.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 1 -- 172.21.15.50:0/2721439310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c809d920 con 0x7f75a800eef0 2023-10-30T12:10:39.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75b7fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:39.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c6d9d640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.118:6789/0 conn(0x7f75a800cc50 legacy=0x7f75a8011860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:39.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800eef0 0x7f75a8009ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): get_auth_request con 0x7f75a800eef0 auth_method 0 2023-10-30T12:10:39.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient(hunting): handle_auth_done global_id 114506 payload 274 2023-10-30T12:10:39.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.118:6789/0 conn(0x7f75a800cc50 legacy=0x7f75a8011860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:39.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c559a640 1 -- 172.21.15.50:0/2721439310 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819561592 0 0) 0x7f75a800f460 con 0x7f75a800e360 2023-10-30T12:10:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 1 -- 172.21.15.50:0/2721439310 >> v1:172.21.15.50:6789/0 conn(0x7f75a800e360 legacy=0x7f75a800e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 1 monclient: found mon.noname-b 2023-10-30T12:10:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:10:39.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c5d9b640 1 -- 172.21.15.50:0/2721439310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a8015100 con 0x7f75a800eef0 2023-10-30T12:10:39.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.009+0000 7f75c559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:10:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.010+0000 7f75c5d9b640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800eef0 0x7f75a8009ef0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0016b0 tx=0x7f75bc02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0d2ab0c2aa6dcf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.010+0000 7f75c559a640 1 -- 172.21.15.50:0/2721439310 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc007480 con 0x7f75a800eef0 2023-10-30T12:10:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:10:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: dump: 2023-10-30T12:10:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.026 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: _finish_auth 0 2023-10-30T12:10:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: _check_auth_tickets 2023-10-30T12:10:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 1 -- 172.21.15.50:0/2721439310 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc007600 con 0x7f75a800eef0 2023-10-30T12:10:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 1 -- 172.21.15.50:0/2721439310 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc005a00 con 0x7f75a800eef0 2023-10-30T12:10:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 5 monclient: authenticate success, global_id 114506 2023-10-30T12:10:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 4 set_mon_vals no callback set 2023-10-30T12:10:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75c559a640 10 monclient: dump: 2023-10-30T12:10:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.030 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 10 monclient: shutdown 2023-10-30T12:10:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 1 -- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800eef0 msgr2=0x7f75a8009ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800eef0 0x7f75a8009ef0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0016b0 tx=0x7f75bc02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 1 -- 172.21.15.50:0/2721439310 shutdown_connections 2023-10-30T12:10:39.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/2721439310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800eef0 0x7f75a8009ef0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.011+0000 7f75ccdd7640 1 -- 172.21.15.50:0/2721439310 >> 172.21.15.50:0/2721439310 conn(0x7f75c80a05c0 msgr2=0x7f75c809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.012+0000 7f75ccdd7640 1 -- 172.21.15.50:0/2721439310 shutdown_connections 2023-10-30T12:10:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.012+0000 7f75ccdd7640 1 -- 172.21.15.50:0/2721439310 wait complete. 2023-10-30T12:10:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 10 monclient: build_initial_monmap 2023-10-30T12:10:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 10 monclient: monmap: 2023-10-30T12:10:39.032 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.034 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:39.034 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:39.034 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:39.034 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 1 librados: starting msgr at 2023-10-30T12:10:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 1 librados: starting objecter 2023-10-30T12:10:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 1 Processor -- start 2023-10-30T12:10:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 1 -- start start 2023-10-30T12:10:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 1 librados: setting wanted keys 2023-10-30T12:10:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 1 librados: calling monclient init 2023-10-30T12:10:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.013+0000 7f75ccdd7640 10 monclient: init 2023-10-30T12:10:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:10:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a800cc50 0x7f75c81c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): picked mon.noname-a con 0x7f75a800cc50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800e360 0x7f75c81c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): picked mon.noname-c con 0x7f75a800e360 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a800eef0 0x7f75c8128630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): picked mon.noname-b con 0x7f75a800eef0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8129740 con 0x7f75a800cc50 2023-10-30T12:10:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c812a850 con 0x7f75a800eef0 2023-10-30T12:10:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c812ab30 con 0x7f75a800e360 2023-10-30T12:10:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75ccdd7640 10 monclient(hunting): authenticate will time out at 1484.196336s 2023-10-30T12:10:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a800cc50 0x7f75c81c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800e360 0x7f75c81c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c6d9d640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a800eef0 msgr2=0x7f75c8128630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a800cc50 0x7f75c81c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60756/0 (socket says 172.21.15.50:60756) 2023-10-30T12:10:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 -- 172.21.15.50:0/714465835 learned_addr learned my addr 172.21.15.50:0/714465835 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): get_auth_request con 0x7f75a800cc50 auth_method 0 2023-10-30T12:10:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c6d9d640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a800eef0 0x7f75c8128630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 10 monclient(hunting): get_auth_request con 0x7f75a800e360 auth_method 0 2023-10-30T12:10:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient(hunting): handle_auth_done global_id 134307 payload 824 2023-10-30T12:10:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 -- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800e360 msgr2=0x7f75c81c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800e360 0x7f75c81c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 -- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a800eef0 msgr2=0x7f75c8128630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:10:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a800eef0 0x7f75c8128630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient: _finish_hunting 0 2023-10-30T12:10:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 monclient: found mon.noname-a 2023-10-30T12:10:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c659c640 1 -- 172.21.15.50:0/714465835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c8131f00 con 0x7f75a800cc50 2023-10-30T12:10:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.014+0000 7f75c5d9b640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800e360 0x7f75c81c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75c659c640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a800cc50 0x7f75c81c5ba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75b8003890 tx=0x7f75b8005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 client.?.objecter ms_handle_connect 0x7f75a800cc50 2023-10-30T12:10:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 1 -- 172.21.15.50:0/714465835 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b80167f0 con 0x7f75a800cc50 2023-10-30T12:10:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: dump: 2023-10-30T12:10:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.048 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.048 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.048 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:10:39.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:10:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 5 monclient: authenticate success, global_id 134307 2023-10-30T12:10:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 10 monclient: _renew_subs 2023-10-30T12:10:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 1 -- 172.21.15.50:0/714465835 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8011040 con 0x7f75a800cc50 2023-10-30T12:10:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c812ae10 con 0x7f75a800cc50 2023-10-30T12:10:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 1 -- 172.21.15.50:0/714465835 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8016bb0 con 0x7f75a800cc50 2023-10-30T12:10:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75b6ffd640 10 monclient: dump: 2023-10-30T12:10:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.052 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75c4d99640 4 set_mon_vals no callback set 2023-10-30T12:10:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 10 client.134307.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 10 monclient: _renew_subs 2023-10-30T12:10:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81c6620 con 0x7f75a800cc50 2023-10-30T12:10:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.015+0000 7f75ccdd7640 1 librados: init done 2023-10-30T12:10:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.016+0000 7f75b6ffd640 1 -- 172.21.15.50:0/714465835 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f75b8016d10 con 0x7f75a800cc50 2023-10-30T12:10:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.016+0000 7f75ccdd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.016+0000 7f75ccdd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.016+0000 7f75ccdd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.016+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c804e160 con 0x7f75a800cc50 2023-10-30T12:10:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75b6ffd640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75a803f230 0x7f75a80416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75b6ffd640 1 -- 172.21.15.50:0/714465835 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(327..327 src has 1..327) v4 ==== 5914+0+0 (secure 0 0 0) 0x7f75b804be60 con 0x7f75a800cc50 2023-10-30T12:10:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75b6ffd640 10 client.134307.objecter ms_dispatch 0x7f75c8000b90 osd_map(327..327 src has 1..327) v4 2023-10-30T12:10:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75b6ffd640 3 client.134307.objecter handle_osd_map got epochs [327,327] > 0 2023-10-30T12:10:39.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75b6ffd640 3 client.134307.objecter handle_osd_map decoding full epoch 327 2023-10-30T12:10:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75b6ffd640 20 client.134307.objecter dump_active .. 0 homeless 2023-10-30T12:10:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75c5d9b640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75a803f230 0x7f75a80416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.017+0000 7f75c5d9b640 10 monclient: get_auth_request con 0x7f75a803f230 auth_method 0 2023-10-30T12:10:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.018+0000 7f75c5d9b640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75a803f230 0x7f75a80416f0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 crypto rx=0x7f75bc034000 tx=0x7f75bc02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.018+0000 7f75b6ffd640 10 client.134307.objecter ms_handle_connect 0x7f75a803f230 2023-10-30T12:10:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.022+0000 7f75b6ffd640 1 -- 172.21.15.50:0/714465835 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75b8005b90 con 0x7f75a800cc50 2023-10-30T12:10:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.022+0000 7f75b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:39.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.022+0000 7f75b6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:39.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.276+0000 7f75ccdd7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:10:39.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.276+0000 7f75ccdd7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:39.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.276+0000 7f75ccdd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.276+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f75c80745c0 con 0x7f75a800cc50 2023-10-30T12:10:39.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.276+0000 7f75b6ffd640 1 -- 172.21.15.50:0/714465835 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f75b801f360 con 0x7f75a800cc50 2023-10-30T12:10:39.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.276+0000 7f75b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:10:39.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.276+0000 7f75b6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:39.278 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:10:39.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 10 librados: watch_flush enter 2023-10-30T12:10:39.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 10 librados: watch_flush exit 2023-10-30T12:10:39.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 20 client.134307.objecter shutdown clearing up homeless session... 2023-10-30T12:10:39.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 10 client.134307.objecter successfully canceled tick 2023-10-30T12:10:39.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75a803f230 msgr2=0x7f75a80416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75a803f230 0x7f75a80416f0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 crypto rx=0x7f75bc034000 tx=0x7f75bc02f000 comp rx=0 tx=0).stop 2023-10-30T12:10:39.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 10 monclient: shutdown 2023-10-30T12:10:39.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:39.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a800cc50 msgr2=0x7f75c81c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a800cc50 0x7f75c81c5ba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75b8003890 tx=0x7f75b8005480 comp rx=0 tx=0).stop 2023-10-30T12:10:39.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 shutdown_connections 2023-10-30T12:10:39.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75a803f230 0x7f75a80416f0 unknown :-1 s=CLOSED pgs=4099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75a800eef0 0x7f75c8128630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75a800e360 0x7f75c81c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 --2- 172.21.15.50:0/714465835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a800cc50 0x7f75c81c5ba0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 >> 172.21.15.50:0/714465835 conn(0x7f75c80a05c0 msgr2=0x7f75c8130660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:39.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 shutdown_connections 2023-10-30T12:10:39.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 -- 172.21.15.50:0/714465835 wait complete. 2023-10-30T12:10:39.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.278+0000 7f75ccdd7640 1 librados: shutdown 2023-10-30T12:10:39.289 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:10:39.289 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:10:39.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:10:39.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.395+0000 7f5494856640 10 monclient: get_monmap_and_config 2023-10-30T12:10:39.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.395+0000 7f5494856640 10 monclient: build_initial_monmap 2023-10-30T12:10:39.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.395+0000 7f5494856640 10 monclient: monmap: 2023-10-30T12:10:39.396 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:39.397 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.397 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:39.395815+0000 2023-10-30T12:10:39.397 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:39.395815+0000 2023-10-30T12:10:39.397 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:39.397 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.398 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:39.398 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:39.398 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:39.398 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:39.398 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:39.398 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 1 Processor -- start 2023-10-30T12:10:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 1 -- start start 2023-10-30T12:10:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient: init 2023-10-30T12:10:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-10-30T12:10:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f549009bb70 0x7f549009bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient(hunting): picked mon.noname-a con 0x7f549009bb70 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f549009d3f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f549009c490 0x7f5490096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient(hunting): picked mon.noname-c con 0x7f549009c490 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5490098170 con 0x7f549009d3f0 2023-10-30T12:10:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549009cf60 con 0x7f549009bb70 2023-10-30T12:10:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5490097b20 con 0x7f549009c490 2023-10-30T12:10:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5494856640 10 monclient(hunting): authenticate will time out at 1484.578858s 2023-10-30T12:10:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f548ed9d640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f549009c490 msgr2=0x7f5490096790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f5485d9b640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f549009d3f0 legacy=0x7f5490096080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f548ed9d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f549009c490 0x7f5490096790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.396+0000 7f548e59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f549009bb70 0x7f549009bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f549009bb70 0x7f549009bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60768/0 (socket says 172.21.15.50:60768) 2023-10-30T12:10:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 -- 172.21.15.50:0/328124872 learned_addr learned my addr 172.21.15.50:0/328124872 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): get_auth_request con 0x7f549009bb70 auth_method 0 2023-10-30T12:10:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient(hunting): handle_auth_done global_id 134310 payload 274 2023-10-30T12:10:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 -- 172.21.15.50:0/328124872 >> v2:172.21.15.118:3300/0 conn(0x7f549009c490 msgr2=0x7f5490096790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 --2- 172.21.15.50:0/328124872 >> v2:172.21.15.118:3300/0 conn(0x7f549009c490 0x7f5490096790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 -- 172.21.15.50:0/328124872 >> v1:172.21.15.118:6789/0 conn(0x7f549009d3f0 legacy=0x7f5490096080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:10:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 monclient: found mon.noname-a 2023-10-30T12:10:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 -- 172.21.15.50:0/328124872 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490097e00 con 0x7f549009bb70 2023-10-30T12:10:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.397+0000 7f548e59c640 1 --2- 172.21.15.50:0/328124872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009bb70 0x7f549009bf50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f549009fe70 tx=0x7f548002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69704645ee6bd6a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 1 -- 172.21.15.50:0/328124872 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480004230 con 0x7f549009bb70 2023-10-30T12:10:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:39.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: dump: 2023-10-30T12:10:39.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.420 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: _finish_auth 0 2023-10-30T12:10:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:10:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 1 -- 172.21.15.50:0/328124872 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5480004390 con 0x7f549009bb70 2023-10-30T12:10:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 5 monclient: authenticate success, global_id 134310 2023-10-30T12:10:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 4 set_mon_vals no callback set 2023-10-30T12:10:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 1 -- 172.21.15.50:0/328124872 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480003810 con 0x7f549009bb70 2023-10-30T12:10:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f548dd9b640 10 monclient: dump: 2023-10-30T12:10:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.426 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 10 monclient: shutdown 2023-10-30T12:10:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 -- 172.21.15.50:0/328124872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009bb70 msgr2=0x7f549009bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 --2- 172.21.15.50:0/328124872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009bb70 0x7f549009bf50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f549009fe70 tx=0x7f548002b7f0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 -- 172.21.15.50:0/328124872 shutdown_connections 2023-10-30T12:10:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 --2- 172.21.15.50:0/328124872 >> v2:172.21.15.118:3300/0 conn(0x7f549009c490 0x7f5490096790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 --2- 172.21.15.50:0/328124872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009bb70 0x7f549009bf50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 -- 172.21.15.50:0/328124872 >> 172.21.15.50:0/328124872 conn(0x7f54900a3980 msgr2=0x7f54900a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 -- 172.21.15.50:0/328124872 shutdown_connections 2023-10-30T12:10:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.398+0000 7f5494856640 1 -- 172.21.15.50:0/328124872 wait complete. 2023-10-30T12:10:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 10 monclient: build_initial_monmap 2023-10-30T12:10:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 10 monclient: monmap: 2023-10-30T12:10:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:39.431 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 1 librados: starting msgr at 2023-10-30T12:10:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 1 librados: starting objecter 2023-10-30T12:10:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 1 Processor -- start 2023-10-30T12:10:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 1 -- start start 2023-10-30T12:10:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 1 librados: setting wanted keys 2023-10-30T12:10:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 1 librados: calling monclient init 2023-10-30T12:10:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.399+0000 7f5494856640 10 monclient: init 2023-10-30T12:10:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:10:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f549009bb70 0x7f5490135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): picked mon.noname-c con 0x7f549009bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009c490 0x7f5490136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): picked mon.noname-a con 0x7f549009c490 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f549009d3f0 0x7f54901366b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): picked mon.noname-b con 0x7f549009d3f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5490137620 con 0x7f549009c490 2023-10-30T12:10:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549013abe0 con 0x7f549009d3f0 2023-10-30T12:10:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5490138f50 con 0x7f549009bb70 2023-10-30T12:10:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5494856640 10 monclient(hunting): authenticate will time out at 1484.582374s 2023-10-30T12:10:39.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009c490 0x7f5490136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f549009bb70 0x7f5490135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009c490 0x7f5490136170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60778/0 (socket says 172.21.15.50:60778) 2023-10-30T12:10:39.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 10 monclient(hunting): get_auth_request con 0x7f549009bb70 auth_method 0 2023-10-30T12:10:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 -- 172.21.15.50:0/2253332100 learned_addr learned my addr 172.21.15.50:0/2253332100 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): get_auth_request con 0x7f549009c490 auth_method 0 2023-10-30T12:10:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548ed9d640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f549009d3f0 msgr2=0x7f54901366b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:39.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548ed9d640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f549009d3f0 0x7f54901366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:39.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient(hunting): handle_auth_done global_id 134313 payload 824 2023-10-30T12:10:39.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f549009bb70 msgr2=0x7f5490135c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f549009bb70 0x7f5490135c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f549009d3f0 msgr2=0x7f54901366b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:10:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f549009d3f0 0x7f54901366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 monclient: found mon.noname-a 2023-10-30T12:10:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 -- 172.21.15.50:0/2253332100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5480009460 con 0x7f549009c490 2023-10-30T12:10:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f548e59c640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f549009bb70 0x7f5490135c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:10:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f5485d9b640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009c490 0x7f5490136170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f547800b550 tx=0x7f547800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.400+0000 7f54877fe640 10 client.?.objecter ms_handle_connect 0x7f549009c490 2023-10-30T12:10:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478003cf0 con 0x7f549009c490 2023-10-30T12:10:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: dump: 2023-10-30T12:10:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.452 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: _finish_auth 0 2023-10-30T12:10:39.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: _check_auth_tickets 2023-10-30T12:10:39.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 5 monclient: authenticate success, global_id 134313 2023-10-30T12:10:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5478003e70 con 0x7f549009c490 2023-10-30T12:10:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:39.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 10 monclient: _renew_subs 2023-10-30T12:10:39.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478005290 con 0x7f549009c490 2023-10-30T12:10:39.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54900976e0 con 0x7f549009c490 2023-10-30T12:10:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f548d59a640 4 set_mon_vals no callback set 2023-10-30T12:10:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:39.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f54877fe640 10 monclient: dump: 2023-10-30T12:10:39.457 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.458 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.458 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.458 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.459 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.459 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.459 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.459 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.460 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.460 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 10 client.134313.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:39.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 10 monclient: _renew_subs 2023-10-30T12:10:39.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5490097180 con 0x7f549009c490 2023-10-30T12:10:39.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 1 librados: init done 2023-10-30T12:10:39.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.401+0000 7f5494856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:39.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f5494856640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549002c5b0 0x7f5490060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f5494856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5490138a70 con 0x7f549002c5b0 2023-10-30T12:10:39.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f54780053f0 con 0x7f549009c490 2023-10-30T12:10:39.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549002c5b0 0x7f5490060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient: get_auth_request con 0x7f549002c5b0 auth_method 0 2023-10-30T12:10:39.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54700392a0 0x7f547003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 10 monclient(hunting): handle_auth_done global_id 134316 payload 824 2023-10-30T12:10:39.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 1 -- 172.21.15.50:0/2253332100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5490097560 con 0x7f549002c5b0 2023-10-30T12:10:39.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(327..327 src has 1..327) v4 ==== 5914+0+0 (secure 0 0 0) 0x7f547804a8b0 con 0x7f549009c490 2023-10-30T12:10:39.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 10 client.134313.objecter ms_dispatch 0x7f5490000b90 osd_map(327..327 src has 1..327) v4 2023-10-30T12:10:39.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 3 client.134313.objecter handle_osd_map got epochs [327,327] > 0 2023-10-30T12:10:39.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 3 client.134313.objecter handle_osd_map decoding full epoch 327 2023-10-30T12:10:39.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 20 client.134313.objecter dump_active .. 0 homeless 2023-10-30T12:10:39.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 10 client.134313.objecter ms_handle_connect 0x7f549002c5b0 2023-10-30T12:10:39.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 10 client.134313.objecter resend_mon_ops 2023-10-30T12:10:39.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548e59c640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549002c5b0 0x7f5490060fc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f549009d840 tx=0x7f54800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480003da0 con 0x7f549002c5b0 2023-10-30T12:10:39.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548ed9d640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54700392a0 0x7f547003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:39.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f54877fe640 10 monclient: dump: 2023-10-30T12:10:39.467 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.467 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.468 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.468 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.468 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.468 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.468 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.469 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.469 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.469 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.402+0000 7f548ed9d640 10 monclient: get_auth_request con 0x7f54700392a0 auth_method 0 2023-10-30T12:10:39.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5480042070 con 0x7f549002c5b0 2023-10-30T12:10:39.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:39.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:39.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549002c5b0 msgr2=0x7f5490060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549002c5b0 0x7f5490060fc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f549009d840 tx=0x7f54800046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549002c5b0 msgr2=0x7f5490060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:39.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549002c5b0 0x7f5490060fc0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f549009d840 tx=0x7f54800046c0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f54877fe640 10 client.134313.objecter ms_handle_connect 0x7f54700392a0 2023-10-30T12:10:39.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.403+0000 7f548ed9d640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54700392a0 0x7f547003b760 secure :-1 s=READY pgs=4100 cs=0 l=1 rev1=1 crypto rx=0x7f5488003cb0 tx=0x7f54880091f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f5494856640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f5494856640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5490061500 0x7f54901d2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f5494856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5490073ff0 con 0x7f5490061500 2023-10-30T12:10:39.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5490061500 0x7f54901d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient: get_auth_request con 0x7f5490061500 auth_method 0 2023-10-30T12:10:39.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 10 monclient(hunting): handle_auth_done global_id 134319 payload 824 2023-10-30T12:10:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 1 -- 172.21.15.50:0/2253332100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f549009c920 con 0x7f5490061500 2023-10-30T12:10:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f54877fe640 10 client.134313.objecter ms_handle_connect 0x7f5490061500 2023-10-30T12:10:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f54877fe640 10 client.134313.objecter resend_mon_ops 2023-10-30T12:10:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.409+0000 7f548e59c640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5490061500 0x7f54901d2960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5480000c00 tx=0x7f5480031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480006900 con 0x7f5490061500 2023-10-30T12:10:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 10 monclient: dump: 2023-10-30T12:10:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.479 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.479 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.479 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.479 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.479 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.480 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.480 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.480 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.480 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7f5480031c90 con 0x7f5490061500 2023-10-30T12:10:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5490061500 msgr2=0x7f54901d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5490061500 0x7f54901d2960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5480000c00 tx=0x7f5480031000 comp rx=0 tx=0).stop 2023-10-30T12:10:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5490061500 msgr2=0x7f54901d2960 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f54877fe640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5490061500 0x7f54901d2960 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5480000c00 tx=0x7f5480031000 comp rx=0 tx=0).stop 2023-10-30T12:10:39.482 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:10:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 98, 2023-10-30T12:10:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:10:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:39.494 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:39.494 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:39.494 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:39.494 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:39.495 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:39.495 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:39.495 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:39.496 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:39.496 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:39.496 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:39.496 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:39.497 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:39.497 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:39.497 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:39.498 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:39.498 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:39.499 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:39.499 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:39.499 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.499 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:39.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.500 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:39.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.500 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:39.500 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:39.501 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:39.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:39.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:39.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:39.502 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.502 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:39.503 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:39.503 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.504 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:39.504 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:39.504 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:39.505 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:39.505 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:39.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.506 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:39.506 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:39.506 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:39.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:39.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.507 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:39.507 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:39.507 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:39.508 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:39.508 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.509 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:39.509 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:39.509 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:39.509 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:39.510 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:39.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.510 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:39.511 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:39.511 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:39.511 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:39.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:39.512 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:39.512 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:39.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:39.513 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.514 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.515 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:39.515 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:39.516 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:39.516 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:39.516 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:39.516 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.517 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.517 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:39.518 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:39.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.518 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:39.519 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:39.519 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:39.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.519 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:10:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:39.520 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:10:39.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.521 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:39.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.522 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:39.522 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:39.522 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:10:39.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.522 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.523 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:10:39.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.523 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:39.523 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:39.523 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:39.524 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.524 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.524 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:39.524 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:39.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 10 librados: watch_flush enter 2023-10-30T12:10:39.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 10 librados: watch_flush exit 2023-10-30T12:10:39.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 20 client.134313.objecter shutdown clearing up homeless session... 2023-10-30T12:10:39.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 10 client.134313.objecter successfully canceled tick 2023-10-30T12:10:39.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54700392a0 msgr2=0x7f547003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54700392a0 0x7f547003b760 secure :-1 s=READY pgs=4100 cs=0 l=1 rev1=1 crypto rx=0x7f5488003cb0 tx=0x7f54880091f0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 10 monclient: shutdown 2023-10-30T12:10:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009c490 msgr2=0x7f5490136170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 --2- 172.21.15.50:0/2253332100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f549009c490 0x7f5490136170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f547800b550 tx=0x7f547800ba10 comp rx=0 tx=0).stop 2023-10-30T12:10:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f548ed9d640 1 -- 172.21.15.50:0/2253332100 reap_dead start 2023-10-30T12:10:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f548ed9d640 1 -- 172.21.15.50:0/2253332100 reap_dead start 2023-10-30T12:10:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 shutdown_connections 2023-10-30T12:10:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 >> 172.21.15.50:0/2253332100 conn(0x7f54900a3980 msgr2=0x7f549013af70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 shutdown_connections 2023-10-30T12:10:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 -- 172.21.15.50:0/2253332100 wait complete. 2023-10-30T12:10:39.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.410+0000 7f5494856640 1 librados: shutdown 2023-10-30T12:10:39.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:10:39.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:10:39.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:10:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.640+0000 7f5f9f290640 10 monclient: get_monmap_and_config 2023-10-30T12:10:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.640+0000 7f5f9f290640 10 monclient: build_initial_monmap 2023-10-30T12:10:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.640+0000 7f5f9f290640 10 monclient: monmap: 2023-10-30T12:10:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:39.640903+0000 2023-10-30T12:10:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:39.640903+0000 2023-10-30T12:10:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:39.644 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.641+0000 7f5f9f290640 1 Processor -- start 2023-10-30T12:10:39.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.641+0000 7f5f9f290640 1 -- start start 2023-10-30T12:10:39.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.641+0000 7f5f9f290640 10 monclient: init 2023-10-30T12:10:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-10-30T12:10:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f98133400 addr v1:172.21.15.118:6789/0 2023-10-30T12:10:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f9809fcc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f98133ef0 0x7f5f9813c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient(hunting): picked mon.noname-a con 0x7f5f98133ef0 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9806e6c0 con 0x7f5f9809fcc0 2023-10-30T12:10:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9806f360 con 0x7f5f98133400 2023-10-30T12:10:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f980717f0 con 0x7f5f98133ef0 2023-10-30T12:10:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9f290640 10 monclient(hunting): authenticate will time out at 1484.824193s 2023-10-30T12:10:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d02c640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f5f98133400 legacy=0x7f5f981337e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:10:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9c82b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5f9809fcc0 0x7f5f98138930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34870/0 (socket says 172.21.15.50:34870) 2023-10-30T12:10:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9c82b640 1 -- 172.21.15.50:0/960867555 learned_addr learned my addr 172.21.15.50:0/960867555 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:10:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 1 --2- 172.21.15.50:0/960867555 >> v2:172.21.15.50:3300/0 conn(0x7f5f98133ef0 0x7f5f9813c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): get_auth_request con 0x7f5f98133ef0 auth_method 0 2023-10-30T12:10:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient(hunting): handle_auth_done global_id 134322 payload 274 2023-10-30T12:10:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 1 -- 172.21.15.50:0/960867555 >> v1:172.21.15.118:6789/0 conn(0x7f5f98133400 legacy=0x7f5f981337e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 1 -- 172.21.15.50:0/960867555 >> v1:172.21.15.50:6789/0 conn(0x7f5f9809fcc0 legacy=0x7f5f98138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient: _finish_hunting 0 2023-10-30T12:10:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 1 monclient: found mon.noname-a 2023-10-30T12:10:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.642+0000 7f5f9d82d640 1 -- 172.21.15.50:0/960867555 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9806fdb0 con 0x7f5f98133ef0 2023-10-30T12:10:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9d82d640 1 --2- 172.21.15.50:0/960867555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133ef0 0x7f5f9813c060 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009d90 tx=0x7f5f8802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219f07cf98f81644 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 1 -- 172.21.15.50:0/960867555 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8802cb40 con 0x7f5f98133ef0 2023-10-30T12:10:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: dump: 2023-10-30T12:10:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.661 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: _finish_auth 0 2023-10-30T12:10:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: _check_auth_tickets 2023-10-30T12:10:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 1 -- 172.21.15.50:0/960867555 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8802cca0 con 0x7f5f98133ef0 2023-10-30T12:10:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 5 monclient: authenticate success, global_id 134322 2023-10-30T12:10:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 1 -- 172.21.15.50:0/960867555 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88034660 con 0x7f5f98133ef0 2023-10-30T12:10:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 4 set_mon_vals no callback set 2023-10-30T12:10:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f97fff640 10 monclient: dump: 2023-10-30T12:10:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.665 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 10 monclient: shutdown 2023-10-30T12:10:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 1 -- 172.21.15.50:0/960867555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133ef0 msgr2=0x7f5f9813c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.643+0000 7f5f9f290640 1 --2- 172.21.15.50:0/960867555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133ef0 0x7f5f9813c060 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009d90 tx=0x7f5f8802bd90 comp rx=0 tx=0).stop 2023-10-30T12:10:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 -- 172.21.15.50:0/960867555 shutdown_connections 2023-10-30T12:10:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 --2- 172.21.15.50:0/960867555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133ef0 0x7f5f9813c060 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 -- 172.21.15.50:0/960867555 >> 172.21.15.50:0/960867555 conn(0x7f5f980a6200 msgr2=0x7f5f980a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 -- 172.21.15.50:0/960867555 shutdown_connections 2023-10-30T12:10:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 -- 172.21.15.50:0/960867555 wait complete. 2023-10-30T12:10:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 10 monclient: build_initial_monmap 2023-10-30T12:10:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 10 monclient: monmap: 2023-10-30T12:10:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:39.669 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 librados: starting msgr at 2023-10-30T12:10:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 librados: starting objecter 2023-10-30T12:10:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.644+0000 7f5f9f290640 1 Processor -- start 2023-10-30T12:10:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 -- start start 2023-10-30T12:10:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 librados: setting wanted keys 2023-10-30T12:10:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 librados: calling monclient init 2023-10-30T12:10:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient: init 2023-10-30T12:10:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:10:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9809fcc0 0x7f5f9812f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9809fcc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133400 0x7f5f9809eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): picked mon.noname-a con 0x7f5f98133400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:39.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98133ef0 0x7f5f98069e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): picked mon.noname-c con 0x7f5f98133ef0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9806f030 con 0x7f5f98133400 2023-10-30T12:10:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f981303a0 con 0x7f5f9809fcc0 2023-10-30T12:10:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9809c590 con 0x7f5f98133ef0 2023-10-30T12:10:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9f290640 10 monclient(hunting): authenticate will time out at 1484.827449s 2023-10-30T12:10:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133400 0x7f5f9809eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d02c640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9809fcc0 msgr2=0x7f5f9812f940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:10:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d02c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9809fcc0 0x7f5f9812f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133400 0x7f5f9809eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60810/0 (socket says 172.21.15.50:60810) 2023-10-30T12:10:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 -- 172.21.15.50:0/4137074833 learned_addr learned my addr 172.21.15.50:0/4137074833 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): get_auth_request con 0x7f5f98133400 auth_method 0 2023-10-30T12:10:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98133ef0 0x7f5f98069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 10 monclient(hunting): get_auth_request con 0x7f5f98133ef0 auth_method 0 2023-10-30T12:10:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9d82d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient(hunting): handle_auth_done global_id 134328 payload 824 2023-10-30T12:10:39.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98133ef0 msgr2=0x7f5f98069e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98133ef0 0x7f5f98069e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9809fcc0 msgr2=0x7f5f9812f940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:10:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9809fcc0 0x7f5f9812f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient: _finish_hunting 0 2023-10-30T12:10:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 monclient: found mon.noname-a 2023-10-30T12:10:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.645+0000 7f5f9c82b640 1 -- 172.21.15.50:0/4137074833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f88009a70 con 0x7f5f98133400 2023-10-30T12:10:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9c82b640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133400 0x7f5f9809eef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00c4a0 tx=0x7f5f8c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 client.?.objecter ms_handle_connect 0x7f5f98133400 2023-10-30T12:10:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c005ce0 con 0x7f5f98133400 2023-10-30T12:10:39.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: dump: 2023-10-30T12:10:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.680 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.682 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: _finish_auth 0 2023-10-30T12:10:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:10:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c005e40 con 0x7f5f98133400 2023-10-30T12:10:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 5 monclient: authenticate success, global_id 134328 2023-10-30T12:10:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 10 monclient: _renew_subs 2023-10-30T12:10:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9809f430 con 0x7f5f98133400 2023-10-30T12:10:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c007270 con 0x7f5f98133400 2023-10-30T12:10:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f95ffb640 10 monclient: dump: 2023-10-30T12:10:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.686 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f977fe640 4 set_mon_vals no callback set 2023-10-30T12:10:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 10 client.134328.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 10 monclient: _renew_subs 2023-10-30T12:10:39.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:39.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9806bab0 con 0x7f5f98133400 2023-10-30T12:10:39.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.646+0000 7f5f9f290640 1 librados: init done 2023-10-30T12:10:39.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9f290640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:39.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5f8c0073d0 con 0x7f5f98133400 2023-10-30T12:10:39.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9f290640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f9806a350 0x7f5f9806a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9f290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f98073ff0 con 0x7f5f9806a350 2023-10-30T12:10:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f9806a350 0x7f5f9806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient: get_auth_request con 0x7f5f9806a350 auth_method 0 2023-10-30T12:10:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f840392a0 0x7f5f8403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(328..328 src has 1..328) v4 ==== 6286+0+0 (secure 0 0 0) 0x7f5f8c04a100 con 0x7f5f98133400 2023-10-30T12:10:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 10 client.134328.objecter ms_dispatch 0x7f5f98000b90 osd_map(328..328 src has 1..328) v4 2023-10-30T12:10:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 3 client.134328.objecter handle_osd_map got epochs [328,328] > 0 2023-10-30T12:10:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 3 client.134328.objecter handle_osd_map decoding full epoch 328 2023-10-30T12:10:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_done global_id 114518 payload 824 2023-10-30T12:10:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 1 -- 172.21.15.50:0/4137074833 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f980a1da0 con 0x7f5f9806a350 2023-10-30T12:10:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 20 client.134328.objecter dump_active .. 0 homeless 2023-10-30T12:10:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 10 client.134328.objecter ms_handle_connect 0x7f5f9806a350 2023-10-30T12:10:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 10 client.134328.objecter resend_mon_ops 2023-10-30T12:10:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d02c640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f9806a350 0x7f5f9806a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f90003930 tx=0x7f5f9000b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f9d82d640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f840392a0 0x7f5f8403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.647+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90003e10 con 0x7f5f9806a350 2023-10-30T12:10:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 10 monclient: dump: 2023-10-30T12:10:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.694 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.694 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.694 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f9d82d640 10 monclient: get_auth_request con 0x7f5f840392a0 auth_method 0 2023-10-30T12:10:39.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5f90005070 con 0x7f5f9806a350 2023-10-30T12:10:39.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:39.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:10:39.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f9806a350 msgr2=0x7f5f9806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f9806a350 0x7f5f9806a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f90003930 tx=0x7f5f9000b8c0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f9806a350 msgr2=0x7f5f9806a730 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:39.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f9806a350 0x7f5f9806a730 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f90003930 tx=0x7f5f9000b8c0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f95ffb640 10 client.134328.objecter ms_handle_connect 0x7f5f840392a0 2023-10-30T12:10:39.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.648+0000 7f5f9d82d640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f840392a0 0x7f5f8403b760 secure :-1 s=READY pgs=4101 cs=0 l=1 rev1=1 crypto rx=0x7f5f88036000 tx=0x7f5f88030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9f290640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:10:39.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9f290640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98131880 0x7f5f98131c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:39.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9f290640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:39.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9809e320 con 0x7f5f98131880 2023-10-30T12:10:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98131880 0x7f5f98131c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient: get_auth_request con 0x7f5f98131880 auth_method 0 2023-10-30T12:10:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:39.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:39.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:39.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 10 monclient(hunting): handle_auth_done global_id 114521 payload 824 2023-10-30T12:10:39.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.654+0000 7f5f9d02c640 1 -- 172.21.15.50:0/4137074833 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5f9806ac70 con 0x7f5f98131880 2023-10-30T12:10:39.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 10 client.134328.objecter ms_handle_connect 0x7f5f98131880 2023-10-30T12:10:39.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9d02c640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98131880 0x7f5f98131c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5f90000c00 tx=0x7f5f90005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:39.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 10 client.134328.objecter resend_mon_ops 2023-10-30T12:10:39.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90020070 con 0x7f5f98131880 2023-10-30T12:10:39.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:39.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:10:39.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 10 monclient: dump: 2023-10-30T12:10:39.702 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:39.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:39.702 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.702 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:39.702 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:39.703 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:39.703 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:39.703 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:39.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:39.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:39.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f5f90010400 con 0x7f5f98131880 2023-10-30T12:10:39.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:10:39.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:10:39.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98131880 msgr2=0x7f5f98131c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98131880 0x7f5f98131c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5f90000c00 tx=0x7f5f90005a20 comp rx=0 tx=0).stop 2023-10-30T12:10:39.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98131880 msgr2=0x7f5f98131c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:10:39.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f95ffb640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f98131880 0x7f5f98131c90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.705 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:10:39.705 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:39.705 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:10:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 98, 2023-10-30T12:10:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:10:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:10:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:10:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:10:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:10:39.707 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:10:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:39.708 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:39.709 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:10:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:10:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:39.710 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:39.711 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:10:39.712 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:10:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:10:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:10:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:10:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:10:39.713 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:10:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:10:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:10:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:10:39.714 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:10:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:10:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:10:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:10:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:10:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:10:39.715 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:10:39.716 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:10:39.716 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:10:39.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:10:39.716 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:10:39.716 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:10:39.717 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:10:39.717 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:10:39.717 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:10:39.717 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:10:39.717 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.717 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:10:39.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.718 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:10:39.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.718 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:10:39.718 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:10:39.719 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:39.719 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:39.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:39.719 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:10:39.720 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.720 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:39.720 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:10:39.720 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.721 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.721 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.721 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:10:39.721 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:10:39.722 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:39.722 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:39.722 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:39.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:10:39.723 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:10:39.723 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:39.723 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:39.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.723 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:39.724 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:10:39.724 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.724 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:39.724 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:10:39.725 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.725 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.725 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:10:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:10:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:10:39.727 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:10:39.727 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:10:39.727 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:10:39.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.728 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:10:39.728 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:10:39.728 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.729 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:10:39.729 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:10:39.729 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:10:39.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.729 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.730 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:10:39.730 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:10:39.730 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:10:39.730 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:10:39.730 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:10:39.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.731 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.731 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:10:39.731 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:10:39.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.732 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:39.732 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:39.732 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:10:39.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.732 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.733 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:10:39.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.733 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:39.733 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:39.733 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:10:39.733 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.734 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:10:39.734 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:10:39.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:10:39.734 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:10:39.734 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:10:39.735 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:10:39.735 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:10:39.735 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:10:39.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:10:39.735 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:10:39.736 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:10:39.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 10 librados: watch_flush enter 2023-10-30T12:10:39.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 10 librados: watch_flush exit 2023-10-30T12:10:39.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 20 client.134328.objecter shutdown clearing up homeless session... 2023-10-30T12:10:39.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 10 client.134328.objecter successfully canceled tick 2023-10-30T12:10:39.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f840392a0 msgr2=0x7f5f8403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f840392a0 0x7f5f8403b760 secure :-1 s=READY pgs=4101 cs=0 l=1 rev1=1 crypto rx=0x7f5f88036000 tx=0x7f5f88030000 comp rx=0 tx=0).stop 2023-10-30T12:10:39.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 10 monclient: shutdown 2023-10-30T12:10:39.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:39.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133400 msgr2=0x7f5f9809eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:39.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133400 0x7f5f9809eef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00c4a0 tx=0x7f5f8c00c960 comp rx=0 tx=0).stop 2023-10-30T12:10:39.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9d82d640 1 -- 172.21.15.50:0/4137074833 reap_dead start 2023-10-30T12:10:39.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 shutdown_connections 2023-10-30T12:10:39.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 --2- 172.21.15.50:0/4137074833 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f98133400 0x7f5f9809eef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:39.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 >> 172.21.15.50:0/4137074833 conn(0x7f5f980a6200 msgr2=0x7f5f980a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:39.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 shutdown_connections 2023-10-30T12:10:39.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.655+0000 7f5f9f290640 1 -- 172.21.15.50:0/4137074833 wait complete. 2023-10-30T12:10:39.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:39.656+0000 7f5f9f290640 1 librados: shutdown 2023-10-30T12:10:39.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:10:39.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:10:59.745 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:10:59.746 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-10-30T12:10:59.746 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:10:59.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:10:59.749 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:10:59.750 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:10:59.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:10:59.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.865+0000 7f661bbd7640 10 monclient: get_monmap_and_config 2023-10-30T12:10:59.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.865+0000 7f661bbd7640 10 monclient: build_initial_monmap 2023-10-30T12:10:59.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.865+0000 7f661bbd7640 10 monclient: monmap: 2023-10-30T12:10:59.867 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:59.867 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:59.867 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:10:59.866094+0000 2023-10-30T12:10:59.868 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:10:59.866094+0000 2023-10-30T12:10:59.868 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:59.868 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:59.868 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:10:59.868 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:10:59.869 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:10:59.869 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:10:59.869 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:10:59.869 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:59.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.865+0000 7f661bbd7640 1 Processor -- start 2023-10-30T12:10:59.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.865+0000 7f661bbd7640 1 -- start start 2023-10-30T12:10:59.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.865+0000 7f661bbd7640 10 monclient: init 2023-10-30T12:10:59.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:59.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-10-30T12:10:59.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 0x7f661409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:59.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient(hunting): picked mon.noname-c con 0x7f6614098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:10:59.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6614098900 0x7f6614098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:59.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f6614098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:10:59.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6614132740 addr v1:172.21.15.50:6789/0 2023-10-30T12:10:59.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661409ad30 con 0x7f6614132740 2023-10-30T12:10:59.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:59.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661409c460 con 0x7f6614098900 2023-10-30T12:10:59.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:59.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661409b260 con 0x7f6614098520 2023-10-30T12:10:59.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:59.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661bbd7640 10 monclient(hunting): authenticate will time out at 1505.049375s 2023-10-30T12:10:59.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6614098900 0x7f6614098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:59.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661a174640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6614132740 0x7f6614132bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37660/0 (socket says 172.21.15.50:37660) 2023-10-30T12:10:59.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f661a174640 1 -- 172.21.15.50:0/270501762 learned_addr learned my addr 172.21.15.50:0/270501762 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:10:59.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6614098900 0x7f6614098d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53010/0 (socket says 172.21.15.50:53010) 2023-10-30T12:10:59.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): get_auth_request con 0x7f6614098900 auth_method 0 2023-10-30T12:10:59.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:59.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:59.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:59.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:59.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:59.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 1 --2- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 0x7f661409bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:59.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:59.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 10 monclient(hunting): get_auth_request con 0x7f6614098520 auth_method 0 2023-10-30T12:10:59.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:59.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:59.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:59.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 1 -- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 msgr2=0x7f661409bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:10:59.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 1 -- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 msgr2=0x7f661409bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:10:59.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 1 --2- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 0x7f661409bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:10:59.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619973640 1 --2- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 0x7f661409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:10:59.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient(hunting): handle_auth_done global_id 134331 payload 274 2023-10-30T12:10:59.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 1 -- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 msgr2=0x7f661409bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:10:59.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 1 --2- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 0x7f661409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:59.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 1 -- 172.21.15.50:0/270501762 >> v1:172.21.15.50:6789/0 conn(0x7f6614132740 legacy=0x7f6614132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:59.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient: _finish_hunting 0 2023-10-30T12:10:59.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 1 monclient: found mon.noname-a 2023-10-30T12:10:59.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6618971640 1 -- 172.21.15.50:0/270501762 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224959333 0 0) 0x7f661409ad30 con 0x7f6614132740 2023-10-30T12:10:59.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:59.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:59.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6619172640 1 -- 172.21.15.50:0/270501762 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614072120 con 0x7f6614098900 2023-10-30T12:10:59.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.866+0000 7f6618971640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:10:59.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6619172640 1 --2- 172.21.15.50:0/270501762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098900 0x7f6614098d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6614073e60 tx=0x7f660402df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855e2cffbe4214b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:59.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 1 -- 172.21.15.50:0/270501762 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660402ed30 con 0x7f6614098900 2023-10-30T12:10:59.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:59.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:59.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: dump: 2023-10-30T12:10:59.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:59.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:59.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.887 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:59.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:59.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:59.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:59.888 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:59.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:59.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: _finish_auth 0 2023-10-30T12:10:59.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: _check_auth_tickets 2023-10-30T12:10:59.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:59.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f661bbd7640 5 monclient: authenticate success, global_id 134331 2023-10-30T12:10:59.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f661bbd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:10:59.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 1 -- 172.21.15.50:0/270501762 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660402ee90 con 0x7f6614098900 2023-10-30T12:10:59.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:59.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f661bbd7640 10 monclient: get_monmap_and_config success 2023-10-30T12:10:59.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 1 -- 172.21.15.50:0/270501762 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604036660 con 0x7f6614098900 2023-10-30T12:10:59.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:59.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f661bbd7640 4 set_mon_vals no callback set 2023-10-30T12:10:59.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:59.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.867+0000 7f6618971640 10 monclient: dump: 2023-10-30T12:10:59.891 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:59.892 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:59.892 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.892 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.892 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:59.892 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:59.893 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:59.893 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:59.893 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:59.893 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:59.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 10 monclient: shutdown 2023-10-30T12:10:59.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:10:59.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 -- 172.21.15.50:0/270501762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098900 msgr2=0x7f6614098d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:59.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 --2- 172.21.15.50:0/270501762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098900 0x7f6614098d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6614073e60 tx=0x7f660402df80 comp rx=0 tx=0).stop 2023-10-30T12:10:59.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 -- 172.21.15.50:0/270501762 shutdown_connections 2023-10-30T12:10:59.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 --2- 172.21.15.50:0/270501762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098900 0x7f6614098d10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:59.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 --2- 172.21.15.50:0/270501762 >> v2:172.21.15.118:3300/0 conn(0x7f6614098520 0x7f661409bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:59.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 -- 172.21.15.50:0/270501762 >> 172.21.15.50:0/270501762 conn(0x7f66140a05c0 msgr2=0x7f661409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:10:59.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 -- 172.21.15.50:0/270501762 shutdown_connections 2023-10-30T12:10:59.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.868+0000 7f661bbd7640 1 -- 172.21.15.50:0/270501762 wait complete. 2023-10-30T12:10:59.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient: build_initial_monmap 2023-10-30T12:10:59.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient: monmap: 2023-10-30T12:10:59.895 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:10:59.896 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:10:59.896 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:10:59.896 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:10:59.896 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:10:59.896 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:59.897 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:10:59.897 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:10:59.897 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:10:59.897 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:59.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 librados: starting msgr at 2023-10-30T12:10:59.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 librados: starting objecter 2023-10-30T12:10:59.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 Processor -- start 2023-10-30T12:10:59.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 -- start start 2023-10-30T12:10:59.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 librados: setting wanted keys 2023-10-30T12:10:59.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 librados: calling monclient init 2023-10-30T12:10:59.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient: init 2023-10-30T12:10:59.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:10:59.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:10:59.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098520 0x7f66141c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:59.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f6614098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:10:59.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6614098900 0x7f66141c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:59.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): picked mon.noname-b con 0x7f6614098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:10:59.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6614132740 0x7f6614129fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:59.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): picked mon.noname-c con 0x7f6614132740 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:10:59.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:59.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6614071df0 con 0x7f6614098520 2023-10-30T12:10:59.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:59.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66140a2d80 con 0x7f6614098900 2023-10-30T12:10:59.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:10:59.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661412dbe0 con 0x7f6614132740 2023-10-30T12:10:59.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:10:59.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.869+0000 7f661bbd7640 10 monclient(hunting): authenticate will time out at 1505.052990s 2023-10-30T12:10:59.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098520 0x7f66141c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:59.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098520 0x7f66141c5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53016/0 (socket says 172.21.15.50:53016) 2023-10-30T12:10:59.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 -- 172.21.15.50:0/2641160718 learned_addr learned my addr 172.21.15.50:0/2641160718 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:10:59.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): get_auth_request con 0x7f6614098520 auth_method 0 2023-10-30T12:10:59.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:59.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:59.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:59.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6614132740 0x7f6614129fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:59.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 10 monclient(hunting): get_auth_request con 0x7f6614132740 auth_method 0 2023-10-30T12:10:59.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:59.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:59.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:59.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:59.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:59.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:59.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619172640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6614098900 0x7f66141c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:59.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:10:59.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:10:59.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661a174640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:10:59.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619172640 10 monclient(hunting): get_auth_request con 0x7f6614098900 auth_method 0 2023-10-30T12:10:59.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619172640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:10:59.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619172640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:10:59.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619172640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:10:59.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient(hunting): handle_auth_done global_id 134337 payload 824 2023-10-30T12:10:59.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 -- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6614132740 msgr2=0x7f6614129fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:59.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6614132740 0x7f6614129fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:59.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 -- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6614098900 msgr2=0x7f66141c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:10:59.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6614098900 0x7f66141c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:10:59.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient: _finish_hunting 0 2023-10-30T12:10:59.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 monclient: found mon.noname-a 2023-10-30T12:10:59.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:10:59.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:10:59.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 -- 172.21.15.50:0/2641160718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660400b6a0 con 0x7f6614098520 2023-10-30T12:10:59.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f6619973640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098520 0x7f66141c5c10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f660c00d800 tx=0x7f660c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:59.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 client.?.objecter ms_handle_connect 0x7f6614098520 2023-10-30T12:10:59.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:10:59.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 1 -- 172.21.15.50:0/2641160718 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c018070 con 0x7f6614098520 2023-10-30T12:10:59.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:59.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:10:59.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: dump: 2023-10-30T12:10:59.911 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:59.911 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:59.911 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.911 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.912 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:59.912 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:59.912 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:59.912 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:59.912 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:59.912 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:59.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:10:59.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:10:59.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:10:59.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 1 -- 172.21.15.50:0/2641160718 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c005990 con 0x7f6614098520 2023-10-30T12:10:59.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:10:59.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 1 -- 172.21.15.50:0/2641160718 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c005b20 con 0x7f6614098520 2023-10-30T12:10:59.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 5 monclient: authenticate success, global_id 134337 2023-10-30T12:10:59.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 10 monclient: _renew_subs 2023-10-30T12:10:59.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:59.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661412d620 con 0x7f6614098520 2023-10-30T12:10:59.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:10:59.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:10:59.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660bfff640 4 set_mon_vals no callback set 2023-10-30T12:10:59.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f660a7fc640 10 monclient: dump: 2023-10-30T12:10:59.915 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:10:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:10:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:10:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:10:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:10:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:10:59.917 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:10:59.917 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:10:59.917 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:10:59.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 10 client.134337.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:10:59.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 10 monclient: _renew_subs 2023-10-30T12:10:59.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:59.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661412e8a0 con 0x7f6614098520 2023-10-30T12:10:59.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.870+0000 7f661bbd7640 1 librados: init done 2023-10-30T12:10:59.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.871+0000 7f661bbd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:59.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.871+0000 7f661bbd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:59.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.871+0000 7f661bbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:10:59.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.871+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661404e160 con 0x7f6614098520 2023-10-30T12:10:59.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.871+0000 7f660a7fc640 1 -- 172.21.15.50:0/2641160718 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f660c00b6c0 con 0x7f6614098520 2023-10-30T12:10:59.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f660a7fc640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65f80392f0 0x7f65f803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:10:59.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f660a7fc640 1 -- 172.21.15.50:0/2641160718 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(339..339 src has 1..339) v4 ==== 5914+0+0 (secure 0 0 0) 0x7f660c04a9c0 con 0x7f6614098520 2023-10-30T12:10:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f660a7fc640 10 client.134337.objecter ms_dispatch 0x7f6614000b90 osd_map(339..339 src has 1..339) v4 2023-10-30T12:10:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f660a7fc640 3 client.134337.objecter handle_osd_map got epochs [339,339] > 0 2023-10-30T12:10:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f660a7fc640 3 client.134337.objecter handle_osd_map decoding full epoch 339 2023-10-30T12:10:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f660a7fc640 20 client.134337.objecter dump_active .. 0 homeless 2023-10-30T12:10:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f6619172640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65f80392f0 0x7f65f803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:10:59.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f6619172640 10 monclient: get_auth_request con 0x7f65f80392f0 auth_method 0 2023-10-30T12:10:59.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f660a7fc640 10 client.134337.objecter ms_handle_connect 0x7f65f80392f0 2023-10-30T12:10:59.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.872+0000 7f6619172640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65f80392f0 0x7f65f803b7b0 secure :-1 s=READY pgs=4103 cs=0 l=1 rev1=1 crypto rx=0x7f6604038000 tx=0x7f6604032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:10:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.877+0000 7f660a7fc640 1 -- 172.21.15.50:0/2641160718 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f660c00b970 con 0x7f6614098520 2023-10-30T12:10:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.877+0000 7f660a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:10:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:10:59.877+0000 7f660a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:00.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.118+0000 7f661bbd7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:00.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.118+0000 7f661bbd7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:00.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.118+0000 7f661bbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:00.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.118+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f66140745c0 con 0x7f6614098520 2023-10-30T12:11:00.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.118+0000 7f660a7fc640 1 -- 172.21.15.50:0/2641160718 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f660c013b80 con 0x7f6614098520 2023-10-30T12:11:00.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.118+0000 7f660a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:00.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.118+0000 7f660a7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:00.122 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:00.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 10 librados: watch_flush enter 2023-10-30T12:11:00.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 10 librados: watch_flush exit 2023-10-30T12:11:00.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 20 client.134337.objecter shutdown clearing up homeless session... 2023-10-30T12:11:00.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 10 client.134337.objecter successfully canceled tick 2023-10-30T12:11:00.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65f80392f0 msgr2=0x7f65f803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:00.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65f80392f0 0x7f65f803b7b0 secure :-1 s=READY pgs=4103 cs=0 l=1 rev1=1 crypto rx=0x7f6604038000 tx=0x7f6604032000 comp rx=0 tx=0).stop 2023-10-30T12:11:00.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 10 monclient: shutdown 2023-10-30T12:11:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098520 msgr2=0x7f66141c5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098520 0x7f66141c5c10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f660c00d800 tx=0x7f660c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:11:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 shutdown_connections 2023-10-30T12:11:00.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65f80392f0 0x7f65f803b7b0 unknown :-1 s=CLOSED pgs=4103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:00.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6614132740 0x7f6614129fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:00.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6614098900 0x7f66141c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:00.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 --2- 172.21.15.50:0/2641160718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6614098520 0x7f66141c5c10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:00.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 >> 172.21.15.50:0/2641160718 conn(0x7f66140a05c0 msgr2=0x7f661409dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 shutdown_connections 2023-10-30T12:11:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 -- 172.21.15.50:0/2641160718 wait complete. 2023-10-30T12:11:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:00.120+0000 7f661bbd7640 1 librados: shutdown 2023-10-30T12:11:03.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.246+0000 7f35f6b25640 10 monclient: get_monmap_and_config 2023-10-30T12:11:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.246+0000 7f35f6b25640 10 monclient: build_initial_monmap 2023-10-30T12:11:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.246+0000 7f35f6b25640 10 monclient: monmap: 2023-10-30T12:11:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:03.247737+0000 2023-10-30T12:11:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:03.247737+0000 2023-10-30T12:11:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:03.250 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:03.250 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:03.250 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:03.250 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:03.250 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:03.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 1 Processor -- start 2023-10-30T12:11:03.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 1 -- start start 2023-10-30T12:11:03.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient: init 2023-10-30T12:11:03.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:03.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T12:11:03.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f35f009f5b0 0x7f35f0099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:03.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient(hunting): picked mon.noname-c con 0x7f35f009f5b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:03.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f0096080 0x7f35f0096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:03.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient(hunting): picked mon.noname-b con 0x7f35f0096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:03.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35f00969a0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:03.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f0098810 con 0x7f35f00969a0 2023-10-30T12:11:03.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:03.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f006aef0 con 0x7f35f009f5b0 2023-10-30T12:11:03.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:03.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f009a1a0 con 0x7f35f0096080 2023-10-30T12:11:03.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:03.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.247+0000 7f35f6b25640 10 monclient(hunting): authenticate will time out at 1508.430933s 2023-10-30T12:11:03.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f0096080 0x7f35f0096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:03.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35f50c2640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f35f00969a0 0x7f35f009cb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37672/0 (socket says 172.21.15.50:37672) 2023-10-30T12:11:03.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f0096080 0x7f35f0096460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44332/0 (socket says 172.21.15.50:44332) 2023-10-30T12:11:03.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35f50c2640 1 -- 172.21.15.50:0/1071928590 learned_addr learned my addr 172.21.15.50:0/1071928590 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:11:03.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): get_auth_request con 0x7f35f0096080 auth_method 0 2023-10-30T12:11:03.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:03.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:03.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:03.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:03.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:03.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:03.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35f48c1640 1 --2- 172.21.15.50:0/1071928590 >> v2:172.21.15.118:3300/0 conn(0x7f35f009f5b0 0x7f35f0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:03.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35f48c1640 10 monclient(hunting): get_auth_request con 0x7f35f009f5b0 auth_method 0 2023-10-30T12:11:03.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35f48c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:03.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35f48c1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:03.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35f48c1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:03.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient(hunting): handle_auth_done global_id 114527 payload 274 2023-10-30T12:11:03.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35ef7fe640 1 -- 172.21.15.50:0/1071928590 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968756140 0 0) 0x7f35f0098810 con 0x7f35f00969a0 2023-10-30T12:11:03.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 1 -- 172.21.15.50:0/1071928590 >> v2:172.21.15.118:3300/0 conn(0x7f35f009f5b0 msgr2=0x7f35f0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:03.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 1 --2- 172.21.15.50:0/1071928590 >> v2:172.21.15.118:3300/0 conn(0x7f35f009f5b0 0x7f35f0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 1 -- 172.21.15.50:0/1071928590 >> v1:172.21.15.50:6789/0 conn(0x7f35f00969a0 legacy=0x7f35f009cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:03.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient: _finish_hunting 0 2023-10-30T12:11:03.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 1 monclient: found mon.noname-b 2023-10-30T12:11:03.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:03.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:03.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35effff640 1 -- 172.21.15.50:0/1071928590 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0098af0 con 0x7f35f0096080 2023-10-30T12:11:03.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.248+0000 7f35ef7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:11:03.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35effff640 1 --2- 172.21.15.50:0/1071928590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f0096080 0x7f35f0096460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0097d0 tx=0x7f35dc02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29e212d34afc1840 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:03.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 1 -- 172.21.15.50:0/1071928590 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0316f0 con 0x7f35f0096080 2023-10-30T12:11:03.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:03.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:03.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: dump: 2023-10-30T12:11:03.268 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:03.268 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:03.268 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.268 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.269 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:03.269 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:03.269 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:03.269 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:03.269 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:03.269 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:03.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:03.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:03.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:03.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 1 -- 172.21.15.50:0/1071928590 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc031850 con 0x7f35f0096080 2023-10-30T12:11:03.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 5 monclient: authenticate success, global_id 114527 2023-10-30T12:11:03.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:03.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:03.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:03.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 1 -- 172.21.15.50:0/1071928590 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc031be0 con 0x7f35f0096080 2023-10-30T12:11:03.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:03.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 4 set_mon_vals no callback set 2023-10-30T12:11:03.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:03.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35ef7fe640 10 monclient: dump: 2023-10-30T12:11:03.272 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:03.272 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:03.273 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.273 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.273 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:03.273 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:03.273 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:03.274 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:03.274 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:03.274 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:03.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 10 monclient: shutdown 2023-10-30T12:11:03.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:03.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 1 -- 172.21.15.50:0/1071928590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f0096080 msgr2=0x7f35f0096460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:03.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 1 --2- 172.21.15.50:0/1071928590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f0096080 0x7f35f0096460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0097d0 tx=0x7f35dc02b790 comp rx=0 tx=0).stop 2023-10-30T12:11:03.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 1 -- 172.21.15.50:0/1071928590 shutdown_connections 2023-10-30T12:11:03.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 1 --2- 172.21.15.50:0/1071928590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f0096080 0x7f35f0096460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 1 --2- 172.21.15.50:0/1071928590 >> v2:172.21.15.118:3300/0 conn(0x7f35f009f5b0 0x7f35f0099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.249+0000 7f35f6b25640 1 -- 172.21.15.50:0/1071928590 >> 172.21.15.50:0/1071928590 conn(0x7f35f00a05c0 msgr2=0x7f35f009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:03.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 -- 172.21.15.50:0/1071928590 shutdown_connections 2023-10-30T12:11:03.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 -- 172.21.15.50:0/1071928590 wait complete. 2023-10-30T12:11:03.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 10 monclient: build_initial_monmap 2023-10-30T12:11:03.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 10 monclient: monmap: 2023-10-30T12:11:03.277 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:03.277 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:03.277 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:03.277 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:03.277 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:03.277 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:03.278 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:03.278 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:03.278 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:03.278 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:03.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 librados: starting msgr at 2023-10-30T12:11:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 librados: starting objecter 2023-10-30T12:11:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 Processor -- start 2023-10-30T12:11:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 -- start start 2023-10-30T12:11:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 librados: setting wanted keys 2023-10-30T12:11:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 1 librados: calling monclient init 2023-10-30T12:11:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.250+0000 7f35f6b25640 10 monclient: init 2023-10-30T12:11:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:11:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 0x7f35f01c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): picked mon.noname-b con 0x7f35f0096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f00969a0 0x7f35f00951a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): picked mon.noname-c con 0x7f35f00969a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35f009f5b0 0x7f35f00956e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): picked mon.noname-a con 0x7f35f009f5b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f008d650 con 0x7f35f009f5b0 2023-10-30T12:11:03.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:03.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f01b95c0 con 0x7f35f0096080 2023-10-30T12:11:03.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:03.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0071a20 con 0x7f35f00969a0 2023-10-30T12:11:03.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:03.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f6b25640 10 monclient(hunting): authenticate will time out at 1508.434680s 2023-10-30T12:11:03.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f00969a0 0x7f35f00951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:03.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f00969a0 0x7f35f00951a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44342/0 (socket says 172.21.15.50:44342) 2023-10-30T12:11:03.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35f009f5b0 0x7f35f00956e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:03.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 1 -- 172.21.15.50:0/607604397 learned_addr learned my addr 172.21.15.50:0/607604397 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:03.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 10 monclient(hunting): get_auth_request con 0x7f35f00969a0 auth_method 0 2023-10-30T12:11:03.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:03.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:03.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:03.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 10 monclient(hunting): get_auth_request con 0x7f35f009f5b0 auth_method 0 2023-10-30T12:11:03.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:03.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:03.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:03.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:03.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:03.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:03.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:03.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:03.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f50c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:03.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f48c1640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 0x7f35f01c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:03.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f48c1640 10 monclient(hunting): get_auth_request con 0x7f35f0096080 auth_method 0 2023-10-30T12:11:03.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f48c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:03.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f48c1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:03.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.251+0000 7f35f48c1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:03.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f48c1640 1 -- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 msgr2=0x7f35f01c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:11:03.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f48c1640 1 -- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 msgr2=0x7f35f01c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:11:03.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 10 monclient(hunting): handle_auth_done global_id 114530 payload 824 2023-10-30T12:11:03.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f48c1640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 0x7f35f01c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:03.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 1 -- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 msgr2=0x7f35f01c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:03.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f48c1640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 0x7f35f01c6350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:03.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 0x7f35f01c6350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 1 -- 172.21.15.50:0/607604397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35f009f5b0 msgr2=0x7f35f00956e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:03.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35f009f5b0 0x7f35f00956e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 10 monclient: _finish_hunting 0 2023-10-30T12:11:03.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 1 monclient: found mon.noname-c 2023-10-30T12:11:03.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:03.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:03.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 1 -- 172.21.15.50:0/607604397 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc009400 con 0x7f35f00969a0 2023-10-30T12:11:03.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35effff640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f00969a0 0x7f35f00951a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000c00 tx=0x7f35dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:03.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 client.?.objecter ms_handle_connect 0x7f35f00969a0 2023-10-30T12:11:03.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:03.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 1 -- 172.21.15.50:0/607604397 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc034890 con 0x7f35f00969a0 2023-10-30T12:11:03.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:03.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:03.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: dump: 2023-10-30T12:11:03.291 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:03.291 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:03.291 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.292 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.292 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:03.292 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:03.292 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:03.292 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:03.293 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:03.293 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:03.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: _finish_auth 0 2023-10-30T12:11:03.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:11:03.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:03.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 1 -- 172.21.15.50:0/607604397 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc0349f0 con 0x7f35f00969a0 2023-10-30T12:11:03.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 5 monclient: authenticate success, global_id 114530 2023-10-30T12:11:03.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 10 monclient: _renew_subs 2023-10-30T12:11:03.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:03.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f008d370 con 0x7f35f00969a0 2023-10-30T12:11:03.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:03.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 1 -- 172.21.15.50:0/607604397 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc034da0 con 0x7f35f00969a0 2023-10-30T12:11:03.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:03.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35ed7fa640 10 monclient: dump: 2023-10-30T12:11:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:03.297 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:03.297 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:03.297 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:03.297 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:03.297 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:03.297 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35eeffd640 4 set_mon_vals no callback set 2023-10-30T12:11:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 10 client.114530.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 10 monclient: _renew_subs 2023-10-30T12:11:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f0095c20 con 0x7f35f00969a0 2023-10-30T12:11:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.252+0000 7f35f6b25640 1 librados: init done 2023-10-30T12:11:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35ed7fa640 1 -- 172.21.15.50:0/607604397 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f35dc03b040 con 0x7f35f00969a0 2023-10-30T12:11:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35f6b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35ed7fa640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35d4039320 0x7f35d403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35f6b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:03.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35f6b25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:03.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f008db50 con 0x7f35f00969a0 2023-10-30T12:11:03.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35ed7fa640 1 -- 172.21.15.50:0/607604397 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(342..342 src has 1..342) v4 ==== 5914+0+0 (secure 0 0 0) 0x7f35dc06c3b0 con 0x7f35f00969a0 2023-10-30T12:11:03.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35ed7fa640 10 client.114530.objecter ms_dispatch 0x7f35f0000b90 osd_map(342..342 src has 1..342) v4 2023-10-30T12:11:03.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35ed7fa640 3 client.114530.objecter handle_osd_map got epochs [342,342] > 0 2023-10-30T12:11:03.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35ed7fa640 3 client.114530.objecter handle_osd_map decoding full epoch 342 2023-10-30T12:11:03.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35ed7fa640 20 client.114530.objecter dump_active .. 0 homeless 2023-10-30T12:11:03.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35f48c1640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35d4039320 0x7f35d403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:03.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.253+0000 7f35f48c1640 10 monclient: get_auth_request con 0x7f35d4039320 auth_method 0 2023-10-30T12:11:03.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.254+0000 7f35ed7fa640 10 client.114530.objecter ms_handle_connect 0x7f35d4039320 2023-10-30T12:11:03.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.254+0000 7f35f48c1640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35d4039320 0x7f35d403b7e0 secure :-1 s=READY pgs=4104 cs=0 l=1 rev1=1 crypto rx=0x7f35e4006a00 tx=0x7f35e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:03.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.258+0000 7f35ed7fa640 1 -- 172.21.15.50:0/607604397 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f35f008db50 con 0x7f35f00969a0 2023-10-30T12:11:03.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.258+0000 7f35ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:03.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.258+0000 7f35ed7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:03.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.505+0000 7f35f6b25640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:03.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.505+0000 7f35f6b25640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.505+0000 7f35f6b25640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.505+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f35f008db50 con 0x7f35f00969a0 2023-10-30T12:11:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.506+0000 7f35ed7fa640 1 -- 172.21.15.50:0/607604397 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f35f008db50 con 0x7f35f00969a0 2023-10-30T12:11:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.506+0000 7f35ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:03.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.506+0000 7f35ed7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:03.511 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 10 librados: watch_flush enter 2023-10-30T12:11:03.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 10 librados: watch_flush exit 2023-10-30T12:11:03.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 20 client.114530.objecter shutdown clearing up homeless session... 2023-10-30T12:11:03.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 10 client.114530.objecter successfully canceled tick 2023-10-30T12:11:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35d4039320 msgr2=0x7f35d403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35d4039320 0x7f35d403b7e0 secure :-1 s=READY pgs=4104 cs=0 l=1 rev1=1 crypto rx=0x7f35e4006a00 tx=0x7f35e4008040 comp rx=0 tx=0).stop 2023-10-30T12:11:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 10 monclient: shutdown 2023-10-30T12:11:03.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:03.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f00969a0 msgr2=0x7f35f00951a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:03.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f00969a0 0x7f35f00951a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000c00 tx=0x7f35dc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 shutdown_connections 2023-10-30T12:11:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f35d4039320 0x7f35d403b7e0 unknown :-1 s=CLOSED pgs=4104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35f009f5b0 0x7f35f00956e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f35f00969a0 0x7f35f00951a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 --2- 172.21.15.50:0/607604397 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35f0096080 0x7f35f01c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:03.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 >> 172.21.15.50:0/607604397 conn(0x7f35f00a05c0 msgr2=0x7f35f00a13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:03.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 shutdown_connections 2023-10-30T12:11:03.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 -- 172.21.15.50:0/607604397 wait complete. 2023-10-30T12:11:03.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:03.508+0000 7f35f6b25640 1 librados: shutdown 2023-10-30T12:11:06.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:06.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.642+0000 7fd3d4b55640 10 monclient: get_monmap_and_config 2023-10-30T12:11:06.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.642+0000 7fd3d4b55640 10 monclient: build_initial_monmap 2023-10-30T12:11:06.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.642+0000 7fd3d4b55640 10 monclient: monmap: 2023-10-30T12:11:06.644 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:06.644 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:06.644 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:06.643391+0000 2023-10-30T12:11:06.645 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:06.643391+0000 2023-10-30T12:11:06.645 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:06.645 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:06.645 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:06.645 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:06.646 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:06.646 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:06.646 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:06.646 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:06.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.642+0000 7fd3d4b55640 1 Processor -- start 2023-10-30T12:11:06.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.642+0000 7fd3d4b55640 1 -- start start 2023-10-30T12:11:06.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.642+0000 7fd3d4b55640 10 monclient: init 2023-10-30T12:11:06.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:06.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:11:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3d00a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd3d0099a40 0x7fd3d013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d0099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3d0132db0 0x7fd3d013cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d0132db0 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d00a4f40 con 0x7fd3d00a4b60 2023-10-30T12:11:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d00a3d70 con 0x7fd3d0132db0 2023-10-30T12:11:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d00a4150 con 0x7fd3d0099a40 2023-10-30T12:11:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3d4b55640 10 monclient(hunting): authenticate will time out at 1511.826734s 2023-10-30T12:11:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3d0132db0 0x7fd3d013cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3d0132db0 0x7fd3d013cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53024/0 (socket says 172.21.15.50:53024) 2023-10-30T12:11:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 1 -- 172.21.15.50:0/1047419313 learned_addr learned my addr 172.21.15.50:0/1047419313 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 10 monclient(hunting): get_auth_request con 0x7fd3d0132db0 auth_method 0 2023-10-30T12:11:06.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:06.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:06.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:06.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:06.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.643+0000 7fd3ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3cdd9b640 1 --2- 172.21.15.50:0/1047419313 >> v2:172.21.15.118:3300/0 conn(0x7fd3d0099a40 0x7fd3d013aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 10 monclient(hunting): handle_auth_done global_id 134346 payload 274 2023-10-30T12:11:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 1 -- 172.21.15.50:0/1047419313 >> v2:172.21.15.118:3300/0 conn(0x7fd3d0099a40 msgr2=0x7fd3d013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 1 --2- 172.21.15.50:0/1047419313 >> v2:172.21.15.118:3300/0 conn(0x7fd3d0099a40 0x7fd3d013aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 1 -- 172.21.15.50:0/1047419313 >> v1:172.21.15.118:6789/0 conn(0x7fd3d00a4b60 legacy=0x7fd3d01326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 10 monclient: _finish_hunting 0 2023-10-30T12:11:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 1 monclient: found mon.noname-a 2023-10-30T12:11:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:06.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 1 -- 172.21.15.50:0/1047419313 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d009a440 con 0x7fd3d0132db0 2023-10-30T12:11:06.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3cdd9b640 10 monclient: get_auth_request con 0x7fd3d0099a40 auth_method 0 2023-10-30T12:11:06.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3cdd9b640 1 --2- 172.21.15.50:0/1047419313 >> v2:172.21.15.118:3300/0 conn(0x7fd3d0099a40 0x7fd3d013aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:11:06.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.644+0000 7fd3ced9d640 1 --2- 172.21.15.50:0/1047419313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d0132db0 0x7fd3d013cfd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0099ed0 tx=0x7fd3c002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6faa2514585bb01 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:06.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 1 -- 172.21.15.50:0/1047419313 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c002c960 con 0x7fd3d0132db0 2023-10-30T12:11:06.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:06.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:06.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: dump: 2023-10-30T12:11:06.662 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:06.662 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:06.664 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: _finish_auth 0 2023-10-30T12:11:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: _check_auth_tickets 2023-10-30T12:11:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 1 -- 172.21.15.50:0/1047419313 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c002cac0 con 0x7fd3d0132db0 2023-10-30T12:11:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 1 -- 172.21.15.50:0/1047419313 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c002ce50 con 0x7fd3d0132db0 2023-10-30T12:11:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 5 monclient: authenticate success, global_id 134346 2023-10-30T12:11:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 4 set_mon_vals no callback set 2023-10-30T12:11:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3cd59a640 10 monclient: dump: 2023-10-30T12:11:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:06.668 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 10 monclient: shutdown 2023-10-30T12:11:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 -- 172.21.15.50:0/1047419313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d0132db0 msgr2=0x7fd3d013cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/1047419313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d0132db0 0x7fd3d013cfd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0099ed0 tx=0x7fd3c002bdc0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 -- 172.21.15.50:0/1047419313 shutdown_connections 2023-10-30T12:11:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/1047419313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d0132db0 0x7fd3d013cfd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/1047419313 >> v2:172.21.15.118:3300/0 conn(0x7fd3d0099a40 0x7fd3d013aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 -- 172.21.15.50:0/1047419313 >> 172.21.15.50:0/1047419313 conn(0x7fd3d00a8060 msgr2=0x7fd3d00a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 -- 172.21.15.50:0/1047419313 shutdown_connections 2023-10-30T12:11:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.645+0000 7fd3d4b55640 1 -- 172.21.15.50:0/1047419313 wait complete. 2023-10-30T12:11:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.646+0000 7fd3d4b55640 10 monclient: build_initial_monmap 2023-10-30T12:11:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.646+0000 7fd3d4b55640 10 monclient: monmap: 2023-10-30T12:11:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:06.672 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.646+0000 7fd3d4b55640 1 librados: starting msgr at 2023-10-30T12:11:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.646+0000 7fd3d4b55640 1 librados: starting objecter 2023-10-30T12:11:06.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.646+0000 7fd3d4b55640 1 Processor -- start 2023-10-30T12:11:06.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 -- start start 2023-10-30T12:11:06.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 librados: setting wanted keys 2023-10-30T12:11:06.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 librados: calling monclient init 2023-10-30T12:11:06.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient: init 2023-10-30T12:11:06.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:06.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:11:06.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d00a4b60 0x7fd3d009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:06.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d00a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:06.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3d0132db0 0x7fd3d009b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:06.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d0132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:06.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3d009b960 0x7fd3d01ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:06.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d009b960 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:06.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:06.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d009dd00 con 0x7fd3d00a4b60 2023-10-30T12:11:06.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:06.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d009ce80 con 0x7fd3d009b960 2023-10-30T12:11:06.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:06.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d013cb40 con 0x7fd3d0132db0 2023-10-30T12:11:06.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:06.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3d4b55640 10 monclient(hunting): authenticate will time out at 1511.830569s 2023-10-30T12:11:06.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d00a4b60 0x7fd3d009aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:06.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3d0132db0 0x7fd3d009b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:06.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d00a4b60 0x7fd3d009aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53040/0 (socket says 172.21.15.50:53040) 2023-10-30T12:11:06.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 1 -- 172.21.15.50:0/4067743866 learned_addr learned my addr 172.21.15.50:0/4067743866 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:06.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3d0132db0 0x7fd3d009b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44358/0 (socket says 172.21.15.50:44358) 2023-10-30T12:11:06.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 10 monclient(hunting): get_auth_request con 0x7fd3d00a4b60 auth_method 0 2023-10-30T12:11:06.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:06.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:06.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:06.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): get_auth_request con 0x7fd3d0132db0 auth_method 0 2023-10-30T12:11:06.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3ced9d640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3d009b960 0x7fd3d01ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 10 monclient(hunting): handle_auth_done global_id 114533 payload 824 2023-10-30T12:11:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 1 -- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3d009b960 msgr2=0x7fd3d01ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:06.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3d009b960 0x7fd3d01ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 1 -- 172.21.15.50:0/4067743866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d00a4b60 msgr2=0x7fd3d009aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.647+0000 7fd3cdd9b640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d00a4b60 0x7fd3d009aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3cdd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:11:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3cdd9b640 1 monclient: found mon.noname-c 2023-10-30T12:11:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3cdd9b640 1 -- 172.21.15.50:0/4067743866 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c0009b30 con 0x7fd3d0132db0 2023-10-30T12:11:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3ced9d640 10 monclient: get_auth_request con 0x7fd3d009b960 auth_method 0 2023-10-30T12:11:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3ced9d640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3d009b960 0x7fd3d01ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:11:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3cdd9b640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3d0132db0 0x7fd3d009b420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c5b0 tx=0x7fd3c400ca70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 client.?.objecter ms_handle_connect 0x7fd3d0132db0 2023-10-30T12:11:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 1 -- 172.21.15.50:0/4067743866 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4011800 con 0x7fd3d0132db0 2023-10-30T12:11:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: dump: 2023-10-30T12:11:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:06.686 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: _finish_auth 0 2023-10-30T12:11:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 1 -- 172.21.15.50:0/4067743866 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c4011960 con 0x7fd3d0132db0 2023-10-30T12:11:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 5 monclient: authenticate success, global_id 114533 2023-10-30T12:11:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 10 monclient: _renew_subs 2023-10-30T12:11:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d00a18d0 con 0x7fd3d0132db0 2023-10-30T12:11:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 1 -- 172.21.15.50:0/4067743866 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4011cf0 con 0x7fd3d0132db0 2023-10-30T12:11:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3ccd99640 4 set_mon_vals no callback set 2023-10-30T12:11:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 10 client.114533.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3beffd640 10 monclient: dump: 2023-10-30T12:11:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:06.691 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 10 monclient: _renew_subs 2023-10-30T12:11:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d009e5a0 con 0x7fd3d0132db0 2023-10-30T12:11:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.648+0000 7fd3d4b55640 1 librados: init done 2023-10-30T12:11:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3beffd640 1 -- 172.21.15.50:0/4067743866 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd3c4018360 con 0x7fd3d0132db0 2023-10-30T12:11:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3beffd640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3b0039320 0x7fd3b003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3beffd640 1 -- 172.21.15.50:0/4067743866 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(346..346 src has 1..346) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fd3c404ac30 con 0x7fd3d0132db0 2023-10-30T12:11:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3beffd640 10 client.114533.objecter ms_dispatch 0x7fd3d0000b90 osd_map(346..346 src has 1..346) v4 2023-10-30T12:11:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3beffd640 3 client.114533.objecter handle_osd_map got epochs [346,346] > 0 2023-10-30T12:11:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3beffd640 3 client.114533.objecter handle_osd_map decoding full epoch 346 2023-10-30T12:11:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3d4b55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3d4b55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3d4b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d004e160 con 0x7fd3d0132db0 2023-10-30T12:11:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3beffd640 20 client.114533.objecter dump_active .. 0 homeless 2023-10-30T12:11:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3ce59c640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3b0039320 0x7fd3b003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.649+0000 7fd3ce59c640 10 monclient: get_auth_request con 0x7fd3b0039320 auth_method 0 2023-10-30T12:11:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.650+0000 7fd3beffd640 10 client.114533.objecter ms_handle_connect 0x7fd3b0039320 2023-10-30T12:11:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.650+0000 7fd3ce59c640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3b0039320 0x7fd3b003b7e0 secure :-1 s=READY pgs=4105 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8005020 tx=0x7fd3c8007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:06.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.654+0000 7fd3beffd640 1 -- 172.21.15.50:0/4067743866 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd3c4018610 con 0x7fd3d0132db0 2023-10-30T12:11:06.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.654+0000 7fd3beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:06.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.654+0000 7fd3beffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:06.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.897+0000 7fd3d4b55640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:06.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.897+0000 7fd3d4b55640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:06.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.897+0000 7fd3d4b55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:06.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.897+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd3d009cb90 con 0x7fd3d0132db0 2023-10-30T12:11:06.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.897+0000 7fd3beffd640 1 -- 172.21.15.50:0/4067743866 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fd3c403c070 con 0x7fd3d0132db0 2023-10-30T12:11:06.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.897+0000 7fd3beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:06.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.897+0000 7fd3beffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:06.900 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:06.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 10 librados: watch_flush enter 2023-10-30T12:11:06.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 10 librados: watch_flush exit 2023-10-30T12:11:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 20 client.114533.objecter shutdown clearing up homeless session... 2023-10-30T12:11:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 10 client.114533.objecter successfully canceled tick 2023-10-30T12:11:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3b0039320 msgr2=0x7fd3b003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3b0039320 0x7fd3b003b7e0 secure :-1 s=READY pgs=4105 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8005020 tx=0x7fd3c8007ef0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 10 monclient: shutdown 2023-10-30T12:11:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3d0132db0 msgr2=0x7fd3d009b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3d0132db0 0x7fd3d009b420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c5b0 tx=0x7fd3c400ca70 comp rx=0 tx=0).stop 2023-10-30T12:11:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 shutdown_connections 2023-10-30T12:11:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3b0039320 0x7fd3b003b7e0 unknown :-1 s=CLOSED pgs=4105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3d009b960 0x7fd3d01ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3d0132db0 0x7fd3d009b420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 --2- 172.21.15.50:0/4067743866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3d00a4b60 0x7fd3d009aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 >> 172.21.15.50:0/4067743866 conn(0x7fd3d00a8060 msgr2=0x7fd3d0136850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 shutdown_connections 2023-10-30T12:11:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 -- 172.21.15.50:0/4067743866 wait complete. 2023-10-30T12:11:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:06.899+0000 7fd3d4b55640 1 librados: shutdown 2023-10-30T12:11:06.911 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:11:06.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:11:06.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:11:07.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.017+0000 7ff76aaf2640 10 monclient: get_monmap_and_config 2023-10-30T12:11:07.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.017+0000 7ff76aaf2640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.017+0000 7ff76aaf2640 10 monclient: monmap: 2023-10-30T12:11:07.034 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.034 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.034 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:07.018168+0000 2023-10-30T12:11:07.035 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:07.018168+0000 2023-10-30T12:11:07.035 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.035 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.036 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:07.036 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:07.036 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:07.036 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:07.036 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:07.037 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.017+0000 7ff76aaf2640 1 Processor -- start 2023-10-30T12:11:07.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.017+0000 7ff76aaf2640 1 -- start start 2023-10-30T12:11:07.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.017+0000 7ff76aaf2640 10 monclient: init 2023-10-30T12:11:07.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-10-30T12:11:07.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7640a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:07.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff7640a2450 0x7ff76413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient(hunting): picked mon.noname-a con 0x7ff7640a2450 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:07.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff76413d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient(hunting): picked mon.noname-b con 0x7ff76413af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7640a4190 con 0x7ff7640a2ae0 2023-10-30T12:11:07.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7640a1630 con 0x7ff7640a2450 2023-10-30T12:11:07.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76409a2e0 con 0x7ff76413af40 2023-10-30T12:11:07.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76aaf2640 10 monclient(hunting): authenticate will time out at 1512.201357s 2023-10-30T12:11:07.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff7640a2450 0x7ff76413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff76413d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76888e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff7640a2ae0 0x7ff7640a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37680/0 (socket says 172.21.15.50:37680) 2023-10-30T12:11:07.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76888e640 1 -- 172.21.15.50:0/3211996558 learned_addr learned my addr 172.21.15.50:0/3211996558 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:11:07.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): get_auth_request con 0x7ff76413af40 auth_method 0 2023-10-30T12:11:07.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff7640a2450 auth_method 0 2023-10-30T12:11:07.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient(hunting): handle_auth_done global_id 114536 payload 274 2023-10-30T12:11:07.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 1 -- 172.21.15.50:0/3211996558 >> v2:172.21.15.50:3300/0 conn(0x7ff7640a2450 msgr2=0x7ff76413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:07.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 1 --2- 172.21.15.50:0/3211996558 >> v2:172.21.15.50:3300/0 conn(0x7ff7640a2450 0x7ff76413aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 1 -- 172.21.15.50:0/3211996558 >> v1:172.21.15.50:6789/0 conn(0x7ff7640a2ae0 legacy=0x7ff7640a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 1 monclient: found mon.noname-b 2023-10-30T12:11:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff76908f640 1 -- 172.21.15.50:0/3211996558 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76409a8a0 con 0x7ff76413af40 2023-10-30T12:11:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff7637fe640 1 -- 172.21.15.50:0/3211996558 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717247044 0 0) 0x7ff74c001110 con 0x7ff7640a2ae0 2023-10-30T12:11:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff7637fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:11:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.018+0000 7ff763fff640 1 --2- 172.21.15.50:0/3211996558 >> v2:172.21.15.50:3300/0 conn(0x7ff7640a2450 0x7ff76413aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:11:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff76908f640 1 --2- 172.21.15.50:0/3211996558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff76413d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009d90 tx=0x7ff75c02bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ec0af8a1daa72b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 1 -- 172.21.15.50:0/3211996558 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c02cb30 con 0x7ff76413af40 2023-10-30T12:11:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:07.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: dump: 2023-10-30T12:11:07.058 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.061 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 1 -- 172.21.15.50:0/3211996558 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c02cc90 con 0x7ff76413af40 2023-10-30T12:11:07.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff76aaf2640 5 monclient: authenticate success, global_id 114536 2023-10-30T12:11:07.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff76aaf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:07.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff76aaf2640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:07.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff76aaf2640 4 set_mon_vals no callback set 2023-10-30T12:11:07.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 1 -- 172.21.15.50:0/3211996558 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c034440 con 0x7ff76413af40 2023-10-30T12:11:07.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.019+0000 7ff7637fe640 10 monclient: dump: 2023-10-30T12:11:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.066 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 10 monclient: shutdown 2023-10-30T12:11:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 -- 172.21.15.50:0/3211996558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 msgr2=0x7ff76413d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 --2- 172.21.15.50:0/3211996558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff76413d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009d90 tx=0x7ff75c02bd80 comp rx=0 tx=0).stop 2023-10-30T12:11:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 -- 172.21.15.50:0/3211996558 shutdown_connections 2023-10-30T12:11:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 --2- 172.21.15.50:0/3211996558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff76413d330 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 --2- 172.21.15.50:0/3211996558 >> v2:172.21.15.50:3300/0 conn(0x7ff7640a2450 0x7ff76413aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 -- 172.21.15.50:0/3211996558 >> 172.21.15.50:0/3211996558 conn(0x7ff7640a80b0 msgr2=0x7ff7640a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 -- 172.21.15.50:0/3211996558 shutdown_connections 2023-10-30T12:11:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 1 -- 172.21.15.50:0/3211996558 wait complete. 2023-10-30T12:11:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.020+0000 7ff76aaf2640 10 monclient: monmap: 2023-10-30T12:11:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:07.071 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 librados: starting msgr at 2023-10-30T12:11:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 librados: starting objecter 2023-10-30T12:11:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 Processor -- start 2023-10-30T12:11:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 -- start start 2023-10-30T12:11:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 librados: setting wanted keys 2023-10-30T12:11:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 librados: calling monclient init 2023-10-30T12:11:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient: init 2023-10-30T12:11:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:11:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7640a2450 0x7ff764134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): picked mon.noname-b con 0x7ff7640a2450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7640a2ae0 0x7ff764135300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): picked mon.noname-a con 0x7ff7640a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff764135840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): picked mon.noname-c con 0x7ff76413af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff764138bd0 con 0x7ff7640a2ae0 2023-10-30T12:11:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff764138510 con 0x7ff7640a2450 2023-10-30T12:11:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76413d540 con 0x7ff76413af40 2023-10-30T12:11:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76aaf2640 10 monclient(hunting): authenticate will time out at 1512.204803s 2023-10-30T12:11:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76908f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff764135840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff763fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7640a2ae0 0x7ff764135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76908f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff764135840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44382/0 (socket says 172.21.15.50:44382) 2023-10-30T12:11:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76908f640 1 -- 172.21.15.50:0/2262772816 learned_addr learned my addr 172.21.15.50:0/2262772816 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff763fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7640a2ae0 0x7ff764135300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53062/0 (socket says 172.21.15.50:53062) 2023-10-30T12:11:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76908f640 10 monclient(hunting): get_auth_request con 0x7ff76413af40 auth_method 0 2023-10-30T12:11:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76908f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76908f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff76908f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff7640a2ae0 auth_method 0 2023-10-30T12:11:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.021+0000 7ff763fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76908f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76908f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76908f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76888e640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7640a2450 0x7ff764134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76888e640 10 monclient(hunting): get_auth_request con 0x7ff7640a2450 auth_method 0 2023-10-30T12:11:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76888e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76888e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76888e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 10 monclient(hunting): handle_auth_done global_id 134358 payload 824 2023-10-30T12:11:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 msgr2=0x7ff764135840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff764135840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7640a2450 msgr2=0x7ff764134dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7640a2450 0x7ff764134dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 1 monclient: found mon.noname-a 2023-10-30T12:11:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 1 -- 172.21.15.50:0/2262772816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff754009400 con 0x7ff7640a2ae0 2023-10-30T12:11:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76908f640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff76413af40 0x7ff764135840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:11:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff763fff640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7640a2ae0 0x7ff764135300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff75400e900 tx=0x7ff75400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 client.?.objecter ms_handle_connect 0x7ff7640a2ae0 2023-10-30T12:11:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75400ca00 con 0x7ff7640a2ae0 2023-10-30T12:11:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: dump: 2023-10-30T12:11:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.085 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75400cb60 con 0x7ff7640a2ae0 2023-10-30T12:11:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754013470 con 0x7ff7640a2ae0 2023-10-30T12:11:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff7617fa640 10 monclient: dump: 2023-10-30T12:11:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.089 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 5 monclient: authenticate success, global_id 134358 2023-10-30T12:11:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 10 monclient: _renew_subs 2023-10-30T12:11:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff762ffd640 4 set_mon_vals no callback set 2023-10-30T12:11:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c009a70 con 0x7ff7640a2ae0 2023-10-30T12:11:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 10 client.134358.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 10 monclient: _renew_subs 2023-10-30T12:11:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff764099bc0 con 0x7ff7640a2ae0 2023-10-30T12:11:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.022+0000 7ff76aaf2640 1 librados: init done 2023-10-30T12:11:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76aaf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76aaf2640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764139540 0x7ff7641321e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76aaf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff764073ff0 con 0x7ff764139540 2023-10-30T12:11:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff7540135d0 con 0x7ff7640a2ae0 2023-10-30T12:11:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764139540 0x7ff7641321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient: get_auth_request con 0x7ff764139540 auth_method 0 2023-10-30T12:11:07.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff7617fa640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7440392f0 0x7ff74403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(346..346 src has 1..346) v4 ==== 6299+0+0 (secure 0 0 0) 0x7ff75404ae70 con 0x7ff7640a2ae0 2023-10-30T12:11:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff7617fa640 10 client.134358.objecter ms_dispatch 0x7ff764000b90 osd_map(346..346 src has 1..346) v4 2023-10-30T12:11:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff7617fa640 3 client.134358.objecter handle_osd_map got epochs [346,346] > 0 2023-10-30T12:11:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff7617fa640 3 client.134358.objecter handle_osd_map decoding full epoch 346 2023-10-30T12:11:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 10 monclient(hunting): handle_auth_done global_id 134361 payload 824 2023-10-30T12:11:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff76888e640 1 -- 172.21.15.50:0/2262772816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff76409a5c0 con 0x7ff764139540 2023-10-30T12:11:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.023+0000 7ff7617fa640 20 client.134358.objecter dump_active .. 0 homeless 2023-10-30T12:11:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff76908f640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7440392f0 0x7ff74403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff76888e640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764139540 0x7ff7641321e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff74c005ed0 tx=0x7ff74c001810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 client.134358.objecter ms_handle_connect 0x7ff764139540 2023-10-30T12:11:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 client.134358.objecter resend_mon_ops 2023-10-30T12:11:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff76908f640 10 monclient: get_auth_request con 0x7ff7440392f0 auth_method 0 2023-10-30T12:11:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c012590 con 0x7ff764139540 2023-10-30T12:11:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 monclient: dump: 2023-10-30T12:11:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.103 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ff74c0126f0 con 0x7ff764139540 2023-10-30T12:11:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764139540 msgr2=0x7ff7641321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764139540 0x7ff7641321e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff74c005ed0 tx=0x7ff74c001810 comp rx=0 tx=0).stop 2023-10-30T12:11:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff76908f640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7440392f0 0x7ff74403b7b0 secure :-1 s=READY pgs=4106 cs=0 l=1 rev1=1 crypto rx=0x7ff75c02bdc0 tx=0x7ff75c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764139540 msgr2=0x7ff7641321e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764139540 0x7ff7641321e0 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff74c005ed0 tx=0x7ff74c001810 comp rx=0 tx=0).stop 2023-10-30T12:11:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.024+0000 7ff7617fa640 10 client.134358.objecter ms_handle_connect 0x7ff7440392f0 2023-10-30T12:11:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76aaf2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76aaf2640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764132720 0x7ff764132b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76aaf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7640a28a0 con 0x7ff764132720 2023-10-30T12:11:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764132720 0x7ff764132b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 10 monclient: get_auth_request con 0x7ff764132720 auth_method 0 2023-10-30T12:11:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.030+0000 7ff76888e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76888e640 10 monclient(hunting): handle_auth_done global_id 134364 payload 824 2023-10-30T12:11:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76888e640 1 -- 172.21.15.50:0/2262772816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff76406b060 con 0x7ff764132720 2023-10-30T12:11:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76888e640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764132720 0x7ff764132b30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff74c005ed0 tx=0x7ff74c001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 10 client.134358.objecter ms_handle_connect 0x7ff764132720 2023-10-30T12:11:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 10 client.134358.objecter resend_mon_ops 2023-10-30T12:11:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c01ebd0 con 0x7ff764132720 2023-10-30T12:11:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 10 monclient: dump: 2023-10-30T12:11:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.111 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7ff74c01ed30 con 0x7ff764132720 2023-10-30T12:11:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764132720 msgr2=0x7ff764132b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764132720 0x7ff764132b30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff74c005ed0 tx=0x7ff74c001080 comp rx=0 tx=0).stop 2023-10-30T12:11:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764132720 msgr2=0x7ff764132b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff7617fa640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff764132720 0x7ff764132b30 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff74c005ed0 tx=0x7ff74c001080 comp rx=0 tx=0).stop 2023-10-30T12:11:07.113 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:07.114 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:07.114 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:07.114 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:11:07.115 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 100, 2023-10-30T12:11:07.115 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:07.116 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:11:07.116 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:07.116 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:11:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:11:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:11:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:11:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:11:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:07.159 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:07.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 10 librados: watch_flush enter 2023-10-30T12:11:07.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 10 librados: watch_flush exit 2023-10-30T12:11:07.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 20 client.134358.objecter shutdown clearing up homeless session... 2023-10-30T12:11:07.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 10 client.134358.objecter successfully canceled tick 2023-10-30T12:11:07.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7440392f0 msgr2=0x7ff74403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff7440392f0 0x7ff74403b7b0 secure :-1 s=READY pgs=4106 cs=0 l=1 rev1=1 crypto rx=0x7ff75c02bdc0 tx=0x7ff75c030000 comp rx=0 tx=0).stop 2023-10-30T12:11:07.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 10 monclient: shutdown 2023-10-30T12:11:07.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:07.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7640a2ae0 msgr2=0x7ff764135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 1 --2- 172.21.15.50:0/2262772816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7640a2ae0 0x7ff764135300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff75400e900 tx=0x7ff75400edc0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76908f640 1 -- 172.21.15.50:0/2262772816 reap_dead start 2023-10-30T12:11:07.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76908f640 1 -- 172.21.15.50:0/2262772816 reap_dead start 2023-10-30T12:11:07.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 shutdown_connections 2023-10-30T12:11:07.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.031+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 >> 172.21.15.50:0/2262772816 conn(0x7ff7640a80b0 msgr2=0x7ff7640a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:07.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.032+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 shutdown_connections 2023-10-30T12:11:07.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.032+0000 7ff76aaf2640 1 -- 172.21.15.50:0/2262772816 wait complete. 2023-10-30T12:11:07.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.032+0000 7ff76aaf2640 1 librados: shutdown 2023-10-30T12:11:07.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:11:07.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:11:07.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:11:07.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.270+0000 7f5102a90640 10 monclient: get_monmap_and_config 2023-10-30T12:11:07.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.270+0000 7f5102a90640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.270+0000 7f5102a90640 10 monclient: monmap: 2023-10-30T12:11:07.272 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.272 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.272 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:07.271128+0000 2023-10-30T12:11:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:07.271128+0000 2023-10-30T12:11:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:07.274 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.270+0000 7f5102a90640 1 Processor -- start 2023-10-30T12:11:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.270+0000 7f5102a90640 1 -- start start 2023-10-30T12:11:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.270+0000 7f5102a90640 10 monclient: init 2023-10-30T12:11:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-10-30T12:11:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50fc0a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f50fc0a2450 0x7f50fc13aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient(hunting): picked mon.noname-c con 0x7f50fc0a2450 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc13d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient(hunting): picked mon.noname-b con 0x7f50fc13af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc0a4190 con 0x7f50fc0a2ae0 2023-10-30T12:11:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50fc0a1630 con 0x7f50fc0a2450 2023-10-30T12:11:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc09a2e0 con 0x7f50fc13af40 2023-10-30T12:11:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f5102a90640 10 monclient(hunting): authenticate will time out at 1512.454283s 2023-10-30T12:11:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc13d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc13d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44396/0 (socket says 172.21.15.50:44396) 2023-10-30T12:11:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 -- 172.21.15.50:0/3404388537 learned_addr learned my addr 172.21.15.50:0/3404388537 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): get_auth_request con 0x7f50fc13af40 auth_method 0 2023-10-30T12:11:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f50fbfff640 1 --2- 172.21.15.50:0/3404388537 >> v2:172.21.15.118:3300/0 conn(0x7f50fc0a2450 0x7f50fc13aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f50fbfff640 10 monclient(hunting): get_auth_request con 0x7f50fc0a2450 auth_method 0 2023-10-30T12:11:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f50fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f50fbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f50fbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient(hunting): handle_auth_done global_id 114542 payload 274 2023-10-30T12:11:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 -- 172.21.15.50:0/3404388537 >> v2:172.21.15.118:3300/0 conn(0x7f50fc0a2450 msgr2=0x7f50fc13aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 --2- 172.21.15.50:0/3404388537 >> v2:172.21.15.118:3300/0 conn(0x7f50fc0a2450 0x7f50fc13aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 -- 172.21.15.50:0/3404388537 >> v1:172.21.15.118:6789/0 conn(0x7f50fc0a2ae0 legacy=0x7f50fc0a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 monclient: found mon.noname-b 2023-10-30T12:11:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.271+0000 7f510102d640 1 -- 172.21.15.50:0/3404388537 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc09a8a0 con 0x7f50fc13af40 2023-10-30T12:11:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f510102d640 1 --2- 172.21.15.50:0/3404388537 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc13d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009870 tx=0x7f50ec02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49d06bc1fd64be64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 1 -- 172.21.15.50:0/3404388537 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec031740 con 0x7f50fc13af40 2023-10-30T12:11:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: dump: 2023-10-30T12:11:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.297 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 1 -- 172.21.15.50:0/3404388537 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ec0318a0 con 0x7f50fc13af40 2023-10-30T12:11:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f5102a90640 5 monclient: authenticate success, global_id 114542 2023-10-30T12:11:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f5102a90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f5102a90640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 1 -- 172.21.15.50:0/3404388537 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec031c30 con 0x7f50fc13af40 2023-10-30T12:11:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f5102a90640 4 set_mon_vals no callback set 2023-10-30T12:11:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.272+0000 7f50fb7fe640 10 monclient: dump: 2023-10-30T12:11:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.302 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 10 monclient: shutdown 2023-10-30T12:11:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 -- 172.21.15.50:0/3404388537 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 msgr2=0x7f50fc13d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 --2- 172.21.15.50:0/3404388537 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc13d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009870 tx=0x7f50ec02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 -- 172.21.15.50:0/3404388537 shutdown_connections 2023-10-30T12:11:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 --2- 172.21.15.50:0/3404388537 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc13d330 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 --2- 172.21.15.50:0/3404388537 >> v2:172.21.15.118:3300/0 conn(0x7f50fc0a2450 0x7f50fc13aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 -- 172.21.15.50:0/3404388537 >> 172.21.15.50:0/3404388537 conn(0x7f50fc0a80b0 msgr2=0x7f50fc0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 -- 172.21.15.50:0/3404388537 shutdown_connections 2023-10-30T12:11:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 -- 172.21.15.50:0/3404388537 wait complete. 2023-10-30T12:11:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 10 monclient: monmap: 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:07.306 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.273+0000 7f5102a90640 1 librados: starting msgr at 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 librados: starting objecter 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 Processor -- start 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 -- start start 2023-10-30T12:11:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 librados: setting wanted keys 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 librados: calling monclient init 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient: init 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50fc0a2450 0x7f50fc1283b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): picked mon.noname-a con 0x7f50fc0a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50fc0a2ae0 0x7f50fc1288f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): picked mon.noname-b con 0x7f50fc0a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc130e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): picked mon.noname-c con 0x7f50fc13af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc0a0720 con 0x7f50fc0a2450 2023-10-30T12:11:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc09c490 con 0x7f50fc0a2ae0 2023-10-30T12:11:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc09bb30 con 0x7f50fc13af40 2023-10-30T12:11:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f5102a90640 10 monclient(hunting): authenticate will time out at 1512.457833s 2023-10-30T12:11:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510082c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50fc0a2450 0x7f50fc1283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510102d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc130e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510082c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50fc0a2450 0x7f50fc1283b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53080/0 (socket says 172.21.15.50:53080) 2023-10-30T12:11:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510082c640 1 -- 172.21.15.50:0/2335872820 learned_addr learned my addr 172.21.15.50:0/2335872820 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510102d640 10 monclient(hunting): get_auth_request con 0x7f50fc13af40 auth_method 0 2023-10-30T12:11:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510102d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510102d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510102d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510082c640 10 monclient(hunting): get_auth_request con 0x7f50fc0a2450 auth_method 0 2023-10-30T12:11:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510082c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510082c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.274+0000 7f510082c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50fc0a2ae0 0x7f50fc1288f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 10 monclient(hunting): get_auth_request con 0x7f50fc0a2ae0 auth_method 0 2023-10-30T12:11:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 10 monclient(hunting): handle_auth_done global_id 114545 payload 824 2023-10-30T12:11:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50fc0a2ae0 msgr2=0x7f50fc1288f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50fc0a2ae0 0x7f50fc1288f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50fc0a2450 msgr2=0x7f50fc1283b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50fc0a2450 0x7f50fc1283b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 1 monclient: found mon.noname-c 2023-10-30T12:11:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 1 -- 172.21.15.50:0/2335872820 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0094d0 con 0x7f50fc13af40 2023-10-30T12:11:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510082c640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50fc0a2450 0x7f50fc1283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:11:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f510102d640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc130e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f50ec033000 tx=0x7f50ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 client.?.objecter ms_handle_connect 0x7f50fc13af40 2023-10-30T12:11:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec031f10 con 0x7f50fc13af40 2023-10-30T12:11:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: dump: 2023-10-30T12:11:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.323 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ec033c70 con 0x7f50fc13af40 2023-10-30T12:11:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 5 monclient: authenticate success, global_id 114545 2023-10-30T12:11:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 10 monclient: _renew_subs 2023-10-30T12:11:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec03c400 con 0x7f50fc13af40 2023-10-30T12:11:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50fc099bc0 con 0x7f50fc13af40 2023-10-30T12:11:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50faffd640 4 set_mon_vals no callback set 2023-10-30T12:11:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f50f97fa640 10 monclient: dump: 2023-10-30T12:11:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.328 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 10 client.114545.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 10 monclient: _renew_subs 2023-10-30T12:11:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50fc09c0b0 con 0x7f50fc13af40 2023-10-30T12:11:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.275+0000 7f5102a90640 1 librados: init done 2023-10-30T12:11:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f50ec03a040 con 0x7f50fc13af40 2023-10-30T12:11:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f5102a90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f5102a90640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc132730 0x7f50fc131350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f5102a90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc073ff0 con 0x7f50fc132730 2023-10-30T12:11:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc132730 0x7f50fc131350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50f97fa640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f50e80392f0 0x7f50e803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient: get_auth_request con 0x7f50fc132730 auth_method 0 2023-10-30T12:11:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(346..346 src has 1..346) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f50ec06c570 con 0x7f50fc13af40 2023-10-30T12:11:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50f97fa640 10 client.114545.objecter ms_dispatch 0x7f50fc000b90 osd_map(346..346 src has 1..346) v4 2023-10-30T12:11:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50f97fa640 3 client.114545.objecter handle_osd_map got epochs [346,346] > 0 2023-10-30T12:11:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50f97fa640 3 client.114545.objecter handle_osd_map decoding full epoch 346 2023-10-30T12:11:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50f97fa640 20 client.114545.objecter dump_active .. 0 homeless 2023-10-30T12:11:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 10 monclient(hunting): handle_auth_done global_id 114548 payload 824 2023-10-30T12:11:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50fbfff640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f50e80392f0 0x7f50e803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f510082c640 1 -- 172.21.15.50:0/2335872820 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50fc09a5c0 con 0x7f50fc132730 2023-10-30T12:11:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.276+0000 7f50fbfff640 10 monclient: get_auth_request con 0x7f50e80392f0 auth_method 0 2023-10-30T12:11:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 client.114545.objecter ms_handle_connect 0x7f50fc132730 2023-10-30T12:11:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 client.114545.objecter resend_mon_ops 2023-10-30T12:11:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f510082c640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc132730 0x7f50fc131350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f50f400fe10 tx=0x7f50f400d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4004980 con 0x7f50fc132730 2023-10-30T12:11:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 monclient: dump: 2023-10-30T12:11:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.338 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 client.114545.objecter ms_handle_connect 0x7f50e80392f0 2023-10-30T12:11:07.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50fbfff640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f50e80392f0 0x7f50e803b7b0 secure :-1 s=READY pgs=4107 cs=0 l=1 rev1=1 crypto rx=0x7f50fc1324a0 tx=0x7f50f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f50f4004ae0 con 0x7f50fc132730 2023-10-30T12:11:07.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:07.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc132730 msgr2=0x7f50fc131350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc132730 0x7f50fc131350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f50f400fe10 tx=0x7f50f400d520 comp rx=0 tx=0).stop 2023-10-30T12:11:07.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc132730 msgr2=0x7f50fc131350 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:07.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.277+0000 7f50f97fa640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc132730 0x7f50fc131350 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f50f400fe10 tx=0x7f50f400d520 comp rx=0 tx=0).stop 2023-10-30T12:11:07.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f5102a90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:07.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f5102a90640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc131890 0x7f50fc131ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f5102a90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc073a20 con 0x7f50fc131890 2023-10-30T12:11:07.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f510082c640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc131890 0x7f50fc131ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f510082c640 10 monclient: get_auth_request con 0x7f50fc131890 auth_method 0 2023-10-30T12:11:07.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f510082c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f510082c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.283+0000 7f510082c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f510082c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f510082c640 10 monclient(hunting): handle_auth_done global_id 114551 payload 824 2023-10-30T12:11:07.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f510082c640 1 -- 172.21.15.50:0/2335872820 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f50fc0a28a0 con 0x7f50fc131890 2023-10-30T12:11:07.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 10 client.114545.objecter ms_handle_connect 0x7f50fc131890 2023-10-30T12:11:07.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 10 client.114545.objecter resend_mon_ops 2023-10-30T12:11:07.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f510082c640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc131890 0x7f50fc131ca0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f50f400fe10 tx=0x7f50f4003ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4024070 con 0x7f50fc131890 2023-10-30T12:11:07.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 10 monclient: dump: 2023-10-30T12:11:07.346 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.346 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.346 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.346 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.346 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.347 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.347 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.347 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.347 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.347 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f50f401f7f0 con 0x7f50fc131890 2023-10-30T12:11:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc131890 msgr2=0x7f50fc131ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc131890 0x7f50fc131ca0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f50f400fe10 tx=0x7f50f4003ae0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc131890 msgr2=0x7f50fc131ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:07.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f50f97fa640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc131890 0x7f50fc131ca0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f50f400fe10 tx=0x7f50f4003ae0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.349 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:07.349 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:11:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 100, 2023-10-30T12:11:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:11:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:11:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:07.368 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.369 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:07.369 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:07.370 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.371 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.371 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.371 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:07.374 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.374 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.375 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:07.375 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:07.375 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.375 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.376 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.376 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:07.376 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:07.376 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.377 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.377 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.377 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:07.377 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.378 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.378 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:07.378 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.379 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.379 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:07.379 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:07.380 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.380 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.380 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.381 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.381 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:07.381 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:07.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.382 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.382 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.382 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:07.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.383 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.383 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:11:07.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.383 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.383 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.384 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:11:07.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.384 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.384 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:07.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.385 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.385 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.386 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:11:07.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.386 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.387 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:07.387 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:07.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.284+0000 7f5102a90640 10 librados: watch_flush enter 2023-10-30T12:11:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 10 librados: watch_flush exit 2023-10-30T12:11:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 20 client.114545.objecter shutdown clearing up homeless session... 2023-10-30T12:11:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 10 client.114545.objecter successfully canceled tick 2023-10-30T12:11:07.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f50e80392f0 msgr2=0x7f50e803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f50e80392f0 0x7f50e803b7b0 secure :-1 s=READY pgs=4107 cs=0 l=1 rev1=1 crypto rx=0x7f50fc1324a0 tx=0x7f50f0008040 comp rx=0 tx=0).stop 2023-10-30T12:11:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 10 monclient: shutdown 2023-10-30T12:11:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 msgr2=0x7f50fc130e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc130e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f50ec033000 tx=0x7f50ec0046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f510102d640 1 -- 172.21.15.50:0/2335872820 reap_dead start 2023-10-30T12:11:07.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 shutdown_connections 2023-10-30T12:11:07.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 --2- 172.21.15.50:0/2335872820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f50fc13af40 0x7f50fc130e10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 >> 172.21.15.50:0/2335872820 conn(0x7f50fc0a80b0 msgr2=0x7f50fc13a760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:07.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 shutdown_connections 2023-10-30T12:11:07.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 -- 172.21.15.50:0/2335872820 wait complete. 2023-10-30T12:11:07.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.285+0000 7f5102a90640 1 librados: shutdown 2023-10-30T12:11:07.396 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:11:07.397 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:11:07.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:11:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.502+0000 7fb4bf232640 10 monclient: get_monmap_and_config 2023-10-30T12:11:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.502+0000 7fb4bf232640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.502+0000 7fb4bf232640 10 monclient: monmap: 2023-10-30T12:11:07.504 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.504 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.504 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:07.503230+0000 2023-10-30T12:11:07.504 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:07.503230+0000 2023-10-30T12:11:07.505 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.505 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.505 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:07.505 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:07.505 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:07.506 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:07.506 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:07.506 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.502+0000 7fb4bf232640 1 Processor -- start 2023-10-30T12:11:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.502+0000 7fb4bf232640 1 -- start start 2023-10-30T12:11:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.502+0000 7fb4bf232640 10 monclient: init 2023-10-30T12:11:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T12:11:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8132970 0x7fb4b8132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b8132970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb4b80a03d0 0x7fb4b80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b80a03d0 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:07.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4b8070d10 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:07.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b80a17d0 con 0x7fb4b8070d10 2023-10-30T12:11:07.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b80a1510 con 0x7fb4b80a03d0 2023-10-30T12:11:07.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b809f680 con 0x7fb4b8132970 2023-10-30T12:11:07.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bf232640 10 monclient(hunting): authenticate will time out at 1512.686402s 2023-10-30T12:11:07.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8132970 0x7fb4b8132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8132970 0x7fb4b8132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44424/0 (socket says 172.21.15.50:44424) 2023-10-30T12:11:07.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 -- 172.21.15.50:0/2323765372 learned_addr learned my addr 172.21.15.50:0/2323765372 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:07.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): get_auth_request con 0x7fb4b8132970 auth_method 0 2023-10-30T12:11:07.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/2323765372 >> v2:172.21.15.118:3300/0 conn(0x7fb4b80a03d0 0x7fb4b80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request con 0x7fb4b80a03d0 auth_method 0 2023-10-30T12:11:07.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4b7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4b7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_done global_id 114554 payload 274 2023-10-30T12:11:07.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 -- 172.21.15.50:0/2323765372 >> v2:172.21.15.118:3300/0 conn(0x7fb4b80a03d0 msgr2=0x7fb4b80707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:07.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 --2- 172.21.15.50:0/2323765372 >> v2:172.21.15.118:3300/0 conn(0x7fb4b80a03d0 0x7fb4b80707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 -- 172.21.15.50:0/2323765372 >> v1:172.21.15.118:6789/0 conn(0x7fb4b8070d10 legacy=0x7fb4b813caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:07.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 monclient: found mon.noname-b 2023-10-30T12:11:07.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.503+0000 7fb4bcfce640 1 -- 172.21.15.50:0/2323765372 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b809f960 con 0x7fb4b8132970 2023-10-30T12:11:07.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4bcfce640 1 --2- 172.21.15.50:0/2323765372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8132970 0x7fb4b8132d50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009870 tx=0x7fb4ac02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea784c6756052c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 1 -- 172.21.15.50:0/2323765372 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac031740 con 0x7fb4b8132970 2023-10-30T12:11:07.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:07.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: dump: 2023-10-30T12:11:07.525 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.525 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.525 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.525 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.526 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.526 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.526 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.526 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.526 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.527 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 1 -- 172.21.15.50:0/2323765372 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac0318a0 con 0x7fb4b8132970 2023-10-30T12:11:07.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4bf232640 5 monclient: authenticate success, global_id 114554 2023-10-30T12:11:07.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4bf232640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:07.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4bf232640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:07.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 1 -- 172.21.15.50:0/2323765372 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac031c30 con 0x7fb4b8132970 2023-10-30T12:11:07.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4bf232640 4 set_mon_vals no callback set 2023-10-30T12:11:07.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.504+0000 7fb4b77fe640 10 monclient: dump: 2023-10-30T12:11:07.529 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.529 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.530 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.530 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.530 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.530 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.530 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.531 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.531 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.531 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 10 monclient: shutdown 2023-10-30T12:11:07.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:07.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 -- 172.21.15.50:0/2323765372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8132970 msgr2=0x7fb4b8132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 --2- 172.21.15.50:0/2323765372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8132970 0x7fb4b8132d50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009870 tx=0x7fb4ac02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 -- 172.21.15.50:0/2323765372 shutdown_connections 2023-10-30T12:11:07.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 --2- 172.21.15.50:0/2323765372 >> v2:172.21.15.118:3300/0 conn(0x7fb4b80a03d0 0x7fb4b80707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 --2- 172.21.15.50:0/2323765372 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8132970 0x7fb4b8132d50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 -- 172.21.15.50:0/2323765372 >> 172.21.15.50:0/2323765372 conn(0x7fb4b80a4800 msgr2=0x7fb4b80a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:07.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 -- 172.21.15.50:0/2323765372 shutdown_connections 2023-10-30T12:11:07.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 1 -- 172.21.15.50:0/2323765372 wait complete. 2023-10-30T12:11:07.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.505+0000 7fb4bf232640 10 monclient: monmap: 2023-10-30T12:11:07.534 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.534 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:07.534 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:07.534 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:07.534 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.535 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.535 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:07.535 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:07.535 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:07.535 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 librados: starting msgr at 2023-10-30T12:11:07.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 librados: starting objecter 2023-10-30T12:11:07.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 Processor -- start 2023-10-30T12:11:07.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 -- start start 2023-10-30T12:11:07.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 librados: setting wanted keys 2023-10-30T12:11:07.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 librados: calling monclient init 2023-10-30T12:11:07.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient: init 2023-10-30T12:11:07.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:11:07.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8070d10 0x7fb4b81ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b8070d10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b8132970 0x7fb4b81cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b8132970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:07.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806c3f0 0x7fb4b806c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b806c3f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:07.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8134ae0 con 0x7fb4b8132970 2023-10-30T12:11:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b806ed40 con 0x7fb4b806c3f0 2023-10-30T12:11:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8125f80 con 0x7fb4b8070d10 2023-10-30T12:11:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bf232640 10 monclient(hunting): authenticate will time out at 1512.689865s 2023-10-30T12:11:07.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bcfce640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8070d10 0x7fb4b81ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bcfce640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8070d10 0x7fb4b81ce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44438/0 (socket says 172.21.15.50:44438) 2023-10-30T12:11:07.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bcfce640 1 -- 172.21.15.50:0/1221847875 learned_addr learned my addr 172.21.15.50:0/1221847875 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:07.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bcfce640 10 monclient(hunting): get_auth_request con 0x7fb4b8070d10 auth_method 0 2023-10-30T12:11:07.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bcfce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bcfce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4bcfce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.506+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b8132970 0x7fb4b81cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request con 0x7fb4b8132970 auth_method 0 2023-10-30T12:11:07.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806c3f0 0x7fb4b806c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 10 monclient(hunting): get_auth_request con 0x7fb4b806c3f0 auth_method 0 2023-10-30T12:11:07.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bd7cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 10 monclient(hunting): handle_auth_done global_id 114557 payload 824 2023-10-30T12:11:07.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806c3f0 msgr2=0x7fb4b806c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806c3f0 0x7fb4b806c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b8132970 msgr2=0x7fb4b81cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b8132970 0x7fb4b81cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 1 monclient: found mon.noname-c 2023-10-30T12:11:07.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 1 -- 172.21.15.50:0/1221847875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac0094d0 con 0x7fb4b8070d10 2023-10-30T12:11:07.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bcfce640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8070d10 0x7fb4b81ce8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0090b0 tx=0x7fb4ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 client.?.objecter ms_handle_connect 0x7fb4b8070d10 2023-10-30T12:11:07.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:07.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac033c70 con 0x7fb4b8070d10 2023-10-30T12:11:07.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:07.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: dump: 2023-10-30T12:11:07.548 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.548 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.549 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.549 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.549 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.549 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.549 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.549 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.550 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.550 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 5 monclient: authenticate success, global_id 114557 2023-10-30T12:11:07.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac033df0 con 0x7fb4b8070d10 2023-10-30T12:11:07.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac03c640 con 0x7fb4b8070d10 2023-10-30T12:11:07.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 10 monclient: _renew_subs 2023-10-30T12:11:07.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b809d7c0 con 0x7fb4b8070d10 2023-10-30T12:11:07.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b6ffd640 4 set_mon_vals no callback set 2023-10-30T12:11:07.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4b57fa640 10 monclient: dump: 2023-10-30T12:11:07.553 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.553 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.553 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.553 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.553 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.553 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.554 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.554 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.554 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.554 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 10 client.114557.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 10 monclient: _renew_subs 2023-10-30T12:11:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b8070460 con 0x7fb4b8070d10 2023-10-30T12:11:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.507+0000 7fb4bf232640 1 librados: init done 2023-10-30T12:11:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4bf232640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4bf232640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b80a03d0 0x7fb4b806cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4bf232640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8071a70 con 0x7fb4b80a03d0 2023-10-30T12:11:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb4ac03a040 con 0x7fb4b8070d10 2023-10-30T12:11:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b80a03d0 0x7fb4b806cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient: get_auth_request con 0x7fb4b80a03d0 auth_method 0 2023-10-30T12:11:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b57fa640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb49c039320 0x7fb49c03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(346..346 src has 1..346) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fb4ac06c830 con 0x7fb4b8070d10 2023-10-30T12:11:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b57fa640 10 client.114557.objecter ms_dispatch 0x7fb4b8000b90 osd_map(346..346 src has 1..346) v4 2023-10-30T12:11:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b57fa640 3 client.114557.objecter handle_osd_map got epochs [346,346] > 0 2023-10-30T12:11:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b57fa640 3 client.114557.objecter handle_osd_map decoding full epoch 346 2023-10-30T12:11:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b57fa640 20 client.114557.objecter dump_active .. 0 homeless 2023-10-30T12:11:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_done global_id 154114 payload 824 2023-10-30T12:11:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.508+0000 7fb4bd7cf640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb49c039320 0x7fb49c03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b7fff640 1 -- 172.21.15.50:0/1221847875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4b8133290 con 0x7fb4b80a03d0 2023-10-30T12:11:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4bd7cf640 10 monclient: get_auth_request con 0x7fb49c039320 auth_method 0 2023-10-30T12:11:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 client.114557.objecter ms_handle_connect 0x7fb4b80a03d0 2023-10-30T12:11:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 client.114557.objecter resend_mon_ops 2023-10-30T12:11:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b80a03d0 0x7fb4b806cdb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006fd0 tx=0x7fb4a4010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4010e60 con 0x7fb4b80a03d0 2023-10-30T12:11:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 monclient: dump: 2023-10-30T12:11:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.562 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.563 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 client.114557.objecter ms_handle_connect 0x7fb49c039320 2023-10-30T12:11:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4bd7cf640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb49c039320 0x7fb49c03b7e0 secure :-1 s=READY pgs=4108 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00055e0 tx=0x7fb4b0003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb4a4004240 con 0x7fb4b80a03d0 2023-10-30T12:11:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b80a03d0 msgr2=0x7fb4b806cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b80a03d0 0x7fb4b806cdb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006fd0 tx=0x7fb4a4010c10 comp rx=0 tx=0).stop 2023-10-30T12:11:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b80a03d0 msgr2=0x7fb4b806cdb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:07.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.509+0000 7fb4b57fa640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b80a03d0 0x7fb4b806cdb0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006fd0 tx=0x7fb4a4010c10 comp rx=0 tx=0).stop 2023-10-30T12:11:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.515+0000 7fb4bf232640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.515+0000 7fb4bf232640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.516+0000 7fb4bf232640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.516+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8074800 con 0x7fb4b806d2f0 2023-10-30T12:11:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.516+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.516+0000 7fb4b7fff640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 msgr2=0x7fb4b811a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:11:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.516+0000 7fb4b7fff640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 msgr2=0x7fb4b811a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:11:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.516+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.516+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:07.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 10 monclient: get_auth_request con 0x7fb4b806d2f0 auth_method 0 2023-10-30T12:11:07.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.717+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_done global_id 154117 payload 824 2023-10-30T12:11:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b7fff640 1 -- 172.21.15.50:0/1221847875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb4b804e160 con 0x7fb4b806d2f0 2023-10-30T12:11:07.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b7fff640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006fd0 tx=0x7fb4a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 10 client.114557.objecter ms_handle_connect 0x7fb4b806d2f0 2023-10-30T12:11:07.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 10 client.114557.objecter resend_mon_ops 2023-10-30T12:11:07.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4025070 con 0x7fb4b806d2f0 2023-10-30T12:11:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 10 monclient: dump: 2023-10-30T12:11:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.724 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.724 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.724 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.724 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.724 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.725 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.725 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.725 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fb4a4003a00 con 0x7fb4b806d2f0 2023-10-30T12:11:07.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:07.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:07.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 msgr2=0x7fb4b811a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006fd0 tx=0x7fb4a40046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 msgr2=0x7fb4b811a430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:07.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.718+0000 7fb4b57fa640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4b806d2f0 0x7fb4b811a430 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006fd0 tx=0x7fb4a40046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.727 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:07.727 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:07.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:07.728 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:11:07.728 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 100, 2023-10-30T12:11:07.728 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:07.728 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:11:07.728 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:07.729 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:07.729 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.729 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:11:07.729 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:07.729 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:07.729 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:07.730 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.730 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.730 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.730 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.730 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.731 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.731 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.731 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.731 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.731 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.732 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.732 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:07.732 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:07.732 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.732 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.733 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.733 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.733 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.733 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.733 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.734 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.734 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.734 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.734 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.735 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:07.735 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:07.735 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:07.735 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:07.735 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:07.735 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:07.736 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:07.736 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:07.736 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:07.736 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:07.736 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:07.737 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:07.737 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:07.737 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:07.737 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:07.737 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:07.738 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:07.738 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:07.738 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:07.738 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:07.739 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:07.739 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:07.739 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:07.739 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:07.739 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:07.740 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:07.740 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:07.740 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.740 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:07.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.740 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:07.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:07.741 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:07.741 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.741 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.742 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.742 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:07.742 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.743 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.743 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:07.743 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.744 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.744 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:07.744 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:07.744 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.745 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.745 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:07.746 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:07.746 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.746 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.746 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.747 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:07.747 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.747 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.748 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:07.748 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.748 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.748 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.748 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:07.749 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:07.749 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.749 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.749 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:07.750 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:07.750 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:07.750 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:07.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.751 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:07.751 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:07.751 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.752 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:07.752 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:07.752 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:07.752 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.752 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.753 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:07.753 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:07.753 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:07.753 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:07.754 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:07.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.754 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.754 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:07.757 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:07.757 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:07.757 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:07.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:07.757 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:07.757 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 10 librados: watch_flush enter 2023-10-30T12:11:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 10 librados: watch_flush exit 2023-10-30T12:11:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 20 client.114557.objecter shutdown clearing up homeless session... 2023-10-30T12:11:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 10 client.114557.objecter successfully canceled tick 2023-10-30T12:11:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb49c039320 msgr2=0x7fb49c03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb49c039320 0x7fb49c03b7e0 secure :-1 s=READY pgs=4108 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00055e0 tx=0x7fb4b0003d10 comp rx=0 tx=0).stop 2023-10-30T12:11:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 10 monclient: shutdown 2023-10-30T12:11:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8070d10 msgr2=0x7fb4b81ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 --2- 172.21.15.50:0/1221847875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb4b8070d10 0x7fb4b81ce8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0090b0 tx=0x7fb4ac0046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bd7cf640 1 -- 172.21.15.50:0/1221847875 reap_dead start 2023-10-30T12:11:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bd7cf640 1 -- 172.21.15.50:0/1221847875 reap_dead start 2023-10-30T12:11:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 shutdown_connections 2023-10-30T12:11:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 >> 172.21.15.50:0/1221847875 conn(0x7fb4b80a4800 msgr2=0x7fb4b80a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 shutdown_connections 2023-10-30T12:11:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 -- 172.21.15.50:0/1221847875 wait complete. 2023-10-30T12:11:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.719+0000 7fb4bf232640 1 librados: shutdown 2023-10-30T12:11:07.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:11:07.763 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:11:07.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:11:07.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.868+0000 7fdc2a7c5640 10 monclient: get_monmap_and_config 2023-10-30T12:11:07.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.868+0000 7fdc2a7c5640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.868+0000 7fdc2a7c5640 10 monclient: monmap: 2023-10-30T12:11:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:07.869698+0000 2023-10-30T12:11:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:07.869698+0000 2023-10-30T12:11:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:07.873 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 1 Processor -- start 2023-10-30T12:11:07.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 1 -- start start 2023-10-30T12:11:07.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient: init 2023-10-30T12:11:07.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-10-30T12:11:07.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc2409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient(hunting): picked mon.noname-b con 0x7fdc24098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc24098900 0x7fdc24098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient(hunting): picked mon.noname-a con 0x7fdc24098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:07.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc24132740 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2409ad30 con 0x7fdc24132740 2023-10-30T12:11:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2409c460 con 0x7fdc24098900 2023-10-30T12:11:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2409b260 con 0x7fdc24098520 2023-10-30T12:11:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.869+0000 7fdc2a7c5640 10 monclient(hunting): authenticate will time out at 1513.052927s 2023-10-30T12:11:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc24098900 0x7fdc24098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc2409bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc28d62640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdc24132740 0x7fdc24132bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:32958/0 (socket says 172.21.15.50:32958) 2023-10-30T12:11:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc28d62640 1 -- 172.21.15.50:0/2508406729 learned_addr learned my addr 172.21.15.50:0/2508406729 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:11:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 10 monclient(hunting): get_auth_request con 0x7fdc24098900 auth_method 0 2023-10-30T12:11:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): get_auth_request con 0x7fdc24098520 auth_method 0 2023-10-30T12:11:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient(hunting): handle_auth_done global_id 114560 payload 274 2023-10-30T12:11:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 1 -- 172.21.15.50:0/2508406729 >> v2:172.21.15.50:3300/0 conn(0x7fdc24098900 msgr2=0x7fdc24098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2508406729 >> v2:172.21.15.50:3300/0 conn(0x7fdc24098900 0x7fdc24098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 1 -- 172.21.15.50:0/2508406729 >> v1:172.21.15.50:6789/0 conn(0x7fdc24132740 legacy=0x7fdc24132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 1 monclient: found mon.noname-b 2023-10-30T12:11:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc23fff640 1 -- 172.21.15.50:0/2508406729 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc24072120 con 0x7fdc24098520 2023-10-30T12:11:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.870+0000 7fdc237fe640 1 --2- 172.21.15.50:0/2508406729 >> v2:172.21.15.50:3300/0 conn(0x7fdc24098900 0x7fdc24098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:11:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc23fff640 1 --2- 172.21.15.50:0/2508406729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc2409bf20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009790 tx=0x7fdc1802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3793557d7556ab15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2508406729 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc180316f0 con 0x7fdc24098520 2023-10-30T12:11:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: dump: 2023-10-30T12:11:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.891 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 5 monclient: authenticate success, global_id 114560 2023-10-30T12:11:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2508406729 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc18031850 con 0x7fdc24098520 2023-10-30T12:11:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 1 -- 172.21.15.50:0/2508406729 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18031be0 con 0x7fdc24098520 2023-10-30T12:11:07.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 4 set_mon_vals no callback set 2023-10-30T12:11:07.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc22ffd640 10 monclient: dump: 2023-10-30T12:11:07.893 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.894 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.894 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.894 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.894 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.894 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.895 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.895 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.895 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.895 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 10 monclient: shutdown 2023-10-30T12:11:07.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:07.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/2508406729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 msgr2=0x7fdc2409bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/2508406729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc2409bf20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009790 tx=0x7fdc1802b790 comp rx=0 tx=0).stop 2023-10-30T12:11:07.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/2508406729 shutdown_connections 2023-10-30T12:11:07.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/2508406729 >> v2:172.21.15.50:3300/0 conn(0x7fdc24098900 0x7fdc24098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/2508406729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc2409bf20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/2508406729 >> 172.21.15.50:0/2508406729 conn(0x7fdc240a05c0 msgr2=0x7fdc2409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:07.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/2508406729 shutdown_connections 2023-10-30T12:11:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.871+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/2508406729 wait complete. 2023-10-30T12:11:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.872+0000 7fdc2a7c5640 10 monclient: build_initial_monmap 2023-10-30T12:11:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.872+0000 7fdc2a7c5640 10 monclient: monmap: 2023-10-30T12:11:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:07.900 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.872+0000 7fdc2a7c5640 1 librados: starting msgr at 2023-10-30T12:11:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.872+0000 7fdc2a7c5640 1 librados: starting objecter 2023-10-30T12:11:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.872+0000 7fdc2a7c5640 1 Processor -- start 2023-10-30T12:11:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 -- start start 2023-10-30T12:11:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 librados: setting wanted keys 2023-10-30T12:11:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 librados: calling monclient init 2023-10-30T12:11:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient: init 2023-10-30T12:11:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:11:07.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc24131bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): picked mon.noname-c con 0x7fdc24098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:07.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc24098900 0x7fdc241320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): picked mon.noname-a con 0x7fdc24098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:07.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc24132740 0x7fdc2412ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): picked mon.noname-b con 0x7fdc24132740 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc241caec0 con 0x7fdc24098900 2023-10-30T12:11:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2412f180 con 0x7fdc24132740 2023-10-30T12:11:07.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:07.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc24071e80 con 0x7fdc24098520 2023-10-30T12:11:07.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:07.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc2a7c5640 10 monclient(hunting): authenticate will time out at 1513.056451s 2023-10-30T12:11:07.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc24098900 0x7fdc241320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc24131bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc24098900 0x7fdc241320f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38542/0 (socket says 172.21.15.50:38542) 2023-10-30T12:11:07.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 1 -- 172.21.15.50:0/536637578 learned_addr learned my addr 172.21.15.50:0/536637578 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:07.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 10 monclient(hunting): get_auth_request con 0x7fdc24098900 auth_method 0 2023-10-30T12:11:07.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc24131bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49610/0 (socket says 172.21.15.50:49610) 2023-10-30T12:11:07.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): get_auth_request con 0x7fdc24098520 auth_method 0 2023-10-30T12:11:07.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:07.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:07.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:07.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:07.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:07.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:07.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient(hunting): handle_auth_done global_id 114563 payload 824 2023-10-30T12:11:07.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc28d62640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc24132740 0x7fdc2412ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 -- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc24132740 msgr2=0x7fdc2412ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc24132740 0x7fdc2412ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 -- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc24098900 msgr2=0x7fdc241320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:07.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc24098900 0x7fdc241320f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:07.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient: _finish_hunting 0 2023-10-30T12:11:07.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 monclient: found mon.noname-c 2023-10-30T12:11:07.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:07.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc23fff640 1 -- 172.21.15.50:0/536637578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc14009400 con 0x7fdc24098520 2023-10-30T12:11:07.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc28d62640 10 monclient: get_auth_request con 0x7fdc24132740 auth_method 0 2023-10-30T12:11:07.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc237fe640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc24098900 0x7fdc241320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:11:07.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.873+0000 7fdc28d62640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc24132740 0x7fdc2412ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:11:07.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc23fff640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc24131bb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc18033000 tx=0x7fdc180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 client.?.objecter ms_handle_connect 0x7fdc24098520 2023-10-30T12:11:07.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:07.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 1 -- 172.21.15.50:0/536637578 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18005b90 con 0x7fdc24098520 2023-10-30T12:11:07.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:07.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: dump: 2023-10-30T12:11:07.913 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.913 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.913 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.913 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.913 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.914 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.914 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.914 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.914 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: _finish_auth 0 2023-10-30T12:11:07.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:11:07.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:07.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 1 -- 172.21.15.50:0/536637578 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc18005d30 con 0x7fdc24098520 2023-10-30T12:11:07.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:07.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 1 -- 172.21.15.50:0/536637578 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18005ea0 con 0x7fdc24098520 2023-10-30T12:11:07.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 5 monclient: authenticate success, global_id 114563 2023-10-30T12:11:07.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 10 monclient: _renew_subs 2023-10-30T12:11:07.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc180093a0 con 0x7fdc24098520 2023-10-30T12:11:07.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:07.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:07.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc20ff9640 10 monclient: dump: 2023-10-30T12:11:07.917 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:07.918 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:07.918 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.918 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:07.918 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:07.919 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:07.919 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:07.919 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:07.919 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:07.919 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:07.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc227fc640 4 set_mon_vals no callback set 2023-10-30T12:11:07.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 10 client.114563.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:07.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 10 monclient: _renew_subs 2023-10-30T12:11:07.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2412ebc0 con 0x7fdc24098520 2023-10-30T12:11:07.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.874+0000 7fdc2a7c5640 1 librados: init done 2023-10-30T12:11:07.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc20ff9640 1 -- 172.21.15.50:0/536637578 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdc1803b040 con 0x7fdc24098520 2023-10-30T12:11:07.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc20ff9640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc10039320 0x7fdc1003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:07.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc2a7c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc2a7c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc2a7c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:07.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2412e980 con 0x7fdc24098520 2023-10-30T12:11:07.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc20ff9640 1 -- 172.21.15.50:0/536637578 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(346..346 src has 1..346) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fdc1806c5d0 con 0x7fdc24098520 2023-10-30T12:11:07.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc20ff9640 10 client.114563.objecter ms_dispatch 0x7fdc24000b90 osd_map(346..346 src has 1..346) v4 2023-10-30T12:11:07.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc20ff9640 3 client.114563.objecter handle_osd_map got epochs [346,346] > 0 2023-10-30T12:11:07.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc20ff9640 3 client.114563.objecter handle_osd_map decoding full epoch 346 2023-10-30T12:11:07.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc20ff9640 20 client.114563.objecter dump_active .. 0 homeless 2023-10-30T12:11:07.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc237fe640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc10039320 0x7fdc1003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:07.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.875+0000 7fdc237fe640 10 monclient: get_auth_request con 0x7fdc10039320 auth_method 0 2023-10-30T12:11:07.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.876+0000 7fdc20ff9640 10 client.114563.objecter ms_handle_connect 0x7fdc10039320 2023-10-30T12:11:07.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.876+0000 7fdc237fe640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc10039320 0x7fdc1003b7e0 secure :-1 s=READY pgs=4109 cs=0 l=1 rev1=1 crypto rx=0x7fdc24071d20 tx=0x7fdc14009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:07.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.879+0000 7fdc20ff9640 1 -- 172.21.15.50:0/536637578 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdc2412e980 con 0x7fdc24098520 2023-10-30T12:11:07.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.879+0000 7fdc20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:07.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:07.879+0000 7fdc20ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.125+0000 7fdc2a7c5640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:11:08.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.125+0000 7fdc2a7c5640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:11:08.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.125+0000 7fdc2a7c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:08.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.125+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fdc2412e980 con 0x7fdc24098520 2023-10-30T12:11:08.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.127+0000 7fdc20ff9640 1 -- 172.21.15.50:0/536637578 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fdc2412e980 con 0x7fdc24098520 2023-10-30T12:11:08.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.127+0000 7fdc20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:11:08.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.127+0000 7fdc20ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:08.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.127+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.127+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=8,logm=57} crc {auth=663737335,config=3140514009,health=3896921429,kv=1634934840,logm=2219016084}) 2023-10-30T12:11:08.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.127+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=11,kv=9,logm=57} crc {auth=3046209764,config=3140514009,health=2181913952,kv=2905431493,logm=1669293716}) 2023-10-30T12:11:08.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.127+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.127+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=8,logm=57} crc {auth=663737335,config=3140514009,health=3896921429,kv=1634934840,logm=2219016084}) 2023-10-30T12:11:08.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.127+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=12,kv=8,logm=59} crc {auth=3642410294,config=3140514009,health=871510025,kv=3134794136,logm=1141184793}) 2023-10-30T12:11:08.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.128+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.128+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517739765}) 2023-10-30T12:11:08.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.128+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407433840}) 2023-10-30T12:11:08.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.128+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.128+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517739765}) 2023-10-30T12:11:08.132 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.128+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415573307}) 2023-10-30T12:11:08.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 10 librados: watch_flush enter 2023-10-30T12:11:08.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 10 librados: watch_flush exit 2023-10-30T12:11:08.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 20 client.114563.objecter shutdown clearing up homeless session... 2023-10-30T12:11:08.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 10 client.114563.objecter successfully canceled tick 2023-10-30T12:11:08.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc10039320 msgr2=0x7fdc1003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:08.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc10039320 0x7fdc1003b7e0 secure :-1 s=READY pgs=4109 cs=0 l=1 rev1=1 crypto rx=0x7fdc24071d20 tx=0x7fdc14009190 comp rx=0 tx=0).stop 2023-10-30T12:11:08.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 10 monclient: shutdown 2023-10-30T12:11:08.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:08.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 msgr2=0x7fdc24131bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:08.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc24131bb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc18033000 tx=0x7fdc180046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:08.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 shutdown_connections 2023-10-30T12:11:08.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc10039320 0x7fdc1003b7e0 unknown :-1 s=CLOSED pgs=4109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:08.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc24132740 0x7fdc2412ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:08.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc24098900 0x7fdc241320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:08.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 --2- 172.21.15.50:0/536637578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc24098520 0x7fdc24131bb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:08.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 >> 172.21.15.50:0/536637578 conn(0x7fdc240a05c0 msgr2=0x7fdc241b98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:08.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 shutdown_connections 2023-10-30T12:11:08.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 -- 172.21.15.50:0/536637578 wait complete. 2023-10-30T12:11:08.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:08.129+0000 7fdc2a7c5640 1 librados: shutdown 2023-10-30T12:11:08.141 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.129+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.141 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.129+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083673499}) 2023-10-30T12:11:08.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.129+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1636817038}) 2023-10-30T12:11:08.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.129+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.129+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083673499}) 2023-10-30T12:11:08.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.129+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3784462287}) 2023-10-30T12:11:08.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139088514}) 2023-10-30T12:11:08.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3801040544}) 2023-10-30T12:11:08.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139088514}) 2023-10-30T12:11:08.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223925458}) 2023-10-30T12:11:08.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714231751}) 2023-10-30T12:11:08.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1043523266}) 2023-10-30T12:11:08.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714231751}) 2023-10-30T12:11:08.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.131+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2747114538}) 2023-10-30T12:11:08.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.132+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.132+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788375493}) 2023-10-30T12:11:08.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.132+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1160327446}) 2023-10-30T12:11:08.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.132+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.132+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788375493}) 2023-10-30T12:11:08.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.132+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2572562283}) 2023-10-30T12:11:08.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.133+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.133+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152608129}) 2023-10-30T12:11:08.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.133+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3763999839}) 2023-10-30T12:11:08.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.133+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.133+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152608129}) 2023-10-30T12:11:08.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.133+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439816580}) 2023-10-30T12:11:08.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.134+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.134+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273800605}) 2023-10-30T12:11:08.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.134+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1213280252}) 2023-10-30T12:11:08.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.134+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.134+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273800605}) 2023-10-30T12:11:08.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.134+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=62350215}) 2023-10-30T12:11:08.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.135+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.135+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075415940}) 2023-10-30T12:11:08.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.135+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501485309}) 2023-10-30T12:11:08.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.135+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.135+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075415940}) 2023-10-30T12:11:08.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.135+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805488254}) 2023-10-30T12:11:08.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.136+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.136+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141634595}) 2023-10-30T12:11:08.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.136+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545325067}) 2023-10-30T12:11:08.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.136+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.136+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141634595}) 2023-10-30T12:11:08.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.136+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178896658}) 2023-10-30T12:11:08.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749601391}) 2023-10-30T12:11:08.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1008475456}) 2023-10-30T12:11:08.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749601391}) 2023-10-30T12:11:08.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1353106342}) 2023-10-30T12:11:08.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4040918693}) 2023-10-30T12:11:08.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321419910}) 2023-10-30T12:11:08.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4040918693}) 2023-10-30T12:11:08.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.137+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576182754}) 2023-10-30T12:11:08.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.138+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.138+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836740222}) 2023-10-30T12:11:08.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.138+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214697552}) 2023-10-30T12:11:08.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.138+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.138+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836740222}) 2023-10-30T12:11:08.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.138+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352326770}) 2023-10-30T12:11:08.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.139+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.139+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147272529}) 2023-10-30T12:11:08.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.139+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246770650}) 2023-10-30T12:11:08.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.139+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.139+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147272529}) 2023-10-30T12:11:08.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.139+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3384865318}) 2023-10-30T12:11:08.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.140+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.140+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293813079}) 2023-10-30T12:11:08.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.140+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1001136353}) 2023-10-30T12:11:08.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.140+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.140+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293813079}) 2023-10-30T12:11:08.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.140+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539213516}) 2023-10-30T12:11:08.159 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484947204}) 2023-10-30T12:11:08.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4103486206}) 2023-10-30T12:11:08.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484947204}) 2023-10-30T12:11:08.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397114095}) 2023-10-30T12:11:08.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:11:08.176 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.176 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878780915}) 2023-10-30T12:11:08.177 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12645646}) 2023-10-30T12:11:08.177 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.177 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.141+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878780915}) 2023-10-30T12:11:08.177 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.142+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117393209}) 2023-10-30T12:11:08.177 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.142+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.178 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.142+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489983671}) 2023-10-30T12:11:08.178 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.142+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15591499}) 2023-10-30T12:11:08.178 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.142+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.178 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.142+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489983671}) 2023-10-30T12:11:08.178 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.142+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739989045}) 2023-10-30T12:11:08.178 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.143+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.179 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.143+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916889153}) 2023-10-30T12:11:08.179 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.143+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2680648372}) 2023-10-30T12:11:08.179 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.143+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.179 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.143+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916889153}) 2023-10-30T12:11:08.179 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.143+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988227587}) 2023-10-30T12:11:08.180 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.180 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113514055}) 2023-10-30T12:11:08.180 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=532548898}) 2023-10-30T12:11:08.180 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.181 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113514055}) 2023-10-30T12:11:08.181 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232984283}) 2023-10-30T12:11:08.181 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.181 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730752514}) 2023-10-30T12:11:08.181 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159512718}) 2023-10-30T12:11:08.182 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.182 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730752514}) 2023-10-30T12:11:08.182 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.144+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1120748587}) 2023-10-30T12:11:08.182 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.145+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.182 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.145+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971241795}) 2023-10-30T12:11:08.183 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.145+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325855171}) 2023-10-30T12:11:08.183 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.145+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.183 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.145+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971241795}) 2023-10-30T12:11:08.184 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.145+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103762729}) 2023-10-30T12:11:08.184 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.184 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399342593}) 2023-10-30T12:11:08.184 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1539164593}) 2023-10-30T12:11:08.184 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.185 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399342593}) 2023-10-30T12:11:08.185 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=724490974}) 2023-10-30T12:11:08.185 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.185 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056987817}) 2023-10-30T12:11:08.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3390469389}) 2023-10-30T12:11:08.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056987817}) 2023-10-30T12:11:08.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.146+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296798455}) 2023-10-30T12:11:08.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.147+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.147+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337644411}) 2023-10-30T12:11:08.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.147+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174867029}) 2023-10-30T12:11:08.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.147+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.147+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337644411}) 2023-10-30T12:11:08.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.147+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3546022572}) 2023-10-30T12:11:08.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803164899}) 2023-10-30T12:11:08.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309776748}) 2023-10-30T12:11:08.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803164899}) 2023-10-30T12:11:08.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975347768}) 2023-10-30T12:11:08.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2917027439}) 2023-10-30T12:11:08.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929671693}) 2023-10-30T12:11:08.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2917027439}) 2023-10-30T12:11:08.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.148+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052835370}) 2023-10-30T12:11:08.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.149+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.149+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=200002532}) 2023-10-30T12:11:08.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.149+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2900619123}) 2023-10-30T12:11:08.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.149+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.149+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=200002532}) 2023-10-30T12:11:08.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.149+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188640687}) 2023-10-30T12:11:08.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.150+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.150+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142462979}) 2023-10-30T12:11:08.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.150+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2846756197}) 2023-10-30T12:11:08.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.150+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.150+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142462979}) 2023-10-30T12:11:08.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.150+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913624290}) 2023-10-30T12:11:08.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164068604}) 2023-10-30T12:11:08.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=454549091}) 2023-10-30T12:11:08.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164068604}) 2023-10-30T12:11:08.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532674272}) 2023-10-30T12:11:08.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080166899}) 2023-10-30T12:11:08.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1172954398}) 2023-10-30T12:11:08.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080166899}) 2023-10-30T12:11:08.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.151+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1552113182}) 2023-10-30T12:11:08.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.152+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.152+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3540396066}) 2023-10-30T12:11:08.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.152+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=773519739}) 2023-10-30T12:11:08.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.152+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.152+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3540396066}) 2023-10-30T12:11:08.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.152+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2386612278}) 2023-10-30T12:11:08.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240589010}) 2023-10-30T12:11:08.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796072743}) 2023-10-30T12:11:08.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240589010}) 2023-10-30T12:11:08.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926670077}) 2023-10-30T12:11:08.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.153+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2212134041}) 2023-10-30T12:11:08.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604310286}) 2023-10-30T12:11:08.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2212134041}) 2023-10-30T12:11:08.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917950354}) 2023-10-30T12:11:08.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092707828}) 2023-10-30T12:11:08.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855439102}) 2023-10-30T12:11:08.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092707828}) 2023-10-30T12:11:08.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.154+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980502387}) 2023-10-30T12:11:08.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248800603}) 2023-10-30T12:11:08.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609839420}) 2023-10-30T12:11:08.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248800603}) 2023-10-30T12:11:08.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013146044}) 2023-10-30T12:11:08.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2948474289}) 2023-10-30T12:11:08.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2318073018}) 2023-10-30T12:11:08.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2948474289}) 2023-10-30T12:11:08.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.155+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1393281372}) 2023-10-30T12:11:08.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.156+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.156+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445676498}) 2023-10-30T12:11:08.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.156+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4187429513}) 2023-10-30T12:11:08.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.156+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.156+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445676498}) 2023-10-30T12:11:08.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.156+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2311045389}) 2023-10-30T12:11:08.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1960584743}) 2023-10-30T12:11:08.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061613323}) 2023-10-30T12:11:08.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1960584743}) 2023-10-30T12:11:08.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964682714}) 2023-10-30T12:11:08.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235194051}) 2023-10-30T12:11:08.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2280189147}) 2023-10-30T12:11:08.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235194051}) 2023-10-30T12:11:08.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.157+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3216909747}) 2023-10-30T12:11:08.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.158+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.158+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326657491}) 2023-10-30T12:11:08.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.158+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872307356}) 2023-10-30T12:11:08.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.158+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.158+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326657491}) 2023-10-30T12:11:08.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.158+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3573777446}) 2023-10-30T12:11:08.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.159+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.159+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937274235}) 2023-10-30T12:11:08.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.159+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1951574338}) 2023-10-30T12:11:08.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.159+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.159+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937274235}) 2023-10-30T12:11:08.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.159+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1729284080}) 2023-10-30T12:11:08.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.159+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=289530192}) 2023-10-30T12:11:08.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733357946}) 2023-10-30T12:11:08.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=289530192}) 2023-10-30T12:11:08.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2872096410}) 2023-10-30T12:11:08.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764594536}) 2023-10-30T12:11:08.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4193299794}) 2023-10-30T12:11:08.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764594536}) 2023-10-30T12:11:08.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.160+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056952834}) 2023-10-30T12:11:08.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.161+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.161+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542398510}) 2023-10-30T12:11:08.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.161+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012668611}) 2023-10-30T12:11:08.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.161+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.161+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542398510}) 2023-10-30T12:11:08.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.161+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458135485}) 2023-10-30T12:11:08.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.161+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3267700676}) 2023-10-30T12:11:08.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2613939377}) 2023-10-30T12:11:08.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3267700676}) 2023-10-30T12:11:08.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187646761}) 2023-10-30T12:11:08.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1873060493}) 2023-10-30T12:11:08.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2938399439}) 2023-10-30T12:11:08.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1873060493}) 2023-10-30T12:11:08.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.162+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3927315398}) 2023-10-30T12:11:08.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2939839118}) 2023-10-30T12:11:08.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2914328107}) 2023-10-30T12:11:08.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2939839118}) 2023-10-30T12:11:08.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747550481}) 2023-10-30T12:11:08.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596775442}) 2023-10-30T12:11:08.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354229642}) 2023-10-30T12:11:08.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596775442}) 2023-10-30T12:11:08.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.163+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658249450}) 2023-10-30T12:11:08.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.164+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.164+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479908366}) 2023-10-30T12:11:08.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.164+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2723145689}) 2023-10-30T12:11:08.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.164+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.164+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479908366}) 2023-10-30T12:11:08.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.164+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656458335}) 2023-10-30T12:11:08.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975364562}) 2023-10-30T12:11:08.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563818240}) 2023-10-30T12:11:08.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975364562}) 2023-10-30T12:11:08.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2211913273}) 2023-10-30T12:11:08.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016956483}) 2023-10-30T12:11:08.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489566813}) 2023-10-30T12:11:08.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016956483}) 2023-10-30T12:11:08.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.165+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766269350}) 2023-10-30T12:11:08.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.166+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.166+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=127097190,mdsmap=3984425268,mgr=1346511200}) 2023-10-30T12:11:08.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.166+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1943763828,mdsmap=3984425268,mgr=1204984937}) 2023-10-30T12:11:08.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.166+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.166+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=127097190,mdsmap=3984425268,mgr=1346511200}) 2023-10-30T12:11:08.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.166+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2843436439,mdsmap=3984425268,mgr=1346511200}) 2023-10-30T12:11:08.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.167+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.167+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=31} crc {mgr=1810321057,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2730086149,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1912977591,osd_snap=8583042,osdmap=971686574}) 2023-10-30T12:11:08.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.167+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=32} crc {mgr=1715906815,mgr_command_descs=3900280244,mgrstat=3738491305,mon_config_key=3217606506,monmap=3324057927,osd_metadata=451695420,osd_pg_creating=1912977591,osd_snap=3350854959,osdmap=2039369855}) 2023-10-30T12:11:08.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.167+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.167+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=31} crc {mgr=1810321057,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2730086149,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1912977591,osd_snap=8583042,osdmap=971686574}) 2023-10-30T12:11:08.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.167+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=32,osdmap=33} crc {mgr=1715906815,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=19325227,mon_config_key=3217606506,monmap=3324057927,osd_metadata=451695420,osd_pg_creating=1912977591,osd_snap=3944313141,osdmap=3457645022}) 2023-10-30T12:11:08.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.168+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.168+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3550291518}) 2023-10-30T12:11:08.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.168+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=17652011}) 2023-10-30T12:11:08.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.168+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.168+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3550291518}) 2023-10-30T12:11:08.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.168+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2903656677}) 2023-10-30T12:11:08.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.169+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.169+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2568311469}) 2023-10-30T12:11:08.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.169+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=509128389}) 2023-10-30T12:11:08.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.169+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.169+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2568311469}) 2023-10-30T12:11:08.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.169+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1597754607}) 2023-10-30T12:11:08.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.170+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.170+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1945439999}) 2023-10-30T12:11:08.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.170+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2504242478}) 2023-10-30T12:11:08.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.170+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.170+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1945439999}) 2023-10-30T12:11:08.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.170+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154304688}) 2023-10-30T12:11:08.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.171+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.171+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4082406557}) 2023-10-30T12:11:08.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.171+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1257527769}) 2023-10-30T12:11:08.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.171+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.171+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4082406557}) 2023-10-30T12:11:08.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.171+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3282996311}) 2023-10-30T12:11:08.230 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.176+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.176+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=84} crc {osdmap=735583421}) 2023-10-30T12:11:08.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.176+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=87} crc {osdmap=3083729764}) 2023-10-30T12:11:08.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.176+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:11:08.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.176+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=84} crc {osdmap=735583421}) 2023-10-30T12:11:08.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:11:08.176+0000 7f62a6af9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=86} crc {osdmap=3648555651}) 2023-10-30T12:11:09.162 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:11:09.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:09.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.277+0000 7f1f05400640 10 monclient: get_monmap_and_config 2023-10-30T12:11:09.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.277+0000 7f1f05400640 10 monclient: build_initial_monmap 2023-10-30T12:11:09.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.277+0000 7f1f05400640 10 monclient: monmap: 2023-10-30T12:11:09.279 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:09.279 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.279 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:09.278360+0000 2023-10-30T12:11:09.280 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:09.278360+0000 2023-10-30T12:11:09.280 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:09.280 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.280 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:09.280 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:09.281 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:09.281 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:09.281 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:09.281 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 1 Processor -- start 2023-10-30T12:11:09.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 1 -- start start 2023-10-30T12:11:09.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient: init 2023-10-30T12:11:09.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:09.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-10-30T12:11:09.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f000a3150 0x7f1f00132490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient(hunting): picked mon.noname-a con 0x7f1f000a3150 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1f000a3530 0x7f1f001354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient(hunting): picked mon.noname-c con 0x7f1f000a3530 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f000a3960 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000a3f60 con 0x7f1f000a3960 2023-10-30T12:11:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0009c650 con 0x7f1f000a3150 2023-10-30T12:11:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0009b350 con 0x7f1f000a3530 2023-10-30T12:11:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1f05400640 10 monclient(hunting): authenticate will time out at 1514.461642s 2023-10-30T12:11:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f000a3150 0x7f1f00132490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f000a3150 0x7f1f00132490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38552/0 (socket says 172.21.15.50:38552) 2023-10-30T12:11:09.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 1 -- 172.21.15.50:0/2944864596 learned_addr learned my addr 172.21.15.50:0/2944864596 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:09.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 10 monclient(hunting): get_auth_request con 0x7f1f000a3150 auth_method 0 2023-10-30T12:11:09.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.278+0000 7f1efeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1ef7fff640 1 --2- 172.21.15.50:0/2944864596 >> v2:172.21.15.118:3300/0 conn(0x7f1f000a3530 0x7f1f001354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 10 monclient(hunting): handle_auth_done global_id 134382 payload 274 2023-10-30T12:11:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 1 -- 172.21.15.50:0/2944864596 >> v2:172.21.15.118:3300/0 conn(0x7f1f000a3530 msgr2=0x7f1f001354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 1 --2- 172.21.15.50:0/2944864596 >> v2:172.21.15.118:3300/0 conn(0x7f1f000a3530 0x7f1f001354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 1 -- 172.21.15.50:0/2944864596 >> v1:172.21.15.118:6789/0 conn(0x7f1f000a3960 legacy=0x7f1f001359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:11:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 1 monclient: found mon.noname-a 2023-10-30T12:11:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:09.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 1 -- 172.21.15.50:0/2944864596 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f001329d0 con 0x7f1f000a3150 2023-10-30T12:11:09.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1ef7fff640 10 monclient: get_auth_request con 0x7f1f000a3530 auth_method 0 2023-10-30T12:11:09.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1ef7fff640 1 --2- 172.21.15.50:0/2944864596 >> v2:172.21.15.118:3300/0 conn(0x7f1f000a3530 0x7f1f001354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:11:09.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.279+0000 7f1efeffd640 1 --2- 172.21.15.50:0/2944864596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f00132490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f000a8850 tx=0x7f1ef002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a15047eeb86ec3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 1 -- 172.21.15.50:0/2944864596 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0031750 con 0x7f1f000a3150 2023-10-30T12:11:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: dump: 2023-10-30T12:11:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.302 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.302 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.302 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.302 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:11:09.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:11:09.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:09.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 5 monclient: authenticate success, global_id 134382 2023-10-30T12:11:09.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:09.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 1 -- 172.21.15.50:0/2944864596 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef00318b0 con 0x7f1f000a3150 2023-10-30T12:11:09.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:09.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:09.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 4 set_mon_vals no callback set 2023-10-30T12:11:09.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 1 -- 172.21.15.50:0/2944864596 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0031c40 con 0x7f1f000a3150 2023-10-30T12:11:09.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1efe7fc640 10 monclient: dump: 2023-10-30T12:11:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.306 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.306 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.306 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.307 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.307 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.307 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.308 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 10 monclient: shutdown 2023-10-30T12:11:09.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:09.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 -- 172.21.15.50:0/2944864596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 msgr2=0x7f1f00132490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 --2- 172.21.15.50:0/2944864596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f00132490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f000a8850 tx=0x7f1ef002b7f0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 -- 172.21.15.50:0/2944864596 shutdown_connections 2023-10-30T12:11:09.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 --2- 172.21.15.50:0/2944864596 >> v2:172.21.15.118:3300/0 conn(0x7f1f000a3530 0x7f1f001354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 --2- 172.21.15.50:0/2944864596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f00132490 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 -- 172.21.15.50:0/2944864596 >> 172.21.15.50:0/2944864596 conn(0x7f1f000a7d80 msgr2=0x7f1f000a41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:09.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 -- 172.21.15.50:0/2944864596 shutdown_connections 2023-10-30T12:11:09.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.280+0000 7f1f05400640 1 -- 172.21.15.50:0/2944864596 wait complete. 2023-10-30T12:11:09.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 10 monclient: build_initial_monmap 2023-10-30T12:11:09.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 10 monclient: monmap: 2023-10-30T12:11:09.311 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:09.311 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:09.311 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:09.311 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:09.311 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:09.312 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.312 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:09.312 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:09.312 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:09.312 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 1 librados: starting msgr at 2023-10-30T12:11:09.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 1 librados: starting objecter 2023-10-30T12:11:09.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 1 Processor -- start 2023-10-30T12:11:09.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 1 -- start start 2023-10-30T12:11:09.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 1 librados: setting wanted keys 2023-10-30T12:11:09.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 1 librados: calling monclient init 2023-10-30T12:11:09.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.281+0000 7f1f05400640 10 monclient: init 2023-10-30T12:11:09.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:09.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:09.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f001ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): picked mon.noname-a con 0x7f1f000a3150 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:09.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f000a3530 0x7f1f001ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): picked mon.noname-b con 0x7f1f000a3530 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:09.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f000a3960 0x7f1f001cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): picked mon.noname-c con 0x7f1f000a3960 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:09.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0011f600 con 0x7f1f000a3150 2023-10-30T12:11:09.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00121150 con 0x7f1f000a3530 2023-10-30T12:11:09.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f001213a0 con 0x7f1f000a3960 2023-10-30T12:11:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1f05400640 10 monclient(hunting): authenticate will time out at 1514.465356s 2023-10-30T12:11:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f001ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f001ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38558/0 (socket says 172.21.15.50:38558) 2023-10-30T12:11:09.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 -- 172.21.15.50:0/1271501675 learned_addr learned my addr 172.21.15.50:0/1271501675 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:09.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): get_auth_request con 0x7f1f000a3150 auth_method 0 2023-10-30T12:11:09.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f000a3960 0x7f1f001cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 10 monclient(hunting): get_auth_request con 0x7f1f000a3960 auth_method 0 2023-10-30T12:11:09.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1ef7fff640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f000a3530 0x7f1f001ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1ef7fff640 10 monclient(hunting): get_auth_request con 0x7f1f000a3530 auth_method 0 2023-10-30T12:11:09.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1ef7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1ef7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1ef7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient(hunting): handle_auth_done global_id 134385 payload 824 2023-10-30T12:11:09.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 -- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f000a3960 msgr2=0x7f1f001cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f000a3960 0x7f1f001cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 -- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f000a3530 msgr2=0x7f1f001ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f000a3530 0x7f1f001ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:11:09.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 monclient: found mon.noname-a 2023-10-30T12:11:09.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:09.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.282+0000 7f1efeffd640 1 -- 172.21.15.50:0/1271501675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00094d0 con 0x7f1f000a3150 2023-10-30T12:11:09.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1efeffd640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f001ca3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009960 tx=0x7f1ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 client.?.objecter ms_handle_connect 0x7f1f000a3150 2023-10-30T12:11:09.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:09.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 1 -- 172.21.15.50:0/1271501675 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0033c80 con 0x7f1f000a3150 2023-10-30T12:11:09.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:09.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: dump: 2023-10-30T12:11:09.326 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.326 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.326 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.326 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.327 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.327 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.327 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.327 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.328 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.328 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:09.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:09.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:09.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 1 -- 172.21.15.50:0/1271501675 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef0033de0 con 0x7f1f000a3150 2023-10-30T12:11:09.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 5 monclient: authenticate success, global_id 134385 2023-10-30T12:11:09.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:09.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 1 -- 172.21.15.50:0/1271501675 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef003c640 con 0x7f1f000a3150 2023-10-30T12:11:09.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:09.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1ef77fe640 10 monclient: dump: 2023-10-30T12:11:09.330 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.330 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.331 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.331 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.331 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.331 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.331 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.332 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.332 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.332 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 10 monclient: _renew_subs 2023-10-30T12:11:09.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0011fc10 con 0x7f1f000a3150 2023-10-30T12:11:09.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1efdffb640 4 set_mon_vals no callback set 2023-10-30T12:11:09.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 10 client.134385.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:09.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 10 monclient: _renew_subs 2023-10-30T12:11:09.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f00135820 con 0x7f1f000a3150 2023-10-30T12:11:09.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.283+0000 7f1f05400640 1 librados: init done 2023-10-30T12:11:09.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef77fe640 1 -- 172.21.15.50:0/1271501675 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1ef003a040 con 0x7f1f000a3150 2023-10-30T12:11:09.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1f05400640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:09.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1f05400640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:09.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1f05400640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0011f090 con 0x7f1f000a3150 2023-10-30T12:11:09.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef77fe640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ee00392f0 0x7f1ee003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef77fe640 1 -- 172.21.15.50:0/1271501675 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f1ef006c9a0 con 0x7f1f000a3150 2023-10-30T12:11:09.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef77fe640 10 client.134385.objecter ms_dispatch 0x7f1f00000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:09.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef77fe640 3 client.134385.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:09.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef77fe640 3 client.134385.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:09.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef77fe640 20 client.134385.objecter dump_active .. 0 homeless 2023-10-30T12:11:09.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef7fff640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ee00392f0 0x7f1ee003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.284+0000 7f1ef7fff640 10 monclient: get_auth_request con 0x7f1ee00392f0 auth_method 0 2023-10-30T12:11:09.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.285+0000 7f1ef77fe640 10 client.134385.objecter ms_handle_connect 0x7f1ee00392f0 2023-10-30T12:11:09.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.285+0000 7f1ef7fff640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ee00392f0 0x7f1ee003b7b0 secure :-1 s=READY pgs=4110 cs=0 l=1 rev1=1 crypto rx=0x7f1f0011f930 tx=0x7f1ee8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.289+0000 7f1ef77fe640 1 -- 172.21.15.50:0/1271501675 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ef003c7a0 con 0x7f1f000a3150 2023-10-30T12:11:09.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.289+0000 7f1ef77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:09.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.289+0000 7f1ef77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:09.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.532+0000 7f1f05400640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:09.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.532+0000 7f1f05400640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:09.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.532+0000 7f1f05400640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.532+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1f001214e0 con 0x7f1f000a3150 2023-10-30T12:11:09.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.532+0000 7f1ef77fe640 1 -- 172.21.15.50:0/1271501675 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f1f001214e0 con 0x7f1f000a3150 2023-10-30T12:11:09.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.532+0000 7f1ef77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:09.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.532+0000 7f1ef77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:09.536 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:09.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 10 librados: watch_flush enter 2023-10-30T12:11:09.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 10 librados: watch_flush exit 2023-10-30T12:11:09.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 20 client.134385.objecter shutdown clearing up homeless session... 2023-10-30T12:11:09.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 10 client.134385.objecter successfully canceled tick 2023-10-30T12:11:09.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ee00392f0 msgr2=0x7f1ee003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ee00392f0 0x7f1ee003b7b0 secure :-1 s=READY pgs=4110 cs=0 l=1 rev1=1 crypto rx=0x7f1f0011f930 tx=0x7f1ee8008040 comp rx=0 tx=0).stop 2023-10-30T12:11:09.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 10 monclient: shutdown 2023-10-30T12:11:09.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:09.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 msgr2=0x7f1f001ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f001ca3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009960 tx=0x7f1ef00046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 shutdown_connections 2023-10-30T12:11:09.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ee00392f0 0x7f1ee003b7b0 unknown :-1 s=CLOSED pgs=4110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f000a3960 0x7f1f001cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f000a3530 0x7f1f001ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 --2- 172.21.15.50:0/1271501675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f000a3150 0x7f1f001ca3a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 >> 172.21.15.50:0/1271501675 conn(0x7f1f000a7d80 msgr2=0x7f1f00135a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:09.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 shutdown_connections 2023-10-30T12:11:09.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 -- 172.21.15.50:0/1271501675 wait complete. 2023-10-30T12:11:09.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.534+0000 7f1f05400640 1 librados: shutdown 2023-10-30T12:11:09.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:11:09.546 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:11:09.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:11:09.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:11:09.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.651+0000 7f6db7125640 10 monclient: get_monmap_and_config 2023-10-30T12:11:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.651+0000 7f6db7125640 10 monclient: build_initial_monmap 2023-10-30T12:11:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.651+0000 7f6db7125640 10 monclient: monmap: 2023-10-30T12:11:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:09.652668+0000 2023-10-30T12:11:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:09.652668+0000 2023-10-30T12:11:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:09.655 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 1 Processor -- start 2023-10-30T12:11:09.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 1 -- start start 2023-10-30T12:11:09.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient: init 2023-10-30T12:11:09.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:09.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T12:11:09.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6db00a0080 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:09.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6db00a0460 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:09.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6db0139bd0 0x7f6db013bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient(hunting): picked mon.noname-a con 0x7f6db0139bd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:09.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db009e440 con 0x7f6db00a0460 2023-10-30T12:11:09.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db0073c30 con 0x7f6db00a0080 2023-10-30T12:11:09.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db009e610 con 0x7f6db0139bd0 2023-10-30T12:11:09.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:09.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db7125640 10 monclient(hunting): authenticate will time out at 1514.835889s 2023-10-30T12:11:09.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6db56c2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6db0139bd0 0x7f6db013bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6daffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6db00a0460 0x7f6db00a0870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:32962/0 (socket says 172.21.15.50:32962) 2023-10-30T12:11:09.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.652+0000 7f6daffff640 1 -- 172.21.15.50:0/3262238788 learned_addr learned my addr 172.21.15.50:0/3262238788 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:11:09.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): get_auth_request con 0x7f6db0139bd0 auth_method 0 2023-10-30T12:11:09.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient(hunting): handle_auth_done global_id 134388 payload 274 2023-10-30T12:11:09.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 1 -- 172.21.15.50:0/3262238788 >> v1:172.21.15.118:6789/0 conn(0x7f6db00a0080 legacy=0x7f6db0098f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 1 -- 172.21.15.50:0/3262238788 >> v1:172.21.15.50:6789/0 conn(0x7f6db00a0460 legacy=0x7f6db00a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient: _finish_hunting 0 2023-10-30T12:11:09.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 1 monclient: found mon.noname-a 2023-10-30T12:11:09.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:09.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6db56c2640 1 -- 172.21.15.50:0/3262238788 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db0074380 con 0x7f6db0139bd0 2023-10-30T12:11:09.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6daf7fe640 1 -- 172.21.15.50:0/3262238788 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250229707 0 0) 0x7f6db009e440 con 0x7f6db00a0460 2023-10-30T12:11:09.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.653+0000 7f6daf7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:11:09.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db56c2640 1 --2- 172.21.15.50:0/3262238788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0139bd0 0x7f6db013bfc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009d90 tx=0x7f6da802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea431a2c0ddd7bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 1 -- 172.21.15.50:0/3262238788 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da802cc60 con 0x7f6db0139bd0 2023-10-30T12:11:09.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:09.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: dump: 2023-10-30T12:11:09.679 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.679 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.679 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.679 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.679 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.679 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.680 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.680 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.680 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.680 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:09.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:09.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:09.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 1 -- 172.21.15.50:0/3262238788 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da802cdc0 con 0x7f6db0139bd0 2023-10-30T12:11:09.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 5 monclient: authenticate success, global_id 134388 2023-10-30T12:11:09.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:09.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:09.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:09.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 1 -- 172.21.15.50:0/3262238788 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8034660 con 0x7f6db0139bd0 2023-10-30T12:11:09.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 4 set_mon_vals no callback set 2023-10-30T12:11:09.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:09.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6daf7fe640 10 monclient: dump: 2023-10-30T12:11:09.683 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.683 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.683 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.683 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.684 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.684 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 10 monclient: shutdown 2023-10-30T12:11:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 1 -- 172.21.15.50:0/3262238788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0139bd0 msgr2=0x7f6db013bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 1 --2- 172.21.15.50:0/3262238788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0139bd0 0x7f6db013bfc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009d90 tx=0x7f6da802beb0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 1 -- 172.21.15.50:0/3262238788 shutdown_connections 2023-10-30T12:11:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 1 --2- 172.21.15.50:0/3262238788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0139bd0 0x7f6db013bfc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 1 -- 172.21.15.50:0/3262238788 >> 172.21.15.50:0/3262238788 conn(0x7f6db00a1170 msgr2=0x7f6db00a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 1 -- 172.21.15.50:0/3262238788 shutdown_connections 2023-10-30T12:11:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.654+0000 7f6db7125640 1 -- 172.21.15.50:0/3262238788 wait complete. 2023-10-30T12:11:09.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 10 monclient: build_initial_monmap 2023-10-30T12:11:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 10 monclient: monmap: 2023-10-30T12:11:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:09.687 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:09.688 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.688 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:09.688 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:09.688 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:09.688 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 1 librados: starting msgr at 2023-10-30T12:11:09.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 1 librados: starting objecter 2023-10-30T12:11:09.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 1 Processor -- start 2023-10-30T12:11:09.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 1 -- start start 2023-10-30T12:11:09.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 1 librados: setting wanted keys 2023-10-30T12:11:09.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 1 librados: calling monclient init 2023-10-30T12:11:09.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.655+0000 7f6db7125640 10 monclient: init 2023-10-30T12:11:09.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:09.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:09.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db00a0080 0x7f6db008c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): picked mon.noname-a con 0x7f6db00a0080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:09.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6db00a0460 0x7f6db008cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): picked mon.noname-b con 0x7f6db00a0460 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:09.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6db0139bd0 0x7f6db008d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): picked mon.noname-c con 0x7f6db0139bd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:09.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db013b770 con 0x7f6db00a0080 2023-10-30T12:11:09.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db008fd20 con 0x7f6db00a0460 2023-10-30T12:11:09.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db0090000 con 0x7f6db0139bd0 2023-10-30T12:11:09.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:09.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db7125640 10 monclient(hunting): authenticate will time out at 1514.839215s 2023-10-30T12:11:09.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db00a0080 0x7f6db008c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6db0139bd0 0x7f6db008d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db00a0080 0x7f6db008c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38574/0 (socket says 172.21.15.50:38574) 2023-10-30T12:11:09.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 1 -- 172.21.15.50:0/2565464369 learned_addr learned my addr 172.21.15.50:0/2565464369 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:09.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 10 monclient(hunting): get_auth_request con 0x7f6db00a0080 auth_method 0 2023-10-30T12:11:09.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): get_auth_request con 0x7f6db0139bd0 auth_method 0 2023-10-30T12:11:09.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6daffff640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6db00a0460 0x7f6db008cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient(hunting): handle_auth_done global_id 114569 payload 824 2023-10-30T12:11:09.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6db00a0460 msgr2=0x7f6db008cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6db00a0460 0x7f6db008cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db00a0080 msgr2=0x7f6db008c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db00a0080 0x7f6db008c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient: _finish_hunting 0 2023-10-30T12:11:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 monclient: found mon.noname-c 2023-10-30T12:11:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:09.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 -- 172.21.15.50:0/2565464369 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da8009a70 con 0x7f6db0139bd0 2023-10-30T12:11:09.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db4ec1640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db00a0080 0x7f6db008c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:11:09.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6db56c2640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6db0139bd0 0x7f6db008d2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6da8033000 tx=0x7f6da8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6dad7fa640 10 client.?.objecter ms_handle_connect 0x7f6db0139bd0 2023-10-30T12:11:09.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6dad7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:09.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8040650 con 0x7f6db0139bd0 2023-10-30T12:11:09.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6dad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:09.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6dad7fa640 10 monclient: dump: 2023-10-30T12:11:09.701 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.701 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.702 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.702 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.702 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.702 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.702 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.703 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.656+0000 7f6dad7fa640 10 monclient: _finish_auth 0 2023-10-30T12:11:09.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:11:09.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:09.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da80407b0 con 0x7f6db0139bd0 2023-10-30T12:11:09.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 5 monclient: authenticate success, global_id 114569 2023-10-30T12:11:09.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:09.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8040b60 con 0x7f6db0139bd0 2023-10-30T12:11:09.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 10 monclient: _renew_subs 2023-10-30T12:11:09.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:09.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db008e180 con 0x7f6db0139bd0 2023-10-30T12:11:09.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6daeffd640 4 set_mon_vals no callback set 2023-10-30T12:11:09.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:09.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 10 monclient: dump: 2023-10-30T12:11:09.707 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.707 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.708 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.708 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.708 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.708 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.708 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.709 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.709 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.709 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 10 client.114569.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:09.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 10 monclient: _renew_subs 2023-10-30T12:11:09.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:09.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db008d800 con 0x7f6db0139bd0 2023-10-30T12:11:09.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 1 librados: init done 2023-10-30T12:11:09.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:09.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0097da0 0x7f6db00704c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db0090bc0 con 0x7f6db0097da0 2023-10-30T12:11:09.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.657+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6da8040cc0 con 0x7f6db0139bd0 2023-10-30T12:11:09.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0097da0 0x7f6db00704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient: get_auth_request con 0x7f6db0097da0 auth_method 0 2023-10-30T12:11:09.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6d940392f0 0x7f6d9403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f6da806da70 con 0x7f6db0139bd0 2023-10-30T12:11:09.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 client.114569.objecter ms_dispatch 0x7f6db0000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:09.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 3 client.114569.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:09.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 3 client.114569.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:09.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_done global_id 134397 payload 824 2023-10-30T12:11:09.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 1 -- 172.21.15.50:0/2565464369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6db0098880 con 0x7f6db0097da0 2023-10-30T12:11:09.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 20 client.114569.objecter dump_active .. 0 homeless 2023-10-30T12:11:09.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6daffff640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6d940392f0 0x7f6d9403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6daffff640 10 monclient: get_auth_request con 0x7f6d940392f0 auth_method 0 2023-10-30T12:11:09.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 client.114569.objecter ms_handle_connect 0x7f6db0097da0 2023-10-30T12:11:09.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 client.114569.objecter resend_mon_ops 2023-10-30T12:11:09.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6db4ec1640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0097da0 0x7f6db00704c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6da400fa30 tx=0x7f6da400fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da40052e0 con 0x7f6db0097da0 2023-10-30T12:11:09.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:09.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 monclient: dump: 2023-10-30T12:11:09.716 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.716 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.716 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.716 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.717 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.717 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.717 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.717 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.717 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.717 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f6da4005440 con 0x7f6db0097da0 2023-10-30T12:11:09.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:09.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:09.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0097da0 msgr2=0x7f6db00704c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0097da0 0x7f6db00704c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6da400fa30 tx=0x7f6da400fef0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6daffff640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6d940392f0 0x7f6d9403b7b0 secure :-1 s=READY pgs=4111 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c007fd0 tx=0x7f6d9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0097da0 msgr2=0x7f6db00704c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:09.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.658+0000 7f6dad7fa640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0097da0 0x7f6db00704c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.659+0000 7f6dad7fa640 10 client.114569.objecter ms_handle_connect 0x7f6d940392f0 2023-10-30T12:11:09.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db7125640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:09.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db7125640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0070a00 0x7f6db0070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db7125640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db006a5a0 con 0x7f6db0070a00 2023-10-30T12:11:09.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0070a00 0x7f6db0070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient: get_auth_request con 0x7f6db0070a00 auth_method 0 2023-10-30T12:11:09.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 10 monclient(hunting): handle_auth_done global_id 134400 payload 824 2023-10-30T12:11:09.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 1 -- 172.21.15.50:0/2565464369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6db008fe60 con 0x7f6db0070a00 2023-10-30T12:11:09.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6db4ec1640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0070a00 0x7f6db0070e10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6da400fa30 tx=0x7f6da4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6dad7fa640 10 client.114569.objecter ms_handle_connect 0x7f6db0070a00 2023-10-30T12:11:09.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6dad7fa640 10 client.114569.objecter resend_mon_ops 2023-10-30T12:11:09.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4006ca0 con 0x7f6db0070a00 2023-10-30T12:11:09.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6dad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:09.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.665+0000 7f6dad7fa640 10 monclient: dump: 2023-10-30T12:11:09.723 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.723 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.724 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.724 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.724 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.724 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.724 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.725 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.725 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.725 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7f6da4006e00 con 0x7f6db0070a00 2023-10-30T12:11:09.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6dad7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:09.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6dad7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:09.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0070a00 msgr2=0x7f6db0070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6dad7fa640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0070a00 0x7f6db0070e10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6da400fa30 tx=0x7f6da4001120 comp rx=0 tx=0).stop 2023-10-30T12:11:09.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6dad7fa640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0070a00 msgr2=0x7f6db0070e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:09.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6dad7fa640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6db0070a00 0x7f6db0070e10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.727 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:09.727 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:09.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:09.728 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:11:09.728 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 100, 2023-10-30T12:11:09.728 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:09.728 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:11:09.729 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:09.729 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:09.729 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.729 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 6, 2023-10-30T12:11:09.730 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:09.730 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:09.730 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:09.730 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:09.731 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:09.731 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:09.731 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:09.731 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:09.732 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:09.732 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:09.732 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:09.732 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:09.732 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:09.733 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.733 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:09.733 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:09.733 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:09.734 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:09.734 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:09.734 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:09.734 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:09.734 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:09.735 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:09.735 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:09.735 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:09.736 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:09.736 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.736 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:09.737 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:09.737 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:09.737 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:09.737 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:09.738 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:09.738 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:09.738 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:09.738 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:09.739 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:09.739 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:09.739 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:09.739 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:09.740 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:09.740 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:09.740 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:09.740 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:09.740 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:09.742 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:09.742 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:09.742 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:09.742 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:09.742 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.742 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:09.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.743 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:09.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.743 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:09.743 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:09.744 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:09.744 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:09.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.744 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:09.744 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:09.744 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.745 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:09.745 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:09.745 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.746 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.746 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:09.746 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:09.746 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:09.747 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:09.747 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:09.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:09.748 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:09.748 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:09.748 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:09.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.748 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:09.748 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:09.749 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.749 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:09.749 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:09.750 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.750 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.750 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:09.751 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:09.751 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:09.751 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:09.751 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:09.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:09.752 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:09.752 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:09.752 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:09.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.753 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:09.753 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:09.753 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.755 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:09.755 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:09.755 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:09.755 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:09.755 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:09.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.756 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.756 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:09.756 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.758 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:11:09.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.758 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:09.758 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:09.758 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:11:09.759 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.759 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.759 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:09.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.759 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:09.760 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:09.760 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:11:09.760 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.760 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.761 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:11:09.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.761 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:09.761 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:09.761 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:09.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.762 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.762 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:09.763 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 10 librados: watch_flush enter 2023-10-30T12:11:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 10 librados: watch_flush exit 2023-10-30T12:11:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 20 client.114569.objecter shutdown clearing up homeless session... 2023-10-30T12:11:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 10 client.114569.objecter successfully canceled tick 2023-10-30T12:11:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6d940392f0 msgr2=0x7f6d9403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6d940392f0 0x7f6d9403b7b0 secure :-1 s=READY pgs=4111 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c007fd0 tx=0x7f6d9c009040 comp rx=0 tx=0).stop 2023-10-30T12:11:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 10 monclient: shutdown 2023-10-30T12:11:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6db0139bd0 msgr2=0x7f6db008d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 --2- 172.21.15.50:0/2565464369 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6db0139bd0 0x7f6db008d2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6da8033000 tx=0x7f6da8004a40 comp rx=0 tx=0).stop 2023-10-30T12:11:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db56c2640 1 -- 172.21.15.50:0/2565464369 reap_dead start 2023-10-30T12:11:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db56c2640 1 -- 172.21.15.50:0/2565464369 reap_dead start 2023-10-30T12:11:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 shutdown_connections 2023-10-30T12:11:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 >> 172.21.15.50:0/2565464369 conn(0x7f6db00a1170 msgr2=0x7f6db013a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 shutdown_connections 2023-10-30T12:11:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 -- 172.21.15.50:0/2565464369 wait complete. 2023-10-30T12:11:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.666+0000 7f6db7125640 1 librados: shutdown 2023-10-30T12:11:09.769 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:11:09.769 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:11:09.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:11:09.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.874+0000 7f15534e5640 10 monclient: get_monmap_and_config 2023-10-30T12:11:09.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.874+0000 7f15534e5640 10 monclient: build_initial_monmap 2023-10-30T12:11:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.874+0000 7f15534e5640 10 monclient: monmap: 2023-10-30T12:11:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:09.875653+0000 2023-10-30T12:11:09.878 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:09.875653+0000 2023-10-30T12:11:09.878 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:09.878 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:09.880 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:09.880 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 Processor -- start 2023-10-30T12:11:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 -- start start 2023-10-30T12:11:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient: init 2023-10-30T12:11:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-10-30T12:11:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f154c132970 0x7f154c132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): picked mon.noname-a con 0x7f154c132970 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:09.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): picked mon.noname-b con 0x7f154c0a03d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:09.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f154c070d10 0x7f154c071140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): picked mon.noname-c con 0x7f154c070d10 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:09.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c071680 con 0x7f154c132970 2023-10-30T12:11:09.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0a17d0 con 0x7f154c070d10 2023-10-30T12:11:09.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0a1510 con 0x7f154c0a03d0 2023-10-30T12:11:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f15534e5640 10 monclient(hunting): authenticate will time out at 1515.058818s 2023-10-30T12:11:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f1550a80640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f1551281640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f154c132970 0x7f154c132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f1551281640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f154c132970 0x7f154c132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38582/0 (socket says 172.21.15.50:38582) 2023-10-30T12:11:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.875+0000 7f1550a80640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49640/0 (socket says 172.21.15.50:49640) 2023-10-30T12:11:09.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 -- 172.21.15.50:0/1981277437 learned_addr learned my addr 172.21.15.50:0/1981277437 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:09.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): get_auth_request con 0x7f154c132970 auth_method 0 2023-10-30T12:11:09.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1550a80640 10 monclient(hunting): get_auth_request con 0x7f154c0a03d0 auth_method 0 2023-10-30T12:11:09.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1550a80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1550a80640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1550a80640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551a82640 1 --2- 172.21.15.50:0/1981277437 >> v2:172.21.15.118:3300/0 conn(0x7f154c070d10 0x7f154c071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551a82640 10 monclient(hunting): get_auth_request con 0x7f154c070d10 auth_method 0 2023-10-30T12:11:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551a82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551a82640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551a82640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient(hunting): handle_auth_done global_id 134403 payload 274 2023-10-30T12:11:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 -- 172.21.15.50:0/1981277437 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 msgr2=0x7f154c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:09.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 --2- 172.21.15.50:0/1981277437 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c0707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 -- 172.21.15.50:0/1981277437 >> v2:172.21.15.118:3300/0 conn(0x7f154c070d10 msgr2=0x7f154c071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:09.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 --2- 172.21.15.50:0/1981277437 >> v2:172.21.15.118:3300/0 conn(0x7f154c070d10 0x7f154c071140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient: _finish_hunting 0 2023-10-30T12:11:09.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 monclient: found mon.noname-a 2023-10-30T12:11:09.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:09.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 -- 172.21.15.50:0/1981277437 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c09f680 con 0x7f154c132970 2023-10-30T12:11:09.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551a82640 1 --2- 172.21.15.50:0/1981277437 >> v2:172.21.15.118:3300/0 conn(0x7f154c070d10 0x7f154c071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:11:09.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.876+0000 7f1551281640 1 --2- 172.21.15.50:0/1981277437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c132970 0x7f154c132d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f154c0a1c90 tx=0x7f154802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb211c80093d00e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 1 -- 172.21.15.50:0/1981277437 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548031740 con 0x7f154c132970 2023-10-30T12:11:09.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:09.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: dump: 2023-10-30T12:11:09.903 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.903 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.903 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.903 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.904 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.904 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.904 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.904 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.905 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.905 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: _finish_auth 0 2023-10-30T12:11:09.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: _check_auth_tickets 2023-10-30T12:11:09.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:09.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 1 -- 172.21.15.50:0/1981277437 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15480318a0 con 0x7f154c132970 2023-10-30T12:11:09.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 5 monclient: authenticate success, global_id 134403 2023-10-30T12:11:09.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:09.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:09.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 1 -- 172.21.15.50:0/1981277437 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548031c30 con 0x7f154c132970 2023-10-30T12:11:09.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:09.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f1543fff640 10 monclient: dump: 2023-10-30T12:11:09.908 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.908 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.909 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.909 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.909 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.909 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.910 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.910 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.910 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.911 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:09.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 4 set_mon_vals no callback set 2023-10-30T12:11:09.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 10 monclient: shutdown 2023-10-30T12:11:09.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:09.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 -- 172.21.15.50:0/1981277437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c132970 msgr2=0x7f154c132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 --2- 172.21.15.50:0/1981277437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c132970 0x7f154c132d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f154c0a1c90 tx=0x7f154802b7e0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 -- 172.21.15.50:0/1981277437 shutdown_connections 2023-10-30T12:11:09.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 --2- 172.21.15.50:0/1981277437 >> v2:172.21.15.118:3300/0 conn(0x7f154c070d10 0x7f154c071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 --2- 172.21.15.50:0/1981277437 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 --2- 172.21.15.50:0/1981277437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c132970 0x7f154c132d50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 -- 172.21.15.50:0/1981277437 >> 172.21.15.50:0/1981277437 conn(0x7f154c0a4800 msgr2=0x7f154c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:09.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 -- 172.21.15.50:0/1981277437 shutdown_connections 2023-10-30T12:11:09.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.877+0000 7f15534e5640 1 -- 172.21.15.50:0/1981277437 wait complete. 2023-10-30T12:11:09.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 10 monclient: build_initial_monmap 2023-10-30T12:11:09.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 10 monclient: monmap: 2023-10-30T12:11:09.914 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:09.914 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:09.914 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:09.915 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:09.915 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:09.915 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.915 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:09.915 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:09.915 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:09.916 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 1 librados: starting msgr at 2023-10-30T12:11:09.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 1 librados: starting objecter 2023-10-30T12:11:09.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 1 Processor -- start 2023-10-30T12:11:09.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 1 -- start start 2023-10-30T12:11:09.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 1 librados: setting wanted keys 2023-10-30T12:11:09.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 1 librados: calling monclient init 2023-10-30T12:11:09.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.878+0000 7f15534e5640 10 monclient: init 2023-10-30T12:11:09.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:09.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:11:09.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c070d10 0x7f154c11d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): picked mon.noname-a con 0x7f154c070d10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:09.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c132970 0x7f154c11dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): picked mon.noname-c con 0x7f154c132970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f154c11e1a0 0x7f154c11e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): picked mon.noname-b con 0x7f154c11e1a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c11eb70 con 0x7f154c070d10 2023-10-30T12:11:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c11f910 con 0x7f154c11e1a0 2023-10-30T12:11:09.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c09ca20 con 0x7f154c132970 2023-10-30T12:11:09.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:09.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient(hunting): authenticate will time out at 1515.062190s 2023-10-30T12:11:09.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c132970 0x7f154c11dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c070d10 0x7f154c11d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c070d10 0x7f154c11d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38594/0 (socket says 172.21.15.50:38594) 2023-10-30T12:11:09.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c132970 0x7f154c11dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49644/0 (socket says 172.21.15.50:49644) 2023-10-30T12:11:09.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 -- 172.21.15.50:0/2507486853 learned_addr learned my addr 172.21.15.50:0/2507486853 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:09.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): get_auth_request con 0x7f154c070d10 auth_method 0 2023-10-30T12:11:09.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 10 monclient(hunting): get_auth_request con 0x7f154c132970 auth_method 0 2023-10-30T12:11:09.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551a82640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f154c11e1a0 0x7f154c11e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient(hunting): handle_auth_done global_id 134406 payload 824 2023-10-30T12:11:09.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c132970 msgr2=0x7f154c11dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c132970 0x7f154c11dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f154c11e1a0 msgr2=0x7f154c11e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f154c11e1a0 0x7f154c11e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient: _finish_hunting 0 2023-10-30T12:11:09.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 monclient: found mon.noname-a 2023-10-30T12:11:09.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:09.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 -- 172.21.15.50:0/2507486853 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480094d0 con 0x7f154c070d10 2023-10-30T12:11:09.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551a82640 10 monclient: get_auth_request con 0x7f154c11e1a0 auth_method 0 2023-10-30T12:11:09.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551a82640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f154c11e1a0 0x7f154c11e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:11:09.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1551281640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c070d10 0x7f154c11d720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f154802b820 tx=0x7f15480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 client.?.objecter ms_handle_connect 0x7f154c070d10 2023-10-30T12:11:09.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:09.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548033a40 con 0x7f154c070d10 2023-10-30T12:11:09.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:09.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: dump: 2023-10-30T12:11:09.928 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.928 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.929 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.929 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.929 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.929 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.930 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.930 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.930 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.931 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: _finish_auth 0 2023-10-30T12:11:09.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:11:09.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:09.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 5 monclient: authenticate success, global_id 134406 2023-10-30T12:11:09.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1548033bc0 con 0x7f154c070d10 2023-10-30T12:11:09.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient: _renew_subs 2023-10-30T12:11:09.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c09d7c0 con 0x7f154c070d10 2023-10-30T12:11:09.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:09.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154803c400 con 0x7f154c070d10 2023-10-30T12:11:09.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:09.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f1541ffb640 10 monclient: dump: 2023-10-30T12:11:09.934 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.934 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.934 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.935 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.935 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.935 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.935 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.935 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.936 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.936 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.879+0000 7f15437fe640 4 set_mon_vals no callback set 2023-10-30T12:11:09.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.880+0000 7f15534e5640 10 client.134406.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:09.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.880+0000 7f15534e5640 10 monclient: _renew_subs 2023-10-30T12:11:09.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.880+0000 7f15534e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:09.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.880+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c130260 con 0x7f154c070d10 2023-10-30T12:11:09.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.880+0000 7f15534e5640 1 librados: init done 2023-10-30T12:11:09.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.880+0000 7f15534e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:09.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f15534e5640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c11a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c071a70 con 0x7f154c0a03d0 2023-10-30T12:11:09.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f154803a040 con 0x7f154c070d10 2023-10-30T12:11:09.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c11a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient: get_auth_request con 0x7f154c0a03d0 auth_method 0 2023-10-30T12:11:09.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f15300392f0 0x7f153003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f154806c890 con 0x7f154c070d10 2023-10-30T12:11:09.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 10 client.134406.objecter ms_dispatch 0x7f154c000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:09.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 3 client.134406.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:09.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 3 client.134406.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:09.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 10 monclient(hunting): handle_auth_done global_id 114578 payload 824 2023-10-30T12:11:09.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 20 client.134406.objecter dump_active .. 0 homeless 2023-10-30T12:11:09.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 1 -- 172.21.15.50:0/2507486853 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f154c09f960 con 0x7f154c0a03d0 2023-10-30T12:11:09.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1551a82640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f15300392f0 0x7f153003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1551a82640 10 monclient: get_auth_request con 0x7f15300392f0 auth_method 0 2023-10-30T12:11:09.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 10 client.134406.objecter ms_handle_connect 0x7f154c0a03d0 2023-10-30T12:11:09.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 10 client.134406.objecter resend_mon_ops 2023-10-30T12:11:09.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1550a80640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c11a430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f154c11ef20 tx=0x7f153c011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c004240 con 0x7f154c0a03d0 2023-10-30T12:11:09.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:09.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 10 monclient: dump: 2023-10-30T12:11:09.945 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.945 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.946 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.946 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.946 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.946 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.946 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.947 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1541ffb640 10 client.134406.objecter ms_handle_connect 0x7f15300392f0 2023-10-30T12:11:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.881+0000 7f1551a82640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f15300392f0 0x7f153003b7b0 secure :-1 s=READY pgs=4112 cs=0 l=1 rev1=1 crypto rx=0x7f15440094c0 tx=0x7f154400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.882+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f153c0043a0 con 0x7f154c0a03d0 2023-10-30T12:11:09.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.882+0000 7f1541ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:09.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.882+0000 7f1541ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:09.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.882+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 msgr2=0x7f154c11a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.882+0000 7f1541ffb640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c11a430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f154c11ef20 tx=0x7f153c011cf0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.882+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 msgr2=0x7f154c11a430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:09.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.882+0000 7f1541ffb640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c0a03d0 0x7f154c11a430 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f154c11ef20 tx=0x7f153c011cf0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f15534e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:09.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f15534e5640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c11a970 0x7f154c11ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:09.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f15534e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:09.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c074800 con 0x7f154c11a970 2023-10-30T12:11:09.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c11a970 0x7f154c11ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:09.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient: get_auth_request con 0x7f154c11a970 auth_method 0 2023-10-30T12:11:09.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:09.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:09.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:09.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:09.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:09.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:09.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 10 monclient(hunting): handle_auth_done global_id 114581 payload 824 2023-10-30T12:11:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 1 -- 172.21.15.50:0/2507486853 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f154c04e160 con 0x7f154c11a970 2023-10-30T12:11:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1541ffb640 10 client.134406.objecter ms_handle_connect 0x7f154c11a970 2023-10-30T12:11:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1541ffb640 10 client.134406.objecter resend_mon_ops 2023-10-30T12:11:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1550a80640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c11a970 0x7f154c11ad80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f153c01b000 tx=0x7f153c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c02c070 con 0x7f154c11a970 2023-10-30T12:11:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:09.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1541ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:09.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.888+0000 7f1541ffb640 10 monclient: dump: 2023-10-30T12:11:09.953 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:09.953 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:09.953 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.954 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:09.954 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:09.954 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:09.954 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:09.955 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:09.955 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:09.955 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:09.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f153c024040 con 0x7f154c11a970 2023-10-30T12:11:09.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1541ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:09.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1541ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:09.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c11a970 msgr2=0x7f154c11ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1541ffb640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c11a970 0x7f154c11ad80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f153c01b000 tx=0x7f153c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1541ffb640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c11a970 msgr2=0x7f154c11ad80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:09.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1541ffb640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f154c11a970 0x7f154c11ad80 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f153c01b000 tx=0x7f153c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.957 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:09.957 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:09.957 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:09.957 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:11:09.958 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 100, 2023-10-30T12:11:09.958 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:09.958 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:11:09.958 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:09.958 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:09.959 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.959 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 6, 2023-10-30T12:11:09.959 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:09.959 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:09.959 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:09.959 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:09.960 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:09.960 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:09.960 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:09.960 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:09.960 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:09.961 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:09.961 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:09.961 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:09.961 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:09.961 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.961 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:09.962 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:09.962 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:09.962 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:09.962 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:09.962 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:09.963 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:09.963 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:09.963 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:09.963 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:09.963 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:09.963 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:09.964 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.964 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:09.964 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:09.965 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:09.965 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:09.965 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:09.965 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:09.966 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:09.966 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:09.966 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:09.966 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:09.966 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:09.967 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:09.967 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:09.967 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:09.967 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:09.968 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:09.968 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:09.968 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:09.968 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:09.968 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:09.969 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:09.969 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:09.970 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:09.970 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:09.970 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:09.970 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:09.970 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:09.971 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.971 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:09.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.971 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:09.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.972 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:09.972 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:09.972 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:09.972 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:09.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.973 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:09.973 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:09.973 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:09.974 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:09.975 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.975 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.975 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:09.976 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:09.976 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:09.976 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:09.976 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:09.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.977 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:09.977 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:09.978 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:09.978 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:09.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:09.979 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:09.979 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.979 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:09.979 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:09.981 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.982 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.983 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:09.983 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:09.983 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:09.983 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:09.984 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:09.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.984 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:09.984 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:09.985 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:09.985 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:09.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.985 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:09.985 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:09.986 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.986 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:09.986 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:09.987 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:09.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.987 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.987 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:09.987 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:09.988 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:09.988 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:09.988 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:09.988 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.988 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.989 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:09.989 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:09.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.989 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:09.990 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:09.990 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:09.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.990 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.990 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:11:09.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.991 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:09.991 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:09.991 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:11:09.991 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.991 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:09.992 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:09.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:09.992 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:09.992 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:09.992 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:11:09.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:09.993 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:09.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:09.993 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:09.993 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:09.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 10 librados: watch_flush enter 2023-10-30T12:11:09.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 10 librados: watch_flush exit 2023-10-30T12:11:09.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 20 client.134406.objecter shutdown clearing up homeless session... 2023-10-30T12:11:09.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 10 client.134406.objecter successfully canceled tick 2023-10-30T12:11:09.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f15300392f0 msgr2=0x7f153003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f15300392f0 0x7f153003b7b0 secure :-1 s=READY pgs=4112 cs=0 l=1 rev1=1 crypto rx=0x7f15440094c0 tx=0x7f154400b040 comp rx=0 tx=0).stop 2023-10-30T12:11:09.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 10 monclient: shutdown 2023-10-30T12:11:09.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:09.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c070d10 msgr2=0x7f154c11d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:09.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c070d10 0x7f154c11d720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f154802b820 tx=0x7f15480046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f1551a82640 1 -- 172.21.15.50:0/2507486853 reap_dead start 2023-10-30T12:11:09.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 shutdown_connections 2023-10-30T12:11:09.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 --2- 172.21.15.50:0/2507486853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f154c070d10 0x7f154c11d720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:09.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 >> 172.21.15.50:0/2507486853 conn(0x7f154c0a4800 msgr2=0x7f154c09cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:09.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 shutdown_connections 2023-10-30T12:11:09.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 -- 172.21.15.50:0/2507486853 wait complete. 2023-10-30T12:11:09.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:09.889+0000 7f15534e5640 1 librados: shutdown 2023-10-30T12:11:09.999 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:11:09.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:11:10.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:11:10.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.105+0000 7f6581aa2640 10 monclient: get_monmap_and_config 2023-10-30T12:11:10.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.105+0000 7f6581aa2640 10 monclient: build_initial_monmap 2023-10-30T12:11:10.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.105+0000 7f6581aa2640 10 monclient: monmap: 2023-10-30T12:11:10.107 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:10.107 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.107 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:10.106271+0000 2023-10-30T12:11:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:10.106271+0000 2023-10-30T12:11:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:10.110 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.105+0000 7f6581aa2640 1 Processor -- start 2023-10-30T12:11:10.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.105+0000 7f6581aa2640 1 -- start start 2023-10-30T12:11:10.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.105+0000 7f6581aa2640 10 monclient: init 2023-10-30T12:11:10.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:10.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-10-30T12:11:10.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c0a8010 0x7f657c132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f657c0a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:10.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f657c099a40 0x7f657c132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f657c099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f657c1356d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c0a7190 con 0x7f657c1356d0 2023-10-30T12:11:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c133110 con 0x7f657c099a40 2023-10-30T12:11:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0a83f0 con 0x7f657c0a8010 2023-10-30T12:11:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f6581aa2640 10 monclient(hunting): authenticate will time out at 1515.289426s 2023-10-30T12:11:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f657c099a40 0x7f657c132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c0a8010 0x7f657c132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f657c1356d0 0x7f657c135b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:32976/0 (socket says 172.21.15.50:32976) 2023-10-30T12:11:10.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f657c099a40 0x7f657c132bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38600/0 (socket says 172.21.15.50:38600) 2023-10-30T12:11:10.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c0a8010 0x7f657c132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49668/0 (socket says 172.21.15.50:49668) 2023-10-30T12:11:10.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657bfff640 1 -- 172.21.15.50:0/1886419407 learned_addr learned my addr 172.21.15.50:0/1886419407 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:11:10.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): get_auth_request con 0x7f657c0a8010 auth_method 0 2023-10-30T12:11:10.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 10 monclient(hunting): get_auth_request con 0x7f657c099a40 auth_method 0 2023-10-30T12:11:10.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient(hunting): handle_auth_done global_id 114584 payload 274 2023-10-30T12:11:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 1 -- 172.21.15.50:0/1886419407 >> v2:172.21.15.50:3300/0 conn(0x7f657c099a40 msgr2=0x7f657c132bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 1 --2- 172.21.15.50:0/1886419407 >> v2:172.21.15.50:3300/0 conn(0x7f657c099a40 0x7f657c132bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 1 -- 172.21.15.50:0/1886419407 >> v1:172.21.15.50:6789/0 conn(0x7f657c1356d0 legacy=0x7f657c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:11:10.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 1 monclient: found mon.noname-b 2023-10-30T12:11:10.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:10.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:10.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657b7fe640 1 -- 172.21.15.50:0/1886419407 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c0a7600 con 0x7f657c0a8010 2023-10-30T12:11:10.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657a7fc640 1 -- 172.21.15.50:0/1886419407 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224488883 0 0) 0x7f657c0a7190 con 0x7f657c1356d0 2023-10-30T12:11:10.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:11:10.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.106+0000 7f657affd640 1 --2- 172.21.15.50:0/1886419407 >> v2:172.21.15.50:3300/0 conn(0x7f657c099a40 0x7f657c132bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:11:10.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657b7fe640 1 --2- 172.21.15.50:0/1886419407 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c0a8010 0x7f657c132690 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f656c0097a0 tx=0x7f656c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed3ca11a19faa4ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 1 -- 172.21.15.50:0/1886419407 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c0316f0 con 0x7f657c0a8010 2023-10-30T12:11:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: dump: 2023-10-30T12:11:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.132 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:11:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:11:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 1 -- 172.21.15.50:0/1886419407 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c031850 con 0x7f657c0a8010 2023-10-30T12:11:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 1 -- 172.21.15.50:0/1886419407 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c031be0 con 0x7f657c0a8010 2023-10-30T12:11:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f6581aa2640 5 monclient: authenticate success, global_id 114584 2023-10-30T12:11:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f6581aa2640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f6581aa2640 4 set_mon_vals no callback set 2023-10-30T12:11:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.107+0000 7f657a7fc640 10 monclient: dump: 2023-10-30T12:11:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.136 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.136 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.136 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.136 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 10 monclient: shutdown 2023-10-30T12:11:10.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:10.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 -- 172.21.15.50:0/1886419407 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c0a8010 msgr2=0x7f657c132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 --2- 172.21.15.50:0/1886419407 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c0a8010 0x7f657c132690 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f656c0097a0 tx=0x7f656c02b790 comp rx=0 tx=0).stop 2023-10-30T12:11:10.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 -- 172.21.15.50:0/1886419407 shutdown_connections 2023-10-30T12:11:10.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 --2- 172.21.15.50:0/1886419407 >> v2:172.21.15.50:3300/0 conn(0x7f657c099a40 0x7f657c132bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 --2- 172.21.15.50:0/1886419407 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c0a8010 0x7f657c132690 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 -- 172.21.15.50:0/1886419407 >> 172.21.15.50:0/1886419407 conn(0x7f657c0a4a10 msgr2=0x7f657c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:10.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 -- 172.21.15.50:0/1886419407 shutdown_connections 2023-10-30T12:11:10.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 1 -- 172.21.15.50:0/1886419407 wait complete. 2023-10-30T12:11:10.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 10 monclient: build_initial_monmap 2023-10-30T12:11:10.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.108+0000 7f6581aa2640 10 monclient: monmap: 2023-10-30T12:11:10.139 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:10.139 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:10.139 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:10.140 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:10.140 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:10.140 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.140 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:10.140 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:10.141 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:10.141 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 librados: starting msgr at 2023-10-30T12:11:10.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 librados: starting objecter 2023-10-30T12:11:10.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 Processor -- start 2023-10-30T12:11:10.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 -- start start 2023-10-30T12:11:10.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 librados: setting wanted keys 2023-10-30T12:11:10.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 librados: calling monclient init 2023-10-30T12:11:10.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient: init 2023-10-30T12:11:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f657c0a8010 0x7f657c11dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f657c0a8010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:10.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1356d0 0x7f657c11e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f657c1356d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:10.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c11ea30 0x7f657c1ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f657c11ea30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c121360 con 0x7f657c0a8010 2023-10-30T12:11:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c1215b0 con 0x7f657c1356d0 2023-10-30T12:11:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c11fe20 con 0x7f657c11ea30 2023-10-30T12:11:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.109+0000 7f6581aa2640 10 monclient(hunting): authenticate will time out at 1515.293001s 2023-10-30T12:11:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f657c0a8010 0x7f657c11dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c11ea30 0x7f657c1ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f657c0a8010 0x7f657c11dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38608/0 (socket says 172.21.15.50:38608) 2023-10-30T12:11:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 -- 172.21.15.50:0/2593414788 learned_addr learned my addr 172.21.15.50:0/2593414788 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): get_auth_request con 0x7f657c0a8010 auth_method 0 2023-10-30T12:11:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 10 monclient(hunting): get_auth_request con 0x7f657c11ea30 auth_method 0 2023-10-30T12:11:10.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657affd640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1356d0 0x7f657c11e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657affd640 10 monclient(hunting): get_auth_request con 0x7f657c1356d0 auth_method 0 2023-10-30T12:11:10.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient(hunting): handle_auth_done global_id 134415 payload 824 2023-10-30T12:11:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c11ea30 msgr2=0x7f657c1ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f657c11ea30 0x7f657c1ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1356d0 msgr2=0x7f657c11e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1356d0 0x7f657c11e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:11:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:11:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:11:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 -- 172.21.15.50:0/2593414788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570009400 con 0x7f657c0a8010 2023-10-30T12:11:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657affd640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1356d0 0x7f657c11e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:11:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f657b7fe640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f657c0a8010 0x7f657c11dfb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f656c000c00 tx=0x7f656c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 client.?.objecter ms_handle_connect 0x7f657c0a8010 2023-10-30T12:11:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:10.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c034680 con 0x7f657c0a8010 2023-10-30T12:11:10.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:11:10.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: dump: 2023-10-30T12:11:10.155 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.155 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.156 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.157 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: _finish_auth 0 2023-10-30T12:11:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: _check_auth_tickets 2023-10-30T12:11:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:10.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 5 monclient: authenticate success, global_id 134415 2023-10-30T12:11:10.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 10 monclient: _renew_subs 2023-10-30T12:11:10.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c034820 con 0x7f657c0a8010 2023-10-30T12:11:10.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c009400 con 0x7f657c0a8010 2023-10-30T12:11:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c034bd0 con 0x7f657c0a8010 2023-10-30T12:11:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6563fff640 10 monclient: dump: 2023-10-30T12:11:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.161 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.161 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.161 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.161 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.161 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6579ffb640 4 set_mon_vals no callback set 2023-10-30T12:11:10.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 10 client.134415.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:10.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 10 monclient: _renew_subs 2023-10-30T12:11:10.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:10.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c131400 con 0x7f657c0a8010 2023-10-30T12:11:10.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.110+0000 7f6581aa2640 1 librados: init done 2023-10-30T12:11:10.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f6581aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:10.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f6581aa2640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c099a40 0x7f657c1cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f6581aa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c073fd0 con 0x7f657c099a40 2023-10-30T12:11:10.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f656c03b040 con 0x7f657c0a8010 2023-10-30T12:11:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c099a40 0x7f657c1cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 10 monclient: get_auth_request con 0x7f657c099a40 auth_method 0 2023-10-30T12:11:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.111+0000 7f6563fff640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6568039320 0x7f656803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f656c06c720 con 0x7f657c0a8010 2023-10-30T12:11:10.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 client.134415.objecter ms_dispatch 0x7f657c000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:10.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 3 client.134415.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:10.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 3 client.134415.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:10.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 20 client.134415.objecter dump_active .. 0 homeless 2023-10-30T12:11:10.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f657affd640 10 monclient(hunting): handle_auth_done global_id 154132 payload 824 2023-10-30T12:11:10.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f657affd640 1 -- 172.21.15.50:0/2593414788 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f657c121720 con 0x7f657c099a40 2023-10-30T12:11:10.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f657bfff640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6568039320 0x7f656803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f657bfff640 10 monclient: get_auth_request con 0x7f6568039320 auth_method 0 2023-10-30T12:11:10.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 client.134415.objecter ms_handle_connect 0x7f657c099a40 2023-10-30T12:11:10.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f657affd640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c099a40 0x7f657c1cab00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f657c0a0dc0 tx=0x7f6570011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 client.134415.objecter resend_mon_ops 2023-10-30T12:11:10.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657000fc10 con 0x7f657c099a40 2023-10-30T12:11:10.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:10.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 monclient: dump: 2023-10-30T12:11:10.168 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.168 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.168 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.169 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.170 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 client.134415.objecter ms_handle_connect 0x7f6568039320 2023-10-30T12:11:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f657bfff640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6568039320 0x7f656803b7e0 secure :-1 s=READY pgs=4113 cs=0 l=1 rev1=1 crypto rx=0x7f65740012e0 tx=0x7f65740016b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f6570011e80 con 0x7f657c099a40 2023-10-30T12:11:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c099a40 msgr2=0x7f657c1cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c099a40 0x7f657c1cab00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f657c0a0dc0 tx=0x7f6570011c90 comp rx=0 tx=0).stop 2023-10-30T12:11:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c099a40 msgr2=0x7f657c1cab00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:10.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.112+0000 7f6563fff640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c099a40 0x7f657c1cab00 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f657c0a0dc0 tx=0x7f6570011c90 comp rx=0 tx=0).stop 2023-10-30T12:11:10.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f6581aa2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:10.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f6581aa2640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1cb040 0x7f657c09cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f6581aa2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c09a720 con 0x7f657c1cb040 2023-10-30T12:11:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1cb040 0x7f657c09cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient: get_auth_request con 0x7f657c1cb040 auth_method 0 2023-10-30T12:11:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.119+0000 7f657affd640 10 monclient(hunting): handle_auth_done global_id 154135 payload 824 2023-10-30T12:11:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f657affd640 1 -- 172.21.15.50:0/2593414788 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f657c09a4b0 con 0x7f657c1cb040 2023-10-30T12:11:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 10 client.134415.objecter ms_handle_connect 0x7f657c1cb040 2023-10-30T12:11:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 10 client.134415.objecter resend_mon_ops 2023-10-30T12:11:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f657affd640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1cb040 0x7f657c09cec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f657c0a0dc0 tx=0x7f6570004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570004030 con 0x7f657c1cb040 2023-10-30T12:11:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 10 monclient: dump: 2023-10-30T12:11:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.181 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f6570004190 con 0x7f657c1cb040 2023-10-30T12:11:10.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:10.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1cb040 msgr2=0x7f657c09cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1cb040 0x7f657c09cec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f657c0a0dc0 tx=0x7f6570004730 comp rx=0 tx=0).stop 2023-10-30T12:11:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1cb040 msgr2=0x7f657c09cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6563fff640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f657c1cb040 0x7f657c09cec0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f657c0a0dc0 tx=0x7f6570004730 comp rx=0 tx=0).stop 2023-10-30T12:11:10.183 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:10.183 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:10.184 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:10.184 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:11:10.184 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 100, 2023-10-30T12:11:10.184 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:10.185 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:11:10.185 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:10.185 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:10.186 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:10.186 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 6, 2023-10-30T12:11:10.186 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:10.186 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:10.186 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:10.187 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:10.187 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:10.187 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:10.187 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:10.187 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:10.188 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:10.188 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:10.188 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:10.188 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:10.188 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:10.188 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:10.189 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:10.189 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:10.189 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:10.189 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:10.189 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:10.190 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:10.190 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:10.190 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:10.190 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:10.190 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:10.191 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:10.192 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:10.192 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:10.192 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:10.192 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:10.192 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:10.193 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:10.193 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:10.193 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:10.193 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:10.193 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:10.193 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:10.194 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:10.194 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:10.194 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:10.194 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:10.194 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:10.195 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:10.195 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:10.195 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:10.195 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:10.195 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:10.196 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:10.196 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:10.196 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:10.196 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:10.196 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:10.196 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:10.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.197 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:10.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.197 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:10.197 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:10.198 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:10.198 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:10.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.198 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:10.198 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:10.198 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:10.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:10.199 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:10.199 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:10.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:10.200 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:10.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.200 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:10.200 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:10.201 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:10.201 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:10.201 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:10.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.201 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:10.202 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:10.202 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:10.202 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:10.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.202 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:10.203 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:10.203 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:10.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.203 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:10.203 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:10.204 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:10.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:10.204 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:10.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.204 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:10.205 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:10.205 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:10.205 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:10.205 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:10.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.206 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:10.206 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:10.206 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:10.207 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:10.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.207 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:10.207 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:10.208 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:10.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.208 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:10.208 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:10.209 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:10.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:10.209 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:10.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.210 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:10.210 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:10.210 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:10.210 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:10.210 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:10.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:10.211 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:10.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.211 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:10.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:10.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.212 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:10.212 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:10.212 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:10.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:10.213 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:10.213 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:10.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:10.214 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:10.214 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:10.214 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:10.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:10.215 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:10.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:10.215 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:10.215 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:10.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.120+0000 7f6581aa2640 10 librados: watch_flush enter 2023-10-30T12:11:10.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 10 librados: watch_flush exit 2023-10-30T12:11:10.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 20 client.134415.objecter shutdown clearing up homeless session... 2023-10-30T12:11:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 10 client.134415.objecter successfully canceled tick 2023-10-30T12:11:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6568039320 msgr2=0x7f656803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6568039320 0x7f656803b7e0 secure :-1 s=READY pgs=4113 cs=0 l=1 rev1=1 crypto rx=0x7f65740012e0 tx=0x7f65740016b0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 10 monclient: shutdown 2023-10-30T12:11:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f657c0a8010 msgr2=0x7f657c11dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 --2- 172.21.15.50:0/2593414788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f657c0a8010 0x7f657c11dfb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f656c000c00 tx=0x7f656c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f657bfff640 1 -- 172.21.15.50:0/2593414788 reap_dead start 2023-10-30T12:11:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f657bfff640 1 -- 172.21.15.50:0/2593414788 reap_dead start 2023-10-30T12:11:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 shutdown_connections 2023-10-30T12:11:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 >> 172.21.15.50:0/2593414788 conn(0x7f657c0a4a10 msgr2=0x7f657c069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:10.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 shutdown_connections 2023-10-30T12:11:10.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 -- 172.21.15.50:0/2593414788 wait complete. 2023-10-30T12:11:10.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.121+0000 7f6581aa2640 1 librados: shutdown 2023-10-30T12:11:10.221 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:11:10.222 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:11:10.222 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:11:10.222 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:11:10.222 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:11:10.223 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:11:10.223 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:11:10.235 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:11:10.235 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:11:10.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:10.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.380+0000 7ff8c92af640 10 monclient: get_monmap_and_config 2023-10-30T12:11:10.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.380+0000 7ff8c92af640 10 monclient: build_initial_monmap 2023-10-30T12:11:10.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.380+0000 7ff8c92af640 10 monclient: monmap: 2023-10-30T12:11:10.383 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:10.383 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.383 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:10.381293+0000 2023-10-30T12:11:10.383 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:10.381293+0000 2023-10-30T12:11:10.383 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:10.384 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.384 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:10.384 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:10.384 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:10.385 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.380+0000 7ff8c92af640 1 Processor -- start 2023-10-30T12:11:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.380+0000 7ff8c92af640 1 -- start start 2023-10-30T12:11:10.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.380+0000 7ff8c92af640 10 monclient: init 2023-10-30T12:11:10.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:10.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:11:10.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff8c40a4b60 0x7ff8c41326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient(hunting): picked mon.noname-a con 0x7ff8c40a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:10.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff8c4099a40 0x7ff8c4132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient(hunting): picked mon.noname-c con 0x7ff8c4099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:10.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8c40a2350 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:10.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c4133120 con 0x7ff8c40a2350 2023-10-30T12:11:10.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c40a3d70 con 0x7ff8c40a4b60 2023-10-30T12:11:10.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c40a4f40 con 0x7ff8c4099a40 2023-10-30T12:11:10.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:10.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c92af640 10 monclient(hunting): authenticate will time out at 1515.564323s 2023-10-30T12:11:10.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c37fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7ff8c40a2350 legacy=0x7ff8c40a27e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:10.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c2ffd640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7ff8c40a4b60 msgr2=0x7ff8c41326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:10.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c2ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff8c40a4b60 0x7ff8c41326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:10.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff8c4099a40 0x7ff8c4132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff8c4099a40 0x7ff8c4132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:46198/0 (socket says 172.21.15.50:46198) 2023-10-30T12:11:10.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 1 -- 172.21.15.50:0/436919718 learned_addr learned my addr 172.21.15.50:0/436919718 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:10.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 10 monclient(hunting): get_auth_request con 0x7ff8c4099a40 auth_method 0 2023-10-30T12:11:10.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.381+0000 7ff8c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 10 monclient(hunting): handle_auth_done global_id 154138 payload 274 2023-10-30T12:11:10.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 1 -- 172.21.15.50:0/436919718 >> v2:172.21.15.50:3300/0 conn(0x7ff8c40a4b60 msgr2=0x7ff8c41326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:10.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 1 --2- 172.21.15.50:0/436919718 >> v2:172.21.15.50:3300/0 conn(0x7ff8c40a4b60 0x7ff8c41326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 1 -- 172.21.15.50:0/436919718 >> v1:172.21.15.50:6789/0 conn(0x7ff8c40a2350 legacy=0x7ff8c40a27e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:10.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 10 monclient: _finish_hunting 0 2023-10-30T12:11:10.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 1 monclient: found mon.noname-c 2023-10-30T12:11:10.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:10.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:11:10.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.382+0000 7ff8c27fc640 1 -- 172.21.15.50:0/436919718 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c40a4150 con 0x7ff8c4099a40 2023-10-30T12:11:10.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c27fc640 1 --2- 172.21.15.50:0/436919718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c4099a40 0x7ff8c4132be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4099ed0 tx=0x7ff8b802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b3903d0eb70612c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 1 -- 172.21.15.50:0/436919718 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8031770 con 0x7ff8c4099a40 2023-10-30T12:11:10.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:10.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: dump: 2023-10-30T12:11:10.401 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.401 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:11:10.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:11:10.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:10.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 1 -- 172.21.15.50:0/436919718 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b80318d0 con 0x7ff8c4099a40 2023-10-30T12:11:10.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 5 monclient: authenticate success, global_id 154138 2023-10-30T12:11:10.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:10.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:10.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:10.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 1 -- 172.21.15.50:0/436919718 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8031c60 con 0x7ff8c4099a40 2023-10-30T12:11:10.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 4 set_mon_vals no callback set 2023-10-30T12:11:10.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:10.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c1ffb640 10 monclient: dump: 2023-10-30T12:11:10.406 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.406 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.406 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.406 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.406 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.407 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.407 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.407 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.407 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.407 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 10 monclient: shutdown 2023-10-30T12:11:10.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:10.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 -- 172.21.15.50:0/436919718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c4099a40 msgr2=0x7ff8c4132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 --2- 172.21.15.50:0/436919718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c4099a40 0x7ff8c4132be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4099ed0 tx=0x7ff8b802b810 comp rx=0 tx=0).stop 2023-10-30T12:11:10.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 -- 172.21.15.50:0/436919718 shutdown_connections 2023-10-30T12:11:10.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 --2- 172.21.15.50:0/436919718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c4099a40 0x7ff8c4132be0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 --2- 172.21.15.50:0/436919718 >> v2:172.21.15.50:3300/0 conn(0x7ff8c40a4b60 0x7ff8c41326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 -- 172.21.15.50:0/436919718 >> 172.21.15.50:0/436919718 conn(0x7ff8c40a8060 msgr2=0x7ff8c40a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:10.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 -- 172.21.15.50:0/436919718 shutdown_connections 2023-10-30T12:11:10.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.383+0000 7ff8c92af640 1 -- 172.21.15.50:0/436919718 wait complete. 2023-10-30T12:11:10.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.384+0000 7ff8c92af640 10 monclient: build_initial_monmap 2023-10-30T12:11:10.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.384+0000 7ff8c92af640 10 monclient: monmap: 2023-10-30T12:11:10.410 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:10.410 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:10.410 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:10.410 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:10.411 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:10.411 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.411 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:10.411 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:10.411 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:10.412 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.384+0000 7ff8c92af640 1 librados: starting msgr at 2023-10-30T12:11:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.384+0000 7ff8c92af640 1 librados: starting objecter 2023-10-30T12:11:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.384+0000 7ff8c92af640 1 Processor -- start 2023-10-30T12:11:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 -- start start 2023-10-30T12:11:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 librados: setting wanted keys 2023-10-30T12:11:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 librados: calling monclient init 2023-10-30T12:11:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient: init 2023-10-30T12:11:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8c40a2350 0x7ff8c41ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): picked mon.noname-a con 0x7ff8c40a2350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:10.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c40a4b60 0x7ff8c41cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): picked mon.noname-b con 0x7ff8c40a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:10.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8c411e7b0 0x7ff8c411ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): picked mon.noname-c con 0x7ff8c411e7b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:10.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c2ffd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8c40a2350 msgr2=0x7ff8c41ca8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:10.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4122000 con 0x7ff8c40a2350 2023-10-30T12:11:10.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4120a00 con 0x7ff8c40a4b60 2023-10-30T12:11:10.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:10.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c41224c0 con 0x7ff8c411e7b0 2023-10-30T12:11:10.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:10.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c92af640 10 monclient(hunting): authenticate will time out at 1515.568537s 2023-10-30T12:11:10.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c2ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8c40a2350 0x7ff8c41ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:10.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8c411e7b0 0x7ff8c411ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8c411e7b0 0x7ff8c411ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49678/0 (socket says 172.21.15.50:49678) 2023-10-30T12:11:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 1 -- 172.21.15.50:0/1105366635 learned_addr learned my addr 172.21.15.50:0/1105366635 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): get_auth_request con 0x7ff8c411e7b0 auth_method 0 2023-10-30T12:11:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c40a4b60 0x7ff8c41cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 10 monclient(hunting): get_auth_request con 0x7ff8c40a4b60 auth_method 0 2023-10-30T12:11:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:10.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 10 monclient(hunting): handle_auth_done global_id 114590 payload 824 2023-10-30T12:11:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 1 -- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c40a4b60 msgr2=0x7ff8c41cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.385+0000 7ff8c37fe640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c40a4b60 0x7ff8c41cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 1 -- 172.21.15.50:0/1105366635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8c40a2350 msgr2=0x7ff8c41ca8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:10.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8c40a2350 0x7ff8c41ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 10 monclient: _finish_hunting 0 2023-10-30T12:11:10.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 1 monclient: found mon.noname-c 2023-10-30T12:11:10.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:10.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:10.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 1 -- 172.21.15.50:0/1105366635 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80094d0 con 0x7ff8c411e7b0 2023-10-30T12:11:10.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c37fe640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8c411e7b0 0x7ff8c411ec60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00be50 tx=0x7ff8bc00be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 client.?.objecter ms_handle_connect 0x7ff8c411e7b0 2023-10-30T12:11:10.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:10.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 1 -- 172.21.15.50:0/1105366635 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc016430 con 0x7ff8c411e7b0 2023-10-30T12:11:10.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:10.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: dump: 2023-10-30T12:11:10.423 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.423 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.423 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.423 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.424 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.424 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.424 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.424 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.424 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:10.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:10.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:10.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 5 monclient: authenticate success, global_id 114590 2023-10-30T12:11:10.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 1 -- 172.21.15.50:0/1105366635 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc016590 con 0x7ff8c411e7b0 2023-10-30T12:11:10.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:10.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 1 -- 172.21.15.50:0/1105366635 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc016980 con 0x7ff8c411e7b0 2023-10-30T12:11:10.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 10 monclient: _renew_subs 2023-10-30T12:11:10.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:10.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c4122800 con 0x7ff8c411e7b0 2023-10-30T12:11:10.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:10.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c17fa640 4 set_mon_vals no callback set 2023-10-30T12:11:10.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:10.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8ab7fe640 10 monclient: dump: 2023-10-30T12:11:10.428 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:10.428 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:10.428 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.428 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:10.429 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:10.429 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:10.429 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:10.429 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:10.429 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:10.430 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:10.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 10 client.114590.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:10.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 10 monclient: _renew_subs 2023-10-30T12:11:10.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:10.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c4135d70 con 0x7ff8c411e7b0 2023-10-30T12:11:10.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.386+0000 7ff8c92af640 1 librados: init done 2023-10-30T12:11:10.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8ab7fe640 1 -- 172.21.15.50:0/1105366635 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff8bc012020 con 0x7ff8c411e7b0 2023-10-30T12:11:10.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8c92af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:10.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8c92af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:10.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8c92af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:10.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c4122140 con 0x7ff8c411e7b0 2023-10-30T12:11:10.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8ab7fe640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8b00392d0 0x7ff8b003b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:10.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8ab7fe640 1 -- 172.21.15.50:0/1105366635 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7ff8bc04af90 con 0x7ff8c411e7b0 2023-10-30T12:11:10.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8ab7fe640 10 client.114590.objecter ms_dispatch 0x7ff8c4000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:10.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8ab7fe640 3 client.114590.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:10.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8ab7fe640 3 client.114590.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:10.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8ab7fe640 20 client.114590.objecter dump_active .. 0 homeless 2023-10-30T12:11:10.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8c2ffd640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8b00392d0 0x7ff8b003b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:10.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.387+0000 7ff8c2ffd640 10 monclient: get_auth_request con 0x7ff8b00392d0 auth_method 0 2023-10-30T12:11:10.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.388+0000 7ff8ab7fe640 10 client.114590.objecter ms_handle_connect 0x7ff8b00392d0 2023-10-30T12:11:10.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.388+0000 7ff8c2ffd640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8b00392d0 0x7ff8b003b790 secure :-1 s=READY pgs=4114 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40038b0 tx=0x7ff8b4006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:10.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.392+0000 7ff8ab7fe640 1 -- 172.21.15.50:0/1105366635 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff8c4122140 con 0x7ff8c411e7b0 2023-10-30T12:11:10.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.392+0000 7ff8ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:10.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.392+0000 7ff8ab7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:10.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.637+0000 7ff8c92af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:10.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.637+0000 7ff8c92af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:10.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.637+0000 7ff8c92af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:10.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.637+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff8c4122140 con 0x7ff8c411e7b0 2023-10-30T12:11:10.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.637+0000 7ff8ab7fe640 1 -- 172.21.15.50:0/1105366635 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7ff8bc03c620 con 0x7ff8c411e7b0 2023-10-30T12:11:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.637+0000 7ff8ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.637+0000 7ff8ab7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:10.640 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 10 librados: watch_flush enter 2023-10-30T12:11:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 10 librados: watch_flush exit 2023-10-30T12:11:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 20 client.114590.objecter shutdown clearing up homeless session... 2023-10-30T12:11:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 10 client.114590.objecter successfully canceled tick 2023-10-30T12:11:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8b00392d0 msgr2=0x7ff8b003b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8b00392d0 0x7ff8b003b790 secure :-1 s=READY pgs=4114 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40038b0 tx=0x7ff8b4006d30 comp rx=0 tx=0).stop 2023-10-30T12:11:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 10 monclient: shutdown 2023-10-30T12:11:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8c411e7b0 msgr2=0x7ff8c411ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8c411e7b0 0x7ff8c411ec60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00be50 tx=0x7ff8bc00be80 comp rx=0 tx=0).stop 2023-10-30T12:11:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 shutdown_connections 2023-10-30T12:11:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8b00392d0 0x7ff8b003b790 unknown :-1 s=CLOSED pgs=4114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8c411e7b0 0x7ff8c411ec60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8c40a4b60 0x7ff8c41cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 --2- 172.21.15.50:0/1105366635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8c40a2350 0x7ff8c41ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 >> 172.21.15.50:0/1105366635 conn(0x7ff8c40a8060 msgr2=0x7ff8c41332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 shutdown_connections 2023-10-30T12:11:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 -- 172.21.15.50:0/1105366635 wait complete. 2023-10-30T12:11:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:10.639+0000 7ff8c92af640 1 librados: shutdown 2023-10-30T12:11:13.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:13.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.766+0000 7f926450f640 10 monclient: get_monmap_and_config 2023-10-30T12:11:13.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.766+0000 7f926450f640 10 monclient: build_initial_monmap 2023-10-30T12:11:13.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.766+0000 7f926450f640 10 monclient: monmap: 2023-10-30T12:11:13.769 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:13.769 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:13.769 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:13.767952+0000 2023-10-30T12:11:13.769 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:13.767952+0000 2023-10-30T12:11:13.769 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:13.770 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:13.770 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:13.770 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:13.770 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:13.770 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:13.771 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:13.771 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.767+0000 7f926450f640 1 Processor -- start 2023-10-30T12:11:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.767+0000 7f926450f640 1 -- start start 2023-10-30T12:11:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.767+0000 7f926450f640 10 monclient: init 2023-10-30T12:11:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-10-30T12:11:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f925c131b20 0x7f925c132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient(hunting): picked mon.noname-a con 0x7f925c131b20 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f925c071bc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071fc0 0x7f925c097b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient(hunting): picked mon.noname-b con 0x7f925c071fc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c09a9b0 con 0x7f925c071bc0 2023-10-30T12:11:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c131f00 con 0x7f925c131b20 2023-10-30T12:11:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c072400 con 0x7f925c071fc0 2023-10-30T12:11:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f926450f640 10 monclient(hunting): authenticate will time out at 1518.951169s 2023-10-30T12:11:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f92622ab640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f925c131b20 msgr2=0x7f925c132270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9261aaa640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f925c071bc0 legacy=0x7f925c097410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f92622ab640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f925c131b20 0x7f925c132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:13.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071fc0 0x7f925c097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:13.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071fc0 0x7f925c097b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49686/0 (socket says 172.21.15.50:49686) 2023-10-30T12:11:13.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 -- 172.21.15.50:0/3091304065 learned_addr learned my addr 172.21.15.50:0/3091304065 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:13.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): get_auth_request con 0x7f925c071fc0 auth_method 0 2023-10-30T12:11:13.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:13.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:13.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:13.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:13.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:13.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:13.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient(hunting): handle_auth_done global_id 114593 payload 274 2023-10-30T12:11:13.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 -- 172.21.15.50:0/3091304065 >> v2:172.21.15.50:3300/0 conn(0x7f925c131b20 msgr2=0x7f925c132270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:13.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 --2- 172.21.15.50:0/3091304065 >> v2:172.21.15.50:3300/0 conn(0x7f925c131b20 0x7f925c132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:13.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 -- 172.21.15.50:0/3091304065 >> v1:172.21.15.50:6789/0 conn(0x7f925c071bc0 legacy=0x7f925c097410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:13.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient: _finish_hunting 0 2023-10-30T12:11:13.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 monclient: found mon.noname-b 2023-10-30T12:11:13.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:13.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:13.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.768+0000 7f9262aac640 1 -- 172.21.15.50:0/3091304065 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c09d920 con 0x7f925c071fc0 2023-10-30T12:11:13.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f9262aac640 1 --2- 172.21.15.50:0/3091304065 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071fc0 0x7f925c097b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f925400af60 tx=0x7f925402ced0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afaf0ca45c149673 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:13.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 1 -- 172.21.15.50:0/3091304065 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925402d870 con 0x7f925c071fc0 2023-10-30T12:11:13.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:13.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:13.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: dump: 2023-10-30T12:11:13.789 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:13.789 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:13.790 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.790 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.790 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:13.790 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:13.790 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:13.791 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:13.791 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:13.791 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:13.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: _finish_auth 0 2023-10-30T12:11:13.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: _check_auth_tickets 2023-10-30T12:11:13.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:13.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f926450f640 5 monclient: authenticate success, global_id 114593 2023-10-30T12:11:13.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f926450f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:13.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 1 -- 172.21.15.50:0/3091304065 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925402d9d0 con 0x7f925c071fc0 2023-10-30T12:11:13.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:13.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f926450f640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:13.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f926450f640 4 set_mon_vals no callback set 2023-10-30T12:11:13.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 1 -- 172.21.15.50:0/3091304065 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925402dd60 con 0x7f925c071fc0 2023-10-30T12:11:13.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:13.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:13.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.769+0000 7f92612a9640 10 monclient: dump: 2023-10-30T12:11:13.794 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:13.794 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:13.794 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.794 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.795 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:13.795 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:13.795 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:13.795 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:13.796 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:13.796 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:13.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 10 monclient: shutdown 2023-10-30T12:11:13.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:13.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 -- 172.21.15.50:0/3091304065 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071fc0 msgr2=0x7f925c097b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:13.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 --2- 172.21.15.50:0/3091304065 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071fc0 0x7f925c097b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f925400af60 tx=0x7f925402ced0 comp rx=0 tx=0).stop 2023-10-30T12:11:13.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 -- 172.21.15.50:0/3091304065 shutdown_connections 2023-10-30T12:11:13.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 --2- 172.21.15.50:0/3091304065 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071fc0 0x7f925c097b20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:13.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 --2- 172.21.15.50:0/3091304065 >> v2:172.21.15.50:3300/0 conn(0x7f925c131b20 0x7f925c132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:13.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 -- 172.21.15.50:0/3091304065 >> 172.21.15.50:0/3091304065 conn(0x7f925c0a05c0 msgr2=0x7f925c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:13.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 -- 172.21.15.50:0/3091304065 shutdown_connections 2023-10-30T12:11:13.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 1 -- 172.21.15.50:0/3091304065 wait complete. 2023-10-30T12:11:13.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 10 monclient: build_initial_monmap 2023-10-30T12:11:13.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.770+0000 7f926450f640 10 monclient: monmap: 2023-10-30T12:11:13.798 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:13.798 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:13.799 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:13.799 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:13.799 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:13.799 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:13.799 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:13.800 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:13.800 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:13.800 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:13.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 librados: starting msgr at 2023-10-30T12:11:13.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 librados: starting objecter 2023-10-30T12:11:13.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 Processor -- start 2023-10-30T12:11:13.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 -- start start 2023-10-30T12:11:13.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 librados: setting wanted keys 2023-10-30T12:11:13.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 librados: calling monclient init 2023-10-30T12:11:13.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient: init 2023-10-30T12:11:13.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:13.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:11:13.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071bc0 0x7f925c1c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:13.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): picked mon.noname-c con 0x7f925c071bc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:13.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f925c071fc0 0x7f925c130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:13.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): picked mon.noname-b con 0x7f925c071fc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:13.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925c131b20 0x7f925c131140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:13.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): picked mon.noname-a con 0x7f925c131b20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:13.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:13.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c12dfd0 con 0x7f925c131b20 2023-10-30T12:11:13.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:13.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c12f3f0 con 0x7f925c071fc0 2023-10-30T12:11:13.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:13.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c12dcb0 con 0x7f925c071bc0 2023-10-30T12:11:13.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:13.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f926450f640 10 monclient(hunting): authenticate will time out at 1518.954643s 2023-10-30T12:11:13.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f9262aac640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925c131b20 msgr2=0x7f925c131140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:13.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f9262aac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925c131b20 0x7f925c131140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:13.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071bc0 0x7f925c1c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:13.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071bc0 0x7f925c1c7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49700/0 (socket says 172.21.15.50:49700) 2023-10-30T12:11:13.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 1 -- 172.21.15.50:0/1557961583 learned_addr learned my addr 172.21.15.50:0/1557961583 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:13.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): get_auth_request con 0x7f925c071bc0 auth_method 0 2023-10-30T12:11:13.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:13.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:13.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:13.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:13.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:13.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:13.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 10 monclient(hunting): handle_auth_done global_id 114596 payload 824 2023-10-30T12:11:13.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 1 -- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f925c071fc0 msgr2=0x7f925c130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:13.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f925c071fc0 0x7f925c130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:13.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.771+0000 7f92622ab640 1 -- 172.21.15.50:0/1557961583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925c131b20 msgr2=0x7f925c131140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:13.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f92622ab640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925c131b20 0x7f925c131140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:13.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f92622ab640 10 monclient: _finish_hunting 0 2023-10-30T12:11:13.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f92622ab640 1 monclient: found mon.noname-c 2023-10-30T12:11:13.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f92622ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:13.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f92622ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:13.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f92622ab640 1 -- 172.21.15.50:0/1557961583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925400ac40 con 0x7f925c071bc0 2023-10-30T12:11:13.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f92622ab640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071bc0 0x7f925c1c7d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f925800b500 tx=0x7f925800b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:13.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 client.?.objecter ms_handle_connect 0x7f925c071bc0 2023-10-30T12:11:13.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:13.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 1 -- 172.21.15.50:0/1557961583 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258003e10 con 0x7f925c071bc0 2023-10-30T12:11:13.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:13.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:13.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: dump: 2023-10-30T12:11:13.810 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:13.811 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:13.812 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:13.812 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:13.812 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:13.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: _finish_auth 0 2023-10-30T12:11:13.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:13.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:13.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 1 -- 172.21.15.50:0/1557961583 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9258005070 con 0x7f925c071bc0 2023-10-30T12:11:13.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 5 monclient: authenticate success, global_id 114596 2023-10-30T12:11:13.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 10 monclient: _renew_subs 2023-10-30T12:11:13.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:13.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c131680 con 0x7f925c071bc0 2023-10-30T12:11:13.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:13.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 1 -- 172.21.15.50:0/1557961583 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258005420 con 0x7f925c071bc0 2023-10-30T12:11:13.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:13.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:13.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9252ffd640 10 monclient: dump: 2023-10-30T12:11:13.815 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:13.815 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:13.815 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.815 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:13.816 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:13.816 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:13.816 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:13.816 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:13.816 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:13.817 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 10 client.114596.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 10 monclient: _renew_subs 2023-10-30T12:11:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f9260aa8640 4 set_mon_vals no callback set 2023-10-30T12:11:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c12d420 con 0x7f925c071bc0 2023-10-30T12:11:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.772+0000 7f926450f640 1 librados: init done 2023-10-30T12:11:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9252ffd640 1 -- 172.21.15.50:0/1557961583 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9258005580 con 0x7f925c071bc0 2023-10-30T12:11:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f926450f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f926450f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f926450f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c12d610 con 0x7f925c071bc0 2023-10-30T12:11:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9252ffd640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92400392a0 0x7f924003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9252ffd640 1 -- 172.21.15.50:0/1557961583 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f925804ac10 con 0x7f925c071bc0 2023-10-30T12:11:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9252ffd640 10 client.114596.objecter ms_dispatch 0x7f925c000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9252ffd640 3 client.114596.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9252ffd640 3 client.114596.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9252ffd640 20 client.114596.objecter dump_active .. 0 homeless 2023-10-30T12:11:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9261aaa640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92400392a0 0x7f924003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.773+0000 7f9261aaa640 10 monclient: get_auth_request con 0x7f92400392a0 auth_method 0 2023-10-30T12:11:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.774+0000 7f9252ffd640 10 client.114596.objecter ms_handle_connect 0x7f92400392a0 2023-10-30T12:11:13.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.774+0000 7f9261aaa640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92400392a0 0x7f924003b760 secure :-1 s=READY pgs=4115 cs=0 l=1 rev1=1 crypto rx=0x7f925c070d50 tx=0x7f924c006cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:13.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.778+0000 7f9252ffd640 1 -- 172.21.15.50:0/1557961583 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f925800d360 con 0x7f925c071bc0 2023-10-30T12:11:13.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.778+0000 7f9252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:13.778+0000 7f9252ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:14.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.021+0000 7f926450f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.021+0000 7f926450f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.021+0000 7f926450f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.021+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f925c12f030 con 0x7f925c071bc0 2023-10-30T12:11:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.021+0000 7f9252ffd640 1 -- 172.21.15.50:0/1557961583 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f92580193c0 con 0x7f925c071bc0 2023-10-30T12:11:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.021+0000 7f9252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:14.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.021+0000 7f9252ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:14.024 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 10 librados: watch_flush enter 2023-10-30T12:11:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 10 librados: watch_flush exit 2023-10-30T12:11:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 20 client.114596.objecter shutdown clearing up homeless session... 2023-10-30T12:11:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 10 client.114596.objecter successfully canceled tick 2023-10-30T12:11:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92400392a0 msgr2=0x7f924003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92400392a0 0x7f924003b760 secure :-1 s=READY pgs=4115 cs=0 l=1 rev1=1 crypto rx=0x7f925c070d50 tx=0x7f924c006cf0 comp rx=0 tx=0).stop 2023-10-30T12:11:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 10 monclient: shutdown 2023-10-30T12:11:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071bc0 msgr2=0x7f925c1c7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071bc0 0x7f925c1c7d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f925800b500 tx=0x7f925800b9c0 comp rx=0 tx=0).stop 2023-10-30T12:11:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 shutdown_connections 2023-10-30T12:11:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92400392a0 0x7f924003b760 unknown :-1 s=CLOSED pgs=4115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925c131b20 0x7f925c131140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f925c071fc0 0x7f925c130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 --2- 172.21.15.50:0/1557961583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f925c071bc0 0x7f925c1c7d40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 >> 172.21.15.50:0/1557961583 conn(0x7f925c0a05c0 msgr2=0x7f925c070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 shutdown_connections 2023-10-30T12:11:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 -- 172.21.15.50:0/1557961583 wait complete. 2023-10-30T12:11:14.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:14.023+0000 7f926450f640 1 librados: shutdown 2023-10-30T12:11:17.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.153+0000 7f981327c640 10 monclient: get_monmap_and_config 2023-10-30T12:11:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.153+0000 7f981327c640 10 monclient: build_initial_monmap 2023-10-30T12:11:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.153+0000 7f981327c640 10 monclient: monmap: 2023-10-30T12:11:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:17.153716+0000 2023-10-30T12:11:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:17.153716+0000 2023-10-30T12:11:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:17.156 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:17.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 1 Processor -- start 2023-10-30T12:11:17.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 1 -- start start 2023-10-30T12:11:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 10 monclient: init 2023-10-30T12:11:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-10-30T12:11:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f980c09f5b0 0x7f980c099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 10 monclient(hunting): picked mon.noname-c con 0x7f980c09f5b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f980c096080 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f980c096ba0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.154+0000 7f981327c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c098810 con 0x7f980c096ba0 2023-10-30T12:11:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f981327c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c06aef0 con 0x7f980c096080 2023-10-30T12:11:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f981327c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f981327c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c09a1a0 con 0x7f980c09f5b0 2023-10-30T12:11:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f981327c640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f981327c640 10 monclient(hunting): authenticate will time out at 1522.337042s 2023-10-30T12:11:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811819640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f980c096ba0 legacy=0x7f980c13ab40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f980c09f5b0 0x7f980c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f980c09f5b0 0x7f980c099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:46230/0 (socket says 172.21.15.50:46230) 2023-10-30T12:11:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 1 -- 172.21.15.50:0/1265076934 learned_addr learned my addr 172.21.15.50:0/1265076934 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): get_auth_request con 0x7f980c09f5b0 auth_method 0 2023-10-30T12:11:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9810817640 1 --1- 172.21.15.50:0/1265076934 >> v1:172.21.15.118:6789/0 conn(0x7f980c096080 0x7f980c096490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:39894/0 (socket says 172.21.15.50:39894) 2023-10-30T12:11:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient(hunting): handle_auth_done global_id 154144 payload 274 2023-10-30T12:11:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 1 -- 172.21.15.50:0/1265076934 >> v1:172.21.15.118:6789/0 conn(0x7f980c096080 legacy=0x7f980c096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 1 -- 172.21.15.50:0/1265076934 >> v1:172.21.15.50:6789/0 conn(0x7f980c096ba0 legacy=0x7f980c13ab40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient: _finish_hunting 0 2023-10-30T12:11:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 1 monclient: found mon.noname-c 2023-10-30T12:11:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:11:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.155+0000 7f9811018640 1 -- 172.21.15.50:0/1265076934 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c098af0 con 0x7f980c09f5b0 2023-10-30T12:11:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9811018640 1 --2- 172.21.15.50:0/1265076934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 0x7f980c099b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9808009850 tx=0x7f980802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4399a9572c4ebf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 1 -- 172.21.15.50:0/1265076934 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980802e070 con 0x7f980c09f5b0 2023-10-30T12:11:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: dump: 2023-10-30T12:11:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:17.175 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: _finish_auth 0 2023-10-30T12:11:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: _check_auth_tickets 2023-10-30T12:11:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 1 -- 172.21.15.50:0/1265076934 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9808004030 con 0x7f980c09f5b0 2023-10-30T12:11:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 1 -- 172.21.15.50:0/1265076934 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808004420 con 0x7f980c09f5b0 2023-10-30T12:11:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f9803fff640 10 monclient: dump: 2023-10-30T12:11:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:17.179 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f981327c640 5 monclient: authenticate success, global_id 154144 2023-10-30T12:11:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f981327c640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.156+0000 7f981327c640 4 set_mon_vals no callback set 2023-10-30T12:11:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 10 monclient: shutdown 2023-10-30T12:11:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 1 -- 172.21.15.50:0/1265076934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 msgr2=0x7f980c099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 1 --2- 172.21.15.50:0/1265076934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 0x7f980c099b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9808009850 tx=0x7f980802b840 comp rx=0 tx=0).stop 2023-10-30T12:11:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 1 -- 172.21.15.50:0/1265076934 shutdown_connections 2023-10-30T12:11:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 1 --2- 172.21.15.50:0/1265076934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 0x7f980c099b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 1 -- 172.21.15.50:0/1265076934 >> 172.21.15.50:0/1265076934 conn(0x7f980c0a05c0 msgr2=0x7f980c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 1 -- 172.21.15.50:0/1265076934 shutdown_connections 2023-10-30T12:11:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 1 -- 172.21.15.50:0/1265076934 wait complete. 2023-10-30T12:11:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 10 monclient: build_initial_monmap 2023-10-30T12:11:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.157+0000 7f981327c640 10 monclient: monmap: 2023-10-30T12:11:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:17.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 librados: starting msgr at 2023-10-30T12:11:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 librados: starting objecter 2023-10-30T12:11:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 Processor -- start 2023-10-30T12:11:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 -- start start 2023-10-30T12:11:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 librados: setting wanted keys 2023-10-30T12:11:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 librados: calling monclient init 2023-10-30T12:11:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient: init 2023-10-30T12:11:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:11:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f980c096080 0x7f980c095130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): picked mon.noname-a con 0x7f980c096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f980c096ba0 0x7f980c095670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): picked mon.noname-c con 0x7f980c096ba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 0x7f980c06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): picked mon.noname-b con 0x7f980c09f5b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f9811018640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f980c096080 msgr2=0x7f980c095130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c070780 con 0x7f980c096080 2023-10-30T12:11:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c08b6c0 con 0x7f980c09f5b0 2023-10-30T12:11:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c072010 con 0x7f980c096ba0 2023-10-30T12:11:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f981327c640 10 monclient(hunting): authenticate will time out at 1522.340952s 2023-10-30T12:11:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.158+0000 7f9811018640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f980c096080 0x7f980c095130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f980c096ba0 0x7f980c095670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f980c096ba0 0x7f980c095670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49708/0 (socket says 172.21.15.50:49708) 2023-10-30T12:11:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 -- 172.21.15.50:0/1729506332 learned_addr learned my addr 172.21.15.50:0/1729506332 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): get_auth_request con 0x7f980c096ba0 auth_method 0 2023-10-30T12:11:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 0x7f980c06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 10 monclient(hunting): get_auth_request con 0x7f980c09f5b0 auth_method 0 2023-10-30T12:11:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9811819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient(hunting): handle_auth_done global_id 114599 payload 824 2023-10-30T12:11:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 -- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 msgr2=0x7f980c06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 0x7f980c06ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 -- 172.21.15.50:0/1729506332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f980c096080 msgr2=0x7f980c095130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f980c096080 0x7f980c095130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient: _finish_hunting 0 2023-10-30T12:11:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 monclient: found mon.noname-c 2023-10-30T12:11:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 -- 172.21.15.50:0/1729506332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9808009460 con 0x7f980c096ba0 2023-10-30T12:11:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9810817640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f980c096ba0 0x7f980c095670 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00c440 tx=0x7f97fc00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 client.?.objecter ms_handle_connect 0x7f980c096ba0 2023-10-30T12:11:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 1 -- 172.21.15.50:0/1729506332 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0074e0 con 0x7f980c096ba0 2023-10-30T12:11:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: dump: 2023-10-30T12:11:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:17.197 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: _finish_auth 0 2023-10-30T12:11:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:11:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f981327c640 5 monclient: authenticate success, global_id 114599 2023-10-30T12:11:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 1 -- 172.21.15.50:0/1729506332 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc007640 con 0x7f980c096ba0 2023-10-30T12:11:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 1 -- 172.21.15.50:0/1729506332 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0049e0 con 0x7f980c096ba0 2023-10-30T12:11:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f981327c640 10 monclient: _renew_subs 2023-10-30T12:11:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f981327c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c070010 con 0x7f980c096ba0 2023-10-30T12:11:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f98037fe640 4 set_mon_vals no callback set 2023-10-30T12:11:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.159+0000 7f9801ffb640 10 monclient: dump: 2023-10-30T12:11:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:17.201 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.160+0000 7f981327c640 10 client.114599.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.160+0000 7f981327c640 10 monclient: _renew_subs 2023-10-30T12:11:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.160+0000 7f981327c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.160+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c06fa90 con 0x7f980c096ba0 2023-10-30T12:11:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.160+0000 7f981327c640 1 librados: init done 2023-10-30T12:11:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.160+0000 7f9801ffb640 1 -- 172.21.15.50:0/1729506332 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f97fc004b40 con 0x7f980c096ba0 2023-10-30T12:11:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.160+0000 7f9801ffb640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f97f00392a0 0x7f97f003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9801ffb640 1 -- 172.21.15.50:0/1729506332 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f97fc04a5a0 con 0x7f980c096ba0 2023-10-30T12:11:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9801ffb640 10 client.114599.objecter ms_dispatch 0x7f980c000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9801ffb640 3 client.114599.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9801ffb640 3 client.114599.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f981327c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f981327c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f981327c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c04e160 con 0x7f980c096ba0 2023-10-30T12:11:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9811018640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f97f00392a0 0x7f97f003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9811018640 10 monclient: get_auth_request con 0x7f97f00392a0 auth_method 0 2023-10-30T12:11:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9801ffb640 20 client.114599.objecter dump_active .. 0 homeless 2023-10-30T12:11:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9801ffb640 10 client.114599.objecter ms_handle_connect 0x7f97f00392a0 2023-10-30T12:11:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.161+0000 7f9811018640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f97f00392a0 0x7f97f003b760 secure :-1 s=READY pgs=4116 cs=0 l=1 rev1=1 crypto rx=0x7f9808034000 tx=0x7f9808008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.165+0000 7f9801ffb640 1 -- 172.21.15.50:0/1729506332 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f97fc0192c0 con 0x7f980c096ba0 2023-10-30T12:11:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.165+0000 7f9801ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.165+0000 7f9801ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.414+0000 7f981327c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.414+0000 7f981327c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.415+0000 7f981327c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.415+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f980c0a3010 con 0x7f980c096ba0 2023-10-30T12:11:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.415+0000 7f9801ffb640 1 -- 172.21.15.50:0/1729506332 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f97fc01aab0 con 0x7f980c096ba0 2023-10-30T12:11:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.415+0000 7f9801ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.415+0000 7f9801ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:17.417 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 10 librados: watch_flush enter 2023-10-30T12:11:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 10 librados: watch_flush exit 2023-10-30T12:11:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 20 client.114599.objecter shutdown clearing up homeless session... 2023-10-30T12:11:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 10 client.114599.objecter successfully canceled tick 2023-10-30T12:11:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f97f00392a0 msgr2=0x7f97f003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f97f00392a0 0x7f97f003b760 secure :-1 s=READY pgs=4116 cs=0 l=1 rev1=1 crypto rx=0x7f9808034000 tx=0x7f9808008030 comp rx=0 tx=0).stop 2023-10-30T12:11:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 10 monclient: shutdown 2023-10-30T12:11:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f980c096ba0 msgr2=0x7f980c095670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f980c096ba0 0x7f980c095670 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00c440 tx=0x7f97fc00c900 comp rx=0 tx=0).stop 2023-10-30T12:11:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 shutdown_connections 2023-10-30T12:11:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f97f00392a0 0x7f97f003b760 unknown :-1 s=CLOSED pgs=4116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980c09f5b0 0x7f980c06ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f980c096ba0 0x7f980c095670 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 --2- 172.21.15.50:0/1729506332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f980c096080 0x7f980c095130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 >> 172.21.15.50:0/1729506332 conn(0x7f980c0a05c0 msgr2=0x7f980c135770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:17.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 shutdown_connections 2023-10-30T12:11:17.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 -- 172.21.15.50:0/1729506332 wait complete. 2023-10-30T12:11:17.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:17.417+0000 7f981327c640 1 librados: shutdown 2023-10-30T12:11:20.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:20.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.544+0000 7f5c90421640 10 monclient: get_monmap_and_config 2023-10-30T12:11:20.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.544+0000 7f5c90421640 10 monclient: build_initial_monmap 2023-10-30T12:11:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.544+0000 7f5c90421640 10 monclient: monmap: 2023-10-30T12:11:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:20.546022+0000 2023-10-30T12:11:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:20.546022+0000 2023-10-30T12:11:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:20.549 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:20.549 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.545+0000 7f5c90421640 1 Processor -- start 2023-10-30T12:11:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.545+0000 7f5c90421640 1 -- start start 2023-10-30T12:11:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.545+0000 7f5c90421640 10 monclient: init 2023-10-30T12:11:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T12:11:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 0x7f5c8809bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): picked mon.noname-a con 0x7f5c88098520 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): picked mon.noname-c con 0x7f5c88098900 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): picked mon.noname-b con 0x7f5c88132740 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8809ad30 con 0x7f5c88098520 2023-10-30T12:11:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8809c460 con 0x7f5c88098900 2023-10-30T12:11:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8809b260 con 0x7f5c88132740 2023-10-30T12:11:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c90421640 10 monclient(hunting): authenticate will time out at 1525.729272s 2023-10-30T12:11:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e1bd640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 msgr2=0x7f5c8809bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e1bd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 0x7f5c8809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56644/0 (socket says 172.21.15.50:56644) 2023-10-30T12:11:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 learned_addr learned my addr 172.21.15.50:0/873498619 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request con 0x7f5c88132740 auth_method 0 2023-10-30T12:11:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:20.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:20.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:11:20.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:20.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:20.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request con 0x7f5c88098900 auth_method 0 2023-10-30T12:11:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:11:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.546+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:20.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 msgr2=0x7f5c8809bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:20.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e1bd640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 0x7f5c8809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:11:20.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:20.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request con 0x7f5c88132740 auth_method 0 2023-10-30T12:11:20.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:20.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:20.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:20.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:11:20.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:20.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:20.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:11:20.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:20.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request con 0x7f5c88098900 auth_method 0 2023-10-30T12:11:20.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:20.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:20.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:20.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:11:20.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:20.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:20.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:20.747+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:11:21.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 msgr2=0x7f5c8809bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:21.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e1bd640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 0x7f5c8809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:11:21.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:21.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request con 0x7f5c88132740 auth_method 0 2023-10-30T12:11:21.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:21.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:21.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:21.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:11:21.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:21.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:21.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:11:21.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:21.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request con 0x7f5c88098900 auth_method 0 2023-10-30T12:11:21.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:21.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:21.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:21.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:11:21.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:21.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:21.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.148+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:11:21.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 msgr2=0x7f5c8809bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:21.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e1bd640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 0x7f5c8809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:11:21.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:21.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request con 0x7f5c88132740 auth_method 0 2023-10-30T12:11:21.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:21.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:21.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:21.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:11:21.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:21.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:21.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:11:21.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request con 0x7f5c88098900 auth_method 0 2023-10-30T12:11:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:21.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:11:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:21.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:21.949+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:11:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 10 monclient(hunting): tick 2023-10-30T12:11:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:11:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 monclient(hunting): continuing hunt 2023-10-30T12:11:23.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:11:23.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 msgr2=0x7f5c88132bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:23.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c88132740 0x7f5c88132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:23.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 msgr2=0x7f5c88098d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:23.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.118:3300/0 conn(0x7f5c88098900 0x7f5c88098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 -- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 msgr2=0x7f5c8809bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 --2- 172.21.15.50:0/873498619 >> v2:172.21.15.50:3300/0 conn(0x7f5c88098520 0x7f5c8809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 10 monclient: _add_conns ranks=[1,0] 2023-10-30T12:11:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c6c003690 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c6c0073c0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 -- 172.21.15.50:0/873498619 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c88072120 con 0x7f5c6c0073c0 2023-10-30T12:11:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 1 -- 172.21.15.50:0/873498619 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c00c140 con 0x7f5c6c003690 2023-10-30T12:11:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c7ffff640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.546+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v1:172.21.15.50:6789/0 conn(0x7f5c6c0073c0 legacy=0x7f5c6c009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:23.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:23.747+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v1:172.21.15.50:6789/0 conn(0x7f5c6c0073c0 legacy=0x7f5c6c009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:24.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.148+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> v1:172.21.15.50:6789/0 conn(0x7f5c6c0073c0 legacy=0x7f5c6c009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.654+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461267597 0 0) 0x7f5c6c00c140 con 0x7f5c6c003690 2023-10-30T12:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.654+0000 7f5c8d1bb640 10 monclient(hunting): _negotiate 2023-10-30T12:11:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.654+0000 7f5c8d1bb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.654+0000 7f5c8d1bb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.654+0000 7f5c8d1bb640 10 monclient(hunting): my global_id is 154150 2023-10-30T12:11:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.654+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c74005730 con 0x7f5c6c003690 2023-10-30T12:11:24.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544186125 0 0) 0x7f5c74005730 con 0x7f5c6c003690 2023-10-30T12:11:24.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 >> v1:172.21.15.50:6789/0 conn(0x7f5c6c0073c0 legacy=0x7f5c6c009880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:24.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: _finish_hunting 0 2023-10-30T12:11:24.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:11:24.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:24.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:11:24.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c00d340 con 0x7f5c6c003690 2023-10-30T12:11:24.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: _finish_auth 0 2023-10-30T12:11:24.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: _check_auth_tickets 2023-10-30T12:11:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f5c84004290 con 0x7f5c6c003690 2023-10-30T12:11:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:11:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: dump: 2023-10-30T12:11:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:24.665 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:11:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 >> v1:172.21.15.118:6789/0 conn(0x7f5c6c003690 legacy=0x7f5c6c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8e9be640 1 -- 172.21.15.50:0/873498619 reap_dead start 2023-10-30T12:11:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c74005b80 0x7f5c74005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): picked mon.a con 0x7f5c74005b80 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c74006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): picked mon.b con 0x7f5c740069d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c74005b80 msgr2=0x7f5c74005f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c740074b0 0x7f5c7400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): picked mon.c con 0x7f5c740074b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8e1bd640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c74005b80 0x7f5c74005f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7400e3a0 con 0x7f5c74005b80 2023-10-30T12:11:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7400e510 con 0x7f5c740069d0 2023-10-30T12:11:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:24.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7400e6d0 con 0x7f5c740074b0 2023-10-30T12:11:24.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:24.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8d1bb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:11:24.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.656+0000 7f5c8e9be640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c740074b0 0x7f5c7400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:24.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c74006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:24.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request con 0x7f5c740074b0 auth_method 0 2023-10-30T12:11:24.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8e9be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:24.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8e9be640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:24.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8e9be640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:24.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request con 0x7f5c740069d0 auth_method 0 2023-10-30T12:11:24.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:24.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:24.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:24.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:24.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:24.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:11:24.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8e9be640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:24.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8e9be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:24.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8e9be640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:11:24.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient(hunting): handle_auth_done global_id 154150 payload 293 2023-10-30T12:11:24.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c740074b0 msgr2=0x7f5c7400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:24.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c740074b0 0x7f5c7400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c74005b80 msgr2=0x7f5c74005f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:24.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c74005b80 0x7f5c74005f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient: _finish_hunting 0 2023-10-30T12:11:24.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 1 monclient: found mon.b 2023-10-30T12:11:24.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:24.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:11:24.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 1 -- 172.21.15.50:0/873498619 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c74005730 con 0x7f5c740069d0 2023-10-30T12:11:24.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient: _finish_auth 0 2023-10-30T12:11:24.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 10 monclient: _check_auth_tickets 2023-10-30T12:11:24.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c8d9bc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:24.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c90421640 5 monclient: authenticate success, global_id 154150 2023-10-30T12:11:24.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.657+0000 7f5c90421640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:24.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c74006e50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c780044f0 tx=0x7f5c74007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54b9b1da03b15485 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:24.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7803a050 con 0x7f5c740069d0 2023-10-30T12:11:24.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:24.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:24.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 10 monclient: dump: 2023-10-30T12:11:24.678 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:24.678 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:24.678 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.678 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.679 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:24.679 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:24.679 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:24.679 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:24.679 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:24.679 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:24.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c90421640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:24.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7803e5f0 con 0x7f5c740069d0 2023-10-30T12:11:24.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:24.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 1 -- 172.21.15.50:0/873498619 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7803e9a0 con 0x7f5c740069d0 2023-10-30T12:11:24.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c90421640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:24.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c90421640 4 set_mon_vals no callback set 2023-10-30T12:11:24.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:24.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:24.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.658+0000 7f5c8d1bb640 10 monclient: dump: 2023-10-30T12:11:24.682 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:24.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:24.682 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.682 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.682 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:24.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:24.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:24.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:24.683 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:24.683 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:24.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 10 monclient: shutdown 2023-10-30T12:11:24.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:24.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 -- 172.21.15.50:0/873498619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 msgr2=0x7f5c74006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:24.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c74006e50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c780044f0 tx=0x7f5c74007a00 comp rx=0 tx=0).stop 2023-10-30T12:11:24.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 -- 172.21.15.50:0/873498619 shutdown_connections 2023-10-30T12:11:24.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c740074b0 0x7f5c7400db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c74006e50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 --2- 172.21.15.50:0/873498619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c74005b80 0x7f5c74005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 -- 172.21.15.50:0/873498619 >> 172.21.15.50:0/873498619 conn(0x7f5c880a05c0 msgr2=0x7f5c8809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:24.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 -- 172.21.15.50:0/873498619 shutdown_connections 2023-10-30T12:11:24.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.659+0000 7f5c90421640 1 -- 172.21.15.50:0/873498619 wait complete. 2023-10-30T12:11:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 10 monclient: build_initial_monmap 2023-10-30T12:11:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 10 monclient: monmap: 2023-10-30T12:11:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:24.693 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 1 librados: starting msgr at 2023-10-30T12:11:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 1 librados: starting objecter 2023-10-30T12:11:24.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 1 Processor -- start 2023-10-30T12:11:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 1 -- start start 2023-10-30T12:11:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 1 librados: setting wanted keys 2023-10-30T12:11:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 1 librados: calling monclient init 2023-10-30T12:11:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.660+0000 7f5c90421640 10 monclient: init 2023-10-30T12:11:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:11:24.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c74005b80 0x7f5c8808afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:24.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): picked mon.noname-c con 0x7f5c74005b80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:24.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c8808b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:24.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): picked mon.noname-b con 0x7f5c740069d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:24.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c740074b0 0x7f5c8808ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:24.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): picked mon.noname-a con 0x7f5c740074b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:24.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:24.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88094190 con 0x7f5c740074b0 2023-10-30T12:11:24.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:24.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8808c740 con 0x7f5c740069d0 2023-10-30T12:11:24.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:24.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88071df0 con 0x7f5c74005b80 2023-10-30T12:11:24.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:24.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c90421640 10 monclient(hunting): authenticate will time out at 1529.844546s 2023-10-30T12:11:24.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e9be640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c740074b0 msgr2=0x7f5c8808ba50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:24.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e9be640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c740074b0 0x7f5c8808ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:24.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c74005b80 0x7f5c8808afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:24.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c74005b80 0x7f5c8808afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56684/0 (socket says 172.21.15.50:56684) 2023-10-30T12:11:24.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/241230583 learned_addr learned my addr 172.21.15.50:0/241230583 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:24.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 10 monclient(hunting): get_auth_request con 0x7f5c74005b80 auth_method 0 2023-10-30T12:11:24.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8e1bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c8808b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request con 0x7f5c740069d0 auth_method 0 2023-10-30T12:11:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8d9bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.661+0000 7f5c8d9bc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 10 monclient(hunting): handle_auth_done global_id 114602 payload 824 2023-10-30T12:11:24.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 msgr2=0x7f5c8808b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:24.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c8808b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/241230583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c740074b0 msgr2=0x7f5c8808ba50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:24.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c740074b0 0x7f5c8808ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 10 monclient: _finish_hunting 0 2023-10-30T12:11:24.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 1 monclient: found mon.noname-c 2023-10-30T12:11:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 1 -- 172.21.15.50:0/241230583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880a2820 con 0x7f5c74005b80 2023-10-30T12:11:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c8808b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:11:24.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8e1bd640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c74005b80 0x7f5c8808afd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400b750 tx=0x7f5c8400bc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:24.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 client.?.objecter ms_handle_connect 0x7f5c74005b80 2023-10-30T12:11:24.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:24.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 1 -- 172.21.15.50:0/241230583 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400c930 con 0x7f5c74005b80 2023-10-30T12:11:24.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:24.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:24.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: dump: 2023-10-30T12:11:24.704 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:24.704 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:24.704 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.704 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:24.705 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:24.705 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: _finish_auth 0 2023-10-30T12:11:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:24.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 1 -- 172.21.15.50:0/241230583 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8400ca90 con 0x7f5c74005b80 2023-10-30T12:11:24.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:24.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 1 -- 172.21.15.50:0/241230583 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400ce80 con 0x7f5c74005b80 2023-10-30T12:11:24.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 5 monclient: authenticate success, global_id 114602 2023-10-30T12:11:24.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 10 monclient: _renew_subs 2023-10-30T12:11:24.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:24.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8808e1d0 con 0x7f5c74005b80 2023-10-30T12:11:24.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c8c9ba640 4 set_mon_vals no callback set 2023-10-30T12:11:24.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:24.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:24.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c7effd640 10 monclient: dump: 2023-10-30T12:11:24.709 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:24.709 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:24.709 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.709 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:24.710 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:24.710 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:24.710 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:24.710 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:24.710 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:24.711 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:24.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 10 client.114602.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:24.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 10 monclient: _renew_subs 2023-10-30T12:11:24.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:24.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8808c910 con 0x7f5c74005b80 2023-10-30T12:11:24.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.662+0000 7f5c90421640 1 librados: init done 2023-10-30T12:11:24.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c7effd640 1 -- 172.21.15.50:0/241230583 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5c84012020 con 0x7f5c74005b80 2023-10-30T12:11:24.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c7effd640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c6c039ad0 0x7f5c6c03bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:24.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c7effd640 1 -- 172.21.15.50:0/241230583 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f5c8404a330 con 0x7f5c74005b80 2023-10-30T12:11:24.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c90421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:24.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c7effd640 10 client.114602.objecter ms_dispatch 0x7f5c88000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:24.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c7effd640 3 client.114602.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:24.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c7effd640 3 client.114602.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:24.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c90421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:24.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c90421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:24.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8804e160 con 0x7f5c74005b80 2023-10-30T12:11:24.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c7effd640 20 client.114602.objecter dump_active .. 0 homeless 2023-10-30T12:11:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.663+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c6c039ad0 0x7f5c6c03bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.664+0000 7f5c8d9bc640 10 monclient: get_auth_request con 0x7f5c6c039ad0 auth_method 0 2023-10-30T12:11:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.664+0000 7f5c8d9bc640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c6c039ad0 0x7f5c6c03bf90 secure :-1 s=READY pgs=4117 cs=0 l=1 rev1=1 crypto rx=0x7f5c78040000 tx=0x7f5c78039000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.664+0000 7f5c7effd640 10 client.114602.objecter ms_handle_connect 0x7f5c6c039ad0 2023-10-30T12:11:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.668+0000 7f5c7effd640 1 -- 172.21.15.50:0/241230583 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5c84014eb0 con 0x7f5c74005b80 2023-10-30T12:11:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.668+0000 7f5c7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:24.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.668+0000 7f5c7effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.921+0000 7f5c90421640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.921+0000 7f5c90421640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.921+0000 7f5c90421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.921+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c880a23d0 con 0x7f5c74005b80 2023-10-30T12:11:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.922+0000 7f5c7effd640 1 -- 172.21.15.50:0/241230583 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f5c84014aa0 con 0x7f5c74005b80 2023-10-30T12:11:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.922+0000 7f5c7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.922+0000 7f5c7effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:24.925 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:24.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 10 librados: watch_flush enter 2023-10-30T12:11:24.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 10 librados: watch_flush exit 2023-10-30T12:11:24.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 20 client.114602.objecter shutdown clearing up homeless session... 2023-10-30T12:11:24.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 10 client.114602.objecter successfully canceled tick 2023-10-30T12:11:24.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c6c039ad0 msgr2=0x7f5c6c03bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c6c039ad0 0x7f5c6c03bf90 secure :-1 s=READY pgs=4117 cs=0 l=1 rev1=1 crypto rx=0x7f5c78040000 tx=0x7f5c78039000 comp rx=0 tx=0).stop 2023-10-30T12:11:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 10 monclient: shutdown 2023-10-30T12:11:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c74005b80 msgr2=0x7f5c8808afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c74005b80 0x7f5c8808afd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400b750 tx=0x7f5c8400bc10 comp rx=0 tx=0).stop 2023-10-30T12:11:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 shutdown_connections 2023-10-30T12:11:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c6c039ad0 0x7f5c6c03bf90 unknown :-1 s=CLOSED pgs=4117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c740074b0 0x7f5c8808ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c740069d0 0x7f5c8808b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 --2- 172.21.15.50:0/241230583 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c74005b80 0x7f5c8808afd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 >> 172.21.15.50:0/241230583 conn(0x7f5c880a05c0 msgr2=0x7f5c88096970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 shutdown_connections 2023-10-30T12:11:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 -- 172.21.15.50:0/241230583 wait complete. 2023-10-30T12:11:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:24.924+0000 7f5c90421640 1 librados: shutdown 2023-10-30T12:11:24.936 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:11:24.936 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:11:24.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:11:25.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.041+0000 7fb8c0826640 10 monclient: get_monmap_and_config 2023-10-30T12:11:25.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.041+0000 7fb8c0826640 10 monclient: build_initial_monmap 2023-10-30T12:11:25.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.041+0000 7fb8c0826640 10 monclient: monmap: 2023-10-30T12:11:25.043 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:25.044 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.044 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:25.042817+0000 2023-10-30T12:11:25.044 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:25.042817+0000 2023-10-30T12:11:25.044 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:25.044 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.045 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:25.045 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:25.045 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:25.045 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:25.046 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 1 Processor -- start 2023-10-30T12:11:25.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 1 -- start start 2023-10-30T12:11:25.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient: init 2023-10-30T12:11:25.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:25.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-10-30T12:11:25.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8bc096540 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:25.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb8bc136960 0x7fb8bc138d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient(hunting): picked mon.noname-c con 0x7fb8bc136960 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:25.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc13b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient(hunting): picked mon.noname-b con 0x7fb8bc139290 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:25.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc09bd30 con 0x7fb8bc096540 2023-10-30T12:11:25.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc09c400 con 0x7fb8bc136960 2023-10-30T12:11:25.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc09c120 con 0x7fb8bc139290 2023-10-30T12:11:25.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:25.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8c0826640 10 monclient(hunting): authenticate will time out at 1530.225968s 2023-10-30T12:11:25.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.042+0000 7fb8ba59c640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fb8bc096540 legacy=0x7fb8bc096920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:25.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc13b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc13b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56688/0 (socket says 172.21.15.50:56688) 2023-10-30T12:11:25.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1357534394 learned_addr learned my addr 172.21.15.50:0/1357534394 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:25.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): get_auth_request con 0x7fb8bc139290 auth_method 0 2023-10-30T12:11:25.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8b3fff640 1 --2- 172.21.15.50:0/1357534394 >> v2:172.21.15.118:3300/0 conn(0x7fb8bc136960 0x7fb8bc138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8b3fff640 10 monclient(hunting): get_auth_request con 0x7fb8bc136960 auth_method 0 2023-10-30T12:11:25.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8b3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8b3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_done global_id 114605 payload 274 2023-10-30T12:11:25.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1357534394 >> v2:172.21.15.118:3300/0 conn(0x7fb8bc136960 msgr2=0x7fb8bc138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:25.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 --2- 172.21.15.50:0/1357534394 >> v2:172.21.15.118:3300/0 conn(0x7fb8bc136960 0x7fb8bc138d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1357534394 >> v1:172.21.15.50:6789/0 conn(0x7fb8bc096540 legacy=0x7fb8bc096920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:25.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:11:25.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 monclient: found mon.noname-b 2023-10-30T12:11:25.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:25.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.043+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1357534394 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc097030 con 0x7fb8bc139290 2023-10-30T12:11:25.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8bad9d640 1 --2- 172.21.15.50:0/1357534394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc13b680 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009e50 tx=0x7fb8b402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b72f09a4058a7e7f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 1 -- 172.21.15.50:0/1357534394 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b402c9e0 con 0x7fb8bc139290 2023-10-30T12:11:25.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:25.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: dump: 2023-10-30T12:11:25.064 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.064 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.064 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.065 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.065 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.065 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.065 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.065 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.066 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.066 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: _finish_auth 0 2023-10-30T12:11:25.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:11:25.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:25.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 1 -- 172.21.15.50:0/1357534394 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b402cb40 con 0x7fb8bc139290 2023-10-30T12:11:25.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:25.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 1 -- 172.21.15.50:0/1357534394 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40343f0 con 0x7fb8bc139290 2023-10-30T12:11:25.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 5 monclient: authenticate success, global_id 114605 2023-10-30T12:11:25.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:25.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 4 set_mon_vals no callback set 2023-10-30T12:11:25.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:25.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8b9d9b640 10 monclient: dump: 2023-10-30T12:11:25.068 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.069 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.069 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.069 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.069 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.070 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.070 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.070 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 10 monclient: shutdown 2023-10-30T12:11:25.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:25.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 -- 172.21.15.50:0/1357534394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 msgr2=0x7fb8bc13b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1357534394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc13b680 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009e50 tx=0x7fb8b402be40 comp rx=0 tx=0).stop 2023-10-30T12:11:25.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 -- 172.21.15.50:0/1357534394 shutdown_connections 2023-10-30T12:11:25.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1357534394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc13b680 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1357534394 >> v2:172.21.15.118:3300/0 conn(0x7fb8bc136960 0x7fb8bc138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 -- 172.21.15.50:0/1357534394 >> 172.21.15.50:0/1357534394 conn(0x7fb8bc0a3980 msgr2=0x7fb8bc0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:25.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 -- 172.21.15.50:0/1357534394 shutdown_connections 2023-10-30T12:11:25.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.044+0000 7fb8c0826640 1 -- 172.21.15.50:0/1357534394 wait complete. 2023-10-30T12:11:25.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.045+0000 7fb8c0826640 10 monclient: build_initial_monmap 2023-10-30T12:11:25.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.045+0000 7fb8c0826640 10 monclient: monmap: 2023-10-30T12:11:25.072 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:25.073 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:25.073 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:25.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:25.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:25.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.074 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:25.074 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:25.074 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:25.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.045+0000 7fb8c0826640 1 librados: starting msgr at 2023-10-30T12:11:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.045+0000 7fb8c0826640 1 librados: starting objecter 2023-10-30T12:11:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.045+0000 7fb8c0826640 1 Processor -- start 2023-10-30T12:11:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 -- start start 2023-10-30T12:11:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 librados: setting wanted keys 2023-10-30T12:11:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 librados: calling monclient init 2023-10-30T12:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient: init 2023-10-30T12:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8bc096540 0x7fb8bc1ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): picked mon.noname-a con 0x7fb8bc096540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8bc136960 0x7fb8bc1cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): picked mon.noname-b con 0x7fb8bc136960 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:25.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc0705f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): picked mon.noname-c con 0x7fb8bc139290 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:25.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0934e0 con 0x7fb8bc096540 2023-10-30T12:11:25.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0a28c0 con 0x7fb8bc136960 2023-10-30T12:11:25.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0992e0 con 0x7fb8bc139290 2023-10-30T12:11:25.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8ba59c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8bc096540 msgr2=0x7fb8bc1ca6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:25.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:25.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8c0826640 10 monclient(hunting): authenticate will time out at 1530.229526s 2023-10-30T12:11:25.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8ba59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8bc096540 0x7fb8bc1ca6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:25.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc0705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc0705f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56698/0 (socket says 172.21.15.50:56698) 2023-10-30T12:11:25.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1268593049 learned_addr learned my addr 172.21.15.50:0/1268593049 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:25.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): get_auth_request con 0x7fb8bc139290 auth_method 0 2023-10-30T12:11:25.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient(hunting): handle_auth_done global_id 114608 payload 824 2023-10-30T12:11:25.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8bc136960 msgr2=0x7fb8bc1cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8bc136960 0x7fb8bc1cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8bc096540 msgr2=0x7fb8bc1ca6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:25.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8bc096540 0x7fb8bc1ca6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:11:25.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 monclient: found mon.noname-c 2023-10-30T12:11:25.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:25.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.046+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1268593049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4009b30 con 0x7fb8bc139290 2023-10-30T12:11:25.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8bad9d640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc0705f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009e50 tx=0x7fb8b4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 client.?.objecter ms_handle_connect 0x7fb8bc139290 2023-10-30T12:11:25.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:25.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4040670 con 0x7fb8bc139290 2023-10-30T12:11:25.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:25.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: dump: 2023-10-30T12:11:25.085 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.085 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.085 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.085 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.085 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.086 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.086 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.086 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.086 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.086 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:11:25.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:11:25.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:25.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 5 monclient: authenticate success, global_id 114608 2023-10-30T12:11:25.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b40407f0 con 0x7fb8bc139290 2023-10-30T12:11:25.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:25.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4040ba0 con 0x7fb8bc139290 2023-10-30T12:11:25.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 10 monclient: _renew_subs 2023-10-30T12:11:25.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc08f510 con 0x7fb8bc139290 2023-10-30T12:11:25.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b959a640 4 set_mon_vals no callback set 2023-10-30T12:11:25.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:25.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8b2ffd640 10 monclient: dump: 2023-10-30T12:11:25.089 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.089 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.089 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.090 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.090 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.090 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.090 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.090 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.091 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.091 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 10 client.114608.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:25.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 10 monclient: _renew_subs 2023-10-30T12:11:25.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc0986a0 con 0x7fb8bc139290 2023-10-30T12:11:25.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.047+0000 7fb8c0826640 1 librados: init done 2023-10-30T12:11:25.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb8b4040d00 con 0x7fb8bc139290 2023-10-30T12:11:25.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8c0826640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:25.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc097d70 0x7fb8bc070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8c0826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc073ff0 con 0x7fb8bc097d70 2023-10-30T12:11:25.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc097d70 0x7fb8bc070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient: get_auth_request con 0x7fb8bc097d70 auth_method 0 2023-10-30T12:11:25.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb89c0392a0 0x7fb89c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fb8b406da30 con 0x7fb8bc139290 2023-10-30T12:11:25.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 client.114608.objecter ms_dispatch 0x7fb8bc000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:25.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 3 client.114608.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:25.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 3 client.114608.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:25.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_done global_id 114611 payload 824 2023-10-30T12:11:25.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 1 -- 172.21.15.50:0/1268593049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8bc09ba70 con 0x7fb8bc097d70 2023-10-30T12:11:25.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 20 client.114608.objecter dump_active .. 0 homeless 2023-10-30T12:11:25.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 client.114608.objecter ms_handle_connect 0x7fb8bc097d70 2023-10-30T12:11:25.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 client.114608.objecter resend_mon_ops 2023-10-30T12:11:25.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8ba59c640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc097d70 0x7fb8bc070b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00b450 tx=0x7fb8ac00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b3fff640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb89c0392a0 0x7fb89c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b3fff640 10 monclient: get_auth_request con 0x7fb89c0392a0 auth_method 0 2023-10-30T12:11:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac014070 con 0x7fb8bc097d70 2023-10-30T12:11:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 monclient: dump: 2023-10-30T12:11:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.099 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb8ac003c60 con 0x7fb8bc097d70 2023-10-30T12:11:25.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:25.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.048+0000 7fb8b2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:25.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.049+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc097d70 msgr2=0x7fb8bc070b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.049+0000 7fb8b2ffd640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc097d70 0x7fb8bc070b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00b450 tx=0x7fb8ac00b910 comp rx=0 tx=0).stop 2023-10-30T12:11:25.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.049+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc097d70 msgr2=0x7fb8bc070b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:25.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.049+0000 7fb8b2ffd640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc097d70 0x7fb8bc070b30 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00b450 tx=0x7fb8ac00b910 comp rx=0 tx=0).stop 2023-10-30T12:11:25.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.049+0000 7fb8b2ffd640 10 client.114608.objecter ms_handle_connect 0x7fb89c0392a0 2023-10-30T12:11:25.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.049+0000 7fb8b3fff640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb89c0392a0 0x7fb89c03b760 secure :-1 s=READY pgs=4118 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc09a260 tx=0x7fb8a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8c0826640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:25.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc071070 0x7fb8bc071480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8c0826640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc094080 con 0x7fb8bc071070 2023-10-30T12:11:25.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc071070 0x7fb8bc071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient: get_auth_request con 0x7fb8bc071070 auth_method 0 2023-10-30T12:11:25.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 10 monclient(hunting): handle_auth_done global_id 114614 payload 824 2023-10-30T12:11:25.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 1 -- 172.21.15.50:0/1268593049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb8bc08f730 con 0x7fb8bc071070 2023-10-30T12:11:25.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8b2ffd640 10 client.114608.objecter ms_handle_connect 0x7fb8bc071070 2023-10-30T12:11:25.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8b2ffd640 10 client.114608.objecter resend_mon_ops 2023-10-30T12:11:25.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8ba59c640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc071070 0x7fb8bc071480 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac012040 tx=0x7fb8ac005a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac01f070 con 0x7fb8bc071070 2023-10-30T12:11:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.055+0000 7fb8b2ffd640 10 monclient: dump: 2023-10-30T12:11:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.109 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.109 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4621 (secure 0 0 0) 0x7fb8ac0058f0 con 0x7fb8bc071070 2023-10-30T12:11:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8b2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc071070 msgr2=0x7fb8bc071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8b2ffd640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc071070 0x7fb8bc071480 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac012040 tx=0x7fb8ac005a70 comp rx=0 tx=0).stop 2023-10-30T12:11:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8b2ffd640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc071070 msgr2=0x7fb8bc071480 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:25.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8b2ffd640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc071070 0x7fb8bc071480 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac012040 tx=0x7fb8ac005a70 comp rx=0 tx=0).stop 2023-10-30T12:11:25.111 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:11:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 102, 2023-10-30T12:11:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:11:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:11:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:11:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.142 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.142 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:25.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.142 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:25.142 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:25.143 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:11:25.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.143 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.143 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:25.143 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:25.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 10 librados: watch_flush enter 2023-10-30T12:11:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 10 librados: watch_flush exit 2023-10-30T12:11:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 20 client.114608.objecter shutdown clearing up homeless session... 2023-10-30T12:11:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 10 client.114608.objecter successfully canceled tick 2023-10-30T12:11:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb89c0392a0 msgr2=0x7fb89c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb89c0392a0 0x7fb89c03b760 secure :-1 s=READY pgs=4118 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc09a260 tx=0x7fb8a4008040 comp rx=0 tx=0).stop 2023-10-30T12:11:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 10 monclient: shutdown 2023-10-30T12:11:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 msgr2=0x7fb8bc0705f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8bad9d640 1 -- 172.21.15.50:0/1268593049 reap_dead start 2023-10-30T12:11:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc0705f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009e50 tx=0x7fb8b4004a40 comp rx=0 tx=0).stop 2023-10-30T12:11:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 shutdown_connections 2023-10-30T12:11:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 --2- 172.21.15.50:0/1268593049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb8bc139290 0x7fb8bc0705f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 >> 172.21.15.50:0/1268593049 conn(0x7fb8bc0a3980 msgr2=0x7fb8bc09a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 shutdown_connections 2023-10-30T12:11:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 -- 172.21.15.50:0/1268593049 wait complete. 2023-10-30T12:11:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.056+0000 7fb8c0826640 1 librados: shutdown 2023-10-30T12:11:25.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:11:25.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:11:25.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:11:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.254+0000 7f4c316e5640 10 monclient: get_monmap_and_config 2023-10-30T12:11:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.254+0000 7f4c316e5640 10 monclient: build_initial_monmap 2023-10-30T12:11:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.254+0000 7f4c316e5640 10 monclient: monmap: 2023-10-30T12:11:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:25.255911+0000 2023-10-30T12:11:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:25.255911+0000 2023-10-30T12:11:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:25.259 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 1 Processor -- start 2023-10-30T12:11:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 1 -- start start 2023-10-30T12:11:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient: init 2023-10-30T12:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-10-30T12:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4c2c09bb70 0x7f4c2c09bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c09bb70 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c09c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c09d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c2c096080 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c098170 con 0x7f4c2c096080 2023-10-30T12:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c09c9d0 con 0x7f4c2c09bb70 2023-10-30T12:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c09cf60 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c316e5640 10 monclient(hunting): authenticate will time out at 1530.438913s 2023-10-30T12:11:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c2affd640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f4c2c09bb70 msgr2=0x7f4c2c09bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c2affd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4c2c09bb70 0x7f4c2c09bf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.255+0000 7f4c2b7fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f4c2c096080 legacy=0x7f4c2c0964f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:11:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c09c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c09c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56722/0 (socket says 172.21.15.50:56722) 2023-10-30T12:11:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 -- 172.21.15.50:0/2299689257 learned_addr learned my addr 172.21.15.50:0/2299689257 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): get_auth_request con 0x7f4c2c09d3f0 auth_method 0 2023-10-30T12:11:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient(hunting): handle_auth_done global_id 114617 payload 274 2023-10-30T12:11:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 -- 172.21.15.50:0/2299689257 >> v2:172.21.15.50:3300/0 conn(0x7f4c2c09bb70 msgr2=0x7f4c2c09bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 --2- 172.21.15.50:0/2299689257 >> v2:172.21.15.50:3300/0 conn(0x7f4c2c09bb70 0x7f4c2c09bf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 -- 172.21.15.50:0/2299689257 >> v1:172.21.15.50:6789/0 conn(0x7f4c2c096080 legacy=0x7f4c2c0964f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 monclient: found mon.noname-b 2023-10-30T12:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.256+0000 7f4c2a7fc640 1 -- 172.21.15.50:0/2299689257 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c097b20 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c2a7fc640 1 --2- 172.21.15.50:0/2299689257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c09c490 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009870 tx=0x7f4c1c02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71fdc0aa4cf2adf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 1 -- 172.21.15.50:0/2299689257 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c031750 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: dump: 2023-10-30T12:11:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.275 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.276 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.276 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.276 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.276 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.276 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: _finish_auth 0 2023-10-30T12:11:25.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:11:25.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:25.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 5 monclient: authenticate success, global_id 114617 2023-10-30T12:11:25.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 1 -- 172.21.15.50:0/2299689257 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1c0318b0 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 4 set_mon_vals no callback set 2023-10-30T12:11:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 1 -- 172.21.15.50:0/2299689257 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c031c40 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c29ffb640 10 monclient: dump: 2023-10-30T12:11:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.279 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.281 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 10 monclient: shutdown 2023-10-30T12:11:25.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:25.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 -- 172.21.15.50:0/2299689257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 msgr2=0x7f4c2c09c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 --2- 172.21.15.50:0/2299689257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c09c490 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009870 tx=0x7f4c1c02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 -- 172.21.15.50:0/2299689257 shutdown_connections 2023-10-30T12:11:25.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 --2- 172.21.15.50:0/2299689257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c09c490 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 --2- 172.21.15.50:0/2299689257 >> v2:172.21.15.50:3300/0 conn(0x7f4c2c09bb70 0x7f4c2c09bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 -- 172.21.15.50:0/2299689257 >> 172.21.15.50:0/2299689257 conn(0x7f4c2c0a3980 msgr2=0x7f4c2c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:25.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 -- 172.21.15.50:0/2299689257 shutdown_connections 2023-10-30T12:11:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.257+0000 7f4c316e5640 1 -- 172.21.15.50:0/2299689257 wait complete. 2023-10-30T12:11:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 10 monclient: build_initial_monmap 2023-10-30T12:11:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 10 monclient: monmap: 2023-10-30T12:11:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:25.285 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 1 librados: starting msgr at 2023-10-30T12:11:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 1 librados: starting objecter 2023-10-30T12:11:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 1 Processor -- start 2023-10-30T12:11:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 1 -- start start 2023-10-30T12:11:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 1 librados: setting wanted keys 2023-10-30T12:11:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 1 librados: calling monclient init 2023-10-30T12:11:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.258+0000 7f4c316e5640 10 monclient: init 2023-10-30T12:11:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:11:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c096080 0x7f4c2c0706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c2c09bb70 0x7f4c2c06bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c09bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c06c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c09d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c06f200 con 0x7f4c2c09bb70 2023-10-30T12:11:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c06f670 con 0x7f4c2c096080 2023-10-30T12:11:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c097120 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c316e5640 10 monclient(hunting): authenticate will time out at 1530.442362s 2023-10-30T12:11:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2a7fc640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c2c09bb70 msgr2=0x7f4c2c06bfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:11:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c2c09bb70 0x7f4c2c06bfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c06c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c06c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56730/0 (socket says 172.21.15.50:56730) 2023-10-30T12:11:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 -- 172.21.15.50:0/3114516242 learned_addr learned my addr 172.21.15.50:0/3114516242 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): get_auth_request con 0x7f4c2c09d3f0 auth_method 0 2023-10-30T12:11:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2affd640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c096080 0x7f4c2c0706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2affd640 10 monclient(hunting): get_auth_request con 0x7f4c2c096080 auth_method 0 2023-10-30T12:11:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient(hunting): handle_auth_done global_id 114620 payload 824 2023-10-30T12:11:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c096080 msgr2=0x7f4c2c0706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c096080 0x7f4c2c0706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c2c09bb70 msgr2=0x7f4c2c06bfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c2c09bb70 0x7f4c2c06bfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:11:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 monclient: found mon.noname-c 2023-10-30T12:11:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 -- 172.21.15.50:0/3114516242 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c0094d0 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c2b7fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c06c520 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400beb0 tx=0x7f4c2400bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c237fe640 10 client.?.objecter ms_handle_connect 0x7f4c2c09d3f0 2023-10-30T12:11:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c237fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24016430 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.259+0000 7f4c237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:25.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: dump: 2023-10-30T12:11:25.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.296 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.296 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.298 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.298 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:25.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:25.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:25.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c240165b0 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 5 monclient: authenticate success, global_id 114620 2023-10-30T12:11:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 10 monclient: _renew_subs 2023-10-30T12:11:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c0976e0 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24016960 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 10 monclient: dump: 2023-10-30T12:11:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.302 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c297fa640 4 set_mon_vals no callback set 2023-10-30T12:11:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 10 client.114620.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 10 monclient: _renew_subs 2023-10-30T12:11:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c06ff10 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 1 librados: init done 2023-10-30T12:11:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c06ca60 0x7f4c2c136af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c073ff0 con 0x7f4c2c06ca60 2023-10-30T12:11:25.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.260+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4c24012020 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c06ca60 0x7f4c2c136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient: get_auth_request con 0x7f4c2c06ca60 auth_method 0 2023-10-30T12:11:25.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c0c0392a0 0x7f4c0c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(347..347 src has 1..347) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f4c2404af50 con 0x7f4c2c09d3f0 2023-10-30T12:11:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 10 client.114620.objecter ms_dispatch 0x7f4c2c000b90 osd_map(347..347 src has 1..347) v4 2023-10-30T12:11:25.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 3 client.114620.objecter handle_osd_map got epochs [347,347] > 0 2023-10-30T12:11:25.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 3 client.114620.objecter handle_osd_map decoding full epoch 347 2023-10-30T12:11:25.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 20 client.114620.objecter dump_active .. 0 homeless 2023-10-30T12:11:25.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2a7fc640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c0c0392a0 0x7f4c0c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2a7fc640 10 monclient: get_auth_request con 0x7f4c0c0392a0 auth_method 0 2023-10-30T12:11:25.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 10 monclient(hunting): handle_auth_done global_id 154162 payload 824 2023-10-30T12:11:25.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 1 -- 172.21.15.50:0/3114516242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c2c097560 con 0x7f4c2c06ca60 2023-10-30T12:11:25.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2affd640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c06ca60 0x7f4c2c136af0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400d700 tx=0x7f4c1400dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 10 client.114620.objecter ms_handle_connect 0x7f4c2c06ca60 2023-10-30T12:11:25.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 10 client.114620.objecter resend_mon_ops 2023-10-30T12:11:25.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c14003a00 con 0x7f4c2c06ca60 2023-10-30T12:11:25.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:25.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 10 monclient: dump: 2023-10-30T12:11:25.309 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.309 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.309 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.309 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.309 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.310 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.310 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.310 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.310 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.310 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c237fe640 10 client.114620.objecter ms_handle_connect 0x7f4c0c0392a0 2023-10-30T12:11:25.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.261+0000 7f4c2a7fc640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c0c0392a0 0x7f4c0c03b760 secure :-1 s=READY pgs=4119 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c035000 tx=0x7f4c1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.262+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f4c14003b60 con 0x7f4c2c06ca60 2023-10-30T12:11:25.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.262+0000 7f4c237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:25.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.262+0000 7f4c237fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:25.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.262+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c06ca60 msgr2=0x7f4c2c136af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.262+0000 7f4c237fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c06ca60 0x7f4c2c136af0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400d700 tx=0x7f4c1400dbc0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.262+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c06ca60 msgr2=0x7f4c2c136af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:25.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.262+0000 7f4c237fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c06ca60 0x7f4c2c136af0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400d700 tx=0x7f4c1400dbc0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c316e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:11:25.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c316e5640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c137030 0x7f4c2c137440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:25.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c316e5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:25.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c097e00 con 0x7f4c2c137030 2023-10-30T12:11:25.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c137030 0x7f4c2c137440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:25.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 10 monclient: get_auth_request con 0x7f4c2c137030 auth_method 0 2023-10-30T12:11:25.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:25.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:25.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:25.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:25.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:25.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.268+0000 7f4c2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:25.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c2affd640 10 monclient(hunting): handle_auth_done global_id 154165 payload 824 2023-10-30T12:11:25.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c2affd640 1 -- 172.21.15.50:0/3114516242 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c2c06ce40 con 0x7f4c2c137030 2023-10-30T12:11:25.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c2affd640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c137030 0x7f4c2c137440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4c14000c00 tx=0x7f4c14005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:25.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 10 client.114620.objecter ms_handle_connect 0x7f4c2c137030 2023-10-30T12:11:25.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 10 client.114620.objecter resend_mon_ops 2023-10-30T12:11:25.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c14022070 con 0x7f4c2c137030 2023-10-30T12:11:25.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:25.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:25.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 10 monclient: dump: 2023-10-30T12:11:25.318 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:25.318 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:25.318 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:25.320 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:25.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4623 (secure 0 0 0) 0x7f4c14005420 con 0x7f4c2c137030 2023-10-30T12:11:25.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:11:25.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:25.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c137030 msgr2=0x7f4c2c137440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c137030 0x7f4c2c137440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4c14000c00 tx=0x7f4c14005a20 comp rx=0 tx=0).stop 2023-10-30T12:11:25.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c137030 msgr2=0x7f4c2c137440 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:11:25.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c237fe640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c2c137030 0x7f4c2c137440 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4c14000c00 tx=0x7f4c14005a20 comp rx=0 tx=0).stop 2023-10-30T12:11:25.321 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:11:25.321 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:25.322 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:25.322 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:11:25.322 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 102, 2023-10-30T12:11:25.322 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:11:25.322 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:11:25.323 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:11:25.323 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.323 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:11:25.323 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:25.323 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:11:25.324 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:11:25.324 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:25.324 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:25.324 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:25.324 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:25.325 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:25.325 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:25.325 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:25.325 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:25.325 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:25.326 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:25.326 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.326 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:11:25.326 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:11:25.326 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:25.327 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:25.327 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:25.327 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:25.327 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:25.327 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.329 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:11:25.329 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:11:25.329 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:11:25.329 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:11:25.330 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:11:25.330 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:11:25.330 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:25.330 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:11:25.330 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:11:25.331 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:11:25.331 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:11:25.331 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:11:25.331 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:11:25.331 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:11:25.331 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:11:25.332 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:11:25.332 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:11:25.332 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:11:25.332 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:11:25.332 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:11:25.333 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:11:25.333 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:11:25.333 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:11:25.333 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:11:25.333 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:11:25.333 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:11:25.334 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:11:25.334 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.334 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:11:25.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.334 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:11:25.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.335 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:11:25.335 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:11:25.335 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:25.335 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:25.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.336 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:25.336 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:11:25.336 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.337 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:25.337 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:11:25.337 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.338 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.338 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:11:25.338 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:11:25.338 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:25.339 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:25.339 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:25.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.339 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:11:25.339 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:11:25.340 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:25.340 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:25.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:25.340 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:11:25.340 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.341 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:25.341 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:11:25.341 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.342 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.342 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:11:25.342 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:11:25.343 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:25.343 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:25.343 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:25.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.344 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:11:25.344 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:11:25.344 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:11:25.344 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:11:25.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:11:25.345 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:11:25.345 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:11:25.345 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:11:25.346 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:11:25.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.346 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.346 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:11:25.347 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:11:25.347 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:11:25.347 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:11:25.347 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:11:25.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.348 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:11:25.348 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:11:25.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.349 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:25.349 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:25.349 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:25.349 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.349 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.349 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:11:25.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.350 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:25.350 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:25.350 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:11:25.350 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.351 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:11:25.351 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:11:25.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:11:25.351 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:11:25.351 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:11:25.352 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:11:25.352 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:11:25.352 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:11:25.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:11:25.352 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:11:25.353 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:11:25.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c316e5640 10 librados: watch_flush enter 2023-10-30T12:11:25.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c316e5640 10 librados: watch_flush exit 2023-10-30T12:11:25.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c316e5640 20 client.114620.objecter shutdown clearing up homeless session... 2023-10-30T12:11:25.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.269+0000 7f4c316e5640 10 client.114620.objecter successfully canceled tick 2023-10-30T12:11:25.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c0c0392a0 msgr2=0x7f4c0c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c0c0392a0 0x7f4c0c03b760 secure :-1 s=READY pgs=4119 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c035000 tx=0x7f4c1c008030 comp rx=0 tx=0).stop 2023-10-30T12:11:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 10 monclient: shutdown 2023-10-30T12:11:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 msgr2=0x7f4c2c06c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c2b7fe640 1 -- 172.21.15.50:0/3114516242 reap_dead start 2023-10-30T12:11:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c06c520 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400beb0 tx=0x7f4c2400bee0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 shutdown_connections 2023-10-30T12:11:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 --2- 172.21.15.50:0/3114516242 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c2c09d3f0 0x7f4c2c06c520 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 >> 172.21.15.50:0/3114516242 conn(0x7f4c2c0a3980 msgr2=0x7f4c2c139fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 shutdown_connections 2023-10-30T12:11:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 -- 172.21.15.50:0/3114516242 wait complete. 2023-10-30T12:11:25.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:25.270+0000 7f4c316e5640 1 librados: shutdown 2023-10-30T12:11:25.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:11:25.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:11:45.359 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:11:45.360 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:11:45.360 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:11:45.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:11:45.364 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:11:45.364 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:11:45.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.479+0000 7f2188010640 10 monclient: get_monmap_and_config 2023-10-30T12:11:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.479+0000 7f2188010640 10 monclient: build_initial_monmap 2023-10-30T12:11:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.479+0000 7f2188010640 10 monclient: monmap: 2023-10-30T12:11:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:45.480187+0000 2023-10-30T12:11:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:45.480187+0000 2023-10-30T12:11:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:45.483 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:45.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.479+0000 7f2188010640 1 Processor -- start 2023-10-30T12:11:45.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.479+0000 7f2188010640 1 -- start start 2023-10-30T12:11:45.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.479+0000 7f2188010640 10 monclient: init 2023-10-30T12:11:45.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:45.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T12:11:45.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f21800a4b60 0x7f21801326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:45.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient(hunting): picked mon.noname-c con 0x7f21800a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:45.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2180099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:45.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2180132be0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:45.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21800a4f40 con 0x7f2180132be0 2023-10-30T12:11:45.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21800a4150 con 0x7f2180099a40 2023-10-30T12:11:45.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:45.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21800a3d70 con 0x7f21800a4b60 2023-10-30T12:11:45.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:45.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2188010640 10 monclient(hunting): authenticate will time out at 1550.663452s 2023-10-30T12:11:45.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f21865ad640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2180132be0 0x7f21800a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56052/0 (socket says 172.21.15.50:56052) 2023-10-30T12:11:45.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f21865ad640 1 -- 172.21.15.50:0/3607389390 learned_addr learned my addr 172.21.15.50:0/3607389390 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:11:45.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2185dac640 1 --2- 172.21.15.50:0/3607389390 >> v2:172.21.15.118:3300/0 conn(0x7f21800a4b60 0x7f21801326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:45.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2185dac640 10 monclient(hunting): get_auth_request con 0x7f21800a4b60 auth_method 0 2023-10-30T12:11:45.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2185dac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:45.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2185dac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:45.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.480+0000 7f2185dac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:45.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:45.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:45.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:45.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 10 monclient(hunting): handle_auth_done global_id 154168 payload 274 2023-10-30T12:11:45.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 1 -- 172.21.15.50:0/3607389390 >> v1:172.21.15.118:6789/0 conn(0x7f2180099a40 legacy=0x7f21800a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:45.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 1 -- 172.21.15.50:0/3607389390 >> v1:172.21.15.50:6789/0 conn(0x7f2180132be0 legacy=0x7f21800a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:45.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 10 monclient: _finish_hunting 0 2023-10-30T12:11:45.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 1 monclient: found mon.noname-c 2023-10-30T12:11:45.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:45.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:11:45.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2185dac640 1 -- 172.21.15.50:0/3607389390 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21800a3170 con 0x7f21800a4b60 2023-10-30T12:11:45.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2184daa640 1 -- 172.21.15.50:0/3607389390 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540474255 0 0) 0x7f21800a4150 con 0x7f2180099a40 2023-10-30T12:11:45.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.481+0000 7f2184daa640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:11:45.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2185dac640 1 --2- 172.21.15.50:0/3607389390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21800a4b60 0x7f21801326a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097a0 tx=0x7f217c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31c498da7d0cb502 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:45.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 1 -- 172.21.15.50:0/3607389390 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c02e070 con 0x7f21800a4b60 2023-10-30T12:11:45.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:45.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:45.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: dump: 2023-10-30T12:11:45.498 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:45.498 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:45.499 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.499 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.499 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:45.499 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:45.499 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:45.499 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:45.500 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:45.500 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:45.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: _finish_auth 0 2023-10-30T12:11:45.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: _check_auth_tickets 2023-10-30T12:11:45.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:45.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 1 -- 172.21.15.50:0/3607389390 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c0315a0 con 0x7f21800a4b60 2023-10-30T12:11:45.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:45.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 1 -- 172.21.15.50:0/3607389390 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c031930 con 0x7f21800a4b60 2023-10-30T12:11:45.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 5 monclient: authenticate success, global_id 154168 2023-10-30T12:11:45.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:45.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 4 set_mon_vals no callback set 2023-10-30T12:11:45.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:45.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:45.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2184daa640 10 monclient: dump: 2023-10-30T12:11:45.502 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:45.502 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:45.503 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.503 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.503 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:45.503 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:45.503 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:45.504 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:45.504 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:45.504 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:45.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 10 monclient: shutdown 2023-10-30T12:11:45.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:45.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 1 -- 172.21.15.50:0/3607389390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21800a4b60 msgr2=0x7f21801326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:45.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 1 --2- 172.21.15.50:0/3607389390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21800a4b60 0x7f21801326a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097a0 tx=0x7f217c02b840 comp rx=0 tx=0).stop 2023-10-30T12:11:45.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 1 -- 172.21.15.50:0/3607389390 shutdown_connections 2023-10-30T12:11:45.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 1 --2- 172.21.15.50:0/3607389390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f21800a4b60 0x7f21801326a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:45.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.482+0000 7f2188010640 1 -- 172.21.15.50:0/3607389390 >> 172.21.15.50:0/3607389390 conn(0x7f21800a8060 msgr2=0x7f21800a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:45.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 -- 172.21.15.50:0/3607389390 shutdown_connections 2023-10-30T12:11:45.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 -- 172.21.15.50:0/3607389390 wait complete. 2023-10-30T12:11:45.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 10 monclient: build_initial_monmap 2023-10-30T12:11:45.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 10 monclient: monmap: 2023-10-30T12:11:45.506 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:45.506 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:45.507 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:45.507 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:45.507 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:45.507 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:45.507 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:45.507 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:45.508 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:45.508 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:45.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 librados: starting msgr at 2023-10-30T12:11:45.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 librados: starting objecter 2023-10-30T12:11:45.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 Processor -- start 2023-10-30T12:11:45.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 -- start start 2023-10-30T12:11:45.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 librados: setting wanted keys 2023-10-30T12:11:45.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 1 librados: calling monclient init 2023-10-30T12:11:45.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.483+0000 7f2188010640 10 monclient: init 2023-10-30T12:11:45.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:45.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:45.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 0x7f2180136e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:45.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): picked mon.noname-a con 0x7f21800a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:45.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2180132be0 0x7f21801373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:45.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): picked mon.noname-b con 0x7f2180132be0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:45.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f218013b030 0x7f2180137910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:45.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): picked mon.noname-c con 0x7f218013b030 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:45.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:45.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2180139990 con 0x7f21800a4b60 2023-10-30T12:11:45.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:45.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21800a03a0 con 0x7f2180132be0 2023-10-30T12:11:45.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:45.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218013a370 con 0x7f218013b030 2023-10-30T12:11:45.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:45.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2188010640 10 monclient(hunting): authenticate will time out at 1550.667389s 2023-10-30T12:11:45.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 0x7f2180136e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:45.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 0x7f2180136e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52738/0 (socket says 172.21.15.50:52738) 2023-10-30T12:11:45.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 1 -- 172.21.15.50:0/529634986 learned_addr learned my addr 172.21.15.50:0/529634986 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:45.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 10 monclient(hunting): get_auth_request con 0x7f21800a4b60 auth_method 0 2023-10-30T12:11:45.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:45.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:45.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:45.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f218013b030 0x7f2180137910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:45.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): get_auth_request con 0x7f218013b030 auth_method 0 2023-10-30T12:11:45.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:45.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:45.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:45.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 1 -- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 msgr2=0x7f2180136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:11:45.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 1 -- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 msgr2=0x7f2180136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:11:45.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 0x7f2180136e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:45.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:45.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:45.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:45.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f2185dac640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 0x7f2180136e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:45.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21855ab640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2180132be0 0x7f21801373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:45.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21855ab640 10 monclient(hunting): get_auth_request con 0x7f2180132be0 auth_method 0 2023-10-30T12:11:45.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21855ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:45.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21855ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:45.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21855ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:45.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient(hunting): handle_auth_done global_id 114623 payload 824 2023-10-30T12:11:45.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 1 -- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2180132be0 msgr2=0x7f21801373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:45.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2180132be0 0x7f21801373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:45.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 1 -- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 msgr2=0x7f2180136e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:45.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 0x7f2180136e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:45.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient: _finish_hunting 0 2023-10-30T12:11:45.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 1 monclient: found mon.noname-c 2023-10-30T12:11:45.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:45.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:45.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.484+0000 7f21865ad640 1 -- 172.21.15.50:0/529634986 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c009400 con 0x7f218013b030 2023-10-30T12:11:45.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21865ad640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f218013b030 0x7f2180137910 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f217800ca70 tx=0x7f217800cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:45.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 client.?.objecter ms_handle_connect 0x7f218013b030 2023-10-30T12:11:45.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:45.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 1 -- 172.21.15.50:0/529634986 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217800ab70 con 0x7f218013b030 2023-10-30T12:11:45.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:45.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:45.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: dump: 2023-10-30T12:11:45.522 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:45.522 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:45.523 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.523 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.523 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:45.523 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:45.523 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:45.523 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:45.524 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:45.524 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:45.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: _finish_auth 0 2023-10-30T12:11:45.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: _check_auth_tickets 2023-10-30T12:11:45.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:45.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 1 -- 172.21.15.50:0/529634986 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217800acd0 con 0x7f218013b030 2023-10-30T12:11:45.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 5 monclient: authenticate success, global_id 114623 2023-10-30T12:11:45.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 10 monclient: _renew_subs 2023-10-30T12:11:45.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:45.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218013a650 con 0x7f218013b030 2023-10-30T12:11:45.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:45.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 1 -- 172.21.15.50:0/529634986 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178012620 con 0x7f218013b030 2023-10-30T12:11:45.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:45.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:45.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f21767fc640 10 monclient: dump: 2023-10-30T12:11:45.526 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:45.527 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:45.527 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.527 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:45.527 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:45.527 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:45.527 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:45.528 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:45.528 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:45.528 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:45.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2177fff640 4 set_mon_vals no callback set 2023-10-30T12:11:45.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 10 client.114623.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 10 monclient: _renew_subs 2023-10-30T12:11:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2180137e50 con 0x7f218013b030 2023-10-30T12:11:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.485+0000 7f2188010640 1 librados: init done 2023-10-30T12:11:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 1 -- 172.21.15.50:0/529634986 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2178007070 con 0x7f218013b030 2023-10-30T12:11:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21640392f0 0x7f216403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 1 -- 172.21.15.50:0/529634986 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(357..357 src has 1..357) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f217804aad0 con 0x7f218013b030 2023-10-30T12:11:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 10 client.114623.objecter ms_dispatch 0x7f2180000b90 osd_map(357..357 src has 1..357) v4 2023-10-30T12:11:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 3 client.114623.objecter handle_osd_map got epochs [357,357] > 0 2023-10-30T12:11:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 3 client.114623.objecter handle_osd_map decoding full epoch 357 2023-10-30T12:11:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f2188010640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f2188010640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f2188010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218004e160 con 0x7f218013b030 2023-10-30T12:11:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 20 client.114623.objecter dump_active .. 0 homeless 2023-10-30T12:11:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f2185dac640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21640392f0 0x7f216403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f2185dac640 10 monclient: get_auth_request con 0x7f21640392f0 auth_method 0 2023-10-30T12:11:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f2185dac640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21640392f0 0x7f216403b7b0 secure :-1 s=READY pgs=4121 cs=0 l=1 rev1=1 crypto rx=0x7f217c000c00 tx=0x7f217c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.486+0000 7f21767fc640 10 client.114623.objecter ms_handle_connect 0x7f21640392f0 2023-10-30T12:11:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.491+0000 7f21767fc640 1 -- 172.21.15.50:0/529634986 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2178017020 con 0x7f218013b030 2023-10-30T12:11:45.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.491+0000 7f21767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:45.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.491+0000 7f21767fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:45.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.735+0000 7f2188010640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:45.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.735+0000 7f2188010640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:45.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.735+0000 7f2188010640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:45.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.735+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f21800745c0 con 0x7f218013b030 2023-10-30T12:11:45.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.735+0000 7f21767fc640 1 -- 172.21.15.50:0/529634986 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f21780198b0 con 0x7f218013b030 2023-10-30T12:11:45.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.735+0000 7f21767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:45.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.735+0000 7f21767fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:45.738 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 10 librados: watch_flush enter 2023-10-30T12:11:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 10 librados: watch_flush exit 2023-10-30T12:11:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 20 client.114623.objecter shutdown clearing up homeless session... 2023-10-30T12:11:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 10 client.114623.objecter successfully canceled tick 2023-10-30T12:11:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21640392f0 msgr2=0x7f216403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21640392f0 0x7f216403b7b0 secure :-1 s=READY pgs=4121 cs=0 l=1 rev1=1 crypto rx=0x7f217c000c00 tx=0x7f217c008030 comp rx=0 tx=0).stop 2023-10-30T12:11:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 10 monclient: shutdown 2023-10-30T12:11:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f218013b030 msgr2=0x7f2180137910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f218013b030 0x7f2180137910 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f217800ca70 tx=0x7f217800cf30 comp rx=0 tx=0).stop 2023-10-30T12:11:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 shutdown_connections 2023-10-30T12:11:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f21640392f0 0x7f216403b7b0 unknown :-1 s=CLOSED pgs=4121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f218013b030 0x7f2180137910 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2180132be0 0x7f21801373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 --2- 172.21.15.50:0/529634986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21800a4b60 0x7f2180136e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 >> 172.21.15.50:0/529634986 conn(0x7f21800a8060 msgr2=0x7f21800a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 shutdown_connections 2023-10-30T12:11:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 -- 172.21.15.50:0/529634986 wait complete. 2023-10-30T12:11:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:45.737+0000 7f2188010640 1 librados: shutdown 2023-10-30T12:11:48.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:48.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.864+0000 7f7028032640 10 monclient: get_monmap_and_config 2023-10-30T12:11:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.864+0000 7f7028032640 10 monclient: build_initial_monmap 2023-10-30T12:11:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.864+0000 7f7028032640 10 monclient: monmap: 2023-10-30T12:11:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:48.865683+0000 2023-10-30T12:11:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:48.865683+0000 2023-10-30T12:11:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:48.868 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:48.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 1 Processor -- start 2023-10-30T12:11:48.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 1 -- start start 2023-10-30T12:11:48.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient: init 2023-10-30T12:11:48.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:48.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-10-30T12:11:48.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 0x7f70200a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient(hunting): picked mon.noname-a con 0x7f70200a0080 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:48.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70200720b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:11:48.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7020073a30 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:48.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702009ccc0 con 0x7f70200720b0 2023-10-30T12:11:48.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7020071e90 con 0x7f7020073a30 2023-10-30T12:11:48.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:48.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7020073620 con 0x7f70200a0080 2023-10-30T12:11:48.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:48.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.865+0000 7f7028032640 10 monclient(hunting): authenticate will time out at 1554.048972s 2023-10-30T12:11:48.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 0x7f70200a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:48.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 0x7f70200a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36410/0 (socket says 172.21.15.50:36410) 2023-10-30T12:11:48.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f70255cd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f70200720b0 0x7f7020097320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52952/0 (socket says 172.21.15.50:52952) 2023-10-30T12:11:48.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 1 -- 172.21.15.50:0/4011714014 learned_addr learned my addr 172.21.15.50:0/4011714014 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 10 monclient(hunting): get_auth_request con 0x7f70200a0080 auth_method 0 2023-10-30T12:11:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 1 -- 172.21.15.50:0/4011714014 >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 msgr2=0x7f70200a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:11:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 1 -- 172.21.15.50:0/4011714014 >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 msgr2=0x7f70200a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 1 --2- 172.21.15.50:0/4011714014 >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 0x7f70200a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7025dce640 1 --2- 172.21.15.50:0/4011714014 >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 0x7f70200a0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719438649 0 0) 0x7f7020071e90 con 0x7f7020073a30 2023-10-30T12:11:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient(hunting): _negotiate 2023-10-30T12:11:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient(hunting): my global_id is 154177 2023-10-30T12:11:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700c003520 con 0x7f7020073a30 2023-10-30T12:11:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801132069 0 0) 0x7f700c003520 con 0x7f7020073a30 2023-10-30T12:11:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 msgr2=0x7f70200a0460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 --2- 172.21.15.50:0/4011714014 >> v2:172.21.15.50:3300/0 conn(0x7f70200a0080 0x7f70200a0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 >> v1:172.21.15.50:6789/0 conn(0x7f70200720b0 legacy=0x7f7020097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: _finish_hunting 0 2023-10-30T12:11:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:11:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:48.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:11:48.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020098140 con 0x7f7020073a30 2023-10-30T12:11:48.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: _finish_auth 0 2023-10-30T12:11:48.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: _check_auth_tickets 2023-10-30T12:11:48.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:48.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f70180019f0 con 0x7f7020073a30 2023-10-30T12:11:48.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7028032640 5 monclient: authenticate success, global_id 154177 2023-10-30T12:11:48.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7028032640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:48.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:48.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:11:48.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: dump: 2023-10-30T12:11:48.888 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:48.888 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:48.888 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.888 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:48.889 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:48.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:11:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.866+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 >> v1:172.21.15.118:6789/0 conn(0x7f7020073a30 legacy=0x7f7020097a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f700c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): picked mon.a con 0x7f700c003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f700c0047c0 0x7f700c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): picked mon.b con 0x7f700c0047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f700c005380 0x7f700c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): picked mon.c con 0x7f700c005380 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020073620 con 0x7f700c003970 2023-10-30T12:11:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c00c220 con 0x7f700c0047c0 2023-10-30T12:11:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c00c3e0 con 0x7f700c005380 2023-10-30T12:11:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7024dcc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:11:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7028032640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f700c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): get_auth_request con 0x7f700c003970 auth_method 0 2023-10-30T12:11:48.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:48.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:48.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:48.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f700c005380 0x7f700c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:48.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f700c0047c0 0x7f700c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:48.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 10 monclient(hunting): get_auth_request con 0x7f700c005380 auth_method 0 2023-10-30T12:11:48.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:48.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:48.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:48.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:48.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:48.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:11:48.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 10 monclient(hunting): get_auth_request con 0x7f700c0047c0 auth_method 0 2023-10-30T12:11:48.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:48.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:48.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:48.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:48.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:48.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:11:48.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:48.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:48.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70255cd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:11:48.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient(hunting): handle_auth_done global_id 154177 payload 293 2023-10-30T12:11:48.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 1 -- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f700c005380 msgr2=0x7f700c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:48.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f700c005380 0x7f700c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:48.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 1 -- 172.21.15.50:0/4011714014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f700c0047c0 msgr2=0x7f700c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:48.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f700c0047c0 0x7f700c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient: _finish_hunting 0 2023-10-30T12:11:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 1 monclient: found mon.a 2023-10-30T12:11:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:11:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 1 -- 172.21.15.50:0/4011714014 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c003520 con 0x7f700c003970 2023-10-30T12:11:48.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient: _finish_auth 0 2023-10-30T12:11:48.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 10 monclient: _check_auth_tickets 2023-10-30T12:11:48.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f7025dce640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:48.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.867+0000 7f70265cf640 1 -- 172.21.15.50:0/4011714014 reap_dead start 2023-10-30T12:11:48.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.868+0000 7f7025dce640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f700c003db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f701c009880 tx=0x7f700c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5225ff3cb09917d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:48.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.868+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c005b90 con 0x7f700c003970 2023-10-30T12:11:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.868+0000 7f7024dcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.868+0000 7f7024dcc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.868+0000 7f7024dcc640 10 monclient: dump: 2023-10-30T12:11:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:48.901 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:48.903 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.868+0000 7f7028032640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:48.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.868+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c005d10 con 0x7f700c003970 2023-10-30T12:11:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7024dcc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7024dcc640 1 -- 172.21.15.50:0/4011714014 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c006290 con 0x7f700c003970 2023-10-30T12:11:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 4 set_mon_vals no callback set 2023-10-30T12:11:48.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7024dcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7024dcc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:11:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7024dcc640 10 monclient: dump: 2023-10-30T12:11:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.905 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:48.907 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 10 monclient: shutdown 2023-10-30T12:11:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 1 -- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 msgr2=0x7f700c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f700c003db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f701c009880 tx=0x7f700c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:11:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 1 -- 172.21.15.50:0/4011714014 shutdown_connections 2023-10-30T12:11:48.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 1 --2- 172.21.15.50:0/4011714014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f700c003db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:48.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 1 -- 172.21.15.50:0/4011714014 >> 172.21.15.50:0/4011714014 conn(0x7f70200a4870 msgr2=0x7f70200a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:48.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 1 -- 172.21.15.50:0/4011714014 shutdown_connections 2023-10-30T12:11:48.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.869+0000 7f7028032640 1 -- 172.21.15.50:0/4011714014 wait complete. 2023-10-30T12:11:48.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.870+0000 7f7028032640 10 monclient: build_initial_monmap 2023-10-30T12:11:48.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.870+0000 7f7028032640 10 monclient: monmap: 2023-10-30T12:11:48.909 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:48.909 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:48.909 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:48.909 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:48.910 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:48.910 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:48.910 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:48.910 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:48.910 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:48.910 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:48.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 librados: starting msgr at 2023-10-30T12:11:48.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 librados: starting objecter 2023-10-30T12:11:48.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 Processor -- start 2023-10-30T12:11:48.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 -- start start 2023-10-30T12:11:48.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 librados: setting wanted keys 2023-10-30T12:11:48.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 librados: calling monclient init 2023-10-30T12:11:48.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 10 monclient: init 2023-10-30T12:11:48.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:48.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:11:48.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f70201c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 10 monclient(hunting): picked mon.noname-a con 0x7f700c003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:48.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70200965c0 0x7f7020096a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 10 monclient(hunting): picked mon.noname-b con 0x7f70200965c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:48.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70201c9e00 0x7f70201ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.871+0000 7f7028032640 10 monclient(hunting): picked mon.noname-c con 0x7f70201c9e00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:48.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:48.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70201c7850 con 0x7f700c003970 2023-10-30T12:11:48.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:48.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020096f80 con 0x7f70200965c0 2023-10-30T12:11:48.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:48.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70200917c0 con 0x7f70201c9e00 2023-10-30T12:11:48.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:48.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f7028032640 10 monclient(hunting): authenticate will time out at 1554.055053s 2023-10-30T12:11:48.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70200965c0 0x7f7020096a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:48.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70200965c0 0x7f7020096a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:37452/0 (socket says 172.21.15.50:37452) 2023-10-30T12:11:48.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 1 -- 172.21.15.50:0/1865385751 learned_addr learned my addr 172.21.15.50:0/1865385751 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 10 monclient(hunting): get_auth_request con 0x7f70200965c0 auth_method 0 2023-10-30T12:11:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.872+0000 7f70255cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 10 monclient(hunting): handle_auth_done global_id 154180 payload 824 2023-10-30T12:11:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 1 -- 172.21.15.50:0/1865385751 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70201c9e00 msgr2=0x7f70201ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70201c9e00 0x7f70201ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 1 -- 172.21.15.50:0/1865385751 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 msgr2=0x7f70201c7310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f70201c7310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 10 monclient: _finish_hunting 0 2023-10-30T12:11:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 1 monclient: found mon.noname-b 2023-10-30T12:11:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:48.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:11:48.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 1 -- 172.21.15.50:0/1865385751 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020091aa0 con 0x7f70200965c0 2023-10-30T12:11:48.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.873+0000 7f70255cd640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70200965c0 0x7f7020096a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f70140077e0 tx=0x7f7014001910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:48.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.874+0000 7f70127fc640 10 client.?.objecter ms_handle_connect 0x7f70200965c0 2023-10-30T12:11:48.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.874+0000 7f70127fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:48.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 1 -- 172.21.15.50:0/1865385751 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014018d80 con 0x7f70200965c0 2023-10-30T12:11:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: dump: 2023-10-30T12:11:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:48.922 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:48.923 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:48.924 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: _finish_auth 0 2023-10-30T12:11:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: _check_auth_tickets 2023-10-30T12:11:48.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 1 -- 172.21.15.50:0/1865385751 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7014021420 con 0x7f70200965c0 2023-10-30T12:11:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 5 monclient: authenticate success, global_id 154180 2023-10-30T12:11:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 10 monclient: _renew_subs 2023-10-30T12:11:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:11:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7020091d80 con 0x7f70200965c0 2023-10-30T12:11:48.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:48.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 1 -- 172.21.15.50:0/1865385751 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014021790 con 0x7f70200965c0 2023-10-30T12:11:48.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:48.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:48.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f70127fc640 10 monclient: dump: 2023-10-30T12:11:48.927 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:48.927 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:48.927 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.927 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:48.927 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:48.928 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:48.928 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:48.928 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:48.928 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:48.928 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:48.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7013fff640 4 set_mon_vals no callback set 2023-10-30T12:11:48.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 10 client.154180.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:48.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 10 monclient: _renew_subs 2023-10-30T12:11:48.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:11:48.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702008fc90 con 0x7f70200965c0 2023-10-30T12:11:48.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.877+0000 7f7028032640 1 librados: init done 2023-10-30T12:11:48.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.880+0000 7f7028032640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:48.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.880+0000 7f7028032640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:48.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.880+0000 7f7028032640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:11:48.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.880+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7020090af0 con 0x7f70200965c0 2023-10-30T12:11:48.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.885+0000 7f70127fc640 1 -- 172.21.15.50:0/1865385751 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f701401e020 con 0x7f70200965c0 2023-10-30T12:11:48.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f70127fc640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7004039340 0x7f700403b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:48.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f70127fc640 1 -- 172.21.15.50:0/1865385751 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(361..361 src has 1..361) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f7014052d00 con 0x7f70200965c0 2023-10-30T12:11:48.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f70127fc640 10 client.154180.objecter ms_dispatch 0x7f7020000b90 osd_map(361..361 src has 1..361) v4 2023-10-30T12:11:48.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f70127fc640 3 client.154180.objecter handle_osd_map got epochs [361,361] > 0 2023-10-30T12:11:48.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f70127fc640 3 client.154180.objecter handle_osd_map decoding full epoch 361 2023-10-30T12:11:48.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f70127fc640 20 client.154180.objecter dump_active .. 0 homeless 2023-10-30T12:11:48.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f7025dce640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7004039340 0x7f700403b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:48.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f7025dce640 10 monclient: get_auth_request con 0x7f7004039340 auth_method 0 2023-10-30T12:11:48.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f7025dce640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7004039340 0x7f700403b800 secure :-1 s=READY pgs=4122 cs=0 l=1 rev1=1 crypto rx=0x7f701c035000 tx=0x7f701c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:48.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.886+0000 7f70127fc640 10 client.154180.objecter ms_handle_connect 0x7f7004039340 2023-10-30T12:11:48.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.890+0000 7f70127fc640 1 -- 172.21.15.50:0/1865385751 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7014028a60 con 0x7f70200965c0 2023-10-30T12:11:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.890+0000 7f70127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:48.890+0000 7f70127fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:49.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.137+0000 7f7028032640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.137+0000 7f7028032640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.137+0000 7f7028032640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:11:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.137+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f702004e160 con 0x7f70200965c0 2023-10-30T12:11:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.138+0000 7f70127fc640 1 -- 172.21.15.50:0/1865385751 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7014026030 con 0x7f70200965c0 2023-10-30T12:11:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.138+0000 7f70127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.138+0000 7f70127fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:49.141 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:49.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 10 librados: watch_flush enter 2023-10-30T12:11:49.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 10 librados: watch_flush exit 2023-10-30T12:11:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 20 client.154180.objecter shutdown clearing up homeless session... 2023-10-30T12:11:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 10 client.154180.objecter successfully canceled tick 2023-10-30T12:11:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7004039340 msgr2=0x7f700403b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7004039340 0x7f700403b800 secure :-1 s=READY pgs=4122 cs=0 l=1 rev1=1 crypto rx=0x7f701c035000 tx=0x7f701c030000 comp rx=0 tx=0).stop 2023-10-30T12:11:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 10 monclient: shutdown 2023-10-30T12:11:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70200965c0 msgr2=0x7f7020096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70200965c0 0x7f7020096a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f70140077e0 tx=0x7f7014001910 comp rx=0 tx=0).stop 2023-10-30T12:11:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 shutdown_connections 2023-10-30T12:11:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7004039340 0x7f700403b800 unknown :-1 s=CLOSED pgs=4122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f70201c9e00 0x7f70201ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:49.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70200965c0 0x7f7020096a40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:49.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 --2- 172.21.15.50:0/1865385751 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f700c003970 0x7f70201c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:49.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 >> 172.21.15.50:0/1865385751 conn(0x7f70200a4870 msgr2=0x7f70201bcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:49.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 shutdown_connections 2023-10-30T12:11:49.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 -- 172.21.15.50:0/1865385751 wait complete. 2023-10-30T12:11:49.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:49.140+0000 7f7028032640 1 librados: shutdown 2023-10-30T12:11:52.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.267+0000 7f2ce5c56640 10 monclient: get_monmap_and_config 2023-10-30T12:11:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.267+0000 7f2ce5c56640 10 monclient: build_initial_monmap 2023-10-30T12:11:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.268+0000 7f2ce5c56640 10 monclient: monmap: 2023-10-30T12:11:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:52.269051+0000 2023-10-30T12:11:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:52.269051+0000 2023-10-30T12:11:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:52.272 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:52.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.268+0000 7f2ce5c56640 1 Processor -- start 2023-10-30T12:11:52.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.268+0000 7f2ce5c56640 1 -- start start 2023-10-30T12:11:52.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.268+0000 7f2ce5c56640 10 monclient: init 2023-10-30T12:11:52.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-10-30T12:11:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce009bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce0098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2ce0098900 0x7f2ce0098d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce0098900 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 0x7f2ce0132b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce01326d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce0099930 con 0x7f2ce01326d0 2023-10-30T12:11:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce009aca0 con 0x7f2ce0098900 2023-10-30T12:11:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce009af80 con 0x7f2ce0098520 2023-10-30T12:11:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2ce5c56640 10 monclient(hunting): authenticate will time out at 1557.452366s 2023-10-30T12:11:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce009bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 0x7f2ce0132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce009bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49448/0 (socket says 172.21.15.50:49448) 2023-10-30T12:11:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 10 monclient(hunting): get_auth_request con 0x7f2ce01326d0 auth_method 0 2023-10-30T12:11:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/2529453502 learned_addr learned my addr 172.21.15.50:0/2529453502 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): get_auth_request con 0x7f2ce0098520 auth_method 0 2023-10-30T12:11:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 1 -- 172.21.15.50:0/2529453502 >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 msgr2=0x7f2ce0132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:11:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 1 -- 172.21.15.50:0/2529453502 >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 msgr2=0x7f2ce0132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 1 --2- 172.21.15.50:0/2529453502 >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 0x7f2ce0132b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdffff640 1 --2- 172.21.15.50:0/2529453502 >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 0x7f2ce0132b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 1 --2- 172.21.15.50:0/2529453502 >> v2:172.21.15.118:3300/0 conn(0x7f2ce0098900 0x7f2ce0098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 10 monclient(hunting): get_auth_request con 0x7f2ce0098900 auth_method 0 2023-10-30T12:11:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:52.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:52.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_done global_id 114626 payload 274 2023-10-30T12:11:52.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/2529453502 >> v2:172.21.15.118:3300/0 conn(0x7f2ce0098900 msgr2=0x7f2ce0098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:11:52.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 --2- 172.21.15.50:0/2529453502 >> v2:172.21.15.118:3300/0 conn(0x7f2ce0098900 0x7f2ce0098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/2529453502 >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 msgr2=0x7f2ce0132b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:52.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 --2- 172.21.15.50:0/2529453502 >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 0x7f2ce0132b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:11:52.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 monclient: found mon.noname-b 2023-10-30T12:11:52.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:52.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:11:52.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.269+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/2529453502 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce009b260 con 0x7f2ce0098520 2023-10-30T12:11:52.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cdf7fe640 1 --2- 172.21.15.50:0/2529453502 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce009bf70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0073db0 tx=0x7f2cd002b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ab8f4b39eaf9791 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:52.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 1 -- 172.21.15.50:0/2529453502 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0004030 con 0x7f2ce0098520 2023-10-30T12:11:52.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:52.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:11:52.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: dump: 2023-10-30T12:11:52.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:52.292 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:52.292 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:52.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:52.293 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:52.293 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:52.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:52.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:52.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: _finish_auth 0 2023-10-30T12:11:52.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:11:52.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:52.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 1 -- 172.21.15.50:0/2529453502 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd0004190 con 0x7f2ce0098520 2023-10-30T12:11:52.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2ce5c56640 5 monclient: authenticate success, global_id 114626 2023-10-30T12:11:52.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2ce5c56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:11:52.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:52.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2ce5c56640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:52.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 1 -- 172.21.15.50:0/2529453502 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0003610 con 0x7f2ce0098520 2023-10-30T12:11:52.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:52.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2ce5c56640 4 set_mon_vals no callback set 2023-10-30T12:11:52.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:52.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.270+0000 7f2cde7fc640 10 monclient: dump: 2023-10-30T12:11:52.297 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:52.297 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:52.297 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.297 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.297 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:52.298 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:52.298 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:52.298 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:52.298 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:52.298 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:52.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 10 monclient: shutdown 2023-10-30T12:11:52.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:52.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 -- 172.21.15.50:0/2529453502 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 msgr2=0x7f2ce009bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:52.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/2529453502 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce009bf70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0073db0 tx=0x7f2cd002b7c0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 -- 172.21.15.50:0/2529453502 shutdown_connections 2023-10-30T12:11:52.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/2529453502 >> v2:172.21.15.50:3300/0 conn(0x7f2ce01326d0 0x7f2ce0132b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/2529453502 >> v2:172.21.15.118:3300/0 conn(0x7f2ce0098900 0x7f2ce0098d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/2529453502 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce009bf70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 -- 172.21.15.50:0/2529453502 >> 172.21.15.50:0/2529453502 conn(0x7f2ce00a05c0 msgr2=0x7f2ce009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:52.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 -- 172.21.15.50:0/2529453502 shutdown_connections 2023-10-30T12:11:52.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.271+0000 7f2ce5c56640 1 -- 172.21.15.50:0/2529453502 wait complete. 2023-10-30T12:11:52.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 10 monclient: build_initial_monmap 2023-10-30T12:11:52.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 10 monclient: monmap: 2023-10-30T12:11:52.301 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:52.301 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:52.302 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:52.302 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:52.302 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:52.302 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:52.302 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:52.303 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:52.303 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:52.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:52.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 1 librados: starting msgr at 2023-10-30T12:11:52.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 1 librados: starting objecter 2023-10-30T12:11:52.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 1 Processor -- start 2023-10-30T12:11:52.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 1 -- start start 2023-10-30T12:11:52.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 1 librados: setting wanted keys 2023-10-30T12:11:52.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 1 librados: calling monclient init 2023-10-30T12:11:52.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.272+0000 7f2ce5c56640 10 monclient: init 2023-10-30T12:11:52.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:52.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:11:52.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce0070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:52.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce0098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:52.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 0x7f2ce0131c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:52.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce0098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:52.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ce01326d0 0x7f2ce0132150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:52.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce01326d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:52.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:52.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce012f6d0 con 0x7f2ce0098900 2023-10-30T12:11:52.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:52.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce012eea0 con 0x7f2ce01326d0 2023-10-30T12:11:52.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:52.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01c7520 con 0x7f2ce0098520 2023-10-30T12:11:52.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:52.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 10 monclient(hunting): authenticate will time out at 1557.456118s 2023-10-30T12:11:52.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce0070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:52.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 0x7f2ce0131c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:52.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce0070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49460/0 (socket says 172.21.15.50:49460) 2023-10-30T12:11:52.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 0x7f2ce0131c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36450/0 (socket says 172.21.15.50:36450) 2023-10-30T12:11:52.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/3961794836 learned_addr learned my addr 172.21.15.50:0/3961794836 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:52.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): get_auth_request con 0x7f2ce0098520 auth_method 0 2023-10-30T12:11:52.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:52.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:52.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:52.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 10 monclient(hunting): get_auth_request con 0x7f2ce0098900 auth_method 0 2023-10-30T12:11:52.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:52.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:52.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:52.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:52.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:52.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:52.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 1 -- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 msgr2=0x7f2ce0131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:11:52.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 1 -- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 msgr2=0x7f2ce0131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:11:52.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 0x7f2ce0131c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:52.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ce01326d0 0x7f2ce0132150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:52.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 10 monclient(hunting): get_auth_request con 0x7f2ce01326d0 auth_method 0 2023-10-30T12:11:52.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:52.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:52.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:52.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdeffd640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 0x7f2ce0131c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:52.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:52.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:52.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:52.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient(hunting): handle_auth_done global_id 114629 payload 824 2023-10-30T12:11:52.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ce01326d0 msgr2=0x7f2ce0132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:52.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ce01326d0 0x7f2ce0132150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 msgr2=0x7f2ce0131c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 0x7f2ce0131c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:11:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 monclient: found mon.noname-c 2023-10-30T12:11:52.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 -- 172.21.15.50:0/3961794836 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0009460 con 0x7f2ce0098520 2023-10-30T12:11:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cdf7fe640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce0070030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0000c00 tx=0x7f2cd00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 10 client.?.objecter ms_handle_connect 0x7f2ce0098520 2023-10-30T12:11:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:52.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 1 -- 172.21.15.50:0/3961794836 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0003b20 con 0x7f2ce0098520 2023-10-30T12:11:52.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:52.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:52.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 10 monclient: dump: 2023-10-30T12:11:52.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:52.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:52.318 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:52.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 10 monclient: _finish_auth 0 2023-10-30T12:11:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:11:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2ce5c56640 5 monclient: authenticate success, global_id 114629 2023-10-30T12:11:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.273+0000 7f2cc7fff640 1 -- 172.21.15.50:0/3961794836 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd0003ca0 con 0x7f2ce0098520 2023-10-30T12:11:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 monclient: _renew_subs 2023-10-30T12:11:52.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce012f110 con 0x7f2ce0098520 2023-10-30T12:11:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2cc7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2cc7fff640 1 -- 172.21.15.50:0/3961794836 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0006280 con 0x7f2ce0098520 2023-10-30T12:11:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2cc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2cc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:52.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2cc7fff640 10 monclient: dump: 2023-10-30T12:11:52.321 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:52.321 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:52.321 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.321 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:52.322 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:52.322 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:52.322 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:52.322 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:52.322 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:52.323 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:52.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 client.114629.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:52.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 monclient: _renew_subs 2023-10-30T12:11:52.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:52.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce01c71c0 con 0x7f2ce0098520 2023-10-30T12:11:52.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2cddffb640 4 set_mon_vals no callback set 2023-10-30T12:11:52.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 1 librados: init done 2023-10-30T12:11:52.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2cc7fff640 1 -- 172.21.15.50:0/3961794836 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2cd00063e0 con 0x7f2ce0098520 2023-10-30T12:11:52.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:52.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:52.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:52.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.274+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce00a25a0 con 0x7f2ce0098520 2023-10-30T12:11:52.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cc7fff640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2ccc0392f0 0x7f2ccc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:52.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cc7fff640 1 -- 172.21.15.50:0/3961794836 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(364..364 src has 1..364) v4 ==== 6286+0+0 (secure 0 0 0) 0x7f2cd006ce80 con 0x7f2ce0098520 2023-10-30T12:11:52.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cc7fff640 10 client.114629.objecter ms_dispatch 0x7f2ce0000b90 osd_map(364..364 src has 1..364) v4 2023-10-30T12:11:52.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cc7fff640 3 client.114629.objecter handle_osd_map got epochs [364,364] > 0 2023-10-30T12:11:52.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cc7fff640 3 client.114629.objecter handle_osd_map decoding full epoch 364 2023-10-30T12:11:52.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cc7fff640 20 client.114629.objecter dump_active .. 0 homeless 2023-10-30T12:11:52.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cdeffd640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2ccc0392f0 0x7f2ccc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:52.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cdeffd640 10 monclient: get_auth_request con 0x7f2ccc0392f0 auth_method 0 2023-10-30T12:11:52.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cc7fff640 10 client.114629.objecter ms_handle_connect 0x7f2ccc0392f0 2023-10-30T12:11:52.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.275+0000 7f2cdeffd640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2ccc0392f0 0x7f2ccc03b7b0 secure :-1 s=READY pgs=4123 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009500 tx=0x7f2cd4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:52.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.279+0000 7f2cc7fff640 1 -- 172.21.15.50:0/3961794836 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2cd0003e10 con 0x7f2ce0098520 2023-10-30T12:11:52.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.279+0000 7f2cc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:52.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.279+0000 7f2cc7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:52.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.522+0000 7f2ce5c56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:52.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.522+0000 7f2ce5c56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:52.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.522+0000 7f2ce5c56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:52.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.522+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2ce004e160 con 0x7f2ce0098520 2023-10-30T12:11:52.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.522+0000 7f2cc7fff640 1 -- 172.21.15.50:0/3961794836 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2cd0038020 con 0x7f2ce0098520 2023-10-30T12:11:52.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.522+0000 7f2cc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:52.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.522+0000 7f2cc7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:52.525 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:52.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 10 librados: watch_flush enter 2023-10-30T12:11:52.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 10 librados: watch_flush exit 2023-10-30T12:11:52.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 20 client.114629.objecter shutdown clearing up homeless session... 2023-10-30T12:11:52.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 10 client.114629.objecter successfully canceled tick 2023-10-30T12:11:52.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2ccc0392f0 msgr2=0x7f2ccc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:52.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2ccc0392f0 0x7f2ccc03b7b0 secure :-1 s=READY pgs=4123 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009500 tx=0x7f2cd4008040 comp rx=0 tx=0).stop 2023-10-30T12:11:52.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 10 monclient: shutdown 2023-10-30T12:11:52.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:52.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 msgr2=0x7f2ce0070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:52.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce0070030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0000c00 tx=0x7f2cd00046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 shutdown_connections 2023-10-30T12:11:52.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2ccc0392f0 0x7f2ccc03b7b0 unknown :-1 s=CLOSED pgs=4123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ce01326d0 0x7f2ce0132150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ce0098900 0x7f2ce0131c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 --2- 172.21.15.50:0/3961794836 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ce0098520 0x7f2ce0070030 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:52.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 >> 172.21.15.50:0/3961794836 conn(0x7f2ce00a05c0 msgr2=0x7f2ce01b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:52.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 shutdown_connections 2023-10-30T12:11:52.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 -- 172.21.15.50:0/3961794836 wait complete. 2023-10-30T12:11:52.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:52.524+0000 7f2ce5c56640 1 librados: shutdown 2023-10-30T12:11:55.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:55.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.651+0000 7fbd3ef90640 10 monclient: get_monmap_and_config 2023-10-30T12:11:55.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.651+0000 7fbd3ef90640 10 monclient: build_initial_monmap 2023-10-30T12:11:55.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.651+0000 7fbd3ef90640 10 monclient: monmap: 2023-10-30T12:11:55.654 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:55.654 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:55.654 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:55.652846+0000 2023-10-30T12:11:55.655 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:55.652846+0000 2023-10-30T12:11:55.655 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:55.655 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:55.655 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:55.655 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:55.656 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:55.656 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:55.656 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:55.656 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:55.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.652+0000 7fbd3ef90640 1 Processor -- start 2023-10-30T12:11:55.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.652+0000 7fbd3ef90640 1 -- start start 2023-10-30T12:11:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.652+0000 7fbd3ef90640 10 monclient: init 2023-10-30T12:11:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:11:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbd38098520 0x7fbd3809bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient(hunting): picked mon.noname-c con 0x7fbd38098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd38098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 0x7fbd38139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient(hunting): picked mon.noname-a con 0x7fbd381379f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:11:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd3809ad30 con 0x7fbd38098900 2023-10-30T12:11:55.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3809c460 con 0x7fbd381379f0 2023-10-30T12:11:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3809b260 con 0x7fbd38098520 2023-10-30T12:11:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:55.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3ef90640 10 monclient(hunting): authenticate will time out at 1560.836101s 2023-10-30T12:11:55.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 0x7fbd38139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:55.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 0x7fbd38139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36472/0 (socket says 172.21.15.50:36472) 2023-10-30T12:11:55.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 1 -- 172.21.15.50:0/2047285929 learned_addr learned my addr 172.21.15.50:0/2047285929 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:55.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 10 monclient(hunting): get_auth_request con 0x7fbd381379f0 auth_method 0 2023-10-30T12:11:55.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:55.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:55.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:55.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 1 -- 172.21.15.50:0/2047285929 >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 msgr2=0x7fbd38139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:11:55.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 1 -- 172.21.15.50:0/2047285929 >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 msgr2=0x7fbd38139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:55.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 1 --2- 172.21.15.50:0/2047285929 >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 0x7fbd38139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:55.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3d52d640 1 --2- 172.21.15.50:0/2047285929 >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 0x7fbd38139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 1 --2- 172.21.15.50:0/2047285929 >> v2:172.21.15.118:3300/0 conn(0x7fbd38098520 0x7fbd3809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): get_auth_request con 0x7fbd38098520 auth_method 0 2023-10-30T12:11:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:55.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:55.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:55.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:55.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_done global_id 154189 payload 274 2023-10-30T12:11:55.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 1 -- 172.21.15.50:0/2047285929 >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 msgr2=0x7fbd38139de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:11:55.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 1 --2- 172.21.15.50:0/2047285929 >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 0x7fbd38139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 1 -- 172.21.15.50:0/2047285929 >> v1:172.21.15.118:6789/0 conn(0x7fbd38098900 legacy=0x7fbd38098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:55.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient: _finish_hunting 0 2023-10-30T12:11:55.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 1 monclient: found mon.noname-c 2023-10-30T12:11:55.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:55.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:11:55.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.653+0000 7fbd3cd2c640 1 -- 172.21.15.50:0/2047285929 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd38072120 con 0x7fbd38098520 2023-10-30T12:11:55.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd3cd2c640 1 --2- 172.21.15.50:0/2047285929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098520 0x7fbd3809bf20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbd38072310 tx=0x7fbd2c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0b4c29673d5b364 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:55.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 1 -- 172.21.15.50:0/2047285929 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c02f050 con 0x7fbd38098520 2023-10-30T12:11:55.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:55.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:55.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: dump: 2023-10-30T12:11:55.674 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:55.674 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:55.674 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.674 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.675 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:55.675 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:55.675 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:55.675 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:55.675 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:55.675 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: _finish_auth 0 2023-10-30T12:11:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: _check_auth_tickets 2023-10-30T12:11:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 1 -- 172.21.15.50:0/2047285929 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c031700 con 0x7fbd38098520 2023-10-30T12:11:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:55.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 1 -- 172.21.15.50:0/2047285929 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c031a90 con 0x7fbd38098520 2023-10-30T12:11:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:11:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd377fe640 10 monclient: dump: 2023-10-30T12:11:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:55.677 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:55.678 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:55.679 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd3ef90640 5 monclient: authenticate success, global_id 154189 2023-10-30T12:11:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd3ef90640 10 monclient: get_monmap_and_config success 2023-10-30T12:11:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.654+0000 7fbd3ef90640 4 set_mon_vals no callback set 2023-10-30T12:11:55.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 10 monclient: shutdown 2023-10-30T12:11:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 -- 172.21.15.50:0/2047285929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098520 msgr2=0x7fbd3809bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/2047285929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098520 0x7fbd3809bf20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbd38072310 tx=0x7fbd2c02b7a0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 -- 172.21.15.50:0/2047285929 shutdown_connections 2023-10-30T12:11:55.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/2047285929 >> v2:172.21.15.50:3300/0 conn(0x7fbd381379f0 0x7fbd38139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/2047285929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098520 0x7fbd3809bf20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 -- 172.21.15.50:0/2047285929 >> 172.21.15.50:0/2047285929 conn(0x7fbd380a05c0 msgr2=0x7fbd3809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 -- 172.21.15.50:0/2047285929 shutdown_connections 2023-10-30T12:11:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.655+0000 7fbd3ef90640 1 -- 172.21.15.50:0/2047285929 wait complete. 2023-10-30T12:11:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 10 monclient: build_initial_monmap 2023-10-30T12:11:55.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 10 monclient: monmap: 2023-10-30T12:11:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:11:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:11:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:11:55.682 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:11:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:11:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:11:55.683 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 1 librados: starting msgr at 2023-10-30T12:11:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 1 librados: starting objecter 2023-10-30T12:11:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 1 Processor -- start 2023-10-30T12:11:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 1 -- start start 2023-10-30T12:11:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 1 librados: setting wanted keys 2023-10-30T12:11:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 1 librados: calling monclient init 2023-10-30T12:11:55.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.656+0000 7fbd3ef90640 10 monclient: init 2023-10-30T12:11:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:11:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbd38098520 0x7fbd381ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): picked mon.noname-c con 0x7fbd38098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098900 0x7fbd380961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): picked mon.noname-b con 0x7fbd38098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 0x7fbd380966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): picked mon.noname-a con 0x7fbd381379f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd38091540 con 0x7fbd381379f0 2023-10-30T12:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd38092740 con 0x7fbd38098900 2023-10-30T12:11:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd381377b0 con 0x7fbd38098520 2023-10-30T12:11:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3ef90640 10 monclient(hunting): authenticate will time out at 1560.840144s 2023-10-30T12:11:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 0x7fbd380966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:55.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbd38098520 0x7fbd381ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 0x7fbd380966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36482/0 (socket says 172.21.15.50:36482) 2023-10-30T12:11:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 1 -- 172.21.15.50:0/1017415447 learned_addr learned my addr 172.21.15.50:0/1017415447 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 10 monclient(hunting): get_auth_request con 0x7fbd381379f0 auth_method 0 2023-10-30T12:11:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:55.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): get_auth_request con 0x7fbd38098520 auth_method 0 2023-10-30T12:11:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:55.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098900 0x7fbd380961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 10 monclient(hunting): get_auth_request con 0x7fbd38098900 auth_method 0 2023-10-30T12:11:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 1 -- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 msgr2=0x7fbd380966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:11:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 1 -- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 msgr2=0x7fbd380966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:11:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 0x7fbd380966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:55.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3d52d640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 0x7fbd380966e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:11:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:11:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:11:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient(hunting): handle_auth_done global_id 114632 payload 824 2023-10-30T12:11:55.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 -- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098900 msgr2=0x7fbd380961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098900 0x7fbd380961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 -- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 msgr2=0x7fbd380966e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:11:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 0x7fbd380966e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient: _finish_hunting 0 2023-10-30T12:11:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 monclient: found mon.noname-c 2023-10-30T12:11:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:11:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:11:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 -- 172.21.15.50:0/1017415447 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c0093a0 con 0x7fbd38098520 2023-10-30T12:11:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd3cd2c640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbd38098520 0x7fbd381ca1f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c033000 tx=0x7fbd2c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd357fa640 10 client.?.objecter ms_handle_connect 0x7fbd38098520 2023-10-30T12:11:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd357fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:11:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd357fa640 1 -- 172.21.15.50:0/1017415447 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c005da0 con 0x7fbd38098520 2023-10-30T12:11:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.657+0000 7fbd357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:11:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: dump: 2023-10-30T12:11:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:55.695 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:55.696 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:55.697 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: _finish_auth 0 2023-10-30T12:11:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: _check_auth_tickets 2023-10-30T12:11:55.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:11:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 5 monclient: authenticate success, global_id 114632 2023-10-30T12:11:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 10 monclient: _renew_subs 2023-10-30T12:11:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 1 -- 172.21.15.50:0/1017415447 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c033820 con 0x7fbd38098520 2023-10-30T12:11:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd38071df0 con 0x7fbd38098520 2023-10-30T12:11:55.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:11:55.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 1 -- 172.21.15.50:0/1017415447 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c033bb0 con 0x7fbd38098520 2023-10-30T12:11:55.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:11:55.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:11:55.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd357fa640 10 monclient: dump: 2023-10-30T12:11:55.699 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:11:55.699 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:55.700 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.700 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:11:55.700 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:11:55.700 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:55.700 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:11:55.700 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:11:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:11:55.701 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd36ffd640 4 set_mon_vals no callback set 2023-10-30T12:11:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 10 client.114632.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:11:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 10 monclient: _renew_subs 2023-10-30T12:11:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd38092460 con 0x7fbd38098520 2023-10-30T12:11:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.658+0000 7fbd3ef90640 1 librados: init done 2023-10-30T12:11:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd357fa640 1 -- 172.21.15.50:0/1017415447 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbd2c03a040 con 0x7fbd38098520 2023-10-30T12:11:55.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd3ef90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd357fa640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbd1c039320 0x7fbd1c03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd3ef90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd3ef90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd3804e160 con 0x7fbd38098520 2023-10-30T12:11:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd357fa640 1 -- 172.21.15.50:0/1017415447 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fbd2c06c370 con 0x7fbd38098520 2023-10-30T12:11:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd357fa640 10 client.114632.objecter ms_dispatch 0x7fbd38000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:11:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd357fa640 3 client.114632.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:11:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd357fa640 3 client.114632.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:11:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd357fa640 20 client.114632.objecter dump_active .. 0 homeless 2023-10-30T12:11:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd37fff640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbd1c039320 0x7fbd1c03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:55.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.659+0000 7fbd37fff640 10 monclient: get_auth_request con 0x7fbd1c039320 auth_method 0 2023-10-30T12:11:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.660+0000 7fbd357fa640 10 client.114632.objecter ms_handle_connect 0x7fbd1c039320 2023-10-30T12:11:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.660+0000 7fbd37fff640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbd1c039320 0x7fbd1c03b7e0 secure :-1 s=READY pgs=4124 cs=0 l=1 rev1=1 crypto rx=0x7fbd24000f60 tx=0x7fbd24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:11:55.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.664+0000 7fbd357fa640 1 -- 172.21.15.50:0/1017415447 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbd2c038030 con 0x7fbd38098520 2023-10-30T12:11:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.664+0000 7fbd357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:11:55.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.664+0000 7fbd357fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:11:55.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.907+0000 7fbd3ef90640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:11:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.907+0000 7fbd3ef90640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.907+0000 7fbd3ef90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:11:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.907+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbd38090720 con 0x7fbd38098520 2023-10-30T12:11:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.907+0000 7fbd357fa640 1 -- 172.21.15.50:0/1017415447 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fbd2c038020 con 0x7fbd38098520 2023-10-30T12:11:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.907+0000 7fbd357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:11:55.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.907+0000 7fbd357fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:11:55.910 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:11:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 10 librados: watch_flush enter 2023-10-30T12:11:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 10 librados: watch_flush exit 2023-10-30T12:11:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 20 client.114632.objecter shutdown clearing up homeless session... 2023-10-30T12:11:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 10 client.114632.objecter successfully canceled tick 2023-10-30T12:11:55.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbd1c039320 msgr2=0x7fbd1c03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbd1c039320 0x7fbd1c03b7e0 secure :-1 s=READY pgs=4124 cs=0 l=1 rev1=1 crypto rx=0x7fbd24000f60 tx=0x7fbd24009040 comp rx=0 tx=0).stop 2023-10-30T12:11:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 10 monclient: shutdown 2023-10-30T12:11:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:11:55.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbd38098520 msgr2=0x7fbd381ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:11:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbd38098520 0x7fbd381ca1f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c033000 tx=0x7fbd2c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 shutdown_connections 2023-10-30T12:11:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbd1c039320 0x7fbd1c03b7e0 unknown :-1 s=CLOSED pgs=4124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd381379f0 0x7fbd380966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd38098900 0x7fbd380961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 --2- 172.21.15.50:0/1017415447 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbd38098520 0x7fbd381ca1f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:11:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 >> 172.21.15.50:0/1017415447 conn(0x7fbd380a05c0 msgr2=0x7fbd381bcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:11:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 shutdown_connections 2023-10-30T12:11:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 -- 172.21.15.50:0/1017415447 wait complete. 2023-10-30T12:11:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:55.909+0000 7fbd3ef90640 1 librados: shutdown 2023-10-30T12:11:58.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:11:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.036+0000 7f9c7134c640 10 monclient: get_monmap_and_config 2023-10-30T12:11:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.036+0000 7f9c7134c640 10 monclient: build_initial_monmap 2023-10-30T12:11:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.036+0000 7f9c7134c640 10 monclient: monmap: 2023-10-30T12:11:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:11:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:11:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:11:59.037996+0000 2023-10-30T12:11:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:11:59.037996+0000 2023-10-30T12:11:59.039 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:11:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:11:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:11:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:11:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:11:59.040 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:11:59.041 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:11:59.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:11:59.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 1 Processor -- start 2023-10-30T12:11:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 1 -- start start 2023-10-30T12:11:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 10 monclient: init 2023-10-30T12:11:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 10 monclient: _reopen_session rank -1 2023-10-30T12:11:59.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-10-30T12:11:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c6c09f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:11:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:11:59.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.037+0000 7f9c7134c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6c096080 addr v2:172.21.15.118:3300/0 2023-10-30T12:11:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c6c0969a0 addr v1:172.21.15.118:6789/0 2023-10-30T12:11:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c098810 con 0x7f9c6c0969a0 2023-10-30T12:11:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c06aef0 con 0x7f9c6c096080 2023-10-30T12:11:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:11:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c09a1a0 con 0x7f9c6c09f5b0 2023-10-30T12:11:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 10 monclient(hunting): _renew_subs 2023-10-30T12:11:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c7134c640 10 monclient(hunting): authenticate will time out at 1564.221041s 2023-10-30T12:11:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45172/0 (socket says 172.21.15.50:45172) 2023-10-30T12:11:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 learned_addr learned my addr 172.21.15.50:0/1819004771 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:11:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c6c09f5b0 auth_method 0 2023-10-30T12:11:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:59.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:11:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:59.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c6c096080 auth_method 0 2023-10-30T12:11:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:11:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:11:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:59.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.038+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:11:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c6c09f5b0 auth_method 0 2023-10-30T12:11:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:11:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:59.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:11:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c6c096080 auth_method 0 2023-10-30T12:11:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:59.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:11:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:59.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.239+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:11:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:59.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c6c09f5b0 auth_method 0 2023-10-30T12:11:59.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:59.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:59.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:59.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:11:59.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:59.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:59.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:11:59.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:11:59.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c6c096080 auth_method 0 2023-10-30T12:11:59.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:11:59.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:11:59.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:11:59.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:11:59.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:11:59.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:11:59.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:11:59.640+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:00.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:00.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c6c09f5b0 auth_method 0 2023-10-30T12:12:00.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:00.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:00.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:00.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:00.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:00.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:00.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:00.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:00.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c6c096080 auth_method 0 2023-10-30T12:12:00.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:00.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:00.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.441+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:00.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.442+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:00.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.442+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:00.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.442+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:00.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:00.442+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:02.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient(hunting): tick 2023-10-30T12:12:02.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:12:02.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:12:02.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:12:02.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 msgr2=0x7f9c6c099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:02.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c09f5b0 0x7f9c6c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:02.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 msgr2=0x7f9c6c096460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:02.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c6c096080 0x7f9c6c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:02.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v1:172.21.15.118:6789/0 conn(0x7f9c6c0969a0 legacy=0x7f9c6c09cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:02.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient: _add_conns ranks=[3,0] 2023-10-30T12:12:02.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:02.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f9c58003710 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:02.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c58006240 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:02.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c098af0 con 0x7f9c58006240 2023-10-30T12:12:02.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:02.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c06aef0 con 0x7f9c58003710 2023-10-30T12:12:02.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c68ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:02.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:02.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58003710 auth_method 0 2023-10-30T12:12:02.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:02.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:02.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:02.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:02.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:02.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:02.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.038+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58003710 auth_method 0 2023-10-30T12:12:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:02.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:02.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:02.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:02.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.239+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:02.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:02.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58003710 auth_method 0 2023-10-30T12:12:02.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:02.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:02.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:02.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:02.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:02.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:02.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:02.640+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:03.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:03.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58003710 auth_method 0 2023-10-30T12:12:03.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:03.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:03.441+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): tick 2023-10-30T12:12:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:12:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:12:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:12:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58003710 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v1:172.21.15.50:6789/0 conn(0x7f9c58006240 legacy=0x7f9c580086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-10-30T12:12:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c5800a050 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 reap_dead start 2023-10-30T12:12:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c6c0969a0 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f9c58008eb0 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5800af90 con 0x7f9c5800a050 2023-10-30T12:12:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c58009290 con 0x7f9c6c0969a0 2023-10-30T12:12:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c06aef0 con 0x7f9c58008eb0 2023-10-30T12:12:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c68ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c58008eb0 auth_method 0 2023-10-30T12:12:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.038+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.039+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.039+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.039+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.039+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c58008eb0 auth_method 0 2023-10-30T12:12:05.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:05.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:05.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.240+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c58008eb0 auth_method 0 2023-10-30T12:12:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:05.641+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c58008eb0 auth_method 0 2023-10-30T12:12:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:06.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:06.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:06.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:06.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:06.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:06.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:06.442+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): tick 2023-10-30T12:12:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:12:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:12:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:12:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 msgr2=0x7f9c58005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c58008eb0 0x7f9c58005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:08.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v1:172.21.15.118:6789/0 conn(0x7f9c6c0969a0 legacy=0x7f9c580102b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:08.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v1:172.21.15.50:6789/0 conn(0x7f9c5800a050 legacy=0x7f9c5800a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:08.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient: _add_conns ranks=[4,2] 2023-10-30T12:12:08.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:08.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f9c58009420 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f9c580119d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c06aef0 con 0x7f9c58009420 2023-10-30T12:12:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c09a1a0 con 0x7f9c580119d0 2023-10-30T12:12:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.038+0000 7f9c68ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c580119d0 auth_method 0 2023-10-30T12:12:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:08.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58009420 auth_method 0 2023-10-30T12:12:08.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:08.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:08.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:08.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.039+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:08.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:08.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c580119d0 auth_method 0 2023-10-30T12:12:08.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:08.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:08.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:08.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:08.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:08.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:08.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:08.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:08.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58009420 auth_method 0 2023-10-30T12:12:08.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:08.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:08.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:08.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:08.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:08.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:08.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.240+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c580119d0 auth_method 0 2023-10-30T12:12:08.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:08.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:08.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:08.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:08.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:08.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:08.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:08.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:08.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58009420 auth_method 0 2023-10-30T12:12:08.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:08.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:08.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:08.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:08.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:08.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:08.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:08.641+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:09.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c580119d0 auth_method 0 2023-10-30T12:12:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58009420 auth_method 0 2023-10-30T12:12:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:09.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:09.442+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.038+0000 7f9c68ff9640 10 monclient(hunting): tick 2023-10-30T12:12:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:12:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:12:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:12:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c580119d0 0x7f9c58013dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 msgr2=0x7f9c58009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c58009420 0x7f9c58009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 1 -- 172.21.15.50:0/1819004771 reap_dead start 2023-10-30T12:12:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-10-30T12:12:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c58008eb0 0x7f9c58009800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f9c58008eb0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c5800a430 0x7f9c58013dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5800a430 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c5800a810 0x7f9c58005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5800a810 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c06aef0 con 0x7f9c5800a430 2023-10-30T12:12:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c09a1a0 con 0x7f9c5800a810 2023-10-30T12:12:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 1 -- 172.21.15.50:0/1819004771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5800b410 con 0x7f9c58008eb0 2023-10-30T12:12:11.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c68ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c58008eb0 0x7f9c58009800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c5800a430 0x7f9c58013dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58008eb0 auth_method 0 2023-10-30T12:12:11.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c5800a430 auth_method 0 2023-10-30T12:12:11.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c5800a810 0x7f9c58005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c5800a810 auth_method 0 2023-10-30T12:12:11.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient(hunting): handle_auth_done global_id 114635 payload 274 2023-10-30T12:12:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c5800a810 msgr2=0x7f9c58005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c5800a810 0x7f9c58005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c5800a430 msgr2=0x7f9c58013dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c5800a430 0x7f9c58013dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 1 monclient: found mon.noname-b 2023-10-30T12:12:11.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:11.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6affd640 1 -- 172.21.15.50:0/1819004771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580086f0 con 0x7f9c58008eb0 2023-10-30T12:12:11.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.039+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c5800a430 0x7f9c58013dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:11.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c58008eb0 0x7f9c58009800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009f20 tx=0x7f9c6402df80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc85f43e8ab76422 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 1 -- 172.21.15.50:0/1819004771 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6402eb20 con 0x7f9c58008eb0 2023-10-30T12:12:11.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:11.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: dump: 2023-10-30T12:12:11.059 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.061 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 1 -- 172.21.15.50:0/1819004771 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6401e040 con 0x7f9c58008eb0 2023-10-30T12:12:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c7134c640 5 monclient: authenticate success, global_id 114635 2023-10-30T12:12:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c7134c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c7134c640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 1 -- 172.21.15.50:0/1819004771 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64036440 con 0x7f9c58008eb0 2023-10-30T12:12:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c7134c640 4 set_mon_vals no callback set 2023-10-30T12:12:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.040+0000 7f9c69ffb640 10 monclient: dump: 2023-10-30T12:12:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.065 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 10 monclient: shutdown 2023-10-30T12:12:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 -- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c58008eb0 msgr2=0x7f9c58009800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c58008eb0 0x7f9c58009800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009f20 tx=0x7f9c6402df80 comp rx=0 tx=0).stop 2023-10-30T12:12:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 -- 172.21.15.50:0/1819004771 shutdown_connections 2023-10-30T12:12:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.118:3300/0 conn(0x7f9c5800a810 0x7f9c58005b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1819004771 >> v2:172.21.15.50:3300/0 conn(0x7f9c5800a430 0x7f9c58013dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1819004771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c58008eb0 0x7f9c58009800 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 -- 172.21.15.50:0/1819004771 >> 172.21.15.50:0/1819004771 conn(0x7f9c6c0a05c0 msgr2=0x7f9c6c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 -- 172.21.15.50:0/1819004771 shutdown_connections 2023-10-30T12:12:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.041+0000 7f9c7134c640 1 -- 172.21.15.50:0/1819004771 wait complete. 2023-10-30T12:12:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.042+0000 7f9c7134c640 10 monclient: build_initial_monmap 2023-10-30T12:12:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.042+0000 7f9c7134c640 10 monclient: monmap: 2023-10-30T12:12:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:11.070 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.043+0000 7f9c7134c640 1 librados: starting msgr at 2023-10-30T12:12:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.043+0000 7f9c7134c640 1 librados: starting objecter 2023-10-30T12:12:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.043+0000 7f9c7134c640 1 Processor -- start 2023-10-30T12:12:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.043+0000 7f9c7134c640 1 -- start start 2023-10-30T12:12:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.043+0000 7f9c7134c640 1 librados: setting wanted keys 2023-10-30T12:12:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.043+0000 7f9c7134c640 1 librados: calling monclient init 2023-10-30T12:12:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.043+0000 7f9c7134c640 10 monclient: init 2023-10-30T12:12:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:12:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c58008eb0 0x7f9c6c08b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): picked mon.noname-a con 0x7f9c58008eb0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c5800a430 0x7f9c6c08bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5800a430 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c095990 0x7f9c6c1c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6c095990 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c08c470 con 0x7f9c58008eb0 2023-10-30T12:12:11.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c1bb780 con 0x7f9c5800a430 2023-10-30T12:12:11.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c095800 con 0x7f9c6c095990 2023-10-30T12:12:11.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c7134c640 10 monclient(hunting): authenticate will time out at 1576.227289s 2023-10-30T12:12:11.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c58008eb0 0x7f9c6c08b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c58008eb0 0x7f9c6c08b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41410/0 (socket says 172.21.15.50:41410) 2023-10-30T12:12:11.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 -- 172.21.15.50:0/1046432543 learned_addr learned my addr 172.21.15.50:0/1046432543 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:11.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): get_auth_request con 0x7f9c58008eb0 auth_method 0 2023-10-30T12:12:11.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c095990 0x7f9c6c1c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c6c095990 auth_method 0 2023-10-30T12:12:11.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c5800a430 0x7f9c6c08bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c5800a430 auth_method 0 2023-10-30T12:12:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient(hunting): handle_auth_done global_id 164100 payload 824 2023-10-30T12:12:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 -- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c095990 msgr2=0x7f9c6c1c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c095990 0x7f9c6c1c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 -- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c5800a430 msgr2=0x7f9c6c08bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c5800a430 0x7f9c6c08bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 monclient: found mon.noname-a 2023-10-30T12:12:11.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6affd640 1 -- 172.21.15.50:0/1046432543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58009290 con 0x7f9c58008eb0 2023-10-30T12:12:11.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.044+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c5800a430 0x7f9c6c08bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:12:11.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c6affd640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c58008eb0 0x7f9c6c08b7f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c64039040 tx=0x7f9c640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 client.?.objecter ms_handle_connect 0x7f9c58008eb0 2023-10-30T12:12:11.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:11.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 1 -- 172.21.15.50:0/1046432543 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64036a80 con 0x7f9c58008eb0 2023-10-30T12:12:11.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:11.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: dump: 2023-10-30T12:12:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 1 -- 172.21.15.50:0/1046432543 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6400c040 con 0x7f9c58008eb0 2023-10-30T12:12:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 1 -- 172.21.15.50:0/1046432543 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64036de0 con 0x7f9c58008eb0 2023-10-30T12:12:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c537fe640 10 monclient: dump: 2023-10-30T12:12:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.086 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.086 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.086 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.086 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.086 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.086 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.087 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 5 monclient: authenticate success, global_id 164100 2023-10-30T12:12:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 10 monclient: _renew_subs 2023-10-30T12:12:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c697fa640 4 set_mon_vals no callback set 2023-10-30T12:12:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6400e040 con 0x7f9c58008eb0 2023-10-30T12:12:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 10 client.164100.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 10 monclient: _renew_subs 2023-10-30T12:12:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c08d7b0 con 0x7f9c58008eb0 2023-10-30T12:12:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.045+0000 7f9c7134c640 1 librados: init done 2023-10-30T12:12:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c537fe640 1 -- 172.21.15.50:0/1046432543 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9c6403d040 con 0x7f9c58008eb0 2023-10-30T12:12:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c537fe640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c5803bf30 0x7f9c5803e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c537fe640 1 -- 172.21.15.50:0/1046432543 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f9c6406e260 con 0x7f9c58008eb0 2023-10-30T12:12:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c537fe640 10 client.164100.objecter ms_dispatch 0x7f9c6c000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c7134c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c537fe640 3 client.164100.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c537fe640 3 client.164100.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c7134c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c7134c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c04e160 con 0x7f9c58008eb0 2023-10-30T12:12:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.046+0000 7f9c537fe640 20 client.164100.objecter dump_active .. 0 homeless 2023-10-30T12:12:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.047+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c5803bf30 0x7f9c5803e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.047+0000 7f9c6a7fc640 10 monclient: get_auth_request con 0x7f9c5803bf30 auth_method 0 2023-10-30T12:12:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.047+0000 7f9c6a7fc640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c5803bf30 0x7f9c5803e3f0 secure :-1 s=READY pgs=4125 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c070ca0 tx=0x7f9c60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.047+0000 7f9c537fe640 10 client.164100.objecter ms_handle_connect 0x7f9c5803bf30 2023-10-30T12:12:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.052+0000 7f9c537fe640 1 -- 172.21.15.50:0/1046432543 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c6403b080 con 0x7f9c58008eb0 2023-10-30T12:12:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.052+0000 7f9c537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.052+0000 7f9c537fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:11.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.297+0000 7f9c7134c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:11.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.297+0000 7f9c7134c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:11.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.297+0000 7f9c7134c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.297+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9c6c0745c0 con 0x7f9c58008eb0 2023-10-30T12:12:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.298+0000 7f9c537fe640 1 -- 172.21.15.50:0/1046432543 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f9c640163d0 con 0x7f9c58008eb0 2023-10-30T12:12:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.298+0000 7f9c537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.298+0000 7f9c537fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:11.301 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 10 librados: watch_flush enter 2023-10-30T12:12:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 10 librados: watch_flush exit 2023-10-30T12:12:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 20 client.164100.objecter shutdown clearing up homeless session... 2023-10-30T12:12:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 10 client.164100.objecter successfully canceled tick 2023-10-30T12:12:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c5803bf30 msgr2=0x7f9c5803e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c5803bf30 0x7f9c5803e3f0 secure :-1 s=READY pgs=4125 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c070ca0 tx=0x7f9c60008040 comp rx=0 tx=0).stop 2023-10-30T12:12:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 10 monclient: shutdown 2023-10-30T12:12:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c58008eb0 msgr2=0x7f9c6c08b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c58008eb0 0x7f9c6c08b7f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c64039040 tx=0x7f9c640046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 shutdown_connections 2023-10-30T12:12:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9c5803bf30 0x7f9c5803e3f0 unknown :-1 s=CLOSED pgs=4125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.299+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9c6c095990 0x7f9c6c1c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.300+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9c5800a430 0x7f9c6c08bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.300+0000 7f9c7134c640 1 --2- 172.21.15.50:0/1046432543 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c58008eb0 0x7f9c6c08b7f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.300+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 >> 172.21.15.50:0/1046432543 conn(0x7f9c6c0a05c0 msgr2=0x7f9c6c0a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.300+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 shutdown_connections 2023-10-30T12:12:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.300+0000 7f9c7134c640 1 -- 172.21.15.50:0/1046432543 wait complete. 2023-10-30T12:12:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.300+0000 7f9c7134c640 1 librados: shutdown 2023-10-30T12:12:11.311 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:12:11.312 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:12:11.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:12:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.417+0000 7f75c59a0640 10 monclient: get_monmap_and_config 2023-10-30T12:12:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.417+0000 7f75c59a0640 10 monclient: build_initial_monmap 2023-10-30T12:12:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.417+0000 7f75c59a0640 10 monclient: monmap: 2023-10-30T12:12:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:11.418187+0000 2023-10-30T12:12:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:11.418187+0000 2023-10-30T12:12:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:11.421 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.417+0000 7f75c59a0640 1 Processor -- start 2023-10-30T12:12:11.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.417+0000 7f75c59a0640 1 -- start start 2023-10-30T12:12:11.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.417+0000 7f75c59a0640 10 monclient: init 2023-10-30T12:12:11.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:11.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:12:11.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c0135950 0x7f75c0135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient(hunting): picked mon.noname-b con 0x7f75c0135950 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:11.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f75c0135460 0x7f75c00a2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient(hunting): picked mon.noname-c con 0x7f75c0135460 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:11.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75c00a2780 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:11.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c009a830 con 0x7f75c00a2780 2023-10-30T12:12:11.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c00a15e0 con 0x7f75c0135460 2023-10-30T12:12:11.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c009a270 con 0x7f75c0135950 2023-10-30T12:12:11.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c59a0640 10 monclient(hunting): authenticate will time out at 1576.601349s 2023-10-30T12:12:11.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c0135950 0x7f75c0135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c0135950 0x7f75c0135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45120/0 (socket says 172.21.15.50:45120) 2023-10-30T12:12:11.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 -- 172.21.15.50:0/1618689858 learned_addr learned my addr 172.21.15.50:0/1618689858 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:11.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): get_auth_request con 0x7f75c0135950 auth_method 0 2023-10-30T12:12:11.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75bffff640 1 --2- 172.21.15.50:0/1618689858 >> v2:172.21.15.118:3300/0 conn(0x7f75c0135460 0x7f75c00a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient(hunting): handle_auth_done global_id 114641 payload 274 2023-10-30T12:12:11.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 -- 172.21.15.50:0/1618689858 >> v2:172.21.15.118:3300/0 conn(0x7f75c0135460 msgr2=0x7f75c00a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:11.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 --2- 172.21.15.50:0/1618689858 >> v2:172.21.15.118:3300/0 conn(0x7f75c0135460 0x7f75c00a2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 -- 172.21.15.50:0/1618689858 >> v1:172.21.15.118:6789/0 conn(0x7f75c00a2780 legacy=0x7f75c013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:11.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 monclient: found mon.noname-b 2023-10-30T12:12:11.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:11.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.418+0000 7f75c499e640 1 -- 172.21.15.50:0/1618689858 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c009a550 con 0x7f75c0135950 2023-10-30T12:12:11.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75c499e640 1 --2- 172.21.15.50:0/1618689858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c0135950 0x7f75c0135d30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009e50 tx=0x7f75b802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=341a4ea385d91e81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 1 -- 172.21.15.50:0/1618689858 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b802c9e0 con 0x7f75c0135950 2023-10-30T12:12:11.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:11.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: dump: 2023-10-30T12:12:11.437 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.437 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.437 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.437 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.437 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.438 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.438 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.438 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.438 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.439 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 1 -- 172.21.15.50:0/1618689858 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b802cb40 con 0x7f75c0135950 2023-10-30T12:12:11.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75c59a0640 5 monclient: authenticate success, global_id 114641 2023-10-30T12:12:11.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75c59a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:11.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75c59a0640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:11.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 1 -- 172.21.15.50:0/1618689858 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b80343f0 con 0x7f75c0135950 2023-10-30T12:12:11.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75c59a0640 4 set_mon_vals no callback set 2023-10-30T12:12:11.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:11.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.419+0000 7f75bf7fe640 10 monclient: dump: 2023-10-30T12:12:11.441 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.441 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.441 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.442 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.442 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.442 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.442 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.442 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 10 monclient: shutdown 2023-10-30T12:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 -- 172.21.15.50:0/1618689858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c0135950 msgr2=0x7f75c0135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 --2- 172.21.15.50:0/1618689858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c0135950 0x7f75c0135d30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009e50 tx=0x7f75b802be40 comp rx=0 tx=0).stop 2023-10-30T12:12:11.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 -- 172.21.15.50:0/1618689858 shutdown_connections 2023-10-30T12:12:11.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 --2- 172.21.15.50:0/1618689858 >> v2:172.21.15.118:3300/0 conn(0x7f75c0135460 0x7f75c00a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 --2- 172.21.15.50:0/1618689858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c0135950 0x7f75c0135d30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 -- 172.21.15.50:0/1618689858 >> 172.21.15.50:0/1618689858 conn(0x7f75c00a7e70 msgr2=0x7f75c00a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 -- 172.21.15.50:0/1618689858 shutdown_connections 2023-10-30T12:12:11.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 1 -- 172.21.15.50:0/1618689858 wait complete. 2023-10-30T12:12:11.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 10 monclient: build_initial_monmap 2023-10-30T12:12:11.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.420+0000 7f75c59a0640 10 monclient: monmap: 2023-10-30T12:12:11.445 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:11.445 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:11.445 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:11.446 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:11.446 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:11.446 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.446 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:11.446 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:11.446 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:11.447 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 librados: starting msgr at 2023-10-30T12:12:11.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 librados: starting objecter 2023-10-30T12:12:11.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 Processor -- start 2023-10-30T12:12:11.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 -- start start 2023-10-30T12:12:11.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 librados: setting wanted keys 2023-10-30T12:12:11.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 librados: calling monclient init 2023-10-30T12:12:11.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient: init 2023-10-30T12:12:11.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:11.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:12:11.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c00a2780 0x7f75c01ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): picked mon.noname-c con 0x7f75c00a2780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:11.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75c0135460 0x7f75c01cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): picked mon.noname-b con 0x7f75c0135460 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:11.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0135950 0x7f75c011f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): picked mon.noname-a con 0x7f75c0135950 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:11.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c0123200 con 0x7f75c0135950 2023-10-30T12:12:11.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c0123450 con 0x7f75c0135460 2023-10-30T12:12:11.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c0121cc0 con 0x7f75c00a2780 2023-10-30T12:12:11.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c59a0640 10 monclient(hunting): authenticate will time out at 1576.604925s 2023-10-30T12:12:11.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.421+0000 7f75c499e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c00a2780 0x7f75c01ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c00a2780 0x7f75c01ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45122/0 (socket says 172.21.15.50:45122) 2023-10-30T12:12:11.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 -- 172.21.15.50:0/4149619722 learned_addr learned my addr 172.21.15.50:0/4149619722 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:11.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): get_auth_request con 0x7f75c00a2780 auth_method 0 2023-10-30T12:12:11.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0135950 0x7f75c011f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 10 monclient(hunting): get_auth_request con 0x7f75c0135950 auth_method 0 2023-10-30T12:12:11.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bffff640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75c0135460 0x7f75c01cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c519f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bffff640 10 monclient(hunting): get_auth_request con 0x7f75c0135460 auth_method 0 2023-10-30T12:12:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient(hunting): handle_auth_done global_id 114644 payload 824 2023-10-30T12:12:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75c0135460 msgr2=0x7f75c01cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75c0135460 0x7f75c01cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0135950 msgr2=0x7f75c011f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0135950 0x7f75c011f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 monclient: found mon.noname-c 2023-10-30T12:12:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 -- 172.21.15.50:0/4149619722 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8009b30 con 0x7f75c00a2780 2023-10-30T12:12:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bffff640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f75c0135460 0x7f75c01cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:12:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c499e640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c00a2780 0x7f75c01ca5e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f75b8036040 tx=0x7f75b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 client.?.objecter ms_handle_connect 0x7f75c00a2780 2023-10-30T12:12:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:11.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8040670 con 0x7f75c00a2780 2023-10-30T12:12:11.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:11.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: dump: 2023-10-30T12:12:11.458 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.459 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.459 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.459 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.459 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.459 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.459 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.460 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.460 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.460 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8040810 con 0x7f75c00a2780 2023-10-30T12:12:11.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 5 monclient: authenticate success, global_id 114644 2023-10-30T12:12:11.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 10 monclient: _renew_subs 2023-10-30T12:12:11.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:11.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c00a1720 con 0x7f75c00a2780 2023-10-30T12:12:11.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8040bc0 con 0x7f75c00a2780 2023-10-30T12:12:11.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:11.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75bd7fa640 10 monclient: dump: 2023-10-30T12:12:11.463 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.463 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.463 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.463 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.463 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.463 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.464 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.464 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.464 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.464 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 10 client.114644.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:11.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 10 monclient: _renew_subs 2023-10-30T12:12:11.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:11.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c0121080 con 0x7f75c00a2780 2023-10-30T12:12:11.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75c59a0640 1 librados: init done 2023-10-30T12:12:11.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.422+0000 7f75beffd640 4 set_mon_vals no callback set 2023-10-30T12:12:11.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75c59a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f75b8040d20 con 0x7f75c00a2780 2023-10-30T12:12:11.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75c59a0640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c011f920 0x7f75c011fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75c59a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c01235e0 con 0x7f75c011f920 2023-10-30T12:12:11.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c011f920 0x7f75c011fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient: get_auth_request con 0x7f75c011f920 auth_method 0 2023-10-30T12:12:11.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bd7fa640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75ac0392f0 0x7f75ac03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f75b806de80 con 0x7f75c00a2780 2023-10-30T12:12:11.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bd7fa640 10 client.114644.objecter ms_dispatch 0x7f75c0000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:11.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bd7fa640 3 client.114644.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:11.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bd7fa640 3 client.114644.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:11.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bd7fa640 20 client.114644.objecter dump_active .. 0 homeless 2023-10-30T12:12:11.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 10 monclient(hunting): handle_auth_done global_id 164106 payload 824 2023-10-30T12:12:11.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75c519f640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75ac0392f0 0x7f75ac03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75bffff640 1 -- 172.21.15.50:0/4149619722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75c0099fd0 con 0x7f75c011f920 2023-10-30T12:12:11.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.423+0000 7f75c519f640 10 monclient: get_auth_request con 0x7f75ac0392f0 auth_method 0 2023-10-30T12:12:11.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 client.114644.objecter ms_handle_connect 0x7f75c011f920 2023-10-30T12:12:11.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 client.114644.objecter resend_mon_ops 2023-10-30T12:12:11.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bffff640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c011f920 0x7f75c011fd00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75b400f4a0 tx=0x7f75b400f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4014890 con 0x7f75c011f920 2023-10-30T12:12:11.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:11.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 monclient: dump: 2023-10-30T12:12:11.471 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.471 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.471 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.471 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.472 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.472 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.472 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.472 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.472 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.472 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 client.114644.objecter ms_handle_connect 0x7f75ac0392f0 2023-10-30T12:12:11.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75c519f640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75ac0392f0 0x7f75ac03b7b0 secure :-1 s=READY pgs=4126 cs=0 l=1 rev1=1 crypto rx=0x7f75c0132c80 tx=0x7f75b0007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f75b400c030 con 0x7f75c011f920 2023-10-30T12:12:11.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:11.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c011f920 msgr2=0x7f75c011fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c011f920 0x7f75c011fd00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75b400f4a0 tx=0x7f75b400f960 comp rx=0 tx=0).stop 2023-10-30T12:12:11.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c011f920 msgr2=0x7f75c011fd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:11.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.424+0000 7f75bd7fa640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c011f920 0x7f75c011fd00 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75b400f4a0 tx=0x7f75b400f960 comp rx=0 tx=0).stop 2023-10-30T12:12:11.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.430+0000 7f75c59a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:11.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.430+0000 7f75c59a0640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0120240 0x7f75c009dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.430+0000 7f75c59a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.430+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c0066190 con 0x7f75c0120240 2023-10-30T12:12:11.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.430+0000 7f75bffff640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0120240 0x7f75c009dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient: get_auth_request con 0x7f75c0120240 auth_method 0 2023-10-30T12:12:11.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 10 monclient(hunting): handle_auth_done global_id 164109 payload 824 2023-10-30T12:12:11.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 1 -- 172.21.15.50:0/4149619722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f75c00a7270 con 0x7f75c0120240 2023-10-30T12:12:11.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bffff640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0120240 0x7f75c009dd30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f75b401a000 tx=0x7f75b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 10 client.114644.objecter ms_handle_connect 0x7f75c0120240 2023-10-30T12:12:11.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 10 client.114644.objecter resend_mon_ops 2023-10-30T12:12:11.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b40275f0 con 0x7f75c0120240 2023-10-30T12:12:11.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:11.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 10 monclient: dump: 2023-10-30T12:12:11.480 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.480 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.480 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.481 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.481 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.481 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.481 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.481 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.482 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.482 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f75b4023040 con 0x7f75c0120240 2023-10-30T12:12:11.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:11.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:11.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0120240 msgr2=0x7f75c009dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0120240 0x7f75c009dd30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f75b401a000 tx=0x7f75b40046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0120240 msgr2=0x7f75c009dd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:11.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75bd7fa640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75c0120240 0x7f75c009dd30 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f75b401a000 tx=0x7f75b40046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.483 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:12:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 114, 2023-10-30T12:12:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:12:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:12:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:11.512 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:11.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75c59a0640 10 librados: watch_flush enter 2023-10-30T12:12:11.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75c59a0640 10 librados: watch_flush exit 2023-10-30T12:12:11.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75c59a0640 20 client.114644.objecter shutdown clearing up homeless session... 2023-10-30T12:12:11.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.431+0000 7f75c59a0640 10 client.114644.objecter successfully canceled tick 2023-10-30T12:12:11.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75ac0392f0 msgr2=0x7f75ac03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f75ac0392f0 0x7f75ac03b7b0 secure :-1 s=READY pgs=4126 cs=0 l=1 rev1=1 crypto rx=0x7f75c0132c80 tx=0x7f75b0007db0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 10 monclient: shutdown 2023-10-30T12:12:11.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c00a2780 msgr2=0x7f75c01ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 --2- 172.21.15.50:0/4149619722 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f75c00a2780 0x7f75c01ca5e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f75b8036040 tx=0x7f75b8004a40 comp rx=0 tx=0).stop 2023-10-30T12:12:11.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c519f640 1 -- 172.21.15.50:0/4149619722 reap_dead start 2023-10-30T12:12:11.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c519f640 1 -- 172.21.15.50:0/4149619722 reap_dead start 2023-10-30T12:12:11.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 shutdown_connections 2023-10-30T12:12:11.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 >> 172.21.15.50:0/4149619722 conn(0x7f75c00a7e70 msgr2=0x7f75c00a5740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 shutdown_connections 2023-10-30T12:12:11.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 -- 172.21.15.50:0/4149619722 wait complete. 2023-10-30T12:12:11.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.432+0000 7f75c59a0640 1 librados: shutdown 2023-10-30T12:12:11.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:12:11.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:12:11.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:12:11.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.624+0000 7f9b365d0640 10 monclient: get_monmap_and_config 2023-10-30T12:12:11.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.624+0000 7f9b365d0640 10 monclient: build_initial_monmap 2023-10-30T12:12:11.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.624+0000 7f9b365d0640 10 monclient: monmap: 2023-10-30T12:12:11.626 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:11.626 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.627 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:11.625584+0000 2023-10-30T12:12:11.627 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:11.625584+0000 2023-10-30T12:12:11.627 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:11.627 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.627 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:11.628 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:11.628 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:11.628 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:11.628 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:11.628 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 1 Processor -- start 2023-10-30T12:12:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 1 -- start start 2023-10-30T12:12:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient: init 2023-10-30T12:12:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T12:12:11.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b30132970 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:11.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b300a03d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:11.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9b3013c200 0x7f9b3013e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3013c200 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:11.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300a17d0 con 0x7f9b300a03d0 2023-10-30T12:12:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300a1510 con 0x7f9b30132970 2023-10-30T12:12:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b30133460 con 0x7f9b3013c200 2023-10-30T12:12:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b365d0640 10 monclient(hunting): authenticate will time out at 1576.808885s 2023-10-30T12:12:11.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b34b6d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9b3013c200 0x7f9b3013e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.625+0000 7f9b2f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9b300a03d0 0x7f9b3013aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50560/0 (socket says 172.21.15.50:50560) 2023-10-30T12:12:11.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2362131347 learned_addr learned my addr 172.21.15.50:0/2362131347 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:12:11.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): get_auth_request con 0x7f9b3013c200 auth_method 0 2023-10-30T12:12:11.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient(hunting): handle_auth_done global_id 164112 payload 274 2023-10-30T12:12:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 1 -- 172.21.15.50:0/2362131347 >> v1:172.21.15.118:6789/0 conn(0x7f9b30132970 legacy=0x7f9b30132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 1 -- 172.21.15.50:0/2362131347 >> v1:172.21.15.50:6789/0 conn(0x7f9b300a03d0 legacy=0x7f9b3013aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 1 monclient: found mon.noname-a 2023-10-30T12:12:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b2effd640 1 -- 172.21.15.50:0/2362131347 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338498644 0 0) 0x7f9b300a17d0 con 0x7f9b300a03d0 2023-10-30T12:12:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b34b6d640 1 -- 172.21.15.50:0/2362131347 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3009f680 con 0x7f9b3013c200 2023-10-30T12:12:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.626+0000 7f9b2effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:12:11.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b34b6d640 1 --2- 172.21.15.50:0/2362131347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3013c200 0x7f9b3013e5f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009d90 tx=0x7f9b2802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51d75d30dd262ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 1 -- 172.21.15.50:0/2362131347 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2802cb40 con 0x7f9b3013c200 2023-10-30T12:12:11.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:11.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: dump: 2023-10-30T12:12:11.645 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.645 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.647 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.647 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.647 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 1 -- 172.21.15.50:0/2362131347 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2802cca0 con 0x7f9b3013c200 2023-10-30T12:12:11.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 5 monclient: authenticate success, global_id 164112 2023-10-30T12:12:11.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:11.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 1 -- 172.21.15.50:0/2362131347 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28034660 con 0x7f9b3013c200 2023-10-30T12:12:11.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:11.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b2effd640 10 monclient: dump: 2023-10-30T12:12:11.649 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.649 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 4 set_mon_vals no callback set 2023-10-30T12:12:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 10 monclient: shutdown 2023-10-30T12:12:11.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 1 -- 172.21.15.50:0/2362131347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3013c200 msgr2=0x7f9b3013e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 1 --2- 172.21.15.50:0/2362131347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3013c200 0x7f9b3013e5f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009d90 tx=0x7f9b2802bd90 comp rx=0 tx=0).stop 2023-10-30T12:12:11.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 1 -- 172.21.15.50:0/2362131347 shutdown_connections 2023-10-30T12:12:11.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 1 --2- 172.21.15.50:0/2362131347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3013c200 0x7f9b3013e5f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 1 -- 172.21.15.50:0/2362131347 >> 172.21.15.50:0/2362131347 conn(0x7f9b300a4800 msgr2=0x7f9b300a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 1 -- 172.21.15.50:0/2362131347 shutdown_connections 2023-10-30T12:12:11.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.627+0000 7f9b365d0640 1 -- 172.21.15.50:0/2362131347 wait complete. 2023-10-30T12:12:11.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 10 monclient: build_initial_monmap 2023-10-30T12:12:11.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 10 monclient: monmap: 2023-10-30T12:12:11.654 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:11.654 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:11.654 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:11.654 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:11.654 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:11.655 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.655 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:11.655 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:11.655 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:11.655 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 1 librados: starting msgr at 2023-10-30T12:12:11.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 1 librados: starting objecter 2023-10-30T12:12:11.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 1 Processor -- start 2023-10-30T12:12:11.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 1 -- start start 2023-10-30T12:12:11.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 1 librados: setting wanted keys 2023-10-30T12:12:11.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 1 librados: calling monclient init 2023-10-30T12:12:11.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.628+0000 7f9b365d0640 10 monclient: init 2023-10-30T12:12:11.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:11.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:11.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30132970 0x7f9b301311a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): picked mon.noname-a con 0x7f9b30132970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:11.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013c200 0x7f9b30139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): picked mon.noname-c con 0x7f9b3013c200 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:11.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b3013a480 0x7f9b3013a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3013a480 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:11.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3011c8f0 con 0x7f9b30132970 2023-10-30T12:12:11.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3011cb40 con 0x7f9b3013a480 2023-10-30T12:12:11.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3009d7c0 con 0x7f9b3013c200 2023-10-30T12:12:11.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b365d0640 10 monclient(hunting): authenticate will time out at 1576.812444s 2023-10-30T12:12:11.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013c200 0x7f9b30139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30132970 0x7f9b301311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30132970 0x7f9b301311a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41450/0 (socket says 172.21.15.50:41450) 2023-10-30T12:12:11.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013c200 0x7f9b30139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45126/0 (socket says 172.21.15.50:45126) 2023-10-30T12:12:11.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 -- 172.21.15.50:0/2015302433 learned_addr learned my addr 172.21.15.50:0/2015302433 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:11.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): get_auth_request con 0x7f9b30132970 auth_method 0 2023-10-30T12:12:11.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b3013c200 auth_method 0 2023-10-30T12:12:11.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b34b6d640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b3013a480 0x7f9b3013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b34b6d640 10 monclient(hunting): get_auth_request con 0x7f9b3013a480 auth_method 0 2023-10-30T12:12:11.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b34b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b34b6d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b34b6d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient(hunting): handle_auth_done global_id 164118 payload 824 2023-10-30T12:12:11.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013c200 msgr2=0x7f9b30139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013c200 0x7f9b30139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b3013a480 msgr2=0x7f9b3013a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b3013a480 0x7f9b3013a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 monclient: found mon.noname-a 2023-10-30T12:12:11.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2ffff640 1 -- 172.21.15.50:0/2015302433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b28009a70 con 0x7f9b30132970 2023-10-30T12:12:11.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013c200 0x7f9b30139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:11.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.629+0000 7f9b34b6d640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b3013a480 0x7f9b3013a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:12:11.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2ffff640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30132970 0x7f9b301311a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c460 tx=0x7f9b2000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 client.?.objecter ms_handle_connect 0x7f9b30132970 2023-10-30T12:12:11.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:11.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20012830 con 0x7f9b30132970 2023-10-30T12:12:11.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:11.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: dump: 2023-10-30T12:12:11.668 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.668 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.668 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.668 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.669 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.669 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.669 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.669 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.669 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.670 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b200129d0 con 0x7f9b30132970 2023-10-30T12:12:11.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 5 monclient: authenticate success, global_id 164118 2023-10-30T12:12:11.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20012d80 con 0x7f9b30132970 2023-10-30T12:12:11.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:11.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2cff9640 10 monclient: dump: 2023-10-30T12:12:11.672 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.672 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.672 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.672 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.672 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.673 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.673 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.673 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.673 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 10 monclient: _renew_subs 2023-10-30T12:12:11.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b2e7fc640 4 set_mon_vals no callback set 2023-10-30T12:12:11.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3013b850 con 0x7f9b30132970 2023-10-30T12:12:11.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 10 client.164118.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:11.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 10 monclient: _renew_subs 2023-10-30T12:12:11.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3011b130 con 0x7f9b30132970 2023-10-30T12:12:11.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.630+0000 7f9b365d0640 1 librados: init done 2023-10-30T12:12:11.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9b2001b400 con 0x7f9b30132970 2023-10-30T12:12:11.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b365d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b365d0640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a03d0 0x7f9b3002c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b365d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3011ccd0 con 0x7f9b300a03d0 2023-10-30T12:12:11.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a03d0 0x7f9b3002c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient: get_auth_request con 0x7f9b300a03d0 auth_method 0 2023-10-30T12:12:11.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f9b2004ae10 con 0x7f9b30132970 2023-10-30T12:12:11.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 10 client.164118.objecter ms_dispatch 0x7f9b30000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:11.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 3 client.164118.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:11.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 3 client.164118.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:11.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 20 client.164118.objecter dump_active .. 0 homeless 2023-10-30T12:12:11.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_done global_id 114650 payload 824 2023-10-30T12:12:11.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2015302433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b3009f960 con 0x7f9b300a03d0 2023-10-30T12:12:11.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b34b6d640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b34b6d640 10 monclient: get_auth_request con 0x7f9b1c0392f0 auth_method 0 2023-10-30T12:12:11.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a03d0 0x7f9b3002c5b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009e010 tx=0x7f9b24001d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 10 client.164118.objecter ms_handle_connect 0x7f9b300a03d0 2023-10-30T12:12:11.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 10 client.164118.objecter resend_mon_ops 2023-10-30T12:12:11.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2400fac0 con 0x7f9b300a03d0 2023-10-30T12:12:11.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:11.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.631+0000 7f9b2cff9640 10 monclient: dump: 2023-10-30T12:12:11.681 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.681 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.681 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.681 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.681 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.682 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.682 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.682 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.682 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.682 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b34b6d640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 secure :-1 s=READY pgs=4127 cs=0 l=1 rev1=1 crypto rx=0x7f9b28036000 tx=0x7f9b28030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 10 client.164118.objecter ms_handle_connect 0x7f9b1c0392f0 2023-10-30T12:12:11.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9b2400fc20 con 0x7f9b300a03d0 2023-10-30T12:12:11.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:11.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a03d0 msgr2=0x7f9b3002c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a03d0 0x7f9b3002c5b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009e010 tx=0x7f9b24001d40 comp rx=0 tx=0).stop 2023-10-30T12:12:11.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a03d0 msgr2=0x7f9b3002c5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:11.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.632+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a03d0 0x7f9b3002c5b0 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009e010 tx=0x7f9b24001d40 comp rx=0 tx=0).stop 2023-10-30T12:12:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b365d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b365d0640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013ae70 0x7f9b30060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b365d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30071a70 con 0x7f9b3013ae70 2023-10-30T12:12:11.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013ae70 0x7f9b30060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient: get_auth_request con 0x7f9b3013ae70 auth_method 0 2023-10-30T12:12:11.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_done global_id 114653 payload 824 2023-10-30T12:12:11.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2015302433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9b3004e160 con 0x7f9b3013ae70 2023-10-30T12:12:11.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2cff9640 10 client.164118.objecter ms_handle_connect 0x7f9b3013ae70 2023-10-30T12:12:11.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013ae70 0x7f9b30060fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9b24014000 tx=0x7f9b24001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.638+0000 7f9b2cff9640 10 client.164118.objecter resend_mon_ops 2023-10-30T12:12:11.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b24022070 con 0x7f9b3013ae70 2023-10-30T12:12:11.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:11.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 10 monclient: dump: 2023-10-30T12:12:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.691 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.691 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.691 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.691 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f9b240136c0 con 0x7f9b3013ae70 2023-10-30T12:12:11.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:11.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:11.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013ae70 msgr2=0x7f9b30060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013ae70 0x7f9b30060fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9b24014000 tx=0x7f9b24001080 comp rx=0 tx=0).stop 2023-10-30T12:12:11.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013ae70 msgr2=0x7f9b30060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:11.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b2cff9640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b3013ae70 0x7f9b30060fc0 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9b24014000 tx=0x7f9b24001080 comp rx=0 tx=0).stop 2023-10-30T12:12:11.693 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:12:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 114, 2023-10-30T12:12:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:12:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:12:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:12:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:12:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:12:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:11.724 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:11.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 10 librados: watch_flush enter 2023-10-30T12:12:11.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 10 librados: watch_flush exit 2023-10-30T12:12:11.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 20 client.164118.objecter shutdown clearing up homeless session... 2023-10-30T12:12:11.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 10 client.164118.objecter successfully canceled tick 2023-10-30T12:12:11.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 msgr2=0x7f9b1c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 secure :-1 s=READY pgs=4127 cs=0 l=1 rev1=1 crypto rx=0x7f9b28036000 tx=0x7f9b28030000 comp rx=0 tx=0).stop 2023-10-30T12:12:11.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 10 monclient: shutdown 2023-10-30T12:12:11.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30132970 msgr2=0x7f9b301311a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 --2- 172.21.15.50:0/2015302433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30132970 0x7f9b301311a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c460 tx=0x7f9b2000c920 comp rx=0 tx=0).stop 2023-10-30T12:12:11.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b34b6d640 1 -- 172.21.15.50:0/2015302433 reap_dead start 2023-10-30T12:12:11.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b34b6d640 1 -- 172.21.15.50:0/2015302433 reap_dead start 2023-10-30T12:12:11.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 shutdown_connections 2023-10-30T12:12:11.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 >> 172.21.15.50:0/2015302433 conn(0x7f9b300a4800 msgr2=0x7f9b300a21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 shutdown_connections 2023-10-30T12:12:11.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 -- 172.21.15.50:0/2015302433 wait complete. 2023-10-30T12:12:11.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.639+0000 7f9b365d0640 1 librados: shutdown 2023-10-30T12:12:11.729 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:12:11.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:12:11.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:12:11.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.834+0000 7f02caada640 10 monclient: get_monmap_and_config 2023-10-30T12:12:11.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.834+0000 7f02caada640 10 monclient: build_initial_monmap 2023-10-30T12:12:11.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.834+0000 7f02caada640 10 monclient: monmap: 2023-10-30T12:12:11.836 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:11.837 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.837 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:11.835782+0000 2023-10-30T12:12:11.837 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:11.835782+0000 2023-10-30T12:12:11.837 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:11.837 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.837 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:11.838 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:11.838 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:11.838 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:11.838 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:11.838 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 1 Processor -- start 2023-10-30T12:12:11.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 1 -- start start 2023-10-30T12:12:11.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient: init 2023-10-30T12:12:11.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:11.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T12:12:11.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02c409bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:11.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f02c409d3f0 0x7f02c4137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient(hunting): picked mon.noname-c con 0x7f02c409d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:11.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409c660 0x7f02c4139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient(hunting): picked mon.noname-b con 0x7f02c409c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:11.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c4098170 con 0x7f02c409bb70 2023-10-30T12:12:11.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c409cf60 con 0x7f02c409d3f0 2023-10-30T12:12:11.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c4097b20 con 0x7f02c409c660 2023-10-30T12:12:11.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.835+0000 7f02caada640 10 monclient(hunting): authenticate will time out at 1577.018970s 2023-10-30T12:12:11.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409c660 0x7f02c4139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409c660 0x7f02c4139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45162/0 (socket says 172.21.15.50:45162) 2023-10-30T12:12:11.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 -- 172.21.15.50:0/3623704519 learned_addr learned my addr 172.21.15.50:0/3623704519 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:11.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): get_auth_request con 0x7f02c409c660 auth_method 0 2023-10-30T12:12:11.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c3fff640 1 --2- 172.21.15.50:0/3623704519 >> v2:172.21.15.118:3300/0 conn(0x7f02c409d3f0 0x7f02c4137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c3fff640 10 monclient(hunting): get_auth_request con 0x7f02c409d3f0 auth_method 0 2023-10-30T12:12:11.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient(hunting): handle_auth_done global_id 114656 payload 274 2023-10-30T12:12:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 -- 172.21.15.50:0/3623704519 >> v2:172.21.15.118:3300/0 conn(0x7f02c409d3f0 msgr2=0x7f02c4137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 --2- 172.21.15.50:0/3623704519 >> v2:172.21.15.118:3300/0 conn(0x7f02c409d3f0 0x7f02c4137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 -- 172.21.15.50:0/3623704519 >> v1:172.21.15.118:6789/0 conn(0x7f02c409bb70 legacy=0x7f02c409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 monclient: found mon.noname-b 2023-10-30T12:12:11.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:11.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.836+0000 7f02c9077640 1 -- 172.21.15.50:0/3623704519 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c4097e00 con 0x7f02c409c660 2023-10-30T12:12:11.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c9077640 1 --2- 172.21.15.50:0/3623704519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409c660 0x7f02c4139eb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00d150 tx=0x7f02bc02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1ffd504bdf79429 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 1 -- 172.21.15.50:0/3623704519 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc02e9e0 con 0x7f02c409c660 2023-10-30T12:12:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: dump: 2023-10-30T12:12:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.855 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.856 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.856 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.856 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.856 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.856 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.857 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 1 -- 172.21.15.50:0/3623704519 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc02eb40 con 0x7f02c409c660 2023-10-30T12:12:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 5 monclient: authenticate success, global_id 114656 2023-10-30T12:12:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 1 -- 172.21.15.50:0/3623704519 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc009600 con 0x7f02c409c660 2023-10-30T12:12:11.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 4 set_mon_vals no callback set 2023-10-30T12:12:11.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:11.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02c37fe640 10 monclient: dump: 2023-10-30T12:12:11.859 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.859 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.860 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.860 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.860 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.860 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.860 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.861 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.861 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.861 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 10 monclient: shutdown 2023-10-30T12:12:11.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 -- 172.21.15.50:0/3623704519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409c660 msgr2=0x7f02c4139eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 --2- 172.21.15.50:0/3623704519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409c660 0x7f02c4139eb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00d150 tx=0x7f02bc02e040 comp rx=0 tx=0).stop 2023-10-30T12:12:11.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 -- 172.21.15.50:0/3623704519 shutdown_connections 2023-10-30T12:12:11.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 --2- 172.21.15.50:0/3623704519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409c660 0x7f02c4139eb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 --2- 172.21.15.50:0/3623704519 >> v2:172.21.15.118:3300/0 conn(0x7f02c409d3f0 0x7f02c4137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 -- 172.21.15.50:0/3623704519 >> 172.21.15.50:0/3623704519 conn(0x7f02c40a3980 msgr2=0x7f02c40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 -- 172.21.15.50:0/3623704519 shutdown_connections 2023-10-30T12:12:11.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.837+0000 7f02caada640 1 -- 172.21.15.50:0/3623704519 wait complete. 2023-10-30T12:12:11.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 10 monclient: build_initial_monmap 2023-10-30T12:12:11.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 10 monclient: monmap: 2023-10-30T12:12:11.863 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:11.863 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:11.864 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:11.864 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:11.864 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:11.864 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.864 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:11.865 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 1 librados: starting msgr at 2023-10-30T12:12:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 1 librados: starting objecter 2023-10-30T12:12:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 1 Processor -- start 2023-10-30T12:12:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 1 -- start start 2023-10-30T12:12:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 1 librados: setting wanted keys 2023-10-30T12:12:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 1 librados: calling monclient init 2023-10-30T12:12:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.838+0000 7f02caada640 10 monclient: init 2023-10-30T12:12:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:12:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02c409bb70 0x7f02c41c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): picked mon.noname-a con 0x7f02c409bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c409c660 0x7f02c4096020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): picked mon.noname-b con 0x7f02c409c660 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409d3f0 0x7f02c4096560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): picked mon.noname-c con 0x7f02c409d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c408d740 con 0x7f02c409bb70 2023-10-30T12:12:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c408f200 con 0x7f02c409c660 2023-10-30T12:12:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c408f4e0 con 0x7f02c409d3f0 2023-10-30T12:12:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02caada640 10 monclient(hunting): authenticate will time out at 1577.022575s 2023-10-30T12:12:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02c409bb70 0x7f02c41c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409d3f0 0x7f02c4096560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02c409bb70 0x7f02c41c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41452/0 (socket says 172.21.15.50:41452) 2023-10-30T12:12:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 1 -- 172.21.15.50:0/707285873 learned_addr learned my addr 172.21.15.50:0/707285873 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409d3f0 0x7f02c4096560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45170/0 (socket says 172.21.15.50:45170) 2023-10-30T12:12:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 10 monclient(hunting): get_auth_request con 0x7f02c409bb70 auth_method 0 2023-10-30T12:12:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 10 monclient(hunting): get_auth_request con 0x7f02c409d3f0 auth_method 0 2023-10-30T12:12:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c8876640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c9077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c409c660 0x7f02c4096020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 10 monclient(hunting): get_auth_request con 0x7f02c409c660 auth_method 0 2023-10-30T12:12:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.839+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 10 monclient(hunting): handle_auth_done global_id 164121 payload 824 2023-10-30T12:12:11.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409d3f0 msgr2=0x7f02c4096560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409d3f0 0x7f02c4096560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c409c660 msgr2=0x7f02c4096020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c409c660 0x7f02c4096020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 10 monclient: _finish_hunting 0 2023-10-30T12:12:11.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 1 monclient: found mon.noname-a 2023-10-30T12:12:11.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 1 -- 172.21.15.50:0/707285873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc00bd40 con 0x7f02c409bb70 2023-10-30T12:12:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c9077640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f02c409d3f0 0x7f02c4096560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c8876640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02c409bb70 0x7f02c41c5e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02b800c450 tx=0x7f02b800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 client.?.objecter ms_handle_connect 0x7f02c409bb70 2023-10-30T12:12:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80052d0 con 0x7f02c409bb70 2023-10-30T12:12:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: dump: 2023-10-30T12:12:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.878 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.878 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.878 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.878 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.878 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.879 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.879 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.879 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: _finish_auth 0 2023-10-30T12:12:11.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: _check_auth_tickets 2023-10-30T12:12:11.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:11.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b8005450 con 0x7f02c409bb70 2023-10-30T12:12:11.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 5 monclient: authenticate success, global_id 164121 2023-10-30T12:12:11.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 10 monclient: _renew_subs 2023-10-30T12:12:11.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c40976e0 con 0x7f02c409bb70 2023-10-30T12:12:11.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:11.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b8003880 con 0x7f02c409bb70 2023-10-30T12:12:11.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:11.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c17fa640 10 monclient: dump: 2023-10-30T12:12:11.881 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.882 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.882 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.882 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.882 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.882 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.883 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.883 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.883 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.883 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02c2ffd640 4 set_mon_vals no callback set 2023-10-30T12:12:11.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 10 client.164121.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 10 monclient: _renew_subs 2023-10-30T12:12:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c408f870 con 0x7f02c409bb70 2023-10-30T12:12:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.840+0000 7f02caada640 1 librados: init done 2023-10-30T12:12:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02caada640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02caada640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c020 0x7f02c408c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02caada640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c408d8b0 con 0x7f02c408c020 2023-10-30T12:12:11.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f02b80039e0 con 0x7f02c409bb70 2023-10-30T12:12:11.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c17fa640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02a80392f0 0x7f02a803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c020 0x7f02c408c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient: get_auth_request con 0x7f02c408c020 auth_method 0 2023-10-30T12:12:11.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f02b804a770 con 0x7f02c409bb70 2023-10-30T12:12:11.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c17fa640 10 client.164121.objecter ms_dispatch 0x7f02c4000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:11.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c17fa640 3 client.164121.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:11.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c17fa640 3 client.164121.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:11.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c9077640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02a80392f0 0x7f02a803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c9077640 10 monclient: get_auth_request con 0x7f02a80392f0 auth_method 0 2023-10-30T12:12:11.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c17fa640 20 client.164121.objecter dump_active .. 0 homeless 2023-10-30T12:12:11.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 10 monclient(hunting): handle_auth_done global_id 154216 payload 824 2023-10-30T12:12:11.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.841+0000 7f02c3fff640 1 -- 172.21.15.50:0/707285873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02c4097560 con 0x7f02c408c020 2023-10-30T12:12:11.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 client.164121.objecter ms_handle_connect 0x7f02c408c020 2023-10-30T12:12:11.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 client.164121.objecter resend_mon_ops 2023-10-30T12:12:11.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c3fff640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c020 0x7f02c408c400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02c4071590 tx=0x7f02b0010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 client.164121.objecter ms_handle_connect 0x7f02a80392f0 2023-10-30T12:12:11.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c9077640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02a80392f0 0x7f02a803b7b0 secure :-1 s=READY pgs=4128 cs=0 l=1 rev1=1 crypto rx=0x7f02bc037000 tx=0x7f02bc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0010e60 con 0x7f02c408c020 2023-10-30T12:12:11.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:11.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 monclient: dump: 2023-10-30T12:12:11.890 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.890 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.890 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.891 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.891 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.891 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.891 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.891 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.892 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.892 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f02b0004240 con 0x7f02c408c020 2023-10-30T12:12:11.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:11.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:11.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c020 msgr2=0x7f02c408c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c020 0x7f02c408c400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02c4071590 tx=0x7f02b0010c10 comp rx=0 tx=0).stop 2023-10-30T12:12:11.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c020 msgr2=0x7f02c408c400 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:11.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.842+0000 7f02c17fa640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c020 0x7f02c408c400 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02c4071590 tx=0x7f02b0010c10 comp rx=0 tx=0).stop 2023-10-30T12:12:11.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.848+0000 7f02caada640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:11.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.848+0000 7f02caada640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c940 0x7f02c406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:11.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.848+0000 7f02caada640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:11.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.848+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c4073ff0 con 0x7f02c408c940 2023-10-30T12:12:11.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c940 0x7f02c406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:11.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient: get_auth_request con 0x7f02c408c940 auth_method 0 2023-10-30T12:12:11.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:11.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:11.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:11.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:11.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:11.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:11.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 10 monclient(hunting): handle_auth_done global_id 154219 payload 824 2023-10-30T12:12:11.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 1 -- 172.21.15.50:0/707285873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02c409f600 con 0x7f02c408c940 2023-10-30T12:12:11.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c3fff640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c940 0x7f02c406e4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f02b0000c00 tx=0x7f02b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:11.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c17fa640 10 client.164121.objecter ms_handle_connect 0x7f02c408c940 2023-10-30T12:12:11.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c17fa640 10 client.164121.objecter resend_mon_ops 2023-10-30T12:12:11.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0025070 con 0x7f02c408c940 2023-10-30T12:12:11.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:11.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:11.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.849+0000 7f02c17fa640 10 monclient: dump: 2023-10-30T12:12:11.899 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:11.899 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:11.899 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.899 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:11.900 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:11.900 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:11.900 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:11.900 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:11.900 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:11.901 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:11.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f02b0003b70 con 0x7f02c408c940 2023-10-30T12:12:11.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:11.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:11.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c940 msgr2=0x7f02c406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c17fa640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c940 0x7f02c406e4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f02b0000c00 tx=0x7f02b00046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c17fa640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c940 msgr2=0x7f02c406e4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:11.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c17fa640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02c408c940 0x7f02c406e4a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:11.902 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:12:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 114, 2023-10-30T12:12:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:12:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:12:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:12:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:12:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:11.932 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:11.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 10 librados: watch_flush enter 2023-10-30T12:12:11.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 10 librados: watch_flush exit 2023-10-30T12:12:11.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 20 client.164121.objecter shutdown clearing up homeless session... 2023-10-30T12:12:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 10 client.164121.objecter successfully canceled tick 2023-10-30T12:12:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02a80392f0 msgr2=0x7f02a803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02a80392f0 0x7f02a803b7b0 secure :-1 s=READY pgs=4128 cs=0 l=1 rev1=1 crypto rx=0x7f02bc037000 tx=0x7f02bc032000 comp rx=0 tx=0).stop 2023-10-30T12:12:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 10 monclient: shutdown 2023-10-30T12:12:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02c409bb70 msgr2=0x7f02c41c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 --2- 172.21.15.50:0/707285873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02c409bb70 0x7f02c41c5e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02b800c450 tx=0x7f02b800c910 comp rx=0 tx=0).stop 2023-10-30T12:12:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c9077640 1 -- 172.21.15.50:0/707285873 reap_dead start 2023-10-30T12:12:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02c9077640 1 -- 172.21.15.50:0/707285873 reap_dead start 2023-10-30T12:12:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 shutdown_connections 2023-10-30T12:12:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 >> 172.21.15.50:0/707285873 conn(0x7f02c40a3980 msgr2=0x7f02c4098ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:11.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 shutdown_connections 2023-10-30T12:12:11.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 -- 172.21.15.50:0/707285873 wait complete. 2023-10-30T12:12:11.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:11.850+0000 7f02caada640 1 librados: shutdown 2023-10-30T12:12:11.937 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:12:11.938 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:12:11.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:12:12.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.041+0000 7fdad1ad4640 10 monclient: get_monmap_and_config 2023-10-30T12:12:12.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.041+0000 7fdad1ad4640 10 monclient: build_initial_monmap 2023-10-30T12:12:12.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.041+0000 7fdad1ad4640 10 monclient: monmap: 2023-10-30T12:12:12.044 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:12.044 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:12.044 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:12.042842+0000 2023-10-30T12:12:12.044 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:12.042842+0000 2023-10-30T12:12:12.044 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:12.045 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:12.045 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:12.045 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:12.045 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:12.045 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:12.046 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:12.046 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:12.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.042+0000 7fdad1ad4640 1 Processor -- start 2023-10-30T12:12:12.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.042+0000 7fdad1ad4640 1 -- start start 2023-10-30T12:12:12.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.042+0000 7fdad1ad4640 10 monclient: init 2023-10-30T12:12:12.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.042+0000 7fdad1ad4640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:12.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.042+0000 7fdad1ad4640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-10-30T12:12:12.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdacc098520 0x7fdacc09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:12.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc098520 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:12.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdacc098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:12.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc1379f0 0x7fdacc139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:12.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc1379f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:12.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc09ad30 con 0x7fdacc098900 2023-10-30T12:12:12.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:12.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdacc09c460 con 0x7fdacc098520 2023-10-30T12:12:12.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:12.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc09b260 con 0x7fdacc1379f0 2023-10-30T12:12:12.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:12.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdad1ad4640 10 monclient(hunting): authenticate will time out at 1577.226100s 2023-10-30T12:12:12.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdacc098520 0x7fdacc09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:12.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc1379f0 0x7fdacc139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:12.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc1379f0 0x7fdacc139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45180/0 (socket says 172.21.15.50:45180) 2023-10-30T12:12:12.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 -- 172.21.15.50:0/2401621435 learned_addr learned my addr 172.21.15.50:0/2401621435 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:12.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdacc098520 0x7fdacc09bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41458/0 (socket says 172.21.15.50:41458) 2023-10-30T12:12:12.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): get_auth_request con 0x7fdacc1379f0 auth_method 0 2023-10-30T12:12:12.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:12.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:12.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 10 monclient(hunting): get_auth_request con 0x7fdacc098520 auth_method 0 2023-10-30T12:12:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:12.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient(hunting): handle_auth_done global_id 114662 payload 274 2023-10-30T12:12:12.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 -- 172.21.15.50:0/2401621435 >> v2:172.21.15.50:3300/0 conn(0x7fdacc098520 msgr2=0x7fdacc09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:12.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 --2- 172.21.15.50:0/2401621435 >> v2:172.21.15.50:3300/0 conn(0x7fdacc098520 0x7fdacc09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 -- 172.21.15.50:0/2401621435 >> v1:172.21.15.118:6789/0 conn(0x7fdacc098900 legacy=0x7fdacc098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:12.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:12:12.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 monclient: found mon.noname-b 2023-10-30T12:12:12.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:12.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:12.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacbfff640 1 -- 172.21.15.50:0/2401621435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc072120 con 0x7fdacc1379f0 2023-10-30T12:12:12.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.043+0000 7fdacb7fe640 1 --2- 172.21.15.50:0/2401621435 >> v2:172.21.15.50:3300/0 conn(0x7fdacc098520 0x7fdacc09bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:12.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdacbfff640 1 --2- 172.21.15.50:0/2401621435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc1379f0 0x7fdacc139de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e50 tx=0x7fdac402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10b0b1ef3e215081 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:12.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 1 -- 172.21.15.50:0/2401621435 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac402c9e0 con 0x7fdacc1379f0 2023-10-30T12:12:12.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:12.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:12.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: dump: 2023-10-30T12:12:12.063 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:12.064 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:12.064 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.064 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.064 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:12.064 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:12.065 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:12.065 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:12.065 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:12.065 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:12.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: _finish_auth 0 2023-10-30T12:12:12.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:12:12.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:12.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 1 -- 172.21.15.50:0/2401621435 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac402cb40 con 0x7fdacc1379f0 2023-10-30T12:12:12.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:12.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 1 -- 172.21.15.50:0/2401621435 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40343f0 con 0x7fdacc1379f0 2023-10-30T12:12:12.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdad1ad4640 5 monclient: authenticate success, global_id 114662 2023-10-30T12:12:12.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdad1ad4640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:12.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdad1ad4640 4 set_mon_vals no callback set 2023-10-30T12:12:12.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:12.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:12.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdaca7fc640 10 monclient: dump: 2023-10-30T12:12:12.068 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:12.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:12.068 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.068 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:12.069 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:12.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:12.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:12.069 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:12.069 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:12.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdad1ad4640 10 monclient: shutdown 2023-10-30T12:12:12.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdad1ad4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:12.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdad1ad4640 1 -- 172.21.15.50:0/2401621435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc1379f0 msgr2=0x7fdacc139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:12.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.044+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/2401621435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc1379f0 0x7fdacc139de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e50 tx=0x7fdac402be40 comp rx=0 tx=0).stop 2023-10-30T12:12:12.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 -- 172.21.15.50:0/2401621435 shutdown_connections 2023-10-30T12:12:12.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/2401621435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc1379f0 0x7fdacc139de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/2401621435 >> v2:172.21.15.50:3300/0 conn(0x7fdacc098520 0x7fdacc09bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 -- 172.21.15.50:0/2401621435 >> 172.21.15.50:0/2401621435 conn(0x7fdacc0a05c0 msgr2=0x7fdacc09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:12.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 -- 172.21.15.50:0/2401621435 shutdown_connections 2023-10-30T12:12:12.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 -- 172.21.15.50:0/2401621435 wait complete. 2023-10-30T12:12:12.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 10 monclient: build_initial_monmap 2023-10-30T12:12:12.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 10 monclient: monmap: 2023-10-30T12:12:12.072 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:12.072 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:12.072 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:12.072 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:12.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:12.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:12.073 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:12.073 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:12.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:12.073 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:12.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 librados: starting msgr at 2023-10-30T12:12:12.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 librados: starting objecter 2023-10-30T12:12:12.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.045+0000 7fdad1ad4640 1 Processor -- start 2023-10-30T12:12:12.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 -- start start 2023-10-30T12:12:12.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 librados: setting wanted keys 2023-10-30T12:12:12.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 librados: calling monclient init 2023-10-30T12:12:12.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient: init 2023-10-30T12:12:12.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:12.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:12.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdacc098520 0x7fdacc095ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:12.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:12.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc098900 0x7fdacc06bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:12.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc098900 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:12.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdacc1379f0 0x7fdacc06c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:12.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc1379f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc096220 con 0x7fdacc098520 2023-10-30T12:12:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc095290 con 0x7fdacc1379f0 2023-10-30T12:12:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc06d470 con 0x7fdacc098900 2023-10-30T12:12:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdad1ad4640 10 monclient(hunting): authenticate will time out at 1577.229713s 2023-10-30T12:12:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc098900 0x7fdacc06bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdacc098520 0x7fdacc095ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdacc098520 0x7fdacc095ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41464/0 (socket says 172.21.15.50:41464) 2023-10-30T12:12:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc098900 0x7fdacc06bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45188/0 (socket says 172.21.15.50:45188) 2023-10-30T12:12:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 1 -- 172.21.15.50:0/3048977409 learned_addr learned my addr 172.21.15.50:0/3048977409 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 10 monclient(hunting): get_auth_request con 0x7fdacc098520 auth_method 0 2023-10-30T12:12:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 10 monclient(hunting): get_auth_request con 0x7fdacc098900 auth_method 0 2023-10-30T12:12:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacbfff640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdacc1379f0 0x7fdacc06c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacbfff640 10 monclient(hunting): get_auth_request con 0x7fdacc1379f0 auth_method 0 2023-10-30T12:12:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.046+0000 7fdacbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 10 monclient(hunting): handle_auth_done global_id 114665 payload 824 2023-10-30T12:12:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 1 -- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdacc1379f0 msgr2=0x7fdacc06c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdacc1379f0 0x7fdacc06c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 1 -- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdacc098520 msgr2=0x7fdacc095ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdacc098520 0x7fdacc095ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:12:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 1 monclient: found mon.noname-c 2023-10-30T12:12:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 1 -- 172.21.15.50:0/3048977409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc0093a0 con 0x7fdacc098900 2023-10-30T12:12:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacb7fe640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdacc098520 0x7fdacc095ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdacaffd640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc098900 0x7fdacc06bb20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdac000c390 tx=0x7fdac000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 client.?.objecter ms_handle_connect 0x7fdacc098900 2023-10-30T12:12:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 1 -- 172.21.15.50:0/3048977409 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac00052d0 con 0x7fdacc098900 2023-10-30T12:12:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: dump: 2023-10-30T12:12:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:12.088 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: _finish_auth 0 2023-10-30T12:12:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:12:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 5 monclient: authenticate success, global_id 114665 2023-10-30T12:12:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 1 -- 172.21.15.50:0/3048977409 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac0005430 con 0x7fdacc098900 2023-10-30T12:12:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 10 monclient: _renew_subs 2023-10-30T12:12:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac4009b30 con 0x7fdacc098900 2023-10-30T12:12:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 1 -- 172.21.15.50:0/3048977409 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac0003830 con 0x7fdacc098900 2023-10-30T12:12:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdab3fff640 10 monclient: dump: 2023-10-30T12:12:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:12.093 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 10 client.114665.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdac9ffb640 4 set_mon_vals no callback set 2023-10-30T12:12:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 10 monclient: _renew_subs 2023-10-30T12:12:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc094cd0 con 0x7fdacc098900 2023-10-30T12:12:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.047+0000 7fdad1ad4640 1 librados: init done 2023-10-30T12:12:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdab3fff640 1 -- 172.21.15.50:0/3048977409 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdac0003990 con 0x7fdacc098900 2023-10-30T12:12:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdad1ad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdad1ad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdad1ad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc099930 con 0x7fdacc098900 2023-10-30T12:12:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdab3fff640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdab80392f0 0x7fdab803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdab3fff640 1 -- 172.21.15.50:0/3048977409 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fdac004a190 con 0x7fdacc098900 2023-10-30T12:12:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdab3fff640 10 client.114665.objecter ms_dispatch 0x7fdacc000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdab3fff640 3 client.114665.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdab3fff640 3 client.114665.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdab3fff640 20 client.114665.objecter dump_active .. 0 homeless 2023-10-30T12:12:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdacb7fe640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdab80392f0 0x7fdab803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.048+0000 7fdacb7fe640 10 monclient: get_auth_request con 0x7fdab80392f0 auth_method 0 2023-10-30T12:12:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.049+0000 7fdacb7fe640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdab80392f0 0x7fdab803b7b0 secure :-1 s=READY pgs=4129 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0114b0 tx=0x7fdabc009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:12.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.049+0000 7fdab3fff640 10 client.114665.objecter ms_handle_connect 0x7fdab80392f0 2023-10-30T12:12:12.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.053+0000 7fdab3fff640 1 -- 172.21.15.50:0/3048977409 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdac0011ab0 con 0x7fdacc098900 2023-10-30T12:12:12.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.053+0000 7fdab3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:12.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.053+0000 7fdab3fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:12.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.296+0000 7fdad1ad4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:12:12.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.296+0000 7fdad1ad4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:12:12.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.297+0000 7fdad1ad4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:12.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.297+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fdacc04e160 con 0x7fdacc098900 2023-10-30T12:12:12.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.298+0000 7fdab3fff640 1 -- 172.21.15.50:0/3048977409 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fdac0019a70 con 0x7fdacc098900 2023-10-30T12:12:12.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.298+0000 7fdab3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:12:12.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.298+0000 7fdab3fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:12.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.298+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.298+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=8,logm=56} crc {auth=112891219,config=3140514009,health=1689223900,kv=3826327778,logm=1835884596}) 2023-10-30T12:12:12.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.298+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=1057107703,config=3140514009,health=1500915751,kv=2905431493,logm=3820121463}) 2023-10-30T12:12:12.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.298+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.298+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=8,logm=56} crc {auth=112891219,config=3140514009,health=1689223900,kv=3826327778,logm=1835884596}) 2023-10-30T12:12:12.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.298+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=9,logm=56} crc {auth=2773572521,config=3140514009,health=1500915751,kv=2905431493,logm=3945391244}) 2023-10-30T12:12:12.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.299+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.299+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339624658}) 2023-10-30T12:12:12.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.299+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643823059}) 2023-10-30T12:12:12.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.299+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.299+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339624658}) 2023-10-30T12:12:12.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.299+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1353633160}) 2023-10-30T12:12:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 10 librados: watch_flush enter 2023-10-30T12:12:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 10 librados: watch_flush exit 2023-10-30T12:12:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 20 client.114665.objecter shutdown clearing up homeless session... 2023-10-30T12:12:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 10 client.114665.objecter successfully canceled tick 2023-10-30T12:12:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdab80392f0 msgr2=0x7fdab803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdab80392f0 0x7fdab803b7b0 secure :-1 s=READY pgs=4129 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0114b0 tx=0x7fdabc009110 comp rx=0 tx=0).stop 2023-10-30T12:12:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 10 monclient: shutdown 2023-10-30T12:12:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc098900 msgr2=0x7fdacc06bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc098900 0x7fdacc06bb20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdac000c390 tx=0x7fdac000c850 comp rx=0 tx=0).stop 2023-10-30T12:12:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 shutdown_connections 2023-10-30T12:12:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdab80392f0 0x7fdab803b7b0 unknown :-1 s=CLOSED pgs=4129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdacc1379f0 0x7fdacc06c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdacc098900 0x7fdacc06bb20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 --2- 172.21.15.50:0/3048977409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdacc098520 0x7fdacc095ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:12.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 >> 172.21.15.50:0/3048977409 conn(0x7fdacc0a05c0 msgr2=0x7fdacc070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:12.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 shutdown_connections 2023-10-30T12:12:12.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 -- 172.21.15.50:0/3048977409 wait complete. 2023-10-30T12:12:12.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:12.300+0000 7fdad1ad4640 1 librados: shutdown 2023-10-30T12:12:12.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.301+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.301+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917757041}) 2023-10-30T12:12:12.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.301+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964647363}) 2023-10-30T12:12:12.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.301+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.301+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917757041}) 2023-10-30T12:12:12.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.301+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367485394}) 2023-10-30T12:12:12.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.302+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.302+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441925596}) 2023-10-30T12:12:12.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.302+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2848663933}) 2023-10-30T12:12:12.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.302+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.302+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441925596}) 2023-10-30T12:12:12.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.302+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=631413821}) 2023-10-30T12:12:12.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.304+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.304+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167100964}) 2023-10-30T12:12:12.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.304+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2041412886}) 2023-10-30T12:12:12.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.304+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.304+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167100964}) 2023-10-30T12:12:12.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.304+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994777866}) 2023-10-30T12:12:12.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.305+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.305+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1932836259}) 2023-10-30T12:12:12.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.305+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545298675}) 2023-10-30T12:12:12.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.305+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.305+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1932836259}) 2023-10-30T12:12:12.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.305+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060065196}) 2023-10-30T12:12:12.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.306+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.306+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165666290}) 2023-10-30T12:12:12.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.306+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1531900608}) 2023-10-30T12:12:12.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.306+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.306+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165666290}) 2023-10-30T12:12:12.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.306+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924166866}) 2023-10-30T12:12:12.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.307+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.307+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180669722}) 2023-10-30T12:12:12.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.307+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4257326932}) 2023-10-30T12:12:12.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.307+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.307+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180669722}) 2023-10-30T12:12:12.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.307+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=582855946}) 2023-10-30T12:12:12.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.308+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.308+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125702463}) 2023-10-30T12:12:12.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.308+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3737680874}) 2023-10-30T12:12:12.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.308+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.308+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125702463}) 2023-10-30T12:12:12.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.308+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638543909}) 2023-10-30T12:12:12.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.309+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.309+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=707693030}) 2023-10-30T12:12:12.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.309+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885451119}) 2023-10-30T12:12:12.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.309+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.309+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=707693030}) 2023-10-30T12:12:12.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.309+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4002484292}) 2023-10-30T12:12:12.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.310+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.310+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1320016009}) 2023-10-30T12:12:12.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.310+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3577008780}) 2023-10-30T12:12:12.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.310+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.310+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1320016009}) 2023-10-30T12:12:12.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.310+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1662117702}) 2023-10-30T12:12:12.326 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:12:12.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.311+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.311+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107584328}) 2023-10-30T12:12:12.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.311+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138968545}) 2023-10-30T12:12:12.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.311+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.311+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107584328}) 2023-10-30T12:12:12.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.311+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191074676}) 2023-10-30T12:12:12.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.312+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.312+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694053768}) 2023-10-30T12:12:12.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.312+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2370718472}) 2023-10-30T12:12:12.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.312+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.312+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694053768}) 2023-10-30T12:12:12.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.312+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3067051258}) 2023-10-30T12:12:12.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236663828}) 2023-10-30T12:12:12.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=557674976}) 2023-10-30T12:12:12.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236663828}) 2023-10-30T12:12:12.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802508889}) 2023-10-30T12:12:12.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573311469}) 2023-10-30T12:12:12.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3289112808}) 2023-10-30T12:12:12.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573311469}) 2023-10-30T12:12:12.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.313+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243547932}) 2023-10-30T12:12:12.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.314+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.314+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863601210}) 2023-10-30T12:12:12.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.314+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038742952}) 2023-10-30T12:12:12.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.314+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.314+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863601210}) 2023-10-30T12:12:12.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.314+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942720531}) 2023-10-30T12:12:12.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.315+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.315+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272943875}) 2023-10-30T12:12:12.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.315+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205166776}) 2023-10-30T12:12:12.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.315+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.315+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272943875}) 2023-10-30T12:12:12.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.315+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253545390}) 2023-10-30T12:12:12.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.316+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.316+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600238588}) 2023-10-30T12:12:12.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.316+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=345665936}) 2023-10-30T12:12:12.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.316+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.316+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600238588}) 2023-10-30T12:12:12.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.316+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3859589236}) 2023-10-30T12:12:12.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825508329}) 2023-10-30T12:12:12.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241124737}) 2023-10-30T12:12:12.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825508329}) 2023-10-30T12:12:12.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012250938}) 2023-10-30T12:12:12.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2509228656}) 2023-10-30T12:12:12.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3959461781}) 2023-10-30T12:12:12.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2509228656}) 2023-10-30T12:12:12.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.317+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=198307376}) 2023-10-30T12:12:12.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.318+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.318+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160662271}) 2023-10-30T12:12:12.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.318+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4230520644}) 2023-10-30T12:12:12.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.318+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.318+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160662271}) 2023-10-30T12:12:12.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.318+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793589655}) 2023-10-30T12:12:12.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.319+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.319+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778734666}) 2023-10-30T12:12:12.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.319+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838825359}) 2023-10-30T12:12:12.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.319+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.319+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778734666}) 2023-10-30T12:12:12.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.319+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1757296549}) 2023-10-30T12:12:12.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638452721}) 2023-10-30T12:12:12.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060279301}) 2023-10-30T12:12:12.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638452721}) 2023-10-30T12:12:12.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646530129}) 2023-10-30T12:12:12.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362895155}) 2023-10-30T12:12:12.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.320+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074531673}) 2023-10-30T12:12:12.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362895155}) 2023-10-30T12:12:12.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861132034}) 2023-10-30T12:12:12.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821330944}) 2023-10-30T12:12:12.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3990336959}) 2023-10-30T12:12:12.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821330944}) 2023-10-30T12:12:12.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.321+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2853841676}) 2023-10-30T12:12:12.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.322+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.322+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1248854706}) 2023-10-30T12:12:12.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.322+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407668799}) 2023-10-30T12:12:12.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.322+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.322+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1248854706}) 2023-10-30T12:12:12.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.322+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1560720337}) 2023-10-30T12:12:12.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.323+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.323+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246984709}) 2023-10-30T12:12:12.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.323+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3908206221}) 2023-10-30T12:12:12.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.323+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.323+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246984709}) 2023-10-30T12:12:12.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.323+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1199495995}) 2023-10-30T12:12:12.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.324+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.324+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998774704}) 2023-10-30T12:12:12.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.324+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=722712087}) 2023-10-30T12:12:12.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.324+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.324+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998774704}) 2023-10-30T12:12:12.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.324+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=863188345}) 2023-10-30T12:12:12.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694485399}) 2023-10-30T12:12:12.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2883753381}) 2023-10-30T12:12:12.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694485399}) 2023-10-30T12:12:12.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013101178}) 2023-10-30T12:12:12.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990525158}) 2023-10-30T12:12:12.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570563099}) 2023-10-30T12:12:12.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990525158}) 2023-10-30T12:12:12.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.325+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068674293}) 2023-10-30T12:12:12.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.326+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.326+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757286818}) 2023-10-30T12:12:12.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.326+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195809594}) 2023-10-30T12:12:12.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.326+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.326+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757286818}) 2023-10-30T12:12:12.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.326+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3078603520}) 2023-10-30T12:12:12.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.327+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.327+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338159736}) 2023-10-30T12:12:12.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.327+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1686955850}) 2023-10-30T12:12:12.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.327+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.327+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338159736}) 2023-10-30T12:12:12.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.327+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517601754}) 2023-10-30T12:12:12.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.328+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.328+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2068065614}) 2023-10-30T12:12:12.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.328+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2131862608}) 2023-10-30T12:12:12.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.328+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.328+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2068065614}) 2023-10-30T12:12:12.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.328+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2072881796}) 2023-10-30T12:12:12.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2096243319}) 2023-10-30T12:12:12.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354975801}) 2023-10-30T12:12:12.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2096243319}) 2023-10-30T12:12:12.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915553482}) 2023-10-30T12:12:12.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555846879}) 2023-10-30T12:12:12.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803968800}) 2023-10-30T12:12:12.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.329+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555846879}) 2023-10-30T12:12:12.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4221968034}) 2023-10-30T12:12:12.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200378707}) 2023-10-30T12:12:12.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024093144}) 2023-10-30T12:12:12.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200378707}) 2023-10-30T12:12:12.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.330+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1590975125}) 2023-10-30T12:12:12.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.331+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.331+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675327963}) 2023-10-30T12:12:12.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.331+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2472911271}) 2023-10-30T12:12:12.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.331+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.331+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675327963}) 2023-10-30T12:12:12.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.331+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198225747}) 2023-10-30T12:12:12.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.332+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.332+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254443289}) 2023-10-30T12:12:12.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.332+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170640277}) 2023-10-30T12:12:12.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.332+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.332+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254443289}) 2023-10-30T12:12:12.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.332+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458605337}) 2023-10-30T12:12:12.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.333+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.333+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620528233}) 2023-10-30T12:12:12.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.333+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2337993833}) 2023-10-30T12:12:12.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.333+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.333+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620528233}) 2023-10-30T12:12:12.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.333+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=131933298}) 2023-10-30T12:12:12.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.334+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.334+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682429186}) 2023-10-30T12:12:12.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.334+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1323606216}) 2023-10-30T12:12:12.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.334+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.334+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682429186}) 2023-10-30T12:12:12.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.334+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=386551258}) 2023-10-30T12:12:12.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467116638}) 2023-10-30T12:12:12.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591428256}) 2023-10-30T12:12:12.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467116638}) 2023-10-30T12:12:12.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1607904896}) 2023-10-30T12:12:12.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797162029}) 2023-10-30T12:12:12.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816715214}) 2023-10-30T12:12:12.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797162029}) 2023-10-30T12:12:12.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.335+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701266947}) 2023-10-30T12:12:12.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.336+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.336+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754345489}) 2023-10-30T12:12:12.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.336+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1957576552}) 2023-10-30T12:12:12.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.336+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.336+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754345489}) 2023-10-30T12:12:12.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.336+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=606007106}) 2023-10-30T12:12:12.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.337+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.337+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1766117808}) 2023-10-30T12:12:12.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.337+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2891503051}) 2023-10-30T12:12:12.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.337+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.337+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1766117808}) 2023-10-30T12:12:12.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.337+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=672496964}) 2023-10-30T12:12:12.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.338+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.338+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760540490}) 2023-10-30T12:12:12.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.338+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899496423}) 2023-10-30T12:12:12.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.338+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.338+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760540490}) 2023-10-30T12:12:12.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.338+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2324376780}) 2023-10-30T12:12:12.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.339+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.339+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996174580}) 2023-10-30T12:12:12.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.339+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1632601233}) 2023-10-30T12:12:12.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.339+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.339+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996174580}) 2023-10-30T12:12:12.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.339+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367697205}) 2023-10-30T12:12:12.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.340+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.340+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397890256}) 2023-10-30T12:12:12.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.340+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1575746087}) 2023-10-30T12:12:12.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.340+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.340+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397890256}) 2023-10-30T12:12:12.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.340+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1896471050}) 2023-10-30T12:12:12.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241050438}) 2023-10-30T12:12:12.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2952786283}) 2023-10-30T12:12:12.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241050438}) 2023-10-30T12:12:12.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4041803318}) 2023-10-30T12:12:12.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693843790}) 2023-10-30T12:12:12.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4256169945}) 2023-10-30T12:12:12.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693843790}) 2023-10-30T12:12:12.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.341+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2893664387}) 2023-10-30T12:12:12.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.342+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.342+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419333642}) 2023-10-30T12:12:12.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.342+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2126832607}) 2023-10-30T12:12:12.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.342+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.342+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419333642}) 2023-10-30T12:12:12.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.342+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359294008}) 2023-10-30T12:12:12.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.343+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.343+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976248715}) 2023-10-30T12:12:12.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.343+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162508574}) 2023-10-30T12:12:12.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.343+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.343+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976248715}) 2023-10-30T12:12:12.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.343+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1566959835}) 2023-10-30T12:12:12.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.344+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.344+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980136043}) 2023-10-30T12:12:12.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.344+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=646632943}) 2023-10-30T12:12:12.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.344+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.344+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980136043}) 2023-10-30T12:12:12.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.344+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2608918547}) 2023-10-30T12:12:12.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.345+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.345+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870103127}) 2023-10-30T12:12:12.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.345+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801650878}) 2023-10-30T12:12:12.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.345+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.345+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870103127}) 2023-10-30T12:12:12.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.345+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618469973}) 2023-10-30T12:12:12.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.346+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.346+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810699919}) 2023-10-30T12:12:12.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.346+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=493615232}) 2023-10-30T12:12:12.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.346+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.346+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810699919}) 2023-10-30T12:12:12.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.346+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798311506}) 2023-10-30T12:12:12.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.347+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.347+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969653133}) 2023-10-30T12:12:12.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.347+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104861956}) 2023-10-30T12:12:12.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.347+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.347+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969653133}) 2023-10-30T12:12:12.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.347+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521510831}) 2023-10-30T12:12:12.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146304829}) 2023-10-30T12:12:12.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1715508721}) 2023-10-30T12:12:12.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146304829}) 2023-10-30T12:12:12.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334934239}) 2023-10-30T12:12:12.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031670219}) 2023-10-30T12:12:12.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533444586}) 2023-10-30T12:12:12.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031670219}) 2023-10-30T12:12:12.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.348+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3347856628}) 2023-10-30T12:12:12.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.349+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.349+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908888638}) 2023-10-30T12:12:12.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.349+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1900348951}) 2023-10-30T12:12:12.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.349+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.349+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908888638}) 2023-10-30T12:12:12.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.349+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867236034}) 2023-10-30T12:12:12.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.350+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.350+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=2,osd_metadata=6,osd_snap=34,osdmap=28} crc {logm=2791030696,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3417517031,mon_config_key=3217606506,monmap=4204254258,osd_metadata=2479346626,osd_snap=2548828034,osdmap=1907255094}) 2023-10-30T12:12:12.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.350+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=30} crc {logm=2791030696,mdsmap=3984425268,mgr=1142129444,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3569795152,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=2979883996,osd_snap=1708336091,osdmap=2607885256}) 2023-10-30T12:12:12.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.350+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.350+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=2,osd_metadata=6,osd_snap=34,osdmap=28} crc {logm=2791030696,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3417517031,mon_config_key=3217606506,monmap=4204254258,osd_metadata=2479346626,osd_snap=2548828034,osdmap=1907255094}) 2023-10-30T12:12:12.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.351+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=31} crc {logm=2791030696,mdsmap=3140514010,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3417517031,mon_config_key=715115367,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=2979883996,osd_snap=268977719,osdmap=680459501}) 2023-10-30T12:12:12.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.352+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.352+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1707476559}) 2023-10-30T12:12:12.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.352+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1713761338}) 2023-10-30T12:12:12.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.352+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.352+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1707476559}) 2023-10-30T12:12:12.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.352+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2535796703}) 2023-10-30T12:12:12.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.353+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.353+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1402916672}) 2023-10-30T12:12:12.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.353+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1841528561}) 2023-10-30T12:12:12.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.353+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.353+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1402916672}) 2023-10-30T12:12:12.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.353+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=533276982}) 2023-10-30T12:12:12.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.354+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.354+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1293624205}) 2023-10-30T12:12:12.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.354+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3906740435}) 2023-10-30T12:12:12.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.354+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.354+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1293624205}) 2023-10-30T12:12:12.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.354+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3664226597}) 2023-10-30T12:12:12.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.356+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.356+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2621980137}) 2023-10-30T12:12:12.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.356+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3515062077}) 2023-10-30T12:12:12.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.356+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.356+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2621980137}) 2023-10-30T12:12:12.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.356+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=487791900}) 2023-10-30T12:12:12.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.357+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.357+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3878046634}) 2023-10-30T12:12:12.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.358+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3938745263}) 2023-10-30T12:12:12.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.358+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.358+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3878046634}) 2023-10-30T12:12:12.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.358+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2720571206}) 2023-10-30T12:12:12.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.363+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.363+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=2762429728}) 2023-10-30T12:12:12.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.363+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=14} crc {osdmap=2666459743}) 2023-10-30T12:12:12.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.363+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:12:12.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.363+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=2762429728}) 2023-10-30T12:12:12.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:12:12.363+0000 7f3c7a277640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=15} crc {osdmap=600211261}) 2023-10-30T12:12:13.328 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:12:13.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.443+0000 7fda68b1d640 10 monclient: get_monmap_and_config 2023-10-30T12:12:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.443+0000 7fda68b1d640 10 monclient: build_initial_monmap 2023-10-30T12:12:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.443+0000 7fda68b1d640 10 monclient: monmap: 2023-10-30T12:12:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:13.446 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.446 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:13.444496+0000 2023-10-30T12:12:13.446 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:13.444496+0000 2023-10-30T12:12:13.446 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:13.446 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:13.448 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.443+0000 7fda68b1d640 1 Processor -- start 2023-10-30T12:12:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 1 -- start start 2023-10-30T12:12:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient: init 2023-10-30T12:12:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T12:12:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fda640a2dd0 0x7fda640a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fda640a2dd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fda6406a8e0 0x7fda640a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fda6406a8e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64071400 0x7fda640a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fda64071400 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda640a31b0 con 0x7fda640a2dd0 2023-10-30T12:12:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6406a6a0 con 0x7fda6406a8e0 2023-10-30T12:12:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6406be90 con 0x7fda64071400 2023-10-30T12:12:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda68b1d640 10 monclient(hunting): authenticate will time out at 1578.627739s 2023-10-30T12:12:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda62d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64071400 0x7fda640a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda6259c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fda640a2dd0 0x7fda640a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda6259c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fda640a2dd0 0x7fda640a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41472/0 (socket says 172.21.15.50:41472) 2023-10-30T12:12:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda62d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64071400 0x7fda640a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45196/0 (socket says 172.21.15.50:45196) 2023-10-30T12:12:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda6259c640 1 -- 172.21.15.50:0/3812270446 learned_addr learned my addr 172.21.15.50:0/3812270446 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda6259c640 10 monclient(hunting): get_auth_request con 0x7fda640a2dd0 auth_method 0 2023-10-30T12:12:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda6259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda6259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda62d9d640 10 monclient(hunting): get_auth_request con 0x7fda64071400 auth_method 0 2023-10-30T12:12:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda62d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.444+0000 7fda62d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda61d9b640 1 --2- 172.21.15.50:0/3812270446 >> v2:172.21.15.118:3300/0 conn(0x7fda6406a8e0 0x7fda640a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda61d9b640 10 monclient(hunting): get_auth_request con 0x7fda6406a8e0 auth_method 0 2023-10-30T12:12:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda61d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda61d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 10 monclient(hunting): handle_auth_done global_id 164130 payload 274 2023-10-30T12:12:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 1 -- 172.21.15.50:0/3812270446 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64071400 msgr2=0x7fda640a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 1 --2- 172.21.15.50:0/3812270446 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64071400 0x7fda640a5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 1 -- 172.21.15.50:0/3812270446 >> v2:172.21.15.118:3300/0 conn(0x7fda6406a8e0 msgr2=0x7fda640a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 1 --2- 172.21.15.50:0/3812270446 >> v2:172.21.15.118:3300/0 conn(0x7fda6406a8e0 0x7fda640a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 10 monclient: _finish_hunting 0 2023-10-30T12:12:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 1 monclient: found mon.noname-a 2023-10-30T12:12:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.445+0000 7fda6259c640 1 -- 172.21.15.50:0/3812270446 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6409e190 con 0x7fda640a2dd0 2023-10-30T12:12:13.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6259c640 1 --2- 172.21.15.50:0/3812270446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda640a2dd0 0x7fda640a46c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda64073f20 tx=0x7fda5402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8b3e7381379401 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 1 -- 172.21.15.50:0/3812270446 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5402c950 con 0x7fda640a2dd0 2023-10-30T12:12:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: dump: 2023-10-30T12:12:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.467 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: _finish_auth 0 2023-10-30T12:12:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: _check_auth_tickets 2023-10-30T12:12:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 1 -- 172.21.15.50:0/3812270446 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5402cab0 con 0x7fda640a2dd0 2023-10-30T12:12:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 5 monclient: authenticate success, global_id 164130 2023-10-30T12:12:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 1 -- 172.21.15.50:0/3812270446 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5402ce40 con 0x7fda640a2dd0 2023-10-30T12:12:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 4 set_mon_vals no callback set 2023-10-30T12:12:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda6159a640 10 monclient: dump: 2023-10-30T12:12:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.471 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 10 monclient: shutdown 2023-10-30T12:12:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 -- 172.21.15.50:0/3812270446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda640a2dd0 msgr2=0x7fda640a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3812270446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda640a2dd0 0x7fda640a46c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda64073f20 tx=0x7fda5402bdb0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 -- 172.21.15.50:0/3812270446 shutdown_connections 2023-10-30T12:12:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3812270446 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64071400 0x7fda640a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3812270446 >> v2:172.21.15.118:3300/0 conn(0x7fda6406a8e0 0x7fda640a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3812270446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda640a2dd0 0x7fda640a46c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 -- 172.21.15.50:0/3812270446 >> 172.21.15.50:0/3812270446 conn(0x7fda640a22b0 msgr2=0x7fda640a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 -- 172.21.15.50:0/3812270446 shutdown_connections 2023-10-30T12:12:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.446+0000 7fda68b1d640 1 -- 172.21.15.50:0/3812270446 wait complete. 2023-10-30T12:12:13.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.447+0000 7fda68b1d640 10 monclient: build_initial_monmap 2023-10-30T12:12:13.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.447+0000 7fda68b1d640 10 monclient: monmap: 2023-10-30T12:12:13.474 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:13.474 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:13.474 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:13.475 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:13.475 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:13.475 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.475 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:13.475 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:13.476 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:13.476 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.447+0000 7fda68b1d640 1 librados: starting msgr at 2023-10-30T12:12:13.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.447+0000 7fda68b1d640 1 librados: starting objecter 2023-10-30T12:12:13.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.447+0000 7fda68b1d640 1 Processor -- start 2023-10-30T12:12:13.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 -- start start 2023-10-30T12:12:13.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 librados: setting wanted keys 2023-10-30T12:12:13.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 librados: calling monclient init 2023-10-30T12:12:13.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient: init 2023-10-30T12:12:13.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:12:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda64071400 0x7fda641ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fda64071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda640a2dd0 0x7fda641225b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fda640a2dd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64122af0 0x7fda64122f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fda64122af0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:13.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64126620 con 0x7fda64071400 2023-10-30T12:12:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6406b0a0 con 0x7fda640a2dd0 2023-10-30T12:12:13.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda64070450 con 0x7fda64122af0 2023-10-30T12:12:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda68b1d640 10 monclient(hunting): authenticate will time out at 1578.631464s 2023-10-30T12:12:13.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda64071400 0x7fda641ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64122af0 0x7fda64122f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64122af0 0x7fda64122f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45212/0 (socket says 172.21.15.50:45212) 2023-10-30T12:12:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda64071400 0x7fda641ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41486/0 (socket says 172.21.15.50:41486) 2023-10-30T12:12:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 -- 172.21.15.50:0/3848225590 learned_addr learned my addr 172.21.15.50:0/3848225590 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): get_auth_request con 0x7fda64122af0 auth_method 0 2023-10-30T12:12:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 10 monclient(hunting): get_auth_request con 0x7fda64071400 auth_method 0 2023-10-30T12:12:13.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda61d9b640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda640a2dd0 0x7fda641225b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient(hunting): handle_auth_done global_id 114671 payload 824 2023-10-30T12:12:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 -- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda640a2dd0 msgr2=0x7fda641225b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda640a2dd0 0x7fda641225b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 -- 172.21.15.50:0/3848225590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda64071400 msgr2=0x7fda641ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda64071400 0x7fda641ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:12:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 monclient: found mon.noname-c 2023-10-30T12:12:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.448+0000 7fda62d9d640 1 -- 172.21.15.50:0/3848225590 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54009b30 con 0x7fda64122af0 2023-10-30T12:12:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda62d9d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64122af0 0x7fda64122f80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00e850 tx=0x7fda5c00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 client.?.objecter ms_handle_connect 0x7fda64122af0 2023-10-30T12:12:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 1 -- 172.21.15.50:0/3848225590 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c009630 con 0x7fda64122af0 2023-10-30T12:12:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: dump: 2023-10-30T12:12:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.489 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.490 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.491 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: _finish_auth 0 2023-10-30T12:12:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:13.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 1 -- 172.21.15.50:0/3848225590 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c00c940 con 0x7fda64122af0 2023-10-30T12:12:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 1 -- 172.21.15.50:0/3848225590 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c00ccf0 con 0x7fda64122af0 2023-10-30T12:12:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:13.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda52ffd640 10 monclient: dump: 2023-10-30T12:12:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.493 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.494 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 5 monclient: authenticate success, global_id 114671 2023-10-30T12:12:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 10 monclient: _renew_subs 2023-10-30T12:12:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda60d99640 4 set_mon_vals no callback set 2023-10-30T12:12:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda641c0f80 con 0x7fda64122af0 2023-10-30T12:12:13.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 10 client.114671.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:13.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 10 monclient: _renew_subs 2023-10-30T12:12:13.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda64124c10 con 0x7fda64122af0 2023-10-30T12:12:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.449+0000 7fda68b1d640 1 librados: init done 2023-10-30T12:12:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda52ffd640 1 -- 172.21.15.50:0/3848225590 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fda5c013470 con 0x7fda64122af0 2023-10-30T12:12:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda68b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda68b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda68b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda52ffd640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda440392f0 0x7fda4403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6404e160 con 0x7fda64122af0 2023-10-30T12:12:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda52ffd640 1 -- 172.21.15.50:0/3848225590 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fda5c04adf0 con 0x7fda64122af0 2023-10-30T12:12:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda52ffd640 10 client.114671.objecter ms_dispatch 0x7fda64000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda52ffd640 3 client.114671.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:13.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda52ffd640 3 client.114671.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda52ffd640 20 client.114671.objecter dump_active .. 0 homeless 2023-10-30T12:12:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda6259c640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda440392f0 0x7fda4403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.450+0000 7fda6259c640 10 monclient: get_auth_request con 0x7fda440392f0 auth_method 0 2023-10-30T12:12:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.451+0000 7fda6259c640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda440392f0 0x7fda4403b7b0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 crypto rx=0x7fda54035000 tx=0x7fda54030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.451+0000 7fda52ffd640 10 client.114671.objecter ms_handle_connect 0x7fda440392f0 2023-10-30T12:12:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.455+0000 7fda52ffd640 1 -- 172.21.15.50:0/3848225590 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fda5c017020 con 0x7fda64122af0 2023-10-30T12:12:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.455+0000 7fda52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:13.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.455+0000 7fda52ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:13.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.696+0000 7fda68b1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:13.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.696+0000 7fda68b1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:13.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.696+0000 7fda68b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.696+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fda640745c0 con 0x7fda64122af0 2023-10-30T12:12:13.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.697+0000 7fda52ffd640 1 -- 172.21.15.50:0/3848225590 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fda5c03c080 con 0x7fda64122af0 2023-10-30T12:12:13.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.697+0000 7fda52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:13.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.697+0000 7fda52ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:13.700 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:13.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 10 librados: watch_flush enter 2023-10-30T12:12:13.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 10 librados: watch_flush exit 2023-10-30T12:12:13.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 20 client.114671.objecter shutdown clearing up homeless session... 2023-10-30T12:12:13.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 10 client.114671.objecter successfully canceled tick 2023-10-30T12:12:13.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda440392f0 msgr2=0x7fda4403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda440392f0 0x7fda4403b7b0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 crypto rx=0x7fda54035000 tx=0x7fda54030000 comp rx=0 tx=0).stop 2023-10-30T12:12:13.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 10 monclient: shutdown 2023-10-30T12:12:13.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64122af0 msgr2=0x7fda64122f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64122af0 0x7fda64122f80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00e850 tx=0x7fda5c00ed10 comp rx=0 tx=0).stop 2023-10-30T12:12:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 shutdown_connections 2023-10-30T12:12:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fda440392f0 0x7fda4403b7b0 unknown :-1 s=CLOSED pgs=4130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fda64122af0 0x7fda64122f80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda640a2dd0 0x7fda641225b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 --2- 172.21.15.50:0/3848225590 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fda64071400 0x7fda641ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 >> 172.21.15.50:0/3848225590 conn(0x7fda640a22b0 msgr2=0x7fda6406e580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 shutdown_connections 2023-10-30T12:12:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 -- 172.21.15.50:0/3848225590 wait complete. 2023-10-30T12:12:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.699+0000 7fda68b1d640 1 librados: shutdown 2023-10-30T12:12:13.710 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:12:13.711 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:12:13.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:12:13.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:12:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.815+0000 7fe46df16640 10 monclient: get_monmap_and_config 2023-10-30T12:12:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.815+0000 7fe46df16640 10 monclient: build_initial_monmap 2023-10-30T12:12:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.815+0000 7fe46df16640 10 monclient: monmap: 2023-10-30T12:12:13.817 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:13.816866+0000 2023-10-30T12:12:13.818 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:13.816866+0000 2023-10-30T12:12:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:13.821 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:13.821 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 1 Processor -- start 2023-10-30T12:12:13.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 1 -- start start 2023-10-30T12:12:13.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 10 monclient: init 2023-10-30T12:12:13.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:12:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe46809fa30 0x7fe4680a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 10 monclient(hunting): picked mon.noname-c con 0x7fe46809fa30 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe46806b720 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.816+0000 7fe46df16640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe46813b070 0x7fe46813d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 10 monclient(hunting): picked mon.noname-a con 0x7fe46813b070 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:13.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe46809e010 con 0x7fe46806b720 2023-10-30T12:12:13.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46809fe10 con 0x7fe46813b070 2023-10-30T12:12:13.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46806bde0 con 0x7fe46809fa30 2023-10-30T12:12:13.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:13.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe46df16640 10 monclient(hunting): authenticate will time out at 1579.000037s 2023-10-30T12:12:13.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe46813b070 0x7fe46813d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe46813b070 0x7fe46813d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41490/0 (socket says 172.21.15.50:41490) 2023-10-30T12:12:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 -- 172.21.15.50:0/2789102485 learned_addr learned my addr 172.21.15.50:0/2789102485 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): get_auth_request con 0x7fe46813b070 auth_method 0 2023-10-30T12:12:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe4677fe640 1 --2- 172.21.15.50:0/2789102485 >> v2:172.21.15.118:3300/0 conn(0x7fe46809fa30 0x7fe4680a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe4677fe640 10 monclient(hunting): get_auth_request con 0x7fe46809fa30 auth_method 0 2023-10-30T12:12:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe4677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe4677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe4677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient(hunting): handle_auth_done global_id 164136 payload 274 2023-10-30T12:12:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 -- 172.21.15.50:0/2789102485 >> v2:172.21.15.118:3300/0 conn(0x7fe46809fa30 msgr2=0x7fe4680a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 --2- 172.21.15.50:0/2789102485 >> v2:172.21.15.118:3300/0 conn(0x7fe46809fa30 0x7fe4680a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 -- 172.21.15.50:0/2789102485 >> v1:172.21.15.118:6789/0 conn(0x7fe46806b720 legacy=0x7fe4680a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient: _finish_hunting 0 2023-10-30T12:12:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 monclient: found mon.noname-a 2023-10-30T12:12:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.817+0000 7fe467fff640 1 -- 172.21.15.50:0/2789102485 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46809e540 con 0x7fe46813b070 2023-10-30T12:12:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe467fff640 1 --2- 172.21.15.50:0/2789102485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46813b070 0x7fe46813d460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe460009e50 tx=0x7fe46002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951d771d2e7aeb87 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 1 -- 172.21.15.50:0/2789102485 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46002c9f0 con 0x7fe46813b070 2023-10-30T12:12:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: dump: 2023-10-30T12:12:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.840 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: _finish_auth 0 2023-10-30T12:12:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: _check_auth_tickets 2023-10-30T12:12:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 1 -- 172.21.15.50:0/2789102485 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46002cb50 con 0x7fe46813b070 2023-10-30T12:12:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 5 monclient: authenticate success, global_id 164136 2023-10-30T12:12:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 1 -- 172.21.15.50:0/2789102485 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600343f0 con 0x7fe46813b070 2023-10-30T12:12:13.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 4 set_mon_vals no callback set 2023-10-30T12:12:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe4667fc640 10 monclient: dump: 2023-10-30T12:12:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.844 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.845 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 10 monclient: shutdown 2023-10-30T12:12:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 1 -- 172.21.15.50:0/2789102485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46813b070 msgr2=0x7fe46813d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 1 --2- 172.21.15.50:0/2789102485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46813b070 0x7fe46813d460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe460009e50 tx=0x7fe46002be50 comp rx=0 tx=0).stop 2023-10-30T12:12:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 1 -- 172.21.15.50:0/2789102485 shutdown_connections 2023-10-30T12:12:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 1 --2- 172.21.15.50:0/2789102485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46813b070 0x7fe46813d460 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 1 --2- 172.21.15.50:0/2789102485 >> v2:172.21.15.118:3300/0 conn(0x7fe46809fa30 0x7fe4680a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.818+0000 7fe46df16640 1 -- 172.21.15.50:0/2789102485 >> 172.21.15.50:0/2789102485 conn(0x7fe4680a6200 msgr2=0x7fe4680a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:13.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.819+0000 7fe46df16640 1 -- 172.21.15.50:0/2789102485 shutdown_connections 2023-10-30T12:12:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.819+0000 7fe46df16640 1 -- 172.21.15.50:0/2789102485 wait complete. 2023-10-30T12:12:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.819+0000 7fe46df16640 10 monclient: build_initial_monmap 2023-10-30T12:12:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.819+0000 7fe46df16640 10 monclient: monmap: 2023-10-30T12:12:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:13.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:13.849 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.819+0000 7fe46df16640 1 librados: starting msgr at 2023-10-30T12:12:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.819+0000 7fe46df16640 1 librados: starting objecter 2023-10-30T12:12:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.819+0000 7fe46df16640 1 Processor -- start 2023-10-30T12:12:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 -- start start 2023-10-30T12:12:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 librados: setting wanted keys 2023-10-30T12:12:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 librados: calling monclient init 2023-10-30T12:12:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient: init 2023-10-30T12:12:13.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:12:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe46809fa30 0x7fe468136b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): picked mon.noname-c con 0x7fe46809fa30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe46813b070 0x7fe4681ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): picked mon.noname-b con 0x7fe46813b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46811cef0 0x7fe46811d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): picked mon.noname-a con 0x7fe46811cef0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46811fde0 con 0x7fe46811cef0 2023-10-30T12:12:13.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468070e50 con 0x7fe46813b070 2023-10-30T12:12:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681385b0 con 0x7fe46809fa30 2023-10-30T12:12:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe46df16640 10 monclient(hunting): authenticate will time out at 1579.003490s 2023-10-30T12:12:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe46809fa30 0x7fe468136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46811cef0 0x7fe46811d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe46809fa30 0x7fe468136b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45216/0 (socket says 172.21.15.50:45216) 2023-10-30T12:12:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 -- 172.21.15.50:0/3321248536 learned_addr learned my addr 172.21.15.50:0/3321248536 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46811cef0 0x7fe46811d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41500/0 (socket says 172.21.15.50:41500) 2023-10-30T12:12:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): get_auth_request con 0x7fe46809fa30 auth_method 0 2023-10-30T12:12:13.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 10 monclient(hunting): get_auth_request con 0x7fe46811cef0 auth_method 0 2023-10-30T12:12:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe466ffd640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe46813b070 0x7fe4681ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe466ffd640 10 monclient(hunting): get_auth_request con 0x7fe46813b070 auth_method 0 2023-10-30T12:12:13.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe466ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe466ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe466ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient(hunting): handle_auth_done global_id 114674 payload 824 2023-10-30T12:12:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe46813b070 msgr2=0x7fe4681ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe46813b070 0x7fe4681ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46811cef0 msgr2=0x7fe46811d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46811cef0 0x7fe46811d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient: _finish_hunting 0 2023-10-30T12:12:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 monclient: found mon.noname-c 2023-10-30T12:12:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe4677fe640 1 -- 172.21.15.50:0/3321248536 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460009b30 con 0x7fe46809fa30 2023-10-30T12:12:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.820+0000 7fe467fff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46811cef0 0x7fe46811d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:13.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe4677fe640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe46809fa30 0x7fe468136b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe45000d800 tx=0x7fe45000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 client.?.objecter ms_handle_connect 0x7fe46809fa30 2023-10-30T12:12:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45000b6c0 con 0x7fe46809fa30 2023-10-30T12:12:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: dump: 2023-10-30T12:12:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.864 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: _finish_auth 0 2023-10-30T12:12:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:12:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45000b840 con 0x7fe46809fa30 2023-10-30T12:12:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 5 monclient: authenticate success, global_id 114674 2023-10-30T12:12:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 10 monclient: _renew_subs 2023-10-30T12:12:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe468120680 con 0x7fe46809fa30 2023-10-30T12:12:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45000bbf0 con 0x7fe46809fa30 2023-10-30T12:12:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe45ffff640 10 monclient: dump: 2023-10-30T12:12:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.869 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe465ffb640 4 set_mon_vals no callback set 2023-10-30T12:12:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 10 client.114674.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 10 monclient: _renew_subs 2023-10-30T12:12:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe468134300 con 0x7fe46809fa30 2023-10-30T12:12:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.821+0000 7fe46df16640 1 librados: init done 2023-10-30T12:12:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe46df16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe46df16640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46806b6a0 0x7fe46811d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe45000bd50 con 0x7fe46809fa30 2023-10-30T12:12:13.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe46df16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468073ff0 con 0x7fe46806b6a0 2023-10-30T12:12:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46806b6a0 0x7fe46811d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient: get_auth_request con 0x7fe46806b6a0 auth_method 0 2023-10-30T12:12:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4480392f0 0x7fe44803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fe45004b120 con 0x7fe46809fa30 2023-10-30T12:12:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 client.114674.objecter ms_dispatch 0x7fe468000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 3 client.114674.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 3 client.114674.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 10 monclient(hunting): handle_auth_done global_id 164142 payload 824 2023-10-30T12:12:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 1 -- 172.21.15.50:0/3321248536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe46806a690 con 0x7fe46806b6a0 2023-10-30T12:12:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 20 client.114674.objecter dump_active .. 0 homeless 2023-10-30T12:12:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 client.114674.objecter ms_handle_connect 0x7fe46806b6a0 2023-10-30T12:12:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 client.114674.objecter resend_mon_ops 2023-10-30T12:12:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe466ffd640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46806b6a0 0x7fe46811d8c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe468071680 tx=0x7fe45800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe467fff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4480392f0 0x7fe44803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe467fff640 10 monclient: get_auth_request con 0x7fe4480392f0 auth_method 0 2023-10-30T12:12:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4580052e0 con 0x7fe46806b6a0 2023-10-30T12:12:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 monclient: dump: 2023-10-30T12:12:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.876 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fe458005440 con 0x7fe46806b6a0 2023-10-30T12:12:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46806b6a0 msgr2=0x7fe46811d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.822+0000 7fe45ffff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46806b6a0 0x7fe46811d8c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe468071680 tx=0x7fe45800e960 comp rx=0 tx=0).stop 2023-10-30T12:12:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.823+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46806b6a0 msgr2=0x7fe46811d8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.823+0000 7fe45ffff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe46806b6a0 0x7fe46811d8c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.823+0000 7fe45ffff640 10 client.114674.objecter ms_handle_connect 0x7fe4480392f0 2023-10-30T12:12:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.823+0000 7fe467fff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4480392f0 0x7fe44803b7b0 secure :-1 s=READY pgs=4131 cs=0 l=1 rev1=1 crypto rx=0x7fe460035000 tx=0x7fe460030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe46df16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe46df16640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe468135260 0x7fe468135670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe46df16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46806ab30 con 0x7fe468135260 2023-10-30T12:12:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe468135260 0x7fe468135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient: get_auth_request con 0x7fe468135260 auth_method 0 2023-10-30T12:12:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:13.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 10 monclient(hunting): handle_auth_done global_id 164145 payload 824 2023-10-30T12:12:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 1 -- 172.21.15.50:0/3321248536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe46806afc0 con 0x7fe468135260 2023-10-30T12:12:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe45ffff640 10 client.114674.objecter ms_handle_connect 0x7fe468135260 2023-10-30T12:12:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe45ffff640 10 client.114674.objecter resend_mon_ops 2023-10-30T12:12:13.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe466ffd640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe468135260 0x7fe468135670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe458016000 tx=0x7fe458001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458014a50 con 0x7fe468135260 2023-10-30T12:12:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe45ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe45ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.829+0000 7fe45ffff640 10 monclient: dump: 2023-10-30T12:12:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:13.884 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:13.885 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:13.886 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7fe458007030 con 0x7fe468135260 2023-10-30T12:12:13.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe45ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe45ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe468135260 msgr2=0x7fe468135670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe45ffff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe468135260 0x7fe468135670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe458016000 tx=0x7fe458001120 comp rx=0 tx=0).stop 2023-10-30T12:12:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe45ffff640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe468135260 msgr2=0x7fe468135670 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:13.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe45ffff640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe468135260 0x7fe468135670 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe458016000 tx=0x7fe458001120 comp rx=0 tx=0).stop 2023-10-30T12:12:13.888 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:13.888 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:13.888 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:13.888 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:12:13.889 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 114, 2023-10-30T12:12:13.889 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:13.889 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:12:13.889 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:13.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:13.890 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:13.890 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:12:13.890 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:13.890 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:13.890 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:13.891 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:13.892 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:13.893 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:13.894 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:13.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:13.896 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:13.897 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:13.898 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:13.899 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:13.900 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.901 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:13.902 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:13.918 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:13.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 10 librados: watch_flush enter 2023-10-30T12:12:13.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 10 librados: watch_flush exit 2023-10-30T12:12:13.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 20 client.114674.objecter shutdown clearing up homeless session... 2023-10-30T12:12:13.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 10 client.114674.objecter successfully canceled tick 2023-10-30T12:12:13.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4480392f0 msgr2=0x7fe44803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe4480392f0 0x7fe44803b7b0 secure :-1 s=READY pgs=4131 cs=0 l=1 rev1=1 crypto rx=0x7fe460035000 tx=0x7fe460030000 comp rx=0 tx=0).stop 2023-10-30T12:12:13.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 10 monclient: shutdown 2023-10-30T12:12:13.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:13.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe46809fa30 msgr2=0x7fe468136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:13.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe467fff640 1 -- 172.21.15.50:0/3321248536 reap_dead start 2023-10-30T12:12:13.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe46809fa30 0x7fe468136b30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe45000d800 tx=0x7fe45000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 shutdown_connections 2023-10-30T12:12:13.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 --2- 172.21.15.50:0/3321248536 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe46809fa30 0x7fe468136b30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:13.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 >> 172.21.15.50:0/3321248536 conn(0x7fe4680a6200 msgr2=0x7fe4680a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:13.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 shutdown_connections 2023-10-30T12:12:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 -- 172.21.15.50:0/3321248536 wait complete. 2023-10-30T12:12:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:13.830+0000 7fe46df16640 1 librados: shutdown 2023-10-30T12:12:13.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:12:13.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:12:13.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:12:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.028+0000 7f9bfb5d0640 10 monclient: get_monmap_and_config 2023-10-30T12:12:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.028+0000 7f9bfb5d0640 10 monclient: build_initial_monmap 2023-10-30T12:12:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.028+0000 7f9bfb5d0640 10 monclient: monmap: 2023-10-30T12:12:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:14.029890+0000 2023-10-30T12:12:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:14.029890+0000 2023-10-30T12:12:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:14.032 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:14.033 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:14.033 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.029+0000 7f9bfb5d0640 1 Processor -- start 2023-10-30T12:12:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.029+0000 7f9bfb5d0640 1 -- start start 2023-10-30T12:12:14.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.029+0000 7f9bfb5d0640 10 monclient: init 2023-10-30T12:12:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.029+0000 7f9bfb5d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.029+0000 7f9bfb5d0640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:12:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bf409bb70 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9bf409d3f0 0x7f9bf4137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf409d3f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9bf409c660 0x7f9bf4139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf409c660 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4098170 con 0x7f9bf409bb70 2023-10-30T12:12:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf409cf60 con 0x7f9bf409d3f0 2023-10-30T12:12:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf4097b20 con 0x7f9bf409c660 2023-10-30T12:12:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bfb5d0640 10 monclient(hunting): authenticate will time out at 1579.213114s 2023-10-30T12:12:14.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9bf409d3f0 0x7f9bf4137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf936c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9bf409bb70 0x7f9bf409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50566/0 (socket says 172.21.15.50:50566) 2023-10-30T12:12:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf936c640 1 -- 172.21.15.50:0/2149288237 learned_addr learned my addr 172.21.15.50:0/2149288237 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:12:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): get_auth_request con 0x7f9bf409d3f0 auth_method 0 2023-10-30T12:12:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf9b6d640 1 --2- 172.21.15.50:0/2149288237 >> v2:172.21.15.118:3300/0 conn(0x7f9bf409c660 0x7f9bf4139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf9b6d640 10 monclient(hunting): get_auth_request con 0x7f9bf409c660 auth_method 0 2023-10-30T12:12:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf9b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf9b6d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf9b6d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_done global_id 164148 payload 274 2023-10-30T12:12:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 1 -- 172.21.15.50:0/2149288237 >> v2:172.21.15.118:3300/0 conn(0x7f9bf409c660 msgr2=0x7f9bf4139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 1 --2- 172.21.15.50:0/2149288237 >> v2:172.21.15.118:3300/0 conn(0x7f9bf409c660 0x7f9bf4139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 1 -- 172.21.15.50:0/2149288237 >> v1:172.21.15.50:6789/0 conn(0x7f9bf409bb70 legacy=0x7f9bf409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient: _finish_hunting 0 2023-10-30T12:12:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 1 monclient: found mon.noname-a 2023-10-30T12:12:14.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bf8b6b640 1 -- 172.21.15.50:0/2149288237 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4097e00 con 0x7f9bf409d3f0 2023-10-30T12:12:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bebfff640 1 -- 172.21.15.50:0/2149288237 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558123504 0 0) 0x7f9bf4098170 con 0x7f9bf409bb70 2023-10-30T12:12:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.030+0000 7f9bebfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:12:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bf8b6b640 1 --2- 172.21.15.50:0/2149288237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409d3f0 0x7f9bf4137960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9bf409fe70 tx=0x7f9be402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3588fb0a94ca77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 1 -- 172.21.15.50:0/2149288237 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4031710 con 0x7f9bf409d3f0 2023-10-30T12:12:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: dump: 2023-10-30T12:12:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.051 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.052 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: _finish_auth 0 2023-10-30T12:12:14.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: _check_auth_tickets 2023-10-30T12:12:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 5 monclient: authenticate success, global_id 164148 2023-10-30T12:12:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 1 -- 172.21.15.50:0/2149288237 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be402f050 con 0x7f9bf409d3f0 2023-10-30T12:12:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:14.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 4 set_mon_vals no callback set 2023-10-30T12:12:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 1 -- 172.21.15.50:0/2149288237 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4031a90 con 0x7f9bf409d3f0 2023-10-30T12:12:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bebfff640 10 monclient: dump: 2023-10-30T12:12:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.056 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.056 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 10 monclient: shutdown 2023-10-30T12:12:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2149288237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409d3f0 msgr2=0x7f9bf4137960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.031+0000 7f9bfb5d0640 1 --2- 172.21.15.50:0/2149288237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409d3f0 0x7f9bf4137960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9bf409fe70 tx=0x7f9be402b7b0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2149288237 shutdown_connections 2023-10-30T12:12:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 --2- 172.21.15.50:0/2149288237 >> v2:172.21.15.118:3300/0 conn(0x7f9bf409c660 0x7f9bf4139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 --2- 172.21.15.50:0/2149288237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409d3f0 0x7f9bf4137960 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2149288237 >> 172.21.15.50:0/2149288237 conn(0x7f9bf40a3980 msgr2=0x7f9bf40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2149288237 shutdown_connections 2023-10-30T12:12:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2149288237 wait complete. 2023-10-30T12:12:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 10 monclient: build_initial_monmap 2023-10-30T12:12:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 10 monclient: monmap: 2023-10-30T12:12:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:14.059 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:14.060 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:14.061 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 librados: starting msgr at 2023-10-30T12:12:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 librados: starting objecter 2023-10-30T12:12:14.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.032+0000 7f9bfb5d0640 1 Processor -- start 2023-10-30T12:12:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 -- start start 2023-10-30T12:12:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 librados: setting wanted keys 2023-10-30T12:12:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 librados: calling monclient init 2023-10-30T12:12:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient: init 2023-10-30T12:12:14.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409bb70 0x7f9bf41ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf409bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf409c660 0x7f9bf40964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf409c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:14.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf409d3f0 0x7f9bf4096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf409d3f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf40927d0 con 0x7f9bf409bb70 2023-10-30T12:12:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4096f70 con 0x7f9bf409d3f0 2023-10-30T12:12:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4097120 con 0x7f9bf409c660 2023-10-30T12:12:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bfb5d0640 10 monclient(hunting): authenticate will time out at 1579.216559s 2023-10-30T12:12:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409bb70 0x7f9bf41ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409bb70 0x7f9bf41ca200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41526/0 (socket says 172.21.15.50:41526) 2023-10-30T12:12:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf409c660 0x7f9bf40964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 -- 172.21.15.50:0/2689351151 learned_addr learned my addr 172.21.15.50:0/2689351151 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): get_auth_request con 0x7f9bf409bb70 auth_method 0 2023-10-30T12:12:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 10 monclient(hunting): get_auth_request con 0x7f9bf409c660 auth_method 0 2023-10-30T12:12:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient(hunting): handle_auth_done global_id 164154 payload 824 2023-10-30T12:12:14.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf409c660 msgr2=0x7f9bf40964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf409c660 0x7f9bf40964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf9b6d640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf409d3f0 0x7f9bf4096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf409d3f0 msgr2=0x7f9bf4096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9bf409d3f0 0x7f9bf4096a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient: _finish_hunting 0 2023-10-30T12:12:14.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 monclient: found mon.noname-a 2023-10-30T12:12:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:12:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.033+0000 7f9bf936c640 1 -- 172.21.15.50:0/2689351151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4009400 con 0x7f9bf409bb70 2023-10-30T12:12:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bf936c640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409bb70 0x7f9bf41ca200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000c520 tx=0x7f9bf000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 client.?.objecter ms_handle_connect 0x7f9bf409bb70 2023-10-30T12:12:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0007480 con 0x7f9bf409bb70 2023-10-30T12:12:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:12:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: dump: 2023-10-30T12:12:14.071 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.072 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.073 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:12:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:12:14.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 5 monclient: authenticate success, global_id 164154 2023-10-30T12:12:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf00075e0 con 0x7f9bf409bb70 2023-10-30T12:12:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf00059c0 con 0x7f9bf409bb70 2023-10-30T12:12:14.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 10 monclient: _renew_subs 2023-10-30T12:12:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf40976e0 con 0x7f9bf409bb70 2023-10-30T12:12:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9beb7fe640 4 set_mon_vals no callback set 2023-10-30T12:12:14.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:12:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9be9ffb640 10 monclient: dump: 2023-10-30T12:12:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.076 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.077 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.077 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 10 client.164154.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 10 monclient: _renew_subs 2023-10-30T12:12:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:12:14.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf40924f0 con 0x7f9bf409bb70 2023-10-30T12:12:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.034+0000 7f9bfb5d0640 1 librados: init done 2023-10-30T12:12:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9bf0005b20 con 0x7f9bf409bb70 2023-10-30T12:12:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bfb5d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bfb5d0640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406bbd0 0x7f9bf406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bfb5d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4073ff0 con 0x7f9bf406bbd0 2023-10-30T12:12:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9bd80392f0 0x7f9bd803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406bbd0 0x7f9bf406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient: get_auth_request con 0x7f9bf406bbd0 auth_method 0 2023-10-30T12:12:14.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f9bf004a210 con 0x7f9bf409bb70 2023-10-30T12:12:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 10 client.164154.objecter ms_dispatch 0x7f9bf4000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:14.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 3 client.164154.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 3 client.164154.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 20 client.164154.objecter dump_active .. 0 homeless 2023-10-30T12:12:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf9b6d640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9bd80392f0 0x7f9bd803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf9b6d640 10 monclient: get_auth_request con 0x7f9bd80392f0 auth_method 0 2023-10-30T12:12:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_done global_id 114680 payload 824 2023-10-30T12:12:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 1 -- 172.21.15.50:0/2689351151 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bf4097560 con 0x7f9bf406bbd0 2023-10-30T12:12:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 10 client.164154.objecter ms_handle_connect 0x7f9bf406bbd0 2023-10-30T12:12:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9be9ffb640 10 client.164154.objecter resend_mon_ops 2023-10-30T12:12:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.035+0000 7f9bf8b6b640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406bbd0 0x7f9bf406bfb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9bf406b140 tx=0x7f9be40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be40339d0 con 0x7f9bf406bbd0 2023-10-30T12:12:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 10 monclient: dump: 2023-10-30T12:12:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.086 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.086 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 10 client.164154.objecter ms_handle_connect 0x7f9bd80392f0 2023-10-30T12:12:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9bf9b6d640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9bd80392f0 0x7f9bd803b7b0 secure :-1 s=READY pgs=4132 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009eb0 tx=0x7f9bec00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9be4041070 con 0x7f9bf406bbd0 2023-10-30T12:12:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:14.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406bbd0 msgr2=0x7f9bf406bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406bbd0 0x7f9bf406bfb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9bf406b140 tx=0x7f9be40046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406bbd0 msgr2=0x7f9bf406bfb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:14.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.036+0000 7f9be9ffb640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406bbd0 0x7f9bf406bfb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bfb5d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bfb5d0640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406c4f0 0x7f9bf406fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bfb5d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf409f600 con 0x7f9bf406c4f0 2023-10-30T12:12:14.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406c4f0 0x7f9bf406fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient: get_auth_request con 0x7f9bf406c4f0 auth_method 0 2023-10-30T12:12:14.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 10 monclient(hunting): handle_auth_done global_id 114683 payload 824 2023-10-30T12:12:14.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.042+0000 7f9bf8b6b640 1 -- 172.21.15.50:0/2689351151 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9bf4092c40 con 0x7f9bf406c4f0 2023-10-30T12:12:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 10 client.164154.objecter ms_handle_connect 0x7f9bf406c4f0 2023-10-30T12:12:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 10 client.164154.objecter resend_mon_ops 2023-10-30T12:12:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bf8b6b640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406c4f0 0x7f9bf406fdc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9be402b7f0 tx=0x7f9be4009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be403d790 con 0x7f9bf406c4f0 2023-10-30T12:12:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 10 monclient: dump: 2023-10-30T12:12:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.094 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.095 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.096 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f9be403d8f0 con 0x7f9bf406c4f0 2023-10-30T12:12:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:14.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406c4f0 msgr2=0x7f9bf406fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406c4f0 0x7f9bf406fdc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9be402b7f0 tx=0x7f9be4009190 comp rx=0 tx=0).stop 2023-10-30T12:12:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406c4f0 msgr2=0x7f9bf406fdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9be9ffb640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9bf406c4f0 0x7f9bf406fdc0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9be402b7f0 tx=0x7f9be4009190 comp rx=0 tx=0).stop 2023-10-30T12:12:14.097 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:12:14.098 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 114, 2023-10-30T12:12:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:12:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.099 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:12:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:14.100 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:14.101 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:14.102 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:14.103 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:14.104 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.105 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:14.106 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:14.107 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:14.108 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:14.109 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:14.110 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.111 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:14.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.114 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:14.115 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:14.116 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:14.117 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:14.118 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:14.119 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:14.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:14.121 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.123 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:14.124 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.125 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:14.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:12:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.127 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:12:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.128 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:12:14.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.130 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:14.130 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:14.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 10 librados: watch_flush enter 2023-10-30T12:12:14.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 10 librados: watch_flush exit 2023-10-30T12:12:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 20 client.164154.objecter shutdown clearing up homeless session... 2023-10-30T12:12:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 10 client.164154.objecter successfully canceled tick 2023-10-30T12:12:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9bd80392f0 msgr2=0x7f9bd803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9bd80392f0 0x7f9bd803b7b0 secure :-1 s=READY pgs=4132 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009eb0 tx=0x7f9bec00b040 comp rx=0 tx=0).stop 2023-10-30T12:12:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 10 monclient: shutdown 2023-10-30T12:12:14.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409bb70 msgr2=0x7f9bf41ca200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 --2- 172.21.15.50:0/2689351151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9bf409bb70 0x7f9bf41ca200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000c520 tx=0x7f9bf000c9e0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bf9b6d640 1 -- 172.21.15.50:0/2689351151 reap_dead start 2023-10-30T12:12:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bf9b6d640 1 -- 172.21.15.50:0/2689351151 reap_dead start 2023-10-30T12:12:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 shutdown_connections 2023-10-30T12:12:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 >> 172.21.15.50:0/2689351151 conn(0x7f9bf40a3980 msgr2=0x7f9bf41ba540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 shutdown_connections 2023-10-30T12:12:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 -- 172.21.15.50:0/2689351151 wait complete. 2023-10-30T12:12:14.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.043+0000 7f9bfb5d0640 1 librados: shutdown 2023-10-30T12:12:14.136 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:12:14.136 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:12:14.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:12:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.241+0000 7f0ca8133640 10 monclient: get_monmap_and_config 2023-10-30T12:12:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.241+0000 7f0ca8133640 10 monclient: build_initial_monmap 2023-10-30T12:12:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.241+0000 7f0ca8133640 10 monclient: monmap: 2023-10-30T12:12:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:14.243005+0000 2023-10-30T12:12:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:14.243005+0000 2023-10-30T12:12:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:14.246 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.242+0000 7f0ca8133640 1 Processor -- start 2023-10-30T12:12:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.242+0000 7f0ca8133640 1 -- start start 2023-10-30T12:12:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.242+0000 7f0ca8133640 10 monclient: init 2023-10-30T12:12:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T12:12:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ca0132970 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0ca00a03d0 0x7f0ca013aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca00a03d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca013b020 0x7f0ca013d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca013b020 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca00a17d0 con 0x7f0ca0132970 2023-10-30T12:12:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca0133460 con 0x7f0ca00a03d0 2023-10-30T12:12:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00a1510 con 0x7f0ca013b020 2023-10-30T12:12:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca8133640 10 monclient(hunting): authenticate will time out at 1579.426188s 2023-10-30T12:12:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0ca00a03d0 0x7f0ca013aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca013b020 0x7f0ca013d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0ca00a03d0 0x7f0ca013aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41528/0 (socket says 172.21.15.50:41528) 2023-10-30T12:12:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca013b020 0x7f0ca013d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45236/0 (socket says 172.21.15.50:45236) 2023-10-30T12:12:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 1 -- 172.21.15.50:0/3032559378 learned_addr learned my addr 172.21.15.50:0/3032559378 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 10 monclient(hunting): get_auth_request con 0x7f0ca00a03d0 auth_method 0 2023-10-30T12:12:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): get_auth_request con 0x7f0ca013b020 auth_method 0 2023-10-30T12:12:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_done global_id 114686 payload 274 2023-10-30T12:12:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3032559378 >> v2:172.21.15.50:3300/0 conn(0x7f0ca00a03d0 msgr2=0x7f0ca013aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 1 --2- 172.21.15.50:0/3032559378 >> v2:172.21.15.50:3300/0 conn(0x7f0ca00a03d0 0x7f0ca013aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3032559378 >> v1:172.21.15.118:6789/0 conn(0x7f0ca0132970 legacy=0x7f0ca0132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient: _finish_hunting 0 2023-10-30T12:12:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 1 monclient: found mon.noname-b 2023-10-30T12:12:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3032559378 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca009f680 con 0x7f0ca013b020 2023-10-30T12:12:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.243+0000 7f0ca56ce640 1 --2- 172.21.15.50:0/3032559378 >> v2:172.21.15.50:3300/0 conn(0x7f0ca00a03d0 0x7f0ca013aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca66d0640 1 --2- 172.21.15.50:0/3032559378 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca013b020 0x7f0ca013d410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0c980097a0 tx=0x7f0c9802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6eea32a3e7d56be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 1 -- 172.21.15.50:0/3032559378 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9802f050 con 0x7f0ca013b020 2023-10-30T12:12:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: dump: 2023-10-30T12:12:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.267 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: _finish_auth 0 2023-10-30T12:12:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca8133640 5 monclient: authenticate success, global_id 114686 2023-10-30T12:12:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 1 -- 172.21.15.50:0/3032559378 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c98031700 con 0x7f0ca013b020 2023-10-30T12:12:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca8133640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca8133640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 1 -- 172.21.15.50:0/3032559378 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98031a90 con 0x7f0ca013b020 2023-10-30T12:12:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca8133640 4 set_mon_vals no callback set 2023-10-30T12:12:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.244+0000 7f0ca4ecd640 10 monclient: dump: 2023-10-30T12:12:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.270 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.271 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 10 monclient: shutdown 2023-10-30T12:12:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 -- 172.21.15.50:0/3032559378 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca013b020 msgr2=0x7f0ca013d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 --2- 172.21.15.50:0/3032559378 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca013b020 0x7f0ca013d410 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0c980097a0 tx=0x7f0c9802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 -- 172.21.15.50:0/3032559378 shutdown_connections 2023-10-30T12:12:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 --2- 172.21.15.50:0/3032559378 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca013b020 0x7f0ca013d410 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 --2- 172.21.15.50:0/3032559378 >> v2:172.21.15.50:3300/0 conn(0x7f0ca00a03d0 0x7f0ca013aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 -- 172.21.15.50:0/3032559378 >> 172.21.15.50:0/3032559378 conn(0x7f0ca00a4800 msgr2=0x7f0ca00a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 -- 172.21.15.50:0/3032559378 shutdown_connections 2023-10-30T12:12:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 -- 172.21.15.50:0/3032559378 wait complete. 2023-10-30T12:12:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 10 monclient: build_initial_monmap 2023-10-30T12:12:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 10 monclient: monmap: 2023-10-30T12:12:14.274 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:14.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:14.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:14.274 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:14.274 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:14.275 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:14.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:14.275 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:14.275 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 librados: starting msgr at 2023-10-30T12:12:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 librados: starting objecter 2023-10-30T12:12:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.245+0000 7f0ca8133640 1 Processor -- start 2023-10-30T12:12:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 -- start start 2023-10-30T12:12:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 librados: setting wanted keys 2023-10-30T12:12:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 librados: calling monclient init 2023-10-30T12:12:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient: init 2023-10-30T12:12:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:12:14.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca0132970 0x7f0ca0131350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca0132970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca013b020 0x7f0ca011a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca013b020 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca011a970 0x7f0ca011ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca011a970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca01cafd0 con 0x7f0ca0132970 2023-10-30T12:12:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca011c2f0 con 0x7f0ca013b020 2023-10-30T12:12:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca009d7c0 con 0x7f0ca011a970 2023-10-30T12:12:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca8133640 10 monclient(hunting): authenticate will time out at 1579.429484s 2023-10-30T12:12:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca0132970 0x7f0ca0131350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca011a970 0x7f0ca011ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca011a970 0x7f0ca011ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45244/0 (socket says 172.21.15.50:45244) 2023-10-30T12:12:14.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3436021737 learned_addr learned my addr 172.21.15.50:0/3436021737 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): get_auth_request con 0x7f0ca011a970 auth_method 0 2023-10-30T12:12:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 10 monclient(hunting): get_auth_request con 0x7f0ca0132970 auth_method 0 2023-10-30T12:12:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca56ce640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca013b020 0x7f0ca011a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca56ce640 10 monclient(hunting): get_auth_request con 0x7f0ca013b020 auth_method 0 2023-10-30T12:12:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca56ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca56ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca56ce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient(hunting): handle_auth_done global_id 114689 payload 824 2023-10-30T12:12:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca013b020 msgr2=0x7f0ca011a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca013b020 0x7f0ca011a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca0132970 msgr2=0x7f0ca0131350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca0132970 0x7f0ca0131350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient: _finish_hunting 0 2023-10-30T12:12:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 monclient: found mon.noname-c 2023-10-30T12:12:14.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:14.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3436021737 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c940094d0 con 0x7f0ca011a970 2023-10-30T12:12:14.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.246+0000 7f0ca5ecf640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ca0132970 0x7f0ca0131350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:14.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca66d0640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca011a970 0x7f0ca011ae00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0c9802b7e0 tx=0x7f0c980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 client.?.objecter ms_handle_connect 0x7f0ca011a970 2023-10-30T12:12:14.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:14.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c980339b0 con 0x7f0ca011a970 2023-10-30T12:12:14.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:14.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: dump: 2023-10-30T12:12:14.288 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.288 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.288 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.288 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.289 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.289 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.289 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.289 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.289 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.289 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:12:14.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:12:14.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:14.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 5 monclient: authenticate success, global_id 114689 2023-10-30T12:12:14.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c98033b10 con 0x7f0ca011a970 2023-10-30T12:12:14.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 10 monclient: _renew_subs 2023-10-30T12:12:14.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c98009400 con 0x7f0ca011a970 2023-10-30T12:12:14.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:14.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9803d410 con 0x7f0ca011a970 2023-10-30T12:12:14.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:14.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9e7fc640 10 monclient: dump: 2023-10-30T12:12:14.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.292 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.292 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.293 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.293 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 10 client.114689.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:14.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 10 monclient: _renew_subs 2023-10-30T12:12:14.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca011dd00 con 0x7f0ca011a970 2023-10-30T12:12:14.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0ca8133640 1 librados: init done 2023-10-30T12:12:14.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.247+0000 7f0c9ffff640 4 set_mon_vals no callback set 2023-10-30T12:12:14.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca8133640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:14.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca8133640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca00a03d0 0x7f0ca002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca8133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca0071a70 con 0x7f0ca00a03d0 2023-10-30T12:12:14.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0c9803a050 con 0x7f0ca011a970 2023-10-30T12:12:14.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0c9e7fc640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c8c0392f0 0x7f0c8c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f0c9806c960 con 0x7f0ca011a970 2023-10-30T12:12:14.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0c9e7fc640 10 client.114689.objecter ms_dispatch 0x7f0ca0000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:14.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0c9e7fc640 3 client.114689.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:14.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0c9e7fc640 3 client.114689.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:14.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0c9e7fc640 20 client.114689.objecter dump_active .. 0 homeless 2023-10-30T12:12:14.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca00a03d0 0x7f0ca002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 10 monclient: get_auth_request con 0x7f0ca00a03d0 auth_method 0 2023-10-30T12:12:14.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca56ce640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c8c0392f0 0x7f0c8c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca56ce640 10 monclient: get_auth_request con 0x7f0c8c0392f0 auth_method 0 2023-10-30T12:12:14.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.248+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_done global_id 154246 payload 824 2023-10-30T12:12:14.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0ca5ecf640 1 -- 172.21.15.50:0/3436021737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ca009f960 con 0x7f0ca00a03d0 2023-10-30T12:12:14.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 client.114689.objecter ms_handle_connect 0x7f0c8c0392f0 2023-10-30T12:12:14.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0ca56ce640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c8c0392f0 0x7f0c8c03b7b0 secure :-1 s=READY pgs=4133 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00a3800 tx=0x7f0c94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 client.114689.objecter ms_handle_connect 0x7f0ca00a03d0 2023-10-30T12:12:14.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 client.114689.objecter resend_mon_ops 2023-10-30T12:12:14.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0ca5ecf640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca00a03d0 0x7f0ca002b430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000ff70 tx=0x7f0c9000d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90017070 con 0x7f0ca00a03d0 2023-10-30T12:12:14.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:14.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 monclient: dump: 2023-10-30T12:12:14.301 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.302 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.302 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.302 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.302 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.302 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.303 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.303 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.303 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0c9000de00 con 0x7f0ca00a03d0 2023-10-30T12:12:14.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:14.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:14.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca00a03d0 msgr2=0x7f0ca002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca00a03d0 0x7f0ca002b430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000ff70 tx=0x7f0c9000d520 comp rx=0 tx=0).stop 2023-10-30T12:12:14.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca00a03d0 msgr2=0x7f0ca002b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:14.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.249+0000 7f0c9e7fc640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca00a03d0 0x7f0ca002b430 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000ff70 tx=0x7f0c9000d520 comp rx=0 tx=0).stop 2023-10-30T12:12:14.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca8133640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:14.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca8133640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca002b970 0x7f0ca002bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca8133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca013a150 con 0x7f0ca002b970 2023-10-30T12:12:14.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca002b970 0x7f0ca002bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient: get_auth_request con 0x7f0ca002b970 auth_method 0 2023-10-30T12:12:14.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 10 monclient(hunting): handle_auth_done global_id 154249 payload 824 2023-10-30T12:12:14.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.256+0000 7f0ca5ecf640 1 -- 172.21.15.50:0/3436021737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0ca004e160 con 0x7f0ca002b970 2023-10-30T12:12:14.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca5ecf640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca002b970 0x7f0ca002bd80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0c90002230 tx=0x7f0c90001250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 10 client.114689.objecter ms_handle_connect 0x7f0ca002b970 2023-10-30T12:12:14.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 10 client.114689.objecter resend_mon_ops 2023-10-30T12:12:14.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90003e10 con 0x7f0ca002b970 2023-10-30T12:12:14.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:14.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 10 monclient: dump: 2023-10-30T12:12:14.311 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.311 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.311 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.311 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.311 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.312 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.312 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.312 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.312 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.312 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f0c90005630 con 0x7f0ca002b970 2023-10-30T12:12:14.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:14.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:14.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca002b970 msgr2=0x7f0ca002bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca002b970 0x7f0ca002bd80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0c90002230 tx=0x7f0c90001250 comp rx=0 tx=0).stop 2023-10-30T12:12:14.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca002b970 msgr2=0x7f0ca002bd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:14.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0c9e7fc640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0ca002b970 0x7f0ca002bd80 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0c90002230 tx=0x7f0c90001250 comp rx=0 tx=0).stop 2023-10-30T12:12:14.314 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:14.314 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:12:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 114, 2023-10-30T12:12:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:12:14.315 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:14.316 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:14.316 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.316 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:12:14.316 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:14.316 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:14.317 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:14.317 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:14.317 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:14.317 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:14.317 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:14.318 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:14.318 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:14.318 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:14.318 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:14.318 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:14.319 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:14.319 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.319 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:14.319 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:14.319 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:14.320 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:14.320 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:14.320 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:14.320 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:14.320 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:14.321 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:14.321 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:14.321 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:14.321 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:14.321 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.322 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:14.322 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:14.322 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:14.322 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:14.323 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:14.323 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:14.323 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:14.323 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:14.323 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:14.323 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:14.338 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.338 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:14.338 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:14.339 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:14.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.339 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.339 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:14.340 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:14.340 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:14.340 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:14.340 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:14.340 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.340 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.341 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:14.341 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:14.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.341 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:14.341 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:14.342 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:14.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.342 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.342 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:14.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.343 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:14.343 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:14.343 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:12:14.343 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.343 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:14.344 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:12:14.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:14.344 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:14.344 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:14.344 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:14.344 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:14.345 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:14.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:14.345 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:14.345 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:14.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 10 librados: watch_flush enter 2023-10-30T12:12:14.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 10 librados: watch_flush exit 2023-10-30T12:12:14.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 20 client.114689.objecter shutdown clearing up homeless session... 2023-10-30T12:12:14.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 10 client.114689.objecter successfully canceled tick 2023-10-30T12:12:14.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c8c0392f0 msgr2=0x7f0c8c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c8c0392f0 0x7f0c8c03b7b0 secure :-1 s=READY pgs=4133 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00a3800 tx=0x7f0c94008040 comp rx=0 tx=0).stop 2023-10-30T12:12:14.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 10 monclient: shutdown 2023-10-30T12:12:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca011a970 msgr2=0x7f0ca011ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 1 --2- 172.21.15.50:0/3436021737 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0ca011a970 0x7f0ca011ae00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0c9802b7e0 tx=0x7f0c980046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3436021737 reap_dead start 2023-10-30T12:12:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca66d0640 1 -- 172.21.15.50:0/3436021737 reap_dead start 2023-10-30T12:12:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 shutdown_connections 2023-10-30T12:12:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 >> 172.21.15.50:0/3436021737 conn(0x7f0ca00a4800 msgr2=0x7f0ca00a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.257+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 shutdown_connections 2023-10-30T12:12:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.258+0000 7f0ca8133640 1 -- 172.21.15.50:0/3436021737 wait complete. 2023-10-30T12:12:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.258+0000 7f0ca8133640 1 librados: shutdown 2023-10-30T12:12:14.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:12:14.351 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:12:14.351 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:12:14.351 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:12:14.351 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:12:14.352 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:12:14.352 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:12:14.362 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:12:14.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:12:14.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:14.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.506+0000 7f03b0525640 10 monclient: get_monmap_and_config 2023-10-30T12:12:14.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.506+0000 7f03b0525640 10 monclient: build_initial_monmap 2023-10-30T12:12:14.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.506+0000 7f03b0525640 10 monclient: monmap: 2023-10-30T12:12:14.508 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:14.508 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.509 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:14.507351+0000 2023-10-30T12:12:14.509 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:14.507351+0000 2023-10-30T12:12:14.509 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:14.509 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.509 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:14.510 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:14.510 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:14.510 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:14.510 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:14.511 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.506+0000 7f03b0525640 1 Processor -- start 2023-10-30T12:12:14.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.506+0000 7f03b0525640 1 -- start start 2023-10-30T12:12:14.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.506+0000 7f03b0525640 10 monclient: init 2023-10-30T12:12:14.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-10-30T12:12:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03a80a7000 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a806abe0 0x7f03a813a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient(hunting): picked mon.noname-b con 0x7f03a806abe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:14.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03a8136290 0x7f03a813cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient(hunting): picked mon.noname-a con 0x7f03a8136290 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a80a73e0 con 0x7f03a80a7000 2023-10-30T12:12:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a809a000 con 0x7f03a8136290 2023-10-30T12:12:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a809a2e0 con 0x7f03a806abe0 2023-10-30T12:12:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03b0525640 10 monclient(hunting): authenticate will time out at 1579.690544s 2023-10-30T12:12:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03ae2c1640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f03a80a7000 legacy=0x7f03a8135b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:12:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03aeac2640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f03a8136290 msgr2=0x7f03a813cee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03aeac2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03a8136290 0x7f03a813cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a806abe0 0x7f03a813a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a806abe0 0x7f03a813a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45248/0 (socket says 172.21.15.50:45248) 2023-10-30T12:12:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 -- 172.21.15.50:0/866283214 learned_addr learned my addr 172.21.15.50:0/866283214 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): get_auth_request con 0x7f03a806abe0 auth_method 0 2023-10-30T12:12:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient(hunting): handle_auth_done global_id 114692 payload 274 2023-10-30T12:12:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 -- 172.21.15.50:0/866283214 >> v2:172.21.15.50:3300/0 conn(0x7f03a8136290 msgr2=0x7f03a813cee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 --2- 172.21.15.50:0/866283214 >> v2:172.21.15.50:3300/0 conn(0x7f03a8136290 0x7f03a813cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 -- 172.21.15.50:0/866283214 >> v1:172.21.15.50:6789/0 conn(0x7f03a80a7000 legacy=0x7f03a8135b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient: _finish_hunting 0 2023-10-30T12:12:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 monclient: found mon.noname-b 2023-10-30T12:12:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.507+0000 7f03adac0640 1 -- 172.21.15.50:0/866283214 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a809a5c0 con 0x7f03a806abe0 2023-10-30T12:12:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03adac0640 1 --2- 172.21.15.50:0/866283214 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a806abe0 0x7f03a813a990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f03a8133210 tx=0x7f039c02b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b536fe478606de3d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 1 -- 172.21.15.50:0/866283214 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c0317e0 con 0x7f03a806abe0 2023-10-30T12:12:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: dump: 2023-10-30T12:12:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.528 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: _finish_auth 0 2023-10-30T12:12:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: _check_auth_tickets 2023-10-30T12:12:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 1 -- 172.21.15.50:0/866283214 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c031940 con 0x7f03a806abe0 2023-10-30T12:12:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03b0525640 5 monclient: authenticate success, global_id 114692 2023-10-30T12:12:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03b0525640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03b0525640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 1 -- 172.21.15.50:0/866283214 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c031cd0 con 0x7f03a806abe0 2023-10-30T12:12:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03b0525640 4 set_mon_vals no callback set 2023-10-30T12:12:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.508+0000 7f03ad2bf640 10 monclient: dump: 2023-10-30T12:12:14.531 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.531 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.531 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.531 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.531 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.531 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.532 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.532 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.532 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.532 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 10 monclient: shutdown 2023-10-30T12:12:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 -- 172.21.15.50:0/866283214 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a806abe0 msgr2=0x7f03a813a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 --2- 172.21.15.50:0/866283214 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a806abe0 0x7f03a813a990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f03a8133210 tx=0x7f039c02b880 comp rx=0 tx=0).stop 2023-10-30T12:12:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 -- 172.21.15.50:0/866283214 shutdown_connections 2023-10-30T12:12:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 --2- 172.21.15.50:0/866283214 >> v2:172.21.15.50:3300/0 conn(0x7f03a8136290 0x7f03a813cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 --2- 172.21.15.50:0/866283214 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a806abe0 0x7f03a813a990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 -- 172.21.15.50:0/866283214 >> 172.21.15.50:0/866283214 conn(0x7f03a8066000 msgr2=0x7f03a80a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 -- 172.21.15.50:0/866283214 shutdown_connections 2023-10-30T12:12:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 1 -- 172.21.15.50:0/866283214 wait complete. 2023-10-30T12:12:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 10 monclient: build_initial_monmap 2023-10-30T12:12:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.509+0000 7f03b0525640 10 monclient: monmap: 2023-10-30T12:12:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:14.536 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 librados: starting msgr at 2023-10-30T12:12:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 librados: starting objecter 2023-10-30T12:12:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 Processor -- start 2023-10-30T12:12:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 -- start start 2023-10-30T12:12:14.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 librados: setting wanted keys 2023-10-30T12:12:14.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 librados: calling monclient init 2023-10-30T12:12:14.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient: init 2023-10-30T12:12:14.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:14.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03a806abe0 0x7f03a809ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): picked mon.noname-a con 0x7f03a806abe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a80a7000 0x7f03a8132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): picked mon.noname-c con 0x7f03a80a7000 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03a8136290 0x7f03a8132a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): picked mon.noname-b con 0x7f03a8136290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:14.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a809f600 con 0x7f03a806abe0 2023-10-30T12:12:14.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a809f850 con 0x7f03a8136290 2023-10-30T12:12:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80a1790 con 0x7f03a80a7000 2023-10-30T12:12:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03b0525640 10 monclient(hunting): authenticate will time out at 1579.693734s 2023-10-30T12:12:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03ae2c1640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03a806abe0 msgr2=0x7f03a809ffe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:14.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03ae2c1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03a806abe0 0x7f03a809ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a80a7000 0x7f03a8132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a80a7000 0x7f03a8132530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45260/0 (socket says 172.21.15.50:45260) 2023-10-30T12:12:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 1 -- 172.21.15.50:0/2681331578 learned_addr learned my addr 172.21.15.50:0/2681331578 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 10 monclient(hunting): get_auth_request con 0x7f03a80a7000 auth_method 0 2023-10-30T12:12:14.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:14.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:14.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.510+0000 7f03adac0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03aeac2640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03a8136290 0x7f03a8132a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03aeac2640 10 monclient(hunting): get_auth_request con 0x7f03a8136290 auth_method 0 2023-10-30T12:12:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03aeac2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03aeac2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03aeac2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 10 monclient(hunting): handle_auth_done global_id 114695 payload 824 2023-10-30T12:12:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 1 -- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03a8136290 msgr2=0x7f03a8132a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03a8136290 0x7f03a8132a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 1 -- 172.21.15.50:0/2681331578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03a806abe0 msgr2=0x7f03a809ffe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:14.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03a806abe0 0x7f03a809ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 10 monclient: _finish_hunting 0 2023-10-30T12:12:14.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 1 monclient: found mon.noname-c 2023-10-30T12:12:14.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:14.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 1 -- 172.21.15.50:0/2681331578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c0094d0 con 0x7f03a80a7000 2023-10-30T12:12:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03adac0640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a80a7000 0x7f03a8132530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f039c000c00 tx=0x7f039c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 client.?.objecter ms_handle_connect 0x7f03a80a7000 2023-10-30T12:12:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 1 -- 172.21.15.50:0/2681331578 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c032880 con 0x7f03a80a7000 2023-10-30T12:12:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:14.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: dump: 2023-10-30T12:12:14.548 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.548 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.548 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.549 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.549 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.549 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.549 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.549 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.549 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.550 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:12:14.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:14.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:14.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 1 -- 172.21.15.50:0/2681331578 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c032a00 con 0x7f03a80a7000 2023-10-30T12:12:14.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 5 monclient: authenticate success, global_id 114695 2023-10-30T12:12:14.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 10 monclient: _renew_subs 2023-10-30T12:12:14.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a813cb90 con 0x7f03a80a7000 2023-10-30T12:12:14.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:14.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 1 -- 172.21.15.50:0/2681331578 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c032db0 con 0x7f03a80a7000 2023-10-30T12:12:14.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:14.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:14.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03a6ffd640 10 monclient: dump: 2023-10-30T12:12:14.552 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:14.553 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:14.553 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.553 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:14.553 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:14.553 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:14.554 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:14.554 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:14.554 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:14.554 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:14.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03acabe640 4 set_mon_vals no callback set 2023-10-30T12:12:14.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 10 client.114695.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:14.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 10 monclient: _renew_subs 2023-10-30T12:12:14.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a809efb0 con 0x7f03a80a7000 2023-10-30T12:12:14.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.511+0000 7f03b0525640 1 librados: init done 2023-10-30T12:12:14.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03a6ffd640 1 -- 172.21.15.50:0/2681331578 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f039c0453f0 con 0x7f03a80a7000 2023-10-30T12:12:14.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03b0525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:14.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03b0525640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:14.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03b0525640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a804e160 con 0x7f03a80a7000 2023-10-30T12:12:14.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03a6ffd640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f03940392a0 0x7f039403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:14.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03a6ffd640 1 -- 172.21.15.50:0/2681331578 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f039c06cf80 con 0x7f03a80a7000 2023-10-30T12:12:14.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03a6ffd640 10 client.114695.objecter ms_dispatch 0x7f03a8000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:14.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03a6ffd640 3 client.114695.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:14.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03a6ffd640 3 client.114695.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:14.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03a6ffd640 20 client.114695.objecter dump_active .. 0 homeless 2023-10-30T12:12:14.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03ae2c1640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f03940392a0 0x7f039403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:14.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.512+0000 7f03ae2c1640 10 monclient: get_auth_request con 0x7f03940392a0 auth_method 0 2023-10-30T12:12:14.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.513+0000 7f03ae2c1640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f03940392a0 0x7f039403b760 secure :-1 s=READY pgs=4134 cs=0 l=1 rev1=1 crypto rx=0x7f03a0003980 tx=0x7f03a0006d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:14.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.513+0000 7f03a6ffd640 10 client.114695.objecter ms_handle_connect 0x7f03940392a0 2023-10-30T12:12:14.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.517+0000 7f03a6ffd640 1 -- 172.21.15.50:0/2681331578 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f039c030330 con 0x7f03a80a7000 2023-10-30T12:12:14.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.517+0000 7f03a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:14.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.517+0000 7f03a6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.759+0000 7f03b0525640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.759+0000 7f03b0525640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.759+0000 7f03b0525640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.759+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f03a8132fb0 con 0x7f03a80a7000 2023-10-30T12:12:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.759+0000 7f03a6ffd640 1 -- 172.21.15.50:0/2681331578 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f039c04d360 con 0x7f03a80a7000 2023-10-30T12:12:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.759+0000 7f03a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.759+0000 7f03a6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:14.763 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 10 librados: watch_flush enter 2023-10-30T12:12:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 10 librados: watch_flush exit 2023-10-30T12:12:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 20 client.114695.objecter shutdown clearing up homeless session... 2023-10-30T12:12:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 10 client.114695.objecter successfully canceled tick 2023-10-30T12:12:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f03940392a0 msgr2=0x7f039403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f03940392a0 0x7f039403b760 secure :-1 s=READY pgs=4134 cs=0 l=1 rev1=1 crypto rx=0x7f03a0003980 tx=0x7f03a0006d70 comp rx=0 tx=0).stop 2023-10-30T12:12:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 10 monclient: shutdown 2023-10-30T12:12:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a80a7000 msgr2=0x7f03a8132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a80a7000 0x7f03a8132530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f039c000c00 tx=0x7f039c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 shutdown_connections 2023-10-30T12:12:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f03940392a0 0x7f039403b760 unknown :-1 s=CLOSED pgs=4134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03a8136290 0x7f03a8132a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03a80a7000 0x7f03a8132530 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 --2- 172.21.15.50:0/2681331578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03a806abe0 0x7f03a809ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 >> 172.21.15.50:0/2681331578 conn(0x7f03a8066000 msgr2=0x7f03a80a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 shutdown_connections 2023-10-30T12:12:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 -- 172.21.15.50:0/2681331578 wait complete. 2023-10-30T12:12:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:14.761+0000 7f03b0525640 1 librados: shutdown 2023-10-30T12:12:14.898 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-10-30T12:12:14.896+0000 7f914ee4a640 -1 snap_mapper.add_oid found existing snaps mapped on 219:cd0cc7f7:test-rados-api-smithi118-87473-80::foo:2, removing 2023-10-30T12:12:14.899 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-10-30T12:12:14.897+0000 7f11953c4640 -1 snap_mapper.add_oid found existing snaps mapped on 219:cd0cc7f7:test-rados-api-smithi118-87473-80::foo:2, removing 2023-10-30T12:12:17.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.888+0000 7ff849e52640 10 monclient: get_monmap_and_config 2023-10-30T12:12:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.888+0000 7ff849e52640 10 monclient: build_initial_monmap 2023-10-30T12:12:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.888+0000 7ff849e52640 10 monclient: monmap: 2023-10-30T12:12:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:17.889896+0000 2023-10-30T12:12:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:17.889896+0000 2023-10-30T12:12:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:17.893 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.889+0000 7ff849e52640 1 Processor -- start 2023-10-30T12:12:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.889+0000 7ff849e52640 1 -- start start 2023-10-30T12:12:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.889+0000 7ff849e52640 10 monclient: init 2023-10-30T12:12:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T12:12:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff84409f5b0 0x7ff844099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient(hunting): picked mon.noname-b con 0x7ff84409f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff844096080 0x7ff844096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient(hunting): picked mon.noname-c con 0x7ff844096080 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8440969a0 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff844098810 con 0x7ff8440969a0 2023-10-30T12:12:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84406aef0 con 0x7ff844096080 2023-10-30T12:12:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84409a1a0 con 0x7ff84409f5b0 2023-10-30T12:12:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff849e52640 10 monclient(hunting): authenticate will time out at 1583.073334s 2023-10-30T12:12:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff843fff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7ff8440969a0 legacy=0x7ff84409cb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:12:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff84409f5b0 0x7ff844099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff84409f5b0 0x7ff844099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48004/0 (socket says 172.21.15.50:48004) 2023-10-30T12:12:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 -- 172.21.15.50:0/788701102 learned_addr learned my addr 172.21.15.50:0/788701102 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): get_auth_request con 0x7ff84409f5b0 auth_method 0 2023-10-30T12:12:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff842ffd640 1 --2- 172.21.15.50:0/788701102 >> v2:172.21.15.118:3300/0 conn(0x7ff844096080 0x7ff844096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient(hunting): handle_auth_done global_id 114698 payload 274 2023-10-30T12:12:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 -- 172.21.15.50:0/788701102 >> v2:172.21.15.118:3300/0 conn(0x7ff844096080 msgr2=0x7ff844096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 --2- 172.21.15.50:0/788701102 >> v2:172.21.15.118:3300/0 conn(0x7ff844096080 0x7ff844096460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 -- 172.21.15.50:0/788701102 >> v1:172.21.15.50:6789/0 conn(0x7ff8440969a0 legacy=0x7ff84409cb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient: _finish_hunting 0 2023-10-30T12:12:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 monclient: found mon.noname-b 2023-10-30T12:12:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff8437fe640 1 -- 172.21.15.50:0/788701102 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844098af0 con 0x7ff84409f5b0 2023-10-30T12:12:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff842ffd640 10 monclient: get_auth_request con 0x7ff844096080 auth_method 0 2023-10-30T12:12:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.890+0000 7ff842ffd640 1 --2- 172.21.15.50:0/788701102 >> v2:172.21.15.118:3300/0 conn(0x7ff844096080 0x7ff844096460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:12:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8437fe640 1 --2- 172.21.15.50:0/788701102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff84409f5b0 0x7ff844099b80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff834009880 tx=0x7ff83402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1d470a21d0f3f05 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 1 -- 172.21.15.50:0/788701102 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83402f050 con 0x7ff84409f5b0 2023-10-30T12:12:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: dump: 2023-10-30T12:12:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:17.911 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:17.911 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: _finish_auth 0 2023-10-30T12:12:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: _check_auth_tickets 2023-10-30T12:12:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 1 -- 172.21.15.50:0/788701102 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834004230 con 0x7ff84409f5b0 2023-10-30T12:12:17.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff849e52640 5 monclient: authenticate success, global_id 114698 2023-10-30T12:12:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff849e52640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff849e52640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 1 -- 172.21.15.50:0/788701102 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834003610 con 0x7ff84409f5b0 2023-10-30T12:12:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:17.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff849e52640 4 set_mon_vals no callback set 2023-10-30T12:12:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.891+0000 7ff8427fc640 10 monclient: dump: 2023-10-30T12:12:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:17.914 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:17.915 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:17.916 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 10 monclient: shutdown 2023-10-30T12:12:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 -- 172.21.15.50:0/788701102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff84409f5b0 msgr2=0x7ff844099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 --2- 172.21.15.50:0/788701102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff84409f5b0 0x7ff844099b80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff834009880 tx=0x7ff83402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:12:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 -- 172.21.15.50:0/788701102 shutdown_connections 2023-10-30T12:12:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 --2- 172.21.15.50:0/788701102 >> v2:172.21.15.118:3300/0 conn(0x7ff844096080 0x7ff844096460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 --2- 172.21.15.50:0/788701102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff84409f5b0 0x7ff844099b80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 -- 172.21.15.50:0/788701102 >> 172.21.15.50:0/788701102 conn(0x7ff8440a05c0 msgr2=0x7ff84409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 -- 172.21.15.50:0/788701102 shutdown_connections 2023-10-30T12:12:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 1 -- 172.21.15.50:0/788701102 wait complete. 2023-10-30T12:12:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 10 monclient: build_initial_monmap 2023-10-30T12:12:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.892+0000 7ff849e52640 10 monclient: monmap: 2023-10-30T12:12:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:17.918 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:17.920 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 librados: starting msgr at 2023-10-30T12:12:17.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 librados: starting objecter 2023-10-30T12:12:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 Processor -- start 2023-10-30T12:12:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 -- start start 2023-10-30T12:12:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 librados: setting wanted keys 2023-10-30T12:12:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 librados: calling monclient init 2023-10-30T12:12:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient: init 2023-10-30T12:12:17.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff844096080 0x7ff844070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): picked mon.noname-a con 0x7ff844096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8440969a0 0x7ff8441b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:17.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): picked mon.noname-c con 0x7ff8440969a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84409f5b0 0x7ff8441ad340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): picked mon.noname-b con 0x7ff84409f5b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8441af880 con 0x7ff844096080 2023-10-30T12:12:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8441b0290 con 0x7ff84409f5b0 2023-10-30T12:12:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff844071930 con 0x7ff8440969a0 2023-10-30T12:12:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff849e52640 10 monclient(hunting): authenticate will time out at 1583.076957s 2023-10-30T12:12:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.893+0000 7ff8437fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff844096080 msgr2=0x7ff844070bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:17.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff8437fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff844096080 0x7ff844070bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8440969a0 0x7ff8441b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8440969a0 0x7ff8441b6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48008/0 (socket says 172.21.15.50:48008) 2023-10-30T12:12:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 -- 172.21.15.50:0/1629200996 learned_addr learned my addr 172.21.15.50:0/1629200996 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): get_auth_request con 0x7ff8440969a0 auth_method 0 2023-10-30T12:12:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff843fff640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84409f5b0 0x7ff8441ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient(hunting): handle_auth_done global_id 114701 payload 824 2023-10-30T12:12:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 -- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84409f5b0 msgr2=0x7ff8441ad340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84409f5b0 0x7ff8441ad340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 -- 172.21.15.50:0/1629200996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff844096080 msgr2=0x7ff844070bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:17.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff844096080 0x7ff844070bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:12:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 monclient: found mon.noname-c 2023-10-30T12:12:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 -- 172.21.15.50:0/1629200996 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834009460 con 0x7ff8440969a0 2023-10-30T12:12:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff843fff640 10 monclient: get_auth_request con 0x7ff84409f5b0 auth_method 0 2023-10-30T12:12:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff843fff640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84409f5b0 0x7ff8441ad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:12:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff842ffd640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8440969a0 0x7ff8441b6a80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff83800c7f0 tx=0x7ff83800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 client.?.objecter ms_handle_connect 0x7ff8440969a0 2023-10-30T12:12:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 1 -- 172.21.15.50:0/1629200996 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff838007820 con 0x7ff8440969a0 2023-10-30T12:12:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: dump: 2023-10-30T12:12:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:17.934 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: _finish_auth 0 2023-10-30T12:12:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:12:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 1 -- 172.21.15.50:0/1629200996 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff838007980 con 0x7ff8440969a0 2023-10-30T12:12:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 5 monclient: authenticate success, global_id 114701 2023-10-30T12:12:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 10 monclient: _renew_subs 2023-10-30T12:12:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8441afc60 con 0x7ff8440969a0 2023-10-30T12:12:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 1 -- 172.21.15.50:0/1629200996 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff838007cf0 con 0x7ff8440969a0 2023-10-30T12:12:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff82bfff640 10 monclient: dump: 2023-10-30T12:12:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:17.939 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff841ffb640 4 set_mon_vals no callback set 2023-10-30T12:12:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 10 client.114701.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 10 monclient: _renew_subs 2023-10-30T12:12:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8441c87e0 con 0x7ff8440969a0 2023-10-30T12:12:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.894+0000 7ff849e52640 1 librados: init done 2023-10-30T12:12:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff82bfff640 1 -- 172.21.15.50:0/1629200996 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff8380113f0 con 0x7ff8440969a0 2023-10-30T12:12:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff849e52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff849e52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff849e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff82bfff640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8300392a0 0x7ff83003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8441c8a30 con 0x7ff8440969a0 2023-10-30T12:12:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff82bfff640 1 -- 172.21.15.50:0/1629200996 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7ff83804af00 con 0x7ff8440969a0 2023-10-30T12:12:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff82bfff640 10 client.114701.objecter ms_dispatch 0x7ff844000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff82bfff640 3 client.114701.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.895+0000 7ff82bfff640 3 client.114701.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.896+0000 7ff82bfff640 20 client.114701.objecter dump_active .. 0 homeless 2023-10-30T12:12:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.896+0000 7ff8437fe640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8300392a0 0x7ff83003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.896+0000 7ff8437fe640 10 monclient: get_auth_request con 0x7ff8300392a0 auth_method 0 2023-10-30T12:12:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.896+0000 7ff8437fe640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8300392a0 0x7ff83003b760 secure :-1 s=READY pgs=4135 cs=0 l=1 rev1=1 crypto rx=0x7ff834035000 tx=0x7ff834008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:17.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.896+0000 7ff82bfff640 10 client.114701.objecter ms_handle_connect 0x7ff8300392a0 2023-10-30T12:12:17.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.900+0000 7ff82bfff640 1 -- 172.21.15.50:0/1629200996 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff8380116a0 con 0x7ff8440969a0 2023-10-30T12:12:17.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.900+0000 7ff82bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:17.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:17.900+0000 7ff82bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.143+0000 7ff849e52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.143+0000 7ff849e52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.143+0000 7ff849e52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.143+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff84404e160 con 0x7ff8440969a0 2023-10-30T12:12:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.143+0000 7ff82bfff640 1 -- 172.21.15.50:0/1629200996 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7ff838035080 con 0x7ff8440969a0 2023-10-30T12:12:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.143+0000 7ff82bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.143+0000 7ff82bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:18.146 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 10 librados: watch_flush enter 2023-10-30T12:12:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 10 librados: watch_flush exit 2023-10-30T12:12:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 20 client.114701.objecter shutdown clearing up homeless session... 2023-10-30T12:12:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 10 client.114701.objecter successfully canceled tick 2023-10-30T12:12:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8300392a0 msgr2=0x7ff83003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8300392a0 0x7ff83003b760 secure :-1 s=READY pgs=4135 cs=0 l=1 rev1=1 crypto rx=0x7ff834035000 tx=0x7ff834008030 comp rx=0 tx=0).stop 2023-10-30T12:12:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 10 monclient: shutdown 2023-10-30T12:12:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8440969a0 msgr2=0x7ff8441b6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8440969a0 0x7ff8441b6a80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff83800c7f0 tx=0x7ff83800ccb0 comp rx=0 tx=0).stop 2023-10-30T12:12:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 shutdown_connections 2023-10-30T12:12:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8300392a0 0x7ff83003b760 unknown :-1 s=CLOSED pgs=4135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84409f5b0 0x7ff8441ad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8440969a0 0x7ff8441b6a80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 --2- 172.21.15.50:0/1629200996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff844096080 0x7ff844070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 >> 172.21.15.50:0/1629200996 conn(0x7ff8440a05c0 msgr2=0x7ff8441b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 shutdown_connections 2023-10-30T12:12:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 -- 172.21.15.50:0/1629200996 wait complete. 2023-10-30T12:12:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:18.145+0000 7ff849e52640 1 librados: shutdown 2023-10-30T12:12:21.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.270+0000 7efe4c990640 10 monclient: get_monmap_and_config 2023-10-30T12:12:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.270+0000 7efe4c990640 10 monclient: build_initial_monmap 2023-10-30T12:12:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.270+0000 7efe4c990640 10 monclient: monmap: 2023-10-30T12:12:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:21.271476+0000 2023-10-30T12:12:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:21.271476+0000 2023-10-30T12:12:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:21.275 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 1 Processor -- start 2023-10-30T12:12:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 1 -- start start 2023-10-30T12:12:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient: init 2023-10-30T12:12:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:12:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe48131b20 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efe48071bc0 0x7efe48071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient(hunting): picked mon.noname-c con 0x7efe48071bc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efe48136e40 0x7efe48139230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient(hunting): picked mon.noname-a con 0x7efe48136e40 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4809a9b0 con 0x7efe48131b20 2023-10-30T12:12:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe48131f00 con 0x7efe48136e40 2023-10-30T12:12:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe48132980 con 0x7efe48071bc0 2023-10-30T12:12:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe4c990640 10 monclient(hunting): authenticate will time out at 1586.454733s 2023-10-30T12:12:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe46d9d640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7efe48136e40 msgr2=0x7efe48139230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.271+0000 7efe46d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efe48136e40 0x7efe48139230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe4659c640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7efe48131b20 0x7efe48132270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:58272/0 (socket says 172.21.15.50:58272) 2023-10-30T12:12:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efe48071bc0 0x7efe48071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe4659c640 1 -- 172.21.15.50:0/17797090 learned_addr learned my addr 172.21.15.50:0/17797090 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:12:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): get_auth_request con 0x7efe48071bc0 auth_method 0 2023-10-30T12:12:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient(hunting): handle_auth_done global_id 154255 payload 274 2023-10-30T12:12:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 1 -- 172.21.15.50:0/17797090 >> v2:172.21.15.50:3300/0 conn(0x7efe48136e40 msgr2=0x7efe48139230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 1 --2- 172.21.15.50:0/17797090 >> v2:172.21.15.50:3300/0 conn(0x7efe48136e40 0x7efe48139230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 1 -- 172.21.15.50:0/17797090 >> v1:172.21.15.118:6789/0 conn(0x7efe48131b20 legacy=0x7efe48132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:12:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 1 monclient: found mon.noname-c 2023-10-30T12:12:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:12:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.272+0000 7efe45d9b640 1 -- 172.21.15.50:0/17797090 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4809d920 con 0x7efe48071bc0 2023-10-30T12:12:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe45d9b640 1 --2- 172.21.15.50:0/17797090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48071bc0 0x7efe48071fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe48098b70 tx=0x7efe3c02b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd01b29d20e75b28 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 1 -- 172.21.15.50:0/17797090 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c031740 con 0x7efe48071bc0 2023-10-30T12:12:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: dump: 2023-10-30T12:12:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:21.295 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: _finish_auth 0 2023-10-30T12:12:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: _check_auth_tickets 2023-10-30T12:12:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 1 -- 172.21.15.50:0/17797090 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c0318a0 con 0x7efe48071bc0 2023-10-30T12:12:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 5 monclient: authenticate success, global_id 154255 2023-10-30T12:12:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 1 -- 172.21.15.50:0/17797090 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c031c30 con 0x7efe48071bc0 2023-10-30T12:12:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 4 set_mon_vals no callback set 2023-10-30T12:12:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4559a640 10 monclient: dump: 2023-10-30T12:12:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:21.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 10 monclient: shutdown 2023-10-30T12:12:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 1 -- 172.21.15.50:0/17797090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48071bc0 msgr2=0x7efe48071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.273+0000 7efe4c990640 1 --2- 172.21.15.50:0/17797090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48071bc0 0x7efe48071fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe48098b70 tx=0x7efe3c02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 1 -- 172.21.15.50:0/17797090 shutdown_connections 2023-10-30T12:12:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 1 --2- 172.21.15.50:0/17797090 >> v2:172.21.15.50:3300/0 conn(0x7efe48136e40 0x7efe48139230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 1 --2- 172.21.15.50:0/17797090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48071bc0 0x7efe48071fc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 1 -- 172.21.15.50:0/17797090 >> 172.21.15.50:0/17797090 conn(0x7efe480a05c0 msgr2=0x7efe4809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:21.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 1 -- 172.21.15.50:0/17797090 shutdown_connections 2023-10-30T12:12:21.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 1 -- 172.21.15.50:0/17797090 wait complete. 2023-10-30T12:12:21.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 10 monclient: build_initial_monmap 2023-10-30T12:12:21.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.274+0000 7efe4c990640 10 monclient: monmap: 2023-10-30T12:12:21.308 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:21.308 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:21.308 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:21.309 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:21.309 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:21.309 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:21.310 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:21.310 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:21.310 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:21.310 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:21.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 librados: starting msgr at 2023-10-30T12:12:21.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 librados: starting objecter 2023-10-30T12:12:21.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 Processor -- start 2023-10-30T12:12:21.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 -- start start 2023-10-30T12:12:21.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 librados: setting wanted keys 2023-10-30T12:12:21.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 librados: calling monclient init 2023-10-30T12:12:21.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient: init 2023-10-30T12:12:21.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:21.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:12:21.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe48071bc0 0x7efe4813a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:21.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): picked mon.noname-c con 0x7efe48071bc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:21.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48131b20 0x7efe48132b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:21.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): picked mon.noname-b con 0x7efe48131b20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:21.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe48136e40 0x7efe48133090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:21.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): picked mon.noname-a con 0x7efe48136e40 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:21.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:21.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe48136b40 con 0x7efe48136e40 2023-10-30T12:12:21.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:21.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4813ae30 con 0x7efe48131b20 2023-10-30T12:12:21.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:21.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4809b9b0 con 0x7efe48071bc0 2023-10-30T12:12:21.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:21.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4c990640 10 monclient(hunting): authenticate will time out at 1586.458741s 2023-10-30T12:12:21.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe46d9d640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe48136e40 msgr2=0x7efe48133090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:21.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe46d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe48136e40 0x7efe48133090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:21.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe48071bc0 0x7efe4813a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:21.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe48071bc0 0x7efe4813a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48024/0 (socket says 172.21.15.50:48024) 2023-10-30T12:12:21.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 1 -- 172.21.15.50:0/2363767073 learned_addr learned my addr 172.21.15.50:0/2363767073 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:21.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 10 monclient(hunting): get_auth_request con 0x7efe48071bc0 auth_method 0 2023-10-30T12:12:21.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:21.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:21.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:21.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:21.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:21.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.275+0000 7efe4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:21.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe45d9b640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48131b20 0x7efe48132b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:21.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe45d9b640 10 monclient(hunting): get_auth_request con 0x7efe48131b20 auth_method 0 2023-10-30T12:12:21.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:21.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe45d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:21.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe45d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:21.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 10 monclient(hunting): handle_auth_done global_id 114704 payload 824 2023-10-30T12:12:21.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 1 -- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48131b20 msgr2=0x7efe48132b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:21.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48131b20 0x7efe48132b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 1 -- 172.21.15.50:0/2363767073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe48136e40 msgr2=0x7efe48133090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe48136e40 0x7efe48133090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 10 monclient: _finish_hunting 0 2023-10-30T12:12:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 1 monclient: found mon.noname-c 2023-10-30T12:12:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 1 -- 172.21.15.50:0/2363767073 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c009400 con 0x7efe48071bc0 2023-10-30T12:12:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe45d9b640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48131b20 0x7efe48132b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:12:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4659c640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe48071bc0 0x7efe4813a8f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c480 tx=0x7efe3800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 client.?.objecter ms_handle_connect 0x7efe48071bc0 2023-10-30T12:12:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 1 -- 172.21.15.50:0/2363767073 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38005d30 con 0x7efe48071bc0 2023-10-30T12:12:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: dump: 2023-10-30T12:12:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:21.325 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: _finish_auth 0 2023-10-30T12:12:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:12:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 5 monclient: authenticate success, global_id 114704 2023-10-30T12:12:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 10 monclient: _renew_subs 2023-10-30T12:12:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe48136480 con 0x7efe48071bc0 2023-10-30T12:12:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 1 -- 172.21.15.50:0/2363767073 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe38005e90 con 0x7efe48071bc0 2023-10-30T12:12:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 1 -- 172.21.15.50:0/2363767073 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38007270 con 0x7efe48071bc0 2023-10-30T12:12:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe36ffd640 10 monclient: dump: 2023-10-30T12:12:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:21.332 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:21.332 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:21.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe44d99640 4 set_mon_vals no callback set 2023-10-30T12:12:21.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 10 client.114704.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:21.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 10 monclient: _renew_subs 2023-10-30T12:12:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe48096c50 con 0x7efe48071bc0 2023-10-30T12:12:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.276+0000 7efe4c990640 1 librados: init done 2023-10-30T12:12:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe36ffd640 1 -- 172.21.15.50:0/2363767073 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efe380073d0 con 0x7efe48071bc0 2023-10-30T12:12:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe4c990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe4c990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe4c990640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe36ffd640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe280392a0 0x7efe2803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe480988a0 con 0x7efe48071bc0 2023-10-30T12:12:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe36ffd640 1 -- 172.21.15.50:0/2363767073 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7efe3804a340 con 0x7efe48071bc0 2023-10-30T12:12:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe36ffd640 10 client.114704.objecter ms_dispatch 0x7efe48000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe36ffd640 3 client.114704.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe36ffd640 3 client.114704.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe36ffd640 20 client.114704.objecter dump_active .. 0 homeless 2023-10-30T12:12:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe45d9b640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe280392a0 0x7efe2803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.277+0000 7efe45d9b640 10 monclient: get_auth_request con 0x7efe280392a0 auth_method 0 2023-10-30T12:12:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.278+0000 7efe45d9b640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe280392a0 0x7efe2803b760 secure :-1 s=READY pgs=4136 cs=0 l=1 rev1=1 crypto rx=0x7efe3c035000 tx=0x7efe3c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.278+0000 7efe36ffd640 10 client.114704.objecter ms_handle_connect 0x7efe280392a0 2023-10-30T12:12:21.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.282+0000 7efe36ffd640 1 -- 172.21.15.50:0/2363767073 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efe38017070 con 0x7efe48071bc0 2023-10-30T12:12:21.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.282+0000 7efe36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:21.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.282+0000 7efe36ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.525+0000 7efe4c990640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.525+0000 7efe4c990640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.525+0000 7efe4c990640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:21.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.525+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efe4804e160 con 0x7efe48071bc0 2023-10-30T12:12:21.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.526+0000 7efe36ffd640 1 -- 172.21.15.50:0/2363767073 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7efe38019730 con 0x7efe48071bc0 2023-10-30T12:12:21.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.526+0000 7efe36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:21.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.526+0000 7efe36ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:21.529 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:21.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 10 librados: watch_flush enter 2023-10-30T12:12:21.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 10 librados: watch_flush exit 2023-10-30T12:12:21.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 20 client.114704.objecter shutdown clearing up homeless session... 2023-10-30T12:12:21.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 10 client.114704.objecter successfully canceled tick 2023-10-30T12:12:21.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe280392a0 msgr2=0x7efe2803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:21.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe280392a0 0x7efe2803b760 secure :-1 s=READY pgs=4136 cs=0 l=1 rev1=1 crypto rx=0x7efe3c035000 tx=0x7efe3c008030 comp rx=0 tx=0).stop 2023-10-30T12:12:21.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 10 monclient: shutdown 2023-10-30T12:12:21.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe48071bc0 msgr2=0x7efe4813a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe48071bc0 0x7efe4813a8f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c480 tx=0x7efe3800c940 comp rx=0 tx=0).stop 2023-10-30T12:12:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 shutdown_connections 2023-10-30T12:12:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efe280392a0 0x7efe2803b760 unknown :-1 s=CLOSED pgs=4136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe48136e40 0x7efe48133090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe48131b20 0x7efe48132b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 --2- 172.21.15.50:0/2363767073 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efe48071bc0 0x7efe4813a8f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 >> 172.21.15.50:0/2363767073 conn(0x7efe480a05c0 msgr2=0x7efe480a1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 shutdown_connections 2023-10-30T12:12:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.527+0000 7efe4c990640 1 -- 172.21.15.50:0/2363767073 wait complete. 2023-10-30T12:12:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:21.528+0000 7efe4c990640 1 librados: shutdown 2023-10-30T12:12:24.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.655+0000 7f4bf8256640 10 monclient: get_monmap_and_config 2023-10-30T12:12:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.655+0000 7f4bf8256640 10 monclient: build_initial_monmap 2023-10-30T12:12:24.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.655+0000 7f4bf8256640 10 monclient: monmap: 2023-10-30T12:12:24.658 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:24.658 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:24.658 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:24.656673+0000 2023-10-30T12:12:24.658 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:24.656673+0000 2023-10-30T12:12:24.659 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:24.659 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:24.659 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:24.659 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:24.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:24.660 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:24.660 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:24.660 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:24.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.656+0000 7f4bf8256640 1 Processor -- start 2023-10-30T12:12:24.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.656+0000 7f4bf8256640 1 -- start start 2023-10-30T12:12:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.656+0000 7f4bf8256640 10 monclient: init 2023-10-30T12:12:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.656+0000 7f4bf8256640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.656+0000 7f4bf8256640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-10-30T12:12:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.656+0000 7f4bf8256640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 10 monclient(hunting): picked mon.noname-c con 0x7f4bf009f5b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bf009a910 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 10 monclient(hunting): picked mon.noname-b con 0x7f4bf0099120 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf009f990 con 0x7f4bf009a910 2023-10-30T12:12:24.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf0099de0 con 0x7f4bf009f5b0 2023-10-30T12:12:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf0099820 con 0x7f4bf0099120 2023-10-30T12:12:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf8256640 10 monclient(hunting): authenticate will time out at 1589.840058s 2023-10-30T12:12:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf57f1640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f4bf009a910 legacy=0x7f4bf0097950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:12:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48028/0 (socket says 172.21.15.50:48028) 2023-10-30T12:12:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 learned_addr learned my addr 172.21.15.50:0/3940815467 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request con 0x7f4bf0099120 auth_method 0 2023-10-30T12:12:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:24.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request con 0x7f4bf009f5b0 auth_method 0 2023-10-30T12:12:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:24.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:24.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.657+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf57f1640 1 -- 172.21.15.50:0/3940815467 >> v1:172.21.15.50:6789/0 conn(0x7f4bf009a910 legacy=0x7f4bf0097950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:12:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request con 0x7f4bf0099120 auth_method 0 2023-10-30T12:12:24.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:24.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:24.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:24.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request con 0x7f4bf009f5b0 auth_method 0 2023-10-30T12:12:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:24.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:24.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:24.858+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf57f1640 1 -- 172.21.15.50:0/3940815467 >> v1:172.21.15.50:6789/0 conn(0x7f4bf009a910 legacy=0x7f4bf0097950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:12:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request con 0x7f4bf0099120 auth_method 0 2023-10-30T12:12:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request con 0x7f4bf009f5b0 auth_method 0 2023-10-30T12:12:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:25.259+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:26.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf57f1640 1 -- 172.21.15.50:0/3940815467 >> v1:172.21.15.50:6789/0 conn(0x7f4bf009a910 legacy=0x7f4bf0097950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:12:26.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request con 0x7f4bf0099120 auth_method 0 2023-10-30T12:12:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:26.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:26.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:26.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:26.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:26.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:26.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request con 0x7f4bf009f5b0 auth_method 0 2023-10-30T12:12:26.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:26.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:26.060+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient(hunting): tick 2023-10-30T12:12:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:12:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:12:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:12:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 msgr2=0x7f4bf0141790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bf0099120 0x7f4bf0141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 msgr2=0x7f4bf0097410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.118:3300/0 conn(0x7f4bf009f5b0 0x7f4bf0097410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 -- 172.21.15.50:0/3940815467 >> v1:172.21.15.50:6789/0 conn(0x7f4bf009a910 legacy=0x7f4bf0097950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient: _add_conns ranks=[3,1] 2023-10-30T12:12:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 0x7f4bd4005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4bd40036b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bd4006220 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 -- 172.21.15.50:0/3940815467 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bf006aef0 con 0x7f4bd4006220 2023-10-30T12:12:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 1 -- 172.21.15.50:0/3940815467 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bf0099de0 con 0x7f4bd40036b0 2023-10-30T12:12:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4be77fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 msgr2=0x7f4bd4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.657+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 0x7f4bd4005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:27.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.858+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 msgr2=0x7f4bd4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:27.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:27.858+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 0x7f4bd4005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:28.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.259+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 msgr2=0x7f4bd4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:28.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.259+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 0x7f4bd4005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.936+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228102498 0 0) 0x7f4bf006aef0 con 0x7f4bd4006220 2023-10-30T12:12:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.936+0000 7f4bf4ff0640 10 monclient(hunting): _negotiate 2023-10-30T12:12:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.936+0000 7f4bf4ff0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.936+0000 7f4bf4ff0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.936+0000 7f4bf4ff0640 10 monclient(hunting): my global_id is 154261 2023-10-30T12:12:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.936+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc003520 con 0x7f4bd4006220 2023-10-30T12:12:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133492652 0 0) 0x7f4bdc003520 con 0x7f4bd4006220 2023-10-30T12:12:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 msgr2=0x7f4bd4005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 --2- 172.21.15.50:0/3940815467 >> v2:172.21.15.50:3300/0 conn(0x7f4bd40036b0 0x7f4bd4005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: _finish_hunting 0 2023-10-30T12:12:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:12:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:12:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf009f990 con 0x7f4bd4006220 2023-10-30T12:12:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: _finish_auth 0 2023-10-30T12:12:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: _check_auth_tickets 2023-10-30T12:12:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f4be0004240 con 0x7f4bd4006220 2023-10-30T12:12:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf8256640 5 monclient: authenticate success, global_id 154261 2023-10-30T12:12:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf8256640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:12:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: dump: 2023-10-30T12:12:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:28.947 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:12:28.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:28.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 >> v1:172.21.15.118:6789/0 conn(0x7f4bd4006220 legacy=0x7f4bd40086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:28.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 1 -- 172.21.15.50:0/3940815467 reap_dead start 2023-10-30T12:12:28.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:12:28.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bdc003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:28.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): picked mon.a con 0x7f4bdc003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:28.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0047c0 0x7f4bdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:28.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): picked mon.b con 0x7f4bdc0047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:28.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0052a0 0x7f4bdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:28.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): picked mon.c con 0x7f4bdc0052a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf0099de0 con 0x7f4bdc003970 2023-10-30T12:12:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc00c1f0 con 0x7f4bdc0047c0 2023-10-30T12:12:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc00c3b0 con 0x7f4bdc0052a0 2023-10-30T12:12:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf4ff0640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:12:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf5ff2640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 msgr2=0x7f4bdc003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf8256640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bdc003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0052a0 0x7f4bdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request con 0x7f4bdc0052a0 auth_method 0 2023-10-30T12:12:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0047c0 0x7f4bdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): get_auth_request con 0x7f4bdc0047c0 auth_method 0 2023-10-30T12:12:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:28.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:28.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:28.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf67f3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:12:28.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:28.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:12:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_done global_id 154261 payload 293 2023-10-30T12:12:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0052a0 msgr2=0x7f4bdc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0052a0 0x7f4bdc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 msgr2=0x7f4bdc003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bdc003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 10 monclient: _finish_hunting 0 2023-10-30T12:12:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.938+0000 7f4bf57f1640 1 monclient: found mon.b 2023-10-30T12:12:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf57f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf57f1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:12:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf57f1640 1 -- 172.21.15.50:0/3940815467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc003520 con 0x7f4bdc0047c0 2023-10-30T12:12:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf57f1640 10 monclient: _finish_auth 0 2023-10-30T12:12:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf57f1640 10 monclient: _check_auth_tickets 2023-10-30T12:12:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf57f1640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0052a0 0x7f4bdc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:12:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.939+0000 7f4bf57f1640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0047c0 0x7f4bdc004c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4be0000c00 tx=0x7f4bdc0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e164204ae37b2e5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0006270 con 0x7f4bdc0047c0 2023-10-30T12:12:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 10 monclient: dump: 2023-10-30T12:12:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:28.960 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be002c8e0 con 0x7f4bdc0047c0 2023-10-30T12:12:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf8256640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf8256640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 1 -- 172.21.15.50:0/3940815467 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be002cc90 con 0x7f4bdc0047c0 2023-10-30T12:12:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf8256640 4 set_mon_vals no callback set 2023-10-30T12:12:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.940+0000 7f4bf4ff0640 10 monclient: dump: 2023-10-30T12:12:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:28.964 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 10 monclient: shutdown 2023-10-30T12:12:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 -- 172.21.15.50:0/3940815467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0047c0 msgr2=0x7f4bdc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0047c0 0x7f4bdc004c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4be0000c00 tx=0x7f4bdc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 -- 172.21.15.50:0/3940815467 shutdown_connections 2023-10-30T12:12:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0052a0 0x7f4bdc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0047c0 0x7f4bdc004c40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 --2- 172.21.15.50:0/3940815467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bdc003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 -- 172.21.15.50:0/3940815467 >> 172.21.15.50:0/3940815467 conn(0x7f4bf00a05c0 msgr2=0x7f4bf009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 -- 172.21.15.50:0/3940815467 shutdown_connections 2023-10-30T12:12:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.941+0000 7f4bf8256640 1 -- 172.21.15.50:0/3940815467 wait complete. 2023-10-30T12:12:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 10 monclient: build_initial_monmap 2023-10-30T12:12:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 10 monclient: monmap: 2023-10-30T12:12:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:28.972 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 1 librados: starting msgr at 2023-10-30T12:12:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 1 librados: starting objecter 2023-10-30T12:12:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 1 Processor -- start 2023-10-30T12:12:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 1 -- start start 2023-10-30T12:12:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 1 librados: setting wanted keys 2023-10-30T12:12:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 1 librados: calling monclient init 2023-10-30T12:12:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.942+0000 7f4bf8256640 10 monclient: init 2023-10-30T12:12:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bf01bb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): picked mon.noname-a con 0x7f4bdc003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0047c0 0x7f4bf01bbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): picked mon.noname-c con 0x7f4bdc0047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0052a0 0x7f4bf00940c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): picked mon.noname-b con 0x7f4bdc0052a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00a2a00 con 0x7f4bdc003970 2023-10-30T12:12:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf006ca30 con 0x7f4bdc0052a0 2023-10-30T12:12:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf006cd10 con 0x7f4bdc0047c0 2023-10-30T12:12:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf8256640 10 monclient(hunting): authenticate will time out at 1594.126242s 2023-10-30T12:12:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf5ff2640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 msgr2=0x7f4bf01bb690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf5ff2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bf01bb690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0047c0 0x7f4bf01bbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0047c0 0x7f4bf01bbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43048/0 (socket says 172.21.15.50:43048) 2023-10-30T12:12:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 -- 172.21.15.50:0/4094039729 learned_addr learned my addr 172.21.15.50:0/4094039729 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): get_auth_request con 0x7f4bdc0047c0 auth_method 0 2023-10-30T12:12:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0052a0 0x7f4bf00940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request con 0x7f4bdc0052a0 auth_method 0 2023-10-30T12:12:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf67f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient(hunting): handle_auth_done global_id 114707 payload 824 2023-10-30T12:12:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 -- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0052a0 msgr2=0x7f4bf00940c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0052a0 0x7f4bf00940c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 -- 172.21.15.50:0/4094039729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 msgr2=0x7f4bf01bb690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bf01bb690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient: _finish_hunting 0 2023-10-30T12:12:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 monclient: found mon.noname-c 2023-10-30T12:12:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.943+0000 7f4bf57f1640 1 -- 172.21.15.50:0/4094039729 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf0096ec0 con 0x7f4bdc0047c0 2023-10-30T12:12:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf57f1640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0047c0 0x7f4bf01bbbd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4be002c550 tx=0x7f4be0006480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 client.?.objecter ms_handle_connect 0x7f4bdc0047c0 2023-10-30T12:12:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 1 -- 172.21.15.50:0/4094039729 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0035670 con 0x7f4bdc0047c0 2023-10-30T12:12:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: dump: 2023-10-30T12:12:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:28.986 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: _finish_auth 0 2023-10-30T12:12:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:12:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 1 -- 172.21.15.50:0/4094039729 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be00357d0 con 0x7f4bdc0047c0 2023-10-30T12:12:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 5 monclient: authenticate success, global_id 114707 2023-10-30T12:12:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 10 monclient: _renew_subs 2023-10-30T12:12:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf0096720 con 0x7f4bdc0047c0 2023-10-30T12:12:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 1 -- 172.21.15.50:0/4094039729 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0035b60 con 0x7f4bdc0047c0 2023-10-30T12:12:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be67fc640 10 monclient: dump: 2023-10-30T12:12:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:28.990 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 10 client.114707.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 10 monclient: _renew_subs 2023-10-30T12:12:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf0095d60 con 0x7f4bdc0047c0 2023-10-30T12:12:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4bf8256640 1 librados: init done 2023-10-30T12:12:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.944+0000 7f4be7fff640 4 set_mon_vals no callback set 2023-10-30T12:12:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.945+0000 7f4bf8256640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.945+0000 7f4bf8256640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.945+0000 7f4bf8256640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.945+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf004e160 con 0x7f4bdc0047c0 2023-10-30T12:12:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4be67fc640 1 -- 172.21.15.50:0/4094039729 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4be0032070 con 0x7f4bdc0047c0 2023-10-30T12:12:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4be67fc640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bd4039920 0x7f4bd403bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4be67fc640 1 -- 172.21.15.50:0/4094039729 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f4be003c040 con 0x7f4bdc0047c0 2023-10-30T12:12:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4be67fc640 10 client.114707.objecter ms_dispatch 0x7f4bf0000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4be67fc640 3 client.114707.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4be67fc640 3 client.114707.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4be67fc640 20 client.114707.objecter dump_active .. 0 homeless 2023-10-30T12:12:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bd4039920 0x7f4bd403bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.947+0000 7f4bf5ff2640 10 monclient: get_auth_request con 0x7f4bd4039920 auth_method 0 2023-10-30T12:12:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.948+0000 7f4bf5ff2640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bd4039920 0x7f4bd403bde0 secure :-1 s=READY pgs=4137 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0095960 tx=0x7f4bec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.948+0000 7f4be67fc640 10 client.114707.objecter ms_handle_connect 0x7f4bd4039920 2023-10-30T12:12:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.952+0000 7f4be67fc640 1 -- 172.21.15.50:0/4094039729 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4be0007dc0 con 0x7f4bdc0047c0 2023-10-30T12:12:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.952+0000 7f4be67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:28.952+0000 7f4be67fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.203+0000 7f4bf8256640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.203+0000 7f4bf8256640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.203+0000 7f4bf8256640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.203+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4bf00745c0 con 0x7f4bdc0047c0 2023-10-30T12:12:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.204+0000 7f4be67fc640 1 -- 172.21.15.50:0/4094039729 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f4be0033b60 con 0x7f4bdc0047c0 2023-10-30T12:12:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.204+0000 7f4be67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.204+0000 7f4be67fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:29.207 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":116,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 10 librados: watch_flush enter 2023-10-30T12:12:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 10 librados: watch_flush exit 2023-10-30T12:12:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 20 client.114707.objecter shutdown clearing up homeless session... 2023-10-30T12:12:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 10 client.114707.objecter successfully canceled tick 2023-10-30T12:12:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bd4039920 msgr2=0x7f4bd403bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bd4039920 0x7f4bd403bde0 secure :-1 s=READY pgs=4137 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0095960 tx=0x7f4bec008040 comp rx=0 tx=0).stop 2023-10-30T12:12:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 10 monclient: shutdown 2023-10-30T12:12:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0047c0 msgr2=0x7f4bf01bbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0047c0 0x7f4bf01bbbd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4be002c550 tx=0x7f4be0006480 comp rx=0 tx=0).stop 2023-10-30T12:12:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 shutdown_connections 2023-10-30T12:12:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4bd4039920 0x7f4bd403bde0 unknown :-1 s=CLOSED pgs=4137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4bdc0052a0 0x7f4bf00940c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4bdc0047c0 0x7f4bf01bbbd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 --2- 172.21.15.50:0/4094039729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4bdc003970 0x7f4bf01bb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 >> 172.21.15.50:0/4094039729 conn(0x7f4bf00a05c0 msgr2=0x7f4bf009ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 shutdown_connections 2023-10-30T12:12:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 -- 172.21.15.50:0/4094039729 wait complete. 2023-10-30T12:12:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.206+0000 7f4bf8256640 1 librados: shutdown 2023-10-30T12:12:29.218 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:12:29.218 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:12:29.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:12:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.323+0000 7fef3f19e640 10 monclient: get_monmap_and_config 2023-10-30T12:12:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.323+0000 7fef3f19e640 10 monclient: build_initial_monmap 2023-10-30T12:12:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.323+0000 7fef3f19e640 10 monclient: monmap: 2023-10-30T12:12:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:29.324192+0000 2023-10-30T12:12:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:29.324192+0000 2023-10-30T12:12:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:29.327 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.323+0000 7fef3f19e640 1 Processor -- start 2023-10-30T12:12:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.323+0000 7fef3f19e640 1 -- start start 2023-10-30T12:12:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.323+0000 7fef3f19e640 10 monclient: init 2023-10-30T12:12:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-10-30T12:12:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef380a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fef380a2450 0x7fef3813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient(hunting): picked mon.noname-c con 0x7fef380a2450 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef3813af40 0x7fef3813d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient(hunting): picked mon.noname-b con 0x7fef3813af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef380a4190 con 0x7fef380a2ae0 2023-10-30T12:12:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef380a1630 con 0x7fef380a2450 2023-10-30T12:12:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3809a2e0 con 0x7fef3813af40 2023-10-30T12:12:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3f19e640 10 monclient(hunting): authenticate will time out at 1594.507359s 2023-10-30T12:12:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef3813af40 0x7fef3813d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef3813af40 0x7fef3813d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43054/0 (socket says 172.21.15.50:43054) 2023-10-30T12:12:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 -- 172.21.15.50:0/4091086759 learned_addr learned my addr 172.21.15.50:0/4091086759 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): get_auth_request con 0x7fef3813af40 auth_method 0 2023-10-30T12:12:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 1 --2- 172.21.15.50:0/4091086759 >> v2:172.21.15.118:3300/0 conn(0x7fef380a2450 0x7fef3813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 10 monclient(hunting): get_auth_request con 0x7fef380a2450 auth_method 0 2023-10-30T12:12:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient(hunting): handle_auth_done global_id 114710 payload 274 2023-10-30T12:12:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 -- 172.21.15.50:0/4091086759 >> v2:172.21.15.118:3300/0 conn(0x7fef380a2450 msgr2=0x7fef3813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 --2- 172.21.15.50:0/4091086759 >> v2:172.21.15.118:3300/0 conn(0x7fef380a2450 0x7fef3813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 -- 172.21.15.50:0/4091086759 >> v1:172.21.15.118:6789/0 conn(0x7fef380a2ae0 legacy=0x7fef380a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient: _finish_hunting 0 2023-10-30T12:12:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 monclient: found mon.noname-b 2023-10-30T12:12:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.324+0000 7fef3d73b640 1 -- 172.21.15.50:0/4091086759 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3809a8a0 con 0x7fef3813af40 2023-10-30T12:12:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef3d73b640 1 --2- 172.21.15.50:0/4091086759 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef3813af40 0x7fef3813d330 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fef30009e50 tx=0x7fef3002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=462e5666ff9cfd5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 1 -- 172.21.15.50:0/4091086759 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3002c9e0 con 0x7fef3813af40 2023-10-30T12:12:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: dump: 2023-10-30T12:12:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.347 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: _finish_auth 0 2023-10-30T12:12:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: _check_auth_tickets 2023-10-30T12:12:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 1 -- 172.21.15.50:0/4091086759 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3002cb40 con 0x7fef3813af40 2023-10-30T12:12:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef3f19e640 5 monclient: authenticate success, global_id 114710 2023-10-30T12:12:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef3f19e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef3f19e640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 1 -- 172.21.15.50:0/4091086759 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef300343f0 con 0x7fef3813af40 2023-10-30T12:12:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef3f19e640 4 set_mon_vals no callback set 2023-10-30T12:12:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.325+0000 7fef377fe640 10 monclient: dump: 2023-10-30T12:12:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.352 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 10 monclient: shutdown 2023-10-30T12:12:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 -- 172.21.15.50:0/4091086759 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef3813af40 msgr2=0x7fef3813d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 --2- 172.21.15.50:0/4091086759 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef3813af40 0x7fef3813d330 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fef30009e50 tx=0x7fef3002be40 comp rx=0 tx=0).stop 2023-10-30T12:12:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 -- 172.21.15.50:0/4091086759 shutdown_connections 2023-10-30T12:12:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 --2- 172.21.15.50:0/4091086759 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef3813af40 0x7fef3813d330 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 --2- 172.21.15.50:0/4091086759 >> v2:172.21.15.118:3300/0 conn(0x7fef380a2450 0x7fef3813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 -- 172.21.15.50:0/4091086759 >> 172.21.15.50:0/4091086759 conn(0x7fef380a80b0 msgr2=0x7fef380a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 -- 172.21.15.50:0/4091086759 shutdown_connections 2023-10-30T12:12:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 1 -- 172.21.15.50:0/4091086759 wait complete. 2023-10-30T12:12:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 10 monclient: build_initial_monmap 2023-10-30T12:12:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.326+0000 7fef3f19e640 10 monclient: monmap: 2023-10-30T12:12:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:29.356 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 librados: starting msgr at 2023-10-30T12:12:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 librados: starting objecter 2023-10-30T12:12:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 Processor -- start 2023-10-30T12:12:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 -- start start 2023-10-30T12:12:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 librados: setting wanted keys 2023-10-30T12:12:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 librados: calling monclient init 2023-10-30T12:12:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient: init 2023-10-30T12:12:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef380a2450 0x7fef38131ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): picked mon.noname-a con 0x7fef380a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef380a2ae0 0x7fef38136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): picked mon.noname-c con 0x7fef380a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3813af40 0x7fef381374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): picked mon.noname-b con 0x7fef3813af40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3811e3e0 con 0x7fef380a2450 2023-10-30T12:12:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3813a7e0 con 0x7fef3813af40 2023-10-30T12:12:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef38139830 con 0x7fef380a2ae0 2023-10-30T12:12:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3cf3a640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef380a2450 msgr2=0x7fef38131ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3f19e640 10 monclient(hunting): authenticate will time out at 1594.510829s 2023-10-30T12:12:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef3cf3a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef380a2450 0x7fef38131ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef37fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef380a2ae0 0x7fef38136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef37fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef380a2ae0 0x7fef38136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43058/0 (socket says 172.21.15.50:43058) 2023-10-30T12:12:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef37fff640 1 -- 172.21.15.50:0/3376583245 learned_addr learned my addr 172.21.15.50:0/3376583245 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef37fff640 10 monclient(hunting): get_auth_request con 0x7fef380a2ae0 auth_method 0 2023-10-30T12:12:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.327+0000 7fef37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3813af40 0x7fef381374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 10 monclient(hunting): get_auth_request con 0x7fef3813af40 auth_method 0 2023-10-30T12:12:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3d73b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 10 monclient(hunting): handle_auth_done global_id 114713 payload 824 2023-10-30T12:12:29.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3813af40 msgr2=0x7fef381374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3813af40 0x7fef381374c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef380a2450 msgr2=0x7fef38131ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:29.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef380a2450 0x7fef38131ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 10 monclient: _finish_hunting 0 2023-10-30T12:12:29.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 1 monclient: found mon.noname-c 2023-10-30T12:12:29.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:29.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 1 -- 172.21.15.50:0/3376583245 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef30009b30 con 0x7fef380a2ae0 2023-10-30T12:12:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef37fff640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef380a2ae0 0x7fef38136f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00ea10 tx=0x7fef2c00eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 client.?.objecter ms_handle_connect 0x7fef380a2ae0 2023-10-30T12:12:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c009670 con 0x7fef380a2ae0 2023-10-30T12:12:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: dump: 2023-10-30T12:12:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.370 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: _finish_auth 0 2023-10-30T12:12:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: _check_auth_tickets 2023-10-30T12:12:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 5 monclient: authenticate success, global_id 114713 2023-10-30T12:12:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c00ca00 con 0x7fef380a2ae0 2023-10-30T12:12:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 10 monclient: _renew_subs 2023-10-30T12:12:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef38099bc0 con 0x7fef380a2ae0 2023-10-30T12:12:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00cdb0 con 0x7fef380a2ae0 2023-10-30T12:12:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef357fa640 10 monclient: dump: 2023-10-30T12:12:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.374 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 10 client.114713.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:29.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 10 monclient: _renew_subs 2023-10-30T12:12:29.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef36ffd640 4 set_mon_vals no callback set 2023-10-30T12:12:29.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef38139540 con 0x7fef380a2ae0 2023-10-30T12:12:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.328+0000 7fef3f19e640 1 librados: init done 2023-10-30T12:12:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3f19e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3f19e640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38137a00 0x7fef38137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3f19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef38073ff0 con 0x7fef38137a00 2023-10-30T12:12:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fef2c013470 con 0x7fef380a2ae0 2023-10-30T12:12:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38137a00 0x7fef38137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient: get_auth_request con 0x7fef38137a00 auth_method 0 2023-10-30T12:12:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef357fa640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef240392a0 0x7fef2403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fef2c04aec0 con 0x7fef380a2ae0 2023-10-30T12:12:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef357fa640 10 client.114713.objecter ms_dispatch 0x7fef38000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef357fa640 3 client.114713.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef357fa640 3 client.114713.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_done global_id 114716 payload 824 2023-10-30T12:12:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3cf3a640 1 -- 172.21.15.50:0/3376583245 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef3809a5c0 con 0x7fef38137a00 2023-10-30T12:12:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef357fa640 20 client.114713.objecter dump_active .. 0 homeless 2023-10-30T12:12:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3d73b640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef240392a0 0x7fef2403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.329+0000 7fef3d73b640 10 monclient: get_auth_request con 0x7fef240392a0 auth_method 0 2023-10-30T12:12:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 client.114713.objecter ms_handle_connect 0x7fef38137a00 2023-10-30T12:12:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 client.114713.objecter resend_mon_ops 2023-10-30T12:12:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef3cf3a640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38137a00 0x7fef38137de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef28001030 tx=0x7fef2800c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef280074e0 con 0x7fef38137a00 2023-10-30T12:12:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 monclient: dump: 2023-10-30T12:12:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.384 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 client.114713.objecter ms_handle_connect 0x7fef240392a0 2023-10-30T12:12:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef3d73b640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef240392a0 0x7fef2403b760 secure :-1 s=READY pgs=4138 cs=0 l=1 rev1=1 crypto rx=0x7fef30035000 tx=0x7fef30030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fef28007640 con 0x7fef38137a00 2023-10-30T12:12:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38137a00 msgr2=0x7fef38137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38137a00 0x7fef38137de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef28001030 tx=0x7fef2800c810 comp rx=0 tx=0).stop 2023-10-30T12:12:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38137a00 msgr2=0x7fef38137de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:29.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.330+0000 7fef357fa640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38137a00 0x7fef38137de0 secure :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef28001030 tx=0x7fef2800c810 comp rx=0 tx=0).stop 2023-10-30T12:12:29.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3f19e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:29.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3f19e640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38133290 0x7fef381336a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3f19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef38073a20 con 0x7fef38133290 2023-10-30T12:12:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38133290 0x7fef381336a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 10 monclient: get_auth_request con 0x7fef38133290 auth_method 0 2023-10-30T12:12:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.336+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3cf3a640 10 monclient(hunting): handle_auth_done global_id 114719 payload 824 2023-10-30T12:12:29.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3cf3a640 1 -- 172.21.15.50:0/3376583245 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fef380a28a0 con 0x7fef38133290 2023-10-30T12:12:29.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 10 client.114713.objecter ms_handle_connect 0x7fef38133290 2023-10-30T12:12:29.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3cf3a640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38133290 0x7fef381336a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fef28000fb0 tx=0x7fef28001150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 10 client.114713.objecter resend_mon_ops 2023-10-30T12:12:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28012ce0 con 0x7fef38133290 2023-10-30T12:12:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 10 monclient: dump: 2023-10-30T12:12:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.394 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.394 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.394 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.394 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.394 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.395 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.395 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.395 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.395 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4621 (secure 0 0 0) 0x7fef28012e40 con 0x7fef38133290 2023-10-30T12:12:29.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:29.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:29.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38133290 msgr2=0x7fef381336a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38133290 0x7fef381336a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fef28000fb0 tx=0x7fef28001150 comp rx=0 tx=0).stop 2023-10-30T12:12:29.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38133290 msgr2=0x7fef381336a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:29.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef357fa640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef38133290 0x7fef381336a0 secure :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fef28000fb0 tx=0x7fef28001150 comp rx=0 tx=0).stop 2023-10-30T12:12:29.397 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:12:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 116, 2023-10-30T12:12:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:12:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.405 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:29.405 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:29.405 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:29.405 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:29.405 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:29.406 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:29.406 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:29.406 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:29.406 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:29.406 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:29.407 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:29.407 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:29.407 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:29.407 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:29.408 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:29.408 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:29.408 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:29.408 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:29.408 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:29.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:29.409 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:29.409 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:29.409 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:29.409 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:29.409 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:29.410 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:29.410 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:29.410 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.410 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:29.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:29.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.411 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:29.412 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:29.412 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:29.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:29.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:29.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:29.413 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:29.414 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:29.414 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.414 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.415 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:29.415 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:29.415 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:29.415 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:29.415 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:29.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.416 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:29.416 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:29.416 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:29.416 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:29.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.417 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:29.417 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:29.417 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:29.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:29.418 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.419 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.419 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:29.419 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:29.419 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:29.420 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:29.420 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:29.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.420 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:29.420 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:29.421 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:29.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:29.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:29.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:29.422 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:29.422 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:29.423 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.423 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.423 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:29.424 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:29.424 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:29.424 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:29.424 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:29.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.424 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.425 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:29.425 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:29.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.425 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:29.426 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:29.426 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:29.426 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.426 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:12:29.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.427 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:29.427 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:29.427 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:12:29.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.428 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:29.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.428 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:29.428 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:29.429 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:12:29.429 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.429 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.429 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:29.430 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:29.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 10 librados: watch_flush enter 2023-10-30T12:12:29.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 10 librados: watch_flush exit 2023-10-30T12:12:29.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 20 client.114713.objecter shutdown clearing up homeless session... 2023-10-30T12:12:29.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 10 client.114713.objecter successfully canceled tick 2023-10-30T12:12:29.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef240392a0 msgr2=0x7fef2403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef240392a0 0x7fef2403b760 secure :-1 s=READY pgs=4138 cs=0 l=1 rev1=1 crypto rx=0x7fef30035000 tx=0x7fef30030000 comp rx=0 tx=0).stop 2023-10-30T12:12:29.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 10 monclient: shutdown 2023-10-30T12:12:29.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:29.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef380a2ae0 msgr2=0x7fef38136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3f19e640 1 --2- 172.21.15.50:0/3376583245 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef380a2ae0 0x7fef38136f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00ea10 tx=0x7fef2c00eed0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3d73b640 1 -- 172.21.15.50:0/3376583245 reap_dead start 2023-10-30T12:12:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.337+0000 7fef3d73b640 1 -- 172.21.15.50:0/3376583245 reap_dead start 2023-10-30T12:12:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.338+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 shutdown_connections 2023-10-30T12:12:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.338+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 >> 172.21.15.50:0/3376583245 conn(0x7fef380a80b0 msgr2=0x7fef380a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:29.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.338+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 shutdown_connections 2023-10-30T12:12:29.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.338+0000 7fef3f19e640 1 -- 172.21.15.50:0/3376583245 wait complete. 2023-10-30T12:12:29.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.338+0000 7fef3f19e640 1 librados: shutdown 2023-10-30T12:12:29.435 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:12:29.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:12:29.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:12:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.541+0000 7ffa1fc10640 10 monclient: get_monmap_and_config 2023-10-30T12:12:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.541+0000 7ffa1fc10640 10 monclient: build_initial_monmap 2023-10-30T12:12:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.541+0000 7ffa1fc10640 10 monclient: monmap: 2023-10-30T12:12:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:29.542896+0000 2023-10-30T12:12:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:29.542896+0000 2023-10-30T12:12:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:29.546 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:29.546 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.542+0000 7ffa1fc10640 1 Processor -- start 2023-10-30T12:12:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.542+0000 7ffa1fc10640 1 -- start start 2023-10-30T12:12:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.542+0000 7ffa1fc10640 10 monclient: init 2023-10-30T12:12:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-10-30T12:12:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa18135ba0 0x7ffa18135f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient(hunting): picked mon.noname-a con 0x7ffa18135ba0 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffa18135670 0x7ffa180703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient(hunting): picked mon.noname-c con 0x7ffa18135670 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa180708f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1809a870 con 0x7ffa180708f0 2023-10-30T12:12:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1809a040 con 0x7ffa18135ba0 2023-10-30T12:12:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1809a2b0 con 0x7ffa18135670 2023-10-30T12:12:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1fc10640 10 monclient(hunting): authenticate will time out at 1594.726119s 2023-10-30T12:12:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d9ac640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7ffa18135ba0 msgr2=0x7ffa18135f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d9ac640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa18135ba0 0x7ffa18135f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffa18135670 0x7ffa180703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1e1ad640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7ffa180708f0 0x7ffa1813c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:52780/0 (socket says 172.21.15.50:52780) 2023-10-30T12:12:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1e1ad640 1 -- 172.21.15.50:0/1563725250 learned_addr learned my addr 172.21.15.50:0/1563725250 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:12:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): get_auth_request con 0x7ffa18135670 auth_method 0 2023-10-30T12:12:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_done global_id 154276 payload 274 2023-10-30T12:12:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 1 -- 172.21.15.50:0/1563725250 >> v2:172.21.15.50:3300/0 conn(0x7ffa18135ba0 msgr2=0x7ffa18135f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 1 --2- 172.21.15.50:0/1563725250 >> v2:172.21.15.50:3300/0 conn(0x7ffa18135ba0 0x7ffa18135f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 1 -- 172.21.15.50:0/1563725250 >> v1:172.21.15.118:6789/0 conn(0x7ffa180708f0 legacy=0x7ffa1813c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient: _finish_hunting 0 2023-10-30T12:12:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 1 monclient: found mon.noname-c 2023-10-30T12:12:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:12:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.543+0000 7ffa1d1ab640 1 -- 172.21.15.50:0/1563725250 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1809a590 con 0x7ffa18135670 2023-10-30T12:12:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1d1ab640 1 --2- 172.21.15.50:0/1563725250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135670 0x7ffa180703b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffa180a39c0 tx=0x7ffa0802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=668d7c5a0b89b174 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 1 -- 172.21.15.50:0/1563725250 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa080317c0 con 0x7ffa18135670 2023-10-30T12:12:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: dump: 2023-10-30T12:12:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.564 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: _finish_auth 0 2023-10-30T12:12:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: _check_auth_tickets 2023-10-30T12:12:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 1 -- 172.21.15.50:0/1563725250 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa08031920 con 0x7ffa18135670 2023-10-30T12:12:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1fc10640 5 monclient: authenticate success, global_id 154276 2023-10-30T12:12:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1fc10640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 1 -- 172.21.15.50:0/1563725250 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa08031cb0 con 0x7ffa18135670 2023-10-30T12:12:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1fc10640 4 set_mon_vals no callback set 2023-10-30T12:12:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.544+0000 7ffa1c9aa640 10 monclient: dump: 2023-10-30T12:12:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.568 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 10 monclient: shutdown 2023-10-30T12:12:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 -- 172.21.15.50:0/1563725250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135670 msgr2=0x7ffa180703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 --2- 172.21.15.50:0/1563725250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135670 0x7ffa180703b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffa180a39c0 tx=0x7ffa0802b860 comp rx=0 tx=0).stop 2023-10-30T12:12:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 -- 172.21.15.50:0/1563725250 shutdown_connections 2023-10-30T12:12:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 --2- 172.21.15.50:0/1563725250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135670 0x7ffa180703b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 --2- 172.21.15.50:0/1563725250 >> v2:172.21.15.50:3300/0 conn(0x7ffa18135ba0 0x7ffa18135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 -- 172.21.15.50:0/1563725250 >> 172.21.15.50:0/1563725250 conn(0x7ffa180a4500 msgr2=0x7ffa180a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 -- 172.21.15.50:0/1563725250 shutdown_connections 2023-10-30T12:12:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 -- 172.21.15.50:0/1563725250 wait complete. 2023-10-30T12:12:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 10 monclient: build_initial_monmap 2023-10-30T12:12:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 10 monclient: monmap: 2023-10-30T12:12:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:29.572 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 librados: starting msgr at 2023-10-30T12:12:29.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.545+0000 7ffa1fc10640 1 librados: starting objecter 2023-10-30T12:12:29.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 Processor -- start 2023-10-30T12:12:29.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 -- start start 2023-10-30T12:12:29.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 librados: setting wanted keys 2023-10-30T12:12:29.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 librados: calling monclient init 2023-10-30T12:12:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient: init 2023-10-30T12:12:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:12:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa180708f0 0x7ffa181cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): picked mon.noname-a con 0x7ffa180708f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa18135670 0x7ffa1811e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): picked mon.noname-c con 0x7ffa18135670 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135ba0 0x7ffa1811edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): picked mon.noname-b con 0x7ffa18135ba0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18122050 con 0x7ffa180708f0 2023-10-30T12:12:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18121160 con 0x7ffa18135ba0 2023-10-30T12:12:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181326a0 con 0x7ffa18135670 2023-10-30T12:12:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1fc10640 10 monclient(hunting): authenticate will time out at 1594.729589s 2023-10-30T12:12:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d9ac640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa180708f0 msgr2=0x7ffa181cac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:12:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d9ac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa180708f0 0x7ffa181cac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:29.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa18135670 0x7ffa1811e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa18135670 0x7ffa1811e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43082/0 (socket says 172.21.15.50:43082) 2023-10-30T12:12:29.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 1 -- 172.21.15.50:0/3950399765 learned_addr learned my addr 172.21.15.50:0/3950399765 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:29.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 10 monclient(hunting): get_auth_request con 0x7ffa18135670 auth_method 0 2023-10-30T12:12:29.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135ba0 0x7ffa1811edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 10 monclient(hunting): get_auth_request con 0x7ffa18135ba0 auth_method 0 2023-10-30T12:12:29.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.546+0000 7ffa1e1ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 10 monclient(hunting): handle_auth_done global_id 114722 payload 824 2023-10-30T12:12:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135ba0 msgr2=0x7ffa1811edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa18135ba0 0x7ffa1811edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa180708f0 msgr2=0x7ffa181cac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:29.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa180708f0 0x7ffa181cac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 10 monclient: _finish_hunting 0 2023-10-30T12:12:29.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 1 monclient: found mon.noname-c 2023-10-30T12:12:29.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:29.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 1 -- 172.21.15.50:0/3950399765 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa08009400 con 0x7ffa18135670 2023-10-30T12:12:29.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1d1ab640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa18135670 0x7ffa1811e880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009890 tx=0x7ffa080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 client.?.objecter ms_handle_connect 0x7ffa18135670 2023-10-30T12:12:29.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:29.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa08032920 con 0x7ffa18135670 2023-10-30T12:12:29.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:29.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: dump: 2023-10-30T12:12:29.584 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.585 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.585 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.585 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.585 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.586 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.586 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.586 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.586 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:12:29.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:12:29.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:29.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa08032ac0 con 0x7ffa18135670 2023-10-30T12:12:29.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 5 monclient: authenticate success, global_id 114722 2023-10-30T12:12:29.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 10 monclient: _renew_subs 2023-10-30T12:12:29.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1809d690 con 0x7ffa18135670 2023-10-30T12:12:29.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:29.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa08032e70 con 0x7ffa18135670 2023-10-30T12:12:29.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:29.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0e7fc640 10 monclient: dump: 2023-10-30T12:12:29.589 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.589 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.589 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.589 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.593 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa0ffff640 4 set_mon_vals no callback set 2023-10-30T12:12:29.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 10 client.114722.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 10 monclient: _renew_subs 2023-10-30T12:12:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa18120520 con 0x7ffa18135670 2023-10-30T12:12:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.547+0000 7ffa1fc10640 1 librados: init done 2023-10-30T12:12:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1fc10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ffa080453f0 con 0x7ffa18135670 2023-10-30T12:12:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1fc10640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa1811fac0 0x7ffa1802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1fc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18066730 con 0x7ffa1811fac0 2023-10-30T12:12:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa0e7fc640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9fc0392a0 0x7ff9fc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(367..367 src has 1..367) v4 ==== 6299+0+0 (secure 0 0 0) 0x7ffa0806d040 con 0x7ffa18135670 2023-10-30T12:12:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa0e7fc640 10 client.114722.objecter ms_dispatch 0x7ffa18000b90 osd_map(367..367 src has 1..367) v4 2023-10-30T12:12:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa0e7fc640 3 client.114722.objecter handle_osd_map got epochs [367,367] > 0 2023-10-30T12:12:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa0e7fc640 3 client.114722.objecter handle_osd_map decoding full epoch 367 2023-10-30T12:12:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa1811fac0 0x7ffa1802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient: get_auth_request con 0x7ffa1811fac0 auth_method 0 2023-10-30T12:12:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa0e7fc640 20 client.114722.objecter dump_active .. 0 homeless 2023-10-30T12:12:29.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1e1ad640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9fc0392a0 0x7ff9fc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1e1ad640 10 monclient: get_auth_request con 0x7ff9fc0392a0 auth_method 0 2023-10-30T12:12:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_done global_id 154285 payload 824 2023-10-30T12:12:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.548+0000 7ffa1d9ac640 1 -- 172.21.15.50:0/3950399765 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa181364c0 con 0x7ffa1811fac0 2023-10-30T12:12:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 client.114722.objecter ms_handle_connect 0x7ffa1811fac0 2023-10-30T12:12:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 client.114722.objecter resend_mon_ops 2023-10-30T12:12:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa1d9ac640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa1811fac0 0x7ffa1802b430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa14003980 tx=0x7ffa1400b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 client.114722.objecter ms_handle_connect 0x7ff9fc0392a0 2023-10-30T12:12:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa1e1ad640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9fc0392a0 0x7ff9fc03b760 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 crypto rx=0x7ffa10005c00 tx=0x7ffa10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14003e10 con 0x7ffa1811fac0 2023-10-30T12:12:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 monclient: dump: 2023-10-30T12:12:29.601 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.601 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.601 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.601 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.601 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.602 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ffa14005070 con 0x7ffa1811fac0 2023-10-30T12:12:29.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:29.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:29.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa1811fac0 msgr2=0x7ffa1802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa1811fac0 0x7ffa1802b430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa14003980 tx=0x7ffa1400b8c0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa1811fac0 msgr2=0x7ffa1802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.549+0000 7ffa0e7fc640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa1811fac0 0x7ffa1802b430 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa14003980 tx=0x7ffa1400b8c0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.555+0000 7ffa1fc10640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:12:29.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.555+0000 7ffa1fc10640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa181cb160 0x7ffa1802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:29.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.555+0000 7ffa1fc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:29.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.555+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1804e160 con 0x7ffa181cb160 2023-10-30T12:12:29.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.555+0000 7ffa1d9ac640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa181cb160 0x7ffa1802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:29.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient: get_auth_request con 0x7ffa181cb160 auth_method 0 2023-10-30T12:12:29.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:29.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:29.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:29.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:29.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:29.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:29.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 10 monclient(hunting): handle_auth_done global_id 154288 payload 824 2023-10-30T12:12:29.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 1 -- 172.21.15.50:0/3950399765 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa18136210 con 0x7ffa181cb160 2023-10-30T12:12:29.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa1d9ac640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa181cb160 0x7ffa1802b970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffa14012040 tx=0x7ffa14005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:29.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 10 client.114722.objecter ms_handle_connect 0x7ffa181cb160 2023-10-30T12:12:29.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 10 client.114722.objecter resend_mon_ops 2023-10-30T12:12:29.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14020070 con 0x7ffa181cb160 2023-10-30T12:12:29.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:29.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:29.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 10 monclient: dump: 2023-10-30T12:12:29.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:29.610 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:29.610 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.610 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:29.610 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:29.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:29.611 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:29.611 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:29.611 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:29.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:29.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4623 (secure 0 0 0) 0x7ffa14010400 con 0x7ffa181cb160 2023-10-30T12:12:29.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:12:29.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:29.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa181cb160 msgr2=0x7ffa1802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa181cb160 0x7ffa1802b970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffa14012040 tx=0x7ffa14005a20 comp rx=0 tx=0).stop 2023-10-30T12:12:29.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa181cb160 msgr2=0x7ffa1802b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:12:29.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.556+0000 7ffa0e7fc640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa181cb160 0x7ffa1802b970 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffa14012040 tx=0x7ffa14005a20 comp rx=0 tx=0).stop 2023-10-30T12:12:29.613 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:12:29.613 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:29.613 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:29.613 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:12:29.614 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 116, 2023-10-30T12:12:29.614 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:12:29.614 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:12:29.614 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:12:29.615 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.615 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:12:29.615 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:29.615 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:12:29.615 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:12:29.615 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:29.616 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:29.616 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:29.616 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:29.616 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:29.616 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:29.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:29.617 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:29.617 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:29.617 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:29.617 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.618 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:12:29.618 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:12:29.618 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:29.618 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:29.618 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:29.618 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:29.619 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:29.619 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:29.619 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:29.619 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:29.619 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:29.620 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:29.620 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.620 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:12:29.620 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:12:29.620 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:12:29.621 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:12:29.621 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:12:29.621 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:12:29.621 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:29.621 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:12:29.621 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:12:29.622 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:12:29.622 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:12:29.622 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:12:29.622 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:12:29.622 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:12:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:12:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:12:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:12:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:12:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:12:29.624 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:12:29.624 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:12:29.624 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:12:29.624 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:12:29.624 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:12:29.625 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:12:29.625 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:12:29.625 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:12:29.625 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.625 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:12:29.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.626 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:12:29.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.626 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:12:29.627 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:12:29.627 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:29.627 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:29.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.627 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:29.628 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:12:29.628 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.628 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:29.629 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:12:29.629 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.629 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.629 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.630 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:12:29.630 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:12:29.630 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:29.630 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:29.630 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:29.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.631 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:12:29.631 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:12:29.631 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:29.631 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:29.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.632 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:29.632 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:12:29.632 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.633 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:29.633 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:12:29.633 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.633 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.633 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.634 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:12:29.634 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:12:29.634 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:29.634 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:29.635 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:29.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.635 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:12:29.635 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:12:29.636 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:12:29.636 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:12:29.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.636 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:12:29.636 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:12:29.637 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.637 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:12:29.637 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:12:29.637 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:12:29.638 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.638 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.639 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:12:29.639 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:12:29.639 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:12:29.639 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:12:29.640 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:12:29.640 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.640 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.640 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:12:29.641 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:12:29.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.641 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:29.641 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:29.641 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:29.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.642 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.642 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:12:29.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.642 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:29.643 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:29.643 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:12:29.643 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.643 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:12:29.643 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:12:29.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:12:29.644 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:12:29.644 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:12:29.644 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:12:29.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:12:29.644 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:12:29.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:12:29.645 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:12:29.645 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:12:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 10 librados: watch_flush enter 2023-10-30T12:12:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 10 librados: watch_flush exit 2023-10-30T12:12:29.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 20 client.114722.objecter shutdown clearing up homeless session... 2023-10-30T12:12:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 10 client.114722.objecter successfully canceled tick 2023-10-30T12:12:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9fc0392a0 msgr2=0x7ff9fc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9fc0392a0 0x7ff9fc03b760 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 crypto rx=0x7ffa10005c00 tx=0x7ffa10009040 comp rx=0 tx=0).stop 2023-10-30T12:12:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 10 monclient: shutdown 2023-10-30T12:12:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa18135670 msgr2=0x7ffa1811e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 --2- 172.21.15.50:0/3950399765 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa18135670 0x7ffa1811e880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009890 tx=0x7ffa080046c0 comp rx=0 tx=0).stop 2023-10-30T12:12:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1e1ad640 1 -- 172.21.15.50:0/3950399765 reap_dead start 2023-10-30T12:12:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1e1ad640 1 -- 172.21.15.50:0/3950399765 reap_dead start 2023-10-30T12:12:29.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 shutdown_connections 2023-10-30T12:12:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 >> 172.21.15.50:0/3950399765 conn(0x7ffa180a4500 msgr2=0x7ffa1809b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 shutdown_connections 2023-10-30T12:12:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 -- 172.21.15.50:0/3950399765 wait complete. 2023-10-30T12:12:29.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:29.557+0000 7ffa1fc10640 1 librados: shutdown 2023-10-30T12:12:29.651 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:12:29.651 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:12:49.652 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:12:49.652 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:12:49.653 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:12:49.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:12:49.657 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:12:49.658 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:12:49.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:49.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.773+0000 7fcda591d640 10 monclient: get_monmap_and_config 2023-10-30T12:12:49.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.773+0000 7fcda591d640 10 monclient: build_initial_monmap 2023-10-30T12:12:49.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.773+0000 7fcda591d640 10 monclient: monmap: 2023-10-30T12:12:49.776 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:49.776 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:49.776 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:49.774862+0000 2023-10-30T12:12:49.776 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:49.774862+0000 2023-10-30T12:12:49.776 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:49.777 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:49.777 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:49.777 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:49.777 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:49.777 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:49.778 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:49.778 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 1 Processor -- start 2023-10-30T12:12:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 1 -- start start 2023-10-30T12:12:49.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient: init 2023-10-30T12:12:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-10-30T12:12:49.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcda00a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:49.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 0x7fcda013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:49.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient(hunting): picked mon.noname-a con 0x7fcda0099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:49.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcda0132db0 0x7fcda013cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:49.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient(hunting): picked mon.noname-c con 0x7fcda0132db0 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:49.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00a4f40 con 0x7fcda00a4b60 2023-10-30T12:12:49.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:49.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda00a3d70 con 0x7fcda0099a40 2023-10-30T12:12:49.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda00a4150 con 0x7fcda0132db0 2023-10-30T12:12:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.774+0000 7fcda591d640 10 monclient(hunting): authenticate will time out at 1614.957948s 2023-10-30T12:12:49.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 0x7fcda013aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:49.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 0x7fcda013aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35058/0 (socket says 172.21.15.50:35058) 2023-10-30T12:12:49.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 1 -- 172.21.15.50:0/2814024769 learned_addr learned my addr 172.21.15.50:0/2814024769 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:49.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 10 monclient(hunting): get_auth_request con 0x7fcda0099a40 auth_method 0 2023-10-30T12:12:49.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:49.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:49.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:49.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 1 --2- 172.21.15.50:0/2814024769 >> v2:172.21.15.118:3300/0 conn(0x7fcda0132db0 0x7fcda013cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:49.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): get_auth_request con 0x7fcda0132db0 auth_method 0 2023-10-30T12:12:49.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 1 -- 172.21.15.50:0/2814024769 >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 msgr2=0x7fcda013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:49.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 1 -- 172.21.15.50:0/2814024769 >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 msgr2=0x7fcda013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 1 --2- 172.21.15.50:0/2814024769 >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 0x7fcda013aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:49.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9e7fc640 1 --2- 172.21.15.50:0/2814024769 >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 0x7fcda013aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:49.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:49.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:49.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:49.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient(hunting): handle_auth_done global_id 154291 payload 274 2023-10-30T12:12:49.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 1 -- 172.21.15.50:0/2814024769 >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 msgr2=0x7fcda013aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:49.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 1 --2- 172.21.15.50:0/2814024769 >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 0x7fcda013aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:49.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 1 -- 172.21.15.50:0/2814024769 >> v1:172.21.15.118:6789/0 conn(0x7fcda00a4b60 legacy=0x7fcda01326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:49.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:12:49.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 1 monclient: found mon.noname-c 2023-10-30T12:12:49.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:49.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:12:49.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9f7fe640 1 -- 172.21.15.50:0/2814024769 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda009a440 con 0x7fcda0132db0 2023-10-30T12:12:49.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9dffb640 1 -- 172.21.15.50:0/2814024769 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219601371 0 0) 0x7fcda00a4f40 con 0x7fcda00a4b60 2023-10-30T12:12:49.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.775+0000 7fcd9dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:12:49.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9f7fe640 1 --2- 172.21.15.50:0/2814024769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda0132db0 0x7fcda013cfd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcda00a3a20 tx=0x7fcd9802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=251397d73e5d927a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:49.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 1 -- 172.21.15.50:0/2814024769 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98031710 con 0x7fcda0132db0 2023-10-30T12:12:49.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:49.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:49.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: dump: 2023-10-30T12:12:49.800 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:49.801 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:49.801 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.801 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.801 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:49.801 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:49.802 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:49.802 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:49.802 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:49.802 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:49.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: _finish_auth 0 2023-10-30T12:12:49.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:12:49.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:49.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 1 -- 172.21.15.50:0/2814024769 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9802f050 con 0x7fcda0132db0 2023-10-30T12:12:49.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:49.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 1 -- 172.21.15.50:0/2814024769 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98031a90 con 0x7fcda0132db0 2023-10-30T12:12:49.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:49.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:12:49.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcd9dffb640 10 monclient: dump: 2023-10-30T12:12:49.804 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:49.804 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:49.804 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.805 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.805 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:49.805 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:49.805 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:49.805 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:49.806 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:49.806 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:49.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcda591d640 5 monclient: authenticate success, global_id 154291 2023-10-30T12:12:49.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcda591d640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:49.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.776+0000 7fcda591d640 4 set_mon_vals no callback set 2023-10-30T12:12:49.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 10 monclient: shutdown 2023-10-30T12:12:49.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:49.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 -- 172.21.15.50:0/2814024769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda0132db0 msgr2=0x7fcda013cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:49.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 --2- 172.21.15.50:0/2814024769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda0132db0 0x7fcda013cfd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcda00a3a20 tx=0x7fcd9802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:12:49.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 -- 172.21.15.50:0/2814024769 shutdown_connections 2023-10-30T12:12:49.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 --2- 172.21.15.50:0/2814024769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda0132db0 0x7fcda013cfd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:49.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 --2- 172.21.15.50:0/2814024769 >> v2:172.21.15.50:3300/0 conn(0x7fcda0099a40 0x7fcda013aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:49.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 -- 172.21.15.50:0/2814024769 >> 172.21.15.50:0/2814024769 conn(0x7fcda00a8060 msgr2=0x7fcda00a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:49.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 -- 172.21.15.50:0/2814024769 shutdown_connections 2023-10-30T12:12:49.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 1 -- 172.21.15.50:0/2814024769 wait complete. 2023-10-30T12:12:49.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 10 monclient: build_initial_monmap 2023-10-30T12:12:49.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.777+0000 7fcda591d640 10 monclient: monmap: 2023-10-30T12:12:49.809 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:49.809 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:49.809 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:49.809 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:49.810 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:49.810 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:49.810 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:49.810 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:49.810 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:49.811 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:49.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 librados: starting msgr at 2023-10-30T12:12:49.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 librados: starting objecter 2023-10-30T12:12:49.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 Processor -- start 2023-10-30T12:12:49.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 -- start start 2023-10-30T12:12:49.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 librados: setting wanted keys 2023-10-30T12:12:49.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 librados: calling monclient init 2023-10-30T12:12:49.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient: init 2023-10-30T12:12:49.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:49.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:12:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda00a4b60 0x7fcda009af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): picked mon.noname-b con 0x7fcda00a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcda0132db0 0x7fcda009b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:49.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): picked mon.noname-c con 0x7fcda0132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 0x7fcda01ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): picked mon.noname-a con 0x7fcda009b9a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:49.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda009e270 con 0x7fcda009b9a0 2023-10-30T12:12:49.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:49.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda009e750 con 0x7fcda00a4b60 2023-10-30T12:12:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda009cfc0 con 0x7fcda0132db0 2023-10-30T12:12:49.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcda591d640 10 monclient(hunting): authenticate will time out at 1614.961836s 2023-10-30T12:12:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 0x7fcda01ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:49.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9e7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcda0132db0 0x7fcda009b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 0x7fcda01ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35060/0 (socket says 172.21.15.50:35060) 2023-10-30T12:12:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9f7fe640 1 -- 172.21.15.50:0/125582064 learned_addr learned my addr 172.21.15.50:0/125582064 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9f7fe640 10 monclient(hunting): get_auth_request con 0x7fcda009b9a0 auth_method 0 2023-10-30T12:12:49.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:49.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:49.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.778+0000 7fcd9f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:49.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): get_auth_request con 0x7fcda0132db0 auth_method 0 2023-10-30T12:12:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:49.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9f7fe640 1 -- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 msgr2=0x7fcda01ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:12:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9f7fe640 1 -- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 msgr2=0x7fcda01ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:12:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9f7fe640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 0x7fcda01ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:49.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda00a4b60 0x7fcda009af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9f7fe640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 0x7fcda01ca540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 10 monclient(hunting): get_auth_request con 0x7fcda00a4b60 auth_method 0 2023-10-30T12:12:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:49.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient(hunting): handle_auth_done global_id 114725 payload 824 2023-10-30T12:12:49.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 1 -- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda00a4b60 msgr2=0x7fcda009af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda00a4b60 0x7fcda009af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 1 -- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 msgr2=0x7fcda01ca540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 0x7fcda01ca540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:12:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 1 monclient: found mon.noname-c 2023-10-30T12:12:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 1 -- 172.21.15.50:0/125582064 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd98009400 con 0x7fcda0132db0 2023-10-30T12:12:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9e7fc640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcda0132db0 0x7fcda009b460 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcda0070610 tx=0x7fcd94007f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 client.?.objecter ms_handle_connect 0x7fcda0132db0 2023-10-30T12:12:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:49.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 1 -- 172.21.15.50:0/125582064 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400dc70 con 0x7fcda0132db0 2023-10-30T12:12:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:49.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: dump: 2023-10-30T12:12:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:49.828 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.829 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:49.830 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:49.831 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:49.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: _finish_auth 0 2023-10-30T12:12:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: _check_auth_tickets 2023-10-30T12:12:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:49.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 1 -- 172.21.15.50:0/125582064 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9400f040 con 0x7fcda0132db0 2023-10-30T12:12:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 5 monclient: authenticate success, global_id 114725 2023-10-30T12:12:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 10 monclient: _renew_subs 2023-10-30T12:12:49.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda00a18d0 con 0x7fcda0132db0 2023-10-30T12:12:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:49.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 1 -- 172.21.15.50:0/125582064 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94015400 con 0x7fcda0132db0 2023-10-30T12:12:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd87fff640 10 monclient: dump: 2023-10-30T12:12:49.836 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:49.836 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:49.837 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:49.838 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:49.839 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 10 client.114725.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 10 monclient: _renew_subs 2023-10-30T12:12:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:49.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda009e470 con 0x7fcda0132db0 2023-10-30T12:12:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcd9d7fa640 4 set_mon_vals no callback set 2023-10-30T12:12:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.779+0000 7fcda591d640 1 librados: init done 2023-10-30T12:12:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcd87fff640 1 -- 172.21.15.50:0/125582064 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcd9401e3b0 con 0x7fcda0132db0 2023-10-30T12:12:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcda591d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcda591d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcda591d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda004e160 con 0x7fcda0132db0 2023-10-30T12:12:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcd87fff640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcd8c0392f0 0x7fcd8c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcd87fff640 1 -- 172.21.15.50:0/125582064 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(387..387 src has 1..387) v4 ==== 5914+0+0 (secure 0 0 0) 0x7fcd9404ab40 con 0x7fcda0132db0 2023-10-30T12:12:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcd87fff640 10 client.114725.objecter ms_dispatch 0x7fcda0000b90 osd_map(387..387 src has 1..387) v4 2023-10-30T12:12:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcd87fff640 3 client.114725.objecter handle_osd_map got epochs [387,387] > 0 2023-10-30T12:12:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcd87fff640 3 client.114725.objecter handle_osd_map decoding full epoch 387 2023-10-30T12:12:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.780+0000 7fcd87fff640 20 client.114725.objecter dump_active .. 0 homeless 2023-10-30T12:12:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.781+0000 7fcd9effd640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcd8c0392f0 0x7fcd8c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:49.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.781+0000 7fcd9effd640 10 monclient: get_auth_request con 0x7fcd8c0392f0 auth_method 0 2023-10-30T12:12:49.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.781+0000 7fcd9effd640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcd8c0392f0 0x7fcd8c03b7b0 secure :-1 s=READY pgs=4141 cs=0 l=1 rev1=1 crypto rx=0x7fcd90000f60 tx=0x7fcd90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:49.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.781+0000 7fcd87fff640 10 client.114725.objecter ms_handle_connect 0x7fcd8c0392f0 2023-10-30T12:12:49.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.785+0000 7fcd87fff640 1 -- 172.21.15.50:0/125582064 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcd940143a0 con 0x7fcda0132db0 2023-10-30T12:12:49.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.785+0000 7fcd87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:49.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:49.785+0000 7fcd87fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:50.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.038+0000 7fcda591d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:50.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.038+0000 7fcda591d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:50.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.038+0000 7fcda591d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:50.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.038+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcda0132170 con 0x7fcda0132db0 2023-10-30T12:12:50.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.039+0000 7fcd87fff640 1 -- 172.21.15.50:0/125582064 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fcd9401a020 con 0x7fcda0132db0 2023-10-30T12:12:50.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.039+0000 7fcd87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:50.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.039+0000 7fcd87fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:50.042 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":116,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:50.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.040+0000 7fcda591d640 10 librados: watch_flush enter 2023-10-30T12:12:50.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.040+0000 7fcda591d640 10 librados: watch_flush exit 2023-10-30T12:12:50.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.040+0000 7fcda591d640 20 client.114725.objecter shutdown clearing up homeless session... 2023-10-30T12:12:50.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.040+0000 7fcda591d640 10 client.114725.objecter successfully canceled tick 2023-10-30T12:12:50.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcd8c0392f0 msgr2=0x7fcd8c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:50.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcd8c0392f0 0x7fcd8c03b7b0 secure :-1 s=READY pgs=4141 cs=0 l=1 rev1=1 crypto rx=0x7fcd90000f60 tx=0x7fcd90009040 comp rx=0 tx=0).stop 2023-10-30T12:12:50.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 10 monclient: shutdown 2023-10-30T12:12:50.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:50.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcda0132db0 msgr2=0x7fcda009b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcda0132db0 0x7fcda009b460 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcda0070610 tx=0x7fcd94007f80 comp rx=0 tx=0).stop 2023-10-30T12:12:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 shutdown_connections 2023-10-30T12:12:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcd8c0392f0 0x7fcd8c03b7b0 unknown :-1 s=CLOSED pgs=4141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcda009b9a0 0x7fcda01ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcda0132db0 0x7fcda009b460 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 --2- 172.21.15.50:0/125582064 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcda00a4b60 0x7fcda009af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 >> 172.21.15.50:0/125582064 conn(0x7fcda00a8060 msgr2=0x7fcda00a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 shutdown_connections 2023-10-30T12:12:50.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 -- 172.21.15.50:0/125582064 wait complete. 2023-10-30T12:12:50.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:50.041+0000 7fcda591d640 1 librados: shutdown 2023-10-30T12:12:53.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:53.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.170+0000 7f12aeab4640 10 monclient: get_monmap_and_config 2023-10-30T12:12:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.170+0000 7f12aeab4640 10 monclient: build_initial_monmap 2023-10-30T12:12:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.170+0000 7f12aeab4640 10 monclient: monmap: 2023-10-30T12:12:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:53.172 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:53.171400+0000 2023-10-30T12:12:53.173 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:53.171400+0000 2023-10-30T12:12:53.173 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:53.173 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:53.173 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:53.173 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:53.174 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:53.174 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:53.174 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:53.174 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 1 Processor -- start 2023-10-30T12:12:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 1 -- start start 2023-10-30T12:12:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient: init 2023-10-30T12:12:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T12:12:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12a8098520 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f12a8098900 0x7f12a8098d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8098900 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:53.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8137980 0x7f12a8139d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:53.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient(hunting): picked mon.noname-b con 0x7f12a8137980 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:53.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a809c680 con 0x7f12a8098520 2023-10-30T12:12:53.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:53.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a8099930 con 0x7f12a8098900 2023-10-30T12:12:53.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:53.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a809aca0 con 0x7f12a8137980 2023-10-30T12:12:53.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:53.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12aeab4640 10 monclient(hunting): authenticate will time out at 1618.354669s 2023-10-30T12:12:53.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8137980 0x7f12a8139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:53.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8137980 0x7f12a8139d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42158/0 (socket says 172.21.15.50:42158) 2023-10-30T12:12:53.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 1 -- 172.21.15.50:0/3951370016 learned_addr learned my addr 172.21.15.50:0/3951370016 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:53.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 10 monclient(hunting): get_auth_request con 0x7f12a8137980 auth_method 0 2023-10-30T12:12:53.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:53.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f129ffff640 1 --2- 172.21.15.50:0/3951370016 >> v2:172.21.15.118:3300/0 conn(0x7f12a8098900 0x7f12a8098d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f129ffff640 10 monclient(hunting): get_auth_request con 0x7f12a8098900 auth_method 0 2023-10-30T12:12:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f129ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f129ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:53.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f129ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:53.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:53.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:53.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.171+0000 7f12ad051640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:53.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f129ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:53.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f129ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:53.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f129ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:53.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 10 monclient(hunting): handle_auth_done global_id 114728 payload 274 2023-10-30T12:12:53.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 1 -- 172.21.15.50:0/3951370016 >> v2:172.21.15.118:3300/0 conn(0x7f12a8098900 msgr2=0x7f12a8098d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:12:53.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 1 --2- 172.21.15.50:0/3951370016 >> v2:172.21.15.118:3300/0 conn(0x7f12a8098900 0x7f12a8098d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 1 -- 172.21.15.50:0/3951370016 >> v1:172.21.15.118:6789/0 conn(0x7f12a8098520 legacy=0x7f12a809bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:53.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 10 monclient: _finish_hunting 0 2023-10-30T12:12:53.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 1 monclient: found mon.noname-b 2023-10-30T12:12:53.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:53.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:12:53.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 1 -- 172.21.15.50:0/3951370016 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a809af80 con 0x7f12a8137980 2023-10-30T12:12:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f12ad051640 1 --2- 172.21.15.50:0/3951370016 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8137980 0x7f12a8139d70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009e50 tx=0x7f12a002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2424b5b43b984a67 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f129f7fe640 1 -- 172.21.15.50:0/3951370016 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a002c9e0 con 0x7f12a8137980 2023-10-30T12:12:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.172+0000 7f129f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:12:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: dump: 2023-10-30T12:12:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:53.192 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:53.192 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.192 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.192 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:53.192 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:53.193 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:53.193 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:53.193 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:53.193 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:53.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:12:53.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:12:53.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:53.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 1 -- 172.21.15.50:0/3951370016 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a002cb40 con 0x7f12a8137980 2023-10-30T12:12:53.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 5 monclient: authenticate success, global_id 114728 2023-10-30T12:12:53.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:12:53.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 10 monclient: get_monmap_and_config success 2023-10-30T12:12:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 1 -- 172.21.15.50:0/3951370016 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a00343f0 con 0x7f12a8137980 2023-10-30T12:12:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 4 set_mon_vals no callback set 2023-10-30T12:12:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:53.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f129f7fe640 10 monclient: dump: 2023-10-30T12:12:53.196 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:53.196 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:53.196 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.196 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:53.197 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 10 monclient: shutdown 2023-10-30T12:12:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 -- 172.21.15.50:0/3951370016 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8137980 msgr2=0x7f12a8139d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 --2- 172.21.15.50:0/3951370016 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8137980 0x7f12a8139d70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009e50 tx=0x7f12a002be40 comp rx=0 tx=0).stop 2023-10-30T12:12:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 -- 172.21.15.50:0/3951370016 shutdown_connections 2023-10-30T12:12:53.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 --2- 172.21.15.50:0/3951370016 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8137980 0x7f12a8139d70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 --2- 172.21.15.50:0/3951370016 >> v2:172.21.15.118:3300/0 conn(0x7f12a8098900 0x7f12a8098d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 -- 172.21.15.50:0/3951370016 >> 172.21.15.50:0/3951370016 conn(0x7f12a80a05c0 msgr2=0x7f12a809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:53.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 -- 172.21.15.50:0/3951370016 shutdown_connections 2023-10-30T12:12:53.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.173+0000 7f12aeab4640 1 -- 172.21.15.50:0/3951370016 wait complete. 2023-10-30T12:12:53.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 10 monclient: build_initial_monmap 2023-10-30T12:12:53.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 10 monclient: monmap: 2023-10-30T12:12:53.200 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:53.200 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:12:53.200 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:12:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:12:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:12:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:12:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:12:53.202 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 1 librados: starting msgr at 2023-10-30T12:12:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 1 librados: starting objecter 2023-10-30T12:12:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 1 Processor -- start 2023-10-30T12:12:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 1 -- start start 2023-10-30T12:12:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 1 librados: setting wanted keys 2023-10-30T12:12:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 1 librados: calling monclient init 2023-10-30T12:12:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.174+0000 7f12aeab4640 10 monclient: init 2023-10-30T12:12:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:12:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8098520 0x7f12a8136a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 0x7f12a8136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): picked mon.noname-a con 0x7f12a8098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:12:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a8137980 0x7f12a80704b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:53.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): picked mon.noname-b con 0x7f12a8137980 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:12:53.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:53.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a809e020 con 0x7f12a8098900 2023-10-30T12:12:53.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:53.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8071d50 con 0x7f12a8137980 2023-10-30T12:12:53.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:53.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a808fa40 con 0x7f12a8098520 2023-10-30T12:12:53.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:53.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12aeab4640 10 monclient(hunting): authenticate will time out at 1618.358149s 2023-10-30T12:12:53.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8098520 0x7f12a8136a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:53.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 0x7f12a8136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:53.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8098520 0x7f12a8136a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42164/0 (socket says 172.21.15.50:42164) 2023-10-30T12:12:53.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 0x7f12a8136f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35074/0 (socket says 172.21.15.50:35074) 2023-10-30T12:12:53.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 -- 172.21.15.50:0/2209873012 learned_addr learned my addr 172.21.15.50:0/2209873012 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:53.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): get_auth_request con 0x7f12a8098520 auth_method 0 2023-10-30T12:12:53.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:53.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:53.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:53.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 10 monclient(hunting): get_auth_request con 0x7f12a8098900 auth_method 0 2023-10-30T12:12:53.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:53.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:53.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:53.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:53.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:53.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 1 -- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 msgr2=0x7f12a8136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:12:53.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 1 -- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 msgr2=0x7f12a8136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:12:53.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 0x7f12a8136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:53.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:53.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129ffff640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 0x7f12a8136f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:53.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a8137980 0x7f12a80704b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:53.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 10 monclient(hunting): get_auth_request con 0x7f12a8137980 auth_method 0 2023-10-30T12:12:53.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:53.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:53.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:53.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:12:53.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:12:53.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ad051640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:12:53.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient(hunting): handle_auth_done global_id 114731 payload 824 2023-10-30T12:12:53.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 -- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a8137980 msgr2=0x7f12a80704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:53.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a8137980 0x7f12a80704b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 -- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 msgr2=0x7f12a8136f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:12:53.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 0x7f12a8136f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient: _finish_hunting 0 2023-10-30T12:12:53.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 monclient: found mon.noname-c 2023-10-30T12:12:53.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:12:53.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:12:53.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 -- 172.21.15.50:0/2209873012 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0009b30 con 0x7f12a8098520 2023-10-30T12:12:53.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f12ac850640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8098520 0x7f12a8136a10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c450 tx=0x7f12a400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:53.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129d7fa640 10 client.?.objecter ms_handle_connect 0x7f12a8098520 2023-10-30T12:12:53.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:12:53.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.175+0000 7f129d7fa640 1 -- 172.21.15.50:0/2209873012 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40052d0 con 0x7f12a8098520 2023-10-30T12:12:53.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:53.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:12:53.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: dump: 2023-10-30T12:12:53.215 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:53.215 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:53.215 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.215 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.215 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:53.216 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:53.216 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:53.216 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:53.216 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:53.216 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:53.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:12:53.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:12:53.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:12:53.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 1 -- 172.21.15.50:0/2209873012 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4005450 con 0x7f12a8098520 2023-10-30T12:12:53.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 5 monclient: authenticate success, global_id 114731 2023-10-30T12:12:53.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 10 monclient: _renew_subs 2023-10-30T12:12:53.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:53.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a813bf10 con 0x7f12a8098520 2023-10-30T12:12:53.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:12:53.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 1 -- 172.21.15.50:0/2209873012 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4003880 con 0x7f12a8098520 2023-10-30T12:12:53.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:12:53.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:12:53.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 10 monclient: dump: 2023-10-30T12:12:53.219 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:12:53.219 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:53.219 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.220 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:12:53.220 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:12:53.220 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:53.220 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:12:53.220 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:12:53.221 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:12:53.221 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:53.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 10 client.114731.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:12:53.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 10 monclient: _renew_subs 2023-10-30T12:12:53.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:53.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a8090420 con 0x7f12a8098520 2023-10-30T12:12:53.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f12aeab4640 1 librados: init done 2023-10-30T12:12:53.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129effd640 4 set_mon_vals no callback set 2023-10-30T12:12:53.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.176+0000 7f129d7fa640 1 -- 172.21.15.50:0/2209873012 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f12a40039e0 con 0x7f12a8098520 2023-10-30T12:12:53.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f12aeab4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:53.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f12aeab4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:53.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f12aeab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:53.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129d7fa640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12940392f0 0x7f129403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:53.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a804e160 con 0x7f12a8098520 2023-10-30T12:12:53.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129d7fa640 1 -- 172.21.15.50:0/2209873012 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(391..391 src has 1..391) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f12a404a780 con 0x7f12a8098520 2023-10-30T12:12:53.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129d7fa640 10 client.114731.objecter ms_dispatch 0x7f12a8000b90 osd_map(391..391 src has 1..391) v4 2023-10-30T12:12:53.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129d7fa640 3 client.114731.objecter handle_osd_map got epochs [391,391] > 0 2023-10-30T12:12:53.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129d7fa640 3 client.114731.objecter handle_osd_map decoding full epoch 391 2023-10-30T12:12:53.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129d7fa640 20 client.114731.objecter dump_active .. 0 homeless 2023-10-30T12:12:53.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129ffff640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12940392f0 0x7f129403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:53.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129ffff640 10 monclient: get_auth_request con 0x7f12940392f0 auth_method 0 2023-10-30T12:12:53.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129d7fa640 10 client.114731.objecter ms_handle_connect 0x7f12940392f0 2023-10-30T12:12:53.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.177+0000 7f129ffff640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12940392f0 0x7f129403b7b0 secure :-1 s=READY pgs=4142 cs=0 l=1 rev1=1 crypto rx=0x7f1298009500 tx=0x7f1298008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:12:53.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.182+0000 7f129d7fa640 1 -- 172.21.15.50:0/2209873012 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12a40102e0 con 0x7f12a8098520 2023-10-30T12:12:53.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.182+0000 7f129d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:12:53.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.182+0000 7f129d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:12:53.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.425+0000 7f12aeab4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:12:53.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.425+0000 7f12aeab4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:53.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.425+0000 7f12aeab4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:12:53.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.425+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f12a808f3e0 con 0x7f12a8098520 2023-10-30T12:12:53.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.426+0000 7f129d7fa640 1 -- 172.21.15.50:0/2209873012 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f12a4019af0 con 0x7f12a8098520 2023-10-30T12:12:53.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.426+0000 7f129d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:12:53.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.426+0000 7f129d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:12:53.429 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":116,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:12:53.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 10 librados: watch_flush enter 2023-10-30T12:12:53.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 10 librados: watch_flush exit 2023-10-30T12:12:53.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 20 client.114731.objecter shutdown clearing up homeless session... 2023-10-30T12:12:53.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 10 client.114731.objecter successfully canceled tick 2023-10-30T12:12:53.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12940392f0 msgr2=0x7f129403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:53.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12940392f0 0x7f129403b7b0 secure :-1 s=READY pgs=4142 cs=0 l=1 rev1=1 crypto rx=0x7f1298009500 tx=0x7f1298008040 comp rx=0 tx=0).stop 2023-10-30T12:12:53.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 10 monclient: shutdown 2023-10-30T12:12:53.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:12:53.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8098520 msgr2=0x7f12a8136a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:53.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8098520 0x7f12a8136a10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c450 tx=0x7f12a400c910 comp rx=0 tx=0).stop 2023-10-30T12:12:53.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 shutdown_connections 2023-10-30T12:12:53.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12940392f0 0x7f129403b7b0 unknown :-1 s=CLOSED pgs=4142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12a8137980 0x7f12a80704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a8098900 0x7f12a8136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 --2- 172.21.15.50:0/2209873012 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12a8098520 0x7f12a8136a10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:53.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 >> 172.21.15.50:0/2209873012 conn(0x7f12a80a05c0 msgr2=0x7f12a809c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:12:53.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 shutdown_connections 2023-10-30T12:12:53.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 -- 172.21.15.50:0/2209873012 wait complete. 2023-10-30T12:12:53.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:53.428+0000 7f12aeab4640 1 librados: shutdown 2023-10-30T12:12:56.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:12:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.556+0000 7efde7bd0640 10 monclient: get_monmap_and_config 2023-10-30T12:12:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.556+0000 7efde7bd0640 10 monclient: build_initial_monmap 2023-10-30T12:12:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.556+0000 7efde7bd0640 10 monclient: monmap: 2023-10-30T12:12:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:12:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:12:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:12:56.557710+0000 2023-10-30T12:12:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:12:56.557710+0000 2023-10-30T12:12:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:12:56.559 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:12:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:12:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:12:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:12:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:12:56.560 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:12:56.561 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:12:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 1 Processor -- start 2023-10-30T12:12:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 1 -- start start 2023-10-30T12:12:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient: init 2023-10-30T12:12:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:12:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-10-30T12:12:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient(hunting): picked mon.noname-b con 0x7efde00707d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:12:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efde00a0170 addr v1:172.21.15.118:6789/0 2023-10-30T12:12:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient(hunting): picked mon.noname-a con 0x7efde00710f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:12:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde00a11f0 con 0x7efde00a0170 2023-10-30T12:12:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde009f4b0 con 0x7efde00710f0 2023-10-30T12:12:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde009d9d0 con 0x7efde00707d0 2023-10-30T12:12:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.557+0000 7efde7bd0640 10 monclient(hunting): authenticate will time out at 1621.740969s 2023-10-30T12:12:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35086/0 (socket says 172.21.15.50:35086) 2023-10-30T12:12:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efde00707d0 auth_method 0 2023-10-30T12:12:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 learned_addr learned my addr 172.21.15.50:0/3303491980 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:12:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 10 monclient(hunting): get_auth_request con 0x7efde00710f0 auth_method 0 2023-10-30T12:12:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.558+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:56.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:56.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:56.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efde00707d0 auth_method 0 2023-10-30T12:12:56.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:56.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 10 monclient(hunting): get_auth_request con 0x7efde00710f0 auth_method 0 2023-10-30T12:12:56.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:56.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:56.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:56.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:56.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:56.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:56.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:56.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:56.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:56.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:56.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:56.759+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:12:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efde00707d0 auth_method 0 2023-10-30T12:12:57.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:57.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:57.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:57.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 10 monclient(hunting): get_auth_request con 0x7efde00710f0 auth_method 0 2023-10-30T12:12:57.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:57.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:57.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:57.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:57.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:57.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:57.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:57.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:57.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:57.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:57.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.160+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:12:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efde00707d0 auth_method 0 2023-10-30T12:12:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 10 monclient(hunting): get_auth_request con 0x7efde00710f0 auth_method 0 2023-10-30T12:12:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:12:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:57.961+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:12:59.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient(hunting): tick 2023-10-30T12:12:59.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:12:59.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:12:59.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:12:59.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 msgr2=0x7efde0070bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:59.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efde00707d0 0x7efde0070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:59.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 msgr2=0x7efde0134d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:12:59.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efde00710f0 0x7efde0134d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:12:59.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 >> v1:172.21.15.118:6789/0 conn(0x7efde00a0170 legacy=0x7efde01345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:12:59.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient: _add_conns ranks=[4,0] 2023-10-30T12:12:59.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:12:59.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient(hunting): picked mon.noname-c con 0x7efdc40036e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:12:59.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdc4006210 addr v1:172.21.15.50:6789/0 2023-10-30T12:12:59.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde00a13f0 con 0x7efdc4006210 2023-10-30T12:12:59.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:12:59.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde009d9d0 con 0x7efdc40036e0 2023-10-30T12:12:59.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efdd37fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:12:59.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:59.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efdc40036e0 auth_method 0 2023-10-30T12:12:59.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:59.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:59.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:12:59.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:12:59.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:59.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:59.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.558+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:12:59.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.758+0000 7efde516b640 1 -- 172.21.15.50:0/3303491980 >> v1:172.21.15.50:6789/0 conn(0x7efdc4006210 legacy=0x7efdc40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-10-30T12:12:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.758+0000 7efde516b640 1 -- 172.21.15.50:0/3303491980 >> v1:172.21.15.50:6789/0 conn(0x7efdc4006210 legacy=0x7efdc40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:12:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.758+0000 7efde516b640 1 --1- 172.21.15.50:0/3303491980 >> v1:172.21.15.50:6789/0 conn(0x7efdc4006210 0x7efdc40086d0 :-1 s=OPENED pgs=24 cs=1 l=1).handle_message read tag failed 2023-10-30T12:12:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.758+0000 7efde516b640 1 --1- 172.21.15.50:0/3303491980 >> v1:172.21.15.50:6789/0 conn(0x7efdc4006210 0x7efdc40086d0 :-1 s=OPENED pgs=24 cs=1 l=1).fault on lossy channel, failing 2023-10-30T12:12:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.758+0000 7efde496a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.50:6789/0 2023-10-30T12:12:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:12:59.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efdc40036e0 auth_method 0 2023-10-30T12:12:59.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:12:59.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:12:59.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:12:59.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:12:59.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:12:59.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:12:59.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:12:59.759+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:13:00.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:00.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efdc40036e0 auth_method 0 2023-10-30T12:13:00.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:00.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:00.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:00.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:13:00.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:00.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.160+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:13:00.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:00.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efdc40036e0 auth_method 0 2023-10-30T12:13:00.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:13:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:00.962+0000 7efde596c640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:13:02.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): tick 2023-10-30T12:13:02.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:13:02.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:13:02.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:13:02.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:02.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc40036e0 0x7efdc4005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 >> v1:172.21.15.50:6789/0 conn(0x7efdc4006210 legacy=0x7efdc40086d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:02.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 reap_dead start 2023-10-30T12:13:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:13:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdc4009fc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc400ab50 0x7efdc4005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): picked mon.noname-c con 0x7efdc400ab50 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efdc4008860 0x7efdc400d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): picked mon.noname-a con 0x7efdc4008860 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc400b020 con 0x7efdc4009fc0 2023-10-30T12:13:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde009d9d0 con 0x7efdc4008860 2023-10-30T12:13:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 1 -- 172.21.15.50:0/3303491980 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc400e920 con 0x7efdc400ab50 2023-10-30T12:13:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efdd37fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.50:3300/0 conn(0x7efdc4008860 0x7efdc400d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): get_auth_request con 0x7efdc4008860 auth_method 0 2023-10-30T12:13:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient(hunting): handle_auth_done global_id 174099 payload 274 2023-10-30T12:13:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde516b640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc400ab50 0x7efdc4005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc400ab50 msgr2=0x7efdc4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc400ab50 0x7efdc4005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 >> v1:172.21.15.50:6789/0 conn(0x7efdc4009fc0 legacy=0x7efdc400a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient: _finish_hunting 0 2023-10-30T12:13:02.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 monclient: found mon.noname-a 2023-10-30T12:13:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 1 -- 172.21.15.50:0/3303491980 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381694509 0 0) 0x7efdc400b020 con 0x7efdc4009fc0 2023-10-30T12:13:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 -- 172.21.15.50:0/3303491980 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc400eae0 con 0x7efdc4008860 2023-10-30T12:13:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde516b640 10 monclient: get_auth_request con 0x7efdc400ab50 auth_method 0 2023-10-30T12:13:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde516b640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc400ab50 0x7efdc4005b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:13:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde616d640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4008860 0x7efdc400d300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efde009b080 tx=0x7efdd802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d8c646d621628c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 1 -- 172.21.15.50:0/3303491980 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8032540 con 0x7efdc4008860 2023-10-30T12:13:02.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:02.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: dump: 2023-10-30T12:13:02.571 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.571 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.571 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.572 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.572 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.572 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.572 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.572 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.572 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.573 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: _finish_auth 0 2023-10-30T12:13:02.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: _check_auth_tickets 2023-10-30T12:13:02.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:02.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 1 -- 172.21.15.50:0/3303491980 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd802f030 con 0x7efdc4008860 2023-10-30T12:13:02.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:02.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 1 -- 172.21.15.50:0/3303491980 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd80326e0 con 0x7efdc4008860 2023-10-30T12:13:02.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde7bd0640 5 monclient: authenticate success, global_id 174099 2023-10-30T12:13:02.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde7bd0640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:02.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde7bd0640 4 set_mon_vals no callback set 2023-10-30T12:13:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.559+0000 7efde496a640 10 monclient: dump: 2023-10-30T12:13:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.575 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.577 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 10 monclient: shutdown 2023-10-30T12:13:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 -- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4008860 msgr2=0x7efdc400d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4008860 0x7efdc400d300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efde009b080 tx=0x7efdd802c650 comp rx=0 tx=0).stop 2023-10-30T12:13:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 -- 172.21.15.50:0/3303491980 shutdown_connections 2023-10-30T12:13:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 --2- 172.21.15.50:0/3303491980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4008860 0x7efdc400d300 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 --2- 172.21.15.50:0/3303491980 >> v2:172.21.15.118:3300/0 conn(0x7efdc400ab50 0x7efdc4005b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 -- 172.21.15.50:0/3303491980 >> 172.21.15.50:0/3303491980 conn(0x7efde00a47f0 msgr2=0x7efde00a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 -- 172.21.15.50:0/3303491980 shutdown_connections 2023-10-30T12:13:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.560+0000 7efde7bd0640 1 -- 172.21.15.50:0/3303491980 wait complete. 2023-10-30T12:13:02.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.561+0000 7efde7bd0640 10 monclient: build_initial_monmap 2023-10-30T12:13:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.561+0000 7efde7bd0640 10 monclient: monmap: 2023-10-30T12:13:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:02.584 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:02.585 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:02.586 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.561+0000 7efde7bd0640 1 librados: starting msgr at 2023-10-30T12:13:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.561+0000 7efde7bd0640 1 librados: starting objecter 2023-10-30T12:13:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.561+0000 7efde7bd0640 1 Processor -- start 2023-10-30T12:13:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 -- start start 2023-10-30T12:13:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 librados: setting wanted keys 2023-10-30T12:13:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 librados: calling monclient init 2023-10-30T12:13:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient: init 2023-10-30T12:13:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:13:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdc4008860 0x7efde01c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): picked mon.noname-c con 0x7efdc4008860 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4009fc0 0x7efde01b98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): picked mon.noname-a con 0x7efdc4009fc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdc400ab50 0x7efde01c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): picked mon.noname-b con 0x7efdc400ab50 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde01a5790 con 0x7efdc4009fc0 2023-10-30T12:13:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde009d710 con 0x7efdc400ab50 2023-10-30T12:13:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde01a7fe0 con 0x7efdc4008860 2023-10-30T12:13:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde7bd0640 10 monclient(hunting): authenticate will time out at 1627.745480s 2023-10-30T12:13:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdc4008860 0x7efde01c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4009fc0 0x7efde01b98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdc4008860 0x7efde01c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50372/0 (socket says 172.21.15.50:50372) 2023-10-30T12:13:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 1 -- 172.21.15.50:0/1861413114 learned_addr learned my addr 172.21.15.50:0/1861413114 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 10 monclient(hunting): get_auth_request con 0x7efdc4009fc0 auth_method 0 2023-10-30T12:13:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 10 monclient(hunting): get_auth_request con 0x7efdc4008860 auth_method 0 2023-10-30T12:13:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdc400ab50 0x7efde01c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 10 monclient(hunting): get_auth_request con 0x7efdc400ab50 auth_method 0 2023-10-30T12:13:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde516b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde596c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.562+0000 7efde616d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 10 monclient(hunting): handle_auth_done global_id 174105 payload 824 2023-10-30T12:13:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 1 -- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdc4008860 msgr2=0x7efde01c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdc4008860 0x7efde01c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 1 -- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdc400ab50 msgr2=0x7efde01c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdc400ab50 0x7efde01c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 10 monclient: _finish_hunting 0 2023-10-30T12:13:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 1 monclient: found mon.noname-a 2023-10-30T12:13:02.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 1 -- 172.21.15.50:0/1861413114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8009780 con 0x7efdc4009fc0 2023-10-30T12:13:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde596c640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdc4008860 0x7efde01c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde516b640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4009fc0 0x7efde01b98d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d540 tx=0x7efdd400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 client.?.objecter ms_handle_connect 0x7efdc4009fc0 2023-10-30T12:13:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 1 -- 172.21.15.50:0/1861413114 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4007480 con 0x7efdc4009fc0 2023-10-30T12:13:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: dump: 2023-10-30T12:13:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.600 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: _finish_auth 0 2023-10-30T12:13:02.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:13:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 1 -- 172.21.15.50:0/1861413114 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd40075e0 con 0x7efdc4009fc0 2023-10-30T12:13:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 5 monclient: authenticate success, global_id 174105 2023-10-30T12:13:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 10 monclient: _renew_subs 2023-10-30T12:13:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde01a7a20 con 0x7efdc4009fc0 2023-10-30T12:13:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 1 -- 172.21.15.50:0/1861413114 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4005a00 con 0x7efdc4009fc0 2023-10-30T12:13:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:02.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd27fc640 10 monclient: dump: 2023-10-30T12:13:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.603 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.604 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efdd3fff640 4 set_mon_vals no callback set 2023-10-30T12:13:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 10 client.174105.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 10 monclient: _renew_subs 2023-10-30T12:13:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde01ca910 con 0x7efdc4009fc0 2023-10-30T12:13:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.563+0000 7efde7bd0640 1 librados: init done 2023-10-30T12:13:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efdd27fc640 1 -- 172.21.15.50:0/1861413114 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efdd4005b60 con 0x7efdc4009fc0 2023-10-30T12:13:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efde7bd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efde7bd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efde7bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde004e160 con 0x7efdc4009fc0 2023-10-30T12:13:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efdd27fc640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efdc403adc0 0x7efdc403d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efdd27fc640 1 -- 172.21.15.50:0/1861413114 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(396..396 src has 1..396) v4 ==== 6482+0+0 (secure 0 0 0) 0x7efdd404acf0 con 0x7efdc4009fc0 2023-10-30T12:13:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efdd27fc640 10 client.174105.objecter ms_dispatch 0x7efde0000b90 osd_map(396..396 src has 1..396) v4 2023-10-30T12:13:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efdd27fc640 3 client.174105.objecter handle_osd_map got epochs [396,396] > 0 2023-10-30T12:13:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.564+0000 7efdd27fc640 3 client.174105.objecter handle_osd_map decoding full epoch 396 2023-10-30T12:13:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.565+0000 7efdd27fc640 20 client.174105.objecter dump_active .. 0 homeless 2023-10-30T12:13:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.565+0000 7efde596c640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efdc403adc0 0x7efdc403d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.565+0000 7efde596c640 10 monclient: get_auth_request con 0x7efdc403adc0 auth_method 0 2023-10-30T12:13:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.565+0000 7efde596c640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efdc403adc0 0x7efdc403d280 secure :-1 s=READY pgs=4143 cs=0 l=1 rev1=1 crypto rx=0x7efddc0175f0 tx=0x7efddc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.565+0000 7efdd27fc640 10 client.174105.objecter ms_handle_connect 0x7efdc403adc0 2023-10-30T12:13:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.569+0000 7efdd27fc640 1 -- 172.21.15.50:0/1861413114 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efdd401a360 con 0x7efdc4009fc0 2023-10-30T12:13:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.569+0000 7efdd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:02.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.569+0000 7efdd27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.813+0000 7efde7bd0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.813+0000 7efde7bd0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.813+0000 7efde7bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.813+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efde009d200 con 0x7efdc4009fc0 2023-10-30T12:13:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.814+0000 7efdd27fc640 1 -- 172.21.15.50:0/1861413114 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7efdd4011370 con 0x7efdc4009fc0 2023-10-30T12:13:02.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.814+0000 7efdd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:02.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.814+0000 7efdd27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:02.819 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.815+0000 7efde7bd0640 10 librados: watch_flush enter 2023-10-30T12:13:02.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.815+0000 7efde7bd0640 10 librados: watch_flush exit 2023-10-30T12:13:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.815+0000 7efde7bd0640 20 client.174105.objecter shutdown clearing up homeless session... 2023-10-30T12:13:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.815+0000 7efde7bd0640 10 client.174105.objecter successfully canceled tick 2023-10-30T12:13:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efdc403adc0 msgr2=0x7efdc403d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efdc403adc0 0x7efdc403d280 secure :-1 s=READY pgs=4143 cs=0 l=1 rev1=1 crypto rx=0x7efddc0175f0 tx=0x7efddc008040 comp rx=0 tx=0).stop 2023-10-30T12:13:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 10 monclient: shutdown 2023-10-30T12:13:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4009fc0 msgr2=0x7efde01b98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4009fc0 0x7efde01b98d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d540 tx=0x7efdd400da00 comp rx=0 tx=0).stop 2023-10-30T12:13:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 shutdown_connections 2023-10-30T12:13:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efdc403adc0 0x7efdc403d280 unknown :-1 s=CLOSED pgs=4143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efdc400ab50 0x7efde01c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdc4009fc0 0x7efde01b98d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 --2- 172.21.15.50:0/1861413114 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7efdc4008860 0x7efde01c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 >> 172.21.15.50:0/1861413114 conn(0x7efde00a47f0 msgr2=0x7efde009b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:02.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 shutdown_connections 2023-10-30T12:13:02.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 -- 172.21.15.50:0/1861413114 wait complete. 2023-10-30T12:13:02.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.816+0000 7efde7bd0640 1 librados: shutdown 2023-10-30T12:13:02.827 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:13:02.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:13:02.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:13:02.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.932+0000 7f5c34a75640 10 monclient: get_monmap_and_config 2023-10-30T12:13:02.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.932+0000 7f5c34a75640 10 monclient: build_initial_monmap 2023-10-30T12:13:02.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.932+0000 7f5c34a75640 10 monclient: monmap: 2023-10-30T12:13:02.934 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:02.935 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.935 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:02.933707+0000 2023-10-30T12:13:02.935 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:02.933707+0000 2023-10-30T12:13:02.935 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:02.935 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.936 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:02.936 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:02.936 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:02.936 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:02.936 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:02.936 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 1 Processor -- start 2023-10-30T12:13:02.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 1 -- start start 2023-10-30T12:13:02.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient: init 2023-10-30T12:13:02.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:02.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-10-30T12:13:02.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c3009e7e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:02.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c3006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3009f2d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:02.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5c30138b60 0x7f5c3013af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient(hunting): picked mon.noname-a con 0x7f5c30138b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:02.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3009d560 con 0x7f5c3009e7e0 2023-10-30T12:13:02.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3009fbf0 con 0x7f5c30138b60 2023-10-30T12:13:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3009fed0 con 0x7f5c3009f2d0 2023-10-30T12:13:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c34a75640 10 monclient(hunting): authenticate will time out at 1628.116866s 2023-10-30T12:13:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c27fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c3006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.933+0000 7f5c2e59c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5c3009e7e0 0x7f5c3009ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47676/0 (socket says 172.21.15.50:47676) 2023-10-30T12:13:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2e59c640 1 -- 172.21.15.50:0/507115327 learned_addr learned my addr 172.21.15.50:0/507115327 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 1 --2- 172.21.15.50:0/507115327 >> v2:172.21.15.50:3300/0 conn(0x7f5c30138b60 0x7f5c3013af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): get_auth_request con 0x7f5c3009f2d0 auth_method 0 2023-10-30T12:13:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 10 monclient(hunting): get_auth_request con 0x7f5c30138b60 auth_method 0 2023-10-30T12:13:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient(hunting): handle_auth_done global_id 114737 payload 274 2023-10-30T12:13:02.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 1 -- 172.21.15.50:0/507115327 >> v2:172.21.15.50:3300/0 conn(0x7f5c30138b60 msgr2=0x7f5c3013af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:02.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 1 --2- 172.21.15.50:0/507115327 >> v2:172.21.15.50:3300/0 conn(0x7f5c30138b60 0x7f5c3013af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 1 -- 172.21.15.50:0/507115327 >> v1:172.21.15.50:6789/0 conn(0x7f5c3009e7e0 legacy=0x7f5c3009ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 1 monclient: found mon.noname-b 2023-10-30T12:13:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c27fff640 1 -- 172.21.15.50:0/507115327 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c300a01b0 con 0x7f5c3009f2d0 2023-10-30T12:13:02.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.934+0000 7f5c2ed9d640 1 --2- 172.21.15.50:0/507115327 >> v2:172.21.15.50:3300/0 conn(0x7f5c30138b60 0x7f5c3013af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:02.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c27fff640 1 --2- 172.21.15.50:0/507115327 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c3006c3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5c180097a0 tx=0x7f5c1802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=194902c8ab8804b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 1 -- 172.21.15.50:0/507115327 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1802f050 con 0x7f5c3009f2d0 2023-10-30T12:13:02.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: dump: 2023-10-30T12:13:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.955 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.955 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.955 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.955 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.955 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: _finish_auth 0 2023-10-30T12:13:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:13:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 1 -- 172.21.15.50:0/507115327 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18031700 con 0x7f5c3009f2d0 2023-10-30T12:13:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 5 monclient: authenticate success, global_id 114737 2023-10-30T12:13:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 4 set_mon_vals no callback set 2023-10-30T12:13:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 1 -- 172.21.15.50:0/507115327 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18031a90 con 0x7f5c3009f2d0 2023-10-30T12:13:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c2dd9b640 10 monclient: dump: 2023-10-30T12:13:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.960 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 10 monclient: shutdown 2023-10-30T12:13:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 -- 172.21.15.50:0/507115327 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 msgr2=0x7f5c3006c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 --2- 172.21.15.50:0/507115327 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c3006c3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5c180097a0 tx=0x7f5c1802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 -- 172.21.15.50:0/507115327 shutdown_connections 2023-10-30T12:13:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 --2- 172.21.15.50:0/507115327 >> v2:172.21.15.50:3300/0 conn(0x7f5c30138b60 0x7f5c3013af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 --2- 172.21.15.50:0/507115327 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c3006c3c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 -- 172.21.15.50:0/507115327 >> 172.21.15.50:0/507115327 conn(0x7f5c300a4800 msgr2=0x7f5c300a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 -- 172.21.15.50:0/507115327 shutdown_connections 2023-10-30T12:13:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.935+0000 7f5c34a75640 1 -- 172.21.15.50:0/507115327 wait complete. 2023-10-30T12:13:02.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 10 monclient: build_initial_monmap 2023-10-30T12:13:02.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 10 monclient: monmap: 2023-10-30T12:13:02.962 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:02.962 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:02.962 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:02.964 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 1 librados: starting msgr at 2023-10-30T12:13:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 1 librados: starting objecter 2023-10-30T12:13:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 1 Processor -- start 2023-10-30T12:13:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 1 -- start start 2023-10-30T12:13:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 1 librados: setting wanted keys 2023-10-30T12:13:02.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 1 librados: calling monclient init 2023-10-30T12:13:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.936+0000 7f5c34a75640 10 monclient: init 2023-10-30T12:13:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:13:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c3009e7e0 0x7f5c301380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3009e7e0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c301c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3009f2d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c30138b60 0x7f5c3011a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): picked mon.noname-a con 0x7f5c30138b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3013ac80 con 0x7f5c30138b60 2023-10-30T12:13:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c30070de0 con 0x7f5c3009e7e0 2023-10-30T12:13:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3011d400 con 0x7f5c3009f2d0 2023-10-30T12:13:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c34a75640 10 monclient(hunting): authenticate will time out at 1628.120363s 2023-10-30T12:13:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c301c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c30138b60 0x7f5c3011a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c30138b60 0x7f5c3011a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56970/0 (socket says 172.21.15.50:56970) 2023-10-30T12:13:02.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c301c89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50398/0 (socket says 172.21.15.50:50398) 2023-10-30T12:13:02.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 -- 172.21.15.50:0/1152477153 learned_addr learned my addr 172.21.15.50:0/1152477153 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:02.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): get_auth_request con 0x7f5c30138b60 auth_method 0 2023-10-30T12:13:02.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 10 monclient(hunting): get_auth_request con 0x7f5c3009f2d0 auth_method 0 2023-10-30T12:13:02.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2e59c640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c3009e7e0 0x7f5c301380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2e59c640 10 monclient(hunting): get_auth_request con 0x7f5c3009e7e0 auth_method 0 2023-10-30T12:13:02.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient(hunting): handle_auth_done global_id 174114 payload 824 2023-10-30T12:13:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 msgr2=0x7f5c301c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5c3009f2d0 0x7f5c301c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c3009e7e0 msgr2=0x7f5c301380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c3009e7e0 0x7f5c301380c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:13:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 monclient: found mon.noname-a 2023-10-30T12:13:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 -- 172.21.15.50:0/1152477153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c28009a70 con 0x7f5c30138b60 2023-10-30T12:13:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c2ed9d640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c30138b60 0x7f5c3011a430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800efb0 tx=0x7f5c2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c26ffd640 10 client.?.objecter ms_handle_connect 0x7f5c30138b60 2023-10-30T12:13:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.937+0000 7f5c26ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28009d50 con 0x7f5c30138b60 2023-10-30T12:13:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: dump: 2023-10-30T12:13:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.978 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 5 monclient: authenticate success, global_id 174114 2023-10-30T12:13:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2800f040 con 0x7f5c30138b60 2023-10-30T12:13:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 10 monclient: _renew_subs 2023-10-30T12:13:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c18009400 con 0x7f5c30138b60 2023-10-30T12:13:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280146a0 con 0x7f5c30138b60 2023-10-30T12:13:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c2d59a640 4 set_mon_vals no callback set 2023-10-30T12:13:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 10 monclient: dump: 2023-10-30T12:13:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.981 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.981 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.981 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.981 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.981 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.982 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 10 client.174114.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 10 monclient: _renew_subs 2023-10-30T12:13:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3011c1f0 con 0x7f5c30138b60 2023-10-30T12:13:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 1 librados: init done 2023-10-30T12:13:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c3011a970 0x7f5c3011ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c300a1280 con 0x7f5c3011a970 2023-10-30T12:13:02.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.938+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5c28014800 con 0x7f5c30138b60 2023-10-30T12:13:02.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c3011a970 0x7f5c3011ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient: get_auth_request con 0x7f5c3011a970 auth_method 0 2023-10-30T12:13:02.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c100392f0 0x7f5c1003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(396..396 src has 1..396) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f5c2804bd40 con 0x7f5c30138b60 2023-10-30T12:13:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 10 client.174114.objecter ms_dispatch 0x7f5c30000b90 osd_map(396..396 src has 1..396) v4 2023-10-30T12:13:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 3 client.174114.objecter handle_osd_map got epochs [396,396] > 0 2023-10-30T12:13:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 3 client.174114.objecter handle_osd_map decoding full epoch 396 2023-10-30T12:13:02.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 20 client.174114.objecter dump_active .. 0 homeless 2023-10-30T12:13:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_done global_id 174117 payload 824 2023-10-30T12:13:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 1 -- 172.21.15.50:0/1152477153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c3011d5d0 con 0x7f5c3011a970 2023-10-30T12:13:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c27fff640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c100392f0 0x7f5c1003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c27fff640 10 monclient: get_auth_request con 0x7f5c100392f0 auth_method 0 2023-10-30T12:13:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 10 client.174114.objecter ms_handle_connect 0x7f5c3011a970 2023-10-30T12:13:02.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 10 client.174114.objecter resend_mon_ops 2023-10-30T12:13:02.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c2e59c640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c3011a970 0x7f5c3011ad50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000f60 tx=0x7f5c2000e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20005ac0 con 0x7f5c3011a970 2023-10-30T12:13:02.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.939+0000 7f5c26ffd640 10 monclient: dump: 2023-10-30T12:13:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:02.991 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 10 client.174114.objecter ms_handle_connect 0x7f5c100392f0 2023-10-30T12:13:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c27fff640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c100392f0 0x7f5c1003b7b0 secure :-1 s=READY pgs=4144 cs=0 l=1 rev1=1 crypto rx=0x7f5c18035000 tx=0x7f5c18008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5c20005c20 con 0x7f5c3011a970 2023-10-30T12:13:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c3011a970 msgr2=0x7f5c3011ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c3011a970 0x7f5c3011ad50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000f60 tx=0x7f5c2000e8b0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c3011a970 msgr2=0x7f5c3011ad50 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.940+0000 7f5c26ffd640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c3011a970 0x7f5c3011ad50 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000f60 tx=0x7f5c2000e8b0 comp rx=0 tx=0).stop 2023-10-30T12:13:02.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c34a75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:02.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c34a75640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c301346b0 0x7f5c30134ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c34a75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3011d790 con 0x7f5c301346b0 2023-10-30T12:13:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c301346b0 0x7f5c30134ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient: get_auth_request con 0x7f5c301346b0 auth_method 0 2023-10-30T12:13:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 10 monclient(hunting): handle_auth_done global_id 174120 payload 824 2023-10-30T12:13:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 1 -- 172.21.15.50:0/1152477153 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5c3009e060 con 0x7f5c301346b0 2023-10-30T12:13:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c2e59c640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c301346b0 0x7f5c30134ac0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000fb0 tx=0x7f5c200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c26ffd640 10 client.174114.objecter ms_handle_connect 0x7f5c301346b0 2023-10-30T12:13:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c26ffd640 10 client.174114.objecter resend_mon_ops 2023-10-30T12:13:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20014890 con 0x7f5c301346b0 2023-10-30T12:13:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.946+0000 7f5c26ffd640 10 monclient: dump: 2023-10-30T12:13:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.000 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f5c20007750 con 0x7f5c301346b0 2023-10-30T12:13:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c26ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c26ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c301346b0 msgr2=0x7f5c30134ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c26ffd640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c301346b0 0x7f5c30134ac0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000fb0 tx=0x7f5c200057f0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c26ffd640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c301346b0 msgr2=0x7f5c30134ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:03.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c26ffd640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c301346b0 0x7f5c30134ac0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000fb0 tx=0x7f5c200057f0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.001 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:03.002 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:03.002 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:03.002 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:13:03.002 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 124, 2023-10-30T12:13:03.002 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:03.003 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:03.003 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:03.003 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:03.003 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.003 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:13:03.004 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:03.004 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:03.004 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:03.004 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.004 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.005 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.005 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.005 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.005 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.005 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.006 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.006 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.006 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.006 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:03.006 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:03.007 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.007 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.007 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.007 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.007 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.008 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.008 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.008 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.008 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.008 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.009 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:03.009 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:03.009 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:03.009 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:03.010 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:03.010 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:03.010 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:03.010 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:03.010 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:03.010 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:03.011 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:03.011 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:03.011 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:03.011 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:03.011 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:03.012 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:03.012 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:03.012 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.012 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.012 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.013 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.013 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.013 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.013 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.013 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.014 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.014 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.014 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:03.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:03.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.015 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:03.015 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:03.015 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.015 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.016 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.016 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:03.016 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.017 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.017 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:03.017 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.017 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.017 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.018 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:03.018 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:03.018 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.018 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.018 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.019 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:03.019 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:03.019 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.020 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.020 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.020 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:03.020 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.021 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.021 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:03.021 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.022 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.022 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:03.022 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:03.023 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.023 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.023 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.023 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:03.024 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:03.024 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.024 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.024 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.025 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:03.025 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.025 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.025 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:03.026 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.026 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.026 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:03.027 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:03.027 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.027 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.027 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.028 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.028 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:03.028 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:03.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.029 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.029 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.029 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:03.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.029 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:03.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.030 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.030 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.030 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:13:03.030 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.031 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.031 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:03.031 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:03.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 10 librados: watch_flush enter 2023-10-30T12:13:03.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 10 librados: watch_flush exit 2023-10-30T12:13:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 20 client.174114.objecter shutdown clearing up homeless session... 2023-10-30T12:13:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 10 client.174114.objecter successfully canceled tick 2023-10-30T12:13:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c100392f0 msgr2=0x7f5c1003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5c100392f0 0x7f5c1003b7b0 secure :-1 s=READY pgs=4144 cs=0 l=1 rev1=1 crypto rx=0x7f5c18035000 tx=0x7f5c18008030 comp rx=0 tx=0).stop 2023-10-30T12:13:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 10 monclient: shutdown 2023-10-30T12:13:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c30138b60 msgr2=0x7f5c3011a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 --2- 172.21.15.50:0/1152477153 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c30138b60 0x7f5c3011a430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800efb0 tx=0x7f5c2800c490 comp rx=0 tx=0).stop 2023-10-30T12:13:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c2ed9d640 1 -- 172.21.15.50:0/1152477153 reap_dead start 2023-10-30T12:13:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c2ed9d640 1 -- 172.21.15.50:0/1152477153 reap_dead start 2023-10-30T12:13:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 shutdown_connections 2023-10-30T12:13:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 >> 172.21.15.50:0/1152477153 conn(0x7f5c300a4800 msgr2=0x7f5c300a1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 shutdown_connections 2023-10-30T12:13:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 -- 172.21.15.50:0/1152477153 wait complete. 2023-10-30T12:13:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:02.947+0000 7f5c34a75640 1 librados: shutdown 2023-10-30T12:13:03.037 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:03.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:13:03.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:13:03.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.142+0000 7f6fc7b68640 10 monclient: get_monmap_and_config 2023-10-30T12:13:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.142+0000 7f6fc7b68640 10 monclient: build_initial_monmap 2023-10-30T12:13:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.142+0000 7f6fc7b68640 10 monclient: monmap: 2023-10-30T12:13:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:03.143672+0000 2023-10-30T12:13:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:03.143672+0000 2023-10-30T12:13:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:03.146 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 1 Processor -- start 2023-10-30T12:13:03.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 1 -- start start 2023-10-30T12:13:03.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 10 monclient: init 2023-10-30T12:13:03.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:03.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T12:13:03.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6fc0132970 0x7f6fc0132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc0132970 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:03.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fc00a03d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:03.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0070ee0 0x7f6fc013db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 10 monclient(hunting): picked mon.noname-b con 0x7f6fc0070ee0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:03.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.143+0000 7f6fc7b68640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00a17d0 con 0x7f6fc00a03d0 2023-10-30T12:13:03.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc7b68640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc7b68640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc00a1510 con 0x7f6fc0132970 2023-10-30T12:13:03.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc7b68640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc7b68640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc009f680 con 0x7f6fc0070ee0 2023-10-30T12:13:03.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc7b68640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:03.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc7b68640 10 monclient(hunting): authenticate will time out at 1628.327045s 2023-10-30T12:13:03.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6fc0132970 0x7f6fc0132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0070ee0 0x7f6fc013db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6fc0132970 0x7f6fc0132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57002/0 (socket says 172.21.15.50:57002) 2023-10-30T12:13:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 -- 172.21.15.50:0/487427181 learned_addr learned my addr 172.21.15.50:0/487427181 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): get_auth_request con 0x7f6fc0132970 auth_method 0 2023-10-30T12:13:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 10 monclient(hunting): get_auth_request con 0x7f6fc0070ee0 auth_method 0 2023-10-30T12:13:03.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc6105640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient(hunting): handle_auth_done global_id 174123 payload 274 2023-10-30T12:13:03.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 -- 172.21.15.50:0/487427181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0070ee0 msgr2=0x7f6fc013db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:03.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 --2- 172.21.15.50:0/487427181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0070ee0 0x7f6fc013db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 -- 172.21.15.50:0/487427181 >> v1:172.21.15.118:6789/0 conn(0x7f6fc00a03d0 legacy=0x7f6fc00707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient: _finish_hunting 0 2023-10-30T12:13:03.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 monclient: found mon.noname-a 2023-10-30T12:13:03.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:03.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:03.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.144+0000 7f6fc5904640 1 -- 172.21.15.50:0/487427181 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc009f960 con 0x7f6fc0132970 2023-10-30T12:13:03.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc5904640 1 --2- 172.21.15.50:0/487427181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc0132d50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009870 tx=0x7f6fbc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7dbe5a62c175f6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 1 -- 172.21.15.50:0/487427181 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0317c0 con 0x7f6fc0132970 2023-10-30T12:13:03.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:03.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: dump: 2023-10-30T12:13:03.164 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.164 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.164 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.166 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: _finish_auth 0 2023-10-30T12:13:03.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: _check_auth_tickets 2023-10-30T12:13:03.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:03.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 1 -- 172.21.15.50:0/487427181 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc031920 con 0x7f6fc0132970 2023-10-30T12:13:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 5 monclient: authenticate success, global_id 174123 2023-10-30T12:13:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 1 -- 172.21.15.50:0/487427181 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc031cb0 con 0x7f6fc0132970 2023-10-30T12:13:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 4 set_mon_vals no callback set 2023-10-30T12:13:03.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:03.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc4902640 10 monclient: dump: 2023-10-30T12:13:03.168 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.168 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.169 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.169 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.169 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.169 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.169 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.169 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.170 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.170 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 10 monclient: shutdown 2023-10-30T12:13:03.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:03.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 1 -- 172.21.15.50:0/487427181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 msgr2=0x7f6fc0132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/487427181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc0132d50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009870 tx=0x7f6fbc02b860 comp rx=0 tx=0).stop 2023-10-30T12:13:03.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 1 -- 172.21.15.50:0/487427181 shutdown_connections 2023-10-30T12:13:03.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/487427181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0070ee0 0x7f6fc013db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/487427181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc0132d50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.145+0000 7f6fc7b68640 1 -- 172.21.15.50:0/487427181 >> 172.21.15.50:0/487427181 conn(0x7f6fc00a4800 msgr2=0x7f6fc00a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 -- 172.21.15.50:0/487427181 shutdown_connections 2023-10-30T12:13:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 -- 172.21.15.50:0/487427181 wait complete. 2023-10-30T12:13:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 10 monclient: build_initial_monmap 2023-10-30T12:13:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 10 monclient: monmap: 2023-10-30T12:13:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:03.173 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:03.173 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:03.173 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:03.173 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.173 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:03.174 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:03.174 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:03.174 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 librados: starting msgr at 2023-10-30T12:13:03.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 librados: starting objecter 2023-10-30T12:13:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 Processor -- start 2023-10-30T12:13:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 -- start start 2023-10-30T12:13:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 librados: setting wanted keys 2023-10-30T12:13:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 1 librados: calling monclient init 2023-10-30T12:13:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.146+0000 7f6fc7b68640 10 monclient: init 2023-10-30T12:13:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:13:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fc0070ee0 0x7f6fc01ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): picked mon.noname-b con 0x7f6fc0070ee0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc01cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): picked mon.noname-a con 0x7f6fc0132970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011dbf0 0x7f6fc011e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): picked mon.noname-c con 0x7f6fc011dbf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc0121a10 con 0x7f6fc0132970 2023-10-30T12:13:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc0121c60 con 0x7f6fc0070ee0 2023-10-30T12:13:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc011fe10 con 0x7f6fc011dbf0 2023-10-30T12:13:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc7b68640 10 monclient(hunting): authenticate will time out at 1628.330336s 2023-10-30T12:13:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011dbf0 0x7f6fc011e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc01cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011dbf0 0x7f6fc011e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50422/0 (socket says 172.21.15.50:50422) 2023-10-30T12:13:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 -- 172.21.15.50:0/3366496875 learned_addr learned my addr 172.21.15.50:0/3366496875 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc01cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57018/0 (socket says 172.21.15.50:57018) 2023-10-30T12:13:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): get_auth_request con 0x7f6fc011dbf0 auth_method 0 2023-10-30T12:13:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 10 monclient(hunting): get_auth_request con 0x7f6fc0132970 auth_method 0 2023-10-30T12:13:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5904640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fc0070ee0 0x7f6fc01ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5904640 10 monclient(hunting): get_auth_request con 0x7f6fc0070ee0 auth_method 0 2023-10-30T12:13:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5904640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5904640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient(hunting): handle_auth_done global_id 114746 payload 824 2023-10-30T12:13:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fc0070ee0 msgr2=0x7f6fc01ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fc0070ee0 0x7f6fc01ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 msgr2=0x7f6fc01cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc01cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient: _finish_hunting 0 2023-10-30T12:13:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 monclient: found mon.noname-c 2023-10-30T12:13:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 -- 172.21.15.50:0/3366496875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc0094d0 con 0x7f6fc011dbf0 2023-10-30T12:13:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc5103640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0132970 0x7f6fc01cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fc6105640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011dbf0 0x7f6fc011e0a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009dd0 tx=0x7f6fb800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fb67fc640 10 client.?.objecter ms_handle_connect 0x7f6fc011dbf0 2023-10-30T12:13:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.147+0000 7f6fb67fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8014480 con 0x7f6fc011dbf0 2023-10-30T12:13:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: dump: 2023-10-30T12:13:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.188 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: _finish_auth 0 2023-10-30T12:13:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:13:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb800f040 con 0x7f6fc011dbf0 2023-10-30T12:13:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8014840 con 0x7f6fc011dbf0 2023-10-30T12:13:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 10 monclient: dump: 2023-10-30T12:13:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.191 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 5 monclient: authenticate success, global_id 114746 2023-10-30T12:13:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb7fff640 4 set_mon_vals no callback set 2023-10-30T12:13:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 10 monclient: _renew_subs 2023-10-30T12:13:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc0134bf0 con 0x7f6fc011dbf0 2023-10-30T12:13:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 10 client.114746.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 10 monclient: _renew_subs 2023-10-30T12:13:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc011f890 con 0x7f6fc011dbf0 2023-10-30T12:13:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 1 librados: init done 2023-10-30T12:13:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc00a03d0 0x7f6fc011e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc0121df0 con 0x7f6fc00a03d0 2023-10-30T12:13:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.148+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6fb80149a0 con 0x7f6fc011dbf0 2023-10-30T12:13:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc00a03d0 0x7f6fc011e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient: get_auth_request con 0x7f6fc00a03d0 auth_method 0 2023-10-30T12:13:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6fa40392f0 0x7f6fa403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 10 monclient(hunting): handle_auth_done global_id 114749 payload 824 2023-10-30T12:13:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(396..396 src has 1..396) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f6fb804bae0 con 0x7f6fc011dbf0 2023-10-30T12:13:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 client.114746.objecter ms_dispatch 0x7f6fc0000b90 osd_map(396..396 src has 1..396) v4 2023-10-30T12:13:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 3 client.114746.objecter handle_osd_map got epochs [396,396] > 0 2023-10-30T12:13:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 3 client.114746.objecter handle_osd_map decoding full epoch 396 2023-10-30T12:13:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 1 -- 172.21.15.50:0/3366496875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fc01214e0 con 0x7f6fc00a03d0 2023-10-30T12:13:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 20 client.114746.objecter dump_active .. 0 homeless 2023-10-30T12:13:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 client.114746.objecter ms_handle_connect 0x7f6fc00a03d0 2023-10-30T12:13:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 client.114746.objecter resend_mon_ops 2023-10-30T12:13:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5904640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc00a03d0 0x7f6fc011e5e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6fc01357c0 tx=0x7f6fbc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5103640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6fa40392f0 0x7f6fa403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5103640 10 monclient: get_auth_request con 0x7f6fa40392f0 auth_method 0 2023-10-30T12:13:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc049070 con 0x7f6fc00a03d0 2023-10-30T12:13:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 monclient: dump: 2023-10-30T12:13:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.201 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f6fbc032660 con 0x7f6fc00a03d0 2023-10-30T12:13:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc00a03d0 msgr2=0x7f6fc011e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc00a03d0 0x7f6fc011e5e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6fc01357c0 tx=0x7f6fbc004730 comp rx=0 tx=0).stop 2023-10-30T12:13:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc00a03d0 msgr2=0x7f6fc011e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc00a03d0 0x7f6fc011e5e0 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6fc01357c0 tx=0x7f6fbc004730 comp rx=0 tx=0).stop 2023-10-30T12:13:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fc5103640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6fa40392f0 0x7f6fa403b7b0 secure :-1 s=READY pgs=4145 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0131b30 tx=0x7f6fb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.149+0000 7f6fb67fc640 10 client.114746.objecter ms_handle_connect 0x7f6fa40392f0 2023-10-30T12:13:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc7b68640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011eb20 0x7f6fc011a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc7b68640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc0071a70 con 0x7f6fc011eb20 2023-10-30T12:13:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011eb20 0x7f6fc011a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient: get_auth_request con 0x7f6fc011eb20 auth_method 0 2023-10-30T12:13:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 10 monclient(hunting): handle_auth_done global_id 114752 payload 824 2023-10-30T12:13:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 1 -- 172.21.15.50:0/3366496875 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6fc00675d0 con 0x7f6fc011eb20 2023-10-30T12:13:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fb67fc640 10 client.114746.objecter ms_handle_connect 0x7f6fc011eb20 2023-10-30T12:13:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fb67fc640 10 client.114746.objecter resend_mon_ops 2023-10-30T12:13:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fc5904640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011eb20 0x7f6fc011a430 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0090b0 tx=0x7f6fbc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc045d20 con 0x7f6fc011eb20 2023-10-30T12:13:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.156+0000 7f6fb67fc640 10 monclient: dump: 2023-10-30T12:13:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.210 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f6fbc058440 con 0x7f6fc011eb20 2023-10-30T12:13:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fb67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fb67fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011eb20 msgr2=0x7f6fc011a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fb67fc640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011eb20 0x7f6fc011a430 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0090b0 tx=0x7f6fbc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fb67fc640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011eb20 msgr2=0x7f6fc011a430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fb67fc640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011eb20 0x7f6fc011a430 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0090b0 tx=0x7f6fbc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.212 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:03.212 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:03.212 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:03.212 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:03.213 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 124, 2023-10-30T12:13:03.213 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:03.213 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:03.213 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:03.213 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:03.213 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.214 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:13:03.214 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:03.214 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:03.214 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:03.214 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.215 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.215 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.215 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.215 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.215 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.216 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.216 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.216 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.216 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.216 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.216 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:03.217 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:03.217 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.217 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.217 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.218 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.218 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.218 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.218 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.218 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.218 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.219 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.219 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:03.219 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:03.219 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:03.220 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:03.220 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:03.220 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:03.220 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:03.220 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:03.221 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:03.221 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:03.221 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:03.221 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:03.221 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:03.221 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:03.222 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:03.222 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:03.222 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:03.222 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.222 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.223 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.223 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.223 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.223 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.223 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.224 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.224 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.224 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.224 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.224 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:03.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.225 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:03.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.225 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:03.225 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:03.225 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.226 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.226 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.226 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:03.226 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.227 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.227 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:03.227 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.229 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.229 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.229 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:03.230 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:03.230 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.230 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.230 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.231 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:03.231 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:03.231 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.231 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.232 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.232 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:03.232 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.233 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.233 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:03.233 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.233 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.234 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:03.234 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:03.234 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.234 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.235 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.235 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:03.235 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:03.236 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.236 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.236 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:03.236 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.237 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.237 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:03.238 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.238 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.238 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.238 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:03.238 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:03.239 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.239 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.239 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.239 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.239 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.241 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:03.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.241 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.241 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:13:03.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.242 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.242 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.242 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:13:03.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.242 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.243 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:03.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.243 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.243 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.243 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:13:03.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.244 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.244 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:03.244 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:03.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 10 librados: watch_flush enter 2023-10-30T12:13:03.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 10 librados: watch_flush exit 2023-10-30T12:13:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 20 client.114746.objecter shutdown clearing up homeless session... 2023-10-30T12:13:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 10 client.114746.objecter successfully canceled tick 2023-10-30T12:13:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6fa40392f0 msgr2=0x7f6fa403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6fa40392f0 0x7f6fa403b7b0 secure :-1 s=READY pgs=4145 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0131b30 tx=0x7f6fb0009040 comp rx=0 tx=0).stop 2023-10-30T12:13:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 10 monclient: shutdown 2023-10-30T12:13:03.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011dbf0 msgr2=0x7f6fc011e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc6105640 1 -- 172.21.15.50:0/3366496875 reap_dead start 2023-10-30T12:13:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011dbf0 0x7f6fc011e0a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009dd0 tx=0x7f6fb800c490 comp rx=0 tx=0).stop 2023-10-30T12:13:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 shutdown_connections 2023-10-30T12:13:03.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 --2- 172.21.15.50:0/3366496875 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc011dbf0 0x7f6fc011e0a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 >> 172.21.15.50:0/3366496875 conn(0x7f6fc00a4800 msgr2=0x7f6fc0130690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 shutdown_connections 2023-10-30T12:13:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 -- 172.21.15.50:0/3366496875 wait complete. 2023-10-30T12:13:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.157+0000 7f6fc7b68640 1 librados: shutdown 2023-10-30T12:13:03.250 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:03.250 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:13:03.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:13:03.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.356+0000 7f92601cc640 10 monclient: get_monmap_and_config 2023-10-30T12:13:03.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.356+0000 7f92601cc640 10 monclient: build_initial_monmap 2023-10-30T12:13:03.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.356+0000 7f92601cc640 10 monclient: monmap: 2023-10-30T12:13:03.358 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:03.359 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.359 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:03.357793+0000 2023-10-30T12:13:03.359 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:03.357793+0000 2023-10-30T12:13:03.359 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:03.359 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.360 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:03.360 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:03.360 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:03.360 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:03.360 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:03.361 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 1 Processor -- start 2023-10-30T12:13:03.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 1 -- start start 2023-10-30T12:13:03.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient: init 2023-10-30T12:13:03.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:03.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-10-30T12:13:03.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9258071c00 0x7f925809a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient(hunting): picked mon.noname-c con 0x7f9258071c00 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:03.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9258097fe0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:03.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f925809b170 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:03.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9258097600 con 0x7f9258097fe0 2023-10-30T12:13:03.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580978e0 con 0x7f925809b170 2023-10-30T12:13:03.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925809bf50 con 0x7f9258071c00 2023-10-30T12:13:03.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:03.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.357+0000 7f92601cc640 10 monclient(hunting): authenticate will time out at 1628.540977s 2023-10-30T12:13:03.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925d767640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9258097fe0 0x7f925809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47678/0 (socket says 172.21.15.50:47678) 2023-10-30T12:13:03.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925d767640 1 -- 172.21.15.50:0/589927686 learned_addr learned my addr 172.21.15.50:0/589927686 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:03.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 1 --2- 172.21.15.50:0/589927686 >> v2:172.21.15.118:3300/0 conn(0x7f9258071c00 0x7f925809a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): get_auth_request con 0x7f9258071c00 auth_method 0 2023-10-30T12:13:03.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 1 -- 172.21.15.50:0/589927686 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580762136 0 0) 0x7f9258097600 con 0x7f9258097fe0 2023-10-30T12:13:03.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 10 monclient(hunting): _negotiate 2023-10-30T12:13:03.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 10 monclient(hunting): my global_id is 174129 2023-10-30T12:13:03.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 1 -- 172.21.15.50:0/589927686 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9244003520 con 0x7f9258097fe0 2023-10-30T12:13:03.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient(hunting): handle_auth_done global_id 154321 payload 274 2023-10-30T12:13:03.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 1 -- 172.21.15.50:0/589927686 >> v1:172.21.15.118:6789/0 conn(0x7f925809b170 legacy=0x7f9258139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 1 -- 172.21.15.50:0/589927686 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 195570614 0 0) 0x7f9244003520 con 0x7f9258097fe0 2023-10-30T12:13:03.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 1 -- 172.21.15.50:0/589927686 >> v1:172.21.15.50:6789/0 conn(0x7f9258097fe0 legacy=0x7f925809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient: _finish_hunting 0 2023-10-30T12:13:03.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 1 monclient: found mon.noname-c 2023-10-30T12:13:03.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:03.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:13:03.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925df68640 1 -- 172.21.15.50:0/589927686 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925809d9b0 con 0x7f9258071c00 2023-10-30T12:13:03.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.358+0000 7f925cf66640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:13:03.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925df68640 1 --2- 172.21.15.50:0/589927686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 0x7f925809a150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9254009790 tx=0x7f925402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c79d9a8159c9042 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 1 -- 172.21.15.50:0/589927686 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248001670 con 0x7f9258071c00 2023-10-30T12:13:03.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: dump: 2023-10-30T12:13:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.383 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.383 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: _finish_auth 0 2023-10-30T12:13:03.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: _check_auth_tickets 2023-10-30T12:13:03.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:03.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 1 -- 172.21.15.50:0/589927686 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925402e050 con 0x7f9258071c00 2023-10-30T12:13:03.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:03.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 1 -- 172.21.15.50:0/589927686 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254030710 con 0x7f9258071c00 2023-10-30T12:13:03.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f92601cc640 5 monclient: authenticate success, global_id 154321 2023-10-30T12:13:03.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f92601cc640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:03.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f92601cc640 4 set_mon_vals no callback set 2023-10-30T12:13:03.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:03.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.359+0000 7f925cf66640 10 monclient: dump: 2023-10-30T12:13:03.385 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.385 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.386 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.386 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.386 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.386 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.386 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.387 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 10 monclient: shutdown 2023-10-30T12:13:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 1 -- 172.21.15.50:0/589927686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 msgr2=0x7f925809a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 1 --2- 172.21.15.50:0/589927686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 0x7f925809a150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9254009790 tx=0x7f925402b710 comp rx=0 tx=0).stop 2023-10-30T12:13:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 1 -- 172.21.15.50:0/589927686 shutdown_connections 2023-10-30T12:13:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 1 --2- 172.21.15.50:0/589927686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 0x7f925809a150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 1 -- 172.21.15.50:0/589927686 >> 172.21.15.50:0/589927686 conn(0x7f92580a3980 msgr2=0x7f92580a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:03.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 1 -- 172.21.15.50:0/589927686 shutdown_connections 2023-10-30T12:13:03.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 1 -- 172.21.15.50:0/589927686 wait complete. 2023-10-30T12:13:03.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 10 monclient: build_initial_monmap 2023-10-30T12:13:03.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.360+0000 7f92601cc640 10 monclient: monmap: 2023-10-30T12:13:03.389 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:03.389 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:03.390 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:03.390 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:03.390 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:03.390 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.390 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:03.391 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:03.391 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:03.391 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 librados: starting msgr at 2023-10-30T12:13:03.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 librados: starting objecter 2023-10-30T12:13:03.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 Processor -- start 2023-10-30T12:13:03.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 -- start start 2023-10-30T12:13:03.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 librados: setting wanted keys 2023-10-30T12:13:03.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 librados: calling monclient init 2023-10-30T12:13:03.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient: init 2023-10-30T12:13:03.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:03.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:13:03.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 0x7f9258096480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): picked mon.noname-b con 0x7f9258071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:03.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9258097fe0 0x7f92580969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): picked mon.noname-c con 0x7f9258097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:03.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925809b170 0x7f92581b7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): picked mon.noname-a con 0x7f925809b170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:03.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9258070f70 con 0x7f925809b170 2023-10-30T12:13:03.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92581ae650 con 0x7f9258071c00 2023-10-30T12:13:03.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92581ae930 con 0x7f9258097fe0 2023-10-30T12:13:03.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:03.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f92601cc640 10 monclient(hunting): authenticate will time out at 1628.544789s 2023-10-30T12:13:03.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925809b170 0x7f92581b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925809b170 0x7f92581b7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57030/0 (socket says 172.21.15.50:57030) 2023-10-30T12:13:03.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 1 -- 172.21.15.50:0/3737888024 learned_addr learned my addr 172.21.15.50:0/3737888024 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:03.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 10 monclient(hunting): get_auth_request con 0x7f925809b170 auth_method 0 2023-10-30T12:13:03.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925d767640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9258097fe0 0x7f92580969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925d767640 10 monclient(hunting): get_auth_request con 0x7f9258097fe0 auth_method 0 2023-10-30T12:13:03.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925d767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925d767640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925d767640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.361+0000 7f925e769640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925d767640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925df68640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 0x7f9258096480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925d767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925d767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925df68640 10 monclient(hunting): get_auth_request con 0x7f9258071c00 auth_method 0 2023-10-30T12:13:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925df68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925df68640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925df68640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 10 monclient(hunting): handle_auth_done global_id 174132 payload 824 2023-10-30T12:13:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9258097fe0 msgr2=0x7f92580969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9258097fe0 0x7f92580969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 msgr2=0x7f9258096480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 0x7f9258096480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 10 monclient: _finish_hunting 0 2023-10-30T12:13:03.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 1 monclient: found mon.noname-a 2023-10-30T12:13:03.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 1 -- 172.21.15.50:0/3737888024 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92540093a0 con 0x7f925809b170 2023-10-30T12:13:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925df68640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9258071c00 0x7f9258096480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:13:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f925e769640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925809b170 0x7f92581b7680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f925000c7a0 tx=0x7f925000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 client.?.objecter ms_handle_connect 0x7f925809b170 2023-10-30T12:13:03.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:03.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250003b70 con 0x7f925809b170 2023-10-30T12:13:03.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:03.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: dump: 2023-10-30T12:13:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.404 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.405 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.405 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:13:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:13:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 5 monclient: authenticate success, global_id 174132 2023-10-30T12:13:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 10 monclient: _renew_subs 2023-10-30T12:13:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250003cf0 con 0x7f925809b170 2023-10-30T12:13:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92581b8f10 con 0x7f925809b170 2023-10-30T12:13:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250003ea0 con 0x7f925809b170 2023-10-30T12:13:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924e7fc640 10 monclient: dump: 2023-10-30T12:13:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f924ffff640 4 set_mon_vals no callback set 2023-10-30T12:13:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 10 client.174132.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 10 monclient: _renew_subs 2023-10-30T12:13:03.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:03.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9258096f00 con 0x7f925809b170 2023-10-30T12:13:03.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.362+0000 7f92601cc640 1 librados: init done 2023-10-30T12:13:03.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f92601cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:03.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f925000aa90 con 0x7f925809b170 2023-10-30T12:13:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f92601cc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b7bc0 0x7f92581b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f92601cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9258073ff0 con 0x7f92581b7bc0 2023-10-30T12:13:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f924e7fc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f923c0392f0 0x7f923c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b7bc0 0x7f92581b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient: get_auth_request con 0x7f92581b7bc0 auth_method 0 2023-10-30T12:13:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(396..396 src has 1..396) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f925004ab20 con 0x7f925809b170 2023-10-30T12:13:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f924e7fc640 10 client.174132.objecter ms_dispatch 0x7f9258000b90 osd_map(396..396 src has 1..396) v4 2023-10-30T12:13:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f924e7fc640 3 client.174132.objecter handle_osd_map got epochs [396,396] > 0 2023-10-30T12:13:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f924e7fc640 3 client.174132.objecter handle_osd_map decoding full epoch 396 2023-10-30T12:13:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925d767640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f923c0392f0 0x7f923c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925d767640 10 monclient: get_auth_request con 0x7f923c0392f0 auth_method 0 2023-10-30T12:13:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f924e7fc640 20 client.174132.objecter dump_active .. 0 homeless 2023-10-30T12:13:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 10 monclient(hunting): handle_auth_done global_id 154330 payload 824 2023-10-30T12:13:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.363+0000 7f925df68640 1 -- 172.21.15.50:0/3737888024 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92581ae7c0 con 0x7f92581b7bc0 2023-10-30T12:13:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f925df68640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b7bc0 0x7f92581b7fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f925806bb30 tx=0x7f92540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 client.174132.objecter ms_handle_connect 0x7f92581b7bc0 2023-10-30T12:13:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 client.174132.objecter resend_mon_ops 2023-10-30T12:13:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f925d767640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f923c0392f0 0x7f923c03b7b0 secure :-1 s=READY pgs=4146 cs=0 l=1 rev1=1 crypto rx=0x7f9258070b10 tx=0x7f9248001da0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 client.174132.objecter ms_handle_connect 0x7f923c0392f0 2023-10-30T12:13:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254032950 con 0x7f92581b7bc0 2023-10-30T12:13:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 monclient: dump: 2023-10-30T12:13:03.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.418 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.418 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.418 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.419 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9254041070 con 0x7f92581b7bc0 2023-10-30T12:13:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b7bc0 msgr2=0x7f92581b7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b7bc0 0x7f92581b7fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f925806bb30 tx=0x7f92540046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b7bc0 msgr2=0x7f92581b7fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:03.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.364+0000 7f924e7fc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b7bc0 0x7f92581b7fa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.370+0000 7f92601cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.370+0000 7f92601cc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b84e0 0x7f925806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.370+0000 7f92601cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.370+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92581ad940 con 0x7f92581b84e0 2023-10-30T12:13:03.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b84e0 0x7f925806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient: get_auth_request con 0x7f92581b84e0 auth_method 0 2023-10-30T12:13:03.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 10 monclient(hunting): handle_auth_done global_id 154333 payload 824 2023-10-30T12:13:03.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 1 -- 172.21.15.50:0/3737888024 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9258071fe0 con 0x7f92581b84e0 2023-10-30T12:13:03.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f925df68640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b84e0 0x7f925806ddb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9254009790 tx=0x7f9254030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 10 client.174132.objecter ms_handle_connect 0x7f92581b84e0 2023-10-30T12:13:03.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 10 client.174132.objecter resend_mon_ops 2023-10-30T12:13:03.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254003aa0 con 0x7f92581b84e0 2023-10-30T12:13:03.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 10 monclient: dump: 2023-10-30T12:13:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.428 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.428 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.428 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.428 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.428 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.429 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.429 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f9254003c00 con 0x7f92581b84e0 2023-10-30T12:13:03.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:03.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:03.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b84e0 msgr2=0x7f925806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b84e0 0x7f925806ddb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9254009790 tx=0x7f9254030000 comp rx=0 tx=0).stop 2023-10-30T12:13:03.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b84e0 msgr2=0x7f925806ddb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:03.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.371+0000 7f924e7fc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92581b84e0 0x7f925806ddb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.431 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 124, 2023-10-30T12:13:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:13:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:13:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:13:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:03.464 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:03.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 10 librados: watch_flush enter 2023-10-30T12:13:03.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 10 librados: watch_flush exit 2023-10-30T12:13:03.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 20 client.174132.objecter shutdown clearing up homeless session... 2023-10-30T12:13:03.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 10 client.174132.objecter successfully canceled tick 2023-10-30T12:13:03.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f923c0392f0 msgr2=0x7f923c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f923c0392f0 0x7f923c03b7b0 secure :-1 s=READY pgs=4146 cs=0 l=1 rev1=1 crypto rx=0x7f9258070b10 tx=0x7f9248001da0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 10 monclient: shutdown 2023-10-30T12:13:03.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:03.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925809b170 msgr2=0x7f92581b7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 --2- 172.21.15.50:0/3737888024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f925809b170 0x7f92581b7680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f925000c7a0 tx=0x7f925000cc60 comp rx=0 tx=0).stop 2023-10-30T12:13:03.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f925e769640 1 -- 172.21.15.50:0/3737888024 reap_dead start 2023-10-30T12:13:03.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f925e769640 1 -- 172.21.15.50:0/3737888024 reap_dead start 2023-10-30T12:13:03.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 shutdown_connections 2023-10-30T12:13:03.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 >> 172.21.15.50:0/3737888024 conn(0x7f92580a3980 msgr2=0x7f9258098440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:03.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 shutdown_connections 2023-10-30T12:13:03.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 -- 172.21.15.50:0/3737888024 wait complete. 2023-10-30T12:13:03.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.372+0000 7f92601cc640 1 librados: shutdown 2023-10-30T12:13:03.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:03.469 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:13:03.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:13:03.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.573+0000 7f681f04b640 10 monclient: get_monmap_and_config 2023-10-30T12:13:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.573+0000 7f681f04b640 10 monclient: build_initial_monmap 2023-10-30T12:13:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.573+0000 7f681f04b640 10 monclient: monmap: 2023-10-30T12:13:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:03.574380+0000 2023-10-30T12:13:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:03.574380+0000 2023-10-30T12:13:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:03.577 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 1 Processor -- start 2023-10-30T12:13:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 1 -- start start 2023-10-30T12:13:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient: init 2023-10-30T12:13:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-10-30T12:13:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6818098520 0x7f681809bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient(hunting): picked mon.noname-b con 0x7f6818098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6818098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f68181379f0 0x7f6818139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient(hunting): picked mon.noname-c con 0x7f68181379f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681809ad30 con 0x7f6818098900 2023-10-30T12:13:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68180993f0 con 0x7f68181379f0 2023-10-30T12:13:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681809c460 con 0x7f6818098520 2023-10-30T12:13:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681f04b640 10 monclient(hunting): authenticate will time out at 1628.757696s 2023-10-30T12:13:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6818098520 0x7f681809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6818098520 0x7f681809bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50444/0 (socket says 172.21.15.50:50444) 2023-10-30T12:13:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 1 -- 172.21.15.50:0/3630969125 learned_addr learned my addr 172.21.15.50:0/3630969125 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 10 monclient(hunting): get_auth_request con 0x7f6818098520 auth_method 0 2023-10-30T12:13:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.574+0000 7f681cde7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681d5e8640 1 --2- 172.21.15.50:0/3630969125 >> v2:172.21.15.118:3300/0 conn(0x7f68181379f0 0x7f6818139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681d5e8640 10 monclient(hunting): get_auth_request con 0x7f68181379f0 auth_method 0 2023-10-30T12:13:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681d5e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681d5e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681d5e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 10 monclient(hunting): handle_auth_done global_id 114758 payload 274 2023-10-30T12:13:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 1 -- 172.21.15.50:0/3630969125 >> v2:172.21.15.118:3300/0 conn(0x7f68181379f0 msgr2=0x7f6818139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 1 --2- 172.21.15.50:0/3630969125 >> v2:172.21.15.118:3300/0 conn(0x7f68181379f0 0x7f6818139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 1 -- 172.21.15.50:0/3630969125 >> v1:172.21.15.118:6789/0 conn(0x7f6818098900 legacy=0x7f6818098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 10 monclient: _finish_hunting 0 2023-10-30T12:13:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 1 monclient: found mon.noname-b 2023-10-30T12:13:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 1 -- 172.21.15.50:0/3630969125 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681809b260 con 0x7f6818098520 2023-10-30T12:13:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.575+0000 7f681cde7640 1 --2- 172.21.15.50:0/3630969125 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6818098520 0x7f681809bf20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f680c009850 tx=0x7f680c02b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4784e981b023c6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 1 -- 172.21.15.50:0/3630969125 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c004230 con 0x7f6818098520 2023-10-30T12:13:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: dump: 2023-10-30T12:13:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.593 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.593 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.593 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.593 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.593 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.594 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.594 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.594 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:03.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:03.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:03.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 1 -- 172.21.15.50:0/3630969125 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c02f050 con 0x7f6818098520 2023-10-30T12:13:03.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 5 monclient: authenticate success, global_id 114758 2023-10-30T12:13:03.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:03.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:03.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:03.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 1 -- 172.21.15.50:0/3630969125 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c003610 con 0x7f6818098520 2023-10-30T12:13:03.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 4 set_mon_vals no callback set 2023-10-30T12:13:03.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:03.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f68177fe640 10 monclient: dump: 2023-10-30T12:13:03.597 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.597 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.597 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.597 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.597 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.597 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.598 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 10 monclient: shutdown 2023-10-30T12:13:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 -- 172.21.15.50:0/3630969125 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6818098520 msgr2=0x7f681809bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 --2- 172.21.15.50:0/3630969125 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6818098520 0x7f681809bf20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f680c009850 tx=0x7f680c02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 -- 172.21.15.50:0/3630969125 shutdown_connections 2023-10-30T12:13:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 --2- 172.21.15.50:0/3630969125 >> v2:172.21.15.118:3300/0 conn(0x7f68181379f0 0x7f6818139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 --2- 172.21.15.50:0/3630969125 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6818098520 0x7f681809bf20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 -- 172.21.15.50:0/3630969125 >> 172.21.15.50:0/3630969125 conn(0x7f68180a05c0 msgr2=0x7f681809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 -- 172.21.15.50:0/3630969125 shutdown_connections 2023-10-30T12:13:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.576+0000 7f681f04b640 1 -- 172.21.15.50:0/3630969125 wait complete. 2023-10-30T12:13:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 10 monclient: build_initial_monmap 2023-10-30T12:13:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 10 monclient: monmap: 2023-10-30T12:13:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:03.602 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 1 librados: starting msgr at 2023-10-30T12:13:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 1 librados: starting objecter 2023-10-30T12:13:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 1 Processor -- start 2023-10-30T12:13:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 1 -- start start 2023-10-30T12:13:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 1 librados: setting wanted keys 2023-10-30T12:13:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 1 librados: calling monclient init 2023-10-30T12:13:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.577+0000 7f681f04b640 10 monclient: init 2023-10-30T12:13:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:13:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6818098520 0x7f68181c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): picked mon.noname-a con 0x7f6818098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6818098900 0x7f6818096530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): picked mon.noname-b con 0x7f6818098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68181379f0 0x7f6818096a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): picked mon.noname-c con 0x7f68181379f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:03.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180927f0 con 0x7f6818098520 2023-10-30T12:13:03.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6818071df0 con 0x7f6818098900 2023-10-30T12:13:03.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:03.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68181377b0 con 0x7f68181379f0 2023-10-30T12:13:03.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:03.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient(hunting): authenticate will time out at 1628.761181s 2023-10-30T12:13:03.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68181379f0 0x7f6818096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6818098520 0x7f68181c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68181379f0 0x7f6818096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50460/0 (socket says 172.21.15.50:50460) 2023-10-30T12:13:03.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 -- 172.21.15.50:0/851241079 learned_addr learned my addr 172.21.15.50:0/851241079 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:03.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): get_auth_request con 0x7f68181379f0 auth_method 0 2023-10-30T12:13:03.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 10 monclient(hunting): get_auth_request con 0x7f6818098520 auth_method 0 2023-10-30T12:13:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:03.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:03.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681cde7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:03.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f6817fff640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6818098900 0x7f6818096530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f6817fff640 10 monclient(hunting): get_auth_request con 0x7f6818098900 auth_method 0 2023-10-30T12:13:03.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f6817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:03.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f6817fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:03.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f6817fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:03.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient(hunting): handle_auth_done global_id 114761 payload 824 2023-10-30T12:13:03.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 -- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6818098900 msgr2=0x7f6818096530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6818098900 0x7f6818096530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 -- 172.21.15.50:0/851241079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6818098520 msgr2=0x7f68181c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6818098520 0x7f68181c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient: _finish_hunting 0 2023-10-30T12:13:03.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 monclient: found mon.noname-c 2023-10-30T12:13:03.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:03.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 -- 172.21.15.50:0/851241079 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c009460 con 0x7f68181379f0 2023-10-30T12:13:03.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f6817fff640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6818098900 0x7f6818096530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:13:03.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681d5e8640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68181379f0 0x7f6818096a70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6810007e40 tx=0x7f681000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 client.?.objecter ms_handle_connect 0x7f68181379f0 2023-10-30T12:13:03.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:03.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 1 -- 172.21.15.50:0/851241079 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810014620 con 0x7f68181379f0 2023-10-30T12:13:03.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:03.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: dump: 2023-10-30T12:13:03.614 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.614 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.614 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.615 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.615 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.615 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.615 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.615 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.616 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.616 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: _finish_auth 0 2023-10-30T12:13:03.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: _check_auth_tickets 2023-10-30T12:13:03.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:03.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 1 -- 172.21.15.50:0/851241079 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681000ee90 con 0x7f68181379f0 2023-10-30T12:13:03.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 5 monclient: authenticate success, global_id 114761 2023-10-30T12:13:03.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient: _renew_subs 2023-10-30T12:13:03.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6818092510 con 0x7f68181379f0 2023-10-30T12:13:03.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:03.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 1 -- 172.21.15.50:0/851241079 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68100149e0 con 0x7f68181379f0 2023-10-30T12:13:03.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:03.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:03.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f68157fa640 10 monclient: dump: 2023-10-30T12:13:03.619 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:03.619 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:03.619 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.619 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:03.619 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:03.619 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:03.620 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:03.620 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:03.620 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:03.620 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:03.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.578+0000 7f6816ffd640 4 set_mon_vals no callback set 2023-10-30T12:13:03.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f681f04b640 10 client.114761.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:03.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f681f04b640 10 monclient: _renew_subs 2023-10-30T12:13:03.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f681f04b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68180909e0 con 0x7f68181379f0 2023-10-30T12:13:03.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f681f04b640 1 librados: init done 2023-10-30T12:13:03.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f68157fa640 1 -- 172.21.15.50:0/851241079 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6810014b40 con 0x7f68181379f0 2023-10-30T12:13:03.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f681f04b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:03.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f681f04b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:03.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.579+0000 7f68157fa640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67fc039320 0x7f67fc03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:03.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f681f04b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68180a15a0 con 0x7f68181379f0 2023-10-30T12:13:03.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f68157fa640 1 -- 172.21.15.50:0/851241079 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(397..397 src has 1..397) v4 ==== 5914+0+0 (secure 0 0 0) 0x7f681004b4a0 con 0x7f68181379f0 2023-10-30T12:13:03.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f68157fa640 10 client.114761.objecter ms_dispatch 0x7f6818000b90 osd_map(397..397 src has 1..397) v4 2023-10-30T12:13:03.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f68157fa640 3 client.114761.objecter handle_osd_map got epochs [397,397] > 0 2023-10-30T12:13:03.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f68157fa640 3 client.114761.objecter handle_osd_map decoding full epoch 397 2023-10-30T12:13:03.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f68157fa640 20 client.114761.objecter dump_active .. 0 homeless 2023-10-30T12:13:03.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f681cde7640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67fc039320 0x7f67fc03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:03.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f681cde7640 10 monclient: get_auth_request con 0x7f67fc039320 auth_method 0 2023-10-30T12:13:03.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f68157fa640 10 client.114761.objecter ms_handle_connect 0x7f67fc039320 2023-10-30T12:13:03.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.580+0000 7f681cde7640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67fc039320 0x7f67fc03b7e0 secure :-1 s=READY pgs=4147 cs=0 l=1 rev1=1 crypto rx=0x7f680c035000 tx=0x7f680c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:03.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.584+0000 7f68157fa640 1 -- 172.21.15.50:0/851241079 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f68100122e0 con 0x7f68181379f0 2023-10-30T12:13:03.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.584+0000 7f68157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:03.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.584+0000 7f68157fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:03.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.829+0000 7f681f04b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:13:03.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.829+0000 7f681f04b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:13:03.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.829+0000 7f681f04b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:03.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.829+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f681804e160 con 0x7f68181379f0 2023-10-30T12:13:03.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f68157fa640 1 -- 172.21.15.50:0/851241079 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f681001bb00 con 0x7f68181379f0 2023-10-30T12:13:03.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f68157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:13:03.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f68157fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:03.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=1,health=11,kv=8,logm=55} crc {auth=413484233,config=1228700967,health=3698204647,kv=4037166911,logm=4272081669}) 2023-10-30T12:13:03.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=4029880599,config=3140514009,health=3698204647,kv=4037166911,logm=516355898}) 2023-10-30T12:13:03.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=1,health=11,kv=8,logm=55} crc {auth=413484233,config=1228700967,health=3698204647,kv=4037166911,logm=4272081669}) 2023-10-30T12:13:03.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.831+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=9,logm=51} crc {auth=4029880599,config=3140514009,health=4209186448,kv=2905431493,logm=3435547958}) 2023-10-30T12:13:03.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.832+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.832+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985607219}) 2023-10-30T12:13:03.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.832+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518777580}) 2023-10-30T12:13:03.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.832+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.832+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985607219}) 2023-10-30T12:13:03.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.832+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=846007624}) 2023-10-30T12:13:03.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.832+0000 7f681f04b640 10 librados: watch_flush enter 2023-10-30T12:13:03.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 10 librados: watch_flush exit 2023-10-30T12:13:03.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 20 client.114761.objecter shutdown clearing up homeless session... 2023-10-30T12:13:03.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 10 client.114761.objecter successfully canceled tick 2023-10-30T12:13:03.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67fc039320 msgr2=0x7f67fc03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67fc039320 0x7f67fc03b7e0 secure :-1 s=READY pgs=4147 cs=0 l=1 rev1=1 crypto rx=0x7f680c035000 tx=0x7f680c008030 comp rx=0 tx=0).stop 2023-10-30T12:13:03.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 10 monclient: shutdown 2023-10-30T12:13:03.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:03.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68181379f0 msgr2=0x7f6818096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:03.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68181379f0 0x7f6818096a70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6810007e40 tx=0x7f681000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 shutdown_connections 2023-10-30T12:13:03.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f67fc039320 0x7f67fc03b7e0 unknown :-1 s=CLOSED pgs=4147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68181379f0 0x7f6818096a70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6818098900 0x7f6818096530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 --2- 172.21.15.50:0/851241079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6818098520 0x7f68181c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:03.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 >> 172.21.15.50:0/851241079 conn(0x7f68180a05c0 msgr2=0x7f6818070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:03.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 shutdown_connections 2023-10-30T12:13:03.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 -- 172.21.15.50:0/851241079 wait complete. 2023-10-30T12:13:03.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f681f04b640 1 librados: shutdown 2023-10-30T12:13:03.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950437093}) 2023-10-30T12:13:03.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=885358969}) 2023-10-30T12:13:03.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950437093}) 2023-10-30T12:13:03.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.833+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681320918}) 2023-10-30T12:13:03.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.835+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.835+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402950015}) 2023-10-30T12:13:03.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.835+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447617656}) 2023-10-30T12:13:03.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.835+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.835+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402950015}) 2023-10-30T12:13:03.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.835+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252170851}) 2023-10-30T12:13:03.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.836+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.836+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369236347}) 2023-10-30T12:13:03.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.836+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3709951641}) 2023-10-30T12:13:03.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.836+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.836+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369236347}) 2023-10-30T12:13:03.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.836+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625771775}) 2023-10-30T12:13:03.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.837+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.837+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878319683}) 2023-10-30T12:13:03.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.837+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4098307297}) 2023-10-30T12:13:03.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.837+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.837+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878319683}) 2023-10-30T12:13:03.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.837+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079699453}) 2023-10-30T12:13:03.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.838+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.838+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605302324}) 2023-10-30T12:13:03.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.838+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885993015}) 2023-10-30T12:13:03.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.838+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.838+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605302324}) 2023-10-30T12:13:03.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.838+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=382846218}) 2023-10-30T12:13:03.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.839+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.839+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860196098}) 2023-10-30T12:13:03.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.839+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930574417}) 2023-10-30T12:13:03.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.839+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.839+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860196098}) 2023-10-30T12:13:03.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.839+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279236772}) 2023-10-30T12:13:03.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.840+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.840+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635842833}) 2023-10-30T12:13:03.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.840+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=42901328}) 2023-10-30T12:13:03.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.840+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.840+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635842833}) 2023-10-30T12:13:03.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.840+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3981308200}) 2023-10-30T12:13:03.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117999359}) 2023-10-30T12:13:03.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1551680922}) 2023-10-30T12:13:03.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117999359}) 2023-10-30T12:13:03.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770406404}) 2023-10-30T12:13:03.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863515249}) 2023-10-30T12:13:03.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937832160}) 2023-10-30T12:13:03.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863515249}) 2023-10-30T12:13:03.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798068096}) 2023-10-30T12:13:03.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696093076}) 2023-10-30T12:13:03.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=344225589}) 2023-10-30T12:13:03.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696093076}) 2023-10-30T12:13:03.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719503924}) 2023-10-30T12:13:03.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126346089}) 2023-10-30T12:13:03.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2121978605}) 2023-10-30T12:13:03.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126346089}) 2023-10-30T12:13:03.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3643666600}) 2023-10-30T12:13:03.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:13:03.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309828567}) 2023-10-30T12:13:03.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628698193}) 2023-10-30T12:13:03.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309828567}) 2023-10-30T12:13:03.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890412152}) 2023-10-30T12:13:03.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939191807}) 2023-10-30T12:13:03.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951121722}) 2023-10-30T12:13:03.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939191807}) 2023-10-30T12:13:03.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027777024}) 2023-10-30T12:13:03.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.846+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.846+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213530622}) 2023-10-30T12:13:03.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.846+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=60093134}) 2023-10-30T12:13:03.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.846+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.846+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213530622}) 2023-10-30T12:13:03.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.846+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3487381414}) 2023-10-30T12:13:03.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556578838}) 2023-10-30T12:13:03.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3850623722}) 2023-10-30T12:13:03.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556578838}) 2023-10-30T12:13:03.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=572727695}) 2023-10-30T12:13:03.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666856718}) 2023-10-30T12:13:03.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4069483609}) 2023-10-30T12:13:03.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666856718}) 2023-10-30T12:13:03.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803364453}) 2023-10-30T12:13:03.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383406138}) 2023-10-30T12:13:03.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769237709}) 2023-10-30T12:13:03.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383406138}) 2023-10-30T12:13:03.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706503686}) 2023-10-30T12:13:03.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892329601}) 2023-10-30T12:13:03.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349815451}) 2023-10-30T12:13:03.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892329601}) 2023-10-30T12:13:03.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666111168}) 2023-10-30T12:13:03.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809108884}) 2023-10-30T12:13:03.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278927417}) 2023-10-30T12:13:03.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809108884}) 2023-10-30T12:13:03.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3372650221}) 2023-10-30T12:13:03.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852038657}) 2023-10-30T12:13:03.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877464737}) 2023-10-30T12:13:03.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852038657}) 2023-10-30T12:13:03.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=802521371}) 2023-10-30T12:13:03.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28110106}) 2023-10-30T12:13:03.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281299297}) 2023-10-30T12:13:03.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28110106}) 2023-10-30T12:13:03.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1669411683}) 2023-10-30T12:13:03.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695451673}) 2023-10-30T12:13:03.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539218201}) 2023-10-30T12:13:03.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695451673}) 2023-10-30T12:13:03.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1591782092}) 2023-10-30T12:13:03.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964942479}) 2023-10-30T12:13:03.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=34258105}) 2023-10-30T12:13:03.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964942479}) 2023-10-30T12:13:03.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2935539266}) 2023-10-30T12:13:03.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90917761}) 2023-10-30T12:13:03.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705276518}) 2023-10-30T12:13:03.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90917761}) 2023-10-30T12:13:03.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243093918}) 2023-10-30T12:13:03.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092035591}) 2023-10-30T12:13:03.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=826044437}) 2023-10-30T12:13:03.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092035591}) 2023-10-30T12:13:03.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3733212107}) 2023-10-30T12:13:03.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057207366}) 2023-10-30T12:13:03.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3490371540}) 2023-10-30T12:13:03.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057207366}) 2023-10-30T12:13:03.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873167064}) 2023-10-30T12:13:03.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199003356}) 2023-10-30T12:13:03.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2133971982}) 2023-10-30T12:13:03.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199003356}) 2023-10-30T12:13:03.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357082524}) 2023-10-30T12:13:03.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303577931}) 2023-10-30T12:13:03.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201071838}) 2023-10-30T12:13:03.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303577931}) 2023-10-30T12:13:03.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3657227509}) 2023-10-30T12:13:03.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459536717}) 2023-10-30T12:13:03.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099877806}) 2023-10-30T12:13:03.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459536717}) 2023-10-30T12:13:03.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2802098893}) 2023-10-30T12:13:03.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306364262}) 2023-10-30T12:13:03.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469956661}) 2023-10-30T12:13:03.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306364262}) 2023-10-30T12:13:03.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396557945}) 2023-10-30T12:13:03.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401221464}) 2023-10-30T12:13:03.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2121077652}) 2023-10-30T12:13:03.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401221464}) 2023-10-30T12:13:03.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416351002}) 2023-10-30T12:13:03.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888586727}) 2023-10-30T12:13:03.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461513810}) 2023-10-30T12:13:03.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888586727}) 2023-10-30T12:13:03.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252598352}) 2023-10-30T12:13:03.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.863+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.863+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139978045}) 2023-10-30T12:13:03.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.863+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229439769}) 2023-10-30T12:13:03.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.863+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.863+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139978045}) 2023-10-30T12:13:03.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.863+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3923999104}) 2023-10-30T12:13:03.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443399599}) 2023-10-30T12:13:03.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2438483582}) 2023-10-30T12:13:03.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443399599}) 2023-10-30T12:13:03.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609230089}) 2023-10-30T12:13:03.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157323656}) 2023-10-30T12:13:03.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=893358441}) 2023-10-30T12:13:03.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157323656}) 2023-10-30T12:13:03.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2343723580}) 2023-10-30T12:13:03.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122091086}) 2023-10-30T12:13:03.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839087194}) 2023-10-30T12:13:03.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122091086}) 2023-10-30T12:13:03.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2008410200}) 2023-10-30T12:13:03.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710196363}) 2023-10-30T12:13:03.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303164315}) 2023-10-30T12:13:03.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710196363}) 2023-10-30T12:13:03.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424405486}) 2023-10-30T12:13:03.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873855796}) 2023-10-30T12:13:03.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495684540}) 2023-10-30T12:13:03.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873855796}) 2023-10-30T12:13:03.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2963949705}) 2023-10-30T12:13:03.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316657875}) 2023-10-30T12:13:03.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171699817}) 2023-10-30T12:13:03.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316657875}) 2023-10-30T12:13:03.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106415864}) 2023-10-30T12:13:03.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148425885}) 2023-10-30T12:13:03.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=593135144}) 2023-10-30T12:13:03.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148425885}) 2023-10-30T12:13:03.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717663542}) 2023-10-30T12:13:03.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181843056}) 2023-10-30T12:13:03.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4195780715}) 2023-10-30T12:13:03.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181843056}) 2023-10-30T12:13:03.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012788250}) 2023-10-30T12:13:03.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467526544}) 2023-10-30T12:13:03.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=638394028}) 2023-10-30T12:13:03.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467526544}) 2023-10-30T12:13:03.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140301481}) 2023-10-30T12:13:03.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242825784}) 2023-10-30T12:13:03.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=392776682}) 2023-10-30T12:13:03.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242825784}) 2023-10-30T12:13:03.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959795746}) 2023-10-30T12:13:03.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301348395}) 2023-10-30T12:13:03.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989260370}) 2023-10-30T12:13:03.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301348395}) 2023-10-30T12:13:03.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1813221382}) 2023-10-30T12:13:03.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738918304}) 2023-10-30T12:13:03.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306280937}) 2023-10-30T12:13:03.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738918304}) 2023-10-30T12:13:03.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309552555}) 2023-10-30T12:13:03.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098520512}) 2023-10-30T12:13:03.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=544615508}) 2023-10-30T12:13:03.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098520512}) 2023-10-30T12:13:03.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2635066576}) 2023-10-30T12:13:03.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428842823}) 2023-10-30T12:13:03.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530471561}) 2023-10-30T12:13:03.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428842823}) 2023-10-30T12:13:03.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348086406}) 2023-10-30T12:13:03.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479004530}) 2023-10-30T12:13:03.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=512432856}) 2023-10-30T12:13:03.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479004530}) 2023-10-30T12:13:03.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1012531994}) 2023-10-30T12:13:03.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446490327}) 2023-10-30T12:13:03.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1937134529}) 2023-10-30T12:13:03.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446490327}) 2023-10-30T12:13:03.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125872604}) 2023-10-30T12:13:03.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735642393}) 2023-10-30T12:13:03.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610602480}) 2023-10-30T12:13:03.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735642393}) 2023-10-30T12:13:03.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=848779804}) 2023-10-30T12:13:03.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885225676}) 2023-10-30T12:13:03.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=224157144}) 2023-10-30T12:13:03.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885225676}) 2023-10-30T12:13:03.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2989441626}) 2023-10-30T12:13:03.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274810100}) 2023-10-30T12:13:03.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4094921580}) 2023-10-30T12:13:03.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274810100}) 2023-10-30T12:13:03.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767876330}) 2023-10-30T12:13:03.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743154081}) 2023-10-30T12:13:03.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2524805123}) 2023-10-30T12:13:03.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743154081}) 2023-10-30T12:13:03.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909697798}) 2023-10-30T12:13:03.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430887518}) 2023-10-30T12:13:03.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024525225}) 2023-10-30T12:13:03.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430887518}) 2023-10-30T12:13:03.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3161434726}) 2023-10-30T12:13:03.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464350507}) 2023-10-30T12:13:03.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047797400}) 2023-10-30T12:13:03.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464350507}) 2023-10-30T12:13:03.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562025712}) 2023-10-30T12:13:03.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275647272}) 2023-10-30T12:13:03.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867832559}) 2023-10-30T12:13:03.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275647272}) 2023-10-30T12:13:03.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1183034705}) 2023-10-30T12:13:03.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369621812}) 2023-10-30T12:13:03.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3624469238}) 2023-10-30T12:13:03.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369621812}) 2023-10-30T12:13:03.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3344733125}) 2023-10-30T12:13:03.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990851635}) 2023-10-30T12:13:03.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=217039468}) 2023-10-30T12:13:03.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990851635}) 2023-10-30T12:13:03.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556403407}) 2023-10-30T12:13:03.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3816365308}) 2023-10-30T12:13:03.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3270049559}) 2023-10-30T12:13:03.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3816365308}) 2023-10-30T12:13:03.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557417424}) 2023-10-30T12:13:03.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821577982}) 2023-10-30T12:13:03.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=856085728}) 2023-10-30T12:13:03.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821577982}) 2023-10-30T12:13:03.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299805747}) 2023-10-30T12:13:03.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285203559}) 2023-10-30T12:13:03.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423564605}) 2023-10-30T12:13:03.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285203559}) 2023-10-30T12:13:03.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417841524}) 2023-10-30T12:13:03.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=2340431821,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2120918898,mon_config_key=3217606506,monmap=3324057927,osd_metadata=234854199}) 2023-10-30T12:13:03.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=4} crc {logm=922441344,mdsmap=1228700967,mgr=1715906815,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2120918898,mon_config_key=712065602,monmap=3324057927,osd_metadata=1330037187}) 2023-10-30T12:13:03.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=2340431821,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2120918898,mon_config_key=3217606506,monmap=3324057927,osd_metadata=234854199}) 2023-10-30T12:13:03.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3} crc {logm=3408990317,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3301977181,mon_config_key=3217606506,monmap=3324057927}) 2023-10-30T12:13:03.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=33,osdmap=62} crc {osd_metadata=1528669153,osd_pg_creating=3108181003,osd_snap=3808724640,osdmap=4016771425}) 2023-10-30T12:13:03.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=60} crc {osd_metadata=3009730626,osd_pg_creating=3108181003,osd_snap=3350854959,osdmap=4188415082}) 2023-10-30T12:13:03.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=33,osdmap=62} crc {osd_metadata=1528669153,osd_pg_creating=3108181003,osd_snap=3808724640,osdmap=4016771425}) 2023-10-30T12:13:03.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=33,osdmap=63} crc {osd_metadata=3927394902,osd_pg_creating=3108181003,osd_snap=253313294,osdmap=3815363624}) 2023-10-30T12:13:03.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=957213781}) 2023-10-30T12:13:03.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3295972042}) 2023-10-30T12:13:03.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=957213781}) 2023-10-30T12:13:03.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1134193233}) 2023-10-30T12:13:03.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4180427012}) 2023-10-30T12:13:03.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1399806923}) 2023-10-30T12:13:03.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4180427012}) 2023-10-30T12:13:03.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1919612895}) 2023-10-30T12:13:03.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1192810621}) 2023-10-30T12:13:03.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3919791689}) 2023-10-30T12:13:03.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1192810621}) 2023-10-30T12:13:03.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3264278148}) 2023-10-30T12:13:03.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3430548474}) 2023-10-30T12:13:03.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=810310457}) 2023-10-30T12:13:03.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3430548474}) 2023-10-30T12:13:03.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1832024321}) 2023-10-30T12:13:03.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=390870193}) 2023-10-30T12:13:03.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1284577582}) 2023-10-30T12:13:03.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=390870193}) 2023-10-30T12:13:03.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2937823947}) 2023-10-30T12:13:03.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=1264333663}) 2023-10-30T12:13:03.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=2834301551}) 2023-10-30T12:13:03.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:03.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=1264333663}) 2023-10-30T12:13:03.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:03.902+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=17} crc {osdmap=99459092}) 2023-10-30T12:13:04.776 INFO:tasks.workunit.client.0.smithi118.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18030 ms) 2023-10-30T12:13:04.777 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-10-30T12:13:04.777 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (36965 ms) 2023-10-30T12:13:04.777 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-10-30T12:13:04.777 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3014 ms) 2023-10-30T12:13:04.777 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-10-30T12:13:04.778 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.778 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2982 ms) 2023-10-30T12:13:04.778 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-10-30T12:13:04.778 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3005 ms) 2023-10-30T12:13:04.778 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-10-30T12:13:04.779 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.779 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3026 ms) 2023-10-30T12:13:04.779 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-10-30T12:13:04.779 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.779 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (9032 ms) 2023-10-30T12:13:04.779 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-10-30T12:13:04.780 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.780 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (66322 ms) 2023-10-30T12:13:04.780 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-10-30T12:13:04.780 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.780 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17051 ms) 2023-10-30T12:13:04.781 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-10-30T12:13:04.781 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.781 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4015 ms) 2023-10-30T12:13:04.781 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-10-30T12:13:04.781 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.782 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3018 ms) 2023-10-30T12:13:04.782 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-10-30T12:13:04.782 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.782 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24185 ms) 2023-10-30T12:13:04.782 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-10-30T12:13:04.783 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.783 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15027 ms) 2023-10-30T12:13:04.783 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-10-30T12:13:04.783 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.783 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4984 ms) 2023-10-30T12:13:04.784 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-10-30T12:13:04.784 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.784 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4019 ms) 2023-10-30T12:13:04.784 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-10-30T12:13:04.784 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.784 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: waiting for scrubs... 2023-10-30T12:13:04.785 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: done waiting 2023-10-30T12:13:04.785 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (39063 ms) 2023-10-30T12:13:04.785 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-10-30T12:13:04.785 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.785 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9931 ms) 2023-10-30T12:13:04.786 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-10-30T12:13:04.789 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.789 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8999 ms) 2023-10-30T12:13:04.790 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-10-30T12:13:04.790 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:13:04.790 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (14576 ms) 2023-10-30T12:13:04.790 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-10-30T12:13:04.862 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:13:04.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.972+0000 7f9b76d33640 10 monclient: get_monmap_and_config 2023-10-30T12:13:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.972+0000 7f9b76d33640 10 monclient: build_initial_monmap 2023-10-30T12:13:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.972+0000 7f9b76d33640 10 monclient: monmap: 2023-10-30T12:13:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:04.973283+0000 2023-10-30T12:13:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:04.973283+0000 2023-10-30T12:13:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:04.976 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:04.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.972+0000 7f9b76d33640 1 Processor -- start 2023-10-30T12:13:04.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 1 -- start start 2023-10-30T12:13:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient: init 2023-10-30T12:13:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T12:13:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient(hunting): picked mon.noname-b con 0x7f9b700a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9b70099a40 0x7f9b70132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient(hunting): picked mon.noname-c con 0x7f9b70099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b700a2350 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b70133120 con 0x7f9b700a2350 2023-10-30T12:13:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b700a3d70 con 0x7f9b70099a40 2023-10-30T12:13:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b700a4f40 con 0x7f9b700a4b60 2023-10-30T12:13:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b76d33640 10 monclient(hunting): authenticate will time out at 1630.156505s 2023-10-30T12:13:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b752d0640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9b700a2350 0x7f9b700a27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47694/0 (socket says 172.21.15.50:47694) 2023-10-30T12:13:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b752d0640 1 -- 172.21.15.50:0/846828675 learned_addr learned my addr 172.21.15.50:0/846828675 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): get_auth_request con 0x7f9b700a4b60 auth_method 0 2023-10-30T12:13:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient(hunting): handle_auth_done global_id 114764 payload 274 2023-10-30T12:13:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b6ffff640 1 --2- 172.21.15.50:0/846828675 >> v2:172.21.15.118:3300/0 conn(0x7f9b70099a40 0x7f9b70132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 1 -- 172.21.15.50:0/846828675 >> v2:172.21.15.118:3300/0 conn(0x7f9b70099a40 msgr2=0x7f9b70132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 1 --2- 172.21.15.50:0/846828675 >> v2:172.21.15.118:3300/0 conn(0x7f9b70099a40 0x7f9b70132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 1 -- 172.21.15.50:0/846828675 >> v1:172.21.15.50:6789/0 conn(0x7f9b700a2350 legacy=0x7f9b700a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient: _finish_hunting 0 2023-10-30T12:13:04.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 1 monclient: found mon.noname-b 2023-10-30T12:13:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b6f7fe640 1 -- 172.21.15.50:0/846828675 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003133379 0 0) 0x7f9b70133120 con 0x7f9b700a2350 2023-10-30T12:13:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.973+0000 7f9b74acf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b74acf640 1 -- 172.21.15.50:0/846828675 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b700a4150 con 0x7f9b700a4b60 2023-10-30T12:13:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6ffff640 10 monclient: get_auth_request con 0x7f9b70099a40 auth_method 0 2023-10-30T12:13:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6ffff640 1 --2- 172.21.15.50:0/846828675 >> v2:172.21.15.118:3300/0 conn(0x7f9b70099a40 0x7f9b70132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:13:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b74acf640 1 --2- 172.21.15.50:0/846828675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701326a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009d90 tx=0x7f9b6802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddaf3f89c22f6fc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 1 -- 172.21.15.50:0/846828675 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6802cb30 con 0x7f9b700a4b60 2023-10-30T12:13:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: dump: 2023-10-30T12:13:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:04.995 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 1 -- 172.21.15.50:0/846828675 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6802cc90 con 0x7f9b700a4b60 2023-10-30T12:13:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b76d33640 5 monclient: authenticate success, global_id 114764 2023-10-30T12:13:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b76d33640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b76d33640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 1 -- 172.21.15.50:0/846828675 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68034440 con 0x7f9b700a4b60 2023-10-30T12:13:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b76d33640 4 set_mon_vals no callback set 2023-10-30T12:13:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:04.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.974+0000 7f9b6f7fe640 10 monclient: dump: 2023-10-30T12:13:04.998 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:04.998 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:04.998 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:04.998 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:04.999 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 10 monclient: shutdown 2023-10-30T12:13:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 -- 172.21.15.50:0/846828675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 msgr2=0x7f9b701326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 --2- 172.21.15.50:0/846828675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701326a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009d90 tx=0x7f9b6802bd80 comp rx=0 tx=0).stop 2023-10-30T12:13:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 -- 172.21.15.50:0/846828675 shutdown_connections 2023-10-30T12:13:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 --2- 172.21.15.50:0/846828675 >> v2:172.21.15.118:3300/0 conn(0x7f9b70099a40 0x7f9b70132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 --2- 172.21.15.50:0/846828675 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701326a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 -- 172.21.15.50:0/846828675 >> 172.21.15.50:0/846828675 conn(0x7f9b700a8060 msgr2=0x7f9b700a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 -- 172.21.15.50:0/846828675 shutdown_connections 2023-10-30T12:13:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.975+0000 7f9b76d33640 1 -- 172.21.15.50:0/846828675 wait complete. 2023-10-30T12:13:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 10 monclient: build_initial_monmap 2023-10-30T12:13:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 10 monclient: monmap: 2023-10-30T12:13:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:05.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 1 librados: starting msgr at 2023-10-30T12:13:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 1 librados: starting objecter 2023-10-30T12:13:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 1 Processor -- start 2023-10-30T12:13:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 1 -- start start 2023-10-30T12:13:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 1 librados: setting wanted keys 2023-10-30T12:13:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 1 librados: calling monclient init 2023-10-30T12:13:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.976+0000 7f9b76d33640 10 monclient: init 2023-10-30T12:13:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:13:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b700a2350 0x7f9b701ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): picked mon.noname-a con 0x7f9b700a2350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): picked mon.noname-c con 0x7f9b700a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b70121590 0x7f9b70121a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): picked mon.noname-b con 0x7f9b70121590 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b701253b0 con 0x7f9b700a2350 2023-10-30T12:13:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b70125600 con 0x7f9b70121590 2023-10-30T12:13:05.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b700a07f0 con 0x7f9b700a4b60 2023-10-30T12:13:05.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:05.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b76d33640 10 monclient(hunting): authenticate will time out at 1630.160247s 2023-10-30T12:13:05.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b700a2350 0x7f9b701ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50480/0 (socket says 172.21.15.50:50480) 2023-10-30T12:13:05.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b700a2350 0x7f9b701ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57070/0 (socket says 172.21.15.50:57070) 2023-10-30T12:13:05.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 -- 172.21.15.50:0/2280118169 learned_addr learned my addr 172.21.15.50:0/2280118169 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:05.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): get_auth_request con 0x7f9b700a4b60 auth_method 0 2023-10-30T12:13:05.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 10 monclient(hunting): get_auth_request con 0x7f9b700a2350 auth_method 0 2023-10-30T12:13:05.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b70121590 0x7f9b70121a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b74acf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 10 monclient(hunting): get_auth_request con 0x7f9b70121590 auth_method 0 2023-10-30T12:13:05.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b752d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_done global_id 114767 payload 824 2023-10-30T12:13:05.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 -- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b70121590 msgr2=0x7f9b70121a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b70121590 0x7f9b70121a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 -- 172.21.15.50:0/2280118169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b700a2350 msgr2=0x7f9b701ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b700a2350 0x7f9b701ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 monclient: found mon.noname-c 2023-10-30T12:13:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 -- 172.21.15.50:0/2280118169 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b68009a70 con 0x7f9b700a4b60 2023-10-30T12:13:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6ffff640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701caac0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c820 tx=0x7f9b6000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6d7fa640 10 client.?.objecter ms_handle_connect 0x7f9b700a4b60 2023-10-30T12:13:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.977+0000 7f9b6d7fa640 1 -- 172.21.15.50:0/2280118169 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b60007a30 con 0x7f9b700a4b60 2023-10-30T12:13:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: dump: 2023-10-30T12:13:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:13:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:13:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 1 -- 172.21.15.50:0/2280118169 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b60007b90 con 0x7f9b700a4b60 2023-10-30T12:13:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 1 -- 172.21.15.50:0/2280118169 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6000ce10 con 0x7f9b700a4b60 2023-10-30T12:13:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 10 monclient: dump: 2023-10-30T12:13:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.023 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 5 monclient: authenticate success, global_id 114767 2023-10-30T12:13:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 10 monclient: _renew_subs 2023-10-30T12:13:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6effd640 4 set_mon_vals no callback set 2023-10-30T12:13:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b700a18d0 con 0x7f9b700a4b60 2023-10-30T12:13:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 10 client.114767.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 10 monclient: _renew_subs 2023-10-30T12:13:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b70123230 con 0x7f9b700a4b60 2023-10-30T12:13:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b76d33640 1 librados: init done 2023-10-30T12:13:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.978+0000 7f9b6d7fa640 1 -- 172.21.15.50:0/2280118169 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9b600113f0 con 0x7f9b700a4b60 2023-10-30T12:13:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b76d33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b76d33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b76d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b70124e80 con 0x7f9b700a4b60 2023-10-30T12:13:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b6d7fa640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b50039320 0x7f9b5003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b6d7fa640 1 -- 172.21.15.50:0/2280118169 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(399..399 src has 1..399) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f9b6004adc0 con 0x7f9b700a4b60 2023-10-30T12:13:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b6d7fa640 10 client.114767.objecter ms_dispatch 0x7f9b70000b90 osd_map(399..399 src has 1..399) v4 2023-10-30T12:13:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b6d7fa640 3 client.114767.objecter handle_osd_map got epochs [399,399] > 0 2023-10-30T12:13:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b6d7fa640 3 client.114767.objecter handle_osd_map decoding full epoch 399 2023-10-30T12:13:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b6d7fa640 20 client.114767.objecter dump_active .. 0 homeless 2023-10-30T12:13:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b74acf640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b50039320 0x7f9b5003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b74acf640 10 monclient: get_auth_request con 0x7f9b50039320 auth_method 0 2023-10-30T12:13:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b6d7fa640 10 client.114767.objecter ms_handle_connect 0x7f9b50039320 2023-10-30T12:13:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.979+0000 7f9b74acf640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b50039320 0x7f9b5003b7e0 secure :-1 s=READY pgs=4148 cs=0 l=1 rev1=1 crypto rx=0x7f9b68036000 tx=0x7f9b68030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.983+0000 7f9b6d7fa640 1 -- 172.21.15.50:0/2280118169 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b60021300 con 0x7f9b700a4b60 2023-10-30T12:13:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.983+0000 7f9b6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:04.983+0000 7f9b6d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.225+0000 7f9b76d33640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.225+0000 7f9b76d33640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.225+0000 7f9b76d33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.225+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9b70125740 con 0x7f9b700a4b60 2023-10-30T12:13:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.226+0000 7f9b6d7fa640 1 -- 172.21.15.50:0/2280118169 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f9b70125740 con 0x7f9b700a4b60 2023-10-30T12:13:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.226+0000 7f9b6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.226+0000 7f9b6d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:05.229 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 10 librados: watch_flush enter 2023-10-30T12:13:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 10 librados: watch_flush exit 2023-10-30T12:13:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 20 client.114767.objecter shutdown clearing up homeless session... 2023-10-30T12:13:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 10 client.114767.objecter successfully canceled tick 2023-10-30T12:13:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b50039320 msgr2=0x7f9b5003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b50039320 0x7f9b5003b7e0 secure :-1 s=READY pgs=4148 cs=0 l=1 rev1=1 crypto rx=0x7f9b68036000 tx=0x7f9b68030000 comp rx=0 tx=0).stop 2023-10-30T12:13:05.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 10 monclient: shutdown 2023-10-30T12:13:05.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 msgr2=0x7f9b701caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701caac0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c820 tx=0x7f9b6000cce0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 shutdown_connections 2023-10-30T12:13:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b50039320 0x7f9b5003b7e0 unknown :-1 s=CLOSED pgs=4148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b70121590 0x7f9b70121a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b700a4b60 0x7f9b701caac0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 --2- 172.21.15.50:0/2280118169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b700a2350 0x7f9b701ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 >> 172.21.15.50:0/2280118169 conn(0x7f9b700a8060 msgr2=0x7f9b700a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 shutdown_connections 2023-10-30T12:13:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 -- 172.21.15.50:0/2280118169 wait complete. 2023-10-30T12:13:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.228+0000 7f9b76d33640 1 librados: shutdown 2023-10-30T12:13:05.239 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:13:05.240 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:13:05.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:13:05.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:13:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.345+0000 7fdca977c640 10 monclient: get_monmap_and_config 2023-10-30T12:13:05.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.345+0000 7fdca977c640 10 monclient: build_initial_monmap 2023-10-30T12:13:05.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.345+0000 7fdca977c640 10 monclient: monmap: 2023-10-30T12:13:05.347 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:05.347 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:05.346663+0000 2023-10-30T12:13:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:05.346663+0000 2023-10-30T12:13:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 1 Processor -- start 2023-10-30T12:13:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 1 -- start start 2023-10-30T12:13:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient: init 2023-10-30T12:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T12:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdca40a2ae0 0x7fdca40a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient(hunting): picked mon.noname-c con 0x7fdca40a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdca40a2ec0 0x7fdca41356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient(hunting): picked mon.noname-a con 0x7fdca40a2ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdca4135c20 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40a4190 con 0x7fdca4135c20 2023-10-30T12:13:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca40a2990 con 0x7fdca40a2ec0 2023-10-30T12:13:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca40a1630 con 0x7fdca40a2ae0 2023-10-30T12:13:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdca977c640 10 monclient(hunting): authenticate will time out at 1630.529788s 2023-10-30T12:13:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdc9a7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdca40a2ec0 0x7fdca41356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdc9a7fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdca40a2ec0 0x7fdca41356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57080/0 (socket says 172.21.15.50:57080) 2023-10-30T12:13:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/2905208268 learned_addr learned my addr 172.21.15.50:0/2905208268 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdc9a7fc640 10 monclient(hunting): get_auth_request con 0x7fdca40a2ec0 auth_method 0 2023-10-30T12:13:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdc9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.346+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdca2ffd640 1 --2- 172.21.15.50:0/2905208268 >> v2:172.21.15.118:3300/0 conn(0x7fdca40a2ae0 0x7fdca40a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_done global_id 174144 payload 274 2023-10-30T12:13:05.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/2905208268 >> v2:172.21.15.118:3300/0 conn(0x7fdca40a2ae0 msgr2=0x7fdca40a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:05.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 1 --2- 172.21.15.50:0/2905208268 >> v2:172.21.15.118:3300/0 conn(0x7fdca40a2ae0 0x7fdca40a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/2905208268 >> v1:172.21.15.118:6789/0 conn(0x7fdca4135c20 legacy=0x7fdca413ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:05.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:13:05.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 1 monclient: found mon.noname-a 2023-10-30T12:13:05.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:05.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:05.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/2905208268 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca409a2e0 con 0x7fdca40a2ec0 2023-10-30T12:13:05.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdca2ffd640 10 monclient: get_auth_request con 0x7fdca40a2ae0 auth_method 0 2023-10-30T12:13:05.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdca2ffd640 1 --2- 172.21.15.50:0/2905208268 >> v2:172.21.15.118:3300/0 conn(0x7fdca40a2ae0 0x7fdca40a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.347+0000 7fdc9a7fc640 1 --2- 172.21.15.50:0/2905208268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca41356e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdca40a3c80 tx=0x7fdc8c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a715f8d08ab121f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 1 -- 172.21.15.50:0/2905208268 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c031750 con 0x7fdca40a2ec0 2023-10-30T12:13:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: dump: 2023-10-30T12:13:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.368 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.368 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.368 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.368 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.368 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: _finish_auth 0 2023-10-30T12:13:05.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:13:05.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:05.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 1 -- 172.21.15.50:0/2905208268 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c0318b0 con 0x7fdca40a2ec0 2023-10-30T12:13:05.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 5 monclient: authenticate success, global_id 174144 2023-10-30T12:13:05.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:05.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:05.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:05.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 4 set_mon_vals no callback set 2023-10-30T12:13:05.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 1 -- 172.21.15.50:0/2905208268 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c031c40 con 0x7fdca40a2ec0 2023-10-30T12:13:05.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:05.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca27fc640 10 monclient: dump: 2023-10-30T12:13:05.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.371 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.371 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.372 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.372 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.372 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.372 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 10 monclient: shutdown 2023-10-30T12:13:05.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 -- 172.21.15.50:0/2905208268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 msgr2=0x7fdca41356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 --2- 172.21.15.50:0/2905208268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca41356e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdca40a3c80 tx=0x7fdc8c02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 -- 172.21.15.50:0/2905208268 shutdown_connections 2023-10-30T12:13:05.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 --2- 172.21.15.50:0/2905208268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca41356e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 --2- 172.21.15.50:0/2905208268 >> v2:172.21.15.118:3300/0 conn(0x7fdca40a2ae0 0x7fdca40a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 -- 172.21.15.50:0/2905208268 >> 172.21.15.50:0/2905208268 conn(0x7fdca40a80b0 msgr2=0x7fdca40a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 -- 172.21.15.50:0/2905208268 shutdown_connections 2023-10-30T12:13:05.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.348+0000 7fdca977c640 1 -- 172.21.15.50:0/2905208268 wait complete. 2023-10-30T12:13:05.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 10 monclient: build_initial_monmap 2023-10-30T12:13:05.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 10 monclient: monmap: 2023-10-30T12:13:05.375 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:05.375 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:05.376 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:05.376 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:05.376 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:05.376 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.376 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:05.377 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:05.377 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:05.377 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 1 librados: starting msgr at 2023-10-30T12:13:05.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 1 librados: starting objecter 2023-10-30T12:13:05.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 1 Processor -- start 2023-10-30T12:13:05.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 1 -- start start 2023-10-30T12:13:05.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 1 librados: setting wanted keys 2023-10-30T12:13:05.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 1 librados: calling monclient init 2023-10-30T12:13:05.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.349+0000 7fdca977c640 10 monclient: init 2023-10-30T12:13:05.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:05.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:13:05.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca40a2ae0 0x7fdca41c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): picked mon.noname-c con 0x7fdca40a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:05.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca4131240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): picked mon.noname-a con 0x7fdca40a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:05.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdca4135c20 0x7fdca4131780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): picked mon.noname-b con 0x7fdca4135c20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:05.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca41ca430 con 0x7fdca40a2ec0 2023-10-30T12:13:05.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4132500 con 0x7fdca4135c20 2023-10-30T12:13:05.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca411e0a0 con 0x7fdca40a2ae0 2023-10-30T12:13:05.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:05.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca977c640 10 monclient(hunting): authenticate will time out at 1630.533193s 2023-10-30T12:13:05.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca4131240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca40a2ae0 0x7fdca41c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca4131240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57086/0 (socket says 172.21.15.50:57086) 2023-10-30T12:13:05.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/3977531649 learned_addr learned my addr 172.21.15.50:0/3977531649 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:05.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): get_auth_request con 0x7fdca40a2ec0 auth_method 0 2023-10-30T12:13:05.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 10 monclient(hunting): get_auth_request con 0x7fdca40a2ae0 auth_method 0 2023-10-30T12:13:05.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient(hunting): handle_auth_done global_id 174147 payload 824 2023-10-30T12:13:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca40a2ae0 msgr2=0x7fdca41c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca40a2ae0 0x7fdca41c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca37fe640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdca4135c20 0x7fdca4131780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdca4135c20 msgr2=0x7fdca4131780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdca4135c20 0x7fdca4131780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:13:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 monclient: found mon.noname-a 2023-10-30T12:13:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:05.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:05.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 -- 172.21.15.50:0/3977531649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c0094d0 con 0x7fdca40a2ec0 2023-10-30T12:13:05.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdca2ffd640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca40a2ae0 0x7fdca41c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:05.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9a7fc640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca4131240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000c00 tx=0x7fdc8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9bfff640 10 client.?.objecter ms_handle_connect 0x7fdca40a2ec0 2023-10-30T12:13:05.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:05.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.350+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c033c80 con 0x7fdca40a2ec0 2023-10-30T12:13:05.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:05.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: dump: 2023-10-30T12:13:05.389 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.389 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.389 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.389 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.389 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.389 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.390 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.390 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.390 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.390 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: _finish_auth 0 2023-10-30T12:13:05.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:13:05.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:05.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c033de0 con 0x7fdca40a2ec0 2023-10-30T12:13:05.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 5 monclient: authenticate success, global_id 174147 2023-10-30T12:13:05.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:05.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 10 monclient: _renew_subs 2023-10-30T12:13:05.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c03c620 con 0x7fdca40a2ec0 2023-10-30T12:13:05.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:05.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca411e3c0 con 0x7fdca40a2ec0 2023-10-30T12:13:05.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca1ffb640 4 set_mon_vals no callback set 2023-10-30T12:13:05.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:05.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdc9bfff640 10 monclient: dump: 2023-10-30T12:13:05.393 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.393 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.393 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.395 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 10 client.174147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 10 monclient: _renew_subs 2023-10-30T12:13:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4131cc0 con 0x7fdca40a2ec0 2023-10-30T12:13:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 1 librados: init done 2023-10-30T12:13:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a0190 0x7fdca409d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.351+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4073ff0 con 0x7fdca40a0190 2023-10-30T12:13:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a0190 0x7fdca409d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient: get_auth_request con 0x7fdca40a0190 auth_method 0 2023-10-30T12:13:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdc8c03a040 con 0x7fdca40a2ec0 2023-10-30T12:13:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc840392f0 0x7fdc8403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_done global_id 174150 payload 824 2023-10-30T12:13:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(399..399 src has 1..399) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fdc8c06c8c0 con 0x7fdca40a2ec0 2023-10-30T12:13:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 1 -- 172.21.15.50:0/3977531649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdca409a5c0 con 0x7fdca40a0190 2023-10-30T12:13:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 client.174147.objecter ms_dispatch 0x7fdca4000b90 osd_map(399..399 src has 1..399) v4 2023-10-30T12:13:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 3 client.174147.objecter handle_osd_map got epochs [399,399] > 0 2023-10-30T12:13:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 3 client.174147.objecter handle_osd_map decoding full epoch 399 2023-10-30T12:13:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 20 client.174147.objecter dump_active .. 0 homeless 2023-10-30T12:13:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca37fe640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc840392f0 0x7fdc8403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 client.174147.objecter ms_handle_connect 0x7fdca40a0190 2023-10-30T12:13:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 client.174147.objecter resend_mon_ops 2023-10-30T12:13:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca2ffd640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a0190 0x7fdca409d8b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400fca0 tx=0x7fdc9400d590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdca37fe640 10 monclient: get_auth_request con 0x7fdc840392f0 auth_method 0 2023-10-30T12:13:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94013640 con 0x7fdca40a0190 2023-10-30T12:13:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 monclient: dump: 2023-10-30T12:13:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fdc9400c040 con 0x7fdca40a0190 2023-10-30T12:13:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a0190 msgr2=0x7fdca409d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a0190 0x7fdca409d8b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400fca0 tx=0x7fdc9400d590 comp rx=0 tx=0).stop 2023-10-30T12:13:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a0190 msgr2=0x7fdca409d8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.352+0000 7fdc9bfff640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a0190 0x7fdca409d8b0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400fca0 tx=0x7fdc9400d590 comp rx=0 tx=0).stop 2023-10-30T12:13:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.353+0000 7fdc9bfff640 10 client.174147.objecter ms_handle_connect 0x7fdc840392f0 2023-10-30T12:13:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.353+0000 7fdca37fe640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc840392f0 0x7fdc8403b7b0 secure :-1 s=READY pgs=4149 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c007fd0 tx=0x7fdc9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca977c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca977c640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca409ddf0 0x7fdca409e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca977c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4099b40 con 0x7fdca409ddf0 2023-10-30T12:13:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca409ddf0 0x7fdca409e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient: get_auth_request con 0x7fdca409ddf0 auth_method 0 2023-10-30T12:13:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 10 monclient(hunting): handle_auth_done global_id 174153 payload 824 2023-10-30T12:13:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 1 -- 172.21.15.50:0/3977531649 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdca406b060 con 0x7fdca409ddf0 2023-10-30T12:13:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdc9bfff640 10 client.174147.objecter ms_handle_connect 0x7fdca409ddf0 2023-10-30T12:13:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdca2ffd640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca409ddf0 0x7fdca409e200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400fca0 tx=0x7fdc940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdc9bfff640 10 client.174147.objecter resend_mon_ops 2023-10-30T12:13:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9401bb40 con 0x7fdca409ddf0 2023-10-30T12:13:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdc9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdc9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.359+0000 7fdc9bfff640 10 monclient: dump: 2023-10-30T12:13:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.413 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7fdc9401bca0 con 0x7fdca409ddf0 2023-10-30T12:13:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdc9bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdc9bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca409ddf0 msgr2=0x7fdca409e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdc9bfff640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca409ddf0 0x7fdca409e200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400fca0 tx=0x7fdc940046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdc9bfff640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca409ddf0 msgr2=0x7fdca409e200 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdc9bfff640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca409ddf0 0x7fdca409e200 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400fca0 tx=0x7fdc940046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.414 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:05.415 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:05.415 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:05.415 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:13:05.415 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 124, 2023-10-30T12:13:05.416 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:05.416 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:05.416 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:05.416 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:05.416 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.417 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:13:05.417 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:05.417 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:05.417 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:05.417 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.417 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.418 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.418 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.418 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.418 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.418 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.419 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.419 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.419 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.419 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.419 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:05.420 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:05.420 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.420 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.420 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.420 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.420 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.421 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.421 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.421 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.421 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.421 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.422 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.422 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:05.422 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:05.422 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:05.423 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:05.423 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:05.423 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:05.423 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:05.423 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:05.424 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:05.424 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:05.424 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:05.424 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:05.424 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:05.424 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:05.425 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:05.425 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:05.425 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:05.425 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.425 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.426 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.426 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.426 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.427 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.427 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.427 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.427 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.427 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:13:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:05.445 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:05.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 10 librados: watch_flush enter 2023-10-30T12:13:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 10 librados: watch_flush exit 2023-10-30T12:13:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 20 client.174147.objecter shutdown clearing up homeless session... 2023-10-30T12:13:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 10 client.174147.objecter successfully canceled tick 2023-10-30T12:13:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc840392f0 msgr2=0x7fdc8403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc840392f0 0x7fdc8403b7b0 secure :-1 s=READY pgs=4149 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c007fd0 tx=0x7fdc9c009040 comp rx=0 tx=0).stop 2023-10-30T12:13:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 10 monclient: shutdown 2023-10-30T12:13:05.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 msgr2=0x7fdca4131240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 --2- 172.21.15.50:0/3977531649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca40a2ec0 0x7fdca4131240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000c00 tx=0x7fdc8c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca37fe640 1 -- 172.21.15.50:0/3977531649 reap_dead start 2023-10-30T12:13:05.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca37fe640 1 -- 172.21.15.50:0/3977531649 reap_dead start 2023-10-30T12:13:05.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 shutdown_connections 2023-10-30T12:13:05.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 >> 172.21.15.50:0/3977531649 conn(0x7fdca40a80b0 msgr2=0x7fdca40a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 shutdown_connections 2023-10-30T12:13:05.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 -- 172.21.15.50:0/3977531649 wait complete. 2023-10-30T12:13:05.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.360+0000 7fdca977c640 1 librados: shutdown 2023-10-30T12:13:05.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:05.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:13:05.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:13:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.556+0000 7f465396c640 10 monclient: get_monmap_and_config 2023-10-30T12:13:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.556+0000 7f465396c640 10 monclient: build_initial_monmap 2023-10-30T12:13:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.556+0000 7f465396c640 10 monclient: monmap: 2023-10-30T12:13:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:05.557139+0000 2023-10-30T12:13:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:05.557139+0000 2023-10-30T12:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:05.560 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:05.560 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.556+0000 7f465396c640 1 Processor -- start 2023-10-30T12:13:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.556+0000 7f465396c640 1 -- start start 2023-10-30T12:13:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.556+0000 7f465396c640 10 monclient: init 2023-10-30T12:13:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:13:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ae0 0x7f464c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): picked mon.noname-b con 0x7f464c0a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f464c0a2ec0 0x7f464c1356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): picked mon.noname-c con 0x7f464c0a2ec0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f464c0a32e0 0x7f464c135c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): picked mon.noname-a con 0x7f464c0a32e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c0a2990 con 0x7f464c0a32e0 2023-10-30T12:13:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c0a1630 con 0x7f464c0a2ec0 2023-10-30T12:13:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0a4190 con 0x7f464c0a2ae0 2023-10-30T12:13:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f465396c640 10 monclient(hunting): authenticate will time out at 1630.740325s 2023-10-30T12:13:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ae0 0x7f464c0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f464c0a32e0 0x7f464c135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f464c0a32e0 0x7f464c135c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57102/0 (socket says 172.21.15.50:57102) 2023-10-30T12:13:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ae0 0x7f464c0a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50498/0 (socket says 172.21.15.50:50498) 2023-10-30T12:13:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 -- 172.21.15.50:0/1029894581 learned_addr learned my addr 172.21.15.50:0/1029894581 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): get_auth_request con 0x7f464c0a32e0 auth_method 0 2023-10-30T12:13:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 10 monclient(hunting): get_auth_request con 0x7f464c0a2ae0 auth_method 0 2023-10-30T12:13:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4650f07640 1 --2- 172.21.15.50:0/1029894581 >> v2:172.21.15.118:3300/0 conn(0x7f464c0a2ec0 0x7f464c1356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient(hunting): handle_auth_done global_id 174156 payload 274 2023-10-30T12:13:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 -- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ae0 msgr2=0x7f464c0a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 --2- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ae0 0x7f464c0a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 -- 172.21.15.50:0/1029894581 >> v2:172.21.15.118:3300/0 conn(0x7f464c0a2ec0 msgr2=0x7f464c1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 --2- 172.21.15.50:0/1029894581 >> v2:172.21.15.118:3300/0 conn(0x7f464c0a2ec0 0x7f464c1356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient: _finish_hunting 0 2023-10-30T12:13:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 monclient: found mon.noname-a 2023-10-30T12:13:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651f09640 1 -- 172.21.15.50:0/1029894581 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c09a2e0 con 0x7f464c0a32e0 2023-10-30T12:13:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4650f07640 10 monclient: get_auth_request con 0x7f464c0a2ec0 auth_method 0 2023-10-30T12:13:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4650f07640 1 --2- 172.21.15.50:0/1029894581 >> v2:172.21.15.118:3300/0 conn(0x7f464c0a2ec0 0x7f464c1356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.557+0000 7f4651708640 1 --2- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ae0 0x7f464c0a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f4651f09640 1 --2- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a32e0 0x7f464c135c20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f464c06b510 tx=0x7f464402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea713539b315c71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 1 -- 172.21.15.50:0/1029894581 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644031740 con 0x7f464c0a32e0 2023-10-30T12:13:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: dump: 2023-10-30T12:13:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.580 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.580 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.580 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.580 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.580 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.581 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: _finish_auth 0 2023-10-30T12:13:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:13:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 1 -- 172.21.15.50:0/1029894581 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46440318a0 con 0x7f464c0a32e0 2023-10-30T12:13:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f465396c640 5 monclient: authenticate success, global_id 174156 2023-10-30T12:13:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f465396c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f465396c640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 1 -- 172.21.15.50:0/1029894581 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644031c30 con 0x7f464c0a32e0 2023-10-30T12:13:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f465396c640 4 set_mon_vals no callback set 2023-10-30T12:13:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.558+0000 7f464bfff640 10 monclient: dump: 2023-10-30T12:13:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.585 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 10 monclient: shutdown 2023-10-30T12:13:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 -- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a32e0 msgr2=0x7f464c135c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 --2- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a32e0 0x7f464c135c20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f464c06b510 tx=0x7f464402b7e0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 -- 172.21.15.50:0/1029894581 shutdown_connections 2023-10-30T12:13:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 --2- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a32e0 0x7f464c135c20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 --2- 172.21.15.50:0/1029894581 >> v2:172.21.15.118:3300/0 conn(0x7f464c0a2ec0 0x7f464c1356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 --2- 172.21.15.50:0/1029894581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ae0 0x7f464c0a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 -- 172.21.15.50:0/1029894581 >> 172.21.15.50:0/1029894581 conn(0x7f464c0a80b0 msgr2=0x7f464c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 -- 172.21.15.50:0/1029894581 shutdown_connections 2023-10-30T12:13:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 1 -- 172.21.15.50:0/1029894581 wait complete. 2023-10-30T12:13:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 10 monclient: build_initial_monmap 2023-10-30T12:13:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.559+0000 7f465396c640 10 monclient: monmap: 2023-10-30T12:13:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:05.589 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:05.589 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:05.589 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.589 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:05.589 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:05.590 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:05.590 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 librados: starting msgr at 2023-10-30T12:13:05.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 librados: starting objecter 2023-10-30T12:13:05.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 Processor -- start 2023-10-30T12:13:05.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 -- start start 2023-10-30T12:13:05.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 librados: setting wanted keys 2023-10-30T12:13:05.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 librados: calling monclient init 2023-10-30T12:13:05.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient: init 2023-10-30T12:13:05.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:05.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:13:05.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a2ae0 0x7f464c1ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): picked mon.noname-a con 0x7f464c0a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:05.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ec0 0x7f464c132570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): picked mon.noname-c con 0x7f464c0a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:05.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f464c132ab0 0x7f464c11e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): picked mon.noname-b con 0x7f464c132ab0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c11f960 con 0x7f464c0a2ae0 2023-10-30T12:13:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c120980 con 0x7f464c132ab0 2023-10-30T12:13:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c120c60 con 0x7f464c0a2ec0 2023-10-30T12:13:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f465396c640 10 monclient(hunting): authenticate will time out at 1630.743827s 2023-10-30T12:13:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4651708640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a2ae0 0x7f464c1ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4650f07640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ec0 0x7f464c132570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4651708640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a2ae0 0x7f464c1ca070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57112/0 (socket says 172.21.15.50:57112) 2023-10-30T12:13:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4651708640 1 -- 172.21.15.50:0/1890976515 learned_addr learned my addr 172.21.15.50:0/1890976515 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4651708640 10 monclient(hunting): get_auth_request con 0x7f464c0a2ae0 auth_method 0 2023-10-30T12:13:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4651708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4651708640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4651708640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4650f07640 10 monclient(hunting): get_auth_request con 0x7f464c0a2ec0 auth_method 0 2023-10-30T12:13:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4650f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4650f07640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.560+0000 7f4650f07640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651f09640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f464c132ab0 0x7f464c11e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651f09640 10 monclient(hunting): get_auth_request con 0x7f464c132ab0 auth_method 0 2023-10-30T12:13:05.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651f09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651f09640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651f09640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 10 monclient(hunting): handle_auth_done global_id 114776 payload 824 2023-10-30T12:13:05.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f464c132ab0 msgr2=0x7f464c11e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f464c132ab0 0x7f464c11e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a2ae0 msgr2=0x7f464c1ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a2ae0 0x7f464c1ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 10 monclient: _finish_hunting 0 2023-10-30T12:13:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 1 monclient: found mon.noname-c 2023-10-30T12:13:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 1 -- 172.21.15.50:0/1890976515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009a70 con 0x7f464c0a2ec0 2023-10-30T12:13:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4651708640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f464c0a2ae0 0x7f464c1ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4650f07640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ec0 0x7f464c132570 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f464000c820 tx=0x7f464000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 client.?.objecter ms_handle_connect 0x7f464c0a2ec0 2023-10-30T12:13:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640004220 con 0x7f464c0a2ec0 2023-10-30T12:13:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: dump: 2023-10-30T12:13:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.604 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: _finish_auth 0 2023-10-30T12:13:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:13:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640004380 con 0x7f464c0a2ec0 2023-10-30T12:13:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 5 monclient: authenticate success, global_id 114776 2023-10-30T12:13:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 10 monclient: _renew_subs 2023-10-30T12:13:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46440094d0 con 0x7f464c0a2ec0 2023-10-30T12:13:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640007a50 con 0x7f464c0a2ec0 2023-10-30T12:13:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f4649ffb640 10 monclient: dump: 2023-10-30T12:13:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.608 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 10 client.114776.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 10 monclient: _renew_subs 2023-10-30T12:13:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f464b7fe640 4 set_mon_vals no callback set 2023-10-30T12:13:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c132f40 con 0x7f464c0a2ec0 2023-10-30T12:13:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.561+0000 7f465396c640 1 librados: init done 2023-10-30T12:13:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4640007bb0 con 0x7f464c0a2ec0 2023-10-30T12:13:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f465396c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f465396c640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a32e0 0x7f464c11e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c073ff0 con 0x7f464c0a32e0 2023-10-30T12:13:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4649ffb640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46380392f0 0x7f463803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(399..399 src has 1..399) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f464004a9e0 con 0x7f464c0a2ec0 2023-10-30T12:13:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4649ffb640 10 client.114776.objecter ms_dispatch 0x7f464c000b90 osd_map(399..399 src has 1..399) v4 2023-10-30T12:13:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4649ffb640 3 client.114776.objecter handle_osd_map got epochs [399,399] > 0 2023-10-30T12:13:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4649ffb640 3 client.114776.objecter handle_osd_map decoding full epoch 399 2023-10-30T12:13:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a32e0 0x7f464c11e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient: get_auth_request con 0x7f464c0a32e0 auth_method 0 2023-10-30T12:13:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4649ffb640 20 client.114776.objecter dump_active .. 0 homeless 2023-10-30T12:13:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 10 monclient(hunting): handle_auth_done global_id 114779 payload 824 2023-10-30T12:13:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.562+0000 7f4651708640 1 -- 172.21.15.50:0/1890976515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f464c09a5c0 con 0x7f464c0a32e0 2023-10-30T12:13:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4651f09640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46380392f0 0x7f463803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4651f09640 10 monclient: get_auth_request con 0x7f46380392f0 auth_method 0 2023-10-30T12:13:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 client.114776.objecter ms_handle_connect 0x7f464c0a32e0 2023-10-30T12:13:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 client.114776.objecter resend_mon_ops 2023-10-30T12:13:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4651708640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a32e0 0x7f464c11e5e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f463c004980 tx=0x7f463c00e980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c015920 con 0x7f464c0a32e0 2023-10-30T12:13:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 monclient: dump: 2023-10-30T12:13:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.617 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 client.114776.objecter ms_handle_connect 0x7f46380392f0 2023-10-30T12:13:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4651f09640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46380392f0 0x7f463803b7b0 secure :-1 s=READY pgs=4150 cs=0 l=1 rev1=1 crypto rx=0x7f4644035000 tx=0x7f4644008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f463c00f040 con 0x7f464c0a32e0 2023-10-30T12:13:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a32e0 msgr2=0x7f464c11e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a32e0 0x7f464c11e5e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f463c004980 tx=0x7f463c00e980 comp rx=0 tx=0).stop 2023-10-30T12:13:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a32e0 msgr2=0x7f464c11e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.563+0000 7f4649ffb640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a32e0 0x7f464c11e5e0 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f463c004980 tx=0x7f463c00e980 comp rx=0 tx=0).stop 2023-10-30T12:13:05.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f465396c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f465396c640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c11eb20 0x7f464c09c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f465396c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c120fa0 con 0x7f464c11eb20 2023-10-30T12:13:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f4651708640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c11eb20 0x7f464c09c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f4651708640 10 monclient: get_auth_request con 0x7f464c11eb20 auth_method 0 2023-10-30T12:13:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f4651708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f4651708640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.569+0000 7f4651708640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4651708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4651708640 10 monclient(hunting): handle_auth_done global_id 114782 payload 824 2023-10-30T12:13:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4651708640 1 -- 172.21.15.50:0/1890976515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f464c099b40 con 0x7f464c11eb20 2023-10-30T12:13:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4651708640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c11eb20 0x7f464c09c2a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f463c00b010 tx=0x7f463c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 10 client.114776.objecter ms_handle_connect 0x7f464c11eb20 2023-10-30T12:13:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 10 client.114776.objecter resend_mon_ops 2023-10-30T12:13:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c023040 con 0x7f464c11eb20 2023-10-30T12:13:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 10 monclient: dump: 2023-10-30T12:13:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.626 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f463c0133d0 con 0x7f464c11eb20 2023-10-30T12:13:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c11eb20 msgr2=0x7f464c09c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c11eb20 0x7f464c09c2a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f463c00b010 tx=0x7f463c004ab0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c11eb20 msgr2=0x7f464c09c2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:05.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.570+0000 7f4649ffb640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c11eb20 0x7f464c09c2a0 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f463c00b010 tx=0x7f463c004ab0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.628 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:05.628 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:05.628 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:05.629 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:05.629 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 124, 2023-10-30T12:13:05.629 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:05.629 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:05.629 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:05.630 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:05.630 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.630 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:13:05.630 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:05.631 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:05.631 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:05.631 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.631 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.631 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.632 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.632 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.632 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.632 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.632 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.633 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.633 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.633 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.633 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:05.633 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:05.634 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.634 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.634 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.634 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.634 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.634 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.635 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.635 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.635 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.635 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.635 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.636 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:05.636 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:05.636 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:05.636 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:05.637 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:05.637 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:05.637 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:05.637 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:05.637 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:05.637 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:05.638 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:05.638 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:05.638 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:05.638 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:05.638 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:05.639 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:05.639 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:05.639 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.639 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.639 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.640 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.640 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.640 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.640 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.640 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.641 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.641 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.641 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.641 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:05.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.641 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:05.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.642 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:05.642 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:05.642 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.642 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.643 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.643 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:05.643 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.644 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.644 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:05.644 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.644 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.645 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:05.645 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:05.645 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.645 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.646 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.646 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:05.646 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:05.647 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.647 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.647 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:05.648 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.648 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.648 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:05.648 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.649 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:05.649 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:05.650 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.650 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.650 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.651 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:05.651 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:05.651 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.651 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.653 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.653 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:05.653 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.653 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.654 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:05.654 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:05.654 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.654 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.654 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.655 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.655 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:05.655 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:05.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.656 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.656 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.656 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:05.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.657 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.657 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:13:05.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.657 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.657 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.658 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:13:05.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.658 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.658 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:05.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:13:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.660 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:05.660 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 10 librados: watch_flush enter 2023-10-30T12:13:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 10 librados: watch_flush exit 2023-10-30T12:13:05.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 20 client.114776.objecter shutdown clearing up homeless session... 2023-10-30T12:13:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 10 client.114776.objecter successfully canceled tick 2023-10-30T12:13:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46380392f0 msgr2=0x7f463803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f46380392f0 0x7f463803b7b0 secure :-1 s=READY pgs=4150 cs=0 l=1 rev1=1 crypto rx=0x7f4644035000 tx=0x7f4644008030 comp rx=0 tx=0).stop 2023-10-30T12:13:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 10 monclient: shutdown 2023-10-30T12:13:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ec0 msgr2=0x7f464c132570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f4651f09640 1 -- 172.21.15.50:0/1890976515 reap_dead start 2023-10-30T12:13:05.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ec0 0x7f464c132570 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f464000c820 tx=0x7f464000cce0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 shutdown_connections 2023-10-30T12:13:05.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 --2- 172.21.15.50:0/1890976515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f464c0a2ec0 0x7f464c132570 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 >> 172.21.15.50:0/1890976515 conn(0x7f464c0a80b0 msgr2=0x7f464c1bb870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 shutdown_connections 2023-10-30T12:13:05.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 -- 172.21.15.50:0/1890976515 wait complete. 2023-10-30T12:13:05.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.571+0000 7f465396c640 1 librados: shutdown 2023-10-30T12:13:05.666 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:05.666 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:13:05.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:13:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.775+0000 7fce6a664640 10 monclient: get_monmap_and_config 2023-10-30T12:13:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.775+0000 7fce6a664640 10 monclient: build_initial_monmap 2023-10-30T12:13:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.775+0000 7fce6a664640 10 monclient: monmap: 2023-10-30T12:13:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:05.776196+0000 2023-10-30T12:13:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:05.776196+0000 2023-10-30T12:13:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:05.779 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.776+0000 7fce6a664640 1 Processor -- start 2023-10-30T12:13:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.776+0000 7fce6a664640 1 -- start start 2023-10-30T12:13:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.776+0000 7fce6a664640 10 monclient: init 2023-10-30T12:13:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-10-30T12:13:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce640659d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient(hunting): picked mon.noname-b con 0x7fce640a0ff0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fce6409e7e0 0x7fce6409ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient(hunting): picked mon.noname-c con 0x7fce6409e7e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce6409f160 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6409b730 con 0x7fce6409f160 2023-10-30T12:13:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6409d3e0 con 0x7fce6409e7e0 2023-10-30T12:13:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce640a13d0 con 0x7fce640a0ff0 2023-10-30T12:13:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce6a664640 10 monclient(hunting): authenticate will time out at 1630.960193s 2023-10-30T12:13:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fce6409e7e0 0x7fce6409ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fce6409e7e0 0x7fce6409ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:59454/0 (socket says 172.21.15.50:59454) 2023-10-30T12:13:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 -- 172.21.15.50:0/2798789622 learned_addr learned my addr 172.21.15.50:0/2798789622 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): get_auth_request con 0x7fce6409e7e0 auth_method 0 2023-10-30T12:13:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 1 --2- 172.21.15.50:0/2798789622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce640659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 10 monclient(hunting): get_auth_request con 0x7fce640a0ff0 auth_method 0 2023-10-30T12:13:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient(hunting): handle_auth_done global_id 154348 payload 274 2023-10-30T12:13:05.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 -- 172.21.15.50:0/2798789622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 msgr2=0x7fce640659d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 --2- 172.21.15.50:0/2798789622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce640659d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 -- 172.21.15.50:0/2798789622 >> v1:172.21.15.50:6789/0 conn(0x7fce6409f160 legacy=0x7fce641396f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient: _finish_hunting 0 2023-10-30T12:13:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 monclient: found mon.noname-c 2023-10-30T12:13:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:13:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce637fe640 1 -- 172.21.15.50:0/2798789622 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce640a0070 con 0x7fce6409e7e0 2023-10-30T12:13:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce62ffd640 1 -- 172.21.15.50:0/2798789622 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995215332 0 0) 0x7fce6409b730 con 0x7fce6409f160 2023-10-30T12:13:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce62ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:13:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.777+0000 7fce63fff640 1 --2- 172.21.15.50:0/2798789622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce640659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce637fe640 1 --2- 172.21.15.50:0/2798789622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 0x7fce6409ec20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fce58009870 tx=0x7fce5802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88370d880bd8d894 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 1 -- 172.21.15.50:0/2798789622 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce580317c0 con 0x7fce6409e7e0 2023-10-30T12:13:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 10 monclient: dump: 2023-10-30T12:13:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.799 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce62ffd640 1 -- 172.21.15.50:0/2798789622 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce58031920 con 0x7fce6409e7e0 2023-10-30T12:13:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce6a664640 5 monclient: authenticate success, global_id 154348 2023-10-30T12:13:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.778+0000 7fce6a664640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce62ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce62ffd640 1 -- 172.21.15.50:0/2798789622 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58031cb0 con 0x7fce6409e7e0 2023-10-30T12:13:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 4 set_mon_vals no callback set 2023-10-30T12:13:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce62ffd640 10 monclient: dump: 2023-10-30T12:13:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.804 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 10 monclient: shutdown 2023-10-30T12:13:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 -- 172.21.15.50:0/2798789622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 msgr2=0x7fce6409ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 --2- 172.21.15.50:0/2798789622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 0x7fce6409ec20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fce58009870 tx=0x7fce5802b860 comp rx=0 tx=0).stop 2023-10-30T12:13:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 -- 172.21.15.50:0/2798789622 shutdown_connections 2023-10-30T12:13:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 --2- 172.21.15.50:0/2798789622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 0x7fce6409ec20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 --2- 172.21.15.50:0/2798789622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce640659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 -- 172.21.15.50:0/2798789622 >> 172.21.15.50:0/2798789622 conn(0x7fce640a4800 msgr2=0x7fce640a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 -- 172.21.15.50:0/2798789622 shutdown_connections 2023-10-30T12:13:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.779+0000 7fce6a664640 1 -- 172.21.15.50:0/2798789622 wait complete. 2023-10-30T12:13:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 10 monclient: build_initial_monmap 2023-10-30T12:13:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 10 monclient: monmap: 2023-10-30T12:13:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:05.808 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 1 librados: starting msgr at 2023-10-30T12:13:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 1 librados: starting objecter 2023-10-30T12:13:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 1 Processor -- start 2023-10-30T12:13:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 1 -- start start 2023-10-30T12:13:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 1 librados: setting wanted keys 2023-10-30T12:13:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 1 librados: calling monclient init 2023-10-30T12:13:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.780+0000 7fce6a664640 10 monclient: init 2023-10-30T12:13:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:13:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 0x7fce6412db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): picked mon.noname-b con 0x7fce6409e7e0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce6409f160 0x7fce64131e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): picked mon.noname-a con 0x7fce6409f160 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce64132390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): picked mon.noname-c con 0x7fce640a0ff0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce641358f0 con 0x7fce6409f160 2023-10-30T12:13:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce64135b60 con 0x7fce6409e7e0 2023-10-30T12:13:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce64135e40 con 0x7fce640a0ff0 2023-10-30T12:13:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce6a664640 10 monclient(hunting): authenticate will time out at 1630.964288s 2023-10-30T12:13:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce6409f160 0x7fce64131e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce64132390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce6409f160 0x7fce64131e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57126/0 (socket says 172.21.15.50:57126) 2023-10-30T12:13:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 1 -- 172.21.15.50:0/3765280627 learned_addr learned my addr 172.21.15.50:0/3765280627 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 10 monclient(hunting): get_auth_request con 0x7fce6409f160 auth_method 0 2023-10-30T12:13:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce64132390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50548/0 (socket says 172.21.15.50:50548) 2023-10-30T12:13:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): get_auth_request con 0x7fce640a0ff0 auth_method 0 2023-10-30T12:13:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 0x7fce6412db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 10 monclient(hunting): get_auth_request con 0x7fce6409e7e0 auth_method 0 2023-10-30T12:13:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient(hunting): handle_auth_done global_id 114788 payload 824 2023-10-30T12:13:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 msgr2=0x7fce6412db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6409e7e0 0x7fce6412db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce6409f160 msgr2=0x7fce64131e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce6409f160 0x7fce64131e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient: _finish_hunting 0 2023-10-30T12:13:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 monclient: found mon.noname-c 2023-10-30T12:13:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 -- 172.21.15.50:0/3765280627 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54009400 con 0x7fce640a0ff0 2023-10-30T12:13:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce637fe640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce6409f160 0x7fce64131e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce68c01640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce64132390 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fce5c001560 tx=0x7fce5c00cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce60ff9640 10 client.?.objecter ms_handle_connect 0x7fce640a0ff0 2023-10-30T12:13:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.781+0000 7fce60ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c003da0 con 0x7fce640a0ff0 2023-10-30T12:13:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: dump: 2023-10-30T12:13:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.823 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: _finish_auth 0 2023-10-30T12:13:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:13:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c00ce30 con 0x7fce640a0ff0 2023-10-30T12:13:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 5 monclient: authenticate success, global_id 114788 2023-10-30T12:13:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c005290 con 0x7fce640a0ff0 2023-10-30T12:13:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 10 monclient: _renew_subs 2023-10-30T12:13:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580094d0 con 0x7fce640a0ff0 2023-10-30T12:13:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce627fc640 4 set_mon_vals no callback set 2023-10-30T12:13:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 10 monclient: dump: 2023-10-30T12:13:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.827 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 10 client.114788.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 10 monclient: _renew_subs 2023-10-30T12:13:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce64133b80 con 0x7fce640a0ff0 2023-10-30T12:13:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 1 librados: init done 2023-10-30T12:13:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fce5c0053f0 con 0x7fce640a0ff0 2023-10-30T12:13:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6412e060 0x7fce641328d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.782+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6409f8e0 con 0x7fce6412e060 2023-10-30T12:13:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce500392f0 0x7fce5003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6412e060 0x7fce641328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(400..400 src has 1..400) v4 ==== 6682+0+0 (secure 0 0 0) 0x7fce5c04a5b0 con 0x7fce640a0ff0 2023-10-30T12:13:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 10 client.114788.objecter ms_dispatch 0x7fce64000b90 osd_map(400..400 src has 1..400) v4 2023-10-30T12:13:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 3 client.114788.objecter handle_osd_map got epochs [400,400] > 0 2023-10-30T12:13:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 3 client.114788.objecter handle_osd_map decoding full epoch 400 2023-10-30T12:13:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient: get_auth_request con 0x7fce6412e060 auth_method 0 2023-10-30T12:13:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 20 client.114788.objecter dump_active .. 0 homeless 2023-10-30T12:13:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce637fe640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce500392f0 0x7fce5003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce637fe640 10 monclient: get_auth_request con 0x7fce500392f0 auth_method 0 2023-10-30T12:13:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 10 monclient(hunting): handle_auth_done global_id 154354 payload 824 2023-10-30T12:13:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 1 -- 172.21.15.50:0/3765280627 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce6409c420 con 0x7fce6412e060 2023-10-30T12:13:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 10 client.114788.objecter ms_handle_connect 0x7fce6412e060 2023-10-30T12:13:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 10 client.114788.objecter resend_mon_ops 2023-10-30T12:13:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce63fff640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6412e060 0x7fce641328d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce54011bd0 tx=0x7fce5400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce54011ed0 con 0x7fce6412e060 2023-10-30T12:13:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 10 monclient: dump: 2023-10-30T12:13:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.836 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce60ff9640 10 client.114788.objecter ms_handle_connect 0x7fce500392f0 2023-10-30T12:13:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.783+0000 7fce637fe640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce500392f0 0x7fce5003b7b0 secure :-1 s=READY pgs=4151 cs=0 l=1 rev1=1 crypto rx=0x7fce58009870 tx=0x7fce5803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.784+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fce5400fe50 con 0x7fce6412e060 2023-10-30T12:13:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.784+0000 7fce60ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.784+0000 7fce60ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.784+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6412e060 msgr2=0x7fce641328d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.784+0000 7fce60ff9640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6412e060 0x7fce641328d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce54011bd0 tx=0x7fce5400f610 comp rx=0 tx=0).stop 2023-10-30T12:13:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.784+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6412e060 msgr2=0x7fce641328d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.784+0000 7fce60ff9640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce6412e060 0x7fce641328d0 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce54011bd0 tx=0x7fce5400f610 comp rx=0 tx=0).stop 2023-10-30T12:13:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce6a664640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce6a664640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce64132e10 0x7fce6412e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce6a664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce64134710 con 0x7fce64132e10 2023-10-30T12:13:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce64132e10 0x7fce6412e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient: get_auth_request con 0x7fce64132e10 auth_method 0 2023-10-30T12:13:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 10 monclient(hunting): handle_auth_done global_id 154357 payload 824 2023-10-30T12:13:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.790+0000 7fce63fff640 1 -- 172.21.15.50:0/3765280627 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce64135cd0 con 0x7fce64132e10 2023-10-30T12:13:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce63fff640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce64132e10 0x7fce6412e780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 10 client.114788.objecter ms_handle_connect 0x7fce64132e10 2023-10-30T12:13:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 10 client.114788.objecter resend_mon_ops 2023-10-30T12:13:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5401e650 con 0x7fce64132e10 2023-10-30T12:13:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 10 monclient: dump: 2023-10-30T12:13:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:05.846 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7fce540283a0 con 0x7fce64132e10 2023-10-30T12:13:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce64132e10 msgr2=0x7fce6412e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce64132e10 0x7fce6412e780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce540046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce64132e10 msgr2=0x7fce6412e780 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce60ff9640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fce64132e10 0x7fce6412e780 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.847 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:05.848 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:05.848 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:05.848 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:05.848 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 124, 2023-10-30T12:13:05.848 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:05.849 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:05.849 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:05.849 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:05.849 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.849 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:13:05.850 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:05.850 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:05.850 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:05.850 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.850 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.851 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.851 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.851 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.851 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.851 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.852 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.852 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.852 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.852 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.852 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:05.853 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:05.853 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.853 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.853 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.853 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.853 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.854 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.854 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.854 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.854 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.854 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.855 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:05.855 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:05.855 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:05.856 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:05.856 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:05.856 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:05.856 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:05.856 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:05.857 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:05.857 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:05.857 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:05.857 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:05.857 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:05.857 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:05.858 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:05.858 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:05.858 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:05.858 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:05.858 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:05.859 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:05.859 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:05.859 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:05.859 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:05.859 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:05.860 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:05.860 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:05.860 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:05.860 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.860 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:05.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.861 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:05.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.861 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:05.861 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:05.861 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.863 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.863 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:05.863 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.863 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.864 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.864 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:05.864 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:05.864 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.865 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.865 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.865 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:05.866 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:05.866 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.866 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.866 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.867 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:05.867 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.867 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.867 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:05.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.871 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:05.871 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:05.871 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:05.872 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:05.872 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:05.872 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.872 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.873 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:05.873 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:05.873 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:05.873 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:05.874 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:05.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.874 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.874 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:05.874 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:05.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.875 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.875 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.875 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:05.875 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.876 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.876 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:05.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.876 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.876 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.877 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:13:05.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.877 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:05.877 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:13:05.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:05.878 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:05.878 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:05.878 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:05.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:05.878 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:05.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:05.879 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:05.879 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:05.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce6a664640 10 librados: watch_flush enter 2023-10-30T12:13:05.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce6a664640 10 librados: watch_flush exit 2023-10-30T12:13:05.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce6a664640 20 client.114788.objecter shutdown clearing up homeless session... 2023-10-30T12:13:05.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.791+0000 7fce6a664640 10 client.114788.objecter successfully canceled tick 2023-10-30T12:13:05.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce500392f0 msgr2=0x7fce5003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fce500392f0 0x7fce5003b7b0 secure :-1 s=READY pgs=4151 cs=0 l=1 rev1=1 crypto rx=0x7fce58009870 tx=0x7fce5803d040 comp rx=0 tx=0).stop 2023-10-30T12:13:05.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 10 monclient: shutdown 2023-10-30T12:13:05.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:05.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 msgr2=0x7fce64132390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:05.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 --2- 172.21.15.50:0/3765280627 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fce640a0ff0 0x7fce64132390 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fce5c001560 tx=0x7fce5c00cbe0 comp rx=0 tx=0).stop 2023-10-30T12:13:05.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce68c01640 1 -- 172.21.15.50:0/3765280627 reap_dead start 2023-10-30T12:13:05.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce68c01640 1 -- 172.21.15.50:0/3765280627 reap_dead start 2023-10-30T12:13:05.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 shutdown_connections 2023-10-30T12:13:05.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 >> 172.21.15.50:0/3765280627 conn(0x7fce640a4800 msgr2=0x7fce6406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:05.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 shutdown_connections 2023-10-30T12:13:05.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 -- 172.21.15.50:0/3765280627 wait complete. 2023-10-30T12:13:05.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:05.792+0000 7fce6a664640 1 librados: shutdown 2023-10-30T12:13:05.884 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:05.885 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-10-30T12:13:05.885 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:13:05.885 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-10-30T12:13:05.885 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:13:05.886 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:13:05.886 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:13:05.902 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:13:05.902 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:13:05.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:06.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.005+0000 7ff9bcdd5640 10 monclient: get_monmap_and_config 2023-10-30T12:13:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.005+0000 7ff9bcdd5640 10 monclient: build_initial_monmap 2023-10-30T12:13:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.005+0000 7ff9bcdd5640 10 monclient: monmap: 2023-10-30T12:13:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:06.006507+0000 2023-10-30T12:13:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:06.006507+0000 2023-10-30T12:13:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:06.009 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:06.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.005+0000 7ff9bcdd5640 1 Processor -- start 2023-10-30T12:13:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 1 -- start start 2023-10-30T12:13:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient: init 2023-10-30T12:13:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-10-30T12:13:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8098520 0x7ff9b809bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): picked mon.noname-b con 0x7ff9b8098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff9b8098900 0x7ff9b8098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): picked mon.noname-a con 0x7ff9b8098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff9b8132740 0x7ff9b8132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): picked mon.noname-c con 0x7ff9b8132740 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b809ad30 con 0x7ff9b8098900 2023-10-30T12:13:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b809c460 con 0x7ff9b8132740 2023-10-30T12:13:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b809b260 con 0x7ff9b8098520 2023-10-30T12:13:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9bcdd5640 10 monclient(hunting): authenticate will time out at 1631.189655s 2023-10-30T12:13:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff9b8098900 0x7ff9b8098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8098520 0x7ff9b809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b6d9d640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7ff9b8132740 msgr2=0x7ff9b8132bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:06.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8098520 0x7ff9b809bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50558/0 (socket says 172.21.15.50:50558) 2023-10-30T12:13:06.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 1 -- 172.21.15.50:0/124724827 learned_addr learned my addr 172.21.15.50:0/124724827 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:06.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff9b8098900 0x7ff9b8098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57128/0 (socket says 172.21.15.50:57128) 2023-10-30T12:13:06.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 10 monclient(hunting): get_auth_request con 0x7ff9b8098520 auth_method 0 2023-10-30T12:13:06.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b6d9d640 1 --2- 172.21.15.50:0/124724827 >> v2:172.21.15.118:3300/0 conn(0x7ff9b8132740 0x7ff9b8132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:06.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:06.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:06.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:06.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 10 monclient(hunting): get_auth_request con 0x7ff9b8098900 auth_method 0 2023-10-30T12:13:06.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:06.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:06.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:06.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:06.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:06.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:06.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:06.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:06.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.006+0000 7ff9b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:06.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 10 monclient(hunting): handle_auth_done global_id 114791 payload 274 2023-10-30T12:13:06.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 1 -- 172.21.15.50:0/124724827 >> v2:172.21.15.118:3300/0 conn(0x7ff9b8132740 msgr2=0x7ff9b8132bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:06.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 1 --2- 172.21.15.50:0/124724827 >> v2:172.21.15.118:3300/0 conn(0x7ff9b8132740 0x7ff9b8132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 1 -- 172.21.15.50:0/124724827 >> v2:172.21.15.50:3300/0 conn(0x7ff9b8098900 msgr2=0x7ff9b8098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:06.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 1 --2- 172.21.15.50:0/124724827 >> v2:172.21.15.50:3300/0 conn(0x7ff9b8098900 0x7ff9b8098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 10 monclient: _finish_hunting 0 2023-10-30T12:13:06.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 1 monclient: found mon.noname-b 2023-10-30T12:13:06.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:06.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:06.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 1 -- 172.21.15.50:0/124724827 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8072120 con 0x7ff9b8098520 2023-10-30T12:13:06.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b5d9b640 1 --2- 172.21.15.50:0/124724827 >> v2:172.21.15.50:3300/0 conn(0x7ff9b8098900 0x7ff9b8098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:06.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.007+0000 7ff9b659c640 1 --2- 172.21.15.50:0/124724827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8098520 0x7ff9b809bf20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff9b806baf0 tx=0x7ff9a002dfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33783ea068cef3fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:06.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 1 -- 172.21.15.50:0/124724827 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a002ed70 con 0x7ff9b8098520 2023-10-30T12:13:06.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:06.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:06.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: dump: 2023-10-30T12:13:06.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:06.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:06.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:06.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:06.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:06.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:06.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:06.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:06.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: _finish_auth 0 2023-10-30T12:13:06.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: _check_auth_tickets 2023-10-30T12:13:06.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:06.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 1 -- 172.21.15.50:0/124724827 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a0004030 con 0x7ff9b8098520 2023-10-30T12:13:06.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 5 monclient: authenticate success, global_id 114791 2023-10-30T12:13:06.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:06.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:06.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 4 set_mon_vals no callback set 2023-10-30T12:13:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 1 -- 172.21.15.50:0/124724827 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0004420 con 0x7ff9b8098520 2023-10-30T12:13:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9b559a640 10 monclient: dump: 2023-10-30T12:13:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:06.034 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:06.034 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:06.034 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:06.034 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:06.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 10 monclient: shutdown 2023-10-30T12:13:06.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:06.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/124724827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8098520 msgr2=0x7ff9b809bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:06.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/124724827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8098520 0x7ff9b809bf20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff9b806baf0 tx=0x7ff9a002dfc0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/124724827 shutdown_connections 2023-10-30T12:13:06.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/124724827 >> v2:172.21.15.118:3300/0 conn(0x7ff9b8132740 0x7ff9b8132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/124724827 >> v2:172.21.15.50:3300/0 conn(0x7ff9b8098900 0x7ff9b8098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/124724827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8098520 0x7ff9b809bf20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/124724827 >> 172.21.15.50:0/124724827 conn(0x7ff9b80a05c0 msgr2=0x7ff9b809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:06.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/124724827 shutdown_connections 2023-10-30T12:13:06.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.008+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/124724827 wait complete. 2023-10-30T12:13:06.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 10 monclient: build_initial_monmap 2023-10-30T12:13:06.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 10 monclient: monmap: 2023-10-30T12:13:06.037 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:06.037 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:06.037 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:06.038 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:06.038 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:06.038 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:06.038 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:06.038 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:06.039 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 1 librados: starting msgr at 2023-10-30T12:13:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 1 librados: starting objecter 2023-10-30T12:13:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 1 Processor -- start 2023-10-30T12:13:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 1 -- start start 2023-10-30T12:13:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 1 librados: setting wanted keys 2023-10-30T12:13:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 1 librados: calling monclient init 2023-10-30T12:13:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.009+0000 7ff9bcdd5640 10 monclient: init 2023-10-30T12:13:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:13:06.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9b8098520 0x7ff9b8131840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:06.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): picked mon.noname-a con 0x7ff9b8098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:06.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9b8098900 0x7ff9b8131d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:06.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): picked mon.noname-b con 0x7ff9b8098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:06.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8132740 0x7ff9b812ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:06.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): picked mon.noname-c con 0x7ff9b8132740 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:06.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:06.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b812f020 con 0x7ff9b8098520 2023-10-30T12:13:06.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:06.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b812ed40 con 0x7ff9b8098900 2023-10-30T12:13:06.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:06.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b812e030 con 0x7ff9b8132740 2023-10-30T12:13:06.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:06.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9bcdd5640 10 monclient(hunting): authenticate will time out at 1631.193378s 2023-10-30T12:13:06.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b5d9b640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9b8098900 msgr2=0x7ff9b8131d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:06.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9b8098520 0x7ff9b8131840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:06.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8132740 0x7ff9b812ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:06.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b5d9b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9b8098900 0x7ff9b8131d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:06.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8132740 0x7ff9b812ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50560/0 (socket says 172.21.15.50:50560) 2023-10-30T12:13:06.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 -- 172.21.15.50:0/2114699082 learned_addr learned my addr 172.21.15.50:0/2114699082 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:06.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): get_auth_request con 0x7ff9b8132740 auth_method 0 2023-10-30T12:13:06.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:06.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:06.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:06.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 10 monclient(hunting): get_auth_request con 0x7ff9b8098520 auth_method 0 2023-10-30T12:13:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient(hunting): handle_auth_done global_id 114794 payload 824 2023-10-30T12:13:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 -- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9b8098900 msgr2=0x7ff9b8131d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9b8098900 0x7ff9b8131d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 -- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9b8098520 msgr2=0x7ff9b8131840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9b8098520 0x7ff9b8131840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:13:06.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 monclient: found mon.noname-c 2023-10-30T12:13:06.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:06.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:06.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 -- 172.21.15.50:0/2114699082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8009400 con 0x7ff9b8132740 2023-10-30T12:13:06.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b659c640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9b8098520 0x7ff9b8131840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9b6d9d640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8132740 0x7ff9b812ca40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000b990 tx=0x7ff9b000be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9aeffd640 10 client.?.objecter ms_handle_connect 0x7ff9b8132740 2023-10-30T12:13:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.010+0000 7ff9aeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 1 -- 172.21.15.50:0/2114699082 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b000cb20 con 0x7ff9b8132740 2023-10-30T12:13:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: dump: 2023-10-30T12:13:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.052 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:06.052 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:06.052 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:06.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:06.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:06.053 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:06.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:06.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:06.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:06.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 1 -- 172.21.15.50:0/2114699082 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b000cc80 con 0x7ff9b8132740 2023-10-30T12:13:06.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 5 monclient: authenticate success, global_id 114794 2023-10-30T12:13:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 1 -- 172.21.15.50:0/2114699082 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b00156b0 con 0x7ff9b8132740 2023-10-30T12:13:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9aeffd640 10 monclient: dump: 2023-10-30T12:13:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:06.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:06.056 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:06.056 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:06.056 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:06.056 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 10 monclient: _renew_subs 2023-10-30T12:13:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9b4d99640 4 set_mon_vals no callback set 2023-10-30T12:13:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a000b700 con 0x7ff9b8132740 2023-10-30T12:13:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 10 client.114794.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 10 monclient: _renew_subs 2023-10-30T12:13:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b8070630 con 0x7ff9b8132740 2023-10-30T12:13:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.011+0000 7ff9bcdd5640 1 librados: init done 2023-10-30T12:13:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9aeffd640 1 -- 172.21.15.50:0/2114699082 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff9b0012070 con 0x7ff9b8132740 2023-10-30T12:13:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9bcdd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9bcdd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9bcdd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b804e160 con 0x7ff9b8132740 2023-10-30T12:13:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9aeffd640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9980392a0 0x7ff99803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9aeffd640 1 -- 172.21.15.50:0/2114699082 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(400..400 src has 1..400) v4 ==== 6682+0+0 (secure 0 0 0) 0x7ff9b004acc0 con 0x7ff9b8132740 2023-10-30T12:13:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9aeffd640 10 client.114794.objecter ms_dispatch 0x7ff9b8000b90 osd_map(400..400 src has 1..400) v4 2023-10-30T12:13:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9aeffd640 3 client.114794.objecter handle_osd_map got epochs [400,400] > 0 2023-10-30T12:13:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9aeffd640 3 client.114794.objecter handle_osd_map decoding full epoch 400 2023-10-30T12:13:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9aeffd640 20 client.114794.objecter dump_active .. 0 homeless 2023-10-30T12:13:06.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9b659c640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9980392a0 0x7ff99803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:06.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.012+0000 7ff9b659c640 10 monclient: get_auth_request con 0x7ff9980392a0 auth_method 0 2023-10-30T12:13:06.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.013+0000 7ff9aeffd640 10 client.114794.objecter ms_handle_connect 0x7ff9980392a0 2023-10-30T12:13:06.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.013+0000 7ff9b659c640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9980392a0 0x7ff99803b760 secure :-1 s=READY pgs=4152 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0037000 tx=0x7ff9a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:06.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.016+0000 7ff9aeffd640 1 -- 172.21.15.50:0/2114699082 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9b00149c0 con 0x7ff9b8132740 2023-10-30T12:13:06.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.016+0000 7ff9aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:06.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.016+0000 7ff9aeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:06.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.261+0000 7ff9bcdd5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:06.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.261+0000 7ff9bcdd5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:06.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.261+0000 7ff9bcdd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:06.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.261+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff9b80745c0 con 0x7ff9b8132740 2023-10-30T12:13:06.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.262+0000 7ff9aeffd640 1 -- 172.21.15.50:0/2114699082 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7ff9b001a030 con 0x7ff9b8132740 2023-10-30T12:13:06.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.262+0000 7ff9aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:06.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.262+0000 7ff9aeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:06.265 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:06.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 10 librados: watch_flush enter 2023-10-30T12:13:06.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 10 librados: watch_flush exit 2023-10-30T12:13:06.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 20 client.114794.objecter shutdown clearing up homeless session... 2023-10-30T12:13:06.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 10 client.114794.objecter successfully canceled tick 2023-10-30T12:13:06.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9980392a0 msgr2=0x7ff99803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:06.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9980392a0 0x7ff99803b760 secure :-1 s=READY pgs=4152 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0037000 tx=0x7ff9a0008040 comp rx=0 tx=0).stop 2023-10-30T12:13:06.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 10 monclient: shutdown 2023-10-30T12:13:06.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:06.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8132740 msgr2=0x7ff9b812ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:06.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8132740 0x7ff9b812ca40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000b990 tx=0x7ff9b000be50 comp rx=0 tx=0).stop 2023-10-30T12:13:06.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 shutdown_connections 2023-10-30T12:13:06.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff9980392a0 0x7ff99803b760 unknown :-1 s=CLOSED pgs=4152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff9b8132740 0x7ff9b812ca40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9b8098900 0x7ff9b8131d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 --2- 172.21.15.50:0/2114699082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9b8098520 0x7ff9b8131840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:06.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 >> 172.21.15.50:0/2114699082 conn(0x7ff9b80a05c0 msgr2=0x7ff9b81b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:06.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 shutdown_connections 2023-10-30T12:13:06.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 -- 172.21.15.50:0/2114699082 wait complete. 2023-10-30T12:13:06.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:06.264+0000 7ff9bcdd5640 1 librados: shutdown 2023-10-30T12:13:09.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:09.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.392+0000 7f8bf7af7640 10 monclient: get_monmap_and_config 2023-10-30T12:13:09.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.392+0000 7f8bf7af7640 10 monclient: build_initial_monmap 2023-10-30T12:13:09.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.392+0000 7f8bf7af7640 10 monclient: monmap: 2023-10-30T12:13:09.395 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:09.395 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:09.395 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:09.392883+0000 2023-10-30T12:13:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:09.392883+0000 2023-10-30T12:13:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:09.398 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.393+0000 7f8bf7af7640 1 Processor -- start 2023-10-30T12:13:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.393+0000 7f8bf7af7640 1 -- start start 2023-10-30T12:13:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.393+0000 7f8bf7af7640 10 monclient: init 2023-10-30T12:13:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-10-30T12:13:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8bf009f5b0 0x7f8bf0099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient(hunting): picked mon.noname-c con 0x7f8bf009f5b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bf0096080 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf0096ba0 0x7f8bf013ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf0096ba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf0098810 con 0x7f8bf0096080 2023-10-30T12:13:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf006aef0 con 0x7f8bf009f5b0 2023-10-30T12:13:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf009a1a0 con 0x7f8bf0096ba0 2023-10-30T12:13:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf7af7640 10 monclient(hunting): authenticate will time out at 1634.576190s 2023-10-30T12:13:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf5092640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f8bf0096080 legacy=0x7f8bf0096490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:13:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf5893640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f8bf009f5b0 msgr2=0x7f8bf0099b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf0096ba0 0x7f8bf013ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf5893640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8bf009f5b0 0x7f8bf0099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf0096ba0 0x7f8bf013ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41438/0 (socket says 172.21.15.50:41438) 2023-10-30T12:13:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 -- 172.21.15.50:0/299841695 learned_addr learned my addr 172.21.15.50:0/299841695 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): get_auth_request con 0x7f8bf0096ba0 auth_method 0 2023-10-30T12:13:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:09.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:09.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:09.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:09.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:09.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient(hunting): handle_auth_done global_id 114797 payload 274 2023-10-30T12:13:09.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 -- 172.21.15.50:0/299841695 >> v2:172.21.15.118:3300/0 conn(0x7f8bf009f5b0 msgr2=0x7f8bf0099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:09.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 --2- 172.21.15.50:0/299841695 >> v2:172.21.15.118:3300/0 conn(0x7f8bf009f5b0 0x7f8bf0099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 -- 172.21.15.50:0/299841695 >> v1:172.21.15.118:6789/0 conn(0x7f8bf0096080 legacy=0x7f8bf0096490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:09.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient: _finish_hunting 0 2023-10-30T12:13:09.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 monclient: found mon.noname-b 2023-10-30T12:13:09.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.394+0000 7f8bf6094640 1 -- 172.21.15.50:0/299841695 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf0098af0 con 0x7f8bf0096ba0 2023-10-30T12:13:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf6094640 1 --2- 172.21.15.50:0/299841695 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf0096ba0 0x7f8bf013ab40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009e50 tx=0x7f8be802bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d2fc03fbf64ff3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 1 -- 172.21.15.50:0/299841695 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be802c950 con 0x7f8bf0096ba0 2023-10-30T12:13:09.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:09.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:09.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: dump: 2023-10-30T12:13:09.419 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:09.419 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:09.420 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.420 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.420 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:09.421 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:09.421 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:09.421 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:09.421 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:09.422 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:09.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: _finish_auth 0 2023-10-30T12:13:09.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: _check_auth_tickets 2023-10-30T12:13:09.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:09.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf7af7640 5 monclient: authenticate success, global_id 114797 2023-10-30T12:13:09.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf7af7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:09.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 1 -- 172.21.15.50:0/299841695 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be802cab0 con 0x7f8bf0096ba0 2023-10-30T12:13:09.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf7af7640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf7af7640 4 set_mon_vals no callback set 2023-10-30T12:13:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 1 -- 172.21.15.50:0/299841695 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be802ce40 con 0x7f8bf0096ba0 2023-10-30T12:13:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.395+0000 7f8bf4891640 10 monclient: dump: 2023-10-30T12:13:09.425 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:09.425 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:09.425 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.425 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.425 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:09.426 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:09.426 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:09.426 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:09.426 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:09.427 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:09.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 10 monclient: shutdown 2023-10-30T12:13:09.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:09.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 -- 172.21.15.50:0/299841695 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf0096ba0 msgr2=0x7f8bf013ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:09.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/299841695 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf0096ba0 0x7f8bf013ab40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009e50 tx=0x7f8be802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 -- 172.21.15.50:0/299841695 shutdown_connections 2023-10-30T12:13:09.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/299841695 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf0096ba0 0x7f8bf013ab40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/299841695 >> v2:172.21.15.118:3300/0 conn(0x7f8bf009f5b0 0x7f8bf0099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 -- 172.21.15.50:0/299841695 >> 172.21.15.50:0/299841695 conn(0x7f8bf00a05c0 msgr2=0x7f8bf009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:09.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 -- 172.21.15.50:0/299841695 shutdown_connections 2023-10-30T12:13:09.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 -- 172.21.15.50:0/299841695 wait complete. 2023-10-30T12:13:09.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 10 monclient: build_initial_monmap 2023-10-30T12:13:09.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 10 monclient: monmap: 2023-10-30T12:13:09.429 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:09.429 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:09.430 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:09.430 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:09.430 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:09.430 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:09.430 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:09.431 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:09.431 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:09.431 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:09.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 librados: starting msgr at 2023-10-30T12:13:09.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.396+0000 7f8bf7af7640 1 librados: starting objecter 2023-10-30T12:13:09.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 Processor -- start 2023-10-30T12:13:09.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 -- start start 2023-10-30T12:13:09.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 librados: setting wanted keys 2023-10-30T12:13:09.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 librados: calling monclient init 2023-10-30T12:13:09.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient: init 2023-10-30T12:13:09.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:09.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:13:09.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bf0096080 0x7f8bf008da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:09.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): picked mon.noname-a con 0x7f8bf0096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:09.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bf0096ba0 0x7f8bf01c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:09.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf0096ba0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:09.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf009f5b0 0x7f8bf01c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:09.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): picked mon.noname-c con 0x7f8bf009f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:09.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:09.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf008df40 con 0x7f8bf0096080 2023-10-30T12:13:09.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:09.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf008d3d0 con 0x7f8bf0096ba0 2023-10-30T12:13:09.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:09.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf008ce10 con 0x7f8bf009f5b0 2023-10-30T12:13:09.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:09.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf7af7640 10 monclient(hunting): authenticate will time out at 1634.579601s 2023-10-30T12:13:09.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf009f5b0 0x7f8bf01c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:09.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bf0096080 0x7f8bf008da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:09.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5092640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bf0096ba0 msgr2=0x7f8bf01c8510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:09.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf009f5b0 0x7f8bf01c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41440/0 (socket says 172.21.15.50:41440) 2023-10-30T12:13:09.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 1 -- 172.21.15.50:0/3774271209 learned_addr learned my addr 172.21.15.50:0/3774271209 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:09.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 10 monclient(hunting): get_auth_request con 0x7f8bf009f5b0 auth_method 0 2023-10-30T12:13:09.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bf0096080 0x7f8bf008da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41728/0 (socket says 172.21.15.50:41728) 2023-10-30T12:13:09.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:09.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:09.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:09.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5092640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bf0096ba0 0x7f8bf01c8510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:09.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 10 monclient(hunting): get_auth_request con 0x7f8bf0096080 auth_method 0 2023-10-30T12:13:09.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:09.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:09.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:09.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:09.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:09.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf6094640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:09.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:09.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:09.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.397+0000 7f8bf5893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:09.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 10 monclient(hunting): handle_auth_done global_id 174174 payload 824 2023-10-30T12:13:09.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 1 -- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf009f5b0 msgr2=0x7f8bf01c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:09.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf009f5b0 0x7f8bf01c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 1 -- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bf0096ba0 msgr2=0x7f8bf01c8510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:09.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bf0096ba0 0x7f8bf01c8510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 10 monclient: _finish_hunting 0 2023-10-30T12:13:09.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 1 monclient: found mon.noname-a 2023-10-30T12:13:09.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:09.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:09.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 1 -- 172.21.15.50:0/3774271209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be8009b30 con 0x7f8bf0096080 2023-10-30T12:13:09.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf6094640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf009f5b0 0x7f8bf01c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:09.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf5893640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bf0096080 0x7f8bf008da00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8bec003980 tx=0x7f8bec00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:09.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 client.?.objecter ms_handle_connect 0x7f8bf0096080 2023-10-30T12:13:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 1 -- 172.21.15.50:0/3774271209 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec003e10 con 0x7f8bf0096080 2023-10-30T12:13:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:09.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: dump: 2023-10-30T12:13:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.446 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:09.447 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:09.448 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:09.448 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:09.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: _finish_auth 0 2023-10-30T12:13:09.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:13:09.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:09.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 1 -- 172.21.15.50:0/3774271209 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec005070 con 0x7f8bf0096080 2023-10-30T12:13:09.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:09.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 1 -- 172.21.15.50:0/3774271209 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec005400 con 0x7f8bf0096080 2023-10-30T12:13:09.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 5 monclient: authenticate success, global_id 174174 2023-10-30T12:13:09.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 10 monclient: _renew_subs 2023-10-30T12:13:09.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:09.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf008d0f0 con 0x7f8bf0096080 2023-10-30T12:13:09.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be3fff640 4 set_mon_vals no callback set 2023-10-30T12:13:09.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:09.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:09.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8be27fc640 10 monclient: dump: 2023-10-30T12:13:09.451 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:09.451 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:09.453 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:09.453 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:09.453 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:09.453 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:09.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 10 client.174174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:09.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 10 monclient: _renew_subs 2023-10-30T12:13:09.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:09.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf01c8f90 con 0x7f8bf0096080 2023-10-30T12:13:09.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.398+0000 7f8bf7af7640 1 librados: init done 2023-10-30T12:13:09.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8be27fc640 1 -- 172.21.15.50:0/3774271209 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8bec005560 con 0x7f8bf0096080 2023-10-30T12:13:09.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8bf7af7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:09.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8be27fc640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8bd40392a0 0x7f8bd403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:09.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8bf7af7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:09.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8bf7af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:09.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf004e160 con 0x7f8bf0096080 2023-10-30T12:13:09.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8be27fc640 1 -- 172.21.15.50:0/3774271209 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(400..400 src has 1..400) v4 ==== 6682+0+0 (secure 0 0 0) 0x7f8bec04a250 con 0x7f8bf0096080 2023-10-30T12:13:09.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8be27fc640 10 client.174174.objecter ms_dispatch 0x7f8bf0000b90 osd_map(400..400 src has 1..400) v4 2023-10-30T12:13:09.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8be27fc640 3 client.174174.objecter handle_osd_map got epochs [400,400] > 0 2023-10-30T12:13:09.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8be27fc640 3 client.174174.objecter handle_osd_map decoding full epoch 400 2023-10-30T12:13:09.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.399+0000 7f8be27fc640 20 client.174174.objecter dump_active .. 0 homeless 2023-10-30T12:13:09.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.400+0000 7f8bf5092640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8bd40392a0 0x7f8bd403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:09.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.400+0000 7f8bf5092640 10 monclient: get_auth_request con 0x7f8bd40392a0 auth_method 0 2023-10-30T12:13:09.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.400+0000 7f8bf5092640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8bd40392a0 0x7f8bd403b760 secure :-1 s=READY pgs=4153 cs=0 l=1 rev1=1 crypto rx=0x7f8be4006fd0 tx=0x7f8be4006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:09.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.400+0000 7f8be27fc640 10 client.174174.objecter ms_handle_connect 0x7f8bd40392a0 2023-10-30T12:13:09.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.404+0000 7f8be27fc640 1 -- 172.21.15.50:0/3774271209 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8bec016020 con 0x7f8bf0096080 2023-10-30T12:13:09.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.404+0000 7f8be27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:09.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.404+0000 7f8be27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.648+0000 7f8bf7af7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.648+0000 7f8bf7af7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.648+0000 7f8bf7af7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.648+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8bf00745c0 con 0x7f8bf0096080 2023-10-30T12:13:09.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.648+0000 7f8be27fc640 1 -- 172.21.15.50:0/3774271209 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f8bec018cd0 con 0x7f8bf0096080 2023-10-30T12:13:09.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.649+0000 7f8be27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:09.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.649+0000 7f8be27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:09.650 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:09.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 10 librados: watch_flush enter 2023-10-30T12:13:09.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 10 librados: watch_flush exit 2023-10-30T12:13:09.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 20 client.174174.objecter shutdown clearing up homeless session... 2023-10-30T12:13:09.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 10 client.174174.objecter successfully canceled tick 2023-10-30T12:13:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8bd40392a0 msgr2=0x7f8bd403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8bd40392a0 0x7f8bd403b760 secure :-1 s=READY pgs=4153 cs=0 l=1 rev1=1 crypto rx=0x7f8be4006fd0 tx=0x7f8be4006d80 comp rx=0 tx=0).stop 2023-10-30T12:13:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 10 monclient: shutdown 2023-10-30T12:13:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.650+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bf0096080 msgr2=0x7f8bf008da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bf0096080 0x7f8bf008da00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8bec003980 tx=0x7f8bec00b8e0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 shutdown_connections 2023-10-30T12:13:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8bd40392a0 0x7f8bd403b760 unknown :-1 s=CLOSED pgs=4153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bf009f5b0 0x7f8bf01c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bf0096ba0 0x7f8bf01c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 --2- 172.21.15.50:0/3774271209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bf0096080 0x7f8bf008da00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 >> 172.21.15.50:0/3774271209 conn(0x7f8bf00a05c0 msgr2=0x7f8bf0138930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 shutdown_connections 2023-10-30T12:13:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 -- 172.21.15.50:0/3774271209 wait complete. 2023-10-30T12:13:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:09.651+0000 7f8bf7af7640 1 librados: shutdown 2023-10-30T12:13:12.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.778+0000 7f4c52716640 10 monclient: get_monmap_and_config 2023-10-30T12:13:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.778+0000 7f4c52716640 10 monclient: build_initial_monmap 2023-10-30T12:13:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.778+0000 7f4c52716640 10 monclient: monmap: 2023-10-30T12:13:12.780 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:12.781 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:12.781 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:12.779780+0000 2023-10-30T12:13:12.781 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:12.779780+0000 2023-10-30T12:13:12.781 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:12.781 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:12.782 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:12.782 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:12.782 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:12.782 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:12.782 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:12.783 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:12.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.779+0000 7f4c52716640 1 Processor -- start 2023-10-30T12:13:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.779+0000 7f4c52716640 1 -- start start 2023-10-30T12:13:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.779+0000 7f4c52716640 10 monclient: init 2023-10-30T12:13:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T12:13:12.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c4c0707d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:12.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4c4c0712c0 0x7f4c4c09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:12.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient(hunting): picked mon.noname-c con 0x7f4c4c0712c0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:12.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c13aba0 0x7f4c4c13cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:12.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient(hunting): picked mon.noname-b con 0x7f4c4c13aba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:12.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c09ee40 con 0x7f4c4c0707d0 2023-10-30T12:13:12.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:12.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c09f4f0 con 0x7f4c4c0712c0 2023-10-30T12:13:12.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:12.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c09f7d0 con 0x7f4c4c13aba0 2023-10-30T12:13:12.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:12.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c52716640 10 monclient(hunting): authenticate will time out at 1637.963133s 2023-10-30T12:13:12.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c4bfff640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f4c4c0707d0 legacy=0x7f4c4c070bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:13:12.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c4b7fe640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f4c4c0712c0 msgr2=0x7f4c4c09e7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:12.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c4b7fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4c4c0712c0 0x7f4c4c09e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:12.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c13aba0 0x7f4c4c13cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:12.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c13aba0 0x7f4c4c13cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41444/0 (socket says 172.21.15.50:41444) 2023-10-30T12:13:12.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 -- 172.21.15.50:0/2819749972 learned_addr learned my addr 172.21.15.50:0/2819749972 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:12.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): get_auth_request con 0x7f4c4c13aba0 auth_method 0 2023-10-30T12:13:12.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:12.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:12.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:12.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:12.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:12.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:12.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient(hunting): handle_auth_done global_id 114803 payload 274 2023-10-30T12:13:12.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 -- 172.21.15.50:0/2819749972 >> v2:172.21.15.118:3300/0 conn(0x7f4c4c0712c0 msgr2=0x7f4c4c09e7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:12.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 --2- 172.21.15.50:0/2819749972 >> v2:172.21.15.118:3300/0 conn(0x7f4c4c0712c0 0x7f4c4c09e7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:12.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 -- 172.21.15.50:0/2819749972 >> v1:172.21.15.118:6789/0 conn(0x7f4c4c0707d0 legacy=0x7f4c4c070bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:12.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient: _finish_hunting 0 2023-10-30T12:13:12.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 monclient: found mon.noname-b 2023-10-30T12:13:12.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:12.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:12.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.780+0000 7f4c50cb3640 1 -- 172.21.15.50:0/2819749972 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c09fab0 con 0x7f4c4c13aba0 2023-10-30T12:13:12.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c50cb3640 1 --2- 172.21.15.50:0/2819749972 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c13aba0 0x7f4c4c13cf90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009e50 tx=0x7f4c4402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df17ae989b34388e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:12.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 1 -- 172.21.15.50:0/2819749972 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4402c980 con 0x7f4c4c13aba0 2023-10-30T12:13:12.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:12.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:12.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: dump: 2023-10-30T12:13:12.798 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:12.798 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:12.798 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.798 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.798 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:12.799 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:12.799 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:12.799 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:12.799 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:12.799 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:12.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: _finish_auth 0 2023-10-30T12:13:12.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:12.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:12.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 1 -- 172.21.15.50:0/2819749972 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4402cae0 con 0x7f4c4c13aba0 2023-10-30T12:13:12.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:12.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 1 -- 172.21.15.50:0/2819749972 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4402ce70 con 0x7f4c4c13aba0 2023-10-30T12:13:12.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c52716640 5 monclient: authenticate success, global_id 114803 2023-10-30T12:13:12.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c52716640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:12.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c52716640 4 set_mon_vals no callback set 2023-10-30T12:13:12.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:12.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:12.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.781+0000 7f4c4affd640 10 monclient: dump: 2023-10-30T12:13:12.802 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:12.802 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:12.802 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.802 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.802 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:12.803 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:12.803 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:12.803 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:12.803 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:12.803 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 10 monclient: shutdown 2023-10-30T12:13:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 -- 172.21.15.50:0/2819749972 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c13aba0 msgr2=0x7f4c4c13cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 --2- 172.21.15.50:0/2819749972 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c13aba0 0x7f4c4c13cf90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009e50 tx=0x7f4c4402bde0 comp rx=0 tx=0).stop 2023-10-30T12:13:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 -- 172.21.15.50:0/2819749972 shutdown_connections 2023-10-30T12:13:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 --2- 172.21.15.50:0/2819749972 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c13aba0 0x7f4c4c13cf90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:12.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 --2- 172.21.15.50:0/2819749972 >> v2:172.21.15.118:3300/0 conn(0x7f4c4c0712c0 0x7f4c4c09e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:12.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 -- 172.21.15.50:0/2819749972 >> 172.21.15.50:0/2819749972 conn(0x7f4c4c0a47f0 msgr2=0x7f4c4c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:12.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 -- 172.21.15.50:0/2819749972 shutdown_connections 2023-10-30T12:13:12.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 -- 172.21.15.50:0/2819749972 wait complete. 2023-10-30T12:13:12.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 10 monclient: build_initial_monmap 2023-10-30T12:13:12.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 10 monclient: monmap: 2023-10-30T12:13:12.806 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:12.806 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:12.806 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:12.806 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:12.807 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:12.807 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:12.807 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:12.807 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:12.807 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:12.808 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 librados: starting msgr at 2023-10-30T12:13:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.782+0000 7f4c52716640 1 librados: starting objecter 2023-10-30T12:13:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 Processor -- start 2023-10-30T12:13:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 -- start start 2023-10-30T12:13:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 librados: setting wanted keys 2023-10-30T12:13:12.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 librados: calling monclient init 2023-10-30T12:13:12.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient: init 2023-10-30T12:13:12.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:12.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:13:12.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c4c0707d0 0x7f4c4c1ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): picked mon.noname-b con 0x7f4c4c0707d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c0712c0 0x7f4c4c11cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): picked mon.noname-c con 0x7f4c4c0712c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c4c13aba0 0x7f4c4c11d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): picked mon.noname-a con 0x7f4c4c13aba0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c120290 con 0x7f4c4c13aba0 2023-10-30T12:13:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c11f410 con 0x7f4c4c0707d0 2023-10-30T12:13:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c09d6a0 con 0x7f4c4c0712c0 2023-10-30T12:13:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c52716640 10 monclient(hunting): authenticate will time out at 1637.966794s 2023-10-30T12:13:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4bfff640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c4c0707d0 msgr2=0x7f4c4c1ca920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4bfff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c4c0707d0 0x7f4c4c1ca920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c0712c0 0x7f4c4c11cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c0712c0 0x7f4c4c11cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41458/0 (socket says 172.21.15.50:41458) 2023-10-30T12:13:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4b7fe640 1 -- 172.21.15.50:0/3272811063 learned_addr learned my addr 172.21.15.50:0/3272811063 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4b7fe640 10 monclient(hunting): get_auth_request con 0x7f4c4c0712c0 auth_method 0 2023-10-30T12:13:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c50cb3640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c4c13aba0 0x7f4c4c11d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c50cb3640 10 monclient(hunting): get_auth_request con 0x7f4c4c13aba0 auth_method 0 2023-10-30T12:13:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c50cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c50cb3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.783+0000 7f4c50cb3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c50cb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c50cb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:12.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c50cb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:12.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 10 monclient(hunting): handle_auth_done global_id 114806 payload 824 2023-10-30T12:13:12.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 1 -- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c4c0707d0 msgr2=0x7f4c4c1ca920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:12.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c4c0707d0 0x7f4c4c1ca920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:12.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 1 -- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c4c13aba0 msgr2=0x7f4c4c11d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:12.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c4c13aba0 0x7f4c4c11d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:12.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:13:12.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 1 monclient: found mon.noname-c 2023-10-30T12:13:12.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:12.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:12.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 1 -- 172.21.15.50:0/3272811063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c44009b30 con 0x7f4c4c0712c0 2023-10-30T12:13:12.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c50cb3640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c4c13aba0 0x7f4c4c11d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:12.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4b7fe640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c0712c0 0x7f4c4c11cb00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400b580 tx=0x7f4c3400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:12.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 client.?.objecter ms_handle_connect 0x7f4c4c0712c0 2023-10-30T12:13:12.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:12.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 1 -- 172.21.15.50:0/3272811063 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c34003dc0 con 0x7f4c4c0712c0 2023-10-30T12:13:12.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:12.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:12.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: dump: 2023-10-30T12:13:12.819 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:12.819 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:12.819 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.820 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.820 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:12.820 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:12.820 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:12.820 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:12.820 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:12.820 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:12.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: _finish_auth 0 2023-10-30T12:13:12.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:13:12.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:12.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 1 -- 172.21.15.50:0/3272811063 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c34005070 con 0x7f4c4c0712c0 2023-10-30T12:13:12.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 5 monclient: authenticate success, global_id 114806 2023-10-30T12:13:12.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:12.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 10 monclient: _renew_subs 2023-10-30T12:13:12.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 1 -- 172.21.15.50:0/3272811063 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c34005420 con 0x7f4c4c0712c0 2023-10-30T12:13:12.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:12.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c09b490 con 0x7f4c4c0712c0 2023-10-30T12:13:12.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:12.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:12.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c48ff9640 10 monclient: dump: 2023-10-30T12:13:12.823 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:12.823 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:12.823 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.823 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:12.824 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:12.824 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:12.824 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:12.824 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:12.824 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:12.824 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:12.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c4a7fc640 4 set_mon_vals no callback set 2023-10-30T12:13:12.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 10 client.114806.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:12.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 10 monclient: _renew_subs 2023-10-30T12:13:12.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:12.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c120900 con 0x7f4c4c0712c0 2023-10-30T12:13:12.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.784+0000 7f4c52716640 1 librados: init done 2023-10-30T12:13:12.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c48ff9640 1 -- 172.21.15.50:0/3272811063 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4c34005580 con 0x7f4c4c0712c0 2023-10-30T12:13:12.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c52716640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:12.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c52716640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:12.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c52716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:12.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c1203d0 con 0x7f4c4c0712c0 2023-10-30T12:13:12.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c48ff9640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c2c0392a0 0x7f4c2c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:12.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c48ff9640 1 -- 172.21.15.50:0/3272811063 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(400..400 src has 1..400) v4 ==== 6682+0+0 (secure 0 0 0) 0x7f4c3404aec0 con 0x7f4c4c0712c0 2023-10-30T12:13:12.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c48ff9640 10 client.114806.objecter ms_dispatch 0x7f4c4c000b90 osd_map(400..400 src has 1..400) v4 2023-10-30T12:13:12.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c48ff9640 3 client.114806.objecter handle_osd_map got epochs [400,400] > 0 2023-10-30T12:13:12.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.785+0000 7f4c48ff9640 3 client.114806.objecter handle_osd_map decoding full epoch 400 2023-10-30T12:13:12.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.786+0000 7f4c48ff9640 20 client.114806.objecter dump_active .. 0 homeless 2023-10-30T12:13:12.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.786+0000 7f4c4bfff640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c2c0392a0 0x7f4c2c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:12.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.786+0000 7f4c4bfff640 10 monclient: get_auth_request con 0x7f4c2c0392a0 auth_method 0 2023-10-30T12:13:12.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.786+0000 7f4c4bfff640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c2c0392a0 0x7f4c2c03b760 secure :-1 s=READY pgs=4154 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c006fd0 tx=0x7f4c3c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:12.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.786+0000 7f4c48ff9640 10 client.114806.objecter ms_handle_connect 0x7f4c2c0392a0 2023-10-30T12:13:12.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.790+0000 7f4c48ff9640 1 -- 172.21.15.50:0/3272811063 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4c4c1203d0 con 0x7f4c4c0712c0 2023-10-30T12:13:12.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.790+0000 7f4c48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:12.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:12.790+0000 7f4c48ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:13.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.033+0000 7f4c52716640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:13.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.033+0000 7f4c52716640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:13.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.033+0000 7f4c52716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:13.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.033+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4c4c1203d0 con 0x7f4c4c0712c0 2023-10-30T12:13:13.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.033+0000 7f4c48ff9640 1 -- 172.21.15.50:0/3272811063 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f4c4c1203d0 con 0x7f4c4c0712c0 2023-10-30T12:13:13.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.033+0000 7f4c48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:13.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.033+0000 7f4c48ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:13.037 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:13.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 10 librados: watch_flush enter 2023-10-30T12:13:13.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 10 librados: watch_flush exit 2023-10-30T12:13:13.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 20 client.114806.objecter shutdown clearing up homeless session... 2023-10-30T12:13:13.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 10 client.114806.objecter successfully canceled tick 2023-10-30T12:13:13.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c2c0392a0 msgr2=0x7f4c2c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:13.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c2c0392a0 0x7f4c2c03b760 secure :-1 s=READY pgs=4154 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c006fd0 tx=0x7f4c3c006d80 comp rx=0 tx=0).stop 2023-10-30T12:13:13.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 10 monclient: shutdown 2023-10-30T12:13:13.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:13.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c0712c0 msgr2=0x7f4c4c11cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:13.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c0712c0 0x7f4c4c11cb00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400b580 tx=0x7f4c3400ba40 comp rx=0 tx=0).stop 2023-10-30T12:13:13.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 shutdown_connections 2023-10-30T12:13:13.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4c2c0392a0 0x7f4c2c03b760 unknown :-1 s=CLOSED pgs=4154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:13.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c4c13aba0 0x7f4c4c11d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:13.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4c4c0712c0 0x7f4c4c11cb00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:13.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 --2- 172.21.15.50:0/3272811063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4c4c0707d0 0x7f4c4c1ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:13.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 >> 172.21.15.50:0/3272811063 conn(0x7f4c4c0a47f0 msgr2=0x7f4c4c0a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:13.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 shutdown_connections 2023-10-30T12:13:13.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 -- 172.21.15.50:0/3272811063 wait complete. 2023-10-30T12:13:13.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:13.035+0000 7f4c52716640 1 librados: shutdown 2023-10-30T12:13:16.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.163+0000 7ff22d78a640 10 monclient: get_monmap_and_config 2023-10-30T12:13:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.163+0000 7ff22d78a640 10 monclient: build_initial_monmap 2023-10-30T12:13:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.163+0000 7ff22d78a640 10 monclient: monmap: 2023-10-30T12:13:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:16.167 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:16.165029+0000 2023-10-30T12:13:16.167 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:16.165029+0000 2023-10-30T12:13:16.167 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:16.167 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:16.167 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:16.168 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:16.168 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:16.168 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:16.168 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:16.168 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:16.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.164+0000 7ff22d78a640 1 Processor -- start 2023-10-30T12:13:16.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.164+0000 7ff22d78a640 1 -- start start 2023-10-30T12:13:16.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.164+0000 7ff22d78a640 10 monclient: init 2023-10-30T12:13:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-10-30T12:13:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient(hunting): picked mon.noname-a con 0x7ff2280a7000 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff22806abe0 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:16.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 0x7ff2280a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:16.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient(hunting): picked mon.noname-c con 0x7ff2281360c0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:16.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280a73e0 con 0x7ff22806abe0 2023-10-30T12:13:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff228136530 con 0x7ff2280a7000 2023-10-30T12:13:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22809a000 con 0x7ff2281360c0 2023-10-30T12:13:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff22d78a640 10 monclient(hunting): authenticate will time out at 1641.348118s 2023-10-30T12:13:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff2277fe640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 msgr2=0x7ff2280a4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff2267fc640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7ff22806abe0 legacy=0x7ff2280a4460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:13:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41746/0 (socket says 172.21.15.50:41746) 2023-10-30T12:13:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff2277fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 0x7ff2280a4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 learned_addr learned my addr 172.21.15.50:0/4267804822 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 10 monclient(hunting): get_auth_request con 0x7ff2280a7000 auth_method 0 2023-10-30T12:13:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:13:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.165+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:16.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:16.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 10 monclient(hunting): get_auth_request con 0x7ff2280a7000 auth_method 0 2023-10-30T12:13:16.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:16.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> v1:172.21.15.118:6789/0 conn(0x7ff22806abe0 legacy=0x7ff2280a4460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:13:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff2277fe640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 msgr2=0x7ff2280a4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff2277fe640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 0x7ff2280a4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:13:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:13:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.366+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:13:16.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:16.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff2277fe640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 msgr2=0x7ff2280a4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:16.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> v1:172.21.15.118:6789/0 conn(0x7ff22806abe0 legacy=0x7ff2280a4460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:13:16.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 10 monclient(hunting): get_auth_request con 0x7ff2280a7000 auth_method 0 2023-10-30T12:13:16.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:16.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:16.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:16.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff2277fe640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 0x7ff2280a4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:13:16.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:13:16.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:16.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:16.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:16.767+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:13:17.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> v1:172.21.15.118:6789/0 conn(0x7ff22806abe0 legacy=0x7ff2280a4460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:13:17.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff2277fe640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 msgr2=0x7ff2280a4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:17.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff2277fe640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 0x7ff2280a4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:13:17.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:17.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 10 monclient(hunting): get_auth_request con 0x7ff2280a7000 auth_method 0 2023-10-30T12:13:17.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:17.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:17.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:17.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:13:17.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:17.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:17.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:17.568+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:13:19.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient(hunting): tick 2023-10-30T12:13:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:13:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:13:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:13:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 msgr2=0x7ff2280a4b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.118:3300/0 conn(0x7ff2281360c0 0x7ff2280a4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 -- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 msgr2=0x7ff228135b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 --2- 172.21.15.50:0/4267804822 >> v2:172.21.15.50:3300/0 conn(0x7ff2280a7000 0x7ff228135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 -- 172.21.15.50:0/4267804822 >> v1:172.21.15.118:6789/0 conn(0x7ff22806abe0 legacy=0x7ff2280a4460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient: _add_conns ranks=[0,2] 2023-10-30T12:13:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2140036b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:19.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:19.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient(hunting): picked mon.noname-b con 0x7ff2140072b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:19.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 -- 172.21.15.50:0/4267804822 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22809a2e0 con 0x7ff2140036b0 2023-10-30T12:13:19.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:19.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 1 -- 172.21.15.50:0/4267804822 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff228136530 con 0x7ff2140072b0 2023-10-30T12:13:19.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff224ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:19.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:19.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 10 monclient(hunting): get_auth_request con 0x7ff2140072b0 auth_method 0 2023-10-30T12:13:19.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:19.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:19.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:19.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:13:19.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:19.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:19.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.165+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:19.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:19.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 10 monclient(hunting): get_auth_request con 0x7ff2140072b0 auth_method 0 2023-10-30T12:13:19.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:19.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:19.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:19.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:13:19.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:19.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:19.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.366+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:13:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 10 monclient(hunting): get_auth_request con 0x7ff2140072b0 auth_method 0 2023-10-30T12:13:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:13:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:19.767+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:13:20.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:20.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 10 monclient(hunting): get_auth_request con 0x7ff2140072b0 auth_method 0 2023-10-30T12:13:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:13:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:20.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:20.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.568+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:13:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.789+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764867760 0 0) 0x7ff22809a2e0 con 0x7ff2140036b0 2023-10-30T12:13:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.789+0000 7ff225ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:13:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.789+0000 7ff225ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.789+0000 7ff225ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.789+0000 7ff225ffb640 10 monclient(hunting): my global_id is 174180 2023-10-30T12:13:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.789+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff210003520 con 0x7ff2140036b0 2023-10-30T12:13:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3306793649 0 0) 0x7ff210003520 con 0x7ff2140036b0 2023-10-30T12:13:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 msgr2=0x7ff214009770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2140072b0 0x7ff214009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: _finish_hunting 0 2023-10-30T12:13:20.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:13:20.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:20.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:13:20.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280a73e0 con 0x7ff2140036b0 2023-10-30T12:13:20.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: _finish_auth 0 2023-10-30T12:13:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:13:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7ff21800c570 con 0x7ff2140036b0 2023-10-30T12:13:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:13:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: dump: 2023-10-30T12:13:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:20.798 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:13:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 >> v1:172.21.15.50:6789/0 conn(0x7ff2140036b0 legacy=0x7ff214005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff2277fe640 1 -- 172.21.15.50:0/4267804822 reap_dead start 2023-10-30T12:13:20.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:13:20.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210003970 0x7ff210003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:20.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): picked mon.c con 0x7ff210003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:20.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff210004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:20.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): picked mon.b con 0x7ff2100047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210005180 0x7ff21000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): picked mon.a con 0x7ff210005180 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff228136530 con 0x7ff210005180 2023-10-30T12:13:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:20.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21000c0b0 con 0x7ff2100047c0 2023-10-30T12:13:20.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:20.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21000c270 con 0x7ff210003970 2023-10-30T12:13:20.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.791+0000 7ff225ffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:13:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2267fc640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 msgr2=0x7ff210004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2267fc640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff210004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210005180 0x7ff21000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:20.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): get_auth_request con 0x7ff210005180 auth_method 0 2023-10-30T12:13:20.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:20.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:20.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:20.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff226ffd640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210003970 0x7ff210003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:20.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff226ffd640 10 monclient(hunting): get_auth_request con 0x7ff210003970 auth_method 0 2023-10-30T12:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff226ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff226ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff226ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient(hunting): handle_auth_done global_id 174180 payload 293 2023-10-30T12:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210003970 msgr2=0x7ff210003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210003970 0x7ff210003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 msgr2=0x7ff210004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff210004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient: _finish_hunting 0 2023-10-30T12:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 1 monclient: found mon.a 2023-10-30T12:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 1 -- 172.21.15.50:0/4267804822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280a73e0 con 0x7ff210005180 2023-10-30T12:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff2277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.792+0000 7ff22d78a640 5 monclient: authenticate success, global_id 174180 2023-10-30T12:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.793+0000 7ff22d78a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.794+0000 7ff2277fe640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210005180 0x7ff21000b980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff22806a6d0 tx=0x7ff2100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94f0295df2d42d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.794+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22002c8f0 con 0x7ff210005180 2023-10-30T12:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.794+0000 7ff225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.794+0000 7ff225ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.794+0000 7ff225ffb640 10 monclient: dump: 2023-10-30T12:13:20.815 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:20.815 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:20.815 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.815 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.815 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:20.815 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.794+0000 7ff22d78a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.794+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22002ca50 con 0x7ff210005180 2023-10-30T12:13:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff225ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff225ffb640 1 -- 172.21.15.50:0/4267804822 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22002ce00 con 0x7ff210005180 2023-10-30T12:13:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 4 set_mon_vals no callback set 2023-10-30T12:13:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff225ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff225ffb640 10 monclient: dump: 2023-10-30T12:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 10 monclient: shutdown 2023-10-30T12:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 1 -- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210005180 msgr2=0x7ff21000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210005180 0x7ff21000b980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff22806a6d0 tx=0x7ff2100057f0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 1 -- 172.21.15.50:0/4267804822 shutdown_connections 2023-10-30T12:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210005180 0x7ff21000b980 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff210004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 1 --2- 172.21.15.50:0/4267804822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210003970 0x7ff210003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.795+0000 7ff22d78a640 1 -- 172.21.15.50:0/4267804822 >> 172.21.15.50:0/4267804822 conn(0x7ff228066000 msgr2=0x7ff2280a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.796+0000 7ff22d78a640 1 -- 172.21.15.50:0/4267804822 shutdown_connections 2023-10-30T12:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.796+0000 7ff22d78a640 1 -- 172.21.15.50:0/4267804822 wait complete. 2023-10-30T12:13:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.796+0000 7ff22d78a640 10 monclient: build_initial_monmap 2023-10-30T12:13:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.796+0000 7ff22d78a640 10 monclient: monmap: 2023-10-30T12:13:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:20.823 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.797+0000 7ff22d78a640 1 librados: starting msgr at 2023-10-30T12:13:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.797+0000 7ff22d78a640 1 librados: starting objecter 2023-10-30T12:13:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.797+0000 7ff22d78a640 1 Processor -- start 2023-10-30T12:13:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.797+0000 7ff22d78a640 1 -- start start 2023-10-30T12:13:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.797+0000 7ff22d78a640 1 librados: setting wanted keys 2023-10-30T12:13:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.797+0000 7ff22d78a640 1 librados: calling monclient init 2023-10-30T12:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.797+0000 7ff22d78a640 10 monclient: init 2023-10-30T12:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210003970 0x7ff2281c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): picked mon.noname-a con 0x7ff210003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff228132020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): picked mon.noname-b con 0x7ff2100047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210005180 0x7ff22809e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): picked mon.noname-c con 0x7ff210005180 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2281bd540 con 0x7ff210003970 2023-10-30T12:13:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210003970 0x7ff2281c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210003970 0x7ff2281c9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55912/0 (socket says 172.21.15.50:55912) 2023-10-30T12:13:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 1 -- 172.21.15.50:0/700102143 learned_addr learned my addr 172.21.15.50:0/700102143 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2267fc640 1 -- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 msgr2=0x7ff228132020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280a0440 con 0x7ff2100047c0 2023-10-30T12:13:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22811f420 con 0x7ff210005180 2023-10-30T12:13:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff22d78a640 10 monclient(hunting): authenticate will time out at 1645.981659s 2023-10-30T12:13:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 10 monclient(hunting): get_auth_request con 0x7ff210003970 auth_method 0 2023-10-30T12:13:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2267fc640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff228132020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210005180 0x7ff22809e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 10 monclient(hunting): get_auth_request con 0x7ff210005180 auth_method 0 2023-10-30T12:13:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff226ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.798+0000 7ff2277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 10 monclient(hunting): handle_auth_done global_id 174183 payload 824 2023-10-30T12:13:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 1 -- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210005180 msgr2=0x7ff22809e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210005180 0x7ff22809e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 1 -- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 msgr2=0x7ff228132020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff228132020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:13:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 1 monclient: found mon.noname-a 2023-10-30T12:13:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 1 -- 172.21.15.50:0/700102143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22811f700 con 0x7ff210003970 2023-10-30T12:13:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff226ffd640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210003970 0x7ff2281c9b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff2180169e0 tx=0x7ff218016ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff20f7fe640 10 client.?.objecter ms_handle_connect 0x7ff210003970 2023-10-30T12:13:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.799+0000 7ff20f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 1 -- 172.21.15.50:0/700102143 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218014cb0 con 0x7ff210003970 2023-10-30T12:13:20.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:20.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:20.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: dump: 2023-10-30T12:13:20.834 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:20.834 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:20.835 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.835 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.835 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:20.835 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:20.835 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:20.836 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:20.836 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:20.836 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:20.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:20.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:20.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:20.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 5 monclient: authenticate success, global_id 174183 2023-10-30T12:13:20.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 1 -- 172.21.15.50:0/700102143 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21800c040 con 0x7ff210003970 2023-10-30T12:13:20.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:20.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 1 -- 172.21.15.50:0/700102143 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218004270 con 0x7ff210003970 2023-10-30T12:13:20.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 10 monclient: _renew_subs 2023-10-30T12:13:20.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:20.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22811f140 con 0x7ff210003970 2023-10-30T12:13:20.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:20.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:20.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff20f7fe640 10 monclient: dump: 2023-10-30T12:13:20.839 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:20.839 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:20.839 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.839 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:20.839 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:20.840 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff2257fa640 4 set_mon_vals no callback set 2023-10-30T12:13:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 10 client.174183.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 10 monclient: _renew_subs 2023-10-30T12:13:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2281ca950 con 0x7ff210003970 2023-10-30T12:13:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.800+0000 7ff22d78a640 1 librados: init done 2023-10-30T12:13:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff20f7fe640 1 -- 172.21.15.50:0/700102143 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff2180043d0 con 0x7ff210003970 2023-10-30T12:13:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff22d78a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff22d78a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff22d78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22804e160 con 0x7ff210003970 2023-10-30T12:13:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff20f7fe640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff214039910 0x7ff21403bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff20f7fe640 1 -- 172.21.15.50:0/700102143 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(400..400 src has 1..400) v4 ==== 6682+0+0 (secure 0 0 0) 0x7ff21804b880 con 0x7ff210003970 2023-10-30T12:13:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff20f7fe640 10 client.174183.objecter ms_dispatch 0x7ff228000b90 osd_map(400..400 src has 1..400) v4 2023-10-30T12:13:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff20f7fe640 3 client.174183.objecter handle_osd_map got epochs [400,400] > 0 2023-10-30T12:13:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff20f7fe640 3 client.174183.objecter handle_osd_map decoding full epoch 400 2023-10-30T12:13:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff20f7fe640 20 client.174183.objecter dump_active .. 0 homeless 2023-10-30T12:13:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff2267fc640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff214039910 0x7ff21403bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.801+0000 7ff2267fc640 10 monclient: get_auth_request con 0x7ff214039910 auth_method 0 2023-10-30T12:13:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.802+0000 7ff2267fc640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff214039910 0x7ff21403bdd0 secure :-1 s=READY pgs=4155 cs=0 l=1 rev1=1 crypto rx=0x7ff22811e6d0 tx=0x7ff21c00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.802+0000 7ff20f7fe640 10 client.174183.objecter ms_handle_connect 0x7ff214039910 2023-10-30T12:13:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.806+0000 7ff20f7fe640 1 -- 172.21.15.50:0/700102143 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff218003610 con 0x7ff210003970 2023-10-30T12:13:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.806+0000 7ff20f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:20.806+0000 7ff20f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.052+0000 7ff22d78a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:21.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.052+0000 7ff22d78a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:21.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.052+0000 7ff22d78a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:21.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.052+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff2280745c0 con 0x7ff210003970 2023-10-30T12:13:21.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.052+0000 7ff20f7fe640 1 -- 172.21.15.50:0/700102143 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7ff21802ab40 con 0x7ff210003970 2023-10-30T12:13:21.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.052+0000 7ff20f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:21.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.052+0000 7ff20f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:21.055 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":128,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:21.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 10 librados: watch_flush enter 2023-10-30T12:13:21.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 10 librados: watch_flush exit 2023-10-30T12:13:21.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 20 client.174183.objecter shutdown clearing up homeless session... 2023-10-30T12:13:21.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 10 client.174183.objecter successfully canceled tick 2023-10-30T12:13:21.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff214039910 msgr2=0x7ff21403bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff214039910 0x7ff21403bdd0 secure :-1 s=READY pgs=4155 cs=0 l=1 rev1=1 crypto rx=0x7ff22811e6d0 tx=0x7ff21c00d040 comp rx=0 tx=0).stop 2023-10-30T12:13:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 10 monclient: shutdown 2023-10-30T12:13:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210003970 msgr2=0x7ff2281c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210003970 0x7ff2281c9b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff2180169e0 tx=0x7ff218016ea0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 shutdown_connections 2023-10-30T12:13:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff214039910 0x7ff21403bdd0 unknown :-1 s=CLOSED pgs=4155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff210005180 0x7ff22809e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff2100047c0 0x7ff228132020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 --2- 172.21.15.50:0/700102143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff210003970 0x7ff2281c9b30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 >> 172.21.15.50:0/700102143 conn(0x7ff228066000 msgr2=0x7ff2280a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 shutdown_connections 2023-10-30T12:13:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 -- 172.21.15.50:0/700102143 wait complete. 2023-10-30T12:13:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.054+0000 7ff22d78a640 1 librados: shutdown 2023-10-30T12:13:21.065 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:13:21.066 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:13:21.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:13:21.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.171+0000 7face9009640 10 monclient: get_monmap_and_config 2023-10-30T12:13:21.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.171+0000 7face9009640 10 monclient: build_initial_monmap 2023-10-30T12:13:21.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.171+0000 7face9009640 10 monclient: monmap: 2023-10-30T12:13:21.173 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:21.173 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.173 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:21.172297+0000 2023-10-30T12:13:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:21.172297+0000 2023-10-30T12:13:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:21.175 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:21.175 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:21.175 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.171+0000 7face9009640 1 Processor -- start 2023-10-30T12:13:21.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.171+0000 7face9009640 1 -- start start 2023-10-30T12:13:21.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.171+0000 7face9009640 10 monclient: init 2023-10-30T12:13:21.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:21.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T12:13:21.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face40a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient(hunting): picked mon.noname-b con 0x7face40a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:21.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7face40a2ec0 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:21.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7face40a32e0 0x7face4135df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient(hunting): picked mon.noname-a con 0x7face40a32e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:21.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face40a4190 con 0x7face40a2ec0 2023-10-30T12:13:21.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face40a2990 con 0x7face40a32e0 2023-10-30T12:13:21.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face40a1630 con 0x7face40a2ae0 2023-10-30T12:13:21.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:21.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face9009640 10 monclient(hunting): authenticate will time out at 1646.355409s 2023-10-30T12:13:21.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face40a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7face40a32e0 0x7face4135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7facdbfff640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7face40a2ec0 legacy=0x7face41356e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:13:21.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face40a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46668/0 (socket says 172.21.15.50:46668) 2023-10-30T12:13:21.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 1 -- 172.21.15.50:0/2581238647 learned_addr learned my addr 172.21.15.50:0/2581238647 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:21.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): get_auth_request con 0x7face40a32e0 auth_method 0 2023-10-30T12:13:21.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 10 monclient(hunting): get_auth_request con 0x7face40a2ae0 auth_method 0 2023-10-30T12:13:21.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient(hunting): handle_auth_done global_id 174186 payload 274 2023-10-30T12:13:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 1 -- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 msgr2=0x7face40a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 1 --2- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face40a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 1 -- 172.21.15.50:0/2581238647 >> v1:172.21.15.118:6789/0 conn(0x7face40a2ec0 legacy=0x7face41356e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient: _finish_hunting 0 2023-10-30T12:13:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 1 monclient: found mon.noname-a 2023-10-30T12:13:21.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:21.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:21.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face359e640 1 -- 172.21.15.50:0/2581238647 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face409a2e0 con 0x7face40a32e0 2023-10-30T12:13:21.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.172+0000 7face2d9d640 1 --2- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face40a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:21.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face359e640 1 --2- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face4135df0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7facdc009d90 tx=0x7facdc02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84b85070aea66f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 1 -- 172.21.15.50:0/2581238647 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc02cab0 con 0x7face40a32e0 2023-10-30T12:13:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: dump: 2023-10-30T12:13:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.195 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.195 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.195 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.195 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.195 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.195 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.196 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.196 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.196 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: _finish_auth 0 2023-10-30T12:13:21.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: _check_auth_tickets 2023-10-30T12:13:21.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:21.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face9009640 5 monclient: authenticate success, global_id 174186 2023-10-30T12:13:21.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 1 -- 172.21.15.50:0/2581238647 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc02cc10 con 0x7face40a32e0 2023-10-30T12:13:21.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face9009640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:21.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:21.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face9009640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:21.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face9009640 4 set_mon_vals no callback set 2023-10-30T12:13:21.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 1 -- 172.21.15.50:0/2581238647 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc034440 con 0x7face40a32e0 2023-10-30T12:13:21.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.173+0000 7face259c640 10 monclient: dump: 2023-10-30T12:13:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.200 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 10 monclient: shutdown 2023-10-30T12:13:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 -- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 msgr2=0x7face4135df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 --2- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face4135df0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7facdc009d90 tx=0x7facdc02bd00 comp rx=0 tx=0).stop 2023-10-30T12:13:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 -- 172.21.15.50:0/2581238647 shutdown_connections 2023-10-30T12:13:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 --2- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face4135df0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 --2- 172.21.15.50:0/2581238647 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face40a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 -- 172.21.15.50:0/2581238647 >> 172.21.15.50:0/2581238647 conn(0x7face40a80b0 msgr2=0x7face40a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:21.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 -- 172.21.15.50:0/2581238647 shutdown_connections 2023-10-30T12:13:21.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.174+0000 7face9009640 1 -- 172.21.15.50:0/2581238647 wait complete. 2023-10-30T12:13:21.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient: build_initial_monmap 2023-10-30T12:13:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient: monmap: 2023-10-30T12:13:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:21.204 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 librados: starting msgr at 2023-10-30T12:13:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 librados: starting objecter 2023-10-30T12:13:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 Processor -- start 2023-10-30T12:13:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 -- start start 2023-10-30T12:13:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 librados: setting wanted keys 2023-10-30T12:13:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 librados: calling monclient init 2023-10-30T12:13:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient: init 2023-10-30T12:13:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:13:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face409f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): picked mon.noname-c con 0x7face40a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face40a2ec0 0x7face4132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): picked mon.noname-b con 0x7face40a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4132a70 0x7face41c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): picked mon.noname-a con 0x7face4132a70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face40a17c0 con 0x7face4132a70 2023-10-30T12:13:21.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4120660 con 0x7face40a2ec0 2023-10-30T12:13:21.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face4120940 con 0x7face40a2ae0 2023-10-30T12:13:21.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:21.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.175+0000 7face9009640 10 monclient(hunting): authenticate will time out at 1646.358943s 2023-10-30T12:13:21.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face409f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4132a70 0x7face41c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face409f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46676/0 (socket says 172.21.15.50:46676) 2023-10-30T12:13:21.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4132a70 0x7face41c9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55940/0 (socket says 172.21.15.50:55940) 2023-10-30T12:13:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 -- 172.21.15.50:0/562405946 learned_addr learned my addr 172.21.15.50:0/562405946 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): get_auth_request con 0x7face40a2ae0 auth_method 0 2023-10-30T12:13:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdbfff640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face40a2ec0 msgr2=0x7face4132530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 10 monclient(hunting): get_auth_request con 0x7face4132a70 auth_method 0 2023-10-30T12:13:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdbfff640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face40a2ec0 0x7face4132530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:21.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient(hunting): handle_auth_done global_id 114815 payload 824 2023-10-30T12:13:21.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face40a2ec0 msgr2=0x7face4132530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:21.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face40a2ec0 0x7face4132530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4132a70 msgr2=0x7face41c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4132a70 0x7face41c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:13:21.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 monclient: found mon.noname-c 2023-10-30T12:13:21.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:21.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:21.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 -- 172.21.15.50:0/562405946 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40094d0 con 0x7face40a2ae0 2023-10-30T12:13:21.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face359e640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4132a70 0x7face41c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:21.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face2d9d640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face409f220 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7facd400e9d0 tx=0x7facd400ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 client.?.objecter ms_handle_connect 0x7face40a2ae0 2023-10-30T12:13:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400ca60 con 0x7face40a2ae0 2023-10-30T12:13:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: dump: 2023-10-30T12:13:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.218 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd400cbc0 con 0x7face40a2ae0 2023-10-30T12:13:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 5 monclient: authenticate success, global_id 114815 2023-10-30T12:13:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 10 monclient: _renew_subs 2023-10-30T12:13:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 1 -- 172.21.15.50:0/562405946 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc009a70 con 0x7face40a2ae0 2023-10-30T12:13:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4013470 con 0x7face40a2ae0 2023-10-30T12:13:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7facdb7fe640 10 monclient: dump: 2023-10-30T12:13:21.221 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.221 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.221 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.221 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.221 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.222 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.222 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.222 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.222 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.222 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face1d9b640 4 set_mon_vals no callback set 2023-10-30T12:13:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 10 client.114815.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 10 monclient: _renew_subs 2023-10-30T12:13:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 1 -- 172.21.15.50:0/562405946 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face411e560 con 0x7face40a2ae0 2023-10-30T12:13:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.176+0000 7face9009640 1 librados: init done 2023-10-30T12:13:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7facd40135d0 con 0x7face40a2ae0 2023-10-30T12:13:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7face9009640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7face9009640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face41ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7face9009640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7face9009640 1 -- 172.21.15.50:0/562405946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4073ff0 con 0x7face40a32e0 2023-10-30T12:13:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face41ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient: get_auth_request con 0x7face40a32e0 auth_method 0 2023-10-30T12:13:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdb7fe640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facc40392a0 0x7facc403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(400..400 src has 1..400) v4 ==== 6682+0+0 (secure 0 0 0) 0x7facd404b130 con 0x7face40a2ae0 2023-10-30T12:13:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdb7fe640 10 client.114815.objecter ms_dispatch 0x7face4000b90 osd_map(400..400 src has 1..400) v4 2023-10-30T12:13:21.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdb7fe640 3 client.114815.objecter handle_osd_map got epochs [400,400] > 0 2023-10-30T12:13:21.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdb7fe640 3 client.114815.objecter handle_osd_map decoding full epoch 400 2023-10-30T12:13:21.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 10 monclient(hunting): handle_auth_done global_id 174192 payload 824 2023-10-30T12:13:21.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdbfff640 1 -- 172.21.15.50:0/562405946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7face409a5c0 con 0x7face40a32e0 2023-10-30T12:13:21.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.177+0000 7facdb7fe640 20 client.114815.objecter dump_active .. 0 homeless 2023-10-30T12:13:21.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7face359e640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facc40392a0 0x7facc403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdbfff640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face41ca310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7face409ae90 tx=0x7faccc00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 client.114815.objecter ms_handle_connect 0x7face40a32e0 2023-10-30T12:13:21.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 client.114815.objecter resend_mon_ops 2023-10-30T12:13:21.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7face359e640 10 monclient: get_auth_request con 0x7facc40392a0 auth_method 0 2023-10-30T12:13:21.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc003e10 con 0x7face40a32e0 2023-10-30T12:13:21.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:21.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 monclient: dump: 2023-10-30T12:13:21.229 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.229 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.229 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.230 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.230 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.230 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.230 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.230 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.231 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7faccc005070 con 0x7face40a32e0 2023-10-30T12:13:21.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:21.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:21.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 msgr2=0x7face41ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face41ca310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7face409ae90 tx=0x7faccc00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 msgr2=0x7face41ca310 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:21.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face40a32e0 0x7face41ca310 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7face409ae90 tx=0x7faccc00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7facdb7fe640 10 client.114815.objecter ms_handle_connect 0x7facc40392a0 2023-10-30T12:13:21.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.178+0000 7face359e640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facc40392a0 0x7facc403b760 secure :-1 s=READY pgs=4156 cs=0 l=1 rev1=1 crypto rx=0x7facdc036000 tx=0x7facdc030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7face9009640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:21.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7face9009640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face41ca850 0x7face409c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7face9009640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7face9009640 1 -- 172.21.15.50:0/562405946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4073a20 con 0x7face41ca850 2023-10-30T12:13:21.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face41ca850 0x7face409c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 10 monclient: get_auth_request con 0x7face41ca850 auth_method 0 2023-10-30T12:13:21.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.184+0000 7facdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdbfff640 10 monclient(hunting): handle_auth_done global_id 174195 payload 824 2023-10-30T12:13:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdbfff640 1 -- 172.21.15.50:0/562405946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7face4099b40 con 0x7face41ca850 2023-10-30T12:13:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdbfff640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face41ca850 0x7face409c420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7face409ae90 tx=0x7faccc005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 10 client.114815.objecter ms_handle_connect 0x7face41ca850 2023-10-30T12:13:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 10 client.114815.objecter resend_mon_ops 2023-10-30T12:13:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc020070 con 0x7face41ca850 2023-10-30T12:13:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 10 monclient: dump: 2023-10-30T12:13:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.240 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7faccc010400 con 0x7face41ca850 2023-10-30T12:13:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face41ca850 msgr2=0x7face409c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face41ca850 0x7face409c420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7face409ae90 tx=0x7faccc005a20 comp rx=0 tx=0).stop 2023-10-30T12:13:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face41ca850 msgr2=0x7face409c420 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7facdb7fe640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face41ca850 0x7face409c420 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7face409ae90 tx=0x7faccc005a20 comp rx=0 tx=0).stop 2023-10-30T12:13:21.242 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:21.242 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:21.242 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:21.242 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:13:21.243 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 128, 2023-10-30T12:13:21.243 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:21.243 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:21.243 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:21.243 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.244 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:13:21.244 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:21.244 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:21.244 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:21.244 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:21.244 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:21.245 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:21.245 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:21.245 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:21.245 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:21.245 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:21.246 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:21.246 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:21.246 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:21.246 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.246 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:21.247 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:21.247 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:21.247 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:21.247 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:21.247 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:21.248 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:21.248 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:21.248 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:21.248 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:21.248 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:21.249 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:21.249 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.249 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:21.249 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:21.249 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:21.250 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:21.250 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:21.250 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:21.250 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:21.250 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:21.251 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:21.251 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:21.251 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:21.251 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:21.251 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:21.252 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:21.252 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:21.252 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:21.252 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:21.252 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:21.252 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:21.253 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:21.253 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:21.253 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:21.253 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:21.253 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:21.254 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:21.254 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:21.254 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:21.254 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.254 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:21.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.255 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:21.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.255 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:21.255 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:21.256 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:21.256 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:21.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:21.256 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:21.256 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.257 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:21.257 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:21.257 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.258 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.258 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:21.258 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:21.259 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:21.259 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:21.259 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:21.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.260 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:21.260 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:21.260 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:21.260 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:21.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.260 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:21.261 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:21.261 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.261 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:21.262 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:21.262 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.262 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.262 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.263 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:21.263 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:21.263 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:21.263 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:21.263 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:21.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.264 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:21.264 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:21.264 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:21.264 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:21.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:21.265 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:21.265 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.266 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:21.266 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:21.266 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.266 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.267 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:21.267 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:21.267 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:21.267 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:21.267 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:21.268 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.268 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.268 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:21.268 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:21.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.269 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:21.269 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:21.269 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:21.269 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.269 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.270 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:21.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.270 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:21.270 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:21.270 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:13:21.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.271 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:21.271 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 10 librados: watch_flush enter 2023-10-30T12:13:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 10 librados: watch_flush exit 2023-10-30T12:13:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 20 client.114815.objecter shutdown clearing up homeless session... 2023-10-30T12:13:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 10 client.114815.objecter successfully canceled tick 2023-10-30T12:13:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facc40392a0 msgr2=0x7facc403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7facc40392a0 0x7facc403b760 secure :-1 s=READY pgs=4156 cs=0 l=1 rev1=1 crypto rx=0x7facdc036000 tx=0x7facdc030000 comp rx=0 tx=0).stop 2023-10-30T12:13:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 10 monclient: shutdown 2023-10-30T12:13:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 1 -- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 msgr2=0x7face409f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 1 --2- 172.21.15.50:0/562405946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7face40a2ae0 0x7face409f220 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7facd400e9d0 tx=0x7facd400ee90 comp rx=0 tx=0).stop 2023-10-30T12:13:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face359e640 1 -- 172.21.15.50:0/562405946 reap_dead start 2023-10-30T12:13:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face359e640 1 -- 172.21.15.50:0/562405946 reap_dead start 2023-10-30T12:13:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 1 -- 172.21.15.50:0/562405946 shutdown_connections 2023-10-30T12:13:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.185+0000 7face9009640 1 -- 172.21.15.50:0/562405946 >> 172.21.15.50:0/562405946 conn(0x7face40a80b0 msgr2=0x7face41b96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.186+0000 7face9009640 1 -- 172.21.15.50:0/562405946 shutdown_connections 2023-10-30T12:13:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.186+0000 7face9009640 1 -- 172.21.15.50:0/562405946 wait complete. 2023-10-30T12:13:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.186+0000 7face9009640 1 librados: shutdown 2023-10-30T12:13:21.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:13:21.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:13:21.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:13:21.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.382+0000 7f8720856640 10 monclient: get_monmap_and_config 2023-10-30T12:13:21.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.382+0000 7f8720856640 10 monclient: build_initial_monmap 2023-10-30T12:13:21.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.382+0000 7f8720856640 10 monclient: monmap: 2023-10-30T12:13:21.384 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:21.384 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.384 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:21.383374+0000 2023-10-30T12:13:21.385 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:21.383374+0000 2023-10-30T12:13:21.385 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:21.385 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.385 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:21.385 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:21.386 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:21.386 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:21.386 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:21.386 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.382+0000 7f8720856640 1 Processor -- start 2023-10-30T12:13:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.382+0000 7f8720856640 1 -- start start 2023-10-30T12:13:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.382+0000 7f8720856640 10 monclient: init 2023-10-30T12:13:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:13:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f871c071c00 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f871c097fe0 0x7f871c09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient(hunting): picked mon.noname-a con 0x7f871c097fe0 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c09b170 0x7f871c139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient(hunting): picked mon.noname-b con 0x7f871c09b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:21.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c097600 con 0x7f871c071c00 2023-10-30T12:13:21.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c0978e0 con 0x7f871c097fe0 2023-10-30T12:13:21.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c09bf50 con 0x7f871c09b170 2023-10-30T12:13:21.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:21.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8720856640 10 monclient(hunting): authenticate will time out at 1646.566403s 2023-10-30T12:13:21.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f871c097fe0 0x7f871c09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c09b170 0x7f871c139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f871c097fe0 0x7f871c09a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55960/0 (socket says 172.21.15.50:55960) 2023-10-30T12:13:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c09b170 0x7f871c139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46682/0 (socket says 172.21.15.50:46682) 2023-10-30T12:13:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 -- 172.21.15.50:0/1418702903 learned_addr learned my addr 172.21.15.50:0/1418702903 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): get_auth_request con 0x7f871c097fe0 auth_method 0 2023-10-30T12:13:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 10 monclient(hunting): get_auth_request con 0x7f871c09b170 auth_method 0 2023-10-30T12:13:21.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient(hunting): handle_auth_done global_id 174198 payload 274 2023-10-30T12:13:21.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 -- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c09b170 msgr2=0x7f871c139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:21.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 --2- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c09b170 0x7f871c139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 -- 172.21.15.50:0/1418702903 >> v1:172.21.15.50:6789/0 conn(0x7f871c071c00 legacy=0x7f871c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:13:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 monclient: found mon.noname-a 2023-10-30T12:13:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f8711d9b640 1 -- 172.21.15.50:0/1418702903 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c09d9b0 con 0x7f871c097fe0 2023-10-30T12:13:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.383+0000 7f871ad9d640 1 --2- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c09b170 0x7f871c139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8711d9b640 1 --2- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c097fe0 0x7f871c09a150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f870400b970 tx=0x7f870402c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9104555622ee0d27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 1 -- 172.21.15.50:0/1418702903 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870402cdb0 con 0x7f871c097fe0 2023-10-30T12:13:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: dump: 2023-10-30T12:13:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.406 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: _finish_auth 0 2023-10-30T12:13:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:13:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8720856640 5 monclient: authenticate success, global_id 174198 2023-10-30T12:13:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 1 -- 172.21.15.50:0/1418702903 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704030070 con 0x7f871c097fe0 2023-10-30T12:13:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8720856640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8720856640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8720856640 4 set_mon_vals no callback set 2023-10-30T12:13:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 1 -- 172.21.15.50:0/1418702903 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704036620 con 0x7f871c097fe0 2023-10-30T12:13:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.384+0000 7f8719d9b640 10 monclient: dump: 2023-10-30T12:13:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.410 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.410 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 10 monclient: shutdown 2023-10-30T12:13:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 -- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c097fe0 msgr2=0x7f871c09a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 --2- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c097fe0 0x7f871c09a150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f870400b970 tx=0x7f870402c060 comp rx=0 tx=0).stop 2023-10-30T12:13:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 -- 172.21.15.50:0/1418702903 shutdown_connections 2023-10-30T12:13:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 --2- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c09b170 0x7f871c139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 --2- 172.21.15.50:0/1418702903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c097fe0 0x7f871c09a150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 -- 172.21.15.50:0/1418702903 >> 172.21.15.50:0/1418702903 conn(0x7f871c0a3980 msgr2=0x7f871c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:21.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 -- 172.21.15.50:0/1418702903 shutdown_connections 2023-10-30T12:13:21.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 1 -- 172.21.15.50:0/1418702903 wait complete. 2023-10-30T12:13:21.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.385+0000 7f8720856640 10 monclient: build_initial_monmap 2023-10-30T12:13:21.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 10 monclient: monmap: 2023-10-30T12:13:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:21.414 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 librados: starting msgr at 2023-10-30T12:13:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 librados: starting objecter 2023-10-30T12:13:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 Processor -- start 2023-10-30T12:13:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 -- start start 2023-10-30T12:13:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 librados: setting wanted keys 2023-10-30T12:13:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 librados: calling monclient init 2023-10-30T12:13:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 10 monclient: init 2023-10-30T12:13:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:13:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c071c00 0x7f871c1c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 10 monclient(hunting): picked mon.noname-a con 0x7f871c071c00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c097fe0 0x7f871c1b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 10 monclient(hunting): picked mon.noname-c con 0x7f871c097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.386+0000 7f8720856640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f871c09b170 0x7f871c1b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient(hunting): picked mon.noname-b con 0x7f871c09b170 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1ad820 con 0x7f871c071c00 2023-10-30T12:13:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1ae4f0 con 0x7f871c09b170 2023-10-30T12:13:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1aea40 con 0x7f871c097fe0 2023-10-30T12:13:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient(hunting): authenticate will time out at 1646.570038s 2023-10-30T12:13:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c097fe0 0x7f871c1b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c071c00 0x7f871c1c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c097fe0 0x7f871c1b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46698/0 (socket says 172.21.15.50:46698) 2023-10-30T12:13:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 1 -- 172.21.15.50:0/1832937440 learned_addr learned my addr 172.21.15.50:0/1832937440 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): get_auth_request con 0x7f871c071c00 auth_method 0 2023-10-30T12:13:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 10 monclient(hunting): get_auth_request con 0x7f871c097fe0 auth_method 0 2023-10-30T12:13:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871ad9d640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f871c09b170 msgr2=0x7f871c1b57e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:13:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871ad9d640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f871c09b170 0x7f871c1b57e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient(hunting): handle_auth_done global_id 174204 payload 824 2023-10-30T12:13:21.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c097fe0 msgr2=0x7f871c1b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c097fe0 0x7f871c1b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f871c09b170 msgr2=0x7f871c1b57e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:21.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f871c09b170 0x7f871c1b57e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:13:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 monclient: found mon.noname-a 2023-10-30T12:13:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 -- 172.21.15.50:0/1832937440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871400f050 con 0x7f871c071c00 2023-10-30T12:13:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871a59c640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c071c00 0x7f871c1c6050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f870c00c3c0 tx=0x7f870c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 client.?.objecter ms_handle_connect 0x7f871c071c00 2023-10-30T12:13:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0035f0 con 0x7f871c071c00 2023-10-30T12:13:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: dump: 2023-10-30T12:13:21.426 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.426 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.426 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.426 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.426 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.427 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.427 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.427 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.427 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.427 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 5 monclient: authenticate success, global_id 174204 2023-10-30T12:13:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient: _renew_subs 2023-10-30T12:13:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870400b5d0 con 0x7f871c071c00 2023-10-30T12:13:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c003750 con 0x7f871c071c00 2023-10-30T12:13:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c003b00 con 0x7f871c071c00 2023-10-30T12:13:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f87137fe640 10 monclient: dump: 2023-10-30T12:13:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.432 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f871959a640 4 set_mon_vals no callback set 2023-10-30T12:13:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 client.174204.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient: _renew_subs 2023-10-30T12:13:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c1ae760 con 0x7f871c071c00 2023-10-30T12:13:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.387+0000 7f8720856640 1 librados: init done 2023-10-30T12:13:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f870c003c60 con 0x7f871c071c00 2023-10-30T12:13:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8720856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8720856640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c071200 0x7f871c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8720856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c073ff0 con 0x7f871c071200 2023-10-30T12:13:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c071200 0x7f871c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient: get_auth_request con 0x7f871c071200 auth_method 0 2023-10-30T12:13:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f87137fe640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86fc0392a0 0x7f86fc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(400..400 src has 1..400) v4 ==== 6682+0+0 (secure 0 0 0) 0x7f870c04a4c0 con 0x7f871c071c00 2023-10-30T12:13:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f87137fe640 10 client.174204.objecter ms_dispatch 0x7f871c000b90 osd_map(400..400 src has 1..400) v4 2023-10-30T12:13:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f87137fe640 3 client.174204.objecter handle_osd_map got epochs [400,400] > 0 2023-10-30T12:13:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f87137fe640 3 client.174204.objecter handle_osd_map decoding full epoch 400 2023-10-30T12:13:21.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 10 monclient(hunting): handle_auth_done global_id 114824 payload 824 2023-10-30T12:13:21.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f8711d9b640 1 -- 172.21.15.50:0/1832937440 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f871c1ad620 con 0x7f871c071200 2023-10-30T12:13:21.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.388+0000 7f87137fe640 20 client.174204.objecter dump_active .. 0 homeless 2023-10-30T12:13:21.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f871ad9d640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86fc0392a0 0x7f86fc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f871ad9d640 10 monclient: get_auth_request con 0x7f86fc0392a0 auth_method 0 2023-10-30T12:13:21.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 client.174204.objecter ms_handle_connect 0x7f871c071200 2023-10-30T12:13:21.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 client.174204.objecter resend_mon_ops 2023-10-30T12:13:21.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f8711d9b640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c071200 0x7f871c06f660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f870400b970 tx=0x7f87040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704049640 con 0x7f871c071200 2023-10-30T12:13:21.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 monclient: dump: 2023-10-30T12:13:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.441 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 client.174204.objecter ms_handle_connect 0x7f86fc0392a0 2023-10-30T12:13:21.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f871ad9d640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86fc0392a0 0x7f86fc03b760 secure :-1 s=READY pgs=4157 cs=0 l=1 rev1=1 crypto rx=0x7f871c1ae3b0 tx=0x7f871400db60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f87040497a0 con 0x7f871c071200 2023-10-30T12:13:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c071200 msgr2=0x7f871c06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c071200 0x7f871c06f660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f870400b970 tx=0x7f87040046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c071200 msgr2=0x7f871c06f660 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.389+0000 7f87137fe640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c071200 0x7f871c06f660 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f870400b970 tx=0x7f87040046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:21.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8720856640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:21.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8720856640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c06fba0 0x7f871c06ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:21.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8720856640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:21.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c071fe0 con 0x7f871c06fba0 2023-10-30T12:13:21.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8711d9b640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c06fba0 0x7f871c06ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:21.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8711d9b640 10 monclient: get_auth_request con 0x7f871c06fba0 auth_method 0 2023-10-30T12:13:21.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8711d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:21.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8711d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:21.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.395+0000 7f8711d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:21.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:21.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:21.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8711d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:21.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8711d9b640 10 monclient(hunting): handle_auth_done global_id 114827 payload 824 2023-10-30T12:13:21.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8711d9b640 1 -- 172.21.15.50:0/1832937440 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f871c0704f0 con 0x7f871c06fba0 2023-10-30T12:13:21.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 10 client.174204.objecter ms_handle_connect 0x7f871c06fba0 2023-10-30T12:13:21.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 10 client.174204.objecter resend_mon_ops 2023-10-30T12:13:21.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8711d9b640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c06fba0 0x7f871c06ffb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f871c070bc0 tx=0x7f8704030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:21.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704049ea0 con 0x7f871c06fba0 2023-10-30T12:13:21.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:21.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:21.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 10 monclient: dump: 2023-10-30T12:13:21.449 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:21.449 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:21.449 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.449 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:21.449 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:21.449 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:21.450 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:21.450 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:21.450 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:21.450 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:21.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4791 (secure 0 0 0) 0x7f870403f930 con 0x7f871c06fba0 2023-10-30T12:13:21.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:21.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:21.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c06fba0 msgr2=0x7f871c06ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c06fba0 0x7f871c06ffb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f871c070bc0 tx=0x7f8704030000 comp rx=0 tx=0).stop 2023-10-30T12:13:21.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c06fba0 msgr2=0x7f871c06ffb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:21.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f87137fe640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f871c06fba0 0x7f871c06ffb0 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f871c070bc0 tx=0x7f8704030000 comp rx=0 tx=0).stop 2023-10-30T12:13:21.452 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:21.452 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:21.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:21.453 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:21.453 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 128, 2023-10-30T12:13:21.453 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:21.453 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:21.454 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:21.454 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.454 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:13:21.454 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:21.454 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:21.455 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:21.455 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:21.455 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:21.455 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:21.455 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:21.455 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:21.456 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:21.456 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:21.456 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:21.456 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:21.456 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:21.457 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.457 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:21.457 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:21.457 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:21.457 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:21.457 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:21.458 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:21.458 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:21.458 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:21.458 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:21.458 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:21.459 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:21.459 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:21.459 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.459 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:21.460 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:21.460 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:21.460 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:21.460 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:21.460 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:21.460 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:21.461 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:21.461 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:21.461 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:21.461 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:21.461 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:21.462 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:21.462 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:21.462 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:21.462 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:21.462 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:21.463 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:21.463 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:21.463 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:21.463 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:21.463 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:21.464 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:21.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:21.464 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:21.464 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:21.464 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:21.464 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.465 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:21.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.465 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:21.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:21.465 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:21.466 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:21.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:21.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:21.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:21.467 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:21.467 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:21.468 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.468 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:21.469 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:21.469 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:21.469 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:21.469 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:21.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:21.470 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:21.470 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:21.470 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:21.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:21.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:21.471 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:21.472 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:21.472 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:21.473 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:21.473 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:21.473 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:21.473 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:21.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:21.474 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:21.474 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:21.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:21.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:21.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:21.475 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.476 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:21.476 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:21.476 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:21.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.477 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.477 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:21.477 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:21.477 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:21.478 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:21.478 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:21.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.478 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.479 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:21.479 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:21.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.479 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:21.479 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:21.479 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:21.480 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.480 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.480 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:13:21.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.480 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:13:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:13:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:13:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:21.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:21.484 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:21.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:21.484 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:21.484 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:21.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8720856640 10 librados: watch_flush enter 2023-10-30T12:13:21.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8720856640 10 librados: watch_flush exit 2023-10-30T12:13:21.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8720856640 20 client.174204.objecter shutdown clearing up homeless session... 2023-10-30T12:13:21.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.396+0000 7f8720856640 10 client.174204.objecter successfully canceled tick 2023-10-30T12:13:21.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86fc0392a0 msgr2=0x7f86fc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86fc0392a0 0x7f86fc03b760 secure :-1 s=READY pgs=4157 cs=0 l=1 rev1=1 crypto rx=0x7f871c1ae3b0 tx=0x7f871400db60 comp rx=0 tx=0).stop 2023-10-30T12:13:21.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 10 monclient: shutdown 2023-10-30T12:13:21.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:21.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c071c00 msgr2=0x7f871c1c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:21.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 --2- 172.21.15.50:0/1832937440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f871c071c00 0x7f871c1c6050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f870c00c3c0 tx=0x7f870c00c880 comp rx=0 tx=0).stop 2023-10-30T12:13:21.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f871ad9d640 1 -- 172.21.15.50:0/1832937440 reap_dead start 2023-10-30T12:13:21.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f871ad9d640 1 -- 172.21.15.50:0/1832937440 reap_dead start 2023-10-30T12:13:21.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 shutdown_connections 2023-10-30T12:13:21.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 >> 172.21.15.50:0/1832937440 conn(0x7f871c0a3980 msgr2=0x7f871c1b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:21.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 shutdown_connections 2023-10-30T12:13:21.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 -- 172.21.15.50:0/1832937440 wait complete. 2023-10-30T12:13:21.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:21.397+0000 7f8720856640 1 librados: shutdown 2023-10-30T12:13:21.490 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:13:21.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:13:41.492 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:13:41.492 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-10-30T12:13:41.492 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:13:41.493 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:13:41.496 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:13:41.496 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:13:41.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.612+0000 7f1952f4c640 10 monclient: get_monmap_and_config 2023-10-30T12:13:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.612+0000 7f1952f4c640 10 monclient: build_initial_monmap 2023-10-30T12:13:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.612+0000 7f1952f4c640 10 monclient: monmap: 2023-10-30T12:13:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:41.613571+0000 2023-10-30T12:13:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:41.613571+0000 2023-10-30T12:13:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:41.617 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 1 Processor -- start 2023-10-30T12:13:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 1 -- start start 2023-10-30T12:13:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient: init 2023-10-30T12:13:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-10-30T12:13:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c09bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient(hunting): picked mon.noname-b con 0x7f194c098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f194c098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 0x7f194c139d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient(hunting): picked mon.noname-c con 0x7f194c137980 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194c099930 con 0x7f194c098900 2023-10-30T12:13:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c09aca0 con 0x7f194c137980 2023-10-30T12:13:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c09af80 con 0x7f194c098520 2023-10-30T12:13:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1952f4c640 10 monclient(hunting): authenticate will time out at 1666.796818s 2023-10-30T12:13:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1950ce8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c09bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f194bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f194c098900 0x7f194c098d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39128/0 (socket says 172.21.15.50:39128) 2023-10-30T12:13:41.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f194bfff640 1 -- 172.21.15.50:0/654488211 learned_addr learned my addr 172.21.15.50:0/654488211 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.613+0000 7f1950ce8640 10 monclient(hunting): get_auth_request con 0x7f194c098520 auth_method 0 2023-10-30T12:13:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 1 --2- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 0x7f194c139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:41.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 10 monclient(hunting): get_auth_request con 0x7f194c137980 auth_method 0 2023-10-30T12:13:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:41.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 1 -- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 msgr2=0x7f194c139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:13:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 1 -- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 msgr2=0x7f194c139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:13:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 1 --2- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 0x7f194c139d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient(hunting): handle_auth_done global_id 114830 payload 274 2023-10-30T12:13:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 1 -- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 msgr2=0x7f194c139d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f194b7fe640 1 -- 172.21.15.50:0/654488211 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388963589 0 0) 0x7f194c099930 con 0x7f194c098900 2023-10-30T12:13:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f19514e9640 1 --2- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 0x7f194c139d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 1 --2- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 0x7f194c139d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 1 -- 172.21.15.50:0/654488211 >> v1:172.21.15.50:6789/0 conn(0x7f194c098900 legacy=0x7f194c098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient: _finish_hunting 0 2023-10-30T12:13:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 1 monclient: found mon.noname-b 2023-10-30T12:13:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f1950ce8640 1 -- 172.21.15.50:0/654488211 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c09b260 con 0x7f194c098520 2023-10-30T12:13:41.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.614+0000 7f194b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:13:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1950ce8640 1 --2- 172.21.15.50:0/654488211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c09bf70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1944009de0 tx=0x7f194402bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1373a5074ce0e046 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 1 -- 172.21.15.50:0/654488211 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194402cb70 con 0x7f194c098520 2023-10-30T12:13:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: dump: 2023-10-30T12:13:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:41.640 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:41.641 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:41.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 5 monclient: authenticate success, global_id 114830 2023-10-30T12:13:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 1 -- 172.21.15.50:0/654488211 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194402ccd0 con 0x7f194c098520 2023-10-30T12:13:41.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 4 set_mon_vals no callback set 2023-10-30T12:13:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 1 -- 172.21.15.50:0/654488211 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944034660 con 0x7f194c098520 2023-10-30T12:13:41.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f194b7fe640 10 monclient: dump: 2023-10-30T12:13:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.644 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:41.646 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 10 monclient: shutdown 2023-10-30T12:13:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:41.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 -- 172.21.15.50:0/654488211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 msgr2=0x7f194c09bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 --2- 172.21.15.50:0/654488211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c09bf70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1944009de0 tx=0x7f194402bdc0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 -- 172.21.15.50:0/654488211 shutdown_connections 2023-10-30T12:13:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 --2- 172.21.15.50:0/654488211 >> v2:172.21.15.118:3300/0 conn(0x7f194c137980 0x7f194c139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 --2- 172.21.15.50:0/654488211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c09bf70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 -- 172.21.15.50:0/654488211 >> 172.21.15.50:0/654488211 conn(0x7f194c0a05c0 msgr2=0x7f194c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:41.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 -- 172.21.15.50:0/654488211 shutdown_connections 2023-10-30T12:13:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.615+0000 7f1952f4c640 1 -- 172.21.15.50:0/654488211 wait complete. 2023-10-30T12:13:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 10 monclient: build_initial_monmap 2023-10-30T12:13:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 10 monclient: monmap: 2023-10-30T12:13:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:41.648 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:41.650 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 1 librados: starting msgr at 2023-10-30T12:13:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 1 librados: starting objecter 2023-10-30T12:13:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 1 Processor -- start 2023-10-30T12:13:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 1 -- start start 2023-10-30T12:13:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 1 librados: setting wanted keys 2023-10-30T12:13:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 1 librados: calling monclient init 2023-10-30T12:13:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.616+0000 7f1952f4c640 10 monclient: init 2023-10-30T12:13:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:13:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c1ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): picked mon.noname-c con 0x7f194c098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 0x7f194c096320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): picked mon.noname-b con 0x7f194c098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:41.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f194c137980 0x7f194c096860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:41.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): picked mon.noname-a con 0x7f194c137980 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:41.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:41.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c071d50 con 0x7f194c137980 2023-10-30T12:13:41.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:41.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0a2c30 con 0x7f194c098900 2023-10-30T12:13:41.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:41.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c092af0 con 0x7f194c098520 2023-10-30T12:13:41.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:41.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1952f4c640 10 monclient(hunting): authenticate will time out at 1666.800572s 2023-10-30T12:13:41.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c1ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:41.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f194c137980 0x7f194c096860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:41.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c1ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53652/0 (socket says 172.21.15.50:53652) 2023-10-30T12:13:41.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 1 -- 172.21.15.50:0/25499138 learned_addr learned my addr 172.21.15.50:0/25499138 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:41.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): get_auth_request con 0x7f194c098520 auth_method 0 2023-10-30T12:13:41.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 10 monclient(hunting): get_auth_request con 0x7f194c137980 auth_method 0 2023-10-30T12:13:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:41.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 0x7f194c096320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 10 monclient(hunting): get_auth_request con 0x7f194c098900 auth_method 0 2023-10-30T12:13:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f19514e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 1 -- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 msgr2=0x7f194c096320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:13:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 1 -- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 msgr2=0x7f194c096320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:13:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 0x7f194c096320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:41.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f194bfff640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 0x7f194c096320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 10 monclient(hunting): handle_auth_done global_id 114833 payload 824 2023-10-30T12:13:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.617+0000 7f1950ce8640 1 -- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 msgr2=0x7f194c096320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 0x7f194c096320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 1 -- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f194c137980 msgr2=0x7f194c096860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f194c137980 0x7f194c096860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 10 monclient: _finish_hunting 0 2023-10-30T12:13:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 1 monclient: found mon.noname-c 2023-10-30T12:13:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 1 -- 172.21.15.50:0/25499138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1944009a70 con 0x7f194c098520 2023-10-30T12:13:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19514e9640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f194c137980 0x7f194c096860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1950ce8640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c1ca270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1944037040 tx=0x7f1944004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 client.?.objecter ms_handle_connect 0x7f194c098520 2023-10-30T12:13:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 1 -- 172.21.15.50:0/25499138 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944040650 con 0x7f194c098520 2023-10-30T12:13:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: dump: 2023-10-30T12:13:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.664 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.664 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:41.664 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:41.664 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:41.664 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:41.664 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:41.665 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:41.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: _finish_auth 0 2023-10-30T12:13:41.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: _check_auth_tickets 2023-10-30T12:13:41.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:41.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 1 -- 172.21.15.50:0/25499138 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19440407b0 con 0x7f194c098520 2023-10-30T12:13:41.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:41.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 1 -- 172.21.15.50:0/25499138 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944040ba0 con 0x7f194c098520 2023-10-30T12:13:41.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 5 monclient: authenticate success, global_id 114833 2023-10-30T12:13:41.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:41.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:41.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f19497fa640 10 monclient: dump: 2023-10-30T12:13:41.667 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:41.667 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:41.667 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.667 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:41.667 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:41.668 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:41.668 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:41.668 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:41.668 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:41.668 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:41.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 10 monclient: _renew_subs 2023-10-30T12:13:41.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:41.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f194affd640 4 set_mon_vals no callback set 2023-10-30T12:13:41.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c091640 con 0x7f194c098520 2023-10-30T12:13:41.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 10 client.114833.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:41.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 10 monclient: _renew_subs 2023-10-30T12:13:41.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:41.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c091190 con 0x7f194c098520 2023-10-30T12:13:41.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.618+0000 7f1952f4c640 1 librados: init done 2023-10-30T12:13:41.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f19497fa640 1 -- 172.21.15.50:0/25499138 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1944040d00 con 0x7f194c098520 2023-10-30T12:13:41.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f19497fa640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1938039320 0x7f193803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:41.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f19497fa640 1 -- 172.21.15.50:0/25499138 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(420..420 src has 1..420) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f194402ce40 con 0x7f194c098520 2023-10-30T12:13:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f19497fa640 10 client.114833.objecter ms_dispatch 0x7f194c000b90 osd_map(420..420 src has 1..420) v4 2023-10-30T12:13:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f19497fa640 3 client.114833.objecter handle_osd_map got epochs [420,420] > 0 2023-10-30T12:13:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f19497fa640 3 client.114833.objecter handle_osd_map decoding full epoch 420 2023-10-30T12:13:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f1952f4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f1952f4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f1952f4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194c04e160 con 0x7f194c098520 2023-10-30T12:13:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f19497fa640 20 client.114833.objecter dump_active .. 0 homeless 2023-10-30T12:13:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f194bfff640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1938039320 0x7f193803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.619+0000 7f194bfff640 10 monclient: get_auth_request con 0x7f1938039320 auth_method 0 2023-10-30T12:13:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.620+0000 7f19497fa640 10 client.114833.objecter ms_handle_connect 0x7f1938039320 2023-10-30T12:13:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.620+0000 7f194bfff640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1938039320 0x7f193803b7e0 secure :-1 s=READY pgs=4159 cs=0 l=1 rev1=1 crypto rx=0x7f194c139ec0 tx=0x7f1940007b50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:41.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.624+0000 7f19497fa640 1 -- 172.21.15.50:0/25499138 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1944039080 con 0x7f194c098520 2023-10-30T12:13:41.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.624+0000 7f19497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:41.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.624+0000 7f19497fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.874+0000 7f1952f4c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:41.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.874+0000 7f1952f4c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.874+0000 7f1952f4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:41.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.874+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f194c0913e0 con 0x7f194c098520 2023-10-30T12:13:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.874+0000 7f19497fa640 1 -- 172.21.15.50:0/25499138 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f1944044030 con 0x7f194c098520 2023-10-30T12:13:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.874+0000 7f19497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:41.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.874+0000 7f19497fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:41.879 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":128,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:41.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 10 librados: watch_flush enter 2023-10-30T12:13:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 10 librados: watch_flush exit 2023-10-30T12:13:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 20 client.114833.objecter shutdown clearing up homeless session... 2023-10-30T12:13:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 10 client.114833.objecter successfully canceled tick 2023-10-30T12:13:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1938039320 msgr2=0x7f193803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1938039320 0x7f193803b7e0 secure :-1 s=READY pgs=4159 cs=0 l=1 rev1=1 crypto rx=0x7f194c139ec0 tx=0x7f1940007b50 comp rx=0 tx=0).stop 2023-10-30T12:13:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 10 monclient: shutdown 2023-10-30T12:13:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 msgr2=0x7f194c1ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c1ca270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1944037040 tx=0x7f1944004a40 comp rx=0 tx=0).stop 2023-10-30T12:13:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 shutdown_connections 2023-10-30T12:13:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1938039320 0x7f193803b7e0 unknown :-1 s=CLOSED pgs=4159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f194c137980 0x7f194c096860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f194c098900 0x7f194c096320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 --2- 172.21.15.50:0/25499138 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f194c098520 0x7f194c1ca270 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 >> 172.21.15.50:0/25499138 conn(0x7f194c0a05c0 msgr2=0x7f194c06da70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:41.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 shutdown_connections 2023-10-30T12:13:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 -- 172.21.15.50:0/25499138 wait complete. 2023-10-30T12:13:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:41.876+0000 7f1952f4c640 1 librados: shutdown 2023-10-30T12:13:44.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.003+0000 7f0a2ac64640 10 monclient: get_monmap_and_config 2023-10-30T12:13:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.004+0000 7f0a2ac64640 10 monclient: build_initial_monmap 2023-10-30T12:13:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.004+0000 7f0a2ac64640 10 monclient: monmap: 2023-10-30T12:13:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:45.005069+0000 2023-10-30T12:13:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:45.005069+0000 2023-10-30T12:13:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:45.008 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.004+0000 7f0a2ac64640 1 Processor -- start 2023-10-30T12:13:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.004+0000 7f0a2ac64640 1 -- start start 2023-10-30T12:13:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.004+0000 7f0a2ac64640 10 monclient: init 2023-10-30T12:13:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:13:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a240a7000 0x7f0a24135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient(hunting): picked mon.noname-a con 0x7f0a240a7000 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0a2406abe0 0x7f0a241360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient(hunting): picked mon.noname-c con 0x7f0a2406abe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a240a4460 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a240a73e0 con 0x7f0a240a4460 2023-10-30T12:13:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a24136600 con 0x7f0a240a7000 2023-10-30T12:13:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a2409a000 con 0x7f0a2406abe0 2023-10-30T12:13:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a2ac64640 10 monclient(hunting): authenticate will time out at 1670.188307s 2023-10-30T12:13:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a240a7000 0x7f0a24135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a240a7000 0x7f0a24135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48012/0 (socket says 172.21.15.50:48012) 2023-10-30T12:13:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 -- 172.21.15.50:0/2685903012 learned_addr learned my addr 172.21.15.50:0/2685903012 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): get_auth_request con 0x7f0a240a7000 auth_method 0 2023-10-30T12:13:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a23fff640 1 --2- 172.21.15.50:0/2685903012 >> v2:172.21.15.118:3300/0 conn(0x7f0a2406abe0 0x7f0a241360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a23fff640 10 monclient(hunting): get_auth_request con 0x7f0a2406abe0 auth_method 0 2023-10-30T12:13:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient(hunting): handle_auth_done global_id 174213 payload 274 2023-10-30T12:13:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 -- 172.21.15.50:0/2685903012 >> v2:172.21.15.118:3300/0 conn(0x7f0a2406abe0 msgr2=0x7f0a241360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 --2- 172.21.15.50:0/2685903012 >> v2:172.21.15.118:3300/0 conn(0x7f0a2406abe0 0x7f0a241360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 -- 172.21.15.50:0/2685903012 >> v1:172.21.15.118:6789/0 conn(0x7f0a240a4460 legacy=0x7f0a240a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient: _finish_hunting 0 2023-10-30T12:13:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 monclient: found mon.noname-a 2023-10-30T12:13:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.005+0000 7f0a28a00640 1 -- 172.21.15.50:0/2685903012 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2409a2e0 con 0x7f0a240a7000 2023-10-30T12:13:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a28a00640 1 --2- 172.21.15.50:0/2685903012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a7000 0x7f0a24135b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a240a76b0 tx=0x7f0a1002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae07d2a84f44e60 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 1 -- 172.21.15.50:0/2685903012 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a100317c0 con 0x7f0a240a7000 2023-10-30T12:13:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: dump: 2023-10-30T12:13:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.024 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:45.026 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:45.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 1 -- 172.21.15.50:0/2685903012 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a10031920 con 0x7f0a240a7000 2023-10-30T12:13:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a2ac64640 5 monclient: authenticate success, global_id 174213 2023-10-30T12:13:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a2ac64640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a2ac64640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:45.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 1 -- 172.21.15.50:0/2685903012 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10031cb0 con 0x7f0a240a7000 2023-10-30T12:13:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a2ac64640 4 set_mon_vals no callback set 2023-10-30T12:13:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.006+0000 7f0a237fe640 10 monclient: dump: 2023-10-30T12:13:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:45.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:45.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 10 monclient: shutdown 2023-10-30T12:13:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:45.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 -- 172.21.15.50:0/2685903012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a7000 msgr2=0x7f0a24135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/2685903012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a7000 0x7f0a24135b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a240a76b0 tx=0x7f0a1002b860 comp rx=0 tx=0).stop 2023-10-30T12:13:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 -- 172.21.15.50:0/2685903012 shutdown_connections 2023-10-30T12:13:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/2685903012 >> v2:172.21.15.118:3300/0 conn(0x7f0a2406abe0 0x7f0a241360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/2685903012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a7000 0x7f0a24135b80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 -- 172.21.15.50:0/2685903012 >> 172.21.15.50:0/2685903012 conn(0x7f0a24066000 msgr2=0x7f0a240a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 -- 172.21.15.50:0/2685903012 shutdown_connections 2023-10-30T12:13:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 1 -- 172.21.15.50:0/2685903012 wait complete. 2023-10-30T12:13:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 10 monclient: build_initial_monmap 2023-10-30T12:13:45.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.007+0000 7f0a2ac64640 10 monclient: monmap: 2023-10-30T12:13:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:45.034 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:45.034 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 librados: starting msgr at 2023-10-30T12:13:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 librados: starting objecter 2023-10-30T12:13:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 Processor -- start 2023-10-30T12:13:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 -- start start 2023-10-30T12:13:45.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 librados: setting wanted keys 2023-10-30T12:13:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 librados: calling monclient init 2023-10-30T12:13:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient: init 2023-10-30T12:13:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:13:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a2406abe0 0x7f0a241ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): picked mon.noname-c con 0x7f0a2406abe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a4460 0x7f0a241cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): picked mon.noname-a con 0x7f0a240a4460 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:45.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 0x7f0a2409e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): picked mon.noname-b con 0x7f0a240a7000 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240a05d0 con 0x7f0a240a4460 2023-10-30T12:13:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240a0be0 con 0x7f0a240a7000 2023-10-30T12:13:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240a1790 con 0x7f0a2406abe0 2023-10-30T12:13:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.008+0000 7f0a2ac64640 10 monclient(hunting): authenticate will time out at 1670.191924s 2023-10-30T12:13:45.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a4460 0x7f0a241cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a4460 0x7f0a241cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48022/0 (socket says 172.21.15.50:48022) 2023-10-30T12:13:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 -- 172.21.15.50:0/3583625876 learned_addr learned my addr 172.21.15.50:0/3583625876 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): get_auth_request con 0x7f0a240a4460 auth_method 0 2023-10-30T12:13:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a2406abe0 0x7f0a241ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 10 monclient(hunting): get_auth_request con 0x7f0a2406abe0 auth_method 0 2023-10-30T12:13:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 0x7f0a2409e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 10 monclient(hunting): get_auth_request con 0x7f0a240a7000 auth_method 0 2023-10-30T12:13:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a28a00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 1 -- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 msgr2=0x7f0a2409e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:13:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 1 -- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 msgr2=0x7f0a2409e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:13:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 0x7f0a2409e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a29201640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 0x7f0a2409e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient(hunting): handle_auth_done global_id 174216 payload 824 2023-10-30T12:13:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 -- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a2406abe0 msgr2=0x7f0a241ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a2406abe0 0x7f0a241ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 -- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 msgr2=0x7f0a2409e300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 0x7f0a2409e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 monclient: found mon.noname-a 2023-10-30T12:13:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 -- 172.21.15.50:0/3583625876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a100094d0 con 0x7f0a240a4460 2023-10-30T12:13:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a23fff640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a4460 0x7f0a241cacf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800d7d0 tx=0x7f0a1800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:45.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 client.?.objecter ms_handle_connect 0x7f0a240a4460 2023-10-30T12:13:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 1 -- 172.21.15.50:0/3583625876 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18018070 con 0x7f0a240a4460 2023-10-30T12:13:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: dump: 2023-10-30T12:13:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:45.049 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: _finish_auth 0 2023-10-30T12:13:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: _check_auth_tickets 2023-10-30T12:13:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 1 -- 172.21.15.50:0/3583625876 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a18004020 con 0x7f0a240a4460 2023-10-30T12:13:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 5 monclient: authenticate success, global_id 174216 2023-10-30T12:13:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 10 monclient: _renew_subs 2023-10-30T12:13:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2411ecc0 con 0x7f0a240a4460 2023-10-30T12:13:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 1 -- 172.21.15.50:0/3583625876 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a180041b0 con 0x7f0a240a4460 2023-10-30T12:13:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a217fa640 10 monclient: dump: 2023-10-30T12:13:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:45.054 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a22ffd640 4 set_mon_vals no callback set 2023-10-30T12:13:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 10 client.174216.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 10 monclient: _renew_subs 2023-10-30T12:13:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2409ffa0 con 0x7f0a240a4460 2023-10-30T12:13:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.009+0000 7f0a2ac64640 1 librados: init done 2023-10-30T12:13:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a217fa640 1 -- 172.21.15.50:0/3583625876 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0a18004310 con 0x7f0a240a4460 2023-10-30T12:13:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a217fa640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a080392f0 0x7f0a0803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a217fa640 1 -- 172.21.15.50:0/3583625876 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f0a1804aad0 con 0x7f0a240a4460 2023-10-30T12:13:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a217fa640 10 client.174216.objecter ms_dispatch 0x7f0a24000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a217fa640 3 client.174216.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a217fa640 3 client.174216.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a2ac64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a2ac64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:45.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a2ac64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:45.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a24073620 con 0x7f0a240a4460 2023-10-30T12:13:45.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.010+0000 7f0a217fa640 20 client.174216.objecter dump_active .. 0 homeless 2023-10-30T12:13:45.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.011+0000 7f0a28a00640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a080392f0 0x7f0a0803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:45.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.011+0000 7f0a28a00640 10 monclient: get_auth_request con 0x7f0a080392f0 auth_method 0 2023-10-30T12:13:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.011+0000 7f0a28a00640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a080392f0 0x7f0a0803b7b0 secure :-1 s=READY pgs=4160 cs=0 l=1 rev1=1 crypto rx=0x7f0a10036000 tx=0x7f0a1003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.011+0000 7f0a217fa640 10 client.174216.objecter ms_handle_connect 0x7f0a080392f0 2023-10-30T12:13:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.015+0000 7f0a217fa640 1 -- 172.21.15.50:0/3583625876 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0a18013df0 con 0x7f0a240a4460 2023-10-30T12:13:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.015+0000 7f0a217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.015+0000 7f0a217fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.258+0000 7f0a2ac64640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.258+0000 7f0a2ac64640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.258+0000 7f0a2ac64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.258+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0a2409a650 con 0x7f0a240a4460 2023-10-30T12:13:45.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.258+0000 7f0a217fa640 1 -- 172.21.15.50:0/3583625876 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f0a18013390 con 0x7f0a240a4460 2023-10-30T12:13:45.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.258+0000 7f0a217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:45.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.258+0000 7f0a217fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:45.261 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":128,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:45.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 10 librados: watch_flush enter 2023-10-30T12:13:45.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 10 librados: watch_flush exit 2023-10-30T12:13:45.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 20 client.174216.objecter shutdown clearing up homeless session... 2023-10-30T12:13:45.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 10 client.174216.objecter successfully canceled tick 2023-10-30T12:13:45.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a080392f0 msgr2=0x7f0a0803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:45.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a080392f0 0x7f0a0803b7b0 secure :-1 s=READY pgs=4160 cs=0 l=1 rev1=1 crypto rx=0x7f0a10036000 tx=0x7f0a1003d040 comp rx=0 tx=0).stop 2023-10-30T12:13:45.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 10 monclient: shutdown 2023-10-30T12:13:45.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:45.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a4460 msgr2=0x7f0a241cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:45.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a4460 0x7f0a241cacf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800d7d0 tx=0x7f0a1800dc90 comp rx=0 tx=0).stop 2023-10-30T12:13:45.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 shutdown_connections 2023-10-30T12:13:45.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0a080392f0 0x7f0a0803b7b0 unknown :-1 s=CLOSED pgs=4160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a240a7000 0x7f0a2409e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a240a4460 0x7f0a241cacf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 --2- 172.21.15.50:0/3583625876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a2406abe0 0x7f0a241ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:45.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 >> 172.21.15.50:0/3583625876 conn(0x7f0a24066000 msgr2=0x7f0a24131b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:45.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 shutdown_connections 2023-10-30T12:13:45.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 -- 172.21.15.50:0/3583625876 wait complete. 2023-10-30T12:13:45.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:45.260+0000 7f0a2ac64640 1 librados: shutdown 2023-10-30T12:13:48.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:48.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.388+0000 7fab384da640 10 monclient: get_monmap_and_config 2023-10-30T12:13:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.388+0000 7fab384da640 10 monclient: build_initial_monmap 2023-10-30T12:13:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.388+0000 7fab384da640 10 monclient: monmap: 2023-10-30T12:13:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:48.391 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:48.389683+0000 2023-10-30T12:13:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:48.389683+0000 2023-10-30T12:13:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.392 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:48.393 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 1 Processor -- start 2023-10-30T12:13:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 1 -- start start 2023-10-30T12:13:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient: init 2023-10-30T12:13:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T12:13:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab30098520 0x7fab3009bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient(hunting): picked mon.noname-b con 0x7fab30098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab30098900 0x7fab30098d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient(hunting): picked mon.noname-a con 0x7fab30098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab301326d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab30099930 con 0x7fab301326d0 2023-10-30T12:13:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3009aca0 con 0x7fab30098900 2023-10-30T12:13:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3009af80 con 0x7fab30098520 2023-10-30T12:13:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab384da640 10 monclient(hunting): authenticate will time out at 1673.572891s 2023-10-30T12:13:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.389+0000 7fab35a75640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab30098900 0x7fab30098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab30098520 0x7fab3009bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab30098900 0x7fab30098d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50538/0 (socket says 172.21.15.50:50538) 2023-10-30T12:13:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 1 -- 172.21.15.50:0/3794214495 learned_addr learned my addr 172.21.15.50:0/3794214495 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): get_auth_request con 0x7fab30098520 auth_method 0 2023-10-30T12:13:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 10 monclient(hunting): get_auth_request con 0x7fab30098900 auth_method 0 2023-10-30T12:13:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient(hunting): handle_auth_done global_id 114839 payload 274 2023-10-30T12:13:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 1 -- 172.21.15.50:0/3794214495 >> v2:172.21.15.50:3300/0 conn(0x7fab30098900 msgr2=0x7fab30098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 1 --2- 172.21.15.50:0/3794214495 >> v2:172.21.15.50:3300/0 conn(0x7fab30098900 0x7fab30098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 1 -- 172.21.15.50:0/3794214495 >> v1:172.21.15.118:6789/0 conn(0x7fab301326d0 legacy=0x7fab30132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient: _finish_hunting 0 2023-10-30T12:13:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 1 monclient: found mon.noname-b 2023-10-30T12:13:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab36276640 1 -- 172.21.15.50:0/3794214495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3009b260 con 0x7fab30098520 2023-10-30T12:13:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.390+0000 7fab35a75640 1 --2- 172.21.15.50:0/3794214495 >> v2:172.21.15.50:3300/0 conn(0x7fab30098900 0x7fab30098d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab36276640 1 --2- 172.21.15.50:0/3794214495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab30098520 0x7fab3009bf70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009790 tx=0x7fab2c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a875eda30ad0097 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 1 -- 172.21.15.50:0/3794214495 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0316f0 con 0x7fab30098520 2023-10-30T12:13:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: dump: 2023-10-30T12:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.412 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: _finish_auth 0 2023-10-30T12:13:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: _check_auth_tickets 2023-10-30T12:13:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 1 -- 172.21.15.50:0/3794214495 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c031850 con 0x7fab30098520 2023-10-30T12:13:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 1 -- 172.21.15.50:0/3794214495 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c031be0 con 0x7fab30098520 2023-10-30T12:13:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 5 monclient: authenticate success, global_id 114839 2023-10-30T12:13:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 4 set_mon_vals no callback set 2023-10-30T12:13:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab35274640 10 monclient: dump: 2023-10-30T12:13:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.416 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 10 monclient: shutdown 2023-10-30T12:13:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 -- 172.21.15.50:0/3794214495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab30098520 msgr2=0x7fab3009bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 --2- 172.21.15.50:0/3794214495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab30098520 0x7fab3009bf70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009790 tx=0x7fab2c02b790 comp rx=0 tx=0).stop 2023-10-30T12:13:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 -- 172.21.15.50:0/3794214495 shutdown_connections 2023-10-30T12:13:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 --2- 172.21.15.50:0/3794214495 >> v2:172.21.15.50:3300/0 conn(0x7fab30098900 0x7fab30098d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 --2- 172.21.15.50:0/3794214495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab30098520 0x7fab3009bf70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 -- 172.21.15.50:0/3794214495 >> 172.21.15.50:0/3794214495 conn(0x7fab300a05c0 msgr2=0x7fab3009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 -- 172.21.15.50:0/3794214495 shutdown_connections 2023-10-30T12:13:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.391+0000 7fab384da640 1 -- 172.21.15.50:0/3794214495 wait complete. 2023-10-30T12:13:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 10 monclient: build_initial_monmap 2023-10-30T12:13:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 10 monclient: monmap: 2023-10-30T12:13:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:48.419 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:48.420 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:48.421 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 1 librados: starting msgr at 2023-10-30T12:13:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 1 librados: starting objecter 2023-10-30T12:13:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 1 Processor -- start 2023-10-30T12:13:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 1 -- start start 2023-10-30T12:13:48.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 1 librados: setting wanted keys 2023-10-30T12:13:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 1 librados: calling monclient init 2023-10-30T12:13:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.392+0000 7fab384da640 10 monclient: init 2023-10-30T12:13:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:13:48.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab30098520 0x7fab30136050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): picked mon.noname-b con 0x7fab30098520 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab30098900 0x7fab30136590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): picked mon.noname-a con 0x7fab30098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab301326d0 0x7fab30136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): picked mon.noname-c con 0x7fab301326d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab301289d0 con 0x7fab30098900 2023-10-30T12:13:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab301395a0 con 0x7fab30098520 2023-10-30T12:13:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab30071dc0 con 0x7fab301326d0 2023-10-30T12:13:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab384da640 10 monclient(hunting): authenticate will time out at 1673.576339s 2023-10-30T12:13:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab30098900 0x7fab30136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab30098900 0x7fab30136590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50544/0 (socket says 172.21.15.50:50544) 2023-10-30T12:13:48.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 -- 172.21.15.50:0/3157762162 learned_addr learned my addr 172.21.15.50:0/3157762162 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab301326d0 0x7fab30136ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): get_auth_request con 0x7fab30098900 auth_method 0 2023-10-30T12:13:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 10 monclient(hunting): get_auth_request con 0x7fab301326d0 auth_method 0 2023-10-30T12:13:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36276640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab30098520 0x7fab30136050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36276640 10 monclient(hunting): get_auth_request con 0x7fab30098520 auth_method 0 2023-10-30T12:13:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36276640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36276640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36276640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient(hunting): handle_auth_done global_id 174222 payload 824 2023-10-30T12:13:48.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 -- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab301326d0 msgr2=0x7fab30136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab301326d0 0x7fab30136ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 -- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab30098520 msgr2=0x7fab30136050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab30098520 0x7fab30136050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient: _finish_hunting 0 2023-10-30T12:13:48.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 monclient: found mon.noname-a 2023-10-30T12:13:48.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:48.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 -- 172.21.15.50:0/3157762162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab200094d0 con 0x7fab30098900 2023-10-30T12:13:48.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab36a77640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab301326d0 0x7fab30136ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:48.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab35a75640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab30098900 0x7fab30136590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fab2000e9d0 tx=0x7fab2000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab26ffd640 10 client.?.objecter ms_handle_connect 0x7fab30098900 2023-10-30T12:13:48.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.393+0000 7fab26ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:48.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 1 -- 172.21.15.50:0/3157762162 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2000ca00 con 0x7fab30098900 2023-10-30T12:13:48.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:48.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: dump: 2023-10-30T12:13:48.433 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.433 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.433 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.433 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.433 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.434 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.434 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.434 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.434 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.434 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:48.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:48.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:48.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 5 monclient: authenticate success, global_id 174222 2023-10-30T12:13:48.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 1 -- 172.21.15.50:0/3157762162 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2000cb60 con 0x7fab30098900 2023-10-30T12:13:48.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:48.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 1 -- 172.21.15.50:0/3157762162 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab20013470 con 0x7fab30098900 2023-10-30T12:13:48.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 10 monclient: _renew_subs 2023-10-30T12:13:48.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c0093a0 con 0x7fab30098900 2023-10-30T12:13:48.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:48.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab34a73640 4 set_mon_vals no callback set 2023-10-30T12:13:48.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab26ffd640 10 monclient: dump: 2023-10-30T12:13:48.437 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.438 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.438 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.438 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.438 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.438 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.438 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.439 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.439 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.439 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 10 client.174222.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:48.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 10 monclient: _renew_subs 2023-10-30T12:13:48.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab30137a70 con 0x7fab30098900 2023-10-30T12:13:48.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.394+0000 7fab384da640 1 librados: init done 2023-10-30T12:13:48.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab26ffd640 1 -- 172.21.15.50:0/3157762162 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fab200135d0 con 0x7fab30098900 2023-10-30T12:13:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab384da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab384da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab384da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab30128810 con 0x7fab30098900 2023-10-30T12:13:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab26ffd640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab140392f0 0x7fab1403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab26ffd640 1 -- 172.21.15.50:0/3157762162 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fab2004af00 con 0x7fab30098900 2023-10-30T12:13:48.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab26ffd640 10 client.174222.objecter ms_dispatch 0x7fab30000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:48.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab26ffd640 3 client.174222.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:48.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab26ffd640 3 client.174222.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:48.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab26ffd640 20 client.174222.objecter dump_active .. 0 homeless 2023-10-30T12:13:48.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab36276640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab140392f0 0x7fab1403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.395+0000 7fab36276640 10 monclient: get_auth_request con 0x7fab140392f0 auth_method 0 2023-10-30T12:13:48.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.396+0000 7fab26ffd640 10 client.174222.objecter ms_handle_connect 0x7fab140392f0 2023-10-30T12:13:48.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.396+0000 7fab36276640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab140392f0 0x7fab1403b7b0 secure :-1 s=READY pgs=4161 cs=0 l=1 rev1=1 crypto rx=0x7fab2c036000 tx=0x7fab2c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.400+0000 7fab26ffd640 1 -- 172.21.15.50:0/3157762162 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab200123d0 con 0x7fab30098900 2023-10-30T12:13:48.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.400+0000 7fab26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:48.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.400+0000 7fab26ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.644+0000 7fab384da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.644+0000 7fab384da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.644+0000 7fab384da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.644+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab3004e160 con 0x7fab30098900 2023-10-30T12:13:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.644+0000 7fab26ffd640 1 -- 172.21.15.50:0/3157762162 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fab20021020 con 0x7fab30098900 2023-10-30T12:13:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.644+0000 7fab26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.644+0000 7fab26ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:48.648 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 10 librados: watch_flush enter 2023-10-30T12:13:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 10 librados: watch_flush exit 2023-10-30T12:13:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 20 client.174222.objecter shutdown clearing up homeless session... 2023-10-30T12:13:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 10 client.174222.objecter successfully canceled tick 2023-10-30T12:13:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab140392f0 msgr2=0x7fab1403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab140392f0 0x7fab1403b7b0 secure :-1 s=READY pgs=4161 cs=0 l=1 rev1=1 crypto rx=0x7fab2c036000 tx=0x7fab2c008030 comp rx=0 tx=0).stop 2023-10-30T12:13:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 10 monclient: shutdown 2023-10-30T12:13:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab30098900 msgr2=0x7fab30136590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab30098900 0x7fab30136590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fab2000e9d0 tx=0x7fab2000ee90 comp rx=0 tx=0).stop 2023-10-30T12:13:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 shutdown_connections 2023-10-30T12:13:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab140392f0 0x7fab1403b7b0 unknown :-1 s=CLOSED pgs=4161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab301326d0 0x7fab30136ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab30098900 0x7fab30136590 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 --2- 172.21.15.50:0/3157762162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab30098520 0x7fab30136050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 >> 172.21.15.50:0/3157762162 conn(0x7fab300a05c0 msgr2=0x7fab30071450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 shutdown_connections 2023-10-30T12:13:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 -- 172.21.15.50:0/3157762162 wait complete. 2023-10-30T12:13:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.646+0000 7fab384da640 1 librados: shutdown 2023-10-30T12:13:48.658 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:13:48.658 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:13:48.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:13:48.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.763+0000 7fb186fd7640 10 monclient: get_monmap_and_config 2023-10-30T12:13:48.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.763+0000 7fb186fd7640 10 monclient: build_initial_monmap 2023-10-30T12:13:48.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.763+0000 7fb186fd7640 10 monclient: monmap: 2023-10-30T12:13:48.765 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:48.765 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.765 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:48.764103+0000 2023-10-30T12:13:48.765 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:48.764103+0000 2023-10-30T12:13:48.765 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:48.766 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.766 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:48.766 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:48.766 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:48.766 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:48.767 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:48.767 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.763+0000 7fb186fd7640 1 Processor -- start 2023-10-30T12:13:48.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.763+0000 7fb186fd7640 1 -- start start 2023-10-30T12:13:48.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.763+0000 7fb186fd7640 10 monclient: init 2023-10-30T12:13:48.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:48.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:13:48.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1800a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:48.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb1800a2450 0x7fb18013aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient(hunting): picked mon.noname-a con 0x7fb1800a2450 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:48.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb18013af40 0x7fb18013d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient(hunting): picked mon.noname-b con 0x7fb18013af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:48.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1800a4190 con 0x7fb1800a2ae0 2023-10-30T12:13:48.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1800a1630 con 0x7fb1800a2450 2023-10-30T12:13:48.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18009a2e0 con 0x7fb18013af40 2023-10-30T12:13:48.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:48.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb186fd7640 10 monclient(hunting): authenticate will time out at 1673.947270s 2023-10-30T12:13:48.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb18013af40 0x7fb18013d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb18013af40 0x7fb18013d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33560/0 (socket says 172.21.15.50:33560) 2023-10-30T12:13:48.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb1800a2450 0x7fb18013aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 1 -- 172.21.15.50:0/2246788654 learned_addr learned my addr 172.21.15.50:0/2246788654 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:48.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 10 monclient(hunting): get_auth_request con 0x7fb18013af40 auth_method 0 2023-10-30T12:13:48.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb184d73640 1 --1- 172.21.15.50:0/2246788654 >> v1:172.21.15.50:6789/0 conn(0x7fb1800a2ae0 0x7fb1800a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51286/0 (socket says 172.21.15.50:51286) 2023-10-30T12:13:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): get_auth_request con 0x7fb1800a2450 auth_method 0 2023-10-30T12:13:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient(hunting): handle_auth_done global_id 174225 payload 274 2023-10-30T12:13:48.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 1 -- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb18013af40 msgr2=0x7fb18013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:48.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 1 --2- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb18013af40 0x7fb18013d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 1 -- 172.21.15.50:0/2246788654 >> v1:172.21.15.50:6789/0 conn(0x7fb1800a2ae0 legacy=0x7fb1800a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:48.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 1 monclient: found mon.noname-a 2023-10-30T12:13:48.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17f7fe640 1 -- 172.21.15.50:0/2246788654 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425959342 0 0) 0x7fb178001490 con 0x7fb1800a2ae0 2023-10-30T12:13:48.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:48.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17ffff640 1 -- 172.21.15.50:0/2246788654 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18009a8a0 con 0x7fb1800a2450 2023-10-30T12:13:48.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb17f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:13:48.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.764+0000 7fb185574640 1 --2- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb18013af40 0x7fb18013d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17ffff640 1 --2- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb18013aa00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb1680097a0 tx=0x7fb16802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c6887aed3d2d3a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 1 -- 172.21.15.50:0/2246788654 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1680316f0 con 0x7fb1800a2450 2023-10-30T12:13:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: dump: 2023-10-30T12:13:48.786 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.786 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.786 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.786 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.786 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.787 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb186fd7640 5 monclient: authenticate success, global_id 174225 2023-10-30T12:13:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 1 -- 172.21.15.50:0/2246788654 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb168031850 con 0x7fb1800a2450 2023-10-30T12:13:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb186fd7640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb186fd7640 4 set_mon_vals no callback set 2023-10-30T12:13:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 1 -- 172.21.15.50:0/2246788654 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb168031be0 con 0x7fb1800a2450 2023-10-30T12:13:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.765+0000 7fb17f7fe640 10 monclient: dump: 2023-10-30T12:13:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.791 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 10 monclient: shutdown 2023-10-30T12:13:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 -- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 msgr2=0x7fb18013aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 --2- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb18013aa00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb1680097a0 tx=0x7fb16802b790 comp rx=0 tx=0).stop 2023-10-30T12:13:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 -- 172.21.15.50:0/2246788654 shutdown_connections 2023-10-30T12:13:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 --2- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb18013af40 0x7fb18013d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 --2- 172.21.15.50:0/2246788654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb18013aa00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 -- 172.21.15.50:0/2246788654 >> 172.21.15.50:0/2246788654 conn(0x7fb1800a80b0 msgr2=0x7fb1800a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:48.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 -- 172.21.15.50:0/2246788654 shutdown_connections 2023-10-30T12:13:48.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 1 -- 172.21.15.50:0/2246788654 wait complete. 2023-10-30T12:13:48.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 10 monclient: build_initial_monmap 2023-10-30T12:13:48.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.766+0000 7fb186fd7640 10 monclient: monmap: 2023-10-30T12:13:48.794 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:48.794 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:48.794 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:48.794 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:48.795 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:48.795 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.795 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:48.795 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:48.797 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 librados: starting msgr at 2023-10-30T12:13:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 librados: starting objecter 2023-10-30T12:13:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 Processor -- start 2023-10-30T12:13:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 -- start start 2023-10-30T12:13:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 librados: setting wanted keys 2023-10-30T12:13:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 librados: calling monclient init 2023-10-30T12:13:48.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient: init 2023-10-30T12:13:48.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:48.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:13:48.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb180131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): picked mon.noname-a con 0x7fb1800a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:48.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1800a2ae0 0x7fb1801315d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): picked mon.noname-c con 0x7fb1800a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:48.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb18013af40 0x7fb180131b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): picked mon.noname-b con 0x7fb18013af40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:48.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18009c1e0 con 0x7fb1800a2450 2023-10-30T12:13:48.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18009b5d0 con 0x7fb18013af40 2023-10-30T12:13:48.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800a0360 con 0x7fb1800a2ae0 2023-10-30T12:13:48.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:48.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb186fd7640 10 monclient(hunting): authenticate will time out at 1673.950820s 2023-10-30T12:13:48.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb184d73640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb180131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb17ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1800a2ae0 0x7fb1801315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb184d73640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb180131090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50574/0 (socket says 172.21.15.50:50574) 2023-10-30T12:13:48.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb17ffff640 10 monclient(hunting): get_auth_request con 0x7fb1800a2ae0 auth_method 0 2023-10-30T12:13:48.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb184d73640 1 -- 172.21.15.50:0/1097826857 learned_addr learned my addr 172.21.15.50:0/1097826857 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:48.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb17ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb17ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb17ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb184d73640 10 monclient(hunting): get_auth_request con 0x7fb1800a2450 auth_method 0 2023-10-30T12:13:48.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb184d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb184d73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb184d73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.767+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 10 monclient(hunting): handle_auth_done global_id 174231 payload 824 2023-10-30T12:13:48.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb185574640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb18013af40 0x7fb180131b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1800a2ae0 msgr2=0x7fb1801315d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1800a2ae0 0x7fb1801315d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb18013af40 msgr2=0x7fb180131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb18013af40 0x7fb180131b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 10 monclient: _finish_hunting 0 2023-10-30T12:13:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 1 monclient: found mon.noname-a 2023-10-30T12:13:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 1 -- 172.21.15.50:0/1097826857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1700094d0 con 0x7fb1800a2450 2023-10-30T12:13:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb185574640 10 monclient: get_auth_request con 0x7fb18013af40 auth_method 0 2023-10-30T12:13:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17ffff640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1800a2ae0 0x7fb1801315d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb185574640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb18013af40 0x7fb180131b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb184d73640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb180131090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c890 tx=0x7fb17800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 client.?.objecter ms_handle_connect 0x7fb1800a2450 2023-10-30T12:13:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17800a640 con 0x7fb1800a2450 2023-10-30T12:13:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: dump: 2023-10-30T12:13:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.810 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:13:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:13:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 5 monclient: authenticate success, global_id 174231 2023-10-30T12:13:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17800a7a0 con 0x7fb1800a2450 2023-10-30T12:13:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 10 monclient: _renew_subs 2023-10-30T12:13:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb168009400 con 0x7fb1800a2450 2023-10-30T12:13:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17800ab30 con 0x7fb1800a2450 2023-10-30T12:13:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17d7fa640 10 monclient: dump: 2023-10-30T12:13:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.814 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb17effd640 4 set_mon_vals no callback set 2023-10-30T12:13:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 10 client.174231.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 10 monclient: _renew_subs 2023-10-30T12:13:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb180099bc0 con 0x7fb1800a2450 2023-10-30T12:13:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.768+0000 7fb186fd7640 1 librados: init done 2023-10-30T12:13:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb17800ac90 con 0x7fb1800a2450 2023-10-30T12:13:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb186fd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb186fd7640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb18009c930 0x7fb1800a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb186fd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb180073ff0 con 0x7fb18009c930 2023-10-30T12:13:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17d7fa640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1600392f0 0x7fb16003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb18009c930 0x7fb1800a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient: get_auth_request con 0x7fb18009c930 auth_method 0 2023-10-30T12:13:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fb17804a340 con 0x7fb1800a2450 2023-10-30T12:13:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17d7fa640 10 client.174231.objecter ms_dispatch 0x7fb180000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17d7fa640 3 client.174231.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17d7fa640 3 client.174231.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17d7fa640 20 client.174231.objecter dump_active .. 0 homeless 2023-10-30T12:13:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb185574640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1600392f0 0x7fb16003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 10 monclient(hunting): handle_auth_done global_id 174234 payload 824 2023-10-30T12:13:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb17ffff640 1 -- 172.21.15.50:0/1097826857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb18009a5c0 con 0x7fb18009c930 2023-10-30T12:13:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.769+0000 7fb185574640 10 monclient: get_auth_request con 0x7fb1600392f0 auth_method 0 2023-10-30T12:13:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17ffff640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb18009c930 0x7fb1800a5850 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1800a3c80 tx=0x7fb1680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 client.174231.objecter ms_handle_connect 0x7fb18009c930 2023-10-30T12:13:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 client.174231.objecter resend_mon_ops 2023-10-30T12:13:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb168034c10 con 0x7fb18009c930 2023-10-30T12:13:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 monclient: dump: 2023-10-30T12:13:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.823 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb168042070 con 0x7fb18009c930 2023-10-30T12:13:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb18009c930 msgr2=0x7fb1800a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb18009c930 0x7fb1800a5850 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1800a3c80 tx=0x7fb1680046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb18009c930 msgr2=0x7fb1800a5850 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb18009c930 0x7fb1800a5850 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1800a3c80 tx=0x7fb1680046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb17d7fa640 10 client.174231.objecter ms_handle_connect 0x7fb1600392f0 2023-10-30T12:13:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.770+0000 7fb185574640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1600392f0 0x7fb16003b7b0 secure :-1 s=READY pgs=4162 cs=0 l=1 rev1=1 crypto rx=0x7fb180133130 tx=0x7fb170008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb186fd7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb186fd7640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a5d90 0x7fb1800a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb186fd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb180073a20 con 0x7fb1800a5d90 2023-10-30T12:13:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a5d90 0x7fb1800a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 10 monclient: get_auth_request con 0x7fb1800a5d90 auth_method 0 2023-10-30T12:13:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.776+0000 7fb17ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17ffff640 10 monclient(hunting): handle_auth_done global_id 174237 payload 824 2023-10-30T12:13:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17ffff640 1 -- 172.21.15.50:0/1097826857 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb1800a28a0 con 0x7fb1800a5d90 2023-10-30T12:13:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 10 client.174231.objecter ms_handle_connect 0x7fb1800a5d90 2023-10-30T12:13:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 10 client.174231.objecter resend_mon_ops 2023-10-30T12:13:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17ffff640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a5d90 0x7fb1800a6170 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb16802b7d0 tx=0x7fb168009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16803e790 con 0x7fb1800a5d90 2023-10-30T12:13:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 10 monclient: dump: 2023-10-30T12:13:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:48.832 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7fb16803e8f0 con 0x7fb1800a5d90 2023-10-30T12:13:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a5d90 msgr2=0x7fb1800a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a5d90 0x7fb1800a6170 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb16802b7d0 tx=0x7fb168009190 comp rx=0 tx=0).stop 2023-10-30T12:13:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a5d90 msgr2=0x7fb1800a6170 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb17d7fa640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a5d90 0x7fb1800a6170 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb16802b7d0 tx=0x7fb168009190 comp rx=0 tx=0).stop 2023-10-30T12:13:48.833 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:48.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:48.834 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:48.834 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:13:48.834 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 130, 2023-10-30T12:13:48.834 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:48.835 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:48.835 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:48.835 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:48.835 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:48.835 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:13:48.835 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:48.836 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:48.836 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:48.836 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:48.836 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:48.836 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:48.837 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:48.837 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:48.837 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:48.837 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:48.837 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:48.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:48.838 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:48.838 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:48.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:48.839 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:48.839 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:48.839 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:48.839 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:48.839 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:48.839 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:48.840 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:48.840 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:48.840 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:48.840 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:48.840 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:48.840 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:48.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.841 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:48.841 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:48.841 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:48.841 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:48.842 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:48.842 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:48.842 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:48.842 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:48.842 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:48.843 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:48.843 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:48.843 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:48.843 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:48.843 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:48.843 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:48.844 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:48.844 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:48.844 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:48.844 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:48.844 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:48.845 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:48.845 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:48.845 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:48.845 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:48.845 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:48.845 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:48.846 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:48.846 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:48.846 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:48.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.846 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:48.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.847 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:48.847 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:48.847 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:48.847 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:48.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:48.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:48.848 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:48.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:48.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:48.849 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:48.849 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:48.849 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:48.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.850 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:48.850 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:48.850 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:48.850 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:48.850 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:48.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.851 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:48.851 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:48.851 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:48.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:48.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.852 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:48.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:48.852 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:48.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:48.853 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:48.853 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:48.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:48.854 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:48.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.854 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:48.854 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:48.854 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:48.855 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:48.855 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:48.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.855 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:48.855 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:48.856 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:48.856 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:48.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:48.856 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:48.857 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:48.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.857 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:48.857 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:48.858 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:48.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:48.858 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:48.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.858 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:48.859 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:48.859 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:48.859 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:48.859 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:48.859 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:48.859 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:13:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:48.863 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:48.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 10 librados: watch_flush enter 2023-10-30T12:13:48.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 10 librados: watch_flush exit 2023-10-30T12:13:48.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 20 client.174231.objecter shutdown clearing up homeless session... 2023-10-30T12:13:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 10 client.174231.objecter successfully canceled tick 2023-10-30T12:13:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1600392f0 msgr2=0x7fb16003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1600392f0 0x7fb16003b7b0 secure :-1 s=READY pgs=4162 cs=0 l=1 rev1=1 crypto rx=0x7fb180133130 tx=0x7fb170008040 comp rx=0 tx=0).stop 2023-10-30T12:13:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 10 monclient: shutdown 2023-10-30T12:13:48.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 msgr2=0x7fb180131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb186fd7640 1 --2- 172.21.15.50:0/1097826857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1800a2450 0x7fb180131090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c890 tx=0x7fb17800cd50 comp rx=0 tx=0).stop 2023-10-30T12:13:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.777+0000 7fb185574640 1 -- 172.21.15.50:0/1097826857 reap_dead start 2023-10-30T12:13:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.778+0000 7fb185574640 1 -- 172.21.15.50:0/1097826857 reap_dead start 2023-10-30T12:13:48.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.778+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 shutdown_connections 2023-10-30T12:13:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.778+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 >> 172.21.15.50:0/1097826857 conn(0x7fb1800a80b0 msgr2=0x7fb18012ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.778+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 shutdown_connections 2023-10-30T12:13:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.778+0000 7fb186fd7640 1 -- 172.21.15.50:0/1097826857 wait complete. 2023-10-30T12:13:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.778+0000 7fb186fd7640 1 librados: shutdown 2023-10-30T12:13:48.869 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:48.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:13:48.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:13:48.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.974+0000 7f5871239640 10 monclient: get_monmap_and_config 2023-10-30T12:13:48.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.974+0000 7f5871239640 10 monclient: build_initial_monmap 2023-10-30T12:13:48.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.974+0000 7f5871239640 10 monclient: monmap: 2023-10-30T12:13:48.976 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:48.977 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:48.977 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:48.975735+0000 2023-10-30T12:13:48.977 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:48.975735+0000 2023-10-30T12:13:48.977 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:48.977 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:48.978 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:48.978 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:48.978 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:48.978 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:48.978 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:48.979 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:48.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 1 Processor -- start 2023-10-30T12:13:48.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 1 -- start start 2023-10-30T12:13:48.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient: init 2023-10-30T12:13:48.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:48.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:13:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f586c09bb70 0x7f586c09bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient(hunting): picked mon.noname-c con 0x7f586c09bb70 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:48.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f586c09d3f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:48.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09c490 0x7f586c096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:48.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient(hunting): picked mon.noname-b con 0x7f586c09c490 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:48.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586c098170 con 0x7f586c09d3f0 2023-10-30T12:13:48.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f586c09cf60 con 0x7f586c09bb70 2023-10-30T12:13:48.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:48.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c097b20 con 0x7f586c09c490 2023-10-30T12:13:49.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:49.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f5871239640 10 monclient(hunting): authenticate will time out at 1674.158882s 2023-10-30T12:13:49.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.975+0000 7f586b59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09c490 0x7f586c096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09c490 0x7f586c096790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33576/0 (socket says 172.21.15.50:33576) 2023-10-30T12:13:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 1 -- 172.21.15.50:0/1235089042 learned_addr learned my addr 172.21.15.50:0/1235089042 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:49.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): get_auth_request con 0x7f586c09c490 auth_method 0 2023-10-30T12:13:49.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586ad9d640 1 --2- 172.21.15.50:0/1235089042 >> v2:172.21.15.118:3300/0 conn(0x7f586c09bb70 0x7f586c09bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586ad9d640 10 monclient(hunting): get_auth_request con 0x7f586c09bb70 auth_method 0 2023-10-30T12:13:49.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient(hunting): handle_auth_done global_id 114851 payload 274 2023-10-30T12:13:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 1 -- 172.21.15.50:0/1235089042 >> v2:172.21.15.118:3300/0 conn(0x7f586c09bb70 msgr2=0x7f586c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 1 --2- 172.21.15.50:0/1235089042 >> v2:172.21.15.118:3300/0 conn(0x7f586c09bb70 0x7f586c09bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 1 -- 172.21.15.50:0/1235089042 >> v1:172.21.15.118:6789/0 conn(0x7f586c09d3f0 legacy=0x7f586c096080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient: _finish_hunting 0 2023-10-30T12:13:49.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 1 monclient: found mon.noname-b 2023-10-30T12:13:49.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.976+0000 7f586b59e640 1 -- 172.21.15.50:0/1235089042 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c097e00 con 0x7f586c09c490 2023-10-30T12:13:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586b59e640 1 --2- 172.21.15.50:0/1235089042 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09c490 0x7f586c096790 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f586400d150 tx=0x7f586402e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a69ae0ee69e1c887 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 1 -- 172.21.15.50:0/1235089042 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586402e9e0 con 0x7f586c09c490 2023-10-30T12:13:49.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:49.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: dump: 2023-10-30T12:13:49.009 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.009 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.010 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.010 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.010 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.011 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: _finish_auth 0 2023-10-30T12:13:49.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: _check_auth_tickets 2023-10-30T12:13:49.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:49.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 1 -- 172.21.15.50:0/1235089042 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586402eb40 con 0x7f586c09c490 2023-10-30T12:13:49.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 5 monclient: authenticate success, global_id 114851 2023-10-30T12:13:49.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:49.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:49.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 1 -- 172.21.15.50:0/1235089042 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864009600 con 0x7f586c09c490 2023-10-30T12:13:49.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:49.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 4 set_mon_vals no callback set 2023-10-30T12:13:49.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:49.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f586a59c640 10 monclient: dump: 2023-10-30T12:13:49.014 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.014 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.014 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.015 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.015 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.015 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.015 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.015 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.016 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.016 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 10 monclient: shutdown 2023-10-30T12:13:49.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:49.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 -- 172.21.15.50:0/1235089042 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09c490 msgr2=0x7f586c096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 --2- 172.21.15.50:0/1235089042 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09c490 0x7f586c096790 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f586400d150 tx=0x7f586402e040 comp rx=0 tx=0).stop 2023-10-30T12:13:49.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 -- 172.21.15.50:0/1235089042 shutdown_connections 2023-10-30T12:13:49.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 --2- 172.21.15.50:0/1235089042 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09c490 0x7f586c096790 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 --2- 172.21.15.50:0/1235089042 >> v2:172.21.15.118:3300/0 conn(0x7f586c09bb70 0x7f586c09bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 -- 172.21.15.50:0/1235089042 >> 172.21.15.50:0/1235089042 conn(0x7f586c0a3980 msgr2=0x7f586c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 -- 172.21.15.50:0/1235089042 shutdown_connections 2023-10-30T12:13:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.977+0000 7f5871239640 1 -- 172.21.15.50:0/1235089042 wait complete. 2023-10-30T12:13:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 10 monclient: build_initial_monmap 2023-10-30T12:13:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 10 monclient: monmap: 2023-10-30T12:13:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:49.020 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.020 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:49.020 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:49.020 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:49.020 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 1 librados: starting msgr at 2023-10-30T12:13:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 1 librados: starting objecter 2023-10-30T12:13:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 1 Processor -- start 2023-10-30T12:13:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 1 -- start start 2023-10-30T12:13:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 1 librados: setting wanted keys 2023-10-30T12:13:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 1 librados: calling monclient init 2023-10-30T12:13:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.978+0000 7f5871239640 10 monclient: init 2023-10-30T12:13:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:13:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09bb70 0x7f586c1c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): picked mon.noname-c con 0x7f586c09bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f586c09c490 0x7f586c0943c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): picked mon.noname-b con 0x7f586c09c490 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f586c09d3f0 0x7f586c094900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): picked mon.noname-a con 0x7f586c09d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c1c36a0 con 0x7f586c09d3f0 2023-10-30T12:13:49.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c08e890 con 0x7f586c09c490 2023-10-30T12:13:49.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c08d040 con 0x7f586c09bb70 2023-10-30T12:13:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5871239640 10 monclient(hunting): authenticate will time out at 1674.162441s 2023-10-30T12:13:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09bb70 0x7f586c1c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09bb70 0x7f586c1c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33580/0 (socket says 172.21.15.50:33580) 2023-10-30T12:13:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 -- 172.21.15.50:0/495483904 learned_addr learned my addr 172.21.15.50:0/495483904 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): get_auth_request con 0x7f586c09bb70 auth_method 0 2023-10-30T12:13:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f586c09d3f0 0x7f586c094900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 10 monclient(hunting): get_auth_request con 0x7f586c09d3f0 auth_method 0 2023-10-30T12:13:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f586c09c490 0x7f586c0943c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 10 monclient(hunting): get_auth_request con 0x7f586c09c490 auth_method 0 2023-10-30T12:13:49.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient(hunting): handle_auth_done global_id 114854 payload 824 2023-10-30T12:13:49.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f586c09c490 msgr2=0x7f586c0943c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f586c09c490 0x7f586c0943c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f586c09d3f0 msgr2=0x7f586c094900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f586c09d3f0 0x7f586c094900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:13:49.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 monclient: found mon.noname-c 2023-10-30T12:13:49.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:49.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:49.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 -- 172.21.15.50:0/495483904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586400bd40 con 0x7f586c09bb70 2023-10-30T12:13:49.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.979+0000 7f586ad9d640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09bb70 0x7f586c1c5e60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f585c00d800 tx=0x7f585c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 client.?.objecter ms_handle_connect 0x7f586c09bb70 2023-10-30T12:13:49.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:49.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c00b6c0 con 0x7f586c09bb70 2023-10-30T12:13:49.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:49.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: dump: 2023-10-30T12:13:49.034 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.034 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.034 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.034 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.036 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: _finish_auth 0 2023-10-30T12:13:49.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: _check_auth_tickets 2023-10-30T12:13:49.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:49.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585c00b820 con 0x7f586c09bb70 2023-10-30T12:13:49.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:49.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c00bbd0 con 0x7f586c09bb70 2023-10-30T12:13:49.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 5 monclient: authenticate success, global_id 114854 2023-10-30T12:13:49.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 10 monclient: _renew_subs 2023-10-30T12:13:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586c0976e0 con 0x7f586c09bb70 2023-10-30T12:13:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5869d9b640 4 set_mon_vals no callback set 2023-10-30T12:13:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f58637fe640 10 monclient: dump: 2023-10-30T12:13:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.041 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.041 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 10 client.114854.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:49.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 10 monclient: _renew_subs 2023-10-30T12:13:49.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:49.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c08e5b0 con 0x7f586c09bb70 2023-10-30T12:13:49.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 1 librados: init done 2023-10-30T12:13:49.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:49.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.980+0000 7f5871239640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08afd0 0x7f586c08b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5871239640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c073ff0 con 0x7f586c08afd0 2023-10-30T12:13:49.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f585c00bd30 con 0x7f586c09bb70 2023-10-30T12:13:49.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08afd0 0x7f586c08b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient: get_auth_request con 0x7f586c08afd0 auth_method 0 2023-10-30T12:13:49.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f585c04b0e0 con 0x7f586c09bb70 2023-10-30T12:13:49.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 client.114854.objecter ms_dispatch 0x7f586c000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:49.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 3 client.114854.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:49.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 3 client.114854.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:49.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 20 client.114854.objecter dump_active .. 0 homeless 2023-10-30T12:13:49.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 10 monclient(hunting): handle_auth_done global_id 114857 payload 824 2023-10-30T12:13:49.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 1 -- 172.21.15.50:0/495483904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f586c097560 con 0x7f586c08afd0 2023-10-30T12:13:49.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f586b59e640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f586b59e640 10 monclient: get_auth_request con 0x7f584c0392f0 auth_method 0 2023-10-30T12:13:49.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 client.114854.objecter ms_handle_connect 0x7f586c08afd0 2023-10-30T12:13:49.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 client.114854.objecter resend_mon_ops 2023-10-30T12:13:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f5863fff640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08afd0 0x7f586c08b3b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f586c08c6c0 tx=0x7f585400fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58540050d0 con 0x7f586c08afd0 2023-10-30T12:13:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 monclient: dump: 2023-10-30T12:13:49.047 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.048 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.049 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 client.114854.objecter ms_handle_connect 0x7f584c0392f0 2023-10-30T12:13:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f586b59e640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 secure :-1 s=READY pgs=4163 cs=0 l=1 rev1=1 crypto rx=0x7f5864037000 tx=0x7f5864032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5854005230 con 0x7f586c08afd0 2023-10-30T12:13:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:49.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08afd0 msgr2=0x7f586c08b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08afd0 0x7f586c08b3b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f586c08c6c0 tx=0x7f585400fc00 comp rx=0 tx=0).stop 2023-10-30T12:13:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.981+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08afd0 msgr2=0x7f586c08b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.982+0000 7f58637fe640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08afd0 0x7f586c08b3b0 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f586c08c6c0 tx=0x7f585400fc00 comp rx=0 tx=0).stop 2023-10-30T12:13:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5871239640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5871239640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08b8f0 0x7f586c06e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5871239640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c09c8f0 con 0x7f586c08b8f0 2023-10-30T12:13:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08b8f0 0x7f586c06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient: get_auth_request con 0x7f586c08b8f0 auth_method 0 2023-10-30T12:13:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 10 monclient(hunting): handle_auth_done global_id 114860 payload 824 2023-10-30T12:13:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 1 -- 172.21.15.50:0/495483904 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f586c04e160 con 0x7f586c08b8f0 2023-10-30T12:13:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f5863fff640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08b8f0 0x7f586c06e560 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f586c08c6c0 tx=0x7f5854001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f58637fe640 10 client.114854.objecter ms_handle_connect 0x7f586c08b8f0 2023-10-30T12:13:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f58637fe640 10 client.114854.objecter resend_mon_ops 2023-10-30T12:13:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5854024070 con 0x7f586c08b8f0 2023-10-30T12:13:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f58637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f58637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.988+0000 7f58637fe640 10 monclient: dump: 2023-10-30T12:13:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.057 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f5854007300 con 0x7f586c08b8f0 2023-10-30T12:13:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f58637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f58637fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08b8f0 msgr2=0x7f586c06e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f58637fe640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08b8f0 0x7f586c06e560 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f586c08c6c0 tx=0x7f5854001120 comp rx=0 tx=0).stop 2023-10-30T12:13:49.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f58637fe640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08b8f0 msgr2=0x7f586c06e560 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:49.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f58637fe640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c08b8f0 0x7f586c06e560 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f586c08c6c0 tx=0x7f5854001120 comp rx=0 tx=0).stop 2023-10-30T12:13:49.059 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:49.059 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:49.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:49.059 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:49.059 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 130, 2023-10-30T12:13:49.060 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:49.060 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:49.060 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:49.060 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:49.061 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.061 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:13:49.061 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:49.061 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:49.061 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:49.062 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:49.062 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:49.062 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:49.062 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:49.063 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:49.063 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:49.063 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:49.064 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:49.064 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:49.064 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:49.064 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.064 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:49.065 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:49.065 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:49.065 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:49.065 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:49.065 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:49.066 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:49.066 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:49.066 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:49.066 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:49.066 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:49.067 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:49.067 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.067 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:49.068 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:49.068 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:49.068 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:49.068 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:49.069 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:49.069 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:49.069 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:49.069 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:49.069 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:13:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:13:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:13:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:13:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:49.099 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 10 librados: watch_flush enter 2023-10-30T12:13:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 10 librados: watch_flush exit 2023-10-30T12:13:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 20 client.114854.objecter shutdown clearing up homeless session... 2023-10-30T12:13:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 10 client.114854.objecter successfully canceled tick 2023-10-30T12:13:49.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 msgr2=0x7f584c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f584c0392f0 0x7f584c03b7b0 secure :-1 s=READY pgs=4163 cs=0 l=1 rev1=1 crypto rx=0x7f5864037000 tx=0x7f5864032000 comp rx=0 tx=0).stop 2023-10-30T12:13:49.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 10 monclient: shutdown 2023-10-30T12:13:49.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:49.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09bb70 msgr2=0x7f586c1c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09bb70 0x7f586c1c5e60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f585c00d800 tx=0x7f585c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f586b59e640 1 -- 172.21.15.50:0/495483904 reap_dead start 2023-10-30T12:13:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 shutdown_connections 2023-10-30T12:13:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 --2- 172.21.15.50:0/495483904 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f586c09bb70 0x7f586c1c5e60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 >> 172.21.15.50:0/495483904 conn(0x7f586c0a3980 msgr2=0x7f586c096500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 shutdown_connections 2023-10-30T12:13:49.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 -- 172.21.15.50:0/495483904 wait complete. 2023-10-30T12:13:49.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:48.989+0000 7f5871239640 1 librados: shutdown 2023-10-30T12:13:49.107 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:49.107 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:13:49.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:13:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.214+0000 7fa417664640 10 monclient: get_monmap_and_config 2023-10-30T12:13:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.214+0000 7fa417664640 10 monclient: build_initial_monmap 2023-10-30T12:13:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.214+0000 7fa417664640 10 monclient: monmap: 2023-10-30T12:13:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:49.215240+0000 2023-10-30T12:13:49.216 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:49.215240+0000 2023-10-30T12:13:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:49.218 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:49.218 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:49.218 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.214+0000 7fa417664640 1 Processor -- start 2023-10-30T12:13:49.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 1 -- start start 2023-10-30T12:13:49.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient: init 2023-10-30T12:13:49.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:49.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:13:49.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4100a2ae0 0x7fa4100a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient(hunting): picked mon.noname-a con 0x7fa4100a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:49.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4100a2ec0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:49.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa410135df0 0x7fa41013da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient(hunting): picked mon.noname-c con 0x7fa410135df0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:49.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4100a4190 con 0x7fa4100a2ec0 2023-10-30T12:13:49.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4100a2990 con 0x7fa4100a2ae0 2023-10-30T12:13:49.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4100a1630 con 0x7fa410135df0 2023-10-30T12:13:49.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:49.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa417664640 10 monclient(hunting): authenticate will time out at 1674.398735s 2023-10-30T12:13:49.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4100a2ae0 0x7fa4100a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa414bff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa4100a2ec0 0x7fa4101356e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51288/0 (socket says 172.21.15.50:51288) 2023-10-30T12:13:49.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa414bff640 1 -- 172.21.15.50:0/538940535 learned_addr learned my addr 172.21.15.50:0/538940535 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:49.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 10 monclient(hunting): get_auth_request con 0x7fa4100a2ae0 auth_method 0 2023-10-30T12:13:49.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.215+0000 7fa415400640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415c01640 1 --2- 172.21.15.50:0/538940535 >> v2:172.21.15.118:3300/0 conn(0x7fa410135df0 0x7fa41013da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 10 monclient(hunting): handle_auth_done global_id 174243 payload 274 2023-10-30T12:13:49.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 1 -- 172.21.15.50:0/538940535 >> v2:172.21.15.118:3300/0 conn(0x7fa410135df0 msgr2=0x7fa41013da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:49.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 1 --2- 172.21.15.50:0/538940535 >> v2:172.21.15.118:3300/0 conn(0x7fa410135df0 0x7fa41013da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 1 -- 172.21.15.50:0/538940535 >> v1:172.21.15.50:6789/0 conn(0x7fa4100a2ec0 legacy=0x7fa4101356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 10 monclient: _finish_hunting 0 2023-10-30T12:13:49.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 1 monclient: found mon.noname-a 2023-10-30T12:13:49.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:49.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 1 -- 172.21.15.50:0/538940535 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41009a2e0 con 0x7fa4100a2ae0 2023-10-30T12:13:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.216+0000 7fa415400640 1 --2- 172.21.15.50:0/538940535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4100a2450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa4100a54d0 tx=0x7fa40c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d31f2b8ffb8f31 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 1 -- 172.21.15.50:0/538940535 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c031710 con 0x7fa4100a2ae0 2023-10-30T12:13:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: dump: 2023-10-30T12:13:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.241 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.241 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.241 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.241 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.241 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.242 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.242 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.242 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.242 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: _finish_auth 0 2023-10-30T12:13:49.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: _check_auth_tickets 2023-10-30T12:13:49.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:49.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 1 -- 172.21.15.50:0/538940535 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c02f050 con 0x7fa4100a2ae0 2023-10-30T12:13:49.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 5 monclient: authenticate success, global_id 174243 2023-10-30T12:13:49.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:49.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:49.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:49.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 1 -- 172.21.15.50:0/538940535 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c031a90 con 0x7fa4100a2ae0 2023-10-30T12:13:49.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 4 set_mon_vals no callback set 2023-10-30T12:13:49.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:49.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa407fff640 10 monclient: dump: 2023-10-30T12:13:49.245 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.245 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.245 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.245 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.246 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.246 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.246 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.246 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.247 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.247 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 10 monclient: shutdown 2023-10-30T12:13:49.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:49.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 -- 172.21.15.50:0/538940535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 msgr2=0x7fa4100a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 --2- 172.21.15.50:0/538940535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4100a2450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa4100a54d0 tx=0x7fa40c02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 -- 172.21.15.50:0/538940535 shutdown_connections 2023-10-30T12:13:49.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 --2- 172.21.15.50:0/538940535 >> v2:172.21.15.118:3300/0 conn(0x7fa410135df0 0x7fa41013da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 --2- 172.21.15.50:0/538940535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4100a2450 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 -- 172.21.15.50:0/538940535 >> 172.21.15.50:0/538940535 conn(0x7fa4100a80b0 msgr2=0x7fa4100a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:49.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 -- 172.21.15.50:0/538940535 shutdown_connections 2023-10-30T12:13:49.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.217+0000 7fa417664640 1 -- 172.21.15.50:0/538940535 wait complete. 2023-10-30T12:13:49.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 10 monclient: build_initial_monmap 2023-10-30T12:13:49.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 10 monclient: monmap: 2023-10-30T12:13:49.251 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:49.251 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:49.251 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:49.252 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:49.252 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:49.252 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.252 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:49.252 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:49.252 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:49.253 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 1 librados: starting msgr at 2023-10-30T12:13:49.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 1 librados: starting objecter 2023-10-30T12:13:49.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 1 Processor -- start 2023-10-30T12:13:49.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 1 -- start start 2023-10-30T12:13:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 1 librados: setting wanted keys 2023-10-30T12:13:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 1 librados: calling monclient init 2023-10-30T12:13:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.218+0000 7fa417664640 10 monclient: init 2023-10-30T12:13:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:13:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4101ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): picked mon.noname-a con 0x7fa4100a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:49.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4100a2ec0 0x7fa41011f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): picked mon.noname-b con 0x7fa4100a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:49.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa410135df0 0x7fa41011fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): picked mon.noname-c con 0x7fa410135df0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:49.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa410123010 con 0x7fa4100a2ae0 2023-10-30T12:13:49.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa410122120 con 0x7fa4100a2ec0 2023-10-30T12:13:49.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100a1790 con 0x7fa410135df0 2023-10-30T12:13:49.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:49.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient(hunting): authenticate will time out at 1674.402191s 2023-10-30T12:13:49.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4101ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa410135df0 0x7fa41011fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4101ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50604/0 (socket says 172.21.15.50:50604) 2023-10-30T12:13:49.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 -- 172.21.15.50:0/2322348822 learned_addr learned my addr 172.21.15.50:0/2322348822 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:49.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa410135df0 0x7fa41011fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33604/0 (socket says 172.21.15.50:33604) 2023-10-30T12:13:49.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): get_auth_request con 0x7fa4100a2ae0 auth_method 0 2023-10-30T12:13:49.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 10 monclient(hunting): get_auth_request con 0x7fa410135df0 auth_method 0 2023-10-30T12:13:49.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4100a2ec0 0x7fa41011f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient(hunting): handle_auth_done global_id 174249 payload 824 2023-10-30T12:13:49.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa410135df0 msgr2=0x7fa41011fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa410135df0 0x7fa41011fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4100a2ec0 msgr2=0x7fa41011f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4100a2ec0 0x7fa41011f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient: _finish_hunting 0 2023-10-30T12:13:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 monclient: found mon.noname-a 2023-10-30T12:13:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 -- 172.21.15.50:0/2322348822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40c009400 con 0x7fa4100a2ae0 2023-10-30T12:13:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa414bff640 10 monclient: get_auth_request con 0x7fa4100a2ec0 auth_method 0 2023-10-30T12:13:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4100a2ec0 0x7fa41011f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415c01640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa410135df0 0x7fa41011fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa415400640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4101ca5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa40c000c00 tx=0x7fa40c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 10 client.?.objecter ms_handle_connect 0x7fa4100a2ae0 2023-10-30T12:13:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c033a50 con 0x7fa4100a2ae0 2023-10-30T12:13:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 10 monclient: dump: 2023-10-30T12:13:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.269 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.269 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.271 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 10 monclient: _finish_auth 0 2023-10-30T12:13:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:13:49.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:49.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c033bb0 con 0x7fa4100a2ae0 2023-10-30T12:13:49.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 5 monclient: authenticate success, global_id 174249 2023-10-30T12:13:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient: _renew_subs 2023-10-30T12:13:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.219+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4101355a0 con 0x7fa4100a2ae0 2023-10-30T12:13:49.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa405ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:49.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c03c400 con 0x7fa4100a2ae0 2023-10-30T12:13:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa405ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa405ffb640 10 monclient: dump: 2023-10-30T12:13:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.277 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa417664640 10 client.174249.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa417664640 10 monclient: _renew_subs 2023-10-30T12:13:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa417664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4101214e0 con 0x7fa4100a2ae0 2023-10-30T12:13:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa4077fe640 4 set_mon_vals no callback set 2023-10-30T12:13:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.220+0000 7fa417664640 1 librados: init done 2023-10-30T12:13:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa40c03a040 con 0x7fa4100a2ae0 2023-10-30T12:13:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa417664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa417664640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4101cab20 0x7fa4101202c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa417664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa410123210 con 0x7fa4101cab20 2023-10-30T12:13:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3f40392f0 0x7fa3f403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4101cab20 0x7fa4101202c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fa40c06c880 con 0x7fa4100a2ae0 2023-10-30T12:13:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 10 client.174249.objecter ms_dispatch 0x7fa410000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 3 client.174249.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 3 client.174249.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient: get_auth_request con 0x7fa4101cab20 auth_method 0 2023-10-30T12:13:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 20 client.174249.objecter dump_active .. 0 homeless 2023-10-30T12:13:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa415c01640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3f40392f0 0x7fa3f403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa415c01640 10 monclient: get_auth_request con 0x7fa3f40392f0 auth_method 0 2023-10-30T12:13:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 10 monclient(hunting): handle_auth_done global_id 184114 payload 824 2023-10-30T12:13:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 1 -- 172.21.15.50:0/2322348822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa41009a5c0 con 0x7fa4101cab20 2023-10-30T12:13:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 10 client.174249.objecter ms_handle_connect 0x7fa4101cab20 2023-10-30T12:13:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 10 client.174249.objecter resend_mon_ops 2023-10-30T12:13:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4101cab20 0x7fa4101202c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa41011f7e0 tx=0x7fa40000d970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400005cc0 con 0x7fa4101cab20 2023-10-30T12:13:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa415c01640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3f40392f0 0x7fa3f403b7b0 secure :-1 s=READY pgs=4164 cs=0 l=1 rev1=1 crypto rx=0x7fa40800d770 tx=0x7fa4080076f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 10 monclient: dump: 2023-10-30T12:13:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.221+0000 7fa405ffb640 10 client.174249.objecter ms_handle_connect 0x7fa3f40392f0 2023-10-30T12:13:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.222+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa400005e20 con 0x7fa4101cab20 2023-10-30T12:13:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.222+0000 7fa405ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.222+0000 7fa405ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.222+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4101cab20 msgr2=0x7fa4101202c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.222+0000 7fa405ffb640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4101cab20 0x7fa4101202c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa41011f7e0 tx=0x7fa40000d970 comp rx=0 tx=0).stop 2023-10-30T12:13:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.222+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4101cab20 msgr2=0x7fa4101202c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.222+0000 7fa405ffb640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa4101cab20 0x7fa4101202c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa417664640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa417664640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa417664640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa410073ff0 con 0x7fa410120800 2023-10-30T12:13:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 10 monclient: get_auth_request con 0x7fa410120800 auth_method 0 2023-10-30T12:13:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.228+0000 7fa414bff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.229+0000 7fa414bff640 10 monclient(hunting): handle_auth_done global_id 184117 payload 824 2023-10-30T12:13:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.229+0000 7fa414bff640 1 -- 172.21.15.50:0/2322348822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa410099b40 con 0x7fa410120800 2023-10-30T12:13:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.229+0000 7fa414bff640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 msgr2=0x7fa41009dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:13:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.229+0000 7fa414bff640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 msgr2=0x7fa41009dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:13:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.229+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fa400013000 tx=0x7fa4000057f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.229+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:49.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: get_auth_request con 0x7fa410120800 auth_method 0 2023-10-30T12:13:49.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: _init_auth method 2 2023-10-30T12:13:49.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: _init_auth already have auth, reseting 2023-10-30T12:13:49.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: handle_auth_reply_more payload 9 2023-10-30T12:13:49.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: handle_auth_reply_more responding with 132 bytes 2023-10-30T12:13:49.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 10 monclient: handle_auth_done global_id 184117 payload 293 2023-10-30T12:13:49.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa414bff640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa400007d80 tx=0x7fa400007000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa405ffb640 10 client.174249.objecter ms_handle_connect 0x7fa410120800 2023-10-30T12:13:49.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa405ffb640 10 client.174249.objecter resend_mon_ops 2023-10-30T12:13:49.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400022070 con 0x7fa410120800 2023-10-30T12:13:49.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa405ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:49.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.430+0000 7fa405ffb640 10 monclient: dump: 2023-10-30T12:13:49.437 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.437 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.437 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.437 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.437 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.438 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.438 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.438 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.438 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.438 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fa4000124c0 con 0x7fa410120800 2023-10-30T12:13:49.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa405ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:49.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa405ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:49.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 msgr2=0x7fa41009dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa405ffb640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa400007d80 tx=0x7fa400007000 comp rx=0 tx=0).stop 2023-10-30T12:13:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa405ffb640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 msgr2=0x7fa41009dde0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa405ffb640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa410120800 0x7fa41009dde0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.440 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:49.440 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:49.441 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:49.441 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:49.441 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 130, 2023-10-30T12:13:49.441 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:49.441 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:49.442 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:49.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:49.442 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.442 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:13:49.442 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:49.443 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:49.443 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:49.443 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:49.443 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:49.443 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:49.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:49.444 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:49.444 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:49.444 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:49.444 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:49.446 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:49.446 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:49.446 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:49.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:49.446 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:49.447 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:49.447 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:49.447 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:49.447 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:49.447 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:49.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.448 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:49.448 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:49.448 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:49.448 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:49.449 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:49.449 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:49.449 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:49.449 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:49.449 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:49.449 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:49.450 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:49.450 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:49.450 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:49.450 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:49.450 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:49.451 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:49.451 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:49.451 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:49.451 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:49.451 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:49.451 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:49.452 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:49.452 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:49.452 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:49.452 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:49.452 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:49.453 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:49.453 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.453 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:49.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.454 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:49.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:49.454 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:49.454 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:49.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:49.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:49.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:49.455 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:49.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:49.456 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.457 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:49.457 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:49.457 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:49.458 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:49.458 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:49.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:49.459 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:49.459 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:49.459 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:49.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.459 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:49.460 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:49.460 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:49.460 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:49.461 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.461 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.461 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:49.462 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:49.462 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:49.462 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:49.463 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:49.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:49.464 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:49.464 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:49.465 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:49.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:49.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:49.466 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:49.468 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:49.468 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:49.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.469 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.469 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:49.470 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:49.470 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:49.471 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:49.471 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:49.471 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.472 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:49.473 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:49.473 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:49.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.473 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:49.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:49.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:49.474 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:49.474 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:49.474 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:49.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:49.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:49.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:49.475 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:49.475 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:49.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 10 librados: watch_flush enter 2023-10-30T12:13:49.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 10 librados: watch_flush exit 2023-10-30T12:13:49.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 20 client.174249.objecter shutdown clearing up homeless session... 2023-10-30T12:13:49.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 10 client.174249.objecter successfully canceled tick 2023-10-30T12:13:49.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3f40392f0 msgr2=0x7fa3f403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3f40392f0 0x7fa3f403b7b0 secure :-1 s=READY pgs=4164 cs=0 l=1 rev1=1 crypto rx=0x7fa40800d770 tx=0x7fa4080076f0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 10 monclient: shutdown 2023-10-30T12:13:49.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:49.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 msgr2=0x7fa4101ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 --2- 172.21.15.50:0/2322348822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4100a2ae0 0x7fa4101ca5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa40c000c00 tx=0x7fa40c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa415c01640 1 -- 172.21.15.50:0/2322348822 reap_dead start 2023-10-30T12:13:49.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa415c01640 1 -- 172.21.15.50:0/2322348822 reap_dead start 2023-10-30T12:13:49.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 shutdown_connections 2023-10-30T12:13:49.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 >> 172.21.15.50:0/2322348822 conn(0x7fa4100a80b0 msgr2=0x7fa4100a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 shutdown_connections 2023-10-30T12:13:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 -- 172.21.15.50:0/2322348822 wait complete. 2023-10-30T12:13:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.431+0000 7fa417664640 1 librados: shutdown 2023-10-30T12:13:49.480 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:49.481 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:13:49.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:13:49.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.584+0000 7fa9e0be7640 10 monclient: get_monmap_and_config 2023-10-30T12:13:49.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.584+0000 7fa9e0be7640 10 monclient: build_initial_monmap 2023-10-30T12:13:49.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.584+0000 7fa9e0be7640 10 monclient: monmap: 2023-10-30T12:13:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:49.585941+0000 2023-10-30T12:13:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:49.585941+0000 2023-10-30T12:13:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:49.590 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:49.590 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:49.590 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.585+0000 7fa9e0be7640 1 Processor -- start 2023-10-30T12:13:49.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.585+0000 7fa9e0be7640 1 -- start start 2023-10-30T12:13:49.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.585+0000 7fa9e0be7640 10 monclient: init 2023-10-30T12:13:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-10-30T12:13:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa9dc0a0080 0x7fa9dc0a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient(hunting): picked mon.noname-c con 0x7fa9dc0a0080 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9dc0720b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9dc073a30 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc09ccc0 con 0x7fa9dc0720b0 2023-10-30T12:13:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc071e90 con 0x7fa9dc073a30 2023-10-30T12:13:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc073620 con 0x7fa9dc0a0080 2023-10-30T12:13:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:49.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9e0be7640 10 monclient(hunting): authenticate will time out at 1674.769106s 2023-10-30T12:13:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d9d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa9dc0720b0 0x7fa9dc097320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51290/0 (socket says 172.21.15.50:51290) 2023-10-30T12:13:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d9d9b640 1 -- 172.21.15.50:0/1219985317 learned_addr learned my addr 172.21.15.50:0/1219985317 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767673300 0 0) 0x7fa9dc09ccc0 con 0x7fa9dc0720b0 2023-10-30T12:13:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): _negotiate 2023-10-30T12:13:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): my global_id is 174252 2023-10-30T12:13:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 1 --2- 172.21.15.50:0/1219985317 >> v2:172.21.15.118:3300/0 conn(0x7fa9dc0a0080 0x7fa9dc0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c8003520 con 0x7fa9dc0720b0 2023-10-30T12:13:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): get_auth_request con 0x7fa9dc0a0080 auth_method 0 2023-10-30T12:13:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368192212 0 0) 0x7fa9c8003520 con 0x7fa9dc0720b0 2023-10-30T12:13:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 >> v2:172.21.15.118:3300/0 conn(0x7fa9dc0a0080 msgr2=0x7fa9dc0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 --2- 172.21.15.50:0/1219985317 >> v2:172.21.15.118:3300/0 conn(0x7fa9dc0a0080 0x7fa9dc0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 >> v1:172.21.15.118:6789/0 conn(0x7fa9dc073a30 legacy=0x7fa9dc097a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: _finish_hunting 0 2023-10-30T12:13:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:13:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:13:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc098140 con 0x7fa9dc0720b0 2023-10-30T12:13:49.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: _finish_auth 0 2023-10-30T12:13:49.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: _check_auth_tickets 2023-10-30T12:13:49.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:49.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fa9c4001670 con 0x7fa9dc0720b0 2023-10-30T12:13:49.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:13:49.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: dump: 2023-10-30T12:13:49.615 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.615 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.615 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.615 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.617 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.617 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.618 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.618 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.618 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.618 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:13:49.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:49.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 >> v1:172.21.15.50:6789/0 conn(0x7fa9dc0720b0 legacy=0x7fa9dc097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:13:49.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9c8003970 0x7fa9c8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): picked mon.b con 0x7fa9c8003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:49.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): picked mon.a con 0x7fa9c80047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:49.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9c80052a0 0x7fa9c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): picked mon.c con 0x7fa9c80052a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:49.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc073620 con 0x7fa9c80047c0 2023-10-30T12:13:49.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c800c170 con 0x7fa9c8003970 2023-10-30T12:13:49.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c800c330 con 0x7fa9c80052a0 2023-10-30T12:13:49.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:49.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d959a640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:13:49.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d9d9b640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d9d9b640 10 monclient(hunting): get_auth_request con 0x7fa9c80047c0 auth_method 0 2023-10-30T12:13:49.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d9d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9d9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9c8003970 0x7fa9c8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): get_auth_request con 0x7fa9c8003970 auth_method 0 2023-10-30T12:13:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:49.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.586+0000 7fa9dad9d640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9c80052a0 0x7fa9c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 10 monclient(hunting): get_auth_request con 0x7fa9c80052a0 auth_method 0 2023-10-30T12:13:49.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:13:49.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:13:49.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:13:49.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:13:49.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient(hunting): handle_auth_done global_id 174252 payload 293 2023-10-30T12:13:49.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 1 -- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9c80052a0 msgr2=0x7fa9c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:49.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9c80052a0 0x7fa9c800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 1 -- 172.21.15.50:0/1219985317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9c8003970 msgr2=0x7fa9c8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:49.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9c8003970 0x7fa9c8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:13:49.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 1 monclient: found mon.a 2023-10-30T12:13:49.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:49.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 1 -- 172.21.15.50:0/1219985317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8003520 con 0x7fa9c80047c0 2023-10-30T12:13:49.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient: _finish_auth 0 2023-10-30T12:13:49.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:13:49.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:49.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9e0be7640 5 monclient: authenticate success, global_id 174252 2023-10-30T12:13:49.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9dad9d640 1 -- 172.21.15.50:0/1219985317 reap_dead start 2023-10-30T12:13:49.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9e0be7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:49.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.587+0000 7fa9da59c640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9c8003970 0x7fa9c8003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:49.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d9d9b640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9c8004c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400c160 tx=0x7fa9c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d1b5d7a6ae2fb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4005ad0 con 0x7fa9c80047c0 2023-10-30T12:13:49.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:49.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 10 monclient: dump: 2023-10-30T12:13:49.635 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.635 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.635 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.635 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.636 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.636 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.636 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.636 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.636 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.636 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:49.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c4005c50 con 0x7fa9c80047c0 2023-10-30T12:13:49.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:49.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 1 -- 172.21.15.50:0/1219985317 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c402c850 con 0x7fa9c80047c0 2023-10-30T12:13:49.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:49.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9d959a640 10 monclient: dump: 2023-10-30T12:13:49.638 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.638 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.638 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.639 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.639 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.639 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.639 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.639 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.640 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.640 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:49.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 4 set_mon_vals no callback set 2023-10-30T12:13:49.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 10 monclient: shutdown 2023-10-30T12:13:49.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:49.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 1 -- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 msgr2=0x7fa9c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9c8004c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400c160 tx=0x7fa9c80057f0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 1 -- 172.21.15.50:0/1219985317 shutdown_connections 2023-10-30T12:13:49.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/1219985317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9c8004c40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 1 -- 172.21.15.50:0/1219985317 >> 172.21.15.50:0/1219985317 conn(0x7fa9dc0a4870 msgr2=0x7fa9dc0a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:49.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 1 -- 172.21.15.50:0/1219985317 shutdown_connections 2023-10-30T12:13:49.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.588+0000 7fa9e0be7640 1 -- 172.21.15.50:0/1219985317 wait complete. 2023-10-30T12:13:49.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.589+0000 7fa9e0be7640 10 monclient: build_initial_monmap 2023-10-30T12:13:49.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.589+0000 7fa9e0be7640 10 monclient: monmap: 2023-10-30T12:13:49.643 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:49.643 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:49.643 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:49.644 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:49.644 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:49.644 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.644 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:49.644 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:49.645 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:49.645 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.589+0000 7fa9e0be7640 1 librados: starting msgr at 2023-10-30T12:13:49.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.589+0000 7fa9e0be7640 1 librados: starting objecter 2023-10-30T12:13:49.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.589+0000 7fa9e0be7640 1 Processor -- start 2023-10-30T12:13:49.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 -- start start 2023-10-30T12:13:49.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 librados: setting wanted keys 2023-10-30T12:13:49.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 librados: calling monclient init 2023-10-30T12:13:49.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient: init 2023-10-30T12:13:49.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:49.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:13:49.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9dc1ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): picked mon.noname-a con 0x7fa9c80047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:49.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9dc1ca570 0x7fa9dc0964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): picked mon.noname-c con 0x7fa9dc1ca570 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:49.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 0x7fa9dc08df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): picked mon.noname-b con 0x7fa9dc096a20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:49.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc08f470 con 0x7fa9c80047c0 2023-10-30T12:13:49.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc073e90 con 0x7fa9dc096a20 2023-10-30T12:13:49.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:49.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0916e0 con 0x7fa9dc1ca570 2023-10-30T12:13:49.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:49.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9e0be7640 10 monclient(hunting): authenticate will time out at 1674.773503s 2023-10-30T12:13:49.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9dc1ca570 0x7fa9dc0964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9dc1ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9dc1ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50622/0 (socket says 172.21.15.50:50622) 2023-10-30T12:13:49.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9dc1ca570 0x7fa9dc0964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33622/0 (socket says 172.21.15.50:33622) 2023-10-30T12:13:49.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 -- 172.21.15.50:0/4276419086 learned_addr learned my addr 172.21.15.50:0/4276419086 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:49.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): get_auth_request con 0x7fa9c80047c0 auth_method 0 2023-10-30T12:13:49.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 10 monclient(hunting): get_auth_request con 0x7fa9dc1ca570 auth_method 0 2023-10-30T12:13:49.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:49.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:49.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:49.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 0x7fa9dc08df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 10 monclient(hunting): get_auth_request con 0x7fa9dc096a20 auth_method 0 2023-10-30T12:13:49.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:49.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:49.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:49.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 1 -- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 msgr2=0x7fa9dc08df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:13:49.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 1 -- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 msgr2=0x7fa9dc08df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:13:49.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 0x7fa9dc08df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:13:49.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9dad9d640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 0x7fa9dc08df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:49.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient(hunting): handle_auth_done global_id 174255 payload 824 2023-10-30T12:13:49.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 -- 172.21.15.50:0/4276419086 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9dc1ca570 msgr2=0x7fa9dc0964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9dc1ca570 0x7fa9dc0964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 -- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 msgr2=0x7fa9dc08df10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:49.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 0x7fa9dc08df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient: _finish_hunting 0 2023-10-30T12:13:49.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 monclient: found mon.noname-a 2023-10-30T12:13:49.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:49.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.590+0000 7fa9da59c640 1 -- 172.21.15.50:0/4276419086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0919c0 con 0x7fa9c80047c0 2023-10-30T12:13:49.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9da59c640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9dc1ca030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc011bd0 tx=0x7fa9cc00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 client.?.objecter ms_handle_connect 0x7fa9c80047c0 2023-10-30T12:13:49.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:49.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 1 -- 172.21.15.50:0/4276419086 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc011cc0 con 0x7fa9c80047c0 2023-10-30T12:13:49.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:49.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: dump: 2023-10-30T12:13:49.660 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.660 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.660 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.660 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.661 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.661 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.661 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.662 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.662 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.662 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:49.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:49.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:49.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 5 monclient: authenticate success, global_id 174255 2023-10-30T12:13:49.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 1 -- 172.21.15.50:0/4276419086 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc011e20 con 0x7fa9c80047c0 2023-10-30T12:13:49.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 10 monclient: _renew_subs 2023-10-30T12:13:49.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc091ca0 con 0x7fa9c80047c0 2023-10-30T12:13:49.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:49.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 1 -- 172.21.15.50:0/4276419086 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc004240 con 0x7fa9c80047c0 2023-10-30T12:13:49.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:49.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:49.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d2ffd640 10 monclient: dump: 2023-10-30T12:13:49.666 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:49.666 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:49.666 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.667 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:49.667 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:49.667 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:49.667 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:49.667 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:49.667 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:49.668 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:49.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9d8d99640 4 set_mon_vals no callback set 2023-10-30T12:13:49.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 10 client.174255.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:49.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 10 monclient: _renew_subs 2023-10-30T12:13:49.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc08fbb0 con 0x7fa9c80047c0 2023-10-30T12:13:49.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.591+0000 7fa9e0be7640 1 librados: init done 2023-10-30T12:13:49.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 1 -- 172.21.15.50:0/4276419086 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa9cc0043a0 con 0x7fa9c80047c0 2023-10-30T12:13:49.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9bc0392f0 0x7fa9bc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:49.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 1 -- 172.21.15.50:0/4276419086 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fa9cc04b1d0 con 0x7fa9c80047c0 2023-10-30T12:13:49.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 10 client.174255.objecter ms_dispatch 0x7fa9dc000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:49.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 3 client.174255.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:49.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 3 client.174255.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:49.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9e0be7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:49.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9e0be7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:49.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9e0be7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc04e160 con 0x7fa9c80047c0 2023-10-30T12:13:49.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 20 client.174255.objecter dump_active .. 0 homeless 2023-10-30T12:13:49.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d9d9b640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9bc0392f0 0x7fa9bc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:49.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d9d9b640 10 monclient: get_auth_request con 0x7fa9bc0392f0 auth_method 0 2023-10-30T12:13:49.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d2ffd640 10 client.174255.objecter ms_handle_connect 0x7fa9bc0392f0 2023-10-30T12:13:49.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.592+0000 7fa9d9d9b640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9bc0392f0 0x7fa9bc03b7b0 secure :-1 s=READY pgs=4165 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4034000 tx=0x7fa9c4007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:49.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.596+0000 7fa9d2ffd640 1 -- 172.21.15.50:0/4276419086 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa9cc022070 con 0x7fa9c80047c0 2023-10-30T12:13:49.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.597+0000 7fa9d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:49.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.597+0000 7fa9d2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:49.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.840+0000 7fa9e0be7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:13:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.840+0000 7fa9e0be7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:13:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.840+0000 7fa9e0be7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.840+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa9dc06ab70 con 0x7fa9c80047c0 2023-10-30T12:13:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.841+0000 7fa9d2ffd640 1 -- 172.21.15.50:0/4276419086 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fa9cc009680 con 0x7fa9c80047c0 2023-10-30T12:13:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.841+0000 7fa9d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:13:49.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.841+0000 7fa9d2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:49.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=971546350,config=3140514009,health=2810010628,kv=1634934840,logm=1740275046}) 2023-10-30T12:13:49.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=10,kv=9,logm=52} crc {auth=3991462514,config=3140514009,health=661029745,kv=3201645878,logm=1564650548}) 2023-10-30T12:13:49.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=971546350,config=3140514009,health=2810010628,kv=1634934840,logm=1740275046}) 2023-10-30T12:13:49.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.841+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=9,logm=51} crc {auth=4210494868,config=3140514009,health=881818072,kv=4169269627,logm=2939548193}) 2023-10-30T12:13:49.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598326030}) 2023-10-30T12:13:49.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2748640747}) 2023-10-30T12:13:49.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.842+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598326030}) 2023-10-30T12:13:49.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.843+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1280558266}) 2023-10-30T12:13:49.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 10 librados: watch_flush enter 2023-10-30T12:13:49.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 10 librados: watch_flush exit 2023-10-30T12:13:49.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 20 client.174255.objecter shutdown clearing up homeless session... 2023-10-30T12:13:49.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 10 client.174255.objecter successfully canceled tick 2023-10-30T12:13:49.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223311683}) 2023-10-30T12:13:49.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470378792}) 2023-10-30T12:13:49.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9bc0392f0 msgr2=0x7fa9bc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9bc0392f0 0x7fa9bc03b7b0 secure :-1 s=READY pgs=4165 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4034000 tx=0x7fa9c4007000 comp rx=0 tx=0).stop 2023-10-30T12:13:49.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 10 monclient: shutdown 2023-10-30T12:13:49.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:49.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 msgr2=0x7fa9dc1ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:49.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9dc1ca030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc011bd0 tx=0x7fa9cc00f610 comp rx=0 tx=0).stop 2023-10-30T12:13:49.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223311683}) 2023-10-30T12:13:49.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.844+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3612017356}) 2023-10-30T12:13:49.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 shutdown_connections 2023-10-30T12:13:49.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9bc0392f0 0x7fa9bc03b7b0 unknown :-1 s=CLOSED pgs=4165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9dc096a20 0x7fa9dc08df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9dc1ca570 0x7fa9dc0964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 --2- 172.21.15.50:0/4276419086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c80047c0 0x7fa9dc1ca030 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:49.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 >> 172.21.15.50:0/4276419086 conn(0x7fa9dc0a4870 msgr2=0x7fa9dc09ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:49.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 shutdown_connections 2023-10-30T12:13:49.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 -- 172.21.15.50:0/4276419086 wait complete. 2023-10-30T12:13:49.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:49.844+0000 7fa9e0be7640 1 librados: shutdown 2023-10-30T12:13:49.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404307981}) 2023-10-30T12:13:49.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775818538}) 2023-10-30T12:13:49.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404307981}) 2023-10-30T12:13:49.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.845+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984592261}) 2023-10-30T12:13:49.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268439338}) 2023-10-30T12:13:49.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=138517484}) 2023-10-30T12:13:49.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268439338}) 2023-10-30T12:13:49.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.847+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988400842}) 2023-10-30T12:13:49.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770996441}) 2023-10-30T12:13:49.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596296284}) 2023-10-30T12:13:49.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770996441}) 2023-10-30T12:13:49.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.848+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448076406}) 2023-10-30T12:13:49.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687349613}) 2023-10-30T12:13:49.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468748161}) 2023-10-30T12:13:49.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687349613}) 2023-10-30T12:13:49.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.849+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=838529984}) 2023-10-30T12:13:49.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3021630562}) 2023-10-30T12:13:49.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2757537785}) 2023-10-30T12:13:49.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3021630562}) 2023-10-30T12:13:49.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.850+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=856582177}) 2023-10-30T12:13:49.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=17051989}) 2023-10-30T12:13:49.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3939914359}) 2023-10-30T12:13:49.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=17051989}) 2023-10-30T12:13:49.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.851+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1570988028}) 2023-10-30T12:13:49.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963139068}) 2023-10-30T12:13:49.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2710993217}) 2023-10-30T12:13:49.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963139068}) 2023-10-30T12:13:49.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.852+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708153516}) 2023-10-30T12:13:49.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397874982}) 2023-10-30T12:13:49.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2642426714}) 2023-10-30T12:13:49.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397874982}) 2023-10-30T12:13:49.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.853+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=541934039}) 2023-10-30T12:13:49.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909778608}) 2023-10-30T12:13:49.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195825503}) 2023-10-30T12:13:49.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909778608}) 2023-10-30T12:13:49.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.854+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562733620}) 2023-10-30T12:13:49.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288429917}) 2023-10-30T12:13:49.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648733557}) 2023-10-30T12:13:49.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288429917}) 2023-10-30T12:13:49.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.855+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785260886}) 2023-10-30T12:13:49.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603426741}) 2023-10-30T12:13:49.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2392311922}) 2023-10-30T12:13:49.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603426741}) 2023-10-30T12:13:49.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.856+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967887774}) 2023-10-30T12:13:49.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274319778}) 2023-10-30T12:13:49.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010001074}) 2023-10-30T12:13:49.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274319778}) 2023-10-30T12:13:49.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.857+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2673977629}) 2023-10-30T12:13:49.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323651082}) 2023-10-30T12:13:49.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227831339}) 2023-10-30T12:13:49.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323651082}) 2023-10-30T12:13:49.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.858+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019119648}) 2023-10-30T12:13:49.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189522567}) 2023-10-30T12:13:49.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2024510707}) 2023-10-30T12:13:49.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189522567}) 2023-10-30T12:13:49.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.859+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2033048399}) 2023-10-30T12:13:49.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:13:49.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385164371}) 2023-10-30T12:13:49.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3083665205}) 2023-10-30T12:13:49.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385164371}) 2023-10-30T12:13:49.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173868140}) 2023-10-30T12:13:49.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3108308292}) 2023-10-30T12:13:49.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=449099834}) 2023-10-30T12:13:49.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.860+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3108308292}) 2023-10-30T12:13:49.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837547004}) 2023-10-30T12:13:49.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048303731}) 2023-10-30T12:13:49.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1636340145}) 2023-10-30T12:13:49.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048303731}) 2023-10-30T12:13:49.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.861+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537514730}) 2023-10-30T12:13:49.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512882165}) 2023-10-30T12:13:49.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=316271703}) 2023-10-30T12:13:49.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512882165}) 2023-10-30T12:13:49.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684393890}) 2023-10-30T12:13:49.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303585541}) 2023-10-30T12:13:49.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113369305}) 2023-10-30T12:13:49.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.862+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303585541}) 2023-10-30T12:13:49.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.863+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2330299274}) 2023-10-30T12:13:49.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211764558}) 2023-10-30T12:13:49.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645881596}) 2023-10-30T12:13:49.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211764558}) 2023-10-30T12:13:49.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.864+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273487994}) 2023-10-30T12:13:49.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951590005}) 2023-10-30T12:13:49.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4230456335}) 2023-10-30T12:13:49.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951590005}) 2023-10-30T12:13:49.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=8682859}) 2023-10-30T12:13:49.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941623135}) 2023-10-30T12:13:49.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000119994}) 2023-10-30T12:13:49.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941623135}) 2023-10-30T12:13:49.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.865+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=531577593}) 2023-10-30T12:13:49.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756827353}) 2023-10-30T12:13:49.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355516539}) 2023-10-30T12:13:49.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756827353}) 2023-10-30T12:13:49.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.866+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2888083750}) 2023-10-30T12:13:49.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627807697}) 2023-10-30T12:13:49.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1141895932}) 2023-10-30T12:13:49.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627807697}) 2023-10-30T12:13:49.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.867+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130640117}) 2023-10-30T12:13:49.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481590433}) 2023-10-30T12:13:49.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640055098}) 2023-10-30T12:13:49.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481590433}) 2023-10-30T12:13:49.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.868+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753428646}) 2023-10-30T12:13:49.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661012707}) 2023-10-30T12:13:49.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=511809493}) 2023-10-30T12:13:49.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661012707}) 2023-10-30T12:13:49.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3275067924}) 2023-10-30T12:13:49.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393301628}) 2023-10-30T12:13:49.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=583620365}) 2023-10-30T12:13:49.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393301628}) 2023-10-30T12:13:49.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.869+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728821977}) 2023-10-30T12:13:49.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427357394}) 2023-10-30T12:13:49.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481939559}) 2023-10-30T12:13:49.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427357394}) 2023-10-30T12:13:49.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.870+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1303651379}) 2023-10-30T12:13:49.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747687553}) 2023-10-30T12:13:49.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3047801330}) 2023-10-30T12:13:49.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747687553}) 2023-10-30T12:13:49.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.871+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494110472}) 2023-10-30T12:13:49.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602133295}) 2023-10-30T12:13:49.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2012096757}) 2023-10-30T12:13:49.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602133295}) 2023-10-30T12:13:49.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.872+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250564060}) 2023-10-30T12:13:49.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693012523}) 2023-10-30T12:13:49.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4030905266}) 2023-10-30T12:13:49.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693012523}) 2023-10-30T12:13:49.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.873+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3130664076}) 2023-10-30T12:13:49.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317420428}) 2023-10-30T12:13:49.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2234606003}) 2023-10-30T12:13:49.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317420428}) 2023-10-30T12:13:49.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248298109}) 2023-10-30T12:13:49.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227131144}) 2023-10-30T12:13:49.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869909147}) 2023-10-30T12:13:49.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227131144}) 2023-10-30T12:13:49.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.874+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612253637}) 2023-10-30T12:13:49.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846620168}) 2023-10-30T12:13:49.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678306257}) 2023-10-30T12:13:49.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846620168}) 2023-10-30T12:13:49.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.875+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078637572}) 2023-10-30T12:13:49.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152930574}) 2023-10-30T12:13:49.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795149878}) 2023-10-30T12:13:49.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152930574}) 2023-10-30T12:13:49.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.876+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2699043970}) 2023-10-30T12:13:49.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704177215}) 2023-10-30T12:13:49.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3444974664}) 2023-10-30T12:13:49.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704177215}) 2023-10-30T12:13:49.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.877+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365563753}) 2023-10-30T12:13:49.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641564403}) 2023-10-30T12:13:49.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3650750584}) 2023-10-30T12:13:49.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641564403}) 2023-10-30T12:13:49.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.878+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385128067}) 2023-10-30T12:13:49.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160133890}) 2023-10-30T12:13:49.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084574228}) 2023-10-30T12:13:49.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160133890}) 2023-10-30T12:13:49.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3506753819}) 2023-10-30T12:13:49.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468933706}) 2023-10-30T12:13:49.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385326518}) 2023-10-30T12:13:49.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468933706}) 2023-10-30T12:13:49.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.879+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827740217}) 2023-10-30T12:13:49.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153712393}) 2023-10-30T12:13:49.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=869826523}) 2023-10-30T12:13:49.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153712393}) 2023-10-30T12:13:49.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.880+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258472856}) 2023-10-30T12:13:49.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325258532}) 2023-10-30T12:13:49.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4223457673}) 2023-10-30T12:13:49.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325258532}) 2023-10-30T12:13:49.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.881+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=836996077}) 2023-10-30T12:13:49.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523592863}) 2023-10-30T12:13:49.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3534493333}) 2023-10-30T12:13:49.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523592863}) 2023-10-30T12:13:49.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.882+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4135990812}) 2023-10-30T12:13:49.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494985997}) 2023-10-30T12:13:49.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069048825}) 2023-10-30T12:13:49.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494985997}) 2023-10-30T12:13:49.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.883+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3105541381}) 2023-10-30T12:13:49.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723881499}) 2023-10-30T12:13:49.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3071408503}) 2023-10-30T12:13:49.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723881499}) 2023-10-30T12:13:49.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411212455}) 2023-10-30T12:13:49.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.884+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285642085}) 2023-10-30T12:13:49.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1811526461}) 2023-10-30T12:13:49.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285642085}) 2023-10-30T12:13:49.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282958042}) 2023-10-30T12:13:49.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086312153}) 2023-10-30T12:13:49.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4215423672}) 2023-10-30T12:13:49.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086312153}) 2023-10-30T12:13:49.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.885+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164895773}) 2023-10-30T12:13:49.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287183981}) 2023-10-30T12:13:49.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=559146423}) 2023-10-30T12:13:49.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287183981}) 2023-10-30T12:13:49.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.886+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068142259}) 2023-10-30T12:13:49.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135145358}) 2023-10-30T12:13:49.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413017866}) 2023-10-30T12:13:49.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135145358}) 2023-10-30T12:13:49.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.887+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2028918987}) 2023-10-30T12:13:49.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290001573}) 2023-10-30T12:13:49.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161455760}) 2023-10-30T12:13:49.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290001573}) 2023-10-30T12:13:49.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.888+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2023627546}) 2023-10-30T12:13:49.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242260482}) 2023-10-30T12:13:49.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1474519449}) 2023-10-30T12:13:49.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242260482}) 2023-10-30T12:13:49.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.889+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2531518976}) 2023-10-30T12:13:49.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=755883001}) 2023-10-30T12:13:49.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=163255836}) 2023-10-30T12:13:49.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=755883001}) 2023-10-30T12:13:49.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.890+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746153729}) 2023-10-30T12:13:49.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266595449}) 2023-10-30T12:13:49.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2281544397}) 2023-10-30T12:13:49.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266595449}) 2023-10-30T12:13:49.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.891+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384948903}) 2023-10-30T12:13:49.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248239073}) 2023-10-30T12:13:49.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3668121149}) 2023-10-30T12:13:49.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248239073}) 2023-10-30T12:13:49.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2699202645}) 2023-10-30T12:13:49.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.892+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955349801}) 2023-10-30T12:13:49.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=595645023}) 2023-10-30T12:13:49.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955349801}) 2023-10-30T12:13:49.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248099523}) 2023-10-30T12:13:49.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179494759}) 2023-10-30T12:13:49.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3681334890}) 2023-10-30T12:13:49.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179494759}) 2023-10-30T12:13:49.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.893+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=803693840}) 2023-10-30T12:13:49.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265938119}) 2023-10-30T12:13:49.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=177012771}) 2023-10-30T12:13:49.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265938119}) 2023-10-30T12:13:49.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.894+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168237321}) 2023-10-30T12:13:49.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.895+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.895+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833622312}) 2023-10-30T12:13:49.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.895+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1840165411}) 2023-10-30T12:13:49.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.895+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833622312}) 2023-10-30T12:13:49.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553718849}) 2023-10-30T12:13:49.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.896+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755863518}) 2023-10-30T12:13:49.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=998151911}) 2023-10-30T12:13:49.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755863518}) 2023-10-30T12:13:49.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2463567731}) 2023-10-30T12:13:49.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=519979648}) 2023-10-30T12:13:49.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1108026274}) 2023-10-30T12:13:49.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=519979648}) 2023-10-30T12:13:49.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.897+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110617263}) 2023-10-30T12:13:49.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.898+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.898+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215663053}) 2023-10-30T12:13:49.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.898+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47807986}) 2023-10-30T12:13:49.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.898+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.898+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215663053}) 2023-10-30T12:13:49.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.898+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043358336}) 2023-10-30T12:13:49.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.899+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.899+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225737884}) 2023-10-30T12:13:49.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.899+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956129268}) 2023-10-30T12:13:49.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.899+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.899+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225737884}) 2023-10-30T12:13:49.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.899+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568816346}) 2023-10-30T12:13:49.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.900+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.900+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9116377}) 2023-10-30T12:13:49.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.900+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1739047818}) 2023-10-30T12:13:49.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.900+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.900+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9116377}) 2023-10-30T12:13:49.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.900+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3629698765}) 2023-10-30T12:13:49.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119296684}) 2023-10-30T12:13:49.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3403288511}) 2023-10-30T12:13:49.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119296684}) 2023-10-30T12:13:49.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.901+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986142055}) 2023-10-30T12:13:49.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.902+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.902+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3757738274}) 2023-10-30T12:13:49.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.902+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2936133254}) 2023-10-30T12:13:49.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.902+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.902+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3757738274}) 2023-10-30T12:13:49.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.902+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156214567}) 2023-10-30T12:13:49.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.903+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.903+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81141523}) 2023-10-30T12:13:49.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.903+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304039936}) 2023-10-30T12:13:49.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.903+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.903+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81141523}) 2023-10-30T12:13:49.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.903+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2677422902}) 2023-10-30T12:13:49.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.904+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.904+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260199800}) 2023-10-30T12:13:49.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.904+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=723249691}) 2023-10-30T12:13:49.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.904+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.904+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260199800}) 2023-10-30T12:13:49.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.904+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637465382}) 2023-10-30T12:13:49.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.905+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.905+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=4} crc {logm=949883050,mdsmap=2441318933,mgr=1142129444,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=954340973,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1945788887,osd_snap=1566761633,osdmap=363744095}) 2023-10-30T12:13:49.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.905+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=3} crc {logm=3254797117,mdsmap=3984425268,mgr=465433054,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=483560336,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1945788887,osd_snap=3747184104,osdmap=4129985701}) 2023-10-30T12:13:49.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.905+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.905+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=4} crc {logm=949883050,mdsmap=2441318933,mgr=1142129444,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=954340973,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1945788887,osd_snap=1566761633,osdmap=363744095}) 2023-10-30T12:13:49.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.905+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=33,osdmap=6} crc {logm=1884666421,mdsmap=3984425268,mgr=370439686,mgr_metadata=2083152833,mgrstat=1453808192,mon_config_key=3217606506,monmap=4204254258,osd_metadata=451695420,osd_pg_creating=1945788887,osd_snap=910308283,osdmap=664816170}) 2023-10-30T12:13:49.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.906+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.906+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3891351026}) 2023-10-30T12:13:49.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.906+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3786416325}) 2023-10-30T12:13:49.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.906+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.906+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3891351026}) 2023-10-30T12:13:49.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.906+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1898229390}) 2023-10-30T12:13:49.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.907+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.907+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2850622766}) 2023-10-30T12:13:49.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.907+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2328949427}) 2023-10-30T12:13:49.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.907+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.907+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2850622766}) 2023-10-30T12:13:49.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.907+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3530469149}) 2023-10-30T12:13:49.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.908+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.908+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=491545190}) 2023-10-30T12:13:49.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.908+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=692535857}) 2023-10-30T12:13:49.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.908+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.908+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=491545190}) 2023-10-30T12:13:49.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.908+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=670516801}) 2023-10-30T12:13:49.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.909+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.909+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2180876497}) 2023-10-30T12:13:49.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.909+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4118249738}) 2023-10-30T12:13:49.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.909+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.909+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2180876497}) 2023-10-30T12:13:49.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.909+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1491262844}) 2023-10-30T12:13:49.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.911+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.911+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2132657391}) 2023-10-30T12:13:49.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.911+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3755721182}) 2023-10-30T12:13:49.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.911+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.911+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2132657391}) 2023-10-30T12:13:49.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.911+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2640079969}) 2023-10-30T12:13:49.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.913+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.913+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1382352382}) 2023-10-30T12:13:49.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.913+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1460062262}) 2023-10-30T12:13:49.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.913+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.913+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1382352382}) 2023-10-30T12:13:49.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.914+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2649391719}) 2023-10-30T12:13:49.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.919+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.919+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=4} crc {osdmap=144109798}) 2023-10-30T12:13:49.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.919+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=7} crc {osdmap=2322716465}) 2023-10-30T12:13:49.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.919+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:13:49.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.919+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=4} crc {osdmap=144109798}) 2023-10-30T12:13:49.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:13:49.920+0000 7f526dac4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=6} crc {osdmap=173094258}) 2023-10-30T12:13:50.879 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:13:50.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:50.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.994+0000 7f2cd69cc640 10 monclient: get_monmap_and_config 2023-10-30T12:13:50.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.994+0000 7f2cd69cc640 10 monclient: build_initial_monmap 2023-10-30T12:13:50.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.994+0000 7f2cd69cc640 10 monclient: monmap: 2023-10-30T12:13:50.997 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:50.997 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:50.997 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:50.996025+0000 2023-10-30T12:13:50.997 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:50.996025+0000 2023-10-30T12:13:50.997 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:50.998 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:50.998 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:50.998 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:50.998 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:50.998 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:50.999 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:50.999 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.995+0000 7f2cd69cc640 1 Processor -- start 2023-10-30T12:13:51.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.995+0000 7f2cd69cc640 1 -- start start 2023-10-30T12:13:51.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.995+0000 7f2cd69cc640 10 monclient: init 2023-10-30T12:13:51.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T12:13:51.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2cd009f5b0 0x7f2cd0099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd009f5b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:51.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2cd0096080 0x7f2cd0096490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd0096080 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:51.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd00969d0 0x7f2cd009cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd00969d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:51.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd0098810 con 0x7f2cd0096080 2023-10-30T12:13:51.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd006aef0 con 0x7f2cd009f5b0 2023-10-30T12:13:51.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd009a1a0 con 0x7f2cd00969d0 2023-10-30T12:13:51.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd69cc640 10 monclient(hunting): authenticate will time out at 1676.179211s 2023-10-30T12:13:51.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2cd0096080 0x7f2cd0096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd00969d0 0x7f2cd009cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2cd0096080 0x7f2cd0096490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50626/0 (socket says 172.21.15.50:50626) 2023-10-30T12:13:51.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd00969d0 0x7f2cd009cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33644/0 (socket says 172.21.15.50:33644) 2023-10-30T12:13:51.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/344552971 learned_addr learned my addr 172.21.15.50:0/344552971 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:51.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): get_auth_request con 0x7f2cd0096080 auth_method 0 2023-10-30T12:13:51.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 10 monclient(hunting): get_auth_request con 0x7f2cd00969d0 auth_method 0 2023-10-30T12:13:51.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccffff640 1 --2- 172.21.15.50:0/344552971 >> v2:172.21.15.118:3300/0 conn(0x7f2cd009f5b0 0x7f2cd0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccffff640 10 monclient(hunting): get_auth_request con 0x7f2cd009f5b0 auth_method 0 2023-10-30T12:13:51.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_done global_id 174258 payload 274 2023-10-30T12:13:51.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd00969d0 msgr2=0x7f2cd009cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:51.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 --2- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd00969d0 0x7f2cd009cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/344552971 >> v2:172.21.15.118:3300/0 conn(0x7f2cd009f5b0 msgr2=0x7f2cd0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:51.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 --2- 172.21.15.50:0/344552971 >> v2:172.21.15.118:3300/0 conn(0x7f2cd009f5b0 0x7f2cd0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 monclient: found mon.noname-a 2023-10-30T12:13:51.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/344552971 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0096e60 con 0x7f2cd0096080 2023-10-30T12:13:51.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.996+0000 7f2cd4f69640 1 --2- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd00969d0 0x7f2cd009cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:51.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2ccf7fe640 1 --2- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd0096080 0x7f2cd0096490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009870 tx=0x7f2cc002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eee3579f63996b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 1 -- 172.21.15.50:0/344552971 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0031740 con 0x7f2cd0096080 2023-10-30T12:13:51.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:51.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: dump: 2023-10-30T12:13:51.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.017 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.017 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.017 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.018 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.018 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.018 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.018 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cd69cc640 5 monclient: authenticate success, global_id 174258 2023-10-30T12:13:51.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cd69cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:51.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 1 -- 172.21.15.50:0/344552971 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc00318a0 con 0x7f2cd0096080 2023-10-30T12:13:51.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cd69cc640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:51.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cd69cc640 4 set_mon_vals no callback set 2023-10-30T12:13:51.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 1 -- 172.21.15.50:0/344552971 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0031c30 con 0x7f2cd0096080 2023-10-30T12:13:51.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.997+0000 7f2cceffd640 10 monclient: dump: 2023-10-30T12:13:51.021 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.021 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.021 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.021 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.022 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.022 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.022 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.022 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.022 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.023 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 10 monclient: shutdown 2023-10-30T12:13:51.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 -- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd0096080 msgr2=0x7f2cd0096490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd0096080 0x7f2cd0096490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009870 tx=0x7f2cc002b7e0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 -- 172.21.15.50:0/344552971 shutdown_connections 2023-10-30T12:13:51.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd00969d0 0x7f2cd009cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/344552971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd0096080 0x7f2cd0096490 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/344552971 >> v2:172.21.15.118:3300/0 conn(0x7f2cd009f5b0 0x7f2cd0099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 -- 172.21.15.50:0/344552971 >> 172.21.15.50:0/344552971 conn(0x7f2cd00a05c0 msgr2=0x7f2cd009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 -- 172.21.15.50:0/344552971 shutdown_connections 2023-10-30T12:13:51.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 1 -- 172.21.15.50:0/344552971 wait complete. 2023-10-30T12:13:51.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 10 monclient: build_initial_monmap 2023-10-30T12:13:51.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.998+0000 7f2cd69cc640 10 monclient: monmap: 2023-10-30T12:13:51.025 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:51.025 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:51.025 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:51.025 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:51.026 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:51.026 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.026 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:51.026 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:51.026 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:51.027 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 librados: starting msgr at 2023-10-30T12:13:51.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 librados: starting objecter 2023-10-30T12:13:51.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 Processor -- start 2023-10-30T12:13:51.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 -- start start 2023-10-30T12:13:51.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 librados: setting wanted keys 2023-10-30T12:13:51.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 librados: calling monclient init 2023-10-30T12:13:51.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient: init 2023-10-30T12:13:51.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:13:51.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cd0096080 0x7f2cd006ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd0096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:51.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd00969d0 0x7f2cd00704a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd00969d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:51.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd009f5b0 0x7f2cd00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd009f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:51.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd009aba0 con 0x7f2cd00969d0 2023-10-30T12:13:51.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd009c6f0 con 0x7f2cd0096080 2023-10-30T12:13:51.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd0072010 con 0x7f2cd009f5b0 2023-10-30T12:13:51.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2cd69cc640 10 monclient(hunting): authenticate will time out at 1676.182915s 2023-10-30T12:13:51.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:50.999+0000 7f2ccf7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd00969d0 0x7f2cd00704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd009f5b0 0x7f2cd00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd00969d0 0x7f2cd00704a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50630/0 (socket says 172.21.15.50:50630) 2023-10-30T12:13:51.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/4005621883 learned_addr learned my addr 172.21.15.50:0/4005621883 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:51.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd009f5b0 0x7f2cd00709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33648/0 (socket says 172.21.15.50:33648) 2023-10-30T12:13:51.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): get_auth_request con 0x7f2cd00969d0 auth_method 0 2023-10-30T12:13:51.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 10 monclient(hunting): get_auth_request con 0x7f2cd009f5b0 auth_method 0 2023-10-30T12:13:51.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cd0096080 0x7f2cd006ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 10 monclient(hunting): get_auth_request con 0x7f2cd0096080 auth_method 0 2023-10-30T12:13:51.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient(hunting): handle_auth_done global_id 174261 payload 824 2023-10-30T12:13:51.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd009f5b0 msgr2=0x7f2cd00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd009f5b0 0x7f2cd00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cd0096080 msgr2=0x7f2cd006ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cd0096080 0x7f2cd006ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 monclient: found mon.noname-a 2023-10-30T12:13:51.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 -- 172.21.15.50:0/4005621883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc8009a70 con 0x7f2cd00969d0 2023-10-30T12:13:51.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd4f69640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd009f5b0 0x7f2cd00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:51.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2ccf7fe640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd00969d0 0x7f2cd00704a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2cc002b820 tx=0x7f2cc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 client.?.objecter ms_handle_connect 0x7f2cd00969d0 2023-10-30T12:13:51.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:51.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 1 -- 172.21.15.50:0/4005621883 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0033940 con 0x7f2cd00969d0 2023-10-30T12:13:51.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:51.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: dump: 2023-10-30T12:13:51.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.040 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.040 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.040 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.040 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.041 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.041 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 1 -- 172.21.15.50:0/4005621883 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc0033ac0 con 0x7f2cd00969d0 2023-10-30T12:13:51.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 5 monclient: authenticate success, global_id 174261 2023-10-30T12:13:51.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 10 monclient: _renew_subs 2023-10-30T12:13:51.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc00094d0 con 0x7f2cd00969d0 2023-10-30T12:13:51.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 1 -- 172.21.15.50:0/4005621883 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0033e70 con 0x7f2cd00969d0 2023-10-30T12:13:51.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cccff9640 10 monclient: dump: 2023-10-30T12:13:51.044 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.044 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.044 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.044 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.044 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.045 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.045 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.045 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.045 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.045 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cce7fc640 4 set_mon_vals no callback set 2023-10-30T12:13:51.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 10 client.174261.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:51.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 10 monclient: _renew_subs 2023-10-30T12:13:51.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd009c410 con 0x7f2cd00969d0 2023-10-30T12:13:51.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.000+0000 7f2cd69cc640 1 librados: init done 2023-10-30T12:13:51.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cccff9640 1 -- 172.21.15.50:0/4005621883 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2cc003a040 con 0x7f2cd00969d0 2023-10-30T12:13:51.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cd69cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cd69cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cd69cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cccff9640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2cbc0392f0 0x7f2cbc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd0096660 con 0x7f2cd00969d0 2023-10-30T12:13:51.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cccff9640 1 -- 172.21.15.50:0/4005621883 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f2cc006c920 con 0x7f2cd00969d0 2023-10-30T12:13:51.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cccff9640 10 client.174261.objecter ms_dispatch 0x7f2cd0000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:51.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cccff9640 3 client.174261.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:51.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.001+0000 7f2cccff9640 3 client.174261.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:51.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.002+0000 7f2cccff9640 20 client.174261.objecter dump_active .. 0 homeless 2023-10-30T12:13:51.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.002+0000 7f2ccffff640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2cbc0392f0 0x7f2cbc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.002+0000 7f2ccffff640 10 monclient: get_auth_request con 0x7f2cbc0392f0 auth_method 0 2023-10-30T12:13:51.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.002+0000 7f2cccff9640 10 client.174261.objecter ms_handle_connect 0x7f2cbc0392f0 2023-10-30T12:13:51.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.002+0000 7f2ccffff640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2cbc0392f0 0x7f2cbc03b7b0 secure :-1 s=READY pgs=4166 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4006a30 tx=0x7f2cc4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.006+0000 7f2cccff9640 1 -- 172.21.15.50:0/4005621883 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2cc0038020 con 0x7f2cd00969d0 2023-10-30T12:13:51.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.006+0000 7f2cccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.006+0000 7f2cccff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:51.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.249+0000 7f2cd69cc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:51.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.249+0000 7f2cd69cc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:51.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.249+0000 7f2cd69cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.249+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2cd009c830 con 0x7f2cd00969d0 2023-10-30T12:13:51.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.250+0000 7f2cccff9640 1 -- 172.21.15.50:0/4005621883 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f2cd009c830 con 0x7f2cd00969d0 2023-10-30T12:13:51.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.250+0000 7f2cccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:51.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.250+0000 7f2cccff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:51.256 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:51.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.251+0000 7f2cd69cc640 10 librados: watch_flush enter 2023-10-30T12:13:51.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 10 librados: watch_flush exit 2023-10-30T12:13:51.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 20 client.174261.objecter shutdown clearing up homeless session... 2023-10-30T12:13:51.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 10 client.174261.objecter successfully canceled tick 2023-10-30T12:13:51.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2cbc0392f0 msgr2=0x7f2cbc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2cbc0392f0 0x7f2cbc03b7b0 secure :-1 s=READY pgs=4166 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4006a30 tx=0x7f2cc4008040 comp rx=0 tx=0).stop 2023-10-30T12:13:51.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 10 monclient: shutdown 2023-10-30T12:13:51.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd00969d0 msgr2=0x7f2cd00704a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd00969d0 0x7f2cd00704a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2cc002b820 tx=0x7f2cc00046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 shutdown_connections 2023-10-30T12:13:51.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2cbc0392f0 0x7f2cbc03b7b0 unknown :-1 s=CLOSED pgs=4166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2cd009f5b0 0x7f2cd00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cd00969d0 0x7f2cd00704a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 --2- 172.21.15.50:0/4005621883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2cd0096080 0x7f2cd006ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 >> 172.21.15.50:0/4005621883 conn(0x7f2cd00a05c0 msgr2=0x7f2cd01b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 shutdown_connections 2023-10-30T12:13:51.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 -- 172.21.15.50:0/4005621883 wait complete. 2023-10-30T12:13:51.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.252+0000 7f2cd69cc640 1 librados: shutdown 2023-10-30T12:13:51.263 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:13:51.264 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:13:51.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:13:51.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:13:51.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.369+0000 7f4d139a9640 10 monclient: get_monmap_and_config 2023-10-30T12:13:51.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.369+0000 7f4d139a9640 10 monclient: build_initial_monmap 2023-10-30T12:13:51.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.369+0000 7f4d139a9640 10 monclient: monmap: 2023-10-30T12:13:51.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:51.372 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.372 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:51.370580+0000 2023-10-30T12:13:51.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:51.370580+0000 2023-10-30T12:13:51.372 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:51.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:51.373 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:51.373 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:51.373 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:51.373 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:51.374 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 1 Processor -- start 2023-10-30T12:13:51.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 1 -- start start 2023-10-30T12:13:51.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient: init 2023-10-30T12:13:51.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-10-30T12:13:51.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4d0c06afc0 0x7f4d0c0a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient(hunting): picked mon.noname-a con 0x7f4d0c06afc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:51.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d0c06b640 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:51.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d0c13b070 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:51.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0c06b3a0 con 0x7f4d0c13b070 2023-10-30T12:13:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0c09e540 con 0x7f4d0c06b640 2023-10-30T12:13:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0c09fa10 con 0x7f4d0c06afc0 2023-10-30T12:13:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d139a9640 10 monclient(hunting): authenticate will time out at 1676.553764s 2023-10-30T12:13:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d11745640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4d0c06afc0 0x7f4d0c0a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d11f46640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4d0c13b070 0x7f4d0c13d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51298/0 (socket says 172.21.15.50:51298) 2023-10-30T12:13:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d11f46640 1 -- 172.21.15.50:0/1546036052 learned_addr learned my addr 172.21.15.50:0/1546036052 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d11745640 10 monclient(hunting): get_auth_request con 0x7f4d0c06afc0 auth_method 0 2023-10-30T12:13:51.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d11745640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d11745640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.370+0000 7f4d11745640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 10 monclient(hunting): handle_auth_done global_id 174264 payload 274 2023-10-30T12:13:51.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 1 -- 172.21.15.50:0/1546036052 >> v1:172.21.15.118:6789/0 conn(0x7f4d0c06b640 legacy=0x7f4d0c0a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 1 -- 172.21.15.50:0/1546036052 >> v1:172.21.15.50:6789/0 conn(0x7f4d0c13b070 legacy=0x7f4d0c13d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 1 monclient: found mon.noname-a 2023-10-30T12:13:51.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 1 -- 172.21.15.50:0/1546036052 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c06a6f0 con 0x7f4d0c06afc0 2023-10-30T12:13:51.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.371+0000 7f4d11745640 1 --2- 172.21.15.50:0/1546036052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06afc0 0x7f4d0c0a1c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4d080097d0 tx=0x7f4d0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba80f21f652ee81a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 1 -- 172.21.15.50:0/1546036052 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0802e070 con 0x7f4d0c06afc0 2023-10-30T12:13:51.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:51.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: dump: 2023-10-30T12:13:51.393 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.394 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.394 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.394 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.395 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.395 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.395 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.395 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.396 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.396 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 1 -- 172.21.15.50:0/1546036052 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d080315a0 con 0x7f4d0c06afc0 2023-10-30T12:13:51.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 5 monclient: authenticate success, global_id 174264 2023-10-30T12:13:51.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:51.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:51.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 1 -- 172.21.15.50:0/1546036052 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08031930 con 0x7f4d0c06afc0 2023-10-30T12:13:51.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 4 set_mon_vals no callback set 2023-10-30T12:13:51.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d03fff640 10 monclient: dump: 2023-10-30T12:13:51.399 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.399 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.399 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.399 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.399 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.399 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.400 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.400 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.400 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.400 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 10 monclient: shutdown 2023-10-30T12:13:51.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 1 -- 172.21.15.50:0/1546036052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06afc0 msgr2=0x7f4d0c0a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 1 --2- 172.21.15.50:0/1546036052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06afc0 0x7f4d0c0a1c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4d080097d0 tx=0x7f4d0802b840 comp rx=0 tx=0).stop 2023-10-30T12:13:51.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 1 -- 172.21.15.50:0/1546036052 shutdown_connections 2023-10-30T12:13:51.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 1 --2- 172.21.15.50:0/1546036052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06afc0 0x7f4d0c0a1c90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 1 -- 172.21.15.50:0/1546036052 >> 172.21.15.50:0/1546036052 conn(0x7f4d0c0a6200 msgr2=0x7f4d0c0a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 1 -- 172.21.15.50:0/1546036052 shutdown_connections 2023-10-30T12:13:51.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.372+0000 7f4d139a9640 1 -- 172.21.15.50:0/1546036052 wait complete. 2023-10-30T12:13:51.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient: build_initial_monmap 2023-10-30T12:13:51.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient: monmap: 2023-10-30T12:13:51.403 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:51.403 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:51.403 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:51.403 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:51.404 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:51.404 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.404 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:51.404 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:51.404 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:51.405 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 librados: starting msgr at 2023-10-30T12:13:51.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 librados: starting objecter 2023-10-30T12:13:51.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 Processor -- start 2023-10-30T12:13:51.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 -- start start 2023-10-30T12:13:51.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 librados: setting wanted keys 2023-10-30T12:13:51.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 librados: calling monclient init 2023-10-30T12:13:51.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient: init 2023-10-30T12:13:51.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:13:51.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d0c06afc0 0x7f4d0c136e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): picked mon.noname-b con 0x7f4d0c06afc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:51.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06b640 0x7f4d0c132da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): picked mon.noname-a con 0x7f4d0c06b640 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:51.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4d0c13b070 0x7f4d0c11bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): picked mon.noname-c con 0x7f4d0c13b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:51.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c070de0 con 0x7f4d0c06b640 2023-10-30T12:13:51.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c11f750 con 0x7f4d0c06afc0 2023-10-30T12:13:51.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c138760 con 0x7f4d0c13b070 2023-10-30T12:13:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.373+0000 7f4d139a9640 10 monclient(hunting): authenticate will time out at 1676.556967s 2023-10-30T12:13:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06b640 0x7f4d0c132da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4d0c13b070 0x7f4d0c11bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06b640 0x7f4d0c132da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50638/0 (socket says 172.21.15.50:50638) 2023-10-30T12:13:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 -- 172.21.15.50:0/3474891899 learned_addr learned my addr 172.21.15.50:0/3474891899 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): get_auth_request con 0x7f4d0c06b640 auth_method 0 2023-10-30T12:13:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 10 monclient(hunting): get_auth_request con 0x7f4d0c13b070 auth_method 0 2023-10-30T12:13:51.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d0c06afc0 0x7f4d0c136e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 10 monclient(hunting): get_auth_request con 0x7f4d0c06afc0 auth_method 0 2023-10-30T12:13:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11f46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient(hunting): handle_auth_done global_id 174270 payload 824 2023-10-30T12:13:51.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4d0c13b070 msgr2=0x7f4d0c11bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4d0c13b070 0x7f4d0c11bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d0c06afc0 msgr2=0x7f4d0c136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4d0c06afc0 0x7f4d0c136e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 monclient: found mon.noname-a 2023-10-30T12:13:51.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 -- 172.21.15.50:0/3474891899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08009400 con 0x7f4d0c06b640 2023-10-30T12:13:51.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d10f44640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06b640 0x7f4d0c132da0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00c450 tx=0x7f4cfc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 client.?.objecter ms_handle_connect 0x7f4d0c06b640 2023-10-30T12:13:51.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:51.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc0052d0 con 0x7f4d0c06b640 2023-10-30T12:13:51.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:51.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: dump: 2023-10-30T12:13:51.417 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.418 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.418 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.418 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.418 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.419 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.419 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 5 monclient: authenticate success, global_id 174270 2023-10-30T12:13:51.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cfc005430 con 0x7f4d0c06b640 2023-10-30T12:13:51.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc012870 con 0x7f4d0c06b640 2023-10-30T12:13:51.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 10 monclient: _renew_subs 2023-10-30T12:13:51.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0c11eeb0 con 0x7f4d0c06b640 2023-10-30T12:13:51.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d037fe640 4 set_mon_vals no callback set 2023-10-30T12:13:51.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d01ffb640 10 monclient: dump: 2023-10-30T12:13:51.422 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.422 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.423 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.423 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.423 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.423 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.423 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.424 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.424 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.424 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 10 client.174270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:51.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 10 monclient: _renew_subs 2023-10-30T12:13:51.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0c11d3a0 con 0x7f4d0c06b640 2023-10-30T12:13:51.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.374+0000 7f4d139a9640 1 librados: init done 2023-10-30T12:13:51.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4cfc0129d0 con 0x7f4d0c06b640 2023-10-30T12:13:51.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d139a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d139a9640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c11c3e0 0x7f4d0c11c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d139a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c073ff0 con 0x7f4d0c11c3e0 2023-10-30T12:13:51.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d01ffb640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4cf00392f0 0x7f4cf003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c11c3e0 0x7f4d0c11c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 10 monclient: get_auth_request con 0x7f4d0c11c3e0 auth_method 0 2023-10-30T12:13:51.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f4cfc04a720 con 0x7f4d0c06b640 2023-10-30T12:13:51.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d01ffb640 10 client.174270.objecter ms_dispatch 0x7f4d0c000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:51.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d01ffb640 3 client.174270.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:51.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d01ffb640 3 client.174270.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:51.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.375+0000 7f4d01ffb640 20 client.174270.objecter dump_active .. 0 homeless 2023-10-30T12:13:51.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d11745640 10 monclient(hunting): handle_auth_done global_id 174273 payload 824 2023-10-30T12:13:51.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d11f46640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4cf00392f0 0x7f4cf003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d11745640 1 -- 172.21.15.50:0/3474891899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d0c09f0b0 con 0x7f4d0c11c3e0 2023-10-30T12:13:51.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d11f46640 10 monclient: get_auth_request con 0x7f4cf00392f0 auth_method 0 2023-10-30T12:13:51.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 client.174270.objecter ms_handle_connect 0x7f4d0c11c3e0 2023-10-30T12:13:51.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d11745640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c11c3e0 0x7f4d0c11c7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d080097a0 tx=0x7f4d08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 client.174270.objecter resend_mon_ops 2023-10-30T12:13:51.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08032a00 con 0x7f4d0c11c3e0 2023-10-30T12:13:51.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 monclient: dump: 2023-10-30T12:13:51.431 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.431 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.431 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.432 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.432 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.432 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.432 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.432 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.432 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.433 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 client.174270.objecter ms_handle_connect 0x7f4cf00392f0 2023-10-30T12:13:51.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d11f46640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4cf00392f0 0x7f4cf003b7b0 secure :-1 s=READY pgs=4167 cs=0 l=1 rev1=1 crypto rx=0x7f4d040012e0 tx=0x7f4d04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f4d08032b60 con 0x7f4d0c11c3e0 2023-10-30T12:13:51.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:51.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c11c3e0 msgr2=0x7f4d0c11c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c11c3e0 0x7f4d0c11c7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d080097a0 tx=0x7f4d08004730 comp rx=0 tx=0).stop 2023-10-30T12:13:51.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c11c3e0 msgr2=0x7f4d0c11c7c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:51.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.376+0000 7f4d01ffb640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c11c3e0 0x7f4d0c11c7c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.382+0000 7f4d139a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:51.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.382+0000 7f4d139a9640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06e4b0 0x7f4d0c06e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.382+0000 7f4d139a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.382+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c11d530 con 0x7f4d0c06e4b0 2023-10-30T12:13:51.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.382+0000 7f4d11745640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06e4b0 0x7f4d0c06e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient: get_auth_request con 0x7f4d0c06e4b0 auth_method 0 2023-10-30T12:13:51.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 10 monclient(hunting): handle_auth_done global_id 174276 payload 824 2023-10-30T12:13:51.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 1 -- 172.21.15.50:0/3474891899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d0c09df80 con 0x7f4d0c06e4b0 2023-10-30T12:13:51.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 10 client.174270.objecter ms_handle_connect 0x7f4d0c06e4b0 2023-10-30T12:13:51.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 10 client.174270.objecter resend_mon_ops 2023-10-30T12:13:51.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d11745640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06e4b0 0x7f4d0c06e8c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d0802b880 tx=0x7f4d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0804f660 con 0x7f4d0c06e4b0 2023-10-30T12:13:51.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 10 monclient: dump: 2023-10-30T12:13:51.444 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.444 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.444 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.444 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.444 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.444 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.445 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.445 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.445 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.445 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f4d08053070 con 0x7f4d0c06e4b0 2023-10-30T12:13:51.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:51.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:51.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06e4b0 msgr2=0x7f4d0c06e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06e4b0 0x7f4d0c06e8c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d0802b880 tx=0x7f4d080046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06e4b0 msgr2=0x7f4d0c06e8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:51.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.383+0000 7f4d01ffb640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06e4b0 0x7f4d0c06e8c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.447 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:51.447 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:51.447 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:51.448 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:13:51.448 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 130, 2023-10-30T12:13:51.448 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:51.448 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:51.448 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:51.448 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:51.449 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.449 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:13:51.449 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:51.449 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:51.449 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:51.449 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.450 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.450 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.450 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.450 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.451 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.451 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.451 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.451 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.451 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.452 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:51.452 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:51.452 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.452 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.452 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.453 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.453 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.453 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.453 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.453 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.454 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.454 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.454 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.454 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:51.455 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:51.455 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:51.455 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:51.455 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:51.455 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:51.456 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:51.456 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:51.456 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:51.456 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:51.456 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:51.457 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:51.457 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:51.457 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:51.457 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:51.457 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:51.457 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:51.458 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.458 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.458 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.458 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:13:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:51.480 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:51.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 10 librados: watch_flush enter 2023-10-30T12:13:51.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 10 librados: watch_flush exit 2023-10-30T12:13:51.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 20 client.174270.objecter shutdown clearing up homeless session... 2023-10-30T12:13:51.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 10 client.174270.objecter successfully canceled tick 2023-10-30T12:13:51.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4cf00392f0 msgr2=0x7f4cf003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4cf00392f0 0x7f4cf003b7b0 secure :-1 s=READY pgs=4167 cs=0 l=1 rev1=1 crypto rx=0x7f4d040012e0 tx=0x7f4d04009040 comp rx=0 tx=0).stop 2023-10-30T12:13:51.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 10 monclient: shutdown 2023-10-30T12:13:51.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06b640 msgr2=0x7f4d0c132da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 --2- 172.21.15.50:0/3474891899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d0c06b640 0x7f4d0c132da0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00c450 tx=0x7f4cfc00c910 comp rx=0 tx=0).stop 2023-10-30T12:13:51.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d11f46640 1 -- 172.21.15.50:0/3474891899 reap_dead start 2023-10-30T12:13:51.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d11f46640 1 -- 172.21.15.50:0/3474891899 reap_dead start 2023-10-30T12:13:51.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 shutdown_connections 2023-10-30T12:13:51.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 >> 172.21.15.50:0/3474891899 conn(0x7f4d0c0a6200 msgr2=0x7f4d0c137fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 shutdown_connections 2023-10-30T12:13:51.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 -- 172.21.15.50:0/3474891899 wait complete. 2023-10-30T12:13:51.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.384+0000 7f4d139a9640 1 librados: shutdown 2023-10-30T12:13:51.487 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:51.487 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:13:51.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:13:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.592+0000 7fa90a84b640 10 monclient: get_monmap_and_config 2023-10-30T12:13:51.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.592+0000 7fa90a84b640 10 monclient: build_initial_monmap 2023-10-30T12:13:51.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.592+0000 7fa90a84b640 10 monclient: monmap: 2023-10-30T12:13:51.594 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:51.594 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.594 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:51.593153+0000 2023-10-30T12:13:51.594 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:51.593153+0000 2023-10-30T12:13:51.595 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:51.595 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.595 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:51.595 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:51.596 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:51.596 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:51.596 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:51.596 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.592+0000 7fa90a84b640 1 Processor -- start 2023-10-30T12:13:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.592+0000 7fa90a84b640 1 -- start start 2023-10-30T12:13:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.592+0000 7fa90a84b640 10 monclient: init 2023-10-30T12:13:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:13:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9040a2ae0 0x7fa9040a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient(hunting): picked mon.noname-a con 0x7fa9040a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9040a2ec0 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa904135df0 0x7fa90413da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient(hunting): picked mon.noname-c con 0x7fa904135df0 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9040a4190 con 0x7fa9040a2ec0 2023-10-30T12:13:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9040a2990 con 0x7fa9040a2ae0 2023-10-30T12:13:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9040a1630 con 0x7fa904135df0 2023-10-30T12:13:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa90a84b640 10 monclient(hunting): authenticate will time out at 1676.776493s 2023-10-30T12:13:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9040a2ae0 0x7fa9040a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa9037fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa9040a2ec0 0x7fa9041356e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51308/0 (socket says 172.21.15.50:51308) 2023-10-30T12:13:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa9037fe640 1 -- 172.21.15.50:0/3037523505 learned_addr learned my addr 172.21.15.50:0/3037523505 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): get_auth_request con 0x7fa9040a2ae0 auth_method 0 2023-10-30T12:13:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa908de8640 1 --2- 172.21.15.50:0/3037523505 >> v2:172.21.15.118:3300/0 conn(0x7fa904135df0 0x7fa90413da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa908de8640 10 monclient(hunting): get_auth_request con 0x7fa904135df0 auth_method 0 2023-10-30T12:13:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa908de8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa908de8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa908de8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa902ffd640 1 -- 172.21.15.50:0/3037523505 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288664190 0 0) 0x7fa8f4001110 con 0x7fa9040a2ec0 2023-10-30T12:13:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa902ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:13:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa902ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa902ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa902ffd640 10 monclient(hunting): my global_id is 174282 2023-10-30T12:13:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa902ffd640 1 -- 172.21.15.50:0/3037523505 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ec003520 con 0x7fa9040a2ec0 2023-10-30T12:13:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient(hunting): handle_auth_done global_id 174279 payload 274 2023-10-30T12:13:51.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 1 -- 172.21.15.50:0/3037523505 >> v2:172.21.15.118:3300/0 conn(0x7fa904135df0 msgr2=0x7fa90413da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 1 --2- 172.21.15.50:0/3037523505 >> v2:172.21.15.118:3300/0 conn(0x7fa904135df0 0x7fa90413da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 1 -- 172.21.15.50:0/3037523505 >> v1:172.21.15.50:6789/0 conn(0x7fa9040a2ec0 legacy=0x7fa9041356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 1 monclient: found mon.noname-a 2023-10-30T12:13:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.593+0000 7fa903fff640 1 -- 172.21.15.50:0/3037523505 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90409a2e0 con 0x7fa9040a2ae0 2023-10-30T12:13:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa908de8640 1 --2- 172.21.15.50:0/3037523505 >> v2:172.21.15.118:3300/0 conn(0x7fa904135df0 0x7fa90413da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:13:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa903fff640 1 --2- 172.21.15.50:0/3037523505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9040a2450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa9040a54d0 tx=0x7fa8f802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820e4c7f69496aa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 1 -- 172.21.15.50:0/3037523505 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8031710 con 0x7fa9040a2ae0 2023-10-30T12:13:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:51.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: dump: 2023-10-30T12:13:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.622 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.623 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.624 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.625 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa90a84b640 5 monclient: authenticate success, global_id 174279 2023-10-30T12:13:51.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa90a84b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 1 -- 172.21.15.50:0/3037523505 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f802f050 con 0x7fa9040a2ae0 2023-10-30T12:13:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa90a84b640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa90a84b640 4 set_mon_vals no callback set 2023-10-30T12:13:51.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 1 -- 172.21.15.50:0/3037523505 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8031a90 con 0x7fa9040a2ae0 2023-10-30T12:13:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.594+0000 7fa902ffd640 10 monclient: dump: 2023-10-30T12:13:51.629 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.630 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.631 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 10 monclient: shutdown 2023-10-30T12:13:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 -- 172.21.15.50:0/3037523505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 msgr2=0x7fa9040a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 --2- 172.21.15.50:0/3037523505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9040a2450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa9040a54d0 tx=0x7fa8f802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 -- 172.21.15.50:0/3037523505 shutdown_connections 2023-10-30T12:13:51.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 --2- 172.21.15.50:0/3037523505 >> v2:172.21.15.118:3300/0 conn(0x7fa904135df0 0x7fa90413da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 --2- 172.21.15.50:0/3037523505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9040a2450 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 -- 172.21.15.50:0/3037523505 >> 172.21.15.50:0/3037523505 conn(0x7fa9040a80b0 msgr2=0x7fa9040a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 -- 172.21.15.50:0/3037523505 shutdown_connections 2023-10-30T12:13:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.595+0000 7fa90a84b640 1 -- 172.21.15.50:0/3037523505 wait complete. 2023-10-30T12:13:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient: build_initial_monmap 2023-10-30T12:13:51.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient: monmap: 2023-10-30T12:13:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:51.634 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:51.635 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:51.635 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 librados: starting msgr at 2023-10-30T12:13:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 librados: starting objecter 2023-10-30T12:13:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 Processor -- start 2023-10-30T12:13:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 -- start start 2023-10-30T12:13:51.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 librados: setting wanted keys 2023-10-30T12:13:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 librados: calling monclient init 2023-10-30T12:13:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient: init 2023-10-30T12:13:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:13:51.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9041ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): picked mon.noname-a con 0x7fa9040a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9040a2ec0 0x7fa9041ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): picked mon.noname-b con 0x7fa9040a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904135df0 0x7fa90411f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): picked mon.noname-c con 0x7fa904135df0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa904133040 con 0x7fa9040a2ae0 2023-10-30T12:13:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9040a1790 con 0x7fa9040a2ec0 2023-10-30T12:13:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa904122200 con 0x7fa904135df0 2023-10-30T12:13:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa90a84b640 10 monclient(hunting): authenticate will time out at 1676.779904s 2023-10-30T12:13:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.596+0000 7fa903fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9041ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9041ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50668/0 (socket says 172.21.15.50:50668) 2023-10-30T12:13:51.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904135df0 0x7fa90411f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 -- 172.21.15.50:0/770631122 learned_addr learned my addr 172.21.15.50:0/770631122 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): get_auth_request con 0x7fa9040a2ae0 auth_method 0 2023-10-30T12:13:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 10 monclient(hunting): get_auth_request con 0x7fa904135df0 auth_method 0 2023-10-30T12:13:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9040a2ec0 0x7fa9041ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 10 monclient(hunting): get_auth_request con 0x7fa9040a2ec0 auth_method 0 2023-10-30T12:13:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa908de8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient(hunting): handle_auth_done global_id 174285 payload 824 2023-10-30T12:13:51.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904135df0 msgr2=0x7fa90411f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904135df0 0x7fa90411f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9040a2ec0 msgr2=0x7fa9041ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9040a2ec0 0x7fa9041ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 monclient: found mon.noname-a 2023-10-30T12:13:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 -- 172.21.15.50:0/770631122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8009400 con 0x7fa9040a2ae0 2023-10-30T12:13:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa903fff640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9041ca290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8000c00 tx=0x7fa8f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 client.?.objecter ms_handle_connect 0x7fa9040a2ae0 2023-10-30T12:13:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:51.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8033890 con 0x7fa9040a2ae0 2023-10-30T12:13:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:13:51.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: dump: 2023-10-30T12:13:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.651 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.652 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.653 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.654 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 5 monclient: authenticate success, global_id 174285 2023-10-30T12:13:51.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f80339f0 con 0x7fa9040a2ae0 2023-10-30T12:13:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 10 monclient: _renew_subs 2023-10-30T12:13:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9041355e0 con 0x7fa9040a2ae0 2023-10-30T12:13:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8033da0 con 0x7fa9040a2ae0 2023-10-30T12:13:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:13:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa900ff9640 10 monclient: dump: 2023-10-30T12:13:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.657 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.658 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.659 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 10 client.174285.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 10 monclient: _renew_subs 2023-10-30T12:13:51.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:13:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9041215c0 con 0x7fa9040a2ae0 2023-10-30T12:13:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa9027fc640 4 set_mon_vals no callback set 2023-10-30T12:13:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.597+0000 7fa90a84b640 1 librados: init done 2023-10-30T12:13:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa90a84b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa8f803a040 con 0x7fa9040a2ae0 2023-10-30T12:13:51.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa90a84b640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa90411fe60 0x7fa904120240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa90a84b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa904073ff0 con 0x7fa90411fe60 2023-10-30T12:13:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa90411fe60 0x7fa904120240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient: get_auth_request con 0x7fa90411fe60 auth_method 0 2023-10-30T12:13:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa900ff9640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f00392f0 0x7fa8f003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fa8f806c1e0 con 0x7fa9040a2ae0 2023-10-30T12:13:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa900ff9640 10 client.174285.objecter ms_dispatch 0x7fa904000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa900ff9640 3 client.174285.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa900ff9640 3 client.174285.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 10 monclient(hunting): handle_auth_done global_id 114881 payload 824 2023-10-30T12:13:51.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa900ff9640 20 client.174285.objecter dump_active .. 0 homeless 2023-10-30T12:13:51.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.598+0000 7fa9037fe640 1 -- 172.21.15.50:0/770631122 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa90409a5c0 con 0x7fa90411fe60 2023-10-30T12:13:51.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 client.174285.objecter ms_handle_connect 0x7fa90411fe60 2023-10-30T12:13:51.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa9037fe640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa90411fe60 0x7fa904120240 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa904122f70 tx=0x7fa8f40078e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 client.174285.objecter resend_mon_ops 2023-10-30T12:13:51.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa908de8640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f00392f0 0x7fa8f003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa908de8640 10 monclient: get_auth_request con 0x7fa8f00392f0 auth_method 0 2023-10-30T12:13:51.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4013600 con 0x7fa90411fe60 2023-10-30T12:13:51.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:51.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 monclient: dump: 2023-10-30T12:13:51.668 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.669 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.669 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.669 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.669 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.669 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.670 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.670 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.670 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.670 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa8f4013760 con 0x7fa90411fe60 2023-10-30T12:13:51.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:51.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa90411fe60 msgr2=0x7fa904120240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa90411fe60 0x7fa904120240 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa904122f70 tx=0x7fa8f40078e0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa90411fe60 msgr2=0x7fa904120240 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:51.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa90411fe60 0x7fa904120240 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa904122f70 tx=0x7fa8f40078e0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa900ff9640 10 client.174285.objecter ms_handle_connect 0x7fa8f00392f0 2023-10-30T12:13:51.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.599+0000 7fa908de8640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f00392f0 0x7fa8f003b7b0 secure :-1 s=READY pgs=4168 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc007440 tx=0x7fa8fc009850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa90a84b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:51.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa90a84b640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904120780 0x7fa90409dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa90a84b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa904099b40 con 0x7fa904120780 2023-10-30T12:13:51.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904120780 0x7fa90409dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 10 monclient: get_auth_request con 0x7fa904120780 auth_method 0 2023-10-30T12:13:51.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.605+0000 7fa9037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa9037fe640 10 monclient(hunting): handle_auth_done global_id 114884 payload 824 2023-10-30T12:13:51.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa9037fe640 1 -- 172.21.15.50:0/770631122 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa90406b060 con 0x7fa904120780 2023-10-30T12:13:51.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa9037fe640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904120780 0x7fa90409dd70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa904121830 tx=0x7fa8f40012e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 10 client.174285.objecter ms_handle_connect 0x7fa904120780 2023-10-30T12:13:51.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 10 client.174285.objecter resend_mon_ops 2023-10-30T12:13:51.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4023070 con 0x7fa904120780 2023-10-30T12:13:51.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:51.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 10 monclient: dump: 2023-10-30T12:13:51.676 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.676 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.676 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.677 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.677 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.677 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.677 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.677 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.677 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.678 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7fa8f4006310 con 0x7fa904120780 2023-10-30T12:13:51.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:51.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:51.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904120780 msgr2=0x7fa90409dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904120780 0x7fa90409dd70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa904121830 tx=0x7fa8f40012e0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904120780 msgr2=0x7fa90409dd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:51.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa900ff9640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa904120780 0x7fa90409dd70 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa904121830 tx=0x7fa8f40012e0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.680 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:51.680 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:51.680 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:51.680 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:51.680 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 130, 2023-10-30T12:13:51.681 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:51.681 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:51.681 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:51.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:51.681 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.682 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:13:51.682 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:51.682 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:51.682 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:51.682 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.683 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.683 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.683 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.683 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.684 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.684 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.684 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.684 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.684 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.685 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:51.685 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:51.685 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.685 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.687 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.687 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.687 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:51.687 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:51.688 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:51.688 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:51.688 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:51.688 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:51.688 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:51.689 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:51.689 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:51.689 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:51.689 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:51.690 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:51.690 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:51.690 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:51.690 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:51.690 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:51.691 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:51.691 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.691 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.691 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.691 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.691 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.692 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.692 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.692 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.692 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.692 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.694 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:51.694 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:51.694 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.695 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:51.695 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:51.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.697 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:51.697 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:51.697 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.697 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.698 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.698 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:51.698 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:51.699 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:51.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:51.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:51.701 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:51.702 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.702 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.702 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.703 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:51.703 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:51.703 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.703 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.703 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:51.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:51.705 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:51.706 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:51.706 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.706 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.706 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.707 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:51.707 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:51.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.708 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.708 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.708 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:51.708 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.709 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:13:51.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.709 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.709 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.709 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:13:51.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.710 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.710 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:51.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.710 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.711 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.711 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:13:51.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.711 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.711 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:13:51.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.712 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.712 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.712 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:51.712 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.712 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.713 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:51.713 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:51.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 10 librados: watch_flush enter 2023-10-30T12:13:51.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 10 librados: watch_flush exit 2023-10-30T12:13:51.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 20 client.174285.objecter shutdown clearing up homeless session... 2023-10-30T12:13:51.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 10 client.174285.objecter successfully canceled tick 2023-10-30T12:13:51.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f00392f0 msgr2=0x7fa8f003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa8f00392f0 0x7fa8f003b7b0 secure :-1 s=READY pgs=4168 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc007440 tx=0x7fa8fc009850 comp rx=0 tx=0).stop 2023-10-30T12:13:51.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 10 monclient: shutdown 2023-10-30T12:13:51.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 msgr2=0x7fa9041ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa90a84b640 1 --2- 172.21.15.50:0/770631122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9040a2ae0 0x7fa9041ca290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8000c00 tx=0x7fa8f80046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.606+0000 7fa908de8640 1 -- 172.21.15.50:0/770631122 reap_dead start 2023-10-30T12:13:51.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.607+0000 7fa908de8640 1 -- 172.21.15.50:0/770631122 reap_dead start 2023-10-30T12:13:51.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.607+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 shutdown_connections 2023-10-30T12:13:51.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.607+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 >> 172.21.15.50:0/770631122 conn(0x7fa9040a80b0 msgr2=0x7fa9040a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.607+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 shutdown_connections 2023-10-30T12:13:51.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.607+0000 7fa90a84b640 1 -- 172.21.15.50:0/770631122 wait complete. 2023-10-30T12:13:51.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.607+0000 7fa90a84b640 1 librados: shutdown 2023-10-30T12:13:51.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:51.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:13:51.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.824+0000 7f37f1b8a640 10 monclient: get_monmap_and_config 2023-10-30T12:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.824+0000 7f37f1b8a640 10 monclient: build_initial_monmap 2023-10-30T12:13:51.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.824+0000 7f37f1b8a640 10 monclient: monmap: 2023-10-30T12:13:51.827 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:51.827 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:51.825915+0000 2023-10-30T12:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:51.825915+0000 2023-10-30T12:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:51.829 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:51.829 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:51.829 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:51.829 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:51.829 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.825+0000 7f37f1b8a640 1 Processor -- start 2023-10-30T12:13:51.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.825+0000 7f37f1b8a640 1 -- start start 2023-10-30T12:13:51.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.825+0000 7f37f1b8a640 10 monclient: init 2023-10-30T12:13:51.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.825+0000 7f37f1b8a640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.825+0000 7f37f1b8a640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-10-30T12:13:51.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.825+0000 7f37f1b8a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37ec097780 addr v1:172.21.15.50:6789/0 2023-10-30T12:13:51.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f37ec073b10 0x7f37ec13aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec073b10 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:51.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec09ccd0 0x7f37ec13cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec09ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:51.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec071ec0 con 0x7f37ec097780 2023-10-30T12:13:51.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ec09bd10 con 0x7f37ec073b10 2023-10-30T12:13:51.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec09bff0 con 0x7f37ec09ccd0 2023-10-30T12:13:51.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37f1b8a640 10 monclient(hunting): authenticate will time out at 1677.009056s 2023-10-30T12:13:51.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec09ccd0 0x7f37ec13cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37eb7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f37ec097780 0x7f37ec097b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51322/0 (socket says 172.21.15.50:51322) 2023-10-30T12:13:51.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37eb7fe640 1 -- 172.21.15.50:0/3691792032 learned_addr learned my addr 172.21.15.50:0/3691792032 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:13:51.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): get_auth_request con 0x7f37ec09ccd0 auth_method 0 2023-10-30T12:13:51.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37eaffd640 1 --2- 172.21.15.50:0/3691792032 >> v2:172.21.15.118:3300/0 conn(0x7f37ec073b10 0x7f37ec13aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37eaffd640 10 monclient(hunting): get_auth_request con 0x7f37ec073b10 auth_method 0 2023-10-30T12:13:51.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37eaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient(hunting): handle_auth_done global_id 114887 payload 274 2023-10-30T12:13:51.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 1 -- 172.21.15.50:0/3691792032 >> v2:172.21.15.118:3300/0 conn(0x7f37ec073b10 msgr2=0x7f37ec13aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:51.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 1 --2- 172.21.15.50:0/3691792032 >> v2:172.21.15.118:3300/0 conn(0x7f37ec073b10 0x7f37ec13aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ea7fc640 1 -- 172.21.15.50:0/3691792032 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952194432 0 0) 0x7f37ec071ec0 con 0x7f37ec097780 2023-10-30T12:13:51.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 1 -- 172.21.15.50:0/3691792032 >> v1:172.21.15.50:6789/0 conn(0x7f37ec097780 legacy=0x7f37ec097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 1 monclient: found mon.noname-b 2023-10-30T12:13:51.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:51.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ebfff640 1 -- 172.21.15.50:0/3691792032 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec071b20 con 0x7f37ec09ccd0 2023-10-30T12:13:51.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.826+0000 7f37ea7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:13:51.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ebfff640 1 --2- 172.21.15.50:0/3691792032 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec09ccd0 0x7f37ec13cfe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009d90 tx=0x7f37e402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc0fa6567ea7ba4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 1 -- 172.21.15.50:0/3691792032 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e402cb30 con 0x7f37ec09ccd0 2023-10-30T12:13:51.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:51.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: dump: 2023-10-30T12:13:51.852 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.852 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.852 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.853 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.853 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.853 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.853 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.853 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.854 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.854 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 5 monclient: authenticate success, global_id 114887 2023-10-30T12:13:51.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 1 -- 172.21.15.50:0/3691792032 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e402cc90 con 0x7f37ec09ccd0 2023-10-30T12:13:51.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:51.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:51.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 4 set_mon_vals no callback set 2023-10-30T12:13:51.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 1 -- 172.21.15.50:0/3691792032 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4034440 con 0x7f37ec09ccd0 2023-10-30T12:13:51.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:51.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37ea7fc640 10 monclient: dump: 2023-10-30T12:13:51.857 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.857 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.857 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.858 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.858 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.859 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.859 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.859 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.859 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.860 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 10 monclient: shutdown 2023-10-30T12:13:51.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 1 -- 172.21.15.50:0/3691792032 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec09ccd0 msgr2=0x7f37ec13cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/3691792032 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec09ccd0 0x7f37ec13cfe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009d90 tx=0x7f37e402bd80 comp rx=0 tx=0).stop 2023-10-30T12:13:51.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 1 -- 172.21.15.50:0/3691792032 shutdown_connections 2023-10-30T12:13:51.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/3691792032 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec09ccd0 0x7f37ec13cfe0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/3691792032 >> v2:172.21.15.118:3300/0 conn(0x7f37ec073b10 0x7f37ec13aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.827+0000 7f37f1b8a640 1 -- 172.21.15.50:0/3691792032 >> 172.21.15.50:0/3691792032 conn(0x7f37ec0a1170 msgr2=0x7f37ec0a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.828+0000 7f37f1b8a640 1 -- 172.21.15.50:0/3691792032 shutdown_connections 2023-10-30T12:13:51.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.828+0000 7f37f1b8a640 1 -- 172.21.15.50:0/3691792032 wait complete. 2023-10-30T12:13:51.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.828+0000 7f37f1b8a640 10 monclient: build_initial_monmap 2023-10-30T12:13:51.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.828+0000 7f37f1b8a640 10 monclient: monmap: 2023-10-30T12:13:51.864 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:51.864 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:51.865 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:51.865 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:51.866 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:51.866 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.866 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:51.866 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:51.869 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:51.869 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.828+0000 7f37f1b8a640 1 librados: starting msgr at 2023-10-30T12:13:51.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.828+0000 7f37f1b8a640 1 librados: starting objecter 2023-10-30T12:13:51.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.828+0000 7f37f1b8a640 1 Processor -- start 2023-10-30T12:13:51.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 -- start start 2023-10-30T12:13:51.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 librados: setting wanted keys 2023-10-30T12:13:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 librados: calling monclient init 2023-10-30T12:13:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient: init 2023-10-30T12:13:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:13:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec073b10 0x7f37ec08bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec073b10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec097780 0x7f37ec1ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec09ccd0 0x7f37ec0948f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec09ccd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:51.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec1ca8b0 con 0x7f37ec09ccd0 2023-10-30T12:13:51.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec1cae10 con 0x7f37ec073b10 2023-10-30T12:13:51.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec0a2b30 con 0x7f37ec097780 2023-10-30T12:13:51.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:51.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37f1b8a640 10 monclient(hunting): authenticate will time out at 1677.012495s 2023-10-30T12:13:51.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec097780 0x7f37ec1ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec09ccd0 0x7f37ec0948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec097780 0x7f37ec1ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33712/0 (socket says 172.21.15.50:33712) 2023-10-30T12:13:51.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec09ccd0 0x7f37ec0948f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50676/0 (socket says 172.21.15.50:50676) 2023-10-30T12:13:51.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 -- 172.21.15.50:0/1180257987 learned_addr learned my addr 172.21.15.50:0/1180257987 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:51.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): get_auth_request con 0x7f37ec097780 auth_method 0 2023-10-30T12:13:51.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 10 monclient(hunting): get_auth_request con 0x7f37ec09ccd0 auth_method 0 2023-10-30T12:13:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eb7fe640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec073b10 0x7f37ec08bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient(hunting): handle_auth_done global_id 114890 payload 824 2023-10-30T12:13:51.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec073b10 msgr2=0x7f37ec08bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec073b10 0x7f37ec08bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec09ccd0 msgr2=0x7f37ec0948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec09ccd0 0x7f37ec0948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:13:51.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 monclient: found mon.noname-c 2023-10-30T12:13:51.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:51.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:51.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eaffd640 1 -- 172.21.15.50:0/1180257987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e4009a70 con 0x7f37ec097780 2023-10-30T12:13:51.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eb7fe640 10 monclient: get_auth_request con 0x7f37ec073b10 auth_method 0 2023-10-30T12:13:51.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37eb7fe640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec073b10 0x7f37ec08bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:51.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.829+0000 7f37ebfff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec09ccd0 0x7f37ec0948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:13:51.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37eaffd640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec097780 0x7f37ec1ca370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f37e000d800 tx=0x7f37e000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 client.?.objecter ms_handle_connect 0x7f37ec097780 2023-10-30T12:13:51.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:51.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e000b6c0 con 0x7f37ec097780 2023-10-30T12:13:51.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:51.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: dump: 2023-10-30T12:13:51.886 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.887 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.888 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.889 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: _finish_auth 0 2023-10-30T12:13:51.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:13:51.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:51.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 5 monclient: authenticate success, global_id 114890 2023-10-30T12:13:51.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e000b820 con 0x7f37ec097780 2023-10-30T12:13:51.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:51.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e000bbb0 con 0x7f37ec097780 2023-10-30T12:13:51.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 10 monclient: _renew_subs 2023-10-30T12:13:51.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:51.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec08b6e0 con 0x7f37ec097780 2023-10-30T12:13:51.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:51.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37e9ffb640 4 set_mon_vals no callback set 2023-10-30T12:13:51.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37d3fff640 10 monclient: dump: 2023-10-30T12:13:51.892 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.892 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.892 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.892 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.893 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.893 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.893 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.893 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.893 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.893 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 10 client.114890.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:51.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 10 monclient: _renew_subs 2023-10-30T12:13:51.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:51.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec08cda0 con 0x7f37ec097780 2023-10-30T12:13:51.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.830+0000 7f37f1b8a640 1 librados: init done 2023-10-30T12:13:51.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37f1b8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f37e000bd10 con 0x7f37ec097780 2023-10-30T12:13:51.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec13d520 0x7f37ec1c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37f1b8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec06a5a0 con 0x7f37ec13d520 2023-10-30T12:13:51.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37d80392f0 0x7f37d803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f37e004b090 con 0x7f37ec097780 2023-10-30T12:13:51.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 10 client.114890.objecter ms_dispatch 0x7f37ec000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:51.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 3 client.114890.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:51.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 3 client.114890.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:51.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec13d520 0x7f37ec1c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient: get_auth_request con 0x7f37ec13d520 auth_method 0 2023-10-30T12:13:51.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 20 client.114890.objecter dump_active .. 0 homeless 2023-10-30T12:13:51.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37ebfff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37d80392f0 0x7f37d803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37ebfff640 10 monclient: get_auth_request con 0x7f37d80392f0 auth_method 0 2023-10-30T12:13:51.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_done global_id 184144 payload 824 2023-10-30T12:13:51.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 1 -- 172.21.15.50:0/1180257987 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37ec08cf60 con 0x7f37ec13d520 2023-10-30T12:13:51.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 10 client.114890.objecter ms_handle_connect 0x7f37d80392f0 2023-10-30T12:13:51.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 10 client.114890.objecter ms_handle_connect 0x7f37ec13d520 2023-10-30T12:13:51.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37d3fff640 10 client.114890.objecter resend_mon_ops 2023-10-30T12:13:51.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37ebfff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37d80392f0 0x7f37d803b7b0 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 crypto rx=0x7f37e4036000 tx=0x7f37e4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.831+0000 7f37eb7fe640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec13d520 0x7f37ec1c6970 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000f60 tx=0x7f37dc00db80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0072d0 con 0x7f37ec13d520 2023-10-30T12:13:51.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:51.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 10 monclient: dump: 2023-10-30T12:13:51.902 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.902 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.902 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.902 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.903 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.903 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.903 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.903 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.903 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.904 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f37dc007430 con 0x7f37ec13d520 2023-10-30T12:13:51.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:51.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:51.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec13d520 msgr2=0x7f37ec1c6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec13d520 0x7f37ec1c6970 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000f60 tx=0x7f37dc00db80 comp rx=0 tx=0).stop 2023-10-30T12:13:51.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec13d520 msgr2=0x7f37ec1c6970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:51.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.832+0000 7f37d3fff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec13d520 0x7f37ec1c6970 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000f60 tx=0x7f37dc00db80 comp rx=0 tx=0).stop 2023-10-30T12:13:51.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37f1b8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:13:51.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec1c6eb0 0x7f37ec1c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:51.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37f1b8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:51.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec1c7800 con 0x7f37ec1c6eb0 2023-10-30T12:13:51.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37eb7fe640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec1c6eb0 0x7f37ec1c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:51.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37eb7fe640 10 monclient: get_auth_request con 0x7f37ec1c6eb0 auth_method 0 2023-10-30T12:13:51.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:51.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37eb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:51.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.838+0000 7f37eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:51.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:51.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:51.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:51.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37eb7fe640 10 monclient(hunting): handle_auth_done global_id 184147 payload 824 2023-10-30T12:13:51.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37eb7fe640 1 -- 172.21.15.50:0/1180257987 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f37ec04e160 con 0x7f37ec1c6eb0 2023-10-30T12:13:51.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37eb7fe640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec1c6eb0 0x7f37ec1c72c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000f60 tx=0x7f37dc001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:51.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37d3fff640 10 client.114890.objecter ms_handle_connect 0x7f37ec1c6eb0 2023-10-30T12:13:51.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.839+0000 7f37d3fff640 10 client.114890.objecter resend_mon_ops 2023-10-30T12:13:51.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00b920 con 0x7f37ec1c6eb0 2023-10-30T12:13:51.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:51.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:13:51.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 10 monclient: dump: 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:51.910 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:51.911 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:51.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f37dc00ba80 con 0x7f37ec1c6eb0 2023-10-30T12:13:51.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:13:51.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:51.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec1c6eb0 msgr2=0x7f37ec1c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec1c6eb0 0x7f37ec1c72c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000f60 tx=0x7f37dc001080 comp rx=0 tx=0).stop 2023-10-30T12:13:51.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec1c6eb0 msgr2=0x7f37ec1c72c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:13:51.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37d3fff640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec1c6eb0 0x7f37ec1c72c0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000f60 tx=0x7f37dc001080 comp rx=0 tx=0).stop 2023-10-30T12:13:51.913 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:13:51.913 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:51.914 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:51.914 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:13:51.914 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 130, 2023-10-30T12:13:51.914 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:13:51.915 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:13:51.915 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:13:51.915 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:13:51.915 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.915 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:13:51.915 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:51.916 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:13:51.916 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:13:51.916 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.916 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.916 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.917 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.917 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.917 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.917 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.918 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.918 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:13:51.918 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:13:51.919 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.919 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.919 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.919 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.919 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.920 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.920 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.920 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.920 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.920 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.921 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:13:51.921 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:13:51.921 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:13:51.921 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:13:51.922 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:13:51.922 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:13:51.922 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:51.922 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:13:51.922 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:13:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:13:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:13:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:13:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:13:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:13:51.923 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:13:51.924 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:13:51.924 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:13:51.924 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:13:51.924 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:13:51.924 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:13:51.925 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:13:51.925 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:13:51.925 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:13:51.925 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:13:51.925 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:13:51.926 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:13:51.926 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:13:51.926 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.926 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:13:51.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.927 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:13:51.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.927 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:13:51.927 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:13:51.927 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.928 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.928 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.928 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:13:51.928 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.929 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:13:51.929 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.930 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:13:51.930 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:13:51.930 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.931 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.931 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.931 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:13:51.932 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:13:51.932 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.932 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.932 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.933 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:13:51.933 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.933 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.933 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:13:51.934 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.934 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.934 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:13:51.935 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:13:51.935 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.935 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.935 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:13:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:13:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:13:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:13:51.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.937 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:13:51.937 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:13:51.937 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:13:51.938 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:13:51.938 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:13:51.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.938 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.939 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:13:51.939 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:13:51.939 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:13:51.939 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:13:51.940 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:13:51.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.940 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.940 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:13:51.940 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:13:51.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.941 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.941 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.941 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:51.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.942 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:13:51.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:13:51.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:13:51.942 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:13:51.942 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:13:51.943 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:13:51.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:13:51.943 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:13:51.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:13:51.943 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:13:51.944 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:13:51.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 10 librados: watch_flush enter 2023-10-30T12:13:51.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 10 librados: watch_flush exit 2023-10-30T12:13:51.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 20 client.114890.objecter shutdown clearing up homeless session... 2023-10-30T12:13:51.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 10 client.114890.objecter successfully canceled tick 2023-10-30T12:13:51.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37d80392f0 msgr2=0x7f37d803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f37d80392f0 0x7f37d803b7b0 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 crypto rx=0x7f37e4036000 tx=0x7f37e4030000 comp rx=0 tx=0).stop 2023-10-30T12:13:51.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 10 monclient: shutdown 2023-10-30T12:13:51.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:51.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec097780 msgr2=0x7f37ec1ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:51.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec097780 0x7f37ec1ca370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f37e000d800 tx=0x7f37e000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37ebfff640 1 -- 172.21.15.50:0/1180257987 reap_dead start 2023-10-30T12:13:51.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 shutdown_connections 2023-10-30T12:13:51.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 --2- 172.21.15.50:0/1180257987 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37ec097780 0x7f37ec1ca370 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:51.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 >> 172.21.15.50:0/1180257987 conn(0x7f37ec0a1170 msgr2=0x7f37ec096230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:51.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 shutdown_connections 2023-10-30T12:13:51.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 -- 172.21.15.50:0/1180257987 wait complete. 2023-10-30T12:13:51.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:51.840+0000 7f37f1b8a640 1 librados: shutdown 2023-10-30T12:13:51.949 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:13:51.949 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:13:51.949 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:13:51.950 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:13:51.950 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:13:51.950 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:13:51.950 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:13:51.960 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:13:51.961 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:13:51.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:52.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.104+0000 7ffa44c10640 10 monclient: get_monmap_and_config 2023-10-30T12:13:52.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.104+0000 7ffa44c10640 10 monclient: build_initial_monmap 2023-10-30T12:13:52.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.104+0000 7ffa44c10640 10 monclient: monmap: 2023-10-30T12:13:52.107 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:52.107 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:52.107 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:52.105292+0000 2023-10-30T12:13:52.108 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:52.105292+0000 2023-10-30T12:13:52.108 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:52.108 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:52.109 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:52.109 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:52.109 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:52.109 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:52.110 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:52.110 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:52.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.104+0000 7ffa44c10640 1 Processor -- start 2023-10-30T12:13:52.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.104+0000 7ffa44c10640 1 -- start start 2023-10-30T12:13:52.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.104+0000 7ffa44c10640 10 monclient: init 2023-10-30T12:13:52.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:52.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-10-30T12:13:52.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa400a4b60 0x7ffa401326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:52.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): picked mon.noname-a con 0x7ffa400a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:13:52.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa40099a40 0x7ffa40132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:52.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): picked mon.noname-b con 0x7ffa40099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:52.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffa400a2350 0x7ffa400a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:52.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): picked mon.noname-c con 0x7ffa400a2350 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:52.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:52.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa400a3d70 con 0x7ffa400a4b60 2023-10-30T12:13:52.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:52.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa40133120 con 0x7ffa400a2350 2023-10-30T12:13:52.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:52.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa400a4f40 con 0x7ffa40099a40 2023-10-30T12:13:52.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:52.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa44c10640 10 monclient(hunting): authenticate will time out at 1677.288451s 2023-10-30T12:13:52.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3e59c640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7ffa400a4b60 msgr2=0x7ffa401326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:13:52.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3e59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa400a4b60 0x7ffa401326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:52.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa40099a40 0x7ffa40132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:52.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa40099a40 0x7ffa40132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33714/0 (socket says 172.21.15.50:33714) 2023-10-30T12:13:52.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/772574940 learned_addr learned my addr 172.21.15.50:0/772574940 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:52.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): get_auth_request con 0x7ffa40099a40 auth_method 0 2023-10-30T12:13:52.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:52.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:52.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:52.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:52.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:52.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:52.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3ed9d640 1 --2- 172.21.15.50:0/772574940 >> v2:172.21.15.118:3300/0 conn(0x7ffa400a2350 0x7ffa400a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:52.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3ed9d640 10 monclient(hunting): get_auth_request con 0x7ffa400a2350 auth_method 0 2023-10-30T12:13:52.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:52.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:52.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:52.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_done global_id 114893 payload 274 2023-10-30T12:13:52.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/772574940 >> v2:172.21.15.118:3300/0 conn(0x7ffa400a2350 msgr2=0x7ffa400a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:52.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 --2- 172.21.15.50:0/772574940 >> v2:172.21.15.118:3300/0 conn(0x7ffa400a2350 0x7ffa400a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/772574940 >> v2:172.21.15.50:3300/0 conn(0x7ffa400a4b60 msgr2=0x7ffa401326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:13:52.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 --2- 172.21.15.50:0/772574940 >> v2:172.21.15.50:3300/0 conn(0x7ffa400a4b60 0x7ffa401326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:13:52.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 monclient: found mon.noname-b 2023-10-30T12:13:52.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:52.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:52.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.105+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/772574940 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400a4150 con 0x7ffa40099a40 2023-10-30T12:13:52.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3dd9b640 1 --2- 172.21.15.50:0/772574940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa40099a40 0x7ffa40132be0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffa400a53f0 tx=0x7ffa2802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c41c8fdf4dfcab6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:52.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 1 -- 172.21.15.50:0/772574940 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28031760 con 0x7ffa40099a40 2023-10-30T12:13:52.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:52.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:52.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: dump: 2023-10-30T12:13:52.133 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:52.133 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:52.133 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.134 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.134 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:52.134 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:52.134 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:52.134 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:52.135 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:52.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:52.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: _finish_auth 0 2023-10-30T12:13:52.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:13:52.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:52.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 1 -- 172.21.15.50:0/772574940 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa280318c0 con 0x7ffa40099a40 2023-10-30T12:13:52.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa44c10640 5 monclient: authenticate success, global_id 114893 2023-10-30T12:13:52.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa44c10640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:52.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:52.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa44c10640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:52.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 1 -- 172.21.15.50:0/772574940 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28031c50 con 0x7ffa40099a40 2023-10-30T12:13:52.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:52.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa44c10640 4 set_mon_vals no callback set 2023-10-30T12:13:52.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:52.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.106+0000 7ffa3d59a640 10 monclient: dump: 2023-10-30T12:13:52.137 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:52.138 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:52.138 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.138 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.138 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:52.138 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:52.139 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:52.139 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:52.139 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:52.139 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:52.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 10 monclient: shutdown 2023-10-30T12:13:52.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:52.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 -- 172.21.15.50:0/772574940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa40099a40 msgr2=0x7ffa40132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:52.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 --2- 172.21.15.50:0/772574940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa40099a40 0x7ffa40132be0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffa400a53f0 tx=0x7ffa2802b800 comp rx=0 tx=0).stop 2023-10-30T12:13:52.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 -- 172.21.15.50:0/772574940 shutdown_connections 2023-10-30T12:13:52.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 --2- 172.21.15.50:0/772574940 >> v2:172.21.15.118:3300/0 conn(0x7ffa400a2350 0x7ffa400a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 --2- 172.21.15.50:0/772574940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa40099a40 0x7ffa40132be0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 --2- 172.21.15.50:0/772574940 >> v2:172.21.15.50:3300/0 conn(0x7ffa400a4b60 0x7ffa401326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 -- 172.21.15.50:0/772574940 >> 172.21.15.50:0/772574940 conn(0x7ffa400a8060 msgr2=0x7ffa400a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:52.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 -- 172.21.15.50:0/772574940 shutdown_connections 2023-10-30T12:13:52.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.107+0000 7ffa44c10640 1 -- 172.21.15.50:0/772574940 wait complete. 2023-10-30T12:13:52.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient: build_initial_monmap 2023-10-30T12:13:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient: monmap: 2023-10-30T12:13:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:52.143 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:52.143 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:52.143 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:52.143 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:52.143 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:52.143 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:52.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 librados: starting msgr at 2023-10-30T12:13:52.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 librados: starting objecter 2023-10-30T12:13:52.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 Processor -- start 2023-10-30T12:13:52.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 -- start start 2023-10-30T12:13:52.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 librados: setting wanted keys 2023-10-30T12:13:52.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 librados: calling monclient init 2023-10-30T12:13:52.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient: init 2023-10-30T12:13:52.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:52.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:13:52.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa400a2350 0x7ffa401cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:52.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): picked mon.noname-a con 0x7ffa400a2350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa400a4b60 0x7ffa4009e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): picked mon.noname-c con 0x7ffa400a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa401cb150 0x7ffa4009ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): picked mon.noname-b con 0x7ffa401cb150 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:52.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4011f540 con 0x7ffa400a2350 2023-10-30T12:13:52.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:52.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4011f790 con 0x7ffa401cb150 2023-10-30T12:13:52.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:52.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4011e000 con 0x7ffa400a4b60 2023-10-30T12:13:52.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa44c10640 10 monclient(hunting): authenticate will time out at 1677.291898s 2023-10-30T12:13:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa3e59c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa400a2350 msgr2=0x7ffa401cac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:13:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa3e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa400a2350 0x7ffa401cac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.108+0000 7ffa3dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa400a4b60 0x7ffa4009e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa400a4b60 0x7ffa4009e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33716/0 (socket says 172.21.15.50:33716) 2023-10-30T12:13:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/4133640887 learned_addr learned my addr 172.21.15.50:0/4133640887 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): get_auth_request con 0x7ffa400a4b60 auth_method 0 2023-10-30T12:13:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3ed9d640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa401cb150 0x7ffa4009ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient(hunting): handle_auth_done global_id 114896 payload 824 2023-10-30T12:13:52.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa401cb150 msgr2=0x7ffa4009ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:52.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa401cb150 0x7ffa4009ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/4133640887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa400a2350 msgr2=0x7ffa401cac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:52.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa400a2350 0x7ffa401cac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:13:52.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 monclient: found mon.noname-c 2023-10-30T12:13:52.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:52.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:52.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 -- 172.21.15.50:0/4133640887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa280094d0 con 0x7ffa400a4b60 2023-10-30T12:13:52.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3ed9d640 10 monclient: get_auth_request con 0x7ffa401cb150 auth_method 0 2023-10-30T12:13:52.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3ed9d640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa401cb150 0x7ffa4009ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:13:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3dd9b640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa400a4b60 0x7ffa4009e6e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffa2802b840 tx=0x7ffa280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 client.?.objecter ms_handle_connect 0x7ffa400a4b60 2023-10-30T12:13:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 1 -- 172.21.15.50:0/4133640887 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2803b800 con 0x7ffa400a4b60 2023-10-30T12:13:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: dump: 2023-10-30T12:13:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:52.156 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: _finish_auth 0 2023-10-30T12:13:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 1 -- 172.21.15.50:0/4133640887 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2803b960 con 0x7ffa400a4b60 2023-10-30T12:13:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 1 -- 172.21.15.50:0/4133640887 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2803bd10 con 0x7ffa400a4b60 2023-10-30T12:13:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa36ffd640 10 monclient: dump: 2023-10-30T12:13:52.158 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:52.158 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:52.158 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.158 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:52.158 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:52.158 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:52.159 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:52.159 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:52.159 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:52.159 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:52.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 5 monclient: authenticate success, global_id 114896 2023-10-30T12:13:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 10 monclient: _renew_subs 2023-10-30T12:13:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa3cd99640 4 set_mon_vals no callback set 2023-10-30T12:13:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa40135cb0 con 0x7ffa400a4b60 2023-10-30T12:13:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 10 client.114896.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:52.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 10 monclient: _renew_subs 2023-10-30T12:13:52.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:52.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4011eef0 con 0x7ffa400a4b60 2023-10-30T12:13:52.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.109+0000 7ffa44c10640 1 librados: init done 2023-10-30T12:13:52.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa36ffd640 1 -- 172.21.15.50:0/4133640887 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ffa280453f0 con 0x7ffa400a4b60 2023-10-30T12:13:52.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa44c10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:52.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa44c10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:52.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa44c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:52.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa40073620 con 0x7ffa400a4b60 2023-10-30T12:13:52.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa36ffd640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa200392d0 0x7ffa2003b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:52.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa36ffd640 1 -- 172.21.15.50:0/4133640887 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7ffa2802f050 con 0x7ffa400a4b60 2023-10-30T12:13:52.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa36ffd640 10 client.114896.objecter ms_dispatch 0x7ffa40000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:52.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa36ffd640 3 client.114896.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:52.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa36ffd640 3 client.114896.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:52.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.110+0000 7ffa36ffd640 20 client.114896.objecter dump_active .. 0 homeless 2023-10-30T12:13:52.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.111+0000 7ffa3e59c640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa200392d0 0x7ffa2003b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:52.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.111+0000 7ffa3e59c640 10 monclient: get_auth_request con 0x7ffa200392d0 auth_method 0 2023-10-30T12:13:52.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.111+0000 7ffa3e59c640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa200392d0 0x7ffa2003b790 secure :-1 s=READY pgs=4170 cs=0 l=1 rev1=1 crypto rx=0x7ffa30003d00 tx=0x7ffa30009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:52.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.111+0000 7ffa36ffd640 10 client.114896.objecter ms_handle_connect 0x7ffa200392d0 2023-10-30T12:13:52.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.115+0000 7ffa36ffd640 1 -- 172.21.15.50:0/4133640887 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffa280456a0 con 0x7ffa400a4b60 2023-10-30T12:13:52.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.115+0000 7ffa36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:52.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.115+0000 7ffa36ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:52.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.359+0000 7ffa44c10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:52.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.359+0000 7ffa44c10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:52.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.360+0000 7ffa44c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:52.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.360+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffa4011f8d0 con 0x7ffa400a4b60 2023-10-30T12:13:52.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.360+0000 7ffa36ffd640 1 -- 172.21.15.50:0/4133640887 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7ffa4011f8d0 con 0x7ffa400a4b60 2023-10-30T12:13:52.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.360+0000 7ffa36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:52.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.360+0000 7ffa36ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:52.363 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 10 librados: watch_flush enter 2023-10-30T12:13:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 10 librados: watch_flush exit 2023-10-30T12:13:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 20 client.114896.objecter shutdown clearing up homeless session... 2023-10-30T12:13:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 10 client.114896.objecter successfully canceled tick 2023-10-30T12:13:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa200392d0 msgr2=0x7ffa2003b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa200392d0 0x7ffa2003b790 secure :-1 s=READY pgs=4170 cs=0 l=1 rev1=1 crypto rx=0x7ffa30003d00 tx=0x7ffa30009270 comp rx=0 tx=0).stop 2023-10-30T12:13:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 10 monclient: shutdown 2023-10-30T12:13:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa400a4b60 msgr2=0x7ffa4009e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa400a4b60 0x7ffa4009e6e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffa2802b840 tx=0x7ffa280046c0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 shutdown_connections 2023-10-30T12:13:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa200392d0 0x7ffa2003b790 unknown :-1 s=CLOSED pgs=4170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa401cb150 0x7ffa4009ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa400a4b60 0x7ffa4009e6e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 --2- 172.21.15.50:0/4133640887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa400a2350 0x7ffa401cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 >> 172.21.15.50:0/4133640887 conn(0x7ffa400a8060 msgr2=0x7ffa40131bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 shutdown_connections 2023-10-30T12:13:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 -- 172.21.15.50:0/4133640887 wait complete. 2023-10-30T12:13:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:52.362+0000 7ffa44c10640 1 librados: shutdown 2023-10-30T12:13:55.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:55.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.490+0000 7f1a58e56640 10 monclient: get_monmap_and_config 2023-10-30T12:13:55.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.490+0000 7f1a58e56640 10 monclient: build_initial_monmap 2023-10-30T12:13:55.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.490+0000 7f1a58e56640 10 monclient: monmap: 2023-10-30T12:13:55.492 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:55.491446+0000 2023-10-30T12:13:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:55.491446+0000 2023-10-30T12:13:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:55.495 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:55.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 1 Processor -- start 2023-10-30T12:13:55.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 1 -- start start 2023-10-30T12:13:55.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient: init 2023-10-30T12:13:55.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:55.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:13:55.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a5409f5b0 0x7f1a54099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:55.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5409f5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1a54096080 0x7f1a54096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient(hunting): picked mon.noname-c con 0x7f1a54096080 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a540969a0 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a54098810 con 0x7f1a540969a0 2023-10-30T12:13:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5406aef0 con 0x7f1a54096080 2023-10-30T12:13:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5409a1a0 con 0x7f1a5409f5b0 2023-10-30T12:13:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a58e56640 10 monclient(hunting): authenticate will time out at 1680.674647s 2023-10-30T12:13:55.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a5409f5b0 0x7f1a54099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a5409f5b0 0x7f1a54099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33726/0 (socket says 172.21.15.50:33726) 2023-10-30T12:13:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 1 -- 172.21.15.50:0/1816465824 learned_addr learned my addr 172.21.15.50:0/1816465824 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 10 monclient(hunting): get_auth_request con 0x7f1a5409f5b0 auth_method 0 2023-10-30T12:13:55.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:55.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.491+0000 7f1a5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 10 monclient(hunting): handle_auth_done global_id 114899 payload 274 2023-10-30T12:13:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a4bfff640 1 --2- 172.21.15.50:0/1816465824 >> v2:172.21.15.118:3300/0 conn(0x7f1a54096080 0x7f1a54096460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 1 -- 172.21.15.50:0/1816465824 >> v2:172.21.15.118:3300/0 conn(0x7f1a54096080 msgr2=0x7f1a54096460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 1 --2- 172.21.15.50:0/1816465824 >> v2:172.21.15.118:3300/0 conn(0x7f1a54096080 0x7f1a54096460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 1 -- 172.21.15.50:0/1816465824 >> v1:172.21.15.118:6789/0 conn(0x7f1a540969a0 legacy=0x7f1a5409cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 10 monclient: _finish_hunting 0 2023-10-30T12:13:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 1 monclient: found mon.noname-b 2023-10-30T12:13:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 1 -- 172.21.15.50:0/1816465824 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54098af0 con 0x7f1a5409f5b0 2023-10-30T12:13:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.492+0000 7f1a5259c640 1 --2- 172.21.15.50:0/1816465824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a5409f5b0 0x7f1a54099b80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009880 tx=0x7f1a4402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f76a057ac5877e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 1 -- 172.21.15.50:0/1816465824 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4402f050 con 0x7f1a5409f5b0 2023-10-30T12:13:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: dump: 2023-10-30T12:13:55.509 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:55.510 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:55.511 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:55.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: _finish_auth 0 2023-10-30T12:13:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:13:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 1 -- 172.21.15.50:0/1816465824 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a44004230 con 0x7f1a5409f5b0 2023-10-30T12:13:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 5 monclient: authenticate success, global_id 114899 2023-10-30T12:13:55.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 1 -- 172.21.15.50:0/1816465824 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44003610 con 0x7f1a5409f5b0 2023-10-30T12:13:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:55.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 4 set_mon_vals no callback set 2023-10-30T12:13:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a51d9b640 10 monclient: dump: 2023-10-30T12:13:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:55.514 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:55.516 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 10 monclient: shutdown 2023-10-30T12:13:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 -- 172.21.15.50:0/1816465824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a5409f5b0 msgr2=0x7f1a54099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:55.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 --2- 172.21.15.50:0/1816465824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a5409f5b0 0x7f1a54099b80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009880 tx=0x7f1a4402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 -- 172.21.15.50:0/1816465824 shutdown_connections 2023-10-30T12:13:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 --2- 172.21.15.50:0/1816465824 >> v2:172.21.15.118:3300/0 conn(0x7f1a54096080 0x7f1a54096460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 --2- 172.21.15.50:0/1816465824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a5409f5b0 0x7f1a54099b80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 -- 172.21.15.50:0/1816465824 >> 172.21.15.50:0/1816465824 conn(0x7f1a540a05c0 msgr2=0x7f1a5409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:55.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 -- 172.21.15.50:0/1816465824 shutdown_connections 2023-10-30T12:13:55.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.493+0000 7f1a58e56640 1 -- 172.21.15.50:0/1816465824 wait complete. 2023-10-30T12:13:55.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 10 monclient: build_initial_monmap 2023-10-30T12:13:55.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 10 monclient: monmap: 2023-10-30T12:13:55.518 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:55.518 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:55.519 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:55.519 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:55.519 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:55.519 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:55.519 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:55.520 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:55.520 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:55.520 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:55.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 1 librados: starting msgr at 2023-10-30T12:13:55.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 1 librados: starting objecter 2023-10-30T12:13:55.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 1 Processor -- start 2023-10-30T12:13:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 1 -- start start 2023-10-30T12:13:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 1 librados: setting wanted keys 2023-10-30T12:13:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 1 librados: calling monclient init 2023-10-30T12:13:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.494+0000 7f1a58e56640 10 monclient: init 2023-10-30T12:13:55.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:13:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a54096080 0x7f1a54095450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): picked mon.noname-a con 0x7f1a54096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a540969a0 0x7f1a54095990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:55.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): picked mon.noname-c con 0x7f1a540969a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a5409f5b0 0x7f1a5408afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5409f5b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5408d920 con 0x7f1a54096080 2023-10-30T12:13:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54095ed0 con 0x7f1a5409f5b0 2023-10-30T12:13:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54072010 con 0x7f1a540969a0 2023-10-30T12:13:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a5259c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a54096080 msgr2=0x7f1a54095450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:13:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a58e56640 10 monclient(hunting): authenticate will time out at 1680.678314s 2023-10-30T12:13:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a5259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a54096080 0x7f1a54095450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a540969a0 0x7f1a54095990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a540969a0 0x7f1a54095990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33738/0 (socket says 172.21.15.50:33738) 2023-10-30T12:13:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 -- 172.21.15.50:0/4006574615 learned_addr learned my addr 172.21.15.50:0/4006574615 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): get_auth_request con 0x7f1a540969a0 auth_method 0 2023-10-30T12:13:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:55.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:55.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a52d9d640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a5409f5b0 0x7f1a5408afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a52d9d640 10 monclient(hunting): get_auth_request con 0x7f1a5409f5b0 auth_method 0 2023-10-30T12:13:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a52d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a52d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:55.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient(hunting): handle_auth_done global_id 114902 payload 824 2023-10-30T12:13:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 -- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a5409f5b0 msgr2=0x7f1a5408afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a5409f5b0 0x7f1a5408afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 -- 172.21.15.50:0/4006574615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a54096080 msgr2=0x7f1a54095450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a54096080 0x7f1a54095450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 monclient: found mon.noname-c 2023-10-30T12:13:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 -- 172.21.15.50:0/4006574615 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a44009460 con 0x7f1a540969a0 2023-10-30T12:13:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a52d9d640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a5409f5b0 0x7f1a5408afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:13:55.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4bfff640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a540969a0 0x7f1a54095990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00c7f0 tx=0x7f1a3c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4affd640 10 client.?.objecter ms_handle_connect 0x7f1a540969a0 2023-10-30T12:13:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.495+0000 7f1a4affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 1 -- 172.21.15.50:0/4006574615 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c004220 con 0x7f1a540969a0 2023-10-30T12:13:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:55.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: dump: 2023-10-30T12:13:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:55.532 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:55.532 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: _finish_auth 0 2023-10-30T12:13:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: _check_auth_tickets 2023-10-30T12:13:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 1 -- 172.21.15.50:0/4006574615 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3c004380 con 0x7f1a540969a0 2023-10-30T12:13:55.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 5 monclient: authenticate success, global_id 114902 2023-10-30T12:13:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 10 monclient: _renew_subs 2023-10-30T12:13:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a541bb960 con 0x7f1a540969a0 2023-10-30T12:13:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:55.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 1 -- 172.21.15.50:0/4006574615 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c007840 con 0x7f1a540969a0 2023-10-30T12:13:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 10 monclient: dump: 2023-10-30T12:13:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:55.535 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:55.536 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:55.537 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a5159a640 4 set_mon_vals no callback set 2023-10-30T12:13:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 10 client.114902.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:55.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 10 monclient: _renew_subs 2023-10-30T12:13:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a54070b20 con 0x7f1a540969a0 2023-10-30T12:13:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a58e56640 1 librados: init done 2023-10-30T12:13:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.496+0000 7f1a4affd640 1 -- 172.21.15.50:0/4006574615 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1a3c007bb0 con 0x7f1a540969a0 2023-10-30T12:13:55.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a58e56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a58e56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a58e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a541bb430 con 0x7f1a540969a0 2023-10-30T12:13:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a4affd640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a340392a0 0x7f1a3403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a4affd640 1 -- 172.21.15.50:0/4006574615 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f1a3c04ab10 con 0x7f1a540969a0 2023-10-30T12:13:55.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a4affd640 10 client.114902.objecter ms_dispatch 0x7f1a54000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a4affd640 3 client.114902.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a4affd640 3 client.114902.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a4affd640 20 client.114902.objecter dump_active .. 0 homeless 2023-10-30T12:13:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a5259c640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a340392a0 0x7f1a3403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:55.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a5259c640 10 monclient: get_auth_request con 0x7f1a340392a0 auth_method 0 2023-10-30T12:13:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a4affd640 10 client.114902.objecter ms_handle_connect 0x7f1a340392a0 2023-10-30T12:13:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.497+0000 7f1a5259c640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a340392a0 0x7f1a3403b760 secure :-1 s=READY pgs=4171 cs=0 l=1 rev1=1 crypto rx=0x7f1a44035000 tx=0x7f1a44008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.501+0000 7f1a4affd640 1 -- 172.21.15.50:0/4006574615 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a3c003990 con 0x7f1a540969a0 2023-10-30T12:13:55.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.501+0000 7f1a4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:55.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.501+0000 7f1a4affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:55.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.744+0000 7f1a58e56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:55.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.744+0000 7f1a58e56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.744+0000 7f1a58e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.744+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1a54099720 con 0x7f1a540969a0 2023-10-30T12:13:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.745+0000 7f1a4affd640 1 -- 172.21.15.50:0/4006574615 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f1a3c017b00 con 0x7f1a540969a0 2023-10-30T12:13:55.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.745+0000 7f1a4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:55.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.745+0000 7f1a4affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:55.748 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:55.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.746+0000 7f1a58e56640 10 librados: watch_flush enter 2023-10-30T12:13:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.746+0000 7f1a58e56640 10 librados: watch_flush exit 2023-10-30T12:13:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.746+0000 7f1a58e56640 20 client.114902.objecter shutdown clearing up homeless session... 2023-10-30T12:13:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.746+0000 7f1a58e56640 10 client.114902.objecter successfully canceled tick 2023-10-30T12:13:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.746+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a340392a0 msgr2=0x7f1a3403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:55.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.746+0000 7f1a58e56640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a340392a0 0x7f1a3403b760 secure :-1 s=READY pgs=4171 cs=0 l=1 rev1=1 crypto rx=0x7f1a44035000 tx=0x7f1a44008030 comp rx=0 tx=0).stop 2023-10-30T12:13:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 10 monclient: shutdown 2023-10-30T12:13:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a540969a0 msgr2=0x7f1a54095990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:55.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a540969a0 0x7f1a54095990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00c7f0 tx=0x7f1a3c00ccb0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 shutdown_connections 2023-10-30T12:13:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a340392a0 0x7f1a3403b760 unknown :-1 s=CLOSED pgs=4171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a5409f5b0 0x7f1a5408afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a540969a0 0x7f1a54095990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 --2- 172.21.15.50:0/4006574615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a54096080 0x7f1a54095450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 >> 172.21.15.50:0/4006574615 conn(0x7f1a540a05c0 msgr2=0x7f1a541bbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 shutdown_connections 2023-10-30T12:13:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 -- 172.21.15.50:0/4006574615 wait complete. 2023-10-30T12:13:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:55.747+0000 7f1a58e56640 1 librados: shutdown 2023-10-30T12:13:58.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:13:58.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.875+0000 7fa699d2c640 10 monclient: get_monmap_and_config 2023-10-30T12:13:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.875+0000 7fa699d2c640 10 monclient: build_initial_monmap 2023-10-30T12:13:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.875+0000 7fa699d2c640 10 monclient: monmap: 2023-10-30T12:13:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:13:58.876356+0000 2023-10-30T12:13:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:13:58.876356+0000 2023-10-30T12:13:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:13:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:13:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:13:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:13:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:13:58.879 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 1 Processor -- start 2023-10-30T12:13:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 1 -- start start 2023-10-30T12:13:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient: init 2023-10-30T12:13:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:13:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa6940a4b60 0x7fa6941326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6940a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:58.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa694099a40 0x7fa694132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:58.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient(hunting): picked mon.noname-c con 0x7fa694099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:13:58.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6940a2350 addr v1:172.21.15.118:6789/0 2023-10-30T12:13:58.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa694133120 con 0x7fa6940a2350 2023-10-30T12:13:58.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6940a3d70 con 0x7fa694099a40 2023-10-30T12:13:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6940a4f40 con 0x7fa6940a4b60 2023-10-30T12:13:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa699d2c640 10 monclient(hunting): authenticate will time out at 1684.059578s 2023-10-30T12:13:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa6940a4b60 0x7fa6941326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa6940a4b60 0x7fa6941326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57132/0 (socket says 172.21.15.50:57132) 2023-10-30T12:13:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 1 -- 172.21.15.50:0/860473411 learned_addr learned my addr 172.21.15.50:0/860473411 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 10 monclient(hunting): get_auth_request con 0x7fa6940a4b60 auth_method 0 2023-10-30T12:13:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa6937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa692ffd640 1 --2- 172.21.15.50:0/860473411 >> v2:172.21.15.118:3300/0 conn(0x7fa694099a40 0x7fa694132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:58.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa692ffd640 10 monclient(hunting): get_auth_request con 0x7fa694099a40 auth_method 0 2023-10-30T12:13:58.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa692ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:58.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa692ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:58.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.876+0000 7fa692ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:58.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 10 monclient(hunting): handle_auth_done global_id 114905 payload 274 2023-10-30T12:13:58.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 1 -- 172.21.15.50:0/860473411 >> v2:172.21.15.118:3300/0 conn(0x7fa694099a40 msgr2=0x7fa694132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:13:58.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 1 --2- 172.21.15.50:0/860473411 >> v2:172.21.15.118:3300/0 conn(0x7fa694099a40 0x7fa694132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:58.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 1 -- 172.21.15.50:0/860473411 >> v1:172.21.15.118:6789/0 conn(0x7fa6940a2350 legacy=0x7fa6940a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:58.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 10 monclient: _finish_hunting 0 2023-10-30T12:13:58.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 1 monclient: found mon.noname-b 2023-10-30T12:13:58.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:58.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:13:58.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 1 -- 172.21.15.50:0/860473411 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940a4150 con 0x7fa6940a4b60 2023-10-30T12:13:58.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6937fe640 1 --2- 172.21.15.50:0/860473411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa6940a4b60 0x7fa6941326a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa684009870 tx=0x7fa68402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34c64adfb9a2fe2f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:58.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6927fc640 1 -- 172.21.15.50:0/860473411 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6840317c0 con 0x7fa6940a4b60 2023-10-30T12:13:58.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.877+0000 7fa6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:58.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:13:58.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: dump: 2023-10-30T12:13:58.895 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:58.895 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:58.896 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.896 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.896 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:58.896 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:58.896 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:58.896 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:58.897 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:58.897 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:58.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: _finish_auth 0 2023-10-30T12:13:58.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: _check_auth_tickets 2023-10-30T12:13:58.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:58.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 1 -- 172.21.15.50:0/860473411 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa684031920 con 0x7fa6940a4b60 2023-10-30T12:13:58.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 5 monclient: authenticate success, global_id 114905 2023-10-30T12:13:58.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:13:58.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:58.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 10 monclient: get_monmap_and_config success 2023-10-30T12:13:58.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 1 -- 172.21.15.50:0/860473411 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684031cb0 con 0x7fa6940a4b60 2023-10-30T12:13:58.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:58.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 4 set_mon_vals no callback set 2023-10-30T12:13:58.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:58.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa6927fc640 10 monclient: dump: 2023-10-30T12:13:58.899 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:58.900 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:58.900 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.900 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.900 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:58.900 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:58.900 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:58.901 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:58.901 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:58.901 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:58.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 10 monclient: shutdown 2023-10-30T12:13:58.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:58.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 -- 172.21.15.50:0/860473411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa6940a4b60 msgr2=0x7fa6941326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:58.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 --2- 172.21.15.50:0/860473411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa6940a4b60 0x7fa6941326a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa684009870 tx=0x7fa68402b860 comp rx=0 tx=0).stop 2023-10-30T12:13:58.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 -- 172.21.15.50:0/860473411 shutdown_connections 2023-10-30T12:13:58.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 --2- 172.21.15.50:0/860473411 >> v2:172.21.15.118:3300/0 conn(0x7fa694099a40 0x7fa694132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:58.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 --2- 172.21.15.50:0/860473411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa6940a4b60 0x7fa6941326a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:58.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 -- 172.21.15.50:0/860473411 >> 172.21.15.50:0/860473411 conn(0x7fa6940a8060 msgr2=0x7fa6940a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:58.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 -- 172.21.15.50:0/860473411 shutdown_connections 2023-10-30T12:13:58.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.878+0000 7fa699d2c640 1 -- 172.21.15.50:0/860473411 wait complete. 2023-10-30T12:13:58.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 10 monclient: build_initial_monmap 2023-10-30T12:13:58.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 10 monclient: monmap: 2023-10-30T12:13:58.904 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:13:58.904 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:13:58.904 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:13:58.904 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:13:58.904 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:13:58.904 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:58.905 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:13:58.905 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:13:58.905 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:13:58.905 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:58.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 1 librados: starting msgr at 2023-10-30T12:13:58.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 1 librados: starting objecter 2023-10-30T12:13:58.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 1 Processor -- start 2023-10-30T12:13:58.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 1 -- start start 2023-10-30T12:13:58.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 1 librados: setting wanted keys 2023-10-30T12:13:58.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 1 librados: calling monclient init 2023-10-30T12:13:58.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.879+0000 7fa699d2c640 10 monclient: init 2023-10-30T12:13:58.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient: _reopen_session rank -1 2023-10-30T12:13:58.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:13:58.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6940a2350 0x7fa6941ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:58.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6940a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:13:58.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6940a4b60 0x7fa694131040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:58.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6940a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:13:58.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa694131580 0x7fa694121a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:58.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): picked mon.noname-c con 0x7fa694131580 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:13:58.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:58.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941254b0 con 0x7fa6940a4b60 2023-10-30T12:13:58.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:58.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa694124060 con 0x7fa6940a2350 2023-10-30T12:13:58.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:13:58.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6940a18d0 con 0x7fa694131580 2023-10-30T12:13:58.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): _renew_subs 2023-10-30T12:13:58.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa699d2c640 10 monclient(hunting): authenticate will time out at 1684.063362s 2023-10-30T12:13:58.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa692ffd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6940a4b60 msgr2=0x7fa694131040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:13:58.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa692ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6940a4b60 0x7fa694131040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:13:58.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa694131580 0x7fa694121a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:58.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa694131580 0x7fa694121a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57142/0 (socket says 172.21.15.50:57142) 2023-10-30T12:13:58.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 -- 172.21.15.50:0/1486269753 learned_addr learned my addr 172.21.15.50:0/1486269753 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:13:58.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): get_auth_request con 0x7fa694131580 auth_method 0 2023-10-30T12:13:58.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:58.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:58.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:58.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6940a2350 0x7fa6941ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:58.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 10 monclient(hunting): get_auth_request con 0x7fa6940a2350 auth_method 0 2023-10-30T12:13:58.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:13:58.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:13:58.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:13:58.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:58.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:58.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:58.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:13:58.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:13:58.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa6937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:13:58.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient(hunting): handle_auth_done global_id 114908 payload 824 2023-10-30T12:13:58.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 -- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6940a2350 msgr2=0x7fa6941ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:58.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6940a2350 0x7fa6941ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:58.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 -- 172.21.15.50:0/1486269753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6940a4b60 msgr2=0x7fa694131040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:13:58.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6940a4b60 0x7fa694131040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:58.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient: _finish_hunting 0 2023-10-30T12:13:58.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 monclient: found mon.noname-c 2023-10-30T12:13:58.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:13:58.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:13:58.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 -- 172.21.15.50:0/1486269753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6840094d0 con 0x7fa694131580 2023-10-30T12:13:58.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa693fff640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa694131580 0x7fa694121a00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00c7a0 tx=0x7fa68c00cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:58.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa67bfff640 10 client.?.objecter ms_handle_connect 0x7fa694131580 2023-10-30T12:13:58.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.880+0000 7fa67bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:13:58.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 1 -- 172.21.15.50:0/1486269753 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c003b70 con 0x7fa694131580 2023-10-30T12:13:58.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:58.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:13:58.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: dump: 2023-10-30T12:13:58.917 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:58.917 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:58.917 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.917 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.917 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:58.918 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:58.918 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:58.918 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:58.918 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:58.918 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:58.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: _finish_auth 0 2023-10-30T12:13:58.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:13:58.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:13:58.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 1 -- 172.21.15.50:0/1486269753 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c003cf0 con 0x7fa694131580 2023-10-30T12:13:58.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:13:58.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 1 -- 172.21.15.50:0/1486269753 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c003ea0 con 0x7fa694131580 2023-10-30T12:13:58.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 5 monclient: authenticate success, global_id 114908 2023-10-30T12:13:58.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 10 monclient: _renew_subs 2023-10-30T12:13:58.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:58.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6940a09a0 con 0x7fa694131580 2023-10-30T12:13:58.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:13:58.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:13:58.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa67bfff640 10 monclient: dump: 2023-10-30T12:13:58.921 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:13:58.921 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:13:58.921 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.921 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:13:58.922 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:13:58.922 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:13:58.922 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:13:58.922 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:13:58.922 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:13:58.923 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:13:58.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa691ffb640 4 set_mon_vals no callback set 2023-10-30T12:13:58.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 10 client.114908.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:13:58.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 10 monclient: _renew_subs 2023-10-30T12:13:58.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:58.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6941251d0 con 0x7fa694131580 2023-10-30T12:13:58.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.881+0000 7fa699d2c640 1 librados: init done 2023-10-30T12:13:58.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa67bfff640 1 -- 172.21.15.50:0/1486269753 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa68c00aa90 con 0x7fa694131580 2023-10-30T12:13:58.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa67bfff640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6800392a0 0x7fa68003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:13:58.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa699d2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:58.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa699d2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:58.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa699d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:58.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa67bfff640 1 -- 172.21.15.50:0/1486269753 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(421..421 src has 1..421) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fa68c04a9e0 con 0x7fa694131580 2023-10-30T12:13:58.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa67bfff640 10 client.114908.objecter ms_dispatch 0x7fa694000b90 osd_map(421..421 src has 1..421) v4 2023-10-30T12:13:58.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa67bfff640 3 client.114908.objecter handle_osd_map got epochs [421,421] > 0 2023-10-30T12:13:58.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa67bfff640 3 client.114908.objecter handle_osd_map decoding full epoch 421 2023-10-30T12:13:58.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6941255f0 con 0x7fa694131580 2023-10-30T12:13:58.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa67bfff640 20 client.114908.objecter dump_active .. 0 homeless 2023-10-30T12:13:58.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa6937fe640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6800392a0 0x7fa68003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:13:58.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.882+0000 7fa6937fe640 10 monclient: get_auth_request con 0x7fa6800392a0 auth_method 0 2023-10-30T12:13:58.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.883+0000 7fa6937fe640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6800392a0 0x7fa68003b760 secure :-1 s=READY pgs=4172 cs=0 l=1 rev1=1 crypto rx=0x7fa684036000 tx=0x7fa68403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:13:58.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.883+0000 7fa67bfff640 10 client.114908.objecter ms_handle_connect 0x7fa6800392a0 2023-10-30T12:13:58.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.887+0000 7fa67bfff640 1 -- 172.21.15.50:0/1486269753 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa68c00ad40 con 0x7fa694131580 2023-10-30T12:13:58.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.887+0000 7fa67bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:13:58.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:58.887+0000 7fa67bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:13:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.132+0000 7fa699d2c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:13:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.132+0000 7fa699d2c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.132+0000 7fa699d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:13:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.132+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa69404e160 con 0x7fa694131580 2023-10-30T12:13:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.133+0000 7fa67bfff640 1 -- 172.21.15.50:0/1486269753 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7fa68c03e620 con 0x7fa694131580 2023-10-30T12:13:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.133+0000 7fa67bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:13:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.133+0000 7fa67bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:13:59.136 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":130,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:13:59.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 10 librados: watch_flush enter 2023-10-30T12:13:59.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 10 librados: watch_flush exit 2023-10-30T12:13:59.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 20 client.114908.objecter shutdown clearing up homeless session... 2023-10-30T12:13:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 10 client.114908.objecter successfully canceled tick 2023-10-30T12:13:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6800392a0 msgr2=0x7fa68003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6800392a0 0x7fa68003b760 secure :-1 s=READY pgs=4172 cs=0 l=1 rev1=1 crypto rx=0x7fa684036000 tx=0x7fa68403d040 comp rx=0 tx=0).stop 2023-10-30T12:13:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 10 monclient: shutdown 2023-10-30T12:13:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:13:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.134+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa694131580 msgr2=0x7fa694121a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:13:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa694131580 0x7fa694121a00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00c7a0 tx=0x7fa68c00cc60 comp rx=0 tx=0).stop 2023-10-30T12:13:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 shutdown_connections 2023-10-30T12:13:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa6800392a0 0x7fa68003b760 unknown :-1 s=CLOSED pgs=4172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa694131580 0x7fa694121a00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6940a4b60 0x7fa694131040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 --2- 172.21.15.50:0/1486269753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa6940a2350 0x7fa6941ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:13:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 >> 172.21.15.50:0/1486269753 conn(0x7fa6940a8060 msgr2=0x7fa69409dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:13:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 shutdown_connections 2023-10-30T12:13:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 -- 172.21.15.50:0/1486269753 wait complete. 2023-10-30T12:13:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:13:59.135+0000 7fa699d2c640 1 librados: shutdown 2023-10-30T12:14:02.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:14:02.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.262+0000 7f92efd97640 10 monclient: get_monmap_and_config 2023-10-30T12:14:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.262+0000 7f92efd97640 10 monclient: build_initial_monmap 2023-10-30T12:14:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.262+0000 7f92efd97640 10 monclient: monmap: 2023-10-30T12:14:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:02.263380+0000 2023-10-30T12:14:02.265 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:02.263380+0000 2023-10-30T12:14:02.265 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:02.265 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:02.265 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:02.265 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:02.266 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.262+0000 7f92efd97640 1 Processor -- start 2023-10-30T12:14:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 1 -- start start 2023-10-30T12:14:02.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient: init 2023-10-30T12:14:02.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:02.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T12:14:02.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:02.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient(hunting): picked mon.noname-b con 0x7f92e8098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92e8098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 0x7f92e8139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient(hunting): picked mon.noname-a con 0x7f92e81379f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e809ad30 con 0x7f92e8098900 2023-10-30T12:14:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e80993f0 con 0x7f92e81379f0 2023-10-30T12:14:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e809c460 con 0x7f92e8098520 2023-10-30T12:14:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92efd97640 10 monclient(hunting): authenticate will time out at 1687.446709s 2023-10-30T12:14:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92ee334640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 msgr2=0x7f92e8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92ee334640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 0x7f92e8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57144/0 (socket says 172.21.15.50:57144) 2023-10-30T12:14:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 learned_addr learned my addr 172.21.15.50:0/1003373363 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 10 monclient(hunting): get_auth_request con 0x7f92e8098520 auth_method 0 2023-10-30T12:14:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.263+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.264+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.464+0000 7f92ee334640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 msgr2=0x7f92e8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.464+0000 7f92ee334640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 0x7f92e8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 10 monclient(hunting): get_auth_request con 0x7f92e8098520 auth_method 0 2023-10-30T12:14:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:02.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:02.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:02.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.465+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:02.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.865+0000 7f92ee334640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 msgr2=0x7f92e8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:02.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.865+0000 7f92ee334640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 0x7f92e8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:02.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:02.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 10 monclient(hunting): get_auth_request con 0x7f92e8098520 auth_method 0 2023-10-30T12:14:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:02.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:02.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:02.866+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:03.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.666+0000 7f92ee334640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 msgr2=0x7f92e8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:03.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.666+0000 7f92ee334640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 0x7f92e8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:03.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:03.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 10 monclient(hunting): get_auth_request con 0x7f92e8098520 auth_method 0 2023-10-30T12:14:03.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:03.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:03.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:03.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:03.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:03.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:03.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:03.667+0000 7f92edb33640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient(hunting): tick 2023-10-30T12:14:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:14:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 monclient(hunting): continuing hunt 2023-10-30T12:14:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:14:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 msgr2=0x7f92e809bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e8098520 0x7f92e809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 msgr2=0x7f92e8139de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.50:3300/0 conn(0x7f92e81379f0 0x7f92e8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 -- 172.21.15.50:0/1003373363 >> v1:172.21.15.118:6789/0 conn(0x7f92e8098900 legacy=0x7f92e8098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient: _add_conns ranks=[0,4] 2023-10-30T12:14:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92cc0036e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f92cc009620 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 -- 172.21.15.50:0/1003373363 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e809b260 con 0x7f92cc0036e0 2023-10-30T12:14:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 1 -- 172.21.15.50:0/1003373363 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e80993f0 con 0x7f92cc009620 2023-10-30T12:14:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92df7fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> v1:172.21.15.50:6789/0 conn(0x7f92cc0036e0 legacy=0x7f92cc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:14:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:05.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 10 monclient(hunting): get_auth_request con 0x7f92cc009620 auth_method 0 2023-10-30T12:14:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 msgr2=0x7f92cc00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 msgr2=0x7f92cc00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.264+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> v1:172.21.15.50:6789/0 conn(0x7f92cc0036e0 legacy=0x7f92cc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:14:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 10 monclient(hunting): get_auth_request con 0x7f92cc009620 auth_method 0 2023-10-30T12:14:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:05.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:05.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:05.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 msgr2=0x7f92cc00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:05.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 msgr2=0x7f92cc00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:05.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:05.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.465+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:05.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> v1:172.21.15.50:6789/0 conn(0x7f92cc0036e0 legacy=0x7f92cc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:14:05.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:05.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 10 monclient(hunting): get_auth_request con 0x7f92cc009620 auth_method 0 2023-10-30T12:14:05.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:05.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:05.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:05.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 msgr2=0x7f92cc00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:05.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 msgr2=0x7f92cc00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:05.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:05.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:05.866+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92edb33640 1 -- 172.21.15.50:0/1003373363 >> v1:172.21.15.50:6789/0 conn(0x7f92cc0036e0 legacy=0x7f92cc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:14:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> v2:172.21.15.118:3300/0 conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 10 monclient(hunting): get_auth_request con 0x7f92cc009620 auth_method 0 2023-10-30T12:14:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.667+0000 7f92ed332640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.668+0000 7f92ed332640 10 monclient(hunting): handle_auth_done global_id 184162 payload 274 2023-10-30T12:14:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.668+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 >> v1:172.21.15.50:6789/0 conn(0x7f92cc0036e0 legacy=0x7f92cc005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.668+0000 7f92ed332640 10 monclient: _finish_hunting 0 2023-10-30T12:14:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.668+0000 7f92ed332640 1 monclient: found mon.noname-c 2023-10-30T12:14:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.668+0000 7f92ed332640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.668+0000 7f92ed332640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:14:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.668+0000 7f92ed332640 1 -- 172.21.15.50:0/1003373363 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc00d1f0 con 0x7f92cc009620 2023-10-30T12:14:06.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ed332640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc009620 0x7f92cc00bae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f92d8007c30 tx=0x7f92d802bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b903fb0c84c6747 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:06.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 1 -- 172.21.15.50:0/1003373363 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d80052e0 con 0x7f92cc009620 2023-10-30T12:14:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: dump: 2023-10-30T12:14:06.679 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:06.681 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:06.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:06.681 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:06.681 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:06.681 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:06.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: _finish_auth 0 2023-10-30T12:14:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: _check_auth_tickets 2023-10-30T12:14:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 1 -- 172.21.15.50:0/1003373363 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d8005440 con 0x7f92cc009620 2023-10-30T12:14:06.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 1 -- 172.21.15.50:0/1003373363 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d802c980 con 0x7f92cc009620 2023-10-30T12:14:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92ecb31640 10 monclient: dump: 2023-10-30T12:14:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:06.684 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:06.685 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92efd97640 5 monclient: authenticate success, global_id 184162 2023-10-30T12:14:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92efd97640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.669+0000 7f92efd97640 4 set_mon_vals no callback set 2023-10-30T12:14:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 10 monclient: shutdown 2023-10-30T12:14:06.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 1 -- 172.21.15.50:0/1003373363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc009620 msgr2=0x7f92cc00bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc009620 0x7f92cc00bae0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f92d8007c30 tx=0x7f92d802bde0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92ee334640 1 -- 172.21.15.50:0/1003373363 reap_dead start 2023-10-30T12:14:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 1 -- 172.21.15.50:0/1003373363 shutdown_connections 2023-10-30T12:14:06.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 1 --2- 172.21.15.50:0/1003373363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92cc009620 0x7f92cc00bae0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 1 -- 172.21.15.50:0/1003373363 >> 172.21.15.50:0/1003373363 conn(0x7f92e80a05c0 msgr2=0x7f92e809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 1 -- 172.21.15.50:0/1003373363 shutdown_connections 2023-10-30T12:14:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 1 -- 172.21.15.50:0/1003373363 wait complete. 2023-10-30T12:14:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 10 monclient: build_initial_monmap 2023-10-30T12:14:06.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.670+0000 7f92efd97640 10 monclient: monmap: 2023-10-30T12:14:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:06.690 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:06.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 librados: starting msgr at 2023-10-30T12:14:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 librados: starting objecter 2023-10-30T12:14:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 Processor -- start 2023-10-30T12:14:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 -- start start 2023-10-30T12:14:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 librados: setting wanted keys 2023-10-30T12:14:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 librados: calling monclient init 2023-10-30T12:14:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient: init 2023-10-30T12:14:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:14:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 0x7f92e8093550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): picked mon.noname-c con 0x7f92cc009620 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 0x7f92e8137310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): picked mon.noname-a con 0x7f92e8136ed0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 0x7f92e808f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): picked mon.noname-b con 0x7f92e808ef90 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e8092b50 con 0x7f92e8136ed0 2023-10-30T12:14:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:06.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e8071e80 con 0x7f92e808ef90 2023-10-30T12:14:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e80922b0 con 0x7f92cc009620 2023-10-30T12:14:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92efd97640 10 monclient(hunting): authenticate will time out at 1691.854939s 2023-10-30T12:14:06.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92ed332640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 msgr2=0x7f92e8137310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:06.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.671+0000 7f92ed332640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 0x7f92e8137310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:06.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 0x7f92e8093550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:06.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 0x7f92e8093550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57182/0 (socket says 172.21.15.50:57182) 2023-10-30T12:14:06.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 learned_addr learned my addr 172.21.15.50:0/3646206259 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:06.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): get_auth_request con 0x7f92cc009620 auth_method 0 2023-10-30T12:14:06.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:06.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:06.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:06.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 0x7f92e808f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:06.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:06.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:06.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:06.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 10 monclient(hunting): get_auth_request con 0x7f92e808ef90 auth_method 0 2023-10-30T12:14:06.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:06.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:06.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:06.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:06.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:06.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92ee334640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:06.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient(hunting): handle_auth_done global_id 114911 payload 824 2023-10-30T12:14:06.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 msgr2=0x7f92e808f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:06.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 0x7f92e808f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 msgr2=0x7f92e8137310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:06.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 0x7f92e8137310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient: _finish_hunting 0 2023-10-30T12:14:06.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 monclient: found mon.noname-c 2023-10-30T12:14:06.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:06.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:06.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d800f040 con 0x7f92cc009620 2023-10-30T12:14:06.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 0x7f92e8093550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c5d0 tx=0x7f92e4006a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:06.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 client.?.objecter ms_handle_connect 0x7f92cc009620 2023-10-30T12:14:06.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:06.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4013560 con 0x7f92cc009620 2023-10-30T12:14:06.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:06.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:06.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: dump: 2023-10-30T12:14:06.702 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:06.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:06.702 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:06.704 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: _finish_auth 0 2023-10-30T12:14:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:14:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e400c040 con 0x7f92cc009620 2023-10-30T12:14:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92efd97640 5 monclient: authenticate success, global_id 114911 2023-10-30T12:14:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40138e0 con 0x7f92cc009620 2023-10-30T12:14:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92de7fc640 10 monclient: dump: 2023-10-30T12:14:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:06.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92efd97640 10 monclient: _renew_subs 2023-10-30T12:14:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92dffff640 4 set_mon_vals no callback set 2023-10-30T12:14:06.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92efd97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:06.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.672+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e8092590 con 0x7f92cc009620 2023-10-30T12:14:06.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.673+0000 7f92efd97640 10 client.114911.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:06.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.673+0000 7f92efd97640 10 monclient: _renew_subs 2023-10-30T12:14:06.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.673+0000 7f92efd97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:06.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.673+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e8090930 con 0x7f92cc009620 2023-10-30T12:14:06.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.673+0000 7f92efd97640 1 librados: init done 2023-10-30T12:14:06.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.673+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f92e4013a40 con 0x7f92cc009620 2023-10-30T12:14:06.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92efd97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:06.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92efd97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:06.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92efd97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:06.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92de7fc640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92cc03a0c0 0x7f92cc03c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:06.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e804e160 con 0x7f92cc009620 2023-10-30T12:14:06.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(423..423 src has 1..423) v4 ==== 6327+0+0 (secure 0 0 0) 0x7f92e404ae90 con 0x7f92cc009620 2023-10-30T12:14:06.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92de7fc640 10 client.114911.objecter ms_dispatch 0x7f92e8000b90 osd_map(423..423 src has 1..423) v4 2023-10-30T12:14:06.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92de7fc640 3 client.114911.objecter handle_osd_map got epochs [423,423] > 0 2023-10-30T12:14:06.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92de7fc640 3 client.114911.objecter handle_osd_map decoding full epoch 423 2023-10-30T12:14:06.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92de7fc640 20 client.114911.objecter dump_active .. 0 homeless 2023-10-30T12:14:06.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92ed332640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92cc03a0c0 0x7f92cc03c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:06.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.674+0000 7f92ed332640 10 monclient: get_auth_request con 0x7f92cc03a0c0 auth_method 0 2023-10-30T12:14:06.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.675+0000 7f92de7fc640 10 client.114911.objecter ms_handle_connect 0x7f92cc03a0c0 2023-10-30T12:14:06.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.675+0000 7f92ed332640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92cc03a0c0 0x7f92cc03c580 secure :-1 s=READY pgs=4173 cs=0 l=1 rev1=1 crypto rx=0x7f92d8035000 tx=0x7f92d8011040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:06.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.679+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f92e4026950 con 0x7f92cc009620 2023-10-30T12:14:06.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.679+0000 7f92de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:06.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.679+0000 7f92de7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:06.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92efd97640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:14:06.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92efd97640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:06.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92efd97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:06.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f92e80745c0 con 0x7f92cc009620 2023-10-30T12:14:06.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 msgr2=0x7f92e8093550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:14:06.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 msgr2=0x7f92e8093550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:14:06.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 0x7f92e8093550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c5d0 tx=0x7f92e4006a70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:06.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 0x7f92e8093550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c5d0 tx=0x7f92e4006a70 comp rx=0 tx=0).stop 2023-10-30T12:14:06.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:06.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:06.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 msgr2=0x7f92e8093550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:06.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc009620 0x7f92e8093550 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:14:06.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc042dc0 0x7f92cc0451d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:06.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient(hunting): picked mon.c con 0x7f92cc042dc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:06.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 0x7f92cc047af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:06.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient(hunting): picked mon.a con 0x7f92e8136ed0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:06.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 0x7f92cc04a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:06.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient(hunting): picked mon.b con 0x7f92e808ef90 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:06.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:06.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc04af80 con 0x7f92e8136ed0 2023-10-30T12:14:06.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:06.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc04b0c0 con 0x7f92e808ef90 2023-10-30T12:14:06.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:06.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc04b250 con 0x7f92cc042dc0 2023-10-30T12:14:06.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92ee334640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 msgr2=0x7f92cc047af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:06.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92de7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:06.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92ee334640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 0x7f92cc047af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:06.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc042dc0 0x7f92cc0451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:06.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): get_auth_request con 0x7f92cc042dc0 auth_method 0 2023-10-30T12:14:06.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:06.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:06.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:06.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:06.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:06.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:06.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 0x7f92cc04a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:06.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient(hunting): handle_auth_done global_id 114911 payload 293 2023-10-30T12:14:06.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 msgr2=0x7f92cc04a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:06.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 0x7f92cc04a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 msgr2=0x7f92cc047af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:06.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 0x7f92cc047af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient: _finish_hunting 0 2023-10-30T12:14:06.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 monclient: found mon.c 2023-10-30T12:14:06.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:06.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:06.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f92cc04b440 con 0x7f92cc042dc0 2023-10-30T12:14:06.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:06.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:06.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.925+0000 7f92edb33640 1 -- 172.21.15.50:0/3646206259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f92e80745c0 con 0x7f92cc042dc0 2023-10-30T12:14:06.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92edb33640 10 monclient: _finish_auth 0 2023-10-30T12:14:06.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92edb33640 10 monclient: _check_auth_tickets 2023-10-30T12:14:06.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92edb33640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:06.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 10 client.114911.objecter ms_handle_connect 0x7f92cc042dc0 2023-10-30T12:14:06.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 10 client.114911.objecter resend_mon_ops 2023-10-30T12:14:06.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92edb33640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc042dc0 0x7f92cc0451d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f92e4049230 tx=0x7f92e4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:06.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e401eec0 con 0x7f92cc042dc0 2023-10-30T12:14:06.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:06.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:06.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 10 monclient: dump: 2023-10-30T12:14:06.939 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:06.940 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:06.940 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.940 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:06.940 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:06.940 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:06.940 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:06.941 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:06.941 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:06.941 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:06.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e40169a0 con 0x7f92cc042dc0 2023-10-30T12:14:06.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:06.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92dffff640 4 set_mon_vals no callback set 2023-10-30T12:14:06.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.926+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f92e40227e0 con 0x7f92cc042dc0 2023-10-30T12:14:06.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.927+0000 7f92de7fc640 1 -- 172.21.15.50:0/3646206259 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f92e404fed0 con 0x7f92cc042dc0 2023-10-30T12:14:06.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.927+0000 7f92de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:06.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.927+0000 7f92de7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:06.943 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":132,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:14:06.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 10 librados: watch_flush enter 2023-10-30T12:14:06.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 10 librados: watch_flush exit 2023-10-30T12:14:06.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 20 client.114911.objecter shutdown clearing up homeless session... 2023-10-30T12:14:06.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 10 client.114911.objecter successfully canceled tick 2023-10-30T12:14:06.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92cc03a0c0 msgr2=0x7f92cc03c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:06.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92cc03a0c0 0x7f92cc03c580 secure :-1 s=READY pgs=4173 cs=0 l=1 rev1=1 crypto rx=0x7f92d8035000 tx=0x7f92d8011040 comp rx=0 tx=0).stop 2023-10-30T12:14:06.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 10 monclient: shutdown 2023-10-30T12:14:06.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:06.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc042dc0 msgr2=0x7f92cc0451d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:06.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc042dc0 0x7f92cc0451d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f92e4049230 tx=0x7f92e4008040 comp rx=0 tx=0).stop 2023-10-30T12:14:06.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 shutdown_connections 2023-10-30T12:14:06.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92e808ef90 0x7f92cc04a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e8136ed0 0x7f92cc047af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92cc042dc0 0x7f92cc0451d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 --2- 172.21.15.50:0/3646206259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f92cc03a0c0 0x7f92cc03c580 unknown :-1 s=CLOSED pgs=4173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:06.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 >> 172.21.15.50:0/3646206259 conn(0x7f92e80a05c0 msgr2=0x7f92e80a1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:06.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 shutdown_connections 2023-10-30T12:14:06.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 -- 172.21.15.50:0/3646206259 wait complete. 2023-10-30T12:14:06.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:06.929+0000 7f92efd97640 1 librados: shutdown 2023-10-30T12:14:06.948 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:14:06.948 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:14:06.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:14:07.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.053+0000 7f3aaacd7640 10 monclient: get_monmap_and_config 2023-10-30T12:14:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.053+0000 7f3aaacd7640 10 monclient: build_initial_monmap 2023-10-30T12:14:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.053+0000 7f3aaacd7640 10 monclient: monmap: 2023-10-30T12:14:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:07.054694+0000 2023-10-30T12:14:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:07.054694+0000 2023-10-30T12:14:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:07.057 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 1 Processor -- start 2023-10-30T12:14:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 1 -- start start 2023-10-30T12:14:07.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient: init 2023-10-30T12:14:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T12:14:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3aa4071c00 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa409a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa4097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3aa409b170 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa4097600 con 0x7f3aa409b170 2023-10-30T12:14:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa40978e0 con 0x7f3aa4071c00 2023-10-30T12:14:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa409bf50 con 0x7f3aa4097fe0 2023-10-30T12:14:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aaacd7640 10 monclient(hunting): authenticate will time out at 1692.237830s 2023-10-30T12:14:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aa9274640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f3aa409b170 legacy=0x7f3aa4139760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:14:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa409a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa409a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57188/0 (socket says 172.21.15.50:57188) 2023-10-30T12:14:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aa3fff640 1 -- 172.21.15.50:0/387698360 learned_addr learned my addr 172.21.15.50:0/387698360 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.054+0000 7f3aa3fff640 10 monclient(hunting): get_auth_request con 0x7f3aa4097fe0 auth_method 0 2023-10-30T12:14:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_done global_id 114914 payload 274 2023-10-30T12:14:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 1 -- 172.21.15.50:0/387698360 >> v1:172.21.15.118:6789/0 conn(0x7f3aa4071c00 legacy=0x7f3aa409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 1 -- 172.21.15.50:0/387698360 >> v1:172.21.15.50:6789/0 conn(0x7f3aa409b170 legacy=0x7f3aa4139760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:14:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 1 monclient: found mon.noname-b 2023-10-30T12:14:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 1 -- 172.21.15.50:0/387698360 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa409d9b0 con 0x7f3aa4097fe0 2023-10-30T12:14:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.055+0000 7f3aa3fff640 1 --2- 172.21.15.50:0/387698360 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa409a150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3a900098a0 tx=0x7f3a9002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aff5f65f3a2e9624 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 1 -- 172.21.15.50:0/387698360 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a900317c0 con 0x7f3aa4097fe0 2023-10-30T12:14:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:14:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: dump: 2023-10-30T12:14:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: _finish_auth 0 2023-10-30T12:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 1 -- 172.21.15.50:0/387698360 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a90031920 con 0x7f3aa4097fe0 2023-10-30T12:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 1 -- 172.21.15.50:0/387698360 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90031cb0 con 0x7f3aa4097fe0 2023-10-30T12:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aa37fe640 10 monclient: dump: 2023-10-30T12:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 5 monclient: authenticate success, global_id 114914 2023-10-30T12:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 4 set_mon_vals no callback set 2023-10-30T12:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 10 monclient: shutdown 2023-10-30T12:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 1 -- 172.21.15.50:0/387698360 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 msgr2=0x7f3aa409a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 1 --2- 172.21.15.50:0/387698360 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa409a150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3a900098a0 tx=0x7f3a9002b860 comp rx=0 tx=0).stop 2023-10-30T12:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 1 -- 172.21.15.50:0/387698360 shutdown_connections 2023-10-30T12:14:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 1 --2- 172.21.15.50:0/387698360 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa409a150 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 1 -- 172.21.15.50:0/387698360 >> 172.21.15.50:0/387698360 conn(0x7f3aa40a3980 msgr2=0x7f3aa40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 1 -- 172.21.15.50:0/387698360 shutdown_connections 2023-10-30T12:14:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.056+0000 7f3aaacd7640 1 -- 172.21.15.50:0/387698360 wait complete. 2023-10-30T12:14:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 10 monclient: build_initial_monmap 2023-10-30T12:14:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 10 monclient: monmap: 2023-10-30T12:14:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:07.081 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 1 librados: starting msgr at 2023-10-30T12:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 1 librados: starting objecter 2023-10-30T12:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 1 Processor -- start 2023-10-30T12:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 1 -- start start 2023-10-30T12:14:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 1 librados: setting wanted keys 2023-10-30T12:14:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 1 librados: calling monclient init 2023-10-30T12:14:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.057+0000 7f3aaacd7640 10 monclient: init 2023-10-30T12:14:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3aa4071c00 0x7f3aa4133360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa4071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa41c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa4097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3aa409b170 0x7f3aa4071270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): picked mon.noname-a con 0x7f3aa409b170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa41c7e90 con 0x7f3aa409b170 2023-10-30T12:14:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa406ca30 con 0x7f3aa4071c00 2023-10-30T12:14:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa413de00 con 0x7f3aa4097fe0 2023-10-30T12:14:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient(hunting): authenticate will time out at 1692.241280s 2023-10-30T12:14:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa9274640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3aa409b170 msgr2=0x7f3aa4071270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:07.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa9274640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3aa409b170 0x7f3aa4071270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa41c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa41c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57196/0 (socket says 172.21.15.50:57196) 2023-10-30T12:14:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 -- 172.21.15.50:0/2749755240 learned_addr learned my addr 172.21.15.50:0/2749755240 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): get_auth_request con 0x7f3aa4097fe0 auth_method 0 2023-10-30T12:14:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa8a73640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3aa4071c00 0x7f3aa4133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa8a73640 10 monclient(hunting): get_auth_request con 0x7f3aa4071c00 auth_method 0 2023-10-30T12:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa8a73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa8a73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa8a73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient(hunting): handle_auth_done global_id 114917 payload 824 2023-10-30T12:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3aa4071c00 msgr2=0x7f3aa4133360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3aa4071c00 0x7f3aa4133360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3aa409b170 msgr2=0x7f3aa4071270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3aa409b170 0x7f3aa4071270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:14:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 monclient: found mon.noname-c 2023-10-30T12:14:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 -- 172.21.15.50:0/2749755240 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a900094d0 con 0x7f3aa4097fe0 2023-10-30T12:14:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa3fff640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa41c6ee0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3a90000c00 tx=0x7f3a900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 client.?.objecter ms_handle_connect 0x7f3aa4097fe0 2023-10-30T12:14:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90032960 con 0x7f3aa4097fe0 2023-10-30T12:14:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: dump: 2023-10-30T12:14:07.092 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.093 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.094 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: _finish_auth 0 2023-10-30T12:14:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: _check_auth_tickets 2023-10-30T12:14:07.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:07.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 5 monclient: authenticate success, global_id 114917 2023-10-30T12:14:07.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient: _renew_subs 2023-10-30T12:14:07.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a90032ae0 con 0x7f3aa4097fe0 2023-10-30T12:14:07.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa406c470 con 0x7f3aa4097fe0 2023-10-30T12:14:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90032e90 con 0x7f3aa4097fe0 2023-10-30T12:14:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa17fa640 10 monclient: dump: 2023-10-30T12:14:07.097 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.097 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.097 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.097 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.097 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.098 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 10 client.114917.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 10 monclient: _renew_subs 2023-10-30T12:14:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.058+0000 7f3aa2ffd640 4 set_mon_vals no callback set 2023-10-30T12:14:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa41c7420 con 0x7f3aa4097fe0 2023-10-30T12:14:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 1 librados: init done 2023-10-30T12:14:07.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa40717b0 0x7f3aa4094a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa4073ff0 con 0x7f3aa40717b0 2023-10-30T12:14:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3a9004d3f0 con 0x7f3aa4097fe0 2023-10-30T12:14:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa40717b0 0x7f3aa4094a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 10 monclient: get_auth_request con 0x7f3aa40717b0 auth_method 0 2023-10-30T12:14:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.059+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_done global_id 114920 payload 824 2023-10-30T12:14:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a880392a0 0x7f3a8803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa8a73640 1 -- 172.21.15.50:0/2749755240 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3aa4071fe0 con 0x7f3aa40717b0 2023-10-30T12:14:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(423..423 src has 1..423) v4 ==== 6327+0+0 (secure 0 0 0) 0x7f3a9006cc80 con 0x7f3aa4097fe0 2023-10-30T12:14:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 client.114917.objecter ms_dispatch 0x7f3aa4000b90 osd_map(423..423 src has 1..423) v4 2023-10-30T12:14:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 3 client.114917.objecter handle_osd_map got epochs [423,423] > 0 2023-10-30T12:14:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 3 client.114917.objecter handle_osd_map decoding full epoch 423 2023-10-30T12:14:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 20 client.114917.objecter dump_active .. 0 homeless 2023-10-30T12:14:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa8a73640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa40717b0 0x7f3aa4094a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800e4d0 tx=0x7f3a9800e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 client.114917.objecter ms_handle_connect 0x7f3aa40717b0 2023-10-30T12:14:07.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 client.114917.objecter resend_mon_ops 2023-10-30T12:14:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a980052e0 con 0x7f3aa40717b0 2023-10-30T12:14:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 monclient: dump: 2023-10-30T12:14:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.105 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.106 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa9274640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a880392a0 0x7f3a8803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa9274640 10 monclient: get_auth_request con 0x7f3a880392a0 auth_method 0 2023-10-30T12:14:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f3a98005440 con 0x7f3aa40717b0 2023-10-30T12:14:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa40717b0 msgr2=0x7f3aa4094a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa40717b0 0x7f3aa4094a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800e4d0 tx=0x7f3a9800e990 comp rx=0 tx=0).stop 2023-10-30T12:14:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa40717b0 msgr2=0x7f3aa4094a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.060+0000 7f3aa17fa640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa40717b0 0x7f3aa4094a20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.061+0000 7f3aa17fa640 10 client.114917.objecter ms_handle_connect 0x7f3a880392a0 2023-10-30T12:14:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.061+0000 7f3aa9274640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a880392a0 0x7f3a8803b760 secure :-1 s=READY pgs=4174 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40965f0 tx=0x7f3a9c009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.066+0000 7f3aaacd7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.066+0000 7f3aaacd7640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4094f60 0x7f3aa4095370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.066+0000 7f3aaacd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.066+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa4073a20 con 0x7f3aa4094f60 2023-10-30T12:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4094f60 0x7f3aa4095370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient: get_auth_request con 0x7f3aa4094f60 auth_method 0 2023-10-30T12:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 10 monclient(hunting): handle_auth_done global_id 114923 payload 824 2023-10-30T12:14:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 1 -- 172.21.15.50:0/2749755240 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3aa40958b0 con 0x7f3aa4094f60 2023-10-30T12:14:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa8a73640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4094f60 0x7f3aa4095370 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800e4d0 tx=0x7f3a98001170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 10 client.114917.objecter ms_handle_connect 0x7f3aa4094f60 2023-10-30T12:14:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 10 client.114917.objecter resend_mon_ops 2023-10-30T12:14:07.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98023070 con 0x7f3aa4094f60 2023-10-30T12:14:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 10 monclient: dump: 2023-10-30T12:14:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.113 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.114 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.115 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.115 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4791 (secure 0 0 0) 0x7f3a9801e030 con 0x7f3aa4094f60 2023-10-30T12:14:07.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:07.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:07.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4094f60 msgr2=0x7f3aa4095370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4094f60 0x7f3aa4095370 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800e4d0 tx=0x7f3a98001170 comp rx=0 tx=0).stop 2023-10-30T12:14:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4094f60 msgr2=0x7f3aa4095370 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:07.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aa17fa640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4094f60 0x7f3aa4095370 secure :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800e4d0 tx=0x7f3a98001170 comp rx=0 tx=0).stop 2023-10-30T12:14:07.116 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:14:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 132, 2023-10-30T12:14:07.117 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:14:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:07.118 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:07.119 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:07.120 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:07.121 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:07.122 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:14:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:14:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:14:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:14:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:07.148 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:07.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aaacd7640 10 librados: watch_flush enter 2023-10-30T12:14:07.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aaacd7640 10 librados: watch_flush exit 2023-10-30T12:14:07.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aaacd7640 20 client.114917.objecter shutdown clearing up homeless session... 2023-10-30T12:14:07.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.067+0000 7f3aaacd7640 10 client.114917.objecter successfully canceled tick 2023-10-30T12:14:07.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a880392a0 msgr2=0x7f3a8803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3a880392a0 0x7f3a8803b760 secure :-1 s=READY pgs=4174 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40965f0 tx=0x7f3a9c009270 comp rx=0 tx=0).stop 2023-10-30T12:14:07.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 10 monclient: shutdown 2023-10-30T12:14:07.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:07.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 msgr2=0x7f3aa41c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aa9274640 1 -- 172.21.15.50:0/2749755240 reap_dead start 2023-10-30T12:14:07.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa41c6ee0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3a90000c00 tx=0x7f3a900046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 shutdown_connections 2023-10-30T12:14:07.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 --2- 172.21.15.50:0/2749755240 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3aa4097fe0 0x7f3aa41c6ee0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 >> 172.21.15.50:0/2749755240 conn(0x7f3aa40a3980 msgr2=0x7f3aa4133ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:07.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 shutdown_connections 2023-10-30T12:14:07.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 -- 172.21.15.50:0/2749755240 wait complete. 2023-10-30T12:14:07.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.068+0000 7f3aaacd7640 1 librados: shutdown 2023-10-30T12:14:07.153 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:14:07.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:14:07.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:14:07.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.258+0000 7fabf4ba5640 10 monclient: get_monmap_and_config 2023-10-30T12:14:07.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.258+0000 7fabf4ba5640 10 monclient: build_initial_monmap 2023-10-30T12:14:07.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.258+0000 7fabf4ba5640 10 monclient: monmap: 2023-10-30T12:14:07.261 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:07.261 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.261 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:07.259692+0000 2023-10-30T12:14:07.261 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:07.259692+0000 2023-10-30T12:14:07.262 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:07.262 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.262 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:07.262 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:07.262 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:07.262 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:07.263 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:07.263 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 1 Processor -- start 2023-10-30T12:14:07.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 1 -- start start 2023-10-30T12:14:07.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient: init 2023-10-30T12:14:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:14:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fabf0071c00 0x7fabf009a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient(hunting): picked mon.noname-a con 0x7fabf0071c00 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabf0097fe0 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabf009b170 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf0097600 con 0x7fabf0097fe0 2023-10-30T12:14:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf00978e0 con 0x7fabf009b170 2023-10-30T12:14:07.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf009bf50 con 0x7fabf0071c00 2023-10-30T12:14:07.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:07.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabf4ba5640 10 monclient(hunting): authenticate will time out at 1692.442720s 2023-10-30T12:14:07.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabe5d9b640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fabf0097fe0 legacy=0x7fabf009aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:14:07.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabee59c640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fabf0071c00 msgr2=0x7fabf009a150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:07.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabee59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fabf0071c00 0x7fabf009a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:07.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabeed9d640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fabf009b170 0x7fabf0139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:41842/0 (socket says 172.21.15.50:41842) 2023-10-30T12:14:07.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.259+0000 7fabeed9d640 1 -- 172.21.15.50:0/4066694241 learned_addr learned my addr 172.21.15.50:0/4066694241 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:14:07.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201535785 0 0) 0x7fabf00978e0 con 0x7fabf009b170 2023-10-30T12:14:07.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): _negotiate 2023-10-30T12:14:07.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): my global_id is 184171 2023-10-30T12:14:07.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd0003440 con 0x7fabf009b170 2023-10-30T12:14:07.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279686578 0 0) 0x7fabd0003440 con 0x7fabf009b170 2023-10-30T12:14:07.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 >> v2:172.21.15.50:3300/0 conn(0x7fabf0071c00 msgr2=0x7fabf009a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:07.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 --2- 172.21.15.50:0/4066694241 >> v2:172.21.15.50:3300/0 conn(0x7fabf0071c00 0x7fabf009a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 >> v1:172.21.15.50:6789/0 conn(0x7fabf0097fe0 legacy=0x7fabf009aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:07.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:14:07.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:14:07.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:07.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:14:07.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf009d9b0 con 0x7fabf009b170 2023-10-30T12:14:07.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: _finish_auth 0 2023-10-30T12:14:07.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:14:07.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fabe80019f0 con 0x7fabf009b170 2023-10-30T12:14:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:14:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: dump: 2023-10-30T12:14:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.282 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.282 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.282 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:14:07.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:07.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 >> v1:172.21.15.118:6789/0 conn(0x7fabf009b170 legacy=0x7fabf0139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:14:07.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0003a60 0x7fabd0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): picked mon.a con 0x7fabd0003a60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:07.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabd0008970 0x7fabd000ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): picked mon.c con 0x7fabd0008970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabd000b6a0 0x7fabd000db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): picked mon.b con 0x7fabd000b6a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf009bf50 con 0x7fabd0003a60 2023-10-30T12:14:07.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd000e440 con 0x7fabd000b6a0 2023-10-30T12:14:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd000e600 con 0x7fabd0008970 2023-10-30T12:14:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:07.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabedd9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:14:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabee59c640 1 -- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0003a60 msgr2=0x7fabd0003e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabee59c640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0003a60 0x7fabd0003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabd0008970 0x7fabd000ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): get_auth_request con 0x7fabd0008970 auth_method 0 2023-10-30T12:14:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabd000b6a0 0x7fabd000db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:07.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 10 monclient(hunting): get_auth_request con 0x7fabd000b6a0 auth_method 0 2023-10-30T12:14:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.260+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_done global_id 184171 payload 293 2023-10-30T12:14:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 1 -- 172.21.15.50:0/4066694241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabd000b6a0 msgr2=0x7fabd000db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabd000b6a0 0x7fabd000db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 1 -- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0003a60 msgr2=0x7fabd0003e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:07.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0003a60 0x7fabd0003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:14:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 1 monclient: found mon.c 2023-10-30T12:14:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 1 -- 172.21.15.50:0/4066694241 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd0003440 con 0x7fabd0008970 2023-10-30T12:14:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:14:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:14:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabf4ba5640 5 monclient: authenticate success, global_id 184171 2023-10-30T12:14:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabf4ba5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabeed9d640 1 -- 172.21.15.50:0/4066694241 reap_dead start 2023-10-30T12:14:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabe5d9b640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabd0008970 0x7fabd000ae00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009900 tx=0x7fabd00036f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6850435039a7185e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8003bb0 con 0x7fabd0008970 2023-10-30T12:14:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabedd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabedd9b640 10 monclient: dump: 2023-10-30T12:14:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.293 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd8003d10 con 0x7fabd0008970 2023-10-30T12:14:07.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabedd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabedd9b640 1 -- 172.21.15.50:0/4066694241 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd80052b0 con 0x7fabd0008970 2023-10-30T12:14:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.261+0000 7fabf4ba5640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 4 set_mon_vals no callback set 2023-10-30T12:14:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabedd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:07.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabedd9b640 10 monclient: dump: 2023-10-30T12:14:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.296 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.297 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 10 monclient: shutdown 2023-10-30T12:14:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 1 -- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabd0008970 msgr2=0x7fabd000ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabd0008970 0x7fabd000ae00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009900 tx=0x7fabd00036f0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 1 -- 172.21.15.50:0/4066694241 shutdown_connections 2023-10-30T12:14:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 1 --2- 172.21.15.50:0/4066694241 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabd0008970 0x7fabd000ae00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 1 -- 172.21.15.50:0/4066694241 >> 172.21.15.50:0/4066694241 conn(0x7fabf00a3980 msgr2=0x7fabf00a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 1 -- 172.21.15.50:0/4066694241 shutdown_connections 2023-10-30T12:14:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.262+0000 7fabf4ba5640 1 -- 172.21.15.50:0/4066694241 wait complete. 2023-10-30T12:14:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 10 monclient: build_initial_monmap 2023-10-30T12:14:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 10 monclient: monmap: 2023-10-30T12:14:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:07.301 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 1 librados: starting msgr at 2023-10-30T12:14:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 1 librados: starting objecter 2023-10-30T12:14:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 1 Processor -- start 2023-10-30T12:14:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 1 -- start start 2023-10-30T12:14:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 1 librados: setting wanted keys 2023-10-30T12:14:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 1 librados: calling monclient init 2023-10-30T12:14:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.263+0000 7fabf4ba5640 10 monclient: init 2023-10-30T12:14:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:14:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0008970 0x7fabf01c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): picked mon.noname-a con 0x7fabd0008970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabf0091cb0 0x7fabf0094750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): picked mon.noname-c con 0x7fabf0091cb0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf0094c90 0x7fabf01c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): picked mon.noname-b con 0x7fabf0094c90 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00913f0 con 0x7fabd0008970 2023-10-30T12:14:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf009cbc0 con 0x7fabf0094c90 2023-10-30T12:14:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf0092fd0 con 0x7fabf0091cb0 2023-10-30T12:14:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabee59c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0008970 msgr2=0x7fabf01c9f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:14:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabf4ba5640 10 monclient(hunting): authenticate will time out at 1692.447402s 2023-10-30T12:14:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabee59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0008970 0x7fabf01c9f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabf0091cb0 0x7fabf0094750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabf0091cb0 0x7fabf0094750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57242/0 (socket says 172.21.15.50:57242) 2023-10-30T12:14:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 -- 172.21.15.50:0/369855190 learned_addr learned my addr 172.21.15.50:0/369855190 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): get_auth_request con 0x7fabf0091cb0 auth_method 0 2023-10-30T12:14:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf0094c90 0x7fabf01c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 10 monclient(hunting): get_auth_request con 0x7fabf0094c90 auth_method 0 2023-10-30T12:14:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient(hunting): handle_auth_done global_id 114926 payload 824 2023-10-30T12:14:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf0094c90 msgr2=0x7fabf01c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf0094c90 0x7fabf01c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0008970 msgr2=0x7fabf01c9f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabd0008970 0x7fabf01c9f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 monclient: found mon.noname-c 2023-10-30T12:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.264+0000 7fabe5d9b640 1 -- 172.21.15.50:0/369855190 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0092980 con 0x7fabf0091cb0 2023-10-30T12:14:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe5d9b640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabf0091cb0 0x7fabf0094750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fabd80090b0 tx=0x7fabd8005a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 client.?.objecter ms_handle_connect 0x7fabf0091cb0 2023-10-30T12:14:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8034670 con 0x7fabf0091cb0 2023-10-30T12:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: dump: 2023-10-30T12:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: _finish_auth 0 2023-10-30T12:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd80347f0 con 0x7fabf0091cb0 2023-10-30T12:14:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 5 monclient: authenticate success, global_id 114926 2023-10-30T12:14:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8034ba0 con 0x7fabf0091cb0 2023-10-30T12:14:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 10 monclient: _renew_subs 2023-10-30T12:14:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf0092fd0 con 0x7fabf0091cb0 2023-10-30T12:14:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabe77fe640 10 monclient: dump: 2023-10-30T12:14:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.318 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabed59a640 4 set_mon_vals no callback set 2023-10-30T12:14:07.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 10 client.114926.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 10 monclient: _renew_subs 2023-10-30T12:14:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf01ca9f0 con 0x7fabf0091cb0 2023-10-30T12:14:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.265+0000 7fabf4ba5640 1 librados: init done 2023-10-30T12:14:07.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fabd8031070 con 0x7fabf0091cb0 2023-10-30T12:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabf4ba5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabf4ba5640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002b430 0x7fabf002b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabf4ba5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf0073ff0 con 0x7fabf002b430 2023-10-30T12:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fabd40392a0 0x7fabd403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(423..423 src has 1..423) v4 ==== 6327+0+0 (secure 0 0 0) 0x7fabd803b040 con 0x7fabf0091cb0 2023-10-30T12:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 10 client.114926.objecter ms_dispatch 0x7fabf0000b90 osd_map(423..423 src has 1..423) v4 2023-10-30T12:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 3 client.114926.objecter handle_osd_map got epochs [423,423] > 0 2023-10-30T12:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 3 client.114926.objecter handle_osd_map decoding full epoch 423 2023-10-30T12:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002b430 0x7fabf002b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient: get_auth_request con 0x7fabf002b430 auth_method 0 2023-10-30T12:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 20 client.114926.objecter dump_active .. 0 homeless 2023-10-30T12:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabeed9d640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fabd40392a0 0x7fabd403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabeed9d640 10 monclient: get_auth_request con 0x7fabd40392a0 auth_method 0 2023-10-30T12:14:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 10 monclient(hunting): handle_auth_done global_id 184177 payload 824 2023-10-30T12:14:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 1 -- 172.21.15.50:0/369855190 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabf0071fe0 con 0x7fabf002b430 2023-10-30T12:14:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 10 client.114926.objecter ms_handle_connect 0x7fabd40392a0 2023-10-30T12:14:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabeed9d640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fabd40392a0 0x7fabd403b760 secure :-1 s=READY pgs=4175 cs=0 l=1 rev1=1 crypto rx=0x7fabf00983c0 tx=0x7fabe8006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 10 client.114926.objecter ms_handle_connect 0x7fabf002b430 2023-10-30T12:14:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabe77fe640 10 client.114926.objecter resend_mon_ops 2023-10-30T12:14:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.266+0000 7fabee59c640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002b430 0x7fabf002b810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fabe000b470 tx=0x7fabe000b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0005070 con 0x7fabf002b430 2023-10-30T12:14:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 10 monclient: dump: 2023-10-30T12:14:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.327 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fabe00051d0 con 0x7fabf002b430 2023-10-30T12:14:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002b430 msgr2=0x7fabf002b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002b430 0x7fabf002b810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fabe000b470 tx=0x7fabe000b930 comp rx=0 tx=0).stop 2023-10-30T12:14:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002b430 msgr2=0x7fabf002b810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:07.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.267+0000 7fabe77fe640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002b430 0x7fabf002b810 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fabe000b470 tx=0x7fabe000b930 comp rx=0 tx=0).stop 2023-10-30T12:14:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabf4ba5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabf4ba5640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002bd50 0x7fabf002c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabf4ba5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf01cabb0 con 0x7fabf002bd50 2023-10-30T12:14:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabee59c640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002bd50 0x7fabf002c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabee59c640 10 monclient: get_auth_request con 0x7fabf002bd50 auth_method 0 2023-10-30T12:14:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabee59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.273+0000 7fabee59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabee59c640 10 monclient(hunting): handle_auth_done global_id 184180 payload 824 2023-10-30T12:14:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabee59c640 1 -- 172.21.15.50:0/369855190 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fabf004e160 con 0x7fabf002bd50 2023-10-30T12:14:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabee59c640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002bd50 0x7fabf002c160 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe000b470 tx=0x7fabe0005b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 10 client.114926.objecter ms_handle_connect 0x7fabf002bd50 2023-10-30T12:14:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 10 client.114926.objecter resend_mon_ops 2023-10-30T12:14:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0021070 con 0x7fabf002bd50 2023-10-30T12:14:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 10 monclient: dump: 2023-10-30T12:14:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:07.335 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:07.335 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7fabe001c2e0 con 0x7fabf002bd50 2023-10-30T12:14:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002bd50 msgr2=0x7fabf002c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002bd50 0x7fabf002c160 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe000b470 tx=0x7fabe0005b00 comp rx=0 tx=0).stop 2023-10-30T12:14:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002bd50 msgr2=0x7fabf002c160 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:07.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.274+0000 7fabe77fe640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fabf002bd50 0x7fabf002c160 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:07.337 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:07.337 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:07.338 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:07.338 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:14:07.338 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 132, 2023-10-30T12:14:07.338 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:07.338 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:07.339 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:07.339 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.339 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:14:07.339 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:07.339 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:07.340 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:07.340 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:07.340 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:07.340 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:07.340 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:07.341 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:07.341 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:07.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:07.341 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:07.341 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:07.341 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:07.342 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.342 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:07.342 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:07.342 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:07.342 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:07.343 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:07.343 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:07.343 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:07.343 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:07.343 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:07.343 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:07.344 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:07.344 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:07.344 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.344 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:07.345 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:07.345 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:07.345 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:07.345 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:07.345 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:07.345 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:07.346 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:07.346 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:07.346 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:07.346 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:07.346 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:07.347 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:07.347 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:07.347 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:07.347 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:07.347 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:07.348 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:07.348 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:07.348 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:07.348 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:07.348 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:07.348 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:07.349 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:07.349 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:07.349 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:07.349 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:07.349 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:07.350 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:07.351 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:07.352 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:07.353 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.354 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.355 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:07.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.357 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:07.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:07.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:07.362 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:07.363 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.364 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:07.365 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:14:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:07.366 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:07.366 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:07.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 10 librados: watch_flush enter 2023-10-30T12:14:07.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 10 librados: watch_flush exit 2023-10-30T12:14:07.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 20 client.114926.objecter shutdown clearing up homeless session... 2023-10-30T12:14:07.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 10 client.114926.objecter successfully canceled tick 2023-10-30T12:14:07.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fabd40392a0 msgr2=0x7fabd403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fabd40392a0 0x7fabd403b760 secure :-1 s=READY pgs=4175 cs=0 l=1 rev1=1 crypto rx=0x7fabf00983c0 tx=0x7fabe8006040 comp rx=0 tx=0).stop 2023-10-30T12:14:07.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 10 monclient: shutdown 2023-10-30T12:14:07.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:07.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabf0091cb0 msgr2=0x7fabf0094750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:07.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 --2- 172.21.15.50:0/369855190 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fabf0091cb0 0x7fabf0094750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fabd80090b0 tx=0x7fabd8005a10 comp rx=0 tx=0).stop 2023-10-30T12:14:07.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabeed9d640 1 -- 172.21.15.50:0/369855190 reap_dead start 2023-10-30T12:14:07.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabeed9d640 1 -- 172.21.15.50:0/369855190 reap_dead start 2023-10-30T12:14:07.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 shutdown_connections 2023-10-30T12:14:07.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 >> 172.21.15.50:0/369855190 conn(0x7fabf00a3980 msgr2=0x7fabf00965b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:07.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 shutdown_connections 2023-10-30T12:14:07.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 -- 172.21.15.50:0/369855190 wait complete. 2023-10-30T12:14:07.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:07.275+0000 7fabf4ba5640 1 librados: shutdown 2023-10-30T12:14:07.372 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:14:07.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:14:27.373 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:14:27.373 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:14:27.373 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:14:27.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:14:27.377 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:14:27.378 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:14:27.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:14:27.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.493+0000 7f39a99a0640 10 monclient: get_monmap_and_config 2023-10-30T12:14:27.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.493+0000 7f39a99a0640 10 monclient: build_initial_monmap 2023-10-30T12:14:27.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.493+0000 7f39a99a0640 10 monclient: monmap: 2023-10-30T12:14:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:27.494888+0000 2023-10-30T12:14:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:27.494888+0000 2023-10-30T12:14:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:27.497 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:27.497 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:27.497 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:27.497 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:27.497 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:27.498 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:27.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.494+0000 7f39a99a0640 1 Processor -- start 2023-10-30T12:14:27.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.494+0000 7f39a99a0640 1 -- start start 2023-10-30T12:14:27.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.494+0000 7f39a99a0640 10 monclient: init 2023-10-30T12:14:27.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:27.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T12:14:27.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f39a41312b0 0x7f39a406b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:27.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient(hunting): picked mon.noname-c con 0x7f39a41312b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:27.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39a4131690 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:27.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39a4131ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:27.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40a2f20 con 0x7f39a4131ae0 2023-10-30T12:14:27.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40a3c20 con 0x7f39a4131690 2023-10-30T12:14:27.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:27.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a406a330 con 0x7f39a41312b0 2023-10-30T12:14:27.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:27.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a99a0640 10 monclient(hunting): authenticate will time out at 1712.678209s 2023-10-30T12:14:27.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a919f640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f39a4131ae0 0x7f39a40a2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50202/0 (socket says 172.21.15.50:50202) 2023-10-30T12:14:27.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a919f640 1 -- 172.21.15.50:0/2876170144 learned_addr learned my addr 172.21.15.50:0/2876170144 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:14:27.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 1 --2- 172.21.15.50:0/2876170144 >> v2:172.21.15.118:3300/0 conn(0x7f39a41312b0 0x7f39a406b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:27.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): get_auth_request con 0x7f39a41312b0 auth_method 0 2023-10-30T12:14:27.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:27.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:27.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:27.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:27.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:27.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:27.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient(hunting): handle_auth_done global_id 184183 payload 274 2023-10-30T12:14:27.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 1 -- 172.21.15.50:0/2876170144 >> v1:172.21.15.118:6789/0 conn(0x7f39a4131690 legacy=0x7f39a40a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:27.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 1 -- 172.21.15.50:0/2876170144 >> v1:172.21.15.50:6789/0 conn(0x7f39a4131ae0 legacy=0x7f39a40a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:27.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient: _finish_hunting 0 2023-10-30T12:14:27.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 1 monclient: found mon.noname-c 2023-10-30T12:14:27.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:27.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:14:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.495+0000 7f39a899e640 1 -- 172.21.15.50:0/2876170144 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a406a610 con 0x7f39a41312b0 2023-10-30T12:14:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f39a899e640 1 --2- 172.21.15.50:0/2876170144 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a41312b0 0x7f39a406b690 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009790 tx=0x7f39a002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de20fa4b15a070aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 1 -- 172.21.15.50:0/2876170144 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0031830 con 0x7f39a41312b0 2023-10-30T12:14:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: dump: 2023-10-30T12:14:27.512 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:27.512 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:27.512 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.512 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.512 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:27.512 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:27.513 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:27.513 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:27.513 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:27.513 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:27.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:14:27.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:14:27.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:27.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 1 -- 172.21.15.50:0/2876170144 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a0031990 con 0x7f39a41312b0 2023-10-30T12:14:27.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f39a99a0640 5 monclient: authenticate success, global_id 184183 2023-10-30T12:14:27.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f39a99a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:27.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:27.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f39a99a0640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:27.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 1 -- 172.21.15.50:0/2876170144 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0031d20 con 0x7f39a41312b0 2023-10-30T12:14:27.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f39a99a0640 4 set_mon_vals no callback set 2023-10-30T12:14:27.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:27.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:27.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.496+0000 7f399b7fe640 10 monclient: dump: 2023-10-30T12:14:27.516 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:27.516 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:27.516 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.516 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.516 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:27.516 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:27.517 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:27.517 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:27.517 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:27.517 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:27.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 10 monclient: shutdown 2023-10-30T12:14:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 1 -- 172.21.15.50:0/2876170144 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a41312b0 msgr2=0x7f39a406b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 1 --2- 172.21.15.50:0/2876170144 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a41312b0 0x7f39a406b690 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009790 tx=0x7f39a002b8d0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 1 -- 172.21.15.50:0/2876170144 shutdown_connections 2023-10-30T12:14:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 1 --2- 172.21.15.50:0/2876170144 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a41312b0 0x7f39a406b690 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 1 -- 172.21.15.50:0/2876170144 >> 172.21.15.50:0/2876170144 conn(0x7f39a40a4200 msgr2=0x7f39a40a4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:27.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 1 -- 172.21.15.50:0/2876170144 shutdown_connections 2023-10-30T12:14:27.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.497+0000 7f39a99a0640 1 -- 172.21.15.50:0/2876170144 wait complete. 2023-10-30T12:14:27.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 10 monclient: build_initial_monmap 2023-10-30T12:14:27.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 10 monclient: monmap: 2023-10-30T12:14:27.519 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:27.520 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:27.520 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:27.520 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:27.520 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:27.520 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:27.521 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:27.521 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:27.521 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:27.521 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:27.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 1 librados: starting msgr at 2023-10-30T12:14:27.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 1 librados: starting objecter 2023-10-30T12:14:27.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 1 Processor -- start 2023-10-30T12:14:27.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 1 -- start start 2023-10-30T12:14:27.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 1 librados: setting wanted keys 2023-10-30T12:14:27.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 1 librados: calling monclient init 2023-10-30T12:14:27.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.498+0000 7f39a99a0640 10 monclient: init 2023-10-30T12:14:27.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:27.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:14:27.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39a41312b0 0x7f39a4133140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:27.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): picked mon.noname-c con 0x7f39a41312b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:27.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 0x7f39a4133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): picked mon.noname-a con 0x7f39a4131690 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a4131ae0 0x7f39a4133bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): picked mon.noname-b con 0x7f39a4131ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4135e20 con 0x7f39a4131690 2023-10-30T12:14:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4134fa0 con 0x7f39a4131ae0 2023-10-30T12:14:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40a0d80 con 0x7f39a41312b0 2023-10-30T12:14:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 10 monclient(hunting): authenticate will time out at 1712.682095s 2023-10-30T12:14:27.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 0x7f39a4133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:27.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39a41312b0 0x7f39a4133140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:27.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 0x7f39a4133680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38888/0 (socket says 172.21.15.50:38888) 2023-10-30T12:14:27.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 1 -- 172.21.15.50:0/764250912 learned_addr learned my addr 172.21.15.50:0/764250912 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:27.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 10 monclient(hunting): get_auth_request con 0x7f39a4131690 auth_method 0 2023-10-30T12:14:27.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:27.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:27.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:27.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): get_auth_request con 0x7f39a41312b0 auth_method 0 2023-10-30T12:14:27.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:27.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:27.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:27.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 1 -- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 msgr2=0x7f39a4133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:14:27.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 1 -- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 msgr2=0x7f39a4133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:14:27.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:27.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 0x7f39a4133680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:27.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:27.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a4131ae0 0x7f39a4133bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:27.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:27.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f399bfff640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 0x7f39a4133680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:27.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 10 monclient(hunting): get_auth_request con 0x7f39a4131ae0 auth_method 0 2023-10-30T12:14:27.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:27.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:27.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:27.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:27.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:27.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a919f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:27.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient(hunting): handle_auth_done global_id 114929 payload 824 2023-10-30T12:14:27.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 -- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a4131ae0 msgr2=0x7f39a4133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:27.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a4131ae0 0x7f39a4133bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 -- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 msgr2=0x7f39a4133680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:27.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 0x7f39a4133680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient: _finish_hunting 0 2023-10-30T12:14:27.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 monclient: found mon.noname-c 2023-10-30T12:14:27.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:27.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:27.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 -- 172.21.15.50:0/764250912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a00093a0 con 0x7f39a41312b0 2023-10-30T12:14:27.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a899e640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39a41312b0 0x7f39a4133140 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f39a0033000 tx=0x7f39a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:27.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 client.?.objecter ms_handle_connect 0x7f39a41312b0 2023-10-30T12:14:27.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:27.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 1 -- 172.21.15.50:0/764250912 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a003ba70 con 0x7f39a41312b0 2023-10-30T12:14:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 monclient: dump: 2023-10-30T12:14:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:27.534 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:27.535 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:27.536 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 monclient: _finish_auth 0 2023-10-30T12:14:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 monclient: _check_auth_tickets 2023-10-30T12:14:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:27.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 1 -- 172.21.15.50:0/764250912 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a003bbf0 con 0x7f39a41312b0 2023-10-30T12:14:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39997fa640 1 -- 172.21.15.50:0/764250912 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00433f0 con 0x7f39a41312b0 2023-10-30T12:14:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.499+0000 7f39a99a0640 5 monclient: authenticate success, global_id 114929 2023-10-30T12:14:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 monclient: _renew_subs 2023-10-30T12:14:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:27.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a40a38f0 con 0x7f39a41312b0 2023-10-30T12:14:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f399affd640 4 set_mon_vals no callback set 2023-10-30T12:14:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39997fa640 10 monclient: dump: 2023-10-30T12:14:27.538 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:27.539 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:27.540 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:27.540 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:27.540 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:27.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 client.114929.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:27.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 monclient: _renew_subs 2023-10-30T12:14:27.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:27.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a4136100 con 0x7f39a41312b0 2023-10-30T12:14:27.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 1 librados: init done 2023-10-30T12:14:27.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39997fa640 1 -- 172.21.15.50:0/764250912 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f39a0030380 con 0x7f39a41312b0 2023-10-30T12:14:27.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:27.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:27.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:27.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.500+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a411e040 con 0x7f39a41312b0 2023-10-30T12:14:27.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f39997fa640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3990039320 0x7f399003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:27.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f39997fa640 1 -- 172.21.15.50:0/764250912 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(429..429 src has 1..429) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f39a006c7a0 con 0x7f39a41312b0 2023-10-30T12:14:27.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f39997fa640 10 client.114929.objecter ms_dispatch 0x7f39a4000b90 osd_map(429..429 src has 1..429) v4 2023-10-30T12:14:27.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f39997fa640 3 client.114929.objecter handle_osd_map got epochs [429,429] > 0 2023-10-30T12:14:27.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f39997fa640 3 client.114929.objecter handle_osd_map decoding full epoch 429 2023-10-30T12:14:27.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f39997fa640 20 client.114929.objecter dump_active .. 0 homeless 2023-10-30T12:14:27.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f399bfff640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3990039320 0x7f399003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:27.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f399bfff640 10 monclient: get_auth_request con 0x7f3990039320 auth_method 0 2023-10-30T12:14:27.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f399bfff640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3990039320 0x7f399003b7e0 secure :-1 s=READY pgs=4177 cs=0 l=1 rev1=1 crypto rx=0x7f39a4134ec0 tx=0x7f3994009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:27.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.501+0000 7f39997fa640 10 client.114929.objecter ms_handle_connect 0x7f3990039320 2023-10-30T12:14:27.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.505+0000 7f39997fa640 1 -- 172.21.15.50:0/764250912 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39a002f2e0 con 0x7f39a41312b0 2023-10-30T12:14:27.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.505+0000 7f39997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:27.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.505+0000 7f39997fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:27.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.749+0000 7f39a99a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:14:27.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.749+0000 7f39a99a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:27.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.749+0000 7f39a99a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:27.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.749+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f39a404e160 con 0x7f39a41312b0 2023-10-30T12:14:27.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.750+0000 7f39997fa640 1 -- 172.21.15.50:0/764250912 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f39a0048620 con 0x7f39a41312b0 2023-10-30T12:14:27.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.750+0000 7f39997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:27.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.750+0000 7f39997fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:27.753 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":132,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:14:27.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 10 librados: watch_flush enter 2023-10-30T12:14:27.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 10 librados: watch_flush exit 2023-10-30T12:14:27.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 20 client.114929.objecter shutdown clearing up homeless session... 2023-10-30T12:14:27.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 10 client.114929.objecter successfully canceled tick 2023-10-30T12:14:27.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3990039320 msgr2=0x7f399003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:27.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3990039320 0x7f399003b7e0 secure :-1 s=READY pgs=4177 cs=0 l=1 rev1=1 crypto rx=0x7f39a4134ec0 tx=0x7f3994009040 comp rx=0 tx=0).stop 2023-10-30T12:14:27.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 10 monclient: shutdown 2023-10-30T12:14:27.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:27.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39a41312b0 msgr2=0x7f39a4133140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:27.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.751+0000 7f39a99a0640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39a41312b0 0x7f39a4133140 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f39a0033000 tx=0x7f39a00046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 shutdown_connections 2023-10-30T12:14:27.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3990039320 0x7f399003b7e0 unknown :-1 s=CLOSED pgs=4177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39a4131ae0 0x7f39a4133bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39a4131690 0x7f39a4133680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 --2- 172.21.15.50:0/764250912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39a41312b0 0x7f39a4133140 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:27.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 >> 172.21.15.50:0/764250912 conn(0x7f39a40a4200 msgr2=0x7f39a4130890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:27.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 shutdown_connections 2023-10-30T12:14:27.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 -- 172.21.15.50:0/764250912 wait complete. 2023-10-30T12:14:27.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:27.752+0000 7f39a99a0640 1 librados: shutdown 2023-10-30T12:14:29.638 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-10-30T12:14:29.636+0000 7f119d3d4640 -1 snap_mapper.add_oid found existing snaps mapped on 58:63ac8bbd:test-rados-api-smithi118-87473-97::foo:31, removing 2023-10-30T12:14:29.639 INFO:tasks.ceph.osd.4.smithi118.stderr:2023-10-30T12:14:29.637+0000 7fc0e5df8640 -1 snap_mapper.add_oid found existing snaps mapped on 58:63ac8bbd:test-rados-api-smithi118-87473-97::foo:31, removing 2023-10-30T12:14:30.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:14:30.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.884+0000 7f32e6ed4640 10 monclient: get_monmap_and_config 2023-10-30T12:14:30.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.884+0000 7f32e6ed4640 10 monclient: build_initial_monmap 2023-10-30T12:14:30.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.884+0000 7f32e6ed4640 10 monclient: monmap: 2023-10-30T12:14:30.887 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:30.887 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:30.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:30.885676+0000 2023-10-30T12:14:30.887 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:30.885676+0000 2023-10-30T12:14:30.887 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:30.888 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:30.888 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:30.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:30.888 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:30.888 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:30.888 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:30.889 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:30.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 Processor -- start 2023-10-30T12:14:30.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 -- start start 2023-10-30T12:14:30.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient: init 2023-10-30T12:14:30.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:30.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-10-30T12:14:30.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098520 0x7f32e009bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:30.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f32e0098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:30.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 0x7f32e0098d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:30.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f32e0098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:30.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f32e01326d0 0x7f32e0132b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:30.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f32e01326d0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:30.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:30.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e0099930 con 0x7f32e0098900 2023-10-30T12:14:30.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:30.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e009aca0 con 0x7f32e01326d0 2023-10-30T12:14:30.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:30.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e009af80 con 0x7f32e0098520 2023-10-30T12:14:30.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:30.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.885+0000 7f32e6ed4640 10 monclient(hunting): authenticate will time out at 1716.068891s 2023-10-30T12:14:30.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 0x7f32e0098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:30.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098520 0x7f32e009bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:30.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 0x7f32e0098d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41494/0 (socket says 172.21.15.50:41494) 2023-10-30T12:14:30.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098520 0x7f32e009bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51482/0 (socket says 172.21.15.50:51482) 2023-10-30T12:14:30.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 1 -- 172.21.15.50:0/666053456 learned_addr learned my addr 172.21.15.50:0/666053456 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:30.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 10 monclient(hunting): get_auth_request con 0x7f32e0098900 auth_method 0 2023-10-30T12:14:30.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:30.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:30.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:30.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): get_auth_request con 0x7f32e0098520 auth_method 0 2023-10-30T12:14:30.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:30.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:30.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:30.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:30.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:30.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 1 -- 172.21.15.50:0/666053456 >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 msgr2=0x7f32e0098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 1 -- 172.21.15.50:0/666053456 >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 msgr2=0x7f32e0098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 0x7f32e0098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32dffff640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 0x7f32e0098d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e5471640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.118:3300/0 conn(0x7f32e01326d0 0x7f32e0132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e5471640 10 monclient(hunting): get_auth_request con 0x7f32e01326d0 auth_method 0 2023-10-30T12:14:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e5471640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e5471640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e5471640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient(hunting): handle_auth_done global_id 114932 payload 274 2023-10-30T12:14:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 -- 172.21.15.50:0/666053456 >> v2:172.21.15.118:3300/0 conn(0x7f32e01326d0 msgr2=0x7f32e0132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.118:3300/0 conn(0x7f32e01326d0 0x7f32e0132b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 -- 172.21.15.50:0/666053456 >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 msgr2=0x7f32e0098d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 0x7f32e0098d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:30.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient: _finish_hunting 0 2023-10-30T12:14:30.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 monclient: found mon.noname-b 2023-10-30T12:14:30.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:30.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:14:30.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e4c70640 1 -- 172.21.15.50:0/666053456 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e009b260 con 0x7f32e0098520 2023-10-30T12:14:30.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.886+0000 7f32e5471640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.118:3300/0 conn(0x7f32e01326d0 0x7f32e0132b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:14:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32e4c70640 1 --2- 172.21.15.50:0/666053456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098520 0x7f32e009bf70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f32e009aba0 tx=0x7f32d402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d5a6943a002548c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 1 -- 172.21.15.50:0/666053456 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d4030710 con 0x7f32e0098520 2023-10-30T12:14:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:14:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: dump: 2023-10-30T12:14:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:30.910 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: _finish_auth 0 2023-10-30T12:14:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:14:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32e6ed4640 5 monclient: authenticate success, global_id 114932 2023-10-30T12:14:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 1 -- 172.21.15.50:0/666053456 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d402e050 con 0x7f32e0098520 2023-10-30T12:14:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32e6ed4640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32e6ed4640 4 set_mon_vals no callback set 2023-10-30T12:14:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 1 -- 172.21.15.50:0/666053456 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d4030a90 con 0x7f32e0098520 2023-10-30T12:14:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.887+0000 7f32df7fe640 10 monclient: dump: 2023-10-30T12:14:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:30.914 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:30.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 10 monclient: shutdown 2023-10-30T12:14:30.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:30.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 -- 172.21.15.50:0/666053456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098520 msgr2=0x7f32e009bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:30.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/666053456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098520 0x7f32e009bf70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f32e009aba0 tx=0x7f32d402b710 comp rx=0 tx=0).stop 2023-10-30T12:14:30.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 -- 172.21.15.50:0/666053456 shutdown_connections 2023-10-30T12:14:30.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.118:3300/0 conn(0x7f32e01326d0 0x7f32e0132b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:30.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/666053456 >> v2:172.21.15.50:3300/0 conn(0x7f32e0098900 0x7f32e0098d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:30.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/666053456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098520 0x7f32e009bf70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:30.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 -- 172.21.15.50:0/666053456 >> 172.21.15.50:0/666053456 conn(0x7f32e00a05c0 msgr2=0x7f32e009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:30.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 -- 172.21.15.50:0/666053456 shutdown_connections 2023-10-30T12:14:30.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 -- 172.21.15.50:0/666053456 wait complete. 2023-10-30T12:14:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 10 monclient: build_initial_monmap 2023-10-30T12:14:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 10 monclient: monmap: 2023-10-30T12:14:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:30.920 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 librados: starting msgr at 2023-10-30T12:14:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.888+0000 7f32e6ed4640 1 librados: starting objecter 2023-10-30T12:14:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 Processor -- start 2023-10-30T12:14:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 -- start start 2023-10-30T12:14:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 librados: setting wanted keys 2023-10-30T12:14:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 librados: calling monclient init 2023-10-30T12:14:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient: init 2023-10-30T12:14:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:30.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:14:30.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 0x7f32e0131990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:30.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): picked mon.noname-a con 0x7f32e0098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:30.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098900 0x7f32e0131ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:30.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): picked mon.noname-c con 0x7f32e0098900 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:30.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32e01326d0 0x7f32e01c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:30.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): picked mon.noname-b con 0x7f32e01326d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:30.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:30.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e012b430 con 0x7f32e0098520 2023-10-30T12:14:30.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:30.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e0071d50 con 0x7f32e01326d0 2023-10-30T12:14:30.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:30.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00a31b0 con 0x7f32e0098900 2023-10-30T12:14:30.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:30.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e6ed4640 10 monclient(hunting): authenticate will time out at 1716.072873s 2023-10-30T12:14:30.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e4c70640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 0x7f32e0131990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:30.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32dffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098900 0x7f32e0131ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.889+0000 7f32e4c70640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 0x7f32e0131990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41498/0 (socket says 172.21.15.50:41498) 2023-10-30T12:14:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): get_auth_request con 0x7f32e0098900 auth_method 0 2023-10-30T12:14:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 1 -- 172.21.15.50:0/949120730 learned_addr learned my addr 172.21.15.50:0/949120730 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 10 monclient(hunting): get_auth_request con 0x7f32e0098520 auth_method 0 2023-10-30T12:14:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32e01326d0 0x7f32e01c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 1 -- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 msgr2=0x7f32e0131990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:14:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 10 monclient(hunting): get_auth_request con 0x7f32e01326d0 auth_method 0 2023-10-30T12:14:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 1 -- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 msgr2=0x7f32e0131990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:14:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 0x7f32e0131990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e4c70640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 0x7f32e0131990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:30.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:30.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:30.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e5471640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:30.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient(hunting): handle_auth_done global_id 114935 payload 824 2023-10-30T12:14:30.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 1 -- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32e01326d0 msgr2=0x7f32e01c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:30.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32e01326d0 0x7f32e01c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:30.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 1 -- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 msgr2=0x7f32e0131990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:30.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 0x7f32e0131990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:30.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient: _finish_hunting 0 2023-10-30T12:14:30.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 1 monclient: found mon.noname-c 2023-10-30T12:14:30.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:30.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:30.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 1 -- 172.21.15.50:0/949120730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40093a0 con 0x7f32e0098900 2023-10-30T12:14:30.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dffff640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098900 0x7f32e0131ed0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f32cc00e7e0 tx=0x7f32cc00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:30.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 client.?.objecter ms_handle_connect 0x7f32e0098900 2023-10-30T12:14:30.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:30.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 1 -- 172.21.15.50:0/949120730 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc004020 con 0x7f32e0098900 2023-10-30T12:14:30.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:30.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:30.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: dump: 2023-10-30T12:14:30.932 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:30.932 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:30.933 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.933 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.933 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:30.933 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:30.933 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:30.934 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:30.934 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:30.934 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:30.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: _finish_auth 0 2023-10-30T12:14:30.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:14:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 1 -- 172.21.15.50:0/949120730 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc004180 con 0x7f32e0098900 2023-10-30T12:14:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 5 monclient: authenticate success, global_id 114935 2023-10-30T12:14:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 10 monclient: _renew_subs 2023-10-30T12:14:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:30.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e012ab90 con 0x7f32e0098900 2023-10-30T12:14:30.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:30.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 1 -- 172.21.15.50:0/949120730 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc00c6d0 con 0x7f32e0098900 2023-10-30T12:14:30.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:30.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:30.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32dd7fa640 10 monclient: dump: 2023-10-30T12:14:30.937 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:30.937 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:30.937 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.937 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:30.938 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32deffd640 4 set_mon_vals no callback set 2023-10-30T12:14:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 10 client.114935.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 10 monclient: _renew_subs 2023-10-30T12:14:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e0132410 con 0x7f32e0098900 2023-10-30T12:14:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.890+0000 7f32e6ed4640 1 librados: init done 2023-10-30T12:14:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.891+0000 7f32dd7fa640 1 -- 172.21.15.50:0/949120730 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f32cc00c830 con 0x7f32e0098900 2023-10-30T12:14:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.891+0000 7f32e6ed4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.891+0000 7f32e6ed4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.891+0000 7f32e6ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.891+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e009c5d0 con 0x7f32e0098900 2023-10-30T12:14:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32dd7fa640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32c40392f0 0x7f32c403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32dd7fa640 1 -- 172.21.15.50:0/949120730 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(430..430 src has 1..430) v4 ==== 5914+0+0 (secure 0 0 0) 0x7f32cc04a4d0 con 0x7f32e0098900 2023-10-30T12:14:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32dd7fa640 10 client.114935.objecter ms_dispatch 0x7f32e0000b90 osd_map(430..430 src has 1..430) v4 2023-10-30T12:14:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32dd7fa640 3 client.114935.objecter handle_osd_map got epochs [430,430] > 0 2023-10-30T12:14:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32dd7fa640 3 client.114935.objecter handle_osd_map decoding full epoch 430 2023-10-30T12:14:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32dd7fa640 20 client.114935.objecter dump_active .. 0 homeless 2023-10-30T12:14:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32e4c70640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32c40392f0 0x7f32c403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32e4c70640 10 monclient: get_auth_request con 0x7f32c40392f0 auth_method 0 2023-10-30T12:14:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32dd7fa640 10 client.114935.objecter ms_handle_connect 0x7f32c40392f0 2023-10-30T12:14:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.892+0000 7f32e4c70640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32c40392f0 0x7f32c403b7b0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 crypto rx=0x7f32d402b750 tx=0x7f32d402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:30.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.896+0000 7f32dd7fa640 1 -- 172.21.15.50:0/949120730 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32cc009390 con 0x7f32e0098900 2023-10-30T12:14:30.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.896+0000 7f32dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:30.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:30.896+0000 7f32dd7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:31.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.140+0000 7f32e6ed4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:14:31.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.140+0000 7f32e6ed4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:31.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.140+0000 7f32e6ed4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:31.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.140+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f32e004e160 con 0x7f32e0098900 2023-10-30T12:14:31.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.140+0000 7f32dd7fa640 1 -- 172.21.15.50:0/949120730 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f32cc020020 con 0x7f32e0098900 2023-10-30T12:14:31.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.140+0000 7f32dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:31.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.140+0000 7f32dd7fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:31.143 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":132,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:14:31.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 10 librados: watch_flush enter 2023-10-30T12:14:31.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 10 librados: watch_flush exit 2023-10-30T12:14:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 20 client.114935.objecter shutdown clearing up homeless session... 2023-10-30T12:14:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 10 client.114935.objecter successfully canceled tick 2023-10-30T12:14:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32c40392f0 msgr2=0x7f32c403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:31.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32c40392f0 0x7f32c403b7b0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 crypto rx=0x7f32d402b750 tx=0x7f32d402f000 comp rx=0 tx=0).stop 2023-10-30T12:14:31.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 10 monclient: shutdown 2023-10-30T12:14:31.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:31.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098900 msgr2=0x7f32e0131ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:31.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098900 0x7f32e0131ed0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f32cc00e7e0 tx=0x7f32cc00eca0 comp rx=0 tx=0).stop 2023-10-30T12:14:31.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 shutdown_connections 2023-10-30T12:14:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32c40392f0 0x7f32c403b7b0 unknown :-1 s=CLOSED pgs=4178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32e01326d0 0x7f32e01c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32e0098900 0x7f32e0131ed0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 --2- 172.21.15.50:0/949120730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32e0098520 0x7f32e0131990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 >> 172.21.15.50:0/949120730 conn(0x7f32e00a05c0 msgr2=0x7f32e01b9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:31.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 shutdown_connections 2023-10-30T12:14:31.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 -- 172.21.15.50:0/949120730 wait complete. 2023-10-30T12:14:31.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:31.142+0000 7f32e6ed4640 1 librados: shutdown 2023-10-30T12:14:34.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:14:34.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.274+0000 7f93c6821640 10 monclient: get_monmap_and_config 2023-10-30T12:14:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.274+0000 7f93c6821640 10 monclient: build_initial_monmap 2023-10-30T12:14:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.274+0000 7f93c6821640 10 monclient: monmap: 2023-10-30T12:14:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:34.276 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:34.275464+0000 2023-10-30T12:14:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:34.275464+0000 2023-10-30T12:14:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:34.277 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:34.278 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 1 Processor -- start 2023-10-30T12:14:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 1 -- start start 2023-10-30T12:14:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient: init 2023-10-30T12:14:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:34.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:14:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient(hunting): picked mon.noname-a con 0x7f93c009f5b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient(hunting): picked mon.noname-c con 0x7f93c0096080 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93c00969d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c0098810 con 0x7f93c00969d0 2023-10-30T12:14:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93c009f5b0 2023-10-30T12:14:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c009a1a0 con 0x7f93c0096080 2023-10-30T12:14:34.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c6821640 10 monclient(hunting): authenticate will time out at 1719.458687s 2023-10-30T12:14:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41510/0 (socket says 172.21.15.50:41510) 2023-10-30T12:14:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93c4dbe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f93c00969d0 0x7f93c009cb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59904/0 (socket says 172.21.15.50:59904) 2023-10-30T12:14:34.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 learned_addr learned my addr 172.21.15.50:0/586146104 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93c009f5b0 auth_method 0 2023-10-30T12:14:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:34.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.275+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:34.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93c0096080 auth_method 0 2023-10-30T12:14:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:34.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:34.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.276+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:34.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93c009f5b0 auth_method 0 2023-10-30T12:14:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:34.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93c0096080 auth_method 0 2023-10-30T12:14:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:34.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:34.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.477+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93c009f5b0 auth_method 0 2023-10-30T12:14:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:34.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:34.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93c0096080 auth_method 0 2023-10-30T12:14:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:34.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:34.878+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:35.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:35.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93c009f5b0 auth_method 0 2023-10-30T12:14:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93c0096080 auth_method 0 2023-10-30T12:14:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:35.679+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient(hunting): tick 2023-10-30T12:14:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:14:37.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:14:37.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:14:37.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 msgr2=0x7f93c0096490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:37.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93c0096080 0x7f93c0096490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:37.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 msgr2=0x7f93c0099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:37.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93c009f5b0 0x7f93c0099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.50:6789/0 conn(0x7f93c00969d0 legacy=0x7f93c009cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient: _add_conns ranks=[1,2] 2023-10-30T12:14:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93ac0036e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac007390 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:37.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c0096e60 con 0x7f93ac0036e0 2023-10-30T12:14:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93ac007390 2023-10-30T12:14:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bdffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:37.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:37.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:37.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:37.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:37.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:37.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:37.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.276+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.477+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:37.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:37.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:37.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:37.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:37.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:37.878+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:38.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:38.679+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:40.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): tick 2023-10-30T12:14:40.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:14:40.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:14:40.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:14:40.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:40.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:40.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.118:6789/0 conn(0x7f93ac0036e0 legacy=0x7f93ac005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:40.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-10-30T12:14:40.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 reap_dead start 2023-10-30T12:14:40.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:40.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac007390 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:40.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93ac00a080 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:40.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:40.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac00a500 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:40.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac00afc0 con 0x7f93ac00a080 2023-10-30T12:14:40.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:40.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93ac00a500 2023-10-30T12:14:40.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:40.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c009a1a0 con 0x7f93ac007390 2023-10-30T12:14:40.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bdffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:40.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:40.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:40.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:40.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:40.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:40.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:40.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:40.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:40.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:40.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:40.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac00a500 auth_method 0 2023-10-30T12:14:40.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:40.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:40.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:40.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:40.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:40.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:40.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.276+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:40.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:40.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:40.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:40.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:40.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:40.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac00a500 auth_method 0 2023-10-30T12:14:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:40.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:40.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.477+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:40.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:40.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:40.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:40.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:40.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:40.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac00a500 auth_method 0 2023-10-30T12:14:40.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:40.878+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:41.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:41.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:41.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:41.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:41.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:41.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:41.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:41.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:41.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:41.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac00a500 auth_method 0 2023-10-30T12:14:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.679+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.680+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:41.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.680+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:41.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.680+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:41.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:41.680+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:43.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient(hunting): tick 2023-10-30T12:14:43.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:14:43.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:14:43.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:14:43.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:43.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:43.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 msgr2=0x7f93ac00a980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:43.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac00a500 0x7f93ac00a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:43.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.50:6789/0 conn(0x7f93ac00a080 legacy=0x7f93ac005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:43.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient: _add_conns ranks=[3,1] 2023-10-30T12:14:43.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac00d4c0 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93ac00dfa0 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac00e4d0 con 0x7f93ac00dfa0 2023-10-30T12:14:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:43.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93ac00d4c0 2023-10-30T12:14:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bdffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac00d4c0 auth_method 0 2023-10-30T12:14:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:43.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.277+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:43.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:43.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac00d4c0 auth_method 0 2023-10-30T12:14:43.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:43.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:43.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:43.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:43.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:43.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:43.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.478+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:43.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:43.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac00d4c0 auth_method 0 2023-10-30T12:14:43.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:43.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:43.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:43.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:43.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:43.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:43.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:43.879+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac00d4c0 auth_method 0 2023-10-30T12:14:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:44.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:44.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:44.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:44.680+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:46.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 10 monclient(hunting): tick 2023-10-30T12:14:46.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:14:46.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:14:46.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:14:46.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 msgr2=0x7f93ac00d8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:46.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac00d4c0 0x7f93ac00d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:46.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.118:6789/0 conn(0x7f93ac00dfa0 legacy=0x7f93ac012750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:46.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 reap_dead start 2023-10-30T12:14:46.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-10-30T12:14:46.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.276+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93ac007390 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:46.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:46.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac00aa20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:46.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:46.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac0147b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:46.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac014fe0 con 0x7f93ac007390 2023-10-30T12:14:46.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:46.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93ac0147b0 2023-10-30T12:14:46.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:46.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c009a1a0 con 0x7f93ac00aa20 2023-10-30T12:14:46.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bdffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:46.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:46.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac00aa20 auth_method 0 2023-10-30T12:14:46.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:46.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:46.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:46.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:46.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:46.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:46.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:46.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac0147b0 auth_method 0 2023-10-30T12:14:46.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:46.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:46.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:46.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:46.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:46.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:46.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:46.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.277+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:46.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:46.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac00aa20 auth_method 0 2023-10-30T12:14:46.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:46.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:46.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:46.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:46.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac0147b0 auth_method 0 2023-10-30T12:14:46.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:46.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:46.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:46.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:46.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:46.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:46.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.478+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac00aa20 auth_method 0 2023-10-30T12:14:46.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac0147b0 auth_method 0 2023-10-30T12:14:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:46.879+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac00aa20 auth_method 0 2023-10-30T12:14:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac0147b0 auth_method 0 2023-10-30T12:14:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:14:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:47.680+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient(hunting): tick 2023-10-30T12:14:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:14:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:14:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:14:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 msgr2=0x7f93ac00d8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac00aa20 0x7f93ac00d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 msgr2=0x7f93ac009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.50:3300/0 conn(0x7f93ac0147b0 0x7f93ac009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.50:6789/0 conn(0x7f93ac007390 legacy=0x7f93ac0140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient: _add_conns ranks=[1,4] 2023-10-30T12:14:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93ac0062f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac006830 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac014bd0 con 0x7f93ac0062f0 2023-10-30T12:14:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93ac006830 2023-10-30T12:14:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bdffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac006830 auth_method 0 2023-10-30T12:14:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.277+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac006830 auth_method 0 2023-10-30T12:14:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:49.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.478+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:49.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:49.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac006830 auth_method 0 2023-10-30T12:14:49.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:49.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:49.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:49.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:49.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:49.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:49.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:49.879+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:50.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:50.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93ac006830 auth_method 0 2023-10-30T12:14:50.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:50.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:50.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:50.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:50.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:50.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:50.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:50.681+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:14:51.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:51.601+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.118:6789/0 conn(0x7f93ac0062f0 legacy=0x7f93ac012750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-10-30T12:14:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:51.601+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.118:6789/0 conn(0x7f93ac0062f0 legacy=0x7f93ac012750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:14:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:51.601+0000 7f93bffff640 1 --1- 172.21.15.50:0/586146104 >> v1:172.21.15.118:6789/0 conn(0x7f93ac0062f0 0x7f93ac012750 :-1 s=OPENED pgs=63 cs=1 l=1).handle_message read tag failed 2023-10-30T12:14:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:51.601+0000 7f93bffff640 1 --1- 172.21.15.50:0/586146104 >> v1:172.21.15.118:6789/0 conn(0x7f93ac0062f0 0x7f93ac012750 :-1 s=OPENED pgs=63 cs=1 l=1).fault on lossy channel, failing 2023-10-30T12:14:51.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:51.601+0000 7f93beffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.118:6789/0 2023-10-30T12:14:52.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): tick 2023-10-30T12:14:52.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:14:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:14:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:14:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac006830 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.118:6789/0 conn(0x7f93ac0062f0 legacy=0x7f93ac012750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-10-30T12:14:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 reap_dead start 2023-10-30T12:14:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac005df0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93ac0062f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac007390 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac005b60 con 0x7f93ac0062f0 2023-10-30T12:14:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93ac005df0 2023-10-30T12:14:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c009a1a0 con 0x7f93ac007390 2023-10-30T12:14:52.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bdffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac005df0 auth_method 0 2023-10-30T12:14:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:52.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:52.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 msgr2=0x7f93ac004b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 msgr2=0x7f93ac004b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.278+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:52.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:52.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:52.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:52.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac005df0 auth_method 0 2023-10-30T12:14:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:52.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 msgr2=0x7f93ac004b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 msgr2=0x7f93ac004b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:52.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.479+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:14:52.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:52.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:52.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:14:52.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:52.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:52.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:52.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac005df0 auth_method 0 2023-10-30T12:14:52.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:52.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 msgr2=0x7f93ac004b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:14:52.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 msgr2=0x7f93ac004b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:14:52.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:14:52.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.880+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:14:52.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.927+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980002943 0 0) 0x7f93ac005b60 con 0x7f93ac0062f0 2023-10-30T12:14:52.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.927+0000 7f93beffd640 10 monclient(hunting): _negotiate 2023-10-30T12:14:52.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.927+0000 7f93beffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.927+0000 7f93beffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:52.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.927+0000 7f93beffd640 10 monclient(hunting): my global_id is 194097 2023-10-30T12:14:52.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.927+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a8003520 con 0x7f93ac0062f0 2023-10-30T12:14:52.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 849265288 0 0) 0x7f93a8003520 con 0x7f93ac0062f0 2023-10-30T12:14:52.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93ac006c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:52.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93ac006c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 msgr2=0x7f93ac004b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:52.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 --2- 172.21.15.50:0/586146104 >> v2:172.21.15.118:3300/0 conn(0x7f93ac005df0 0x7f93ac004b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: _finish_hunting 0 2023-10-30T12:14:52.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:14:52.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:52.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:14:52.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac0058d0 con 0x7f93ac0062f0 2023-10-30T12:14:52.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: _finish_auth 0 2023-10-30T12:14:52.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: _check_auth_tickets 2023-10-30T12:14:52.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:52.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f93b4017660 con 0x7f93ac0062f0 2023-10-30T12:14:52.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93c6821640 5 monclient: authenticate success, global_id 194097 2023-10-30T12:14:52.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93c6821640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:52.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:52.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:14:52.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: dump: 2023-10-30T12:14:52.939 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:52.939 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:52.939 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.939 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.939 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:52.939 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:52.940 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:52.940 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:52.940 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:52.940 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:52.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:14:52.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:52.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 >> v1:172.21.15.50:6789/0 conn(0x7f93ac0062f0 legacy=0x7f93ac005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:52.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:14:52.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93a8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): picked mon.c con 0x7f93ac007390 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:52.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93a8003fa0 0x7f93a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): picked mon.a con 0x7f93a8003fa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:52.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8004d00 0x7f93a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): picked mon.b con 0x7f93a8004d00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:52.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c006aef0 con 0x7f93a8003fa0 2023-10-30T12:14:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c009a1a0 con 0x7f93a8004d00 2023-10-30T12:14:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a800bb40 con 0x7f93ac007390 2023-10-30T12:14:52.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:52.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.930+0000 7f93beffd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:14:52.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c6821640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:52.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93a8003fa0 0x7f93a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93a8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93a8003fa0 auth_method 0 2023-10-30T12:14:52.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:52.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93ac007390 auth_method 0 2023-10-30T12:14:52.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:52.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:52.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:52.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8004d00 0x7f93a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93a8004d00 auth_method 0 2023-10-30T12:14:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:52.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:52.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:52.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:52.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93c4dbe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient(hunting): handle_auth_done global_id 194097 payload 293 2023-10-30T12:14:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8004d00 msgr2=0x7f93a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8004d00 0x7f93a8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93a8003fa0 msgr2=0x7f93a80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93a8003fa0 0x7f93a80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient: _finish_hunting 0 2023-10-30T12:14:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 1 monclient: found mon.c 2023-10-30T12:14:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 1 -- 172.21.15.50:0/586146104 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a8003520 con 0x7f93ac007390 2023-10-30T12:14:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient: _finish_auth 0 2023-10-30T12:14:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 10 monclient: _check_auth_tickets 2023-10-30T12:14:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.931+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93a8003fa0 0x7f93a80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93bffff640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93a8003970 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f93b0016d50 tx=0x7f93a80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be29fa6973492f3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0004230 con 0x7f93ac007390 2023-10-30T12:14:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 10 monclient: dump: 2023-10-30T12:14:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:52.954 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:52.954 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:52.954 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:52.954 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:52.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b0033070 con 0x7f93ac007390 2023-10-30T12:14:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 1 -- 172.21.15.50:0/586146104 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b002f7d0 con 0x7f93ac007390 2023-10-30T12:14:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93c6821640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93c6821640 4 set_mon_vals no callback set 2023-10-30T12:14:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:52.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.932+0000 7f93beffd640 10 monclient: dump: 2023-10-30T12:14:52.956 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:52.956 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:52.956 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.956 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:52.958 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 10 monclient: shutdown 2023-10-30T12:14:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 1 -- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 msgr2=0x7f93a8003970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 1 --2- 172.21.15.50:0/586146104 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93ac007390 0x7f93a8003970 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f93b0016d50 tx=0x7f93a80057f0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c4dbe640 1 -- 172.21.15.50:0/586146104 reap_dead start 2023-10-30T12:14:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 1 -- 172.21.15.50:0/586146104 shutdown_connections 2023-10-30T12:14:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 1 -- 172.21.15.50:0/586146104 >> 172.21.15.50:0/586146104 conn(0x7f93c00a05c0 msgr2=0x7f93c009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 1 -- 172.21.15.50:0/586146104 shutdown_connections 2023-10-30T12:14:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.933+0000 7f93c6821640 1 -- 172.21.15.50:0/586146104 wait complete. 2023-10-30T12:14:52.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.934+0000 7f93c6821640 10 monclient: build_initial_monmap 2023-10-30T12:14:52.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.934+0000 7f93c6821640 10 monclient: monmap: 2023-10-30T12:14:52.964 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:52.964 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:52.964 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:52.964 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:52.966 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.934+0000 7f93c6821640 1 librados: starting msgr at 2023-10-30T12:14:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.934+0000 7f93c6821640 1 librados: starting objecter 2023-10-30T12:14:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.934+0000 7f93c6821640 1 Processor -- start 2023-10-30T12:14:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 -- start start 2023-10-30T12:14:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 librados: setting wanted keys 2023-10-30T12:14:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 librados: calling monclient init 2023-10-30T12:14:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient: init 2023-10-30T12:14:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:14:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93c0095220 0x7f93c01ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): picked mon.noname-a con 0x7f93c0095220 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:52.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93c0095600 0x7f93c008c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): picked mon.noname-b con 0x7f93c0095600 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:52.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93c0095a40 0x7f93c008c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): picked mon.noname-c con 0x7f93c0095a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:52.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c008fad0 con 0x7f93c0095220 2023-10-30T12:14:52.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c008f410 con 0x7f93c0095600 2023-10-30T12:14:52.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:52.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c008de90 con 0x7f93c0095a40 2023-10-30T12:14:52.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:52.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93c6821640 10 monclient(hunting): authenticate will time out at 1738.118863s 2023-10-30T12:14:52.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.935+0000 7f93bffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93c0095220 0x7f93c01ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93c0095a40 0x7f93c008c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93c0095220 0x7f93c01ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55312/0 (socket says 172.21.15.50:55312) 2023-10-30T12:14:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93c0095a40 0x7f93c008c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34098/0 (socket says 172.21.15.50:34098) 2023-10-30T12:14:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 -- 172.21.15.50:0/451465988 learned_addr learned my addr 172.21.15.50:0/451465988 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): get_auth_request con 0x7f93c0095220 auth_method 0 2023-10-30T12:14:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:52.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request con 0x7f93c0095a40 auth_method 0 2023-10-30T12:14:52.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:52.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93c0095600 0x7f93c008c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request con 0x7f93c0095600 auth_method 0 2023-10-30T12:14:52.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:52.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:52.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:52.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:52.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:52.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:52.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:52.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:52.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient(hunting): handle_auth_done global_id 194100 payload 824 2023-10-30T12:14:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 -- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93c0095a40 msgr2=0x7f93c008c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93c0095a40 0x7f93c008c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 -- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93c0095600 msgr2=0x7f93c008c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93c0095600 0x7f93c008c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient: _finish_hunting 0 2023-10-30T12:14:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 monclient: found mon.noname-a 2023-10-30T12:14:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 -- 172.21.15.50:0/451465988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c0072010 con 0x7f93c0095220 2023-10-30T12:14:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c4dbe640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93c0095a40 0x7f93c008c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bffff640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93c0095220 0x7f93c01ca320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f93b0001eb0 tx=0x7f93b0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 10 client.?.objecter ms_handle_connect 0x7f93c0095220 2023-10-30T12:14:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 1 -- 172.21.15.50:0/451465988 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0046070 con 0x7f93c0095220 2023-10-30T12:14:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:52.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 10 monclient: dump: 2023-10-30T12:14:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:52.980 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 10 monclient: _finish_auth 0 2023-10-30T12:14:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 10 monclient: _check_auth_tickets 2023-10-30T12:14:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93bcff9640 1 -- 172.21.15.50:0/451465988 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b001e040 con 0x7f93c0095220 2023-10-30T12:14:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c6821640 5 monclient: authenticate success, global_id 194100 2023-10-30T12:14:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c6821640 10 monclient: _renew_subs 2023-10-30T12:14:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c6821640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.936+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c01bb6e0 con 0x7f93c0095220 2023-10-30T12:14:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93bcff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93bcff9640 1 -- 172.21.15.50:0/451465988 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b002fa30 con 0x7f93c0095220 2023-10-30T12:14:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93bcff9640 10 monclient: dump: 2023-10-30T12:14:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:52.985 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93c6821640 10 client.194100.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93c6821640 10 monclient: _renew_subs 2023-10-30T12:14:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93be7fc640 4 set_mon_vals no callback set 2023-10-30T12:14:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93c6821640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c008d900 con 0x7f93c0095220 2023-10-30T12:14:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93c6821640 1 librados: init done 2023-10-30T12:14:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.937+0000 7f93bcff9640 1 -- 172.21.15.50:0/451465988 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f93b002fb90 con 0x7f93c0095220 2023-10-30T12:14:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bcff9640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f93ac03c090 0x7f93ac03e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bcff9640 1 -- 172.21.15.50:0/451465988 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(433..433 src has 1..433) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f93b006e180 con 0x7f93c0095220 2023-10-30T12:14:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bcff9640 10 client.194100.objecter ms_dispatch 0x7f93c0000b90 osd_map(433..433 src has 1..433) v4 2023-10-30T12:14:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bcff9640 3 client.194100.objecter handle_osd_map got epochs [433,433] > 0 2023-10-30T12:14:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bcff9640 3 client.194100.objecter handle_osd_map decoding full epoch 433 2023-10-30T12:14:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bcff9640 20 client.194100.objecter dump_active .. 0 homeless 2023-10-30T12:14:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f93ac03c090 0x7f93ac03e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93bf7fe640 10 monclient: get_auth_request con 0x7f93ac03c090 auth_method 0 2023-10-30T12:14:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93c6821640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93c6821640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93c6821640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.938+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c01bb6e0 con 0x7f93c0095220 2023-10-30T12:14:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.939+0000 7f93bf7fe640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f93ac03c090 0x7f93ac03e550 secure :-1 s=READY pgs=4179 cs=0 l=1 rev1=1 crypto rx=0x7f93ac007220 tx=0x7f93b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.939+0000 7f93bcff9640 10 client.194100.objecter ms_handle_connect 0x7f93ac03c090 2023-10-30T12:14:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.943+0000 7f93bcff9640 1 -- 172.21.15.50:0/451465988 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f93b002fe40 con 0x7f93c0095220 2023-10-30T12:14:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.943+0000 7f93bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:52.943+0000 7f93bcff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.195+0000 7f93c6821640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:14:53.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.195+0000 7f93c6821640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.195+0000 7f93c6821640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.195+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f93c01bb6e0 con 0x7f93c0095220 2023-10-30T12:14:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.196+0000 7f93bcff9640 1 -- 172.21.15.50:0/451465988 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f93b0041390 con 0x7f93c0095220 2023-10-30T12:14:53.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.196+0000 7f93bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:53.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.196+0000 7f93bcff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:53.199 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:14:53.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 10 librados: watch_flush enter 2023-10-30T12:14:53.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 10 librados: watch_flush exit 2023-10-30T12:14:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 20 client.194100.objecter shutdown clearing up homeless session... 2023-10-30T12:14:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 10 client.194100.objecter successfully canceled tick 2023-10-30T12:14:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f93ac03c090 msgr2=0x7f93ac03e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f93ac03c090 0x7f93ac03e550 secure :-1 s=READY pgs=4179 cs=0 l=1 rev1=1 crypto rx=0x7f93ac007220 tx=0x7f93b4008040 comp rx=0 tx=0).stop 2023-10-30T12:14:53.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 10 monclient: shutdown 2023-10-30T12:14:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93c0095220 msgr2=0x7f93c01ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93c0095220 0x7f93c01ca320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f93b0001eb0 tx=0x7f93b0008040 comp rx=0 tx=0).stop 2023-10-30T12:14:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 shutdown_connections 2023-10-30T12:14:53.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f93ac03c090 0x7f93ac03e550 unknown :-1 s=CLOSED pgs=4179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f93c0095a40 0x7f93c008c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93c0095600 0x7f93c008c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 --2- 172.21.15.50:0/451465988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93c0095220 0x7f93c01ca320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 >> 172.21.15.50:0/451465988 conn(0x7f93c00a05c0 msgr2=0x7f93c0070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 shutdown_connections 2023-10-30T12:14:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 -- 172.21.15.50:0/451465988 wait complete. 2023-10-30T12:14:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.198+0000 7f93c6821640 1 librados: shutdown 2023-10-30T12:14:53.210 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:14:53.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:14:53.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:14:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.315+0000 7f4022100640 10 monclient: get_monmap_and_config 2023-10-30T12:14:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.315+0000 7f4022100640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.315+0000 7f4022100640 10 monclient: monmap: 2023-10-30T12:14:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:53.316817+0000 2023-10-30T12:14:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:53.316817+0000 2023-10-30T12:14:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:53.320 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 1 Processor -- start 2023-10-30T12:14:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 1 -- start start 2023-10-30T12:14:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient: init 2023-10-30T12:14:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:14:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f401c071c00 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:53.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f401c097fe0 0x7f401c09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient(hunting): picked mon.noname-c con 0x7f401c097fe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:53.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f401c09b170 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:53.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c097600 con 0x7f401c09b170 2023-10-30T12:14:53.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c0978e0 con 0x7f401c071c00 2023-10-30T12:14:53.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401c09bf50 con 0x7f401c097fe0 2023-10-30T12:14:53.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:53.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.316+0000 7f4022100640 10 monclient(hunting): authenticate will time out at 1738.499960s 2023-10-30T12:14:53.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f401c09b170 0x7f401c139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40444/0 (socket says 172.21.15.50:40444) 2023-10-30T12:14:53.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401bfff640 1 -- 172.21.15.50:0/3321118303 learned_addr learned my addr 172.21.15.50:0/3321118303 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:14:53.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 1 --2- 172.21.15.50:0/3321118303 >> v2:172.21.15.118:3300/0 conn(0x7f401c097fe0 0x7f401c09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): get_auth_request con 0x7f401c097fe0 auth_method 0 2023-10-30T12:14:53.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 1 -- 172.21.15.50:0/3321118303 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271208333 0 0) 0x7f401c097600 con 0x7f401c09b170 2023-10-30T12:14:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:14:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 10 monclient(hunting): my global_id is 194103 2023-10-30T12:14:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 1 -- 172.21.15.50:0/3321118303 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4004003520 con 0x7f401c09b170 2023-10-30T12:14:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient(hunting): handle_auth_done global_id 184201 payload 274 2023-10-30T12:14:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 1 -- 172.21.15.50:0/3321118303 >> v1:172.21.15.118:6789/0 conn(0x7f401c071c00 legacy=0x7f401c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 1 -- 172.21.15.50:0/3321118303 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188725949 0 0) 0x7f401c0978e0 con 0x7f401c071c00 2023-10-30T12:14:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 1 -- 172.21.15.50:0/3321118303 >> v1:172.21.15.50:6789/0 conn(0x7f401c09b170 legacy=0x7f401c139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient: _finish_hunting 0 2023-10-30T12:14:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 1 monclient: found mon.noname-c 2023-10-30T12:14:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:14:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401affd640 1 -- 172.21.15.50:0/3321118303 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c09d9b0 con 0x7f401c097fe0 2023-10-30T12:14:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.317+0000 7f401a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:14:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401affd640 1 --2- 172.21.15.50:0/3321118303 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c097fe0 0x7f401c09a150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40100097a0 tx=0x7f401002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45fcc28e92a07714 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 1 -- 172.21.15.50:0/3321118303 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401002e070 con 0x7f401c097fe0 2023-10-30T12:14:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:53.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: dump: 2023-10-30T12:14:53.337 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.337 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.337 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.337 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.338 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.338 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.338 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.338 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.338 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.339 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:14:53.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:14:53.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:53.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f4022100640 5 monclient: authenticate success, global_id 184201 2023-10-30T12:14:53.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f4022100640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:53.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 1 -- 172.21.15.50:0/3321118303 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40100315a0 con 0x7f401c097fe0 2023-10-30T12:14:53.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:53.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f4022100640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:53.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f4022100640 4 set_mon_vals no callback set 2023-10-30T12:14:53.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 1 -- 172.21.15.50:0/3321118303 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010031930 con 0x7f401c097fe0 2023-10-30T12:14:53.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:53.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.318+0000 7f401a7fc640 10 monclient: dump: 2023-10-30T12:14:53.341 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.341 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.342 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.342 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.342 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.342 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.342 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.343 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.343 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.343 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 10 monclient: shutdown 2023-10-30T12:14:53.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 -- 172.21.15.50:0/3321118303 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c097fe0 msgr2=0x7f401c09a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 --2- 172.21.15.50:0/3321118303 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c097fe0 0x7f401c09a150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40100097a0 tx=0x7f401002b840 comp rx=0 tx=0).stop 2023-10-30T12:14:53.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 -- 172.21.15.50:0/3321118303 shutdown_connections 2023-10-30T12:14:53.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 --2- 172.21.15.50:0/3321118303 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c097fe0 0x7f401c09a150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 -- 172.21.15.50:0/3321118303 >> 172.21.15.50:0/3321118303 conn(0x7f401c0a3980 msgr2=0x7f401c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 -- 172.21.15.50:0/3321118303 shutdown_connections 2023-10-30T12:14:53.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 -- 172.21.15.50:0/3321118303 wait complete. 2023-10-30T12:14:53.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 10 monclient: monmap: 2023-10-30T12:14:53.345 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.345 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:53.346 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:53.346 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:53.346 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.346 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.346 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:53.347 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 librados: starting msgr at 2023-10-30T12:14:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 librados: starting objecter 2023-10-30T12:14:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.319+0000 7f4022100640 1 Processor -- start 2023-10-30T12:14:53.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 -- start start 2023-10-30T12:14:53.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 librados: setting wanted keys 2023-10-30T12:14:53.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 librados: calling monclient init 2023-10-30T12:14:53.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient: init 2023-10-30T12:14:53.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:14:53.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c071c00 0x7f401c1367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): picked mon.noname-b con 0x7f401c071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:53.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f401c097fe0 0x7f401c136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): picked mon.noname-c con 0x7f401c097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:53.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c09b170 0x7f401c137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): picked mon.noname-a con 0x7f401c09b170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:53.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1388a0 con 0x7f401c09b170 2023-10-30T12:14:53.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c13a190 con 0x7f401c071c00 2023-10-30T12:14:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c13a470 con 0x7f401c097fe0 2023-10-30T12:14:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f4022100640 10 monclient(hunting): authenticate will time out at 1738.503595s 2023-10-30T12:14:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c09b170 0x7f401c137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f401c097fe0 0x7f401c136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c09b170 0x7f401c137230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55318/0 (socket says 172.21.15.50:55318) 2023-10-30T12:14:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 1 -- 172.21.15.50:0/2784109118 learned_addr learned my addr 172.21.15.50:0/2784109118 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 10 monclient(hunting): get_auth_request con 0x7f401c09b170 auth_method 0 2023-10-30T12:14:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 10 monclient(hunting): get_auth_request con 0x7f401c097fe0 auth_method 0 2023-10-30T12:14:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c071c00 0x7f401c1367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 10 monclient(hunting): get_auth_request con 0x7f401c071c00 auth_method 0 2023-10-30T12:14:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.320+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 10 monclient(hunting): handle_auth_done global_id 194106 payload 824 2023-10-30T12:14:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f401c097fe0 msgr2=0x7f401c136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f401c097fe0 0x7f401c136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c071c00 msgr2=0x7f401c1367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f401c071c00 0x7f401c1367b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:14:53.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 1 monclient: found mon.noname-a 2023-10-30T12:14:53.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:53.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 1 -- 172.21.15.50:0/2784109118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4010009400 con 0x7f401c09b170 2023-10-30T12:14:53.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f401bfff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c09b170 0x7f401c137230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f401400cb10 tx=0x7f401400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 client.?.objecter ms_handle_connect 0x7f401c09b170 2023-10-30T12:14:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401400ac70 con 0x7f401c09b170 2023-10-30T12:14:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:53.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: dump: 2023-10-30T12:14:53.360 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.360 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.360 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.360 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.360 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.361 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.361 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.361 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.361 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.361 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: _finish_auth 0 2023-10-30T12:14:53.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: _check_auth_tickets 2023-10-30T12:14:53.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:53.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 5 monclient: authenticate success, global_id 194106 2023-10-30T12:14:53.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 10 monclient: _renew_subs 2023-10-30T12:14:53.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401400add0 con 0x7f401c09b170 2023-10-30T12:14:53.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c138ce0 con 0x7f401c09b170 2023-10-30T12:14:53.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:53.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4014012620 con 0x7f401c09b170 2023-10-30T12:14:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4003fff640 10 monclient: dump: 2023-10-30T12:14:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.365 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.365 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.365 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.365 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.365 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.366 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.366 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4019ffb640 4 set_mon_vals no callback set 2023-10-30T12:14:53.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 10 client.194106.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:53.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 10 monclient: _renew_subs 2023-10-30T12:14:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c13c4f0 con 0x7f401c09b170 2023-10-30T12:14:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.321+0000 7f4022100640 1 librados: init done 2023-10-30T12:14:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4014007070 con 0x7f401c09b170 2023-10-30T12:14:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4022100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:53.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4022100640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1381c0 0x7f401c1c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4022100640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c073ff0 con 0x7f401c1381c0 2023-10-30T12:14:53.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1381c0 0x7f401c1c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40080392f0 0x7f400803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient: get_auth_request con 0x7f401c1381c0 auth_method 0 2023-10-30T12:14:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(433..433 src has 1..433) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f401404ac50 con 0x7f401c09b170 2023-10-30T12:14:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 10 client.194106.objecter ms_dispatch 0x7f401c000b90 osd_map(433..433 src has 1..433) v4 2023-10-30T12:14:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 3 client.194106.objecter handle_osd_map got epochs [433,433] > 0 2023-10-30T12:14:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 3 client.194106.objecter handle_osd_map decoding full epoch 433 2023-10-30T12:14:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 20 client.194106.objecter dump_active .. 0 homeless 2023-10-30T12:14:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401affd640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40080392f0 0x7f400803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 10 monclient(hunting): handle_auth_done global_id 194109 payload 824 2023-10-30T12:14:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 1 -- 172.21.15.50:0/2784109118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f401c071fe0 con 0x7f401c1381c0 2023-10-30T12:14:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401affd640 10 monclient: get_auth_request con 0x7f40080392f0 auth_method 0 2023-10-30T12:14:53.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 10 client.194106.objecter ms_handle_connect 0x7f401c1381c0 2023-10-30T12:14:53.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f4003fff640 10 client.194106.objecter resend_mon_ops 2023-10-30T12:14:53.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.322+0000 7f401b7fe640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1381c0 0x7f401c1c7630 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f770 tx=0x7f400c00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c0050d0 con 0x7f401c1381c0 2023-10-30T12:14:53.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 10 monclient: dump: 2023-10-30T12:14:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.374 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.374 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.374 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.374 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.374 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.375 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.375 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 10 client.194106.objecter ms_handle_connect 0x7f40080392f0 2023-10-30T12:14:53.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f401affd640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40080392f0 0x7f400803b7b0 secure :-1 s=READY pgs=4180 cs=0 l=1 rev1=1 crypto rx=0x7f4010036000 tx=0x7f401003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f400c005230 con 0x7f401c1381c0 2023-10-30T12:14:53.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:53.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:53.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1381c0 msgr2=0x7f401c1c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1381c0 0x7f401c1c7630 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f770 tx=0x7f400c00fc30 comp rx=0 tx=0).stop 2023-10-30T12:14:53.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1381c0 msgr2=0x7f401c1c7630 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:53.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.323+0000 7f4003fff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1381c0 0x7f401c1c7630 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f770 tx=0x7f400c00fc30 comp rx=0 tx=0).stop 2023-10-30T12:14:53.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f4022100640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:53.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f4022100640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1c7b70 0x7f401c13cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f4022100640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c073a20 con 0x7f401c1c7b70 2023-10-30T12:14:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1c7b70 0x7f401c13cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 10 monclient: get_auth_request con 0x7f401c1c7b70 auth_method 0 2023-10-30T12:14:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.329+0000 7f401b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f401b7fe640 10 monclient(hunting): handle_auth_done global_id 194112 payload 824 2023-10-30T12:14:53.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f401b7fe640 1 -- 172.21.15.50:0/2784109118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f401c04e160 con 0x7f401c1c7b70 2023-10-30T12:14:53.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 10 client.194106.objecter ms_handle_connect 0x7f401c1c7b70 2023-10-30T12:14:53.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 10 client.194106.objecter resend_mon_ops 2023-10-30T12:14:53.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f401b7fe640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1c7b70 0x7f401c13cd90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f770 tx=0x7f400c0010d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c024070 con 0x7f401c1c7b70 2023-10-30T12:14:53.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:53.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 10 monclient: dump: 2023-10-30T12:14:53.382 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.383 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.383 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.383 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.383 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.383 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.384 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.384 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.384 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.384 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7f400c003970 con 0x7f401c1c7b70 2023-10-30T12:14:53.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:53.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:53.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1c7b70 msgr2=0x7f401c13cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1c7b70 0x7f401c13cd90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f770 tx=0x7f400c0010d0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1c7b70 msgr2=0x7f401c13cd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:53.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4003fff640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c1c7b70 0x7f401c13cd90 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f770 tx=0x7f400c0010d0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.386 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:53.386 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:14:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 150, 2023-10-30T12:14:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:14:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:14:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:53.396 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:14:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:14:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:14:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:14:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:53.419 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:53.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 10 librados: watch_flush enter 2023-10-30T12:14:53.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 10 librados: watch_flush exit 2023-10-30T12:14:53.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 20 client.194106.objecter shutdown clearing up homeless session... 2023-10-30T12:14:53.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 10 client.194106.objecter successfully canceled tick 2023-10-30T12:14:53.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40080392f0 msgr2=0x7f400803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f40080392f0 0x7f400803b7b0 secure :-1 s=READY pgs=4180 cs=0 l=1 rev1=1 crypto rx=0x7f4010036000 tx=0x7f401003d040 comp rx=0 tx=0).stop 2023-10-30T12:14:53.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 10 monclient: shutdown 2023-10-30T12:14:53.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c09b170 msgr2=0x7f401c137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 1 --2- 172.21.15.50:0/2784109118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f401c09b170 0x7f401c137230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f401400cb10 tx=0x7f401400a060 comp rx=0 tx=0).stop 2023-10-30T12:14:53.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f401bfff640 1 -- 172.21.15.50:0/2784109118 reap_dead start 2023-10-30T12:14:53.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f401bfff640 1 -- 172.21.15.50:0/2784109118 reap_dead start 2023-10-30T12:14:53.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 shutdown_connections 2023-10-30T12:14:53.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.330+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 >> 172.21.15.50:0/2784109118 conn(0x7f401c0a3980 msgr2=0x7f401c06dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.331+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 shutdown_connections 2023-10-30T12:14:53.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.331+0000 7f4022100640 1 -- 172.21.15.50:0/2784109118 wait complete. 2023-10-30T12:14:53.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.331+0000 7f4022100640 1 librados: shutdown 2023-10-30T12:14:53.425 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:14:53.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:14:53.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:14:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.530+0000 7f5f32e09640 10 monclient: get_monmap_and_config 2023-10-30T12:14:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.530+0000 7f5f32e09640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.530+0000 7f5f32e09640 10 monclient: monmap: 2023-10-30T12:14:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:53.531978+0000 2023-10-30T12:14:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:53.531978+0000 2023-10-30T12:14:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:53.535 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.531+0000 7f5f32e09640 1 Processor -- start 2023-10-30T12:14:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.531+0000 7f5f32e09640 1 -- start start 2023-10-30T12:14:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.531+0000 7f5f32e09640 10 monclient: init 2023-10-30T12:14:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-10-30T12:14:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f2c09bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5f2c09d3f0 0x7f5f2c137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient(hunting): picked mon.noname-c con 0x7f5f2c09d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09c660 0x7f5f2c139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient(hunting): picked mon.noname-b con 0x7f5f2c09c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:53.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2c098170 con 0x7f5f2c09bb70 2023-10-30T12:14:53.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f2c09cf60 con 0x7f5f2c09d3f0 2023-10-30T12:14:53.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c097b20 con 0x7f5f2c09c660 2023-10-30T12:14:53.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:53.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f32e09640 10 monclient(hunting): authenticate will time out at 1738.715210s 2023-10-30T12:14:53.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09c660 0x7f5f2c139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09c660 0x7f5f2c139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34110/0 (socket says 172.21.15.50:34110) 2023-10-30T12:14:53.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 -- 172.21.15.50:0/1684749334 learned_addr learned my addr 172.21.15.50:0/1684749334 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:53.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): get_auth_request con 0x7f5f2c09c660 auth_method 0 2023-10-30T12:14:53.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 1 --2- 172.21.15.50:0/1684749334 >> v2:172.21.15.118:3300/0 conn(0x7f5f2c09d3f0 0x7f5f2c137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request con 0x7f5f2c09d3f0 auth_method 0 2023-10-30T12:14:53.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient(hunting): handle_auth_done global_id 114944 payload 274 2023-10-30T12:14:53.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 -- 172.21.15.50:0/1684749334 >> v2:172.21.15.118:3300/0 conn(0x7f5f2c09d3f0 msgr2=0x7f5f2c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:53.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 --2- 172.21.15.50:0/1684749334 >> v2:172.21.15.118:3300/0 conn(0x7f5f2c09d3f0 0x7f5f2c137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 -- 172.21.15.50:0/1684749334 >> v1:172.21.15.118:6789/0 conn(0x7f5f2c09bb70 legacy=0x7f5f2c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient: _finish_hunting 0 2023-10-30T12:14:53.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 monclient: found mon.noname-b 2023-10-30T12:14:53.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:53.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:14:53.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.532+0000 7f5f313a6640 1 -- 172.21.15.50:0/1684749334 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c097e00 con 0x7f5f2c09c660 2023-10-30T12:14:53.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f313a6640 1 --2- 172.21.15.50:0/1684749334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09c660 0x7f5f2c139eb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009e50 tx=0x7f5f2402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb2d309b9af362d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 1 -- 172.21.15.50:0/1684749334 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2402c9e0 con 0x7f5f2c09c660 2023-10-30T12:14:53.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:14:53.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: dump: 2023-10-30T12:14:53.553 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.553 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.554 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.554 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.554 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.554 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.555 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:14:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:14:53.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:53.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 5 monclient: authenticate success, global_id 114944 2023-10-30T12:14:53.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 1 -- 172.21.15.50:0/1684749334 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2402cb40 con 0x7f5f2c09c660 2023-10-30T12:14:53.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:53.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:53.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:53.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 4 set_mon_vals no callback set 2023-10-30T12:14:53.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 1 -- 172.21.15.50:0/1684749334 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f240343f0 con 0x7f5f2c09c660 2023-10-30T12:14:53.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:53.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f2b7fe640 10 monclient: dump: 2023-10-30T12:14:53.558 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.558 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.558 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.560 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 10 monclient: shutdown 2023-10-30T12:14:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 1 -- 172.21.15.50:0/1684749334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09c660 msgr2=0x7f5f2c139eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.533+0000 7f5f32e09640 1 --2- 172.21.15.50:0/1684749334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09c660 0x7f5f2c139eb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009e50 tx=0x7f5f2402be40 comp rx=0 tx=0).stop 2023-10-30T12:14:53.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 -- 172.21.15.50:0/1684749334 shutdown_connections 2023-10-30T12:14:53.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 --2- 172.21.15.50:0/1684749334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09c660 0x7f5f2c139eb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 --2- 172.21.15.50:0/1684749334 >> v2:172.21.15.118:3300/0 conn(0x7f5f2c09d3f0 0x7f5f2c137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 -- 172.21.15.50:0/1684749334 >> 172.21.15.50:0/1684749334 conn(0x7f5f2c0a3980 msgr2=0x7f5f2c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 -- 172.21.15.50:0/1684749334 shutdown_connections 2023-10-30T12:14:53.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 -- 172.21.15.50:0/1684749334 wait complete. 2023-10-30T12:14:53.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 10 monclient: monmap: 2023-10-30T12:14:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:53.563 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:53.564 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 librados: starting msgr at 2023-10-30T12:14:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 librados: starting objecter 2023-10-30T12:14:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.534+0000 7f5f32e09640 1 Processor -- start 2023-10-30T12:14:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 -- start start 2023-10-30T12:14:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 librados: setting wanted keys 2023-10-30T12:14:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 librados: calling monclient init 2023-10-30T12:14:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient: init 2023-10-30T12:14:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:14:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09bb70 0x7f5f2c06bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): picked mon.noname-c con 0x7f5f2c09bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c09c660 0x7f5f2c06c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): picked mon.noname-a con 0x7f5f2c09c660 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f2c09d3f0 0x7f5f2c06c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): picked mon.noname-b con 0x7f5f2c09d3f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c094060 con 0x7f5f2c09c660 2023-10-30T12:14:53.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c091bd0 con 0x7f5f2c09d3f0 2023-10-30T12:14:53.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c092390 con 0x7f5f2c09bb70 2023-10-30T12:14:53.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:53.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f32e09640 10 monclient(hunting): authenticate will time out at 1738.718692s 2023-10-30T12:14:53.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09bb70 0x7f5f2c06bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09bb70 0x7f5f2c06bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34112/0 (socket says 172.21.15.50:34112) 2023-10-30T12:14:53.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 1 -- 172.21.15.50:0/3916423920 learned_addr learned my addr 172.21.15.50:0/3916423920 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:53.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 10 monclient(hunting): get_auth_request con 0x7f5f2c09bb70 auth_method 0 2023-10-30T12:14:53.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c09c660 0x7f5f2c06c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request con 0x7f5f2c09c660 auth_method 0 2023-10-30T12:14:53.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f313a6640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f2c09d3f0 0x7f5f2c06c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f30ba5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f313a6640 10 monclient(hunting): get_auth_request con 0x7f5f2c09d3f0 auth_method 0 2023-10-30T12:14:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f313a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f313a6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f313a6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.535+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f313a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f313a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f313a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 10 monclient(hunting): handle_auth_done global_id 114947 payload 824 2023-10-30T12:14:53.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f2c09d3f0 msgr2=0x7f5f2c06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f2c09d3f0 0x7f5f2c06c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c09c660 msgr2=0x7f5f2c06c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c09c660 0x7f5f2c06c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 10 monclient: _finish_hunting 0 2023-10-30T12:14:53.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 1 monclient: found mon.noname-c 2023-10-30T12:14:53.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:53.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:53.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 1 -- 172.21.15.50:0/3916423920 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24009b30 con 0x7f5f2c09bb70 2023-10-30T12:14:53.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f2bfff640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f2c09c660 0x7f5f2c06c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:53.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f30ba5640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09bb70 0x7f5f2c06bba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00c480 tx=0x7f5f1c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 client.?.objecter ms_handle_connect 0x7f5f2c09bb70 2023-10-30T12:14:53.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:53.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0052d0 con 0x7f5f2c09bb70 2023-10-30T12:14:53.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:53.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: dump: 2023-10-30T12:14:53.577 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.577 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.577 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.578 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.578 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.578 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.578 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.578 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.581 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.581 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: _finish_auth 0 2023-10-30T12:14:53.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: _check_auth_tickets 2023-10-30T12:14:53.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:53.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c005450 con 0x7f5f2c09bb70 2023-10-30T12:14:53.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 5 monclient: authenticate success, global_id 114947 2023-10-30T12:14:53.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 10 monclient: _renew_subs 2023-10-30T12:14:53.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:53.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c0976e0 con 0x7f5f2c09bb70 2023-10-30T12:14:53.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:53.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c003880 con 0x7f5f2c09bb70 2023-10-30T12:14:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f297fa640 10 monclient: dump: 2023-10-30T12:14:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.586 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.586 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.586 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 10 client.114947.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:53.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f2affd640 4 set_mon_vals no callback set 2023-10-30T12:14:53.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 10 monclient: _renew_subs 2023-10-30T12:14:53.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:53.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c06d760 con 0x7f5f2c09bb70 2023-10-30T12:14:53.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.536+0000 7f5f32e09640 1 librados: init done 2023-10-30T12:14:53.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5f1c0039e0 con 0x7f5f2c09bb70 2023-10-30T12:14:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f32e09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f32e09640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c095510 0x7f5f2c1ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f32e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c073ff0 con 0x7f5f2c095510 2023-10-30T12:14:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c095510 0x7f5f2c1ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient: get_auth_request con 0x7f5f2c095510 auth_method 0 2023-10-30T12:14:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f180392f0 0x7f5f1803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(433..433 src has 1..433) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f5f1c04a8f0 con 0x7f5f2c09bb70 2023-10-30T12:14:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 10 client.114947.objecter ms_dispatch 0x7f5f2c000b90 osd_map(433..433 src has 1..433) v4 2023-10-30T12:14:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 3 client.114947.objecter handle_osd_map got epochs [433,433] > 0 2023-10-30T12:14:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 3 client.114947.objecter handle_osd_map decoding full epoch 433 2023-10-30T12:14:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_done global_id 114950 payload 824 2023-10-30T12:14:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 1 -- 172.21.15.50:0/3916423920 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f2c092220 con 0x7f5f2c095510 2023-10-30T12:14:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f313a6640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f180392f0 0x7f5f1803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 20 client.114947.objecter dump_active .. 0 homeless 2023-10-30T12:14:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f313a6640 10 monclient: get_auth_request con 0x7f5f180392f0 auth_method 0 2023-10-30T12:14:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 10 client.114947.objecter ms_handle_connect 0x7f5f2c095510 2023-10-30T12:14:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 10 client.114947.objecter resend_mon_ops 2023-10-30T12:14:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f2bfff640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c095510 0x7f5f2c1ca7b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c09eea0 tx=0x7f5f2000f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.537+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20004030 con 0x7f5f2c095510 2023-10-30T12:14:53.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:53.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 10 monclient: dump: 2023-10-30T12:14:53.593 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.593 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.593 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.594 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.594 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.594 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.594 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.594 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.595 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.595 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5f20004190 con 0x7f5f2c095510 2023-10-30T12:14:53.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:53.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:53.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c095510 msgr2=0x7f5f2c1ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c095510 0x7f5f2c1ca7b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c09eea0 tx=0x7f5f2000f610 comp rx=0 tx=0).stop 2023-10-30T12:14:53.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c095510 msgr2=0x7f5f2c1ca7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:53.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c095510 0x7f5f2c1ca7b0 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c09eea0 tx=0x7f5f2000f610 comp rx=0 tx=0).stop 2023-10-30T12:14:53.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f297fa640 10 client.114947.objecter ms_handle_connect 0x7f5f180392f0 2023-10-30T12:14:53.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.538+0000 7f5f313a6640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f180392f0 0x7f5f1803b7b0 secure :-1 s=READY pgs=4181 cs=0 l=1 rev1=1 crypto rx=0x7f5f24035000 tx=0x7f5f24030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f32e09640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:53.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f32e09640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c1cacf0 0x7f5f2c1cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f32e09640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c097560 con 0x7f5f2c1cacf0 2023-10-30T12:14:53.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c1cacf0 0x7f5f2c1cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient: get_auth_request con 0x7f5f2c1cacf0 auth_method 0 2023-10-30T12:14:53.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 10 monclient(hunting): handle_auth_done global_id 114953 payload 824 2023-10-30T12:14:53.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.544+0000 7f5f2bfff640 1 -- 172.21.15.50:0/3916423920 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5f2c09f600 con 0x7f5f2c1cacf0 2023-10-30T12:14:53.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f2bfff640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c1cacf0 0x7f5f2c1cb100 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c09eea0 tx=0x7f5f200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 10 client.114947.objecter ms_handle_connect 0x7f5f2c1cacf0 2023-10-30T12:14:53.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 10 client.114947.objecter resend_mon_ops 2023-10-30T12:14:53.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2002c070 con 0x7f5f2c1cacf0 2023-10-30T12:14:53.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:53.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 10 monclient: dump: 2023-10-30T12:14:53.602 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.602 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.602 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.602 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.603 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.603 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.603 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.603 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.603 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.604 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f5f20024040 con 0x7f5f2c1cacf0 2023-10-30T12:14:53.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:53.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:53.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c1cacf0 msgr2=0x7f5f2c1cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c1cacf0 0x7f5f2c1cb100 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c09eea0 tx=0x7f5f200046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c1cacf0 msgr2=0x7f5f2c1cb100 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:53.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f297fa640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c1cacf0 0x7f5f2c1cb100 secure :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c09eea0 tx=0x7f5f200046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.605 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:53.606 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:53.606 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:53.606 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:14:53.606 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 150, 2023-10-30T12:14:53.607 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:53.607 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:14:53.607 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:53.607 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:53.607 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.608 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:14:53.608 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:53.608 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:53.608 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:53.608 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.609 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.609 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.609 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.609 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.609 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.609 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.610 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.610 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.610 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.610 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.610 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:53.611 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:53.611 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.611 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.611 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.611 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.612 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.612 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.612 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.612 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.612 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.612 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.613 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.613 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:53.613 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:53.613 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:53.613 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:53.614 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:53.614 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:53.614 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:53.614 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:53.614 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:53.615 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:53.615 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:53.615 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:53.615 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:53.615 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:53.616 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:53.616 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:53.616 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:53.616 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.616 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.617 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.617 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.617 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.617 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.618 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.618 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.618 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.618 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.618 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:53.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.619 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:53.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.619 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:53.619 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:53.620 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.620 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.620 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:53.620 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:53.621 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.622 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:53.622 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:53.622 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.623 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.623 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.623 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:53.623 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:53.624 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.624 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.624 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.624 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:53.625 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.625 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:53.625 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.626 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.626 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.626 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:53.626 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:53.627 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.627 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.627 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.628 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:53.628 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:53.628 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.628 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.628 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:53.630 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.630 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:53.631 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:53.631 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.631 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.631 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.632 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:53.632 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:53.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.632 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.633 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.633 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:53.633 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.633 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:53.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.634 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.634 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.634 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:14:53.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.635 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.635 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:53.635 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 10 librados: watch_flush enter 2023-10-30T12:14:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 10 librados: watch_flush exit 2023-10-30T12:14:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 20 client.114947.objecter shutdown clearing up homeless session... 2023-10-30T12:14:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 10 client.114947.objecter successfully canceled tick 2023-10-30T12:14:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f180392f0 msgr2=0x7f5f1803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5f180392f0 0x7f5f1803b7b0 secure :-1 s=READY pgs=4181 cs=0 l=1 rev1=1 crypto rx=0x7f5f24035000 tx=0x7f5f24030000 comp rx=0 tx=0).stop 2023-10-30T12:14:53.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 10 monclient: shutdown 2023-10-30T12:14:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09bb70 msgr2=0x7f5f2c06bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09bb70 0x7f5f2c06bba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00c480 tx=0x7f5f1c00c940 comp rx=0 tx=0).stop 2023-10-30T12:14:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f313a6640 1 -- 172.21.15.50:0/3916423920 reap_dead start 2023-10-30T12:14:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 shutdown_connections 2023-10-30T12:14:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 --2- 172.21.15.50:0/3916423920 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f2c09bb70 0x7f5f2c06bba0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 >> 172.21.15.50:0/3916423920 conn(0x7f5f2c0a3980 msgr2=0x7f5f2c096590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 shutdown_connections 2023-10-30T12:14:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.545+0000 7f5f32e09640 1 -- 172.21.15.50:0/3916423920 wait complete. 2023-10-30T12:14:53.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.546+0000 7f5f32e09640 1 librados: shutdown 2023-10-30T12:14:53.640 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:14:53.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:14:53.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:14:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.745+0000 7fcfb261f640 10 monclient: get_monmap_and_config 2023-10-30T12:14:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.745+0000 7fcfb261f640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.745+0000 7fcfb261f640 10 monclient: monmap: 2023-10-30T12:14:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:53.746791+0000 2023-10-30T12:14:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:53.746791+0000 2023-10-30T12:14:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:53.750 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 1 Processor -- start 2023-10-30T12:14:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 1 -- start start 2023-10-30T12:14:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient: init 2023-10-30T12:14:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:14:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfac071c00 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcfac097fe0 0x7fcfac09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient(hunting): picked mon.noname-c con 0x7fcfac097fe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcfac09b170 0x7fcfac139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient(hunting): picked mon.noname-a con 0x7fcfac09b170 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac097600 con 0x7fcfac071c00 2023-10-30T12:14:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac0978e0 con 0x7fcfac09b170 2023-10-30T12:14:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac09bf50 con 0x7fcfac097fe0 2023-10-30T12:14:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.746+0000 7fcfb261f640 10 monclient(hunting): authenticate will time out at 1738.929919s 2023-10-30T12:14:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcfac09b170 0x7fcfac139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcfac09b170 0x7fcfac139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55360/0 (socket says 172.21.15.50:55360) 2023-10-30T12:14:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 -- 172.21.15.50:0/2389687253 learned_addr learned my addr 172.21.15.50:0/2389687253 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): get_auth_request con 0x7fcfac09b170 auth_method 0 2023-10-30T12:14:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfab7fe640 1 --2- 172.21.15.50:0/2389687253 >> v2:172.21.15.118:3300/0 conn(0x7fcfac097fe0 0x7fcfac09a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfab7fe640 10 monclient(hunting): get_auth_request con 0x7fcfac097fe0 auth_method 0 2023-10-30T12:14:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient(hunting): handle_auth_done global_id 194118 payload 274 2023-10-30T12:14:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 -- 172.21.15.50:0/2389687253 >> v2:172.21.15.118:3300/0 conn(0x7fcfac097fe0 msgr2=0x7fcfac09a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 --2- 172.21.15.50:0/2389687253 >> v2:172.21.15.118:3300/0 conn(0x7fcfac097fe0 0x7fcfac09a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 -- 172.21.15.50:0/2389687253 >> v1:172.21.15.118:6789/0 conn(0x7fcfac071c00 legacy=0x7fcfac09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient: _finish_hunting 0 2023-10-30T12:14:53.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 monclient: found mon.noname-a 2023-10-30T12:14:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.747+0000 7fcfb0bbc640 1 -- 172.21.15.50:0/2389687253 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac09d9b0 con 0x7fcfac09b170 2023-10-30T12:14:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb0bbc640 1 --2- 172.21.15.50:0/2389687253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac09b170 0x7fcfac139760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcfac06bb30 tx=0x7fcfa402e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59445987c793db7a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 1 -- 172.21.15.50:0/2389687253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa402ea00 con 0x7fcfac09b170 2023-10-30T12:14:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: dump: 2023-10-30T12:14:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.770 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.772 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: _finish_auth 0 2023-10-30T12:14:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:14:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 5 monclient: authenticate success, global_id 194118 2023-10-30T12:14:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 1 -- 172.21.15.50:0/2389687253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa402eb60 con 0x7fcfac09b170 2023-10-30T12:14:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 1 -- 172.21.15.50:0/2389687253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4009600 con 0x7fcfac09b170 2023-10-30T12:14:53.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:53.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfaaffd640 10 monclient: dump: 2023-10-30T12:14:53.774 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.774 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.775 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.775 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.775 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.775 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.775 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.776 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 4 set_mon_vals no callback set 2023-10-30T12:14:53.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 10 monclient: shutdown 2023-10-30T12:14:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 -- 172.21.15.50:0/2389687253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac09b170 msgr2=0x7fcfac139760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 --2- 172.21.15.50:0/2389687253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac09b170 0x7fcfac139760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcfac06bb30 tx=0x7fcfa402e060 comp rx=0 tx=0).stop 2023-10-30T12:14:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 -- 172.21.15.50:0/2389687253 shutdown_connections 2023-10-30T12:14:53.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 --2- 172.21.15.50:0/2389687253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac09b170 0x7fcfac139760 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 --2- 172.21.15.50:0/2389687253 >> v2:172.21.15.118:3300/0 conn(0x7fcfac097fe0 0x7fcfac09a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 -- 172.21.15.50:0/2389687253 >> 172.21.15.50:0/2389687253 conn(0x7fcfac0a3980 msgr2=0x7fcfac0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 -- 172.21.15.50:0/2389687253 shutdown_connections 2023-10-30T12:14:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.748+0000 7fcfb261f640 1 -- 172.21.15.50:0/2389687253 wait complete. 2023-10-30T12:14:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.749+0000 7fcfb261f640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.749+0000 7fcfb261f640 10 monclient: monmap: 2023-10-30T12:14:53.779 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.779 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:53.779 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:53.780 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:53.781 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.749+0000 7fcfb261f640 1 librados: starting msgr at 2023-10-30T12:14:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.749+0000 7fcfb261f640 1 librados: starting objecter 2023-10-30T12:14:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.749+0000 7fcfb261f640 1 Processor -- start 2023-10-30T12:14:53.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 -- start start 2023-10-30T12:14:53.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 librados: setting wanted keys 2023-10-30T12:14:53.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 librados: calling monclient init 2023-10-30T12:14:53.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient: init 2023-10-30T12:14:53.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:14:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac071c00 0x7fcfac070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): picked mon.noname-b con 0x7fcfac071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac097fe0 0x7fcfac1c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): picked mon.noname-a con 0x7fcfac097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:53.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfac09b170 0x7fcfac094540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): picked mon.noname-c con 0x7fcfac09b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:53.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac1c33b0 con 0x7fcfac097fe0 2023-10-30T12:14:53.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac09cbc0 con 0x7fcfac071c00 2023-10-30T12:14:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac08d720 con 0x7fcfac09b170 2023-10-30T12:14:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb261f640 10 monclient(hunting): authenticate will time out at 1738.933569s 2023-10-30T12:14:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac097fe0 0x7fcfac1c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac097fe0 0x7fcfac1c2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55364/0 (socket says 172.21.15.50:55364) 2023-10-30T12:14:53.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 -- 172.21.15.50:0/3181739519 learned_addr learned my addr 172.21.15.50:0/3181739519 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:53.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfac09b170 0x7fcfac094540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): get_auth_request con 0x7fcfac097fe0 auth_method 0 2023-10-30T12:14:53.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 10 monclient(hunting): get_auth_request con 0x7fcfac09b170 auth_method 0 2023-10-30T12:14:53.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfb0bbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfabfff640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac071c00 0x7fcfac070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfabfff640 10 monclient(hunting): get_auth_request con 0x7fcfac071c00 auth_method 0 2023-10-30T12:14:53.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient(hunting): handle_auth_done global_id 194121 payload 824 2023-10-30T12:14:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfac09b170 msgr2=0x7fcfac094540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfac09b170 0x7fcfac094540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac071c00 msgr2=0x7fcfac070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac071c00 0x7fcfac070c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:14:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 monclient: found mon.noname-a 2023-10-30T12:14:53.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.750+0000 7fcfab7fe640 1 -- 172.21.15.50:0/3181739519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa400bd40 con 0x7fcfac097fe0 2023-10-30T12:14:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfabfff640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac071c00 0x7fcfac070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:14:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfab7fe640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac097fe0 0x7fcfac1c2e70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00d800 tx=0x7fcf9c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 client.?.objecter ms_handle_connect 0x7fcfac097fe0 2023-10-30T12:14:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00b6c0 con 0x7fcfac097fe0 2023-10-30T12:14:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: dump: 2023-10-30T12:14:53.793 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.794 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.795 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: _finish_auth 0 2023-10-30T12:14:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:14:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 5 monclient: authenticate success, global_id 194121 2023-10-30T12:14:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c00b840 con 0x7fcfac097fe0 2023-10-30T12:14:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:53.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00bbf0 con 0x7fcfac097fe0 2023-10-30T12:14:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfaa7fc640 4 set_mon_vals no callback set 2023-10-30T12:14:53.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfa8ff9640 10 monclient: dump: 2023-10-30T12:14:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.799 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.799 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 10 monclient: _renew_subs 2023-10-30T12:14:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac08da00 con 0x7fcfac097fe0 2023-10-30T12:14:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 10 client.194121.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:53.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 10 monclient: _renew_subs 2023-10-30T12:14:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac094a80 con 0x7fcfac097fe0 2023-10-30T12:14:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.751+0000 7fcfb261f640 1 librados: init done 2023-10-30T12:14:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcf9c00bd50 con 0x7fcfac097fe0 2023-10-30T12:14:53.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfb261f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfb261f640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c5eb0 0x7fcfac1c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfb261f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac073ff0 con 0x7fcfac1c5eb0 2023-10-30T12:14:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfa8ff9640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcf98039320 0x7fcf9803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(433..433 src has 1..433) v4 ==== 6307+0+0 (secure 0 0 0) 0x7fcf9c04b140 con 0x7fcfac097fe0 2023-10-30T12:14:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfa8ff9640 10 client.194121.objecter ms_dispatch 0x7fcfac000b90 osd_map(433..433 src has 1..433) v4 2023-10-30T12:14:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfa8ff9640 3 client.194121.objecter handle_osd_map got epochs [433,433] > 0 2023-10-30T12:14:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfa8ff9640 3 client.194121.objecter handle_osd_map decoding full epoch 433 2023-10-30T12:14:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c5eb0 0x7fcfac1c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient: get_auth_request con 0x7fcfac1c5eb0 auth_method 0 2023-10-30T12:14:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfa8ff9640 20 client.194121.objecter dump_active .. 0 homeless 2023-10-30T12:14:53.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfb0bbc640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcf98039320 0x7fcf9803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfb0bbc640 10 monclient: get_auth_request con 0x7fcf98039320 auth_method 0 2023-10-30T12:14:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 10 monclient(hunting): handle_auth_done global_id 184228 payload 824 2023-10-30T12:14:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.752+0000 7fcfabfff640 1 -- 172.21.15.50:0/3181739519 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac071fe0 con 0x7fcfac1c5eb0 2023-10-30T12:14:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 client.194121.objecter ms_handle_connect 0x7fcfac1c5eb0 2023-10-30T12:14:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 client.194121.objecter resend_mon_ops 2023-10-30T12:14:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfabfff640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c5eb0 0x7fcfac1c6290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000e700 tx=0x7fcfa000ebc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 client.194121.objecter ms_handle_connect 0x7fcf98039320 2023-10-30T12:14:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfb0bbc640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcf98039320 0x7fcf9803b7e0 secure :-1 s=READY pgs=4182 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4037000 tx=0x7fcfa4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00050d0 con 0x7fcfac1c5eb0 2023-10-30T12:14:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 monclient: dump: 2023-10-30T12:14:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.809 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fcfa0005230 con 0x7fcfac1c5eb0 2023-10-30T12:14:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c5eb0 msgr2=0x7fcfac1c6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c5eb0 0x7fcfac1c6290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000e700 tx=0x7fcfa000ebc0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c5eb0 msgr2=0x7fcfac1c6290 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:53.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.753+0000 7fcfa8ff9640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c5eb0 0x7fcfac1c6290 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.759+0000 7fcfb261f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.759+0000 7fcfb261f640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c67d0 0x7fcfac06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.759+0000 7fcfb261f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.759+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac08ca70 con 0x7fcfac1c67d0 2023-10-30T12:14:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c67d0 0x7fcfac06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient: get_auth_request con 0x7fcfac1c67d0 auth_method 0 2023-10-30T12:14:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 10 monclient(hunting): handle_auth_done global_id 184231 payload 824 2023-10-30T12:14:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 1 -- 172.21.15.50:0/3181739519 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcfac08dbf0 con 0x7fcfac1c67d0 2023-10-30T12:14:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfabfff640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c67d0 0x7fcfac06dbb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000e700 tx=0x7fcfa0001170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfa8ff9640 10 client.194121.objecter ms_handle_connect 0x7fcfac1c67d0 2023-10-30T12:14:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfa8ff9640 10 client.194121.objecter resend_mon_ops 2023-10-30T12:14:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0022070 con 0x7fcfac1c67d0 2023-10-30T12:14:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfa8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:53.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.760+0000 7fcfa8ff9640 10 monclient: dump: 2023-10-30T12:14:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.817 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.818 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.818 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.818 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.818 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.818 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7fcfa0003910 con 0x7fcfac1c67d0 2023-10-30T12:14:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfa8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfa8ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c67d0 msgr2=0x7fcfac06dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfa8ff9640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c67d0 0x7fcfac06dbb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000e700 tx=0x7fcfa0001170 comp rx=0 tx=0).stop 2023-10-30T12:14:53.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfa8ff9640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c67d0 msgr2=0x7fcfac06dbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:53.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfa8ff9640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcfac1c67d0 0x7fcfac06dbb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 150, 2023-10-30T12:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:53.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 10 librados: watch_flush enter 2023-10-30T12:14:53.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 10 librados: watch_flush exit 2023-10-30T12:14:53.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 20 client.194121.objecter shutdown clearing up homeless session... 2023-10-30T12:14:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 10 client.194121.objecter successfully canceled tick 2023-10-30T12:14:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcf98039320 msgr2=0x7fcf9803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcf98039320 0x7fcf9803b7e0 secure :-1 s=READY pgs=4182 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4037000 tx=0x7fcfa4032000 comp rx=0 tx=0).stop 2023-10-30T12:14:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 10 monclient: shutdown 2023-10-30T12:14:53.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac097fe0 msgr2=0x7fcfac1c2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 --2- 172.21.15.50:0/3181739519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfac097fe0 0x7fcfac1c2e70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00d800 tx=0x7fcf9c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb0bbc640 1 -- 172.21.15.50:0/3181739519 reap_dead start 2023-10-30T12:14:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb0bbc640 1 -- 172.21.15.50:0/3181739519 reap_dead start 2023-10-30T12:14:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 shutdown_connections 2023-10-30T12:14:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 >> 172.21.15.50:0/3181739519 conn(0x7fcfac0a3980 msgr2=0x7fcfac096650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 shutdown_connections 2023-10-30T12:14:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 -- 172.21.15.50:0/3181739519 wait complete. 2023-10-30T12:14:53.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.761+0000 7fcfb261f640 1 librados: shutdown 2023-10-30T12:14:53.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:14:53.858 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:14:53.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:14:53.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.966+0000 7fcde3aec640 10 monclient: get_monmap_and_config 2023-10-30T12:14:53.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.966+0000 7fcde3aec640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.966+0000 7fcde3aec640 10 monclient: monmap: 2023-10-30T12:14:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.968 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:53.967153+0000 2023-10-30T12:14:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:53.967153+0000 2023-10-30T12:14:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:53.970 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:53.970 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.966+0000 7fcde3aec640 1 Processor -- start 2023-10-30T12:14:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.966+0000 7fcde3aec640 1 -- start start 2023-10-30T12:14:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.966+0000 7fcde3aec640 10 monclient: init 2023-10-30T12:14:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-10-30T12:14:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcddc0a4b60 0x7fcddc1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient(hunting): picked mon.noname-a con 0x7fcddc0a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcddc099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcddc132be0 0x7fcddc0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient(hunting): picked mon.noname-c con 0x7fcddc132be0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcddc0a4f40 con 0x7fcddc099a40 2023-10-30T12:14:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0a3d70 con 0x7fcddc0a4b60 2023-10-30T12:14:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0a4150 con 0x7fcddc132be0 2023-10-30T12:14:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde3aec640 10 monclient(hunting): authenticate will time out at 1739.150318s 2023-10-30T12:14:53.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcddc0a4b60 0x7fcddc1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1087640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcddc099a40 0x7fcddc0a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40454/0 (socket says 172.21.15.50:40454) 2023-10-30T12:14:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1087640 1 -- 172.21.15.50:0/4201353859 learned_addr learned my addr 172.21.15.50:0/4201353859 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:14:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): get_auth_request con 0x7fcddc0a4b60 auth_method 0 2023-10-30T12:14:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:53.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient(hunting): handle_auth_done global_id 194124 payload 274 2023-10-30T12:14:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde2089640 1 --2- 172.21.15.50:0/4201353859 >> v2:172.21.15.118:3300/0 conn(0x7fcddc132be0 0x7fcddc0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:53.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 1 -- 172.21.15.50:0/4201353859 >> v2:172.21.15.118:3300/0 conn(0x7fcddc132be0 msgr2=0x7fcddc0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 1 --2- 172.21.15.50:0/4201353859 >> v2:172.21.15.118:3300/0 conn(0x7fcddc132be0 0x7fcddc0a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 1 -- 172.21.15.50:0/4201353859 >> v1:172.21.15.50:6789/0 conn(0x7fcddc099a40 legacy=0x7fcddc0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient: _finish_hunting 0 2023-10-30T12:14:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 1 monclient: found mon.noname-a 2023-10-30T12:14:53.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:53.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:53.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde0886640 1 -- 172.21.15.50:0/4201353859 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405645830 0 0) 0x7fcddc0a4f40 con 0x7fcddc099a40 2023-10-30T12:14:53.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde1888640 1 -- 172.21.15.50:0/4201353859 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc09a440 con 0x7fcddc0a4b60 2023-10-30T12:14:53.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde2089640 10 monclient: get_auth_request con 0x7fcddc132be0 auth_method 0 2023-10-30T12:14:53.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde2089640 1 --2- 172.21.15.50:0/4201353859 >> v2:172.21.15.118:3300/0 conn(0x7fcddc132be0 0x7fcddc0a2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:14:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.967+0000 7fcde0886640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:14:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde1888640 1 --2- 172.21.15.50:0/4201353859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc0a4b60 0x7fcddc1326a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0a3a20 tx=0x7fcdd402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df8044213dfa31e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 1 -- 172.21.15.50:0/4201353859 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4031710 con 0x7fcddc0a4b60 2023-10-30T12:14:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: dump: 2023-10-30T12:14:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.986 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.987 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.988 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: _finish_auth 0 2023-10-30T12:14:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: _check_auth_tickets 2023-10-30T12:14:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:53.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 1 -- 172.21.15.50:0/4201353859 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd402f050 con 0x7fcddc0a4b60 2023-10-30T12:14:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 1 -- 172.21.15.50:0/4201353859 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4031a90 con 0x7fcddc0a4b60 2023-10-30T12:14:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde3aec640 5 monclient: authenticate success, global_id 194124 2023-10-30T12:14:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde3aec640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde3aec640 4 set_mon_vals no callback set 2023-10-30T12:14:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.968+0000 7fcde0886640 10 monclient: dump: 2023-10-30T12:14:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:53.990 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:53.991 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:53.993 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 10 monclient: shutdown 2023-10-30T12:14:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 -- 172.21.15.50:0/4201353859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc0a4b60 msgr2=0x7fcddc1326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 --2- 172.21.15.50:0/4201353859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc0a4b60 0x7fcddc1326a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0a3a20 tx=0x7fcdd402b7b0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 -- 172.21.15.50:0/4201353859 shutdown_connections 2023-10-30T12:14:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 --2- 172.21.15.50:0/4201353859 >> v2:172.21.15.118:3300/0 conn(0x7fcddc132be0 0x7fcddc0a2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 --2- 172.21.15.50:0/4201353859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc0a4b60 0x7fcddc1326a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 -- 172.21.15.50:0/4201353859 >> 172.21.15.50:0/4201353859 conn(0x7fcddc0a8060 msgr2=0x7fcddc0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 -- 172.21.15.50:0/4201353859 shutdown_connections 2023-10-30T12:14:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 1 -- 172.21.15.50:0/4201353859 wait complete. 2023-10-30T12:14:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 10 monclient: build_initial_monmap 2023-10-30T12:14:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.969+0000 7fcde3aec640 10 monclient: monmap: 2023-10-30T12:14:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:53.996 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:53.998 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 librados: starting msgr at 2023-10-30T12:14:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 librados: starting objecter 2023-10-30T12:14:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 Processor -- start 2023-10-30T12:14:53.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 -- start start 2023-10-30T12:14:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 librados: setting wanted keys 2023-10-30T12:14:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 librados: calling monclient init 2023-10-30T12:14:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient: init 2023-10-30T12:14:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:53.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:14:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcddc0a4b60 0x7fcddc1ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): picked mon.noname-c con 0x7fcddc0a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcddc132be0 0x7fcddc1cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): picked mon.noname-b con 0x7fcddc132be0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc1215f0 0x7fcddc121aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): picked mon.noname-a con 0x7fcddc1215f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc123d00 con 0x7fcddc1215f0 2023-10-30T12:14:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0a18d0 con 0x7fcddc132be0 2023-10-30T12:14:54.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0a0880 con 0x7fcddc0a4b60 2023-10-30T12:14:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde3aec640 10 monclient(hunting): authenticate will time out at 1739.153819s 2023-10-30T12:14:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde2089640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc1215f0 0x7fcddc121aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:54.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde1888640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcddc0a4b60 0x7fcddc1ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde2089640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc1215f0 0x7fcddc121aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55376/0 (socket says 172.21.15.50:55376) 2023-10-30T12:14:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde1888640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcddc0a4b60 0x7fcddc1ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34138/0 (socket says 172.21.15.50:34138) 2023-10-30T12:14:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde2089640 1 -- 172.21.15.50:0/1263253891 learned_addr learned my addr 172.21.15.50:0/1263253891 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde2089640 10 monclient(hunting): get_auth_request con 0x7fcddc1215f0 auth_method 0 2023-10-30T12:14:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde2089640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:54.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde2089640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde2089640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde1888640 10 monclient(hunting): get_auth_request con 0x7fcddc0a4b60 auth_method 0 2023-10-30T12:14:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde1888640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde1888640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:54.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.970+0000 7fcde1888640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1087640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcddc132be0 0x7fcddc1cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1888640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:54.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1888640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1888640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1087640 10 monclient(hunting): get_auth_request con 0x7fcddc132be0 auth_method 0 2023-10-30T12:14:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1087640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1087640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:54.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1087640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 10 monclient(hunting): handle_auth_done global_id 194130 payload 824 2023-10-30T12:14:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 1 -- 172.21.15.50:0/1263253891 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcddc0a4b60 msgr2=0x7fcddc1ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcddc0a4b60 0x7fcddc1ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 1 -- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcddc132be0 msgr2=0x7fcddc1cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcddc132be0 0x7fcddc1cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 10 monclient: _finish_hunting 0 2023-10-30T12:14:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 1 monclient: found mon.noname-a 2023-10-30T12:14:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 1 -- 172.21.15.50:0/1263253891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd4009400 con 0x7fcddc1215f0 2023-10-30T12:14:54.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde1087640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcddc132be0 0x7fcddc1cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:14:54.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde2089640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc1215f0 0x7fcddc121aa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00cef0 tx=0x7fcdcc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:54.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 client.?.objecter ms_handle_connect 0x7fcddc1215f0 2023-10-30T12:14:54.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:54.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 1 -- 172.21.15.50:0/1263253891 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00d840 con 0x7fcddc1215f0 2023-10-30T12:14:54.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:54.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:54.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: dump: 2023-10-30T12:14:54.010 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:54.010 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:54.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:54.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:54.012 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: _finish_auth 0 2023-10-30T12:14:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:14:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 1 -- 172.21.15.50:0/1263253891 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc00d9c0 con 0x7fcddc1215f0 2023-10-30T12:14:54.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 1 -- 172.21.15.50:0/1263253891 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00dd70 con 0x7fcddc1215f0 2023-10-30T12:14:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 5 monclient: authenticate success, global_id 194130 2023-10-30T12:14:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 10 monclient: _renew_subs 2023-10-30T12:14:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:54.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc125660 con 0x7fcddc1215f0 2023-10-30T12:14:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcddbfff640 4 set_mon_vals no callback set 2023-10-30T12:14:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcdda7fc640 10 monclient: dump: 2023-10-30T12:14:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:54.014 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:54.015 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:54.016 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 10 client.194130.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 10 monclient: _renew_subs 2023-10-30T12:14:54.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc123290 con 0x7fcddc1215f0 2023-10-30T12:14:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.971+0000 7fcde3aec640 1 librados: init done 2023-10-30T12:14:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcdda7fc640 1 -- 172.21.15.50:0/1263253891 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcdcc01e3b0 con 0x7fcddc1215f0 2023-10-30T12:14:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcde3aec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:54.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcde3aec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcde3aec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc04e160 con 0x7fcddc1215f0 2023-10-30T12:14:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcdda7fc640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdc80392f0 0x7fcdc803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcdda7fc640 1 -- 172.21.15.50:0/1263253891 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(434..434 src has 1..434) v4 ==== 6307+0+0 (secure 0 0 0) 0x7fcdcc04b7a0 con 0x7fcddc1215f0 2023-10-30T12:14:54.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcdda7fc640 10 client.194130.objecter ms_dispatch 0x7fcddc000b90 osd_map(434..434 src has 1..434) v4 2023-10-30T12:14:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcdda7fc640 3 client.194130.objecter handle_osd_map got epochs [434,434] > 0 2023-10-30T12:14:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcdda7fc640 3 client.194130.objecter handle_osd_map decoding full epoch 434 2023-10-30T12:14:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.972+0000 7fcdda7fc640 20 client.194130.objecter dump_active .. 0 homeless 2023-10-30T12:14:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.973+0000 7fcde1888640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdc80392f0 0x7fcdc803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.973+0000 7fcde1888640 10 monclient: get_auth_request con 0x7fcdc80392f0 auth_method 0 2023-10-30T12:14:54.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.973+0000 7fcde1888640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdc80392f0 0x7fcdc803b7b0 secure :-1 s=READY pgs=4183 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4035000 tx=0x7fcdd4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.973+0000 7fcdda7fc640 10 client.194130.objecter ms_handle_connect 0x7fcdc80392f0 2023-10-30T12:14:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.977+0000 7fcdda7fc640 1 -- 172.21.15.50:0/1263253891 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcdcc01d360 con 0x7fcddc1215f0 2023-10-30T12:14:54.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.977+0000 7fcdda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:54.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:53.977+0000 7fcdda7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:54.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.221+0000 7fcde3aec640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:14:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.221+0000 7fcde3aec640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:14:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.221+0000 7fcde3aec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.221+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fcddc0745c0 con 0x7fcddc1215f0 2023-10-30T12:14:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.222+0000 7fcdda7fc640 1 -- 172.21.15.50:0/1263253891 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fcdcc01c2c0 con 0x7fcddc1215f0 2023-10-30T12:14:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.222+0000 7fcdda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:14:54.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.222+0000 7fcdda7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:54.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.222+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.222+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=896424701,config=3140514009,health=2019044389,kv=3826327778,logm=1458989148}) 2023-10-30T12:14:54.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.222+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=9,logm=51} crc {auth=1654303754,config=3140514009,health=3536970861,kv=2905431493,logm=3896595970}) 2023-10-30T12:14:54.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.222+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.222+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=896424701,config=3140514009,health=2019044389,kv=3826327778,logm=1458989148}) 2023-10-30T12:14:54.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.222+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=9,logm=50} crc {auth=1944817351,config=3140514009,health=264023753,kv=2905431493,logm=3323842786}) 2023-10-30T12:14:54.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.224+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.224+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068280235}) 2023-10-30T12:14:54.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.224+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4164264837}) 2023-10-30T12:14:54.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.224+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.224+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068280235}) 2023-10-30T12:14:54.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 10 librados: watch_flush enter 2023-10-30T12:14:54.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.224+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949719608}) 2023-10-30T12:14:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 10 librados: watch_flush exit 2023-10-30T12:14:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 20 client.194130.objecter shutdown clearing up homeless session... 2023-10-30T12:14:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 10 client.194130.objecter successfully canceled tick 2023-10-30T12:14:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdc80392f0 msgr2=0x7fcdc803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:54.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdc80392f0 0x7fcdc803b7b0 secure :-1 s=READY pgs=4183 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4035000 tx=0x7fcdd4008030 comp rx=0 tx=0).stop 2023-10-30T12:14:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 10 monclient: shutdown 2023-10-30T12:14:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc1215f0 msgr2=0x7fcddc121aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc1215f0 0x7fcddc121aa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00cef0 tx=0x7fcdcc00a710 comp rx=0 tx=0).stop 2023-10-30T12:14:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 shutdown_connections 2023-10-30T12:14:54.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdc80392f0 0x7fcdc803b7b0 unknown :-1 s=CLOSED pgs=4183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:54.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcddc1215f0 0x7fcddc121aa0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:54.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcddc132be0 0x7fcddc1cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:54.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 --2- 172.21.15.50:0/1263253891 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcddc0a4b60 0x7fcddc1ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:54.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 >> 172.21.15.50:0/1263253891 conn(0x7fcddc0a8060 msgr2=0x7fcddc0a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:54.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 shutdown_connections 2023-10-30T12:14:54.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 -- 172.21.15.50:0/1263253891 wait complete. 2023-10-30T12:14:54.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:54.224+0000 7fcde3aec640 1 librados: shutdown 2023-10-30T12:14:54.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.225+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.225+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229882018}) 2023-10-30T12:14:54.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.225+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=304667302}) 2023-10-30T12:14:54.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.225+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.225+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229882018}) 2023-10-30T12:14:54.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.225+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1091806631}) 2023-10-30T12:14:54.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.226+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.226+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1633301871}) 2023-10-30T12:14:54.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.226+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2280005810}) 2023-10-30T12:14:54.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.226+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.226+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1633301871}) 2023-10-30T12:14:54.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.226+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576161854}) 2023-10-30T12:14:54.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276571115}) 2023-10-30T12:14:54.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3384053594}) 2023-10-30T12:14:54.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276571115}) 2023-10-30T12:14:54.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259643522}) 2023-10-30T12:14:54.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2548490628}) 2023-10-30T12:14:54.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1862732026}) 2023-10-30T12:14:54.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2548490628}) 2023-10-30T12:14:54.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.228+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362625372}) 2023-10-30T12:14:54.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.229+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.229+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509062418}) 2023-10-30T12:14:54.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.229+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3340842256}) 2023-10-30T12:14:54.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.229+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.229+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509062418}) 2023-10-30T12:14:54.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.229+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1189060297}) 2023-10-30T12:14:54.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.230+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.230+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319858117}) 2023-10-30T12:14:54.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.230+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763802177}) 2023-10-30T12:14:54.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.230+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.230+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319858117}) 2023-10-30T12:14:54.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.230+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1253343490}) 2023-10-30T12:14:54.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291458918}) 2023-10-30T12:14:54.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327859464}) 2023-10-30T12:14:54.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291458918}) 2023-10-30T12:14:54.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4062838312}) 2023-10-30T12:14:54.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180219071}) 2023-10-30T12:14:54.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2997990860}) 2023-10-30T12:14:54.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180219071}) 2023-10-30T12:14:54.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192378782}) 2023-10-30T12:14:54.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886225177}) 2023-10-30T12:14:54.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3433815861}) 2023-10-30T12:14:54.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886225177}) 2023-10-30T12:14:54.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187451430}) 2023-10-30T12:14:54.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762969949}) 2023-10-30T12:14:54.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.233+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4184619701}) 2023-10-30T12:14:54.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762969949}) 2023-10-30T12:14:54.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353302912}) 2023-10-30T12:14:54.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2585527460}) 2023-10-30T12:14:54.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866006226}) 2023-10-30T12:14:54.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2585527460}) 2023-10-30T12:14:54.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785486278}) 2023-10-30T12:14:54.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176890354}) 2023-10-30T12:14:54.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943485164}) 2023-10-30T12:14:54.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176890354}) 2023-10-30T12:14:54.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=654899634}) 2023-10-30T12:14:54.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:14:54.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.236+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.236+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216598890}) 2023-10-30T12:14:54.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.236+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2328951682}) 2023-10-30T12:14:54.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.236+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.236+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216598890}) 2023-10-30T12:14:54.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.236+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383622579}) 2023-10-30T12:14:54.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708434094}) 2023-10-30T12:14:54.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4274259597}) 2023-10-30T12:14:54.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708434094}) 2023-10-30T12:14:54.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3333999490}) 2023-10-30T12:14:54.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869748197}) 2023-10-30T12:14:54.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3454001211}) 2023-10-30T12:14:54.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869748197}) 2023-10-30T12:14:54.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3368017144}) 2023-10-30T12:14:54.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505403698}) 2023-10-30T12:14:54.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3088339727}) 2023-10-30T12:14:54.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505403698}) 2023-10-30T12:14:54.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369861720}) 2023-10-30T12:14:54.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554309328}) 2023-10-30T12:14:54.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1726499946}) 2023-10-30T12:14:54.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554309328}) 2023-10-30T12:14:54.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3231884655}) 2023-10-30T12:14:54.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815828021}) 2023-10-30T12:14:54.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780653777}) 2023-10-30T12:14:54.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815828021}) 2023-10-30T12:14:54.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394297573}) 2023-10-30T12:14:54.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247445601}) 2023-10-30T12:14:54.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4062358292}) 2023-10-30T12:14:54.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247445601}) 2023-10-30T12:14:54.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=764652715}) 2023-10-30T12:14:54.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558283019}) 2023-10-30T12:14:54.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4172899199}) 2023-10-30T12:14:54.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558283019}) 2023-10-30T12:14:54.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=414623746}) 2023-10-30T12:14:54.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825093841}) 2023-10-30T12:14:54.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769480995}) 2023-10-30T12:14:54.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825093841}) 2023-10-30T12:14:54.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845140351}) 2023-10-30T12:14:54.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216853151}) 2023-10-30T12:14:54.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869178954}) 2023-10-30T12:14:54.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216853151}) 2023-10-30T12:14:54.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660409314}) 2023-10-30T12:14:54.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363499440}) 2023-10-30T12:14:54.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2943525609}) 2023-10-30T12:14:54.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363499440}) 2023-10-30T12:14:54.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2009578855}) 2023-10-30T12:14:54.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235015448}) 2023-10-30T12:14:54.274 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461513544}) 2023-10-30T12:14:54.274 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.274 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235015448}) 2023-10-30T12:14:54.274 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332118106}) 2023-10-30T12:14:54.274 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.275 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442836485}) 2023-10-30T12:14:54.275 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=525540155}) 2023-10-30T12:14:54.275 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.275 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442836485}) 2023-10-30T12:14:54.275 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=902036137}) 2023-10-30T12:14:54.275 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263156421}) 2023-10-30T12:14:54.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=509042268}) 2023-10-30T12:14:54.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263156421}) 2023-10-30T12:14:54.276 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010037558}) 2023-10-30T12:14:54.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434817432}) 2023-10-30T12:14:54.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3925335479}) 2023-10-30T12:14:54.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434817432}) 2023-10-30T12:14:54.277 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258435658}) 2023-10-30T12:14:54.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057620623}) 2023-10-30T12:14:54.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4033880651}) 2023-10-30T12:14:54.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057620623}) 2023-10-30T12:14:54.278 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3836517086}) 2023-10-30T12:14:54.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857742663}) 2023-10-30T12:14:54.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1308746589}) 2023-10-30T12:14:54.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.279 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857742663}) 2023-10-30T12:14:54.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1896592942}) 2023-10-30T12:14:54.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086988009}) 2023-10-30T12:14:54.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2656234026}) 2023-10-30T12:14:54.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.280 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086988009}) 2023-10-30T12:14:54.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2102769022}) 2023-10-30T12:14:54.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773417014}) 2023-10-30T12:14:54.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1900109614}) 2023-10-30T12:14:54.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.281 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773417014}) 2023-10-30T12:14:54.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666280963}) 2023-10-30T12:14:54.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941183725}) 2023-10-30T12:14:54.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4217890285}) 2023-10-30T12:14:54.282 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941183725}) 2023-10-30T12:14:54.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645271258}) 2023-10-30T12:14:54.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3615874608}) 2023-10-30T12:14:54.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=95813089}) 2023-10-30T12:14:54.283 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3615874608}) 2023-10-30T12:14:54.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477574028}) 2023-10-30T12:14:54.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503120077}) 2023-10-30T12:14:54.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1662453705}) 2023-10-30T12:14:54.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.284 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503120077}) 2023-10-30T12:14:54.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4088968246}) 2023-10-30T12:14:54.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924829438}) 2023-10-30T12:14:54.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3963412202}) 2023-10-30T12:14:54.285 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924829438}) 2023-10-30T12:14:54.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2419266108}) 2023-10-30T12:14:54.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902328849}) 2023-10-30T12:14:54.286 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=888894106}) 2023-10-30T12:14:54.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902328849}) 2023-10-30T12:14:54.287 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963410566}) 2023-10-30T12:14:54.305 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4196154135}) 2023-10-30T12:14:54.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931603456}) 2023-10-30T12:14:54.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4196154135}) 2023-10-30T12:14:54.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1210289799}) 2023-10-30T12:14:54.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791044328}) 2023-10-30T12:14:54.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759435646}) 2023-10-30T12:14:54.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791044328}) 2023-10-30T12:14:54.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082421648}) 2023-10-30T12:14:54.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281238399}) 2023-10-30T12:14:54.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=543778810}) 2023-10-30T12:14:54.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281238399}) 2023-10-30T12:14:54.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714023123}) 2023-10-30T12:14:54.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2627022380}) 2023-10-30T12:14:54.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2774351551}) 2023-10-30T12:14:54.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2627022380}) 2023-10-30T12:14:54.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563262746}) 2023-10-30T12:14:54.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281761110}) 2023-10-30T12:14:54.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=938326287}) 2023-10-30T12:14:54.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281761110}) 2023-10-30T12:14:54.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660688691}) 2023-10-30T12:14:54.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917176685}) 2023-10-30T12:14:54.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670898245}) 2023-10-30T12:14:54.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917176685}) 2023-10-30T12:14:54.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714088328}) 2023-10-30T12:14:54.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022696274}) 2023-10-30T12:14:54.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805976238}) 2023-10-30T12:14:54.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022696274}) 2023-10-30T12:14:54.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253971884}) 2023-10-30T12:14:54.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611361787}) 2023-10-30T12:14:54.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011509699}) 2023-10-30T12:14:54.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611361787}) 2023-10-30T12:14:54.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141347438}) 2023-10-30T12:14:54.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994707948}) 2023-10-30T12:14:54.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2783965396}) 2023-10-30T12:14:54.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994707948}) 2023-10-30T12:14:54.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782383450}) 2023-10-30T12:14:54.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169690244}) 2023-10-30T12:14:54.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313807185}) 2023-10-30T12:14:54.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169690244}) 2023-10-30T12:14:54.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882488900}) 2023-10-30T12:14:54.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442352316}) 2023-10-30T12:14:54.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015258517}) 2023-10-30T12:14:54.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442352316}) 2023-10-30T12:14:54.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=401949562}) 2023-10-30T12:14:54.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074524496}) 2023-10-30T12:14:54.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1540144248}) 2023-10-30T12:14:54.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074524496}) 2023-10-30T12:14:54.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485658516}) 2023-10-30T12:14:54.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264749517}) 2023-10-30T12:14:54.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=21550391}) 2023-10-30T12:14:54.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264749517}) 2023-10-30T12:14:54.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2455314353}) 2023-10-30T12:14:54.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705206632}) 2023-10-30T12:14:54.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245280992}) 2023-10-30T12:14:54.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705206632}) 2023-10-30T12:14:54.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469079437}) 2023-10-30T12:14:54.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154965545}) 2023-10-30T12:14:54.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3183020145}) 2023-10-30T12:14:54.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154965545}) 2023-10-30T12:14:54.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003496095}) 2023-10-30T12:14:54.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289498543}) 2023-10-30T12:14:54.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110597243}) 2023-10-30T12:14:54.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289498543}) 2023-10-30T12:14:54.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278348753}) 2023-10-30T12:14:54.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42483585}) 2023-10-30T12:14:54.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030751812}) 2023-10-30T12:14:54.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42483585}) 2023-10-30T12:14:54.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912243119}) 2023-10-30T12:14:54.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101075919}) 2023-10-30T12:14:54.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1523702965}) 2023-10-30T12:14:54.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101075919}) 2023-10-30T12:14:54.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51716173}) 2023-10-30T12:14:54.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596087917}) 2023-10-30T12:14:54.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044105737}) 2023-10-30T12:14:54.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596087917}) 2023-10-30T12:14:54.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832511776}) 2023-10-30T12:14:54.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281228107}) 2023-10-30T12:14:54.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260603063}) 2023-10-30T12:14:54.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281228107}) 2023-10-30T12:14:54.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3180551803}) 2023-10-30T12:14:54.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869631441}) 2023-10-30T12:14:54.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1038847816}) 2023-10-30T12:14:54.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869631441}) 2023-10-30T12:14:54.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3486526571}) 2023-10-30T12:14:54.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246447691}) 2023-10-30T12:14:54.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495923584}) 2023-10-30T12:14:54.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246447691}) 2023-10-30T12:14:54.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538457282}) 2023-10-30T12:14:54.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214628557}) 2023-10-30T12:14:54.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=828594918}) 2023-10-30T12:14:54.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214628557}) 2023-10-30T12:14:54.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897437498}) 2023-10-30T12:14:54.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842313651}) 2023-10-30T12:14:54.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1414268966}) 2023-10-30T12:14:54.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842313651}) 2023-10-30T12:14:54.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3991063196}) 2023-10-30T12:14:54.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727099324}) 2023-10-30T12:14:54.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3599925132}) 2023-10-30T12:14:54.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727099324}) 2023-10-30T12:14:54.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812188059}) 2023-10-30T12:14:54.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037077561}) 2023-10-30T12:14:54.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748491706}) 2023-10-30T12:14:54.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037077561}) 2023-10-30T12:14:54.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387999900}) 2023-10-30T12:14:54.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199354052}) 2023-10-30T12:14:54.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505029314}) 2023-10-30T12:14:54.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199354052}) 2023-10-30T12:14:54.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1847731457}) 2023-10-30T12:14:54.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513675608}) 2023-10-30T12:14:54.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265157663}) 2023-10-30T12:14:54.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513675608}) 2023-10-30T12:14:54.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3243228344}) 2023-10-30T12:14:54.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030350419}) 2023-10-30T12:14:54.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902176879}) 2023-10-30T12:14:54.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030350419}) 2023-10-30T12:14:54.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560301816}) 2023-10-30T12:14:54.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964407258}) 2023-10-30T12:14:54.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29501780}) 2023-10-30T12:14:54.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964407258}) 2023-10-30T12:14:54.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3755789951}) 2023-10-30T12:14:54.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479177219}) 2023-10-30T12:14:54.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3306674804}) 2023-10-30T12:14:54.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479177219}) 2023-10-30T12:14:54.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1628185013}) 2023-10-30T12:14:54.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771180740}) 2023-10-30T12:14:54.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=279086594}) 2023-10-30T12:14:54.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771180740}) 2023-10-30T12:14:54.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2083673822}) 2023-10-30T12:14:54.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887583454}) 2023-10-30T12:14:54.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974585059}) 2023-10-30T12:14:54.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887583454}) 2023-10-30T12:14:54.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455970078}) 2023-10-30T12:14:54.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132120434}) 2023-10-30T12:14:54.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1464609257}) 2023-10-30T12:14:54.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132120434}) 2023-10-30T12:14:54.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=675302638}) 2023-10-30T12:14:54.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630184328}) 2023-10-30T12:14:54.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1323519589}) 2023-10-30T12:14:54.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630184328}) 2023-10-30T12:14:54.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1463707456}) 2023-10-30T12:14:54.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002844533}) 2023-10-30T12:14:54.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3949966457}) 2023-10-30T12:14:54.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002844533}) 2023-10-30T12:14:54.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1076252886}) 2023-10-30T12:14:54.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369130561}) 2023-10-30T12:14:54.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=884853889}) 2023-10-30T12:14:54.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369130561}) 2023-10-30T12:14:54.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343536855}) 2023-10-30T12:14:54.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36} crc {logm=1491867847,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3245056783,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1718653645,osd_snap=542912528}) 2023-10-30T12:14:54.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=22,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=4} crc {logm=3339114861,mdsmap=3984425268,mgr=1715906815,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3245056783,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1718653645,osd_snap=3285409935,osdmap=363744095}) 2023-10-30T12:14:54.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36} crc {logm=1491867847,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3245056783,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1718653645,osd_snap=542912528}) 2023-10-30T12:14:54.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=3} crc {logm=838636219,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3245056783,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1718653645,osd_snap=3470052660,osdmap=4129985701}) 2023-10-30T12:14:54.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2040842389}) 2023-10-30T12:14:54.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2227400493}) 2023-10-30T12:14:54.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2040842389}) 2023-10-30T12:14:54.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2286091128}) 2023-10-30T12:14:54.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3638905164}) 2023-10-30T12:14:54.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3249399510}) 2023-10-30T12:14:54.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3638905164}) 2023-10-30T12:14:54.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1467243310}) 2023-10-30T12:14:54.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=112065446}) 2023-10-30T12:14:54.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=357335367}) 2023-10-30T12:14:54.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=112065446}) 2023-10-30T12:14:54.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1356751068}) 2023-10-30T12:14:54.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3053317525}) 2023-10-30T12:14:54.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1540128265}) 2023-10-30T12:14:54.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3053317525}) 2023-10-30T12:14:54.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1587872839}) 2023-10-30T12:14:54.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2675252022}) 2023-10-30T12:14:54.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2597608529}) 2023-10-30T12:14:54.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2675252022}) 2023-10-30T12:14:54.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1542099551}) 2023-10-30T12:14:54.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3916309600}) 2023-10-30T12:14:54.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4163507167}) 2023-10-30T12:14:54.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3916309600}) 2023-10-30T12:14:54.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2217524953}) 2023-10-30T12:14:54.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.303+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.303+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3258987249}) 2023-10-30T12:14:54.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.303+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=23} crc {osdmap=4025446644}) 2023-10-30T12:14:54.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.303+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:14:54.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.303+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3258987249}) 2023-10-30T12:14:54.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:14:54.303+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=3789739275}) 2023-10-30T12:14:55.253 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:14:55.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:14:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.369+0000 7f0dbca6b640 10 monclient: get_monmap_and_config 2023-10-30T12:14:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.369+0000 7f0dbca6b640 10 monclient: build_initial_monmap 2023-10-30T12:14:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.369+0000 7f0dbca6b640 10 monclient: monmap: 2023-10-30T12:14:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:55.370290+0000 2023-10-30T12:14:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:55.370290+0000 2023-10-30T12:14:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:55.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.369+0000 7f0dbca6b640 1 Processor -- start 2023-10-30T12:14:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.369+0000 7f0dbca6b640 1 -- start start 2023-10-30T12:14:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.369+0000 7f0dbca6b640 10 monclient: init 2023-10-30T12:14:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-10-30T12:14:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0db80a4b60 0x7f0db81326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): picked mon.noname-c con 0x7f0db80a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db8099a40 0x7f0db8132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): picked mon.noname-b con 0x7f0db8099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0db80a2350 0x7f0db80a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): picked mon.noname-a con 0x7f0db80a2350 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:55.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80a3d70 con 0x7f0db80a2350 2023-10-30T12:14:55.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db8133120 con 0x7f0db80a4b60 2023-10-30T12:14:55.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80a4f40 con 0x7f0db8099a40 2023-10-30T12:14:55.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:55.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0dbca6b640 10 monclient(hunting): authenticate will time out at 1740.553544s 2023-10-30T12:14:55.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0db80a2350 0x7f0db80a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db8099a40 0x7f0db8132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db8099a40 0x7f0db8132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34146/0 (socket says 172.21.15.50:34146) 2023-10-30T12:14:55.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0db80a2350 0x7f0db80a27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55380/0 (socket says 172.21.15.50:55380) 2023-10-30T12:14:55.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 1 -- 172.21.15.50:0/1046198682 learned_addr learned my addr 172.21.15.50:0/1046198682 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:55.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 10 monclient(hunting): get_auth_request con 0x7f0db8099a40 auth_method 0 2023-10-30T12:14:55.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 10 monclient(hunting): get_auth_request con 0x7f0db80a2350 auth_method 0 2023-10-30T12:14:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db659c640 1 --2- 172.21.15.50:0/1046198682 >> v2:172.21.15.118:3300/0 conn(0x7f0db80a4b60 0x7f0db81326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db659c640 10 monclient(hunting): get_auth_request con 0x7f0db80a4b60 auth_method 0 2023-10-30T12:14:55.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.370+0000 7f0db659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 10 monclient(hunting): handle_auth_done global_id 114962 payload 274 2023-10-30T12:14:55.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 1 -- 172.21.15.50:0/1046198682 >> v2:172.21.15.118:3300/0 conn(0x7f0db80a4b60 msgr2=0x7f0db81326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:55.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 1 --2- 172.21.15.50:0/1046198682 >> v2:172.21.15.118:3300/0 conn(0x7f0db80a4b60 0x7f0db81326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 1 -- 172.21.15.50:0/1046198682 >> v2:172.21.15.50:3300/0 conn(0x7f0db80a2350 msgr2=0x7f0db80a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:55.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 1 --2- 172.21.15.50:0/1046198682 >> v2:172.21.15.50:3300/0 conn(0x7f0db80a2350 0x7f0db80a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:14:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 1 monclient: found mon.noname-b 2023-10-30T12:14:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:14:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 1 -- 172.21.15.50:0/1046198682 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80a4150 con 0x7f0db8099a40 2023-10-30T12:14:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db6d9d640 1 --2- 172.21.15.50:0/1046198682 >> v2:172.21.15.50:3300/0 conn(0x7f0db80a2350 0x7f0db80a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db5d9b640 1 --2- 172.21.15.50:0/1046198682 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db8099a40 0x7f0db8132be0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0db80a3820 tx=0x7f0dac02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=271f60475d56d4f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.371+0000 7f0db559a640 1 -- 172.21.15.50:0/1046198682 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac031740 con 0x7f0db8099a40 2023-10-30T12:14:55.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:14:55.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: dump: 2023-10-30T12:14:55.390 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.391 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.391 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.391 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.391 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.391 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.392 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.392 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.392 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.392 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: _finish_auth 0 2023-10-30T12:14:55.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: _check_auth_tickets 2023-10-30T12:14:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 5 monclient: authenticate success, global_id 114962 2023-10-30T12:14:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 1 -- 172.21.15.50:0/1046198682 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac0318a0 con 0x7f0db8099a40 2023-10-30T12:14:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 1 -- 172.21.15.50:0/1046198682 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac031c30 con 0x7f0db8099a40 2023-10-30T12:14:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:55.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0db559a640 10 monclient: dump: 2023-10-30T12:14:55.394 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.394 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.394 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.395 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.395 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.395 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.395 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.395 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.395 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.396 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:55.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 4 set_mon_vals no callback set 2023-10-30T12:14:55.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 10 monclient: shutdown 2023-10-30T12:14:55.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:55.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 -- 172.21.15.50:0/1046198682 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db8099a40 msgr2=0x7f0db8132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/1046198682 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db8099a40 0x7f0db8132be0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0db80a3820 tx=0x7f0dac02b7e0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 -- 172.21.15.50:0/1046198682 shutdown_connections 2023-10-30T12:14:55.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/1046198682 >> v2:172.21.15.50:3300/0 conn(0x7f0db80a2350 0x7f0db80a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/1046198682 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db8099a40 0x7f0db8132be0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/1046198682 >> v2:172.21.15.118:3300/0 conn(0x7f0db80a4b60 0x7f0db81326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 -- 172.21.15.50:0/1046198682 >> 172.21.15.50:0/1046198682 conn(0x7f0db80a8060 msgr2=0x7f0db80a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 -- 172.21.15.50:0/1046198682 shutdown_connections 2023-10-30T12:14:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.372+0000 7f0dbca6b640 1 -- 172.21.15.50:0/1046198682 wait complete. 2023-10-30T12:14:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 10 monclient: build_initial_monmap 2023-10-30T12:14:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 10 monclient: monmap: 2023-10-30T12:14:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:55.400 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 1 librados: starting msgr at 2023-10-30T12:14:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 1 librados: starting objecter 2023-10-30T12:14:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 1 Processor -- start 2023-10-30T12:14:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 1 -- start start 2023-10-30T12:14:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 1 librados: setting wanted keys 2023-10-30T12:14:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 1 librados: calling monclient init 2023-10-30T12:14:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.373+0000 7f0dbca6b640 10 monclient: init 2023-10-30T12:14:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:14:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db80a2350 0x7f0db809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): picked mon.noname-c con 0x7f0db80a2350 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80a4b60 0x7f0db809ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): picked mon.noname-b con 0x7f0db80a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db809ef60 0x7f0db81d0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): picked mon.noname-a con 0x7f0db809ef60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db812e0d0 con 0x7f0db809ef60 2023-10-30T12:14:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8135cb0 con 0x7f0db80a4b60 2023-10-30T12:14:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80a18d0 con 0x7f0db80a2350 2023-10-30T12:14:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0dbca6b640 10 monclient(hunting): authenticate will time out at 1740.557237s 2023-10-30T12:14:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db80a2350 0x7f0db809e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db80a2350 0x7f0db809e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34156/0 (socket says 172.21.15.50:34156) 2023-10-30T12:14:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 -- 172.21.15.50:0/2989686392 learned_addr learned my addr 172.21.15.50:0/2989686392 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): get_auth_request con 0x7f0db80a2350 auth_method 0 2023-10-30T12:14:55.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db809ef60 0x7f0db81d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 10 monclient(hunting): get_auth_request con 0x7f0db809ef60 auth_method 0 2023-10-30T12:14:55.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db5d9b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80a4b60 0x7f0db809ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db5d9b640 10 monclient(hunting): get_auth_request con 0x7f0db80a4b60 auth_method 0 2023-10-30T12:14:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient(hunting): handle_auth_done global_id 114965 payload 824 2023-10-30T12:14:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 -- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80a4b60 msgr2=0x7f0db809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80a4b60 0x7f0db809ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 -- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db809ef60 msgr2=0x7f0db81d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db809ef60 0x7f0db81d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient: _finish_hunting 0 2023-10-30T12:14:55.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 monclient: found mon.noname-c 2023-10-30T12:14:55.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:55.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:55.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 -- 172.21.15.50:0/2989686392 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0009a70 con 0x7f0db80a2350 2023-10-30T12:14:55.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db6d9d640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db809ef60 0x7f0db81d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:55.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0db659c640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db80a2350 0x7f0db809e4e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0da800d800 tx=0x7f0da800dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0da6ffd640 10 client.?.objecter ms_handle_connect 0x7f0db80a2350 2023-10-30T12:14:55.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.374+0000 7f0da6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:55.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 1 -- 172.21.15.50:0/2989686392 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da800b6c0 con 0x7f0db80a2350 2023-10-30T12:14:55.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:55.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: dump: 2023-10-30T12:14:55.411 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.411 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.411 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.412 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.412 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.412 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.412 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.412 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.412 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.412 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:14:55.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:14:55.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:55.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 5 monclient: authenticate success, global_id 114965 2023-10-30T12:14:55.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 1 -- 172.21.15.50:0/2989686392 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da800b820 con 0x7f0db80a2350 2023-10-30T12:14:55.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 10 monclient: _renew_subs 2023-10-30T12:14:55.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:55.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac0094d0 con 0x7f0db80a2350 2023-10-30T12:14:55.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:55.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 1 -- 172.21.15.50:0/2989686392 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da800bbb0 con 0x7f0db80a2350 2023-10-30T12:14:55.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:55.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 10 monclient: dump: 2023-10-30T12:14:55.415 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.415 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.415 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.415 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.416 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.416 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.416 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.416 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.416 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0db4d99640 4 set_mon_vals no callback set 2023-10-30T12:14:55.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 10 client.114965.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:55.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 10 monclient: _renew_subs 2023-10-30T12:14:55.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:55.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db809ff10 con 0x7f0db80a2350 2023-10-30T12:14:55.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0dbca6b640 1 librados: init done 2023-10-30T12:14:55.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.375+0000 7f0da6ffd640 1 -- 172.21.15.50:0/2989686392 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0da800bd10 con 0x7f0db80a2350 2023-10-30T12:14:55.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0da6ffd640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0d980392f0 0x7f0d9803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0da6ffd640 1 -- 172.21.15.50:0/2989686392 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(435..435 src has 1..435) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f0da804b160 con 0x7f0db80a2350 2023-10-30T12:14:55.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0da6ffd640 10 client.114965.objecter ms_dispatch 0x7f0db8000b90 osd_map(435..435 src has 1..435) v4 2023-10-30T12:14:55.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0da6ffd640 3 client.114965.objecter handle_osd_map got epochs [435,435] > 0 2023-10-30T12:14:55.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0da6ffd640 3 client.114965.objecter handle_osd_map decoding full epoch 435 2023-10-30T12:14:55.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0dbca6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:55.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0dbca6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:55.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0dbca6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:55.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0da6ffd640 20 client.114965.objecter dump_active .. 0 homeless 2023-10-30T12:14:55.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db811f200 con 0x7f0db80a2350 2023-10-30T12:14:55.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0db5d9b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0d980392f0 0x7f0d9803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.376+0000 7f0db5d9b640 10 monclient: get_auth_request con 0x7f0d980392f0 auth_method 0 2023-10-30T12:14:55.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.377+0000 7f0da6ffd640 10 client.114965.objecter ms_handle_connect 0x7f0d980392f0 2023-10-30T12:14:55.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.377+0000 7f0db5d9b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0d980392f0 0x7f0d9803b7b0 secure :-1 s=READY pgs=4184 cs=0 l=1 rev1=1 crypto rx=0x7f0dac035000 tx=0x7f0dac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.381+0000 7f0da6ffd640 1 -- 172.21.15.50:0/2989686392 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0da8014430 con 0x7f0db80a2350 2023-10-30T12:14:55.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.381+0000 7f0da6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:55.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.381+0000 7f0da6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:55.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.632+0000 7f0dbca6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:14:55.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.632+0000 7f0dbca6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:55.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.632+0000 7f0dbca6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:55.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.632+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0db811ebc0 con 0x7f0db80a2350 2023-10-30T12:14:55.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.632+0000 7f0da6ffd640 1 -- 172.21.15.50:0/2989686392 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f0da801d3c0 con 0x7f0db80a2350 2023-10-30T12:14:55.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.632+0000 7f0da6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:55.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.632+0000 7f0da6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:55.635 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:14:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 10 librados: watch_flush enter 2023-10-30T12:14:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 10 librados: watch_flush exit 2023-10-30T12:14:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 20 client.114965.objecter shutdown clearing up homeless session... 2023-10-30T12:14:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 10 client.114965.objecter successfully canceled tick 2023-10-30T12:14:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0d980392f0 msgr2=0x7f0d9803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0d980392f0 0x7f0d9803b7b0 secure :-1 s=READY pgs=4184 cs=0 l=1 rev1=1 crypto rx=0x7f0dac035000 tx=0x7f0dac008030 comp rx=0 tx=0).stop 2023-10-30T12:14:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 10 monclient: shutdown 2023-10-30T12:14:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db80a2350 msgr2=0x7f0db809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db80a2350 0x7f0db809e4e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0da800d800 tx=0x7f0da800dc00 comp rx=0 tx=0).stop 2023-10-30T12:14:55.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 shutdown_connections 2023-10-30T12:14:55.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0d980392f0 0x7f0d9803b7b0 unknown :-1 s=CLOSED pgs=4184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db809ef60 0x7f0db81d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0db80a4b60 0x7f0db809ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 --2- 172.21.15.50:0/2989686392 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0db80a2350 0x7f0db809e4e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 >> 172.21.15.50:0/2989686392 conn(0x7f0db80a8060 msgr2=0x7f0db80a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:55.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 shutdown_connections 2023-10-30T12:14:55.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.634+0000 7f0dbca6b640 1 -- 172.21.15.50:0/2989686392 wait complete. 2023-10-30T12:14:55.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.635+0000 7f0dbca6b640 1 librados: shutdown 2023-10-30T12:14:55.646 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:14:55.646 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:14:55.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:14:55.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:14:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.751+0000 7f9f80905640 10 monclient: get_monmap_and_config 2023-10-30T12:14:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.751+0000 7f9f80905640 10 monclient: build_initial_monmap 2023-10-30T12:14:55.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.751+0000 7f9f80905640 10 monclient: monmap: 2023-10-30T12:14:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:55.752880+0000 2023-10-30T12:14:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:55.752880+0000 2023-10-30T12:14:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:55.755 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:55.756 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.752+0000 7f9f80905640 1 Processor -- start 2023-10-30T12:14:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.752+0000 7f9f80905640 1 -- start start 2023-10-30T12:14:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.752+0000 7f9f80905640 10 monclient: init 2023-10-30T12:14:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.752+0000 7f9f80905640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.752+0000 7f9f80905640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:14:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c09fa30 0x7f9f7c0a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): picked mon.noname-b con 0x7f9f7c09fa30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9f7c06b720 0x7f9f7c0a2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): picked mon.noname-c con 0x7f9f7c06b720 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f7c0a27b0 0x7f9f7c13c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): picked mon.noname-a con 0x7f9f7c0a27b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7c0a2be0 con 0x7f9f7c0a27b0 2023-10-30T12:14:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7c09e010 con 0x7f9f7c06b720 2023-10-30T12:14:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c09fe10 con 0x7f9f7c09fa30 2023-10-30T12:14:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f80905640 10 monclient(hunting): authenticate will time out at 1740.936059s 2023-10-30T12:14:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f7c0a27b0 0x7f9f7c13c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c09fa30 0x7f9f7c0a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c09fa30 0x7f9f7c0a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34170/0 (socket says 172.21.15.50:34170) 2023-10-30T12:14:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f7c0a27b0 0x7f9f7c13c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55402/0 (socket says 172.21.15.50:55402) 2023-10-30T12:14:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 1 -- 172.21.15.50:0/171328581 learned_addr learned my addr 172.21.15.50:0/171328581 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 10 monclient(hunting): get_auth_request con 0x7f9f7c09fa30 auth_method 0 2023-10-30T12:14:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): get_auth_request con 0x7f9f7c0a27b0 auth_method 0 2023-10-30T12:14:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f79d9b640 1 --2- 172.21.15.50:0/171328581 >> v2:172.21.15.118:3300/0 conn(0x7f9f7c06b720 0x7f9f7c0a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f79d9b640 10 monclient(hunting): get_auth_request con 0x7f9f7c06b720 auth_method 0 2023-10-30T12:14:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f79d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f79d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_done global_id 194139 payload 274 2023-10-30T12:14:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c09fa30 msgr2=0x7f9f7c0a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 --2- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c09fa30 0x7f9f7c0a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/171328581 >> v2:172.21.15.118:3300/0 conn(0x7f9f7c06b720 msgr2=0x7f9f7c0a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 --2- 172.21.15.50:0/171328581 >> v2:172.21.15.118:3300/0 conn(0x7f9f7c06b720 0x7f9f7c0a2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:14:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 monclient: found mon.noname-a 2023-10-30T12:14:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:55.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/171328581 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c06bde0 con 0x7f9f7c0a27b0 2023-10-30T12:14:55.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f79d9b640 1 --2- 172.21.15.50:0/171328581 >> v2:172.21.15.118:3300/0 conn(0x7f9f7c06b720 0x7f9f7c0a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:14:55.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.753+0000 7f9f7a59c640 1 --2- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c09fa30 0x7f9f7c0a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:55.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7ad9d640 1 --2- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c0a27b0 0x7f9f7c13c070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c069e60 tx=0x7f9f6c02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2a27747660c630 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 1 -- 172.21.15.50:0/171328581 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c02cab0 con 0x7f9f7c0a27b0 2023-10-30T12:14:55.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:55.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: dump: 2023-10-30T12:14:55.779 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.779 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.779 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.781 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.781 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: _finish_auth 0 2023-10-30T12:14:55.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: _check_auth_tickets 2023-10-30T12:14:55.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:55.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 1 -- 172.21.15.50:0/171328581 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c02cc10 con 0x7f9f7c0a27b0 2023-10-30T12:14:55.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:55.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f80905640 5 monclient: authenticate success, global_id 194139 2023-10-30T12:14:55.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f80905640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:55.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f80905640 4 set_mon_vals no callback set 2023-10-30T12:14:55.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 1 -- 172.21.15.50:0/171328581 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c034440 con 0x7f9f7c0a27b0 2023-10-30T12:14:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f7959a640 10 monclient: dump: 2023-10-30T12:14:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.785 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.785 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f80905640 10 monclient: shutdown 2023-10-30T12:14:55.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f80905640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:55.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f80905640 1 -- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c0a27b0 msgr2=0x7f9f7c13c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.754+0000 7f9f80905640 1 --2- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c0a27b0 0x7f9f7c13c070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c069e60 tx=0x7f9f6c02bd00 comp rx=0 tx=0).stop 2023-10-30T12:14:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 1 -- 172.21.15.50:0/171328581 shutdown_connections 2023-10-30T12:14:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 1 --2- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c0a27b0 0x7f9f7c13c070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 1 --2- 172.21.15.50:0/171328581 >> v2:172.21.15.118:3300/0 conn(0x7f9f7c06b720 0x7f9f7c0a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 1 --2- 172.21.15.50:0/171328581 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c09fa30 0x7f9f7c0a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 1 -- 172.21.15.50:0/171328581 >> 172.21.15.50:0/171328581 conn(0x7f9f7c0a6200 msgr2=0x7f9f7c0a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 1 -- 172.21.15.50:0/171328581 shutdown_connections 2023-10-30T12:14:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 1 -- 172.21.15.50:0/171328581 wait complete. 2023-10-30T12:14:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 10 monclient: build_initial_monmap 2023-10-30T12:14:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.755+0000 7f9f80905640 10 monclient: monmap: 2023-10-30T12:14:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:55.789 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:55.789 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.789 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:55.789 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:55.789 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:55.790 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 librados: starting msgr at 2023-10-30T12:14:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 librados: starting objecter 2023-10-30T12:14:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 Processor -- start 2023-10-30T12:14:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 -- start start 2023-10-30T12:14:55.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 librados: setting wanted keys 2023-10-30T12:14:55.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 librados: calling monclient init 2023-10-30T12:14:55.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient: init 2023-10-30T12:14:55.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:55.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:14:55.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f7c09fa30 0x7f9f7c1c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): picked mon.noname-b con 0x7f9f7c09fa30 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:55.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c0a27b0 0x7f9f7c1c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): picked mon.noname-c con 0x7f9f7c0a27b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:55.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c1c6c90 0x7f9f7c1c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): picked mon.noname-a con 0x7f9f7c1c6c90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:55.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c070ee0 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c1a8890 con 0x7f9f7c09fa30 2023-10-30T12:14:55.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c0705f0 con 0x7f9f7c0a27b0 2023-10-30T12:14:55.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:55.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f80905640 10 monclient(hunting): authenticate will time out at 1740.939675s 2023-10-30T12:14:55.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c0a27b0 0x7f9f7c1c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c1c6c90 0x7f9f7c1c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c0a27b0 0x7f9f7c1c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34178/0 (socket says 172.21.15.50:34178) 2023-10-30T12:14:55.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 1 -- 172.21.15.50:0/245008379 learned_addr learned my addr 172.21.15.50:0/245008379 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:55.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 10 monclient(hunting): get_auth_request con 0x7f9f7c0a27b0 auth_method 0 2023-10-30T12:14:55.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 10 monclient(hunting): get_auth_request con 0x7f9f7c1c6c90 auth_method 0 2023-10-30T12:14:55.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7a59c640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f7c09fa30 0x7f9f7c1c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7a59c640 10 monclient(hunting): get_auth_request con 0x7f9f7c09fa30 auth_method 0 2023-10-30T12:14:55.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.756+0000 7f9f7a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 10 monclient(hunting): handle_auth_done global_id 194142 payload 824 2023-10-30T12:14:55.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c0a27b0 msgr2=0x7f9f7c1c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f7c0a27b0 0x7f9f7c1c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f7c09fa30 msgr2=0x7f9f7c1c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f7c09fa30 0x7f9f7c1c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:14:55.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 1 monclient: found mon.noname-a 2023-10-30T12:14:55.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:55.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:55.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/245008379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f740094d0 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7a59c640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f7c09fa30 0x7f9f7c1c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:14:55.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f7ad9d640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c1c6c90 0x7f9f7c1c9c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c033000 tx=0x7f9f6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 client.?.objecter ms_handle_connect 0x7f9f7c1c6c90 2023-10-30T12:14:55.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:55.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c03e610 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:55.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: dump: 2023-10-30T12:14:55.802 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.803 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.803 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.803 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.803 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.803 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.803 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.804 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.804 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.804 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: _finish_auth 0 2023-10-30T12:14:55.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:14:55.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:55.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c03e770 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 5 monclient: authenticate success, global_id 194142 2023-10-30T12:14:55.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 10 monclient: _renew_subs 2023-10-30T12:14:55.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:55.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c009a70 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:55.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c03eb20 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:55.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f72ffd640 10 monclient: dump: 2023-10-30T12:14:55.807 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.807 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.807 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.807 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.807 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.808 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.808 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.808 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.808 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.808 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f78d99640 4 set_mon_vals no callback set 2023-10-30T12:14:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 10 client.194142.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 10 monclient: _renew_subs 2023-10-30T12:14:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7c1ab150 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.757+0000 7f9f80905640 1 librados: init done 2023-10-30T12:14:55.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f80905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:55.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f80905640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c06b720 0x7f9f7c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c073fd0 con 0x7f9f7c06b720 2023-10-30T12:14:55.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9f6c03b050 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c06b720 0x7f9f7c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient: get_auth_request con 0x7f9f7c06b720 auth_method 0 2023-10-30T12:14:55.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f5c0392f0 0x7f9f5c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(435..435 src has 1..435) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f9f6c06ced0 con 0x7f9f7c1c6c90 2023-10-30T12:14:55.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 10 client.194142.objecter ms_dispatch 0x7f9f7c000b90 osd_map(435..435 src has 1..435) v4 2023-10-30T12:14:55.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 3 client.194142.objecter handle_osd_map got epochs [435,435] > 0 2023-10-30T12:14:55.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 3 client.194142.objecter handle_osd_map decoding full epoch 435 2023-10-30T12:14:55.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_done global_id 194145 payload 824 2023-10-30T12:14:55.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 20 client.194142.objecter dump_active .. 0 homeless 2023-10-30T12:14:55.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 1 -- 172.21.15.50:0/245008379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f7c1aad00 con 0x7f9f7c06b720 2023-10-30T12:14:55.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f79d9b640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f5c0392f0 0x7f9f5c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f79d9b640 10 monclient: get_auth_request con 0x7f9f5c0392f0 auth_method 0 2023-10-30T12:14:55.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 10 client.194142.objecter ms_handle_connect 0x7f9f7c06b720 2023-10-30T12:14:55.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f72ffd640 10 client.194142.objecter resend_mon_ops 2023-10-30T12:14:55.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.758+0000 7f9f7a59c640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c06b720 0x7f9f7c02b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f74011800 tx=0x7f9f74011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7400fc40 con 0x7f9f7c06b720 2023-10-30T12:14:55.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:55.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 10 monclient: dump: 2023-10-30T12:14:55.816 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.816 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.816 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.816 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.816 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.816 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.817 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.817 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.817 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.817 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 10 client.194142.objecter ms_handle_connect 0x7f9f5c0392f0 2023-10-30T12:14:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f79d9b640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f5c0392f0 0x7f9f5c03b7b0 secure :-1 s=READY pgs=4185 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c1cad50 tx=0x7f9f64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9f74011eb0 con 0x7f9f7c06b720 2023-10-30T12:14:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c06b720 msgr2=0x7f9f7c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c06b720 0x7f9f7c02b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f74011800 tx=0x7f9f74011cc0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c06b720 msgr2=0x7f9f7c02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:55.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.759+0000 7f9f72ffd640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c06b720 0x7f9f7c02b430 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f74011800 tx=0x7f9f74011cc0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f80905640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:55.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f80905640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c02b970 0x7f9f7c02bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f80905640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7c1aab90 con 0x7f9f7c02b970 2023-10-30T12:14:55.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c02b970 0x7f9f7c02bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient: get_auth_request con 0x7f9f7c02b970 auth_method 0 2023-10-30T12:14:55.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 10 monclient(hunting): handle_auth_done global_id 194148 payload 824 2023-10-30T12:14:55.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.765+0000 7f9f7a59c640 1 -- 172.21.15.50:0/245008379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9f7c06ab30 con 0x7f9f7c02b970 2023-10-30T12:14:55.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 10 client.194142.objecter ms_handle_connect 0x7f9f7c02b970 2023-10-30T12:14:55.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 10 client.194142.objecter resend_mon_ops 2023-10-30T12:14:55.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f7a59c640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c02b970 0x7f9f7c02bd80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f74011800 tx=0x7f9f74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74015660 con 0x7f9f7c02b970 2023-10-30T12:14:55.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:55.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 10 monclient: dump: 2023-10-30T12:14:55.825 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.826 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.826 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.826 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.826 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.826 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.827 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.827 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.827 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.827 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7f9f74006850 con 0x7f9f7c02b970 2023-10-30T12:14:55.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:55.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:55.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c02b970 msgr2=0x7f9f7c02bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c02b970 0x7f9f7c02bd80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f74011800 tx=0x7f9f74004730 comp rx=0 tx=0).stop 2023-10-30T12:14:55.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c02b970 msgr2=0x7f9f7c02bd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:55.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f72ffd640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c02b970 0x7f9f7c02bd80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.829 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:55.829 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:55.829 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:55.829 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:14:55.830 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 150, 2023-10-30T12:14:55.830 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:55.830 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:14:55.830 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:55.830 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:55.831 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:55.831 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:14:55.831 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:55.831 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:55.831 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:55.832 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:55.832 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:55.832 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:55.832 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:55.832 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:55.832 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:55.833 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:55.833 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:55.833 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:55.833 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:55.834 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:55.834 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:55.834 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:55.834 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:55.834 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:55.835 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:55.835 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:55.835 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:55.835 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:55.835 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:55.835 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:55.836 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:55.836 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:55.836 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:55.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.836 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:55.837 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:55.837 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:55.837 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:55.837 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:55.837 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:55.838 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:55.838 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:55.838 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:55.838 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:55.838 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:55.839 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:55.839 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:55.839 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:55.839 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:55.839 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:55.840 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:55.840 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:55.840 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:55.840 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:55.840 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:55.841 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:55.841 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:55.841 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:55.841 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:55.841 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:55.841 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:55.842 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:55.842 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:55.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.842 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:55.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.843 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:55.843 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:55.843 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:55.843 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:55.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.844 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:55.844 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:55.844 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:55.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:55.845 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:55.845 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:55.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.845 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:55.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.846 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:55.846 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:55.846 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:55.846 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:55.846 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:55.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.847 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:55.847 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:55.847 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:55.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:55.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:55.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:55.848 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:55.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:55.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:55.849 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:55.849 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.850 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:55.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.850 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:55.850 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:55.850 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:55.851 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:55.851 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:55.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.851 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:55.851 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:55.852 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:55.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:55.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.852 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:55.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:55.853 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:55.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:55.853 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:55.854 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:55.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.854 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:55.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.854 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:55.855 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:55.855 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:55.855 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:55.855 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:55.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:55.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.856 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:55.856 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:55.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.856 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:55.857 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:55.857 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:55.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.857 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:55.857 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:14:55.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.858 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:55.858 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:55.858 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:14:55.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.858 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:55.859 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:55.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.859 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:55.859 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:55.859 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:14:55.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.860 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:55.860 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:14:55.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:55.860 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:55.860 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:55.861 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:55.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:55.861 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:55.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:55.861 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:55.862 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:55.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 10 librados: watch_flush enter 2023-10-30T12:14:55.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 10 librados: watch_flush exit 2023-10-30T12:14:55.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 20 client.194142.objecter shutdown clearing up homeless session... 2023-10-30T12:14:55.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 10 client.194142.objecter successfully canceled tick 2023-10-30T12:14:55.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f5c0392f0 msgr2=0x7f9f5c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f5c0392f0 0x7f9f5c03b7b0 secure :-1 s=READY pgs=4185 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c1cad50 tx=0x7f9f64008040 comp rx=0 tx=0).stop 2023-10-30T12:14:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 10 monclient: shutdown 2023-10-30T12:14:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c1c6c90 msgr2=0x7f9f7c1c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 --2- 172.21.15.50:0/245008379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f7c1c6c90 0x7f9f7c1c9c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c033000 tx=0x7f9f6c004a40 comp rx=0 tx=0).stop 2023-10-30T12:14:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/245008379 reap_dead start 2023-10-30T12:14:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f7ad9d640 1 -- 172.21.15.50:0/245008379 reap_dead start 2023-10-30T12:14:55.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 shutdown_connections 2023-10-30T12:14:55.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 >> 172.21.15.50:0/245008379 conn(0x7f9f7c0a6200 msgr2=0x7f9f7c0a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:55.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 shutdown_connections 2023-10-30T12:14:55.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 -- 172.21.15.50:0/245008379 wait complete. 2023-10-30T12:14:55.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.766+0000 7f9f80905640 1 librados: shutdown 2023-10-30T12:14:55.867 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:14:55.867 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:14:55.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:14:55.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.974+0000 7f839cf05640 10 monclient: get_monmap_and_config 2023-10-30T12:14:55.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.974+0000 7f839cf05640 10 monclient: build_initial_monmap 2023-10-30T12:14:55.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.974+0000 7f839cf05640 10 monclient: monmap: 2023-10-30T12:14:55.976 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:55.976 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.976 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:55.975291+0000 2023-10-30T12:14:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:55.975291+0000 2023-10-30T12:14:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:55.978 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:55.978 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:55.978 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.974+0000 7f839cf05640 1 Processor -- start 2023-10-30T12:14:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.974+0000 7f839cf05640 1 -- start start 2023-10-30T12:14:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.974+0000 7f839cf05640 10 monclient: init 2023-10-30T12:14:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-10-30T12:14:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83980a2ae0 0x7f83980a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient(hunting): picked mon.noname-a con 0x7f83980a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f83980a2ec0 0x7f83981356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:55.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient(hunting): picked mon.noname-c con 0x7f83980a2ec0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8398135c20 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83980a4190 con 0x7f8398135c20 2023-10-30T12:14:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83980a2990 con 0x7f83980a2ae0 2023-10-30T12:14:55.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:55.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83980a1630 con 0x7f83980a2ec0 2023-10-30T12:14:55.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:55.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839cf05640 10 monclient(hunting): authenticate will time out at 1741.158478s 2023-10-30T12:14:55.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83980a2ae0 0x7f83980a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83980a2ae0 0x7f83980a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55432/0 (socket says 172.21.15.50:55432) 2023-10-30T12:14:55.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 -- 172.21.15.50:0/62663206 learned_addr learned my addr 172.21.15.50:0/62663206 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:55.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): get_auth_request con 0x7f83980a2ae0 auth_method 0 2023-10-30T12:14:55.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:55.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:55.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:55.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f838ffff640 1 --2- 172.21.15.50:0/62663206 >> v2:172.21.15.118:3300/0 conn(0x7f83980a2ec0 0x7f83981356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:55.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f838ffff640 10 monclient(hunting): get_auth_request con 0x7f83980a2ec0 auth_method 0 2023-10-30T12:14:55.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f838ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:55.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f838ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:55.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f838ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:55.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient(hunting): handle_auth_done global_id 194151 payload 274 2023-10-30T12:14:55.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 -- 172.21.15.50:0/62663206 >> v2:172.21.15.118:3300/0 conn(0x7f83980a2ec0 msgr2=0x7f83981356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:55.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 --2- 172.21.15.50:0/62663206 >> v2:172.21.15.118:3300/0 conn(0x7f83980a2ec0 0x7f83981356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:55.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 -- 172.21.15.50:0/62663206 >> v1:172.21.15.118:6789/0 conn(0x7f8398135c20 legacy=0x7f839813ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:55.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient: _finish_hunting 0 2023-10-30T12:14:55.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 monclient: found mon.noname-a 2023-10-30T12:14:55.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:55.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:55.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.975+0000 7f839659c640 1 -- 172.21.15.50:0/62663206 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839809a2e0 con 0x7f83980a2ae0 2023-10-30T12:14:55.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f839659c640 1 --2- 172.21.15.50:0/62663206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83980a2ae0 0x7f83980a2450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f83980a4c80 tx=0x7f838802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d6703a576d783 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:55.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 1 -- 172.21.15.50:0/62663206 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83880317e0 con 0x7f83980a2ae0 2023-10-30T12:14:55.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:55.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: dump: 2023-10-30T12:14:55.994 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.995 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.995 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.995 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.995 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.995 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.995 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:55.996 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:55.996 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:55.996 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:55.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: _finish_auth 0 2023-10-30T12:14:55.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:14:55.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:55.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f839cf05640 5 monclient: authenticate success, global_id 194151 2023-10-30T12:14:55.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f839cf05640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:55.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 1 -- 172.21.15.50:0/62663206 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8388031940 con 0x7f83980a2ae0 2023-10-30T12:14:55.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:55.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 1 -- 172.21.15.50:0/62663206 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388031cd0 con 0x7f83980a2ae0 2023-10-30T12:14:55.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:55.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:55.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f8395d9b640 10 monclient: dump: 2023-10-30T12:14:55.998 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:55.998 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:55.999 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.999 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:55.999 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:55.999 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:55.999 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.000 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.000 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.000 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f839cf05640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:56.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.976+0000 7f839cf05640 4 set_mon_vals no callback set 2023-10-30T12:14:56.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 10 monclient: shutdown 2023-10-30T12:14:56.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:56.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 -- 172.21.15.50:0/62663206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83980a2ae0 msgr2=0x7f83980a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 --2- 172.21.15.50:0/62663206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83980a2ae0 0x7f83980a2450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f83980a4c80 tx=0x7f838802b880 comp rx=0 tx=0).stop 2023-10-30T12:14:56.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 -- 172.21.15.50:0/62663206 shutdown_connections 2023-10-30T12:14:56.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 --2- 172.21.15.50:0/62663206 >> v2:172.21.15.118:3300/0 conn(0x7f83980a2ec0 0x7f83981356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 --2- 172.21.15.50:0/62663206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83980a2ae0 0x7f83980a2450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 -- 172.21.15.50:0/62663206 >> 172.21.15.50:0/62663206 conn(0x7f83980a80b0 msgr2=0x7f83980a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:56.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 -- 172.21.15.50:0/62663206 shutdown_connections 2023-10-30T12:14:56.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.977+0000 7f839cf05640 1 -- 172.21.15.50:0/62663206 wait complete. 2023-10-30T12:14:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 10 monclient: build_initial_monmap 2023-10-30T12:14:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 10 monclient: monmap: 2023-10-30T12:14:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:56.004 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:56.004 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.004 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:56.004 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:56.004 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:56.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 1 librados: starting msgr at 2023-10-30T12:14:56.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 1 librados: starting objecter 2023-10-30T12:14:56.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 1 Processor -- start 2023-10-30T12:14:56.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 1 -- start start 2023-10-30T12:14:56.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 1 librados: setting wanted keys 2023-10-30T12:14:56.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 1 librados: calling monclient init 2023-10-30T12:14:56.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.978+0000 7f839cf05640 10 monclient: init 2023-10-30T12:14:56.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:56.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:14:56.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83980a2ae0 0x7f83981ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): picked mon.noname-b con 0x7f83980a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:56.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83980a2ec0 0x7f83981322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): picked mon.noname-c con 0x7f83980a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:56.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8398135c20 0x7f8398132820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): picked mon.noname-a con 0x7f8398135c20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:56.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8398121c80 con 0x7f8398135c20 2023-10-30T12:14:56.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83980a1790 con 0x7f83980a2ae0 2023-10-30T12:14:56.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8398135530 con 0x7f83980a2ec0 2023-10-30T12:14:56.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:56.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient(hunting): authenticate will time out at 1741.162146s 2023-10-30T12:14:56.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8398135c20 0x7f8398132820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83980a2ec0 0x7f83981322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8398135c20 0x7f8398132820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55444/0 (socket says 172.21.15.50:55444) 2023-10-30T12:14:56.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83980a2ec0 0x7f83981322e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34188/0 (socket says 172.21.15.50:34188) 2023-10-30T12:14:56.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 -- 172.21.15.50:0/818039299 learned_addr learned my addr 172.21.15.50:0/818039299 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:56.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): get_auth_request con 0x7f8398135c20 auth_method 0 2023-10-30T12:14:56.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 10 monclient(hunting): get_auth_request con 0x7f83980a2ec0 auth_method 0 2023-10-30T12:14:56.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83980a2ae0 0x7f83981ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 10 monclient(hunting): get_auth_request con 0x7f83980a2ae0 auth_method 0 2023-10-30T12:14:56.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient(hunting): handle_auth_done global_id 194154 payload 824 2023-10-30T12:14:56.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83980a2ec0 msgr2=0x7f83981322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83980a2ec0 0x7f83981322e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83980a2ae0 msgr2=0x7f83981ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f83980a2ae0 0x7f83981ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:14:56.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 monclient: found mon.noname-a 2023-10-30T12:14:56.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:56.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 -- 172.21.15.50:0/818039299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880094d0 con 0x7f8398135c20 2023-10-30T12:14:56.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f8396d9d640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8398135c20 0x7f8398132820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f839000c7a0 tx=0x7f839000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 client.?.objecter ms_handle_connect 0x7f8398135c20 2023-10-30T12:14:56.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:56.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390003b70 con 0x7f8398135c20 2023-10-30T12:14:56.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:56.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: dump: 2023-10-30T12:14:56.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.017 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.017 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.018 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.018 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.018 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: _finish_auth 0 2023-10-30T12:14:56.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: _check_auth_tickets 2023-10-30T12:14:56.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:56.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390003cf0 con 0x7f8398135c20 2023-10-30T12:14:56.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 5 monclient: authenticate success, global_id 194154 2023-10-30T12:14:56.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient: _renew_subs 2023-10-30T12:14:56.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8398123ab0 con 0x7f8398135c20 2023-10-30T12:14:56.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:56.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390003ea0 con 0x7f8398135c20 2023-10-30T12:14:56.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:56.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f838effd640 10 monclient: dump: 2023-10-30T12:14:56.021 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.021 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.022 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.022 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.022 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.022 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.022 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.023 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.023 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.023 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 client.194154.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:56.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839559a640 4 set_mon_vals no callback set 2023-10-30T12:14:56.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient: _renew_subs 2023-10-30T12:14:56.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83981216e0 con 0x7f8398135c20 2023-10-30T12:14:56.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.979+0000 7f839cf05640 1 librados: init done 2023-10-30T12:14:56.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839cf05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:56.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839cf05640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f839811fa40 0x7f839811fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839cf05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83981cae70 con 0x7f839811fa40 2023-10-30T12:14:56.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f839000aa90 con 0x7f8398135c20 2023-10-30T12:14:56.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f839811fa40 0x7f839811fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 10 monclient: get_auth_request con 0x7f839811fa40 auth_method 0 2023-10-30T12:14:56.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.980+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8378039320 0x7f837803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f839659c640 10 monclient(hunting): handle_auth_done global_id 114977 payload 824 2023-10-30T12:14:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f839659c640 1 -- 172.21.15.50:0/818039299 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f839809a5c0 con 0x7f839811fa40 2023-10-30T12:14:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f839004a870 con 0x7f8398135c20 2023-10-30T12:14:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 client.194154.objecter ms_dispatch 0x7f8398000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:14:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 3 client.194154.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:14:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 3 client.194154.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:14:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 20 client.194154.objecter dump_active .. 0 homeless 2023-10-30T12:14:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f839659c640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f839811fa40 0x7f839811fe20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f83980a54d0 tx=0x7f8388004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 client.194154.objecter ms_handle_connect 0x7f839811fa40 2023-10-30T12:14:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 client.194154.objecter resend_mon_ops 2023-10-30T12:14:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838ffff640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8378039320 0x7f837803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838ffff640 10 monclient: get_auth_request con 0x7f8378039320 auth_method 0 2023-10-30T12:14:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83880329c0 con 0x7f839811fa40 2023-10-30T12:14:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 monclient: dump: 2023-10-30T12:14:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.031 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 client.194154.objecter ms_handle_connect 0x7f8378039320 2023-10-30T12:14:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838ffff640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8378039320 0x7f837803b7e0 secure :-1 s=READY pgs=4186 cs=0 l=1 rev1=1 crypto rx=0x7f8398121a60 tx=0x7f8380008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8388032b20 con 0x7f839811fa40 2023-10-30T12:14:56.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:56.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:56.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f839811fa40 msgr2=0x7f839811fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f839811fa40 0x7f839811fe20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f83980a54d0 tx=0x7f8388004730 comp rx=0 tx=0).stop 2023-10-30T12:14:56.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f839811fa40 msgr2=0x7f839811fe20 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.981+0000 7f838effd640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f839811fa40 0x7f839811fe20 secure :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f83980a54d0 tx=0x7f8388004730 comp rx=0 tx=0).stop 2023-10-30T12:14:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.987+0000 7f839cf05640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.987+0000 7f839cf05640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8398120360 0x7f839809dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.987+0000 7f839cf05640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.987+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8398073ff0 con 0x7f8398120360 2023-10-30T12:14:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8398120360 0x7f839809dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient: get_auth_request con 0x7f8398120360 auth_method 0 2023-10-30T12:14:56.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 10 monclient(hunting): handle_auth_done global_id 114980 payload 824 2023-10-30T12:14:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 1 -- 172.21.15.50:0/818039299 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f839806b060 con 0x7f8398120360 2023-10-30T12:14:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f839659c640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8398120360 0x7f839809dde0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f838802b8c0 tx=0x7f83880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 10 client.194154.objecter ms_handle_connect 0x7f8398120360 2023-10-30T12:14:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 10 client.194154.objecter resend_mon_ops 2023-10-30T12:14:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838804f660 con 0x7f8398120360 2023-10-30T12:14:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 10 monclient: dump: 2023-10-30T12:14:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.041 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.041 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f8388053070 con 0x7f8398120360 2023-10-30T12:14:56.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:56.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:56.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8398120360 msgr2=0x7f839809dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8398120360 0x7f839809dde0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f838802b8c0 tx=0x7f83880046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8398120360 msgr2=0x7f839809dde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:56.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.988+0000 7f838effd640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8398120360 0x7f839809dde0 secure :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f838802b8c0 tx=0x7f83880046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.043 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:56.043 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:56.043 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:56.043 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:14:56.044 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 150, 2023-10-30T12:14:56.044 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:56.044 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:14:56.044 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:56.044 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:56.044 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.045 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:14:56.045 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:56.045 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:56.045 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:56.045 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:56.046 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:56.046 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:56.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:56.046 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:56.046 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:56.046 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:56.047 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:56.047 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:56.047 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:56.047 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.047 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:56.048 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:56.048 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:56.048 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:56.048 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:56.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:56.049 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:56.049 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:56.049 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:56.049 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:56.049 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:56.049 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:56.050 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.050 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:56.050 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:56.050 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:56.051 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:56.051 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:56.051 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:56.051 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:56.051 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:56.051 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:56.052 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:56.052 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:56.052 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:56.052 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:56.052 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:56.052 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:56.053 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:56.053 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:56.053 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:56.053 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:56.053 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:56.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:56.054 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:56.054 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:56.054 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:56.054 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:56.054 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:56.055 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:56.055 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.055 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:56.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.055 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:56.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.056 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:56.056 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:56.056 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:56.056 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:56.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:56.057 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:56.057 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.057 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:56.058 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:56.058 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.058 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.059 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:56.059 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:56.059 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:56.059 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:56.059 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:56.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:56.060 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:56.060 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:56.060 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:56.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.061 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:56.061 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:56.061 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.062 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:56.062 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:56.062 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.063 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:56.063 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:56.063 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:56.063 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:56.063 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:56.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:56.064 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:56.064 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:56.064 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:56.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.065 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:56.065 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:56.065 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.066 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:56.066 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:56.066 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.066 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.067 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:56.067 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:56.067 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:56.067 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:56.068 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:56.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.068 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.068 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:56.069 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:56.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.069 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:56.069 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:56.069 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:56.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.070 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.070 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:56.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.070 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:56.070 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:56.071 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:14:56.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.071 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.071 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:56.072 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 10 librados: watch_flush enter 2023-10-30T12:14:56.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 10 librados: watch_flush exit 2023-10-30T12:14:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 20 client.194154.objecter shutdown clearing up homeless session... 2023-10-30T12:14:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 10 client.194154.objecter successfully canceled tick 2023-10-30T12:14:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8378039320 msgr2=0x7f837803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8378039320 0x7f837803b7e0 secure :-1 s=READY pgs=4186 cs=0 l=1 rev1=1 crypto rx=0x7f8398121a60 tx=0x7f8380008040 comp rx=0 tx=0).stop 2023-10-30T12:14:56.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 10 monclient: shutdown 2023-10-30T12:14:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8398135c20 msgr2=0x7f8398132820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 --2- 172.21.15.50:0/818039299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8398135c20 0x7f8398132820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f839000c7a0 tx=0x7f839000cc60 comp rx=0 tx=0).stop 2023-10-30T12:14:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f8396d9d640 1 -- 172.21.15.50:0/818039299 reap_dead start 2023-10-30T12:14:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f8396d9d640 1 -- 172.21.15.50:0/818039299 reap_dead start 2023-10-30T12:14:56.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 shutdown_connections 2023-10-30T12:14:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 >> 172.21.15.50:0/818039299 conn(0x7f83980a80b0 msgr2=0x7f83980a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 shutdown_connections 2023-10-30T12:14:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 -- 172.21.15.50:0/818039299 wait complete. 2023-10-30T12:14:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:55.989+0000 7f839cf05640 1 librados: shutdown 2023-10-30T12:14:56.077 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:14:56.077 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:14:56.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:14:56.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.182+0000 7f097b8b0640 10 monclient: get_monmap_and_config 2023-10-30T12:14:56.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.182+0000 7f097b8b0640 10 monclient: build_initial_monmap 2023-10-30T12:14:56.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.182+0000 7f097b8b0640 10 monclient: monmap: 2023-10-30T12:14:56.184 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:56.184 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.184 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:56.183252+0000 2023-10-30T12:14:56.185 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:56.183252+0000 2023-10-30T12:14:56.185 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:56.185 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.185 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:56.185 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:56.185 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:56.186 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:56.186 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:56.186 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.182+0000 7f097b8b0640 1 Processor -- start 2023-10-30T12:14:56.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.182+0000 7f097b8b0640 1 -- start start 2023-10-30T12:14:56.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.182+0000 7f097b8b0640 10 monclient: init 2023-10-30T12:14:56.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:56.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-10-30T12:14:56.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0974097780 0x7f0974097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient(hunting): picked mon.noname-a con 0x7f0974097780 addr v2:172.21.15.50:3300/0 2023-10-30T12:14:56.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974073b10 0x7f09740a0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient(hunting): picked mon.noname-b con 0x7f0974073b10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:56.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f097409ccd0 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:56.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0974071ec0 con 0x7f097409ccd0 2023-10-30T12:14:56.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097409bd10 con 0x7f0974097780 2023-10-30T12:14:56.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097409bff0 con 0x7f0974073b10 2023-10-30T12:14:56.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:56.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097b8b0640 10 monclient(hunting): authenticate will time out at 1741.366500s 2023-10-30T12:14:56.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0974097780 0x7f0974097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974073b10 0x7f09740a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0974097780 0x7f0974097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55450/0 (socket says 172.21.15.50:55450) 2023-10-30T12:14:56.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974073b10 0x7f09740a0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34198/0 (socket says 172.21.15.50:34198) 2023-10-30T12:14:56.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 1 -- 172.21.15.50:0/2674813390 learned_addr learned my addr 172.21.15.50:0/2674813390 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:56.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 10 monclient(hunting): get_auth_request con 0x7f0974097780 auth_method 0 2023-10-30T12:14:56.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0979e4d640 1 --1- 172.21.15.50:0/2674813390 >> v1:172.21.15.50:6789/0 conn(0x7f097409ccd0 0x7f09740a05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40462/0 (socket says 172.21.15.50:40462) 2023-10-30T12:14:56.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): get_auth_request con 0x7f0974073b10 auth_method 0 2023-10-30T12:14:56.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f097964c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient(hunting): handle_auth_done global_id 114983 payload 274 2023-10-30T12:14:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 1 -- 172.21.15.50:0/2674813390 >> v2:172.21.15.50:3300/0 conn(0x7f0974097780 msgr2=0x7f0974097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 1 --2- 172.21.15.50:0/2674813390 >> v2:172.21.15.50:3300/0 conn(0x7f0974097780 0x7f0974097b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 1 -- 172.21.15.50:0/2674813390 >> v1:172.21.15.50:6789/0 conn(0x7f097409ccd0 legacy=0x7f09740a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient: _finish_hunting 0 2023-10-30T12:14:56.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 1 monclient: found mon.noname-b 2023-10-30T12:14:56.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:56.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:14:56.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0967fff640 1 -- 172.21.15.50:0/2674813390 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679576356 0 0) 0x7f0974071ec0 con 0x7f097409ccd0 2023-10-30T12:14:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.183+0000 7f0978e4b640 1 -- 172.21.15.50:0/2674813390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09740980a0 con 0x7f0974073b10 2023-10-30T12:14:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f097964c640 1 --2- 172.21.15.50:0/2674813390 >> v2:172.21.15.50:3300/0 conn(0x7f0974097780 0x7f0974097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:14:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0978e4b640 1 --2- 172.21.15.50:0/2674813390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974073b10 0x7f09740a0080 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f09700097a0 tx=0x7f097002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0d8b76884bbb19 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 1 -- 172.21.15.50:0/2674813390 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700316f0 con 0x7f0974073b10 2023-10-30T12:14:56.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:14:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: dump: 2023-10-30T12:14:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.205 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.206 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.207 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: _finish_auth 0 2023-10-30T12:14:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: _check_auth_tickets 2023-10-30T12:14:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:56.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f097b8b0640 5 monclient: authenticate success, global_id 114983 2023-10-30T12:14:56.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 1 -- 172.21.15.50:0/2674813390 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0970031850 con 0x7f0974073b10 2023-10-30T12:14:56.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:56.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 1 -- 172.21.15.50:0/2674813390 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970031be0 con 0x7f0974073b10 2023-10-30T12:14:56.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:56.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f0967fff640 10 monclient: dump: 2023-10-30T12:14:56.209 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.209 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.209 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.209 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.209 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.210 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.210 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.210 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.210 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.210 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f097b8b0640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:56.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.184+0000 7f097b8b0640 4 set_mon_vals no callback set 2023-10-30T12:14:56.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 10 monclient: shutdown 2023-10-30T12:14:56.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:56.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 -- 172.21.15.50:0/2674813390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974073b10 msgr2=0x7f09740a0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 --2- 172.21.15.50:0/2674813390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974073b10 0x7f09740a0080 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f09700097a0 tx=0x7f097002b790 comp rx=0 tx=0).stop 2023-10-30T12:14:56.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 -- 172.21.15.50:0/2674813390 shutdown_connections 2023-10-30T12:14:56.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 --2- 172.21.15.50:0/2674813390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974073b10 0x7f09740a0080 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 --2- 172.21.15.50:0/2674813390 >> v2:172.21.15.50:3300/0 conn(0x7f0974097780 0x7f0974097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 -- 172.21.15.50:0/2674813390 >> 172.21.15.50:0/2674813390 conn(0x7f09740a1170 msgr2=0x7f09740a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:56.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 -- 172.21.15.50:0/2674813390 shutdown_connections 2023-10-30T12:14:56.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.185+0000 7f097b8b0640 1 -- 172.21.15.50:0/2674813390 wait complete. 2023-10-30T12:14:56.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient: build_initial_monmap 2023-10-30T12:14:56.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient: monmap: 2023-10-30T12:14:56.213 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:56.213 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:56.213 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:56.214 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:56.214 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:56.214 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.214 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:56.214 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:56.215 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:56.215 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 librados: starting msgr at 2023-10-30T12:14:56.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 librados: starting objecter 2023-10-30T12:14:56.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 Processor -- start 2023-10-30T12:14:56.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 -- start start 2023-10-30T12:14:56.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 librados: setting wanted keys 2023-10-30T12:14:56.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 librados: calling monclient init 2023-10-30T12:14:56.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient: init 2023-10-30T12:14:56.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:56.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:14:56.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0974073b10 0x7f097408f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): picked mon.noname-a con 0x7f0974073b10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:56.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974097780 0x7f097408fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): picked mon.noname-c con 0x7f0974097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:56.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097409ccd0 0x7f097408fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): picked mon.noname-b con 0x7f097409ccd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:56.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0974092a80 con 0x7f0974073b10 2023-10-30T12:14:56.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09740919c0 con 0x7f097409ccd0 2023-10-30T12:14:56.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0974073f70 con 0x7f0974097780 2023-10-30T12:14:56.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:56.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.186+0000 7f097b8b0640 10 monclient(hunting): authenticate will time out at 1741.370003s 2023-10-30T12:14:56.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0974073b10 0x7f097408f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974097780 0x7f097408fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0974073b10 0x7f097408f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55462/0 (socket says 172.21.15.50:55462) 2023-10-30T12:14:56.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 -- 172.21.15.50:0/157414444 learned_addr learned my addr 172.21.15.50:0/157414444 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:56.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): get_auth_request con 0x7f0974073b10 auth_method 0 2023-10-30T12:14:56.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 10 monclient(hunting): get_auth_request con 0x7f0974097780 auth_method 0 2023-10-30T12:14:56.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0979e4d640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097409ccd0 0x7f097408fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient(hunting): handle_auth_done global_id 194163 payload 824 2023-10-30T12:14:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974097780 msgr2=0x7f097408fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974097780 0x7f097408fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097409ccd0 msgr2=0x7f097408fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097409ccd0 0x7f097408fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient: _finish_hunting 0 2023-10-30T12:14:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 monclient: found mon.noname-a 2023-10-30T12:14:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 -- 172.21.15.50:0/157414444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680094d0 con 0x7f0974073b10 2023-10-30T12:14:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0979e4d640 10 monclient: get_auth_request con 0x7f097409ccd0 auth_method 0 2023-10-30T12:14:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0979e4d640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097409ccd0 0x7f097408fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:14:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0978e4b640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0974097780 0x7f097408fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097964c640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0974073b10 0x7f097408f570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f096800ea00 tx=0x7f096800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 client.?.objecter ms_handle_connect 0x7f0974073b10 2023-10-30T12:14:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096800ca60 con 0x7f0974073b10 2023-10-30T12:14:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: dump: 2023-10-30T12:14:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.227 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.228 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.228 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.228 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.228 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: _finish_auth 0 2023-10-30T12:14:56.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:14:56.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:56.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097b8b0640 5 monclient: authenticate success, global_id 194163 2023-10-30T12:14:56.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096800cbc0 con 0x7f0974073b10 2023-10-30T12:14:56.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:56.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968013470 con 0x7f0974073b10 2023-10-30T12:14:56.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097b8b0640 10 monclient: _renew_subs 2023-10-30T12:14:56.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097b8b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0970009400 con 0x7f0974073b10 2023-10-30T12:14:56.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f09677fe640 4 set_mon_vals no callback set 2023-10-30T12:14:56.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:56.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.187+0000 7f0965ffb640 10 monclient: dump: 2023-10-30T12:14:56.231 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.231 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.231 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.231 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.232 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.232 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.232 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.232 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.232 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.233 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 10 client.194163.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:56.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 10 monclient: _renew_subs 2023-10-30T12:14:56.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0974093320 con 0x7f0974073b10 2023-10-30T12:14:56.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 1 librados: init done 2023-10-30T12:14:56.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f09680135d0 con 0x7f0974073b10 2023-10-30T12:14:56.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:56.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408bfe0 0x7f097408c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097406a5a0 con 0x7f097408bfe0 2023-10-30T12:14:56.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f0965ffb640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f09580392f0 0x7f095803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f096804b350 con 0x7f0974073b10 2023-10-30T12:14:56.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f0965ffb640 10 client.194163.objecter ms_dispatch 0x7f0974000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:14:56.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f0965ffb640 3 client.194163.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:14:56.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.188+0000 7f0965ffb640 3 client.194163.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:14:56.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408bfe0 0x7f097408c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 20 client.194163.objecter dump_active .. 0 homeless 2023-10-30T12:14:56.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient: get_auth_request con 0x7f097408bfe0 auth_method 0 2023-10-30T12:14:56.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0979e4d640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f09580392f0 0x7f095803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0979e4d640 10 monclient: get_auth_request con 0x7f09580392f0 auth_method 0 2023-10-30T12:14:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 10 monclient(hunting): handle_auth_done global_id 184258 payload 824 2023-10-30T12:14:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 1 -- 172.21.15.50:0/157414444 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09740a0cd0 con 0x7f097408bfe0 2023-10-30T12:14:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0979e4d640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f09580392f0 0x7f095803b7b0 secure :-1 s=READY pgs=4187 cs=0 l=1 rev1=1 crypto rx=0x7f0974090c40 tx=0x7f096c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 10 client.194163.objecter ms_handle_connect 0x7f09580392f0 2023-10-30T12:14:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 10 client.194163.objecter ms_handle_connect 0x7f097408bfe0 2023-10-30T12:14:56.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 10 client.194163.objecter resend_mon_ops 2023-10-30T12:14:56.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0978e4b640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408bfe0 0x7f097408c3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f097409e240 tx=0x7f09700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970034c10 con 0x7f097408bfe0 2023-10-30T12:14:56.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:56.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.189+0000 7f0965ffb640 10 monclient: dump: 2023-10-30T12:14:56.242 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.242 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.242 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.242 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.242 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.243 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.243 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.243 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.243 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.243 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.190+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0970042070 con 0x7f097408bfe0 2023-10-30T12:14:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.190+0000 7f0965ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.190+0000 7f0965ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.190+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408bfe0 msgr2=0x7f097408c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.190+0000 7f0965ffb640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408bfe0 0x7f097408c3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f097409e240 tx=0x7f09700046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.190+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408bfe0 msgr2=0x7f097408c3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.190+0000 7f0965ffb640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408bfe0 0x7f097408c3c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f097b8b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:14:56.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f097b8b0640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408c900 0x7f09741d46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f097b8b0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0974092f60 con 0x7f097408c900 2023-10-30T12:14:56.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408c900 0x7f09741d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 10 monclient: get_auth_request con 0x7f097408c900 auth_method 0 2023-10-30T12:14:56.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.196+0000 7f0978e4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0978e4b640 10 monclient(hunting): handle_auth_done global_id 184261 payload 824 2023-10-30T12:14:56.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0978e4b640 1 -- 172.21.15.50:0/157414444 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f097404e160 con 0x7f097408c900 2023-10-30T12:14:56.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0978e4b640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408c900 0x7f09741d46b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f097002b7d0 tx=0x7f0970031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 10 client.194163.objecter ms_handle_connect 0x7f097408c900 2023-10-30T12:14:56.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 10 client.194163.objecter resend_mon_ops 2023-10-30T12:14:56.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970047ab0 con 0x7f097408c900 2023-10-30T12:14:56.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:14:56.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 10 monclient: dump: 2023-10-30T12:14:56.250 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.250 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.251 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.251 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.251 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.251 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.251 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.251 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.252 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f0970047c10 con 0x7f097408c900 2023-10-30T12:14:56.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:14:56.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:56.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408c900 msgr2=0x7f09741d46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408c900 0x7f09741d46b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f097002b7d0 tx=0x7f0970031000 comp rx=0 tx=0).stop 2023-10-30T12:14:56.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408c900 msgr2=0x7f09741d46b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:14:56.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.197+0000 7f0965ffb640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f097408c900 0x7f09741d46b0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f097002b7d0 tx=0x7f0970031000 comp rx=0 tx=0).stop 2023-10-30T12:14:56.253 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:14:56.254 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:56.254 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:56.254 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:14:56.254 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 150, 2023-10-30T12:14:56.255 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:14:56.255 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:14:56.255 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:14:56.255 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:14:56.255 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.255 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:14:56.256 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:56.256 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:14:56.256 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:14:56.256 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:56.256 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:56.257 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:56.257 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:56.257 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:56.257 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:56.257 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:56.257 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:56.258 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:56.258 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:56.258 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.258 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:14:56.258 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:14:56.259 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:56.259 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:56.259 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:56.259 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:56.259 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:56.259 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:56.260 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:56.260 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:56.260 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:56.260 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:56.260 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:14:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:14:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:14:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:14:56.261 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:14:56.262 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:14:56.262 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:56.262 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:14:56.262 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:14:56.262 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:14:56.262 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:14:56.263 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:14:56.263 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:14:56.263 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:14:56.263 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:14:56.263 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:14:56.263 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:14:56.264 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:14:56.264 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:14:56.264 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:14:56.264 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:14:56.264 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:14:56.264 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:14:56.265 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:14:56.265 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:14:56.265 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:14:56.265 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:14:56.265 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.266 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:14:56.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.266 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:14:56.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.266 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:14:56.266 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:14:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:14:56.267 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.268 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:56.268 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:14:56.268 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.268 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.269 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.269 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:14:56.269 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:14:56.269 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:56.270 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:56.270 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:56.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.270 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:14:56.270 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:14:56.271 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:56.271 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:56.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.271 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:56.271 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:14:56.271 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.272 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:56.272 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:14:56.272 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.272 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.273 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.273 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:14:56.273 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:14:56.273 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:56.274 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:56.274 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:56.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.274 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:14:56.274 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:14:56.275 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:14:56.275 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:14:56.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.275 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:14:56.275 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:14:56.275 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.276 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:14:56.276 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:14:56.276 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:14:56.276 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.277 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.277 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:14:56.277 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:14:56.277 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:14:56.277 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:14:56.278 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:14:56.278 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.278 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.278 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:14:56.279 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:14:56.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.279 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:56.279 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:56.279 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:56.279 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.280 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.280 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:14:56.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.280 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:56.280 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:56.281 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:14:56.281 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.281 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:14:56.281 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:14:56.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:14:56.282 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:14:56.282 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:14:56.282 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:14:56.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:14:56.282 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:14:56.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:14:56.283 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:14:56.283 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:14:56.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 10 librados: watch_flush enter 2023-10-30T12:14:56.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 10 librados: watch_flush exit 2023-10-30T12:14:56.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 20 client.194163.objecter shutdown clearing up homeless session... 2023-10-30T12:14:56.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 10 client.194163.objecter successfully canceled tick 2023-10-30T12:14:56.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f09580392f0 msgr2=0x7f095803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f09580392f0 0x7f095803b7b0 secure :-1 s=READY pgs=4187 cs=0 l=1 rev1=1 crypto rx=0x7f0974090c40 tx=0x7f096c009040 comp rx=0 tx=0).stop 2023-10-30T12:14:56.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 10 monclient: shutdown 2023-10-30T12:14:56.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:56.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0974073b10 msgr2=0x7f097408f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 --2- 172.21.15.50:0/157414444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0974073b10 0x7f097408f570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f096800ea00 tx=0x7f096800eec0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f0979e4d640 1 -- 172.21.15.50:0/157414444 reap_dead start 2023-10-30T12:14:56.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f0979e4d640 1 -- 172.21.15.50:0/157414444 reap_dead start 2023-10-30T12:14:56.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 shutdown_connections 2023-10-30T12:14:56.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 >> 172.21.15.50:0/157414444 conn(0x7f09740a1170 msgr2=0x7f09740a2570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:56.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 shutdown_connections 2023-10-30T12:14:56.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 -- 172.21.15.50:0/157414444 wait complete. 2023-10-30T12:14:56.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.198+0000 7f097b8b0640 1 librados: shutdown 2023-10-30T12:14:56.288 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:14:56.289 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-10-30T12:14:56.289 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:14:56.289 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-10-30T12:14:56.289 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:14:56.290 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:14:56.290 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:14:56.303 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:14:56.303 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:14:56.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:14:56.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.406+0000 7f8ffca10640 10 monclient: get_monmap_and_config 2023-10-30T12:14:56.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.406+0000 7f8ffca10640 10 monclient: build_initial_monmap 2023-10-30T12:14:56.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.406+0000 7f8ffca10640 10 monclient: monmap: 2023-10-30T12:14:56.408 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:56.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:56.407755+0000 2023-10-30T12:14:56.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:56.407755+0000 2023-10-30T12:14:56.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:56.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.410 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:56.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:56.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:56.410 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:56.410 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:56.411 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 1 Processor -- start 2023-10-30T12:14:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 1 -- start start 2023-10-30T12:14:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient: init 2023-10-30T12:14:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:14:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098520 0x7f8ff809bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient(hunting): picked mon.noname-b con 0x7f8ff8098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ff8098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ff8137980 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff8099930 con 0x7f8ff8098900 2023-10-30T12:14:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff809aca0 con 0x7f8ff8137980 2023-10-30T12:14:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff809af80 con 0x7f8ff8098520 2023-10-30T12:14:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.407+0000 7f8ffca10640 10 monclient(hunting): authenticate will time out at 1741.590958s 2023-10-30T12:14:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098520 0x7f8ff809bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff6d9d640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f8ff8137980 legacy=0x7f8ff8139d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:14:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098520 0x7f8ff809bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34214/0 (socket says 172.21.15.50:34214) 2023-10-30T12:14:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 1 -- 172.21.15.50:0/1173987935 learned_addr learned my addr 172.21.15.50:0/1173987935 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): get_auth_request con 0x7f8ff8098520 auth_method 0 2023-10-30T12:14:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8feffff640 1 --1- 172.21.15.50:0/1173987935 >> v1:172.21.15.50:6789/0 conn(0x7f8ff8098900 0x7f8ff8098d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40478/0 (socket says 172.21.15.50:40478) 2023-10-30T12:14:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient(hunting): handle_auth_done global_id 114989 payload 274 2023-10-30T12:14:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 1 -- 172.21.15.50:0/1173987935 >> v1:172.21.15.118:6789/0 conn(0x7f8ff8137980 legacy=0x7f8ff8139d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff5d9b640 1 -- 172.21.15.50:0/1173987935 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813037015 0 0) 0x7f8ff8099930 con 0x7f8ff8098900 2023-10-30T12:14:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 1 -- 172.21.15.50:0/1173987935 >> v1:172.21.15.50:6789/0 conn(0x7f8ff8098900 legacy=0x7f8ff8098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient: _finish_hunting 0 2023-10-30T12:14:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 1 monclient: found mon.noname-b 2023-10-30T12:14:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:14:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff659c640 1 -- 172.21.15.50:0/1173987935 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff809b260 con 0x7f8ff8098520 2023-10-30T12:14:56.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.408+0000 7f8ff5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:14:56.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff659c640 1 --2- 172.21.15.50:0/1173987935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098520 0x7f8ff809bf70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009880 tx=0x7f8fe802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208c703b460746bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 1 -- 172.21.15.50:0/1173987935 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe802e070 con 0x7f8ff8098520 2023-10-30T12:14:56.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:14:56.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: dump: 2023-10-30T12:14:56.426 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.426 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.426 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.426 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.427 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.427 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.427 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.427 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.427 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.428 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:14:56.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:14:56.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:56.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 1 -- 172.21.15.50:0/1173987935 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe8004030 con 0x7f8ff8098520 2023-10-30T12:14:56.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 5 monclient: authenticate success, global_id 114989 2023-10-30T12:14:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 1 -- 172.21.15.50:0/1173987935 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe80043a0 con 0x7f8ff8098520 2023-10-30T12:14:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 4 set_mon_vals no callback set 2023-10-30T12:14:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ff5d9b640 10 monclient: dump: 2023-10-30T12:14:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.432 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 10 monclient: shutdown 2023-10-30T12:14:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 1 -- 172.21.15.50:0/1173987935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098520 msgr2=0x7f8ff809bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 1 --2- 172.21.15.50:0/1173987935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098520 0x7f8ff809bf70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009880 tx=0x7f8fe802b840 comp rx=0 tx=0).stop 2023-10-30T12:14:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 1 -- 172.21.15.50:0/1173987935 shutdown_connections 2023-10-30T12:14:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 1 --2- 172.21.15.50:0/1173987935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098520 0x7f8ff809bf70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 1 -- 172.21.15.50:0/1173987935 >> 172.21.15.50:0/1173987935 conn(0x7f8ff80a05c0 msgr2=0x7f8ff809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 1 -- 172.21.15.50:0/1173987935 shutdown_connections 2023-10-30T12:14:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.409+0000 7f8ffca10640 1 -- 172.21.15.50:0/1173987935 wait complete. 2023-10-30T12:14:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 10 monclient: build_initial_monmap 2023-10-30T12:14:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 10 monclient: monmap: 2023-10-30T12:14:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:56.435 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 1 librados: starting msgr at 2023-10-30T12:14:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 1 librados: starting objecter 2023-10-30T12:14:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 1 Processor -- start 2023-10-30T12:14:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 1 -- start start 2023-10-30T12:14:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 1 librados: setting wanted keys 2023-10-30T12:14:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 1 librados: calling monclient init 2023-10-30T12:14:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.410+0000 7f8ffca10640 10 monclient: init 2023-10-30T12:14:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:14:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ff8098520 0x7f8ff8136980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): picked mon.noname-a con 0x7f8ff8098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098900 0x7f8ff8136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): picked mon.noname-c con 0x7f8ff8098900 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ff8137980 0x7f8ff808be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): picked mon.noname-b con 0x7f8ff8137980 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:56.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80a2b90 con 0x7f8ff8098520 2023-10-30T12:14:56.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff808f8e0 con 0x7f8ff8137980 2023-10-30T12:14:56.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:56.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff8071d50 con 0x7f8ff8098900 2023-10-30T12:14:56.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:56.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ffca10640 10 monclient(hunting): authenticate will time out at 1741.594400s 2023-10-30T12:14:56.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098900 0x7f8ff8136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ff8098520 0x7f8ff8136980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff6d9d640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ff8137980 msgr2=0x7f8ff808be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:14:56.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098900 0x7f8ff8136ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34220/0 (socket says 172.21.15.50:34220) 2023-10-30T12:14:56.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 1 -- 172.21.15.50:0/2792334796 learned_addr learned my addr 172.21.15.50:0/2792334796 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:56.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): get_auth_request con 0x7f8ff8098520 auth_method 0 2023-10-30T12:14:56.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff6d9d640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ff8137980 0x7f8ff808be10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:56.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 10 monclient(hunting): get_auth_request con 0x7f8ff8098900 auth_method 0 2023-10-30T12:14:56.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:56.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:56.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:56.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:56.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:56.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:56.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient(hunting): handle_auth_done global_id 194169 payload 824 2023-10-30T12:14:56.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 -- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098900 msgr2=0x7f8ff8136ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098900 0x7f8ff8136ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 -- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ff8137980 msgr2=0x7f8ff808be10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:56.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ff8137980 0x7f8ff808be10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient: _finish_hunting 0 2023-10-30T12:14:56.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 monclient: found mon.noname-a 2023-10-30T12:14:56.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:56.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 -- 172.21.15.50:0/2792334796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe8009460 con 0x7f8ff8098520 2023-10-30T12:14:56.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feffff640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098900 0x7f8ff8136ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:14:56.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8ff659c640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ff8098520 0x7f8ff8136980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8004000 tx=0x7f8fe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feeffd640 10 client.?.objecter ms_handle_connect 0x7f8ff8098520 2023-10-30T12:14:56.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.411+0000 7f8feeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:56.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 1 -- 172.21.15.50:0/2792334796 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8003ce0 con 0x7f8ff8098520 2023-10-30T12:14:56.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:14:56.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: dump: 2023-10-30T12:14:56.447 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.448 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.448 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.448 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.448 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.448 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.448 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.449 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.449 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.449 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: _finish_auth 0 2023-10-30T12:14:56.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:14:56.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:56.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 5 monclient: authenticate success, global_id 194169 2023-10-30T12:14:56.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 1 -- 172.21.15.50:0/2792334796 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe8003e40 con 0x7f8ff8098520 2023-10-30T12:14:56.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:56.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 1 -- 172.21.15.50:0/2792334796 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8006280 con 0x7f8ff8098520 2023-10-30T12:14:56.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 10 monclient: _renew_subs 2023-10-30T12:14:56.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff808fae0 con 0x7f8ff8098520 2023-10-30T12:14:56.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ff559a640 4 set_mon_vals no callback set 2023-10-30T12:14:56.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:56.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:56.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8feeffd640 10 monclient: dump: 2023-10-30T12:14:56.452 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:56.452 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:56.452 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.452 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:56.453 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:56.453 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:56.453 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:56.453 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:56.453 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:56.454 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:56.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 10 client.194169.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:56.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 10 monclient: _renew_subs 2023-10-30T12:14:56.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff808d980 con 0x7f8ff8098520 2023-10-30T12:14:56.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.412+0000 7f8ffca10640 1 librados: init done 2023-10-30T12:14:56.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feeffd640 1 -- 172.21.15.50:0/2792334796 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8fe8039050 con 0x7f8ff8098520 2023-10-30T12:14:56.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8ffca10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:56.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8ffca10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:56.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8ffca10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff808e840 con 0x7f8ff8098520 2023-10-30T12:14:56.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feeffd640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fd80392a0 0x7f8fd803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:56.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feeffd640 1 -- 172.21.15.50:0/2792334796 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f8fe806c670 con 0x7f8ff8098520 2023-10-30T12:14:56.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feeffd640 10 client.194169.objecter ms_dispatch 0x7f8ff8000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:14:56.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feeffd640 3 client.194169.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:14:56.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feeffd640 3 client.194169.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:14:56.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feeffd640 20 client.194169.objecter dump_active .. 0 homeless 2023-10-30T12:14:56.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feffff640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fd80392a0 0x7f8fd803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:56.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.413+0000 7f8feffff640 10 monclient: get_auth_request con 0x7f8fd80392a0 auth_method 0 2023-10-30T12:14:56.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.414+0000 7f8feeffd640 10 client.194169.objecter ms_handle_connect 0x7f8fd80392a0 2023-10-30T12:14:56.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.414+0000 7f8feffff640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fd80392a0 0x7f8fd803b760 secure :-1 s=READY pgs=4188 cs=0 l=1 rev1=1 crypto rx=0x7f8ff81c9400 tx=0x7f8fe0007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:56.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.417+0000 7f8feeffd640 1 -- 172.21.15.50:0/2792334796 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ff808e840 con 0x7f8ff8098520 2023-10-30T12:14:56.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.417+0000 7f8feeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:56.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.417+0000 7f8feeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:14:56.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.663+0000 7f8ffca10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:14:56.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.663+0000 7f8ffca10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:56.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.663+0000 7f8ffca10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:56.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.663+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ff808e840 con 0x7f8ff8098520 2023-10-30T12:14:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.663+0000 7f8feeffd640 1 -- 172.21.15.50:0/2792334796 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f8ff808e840 con 0x7f8ff8098520 2023-10-30T12:14:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.663+0000 7f8feeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:14:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.663+0000 7f8feeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:14:56.666 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:14:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 10 librados: watch_flush enter 2023-10-30T12:14:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 10 librados: watch_flush exit 2023-10-30T12:14:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 20 client.194169.objecter shutdown clearing up homeless session... 2023-10-30T12:14:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 10 client.194169.objecter successfully canceled tick 2023-10-30T12:14:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fd80392a0 msgr2=0x7f8fd803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fd80392a0 0x7f8fd803b760 secure :-1 s=READY pgs=4188 cs=0 l=1 rev1=1 crypto rx=0x7f8ff81c9400 tx=0x7f8fe0007e10 comp rx=0 tx=0).stop 2023-10-30T12:14:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 10 monclient: shutdown 2023-10-30T12:14:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ff8098520 msgr2=0x7f8ff8136980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ff8098520 0x7f8ff8136980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8004000 tx=0x7f8fe80046c0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 shutdown_connections 2023-10-30T12:14:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8fd80392a0 0x7f8fd803b760 unknown :-1 s=CLOSED pgs=4188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8ff8137980 0x7f8ff808be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ff8098900 0x7f8ff8136ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 --2- 172.21.15.50:0/2792334796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ff8098520 0x7f8ff8136980 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 >> 172.21.15.50:0/2792334796 conn(0x7f8ff80a05c0 msgr2=0x7f8ff8097410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 shutdown_connections 2023-10-30T12:14:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 -- 172.21.15.50:0/2792334796 wait complete. 2023-10-30T12:14:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:56.665+0000 7f8ffca10640 1 librados: shutdown 2023-10-30T12:14:59.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:14:59.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.793+0000 7f2fdd9fc640 10 monclient: get_monmap_and_config 2023-10-30T12:14:59.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.793+0000 7f2fdd9fc640 10 monclient: build_initial_monmap 2023-10-30T12:14:59.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.793+0000 7f2fdd9fc640 10 monclient: monmap: 2023-10-30T12:14:59.795 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:59.795 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:59.796 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:14:59.794157+0000 2023-10-30T12:14:59.796 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:14:59.794157+0000 2023-10-30T12:14:59.796 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:59.796 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:59.796 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:14:59.797 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:14:59.797 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:14:59.797 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:14:59.797 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:14:59.797 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:59.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.793+0000 7f2fdd9fc640 1 Processor -- start 2023-10-30T12:14:59.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.793+0000 7f2fdd9fc640 1 -- start start 2023-10-30T12:14:59.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.793+0000 7f2fdd9fc640 10 monclient: init 2023-10-30T12:14:59.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:59.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:14:59.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fd80a2dd0 addr v1:172.21.15.118:6789/0 2023-10-30T12:14:59.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2fd806a8e0 0x7f2fd80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd806a8e0 addr v2:172.21.15.118:3300/0 2023-10-30T12:14:59.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fd8071400 addr v1:172.21.15.50:6789/0 2023-10-30T12:14:59.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd80a31b0 con 0x7f2fd8071400 2023-10-30T12:14:59.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd806a6a0 con 0x7f2fd80a2dd0 2023-10-30T12:14:59.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:59.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd806be90 con 0x7f2fd806a8e0 2023-10-30T12:14:59.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:59.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 10 monclient(hunting): authenticate will time out at 1744.977227s 2023-10-30T12:14:59.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd67fc640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f2fd806a8e0 msgr2=0x7f2fd80a4dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:14:59.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd6ffd640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f2fd80a2dd0 legacy=0x7f2fd80a46c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:14:59.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd77fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2fd8071400 0x7f2fd813cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50446/0 (socket says 172.21.15.50:50446) 2023-10-30T12:14:59.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd77fe640 1 -- 172.21.15.50:0/455579143 learned_addr learned my addr 172.21.15.50:0/455579143 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:14:59.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/455579143 >> v2:172.21.15.118:3300/0 conn(0x7f2fd806a8e0 0x7f2fd80a4dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:59.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353963028 0 0) 0x7f2fd80a31b0 con 0x7f2fd8071400 2023-10-30T12:14:59.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:14:59.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:59.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:59.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient(hunting): my global_id is 194172 2023-10-30T12:14:59.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc4003520 con 0x7f2fd8071400 2023-10-30T12:14:59.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1925515391 0 0) 0x7f2fc4003520 con 0x7f2fd8071400 2023-10-30T12:14:59.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 >> v2:172.21.15.118:3300/0 conn(0x7f2fd806a8e0 msgr2=0x7f2fd80a4dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:59.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 --2- 172.21.15.50:0/455579143 >> v2:172.21.15.118:3300/0 conn(0x7f2fd806a8e0 0x7f2fd80a4dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:59.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 >> v1:172.21.15.118:6789/0 conn(0x7f2fd80a2dd0 legacy=0x7f2fd80a46c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:59.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: _finish_hunting 0 2023-10-30T12:14:59.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:14:59.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:59.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:14:59.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd809e190 con 0x7f2fd8071400 2023-10-30T12:14:59.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:14:59.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:14:59.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:59.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 5 monclient: authenticate success, global_id 194172 2023-10-30T12:14:59.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f2fd0001670 con 0x7f2fd8071400 2023-10-30T12:14:59.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fdd9fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:59.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:59.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:14:59.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: dump: 2023-10-30T12:14:59.814 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:59.814 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:59.814 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.814 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.814 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:59.815 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:59.815 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:59.815 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:59.815 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:59.815 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:59.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:14:59.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:59.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 >> v1:172.21.15.50:6789/0 conn(0x7f2fd8071400 legacy=0x7f2fd813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:59.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:14:59.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.794+0000 7f2fd5ffb640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fc4003970 0x7f2fc4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): picked mon.c con 0x7f2fc4003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:59.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fc40047c0 0x7f2fc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): picked mon.a con 0x7f2fc40047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:59.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc4005330 0x7f2fc400b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): picked mon.b con 0x7f2fc4005330 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:59.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:59.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd806be90 con 0x7f2fc40047c0 2023-10-30T12:14:59.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:59.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc400c280 con 0x7f2fc4005330 2023-10-30T12:14:59.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:59.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc400c440 con 0x7f2fc4003970 2023-10-30T12:14:59.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:59.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd5ffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:14:59.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fdd9fc640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:59.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fc4003970 0x7f2fc4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:59.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fc40047c0 0x7f2fc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:59.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 10 monclient(hunting): get_auth_request con 0x7f2fc4003970 auth_method 0 2023-10-30T12:14:59.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:59.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:59.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:59.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): get_auth_request con 0x7f2fc40047c0 auth_method 0 2023-10-30T12:14:59.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:59.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:59.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:14:59.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd77fe640 1 -- 172.21.15.50:0/455579143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc4005330 msgr2=0x7f2fc400b9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:14:59.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd77fe640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc4005330 0x7f2fc400b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:59.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:59.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:59.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:59.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:59.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:59.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:14:59.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_done global_id 194172 payload 293 2023-10-30T12:14:59.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 1 -- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fc4003970 msgr2=0x7f2fc4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:14:59.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fc4003970 0x7f2fc4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:59.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 1 -- 172.21.15.50:0/455579143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc4005330 msgr2=0x7f2fc400b9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:14:59.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc4005330 0x7f2fc400b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:59.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient: _finish_hunting 0 2023-10-30T12:14:59.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 1 monclient: found mon.a 2023-10-30T12:14:59.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:59.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:14:59.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd77fe640 1 -- 172.21.15.50:0/455579143 reap_dead start 2023-10-30T12:14:59.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 1 -- 172.21.15.50:0/455579143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4003520 con 0x7f2fc40047c0 2023-10-30T12:14:59.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient: _finish_auth 0 2023-10-30T12:14:59.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:14:59.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.795+0000 7f2fd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:59.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fc40047c0 0x7f2fc4004c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009900 tx=0x7f2fc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f936d54d19705b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:59.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0003bb0 con 0x7f2fc40047c0 2023-10-30T12:14:59.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:59.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:59.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 10 monclient: dump: 2023-10-30T12:14:59.826 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:59.827 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:59.827 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.827 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.827 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:59.827 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:59.827 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:59.828 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:59.828 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:59.828 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:59.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc0003d10 con 0x7f2fc40047c0 2023-10-30T12:14:59.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:14:59.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:59.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 10 monclient: get_monmap_and_config success 2023-10-30T12:14:59.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 4 set_mon_vals no callback set 2023-10-30T12:14:59.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 1 -- 172.21.15.50:0/455579143 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00052b0 con 0x7f2fc40047c0 2023-10-30T12:14:59.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:59.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:14:59.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fd5ffb640 10 monclient: dump: 2023-10-30T12:14:59.830 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:59.830 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:59.830 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.831 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.831 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:59.831 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:59.831 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:59.831 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:59.831 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:59.832 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:59.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 10 monclient: shutdown 2023-10-30T12:14:59.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:14:59.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fc40047c0 msgr2=0x7f2fc4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:59.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fc40047c0 0x7f2fc4004c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009900 tx=0x7f2fc40057f0 comp rx=0 tx=0).stop 2023-10-30T12:14:59.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/455579143 shutdown_connections 2023-10-30T12:14:59.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/455579143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fc40047c0 0x7f2fc4004c40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:59.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.796+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/455579143 >> 172.21.15.50:0/455579143 conn(0x7f2fd80a22b0 msgr2=0x7f2fd80a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:14:59.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.797+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/455579143 shutdown_connections 2023-10-30T12:14:59.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.797+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/455579143 wait complete. 2023-10-30T12:14:59.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.797+0000 7f2fdd9fc640 10 monclient: build_initial_monmap 2023-10-30T12:14:59.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.797+0000 7f2fdd9fc640 10 monclient: monmap: 2023-10-30T12:14:59.834 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:14:59.834 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:14:59.834 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:14:59.834 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:14:59.835 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:14:59.835 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:59.835 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:14:59.835 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:14:59.835 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:14:59.836 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:59.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.797+0000 7f2fdd9fc640 1 librados: starting msgr at 2023-10-30T12:14:59.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.797+0000 7f2fdd9fc640 1 librados: starting objecter 2023-10-30T12:14:59.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.797+0000 7f2fdd9fc640 1 Processor -- start 2023-10-30T12:14:59.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 -- start start 2023-10-30T12:14:59.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 librados: setting wanted keys 2023-10-30T12:14:59.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 librados: calling monclient init 2023-10-30T12:14:59.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient: init 2023-10-30T12:14:59.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:14:59.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:14:59.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc40047c0 0x7f2fd81c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): picked mon.noname-b con 0x7f2fc40047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:14:59.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fd81c68d0 0x7f2fd81c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd81c68d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:14:59.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fd81c9e10 0x7f2fd81ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd81c9e10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:14:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd806b130 con 0x7f2fd81c9e10 2023-10-30T12:14:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd81a8a10 con 0x7f2fc40047c0 2023-10-30T12:14:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:14:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80706f0 con 0x7f2fd81c68d0 2023-10-30T12:14:59.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:14:59.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fdd9fc640 10 monclient(hunting): authenticate will time out at 1744.981592s 2023-10-30T12:14:59.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd6ffd640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc40047c0 msgr2=0x7f2fd81c3c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:14:59.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fd81c68d0 0x7f2fd81c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:59.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fd81c9e10 0x7f2fd81ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:59.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc40047c0 0x7f2fd81c3c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:14:59.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fd81c68d0 0x7f2fd81c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52474/0 (socket says 172.21.15.50:52474) 2023-10-30T12:14:59.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 1 -- 172.21.15.50:0/1332817425 learned_addr learned my addr 172.21.15.50:0/1332817425 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:14:59.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 10 monclient(hunting): get_auth_request con 0x7f2fd81c68d0 auth_method 0 2023-10-30T12:14:59.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fd81c9e10 0x7f2fd81ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56348/0 (socket says 172.21.15.50:56348) 2023-10-30T12:14:59.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 10 monclient(hunting): get_auth_request con 0x7f2fd81c9e10 auth_method 0 2023-10-30T12:14:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:14:59.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:14:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:14:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:14:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:14:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.798+0000 7f2fd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:14:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 10 monclient(hunting): handle_auth_done global_id 114995 payload 824 2023-10-30T12:14:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 1 -- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc40047c0 msgr2=0x7f2fd81c3c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:14:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc40047c0 0x7f2fd81c3c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 1 -- 172.21.15.50:0/1332817425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fd81c9e10 msgr2=0x7f2fd81ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:14:59.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fd81c9e10 0x7f2fd81ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:14:59.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 10 monclient: _finish_hunting 0 2023-10-30T12:14:59.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 1 monclient: found mon.noname-c 2023-10-30T12:14:59.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:14:59.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:14:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 1 -- 172.21.15.50:0/1332817425 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd81aad10 con 0x7f2fd81c68d0 2023-10-30T12:14:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd67fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fd81c68d0 0x7f2fd81c6d50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2fc00090b0 tx=0x7f2fc0005a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 client.?.objecter ms_handle_connect 0x7f2fd81c68d0 2023-10-30T12:14:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:14:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 1 -- 172.21.15.50:0/1332817425 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0034670 con 0x7f2fd81c68d0 2023-10-30T12:14:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:14:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: dump: 2023-10-30T12:14:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:59.849 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:59.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: _finish_auth 0 2023-10-30T12:14:59.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: _check_auth_tickets 2023-10-30T12:14:59.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:14:59.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 1 -- 172.21.15.50:0/1332817425 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00347d0 con 0x7f2fd81c68d0 2023-10-30T12:14:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:14:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 1 -- 172.21.15.50:0/1332817425 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0034b60 con 0x7f2fd81c68d0 2023-10-30T12:14:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:14:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:14:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fcffff640 10 monclient: dump: 2023-10-30T12:14:59.851 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:14:59.851 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:14:59.851 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.851 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:14:59.851 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:14:59.851 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:14:59.852 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:14:59.852 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:14:59.852 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:14:59.852 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:14:59.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 5 monclient: authenticate success, global_id 114995 2023-10-30T12:14:59.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 10 monclient: _renew_subs 2023-10-30T12:14:59.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:59.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd81aaff0 con 0x7f2fd81c68d0 2023-10-30T12:14:59.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fd57fa640 4 set_mon_vals no callback set 2023-10-30T12:14:59.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 10 client.114995.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:14:59.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 10 monclient: _renew_subs 2023-10-30T12:14:59.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:59.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd81c7290 con 0x7f2fd81c68d0 2023-10-30T12:14:59.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.799+0000 7f2fdd9fc640 1 librados: init done 2023-10-30T12:14:59.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fcffff640 1 -- 172.21.15.50:0/1332817425 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2fc0031070 con 0x7f2fd81c68d0 2023-10-30T12:14:59.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fdd9fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:59.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fdd9fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:59.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fdd9fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:14:59.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fcffff640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fb80392a0 0x7f2fb803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:14:59.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd804e160 con 0x7f2fd81c68d0 2023-10-30T12:14:59.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fcffff640 1 -- 172.21.15.50:0/1332817425 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f2fc003b040 con 0x7f2fd81c68d0 2023-10-30T12:14:59.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fcffff640 10 client.114995.objecter ms_dispatch 0x7f2fd8000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:14:59.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fcffff640 3 client.114995.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:14:59.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fcffff640 3 client.114995.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:14:59.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fcffff640 20 client.114995.objecter dump_active .. 0 homeless 2023-10-30T12:14:59.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fd6ffd640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fb80392a0 0x7f2fb803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:14:59.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.800+0000 7f2fd6ffd640 10 monclient: get_auth_request con 0x7f2fb80392a0 auth_method 0 2023-10-30T12:14:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.801+0000 7f2fd6ffd640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fb80392a0 0x7f2fb803b760 secure :-1 s=READY pgs=4189 cs=0 l=1 rev1=1 crypto rx=0x7f2fd81a8ce0 tx=0x7f2fc8009520 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:14:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.801+0000 7f2fcffff640 10 client.114995.objecter ms_handle_connect 0x7f2fb80392a0 2023-10-30T12:14:59.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.805+0000 7f2fcffff640 1 -- 172.21.15.50:0/1332817425 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2fc00479b0 con 0x7f2fd81c68d0 2023-10-30T12:14:59.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.805+0000 7f2fcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:14:59.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:14:59.805+0000 7f2fcffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:00.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.049+0000 7f2fdd9fc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:00.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.049+0000 7f2fdd9fc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.049+0000 7f2fdd9fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.049+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2fd81a4230 con 0x7f2fd81c68d0 2023-10-30T12:15:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.050+0000 7f2fcffff640 1 -- 172.21.15.50:0/1332817425 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f2fc0047360 con 0x7f2fd81c68d0 2023-10-30T12:15:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.050+0000 7f2fcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.050+0000 7f2fcffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:00.053 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:00.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 10 librados: watch_flush enter 2023-10-30T12:15:00.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 10 librados: watch_flush exit 2023-10-30T12:15:00.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 20 client.114995.objecter shutdown clearing up homeless session... 2023-10-30T12:15:00.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 10 client.114995.objecter successfully canceled tick 2023-10-30T12:15:00.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fb80392a0 msgr2=0x7f2fb803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:00.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fb80392a0 0x7f2fb803b760 secure :-1 s=READY pgs=4189 cs=0 l=1 rev1=1 crypto rx=0x7f2fd81a8ce0 tx=0x7f2fc8009520 comp rx=0 tx=0).stop 2023-10-30T12:15:00.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 10 monclient: shutdown 2023-10-30T12:15:00.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:00.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fd81c68d0 msgr2=0x7f2fd81c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:00.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fd81c68d0 0x7f2fd81c6d50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2fc00090b0 tx=0x7f2fc0005a10 comp rx=0 tx=0).stop 2023-10-30T12:15:00.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 shutdown_connections 2023-10-30T12:15:00.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fb80392a0 0x7f2fb803b760 unknown :-1 s=CLOSED pgs=4189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:00.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fd81c9e10 0x7f2fd81ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:00.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fd81c68d0 0x7f2fd81c6d50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:00.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 --2- 172.21.15.50:0/1332817425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fc40047c0 0x7f2fd81c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:00.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 >> 172.21.15.50:0/1332817425 conn(0x7f2fd80a22b0 msgr2=0x7f2fd806e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:00.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 shutdown_connections 2023-10-30T12:15:00.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 -- 172.21.15.50:0/1332817425 wait complete. 2023-10-30T12:15:00.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:00.052+0000 7f2fdd9fc640 1 librados: shutdown 2023-10-30T12:15:03.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.180+0000 7f226a5b6640 10 monclient: get_monmap_and_config 2023-10-30T12:15:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.180+0000 7f226a5b6640 10 monclient: build_initial_monmap 2023-10-30T12:15:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.180+0000 7f226a5b6640 10 monclient: monmap: 2023-10-30T12:15:03.181 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:03.180653+0000 2023-10-30T12:15:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:03.180653+0000 2023-10-30T12:15:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:03.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 1 Processor -- start 2023-10-30T12:15:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 1 -- start start 2023-10-30T12:15:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient: init 2023-10-30T12:15:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-10-30T12:15:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2264098520 0x7f226409bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient(hunting): picked mon.noname-c con 0x7f2264098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2264098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264137980 0x7f2264139d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient(hunting): picked mon.noname-b con 0x7f2264137980 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2264099930 con 0x7f2264098900 2023-10-30T12:15:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226409aca0 con 0x7f2264098520 2023-10-30T12:15:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226409af80 con 0x7f2264137980 2023-10-30T12:15:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f226a5b6640 10 monclient(hunting): authenticate will time out at 1748.363937s 2023-10-30T12:15:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.181+0000 7f2263fff640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f2264098520 msgr2=0x7f226409bf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2263fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2264098520 0x7f226409bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264137980 0x7f2264139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f22637fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2264098900 0x7f2264098d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50462/0 (socket says 172.21.15.50:50462) 2023-10-30T12:15:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f22637fe640 1 -- 172.21.15.50:0/4081839937 learned_addr learned my addr 172.21.15.50:0/4081839937 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): get_auth_request con 0x7f2264137980 auth_method 0 2023-10-30T12:15:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient(hunting): handle_auth_done global_id 114998 payload 274 2023-10-30T12:15:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 1 -- 172.21.15.50:0/4081839937 >> v2:172.21.15.118:3300/0 conn(0x7f2264098520 msgr2=0x7f226409bf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 1 --2- 172.21.15.50:0/4081839937 >> v2:172.21.15.118:3300/0 conn(0x7f2264098520 0x7f226409bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 1 -- 172.21.15.50:0/4081839937 >> v1:172.21.15.50:6789/0 conn(0x7f2264098900 legacy=0x7f2264098d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient: _finish_hunting 0 2023-10-30T12:15:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 1 monclient: found mon.noname-b 2023-10-30T12:15:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.182+0000 7f2268b53640 1 -- 172.21.15.50:0/4081839937 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226409b260 con 0x7f2264137980 2023-10-30T12:15:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2268b53640 1 --2- 172.21.15.50:0/4081839937 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264137980 0x7f2264139d70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f225c009d90 tx=0x7f225c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c1c34eeef3eeec1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 1 -- 172.21.15.50:0/4081839937 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c02cb60 con 0x7f2264137980 2023-10-30T12:15:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: dump: 2023-10-30T12:15:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:03.201 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: _finish_auth 0 2023-10-30T12:15:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 1 -- 172.21.15.50:0/4081839937 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c02ccc0 con 0x7f2264137980 2023-10-30T12:15:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 1 -- 172.21.15.50:0/4081839937 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c034660 con 0x7f2264137980 2023-10-30T12:15:03.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 5 monclient: authenticate success, global_id 114998 2023-10-30T12:15:03.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:03.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 4 set_mon_vals no callback set 2023-10-30T12:15:03.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:03.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f2262ffd640 10 monclient: dump: 2023-10-30T12:15:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:03.205 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 10 monclient: shutdown 2023-10-30T12:15:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 -- 172.21.15.50:0/4081839937 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264137980 msgr2=0x7f2264139d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 --2- 172.21.15.50:0/4081839937 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264137980 0x7f2264139d70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f225c009d90 tx=0x7f225c02bdb0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 -- 172.21.15.50:0/4081839937 shutdown_connections 2023-10-30T12:15:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 --2- 172.21.15.50:0/4081839937 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264137980 0x7f2264139d70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 --2- 172.21.15.50:0/4081839937 >> v2:172.21.15.118:3300/0 conn(0x7f2264098520 0x7f226409bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 -- 172.21.15.50:0/4081839937 >> 172.21.15.50:0/4081839937 conn(0x7f22640a05c0 msgr2=0x7f226409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 -- 172.21.15.50:0/4081839937 shutdown_connections 2023-10-30T12:15:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.183+0000 7f226a5b6640 1 -- 172.21.15.50:0/4081839937 wait complete. 2023-10-30T12:15:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.184+0000 7f226a5b6640 10 monclient: build_initial_monmap 2023-10-30T12:15:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.184+0000 7f226a5b6640 10 monclient: monmap: 2023-10-30T12:15:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:03.211 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:03.212 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:03.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.184+0000 7f226a5b6640 1 librados: starting msgr at 2023-10-30T12:15:03.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.184+0000 7f226a5b6640 1 librados: starting objecter 2023-10-30T12:15:03.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.184+0000 7f226a5b6640 1 Processor -- start 2023-10-30T12:15:03.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 -- start start 2023-10-30T12:15:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 librados: setting wanted keys 2023-10-30T12:15:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 librados: calling monclient init 2023-10-30T12:15:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient: init 2023-10-30T12:15:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:15:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264098520 0x7f22641ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:03.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): picked mon.noname-c con 0x7f2264098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:03.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2264098900 0x7f22640961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:03.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): picked mon.noname-b con 0x7f2264098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:03.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2264137980 0x7f2264096730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:03.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): picked mon.noname-a con 0x7f2264137980 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2264092890 con 0x7f2264137980 2023-10-30T12:15:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22640921d0 con 0x7f2264098900 2023-10-30T12:15:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22641bda80 con 0x7f2264098520 2023-10-30T12:15:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f226a5b6640 10 monclient(hunting): authenticate will time out at 1748.367600s 2023-10-30T12:15:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264098520 0x7f22641ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264098520 0x7f22641ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52488/0 (socket says 172.21.15.50:52488) 2023-10-30T12:15:03.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 -- 172.21.15.50:0/3030871017 learned_addr learned my addr 172.21.15.50:0/3030871017 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:03.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): get_auth_request con 0x7f2264098520 auth_method 0 2023-10-30T12:15:03.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:03.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:03.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:03.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2264137980 0x7f2264096730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:03.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 10 monclient(hunting): get_auth_request con 0x7f2264137980 auth_method 0 2023-10-30T12:15:03.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:03.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:03.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:03.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f22637fe640 1 -- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2264098900 msgr2=0x7f22640961f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:03.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f22637fe640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2264098900 0x7f22640961f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:03.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:03.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:03.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:03.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:03.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:03.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2268b53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:03.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient(hunting): handle_auth_done global_id 115001 payload 824 2023-10-30T12:15:03.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 -- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2264098900 msgr2=0x7f22640961f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:03.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2264098900 0x7f22640961f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 -- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2264137980 msgr2=0x7f2264096730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2264137980 0x7f2264096730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 1 monclient: found mon.noname-c 2023-10-30T12:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.185+0000 7f2263fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2263fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2263fff640 1 -- 172.21.15.50:0/3030871017 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c009a70 con 0x7f2264098520 2023-10-30T12:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2263fff640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264098520 0x7f22641ca240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f225800b530 tx=0x7f225800b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 client.?.objecter ms_handle_connect 0x7f2264098520 2023-10-30T12:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 1 -- 172.21.15.50:0/3030871017 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258003e10 con 0x7f2264098520 2023-10-30T12:15:03.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:03.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:03.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: dump: 2023-10-30T12:15:03.223 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:03.223 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:03.224 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.224 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.224 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:03.224 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:03.224 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:03.225 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:03.225 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:03.225 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:03.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: _finish_auth 0 2023-10-30T12:15:03.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:15:03.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:03.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 1 -- 172.21.15.50:0/3030871017 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2258005070 con 0x7f2264098520 2023-10-30T12:15:03.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:03.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 1 -- 172.21.15.50:0/3030871017 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258005420 con 0x7f2264098520 2023-10-30T12:15:03.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:03.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:03.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f2260ff9640 10 monclient: dump: 2023-10-30T12:15:03.227 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:03.227 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:03.227 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.227 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:03.228 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:03.228 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:03.228 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:03.228 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:03.228 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:03.228 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:03.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 5 monclient: authenticate success, global_id 115001 2023-10-30T12:15:03.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 10 monclient: _renew_subs 2023-10-30T12:15:03.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:03.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2264071d50 con 0x7f2264098520 2023-10-30T12:15:03.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f22627fc640 4 set_mon_vals no callback set 2023-10-30T12:15:03.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 10 client.115001.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:03.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 10 monclient: _renew_subs 2023-10-30T12:15:03.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:03.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2264090be0 con 0x7f2264098520 2023-10-30T12:15:03.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.186+0000 7f226a5b6640 1 librados: init done 2023-10-30T12:15:03.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f2260ff9640 1 -- 172.21.15.50:0/3030871017 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2258005580 con 0x7f2264098520 2023-10-30T12:15:03.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f226a5b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:03.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f226a5b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:03.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f226a5b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:03.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226409c470 con 0x7f2264098520 2023-10-30T12:15:03.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f2260ff9640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f22500392a0 0x7f225003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f2260ff9640 1 -- 172.21.15.50:0/3030871017 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f225804aa20 con 0x7f2264098520 2023-10-30T12:15:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f2260ff9640 10 client.115001.objecter ms_dispatch 0x7f2264000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:15:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f2260ff9640 3 client.115001.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:15:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f2260ff9640 3 client.115001.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:15:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f2260ff9640 20 client.115001.objecter dump_active .. 0 homeless 2023-10-30T12:15:03.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f22637fe640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f22500392a0 0x7f225003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:03.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.187+0000 7f22637fe640 10 monclient: get_auth_request con 0x7f22500392a0 auth_method 0 2023-10-30T12:15:03.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.188+0000 7f22637fe640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f22500392a0 0x7f225003b760 secure :-1 s=READY pgs=4190 cs=0 l=1 rev1=1 crypto rx=0x7f2254007fd0 tx=0x7f2254007aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:03.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.188+0000 7f2260ff9640 10 client.115001.objecter ms_handle_connect 0x7f22500392a0 2023-10-30T12:15:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.192+0000 7f2260ff9640 1 -- 172.21.15.50:0/3030871017 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2258020350 con 0x7f2264098520 2023-10-30T12:15:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.192+0000 7f2260ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.192+0000 7f2260ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:03.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.440+0000 7f226a5b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:03.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.440+0000 7f226a5b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:03.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.440+0000 7f226a5b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:03.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.440+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2264092640 con 0x7f2264098520 2023-10-30T12:15:03.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.440+0000 7f2260ff9640 1 -- 172.21.15.50:0/3030871017 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f2258013020 con 0x7f2264098520 2023-10-30T12:15:03.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.440+0000 7f2260ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:03.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.440+0000 7f2260ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:03.442 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:03.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 10 librados: watch_flush enter 2023-10-30T12:15:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 10 librados: watch_flush exit 2023-10-30T12:15:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 20 client.115001.objecter shutdown clearing up homeless session... 2023-10-30T12:15:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 10 client.115001.objecter successfully canceled tick 2023-10-30T12:15:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f22500392a0 msgr2=0x7f225003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:03.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f22500392a0 0x7f225003b760 secure :-1 s=READY pgs=4190 cs=0 l=1 rev1=1 crypto rx=0x7f2254007fd0 tx=0x7f2254007aa0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 10 monclient: shutdown 2023-10-30T12:15:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264098520 msgr2=0x7f22641ca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264098520 0x7f22641ca240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f225800b530 tx=0x7f225800b9f0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 shutdown_connections 2023-10-30T12:15:03.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f22500392a0 0x7f225003b760 unknown :-1 s=CLOSED pgs=4190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2264137980 0x7f2264096730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2264098900 0x7f22640961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 --2- 172.21.15.50:0/3030871017 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2264098520 0x7f22641ca240 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 >> 172.21.15.50:0/3030871017 conn(0x7f22640a05c0 msgr2=0x7f22641bdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 shutdown_connections 2023-10-30T12:15:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 -- 172.21.15.50:0/3030871017 wait complete. 2023-10-30T12:15:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:03.442+0000 7f226a5b6640 1 librados: shutdown 2023-10-30T12:15:06.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:06.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.569+0000 7f951e4bb640 10 monclient: get_monmap_and_config 2023-10-30T12:15:06.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.569+0000 7f951e4bb640 10 monclient: build_initial_monmap 2023-10-30T12:15:06.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.569+0000 7f951e4bb640 10 monclient: monmap: 2023-10-30T12:15:06.571 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:06.570694+0000 2023-10-30T12:15:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:06.570694+0000 2023-10-30T12:15:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:06.573 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:06.574 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:06.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 1 Processor -- start 2023-10-30T12:15:06.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 1 -- start start 2023-10-30T12:15:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient: init 2023-10-30T12:15:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-10-30T12:15:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 0x7f9518070fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient(hunting): picked mon.noname-c con 0x7f9518130290 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:06.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9518130670 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9518130ab0 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9518070000 con 0x7f9518130670 2023-10-30T12:15:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95180a54e0 con 0x7f9518130ab0 2023-10-30T12:15:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95180714e0 con 0x7f9518130290 2023-10-30T12:15:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.570+0000 7f951e4bb640 10 monclient(hunting): authenticate will time out at 1751.753995s 2023-10-30T12:15:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.571+0000 7f951ca58640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f9518130ab0 legacy=0x7f95180a4dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:15:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.571+0000 7f9517fff640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 msgr2=0x7f9518070fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:06.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.571+0000 7f9517fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 0x7f9518070fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.571+0000 7f95177fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9518130670 0x7f95180a46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50466/0 (socket says 172.21.15.50:50466) 2023-10-30T12:15:06.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.571+0000 7f95177fe640 1 -- 172.21.15.50:0/723911373 learned_addr learned my addr 172.21.15.50:0/723911373 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:06.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.772+0000 7f951ca58640 1 -- 172.21.15.50:0/723911373 >> v1:172.21.15.118:6789/0 conn(0x7f9518130ab0 legacy=0x7f95180a4dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:15:06.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.772+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 msgr2=0x7f9518070fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:06.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:06.772+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 0x7f9518070fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:15:07.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:07.173+0000 7f951ca58640 1 -- 172.21.15.50:0/723911373 >> v1:172.21.15.118:6789/0 conn(0x7f9518130ab0 legacy=0x7f95180a4dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:15:07.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:07.173+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 msgr2=0x7f9518070fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:07.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:07.173+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 0x7f9518070fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:15:07.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:07.974+0000 7f951ca58640 1 -- 172.21.15.50:0/723911373 >> v1:172.21.15.118:6789/0 conn(0x7f9518130ab0 legacy=0x7f95180a4dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:15:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:07.974+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 msgr2=0x7f9518070fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:07.974+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 0x7f9518070fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:15:09.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): tick 2023-10-30T12:15:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:15:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 monclient(hunting): continuing hunt 2023-10-30T12:15:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:15:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 msgr2=0x7f9518070fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.118:3300/0 conn(0x7f9518130290 0x7f9518070fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 -- 172.21.15.50:0/723911373 >> v1:172.21.15.118:6789/0 conn(0x7f9518130ab0 legacy=0x7f95180a4dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 -- 172.21.15.50:0/723911373 >> v1:172.21.15.50:6789/0 conn(0x7f9518130670 legacy=0x7f95180a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient: _add_conns ranks=[2,3] 2023-10-30T12:15:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f95040036c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:09.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:09.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9504006240 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:09.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:09.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 -- 172.21.15.50:0/723911373 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9504008d90 con 0x7f9504006240 2023-10-30T12:15:09.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 1 -- 172.21.15.50:0/723911373 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9504008f00 con 0x7f95040036c0 2023-10-30T12:15:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9515ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f95040036c0 auth_method 0 2023-10-30T12:15:09.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:09.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:09.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:09.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 10 monclient(hunting): get_auth_request con 0x7f9504006240 auth_method 0 2023-10-30T12:15:09.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f9504005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:15:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f9504005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 msgr2=0x7f9504008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:15:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 msgr2=0x7f9504008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.571+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:09.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:09.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:09.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f95040036c0 auth_method 0 2023-10-30T12:15:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 10 monclient(hunting): get_auth_request con 0x7f9504006240 auth_method 0 2023-10-30T12:15:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f9504005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:15:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f9504005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 msgr2=0x7f9504008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:15:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 msgr2=0x7f9504008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:15:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:09.772+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:15:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f95040036c0 auth_method 0 2023-10-30T12:15:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 10 monclient(hunting): get_auth_request con 0x7f9504006240 auth_method 0 2023-10-30T12:15:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f9504005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:15:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f9504005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 msgr2=0x7f9504008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:15:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:10.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 msgr2=0x7f9504008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:15:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.173+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:15:10.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:10.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f95040036c0 auth_method 0 2023-10-30T12:15:10.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f95177fe640 10 monclient(hunting): get_auth_request con 0x7f9504006240 auth_method 0 2023-10-30T12:15:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f95177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f95177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f95177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient(hunting): handle_auth_done global_id 115004 payload 274 2023-10-30T12:15:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 msgr2=0x7f9504008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 1 monclient: found mon.noname-b 2023-10-30T12:15:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f9517fff640 1 -- 172.21.15.50:0/723911373 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951806f2d0 con 0x7f95040036c0 2023-10-30T12:15:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.974+0000 7f95177fe640 1 --2- 172.21.15.50:0/723911373 >> v2:172.21.15.50:3300/0 conn(0x7f9504006240 0x7f9504008700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:15:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9517fff640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f95080038b0 tx=0x7f950802be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c79157f8900af487 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 1 -- 172.21.15.50:0/723911373 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508003b70 con 0x7f95040036c0 2023-10-30T12:15:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: dump: 2023-10-30T12:15:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:10.985 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: _finish_auth 0 2023-10-30T12:15:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 1 -- 172.21.15.50:0/723911373 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9508003cd0 con 0x7f95040036c0 2023-10-30T12:15:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f951e4bb640 5 monclient: authenticate success, global_id 115004 2023-10-30T12:15:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f951e4bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f951e4bb640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f951e4bb640 4 set_mon_vals no callback set 2023-10-30T12:15:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 1 -- 172.21.15.50:0/723911373 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950802ca10 con 0x7f95040036c0 2023-10-30T12:15:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.975+0000 7f9516ffd640 10 monclient: dump: 2023-10-30T12:15:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:10.990 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:10.990 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:10.990 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 10 monclient: shutdown 2023-10-30T12:15:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 1 -- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f9504005b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 1 --2- 172.21.15.50:0/723911373 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f9504005b10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f95080038b0 tx=0x7f950802be30 comp rx=0 tx=0).stop 2023-10-30T12:15:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951ca58640 1 -- 172.21.15.50:0/723911373 reap_dead start 2023-10-30T12:15:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 1 -- 172.21.15.50:0/723911373 shutdown_connections 2023-10-30T12:15:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 1 -- 172.21.15.50:0/723911373 >> 172.21.15.50:0/723911373 conn(0x7f95180a22b0 msgr2=0x7f95180a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 1 -- 172.21.15.50:0/723911373 shutdown_connections 2023-10-30T12:15:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.976+0000 7f951e4bb640 1 -- 172.21.15.50:0/723911373 wait complete. 2023-10-30T12:15:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.977+0000 7f951e4bb640 10 monclient: build_initial_monmap 2023-10-30T12:15:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.977+0000 7f951e4bb640 10 monclient: monmap: 2023-10-30T12:15:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:11.002 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.977+0000 7f951e4bb640 1 librados: starting msgr at 2023-10-30T12:15:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.977+0000 7f951e4bb640 1 librados: starting objecter 2023-10-30T12:15:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.977+0000 7f951e4bb640 1 Processor -- start 2023-10-30T12:15:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 -- start start 2023-10-30T12:15:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 librados: setting wanted keys 2023-10-30T12:15:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 librados: calling monclient init 2023-10-30T12:15:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient: init 2023-10-30T12:15:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:15:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f95181c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): picked mon.noname-c con 0x7f95040036c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95181c52c0 0x7f951812e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): picked mon.noname-b con 0x7f95181c52c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f951812ec20 0x7f95181c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): picked mon.noname-a con 0x7f951812ec20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951809e1a0 con 0x7f951812ec20 2023-10-30T12:15:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951811ce60 con 0x7f95181c52c0 2023-10-30T12:15:11.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951809f570 con 0x7f95040036c0 2023-10-30T12:15:11.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:11.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951e4bb640 10 monclient(hunting): authenticate will time out at 1756.161758s 2023-10-30T12:15:11.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f9517fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f95181c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f9517fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f95181c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46494/0 (socket says 172.21.15.50:46494) 2023-10-30T12:15:11.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f9517fff640 1 -- 172.21.15.50:0/887865719 learned_addr learned my addr 172.21.15.50:0/887865719 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:11.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f9517fff640 10 monclient(hunting): get_auth_request con 0x7f95040036c0 auth_method 0 2023-10-30T12:15:11.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f95177fe640 1 -- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95181c52c0 msgr2=0x7f951812e6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:11.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f9517fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951ca58640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f951812ec20 0x7f95181c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f9517fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f95177fe640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95181c52c0 0x7f951812e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:11.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951ca58640 10 monclient(hunting): get_auth_request con 0x7f951812ec20 auth_method 0 2023-10-30T12:15:11.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951ca58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951ca58640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.978+0000 7f951ca58640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951ca58640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951ca58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951ca58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 10 monclient(hunting): handle_auth_done global_id 115007 payload 824 2023-10-30T12:15:11.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 1 -- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95181c52c0 msgr2=0x7f951812e6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:11.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95181c52c0 0x7f951812e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 1 -- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f951812ec20 msgr2=0x7f95181c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f951812ec20 0x7f95181c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:11.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 1 monclient: found mon.noname-c 2023-10-30T12:15:11.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:11.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:11.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 1 -- 172.21.15.50:0/887865719 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950800e040 con 0x7f95040036c0 2023-10-30T12:15:11.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951ca58640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f951812ec20 0x7f95181c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9517fff640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f95181c4860 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f950802c440 tx=0x7f9508005a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 client.?.objecter ms_handle_connect 0x7f95040036c0 2023-10-30T12:15:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 1 -- 172.21.15.50:0/887865719 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508003b70 con 0x7f95040036c0 2023-10-30T12:15:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: dump: 2023-10-30T12:15:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.017 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.017 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.017 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.017 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.018 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.018 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.018 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: _finish_auth 0 2023-10-30T12:15:11.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:15:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 1 -- 172.21.15.50:0/887865719 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95080100c0 con 0x7f95040036c0 2023-10-30T12:15:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 1 -- 172.21.15.50:0/887865719 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508003e40 con 0x7f95040036c0 2023-10-30T12:15:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 5 monclient: authenticate success, global_id 115007 2023-10-30T12:15:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f9514ff9640 10 monclient: dump: 2023-10-30T12:15:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.022 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f95167fc640 4 set_mon_vals no callback set 2023-10-30T12:15:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 10 monclient: _renew_subs 2023-10-30T12:15:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951811c810 con 0x7f95040036c0 2023-10-30T12:15:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 10 client.115007.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 10 monclient: _renew_subs 2023-10-30T12:15:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951809f7b0 con 0x7f95040036c0 2023-10-30T12:15:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.979+0000 7f951e4bb640 1 librados: init done 2023-10-30T12:15:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.980+0000 7f9514ff9640 1 -- 172.21.15.50:0/887865719 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f950803e450 con 0x7f95040036c0 2023-10-30T12:15:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f951e4bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:11.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f951e4bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:11.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f951e4bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:11.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951804e160 con 0x7f95040036c0 2023-10-30T12:15:11.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f9514ff9640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f950403a190 0x7f950403c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f9514ff9640 1 -- 172.21.15.50:0/887865719 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f950806cf00 con 0x7f95040036c0 2023-10-30T12:15:11.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f9514ff9640 10 client.115007.objecter ms_dispatch 0x7f9518000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:15:11.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f9514ff9640 3 client.115007.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:15:11.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f9514ff9640 3 client.115007.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:15:11.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f9514ff9640 20 client.115007.objecter dump_active .. 0 homeless 2023-10-30T12:15:11.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f95177fe640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f950403a190 0x7f950403c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.981+0000 7f95177fe640 10 monclient: get_auth_request con 0x7f950403a190 auth_method 0 2023-10-30T12:15:11.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.982+0000 7f95177fe640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f950403a190 0x7f950403c650 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 crypto rx=0x7f950c000c00 tx=0x7f950c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.982+0000 7f9514ff9640 10 client.115007.objecter ms_handle_connect 0x7f950403a190 2023-10-30T12:15:11.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.986+0000 7f9514ff9640 1 -- 172.21.15.50:0/887865719 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9508033760 con 0x7f95040036c0 2023-10-30T12:15:11.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.986+0000 7f9514ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:11.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:10.986+0000 7f9514ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.231+0000 7f951e4bb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.231+0000 7f951e4bb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.231+0000 7f951e4bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.231+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f95180a13a0 con 0x7f95040036c0 2023-10-30T12:15:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.232+0000 7f9514ff9640 1 -- 172.21.15.50:0/887865719 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f9508042070 con 0x7f95040036c0 2023-10-30T12:15:11.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.232+0000 7f9514ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.232+0000 7f9514ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:11.235 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:11.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 10 librados: watch_flush enter 2023-10-30T12:15:11.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 10 librados: watch_flush exit 2023-10-30T12:15:11.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 20 client.115007.objecter shutdown clearing up homeless session... 2023-10-30T12:15:11.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 10 client.115007.objecter successfully canceled tick 2023-10-30T12:15:11.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f950403a190 msgr2=0x7f950403c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f950403a190 0x7f950403c650 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 crypto rx=0x7f950c000c00 tx=0x7f950c009040 comp rx=0 tx=0).stop 2023-10-30T12:15:11.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 10 monclient: shutdown 2023-10-30T12:15:11.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:11.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 msgr2=0x7f95181c4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.233+0000 7f951e4bb640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f95181c4860 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f950802c440 tx=0x7f9508005a30 comp rx=0 tx=0).stop 2023-10-30T12:15:11.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 shutdown_connections 2023-10-30T12:15:11.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f950403a190 0x7f950403c650 unknown :-1 s=CLOSED pgs=4191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f951812ec20 0x7f95181c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95181c52c0 0x7f951812e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 --2- 172.21.15.50:0/887865719 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95040036c0 0x7f95181c4860 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 >> 172.21.15.50:0/887865719 conn(0x7f95180a22b0 msgr2=0x7f95181b9160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:11.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 shutdown_connections 2023-10-30T12:15:11.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 -- 172.21.15.50:0/887865719 wait complete. 2023-10-30T12:15:11.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.234+0000 7f951e4bb640 1 librados: shutdown 2023-10-30T12:15:11.245 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:15:11.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:15:11.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:15:11.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.350+0000 7f4a91cc2640 10 monclient: get_monmap_and_config 2023-10-30T12:15:11.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.350+0000 7f4a91cc2640 10 monclient: build_initial_monmap 2023-10-30T12:15:11.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.350+0000 7f4a91cc2640 10 monclient: monmap: 2023-10-30T12:15:11.352 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:11.353 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.353 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:11.351823+0000 2023-10-30T12:15:11.353 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:11.351823+0000 2023-10-30T12:15:11.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:11.353 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.354 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:11.354 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:11.354 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:11.354 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:11.354 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:11.355 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 Processor -- start 2023-10-30T12:15:11.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 -- start start 2023-10-30T12:15:11.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient: init 2023-10-30T12:15:11.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:11.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-10-30T12:15:11.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4a8c0a2ae0 0x7f4a8c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c0a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:11.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4a8c0a2ec0 0x7f4a8c1356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c0a2ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:11.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c136070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c135c20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:11.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a8c1365b0 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a8c0a2990 con 0x7f4a8c0a2ae0 2023-10-30T12:15:11.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c0a1630 con 0x7f4a8c135c20 2023-10-30T12:15:11.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:11.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.351+0000 7f4a91cc2640 10 monclient(hunting): authenticate will time out at 1756.534973s 2023-10-30T12:15:11.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4a8c0a2ec0 0x7f4a8c1356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8b7fe640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f4a8c0a2ae0 msgr2=0x7f4a8c0a2450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:11.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8b7fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4a8c0a2ae0 0x7f4a8c0a2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:11.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4a8c0a2ec0 0x7f4a8c1356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46252/0 (socket says 172.21.15.50:46252) 2023-10-30T12:15:11.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c136070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46502/0 (socket says 172.21.15.50:46502) 2023-10-30T12:15:11.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 1 -- 172.21.15.50:0/1922620909 learned_addr learned my addr 172.21.15.50:0/1922620909 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:11.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 10 monclient(hunting): get_auth_request con 0x7f4a8c0a2ec0 auth_method 0 2023-10-30T12:15:11.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request con 0x7f4a8c135c20 auth_method 0 2023-10-30T12:15:11.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient(hunting): handle_auth_done global_id 115010 payload 274 2023-10-30T12:15:11.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 -- 172.21.15.50:0/1922620909 >> v2:172.21.15.118:3300/0 conn(0x7f4a8c0a2ae0 msgr2=0x7f4a8c0a2450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:11.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 --2- 172.21.15.50:0/1922620909 >> v2:172.21.15.118:3300/0 conn(0x7f4a8c0a2ae0 0x7f4a8c0a2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 -- 172.21.15.50:0/1922620909 >> v2:172.21.15.50:3300/0 conn(0x7f4a8c0a2ec0 msgr2=0x7f4a8c1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:11.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 --2- 172.21.15.50:0/1922620909 >> v2:172.21.15.50:3300/0 conn(0x7f4a8c0a2ec0 0x7f4a8c1356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 monclient: found mon.noname-b 2023-10-30T12:15:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.352+0000 7f4a8bfff640 1 -- 172.21.15.50:0/1922620909 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c0a4190 con 0x7f4a8c135c20 2023-10-30T12:15:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8bfff640 1 --2- 172.21.15.50:0/1922620909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c136070 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0a3c80 tx=0x7f4a8402bd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b2cd7714d35b8d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 1 -- 172.21.15.50:0/1922620909 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8402cad0 con 0x7f4a8c135c20 2023-10-30T12:15:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: dump: 2023-10-30T12:15:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.373 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.373 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.373 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.374 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.374 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.374 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.374 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:15:11.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:15:11.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:11.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 1 -- 172.21.15.50:0/1922620909 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8402cc30 con 0x7f4a8c135c20 2023-10-30T12:15:11.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 5 monclient: authenticate success, global_id 115010 2023-10-30T12:15:11.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:11.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:11.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:11.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 1 -- 172.21.15.50:0/1922620909 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84034440 con 0x7f4a8c135c20 2023-10-30T12:15:11.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 4 set_mon_vals no callback set 2023-10-30T12:15:11.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:11.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a8a7fc640 10 monclient: dump: 2023-10-30T12:15:11.377 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.377 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.377 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.377 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.378 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.378 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.378 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.378 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.378 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.379 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 10 monclient: shutdown 2023-10-30T12:15:11.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:11.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 1 -- 172.21.15.50:0/1922620909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 msgr2=0x7f4a8c136070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/1922620909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c136070 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0a3c80 tx=0x7f4a8402bd20 comp rx=0 tx=0).stop 2023-10-30T12:15:11.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 1 -- 172.21.15.50:0/1922620909 shutdown_connections 2023-10-30T12:15:11.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/1922620909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c136070 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/1922620909 >> v2:172.21.15.50:3300/0 conn(0x7f4a8c0a2ec0 0x7f4a8c1356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/1922620909 >> v2:172.21.15.118:3300/0 conn(0x7f4a8c0a2ae0 0x7f4a8c0a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.353+0000 7f4a91cc2640 1 -- 172.21.15.50:0/1922620909 >> 172.21.15.50:0/1922620909 conn(0x7f4a8c0a80b0 msgr2=0x7f4a8c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:11.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.354+0000 7f4a91cc2640 1 -- 172.21.15.50:0/1922620909 shutdown_connections 2023-10-30T12:15:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.354+0000 7f4a91cc2640 1 -- 172.21.15.50:0/1922620909 wait complete. 2023-10-30T12:15:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.354+0000 7f4a91cc2640 10 monclient: build_initial_monmap 2023-10-30T12:15:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.354+0000 7f4a91cc2640 10 monclient: monmap: 2023-10-30T12:15:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:11.383 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:11.383 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:11.383 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.354+0000 7f4a91cc2640 1 librados: starting msgr at 2023-10-30T12:15:11.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.354+0000 7f4a91cc2640 1 librados: starting objecter 2023-10-30T12:15:11.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.354+0000 7f4a91cc2640 1 Processor -- start 2023-10-30T12:15:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 -- start start 2023-10-30T12:15:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 librados: setting wanted keys 2023-10-30T12:15:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 librados: calling monclient init 2023-10-30T12:15:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient: init 2023-10-30T12:15:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:15:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0a2ae0 0x7f4a8c1ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c0a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c0a2ec0 0x7f4a8c130640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c0a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c135c20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c09f260 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c11f080 con 0x7f4a8c0a2ae0 2023-10-30T12:15:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c0a1820 con 0x7f4a8c135c20 2023-10-30T12:15:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a91cc2640 10 monclient(hunting): authenticate will time out at 1756.538581s 2023-10-30T12:15:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8b7fe640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0a2ae0 msgr2=0x7f4a8c1ca8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:11.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c0a2ec0 0x7f4a8c130640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0a2ae0 0x7f4a8c1ca8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:11.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c0a2ec0 0x7f4a8c130640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46254/0 (socket says 172.21.15.50:46254) 2023-10-30T12:15:11.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 -- 172.21.15.50:0/2441767712 learned_addr learned my addr 172.21.15.50:0/2441767712 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:11.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): get_auth_request con 0x7f4a8c0a2ec0 auth_method 0 2023-10-30T12:15:11.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request con 0x7f4a8c135c20 auth_method 0 2023-10-30T12:15:11.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient(hunting): handle_auth_done global_id 194190 payload 824 2023-10-30T12:15:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 msgr2=0x7f4a8c130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c130b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0a2ae0 msgr2=0x7f4a8c1ca8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:11.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4a8c0a2ae0 0x7f4a8c1ca8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient: _finish_hunting 0 2023-10-30T12:15:11.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 monclient: found mon.noname-a 2023-10-30T12:15:11.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:11.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:11.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8affd640 1 -- 172.21.15.50:0/2441767712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84009a70 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.355+0000 7f4a8bfff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4a8c135c20 0x7f4a8c130b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:11.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a8affd640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c0a2ec0 0x7f4a8c130640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009fd0 tx=0x7f4a8000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 client.?.objecter ms_handle_connect 0x7f4a8c0a2ec0 2023-10-30T12:15:11.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:11.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000c940 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:11.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: dump: 2023-10-30T12:15:11.394 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.395 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.395 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.395 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.395 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.395 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.396 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.396 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.396 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.396 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: _finish_auth 0 2023-10-30T12:15:11.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: _check_auth_tickets 2023-10-30T12:15:11.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:11.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 5 monclient: authenticate success, global_id 194190 2023-10-30T12:15:11.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8000caa0 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:11.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 10 monclient: _renew_subs 2023-10-30T12:15:11.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000ce30 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:11.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8c11ed30 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:11.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a73fff640 10 monclient: dump: 2023-10-30T12:15:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.402 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a89ffb640 4 set_mon_vals no callback set 2023-10-30T12:15:11.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 10 client.194190.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:11.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 10 monclient: _renew_subs 2023-10-30T12:15:11.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8c09f950 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.356+0000 7f4a91cc2640 1 librados: init done 2023-10-30T12:15:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4a80013470 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a91cc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c68f0 0x7f4a8c1c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c073ff0 con 0x7f4a8c1c68f0 2023-10-30T12:15:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a780392a0 0x7f4a7803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c68f0 0x7f4a8c1c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient: get_auth_request con 0x7f4a8c1c68f0 auth_method 0 2023-10-30T12:15:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f4a8004ada0 con 0x7f4a8c0a2ec0 2023-10-30T12:15:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 10 client.194190.objecter ms_dispatch 0x7f4a8c000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:15:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 3 client.194190.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:15:11.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 3 client.194190.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:15:11.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 20 client.194190.objecter dump_active .. 0 homeless 2023-10-30T12:15:11.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_done global_id 194193 payload 824 2023-10-30T12:15:11.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 1 -- 172.21.15.50:0/2441767712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a8c099b40 con 0x7f4a8c1c68f0 2023-10-30T12:15:11.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8bfff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a780392a0 0x7f4a7803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8bfff640 10 monclient: get_auth_request con 0x7f4a780392a0 auth_method 0 2023-10-30T12:15:11.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 10 client.194190.objecter ms_handle_connect 0x7f4a8c1c68f0 2023-10-30T12:15:11.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 10 client.194190.objecter resend_mon_ops 2023-10-30T12:15:11.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a8b7fe640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c68f0 0x7f4a8c1c6cd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c1caf70 tx=0x7f4a7c00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c003e10 con 0x7f4a8c1c68f0 2023-10-30T12:15:11.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:11.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.357+0000 7f4a73fff640 10 monclient: dump: 2023-10-30T12:15:11.409 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 10 client.194190.objecter ms_handle_connect 0x7f4a780392a0 2023-10-30T12:15:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a8bfff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a780392a0 0x7f4a7803b760 secure :-1 s=READY pgs=4192 cs=0 l=1 rev1=1 crypto rx=0x7f4a84036000 tx=0x7f4a84030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f4a7c005070 con 0x7f4a8c1c68f0 2023-10-30T12:15:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c68f0 msgr2=0x7f4a8c1c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c68f0 0x7f4a8c1c6cd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c1caf70 tx=0x7f4a7c00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c68f0 msgr2=0x7f4a8c1c6cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:11.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.358+0000 7f4a73fff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c68f0 0x7f4a8c1c6cd0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c1caf70 tx=0x7f4a7c00b8c0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a91cc2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:11.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c7210 0x7f4a8c09ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a91cc2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c09a8a0 con 0x7f4a8c1c7210 2023-10-30T12:15:11.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c7210 0x7f4a8c09ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient: get_auth_request con 0x7f4a8c1c7210 auth_method 0 2023-10-30T12:15:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_done global_id 194196 payload 824 2023-10-30T12:15:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.364+0000 7f4a8b7fe640 1 -- 172.21.15.50:0/2441767712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4a8c09a5c0 con 0x7f4a8c1c7210 2023-10-30T12:15:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a8b7fe640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c7210 0x7f4a8c09ad50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c012040 tx=0x7f4a7c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 10 client.194190.objecter ms_handle_connect 0x7f4a8c1c7210 2023-10-30T12:15:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 10 client.194190.objecter resend_mon_ops 2023-10-30T12:15:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c020070 con 0x7f4a8c1c7210 2023-10-30T12:15:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 10 monclient: dump: 2023-10-30T12:15:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.419 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4793 (secure 0 0 0) 0x7f4a7c010400 con 0x7f4a8c1c7210 2023-10-30T12:15:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c7210 msgr2=0x7f4a8c09ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c7210 0x7f4a8c09ad50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c012040 tx=0x7f4a7c005a20 comp rx=0 tx=0).stop 2023-10-30T12:15:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c7210 msgr2=0x7f4a8c09ad50 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a73fff640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c1c7210 0x7f4a8c09ad50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.421 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:15:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 154, 2023-10-30T12:15:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:15:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:15:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:15:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:15:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:11.453 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 10 librados: watch_flush enter 2023-10-30T12:15:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 10 librados: watch_flush exit 2023-10-30T12:15:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 20 client.194190.objecter shutdown clearing up homeless session... 2023-10-30T12:15:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 10 client.194190.objecter successfully canceled tick 2023-10-30T12:15:11.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a780392a0 msgr2=0x7f4a7803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a780392a0 0x7f4a7803b760 secure :-1 s=READY pgs=4192 cs=0 l=1 rev1=1 crypto rx=0x7f4a84036000 tx=0x7f4a84030000 comp rx=0 tx=0).stop 2023-10-30T12:15:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 10 monclient: shutdown 2023-10-30T12:15:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c0a2ec0 msgr2=0x7f4a8c130640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 --2- 172.21.15.50:0/2441767712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a8c0a2ec0 0x7f4a8c130640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009fd0 tx=0x7f4a8000ed40 comp rx=0 tx=0).stop 2023-10-30T12:15:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a8bfff640 1 -- 172.21.15.50:0/2441767712 reap_dead start 2023-10-30T12:15:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a8bfff640 1 -- 172.21.15.50:0/2441767712 reap_dead start 2023-10-30T12:15:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 shutdown_connections 2023-10-30T12:15:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 >> 172.21.15.50:0/2441767712 conn(0x7f4a8c0a80b0 msgr2=0x7f4a8c132940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 shutdown_connections 2023-10-30T12:15:11.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 -- 172.21.15.50:0/2441767712 wait complete. 2023-10-30T12:15:11.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.365+0000 7f4a91cc2640 1 librados: shutdown 2023-10-30T12:15:11.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:15:11.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:15:11.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:15:11.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.563+0000 7f1475209640 10 monclient: get_monmap_and_config 2023-10-30T12:15:11.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.563+0000 7f1475209640 10 monclient: build_initial_monmap 2023-10-30T12:15:11.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.563+0000 7f1475209640 10 monclient: monmap: 2023-10-30T12:15:11.565 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:11.565 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.566 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:11.564677+0000 2023-10-30T12:15:11.566 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:11.564677+0000 2023-10-30T12:15:11.566 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:11.566 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.566 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:11.567 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:11.567 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:11.567 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:11.567 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:11.567 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 1 Processor -- start 2023-10-30T12:15:11.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 1 -- start start 2023-10-30T12:15:11.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient: init 2023-10-30T12:15:11.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:11.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-10-30T12:15:11.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 0x7f147009a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient(hunting): picked mon.noname-c con 0x7f1470071c00 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:11.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1470097fe0 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:11.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f147009b170 0x7f1470139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient(hunting): picked mon.noname-a con 0x7f147009b170 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:11.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1470097600 con 0x7f1470097fe0 2023-10-30T12:15:11.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14700978e0 con 0x7f147009b170 2023-10-30T12:15:11.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147009bf50 con 0x7f1470071c00 2023-10-30T12:15:11.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:11.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1475209640 10 monclient(hunting): authenticate will time out at 1756.747792s 2023-10-30T12:15:11.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f1467fff640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f1470097fe0 legacy=0x7f147009aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:15:11.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146ed9d640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 msgr2=0x7f147009a150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:11.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146f59e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f147009b170 0x7f1470139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146ed9d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 0x7f147009a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:11.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146f59e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f147009b170 0x7f1470139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46286/0 (socket says 172.21.15.50:46286) 2023-10-30T12:15:11.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146f59e640 1 -- 172.21.15.50:0/1296056438 learned_addr learned my addr 172.21.15.50:0/1296056438 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:11.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146f59e640 10 monclient(hunting): get_auth_request con 0x7f147009b170 auth_method 0 2023-10-30T12:15:11.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.564+0000 7f146f59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.565+0000 7f146f59e640 1 -- 172.21.15.50:0/1296056438 >> v2:172.21.15.50:3300/0 conn(0x7f147009b170 msgr2=0x7f1470139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2023-10-30T12:15:11.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.565+0000 7f146f59e640 1 --2- 172.21.15.50:0/1296056438 >> v2:172.21.15.50:3300/0 conn(0x7f147009b170 0x7f1470139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2023-10-30T12:15:11.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.565+0000 7f146f59e640 1 --2- 172.21.15.50:0/1296056438 >> v2:172.21.15.50:3300/0 conn(0x7f147009b170 0x7f1470139760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:11.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.765+0000 7f1467fff640 1 -- 172.21.15.50:0/1296056438 >> v1:172.21.15.118:6789/0 conn(0x7f1470097fe0 legacy=0x7f147009aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:15:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.765+0000 7f146ed9d640 1 -- 172.21.15.50:0/1296056438 >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 msgr2=0x7f147009a150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.765+0000 7f146ed9d640 1 --2- 172.21.15.50:0/1296056438 >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 0x7f147009a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:15:11.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 1 --2- 172.21.15.50:0/1296056438 >> v2:172.21.15.50:3300/0 conn(0x7f147009b170 0x7f1470139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): get_auth_request con 0x7f147009b170 auth_method 0 2023-10-30T12:15:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient(hunting): handle_auth_done global_id 194199 payload 274 2023-10-30T12:15:11.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 1 -- 172.21.15.50:0/1296056438 >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 msgr2=0x7f147009a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:11.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 1 --2- 172.21.15.50:0/1296056438 >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 0x7f147009a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 1 -- 172.21.15.50:0/1296056438 >> v1:172.21.15.118:6789/0 conn(0x7f1470097fe0 legacy=0x7f147009aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient: _finish_hunting 0 2023-10-30T12:15:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 1 monclient: found mon.noname-a 2023-10-30T12:15:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.766+0000 7f146f59e640 1 -- 172.21.15.50:0/1296056438 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147009d9b0 con 0x7f147009b170 2023-10-30T12:15:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146f59e640 1 --2- 172.21.15.50:0/1296056438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f1470139760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f147009fdf0 tx=0x7f146802bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c9afb20d3794a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 1 -- 172.21.15.50:0/1296056438 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146802cd50 con 0x7f147009b170 2023-10-30T12:15:11.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:11.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: dump: 2023-10-30T12:15:11.773 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.774 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.774 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.774 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.774 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.774 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.775 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.775 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.775 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.775 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: _finish_auth 0 2023-10-30T12:15:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: _check_auth_tickets 2023-10-30T12:15:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 1 -- 172.21.15.50:0/1296056438 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468030070 con 0x7f147009b170 2023-10-30T12:15:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 1 -- 172.21.15.50:0/1296056438 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468036600 con 0x7f147009b170 2023-10-30T12:15:11.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 5 monclient: authenticate success, global_id 194199 2023-10-30T12:15:11.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:11.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 4 set_mon_vals no callback set 2023-10-30T12:15:11.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:11.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f146e59c640 10 monclient: dump: 2023-10-30T12:15:11.778 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.778 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.778 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.778 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.779 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.779 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.779 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.779 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.779 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.779 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 10 monclient: shutdown 2023-10-30T12:15:11.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:11.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 -- 172.21.15.50:0/1296056438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 msgr2=0x7f1470139760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 --2- 172.21.15.50:0/1296056438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f1470139760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f147009fdf0 tx=0x7f146802bfa0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 -- 172.21.15.50:0/1296056438 shutdown_connections 2023-10-30T12:15:11.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 --2- 172.21.15.50:0/1296056438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f1470139760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 --2- 172.21.15.50:0/1296056438 >> v2:172.21.15.118:3300/0 conn(0x7f1470071c00 0x7f147009a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 -- 172.21.15.50:0/1296056438 >> 172.21.15.50:0/1296056438 conn(0x7f14700a3980 msgr2=0x7f14700a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:11.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 -- 172.21.15.50:0/1296056438 shutdown_connections 2023-10-30T12:15:11.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.767+0000 7f1475209640 1 -- 172.21.15.50:0/1296056438 wait complete. 2023-10-30T12:15:11.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.768+0000 7f1475209640 10 monclient: build_initial_monmap 2023-10-30T12:15:11.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.768+0000 7f1475209640 10 monclient: monmap: 2023-10-30T12:15:11.789 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:11.789 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:11.789 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:11.790 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:11.790 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:11.790 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.790 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:11.790 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:11.791 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:11.791 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.768+0000 7f1475209640 1 librados: starting msgr at 2023-10-30T12:15:11.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.768+0000 7f1475209640 1 librados: starting objecter 2023-10-30T12:15:11.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.768+0000 7f1475209640 1 Processor -- start 2023-10-30T12:15:11.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 -- start start 2023-10-30T12:15:11.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 librados: setting wanted keys 2023-10-30T12:15:11.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 librados: calling monclient init 2023-10-30T12:15:11.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient: init 2023-10-30T12:15:11.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:11.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:15:11.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1470071c00 0x7f14701c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): picked mon.noname-c con 0x7f1470071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:11.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1470097fe0 0x7f14701b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): picked mon.noname-b con 0x7f1470097fe0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:11.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f14701c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): picked mon.noname-a con 0x7f147009b170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:11.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701ae520 con 0x7f147009b170 2023-10-30T12:15:11.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14700700f0 con 0x7f1470097fe0 2023-10-30T12:15:11.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14701adb80 con 0x7f1470071c00 2023-10-30T12:15:11.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:11.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1475209640 10 monclient(hunting): authenticate will time out at 1756.952492s 2023-10-30T12:15:11.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1470071c00 0x7f14701c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f14701c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f14701c2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46316/0 (socket says 172.21.15.50:46316) 2023-10-30T12:15:11.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1470071c00 0x7f14701c6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46526/0 (socket says 172.21.15.50:46526) 2023-10-30T12:15:11.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 -- 172.21.15.50:0/3950215858 learned_addr learned my addr 172.21.15.50:0/3950215858 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:11.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): get_auth_request con 0x7f147009b170 auth_method 0 2023-10-30T12:15:11.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1467fff640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1470097fe0 msgr2=0x7f14701b59b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:15:11.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 10 monclient(hunting): get_auth_request con 0x7f1470071c00 auth_method 0 2023-10-30T12:15:11.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f1467fff640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1470097fe0 0x7f14701b59b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:11.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient(hunting): handle_auth_done global_id 194202 payload 824 2023-10-30T12:15:11.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1470071c00 msgr2=0x7f14701c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1470071c00 0x7f14701c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1470097fe0 msgr2=0x7f14701b59b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:11.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1470097fe0 0x7f14701b59b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient: _finish_hunting 0 2023-10-30T12:15:11.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 monclient: found mon.noname-a 2023-10-30T12:15:11.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:11.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:11.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.769+0000 7f146f59e640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680097e0 con 0x7f147009b170 2023-10-30T12:15:11.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f146f59e640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f14701c2d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f146802c5b0 tx=0x7f1468004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 client.?.objecter ms_handle_connect 0x7f147009b170 2023-10-30T12:15:11.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:11.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468044070 con 0x7f147009b170 2023-10-30T12:15:11.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:11.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: dump: 2023-10-30T12:15:11.802 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.803 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.803 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.803 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.803 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.803 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.804 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.804 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.804 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.804 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: _finish_auth 0 2023-10-30T12:15:11.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: _check_auth_tickets 2023-10-30T12:15:11.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:11.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468036d90 con 0x7f147009b170 2023-10-30T12:15:11.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 5 monclient: authenticate success, global_id 194202 2023-10-30T12:15:11.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 10 monclient: _renew_subs 2023-10-30T12:15:11.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:11.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701ade60 con 0x7f147009b170 2023-10-30T12:15:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468003bd0 con 0x7f147009b170 2023-10-30T12:15:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f14677fe640 10 monclient: dump: 2023-10-30T12:15:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.807 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.807 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.807 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.807 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.807 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.808 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.808 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.808 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.808 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f146dd9b640 4 set_mon_vals no callback set 2023-10-30T12:15:11.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 10 client.194202.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:11.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 10 monclient: _renew_subs 2023-10-30T12:15:11.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:11.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1470071870 con 0x7f147009b170 2023-10-30T12:15:11.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.770+0000 7f1475209640 1 librados: init done 2023-10-30T12:15:11.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1468003d30 con 0x7f147009b170 2023-10-30T12:15:11.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f1475209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:11.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f1475209640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701c6c20 0x7f14701c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f1475209640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1470073ff0 con 0x7f14701c6c20 2023-10-30T12:15:11.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701c6c20 0x7f14701c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14500392a0 0x7f145003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient: get_auth_request con 0x7f14701c6c20 auth_method 0 2023-10-30T12:15:11.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(436..436 src has 1..436) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f146806cba0 con 0x7f147009b170 2023-10-30T12:15:11.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 10 client.194202.objecter ms_dispatch 0x7f1470000b90 osd_map(436..436 src has 1..436) v4 2023-10-30T12:15:11.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 3 client.194202.objecter handle_osd_map got epochs [436,436] > 0 2023-10-30T12:15:11.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 3 client.194202.objecter handle_osd_map decoding full epoch 436 2023-10-30T12:15:11.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 20 client.194202.objecter dump_active .. 0 homeless 2023-10-30T12:15:11.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 10 monclient(hunting): handle_auth_done global_id 115019 payload 824 2023-10-30T12:15:11.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1470071fe0 con 0x7f14701c6c20 2023-10-30T12:15:11.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f1467fff640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14500392a0 0x7f145003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f1467fff640 10 monclient: get_auth_request con 0x7f14500392a0 auth_method 0 2023-10-30T12:15:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 10 client.194202.objecter ms_handle_connect 0x7f14701c6c20 2023-10-30T12:15:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 10 client.194202.objecter resend_mon_ops 2023-10-30T12:15:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701c6c20 0x7f14701c3270 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1460003980 tx=0x7f1460009a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460012600 con 0x7f14701c6c20 2023-10-30T12:15:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:11.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.771+0000 7f14677fe640 10 monclient: dump: 2023-10-30T12:15:11.815 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.815 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.815 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.816 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.816 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.816 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.816 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.816 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.817 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.817 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 10 client.194202.objecter ms_handle_connect 0x7f14500392a0 2023-10-30T12:15:11.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f1467fff640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14500392a0 0x7f145003b760 secure :-1 s=READY pgs=4193 cs=0 l=1 rev1=1 crypto rx=0x7f14701ad890 tx=0x7f1458006d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1460012760 con 0x7f14701c6c20 2023-10-30T12:15:11.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:11.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:11.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701c6c20 msgr2=0x7f14701c3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701c6c20 0x7f14701c3270 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1460003980 tx=0x7f1460009a30 comp rx=0 tx=0).stop 2023-10-30T12:15:11.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701c6c20 msgr2=0x7f14701c3270 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:11.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.772+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701c6c20 0x7f14701c3270 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f1475209640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:11.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f1475209640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 0x7f14701c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f1475209640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14701ae0a0 con 0x7f14701b5ef0 2023-10-30T12:15:11.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 0x7f14701c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient: get_auth_request con 0x7f14701b5ef0 auth_method 0 2023-10-30T12:15:11.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 10 monclient(hunting): handle_auth_done global_id 115022 payload 824 2023-10-30T12:15:11.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.778+0000 7f146ed9d640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f14701ae250 con 0x7f14701b5ef0 2023-10-30T12:15:11.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 0x7f14701c37b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1460006a30 tx=0x7f1460003910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 10 client.194202.objecter ms_handle_connect 0x7f14701b5ef0 2023-10-30T12:15:11.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 10 client.194202.objecter resend_mon_ops 2023-10-30T12:15:11.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460023070 con 0x7f14701b5ef0 2023-10-30T12:15:11.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:11.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 10 monclient: dump: 2023-10-30T12:15:11.823 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.823 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.823 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.823 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.823 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.824 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.824 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.824 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.824 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.824 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 msgr2=0x7f14701c37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:15:11.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 msgr2=0x7f14701c37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:15:11.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 0x7f14701c37b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1460006a30 tx=0x7f1460003910 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:11.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 0x7f14701c37b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1460006a30 tx=0x7f1460003910 comp rx=0 tx=0).stop 2023-10-30T12:15:11.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 msgr2=0x7f14701c37b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:11.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 0x7f14701c37b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1450041dc0 0x7f14500441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:11.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 msgr2=0x7f14701c37b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:11.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14701b5ef0 0x7f14701c37b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:11.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1450044790 con 0x7f1450041dc0 2023-10-30T12:15:11.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1450041dc0 0x7f14500441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:11.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient: get_auth_request con 0x7f1450041dc0 auth_method 0 2023-10-30T12:15:11.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:11.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:11.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:11.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:11.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:11.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:11.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 10 monclient(hunting): handle_auth_done global_id 115025 payload 824 2023-10-30T12:15:11.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.779+0000 7f146ed9d640 1 -- 172.21.15.50:0/3950215858 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1460012760 con 0x7f1450041dc0 2023-10-30T12:15:11.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f146ed9d640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1450041dc0 0x7f14500441b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1470069d90 tx=0x7f1460009a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:11.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 10 client.194202.objecter ms_handle_connect 0x7f1450041dc0 2023-10-30T12:15:11.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 10 client.194202.objecter resend_mon_ops 2023-10-30T12:15:11.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146002d030 con 0x7f1450041dc0 2023-10-30T12:15:11.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:11.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:11.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 10 monclient: dump: 2023-10-30T12:15:11.830 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:11.830 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:11.830 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.831 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:11.831 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:11.831 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:11.831 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:11.831 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:11.831 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:11.832 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:11.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4621 (secure 0 0 0) 0x7f146001c670 con 0x7f1450041dc0 2023-10-30T12:15:11.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:11.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:11.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1450041dc0 msgr2=0x7f14500441b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1450041dc0 0x7f14500441b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1470069d90 tx=0x7f1460009a30 comp rx=0 tx=0).stop 2023-10-30T12:15:11.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1450041dc0 msgr2=0x7f14500441b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:11.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f14677fe640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1450041dc0 0x7f14500441b0 secure :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1470069d90 tx=0x7f1460009a30 comp rx=0 tx=0).stop 2023-10-30T12:15:11.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f146f59e640 1 -- 172.21.15.50:0/3950215858 reap_dead start 2023-10-30T12:15:11.833 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:11.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:11.834 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:11.834 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:15:11.834 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 154, 2023-10-30T12:15:11.835 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:11.835 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:11.835 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:11.835 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.835 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:15:11.836 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:11.836 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:11.836 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:11.836 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:11.836 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:11.836 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:11.837 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:11.837 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:11.837 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:11.837 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:11.837 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:11.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:11.838 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:11.838 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:11.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:11.838 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:11.839 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:11.839 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:11.839 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:11.839 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:11.839 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:11.840 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:11.840 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:11.840 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:11.840 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:11.840 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.841 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:11.841 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:11.841 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:11.841 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:11.841 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:11.841 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:11.842 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:11.842 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:11.842 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:11.842 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:11.842 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:11.843 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:11.843 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:11.843 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:11.843 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:11.843 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:11.843 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:11.844 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:11.844 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:11.844 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:11.844 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:11.844 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:11.845 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:11.845 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:11.845 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:11.845 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:11.845 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:11.846 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.846 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:11.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.846 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:11.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.851 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:11.851 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:11.851 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:11.851 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:11.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.851 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:11.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:11.852 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.852 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:11.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:11.853 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.853 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.853 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:11.853 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:11.854 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:11.854 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:11.854 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:11.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.855 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:11.855 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:11.855 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:11.855 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:11.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.855 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:11.856 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:11.856 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:11.856 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:11.857 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:11.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.857 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:11.857 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:11.858 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:11.858 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:11.858 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:11.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.858 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.859 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:11.859 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:11.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.859 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:11.860 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:11.860 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:11.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.860 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.860 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:11.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.861 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:11.861 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:11.861 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:11.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.861 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:11.862 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:11.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:11.862 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:11.862 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:11.862 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:15:11.862 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:11.863 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:11.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:11.863 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:11.863 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 10 librados: watch_flush enter 2023-10-30T12:15:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 10 librados: watch_flush exit 2023-10-30T12:15:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 20 client.194202.objecter shutdown clearing up homeless session... 2023-10-30T12:15:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 10 client.194202.objecter successfully canceled tick 2023-10-30T12:15:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14500392a0 msgr2=0x7f145003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14500392a0 0x7f145003b760 secure :-1 s=READY pgs=4193 cs=0 l=1 rev1=1 crypto rx=0x7f14701ad890 tx=0x7f1458006d70 comp rx=0 tx=0).stop 2023-10-30T12:15:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 10 monclient: shutdown 2023-10-30T12:15:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 msgr2=0x7f14701c2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:11.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f14701c2d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f146802c5b0 tx=0x7f1468004a40 comp rx=0 tx=0).stop 2023-10-30T12:15:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 shutdown_connections 2023-10-30T12:15:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14500392a0 0x7f145003b760 unknown :-1 s=CLOSED pgs=4193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 --2- 172.21.15.50:0/3950215858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f147009b170 0x7f14701c2d30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 >> 172.21.15.50:0/3950215858 conn(0x7f14700a3980 msgr2=0x7f14700965b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:11.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 shutdown_connections 2023-10-30T12:15:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 -- 172.21.15.50:0/3950215858 wait complete. 2023-10-30T12:15:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:11.780+0000 7f1475209640 1 librados: shutdown 2023-10-30T12:15:11.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:15:11.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:15:31.869 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:15:31.870 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-10-30T12:15:31.870 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:15:31.871 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:15:31.874 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:15:31.874 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:15:31.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:31.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.990+0000 7f2b249d0640 10 monclient: get_monmap_and_config 2023-10-30T12:15:31.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.990+0000 7f2b249d0640 10 monclient: build_initial_monmap 2023-10-30T12:15:31.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.990+0000 7f2b249d0640 10 monclient: monmap: 2023-10-30T12:15:31.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:31.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:31.991108+0000 2023-10-30T12:15:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:31.991108+0000 2023-10-30T12:15:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:31.994 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.990+0000 7f2b249d0640 1 Processor -- start 2023-10-30T12:15:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.990+0000 7f2b249d0640 1 -- start start 2023-10-30T12:15:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.990+0000 7f2b249d0640 10 monclient: init 2023-10-30T12:15:32.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:32.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:15:32.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 0x7f2b201326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:32.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2b200a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:32.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b20099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:32.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b200a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:32.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2b20132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:32.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200a4f40 con 0x7f2b20099a40 2023-10-30T12:15:32.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:32.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b200a3d70 con 0x7f2b200a4b60 2023-10-30T12:15:32.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:32.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200a4150 con 0x7f2b20132be0 2023-10-30T12:15:32.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:32.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b249d0640 10 monclient(hunting): authenticate will time out at 1777.174371s 2023-10-30T12:15:32.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b200a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b200a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53162/0 (socket says 172.21.15.50:53162) 2023-10-30T12:15:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 -- 172.21.15.50:0/775594083 learned_addr learned my addr 172.21.15.50:0/775594083 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): get_auth_request con 0x7f2b20132be0 auth_method 0 2023-10-30T12:15:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 1 --2- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 0x7f2b201326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 10 monclient(hunting): get_auth_request con 0x7f2b200a4b60 auth_method 0 2023-10-30T12:15:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 1 -- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 msgr2=0x7f2b201326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:15:32.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 1 -- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 msgr2=0x7f2b201326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:32.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 1 --2- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 0x7f2b201326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:32.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1e59c640 1 --2- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 0x7f2b201326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:32.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient(hunting): handle_auth_done global_id 115028 payload 274 2023-10-30T12:15:32.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 -- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 msgr2=0x7f2b201326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:32.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 --2- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 0x7f2b201326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 -- 172.21.15.50:0/775594083 >> v1:172.21.15.118:6789/0 conn(0x7f2b20099a40 legacy=0x7f2b200a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:32.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:15:32.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 monclient: found mon.noname-b 2023-10-30T12:15:32.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:32.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:32.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.991+0000 7f2b1ed9d640 1 -- 172.21.15.50:0/775594083 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2009a440 con 0x7f2b20132be0 2023-10-30T12:15:32.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1ed9d640 1 --2- 172.21.15.50:0/775594083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b200a2a60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009e50 tx=0x7f2b1802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51e6fcee6034830d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:32.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 1 -- 172.21.15.50:0/775594083 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1802c9e0 con 0x7f2b20132be0 2023-10-30T12:15:32.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:32.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:32.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: dump: 2023-10-30T12:15:32.013 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:32.013 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:32.013 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.014 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.014 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:32.014 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:32.014 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:32.014 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:32.014 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:32.015 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:32.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: _finish_auth 0 2023-10-30T12:15:32.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:15:32.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:32.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 1 -- 172.21.15.50:0/775594083 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1802cb40 con 0x7f2b20132be0 2023-10-30T12:15:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 1 -- 172.21.15.50:0/775594083 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180343f0 con 0x7f2b20132be0 2023-10-30T12:15:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b249d0640 5 monclient: authenticate success, global_id 115028 2023-10-30T12:15:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b249d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b249d0640 4 set_mon_vals no callback set 2023-10-30T12:15:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.992+0000 7f2b1d59a640 10 monclient: dump: 2023-10-30T12:15:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:32.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 10 monclient: shutdown 2023-10-30T12:15:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 -- 172.21.15.50:0/775594083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 msgr2=0x7f2b200a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 --2- 172.21.15.50:0/775594083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b200a2a60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009e50 tx=0x7f2b1802be40 comp rx=0 tx=0).stop 2023-10-30T12:15:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 -- 172.21.15.50:0/775594083 shutdown_connections 2023-10-30T12:15:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 --2- 172.21.15.50:0/775594083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b200a2a60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 --2- 172.21.15.50:0/775594083 >> v2:172.21.15.118:3300/0 conn(0x7f2b200a4b60 0x7f2b201326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 -- 172.21.15.50:0/775594083 >> 172.21.15.50:0/775594083 conn(0x7f2b200a8060 msgr2=0x7f2b200a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:32.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 -- 172.21.15.50:0/775594083 shutdown_connections 2023-10-30T12:15:32.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.993+0000 7f2b249d0640 1 -- 172.21.15.50:0/775594083 wait complete. 2023-10-30T12:15:32.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 10 monclient: build_initial_monmap 2023-10-30T12:15:32.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 10 monclient: monmap: 2023-10-30T12:15:32.021 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:32.021 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:32.023 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 1 librados: starting msgr at 2023-10-30T12:15:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 1 librados: starting objecter 2023-10-30T12:15:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 1 Processor -- start 2023-10-30T12:15:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 1 -- start start 2023-10-30T12:15:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 1 librados: setting wanted keys 2023-10-30T12:15:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 1 librados: calling monclient init 2023-10-30T12:15:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.994+0000 7f2b249d0640 10 monclient: init 2023-10-30T12:15:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:32.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:15:32.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b200a4b60 0x7f2b201ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:32.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2b200a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:32.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b201cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:32.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2b20132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:32.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b2011e690 0x7f2b2011eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2b2011e690 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20120f30 con 0x7f2b200a4b60 2023-10-30T12:15:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20122430 con 0x7f2b2011e690 2023-10-30T12:15:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20122680 con 0x7f2b20132be0 2023-10-30T12:15:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b249d0640 10 monclient(hunting): authenticate will time out at 1777.178375s 2023-10-30T12:15:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b201cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b200a4b60 0x7f2b201ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b201cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53166/0 (socket says 172.21.15.50:53166) 2023-10-30T12:15:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b200a4b60 0x7f2b201ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37734/0 (socket says 172.21.15.50:37734) 2023-10-30T12:15:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 -- 172.21.15.50:0/1382327731 learned_addr learned my addr 172.21.15.50:0/1382327731 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): get_auth_request con 0x7f2b20132be0 auth_method 0 2023-10-30T12:15:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 10 monclient(hunting): get_auth_request con 0x7f2b200a4b60 auth_method 0 2023-10-30T12:15:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1ed9d640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b2011e690 0x7f2b2011eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient(hunting): handle_auth_done global_id 115031 payload 824 2023-10-30T12:15:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 -- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b2011e690 msgr2=0x7f2b2011eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b2011e690 0x7f2b2011eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 -- 172.21.15.50:0/1382327731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b200a4b60 msgr2=0x7f2b201ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b200a4b60 0x7f2b201ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:15:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 monclient: found mon.noname-c 2023-10-30T12:15:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 -- 172.21.15.50:0/1382327731 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18009b30 con 0x7f2b20132be0 2023-10-30T12:15:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1ed9d640 10 monclient: get_auth_request con 0x7f2b2011e690 auth_method 0 2023-10-30T12:15:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1ed9d640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b2011e690 0x7f2b2011eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:15:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b1dd9b640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b201cae00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400c3f0 tx=0x7f2b1400c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b0effd640 10 client.?.objecter ms_handle_connect 0x7f2b20132be0 2023-10-30T12:15:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.995+0000 7f2b0effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 1 -- 172.21.15.50:0/1382327731 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b140052d0 con 0x7f2b20132be0 2023-10-30T12:15:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: dump: 2023-10-30T12:15:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:32.037 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: _finish_auth 0 2023-10-30T12:15:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 5 monclient: authenticate success, global_id 115031 2023-10-30T12:15:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 1 -- 172.21.15.50:0/1382327731 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b14005450 con 0x7f2b20132be0 2023-10-30T12:15:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 1 -- 172.21.15.50:0/1382327731 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b14007260 con 0x7f2b20132be0 2023-10-30T12:15:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 10 monclient: _renew_subs 2023-10-30T12:15:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b200a18d0 con 0x7f2b20132be0 2023-10-30T12:15:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b1cd99640 4 set_mon_vals no callback set 2023-10-30T12:15:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b0effd640 10 monclient: dump: 2023-10-30T12:15:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:32.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 10 client.115031.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 10 monclient: _renew_subs 2023-10-30T12:15:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b201320a0 con 0x7f2b20132be0 2023-10-30T12:15:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.996+0000 7f2b249d0640 1 librados: init done 2023-10-30T12:15:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b0effd640 1 -- 172.21.15.50:0/1382327731 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2b140073c0 con 0x7f2b20132be0 2023-10-30T12:15:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b0effd640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2b000392f0 0x7f2b0003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b249d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b249d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b249d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b0effd640 1 -- 172.21.15.50:0/1382327731 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(456..456 src has 417..456) v4 ==== 6300+0+0 (secure 0 0 0) 0x7f2b14049f50 con 0x7f2b20132be0 2023-10-30T12:15:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b0effd640 10 client.115031.objecter ms_dispatch 0x7f2b20000b90 osd_map(456..456 src has 417..456) v4 2023-10-30T12:15:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b0effd640 3 client.115031.objecter handle_osd_map got epochs [456,456] > 0 2023-10-30T12:15:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b0effd640 3 client.115031.objecter handle_osd_map decoding full epoch 456 2023-10-30T12:15:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2004e160 con 0x7f2b20132be0 2023-10-30T12:15:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b0effd640 20 client.115031.objecter dump_active .. 0 homeless 2023-10-30T12:15:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b1e59c640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2b000392f0 0x7f2b0003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.997+0000 7f2b1e59c640 10 monclient: get_auth_request con 0x7f2b000392f0 auth_method 0 2023-10-30T12:15:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.998+0000 7f2b0effd640 10 client.115031.objecter ms_handle_connect 0x7f2b000392f0 2023-10-30T12:15:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:31.998+0000 7f2b1e59c640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2b000392f0 0x7f2b0003b7b0 secure :-1 s=READY pgs=4195 cs=0 l=1 rev1=1 crypto rx=0x7f2b10006fd0 tx=0x7f2b10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.002+0000 7f2b0effd640 1 -- 172.21.15.50:0/1382327731 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b140035f0 con 0x7f2b20132be0 2023-10-30T12:15:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.002+0000 7f2b0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.002+0000 7f2b0effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.246+0000 7f2b249d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.246+0000 7f2b249d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:32.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.246+0000 7f2b249d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.246+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2b201227c0 con 0x7f2b20132be0 2023-10-30T12:15:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.247+0000 7f2b0effd640 1 -- 172.21.15.50:0/1382327731 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2b201227c0 con 0x7f2b20132be0 2023-10-30T12:15:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.247+0000 7f2b0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:32.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.247+0000 7f2b0effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:32.250 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.248+0000 7f2b249d0640 10 librados: watch_flush enter 2023-10-30T12:15:32.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.248+0000 7f2b249d0640 10 librados: watch_flush exit 2023-10-30T12:15:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.248+0000 7f2b249d0640 20 client.115031.objecter shutdown clearing up homeless session... 2023-10-30T12:15:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.248+0000 7f2b249d0640 10 client.115031.objecter successfully canceled tick 2023-10-30T12:15:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2b000392f0 msgr2=0x7f2b0003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2b000392f0 0x7f2b0003b7b0 secure :-1 s=READY pgs=4195 cs=0 l=1 rev1=1 crypto rx=0x7f2b10006fd0 tx=0x7f2b10008040 comp rx=0 tx=0).stop 2023-10-30T12:15:32.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 10 monclient: shutdown 2023-10-30T12:15:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 msgr2=0x7f2b201cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b201cae00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400c3f0 tx=0x7f2b1400c7f0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 shutdown_connections 2023-10-30T12:15:32.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2b000392f0 0x7f2b0003b7b0 unknown :-1 s=CLOSED pgs=4195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b2011e690 0x7f2b2011eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2b20132be0 0x7f2b201cae00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 --2- 172.21.15.50:0/1382327731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b200a4b60 0x7f2b201ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 >> 172.21.15.50:0/1382327731 conn(0x7f2b200a8060 msgr2=0x7f2b201344a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 shutdown_connections 2023-10-30T12:15:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 -- 172.21.15.50:0/1382327731 wait complete. 2023-10-30T12:15:32.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:32.249+0000 7f2b249d0640 1 librados: shutdown 2023-10-30T12:15:35.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:35.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.377+0000 7f6e1c275640 10 monclient: get_monmap_and_config 2023-10-30T12:15:35.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.377+0000 7f6e1c275640 10 monclient: build_initial_monmap 2023-10-30T12:15:35.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.377+0000 7f6e1c275640 10 monclient: monmap: 2023-10-30T12:15:35.379 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:35.379 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:35.379 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:35.378173+0000 2023-10-30T12:15:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:35.378173+0000 2023-10-30T12:15:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:35.380 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:35.381 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.377+0000 7f6e1c275640 1 Processor -- start 2023-10-30T12:15:35.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.377+0000 7f6e1c275640 1 -- start start 2023-10-30T12:15:35.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.377+0000 7f6e1c275640 10 monclient: init 2023-10-30T12:15:35.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:35.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:15:35.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e140a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:35.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6e14099a40 0x7f6e1413aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient(hunting): picked mon.noname-a con 0x7f6e14099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e1413cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient(hunting): picked mon.noname-b con 0x7f6e14132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e140a4f40 con 0x7f6e140a4b60 2023-10-30T12:15:35.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e140a3d70 con 0x7f6e14099a40 2023-10-30T12:15:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e140a4150 con 0x7f6e14132db0 2023-10-30T12:15:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1c275640 10 monclient(hunting): authenticate will time out at 1780.561553s 2023-10-30T12:15:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a011640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6e140a4b60 0x7f6e141326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55278/0 (socket says 172.21.15.50:55278) 2023-10-30T12:15:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6e14099a40 0x7f6e1413aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a011640 1 -- 172.21.15.50:0/3726189282 learned_addr learned my addr 172.21.15.50:0/3726189282 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 1 --2- 172.21.15.50:0/3726189282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e1413cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 10 monclient(hunting): get_auth_request con 0x7f6e14099a40 auth_method 0 2023-10-30T12:15:35.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 10 monclient(hunting): get_auth_request con 0x7f6e14132db0 auth_method 0 2023-10-30T12:15:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e1a812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.378+0000 7f6e19810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 10 monclient(hunting): handle_auth_done global_id 115034 payload 274 2023-10-30T12:15:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 1 -- 172.21.15.50:0/3726189282 >> v2:172.21.15.50:3300/0 conn(0x7f6e14099a40 msgr2=0x7f6e1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 1 --2- 172.21.15.50:0/3726189282 >> v2:172.21.15.50:3300/0 conn(0x7f6e14099a40 0x7f6e1413aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 1 -- 172.21.15.50:0/3726189282 >> v1:172.21.15.50:6789/0 conn(0x7f6e140a4b60 legacy=0x7f6e141326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:35.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 10 monclient: _finish_hunting 0 2023-10-30T12:15:35.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 1 monclient: found mon.noname-b 2023-10-30T12:15:35.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:35.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:35.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 1 -- 172.21.15.50:0/3726189282 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1409a440 con 0x7f6e14132db0 2023-10-30T12:15:35.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e19810640 1 --2- 172.21.15.50:0/3726189282 >> v2:172.21.15.50:3300/0 conn(0x7f6e14099a40 0x7f6e1413aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:35.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1a812640 1 --2- 172.21.15.50:0/3726189282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e1413cfd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009d90 tx=0x7f6e0c02bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=253b7836b1b54cbe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:35.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1900f640 1 -- 172.21.15.50:0/3726189282 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c02cb40 con 0x7f6e14132db0 2023-10-30T12:15:35.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1900f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:35.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1900f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:35.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1900f640 10 monclient: dump: 2023-10-30T12:15:35.398 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:35.398 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:35.398 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.398 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.398 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:35.398 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:35.399 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:35.399 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:35.399 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:35.399 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:35.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1900f640 10 monclient: _finish_auth 0 2023-10-30T12:15:35.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.379+0000 7f6e1900f640 10 monclient: _check_auth_tickets 2023-10-30T12:15:35.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1900f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:35.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1900f640 1 -- 172.21.15.50:0/3726189282 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c02cca0 con 0x7f6e14132db0 2023-10-30T12:15:35.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 5 monclient: authenticate success, global_id 115034 2023-10-30T12:15:35.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:35.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1900f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:35.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:35.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1900f640 1 -- 172.21.15.50:0/3726189282 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c034660 con 0x7f6e14132db0 2023-10-30T12:15:35.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1900f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:35.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 4 set_mon_vals no callback set 2023-10-30T12:15:35.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1900f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:35.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1900f640 10 monclient: dump: 2023-10-30T12:15:35.402 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:35.402 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:35.402 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:35.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 10 monclient: shutdown 2023-10-30T12:15:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 -- 172.21.15.50:0/3726189282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 msgr2=0x7f6e1413cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 --2- 172.21.15.50:0/3726189282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e1413cfd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009d90 tx=0x7f6e0c02bd90 comp rx=0 tx=0).stop 2023-10-30T12:15:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 -- 172.21.15.50:0/3726189282 shutdown_connections 2023-10-30T12:15:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 --2- 172.21.15.50:0/3726189282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e1413cfd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 --2- 172.21.15.50:0/3726189282 >> v2:172.21.15.50:3300/0 conn(0x7f6e14099a40 0x7f6e1413aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 -- 172.21.15.50:0/3726189282 >> 172.21.15.50:0/3726189282 conn(0x7f6e140a8060 msgr2=0x7f6e140a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 -- 172.21.15.50:0/3726189282 shutdown_connections 2023-10-30T12:15:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.380+0000 7f6e1c275640 1 -- 172.21.15.50:0/3726189282 wait complete. 2023-10-30T12:15:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient: build_initial_monmap 2023-10-30T12:15:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient: monmap: 2023-10-30T12:15:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:35.407 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 librados: starting msgr at 2023-10-30T12:15:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 librados: starting objecter 2023-10-30T12:15:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 Processor -- start 2023-10-30T12:15:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 -- start start 2023-10-30T12:15:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 librados: setting wanted keys 2023-10-30T12:15:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 librados: calling monclient init 2023-10-30T12:15:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient: init 2023-10-30T12:15:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:15:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 0x7f6e14136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): picked mon.noname-b con 0x7f6e140a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e14136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): picked mon.noname-c con 0x7f6e14132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e141372d0 0x7f6e141ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): picked mon.noname-a con 0x7f6e141372d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e14139f80 con 0x7f6e141372d0 2023-10-30T12:15:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e140a18d0 con 0x7f6e140a4b60 2023-10-30T12:15:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1413cb60 con 0x7f6e14132db0 2023-10-30T12:15:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.381+0000 7f6e1c275640 10 monclient(hunting): authenticate will time out at 1780.564984s 2023-10-30T12:15:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e141372d0 0x7f6e141ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e14136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e14136d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53174/0 (socket says 172.21.15.50:53174) 2023-10-30T12:15:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e141372d0 0x7f6e141ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37754/0 (socket says 172.21.15.50:37754) 2023-10-30T12:15:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 -- 172.21.15.50:0/690842415 learned_addr learned my addr 172.21.15.50:0/690842415 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): get_auth_request con 0x7f6e14132db0 auth_method 0 2023-10-30T12:15:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 10 monclient(hunting): get_auth_request con 0x7f6e141372d0 auth_method 0 2023-10-30T12:15:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 0x7f6e14136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 10 monclient(hunting): get_auth_request con 0x7f6e140a4b60 auth_method 0 2023-10-30T12:15:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 1 -- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 msgr2=0x7f6e14136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:15:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 1 -- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 msgr2=0x7f6e14136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:15:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 0x7f6e14136850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient(hunting): handle_auth_done global_id 115037 payload 824 2023-10-30T12:15:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a011640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 0x7f6e14136850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 -- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 msgr2=0x7f6e14136850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 0x7f6e14136850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 -- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e141372d0 msgr2=0x7f6e141ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e141372d0 0x7f6e141ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient: _finish_hunting 0 2023-10-30T12:15:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 monclient: found mon.noname-c 2023-10-30T12:15:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 -- 172.21.15.50:0/690842415 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04009400 con 0x7f6e14132db0 2023-10-30T12:15:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1a812640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e141372d0 0x7f6e141ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e19810640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e14136d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400e900 tx=0x7f6e0400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 client.?.objecter ms_handle_connect 0x7f6e14132db0 2023-10-30T12:15:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 1 -- 172.21.15.50:0/690842415 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04009660 con 0x7f6e14132db0 2023-10-30T12:15:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: dump: 2023-10-30T12:15:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:35.422 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: _finish_auth 0 2023-10-30T12:15:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 5 monclient: authenticate success, global_id 115037 2023-10-30T12:15:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 1 -- 172.21.15.50:0/690842415 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0400ca00 con 0x7f6e14132db0 2023-10-30T12:15:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 1 -- 172.21.15.50:0/690842415 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400cdb0 con 0x7f6e14132db0 2023-10-30T12:15:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e0affd640 10 monclient: dump: 2023-10-30T12:15:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:35.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1880e640 4 set_mon_vals no callback set 2023-10-30T12:15:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 10 monclient: _renew_subs 2023-10-30T12:15:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c009a70 con 0x7f6e14132db0 2023-10-30T12:15:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 10 client.115037.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 10 monclient: _renew_subs 2023-10-30T12:15:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14139650 con 0x7f6e14132db0 2023-10-30T12:15:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.382+0000 7f6e1c275640 1 librados: init done 2023-10-30T12:15:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.383+0000 7f6e0affd640 1 -- 172.21.15.50:0/690842415 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6e04013470 con 0x7f6e14132db0 2023-10-30T12:15:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.383+0000 7f6e1c275640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.383+0000 7f6e1c275640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.383+0000 7f6e1c275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.383+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1404e160 con 0x7f6e14132db0 2023-10-30T12:15:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e0affd640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df8039320 0x7f6df803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e0affd640 1 -- 172.21.15.50:0/690842415 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(460..460 src has 417..460) v4 ==== 5914+0+0 (secure 0 0 0) 0x7f6e0404a930 con 0x7f6e14132db0 2023-10-30T12:15:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e0affd640 10 client.115037.objecter ms_dispatch 0x7f6e14000b90 osd_map(460..460 src has 417..460) v4 2023-10-30T12:15:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e0affd640 3 client.115037.objecter handle_osd_map got epochs [460,460] > 0 2023-10-30T12:15:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e0affd640 3 client.115037.objecter handle_osd_map decoding full epoch 460 2023-10-30T12:15:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e0affd640 20 client.115037.objecter dump_active .. 0 homeless 2023-10-30T12:15:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e1a011640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df8039320 0x7f6df803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e1a011640 10 monclient: get_auth_request con 0x7f6df8039320 auth_method 0 2023-10-30T12:15:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e0affd640 10 client.115037.objecter ms_handle_connect 0x7f6df8039320 2023-10-30T12:15:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.384+0000 7f6e1a011640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df8039320 0x7f6df803b7e0 secure :-1 s=READY pgs=4196 cs=0 l=1 rev1=1 crypto rx=0x7f6e10005880 tx=0x7f6e10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.388+0000 7f6e0affd640 1 -- 172.21.15.50:0/690842415 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e040102b0 con 0x7f6e14132db0 2023-10-30T12:15:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.388+0000 7f6e0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.388+0000 7f6e0affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:35.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.635+0000 7f6e1c275640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:35.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.635+0000 7f6e1c275640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:35.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.635+0000 7f6e1c275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.635+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e14138980 con 0x7f6e14132db0 2023-10-30T12:15:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.635+0000 7f6e0affd640 1 -- 172.21.15.50:0/690842415 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f6e04017080 con 0x7f6e14132db0 2023-10-30T12:15:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.636+0000 7f6e0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.636+0000 7f6e0affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:35.639 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:35.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 10 librados: watch_flush enter 2023-10-30T12:15:35.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 10 librados: watch_flush exit 2023-10-30T12:15:35.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 20 client.115037.objecter shutdown clearing up homeless session... 2023-10-30T12:15:35.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 10 client.115037.objecter successfully canceled tick 2023-10-30T12:15:35.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df8039320 msgr2=0x7f6df803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:35.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df8039320 0x7f6df803b7e0 secure :-1 s=READY pgs=4196 cs=0 l=1 rev1=1 crypto rx=0x7f6e10005880 tx=0x7f6e10009040 comp rx=0 tx=0).stop 2023-10-30T12:15:35.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 10 monclient: shutdown 2023-10-30T12:15:35.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:35.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 msgr2=0x7f6e14136d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:35.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e14136d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400e900 tx=0x7f6e0400edc0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 shutdown_connections 2023-10-30T12:15:35.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6df8039320 0x7f6df803b7e0 unknown :-1 s=CLOSED pgs=4196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e141372d0 0x7f6e141ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e14132db0 0x7f6e14136d90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 --2- 172.21.15.50:0/690842415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e140a4b60 0x7f6e14136850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:35.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 >> 172.21.15.50:0/690842415 conn(0x7f6e140a8060 msgr2=0x7f6e1409aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:35.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.637+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 shutdown_connections 2023-10-30T12:15:35.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.638+0000 7f6e1c275640 1 -- 172.21.15.50:0/690842415 wait complete. 2023-10-30T12:15:35.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:35.638+0000 7f6e1c275640 1 librados: shutdown 2023-10-30T12:15:38.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.766+0000 7f42c1c75640 10 monclient: get_monmap_and_config 2023-10-30T12:15:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.766+0000 7f42c1c75640 10 monclient: build_initial_monmap 2023-10-30T12:15:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.766+0000 7f42c1c75640 10 monclient: monmap: 2023-10-30T12:15:38.768 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:38.767411+0000 2023-10-30T12:15:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:38.767411+0000 2023-10-30T12:15:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:38.769 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:38.770 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:38.771 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 1 Processor -- start 2023-10-30T12:15:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 1 -- start start 2023-10-30T12:15:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient: init 2023-10-30T12:15:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T12:15:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc0707d0 0x7f42bc09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc0707d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f42bc070bb0 0x7f42bc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc070bb0 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:38.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42bc1345f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:38.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc09ee40 con 0x7f42bc1345f0 2023-10-30T12:15:38.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:38.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42bc09f4f0 con 0x7f42bc070bb0 2023-10-30T12:15:38.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:38.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc09f7d0 con 0x7f42bc0707d0 2023-10-30T12:15:38.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:38.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42c1c75640 10 monclient(hunting): authenticate will time out at 1783.950722s 2023-10-30T12:15:38.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc0707d0 0x7f42bc09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:38.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bbfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f42bc1345f0 0x7f42bc134a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47918/0 (socket says 172.21.15.50:47918) 2023-10-30T12:15:38.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bbfff640 1 -- 172.21.15.50:0/3770328149 learned_addr learned my addr 172.21.15.50:0/3770328149 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:38.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 10 monclient(hunting): get_auth_request con 0x7f42bc0707d0 auth_method 0 2023-10-30T12:15:38.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:38.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:38.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:38.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:38.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:38.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.767+0000 7f42bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:38.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42baffd640 1 --2- 172.21.15.50:0/3770328149 >> v2:172.21.15.118:3300/0 conn(0x7f42bc070bb0 0x7f42bc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:38.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42baffd640 10 monclient(hunting): get_auth_request con 0x7f42bc070bb0 auth_method 0 2023-10-30T12:15:38.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:38.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42baffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:38.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42baffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:38.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42ba7fc640 1 -- 172.21.15.50:0/3770328149 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477937036 0 0) 0x7f42bc09ee40 con 0x7f42bc1345f0 2023-10-30T12:15:38.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 10 monclient(hunting): handle_auth_done global_id 115043 payload 274 2023-10-30T12:15:38.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 1 -- 172.21.15.50:0/3770328149 >> v2:172.21.15.118:3300/0 conn(0x7f42bc070bb0 msgr2=0x7f42bc070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:38.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 1 --2- 172.21.15.50:0/3770328149 >> v2:172.21.15.118:3300/0 conn(0x7f42bc070bb0 0x7f42bc070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:38.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 1 -- 172.21.15.50:0/3770328149 >> v1:172.21.15.50:6789/0 conn(0x7f42bc1345f0 legacy=0x7f42bc134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:38.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:15:38.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 1 monclient: found mon.noname-b 2023-10-30T12:15:38.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:38.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:38.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 1 -- 172.21.15.50:0/3770328149 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc09fab0 con 0x7f42bc0707d0 2023-10-30T12:15:38.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42ba7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:15:38.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.768+0000 7f42bb7fe640 1 --2- 172.21.15.50:0/3770328149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc0707d0 0x7f42bc09e7e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f42ac0097a0 tx=0x7f42ac02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80f7f0c627e33b11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 1 -- 172.21.15.50:0/3770328149 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac02f050 con 0x7f42bc0707d0 2023-10-30T12:15:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: dump: 2023-10-30T12:15:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:38.791 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: _finish_auth 0 2023-10-30T12:15:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:15:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 1 -- 172.21.15.50:0/3770328149 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ac031700 con 0x7f42bc0707d0 2023-10-30T12:15:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 5 monclient: authenticate success, global_id 115043 2023-10-30T12:15:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 4 set_mon_vals no callback set 2023-10-30T12:15:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 1 -- 172.21.15.50:0/3770328149 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac031a90 con 0x7f42bc0707d0 2023-10-30T12:15:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42ba7fc640 10 monclient: dump: 2023-10-30T12:15:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:38.794 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 10 monclient: shutdown 2023-10-30T12:15:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 -- 172.21.15.50:0/3770328149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc0707d0 msgr2=0x7f42bc09e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 --2- 172.21.15.50:0/3770328149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc0707d0 0x7f42bc09e7e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f42ac0097a0 tx=0x7f42ac02b7a0 comp rx=0 tx=0).stop 2023-10-30T12:15:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 -- 172.21.15.50:0/3770328149 shutdown_connections 2023-10-30T12:15:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 --2- 172.21.15.50:0/3770328149 >> v2:172.21.15.118:3300/0 conn(0x7f42bc070bb0 0x7f42bc070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 --2- 172.21.15.50:0/3770328149 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc0707d0 0x7f42bc09e7e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 -- 172.21.15.50:0/3770328149 >> 172.21.15.50:0/3770328149 conn(0x7f42bc0a47f0 msgr2=0x7f42bc0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 -- 172.21.15.50:0/3770328149 shutdown_connections 2023-10-30T12:15:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.769+0000 7f42c1c75640 1 -- 172.21.15.50:0/3770328149 wait complete. 2023-10-30T12:15:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 10 monclient: build_initial_monmap 2023-10-30T12:15:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 10 monclient: monmap: 2023-10-30T12:15:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:38.798 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 1 librados: starting msgr at 2023-10-30T12:15:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 1 librados: starting objecter 2023-10-30T12:15:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 1 Processor -- start 2023-10-30T12:15:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 1 -- start start 2023-10-30T12:15:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 1 librados: setting wanted keys 2023-10-30T12:15:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 1 librados: calling monclient init 2023-10-30T12:15:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.770+0000 7f42c1c75640 10 monclient: init 2023-10-30T12:15:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:15:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42bc0707d0 0x7f42bc1c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): picked mon.noname-a con 0x7f42bc0707d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 0x7f42bc131930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc070bb0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc1345f0 0x7f42bc131e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc1345f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc09d6a0 con 0x7f42bc0707d0 2023-10-30T12:15:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc1323b0 con 0x7f42bc070bb0 2023-10-30T12:15:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc1ca5f0 con 0x7f42bc1345f0 2023-10-30T12:15:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42c1c75640 10 monclient(hunting): authenticate will time out at 1783.954457s 2023-10-30T12:15:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42bc0707d0 0x7f42bc1c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42bc0707d0 0x7f42bc1c9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35972/0 (socket says 172.21.15.50:35972) 2023-10-30T12:15:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 1 -- 172.21.15.50:0/1355588354 learned_addr learned my addr 172.21.15.50:0/1355588354 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 10 monclient(hunting): get_auth_request con 0x7f42bc0707d0 auth_method 0 2023-10-30T12:15:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc1345f0 0x7f42bc131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): get_auth_request con 0x7f42bc1345f0 auth_method 0 2023-10-30T12:15:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 0x7f42bc131930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 10 monclient(hunting): get_auth_request con 0x7f42bc070bb0 auth_method 0 2023-10-30T12:15:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 1 -- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 msgr2=0x7f42bc131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:15:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 1 -- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 msgr2=0x7f42bc131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:15:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 0x7f42bc131930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42baffd640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 0x7f42bc131930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.771+0000 7f42bbfff640 10 monclient(hunting): handle_auth_done global_id 115046 payload 824 2023-10-30T12:15:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 1 -- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 msgr2=0x7f42bc131930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 0x7f42bc131930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 1 -- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42bc0707d0 msgr2=0x7f42bc1c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42bc0707d0 0x7f42bc1c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 1 monclient: found mon.noname-c 2023-10-30T12:15:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 1 -- 172.21.15.50:0/1355588354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac009400 con 0x7f42bc1345f0 2023-10-30T12:15:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bb7fe640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42bc0707d0 0x7f42bc1c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42bbfff640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc1345f0 0x7f42bc131e70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f42b400c6d0 tx=0x7f42b400cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 client.?.objecter ms_handle_connect 0x7f42bc1345f0 2023-10-30T12:15:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 1 -- 172.21.15.50:0/1355588354 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4004d60 con 0x7f42bc1345f0 2023-10-30T12:15:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: dump: 2023-10-30T12:15:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:38.813 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: _finish_auth 0 2023-10-30T12:15:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:15:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 1 -- 172.21.15.50:0/1355588354 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b400a640 con 0x7f42bc1345f0 2023-10-30T12:15:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 5 monclient: authenticate success, global_id 115046 2023-10-30T12:15:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 10 monclient: _renew_subs 2023-10-30T12:15:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc11e8f0 con 0x7f42bc1345f0 2023-10-30T12:15:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 1 -- 172.21.15.50:0/1355588354 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400aa30 con 0x7f42bc1345f0 2023-10-30T12:15:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42a3fff640 10 monclient: dump: 2023-10-30T12:15:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:38.817 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42b9ffb640 4 set_mon_vals no callback set 2023-10-30T12:15:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 10 client.115046.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 10 monclient: _renew_subs 2023-10-30T12:15:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc11fd40 con 0x7f42bc1345f0 2023-10-30T12:15:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.772+0000 7f42c1c75640 1 librados: init done 2023-10-30T12:15:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42a3fff640 1 -- 172.21.15.50:0/1355588354 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f42b40070d0 con 0x7f42bc1345f0 2023-10-30T12:15:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42c1c75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42c1c75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42c1c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42a3fff640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42a80392f0 0x7f42a803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc04e160 con 0x7f42bc1345f0 2023-10-30T12:15:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42a3fff640 1 -- 172.21.15.50:0/1355588354 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(464..464 src has 417..464) v4 ==== 6193+0+0 (secure 0 0 0) 0x7f42b404a520 con 0x7f42bc1345f0 2023-10-30T12:15:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42a3fff640 10 client.115046.objecter ms_dispatch 0x7f42bc000b90 osd_map(464..464 src has 417..464) v4 2023-10-30T12:15:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42a3fff640 3 client.115046.objecter handle_osd_map got epochs [464,464] > 0 2023-10-30T12:15:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42a3fff640 3 client.115046.objecter handle_osd_map decoding full epoch 464 2023-10-30T12:15:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42a3fff640 20 client.115046.objecter dump_active .. 0 homeless 2023-10-30T12:15:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42bb7fe640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42a80392f0 0x7f42a803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.773+0000 7f42bb7fe640 10 monclient: get_auth_request con 0x7f42a80392f0 auth_method 0 2023-10-30T12:15:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.774+0000 7f42bb7fe640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42a80392f0 0x7f42a803b7b0 secure :-1 s=READY pgs=4198 cs=0 l=1 rev1=1 crypto rx=0x7f42bc1caeb0 tx=0x7f42ac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.774+0000 7f42a3fff640 10 client.115046.objecter ms_handle_connect 0x7f42a80392f0 2023-10-30T12:15:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.778+0000 7f42a3fff640 1 -- 172.21.15.50:0/1355588354 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42b4018030 con 0x7f42bc1345f0 2023-10-30T12:15:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.778+0000 7f42a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:38.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:38.778+0000 7f42a3fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:39.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.020+0000 7f42c1c75640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.020+0000 7f42c1c75640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.020+0000 7f42c1c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.020+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f42bc1200b0 con 0x7f42bc1345f0 2023-10-30T12:15:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.021+0000 7f42a3fff640 1 -- 172.21.15.50:0/1355588354 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f42b4018020 con 0x7f42bc1345f0 2023-10-30T12:15:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.021+0000 7f42a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.021+0000 7f42a3fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:39.024 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 10 librados: watch_flush enter 2023-10-30T12:15:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 10 librados: watch_flush exit 2023-10-30T12:15:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 20 client.115046.objecter shutdown clearing up homeless session... 2023-10-30T12:15:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 10 client.115046.objecter successfully canceled tick 2023-10-30T12:15:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42a80392f0 msgr2=0x7f42a803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42a80392f0 0x7f42a803b7b0 secure :-1 s=READY pgs=4198 cs=0 l=1 rev1=1 crypto rx=0x7f42bc1caeb0 tx=0x7f42ac008030 comp rx=0 tx=0).stop 2023-10-30T12:15:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 10 monclient: shutdown 2023-10-30T12:15:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc1345f0 msgr2=0x7f42bc131e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc1345f0 0x7f42bc131e70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f42b400c6d0 tx=0x7f42b400cb90 comp rx=0 tx=0).stop 2023-10-30T12:15:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 shutdown_connections 2023-10-30T12:15:39.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f42a80392f0 0x7f42a803b7b0 unknown :-1 s=CLOSED pgs=4198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42bc1345f0 0x7f42bc131e70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f42bc070bb0 0x7f42bc131930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 --2- 172.21.15.50:0/1355588354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42bc0707d0 0x7f42bc1c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 >> 172.21.15.50:0/1355588354 conn(0x7f42bc0a47f0 msgr2=0x7f42bc1bccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:39.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 shutdown_connections 2023-10-30T12:15:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 -- 172.21.15.50:0/1355588354 wait complete. 2023-10-30T12:15:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.023+0000 7f42c1c75640 1 librados: shutdown 2023-10-30T12:15:39.035 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:15:39.035 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:15:39.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:15:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.139+0000 7f790349b640 10 monclient: get_monmap_and_config 2023-10-30T12:15:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.139+0000 7f790349b640 10 monclient: build_initial_monmap 2023-10-30T12:15:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.139+0000 7f790349b640 10 monclient: monmap: 2023-10-30T12:15:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:39.140831+0000 2023-10-30T12:15:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:39.140831+0000 2023-10-30T12:15:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:39.145 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 1 Processor -- start 2023-10-30T12:15:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 1 -- start start 2023-10-30T12:15:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 10 monclient: init 2023-10-30T12:15:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-10-30T12:15:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc097780 0x7f78fc097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 10 monclient(hunting): picked mon.noname-b con 0x7f78fc097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f78fc073b10 0x7f78fc0a0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 10 monclient(hunting): picked mon.noname-a con 0x7f78fc073b10 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f78fc09ccd0 0x7f78fc0a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 10 monclient(hunting): picked mon.noname-c con 0x7f78fc09ccd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.140+0000 7f790349b640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc071ec0 con 0x7f78fc073b10 2023-10-30T12:15:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f790349b640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc09bd10 con 0x7f78fc09ccd0 2023-10-30T12:15:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f790349b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc09bff0 con 0x7f78fc097780 2023-10-30T12:15:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f790349b640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f790349b640 10 monclient(hunting): authenticate will time out at 1784.324024s 2023-10-30T12:15:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f78fc073b10 0x7f78fc0a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc097780 0x7f78fc097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc097780 0x7f78fc097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35428/0 (socket says 172.21.15.50:35428) 2023-10-30T12:15:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f78fc073b10 0x7f78fc0a0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35986/0 (socket says 172.21.15.50:35986) 2023-10-30T12:15:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 1 -- 172.21.15.50:0/3196040739 learned_addr learned my addr 172.21.15.50:0/3196040739 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 10 monclient(hunting): get_auth_request con 0x7f78fc097780 auth_method 0 2023-10-30T12:15:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): get_auth_request con 0x7f78fc073b10 auth_method 0 2023-10-30T12:15:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901a38640 1 --2- 172.21.15.50:0/3196040739 >> v2:172.21.15.118:3300/0 conn(0x7f78fc09ccd0 0x7f78fc0a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901a38640 10 monclient(hunting): get_auth_request con 0x7f78fc09ccd0 auth_method 0 2023-10-30T12:15:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901a38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901a38640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901a38640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient(hunting): handle_auth_done global_id 194229 payload 274 2023-10-30T12:15:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 -- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc097780 msgr2=0x7f78fc097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 --2- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc097780 0x7f78fc097b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 -- 172.21.15.50:0/3196040739 >> v2:172.21.15.118:3300/0 conn(0x7f78fc09ccd0 msgr2=0x7f78fc0a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 --2- 172.21.15.50:0/3196040739 >> v2:172.21.15.118:3300/0 conn(0x7f78fc09ccd0 0x7f78fc0a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient: _finish_hunting 0 2023-10-30T12:15:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 monclient: found mon.noname-a 2023-10-30T12:15:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7900a36640 1 -- 172.21.15.50:0/3196040739 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc0980a0 con 0x7f78fc073b10 2023-10-30T12:15:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.141+0000 7f7901237640 1 --2- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc097780 0x7f78fc097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f7900a36640 1 --2- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc073b10 0x7f78fc0a0080 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f78fc09e240 tx=0x7f78ec02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e942355a51f3fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 1 -- 172.21.15.50:0/3196040739 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec030710 con 0x7f78fc073b10 2023-10-30T12:15:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: dump: 2023-10-30T12:15:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.173 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: _finish_auth 0 2023-10-30T12:15:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:15:39.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 5 monclient: authenticate success, global_id 194229 2023-10-30T12:15:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 1 -- 172.21.15.50:0/3196040739 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec02e050 con 0x7f78fc073b10 2023-10-30T12:15:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 4 set_mon_vals no callback set 2023-10-30T12:15:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 1 -- 172.21.15.50:0/3196040739 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec030a90 con 0x7f78fc073b10 2023-10-30T12:15:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f78f3fff640 10 monclient: dump: 2023-10-30T12:15:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.178 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 10 monclient: shutdown 2023-10-30T12:15:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 1 -- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc073b10 msgr2=0x7f78fc0a0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 1 --2- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc073b10 0x7f78fc0a0080 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f78fc09e240 tx=0x7f78ec02b710 comp rx=0 tx=0).stop 2023-10-30T12:15:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 1 -- 172.21.15.50:0/3196040739 shutdown_connections 2023-10-30T12:15:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 1 --2- 172.21.15.50:0/3196040739 >> v2:172.21.15.118:3300/0 conn(0x7f78fc09ccd0 0x7f78fc0a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 1 --2- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc073b10 0x7f78fc0a0080 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 1 --2- 172.21.15.50:0/3196040739 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc097780 0x7f78fc097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.142+0000 7f790349b640 1 -- 172.21.15.50:0/3196040739 >> 172.21.15.50:0/3196040739 conn(0x7f78fc0a1170 msgr2=0x7f78fc0a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.143+0000 7f790349b640 1 -- 172.21.15.50:0/3196040739 shutdown_connections 2023-10-30T12:15:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.143+0000 7f790349b640 1 -- 172.21.15.50:0/3196040739 wait complete. 2023-10-30T12:15:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.143+0000 7f790349b640 10 monclient: build_initial_monmap 2023-10-30T12:15:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.143+0000 7f790349b640 10 monclient: monmap: 2023-10-30T12:15:39.181 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:39.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.143+0000 7f790349b640 1 librados: starting msgr at 2023-10-30T12:15:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.143+0000 7f790349b640 1 librados: starting objecter 2023-10-30T12:15:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.143+0000 7f790349b640 1 Processor -- start 2023-10-30T12:15:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 -- start start 2023-10-30T12:15:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 librados: setting wanted keys 2023-10-30T12:15:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 librados: calling monclient init 2023-10-30T12:15:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient: init 2023-10-30T12:15:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:15:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc073b10 0x7f78fc1ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): picked mon.noname-c con 0x7f78fc073b10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc097780 0x7f78fc1cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): picked mon.noname-a con 0x7f78fc097780 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78fc09ccd0 0x7f78fc08f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): picked mon.noname-b con 0x7f78fc09ccd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0935e0 con 0x7f78fc097780 2023-10-30T12:15:39.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0a3a60 con 0x7f78fc09ccd0 2023-10-30T12:15:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc073f70 con 0x7f78fc073b10 2023-10-30T12:15:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:39.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f790349b640 10 monclient(hunting): authenticate will time out at 1784.327500s 2023-10-30T12:15:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc097780 0x7f78fc1cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc073b10 0x7f78fc1ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc097780 0x7f78fc1cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35992/0 (socket says 172.21.15.50:35992) 2023-10-30T12:15:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 -- 172.21.15.50:0/3383309059 learned_addr learned my addr 172.21.15.50:0/3383309059 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:39.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc073b10 0x7f78fc1ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35430/0 (socket says 172.21.15.50:35430) 2023-10-30T12:15:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): get_auth_request con 0x7f78fc097780 auth_method 0 2023-10-30T12:15:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 10 monclient(hunting): get_auth_request con 0x7f78fc073b10 auth_method 0 2023-10-30T12:15:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient(hunting): handle_auth_done global_id 194232 payload 824 2023-10-30T12:15:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901a38640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78fc09ccd0 0x7f78fc08f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc073b10 msgr2=0x7f78fc1ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc073b10 0x7f78fc1ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78fc09ccd0 msgr2=0x7f78fc08f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78fc09ccd0 0x7f78fc08f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient: _finish_hunting 0 2023-10-30T12:15:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 monclient: found mon.noname-a 2023-10-30T12:15:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7900a36640 1 -- 172.21.15.50:0/3383309059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80094d0 con 0x7f78fc097780 2023-10-30T12:15:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901a38640 10 monclient: get_auth_request con 0x7f78fc09ccd0 auth_method 0 2023-10-30T12:15:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901237640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78fc073b10 0x7f78fc1ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.144+0000 7f7901a38640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78fc09ccd0 0x7f78fc08f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:15:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f7900a36640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc097780 0x7f78fc1cae10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78ec000c00 tx=0x7f78ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 client.?.objecter ms_handle_connect 0x7f78fc097780 2023-10-30T12:15:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec032890 con 0x7f78fc097780 2023-10-30T12:15:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: dump: 2023-10-30T12:15:39.198 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.198 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.198 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.200 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:15:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:15:39.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:39.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec032a10 con 0x7f78fc097780 2023-10-30T12:15:39.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 5 monclient: authenticate success, global_id 194232 2023-10-30T12:15:39.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 10 monclient: _renew_subs 2023-10-30T12:15:39.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ec009400 con 0x7f78fc097780 2023-10-30T12:15:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec032dc0 con 0x7f78fc097780 2023-10-30T12:15:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f1ffb640 10 monclient: dump: 2023-10-30T12:15:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.205 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 10 client.194232.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 10 monclient: _renew_subs 2023-10-30T12:15:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f78f37fe640 4 set_mon_vals no callback set 2023-10-30T12:15:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc092cb0 con 0x7f78fc097780 2023-10-30T12:15:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.145+0000 7f790349b640 1 librados: init done 2023-10-30T12:15:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f790349b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f790349b640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc08faf0 0x7f78fc08fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f78ec039040 con 0x7f78fc097780 2023-10-30T12:15:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc093400 con 0x7f78fc08faf0 2023-10-30T12:15:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc08faf0 0x7f78fc08fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient: get_auth_request con 0x7f78fc08faf0 auth_method 0 2023-10-30T12:15:39.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78e00392f0 0x7f78e003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(464..464 src has 417..464) v4 ==== 6193+0+0 (secure 0 0 0) 0x7f78ec06c440 con 0x7f78fc097780 2023-10-30T12:15:39.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 10 client.194232.objecter ms_dispatch 0x7f78fc000b90 osd_map(464..464 src has 417..464) v4 2023-10-30T12:15:39.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 3 client.194232.objecter handle_osd_map got epochs [464,464] > 0 2023-10-30T12:15:39.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 3 client.194232.objecter handle_osd_map decoding full epoch 464 2023-10-30T12:15:39.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 20 client.194232.objecter dump_active .. 0 homeless 2023-10-30T12:15:39.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 10 monclient(hunting): handle_auth_done global_id 194235 payload 824 2023-10-30T12:15:39.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 1 -- 172.21.15.50:0/3383309059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78fc0a0b00 con 0x7f78fc08faf0 2023-10-30T12:15:39.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901a38640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78e00392f0 0x7f78e003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901a38640 10 monclient: get_auth_request con 0x7f78e00392f0 auth_method 0 2023-10-30T12:15:39.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f7901237640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc08faf0 0x7f78fc08fed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f78f8011e70 tx=0x7f78f800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 10 client.194232.objecter ms_handle_connect 0x7f78fc08faf0 2023-10-30T12:15:39.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.146+0000 7f78f1ffb640 10 client.194232.objecter resend_mon_ops 2023-10-30T12:15:39.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f800fe50 con 0x7f78fc08faf0 2023-10-30T12:15:39.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:39.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 10 monclient: dump: 2023-10-30T12:15:39.213 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.213 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.213 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.213 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.214 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.214 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.214 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.214 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.215 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.215 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 10 client.194232.objecter ms_handle_connect 0x7f78e00392f0 2023-10-30T12:15:39.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f7901a38640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78e00392f0 0x7f78e003b7b0 secure :-1 s=READY pgs=4199 cs=0 l=1 rev1=1 crypto rx=0x7f78fc091f90 tx=0x7f78f400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f78f8014650 con 0x7f78fc08faf0 2023-10-30T12:15:39.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:39.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:39.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc08faf0 msgr2=0x7f78fc08fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc08faf0 0x7f78fc08fed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f78f8011e70 tx=0x7f78f800f610 comp rx=0 tx=0).stop 2023-10-30T12:15:39.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc08faf0 msgr2=0x7f78fc08fed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:39.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.147+0000 7f78f1ffb640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc08faf0 0x7f78fc08fed0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f78f8011e70 tx=0x7f78f800f610 comp rx=0 tx=0).stop 2023-10-30T12:15:39.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f790349b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:39.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f790349b640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc090410 0x7f78fc08bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f790349b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc06a5a0 con 0x7f78fc090410 2023-10-30T12:15:39.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc090410 0x7f78fc08bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient: get_auth_request con 0x7f78fc090410 auth_method 0 2023-10-30T12:15:39.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 10 monclient(hunting): handle_auth_done global_id 194238 payload 824 2023-10-30T12:15:39.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.153+0000 7f7901237640 1 -- 172.21.15.50:0/3383309059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f78fc092ea0 con 0x7f78fc090410 2023-10-30T12:15:39.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f7901237640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc090410 0x7f78fc08bfe0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78f8000c00 tx=0x7f78f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 10 client.194232.objecter ms_handle_connect 0x7f78fc090410 2023-10-30T12:15:39.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 10 client.194232.objecter resend_mon_ops 2023-10-30T12:15:39.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f801dc40 con 0x7f78fc090410 2023-10-30T12:15:39.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:39.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 10 monclient: dump: 2023-10-30T12:15:39.222 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.222 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.222 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.222 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.223 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.223 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.223 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.223 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.223 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.224 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7f78f801dda0 con 0x7f78fc090410 2023-10-30T12:15:39.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:39.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:39.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc090410 msgr2=0x7f78fc08bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc090410 0x7f78fc08bfe0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78f8000c00 tx=0x7f78f80046c0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc090410 msgr2=0x7f78fc08bfe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:39.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f78f1ffb640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc090410 0x7f78fc08bfe0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78f8000c00 tx=0x7f78f80046c0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.225 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:15:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 156, 2023-10-30T12:15:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:15:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:15:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:15:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:15:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:39.261 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:39.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 10 librados: watch_flush enter 2023-10-30T12:15:39.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 10 librados: watch_flush exit 2023-10-30T12:15:39.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 20 client.194232.objecter shutdown clearing up homeless session... 2023-10-30T12:15:39.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 10 client.194232.objecter successfully canceled tick 2023-10-30T12:15:39.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78e00392f0 msgr2=0x7f78e003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78e00392f0 0x7f78e003b7b0 secure :-1 s=READY pgs=4199 cs=0 l=1 rev1=1 crypto rx=0x7f78fc091f90 tx=0x7f78f400b040 comp rx=0 tx=0).stop 2023-10-30T12:15:39.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 10 monclient: shutdown 2023-10-30T12:15:39.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:39.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc097780 msgr2=0x7f78fc1cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 --2- 172.21.15.50:0/3383309059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78fc097780 0x7f78fc1cae10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78ec000c00 tx=0x7f78ec0046c0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f7901a38640 1 -- 172.21.15.50:0/3383309059 reap_dead start 2023-10-30T12:15:39.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f7901a38640 1 -- 172.21.15.50:0/3383309059 reap_dead start 2023-10-30T12:15:39.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 shutdown_connections 2023-10-30T12:15:39.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 >> 172.21.15.50:0/3383309059 conn(0x7f78fc0a1170 msgr2=0x7f78fc0a2570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:39.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 shutdown_connections 2023-10-30T12:15:39.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 -- 172.21.15.50:0/3383309059 wait complete. 2023-10-30T12:15:39.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.154+0000 7f790349b640 1 librados: shutdown 2023-10-30T12:15:39.267 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:15:39.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:15:39.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:15:39.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.372+0000 7fb35b30c640 10 monclient: get_monmap_and_config 2023-10-30T12:15:39.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.372+0000 7fb35b30c640 10 monclient: build_initial_monmap 2023-10-30T12:15:39.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.372+0000 7fb35b30c640 10 monclient: monmap: 2023-10-30T12:15:39.375 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:39.376 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.376 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:39.373964+0000 2023-10-30T12:15:39.376 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:39.373964+0000 2023-10-30T12:15:39.377 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:39.377 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.378 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:39.378 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:39.378 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:39.379 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:39.379 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:39.380 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.373+0000 7fb35b30c640 1 Processor -- start 2023-10-30T12:15:39.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.373+0000 7fb35b30c640 1 -- start start 2023-10-30T12:15:39.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.373+0000 7fb35b30c640 10 monclient: init 2023-10-30T12:15:39.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:39.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T12:15:39.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb354097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient(hunting): picked mon.noname-b con 0x7fb354097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:39.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb354073b10 0x7fb3540a0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient(hunting): picked mon.noname-a con 0x7fb354073b10 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:39.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb35409ccd0 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:39.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb354071ec0 con 0x7fb35409ccd0 2023-10-30T12:15:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35409bd10 con 0x7fb354073b10 2023-10-30T12:15:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35409bff0 con 0x7fb354097780 2023-10-30T12:15:39.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb35b30c640 10 monclient(hunting): authenticate will time out at 1784.557170s 2023-10-30T12:15:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb354073b10 0x7fb3540a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb354097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb354073b10 0x7fb3540a0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36028/0 (socket says 172.21.15.50:36028) 2023-10-30T12:15:39.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 1 -- 172.21.15.50:0/3205865117 learned_addr learned my addr 172.21.15.50:0/3205865117 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:39.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 10 monclient(hunting): get_auth_request con 0x7fb354073b10 auth_method 0 2023-10-30T12:15:39.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb354097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35446/0 (socket says 172.21.15.50:35446) 2023-10-30T12:15:39.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): get_auth_request con 0x7fb354097780 auth_method 0 2023-10-30T12:15:39.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient(hunting): handle_auth_done global_id 115055 payload 274 2023-10-30T12:15:39.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 1 -- 172.21.15.50:0/3205865117 >> v2:172.21.15.50:3300/0 conn(0x7fb354073b10 msgr2=0x7fb3540a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:39.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 1 --2- 172.21.15.50:0/3205865117 >> v2:172.21.15.50:3300/0 conn(0x7fb354073b10 0x7fb3540a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 1 -- 172.21.15.50:0/3205865117 >> v1:172.21.15.118:6789/0 conn(0x7fb35409ccd0 legacy=0x7fb3540a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient: _finish_hunting 0 2023-10-30T12:15:39.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 1 monclient: found mon.noname-b 2023-10-30T12:15:39.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3590a8640 1 -- 172.21.15.50:0/3205865117 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3540980a0 con 0x7fb354097780 2023-10-30T12:15:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.374+0000 7fb3588a7640 1 --2- 172.21.15.50:0/3205865117 >> v2:172.21.15.50:3300/0 conn(0x7fb354073b10 0x7fb3540a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb3590a8640 1 --2- 172.21.15.50:0/3205865117 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb354097b60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb344009d90 tx=0x7fb34402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe97c447415be28 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 1 -- 172.21.15.50:0/3205865117 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34402cb30 con 0x7fb354097780 2023-10-30T12:15:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: dump: 2023-10-30T12:15:39.408 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.409 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: _finish_auth 0 2023-10-30T12:15:39.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: _check_auth_tickets 2023-10-30T12:15:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 1 -- 172.21.15.50:0/3205865117 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34402cc90 con 0x7fb354097780 2023-10-30T12:15:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb35b30c640 5 monclient: authenticate success, global_id 115055 2023-10-30T12:15:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb35b30c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:39.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb35b30c640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb35b30c640 4 set_mon_vals no callback set 2023-10-30T12:15:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 1 -- 172.21.15.50:0/3205865117 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb344034440 con 0x7fb354097780 2023-10-30T12:15:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:39.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.375+0000 7fb353fff640 10 monclient: dump: 2023-10-30T12:15:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.414 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 10 monclient: shutdown 2023-10-30T12:15:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 -- 172.21.15.50:0/3205865117 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 msgr2=0x7fb354097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3205865117 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb354097b60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb344009d90 tx=0x7fb34402bd80 comp rx=0 tx=0).stop 2023-10-30T12:15:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 -- 172.21.15.50:0/3205865117 shutdown_connections 2023-10-30T12:15:39.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3205865117 >> v2:172.21.15.50:3300/0 conn(0x7fb354073b10 0x7fb3540a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3205865117 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb354097b60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 -- 172.21.15.50:0/3205865117 >> 172.21.15.50:0/3205865117 conn(0x7fb3540a1170 msgr2=0x7fb3540a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 -- 172.21.15.50:0/3205865117 shutdown_connections 2023-10-30T12:15:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.376+0000 7fb35b30c640 1 -- 172.21.15.50:0/3205865117 wait complete. 2023-10-30T12:15:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 10 monclient: build_initial_monmap 2023-10-30T12:15:39.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 10 monclient: monmap: 2023-10-30T12:15:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:39.417 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.418 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:39.418 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:39.418 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:39.418 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 1 librados: starting msgr at 2023-10-30T12:15:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 1 librados: starting objecter 2023-10-30T12:15:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 1 Processor -- start 2023-10-30T12:15:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 1 -- start start 2023-10-30T12:15:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 1 librados: setting wanted keys 2023-10-30T12:15:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 1 librados: calling monclient init 2023-10-30T12:15:39.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.377+0000 7fb35b30c640 10 monclient: init 2023-10-30T12:15:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:15:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 0x7fb35408e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): picked mon.noname-b con 0x7fb354073b10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb35408ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): picked mon.noname-c con 0x7fb354097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35409ccd0 0x7fb35408f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): picked mon.noname-a con 0x7fb35409ccd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb354094fc0 con 0x7fb35409ccd0 2023-10-30T12:15:39.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb354090ca0 con 0x7fb354073b10 2023-10-30T12:15:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb354092a90 con 0x7fb354097780 2023-10-30T12:15:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:39.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb35b30c640 10 monclient(hunting): authenticate will time out at 1784.561347s 2023-10-30T12:15:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb35408ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb35408ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35448/0 (socket says 172.21.15.50:35448) 2023-10-30T12:15:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 -- 172.21.15.50:0/3387309769 learned_addr learned my addr 172.21.15.50:0/3387309769 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): get_auth_request con 0x7fb354097780 auth_method 0 2023-10-30T12:15:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35409ccd0 0x7fb35408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3590a8640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 0x7fb35408e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 10 monclient(hunting): get_auth_request con 0x7fb35409ccd0 auth_method 0 2023-10-30T12:15:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3590a8640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 msgr2=0x7fb35408e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:15:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3590a8640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 msgr2=0x7fb35408e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:15:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3590a8640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 0x7fb35408e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3590a8640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 0x7fb35408e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient(hunting): handle_auth_done global_id 115058 payload 824 2023-10-30T12:15:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 msgr2=0x7fb35408e8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb354073b10 0x7fb35408e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35409ccd0 msgr2=0x7fb35408f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35409ccd0 0x7fb35408f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient: _finish_hunting 0 2023-10-30T12:15:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 monclient: found mon.noname-c 2023-10-30T12:15:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 -- 172.21.15.50:0/3387309769 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480094d0 con 0x7fb354097780 2023-10-30T12:15:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3598a9640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35409ccd0 0x7fb35408f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb3588a7640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb35408ee30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb34800e9d0 tx=0x7fb34800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb351ffb640 10 client.?.objecter ms_handle_connect 0x7fb354097780 2023-10-30T12:15:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.378+0000 7fb351ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34800ca60 con 0x7fb354097780 2023-10-30T12:15:39.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: dump: 2023-10-30T12:15:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.430 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.431 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.432 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: _finish_auth 0 2023-10-30T12:15:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:15:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34800cbc0 con 0x7fb354097780 2023-10-30T12:15:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 5 monclient: authenticate success, global_id 115058 2023-10-30T12:15:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 10 monclient: _renew_subs 2023-10-30T12:15:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb344009a70 con 0x7fb354097780 2023-10-30T12:15:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348013470 con 0x7fb354097780 2023-10-30T12:15:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 10 monclient: dump: 2023-10-30T12:15:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.434 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.436 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb3537fe640 4 set_mon_vals no callback set 2023-10-30T12:15:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 10 client.115058.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 10 monclient: _renew_subs 2023-10-30T12:15:39.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:39.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb354092390 con 0x7fb354097780 2023-10-30T12:15:39.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb35b30c640 1 librados: init done 2023-10-30T12:15:39.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.379+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb3480135d0 con 0x7fb354097780 2023-10-30T12:15:39.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb3400392f0 0x7fb34003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(464..464 src has 417..464) v4 ==== 6193+0+0 (secure 0 0 0) 0x7fb34804afe0 con 0x7fb354097780 2023-10-30T12:15:39.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 10 client.115058.objecter ms_dispatch 0x7fb354000b90 osd_map(464..464 src has 417..464) v4 2023-10-30T12:15:39.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 3 client.115058.objecter handle_osd_map got epochs [464,464] > 0 2023-10-30T12:15:39.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 3 client.115058.objecter handle_osd_map decoding full epoch 464 2023-10-30T12:15:39.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb35b30c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 20 client.115058.objecter dump_active .. 0 homeless 2023-10-30T12:15:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408afd0 0x7fb35408b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb35b30c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35406a5a0 con 0x7fb35408afd0 2023-10-30T12:15:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408afd0 0x7fb35408b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient: get_auth_request con 0x7fb35408afd0 auth_method 0 2023-10-30T12:15:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3590a8640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb3400392f0 0x7fb34003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3590a8640 10 monclient: get_auth_request con 0x7fb3400392f0 auth_method 0 2023-10-30T12:15:39.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 10 monclient(hunting): handle_auth_done global_id 115061 payload 824 2023-10-30T12:15:39.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 1 -- 172.21.15.50:0/3387309769 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3540a0cd0 con 0x7fb35408afd0 2023-10-30T12:15:39.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb3598a9640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408afd0 0x7fb35408b3b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fae0 tx=0x7fb34c001cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 10 client.115058.objecter ms_handle_connect 0x7fb35408afd0 2023-10-30T12:15:39.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.380+0000 7fb351ffb640 10 client.115058.objecter resend_mon_ops 2023-10-30T12:15:39.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0052e0 con 0x7fb35408afd0 2023-10-30T12:15:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 10 monclient: dump: 2023-10-30T12:15:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.445 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.446 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.447 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 10 client.115058.objecter ms_handle_connect 0x7fb3400392f0 2023-10-30T12:15:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb3590a8640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb3400392f0 0x7fb34003b7b0 secure :-1 s=READY pgs=4200 cs=0 l=1 rev1=1 crypto rx=0x7fb344036000 tx=0x7fb344030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb34c005440 con 0x7fb35408afd0 2023-10-30T12:15:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408afd0 msgr2=0x7fb35408b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408afd0 0x7fb35408b3b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fae0 tx=0x7fb34c001cf0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408afd0 msgr2=0x7fb35408b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.381+0000 7fb351ffb640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408afd0 0x7fb35408b3b0 secure :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fae0 tx=0x7fb34c001cf0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb35b30c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408b8f0 0x7fb3541d46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb35b30c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb354092010 con 0x7fb35408b8f0 2023-10-30T12:15:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408b8f0 0x7fb3541d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient: get_auth_request con 0x7fb35408b8f0 auth_method 0 2023-10-30T12:15:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 10 monclient(hunting): handle_auth_done global_id 115064 payload 824 2023-10-30T12:15:39.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.387+0000 7fb3598a9640 1 -- 172.21.15.50:0/3387309769 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb35404e160 con 0x7fb35408b8f0 2023-10-30T12:15:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb3598a9640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408b8f0 0x7fb3541d46b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fae0 tx=0x7fb34c001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 10 client.115058.objecter ms_handle_connect 0x7fb35408b8f0 2023-10-30T12:15:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 10 client.115058.objecter resend_mon_ops 2023-10-30T12:15:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c023070 con 0x7fb35408b8f0 2023-10-30T12:15:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:39.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 10 monclient: dump: 2023-10-30T12:15:39.453 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.453 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.453 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.455 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.455 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7fb34c003e50 con 0x7fb35408b8f0 2023-10-30T12:15:39.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:39.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408b8f0 msgr2=0x7fb3541d46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408b8f0 0x7fb3541d46b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fae0 tx=0x7fb34c001120 comp rx=0 tx=0).stop 2023-10-30T12:15:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408b8f0 msgr2=0x7fb3541d46b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:39.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb351ffb640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb35408b8f0 0x7fb3541d46b0 secure :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fae0 tx=0x7fb34c001120 comp rx=0 tx=0).stop 2023-10-30T12:15:39.457 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:15:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 156, 2023-10-30T12:15:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:15:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:15:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:39.491 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 10 librados: watch_flush enter 2023-10-30T12:15:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 10 librados: watch_flush exit 2023-10-30T12:15:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 20 client.115058.objecter shutdown clearing up homeless session... 2023-10-30T12:15:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 10 client.115058.objecter successfully canceled tick 2023-10-30T12:15:39.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb3400392f0 msgr2=0x7fb34003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb3400392f0 0x7fb34003b7b0 secure :-1 s=READY pgs=4200 cs=0 l=1 rev1=1 crypto rx=0x7fb344036000 tx=0x7fb344030000 comp rx=0 tx=0).stop 2023-10-30T12:15:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 10 monclient: shutdown 2023-10-30T12:15:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 msgr2=0x7fb35408ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb3598a9640 1 -- 172.21.15.50:0/3387309769 reap_dead start 2023-10-30T12:15:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb35408ee30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb34800e9d0 tx=0x7fb34800ee90 comp rx=0 tx=0).stop 2023-10-30T12:15:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 shutdown_connections 2023-10-30T12:15:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 --2- 172.21.15.50:0/3387309769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb354097780 0x7fb35408ee30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 >> 172.21.15.50:0/3387309769 conn(0x7fb3540a1170 msgr2=0x7fb3540952a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:39.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.388+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 shutdown_connections 2023-10-30T12:15:39.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.389+0000 7fb35b30c640 1 -- 172.21.15.50:0/3387309769 wait complete. 2023-10-30T12:15:39.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.389+0000 7fb35b30c640 1 librados: shutdown 2023-10-30T12:15:39.496 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:15:39.496 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:15:39.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:15:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.607+0000 7fd59a632640 10 monclient: get_monmap_and_config 2023-10-30T12:15:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.607+0000 7fd59a632640 10 monclient: build_initial_monmap 2023-10-30T12:15:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.607+0000 7fd59a632640 10 monclient: monmap: 2023-10-30T12:15:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:39.608194+0000 2023-10-30T12:15:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:39.608194+0000 2023-10-30T12:15:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:39.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.607+0000 7fd59a632640 1 Processor -- start 2023-10-30T12:15:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.607+0000 7fd59a632640 1 -- start start 2023-10-30T12:15:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.607+0000 7fd59a632640 10 monclient: init 2023-10-30T12:15:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T12:15:39.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd5940a2ae0 0x7fd5940a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient(hunting): picked mon.noname-c con 0x7fd5940a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:39.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd5940a2ec0 0x7fd5941356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient(hunting): picked mon.noname-a con 0x7fd5940a2ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:39.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd594135c20 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940a4190 con 0x7fd594135c20 2023-10-30T12:15:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5940a2990 con 0x7fd5940a2ec0 2023-10-30T12:15:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5940a1630 con 0x7fd5940a2ae0 2023-10-30T12:15:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd59a632640 10 monclient(hunting): authenticate will time out at 1784.791407s 2023-10-30T12:15:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd5940a2ec0 0x7fd5941356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd598bcf640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd594135c20 0x7fd59413ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47930/0 (socket says 172.21.15.50:47930) 2023-10-30T12:15:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): get_auth_request con 0x7fd5940a2ec0 auth_method 0 2023-10-30T12:15:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd598bcf640 1 -- 172.21.15.50:0/377609155 learned_addr learned my addr 172.21.15.50:0/377609155 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 1 --2- 172.21.15.50:0/377609155 >> v2:172.21.15.118:3300/0 conn(0x7fd5940a2ae0 0x7fd5940a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 10 monclient(hunting): get_auth_request con 0x7fd5940a2ae0 auth_method 0 2023-10-30T12:15:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd593fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient(hunting): handle_auth_done global_id 194256 payload 274 2023-10-30T12:15:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 1 -- 172.21.15.50:0/377609155 >> v2:172.21.15.118:3300/0 conn(0x7fd5940a2ae0 msgr2=0x7fd5940a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 1 --2- 172.21.15.50:0/377609155 >> v2:172.21.15.118:3300/0 conn(0x7fd5940a2ae0 0x7fd5940a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 1 -- 172.21.15.50:0/377609155 >> v1:172.21.15.50:6789/0 conn(0x7fd594135c20 legacy=0x7fd59413ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient: _finish_hunting 0 2023-10-30T12:15:39.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 1 monclient: found mon.noname-a 2023-10-30T12:15:39.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:39.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd5937fe640 1 -- 172.21.15.50:0/377609155 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59409a2e0 con 0x7fd5940a2ec0 2023-10-30T12:15:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd592ffd640 1 -- 172.21.15.50:0/377609155 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691852449 0 0) 0x7fd58c001490 con 0x7fd594135c20 2023-10-30T12:15:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.608+0000 7fd592ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:15:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd5937fe640 1 --2- 172.21.15.50:0/377609155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ec0 0x7fd5941356e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5940a3c80 tx=0x7fd57c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42206067aedd757 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 1 -- 172.21.15.50:0/377609155 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031710 con 0x7fd5940a2ec0 2023-10-30T12:15:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: dump: 2023-10-30T12:15:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.630 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: _finish_auth 0 2023-10-30T12:15:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 1 -- 172.21.15.50:0/377609155 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c02f050 con 0x7fd5940a2ec0 2023-10-30T12:15:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd59a632640 5 monclient: authenticate success, global_id 194256 2023-10-30T12:15:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd59a632640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd59a632640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd59a632640 4 set_mon_vals no callback set 2023-10-30T12:15:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 1 -- 172.21.15.50:0/377609155 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031a90 con 0x7fd5940a2ec0 2023-10-30T12:15:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.609+0000 7fd592ffd640 10 monclient: dump: 2023-10-30T12:15:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.632 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.634 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.634 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 10 monclient: shutdown 2023-10-30T12:15:39.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 -- 172.21.15.50:0/377609155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ec0 msgr2=0x7fd5941356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 --2- 172.21.15.50:0/377609155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ec0 0x7fd5941356e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5940a3c80 tx=0x7fd57c02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 -- 172.21.15.50:0/377609155 shutdown_connections 2023-10-30T12:15:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 --2- 172.21.15.50:0/377609155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ec0 0x7fd5941356e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 --2- 172.21.15.50:0/377609155 >> v2:172.21.15.118:3300/0 conn(0x7fd5940a2ae0 0x7fd5940a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 -- 172.21.15.50:0/377609155 >> 172.21.15.50:0/377609155 conn(0x7fd5940a80b0 msgr2=0x7fd5940a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:39.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 -- 172.21.15.50:0/377609155 shutdown_connections 2023-10-30T12:15:39.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 1 -- 172.21.15.50:0/377609155 wait complete. 2023-10-30T12:15:39.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 10 monclient: build_initial_monmap 2023-10-30T12:15:39.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.610+0000 7fd59a632640 10 monclient: monmap: 2023-10-30T12:15:39.636 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:39.638 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:39.638 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:39.638 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 librados: starting msgr at 2023-10-30T12:15:39.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 librados: starting objecter 2023-10-30T12:15:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 Processor -- start 2023-10-30T12:15:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 -- start start 2023-10-30T12:15:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 librados: setting wanted keys 2023-10-30T12:15:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 librados: calling monclient init 2023-10-30T12:15:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient: init 2023-10-30T12:15:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:15:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ae0 0x7fd5941ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): picked mon.noname-a con 0x7fd5940a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5940a2ec0 0x7fd594132570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): picked mon.noname-b con 0x7fd5940a2ec0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd594135c20 0x7fd594132ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): picked mon.noname-c con 0x7fd594135c20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5940a0a20 con 0x7fd5940a2ae0 2023-10-30T12:15:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594120980 con 0x7fd5940a2ec0 2023-10-30T12:15:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd594120c60 con 0x7fd594135c20 2023-10-30T12:15:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.611+0000 7fd59a632640 10 monclient(hunting): authenticate will time out at 1784.794933s 2023-10-30T12:15:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ae0 0x7fd5941ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd594135c20 0x7fd594132ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ae0 0x7fd5941ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36052/0 (socket says 172.21.15.50:36052) 2023-10-30T12:15:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 -- 172.21.15.50:0/1013534814 learned_addr learned my addr 172.21.15.50:0/1013534814 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): get_auth_request con 0x7fd5940a2ae0 auth_method 0 2023-10-30T12:15:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 10 monclient(hunting): get_auth_request con 0x7fd594135c20 auth_method 0 2023-10-30T12:15:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5940a2ec0 0x7fd594132570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd598bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 10 monclient(hunting): get_auth_request con 0x7fd5940a2ec0 auth_method 0 2023-10-30T12:15:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient(hunting): handle_auth_done global_id 194262 payload 824 2023-10-30T12:15:39.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd594135c20 msgr2=0x7fd594132ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd594135c20 0x7fd594132ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5940a2ec0 msgr2=0x7fd594132570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5940a2ec0 0x7fd594132570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:39.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 monclient: found mon.noname-a 2023-10-30T12:15:39.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:39.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 -- 172.21.15.50:0/1013534814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c009400 con 0x7fd5940a2ae0 2023-10-30T12:15:39.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd593fff640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ae0 0x7fd5941ca240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd58400e9b0 tx=0x7fd58400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 client.?.objecter ms_handle_connect 0x7fd5940a2ae0 2023-10-30T12:15:39.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:39.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584009670 con 0x7fd5940a2ae0 2023-10-30T12:15:39.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:39.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: dump: 2023-10-30T12:15:39.650 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.650 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.651 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.651 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.651 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.651 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.651 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.652 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.652 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: _finish_auth 0 2023-10-30T12:15:39.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:15:39.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58400c9a0 con 0x7fd5940a2ae0 2023-10-30T12:15:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 5 monclient: authenticate success, global_id 194262 2023-10-30T12:15:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58400cd50 con 0x7fd5940a2ae0 2023-10-30T12:15:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd590ff9640 10 monclient: dump: 2023-10-30T12:15:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.655 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.656 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 10 monclient: _renew_subs 2023-10-30T12:15:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd5927fc640 4 set_mon_vals no callback set 2023-10-30T12:15:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd594120f40 con 0x7fd5940a2ae0 2023-10-30T12:15:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 10 client.194262.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 10 monclient: _renew_subs 2023-10-30T12:15:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59409fee0 con 0x7fd5940a2ae0 2023-10-30T12:15:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.612+0000 7fd59a632640 1 librados: init done 2023-10-30T12:15:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd59a632640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd59a632640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e0a0 0x7fd59411e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd584013470 con 0x7fd5940a2ae0 2023-10-30T12:15:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd59a632640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594073ff0 con 0x7fd59411e0a0 2023-10-30T12:15:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd590ff9640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5740392f0 0x7fd57403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e0a0 0x7fd59411e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(465..465 src has 417..465) v4 ==== 6193+0+0 (secure 0 0 0) 0x7fd58404afa0 con 0x7fd5940a2ae0 2023-10-30T12:15:39.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd590ff9640 10 client.194262.objecter ms_dispatch 0x7fd594000b90 osd_map(465..465 src has 417..465) v4 2023-10-30T12:15:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd590ff9640 3 client.194262.objecter handle_osd_map got epochs [465,465] > 0 2023-10-30T12:15:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd590ff9640 3 client.194262.objecter handle_osd_map decoding full epoch 465 2023-10-30T12:15:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 10 monclient: get_auth_request con 0x7fd59411e0a0 auth_method 0 2023-10-30T12:15:39.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd590ff9640 20 client.194262.objecter dump_active .. 0 homeless 2023-10-30T12:15:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd598bcf640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5740392f0 0x7fd57403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.613+0000 7fd598bcf640 10 monclient: get_auth_request con 0x7fd5740392f0 auth_method 0 2023-10-30T12:15:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd5937fe640 10 monclient(hunting): handle_auth_done global_id 204109 payload 824 2023-10-30T12:15:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd5937fe640 1 -- 172.21.15.50:0/1013534814 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd59409a5c0 con 0x7fd59411e0a0 2023-10-30T12:15:39.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 client.194262.objecter ms_handle_connect 0x7fd59411e0a0 2023-10-30T12:15:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 client.194262.objecter resend_mon_ops 2023-10-30T12:15:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e0a0 0x7fd59411e480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd5940a54d0 tx=0x7fd57c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 client.194262.objecter ms_handle_connect 0x7fd5740392f0 2023-10-30T12:15:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd598bcf640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5740392f0 0x7fd57403b7b0 secure :-1 s=READY pgs=4202 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0012e0 tx=0x7fd58c007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c003a70 con 0x7fd59411e0a0 2023-10-30T12:15:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 monclient: dump: 2023-10-30T12:15:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.665 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.666 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fd57c042070 con 0x7fd59411e0a0 2023-10-30T12:15:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:39.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e0a0 msgr2=0x7fd59411e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e0a0 0x7fd59411e480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd5940a54d0 tx=0x7fd57c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e0a0 msgr2=0x7fd59411e480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.614+0000 7fd590ff9640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e0a0 0x7fd59411e480 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd59a632640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd59a632640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd59a632640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594073a20 con 0x7fd59411e9c0 2023-10-30T12:15:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd5937fe640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 msgr2=0x7fd59409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:15:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd5937fe640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 msgr2=0x7fd59409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:15:39.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.621+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:39.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient: get_auth_request con 0x7fd59411e9c0 auth_method 0 2023-10-30T12:15:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 10 monclient(hunting): handle_auth_done global_id 204112 payload 824 2023-10-30T12:15:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 1 -- 172.21.15.50:0/1013534814 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd594099b40 con 0x7fd59411e9c0 2023-10-30T12:15:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd5937fe640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd57c000c00 tx=0x7fd57c031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 10 client.194262.objecter ms_handle_connect 0x7fd59411e9c0 2023-10-30T12:15:39.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 10 client.194262.objecter resend_mon_ops 2023-10-30T12:15:39.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031ee0 con 0x7fd59411e9c0 2023-10-30T12:15:39.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:39.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 10 monclient: dump: 2023-10-30T12:15:39.828 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.828 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.828 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.828 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.828 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.828 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.829 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.829 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.829 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.829 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fd57c051650 con 0x7fd59411e9c0 2023-10-30T12:15:39.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:39.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:39.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 msgr2=0x7fd59409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd57c000c00 tx=0x7fd57c031000 comp rx=0 tx=0).stop 2023-10-30T12:15:39.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 msgr2=0x7fd59409cd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:39.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd590ff9640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd59411e9c0 0x7fd59409cd60 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd57c000c00 tx=0x7fd57c031000 comp rx=0 tx=0).stop 2023-10-30T12:15:39.831 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:39.831 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:39.832 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:39.832 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:15:39.832 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 156, 2023-10-30T12:15:39.832 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:39.833 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:39.833 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:39.833 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:39.833 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.833 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:15:39.834 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:39.834 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:39.834 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:39.834 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.834 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.835 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.835 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.835 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.835 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.835 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.835 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.836 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.836 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.836 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.836 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:39.837 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:39.837 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.837 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.837 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.837 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.838 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.838 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.838 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.838 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.839 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.839 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.839 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:39.839 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:39.840 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:39.840 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:39.840 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:39.840 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:39.840 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:39.840 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:39.841 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:39.841 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:39.841 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:39.841 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:39.841 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:39.841 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:39.842 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:39.842 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:39.842 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:39.842 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:39.842 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:39.843 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:39.843 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:39.843 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:39.843 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:39.843 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:39.843 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:39.844 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:39.844 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:39.844 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.844 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:39.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.845 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:39.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.845 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:39.845 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:39.845 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.846 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.846 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.846 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:39.846 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.847 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:39.847 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.848 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.848 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:39.848 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:39.848 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.849 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.849 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.849 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:39.850 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:39.850 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.850 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.850 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.851 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:39.851 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.851 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:39.852 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.852 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:39.853 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:39.853 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.853 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.853 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.854 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:39.854 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:39.854 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:39.854 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:39.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.855 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:39.855 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:39.855 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:39.856 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:39.856 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:39.856 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.857 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:39.857 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:39.857 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:39.857 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:39.858 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:39.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.858 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.858 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:39.859 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:39.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.859 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.859 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.859 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:39.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.860 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:39.860 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:39.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:39.860 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:39.860 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:39.861 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:39.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:39.861 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:39.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:39.861 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:39.862 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:39.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 10 librados: watch_flush enter 2023-10-30T12:15:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 10 librados: watch_flush exit 2023-10-30T12:15:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 20 client.194262.objecter shutdown clearing up homeless session... 2023-10-30T12:15:39.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 10 client.194262.objecter successfully canceled tick 2023-10-30T12:15:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5740392f0 msgr2=0x7fd57403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5740392f0 0x7fd57403b7b0 secure :-1 s=READY pgs=4202 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0012e0 tx=0x7fd58c007e90 comp rx=0 tx=0).stop 2023-10-30T12:15:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 10 monclient: shutdown 2023-10-30T12:15:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ae0 msgr2=0x7fd5941ca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 --2- 172.21.15.50:0/1013534814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5940a2ae0 0x7fd5941ca240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd58400e9b0 tx=0x7fd58400ee70 comp rx=0 tx=0).stop 2023-10-30T12:15:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd598bcf640 1 -- 172.21.15.50:0/1013534814 reap_dead start 2023-10-30T12:15:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd598bcf640 1 -- 172.21.15.50:0/1013534814 reap_dead start 2023-10-30T12:15:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 shutdown_connections 2023-10-30T12:15:39.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 >> 172.21.15.50:0/1013534814 conn(0x7fd5940a80b0 msgr2=0x7fd594133fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 shutdown_connections 2023-10-30T12:15:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.823+0000 7fd59a632640 1 -- 172.21.15.50:0/1013534814 wait complete. 2023-10-30T12:15:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.824+0000 7fd59a632640 1 librados: shutdown 2023-10-30T12:15:39.870 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:15:39.870 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:15:39.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:15:39.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.975+0000 7f8fa7c5d640 10 monclient: get_monmap_and_config 2023-10-30T12:15:39.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.975+0000 7f8fa7c5d640 10 monclient: build_initial_monmap 2023-10-30T12:15:39.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.975+0000 7f8fa7c5d640 10 monclient: monmap: 2023-10-30T12:15:39.978 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:39.978 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.978 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:39.976908+0000 2023-10-30T12:15:39.979 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:39.976908+0000 2023-10-30T12:15:39.979 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:39.979 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.979 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:39.979 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:39.980 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:39.980 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:39.980 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:39.980 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.976+0000 7f8fa7c5d640 1 Processor -- start 2023-10-30T12:15:39.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.976+0000 7f8fa7c5d640 1 -- start start 2023-10-30T12:15:39.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.976+0000 7f8fa7c5d640 10 monclient: init 2023-10-30T12:15:39.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.976+0000 7f8fa7c5d640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:39.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-10-30T12:15:39.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8fa00a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:39.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fa0099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:39.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8fa0132db0 0x7f8fa013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:39.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa0132db0 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:39.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00a4f40 con 0x7f8fa00a4b60 2023-10-30T12:15:39.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00a4150 con 0x7f8fa0099a40 2023-10-30T12:15:39.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:39.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00a3d70 con 0x7f8fa0132db0 2023-10-30T12:15:39.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:39.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa7c5d640 10 monclient(hunting): authenticate will time out at 1785.160090s 2023-10-30T12:15:39.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8fa0132db0 0x7f8fa013e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:39.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa59f9640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8fa00a4b60 0x7f8fa01326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47932/0 (socket says 172.21.15.50:47932) 2023-10-30T12:15:39.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa59f9640 1 -- 172.21.15.50:0/2730329720 learned_addr learned my addr 172.21.15.50:0/2730329720 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): get_auth_request con 0x7f8fa0132db0 auth_method 0 2023-10-30T12:15:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient(hunting): handle_auth_done global_id 194265 payload 274 2023-10-30T12:15:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 1 -- 172.21.15.50:0/2730329720 >> v1:172.21.15.118:6789/0 conn(0x7f8fa0099a40 legacy=0x7f8fa013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 1 -- 172.21.15.50:0/2730329720 >> v1:172.21.15.50:6789/0 conn(0x7f8fa00a4b60 legacy=0x7f8fa01326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient: _finish_hunting 0 2023-10-30T12:15:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 1 monclient: found mon.noname-a 2023-10-30T12:15:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa61fa640 1 -- 172.21.15.50:0/2730329720 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa009a440 con 0x7f8fa0132db0 2023-10-30T12:15:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa49f7640 1 -- 172.21.15.50:0/2730329720 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097128631 0 0) 0x7f8fa00a4f40 con 0x7f8fa00a4b60 2023-10-30T12:15:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.977+0000 7f8fa49f7640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:15:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa61fa640 1 --2- 172.21.15.50:0/2730329720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa0132db0 0x7f8fa013e1b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009d90 tx=0x7f8f9002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82fa0369cc403f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 1 -- 172.21.15.50:0/2730329720 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9002cbd0 con 0x7f8fa0132db0 2023-10-30T12:15:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: dump: 2023-10-30T12:15:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.995 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.995 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.995 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:39.995 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:39.996 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: _finish_auth 0 2023-10-30T12:15:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: _check_auth_tickets 2023-10-30T12:15:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 1 -- 172.21.15.50:0/2730329720 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9002cd30 con 0x7f8fa0132db0 2023-10-30T12:15:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 5 monclient: authenticate success, global_id 194265 2023-10-30T12:15:39.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:39.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:39.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:39.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 1 -- 172.21.15.50:0/2730329720 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9002cea0 con 0x7f8fa0132db0 2023-10-30T12:15:39.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 4 set_mon_vals no callback set 2023-10-30T12:15:39.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa49f7640 10 monclient: dump: 2023-10-30T12:15:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.999 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:39.999 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:39.999 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:39.999 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:39.999 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:40.000 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 10 monclient: shutdown 2023-10-30T12:15:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/2730329720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa0132db0 msgr2=0x7f8fa013e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/2730329720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa0132db0 0x7f8fa013e1b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009d90 tx=0x7f8f9002be20 comp rx=0 tx=0).stop 2023-10-30T12:15:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/2730329720 shutdown_connections 2023-10-30T12:15:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/2730329720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa0132db0 0x7f8fa013e1b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.978+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/2730329720 >> 172.21.15.50:0/2730329720 conn(0x7f8fa00a8060 msgr2=0x7f8fa00a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/2730329720 shutdown_connections 2023-10-30T12:15:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/2730329720 wait complete. 2023-10-30T12:15:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 10 monclient: build_initial_monmap 2023-10-30T12:15:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 10 monclient: monmap: 2023-10-30T12:15:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:40.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:40.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 librados: starting msgr at 2023-10-30T12:15:40.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 librados: starting objecter 2023-10-30T12:15:40.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 Processor -- start 2023-10-30T12:15:40.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 -- start start 2023-10-30T12:15:40.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 librados: setting wanted keys 2023-10-30T12:15:40.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 1 librados: calling monclient init 2023-10-30T12:15:40.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.979+0000 7f8fa7c5d640 10 monclient: init 2023-10-30T12:15:40.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:40.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:15:40.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa00a4b60 0x7f8fa0136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:40.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa00a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:40.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fa0132db0 0x7f8fa0136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:40.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa0132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:40.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fa01372d0 0x7f8fa01ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:40.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa01372d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa013cb50 con 0x7f8fa00a4b60 2023-10-30T12:15:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa0139c50 con 0x7f8fa01372d0 2023-10-30T12:15:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:40.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00a18d0 con 0x7f8fa0132db0 2023-10-30T12:15:40.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:40.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa7c5d640 10 monclient(hunting): authenticate will time out at 1785.163354s 2023-10-30T12:15:40.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa00a4b60 0x7f8fa0136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:40.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fa0132db0 0x7f8fa0136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa00a4b60 0x7f8fa0136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36064/0 (socket says 172.21.15.50:36064) 2023-10-30T12:15:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 -- 172.21.15.50:0/956049047 learned_addr learned my addr 172.21.15.50:0/956049047 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 10 monclient(hunting): get_auth_request con 0x7f8fa0132db0 auth_method 0 2023-10-30T12:15:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): get_auth_request con 0x7f8fa00a4b60 auth_method 0 2023-10-30T12:15:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:40.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:40.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa51f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:40.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:40.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:40.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:40.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa61fa640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fa01372d0 0x7f8fa01ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:40.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient(hunting): handle_auth_done global_id 194271 payload 824 2023-10-30T12:15:40.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 -- 172.21.15.50:0/956049047 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fa0132db0 msgr2=0x7f8fa0136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:40.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fa0132db0 0x7f8fa0136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:40.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 -- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fa01372d0 msgr2=0x7f8fa01ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fa01372d0 0x7f8fa01ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient: _finish_hunting 0 2023-10-30T12:15:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 monclient: found mon.noname-a 2023-10-30T12:15:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 -- 172.21.15.50:0/956049047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f90009a70 con 0x7f8fa00a4b60 2023-10-30T12:15:40.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa61fa640 10 monclient: get_auth_request con 0x7f8fa01372d0 auth_method 0 2023-10-30T12:15:40.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa61fa640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fa01372d0 0x7f8fa01ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:15:40.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8fa59f9640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa00a4b60 0x7f8fa0136850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c4d0 tx=0x7f8f9800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:40.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8f9e7fc640 10 client.?.objecter ms_handle_connect 0x7f8fa00a4b60 2023-10-30T12:15:40.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.980+0000 7f8f9e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:40.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 1 -- 172.21.15.50:0/956049047 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f980074e0 con 0x7f8fa00a4b60 2023-10-30T12:15:40.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:40.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:40.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: dump: 2023-10-30T12:15:40.015 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:40.015 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:40.017 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:15:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:15:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 5 monclient: authenticate success, global_id 194271 2023-10-30T12:15:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 1 -- 172.21.15.50:0/956049047 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f98007640 con 0x7f8fa00a4b60 2023-10-30T12:15:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 10 monclient: _renew_subs 2023-10-30T12:15:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa013da50 con 0x7f8fa00a4b60 2023-10-30T12:15:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 1 -- 172.21.15.50:0/956049047 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98004a20 con 0x7f8fa00a4b60 2023-10-30T12:15:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9e7fc640 10 monclient: dump: 2023-10-30T12:15:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:40.022 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:40.022 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:40.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8f9ffff640 4 set_mon_vals no callback set 2023-10-30T12:15:40.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 10 client.194271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:40.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 10 monclient: _renew_subs 2023-10-30T12:15:40.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:40.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa0138490 con 0x7f8fa00a4b60 2023-10-30T12:15:40.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.981+0000 7f8fa7c5d640 1 librados: init done 2023-10-30T12:15:40.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8f9e7fc640 1 -- 172.21.15.50:0/956049047 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8f98004b80 con 0x7f8fa00a4b60 2023-10-30T12:15:40.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8fa7c5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:40.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8fa7c5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:40.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8fa7c5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:40.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa004e160 con 0x7f8fa00a4b60 2023-10-30T12:15:40.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8f9e7fc640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f8c0392f0 0x7f8f8c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:40.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8f9e7fc640 1 -- 172.21.15.50:0/956049047 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(465..465 src has 417..465) v4 ==== 6193+0+0 (secure 0 0 0) 0x7f8f9804a0e0 con 0x7f8fa00a4b60 2023-10-30T12:15:40.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8f9e7fc640 10 client.194271.objecter ms_dispatch 0x7f8fa0000b90 osd_map(465..465 src has 417..465) v4 2023-10-30T12:15:40.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8f9e7fc640 3 client.194271.objecter handle_osd_map got epochs [465,465] > 0 2023-10-30T12:15:40.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8f9e7fc640 3 client.194271.objecter handle_osd_map decoding full epoch 465 2023-10-30T12:15:40.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8f9e7fc640 20 client.194271.objecter dump_active .. 0 homeless 2023-10-30T12:15:40.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8fa51f8640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f8c0392f0 0x7f8f8c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:40.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.982+0000 7f8fa51f8640 10 monclient: get_auth_request con 0x7f8f8c0392f0 auth_method 0 2023-10-30T12:15:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.983+0000 7f8f9e7fc640 10 client.194271.objecter ms_handle_connect 0x7f8f8c0392f0 2023-10-30T12:15:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.983+0000 7f8fa51f8640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f8c0392f0 0x7f8f8c03b7b0 secure :-1 s=READY pgs=4203 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00a0e20 tx=0x7f8f94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.987+0000 7f8f9e7fc640 1 -- 172.21.15.50:0/956049047 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8f9801ab40 con 0x7f8fa00a4b60 2023-10-30T12:15:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.987+0000 7f8f9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:39.987+0000 7f8f9e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:40.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.229+0000 7f8fa7c5d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:15:40.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.229+0000 7f8fa7c5d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:15:40.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.229+0000 7f8fa7c5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:40.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.229+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8fa00745c0 con 0x7f8fa00a4b60 2023-10-30T12:15:40.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.230+0000 7f8f9e7fc640 1 -- 172.21.15.50:0/956049047 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f8f98011e20 con 0x7f8fa00a4b60 2023-10-30T12:15:40.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.230+0000 7f8f9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:15:40.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.230+0000 7f8f9e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:40.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=11,kv=9,logm=48} crc {auth=3682685307,config=3140514009,health=3742334251,kv=2905431493,logm=2966042887}) 2023-10-30T12:15:40.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=8,logm=52} crc {auth=1792973492,config=3140514009,health=3603891712,kv=474600764,logm=3337489612}) 2023-10-30T12:15:40.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=11,kv=9,logm=48} crc {auth=3682685307,config=3140514009,health=3742334251,kv=2905431493,logm=2966042887}) 2023-10-30T12:15:40.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.231+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=1,health=11,kv=8,logm=50} crc {auth=3819119062,config=1228700967,health=3772018171,kv=4037166911,logm=4294084500}) 2023-10-30T12:15:40.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515138255}) 2023-10-30T12:15:40.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=205958818}) 2023-10-30T12:15:40.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515138255}) 2023-10-30T12:15:40.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.232+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2060482626}) 2023-10-30T12:15:40.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.233+0000 7f8fa7c5d640 10 librados: watch_flush enter 2023-10-30T12:15:40.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.233+0000 7f8fa7c5d640 10 librados: watch_flush exit 2023-10-30T12:15:40.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.233+0000 7f8fa7c5d640 20 client.194271.objecter shutdown clearing up homeless session... 2023-10-30T12:15:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.233+0000 7f8fa7c5d640 10 client.194271.objecter successfully canceled tick 2023-10-30T12:15:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f8c0392f0 msgr2=0x7f8f8c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f8c0392f0 0x7f8f8c03b7b0 secure :-1 s=READY pgs=4203 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00a0e20 tx=0x7f8f94009040 comp rx=0 tx=0).stop 2023-10-30T12:15:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 10 monclient: shutdown 2023-10-30T12:15:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa00a4b60 msgr2=0x7f8fa0136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa00a4b60 0x7f8fa0136850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c4d0 tx=0x7f8f9800c990 comp rx=0 tx=0).stop 2023-10-30T12:15:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 shutdown_connections 2023-10-30T12:15:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f8c0392f0 0x7f8f8c03b7b0 unknown :-1 s=CLOSED pgs=4203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8fa01372d0 0x7f8fa01ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fa0132db0 0x7f8fa0136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 --2- 172.21.15.50:0/956049047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fa00a4b60 0x7f8fa0136850 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 >> 172.21.15.50:0/956049047 conn(0x7f8fa00a8060 msgr2=0x7f8fa00a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 shutdown_connections 2023-10-30T12:15:40.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 -- 172.21.15.50:0/956049047 wait complete. 2023-10-30T12:15:40.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446098091}) 2023-10-30T12:15:40.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1194175279}) 2023-10-30T12:15:40.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446098091}) 2023-10-30T12:15:40.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=66684010}) 2023-10-30T12:15:40.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:40.234+0000 7f8fa7c5d640 1 librados: shutdown 2023-10-30T12:15:40.255 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.255 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555134873}) 2023-10-30T12:15:40.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076985852}) 2023-10-30T12:15:40.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555134873}) 2023-10-30T12:15:40.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.235+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660271815}) 2023-10-30T12:15:40.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758208830}) 2023-10-30T12:15:40.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404483831}) 2023-10-30T12:15:40.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758208830}) 2023-10-30T12:15:40.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.237+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110124366}) 2023-10-30T12:15:40.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544560324}) 2023-10-30T12:15:40.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=313275204}) 2023-10-30T12:15:40.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544560324}) 2023-10-30T12:15:40.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.238+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1424387980}) 2023-10-30T12:15:40.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1623499136}) 2023-10-30T12:15:40.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628617154}) 2023-10-30T12:15:40.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1623499136}) 2023-10-30T12:15:40.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.239+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1181747567}) 2023-10-30T12:15:40.262 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693067845}) 2023-10-30T12:15:40.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949140435}) 2023-10-30T12:15:40.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693067845}) 2023-10-30T12:15:40.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.240+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84444328}) 2023-10-30T12:15:40.263 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310353642}) 2023-10-30T12:15:40.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641424905}) 2023-10-30T12:15:40.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.264 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310353642}) 2023-10-30T12:15:40.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.241+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744498203}) 2023-10-30T12:15:40.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347288964}) 2023-10-30T12:15:40.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3085892130}) 2023-10-30T12:15:40.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.265 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347288964}) 2023-10-30T12:15:40.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.242+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380953094}) 2023-10-30T12:15:40.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963814006}) 2023-10-30T12:15:40.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1282542978}) 2023-10-30T12:15:40.266 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963814006}) 2023-10-30T12:15:40.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46088520}) 2023-10-30T12:15:40.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.243+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584023243}) 2023-10-30T12:15:40.267 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3957363480}) 2023-10-30T12:15:40.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584023243}) 2023-10-30T12:15:40.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2928197811}) 2023-10-30T12:15:40.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.268 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2803266930}) 2023-10-30T12:15:40.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193842558}) 2023-10-30T12:15:40.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2803266930}) 2023-10-30T12:15:40.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.244+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=177863393}) 2023-10-30T12:15:40.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.269 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930275993}) 2023-10-30T12:15:40.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45798906}) 2023-10-30T12:15:40.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930275993}) 2023-10-30T12:15:40.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.245+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239629432}) 2023-10-30T12:15:40.270 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526512720}) 2023-10-30T12:15:40.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088495582}) 2023-10-30T12:15:40.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526512720}) 2023-10-30T12:15:40.271 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.246+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864156465}) 2023-10-30T12:15:40.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387720893}) 2023-10-30T12:15:40.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837833144}) 2023-10-30T12:15:40.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387720893}) 2023-10-30T12:15:40.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.247+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1998685435}) 2023-10-30T12:15:40.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142039000}) 2023-10-30T12:15:40.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619583606}) 2023-10-30T12:15:40.273 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.274 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142039000}) 2023-10-30T12:15:40.274 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.248+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=737380044}) 2023-10-30T12:15:40.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:15:40.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745651435}) 2023-10-30T12:15:40.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2282082121}) 2023-10-30T12:15:40.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745651435}) 2023-10-30T12:15:40.312 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.249+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=117507599}) 2023-10-30T12:15:40.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521739864}) 2023-10-30T12:15:40.313 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094322044}) 2023-10-30T12:15:40.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521739864}) 2023-10-30T12:15:40.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565087587}) 2023-10-30T12:15:40.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3300628517}) 2023-10-30T12:15:40.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2261880177}) 2023-10-30T12:15:40.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3300628517}) 2023-10-30T12:15:40.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.250+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=81047764}) 2023-10-30T12:15:40.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044781916}) 2023-10-30T12:15:40.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45515295}) 2023-10-30T12:15:40.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044781916}) 2023-10-30T12:15:40.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.251+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568326955}) 2023-10-30T12:15:40.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276467903}) 2023-10-30T12:15:40.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2396358633}) 2023-10-30T12:15:40.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276467903}) 2023-10-30T12:15:40.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.252+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2709572553}) 2023-10-30T12:15:40.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031539104}) 2023-10-30T12:15:40.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2120892250}) 2023-10-30T12:15:40.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031539104}) 2023-10-30T12:15:40.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.253+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817527462}) 2023-10-30T12:15:40.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448280800}) 2023-10-30T12:15:40.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568119938}) 2023-10-30T12:15:40.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448280800}) 2023-10-30T12:15:40.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.254+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074485090}) 2023-10-30T12:15:40.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222585204}) 2023-10-30T12:15:40.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071582684}) 2023-10-30T12:15:40.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222585204}) 2023-10-30T12:15:40.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910120094}) 2023-10-30T12:15:40.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553394310}) 2023-10-30T12:15:40.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467359577}) 2023-10-30T12:15:40.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553394310}) 2023-10-30T12:15:40.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.255+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2762889766}) 2023-10-30T12:15:40.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603939102}) 2023-10-30T12:15:40.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3115546273}) 2023-10-30T12:15:40.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603939102}) 2023-10-30T12:15:40.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.256+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458289902}) 2023-10-30T12:15:40.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738978289}) 2023-10-30T12:15:40.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314798518}) 2023-10-30T12:15:40.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738978289}) 2023-10-30T12:15:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2739221975}) 2023-10-30T12:15:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.257+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820990484}) 2023-10-30T12:15:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818996608}) 2023-10-30T12:15:40.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820990484}) 2023-10-30T12:15:40.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613800113}) 2023-10-30T12:15:40.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420955797}) 2023-10-30T12:15:40.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168899672}) 2023-10-30T12:15:40.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420955797}) 2023-10-30T12:15:40.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.258+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317601753}) 2023-10-30T12:15:40.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273738761}) 2023-10-30T12:15:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=507199771}) 2023-10-30T12:15:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273738761}) 2023-10-30T12:15:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.259+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2769457066}) 2023-10-30T12:15:40.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480677506}) 2023-10-30T12:15:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2415682531}) 2023-10-30T12:15:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480677506}) 2023-10-30T12:15:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.260+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=71255198}) 2023-10-30T12:15:40.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647767269}) 2023-10-30T12:15:40.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2141613929}) 2023-10-30T12:15:40.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647767269}) 2023-10-30T12:15:40.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1724616798}) 2023-10-30T12:15:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1139400836}) 2023-10-30T12:15:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=661288740}) 2023-10-30T12:15:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1139400836}) 2023-10-30T12:15:40.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.261+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531254993}) 2023-10-30T12:15:40.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964615886}) 2023-10-30T12:15:40.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255290950}) 2023-10-30T12:15:40.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964615886}) 2023-10-30T12:15:40.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.262+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751376739}) 2023-10-30T12:15:40.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467387522}) 2023-10-30T12:15:40.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2324636018}) 2023-10-30T12:15:40.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467387522}) 2023-10-30T12:15:40.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.263+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7848790}) 2023-10-30T12:15:40.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935377066}) 2023-10-30T12:15:40.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3405830864}) 2023-10-30T12:15:40.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=935377066}) 2023-10-30T12:15:40.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013182878}) 2023-10-30T12:15:40.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1902887099}) 2023-10-30T12:15:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=401436852}) 2023-10-30T12:15:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.264+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1902887099}) 2023-10-30T12:15:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322971329}) 2023-10-30T12:15:40.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650490800}) 2023-10-30T12:15:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607368777}) 2023-10-30T12:15:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650490800}) 2023-10-30T12:15:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.265+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1870219972}) 2023-10-30T12:15:40.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485580215}) 2023-10-30T12:15:40.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2104736160}) 2023-10-30T12:15:40.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485580215}) 2023-10-30T12:15:40.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.266+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522011934}) 2023-10-30T12:15:40.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764857456}) 2023-10-30T12:15:40.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3870878614}) 2023-10-30T12:15:40.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764857456}) 2023-10-30T12:15:40.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.267+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80963303}) 2023-10-30T12:15:40.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597036183}) 2023-10-30T12:15:40.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565703434}) 2023-10-30T12:15:40.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597036183}) 2023-10-30T12:15:40.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93153614}) 2023-10-30T12:15:40.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133979269}) 2023-10-30T12:15:40.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911901643}) 2023-10-30T12:15:40.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133979269}) 2023-10-30T12:15:40.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.268+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4051228300}) 2023-10-30T12:15:40.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223699820}) 2023-10-30T12:15:40.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=868286399}) 2023-10-30T12:15:40.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223699820}) 2023-10-30T12:15:40.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.269+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718108372}) 2023-10-30T12:15:40.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951140278}) 2023-10-30T12:15:40.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962538057}) 2023-10-30T12:15:40.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951140278}) 2023-10-30T12:15:40.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.270+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1669374453}) 2023-10-30T12:15:40.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3266988894}) 2023-10-30T12:15:40.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=413194141}) 2023-10-30T12:15:40.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3266988894}) 2023-10-30T12:15:40.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.271+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1138545045}) 2023-10-30T12:15:40.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590141059}) 2023-10-30T12:15:40.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809042747}) 2023-10-30T12:15:40.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590141059}) 2023-10-30T12:15:40.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746815730}) 2023-10-30T12:15:40.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967452346}) 2023-10-30T12:15:40.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=873885510}) 2023-10-30T12:15:40.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967452346}) 2023-10-30T12:15:40.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.272+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172533642}) 2023-10-30T12:15:40.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654654869}) 2023-10-30T12:15:40.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223162254}) 2023-10-30T12:15:40.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654654869}) 2023-10-30T12:15:40.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.273+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268092296}) 2023-10-30T12:15:40.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402800379}) 2023-10-30T12:15:40.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211294657}) 2023-10-30T12:15:40.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402800379}) 2023-10-30T12:15:40.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.274+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780888786}) 2023-10-30T12:15:40.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738059228}) 2023-10-30T12:15:40.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162564760}) 2023-10-30T12:15:40.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2738059228}) 2023-10-30T12:15:40.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.275+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122720693}) 2023-10-30T12:15:40.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297426261}) 2023-10-30T12:15:40.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520626666}) 2023-10-30T12:15:40.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297426261}) 2023-10-30T12:15:40.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2925219387}) 2023-10-30T12:15:40.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367607723}) 2023-10-30T12:15:40.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2318259951}) 2023-10-30T12:15:40.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367607723}) 2023-10-30T12:15:40.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.276+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143210599}) 2023-10-30T12:15:40.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2498106548}) 2023-10-30T12:15:40.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2057354155}) 2023-10-30T12:15:40.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2498106548}) 2023-10-30T12:15:40.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.277+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=18095017}) 2023-10-30T12:15:40.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226459913}) 2023-10-30T12:15:40.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3172639727}) 2023-10-30T12:15:40.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226459913}) 2023-10-30T12:15:40.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.278+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231432783}) 2023-10-30T12:15:40.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117155330}) 2023-10-30T12:15:40.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4121918358}) 2023-10-30T12:15:40.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117155330}) 2023-10-30T12:15:40.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2338749463}) 2023-10-30T12:15:40.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.279+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1531194309}) 2023-10-30T12:15:40.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=333777942}) 2023-10-30T12:15:40.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1531194309}) 2023-10-30T12:15:40.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145671127}) 2023-10-30T12:15:40.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860947363}) 2023-10-30T12:15:40.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657545571}) 2023-10-30T12:15:40.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860947363}) 2023-10-30T12:15:40.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.280+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732090609}) 2023-10-30T12:15:40.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119547656}) 2023-10-30T12:15:40.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=163441881}) 2023-10-30T12:15:40.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119547656}) 2023-10-30T12:15:40.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.281+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1139365532}) 2023-10-30T12:15:40.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895313840}) 2023-10-30T12:15:40.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37017954}) 2023-10-30T12:15:40.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895313840}) 2023-10-30T12:15:40.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.282+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4076548110}) 2023-10-30T12:15:40.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539201481}) 2023-10-30T12:15:40.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=106600634}) 2023-10-30T12:15:40.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539201481}) 2023-10-30T12:15:40.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1460915318}) 2023-10-30T12:15:40.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.283+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057066712}) 2023-10-30T12:15:40.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=479486885}) 2023-10-30T12:15:40.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057066712}) 2023-10-30T12:15:40.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3762901242}) 2023-10-30T12:15:40.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256365093}) 2023-10-30T12:15:40.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4246661501}) 2023-10-30T12:15:40.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256365093}) 2023-10-30T12:15:40.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.284+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=547153851}) 2023-10-30T12:15:40.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259025176}) 2023-10-30T12:15:40.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278214375}) 2023-10-30T12:15:40.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259025176}) 2023-10-30T12:15:40.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.285+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876782633}) 2023-10-30T12:15:40.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645084507}) 2023-10-30T12:15:40.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365908716}) 2023-10-30T12:15:40.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645084507}) 2023-10-30T12:15:40.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.286+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3895158692}) 2023-10-30T12:15:40.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555398971}) 2023-10-30T12:15:40.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839340431}) 2023-10-30T12:15:40.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555398971}) 2023-10-30T12:15:40.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548278347}) 2023-10-30T12:15:40.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.287+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438235405}) 2023-10-30T12:15:40.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113695085}) 2023-10-30T12:15:40.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438235405}) 2023-10-30T12:15:40.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2135293344}) 2023-10-30T12:15:40.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=87328199}) 2023-10-30T12:15:40.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071821922}) 2023-10-30T12:15:40.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=87328199}) 2023-10-30T12:15:40.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.288+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=690773121}) 2023-10-30T12:15:40.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691815767}) 2023-10-30T12:15:40.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=808179080}) 2023-10-30T12:15:40.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691815767}) 2023-10-30T12:15:40.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.289+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=362399592}) 2023-10-30T12:15:40.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532137408}) 2023-10-30T12:15:40.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1592193309}) 2023-10-30T12:15:40.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532137408}) 2023-10-30T12:15:40.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.290+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3497842926}) 2023-10-30T12:15:40.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479380729}) 2023-10-30T12:15:40.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1010598443}) 2023-10-30T12:15:40.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479380729}) 2023-10-30T12:15:40.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204572495}) 2023-10-30T12:15:40.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906869755}) 2023-10-30T12:15:40.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559694401}) 2023-10-30T12:15:40.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906869755}) 2023-10-30T12:15:40.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.291+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=980352372}) 2023-10-30T12:15:40.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183704145}) 2023-10-30T12:15:40.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773619385}) 2023-10-30T12:15:40.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183704145}) 2023-10-30T12:15:40.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.292+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2008411002}) 2023-10-30T12:15:40.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416007249}) 2023-10-30T12:15:40.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1529134363}) 2023-10-30T12:15:40.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416007249}) 2023-10-30T12:15:40.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.293+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202903527}) 2023-10-30T12:15:40.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406882563}) 2023-10-30T12:15:40.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051774079}) 2023-10-30T12:15:40.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406882563}) 2023-10-30T12:15:40.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4060067165}) 2023-10-30T12:15:40.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041873306}) 2023-10-30T12:15:40.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2817017279}) 2023-10-30T12:15:40.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041873306}) 2023-10-30T12:15:40.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.294+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3305385169}) 2023-10-30T12:15:40.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191874452}) 2023-10-30T12:15:40.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2891997541}) 2023-10-30T12:15:40.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191874452}) 2023-10-30T12:15:40.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.295+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469327387}) 2023-10-30T12:15:40.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.296+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.296+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662146387}) 2023-10-30T12:15:40.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.296+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556506950}) 2023-10-30T12:15:40.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.296+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.296+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662146387}) 2023-10-30T12:15:40.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.296+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951393403}) 2023-10-30T12:15:40.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908485256}) 2023-10-30T12:15:40.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225397901}) 2023-10-30T12:15:40.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908485256}) 2023-10-30T12:15:40.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.297+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1656955133}) 2023-10-30T12:15:40.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830234152}) 2023-10-30T12:15:40.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=182227718}) 2023-10-30T12:15:40.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830234152}) 2023-10-30T12:15:40.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1821376843}) 2023-10-30T12:15:40.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044621188}) 2023-10-30T12:15:40.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=750924099}) 2023-10-30T12:15:40.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044621188}) 2023-10-30T12:15:40.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.298+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2361840312}) 2023-10-30T12:15:40.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=2,osd_metadata=5} crc {logm=890569820,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=357425066,mon_config_key=3965234311,monmap=3140514009,osd_metadata=2398621561}) 2023-10-30T12:15:40.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5} crc {logm=140222855,mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2376284714,mon_config_key=3217606506,monmap=3324057927,osd_metadata=451695420}) 2023-10-30T12:15:40.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=2,osd_metadata=5} crc {logm=890569820,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=357425066,mon_config_key=3965234311,monmap=3140514009,osd_metadata=2398621561}) 2023-10-30T12:15:40.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.299+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=4} crc {logm=4032881371,mdsmap=3984425269,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=357425066,mon_config_key=3217606506,monmap=3324057927,osd_metadata=1319659112}) 2023-10-30T12:15:40.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.300+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.300+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_pg_creating=1,osd_snap=36,osdmap=63} crc {osd_pg_creating=785723747,osd_snap=988802183,osdmap=1307834917}) 2023-10-30T12:15:40.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.301+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=34,osdmap=65} crc {osd_pg_creating=785723747,osd_snap=1203473447,osdmap=2350378258}) 2023-10-30T12:15:40.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.301+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.301+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_pg_creating=1,osd_snap=36,osdmap=63} crc {osd_pg_creating=785723747,osd_snap=988802183,osdmap=1307834917}) 2023-10-30T12:15:40.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.301+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_pg_creating=1,osd_snap=36,osdmap=63} crc {osd_pg_creating=785723747,osd_snap=1769035770,osdmap=1380934878}) 2023-10-30T12:15:40.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.305+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.305+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=1583097555}) 2023-10-30T12:15:40.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.305+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=23} crc {osdmap=1233601711}) 2023-10-30T12:15:40.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.305+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:15:40.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.305+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=1583097555}) 2023-10-30T12:15:40.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:15:40.305+0000 7f0c45944640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=2196986432}) 2023-10-30T12:15:41.276 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:15:41.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.392+0000 7faaf5421640 10 monclient: get_monmap_and_config 2023-10-30T12:15:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.392+0000 7faaf5421640 10 monclient: build_initial_monmap 2023-10-30T12:15:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.392+0000 7faaf5421640 10 monclient: monmap: 2023-10-30T12:15:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:41.392714+0000 2023-10-30T12:15:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:41.392714+0000 2023-10-30T12:15:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:41.396 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 1 Processor -- start 2023-10-30T12:15:41.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 1 -- start start 2023-10-30T12:15:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient: init 2023-10-30T12:15:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-10-30T12:15:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7faaf0131b20 0x7faaf0132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient(hunting): picked mon.noname-c con 0x7faaf0131b20 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf0071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient(hunting): picked mon.noname-b con 0x7faaf0071bc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaf0097410 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf009a9b0 con 0x7faaf0097410 2023-10-30T12:15:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf0131f00 con 0x7faaf0131b20 2023-10-30T12:15:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf009d920 con 0x7faaf0071bc0 2023-10-30T12:15:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.393+0000 7faaf5421640 10 monclient(hunting): authenticate will time out at 1786.575928s 2023-10-30T12:15:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf0071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf0071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35514/0 (socket says 172.21.15.50:35514) 2023-10-30T12:15:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 -- 172.21.15.50:0/3718941091 learned_addr learned my addr 172.21.15.50:0/3718941091 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): get_auth_request con 0x7faaf0071bc0 auth_method 0 2023-10-30T12:15:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 1 --2- 172.21.15.50:0/3718941091 >> v2:172.21.15.118:3300/0 conn(0x7faaf0131b20 0x7faaf0132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 10 monclient(hunting): get_auth_request con 0x7faaf0131b20 auth_method 0 2023-10-30T12:15:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faaeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient(hunting): handle_auth_done global_id 115079 payload 274 2023-10-30T12:15:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 -- 172.21.15.50:0/3718941091 >> v2:172.21.15.118:3300/0 conn(0x7faaf0131b20 msgr2=0x7faaf0132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 --2- 172.21.15.50:0/3718941091 >> v2:172.21.15.118:3300/0 conn(0x7faaf0131b20 0x7faaf0132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 -- 172.21.15.50:0/3718941091 >> v1:172.21.15.118:6789/0 conn(0x7faaf0097410 legacy=0x7faaf00978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 monclient: found mon.noname-b 2023-10-30T12:15:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.394+0000 7faae7fff640 1 -- 172.21.15.50:0/3718941091 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf009a0b0 con 0x7faaf0071bc0 2023-10-30T12:15:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faae7fff640 1 --2- 172.21.15.50:0/3718941091 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf0071fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faad800bb10 tx=0x7faad802e030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17fff98becb120bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 1 -- 172.21.15.50:0/3718941091 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad802ede0 con 0x7faaf0071bc0 2023-10-30T12:15:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: dump: 2023-10-30T12:15:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.415 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.415 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.415 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: _finish_auth 0 2023-10-30T12:15:41.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:15:41.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:41.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 1 -- 172.21.15.50:0/3718941091 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad8036440 con 0x7faaf0071bc0 2023-10-30T12:15:41.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 5 monclient: authenticate success, global_id 115079 2023-10-30T12:15:41.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:41.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:41.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:41.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 1 -- 172.21.15.50:0/3718941091 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad80367d0 con 0x7faaf0071bc0 2023-10-30T12:15:41.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 4 set_mon_vals no callback set 2023-10-30T12:15:41.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:41.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaee7fc640 10 monclient: dump: 2023-10-30T12:15:41.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.418 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.418 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.419 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.419 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.419 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.419 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 10 monclient: shutdown 2023-10-30T12:15:41.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:41.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 1 -- 172.21.15.50:0/3718941091 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 msgr2=0x7faaf0071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 1 --2- 172.21.15.50:0/3718941091 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf0071fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faad800bb10 tx=0x7faad802e030 comp rx=0 tx=0).stop 2023-10-30T12:15:41.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 1 -- 172.21.15.50:0/3718941091 shutdown_connections 2023-10-30T12:15:41.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 1 --2- 172.21.15.50:0/3718941091 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf0071fc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 1 --2- 172.21.15.50:0/3718941091 >> v2:172.21.15.118:3300/0 conn(0x7faaf0131b20 0x7faaf0132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.395+0000 7faaf5421640 1 -- 172.21.15.50:0/3718941091 >> 172.21.15.50:0/3718941091 conn(0x7faaf00a05c0 msgr2=0x7faaf009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:41.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 -- 172.21.15.50:0/3718941091 shutdown_connections 2023-10-30T12:15:41.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 -- 172.21.15.50:0/3718941091 wait complete. 2023-10-30T12:15:41.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 10 monclient: build_initial_monmap 2023-10-30T12:15:41.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 10 monclient: monmap: 2023-10-30T12:15:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:41.423 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 librados: starting msgr at 2023-10-30T12:15:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 librados: starting objecter 2023-10-30T12:15:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 Processor -- start 2023-10-30T12:15:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 -- start start 2023-10-30T12:15:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 librados: setting wanted keys 2023-10-30T12:15:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 1 librados: calling monclient init 2023-10-30T12:15:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.396+0000 7faaf5421640 10 monclient: init 2023-10-30T12:15:41.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:15:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf01c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): picked mon.noname-c con 0x7faaf0071bc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf0097410 0x7faaf0096360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): picked mon.noname-b con 0x7faaf0097410 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faaf0131b20 0x7faaf00968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): picked mon.noname-a con 0x7faaf0131b20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:41.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf0099430 con 0x7faaf0131b20 2023-10-30T12:15:41.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf009bd80 con 0x7faaf0097410 2023-10-30T12:15:41.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf0130ff0 con 0x7faaf0071bc0 2023-10-30T12:15:41.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:41.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaf5421640 10 monclient(hunting): authenticate will time out at 1786.579396s 2023-10-30T12:15:41.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf01c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf01c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35516/0 (socket says 172.21.15.50:35516) 2023-10-30T12:15:41.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 -- 172.21.15.50:0/4247174714 learned_addr learned my addr 172.21.15.50:0/4247174714 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:41.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): get_auth_request con 0x7faaf0071bc0 auth_method 0 2023-10-30T12:15:41.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faaf0131b20 0x7faaf00968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 10 monclient(hunting): get_auth_request con 0x7faaf0131b20 auth_method 0 2023-10-30T12:15:41.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf0097410 0x7faaf0096360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 10 monclient(hunting): get_auth_request con 0x7faaf0097410 auth_method 0 2023-10-30T12:15:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faae7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient(hunting): handle_auth_done global_id 115082 payload 824 2023-10-30T12:15:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 -- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf0097410 msgr2=0x7faaf0096360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf0097410 0x7faaf0096360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 -- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faaf0131b20 msgr2=0x7faaf00968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faaf0131b20 0x7faaf00968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:15:41.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 monclient: found mon.noname-c 2023-10-30T12:15:41.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:41.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.397+0000 7faaeeffd640 1 -- 172.21.15.50:0/4247174714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad800b770 con 0x7faaf0071bc0 2023-10-30T12:15:41.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaeeffd640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf01c9fb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faae000e900 tx=0x7faae000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 client.?.objecter ms_handle_connect 0x7faaf0071bc0 2023-10-30T12:15:41.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:41.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 1 -- 172.21.15.50:0/4247174714 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0009660 con 0x7faaf0071bc0 2023-10-30T12:15:41.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:41.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: dump: 2023-10-30T12:15:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.437 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: _finish_auth 0 2023-10-30T12:15:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:15:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 5 monclient: authenticate success, global_id 115082 2023-10-30T12:15:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 1 -- 172.21.15.50:0/4247174714 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae000ca00 con 0x7faaf0071bc0 2023-10-30T12:15:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 1 -- 172.21.15.50:0/4247174714 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae000cd70 con 0x7faaf0071bc0 2023-10-30T12:15:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 10 monclient: _renew_subs 2023-10-30T12:15:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf01312d0 con 0x7faaf0071bc0 2023-10-30T12:15:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaedffb640 4 set_mon_vals no callback set 2023-10-30T12:15:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faae77fe640 10 monclient: dump: 2023-10-30T12:15:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.441 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 10 client.115082.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 10 monclient: _renew_subs 2023-10-30T12:15:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf012f540 con 0x7faaf0071bc0 2023-10-30T12:15:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.398+0000 7faaf5421640 1 librados: init done 2023-10-30T12:15:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae77fe640 1 -- 172.21.15.50:0/4247174714 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7faae0013470 con 0x7faaf0071bc0 2023-10-30T12:15:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faaf5421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faaf5421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:41.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faaf5421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf004e160 con 0x7faaf0071bc0 2023-10-30T12:15:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae77fe640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faad0039320 0x7faad003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae77fe640 1 -- 172.21.15.50:0/4247174714 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(466..466 src has 417..466) v4 ==== 6566+0+0 (secure 0 0 0) 0x7faae004b290 con 0x7faaf0071bc0 2023-10-30T12:15:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae77fe640 10 client.115082.objecter ms_dispatch 0x7faaf0000b90 osd_map(466..466 src has 417..466) v4 2023-10-30T12:15:41.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae77fe640 3 client.115082.objecter handle_osd_map got epochs [466,466] > 0 2023-10-30T12:15:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae77fe640 3 client.115082.objecter handle_osd_map decoding full epoch 466 2023-10-30T12:15:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae77fe640 20 client.115082.objecter dump_active .. 0 homeless 2023-10-30T12:15:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae7fff640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faad0039320 0x7faad003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.399+0000 7faae7fff640 10 monclient: get_auth_request con 0x7faad0039320 auth_method 0 2023-10-30T12:15:41.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.400+0000 7faae77fe640 10 client.115082.objecter ms_handle_connect 0x7faad0039320 2023-10-30T12:15:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.400+0000 7faae7fff640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faad0039320 0x7faad003b7e0 secure :-1 s=READY pgs=4204 cs=0 l=1 rev1=1 crypto rx=0x7faad8038000 tx=0x7faad8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.404+0000 7faae77fe640 1 -- 172.21.15.50:0/4247174714 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faae001b3b0 con 0x7faaf0071bc0 2023-10-30T12:15:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.404+0000 7faae77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:41.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.404+0000 7faae77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.656+0000 7faaf5421640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.656+0000 7faaf5421640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.656+0000 7faaf5421640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.656+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faaf00988b0 con 0x7faaf0071bc0 2023-10-30T12:15:41.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.656+0000 7faae77fe640 1 -- 172.21.15.50:0/4247174714 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7faae0017020 con 0x7faaf0071bc0 2023-10-30T12:15:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.656+0000 7faae77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:41.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.656+0000 7faae77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:41.658 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 10 librados: watch_flush enter 2023-10-30T12:15:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 10 librados: watch_flush exit 2023-10-30T12:15:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 20 client.115082.objecter shutdown clearing up homeless session... 2023-10-30T12:15:41.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 10 client.115082.objecter successfully canceled tick 2023-10-30T12:15:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faad0039320 msgr2=0x7faad003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faad0039320 0x7faad003b7e0 secure :-1 s=READY pgs=4204 cs=0 l=1 rev1=1 crypto rx=0x7faad8038000 tx=0x7faad8032000 comp rx=0 tx=0).stop 2023-10-30T12:15:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 10 monclient: shutdown 2023-10-30T12:15:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:41.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 msgr2=0x7faaf01c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf01c9fb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faae000e900 tx=0x7faae000edc0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 shutdown_connections 2023-10-30T12:15:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faad0039320 0x7faad003b7e0 unknown :-1 s=CLOSED pgs=4204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faaf0131b20 0x7faaf00968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf0097410 0x7faaf0096360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 --2- 172.21.15.50:0/4247174714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faaf0071bc0 0x7faaf01c9fb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 >> 172.21.15.50:0/4247174714 conn(0x7faaf00a05c0 msgr2=0x7faaf009ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 shutdown_connections 2023-10-30T12:15:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 -- 172.21.15.50:0/4247174714 wait complete. 2023-10-30T12:15:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.658+0000 7faaf5421640 1 librados: shutdown 2023-10-30T12:15:41.669 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:15:41.669 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:15:41.670 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:15:41.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:15:41.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.774+0000 7fefdb887640 10 monclient: get_monmap_and_config 2023-10-30T12:15:41.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.774+0000 7fefdb887640 10 monclient: build_initial_monmap 2023-10-30T12:15:41.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.774+0000 7fefdb887640 10 monclient: monmap: 2023-10-30T12:15:41.777 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:41.777 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.777 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:41.775718+0000 2023-10-30T12:15:41.777 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:41.775718+0000 2023-10-30T12:15:41.777 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:41.778 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.778 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:41.778 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:41.778 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:41.778 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:41.779 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:41.779 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 Processor -- start 2023-10-30T12:15:41.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 -- start start 2023-10-30T12:15:41.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient: init 2023-10-30T12:15:41.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:41.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-10-30T12:15:41.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fefd4096540 0x7fefd4096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): picked mon.noname-a con 0x7fefd4096540 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:41.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fefd4096e60 0x7fefd409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): picked mon.noname-c con 0x7fefd4096e60 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:41.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd409afa0 0x7fefd409b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): picked mon.noname-b con 0x7fefd409afa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd409bd30 con 0x7fefd4096540 2023-10-30T12:15:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd409c400 con 0x7fefd4096e60 2023-10-30T12:15:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd409c120 con 0x7fefd409afa0 2023-10-30T12:15:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.775+0000 7fefdb887640 10 monclient(hunting): authenticate will time out at 1786.958978s 2023-10-30T12:15:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd409afa0 0x7fefd409b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fefd4096540 0x7fefd4096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd409afa0 0x7fefd409b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35522/0 (socket says 172.21.15.50:35522) 2023-10-30T12:15:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fefd4096540 0x7fefd4096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36080/0 (socket says 172.21.15.50:36080) 2023-10-30T12:15:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 -- 172.21.15.50:0/51309817 learned_addr learned my addr 172.21.15.50:0/51309817 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): get_auth_request con 0x7fefd409afa0 auth_method 0 2023-10-30T12:15:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 10 monclient(hunting): get_auth_request con 0x7fefd4096540 auth_method 0 2023-10-30T12:15:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd8e22640 1 --2- 172.21.15.50:0/51309817 >> v2:172.21.15.118:3300/0 conn(0x7fefd4096e60 0x7fefd409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd8e22640 10 monclient(hunting): get_auth_request con 0x7fefd4096e60 auth_method 0 2023-10-30T12:15:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd8e22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd8e22640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd8e22640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient(hunting): handle_auth_done global_id 115085 payload 274 2023-10-30T12:15:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 -- 172.21.15.50:0/51309817 >> v2:172.21.15.118:3300/0 conn(0x7fefd4096e60 msgr2=0x7fefd409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:41.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 --2- 172.21.15.50:0/51309817 >> v2:172.21.15.118:3300/0 conn(0x7fefd4096e60 0x7fefd409aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 -- 172.21.15.50:0/51309817 >> v2:172.21.15.50:3300/0 conn(0x7fefd4096540 msgr2=0x7fefd4096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:41.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 --2- 172.21.15.50:0/51309817 >> v2:172.21.15.50:3300/0 conn(0x7fefd4096540 0x7fefd4096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient: _finish_hunting 0 2023-10-30T12:15:41.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 monclient: found mon.noname-b 2023-10-30T12:15:41.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:41.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9e24640 1 -- 172.21.15.50:0/51309817 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd409f600 con 0x7fefd409afa0 2023-10-30T12:15:41.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.776+0000 7fefd9623640 1 --2- 172.21.15.50:0/51309817 >> v2:172.21.15.50:3300/0 conn(0x7fefd4096540 0x7fefd4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:41.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefd9e24640 1 --2- 172.21.15.50:0/51309817 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd409afa0 0x7fefd409b430 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fefd4074290 tx=0x7fefcc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5cce77aa7b204b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 1 -- 172.21.15.50:0/51309817 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc02ca70 con 0x7fefd409afa0 2023-10-30T12:15:41.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:41.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: dump: 2023-10-30T12:15:41.799 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.799 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.800 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.800 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.800 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.800 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.800 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.800 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.801 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.801 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: _finish_auth 0 2023-10-30T12:15:41.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:15:41.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:41.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 1 -- 172.21.15.50:0/51309817 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefcc02cbd0 con 0x7fefd409afa0 2023-10-30T12:15:41.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 5 monclient: authenticate success, global_id 115085 2023-10-30T12:15:41.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:41.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:41.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 1 -- 172.21.15.50:0/51309817 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc0343f0 con 0x7fefd409afa0 2023-10-30T12:15:41.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:41.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 4 set_mon_vals no callback set 2023-10-30T12:15:41.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:41.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefcbfff640 10 monclient: dump: 2023-10-30T12:15:41.803 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.804 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.804 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.804 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.804 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.804 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.805 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.805 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.805 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.805 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 10 monclient: shutdown 2023-10-30T12:15:41.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:41.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 -- 172.21.15.50:0/51309817 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd409afa0 msgr2=0x7fefd409b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 --2- 172.21.15.50:0/51309817 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd409afa0 0x7fefd409b430 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fefd4074290 tx=0x7fefcc02bed0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 -- 172.21.15.50:0/51309817 shutdown_connections 2023-10-30T12:15:41.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 --2- 172.21.15.50:0/51309817 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd409afa0 0x7fefd409b430 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 --2- 172.21.15.50:0/51309817 >> v2:172.21.15.118:3300/0 conn(0x7fefd4096e60 0x7fefd409aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 --2- 172.21.15.50:0/51309817 >> v2:172.21.15.50:3300/0 conn(0x7fefd4096540 0x7fefd4096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 -- 172.21.15.50:0/51309817 >> 172.21.15.50:0/51309817 conn(0x7fefd40a3980 msgr2=0x7fefd40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:41.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 -- 172.21.15.50:0/51309817 shutdown_connections 2023-10-30T12:15:41.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.777+0000 7fefdb887640 1 -- 172.21.15.50:0/51309817 wait complete. 2023-10-30T12:15:41.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 10 monclient: build_initial_monmap 2023-10-30T12:15:41.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 10 monclient: monmap: 2023-10-30T12:15:41.808 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:41.808 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:41.808 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:41.808 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:41.809 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:41.809 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.809 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:41.809 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:41.809 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:41.809 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 1 librados: starting msgr at 2023-10-30T12:15:41.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 1 librados: starting objecter 2023-10-30T12:15:41.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 1 Processor -- start 2023-10-30T12:15:41.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 1 -- start start 2023-10-30T12:15:41.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 1 librados: setting wanted keys 2023-10-30T12:15:41.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 1 librados: calling monclient init 2023-10-30T12:15:41.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.778+0000 7fefdb887640 10 monclient: init 2023-10-30T12:15:41.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:41.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:15:41.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd4096540 0x7fefd41c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): picked mon.noname-c con 0x7fefd4096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefd4096e60 0x7fefd40936c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): picked mon.noname-b con 0x7fefd4096e60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd409afa0 0x7fefd41c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): picked mon.noname-a con 0x7fefd409afa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:41.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd40902d0 con 0x7fefd409afa0 2023-10-30T12:15:41.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd41c7560 con 0x7fefd4096e60 2023-10-30T12:15:41.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd40908e0 con 0x7fefd4096540 2023-10-30T12:15:41.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:41.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefdb887640 10 monclient(hunting): authenticate will time out at 1786.962303s 2023-10-30T12:15:41.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd4096540 0x7fefd41c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd4096540 0x7fefd41c6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35532/0 (socket says 172.21.15.50:35532) 2023-10-30T12:15:41.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd409afa0 0x7fefd41c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 -- 172.21.15.50:0/2488002946 learned_addr learned my addr 172.21.15.50:0/2488002946 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:41.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): get_auth_request con 0x7fefd4096540 auth_method 0 2023-10-30T12:15:41.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 10 monclient(hunting): get_auth_request con 0x7fefd409afa0 auth_method 0 2023-10-30T12:15:41.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefd4096e60 0x7fefd40936c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 10 monclient(hunting): get_auth_request con 0x7fefd4096e60 auth_method 0 2023-10-30T12:15:41.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient(hunting): handle_auth_done global_id 115088 payload 824 2023-10-30T12:15:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefd4096e60 msgr2=0x7fefd40936c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefd4096e60 0x7fefd40936c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd409afa0 msgr2=0x7fefd41c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd409afa0 0x7fefd41c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient: _finish_hunting 0 2023-10-30T12:15:41.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 monclient: found mon.noname-c 2023-10-30T12:15:41.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:41.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 -- 172.21.15.50:0/2488002946 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00093a0 con 0x7fefd4096540 2023-10-30T12:15:41.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9e24640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd409afa0 0x7fefd41c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:41.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefd9623640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd4096540 0x7fefd41c6810 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fefd000ec10 tx=0x7fefd000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefc9ffb640 10 client.?.objecter ms_handle_connect 0x7fefd4096540 2023-10-30T12:15:41.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.779+0000 7fefc9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:41.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd0009750 con 0x7fefd4096540 2023-10-30T12:15:41.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:41.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: dump: 2023-10-30T12:15:41.823 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.823 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.823 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.823 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.824 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.824 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.824 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.824 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.824 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.824 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:15:41.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:15:41.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:41.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd000cc30 con 0x7fefd4096540 2023-10-30T12:15:41.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 5 monclient: authenticate success, global_id 115088 2023-10-30T12:15:41.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:41.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd0005990 con 0x7fefd4096540 2023-10-30T12:15:41.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:41.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 10 monclient: dump: 2023-10-30T12:15:41.826 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.827 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.827 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.827 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.827 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.827 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.827 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.828 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.828 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.828 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 10 monclient: _renew_subs 2023-10-30T12:15:41.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefcb7fe640 4 set_mon_vals no callback set 2023-10-30T12:15:41.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc009b30 con 0x7fefd4096540 2023-10-30T12:15:41.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 10 client.115088.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:41.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 10 monclient: _renew_subs 2023-10-30T12:15:41.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:41.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd4090bc0 con 0x7fefd4096540 2023-10-30T12:15:41.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefdb887640 1 librados: init done 2023-10-30T12:15:41.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.780+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fefd0005af0 con 0x7fefd4096540 2023-10-30T12:15:41.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefdb887640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:41.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefdb887640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd408fa80 0x7fefd402b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd4073ff0 con 0x7fefd408fa80 2023-10-30T12:15:41.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fefb80392f0 0x7fefb803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(467..467 src has 417..467) v4 ==== 6579+0+0 (secure 0 0 0) 0x7fefd004b340 con 0x7fefd4096540 2023-10-30T12:15:41.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 10 client.115088.objecter ms_dispatch 0x7fefd4000b90 osd_map(467..467 src has 417..467) v4 2023-10-30T12:15:41.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 3 client.115088.objecter handle_osd_map got epochs [467,467] > 0 2023-10-30T12:15:41.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 3 client.115088.objecter handle_osd_map decoding full epoch 467 2023-10-30T12:15:41.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd408fa80 0x7fefd402b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient: get_auth_request con 0x7fefd408fa80 auth_method 0 2023-10-30T12:15:41.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 20 client.115088.objecter dump_active .. 0 homeless 2023-10-30T12:15:41.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 10 monclient(hunting): handle_auth_done global_id 194283 payload 824 2023-10-30T12:15:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd9e24640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fefb80392f0 0x7fefb803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 1 -- 172.21.15.50:0/2488002946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd4099110 con 0x7fefd408fa80 2023-10-30T12:15:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd9e24640 10 monclient: get_auth_request con 0x7fefb80392f0 auth_method 0 2023-10-30T12:15:41.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 10 client.115088.objecter ms_handle_connect 0x7fefd408fa80 2023-10-30T12:15:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 10 client.115088.objecter resend_mon_ops 2023-10-30T12:15:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd8e22640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd408fa80 0x7fefd402b430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc4010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc400ea00 con 0x7fefd408fa80 2023-10-30T12:15:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 10 monclient: dump: 2023-10-30T12:15:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.836 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.837 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.837 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.837 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.837 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.837 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.838 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefd9e24640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fefb80392f0 0x7fefb803b7b0 secure :-1 s=READY pgs=4205 cs=0 l=1 rev1=1 crypto rx=0x7fefcc035000 tx=0x7fefcc030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.781+0000 7fefc9ffb640 10 client.115088.objecter ms_handle_connect 0x7fefb80392f0 2023-10-30T12:15:41.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.782+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fefc400eb60 con 0x7fefd408fa80 2023-10-30T12:15:41.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.782+0000 7fefc9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:41.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.782+0000 7fefc9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:41.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.782+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd408fa80 msgr2=0x7fefd402b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.782+0000 7fefc9ffb640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd408fa80 0x7fefd402b430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc4010c80 comp rx=0 tx=0).stop 2023-10-30T12:15:41.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.782+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd408fa80 msgr2=0x7fefd402b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:41.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.782+0000 7fefc9ffb640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd408fa80 0x7fefd402b430 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc4010c80 comp rx=0 tx=0).stop 2023-10-30T12:15:41.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefdb887640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:41.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefdb887640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd4093c00 0x7fefd402b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:41.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefdb887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:41.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd409b970 con 0x7fefd4093c00 2023-10-30T12:15:41.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd4093c00 0x7fefd402b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:41.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient: get_auth_request con 0x7fefd4093c00 auth_method 0 2023-10-30T12:15:41.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:41.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:41.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:41.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:41.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:41.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:41.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 10 monclient(hunting): handle_auth_done global_id 194286 payload 824 2023-10-30T12:15:41.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 1 -- 172.21.15.50:0/2488002946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fefd404e160 con 0x7fefd4093c00 2023-10-30T12:15:41.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefc9ffb640 10 client.115088.objecter ms_handle_connect 0x7fefd4093c00 2023-10-30T12:15:41.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefc9ffb640 10 client.115088.objecter resend_mon_ops 2023-10-30T12:15:41.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefd8e22640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd4093c00 0x7fefd402b970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:41.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc402b070 con 0x7fefd4093c00 2023-10-30T12:15:41.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.788+0000 7fefc9ffb640 10 monclient: dump: 2023-10-30T12:15:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.845 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:41.846 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:41.847 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7fefc4023040 con 0x7fefd4093c00 2023-10-30T12:15:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefc9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefc9ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd4093c00 msgr2=0x7fefd402b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefc9ffb640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd4093c00 0x7fefd402b970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc40046c0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefc9ffb640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd4093c00 msgr2=0x7fefd402b970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:41.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefc9ffb640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefd4093c00 0x7fefd402b970 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:41.848 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:41.848 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:41.849 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:41.849 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:15:41.849 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 156, 2023-10-30T12:15:41.849 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:41.849 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:41.850 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:41.850 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:41.850 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:41.850 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:15:41.850 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:41.850 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:41.851 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:41.851 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:41.851 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:41.851 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:41.851 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:41.852 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:41.852 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:41.852 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:41.852 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:41.852 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:41.853 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:41.854 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:41.854 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:41.854 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:41.854 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:41.854 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:41.855 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:41.855 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:41.855 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:41.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:41.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.856 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:41.856 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:41.856 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:41.856 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:41.856 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:41.857 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:41.857 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:41.857 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:41.857 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:41.857 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:41.857 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:41.858 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:41.858 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:41.858 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:41.858 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:41.858 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:41.859 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:41.859 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:41.859 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:41.859 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:41.859 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:41.860 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:41.860 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:41.860 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:41.860 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:41.860 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:41.860 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:41.861 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:41.861 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:41.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.861 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:41.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.861 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:41.862 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:41.862 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:41.862 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:41.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.862 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:41.863 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:41.863 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:41.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.863 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:41.864 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:41.864 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:41.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.864 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:41.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.864 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:41.865 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:41.865 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:41.865 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:41.865 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:41.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.866 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:41.866 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:41.866 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:41.866 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:41.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.867 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:41.867 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:41.867 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:41.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.868 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:41.868 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:41.868 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:41.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.868 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:41.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.869 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:41.869 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:41.869 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:41.869 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:41.869 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:41.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.870 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:41.870 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:41.870 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:41.871 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:41.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.871 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:41.871 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:41.871 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:41.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:41.872 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:41.872 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:41.875 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:41.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.875 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:41.875 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:41.875 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:41.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.876 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:41.876 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:41.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.876 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:41.877 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:41.877 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:15:41.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.877 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:41.877 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:41.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.878 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:41.878 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:41.878 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:15:41.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.879 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:41.879 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:15:41.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:41.879 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:41.879 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:41.879 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:41.880 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 10 librados: watch_flush enter 2023-10-30T12:15:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 10 librados: watch_flush exit 2023-10-30T12:15:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 20 client.115088.objecter shutdown clearing up homeless session... 2023-10-30T12:15:41.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 10 client.115088.objecter successfully canceled tick 2023-10-30T12:15:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fefb80392f0 msgr2=0x7fefb803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fefb80392f0 0x7fefb803b7b0 secure :-1 s=READY pgs=4205 cs=0 l=1 rev1=1 crypto rx=0x7fefcc035000 tx=0x7fefcc030000 comp rx=0 tx=0).stop 2023-10-30T12:15:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 10 monclient: shutdown 2023-10-30T12:15:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd4096540 msgr2=0x7fefd41c6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 --2- 172.21.15.50:0/2488002946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefd4096540 0x7fefd41c6810 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fefd000ec10 tx=0x7fefd000c520 comp rx=0 tx=0).stop 2023-10-30T12:15:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefd9e24640 1 -- 172.21.15.50:0/2488002946 reap_dead start 2023-10-30T12:15:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefd9e24640 1 -- 172.21.15.50:0/2488002946 reap_dead start 2023-10-30T12:15:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 shutdown_connections 2023-10-30T12:15:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 >> 172.21.15.50:0/2488002946 conn(0x7fefd40a3980 msgr2=0x7fefd41ba890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 shutdown_connections 2023-10-30T12:15:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 -- 172.21.15.50:0/2488002946 wait complete. 2023-10-30T12:15:41.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.789+0000 7fefdb887640 1 librados: shutdown 2023-10-30T12:15:41.885 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:15:41.886 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:15:41.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:15:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.990+0000 7f14ab5f5640 10 monclient: get_monmap_and_config 2023-10-30T12:15:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.990+0000 7f14ab5f5640 10 monclient: build_initial_monmap 2023-10-30T12:15:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.990+0000 7f14ab5f5640 10 monclient: monmap: 2023-10-30T12:15:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:41.991943+0000 2023-10-30T12:15:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:41.991943+0000 2023-10-30T12:15:41.994 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:41.994 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:41.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:41.994 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:41.995 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:41.995 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:41.995 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:41.995 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.991+0000 7f14ab5f5640 1 Processor -- start 2023-10-30T12:15:42.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.991+0000 7f14ab5f5640 1 -- start start 2023-10-30T12:15:42.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.991+0000 7f14ab5f5640 10 monclient: init 2023-10-30T12:15:42.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:42.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:15:42.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14a406afc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:42.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f14a406b640 0x7f14a40a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient(hunting): picked mon.noname-c con 0x7f14a406b640 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:42.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14a413b070 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:42.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a406b3a0 con 0x7f14a413b070 2023-10-30T12:15:42.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a409e540 con 0x7f14a406afc0 2023-10-30T12:15:42.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a409fa10 con 0x7f14a406b640 2023-10-30T12:15:42.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:42.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14ab5f5640 10 monclient(hunting): authenticate will time out at 1787.175251s 2023-10-30T12:15:42.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a9b92640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f14a413b070 0x7f14a413d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47944/0 (socket says 172.21.15.50:47944) 2023-10-30T12:15:42.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a9b92640 1 -- 172.21.15.50:0/1285483170 learned_addr learned my addr 172.21.15.50:0/1285483170 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:42.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 1 --2- 172.21.15.50:0/1285483170 >> v2:172.21.15.118:3300/0 conn(0x7f14a406b640 0x7f14a40a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 10 monclient(hunting): get_auth_request con 0x7f14a406b640 auth_method 0 2023-10-30T12:15:42.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74771831 0 0) 0x7f14a406b3a0 con 0x7f14a413b070 2023-10-30T12:15:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): _negotiate 2023-10-30T12:15:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): my global_id is 194289 2023-10-30T12:15:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148c003520 con 0x7f14a413b070 2023-10-30T12:15:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2878502333 0 0) 0x7f148c003520 con 0x7f14a413b070 2023-10-30T12:15:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 >> v2:172.21.15.118:3300/0 conn(0x7f14a406b640 msgr2=0x7f14a40a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 --2- 172.21.15.50:0/1285483170 >> v2:172.21.15.118:3300/0 conn(0x7f14a406b640 0x7f14a40a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 >> v1:172.21.15.118:6789/0 conn(0x7f14a406afc0 legacy=0x7f14a40a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:15:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:15:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a406a6f0 con 0x7f14a413b070 2023-10-30T12:15:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: _finish_auth 0 2023-10-30T12:15:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:15:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f1494001a10 con 0x7f14a413b070 2023-10-30T12:15:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:15:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: dump: 2023-10-30T12:15:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.016 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:15:42.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 >> v1:172.21.15.50:6789/0 conn(0x7f14a413b070 legacy=0x7f14a413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:15:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f148c0039c0 0x7f148c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): picked mon.a con 0x7f148c0039c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0047c0 0x7f148c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): picked mon.b con 0x7f148c0047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:42.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f148c0052a0 0x7f148c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): picked mon.c con 0x7f148c0052a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:42.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a409fa10 con 0x7f148c0039c0 2023-10-30T12:15:42.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c00c1a0 con 0x7f148c0047c0 2023-10-30T12:15:42.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c00c360 con 0x7f148c0052a0 2023-10-30T12:15:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a3fff640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:15:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.992+0000 7f14a9391640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f148c0039c0 0x7f148c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): get_auth_request con 0x7f148c0039c0 auth_method 0 2023-10-30T12:15:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0047c0 0x7f148c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:15:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 10 monclient(hunting): get_auth_request con 0x7f148c0047c0 auth_method 0 2023-10-30T12:15:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f148c0052a0 0x7f148c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 10 monclient(hunting): get_auth_request con 0x7f148c0052a0 auth_method 0 2023-10-30T12:15:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:15:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:15:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient(hunting): handle_auth_done global_id 194289 payload 293 2023-10-30T12:15:42.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 1 -- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f148c0052a0 msgr2=0x7f148c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:42.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f148c0052a0 0x7f148c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 1 -- 172.21.15.50:0/1285483170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0047c0 msgr2=0x7f148c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:42.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0047c0 0x7f148c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient: _finish_hunting 0 2023-10-30T12:15:42.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 1 monclient: found mon.a 2023-10-30T12:15:42.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:42.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 1 -- 172.21.15.50:0/1285483170 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c003520 con 0x7f148c0039c0 2023-10-30T12:15:42.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient: _finish_auth 0 2023-10-30T12:15:42.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 10 monclient: _check_auth_tickets 2023-10-30T12:15:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9391640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14a9b92640 1 -- 172.21.15.50:0/1285483170 reap_dead start 2023-10-30T12:15:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14ab5f5640 5 monclient: authenticate success, global_id 194289 2023-10-30T12:15:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.993+0000 7f14ab5f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a9391640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f148c0039c0 0x7f148c003de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f149c001750 tx=0x7f148c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50c2a454f29339d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c0052d0 con 0x7f148c0039c0 2023-10-30T12:15:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 10 monclient: dump: 2023-10-30T12:15:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c02e050 con 0x7f148c0039c0 2023-10-30T12:15:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 1 -- 172.21.15.50:0/1285483170 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c003830 con 0x7f148c0039c0 2023-10-30T12:15:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 4 set_mon_vals no callback set 2023-10-30T12:15:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14a3fff640 10 monclient: dump: 2023-10-30T12:15:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.033 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.033 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.033 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.033 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 10 monclient: shutdown 2023-10-30T12:15:42.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 1 -- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f148c0039c0 msgr2=0x7f148c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f148c0039c0 0x7f148c003de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f149c001750 tx=0x7f148c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 1 -- 172.21.15.50:0/1285483170 shutdown_connections 2023-10-30T12:15:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 1 --2- 172.21.15.50:0/1285483170 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f148c0039c0 0x7f148c003de0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 1 -- 172.21.15.50:0/1285483170 >> 172.21.15.50:0/1285483170 conn(0x7f14a40a6200 msgr2=0x7f14a40a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 1 -- 172.21.15.50:0/1285483170 shutdown_connections 2023-10-30T12:15:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.994+0000 7f14ab5f5640 1 -- 172.21.15.50:0/1285483170 wait complete. 2023-10-30T12:15:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 10 monclient: build_initial_monmap 2023-10-30T12:15:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 10 monclient: monmap: 2023-10-30T12:15:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:42.037 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:42.037 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 1 librados: starting msgr at 2023-10-30T12:15:42.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 1 librados: starting objecter 2023-10-30T12:15:42.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 1 Processor -- start 2023-10-30T12:15:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 1 -- start start 2023-10-30T12:15:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 1 librados: setting wanted keys 2023-10-30T12:15:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 1 librados: calling monclient init 2023-10-30T12:15:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.995+0000 7f14ab5f5640 10 monclient: init 2023-10-30T12:15:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:15:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0039c0 0x7f14a41ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): picked mon.noname-b con 0x7f148c0039c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a411c680 0x7f14a411cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): picked mon.noname-c con 0x7f14a411c680 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:42.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a411d020 0x7f14a411d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): picked mon.noname-a con 0x7f14a411d020 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a411fe50 con 0x7f14a411d020 2023-10-30T12:15:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a41385b0 con 0x7f148c0039c0 2023-10-30T12:15:42.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a4070de0 con 0x7f14a411c680 2023-10-30T12:15:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14ab5f5640 10 monclient(hunting): authenticate will time out at 1787.179216s 2023-10-30T12:15:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a411c680 0x7f14a411cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a411d020 0x7f14a411d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a411c680 0x7f14a411cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35546/0 (socket says 172.21.15.50:35546) 2023-10-30T12:15:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 1 -- 172.21.15.50:0/2862449734 learned_addr learned my addr 172.21.15.50:0/2862449734 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a411d020 0x7f14a411d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36106/0 (socket says 172.21.15.50:36106) 2023-10-30T12:15:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 10 monclient(hunting): get_auth_request con 0x7f14a411c680 auth_method 0 2023-10-30T12:15:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): get_auth_request con 0x7f14a411d020 auth_method 0 2023-10-30T12:15:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0039c0 0x7f14a41ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 10 monclient(hunting): get_auth_request con 0x7f148c0039c0 auth_method 0 2023-10-30T12:15:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient(hunting): handle_auth_done global_id 194292 payload 824 2023-10-30T12:15:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a411c680 msgr2=0x7f14a411cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a411c680 0x7f14a411cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0039c0 msgr2=0x7f14a41ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f148c0039c0 0x7f14a41ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient: _finish_hunting 0 2023-10-30T12:15:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 monclient: found mon.noname-a 2023-10-30T12:15:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:42.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 -- 172.21.15.50:0/2862449734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a411ece0 con 0x7f14a411d020 2023-10-30T12:15:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a8b90640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a411c680 0x7f14a411cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a9b92640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a411d020 0x7f14a411d4a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f14a406a4b0 tx=0x7f1494007aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 10 client.?.objecter ms_handle_connect 0x7f14a411d020 2023-10-30T12:15:42.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149400a9f0 con 0x7f14a411d020 2023-10-30T12:15:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 10 monclient: dump: 2023-10-30T12:15:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.050 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.051 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:15:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:15:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.996+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149400ab90 con 0x7f14a411d020 2023-10-30T12:15:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 5 monclient: authenticate success, global_id 194292 2023-10-30T12:15:42.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 10 monclient: _renew_subs 2023-10-30T12:15:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a41206f0 con 0x7f14a411d020 2023-10-30T12:15:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14a1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14940133f0 con 0x7f14a411d020 2023-10-30T12:15:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14a1ffb640 10 monclient: dump: 2023-10-30T12:15:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.054 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.055 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14a37fe640 4 set_mon_vals no callback set 2023-10-30T12:15:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 10 client.194292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 10 monclient: _renew_subs 2023-10-30T12:15:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a411e320 con 0x7f14a411d020 2023-10-30T12:15:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.997+0000 7f14ab5f5640 1 librados: init done 2023-10-30T12:15:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14ab5f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14ab5f5640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a402c5b0 0x7f14a4060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14ab5f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a4073ff0 con 0x7f14a402c5b0 2023-10-30T12:15:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1494013550 con 0x7f14a411d020 2023-10-30T12:15:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a402c5b0 0x7f14a4060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient: get_auth_request con 0x7f14a402c5b0 auth_method 0 2023-10-30T12:15:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14900392f0 0x7f149003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 10 monclient(hunting): handle_auth_done global_id 115094 payload 824 2023-10-30T12:15:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(467..467 src has 417..467) v4 ==== 6579+0+0 (secure 0 0 0) 0x7f149404b1a0 con 0x7f14a411d020 2023-10-30T12:15:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 1 -- 172.21.15.50:0/2862449734 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14a409f0b0 con 0x7f14a402c5b0 2023-10-30T12:15:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 10 client.194292.objecter ms_dispatch 0x7f14a4000b90 osd_map(467..467 src has 417..467) v4 2023-10-30T12:15:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 3 client.194292.objecter handle_osd_map got epochs [467,467] > 0 2023-10-30T12:15:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 3 client.194292.objecter handle_osd_map decoding full epoch 467 2023-10-30T12:15:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 20 client.194292.objecter dump_active .. 0 homeless 2023-10-30T12:15:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a9391640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a402c5b0 0x7f14a4060fc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f149c001720 tx=0x7f149c001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 10 client.194292.objecter ms_handle_connect 0x7f14a402c5b0 2023-10-30T12:15:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 10 client.194292.objecter resend_mon_ops 2023-10-30T12:15:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a8b90640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14900392f0 0x7f149003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a8b90640 10 monclient: get_auth_request con 0x7f14900392f0 auth_method 0 2023-10-30T12:15:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c007a30 con 0x7f14a402c5b0 2023-10-30T12:15:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.998+0000 7f14a1ffb640 10 monclient: dump: 2023-10-30T12:15:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.064 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f149c0069e0 con 0x7f14a402c5b0 2023-10-30T12:15:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a402c5b0 msgr2=0x7f14a4060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a402c5b0 0x7f14a4060fc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f149c001720 tx=0x7f149c001120 comp rx=0 tx=0).stop 2023-10-30T12:15:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a402c5b0 msgr2=0x7f14a4060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a402c5b0 0x7f14a4060fc0 secure :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f149c001720 tx=0x7f149c001120 comp rx=0 tx=0).stop 2023-10-30T12:15:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a1ffb640 10 client.194292.objecter ms_handle_connect 0x7f14900392f0 2023-10-30T12:15:42.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:41.999+0000 7f14a8b90640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14900392f0 0x7f149003b7b0 secure :-1 s=READY pgs=4206 cs=0 l=1 rev1=1 crypto rx=0x7f14a411df20 tx=0x7f1498009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14ab5f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14ab5f5640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a4061500 0x7f14a41d5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14ab5f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a409df80 con 0x7f14a4061500 2023-10-30T12:15:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a4061500 0x7f14a41d5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient: get_auth_request con 0x7f14a4061500 auth_method 0 2023-10-30T12:15:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 10 monclient(hunting): handle_auth_done global_id 115097 payload 824 2023-10-30T12:15:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.005+0000 7f14a9391640 1 -- 172.21.15.50:0/2862449734 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f14a406aa80 con 0x7f14a4061500 2023-10-30T12:15:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a9391640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a4061500 0x7f14a41d5470 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f149c02bc00 tx=0x7f149c030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 10 client.194292.objecter ms_handle_connect 0x7f14a4061500 2023-10-30T12:15:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 10 client.194292.objecter resend_mon_ops 2023-10-30T12:15:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c046660 con 0x7f14a4061500 2023-10-30T12:15:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 10 monclient: dump: 2023-10-30T12:15:42.071 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.072 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.073 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f149c03e420 con 0x7f14a4061500 2023-10-30T12:15:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a4061500 msgr2=0x7f14a41d5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a4061500 0x7f14a41d5470 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f149c02bc00 tx=0x7f149c030000 comp rx=0 tx=0).stop 2023-10-30T12:15:42.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a4061500 msgr2=0x7f14a41d5470 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a1ffb640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f14a4061500 0x7f14a41d5470 secure :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f149c02bc00 tx=0x7f149c030000 comp rx=0 tx=0).stop 2023-10-30T12:15:42.075 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:42.075 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:42.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:42.076 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:15:42.076 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 156, 2023-10-30T12:15:42.076 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:42.076 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:42.077 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:42.077 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:42.077 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.077 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:15:42.077 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:42.078 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:42.078 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:42.078 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:42.078 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:42.078 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:42.079 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:42.079 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:42.079 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:42.079 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:42.079 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:42.079 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:42.080 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:42.080 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.080 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:42.080 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:42.080 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:42.081 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:42.081 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:42.081 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:42.081 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:42.081 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:42.082 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:42.082 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:42.082 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:42.082 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:42.082 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.083 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:42.083 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:42.083 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:42.083 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:42.083 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:42.084 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:42.084 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:42.084 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:42.084 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:42.084 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:42.085 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:42.085 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:42.085 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:42.085 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:42.085 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:42.086 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:42.086 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:42.086 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:42.086 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:42.086 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:42.087 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:42.087 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:42.087 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:42.087 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:42.087 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:42.087 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:42.088 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:42.088 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.088 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:42.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.088 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:42.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:42.089 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:42.089 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:42.089 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:42.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.090 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:42.090 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:42.090 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.091 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:42.091 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:42.091 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.091 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.091 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.092 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:42.092 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:42.092 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:42.092 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:42.093 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:42.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:42.093 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:42.094 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:42.094 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:42.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.094 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:42.094 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:42.095 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.095 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:42.095 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:42.096 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.096 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.096 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:42.096 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:42.097 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:42.097 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:42.097 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:42.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:15:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:42.106 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 10 librados: watch_flush enter 2023-10-30T12:15:42.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 10 librados: watch_flush exit 2023-10-30T12:15:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 20 client.194292.objecter shutdown clearing up homeless session... 2023-10-30T12:15:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 10 client.194292.objecter successfully canceled tick 2023-10-30T12:15:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14900392f0 msgr2=0x7f149003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f14900392f0 0x7f149003b7b0 secure :-1 s=READY pgs=4206 cs=0 l=1 rev1=1 crypto rx=0x7f14a411df20 tx=0x7f1498009190 comp rx=0 tx=0).stop 2023-10-30T12:15:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 10 monclient: shutdown 2023-10-30T12:15:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a411d020 msgr2=0x7f14a411d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 --2- 172.21.15.50:0/2862449734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a411d020 0x7f14a411d4a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f14a406a4b0 tx=0x7f1494007aa0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a9b92640 1 -- 172.21.15.50:0/2862449734 reap_dead start 2023-10-30T12:15:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14a9b92640 1 -- 172.21.15.50:0/2862449734 reap_dead start 2023-10-30T12:15:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 shutdown_connections 2023-10-30T12:15:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 >> 172.21.15.50:0/2862449734 conn(0x7f14a40a6200 msgr2=0x7f14a4137e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 shutdown_connections 2023-10-30T12:15:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 -- 172.21.15.50:0/2862449734 wait complete. 2023-10-30T12:15:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.006+0000 7f14ab5f5640 1 librados: shutdown 2023-10-30T12:15:42.112 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:15:42.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:15:42.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:15:42.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.217+0000 7f2a263a5640 10 monclient: get_monmap_and_config 2023-10-30T12:15:42.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.217+0000 7f2a263a5640 10 monclient: build_initial_monmap 2023-10-30T12:15:42.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.217+0000 7f2a263a5640 10 monclient: monmap: 2023-10-30T12:15:42.219 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:42.219 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.220 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:42.218538+0000 2023-10-30T12:15:42.220 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:42.218538+0000 2023-10-30T12:15:42.220 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:42.220 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.220 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:42.221 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:42.221 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:42.221 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:42.221 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:42.221 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 1 Processor -- start 2023-10-30T12:15:42.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 1 -- start start 2023-10-30T12:15:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient: init 2023-10-30T12:15:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:15:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a20071c00 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2a20097fe0 0x7f2a2009a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a20097fe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a2009b170 0x7f2a20139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2009b170 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a20097600 con 0x7f2a20071c00 2023-10-30T12:15:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a200978e0 con 0x7f2a2009b170 2023-10-30T12:15:42.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2009bf50 con 0x7f2a20097fe0 2023-10-30T12:15:42.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a263a5640 10 monclient(hunting): authenticate will time out at 1787.401672s 2023-10-30T12:15:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a24942640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a2009b170 0x7f2a20139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a1ffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2a20071c00 0x7f2a2009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47950/0 (socket says 172.21.15.50:47950) 2023-10-30T12:15:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a1ffff640 1 -- 172.21.15.50:0/1341766 learned_addr learned my addr 172.21.15.50:0/1341766 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:15:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a24942640 10 monclient(hunting): get_auth_request con 0x7f2a2009b170 auth_method 0 2023-10-30T12:15:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a24942640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a24942640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a24942640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a1f7fe640 1 --2- 172.21.15.50:0/1341766 >> v2:172.21.15.118:3300/0 conn(0x7f2a20097fe0 0x7f2a2009a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f2a20097fe0 auth_method 0 2023-10-30T12:15:42.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a1f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.218+0000 7f2a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1effd640 1 -- 172.21.15.50:0/1341766 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354703227 0 0) 0x7f2a20097600 con 0x7f2a20071c00 2023-10-30T12:15:42.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1effd640 10 monclient(hunting): _negotiate 2023-10-30T12:15:42.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1effd640 10 monclient(hunting): my global_id is 194298 2023-10-30T12:15:42.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1effd640 1 -- 172.21.15.50:0/1341766 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a08004630 con 0x7f2a20071c00 2023-10-30T12:15:42.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 10 monclient(hunting): handle_auth_done global_id 194295 payload 274 2023-10-30T12:15:42.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 1 -- 172.21.15.50:0/1341766 >> v2:172.21.15.118:3300/0 conn(0x7f2a20097fe0 msgr2=0x7f2a2009a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:42.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 1 --2- 172.21.15.50:0/1341766 >> v2:172.21.15.118:3300/0 conn(0x7f2a20097fe0 0x7f2a2009a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 1 -- 172.21.15.50:0/1341766 >> v1:172.21.15.50:6789/0 conn(0x7f2a20071c00 legacy=0x7f2a2009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 10 monclient: _finish_hunting 0 2023-10-30T12:15:42.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 1 monclient: found mon.noname-a 2023-10-30T12:15:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 1 -- 172.21.15.50:0/1341766 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2009d9b0 con 0x7f2a2009b170 2023-10-30T12:15:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a1f7fe640 1 --2- 172.21.15.50:0/1341766 >> v2:172.21.15.118:3300/0 conn(0x7f2a20097fe0 0x7f2a2009a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:15:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.219+0000 7f2a24942640 1 --2- 172.21.15.50:0/1341766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a2009b170 0x7f2a20139760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2a2006bb30 tx=0x7f2a1802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ac9fb73ba47a22 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 1 -- 172.21.15.50:0/1341766 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1802cb60 con 0x7f2a2009b170 2023-10-30T12:15:42.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:42.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: dump: 2023-10-30T12:15:42.241 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.241 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.241 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.242 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.242 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.242 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.242 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.242 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.243 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.243 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: _finish_auth 0 2023-10-30T12:15:42.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:42.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:42.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 5 monclient: authenticate success, global_id 194295 2023-10-30T12:15:42.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:42.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 1 -- 172.21.15.50:0/1341766 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1802ccc0 con 0x7f2a2009b170 2023-10-30T12:15:42.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:42.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 1 -- 172.21.15.50:0/1341766 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18034640 con 0x7f2a2009b170 2023-10-30T12:15:42.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:42.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a1effd640 10 monclient: dump: 2023-10-30T12:15:42.245 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.246 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.246 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.246 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.246 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.246 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.246 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.247 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.247 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.247 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:42.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 4 set_mon_vals no callback set 2023-10-30T12:15:42.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 10 monclient: shutdown 2023-10-30T12:15:42.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:42.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 -- 172.21.15.50:0/1341766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a2009b170 msgr2=0x7f2a20139760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 --2- 172.21.15.50:0/1341766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a2009b170 0x7f2a20139760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2a2006bb30 tx=0x7f2a1802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 -- 172.21.15.50:0/1341766 shutdown_connections 2023-10-30T12:15:42.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 --2- 172.21.15.50:0/1341766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a2009b170 0x7f2a20139760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 --2- 172.21.15.50:0/1341766 >> v2:172.21.15.118:3300/0 conn(0x7f2a20097fe0 0x7f2a2009a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 -- 172.21.15.50:0/1341766 >> 172.21.15.50:0/1341766 conn(0x7f2a200a3980 msgr2=0x7f2a200a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 -- 172.21.15.50:0/1341766 shutdown_connections 2023-10-30T12:15:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.220+0000 7f2a263a5640 1 -- 172.21.15.50:0/1341766 wait complete. 2023-10-30T12:15:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 10 monclient: build_initial_monmap 2023-10-30T12:15:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 10 monclient: monmap: 2023-10-30T12:15:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:42.251 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 1 librados: starting msgr at 2023-10-30T12:15:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 1 librados: starting objecter 2023-10-30T12:15:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 1 Processor -- start 2023-10-30T12:15:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 1 -- start start 2023-10-30T12:15:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 1 librados: setting wanted keys 2023-10-30T12:15:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 1 librados: calling monclient init 2023-10-30T12:15:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.221+0000 7f2a263a5640 10 monclient: init 2023-10-30T12:15:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:15:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20071c00 0x7f2a20094710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a20071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a20097fe0 0x7f2a201c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a20097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a2009b170 0x7f2a201c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a2009b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2008c440 con 0x7f2a20097fe0 2023-10-30T12:15:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20092940 con 0x7f2a20071c00 2023-10-30T12:15:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20092c20 con 0x7f2a2009b170 2023-10-30T12:15:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a263a5640 10 monclient(hunting): authenticate will time out at 1787.405428s 2023-10-30T12:15:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a20097fe0 0x7f2a201c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a2009b170 0x7f2a201c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a20097fe0 0x7f2a201c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36114/0 (socket says 172.21.15.50:36114) 2023-10-30T12:15:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 -- 172.21.15.50:0/4192458961 learned_addr learned my addr 172.21.15.50:0/4192458961 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f2a20097fe0 auth_method 0 2023-10-30T12:15:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 10 monclient(hunting): get_auth_request con 0x7f2a2009b170 auth_method 0 2023-10-30T12:15:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20071c00 0x7f2a20094710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a24942640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 10 monclient(hunting): get_auth_request con 0x7f2a20071c00 auth_method 0 2023-10-30T12:15:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient(hunting): handle_auth_done global_id 194301 payload 824 2023-10-30T12:15:42.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a2009b170 msgr2=0x7f2a201c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2a2009b170 0x7f2a201c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20071c00 msgr2=0x7f2a20094710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20071c00 0x7f2a20094710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:15:42.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:15:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.222+0000 7f2a1f7fe640 1 -- 172.21.15.50:0/4192458961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a14009400 con 0x7f2a20097fe0 2023-10-30T12:15:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1f7fe640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a20097fe0 0x7f2a201c6a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400ebe0 tx=0x7f2a1400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 client.?.objecter ms_handle_connect 0x7f2a20097fe0 2023-10-30T12:15:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14009760 con 0x7f2a20097fe0 2023-10-30T12:15:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:15:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: dump: 2023-10-30T12:15:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.265 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: _finish_auth 0 2023-10-30T12:15:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:15:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 5 monclient: authenticate success, global_id 194301 2023-10-30T12:15:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1400cc30 con 0x7f2a20097fe0 2023-10-30T12:15:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400eee0 con 0x7f2a20097fe0 2023-10-30T12:15:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 10 monclient: _renew_subs 2023-10-30T12:15:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a18009a70 con 0x7f2a20097fe0 2023-10-30T12:15:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:15:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1e7fc640 4 set_mon_vals no callback set 2023-10-30T12:15:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a1cff9640 10 monclient: dump: 2023-10-30T12:15:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.269 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 10 client.194301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 10 monclient: _renew_subs 2023-10-30T12:15:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:15:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a201ca200 con 0x7f2a20097fe0 2023-10-30T12:15:42.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 1 librados: init done 2023-10-30T12:15:42.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:42.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.223+0000 7f2a263a5640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20091c90 0x7f2a2002c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a263a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20073ff0 con 0x7f2a20091c90 2023-10-30T12:15:42.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2a14013470 con 0x7f2a20097fe0 2023-10-30T12:15:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20091c90 0x7f2a2002c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a0c0392f0 0x7f2a0c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient: get_auth_request con 0x7f2a20091c90 auth_method 0 2023-10-30T12:15:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(467..467 src has 417..467) v4 ==== 6579+0+0 (secure 0 0 0) 0x7f2a1404b040 con 0x7f2a20097fe0 2023-10-30T12:15:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 10 client.194301.objecter ms_dispatch 0x7f2a20000b90 osd_map(467..467 src has 417..467) v4 2023-10-30T12:15:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 3 client.194301.objecter handle_osd_map got epochs [467,467] > 0 2023-10-30T12:15:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 3 client.194301.objecter handle_osd_map decoding full epoch 467 2023-10-30T12:15:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 20 client.194301.objecter dump_active .. 0 homeless 2023-10-30T12:15:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a24942640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a0c0392f0 0x7f2a0c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a24942640 10 monclient: get_auth_request con 0x7f2a0c0392f0 auth_method 0 2023-10-30T12:15:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_done global_id 204145 payload 824 2023-10-30T12:15:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 1 -- 172.21.15.50:0/4192458961 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a20071fe0 con 0x7f2a20091c90 2023-10-30T12:15:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 10 client.194301.objecter ms_handle_connect 0x7f2a20091c90 2023-10-30T12:15:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1cff9640 10 client.194301.objecter resend_mon_ops 2023-10-30T12:15:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.224+0000 7f2a1ffff640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20091c90 0x7f2a2002c5b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a10001260 tx=0x7f2a1000fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 10 client.194301.objecter ms_handle_connect 0x7f2a0c0392f0 2023-10-30T12:15:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a24942640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a0c0392f0 0x7f2a0c03b7b0 secure :-1 s=READY pgs=4207 cs=0 l=1 rev1=1 crypto rx=0x7f2a18036000 tx=0x7f2a18030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100070d0 con 0x7f2a20091c90 2023-10-30T12:15:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 10 monclient: dump: 2023-10-30T12:15:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.275 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.277 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f2a10007230 con 0x7f2a20091c90 2023-10-30T12:15:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20091c90 msgr2=0x7f2a2002c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20091c90 0x7f2a2002c5b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a10001260 tx=0x7f2a1000fc00 comp rx=0 tx=0).stop 2023-10-30T12:15:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20091c90 msgr2=0x7f2a2002c5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.225+0000 7f2a1cff9640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a20091c90 0x7f2a2002c5b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.231+0000 7f2a263a5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.231+0000 7f2a263a5640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a201ca9e0 0x7f2a201c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.231+0000 7f2a263a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.231+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20093350 con 0x7f2a201ca9e0 2023-10-30T12:15:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a201ca9e0 0x7f2a201c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient: get_auth_request con 0x7f2a201ca9e0 auth_method 0 2023-10-30T12:15:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 10 monclient(hunting): handle_auth_done global_id 204148 payload 824 2023-10-30T12:15:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 1 -- 172.21.15.50:0/4192458961 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a20093140 con 0x7f2a201ca9e0 2023-10-30T12:15:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1ffff640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a201ca9e0 0x7f2a201c7510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a10001260 tx=0x7f2a10000f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1cff9640 10 client.194301.objecter ms_handle_connect 0x7f2a201ca9e0 2023-10-30T12:15:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1cff9640 10 client.194301.objecter resend_mon_ops 2023-10-30T12:15:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10024070 con 0x7f2a201ca9e0 2023-10-30T12:15:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1cff9640 10 monclient: dump: 2023-10-30T12:15:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.285 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.232+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f2a1000dc80 con 0x7f2a201ca9e0 2023-10-30T12:15:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a1cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a1cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a201ca9e0 msgr2=0x7f2a201c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a1cff9640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a201ca9e0 0x7f2a201c7510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a10001260 tx=0x7f2a10000f70 comp rx=0 tx=0).stop 2023-10-30T12:15:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a1cff9640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a201ca9e0 msgr2=0x7f2a201c7510 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a1cff9640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a201ca9e0 0x7f2a201c7510 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a10001260 tx=0x7f2a10000f70 comp rx=0 tx=0).stop 2023-10-30T12:15:42.287 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:42.287 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:42.287 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:42.287 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:15:42.287 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 156, 2023-10-30T12:15:42.288 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:42.288 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:15:42.288 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:42.288 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:42.288 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.288 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:15:42.289 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:42.289 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:42.289 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:42.289 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:42.289 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:42.290 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:42.290 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:42.290 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:42.290 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:42.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:42.290 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:42.291 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:42.291 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:42.291 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.291 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:42.291 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:42.291 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:42.292 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:42.293 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:42.293 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:42.293 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:42.293 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.294 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:42.294 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:42.294 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:42.294 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:42.294 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:42.295 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:42.295 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:42.295 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:42.295 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:42.296 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:42.296 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:42.296 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:42.296 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:42.296 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:42.296 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:42.297 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:42.297 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:42.297 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:42.297 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:42.297 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:42.298 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:42.298 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:42.298 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:42.298 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:42.298 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:42.298 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:42.299 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:42.299 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.299 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:42.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.299 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:42.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.300 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:42.300 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:42.300 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:42.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:42.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:42.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:42.301 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:42.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:42.302 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.303 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:42.303 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:42.303 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:42.303 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:42.304 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:42.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.304 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:42.304 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:42.304 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:42.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:42.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.305 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:42.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:42.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.306 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:42.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:42.306 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.307 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:42.307 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:42.307 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:42.307 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:42.308 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:42.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.308 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:42.308 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:42.309 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:42.309 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:42.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:42.309 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:42.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:42.311 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:42.311 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.311 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.311 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:42.311 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:42.312 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:42.312 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:42.312 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:42.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.312 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.313 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:42.313 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:42.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.313 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:42.313 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:42.314 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:42.314 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.314 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:42.314 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:42.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:42.314 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:42.316 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 10 librados: watch_flush enter 2023-10-30T12:15:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 10 librados: watch_flush exit 2023-10-30T12:15:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 20 client.194301.objecter shutdown clearing up homeless session... 2023-10-30T12:15:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 10 client.194301.objecter successfully canceled tick 2023-10-30T12:15:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a0c0392f0 msgr2=0x7f2a0c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2a0c0392f0 0x7f2a0c03b7b0 secure :-1 s=READY pgs=4207 cs=0 l=1 rev1=1 crypto rx=0x7f2a18036000 tx=0x7f2a18030000 comp rx=0 tx=0).stop 2023-10-30T12:15:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 10 monclient: shutdown 2023-10-30T12:15:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a20097fe0 msgr2=0x7f2a201c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 --2- 172.21.15.50:0/4192458961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a20097fe0 0x7f2a201c6a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400ebe0 tx=0x7f2a1400c520 comp rx=0 tx=0).stop 2023-10-30T12:15:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a24942640 1 -- 172.21.15.50:0/4192458961 reap_dead start 2023-10-30T12:15:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a24942640 1 -- 172.21.15.50:0/4192458961 reap_dead start 2023-10-30T12:15:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 shutdown_connections 2023-10-30T12:15:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 >> 172.21.15.50:0/4192458961 conn(0x7f2a200a3980 msgr2=0x7f2a200965b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 shutdown_connections 2023-10-30T12:15:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 -- 172.21.15.50:0/4192458961 wait complete. 2023-10-30T12:15:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.233+0000 7f2a263a5640 1 librados: shutdown 2023-10-30T12:15:42.321 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:15:42.321 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:15:42.321 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:15:42.321 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:15:42.321 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:15:42.322 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:15:42.322 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:15:42.335 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:15:42.335 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:15:42.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:42.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.478+0000 7f8318e96640 10 monclient: get_monmap_and_config 2023-10-30T12:15:42.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.478+0000 7f8318e96640 10 monclient: build_initial_monmap 2023-10-30T12:15:42.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.478+0000 7f8318e96640 10 monclient: monmap: 2023-10-30T12:15:42.480 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:42.480 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.481 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:42.479350+0000 2023-10-30T12:15:42.481 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:42.479350+0000 2023-10-30T12:15:42.481 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:42.481 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.481 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:42.481 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:42.482 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:42.482 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:42.482 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:42.482 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.478+0000 7f8318e96640 1 Processor -- start 2023-10-30T12:15:42.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.478+0000 7f8318e96640 1 -- start start 2023-10-30T12:15:42.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.478+0000 7f8318e96640 10 monclient: init 2023-10-30T12:15:42.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:42.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:15:42.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83140a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient(hunting): picked mon.noname-b con 0x7f83140a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:42.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f831406a8e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:42.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8314071400 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:42.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83140a31b0 con 0x7f831406a8e0 2023-10-30T12:15:42.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831406a6a0 con 0x7f8314071400 2023-10-30T12:15:42.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831406be90 con 0x7f83140a2dd0 2023-10-30T12:15:42.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:42.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8318e96640 10 monclient(hunting): authenticate will time out at 1787.662681s 2023-10-30T12:15:42.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f8311d9b640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f831406a8e0 legacy=0x7f83140a4c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:42.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83140a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83140a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35624/0 (socket says 172.21.15.50:35624) 2023-10-30T12:15:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 1 -- 172.21.15.50:0/100248067 learned_addr learned my addr 172.21.15.50:0/100248067 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 10 monclient(hunting): get_auth_request con 0x7f83140a2dd0 auth_method 0 2023-10-30T12:15:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.479+0000 7f831259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 10 monclient(hunting): handle_auth_done global_id 115103 payload 274 2023-10-30T12:15:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 1 -- 172.21.15.50:0/100248067 >> v1:172.21.15.118:6789/0 conn(0x7f8314071400 legacy=0x7f831413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 1 -- 172.21.15.50:0/100248067 >> v1:172.21.15.50:6789/0 conn(0x7f831406a8e0 legacy=0x7f83140a4c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:42.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 10 monclient: _finish_hunting 0 2023-10-30T12:15:42.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 1 monclient: found mon.noname-b 2023-10-30T12:15:42.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:42.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:42.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 1 -- 172.21.15.50:0/100248067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831409e190 con 0x7f83140a2dd0 2023-10-30T12:15:42.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831259c640 1 --2- 172.21.15.50:0/100248067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83140a46c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f83040098d0 tx=0x7f830402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e24a90d291ad4e9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 1 -- 172.21.15.50:0/100248067 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830402e070 con 0x7f83140a2dd0 2023-10-30T12:15:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: dump: 2023-10-30T12:15:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.499 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.500 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: _finish_auth 0 2023-10-30T12:15:42.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: _check_auth_tickets 2023-10-30T12:15:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 1 -- 172.21.15.50:0/100248067 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83040315a0 con 0x7f83140a2dd0 2023-10-30T12:15:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f8318e96640 5 monclient: authenticate success, global_id 115103 2023-10-30T12:15:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f8318e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:42.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f8318e96640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f8318e96640 4 set_mon_vals no callback set 2023-10-30T12:15:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 1 -- 172.21.15.50:0/100248067 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304031930 con 0x7f83140a2dd0 2023-10-30T12:15:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:42.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.480+0000 7f831159a640 10 monclient: dump: 2023-10-30T12:15:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.503 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.504 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.504 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.504 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.504 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.504 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 10 monclient: shutdown 2023-10-30T12:15:42.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:42.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 1 -- 172.21.15.50:0/100248067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 msgr2=0x7f83140a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 1 --2- 172.21.15.50:0/100248067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83140a46c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f83040098d0 tx=0x7f830402b840 comp rx=0 tx=0).stop 2023-10-30T12:15:42.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 1 -- 172.21.15.50:0/100248067 shutdown_connections 2023-10-30T12:15:42.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 1 --2- 172.21.15.50:0/100248067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83140a46c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 1 -- 172.21.15.50:0/100248067 >> 172.21.15.50:0/100248067 conn(0x7f83140a22b0 msgr2=0x7f83140a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:42.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 1 -- 172.21.15.50:0/100248067 shutdown_connections 2023-10-30T12:15:42.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.481+0000 7f8318e96640 1 -- 172.21.15.50:0/100248067 wait complete. 2023-10-30T12:15:42.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 10 monclient: build_initial_monmap 2023-10-30T12:15:42.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 10 monclient: monmap: 2023-10-30T12:15:42.507 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:42.507 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:42.507 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:42.507 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:42.507 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:42.508 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.508 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:42.508 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:42.508 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:42.508 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 1 librados: starting msgr at 2023-10-30T12:15:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 1 librados: starting objecter 2023-10-30T12:15:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 1 Processor -- start 2023-10-30T12:15:42.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 1 -- start start 2023-10-30T12:15:42.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 1 librados: setting wanted keys 2023-10-30T12:15:42.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 1 librados: calling monclient init 2023-10-30T12:15:42.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.482+0000 7f8318e96640 10 monclient: init 2023-10-30T12:15:42.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:42.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:15:42.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8314071400 0x7f83141ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): picked mon.noname-a con 0x7f8314071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:42.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83141cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): picked mon.noname-c con 0x7f83140a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:42.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f831411e1c0 0x7f831411e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): picked mon.noname-b con 0x7f831411e1c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:42.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8314139020 con 0x7f8314071400 2023-10-30T12:15:42.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f831259c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8314071400 msgr2=0x7f83141ca780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:42.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83141208d0 con 0x7f831411e1c0 2023-10-30T12:15:42.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:42.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831406b0a0 con 0x7f83140a2dd0 2023-10-30T12:15:42.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:42.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient(hunting): authenticate will time out at 1787.666151s 2023-10-30T12:15:42.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f831259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8314071400 0x7f83141ca780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:42.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83141cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83141cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35634/0 (socket says 172.21.15.50:35634) 2023-10-30T12:15:42.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 -- 172.21.15.50:0/3025613460 learned_addr learned my addr 172.21.15.50:0/3025613460 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): get_auth_request con 0x7f83140a2dd0 auth_method 0 2023-10-30T12:15:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f831411e1c0 0x7f831411e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 10 monclient(hunting): get_auth_request con 0x7f831411e1c0 auth_method 0 2023-10-30T12:15:42.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:42.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:42.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:42.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:42.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:42.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8312d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:42.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient(hunting): handle_auth_done global_id 115106 payload 824 2023-10-30T12:15:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 -- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f831411e1c0 msgr2=0x7f831411e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f831411e1c0 0x7f831411e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 -- 172.21.15.50:0/3025613460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8314071400 msgr2=0x7f83141ca780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8314071400 0x7f83141ca780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:15:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 monclient: found mon.noname-c 2023-10-30T12:15:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 -- 172.21.15.50:0/3025613460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040094d0 con 0x7f83140a2dd0 2023-10-30T12:15:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8311d9b640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83141cacc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f830800b580 tx=0x7f830800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 client.?.objecter ms_handle_connect 0x7f83140a2dd0 2023-10-30T12:15:42.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:42.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 1 -- 172.21.15.50:0/3025613460 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308003dc0 con 0x7f83140a2dd0 2023-10-30T12:15:42.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: dump: 2023-10-30T12:15:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.522 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: _finish_auth 0 2023-10-30T12:15:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 5 monclient: authenticate success, global_id 115106 2023-10-30T12:15:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 1 -- 172.21.15.50:0/3025613460 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8308005070 con 0x7f83140a2dd0 2023-10-30T12:15:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 1 -- 172.21.15.50:0/3025613460 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308005420 con 0x7f83140a2dd0 2023-10-30T12:15:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8302ffd640 10 monclient: dump: 2023-10-30T12:15:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:42.526 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8310d99640 4 set_mon_vals no callback set 2023-10-30T12:15:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.483+0000 7f8318e96640 10 monclient: _renew_subs 2023-10-30T12:15:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.484+0000 7f8318e96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.484+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83140a1d00 con 0x7f83140a2dd0 2023-10-30T12:15:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.484+0000 7f8318e96640 10 client.115106.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.484+0000 7f8318e96640 10 monclient: _renew_subs 2023-10-30T12:15:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.484+0000 7f8318e96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.484+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8314121990 con 0x7f83140a2dd0 2023-10-30T12:15:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.484+0000 7f8318e96640 1 librados: init done 2023-10-30T12:15:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8302ffd640 1 -- 172.21.15.50:0/3025613460 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8308005580 con 0x7f83140a2dd0 2023-10-30T12:15:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8318e96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8318e96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8318e96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83140739e0 con 0x7f83140a2dd0 2023-10-30T12:15:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8302ffd640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f82f40392a0 0x7f82f403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8302ffd640 1 -- 172.21.15.50:0/3025613460 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(467..467 src has 417..467) v4 ==== 6579+0+0 (secure 0 0 0) 0x7f830804ad60 con 0x7f83140a2dd0 2023-10-30T12:15:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8302ffd640 10 client.115106.objecter ms_dispatch 0x7f8314000b90 osd_map(467..467 src has 417..467) v4 2023-10-30T12:15:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8302ffd640 3 client.115106.objecter handle_osd_map got epochs [467,467] > 0 2023-10-30T12:15:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8302ffd640 3 client.115106.objecter handle_osd_map decoding full epoch 467 2023-10-30T12:15:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f8302ffd640 20 client.115106.objecter dump_active .. 0 homeless 2023-10-30T12:15:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f831259c640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f82f40392a0 0x7f82f403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.485+0000 7f831259c640 10 monclient: get_auth_request con 0x7f82f40392a0 auth_method 0 2023-10-30T12:15:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.486+0000 7f8302ffd640 10 client.115106.objecter ms_handle_connect 0x7f82f40392a0 2023-10-30T12:15:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.486+0000 7f831259c640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f82f40392a0 0x7f82f403b760 secure :-1 s=READY pgs=4208 cs=0 l=1 rev1=1 crypto rx=0x7f8304036000 tx=0x7f8304008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.490+0000 7f8302ffd640 1 -- 172.21.15.50:0/3025613460 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8308020350 con 0x7f83140a2dd0 2023-10-30T12:15:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.490+0000 7f8302ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.490+0000 7f8302ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.732+0000 7f8318e96640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.732+0000 7f8318e96640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.732+0000 7f8318e96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.732+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f831404e160 con 0x7f83140a2dd0 2023-10-30T12:15:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.733+0000 7f8302ffd640 1 -- 172.21.15.50:0/3025613460 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f8308013030 con 0x7f83140a2dd0 2023-10-30T12:15:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.733+0000 7f8302ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.733+0000 7f8302ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:42.736 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 10 librados: watch_flush enter 2023-10-30T12:15:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 10 librados: watch_flush exit 2023-10-30T12:15:42.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 20 client.115106.objecter shutdown clearing up homeless session... 2023-10-30T12:15:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 10 client.115106.objecter successfully canceled tick 2023-10-30T12:15:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f82f40392a0 msgr2=0x7f82f403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f82f40392a0 0x7f82f403b760 secure :-1 s=READY pgs=4208 cs=0 l=1 rev1=1 crypto rx=0x7f8304036000 tx=0x7f8304008030 comp rx=0 tx=0).stop 2023-10-30T12:15:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 10 monclient: shutdown 2023-10-30T12:15:42.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 msgr2=0x7f83141cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83141cacc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f830800b580 tx=0x7f830800ba40 comp rx=0 tx=0).stop 2023-10-30T12:15:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 shutdown_connections 2023-10-30T12:15:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f82f40392a0 0x7f82f403b760 unknown :-1 s=CLOSED pgs=4208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f831411e1c0 0x7f831411e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f83140a2dd0 0x7f83141cacc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 --2- 172.21.15.50:0/3025613460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8314071400 0x7f83141ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 >> 172.21.15.50:0/3025613460 conn(0x7f83140a22b0 msgr2=0x7f831406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 shutdown_connections 2023-10-30T12:15:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.734+0000 7f8318e96640 1 -- 172.21.15.50:0/3025613460 wait complete. 2023-10-30T12:15:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:42.735+0000 7f8318e96640 1 librados: shutdown 2023-10-30T12:15:45.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:45.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.862+0000 7f9f52a52640 10 monclient: get_monmap_and_config 2023-10-30T12:15:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.862+0000 7f9f52a52640 10 monclient: build_initial_monmap 2023-10-30T12:15:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.862+0000 7f9f52a52640 10 monclient: monmap: 2023-10-30T12:15:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:45.864 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:45.863526+0000 2023-10-30T12:15:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:45.863526+0000 2023-10-30T12:15:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:45.866 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:45.866 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:45.866 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:45.866 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:45.866 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 1 Processor -- start 2023-10-30T12:15:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 1 -- start start 2023-10-30T12:15:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient: init 2023-10-30T12:15:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-10-30T12:15:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9f4c098520 0x7f9f4c09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient(hunting): picked mon.noname-c con 0x7f9f4c098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f4c098900 0x7f9f4c098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient(hunting): picked mon.noname-a con 0x7f9f4c098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f4c132740 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c09ad30 con 0x7f9f4c132740 2023-10-30T12:15:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f4c09c460 con 0x7f9f4c098900 2023-10-30T12:15:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f4c09b260 con 0x7f9f4c098520 2023-10-30T12:15:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:45.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f52a52640 10 monclient(hunting): authenticate will time out at 1791.046905s 2023-10-30T12:15:45.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f50fef640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f9f4c132740 legacy=0x7f9f4c132bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:45.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f4b7fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f9f4c098900 msgr2=0x7f9f4c098d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:45.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.863+0000 7f9f4b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f4c098900 0x7f9f4c098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:45.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9f4c098520 0x7f9f4c09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:45.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9f4c098520 0x7f9f4c09bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:51976/0 (socket says 172.21.15.50:51976) 2023-10-30T12:15:45.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 -- 172.21.15.50:0/3319385428 learned_addr learned my addr 172.21.15.50:0/3319385428 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:45.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): get_auth_request con 0x7f9f4c098520 auth_method 0 2023-10-30T12:15:45.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:45.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_done global_id 204157 payload 274 2023-10-30T12:15:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 -- 172.21.15.50:0/3319385428 >> v2:172.21.15.50:3300/0 conn(0x7f9f4c098900 msgr2=0x7f9f4c098d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 --2- 172.21.15.50:0/3319385428 >> v2:172.21.15.50:3300/0 conn(0x7f9f4c098900 0x7f9f4c098d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 -- 172.21.15.50:0/3319385428 >> v1:172.21.15.50:6789/0 conn(0x7f9f4c132740 legacy=0x7f9f4c132bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 monclient: found mon.noname-c 2023-10-30T12:15:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:15:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.864+0000 7f9f4bfff640 1 -- 172.21.15.50:0/3319385428 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c072120 con 0x7f9f4c098520 2023-10-30T12:15:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4bfff640 1 --2- 172.21.15.50:0/3319385428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c098520 0x7f9f4c09bf20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c073e60 tx=0x7f9f3c02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e39d33193ccf62 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 1 -- 172.21.15.50:0/3319385428 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c004230 con 0x7f9f4c098520 2023-10-30T12:15:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:45.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: dump: 2023-10-30T12:15:45.882 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:45.882 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:45.882 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.882 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.883 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:45.883 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:45.883 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:45.883 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:45.883 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:45.884 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:45.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: _finish_auth 0 2023-10-30T12:15:45.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:45.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:45.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f52a52640 5 monclient: authenticate success, global_id 204157 2023-10-30T12:15:45.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 1 -- 172.21.15.50:0/3319385428 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c004390 con 0x7f9f4c098520 2023-10-30T12:15:45.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f52a52640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:45.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:45.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f52a52640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:45.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f52a52640 4 set_mon_vals no callback set 2023-10-30T12:15:45.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 1 -- 172.21.15.50:0/3319385428 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c003810 con 0x7f9f4c098520 2023-10-30T12:15:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.865+0000 7f9f4affd640 10 monclient: dump: 2023-10-30T12:15:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:45.887 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.887 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.887 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:45.887 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:45.887 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:45.888 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:45.888 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:45.888 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:45.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 10 monclient: shutdown 2023-10-30T12:15:45.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:45.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 -- 172.21.15.50:0/3319385428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c098520 msgr2=0x7f9f4c09bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:45.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 --2- 172.21.15.50:0/3319385428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c098520 0x7f9f4c09bf20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c073e60 tx=0x7f9f3c02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:15:45.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 -- 172.21.15.50:0/3319385428 shutdown_connections 2023-10-30T12:15:45.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 --2- 172.21.15.50:0/3319385428 >> v2:172.21.15.50:3300/0 conn(0x7f9f4c098900 0x7f9f4c098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:45.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 --2- 172.21.15.50:0/3319385428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c098520 0x7f9f4c09bf20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:45.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 -- 172.21.15.50:0/3319385428 >> 172.21.15.50:0/3319385428 conn(0x7f9f4c0a05c0 msgr2=0x7f9f4c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:45.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 -- 172.21.15.50:0/3319385428 shutdown_connections 2023-10-30T12:15:45.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.866+0000 7f9f52a52640 1 -- 172.21.15.50:0/3319385428 wait complete. 2023-10-30T12:15:45.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient: build_initial_monmap 2023-10-30T12:15:45.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient: monmap: 2023-10-30T12:15:45.890 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:45.891 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:45.891 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:45.891 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:45.891 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:45.891 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:45.892 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:45.892 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:45.892 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:45.892 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:45.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 librados: starting msgr at 2023-10-30T12:15:45.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 librados: starting objecter 2023-10-30T12:15:45.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 Processor -- start 2023-10-30T12:15:45.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 -- start start 2023-10-30T12:15:45.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 librados: setting wanted keys 2023-10-30T12:15:45.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 librados: calling monclient init 2023-10-30T12:15:45.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient: init 2023-10-30T12:15:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:15:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f4c098520 0x7f9f4c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): picked mon.noname-c con 0x7f9f4c098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f4c098900 0x7f9f4c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): picked mon.noname-a con 0x7f9f4c098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c132740 0x7f9f4c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): picked mon.noname-b con 0x7f9f4c132740 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c096120 con 0x7f9f4c098900 2023-10-30T12:15:45.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:45.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c071df0 con 0x7f9f4c132740 2023-10-30T12:15:45.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:45.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0a3080 con 0x7f9f4c098520 2023-10-30T12:15:45.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:45.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.867+0000 7f9f52a52640 10 monclient(hunting): authenticate will time out at 1791.050981s 2023-10-30T12:15:45.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4b7fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f4c098900 msgr2=0x7f9f4c06eb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:45.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f4c098900 0x7f9f4c06eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:45.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f4c098520 0x7f9f4c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:45.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f4c098520 0x7f9f4c06e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35646/0 (socket says 172.21.15.50:35646) 2023-10-30T12:15:45.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 -- 172.21.15.50:0/1173862460 learned_addr learned my addr 172.21.15.50:0/1173862460 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:45.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): get_auth_request con 0x7f9f4c098520 auth_method 0 2023-10-30T12:15:45.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:45.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:45.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:45.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:45.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:45.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:45.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f50fef640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c132740 0x7f9f4c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:45.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f50fef640 10 monclient(hunting): get_auth_request con 0x7f9f4c132740 auth_method 0 2023-10-30T12:15:45.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f50fef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:45.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f50fef640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f50fef640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient(hunting): handle_auth_done global_id 115109 payload 824 2023-10-30T12:15:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 -- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c132740 msgr2=0x7f9f4c06f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c132740 0x7f9f4c06f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 -- 172.21.15.50:0/1173862460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f4c098900 msgr2=0x7f9f4c06eb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f4c098900 0x7f9f4c06eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:15:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 monclient: found mon.noname-c 2023-10-30T12:15:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 -- 172.21.15.50:0/1173862460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c009460 con 0x7f9f4c098520 2023-10-30T12:15:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4bfff640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f4c098520 0x7f9f4c06e650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c036040 tx=0x7f9f3c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 client.?.objecter ms_handle_connect 0x7f9f4c098520 2023-10-30T12:15:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 1 -- 172.21.15.50:0/1173862460 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c003e50 con 0x7f9f4c098520 2023-10-30T12:15:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: dump: 2023-10-30T12:15:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:45.905 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:45.905 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:45.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: _finish_auth 0 2023-10-30T12:15:45.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:15:45.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:45.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 1 -- 172.21.15.50:0/1173862460 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c004500 con 0x7f9f4c098520 2023-10-30T12:15:45.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:45.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 1 -- 172.21.15.50:0/1173862460 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c006240 con 0x7f9f4c098520 2023-10-30T12:15:45.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 5 monclient: authenticate success, global_id 115109 2023-10-30T12:15:45.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 10 monclient: _renew_subs 2023-10-30T12:15:45.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:45.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4c070a10 con 0x7f9f4c098520 2023-10-30T12:15:45.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:45.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:45.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f4a7fc640 4 set_mon_vals no callback set 2023-10-30T12:15:45.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f48ff9640 10 monclient: dump: 2023-10-30T12:15:45.908 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:45.908 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:45.908 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.908 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:45.908 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:45.909 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:45.909 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:45.909 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:45.909 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:45.909 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 10 client.115109.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 10 monclient: _renew_subs 2023-10-30T12:15:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4c06fdd0 con 0x7f9f4c098520 2023-10-30T12:15:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.868+0000 7f9f52a52640 1 librados: init done 2023-10-30T12:15:45.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f48ff9640 1 -- 172.21.15.50:0/1173862460 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9f3c03a050 con 0x7f9f4c098520 2023-10-30T12:15:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f48ff9640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f2c0392a0 0x7f9f2c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f52a52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f48ff9640 1 -- 172.21.15.50:0/1173862460 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(467..467 src has 417..467) v4 ==== 6579+0+0 (secure 0 0 0) 0x7f9f3c06cd60 con 0x7f9f4c098520 2023-10-30T12:15:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f48ff9640 10 client.115109.objecter ms_dispatch 0x7f9f4c000b90 osd_map(467..467 src has 417..467) v4 2023-10-30T12:15:45.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f52a52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:45.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f48ff9640 3 client.115109.objecter handle_osd_map got epochs [467,467] > 0 2023-10-30T12:15:45.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f48ff9640 3 client.115109.objecter handle_osd_map decoding full epoch 467 2023-10-30T12:15:45.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f52a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:45.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.869+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4c0a1d70 con 0x7f9f4c098520 2023-10-30T12:15:45.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.870+0000 7f9f48ff9640 20 client.115109.objecter dump_active .. 0 homeless 2023-10-30T12:15:45.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.870+0000 7f9f4b7fe640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f2c0392a0 0x7f9f2c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:45.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.870+0000 7f9f4b7fe640 10 monclient: get_auth_request con 0x7f9f2c0392a0 auth_method 0 2023-10-30T12:15:45.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.870+0000 7f9f48ff9640 10 client.115109.objecter ms_handle_connect 0x7f9f2c0392a0 2023-10-30T12:15:45.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.870+0000 7f9f4b7fe640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f2c0392a0 0x7f9f2c03b760 secure :-1 s=READY pgs=4212 cs=0 l=1 rev1=1 crypto rx=0x7f9f34006fd0 tx=0x7f9f34006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:45.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.874+0000 7f9f48ff9640 1 -- 172.21.15.50:0/1173862460 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9f3c038070 con 0x7f9f4c098520 2023-10-30T12:15:45.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.874+0000 7f9f48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:45.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:45.874+0000 7f9f48ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:46.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.121+0000 7f9f52a52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:46.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.121+0000 7f9f52a52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:46.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.121+0000 7f9f52a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:46.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.121+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9f4c04e160 con 0x7f9f4c098520 2023-10-30T12:15:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.121+0000 7f9f48ff9640 1 -- 172.21.15.50:0/1173862460 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f9f3c056080 con 0x7f9f4c098520 2023-10-30T12:15:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.121+0000 7f9f48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:46.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.121+0000 7f9f48ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:46.125 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 10 librados: watch_flush enter 2023-10-30T12:15:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 10 librados: watch_flush exit 2023-10-30T12:15:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 20 client.115109.objecter shutdown clearing up homeless session... 2023-10-30T12:15:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 10 client.115109.objecter successfully canceled tick 2023-10-30T12:15:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f2c0392a0 msgr2=0x7f9f2c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f2c0392a0 0x7f9f2c03b760 secure :-1 s=READY pgs=4212 cs=0 l=1 rev1=1 crypto rx=0x7f9f34006fd0 tx=0x7f9f34006d80 comp rx=0 tx=0).stop 2023-10-30T12:15:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 10 monclient: shutdown 2023-10-30T12:15:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f4c098520 msgr2=0x7f9f4c06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f4c098520 0x7f9f4c06e650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c036040 tx=0x7f9f3c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:15:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 shutdown_connections 2023-10-30T12:15:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9f2c0392a0 0x7f9f2c03b760 unknown :-1 s=CLOSED pgs=4212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f4c132740 0x7f9f4c06f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f4c098900 0x7f9f4c06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 --2- 172.21.15.50:0/1173862460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9f4c098520 0x7f9f4c06e650 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 >> 172.21.15.50:0/1173862460 conn(0x7f9f4c0a05c0 msgr2=0x7f9f4c131fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 shutdown_connections 2023-10-30T12:15:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 -- 172.21.15.50:0/1173862460 wait complete. 2023-10-30T12:15:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:46.123+0000 7f9f52a52640 1 librados: shutdown 2023-10-30T12:15:49.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:49.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.252+0000 7f515493c640 10 monclient: get_monmap_and_config 2023-10-30T12:15:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.252+0000 7f515493c640 10 monclient: build_initial_monmap 2023-10-30T12:15:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.252+0000 7f515493c640 10 monclient: monmap: 2023-10-30T12:15:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:49.254 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:49.253222+0000 2023-10-30T12:15:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:49.253222+0000 2023-10-30T12:15:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:49.255 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:49.256 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:49.256 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:49.256 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:49.256 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:49.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.252+0000 7f515493c640 1 Processor -- start 2023-10-30T12:15:49.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.252+0000 7f515493c640 1 -- start start 2023-10-30T12:15:49.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.252+0000 7f515493c640 10 monclient: init 2023-10-30T12:15:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:15:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51500a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5150099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:49.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5150132db0 0x7f515013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient(hunting): picked mon.noname-b con 0x7f5150132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51500a4f40 con 0x7f51500a4b60 2023-10-30T12:15:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51500a4150 con 0x7f5150099a40 2023-10-30T12:15:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:49.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500a3d70 con 0x7f5150132db0 2023-10-30T12:15:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f515493c640 10 monclient(hunting): authenticate will time out at 1794.436506s 2023-10-30T12:15:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514e59c640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f51500a4b60 legacy=0x7f51501326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5150132db0 0x7f515013e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:49.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5150132db0 0x7f515013e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38778/0 (socket says 172.21.15.50:38778) 2023-10-30T12:15:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 1 -- 172.21.15.50:0/3232314383 learned_addr learned my addr 172.21.15.50:0/3232314383 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): get_auth_request con 0x7f5150132db0 auth_method 0 2023-10-30T12:15:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:49.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 10 monclient(hunting): handle_auth_done global_id 115112 payload 274 2023-10-30T12:15:49.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 1 -- 172.21.15.50:0/3232314383 >> v1:172.21.15.118:6789/0 conn(0x7f5150099a40 legacy=0x7f515013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:49.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.253+0000 7f514ed9d640 1 -- 172.21.15.50:0/3232314383 >> v1:172.21.15.50:6789/0 conn(0x7f51500a4b60 legacy=0x7f51501326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:49.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:15:49.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514ed9d640 1 monclient: found mon.noname-b 2023-10-30T12:15:49.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:49.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514ed9d640 1 -- 172.21.15.50:0/3232314383 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515009a440 con 0x7f5150132db0 2023-10-30T12:15:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514ed9d640 1 --2- 172.21.15.50:0/3232314383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5150132db0 0x7f515013e1b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5148009e80 tx=0x7f514802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86c9e36a45bd5071 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 1 -- 172.21.15.50:0/3232314383 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514802c9c0 con 0x7f5150132db0 2023-10-30T12:15:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: dump: 2023-10-30T12:15:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.271 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:49.272 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:49.272 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:49.272 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:49.272 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:49.272 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: _finish_auth 0 2023-10-30T12:15:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:15:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 1 -- 172.21.15.50:0/3232314383 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514802cb20 con 0x7f5150132db0 2023-10-30T12:15:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:49.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 1 -- 172.21.15.50:0/3232314383 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480343f0 con 0x7f5150132db0 2023-10-30T12:15:49.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f515493c640 5 monclient: authenticate success, global_id 115112 2023-10-30T12:15:49.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f515493c640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:49.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f515493c640 4 set_mon_vals no callback set 2023-10-30T12:15:49.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:49.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.254+0000 7f514d59a640 10 monclient: dump: 2023-10-30T12:15:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.275 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:49.276 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:49.277 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 10 monclient: shutdown 2023-10-30T12:15:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 1 -- 172.21.15.50:0/3232314383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5150132db0 msgr2=0x7f515013e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 1 --2- 172.21.15.50:0/3232314383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5150132db0 0x7f515013e1b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5148009e80 tx=0x7f514802be20 comp rx=0 tx=0).stop 2023-10-30T12:15:49.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 1 -- 172.21.15.50:0/3232314383 shutdown_connections 2023-10-30T12:15:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 1 --2- 172.21.15.50:0/3232314383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5150132db0 0x7f515013e1b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 1 -- 172.21.15.50:0/3232314383 >> 172.21.15.50:0/3232314383 conn(0x7f51500a8060 msgr2=0x7f51500a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 1 -- 172.21.15.50:0/3232314383 shutdown_connections 2023-10-30T12:15:49.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.255+0000 7f515493c640 1 -- 172.21.15.50:0/3232314383 wait complete. 2023-10-30T12:15:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 10 monclient: build_initial_monmap 2023-10-30T12:15:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 10 monclient: monmap: 2023-10-30T12:15:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:49.279 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:49.280 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:49.281 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 1 librados: starting msgr at 2023-10-30T12:15:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 1 librados: starting objecter 2023-10-30T12:15:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 1 Processor -- start 2023-10-30T12:15:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 1 -- start start 2023-10-30T12:15:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 1 librados: setting wanted keys 2023-10-30T12:15:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 1 librados: calling monclient init 2023-10-30T12:15:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.256+0000 7f515493c640 10 monclient: init 2023-10-30T12:15:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:49.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:15:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51500a4b60 0x7f515012db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): picked mon.noname-a con 0x7f51500a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5150132db0 0x7f515012e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): picked mon.noname-b con 0x7f5150132db0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:49.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f515012e610 0x7f51501ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): picked mon.noname-c con 0x7f515012e610 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515012f620 con 0x7f51500a4b60 2023-10-30T12:15:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515012fae0 con 0x7f5150132db0 2023-10-30T12:15:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500a18d0 con 0x7f515012e610 2023-10-30T12:15:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514e59c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51500a4b60 msgr2=0x7f515012db90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient(hunting): authenticate will time out at 1794.440164s 2023-10-30T12:15:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51500a4b60 0x7f515012db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f515012e610 0x7f51501ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f515012e610 0x7f51501ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38780/0 (socket says 172.21.15.50:38780) 2023-10-30T12:15:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 -- 172.21.15.50:0/2676412770 learned_addr learned my addr 172.21.15.50:0/2676412770 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): get_auth_request con 0x7f515012e610 auth_method 0 2023-10-30T12:15:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5150132db0 0x7f515012e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 10 monclient(hunting): get_auth_request con 0x7f5150132db0 auth_method 0 2023-10-30T12:15:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient(hunting): handle_auth_done global_id 115115 payload 824 2023-10-30T12:15:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 -- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5150132db0 msgr2=0x7f515012e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5150132db0 0x7f515012e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 -- 172.21.15.50:0/2676412770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51500a4b60 msgr2=0x7f515012db90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51500a4b60 0x7f515012db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:15:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 monclient: found mon.noname-c 2023-10-30T12:15:49.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 -- 172.21.15.50:0/2676412770 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5148009b30 con 0x7f515012e610 2023-10-30T12:15:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514ed9d640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f515012e610 0x7f51501ca7f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5148009fa0 tx=0x7f5148004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 client.?.objecter ms_handle_connect 0x7f515012e610 2023-10-30T12:15:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 1 -- 172.21.15.50:0/2676412770 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514803e450 con 0x7f515012e610 2023-10-30T12:15:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: dump: 2023-10-30T12:15:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:49.294 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:49.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:49.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:49.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:49.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: _finish_auth 0 2023-10-30T12:15:49.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 1 -- 172.21.15.50:0/2676412770 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514803e5d0 con 0x7f515012e610 2023-10-30T12:15:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 5 monclient: authenticate success, global_id 115115 2023-10-30T12:15:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient: _renew_subs 2023-10-30T12:15:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515013da50 con 0x7f515012e610 2023-10-30T12:15:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 1 -- 172.21.15.50:0/2676412770 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514803e980 con 0x7f515012e610 2023-10-30T12:15:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f513effd640 10 monclient: dump: 2023-10-30T12:15:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:49.298 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.257+0000 7f514cd99640 4 set_mon_vals no callback set 2023-10-30T12:15:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.258+0000 7f515493c640 10 client.115115.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.258+0000 7f515493c640 10 monclient: _renew_subs 2023-10-30T12:15:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.258+0000 7f515493c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.258+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515012ffc0 con 0x7f515012e610 2023-10-30T12:15:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.258+0000 7f515493c640 1 librados: init done 2023-10-30T12:15:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.258+0000 7f513effd640 1 -- 172.21.15.50:0/2676412770 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f514803eae0 con 0x7f515012e610 2023-10-30T12:15:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f515493c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f515493c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f515493c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515004e160 con 0x7f515012e610 2023-10-30T12:15:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f513effd640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51300392d0 0x7f513003b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f513effd640 1 -- 172.21.15.50:0/2676412770 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(467..467 src has 417..467) v4 ==== 6579+0+0 (secure 0 0 0) 0x7f514806da10 con 0x7f515012e610 2023-10-30T12:15:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f513effd640 10 client.115115.objecter ms_dispatch 0x7f5150000b90 osd_map(467..467 src has 417..467) v4 2023-10-30T12:15:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f513effd640 3 client.115115.objecter handle_osd_map got epochs [467,467] > 0 2023-10-30T12:15:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f513effd640 3 client.115115.objecter handle_osd_map decoding full epoch 467 2023-10-30T12:15:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f513effd640 20 client.115115.objecter dump_active .. 0 homeless 2023-10-30T12:15:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f514e59c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51300392d0 0x7f513003b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f514e59c640 10 monclient: get_auth_request con 0x7f51300392d0 auth_method 0 2023-10-30T12:15:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f513effd640 10 client.115115.objecter ms_handle_connect 0x7f51300392d0 2023-10-30T12:15:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.259+0000 7f514e59c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51300392d0 0x7f513003b790 secure :-1 s=READY pgs=4213 cs=0 l=1 rev1=1 crypto rx=0x7f5140006fd0 tx=0x7f5140006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:49.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.263+0000 7f513effd640 1 -- 172.21.15.50:0/2676412770 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5148033d70 con 0x7f515012e610 2023-10-30T12:15:49.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.263+0000 7f513effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:49.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.263+0000 7f513effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.506+0000 7f515493c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.506+0000 7f515493c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.506+0000 7f515493c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.506+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f51500745c0 con 0x7f515012e610 2023-10-30T12:15:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.506+0000 7f513effd640 1 -- 172.21.15.50:0/2676412770 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f5148033b90 con 0x7f515012e610 2023-10-30T12:15:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.506+0000 7f513effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.506+0000 7f513effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:49.509 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:49.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 10 librados: watch_flush enter 2023-10-30T12:15:49.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 10 librados: watch_flush exit 2023-10-30T12:15:49.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 20 client.115115.objecter shutdown clearing up homeless session... 2023-10-30T12:15:49.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 10 client.115115.objecter successfully canceled tick 2023-10-30T12:15:49.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51300392d0 msgr2=0x7f513003b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:49.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51300392d0 0x7f513003b790 secure :-1 s=READY pgs=4213 cs=0 l=1 rev1=1 crypto rx=0x7f5140006fd0 tx=0x7f5140006d80 comp rx=0 tx=0).stop 2023-10-30T12:15:49.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 10 monclient: shutdown 2023-10-30T12:15:49.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:49.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f515012e610 msgr2=0x7f51501ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:49.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f515012e610 0x7f51501ca7f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5148009fa0 tx=0x7f5148004a40 comp rx=0 tx=0).stop 2023-10-30T12:15:49.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 shutdown_connections 2023-10-30T12:15:49.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51300392d0 0x7f513003b790 unknown :-1 s=CLOSED pgs=4213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:49.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f515012e610 0x7f51501ca7f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:49.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5150132db0 0x7f515012e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:49.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 --2- 172.21.15.50:0/2676412770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51500a4b60 0x7f515012db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:49.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 >> 172.21.15.50:0/2676412770 conn(0x7f51500a8060 msgr2=0x7f51500a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:49.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 shutdown_connections 2023-10-30T12:15:49.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 -- 172.21.15.50:0/2676412770 wait complete. 2023-10-30T12:15:49.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:49.508+0000 7f515493c640 1 librados: shutdown 2023-10-30T12:15:52.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:15:52.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.637+0000 7fcbac9e2640 10 monclient: get_monmap_and_config 2023-10-30T12:15:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.637+0000 7fcbac9e2640 10 monclient: build_initial_monmap 2023-10-30T12:15:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.637+0000 7fcbac9e2640 10 monclient: monmap: 2023-10-30T12:15:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:52.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:52.638982+0000 2023-10-30T12:15:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:52.638982+0000 2023-10-30T12:15:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:52.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:52.642 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:52.642 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.638+0000 7fcbac9e2640 1 Processor -- start 2023-10-30T12:15:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.638+0000 7fcbac9e2640 1 -- start start 2023-10-30T12:15:52.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.638+0000 7fcbac9e2640 10 monclient: init 2023-10-30T12:15:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T12:15:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient(hunting): picked mon.noname-b con 0x7fcba8098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcba8098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 0x7fcba8139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient(hunting): picked mon.noname-a con 0x7fcba81379f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba809ad30 con 0x7fcba8098900 2023-10-30T12:15:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:52.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba80993f0 con 0x7fcba81379f0 2023-10-30T12:15:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba809c460 con 0x7fcba8098520 2023-10-30T12:15:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcbac9e2640 10 monclient(hunting): authenticate will time out at 1797.822225s 2023-10-30T12:15:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba6d9d640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 msgr2=0x7fcba8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:52.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba6d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 0x7fcba8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38782/0 (socket says 172.21.15.50:38782) 2023-10-30T12:15:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 learned_addr learned my addr 172.21.15.50:0/1950985557 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcba8098520 auth_method 0 2023-10-30T12:15:52.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:15:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.639+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:52.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba6d9d640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 msgr2=0x7fcba8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:52.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba6d9d640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 0x7fcba8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:15:52.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:52.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcba8098520 auth_method 0 2023-10-30T12:15:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:15:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:52.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:52.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:52.840+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:15:53.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba6d9d640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 msgr2=0x7fcba8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:53.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba6d9d640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 0x7fcba8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:15:53.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:53.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcba8098520 auth_method 0 2023-10-30T12:15:53.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:53.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:53.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:53.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:15:53.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:53.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:53.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:53.241+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:15:54.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba6d9d640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 msgr2=0x7fcba8139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:54.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba6d9d640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 0x7fcba8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:15:54.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:54.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcba8098520 auth_method 0 2023-10-30T12:15:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:15:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:54.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:54.042+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:15:55.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient(hunting): tick 2023-10-30T12:15:55.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:15:55.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:15:55.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:15:55.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 msgr2=0x7fcba809bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:55.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcba8098520 0x7fcba809bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:55.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 msgr2=0x7fcba8139de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:55.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.50:3300/0 conn(0x7fcba81379f0 0x7fcba8139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:55.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 -- 172.21.15.50:0/1950985557 >> v1:172.21.15.118:6789/0 conn(0x7fcba8098900 legacy=0x7fcba8098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:55.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient: _add_conns ranks=[4,0] 2023-10-30T12:15:55.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:55.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient(hunting): picked mon.noname-c con 0x7fcb880058f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:55.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb88008440 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:55.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 -- 172.21.15.50:0/1950985557 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba809b260 con 0x7fcb88008440 2023-10-30T12:15:55.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:55.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 1 -- 172.21.15.50:0/1950985557 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba80993f0 con 0x7fcb880058f0 2023-10-30T12:15:55.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcb97fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:55.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcba5d9b640 1 -- 172.21.15.50:0/1950985557 >> v1:172.21.15.50:6789/0 conn(0x7fcb88008440 legacy=0x7fcb8800a8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:55.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:55.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcb880058f0 auth_method 0 2023-10-30T12:15:55.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:55.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:55.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.639+0000 7fcba659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:55.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.640+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 msgr2=0x7fcb88007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:15:55.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.640+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 msgr2=0x7fcb88007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:55.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.640+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:55.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.640+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:55.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.840+0000 7fcba5d9b640 1 -- 172.21.15.50:0/1950985557 >> v1:172.21.15.50:6789/0 conn(0x7fcb88008440 legacy=0x7fcb8800a8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:55.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcb880058f0 auth_method 0 2023-10-30T12:15:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 msgr2=0x7fcb88007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:15:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 msgr2=0x7fcb88007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:55.841+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:15:56.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.241+0000 7fcba5d9b640 1 -- 172.21.15.50:0/1950985557 >> v1:172.21.15.50:6789/0 conn(0x7fcb88008440 legacy=0x7fcb8800a8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:56.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcb880058f0 auth_method 0 2023-10-30T12:15:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:56.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 msgr2=0x7fcb88007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:15:56.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 msgr2=0x7fcb88007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:15:56.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:15:56.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:56.242+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:15:57.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.042+0000 7fcba5d9b640 1 -- 172.21.15.50:0/1950985557 >> v1:172.21.15.50:6789/0 conn(0x7fcb88008440 legacy=0x7fcb8800a8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:57.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> v2:172.21.15.118:3300/0 conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcb880058f0 auth_method 0 2023-10-30T12:15:57.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:15:57.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient(hunting): handle_auth_done global_id 204166 payload 274 2023-10-30T12:15:57.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 >> v1:172.21.15.50:6789/0 conn(0x7fcb88008440 legacy=0x7fcb8800a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:57.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient: _finish_hunting 0 2023-10-30T12:15:57.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 1 monclient: found mon.noname-c 2023-10-30T12:15:57.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:57.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:15:57.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.043+0000 7fcba659c640 1 -- 172.21.15.50:0/1950985557 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8800d1d0 con 0x7fcb880058f0 2023-10-30T12:15:57.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.044+0000 7fcba659c640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb880058f0 0x7fcb88007d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcb98006f90 tx=0x7fcb9802d700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fc76f657da8ef27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 1 -- 172.21.15.50:0/1950985557 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98004020 con 0x7fcb880058f0 2023-10-30T12:15:57.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:57.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: dump: 2023-10-30T12:15:57.052 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.052 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.052 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.052 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.052 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.053 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.053 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.053 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.053 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.053 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: _finish_auth 0 2023-10-30T12:15:57.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: _check_auth_tickets 2023-10-30T12:15:57.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:57.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 1 -- 172.21.15.50:0/1950985557 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb98030030 con 0x7fcb880058f0 2023-10-30T12:15:57.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:57.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 1 -- 172.21.15.50:0/1950985557 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb980043c0 con 0x7fcb880058f0 2023-10-30T12:15:57.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:57.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba559a640 10 monclient: dump: 2023-10-30T12:15:57.055 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.056 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.056 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.056 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.056 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.057 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 5 monclient: authenticate success, global_id 204166 2023-10-30T12:15:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 4 set_mon_vals no callback set 2023-10-30T12:15:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 10 monclient: shutdown 2023-10-30T12:15:57.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 1 -- 172.21.15.50:0/1950985557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb880058f0 msgr2=0x7fcb88007d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb880058f0 0x7fcb88007d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcb98006f90 tx=0x7fcb9802d700 comp rx=0 tx=0).stop 2023-10-30T12:15:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcba6d9d640 1 -- 172.21.15.50:0/1950985557 reap_dead start 2023-10-30T12:15:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 1 -- 172.21.15.50:0/1950985557 shutdown_connections 2023-10-30T12:15:57.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/1950985557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb880058f0 0x7fcb88007d70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.045+0000 7fcbac9e2640 1 -- 172.21.15.50:0/1950985557 >> 172.21.15.50:0/1950985557 conn(0x7fcba80a05c0 msgr2=0x7fcba809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:57.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.046+0000 7fcbac9e2640 1 -- 172.21.15.50:0/1950985557 shutdown_connections 2023-10-30T12:15:57.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.046+0000 7fcbac9e2640 1 -- 172.21.15.50:0/1950985557 wait complete. 2023-10-30T12:15:57.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.046+0000 7fcbac9e2640 10 monclient: build_initial_monmap 2023-10-30T12:15:57.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 10 monclient: monmap: 2023-10-30T12:15:57.066 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:57.066 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:57.066 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:57.066 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:57.068 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 librados: starting msgr at 2023-10-30T12:15:57.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 librados: starting objecter 2023-10-30T12:15:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 Processor -- start 2023-10-30T12:15:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 -- start start 2023-10-30T12:15:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 librados: setting wanted keys 2023-10-30T12:15:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 librados: calling monclient init 2023-10-30T12:15:57.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 10 monclient: init 2023-10-30T12:15:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:15:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb880058f0 0x7fcba81ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 10 monclient(hunting): picked mon.noname-c con 0x7fcb880058f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8096420 0x7fcba8096860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 10 monclient(hunting): picked mon.noname-b con 0x7fcba8096420 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcba8091060 0x7fcba80914f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.047+0000 7fcbac9e2640 10 monclient(hunting): picked mon.noname-a con 0x7fcba8091060 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:57.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81378b0 con 0x7fcba8091060 2023-10-30T12:15:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81cada0 con 0x7fcba8096420 2023-10-30T12:15:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba8097e80 con 0x7fcb880058f0 2023-10-30T12:15:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcbac9e2640 10 monclient(hunting): authenticate will time out at 1802.231032s 2023-10-30T12:15:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba6d9d640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcba8091060 msgr2=0x7fcba80914f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:57.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcba8091060 0x7fcba80914f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb880058f0 0x7fcba81ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb880058f0 0x7fcba81ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38808/0 (socket says 172.21.15.50:38808) 2023-10-30T12:15:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 -- 172.21.15.50:0/3660518055 learned_addr learned my addr 172.21.15.50:0/3660518055 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:57.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): get_auth_request con 0x7fcb880058f0 auth_method 0 2023-10-30T12:15:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8096420 0x7fcba8096860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 10 monclient(hunting): get_auth_request con 0x7fcba8096420 auth_method 0 2023-10-30T12:15:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient(hunting): handle_auth_done global_id 115118 payload 824 2023-10-30T12:15:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 -- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8096420 msgr2=0x7fcba8096860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8096420 0x7fcba8096860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 -- 172.21.15.50:0/3660518055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcba8091060 msgr2=0x7fcba80914f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcba8091060 0x7fcba80914f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient: _finish_hunting 0 2023-10-30T12:15:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 monclient: found mon.noname-c 2023-10-30T12:15:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 -- 172.21.15.50:0/3660518055 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb98018810 con 0x7fcb880058f0 2023-10-30T12:15:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcba659c640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb880058f0 0x7fcba81ca0a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcb98006fd0 tx=0x7fcb98004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcb96ffd640 10 client.?.objecter ms_handle_connect 0x7fcb880058f0 2023-10-30T12:15:57.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcb96ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcb96ffd640 1 -- 172.21.15.50:0/3660518055 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98003c30 con 0x7fcb880058f0 2023-10-30T12:15:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcb96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcb96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.048+0000 7fcb96ffd640 10 monclient: dump: 2023-10-30T12:15:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.083 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 10 monclient: _finish_auth 0 2023-10-30T12:15:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 1 -- 172.21.15.50:0/3660518055 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9800c040 con 0x7fcb880058f0 2023-10-30T12:15:57.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 5 monclient: authenticate success, global_id 115118 2023-10-30T12:15:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 10 monclient: _renew_subs 2023-10-30T12:15:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba8093760 con 0x7fcb880058f0 2023-10-30T12:15:57.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 1 -- 172.21.15.50:0/3660518055 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98005990 con 0x7fcb880058f0 2023-10-30T12:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 10 monclient: dump: 2023-10-30T12:15:57.086 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.087 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.088 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.089 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb97fff640 4 set_mon_vals no callback set 2023-10-30T12:15:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 10 client.115118.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 10 monclient: _renew_subs 2023-10-30T12:15:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba8071df0 con 0x7fcb880058f0 2023-10-30T12:15:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcbac9e2640 1 librados: init done 2023-10-30T12:15:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.049+0000 7fcb96ffd640 1 -- 172.21.15.50:0/3660518055 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcb98005af0 con 0x7fcb880058f0 2023-10-30T12:15:57.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcb96ffd640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb8803a0f0 0x7fcb8803c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcb96ffd640 1 -- 172.21.15.50:0/3660518055 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(469..469 src has 417..469) v4 ==== 6971+0+0 (secure 0 0 0) 0x7fcb9806e5b0 con 0x7fcb880058f0 2023-10-30T12:15:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcb96ffd640 10 client.115118.objecter ms_dispatch 0x7fcba8000b90 osd_map(469..469 src has 417..469) v4 2023-10-30T12:15:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcb96ffd640 3 client.115118.objecter handle_osd_map got epochs [469,469] > 0 2023-10-30T12:15:57.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcb96ffd640 3 client.115118.objecter handle_osd_map decoding full epoch 469 2023-10-30T12:15:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcb96ffd640 20 client.115118.objecter dump_active .. 0 homeless 2023-10-30T12:15:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcbac9e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcbac9e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcbac9e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba804e160 con 0x7fcb880058f0 2023-10-30T12:15:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcba5d9b640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb8803a0f0 0x7fcb8803c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcba5d9b640 10 monclient: get_auth_request con 0x7fcb8803a0f0 auth_method 0 2023-10-30T12:15:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcba5d9b640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb8803a0f0 0x7fcb8803c5b0 secure :-1 s=READY pgs=4215 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007b40 tx=0x7fcb9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.050+0000 7fcb96ffd640 10 client.115118.objecter ms_handle_connect 0x7fcb8803a0f0 2023-10-30T12:15:57.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.055+0000 7fcb96ffd640 1 -- 172.21.15.50:0/3660518055 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcb9800f040 con 0x7fcb880058f0 2023-10-30T12:15:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.055+0000 7fcb96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.055+0000 7fcb96ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:57.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.299+0000 7fcbac9e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:15:57.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.299+0000 7fcbac9e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:57.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.299+0000 7fcbac9e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.299+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcba8094650 con 0x7fcb880058f0 2023-10-30T12:15:57.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.299+0000 7fcb96ffd640 1 -- 172.21.15.50:0/3660518055 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fcb98035ba0 con 0x7fcb880058f0 2023-10-30T12:15:57.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.299+0000 7fcb96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:15:57.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.299+0000 7fcb96ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:57.303 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":158,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:15:57.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 10 librados: watch_flush enter 2023-10-30T12:15:57.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 10 librados: watch_flush exit 2023-10-30T12:15:57.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 20 client.115118.objecter shutdown clearing up homeless session... 2023-10-30T12:15:57.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 10 client.115118.objecter successfully canceled tick 2023-10-30T12:15:57.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb8803a0f0 msgr2=0x7fcb8803c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb8803a0f0 0x7fcb8803c5b0 secure :-1 s=READY pgs=4215 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007b40 tx=0x7fcb9c009040 comp rx=0 tx=0).stop 2023-10-30T12:15:57.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 10 monclient: shutdown 2023-10-30T12:15:57.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:57.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb880058f0 msgr2=0x7fcba81ca0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb880058f0 0x7fcba81ca0a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcb98006fd0 tx=0x7fcb98004730 comp rx=0 tx=0).stop 2023-10-30T12:15:57.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 shutdown_connections 2023-10-30T12:15:57.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb8803a0f0 0x7fcb8803c5b0 unknown :-1 s=CLOSED pgs=4215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcba8091060 0x7fcba80914f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8096420 0x7fcba8096860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 --2- 172.21.15.50:0/3660518055 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb880058f0 0x7fcba81ca0a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 >> 172.21.15.50:0/3660518055 conn(0x7fcba80a05c0 msgr2=0x7fcba809dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:57.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 shutdown_connections 2023-10-30T12:15:57.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 -- 172.21.15.50:0/3660518055 wait complete. 2023-10-30T12:15:57.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.301+0000 7fcbac9e2640 1 librados: shutdown 2023-10-30T12:15:57.313 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:15:57.313 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:15:57.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:15:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.418+0000 7f24b73a5640 10 monclient: get_monmap_and_config 2023-10-30T12:15:57.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.418+0000 7f24b73a5640 10 monclient: build_initial_monmap 2023-10-30T12:15:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.418+0000 7f24b73a5640 10 monclient: monmap: 2023-10-30T12:15:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:57.419091+0000 2023-10-30T12:15:57.420 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:57.419091+0000 2023-10-30T12:15:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:57.421 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:57.422 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:57.422 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.418+0000 7f24b73a5640 1 Processor -- start 2023-10-30T12:15:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.418+0000 7f24b73a5640 1 -- start start 2023-10-30T12:15:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.418+0000 7f24b73a5640 10 monclient: init 2023-10-30T12:15:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:15:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f24b0132970 0x7f24b0132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): picked mon.noname-a con 0x7f24b0132970 addr v2:172.21.15.50:3300/0 2023-10-30T12:15:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b00707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): picked mon.noname-b con 0x7f24b00a03d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f24b0070d10 0x7f24b0071140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): picked mon.noname-c con 0x7f24b0070d10 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:57.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b0071680 con 0x7f24b0132970 2023-10-30T12:15:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b00a17d0 con 0x7f24b0070d10 2023-10-30T12:15:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00a1510 con 0x7f24b00a03d0 2023-10-30T12:15:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b73a5640 10 monclient(hunting): authenticate will time out at 1802.602229s 2023-10-30T12:15:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b5141640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f24b0132970 msgr2=0x7f24b0132d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b5141640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f24b0132970 0x7f24b0132d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:57.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b00707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b00707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38816/0 (socket says 172.21.15.50:38816) 2023-10-30T12:15:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 -- 172.21.15.50:0/3624760490 learned_addr learned my addr 172.21.15.50:0/3624760490 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): get_auth_request con 0x7f24b00a03d0 auth_method 0 2023-10-30T12:15:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b5942640 1 --2- 172.21.15.50:0/3624760490 >> v2:172.21.15.118:3300/0 conn(0x7f24b0070d10 0x7f24b0071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b5942640 10 monclient(hunting): get_auth_request con 0x7f24b0070d10 auth_method 0 2023-10-30T12:15:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b5942640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b5942640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b5942640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient(hunting): handle_auth_done global_id 115121 payload 274 2023-10-30T12:15:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 -- 172.21.15.50:0/3624760490 >> v2:172.21.15.118:3300/0 conn(0x7f24b0070d10 msgr2=0x7f24b0071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:15:57.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 --2- 172.21.15.50:0/3624760490 >> v2:172.21.15.118:3300/0 conn(0x7f24b0070d10 0x7f24b0071140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 -- 172.21.15.50:0/3624760490 >> v2:172.21.15.50:3300/0 conn(0x7f24b0132970 msgr2=0x7f24b0132d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:57.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 --2- 172.21.15.50:0/3624760490 >> v2:172.21.15.50:3300/0 conn(0x7f24b0132970 0x7f24b0132d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient: _finish_hunting 0 2023-10-30T12:15:57.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 monclient: found mon.noname-b 2023-10-30T12:15:57.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:57.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.419+0000 7f24b4940640 1 -- 172.21.15.50:0/3624760490 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b009f680 con 0x7f24b00a03d0 2023-10-30T12:15:57.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b4940640 1 --2- 172.21.15.50:0/3624760490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b00707d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f24b00a1c90 tx=0x7f24a002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d276c0c19dde7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 1 -- 172.21.15.50:0/3624760490 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0031760 con 0x7f24b00a03d0 2023-10-30T12:15:57.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:15:57.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: dump: 2023-10-30T12:15:57.440 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.440 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.440 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.440 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.440 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.441 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.441 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.441 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.441 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.441 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: _finish_auth 0 2023-10-30T12:15:57.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:15:57.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:57.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 1 -- 172.21.15.50:0/3624760490 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a00318c0 con 0x7f24b00a03d0 2023-10-30T12:15:57.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 5 monclient: authenticate success, global_id 115121 2023-10-30T12:15:57.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:15:57.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:57.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:57.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 1 -- 172.21.15.50:0/3624760490 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0031c50 con 0x7f24b00a03d0 2023-10-30T12:15:57.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 4 set_mon_vals no callback set 2023-10-30T12:15:57.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:57.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24a7fff640 10 monclient: dump: 2023-10-30T12:15:57.444 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.444 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.444 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.444 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.445 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.445 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.445 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.445 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.445 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.445 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 10 monclient: shutdown 2023-10-30T12:15:57.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:57.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 1 -- 172.21.15.50:0/3624760490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 msgr2=0x7f24b00707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.420+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3624760490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b00707d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f24b00a1c90 tx=0x7f24a002b800 comp rx=0 tx=0).stop 2023-10-30T12:15:57.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 -- 172.21.15.50:0/3624760490 shutdown_connections 2023-10-30T12:15:57.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3624760490 >> v2:172.21.15.118:3300/0 conn(0x7f24b0070d10 0x7f24b0071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3624760490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b00707d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3624760490 >> v2:172.21.15.50:3300/0 conn(0x7f24b0132970 0x7f24b0132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 -- 172.21.15.50:0/3624760490 >> 172.21.15.50:0/3624760490 conn(0x7f24b00a4800 msgr2=0x7f24b00a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:57.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 -- 172.21.15.50:0/3624760490 shutdown_connections 2023-10-30T12:15:57.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 -- 172.21.15.50:0/3624760490 wait complete. 2023-10-30T12:15:57.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 10 monclient: build_initial_monmap 2023-10-30T12:15:57.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 10 monclient: monmap: 2023-10-30T12:15:57.448 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:57.448 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:57.449 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:57.449 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:57.449 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:57.449 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.449 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:57.449 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:57.450 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:57.450 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 librados: starting msgr at 2023-10-30T12:15:57.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 librados: starting objecter 2023-10-30T12:15:57.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.421+0000 7f24b73a5640 1 Processor -- start 2023-10-30T12:15:57.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 -- start start 2023-10-30T12:15:57.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 librados: setting wanted keys 2023-10-30T12:15:57.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 librados: calling monclient init 2023-10-30T12:15:57.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient: init 2023-10-30T12:15:57.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:57.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:15:57.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b0070d10 0x7f24b01ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): picked mon.noname-c con 0x7f24b0070d10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:57.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24b0132970 0x7f24b01cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): picked mon.noname-b con 0x7f24b0132970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:57.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24b011f7a0 0x7f24b011fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): picked mon.noname-a con 0x7f24b011f7a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:57.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0123350 con 0x7f24b011f7a0 2023-10-30T12:15:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0123c30 con 0x7f24b0132970 2023-10-30T12:15:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0131550 con 0x7f24b0070d10 2023-10-30T12:15:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b73a5640 10 monclient(hunting): authenticate will time out at 1802.605497s 2023-10-30T12:15:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5942640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24b011f7a0 msgr2=0x7f24b011fc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5942640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24b011f7a0 0x7f24b011fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b0070d10 0x7f24b01ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b0070d10 0x7f24b01ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38830/0 (socket says 172.21.15.50:38830) 2023-10-30T12:15:57.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 -- 172.21.15.50:0/3021535024 learned_addr learned my addr 172.21.15.50:0/3021535024 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:57.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): get_auth_request con 0x7f24b0070d10 auth_method 0 2023-10-30T12:15:57.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient(hunting): handle_auth_done global_id 115124 payload 824 2023-10-30T12:15:57.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24b0132970 msgr2=0x7f24b01cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24b0132970 0x7f24b01cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24b011f7a0 msgr2=0x7f24b011fc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:57.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24b011f7a0 0x7f24b011fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient: _finish_hunting 0 2023-10-30T12:15:57.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 monclient: found mon.noname-c 2023-10-30T12:15:57.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:57.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 -- 172.21.15.50:0/3021535024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a00094d0 con 0x7f24b0070d10 2023-10-30T12:15:57.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24b5141640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b0070d10 0x7f24b01ca8b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00b550 tx=0x7f24ac00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24a5ffb640 10 client.?.objecter ms_handle_connect 0x7f24b0070d10 2023-10-30T12:15:57.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.422+0000 7f24a5ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:57.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac003dc0 con 0x7f24b0070d10 2023-10-30T12:15:57.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:57.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: dump: 2023-10-30T12:15:57.459 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.459 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.459 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.460 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.460 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.460 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.460 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.460 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.460 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.461 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:15:57.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:15:57.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:57.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac005070 con 0x7f24b0070d10 2023-10-30T12:15:57.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 5 monclient: authenticate success, global_id 115124 2023-10-30T12:15:57.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 10 monclient: _renew_subs 2023-10-30T12:15:57.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b0123810 con 0x7f24b0070d10 2023-10-30T12:15:57.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:57.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac005420 con 0x7f24b0070d10 2023-10-30T12:15:57.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:57.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a5ffb640 10 monclient: dump: 2023-10-30T12:15:57.463 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.463 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.463 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.463 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.464 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.464 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.464 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.464 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.464 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.464 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24a77fe640 4 set_mon_vals no callback set 2023-10-30T12:15:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 10 client.115124.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 10 monclient: _renew_subs 2023-10-30T12:15:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b0122f70 con 0x7f24b0070d10 2023-10-30T12:15:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.423+0000 7f24b73a5640 1 librados: init done 2023-10-30T12:15:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f24ac005580 con 0x7f24b0070d10 2023-10-30T12:15:57.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b73a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:57.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b0120190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0071a70 con 0x7f24b00a03d0 2023-10-30T12:15:57.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24940392a0 0x7f249403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(469..469 src has 417..469) v4 ==== 6971+0+0 (secure 0 0 0) 0x7f24ac04ad40 con 0x7f24b0070d10 2023-10-30T12:15:57.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 10 client.115124.objecter ms_dispatch 0x7f24b0000b90 osd_map(469..469 src has 417..469) v4 2023-10-30T12:15:57.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 3 client.115124.objecter handle_osd_map got epochs [469,469] > 0 2023-10-30T12:15:57.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 3 client.115124.objecter handle_osd_map decoding full epoch 469 2023-10-30T12:15:57.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b0120190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient: get_auth_request con 0x7f24b00a03d0 auth_method 0 2023-10-30T12:15:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 20 client.115124.objecter dump_active .. 0 homeless 2023-10-30T12:15:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b5942640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24940392a0 0x7f249403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b5942640 10 monclient: get_auth_request con 0x7f24940392a0 auth_method 0 2023-10-30T12:15:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 10 monclient(hunting): handle_auth_done global_id 115127 payload 824 2023-10-30T12:15:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 1 -- 172.21.15.50:0/3021535024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b009f960 con 0x7f24b00a03d0 2023-10-30T12:15:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 10 client.115124.objecter ms_handle_connect 0x7f24b00a03d0 2023-10-30T12:15:57.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 10 client.115124.objecter resend_mon_ops 2023-10-30T12:15:57.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b4940640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b0120190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f24b009aef0 tx=0x7f24a0031c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 10 client.115124.objecter ms_handle_connect 0x7f24940392a0 2023-10-30T12:15:57.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24b5942640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24940392a0 0x7f249403b760 secure :-1 s=READY pgs=4216 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009660 tx=0x7f24a80098b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a003b600 con 0x7f24b00a03d0 2023-10-30T12:15:57.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:57.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.424+0000 7f24a5ffb640 10 monclient: dump: 2023-10-30T12:15:57.471 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.471 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.471 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.471 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.472 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.472 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.472 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.472 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.472 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.472 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.425+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f24a003b760 con 0x7f24b00a03d0 2023-10-30T12:15:57.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.425+0000 7f24a5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:57.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.425+0000 7f24a5ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:57.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.425+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 msgr2=0x7f24b0120190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.425+0000 7f24a5ffb640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b0120190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f24b009aef0 tx=0x7f24a0031c60 comp rx=0 tx=0).stop 2023-10-30T12:15:57.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.425+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 msgr2=0x7f24b0120190 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:57.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.425+0000 7f24a5ffb640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b00a03d0 0x7f24b0120190 secure :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f24b009aef0 tx=0x7f24a0031c60 comp rx=0 tx=0).stop 2023-10-30T12:15:57.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b73a5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:57.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b01206d0 0x7f24b011c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b73a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0074800 con 0x7f24b01206d0 2023-10-30T12:15:57.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b01206d0 0x7f24b011c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient: get_auth_request con 0x7f24b01206d0 auth_method 0 2023-10-30T12:15:57.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 10 monclient(hunting): handle_auth_done global_id 115130 payload 824 2023-10-30T12:15:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 1 -- 172.21.15.50:0/3021535024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f24b0123650 con 0x7f24b01206d0 2023-10-30T12:15:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24b4940640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b01206d0 0x7f24b011c440 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f24a0043000 tx=0x7f24a0031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24a5ffb640 10 client.115124.objecter ms_handle_connect 0x7f24b01206d0 2023-10-30T12:15:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.431+0000 7f24a5ffb640 10 client.115124.objecter resend_mon_ops 2023-10-30T12:15:57.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0006530 con 0x7f24b01206d0 2023-10-30T12:15:57.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:57.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 10 monclient: dump: 2023-10-30T12:15:57.479 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.479 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.479 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.480 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.480 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.480 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.480 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.480 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.480 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.481 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4791 (secure 0 0 0) 0x7f24a0006690 con 0x7f24b01206d0 2023-10-30T12:15:57.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:57.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:57.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b01206d0 msgr2=0x7f24b011c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b01206d0 0x7f24b011c440 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f24a0043000 tx=0x7f24a0031000 comp rx=0 tx=0).stop 2023-10-30T12:15:57.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b01206d0 msgr2=0x7f24b011c440 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:57.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24a5ffb640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b01206d0 0x7f24b011c440 secure :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f24a0043000 tx=0x7f24a0031000 comp rx=0 tx=0).stop 2023-10-30T12:15:57.482 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:15:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 158, 2023-10-30T12:15:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:15:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:15:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:15:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:15:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:57.513 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:57.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 10 librados: watch_flush enter 2023-10-30T12:15:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 10 librados: watch_flush exit 2023-10-30T12:15:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 20 client.115124.objecter shutdown clearing up homeless session... 2023-10-30T12:15:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 10 client.115124.objecter successfully canceled tick 2023-10-30T12:15:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24940392a0 msgr2=0x7f249403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24940392a0 0x7f249403b760 secure :-1 s=READY pgs=4216 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009660 tx=0x7f24a80098b0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 10 monclient: shutdown 2023-10-30T12:15:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b0070d10 msgr2=0x7f24b01ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 --2- 172.21.15.50:0/3021535024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24b0070d10 0x7f24b01ca8b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00b550 tx=0x7f24ac00ba10 comp rx=0 tx=0).stop 2023-10-30T12:15:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b5942640 1 -- 172.21.15.50:0/3021535024 reap_dead start 2023-10-30T12:15:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b5942640 1 -- 172.21.15.50:0/3021535024 reap_dead start 2023-10-30T12:15:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 shutdown_connections 2023-10-30T12:15:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 >> 172.21.15.50:0/3021535024 conn(0x7f24b00a4800 msgr2=0x7f24b00a34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 shutdown_connections 2023-10-30T12:15:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 -- 172.21.15.50:0/3021535024 wait complete. 2023-10-30T12:15:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.432+0000 7f24b73a5640 1 librados: shutdown 2023-10-30T12:15:57.519 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:15:57.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:15:57.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:15:57.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.625+0000 7f8d7e55a640 10 monclient: get_monmap_and_config 2023-10-30T12:15:57.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.625+0000 7f8d7e55a640 10 monclient: build_initial_monmap 2023-10-30T12:15:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.625+0000 7f8d7e55a640 10 monclient: monmap: 2023-10-30T12:15:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:15:57.626160+0000 2023-10-30T12:15:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:15:57.626160+0000 2023-10-30T12:15:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:15:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:15:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:15:57.629 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:15:57.629 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:15:57.629 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.625+0000 7f8d7e55a640 1 Processor -- start 2023-10-30T12:15:57.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.625+0000 7f8d7e55a640 1 -- start start 2023-10-30T12:15:57.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.625+0000 7f8d7e55a640 10 monclient: init 2023-10-30T12:15:57.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:57.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-10-30T12:15:57.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d780a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:15:57.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d780a2450 addr v1:172.21.15.50:6789/0 2023-10-30T12:15:57.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8d7813c120 0x7f8d7813e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7813c120 addr v2:172.21.15.118:3300/0 2023-10-30T12:15:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d780a4190 con 0x7f8d780a2450 2023-10-30T12:15:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7809a2e0 con 0x7f8d780a2ae0 2023-10-30T12:15:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d780a1630 con 0x7f8d7813c120 2023-10-30T12:15:57.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:57.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7e55a640 10 monclient(hunting): authenticate will time out at 1802.809345s 2023-10-30T12:15:57.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d777fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8d780a2450 legacy=0x7f8d7813aa00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:15:57.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8d7813c120 0x7f8d7813e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8d7813c120 0x7f8d7813e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:41652/0 (socket says 172.21.15.50:41652) 2023-10-30T12:15:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d77fff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f8d780a2ae0 0x7f8d780a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:54884/0 (socket says 172.21.15.50:54884) 2023-10-30T12:15:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 1 -- 172.21.15.50:0/1477187056 learned_addr learned my addr 172.21.15.50:0/1477187056 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 10 monclient(hunting): get_auth_request con 0x7f8d7813c120 auth_method 0 2023-10-30T12:15:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.626+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_done global_id 204175 payload 274 2023-10-30T12:15:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 1 -- 172.21.15.50:0/1477187056 >> v1:172.21.15.118:6789/0 conn(0x7f8d780a2ae0 legacy=0x7f8d780a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 1 -- 172.21.15.50:0/1477187056 >> v1:172.21.15.50:6789/0 conn(0x7f8d780a2450 legacy=0x7f8d7813aa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:15:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 10 monclient: _finish_hunting 0 2023-10-30T12:15:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 1 monclient: found mon.noname-c 2023-10-30T12:15:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:15:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 1 -- 172.21.15.50:0/1477187056 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7809a8a0 con 0x7f8d7813c120 2023-10-30T12:15:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d7caf7640 1 --2- 172.21.15.50:0/1477187056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d7813c120 0x7f8d7813e510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009e50 tx=0x7f8d7002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab19c6a3a6430eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d76ffd640 1 -- 172.21.15.50:0/1477187056 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7002c9f0 con 0x7f8d7813c120 2023-10-30T12:15:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.627+0000 7f8d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:57.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: dump: 2023-10-30T12:15:57.645 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.645 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.645 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.645 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.646 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.646 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.646 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.646 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.646 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.647 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: _finish_auth 0 2023-10-30T12:15:57.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:15:57.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:57.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 1 -- 172.21.15.50:0/1477187056 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7002cb50 con 0x7f8d7813c120 2023-10-30T12:15:57.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:57.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 1 -- 172.21.15.50:0/1477187056 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d700343f0 con 0x7f8d7813c120 2023-10-30T12:15:57.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 5 monclient: authenticate success, global_id 204175 2023-10-30T12:15:57.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 10 monclient: get_monmap_and_config success 2023-10-30T12:15:57.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 4 set_mon_vals no callback set 2023-10-30T12:15:57.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:57.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d76ffd640 10 monclient: dump: 2023-10-30T12:15:57.649 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.649 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.649 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.650 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.650 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.650 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 10 monclient: shutdown 2023-10-30T12:15:57.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:57.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 1 -- 172.21.15.50:0/1477187056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d7813c120 msgr2=0x7f8d7813e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 1 --2- 172.21.15.50:0/1477187056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d7813c120 0x7f8d7813e510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009e50 tx=0x7f8d7002be50 comp rx=0 tx=0).stop 2023-10-30T12:15:57.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 1 -- 172.21.15.50:0/1477187056 shutdown_connections 2023-10-30T12:15:57.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 1 --2- 172.21.15.50:0/1477187056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d7813c120 0x7f8d7813e510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 1 -- 172.21.15.50:0/1477187056 >> 172.21.15.50:0/1477187056 conn(0x7f8d780a80b0 msgr2=0x7f8d780a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:57.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 1 -- 172.21.15.50:0/1477187056 shutdown_connections 2023-10-30T12:15:57.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.628+0000 7f8d7e55a640 1 -- 172.21.15.50:0/1477187056 wait complete. 2023-10-30T12:15:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 10 monclient: build_initial_monmap 2023-10-30T12:15:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 10 monclient: monmap: 2023-10-30T12:15:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:15:57.653 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:15:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:15:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:15:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:15:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.654 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:15:57.655 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:15:57.655 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:15:57.655 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 1 librados: starting msgr at 2023-10-30T12:15:57.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 1 librados: starting objecter 2023-10-30T12:15:57.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 1 Processor -- start 2023-10-30T12:15:57.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 1 -- start start 2023-10-30T12:15:57.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 1 librados: setting wanted keys 2023-10-30T12:15:57.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 1 librados: calling monclient init 2023-10-30T12:15:57.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 10 monclient: init 2023-10-30T12:15:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 10 monclient: _reopen_session rank -1 2023-10-30T12:15:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:15:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d780a2450 0x7f8d781ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.629+0000 7f8d7e55a640 10 monclient(hunting): picked mon.noname-b con 0x7f8d780a2450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:15:57.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d780a2ae0 0x7f8d781ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient(hunting): picked mon.noname-a con 0x7f8d780a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:15:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8d7813c120 0x7f8d781cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7813c120 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:15:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7813ab20 con 0x7f8d780a2ae0 2023-10-30T12:15:57.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d78133050 con 0x7f8d780a2450 2023-10-30T12:15:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7813a8d0 con 0x7f8d7813c120 2023-10-30T12:15:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient(hunting): _renew_subs 2023-10-30T12:15:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient(hunting): authenticate will time out at 1802.813048s 2023-10-30T12:15:57.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d777fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d780a2ae0 msgr2=0x7f8d781ca7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:15:57.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d777fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d780a2ae0 0x7f8d781ca7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:15:57.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8d7813c120 0x7f8d781cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8d7813c120 0x7f8d781cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57582/0 (socket says 172.21.15.50:57582) 2023-10-30T12:15:57.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 -- 172.21.15.50:0/2318324671 learned_addr learned my addr 172.21.15.50:0/2318324671 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:15:57.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): get_auth_request con 0x7f8d7813c120 auth_method 0 2023-10-30T12:15:57.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d780a2450 0x7f8d781ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 10 monclient(hunting): get_auth_request con 0x7f8d780a2450 auth_method 0 2023-10-30T12:15:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient(hunting): handle_auth_done global_id 115133 payload 824 2023-10-30T12:15:57.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d780a2450 msgr2=0x7f8d781ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d780a2450 0x7f8d781ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d780a2ae0 msgr2=0x7f8d781ca7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:15:57.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d780a2ae0 0x7f8d781ca7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient: _finish_hunting 0 2023-10-30T12:15:57.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 monclient: found mon.noname-c 2023-10-30T12:15:57.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:15:57.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 -- 172.21.15.50:0/2318324671 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70009b30 con 0x7f8d7813c120 2023-10-30T12:15:57.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7caf7640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8d7813c120 0x7f8d781cad20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009f70 tx=0x7f8d70004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 client.?.objecter ms_handle_connect 0x7f8d7813c120 2023-10-30T12:15:57.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:15:57.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7003e450 con 0x7f8d7813c120 2023-10-30T12:15:57.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:15:57.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: dump: 2023-10-30T12:15:57.666 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.667 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.667 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.667 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.667 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.667 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.667 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.668 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.668 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.668 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: _finish_auth 0 2023-10-30T12:15:57.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:15:57.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:15:57.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7003e5d0 con 0x7f8d7813c120 2023-10-30T12:15:57.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 5 monclient: authenticate success, global_id 115133 2023-10-30T12:15:57.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient: _renew_subs 2023-10-30T12:15:57.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d780a1790 con 0x7f8d7813c120 2023-10-30T12:15:57.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:15:57.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7003e980 con 0x7f8d7813c120 2023-10-30T12:15:57.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:15:57.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d74ff9640 10 monclient: dump: 2023-10-30T12:15:57.671 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.671 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.671 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.671 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.672 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.672 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.672 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.672 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.672 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.672 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d767fc640 4 set_mon_vals no callback set 2023-10-30T12:15:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 client.115133.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:15:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient: _renew_subs 2023-10-30T12:15:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:15:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d78138750 con 0x7f8d7813c120 2023-10-30T12:15:57.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.630+0000 7f8d7e55a640 1 librados: init done 2023-10-30T12:15:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8d7003eae0 con 0x7f8d7813c120 2023-10-30T12:15:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d74ff9640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8d580392a0 0x7f8d5803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d7e55a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d7e55a640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d781332d0 0x7f8d781336b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d7e55a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7813acb0 con 0x7f8d781332d0 2023-10-30T12:15:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(469..469 src has 417..469) v4 ==== 6971+0+0 (secure 0 0 0) 0x7f8d7006dad0 con 0x7f8d7813c120 2023-10-30T12:15:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d74ff9640 10 client.115133.objecter ms_dispatch 0x7f8d78000b90 osd_map(469..469 src has 417..469) v4 2023-10-30T12:15:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d74ff9640 3 client.115133.objecter handle_osd_map got epochs [469,469] > 0 2023-10-30T12:15:57.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d74ff9640 3 client.115133.objecter handle_osd_map decoding full epoch 469 2023-10-30T12:15:57.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.631+0000 7f8d74ff9640 20 client.115133.objecter dump_active .. 0 homeless 2023-10-30T12:15:57.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d781332d0 0x7f8d781336b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient: get_auth_request con 0x7f8d781332d0 auth_method 0 2023-10-30T12:15:57.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d77fff640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8d580392a0 0x7f8d5803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d77fff640 10 monclient: get_auth_request con 0x7f8d580392a0 auth_method 0 2023-10-30T12:15:57.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 10 monclient(hunting): handle_auth_done global_id 204184 payload 824 2023-10-30T12:15:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 1 -- 172.21.15.50:0/2318324671 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d7809a5c0 con 0x7f8d781332d0 2023-10-30T12:15:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d777fe640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d781332d0 0x7f8d781336b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d780a0450 tx=0x7f8d6000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d77fff640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8d580392a0 0x7f8d5803b760 secure :-1 s=READY pgs=4217 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000f60 tx=0x7f8d68000ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d74ff9640 10 client.115133.objecter ms_handle_connect 0x7f8d781332d0 2023-10-30T12:15:57.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d74ff9640 10 client.115133.objecter resend_mon_ops 2023-10-30T12:15:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d74ff9640 10 client.115133.objecter ms_handle_connect 0x7f8d580392a0 2023-10-30T12:15:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60003e10 con 0x7f8d781332d0 2023-10-30T12:15:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.632+0000 7f8d74ff9640 10 monclient: dump: 2023-10-30T12:15:57.679 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.680 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.681 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.681 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.681 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.633+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8d60005070 con 0x7f8d781332d0 2023-10-30T12:15:57.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.633+0000 7f8d74ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:15:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.633+0000 7f8d74ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:15:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.633+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d781332d0 msgr2=0x7f8d781336b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.633+0000 7f8d74ff9640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d781332d0 0x7f8d781336b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d780a0450 tx=0x7f8d6000b8c0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.633+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d781332d0 msgr2=0x7f8d781336b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:57.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.633+0000 7f8d74ff9640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d781332d0 0x7f8d781336b0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d780a0450 tx=0x7f8d6000b8c0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d7e55a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:15:57.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d7e55a640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d78133bf0 0x7f8d78136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:15:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d7e55a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:15:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d78073ff0 con 0x7f8d78133bf0 2023-10-30T12:15:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d78133bf0 0x7f8d78136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:15:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient: get_auth_request con 0x7f8d78133bf0 auth_method 0 2023-10-30T12:15:57.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:15:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:15:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:15:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:15:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:15:57.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:15:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 10 monclient(hunting): handle_auth_done global_id 204187 payload 824 2023-10-30T12:15:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.639+0000 7f8d777fe640 1 -- 172.21.15.50:0/2318324671 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8d78099b80 con 0x7f8d78133bf0 2023-10-30T12:15:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d777fe640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d78133bf0 0x7f8d78136b40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d60014000 tx=0x7f8d60005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:15:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 10 client.115133.objecter ms_handle_connect 0x7f8d78133bf0 2023-10-30T12:15:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 10 client.115133.objecter resend_mon_ops 2023-10-30T12:15:57.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60020070 con 0x7f8d78133bf0 2023-10-30T12:15:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:15:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:15:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 10 monclient: dump: 2023-10-30T12:15:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:15:57.688 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:15:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:15:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:15:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:15:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:15:57.689 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:15:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:15:57.690 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:15:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4623 (secure 0 0 0) 0x7f8d60010400 con 0x7f8d78133bf0 2023-10-30T12:15:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:15:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:15:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d78133bf0 msgr2=0x7f8d78136b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d78133bf0 0x7f8d78136b40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d60014000 tx=0x7f8d60005a20 comp rx=0 tx=0).stop 2023-10-30T12:15:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d78133bf0 msgr2=0x7f8d78136b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:15:57.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.640+0000 7f8d74ff9640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d78133bf0 0x7f8d78136b40 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d60014000 tx=0x7f8d60005a20 comp rx=0 tx=0).stop 2023-10-30T12:15:57.691 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:15:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:15:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 158, 2023-10-30T12:15:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:15:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:15:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:15:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:15:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:15:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:15:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:15:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:15:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:15:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:15:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:15:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:15:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:15:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:15:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:15:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:15:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:15:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:15:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:15:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:15:57.702 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:15:57.702 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:15:57.702 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:15:57.702 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:15:57.702 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:15:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:15:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:15:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:15:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:15:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:15:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:15:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:15:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:15:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:15:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:15:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:15:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:15:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:15:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:15:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:15:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:15:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:15:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:15:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:15:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:15:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:15:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:15:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:15:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:15:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:15:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:15:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:15:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:15:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:15:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:15:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:15:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:15:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:15:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:15:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:15:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:15:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:15:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:15:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:15:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:15:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:15:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:15:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:15:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:15:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:15:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:15:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:15:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:15:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:15:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:15:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:15:57.725 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:15:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 10 librados: watch_flush enter 2023-10-30T12:15:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 10 librados: watch_flush exit 2023-10-30T12:15:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 20 client.115133.objecter shutdown clearing up homeless session... 2023-10-30T12:15:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 10 client.115133.objecter successfully canceled tick 2023-10-30T12:15:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8d580392a0 msgr2=0x7f8d5803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8d580392a0 0x7f8d5803b760 secure :-1 s=READY pgs=4217 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000f60 tx=0x7f8d68000ed0 comp rx=0 tx=0).stop 2023-10-30T12:15:57.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 10 monclient: shutdown 2023-10-30T12:15:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:15:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8d7813c120 msgr2=0x7f8d781cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:15:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 --2- 172.21.15.50:0/2318324671 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8d7813c120 0x7f8d781cad20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009f70 tx=0x7f8d70004a40 comp rx=0 tx=0).stop 2023-10-30T12:15:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7caf7640 1 -- 172.21.15.50:0/2318324671 reap_dead start 2023-10-30T12:15:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7caf7640 1 -- 172.21.15.50:0/2318324671 reap_dead start 2023-10-30T12:15:57.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 shutdown_connections 2023-10-30T12:15:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 >> 172.21.15.50:0/2318324671 conn(0x7f8d780a80b0 msgr2=0x7f8d780a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:15:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 shutdown_connections 2023-10-30T12:15:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 -- 172.21.15.50:0/2318324671 wait complete. 2023-10-30T12:15:57.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:15:57.641+0000 7f8d7e55a640 1 librados: shutdown 2023-10-30T12:15:57.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:15:57.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:16:17.732 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:16:17.732 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:16:17.732 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:16:17.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:16:17.736 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:16:17.736 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:16:17.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:17.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient: get_monmap_and_config 2023-10-30T12:16:17.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient: build_initial_monmap 2023-10-30T12:16:17.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient: monmap: 2023-10-30T12:16:17.857 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:17.856714+0000 2023-10-30T12:16:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:17.856714+0000 2023-10-30T12:16:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:17.858 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:17.859 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:17.859 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:17.859 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:17.859 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:17.859 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:17.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 1 Processor -- start 2023-10-30T12:16:17.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 1 -- start start 2023-10-30T12:16:17.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient: init 2023-10-30T12:16:17.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:17.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-10-30T12:16:17.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1144135b90 0x7f1144135600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:17.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient(hunting): picked mon.noname-b con 0x7f1144135b90 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:17.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1144135f70 0x7f11440a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:17.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient(hunting): picked mon.noname-c con 0x7f1144135f70 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:17.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11440a2990 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:17.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1144099f90 con 0x7f11440a2990 2023-10-30T12:16:17.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114409a200 con 0x7f1144135f70 2023-10-30T12:16:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11440a14c0 con 0x7f1144135b90 2023-10-30T12:16:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.856+0000 7f1148fe9640 10 monclient(hunting): authenticate will time out at 1823.040001s 2023-10-30T12:16:17.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1144135b90 0x7f1144135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1144135b90 0x7f1144135600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60888/0 (socket says 172.21.15.50:60888) 2023-10-30T12:16:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 -- 172.21.15.50:0/4209043740 learned_addr learned my addr 172.21.15.50:0/4209043740 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): get_auth_request con 0x7f1144135b90 auth_method 0 2023-10-30T12:16:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:17.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:17.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:17.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:17.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:17.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f113bfff640 1 --2- 172.21.15.50:0/4209043740 >> v2:172.21.15.118:3300/0 conn(0x7f1144135f70 0x7f11440a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:17.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f113bfff640 10 monclient(hunting): get_auth_request con 0x7f1144135f70 auth_method 0 2023-10-30T12:16:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f113bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f113bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f113bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient(hunting): handle_auth_done global_id 115142 payload 274 2023-10-30T12:16:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 -- 172.21.15.50:0/4209043740 >> v2:172.21.15.118:3300/0 conn(0x7f1144135f70 msgr2=0x7f11440a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 --2- 172.21.15.50:0/4209043740 >> v2:172.21.15.118:3300/0 conn(0x7f1144135f70 0x7f11440a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:17.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 -- 172.21.15.50:0/4209043740 >> v1:172.21.15.118:6789/0 conn(0x7f11440a2990 legacy=0x7f114413c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:17.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient: _finish_hunting 0 2023-10-30T12:16:17.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 monclient: found mon.noname-b 2023-10-30T12:16:17.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:17.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:16:17.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.857+0000 7f114259c640 1 -- 172.21.15.50:0/4209043740 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114409a4e0 con 0x7f1144135b90 2023-10-30T12:16:17.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f114259c640 1 --2- 172.21.15.50:0/4209043740 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1144135b90 0x7f1144135600 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1134009870 tx=0x7f113402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1de5858117e4b42 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:17.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 1 -- 172.21.15.50:0/4209043740 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11340317c0 con 0x7f1144135b90 2023-10-30T12:16:17.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:17.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:16:17.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: dump: 2023-10-30T12:16:17.876 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:17.876 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:17.876 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.876 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.876 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:17.877 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:17.877 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:17.877 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:17.877 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:17.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:17.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: _finish_auth 0 2023-10-30T12:16:17.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:16:17.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:17.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 5 monclient: authenticate success, global_id 115142 2023-10-30T12:16:17.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 1 -- 172.21.15.50:0/4209043740 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1134031920 con 0x7f1144135b90 2023-10-30T12:16:17.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:17.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 4 set_mon_vals no callback set 2023-10-30T12:16:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 1 -- 172.21.15.50:0/4209043740 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134031cb0 con 0x7f1144135b90 2023-10-30T12:16:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1141d9b640 10 monclient: dump: 2023-10-30T12:16:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:17.881 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:17.881 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:17.881 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:17.881 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:17.881 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 10 monclient: shutdown 2023-10-30T12:16:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 1 -- 172.21.15.50:0/4209043740 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1144135b90 msgr2=0x7f1144135600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4209043740 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1144135b90 0x7f1144135600 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1134009870 tx=0x7f113402b860 comp rx=0 tx=0).stop 2023-10-30T12:16:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 1 -- 172.21.15.50:0/4209043740 shutdown_connections 2023-10-30T12:16:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4209043740 >> v2:172.21.15.118:3300/0 conn(0x7f1144135f70 0x7f11440a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4209043740 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1144135b90 0x7f1144135600 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 1 -- 172.21.15.50:0/4209043740 >> 172.21.15.50:0/4209043740 conn(0x7f11440a82b0 msgr2=0x7f11440a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.858+0000 7f1148fe9640 1 -- 172.21.15.50:0/4209043740 shutdown_connections 2023-10-30T12:16:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.859+0000 7f1148fe9640 1 -- 172.21.15.50:0/4209043740 wait complete. 2023-10-30T12:16:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.859+0000 7f1148fe9640 10 monclient: build_initial_monmap 2023-10-30T12:16:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.859+0000 7f1148fe9640 10 monclient: monmap: 2023-10-30T12:16:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:17.886 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.859+0000 7f1148fe9640 1 librados: starting msgr at 2023-10-30T12:16:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.859+0000 7f1148fe9640 1 librados: starting objecter 2023-10-30T12:16:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 Processor -- start 2023-10-30T12:16:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 -- start start 2023-10-30T12:16:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 librados: setting wanted keys 2023-10-30T12:16:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 librados: calling monclient init 2023-10-30T12:16:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient: init 2023-10-30T12:16:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:16:17.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11440a2990 0x7f11441ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): picked mon.noname-c con 0x7f11440a2990 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 0x7f11441ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): picked mon.noname-a con 0x7f1144135b90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1144135f70 0x7f11441cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:17.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): picked mon.noname-b con 0x7f1144135f70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1144122a10 con 0x7f1144135b90 2023-10-30T12:16:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1144121d50 con 0x7f1144135f70 2023-10-30T12:16:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1144120330 con 0x7f11440a2990 2023-10-30T12:16:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1148fe9640 10 monclient(hunting): authenticate will time out at 1823.043615s 2023-10-30T12:16:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11440a2990 0x7f11441ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 0x7f11441ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 0x7f11441ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47182/0 (socket says 172.21.15.50:47182) 2023-10-30T12:16:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 1 -- 172.21.15.50:0/4212520743 learned_addr learned my addr 172.21.15.50:0/4212520743 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11440a2990 0x7f11441ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60892/0 (socket says 172.21.15.50:60892) 2023-10-30T12:16:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 10 monclient(hunting): get_auth_request con 0x7f1144135b90 auth_method 0 2023-10-30T12:16:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:17.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 10 monclient(hunting): get_auth_request con 0x7f11440a2990 auth_method 0 2023-10-30T12:16:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:17.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 1 -- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 msgr2=0x7f11441ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:16:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 1 -- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 msgr2=0x7f11441ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:16:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f114259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 0x7f11441ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1142d9d640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1144135f70 0x7f11441cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f113bfff640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 0x7f11441ca8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1142d9d640 10 monclient(hunting): get_auth_request con 0x7f1144135f70 auth_method 0 2023-10-30T12:16:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1142d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:17.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1142d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.860+0000 7f1142d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1142d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1142d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1142d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 10 monclient(hunting): handle_auth_done global_id 115145 payload 824 2023-10-30T12:16:17.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 1 -- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1144135f70 msgr2=0x7f11441cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1144135f70 0x7f11441cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 1 -- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 msgr2=0x7f11441ca8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 0x7f11441ca8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 10 monclient: _finish_hunting 0 2023-10-30T12:16:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 1 monclient: found mon.noname-c 2023-10-30T12:16:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 1 -- 172.21.15.50:0/4212520743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11340094d0 con 0x7f11440a2990 2023-10-30T12:16:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114259c640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11440a2990 0x7f11441ca3a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1134033000 tx=0x7f11340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 client.?.objecter ms_handle_connect 0x7f11440a2990 2023-10-30T12:16:17.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 1 -- 172.21.15.50:0/4212520743 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134032860 con 0x7f11440a2990 2023-10-30T12:16:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: dump: 2023-10-30T12:16:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:17.898 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:17.899 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:17.900 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: _finish_auth 0 2023-10-30T12:16:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: _check_auth_tickets 2023-10-30T12:16:17.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 1 -- 172.21.15.50:0/4212520743 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11340329e0 con 0x7f11440a2990 2023-10-30T12:16:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 5 monclient: authenticate success, global_id 115145 2023-10-30T12:16:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 1 -- 172.21.15.50:0/4212520743 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134032d90 con 0x7f11440a2990 2023-10-30T12:16:17.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f114159a640 4 set_mon_vals no callback set 2023-10-30T12:16:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f113affd640 10 monclient: dump: 2023-10-30T12:16:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:17.902 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:17.903 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:17.904 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 10 monclient: _renew_subs 2023-10-30T12:16:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1144120ed0 con 0x7f11440a2990 2023-10-30T12:16:17.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 10 client.115145.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 10 monclient: _renew_subs 2023-10-30T12:16:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11440a1710 con 0x7f11440a2990 2023-10-30T12:16:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.861+0000 7f1148fe9640 1 librados: init done 2023-10-30T12:16:17.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113affd640 1 -- 172.21.15.50:0/4212520743 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f113404e450 con 0x7f11440a2990 2023-10-30T12:16:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f1148fe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f1148fe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f1148fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11441221c0 con 0x7f11440a2990 2023-10-30T12:16:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113affd640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f11240392f0 0x7f112403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:17.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113affd640 1 -- 172.21.15.50:0/4212520743 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(489..489 src has 417..489) v4 ==== 7367+0+0 (secure 0 0 0) 0x7f113406d410 con 0x7f11440a2990 2023-10-30T12:16:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113affd640 10 client.115145.objecter ms_dispatch 0x7f1144000b90 osd_map(489..489 src has 417..489) v4 2023-10-30T12:16:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113affd640 3 client.115145.objecter handle_osd_map got epochs [489,489] > 0 2023-10-30T12:16:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113affd640 3 client.115145.objecter handle_osd_map decoding full epoch 489 2023-10-30T12:16:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113affd640 20 client.115145.objecter dump_active .. 0 homeless 2023-10-30T12:16:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113bfff640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f11240392f0 0x7f112403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.862+0000 7f113bfff640 10 monclient: get_auth_request con 0x7f11240392f0 auth_method 0 2023-10-30T12:16:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.863+0000 7f113bfff640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f11240392f0 0x7f112403b7b0 secure :-1 s=READY pgs=4220 cs=0 l=1 rev1=1 crypto rx=0x7f114411fe20 tx=0x7f112c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:17.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.863+0000 7f113affd640 10 client.115145.objecter ms_handle_connect 0x7f11240392f0 2023-10-30T12:16:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.868+0000 7f113affd640 1 -- 172.21.15.50:0/4212520743 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f113404e700 con 0x7f11440a2990 2023-10-30T12:16:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.868+0000 7f113affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:17.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:17.868+0000 7f113affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:18.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.109+0000 7f1148fe9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:18.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.109+0000 7f1148fe9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:18.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.109+0000 7f1148fe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:18.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.109+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f114404e160 con 0x7f11440a2990 2023-10-30T12:16:18.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.110+0000 7f113affd640 1 -- 172.21.15.50:0/4212520743 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f113404d3b0 con 0x7f11440a2990 2023-10-30T12:16:18.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.110+0000 7f113affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:18.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.110+0000 7f113affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:18.113 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":158,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:18.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 10 librados: watch_flush enter 2023-10-30T12:16:18.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 10 librados: watch_flush exit 2023-10-30T12:16:18.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 20 client.115145.objecter shutdown clearing up homeless session... 2023-10-30T12:16:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 10 client.115145.objecter successfully canceled tick 2023-10-30T12:16:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f11240392f0 msgr2=0x7f112403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f11240392f0 0x7f112403b7b0 secure :-1 s=READY pgs=4220 cs=0 l=1 rev1=1 crypto rx=0x7f114411fe20 tx=0x7f112c008040 comp rx=0 tx=0).stop 2023-10-30T12:16:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 10 monclient: shutdown 2023-10-30T12:16:18.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11440a2990 msgr2=0x7f11441ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11440a2990 0x7f11441ca3a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1134033000 tx=0x7f11340046c0 comp rx=0 tx=0).stop 2023-10-30T12:16:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 shutdown_connections 2023-10-30T12:16:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f11240392f0 0x7f112403b7b0 unknown :-1 s=CLOSED pgs=4220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1144135f70 0x7f11441cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1144135b90 0x7f11441ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.111+0000 7f1148fe9640 1 --2- 172.21.15.50:0/4212520743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f11440a2990 0x7f11441ca3a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.112+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 >> 172.21.15.50:0/4212520743 conn(0x7f11440a82b0 msgr2=0x7f1144135300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.112+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 shutdown_connections 2023-10-30T12:16:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.112+0000 7f1148fe9640 1 -- 172.21.15.50:0/4212520743 wait complete. 2023-10-30T12:16:18.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:18.112+0000 7f1148fe9640 1 librados: shutdown 2023-10-30T12:16:21.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.239+0000 7f1372725640 10 monclient: get_monmap_and_config 2023-10-30T12:16:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.239+0000 7f1372725640 10 monclient: build_initial_monmap 2023-10-30T12:16:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.239+0000 7f1372725640 10 monclient: monmap: 2023-10-30T12:16:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:21.240932+0000 2023-10-30T12:16:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:21.240932+0000 2023-10-30T12:16:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:21.244 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:21.244 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.240+0000 7f1372725640 1 Processor -- start 2023-10-30T12:16:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.240+0000 7f1372725640 1 -- start start 2023-10-30T12:16:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.240+0000 7f1372725640 10 monclient: init 2023-10-30T12:16:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-10-30T12:16:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 0x7f136c132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient(hunting): picked mon.noname-a con 0x7f136c131b20 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f136c071bc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f136c071fc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c09a9b0 con 0x7f136c071fc0 2023-10-30T12:16:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c131f00 con 0x7f136c071bc0 2023-10-30T12:16:21.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:21.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136c098230 con 0x7f136c131b20 2023-10-30T12:16:21.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:21.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 10 monclient(hunting): authenticate will time out at 1826.424134s 2023-10-30T12:16:21.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 0x7f136c132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 0x7f136c132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47196/0 (socket says 172.21.15.50:47196) 2023-10-30T12:16:21.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1370cc2640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f136c071fc0 0x7f136c097b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34948/0 (socket says 172.21.15.50:34948) 2023-10-30T12:16:21.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 1 -- 172.21.15.50:0/1100933478 learned_addr learned my addr 172.21.15.50:0/1100933478 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:21.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 10 monclient(hunting): get_auth_request con 0x7f136c131b20 auth_method 0 2023-10-30T12:16:21.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:21.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:21.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 1 -- 172.21.15.50:0/1100933478 >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 msgr2=0x7f136c132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:21.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 1 -- 172.21.15.50:0/1100933478 >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 msgr2=0x7f136c132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:21.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 1 --2- 172.21.15.50:0/1100933478 >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 0x7f136c132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136bfff640 1 --2- 172.21.15.50:0/1100933478 >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 0x7f136c132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551913576 0 0) 0x7f136c131f00 con 0x7f136c071bc0 2023-10-30T12:16:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient(hunting): _negotiate 2023-10-30T12:16:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient(hunting): my global_id is 204202 2023-10-30T12:16:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1354005730 con 0x7f136c071bc0 2023-10-30T12:16:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1802997587 0 0) 0x7f1354005730 con 0x7f136c071bc0 2023-10-30T12:16:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 msgr2=0x7f136c132270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 --2- 172.21.15.50:0/1100933478 >> v2:172.21.15.50:3300/0 conn(0x7f136c131b20 0x7f136c132270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 >> v1:172.21.15.50:6789/0 conn(0x7f136c071fc0 legacy=0x7f136c097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient: _finish_hunting 0 2023-10-30T12:16:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:16:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:16:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c09d920 con 0x7f136c071bc0 2023-10-30T12:16:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient: _finish_auth 0 2023-10-30T12:16:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient: _check_auth_tickets 2023-10-30T12:16:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f135c001670 con 0x7f136c071bc0 2023-10-30T12:16:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f1372725640 5 monclient: authenticate success, global_id 204202 2023-10-30T12:16:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.241+0000 7f136affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:16:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient: dump: 2023-10-30T12:16:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.264 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:21.264 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:21.264 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:21.264 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:21.264 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:21.264 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:16:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 >> v1:172.21.15.118:6789/0 conn(0x7f136c071bc0 legacy=0x7f136c097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:16:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1354005b80 0x7f1354005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): picked mon.b con 0x7f1354005b80 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:21.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13540069d0 0x7f1354006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): picked mon.c con 0x7f13540069d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:21.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354007590 0x7f135400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): picked mon.a con 0x7f1354007590 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c098230 con 0x7f1354007590 2023-10-30T12:16:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135400e430 con 0x7f1354005b80 2023-10-30T12:16:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135400e5f0 con 0x7f13540069d0 2023-10-30T12:16:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136affd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:16:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1372725640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13540069d0 0x7f1354006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354007590 0x7f135400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): get_auth_request con 0x7f13540069d0 auth_method 0 2023-10-30T12:16:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1354005b80 0x7f1354005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 10 monclient(hunting): get_auth_request con 0x7f1354007590 auth_method 0 2023-10-30T12:16:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 10 monclient(hunting): get_auth_request con 0x7f1354005b80 auth_method 0 2023-10-30T12:16:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient(hunting): handle_auth_done global_id 204202 payload 293 2023-10-30T12:16:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 1 -- 172.21.15.50:0/1100933478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1354005b80 msgr2=0x7f1354005fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1354005b80 0x7f1354005fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 1 -- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354007590 msgr2=0x7f135400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354007590 0x7f135400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:16:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 1 monclient: found mon.c 2023-10-30T12:16:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 1 -- 172.21.15.50:0/1100933478 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1354005730 con 0x7f13540069d0 2023-10-30T12:16:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:16:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:16:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f136b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1354007590 0x7f135400db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.242+0000 7f1370cc2640 1 -- 172.21.15.50:0/1100933478 reap_dead start 2023-10-30T12:16:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136b7fe640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13540069d0 0x7f1354006e50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c0e0 tx=0x7f1354007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8539acce061bb1bf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c02c9d0 con 0x7f13540069d0 2023-10-30T12:16:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 10 monclient: dump: 2023-10-30T12:16:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.277 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:21.277 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:21.277 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:21.277 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:21.277 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:21.277 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:21.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c02cb50 con 0x7f13540069d0 2023-10-30T12:16:21.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f1372725640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:21.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:21.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f1372725640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:21.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 1 -- 172.21.15.50:0/1100933478 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c034440 con 0x7f13540069d0 2023-10-30T12:16:21.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:21.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f1372725640 4 set_mon_vals no callback set 2023-10-30T12:16:21.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:21.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f136affd640 10 monclient: dump: 2023-10-30T12:16:21.279 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:21.279 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:21.280 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.280 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.280 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:21.280 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:21.280 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:21.280 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:21.281 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f1372725640 10 monclient: shutdown 2023-10-30T12:16:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f1372725640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f1372725640 1 -- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13540069d0 msgr2=0x7f1354006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.243+0000 7f1372725640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13540069d0 0x7f1354006e50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c0e0 tx=0x7f1354007a00 comp rx=0 tx=0).stop 2023-10-30T12:16:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 -- 172.21.15.50:0/1100933478 shutdown_connections 2023-10-30T12:16:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 --2- 172.21.15.50:0/1100933478 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f13540069d0 0x7f1354006e50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 -- 172.21.15.50:0/1100933478 >> 172.21.15.50:0/1100933478 conn(0x7f136c0a05c0 msgr2=0x7f136c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 -- 172.21.15.50:0/1100933478 shutdown_connections 2023-10-30T12:16:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 -- 172.21.15.50:0/1100933478 wait complete. 2023-10-30T12:16:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 10 monclient: build_initial_monmap 2023-10-30T12:16:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 10 monclient: monmap: 2023-10-30T12:16:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:21.284 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 librados: starting msgr at 2023-10-30T12:16:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 librados: starting objecter 2023-10-30T12:16:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.244+0000 7f1372725640 1 Processor -- start 2023-10-30T12:16:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 -- start start 2023-10-30T12:16:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 librados: setting wanted keys 2023-10-30T12:16:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 librados: calling monclient init 2023-10-30T12:16:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient: init 2023-10-30T12:16:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:16:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13540069d0 0x7f136c1c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): picked mon.noname-b con 0x7f13540069d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f136c130f60 0x7f136c1313e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): picked mon.noname-c con 0x7f136c130f60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 0x7f136c1ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): picked mon.noname-a con 0x7f136c1c9ba0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c12e740 con 0x7f136c1c9ba0 2023-10-30T12:16:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c099430 con 0x7f13540069d0 2023-10-30T12:16:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136c12e460 con 0x7f136c130f60 2023-10-30T12:16:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1372725640 10 monclient(hunting): authenticate will time out at 1826.428733s 2023-10-30T12:16:21.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f136c130f60 0x7f136c1313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f136c130f60 0x7f136c1313e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60924/0 (socket says 172.21.15.50:60924) 2023-10-30T12:16:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1370cc2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 0x7f136c1ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 1 -- 172.21.15.50:0/3177724423 learned_addr learned my addr 172.21.15.50:0/3177724423 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 10 monclient(hunting): get_auth_request con 0x7f136c130f60 auth_method 0 2023-10-30T12:16:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:21.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1370cc2640 10 monclient(hunting): get_auth_request con 0x7f136c1c9ba0 auth_method 0 2023-10-30T12:16:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1370cc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1370cc2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f1370cc2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136bfff640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13540069d0 0x7f136c1c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.245+0000 7f136bfff640 10 monclient(hunting): get_auth_request con 0x7f13540069d0 auth_method 0 2023-10-30T12:16:21.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1370cc2640 1 -- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 msgr2=0x7f136c1ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:16:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1370cc2640 1 -- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 msgr2=0x7f136c1ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:16:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1370cc2640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 0x7f136c1ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1370cc2640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 0x7f136c1ca050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 10 monclient(hunting): handle_auth_done global_id 115148 payload 824 2023-10-30T12:16:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 1 -- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13540069d0 msgr2=0x7f136c1c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13540069d0 0x7f136c1c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 1 -- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 msgr2=0x7f136c1ca050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 0x7f136c1ca050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:16:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 1 monclient: found mon.noname-c 2023-10-30T12:16:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 1 -- 172.21.15.50:0/3177724423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c1b90d0 con 0x7f136c130f60 2023-10-30T12:16:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136b7fe640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f136c130f60 0x7f136c1313e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c200 tx=0x7f136c09b140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:21.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 client.?.objecter ms_handle_connect 0x7f136c130f60 2023-10-30T12:16:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 1 -- 172.21.15.50:0/3177724423 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c034ea0 con 0x7f136c130f60 2023-10-30T12:16:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: dump: 2023-10-30T12:16:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:21.298 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: _finish_auth 0 2023-10-30T12:16:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:16:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 1 -- 172.21.15.50:0/3177724423 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c005070 con 0x7f136c130f60 2023-10-30T12:16:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 5 monclient: authenticate success, global_id 115148 2023-10-30T12:16:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 10 monclient: _renew_subs 2023-10-30T12:16:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c131920 con 0x7f136c130f60 2023-10-30T12:16:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 1 -- 172.21.15.50:0/3177724423 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c02c9f0 con 0x7f136c130f60 2023-10-30T12:16:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:21.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1368ff9640 10 monclient: dump: 2023-10-30T12:16:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:21.302 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 10 client.115148.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 10 monclient: _renew_subs 2023-10-30T12:16:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136c12e180 con 0x7f136c130f60 2023-10-30T12:16:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f1372725640 1 librados: init done 2023-10-30T12:16:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.246+0000 7f136a7fc640 4 set_mon_vals no callback set 2023-10-30T12:16:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1368ff9640 1 -- 172.21.15.50:0/3177724423 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f135c03b040 con 0x7f136c130f60 2023-10-30T12:16:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1372725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1372725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1372725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136c12e880 con 0x7f136c130f60 2023-10-30T12:16:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1368ff9640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f13580392f0 0x7f135803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1368ff9640 1 -- 172.21.15.50:0/3177724423 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(493..493 src has 417..493) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f135c06cbe0 con 0x7f136c130f60 2023-10-30T12:16:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1368ff9640 10 client.115148.objecter ms_dispatch 0x7f136c000b90 osd_map(493..493 src has 417..493) v4 2023-10-30T12:16:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1368ff9640 3 client.115148.objecter handle_osd_map got epochs [493,493] > 0 2023-10-30T12:16:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1368ff9640 3 client.115148.objecter handle_osd_map decoding full epoch 493 2023-10-30T12:16:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.247+0000 7f1368ff9640 20 client.115148.objecter dump_active .. 0 homeless 2023-10-30T12:16:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.248+0000 7f136bfff640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f13580392f0 0x7f135803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.248+0000 7f136bfff640 10 monclient: get_auth_request con 0x7f13580392f0 auth_method 0 2023-10-30T12:16:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.248+0000 7f136bfff640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f13580392f0 0x7f135803b7b0 secure :-1 s=READY pgs=4221 cs=0 l=1 rev1=1 crypto rx=0x7f136c0713c0 tx=0x7f1360008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.248+0000 7f1368ff9640 10 client.115148.objecter ms_handle_connect 0x7f13580392f0 2023-10-30T12:16:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.252+0000 7f1368ff9640 1 -- 172.21.15.50:0/3177724423 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f136c12e880 con 0x7f136c130f60 2023-10-30T12:16:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.252+0000 7f1368ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:21.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.252+0000 7f1368ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:21.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.496+0000 7f1372725640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:21.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.496+0000 7f1372725640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:21.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.496+0000 7f1372725640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:21.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.496+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f136c12e880 con 0x7f136c130f60 2023-10-30T12:16:21.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.497+0000 7f1368ff9640 1 -- 172.21.15.50:0/3177724423 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f136c12e880 con 0x7f136c130f60 2023-10-30T12:16:21.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.497+0000 7f1368ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:21.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.497+0000 7f1368ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:21.500 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":158,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:21.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 10 librados: watch_flush enter 2023-10-30T12:16:21.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 10 librados: watch_flush exit 2023-10-30T12:16:21.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 20 client.115148.objecter shutdown clearing up homeless session... 2023-10-30T12:16:21.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 10 client.115148.objecter successfully canceled tick 2023-10-30T12:16:21.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f13580392f0 msgr2=0x7f135803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:21.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f13580392f0 0x7f135803b7b0 secure :-1 s=READY pgs=4221 cs=0 l=1 rev1=1 crypto rx=0x7f136c0713c0 tx=0x7f1360008040 comp rx=0 tx=0).stop 2023-10-30T12:16:21.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 10 monclient: shutdown 2023-10-30T12:16:21.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:21.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f136c130f60 msgr2=0x7f136c1313e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:21.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f136c130f60 0x7f136c1313e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c200 tx=0x7f136c09b140 comp rx=0 tx=0).stop 2023-10-30T12:16:21.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 shutdown_connections 2023-10-30T12:16:21.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f13580392f0 0x7f135803b7b0 unknown :-1 s=CLOSED pgs=4221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f136c1c9ba0 0x7f136c1ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f136c130f60 0x7f136c1313e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 --2- 172.21.15.50:0/3177724423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13540069d0 0x7f136c1c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:21.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.498+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 >> 172.21.15.50:0/3177724423 conn(0x7f136c0a05c0 msgr2=0x7f136c09c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:21.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.499+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 shutdown_connections 2023-10-30T12:16:21.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.499+0000 7f1372725640 1 -- 172.21.15.50:0/3177724423 wait complete. 2023-10-30T12:16:21.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:21.499+0000 7f1372725640 1 librados: shutdown 2023-10-30T12:16:24.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:24.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.627+0000 7f98b470f640 10 monclient: get_monmap_and_config 2023-10-30T12:16:24.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.627+0000 7f98b470f640 10 monclient: build_initial_monmap 2023-10-30T12:16:24.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.627+0000 7f98b470f640 10 monclient: monmap: 2023-10-30T12:16:24.629 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:24.629 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:24.630 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:24.628573+0000 2023-10-30T12:16:24.630 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:24.628573+0000 2023-10-30T12:16:24.630 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:24.630 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:24.630 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:24.630 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:24.631 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:24.631 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:24.631 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:24.631 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:24.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 1 Processor -- start 2023-10-30T12:16:24.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 1 -- start start 2023-10-30T12:16:24.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient: init 2023-10-30T12:16:24.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:24.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-10-30T12:16:24.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:24.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient(hunting): picked mon.noname-b con 0x7f98ac0a0080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:24.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98ac0720b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:24.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:24.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient(hunting): picked mon.noname-c con 0x7f98ac073a30 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:24.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac09ccc0 con 0x7f98ac0720b0 2023-10-30T12:16:24.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:24.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac073620 con 0x7f98ac073a30 2023-10-30T12:16:24.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:24.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0a0ee0 con 0x7f98ac0a0080 2023-10-30T12:16:24.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:24.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b470f640 10 monclient(hunting): authenticate will time out at 1829.811786s 2023-10-30T12:16:24.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b24ab640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:24.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b1caa640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f98ac0720b0 0x7f98ac097320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34958/0 (socket says 172.21.15.50:34958) 2023-10-30T12:16:24.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 learned_addr learned my addr 172.21.15.50:0/1197621721 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:24.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.628+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98ac0a0080 auth_method 0 2023-10-30T12:16:24.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:24.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:24.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b24ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:24.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:24.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f98ac073a30 auth_method 0 2023-10-30T12:16:24.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:24.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:24.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:24.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:24.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:24.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:24.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:24.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:24.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:24.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:24.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.629+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98ac0a0080 auth_method 0 2023-10-30T12:16:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f98ac073a30 auth_method 0 2023-10-30T12:16:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:24.830+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:25.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98ac0a0080 auth_method 0 2023-10-30T12:16:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f98ac073a30 auth_method 0 2023-10-30T12:16:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:25.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:25.231+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:26.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:26.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98ac0a0080 auth_method 0 2023-10-30T12:16:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f98ac073a30 auth_method 0 2023-10-30T12:16:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:26.032+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient(hunting): tick 2023-10-30T12:16:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 msgr2=0x7f98ac0a0460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98ac0a0080 0x7f98ac0a0460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 msgr2=0x7f98ac0a09a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f98ac073a30 0x7f98ac0a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> v1:172.21.15.50:6789/0 conn(0x7f98ac0720b0 legacy=0x7f98ac097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient: _add_conns ranks=[3,1] 2023-10-30T12:16:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f98900036a0 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9890006210 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac071e90 con 0x7f9890006210 2023-10-30T12:16:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac073620 con 0x7f98900036a0 2023-10-30T12:16:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98a3fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:27.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:27.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:27.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:27.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:27.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:27.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:27.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:27.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:27.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:27.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:28.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:28.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:28.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:28.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:28.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:28.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:28.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:28.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:28.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:28.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:29.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:29.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:29.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:29.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.085+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783178597 0 0) 0x7f98ac071e90 con 0x7f9890006210 2023-10-30T12:16:29.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.085+0000 7f98b14a9640 10 monclient(hunting): _negotiate 2023-10-30T12:16:29.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.085+0000 7f98b14a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:29.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.085+0000 7f98b14a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:29.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.085+0000 7f98b14a9640 10 monclient(hunting): my global_id is 204208 2023-10-30T12:16:29.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:29.085+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9898003520 con 0x7f9890006210 2023-10-30T12:16:30.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): tick 2023-10-30T12:16:30.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:30.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:30.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:30.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:30.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:30.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> v1:172.21.15.118:6789/0 conn(0x7f9890006210 legacy=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:30.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T12:16:30.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:30.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 reap_dead start 2023-10-30T12:16:30.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f98900036a0 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f989000a060 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f989000a440 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac073620 con 0x7f98900036a0 2023-10-30T12:16:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac0a0ee0 con 0x7f989000a060 2023-10-30T12:16:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9890008ea0 con 0x7f989000a440 2023-10-30T12:16:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98a3fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:30.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:30.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:30.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:30.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:30.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 10 monclient(hunting): get_auth_request con 0x7f989000a060 auth_method 0 2023-10-30T12:16:30.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:30.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:30.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:30.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:30.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:30.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:30.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.629+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:30.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:30.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:30.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f989000a440 auth_method 0 2023-10-30T12:16:30.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:30.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:30.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:30.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:30.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:30.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:30.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:30.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:30.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 10 monclient(hunting): get_auth_request con 0x7f989000a060 auth_method 0 2023-10-30T12:16:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:30.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:30.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:30.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:30.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:30.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:30.830+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:31.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f989000a440 auth_method 0 2023-10-30T12:16:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:31.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 10 monclient(hunting): get_auth_request con 0x7f989000a060 auth_method 0 2023-10-30T12:16:31.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:31.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:31.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:31.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:31.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:31.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:31.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:31.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:31.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:31.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:31.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:31.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:31.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:31.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:31.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:31.231+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f98900036a0 auth_method 0 2023-10-30T12:16:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f989000a440 auth_method 0 2023-10-30T12:16:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:16:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b1caa640 10 monclient(hunting): get_auth_request con 0x7f989000a060 auth_method 0 2023-10-30T12:16:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b1caa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b1caa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b1caa640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.032+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.033+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:16:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.033+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.033+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:32.033+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:16:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 10 monclient(hunting): tick 2023-10-30T12:16:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:16:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:16:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:16:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 msgr2=0x7f989000a8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f989000a440 0x7f989000a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 msgr2=0x7f98900086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.118:3300/0 conn(0x7f989000a060 0x7f98900086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 msgr2=0x7f9890005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 --2- 172.21.15.50:0/1197621721 >> v2:172.21.15.50:3300/0 conn(0x7f98900036a0 0x7f9890005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 10 monclient: _add_conns ranks=[1,0] 2023-10-30T12:16:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9890009210 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9890010340 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989000b020 con 0x7f9890010340 2023-10-30T12:16:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 1 -- 172.21.15.50:0/1197621721 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9890014f30 con 0x7f9890009210 2023-10-30T12:16:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:33.629+0000 7f98a3fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:34.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.103+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321925299 0 0) 0x7f989000b020 con 0x7f9890010340 2023-10-30T12:16:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.103+0000 7f98b14a9640 10 monclient(hunting): _negotiate 2023-10-30T12:16:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.103+0000 7f98b14a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.103+0000 7f98b14a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.103+0000 7f98b14a9640 10 monclient(hunting): my global_id is 214098 2023-10-30T12:16:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.103+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ac071e90 con 0x7f9890010340 2023-10-30T12:16:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894121246 0 0) 0x7f98ac071e90 con 0x7f9890010340 2023-10-30T12:16:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 >> v1:172.21.15.118:6789/0 conn(0x7f9890009210 legacy=0x7f9890009620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:34.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: _finish_hunting 0 2023-10-30T12:16:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:16:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:16:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9890016110 con 0x7f9890010340 2023-10-30T12:16:34.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: _finish_auth 0 2023-10-30T12:16:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: _check_auth_tickets 2023-10-30T12:16:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f989c0015b0 con 0x7f9890010340 2023-10-30T12:16:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:16:34.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: dump: 2023-10-30T12:16:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:34.113 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:16:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 >> v1:172.21.15.50:6789/0 conn(0x7f9890010340 legacy=0x7f9890012730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:16:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 1 -- 172.21.15.50:0/1197621721 reap_dead start 2023-10-30T12:16:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898003b30 0x7f9898003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): picked mon.a con 0x7f9898003b30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98980047c0 0x7f9898004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): picked mon.c con 0x7f98980047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9898009970 0x7f989800bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): picked mon.b con 0x7f9898009970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989800c3a0 con 0x7f9898003b30 2023-10-30T12:16:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989800c510 con 0x7f9898009970 2023-10-30T12:16:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989800c6d0 con 0x7f98980047c0 2023-10-30T12:16:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b14a9640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:16:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898003b30 0x7f9898003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98980047c0 0x7f9898004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f9898003b30 auth_method 0 2023-10-30T12:16:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): get_auth_request con 0x7f98980047c0 auth_method 0 2023-10-30T12:16:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9898009970 0x7f989800bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f9898009970 auth_method 0 2023-10-30T12:16:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient(hunting): handle_auth_done global_id 214098 payload 293 2023-10-30T12:16:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9898009970 msgr2=0x7f989800bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9898009970 0x7f989800bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898003b30 msgr2=0x7f9898003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898003b30 0x7f9898003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient: _finish_hunting 0 2023-10-30T12:16:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 1 monclient: found mon.c 2023-10-30T12:16:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 1 -- 172.21.15.50:0/1197621721 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac071e90 con 0x7f98980047c0 2023-10-30T12:16:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient: _finish_auth 0 2023-10-30T12:16:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 10 monclient: _check_auth_tickets 2023-10-30T12:16:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b1caa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b470f640 5 monclient: authenticate success, global_id 214098 2023-10-30T12:16:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b470f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.104+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898003b30 0x7f9898003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.105+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98980047c0 0x7f9898004c40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f989c007d00 tx=0x7f98980057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f79ad8adcfbb3b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c036070 con 0x7f98980047c0 2023-10-30T12:16:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 10 monclient: dump: 2023-10-30T12:16:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:34.132 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c02e040 con 0x7f98980047c0 2023-10-30T12:16:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:34.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 1 -- 172.21.15.50:0/1197621721 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c005ad0 con 0x7f98980047c0 2023-10-30T12:16:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b470f640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b470f640 4 set_mon_vals no callback set 2023-10-30T12:16:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.114+0000 7f98b14a9640 10 monclient: dump: 2023-10-30T12:16:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:34.134 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:34.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 10 monclient: shutdown 2023-10-30T12:16:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 -- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98980047c0 msgr2=0x7f9898004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98980047c0 0x7f9898004c40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f989c007d00 tx=0x7f98980057f0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 -- 172.21.15.50:0/1197621721 shutdown_connections 2023-10-30T12:16:34.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9898009970 0x7f989800bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98980047c0 0x7f9898004c40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 --2- 172.21.15.50:0/1197621721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898003b30 0x7f9898003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 -- 172.21.15.50:0/1197621721 >> 172.21.15.50:0/1197621721 conn(0x7f98ac0a4870 msgr2=0x7f98ac0a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 -- 172.21.15.50:0/1197621721 shutdown_connections 2023-10-30T12:16:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.115+0000 7f98b470f640 1 -- 172.21.15.50:0/1197621721 wait complete. 2023-10-30T12:16:34.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.116+0000 7f98b470f640 10 monclient: build_initial_monmap 2023-10-30T12:16:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.116+0000 7f98b470f640 10 monclient: monmap: 2023-10-30T12:16:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:34.139 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:34.140 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 librados: starting msgr at 2023-10-30T12:16:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 librados: starting objecter 2023-10-30T12:16:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 Processor -- start 2023-10-30T12:16:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 -- start start 2023-10-30T12:16:34.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 librados: setting wanted keys 2023-10-30T12:16:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 librados: calling monclient init 2023-10-30T12:16:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient: init 2023-10-30T12:16:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:16:34.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f98ac1c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): picked mon.noname-c con 0x7f9898003b30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f98ac093f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): picked mon.noname-b con 0x7f98980047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): picked mon.noname-a con 0x7f9898009970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0901f0 con 0x7f9898009970 2023-10-30T12:16:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac090460 con 0x7f98980047c0 2023-10-30T12:16:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac090740 con 0x7f9898003b30 2023-10-30T12:16:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b470f640 10 monclient(hunting): authenticate will time out at 1839.300803s 2023-10-30T12:16:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b24ab640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f98ac1c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b24ab640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f98ac1c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42874/0 (socket says 172.21.15.50:42874) 2023-10-30T12:16:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b24ab640 1 -- 172.21.15.50:0/1831297926 learned_addr learned my addr 172.21.15.50:0/1831297926 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f9898003b30 auth_method 0 2023-10-30T12:16:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b2cac640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.117+0000 7f98b24ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f9898009970 auth_method 0 2023-10-30T12:16:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f98ac093f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b24ab640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 msgr2=0x7f98ac1c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:16:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b24ab640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 msgr2=0x7f98ac1c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:16:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f98ac1c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 10 monclient(hunting): get_auth_request con 0x7f98980047c0 auth_method 0 2023-10-30T12:16:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:34.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:34.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:34.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f98ac1c9d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:34.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:34.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 msgr2=0x7f98ac093f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:16:34.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 msgr2=0x7f98ac093f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:16:34.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f98ac093f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:34.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f98ac093f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:34.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient(hunting): handle_auth_done global_id 214101 payload 824 2023-10-30T12:16:34.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 msgr2=0x7f98ac1c9d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:34.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f98ac1c9d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 msgr2=0x7f98ac093f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:34.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f98ac093f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient: _finish_hunting 0 2023-10-30T12:16:34.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 monclient: found mon.noname-a 2023-10-30T12:16:34.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:34.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:34.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac08e640 con 0x7f9898009970 2023-10-30T12:16:34.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 msgr2=0x7f98ac0944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:16:34.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 msgr2=0x7f98ac0944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:16:34.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f98ac09c760 tx=0x7f98a4017780 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:34.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.118+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:34.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient: get_auth_request con 0x7f9898009970 auth_method 0 2023-10-30T12:16:34.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 0 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-30T12:16:34.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:34.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:34.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 msgr2=0x7f98ac0944b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:34.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9898009970 0x7f98ac0944b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:16:34.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f989000a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f9898003b30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:34.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98900095f0 0x7f9890005a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f98900095f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f9890002ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f98980047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9890014680 con 0x7f98900095f0 2023-10-30T12:16:34.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900147c0 con 0x7f98980047c0 2023-10-30T12:16:34.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:34.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9890014980 con 0x7f9898003b30 2023-10-30T12:16:34.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98a2ffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:34.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98900095f0 0x7f9890005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f989000a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 10 monclient(hunting): get_auth_request con 0x7f98900095f0 auth_method 0 2023-10-30T12:16:34.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:34.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 10 monclient(hunting): get_auth_request con 0x7f9898003b30 auth_method 0 2023-10-30T12:16:34.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:34.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f9890002ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:34.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:34.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b1caa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:34.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient(hunting): get_auth_request con 0x7f98980047c0 auth_method 0 2023-10-30T12:16:34.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:34.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:34.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:34.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:34.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:34.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b24ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:34.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:34.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:34.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.319+0000 7f98b2cac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:34.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 10 monclient(hunting): handle_auth_done global_id 214101 payload 293 2023-10-30T12:16:34.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 msgr2=0x7f989000a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f989000a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 msgr2=0x7f9890002ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f9890002ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 10 monclient: _finish_hunting 0 2023-10-30T12:16:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 1 monclient: found mon.noname-a 2023-10-30T12:16:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9890014c00 con 0x7f98900095f0 2023-10-30T12:16:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b1caa640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98900095f0 0x7f9890005a90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f989c033000 tx=0x7f98a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 client.?.objecter ms_handle_connect 0x7f98900095f0 2023-10-30T12:16:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c036040 con 0x7f98900095f0 2023-10-30T12:16:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: dump: 2023-10-30T12:16:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:34.337 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:34.337 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.337 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.337 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:34.337 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:34.338 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:34.338 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:34.338 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:34.338 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:34.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:16:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:16:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c0058d0 con 0x7f98900095f0 2023-10-30T12:16:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 5 monclient: authenticate success, global_id 214101 2023-10-30T12:16:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 10 monclient: _renew_subs 2023-10-30T12:16:34.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:34.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac08f320 con 0x7f98900095f0 2023-10-30T12:16:34.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:34.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c005c60 con 0x7f98900095f0 2023-10-30T12:16:34.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:34.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:34.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a2ffd640 10 monclient: dump: 2023-10-30T12:16:34.341 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:34.341 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:34.341 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.341 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:34.341 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:34.342 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:34.342 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:34.342 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:34.342 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:34.342 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:34.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 10 client.214101.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:34.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98a3fff640 4 set_mon_vals no callback set 2023-10-30T12:16:34.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 10 monclient: _renew_subs 2023-10-30T12:16:34.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:34.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac1c9f50 con 0x7f98900095f0 2023-10-30T12:16:34.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.320+0000 7f98b470f640 1 librados: init done 2023-10-30T12:16:34.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f989c0047e0 con 0x7f98900095f0 2023-10-30T12:16:34.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98b470f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:34.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98b470f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:34.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98a2ffd640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9898009970 0x7f9890045390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:34.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98b470f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:34.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac1c9f50 con 0x7f98900095f0 2023-10-30T12:16:34.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(496..496 src has 417..496) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f989c06be60 con 0x7f98900095f0 2023-10-30T12:16:34.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98a2ffd640 10 client.214101.objecter ms_dispatch 0x7f98ac000b90 osd_map(496..496 src has 417..496) v4 2023-10-30T12:16:34.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98a2ffd640 3 client.214101.objecter handle_osd_map got epochs [496,496] > 0 2023-10-30T12:16:34.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98a2ffd640 3 client.214101.objecter handle_osd_map decoding full epoch 496 2023-10-30T12:16:34.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.321+0000 7f98a2ffd640 20 client.214101.objecter dump_active .. 0 homeless 2023-10-30T12:16:34.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.322+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9898009970 0x7f9890045390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:34.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.322+0000 7f98b24ab640 10 monclient: get_auth_request con 0x7f9898009970 auth_method 0 2023-10-30T12:16:34.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.322+0000 7f98b24ab640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9898009970 0x7f9890045390 secure :-1 s=READY pgs=4222 cs=0 l=1 rev1=1 crypto rx=0x7f989000af90 tx=0x7f98a801a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:34.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.322+0000 7f98a2ffd640 10 client.214101.objecter ms_handle_connect 0x7f9898009970 2023-10-30T12:16:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.327+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f989c040020 con 0x7f98900095f0 2023-10-30T12:16:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.327+0000 7f98a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.327+0000 7f98a2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:34.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.575+0000 7f98b470f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:34.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.575+0000 7f98b470f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:34.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.575+0000 7f98b470f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:34.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.575+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f98ac1c9f50 con 0x7f98900095f0 2023-10-30T12:16:34.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.575+0000 7f98a2ffd640 1 -- 172.21.15.50:0/1831297926 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f989c0070d0 con 0x7f98900095f0 2023-10-30T12:16:34.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.575+0000 7f98a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:34.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.575+0000 7f98a2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:34.578 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":166,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 10 librados: watch_flush enter 2023-10-30T12:16:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 10 librados: watch_flush exit 2023-10-30T12:16:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 20 client.214101.objecter shutdown clearing up homeless session... 2023-10-30T12:16:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 10 client.214101.objecter successfully canceled tick 2023-10-30T12:16:34.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9898009970 msgr2=0x7f9890045390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9898009970 0x7f9890045390 secure :-1 s=READY pgs=4222 cs=0 l=1 rev1=1 crypto rx=0x7f989000af90 tx=0x7f98a801a040 comp rx=0 tx=0).stop 2023-10-30T12:16:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 10 monclient: shutdown 2023-10-30T12:16:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98900095f0 msgr2=0x7f9890005a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:34.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98900095f0 0x7f9890005a90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f989c033000 tx=0x7f98a400b040 comp rx=0 tx=0).stop 2023-10-30T12:16:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 shutdown_connections 2023-10-30T12:16:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9898009970 0x7f9890045390 unknown :-1 s=CLOSED pgs=4222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98980047c0 0x7f9890002ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98900095f0 0x7f9890005a90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 --2- 172.21.15.50:0/1831297926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9898003b30 0x7f989000a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 >> 172.21.15.50:0/1831297926 conn(0x7f98ac0a4870 msgr2=0x7f98ac1bd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 shutdown_connections 2023-10-30T12:16:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 -- 172.21.15.50:0/1831297926 wait complete. 2023-10-30T12:16:34.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:34.577+0000 7f98b470f640 1 librados: shutdown 2023-10-30T12:16:37.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:37.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.704+0000 7f9e17fff640 10 monclient: get_monmap_and_config 2023-10-30T12:16:37.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.704+0000 7f9e17fff640 10 monclient: build_initial_monmap 2023-10-30T12:16:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.704+0000 7f9e17fff640 10 monclient: monmap: 2023-10-30T12:16:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:37.705882+0000 2023-10-30T12:16:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:37.705882+0000 2023-10-30T12:16:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:37.709 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.705+0000 7f9e17fff640 1 Processor -- start 2023-10-30T12:16:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.705+0000 7f9e17fff640 1 -- start start 2023-10-30T12:16:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.705+0000 7f9e17fff640 10 monclient: init 2023-10-30T12:16:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.705+0000 7f9e17fff640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.705+0000 7f9e17fff640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:16:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.705+0000 7f9e17fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e181312b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.705+0000 7f9e17fff640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9e18131690 0x7f9e1813a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 10 monclient(hunting): picked mon.noname-c con 0x7f9e18131690 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9e18131ae0 0x7f9e1813cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 10 monclient(hunting): picked mon.noname-a con 0x7f9e18131ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1806bda0 con 0x7f9e181312b0 2023-10-30T12:16:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e180a3c20 con 0x7f9e18131ae0 2023-10-30T12:16:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1806a330 con 0x7f9e18131690 2023-10-30T12:16:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e17fff640 10 monclient(hunting): authenticate will time out at 1842.889083s 2023-10-30T12:16:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e16ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9e181312b0 0x7f9e1806b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50196/0 (socket says 172.21.15.50:50196) 2023-10-30T12:16:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e16ffd640 1 -- 172.21.15.50:0/1435025178 learned_addr learned my addr 172.21.15.50:0/1435025178 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 1 --2- 172.21.15.50:0/1435025178 >> v2:172.21.15.50:3300/0 conn(0x7f9e18131ae0 0x7f9e1813cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): get_auth_request con 0x7f9e18131ae0 auth_method 0 2023-10-30T12:16:37.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:37.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:37.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:37.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:37.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:37.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:37.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e167fc640 1 --2- 172.21.15.50:0/1435025178 >> v2:172.21.15.118:3300/0 conn(0x7f9e18131690 0x7f9e1813a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:37.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e167fc640 10 monclient(hunting): get_auth_request con 0x7f9e18131690 auth_method 0 2023-10-30T12:16:37.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:37.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e167fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:37.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e167fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:37.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient(hunting): handle_auth_done global_id 214104 payload 274 2023-10-30T12:16:37.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 1 -- 172.21.15.50:0/1435025178 >> v2:172.21.15.118:3300/0 conn(0x7f9e18131690 msgr2=0x7f9e1813a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:37.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 1 --2- 172.21.15.50:0/1435025178 >> v2:172.21.15.118:3300/0 conn(0x7f9e18131690 0x7f9e1813a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 1 -- 172.21.15.50:0/1435025178 >> v1:172.21.15.50:6789/0 conn(0x7f9e181312b0 legacy=0x7f9e1806b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:37.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient: _finish_hunting 0 2023-10-30T12:16:37.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 1 monclient: found mon.noname-a 2023-10-30T12:16:37.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:37.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:37.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.706+0000 7f9e177fe640 1 -- 172.21.15.50:0/1435025178 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1806a610 con 0x7f9e18131ae0 2023-10-30T12:16:37.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e177fe640 1 --2- 172.21.15.50:0/1435025178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e18131ae0 0x7f9e1813cdc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e180746d0 tx=0x7f9e1002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869265a8d058f1c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:37.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 1 -- 172.21.15.50:0/1435025178 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1002cbf0 con 0x7f9e18131ae0 2023-10-30T12:16:37.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:37.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:37.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: dump: 2023-10-30T12:16:37.726 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:37.726 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:37.726 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.726 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.727 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:37.727 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:37.727 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:37.727 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:37.727 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:37.728 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:37.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: _finish_auth 0 2023-10-30T12:16:37.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:16:37.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:37.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 5 monclient: authenticate success, global_id 214104 2023-10-30T12:16:37.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:37.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 1 -- 172.21.15.50:0/1435025178 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1002cd50 con 0x7f9e18131ae0 2023-10-30T12:16:37.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:37.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:37.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 4 set_mon_vals no callback set 2023-10-30T12:16:37.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 1 -- 172.21.15.50:0/1435025178 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10034660 con 0x7f9e18131ae0 2023-10-30T12:16:37.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:37.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:37.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e15ffb640 10 monclient: dump: 2023-10-30T12:16:37.730 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:37.730 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:37.731 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.731 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.731 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:37.731 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:37.731 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:37.731 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:37.732 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:37.732 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:37.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 10 monclient: shutdown 2023-10-30T12:16:37.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:37.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 1 -- 172.21.15.50:0/1435025178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e18131ae0 msgr2=0x7f9e1813cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:37.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.707+0000 7f9e17fff640 1 --2- 172.21.15.50:0/1435025178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e18131ae0 0x7f9e1813cdc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e180746d0 tx=0x7f9e1002be40 comp rx=0 tx=0).stop 2023-10-30T12:16:37.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 -- 172.21.15.50:0/1435025178 shutdown_connections 2023-10-30T12:16:37.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 --2- 172.21.15.50:0/1435025178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e18131ae0 0x7f9e1813cdc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 --2- 172.21.15.50:0/1435025178 >> v2:172.21.15.118:3300/0 conn(0x7f9e18131690 0x7f9e1813a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 -- 172.21.15.50:0/1435025178 >> 172.21.15.50:0/1435025178 conn(0x7f9e180a4200 msgr2=0x7f9e180a4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:37.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 -- 172.21.15.50:0/1435025178 shutdown_connections 2023-10-30T12:16:37.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 -- 172.21.15.50:0/1435025178 wait complete. 2023-10-30T12:16:37.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 10 monclient: build_initial_monmap 2023-10-30T12:16:37.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 10 monclient: monmap: 2023-10-30T12:16:37.735 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:37.735 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:37.735 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:37.735 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:37.735 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:37.735 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:37.736 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 librados: starting msgr at 2023-10-30T12:16:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.708+0000 7f9e17fff640 1 librados: starting objecter 2023-10-30T12:16:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 Processor -- start 2023-10-30T12:16:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 -- start start 2023-10-30T12:16:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 librados: setting wanted keys 2023-10-30T12:16:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 librados: calling monclient init 2023-10-30T12:16:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient: init 2023-10-30T12:16:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:16:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e181312b0 0x7f9e1812f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): picked mon.noname-a con 0x7f9e181312b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 0x7f9e1812f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): picked mon.noname-c con 0x7f9e18131690 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 0x7f9e181c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): picked mon.noname-b con 0x7f9e18131ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e180a38f0 con 0x7f9e181312b0 2023-10-30T12:16:37.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:37.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e18133b90 con 0x7f9e18131ae0 2023-10-30T12:16:37.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:37.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e181bd4a0 con 0x7f9e18131690 2023-10-30T12:16:37.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:37.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e17fff640 10 monclient(hunting): authenticate will time out at 1842.892736s 2023-10-30T12:16:37.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e181312b0 0x7f9e1812f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:37.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 0x7f9e1812f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:37.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e181312b0 0x7f9e1812f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43092/0 (socket says 172.21.15.50:43092) 2023-10-30T12:16:37.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 1 -- 172.21.15.50:0/518026897 learned_addr learned my addr 172.21.15.50:0/518026897 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:37.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 10 monclient(hunting): get_auth_request con 0x7f9e181312b0 auth_method 0 2023-10-30T12:16:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 10 monclient(hunting): get_auth_request con 0x7f9e18131690 auth_method 0 2023-10-30T12:16:37.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:37.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e177fe640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 0x7f9e181c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:37.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:37.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 msgr2=0x7f9e1812f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:16:37.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 msgr2=0x7f9e1812f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:16:37.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e177fe640 10 monclient(hunting): get_auth_request con 0x7f9e18131ae0 auth_method 0 2023-10-30T12:16:37.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:37.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e167fc640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 0x7f9e1812f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:37.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:37.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.709+0000 7f9e177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:37.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e167fc640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 0x7f9e1812f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:37.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e177fe640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 msgr2=0x7f9e181c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:16:37.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e177fe640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 msgr2=0x7f9e181c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:16:37.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e177fe640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 0x7f9e181c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:37.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e177fe640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 0x7f9e181c9bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:37.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 10 monclient(hunting): handle_auth_done global_id 214110 payload 824 2023-10-30T12:16:37.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 msgr2=0x7f9e1812f860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:37.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 0x7f9e1812f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 msgr2=0x7f9e181c9bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:37.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 0x7f9e181c9bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:16:37.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 1 monclient: found mon.noname-a 2023-10-30T12:16:37.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:37.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:37.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 1 -- 172.21.15.50:0/518026897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e10009a70 con 0x7f9e181312b0 2023-10-30T12:16:37.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e16ffd640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e181312b0 0x7f9e1812f320 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c400 tx=0x7f9e0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:37.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 client.?.objecter ms_handle_connect 0x7f9e181312b0 2023-10-30T12:16:37.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:37.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 1 -- 172.21.15.50:0/518026897 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080074e0 con 0x7f9e181312b0 2023-10-30T12:16:37.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:37.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:37.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: dump: 2023-10-30T12:16:37.749 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:37.749 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:37.751 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: _finish_auth 0 2023-10-30T12:16:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:16:37.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:37.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 5 monclient: authenticate success, global_id 214110 2023-10-30T12:16:37.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 1 -- 172.21.15.50:0/518026897 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e08007640 con 0x7f9e181312b0 2023-10-30T12:16:37.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 10 monclient: _renew_subs 2023-10-30T12:16:37.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:37.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e181347d0 con 0x7f9e181312b0 2023-10-30T12:16:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 1 -- 172.21.15.50:0/518026897 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080059a0 con 0x7f9e181312b0 2023-10-30T12:16:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:37.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9dff7fe640 10 monclient: dump: 2023-10-30T12:16:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:37.755 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:37.755 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 10 client.214110.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 10 monclient: _renew_subs 2023-10-30T12:16:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e157fa640 4 set_mon_vals no callback set 2023-10-30T12:16:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e180a1830 con 0x7f9e181312b0 2023-10-30T12:16:37.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.710+0000 7f9e17fff640 1 librados: init done 2023-10-30T12:16:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9dff7fe640 1 -- 172.21.15.50:0/518026897 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9e08005b00 con 0x7f9e181312b0 2023-10-30T12:16:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9dff7fe640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e040392f0 0x7f9e0403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9dff7fe640 1 -- 172.21.15.50:0/518026897 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(496..496 src has 417..496) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f9e08049ee0 con 0x7f9e181312b0 2023-10-30T12:16:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9dff7fe640 10 client.214110.objecter ms_dispatch 0x7f9e18000b90 osd_map(496..496 src has 417..496) v4 2023-10-30T12:16:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9dff7fe640 3 client.214110.objecter handle_osd_map got epochs [496,496] > 0 2023-10-30T12:16:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9dff7fe640 3 client.214110.objecter handle_osd_map decoding full epoch 496 2023-10-30T12:16:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9dff7fe640 20 client.214110.objecter dump_active .. 0 homeless 2023-10-30T12:16:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9e17fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9e17fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9e167fc640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e040392f0 0x7f9e0403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9e17fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1804e160 con 0x7f9e181312b0 2023-10-30T12:16:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.711+0000 7f9e167fc640 10 monclient: get_auth_request con 0x7f9e040392f0 auth_method 0 2023-10-30T12:16:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.712+0000 7f9e167fc640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e040392f0 0x7f9e0403b7b0 secure :-1 s=READY pgs=4223 cs=0 l=1 rev1=1 crypto rx=0x7f9e18134700 tx=0x7f9e0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.712+0000 7f9dff7fe640 10 client.214110.objecter ms_handle_connect 0x7f9e040392f0 2023-10-30T12:16:37.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.716+0000 7f9dff7fe640 1 -- 172.21.15.50:0/518026897 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9e08017020 con 0x7f9e181312b0 2023-10-30T12:16:37.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.716+0000 7f9dff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:37.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.716+0000 7f9dff7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:37.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.958+0000 7f9e17fff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:37.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.958+0000 7f9e17fff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:37.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.958+0000 7f9e17fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:37.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.958+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e18074390 con 0x7f9e181312b0 2023-10-30T12:16:37.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.959+0000 7f9dff7fe640 1 -- 172.21.15.50:0/518026897 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f9e08011910 con 0x7f9e181312b0 2023-10-30T12:16:37.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.959+0000 7f9dff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:37.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.959+0000 7f9dff7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:37.962 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":166,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:37.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 10 librados: watch_flush enter 2023-10-30T12:16:37.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 10 librados: watch_flush exit 2023-10-30T12:16:37.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 20 client.214110.objecter shutdown clearing up homeless session... 2023-10-30T12:16:37.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 10 client.214110.objecter successfully canceled tick 2023-10-30T12:16:37.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e040392f0 msgr2=0x7f9e0403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:37.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e040392f0 0x7f9e0403b7b0 secure :-1 s=READY pgs=4223 cs=0 l=1 rev1=1 crypto rx=0x7f9e18134700 tx=0x7f9e0c008040 comp rx=0 tx=0).stop 2023-10-30T12:16:37.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 10 monclient: shutdown 2023-10-30T12:16:37.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:37.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e181312b0 msgr2=0x7f9e1812f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:37.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e181312b0 0x7f9e1812f320 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c400 tx=0x7f9e0800c8c0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 shutdown_connections 2023-10-30T12:16:37.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9e040392f0 0x7f9e0403b7b0 unknown :-1 s=CLOSED pgs=4223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e18131ae0 0x7f9e181c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9e18131690 0x7f9e1812f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 --2- 172.21.15.50:0/518026897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e181312b0 0x7f9e1812f320 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:37.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 >> 172.21.15.50:0/518026897 conn(0x7f9e180a4200 msgr2=0x7f9e181308e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:37.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.960+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 shutdown_connections 2023-10-30T12:16:37.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.961+0000 7f9e17fff640 1 -- 172.21.15.50:0/518026897 wait complete. 2023-10-30T12:16:37.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:37.961+0000 7f9e17fff640 1 librados: shutdown 2023-10-30T12:16:40.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.087+0000 7f05177b6640 10 monclient: get_monmap_and_config 2023-10-30T12:16:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.087+0000 7f05177b6640 10 monclient: build_initial_monmap 2023-10-30T12:16:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.087+0000 7f05177b6640 10 monclient: monmap: 2023-10-30T12:16:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:41.088916+0000 2023-10-30T12:16:41.090 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:41.088916+0000 2023-10-30T12:16:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:41.091 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:41.092 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:41.092 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:41.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.088+0000 7f05177b6640 1 Processor -- start 2023-10-30T12:16:41.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.088+0000 7f05177b6640 1 -- start start 2023-10-30T12:16:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.088+0000 7f05177b6640 10 monclient: init 2023-10-30T12:16:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-10-30T12:16:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0510131b20 0x7f0510132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0510131b20 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:41.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0510071bc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:41.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0510071fc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:41.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051009a9b0 con 0x7f0510071bc0 2023-10-30T12:16:41.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0510131f00 con 0x7f0510071fc0 2023-10-30T12:16:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0510098230 con 0x7f0510131b20 2023-10-30T12:16:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f05177b6640 10 monclient(hunting): authenticate will time out at 1846.272163s 2023-10-30T12:16:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0514d51640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0510071bc0 0x7f0510097410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50204/0 (socket says 172.21.15.50:50204) 2023-10-30T12:16:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0514d51640 1 -- 172.21.15.50:0/1562529755 learned_addr learned my addr 172.21.15.50:0/1562529755 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 1 --2- 172.21.15.50:0/1562529755 >> v2:172.21.15.118:3300/0 conn(0x7f0510131b20 0x7f0510132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): get_auth_request con 0x7f0510131b20 auth_method 0 2023-10-30T12:16:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0507fff640 1 -- 172.21.15.50:0/1562529755 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298114317 0 0) 0x7f0510131f00 con 0x7f0510071fc0 2023-10-30T12:16:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0507fff640 10 monclient(hunting): _negotiate 2023-10-30T12:16:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0507fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0507fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0507fff640 10 monclient(hunting): my global_id is 204217 2023-10-30T12:16:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0507fff640 1 -- 172.21.15.50:0/1562529755 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc003440 con 0x7f0510071fc0 2023-10-30T12:16:41.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient(hunting): handle_auth_done global_id 204214 payload 274 2023-10-30T12:16:41.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 1 -- 172.21.15.50:0/1562529755 >> v1:172.21.15.118:6789/0 conn(0x7f0510071fc0 legacy=0x7f0510097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:41.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 1 -- 172.21.15.50:0/1562529755 >> v1:172.21.15.50:6789/0 conn(0x7f0510071bc0 legacy=0x7f0510097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:41.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient: _finish_hunting 0 2023-10-30T12:16:41.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 1 monclient: found mon.noname-c 2023-10-30T12:16:41.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:41.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:16:41.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.089+0000 7f0515552640 1 -- 172.21.15.50:0/1562529755 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051009d920 con 0x7f0510131b20 2023-10-30T12:16:41.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0515552640 1 --2- 172.21.15.50:0/1562529755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510131b20 0x7f0510132270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f051009db10 tx=0x7f050c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f5c5e4a7512577 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:41.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 1 -- 172.21.15.50:0/1562529755 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c031770 con 0x7f0510131b20 2023-10-30T12:16:41.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:41.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:41.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: dump: 2023-10-30T12:16:41.110 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:41.110 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:41.112 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: _finish_auth 0 2023-10-30T12:16:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: _check_auth_tickets 2023-10-30T12:16:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f05177b6640 5 monclient: authenticate success, global_id 204214 2023-10-30T12:16:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 1 -- 172.21.15.50:0/1562529755 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c0318d0 con 0x7f0510131b20 2023-10-30T12:16:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f05177b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:41.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f05177b6640 4 set_mon_vals no callback set 2023-10-30T12:16:41.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 1 -- 172.21.15.50:0/1562529755 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c031c60 con 0x7f0510131b20 2023-10-30T12:16:41.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:41.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:16:41.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.090+0000 7f0507fff640 10 monclient: dump: 2023-10-30T12:16:41.114 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:41.115 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:41.115 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.115 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.115 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:41.115 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:41.116 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:41.116 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:41.116 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:41.116 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:41.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 10 monclient: shutdown 2023-10-30T12:16:41.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 1 -- 172.21.15.50:0/1562529755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510131b20 msgr2=0x7f0510132270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 1 --2- 172.21.15.50:0/1562529755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510131b20 0x7f0510132270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f051009db10 tx=0x7f050c02b810 comp rx=0 tx=0).stop 2023-10-30T12:16:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 1 -- 172.21.15.50:0/1562529755 shutdown_connections 2023-10-30T12:16:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 1 --2- 172.21.15.50:0/1562529755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510131b20 0x7f0510132270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 1 -- 172.21.15.50:0/1562529755 >> 172.21.15.50:0/1562529755 conn(0x7f05100a05c0 msgr2=0x7f051009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:41.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 1 -- 172.21.15.50:0/1562529755 shutdown_connections 2023-10-30T12:16:41.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.091+0000 7f05177b6640 1 -- 172.21.15.50:0/1562529755 wait complete. 2023-10-30T12:16:41.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient: build_initial_monmap 2023-10-30T12:16:41.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient: monmap: 2023-10-30T12:16:41.120 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:41.120 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:41.120 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:41.120 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:41.121 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:41.121 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:41.121 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:41.121 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:41.121 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:41.122 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:41.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 librados: starting msgr at 2023-10-30T12:16:41.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 librados: starting objecter 2023-10-30T12:16:41.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 Processor -- start 2023-10-30T12:16:41.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 -- start start 2023-10-30T12:16:41.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 librados: setting wanted keys 2023-10-30T12:16:41.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 librados: calling monclient init 2023-10-30T12:16:41.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient: init 2023-10-30T12:16:41.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:41.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:16:41.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510071bc0 0x7f05101c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:41.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): picked mon.noname-b con 0x7f0510071bc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:41.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0510071fc0 0x7f05101308f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:41.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0510071fc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:41.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0510131b20 0x7f0510130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:41.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): picked mon.noname-a con 0x7f0510131b20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:41.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:41.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05101c6920 con 0x7f0510131b20 2023-10-30T12:16:41.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:41.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051012d0c0 con 0x7f0510071bc0 2023-10-30T12:16:41.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:41.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05100993c0 con 0x7f0510071fc0 2023-10-30T12:16:41.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:41.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.092+0000 7f05177b6640 10 monclient(hunting): authenticate will time out at 1846.275932s 2023-10-30T12:16:41.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0510131b20 0x7f0510130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:41.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0510071fc0 0x7f05101308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:41.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0510071fc0 0x7f05101308f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53664/0 (socket says 172.21.15.50:53664) 2023-10-30T12:16:41.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 -- 172.21.15.50:0/1501703887 learned_addr learned my addr 172.21.15.50:0/1501703887 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:41.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): get_auth_request con 0x7f0510071fc0 auth_method 0 2023-10-30T12:16:41.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:41.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0510131b20 0x7f0510130e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43100/0 (socket says 172.21.15.50:43100) 2023-10-30T12:16:41.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:41.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:41.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510071bc0 0x7f05101c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:41.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 10 monclient(hunting): get_auth_request con 0x7f0510131b20 auth_method 0 2023-10-30T12:16:41.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 10 monclient(hunting): get_auth_request con 0x7f0510071bc0 auth_method 0 2023-10-30T12:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:41.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:41.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:41.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:41.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:41.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:41.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:41.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515552640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:41.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient(hunting): handle_auth_done global_id 115157 payload 824 2023-10-30T12:16:41.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 -- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510071bc0 msgr2=0x7f05101c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:41.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510071bc0 0x7f05101c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:41.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 -- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0510131b20 msgr2=0x7f0510130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0510131b20 0x7f0510130e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient: _finish_hunting 0 2023-10-30T12:16:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 monclient: found mon.noname-c 2023-10-30T12:16:41.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 -- 172.21.15.50:0/1501703887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c0093a0 con 0x7f0510071fc0 2023-10-30T12:16:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0515d53640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0510131b20 0x7f0510130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0514d51640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0510071fc0 0x7f05101308f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f050000c4d0 tx=0x7f050000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 client.?.objecter ms_handle_connect 0x7f0510071fc0 2023-10-30T12:16:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 1 -- 172.21.15.50:0/1501703887 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500005ce0 con 0x7f0510071fc0 2023-10-30T12:16:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:41.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: dump: 2023-10-30T12:16:41.137 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:41.137 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:41.137 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.137 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.137 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:41.138 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:41.138 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:41.138 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:41.138 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:41.139 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:41.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: _finish_auth 0 2023-10-30T12:16:41.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:16:41.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:41.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 1 -- 172.21.15.50:0/1501703887 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0500005e40 con 0x7f0510071fc0 2023-10-30T12:16:41.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f05177b6640 5 monclient: authenticate success, global_id 115157 2023-10-30T12:16:41.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f05177b6640 10 monclient: _renew_subs 2023-10-30T12:16:41.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f05177b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:41.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051012bc10 con 0x7f0510071fc0 2023-10-30T12:16:41.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:41.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 1 -- 172.21.15.50:0/1501703887 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500007270 con 0x7f0510071fc0 2023-10-30T12:16:41.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:41.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:41.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f0505ffb640 10 monclient: dump: 2023-10-30T12:16:41.141 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:41.141 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:41.142 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.142 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:41.142 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:41.142 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:41.142 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:41.142 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:41.143 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:41.143 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:41.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.093+0000 7f05077fe640 4 set_mon_vals no callback set 2023-10-30T12:16:41.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.094+0000 7f05177b6640 10 client.115157.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:41.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.094+0000 7f05177b6640 10 monclient: _renew_subs 2023-10-30T12:16:41.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.094+0000 7f05177b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.094+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051012afd0 con 0x7f0510071fc0 2023-10-30T12:16:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.094+0000 7f05177b6640 1 librados: init done 2023-10-30T12:16:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.094+0000 7f0505ffb640 1 -- 172.21.15.50:0/1501703887 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f05000073d0 con 0x7f0510071fc0 2023-10-30T12:16:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f05177b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f05177b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:41.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f05177b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051004e160 con 0x7f0510071fc0 2023-10-30T12:16:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0505ffb640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f40392f0 0x7f04f403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0505ffb640 1 -- 172.21.15.50:0/1501703887 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(497..497 src has 417..497) v4 ==== 5398+0+0 (secure 0 0 0) 0x7f0500049dc0 con 0x7f0510071fc0 2023-10-30T12:16:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0505ffb640 10 client.115157.objecter ms_dispatch 0x7f0510000b90 osd_map(497..497 src has 417..497) v4 2023-10-30T12:16:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0505ffb640 3 client.115157.objecter handle_osd_map got epochs [497,497] > 0 2023-10-30T12:16:41.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0505ffb640 3 client.115157.objecter handle_osd_map decoding full epoch 497 2023-10-30T12:16:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0505ffb640 20 client.115157.objecter dump_active .. 0 homeless 2023-10-30T12:16:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0515552640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f40392f0 0x7f04f403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0515552640 10 monclient: get_auth_request con 0x7f04f40392f0 auth_method 0 2023-10-30T12:16:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0505ffb640 10 client.115157.objecter ms_handle_connect 0x7f04f40392f0 2023-10-30T12:16:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.095+0000 7f0515552640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f40392f0 0x7f04f403b7b0 secure :-1 s=READY pgs=4225 cs=0 l=1 rev1=1 crypto rx=0x7f050c036000 tx=0x7f050c03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.100+0000 7f0505ffb640 1 -- 172.21.15.50:0/1501703887 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f050001b3b0 con 0x7f0510071fc0 2023-10-30T12:16:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.100+0000 7f0505ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:41.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.100+0000 7f0505ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.343+0000 7f05177b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:41.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.343+0000 7f05177b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:41.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.343+0000 7f05177b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.343+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f05100745c0 con 0x7f0510071fc0 2023-10-30T12:16:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.344+0000 7f0505ffb640 1 -- 172.21.15.50:0/1501703887 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f0500017070 con 0x7f0510071fc0 2023-10-30T12:16:41.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.344+0000 7f0505ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.344+0000 7f0505ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:41.384 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":168,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.345+0000 7f05177b6640 10 librados: watch_flush enter 2023-10-30T12:16:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 10 librados: watch_flush exit 2023-10-30T12:16:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 20 client.115157.objecter shutdown clearing up homeless session... 2023-10-30T12:16:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 10 client.115157.objecter successfully canceled tick 2023-10-30T12:16:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f40392f0 msgr2=0x7f04f403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f40392f0 0x7f04f403b7b0 secure :-1 s=READY pgs=4225 cs=0 l=1 rev1=1 crypto rx=0x7f050c036000 tx=0x7f050c03d040 comp rx=0 tx=0).stop 2023-10-30T12:16:41.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 10 monclient: shutdown 2023-10-30T12:16:41.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:41.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0510071fc0 msgr2=0x7f05101308f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0510071fc0 0x7f05101308f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f050000c4d0 tx=0x7f050000c990 comp rx=0 tx=0).stop 2023-10-30T12:16:41.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 shutdown_connections 2023-10-30T12:16:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04f40392f0 0x7f04f403b7b0 unknown :-1 s=CLOSED pgs=4225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0510131b20 0x7f0510130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0510071fc0 0x7f05101308f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 --2- 172.21.15.50:0/1501703887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0510071bc0 0x7f05101c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:41.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 >> 172.21.15.50:0/1501703887 conn(0x7f05100a05c0 msgr2=0x7f051009ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 shutdown_connections 2023-10-30T12:16:41.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 -- 172.21.15.50:0/1501703887 wait complete. 2023-10-30T12:16:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:41.346+0000 7f05177b6640 1 librados: shutdown 2023-10-30T12:16:44.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:44.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.509+0000 7f3f5611d640 10 monclient: get_monmap_and_config 2023-10-30T12:16:44.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.509+0000 7f3f5611d640 10 monclient: build_initial_monmap 2023-10-30T12:16:44.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.509+0000 7f3f5611d640 10 monclient: monmap: 2023-10-30T12:16:44.511 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:44.512 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.512 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:44.510331+0000 2023-10-30T12:16:44.512 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:44.510331+0000 2023-10-30T12:16:44.512 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:44.513 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.513 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:44.513 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:44.513 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:44.513 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:44.514 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:44.514 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.509+0000 7f3f5611d640 1 Processor -- start 2023-10-30T12:16:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 1 -- start start 2023-10-30T12:16:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient: init 2023-10-30T12:16:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:16:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3f500a4b60 0x7f3f501326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f500a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f50099a40 0x7f3f50132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f50099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f500a2350 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f50133120 con 0x7f3f500a2350 2023-10-30T12:16:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f500a3d70 con 0x7f3f500a4b60 2023-10-30T12:16:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500a4f40 con 0x7f3f50099a40 2023-10-30T12:16:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f5611d640 10 monclient(hunting): authenticate will time out at 1849.693553s 2023-10-30T12:16:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f50099a40 0x7f3f50132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f50099a40 0x7f3f50132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53674/0 (socket says 172.21.15.50:53674) 2023-10-30T12:16:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 1 -- 172.21.15.50:0/3697753867 learned_addr learned my addr 172.21.15.50:0/3697753867 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): get_auth_request con 0x7f3f50099a40 auth_method 0 2023-10-30T12:16:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 1 --2- 172.21.15.50:0/3697753867 >> v2:172.21.15.118:3300/0 conn(0x7f3f500a4b60 0x7f3f501326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 10 monclient(hunting): get_auth_request con 0x7f3f500a4b60 auth_method 0 2023-10-30T12:16:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient(hunting): handle_auth_done global_id 115160 payload 274 2023-10-30T12:16:44.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 1 -- 172.21.15.50:0/3697753867 >> v2:172.21.15.118:3300/0 conn(0x7f3f500a4b60 msgr2=0x7f3f501326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:44.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 1 --2- 172.21.15.50:0/3697753867 >> v2:172.21.15.118:3300/0 conn(0x7f3f500a4b60 0x7f3f501326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 1 -- 172.21.15.50:0/3697753867 >> v1:172.21.15.118:6789/0 conn(0x7f3f500a2350 legacy=0x7f3f500a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.510+0000 7f3f4effd640 10 monclient: _finish_hunting 0 2023-10-30T12:16:44.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4effd640 1 monclient: found mon.noname-b 2023-10-30T12:16:44.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:16:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4effd640 1 -- 172.21.15.50:0/3697753867 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f500a4150 con 0x7f3f50099a40 2023-10-30T12:16:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4effd640 1 --2- 172.21.15.50:0/3697753867 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f50099a40 0x7f3f50132be0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f4402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359e514083583871 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 1 -- 172.21.15.50:0/3697753867 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440317c0 con 0x7f3f50099a40 2023-10-30T12:16:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:16:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: dump: 2023-10-30T12:16:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.532 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:16:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:16:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f5611d640 5 monclient: authenticate success, global_id 115160 2023-10-30T12:16:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f5611d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 1 -- 172.21.15.50:0/3697753867 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44031920 con 0x7f3f50099a40 2023-10-30T12:16:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f5611d640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f5611d640 4 set_mon_vals no callback set 2023-10-30T12:16:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 1 -- 172.21.15.50:0/3697753867 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44031cb0 con 0x7f3f50099a40 2023-10-30T12:16:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:44.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.511+0000 7f3f4e7fc640 10 monclient: dump: 2023-10-30T12:16:44.535 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.535 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.535 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.536 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.536 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.536 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.536 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.536 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.537 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.537 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 10 monclient: shutdown 2023-10-30T12:16:44.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:44.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 -- 172.21.15.50:0/3697753867 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f50099a40 msgr2=0x7f3f50132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3697753867 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f50099a40 0x7f3f50132be0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f4402b860 comp rx=0 tx=0).stop 2023-10-30T12:16:44.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 -- 172.21.15.50:0/3697753867 shutdown_connections 2023-10-30T12:16:44.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3697753867 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f50099a40 0x7f3f50132be0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3697753867 >> v2:172.21.15.118:3300/0 conn(0x7f3f500a4b60 0x7f3f501326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 -- 172.21.15.50:0/3697753867 >> 172.21.15.50:0/3697753867 conn(0x7f3f500a8060 msgr2=0x7f3f500a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:44.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 -- 172.21.15.50:0/3697753867 shutdown_connections 2023-10-30T12:16:44.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.512+0000 7f3f5611d640 1 -- 172.21.15.50:0/3697753867 wait complete. 2023-10-30T12:16:44.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 10 monclient: build_initial_monmap 2023-10-30T12:16:44.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 10 monclient: monmap: 2023-10-30T12:16:44.539 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:44.539 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:44.539 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:44.541 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:44.541 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 librados: starting msgr at 2023-10-30T12:16:44.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 librados: starting objecter 2023-10-30T12:16:44.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 Processor -- start 2023-10-30T12:16:44.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 -- start start 2023-10-30T12:16:44.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 librados: setting wanted keys 2023-10-30T12:16:44.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 librados: calling monclient init 2023-10-30T12:16:44.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 10 monclient: init 2023-10-30T12:16:44.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:44.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:16:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f500a2350 0x7f3f501ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f500a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f500a4b60 0x7f3f501cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.513+0000 7f3f5611d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f500a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f5011e480 0x7f3f5011e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f5011e480 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f501311e0 con 0x7f3f500a4b60 2023-10-30T12:16:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f50120d70 con 0x7f3f500a2350 2023-10-30T12:16:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f501228b0 con 0x7f3f5011e480 2023-10-30T12:16:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient(hunting): authenticate will time out at 1849.697049s 2023-10-30T12:16:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f5011e480 0x7f3f5011e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f500a4b60 0x7f3f501cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f5011e480 0x7f3f5011e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53678/0 (socket says 172.21.15.50:53678) 2023-10-30T12:16:44.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 -- 172.21.15.50:0/3549342823 learned_addr learned my addr 172.21.15.50:0/3549342823 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:44.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): get_auth_request con 0x7f3f5011e480 auth_method 0 2023-10-30T12:16:44.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 10 monclient(hunting): get_auth_request con 0x7f3f500a4b60 auth_method 0 2023-10-30T12:16:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f500a2350 0x7f3f501ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 10 monclient(hunting): get_auth_request con 0x7f3f500a2350 auth_method 0 2023-10-30T12:16:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient(hunting): handle_auth_done global_id 115163 payload 824 2023-10-30T12:16:44.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 -- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f500a2350 msgr2=0x7f3f501ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f500a2350 0x7f3f501ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 -- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f500a4b60 msgr2=0x7f3f501cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f500a4b60 0x7f3f501cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:16:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 monclient: found mon.noname-c 2023-10-30T12:16:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 -- 172.21.15.50:0/3549342823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f440094d0 con 0x7f3f5011e480 2023-10-30T12:16:44.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4effd640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f500a4b60 0x7f3f501cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:44.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4ffff640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f5011e480 0x7f3f5011e910 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800c7d0 tx=0x7f3f4800cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 client.?.objecter ms_handle_connect 0x7f3f5011e480 2023-10-30T12:16:44.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:44.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 1 -- 172.21.15.50:0/3549342823 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48003b70 con 0x7f3f5011e480 2023-10-30T12:16:44.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: dump: 2023-10-30T12:16:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.555 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: _finish_auth 0 2023-10-30T12:16:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: _check_auth_tickets 2023-10-30T12:16:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 1 -- 172.21.15.50:0/3549342823 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f48003cf0 con 0x7f3f5011e480 2023-10-30T12:16:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 5 monclient: authenticate success, global_id 115163 2023-10-30T12:16:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 1 -- 172.21.15.50:0/3549342823 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800aa90 con 0x7f3f5011e480 2023-10-30T12:16:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient: _renew_subs 2023-10-30T12:16:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:44.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f50135e60 con 0x7f3f5011e480 2023-10-30T12:16:44.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f4dffb640 4 set_mon_vals no callback set 2023-10-30T12:16:44.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:44.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f37fff640 10 monclient: dump: 2023-10-30T12:16:44.557 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.558 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.558 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.558 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.558 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.558 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.559 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.559 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.559 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.559 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 client.115163.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:44.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient: _renew_subs 2023-10-30T12:16:44.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:44.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f50122490 con 0x7f3f5011e480 2023-10-30T12:16:44.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.514+0000 7f3f5611d640 1 librados: init done 2023-10-30T12:16:44.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.515+0000 7f3f37fff640 1 -- 172.21.15.50:0/3549342823 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3f4800abf0 con 0x7f3f5011e480 2023-10-30T12:16:44.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.515+0000 7f3f5611d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:44.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.515+0000 7f3f5611d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:44.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.515+0000 7f3f5611d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:44.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.515+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5004e160 con 0x7f3f5011e480 2023-10-30T12:16:44.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f37fff640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f3c0392f0 0x7f3f3c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f37fff640 1 -- 172.21.15.50:0/3549342823 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(502..502 src has 417..502) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f3f4804ab60 con 0x7f3f5011e480 2023-10-30T12:16:44.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f37fff640 10 client.115163.objecter ms_dispatch 0x7f3f50000b90 osd_map(502..502 src has 417..502) v4 2023-10-30T12:16:44.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f37fff640 3 client.115163.objecter handle_osd_map got epochs [502,502] > 0 2023-10-30T12:16:44.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f37fff640 3 client.115163.objecter handle_osd_map decoding full epoch 502 2023-10-30T12:16:44.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f37fff640 20 client.115163.objecter dump_active .. 0 homeless 2023-10-30T12:16:44.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f4f7fe640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f3c0392f0 0x7f3f3c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f4f7fe640 10 monclient: get_auth_request con 0x7f3f3c0392f0 auth_method 0 2023-10-30T12:16:44.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f37fff640 10 client.115163.objecter ms_handle_connect 0x7f3f3c0392f0 2023-10-30T12:16:44.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.516+0000 7f3f4f7fe640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f3c0392f0 0x7f3f3c03b7b0 secure :-1 s=READY pgs=4226 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009b40 tx=0x7f3f40009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.520+0000 7f3f37fff640 1 -- 172.21.15.50:0/3549342823 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3f48003e60 con 0x7f3f5011e480 2023-10-30T12:16:44.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.520+0000 7f3f37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:44.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.520+0000 7f3f37fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:44.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.768+0000 7f3f5611d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:44.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.768+0000 7f3f5611d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:44.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.768+0000 7f3f5611d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:44.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.768+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3f500745c0 con 0x7f3f5011e480 2023-10-30T12:16:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.768+0000 7f3f37fff640 1 -- 172.21.15.50:0/3549342823 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f3f48011380 con 0x7f3f5011e480 2023-10-30T12:16:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.768+0000 7f3f37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.768+0000 7f3f37fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:44.771 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:44.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 10 librados: watch_flush enter 2023-10-30T12:16:44.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 10 librados: watch_flush exit 2023-10-30T12:16:44.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 20 client.115163.objecter shutdown clearing up homeless session... 2023-10-30T12:16:44.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 10 client.115163.objecter successfully canceled tick 2023-10-30T12:16:44.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f3c0392f0 msgr2=0x7f3f3c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f3c0392f0 0x7f3f3c03b7b0 secure :-1 s=READY pgs=4226 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009b40 tx=0x7f3f40009190 comp rx=0 tx=0).stop 2023-10-30T12:16:44.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 10 monclient: shutdown 2023-10-30T12:16:44.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:44.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f5011e480 msgr2=0x7f3f5011e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f5011e480 0x7f3f5011e910 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800c7d0 tx=0x7f3f4800cc90 comp rx=0 tx=0).stop 2023-10-30T12:16:44.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 shutdown_connections 2023-10-30T12:16:44.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f3c0392f0 0x7f3f3c03b7b0 unknown :-1 s=CLOSED pgs=4226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f5011e480 0x7f3f5011e910 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f500a4b60 0x7f3f501cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 --2- 172.21.15.50:0/3549342823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f500a2350 0x7f3f501ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 >> 172.21.15.50:0/3549342823 conn(0x7f3f500a8060 msgr2=0x7f3f50134020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:44.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 shutdown_connections 2023-10-30T12:16:44.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 -- 172.21.15.50:0/3549342823 wait complete. 2023-10-30T12:16:44.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.770+0000 7f3f5611d640 1 librados: shutdown 2023-10-30T12:16:44.782 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:16:44.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:16:44.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:16:44.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.887+0000 7fc6faae5640 10 monclient: get_monmap_and_config 2023-10-30T12:16:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.887+0000 7fc6faae5640 10 monclient: build_initial_monmap 2023-10-30T12:16:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.887+0000 7fc6faae5640 10 monclient: monmap: 2023-10-30T12:16:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.889 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:44.888080+0000 2023-10-30T12:16:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:44.888080+0000 2023-10-30T12:16:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:44.891 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:44.892 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.887+0000 7fc6faae5640 1 Processor -- start 2023-10-30T12:16:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.887+0000 7fc6faae5640 1 -- start start 2023-10-30T12:16:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.887+0000 7fc6faae5640 10 monclient: init 2023-10-30T12:16:44.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:44.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-10-30T12:16:44.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6f4096540 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:44.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc6f4136960 0x7fc6f4138d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient(hunting): picked mon.noname-a con 0x7fc6f4136960 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:44.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6f4139290 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:44.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f409bd30 con 0x7fc6f4139290 2023-10-30T12:16:44.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f409c400 con 0x7fc6f4096540 2023-10-30T12:16:44.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f409c120 con 0x7fc6f4136960 2023-10-30T12:16:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6faae5640 10 monclient(hunting): authenticate will time out at 1850.071223s 2023-10-30T12:16:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc6f4136960 0x7fc6f4138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f9082640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc6f4139290 0x7fc6f413b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50216/0 (socket says 172.21.15.50:50216) 2023-10-30T12:16:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f9082640 1 -- 172.21.15.50:0/568271837 learned_addr learned my addr 172.21.15.50:0/568271837 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): get_auth_request con 0x7fc6f4136960 auth_method 0 2023-10-30T12:16:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_done global_id 214125 payload 274 2023-10-30T12:16:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 1 -- 172.21.15.50:0/568271837 >> v1:172.21.15.118:6789/0 conn(0x7fc6f4096540 legacy=0x7fc6f4096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 1 -- 172.21.15.50:0/568271837 >> v1:172.21.15.50:6789/0 conn(0x7fc6f4139290 legacy=0x7fc6f413b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:16:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 1 monclient: found mon.noname-a 2023-10-30T12:16:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:44.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f37fe640 1 -- 172.21.15.50:0/568271837 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123669470 0 0) 0x7fc6f409bd30 con 0x7fc6f4139290 2023-10-30T12:16:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f3fff640 1 -- 172.21.15.50:0/568271837 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4097030 con 0x7fc6f4136960 2023-10-30T12:16:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.888+0000 7fc6f37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:16:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f3fff640 1 --2- 172.21.15.50:0/568271837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4136960 0x7fc6f4138d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80097a0 tx=0x7fc6e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cc7004526d7a93 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 1 -- 172.21.15.50:0/568271837 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e802e070 con 0x7fc6f4136960 2023-10-30T12:16:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:44.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: dump: 2023-10-30T12:16:44.913 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.913 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.913 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.914 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.914 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.914 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.914 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.915 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.915 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: _finish_auth 0 2023-10-30T12:16:44.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:16:44.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:44.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 1 -- 172.21.15.50:0/568271837 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e80315a0 con 0x7fc6f4136960 2023-10-30T12:16:44.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6faae5640 5 monclient: authenticate success, global_id 214125 2023-10-30T12:16:44.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6faae5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:44.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:44.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6faae5640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:44.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6faae5640 4 set_mon_vals no callback set 2023-10-30T12:16:44.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 1 -- 172.21.15.50:0/568271837 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8031930 con 0x7fc6f4136960 2023-10-30T12:16:44.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:44.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.889+0000 7fc6f37fe640 10 monclient: dump: 2023-10-30T12:16:44.918 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.918 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.919 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.919 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.919 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.919 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.919 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.920 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.920 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.920 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 10 monclient: shutdown 2023-10-30T12:16:44.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:44.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 -- 172.21.15.50:0/568271837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4136960 msgr2=0x7fc6f4138d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 --2- 172.21.15.50:0/568271837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4136960 0x7fc6f4138d50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80097a0 tx=0x7fc6e802b840 comp rx=0 tx=0).stop 2023-10-30T12:16:44.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 -- 172.21.15.50:0/568271837 shutdown_connections 2023-10-30T12:16:44.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 --2- 172.21.15.50:0/568271837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4136960 0x7fc6f4138d50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 -- 172.21.15.50:0/568271837 >> 172.21.15.50:0/568271837 conn(0x7fc6f40a3980 msgr2=0x7fc6f40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:44.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 -- 172.21.15.50:0/568271837 shutdown_connections 2023-10-30T12:16:44.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 -- 172.21.15.50:0/568271837 wait complete. 2023-10-30T12:16:44.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 10 monclient: build_initial_monmap 2023-10-30T12:16:44.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 10 monclient: monmap: 2023-10-30T12:16:44.924 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:44.924 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:44.924 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:44.925 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:44.925 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:44.925 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.926 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:44.926 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:44.926 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:44.927 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 librados: starting msgr at 2023-10-30T12:16:44.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.890+0000 7fc6faae5640 1 librados: starting objecter 2023-10-30T12:16:44.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 Processor -- start 2023-10-30T12:16:44.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 -- start start 2023-10-30T12:16:44.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 librados: setting wanted keys 2023-10-30T12:16:44.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 librados: calling monclient init 2023-10-30T12:16:44.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient: init 2023-10-30T12:16:44.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:44.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:16:44.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4096540 0x7fc6f41c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): picked mon.noname-a con 0x7fc6f4096540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:44.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6f4136960 0x7fc6f41c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): picked mon.noname-c con 0x7fc6f4136960 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:44.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6f4139290 0x7fc6f41c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): picked mon.noname-b con 0x7fc6f4139290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:44.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f408d130 con 0x7fc6f4096540 2023-10-30T12:16:44.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f408b3a0 con 0x7fc6f4139290 2023-10-30T12:16:44.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f4093e30 con 0x7fc6f4136960 2023-10-30T12:16:44.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:44.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6faae5640 10 monclient(hunting): authenticate will time out at 1850.074763s 2023-10-30T12:16:44.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4096540 0x7fc6f41c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6f4136960 0x7fc6f41c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4096540 0x7fc6f41c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43118/0 (socket says 172.21.15.50:43118) 2023-10-30T12:16:44.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 1 -- 172.21.15.50:0/1530558398 learned_addr learned my addr 172.21.15.50:0/1530558398 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:44.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 10 monclient(hunting): get_auth_request con 0x7fc6f4096540 auth_method 0 2023-10-30T12:16:44.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f3fff640 10 monclient(hunting): get_auth_request con 0x7fc6f4136960 auth_method 0 2023-10-30T12:16:44.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f9082640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6f4139290 0x7fc6f41c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f8881640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f9082640 10 monclient(hunting): get_auth_request con 0x7fc6f4139290 auth_method 0 2023-10-30T12:16:44.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f9082640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f9082640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.891+0000 7fc6f9082640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f9082640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f9082640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f9082640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 10 monclient(hunting): handle_auth_done global_id 214131 payload 824 2023-10-30T12:16:44.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6f4136960 msgr2=0x7fc6f41c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6f4136960 0x7fc6f41c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6f4139290 msgr2=0x7fc6f41c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6f4139290 0x7fc6f41c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 10 monclient: _finish_hunting 0 2023-10-30T12:16:44.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 1 monclient: found mon.noname-a 2023-10-30T12:16:44.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:44.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:44.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 1 -- 172.21.15.50:0/1530558398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e8009400 con 0x7fc6f4096540 2023-10-30T12:16:44.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f8881640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4096540 0x7fc6f41c8420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4004640 tx=0x7fc6e400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 client.?.objecter ms_handle_connect 0x7fc6f4096540 2023-10-30T12:16:44.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:44.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4011790 con 0x7fc6f4096540 2023-10-30T12:16:44.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:44.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: dump: 2023-10-30T12:16:44.943 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.943 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.943 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.944 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.944 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.944 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.944 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.944 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.945 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.945 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: _finish_auth 0 2023-10-30T12:16:44.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: _check_auth_tickets 2023-10-30T12:16:44.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:44.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 5 monclient: authenticate success, global_id 214131 2023-10-30T12:16:44.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e40118f0 con 0x7fc6f4096540 2023-10-30T12:16:44.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 10 monclient: _renew_subs 2023-10-30T12:16:44.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:44.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f408d5f0 con 0x7fc6f4096540 2023-10-30T12:16:44.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:44.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4011ce0 con 0x7fc6f4096540 2023-10-30T12:16:44.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:44.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f17fa640 10 monclient: dump: 2023-10-30T12:16:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.950 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.951 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6f2ffd640 4 set_mon_vals no callback set 2023-10-30T12:16:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 10 client.214131.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:44.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 10 monclient: _renew_subs 2023-10-30T12:16:44.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:44.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f409a080 con 0x7fc6f4096540 2023-10-30T12:16:44.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.892+0000 7fc6faae5640 1 librados: init done 2023-10-30T12:16:44.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6faae5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:44.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6faae5640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406cbb0 0x7fc6f406cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6faae5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f4073ff0 con 0x7fc6f406cbb0 2023-10-30T12:16:44.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc6e401c460 con 0x7fc6f4096540 2023-10-30T12:16:44.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406cbb0 0x7fc6f406cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient: get_auth_request con 0x7fc6f406cbb0 auth_method 0 2023-10-30T12:16:44.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6e00392f0 0x7fc6e003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(502..502 src has 417..502) v4 ==== 6571+0+0 (secure 0 0 0) 0x7fc6e404ab50 con 0x7fc6f4096540 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 10 client.214131.objecter ms_dispatch 0x7fc6f4000b90 osd_map(502..502 src has 417..502) v4 2023-10-30T12:16:44.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 3 client.214131.objecter handle_osd_map got epochs [502,502] > 0 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 3 client.214131.objecter handle_osd_map decoding full epoch 502 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_done global_id 214134 payload 824 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 20 client.214131.objecter dump_active .. 0 homeless 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 1 -- 172.21.15.50:0/1530558398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6f409ba70 con 0x7fc6f406cbb0 2023-10-30T12:16:44.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f9082640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6e00392f0 0x7fc6e003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f9082640 10 monclient: get_auth_request con 0x7fc6e00392f0 auth_method 0 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 10 client.214131.objecter ms_handle_connect 0x7fc6f406cbb0 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f17fa640 10 client.214131.objecter resend_mon_ops 2023-10-30T12:16:44.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.893+0000 7fc6f3fff640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406cbb0 0x7fc6f406cf90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6f409ffe0 tx=0x7fc6e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8041040 con 0x7fc6f406cbb0 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:44.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 10 monclient: dump: 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.959 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.960 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f9082640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6e00392f0 0x7fc6e003b7b0 secure :-1 s=READY pgs=4227 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0018a0 tx=0x7fc6ec001830 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 10 client.214131.objecter ms_handle_connect 0x7fc6e00392f0 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc6e8003610 con 0x7fc6f406cbb0 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:44.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406cbb0 msgr2=0x7fc6f406cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406cbb0 0x7fc6f406cf90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6f409ffe0 tx=0x7fc6e8004730 comp rx=0 tx=0).stop 2023-10-30T12:16:44.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406cbb0 msgr2=0x7fc6f406cf90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:44.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.894+0000 7fc6f17fa640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406cbb0 0x7fc6f406cf90 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6f409ffe0 tx=0x7fc6e8004730 comp rx=0 tx=0).stop 2023-10-30T12:16:44.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6faae5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:16:44.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6faae5640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406d4d0 0x7fc6f4070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:44.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6faae5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:44.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f404e160 con 0x7fc6f406d4d0 2023-10-30T12:16:44.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6f3fff640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406d4d0 0x7fc6f4070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:44.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6f3fff640 10 monclient: get_auth_request con 0x7fc6f406d4d0 auth_method 0 2023-10-30T12:16:44.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:44.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6f3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:44.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6f3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:44.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:44.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.900+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:44.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:44.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f3fff640 10 monclient(hunting): handle_auth_done global_id 214137 payload 824 2023-10-30T12:16:44.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f3fff640 1 -- 172.21.15.50:0/1530558398 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc6f408cee0 con 0x7fc6f406d4d0 2023-10-30T12:16:44.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 10 client.214131.objecter ms_handle_connect 0x7fc6f406d4d0 2023-10-30T12:16:44.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f3fff640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406d4d0 0x7fc6f4070400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6e802b880 tx=0x7fc6e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:44.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 10 client.214131.objecter resend_mon_ops 2023-10-30T12:16:44.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8046c40 con 0x7fc6f406d4d0 2023-10-30T12:16:44.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:44.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:44.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 10 monclient: dump: 2023-10-30T12:16:44.967 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:44.967 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:44.967 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.967 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:44.967 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:44.968 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:44.968 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:44.968 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:44.968 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:44.968 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:44.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7fc6e8046da0 con 0x7fc6f406d4d0 2023-10-30T12:16:44.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:16:44.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:44.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406d4d0 msgr2=0x7fc6f4070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:44.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406d4d0 0x7fc6f4070400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6e802b880 tx=0x7fc6e80046c0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406d4d0 msgr2=0x7fc6f4070400 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:44.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6f17fa640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f406d4d0 0x7fc6f4070400 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6e802b880 tx=0x7fc6e80046c0 comp rx=0 tx=0).stop 2023-10-30T12:16:44.970 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:16:44.971 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:44.971 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:44.971 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:16:44.971 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 170, 2023-10-30T12:16:44.971 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:16:44.972 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:16:44.972 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:16:44.972 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:16:44.972 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:44.972 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:16:44.973 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:44.973 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:16:44.973 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:16:44.973 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:44.973 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:44.974 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:44.974 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:44.974 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:44.974 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:44.974 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:44.975 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:44.975 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:44.975 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:44.976 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:44.976 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:16:44.976 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:16:44.976 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:44.977 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:44.977 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:44.977 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:44.977 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:44.977 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:44.978 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:44.978 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:44.978 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:44.979 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:44.979 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:44.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.979 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:16:44.980 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:16:44.980 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:16:44.980 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:16:44.980 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:16:44.981 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:16:44.981 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:44.981 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:44.981 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:16:44.982 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:16:44.982 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:16:44.982 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:16:44.982 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:16:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:16:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:16:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:16:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:44.984 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:44.984 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:44.984 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:44.984 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:44.984 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:44.985 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:44.985 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:44.985 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:44.985 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:44.985 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:16:44.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.986 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:16:44.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.986 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:44.986 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:44.987 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:44.987 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:44.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.987 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:44.987 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:16:44.988 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:44.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.988 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:44.989 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:16:44.989 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:44.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:44.989 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:44.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.990 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:16:44.990 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:16:44.990 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:44.990 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:44.991 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:44.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.991 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:44.991 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:44.992 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:44.992 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:44.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.992 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:44.992 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:16:44.993 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:44.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.993 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:44.993 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:16:44.993 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:44.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:44.994 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:44.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.994 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:16:44.995 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:16:44.995 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:44.995 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:44.995 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:44.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.996 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:44.996 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:44.996 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:44.996 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:44.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.997 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:44.997 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:16:44.997 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:44.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:44.998 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:44.998 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:16:44.998 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:44.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:44.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:44.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:44.999 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:16:44.999 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:16:44.999 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:44.999 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:45.000 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:45.000 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.000 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.000 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:16:45.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:16:45.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.001 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.001 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.001 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:45.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.002 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.002 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:16:45.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.002 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.003 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.003 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:16:45.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.003 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.004 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:16:45.004 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:16:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6faae5640 10 librados: watch_flush enter 2023-10-30T12:16:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6faae5640 10 librados: watch_flush exit 2023-10-30T12:16:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6faae5640 20 client.214131.objecter shutdown clearing up homeless session... 2023-10-30T12:16:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.901+0000 7fc6faae5640 10 client.214131.objecter successfully canceled tick 2023-10-30T12:16:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6e00392f0 msgr2=0x7fc6e003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6e00392f0 0x7fc6e003b7b0 secure :-1 s=READY pgs=4227 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0018a0 tx=0x7fc6ec001830 comp rx=0 tx=0).stop 2023-10-30T12:16:45.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 10 monclient: shutdown 2023-10-30T12:16:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4096540 msgr2=0x7fc6f41c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 --2- 172.21.15.50:0/1530558398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6f4096540 0x7fc6f41c8420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4004640 tx=0x7fc6e400c8e0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6f9082640 1 -- 172.21.15.50:0/1530558398 reap_dead start 2023-10-30T12:16:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6f9082640 1 -- 172.21.15.50:0/1530558398 reap_dead start 2023-10-30T12:16:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 shutdown_connections 2023-10-30T12:16:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 >> 172.21.15.50:0/1530558398 conn(0x7fc6f40a3980 msgr2=0x7fc6f409a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 shutdown_connections 2023-10-30T12:16:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 -- 172.21.15.50:0/1530558398 wait complete. 2023-10-30T12:16:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:44.902+0000 7fc6faae5640 1 librados: shutdown 2023-10-30T12:16:45.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:16:45.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:16:45.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:16:45.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.116+0000 7f66f7f4e640 10 monclient: get_monmap_and_config 2023-10-30T12:16:45.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.116+0000 7f66f7f4e640 10 monclient: build_initial_monmap 2023-10-30T12:16:45.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.116+0000 7f66f7f4e640 10 monclient: monmap: 2023-10-30T12:16:45.118 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:45.118 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.118 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:45.117309+0000 2023-10-30T12:16:45.119 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:45.117309+0000 2023-10-30T12:16:45.119 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:45.119 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.120 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:45.120 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:45.120 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:45.121 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:45.121 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:45.122 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.116+0000 7f66f7f4e640 1 Processor -- start 2023-10-30T12:16:45.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 1 -- start start 2023-10-30T12:16:45.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient: init 2023-10-30T12:16:45.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:45.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-10-30T12:16:45.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66f009e7e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:45.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient(hunting): picked mon.noname-b con 0x7f66f009f2d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:45.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66f0138b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:45.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f009d560 con 0x7f66f009e7e0 2023-10-30T12:16:45.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f009fbf0 con 0x7f66f0138b60 2023-10-30T12:16:45.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f009fed0 con 0x7f66f009f2d0 2023-10-30T12:16:45.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:45.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f7f4e640 10 monclient(hunting): authenticate will time out at 1850.300599s 2023-10-30T12:16:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f5cea640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f66f009e7e0 0x7f66f009ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50230/0 (socket says 172.21.15.50:50230) 2023-10-30T12:16:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f5cea640 1 -- 172.21.15.50:0/592906337 learned_addr learned my addr 172.21.15.50:0/592906337 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 1 --2- 172.21.15.50:0/592906337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): get_auth_request con 0x7f66f009f2d0 auth_method 0 2023-10-30T12:16:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f4ce8640 1 -- 172.21.15.50:0/592906337 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972567896 0 0) 0x7f66f009d560 con 0x7f66f009e7e0 2023-10-30T12:16:45.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f4ce8640 10 monclient(hunting): _negotiate 2023-10-30T12:16:45.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f4ce8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f4ce8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f4ce8640 10 monclient(hunting): my global_id is 214140 2023-10-30T12:16:45.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f4ce8640 1 -- 172.21.15.50:0/592906337 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d8003520 con 0x7f66f009e7e0 2023-10-30T12:16:45.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 10 monclient(hunting): handle_auth_done global_id 115169 payload 274 2023-10-30T12:16:45.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.117+0000 7f66f54e9640 1 -- 172.21.15.50:0/592906337 >> v1:172.21.15.118:6789/0 conn(0x7f66f0138b60 legacy=0x7f66f013af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:45.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f54e9640 1 -- 172.21.15.50:0/592906337 >> v1:172.21.15.50:6789/0 conn(0x7f66f009e7e0 legacy=0x7f66f009ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f54e9640 10 monclient: _finish_hunting 0 2023-10-30T12:16:45.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f54e9640 1 monclient: found mon.noname-b 2023-10-30T12:16:45.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f54e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:45.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f54e9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:16:45.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f54e9640 1 -- 172.21.15.50:0/592906337 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00a01b0 con 0x7f66f009f2d0 2023-10-30T12:16:45.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f54e9640 1 --2- 172.21.15.50:0/592906337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f006c3c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f66e40097d0 tx=0x7f66e402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4fc12e3ce8b2a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 1 -- 172.21.15.50:0/592906337 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40316f0 con 0x7f66f009f2d0 2023-10-30T12:16:45.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:16:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: dump: 2023-10-30T12:16:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.151 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: _finish_auth 0 2023-10-30T12:16:45.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: _check_auth_tickets 2023-10-30T12:16:45.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:45.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 1 -- 172.21.15.50:0/592906337 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e4031850 con 0x7f66f009f2d0 2023-10-30T12:16:45.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:45.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 1 -- 172.21.15.50:0/592906337 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e4031be0 con 0x7f66f009f2d0 2023-10-30T12:16:45.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f7f4e640 5 monclient: authenticate success, global_id 115169 2023-10-30T12:16:45.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f7f4e640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:45.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f7f4e640 4 set_mon_vals no callback set 2023-10-30T12:16:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.118+0000 7f66f4ce8640 10 monclient: dump: 2023-10-30T12:16:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.153 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.154 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 10 monclient: shutdown 2023-10-30T12:16:45.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:45.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 1 -- 172.21.15.50:0/592906337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 msgr2=0x7f66f006c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 1 --2- 172.21.15.50:0/592906337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f006c3c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f66e40097d0 tx=0x7f66e402b790 comp rx=0 tx=0).stop 2023-10-30T12:16:45.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 1 -- 172.21.15.50:0/592906337 shutdown_connections 2023-10-30T12:16:45.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 1 --2- 172.21.15.50:0/592906337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f006c3c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 1 -- 172.21.15.50:0/592906337 >> 172.21.15.50:0/592906337 conn(0x7f66f00a4800 msgr2=0x7f66f00a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:45.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 1 -- 172.21.15.50:0/592906337 shutdown_connections 2023-10-30T12:16:45.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.119+0000 7f66f7f4e640 1 -- 172.21.15.50:0/592906337 wait complete. 2023-10-30T12:16:45.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient: build_initial_monmap 2023-10-30T12:16:45.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient: monmap: 2023-10-30T12:16:45.157 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:45.158 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:45.158 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:45.158 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:45.158 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:45.158 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.159 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:45.159 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:45.159 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:45.159 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 librados: starting msgr at 2023-10-30T12:16:45.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 librados: starting objecter 2023-10-30T12:16:45.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 Processor -- start 2023-10-30T12:16:45.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 -- start start 2023-10-30T12:16:45.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 librados: setting wanted keys 2023-10-30T12:16:45.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 librados: calling monclient init 2023-10-30T12:16:45.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient: init 2023-10-30T12:16:45.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:45.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:16:45.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66f009e7e0 0x7f66f009c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient(hunting): picked mon.noname-b con 0x7f66f009e7e0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f0133320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient(hunting): picked mon.noname-c con 0x7f66f009f2d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66f0138b60 0x7f66f0136e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient(hunting): picked mon.noname-a con 0x7f66f0138b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:45.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f011ea70 con 0x7f66f0138b60 2023-10-30T12:16:45.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f013ba10 con 0x7f66f009e7e0 2023-10-30T12:16:45.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.120+0000 7f66f7f4e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f01cb3a0 con 0x7f66f009f2d0 2023-10-30T12:16:45.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:45.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 10 monclient(hunting): authenticate will time out at 1850.304016s 2023-10-30T12:16:45.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66f0138b60 0x7f66f0136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f0133320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66f0138b60 0x7f66f0136e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43150/0 (socket says 172.21.15.50:43150) 2023-10-30T12:16:45.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 -- 172.21.15.50:0/2768431355 learned_addr learned my addr 172.21.15.50:0/2768431355 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:45.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): get_auth_request con 0x7f66f0138b60 auth_method 0 2023-10-30T12:16:45.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66f009e7e0 0x7f66f009c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 10 monclient(hunting): get_auth_request con 0x7f66f009f2d0 auth_method 0 2023-10-30T12:16:45.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 10 monclient(hunting): get_auth_request con 0x7f66f009e7e0 auth_method 0 2023-10-30T12:16:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient(hunting): handle_auth_done global_id 214143 payload 824 2023-10-30T12:16:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 msgr2=0x7f66f0133320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f0133320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66f009e7e0 msgr2=0x7f66f009c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66f009e7e0 0x7f66f009c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient: _finish_hunting 0 2023-10-30T12:16:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 monclient: found mon.noname-a 2023-10-30T12:16:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 -- 172.21.15.50:0/2768431355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e4009400 con 0x7f66f0138b60 2023-10-30T12:16:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f54e9640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f009f2d0 0x7f66f0133320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f5cea640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f66f009e7e0 0x7f66f009c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f64eb640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66f0138b60 0x7f66f0136e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c700 tx=0x7f66e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 client.?.objecter ms_handle_connect 0x7f66f0138b60 2023-10-30T12:16:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e000aa50 con 0x7f66f0138b60 2023-10-30T12:16:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: dump: 2023-10-30T12:16:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.177 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.177 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.177 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.177 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.177 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.178 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.178 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: _finish_auth 0 2023-10-30T12:16:45.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:16:45.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:45.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 5 monclient: authenticate success, global_id 214143 2023-10-30T12:16:45.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e000abb0 con 0x7f66f0138b60 2023-10-30T12:16:45.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:45.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e000ce10 con 0x7f66f0138b60 2023-10-30T12:16:45.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:45.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66ee7fc640 10 monclient: dump: 2023-10-30T12:16:45.180 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.180 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.181 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.181 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.181 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.182 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.182 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.182 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.183 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66effff640 4 set_mon_vals no callback set 2023-10-30T12:16:45.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 10 monclient: _renew_subs 2023-10-30T12:16:45.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f009cbe0 con 0x7f66f0138b60 2023-10-30T12:16:45.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 10 client.214143.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:45.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 10 monclient: _renew_subs 2023-10-30T12:16:45.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f009b570 con 0x7f66f0138b60 2023-10-30T12:16:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.121+0000 7f66f7f4e640 1 librados: init done 2023-10-30T12:16:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f66e0012420 con 0x7f66f0138b60 2023-10-30T12:16:45.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f7f4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:45.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f7f4e640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f01c14a0 0x7f66f002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f7f4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00a1280 con 0x7f66f01c14a0 2023-10-30T12:16:45.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f01c14a0 0x7f66f002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient: get_auth_request con 0x7f66f01c14a0 auth_method 0 2023-10-30T12:16:45.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66ee7fc640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66dc0392f0 0x7f66dc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(502..502 src has 417..502) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f66e004b150 con 0x7f66f0138b60 2023-10-30T12:16:45.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66ee7fc640 10 client.214143.objecter ms_dispatch 0x7f66f0000b90 osd_map(502..502 src has 417..502) v4 2023-10-30T12:16:45.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66ee7fc640 3 client.214143.objecter handle_osd_map got epochs [502,502] > 0 2023-10-30T12:16:45.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66ee7fc640 3 client.214143.objecter handle_osd_map decoding full epoch 502 2023-10-30T12:16:45.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 10 monclient(hunting): handle_auth_done global_id 115175 payload 824 2023-10-30T12:16:45.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.122+0000 7f66f5cea640 1 -- 172.21.15.50:0/2768431355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66f009e060 con 0x7f66f01c14a0 2023-10-30T12:16:45.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66f54e9640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66dc0392f0 0x7f66dc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 20 client.214143.objecter dump_active .. 0 homeless 2023-10-30T12:16:45.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66f54e9640 10 monclient: get_auth_request con 0x7f66dc0392f0 auth_method 0 2023-10-30T12:16:45.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 client.214143.objecter ms_handle_connect 0x7f66f01c14a0 2023-10-30T12:16:45.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 client.214143.objecter resend_mon_ops 2023-10-30T12:16:45.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66f5cea640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f01c14a0 0x7f66f002b430 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f66e8000f60 tx=0x7f66e8001ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e800f9e0 con 0x7f66f01c14a0 2023-10-30T12:16:45.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:45.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 monclient: dump: 2023-10-30T12:16:45.197 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.197 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.197 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.198 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.198 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.198 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.199 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.199 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.199 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.200 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 client.214143.objecter ms_handle_connect 0x7f66dc0392f0 2023-10-30T12:16:45.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66f54e9640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66dc0392f0 0x7f66dc03b7b0 secure :-1 s=READY pgs=4228 cs=0 l=1 rev1=1 crypto rx=0x7f66e4036000 tx=0x7f66e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f66e800fb40 con 0x7f66f01c14a0 2023-10-30T12:16:45.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:45.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:45.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f01c14a0 msgr2=0x7f66f002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f01c14a0 0x7f66f002b430 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f66e8000f60 tx=0x7f66e8001ca0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f01c14a0 msgr2=0x7f66f002b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:45.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.123+0000 7f66ee7fc640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f01c14a0 0x7f66f002b430 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f7f4e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:16:45.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f7f4e640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f002b970 0x7f66f002bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f7f4e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0074800 con 0x7f66f002b970 2023-10-30T12:16:45.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f5cea640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f002b970 0x7f66f002bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f5cea640 10 monclient: get_auth_request con 0x7f66f002b970 auth_method 0 2023-10-30T12:16:45.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f5cea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f5cea640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.129+0000 7f66f5cea640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f5cea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f5cea640 10 monclient(hunting): handle_auth_done global_id 115178 payload 824 2023-10-30T12:16:45.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f5cea640 1 -- 172.21.15.50:0/2768431355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f66f0070e40 con 0x7f66f002b970 2023-10-30T12:16:45.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 10 client.214143.objecter ms_handle_connect 0x7f66f002b970 2023-10-30T12:16:45.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 10 client.214143.objecter resend_mon_ops 2023-10-30T12:16:45.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f5cea640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f002b970 0x7f66f002bd80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f66e8018000 tx=0x7f66e8001150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8023070 con 0x7f66f002b970 2023-10-30T12:16:45.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:45.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 10 monclient: dump: 2023-10-30T12:16:45.211 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.211 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.211 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.212 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.212 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.213 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.213 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.213 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.214 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.214 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f66e8028470 con 0x7f66f002b970 2023-10-30T12:16:45.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:16:45.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:45.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f002b970 msgr2=0x7f66f002bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f002b970 0x7f66f002bd80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f66e8018000 tx=0x7f66e8001150 comp rx=0 tx=0).stop 2023-10-30T12:16:45.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f002b970 msgr2=0x7f66f002bd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:45.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66ee7fc640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66f002b970 0x7f66f002bd80 secure :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f66e8018000 tx=0x7f66e8001150 comp rx=0 tx=0).stop 2023-10-30T12:16:45.217 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:16:45.217 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:45.218 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:45.218 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:16:45.218 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 170, 2023-10-30T12:16:45.218 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:16:45.219 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:16:45.219 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:16:45.219 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:16:45.219 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.220 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:16:45.220 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:45.220 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:16:45.220 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:16:45.220 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:45.221 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:45.221 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:45.221 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:45.221 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:45.221 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:45.221 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:45.222 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:45.222 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:45.222 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:45.222 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.222 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:16:45.223 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:16:45.223 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:45.223 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:45.223 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:45.223 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:45.224 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:45.224 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:45.224 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:45.224 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:16:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:16:45.226 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:16:45.226 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:16:45.226 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:16:45.226 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:16:45.226 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:45.227 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:45.227 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:16:45.227 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:16:45.227 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:16:45.227 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:16:45.228 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:16:45.228 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:16:45.228 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:16:45.228 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:45.228 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:16:45.229 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:45.229 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:45.229 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:45.229 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:45.229 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:45.229 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:45.230 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:45.230 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:45.230 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:45.230 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:45.231 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.231 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:16:45.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.231 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:16:45.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.232 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:45.232 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:45.232 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:45.232 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:45.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.233 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:45.233 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:16:45.234 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.235 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:45.235 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:16:45.235 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.235 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.236 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.236 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:16:45.237 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:16:45.237 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:45.237 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:45.238 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:45.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.238 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:45.238 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:45.239 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:45.239 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:45.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.239 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:45.239 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:16:45.240 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.240 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:45.240 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:16:45.241 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.241 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.241 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:16:45.242 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:16:45.242 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:45.242 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:45.243 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:45.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.244 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:45.244 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:45.244 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:45.245 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:45.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.245 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:45.246 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:16:45.246 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.247 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:45.247 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:16:45.247 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.247 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.248 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:16:45.248 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:16:45.248 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:45.248 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:45.249 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:45.249 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.249 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.249 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:16:45.250 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:16:45.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.250 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.250 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.250 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:45.250 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.251 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.251 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:16:45.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.251 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.251 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.252 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:16:45.252 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.252 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.252 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:16:45.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.252 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.253 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.253 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:16:45.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.253 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.253 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:16:45.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.254 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.254 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.254 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:45.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.254 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.255 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:16:45.255 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:16:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f7f4e640 10 librados: watch_flush enter 2023-10-30T12:16:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f7f4e640 10 librados: watch_flush exit 2023-10-30T12:16:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f7f4e640 20 client.214143.objecter shutdown clearing up homeless session... 2023-10-30T12:16:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.130+0000 7f66f7f4e640 10 client.214143.objecter successfully canceled tick 2023-10-30T12:16:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66dc0392f0 msgr2=0x7f66dc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f66dc0392f0 0x7f66dc03b7b0 secure :-1 s=READY pgs=4228 cs=0 l=1 rev1=1 crypto rx=0x7f66e4036000 tx=0x7f66e4008030 comp rx=0 tx=0).stop 2023-10-30T12:16:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 10 monclient: shutdown 2023-10-30T12:16:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66f0138b60 msgr2=0x7f66f0136e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 --2- 172.21.15.50:0/2768431355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66f0138b60 0x7f66f0136e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c700 tx=0x7f66e000cbc0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f64eb640 1 -- 172.21.15.50:0/2768431355 reap_dead start 2023-10-30T12:16:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f64eb640 1 -- 172.21.15.50:0/2768431355 reap_dead start 2023-10-30T12:16:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 shutdown_connections 2023-10-30T12:16:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 >> 172.21.15.50:0/2768431355 conn(0x7f66f00a4800 msgr2=0x7f66f013a170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 shutdown_connections 2023-10-30T12:16:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 -- 172.21.15.50:0/2768431355 wait complete. 2023-10-30T12:16:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.131+0000 7f66f7f4e640 1 librados: shutdown 2023-10-30T12:16:45.260 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:16:45.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:16:45.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:16:45.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.369+0000 7fcc8b9c5640 10 monclient: get_monmap_and_config 2023-10-30T12:16:45.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.369+0000 7fcc8b9c5640 10 monclient: build_initial_monmap 2023-10-30T12:16:45.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.369+0000 7fcc8b9c5640 10 monclient: monmap: 2023-10-30T12:16:45.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:45.372 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.372 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:45.370229+0000 2023-10-30T12:16:45.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:45.370229+0000 2023-10-30T12:16:45.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:45.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:45.373 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:45.374 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:45.374 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:45.374 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:45.375 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.369+0000 7fcc8b9c5640 1 Processor -- start 2023-10-30T12:16:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.369+0000 7fcc8b9c5640 1 -- start start 2023-10-30T12:16:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.369+0000 7fcc8b9c5640 10 monclient: init 2023-10-30T12:16:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:16:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc840a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:45.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcc840a2450 0x7fcc8413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient(hunting): picked mon.noname-c con 0x7fcc840a2450 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:45.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc8413af40 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:45.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc840a4190 con 0x7fcc8413af40 2023-10-30T12:16:45.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8409a2e0 con 0x7fcc840a2ae0 2023-10-30T12:16:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840a1630 con 0x7fcc840a2450 2023-10-30T12:16:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc8b9c5640 10 monclient(hunting): authenticate will time out at 1850.553433s 2023-10-30T12:16:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc89f62640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcc8413af40 0x7fcc8413d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50242/0 (socket says 172.21.15.50:50242) 2023-10-30T12:16:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc89f62640 1 -- 172.21.15.50:0/4209239135 learned_addr learned my addr 172.21.15.50:0/4209239135 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:45.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796766849 0 0) 0x7fcc74001490 con 0x7fcc8413af40 2023-10-30T12:16:45.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 10 monclient(hunting): _negotiate 2023-10-30T12:16:45.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 1 --2- 172.21.15.50:0/4209239135 >> v2:172.21.15.118:3300/0 conn(0x7fcc840a2450 0x7fcc8413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 10 monclient(hunting): my global_id is 214146 2023-10-30T12:16:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6c003520 con 0x7fcc8413af40 2023-10-30T12:16:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 10 monclient(hunting): get_auth_request con 0x7fcc840a2450 auth_method 0 2023-10-30T12:16:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1545788044 0 0) 0x7fcc6c003520 con 0x7fcc8413af40 2023-10-30T12:16:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 >> v2:172.21.15.118:3300/0 conn(0x7fcc840a2450 msgr2=0x7fcc8413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.370+0000 7fcc83fff640 1 --2- 172.21.15.50:0/4209239135 >> v2:172.21.15.118:3300/0 conn(0x7fcc840a2450 0x7fcc8413aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 >> v1:172.21.15.118:6789/0 conn(0x7fcc840a2ae0 legacy=0x7fcc840a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: _finish_hunting 0 2023-10-30T12:16:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:16:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:16:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8409a8a0 con 0x7fcc8413af40 2023-10-30T12:16:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: _finish_auth 0 2023-10-30T12:16:45.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: _check_auth_tickets 2023-10-30T12:16:45.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:45.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc8b9c5640 5 monclient: authenticate success, global_id 214146 2023-10-30T12:16:45.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fcc74001b80 con 0x7fcc8413af40 2023-10-30T12:16:45.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc8b9c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:45.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:16:45.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: dump: 2023-10-30T12:16:45.401 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.401 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.401 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.401 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.402 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.402 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:16:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 >> v1:172.21.15.50:6789/0 conn(0x7fcc8413af40 legacy=0x7fcc8413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:16:45.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c003970 0x7fcc6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): picked mon.b con 0x7fcc6c003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:45.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c0047c0 0x7fcc6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): picked mon.a con 0x7fcc6c0047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:45.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc6c0052a0 0x7fcc6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): picked mon.c con 0x7fcc6c0052a0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:45.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840a1630 con 0x7fcc6c0047c0 2023-10-30T12:16:45.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c00c170 con 0x7fcc6c003970 2023-10-30T12:16:45.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c00c330 con 0x7fcc6c0052a0 2023-10-30T12:16:45.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:45.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc83fff640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:16:45.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc8b9c5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:45.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c0047c0 0x7fcc6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc6c0052a0 0x7fcc6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): get_auth_request con 0x7fcc6c0047c0 auth_method 0 2023-10-30T12:16:45.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:45.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 10 monclient(hunting): get_auth_request con 0x7fcc6c0052a0 auth_method 0 2023-10-30T12:16:45.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c003970 0x7fcc6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:45.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 10 monclient(hunting): get_auth_request con 0x7fcc6c003970 auth_method 0 2023-10-30T12:16:45.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:45.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:16:45.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient(hunting): handle_auth_done global_id 214146 payload 293 2023-10-30T12:16:45.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 1 -- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc6c0052a0 msgr2=0x7fcc6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:45.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc6c0052a0 0x7fcc6c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 1 -- 172.21.15.50:0/4209239135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c003970 msgr2=0x7fcc6c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:45.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c003970 0x7fcc6c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient: _finish_hunting 0 2023-10-30T12:16:45.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 1 monclient: found mon.a 2023-10-30T12:16:45.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:45.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 1 -- 172.21.15.50:0/4209239135 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c003520 con 0x7fcc6c0047c0 2023-10-30T12:16:45.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient: _finish_auth 0 2023-10-30T12:16:45.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 10 monclient: _check_auth_tickets 2023-10-30T12:16:45.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc88f60640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.371+0000 7fcc89f62640 1 -- 172.21.15.50:0/4209239135 reap_dead start 2023-10-30T12:16:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc88f60640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c0047c0 0x7fcc6c004c40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009a90 tx=0x7fcc6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7a60753abd15dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7802c680 con 0x7fcc6c0047c0 2023-10-30T12:16:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 10 monclient: dump: 2023-10-30T12:16:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.421 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.422 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7802c7e0 con 0x7fcc6c0047c0 2023-10-30T12:16:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc8b9c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:45.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 1 -- 172.21.15.50:0/4209239135 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7802cb70 con 0x7fcc6c0047c0 2023-10-30T12:16:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc8b9c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc8b9c5640 4 set_mon_vals no callback set 2023-10-30T12:16:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc83fff640 10 monclient: dump: 2023-10-30T12:16:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.424 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.425 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc8b9c5640 10 monclient: shutdown 2023-10-30T12:16:45.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc8b9c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:45.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c0047c0 msgr2=0x7fcc6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.372+0000 7fcc8b9c5640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c0047c0 0x7fcc6c004c40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009a90 tx=0x7fcc6c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/4209239135 shutdown_connections 2023-10-30T12:16:45.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 1 --2- 172.21.15.50:0/4209239135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c0047c0 0x7fcc6c004c40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/4209239135 >> 172.21.15.50:0/4209239135 conn(0x7fcc840a80b0 msgr2=0x7fcc840a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:45.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/4209239135 shutdown_connections 2023-10-30T12:16:45.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/4209239135 wait complete. 2023-10-30T12:16:45.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 10 monclient: build_initial_monmap 2023-10-30T12:16:45.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 10 monclient: monmap: 2023-10-30T12:16:45.428 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:45.428 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:45.428 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:45.428 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:45.428 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:45.429 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.429 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:45.429 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:45.429 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:45.430 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 1 librados: starting msgr at 2023-10-30T12:16:45.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.373+0000 7fcc8b9c5640 1 librados: starting objecter 2023-10-30T12:16:45.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 Processor -- start 2023-10-30T12:16:45.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 -- start start 2023-10-30T12:16:45.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 librados: setting wanted keys 2023-10-30T12:16:45.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 librados: calling monclient init 2023-10-30T12:16:45.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient: init 2023-10-30T12:16:45.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:45.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:16:45.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c0047c0 0x7fcc841ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): picked mon.noname-b con 0x7fcc6c0047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:45.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc84132430 0x7fcc841328b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): picked mon.noname-c con 0x7fcc84132430 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:45.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc8409db70 0x7fcc8409e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): picked mon.noname-a con 0x7fcc8409db70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:45.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8411e840 con 0x7fcc8409db70 2023-10-30T12:16:45.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8411e560 con 0x7fcc6c0047c0 2023-10-30T12:16:45.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840a01d0 con 0x7fcc84132430 2023-10-30T12:16:45.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:45.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc8b9c5640 10 monclient(hunting): authenticate will time out at 1850.557595s 2023-10-30T12:16:45.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc8409db70 0x7fcc8409e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc84132430 0x7fcc841328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc8409db70 0x7fcc8409e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43168/0 (socket says 172.21.15.50:43168) 2023-10-30T12:16:45.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc84132430 0x7fcc841328b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53744/0 (socket says 172.21.15.50:53744) 2023-10-30T12:16:45.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 1 -- 172.21.15.50:0/3239222916 learned_addr learned my addr 172.21.15.50:0/3239222916 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:45.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 10 monclient(hunting): get_auth_request con 0x7fcc8409db70 auth_method 0 2023-10-30T12:16:45.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): get_auth_request con 0x7fcc84132430 auth_method 0 2023-10-30T12:16:45.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c0047c0 0x7fcc841ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 10 monclient(hunting): get_auth_request con 0x7fcc6c0047c0 auth_method 0 2023-10-30T12:16:45.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 10 monclient(hunting): handle_auth_done global_id 115181 payload 824 2023-10-30T12:16:45.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.374+0000 7fcc88f60640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c0047c0 msgr2=0x7fcc841ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc6c0047c0 0x7fcc841ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc8409db70 msgr2=0x7fcc8409e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc8409db70 0x7fcc8409e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 10 monclient: _finish_hunting 0 2023-10-30T12:16:45.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 1 monclient: found mon.noname-c 2023-10-30T12:16:45.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:45.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:45.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 1 -- 172.21.15.50:0/3239222916 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc840a1790 con 0x7fcc84132430 2023-10-30T12:16:45.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc89f62640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc8409db70 0x7fcc8409e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:45.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc88f60640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc84132430 0x7fcc841328b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00c140 tx=0x7fcc78004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 client.?.objecter ms_handle_connect 0x7fcc84132430 2023-10-30T12:16:45.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:45.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78004520 con 0x7fcc84132430 2023-10-30T12:16:45.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:45.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: dump: 2023-10-30T12:16:45.448 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.448 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.448 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.448 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.449 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.449 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.449 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.449 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.449 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.450 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: _finish_auth 0 2023-10-30T12:16:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:16:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78003810 con 0x7fcc84132430 2023-10-30T12:16:45.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 5 monclient: authenticate success, global_id 115181 2023-10-30T12:16:45.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 10 monclient: _renew_subs 2023-10-30T12:16:45.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:45.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc84133570 con 0x7fcc84132430 2023-10-30T12:16:45.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:45.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78003ba0 con 0x7fcc84132430 2023-10-30T12:16:45.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:45.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc81ffb640 10 monclient: dump: 2023-10-30T12:16:45.452 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.453 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.453 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.453 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.453 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.453 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.454 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.454 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.454 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.454 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 10 client.115181.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:45.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc837fe640 4 set_mon_vals no callback set 2023-10-30T12:16:45.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 10 monclient: _renew_subs 2023-10-30T12:16:45.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:45.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8411e0a0 con 0x7fcc84132430 2023-10-30T12:16:45.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.375+0000 7fcc8b9c5640 1 librados: init done 2023-10-30T12:16:45.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc8b9c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:45.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcc78003d00 con 0x7fcc84132430 2023-10-30T12:16:45.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc8b9c5640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc8402c5b0 0x7fcc8409e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc8b9c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc84073ff0 con 0x7fcc8402c5b0 2023-10-30T12:16:45.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc81ffb640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc70039320 0x7fcc7003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(503..503 src has 417..503) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fcc7806cca0 con 0x7fcc84132430 2023-10-30T12:16:45.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc81ffb640 10 client.115181.objecter ms_dispatch 0x7fcc84000b90 osd_map(503..503 src has 417..503) v4 2023-10-30T12:16:45.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc81ffb640 3 client.115181.objecter handle_osd_map got epochs [503,503] > 0 2023-10-30T12:16:45.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc81ffb640 3 client.115181.objecter handle_osd_map decoding full epoch 503 2023-10-30T12:16:45.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc8402c5b0 0x7fcc8409e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 10 monclient: get_auth_request con 0x7fcc8402c5b0 auth_method 0 2023-10-30T12:16:45.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc81ffb640 20 client.115181.objecter dump_active .. 0 homeless 2023-10-30T12:16:45.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89f62640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc70039320 0x7fcc7003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.376+0000 7fcc89f62640 10 monclient: get_auth_request con 0x7fcc70039320 auth_method 0 2023-10-30T12:16:45.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc89761640 10 monclient(hunting): handle_auth_done global_id 204247 payload 824 2023-10-30T12:16:45.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc89761640 1 -- 172.21.15.50:0/3239222916 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc8409a5c0 con 0x7fcc8402c5b0 2023-10-30T12:16:45.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc89761640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc8402c5b0 0x7fcc8409e560 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcc8409f410 tx=0x7fcc7c012ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 client.115181.objecter ms_handle_connect 0x7fcc8402c5b0 2023-10-30T12:16:45.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 client.115181.objecter resend_mon_ops 2023-10-30T12:16:45.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 client.115181.objecter ms_handle_connect 0x7fcc70039320 2023-10-30T12:16:45.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc89f62640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc70039320 0x7fcc7003b7e0 secure :-1 s=READY pgs=4229 cs=0 l=1 rev1=1 crypto rx=0x7fcc74001710 tx=0x7fcc74006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c0053e0 con 0x7fcc8402c5b0 2023-10-30T12:16:45.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:16:45.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 monclient: dump: 2023-10-30T12:16:45.466 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.466 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.466 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.467 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.467 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.467 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.467 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.468 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fcc7c007030 con 0x7fcc8402c5b0 2023-10-30T12:16:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc8402c5b0 msgr2=0x7fcc8409e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc8402c5b0 0x7fcc8409e560 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcc8409f410 tx=0x7fcc7c012ac0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc8402c5b0 msgr2=0x7fcc8409e560 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.377+0000 7fcc81ffb640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc8402c5b0 0x7fcc8409e560 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcc8409f410 tx=0x7fcc7c012ac0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.383+0000 7fcc8b9c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:16:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc8b9c5640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc84060fc0 0x7fcc840613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc8b9c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc84099b40 con 0x7fcc84060fc0 2023-10-30T12:16:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc84060fc0 0x7fcc840613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient: get_auth_request con 0x7fcc84060fc0 auth_method 0 2023-10-30T12:16:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 10 monclient(hunting): handle_auth_done global_id 204250 payload 824 2023-10-30T12:16:45.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 1 -- 172.21.15.50:0/3239222916 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcc840a28a0 con 0x7fcc84060fc0 2023-10-30T12:16:45.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc89761640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc84060fc0 0x7fcc840613d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc8409f410 tx=0x7fcc7c001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc81ffb640 10 client.115181.objecter ms_handle_connect 0x7fcc84060fc0 2023-10-30T12:16:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc81ffb640 10 client.115181.objecter resend_mon_ops 2023-10-30T12:16:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.384+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c026070 con 0x7fcc84060fc0 2023-10-30T12:16:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:16:45.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 10 monclient: dump: 2023-10-30T12:16:45.475 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.475 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.475 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.477 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7fcc7c003800 con 0x7fcc84060fc0 2023-10-30T12:16:45.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:16:45.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:45.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc84060fc0 msgr2=0x7fcc840613d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc84060fc0 0x7fcc840613d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc8409f410 tx=0x7fcc7c001120 comp rx=0 tx=0).stop 2023-10-30T12:16:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc84060fc0 msgr2=0x7fcc840613d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc81ffb640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc84060fc0 0x7fcc840613d0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc8409f410 tx=0x7fcc7c001120 comp rx=0 tx=0).stop 2023-10-30T12:16:45.480 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:16:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:16:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 170, 2023-10-30T12:16:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:16:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:16:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:16:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:16:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:16:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:16:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:16:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:16:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:16:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:16:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:16:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:16:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:16:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:16:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:16:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:16:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:16:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:16:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:16:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:16:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:16:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:16:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:16:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:16:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:16:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:16:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:16:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:16:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:16:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:16:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:16:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:16:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:16:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:16:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:16:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:16:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:16:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:16:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:16:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:16:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:16:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:16:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:16:45.527 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:16:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 10 librados: watch_flush enter 2023-10-30T12:16:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 10 librados: watch_flush exit 2023-10-30T12:16:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 20 client.115181.objecter shutdown clearing up homeless session... 2023-10-30T12:16:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 10 client.115181.objecter successfully canceled tick 2023-10-30T12:16:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc70039320 msgr2=0x7fcc7003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc70039320 0x7fcc7003b7e0 secure :-1 s=READY pgs=4229 cs=0 l=1 rev1=1 crypto rx=0x7fcc74001710 tx=0x7fcc74006040 comp rx=0 tx=0).stop 2023-10-30T12:16:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 10 monclient: shutdown 2023-10-30T12:16:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc84132430 msgr2=0x7fcc841328b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc84132430 0x7fcc841328b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00c140 tx=0x7fcc78004730 comp rx=0 tx=0).stop 2023-10-30T12:16:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc89f62640 1 -- 172.21.15.50:0/3239222916 reap_dead start 2023-10-30T12:16:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 shutdown_connections 2023-10-30T12:16:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 --2- 172.21.15.50:0/3239222916 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc84132430 0x7fcc841328b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 >> 172.21.15.50:0/3239222916 conn(0x7fcc840a80b0 msgr2=0x7fcc84134ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:45.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 shutdown_connections 2023-10-30T12:16:45.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 -- 172.21.15.50:0/3239222916 wait complete. 2023-10-30T12:16:45.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.385+0000 7fcc8b9c5640 1 librados: shutdown 2023-10-30T12:16:45.534 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:16:45.534 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:16:45.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:16:45.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.638+0000 7fb2c3abb640 10 monclient: get_monmap_and_config 2023-10-30T12:16:45.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.638+0000 7fb2c3abb640 10 monclient: build_initial_monmap 2023-10-30T12:16:45.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.638+0000 7fb2c3abb640 10 monclient: monmap: 2023-10-30T12:16:45.641 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:45.641 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.642 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:45.639598+0000 2023-10-30T12:16:45.642 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:45.639598+0000 2023-10-30T12:16:45.642 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:45.643 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:45.643 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:45.644 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:45.644 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:45.645 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:45.645 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 1 Processor -- start 2023-10-30T12:16:45.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 1 -- start start 2023-10-30T12:16:45.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient: init 2023-10-30T12:16:45.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:45.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T12:16:45.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb2bc098520 0x7fb2bc09bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient(hunting): picked mon.noname-c con 0x7fb2bc098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:45.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2bc098900 0x7fb2bc098d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient(hunting): picked mon.noname-a con 0x7fb2bc098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:45.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2bc1326d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:45.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc099930 con 0x7fb2bc1326d0 2023-10-30T12:16:45.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc09aca0 con 0x7fb2bc098900 2023-10-30T12:16:45.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc09af80 con 0x7fb2bc098520 2023-10-30T12:16:45.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:45.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c3abb640 10 monclient(hunting): authenticate will time out at 1850.822735s 2023-10-30T12:16:45.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c1056640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2bc098900 0x7fb2bc098d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c1056640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2bc098900 0x7fb2bc098d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43172/0 (socket says 172.21.15.50:43172) 2023-10-30T12:16:45.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c1056640 1 -- 172.21.15.50:0/733883038 learned_addr learned my addr 172.21.15.50:0/733883038 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:45.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c1056640 10 monclient(hunting): get_auth_request con 0x7fb2bc098900 auth_method 0 2023-10-30T12:16:45.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c1056640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c1056640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.639+0000 7fb2c1056640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1857640 1 --2- 172.21.15.50:0/733883038 >> v2:172.21.15.118:3300/0 conn(0x7fb2bc098520 0x7fb2bc09bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1857640 10 monclient(hunting): get_auth_request con 0x7fb2bc098520 auth_method 0 2023-10-30T12:16:45.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1857640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1857640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1857640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 10 monclient(hunting): handle_auth_done global_id 214152 payload 274 2023-10-30T12:16:45.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 1 -- 172.21.15.50:0/733883038 >> v2:172.21.15.118:3300/0 conn(0x7fb2bc098520 msgr2=0x7fb2bc09bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:45.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 1 --2- 172.21.15.50:0/733883038 >> v2:172.21.15.118:3300/0 conn(0x7fb2bc098520 0x7fb2bc09bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 1 -- 172.21.15.50:0/733883038 >> v1:172.21.15.118:6789/0 conn(0x7fb2bc1326d0 legacy=0x7fb2bc132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:45.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 10 monclient: _finish_hunting 0 2023-10-30T12:16:45.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 1 monclient: found mon.noname-a 2023-10-30T12:16:45.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:45.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 1 -- 172.21.15.50:0/733883038 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc09b260 con 0x7fb2bc098900 2023-10-30T12:16:45.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.640+0000 7fb2c1056640 1 --2- 172.21.15.50:0/733883038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098900 0x7fb2bc098d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc06b170 tx=0x7fb2ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b1840191e0ba10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 1 -- 172.21.15.50:0/733883038 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0317e0 con 0x7fb2bc098900 2023-10-30T12:16:45.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:45.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: dump: 2023-10-30T12:16:45.671 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.671 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.673 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.673 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: _finish_auth 0 2023-10-30T12:16:45.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: _check_auth_tickets 2023-10-30T12:16:45.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 1 -- 172.21.15.50:0/733883038 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac031940 con 0x7fb2bc098900 2023-10-30T12:16:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 1 -- 172.21.15.50:0/733883038 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac031cd0 con 0x7fb2bc098900 2023-10-30T12:16:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 5 monclient: authenticate success, global_id 214152 2023-10-30T12:16:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:45.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 4 set_mon_vals no callback set 2023-10-30T12:16:45.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:45.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c0855640 10 monclient: dump: 2023-10-30T12:16:45.675 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.676 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.676 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.676 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.676 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.676 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.677 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.677 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.677 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.677 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 10 monclient: shutdown 2023-10-30T12:16:45.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:45.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 -- 172.21.15.50:0/733883038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098900 msgr2=0x7fb2bc098d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/733883038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098900 0x7fb2bc098d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc06b170 tx=0x7fb2ac02b880 comp rx=0 tx=0).stop 2023-10-30T12:16:45.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 -- 172.21.15.50:0/733883038 shutdown_connections 2023-10-30T12:16:45.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/733883038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098900 0x7fb2bc098d70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/733883038 >> v2:172.21.15.118:3300/0 conn(0x7fb2bc098520 0x7fb2bc09bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 -- 172.21.15.50:0/733883038 >> 172.21.15.50:0/733883038 conn(0x7fb2bc0a05c0 msgr2=0x7fb2bc09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 -- 172.21.15.50:0/733883038 shutdown_connections 2023-10-30T12:16:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.641+0000 7fb2c3abb640 1 -- 172.21.15.50:0/733883038 wait complete. 2023-10-30T12:16:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 10 monclient: build_initial_monmap 2023-10-30T12:16:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 10 monclient: monmap: 2023-10-30T12:16:45.680 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:45.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:45.680 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:45.680 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:45.680 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:45.681 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:45.681 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:45.681 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:45.681 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 1 librados: starting msgr at 2023-10-30T12:16:45.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 1 librados: starting objecter 2023-10-30T12:16:45.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 1 Processor -- start 2023-10-30T12:16:45.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 1 -- start start 2023-10-30T12:16:45.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 1 librados: setting wanted keys 2023-10-30T12:16:45.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 1 librados: calling monclient init 2023-10-30T12:16:45.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 10 monclient: init 2023-10-30T12:16:45.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:45.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:16:45.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.642+0000 7fb2c3abb640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098520 0x7fb2bc131150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): picked mon.noname-a con 0x7fb2bc098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:45.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2bc098900 0x7fb2bc12ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): picked mon.noname-b con 0x7fb2bc098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:45.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2bc1326d0 0x7fb2bc12cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): picked mon.noname-c con 0x7fb2bc1326d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:45.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc130400 con 0x7fb2bc098520 2023-10-30T12:16:45.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc12edc0 con 0x7fb2bc098900 2023-10-30T12:16:45.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:45.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc1309c0 con 0x7fb2bc1326d0 2023-10-30T12:16:45.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:45.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient(hunting): authenticate will time out at 1850.826066s 2023-10-30T12:16:45.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098520 0x7fb2bc131150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098520 0x7fb2bc131150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43180/0 (socket says 172.21.15.50:43180) 2023-10-30T12:16:45.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 -- 172.21.15.50:0/246195792 learned_addr learned my addr 172.21.15.50:0/246195792 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:45.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): get_auth_request con 0x7fb2bc098520 auth_method 0 2023-10-30T12:16:45.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2bc1326d0 0x7fb2bc12cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 10 monclient(hunting): get_auth_request con 0x7fb2bc1326d0 auth_method 0 2023-10-30T12:16:45.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2bc098900 0x7fb2bc12ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 10 monclient(hunting): get_auth_request con 0x7fb2bc098900 auth_method 0 2023-10-30T12:16:45.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:45.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:45.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:45.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:45.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:45.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1056640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:45.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient(hunting): handle_auth_done global_id 214155 payload 824 2023-10-30T12:16:45.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 -- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2bc1326d0 msgr2=0x7fb2bc12cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2bc1326d0 0x7fb2bc12cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 -- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2bc098900 msgr2=0x7fb2bc12ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2bc098900 0x7fb2bc12ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient: _finish_hunting 0 2023-10-30T12:16:45.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 monclient: found mon.noname-a 2023-10-30T12:16:45.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:45.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 -- 172.21.15.50:0/246195792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac0094d0 con 0x7fb2bc098520 2023-10-30T12:16:45.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c2058640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2bc1326d0 0x7fb2bc12cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:45.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c1857640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098520 0x7fb2bc131150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8007ef0 tx=0x7fb2b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 client.?.objecter ms_handle_connect 0x7fb2bc098520 2023-10-30T12:16:45.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:45.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 1 -- 172.21.15.50:0/246195792 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8014620 con 0x7fb2bc098520 2023-10-30T12:16:45.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:45.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: dump: 2023-10-30T12:16:45.694 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.694 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.695 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.695 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.695 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.695 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.695 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.696 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.696 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.696 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: _finish_auth 0 2023-10-30T12:16:45.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:16:45.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:45.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 1 -- 172.21.15.50:0/246195792 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b800ee90 con 0x7fb2bc098520 2023-10-30T12:16:45.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:45.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 1 -- 172.21.15.50:0/246195792 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80149c0 con 0x7fb2bc098520 2023-10-30T12:16:45.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 5 monclient: authenticate success, global_id 214155 2023-10-30T12:16:45.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:45.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:45.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b27fc640 10 monclient: dump: 2023-10-30T12:16:45.698 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:45.698 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:45.698 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.699 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:45.699 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:45.699 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:45.699 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:45.699 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:45.699 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:45.700 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:45.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2b3fff640 4 set_mon_vals no callback set 2023-10-30T12:16:45.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient: _renew_subs 2023-10-30T12:16:45.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc071d50 con 0x7fb2bc098520 2023-10-30T12:16:45.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 client.214155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:45.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient: _renew_subs 2023-10-30T12:16:45.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc130120 con 0x7fb2bc098520 2023-10-30T12:16:45.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.643+0000 7fb2c3abb640 1 librados: init done 2023-10-30T12:16:45.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2b27fc640 1 -- 172.21.15.50:0/246195792 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb2b8014b20 con 0x7fb2bc098520 2023-10-30T12:16:45.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2c3abb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:45.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2b27fc640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb2a00392f0 0x7fb2a003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:45.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2c3abb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:45.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2c3abb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc04e160 con 0x7fb2bc098520 2023-10-30T12:16:45.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2b27fc640 1 -- 172.21.15.50:0/246195792 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(503..503 src has 417..503) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fb2b804b670 con 0x7fb2bc098520 2023-10-30T12:16:45.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2b27fc640 10 client.214155.objecter ms_dispatch 0x7fb2bc000b90 osd_map(503..503 src has 417..503) v4 2023-10-30T12:16:45.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2b27fc640 3 client.214155.objecter handle_osd_map got epochs [503,503] > 0 2023-10-30T12:16:45.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2b27fc640 3 client.214155.objecter handle_osd_map decoding full epoch 503 2023-10-30T12:16:45.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2b27fc640 20 client.214155.objecter dump_active .. 0 homeless 2023-10-30T12:16:45.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.644+0000 7fb2c1056640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb2a00392f0 0x7fb2a003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:45.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.645+0000 7fb2c1056640 10 monclient: get_auth_request con 0x7fb2a00392f0 auth_method 0 2023-10-30T12:16:45.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.645+0000 7fb2b27fc640 10 client.214155.objecter ms_handle_connect 0x7fb2a00392f0 2023-10-30T12:16:45.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.645+0000 7fb2c1056640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb2a00392f0 0x7fb2a003b7b0 secure :-1 s=READY pgs=4230 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac036000 tx=0x7fb2ac03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:45.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.649+0000 7fb2b27fc640 1 -- 172.21.15.50:0/246195792 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb2b80122e0 con 0x7fb2bc098520 2023-10-30T12:16:45.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.649+0000 7fb2b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:45.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.649+0000 7fb2b27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.892+0000 7fb2c3abb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:16:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.892+0000 7fb2c3abb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:16:45.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.892+0000 7fb2c3abb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.892+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb2bc0a1dc0 con 0x7fb2bc098520 2023-10-30T12:16:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.893+0000 7fb2b27fc640 1 -- 172.21.15.50:0/246195792 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb2b803d080 con 0x7fb2bc098520 2023-10-30T12:16:45.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.893+0000 7fb2b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:16:45.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.893+0000 7fb2b27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:45.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.894+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.894+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=9,kv=8,logm=50} crc {auth=755672380,config=3140514009,health=1769428655,kv=474600764,logm=1868007715}) 2023-10-30T12:16:45.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.894+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=10,kv=9,logm=54} crc {auth=3439823833,config=3140514009,health=3788137532,kv=2905431493,logm=3717420638}) 2023-10-30T12:16:45.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.894+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.894+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=9,kv=8,logm=50} crc {auth=755672380,config=3140514009,health=1769428655,kv=474600764,logm=1868007715}) 2023-10-30T12:16:45.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.894+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=1674531716,config=3140514009,health=149930001,kv=2905431493,logm=2225181631}) 2023-10-30T12:16:45.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.895+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.895+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125850353}) 2023-10-30T12:16:45.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.895+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780070396}) 2023-10-30T12:16:45.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.895+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.895+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125850353}) 2023-10-30T12:16:45.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.895+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847294419}) 2023-10-30T12:16:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.895+0000 7fb2c3abb640 10 librados: watch_flush enter 2023-10-30T12:16:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.895+0000 7fb2c3abb640 10 librados: watch_flush exit 2023-10-30T12:16:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.895+0000 7fb2c3abb640 20 client.214155.objecter shutdown clearing up homeless session... 2023-10-30T12:16:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.895+0000 7fb2c3abb640 10 client.214155.objecter successfully canceled tick 2023-10-30T12:16:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb2a00392f0 msgr2=0x7fb2a003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb2a00392f0 0x7fb2a003b7b0 secure :-1 s=READY pgs=4230 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac036000 tx=0x7fb2ac03d040 comp rx=0 tx=0).stop 2023-10-30T12:16:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 10 monclient: shutdown 2023-10-30T12:16:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098520 msgr2=0x7fb2bc131150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098520 0x7fb2bc131150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8007ef0 tx=0x7fb2b800e4b0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 shutdown_connections 2023-10-30T12:16:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb2a00392f0 0x7fb2a003b7b0 unknown :-1 s=CLOSED pgs=4230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2bc1326d0 0x7fb2bc12cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2bc098900 0x7fb2bc12ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 --2- 172.21.15.50:0/246195792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2bc098520 0x7fb2bc131150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:45.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 >> 172.21.15.50:0/246195792 conn(0x7fb2bc0a05c0 msgr2=0x7fb2bc070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 shutdown_connections 2023-10-30T12:16:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 -- 172.21.15.50:0/246195792 wait complete. 2023-10-30T12:16:45.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:45.896+0000 7fb2c3abb640 1 librados: shutdown 2023-10-30T12:16:45.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.896+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.896+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007914984}) 2023-10-30T12:16:45.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.896+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852390995}) 2023-10-30T12:16:45.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.896+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.896+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007914984}) 2023-10-30T12:16:45.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.896+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3475713854}) 2023-10-30T12:16:45.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.898+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.898+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951968092}) 2023-10-30T12:16:45.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.898+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2048773406}) 2023-10-30T12:16:45.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.898+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.898+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951968092}) 2023-10-30T12:16:45.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.898+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864757673}) 2023-10-30T12:16:45.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.899+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.899+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026059319}) 2023-10-30T12:16:45.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.899+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4172824926}) 2023-10-30T12:16:45.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.899+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.899+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026059319}) 2023-10-30T12:16:45.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.899+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877760823}) 2023-10-30T12:16:45.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.900+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.900+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376418388}) 2023-10-30T12:16:45.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.900+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230468644}) 2023-10-30T12:16:45.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.900+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.900+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376418388}) 2023-10-30T12:16:45.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.900+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656804227}) 2023-10-30T12:16:45.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.901+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.901+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840175857}) 2023-10-30T12:16:45.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.901+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=620126616}) 2023-10-30T12:16:45.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.901+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.901+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840175857}) 2023-10-30T12:16:45.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.901+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2391190462}) 2023-10-30T12:16:45.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2109478125}) 2023-10-30T12:16:45.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938626889}) 2023-10-30T12:16:45.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2109478125}) 2023-10-30T12:16:45.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183569906}) 2023-10-30T12:16:45.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451357529}) 2023-10-30T12:16:45.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2285157312}) 2023-10-30T12:16:45.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451357529}) 2023-10-30T12:16:45.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.902+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118479473}) 2023-10-30T12:16:45.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.903+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.903+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966436846}) 2023-10-30T12:16:45.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.903+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1113706744}) 2023-10-30T12:16:45.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.903+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.903+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966436846}) 2023-10-30T12:16:45.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.903+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753794549}) 2023-10-30T12:16:45.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.904+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.904+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136874823}) 2023-10-30T12:16:45.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.904+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931127332}) 2023-10-30T12:16:45.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.904+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.904+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136874823}) 2023-10-30T12:16:45.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.904+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243553311}) 2023-10-30T12:16:45.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.905+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.905+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229131673}) 2023-10-30T12:16:45.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.905+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157564493}) 2023-10-30T12:16:45.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.905+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.905+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229131673}) 2023-10-30T12:16:45.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.905+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2044294210}) 2023-10-30T12:16:45.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.906+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.906+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543399922}) 2023-10-30T12:16:45.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.906+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=337298099}) 2023-10-30T12:16:45.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.906+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.906+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543399922}) 2023-10-30T12:16:45.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.906+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776463713}) 2023-10-30T12:16:45.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001673664}) 2023-10-30T12:16:45.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4269689174}) 2023-10-30T12:16:45.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001673664}) 2023-10-30T12:16:45.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=209992024}) 2023-10-30T12:16:45.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547769180}) 2023-10-30T12:16:45.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122657182}) 2023-10-30T12:16:45.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547769180}) 2023-10-30T12:16:45.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.907+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227826377}) 2023-10-30T12:16:45.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:16:45.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.908+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.908+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391346781}) 2023-10-30T12:16:45.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.908+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=158402123}) 2023-10-30T12:16:45.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.908+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.908+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391346781}) 2023-10-30T12:16:45.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.908+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104686848}) 2023-10-30T12:16:45.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.909+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.909+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644837656}) 2023-10-30T12:16:45.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.909+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=686825431}) 2023-10-30T12:16:45.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.909+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.909+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644837656}) 2023-10-30T12:16:45.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.909+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953950597}) 2023-10-30T12:16:45.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.910+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.910+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736078571}) 2023-10-30T12:16:45.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.910+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4081042894}) 2023-10-30T12:16:45.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.910+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.910+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736078571}) 2023-10-30T12:16:45.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.910+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=937639413}) 2023-10-30T12:16:45.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.911+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.911+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252509518}) 2023-10-30T12:16:45.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.911+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1998072580}) 2023-10-30T12:16:45.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.911+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.911+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252509518}) 2023-10-30T12:16:45.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.911+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760302596}) 2023-10-30T12:16:45.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.912+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.912+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2503862189}) 2023-10-30T12:16:45.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.912+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=898936751}) 2023-10-30T12:16:45.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.912+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.912+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2503862189}) 2023-10-30T12:16:45.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.912+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3839505618}) 2023-10-30T12:16:45.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225839280}) 2023-10-30T12:16:45.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227885290}) 2023-10-30T12:16:45.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225839280}) 2023-10-30T12:16:45.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943746758}) 2023-10-30T12:16:45.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172853034}) 2023-10-30T12:16:45.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1429663192}) 2023-10-30T12:16:45.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172853034}) 2023-10-30T12:16:45.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.913+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918105308}) 2023-10-30T12:16:45.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.914+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.914+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213361871}) 2023-10-30T12:16:45.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.914+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106693933}) 2023-10-30T12:16:45.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.914+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.914+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213361871}) 2023-10-30T12:16:45.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.914+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716606779}) 2023-10-30T12:16:45.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.915+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.915+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223075522}) 2023-10-30T12:16:45.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.915+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899739994}) 2023-10-30T12:16:45.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.915+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.915+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223075522}) 2023-10-30T12:16:45.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.915+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094820879}) 2023-10-30T12:16:45.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.916+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.916+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131061970}) 2023-10-30T12:16:45.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.916+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=444454571}) 2023-10-30T12:16:45.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.916+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.916+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131061970}) 2023-10-30T12:16:45.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.916+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660329553}) 2023-10-30T12:16:45.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974197613}) 2023-10-30T12:16:45.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4042577765}) 2023-10-30T12:16:45.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974197613}) 2023-10-30T12:16:45.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316090577}) 2023-10-30T12:16:45.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053379923}) 2023-10-30T12:16:45.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156693374}) 2023-10-30T12:16:45.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053379923}) 2023-10-30T12:16:45.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.917+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664731195}) 2023-10-30T12:16:45.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.918+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.918+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594012124}) 2023-10-30T12:16:45.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.918+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1988201994}) 2023-10-30T12:16:45.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.918+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.918+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594012124}) 2023-10-30T12:16:45.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.918+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795732485}) 2023-10-30T12:16:45.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.919+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.919+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203182588}) 2023-10-30T12:16:45.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.919+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663688636}) 2023-10-30T12:16:45.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.919+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.919+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203182588}) 2023-10-30T12:16:45.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.919+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127147869}) 2023-10-30T12:16:45.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613827785}) 2023-10-30T12:16:45.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=816436659}) 2023-10-30T12:16:45.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613827785}) 2023-10-30T12:16:45.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1146608045}) 2023-10-30T12:16:45.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.920+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102357486}) 2023-10-30T12:16:45.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2631502359}) 2023-10-30T12:16:45.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102357486}) 2023-10-30T12:16:45.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864646941}) 2023-10-30T12:16:45.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270374153}) 2023-10-30T12:16:45.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816195399}) 2023-10-30T12:16:45.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270374153}) 2023-10-30T12:16:45.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.921+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82032090}) 2023-10-30T12:16:45.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.922+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.922+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587317819}) 2023-10-30T12:16:45.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.922+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=876255860}) 2023-10-30T12:16:45.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.922+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.922+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587317819}) 2023-10-30T12:16:45.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.922+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1912500164}) 2023-10-30T12:16:45.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.923+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.923+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246497087}) 2023-10-30T12:16:45.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.923+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397166090}) 2023-10-30T12:16:45.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.923+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.923+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246497087}) 2023-10-30T12:16:45.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.923+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456127219}) 2023-10-30T12:16:45.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.923+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907950035}) 2023-10-30T12:16:45.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505511332}) 2023-10-30T12:16:45.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907950035}) 2023-10-30T12:16:45.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192189465}) 2023-10-30T12:16:45.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581708746}) 2023-10-30T12:16:45.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=463019854}) 2023-10-30T12:16:45.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581708746}) 2023-10-30T12:16:45.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.924+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246490671}) 2023-10-30T12:16:45.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.925+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.925+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285851621}) 2023-10-30T12:16:45.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.925+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1109496773}) 2023-10-30T12:16:45.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.925+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.925+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285851621}) 2023-10-30T12:16:45.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.925+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2167374521}) 2023-10-30T12:16:45.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.926+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.926+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869582032}) 2023-10-30T12:16:45.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.926+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2411237426}) 2023-10-30T12:16:45.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.926+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.926+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869582032}) 2023-10-30T12:16:45.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.926+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2044190296}) 2023-10-30T12:16:45.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.927+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.927+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392078534}) 2023-10-30T12:16:45.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.927+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754529618}) 2023-10-30T12:16:45.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.927+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.927+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392078534}) 2023-10-30T12:16:45.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.927+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2338670692}) 2023-10-30T12:16:45.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3615495118}) 2023-10-30T12:16:45.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4247461769}) 2023-10-30T12:16:45.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3615495118}) 2023-10-30T12:16:45.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3372688339}) 2023-10-30T12:16:45.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49176124}) 2023-10-30T12:16:45.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872325562}) 2023-10-30T12:16:45.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49176124}) 2023-10-30T12:16:45.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.928+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705244347}) 2023-10-30T12:16:45.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.929+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.929+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142751835}) 2023-10-30T12:16:45.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.929+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=807670907}) 2023-10-30T12:16:45.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.929+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.929+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142751835}) 2023-10-30T12:16:45.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.929+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3279659033}) 2023-10-30T12:16:45.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.930+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.930+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658397082}) 2023-10-30T12:16:45.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.930+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4274588336}) 2023-10-30T12:16:45.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.930+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.930+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658397082}) 2023-10-30T12:16:45.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.930+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371127165}) 2023-10-30T12:16:45.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.931+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.931+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801294224}) 2023-10-30T12:16:45.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.931+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=590410155}) 2023-10-30T12:16:45.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.931+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.931+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801294224}) 2023-10-30T12:16:45.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.931+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992214547}) 2023-10-30T12:16:45.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.932+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.932+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496953291}) 2023-10-30T12:16:45.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.932+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=822338479}) 2023-10-30T12:16:45.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.932+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.932+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496953291}) 2023-10-30T12:16:45.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.932+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4113495046}) 2023-10-30T12:16:45.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257709782}) 2023-10-30T12:16:45.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=996476221}) 2023-10-30T12:16:45.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257709782}) 2023-10-30T12:16:45.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941157462}) 2023-10-30T12:16:45.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.933+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551791184}) 2023-10-30T12:16:45.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.934+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1747474446}) 2023-10-30T12:16:45.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.934+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.934+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551791184}) 2023-10-30T12:16:45.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.934+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3370119500}) 2023-10-30T12:16:45.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.934+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.934+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597113159}) 2023-10-30T12:16:45.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624500960}) 2023-10-30T12:16:45.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597113159}) 2023-10-30T12:16:45.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899740573}) 2023-10-30T12:16:45.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2357004749}) 2023-10-30T12:16:45.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487761874}) 2023-10-30T12:16:45.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2357004749}) 2023-10-30T12:16:45.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.935+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107803947}) 2023-10-30T12:16:45.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.936+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.936+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602395178}) 2023-10-30T12:16:45.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.936+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128935002}) 2023-10-30T12:16:45.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.936+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.936+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602395178}) 2023-10-30T12:16:45.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.936+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75198486}) 2023-10-30T12:16:45.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220328456}) 2023-10-30T12:16:45.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004132217}) 2023-10-30T12:16:45.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220328456}) 2023-10-30T12:16:45.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535802421}) 2023-10-30T12:16:45.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.937+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838977110}) 2023-10-30T12:16:45.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3180918552}) 2023-10-30T12:16:45.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838977110}) 2023-10-30T12:16:45.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676944395}) 2023-10-30T12:16:45.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683160985}) 2023-10-30T12:16:45.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3656120963}) 2023-10-30T12:16:45.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683160985}) 2023-10-30T12:16:45.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.938+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2755545410}) 2023-10-30T12:16:45.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.939+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.939+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3668614566}) 2023-10-30T12:16:45.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.939+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773378879}) 2023-10-30T12:16:45.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.939+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.939+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3668614566}) 2023-10-30T12:16:45.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.939+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=418091253}) 2023-10-30T12:16:45.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.940+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.940+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832422639}) 2023-10-30T12:16:45.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.940+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251496315}) 2023-10-30T12:16:45.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.940+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.940+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832422639}) 2023-10-30T12:16:45.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.940+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415679335}) 2023-10-30T12:16:45.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036662670}) 2023-10-30T12:16:45.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151705984}) 2023-10-30T12:16:45.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036662670}) 2023-10-30T12:16:45.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3864086543}) 2023-10-30T12:16:45.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735378030}) 2023-10-30T12:16:45.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208574735}) 2023-10-30T12:16:45.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.941+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735378030}) 2023-10-30T12:16:45.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.942+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909419448}) 2023-10-30T12:16:45.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.942+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.942+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121883381}) 2023-10-30T12:16:45.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.942+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318183523}) 2023-10-30T12:16:45.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.942+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.942+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121883381}) 2023-10-30T12:16:45.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.942+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517527090}) 2023-10-30T12:16:45.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.943+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.943+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296621003}) 2023-10-30T12:16:45.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.943+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4135642045}) 2023-10-30T12:16:45.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.943+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.943+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296621003}) 2023-10-30T12:16:45.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.943+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830714188}) 2023-10-30T12:16:45.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.944+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.944+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066210132}) 2023-10-30T12:16:45.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.944+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271614006}) 2023-10-30T12:16:45.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.944+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:45.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.944+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066210132}) 2023-10-30T12:16:45.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.944+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416925889}) 2023-10-30T12:16:46.010 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.010 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310971552}) 2023-10-30T12:16:46.010 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542178319}) 2023-10-30T12:16:46.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310971552}) 2023-10-30T12:16:46.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2900091374}) 2023-10-30T12:16:46.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=698296259}) 2023-10-30T12:16:46.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517860286}) 2023-10-30T12:16:46.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.945+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=698296259}) 2023-10-30T12:16:46.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1297497041}) 2023-10-30T12:16:46.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377163979}) 2023-10-30T12:16:46.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1475259366}) 2023-10-30T12:16:46.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377163979}) 2023-10-30T12:16:46.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.946+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643487157}) 2023-10-30T12:16:46.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.947+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.947+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034550045}) 2023-10-30T12:16:46.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.947+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2221914189}) 2023-10-30T12:16:46.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.947+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.947+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034550045}) 2023-10-30T12:16:46.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.947+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281726616}) 2023-10-30T12:16:46.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.948+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.948+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040212619}) 2023-10-30T12:16:46.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.948+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=727438820}) 2023-10-30T12:16:46.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.948+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.948+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040212619}) 2023-10-30T12:16:46.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.948+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712035462}) 2023-10-30T12:16:46.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905136856}) 2023-10-30T12:16:46.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=499304154}) 2023-10-30T12:16:46.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905136856}) 2023-10-30T12:16:46.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422831733}) 2023-10-30T12:16:46.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771951579}) 2023-10-30T12:16:46.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3577006025}) 2023-10-30T12:16:46.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771951579}) 2023-10-30T12:16:46.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.949+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1292311539}) 2023-10-30T12:16:46.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.950+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.950+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207522636}) 2023-10-30T12:16:46.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.950+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3318535631}) 2023-10-30T12:16:46.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.950+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.950+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207522636}) 2023-10-30T12:16:46.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.950+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1784842200}) 2023-10-30T12:16:46.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.951+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.951+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242761428}) 2023-10-30T12:16:46.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.951+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170446477}) 2023-10-30T12:16:46.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.951+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.951+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242761428}) 2023-10-30T12:16:46.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.951+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=751864424}) 2023-10-30T12:16:46.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.952+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.952+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278781252}) 2023-10-30T12:16:46.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.952+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3681593385}) 2023-10-30T12:16:46.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.952+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.952+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278781252}) 2023-10-30T12:16:46.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.952+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3092873238}) 2023-10-30T12:16:46.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400100742}) 2023-10-30T12:16:46.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726122561}) 2023-10-30T12:16:46.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400100742}) 2023-10-30T12:16:46.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864578783}) 2023-10-30T12:16:46.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155435271}) 2023-10-30T12:16:46.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3007075594}) 2023-10-30T12:16:46.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155435271}) 2023-10-30T12:16:46.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.953+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3489351249}) 2023-10-30T12:16:46.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.954+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.954+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741575510}) 2023-10-30T12:16:46.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.954+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350776368}) 2023-10-30T12:16:46.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.954+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.025 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.954+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741575510}) 2023-10-30T12:16:46.025 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.954+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684259692}) 2023-10-30T12:16:46.025 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.955+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.025 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.955+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859289938}) 2023-10-30T12:16:46.025 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.955+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481230039}) 2023-10-30T12:16:46.025 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.955+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.955+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859289938}) 2023-10-30T12:16:46.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.955+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317430429}) 2023-10-30T12:16:46.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.956+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.956+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504749503}) 2023-10-30T12:16:46.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.956+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4164220726}) 2023-10-30T12:16:46.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.956+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.956+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504749503}) 2023-10-30T12:16:46.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.956+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2032456240}) 2023-10-30T12:16:46.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.957+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.957+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572611807}) 2023-10-30T12:16:46.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.957+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2925485770}) 2023-10-30T12:16:46.028 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.957+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.028 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.957+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572611807}) 2023-10-30T12:16:46.028 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.957+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1521280632}) 2023-10-30T12:16:46.028 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.028 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933460107}) 2023-10-30T12:16:46.029 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363783711}) 2023-10-30T12:16:46.029 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.029 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933460107}) 2023-10-30T12:16:46.029 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1854352054}) 2023-10-30T12:16:46.029 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.029 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601316018}) 2023-10-30T12:16:46.030 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240506672}) 2023-10-30T12:16:46.030 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.030 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601316018}) 2023-10-30T12:16:46.030 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.958+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=162143603}) 2023-10-30T12:16:46.030 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.959+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.031 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.959+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615331787}) 2023-10-30T12:16:46.031 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.959+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588717407}) 2023-10-30T12:16:46.031 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.959+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.031 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.959+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615331787}) 2023-10-30T12:16:46.031 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.959+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937714228}) 2023-10-30T12:16:46.031 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.032 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032517598}) 2023-10-30T12:16:46.032 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3364263811}) 2023-10-30T12:16:46.032 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.032 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032517598}) 2023-10-30T12:16:46.032 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693284607}) 2023-10-30T12:16:46.032 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.033 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754114362}) 2023-10-30T12:16:46.033 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228406467}) 2023-10-30T12:16:46.033 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.033 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754114362}) 2023-10-30T12:16:46.033 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.961+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=581381956}) 2023-10-30T12:16:46.034 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.962+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.034 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.962+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511938689}) 2023-10-30T12:16:46.034 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.962+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4200417715}) 2023-10-30T12:16:46.034 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.962+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.034 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.962+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511938689}) 2023-10-30T12:16:46.035 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.962+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089595980}) 2023-10-30T12:16:46.035 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.963+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.035 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.963+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485257718}) 2023-10-30T12:16:46.035 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.963+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238879440}) 2023-10-30T12:16:46.035 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.963+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.036 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.963+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485257718}) 2023-10-30T12:16:46.036 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.963+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=245140838}) 2023-10-30T12:16:46.036 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.964+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.036 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.964+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906736664}) 2023-10-30T12:16:46.036 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.964+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=862798220}) 2023-10-30T12:16:46.036 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.964+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.037 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.964+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906736664}) 2023-10-30T12:16:46.037 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.964+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774029499}) 2023-10-30T12:16:46.037 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.037 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765453359}) 2023-10-30T12:16:46.037 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2034262595}) 2023-10-30T12:16:46.038 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.038 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765453359}) 2023-10-30T12:16:46.038 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139983233}) 2023-10-30T12:16:46.038 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.038 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169052951}) 2023-10-30T12:16:46.039 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=618295069}) 2023-10-30T12:16:46.039 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.039 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169052951}) 2023-10-30T12:16:46.039 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.965+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563705242}) 2023-10-30T12:16:46.039 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.039 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501559472}) 2023-10-30T12:16:46.040 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427292748}) 2023-10-30T12:16:46.040 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.040 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501559472}) 2023-10-30T12:16:46.040 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532617300}) 2023-10-30T12:16:46.040 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.041 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626163522}) 2023-10-30T12:16:46.041 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698717890}) 2023-10-30T12:16:46.041 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.041 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626163522}) 2023-10-30T12:16:46.041 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.966+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=540749543}) 2023-10-30T12:16:46.041 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.967+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.042 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.967+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=5,osd_snap=15} crc {logm=2445066497,mdsmap=3984425268,mgr=1711728957,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3884582629,mon_config_key=1505500295,monmap=3324057927,osd_metadata=2801852593,osd_snap=2172385760}) 2023-10-30T12:16:46.042 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.967+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=4212780541,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3884582629,mon_config_key=3217606506,monmap=4204254258,osd_metadata=2801852593,osd_pg_creating=3530178349,osd_snap=3765965428}) 2023-10-30T12:16:46.042 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.967+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.042 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.967+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=5,osd_snap=15} crc {logm=2445066497,mdsmap=3984425268,mgr=1711728957,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3884582629,mon_config_key=1505500295,monmap=3324057927,osd_metadata=2801852593,osd_snap=2172385760}) 2023-10-30T12:16:46.042 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.967+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=2927767097,mdsmap=3984425269,mgr=3853015086,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=535444230,mon_config_key=715115367,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=3530178349,osd_snap=2172385761}) 2023-10-30T12:16:46.043 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.969+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.043 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.969+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=1363000781,osdmap=3264618825}) 2023-10-30T12:16:46.043 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.969+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=22,osdmap=78} crc {osd_snap=2164767813,osdmap=3949821268}) 2023-10-30T12:16:46.043 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.969+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.043 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.969+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=1363000781,osdmap=3264618825}) 2023-10-30T12:16:46.044 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.969+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=22,osdmap=78} crc {osd_snap=3636700290,osdmap=667724150}) 2023-10-30T12:16:46.044 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.976+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.044 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.976+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=4167809609}) 2023-10-30T12:16:46.044 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.976+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=58} crc {osdmap=3391224942}) 2023-10-30T12:16:46.044 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.976+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:16:46.045 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.976+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=4167809609}) 2023-10-30T12:16:46.045 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:16:45.976+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=58} crc {osdmap=13377957}) 2023-10-30T12:16:46.926 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:16:46.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.041+0000 7f1f4c66f640 10 monclient: get_monmap_and_config 2023-10-30T12:16:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.041+0000 7f1f4c66f640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.041+0000 7f1f4c66f640 10 monclient: monmap: 2023-10-30T12:16:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:47.042347+0000 2023-10-30T12:16:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:47.042347+0000 2023-10-30T12:16:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:47.045 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:47.045 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:47.045 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:47.045 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.041+0000 7f1f4c66f640 1 Processor -- start 2023-10-30T12:16:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.041+0000 7f1f4c66f640 1 -- start start 2023-10-30T12:16:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.041+0000 7f1f4c66f640 10 monclient: init 2023-10-30T12:16:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-10-30T12:16:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1f44135d20 0x7f1f44136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f44135d20 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:47.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f44135670 0x7f1f44070390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): picked mon.noname-b con 0x7f1f44135670 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:47.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f440708d0 0x7f1f44070d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): picked mon.noname-a con 0x7f1f440708d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:47.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4409d4e0 con 0x7f1f440708d0 2023-10-30T12:16:47.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f44071620 con 0x7f1f44135d20 2023-10-30T12:16:47.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440a01a0 con 0x7f1f44135670 2023-10-30T12:16:47.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4c66f640 10 monclient(hunting): authenticate will time out at 1852.225561s 2023-10-30T12:16:47.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f44135670 0x7f1f44070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f440708d0 0x7f1f44070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f440708d0 0x7f1f44070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43196/0 (socket says 172.21.15.50:43196) 2023-10-30T12:16:47.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 1 -- 172.21.15.50:0/1030779083 learned_addr learned my addr 172.21.15.50:0/1030779083 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:47.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f44135670 0x7f1f44070390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53790/0 (socket says 172.21.15.50:53790) 2023-10-30T12:16:47.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 10 monclient(hunting): get_auth_request con 0x7f1f440708d0 auth_method 0 2023-10-30T12:16:47.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 10 monclient(hunting): get_auth_request con 0x7f1f44135670 auth_method 0 2023-10-30T12:16:47.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 1 --2- 172.21.15.50:0/1030779083 >> v2:172.21.15.118:3300/0 conn(0x7f1f44135d20 0x7f1f44136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 10 monclient(hunting): get_auth_request con 0x7f1f44135d20 auth_method 0 2023-10-30T12:16:47.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4ac0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f49c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.042+0000 7f1f4a40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 10 monclient(hunting): handle_auth_done global_id 214158 payload 274 2023-10-30T12:16:47.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 1 -- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f44135670 msgr2=0x7f1f44070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:47.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 1 --2- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f44135670 0x7f1f44070390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 1 -- 172.21.15.50:0/1030779083 >> v2:172.21.15.118:3300/0 conn(0x7f1f44135d20 msgr2=0x7f1f44136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:47.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 1 --2- 172.21.15.50:0/1030779083 >> v2:172.21.15.118:3300/0 conn(0x7f1f44135d20 0x7f1f44136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 1 monclient: found mon.noname-a 2023-10-30T12:16:47.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:47.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 1 -- 172.21.15.50:0/1030779083 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f44099f60 con 0x7f1f440708d0 2023-10-30T12:16:47.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49c0a640 1 --2- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f44135670 0x7f1f44070390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:47.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4ac0c640 1 --2- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f440708d0 0x7f1f44070d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f44067130 tx=0x7f1f3c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20289758912ead5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 1 -- 172.21.15.50:0/1030779083 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c02c960 con 0x7f1f440708d0 2023-10-30T12:16:47.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:47.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 10 monclient: dump: 2023-10-30T12:16:47.067 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.068 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.068 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.068 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.069 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 1 -- 172.21.15.50:0/1030779083 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c02cac0 con 0x7f1f440708d0 2023-10-30T12:16:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4c66f640 5 monclient: authenticate success, global_id 214158 2023-10-30T12:16:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f4c66f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.043+0000 7f1f49409640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f49409640 1 -- 172.21.15.50:0/1030779083 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c02ce50 con 0x7f1f440708d0 2023-10-30T12:16:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 4 set_mon_vals no callback set 2023-10-30T12:16:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f49409640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f49409640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f49409640 10 monclient: dump: 2023-10-30T12:16:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.072 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.074 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.074 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 10 monclient: shutdown 2023-10-30T12:16:47.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 -- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f440708d0 msgr2=0x7f1f44070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f440708d0 0x7f1f44070d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f44067130 tx=0x7f1f3c02bdc0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 -- 172.21.15.50:0/1030779083 shutdown_connections 2023-10-30T12:16:47.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f440708d0 0x7f1f44070d40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/1030779083 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f44135670 0x7f1f44070390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/1030779083 >> v2:172.21.15.118:3300/0 conn(0x7f1f44135d20 0x7f1f44136100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 -- 172.21.15.50:0/1030779083 >> 172.21.15.50:0/1030779083 conn(0x7f1f440a4670 msgr2=0x7f1f440a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 -- 172.21.15.50:0/1030779083 shutdown_connections 2023-10-30T12:16:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.044+0000 7f1f4c66f640 1 -- 172.21.15.50:0/1030779083 wait complete. 2023-10-30T12:16:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 10 monclient: monmap: 2023-10-30T12:16:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:47.078 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:47.078 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.078 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.078 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:47.078 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:47.079 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:47.079 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 1 librados: starting msgr at 2023-10-30T12:16:47.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 1 librados: starting objecter 2023-10-30T12:16:47.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 1 Processor -- start 2023-10-30T12:16:47.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 1 -- start start 2023-10-30T12:16:47.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 1 librados: setting wanted keys 2023-10-30T12:16:47.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 1 librados: calling monclient init 2023-10-30T12:16:47.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.045+0000 7f1f4c66f640 10 monclient: init 2023-10-30T12:16:47.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:16:47.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f440708d0 0x7f1f441ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f440708d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:47.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f44135670 0x7f1f4411f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): picked mon.noname-b con 0x7f1f44135670 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:47.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f44135d20 0x7f1f4411fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): picked mon.noname-a con 0x7f1f44135d20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:47.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f44121b60 con 0x7f1f44135d20 2023-10-30T12:16:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f441220d0 con 0x7f1f44135670 2023-10-30T12:16:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4409d6b0 con 0x7f1f440708d0 2023-10-30T12:16:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 10 monclient(hunting): authenticate will time out at 1852.229250s 2023-10-30T12:16:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f440708d0 0x7f1f441ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f440708d0 0x7f1f441ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53800/0 (socket says 172.21.15.50:53800) 2023-10-30T12:16:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 -- 172.21.15.50:0/3839778222 learned_addr learned my addr 172.21.15.50:0/3839778222 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): get_auth_request con 0x7f1f440708d0 auth_method 0 2023-10-30T12:16:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f44135d20 0x7f1f4411fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 10 monclient(hunting): get_auth_request con 0x7f1f44135d20 auth_method 0 2023-10-30T12:16:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f49c0a640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f44135670 0x7f1f4411f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f49c0a640 10 monclient(hunting): get_auth_request con 0x7f1f44135670 auth_method 0 2023-10-30T12:16:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f49c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f49c0a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f49c0a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient(hunting): handle_auth_done global_id 115190 payload 824 2023-10-30T12:16:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 -- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f44135670 msgr2=0x7f1f4411f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f44135670 0x7f1f4411f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 -- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f44135d20 msgr2=0x7f1f4411fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f44135d20 0x7f1f4411fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 monclient: found mon.noname-c 2023-10-30T12:16:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 -- 172.21.15.50:0/3839778222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f34009400 con 0x7f1f440708d0 2023-10-30T12:16:47.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4ac0c640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f44135d20 0x7f1f4411fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4a40b640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f440708d0 0x7f1f441ca4c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e9b0 tx=0x7f1f4000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 10 client.?.objecter ms_handle_connect 0x7f1f440708d0 2023-10-30T12:16:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 1 -- 172.21.15.50:0/3839778222 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40009670 con 0x7f1f440708d0 2023-10-30T12:16:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 10 monclient: dump: 2023-10-30T12:16:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.097 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f4c66f640 5 monclient: authenticate success, global_id 115190 2023-10-30T12:16:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.046+0000 7f1f3affd640 1 -- 172.21.15.50:0/3839778222 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4000c9a0 con 0x7f1f440708d0 2023-10-30T12:16:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 monclient: _renew_subs 2023-10-30T12:16:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c009b30 con 0x7f1f440708d0 2023-10-30T12:16:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f3affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f3affd640 1 -- 172.21.15.50:0/3839778222 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000cd50 con 0x7f1f440708d0 2023-10-30T12:16:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f3affd640 10 monclient: dump: 2023-10-30T12:16:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.101 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 client.115190.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 monclient: _renew_subs 2023-10-30T12:16:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f48c08640 4 set_mon_vals no callback set 2023-10-30T12:16:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f44123860 con 0x7f1f440708d0 2023-10-30T12:16:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 1 librados: init done 2023-10-30T12:16:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f3affd640 1 -- 172.21.15.50:0/3839778222 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1f40013470 con 0x7f1f440708d0 2023-10-30T12:16:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.047+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f44123430 con 0x7f1f440708d0 2023-10-30T12:16:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f3affd640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f280392f0 0x7f1f2803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f3affd640 1 -- 172.21.15.50:0/3839778222 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(504..504 src has 417..504) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f1f4004b070 con 0x7f1f440708d0 2023-10-30T12:16:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f3affd640 10 client.115190.objecter ms_dispatch 0x7f1f44000b90 osd_map(504..504 src has 417..504) v4 2023-10-30T12:16:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f3affd640 3 client.115190.objecter handle_osd_map got epochs [504,504] > 0 2023-10-30T12:16:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f3affd640 3 client.115190.objecter handle_osd_map decoding full epoch 504 2023-10-30T12:16:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f3affd640 20 client.115190.objecter dump_active .. 0 homeless 2023-10-30T12:16:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f49c0a640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f280392f0 0x7f1f2803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f49c0a640 10 monclient: get_auth_request con 0x7f1f280392f0 auth_method 0 2023-10-30T12:16:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f49c0a640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f280392f0 0x7f1f2803b7b0 secure :-1 s=READY pgs=4231 cs=0 l=1 rev1=1 crypto rx=0x7f1f44122060 tx=0x7f1f34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.048+0000 7f1f3affd640 10 client.115190.objecter ms_handle_connect 0x7f1f280392f0 2023-10-30T12:16:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.052+0000 7f1f3affd640 1 -- 172.21.15.50:0/3839778222 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1f400232f0 con 0x7f1f440708d0 2023-10-30T12:16:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.052+0000 7f1f3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.052+0000 7f1f3affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:47.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.297+0000 7f1f4c66f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:47.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.297+0000 7f1f4c66f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:47.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.297+0000 7f1f4c66f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.297+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1f4404e160 con 0x7f1f440708d0 2023-10-30T12:16:47.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.297+0000 7f1f3affd640 1 -- 172.21.15.50:0/3839778222 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f1f40021020 con 0x7f1f440708d0 2023-10-30T12:16:47.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.297+0000 7f1f3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:47.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.297+0000 7f1f3affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:47.300 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 10 librados: watch_flush enter 2023-10-30T12:16:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 10 librados: watch_flush exit 2023-10-30T12:16:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 20 client.115190.objecter shutdown clearing up homeless session... 2023-10-30T12:16:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 10 client.115190.objecter successfully canceled tick 2023-10-30T12:16:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f280392f0 msgr2=0x7f1f2803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f280392f0 0x7f1f2803b7b0 secure :-1 s=READY pgs=4231 cs=0 l=1 rev1=1 crypto rx=0x7f1f44122060 tx=0x7f1f34008040 comp rx=0 tx=0).stop 2023-10-30T12:16:47.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 10 monclient: shutdown 2023-10-30T12:16:47.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f440708d0 msgr2=0x7f1f441ca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f440708d0 0x7f1f441ca4c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e9b0 tx=0x7f1f4000ee70 comp rx=0 tx=0).stop 2023-10-30T12:16:47.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 shutdown_connections 2023-10-30T12:16:47.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1f280392f0 0x7f1f2803b7b0 unknown :-1 s=CLOSED pgs=4231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f44135d20 0x7f1f4411fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1f44135670 0x7f1f4411f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 --2- 172.21.15.50:0/3839778222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f440708d0 0x7f1f441ca4c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 >> 172.21.15.50:0/3839778222 conn(0x7f1f440a4670 msgr2=0x7f1f440a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 shutdown_connections 2023-10-30T12:16:47.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 -- 172.21.15.50:0/3839778222 wait complete. 2023-10-30T12:16:47.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.299+0000 7f1f4c66f640 1 librados: shutdown 2023-10-30T12:16:47.311 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:16:47.311 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:16:47.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:16:47.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:16:47.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.419+0000 7f9b35ec2640 10 monclient: get_monmap_and_config 2023-10-30T12:16:47.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.419+0000 7f9b35ec2640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.419+0000 7f9b35ec2640 10 monclient: monmap: 2023-10-30T12:16:47.421 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.421 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.421 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:47.420180+0000 2023-10-30T12:16:47.422 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:47.420180+0000 2023-10-30T12:16:47.422 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.422 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.422 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:47.422 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:47.422 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:47.423 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:47.423 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:47.423 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.419+0000 7f9b35ec2640 1 Processor -- start 2023-10-30T12:16:47.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.419+0000 7f9b35ec2640 1 -- start start 2023-10-30T12:16:47.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.419+0000 7f9b35ec2640 10 monclient: init 2023-10-30T12:16:47.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-10-30T12:16:47.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9b300a8010 0x7f9b30132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient(hunting): picked mon.noname-c con 0x7f9b300a8010 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:47.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9b30099a40 0x7f9b30132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient(hunting): picked mon.noname-a con 0x7f9b30099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:47.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b301356d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:47.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300a7190 con 0x7f9b301356d0 2023-10-30T12:16:47.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b30133110 con 0x7f9b30099a40 2023-10-30T12:16:47.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b300a83f0 con 0x7f9b300a8010 2023-10-30T12:16:47.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b35ec2640 10 monclient(hunting): authenticate will time out at 1852.603411s 2023-10-30T12:16:47.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9b30099a40 0x7f9b30132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9b30099a40 0x7f9b30132bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43208/0 (socket says 172.21.15.50:43208) 2023-10-30T12:16:47.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 -- 172.21.15.50:0/3639390705 learned_addr learned my addr 172.21.15.50:0/3639390705 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:47.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b30099a40 auth_method 0 2023-10-30T12:16:47.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/3639390705 >> v2:172.21.15.118:3300/0 conn(0x7f9b300a8010 0x7f9b30132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b300a8010 auth_method 0 2023-10-30T12:16:47.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient(hunting): handle_auth_done global_id 214164 payload 274 2023-10-30T12:16:47.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 -- 172.21.15.50:0/3639390705 >> v2:172.21.15.118:3300/0 conn(0x7f9b300a8010 msgr2=0x7f9b30132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:47.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 --2- 172.21.15.50:0/3639390705 >> v2:172.21.15.118:3300/0 conn(0x7f9b300a8010 0x7f9b30132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 -- 172.21.15.50:0/3639390705 >> v1:172.21.15.118:6789/0 conn(0x7f9b301356d0 legacy=0x7f9b30135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 monclient: found mon.noname-a 2023-10-30T12:16:47.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:47.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.420+0000 7f9b2effd640 1 -- 172.21.15.50:0/3639390705 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300a7600 con 0x7f9b30099a40 2023-10-30T12:16:47.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2effd640 1 --2- 172.21.15.50:0/3639390705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b30132bd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b300a8640 tx=0x7f9b2402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559254891d8fe8f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/3639390705 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b240317e0 con 0x7f9b30099a40 2023-10-30T12:16:47.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:47.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: dump: 2023-10-30T12:16:47.441 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.441 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.441 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.442 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.442 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.442 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.442 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.442 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.443 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.443 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/3639390705 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b24031940 con 0x7f9b30099a40 2023-10-30T12:16:47.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 1 -- 172.21.15.50:0/3639390705 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b24031cd0 con 0x7f9b30099a40 2023-10-30T12:16:47.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b35ec2640 5 monclient: authenticate success, global_id 214164 2023-10-30T12:16:47.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b35ec2640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:47.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b35ec2640 4 set_mon_vals no callback set 2023-10-30T12:16:47.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:47.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.421+0000 7f9b2e7fc640 10 monclient: dump: 2023-10-30T12:16:47.445 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.446 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.446 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.446 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.446 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.446 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.447 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.447 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.447 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.447 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 10 monclient: shutdown 2023-10-30T12:16:47.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 -- 172.21.15.50:0/3639390705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 msgr2=0x7f9b30132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 --2- 172.21.15.50:0/3639390705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b30132bd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b300a8640 tx=0x7f9b2402b880 comp rx=0 tx=0).stop 2023-10-30T12:16:47.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 -- 172.21.15.50:0/3639390705 shutdown_connections 2023-10-30T12:16:47.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 --2- 172.21.15.50:0/3639390705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b30132bd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 --2- 172.21.15.50:0/3639390705 >> v2:172.21.15.118:3300/0 conn(0x7f9b300a8010 0x7f9b30132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 -- 172.21.15.50:0/3639390705 >> 172.21.15.50:0/3639390705 conn(0x7f9b300a4a10 msgr2=0x7f9b300a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 -- 172.21.15.50:0/3639390705 shutdown_connections 2023-10-30T12:16:47.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 1 -- 172.21.15.50:0/3639390705 wait complete. 2023-10-30T12:16:47.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.422+0000 7f9b35ec2640 10 monclient: monmap: 2023-10-30T12:16:47.451 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.451 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:47.451 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:47.452 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:47.452 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.452 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.452 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:47.452 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:47.453 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:47.453 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 librados: starting msgr at 2023-10-30T12:16:47.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 librados: starting objecter 2023-10-30T12:16:47.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 Processor -- start 2023-10-30T12:16:47.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 -- start start 2023-10-30T12:16:47.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 librados: setting wanted keys 2023-10-30T12:16:47.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 librados: calling monclient init 2023-10-30T12:16:47.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient: init 2023-10-30T12:16:47.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:16:47.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a8010 0x7f9b301ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): picked mon.noname-c con 0x7f9b300a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:47.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b301356d0 0x7f9b301cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): picked mon.noname-b con 0x7f9b301356d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:47.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011e580 0x7f9b3011ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3011e580 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:47.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30120e60 con 0x7f9b3011e580 2023-10-30T12:16:47.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30134b60 con 0x7f9b301356d0 2023-10-30T12:16:47.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b300a18d0 con 0x7f9b300a8010 2023-10-30T12:16:47.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b35ec2640 10 monclient(hunting): authenticate will time out at 1852.606868s 2023-10-30T12:16:47.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b2ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011e580 0x7f9b3011ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a8010 0x7f9b301ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.423+0000 7f9b2ffff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011e580 0x7f9b3011ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43214/0 (socket says 172.21.15.50:43214) 2023-10-30T12:16:47.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b300a8010 auth_method 0 2023-10-30T12:16:47.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 1 -- 172.21.15.50:0/2625755821 learned_addr learned my addr 172.21.15.50:0/2625755821 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:47.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 10 monclient(hunting): get_auth_request con 0x7f9b3011e580 auth_method 0 2023-10-30T12:16:47.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b301356d0 0x7f9b301cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 10 monclient(hunting): get_auth_request con 0x7f9b301356d0 auth_method 0 2023-10-30T12:16:47.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient(hunting): handle_auth_done global_id 115193 payload 824 2023-10-30T12:16:47.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b301356d0 msgr2=0x7f9b301cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9b301356d0 0x7f9b301cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011e580 msgr2=0x7f9b3011ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011e580 0x7f9b3011ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 1 monclient: found mon.noname-c 2023-10-30T12:16:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 1 -- 172.21.15.50:0/2625755821 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240094d0 con 0x7f9b300a8010 2023-10-30T12:16:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2ffff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011e580 0x7f9b3011ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2f7fe640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a8010 0x7f9b301ca800 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000e870 tx=0x7f9b2000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 client.?.objecter ms_handle_connect 0x7f9b300a8010 2023-10-30T12:16:47.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:47.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20009590 con 0x7f9b300a8010 2023-10-30T12:16:47.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: dump: 2023-10-30T12:16:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.468 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2000c940 con 0x7f9b300a8010 2023-10-30T12:16:47.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2000ccd0 con 0x7f9b300a8010 2023-10-30T12:16:47.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 5 monclient: authenticate success, global_id 115193 2023-10-30T12:16:47.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 10 monclient: _renew_subs 2023-10-30T12:16:47.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b30121d50 con 0x7f9b300a8010 2023-10-30T12:16:47.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b2dffb640 4 set_mon_vals no callback set 2023-10-30T12:16:47.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:47.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b17fff640 10 monclient: dump: 2023-10-30T12:16:47.471 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.474 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 10 client.115193.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:47.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 10 monclient: _renew_subs 2023-10-30T12:16:47.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b30120220 con 0x7f9b300a8010 2023-10-30T12:16:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.424+0000 7f9b35ec2640 1 librados: init done 2023-10-30T12:16:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b35ec2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b35ec2640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b3002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b35ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30073fd0 con 0x7f9b30099a40 2023-10-30T12:16:47.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9b20013470 con 0x7f9b300a8010 2023-10-30T12:16:47.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b3002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 10 monclient: get_auth_request con 0x7f9b30099a40 auth_method 0 2023-10-30T12:16:47.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.425+0000 7f9b17fff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(505..505 src has 417..505) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f9b2004ac20 con 0x7f9b300a8010 2023-10-30T12:16:47.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 client.115193.objecter ms_dispatch 0x7f9b30000b90 osd_map(505..505 src has 417..505) v4 2023-10-30T12:16:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 3 client.115193.objecter handle_osd_map got epochs [505,505] > 0 2023-10-30T12:16:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 3 client.115193.objecter handle_osd_map decoding full epoch 505 2023-10-30T12:16:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b2effd640 10 monclient(hunting): handle_auth_done global_id 214170 payload 824 2023-10-30T12:16:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b2effd640 1 -- 172.21.15.50:0/2625755821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b3009a720 con 0x7f9b30099a40 2023-10-30T12:16:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 20 client.115193.objecter dump_active .. 0 homeless 2023-10-30T12:16:47.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b2ffff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b2ffff640 10 monclient: get_auth_request con 0x7f9b1c0392f0 auth_method 0 2023-10-30T12:16:47.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 client.115193.objecter ms_handle_connect 0x7f9b30099a40 2023-10-30T12:16:47.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 client.115193.objecter resend_mon_ops 2023-10-30T12:16:47.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b3002b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009a210 tx=0x7f9b24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b24032880 con 0x7f9b30099a40 2023-10-30T12:16:47.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:47.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 monclient: dump: 2023-10-30T12:16:47.481 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.482 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.482 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.482 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.482 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.482 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.483 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.483 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.483 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.483 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 client.115193.objecter ms_handle_connect 0x7f9b1c0392f0 2023-10-30T12:16:47.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b2ffff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 secure :-1 s=READY pgs=4232 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800f290 tx=0x7f9b28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9b240329e0 con 0x7f9b30099a40 2023-10-30T12:16:47.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:47.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 msgr2=0x7f9b3002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b3002b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009a210 tx=0x7f9b24004730 comp rx=0 tx=0).stop 2023-10-30T12:16:47.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 msgr2=0x7f9b3002b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:47.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.426+0000 7f9b17fff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b30099a40 0x7f9b3002b430 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b3009a210 tx=0x7f9b24004730 comp rx=0 tx=0).stop 2023-10-30T12:16:47.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.432+0000 7f9b35ec2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:16:47.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.432+0000 7f9b35ec2640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011f710 0x7f9b3002b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.432+0000 7f9b35ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.432+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30121ee0 con 0x7f9b3011f710 2023-10-30T12:16:47.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011f710 0x7f9b3002b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient: get_auth_request con 0x7f9b3011f710 auth_method 0 2023-10-30T12:16:47.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 10 monclient(hunting): handle_auth_done global_id 214173 payload 824 2023-10-30T12:16:47.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 1 -- 172.21.15.50:0/2625755821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9b3009a500 con 0x7f9b3011f710 2023-10-30T12:16:47.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b2effd640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011f710 0x7f9b3002b970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b2402b8c0 tx=0x7f9b240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 10 client.115193.objecter ms_handle_connect 0x7f9b3011f710 2023-10-30T12:16:47.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 10 client.115193.objecter resend_mon_ops 2023-10-30T12:16:47.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2404fb40 con 0x7f9b3011f710 2023-10-30T12:16:47.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:47.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 10 monclient: dump: 2023-10-30T12:16:47.491 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.491 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.491 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.491 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.492 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.492 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.492 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.492 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.492 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.492 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f9b2404fca0 con 0x7f9b3011f710 2023-10-30T12:16:47.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:16:47.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:47.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011f710 msgr2=0x7f9b3002b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011f710 0x7f9b3002b970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b2402b8c0 tx=0x7f9b240046c0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011f710 msgr2=0x7f9b3002b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:47.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.433+0000 7f9b17fff640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b3011f710 0x7f9b3002b970 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.494 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:16:47.494 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:47.495 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:47.495 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:16:47.495 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 170, 2023-10-30T12:16:47.495 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:16:47.495 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:16:47.496 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:16:47.496 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:16:47.496 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.496 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:16:47.496 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:47.497 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:16:47.497 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:16:47.497 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.497 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.497 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.497 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.498 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.498 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.498 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.498 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.501 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.501 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.501 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.501 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:16:47.502 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:16:47.502 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.502 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.502 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.502 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.503 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.503 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.503 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.503 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.503 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.504 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.504 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:16:47.504 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:16:47.505 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:16:47.505 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:16:47.505 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:16:47.505 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:16:47.505 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:47.505 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:47.506 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:16:47.506 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:16:47.506 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:16:47.506 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:16:47.506 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:16:47.507 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:16:47.507 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:16:47.507 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:47.507 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:16:47.507 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.508 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.508 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.508 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.508 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.508 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.508 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.509 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.509 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.509 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.509 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.509 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:16:47.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.510 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:16:47.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.510 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:47.510 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:47.511 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.511 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.511 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:16:47.512 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.512 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:16:47.513 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.513 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.514 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:16:47.514 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:16:47.514 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.515 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.515 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.516 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:47.516 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:47.516 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.516 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.516 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.517 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:16:47.517 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.517 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.518 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:16:47.518 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.518 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.519 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:16:47.519 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:16:47.519 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.519 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.519 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.520 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:47.520 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:47.520 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.520 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.521 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.521 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:16:47.521 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.522 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.522 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:16:47.522 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.522 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.523 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:16:47.523 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:16:47.523 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.523 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.523 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.524 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.524 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.524 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:16:47.524 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:16:47.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.525 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.525 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.525 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:47.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.525 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.526 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:16:47.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.526 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.526 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.526 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:16:47.527 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:16:47.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 10 librados: watch_flush enter 2023-10-30T12:16:47.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 10 librados: watch_flush exit 2023-10-30T12:16:47.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 20 client.115193.objecter shutdown clearing up homeless session... 2023-10-30T12:16:47.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 10 client.115193.objecter successfully canceled tick 2023-10-30T12:16:47.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 msgr2=0x7f9b1c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9b1c0392f0 0x7f9b1c03b7b0 secure :-1 s=READY pgs=4232 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800f290 tx=0x7f9b28009040 comp rx=0 tx=0).stop 2023-10-30T12:16:47.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 10 monclient: shutdown 2023-10-30T12:16:47.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a8010 msgr2=0x7f9b301ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 --2- 172.21.15.50:0/2625755821 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b300a8010 0x7f9b301ca800 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000e870 tx=0x7f9b2000ed30 comp rx=0 tx=0).stop 2023-10-30T12:16:47.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b2ffff640 1 -- 172.21.15.50:0/2625755821 reap_dead start 2023-10-30T12:16:47.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b2ffff640 1 -- 172.21.15.50:0/2625755821 reap_dead start 2023-10-30T12:16:47.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 shutdown_connections 2023-10-30T12:16:47.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 >> 172.21.15.50:0/2625755821 conn(0x7f9b300a4a10 msgr2=0x7f9b301332c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 shutdown_connections 2023-10-30T12:16:47.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 -- 172.21.15.50:0/2625755821 wait complete. 2023-10-30T12:16:47.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.434+0000 7f9b35ec2640 1 librados: shutdown 2023-10-30T12:16:47.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:16:47.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:16:47.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:16:47.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.638+0000 7fe5287c5640 10 monclient: get_monmap_and_config 2023-10-30T12:16:47.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.638+0000 7fe5287c5640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.638+0000 7fe5287c5640 10 monclient: monmap: 2023-10-30T12:16:47.641 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.641 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.641 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:47.640013+0000 2023-10-30T12:16:47.641 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:47.640013+0000 2023-10-30T12:16:47.641 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.642 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.642 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:47.642 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:47.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:47.642 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:47.643 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:47.643 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.639+0000 7fe5287c5640 1 Processor -- start 2023-10-30T12:16:47.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.639+0000 7fe5287c5640 1 -- start start 2023-10-30T12:16:47.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.639+0000 7fe5287c5640 10 monclient: init 2023-10-30T12:16:47.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-10-30T12:16:47.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe520096540 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe520136960 0x7fe520138d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient(hunting): picked mon.noname-a con 0x7fe520136960 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe520139290 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52009bd30 con 0x7fe520139290 2023-10-30T12:16:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52009c400 con 0x7fe520096540 2023-10-30T12:16:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52009c120 con 0x7fe520136960 2023-10-30T12:16:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe5287c5640 10 monclient(hunting): authenticate will time out at 1852.823177s 2023-10-30T12:16:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe520136960 0x7fe520138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe526d62640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe520139290 0x7fe52013b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57360/0 (socket says 172.21.15.50:57360) 2023-10-30T12:16:47.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe526d62640 1 -- 172.21.15.50:0/2667990707 learned_addr learned my addr 172.21.15.50:0/2667990707 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:47.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe520136960 0x7fe520138d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48500/0 (socket says 172.21.15.50:48500) 2023-10-30T12:16:47.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): get_auth_request con 0x7fe520136960 auth_method 0 2023-10-30T12:16:47.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient(hunting): handle_auth_done global_id 214176 payload 274 2023-10-30T12:16:47.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 1 -- 172.21.15.50:0/2667990707 >> v1:172.21.15.118:6789/0 conn(0x7fe520096540 legacy=0x7fe520096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:47.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 1 -- 172.21.15.50:0/2667990707 >> v1:172.21.15.50:6789/0 conn(0x7fe520139290 legacy=0x7fe52013b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 1 monclient: found mon.noname-a 2023-10-30T12:16:47.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:47.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe525d60640 1 -- 172.21.15.50:0/2667990707 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe520097030 con 0x7fe520136960 2023-10-30T12:16:47.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe52555f640 1 -- 172.21.15.50:0/2667990707 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835951556 0 0) 0x7fe52009bd30 con 0x7fe520139290 2023-10-30T12:16:47.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.640+0000 7fe52555f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:16:47.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe525d60640 1 --2- 172.21.15.50:0/2667990707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520136960 0x7fe520138d50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0097a0 tx=0x7fe51c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98360ea62921d74e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 1 -- 172.21.15.50:0/2667990707 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c02e070 con 0x7fe520136960 2023-10-30T12:16:47.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:47.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: dump: 2023-10-30T12:16:47.658 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.658 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.658 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.659 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.659 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.659 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.659 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.659 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.660 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 1 -- 172.21.15.50:0/2667990707 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c0315a0 con 0x7fe520136960 2023-10-30T12:16:47.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe5287c5640 5 monclient: authenticate success, global_id 214176 2023-10-30T12:16:47.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe5287c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:47.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe5287c5640 4 set_mon_vals no callback set 2023-10-30T12:16:47.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 1 -- 172.21.15.50:0/2667990707 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c031930 con 0x7fe520136960 2023-10-30T12:16:47.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:47.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.641+0000 7fe52555f640 10 monclient: dump: 2023-10-30T12:16:47.662 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.662 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.662 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.664 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.664 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 10 monclient: shutdown 2023-10-30T12:16:47.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 -- 172.21.15.50:0/2667990707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520136960 msgr2=0x7fe520138d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 --2- 172.21.15.50:0/2667990707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520136960 0x7fe520138d50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0097a0 tx=0x7fe51c02b840 comp rx=0 tx=0).stop 2023-10-30T12:16:47.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 -- 172.21.15.50:0/2667990707 shutdown_connections 2023-10-30T12:16:47.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 --2- 172.21.15.50:0/2667990707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520136960 0x7fe520138d50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 -- 172.21.15.50:0/2667990707 >> 172.21.15.50:0/2667990707 conn(0x7fe5200a3980 msgr2=0x7fe5200a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 -- 172.21.15.50:0/2667990707 shutdown_connections 2023-10-30T12:16:47.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 -- 172.21.15.50:0/2667990707 wait complete. 2023-10-30T12:16:47.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 10 monclient: monmap: 2023-10-30T12:16:47.666 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.666 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:47.666 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:47.668 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 librados: starting msgr at 2023-10-30T12:16:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.642+0000 7fe5287c5640 1 librados: starting objecter 2023-10-30T12:16:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 Processor -- start 2023-10-30T12:16:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 -- start start 2023-10-30T12:16:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 librados: setting wanted keys 2023-10-30T12:16:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 librados: calling monclient init 2023-10-30T12:16:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient: init 2023-10-30T12:16:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:16:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520096540 0x7fe5201ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): picked mon.noname-c con 0x7fe520096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe520136960 0x7fe5201cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): picked mon.noname-b con 0x7fe520136960 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520139290 0x7fe520070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): picked mon.noname-a con 0x7fe520139290 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52008f5a0 con 0x7fe520139290 2023-10-30T12:16:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52008f7f0 con 0x7fe520136960 2023-10-30T12:16:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52009a080 con 0x7fe520096540 2023-10-30T12:16:47.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe5287c5640 10 monclient(hunting): authenticate will time out at 1852.826603s 2023-10-30T12:16:47.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520139290 0x7fe520070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520096540 0x7fe5201ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520096540 0x7fe5201ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55204/0 (socket says 172.21.15.50:55204) 2023-10-30T12:16:47.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520139290 0x7fe520070400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48512/0 (socket says 172.21.15.50:48512) 2023-10-30T12:16:47.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 1 -- 172.21.15.50:0/1045507230 learned_addr learned my addr 172.21.15.50:0/1045507230 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:47.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 10 monclient(hunting): get_auth_request con 0x7fe520096540 auth_method 0 2023-10-30T12:16:47.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): get_auth_request con 0x7fe520139290 auth_method 0 2023-10-30T12:16:47.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe525d60640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe520136960 0x7fe5201cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe525d60640 10 monclient(hunting): get_auth_request con 0x7fe520136960 auth_method 0 2023-10-30T12:16:47.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe525d60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe525d60640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe525d60640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.643+0000 7fe526d62640 10 monclient(hunting): handle_auth_done global_id 214182 payload 824 2023-10-30T12:16:47.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520096540 msgr2=0x7fe5201ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520096540 0x7fe5201ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe520136960 msgr2=0x7fe5201cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe520136960 0x7fe5201cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 1 monclient: found mon.noname-a 2023-10-30T12:16:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 1 -- 172.21.15.50:0/1045507230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c009400 con 0x7fe520139290 2023-10-30T12:16:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe525d60640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe520136960 0x7fe5201cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:16:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe526d62640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520139290 0x7fe520070400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c6d0 tx=0x7fe51800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 client.?.objecter ms_handle_connect 0x7fe520139290 2023-10-30T12:16:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518004d60 con 0x7fe520139290 2023-10-30T12:16:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: dump: 2023-10-30T12:16:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.681 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.681 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.681 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.681 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 5 monclient: authenticate success, global_id 214182 2023-10-30T12:16:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51800a640 con 0x7fe520139290 2023-10-30T12:16:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 10 monclient: _renew_subs 2023-10-30T12:16:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe520093c10 con 0x7fe520139290 2023-10-30T12:16:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51800a9f0 con 0x7fe520139290 2023-10-30T12:16:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe512ffd640 10 monclient: dump: 2023-10-30T12:16:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.686 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe524d5e640 4 set_mon_vals no callback set 2023-10-30T12:16:47.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 10 client.214182.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:47.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 10 monclient: _renew_subs 2023-10-30T12:16:47.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:47.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5201cb130 con 0x7fe520139290 2023-10-30T12:16:47.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.644+0000 7fe5287c5640 1 librados: init done 2023-10-30T12:16:47.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe5180070d0 con 0x7fe520139290 2023-10-30T12:16:47.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe5287c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe5287c5640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520070940 0x7fe520070d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe5287c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe520097df0 con 0x7fe520070940 2023-10-30T12:16:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe504039320 0x7fe50403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520070940 0x7fe520070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(505..505 src has 417..505) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fe51804a610 con 0x7fe520139290 2023-10-30T12:16:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 10 client.214182.objecter ms_dispatch 0x7fe520000b90 osd_map(505..505 src has 417..505) v4 2023-10-30T12:16:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 3 client.214182.objecter handle_osd_map got epochs [505,505] > 0 2023-10-30T12:16:47.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient: get_auth_request con 0x7fe520070940 auth_method 0 2023-10-30T12:16:47.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 3 client.214182.objecter handle_osd_map decoding full epoch 505 2023-10-30T12:16:47.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 20 client.214182.objecter dump_active .. 0 homeless 2023-10-30T12:16:47.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 10 monclient(hunting): handle_auth_done global_id 115199 payload 824 2023-10-30T12:16:47.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 1 -- 172.21.15.50:0/1045507230 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe52008f960 con 0x7fe520070940 2023-10-30T12:16:47.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe525d60640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe504039320 0x7fe50403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe525d60640 10 monclient: get_auth_request con 0x7fe504039320 auth_method 0 2023-10-30T12:16:47.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 10 client.214182.objecter ms_handle_connect 0x7fe520070940 2023-10-30T12:16:47.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe526561640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520070940 0x7fe520070d20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe51400f770 tx=0x7fe51400fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 10 client.214182.objecter resend_mon_ops 2023-10-30T12:16:47.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514003a50 con 0x7fe520070940 2023-10-30T12:16:47.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:47.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.645+0000 7fe512ffd640 10 monclient: dump: 2023-10-30T12:16:47.692 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.692 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.693 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.693 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.693 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.693 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.693 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.694 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.694 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.694 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 10 client.214182.objecter ms_handle_connect 0x7fe504039320 2023-10-30T12:16:47.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe525d60640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe504039320 0x7fe50403b7e0 secure :-1 s=READY pgs=4233 cs=0 l=1 rev1=1 crypto rx=0x7fe51c036000 tx=0x7fe51c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fe514003bb0 con 0x7fe520070940 2023-10-30T12:16:47.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:47.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520070940 msgr2=0x7fe520070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520070940 0x7fe520070d20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe51400f770 tx=0x7fe51400fc30 comp rx=0 tx=0).stop 2023-10-30T12:16:47.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520070940 msgr2=0x7fe520070d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:47.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.646+0000 7fe512ffd640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520070940 0x7fe520070d20 secure :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe51400f770 tx=0x7fe51400fc30 comp rx=0 tx=0).stop 2023-10-30T12:16:47.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe5287c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:16:47.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe5287c5640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520071260 0x7fe52006cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe5287c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe520073ff0 con 0x7fe520071260 2023-10-30T12:16:47.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520071260 0x7fe52006cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient: get_auth_request con 0x7fe520071260 auth_method 0 2023-10-30T12:16:47.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 10 monclient(hunting): handle_auth_done global_id 115202 payload 824 2023-10-30T12:16:47.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.652+0000 7fe526561640 1 -- 172.21.15.50:0/1045507230 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe52009ba70 con 0x7fe520071260 2023-10-30T12:16:47.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 10 client.214182.objecter ms_handle_connect 0x7fe520071260 2023-10-30T12:16:47.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe526561640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520071260 0x7fe52006cbb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe51400f770 tx=0x7fe5140037b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 10 client.214182.objecter resend_mon_ops 2023-10-30T12:16:47.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514023070 con 0x7fe520071260 2023-10-30T12:16:47.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:47.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 10 monclient: dump: 2023-10-30T12:16:47.702 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.702 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.702 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.702 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.703 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.703 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.703 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7fe514004840 con 0x7fe520071260 2023-10-30T12:16:47.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:16:47.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:47.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520071260 msgr2=0x7fe52006cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520071260 0x7fe52006cbb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe51400f770 tx=0x7fe5140037b0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520071260 msgr2=0x7fe52006cbb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:47.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe512ffd640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe520071260 0x7fe52006cbb0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.705 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:16:47.705 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:47.706 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:47.706 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:16:47.706 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 170, 2023-10-30T12:16:47.706 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:16:47.706 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:16:47.707 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:16:47.707 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:16:47.707 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.707 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:16:47.707 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:47.709 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:16:47.709 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:16:47.709 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.709 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.710 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.710 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.710 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.710 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:16:47.712 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:16:47.712 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.712 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.712 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.712 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.713 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.713 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.713 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.713 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.713 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.714 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:16:47.714 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:16:47.714 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:16:47.715 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:16:47.715 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:16:47.715 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:16:47.715 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:47.715 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:47.715 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:16:47.716 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:16:47.716 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:16:47.716 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:16:47.716 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:16:47.716 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:16:47.717 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:16:47.717 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:47.717 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:16:47.717 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.717 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.717 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.718 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.718 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.718 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.718 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.719 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.719 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.719 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.719 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:16:47.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.720 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:16:47.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.720 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:47.720 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:47.720 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.720 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.721 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.721 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:16:47.721 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.722 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:16:47.722 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.723 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.723 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:16:47.723 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:16:47.723 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.724 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.724 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:47.724 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:47.725 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.725 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.725 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.725 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:16:47.726 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.726 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.726 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:16:47.727 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.727 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.727 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:16:47.727 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:16:47.728 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.728 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.728 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:47.729 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:47.729 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.729 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.729 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.730 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:16:47.730 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.730 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.731 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:16:47.731 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.731 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.732 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:16:47.732 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:16:47.732 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.732 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.732 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.733 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.733 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:16:47.733 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:16:47.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.734 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.734 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.734 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:47.734 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.734 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.735 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:16:47.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.735 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.735 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.735 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:16:47.735 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:16:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:16:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:16:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.739 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:16:47.739 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:16:47.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 10 librados: watch_flush enter 2023-10-30T12:16:47.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 10 librados: watch_flush exit 2023-10-30T12:16:47.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 20 client.214182.objecter shutdown clearing up homeless session... 2023-10-30T12:16:47.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 10 client.214182.objecter successfully canceled tick 2023-10-30T12:16:47.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe504039320 msgr2=0x7fe50403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe504039320 0x7fe50403b7e0 secure :-1 s=READY pgs=4233 cs=0 l=1 rev1=1 crypto rx=0x7fe51c036000 tx=0x7fe51c008030 comp rx=0 tx=0).stop 2023-10-30T12:16:47.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 10 monclient: shutdown 2023-10-30T12:16:47.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520139290 msgr2=0x7fe520070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520139290 0x7fe520070400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c6d0 tx=0x7fe51800cb90 comp rx=0 tx=0).stop 2023-10-30T12:16:47.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe526d62640 1 -- 172.21.15.50:0/1045507230 reap_dead start 2023-10-30T12:16:47.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 shutdown_connections 2023-10-30T12:16:47.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 --2- 172.21.15.50:0/1045507230 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe520139290 0x7fe520070400 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 >> 172.21.15.50:0/1045507230 conn(0x7fe5200a3980 msgr2=0x7fe52009a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 shutdown_connections 2023-10-30T12:16:47.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.653+0000 7fe5287c5640 1 -- 172.21.15.50:0/1045507230 wait complete. 2023-10-30T12:16:47.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.654+0000 7fe5287c5640 1 librados: shutdown 2023-10-30T12:16:47.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:16:47.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:16:47.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:16:47.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.852+0000 7f2d6fc75640 10 monclient: get_monmap_and_config 2023-10-30T12:16:47.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.852+0000 7f2d6fc75640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.852+0000 7f2d6fc75640 10 monclient: monmap: 2023-10-30T12:16:47.854 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.855 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.855 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:47.853605+0000 2023-10-30T12:16:47.855 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:47.853605+0000 2023-10-30T12:16:47.855 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.855 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.855 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:47.856 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:47.856 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:47.856 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:47.856 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:47.856 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 1 Processor -- start 2023-10-30T12:16:47.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 1 -- start start 2023-10-30T12:16:47.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient: init 2023-10-30T12:16:47.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-10-30T12:16:47.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d680a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:47.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2450 0x7f2d6813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient(hunting): picked mon.noname-b con 0x7f2d680a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:47.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2d6813af40 0x7f2d6813d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6813af40 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:47.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680a4190 con 0x7f2d680a2ae0 2023-10-30T12:16:47.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d680a1630 con 0x7f2d6813af40 2023-10-30T12:16:47.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6809a2e0 con 0x7f2d680a2450 2023-10-30T12:16:47.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.853+0000 7f2d6fc75640 10 monclient(hunting): authenticate will time out at 1853.036969s 2023-10-30T12:16:47.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2450 0x7f2d6813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2d6813af40 0x7f2d6813d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2450 0x7f2d6813aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55242/0 (socket says 172.21.15.50:55242) 2023-10-30T12:16:47.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 1 -- 172.21.15.50:0/278066273 learned_addr learned my addr 172.21.15.50:0/278066273 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:47.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 10 monclient(hunting): get_auth_request con 0x7f2d680a2450 auth_method 0 2023-10-30T12:16:47.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): get_auth_request con 0x7f2d6813af40 auth_method 0 2023-10-30T12:16:47.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient(hunting): handle_auth_done global_id 214185 payload 274 2023-10-30T12:16:47.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 1 -- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2450 msgr2=0x7f2d6813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:47.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 1 --2- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2450 0x7f2d6813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 1 -- 172.21.15.50:0/278066273 >> v1:172.21.15.118:6789/0 conn(0x7f2d680a2ae0 legacy=0x7f2d680a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 1 monclient: found mon.noname-a 2023-10-30T12:16:47.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:47.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6ca0f640 1 -- 172.21.15.50:0/278066273 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377250330 0 0) 0x7f2d64001110 con 0x7f2d680a2ae0 2023-10-30T12:16:47.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6e212640 1 -- 172.21.15.50:0/278066273 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6809a8a0 con 0x7f2d6813af40 2023-10-30T12:16:47.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6d210640 1 --2- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2450 0x7f2d6813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:16:47.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.854+0000 7f2d6ca0f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:16:47.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.855+0000 7f2d6e212640 1 --2- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d6813af40 0x7f2d6813d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009d90 tx=0x7f2d6002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06bb22e1b52d40b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.855+0000 7f2d6ca0f640 1 -- 172.21.15.50:0/278066273 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6002cb30 con 0x7f2d6813af40 2023-10-30T12:16:47.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.855+0000 7f2d6ca0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:47.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: dump: 2023-10-30T12:16:47.875 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.875 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.875 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 1 -- 172.21.15.50:0/278066273 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6002cc90 con 0x7f2d6813af40 2023-10-30T12:16:47.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 1 -- 172.21.15.50:0/278066273 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60034440 con 0x7f2d6813af40 2023-10-30T12:16:47.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6fc75640 5 monclient: authenticate success, global_id 214185 2023-10-30T12:16:47.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6fc75640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:47.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6fc75640 4 set_mon_vals no callback set 2023-10-30T12:16:47.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:47.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.856+0000 7f2d6ca0f640 10 monclient: dump: 2023-10-30T12:16:47.879 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.879 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.879 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.880 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.880 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.880 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.880 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.880 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.881 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.881 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 10 monclient: shutdown 2023-10-30T12:16:47.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 -- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d6813af40 msgr2=0x7f2d6813d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d6813af40 0x7f2d6813d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009d90 tx=0x7f2d6002bd80 comp rx=0 tx=0).stop 2023-10-30T12:16:47.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 -- 172.21.15.50:0/278066273 shutdown_connections 2023-10-30T12:16:47.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d6813af40 0x7f2d6813d330 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/278066273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2450 0x7f2d6813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 -- 172.21.15.50:0/278066273 >> 172.21.15.50:0/278066273 conn(0x7f2d680a80b0 msgr2=0x7f2d680a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 -- 172.21.15.50:0/278066273 shutdown_connections 2023-10-30T12:16:47.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 1 -- 172.21.15.50:0/278066273 wait complete. 2023-10-30T12:16:47.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 10 monclient: build_initial_monmap 2023-10-30T12:16:47.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.857+0000 7f2d6fc75640 10 monclient: monmap: 2023-10-30T12:16:47.883 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:47.883 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:47.884 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:47.884 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:47.884 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:47.884 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.884 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:47.884 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:47.885 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:47.885 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 librados: starting msgr at 2023-10-30T12:16:47.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 librados: starting objecter 2023-10-30T12:16:47.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 Processor -- start 2023-10-30T12:16:47.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 -- start start 2023-10-30T12:16:47.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 librados: setting wanted keys 2023-10-30T12:16:47.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 librados: calling monclient init 2023-10-30T12:16:47.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient: init 2023-10-30T12:16:47.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:47.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:16:47.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d680a2450 0x7f2d681ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): picked mon.noname-a con 0x7f2d680a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:47.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2ae0 0x7f2d68136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): picked mon.noname-c con 0x7f2d680a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:47.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d6813af40 0x7f2d68137310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): picked mon.noname-b con 0x7f2d6813af40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:47.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68139000 con 0x7f2d680a2450 2023-10-30T12:16:47.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68139610 con 0x7f2d6813af40 2023-10-30T12:16:47.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680a1790 con 0x7f2d680a2ae0 2023-10-30T12:16:47.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:47.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.858+0000 7f2d6fc75640 10 monclient(hunting): authenticate will time out at 1853.041930s 2023-10-30T12:16:47.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2ae0 0x7f2d68136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d680a2450 0x7f2d681ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2ae0 0x7f2d68136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55252/0 (socket says 172.21.15.50:55252) 2023-10-30T12:16:47.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 -- 172.21.15.50:0/2768470 learned_addr learned my addr 172.21.15.50:0/2768470 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:47.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 10 monclient(hunting): get_auth_request con 0x7f2d680a2450 auth_method 0 2023-10-30T12:16:47.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): get_auth_request con 0x7f2d680a2ae0 auth_method 0 2023-10-30T12:16:47.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d6813af40 0x7f2d68137310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 10 monclient(hunting): get_auth_request con 0x7f2d6813af40 auth_method 0 2023-10-30T12:16:47.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6e212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient(hunting): handle_auth_done global_id 115208 payload 824 2023-10-30T12:16:47.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d6813af40 msgr2=0x7f2d68137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d6813af40 0x7f2d68137310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d680a2450 msgr2=0x7f2d681ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d680a2450 0x7f2d681ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient: _finish_hunting 0 2023-10-30T12:16:47.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 monclient: found mon.noname-c 2023-10-30T12:16:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 -- 172.21.15.50:0/2768470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580094d0 con 0x7f2d680a2ae0 2023-10-30T12:16:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6d210640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2ae0 0x7f2d68136dd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800ea00 tx=0x7f2d5800eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 10 client.?.objecter ms_handle_connect 0x7f2d680a2ae0 2023-10-30T12:16:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:47.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800ca60 con 0x7f2d680a2ae0 2023-10-30T12:16:47.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:47.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 10 monclient: dump: 2023-10-30T12:16:47.897 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.898 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.898 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.898 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.898 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.898 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.898 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.899 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.899 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.899 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:16:47.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:16:47.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:47.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5800cbc0 con 0x7f2d680a2ae0 2023-10-30T12:16:47.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6fc75640 5 monclient: authenticate success, global_id 115208 2023-10-30T12:16:47.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6fc75640 10 monclient: _renew_subs 2023-10-30T12:16:47.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6fc75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.859+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d60009a70 con 0x7f2d680a2ae0 2023-10-30T12:16:47.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d5e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:47.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58013470 con 0x7f2d680a2ae0 2023-10-30T12:16:47.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:16:47.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d5e7fc640 10 monclient: dump: 2023-10-30T12:16:47.902 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.902 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.902 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.902 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.902 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.902 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.903 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.903 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.903 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.903 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d5ffff640 4 set_mon_vals no callback set 2023-10-30T12:16:47.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 10 client.115208.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:47.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 10 monclient: _renew_subs 2023-10-30T12:16:47.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:16:47.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6811e800 con 0x7f2d680a2ae0 2023-10-30T12:16:47.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 1 librados: init done 2023-10-30T12:16:47.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d681cac90 0x7f2d68137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.860+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6811e300 con 0x7f2d681cac90 2023-10-30T12:16:47.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d681cac90 0x7f2d68137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient: get_auth_request con 0x7f2d681cac90 auth_method 0 2023-10-30T12:16:47.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2d580135d0 con 0x7f2d680a2ae0 2023-10-30T12:16:47.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2d4c0392f0 0x7f2d4c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(505..505 src has 417..505) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f2d5804b0a0 con 0x7f2d680a2ae0 2023-10-30T12:16:47.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 10 client.115208.objecter ms_dispatch 0x7f2d68000b90 osd_map(505..505 src has 417..505) v4 2023-10-30T12:16:47.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 3 client.115208.objecter handle_osd_map got epochs [505,505] > 0 2023-10-30T12:16:47.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 3 client.115208.objecter handle_osd_map decoding full epoch 505 2023-10-30T12:16:47.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 10 monclient(hunting): handle_auth_done global_id 204280 payload 824 2023-10-30T12:16:47.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 1 -- 172.21.15.50:0/2768470 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d6809a5c0 con 0x7f2d681cac90 2023-10-30T12:16:47.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 20 client.115208.objecter dump_active .. 0 homeless 2023-10-30T12:16:47.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6e212640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2d4c0392f0 0x7f2d4c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6e212640 10 monclient: get_auth_request con 0x7f2d4c0392f0 auth_method 0 2023-10-30T12:16:47.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 10 client.115208.objecter ms_handle_connect 0x7f2d681cac90 2023-10-30T12:16:47.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d5e7fc640 10 client.115208.objecter resend_mon_ops 2023-10-30T12:16:47.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.861+0000 7f2d6da11640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d681cac90 0x7f2d68137850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d64001370 tx=0x7f2d6400fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d640050d0 con 0x7f2d681cac90 2023-10-30T12:16:47.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:16:47.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 10 monclient: dump: 2023-10-30T12:16:47.912 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.912 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.912 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.912 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.912 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.913 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 10 client.115208.objecter ms_handle_connect 0x7f2d4c0392f0 2023-10-30T12:16:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d6e212640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2d4c0392f0 0x7f2d4c03b7b0 secure :-1 s=READY pgs=4234 cs=0 l=1 rev1=1 crypto rx=0x7f2d60036000 tx=0x7f2d60030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f2d64005230 con 0x7f2d681cac90 2023-10-30T12:16:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d681cac90 msgr2=0x7f2d68137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d681cac90 0x7f2d68137850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d64001370 tx=0x7f2d6400fc00 comp rx=0 tx=0).stop 2023-10-30T12:16:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d681cac90 msgr2=0x7f2d68137850 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.862+0000 7f2d5e7fc640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d681cac90 0x7f2d68137850 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2d64001370 tx=0x7f2d6400fc00 comp rx=0 tx=0).stop 2023-10-30T12:16:47.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6fc75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:16:47.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d68137d90 0x7f2d68133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:47.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6fc75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:47.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68073ff0 con 0x7f2d68137d90 2023-10-30T12:16:47.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6da11640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d68137d90 0x7f2d68133290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:47.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6da11640 10 monclient: get_auth_request con 0x7f2d68137d90 auth_method 0 2023-10-30T12:16:47.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6da11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:47.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6da11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.868+0000 7f2d6da11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d6da11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d6da11640 10 monclient(hunting): handle_auth_done global_id 204283 payload 824 2023-10-30T12:16:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d6da11640 1 -- 172.21.15.50:0/2768470 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2d68099b70 con 0x7f2d68137d90 2023-10-30T12:16:47.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d6da11640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d68137d90 0x7f2d68133290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d64000fb0 tx=0x7f2d64001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:47.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 10 client.115208.objecter ms_handle_connect 0x7f2d68137d90 2023-10-30T12:16:47.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 10 client.115208.objecter resend_mon_ops 2023-10-30T12:16:47.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64024070 con 0x7f2d68137d90 2023-10-30T12:16:47.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:47.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:16:47.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 10 monclient: dump: 2023-10-30T12:16:47.921 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:47.921 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:47.921 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.921 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:47.923 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:47.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f2d64003b50 con 0x7f2d68137d90 2023-10-30T12:16:47.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:16:47.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:47.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d68137d90 msgr2=0x7f2d68133290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d68137d90 0x7f2d68133290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d64000fb0 tx=0x7f2d64001080 comp rx=0 tx=0).stop 2023-10-30T12:16:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d68137d90 msgr2=0x7f2d68133290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:16:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.869+0000 7f2d5e7fc640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d68137d90 0x7f2d68133290 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.924 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:16:47.924 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:47.925 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:47.925 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:16:47.925 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 170, 2023-10-30T12:16:47.925 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:16:47.925 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:16:47.926 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:16:47.926 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:16:47.926 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.926 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:16:47.926 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:47.927 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:16:47.927 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:16:47.927 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.927 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.927 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.927 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.928 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.928 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.928 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.928 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.928 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.929 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.929 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.929 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:16:47.929 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:16:47.929 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.929 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.930 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.930 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.930 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.930 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.930 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.931 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.931 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.931 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.931 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.932 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:16:47.932 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:16:47.932 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:16:47.932 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:16:47.932 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:16:47.932 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:16:47.933 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:47.933 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:16:47.933 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:16:47.933 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:16:47.933 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:16:47.934 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:16:47.934 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:16:47.934 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:16:47.934 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:16:47.934 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:16:47.934 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:16:47.935 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:16:47.935 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:16:47.935 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:16:47.935 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:16:47.935 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:16:47.936 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:16:47.936 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:16:47.936 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:16:47.936 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:16:47.936 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:16:47.936 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.937 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:16:47.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:16:47.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.937 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:16:47.938 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:16:47.938 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.938 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.938 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:16:47.939 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.939 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:16:47.940 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.940 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.940 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:16:47.940 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:16:47.941 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.941 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.941 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.942 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:16:47.942 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:16:47.942 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.942 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.942 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.943 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:16:47.943 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.943 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.944 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:16:47.944 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.944 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.944 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:16:47.945 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:16:47.945 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.945 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.945 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.946 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:16:47.946 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:16:47.946 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:16:47.946 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:16:47.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:16:47.947 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:16:47.947 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.948 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:16:47.948 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:16:47.948 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:16:47.948 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.948 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.949 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:16:47.949 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:16:47.949 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:16:47.949 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:16:47.949 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:16:47.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.950 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.950 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:16:47.950 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:16:47.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.951 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.951 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.951 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:47.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.951 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.952 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:16:47.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.952 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.952 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.952 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:16:47.952 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.953 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:16:47.953 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:16:47.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:16:47.953 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:16:47.953 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:16:47.954 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:16:47.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:16:47.954 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:16:47.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:16:47.954 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:16:47.954 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:16:47.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 10 librados: watch_flush enter 2023-10-30T12:16:47.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 10 librados: watch_flush exit 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 20 client.115208.objecter shutdown clearing up homeless session... 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 10 client.115208.objecter successfully canceled tick 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2d4c0392f0 msgr2=0x7f2d4c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2d4c0392f0 0x7f2d4c03b7b0 secure :-1 s=READY pgs=4234 cs=0 l=1 rev1=1 crypto rx=0x7f2d60036000 tx=0x7f2d60030000 comp rx=0 tx=0).stop 2023-10-30T12:16:47.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 10 monclient: shutdown 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2ae0 msgr2=0x7f2d68136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6e212640 1 -- 172.21.15.50:0/2768470 reap_dead start 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2ae0 0x7f2d68136dd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800ea00 tx=0x7f2d5800eec0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 shutdown_connections 2023-10-30T12:16:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 --2- 172.21.15.50:0/2768470 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2d680a2ae0 0x7f2d68136dd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:47.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 >> 172.21.15.50:0/2768470 conn(0x7f2d680a80b0 msgr2=0x7f2d6809dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:47.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 shutdown_connections 2023-10-30T12:16:47.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 -- 172.21.15.50:0/2768470 wait complete. 2023-10-30T12:16:47.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:47.870+0000 7f2d6fc75640 1 librados: shutdown 2023-10-30T12:16:47.960 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:16:47.960 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-10-30T12:16:47.960 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:16:47.960 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-10-30T12:16:47.961 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:16:47.961 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:16:47.961 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:16:47.978 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:16:47.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:16:47.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:48.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.121+0000 7fab2af16640 10 monclient: get_monmap_and_config 2023-10-30T12:16:48.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.121+0000 7fab2af16640 10 monclient: build_initial_monmap 2023-10-30T12:16:48.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.121+0000 7fab2af16640 10 monclient: monmap: 2023-10-30T12:16:48.123 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:48.123 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:48.123 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:48.122227+0000 2023-10-30T12:16:48.123 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:48.122227+0000 2023-10-30T12:16:48.124 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:48.124 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:48.124 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:48.124 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:48.124 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:48.124 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:48.125 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:48.125 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.121+0000 7fab2af16640 1 Processor -- start 2023-10-30T12:16:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.121+0000 7fab2af16640 1 -- start start 2023-10-30T12:16:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.121+0000 7fab2af16640 10 monclient: init 2023-10-30T12:16:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T12:16:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fab240a4b60 0x7fab241326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient(hunting): picked mon.noname-c con 0x7fab240a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:48.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab24099a40 0x7fab24132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:48.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient(hunting): picked mon.noname-a con 0x7fab24099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:48.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab240a2350 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:48.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab24133120 con 0x7fab240a2350 2023-10-30T12:16:48.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:48.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab240a3d70 con 0x7fab24099a40 2023-10-30T12:16:48.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:48.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab240a4f40 con 0x7fab240a4b60 2023-10-30T12:16:48.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:48.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab2af16640 10 monclient(hunting): authenticate will time out at 1853.305433s 2023-10-30T12:16:48.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab24099a40 0x7fab24132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:48.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab24099a40 0x7fab24132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48558/0 (socket says 172.21.15.50:48558) 2023-10-30T12:16:48.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 -- 172.21.15.50:0/2055795114 learned_addr learned my addr 172.21.15.50:0/2055795114 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:48.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): get_auth_request con 0x7fab24099a40 auth_method 0 2023-10-30T12:16:48.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:48.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:48.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:48.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:48.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:48.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:48.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab28cb2640 1 --2- 172.21.15.50:0/2055795114 >> v2:172.21.15.118:3300/0 conn(0x7fab240a4b60 0x7fab241326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:48.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab28cb2640 10 monclient(hunting): get_auth_request con 0x7fab240a4b60 auth_method 0 2023-10-30T12:16:48.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab28cb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:48.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab28cb2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:48.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab28cb2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:48.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient(hunting): handle_auth_done global_id 214191 payload 274 2023-10-30T12:16:48.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 -- 172.21.15.50:0/2055795114 >> v2:172.21.15.118:3300/0 conn(0x7fab240a4b60 msgr2=0x7fab241326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 --2- 172.21.15.50:0/2055795114 >> v2:172.21.15.118:3300/0 conn(0x7fab240a4b60 0x7fab241326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 -- 172.21.15.50:0/2055795114 >> v1:172.21.15.118:6789/0 conn(0x7fab240a2350 legacy=0x7fab240a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient: _finish_hunting 0 2023-10-30T12:16:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 monclient: found mon.noname-a 2023-10-30T12:16:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.122+0000 7fab23fff640 1 -- 172.21.15.50:0/2055795114 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab240a4150 con 0x7fab24099a40 2023-10-30T12:16:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab23fff640 1 --2- 172.21.15.50:0/2055795114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab24099a40 0x7fab24132be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab240a3820 tx=0x7fab1402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94f61890fde0776 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 1 -- 172.21.15.50:0/2055795114 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1402c960 con 0x7fab24099a40 2023-10-30T12:16:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: dump: 2023-10-30T12:16:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:48.141 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: _finish_auth 0 2023-10-30T12:16:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: _check_auth_tickets 2023-10-30T12:16:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 1 -- 172.21.15.50:0/2055795114 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1402cac0 con 0x7fab24099a40 2023-10-30T12:16:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 1 -- 172.21.15.50:0/2055795114 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1402ce50 con 0x7fab24099a40 2023-10-30T12:16:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab2af16640 5 monclient: authenticate success, global_id 214191 2023-10-30T12:16:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab2af16640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab2af16640 4 set_mon_vals no callback set 2023-10-30T12:16:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.123+0000 7fab237fe640 10 monclient: dump: 2023-10-30T12:16:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:48.145 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:48.145 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:48.145 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:48.145 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:48.145 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:48.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 10 monclient: shutdown 2023-10-30T12:16:48.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:48.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 -- 172.21.15.50:0/2055795114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab24099a40 msgr2=0x7fab24132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:48.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 --2- 172.21.15.50:0/2055795114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab24099a40 0x7fab24132be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab240a3820 tx=0x7fab1402bdc0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 -- 172.21.15.50:0/2055795114 shutdown_connections 2023-10-30T12:16:48.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 --2- 172.21.15.50:0/2055795114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab24099a40 0x7fab24132be0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 --2- 172.21.15.50:0/2055795114 >> v2:172.21.15.118:3300/0 conn(0x7fab240a4b60 0x7fab241326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 -- 172.21.15.50:0/2055795114 >> 172.21.15.50:0/2055795114 conn(0x7fab240a8060 msgr2=0x7fab240a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 -- 172.21.15.50:0/2055795114 shutdown_connections 2023-10-30T12:16:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.124+0000 7fab2af16640 1 -- 172.21.15.50:0/2055795114 wait complete. 2023-10-30T12:16:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient: build_initial_monmap 2023-10-30T12:16:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient: monmap: 2023-10-30T12:16:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:48.149 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 librados: starting msgr at 2023-10-30T12:16:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 librados: starting objecter 2023-10-30T12:16:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 Processor -- start 2023-10-30T12:16:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 -- start start 2023-10-30T12:16:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 librados: setting wanted keys 2023-10-30T12:16:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 librados: calling monclient init 2023-10-30T12:16:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient: init 2023-10-30T12:16:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:16:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab240a2350 0x7fab241ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): picked mon.noname-c con 0x7fab240a2350 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab240a4b60 0x7fab241cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): picked mon.noname-a con 0x7fab240a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab2411e6d0 0x7fab2411eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): picked mon.noname-b con 0x7fab2411e6d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab241224f0 con 0x7fab240a4b60 2023-10-30T12:16:48.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:48.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab24122740 con 0x7fab2411e6d0 2023-10-30T12:16:48.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:48.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab24120fb0 con 0x7fab240a2350 2023-10-30T12:16:48.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:48.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.125+0000 7fab2af16640 10 monclient(hunting): authenticate will time out at 1853.309003s 2023-10-30T12:16:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab28cb2640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab240a2350 msgr2=0x7fab241ca8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:16:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab28cb2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab240a2350 0x7fab241ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab240a4b60 0x7fab241cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab240a4b60 0x7fab241cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48564/0 (socket says 172.21.15.50:48564) 2023-10-30T12:16:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 -- 172.21.15.50:0/4085524412 learned_addr learned my addr 172.21.15.50:0/4085524412 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): get_auth_request con 0x7fab240a4b60 auth_method 0 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab294b3640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab2411e6d0 0x7fab2411eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient(hunting): handle_auth_done global_id 214194 payload 824 2023-10-30T12:16:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 -- 172.21.15.50:0/4085524412 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab240a2350 msgr2=0x7fab241ca8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab240a2350 0x7fab241ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 -- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab2411e6d0 msgr2=0x7fab2411eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:48.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab2411e6d0 0x7fab2411eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient: _finish_hunting 0 2023-10-30T12:16:48.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 monclient: found mon.noname-a 2023-10-30T12:16:48.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:48.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:48.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 -- 172.21.15.50:0/4085524412 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14009b30 con 0x7fab240a4b60 2023-10-30T12:16:48.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab294b3640 10 monclient: get_auth_request con 0x7fab2411e6d0 auth_method 0 2023-10-30T12:16:48.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab294b3640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab2411e6d0 0x7fab2411eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:16:48.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab23fff640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab240a4b60 0x7fab241cae00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab14036040 tx=0x7fab14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:48.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 client.?.objecter ms_handle_connect 0x7fab240a4b60 2023-10-30T12:16:48.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:48.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 1 -- 172.21.15.50:0/4085524412 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab14009510 con 0x7fab240a4b60 2023-10-30T12:16:48.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:48.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:48.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: dump: 2023-10-30T12:16:48.160 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:48.160 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:48.160 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.160 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.160 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:48.160 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:48.161 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:48.161 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:48.161 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:48.161 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:48.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: _finish_auth 0 2023-10-30T12:16:48.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: _check_auth_tickets 2023-10-30T12:16:48.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:48.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 1 -- 172.21.15.50:0/4085524412 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1403e450 con 0x7fab240a4b60 2023-10-30T12:16:48.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 5 monclient: authenticate success, global_id 214194 2023-10-30T12:16:48.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 10 monclient: _renew_subs 2023-10-30T12:16:48.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:48.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab240a18d0 con 0x7fab240a4b60 2023-10-30T12:16:48.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:48.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 1 -- 172.21.15.50:0/4085524412 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1403e800 con 0x7fab240a4b60 2023-10-30T12:16:48.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:48.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:48.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab217fa640 10 monclient: dump: 2023-10-30T12:16:48.164 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:48.164 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:48.164 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.164 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:48.164 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:48.165 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:48.165 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:48.165 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:48.165 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:48.165 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:48.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 10 client.214194.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:48.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 10 monclient: _renew_subs 2023-10-30T12:16:48.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab22ffd640 4 set_mon_vals no callback set 2023-10-30T12:16:48.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:48.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab24120370 con 0x7fab240a4b60 2023-10-30T12:16:48.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.126+0000 7fab2af16640 1 librados: init done 2023-10-30T12:16:48.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.127+0000 7fab217fa640 1 -- 172.21.15.50:0/4085524412 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fab1403e960 con 0x7fab240a4b60 2023-10-30T12:16:48.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.127+0000 7fab2af16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:48.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.127+0000 7fab2af16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:48.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.127+0000 7fab2af16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:48.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.127+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab24122880 con 0x7fab240a4b60 2023-10-30T12:16:48.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.127+0000 7fab217fa640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab040392a0 0x7fab0403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:48.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.127+0000 7fab217fa640 1 -- 172.21.15.50:0/4085524412 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(505..505 src has 417..505) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fab1406dc20 con 0x7fab240a4b60 2023-10-30T12:16:48.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab217fa640 10 client.214194.objecter ms_dispatch 0x7fab24000b90 osd_map(505..505 src has 417..505) v4 2023-10-30T12:16:48.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab217fa640 3 client.214194.objecter handle_osd_map got epochs [505,505] > 0 2023-10-30T12:16:48.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab217fa640 3 client.214194.objecter handle_osd_map decoding full epoch 505 2023-10-30T12:16:48.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab217fa640 20 client.214194.objecter dump_active .. 0 homeless 2023-10-30T12:16:48.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab28cb2640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab040392a0 0x7fab0403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:48.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab28cb2640 10 monclient: get_auth_request con 0x7fab040392a0 auth_method 0 2023-10-30T12:16:48.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab217fa640 10 client.214194.objecter ms_handle_connect 0x7fab040392a0 2023-10-30T12:16:48.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.128+0000 7fab28cb2640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab040392a0 0x7fab0403b760 secure :-1 s=READY pgs=4235 cs=0 l=1 rev1=1 crypto rx=0x7fab1c006fd0 tx=0x7fab1c0092e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:48.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.132+0000 7fab217fa640 1 -- 172.21.15.50:0/4085524412 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab24122880 con 0x7fab240a4b60 2023-10-30T12:16:48.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.132+0000 7fab217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:48.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.132+0000 7fab217fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:48.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.373+0000 7fab2af16640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:48.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.373+0000 7fab2af16640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:48.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.374+0000 7fab2af16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:48.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.374+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab24122880 con 0x7fab240a4b60 2023-10-30T12:16:48.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.374+0000 7fab217fa640 1 -- 172.21.15.50:0/4085524412 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fab24122880 con 0x7fab240a4b60 2023-10-30T12:16:48.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.374+0000 7fab217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:48.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.374+0000 7fab217fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:48.377 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:48.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 10 librados: watch_flush enter 2023-10-30T12:16:48.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 10 librados: watch_flush exit 2023-10-30T12:16:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 20 client.214194.objecter shutdown clearing up homeless session... 2023-10-30T12:16:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 10 client.214194.objecter successfully canceled tick 2023-10-30T12:16:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab040392a0 msgr2=0x7fab0403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab040392a0 0x7fab0403b760 secure :-1 s=READY pgs=4235 cs=0 l=1 rev1=1 crypto rx=0x7fab1c006fd0 tx=0x7fab1c0092e0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 10 monclient: shutdown 2023-10-30T12:16:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab240a4b60 msgr2=0x7fab241cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab240a4b60 0x7fab241cae00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab14036040 tx=0x7fab14004a40 comp rx=0 tx=0).stop 2023-10-30T12:16:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 shutdown_connections 2023-10-30T12:16:48.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fab040392a0 0x7fab0403b760 unknown :-1 s=CLOSED pgs=4235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab2411e6d0 0x7fab2411eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab240a4b60 0x7fab241cae00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 --2- 172.21.15.50:0/4085524412 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fab240a2350 0x7fab241ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 >> 172.21.15.50:0/4085524412 conn(0x7fab240a8060 msgr2=0x7fab241332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 shutdown_connections 2023-10-30T12:16:48.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 -- 172.21.15.50:0/4085524412 wait complete. 2023-10-30T12:16:48.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:48.376+0000 7fab2af16640 1 librados: shutdown 2023-10-30T12:16:51.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:51.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.504+0000 7f592ad73640 10 monclient: get_monmap_and_config 2023-10-30T12:16:51.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.504+0000 7f592ad73640 10 monclient: build_initial_monmap 2023-10-30T12:16:51.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.504+0000 7f592ad73640 10 monclient: monmap: 2023-10-30T12:16:51.506 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:51.506 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:51.506 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:51.505232+0000 2023-10-30T12:16:51.507 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:51.505232+0000 2023-10-30T12:16:51.507 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:51.507 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:51.507 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:51.507 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:51.508 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:51.508 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:51.508 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:51.508 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.504+0000 7f592ad73640 1 Processor -- start 2023-10-30T12:16:51.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 1 -- start start 2023-10-30T12:16:51.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient: init 2023-10-30T12:16:51.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:51.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:16:51.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f59240a4b60 0x7f59241326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:51.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient(hunting): picked mon.noname-c con 0x7f59240a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:51.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5924099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:51.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5924132be0 0x7f59240a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:51.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient(hunting): picked mon.noname-a con 0x7f5924132be0 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:51.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240a4f40 con 0x7f5924099a40 2023-10-30T12:16:51.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:51.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59240a3d70 con 0x7f5924132be0 2023-10-30T12:16:51.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:51.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59240a4150 con 0x7f59240a4b60 2023-10-30T12:16:51.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:51.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f592ad73640 10 monclient(hunting): authenticate will time out at 1856.688580s 2023-10-30T12:16:51.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5924132be0 0x7f59240a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:51.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5924132be0 0x7f59240a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48572/0 (socket says 172.21.15.50:48572) 2023-10-30T12:16:51.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 1 -- 172.21.15.50:0/2457822197 learned_addr learned my addr 172.21.15.50:0/2457822197 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:51.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 10 monclient(hunting): get_auth_request con 0x7f5924132be0 auth_method 0 2023-10-30T12:16:51.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:51.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:51.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:51.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:51.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:51.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5929310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:51.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5928b0f640 1 --2- 172.21.15.50:0/2457822197 >> v2:172.21.15.118:3300/0 conn(0x7f59240a4b60 0x7f59241326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:51.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5928b0f640 10 monclient(hunting): get_auth_request con 0x7f59240a4b60 auth_method 0 2023-10-30T12:16:51.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5928b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:51.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5928b0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:51.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.505+0000 7f5928b0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:51.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 10 monclient(hunting): handle_auth_done global_id 214197 payload 274 2023-10-30T12:16:51.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 1 -- 172.21.15.50:0/2457822197 >> v2:172.21.15.118:3300/0 conn(0x7f59240a4b60 msgr2=0x7f59241326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:51.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 1 --2- 172.21.15.50:0/2457822197 >> v2:172.21.15.118:3300/0 conn(0x7f59240a4b60 0x7f59241326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 1 -- 172.21.15.50:0/2457822197 >> v1:172.21.15.118:6789/0 conn(0x7f5924099a40 legacy=0x7f59240a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:16:51.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 10 monclient: _finish_hunting 0 2023-10-30T12:16:51.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 1 monclient: found mon.noname-a 2023-10-30T12:16:51.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:51.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:51.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 1 -- 172.21.15.50:0/2457822197 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592409a440 con 0x7f5924132be0 2023-10-30T12:16:51.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.506+0000 7f5929310640 1 --2- 172.21.15.50:0/2457822197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5924132be0 0x7f59240a2a60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59240a3820 tx=0x7f591c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3406cb6e841153d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:51.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 1 -- 172.21.15.50:0/2457822197 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c02c960 con 0x7f5924132be0 2023-10-30T12:16:51.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:51.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:51.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: dump: 2023-10-30T12:16:51.530 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:51.530 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:51.530 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:51.532 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: _finish_auth 0 2023-10-30T12:16:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: _check_auth_tickets 2023-10-30T12:16:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:51.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 1 -- 172.21.15.50:0/2457822197 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c02cac0 con 0x7f5924132be0 2023-10-30T12:16:51.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 5 monclient: authenticate success, global_id 214197 2023-10-30T12:16:51.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:51.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:51.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:51.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 1 -- 172.21.15.50:0/2457822197 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c02ce50 con 0x7f5924132be0 2023-10-30T12:16:51.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:51.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 4 set_mon_vals no callback set 2023-10-30T12:16:51.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:51.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f59237fe640 10 monclient: dump: 2023-10-30T12:16:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:51.536 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 10 monclient: shutdown 2023-10-30T12:16:51.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:51.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 -- 172.21.15.50:0/2457822197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5924132be0 msgr2=0x7f59240a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:51.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 --2- 172.21.15.50:0/2457822197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5924132be0 0x7f59240a2a60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59240a3820 tx=0x7f591c02bdc0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 -- 172.21.15.50:0/2457822197 shutdown_connections 2023-10-30T12:16:51.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 --2- 172.21.15.50:0/2457822197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5924132be0 0x7f59240a2a60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 --2- 172.21.15.50:0/2457822197 >> v2:172.21.15.118:3300/0 conn(0x7f59240a4b60 0x7f59241326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 -- 172.21.15.50:0/2457822197 >> 172.21.15.50:0/2457822197 conn(0x7f59240a8060 msgr2=0x7f59240a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:51.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 -- 172.21.15.50:0/2457822197 shutdown_connections 2023-10-30T12:16:51.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.507+0000 7f592ad73640 1 -- 172.21.15.50:0/2457822197 wait complete. 2023-10-30T12:16:51.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 10 monclient: build_initial_monmap 2023-10-30T12:16:51.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 10 monclient: monmap: 2023-10-30T12:16:51.539 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:51.539 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:51.540 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:51.540 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:51.540 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:51.540 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:51.541 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:51.541 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:51.541 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:51.541 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:51.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 1 librados: starting msgr at 2023-10-30T12:16:51.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 1 librados: starting objecter 2023-10-30T12:16:51.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 1 Processor -- start 2023-10-30T12:16:51.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 1 -- start start 2023-10-30T12:16:51.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 1 librados: setting wanted keys 2023-10-30T12:16:51.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 1 librados: calling monclient init 2023-10-30T12:16:51.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.508+0000 7f592ad73640 10 monclient: init 2023-10-30T12:16:51.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:51.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:16:51.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59240a4b60 0x7f59241ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:51.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): picked mon.noname-c con 0x7f59240a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:51.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5924132be0 0x7f59241cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:51.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): picked mon.noname-b con 0x7f5924132be0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:51.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592411e440 0x7f592411e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:51.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): picked mon.noname-a con 0x7f592411e440 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:51.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:51.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924121ca0 con 0x7f592411e440 2023-10-30T12:16:51.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:51.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59241221d0 con 0x7f5924132be0 2023-10-30T12:16:51.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:51.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59241224b0 con 0x7f59240a4b60 2023-10-30T12:16:51.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:51.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f592ad73640 10 monclient(hunting): authenticate will time out at 1856.692413s 2023-10-30T12:16:51.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5928b0f640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59240a4b60 msgr2=0x7f59241ca8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:16:51.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5928b0f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59240a4b60 0x7f59241ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:51.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592411e440 0x7f592411e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:51.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592411e440 0x7f592411e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48574/0 (socket says 172.21.15.50:48574) 2023-10-30T12:16:51.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 -- 172.21.15.50:0/1038984640 learned_addr learned my addr 172.21.15.50:0/1038984640 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:51.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): get_auth_request con 0x7f592411e440 auth_method 0 2023-10-30T12:16:51.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:51.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:51.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:51.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:51.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:51.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:51.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5923fff640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5924132be0 0x7f59241cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:51.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient(hunting): handle_auth_done global_id 214200 payload 824 2023-10-30T12:16:51.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 -- 172.21.15.50:0/1038984640 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59240a4b60 msgr2=0x7f59241ca8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:51.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59240a4b60 0x7f59241ca8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 -- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5924132be0 msgr2=0x7f59241cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:51.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5924132be0 0x7f59241cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient: _finish_hunting 0 2023-10-30T12:16:51.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 monclient: found mon.noname-a 2023-10-30T12:16:51.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:51.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:51.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5929310640 1 -- 172.21.15.50:0/1038984640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c009b30 con 0x7f592411e440 2023-10-30T12:16:51.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5923fff640 10 monclient: get_auth_request con 0x7f5924132be0 auth_method 0 2023-10-30T12:16:51.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.509+0000 7f5923fff640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5924132be0 0x7f59241cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:16:51.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f5929310640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592411e440 0x7f592411e8f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f591c036040 tx=0x7f591c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:51.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 client.?.objecter ms_handle_connect 0x7f592411e440 2023-10-30T12:16:51.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:51.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 1 -- 172.21.15.50:0/1038984640 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c009510 con 0x7f592411e440 2023-10-30T12:16:51.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:51.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: dump: 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.555 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:51.556 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:51.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: _finish_auth 0 2023-10-30T12:16:51.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: _check_auth_tickets 2023-10-30T12:16:51.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:51.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 5 monclient: authenticate success, global_id 214200 2023-10-30T12:16:51.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 1 -- 172.21.15.50:0/1038984640 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c03e450 con 0x7f592411e440 2023-10-30T12:16:51.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:51.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 1 -- 172.21.15.50:0/1038984640 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c03e800 con 0x7f592411e440 2023-10-30T12:16:51.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 10 monclient: _renew_subs 2023-10-30T12:16:51.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:51.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59240a18d0 con 0x7f592411e440 2023-10-30T12:16:51.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f5922ffd640 4 set_mon_vals no callback set 2023-10-30T12:16:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f59217fa640 10 monclient: dump: 2023-10-30T12:16:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:51.560 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.560 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:51.560 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:51.560 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:51.560 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:51.560 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:51.561 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:51.561 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:51.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 10 client.214200.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:51.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 10 monclient: _renew_subs 2023-10-30T12:16:51.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:51.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5924131f40 con 0x7f592411e440 2023-10-30T12:16:51.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.510+0000 7f592ad73640 1 librados: init done 2023-10-30T12:16:51.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f59217fa640 1 -- 172.21.15.50:0/1038984640 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f591c03e960 con 0x7f592411e440 2023-10-30T12:16:51.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f592ad73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:51.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f592ad73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:51.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f592ad73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:51.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f59217fa640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59080392a0 0x7f590803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:51.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592404e160 con 0x7f592411e440 2023-10-30T12:16:51.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f59217fa640 1 -- 172.21.15.50:0/1038984640 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(505..505 src has 417..505) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f591c06d930 con 0x7f592411e440 2023-10-30T12:16:51.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f59217fa640 10 client.214200.objecter ms_dispatch 0x7f5924000b90 osd_map(505..505 src has 417..505) v4 2023-10-30T12:16:51.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f59217fa640 3 client.214200.objecter handle_osd_map got epochs [505,505] > 0 2023-10-30T12:16:51.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f59217fa640 3 client.214200.objecter handle_osd_map decoding full epoch 505 2023-10-30T12:16:51.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f59217fa640 20 client.214200.objecter dump_active .. 0 homeless 2023-10-30T12:16:51.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f5928b0f640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59080392a0 0x7f590803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:51.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.511+0000 7f5928b0f640 10 monclient: get_auth_request con 0x7f59080392a0 auth_method 0 2023-10-30T12:16:51.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.512+0000 7f59217fa640 10 client.214200.objecter ms_handle_connect 0x7f59080392a0 2023-10-30T12:16:51.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.512+0000 7f5928b0f640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59080392a0 0x7f590803b760 secure :-1 s=READY pgs=4236 cs=0 l=1 rev1=1 crypto rx=0x7f5918006fd0 tx=0x7f59180092e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:51.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.516+0000 7f59217fa640 1 -- 172.21.15.50:0/1038984640 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f591c047d80 con 0x7f592411e440 2023-10-30T12:16:51.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.516+0000 7f59217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:51.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.516+0000 7f59217fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:51.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.760+0000 7f592ad73640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:51.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.760+0000 7f592ad73640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:51.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.760+0000 7f592ad73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.760+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f59240745c0 con 0x7f592411e440 2023-10-30T12:16:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.760+0000 7f59217fa640 1 -- 172.21.15.50:0/1038984640 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f591c043020 con 0x7f592411e440 2023-10-30T12:16:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.760+0000 7f59217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:51.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.760+0000 7f59217fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:51.764 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 10 librados: watch_flush enter 2023-10-30T12:16:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 10 librados: watch_flush exit 2023-10-30T12:16:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 20 client.214200.objecter shutdown clearing up homeless session... 2023-10-30T12:16:51.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 10 client.214200.objecter successfully canceled tick 2023-10-30T12:16:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59080392a0 msgr2=0x7f590803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59080392a0 0x7f590803b760 secure :-1 s=READY pgs=4236 cs=0 l=1 rev1=1 crypto rx=0x7f5918006fd0 tx=0x7f59180092e0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 10 monclient: shutdown 2023-10-30T12:16:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:51.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592411e440 msgr2=0x7f592411e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592411e440 0x7f592411e8f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f591c036040 tx=0x7f591c004a40 comp rx=0 tx=0).stop 2023-10-30T12:16:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 shutdown_connections 2023-10-30T12:16:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59080392a0 0x7f590803b760 unknown :-1 s=CLOSED pgs=4236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f592411e440 0x7f592411e8f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5924132be0 0x7f59241cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 --2- 172.21.15.50:0/1038984640 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f59240a4b60 0x7f59241ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:51.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 >> 172.21.15.50:0/1038984640 conn(0x7f59240a8060 msgr2=0x7f59241345a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:51.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 shutdown_connections 2023-10-30T12:16:51.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.762+0000 7f592ad73640 1 -- 172.21.15.50:0/1038984640 wait complete. 2023-10-30T12:16:51.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:51.763+0000 7f592ad73640 1 librados: shutdown 2023-10-30T12:16:54.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:54.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.892+0000 7f4ece621640 10 monclient: get_monmap_and_config 2023-10-30T12:16:54.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.892+0000 7f4ece621640 10 monclient: build_initial_monmap 2023-10-30T12:16:54.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.892+0000 7f4ece621640 10 monclient: monmap: 2023-10-30T12:16:54.894 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:54.895 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:54.895 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:54.893121+0000 2023-10-30T12:16:54.895 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:54.893121+0000 2023-10-30T12:16:54.895 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:54.895 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:54.896 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:54.896 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:54.896 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:54.896 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:54.896 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:54.896 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.892+0000 7f4ece621640 1 Processor -- start 2023-10-30T12:16:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.892+0000 7f4ece621640 1 -- start start 2023-10-30T12:16:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.892+0000 7f4ece621640 10 monclient: init 2023-10-30T12:16:54.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:54.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-10-30T12:16:54.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4ec80a4b60 0x7f4ec81326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:54.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec80a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:16:54.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec8099a40 0x7f4ec8132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:54.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec8099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:54.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ec80a2350 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:54.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec8133120 con 0x7f4ec80a2350 2023-10-30T12:16:54.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:54.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec80a3d70 con 0x7f4ec80a4b60 2023-10-30T12:16:54.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:54.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80a4f40 con 0x7f4ec8099a40 2023-10-30T12:16:54.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:54.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ece621640 10 monclient(hunting): authenticate will time out at 1860.076386s 2023-10-30T12:16:54.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec77fe640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec8099a40 msgr2=0x7f4ec8132be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:16:54.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec8099a40 0x7f4ec8132be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:54.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4eccbbe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4ec80a2350 0x7f4ec80a27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57374/0 (socket says 172.21.15.50:57374) 2023-10-30T12:16:54.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4eccbbe640 1 -- 172.21.15.50:0/3548417574 learned_addr learned my addr 172.21.15.50:0/3548417574 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:16:54.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 1 --2- 172.21.15.50:0/3548417574 >> v2:172.21.15.118:3300/0 conn(0x7f4ec80a4b60 0x7f4ec81326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:54.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request con 0x7f4ec80a4b60 auth_method 0 2023-10-30T12:16:54.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:54.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:54.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:54.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:54.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:54.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.893+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:54.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_done global_id 204292 payload 274 2023-10-30T12:16:54.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 1 -- 172.21.15.50:0/3548417574 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec8099a40 msgr2=0x7f4ec8132be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:16:54.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 1 --2- 172.21.15.50:0/3548417574 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec8099a40 0x7f4ec8132be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:54.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 1 -- 172.21.15.50:0/3548417574 >> v1:172.21.15.50:6789/0 conn(0x7f4ec80a2350 legacy=0x7f4ec80a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:54.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:16:54.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 1 monclient: found mon.noname-c 2023-10-30T12:16:54.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:54.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:16:54.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 1 -- 172.21.15.50:0/3548417574 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec80a4150 con 0x7f4ec80a4b60 2023-10-30T12:16:54.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.894+0000 7f4ec7fff640 1 --2- 172.21.15.50:0/3548417574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec80a4b60 0x7f4ec81326a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4eb80097a0 tx=0x7f4eb802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=524344dcb3b01837 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 1 -- 172.21.15.50:0/3548417574 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8031710 con 0x7f4ec80a4b60 2023-10-30T12:16:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:16:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: dump: 2023-10-30T12:16:54.912 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:54.913 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:54.913 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.913 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.913 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:54.913 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:54.914 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:54.914 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:54.914 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:54.914 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:54.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:16:54.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:16:54.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:54.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 1 -- 172.21.15.50:0/3548417574 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb802f050 con 0x7f4ec80a4b60 2023-10-30T12:16:54.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 5 monclient: authenticate success, global_id 204292 2023-10-30T12:16:54.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:16:54.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 1 -- 172.21.15.50:0/3548417574 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8031a90 con 0x7f4ec80a4b60 2023-10-30T12:16:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 10 monclient: get_monmap_and_config success 2023-10-30T12:16:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 4 set_mon_vals no callback set 2023-10-30T12:16:54.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:16:54.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ec6ffd640 10 monclient: dump: 2023-10-30T12:16:54.917 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:54.917 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:54.917 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.917 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.917 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:54.918 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:54.918 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:54.918 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:54.918 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:54.918 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:54.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 10 monclient: shutdown 2023-10-30T12:16:54.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:54.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 -- 172.21.15.50:0/3548417574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec80a4b60 msgr2=0x7f4ec81326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:54.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 --2- 172.21.15.50:0/3548417574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec80a4b60 0x7f4ec81326a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4eb80097a0 tx=0x7f4eb802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:16:54.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 -- 172.21.15.50:0/3548417574 shutdown_connections 2023-10-30T12:16:54.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 --2- 172.21.15.50:0/3548417574 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec8099a40 0x7f4ec8132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:54.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 --2- 172.21.15.50:0/3548417574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec80a4b60 0x7f4ec81326a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:54.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 -- 172.21.15.50:0/3548417574 >> 172.21.15.50:0/3548417574 conn(0x7f4ec80a8060 msgr2=0x7f4ec80a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:54.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 -- 172.21.15.50:0/3548417574 shutdown_connections 2023-10-30T12:16:54.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.895+0000 7f4ece621640 1 -- 172.21.15.50:0/3548417574 wait complete. 2023-10-30T12:16:54.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.896+0000 7f4ece621640 10 monclient: build_initial_monmap 2023-10-30T12:16:54.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.896+0000 7f4ece621640 10 monclient: monmap: 2023-10-30T12:16:54.921 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:54.921 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:16:54.921 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:16:54.921 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:16:54.922 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:54.922 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:54.922 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:16:54.922 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:16:54.922 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:16:54.922 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:54.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.896+0000 7f4ece621640 1 librados: starting msgr at 2023-10-30T12:16:54.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.896+0000 7f4ece621640 1 librados: starting objecter 2023-10-30T12:16:54.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.896+0000 7f4ece621640 1 Processor -- start 2023-10-30T12:16:54.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 -- start start 2023-10-30T12:16:54.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 librados: setting wanted keys 2023-10-30T12:16:54.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 librados: calling monclient init 2023-10-30T12:16:54.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient: init 2023-10-30T12:16:54.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:54.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:16:54.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ec80a2350 0x7f4ec81ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:54.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec80a2350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:16:54.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec80a4b60 0x7f4ec8121830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:54.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec80a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:16:54.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec81cae80 0x7f4ec8121d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:54.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec81cae80 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:16:54.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:54.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8125000 con 0x7f4ec80a2350 2023-10-30T12:16:54.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:54.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80a18d0 con 0x7f4ec81cae80 2023-10-30T12:16:54.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:54.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80a09c0 con 0x7f4ec80a4b60 2023-10-30T12:16:54.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:54.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ece621640 10 monclient(hunting): authenticate will time out at 1860.080571s 2023-10-30T12:16:54.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec77fe640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec80a4b60 msgr2=0x7f4ec8121830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:16:54.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec80a4b60 0x7f4ec8121830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:54.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ec80a2350 0x7f4ec81ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:54.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ec80a2350 0x7f4ec81ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48578/0 (socket says 172.21.15.50:48578) 2023-10-30T12:16:54.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 -- 172.21.15.50:0/2629232819 learned_addr learned my addr 172.21.15.50:0/2629232819 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:54.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request con 0x7f4ec80a2350 auth_method 0 2023-10-30T12:16:54.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:54.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:54.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:54.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:16:54.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:16:54.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:16:54.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4eccbbe640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec81cae80 0x7f4ec8121d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:54.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4eccbbe640 10 monclient(hunting): get_auth_request con 0x7f4ec81cae80 auth_method 0 2023-10-30T12:16:54.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4eccbbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:54.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4eccbbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:54.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4eccbbe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:54.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_done global_id 214203 payload 824 2023-10-30T12:16:54.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 -- 172.21.15.50:0/2629232819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec80a4b60 msgr2=0x7f4ec8121830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:16:54.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec80a4b60 0x7f4ec8121830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:54.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 -- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec81cae80 msgr2=0x7f4ec8121d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:54.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec81cae80 0x7f4ec8121d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:54.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:16:54.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 monclient: found mon.noname-a 2023-10-30T12:16:54.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:16:54.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:16:54.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.897+0000 7f4ec7fff640 1 -- 172.21.15.50:0/2629232819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb8009400 con 0x7f4ec80a2350 2023-10-30T12:16:54.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec7fff640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ec80a2350 0x7f4ec81ca940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8000c00 tx=0x7f4eb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:54.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 client.?.objecter ms_handle_connect 0x7f4ec80a2350 2023-10-30T12:16:54.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:16:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 1 -- 172.21.15.50:0/2629232819 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8033a50 con 0x7f4ec80a2350 2023-10-30T12:16:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:16:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: dump: 2023-10-30T12:16:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:54.933 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:54.934 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.934 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.934 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:54.934 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:54.934 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:54.935 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:16:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:16:54.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:16:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 1 -- 172.21.15.50:0/2629232819 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb8033bb0 con 0x7f4ec80a2350 2023-10-30T12:16:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 5 monclient: authenticate success, global_id 214203 2023-10-30T12:16:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 10 monclient: _renew_subs 2023-10-30T12:16:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec81258a0 con 0x7f4ec80a2350 2023-10-30T12:16:54.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:16:54.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 1 -- 172.21.15.50:0/2629232819 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb803d470 con 0x7f4ec80a2350 2023-10-30T12:16:54.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:16:54.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:16:54.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec4ff9640 10 monclient: dump: 2023-10-30T12:16:54.937 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:16:54.938 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:54.938 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.938 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:16:54.938 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:16:54.938 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:54.939 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:16:54.939 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:16:54.939 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:16:54.939 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:54.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ec67fc640 4 set_mon_vals no callback set 2023-10-30T12:16:54.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 10 client.214203.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:16:54.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 10 monclient: _renew_subs 2023-10-30T12:16:54.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:54.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec81234d0 con 0x7f4ec80a2350 2023-10-30T12:16:54.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.898+0000 7f4ece621640 1 librados: init done 2023-10-30T12:16:54.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec4ff9640 1 -- 172.21.15.50:0/2629232819 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4eb803a040 con 0x7f4ec80a2350 2023-10-30T12:16:54.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ece621640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:54.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ece621640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:54.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ece621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:54.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec4ff9640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ea80392d0 0x7f4ea803b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:54.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec80a2e50 con 0x7f4ec80a2350 2023-10-30T12:16:54.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec4ff9640 1 -- 172.21.15.50:0/2629232819 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(505..505 src has 417..505) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f4eb806c830 con 0x7f4ec80a2350 2023-10-30T12:16:54.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec4ff9640 10 client.214203.objecter ms_dispatch 0x7f4ec8000b90 osd_map(505..505 src has 417..505) v4 2023-10-30T12:16:54.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec4ff9640 3 client.214203.objecter handle_osd_map got epochs [505,505] > 0 2023-10-30T12:16:54.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec4ff9640 3 client.214203.objecter handle_osd_map decoding full epoch 505 2023-10-30T12:16:54.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec4ff9640 20 client.214203.objecter dump_active .. 0 homeless 2023-10-30T12:16:54.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec77fe640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ea80392d0 0x7f4ea803b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:54.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.899+0000 7f4ec77fe640 10 monclient: get_auth_request con 0x7f4ea80392d0 auth_method 0 2023-10-30T12:16:54.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.900+0000 7f4ec4ff9640 10 client.214203.objecter ms_handle_connect 0x7f4ea80392d0 2023-10-30T12:16:54.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.900+0000 7f4ec77fe640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ea80392d0 0x7f4ea803b790 secure :-1 s=READY pgs=4237 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0006fd0 tx=0x7f4eb0006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:16:54.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.904+0000 7f4ec4ff9640 1 -- 172.21.15.50:0/2629232819 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4eb80483b0 con 0x7f4ec80a2350 2023-10-30T12:16:54.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.904+0000 7f4ec4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:16:54.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:54.904+0000 7f4ec4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:16:55.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.150+0000 7f4ece621640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:16:55.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.150+0000 7f4ece621640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.150+0000 7f4ece621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:16:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.150+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4ec804e160 con 0x7f4ec80a2350 2023-10-30T12:16:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.151+0000 7f4ec4ff9640 1 -- 172.21.15.50:0/2629232819 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f4eb8038620 con 0x7f4ec80a2350 2023-10-30T12:16:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.151+0000 7f4ec4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:16:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.151+0000 7f4ec4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:16:55.154 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:16:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 10 librados: watch_flush enter 2023-10-30T12:16:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 10 librados: watch_flush exit 2023-10-30T12:16:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 20 client.214203.objecter shutdown clearing up homeless session... 2023-10-30T12:16:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 10 client.214203.objecter successfully canceled tick 2023-10-30T12:16:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ea80392d0 msgr2=0x7f4ea803b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ea80392d0 0x7f4ea803b790 secure :-1 s=READY pgs=4237 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0006fd0 tx=0x7f4eb0006d80 comp rx=0 tx=0).stop 2023-10-30T12:16:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 10 monclient: shutdown 2023-10-30T12:16:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:16:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ec80a2350 msgr2=0x7f4ec81ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:16:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.152+0000 7f4ece621640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ec80a2350 0x7f4ec81ca940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8000c00 tx=0x7f4eb80046c0 comp rx=0 tx=0).stop 2023-10-30T12:16:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 shutdown_connections 2023-10-30T12:16:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4ea80392d0 0x7f4ea803b790 unknown :-1 s=CLOSED pgs=4237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ec81cae80 0x7f4ec8121d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:55.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4ec80a4b60 0x7f4ec8121830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:55.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 --2- 172.21.15.50:0/2629232819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ec80a2350 0x7f4ec81ca940 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:16:55.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 >> 172.21.15.50:0/2629232819 conn(0x7f4ec80a8060 msgr2=0x7f4ec80a5830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:16:55.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 shutdown_connections 2023-10-30T12:16:55.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 -- 172.21.15.50:0/2629232819 wait complete. 2023-10-30T12:16:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:55.153+0000 7f4ece621640 1 librados: shutdown 2023-10-30T12:16:58.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:16:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.281+0000 7fdf18fb6640 10 monclient: get_monmap_and_config 2023-10-30T12:16:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.281+0000 7fdf18fb6640 10 monclient: build_initial_monmap 2023-10-30T12:16:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.281+0000 7fdf18fb6640 10 monclient: monmap: 2023-10-30T12:16:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:16:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:16:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:16:58.282253+0000 2023-10-30T12:16:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:16:58.282253+0000 2023-10-30T12:16:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:16:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:16:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:16:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:16:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:16:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:16:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:16:58.285 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:16:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.281+0000 7fdf18fb6640 1 Processor -- start 2023-10-30T12:16:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.281+0000 7fdf18fb6640 1 -- start start 2023-10-30T12:16:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.281+0000 7fdf18fb6640 10 monclient: init 2023-10-30T12:16:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient: _reopen_session rank -1 2023-10-30T12:16:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-10-30T12:16:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf140a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:16:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:16:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient(hunting): picked mon.noname-a con 0x7fdf14099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:16:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf14132db0 addr v1:172.21.15.50:6789/0 2023-10-30T12:16:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf140a4f40 con 0x7fdf14132db0 2023-10-30T12:16:58.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf140a4150 con 0x7fdf140a4b60 2023-10-30T12:16:58.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:16:58.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf140a3d70 con 0x7fdf14099a40 2023-10-30T12:16:58.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient(hunting): _renew_subs 2023-10-30T12:16:58.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf18fb6640 10 monclient(hunting): authenticate will time out at 1863.465481s 2023-10-30T12:16:58.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:58.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49170/0 (socket says 172.21.15.50:49170) 2023-10-30T12:16:58.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf12d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdf14132db0 0x7fdf1413cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52266/0 (socket says 172.21.15.50:52266) 2023-10-30T12:16:58.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 learned_addr learned my addr 172.21.15.50:0/4124162236 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:16:58.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request con 0x7fdf14099a40 auth_method 0 2023-10-30T12:16:58.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:58.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:58.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:16:58.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:58.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:58.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.282+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:16:58.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:58.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request con 0x7fdf14099a40 auth_method 0 2023-10-30T12:16:58.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:58.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:58.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:58.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:58.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:58.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:58.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.483+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:16:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request con 0x7fdf14099a40 auth_method 0 2023-10-30T12:16:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:58.884+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:16:59.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:16:59.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request con 0x7fdf14099a40 auth_method 0 2023-10-30T12:16:59.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:16:59.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:16:59.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:16:59.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:16:59.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:16:59.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:16:59.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:16:59.685+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): tick 2023-10-30T12:17:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:17:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 monclient(hunting): continuing hunt 2023-10-30T12:17:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:17:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 msgr2=0x7fdf1413aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdf14099a40 0x7fdf1413aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 >> v1:172.21.15.118:6789/0 conn(0x7fdf140a4b60 legacy=0x7fdf141326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 >> v1:172.21.15.50:6789/0 conn(0x7fdf14132db0 legacy=0x7fdf1413cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient: _add_conns ranks=[4,2] 2023-10-30T12:17:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fdef40036c0 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 0x7fdef40086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fdef4006220 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef4008dc0 con 0x7fdef40036c0 2023-10-30T12:17:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4008f30 con 0x7fdef4006220 2023-10-30T12:17:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf0bfff640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 msgr2=0x7fdef40086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.282+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 0x7fdef40086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 10 monclient(hunting): get_auth_request con 0x7fdef40036c0 auth_method 0 2023-10-30T12:17:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.283+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:01.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.483+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 msgr2=0x7fdef40086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:01.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.483+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 0x7fdef40086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:01.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:01.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 10 monclient(hunting): get_auth_request con 0x7fdef40036c0 auth_method 0 2023-10-30T12:17:01.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:01.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:01.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:01.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:01.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:01.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:01.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.484+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:01.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.884+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 msgr2=0x7fdef40086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:01.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.884+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 0x7fdef40086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:01.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:01.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 10 monclient(hunting): get_auth_request con 0x7fdef40036c0 auth_method 0 2023-10-30T12:17:01.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:01.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:01.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:01.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:01.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:01.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:01.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:01.885+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.685+0000 7fdf11d9b640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 msgr2=0x7fdef40086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.685+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 0x7fdef40086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:02.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:02.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 10 monclient(hunting): get_auth_request con 0x7fdef40036c0 auth_method 0 2023-10-30T12:17:02.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:02.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:02.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:02.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:02.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:02.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:02.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:02.687+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:04.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): tick 2023-10-30T12:17:04.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:17:04.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 monclient(hunting): continuing hunt 2023-10-30T12:17:04.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:17:04.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 msgr2=0x7fdef40086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006220 0x7fdef40086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 msgr2=0x7fdef4005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef40036c0 0x7fdef4005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-10-30T12:17:04.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf12d9d640 1 -- 172.21.15.50:0/4124162236 reap_dead start 2023-10-30T12:17:04.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdef4009350 0x7fdef400b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fdef4009350 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:04.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef4003b30 0x7fdef40066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fdef4003b30 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:04.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006bf0 0x7fdef4007030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fdef4006bf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:04.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef4008dc0 con 0x7fdef4009350 2023-10-30T12:17:04.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef4008f30 con 0x7fdef4003b30 2023-10-30T12:17:04.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 1 -- 172.21.15.50:0/4124162236 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4007640 con 0x7fdef4006bf0 2023-10-30T12:17:04.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf0bfff640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf12d9d640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006bf0 msgr2=0x7fdef4007030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:04.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf12d9d640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006bf0 0x7fdef4007030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:04.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.50:3300/0 conn(0x7fdef4009350 0x7fdef400b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): get_auth_request con 0x7fdef4009350 auth_method 0 2023-10-30T12:17:04.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef4003b30 0x7fdef40066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request con 0x7fdef4003b30 auth_method 0 2023-10-30T12:17:04.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf11d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf11d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient(hunting): handle_auth_done global_id 214206 payload 274 2023-10-30T12:17:04.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006bf0 msgr2=0x7fdef4007030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006bf0 0x7fdef4007030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef4003b30 msgr2=0x7fdef40066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:04.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef4003b30 0x7fdef40066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 1 monclient: found mon.noname-a 2023-10-30T12:17:04.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.283+0000 7fdf1259c640 1 -- 172.21.15.50:0/4124162236 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1409a440 con 0x7fdef4009350 2023-10-30T12:17:04.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1259c640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdef400b740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdf140a3a20 tx=0x7fdf0c02c020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deaf91e1b66a463d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 1 -- 172.21.15.50:0/4124162236 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c02cc90 con 0x7fdef4009350 2023-10-30T12:17:04.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:04.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: dump: 2023-10-30T12:17:04.297 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.297 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.298 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.298 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.298 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.298 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.298 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.299 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.299 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.299 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 1 -- 172.21.15.50:0/4124162236 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0c02cdf0 con 0x7fdef4009350 2023-10-30T12:17:04.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf18fb6640 5 monclient: authenticate success, global_id 214206 2023-10-30T12:17:04.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf18fb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:04.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf18fb6640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:04.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 1 -- 172.21.15.50:0/4124162236 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c0376a0 con 0x7fdef4009350 2023-10-30T12:17:04.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf18fb6640 4 set_mon_vals no callback set 2023-10-30T12:17:04.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.284+0000 7fdf1159a640 10 monclient: dump: 2023-10-30T12:17:04.302 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.302 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.302 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.302 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.302 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.303 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.303 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.303 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.303 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 10 monclient: shutdown 2023-10-30T12:17:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 -- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 msgr2=0x7fdef400b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdef400b740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdf140a3a20 tx=0x7fdf0c02c020 comp rx=0 tx=0).stop 2023-10-30T12:17:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 -- 172.21.15.50:0/4124162236 shutdown_connections 2023-10-30T12:17:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4006bf0 0x7fdef4007030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/4124162236 >> v2:172.21.15.118:3300/0 conn(0x7fdef4003b30 0x7fdef40066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/4124162236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdef400b740 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 -- 172.21.15.50:0/4124162236 >> 172.21.15.50:0/4124162236 conn(0x7fdf140a8060 msgr2=0x7fdf140a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:04.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 -- 172.21.15.50:0/4124162236 shutdown_connections 2023-10-30T12:17:04.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.285+0000 7fdf18fb6640 1 -- 172.21.15.50:0/4124162236 wait complete. 2023-10-30T12:17:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.286+0000 7fdf18fb6640 10 monclient: build_initial_monmap 2023-10-30T12:17:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.286+0000 7fdf18fb6640 10 monclient: monmap: 2023-10-30T12:17:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:04.312 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 librados: starting msgr at 2023-10-30T12:17:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 librados: starting objecter 2023-10-30T12:17:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 Processor -- start 2023-10-30T12:17:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 -- start start 2023-10-30T12:17:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 librados: setting wanted keys 2023-10-30T12:17:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 librados: calling monclient init 2023-10-30T12:17:04.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient: init 2023-10-30T12:17:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:17:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4003b30 0x7fdf1411e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): picked mon.noname-c con 0x7fdef4003b30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:04.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdef4006bf0 0x7fdf1409ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): picked mon.noname-b con 0x7fdef4006bf0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdf14131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): picked mon.noname-a con 0x7fdef4009350 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140a1940 con 0x7fdef4009350 2023-10-30T12:17:04.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1411fd70 con 0x7fdef4006bf0 2023-10-30T12:17:04.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1411f7b0 con 0x7fdef4003b30 2023-10-30T12:17:04.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf18fb6640 10 monclient(hunting): authenticate will time out at 1869.470896s 2023-10-30T12:17:04.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf1259c640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4003b30 msgr2=0x7fdf1411e8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:04.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.287+0000 7fdf1259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4003b30 0x7fdf1411e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:04.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdf14131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdf14131660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49208/0 (socket says 172.21.15.50:49208) 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 -- 172.21.15.50:0/3352899383 learned_addr learned my addr 172.21.15.50:0/3352899383 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): get_auth_request con 0x7fdef4009350 auth_method 0 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdef4006bf0 0x7fdf1409ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request con 0x7fdef4006bf0 auth_method 0 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient(hunting): handle_auth_done global_id 214209 payload 824 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 -- 172.21.15.50:0/3352899383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4003b30 msgr2=0x7fdf1411e8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4003b30 0x7fdf1411e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 -- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdef4006bf0 msgr2=0x7fdf1409ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdef4006bf0 0x7fdf1409ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 monclient: found mon.noname-a 2023-10-30T12:17:04.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 -- 172.21.15.50:0/3352899383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c00f040 con 0x7fdef4009350 2023-10-30T12:17:04.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf12d9d640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdf14131660 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf140a5300 tx=0x7fdf0400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 client.?.objecter ms_handle_connect 0x7fdef4009350 2023-10-30T12:17:04.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:04.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 1 -- 172.21.15.50:0/3352899383 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04004d60 con 0x7fdef4009350 2023-10-30T12:17:04.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:04.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 monclient: dump: 2023-10-30T12:17:04.324 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.324 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.325 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.325 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.325 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.325 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.325 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.325 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.326 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.326 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 1 -- 172.21.15.50:0/3352899383 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0400a610 con 0x7fdef4009350 2023-10-30T12:17:04.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.288+0000 7fdf0affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf0affd640 1 -- 172.21.15.50:0/3352899383 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0400a9a0 con 0x7fdef4009350 2023-10-30T12:17:04.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 5 monclient: authenticate success, global_id 214209 2023-10-30T12:17:04.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 10 monclient: _renew_subs 2023-10-30T12:17:04.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1411faf0 con 0x7fdef4009350 2023-10-30T12:17:04.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf10d99640 4 set_mon_vals no callback set 2023-10-30T12:17:04.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf0affd640 10 monclient: dump: 2023-10-30T12:17:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.331 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 10 client.214209.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:04.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 10 monclient: _renew_subs 2023-10-30T12:17:04.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1409f3a0 con 0x7fdef4009350 2023-10-30T12:17:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.289+0000 7fdf18fb6640 1 librados: init done 2023-10-30T12:17:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf0affd640 1 -- 172.21.15.50:0/3352899383 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdf0400ab00 con 0x7fdef4009350 2023-10-30T12:17:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf18fb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf18fb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf18fb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1404e160 con 0x7fdef4009350 2023-10-30T12:17:04.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf0affd640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdef403b0d0 0x7fdef403d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf0affd640 1 -- 172.21.15.50:0/3352899383 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(507..507 src has 417..507) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fdf04049cd0 con 0x7fdef4009350 2023-10-30T12:17:04.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf0affd640 10 client.214209.objecter ms_dispatch 0x7fdf14000b90 osd_map(507..507 src has 417..507) v4 2023-10-30T12:17:04.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf0affd640 3 client.214209.objecter handle_osd_map got epochs [507,507] > 0 2023-10-30T12:17:04.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf0affd640 3 client.214209.objecter handle_osd_map decoding full epoch 507 2023-10-30T12:17:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.290+0000 7fdf0affd640 20 client.214209.objecter dump_active .. 0 homeless 2023-10-30T12:17:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.291+0000 7fdf1259c640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdef403b0d0 0x7fdef403d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.291+0000 7fdf1259c640 10 monclient: get_auth_request con 0x7fdef403b0d0 auth_method 0 2023-10-30T12:17:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.291+0000 7fdf0affd640 10 client.214209.objecter ms_handle_connect 0x7fdef403b0d0 2023-10-30T12:17:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.291+0000 7fdf1259c640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdef403b0d0 0x7fdef403d590 secure :-1 s=READY pgs=4238 cs=0 l=1 rev1=1 crypto rx=0x7fdf1406ae30 tx=0x7fdf0c011040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.295+0000 7fdf0affd640 1 -- 172.21.15.50:0/3352899383 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf04019270 con 0x7fdef4009350 2023-10-30T12:17:04.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.295+0000 7fdf0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.295+0000 7fdf0affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:04.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.541+0000 7fdf18fb6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:17:04.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.541+0000 7fdf18fb6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:04.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.541+0000 7fdf18fb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.541+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdf1406b110 con 0x7fdef4009350 2023-10-30T12:17:04.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.542+0000 7fdf0affd640 1 -- 172.21.15.50:0/3352899383 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fdf04018070 con 0x7fdef4009350 2023-10-30T12:17:04.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.542+0000 7fdf0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.542+0000 7fdf0affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:04.545 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":174,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:17:04.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.543+0000 7fdf18fb6640 10 librados: watch_flush enter 2023-10-30T12:17:04.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.543+0000 7fdf18fb6640 10 librados: watch_flush exit 2023-10-30T12:17:04.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.543+0000 7fdf18fb6640 20 client.214209.objecter shutdown clearing up homeless session... 2023-10-30T12:17:04.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.543+0000 7fdf18fb6640 10 client.214209.objecter successfully canceled tick 2023-10-30T12:17:04.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdef403b0d0 msgr2=0x7fdef403d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdef403b0d0 0x7fdef403d590 secure :-1 s=READY pgs=4238 cs=0 l=1 rev1=1 crypto rx=0x7fdf1406ae30 tx=0x7fdf0c011040 comp rx=0 tx=0).stop 2023-10-30T12:17:04.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 10 monclient: shutdown 2023-10-30T12:17:04.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:04.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 msgr2=0x7fdf14131660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdf14131660 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf140a5300 tx=0x7fdf0400cb90 comp rx=0 tx=0).stop 2023-10-30T12:17:04.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 shutdown_connections 2023-10-30T12:17:04.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdef403b0d0 0x7fdef403d590 unknown :-1 s=CLOSED pgs=4238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef4009350 0x7fdf14131660 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdef4006bf0 0x7fdf1409ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 --2- 172.21.15.50:0/3352899383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef4003b30 0x7fdf1411e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 >> 172.21.15.50:0/3352899383 conn(0x7fdf140a8060 msgr2=0x7fdf141b97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:04.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 shutdown_connections 2023-10-30T12:17:04.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 -- 172.21.15.50:0/3352899383 wait complete. 2023-10-30T12:17:04.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.544+0000 7fdf18fb6640 1 librados: shutdown 2023-10-30T12:17:04.557 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:17:04.557 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:17:04.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:17:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.663+0000 7fe1fc9c9640 10 monclient: get_monmap_and_config 2023-10-30T12:17:04.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.663+0000 7fe1fc9c9640 10 monclient: build_initial_monmap 2023-10-30T12:17:04.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.663+0000 7fe1fc9c9640 10 monclient: monmap: 2023-10-30T12:17:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:04.664292+0000 2023-10-30T12:17:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:04.664292+0000 2023-10-30T12:17:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:04.670 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.663+0000 7fe1fc9c9640 1 Processor -- start 2023-10-30T12:17:04.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 1 -- start start 2023-10-30T12:17:04.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient: init 2023-10-30T12:17:04.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-10-30T12:17:04.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1f80a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:04.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 0x7fe1f813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f80a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:04.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1f813af40 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:04.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80a4190 con 0x7fe1f80a2ae0 2023-10-30T12:17:04.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f809a2e0 con 0x7fe1f813af40 2023-10-30T12:17:04.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80a1630 con 0x7fe1f80a2450 2023-10-30T12:17:04.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1effff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 msgr2=0x7fe1f813aa00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:04.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1fc9c9640 10 monclient(hunting): authenticate will time out at 1869.847605s 2023-10-30T12:17:04.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1effff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 0x7fe1f813aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:04.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f659c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe1f80a2ae0 0x7fe1f80a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52280/0 (socket says 172.21.15.50:52280) 2023-10-30T12:17:04.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f659c640 1 -- 172.21.15.50:0/1905068571 learned_addr learned my addr 172.21.15.50:0/1905068571 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:17:04.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334998699 0 0) 0x7fe1e8001110 con 0x7fe1f80a2ae0 2023-10-30T12:17:04.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f5d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:17:04.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f5d9b640 10 monclient(hunting): my global_id is 214212 2023-10-30T12:17:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.664+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e4003520 con 0x7fe1f80a2ae0 2023-10-30T12:17:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906801942 0 0) 0x7fe1e4003520 con 0x7fe1f80a2ae0 2023-10-30T12:17:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 msgr2=0x7fe1f813aa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 0x7fe1f813aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 >> v1:172.21.15.118:6789/0 conn(0x7fe1f813af40 legacy=0x7fe1f813d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:17:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:17:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f809a8a0 con 0x7fe1f80a2ae0 2023-10-30T12:17:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fe1e8001800 con 0x7fe1f80a2ae0 2023-10-30T12:17:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:17:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: dump: 2023-10-30T12:17:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.708 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.708 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.708 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.709 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.709 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.709 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.710 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.710 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.710 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:17:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 >> v1:172.21.15.50:6789/0 conn(0x7fe1f80a2ae0 legacy=0x7fe1f80a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1e40039a0 0x7fe1e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): picked mon.a con 0x7fe1e40039a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:04.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40047c0 0x7fe1e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): picked mon.b con 0x7fe1e40047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:04.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 0x7fe1e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): picked mon.c con 0x7fe1f80a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:04.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80a1630 con 0x7fe1e40039a0 2023-10-30T12:17:04.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40055b0 con 0x7fe1e40047c0 2023-10-30T12:17:04.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e400bec0 con 0x7fe1f80a2450 2023-10-30T12:17:04.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f5d9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:17:04.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1e40039a0 0x7fe1e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f6d9d640 1 -- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 msgr2=0x7fe1e400b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:04.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f6d9d640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 0x7fe1e400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:04.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): get_auth_request con 0x7fe1e40039a0 auth_method 0 2023-10-30T12:17:04.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:04.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40047c0 0x7fe1e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 10 monclient(hunting): get_auth_request con 0x7fe1e40047c0 auth_method 0 2023-10-30T12:17:04.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:04.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:04.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1effff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:04.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient(hunting): handle_auth_done global_id 214212 payload 293 2023-10-30T12:17:04.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 1 -- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 msgr2=0x7fe1e400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f80a2450 0x7fe1e400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 1 -- 172.21.15.50:0/1905068571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40047c0 msgr2=0x7fe1e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:04.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40047c0 0x7fe1e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 1 monclient: found mon.a 2023-10-30T12:17:04.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 1 -- 172.21.15.50:0/1905068571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e4003520 con 0x7fe1e40039a0 2023-10-30T12:17:04.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1f659c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1fc9c9640 5 monclient: authenticate success, global_id 214212 2023-10-30T12:17:04.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.665+0000 7fe1fc9c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:04.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f659c640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1e40039a0 0x7fe1e4003dc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c160 tx=0x7fe1e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc1f0250df507ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8005ad0 con 0x7fe1e40039a0 2023-10-30T12:17:04.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 10 monclient: dump: 2023-10-30T12:17:04.731 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.732 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.732 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.732 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.733 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.734 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.734 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.735 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1fc9c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:04.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e8005c50 con 0x7fe1e40039a0 2023-10-30T12:17:04.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1fc9c9640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:04.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 1 -- 172.21.15.50:0/1905068571 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e802c830 con 0x7fe1e40039a0 2023-10-30T12:17:04.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1fc9c9640 4 set_mon_vals no callback set 2023-10-30T12:17:04.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.666+0000 7fe1f5d9b640 10 monclient: dump: 2023-10-30T12:17:04.738 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.741 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 10 monclient: shutdown 2023-10-30T12:17:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1e40039a0 msgr2=0x7fe1e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 1 --2- 172.21.15.50:0/1905068571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1e40039a0 0x7fe1e4003dc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c160 tx=0x7fe1e40057f0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1f6d9d640 1 -- 172.21.15.50:0/1905068571 reap_dead start 2023-10-30T12:17:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/1905068571 shutdown_connections 2023-10-30T12:17:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/1905068571 >> 172.21.15.50:0/1905068571 conn(0x7fe1f80a80b0 msgr2=0x7fe1f80a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:04.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/1905068571 shutdown_connections 2023-10-30T12:17:04.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/1905068571 wait complete. 2023-10-30T12:17:04.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 10 monclient: build_initial_monmap 2023-10-30T12:17:04.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.667+0000 7fe1fc9c9640 10 monclient: monmap: 2023-10-30T12:17:04.743 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.745 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:04.745 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:04.745 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:04.745 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 librados: starting msgr at 2023-10-30T12:17:04.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 librados: starting objecter 2023-10-30T12:17:04.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 Processor -- start 2023-10-30T12:17:04.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 -- start start 2023-10-30T12:17:04.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 librados: setting wanted keys 2023-10-30T12:17:04.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 librados: calling monclient init 2023-10-30T12:17:04.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient: init 2023-10-30T12:17:04.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:04.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40039a0 0x7fe1f81ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe1e40039a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:04.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809f4e0 0x7fe1f8131340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f809f4e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f8131880 0x7fe1f809ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f8131880 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f811e0a0 con 0x7fe1f809f4e0 2023-10-30T12:17:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80a0ae0 con 0x7fe1e40039a0 2023-10-30T12:17:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80a1790 con 0x7fe1f8131880 2023-10-30T12:17:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1fc9c9640 10 monclient(hunting): authenticate will time out at 1869.851660s 2023-10-30T12:17:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1f6d9d640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f8131880 msgr2=0x7fe1f809ca80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f8131880 0x7fe1f809ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809f4e0 0x7fe1f8131340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809f4e0 0x7fe1f8131340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49232/0 (socket says 172.21.15.50:49232) 2023-10-30T12:17:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 1 -- 172.21.15.50:0/2420795110 learned_addr learned my addr 172.21.15.50:0/2420795110 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): get_auth_request con 0x7fe1f809f4e0 auth_method 0 2023-10-30T12:17:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1f659c640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40039a0 0x7fe1f81ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1f659c640 10 monclient(hunting): get_auth_request con 0x7fe1e40039a0 auth_method 0 2023-10-30T12:17:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1f659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.668+0000 7fe1effff640 10 monclient(hunting): handle_auth_done global_id 214215 payload 824 2023-10-30T12:17:04.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f8131880 msgr2=0x7fe1f809ca80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:04.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe1f8131880 0x7fe1f809ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40039a0 msgr2=0x7fe1f81ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40039a0 0x7fe1f81ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 1 monclient: found mon.noname-a 2023-10-30T12:17:04.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 1 -- 172.21.15.50:0/2420795110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8133700 con 0x7fe1f809f4e0 2023-10-30T12:17:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1f659c640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1e40039a0 0x7fe1f81ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:17:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1effff640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809f4e0 0x7fe1f8131340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00096c0 tx=0x7fe1e0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 client.?.objecter ms_handle_connect 0x7fe1f809f4e0 2023-10-30T12:17:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:04.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000e7b0 con 0x7fe1f809f4e0 2023-10-30T12:17:04.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:04.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: dump: 2023-10-30T12:17:04.758 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.759 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.760 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.760 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.760 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.760 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e000e910 con 0x7fe1f809f4e0 2023-10-30T12:17:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000ecc0 con 0x7fe1f809f4e0 2023-10-30T12:17:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1eeffd640 10 monclient: dump: 2023-10-30T12:17:04.762 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.762 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.762 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.763 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.763 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.763 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.763 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.763 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.764 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.764 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 5 monclient: authenticate success, global_id 214215 2023-10-30T12:17:04.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 10 monclient: _renew_subs 2023-10-30T12:17:04.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f80a01b0 con 0x7fe1f809f4e0 2023-10-30T12:17:04.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1f559a640 4 set_mon_vals no callback set 2023-10-30T12:17:04.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 10 client.214215.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:04.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 10 monclient: _renew_subs 2023-10-30T12:17:04.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f809e680 con 0x7fe1f809f4e0 2023-10-30T12:17:04.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.669+0000 7fe1fc9c9640 1 librados: init done 2023-10-30T12:17:04.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe1e00123f0 con 0x7fe1f809f4e0 2023-10-30T12:17:04.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1fc9c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1fc9c9640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809cfc0 0x7fe1f809d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1fc9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f8073ff0 con 0x7fe1f809cfc0 2023-10-30T12:17:04.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809cfc0 0x7fe1f809d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient: get_auth_request con 0x7fe1f809cfc0 auth_method 0 2023-10-30T12:17:04.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe1d80392a0 0x7fe1d803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(507..507 src has 417..507) v4 ==== 6185+0+0 (secure 0 0 0) 0x7fe1e004a950 con 0x7fe1f809f4e0 2023-10-30T12:17:04.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 10 client.214215.objecter ms_dispatch 0x7fe1f8000b90 osd_map(507..507 src has 417..507) v4 2023-10-30T12:17:04.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 3 client.214215.objecter handle_osd_map got epochs [507,507] > 0 2023-10-30T12:17:04.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 3 client.214215.objecter handle_osd_map decoding full epoch 507 2023-10-30T12:17:04.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 20 client.214215.objecter dump_active .. 0 homeless 2023-10-30T12:17:04.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 10 monclient(hunting): handle_auth_done global_id 214218 payload 824 2023-10-30T12:17:04.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f6d9d640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe1d80392a0 0x7fe1d803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 1 -- 172.21.15.50:0/2420795110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f8131cb0 con 0x7fe1f809cfc0 2023-10-30T12:17:04.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f6d9d640 10 monclient: get_auth_request con 0x7fe1d80392a0 auth_method 0 2023-10-30T12:17:04.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 10 client.214215.objecter ms_handle_connect 0x7fe1f809cfc0 2023-10-30T12:17:04.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1f659c640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809cfc0 0x7fe1f809d3a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8001eb0 tx=0x7fe1f80a58e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 10 client.214215.objecter resend_mon_ops 2023-10-30T12:17:04.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.670+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e803c040 con 0x7fe1f809cfc0 2023-10-30T12:17:04.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 10 monclient: dump: 2023-10-30T12:17:04.772 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.772 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.772 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.772 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.772 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.773 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.773 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.773 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.773 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.773 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 10 client.214215.objecter ms_handle_connect 0x7fe1d80392a0 2023-10-30T12:17:04.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1f6d9d640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe1d80392a0 0x7fe1d803b760 secure :-1 s=READY pgs=4239 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80a4430 tx=0x7fe1f0007cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fe1e802ca30 con 0x7fe1f809cfc0 2023-10-30T12:17:04.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:04.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809cfc0 msgr2=0x7fe1f809d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809cfc0 0x7fe1f809d3a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8001eb0 tx=0x7fe1f80a58e0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809cfc0 msgr2=0x7fe1f809d3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:04.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.671+0000 7fe1eeffd640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809cfc0 0x7fe1f809d3a0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8001eb0 tx=0x7fe1f80a58e0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1fc9c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:04.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1fc9c9640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f81387b0 0x7fe1f8138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1fc9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f809a5c0 con 0x7fe1f81387b0 2023-10-30T12:17:04.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f81387b0 0x7fe1f8138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient: get_auth_request con 0x7fe1f81387b0 auth_method 0 2023-10-30T12:17:04.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 10 monclient(hunting): handle_auth_done global_id 214221 payload 824 2023-10-30T12:17:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.677+0000 7fe1f659c640 1 -- 172.21.15.50:0/2420795110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe1f8099b40 con 0x7fe1f81387b0 2023-10-30T12:17:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1f659c640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f81387b0 0x7fe1f8138bc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c160 tx=0x7fe1e8009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 10 client.214215.objecter ms_handle_connect 0x7fe1f81387b0 2023-10-30T12:17:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 10 client.214215.objecter resend_mon_ops 2023-10-30T12:17:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8052070 con 0x7fe1f81387b0 2023-10-30T12:17:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 10 monclient: dump: 2023-10-30T12:17:04.780 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.780 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.780 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.780 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.780 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.781 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4793 (secure 0 0 0) 0x7fe1e8035680 con 0x7fe1f81387b0 2023-10-30T12:17:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f81387b0 msgr2=0x7fe1f8138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f81387b0 0x7fe1f8138bc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c160 tx=0x7fe1e8009030 comp rx=0 tx=0).stop 2023-10-30T12:17:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f81387b0 msgr2=0x7fe1f8138bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:04.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1eeffd640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f81387b0 0x7fe1f8138bc0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c160 tx=0x7fe1e8009030 comp rx=0 tx=0).stop 2023-10-30T12:17:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:04.783 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:04.784 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:04.784 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:17:04.784 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 174, 2023-10-30T12:17:04.784 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:04.784 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:04.785 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:17:04.785 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:04.785 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:17:04.785 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:04.785 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:04.786 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:04.786 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:04.786 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:04.786 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:04.786 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:04.787 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:04.787 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:04.787 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:04.787 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:04.787 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:04.788 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:04.788 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:04.788 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:04.788 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:04.788 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:04.789 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:04.789 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:04.789 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:04.789 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:04.789 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:04.790 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:04.790 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:04.790 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:04.790 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:04.790 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:04.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.791 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:04.791 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:04.791 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:04.791 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:04.793 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:04.793 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:04.793 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:04.793 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:04.793 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:04.793 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:04.794 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:04.795 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:04.795 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:04.795 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:04.795 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:04.795 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:04.796 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:04.796 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:04.796 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:04.796 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:04.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.797 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:04.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.797 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:04.797 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:04.797 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:04.798 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:04.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.798 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:04.798 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:04.798 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:04.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.799 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:04.799 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:04.799 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:04.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.800 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:04.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.800 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:04.800 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:04.801 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:04.801 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:04.801 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:04.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.802 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:04.802 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:04.802 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:04.802 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:04.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.803 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:04.803 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:04.803 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:04.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.804 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:04.804 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:04.804 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:04.804 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.804 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:04.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.805 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:04.805 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:04.805 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:04.806 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:04.806 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:04.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.806 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:04.807 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:04.807 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:04.807 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:04.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.807 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:04.807 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:04.808 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:04.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.808 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:04.808 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:04.809 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:04.809 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.809 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:04.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.809 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:04.810 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:04.810 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:04.810 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:04.810 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:04.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.811 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:04.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.811 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:04.811 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:04.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.811 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:04.812 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:04.812 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:04.812 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.812 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:04.812 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:17:04.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.813 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:04.813 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:04.813 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:17:04.813 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.814 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:04.814 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:04.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.814 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:04.814 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:04.815 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:17:04.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.815 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:04.815 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:17:04.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:04.816 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:04.816 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:04.816 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:04.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:04.816 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:04.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:04.817 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 10 librados: watch_flush enter 2023-10-30T12:17:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 10 librados: watch_flush exit 2023-10-30T12:17:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 20 client.214215.objecter shutdown clearing up homeless session... 2023-10-30T12:17:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 10 client.214215.objecter successfully canceled tick 2023-10-30T12:17:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe1d80392a0 msgr2=0x7fe1d803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe1d80392a0 0x7fe1d803b760 secure :-1 s=READY pgs=4239 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80a4430 tx=0x7fe1f0007cf0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 10 monclient: shutdown 2023-10-30T12:17:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:04.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809f4e0 msgr2=0x7fe1f8131340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 --2- 172.21.15.50:0/2420795110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f809f4e0 0x7fe1f8131340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00096c0 tx=0x7fe1e0009b80 comp rx=0 tx=0).stop 2023-10-30T12:17:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1f6d9d640 1 -- 172.21.15.50:0/2420795110 reap_dead start 2023-10-30T12:17:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1f6d9d640 1 -- 172.21.15.50:0/2420795110 reap_dead start 2023-10-30T12:17:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 shutdown_connections 2023-10-30T12:17:04.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 >> 172.21.15.50:0/2420795110 conn(0x7fe1f80a80b0 msgr2=0x7fe1f8134f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 shutdown_connections 2023-10-30T12:17:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 -- 172.21.15.50:0/2420795110 wait complete. 2023-10-30T12:17:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.678+0000 7fe1fc9c9640 1 librados: shutdown 2023-10-30T12:17:04.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:17:04.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:17:04.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:17:04.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.933+0000 7f58a436c640 10 monclient: get_monmap_and_config 2023-10-30T12:17:04.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.933+0000 7f58a436c640 10 monclient: build_initial_monmap 2023-10-30T12:17:04.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.933+0000 7f58a436c640 10 monclient: monmap: 2023-10-30T12:17:04.935 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:04.935 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:04.934567+0000 2023-10-30T12:17:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:04.934567+0000 2023-10-30T12:17:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:04.937 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:04.937 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:04.937 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:04.937 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:04.937 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 1 Processor -- start 2023-10-30T12:17:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 1 -- start start 2023-10-30T12:17:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient: init 2023-10-30T12:17:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-10-30T12:17:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c071c00 0x7f589c09a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient(hunting): picked mon.noname-b con 0x7f589c071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:04.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f589c097fe0 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:04.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f589c09b170 0x7f589c139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient(hunting): picked mon.noname-c con 0x7f589c09b170 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:04.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589c097600 con 0x7f589c097fe0 2023-10-30T12:17:04.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589c0978e0 con 0x7f589c09b170 2023-10-30T12:17:04.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c09bf50 con 0x7f589c071c00 2023-10-30T12:17:04.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a436c640 10 monclient(hunting): authenticate will time out at 1870.117731s 2023-10-30T12:17:04.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a2108640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c071c00 msgr2=0x7f589c09a150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:04.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.934+0000 7f58a2108640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c071c00 0x7f589c09a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:04.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f589c09b170 0x7f589c139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f589c09b170 0x7f589c139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:34340/0 (socket says 172.21.15.50:34340) 2023-10-30T12:17:04.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 -- 172.21.15.50:0/2750868350 learned_addr learned my addr 172.21.15.50:0/2750868350 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:04.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): get_auth_request con 0x7f589c09b170 auth_method 0 2023-10-30T12:17:04.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a1907640 1 --1- 172.21.15.50:0/2750868350 >> v1:172.21.15.118:6789/0 conn(0x7f589c097fe0 0x7f589c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:53864/0 (socket says 172.21.15.50:53864) 2023-10-30T12:17:04.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient(hunting): handle_auth_done global_id 204310 payload 274 2023-10-30T12:17:04.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 -- 172.21.15.50:0/2750868350 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c071c00 msgr2=0x7f589c09a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:04.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 --2- 172.21.15.50:0/2750868350 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c071c00 0x7f589c09a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 -- 172.21.15.50:0/2750868350 >> v1:172.21.15.118:6789/0 conn(0x7f589c097fe0 legacy=0x7f589c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 monclient: found mon.noname-c 2023-10-30T12:17:04.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:17:04.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.935+0000 7f58a2909640 1 -- 172.21.15.50:0/2750868350 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c09d9b0 con 0x7f589c09b170 2023-10-30T12:17:04.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a2909640 1 --2- 172.21.15.50:0/2750868350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c09b170 0x7f589c139760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58940083f0 tx=0x7f589400a870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=537b3df032535540 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 1 -- 172.21.15.50:0/2750868350 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589402c7f0 con 0x7f589c09b170 2023-10-30T12:17:04.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:04.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: dump: 2023-10-30T12:17:04.955 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.955 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.955 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.956 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.956 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.956 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.956 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.956 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.956 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.957 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 1 -- 172.21.15.50:0/2750868350 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589402c950 con 0x7f589c09b170 2023-10-30T12:17:04.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a436c640 5 monclient: authenticate success, global_id 204310 2023-10-30T12:17:04.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a436c640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:04.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 1 -- 172.21.15.50:0/2750868350 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589402ccc0 con 0x7f589c09b170 2023-10-30T12:17:04.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a436c640 4 set_mon_vals no callback set 2023-10-30T12:17:04.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:04.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.936+0000 7f58a1106640 10 monclient: dump: 2023-10-30T12:17:04.959 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.959 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.960 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.960 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.960 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.960 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.960 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.961 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.961 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.961 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 10 monclient: shutdown 2023-10-30T12:17:04.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:04.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 -- 172.21.15.50:0/2750868350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c09b170 msgr2=0x7f589c139760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 --2- 172.21.15.50:0/2750868350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c09b170 0x7f589c139760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58940083f0 tx=0x7f589400a870 comp rx=0 tx=0).stop 2023-10-30T12:17:04.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 -- 172.21.15.50:0/2750868350 shutdown_connections 2023-10-30T12:17:04.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 --2- 172.21.15.50:0/2750868350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c09b170 0x7f589c139760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 --2- 172.21.15.50:0/2750868350 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c071c00 0x7f589c09a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 -- 172.21.15.50:0/2750868350 >> 172.21.15.50:0/2750868350 conn(0x7f589c0a3980 msgr2=0x7f589c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:04.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 -- 172.21.15.50:0/2750868350 shutdown_connections 2023-10-30T12:17:04.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 1 -- 172.21.15.50:0/2750868350 wait complete. 2023-10-30T12:17:04.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 10 monclient: build_initial_monmap 2023-10-30T12:17:04.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.937+0000 7f58a436c640 10 monclient: monmap: 2023-10-30T12:17:04.966 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:04.966 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:04.966 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:04.967 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:04.967 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:04.967 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.967 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:04.967 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:04.968 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:04.968 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 librados: starting msgr at 2023-10-30T12:17:04.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 librados: starting objecter 2023-10-30T12:17:04.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 Processor -- start 2023-10-30T12:17:04.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 -- start start 2023-10-30T12:17:04.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 librados: setting wanted keys 2023-10-30T12:17:04.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 librados: calling monclient init 2023-10-30T12:17:04.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient: init 2023-10-30T12:17:04.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:04.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:17:04.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c071c00 0x7f589c08f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): picked mon.noname-b con 0x7f589c071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:04.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c097fe0 0x7f589c08f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): picked mon.noname-c con 0x7f589c097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:04.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f589c09b170 0x7f589c08fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): picked mon.noname-a con 0x7f589c09b170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:04.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c093f90 con 0x7f589c09b170 2023-10-30T12:17:04.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c092010 con 0x7f589c071c00 2023-10-30T12:17:04.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c091d30 con 0x7f589c097fe0 2023-10-30T12:17:04.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:04.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a436c640 10 monclient(hunting): authenticate will time out at 1870.121653s 2023-10-30T12:17:04.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a1907640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c097fe0 msgr2=0x7f589c08f920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:17:04.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a1907640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c097fe0 0x7f589c08f920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f589c09b170 0x7f589c08fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f589c09b170 0x7f589c08fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49260/0 (socket says 172.21.15.50:49260) 2023-10-30T12:17:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 1 -- 172.21.15.50:0/1301827978 learned_addr learned my addr 172.21.15.50:0/1301827978 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 10 monclient(hunting): get_auth_request con 0x7f589c09b170 auth_method 0 2023-10-30T12:17:04.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c071c00 0x7f589c08f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 10 monclient(hunting): get_auth_request con 0x7f589c071c00 auth_method 0 2023-10-30T12:17:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.938+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 10 monclient(hunting): handle_auth_done global_id 214224 payload 824 2023-10-30T12:17:04.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c097fe0 msgr2=0x7f589c08f920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f589c097fe0 0x7f589c08f920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c071c00 msgr2=0x7f589c08f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c071c00 0x7f589c08f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 10 monclient: _finish_hunting 0 2023-10-30T12:17:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 1 monclient: found mon.noname-a 2023-10-30T12:17:04.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 1 -- 172.21.15.50:0/1301827978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589400ab50 con 0x7f589c09b170 2023-10-30T12:17:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2108640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c071c00 0x7f589c08f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a2909640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f589c09b170 0x7f589c08fe60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f589400a8b0 tx=0x7f5894004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 client.?.objecter ms_handle_connect 0x7f589c09b170 2023-10-30T12:17:04.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:04.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894003e10 con 0x7f589c09b170 2023-10-30T12:17:04.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:04.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: dump: 2023-10-30T12:17:04.980 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.980 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.980 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.980 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.980 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.981 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.981 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.981 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.981 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.981 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: _finish_auth 0 2023-10-30T12:17:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5894004250 con 0x7f589c09b170 2023-10-30T12:17:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 5 monclient: authenticate success, global_id 214224 2023-10-30T12:17:04.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 10 monclient: _renew_subs 2023-10-30T12:17:04.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589c09cbc0 con 0x7f589c09b170 2023-10-30T12:17:04.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:04.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894004600 con 0x7f589c09b170 2023-10-30T12:17:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f588effd640 10 monclient: dump: 2023-10-30T12:17:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.984 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.985 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.986 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a0905640 4 set_mon_vals no callback set 2023-10-30T12:17:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 10 client.214224.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:04.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 10 monclient: _renew_subs 2023-10-30T12:17:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589c071720 con 0x7f589c09b170 2023-10-30T12:17:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.939+0000 7f58a436c640 1 librados: init done 2023-10-30T12:17:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a436c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a436c640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1ca6b0 0x7f589c1caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a436c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c073ff0 con 0x7f589c1ca6b0 2023-10-30T12:17:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f589403e450 con 0x7f589c09b170 2023-10-30T12:17:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1ca6b0 0x7f589c1caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient: get_auth_request con 0x7f589c1ca6b0 auth_method 0 2023-10-30T12:17:04.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f58800392a0 0x7f588003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:04.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(508..508 src has 417..508) v4 ==== 6558+0+0 (secure 0 0 0) 0x7f589406da70 con 0x7f589c09b170 2023-10-30T12:17:04.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 10 client.214224.objecter ms_dispatch 0x7f589c000b90 osd_map(508..508 src has 417..508) v4 2023-10-30T12:17:04.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 3 client.214224.objecter handle_osd_map got epochs [508,508] > 0 2023-10-30T12:17:04.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 3 client.214224.objecter handle_osd_map decoding full epoch 508 2023-10-30T12:17:04.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 20 client.214224.objecter dump_active .. 0 homeless 2023-10-30T12:17:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 10 monclient(hunting): handle_auth_done global_id 204319 payload 824 2023-10-30T12:17:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a1907640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f58800392a0 0x7f588003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 1 -- 172.21.15.50:0/1301827978 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f589c091bb0 con 0x7f589c1ca6b0 2023-10-30T12:17:04.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a1907640 10 monclient: get_auth_request con 0x7f58800392a0 auth_method 0 2023-10-30T12:17:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 10 client.214224.objecter ms_handle_connect 0x7f589c1ca6b0 2023-10-30T12:17:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f588effd640 10 client.214224.objecter resend_mon_ops 2023-10-30T12:17:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.940+0000 7f58a2108640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1ca6b0 0x7f589c1caa90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58980055f0 tx=0x7f589800e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898003b80 con 0x7f589c1ca6b0 2023-10-30T12:17:04.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 10 monclient: dump: 2023-10-30T12:17:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:04.994 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:04.995 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 10 client.214224.objecter ms_handle_connect 0x7f58800392a0 2023-10-30T12:17:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f58a1907640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f58800392a0 0x7f588003b760 secure :-1 s=READY pgs=4240 cs=0 l=1 rev1=1 crypto rx=0x7f589c0712c0 tx=0x7f5890007aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f589800eca0 con 0x7f589c1ca6b0 2023-10-30T12:17:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1ca6b0 msgr2=0x7f589c1caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1ca6b0 0x7f589c1caa90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58980055f0 tx=0x7f589800e5a0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1ca6b0 msgr2=0x7f589c1caa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:04.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.941+0000 7f588effd640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1ca6b0 0x7f589c1caa90 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58980055f0 tx=0x7f589800e5a0 comp rx=0 tx=0).stop 2023-10-30T12:17:04.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.947+0000 7f58a436c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.947+0000 7f58a436c640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1cafd0 0x7f589c06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.947+0000 7f58a436c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.947+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c071fe0 con 0x7f589c1cafd0 2023-10-30T12:17:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1cafd0 0x7f589c06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:04.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient: get_auth_request con 0x7f589c1cafd0 auth_method 0 2023-10-30T12:17:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 10 monclient(hunting): handle_auth_done global_id 204322 payload 824 2023-10-30T12:17:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 1 -- 172.21.15.50:0/1301827978 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f589c092250 con 0x7f589c1cafd0 2023-10-30T12:17:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f58a2108640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1cafd0 0x7f589c06dbb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5898018000 tx=0x7f5898005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f588effd640 10 client.214224.objecter ms_handle_connect 0x7f589c1cafd0 2023-10-30T12:17:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f588effd640 10 client.214224.objecter resend_mon_ops 2023-10-30T12:17:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898012800 con 0x7f589c1cafd0 2023-10-30T12:17:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f588effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f588effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.948+0000 7f588effd640 10 monclient: dump: 2023-10-30T12:17:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:05.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4451 (secure 0 0 0) 0x7f5898012960 con 0x7f589c1cafd0 2023-10-30T12:17:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f588effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f588effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1cafd0 msgr2=0x7f589c06dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f588effd640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1cafd0 0x7f589c06dbb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5898018000 tx=0x7f5898005a20 comp rx=0 tx=0).stop 2023-10-30T12:17:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f588effd640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1cafd0 msgr2=0x7f589c06dbb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f588effd640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f589c1cafd0 0x7f589c06dbb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.006 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:05.006 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:05.006 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:05.007 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:17:05.007 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 174, 2023-10-30T12:17:05.007 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:05.007 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:05.007 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:17:05.008 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:05.008 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:17:05.008 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:05.008 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:05.008 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:05.009 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:05.009 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:05.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:05.009 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:05.009 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:05.010 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:05.010 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:05.010 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:05.010 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:05.010 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:05.011 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:05.011 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:05.011 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:05.011 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:05.011 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:05.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:05.012 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:05.012 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:05.012 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:05.012 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:05.012 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.013 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:05.014 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:05.016 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:05.016 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:05.016 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:05.016 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:05.016 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:05.017 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:05.017 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:05.017 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:05.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:05.017 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:05.018 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:05.018 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:05.018 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:05.018 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:05.018 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:05.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.020 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:05.020 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:05.020 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:05.020 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:05.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:05.022 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:05.023 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:05.024 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:05.025 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:05.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.025 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:05.025 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:05.025 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:17:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 10 librados: watch_flush enter 2023-10-30T12:17:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 10 librados: watch_flush exit 2023-10-30T12:17:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 20 client.214224.objecter shutdown clearing up homeless session... 2023-10-30T12:17:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 10 client.214224.objecter successfully canceled tick 2023-10-30T12:17:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f58800392a0 msgr2=0x7f588003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f58800392a0 0x7f588003b760 secure :-1 s=READY pgs=4240 cs=0 l=1 rev1=1 crypto rx=0x7f589c0712c0 tx=0x7f5890007aa0 comp rx=0 tx=0).stop 2023-10-30T12:17:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 10 monclient: shutdown 2023-10-30T12:17:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f589c09b170 msgr2=0x7f589c08fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 --2- 172.21.15.50:0/1301827978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f589c09b170 0x7f589c08fe60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f589400a8b0 tx=0x7f5894004a40 comp rx=0 tx=0).stop 2023-10-30T12:17:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a2909640 1 -- 172.21.15.50:0/1301827978 reap_dead start 2023-10-30T12:17:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a2909640 1 -- 172.21.15.50:0/1301827978 reap_dead start 2023-10-30T12:17:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 shutdown_connections 2023-10-30T12:17:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 >> 172.21.15.50:0/1301827978 conn(0x7f589c0a3980 msgr2=0x7f589c0966e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 shutdown_connections 2023-10-30T12:17:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 -- 172.21.15.50:0/1301827978 wait complete. 2023-10-30T12:17:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:04.949+0000 7f58a436c640 1 librados: shutdown 2023-10-30T12:17:05.043 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:17:05.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:17:25.044 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:17:25.044 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-10-30T12:17:25.045 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:17:25.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:17:25.048 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:17:25.049 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:17:25.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:17:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.163+0000 7fd18dc10640 10 monclient: get_monmap_and_config 2023-10-30T12:17:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.163+0000 7fd18dc10640 10 monclient: build_initial_monmap 2023-10-30T12:17:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.163+0000 7fd18dc10640 10 monclient: monmap: 2023-10-30T12:17:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:25.164841+0000 2023-10-30T12:17:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:25.164841+0000 2023-10-30T12:17:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:25.168 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:25.168 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.164+0000 7fd18dc10640 1 Processor -- start 2023-10-30T12:17:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.164+0000 7fd18dc10640 1 -- start start 2023-10-30T12:17:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.164+0000 7fd18dc10640 10 monclient: init 2023-10-30T12:17:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T12:17:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 0x7fd188135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient(hunting): picked mon.noname-b con 0x7fd1880a7000 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd18806abe0 0x7fd1881360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient(hunting): picked mon.noname-a con 0x7fd18806abe0 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1880a4460 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1880a73e0 con 0x7fd1880a4460 2023-10-30T12:17:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd188136600 con 0x7fd18806abe0 2023-10-30T12:17:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18809a000 con 0x7fd1880a7000 2023-10-30T12:17:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd18dc10640 10 monclient(hunting): authenticate will time out at 1890.348137s 2023-10-30T12:17:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 0x7fd188135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd18806abe0 0x7fd1881360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 0x7fd188135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56560/0 (socket says 172.21.15.50:56560) 2023-10-30T12:17:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 1 -- 172.21.15.50:0/2200642292 learned_addr learned my addr 172.21.15.50:0/2200642292 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd18806abe0 0x7fd1881360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45030/0 (socket says 172.21.15.50:45030) 2023-10-30T12:17:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 10 monclient(hunting): get_auth_request con 0x7fd1880a7000 auth_method 0 2023-10-30T12:17:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): get_auth_request con 0x7fd18806abe0 auth_method 0 2023-10-30T12:17:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 1 -- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 msgr2=0x7fd188135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 1 -- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 msgr2=0x7fd188135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 1 --2- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 0x7fd188135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd1877fe640 1 --2- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 0x7fd188135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient(hunting): handle_auth_done global_id 214227 payload 274 2023-10-30T12:17:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 1 -- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 msgr2=0x7fd188135b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 1 --2- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 0x7fd188135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 1 -- 172.21.15.50:0/2200642292 >> v1:172.21.15.118:6789/0 conn(0x7fd1880a4460 legacy=0x7fd1880a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 1 monclient: found mon.noname-a 2023-10-30T12:17:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.165+0000 7fd186ffd640 1 -- 172.21.15.50:0/2200642292 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18809a2e0 con 0x7fd18806abe0 2023-10-30T12:17:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd186ffd640 1 --2- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd1881360c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0097a0 tx=0x7fd17c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5dd8ccc5140c8e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 1 -- 172.21.15.50:0/2200642292 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c02f050 con 0x7fd18806abe0 2023-10-30T12:17:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: dump: 2023-10-30T12:17:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:25.188 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:25.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:25.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd18dc10640 5 monclient: authenticate success, global_id 214227 2023-10-30T12:17:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd18dc10640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 1 -- 172.21.15.50:0/2200642292 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c031700 con 0x7fd18806abe0 2023-10-30T12:17:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd18dc10640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd18dc10640 4 set_mon_vals no callback set 2023-10-30T12:17:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 1 -- 172.21.15.50:0/2200642292 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c031a90 con 0x7fd18806abe0 2023-10-30T12:17:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.166+0000 7fd1867fc640 10 monclient: dump: 2023-10-30T12:17:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:25.193 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 10 monclient: shutdown 2023-10-30T12:17:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 -- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 msgr2=0x7fd1881360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 --2- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd1881360c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0097a0 tx=0x7fd17c02b7a0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 -- 172.21.15.50:0/2200642292 shutdown_connections 2023-10-30T12:17:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 --2- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd1881360c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 --2- 172.21.15.50:0/2200642292 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a7000 0x7fd188135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 -- 172.21.15.50:0/2200642292 >> 172.21.15.50:0/2200642292 conn(0x7fd188066000 msgr2=0x7fd1880a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 -- 172.21.15.50:0/2200642292 shutdown_connections 2023-10-30T12:17:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 1 -- 172.21.15.50:0/2200642292 wait complete. 2023-10-30T12:17:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 10 monclient: build_initial_monmap 2023-10-30T12:17:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.167+0000 7fd18dc10640 10 monclient: monmap: 2023-10-30T12:17:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:25.197 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 1 librados: starting msgr at 2023-10-30T12:17:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 1 librados: starting objecter 2023-10-30T12:17:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 1 Processor -- start 2023-10-30T12:17:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 1 -- start start 2023-10-30T12:17:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 1 librados: setting wanted keys 2023-10-30T12:17:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 1 librados: calling monclient init 2023-10-30T12:17:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 10 monclient: init 2023-10-30T12:17:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:17:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd18811e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.168+0000 7fd18dc10640 10 monclient(hunting): picked mon.noname-a con 0x7fd18806abe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 0x7fd18811e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient(hunting): picked mon.noname-c con 0x7fd1880a4460 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1880a7000 0x7fd18811eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient(hunting): picked mon.noname-b con 0x7fd1880a7000 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd188130170 con 0x7fd18806abe0 2023-10-30T12:17:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd18811e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd188121480 con 0x7fd1880a7000 2023-10-30T12:17:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd18811e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45046/0 (socket says 172.21.15.50:45046) 2023-10-30T12:17:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 -- 172.21.15.50:0/3755082530 learned_addr learned my addr 172.21.15.50:0/3755082530 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd188130f10 con 0x7fd1880a4460 2023-10-30T12:17:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 0x7fd18811e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient(hunting): authenticate will time out at 1890.352210s 2023-10-30T12:17:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): get_auth_request con 0x7fd18806abe0 auth_method 0 2023-10-30T12:17:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 10 monclient(hunting): get_auth_request con 0x7fd1880a4460 auth_method 0 2023-10-30T12:17:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 1 -- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 msgr2=0x7fd18811e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:17:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 1 -- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 msgr2=0x7fd18811e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 0x7fd18811e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd186ffd640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 0x7fd18811e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd187fff640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1880a7000 0x7fd18811eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd187fff640 10 monclient(hunting): get_auth_request con 0x7fd1880a7000 auth_method 0 2023-10-30T12:17:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd187fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd187fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient(hunting): handle_auth_done global_id 214230 payload 824 2023-10-30T12:17:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 -- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 msgr2=0x7fd18811e570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 0x7fd18811e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 -- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1880a7000 msgr2=0x7fd18811eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1880a7000 0x7fd18811eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 monclient: found mon.noname-a 2023-10-30T12:17:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 -- 172.21.15.50:0/3755082530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c009400 con 0x7fd18806abe0 2023-10-30T12:17:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd1877fe640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd18811e030 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e840 tx=0x7fd17800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 client.?.objecter ms_handle_connect 0x7fd18806abe0 2023-10-30T12:17:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 1 -- 172.21.15.50:0/3755082530 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178004020 con 0x7fd18806abe0 2023-10-30T12:17:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 monclient: dump: 2023-10-30T12:17:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:25.211 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 monclient: _finish_auth 0 2023-10-30T12:17:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:17:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 1 -- 172.21.15.50:0/3755082530 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd178004180 con 0x7fd18806abe0 2023-10-30T12:17:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd16ffff640 1 -- 172.21.15.50:0/3755082530 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17800c6d0 con 0x7fd18806abe0 2023-10-30T12:17:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 5 monclient: authenticate success, global_id 214230 2023-10-30T12:17:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient: _renew_subs 2023-10-30T12:17:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1880a1790 con 0x7fd18806abe0 2023-10-30T12:17:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.169+0000 7fd185ffb640 4 set_mon_vals no callback set 2023-10-30T12:17:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 10 client.214230.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 10 monclient: _renew_subs 2023-10-30T12:17:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18811fc60 con 0x7fd18806abe0 2023-10-30T12:17:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 1 librados: init done 2023-10-30T12:17:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd16ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd16ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd16ffff640 10 monclient: dump: 2023-10-30T12:17:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:25.216 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd16ffff640 1 -- 172.21.15.50:0/3755082530 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd17800c830 con 0x7fd18806abe0 2023-10-30T12:17:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.170+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188121230 con 0x7fd18806abe0 2023-10-30T12:17:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd16ffff640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd1740392f0 0x7fd17403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd16ffff640 1 -- 172.21.15.50:0/3755082530 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fd17804a890 con 0x7fd18806abe0 2023-10-30T12:17:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd16ffff640 10 client.214230.objecter ms_dispatch 0x7fd188000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd16ffff640 3 client.214230.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd16ffff640 3 client.214230.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd16ffff640 20 client.214230.objecter dump_active .. 0 homeless 2023-10-30T12:17:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd186ffd640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd1740392f0 0x7fd17403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd186ffd640 10 monclient: get_auth_request con 0x7fd1740392f0 auth_method 0 2023-10-30T12:17:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd16ffff640 10 client.214230.objecter ms_handle_connect 0x7fd1740392f0 2023-10-30T12:17:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.171+0000 7fd186ffd640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd1740392f0 0x7fd17403b7b0 secure :-1 s=READY pgs=4243 cs=0 l=1 rev1=1 crypto rx=0x7fd17c035000 tx=0x7fd17c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:25.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.175+0000 7fd16ffff640 1 -- 172.21.15.50:0/3755082530 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd17800ce60 con 0x7fd18806abe0 2023-10-30T12:17:25.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.175+0000 7fd16ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:25.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.175+0000 7fd16ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:25.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.419+0000 7fd18dc10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:17:25.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.419+0000 7fd18dc10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:25.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.419+0000 7fd18dc10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:25.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.419+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd18804e160 con 0x7fd18806abe0 2023-10-30T12:17:25.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.420+0000 7fd16ffff640 1 -- 172.21.15.50:0/3755082530 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd178022300 con 0x7fd18806abe0 2023-10-30T12:17:25.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.420+0000 7fd16ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:25.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.420+0000 7fd16ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:25.423 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":174,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:17:25.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.421+0000 7fd18dc10640 10 librados: watch_flush enter 2023-10-30T12:17:25.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.421+0000 7fd18dc10640 10 librados: watch_flush exit 2023-10-30T12:17:25.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.421+0000 7fd18dc10640 20 client.214230.objecter shutdown clearing up homeless session... 2023-10-30T12:17:25.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.421+0000 7fd18dc10640 10 client.214230.objecter successfully canceled tick 2023-10-30T12:17:25.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd1740392f0 msgr2=0x7fd17403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:25.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd1740392f0 0x7fd17403b7b0 secure :-1 s=READY pgs=4243 cs=0 l=1 rev1=1 crypto rx=0x7fd17c035000 tx=0x7fd17c008030 comp rx=0 tx=0).stop 2023-10-30T12:17:25.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 10 monclient: shutdown 2023-10-30T12:17:25.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:25.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 msgr2=0x7fd18811e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:25.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd18811e030 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e840 tx=0x7fd17800ed00 comp rx=0 tx=0).stop 2023-10-30T12:17:25.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 shutdown_connections 2023-10-30T12:17:25.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd1740392f0 0x7fd17403b7b0 unknown :-1 s=CLOSED pgs=4243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1880a7000 0x7fd18811eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd1880a4460 0x7fd18811e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 --2- 172.21.15.50:0/3755082530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd18806abe0 0x7fd18811e030 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:25.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 >> 172.21.15.50:0/3755082530 conn(0x7fd188066000 msgr2=0x7fd188131ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:25.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 shutdown_connections 2023-10-30T12:17:25.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 -- 172.21.15.50:0/3755082530 wait complete. 2023-10-30T12:17:25.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:25.422+0000 7fd18dc10640 1 librados: shutdown 2023-10-30T12:17:28.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:17:28.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.566+0000 7fb12016e640 10 monclient: get_monmap_and_config 2023-10-30T12:17:28.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.566+0000 7fb12016e640 10 monclient: build_initial_monmap 2023-10-30T12:17:28.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.566+0000 7fb12016e640 10 monclient: monmap: 2023-10-30T12:17:28.568 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:28.569 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.569 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:28.567527+0000 2023-10-30T12:17:28.569 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:28.567527+0000 2023-10-30T12:17:28.569 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:28.569 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:28.571 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 1 Processor -- start 2023-10-30T12:17:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 1 -- start start 2023-10-30T12:17:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient: init 2023-10-30T12:17:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-10-30T12:17:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1180a7000 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb11806abe0 0x7fb11813a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient(hunting): picked mon.noname-c con 0x7fb11806abe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb118136290 0x7fb11813cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient(hunting): picked mon.noname-a con 0x7fb118136290 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1180a73e0 con 0x7fb1180a7000 2023-10-30T12:17:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11809a000 con 0x7fb118136290 2023-10-30T12:17:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11809a2e0 con 0x7fb11806abe0 2023-10-30T12:17:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb12016e640 10 monclient(hunting): authenticate will time out at 1893.750824s 2023-10-30T12:17:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.567+0000 7fb11e70b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb118136290 0x7fb11813cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb118136290 0x7fb11813cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51444/0 (socket says 172.21.15.50:51444) 2023-10-30T12:17:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 1 -- 172.21.15.50:0/3017723721 learned_addr learned my addr 172.21.15.50:0/3017723721 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): get_auth_request con 0x7fb118136290 auth_method 0 2023-10-30T12:17:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11d709640 1 --2- 172.21.15.50:0/3017723721 >> v2:172.21.15.118:3300/0 conn(0x7fb11806abe0 0x7fb11813a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient(hunting): handle_auth_done global_id 214233 payload 274 2023-10-30T12:17:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 1 -- 172.21.15.50:0/3017723721 >> v2:172.21.15.118:3300/0 conn(0x7fb11806abe0 msgr2=0x7fb11813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 1 --2- 172.21.15.50:0/3017723721 >> v2:172.21.15.118:3300/0 conn(0x7fb11806abe0 0x7fb11813a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 1 -- 172.21.15.50:0/3017723721 >> v1:172.21.15.118:6789/0 conn(0x7fb1180a7000 legacy=0x7fb118135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient: _finish_hunting 0 2023-10-30T12:17:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 1 monclient: found mon.noname-a 2023-10-30T12:17:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.568+0000 7fb11e70b640 1 -- 172.21.15.50:0/3017723721 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11809a5c0 con 0x7fb118136290 2023-10-30T12:17:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11e70b640 1 --2- 172.21.15.50:0/3017723721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11813cee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb1180a1710 tx=0x7fb11002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15f53f6ad88bf98 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 1 -- 172.21.15.50:0/3017723721 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11002c9d0 con 0x7fb118136290 2023-10-30T12:17:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:28.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:28.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: dump: 2023-10-30T12:17:28.585 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:28.585 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.585 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.586 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.586 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:28.586 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.586 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:28.586 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:28.586 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:28.587 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: _finish_auth 0 2023-10-30T12:17:28.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: _check_auth_tickets 2023-10-30T12:17:28.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:28.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 5 monclient: authenticate success, global_id 214233 2023-10-30T12:17:28.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:28.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 1 -- 172.21.15.50:0/3017723721 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11002cb30 con 0x7fb118136290 2023-10-30T12:17:28.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:28.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:28.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 4 set_mon_vals no callback set 2023-10-30T12:17:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 1 -- 172.21.15.50:0/3017723721 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1100343f0 con 0x7fb118136290 2023-10-30T12:17:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb11cf08640 10 monclient: dump: 2023-10-30T12:17:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:28.591 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 10 monclient: shutdown 2023-10-30T12:17:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 1 -- 172.21.15.50:0/3017723721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 msgr2=0x7fb11813cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 1 --2- 172.21.15.50:0/3017723721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11813cee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb1180a1710 tx=0x7fb11002be30 comp rx=0 tx=0).stop 2023-10-30T12:17:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 1 -- 172.21.15.50:0/3017723721 shutdown_connections 2023-10-30T12:17:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 1 --2- 172.21.15.50:0/3017723721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11813cee0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 1 --2- 172.21.15.50:0/3017723721 >> v2:172.21.15.118:3300/0 conn(0x7fb11806abe0 0x7fb11813a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.569+0000 7fb12016e640 1 -- 172.21.15.50:0/3017723721 >> 172.21.15.50:0/3017723721 conn(0x7fb118066000 msgr2=0x7fb1180a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 -- 172.21.15.50:0/3017723721 shutdown_connections 2023-10-30T12:17:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 -- 172.21.15.50:0/3017723721 wait complete. 2023-10-30T12:17:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 10 monclient: build_initial_monmap 2023-10-30T12:17:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 10 monclient: monmap: 2023-10-30T12:17:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:28.595 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 librados: starting msgr at 2023-10-30T12:17:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 librados: starting objecter 2023-10-30T12:17:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 Processor -- start 2023-10-30T12:17:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 -- start start 2023-10-30T12:17:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 librados: setting wanted keys 2023-10-30T12:17:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 1 librados: calling monclient init 2023-10-30T12:17:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.570+0000 7fb12016e640 10 monclient: init 2023-10-30T12:17:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:17:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb11806abe0 0x7fb11809fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): picked mon.noname-b con 0x7fb11806abe0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 0x7fb11809b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1180a7000 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11809b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): picked mon.noname-a con 0x7fb118136290 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11809ec50 con 0x7fb118136290 2023-10-30T12:17:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11809f2f0 con 0x7fb11806abe0 2023-10-30T12:17:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11809dae0 con 0x7fb1180a7000 2023-10-30T12:17:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb12016e640 10 monclient(hunting): authenticate will time out at 1893.754405s 2023-10-30T12:17:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11809b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11809b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51458/0 (socket says 172.21.15.50:51458) 2023-10-30T12:17:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 0x7fb11809b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 -- 172.21.15.50:0/2507659020 learned_addr learned my addr 172.21.15.50:0/2507659020 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): get_auth_request con 0x7fb118136290 auth_method 0 2023-10-30T12:17:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 10 monclient(hunting): get_auth_request con 0x7fb1180a7000 auth_method 0 2023-10-30T12:17:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 1 -- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 msgr2=0x7fb11809b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:17:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 1 -- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 msgr2=0x7fb11809b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 0x7fb11809b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11d709640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 0x7fb11809b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11df0a640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb11806abe0 0x7fb11809fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient(hunting): handle_auth_done global_id 214236 payload 824 2023-10-30T12:17:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 -- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 msgr2=0x7fb11809b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 0x7fb11809b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 -- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb11806abe0 msgr2=0x7fb11809fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb11806abe0 0x7fb11809fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient: _finish_hunting 0 2023-10-30T12:17:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 monclient: found mon.noname-a 2023-10-30T12:17:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 -- 172.21.15.50:0/2507659020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110009b30 con 0x7fb118136290 2023-10-30T12:17:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11df0a640 10 monclient: get_auth_request con 0x7fb11806abe0 auth_method 0 2023-10-30T12:17:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11df0a640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb11806abe0 0x7fb11809fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:17:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb11e70b640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11809b9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb110036040 tx=0x7fb110004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.571+0000 7fb10e7fc640 10 client.?.objecter ms_handle_connect 0x7fb118136290 2023-10-30T12:17:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 1 -- 172.21.15.50:0/2507659020 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11003e450 con 0x7fb118136290 2023-10-30T12:17:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: dump: 2023-10-30T12:17:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:28.608 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 5 monclient: authenticate success, global_id 214236 2023-10-30T12:17:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 1 -- 172.21.15.50:0/2507659020 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11003e5d0 con 0x7fb118136290 2023-10-30T12:17:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 10 monclient: _renew_subs 2023-10-30T12:17:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 1 -- 172.21.15.50:0/2507659020 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11003e980 con 0x7fb118136290 2023-10-30T12:17:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1180a1790 con 0x7fb118136290 2023-10-30T12:17:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10ffff640 4 set_mon_vals no callback set 2023-10-30T12:17:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb10e7fc640 10 monclient: dump: 2023-10-30T12:17:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:28.613 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 10 client.214236.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 10 monclient: _renew_subs 2023-10-30T12:17:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11813c930 con 0x7fb118136290 2023-10-30T12:17:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.572+0000 7fb12016e640 1 librados: init done 2023-10-30T12:17:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb10e7fc640 1 -- 172.21.15.50:0/2507659020 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb11003eae0 con 0x7fb118136290 2023-10-30T12:17:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb12016e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb12016e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb12016e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11804e160 con 0x7fb118136290 2023-10-30T12:17:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb10e7fc640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0fc0392f0 0x7fb0fc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb10e7fc640 1 -- 172.21.15.50:0/2507659020 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fb11006dc80 con 0x7fb118136290 2023-10-30T12:17:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb10e7fc640 10 client.214236.objecter ms_dispatch 0x7fb118000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:28.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb10e7fc640 3 client.214236.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:28.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb10e7fc640 3 client.214236.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:28.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb10e7fc640 20 client.214236.objecter dump_active .. 0 homeless 2023-10-30T12:17:28.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb11df0a640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0fc0392f0 0x7fb0fc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.573+0000 7fb11df0a640 10 monclient: get_auth_request con 0x7fb0fc0392f0 auth_method 0 2023-10-30T12:17:28.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.574+0000 7fb11df0a640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0fc0392f0 0x7fb0fc03b7b0 secure :-1 s=READY pgs=4244 cs=0 l=1 rev1=1 crypto rx=0x7fb114003eb0 tx=0x7fb114009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:28.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.574+0000 7fb10e7fc640 10 client.214236.objecter ms_handle_connect 0x7fb0fc0392f0 2023-10-30T12:17:28.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.578+0000 7fb10e7fc640 1 -- 172.21.15.50:0/2507659020 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb110073020 con 0x7fb118136290 2023-10-30T12:17:28.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.578+0000 7fb10e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:28.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.578+0000 7fb10e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:28.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.821+0000 7fb12016e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:17:28.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.821+0000 7fb12016e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.821+0000 7fb12016e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.821+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb1180745c0 con 0x7fb118136290 2023-10-30T12:17:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.821+0000 7fb10e7fc640 1 -- 172.21.15.50:0/2507659020 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fb110043070 con 0x7fb118136290 2023-10-30T12:17:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.821+0000 7fb10e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.821+0000 7fb10e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:28.824 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:17:28.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 10 librados: watch_flush enter 2023-10-30T12:17:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 10 librados: watch_flush exit 2023-10-30T12:17:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 20 client.214236.objecter shutdown clearing up homeless session... 2023-10-30T12:17:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 10 client.214236.objecter successfully canceled tick 2023-10-30T12:17:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0fc0392f0 msgr2=0x7fb0fc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0fc0392f0 0x7fb0fc03b7b0 secure :-1 s=READY pgs=4244 cs=0 l=1 rev1=1 crypto rx=0x7fb114003eb0 tx=0x7fb114009040 comp rx=0 tx=0).stop 2023-10-30T12:17:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 10 monclient: shutdown 2023-10-30T12:17:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 msgr2=0x7fb11809b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11809b9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb110036040 tx=0x7fb110004a40 comp rx=0 tx=0).stop 2023-10-30T12:17:28.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 shutdown_connections 2023-10-30T12:17:28.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb0fc0392f0 0x7fb0fc03b7b0 unknown :-1 s=CLOSED pgs=4244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb118136290 0x7fb11809b9c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1180a7000 0x7fb11809b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 --2- 172.21.15.50:0/2507659020 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb11806abe0 0x7fb11809fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 >> 172.21.15.50:0/2507659020 conn(0x7fb118066000 msgr2=0x7fb1180a3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 shutdown_connections 2023-10-30T12:17:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 -- 172.21.15.50:0/2507659020 wait complete. 2023-10-30T12:17:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.823+0000 7fb12016e640 1 librados: shutdown 2023-10-30T12:17:28.835 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:17:28.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:17:28.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:17:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.940+0000 7fc035b7d640 10 monclient: get_monmap_and_config 2023-10-30T12:17:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.940+0000 7fc035b7d640 10 monclient: build_initial_monmap 2023-10-30T12:17:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.940+0000 7fc035b7d640 10 monclient: monmap: 2023-10-30T12:17:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:28.941390+0000 2023-10-30T12:17:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:28.941390+0000 2023-10-30T12:17:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:28.944 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 1 Processor -- start 2023-10-30T12:17:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 1 -- start start 2023-10-30T12:17:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient: init 2023-10-30T12:17:28.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-10-30T12:17:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc03009bb70 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc03009d3f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc03013b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient(hunting): picked mon.noname-b con 0x7fc03009c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc030098170 con 0x7fc03009bb70 2023-10-30T12:17:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03009cf60 con 0x7fc03009d3f0 2023-10-30T12:17:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc030097b20 con 0x7fc03009c660 2023-10-30T12:17:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc035b7d640 10 monclient(hunting): authenticate will time out at 1894.124603s 2023-10-30T12:17:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc03013b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc03009bb70 0x7fc03009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45516/0 (socket says 172.21.15.50:45516) 2023-10-30T12:17:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02f7fe640 1 -- 172.21.15.50:0/1170426818 learned_addr learned my addr 172.21.15.50:0/1170426818 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:17:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 10 monclient(hunting): get_auth_request con 0x7fc03009c660 auth_method 0 2023-10-30T12:17:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 1 -- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 msgr2=0x7fc03013b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 1 -- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 msgr2=0x7fc03013b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc03013b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.941+0000 7fc02ffff640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc03013b090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614268118 0 0) 0x7fc030098170 con 0x7fc03009bb70 2023-10-30T12:17:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:17:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): my global_id is 214239 2023-10-30T12:17:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc01c005730 con 0x7fc03009bb70 2023-10-30T12:17:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279714111 0 0) 0x7fc03009cf60 con 0x7fc03009d3f0 2023-10-30T12:17:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:17:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): my global_id is 204328 2023-10-30T12:17:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030098170 con 0x7fc03009d3f0 2023-10-30T12:17:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431517035 0 0) 0x7fc01c005730 con 0x7fc03009bb70 2023-10-30T12:17:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 msgr2=0x7fc03013b090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc03013b090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 >> v1:172.21.15.118:6789/0 conn(0x7fc03009d3f0 legacy=0x7fc030137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:17:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:17:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:17:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030097e00 con 0x7fc03009bb70 2023-10-30T12:17:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fc0180017f0 con 0x7fc03009bb70 2023-10-30T12:17:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc035b7d640 5 monclient: authenticate success, global_id 214239 2023-10-30T12:17:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:17:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: dump: 2023-10-30T12:17:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:28.967 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:17:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 >> v1:172.21.15.50:6789/0 conn(0x7fc03009bb70 legacy=0x7fc03009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc01c005d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): picked mon.c con 0x7fc03009c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc01c0062a0 0x7fc01c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): picked mon.a con 0x7fc01c0062a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c006f60 0x7fc01c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): picked mon.b con 0x7fc01c006f60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc030097b20 con 0x7fc01c0062a0 2023-10-30T12:17:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c00de40 con 0x7fc01c006f60 2023-10-30T12:17:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c00e000 con 0x7fc03009c660 2023-10-30T12:17:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02e7fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:17:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc035b7d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc01c005d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc01c0062a0 0x7fc01c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 10 monclient(hunting): get_auth_request con 0x7fc03009c660 auth_method 0 2023-10-30T12:17:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): get_auth_request con 0x7fc01c0062a0 auth_method 0 2023-10-30T12:17:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c006f60 0x7fc01c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 10 monclient(hunting): get_auth_request con 0x7fc01c006f60 auth_method 0 2023-10-30T12:17:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient(hunting): handle_auth_done global_id 214239 payload 293 2023-10-30T12:17:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 1 -- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 msgr2=0x7fc01c005d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc01c005d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 1 -- 172.21.15.50:0/1170426818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c006f60 msgr2=0x7fc01c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c006f60 0x7fc01c0073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 1 monclient: found mon.a 2023-10-30T12:17:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 1 -- 172.21.15.50:0/1170426818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c005730 con 0x7fc01c0062a0 2023-10-30T12:17:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient: _finish_auth 0 2023-10-30T12:17:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.942+0000 7fc02f7fe640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03009c660 0x7fc01c005d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02effd640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc01c0062a0 0x7fc01c0069d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc0200018d0 tx=0x7fc01c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7801681969a202 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0200052d0 con 0x7fc01c0062a0 2023-10-30T12:17:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 10 monclient: dump: 2023-10-30T12:17:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:28.981 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc035b7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc020005430 con 0x7fc01c0062a0 2023-10-30T12:17:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc035b7d640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc035b7d640 4 set_mon_vals no callback set 2023-10-30T12:17:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 1 -- 172.21.15.50:0/1170426818 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020003810 con 0x7fc01c0062a0 2023-10-30T12:17:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.943+0000 7fc02e7fc640 10 monclient: dump: 2023-10-30T12:17:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:28.984 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 10 monclient: shutdown 2023-10-30T12:17:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 1 -- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc01c0062a0 msgr2=0x7fc01c0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 1 --2- 172.21.15.50:0/1170426818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc01c0062a0 0x7fc01c0069d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc0200018d0 tx=0x7fc01c007a00 comp rx=0 tx=0).stop 2023-10-30T12:17:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc02ffff640 1 -- 172.21.15.50:0/1170426818 reap_dead start 2023-10-30T12:17:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 1 -- 172.21.15.50:0/1170426818 shutdown_connections 2023-10-30T12:17:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 1 -- 172.21.15.50:0/1170426818 >> 172.21.15.50:0/1170426818 conn(0x7fc0300a3980 msgr2=0x7fc0300a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 1 -- 172.21.15.50:0/1170426818 shutdown_connections 2023-10-30T12:17:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.944+0000 7fc035b7d640 1 -- 172.21.15.50:0/1170426818 wait complete. 2023-10-30T12:17:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 10 monclient: build_initial_monmap 2023-10-30T12:17:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 10 monclient: monmap: 2023-10-30T12:17:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:28.988 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 1 librados: starting msgr at 2023-10-30T12:17:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 1 librados: starting objecter 2023-10-30T12:17:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 1 Processor -- start 2023-10-30T12:17:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 1 -- start start 2023-10-30T12:17:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 1 librados: setting wanted keys 2023-10-30T12:17:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 1 librados: calling monclient init 2023-10-30T12:17:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.945+0000 7fc035b7d640 10 monclient: init 2023-10-30T12:17:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:17:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c0062a0 0x7fc030070e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): picked mon.noname-b con 0x7fc01c0062a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0301c6bd0 0x7fc030094710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): picked mon.noname-a con 0x7fc0301c6bd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 0x7fc0301c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): picked mon.noname-c con 0x7fc0301c7050 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc030071860 con 0x7fc0301c6bd0 2023-10-30T12:17:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc030092c20 con 0x7fc01c0062a0 2023-10-30T12:17:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc030092f00 con 0x7fc0301c7050 2023-10-30T12:17:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc035b7d640 10 monclient(hunting): authenticate will time out at 1894.129157s 2023-10-30T12:17:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 0x7fc0301c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0301c6bd0 0x7fc030094710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 0x7fc0301c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52078/0 (socket says 172.21.15.50:52078) 2023-10-30T12:17:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0301c6bd0 0x7fc030094710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51480/0 (socket says 172.21.15.50:51480) 2023-10-30T12:17:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 1 -- 172.21.15.50:0/2930336566 learned_addr learned my addr 172.21.15.50:0/2930336566 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 10 monclient(hunting): get_auth_request con 0x7fc0301c7050 auth_method 0 2023-10-30T12:17:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): get_auth_request con 0x7fc0301c6bd0 auth_method 0 2023-10-30T12:17:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c0062a0 0x7fc030070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 msgr2=0x7fc0301c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:17:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 msgr2=0x7fc0301c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 10 monclient(hunting): get_auth_request con 0x7fc01c0062a0 auth_method 0 2023-10-30T12:17:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 0x7fc0301c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02ffff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 0x7fc0301c9f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient(hunting): handle_auth_done global_id 214242 payload 824 2023-10-30T12:17:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 msgr2=0x7fc0301c9f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0301c7050 0x7fc0301c9f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c0062a0 msgr2=0x7fc030070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc01c0062a0 0x7fc030070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 monclient: found mon.noname-a 2023-10-30T12:17:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 -- 172.21.15.50:0/2930336566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300976e0 con 0x7fc0301c6bd0 2023-10-30T12:17:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc02effd640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0301c6bd0 0x7fc030094710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc020004000 tx=0x7fc0200012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc027fff640 10 client.?.objecter ms_handle_connect 0x7fc0301c6bd0 2023-10-30T12:17:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.946+0000 7fc027fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020007250 con 0x7fc0301c6bd0 2023-10-30T12:17:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: dump: 2023-10-30T12:17:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.005 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: _finish_auth 0 2023-10-30T12:17:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: _check_auth_tickets 2023-10-30T12:17:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 5 monclient: authenticate success, global_id 214242 2023-10-30T12:17:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0200073b0 con 0x7fc0301c6bd0 2023-10-30T12:17:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 10 monclient: _renew_subs 2023-10-30T12:17:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0300931e0 con 0x7fc0301c6bd0 2023-10-30T12:17:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020007720 con 0x7fc0301c6bd0 2023-10-30T12:17:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 10 monclient: dump: 2023-10-30T12:17:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.010 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc02dffb640 4 set_mon_vals no callback set 2023-10-30T12:17:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 10 client.214242.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 10 monclient: _renew_subs 2023-10-30T12:17:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc030094e30 con 0x7fc0301c6bd0 2023-10-30T12:17:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 1 librados: init done 2023-10-30T12:17:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc0200035f0 con 0x7fc0301c6bd0 2023-10-30T12:17:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002b430 0x7fc03002b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.947+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc030073ff0 con 0x7fc03002b430 2023-10-30T12:17:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002b430 0x7fc03002b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient: get_auth_request con 0x7fc03002b430 auth_method 0 2023-10-30T12:17:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc0100392f0 0x7fc01003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fc02006c880 con 0x7fc0301c6bd0 2023-10-30T12:17:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 client.214242.objecter ms_dispatch 0x7fc030000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 3 client.214242.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 3 client.214242.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 20 client.214242.objecter dump_active .. 0 homeless 2023-10-30T12:17:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_done global_id 214245 payload 824 2023-10-30T12:17:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02ffff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc0100392f0 0x7fc01003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 1 -- 172.21.15.50:0/2930336566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc030097560 con 0x7fc03002b430 2023-10-30T12:17:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02ffff640 10 monclient: get_auth_request con 0x7fc0100392f0 auth_method 0 2023-10-30T12:17:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 client.214242.objecter ms_handle_connect 0x7fc03002b430 2023-10-30T12:17:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 client.214242.objecter resend_mon_ops 2023-10-30T12:17:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02f7fe640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002b430 0x7fc03002b810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc018007c30 tx=0x7fc0180018e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018010eb0 con 0x7fc03002b430 2023-10-30T12:17:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 monclient: dump: 2023-10-30T12:17:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.018 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 client.214242.objecter ms_handle_connect 0x7fc0100392f0 2023-10-30T12:17:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc02ffff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc0100392f0 0x7fc01003b7b0 secure :-1 s=READY pgs=4245 cs=0 l=1 rev1=1 crypto rx=0x7fc03009cd10 tx=0x7fc028009770 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc018007e00 con 0x7fc03002b430 2023-10-30T12:17:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002b430 msgr2=0x7fc03002b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002b430 0x7fc03002b810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc018007c30 tx=0x7fc0180018e0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002b430 msgr2=0x7fc03002b810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.948+0000 7fc027fff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002b430 0x7fc03002b810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc035b7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc035b7d640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002bd50 0x7fc03002c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc035b7d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03009f600 con 0x7fc03002bd50 2023-10-30T12:17:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002bd50 0x7fc03002c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient: get_auth_request con 0x7fc03002bd50 auth_method 0 2023-10-30T12:17:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 10 monclient(hunting): handle_auth_done global_id 214248 payload 824 2023-10-30T12:17:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 1 -- 172.21.15.50:0/2930336566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc03004e160 con 0x7fc03002bd50 2023-10-30T12:17:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc027fff640 10 client.214242.objecter ms_handle_connect 0x7fc03002bd50 2023-10-30T12:17:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc027fff640 10 client.214242.objecter resend_mon_ops 2023-10-30T12:17:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc02f7fe640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002bd50 0x7fc03002c160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc018011010 tx=0x7fc018001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018026070 con 0x7fc03002bd50 2023-10-30T12:17:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc027fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.955+0000 7fc027fff640 10 monclient: dump: 2023-10-30T12:17:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.027 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.027 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.027 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.027 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.027 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.028 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7fc018021860 con 0x7fc03002bd50 2023-10-30T12:17:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc027fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc027fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002bd50 msgr2=0x7fc03002c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc027fff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002bd50 0x7fc03002c160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc018011010 tx=0x7fc018001b00 comp rx=0 tx=0).stop 2023-10-30T12:17:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc027fff640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002bd50 msgr2=0x7fc03002c160 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc027fff640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc03002bd50 0x7fc03002c160 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.029 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:17:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 176, 2023-10-30T12:17:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:17:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:17:29.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.058 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:29.058 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:29.058 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:17:29.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.059 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:29.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:29.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.059 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:29.059 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:29.060 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:17:29.060 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.060 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:29.060 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:17:29.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:29.060 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:29.061 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:29.061 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:29.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:29.061 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:29.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:29.062 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:29.062 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:29.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 10 librados: watch_flush enter 2023-10-30T12:17:29.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 10 librados: watch_flush exit 2023-10-30T12:17:29.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 20 client.214242.objecter shutdown clearing up homeless session... 2023-10-30T12:17:29.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 10 client.214242.objecter successfully canceled tick 2023-10-30T12:17:29.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc0100392f0 msgr2=0x7fc01003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc0100392f0 0x7fc01003b7b0 secure :-1 s=READY pgs=4245 cs=0 l=1 rev1=1 crypto rx=0x7fc03009cd10 tx=0x7fc028009770 comp rx=0 tx=0).stop 2023-10-30T12:17:29.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 10 monclient: shutdown 2023-10-30T12:17:29.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:29.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0301c6bd0 msgr2=0x7fc030094710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0301c6bd0 0x7fc030094710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc020004000 tx=0x7fc0200012d0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc02ffff640 1 -- 172.21.15.50:0/2930336566 reap_dead start 2023-10-30T12:17:29.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 shutdown_connections 2023-10-30T12:17:29.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 --2- 172.21.15.50:0/2930336566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0301c6bd0 0x7fc030094710 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 >> 172.21.15.50:0/2930336566 conn(0x7fc0300a3980 msgr2=0x7fc030096500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:29.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 shutdown_connections 2023-10-30T12:17:29.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 -- 172.21.15.50:0/2930336566 wait complete. 2023-10-30T12:17:29.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:28.956+0000 7fc035b7d640 1 librados: shutdown 2023-10-30T12:17:29.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:17:29.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:17:29.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:17:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.172+0000 7fdc92840640 10 monclient: get_monmap_and_config 2023-10-30T12:17:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.172+0000 7fdc92840640 10 monclient: build_initial_monmap 2023-10-30T12:17:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.172+0000 7fdc92840640 10 monclient: monmap: 2023-10-30T12:17:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:29.173945+0000 2023-10-30T12:17:29.175 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:29.173945+0000 2023-10-30T12:17:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:29.177 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.173+0000 7fdc92840640 1 Processor -- start 2023-10-30T12:17:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.173+0000 7fdc92840640 1 -- start start 2023-10-30T12:17:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.173+0000 7fdc92840640 10 monclient: init 2023-10-30T12:17:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:17:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc8c097780 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc8c073b10 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c13e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient(hunting): picked mon.noname-b con 0x7fdc8c09ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c071ec0 con 0x7fdc8c097780 2023-10-30T12:17:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c09bd10 con 0x7fdc8c073b10 2023-10-30T12:17:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c09bff0 con 0x7fdc8c09ccd0 2023-10-30T12:17:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 10 monclient(hunting): authenticate will time out at 1894.357108s 2023-10-30T12:17:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdc8c097780 0x7fdc8c097b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45528/0 (socket says 172.21.15.50:45528) 2023-10-30T12:17:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c13e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3562125880 learned_addr learned my addr 172.21.15.50:0/3562125880 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:17:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 10 monclient(hunting): get_auth_request con 0x7fdc8c09ccd0 auth_method 0 2023-10-30T12:17:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 msgr2=0x7fdc8c13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:17:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 msgr2=0x7fdc8c13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c13e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c13e1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48738927 0 0) 0x7fdc8c071ec0 con 0x7fdc8c097780 2023-10-30T12:17:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): _negotiate 2023-10-30T12:17:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): my global_id is 214251 2023-10-30T12:17:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc74003520 con 0x7fdc8c097780 2023-10-30T12:17:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170313282 0 0) 0x7fdc74003520 con 0x7fdc8c097780 2023-10-30T12:17:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 msgr2=0x7fdc8c13e1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c13e1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 >> v1:172.21.15.118:6789/0 conn(0x7fdc8c073b10 legacy=0x7fdc8c13aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:17:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:17:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c071b20 con 0x7fdc8c097780 2023-10-30T12:17:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: _finish_auth 0 2023-10-30T12:17:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fdc80001670 con 0x7fdc8c097780 2023-10-30T12:17:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 5 monclient: authenticate success, global_id 214251 2023-10-30T12:17:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:17:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: dump: 2023-10-30T12:17:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.198 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:17:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 >> v1:172.21.15.50:6789/0 conn(0x7fdc8c097780 legacy=0x7fdc8c097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:17:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc740039a0 0x7fdc74003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): picked mon.b con 0x7fdc740039a0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): picked mon.c con 0x7fdc8c09ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc74004d50 0x7fdc740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): picked mon.a con 0x7fdc74004d50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c09bff0 con 0x7fdc74004d50 2023-10-30T12:17:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc74014440 con 0x7fdc740039a0 2023-10-30T12:17:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc74014600 con 0x7fdc8c09ccd0 2023-10-30T12:17:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc8affd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:17:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.174+0000 7fdc92840640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc74004d50 0x7fdc740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): get_auth_request con 0x7fdc8c09ccd0 auth_method 0 2023-10-30T12:17:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc740039a0 0x7fdc74003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 10 monclient(hunting): get_auth_request con 0x7fdc74004d50 auth_method 0 2023-10-30T12:17:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request con 0x7fdc740039a0 auth_method 0 2023-10-30T12:17:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient(hunting): handle_auth_done global_id 214251 payload 293 2023-10-30T12:17:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 1 -- 172.21.15.50:0/3562125880 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc740039a0 msgr2=0x7fdc74003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc740039a0 0x7fdc74003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 1 -- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc74004d50 msgr2=0x7fdc740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc74004d50 0x7fdc740051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 1 monclient: found mon.c 2023-10-30T12:17:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 1 -- 172.21.15.50:0/3562125880 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc74003520 con 0x7fdc8c09ccd0 2023-10-30T12:17:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.175+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc74004d50 0x7fdc740051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8b7fe640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc740047c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c001720 tx=0x7fdc740057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6507dbf139d3968a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0052d0 con 0x7fdc8c09ccd0 2023-10-30T12:17:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 10 monclient: dump: 2023-10-30T12:17:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.212 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c02e050 con 0x7fdc8c09ccd0 2023-10-30T12:17:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 1 -- 172.21.15.50:0/3562125880 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0035f0 con 0x7fdc8c09ccd0 2023-10-30T12:17:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 4 set_mon_vals no callback set 2023-10-30T12:17:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc8affd640 10 monclient: dump: 2023-10-30T12:17:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.215 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 10 monclient: shutdown 2023-10-30T12:17:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 1 -- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 msgr2=0x7fdc740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 1 --2- 172.21.15.50:0/3562125880 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc740047c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c001720 tx=0x7fdc740057f0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3562125880 reap_dead start 2023-10-30T12:17:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 1 -- 172.21.15.50:0/3562125880 shutdown_connections 2023-10-30T12:17:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 1 -- 172.21.15.50:0/3562125880 >> 172.21.15.50:0/3562125880 conn(0x7fdc8c0a1170 msgr2=0x7fdc8c0a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 1 -- 172.21.15.50:0/3562125880 shutdown_connections 2023-10-30T12:17:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.176+0000 7fdc92840640 1 -- 172.21.15.50:0/3562125880 wait complete. 2023-10-30T12:17:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.177+0000 7fdc92840640 10 monclient: build_initial_monmap 2023-10-30T12:17:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.177+0000 7fdc92840640 10 monclient: monmap: 2023-10-30T12:17:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:29.219 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.177+0000 7fdc92840640 1 librados: starting msgr at 2023-10-30T12:17:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.177+0000 7fdc92840640 1 librados: starting objecter 2023-10-30T12:17:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.177+0000 7fdc92840640 1 Processor -- start 2023-10-30T12:17:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 -- start start 2023-10-30T12:17:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 librados: setting wanted keys 2023-10-30T12:17:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 librados: calling monclient init 2023-10-30T12:17:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient: init 2023-10-30T12:17:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:17:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): picked mon.noname-c con 0x7fdc8c09ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8c1c28d0 0x7fdc8c1c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): picked mon.noname-b con 0x7fdc8c1c28d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8c1c3270 0x7fdc8c1c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): picked mon.noname-a con 0x7fdc8c1c3270 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c08dfa0 con 0x7fdc8c1c3270 2023-10-30T12:17:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c1c36f0 con 0x7fdc8c1c28d0 2023-10-30T12:17:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c094ec0 con 0x7fdc8c09ccd0 2023-10-30T12:17:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc92840640 10 monclient(hunting): authenticate will time out at 1894.361543s 2023-10-30T12:17:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c09aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52100/0 (socket says 172.21.15.50:52100) 2023-10-30T12:17:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 learned_addr learned my addr 172.21.15.50:0/3603889354 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request con 0x7fdc8c09ccd0 auth_method 0 2023-10-30T12:17:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8c1c3270 0x7fdc8c1c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 msgr2=0x7fdc8c09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:17:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): get_auth_request con 0x7fdc8c1c3270 auth_method 0 2023-10-30T12:17:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 msgr2=0x7fdc8c09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c09aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c09aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8b7fe640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8c1c28d0 0x7fdc8c1c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8b7fe640 10 monclient(hunting): get_auth_request con 0x7fdc8c1c28d0 auth_method 0 2023-10-30T12:17:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc8b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient(hunting): handle_auth_done global_id 214254 payload 824 2023-10-30T12:17:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 msgr2=0x7fdc8c09aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c09ccd0 0x7fdc8c09aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8c1c28d0 msgr2=0x7fdc8c1c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:29.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc8c1c28d0 0x7fdc8c1c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:29.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:29.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.178+0000 7fdc90ddd640 1 monclient: found mon.noname-a 2023-10-30T12:17:29.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc90ddd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:29.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc90ddd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3603889354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c08dcc0 con 0x7fdc8c1c3270 2023-10-30T12:17:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc90ddd640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8c1c3270 0x7fdc8c1c5e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0725c0 tx=0x7fdc840077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 client.?.objecter ms_handle_connect 0x7fdc8c1c3270 2023-10-30T12:17:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84013780 con 0x7fdc8c1c3270 2023-10-30T12:17:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: dump: 2023-10-30T12:17:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.233 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: _finish_auth 0 2023-10-30T12:17:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:17:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc840138e0 con 0x7fdc8c1c3270 2023-10-30T12:17:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 5 monclient: authenticate success, global_id 214254 2023-10-30T12:17:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 10 monclient: _renew_subs 2023-10-30T12:17:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c08d9e0 con 0x7fdc8c1c3270 2023-10-30T12:17:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84013c90 con 0x7fdc8c1c3270 2023-10-30T12:17:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc88ff9640 10 monclient: dump: 2023-10-30T12:17:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:29.237 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc8a7fc640 4 set_mon_vals no callback set 2023-10-30T12:17:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 10 client.214254.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 10 monclient: _renew_subs 2023-10-30T12:17:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c1c6530 con 0x7fdc8c1c3270 2023-10-30T12:17:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.179+0000 7fdc92840640 1 librados: init done 2023-10-30T12:17:29.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc92840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc92840640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc92840640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c06a5a0 con 0x7fdc8c06d4a0 2023-10-30T12:17:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdc84014410 con 0x7fdc8c1c3270 2023-10-30T12:17:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 10 monclient: get_auth_request con 0x7fdc8c06d4a0 auth_method 0 2023-10-30T12:17:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:17:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:29.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:29.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:29.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc88ff9640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc780392f0 0x7fdc7803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:29.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7fdc8404b8a0 con 0x7fdc8c1c3270 2023-10-30T12:17:29.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc88ff9640 10 client.214254.objecter ms_dispatch 0x7fdc8c000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc88ff9640 3 client.214254.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc88ff9640 3 client.214254.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.180+0000 7fdc88ff9640 20 client.214254.objecter dump_active .. 0 homeless 2023-10-30T12:17:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.181+0000 7fdc8b7fe640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc780392f0 0x7fdc7803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.181+0000 7fdc8b7fe640 10 monclient: get_auth_request con 0x7fdc780392f0 auth_method 0 2023-10-30T12:17:29.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.181+0000 7fdc8b7fe640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc780392f0 0x7fdc7803b7b0 secure :-1 s=READY pgs=4246 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c033000 tx=0x7fdc7c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:29.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.181+0000 7fdc88ff9640 10 client.214254.objecter ms_handle_connect 0x7fdc780392f0 2023-10-30T12:17:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 10 monclient: get_auth_request con 0x7fdc8c06d4a0 auth_method 0 2023-10-30T12:17:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:17:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.381+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:29.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:29.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 10 monclient: get_auth_request con 0x7fdc8c06d4a0 auth_method 0 2023-10-30T12:17:29.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:29.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:29.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:29.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:17:29.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:29.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:29.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:29.782+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:30.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient: get_auth_request con 0x7fdc8c06d4a0 auth_method 0 2023-10-30T12:17:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_done global_id 224099 payload 824 2023-10-30T12:17:30.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc8c0a2d90 con 0x7fdc8c06d4a0 2023-10-30T12:17:30.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdc800017a0 tx=0x7fdc80007720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc88ff9640 10 client.214254.objecter ms_handle_connect 0x7fdc8c06d4a0 2023-10-30T12:17:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.583+0000 7fdc88ff9640 10 client.214254.objecter resend_mon_ops 2023-10-30T12:17:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80013de0 con 0x7fdc8c06d4a0 2023-10-30T12:17:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 10 monclient: dump: 2023-10-30T12:17:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.590 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.590 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.590 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.590 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.590 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fdc8001d420 con 0x7fdc8c06d4a0 2023-10-30T12:17:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdc800017a0 tx=0x7fdc80007720 comp rx=0 tx=0).stop 2023-10-30T12:17:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 msgr2=0x7fdc8c02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.584+0000 7fdc88ff9640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c06d4a0 0x7fdc8c02b430 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdc800017a0 tx=0x7fdc80007720 comp rx=0 tx=0).stop 2023-10-30T12:17:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc92840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc92840640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c02b970 0x7fdc8c02bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc92840640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c069fd0 con 0x7fdc8c02b970 2023-10-30T12:17:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c02b970 0x7fdc8c02bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 10 monclient: get_auth_request con 0x7fdc8c02b970 auth_method 0 2023-10-30T12:17:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.591+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc8bfff640 10 monclient(hunting): handle_auth_done global_id 224102 payload 824 2023-10-30T12:17:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc8bfff640 1 -- 172.21.15.50:0/3603889354 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdc8c04e160 con 0x7fdc8c02b970 2023-10-30T12:17:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc8bfff640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c02b970 0x7fdc8c02bd80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc800017a0 tx=0x7fdc80001a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 10 client.214254.objecter ms_handle_connect 0x7fdc8c02b970 2023-10-30T12:17:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 10 client.214254.objecter resend_mon_ops 2023-10-30T12:17:30.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc800149d0 con 0x7fdc8c02b970 2023-10-30T12:17:30.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:30.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 10 monclient: dump: 2023-10-30T12:17:30.598 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.600 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fdc80014b30 con 0x7fdc8c02b970 2023-10-30T12:17:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:30.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:30.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c02b970 msgr2=0x7fdc8c02bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c02b970 0x7fdc8c02bd80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc800017a0 tx=0x7fdc80001a00 comp rx=0 tx=0).stop 2023-10-30T12:17:30.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c02b970 msgr2=0x7fdc8c02bd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:30.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.592+0000 7fdc88ff9640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc8c02b970 0x7fdc8c02bd80 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc800017a0 tx=0x7fdc80001a00 comp rx=0 tx=0).stop 2023-10-30T12:17:30.602 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:30.602 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:30.602 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:30.603 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:17:30.603 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 176, 2023-10-30T12:17:30.603 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:30.603 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:30.603 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:30.604 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:30.604 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.604 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:17:30.604 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:30.604 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:30.605 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:30.605 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:30.605 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:30.605 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:30.605 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:30.606 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:30.606 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:30.606 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:30.606 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:30.606 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:30.607 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:30.607 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.607 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:30.607 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:30.607 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:30.608 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:30.608 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:30.608 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:30.608 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:30.608 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:30.609 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:30.609 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:30.609 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:30.609 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:30.609 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.610 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:30.610 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:30.610 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:30.610 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:30.632 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 10 librados: watch_flush enter 2023-10-30T12:17:30.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 10 librados: watch_flush exit 2023-10-30T12:17:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 20 client.214254.objecter shutdown clearing up homeless session... 2023-10-30T12:17:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 10 client.214254.objecter successfully canceled tick 2023-10-30T12:17:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc780392f0 msgr2=0x7fdc7803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdc780392f0 0x7fdc7803b7b0 secure :-1 s=READY pgs=4246 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c033000 tx=0x7fdc7c009030 comp rx=0 tx=0).stop 2023-10-30T12:17:30.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 10 monclient: shutdown 2023-10-30T12:17:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8c1c3270 msgr2=0x7fdc8c1c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 --2- 172.21.15.50:0/3603889354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8c1c3270 0x7fdc8c1c5e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0725c0 tx=0x7fdc840077c0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3603889354 reap_dead start 2023-10-30T12:17:30.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc90ddd640 1 -- 172.21.15.50:0/3603889354 reap_dead start 2023-10-30T12:17:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 shutdown_connections 2023-10-30T12:17:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 >> 172.21.15.50:0/3603889354 conn(0x7fdc8c0a1170 msgr2=0x7fdc8c0710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 shutdown_connections 2023-10-30T12:17:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 -- 172.21.15.50:0/3603889354 wait complete. 2023-10-30T12:17:30.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.593+0000 7fdc92840640 1 librados: shutdown 2023-10-30T12:17:30.638 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:17:30.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:17:30.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:17:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.753+0000 7f873e8a9640 10 monclient: get_monmap_and_config 2023-10-30T12:17:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.753+0000 7f873e8a9640 10 monclient: build_initial_monmap 2023-10-30T12:17:30.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.753+0000 7f873e8a9640 10 monclient: monmap: 2023-10-30T12:17:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:30.754803+0000 2023-10-30T12:17:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:30.754803+0000 2023-10-30T12:17:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:30.757 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:30.758 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 1 Processor -- start 2023-10-30T12:17:30.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 1 -- start start 2023-10-30T12:17:30.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 10 monclient: init 2023-10-30T12:17:30.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:30.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-10-30T12:17:30.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f8738096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f8738096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:30.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8738096e60 0x7f873809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 10 monclient(hunting): picked mon.noname-c con 0x7f8738096e60 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:30.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f873809afa0 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:30.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.754+0000 7f873e8a9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873809bb20 con 0x7f873809afa0 2023-10-30T12:17:30.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873e8a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873e8a9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873809bd30 con 0x7f8738096e60 2023-10-30T12:17:30.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873e8a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873e8a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873809c400 con 0x7f8738096540 2023-10-30T12:17:30.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873e8a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:30.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873e8a9640 10 monclient(hunting): authenticate will time out at 1895.938032s 2023-10-30T12:17:30.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873ce46640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f873809afa0 0x7f873809b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45544/0 (socket says 172.21.15.50:45544) 2023-10-30T12:17:30.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8737fff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 msgr2=0x7f8738096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:30.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f873ce46640 1 -- 172.21.15.50:0/3089321131 learned_addr learned my addr 172.21.15.50:0/3089321131 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:17:30.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8737fff640 1 -- 172.21.15.50:0/3089321131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 msgr2=0x7f8738096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:30.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8737fff640 1 --2- 172.21.15.50:0/3089321131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f8738096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2023-10-30T12:17:30.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8737fff640 1 --2- 172.21.15.50:0/3089321131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f8738096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:30.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 1 --2- 172.21.15.50:0/3089321131 >> v2:172.21.15.118:3300/0 conn(0x7f8738096e60 0x7f873809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): get_auth_request con 0x7f8738096e60 auth_method 0 2023-10-30T12:17:30.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 1 -- 172.21.15.50:0/3089321131 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878661913 0 0) 0x7f873809bb20 con 0x7f873809afa0 2023-10-30T12:17:30.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:17:30.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 10 monclient(hunting): my global_id is 214257 2023-10-30T12:17:30.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 1 -- 172.21.15.50:0/3089321131 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8724003520 con 0x7f873809afa0 2023-10-30T12:17:30.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient(hunting): handle_auth_done global_id 204340 payload 274 2023-10-30T12:17:30.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 1 -- 172.21.15.50:0/3089321131 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 564575594 0 0) 0x7f8724003520 con 0x7f873809afa0 2023-10-30T12:17:30.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 1 -- 172.21.15.50:0/3089321131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 msgr2=0x7f8738096920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:30.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 1 --2- 172.21.15.50:0/3089321131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f8738096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 1 -- 172.21.15.50:0/3089321131 >> v1:172.21.15.50:6789/0 conn(0x7f873809afa0 legacy=0x7f873809b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:30.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 1 monclient: found mon.noname-c 2023-10-30T12:17:30.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:30.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:17:30.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f87377fe640 1 -- 172.21.15.50:0/3089321131 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873809c120 con 0x7f8738096e60 2023-10-30T12:17:30.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.755+0000 7f8736ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:17:30.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f87377fe640 1 --2- 172.21.15.50:0/3089321131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8738096e60 0x7f873809aa60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f87280097a0 tx=0x7f872802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c77a0237186a2600 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 1 -- 172.21.15.50:0/3089321131 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87300019f0 con 0x7f8738096e60 2023-10-30T12:17:30.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:30.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: dump: 2023-10-30T12:17:30.777 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.777 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.777 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.778 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.778 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.778 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.778 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.778 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.779 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.779 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:30.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:30.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 1 -- 172.21.15.50:0/3089321131 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872802e050 con 0x7f8738096e60 2023-10-30T12:17:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f873e8a9640 5 monclient: authenticate success, global_id 204340 2023-10-30T12:17:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f873e8a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f873e8a9640 4 set_mon_vals no callback set 2023-10-30T12:17:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 1 -- 172.21.15.50:0/3089321131 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728030710 con 0x7f8738096e60 2023-10-30T12:17:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.756+0000 7f8736ffd640 10 monclient: dump: 2023-10-30T12:17:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.783 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 10 monclient: shutdown 2023-10-30T12:17:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 -- 172.21.15.50:0/3089321131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8738096e60 msgr2=0x7f873809aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 --2- 172.21.15.50:0/3089321131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8738096e60 0x7f873809aa60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f87280097a0 tx=0x7f872802b790 comp rx=0 tx=0).stop 2023-10-30T12:17:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 -- 172.21.15.50:0/3089321131 shutdown_connections 2023-10-30T12:17:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 --2- 172.21.15.50:0/3089321131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8738096e60 0x7f873809aa60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 --2- 172.21.15.50:0/3089321131 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f8738096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 -- 172.21.15.50:0/3089321131 >> 172.21.15.50:0/3089321131 conn(0x7f87380a3980 msgr2=0x7f87380a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 -- 172.21.15.50:0/3089321131 shutdown_connections 2023-10-30T12:17:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 1 -- 172.21.15.50:0/3089321131 wait complete. 2023-10-30T12:17:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 10 monclient: build_initial_monmap 2023-10-30T12:17:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.757+0000 7f873e8a9640 10 monclient: monmap: 2023-10-30T12:17:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:30.787 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 librados: starting msgr at 2023-10-30T12:17:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 librados: starting objecter 2023-10-30T12:17:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 Processor -- start 2023-10-30T12:17:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 -- start start 2023-10-30T12:17:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 librados: setting wanted keys 2023-10-30T12:17:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 librados: calling monclient init 2023-10-30T12:17:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient: init 2023-10-30T12:17:30.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:30.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:30.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f87381c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient(hunting): picked mon.noname-c con 0x7f8738096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:30.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8738096e60 0x7f87380942c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f8738096e60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873809afa0 0x7f8738094800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f873809afa0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873808f4c0 con 0x7f8738096e60 2023-10-30T12:17:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873808f7a0 con 0x7f873809afa0 2023-10-30T12:17:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873808fa80 con 0x7f8738096540 2023-10-30T12:17:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.758+0000 7f873e8a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 10 monclient(hunting): authenticate will time out at 1895.942010s 2023-10-30T12:17:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f87381c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8738096e60 0x7f87380942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f87381c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52174/0 (socket says 172.21.15.50:52174) 2023-10-30T12:17:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 -- 172.21.15.50:0/2714295204 learned_addr learned my addr 172.21.15.50:0/2714295204 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): get_auth_request con 0x7f8738096540 auth_method 0 2023-10-30T12:17:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 10 monclient(hunting): get_auth_request con 0x7f8738096e60 auth_method 0 2023-10-30T12:17:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873ce46640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873809afa0 0x7f8738094800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873ce46640 10 monclient(hunting): get_auth_request con 0x7f873809afa0 auth_method 0 2023-10-30T12:17:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873ce46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873ce46640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873ce46640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient(hunting): handle_auth_done global_id 224105 payload 824 2023-10-30T12:17:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873809afa0 msgr2=0x7f8738094800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873809afa0 0x7f8738094800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8738096e60 msgr2=0x7f87380942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8738096e60 0x7f87380942c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 monclient: found mon.noname-c 2023-10-30T12:17:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 -- 172.21.15.50:0/2714295204 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728009400 con 0x7f8738096540 2023-10-30T12:17:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87377fe640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8738096e60 0x7f87380942c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8737fff640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f87381c7eb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f872000c480 tx=0x7f872000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 client.?.objecter ms_handle_connect 0x7f8738096540 2023-10-30T12:17:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720005de0 con 0x7f8738096540 2023-10-30T12:17:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: dump: 2023-10-30T12:17:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.801 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: _finish_auth 0 2023-10-30T12:17:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:17:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8720007070 con 0x7f8738096540 2023-10-30T12:17:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 5 monclient: authenticate success, global_id 224105 2023-10-30T12:17:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 10 monclient: _renew_subs 2023-10-30T12:17:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87381c83f0 con 0x7f8738096540 2023-10-30T12:17:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720007420 con 0x7f8738096540 2023-10-30T12:17:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f8734ff9640 10 monclient: dump: 2023-10-30T12:17:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.805 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f87367fc640 4 set_mon_vals no callback set 2023-10-30T12:17:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 10 client.224105.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 10 monclient: _renew_subs 2023-10-30T12:17:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873808da80 con 0x7f8738096540 2023-10-30T12:17:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.759+0000 7f873e8a9640 1 librados: init done 2023-10-30T12:17:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f873e8a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8720007580 con 0x7f8738096540 2023-10-30T12:17:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f873e8a9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c020 0x7f873808c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f873e8a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8738073ff0 con 0x7f873808c020 2023-10-30T12:17:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f87377fe640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c020 0x7f873808c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f87377fe640 10 monclient: get_auth_request con 0x7f873808c020 auth_method 0 2023-10-30T12:17:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f87377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f87377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f87377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.760+0000 7f8734ff9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f87180392f0 0x7f871803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f872004a370 con 0x7f8738096540 2023-10-30T12:17:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 10 client.224105.objecter ms_dispatch 0x7f8738000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 3 client.224105.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 3 client.224105.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f873ce46640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f87180392f0 0x7f871803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 20 client.224105.objecter dump_active .. 0 homeless 2023-10-30T12:17:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f873ce46640 10 monclient: get_auth_request con 0x7f87180392f0 auth_method 0 2023-10-30T12:17:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f87377fe640 10 monclient(hunting): handle_auth_done global_id 204346 payload 824 2023-10-30T12:17:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f87377fe640 1 -- 172.21.15.50:0/2714295204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f873808dd30 con 0x7f873808c020 2023-10-30T12:17:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 10 client.224105.objecter ms_handle_connect 0x7f873808c020 2023-10-30T12:17:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 10 client.224105.objecter resend_mon_ops 2023-10-30T12:17:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f87377fe640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c020 0x7f873808c400 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f873809c8f0 tx=0x7f87280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f873ce46640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f87180392f0 0x7f871803b7b0 secure :-1 s=READY pgs=4247 cs=0 l=1 rev1=1 crypto rx=0x7f873000e1b0 tx=0x7f8730009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 10 client.224105.objecter ms_handle_connect 0x7f87180392f0 2023-10-30T12:17:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728032a50 con 0x7f873808c020 2023-10-30T12:17:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 10 monclient: dump: 2023-10-30T12:17:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.814 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.761+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8728032bb0 con 0x7f873808c020 2023-10-30T12:17:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.762+0000 7f8734ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.762+0000 7f8734ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:30.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.762+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c020 msgr2=0x7f873808c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.762+0000 7f8734ff9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c020 0x7f873808c400 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f873809c8f0 tx=0x7f87280046c0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.762+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c020 msgr2=0x7f873808c400 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:30.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.762+0000 7f8734ff9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c020 0x7f873808c400 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f873e8a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:30.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f873e8a9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c940 0x7f873806e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f873e8a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873808f630 con 0x7f873808c940 2023-10-30T12:17:30.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c940 0x7f873806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient: get_auth_request con 0x7f873808c940 auth_method 0 2023-10-30T12:17:30.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 10 monclient(hunting): handle_auth_done global_id 204349 payload 824 2023-10-30T12:17:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.768+0000 7f87377fe640 1 -- 172.21.15.50:0/2714295204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f873809d3a0 con 0x7f873808c940 2023-10-30T12:17:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f87377fe640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c940 0x7f873806e430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8728000c00 tx=0x7f8728008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 10 client.224105.objecter ms_handle_connect 0x7f873808c940 2023-10-30T12:17:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 10 client.224105.objecter resend_mon_ops 2023-10-30T12:17:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872804f670 con 0x7f873808c940 2023-10-30T12:17:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:30.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 10 monclient: dump: 2023-10-30T12:17:30.821 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.821 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.821 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.822 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.822 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.822 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.822 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.822 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.822 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.823 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f872804f7d0 con 0x7f873808c940 2023-10-30T12:17:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c940 msgr2=0x7f873806e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c940 0x7f873806e430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8728000c00 tx=0x7f8728008030 comp rx=0 tx=0).stop 2023-10-30T12:17:30.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c940 msgr2=0x7f873806e430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:30.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f8734ff9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873808c940 0x7f873806e430 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8728000c00 tx=0x7f8728008030 comp rx=0 tx=0).stop 2023-10-30T12:17:30.824 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:17:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 176, 2023-10-30T12:17:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:17:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:30.828 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:30.828 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:30.828 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:30.828 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:30.828 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:30.829 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:30.829 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:30.829 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.829 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:30.829 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:30.830 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:30.830 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:30.830 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:30.830 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:30.830 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:30.831 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:30.831 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:30.831 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:30.831 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:30.831 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:30.832 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.832 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:30.832 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:30.832 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:30.832 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:30.833 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:30.833 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:30.833 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:30.833 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:30.833 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:30.834 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:30.834 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:30.834 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:30.834 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:30.834 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:30.834 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:30.835 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:30.835 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:30.835 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:30.835 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:30.835 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:30.836 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:30.836 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:30.836 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:30.836 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:30.836 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:30.836 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:30.837 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:30.837 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.837 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:30.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.837 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:30.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.838 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:30.838 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:30.838 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:30.838 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:17:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:30.854 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:30.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 10 librados: watch_flush enter 2023-10-30T12:17:30.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 10 librados: watch_flush exit 2023-10-30T12:17:30.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 20 client.224105.objecter shutdown clearing up homeless session... 2023-10-30T12:17:30.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 10 client.224105.objecter successfully canceled tick 2023-10-30T12:17:30.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f87180392f0 msgr2=0x7f871803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f87180392f0 0x7f871803b7b0 secure :-1 s=READY pgs=4247 cs=0 l=1 rev1=1 crypto rx=0x7f873000e1b0 tx=0x7f8730009040 comp rx=0 tx=0).stop 2023-10-30T12:17:30.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 10 monclient: shutdown 2023-10-30T12:17:30.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:30.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 msgr2=0x7f87381c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f87381c7eb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f872000c480 tx=0x7f872000c940 comp rx=0 tx=0).stop 2023-10-30T12:17:30.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873ce46640 1 -- 172.21.15.50:0/2714295204 reap_dead start 2023-10-30T12:17:30.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 shutdown_connections 2023-10-30T12:17:30.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 1 --2- 172.21.15.50:0/2714295204 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8738096540 0x7f87381c7eb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.769+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 >> 172.21.15.50:0/2714295204 conn(0x7f87380a3980 msgr2=0x7f87381bb680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:30.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.770+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 shutdown_connections 2023-10-30T12:17:30.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.770+0000 7f873e8a9640 1 -- 172.21.15.50:0/2714295204 wait complete. 2023-10-30T12:17:30.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.770+0000 7f873e8a9640 1 librados: shutdown 2023-10-30T12:17:30.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:17:30.859 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:17:30.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:17:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.965+0000 7f45dbcd7640 10 monclient: get_monmap_and_config 2023-10-30T12:17:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.965+0000 7f45dbcd7640 10 monclient: build_initial_monmap 2023-10-30T12:17:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.965+0000 7f45dbcd7640 10 monclient: monmap: 2023-10-30T12:17:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:30.966882+0000 2023-10-30T12:17:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:30.966882+0000 2023-10-30T12:17:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:30.970 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.966+0000 7f45dbcd7640 1 Processor -- start 2023-10-30T12:17:30.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.966+0000 7f45dbcd7640 1 -- start start 2023-10-30T12:17:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.966+0000 7f45dbcd7640 10 monclient: init 2023-10-30T12:17:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.966+0000 7f45dbcd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-10-30T12:17:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45d4098520 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f45d4098900 0x7f45d4098d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient(hunting): picked mon.noname-a con 0x7f45d4098900 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4137980 0x7f45d4139d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient(hunting): picked mon.noname-b con 0x7f45d4137980 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d409c680 con 0x7f45d4098520 2023-10-30T12:17:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d4099930 con 0x7f45d4098900 2023-10-30T12:17:30.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:30.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d409aca0 con 0x7f45d4137980 2023-10-30T12:17:30.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:30.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45dbcd7640 10 monclient(hunting): authenticate will time out at 1896.150110s 2023-10-30T12:17:30.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f45d4098900 0x7f45d4098d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4137980 0x7f45d4139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:30.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f45d4098900 0x7f45d4098d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51534/0 (socket says 172.21.15.50:51534) 2023-10-30T12:17:30.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 -- 172.21.15.50:0/1677397352 learned_addr learned my addr 172.21.15.50:0/1677397352 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:30.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 10 monclient(hunting): get_auth_request con 0x7f45d4137980 auth_method 0 2023-10-30T12:17:30.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): get_auth_request con 0x7f45d4098900 auth_method 0 2023-10-30T12:17:30.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:30.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:30.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:30.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:30.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:30.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45da274640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:30.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient(hunting): handle_auth_done global_id 214263 payload 274 2023-10-30T12:17:30.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 -- 172.21.15.50:0/1677397352 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4137980 msgr2=0x7f45d4139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:30.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 --2- 172.21.15.50:0/1677397352 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4137980 0x7f45d4139d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 -- 172.21.15.50:0/1677397352 >> v1:172.21.15.118:6789/0 conn(0x7f45d4098520 legacy=0x7f45d409bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient: _finish_hunting 0 2023-10-30T12:17:30.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 monclient: found mon.noname-a 2023-10-30T12:17:30.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:30.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:30.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.967+0000 7f45d9272640 1 -- 172.21.15.50:0/1677397352 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d409af80 con 0x7f45d4098900 2023-10-30T12:17:30.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d9272640 1 --2- 172.21.15.50:0/1677397352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4098900 0x7f45d4098d70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009870 tx=0x7f45d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42d07f5201971d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:30.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 1 -- 172.21.15.50:0/1677397352 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d00317c0 con 0x7f45d4098900 2023-10-30T12:17:30.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: dump: 2023-10-30T12:17:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.989 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.989 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: _finish_auth 0 2023-10-30T12:17:30.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: _check_auth_tickets 2023-10-30T12:17:30.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:30.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 1 -- 172.21.15.50:0/1677397352 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0031920 con 0x7f45d4098900 2023-10-30T12:17:30.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 5 monclient: authenticate success, global_id 214263 2023-10-30T12:17:30.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:30.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:30.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:30.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 1 -- 172.21.15.50:0/1677397352 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0031cb0 con 0x7f45d4098900 2023-10-30T12:17:30.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:30.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 4 set_mon_vals no callback set 2023-10-30T12:17:30.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:30.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45d8a71640 10 monclient: dump: 2023-10-30T12:17:30.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:30.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:30.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:30.992 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:30.992 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.993 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:30.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:30.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:30.993 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 10 monclient: shutdown 2023-10-30T12:17:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 1 -- 172.21.15.50:0/1677397352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4098900 msgr2=0x7f45d4098d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/1677397352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4098900 0x7f45d4098d70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009870 tx=0x7f45d002b860 comp rx=0 tx=0).stop 2023-10-30T12:17:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 1 -- 172.21.15.50:0/1677397352 shutdown_connections 2023-10-30T12:17:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/1677397352 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4137980 0x7f45d4139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/1677397352 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4098900 0x7f45d4098d70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:30.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.968+0000 7f45dbcd7640 1 -- 172.21.15.50:0/1677397352 >> 172.21.15.50:0/1677397352 conn(0x7f45d40a05c0 msgr2=0x7f45d409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:30.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 -- 172.21.15.50:0/1677397352 shutdown_connections 2023-10-30T12:17:30.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 -- 172.21.15.50:0/1677397352 wait complete. 2023-10-30T12:17:30.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 10 monclient: build_initial_monmap 2023-10-30T12:17:30.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 10 monclient: monmap: 2023-10-30T12:17:30.996 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:30.996 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:30.996 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:30.996 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:30.996 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:30.997 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:30.997 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:30.997 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:30.997 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:30.997 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:30.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 librados: starting msgr at 2023-10-30T12:17:30.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 librados: starting objecter 2023-10-30T12:17:30.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 Processor -- start 2023-10-30T12:17:30.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 -- start start 2023-10-30T12:17:30.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 librados: setting wanted keys 2023-10-30T12:17:30.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 1 librados: calling monclient init 2023-10-30T12:17:30.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.969+0000 7f45dbcd7640 10 monclient: init 2023-10-30T12:17:30.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:30.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:17:30.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4098520 0x7f45d408c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:30.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): picked mon.noname-c con 0x7f45d4098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:31.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45d4098900 0x7f45d408ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:31.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): picked mon.noname-b con 0x7f45d4098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:31.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4137980 0x7f45d41c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:31.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): picked mon.noname-a con 0x7f45d4137980 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:31.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d408e3d0 con 0x7f45d4137980 2023-10-30T12:17:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d4096a30 con 0x7f45d4098900 2023-10-30T12:17:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d408e640 con 0x7f45d4098520 2023-10-30T12:17:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:31.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45dbcd7640 10 monclient(hunting): authenticate will time out at 1896.153382s 2023-10-30T12:17:31.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4098520 0x7f45d408c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:31.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4137980 0x7f45d41c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:31.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4098520 0x7f45d408c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52192/0 (socket says 172.21.15.50:52192) 2023-10-30T12:17:31.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 -- 172.21.15.50:0/2257010355 learned_addr learned my addr 172.21.15.50:0/2257010355 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:31.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): get_auth_request con 0x7f45d4098520 auth_method 0 2023-10-30T12:17:31.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:31.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:31.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:31.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 10 monclient(hunting): get_auth_request con 0x7f45d4137980 auth_method 0 2023-10-30T12:17:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:31.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:31.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:31.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:31.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:31.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9272640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45d4098900 0x7f45d408ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient(hunting): handle_auth_done global_id 224111 payload 824 2023-10-30T12:17:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 -- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45d4098900 msgr2=0x7f45d408ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45d4098900 0x7f45d408ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 -- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4137980 msgr2=0x7f45d41c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4137980 0x7f45d41c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient: _finish_hunting 0 2023-10-30T12:17:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 monclient: found mon.noname-c 2023-10-30T12:17:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 -- 172.21.15.50:0/2257010355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00094d0 con 0x7f45d4098520 2023-10-30T12:17:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9272640 10 monclient: get_auth_request con 0x7f45d4098900 auth_method 0 2023-10-30T12:17:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9272640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45d4098900 0x7f45d408ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:17:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45da274640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4137980 0x7f45d41c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45d9a73640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4098520 0x7f45d408c4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c480 tx=0x7f45c400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:31.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.970+0000 7f45ca7fc640 10 client.?.objecter ms_handle_connect 0x7f45d4098520 2023-10-30T12:17:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 1 -- 172.21.15.50:0/2257010355 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c40052d0 con 0x7f45d4098520 2023-10-30T12:17:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: dump: 2023-10-30T12:17:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:31.010 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:31.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:31.010 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:31.010 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:31.011 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:31.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:31.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:31.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:31.011 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:31.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:31.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:31.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:31.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 5 monclient: authenticate success, global_id 224111 2023-10-30T12:17:31.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 1 -- 172.21.15.50:0/2257010355 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c4005450 con 0x7f45d4098520 2023-10-30T12:17:31.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:31.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 1 -- 172.21.15.50:0/2257010355 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c4003880 con 0x7f45d4098520 2023-10-30T12:17:31.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 10 monclient: _renew_subs 2023-10-30T12:17:31.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:31.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d408e920 con 0x7f45d4098520 2023-10-30T12:17:31.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45cbfff640 4 set_mon_vals no callback set 2023-10-30T12:17:31.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:31.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 10 client.224111.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:31.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:31.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45ca7fc640 10 monclient: dump: 2023-10-30T12:17:31.014 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:31.016 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 10 monclient: _renew_subs 2023-10-30T12:17:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:31.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d408cf60 con 0x7f45d4098520 2023-10-30T12:17:31.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.971+0000 7f45dbcd7640 1 librados: init done 2023-10-30T12:17:31.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45ca7fc640 1 -- 172.21.15.50:0/2257010355 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f45c40039e0 con 0x7f45d4098520 2023-10-30T12:17:31.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45dbcd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:31.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45dbcd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:31.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45dbcd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:31.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d408eb60 con 0x7f45d4098520 2023-10-30T12:17:31.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45ca7fc640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:31.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45ca7fc640 1 -- 172.21.15.50:0/2257010355 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f45c404a8f0 con 0x7f45d4098520 2023-10-30T12:17:31.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45ca7fc640 10 client.224111.objecter ms_dispatch 0x7f45d4000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:31.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45ca7fc640 3 client.224111.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:31.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45ca7fc640 3 client.224111.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:31.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45ca7fc640 20 client.224111.objecter dump_active .. 0 homeless 2023-10-30T12:17:31.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45d9272640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:31.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45d9272640 10 monclient: get_auth_request con 0x7f45b8039320 auth_method 0 2023-10-30T12:17:31.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45d9272640 1 -- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 msgr2=0x7f45b803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:17:31.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45d9272640 1 -- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 msgr2=0x7f45b803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:31.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45d9272640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f45d0036000 tx=0x7f45d003d040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:31.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.972+0000 7f45d9272640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:31.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.977+0000 7f45ca7fc640 1 -- 172.21.15.50:0/2257010355 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f45d408eb60 con 0x7f45d4098520 2023-10-30T12:17:31.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.977+0000 7f45ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:31.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:30.977+0000 7f45ca7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:31.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.173+0000 7f45d9272640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:31.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.173+0000 7f45d9272640 10 monclient: get_auth_request con 0x7f45b8039320 auth_method 0 2023-10-30T12:17:31.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.173+0000 7f45d9272640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 secure :-1 s=READY pgs=4248 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:31.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.173+0000 7f45ca7fc640 10 client.224111.objecter ms_handle_connect 0x7f45b8039320 2023-10-30T12:17:31.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.223+0000 7f45dbcd7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:17:31.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.223+0000 7f45dbcd7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:17:31.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.223+0000 7f45dbcd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:31.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.223+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f45d408eb60 con 0x7f45d4098520 2023-10-30T12:17:31.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f45ca7fc640 1 -- 172.21.15.50:0/2257010355 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f45d408eb60 con 0x7f45d4098520 2023-10-30T12:17:31.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f45ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:17:31.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f45ca7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:31.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=10,kv=9,logm=54} crc {auth=2859703620,config=3140514009,health=1084738009,kv=2905431493,logm=1904985195}) 2023-10-30T12:17:31.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3224033659,config=3140514009,health=1681288197,kv=187927950,logm=1631744183}) 2023-10-30T12:17:31.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=10,kv=9,logm=54} crc {auth=2859703620,config=3140514009,health=1084738009,kv=2905431493,logm=1904985195}) 2023-10-30T12:17:31.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.224+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=10,kv=8,logm=54} crc {auth=106861867,config=3140514009,health=3369061496,kv=784550477,logm=2875197937}) 2023-10-30T12:17:31.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850252898}) 2023-10-30T12:17:31.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2689986806}) 2023-10-30T12:17:31.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850252898}) 2023-10-30T12:17:31.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733027803}) 2023-10-30T12:17:31.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 10 librados: watch_flush enter 2023-10-30T12:17:31.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 10 librados: watch_flush exit 2023-10-30T12:17:31.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 20 client.224111.objecter shutdown clearing up homeless session... 2023-10-30T12:17:31.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 10 client.224111.objecter successfully canceled tick 2023-10-30T12:17:31.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 msgr2=0x7f45b803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:31.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 secure :-1 s=READY pgs=4248 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d003d040 comp rx=0 tx=0).stop 2023-10-30T12:17:31.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 10 monclient: shutdown 2023-10-30T12:17:31.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:31.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4098520 msgr2=0x7f45d408c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:31.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4098520 0x7f45d408c4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c480 tx=0x7f45c400c940 comp rx=0 tx=0).stop 2023-10-30T12:17:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 shutdown_connections 2023-10-30T12:17:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f45b8039320 0x7f45b803b7e0 unknown :-1 s=CLOSED pgs=4248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45d4137980 0x7f45d41c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45d4098900 0x7f45d408ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 --2- 172.21.15.50:0/2257010355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f45d4098520 0x7f45d408c4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 >> 172.21.15.50:0/2257010355 conn(0x7f45d40a05c0 msgr2=0x7f45d4135980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 shutdown_connections 2023-10-30T12:17:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 -- 172.21.15.50:0/2257010355 wait complete. 2023-10-30T12:17:31.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:31.226+0000 7f45dbcd7640 1 librados: shutdown 2023-10-30T12:17:31.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.227+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.227+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945656738}) 2023-10-30T12:17:31.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.227+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272702406}) 2023-10-30T12:17:31.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.227+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.227+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945656738}) 2023-10-30T12:17:31.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.227+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380937619}) 2023-10-30T12:17:31.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.228+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.228+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217619965}) 2023-10-30T12:17:31.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.228+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=871560923}) 2023-10-30T12:17:31.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.228+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.228+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217619965}) 2023-10-30T12:17:31.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.228+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2500939556}) 2023-10-30T12:17:31.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.229+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.229+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183725681}) 2023-10-30T12:17:31.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.229+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697165185}) 2023-10-30T12:17:31.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.229+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.229+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183725681}) 2023-10-30T12:17:31.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.229+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3832263750}) 2023-10-30T12:17:31.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.230+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.230+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266448557}) 2023-10-30T12:17:31.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.230+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1554703342}) 2023-10-30T12:17:31.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.230+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.230+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266448557}) 2023-10-30T12:17:31.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.230+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714668614}) 2023-10-30T12:17:31.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.231+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.231+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008833303}) 2023-10-30T12:17:31.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.231+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3201580360}) 2023-10-30T12:17:31.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.231+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.231+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008833303}) 2023-10-30T12:17:31.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.231+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3656183745}) 2023-10-30T12:17:31.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870588095}) 2023-10-30T12:17:31.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519294494}) 2023-10-30T12:17:31.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870588095}) 2023-10-30T12:17:31.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3338587575}) 2023-10-30T12:17:31.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761676040}) 2023-10-30T12:17:31.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266852282}) 2023-10-30T12:17:31.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761676040}) 2023-10-30T12:17:31.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.232+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288875081}) 2023-10-30T12:17:31.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.233+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.233+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255151740}) 2023-10-30T12:17:31.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.233+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3507672868}) 2023-10-30T12:17:31.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.233+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.233+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255151740}) 2023-10-30T12:17:31.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.233+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508724051}) 2023-10-30T12:17:31.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3581028832}) 2023-10-30T12:17:31.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821659241}) 2023-10-30T12:17:31.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3581028832}) 2023-10-30T12:17:31.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887028676}) 2023-10-30T12:17:31.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024897703}) 2023-10-30T12:17:31.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3184109617}) 2023-10-30T12:17:31.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024897703}) 2023-10-30T12:17:31.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.234+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3660594864}) 2023-10-30T12:17:31.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.235+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.254 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.235+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316615513}) 2023-10-30T12:17:31.254 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.235+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3794030618}) 2023-10-30T12:17:31.254 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.235+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.254 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.235+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1316615513}) 2023-10-30T12:17:31.254 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.235+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750493692}) 2023-10-30T12:17:31.255 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.236+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.255 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.236+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360160411}) 2023-10-30T12:17:31.255 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.236+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1968942389}) 2023-10-30T12:17:31.255 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.236+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.236+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360160411}) 2023-10-30T12:17:31.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.236+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899863871}) 2023-10-30T12:17:31.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790611341}) 2023-10-30T12:17:31.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244241696}) 2023-10-30T12:17:31.256 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790611341}) 2023-10-30T12:17:31.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909251004}) 2023-10-30T12:17:31.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.257 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496077459}) 2023-10-30T12:17:31.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187885055}) 2023-10-30T12:17:31.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.237+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496077459}) 2023-10-30T12:17:31.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3868057592}) 2023-10-30T12:17:31.258 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821754496}) 2023-10-30T12:17:31.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2379332975}) 2023-10-30T12:17:31.259 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821754496}) 2023-10-30T12:17:31.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.238+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1650137987}) 2023-10-30T12:17:31.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.239+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.239+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533169933}) 2023-10-30T12:17:31.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.239+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2122988637}) 2023-10-30T12:17:31.260 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.239+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.261 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.239+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533169933}) 2023-10-30T12:17:31.262 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:17:31.299 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.239+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787953703}) 2023-10-30T12:17:31.299 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.300 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815564348}) 2023-10-30T12:17:31.300 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3890455836}) 2023-10-30T12:17:31.300 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.300 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815564348}) 2023-10-30T12:17:31.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040745192}) 2023-10-30T12:17:31.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801770781}) 2023-10-30T12:17:31.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3574677613}) 2023-10-30T12:17:31.301 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801770781}) 2023-10-30T12:17:31.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.240+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768729541}) 2023-10-30T12:17:31.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.241+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.241+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924322360}) 2023-10-30T12:17:31.302 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.241+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1545061574}) 2023-10-30T12:17:31.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.241+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.241+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924322360}) 2023-10-30T12:17:31.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.241+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517769325}) 2023-10-30T12:17:31.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.242+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.303 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.242+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613907353}) 2023-10-30T12:17:31.304 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.242+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3223943492}) 2023-10-30T12:17:31.304 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.242+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.304 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.242+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613907353}) 2023-10-30T12:17:31.304 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.242+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121879904}) 2023-10-30T12:17:31.304 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.243+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.304 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.243+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270806268}) 2023-10-30T12:17:31.305 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.243+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1193391220}) 2023-10-30T12:17:31.305 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.243+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.305 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.243+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270806268}) 2023-10-30T12:17:31.305 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.243+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941314800}) 2023-10-30T12:17:31.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388880853}) 2023-10-30T12:17:31.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948796895}) 2023-10-30T12:17:31.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.306 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388880853}) 2023-10-30T12:17:31.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1181881789}) 2023-10-30T12:17:31.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266464648}) 2023-10-30T12:17:31.307 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333238515}) 2023-10-30T12:17:31.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266464648}) 2023-10-30T12:17:31.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.244+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943602348}) 2023-10-30T12:17:31.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.245+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.245+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3911689302}) 2023-10-30T12:17:31.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.245+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293613753}) 2023-10-30T12:17:31.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.245+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.245+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3911689302}) 2023-10-30T12:17:31.309 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.245+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3056018083}) 2023-10-30T12:17:31.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243055293}) 2023-10-30T12:17:31.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3007625031}) 2023-10-30T12:17:31.310 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243055293}) 2023-10-30T12:17:31.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735903237}) 2023-10-30T12:17:31.314 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767979142}) 2023-10-30T12:17:31.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2965899811}) 2023-10-30T12:17:31.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767979142}) 2023-10-30T12:17:31.315 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.246+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1818070432}) 2023-10-30T12:17:31.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.247+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.247+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924774371}) 2023-10-30T12:17:31.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.247+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499615783}) 2023-10-30T12:17:31.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.247+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.247+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924774371}) 2023-10-30T12:17:31.316 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.247+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417811343}) 2023-10-30T12:17:31.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615524323}) 2023-10-30T12:17:31.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154331489}) 2023-10-30T12:17:31.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.317 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615524323}) 2023-10-30T12:17:31.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2509707409}) 2023-10-30T12:17:31.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958982953}) 2023-10-30T12:17:31.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4073723618}) 2023-10-30T12:17:31.318 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958982953}) 2023-10-30T12:17:31.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.248+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2674081463}) 2023-10-30T12:17:31.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.249+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.319 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.249+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258983518}) 2023-10-30T12:17:31.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.249+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436815041}) 2023-10-30T12:17:31.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.249+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.249+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258983518}) 2023-10-30T12:17:31.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.249+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909407853}) 2023-10-30T12:17:31.320 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090713832}) 2023-10-30T12:17:31.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889666111}) 2023-10-30T12:17:31.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090713832}) 2023-10-30T12:17:31.321 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=230366001}) 2023-10-30T12:17:31.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613300995}) 2023-10-30T12:17:31.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388325175}) 2023-10-30T12:17:31.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.322 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613300995}) 2023-10-30T12:17:31.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.250+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410596808}) 2023-10-30T12:17:31.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.251+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.251+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778241734}) 2023-10-30T12:17:31.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.251+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3914620776}) 2023-10-30T12:17:31.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.251+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.323 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.251+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778241734}) 2023-10-30T12:17:31.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.251+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714202774}) 2023-10-30T12:17:31.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153488979}) 2023-10-30T12:17:31.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376658231}) 2023-10-30T12:17:31.324 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153488979}) 2023-10-30T12:17:31.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1213394779}) 2023-10-30T12:17:31.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549863551}) 2023-10-30T12:17:31.325 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495264164}) 2023-10-30T12:17:31.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549863551}) 2023-10-30T12:17:31.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.252+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1826561755}) 2023-10-30T12:17:31.326 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.253+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.253+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=635238761}) 2023-10-30T12:17:31.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.253+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670416163}) 2023-10-30T12:17:31.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.253+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.253+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=635238761}) 2023-10-30T12:17:31.327 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.253+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1837290191}) 2023-10-30T12:17:31.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479743624}) 2023-10-30T12:17:31.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966005455}) 2023-10-30T12:17:31.328 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479743624}) 2023-10-30T12:17:31.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=764038987}) 2023-10-30T12:17:31.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.254+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542930064}) 2023-10-30T12:17:31.329 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752819674}) 2023-10-30T12:17:31.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542930064}) 2023-10-30T12:17:31.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119484666}) 2023-10-30T12:17:31.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.330 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324496614}) 2023-10-30T12:17:31.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2677725566}) 2023-10-30T12:17:31.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324496614}) 2023-10-30T12:17:31.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.255+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047507540}) 2023-10-30T12:17:31.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.256+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.256+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478412743}) 2023-10-30T12:17:31.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.256+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494980940}) 2023-10-30T12:17:31.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.256+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.256+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478412743}) 2023-10-30T12:17:31.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.256+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1329087953}) 2023-10-30T12:17:31.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755039544}) 2023-10-30T12:17:31.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985031962}) 2023-10-30T12:17:31.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755039544}) 2023-10-30T12:17:31.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042128423}) 2023-10-30T12:17:31.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478177369}) 2023-10-30T12:17:31.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2334472863}) 2023-10-30T12:17:31.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478177369}) 2023-10-30T12:17:31.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.257+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3705886338}) 2023-10-30T12:17:31.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.258+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.258+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588916119}) 2023-10-30T12:17:31.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.258+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=776283699}) 2023-10-30T12:17:31.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.258+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.258+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588916119}) 2023-10-30T12:17:31.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.258+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494214705}) 2023-10-30T12:17:31.336 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.259+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.259+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457761613}) 2023-10-30T12:17:31.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.259+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055452519}) 2023-10-30T12:17:31.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.259+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.259+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457761613}) 2023-10-30T12:17:31.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.259+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1388589483}) 2023-10-30T12:17:31.337 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372789373}) 2023-10-30T12:17:31.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3932108625}) 2023-10-30T12:17:31.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372789373}) 2023-10-30T12:17:31.338 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149911768}) 2023-10-30T12:17:31.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075709426}) 2023-10-30T12:17:31.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.260+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1459492784}) 2023-10-30T12:17:31.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075709426}) 2023-10-30T12:17:31.339 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143164725}) 2023-10-30T12:17:31.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784347110}) 2023-10-30T12:17:31.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4158185830}) 2023-10-30T12:17:31.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.340 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784347110}) 2023-10-30T12:17:31.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.261+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3190799118}) 2023-10-30T12:17:31.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.262+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.262+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809425617}) 2023-10-30T12:17:31.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.262+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1006297116}) 2023-10-30T12:17:31.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.262+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.262+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809425617}) 2023-10-30T12:17:31.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.262+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752326356}) 2023-10-30T12:17:31.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632502590}) 2023-10-30T12:17:31.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578384613}) 2023-10-30T12:17:31.342 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632502590}) 2023-10-30T12:17:31.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3347699844}) 2023-10-30T12:17:31.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850591978}) 2023-10-30T12:17:31.343 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443180785}) 2023-10-30T12:17:31.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850591978}) 2023-10-30T12:17:31.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.263+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3442359059}) 2023-10-30T12:17:31.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.264+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.264+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235577435}) 2023-10-30T12:17:31.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.264+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3088029913}) 2023-10-30T12:17:31.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.264+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.264+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235577435}) 2023-10-30T12:17:31.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.264+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766689207}) 2023-10-30T12:17:31.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464178323}) 2023-10-30T12:17:31.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128793911}) 2023-10-30T12:17:31.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464178323}) 2023-10-30T12:17:31.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896884457}) 2023-10-30T12:17:31.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103384331}) 2023-10-30T12:17:31.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777103534}) 2023-10-30T12:17:31.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103384331}) 2023-10-30T12:17:31.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.265+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594463675}) 2023-10-30T12:17:31.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.266+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.266+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778376838}) 2023-10-30T12:17:31.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.266+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250253319}) 2023-10-30T12:17:31.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.266+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.266+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778376838}) 2023-10-30T12:17:31.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.266+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3488748085}) 2023-10-30T12:17:31.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143403813}) 2023-10-30T12:17:31.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3733509167}) 2023-10-30T12:17:31.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143403813}) 2023-10-30T12:17:31.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1106759685}) 2023-10-30T12:17:31.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283748995}) 2023-10-30T12:17:31.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=168487177}) 2023-10-30T12:17:31.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283748995}) 2023-10-30T12:17:31.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.267+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585232498}) 2023-10-30T12:17:31.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.268+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.268+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712068276}) 2023-10-30T12:17:31.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.268+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013162532}) 2023-10-30T12:17:31.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.268+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.268+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712068276}) 2023-10-30T12:17:31.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.268+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3199280793}) 2023-10-30T12:17:31.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263646692}) 2023-10-30T12:17:31.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3429421120}) 2023-10-30T12:17:31.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263646692}) 2023-10-30T12:17:31.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700593727}) 2023-10-30T12:17:31.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127640956}) 2023-10-30T12:17:31.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3477793249}) 2023-10-30T12:17:31.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127640956}) 2023-10-30T12:17:31.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.269+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1075228372}) 2023-10-30T12:17:31.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.270+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.270+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481400276}) 2023-10-30T12:17:31.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.270+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4248215943}) 2023-10-30T12:17:31.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.270+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.270+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481400276}) 2023-10-30T12:17:31.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.270+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1087442050}) 2023-10-30T12:17:31.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535904404}) 2023-10-30T12:17:31.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3790124172}) 2023-10-30T12:17:31.357 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535904404}) 2023-10-30T12:17:31.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2802944506}) 2023-10-30T12:17:31.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=575971623}) 2023-10-30T12:17:31.358 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2095684350}) 2023-10-30T12:17:31.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=575971623}) 2023-10-30T12:17:31.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.271+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967494757}) 2023-10-30T12:17:31.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.272+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.272+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956067111}) 2023-10-30T12:17:31.359 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.272+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3620457204}) 2023-10-30T12:17:31.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.272+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.272+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956067111}) 2023-10-30T12:17:31.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.272+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281057084}) 2023-10-30T12:17:31.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.273+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.360 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.273+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317973738}) 2023-10-30T12:17:31.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.273+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2229110994}) 2023-10-30T12:17:31.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.273+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.273+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317973738}) 2023-10-30T12:17:31.361 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.273+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087246728}) 2023-10-30T12:17:31.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901028267}) 2023-10-30T12:17:31.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713617258}) 2023-10-30T12:17:31.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.362 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901028267}) 2023-10-30T12:17:31.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2374544525}) 2023-10-30T12:17:31.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879229920}) 2023-10-30T12:17:31.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2597130481}) 2023-10-30T12:17:31.363 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879229920}) 2023-10-30T12:17:31.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.274+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3131467983}) 2023-10-30T12:17:31.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.275+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.364 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.275+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236170891}) 2023-10-30T12:17:31.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.275+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436390255}) 2023-10-30T12:17:31.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.275+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.275+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236170891}) 2023-10-30T12:17:31.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.275+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3113432275}) 2023-10-30T12:17:31.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.276+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.365 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.276+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733334707}) 2023-10-30T12:17:31.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.276+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2285693587}) 2023-10-30T12:17:31.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.276+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.276+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733334707}) 2023-10-30T12:17:31.366 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.276+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626886911}) 2023-10-30T12:17:31.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836628723}) 2023-10-30T12:17:31.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1637338785}) 2023-10-30T12:17:31.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836628723}) 2023-10-30T12:17:31.367 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250141545}) 2023-10-30T12:17:31.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050047113}) 2023-10-30T12:17:31.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979910105}) 2023-10-30T12:17:31.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.368 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050047113}) 2023-10-30T12:17:31.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.277+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545188319}) 2023-10-30T12:17:31.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.278+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.278+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150611002}) 2023-10-30T12:17:31.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.278+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2569844499}) 2023-10-30T12:17:31.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.278+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.278+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150611002}) 2023-10-30T12:17:31.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.278+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1954254104}) 2023-10-30T12:17:31.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.370 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510525122}) 2023-10-30T12:17:31.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2597007208}) 2023-10-30T12:17:31.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510525122}) 2023-10-30T12:17:31.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416809484}) 2023-10-30T12:17:31.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.371 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237956348}) 2023-10-30T12:17:31.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3738303556}) 2023-10-30T12:17:31.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237956348}) 2023-10-30T12:17:31.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.279+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2625509316}) 2023-10-30T12:17:31.372 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.280+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.280+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385727810}) 2023-10-30T12:17:31.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.280+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466441962}) 2023-10-30T12:17:31.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.280+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.280+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385727810}) 2023-10-30T12:17:31.373 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.280+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149547956}) 2023-10-30T12:17:31.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.281+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.281+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351656149}) 2023-10-30T12:17:31.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.281+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045868758}) 2023-10-30T12:17:31.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.281+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.374 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.281+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351656149}) 2023-10-30T12:17:31.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.281+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247735027}) 2023-10-30T12:17:31.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422926662}) 2023-10-30T12:17:31.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4094403054}) 2023-10-30T12:17:31.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422926662}) 2023-10-30T12:17:31.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1677923335}) 2023-10-30T12:17:31.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953499396}) 2023-10-30T12:17:31.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2477324580}) 2023-10-30T12:17:31.376 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953499396}) 2023-10-30T12:17:31.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.282+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571599711}) 2023-10-30T12:17:31.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.283+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.283+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2580041841}) 2023-10-30T12:17:31.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.283+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1567923197}) 2023-10-30T12:17:31.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.283+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.283+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2580041841}) 2023-10-30T12:17:31.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.283+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3976969056}) 2023-10-30T12:17:31.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373153571}) 2023-10-30T12:17:31.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050304141}) 2023-10-30T12:17:31.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373153571}) 2023-10-30T12:17:31.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757897227}) 2023-10-30T12:17:31.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263900479}) 2023-10-30T12:17:31.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068320800}) 2023-10-30T12:17:31.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.284+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263900479}) 2023-10-30T12:17:31.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333842011}) 2023-10-30T12:17:31.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770733446}) 2023-10-30T12:17:31.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353680679}) 2023-10-30T12:17:31.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770733446}) 2023-10-30T12:17:31.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.285+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590841507}) 2023-10-30T12:17:31.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.286+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.286+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384901889}) 2023-10-30T12:17:31.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.286+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=393898023}) 2023-10-30T12:17:31.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.286+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.286+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384901889}) 2023-10-30T12:17:31.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.286+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75356506}) 2023-10-30T12:17:31.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438793640}) 2023-10-30T12:17:31.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769856611}) 2023-10-30T12:17:31.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438793640}) 2023-10-30T12:17:31.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2255556162}) 2023-10-30T12:17:31.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570688147}) 2023-10-30T12:17:31.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3005210983}) 2023-10-30T12:17:31.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570688147}) 2023-10-30T12:17:31.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.287+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737280035}) 2023-10-30T12:17:31.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.288+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.288+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369611113}) 2023-10-30T12:17:31.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.288+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678391721}) 2023-10-30T12:17:31.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.288+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.288+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369611113}) 2023-10-30T12:17:31.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.288+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988364466}) 2023-10-30T12:17:31.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.289+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.289+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172434333}) 2023-10-30T12:17:31.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.289+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614286490}) 2023-10-30T12:17:31.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.289+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.289+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172434333}) 2023-10-30T12:17:31.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.289+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20830504}) 2023-10-30T12:17:31.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398923672}) 2023-10-30T12:17:31.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190686501}) 2023-10-30T12:17:31.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398923672}) 2023-10-30T12:17:31.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284550085}) 2023-10-30T12:17:31.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602593295}) 2023-10-30T12:17:31.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=797137845}) 2023-10-30T12:17:31.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602593295}) 2023-10-30T12:17:31.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.290+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1001072237}) 2023-10-30T12:17:31.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.291+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.291+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230836466}) 2023-10-30T12:17:31.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.291+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2426286362}) 2023-10-30T12:17:31.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.291+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.291+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230836466}) 2023-10-30T12:17:31.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.291+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082988058}) 2023-10-30T12:17:31.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127248359}) 2023-10-30T12:17:31.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1551955593}) 2023-10-30T12:17:31.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127248359}) 2023-10-30T12:17:31.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111700539}) 2023-10-30T12:17:31.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672844746}) 2023-10-30T12:17:31.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2426344696}) 2023-10-30T12:17:31.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672844746}) 2023-10-30T12:17:31.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.292+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474467467}) 2023-10-30T12:17:31.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.293+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.293+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232269320}) 2023-10-30T12:17:31.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.293+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878795111}) 2023-10-30T12:17:31.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.293+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.293+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232269320}) 2023-10-30T12:17:31.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.293+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553224946}) 2023-10-30T12:17:31.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.294+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.294+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3158233217,mdsmap=361818223}) 2023-10-30T12:17:31.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.294+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=2702661626,mdsmap=3984425268,mgr=11346411}) 2023-10-30T12:17:31.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.294+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.294+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3158233217,mdsmap=361818223}) 2023-10-30T12:17:31.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.294+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=4042785784,mdsmap=3984425268}) 2023-10-30T12:17:31.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.295+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.295+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=26} crc {mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1759322679,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=3105363944,osd_snap=2049487035,osdmap=300303636}) 2023-10-30T12:17:31.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.295+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=25} crc {mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1759322679,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=3105363944,osd_snap=2566033224,osdmap=15192974}) 2023-10-30T12:17:31.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.295+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.295+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=26} crc {mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1759322679,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=3105363944,osd_snap=2049487035,osdmap=300303636}) 2023-10-30T12:17:31.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.295+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=26} crc {mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=677568569,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=3105363944,osd_snap=1713215561,osdmap=1360972147}) 2023-10-30T12:17:31.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.299+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.299+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=38} crc {osdmap=596722202}) 2023-10-30T12:17:31.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.299+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=38} crc {osdmap=2255454458}) 2023-10-30T12:17:31.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.299+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:17:31.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.299+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=38} crc {osdmap=596722202}) 2023-10-30T12:17:31.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:17:31.299+0000 7f7094d36640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=1641979693}) 2023-10-30T12:17:32.262 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:17:32.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:17:32.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.378+0000 7f5a545e9640 10 monclient: get_monmap_and_config 2023-10-30T12:17:32.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.378+0000 7f5a545e9640 10 monclient: build_initial_monmap 2023-10-30T12:17:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.378+0000 7f5a545e9640 10 monclient: monmap: 2023-10-30T12:17:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:32.379135+0000 2023-10-30T12:17:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:32.379135+0000 2023-10-30T12:17:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:32.382 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.378+0000 7f5a545e9640 1 Processor -- start 2023-10-30T12:17:32.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.378+0000 7f5a545e9640 1 -- start start 2023-10-30T12:17:32.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.378+0000 7f5a545e9640 10 monclient: init 2023-10-30T12:17:32.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:32.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-10-30T12:17:32.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5a4c0a4b60 0x7f5a4c1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c0a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:32.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a4c099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:32.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c132be0 0x7f5a4c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient(hunting): picked mon.noname-b con 0x7f5a4c132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:32.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0a4f40 con 0x7f5a4c099a40 2023-10-30T12:17:32.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4c0a3d70 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0a4150 con 0x7f5a4c132be0 2023-10-30T12:17:32.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:32.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a545e9640 10 monclient(hunting): authenticate will time out at 1897.562420s 2023-10-30T12:17:32.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c132be0 0x7f5a4c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c132be0 0x7f5a4c0a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52216/0 (socket says 172.21.15.50:52216) 2023-10-30T12:17:32.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 -- 172.21.15.50:0/3821156776 learned_addr learned my addr 172.21.15.50:0/3821156776 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): get_auth_request con 0x7f5a4c132be0 auth_method 0 2023-10-30T12:17:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52385640 1 --2- 172.21.15.50:0/3821156776 >> v2:172.21.15.118:3300/0 conn(0x7f5a4c0a4b60 0x7f5a4c1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52385640 10 monclient(hunting): get_auth_request con 0x7f5a4c0a4b60 auth_method 0 2023-10-30T12:17:32.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52385640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52385640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient(hunting): handle_auth_done global_id 224114 payload 274 2023-10-30T12:17:32.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 -- 172.21.15.50:0/3821156776 >> v2:172.21.15.118:3300/0 conn(0x7f5a4c0a4b60 msgr2=0x7f5a4c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:32.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 --2- 172.21.15.50:0/3821156776 >> v2:172.21.15.118:3300/0 conn(0x7f5a4c0a4b60 0x7f5a4c1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 -- 172.21.15.50:0/3821156776 >> v1:172.21.15.118:6789/0 conn(0x7f5a4c099a40 legacy=0x7f5a4c0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:32.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient: _finish_hunting 0 2023-10-30T12:17:32.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 monclient: found mon.noname-b 2023-10-30T12:17:32.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:17:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.379+0000 7f5a52b86640 1 -- 172.21.15.50:0/3821156776 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c09a440 con 0x7f5a4c132be0 2023-10-30T12:17:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a52b86640 1 --2- 172.21.15.50:0/3821156776 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c132be0 0x7f5a4c0a2a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5a44009870 tx=0x7f5a4402b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d464e86708e353 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 1 -- 172.21.15.50:0/3821156776 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a44031740 con 0x7f5a4c132be0 2023-10-30T12:17:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: dump: 2023-10-30T12:17:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.399 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.400 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.400 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.400 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.400 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.400 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.401 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: _finish_auth 0 2023-10-30T12:17:32.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: _check_auth_tickets 2023-10-30T12:17:32.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:32.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 1 -- 172.21.15.50:0/3821156776 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a440318a0 con 0x7f5a4c132be0 2023-10-30T12:17:32.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a545e9640 5 monclient: authenticate success, global_id 224114 2023-10-30T12:17:32.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a545e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:32.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:32.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a545e9640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:32.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 1 -- 172.21.15.50:0/3821156776 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a44031c30 con 0x7f5a4c132be0 2023-10-30T12:17:32.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a545e9640 4 set_mon_vals no callback set 2023-10-30T12:17:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.380+0000 7f5a51383640 10 monclient: dump: 2023-10-30T12:17:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.404 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.404 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.404 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.404 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.404 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.404 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.405 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.405 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 10 monclient: shutdown 2023-10-30T12:17:32.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:32.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 -- 172.21.15.50:0/3821156776 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c132be0 msgr2=0x7f5a4c0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 --2- 172.21.15.50:0/3821156776 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c132be0 0x7f5a4c0a2a60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5a44009870 tx=0x7f5a4402b7e0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 -- 172.21.15.50:0/3821156776 shutdown_connections 2023-10-30T12:17:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 --2- 172.21.15.50:0/3821156776 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c132be0 0x7f5a4c0a2a60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 --2- 172.21.15.50:0/3821156776 >> v2:172.21.15.118:3300/0 conn(0x7f5a4c0a4b60 0x7f5a4c1326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 -- 172.21.15.50:0/3821156776 >> 172.21.15.50:0/3821156776 conn(0x7f5a4c0a8060 msgr2=0x7f5a4c0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 -- 172.21.15.50:0/3821156776 shutdown_connections 2023-10-30T12:17:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.381+0000 7f5a545e9640 1 -- 172.21.15.50:0/3821156776 wait complete. 2023-10-30T12:17:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient: build_initial_monmap 2023-10-30T12:17:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient: monmap: 2023-10-30T12:17:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:32.409 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 librados: starting msgr at 2023-10-30T12:17:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 librados: starting objecter 2023-10-30T12:17:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 Processor -- start 2023-10-30T12:17:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 -- start start 2023-10-30T12:17:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 librados: setting wanted keys 2023-10-30T12:17:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 librados: calling monclient init 2023-10-30T12:17:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient: init 2023-10-30T12:17:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c0a4b60 0x7f5a4c1ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c0a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a4c132be0 0x7f5a4c1cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4c132be0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:32.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a4c11e5d0 0x7f5a4c11ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): picked mon.noname-b con 0x7f5a4c11e5d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:32.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c1223f0 con 0x7f5a4c132be0 2023-10-30T12:17:32.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c122640 con 0x7f5a4c11e5d0 2023-10-30T12:17:32.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c120eb0 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.382+0000 7f5a545e9640 10 monclient(hunting): authenticate will time out at 1897.565926s 2023-10-30T12:17:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c0a4b60 0x7f5a4c1ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c0a4b60 0x7f5a4c1ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52220/0 (socket says 172.21.15.50:52220) 2023-10-30T12:17:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 -- 172.21.15.50:0/294725331 learned_addr learned my addr 172.21.15.50:0/294725331 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): get_auth_request con 0x7f5a4c0a4b60 auth_method 0 2023-10-30T12:17:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a4c132be0 0x7f5a4c1cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 10 monclient(hunting): get_auth_request con 0x7f5a4c132be0 auth_method 0 2023-10-30T12:17:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52b86640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a4c11e5d0 0x7f5a4c11ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52b86640 10 monclient(hunting): get_auth_request con 0x7f5a4c11e5d0 auth_method 0 2023-10-30T12:17:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52b86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52b86640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52b86640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a51b84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient(hunting): handle_auth_done global_id 224117 payload 824 2023-10-30T12:17:32.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 -- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a4c11e5d0 msgr2=0x7f5a4c11ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a4c11e5d0 0x7f5a4c11ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 -- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a4c132be0 msgr2=0x7f5a4c1cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a4c132be0 0x7f5a4c1cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient: _finish_hunting 0 2023-10-30T12:17:32.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 monclient: found mon.noname-c 2023-10-30T12:17:32.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:32.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:32.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 -- 172.21.15.50:0/294725331 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a440094d0 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52b86640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a4c11e5d0 0x7f5a4c11ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:17:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a52385640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c0a4b60 0x7f5a4c1ca8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a48007f20 tx=0x7f5a4800e570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 client.?.objecter ms_handle_connect 0x7f5a4c0a4b60 2023-10-30T12:17:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 1 -- 172.21.15.50:0/294725331 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48014620 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:32.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: dump: 2023-10-30T12:17:32.422 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.422 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.422 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.422 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.423 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.423 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.423 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.423 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.423 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.424 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:32.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:32.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:32.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 5 monclient: authenticate success, global_id 224117 2023-10-30T12:17:32.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 1 -- 172.21.15.50:0/294725331 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a48014780 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:32.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 1 -- 172.21.15.50:0/294725331 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48014b30 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 10 monclient: _renew_subs 2023-10-30T12:17:32.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:32.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c135d80 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a50b82640 4 set_mon_vals no callback set 2023-10-30T12:17:32.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:32.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a42ffd640 10 monclient: dump: 2023-10-30T12:17:32.427 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.427 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.427 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.427 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.427 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.427 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.428 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.428 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.428 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.428 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 10 client.224117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:32.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 10 monclient: _renew_subs 2023-10-30T12:17:32.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:32.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c120270 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.383+0000 7f5a545e9640 1 librados: init done 2023-10-30T12:17:32.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.384+0000 7f5a42ffd640 1 -- 172.21.15.50:0/294725331 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5a48014c90 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a545e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a545e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a545e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a42ffd640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a300392f0 0x7f5a3003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4c04e160 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a42ffd640 1 -- 172.21.15.50:0/294725331 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f5a4804b690 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a42ffd640 10 client.224117.objecter ms_dispatch 0x7f5a4c000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a42ffd640 3 client.224117.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a42ffd640 3 client.224117.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a42ffd640 20 client.224117.objecter dump_active .. 0 homeless 2023-10-30T12:17:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a51b84640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a300392f0 0x7f5a3003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a51b84640 10 monclient: get_auth_request con 0x7f5a300392f0 auth_method 0 2023-10-30T12:17:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a42ffd640 10 client.224117.objecter ms_handle_connect 0x7f5a300392f0 2023-10-30T12:17:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.385+0000 7f5a51b84640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a300392f0 0x7f5a3003b7b0 secure :-1 s=READY pgs=4249 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c1350f0 tx=0x7f5a3c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.389+0000 7f5a42ffd640 1 -- 172.21.15.50:0/294725331 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a4801d3c0 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.389+0000 7f5a42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.389+0000 7f5a42ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:32.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.632+0000 7f5a545e9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:17:32.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.632+0000 7f5a545e9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:32.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.632+0000 7f5a545e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.632+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5a4c122780 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.633+0000 7f5a42ffd640 1 -- 172.21.15.50:0/294725331 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f5a4c122780 con 0x7f5a4c0a4b60 2023-10-30T12:17:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.633+0000 7f5a42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:32.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.633+0000 7f5a42ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:32.636 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:17:32.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.634+0000 7f5a545e9640 10 librados: watch_flush enter 2023-10-30T12:17:32.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 10 librados: watch_flush exit 2023-10-30T12:17:32.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 20 client.224117.objecter shutdown clearing up homeless session... 2023-10-30T12:17:32.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 10 client.224117.objecter successfully canceled tick 2023-10-30T12:17:32.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a300392f0 msgr2=0x7f5a3003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a300392f0 0x7f5a3003b7b0 secure :-1 s=READY pgs=4249 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c1350f0 tx=0x7f5a3c009060 comp rx=0 tx=0).stop 2023-10-30T12:17:32.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 10 monclient: shutdown 2023-10-30T12:17:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c0a4b60 msgr2=0x7f5a4c1ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c0a4b60 0x7f5a4c1ca8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a48007f20 tx=0x7f5a4800e570 comp rx=0 tx=0).stop 2023-10-30T12:17:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 shutdown_connections 2023-10-30T12:17:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a300392f0 0x7f5a3003b7b0 unknown :-1 s=CLOSED pgs=4249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a4c11e5d0 0x7f5a4c11ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a4c132be0 0x7f5a4c1cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 --2- 172.21.15.50:0/294725331 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a4c0a4b60 0x7f5a4c1ca8c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 >> 172.21.15.50:0/294725331 conn(0x7f5a4c0a8060 msgr2=0x7f5a4c1332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:32.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 shutdown_connections 2023-10-30T12:17:32.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 -- 172.21.15.50:0/294725331 wait complete. 2023-10-30T12:17:32.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.635+0000 7f5a545e9640 1 librados: shutdown 2023-10-30T12:17:32.646 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:17:32.647 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:17:32.647 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:17:32.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:17:32.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.751+0000 7f0f12dc5640 10 monclient: get_monmap_and_config 2023-10-30T12:17:32.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.751+0000 7f0f12dc5640 10 monclient: build_initial_monmap 2023-10-30T12:17:32.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.751+0000 7f0f12dc5640 10 monclient: monmap: 2023-10-30T12:17:32.754 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:32.754 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.754 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:32.752939+0000 2023-10-30T12:17:32.755 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:32.752939+0000 2023-10-30T12:17:32.755 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:32.755 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.755 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:32.755 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:32.755 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:32.756 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:32.756 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:32.756 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.752+0000 7f0f12dc5640 1 Processor -- start 2023-10-30T12:17:32.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.752+0000 7f0f12dc5640 1 -- start start 2023-10-30T12:17:32.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.752+0000 7f0f12dc5640 10 monclient: init 2023-10-30T12:17:32.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:32.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-10-30T12:17:32.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c0a8010 0x7f0f0c132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient(hunting): picked mon.noname-b con 0x7f0f0c0a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:32.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f0c099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:32.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0f0c132bd0 0x7f0f0c135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient(hunting): picked mon.noname-c con 0x7f0f0c132bd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:32.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c0a7190 con 0x7f0f0c099a40 2023-10-30T12:17:32.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0c0a83f0 con 0x7f0f0c132bd0 2023-10-30T12:17:32.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0a7600 con 0x7f0f0c0a8010 2023-10-30T12:17:32.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:32.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f12dc5640 10 monclient(hunting): authenticate will time out at 1897.936217s 2023-10-30T12:17:32.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c0a8010 0x7f0f0c132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c0a8010 0x7f0f0c132690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52230/0 (socket says 172.21.15.50:52230) 2023-10-30T12:17:32.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 -- 172.21.15.50:0/3772174436 learned_addr learned my addr 172.21.15.50:0/3772174436 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:32.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): get_auth_request con 0x7f0f0c0a8010 auth_method 0 2023-10-30T12:17:32.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f11362640 1 --2- 172.21.15.50:0/3772174436 >> v2:172.21.15.118:3300/0 conn(0x7f0f0c132bd0 0x7f0f0c135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f11362640 10 monclient(hunting): get_auth_request con 0x7f0f0c132bd0 auth_method 0 2023-10-30T12:17:32.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f11362640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f11362640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f11362640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient(hunting): handle_auth_done global_id 224120 payload 274 2023-10-30T12:17:32.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 -- 172.21.15.50:0/3772174436 >> v2:172.21.15.118:3300/0 conn(0x7f0f0c132bd0 msgr2=0x7f0f0c135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:32.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 --2- 172.21.15.50:0/3772174436 >> v2:172.21.15.118:3300/0 conn(0x7f0f0c132bd0 0x7f0f0c135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 -- 172.21.15.50:0/3772174436 >> v1:172.21.15.118:6789/0 conn(0x7f0f0c099a40 legacy=0x7f0f0c1356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient: _finish_hunting 0 2023-10-30T12:17:32.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 monclient: found mon.noname-b 2023-10-30T12:17:32.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:32.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:17:32.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.753+0000 7f0f10b61640 1 -- 172.21.15.50:0/3772174436 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c09a440 con 0x7f0f0c0a8010 2023-10-30T12:17:32.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f10b61640 1 --2- 172.21.15.50:0/3772174436 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c0a8010 0x7f0f0c132690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009e50 tx=0x7f0f0402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4d9ce159b6e4f98 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 1 -- 172.21.15.50:0/3772174436 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0402c9e0 con 0x7f0f0c0a8010 2023-10-30T12:17:32.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:32.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: dump: 2023-10-30T12:17:32.776 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.776 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.776 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.776 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.777 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.777 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.777 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.777 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.777 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.778 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:32.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:32.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:32.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 1 -- 172.21.15.50:0/3772174436 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0402cb40 con 0x7f0f0c0a8010 2023-10-30T12:17:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 1 -- 172.21.15.50:0/3772174436 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f040343f0 con 0x7f0f0c0a8010 2023-10-30T12:17:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f12dc5640 5 monclient: authenticate success, global_id 224120 2023-10-30T12:17:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f12dc5640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f12dc5640 4 set_mon_vals no callback set 2023-10-30T12:17:32.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:32.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.754+0000 7f0f0b7fe640 10 monclient: dump: 2023-10-30T12:17:32.780 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.781 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.781 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.781 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.781 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.781 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.782 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.782 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.782 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.782 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 10 monclient: shutdown 2023-10-30T12:17:32.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:32.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3772174436 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c0a8010 msgr2=0x7f0f0c132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 --2- 172.21.15.50:0/3772174436 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c0a8010 0x7f0f0c132690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009e50 tx=0x7f0f0402be40 comp rx=0 tx=0).stop 2023-10-30T12:17:32.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3772174436 shutdown_connections 2023-10-30T12:17:32.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 --2- 172.21.15.50:0/3772174436 >> v2:172.21.15.118:3300/0 conn(0x7f0f0c132bd0 0x7f0f0c135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 --2- 172.21.15.50:0/3772174436 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c0a8010 0x7f0f0c132690 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3772174436 >> 172.21.15.50:0/3772174436 conn(0x7f0f0c0a4a10 msgr2=0x7f0f0c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:32.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3772174436 shutdown_connections 2023-10-30T12:17:32.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3772174436 wait complete. 2023-10-30T12:17:32.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 10 monclient: build_initial_monmap 2023-10-30T12:17:32.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 10 monclient: monmap: 2023-10-30T12:17:32.785 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:32.787 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 librados: starting msgr at 2023-10-30T12:17:32.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 librados: starting objecter 2023-10-30T12:17:32.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.755+0000 7f0f12dc5640 1 Processor -- start 2023-10-30T12:17:32.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 -- start start 2023-10-30T12:17:32.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 librados: setting wanted keys 2023-10-30T12:17:32.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 librados: calling monclient init 2023-10-30T12:17:32.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient: init 2023-10-30T12:17:32.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:32.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:17:32.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c0a8010 0x7f0f0c1ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): picked mon.noname-a con 0x7f0f0c0a8010 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:32.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c132bd0 0x7f0f0c1cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): picked mon.noname-c con 0x7f0f0c132bd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:32.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f0c11e6f0 0x7f0f0c11eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): picked mon.noname-b con 0x7f0f0c11e6f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:32.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c121040 con 0x7f0f0c0a8010 2023-10-30T12:17:32.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c135940 con 0x7f0f0c11e6f0 2023-10-30T12:17:32.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0a18d0 con 0x7f0f0c132bd0 2023-10-30T12:17:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f12dc5640 10 monclient(hunting): authenticate will time out at 1897.939608s 2023-10-30T12:17:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c0a8010 0x7f0f0c1ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c132bd0 0x7f0f0c1cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c0a8010 0x7f0f0c1ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51564/0 (socket says 172.21.15.50:51564) 2023-10-30T12:17:32.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c132bd0 0x7f0f0c1cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52234/0 (socket says 172.21.15.50:52234) 2023-10-30T12:17:32.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 1 -- 172.21.15.50:0/3770053394 learned_addr learned my addr 172.21.15.50:0/3770053394 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:32.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 10 monclient(hunting): get_auth_request con 0x7f0f0c0a8010 auth_method 0 2023-10-30T12:17:32.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f0c11e6f0 0x7f0f0c11eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 10 monclient(hunting): get_auth_request con 0x7f0f0c132bd0 auth_method 0 2023-10-30T12:17:32.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 10 monclient(hunting): get_auth_request con 0x7f0f0c11e6f0 auth_method 0 2023-10-30T12:17:32.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f10b61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.756+0000 7f0f11362640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 10 monclient(hunting): handle_auth_done global_id 214272 payload 824 2023-10-30T12:17:32.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c132bd0 msgr2=0x7f0f0c1cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f0c132bd0 0x7f0f0c1cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f0c11e6f0 msgr2=0x7f0f0c11eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f0c11e6f0 0x7f0f0c11eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 10 monclient: _finish_hunting 0 2023-10-30T12:17:32.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 1 monclient: found mon.noname-a 2023-10-30T12:17:32.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:32.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:32.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 1 -- 172.21.15.50:0/3770053394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f04009b30 con 0x7f0f0c0a8010 2023-10-30T12:17:32.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f10b61640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c0a8010 0x7f0f0c1ca630 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0f04000f80 tx=0x7f0f04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 client.?.objecter ms_handle_connect 0x7f0f0c0a8010 2023-10-30T12:17:32.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:32.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0403e450 con 0x7f0f0c0a8010 2023-10-30T12:17:32.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:32.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: dump: 2023-10-30T12:17:32.802 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.802 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.802 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.802 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.803 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.803 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.803 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.803 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.803 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.804 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: _finish_auth 0 2023-10-30T12:17:32.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: _check_auth_tickets 2023-10-30T12:17:32.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:32.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0403e5b0 con 0x7f0f0c0a8010 2023-10-30T12:17:32.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 5 monclient: authenticate success, global_id 214272 2023-10-30T12:17:32.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 10 monclient: _renew_subs 2023-10-30T12:17:32.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:32.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c122760 con 0x7f0f0c0a8010 2023-10-30T12:17:32.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:32.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0403e960 con 0x7f0f0c0a8010 2023-10-30T12:17:32.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:32.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f097fa640 10 monclient: dump: 2023-10-30T12:17:32.807 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.807 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.808 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.808 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.808 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.808 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.808 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.809 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.809 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.809 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 10 client.214272.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:32.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 10 monclient: _renew_subs 2023-10-30T12:17:32.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:32.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c120390 con 0x7f0f0c0a8010 2023-10-30T12:17:32.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f12dc5640 1 librados: init done 2023-10-30T12:17:32.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.757+0000 7f0f0affd640 4 set_mon_vals no callback set 2023-10-30T12:17:32.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0f0403eac0 con 0x7f0f0c0a8010 2023-10-30T12:17:32.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f12dc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:32.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f12dc5640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c099a40 0x7f0f0c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f12dc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c073fd0 con 0x7f0f0c099a40 2023-10-30T12:17:32.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c099a40 0x7f0f0c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient: get_auth_request con 0x7f0f0c099a40 auth_method 0 2023-10-30T12:17:32.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f097fa640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0eec0392f0 0x7f0eec03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f0f0406dc30 con 0x7f0f0c0a8010 2023-10-30T12:17:32.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f097fa640 10 client.214272.objecter ms_dispatch 0x7f0f0c000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:32.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f097fa640 3 client.214272.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:32.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f097fa640 3 client.214272.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:32.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f097fa640 20 client.214272.objecter dump_active .. 0 homeless 2023-10-30T12:17:32.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f11362640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0eec0392f0 0x7f0eec03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f11362640 10 monclient: get_auth_request con 0x7f0eec0392f0 auth_method 0 2023-10-30T12:17:32.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_done global_id 214275 payload 824 2023-10-30T12:17:32.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.758+0000 7f0f0bfff640 1 -- 172.21.15.50:0/3770053394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f0c09aa00 con 0x7f0f0c099a40 2023-10-30T12:17:32.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 client.214272.objecter ms_handle_connect 0x7f0f0c099a40 2023-10-30T12:17:32.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 client.214272.objecter resend_mon_ops 2023-10-30T12:17:32.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f0bfff640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c099a40 0x7f0f0c02b430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c11ff90 tx=0x7f0efc00f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc0052e0 con 0x7f0f0c099a40 2023-10-30T12:17:32.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:32.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 monclient: dump: 2023-10-30T12:17:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.817 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.818 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.818 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.818 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.818 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.818 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 client.214272.objecter ms_handle_connect 0x7f0eec0392f0 2023-10-30T12:17:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f11362640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0eec0392f0 0x7f0eec03b7b0 secure :-1 s=READY pgs=4250 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4006fd0 tx=0x7f0ef4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0efc005440 con 0x7f0f0c099a40 2023-10-30T12:17:32.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:32.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:32.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c099a40 msgr2=0x7f0f0c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c099a40 0x7f0f0c02b430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c11ff90 tx=0x7f0efc00f9d0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c099a40 msgr2=0x7f0f0c02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:32.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.759+0000 7f0f097fa640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c099a40 0x7f0f0c02b430 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c11ff90 tx=0x7f0efc00f9d0 comp rx=0 tx=0).stop 2023-10-30T12:17:32.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f12dc5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f12dc5640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c11f880 0x7f0f0c02b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f12dc5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c122260 con 0x7f0f0c11f880 2023-10-30T12:17:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c11f880 0x7f0f0c02b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 10 monclient: get_auth_request con 0x7f0f0c11f880 auth_method 0 2023-10-30T12:17:32.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.765+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f0bfff640 10 monclient(hunting): handle_auth_done global_id 214278 payload 824 2023-10-30T12:17:32.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f0bfff640 1 -- 172.21.15.50:0/3770053394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0f0c09a720 con 0x7f0f0c11f880 2023-10-30T12:17:32.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f0bfff640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c11f880 0x7f0f0c02b970 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c11ff90 tx=0x7f0efc001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:32.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 10 client.214272.objecter ms_handle_connect 0x7f0f0c11f880 2023-10-30T12:17:32.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 10 client.214272.objecter resend_mon_ops 2023-10-30T12:17:32.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc024070 con 0x7f0f0c11f880 2023-10-30T12:17:32.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:32.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:32.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 10 monclient: dump: 2023-10-30T12:17:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:32.832 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7f0efc007030 con 0x7f0f0c11f880 2023-10-30T12:17:32.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:32.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:32.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c11f880 msgr2=0x7f0f0c02b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c11f880 0x7f0f0c02b970 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c11ff90 tx=0x7f0efc001080 comp rx=0 tx=0).stop 2023-10-30T12:17:32.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c11f880 msgr2=0x7f0f0c02b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:32.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f097fa640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c11f880 0x7f0f0c02b970 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c11ff90 tx=0x7f0efc001080 comp rx=0 tx=0).stop 2023-10-30T12:17:32.834 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:17:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 176, 2023-10-30T12:17:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:17:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.858 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:32.858 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:32.858 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:32.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.859 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:32.859 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:32.859 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:32.859 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.859 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:32.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.860 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:32.860 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:32.860 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:32.860 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:32.861 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:32.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.861 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:32.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.861 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:32.861 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:32.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.862 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:32.862 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:32.862 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:32.862 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.863 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:32.863 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:17:32.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.863 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:32.863 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:32.864 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:17:32.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.864 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:32.864 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:32.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.865 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:32.865 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:32.865 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:17:32.865 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.865 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:32.865 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:17:32.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:32.866 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:32.866 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:32.866 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:32.866 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:32.867 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:32.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:32.867 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:32.867 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:32.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 10 librados: watch_flush enter 2023-10-30T12:17:32.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 10 librados: watch_flush exit 2023-10-30T12:17:32.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 20 client.214272.objecter shutdown clearing up homeless session... 2023-10-30T12:17:32.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 10 client.214272.objecter successfully canceled tick 2023-10-30T12:17:32.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0eec0392f0 msgr2=0x7f0eec03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0eec0392f0 0x7f0eec03b7b0 secure :-1 s=READY pgs=4250 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4006fd0 tx=0x7f0ef4008040 comp rx=0 tx=0).stop 2023-10-30T12:17:32.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 10 monclient: shutdown 2023-10-30T12:17:32.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:32.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c0a8010 msgr2=0x7f0f0c1ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:32.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f12dc5640 1 --2- 172.21.15.50:0/3770053394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f0c0a8010 0x7f0f0c1ca630 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0f04000f80 tx=0x7f0f04004a40 comp rx=0 tx=0).stop 2023-10-30T12:17:32.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.766+0000 7f0f11362640 1 -- 172.21.15.50:0/3770053394 reap_dead start 2023-10-30T12:17:32.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.767+0000 7f0f11362640 1 -- 172.21.15.50:0/3770053394 reap_dead start 2023-10-30T12:17:32.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.767+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 shutdown_connections 2023-10-30T12:17:32.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.767+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 >> 172.21.15.50:0/3770053394 conn(0x7f0f0c0a4a10 msgr2=0x7f0f0c135b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:32.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.767+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 shutdown_connections 2023-10-30T12:17:32.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.767+0000 7f0f12dc5640 1 -- 172.21.15.50:0/3770053394 wait complete. 2023-10-30T12:17:32.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.767+0000 7f0f12dc5640 1 librados: shutdown 2023-10-30T12:17:32.873 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:17:32.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:17:32.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:17:32.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.980+0000 7f3e4efd0640 10 monclient: get_monmap_and_config 2023-10-30T12:17:32.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.980+0000 7f3e4efd0640 10 monclient: build_initial_monmap 2023-10-30T12:17:32.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.980+0000 7f3e4efd0640 10 monclient: monmap: 2023-10-30T12:17:32.982 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:32.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:32.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:32.981692+0000 2023-10-30T12:17:32.983 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:32.981692+0000 2023-10-30T12:17:32.983 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:32.983 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:32.983 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:32.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:32.984 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:32.984 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:32.984 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:32.984 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:32.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 1 Processor -- start 2023-10-30T12:17:32.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 1 -- start start 2023-10-30T12:17:32.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient: init 2023-10-30T12:17:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T12:17:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e4806afc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e480a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:32.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4806b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e4813b070 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4806bde0 con 0x7f3e4813b070 2023-10-30T12:17:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4806b3a0 con 0x7f3e4806afc0 2023-10-30T12:17:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4809e540 con 0x7f3e4806b640 2023-10-30T12:17:32.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:32.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4efd0640 10 monclient(hunting): authenticate will time out at 1898.164863s 2023-10-30T12:17:32.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e480a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:32.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.981+0000 7f3e4d56d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3e4813b070 0x7f3e4813d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45546/0 (socket says 172.21.15.50:45546) 2023-10-30T12:17:32.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e480a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52250/0 (socket says 172.21.15.50:52250) 2023-10-30T12:17:32.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e4d56d640 1 -- 172.21.15.50:0/2855547927 learned_addr learned my addr 172.21.15.50:0/2855547927 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:17:32.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): get_auth_request con 0x7f3e4806b640 auth_method 0 2023-10-30T12:17:32.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:32.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:32.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:32.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:32.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e477fe640 1 -- 172.21.15.50:0/2855547927 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623093870 0 0) 0x7f3e4806bde0 con 0x7f3e4813b070 2023-10-30T12:17:32.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e477fe640 10 monclient(hunting): _negotiate 2023-10-30T12:17:32.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e477fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:32.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e477fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:32.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e477fe640 10 monclient(hunting): my global_id is 214281 2023-10-30T12:17:32.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e477fe640 1 -- 172.21.15.50:0/2855547927 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e38003520 con 0x7f3e4813b070 2023-10-30T12:17:32.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient(hunting): handle_auth_done global_id 224126 payload 274 2023-10-30T12:17:32.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 1 -- 172.21.15.50:0/2855547927 >> v1:172.21.15.118:6789/0 conn(0x7f3e4806afc0 legacy=0x7f3e480a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:32.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 1 -- 172.21.15.50:0/2855547927 >> v1:172.21.15.50:6789/0 conn(0x7f3e4813b070 legacy=0x7f3e4813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:33.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 1 monclient: found mon.noname-b 2023-10-30T12:17:33.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:33.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.982+0000 7f3e47fff640 1 -- 172.21.15.50:0/2855547927 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4809fa10 con 0x7f3e4806b640 2023-10-30T12:17:33.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e47fff640 1 --2- 172.21.15.50:0/2855547927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e480a23a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3e4806b530 tx=0x7f3e3402b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e6bcd4bd5389302 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 1 -- 172.21.15.50:0/2855547927 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34030710 con 0x7f3e4806b640 2023-10-30T12:17:33.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:33.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: dump: 2023-10-30T12:17:33.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.002 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.002 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.002 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.003 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:33.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:33.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:33.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 1 -- 172.21.15.50:0/2855547927 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3402e050 con 0x7f3e4806b640 2023-10-30T12:17:33.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 5 monclient: authenticate success, global_id 224126 2023-10-30T12:17:33.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:33.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:33.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:33.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 4 set_mon_vals no callback set 2023-10-30T12:17:33.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 1 -- 172.21.15.50:0/2855547927 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34030a90 con 0x7f3e4806b640 2023-10-30T12:17:33.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e477fe640 10 monclient: dump: 2023-10-30T12:17:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.006 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.007 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.007 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 10 monclient: shutdown 2023-10-30T12:17:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 1 -- 172.21.15.50:0/2855547927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 msgr2=0x7f3e480a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 1 --2- 172.21.15.50:0/2855547927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e480a23a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3e4806b530 tx=0x7f3e3402b780 comp rx=0 tx=0).stop 2023-10-30T12:17:33.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 1 -- 172.21.15.50:0/2855547927 shutdown_connections 2023-10-30T12:17:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 1 --2- 172.21.15.50:0/2855547927 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e480a23a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 1 -- 172.21.15.50:0/2855547927 >> 172.21.15.50:0/2855547927 conn(0x7f3e480a6200 msgr2=0x7f3e480a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 1 -- 172.21.15.50:0/2855547927 shutdown_connections 2023-10-30T12:17:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.983+0000 7f3e4efd0640 1 -- 172.21.15.50:0/2855547927 wait complete. 2023-10-30T12:17:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 10 monclient: build_initial_monmap 2023-10-30T12:17:33.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 10 monclient: monmap: 2023-10-30T12:17:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:33.010 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:33.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:33.011 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 1 librados: starting msgr at 2023-10-30T12:17:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 1 librados: starting objecter 2023-10-30T12:17:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 1 Processor -- start 2023-10-30T12:17:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 1 -- start start 2023-10-30T12:17:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 1 librados: setting wanted keys 2023-10-30T12:17:33.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 1 librados: calling monclient init 2023-10-30T12:17:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.984+0000 7f3e4efd0640 10 monclient: init 2023-10-30T12:17:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:17:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4806afc0 0x7f3e481ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4806afc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e481ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4806b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4813b070 0x7f3e4811c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): picked mon.noname-a con 0x7f3e4813b070 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4811eae0 con 0x7f3e4813b070 2023-10-30T12:17:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48138490 con 0x7f3e4806afc0 2023-10-30T12:17:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4811fba0 con 0x7f3e4806b640 2023-10-30T12:17:33.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4efd0640 10 monclient(hunting): authenticate will time out at 1898.168475s 2023-10-30T12:17:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4813b070 0x7f3e4811c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e481ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4813b070 0x7f3e4811c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51584/0 (socket says 172.21.15.50:51584) 2023-10-30T12:17:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e481ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52262/0 (socket says 172.21.15.50:52262) 2023-10-30T12:17:33.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 1 -- 172.21.15.50:0/1816169259 learned_addr learned my addr 172.21.15.50:0/1816169259 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 10 monclient(hunting): get_auth_request con 0x7f3e4813b070 auth_method 0 2023-10-30T12:17:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): get_auth_request con 0x7f3e4806b640 auth_method 0 2023-10-30T12:17:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4806afc0 0x7f3e481ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 10 monclient(hunting): get_auth_request con 0x7f3e4806afc0 auth_method 0 2023-10-30T12:17:33.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient(hunting): handle_auth_done global_id 224129 payload 824 2023-10-30T12:17:33.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4806afc0 msgr2=0x7f3e481ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4806afc0 0x7f3e481ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4813b070 msgr2=0x7f3e4811c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4813b070 0x7f3e4811c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:33.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 monclient: found mon.noname-c 2023-10-30T12:17:33.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:33.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e47fff640 1 -- 172.21.15.50:0/1816169259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e34009400 con 0x7f3e4806b640 2023-10-30T12:17:33.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.985+0000 7f3e4d56d640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4813b070 0x7f3e4811c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:33.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e47fff640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e481ca9b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e4806a4f0 tx=0x7f3e340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 client.?.objecter ms_handle_connect 0x7f3e4806b640 2023-10-30T12:17:33.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:33.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34032950 con 0x7f3e4806b640 2023-10-30T12:17:33.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:33.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: dump: 2023-10-30T12:17:33.024 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.024 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.024 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.025 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.025 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.025 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.025 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.025 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.026 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.026 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: _finish_auth 0 2023-10-30T12:17:33.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: _check_auth_tickets 2023-10-30T12:17:33.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:33.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e34032ab0 con 0x7f3e4806b640 2023-10-30T12:17:33.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 5 monclient: authenticate success, global_id 224129 2023-10-30T12:17:33.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 10 monclient: _renew_subs 2023-10-30T12:17:33.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4811fe80 con 0x7f3e4806b640 2023-10-30T12:17:33.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34032e60 con 0x7f3e4806b640 2023-10-30T12:17:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e457fa640 10 monclient: dump: 2023-10-30T12:17:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.030 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e46ffd640 4 set_mon_vals no callback set 2023-10-30T12:17:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 10 client.224129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 10 monclient: _renew_subs 2023-10-30T12:17:33.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e48070de0 con 0x7f3e4806b640 2023-10-30T12:17:33.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.986+0000 7f3e4efd0640 1 librados: init done 2023-10-30T12:17:33.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4efd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:33.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3e34039040 con 0x7f3e4806b640 2023-10-30T12:17:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4efd0640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d0d0 0x7f3e4802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4efd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48120220 con 0x7f3e4811d0d0 2023-10-30T12:17:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d0d0 0x7f3e4802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient: get_auth_request con 0x7f3e4811d0d0 auth_method 0 2023-10-30T12:17:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e2c0392f0 0x7f3e2c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f3e3406c540 con 0x7f3e4806b640 2023-10-30T12:17:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 10 client.224129.objecter ms_dispatch 0x7f3e48000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 3 client.224129.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 3 client.224129.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_done global_id 224132 payload 824 2023-10-30T12:17:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 1 -- 172.21.15.50:0/1816169259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e4809f0b0 con 0x7f3e4811d0d0 2023-10-30T12:17:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 20 client.224129.objecter dump_active .. 0 homeless 2023-10-30T12:17:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 10 client.224129.objecter ms_handle_connect 0x7f3e4811d0d0 2023-10-30T12:17:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 10 client.224129.objecter resend_mon_ops 2023-10-30T12:17:33.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4cd6c640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d0d0 0x7f3e4802b430 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c00fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4d56d640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e2c0392f0 0x7f3e2c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c004a40 con 0x7f3e4811d0d0 2023-10-30T12:17:33.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:33.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e457fa640 10 monclient: dump: 2023-10-30T12:17:33.037 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.037 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.037 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.037 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.037 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.038 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.987+0000 7f3e4d56d640 10 monclient: get_auth_request con 0x7f3e2c0392f0 auth_method 0 2023-10-30T12:17:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f3e3c003940 con 0x7f3e4811d0d0 2023-10-30T12:17:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d0d0 msgr2=0x7f3e4802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d0d0 0x7f3e4802b430 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c00fc60 comp rx=0 tx=0).stop 2023-10-30T12:17:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d0d0 msgr2=0x7f3e4802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d0d0 0x7f3e4802b430 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e457fa640 10 client.224129.objecter ms_handle_connect 0x7f3e2c0392f0 2023-10-30T12:17:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.988+0000 7f3e4d56d640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e2c0392f0 0x7f3e2c03b7b0 secure :-1 s=READY pgs=4251 cs=0 l=1 rev1=1 crypto rx=0x7f3e48134340 tx=0x7f3e40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4efd0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:33.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4efd0640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d4b0 0x7f3e4802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4efd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48073ff0 con 0x7f3e4811d4b0 2023-10-30T12:17:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d4b0 0x7f3e4802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient: get_auth_request con 0x7f3e4811d4b0 auth_method 0 2023-10-30T12:17:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 10 monclient(hunting): handle_auth_done global_id 224135 payload 824 2023-10-30T12:17:33.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 1 -- 172.21.15.50:0/1816169259 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3e4809df80 con 0x7f3e4811d4b0 2023-10-30T12:17:33.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e457fa640 10 client.224129.objecter ms_handle_connect 0x7f3e4811d4b0 2023-10-30T12:17:33.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e457fa640 10 client.224129.objecter resend_mon_ops 2023-10-30T12:17:33.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.994+0000 7f3e4cd6c640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d4b0 0x7f3e4802b970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c023070 con 0x7f3e4811d4b0 2023-10-30T12:17:33.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:33.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 10 monclient: dump: 2023-10-30T12:17:33.047 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.047 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.047 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.048 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.048 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.048 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.048 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.048 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.049 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.049 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f3e3c004560 con 0x7f3e4811d4b0 2023-10-30T12:17:33.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:33.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:33.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d4b0 msgr2=0x7f3e4802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d4b0 0x7f3e4802b970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c003760 comp rx=0 tx=0).stop 2023-10-30T12:17:33.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d4b0 msgr2=0x7f3e4802b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:33.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e457fa640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4811d4b0 0x7f3e4802b970 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00f7a0 tx=0x7f3e3c003760 comp rx=0 tx=0).stop 2023-10-30T12:17:33.050 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:17:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 176, 2023-10-30T12:17:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:17:33.053 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:33.053 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:33.053 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:33.053 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:33.053 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:33.054 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:33.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:33.054 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:33.054 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:33.054 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:33.054 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:33.055 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:33.055 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:33.055 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.055 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:33.055 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:33.056 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:33.056 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:33.056 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:33.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:33.056 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:33.057 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:33.057 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:33.057 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:33.057 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:33.057 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:33.057 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.058 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:33.058 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:33.058 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:33.058 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:33.059 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:33.059 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:33.059 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:33.059 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:33.059 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:33.060 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:33.060 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:33.060 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:33.060 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:33.060 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:33.060 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:33.061 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:33.061 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:33.061 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:33.061 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:33.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:33.074 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:33.074 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:33.077 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.078 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:17:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.080 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:33.080 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:33.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 10 librados: watch_flush enter 2023-10-30T12:17:33.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 10 librados: watch_flush exit 2023-10-30T12:17:33.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 20 client.224129.objecter shutdown clearing up homeless session... 2023-10-30T12:17:33.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 10 client.224129.objecter successfully canceled tick 2023-10-30T12:17:33.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e2c0392f0 msgr2=0x7f3e2c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e2c0392f0 0x7f3e2c03b7b0 secure :-1 s=READY pgs=4251 cs=0 l=1 rev1=1 crypto rx=0x7f3e48134340 tx=0x7f3e40009040 comp rx=0 tx=0).stop 2023-10-30T12:17:33.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 10 monclient: shutdown 2023-10-30T12:17:33.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:33.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 msgr2=0x7f3e481ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e481ca9b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e4806a4f0 tx=0x7f3e340046c0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4d56d640 1 -- 172.21.15.50:0/1816169259 reap_dead start 2023-10-30T12:17:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 shutdown_connections 2023-10-30T12:17:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 --2- 172.21.15.50:0/1816169259 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e4806b640 0x7f3e481ca9b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 >> 172.21.15.50:0/1816169259 conn(0x7f3e480a6200 msgr2=0x7f3e48070630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 shutdown_connections 2023-10-30T12:17:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 -- 172.21.15.50:0/1816169259 wait complete. 2023-10-30T12:17:33.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:32.995+0000 7f3e4efd0640 1 librados: shutdown 2023-10-30T12:17:33.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:17:33.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:17:33.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:17:33.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.192+0000 7f946eb00640 10 monclient: get_monmap_and_config 2023-10-30T12:17:33.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.192+0000 7f946eb00640 10 monclient: build_initial_monmap 2023-10-30T12:17:33.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.192+0000 7f946eb00640 10 monclient: monmap: 2023-10-30T12:17:33.206 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:33.206 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.207 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:33.193393+0000 2023-10-30T12:17:33.207 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:33.193393+0000 2023-10-30T12:17:33.207 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:33.207 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.207 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:33.207 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:33.208 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:33.208 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:33.208 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:33.208 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 1 Processor -- start 2023-10-30T12:17:33.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 1 -- start start 2023-10-30T12:17:33.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient: init 2023-10-30T12:17:33.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:33.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:17:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f946806afc0 0x7f94680a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient(hunting): picked mon.noname-a con 0x7f946806afc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f946806b640 0x7f94680a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient(hunting): picked mon.noname-c con 0x7f946806b640 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94680a2710 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:33.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946806bde0 con 0x7f94680a2710 2023-10-30T12:17:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946806b3a0 con 0x7f946806afc0 2023-10-30T12:17:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946809e540 con 0x7f946806b640 2023-10-30T12:17:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946eb00640 10 monclient(hunting): authenticate will time out at 1898.376546s 2023-10-30T12:17:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f946806afc0 0x7f94680a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f946806afc0 0x7f94680a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51596/0 (socket says 172.21.15.50:51596) 2023-10-30T12:17:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 1 -- 172.21.15.50:0/1907586045 learned_addr learned my addr 172.21.15.50:0/1907586045 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:33.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): get_auth_request con 0x7f946806afc0 auth_method 0 2023-10-30T12:17:33.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 1 --2- 172.21.15.50:0/1907586045 >> v2:172.21.15.118:3300/0 conn(0x7f946806b640 0x7f94680a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 10 monclient(hunting): get_auth_request con 0x7f946806b640 auth_method 0 2023-10-30T12:17:33.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.193+0000 7f946c89c640 10 monclient(hunting): handle_auth_done global_id 214287 payload 274 2023-10-30T12:17:33.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 1 -- 172.21.15.50:0/1907586045 >> v2:172.21.15.118:3300/0 conn(0x7f946806b640 msgr2=0x7f94680a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:33.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 1 --2- 172.21.15.50:0/1907586045 >> v2:172.21.15.118:3300/0 conn(0x7f946806b640 0x7f94680a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 1 -- 172.21.15.50:0/1907586045 >> v1:172.21.15.118:6789/0 conn(0x7f94680a2710 legacy=0x7f946813c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:33.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 1 monclient: found mon.noname-a 2023-10-30T12:17:33.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:33.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 1 -- 172.21.15.50:0/1907586045 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946809fa10 con 0x7f946806afc0 2023-10-30T12:17:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946c89c640 1 --2- 172.21.15.50:0/1907586045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94680a1c90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f945c009870 tx=0x7f945c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd5218165c466f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 1 -- 172.21.15.50:0/1907586045 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c0317e0 con 0x7f946806afc0 2023-10-30T12:17:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: dump: 2023-10-30T12:17:33.218 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.218 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.218 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.219 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.219 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.219 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.220 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.220 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.220 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.221 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:33.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:33.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:33.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946eb00640 5 monclient: authenticate success, global_id 214287 2023-10-30T12:17:33.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 1 -- 172.21.15.50:0/1907586045 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c031940 con 0x7f946806afc0 2023-10-30T12:17:33.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946eb00640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:33.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:33.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946eb00640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:33.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 1 -- 172.21.15.50:0/1907586045 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c031cd0 con 0x7f946806afc0 2023-10-30T12:17:33.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f946eb00640 4 set_mon_vals no callback set 2023-10-30T12:17:33.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:33.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.194+0000 7f94677fe640 10 monclient: dump: 2023-10-30T12:17:33.225 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.226 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.226 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.227 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.227 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.227 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.228 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.228 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.228 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.228 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 10 monclient: shutdown 2023-10-30T12:17:33.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:33.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 -- 172.21.15.50:0/1907586045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 msgr2=0x7f94680a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 --2- 172.21.15.50:0/1907586045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94680a1c90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f945c009870 tx=0x7f945c02b880 comp rx=0 tx=0).stop 2023-10-30T12:17:33.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 -- 172.21.15.50:0/1907586045 shutdown_connections 2023-10-30T12:17:33.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 --2- 172.21.15.50:0/1907586045 >> v2:172.21.15.118:3300/0 conn(0x7f946806b640 0x7f94680a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 --2- 172.21.15.50:0/1907586045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94680a1c90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 -- 172.21.15.50:0/1907586045 >> 172.21.15.50:0/1907586045 conn(0x7f94680a6200 msgr2=0x7f94680a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:33.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 -- 172.21.15.50:0/1907586045 shutdown_connections 2023-10-30T12:17:33.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.195+0000 7f946eb00640 1 -- 172.21.15.50:0/1907586045 wait complete. 2023-10-30T12:17:33.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 10 monclient: build_initial_monmap 2023-10-30T12:17:33.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 10 monclient: monmap: 2023-10-30T12:17:33.231 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:33.231 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:33.231 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:33.231 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:33.231 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:33.232 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:33.232 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 1 librados: starting msgr at 2023-10-30T12:17:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 1 librados: starting objecter 2023-10-30T12:17:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 1 Processor -- start 2023-10-30T12:17:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 1 -- start start 2023-10-30T12:17:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 1 librados: setting wanted keys 2023-10-30T12:17:33.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 1 librados: calling monclient init 2023-10-30T12:17:33.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 10 monclient: init 2023-10-30T12:17:33.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.196+0000 7f946eb00640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:33.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:33.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94681c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): picked mon.noname-a con 0x7f946806afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:33.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f946806b640 0x7f94681c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): picked mon.noname-b con 0x7f946806b640 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:33.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f94680a2710 0x7f94681c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): picked mon.noname-c con 0x7f94680a2710 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:33.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681c7470 con 0x7f946806afc0 2023-10-30T12:17:33.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681a8b30 con 0x7f946806b640 2023-10-30T12:17:33.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94680706c0 con 0x7f94680a2710 2023-10-30T12:17:33.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:33.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient(hunting): authenticate will time out at 1898.380079s 2023-10-30T12:17:33.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94681c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94681c3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51602/0 (socket says 172.21.15.50:51602) 2023-10-30T12:17:33.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 -- 172.21.15.50:0/1317142393 learned_addr learned my addr 172.21.15.50:0/1317142393 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:33.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): get_auth_request con 0x7f946806afc0 auth_method 0 2023-10-30T12:17:33.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f94680a2710 0x7f94681c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 10 monclient(hunting): get_auth_request con 0x7f94680a2710 auth_method 0 2023-10-30T12:17:33.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946d09d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient(hunting): handle_auth_done global_id 214290 payload 824 2023-10-30T12:17:33.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f9467fff640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f946806b640 0x7f94681c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f94680a2710 msgr2=0x7f94681c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f94680a2710 0x7f94681c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f946806b640 msgr2=0x7f94681c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f946806b640 0x7f94681c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:33.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 monclient: found mon.noname-a 2023-10-30T12:17:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:17:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 -- 172.21.15.50:0/1317142393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0094d0 con 0x7f946806afc0 2023-10-30T12:17:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946c89c640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94681c3d70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f945c000c00 tx=0x7f945c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 client.?.objecter ms_handle_connect 0x7f946806afc0 2023-10-30T12:17:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c032880 con 0x7f946806afc0 2023-10-30T12:17:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:17:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: dump: 2023-10-30T12:17:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.245 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.246 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: _finish_auth 0 2023-10-30T12:17:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: _check_auth_tickets 2023-10-30T12:17:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c0329e0 con 0x7f946806afc0 2023-10-30T12:17:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 5 monclient: authenticate success, global_id 214290 2023-10-30T12:17:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient: _renew_subs 2023-10-30T12:17:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94681aae30 con 0x7f946806afc0 2023-10-30T12:17:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:33.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c032d90 con 0x7f946806afc0 2023-10-30T12:17:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:17:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f94657fa640 10 monclient: dump: 2023-10-30T12:17:33.249 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.252 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f9466ffd640 4 set_mon_vals no callback set 2023-10-30T12:17:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 client.214290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:33.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient: _renew_subs 2023-10-30T12:17:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:17:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94681ab3f0 con 0x7f946806afc0 2023-10-30T12:17:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.197+0000 7f946eb00640 1 librados: init done 2023-10-30T12:17:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f946eb00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f946eb00640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681c9f30 0x7f94681ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f946eb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468073ff0 con 0x7f94681c9f30 2023-10-30T12:17:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f945c04e450 con 0x7f946806afc0 2023-10-30T12:17:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f94657fa640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f944c0392f0 0x7f944c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f945c06ce40 con 0x7f946806afc0 2023-10-30T12:17:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f94657fa640 10 client.214290.objecter ms_dispatch 0x7f9468000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f94657fa640 3 client.214290.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.198+0000 7f94657fa640 3 client.214290.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681c9f30 0x7f94681ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient: get_auth_request con 0x7f94681c9f30 auth_method 0 2023-10-30T12:17:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 20 client.214290.objecter dump_active .. 0 homeless 2023-10-30T12:17:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f946d09d640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f944c0392f0 0x7f944c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f946d09d640 10 monclient: get_auth_request con 0x7f944c0392f0 auth_method 0 2023-10-30T12:17:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 10 monclient(hunting): handle_auth_done global_id 204376 payload 824 2023-10-30T12:17:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 1 -- 172.21.15.50:0/1317142393 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f946809df80 con 0x7f94681c9f30 2023-10-30T12:17:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 10 client.214290.objecter ms_handle_connect 0x7f94681c9f30 2023-10-30T12:17:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 10 client.214290.objecter resend_mon_ops 2023-10-30T12:17:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f9467fff640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681c9f30 0x7f94681ca310 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9454010720 tx=0x7f9454010be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 10 client.214290.objecter ms_handle_connect 0x7f944c0392f0 2023-10-30T12:17:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f946d09d640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f944c0392f0 0x7f944c03b7b0 secure :-1 s=READY pgs=4252 cs=0 l=1 rev1=1 crypto rx=0x7f94600012e0 tx=0x7f9460009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945400e9c0 con 0x7f94681c9f30 2023-10-30T12:17:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.199+0000 7f94657fa640 10 monclient: dump: 2023-10-30T12:17:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.264 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.265 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.200+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f945400eb20 con 0x7f94681c9f30 2023-10-30T12:17:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.200+0000 7f94657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.200+0000 7f94657fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.200+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681c9f30 msgr2=0x7f94681ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.200+0000 7f94657fa640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681c9f30 0x7f94681ca310 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9454010720 tx=0x7f9454010be0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.200+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681c9f30 msgr2=0x7f94681ca310 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.200+0000 7f94657fa640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681c9f30 0x7f94681ca310 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9454010720 tx=0x7f9454010be0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f946eb00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f946eb00640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681ca850 0x7f94681a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f946eb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681aafc0 con 0x7f94681ca850 2023-10-30T12:17:33.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681ca850 0x7f94681a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient: get_auth_request con 0x7f94681ca850 auth_method 0 2023-10-30T12:17:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 10 monclient(hunting): handle_auth_done global_id 204379 payload 824 2023-10-30T12:17:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.206+0000 7f9467fff640 1 -- 172.21.15.50:0/1317142393 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f946806a6f0 con 0x7f94681ca850 2023-10-30T12:17:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f9467fff640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681ca850 0x7f94681a3bd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9454010720 tx=0x7f9454004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 10 client.214290.objecter ms_handle_connect 0x7f94681ca850 2023-10-30T12:17:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 10 client.214290.objecter resend_mon_ops 2023-10-30T12:17:33.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454004050 con 0x7f94681ca850 2023-10-30T12:17:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 10 monclient: dump: 2023-10-30T12:17:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.273 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.275 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f94540041b0 con 0x7f94681ca850 2023-10-30T12:17:33.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681ca850 msgr2=0x7f94681a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681ca850 0x7f94681a3bd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9454010720 tx=0x7f9454004730 comp rx=0 tx=0).stop 2023-10-30T12:17:33.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681ca850 msgr2=0x7f94681a3bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:33.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f94657fa640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94681ca850 0x7f94681a3bd0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9454010720 tx=0x7f9454004730 comp rx=0 tx=0).stop 2023-10-30T12:17:33.277 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:33.277 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:33.277 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:33.278 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:17:33.278 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 176, 2023-10-30T12:17:33.278 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:33.278 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:17:33.278 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:33.279 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:33.279 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.279 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:17:33.280 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:33.280 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:33.280 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:33.280 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:33.280 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:33.280 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:33.281 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:33.281 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:33.281 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:33.281 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:33.281 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:33.292 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:33.292 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:33.292 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:33.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.292 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:33.292 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:33.293 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.293 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:33.293 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:33.294 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.294 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:33.295 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:33.295 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:33.295 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:33.295 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:33.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.296 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:33.296 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:33.296 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:33.296 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:33.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.297 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:33.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:33.297 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.298 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:33.298 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:33.298 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.299 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.299 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:33.299 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:33.299 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:33.300 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:33.300 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:33.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:33.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:33.302 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:33.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:33.303 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:33.303 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.303 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:33.304 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:33.304 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:33.304 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:33.304 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:33.305 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.305 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.305 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:33.305 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:33.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.306 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:33.306 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:33.306 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:33.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.307 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:33.307 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:33.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:33.307 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:33.308 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:33.308 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:17:33.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:33.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:33.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:33.308 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:33.309 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:33.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 10 librados: watch_flush enter 2023-10-30T12:17:33.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 10 librados: watch_flush exit 2023-10-30T12:17:33.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 20 client.214290.objecter shutdown clearing up homeless session... 2023-10-30T12:17:33.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 10 client.214290.objecter successfully canceled tick 2023-10-30T12:17:33.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f944c0392f0 msgr2=0x7f944c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f944c0392f0 0x7f944c03b7b0 secure :-1 s=READY pgs=4252 cs=0 l=1 rev1=1 crypto rx=0x7f94600012e0 tx=0x7f9460009040 comp rx=0 tx=0).stop 2023-10-30T12:17:33.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 10 monclient: shutdown 2023-10-30T12:17:33.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:33.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 msgr2=0x7f94681c3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 --2- 172.21.15.50:0/1317142393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f946806afc0 0x7f94681c3d70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f945c000c00 tx=0x7f945c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946d09d640 1 -- 172.21.15.50:0/1317142393 reap_dead start 2023-10-30T12:17:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946d09d640 1 -- 172.21.15.50:0/1317142393 reap_dead start 2023-10-30T12:17:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 shutdown_connections 2023-10-30T12:17:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 >> 172.21.15.50:0/1317142393 conn(0x7f94680a6200 msgr2=0x7f946806e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:33.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 shutdown_connections 2023-10-30T12:17:33.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 -- 172.21.15.50:0/1317142393 wait complete. 2023-10-30T12:17:33.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.207+0000 7f946eb00640 1 librados: shutdown 2023-10-30T12:17:33.314 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:17:33.315 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:17:33.315 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:17:33.315 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:17:33.315 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:17:33.316 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:17:33.316 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:17:33.329 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:17:33.329 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:17:33.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:17:33.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient: get_monmap_and_config 2023-10-30T12:17:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient: build_initial_monmap 2023-10-30T12:17:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient: monmap: 2023-10-30T12:17:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:33.474648+0000 2023-10-30T12:17:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:33.474648+0000 2023-10-30T12:17:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:33.477 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 1 Processor -- start 2023-10-30T12:17:33.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 1 -- start start 2023-10-30T12:17:33.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient: init 2023-10-30T12:17:33.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:33.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-10-30T12:17:33.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4dc40a2dd0 0x7f4dc40a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc40a2dd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:33.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4dc406a8e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:33.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4dc4071400 0x7f4dc413cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc4071400 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40a31b0 con 0x7f4dc406a8e0 2023-10-30T12:17:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc406a6a0 con 0x7f4dc4071400 2023-10-30T12:17:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc406be90 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dcbb00640 10 monclient(hunting): authenticate will time out at 1898.657809s 2023-10-30T12:17:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dca09d640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f4dc4071400 msgr2=0x7f4dc413cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.474+0000 7f4dca09d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4dc4071400 0x7f4dc413cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4dc40a2dd0 0x7f4dc40a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4dc40a2dd0 0x7f4dc40a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:53328/0 (socket says 172.21.15.50:53328) 2023-10-30T12:17:33.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 -- 172.21.15.50:0/2714859619 learned_addr learned my addr 172.21.15.50:0/2714859619 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:33.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): get_auth_request con 0x7f4dc40a2dd0 auth_method 0 2023-10-30T12:17:33.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient(hunting): handle_auth_done global_id 204382 payload 274 2023-10-30T12:17:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 -- 172.21.15.50:0/2714859619 >> v2:172.21.15.50:3300/0 conn(0x7f4dc4071400 msgr2=0x7f4dc413cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 --2- 172.21.15.50:0/2714859619 >> v2:172.21.15.50:3300/0 conn(0x7f4dc4071400 0x7f4dc413cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 -- 172.21.15.50:0/2714859619 >> v1:172.21.15.118:6789/0 conn(0x7f4dc406a8e0 legacy=0x7f4dc40a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 monclient: found mon.noname-c 2023-10-30T12:17:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:17:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.475+0000 7f4dc989c640 1 -- 172.21.15.50:0/2714859619 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc409e190 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc989c640 1 --2- 172.21.15.50:0/2714859619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc40a2dd0 0x7f4dc40a46c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4dc406af70 tx=0x7f4dc002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=323f95a78ff6afc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 1 -- 172.21.15.50:0/2714859619 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0031800 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: dump: 2023-10-30T12:17:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.497 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: _finish_auth 0 2023-10-30T12:17:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: _check_auth_tickets 2023-10-30T12:17:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 1 -- 172.21.15.50:0/2714859619 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc0031960 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 1 -- 172.21.15.50:0/2714859619 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0031cf0 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 5 monclient: authenticate success, global_id 204382 2023-10-30T12:17:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 4 set_mon_vals no callback set 2023-10-30T12:17:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dc889a640 10 monclient: dump: 2023-10-30T12:17:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.501 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 10 monclient: shutdown 2023-10-30T12:17:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 1 -- 172.21.15.50:0/2714859619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc40a2dd0 msgr2=0x7f4dc40a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/2714859619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc40a2dd0 0x7f4dc40a46c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4dc406af70 tx=0x7f4dc002b8a0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 1 -- 172.21.15.50:0/2714859619 shutdown_connections 2023-10-30T12:17:33.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/2714859619 >> v2:172.21.15.50:3300/0 conn(0x7f4dc4071400 0x7f4dc413cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/2714859619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc40a2dd0 0x7f4dc40a46c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.476+0000 7f4dcbb00640 1 -- 172.21.15.50:0/2714859619 >> 172.21.15.50:0/2714859619 conn(0x7f4dc40a22b0 msgr2=0x7f4dc40a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:33.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.477+0000 7f4dcbb00640 1 -- 172.21.15.50:0/2714859619 shutdown_connections 2023-10-30T12:17:33.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.477+0000 7f4dcbb00640 1 -- 172.21.15.50:0/2714859619 wait complete. 2023-10-30T12:17:33.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.477+0000 7f4dcbb00640 10 monclient: build_initial_monmap 2023-10-30T12:17:33.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.477+0000 7f4dcbb00640 10 monclient: monmap: 2023-10-30T12:17:33.503 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:33.503 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:33.503 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:33.505 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.477+0000 7f4dcbb00640 1 librados: starting msgr at 2023-10-30T12:17:33.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.477+0000 7f4dcbb00640 1 librados: starting objecter 2023-10-30T12:17:33.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.477+0000 7f4dcbb00640 1 Processor -- start 2023-10-30T12:17:33.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 -- start start 2023-10-30T12:17:33.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 librados: setting wanted keys 2023-10-30T12:17:33.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 librados: calling monclient init 2023-10-30T12:17:33.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient: init 2023-10-30T12:17:33.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:33.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:17:33.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 0x7f4dc41ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc4071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 0x7f4dc41bdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc40a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4dc41a7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc41be0c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:33.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc41aaf80 con 0x7f4dc4071400 2023-10-30T12:17:33.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40a1da0 con 0x7f4dc41be0c0 2023-10-30T12:17:33.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc41aaca0 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:33.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dcbb00640 10 monclient(hunting): authenticate will time out at 1898.661465s 2023-10-30T12:17:33.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc989c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 msgr2=0x7f4dc41ca290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:33.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc989c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 0x7f4dc41ca290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:33.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 0x7f4dc41bdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 0x7f4dc41bdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52330/0 (socket says 172.21.15.50:52330) 2023-10-30T12:17:33.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 -- 172.21.15.50:0/3688824912 learned_addr learned my addr 172.21.15.50:0/3688824912 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:33.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): get_auth_request con 0x7f4dc40a2dd0 auth_method 0 2023-10-30T12:17:33.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:33.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:33.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dca09d640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4dc41a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient(hunting): handle_auth_done global_id 224141 payload 824 2023-10-30T12:17:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 msgr2=0x7f4dc41a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4dc41a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 msgr2=0x7f4dc41ca290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 0x7f4dc41ca290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient: _finish_hunting 0 2023-10-30T12:17:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 monclient: found mon.noname-c 2023-10-30T12:17:33.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:33.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dc909b640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc00094d0 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dca09d640 10 monclient: get_auth_request con 0x7f4dc41be0c0 auth_method 0 2023-10-30T12:17:33.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.478+0000 7f4dca09d640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4dc41a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:17:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 0x7f4dc41bdb80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c400 tx=0x7f4db400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 client.?.objecter ms_handle_connect 0x7f4dc40a2dd0 2023-10-30T12:17:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db40052d0 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:33.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: dump: 2023-10-30T12:17:33.515 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.515 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.515 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.515 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.516 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.516 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: _finish_auth 0 2023-10-30T12:17:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:17:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 5 monclient: authenticate success, global_id 224141 2023-10-30T12:17:33.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 10 monclient: _renew_subs 2023-10-30T12:17:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4005430 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc406b0a0 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4003830 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 10 client.224141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dbbfff640 4 set_mon_vals no callback set 2023-10-30T12:17:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: dump: 2023-10-30T12:17:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.521 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.521 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.521 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 10 monclient: _renew_subs 2023-10-30T12:17:33.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc41a9170 con 0x7f4dc40a2dd0 2023-10-30T12:17:33.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dcbb00640 1 librados: init done 2023-10-30T12:17:33.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc909b640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 msgr2=0x7f4dc41bdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:17:33.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc909b640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 msgr2=0x7f4dc41bdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:17:33.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 0x7f4dc41bdb80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c400 tx=0x7f4db400c8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:33.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 0x7f4dc41bdb80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c400 tx=0x7f4db400c8c0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 msgr2=0x7f4dc41bdb80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dc40a2dd0 0x7f4dc41bdb80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4da80051d0 0x7f4da80075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient(hunting): picked mon.c con 0x7f4da80051d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 0x7f4da800a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient(hunting): picked mon.a con 0x7f4dc4071400 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4da800cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient(hunting): picked mon.b con 0x7f4dc41be0c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da800d460 con 0x7f4dc4071400 2023-10-30T12:17:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da800d5a0 con 0x7f4dc41be0c0 2023-10-30T12:17:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da800d710 con 0x7f4da80051d0 2023-10-30T12:17:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc909b640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 msgr2=0x7f4da800a0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 0x7f4da800a0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dba7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc989c640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4da80051d0 0x7f4da80075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc989c640 10 monclient(hunting): get_auth_request con 0x7f4da80051d0 auth_method 0 2023-10-30T12:17:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc989c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc989c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.479+0000 7f4dc989c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4da800cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 10 monclient(hunting): get_auth_request con 0x7f4dc41be0c0 auth_method 0 2023-10-30T12:17:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:33.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:33.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:33.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dca09d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:33.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient(hunting): handle_auth_done global_id 224141 payload 293 2023-10-30T12:17:33.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 msgr2=0x7f4da800cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4da800cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 msgr2=0x7f4da800a0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 0x7f4da800a0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient: _finish_hunting 0 2023-10-30T12:17:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 1 monclient: found mon.c 2023-10-30T12:17:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f4da800da00 con 0x7f4da80051d0 2023-10-30T12:17:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient: _finish_auth 0 2023-10-30T12:17:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 10 monclient: _check_auth_tickets 2023-10-30T12:17:33.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:33.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dcbb00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:33.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dcbb00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:33.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dcbb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc41ab150 con 0x7f4da80051d0 2023-10-30T12:17:33.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 10 client.224141.objecter ms_handle_connect 0x7f4da80051d0 2023-10-30T12:17:33.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 10 client.224141.objecter resend_mon_ops 2023-10-30T12:17:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dc989c640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4da80051d0 0x7f4da80075c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0033000 tx=0x7f4db4007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0031ad0 con 0x7f4da80051d0 2023-10-30T12:17:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 10 monclient: dump: 2023-10-30T12:17:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:33.534 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:33.534 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.534 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:33.534 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:33.534 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:33.535 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc0031e90 con 0x7f4da80051d0 2023-10-30T12:17:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dba7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.480+0000 7f4dbb7fe640 4 set_mon_vals no callback set 2023-10-30T12:17:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4dc003a420 con 0x7f4da80051d0 2023-10-30T12:17:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dba7fc640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4dc40a2dd0 0x7f4da80439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 4 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f4dc002e070 con 0x7f4da80051d0 2023-10-30T12:17:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dba7fc640 10 client.224141.objecter ms_dispatch 0x7f4dc4000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:33.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dba7fc640 3 client.224141.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:33.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dba7fc640 3 client.224141.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:33.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dba7fc640 20 client.224141.objecter dump_active .. 0 homeless 2023-10-30T12:17:33.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4dc40a2dd0 0x7f4da80439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:33.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.481+0000 7f4dc909b640 10 monclient: get_auth_request con 0x7f4dc40a2dd0 auth_method 0 2023-10-30T12:17:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.482+0000 7f4dba7fc640 10 client.224141.objecter ms_handle_connect 0x7f4dc40a2dd0 2023-10-30T12:17:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.482+0000 7f4dc909b640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4dc40a2dd0 0x7f4da80439b0 secure :-1 s=READY pgs=4253 cs=0 l=1 rev1=1 crypto rx=0x7f4dc41a8d70 tx=0x7f4db4009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.485+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4dc002fe40 con 0x7f4da80051d0 2023-10-30T12:17:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.485+0000 7f4dba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.485+0000 7f4dba7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:33.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.726+0000 7f4dcbb00640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:17:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.726+0000 7f4dcbb00640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.726+0000 7f4dcbb00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.726+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4dc41ab650 con 0x7f4da80051d0 2023-10-30T12:17:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.726+0000 7f4dba7fc640 1 -- 172.21.15.50:0/3688824912 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f4dc002fa50 con 0x7f4da80051d0 2023-10-30T12:17:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.726+0000 7f4dba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.726+0000 7f4dba7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:33.729 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:17:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 10 librados: watch_flush enter 2023-10-30T12:17:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 10 librados: watch_flush exit 2023-10-30T12:17:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 20 client.224141.objecter shutdown clearing up homeless session... 2023-10-30T12:17:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 10 client.224141.objecter successfully canceled tick 2023-10-30T12:17:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4dc40a2dd0 msgr2=0x7f4da80439b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4dc40a2dd0 0x7f4da80439b0 secure :-1 s=READY pgs=4253 cs=0 l=1 rev1=1 crypto rx=0x7f4dc41a8d70 tx=0x7f4db4009030 comp rx=0 tx=0).stop 2023-10-30T12:17:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 10 monclient: shutdown 2023-10-30T12:17:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4da80051d0 msgr2=0x7f4da80075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:33.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4da80051d0 0x7f4da80075c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0033000 tx=0x7f4db4007ec0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 shutdown_connections 2023-10-30T12:17:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4dc40a2dd0 0x7f4da80439b0 unknown :-1 s=CLOSED pgs=4253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4dc41be0c0 0x7f4da800cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dc4071400 0x7f4da800a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 --2- 172.21.15.50:0/3688824912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4da80051d0 0x7f4da80075c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 >> 172.21.15.50:0/3688824912 conn(0x7f4dc40a22b0 msgr2=0x7f4dc406fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 shutdown_connections 2023-10-30T12:17:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 -- 172.21.15.50:0/3688824912 wait complete. 2023-10-30T12:17:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:33.729+0000 7f4dcbb00640 1 librados: shutdown 2023-10-30T12:17:36.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:17:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.857+0000 7f2ffa54c640 10 monclient: get_monmap_and_config 2023-10-30T12:17:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.857+0000 7f2ffa54c640 10 monclient: build_initial_monmap 2023-10-30T12:17:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.857+0000 7f2ffa54c640 10 monclient: monmap: 2023-10-30T12:17:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:36.858288+0000 2023-10-30T12:17:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:36.858288+0000 2023-10-30T12:17:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:36.861 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:36.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.857+0000 7f2ffa54c640 1 Processor -- start 2023-10-30T12:17:36.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 1 -- start start 2023-10-30T12:17:36.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient: init 2023-10-30T12:17:36.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:36.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T12:17:36.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ff40a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ff4099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2ff4132db0 0x7f2ff413e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff4132db0 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40a4f40 con 0x7f2ff4099a40 2023-10-30T12:17:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40a4150 con 0x7f2ff40a4b60 2023-10-30T12:17:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:36.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff40a3d70 con 0x7f2ff4132db0 2023-10-30T12:17:36.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:36.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ffa54c640 10 monclient(hunting): authenticate will time out at 1902.041536s 2023-10-30T12:17:36.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ff37fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f2ff4099a40 legacy=0x7f2ff413aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:17:36.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ff8ae9640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f2ff4132db0 msgr2=0x7f2ff413e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:36.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ff8ae9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2ff4132db0 0x7f2ff413e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:36.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ff3fff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f2ff40a4b60 0x7f2ff41326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:55072/0 (socket says 172.21.15.50:55072) 2023-10-30T12:17:36.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.858+0000 7f2ff3fff640 1 -- 172.21.15.50:0/1889172390 learned_addr learned my addr 172.21.15.50:0/1889172390 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:17:36.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008780874 0 0) 0x7f2ff40a4150 con 0x7f2ff40a4b60 2023-10-30T12:17:36.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:17:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): my global_id is 204388 2023-10-30T12:17:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fdc003520 con 0x7f2ff40a4b60 2023-10-30T12:17:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019801983 0 0) 0x7f2fdc003520 con 0x7f2ff40a4b60 2023-10-30T12:17:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 >> v2:172.21.15.50:3300/0 conn(0x7f2ff4132db0 msgr2=0x7f2ff413e1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:36.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 --2- 172.21.15.50:0/1889172390 >> v2:172.21.15.50:3300/0 conn(0x7f2ff4132db0 0x7f2ff413e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:36.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 >> v1:172.21.15.50:6789/0 conn(0x7f2ff4099a40 legacy=0x7f2ff413aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:36.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:36.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:17:36.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:36.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:17:36.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff409a440 con 0x7f2ff40a4b60 2023-10-30T12:17:36.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:17:36.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:36.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f2fe4001670 con 0x7f2ff40a4b60 2023-10-30T12:17:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:17:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: dump: 2023-10-30T12:17:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:36.880 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:17:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 >> v1:172.21.15.118:6789/0 conn(0x7f2ff40a4b60 legacy=0x7f2ff41326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fdc003970 0x7f2fdc003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): picked mon.c con 0x7f2fdc003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc0047c0 0x7f2fdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): picked mon.a con 0x7f2fdc0047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fdc005380 0x7f2fdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): picked mon.b con 0x7f2fdc005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00c1f0 con 0x7f2fdc0047c0 2023-10-30T12:17:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00c360 con 0x7f2fdc005380 2023-10-30T12:17:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc00c520 con 0x7f2fdc003970 2023-10-30T12:17:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff2ffd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:17:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff37fe640 1 -- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc0047c0 msgr2=0x7f2fdc004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff37fe640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc0047c0 0x7f2fdc004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fdc003970 0x7f2fdc003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): get_auth_request con 0x7f2fdc003970 auth_method 0 2023-10-30T12:17:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fdc005380 0x7f2fdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 10 monclient(hunting): get_auth_request con 0x7f2fdc005380 auth_method 0 2023-10-30T12:17:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_done global_id 204388 payload 293 2023-10-30T12:17:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 1 -- 172.21.15.50:0/1889172390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fdc005380 msgr2=0x7f2fdc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2fdc005380 0x7f2fdc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 1 -- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc0047c0 msgr2=0x7f2fdc004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc0047c0 0x7f2fdc004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:17:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 1 monclient: found mon.c 2023-10-30T12:17:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 1 -- 172.21.15.50:0/1889172390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc003520 con 0x7f2fdc003970 2023-10-30T12:17:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient: _finish_auth 0 2023-10-30T12:17:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:17:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ff8ae9640 1 -- 172.21.15.50:0/1889172390 reap_dead start 2023-10-30T12:17:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ffa54c640 5 monclient: authenticate success, global_id 204388 2023-10-30T12:17:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.859+0000 7f2ffa54c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff3fff640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fdc003970 0x7f2fdc003d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c190 tx=0x7f2fdc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ec2590c97c0286f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4005ad0 con 0x7f2fdc003970 2023-10-30T12:17:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 10 monclient: dump: 2023-10-30T12:17:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:36.894 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe4005c30 con 0x7f2fdc003970 2023-10-30T12:17:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ffa54c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 1 -- 172.21.15.50:0/1889172390 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe402c890 con 0x7f2fdc003970 2023-10-30T12:17:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ffa54c640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ffa54c640 4 set_mon_vals no callback set 2023-10-30T12:17:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.860+0000 7f2ff2ffd640 10 monclient: dump: 2023-10-30T12:17:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:36.898 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 10 monclient: shutdown 2023-10-30T12:17:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 1 -- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fdc003970 msgr2=0x7f2fdc003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fdc003970 0x7f2fdc003d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c190 tx=0x7f2fdc0057f0 comp rx=0 tx=0).stop 2023-10-30T12:17:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 1 -- 172.21.15.50:0/1889172390 shutdown_connections 2023-10-30T12:17:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/1889172390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2fdc003970 0x7f2fdc003d70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 1 -- 172.21.15.50:0/1889172390 >> 172.21.15.50:0/1889172390 conn(0x7f2ff40a8060 msgr2=0x7f2ff40a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 1 -- 172.21.15.50:0/1889172390 shutdown_connections 2023-10-30T12:17:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.861+0000 7f2ffa54c640 1 -- 172.21.15.50:0/1889172390 wait complete. 2023-10-30T12:17:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 10 monclient: build_initial_monmap 2023-10-30T12:17:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 10 monclient: monmap: 2023-10-30T12:17:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:36.902 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 1 librados: starting msgr at 2023-10-30T12:17:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 1 librados: starting objecter 2023-10-30T12:17:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 1 Processor -- start 2023-10-30T12:17:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 1 -- start start 2023-10-30T12:17:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 1 librados: setting wanted keys 2023-10-30T12:17:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 1 librados: calling monclient init 2023-10-30T12:17:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.862+0000 7f2ffa54c640 10 monclient: init 2023-10-30T12:17:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc003970 0x7f2ff41ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): picked mon.noname-a con 0x7f2fdc003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ff41315c0 0x7f2ff4131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff41315c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ff409e460 0x7f2ff409e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff409e460 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff411faf0 con 0x7f2fdc003970 2023-10-30T12:17:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff411e420 con 0x7f2ff41315c0 2023-10-30T12:17:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4131f80 con 0x7f2ff409e460 2023-10-30T12:17:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff3fff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc003970 msgr2=0x7f2ff41ca040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ffa54c640 10 monclient(hunting): authenticate will time out at 1902.046208s 2023-10-30T12:17:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc003970 0x7f2ff41ca040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ff409e460 0x7f2ff409e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ff409e460 0x7f2ff409e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52356/0 (socket says 172.21.15.50:52356) 2023-10-30T12:17:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 -- 172.21.15.50:0/3446800981 learned_addr learned my addr 172.21.15.50:0/3446800981 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): get_auth_request con 0x7f2ff409e460 auth_method 0 2023-10-30T12:17:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ff41315c0 0x7f2ff4131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 10 monclient(hunting): get_auth_request con 0x7f2ff41315c0 auth_method 0 2023-10-30T12:17:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient(hunting): handle_auth_done global_id 224144 payload 824 2023-10-30T12:17:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 -- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ff41315c0 msgr2=0x7f2ff4131a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ff41315c0 0x7f2ff4131a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 -- 172.21.15.50:0/3446800981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc003970 msgr2=0x7f2ff41ca040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc003970 0x7f2ff41ca040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient: _finish_hunting 0 2023-10-30T12:17:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 monclient: found mon.noname-c 2023-10-30T12:17:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 -- 172.21.15.50:0/3446800981 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff409f930 con 0x7f2ff409e460 2023-10-30T12:17:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff8ae9640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ff409e460 0x7f2ff409e910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40a5300 tx=0x7f2fec00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ff409e460 2023-10-30T12:17:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff0ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff0ff9640 1 -- 172.21.15.50:0/3446800981 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00cd80 con 0x7f2ff409e460 2023-10-30T12:17:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.863+0000 7f2ff0ff9640 10 monclient: dump: 2023-10-30T12:17:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:36.915 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 10 monclient: _finish_auth 0 2023-10-30T12:17:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:17:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 1 -- 172.21.15.50:0/3446800981 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec017440 con 0x7f2ff409e460 2023-10-30T12:17:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 5 monclient: authenticate success, global_id 224144 2023-10-30T12:17:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 10 monclient: _renew_subs 2023-10-30T12:17:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff40a18d0 con 0x7f2ff409e460 2023-10-30T12:17:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 1 -- 172.21.15.50:0/3446800981 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00cee0 con 0x7f2ff409e460 2023-10-30T12:17:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 10 monclient: dump: 2023-10-30T12:17:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:36.919 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff27fc640 4 set_mon_vals no callback set 2023-10-30T12:17:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 10 client.224144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 10 monclient: _renew_subs 2023-10-30T12:17:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff40a0100 con 0x7f2ff409e460 2023-10-30T12:17:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ffa54c640 1 librados: init done 2023-10-30T12:17:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.864+0000 7f2ff0ff9640 1 -- 172.21.15.50:0/3446800981 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2fec0177d0 con 0x7f2ff409e460 2023-10-30T12:17:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ffa54c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ffa54c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ffa54c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff411fc30 con 0x7f2ff409e460 2023-10-30T12:17:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff0ff9640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fe00392d0 0x7f2fe003b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff0ff9640 1 -- 172.21.15.50:0/3446800981 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f2fec04a9a0 con 0x7f2ff409e460 2023-10-30T12:17:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff0ff9640 10 client.224144.objecter ms_dispatch 0x7f2ff4000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff0ff9640 3 client.224144.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff0ff9640 3 client.224144.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff0ff9640 20 client.224144.objecter dump_active .. 0 homeless 2023-10-30T12:17:36.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff3fff640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fe00392d0 0x7f2fe003b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:36.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff3fff640 10 monclient: get_auth_request con 0x7f2fe00392d0 auth_method 0 2023-10-30T12:17:36.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff0ff9640 10 client.224144.objecter ms_handle_connect 0x7f2fe00392d0 2023-10-30T12:17:36.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.865+0000 7f2ff3fff640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fe00392d0 0x7f2fe003b790 secure :-1 s=READY pgs=4254 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4034000 tx=0x7f2fe4007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:36.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.869+0000 7f2ff0ff9640 1 -- 172.21.15.50:0/3446800981 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ff411fc30 con 0x7f2ff409e460 2023-10-30T12:17:36.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.869+0000 7f2ff0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:36.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:36.869+0000 7f2ff0ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.114+0000 7f2ffa54c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:17:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.114+0000 7f2ffa54c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.114+0000 7f2ffa54c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.114+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2ff411fc30 con 0x7f2ff409e460 2023-10-30T12:17:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.115+0000 7f2ff0ff9640 1 -- 172.21.15.50:0/3446800981 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f2ff411fc30 con 0x7f2ff409e460 2023-10-30T12:17:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.115+0000 7f2ff0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.115+0000 7f2ff0ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:37.118 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:17:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.116+0000 7f2ffa54c640 10 librados: watch_flush enter 2023-10-30T12:17:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 10 librados: watch_flush exit 2023-10-30T12:17:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 20 client.224144.objecter shutdown clearing up homeless session... 2023-10-30T12:17:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 10 client.224144.objecter successfully canceled tick 2023-10-30T12:17:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fe00392d0 msgr2=0x7f2fe003b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fe00392d0 0x7f2fe003b790 secure :-1 s=READY pgs=4254 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4034000 tx=0x7f2fe4007000 comp rx=0 tx=0).stop 2023-10-30T12:17:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 10 monclient: shutdown 2023-10-30T12:17:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ff409e460 msgr2=0x7f2ff409e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ff409e460 0x7f2ff409e910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40a5300 tx=0x7f2fec00ee30 comp rx=0 tx=0).stop 2023-10-30T12:17:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 shutdown_connections 2023-10-30T12:17:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2fe00392d0 0x7f2fe003b790 unknown :-1 s=CLOSED pgs=4254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ff409e460 0x7f2ff409e910 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ff41315c0 0x7f2ff4131a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 --2- 172.21.15.50:0/3446800981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fdc003970 0x7f2ff41ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 >> 172.21.15.50:0/3446800981 conn(0x7f2ff40a8060 msgr2=0x7f2ff40a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 shutdown_connections 2023-10-30T12:17:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 -- 172.21.15.50:0/3446800981 wait complete. 2023-10-30T12:17:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:37.117+0000 7f2ffa54c640 1 librados: shutdown 2023-10-30T12:17:40.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:17:40.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.245+0000 7f5ea167c640 10 monclient: get_monmap_and_config 2023-10-30T12:17:40.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.245+0000 7f5ea167c640 10 monclient: build_initial_monmap 2023-10-30T12:17:40.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.245+0000 7f5ea167c640 10 monclient: monmap: 2023-10-30T12:17:40.247 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:40.247 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:40.248 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:40.246349+0000 2023-10-30T12:17:40.248 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:40.246349+0000 2023-10-30T12:17:40.248 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:40.248 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:40.248 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:40.248 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:40.249 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:40.249 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:40.249 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:40.249 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:40.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.245+0000 7f5ea167c640 1 Processor -- start 2023-10-30T12:17:40.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 1 -- start start 2023-10-30T12:17:40.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient: init 2023-10-30T12:17:40.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:40.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T12:17:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e9c098520 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e9c098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 0x7f5e9c13ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient(hunting): picked mon.noname-a con 0x7f5e9c138990 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c09c680 con 0x7f5e9c098900 2023-10-30T12:17:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c099930 con 0x7f5e9c098520 2023-10-30T12:17:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9c09aca0 con 0x7f5e9c138990 2023-10-30T12:17:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5ea167c640 10 monclient(hunting): authenticate will time out at 1905.429487s 2023-10-30T12:17:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5e9a7fc640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f5e9c098900 legacy=0x7f5e9c098d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:17:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5e9b7fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 msgr2=0x7f5e9c13ad80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5e9b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 0x7f5e9c13ad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5e9affd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f5e9c098520 0x7f5e9c09bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:36508/0 (socket says 172.21.15.50:36508) 2023-10-30T12:17:40.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.246+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 learned_addr learned my addr 172.21.15.50:0/1963492745 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:17:40.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.447+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 msgr2=0x7f5e9c13ad80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:40.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.447+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v1:172.21.15.50:6789/0 conn(0x7f5e9c098900 legacy=0x7f5e9c098d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:17:40.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.447+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 0x7f5e9c13ad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:40.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.848+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 msgr2=0x7f5e9c13ad80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:40.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.848+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v1:172.21.15.50:6789/0 conn(0x7f5e9c098900 legacy=0x7f5e9c098d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:17:40.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:40.848+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 0x7f5e9c13ad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:41.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:41.649+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 msgr2=0x7f5e9c13ad80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:41.649+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v1:172.21.15.50:6789/0 conn(0x7f5e9c098900 legacy=0x7f5e9c098d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:17:41.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:41.649+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 0x7f5e9c13ad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:43.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): tick 2023-10-30T12:17:43.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:17:43.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:17:43.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:17:43.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 msgr2=0x7f5e9c13ad80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:43.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e9c138990 0x7f5e9c13ad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:43.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 >> v1:172.21.15.118:6789/0 conn(0x7f5e9c098520 legacy=0x7f5e9c09bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:43.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 >> v1:172.21.15.50:6789/0 conn(0x7f5e9c098900 legacy=0x7f5e9c098d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:43.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient: _add_conns ranks=[4,2] 2023-10-30T12:17:43.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:43.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e880036c0 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:43.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:43.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f5e880061d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:43.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:43.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9c09aca0 con 0x7f5e880036c0 2023-10-30T12:17:43.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:43.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88008da0 con 0x7f5e880061d0 2023-10-30T12:17:43.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e98ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:43.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e880061d0 auth_method 0 2023-10-30T12:17:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.246+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:43.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 10 monclient(hunting): get_auth_request con 0x7f5e880036c0 auth_method 0 2023-10-30T12:17:43.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:43.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:43.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:43.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:43.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:43.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:43.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:43.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.247+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:43.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:43.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e880061d0 auth_method 0 2023-10-30T12:17:43.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:43.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:43.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:43.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:43.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:43.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:43.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:43.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:43.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 10 monclient(hunting): get_auth_request con 0x7f5e880036c0 auth_method 0 2023-10-30T12:17:43.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:43.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:43.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:43.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:43.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:43.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:43.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.448+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:43.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:43.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e880061d0 auth_method 0 2023-10-30T12:17:43.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:43.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:43.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:43.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:43.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:43.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:43.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:43.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:43.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 10 monclient(hunting): get_auth_request con 0x7f5e880036c0 auth_method 0 2023-10-30T12:17:43.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:43.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:43.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:43.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:43.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:43.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:43.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:43.849+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e880061d0 auth_method 0 2023-10-30T12:17:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:17:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:44.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:44.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:44.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9affd640 10 monclient(hunting): get_auth_request con 0x7f5e880036c0 auth_method 0 2023-10-30T12:17:44.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:44.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:44.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.650+0000 7f5e9affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:44.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.651+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:44.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.651+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:44.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.651+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:44.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:44.651+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:46.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): tick 2023-10-30T12:17:46.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:17:46.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:17:46.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:17:46.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 msgr2=0x7f5e88008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:46.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e880061d0 0x7f5e88008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:46.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 msgr2=0x7f5e88005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:46.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e880036c0 0x7f5e88005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:46.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-10-30T12:17:46.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 reap_dead start 2023-10-30T12:17:46.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e9c098520 addr v1:172.21.15.118:6789/0 2023-10-30T12:17:46.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:46.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e88003b30 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:46.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 0x7f5e88007070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:46.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f5e88006bc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:46.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c09af80 con 0x7f5e9c098520 2023-10-30T12:17:46.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:46.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9c09aca0 con 0x7f5e88006bc0 2023-10-30T12:17:46.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:46.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 1 -- 172.21.15.50:0/1963492745 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e88008da0 con 0x7f5e88003b30 2023-10-30T12:17:46.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e98ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:46.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 msgr2=0x7f5e88007070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:46.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 0x7f5e88007070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:46.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:46.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e88003b30 auth_method 0 2023-10-30T12:17:46.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:46.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:46.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:46.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:46.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:46.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:46.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.247+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:46.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 msgr2=0x7f5e88007070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:46.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 0x7f5e88007070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:46.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:46.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e88003b30 auth_method 0 2023-10-30T12:17:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:17:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:46.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:46.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.448+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:17:46.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 msgr2=0x7f5e88007070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:46.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 0x7f5e88007070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:46.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:46.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e88003b30 auth_method 0 2023-10-30T12:17:46.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:46.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:46.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:46.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:17:46.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:46.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:46.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:46.849+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:17:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 msgr2=0x7f5e88007070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 0x7f5e88007070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:47.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e88003b30 auth_method 0 2023-10-30T12:17:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:17:47.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:17:47.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:17:47.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:47.650+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:17:48.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.196+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207986247 0 0) 0x7f5e9c09af80 con 0x7f5e9c098520 2023-10-30T12:17:48.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.196+0000 7f5e99ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:17:48.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.196+0000 7f5e99ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.196+0000 7f5e99ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.196+0000 7f5e99ffb640 10 monclient(hunting): my global_id is 204394 2023-10-30T12:17:48.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.196+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e84003520 con 0x7f5e9c098520 2023-10-30T12:17:48.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575634710 0 0) 0x7f5e84003520 con 0x7f5e9c098520 2023-10-30T12:17:48.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 msgr2=0x7f5e88006680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:48.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.118:3300/0 conn(0x7f5e88003b30 0x7f5e88006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 msgr2=0x7f5e88007070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 --2- 172.21.15.50:0/1963492745 >> v2:172.21.15.50:3300/0 conn(0x7f5e88006bc0 0x7f5e88007070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: _finish_hunting 0 2023-10-30T12:17:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:17:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:17:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c09c680 con 0x7f5e9c098520 2023-10-30T12:17:48.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:48.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:48.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:48.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f5e8c001610 con 0x7f5e9c098520 2023-10-30T12:17:48.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5ea167c640 5 monclient: authenticate success, global_id 204394 2023-10-30T12:17:48.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5ea167c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:48.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:17:48.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: dump: 2023-10-30T12:17:48.205 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.205 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.206 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.206 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.206 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.206 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.206 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.207 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.207 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.207 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:17:48.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:48.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 >> v1:172.21.15.118:6789/0 conn(0x7f5e9c098520 legacy=0x7f5e8800d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:17:48.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e84003970 0x7f5e84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.197+0000 7f5e99ffb640 10 monclient(hunting): picked mon.c con 0x7f5e84003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:48.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e840047c0 0x7f5e84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 10 monclient(hunting): picked mon.b con 0x7f5e840047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:48.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840051d0 0x7f5e8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 10 monclient(hunting): picked mon.a con 0x7f5e840051d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:48.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c09aca0 con 0x7f5e840051d0 2023-10-30T12:17:48.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88008da0 con 0x7f5e840047c0 2023-10-30T12:17:48.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8400c1a0 con 0x7f5e84003970 2023-10-30T12:17:48.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:48.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e99ffb640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:17:48.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5ea167c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:48.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840051d0 msgr2=0x7f5e8400b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:48.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840051d0 0x7f5e8400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:48.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e84003970 0x7f5e84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): get_auth_request con 0x7f5e84003970 auth_method 0 2023-10-30T12:17:48.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:48.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e840047c0 0x7f5e84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e840047c0 auth_method 0 2023-10-30T12:17:48.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:17:48.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:48.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:17:48.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient(hunting): handle_auth_done global_id 204394 payload 293 2023-10-30T12:17:48.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e840047c0 msgr2=0x7f5e84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:17:48.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e840047c0 0x7f5e84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840051d0 msgr2=0x7f5e8400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:48.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840051d0 0x7f5e8400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:48.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 1 monclient: found mon.c 2023-10-30T12:17:48.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:48.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 1 -- 172.21.15.50:0/1963492745 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e84003520 con 0x7f5e84003970 2023-10-30T12:17:48.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient: _finish_auth 0 2023-10-30T12:17:48.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:48.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:48.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.198+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1963492745 reap_dead start 2023-10-30T12:17:48.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.199+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e84003970 0x7f5e84003d90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c007d70 tx=0x7f5e840057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d83f35fcdae98070 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c032430 con 0x7f5e84003970 2023-10-30T12:17:48.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:48.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 10 monclient: dump: 2023-10-30T12:17:48.218 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.218 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.218 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.218 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.218 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.218 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.219 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.219 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.219 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.219 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c032590 con 0x7f5e84003970 2023-10-30T12:17:48.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:48.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 1 -- 172.21.15.50:0/1963492745 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c032940 con 0x7f5e84003970 2023-10-30T12:17:48.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:48.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5e99ffb640 10 monclient: dump: 2023-10-30T12:17:48.220 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.221 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.221 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.221 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.221 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.221 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.222 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.222 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.222 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.222 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5ea167c640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:48.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.200+0000 7f5ea167c640 4 set_mon_vals no callback set 2023-10-30T12:17:48.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 10 monclient: shutdown 2023-10-30T12:17:48.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:48.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 1 -- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e84003970 msgr2=0x7f5e84003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e84003970 0x7f5e84003d90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c007d70 tx=0x7f5e840057f0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 1 -- 172.21.15.50:0/1963492745 shutdown_connections 2023-10-30T12:17:48.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1963492745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e84003970 0x7f5e84003d90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 1 -- 172.21.15.50:0/1963492745 >> 172.21.15.50:0/1963492745 conn(0x7f5e9c0a05c0 msgr2=0x7f5e9c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:48.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 1 -- 172.21.15.50:0/1963492745 shutdown_connections 2023-10-30T12:17:48.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.201+0000 7f5ea167c640 1 -- 172.21.15.50:0/1963492745 wait complete. 2023-10-30T12:17:48.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.202+0000 7f5ea167c640 10 monclient: build_initial_monmap 2023-10-30T12:17:48.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.202+0000 7f5ea167c640 10 monclient: monmap: 2023-10-30T12:17:48.229 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:48.230 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:48.230 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:48.230 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:48.230 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:48.230 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.231 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:48.231 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:48.231 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:48.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.202+0000 7f5ea167c640 1 librados: starting msgr at 2023-10-30T12:17:48.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.202+0000 7f5ea167c640 1 librados: starting objecter 2023-10-30T12:17:48.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 Processor -- start 2023-10-30T12:17:48.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 -- start start 2023-10-30T12:17:48.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 librados: setting wanted keys 2023-10-30T12:17:48.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 librados: calling monclient init 2023-10-30T12:17:48.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient: init 2023-10-30T12:17:48.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:48.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:17:48.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e84003970 0x7f5e9c070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): picked mon.noname-a con 0x7f5e84003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:48.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e9c08c9e0 0x7f5e9c0965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): picked mon.noname-b con 0x7f5e9c08c9e0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:48.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9c096b30 0x7f5e9c1c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): picked mon.noname-c con 0x7f5e9c096b30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:48.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c08e050 con 0x7f5e84003970 2023-10-30T12:17:48.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c071060 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c08d7b0 con 0x7f5e9c096b30 2023-10-30T12:17:48.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:48.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5ea167c640 10 monclient(hunting): authenticate will time out at 1913.386788s 2023-10-30T12:17:48.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9affd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e84003970 msgr2=0x7f5e9c070380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:48.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e84003970 0x7f5e9c070380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:48.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9c096b30 0x7f5e9c1c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9c096b30 0x7f5e9c1c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56150/0 (socket says 172.21.15.50:56150) 2023-10-30T12:17:48.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9a7fc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e9c08c9e0 0x7f5e9c0965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9b7fe640 1 -- 172.21.15.50:0/1984485273 learned_addr learned my addr 172.21.15.50:0/1984485273 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:48.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e9c096b30 auth_method 0 2023-10-30T12:17:48.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.203+0000 7f5e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e9c08c9e0 auth_method 0 2023-10-30T12:17:48.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient(hunting): handle_auth_done global_id 204397 payload 824 2023-10-30T12:17:48.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1984485273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9c096b30 msgr2=0x7f5e9c1c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9c096b30 0x7f5e9c1c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1984485273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e84003970 msgr2=0x7f5e9c070380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:48.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e84003970 0x7f5e9c070380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:17:48.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 1 monclient: found mon.noname-b 2023-10-30T12:17:48.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:48.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:17:48.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 1 -- 172.21.15.50:0/1984485273 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c08da90 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9b7fe640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9c096b30 0x7f5e9c1c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:17:48.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e9a7fc640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e9c08c9e0 0x7f5e9c0965f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e90003980 tx=0x7f5e90019b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 client.?.objecter ms_handle_connect 0x7f5e9c08c9e0 2023-10-30T12:17:48.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:48.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 1 -- 172.21.15.50:0/1984485273 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9001e050 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:48.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: dump: 2023-10-30T12:17:48.243 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.243 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.243 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.243 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.244 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.244 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.244 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.244 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.244 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.244 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:48.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:48.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:48.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 1 -- 172.21.15.50:0/1984485273 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9001f040 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:48.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 1 -- 172.21.15.50:0/1984485273 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90019e80 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:48.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e837fe640 10 monclient: dump: 2023-10-30T12:17:48.246 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.246 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.247 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.247 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.247 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.247 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.247 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.248 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.248 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.248 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5ea167c640 5 monclient: authenticate success, global_id 204397 2023-10-30T12:17:48.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5ea167c640 10 monclient: _renew_subs 2023-10-30T12:17:48.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5ea167c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:17:48.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5e997fa640 4 set_mon_vals no callback set 2023-10-30T12:17:48.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.204+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c08dd70 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.205+0000 7f5ea167c640 10 client.204397.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:48.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.205+0000 7f5ea167c640 10 monclient: _renew_subs 2023-10-30T12:17:48.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.205+0000 7f5ea167c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:17:48.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.205+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c1c3180 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.205+0000 7f5ea167c640 1 librados: init done 2023-10-30T12:17:48.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e837fe640 1 -- 172.21.15.50:0/1984485273 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5e90024460 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5ea167c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:48.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5ea167c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:48.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5ea167c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:17:48.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c09cf40 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e837fe640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5e8803c920 0x7f5e8803ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e837fe640 1 -- 172.21.15.50:0/1984485273 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(521..521 src has 490..521) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f5e9004c4b0 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e837fe640 10 client.204397.objecter ms_dispatch 0x7f5e9c000b90 osd_map(521..521 src has 490..521) v4 2023-10-30T12:17:48.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e837fe640 3 client.204397.objecter handle_osd_map got epochs [521,521] > 0 2023-10-30T12:17:48.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e837fe640 3 client.204397.objecter handle_osd_map decoding full epoch 521 2023-10-30T12:17:48.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e837fe640 20 client.204397.objecter dump_active .. 0 homeless 2023-10-30T12:17:48.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5e8803c920 0x7f5e8803ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.206+0000 7f5e9affd640 10 monclient: get_auth_request con 0x7f5e8803c920 auth_method 0 2023-10-30T12:17:48.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.207+0000 7f5e837fe640 10 client.204397.objecter ms_handle_connect 0x7f5e8803c920 2023-10-30T12:17:48.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.207+0000 7f5e9affd640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5e8803c920 0x7f5e8803ede0 secure :-1 s=READY pgs=4255 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c038000 tx=0x7f5e8c031000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.212+0000 7f5e837fe640 1 -- 172.21.15.50:0/1984485273 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5e90022360 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.212+0000 7f5e837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:48.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.212+0000 7f5e837fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:48.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.465+0000 7f5ea167c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:17:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.465+0000 7f5ea167c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.465+0000 7f5ea167c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:17:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.465+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e9c08d8f0 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.466+0000 7f5e837fe640 1 -- 172.21.15.50:0/1984485273 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f5e9c08d8f0 con 0x7f5e9c08c9e0 2023-10-30T12:17:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.466+0000 7f5e837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:17:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.466+0000 7f5e837fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:48.469 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":178,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:17:48.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 10 librados: watch_flush enter 2023-10-30T12:17:48.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 10 librados: watch_flush exit 2023-10-30T12:17:48.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 20 client.204397.objecter shutdown clearing up homeless session... 2023-10-30T12:17:48.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 10 client.204397.objecter successfully canceled tick 2023-10-30T12:17:48.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5e8803c920 msgr2=0x7f5e8803ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5e8803c920 0x7f5e8803ede0 secure :-1 s=READY pgs=4255 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c038000 tx=0x7f5e8c031000 comp rx=0 tx=0).stop 2023-10-30T12:17:48.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 10 monclient: shutdown 2023-10-30T12:17:48.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:48.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e9c08c9e0 msgr2=0x7f5e9c0965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e9c08c9e0 0x7f5e9c0965f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e90003980 tx=0x7f5e90019b70 comp rx=0 tx=0).stop 2023-10-30T12:17:48.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 shutdown_connections 2023-10-30T12:17:48.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5e8803c920 0x7f5e8803ede0 unknown :-1 s=CLOSED pgs=4255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9c096b30 0x7f5e9c1c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e9c08c9e0 0x7f5e9c0965f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 --2- 172.21.15.50:0/1984485273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e84003970 0x7f5e9c070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 >> 172.21.15.50:0/1984485273 conn(0x7f5e9c0a05c0 msgr2=0x7f5e9c1379b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:48.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 shutdown_connections 2023-10-30T12:17:48.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 -- 172.21.15.50:0/1984485273 wait complete. 2023-10-30T12:17:48.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.467+0000 7f5ea167c640 1 librados: shutdown 2023-10-30T12:17:48.480 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:17:48.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:17:48.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:17:48.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.585+0000 7f5386516640 10 monclient: get_monmap_and_config 2023-10-30T12:17:48.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.585+0000 7f5386516640 10 monclient: build_initial_monmap 2023-10-30T12:17:48.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.585+0000 7f5386516640 10 monclient: monmap: 2023-10-30T12:17:48.587 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:48.587 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.587 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:48.586444+0000 2023-10-30T12:17:48.588 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:48.586444+0000 2023-10-30T12:17:48.588 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:48.588 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.588 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:48.588 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:48.589 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:48.589 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:48.589 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:48.589 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 1 Processor -- start 2023-10-30T12:17:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 1 -- start start 2023-10-30T12:17:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient: init 2023-10-30T12:17:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-10-30T12:17:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5380097780 0x7f5380097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient(hunting): picked mon.noname-a con 0x7f5380097780 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5380073b10 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f538009ccd0 0x7f53800a0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient(hunting): picked mon.noname-c con 0x7f538009ccd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:17:48.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5380071ec0 con 0x7f5380073b10 2023-10-30T12:17:48.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538009bd10 con 0x7f5380097780 2023-10-30T12:17:48.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538009bff0 con 0x7f538009ccd0 2023-10-30T12:17:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5386516640 10 monclient(hunting): authenticate will time out at 1913.769581s 2023-10-30T12:17:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f537ffff640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f5380097780 msgr2=0x7f5380097b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f537f7fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f5380073b10 legacy=0x7f53800a0080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:17:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f537ffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5380097780 0x7f5380097b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5384ab3640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f538009ccd0 0x7f53800a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5384ab3640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f538009ccd0 0x7f53800a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:50148/0 (socket says 172.21.15.50:50148) 2023-10-30T12:17:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5384ab3640 1 -- 172.21.15.50:0/3359988077 learned_addr learned my addr 172.21.15.50:0/3359988077 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5384ab3640 10 monclient(hunting): get_auth_request con 0x7f538009ccd0 auth_method 0 2023-10-30T12:17:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5384ab3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5384ab3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.586+0000 7f5384ab3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 10 monclient(hunting): handle_auth_done global_id 204400 payload 274 2023-10-30T12:17:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 1 -- 172.21.15.50:0/3359988077 >> v2:172.21.15.50:3300/0 conn(0x7f5380097780 msgr2=0x7f5380097b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 1 --2- 172.21.15.50:0/3359988077 >> v2:172.21.15.50:3300/0 conn(0x7f5380097780 0x7f5380097b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 1 -- 172.21.15.50:0/3359988077 >> v1:172.21.15.50:6789/0 conn(0x7f5380073b10 legacy=0x7f53800a0080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 10 monclient: _finish_hunting 0 2023-10-30T12:17:48.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 1 monclient: found mon.noname-c 2023-10-30T12:17:48.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:48.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:17:48.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 1 -- 172.21.15.50:0/3359988077 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53800980a0 con 0x7f538009ccd0 2023-10-30T12:17:48.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.587+0000 7f5384ab3640 1 --2- 172.21.15.50:0/3359988077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f538009ccd0 0x7f53800a0790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f538009e770 tx=0x7f537802bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4b86ab45beca05d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 1 -- 172.21.15.50:0/3359988077 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537802c980 con 0x7f538009ccd0 2023-10-30T12:17:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: dump: 2023-10-30T12:17:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.607 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.608 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.608 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.608 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: _finish_auth 0 2023-10-30T12:17:48.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: _check_auth_tickets 2023-10-30T12:17:48.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:48.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 5 monclient: authenticate success, global_id 204400 2023-10-30T12:17:48.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:48.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 1 -- 172.21.15.50:0/3359988077 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537802cae0 con 0x7f538009ccd0 2023-10-30T12:17:48.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:48.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:48.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 4 set_mon_vals no callback set 2023-10-30T12:17:48.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 1 -- 172.21.15.50:0/3359988077 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537802ce70 con 0x7f538009ccd0 2023-10-30T12:17:48.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:48.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f537effd640 10 monclient: dump: 2023-10-30T12:17:48.611 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.611 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.611 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.611 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.612 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 10 monclient: shutdown 2023-10-30T12:17:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 -- 172.21.15.50:0/3359988077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f538009ccd0 msgr2=0x7f53800a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 --2- 172.21.15.50:0/3359988077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f538009ccd0 0x7f53800a0790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f538009e770 tx=0x7f537802bde0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 -- 172.21.15.50:0/3359988077 shutdown_connections 2023-10-30T12:17:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 --2- 172.21.15.50:0/3359988077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f538009ccd0 0x7f53800a0790 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 --2- 172.21.15.50:0/3359988077 >> v2:172.21.15.50:3300/0 conn(0x7f5380097780 0x7f5380097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 -- 172.21.15.50:0/3359988077 >> 172.21.15.50:0/3359988077 conn(0x7f53800a1170 msgr2=0x7f53800a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 -- 172.21.15.50:0/3359988077 shutdown_connections 2023-10-30T12:17:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.588+0000 7f5386516640 1 -- 172.21.15.50:0/3359988077 wait complete. 2023-10-30T12:17:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 10 monclient: build_initial_monmap 2023-10-30T12:17:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 10 monclient: monmap: 2023-10-30T12:17:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:48.617 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 1 librados: starting msgr at 2023-10-30T12:17:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 1 librados: starting objecter 2023-10-30T12:17:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 1 Processor -- start 2023-10-30T12:17:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 1 -- start start 2023-10-30T12:17:48.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 1 librados: setting wanted keys 2023-10-30T12:17:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 1 librados: calling monclient init 2023-10-30T12:17:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.589+0000 7f5386516640 10 monclient: init 2023-10-30T12:17:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:17:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5380073b10 0x7f538008f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): picked mon.noname-b con 0x7f5380073b10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380097780 0x7f538008fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): picked mon.noname-c con 0x7f5380097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f538009ccd0 0x7f53800901a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): picked mon.noname-a con 0x7f538009ccd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:48.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5380093820 con 0x7f538009ccd0 2023-10-30T12:17:48.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5380091aa0 con 0x7f5380073b10 2023-10-30T12:17:48.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53800a3bc0 con 0x7f5380097780 2023-10-30T12:17:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5386516640 10 monclient(hunting): authenticate will time out at 1913.773235s 2023-10-30T12:17:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5384ab3640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f538009ccd0 msgr2=0x7f53800901a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f5384ab3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f538009ccd0 0x7f53800901a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380097780 0x7f538008fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380097780 0x7f538008fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56162/0 (socket says 172.21.15.50:56162) 2023-10-30T12:17:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 -- 172.21.15.50:0/2585179172 learned_addr learned my addr 172.21.15.50:0/2585179172 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): get_auth_request con 0x7f5380097780 auth_method 0 2023-10-30T12:17:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537ffff640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5380073b10 0x7f538008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537ffff640 10 monclient(hunting): get_auth_request con 0x7f5380073b10 auth_method 0 2023-10-30T12:17:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient(hunting): handle_auth_done global_id 224150 payload 824 2023-10-30T12:17:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5380073b10 msgr2=0x7f538008f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5380073b10 0x7f538008f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f538009ccd0 msgr2=0x7f53800901a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:48.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f538009ccd0 0x7f53800901a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:17:48.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 monclient: found mon.noname-c 2023-10-30T12:17:48.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:48.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 -- 172.21.15.50:0/2585179172 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5378009b30 con 0x7f5380097780 2023-10-30T12:17:48.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537f7fe640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380097780 0x7f538008fc60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f537400b550 tx=0x7f537400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537cff9640 10 client.?.objecter ms_handle_connect 0x7f5380097780 2023-10-30T12:17:48.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537cff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:48.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.590+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374003cf0 con 0x7f5380097780 2023-10-30T12:17:48.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:48.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: dump: 2023-10-30T12:17:48.627 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.628 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.628 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.628 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.628 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.628 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.629 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.629 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.629 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.629 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: _finish_auth 0 2023-10-30T12:17:48.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: _check_auth_tickets 2023-10-30T12:17:48.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:48.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 5 monclient: authenticate success, global_id 224150 2023-10-30T12:17:48.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 10 monclient: _renew_subs 2023-10-30T12:17:48.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374003e70 con 0x7f5380097780 2023-10-30T12:17:48.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5380073fd0 con 0x7f5380097780 2023-10-30T12:17:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374005290 con 0x7f5380097780 2023-10-30T12:17:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 10 monclient: dump: 2023-10-30T12:17:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.632 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.633 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 10 client.224150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 10 monclient: _renew_subs 2023-10-30T12:17:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5380095dc0 con 0x7f5380097780 2023-10-30T12:17:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537e7fc640 4 set_mon_vals no callback set 2023-10-30T12:17:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 1 librados: init done 2023-10-30T12:17:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f53740053f0 con 0x7f5380097780 2023-10-30T12:17:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f538002c5b0 0x7f5380060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538006a5a0 con 0x7f538002c5b0 2023-10-30T12:17:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537ffff640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f538002c5b0 0x7f5380060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.591+0000 7f537ffff640 10 monclient: get_auth_request con 0x7f538002c5b0 auth_method 0 2023-10-30T12:17:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f536c0392a0 0x7f536c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(522..522 src has 490..522) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f537404ac40 con 0x7f5380097780 2023-10-30T12:17:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 client.224150.objecter ms_dispatch 0x7f5380000b90 osd_map(522..522 src has 490..522) v4 2023-10-30T12:17:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 3 client.224150.objecter handle_osd_map got epochs [522,522] > 0 2023-10-30T12:17:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 3 client.224150.objecter handle_osd_map decoding full epoch 522 2023-10-30T12:17:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 10 monclient(hunting): handle_auth_done global_id 224153 payload 824 2023-10-30T12:17:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 1 -- 172.21.15.50:0/2585179172 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53800a0cd0 con 0x7f538002c5b0 2023-10-30T12:17:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 20 client.224150.objecter dump_active .. 0 homeless 2023-10-30T12:17:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f5384ab3640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f536c0392a0 0x7f536c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f5384ab3640 10 monclient: get_auth_request con 0x7f536c0392a0 auth_method 0 2023-10-30T12:17:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 client.224150.objecter ms_handle_connect 0x7f538002c5b0 2023-10-30T12:17:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537ffff640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f538002c5b0 0x7f5380060fc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f537000d700 tx=0x7f537000dbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 client.224150.objecter resend_mon_ops 2023-10-30T12:17:48.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53700039b0 con 0x7f538002c5b0 2023-10-30T12:17:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 monclient: dump: 2023-10-30T12:17:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.642 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 client.224150.objecter ms_handle_connect 0x7f536c0392a0 2023-10-30T12:17:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f5384ab3640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f536c0392a0 0x7f536c03b760 secure :-1 s=READY pgs=4256 cs=0 l=1 rev1=1 crypto rx=0x7f5378035000 tx=0x7f5378030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5370003b10 con 0x7f538002c5b0 2023-10-30T12:17:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f538002c5b0 msgr2=0x7f5380060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f538002c5b0 0x7f5380060fc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f537000d700 tx=0x7f537000dbc0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f538002c5b0 msgr2=0x7f5380060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.592+0000 7f537cff9640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f538002c5b0 0x7f5380060fc0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f537000d700 tx=0x7f537000dbc0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f5386516640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f5386516640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380061500 0x7f53801d46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f5386516640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5380069fd0 con 0x7f5380061500 2023-10-30T12:17:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380061500 0x7f53801d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient: get_auth_request con 0x7f5380061500 auth_method 0 2023-10-30T12:17:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 10 monclient(hunting): handle_auth_done global_id 224156 payload 824 2023-10-30T12:17:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 1 -- 172.21.15.50:0/2585179172 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f538002c990 con 0x7f5380061500 2023-10-30T12:17:48.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537ffff640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380061500 0x7f53801d46b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5370000c00 tx=0x7f53700059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537cff9640 10 client.224150.objecter ms_handle_connect 0x7f5380061500 2023-10-30T12:17:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537cff9640 10 client.224150.objecter resend_mon_ops 2023-10-30T12:17:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5370022070 con 0x7f5380061500 2023-10-30T12:17:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.599+0000 7f537cff9640 10 monclient: dump: 2023-10-30T12:17:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4791 (secure 0 0 0) 0x7f5370005420 con 0x7f5380061500 2023-10-30T12:17:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f537cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f537cff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380061500 msgr2=0x7f53801d46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f537cff9640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380061500 0x7f53801d46b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5370000c00 tx=0x7f53700059d0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f537cff9640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380061500 msgr2=0x7f53801d46b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:48.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f537cff9640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380061500 0x7f53801d46b0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5370000c00 tx=0x7f53700059d0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.653 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:48.653 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:48.654 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:48.654 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:17:48.654 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 178, 2023-10-30T12:17:48.654 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:48.654 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:48.655 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:48.655 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.655 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:17:48.655 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:48.655 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:48.655 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:48.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:48.656 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:48.656 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:48.656 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:48.656 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:48.657 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:48.657 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:48.657 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:48.657 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:48.657 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:48.657 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.658 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:48.658 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:48.658 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:48.658 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:48.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:48.659 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:48.659 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:48.659 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:48.659 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:48.659 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:48.660 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:48.660 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:48.660 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.660 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:48.660 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:48.661 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:48.661 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:48.661 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:48.661 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:48.661 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:48.662 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:48.662 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:48.662 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:48.662 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:48.662 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:48.662 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:48.663 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:48.663 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:48.663 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:48.663 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:48.663 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:48.664 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:48.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:48.664 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:48.664 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:48.664 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:48.664 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:48.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.666 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:48.666 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:48.666 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:48.667 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:48.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.667 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:48.667 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:48.667 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:48.668 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:48.668 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.668 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.669 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.669 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:48.669 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:48.669 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:48.670 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:48.670 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:48.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.670 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:48.670 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:48.671 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:48.671 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:48.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.671 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:48.671 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:48.672 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.672 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:48.672 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:48.672 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.673 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.673 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:48.673 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:48.674 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:48.674 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:48.674 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:48.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.675 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:48.675 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:48.675 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:48.675 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:48.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.675 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:48.676 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:48.676 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.676 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:17:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:17:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:17:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:17:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:48.685 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 10 librados: watch_flush enter 2023-10-30T12:17:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 10 librados: watch_flush exit 2023-10-30T12:17:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 20 client.224150.objecter shutdown clearing up homeless session... 2023-10-30T12:17:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 10 client.224150.objecter successfully canceled tick 2023-10-30T12:17:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f536c0392a0 msgr2=0x7f536c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f536c0392a0 0x7f536c03b760 secure :-1 s=READY pgs=4256 cs=0 l=1 rev1=1 crypto rx=0x7f5378035000 tx=0x7f5378030000 comp rx=0 tx=0).stop 2023-10-30T12:17:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 10 monclient: shutdown 2023-10-30T12:17:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380097780 msgr2=0x7f538008fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380097780 0x7f538008fc60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f537400b550 tx=0x7f537400ba10 comp rx=0 tx=0).stop 2023-10-30T12:17:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5384ab3640 1 -- 172.21.15.50:0/2585179172 reap_dead start 2023-10-30T12:17:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 shutdown_connections 2023-10-30T12:17:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 --2- 172.21.15.50:0/2585179172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5380097780 0x7f538008fc60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 >> 172.21.15.50:0/2585179172 conn(0x7f53800a1170 msgr2=0x7f538009a580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 shutdown_connections 2023-10-30T12:17:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 -- 172.21.15.50:0/2585179172 wait complete. 2023-10-30T12:17:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.600+0000 7f5386516640 1 librados: shutdown 2023-10-30T12:17:48.690 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:17:48.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:17:48.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:17:48.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.794+0000 7fdbfd7a4640 10 monclient: get_monmap_and_config 2023-10-30T12:17:48.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.794+0000 7fdbfd7a4640 10 monclient: build_initial_monmap 2023-10-30T12:17:48.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.794+0000 7fdbfd7a4640 10 monclient: monmap: 2023-10-30T12:17:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:17:48.795954+0000 2023-10-30T12:17:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:17:48.795954+0000 2023-10-30T12:17:48.797 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:17:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:17:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:17:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:17:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:17:48.799 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 1 Processor -- start 2023-10-30T12:17:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 1 -- start start 2023-10-30T12:17:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 10 monclient: init 2023-10-30T12:17:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-10-30T12:17:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 10 monclient(hunting): picked mon.noname-b con 0x7fdbf809e7e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbf809ebc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:17:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdbf8138b60 0x7fdbf813af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 10 monclient(hunting): picked mon.noname-a con 0x7fdbf8138b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:17:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.795+0000 7fdbfd7a4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf809d560 con 0x7fdbf809ebc0 2023-10-30T12:17:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbfd7a4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbfd7a4640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf809fbf0 con 0x7fdbf8138b60 2023-10-30T12:17:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbfd7a4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbfd7a4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf809fed0 con 0x7fdbf809e7e0 2023-10-30T12:17:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbfd7a4640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:48.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf67fc640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fdbf809ebc0 legacy=0x7fdbf809f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:17:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbfd7a4640 10 monclient(hunting): authenticate will time out at 1913.979034s 2023-10-30T12:17:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf77fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fdbf8138b60 msgr2=0x7fdbf813af50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf77fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdbf8138b60 0x7fdbf813af50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf806c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56196/0 (socket says 172.21.15.50:56196) 2023-10-30T12:17:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/2744504366 learned_addr learned my addr 172.21.15.50:0/2744504366 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): get_auth_request con 0x7fdbf809e7e0 auth_method 0 2023-10-30T12:17:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_done global_id 224159 payload 274 2023-10-30T12:17:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/2744504366 >> v2:172.21.15.50:3300/0 conn(0x7fdbf8138b60 msgr2=0x7fdbf813af50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:48.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 --2- 172.21.15.50:0/2744504366 >> v2:172.21.15.50:3300/0 conn(0x7fdbf8138b60 0x7fdbf813af50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/2744504366 >> v1:172.21.15.50:6789/0 conn(0x7fdbf809ebc0 legacy=0x7fdbf809f020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:17:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 monclient: found mon.noname-b 2023-10-30T12:17:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.796+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/2744504366 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80a01b0 con 0x7fdbf809e7e0 2023-10-30T12:17:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf6ffd640 1 --2- 172.21.15.50:0/2744504366 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf806c3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009e50 tx=0x7fdbe802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f425da0bf1937ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 1 -- 172.21.15.50:0/2744504366 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe802c980 con 0x7fdbf809e7e0 2023-10-30T12:17:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:17:48.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: dump: 2023-10-30T12:17:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.815 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.816 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: _finish_auth 0 2023-10-30T12:17:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:17:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 1 -- 172.21.15.50:0/2744504366 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe802cae0 con 0x7fdbf809e7e0 2023-10-30T12:17:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 5 monclient: authenticate success, global_id 224159 2023-10-30T12:17:48.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:17:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 10 monclient: get_monmap_and_config success 2023-10-30T12:17:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 1 -- 172.21.15.50:0/2744504366 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe802ce70 con 0x7fdbf809e7e0 2023-10-30T12:17:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 4 set_mon_vals no callback set 2023-10-30T12:17:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbf5ffb640 10 monclient: dump: 2023-10-30T12:17:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.819 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.820 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.820 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 10 monclient: shutdown 2023-10-30T12:17:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/2744504366 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 msgr2=0x7fdbf806c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/2744504366 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf806c3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009e50 tx=0x7fdbe802bde0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/2744504366 shutdown_connections 2023-10-30T12:17:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/2744504366 >> v2:172.21.15.50:3300/0 conn(0x7fdbf8138b60 0x7fdbf813af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/2744504366 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf806c3c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/2744504366 >> 172.21.15.50:0/2744504366 conn(0x7fdbf80a4800 msgr2=0x7fdbf80a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/2744504366 shutdown_connections 2023-10-30T12:17:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.797+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/2744504366 wait complete. 2023-10-30T12:17:48.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.798+0000 7fdbfd7a4640 10 monclient: build_initial_monmap 2023-10-30T12:17:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.798+0000 7fdbfd7a4640 10 monclient: monmap: 2023-10-30T12:17:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:17:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:17:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:17:48.823 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:17:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:17:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:17:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:17:48.824 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:17:48.824 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.798+0000 7fdbfd7a4640 1 librados: starting msgr at 2023-10-30T12:17:48.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.798+0000 7fdbfd7a4640 1 librados: starting objecter 2023-10-30T12:17:48.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.798+0000 7fdbfd7a4640 1 Processor -- start 2023-10-30T12:17:48.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 -- start start 2023-10-30T12:17:48.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 librados: setting wanted keys 2023-10-30T12:17:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 librados: calling monclient init 2023-10-30T12:17:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient: init 2023-10-30T12:17:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient: _reopen_session rank -1 2023-10-30T12:17:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:17:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf81ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): picked mon.noname-c con 0x7fdbf809e7e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:17:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbf809ebc0 0x7fdbf8132a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): picked mon.noname-a con 0x7fdbf809ebc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:17:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf8138b60 0x7fdbf8132f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): picked mon.noname-b con 0x7fdbf8138b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:17:48.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8120b50 con 0x7fdbf809ebc0 2023-10-30T12:17:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8133480 con 0x7fdbf8138b60 2023-10-30T12:17:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf81cb120 con 0x7fdbf809e7e0 2023-10-30T12:17:48.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): _renew_subs 2023-10-30T12:17:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbfd7a4640 10 monclient(hunting): authenticate will time out at 1913.982502s 2023-10-30T12:17:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf67fc640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbf809ebc0 msgr2=0x7fdbf8132a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:17:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf67fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbf809ebc0 0x7fdbf8132a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:17:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf81ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf81ca4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56204/0 (socket says 172.21.15.50:56204) 2023-10-30T12:17:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/3012974456 learned_addr learned my addr 172.21.15.50:0/3012974456 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:17:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): get_auth_request con 0x7fdbf809e7e0 auth_method 0 2023-10-30T12:17:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient(hunting): handle_auth_done global_id 224162 payload 824 2023-10-30T12:17:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf8138b60 msgr2=0x7fdbf8132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf8138b60 0x7fdbf8132f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbf809ebc0 msgr2=0x7fdbf8132a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:17:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbf809ebc0 0x7fdbf8132a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:17:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 monclient: found mon.noname-c 2023-10-30T12:17:48.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:17:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.799+0000 7fdbf6ffd640 1 -- 172.21.15.50:0/3012974456 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8009b30 con 0x7fdbf809e7e0 2023-10-30T12:17:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbf6ffd640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf81ca4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8036040 tx=0x7fdbe8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 client.?.objecter ms_handle_connect 0x7fdbf809e7e0 2023-10-30T12:17:48.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:17:48.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8007ab0 con 0x7fdbf809e7e0 2023-10-30T12:17:48.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:17:48.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: dump: 2023-10-30T12:17:48.834 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.835 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.835 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.835 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.835 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.835 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.835 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.836 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.836 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.836 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: _finish_auth 0 2023-10-30T12:17:48.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:17:48.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:17:48.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 5 monclient: authenticate success, global_id 224162 2023-10-30T12:17:48.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe8007c50 con 0x7fdbf809e7e0 2023-10-30T12:17:48.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 10 monclient: _renew_subs 2023-10-30T12:17:48.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf8070de0 con 0x7fdbf809e7e0 2023-10-30T12:17:48.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:17:48.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe803e450 con 0x7fdbf809e7e0 2023-10-30T12:17:48.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:17:48.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 10 monclient: dump: 2023-10-30T12:17:48.839 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.839 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.839 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.839 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.839 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.839 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.840 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.840 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.840 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.840 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 10 client.224162.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:17:48.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 10 monclient: _renew_subs 2023-10-30T12:17:48.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbf57fa640 4 set_mon_vals no callback set 2023-10-30T12:17:48.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:17:48.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf81213f0 con 0x7fdbf809e7e0 2023-10-30T12:17:48.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 1 librados: init done 2023-10-30T12:17:48.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fdbe803e5b0 con 0x7fdbf809e7e0 2023-10-30T12:17:48.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.800+0000 7fdbfd7a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:48.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809b310 0x7fdbf809b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbfd7a4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80a1280 con 0x7fdbf809b310 2023-10-30T12:17:48.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbd80392a0 0x7fdbd803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(522..522 src has 490..522) v4 ==== 6678+0+0 (secure 0 0 0) 0x7fdbe806d690 con 0x7fdbf809e7e0 2023-10-30T12:17:48.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 10 client.224162.objecter ms_dispatch 0x7fdbf8000b90 osd_map(522..522 src has 490..522) v4 2023-10-30T12:17:48.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 3 client.224162.objecter handle_osd_map got epochs [522,522] > 0 2023-10-30T12:17:48.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 3 client.224162.objecter handle_osd_map decoding full epoch 522 2023-10-30T12:17:48.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809b310 0x7fdbf809b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient: get_auth_request con 0x7fdbf809b310 auth_method 0 2023-10-30T12:17:48.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf77fe640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbd80392a0 0x7fdbd803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 20 client.224162.objecter dump_active .. 0 homeless 2023-10-30T12:17:48.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf77fe640 10 monclient: get_auth_request con 0x7fdbd80392a0 auth_method 0 2023-10-30T12:17:48.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_done global_id 204406 payload 824 2023-10-30T12:17:48.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 1 -- 172.21.15.50:0/3012974456 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdbf809e040 con 0x7fdbf809b310 2023-10-30T12:17:48.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 10 client.224162.objecter ms_handle_connect 0x7fdbd80392a0 2023-10-30T12:17:48.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf77fe640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbd80392a0 0x7fdbd803b760 secure :-1 s=READY pgs=4257 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8070db0 tx=0x7fdbf0006bf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 10 client.224162.objecter ms_handle_connect 0x7fdbf809b310 2023-10-30T12:17:48.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbf67fc640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809b310 0x7fdbf809b6f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdbf809cbb0 tx=0x7fdbe000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 10 client.224162.objecter resend_mon_ops 2023-10-30T12:17:48.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.801+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0003e10 con 0x7fdbf809b310 2023-10-30T12:17:48.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:48.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 10 monclient: dump: 2023-10-30T12:17:48.848 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.848 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.848 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.848 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.848 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.848 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.849 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.849 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.849 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.849 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fdbe0005070 con 0x7fdbf809b310 2023-10-30T12:17:48.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:17:48.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:17:48.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809b310 msgr2=0x7fdbf809b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809b310 0x7fdbf809b6f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdbf809cbb0 tx=0x7fdbe000b8c0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809b310 msgr2=0x7fdbf809b6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:48.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.802+0000 7fdbef7fe640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809b310 0x7fdbf809b6f0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdbf809cbb0 tx=0x7fdbe000b8c0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbfd7a4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:17:48.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809bc30 0x7fdbf81366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:17:48.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbfd7a4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:17:48.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8074800 con 0x7fdbf809bc30 2023-10-30T12:17:48.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbf67fc640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809bc30 0x7fdbf81366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:17:48.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbf67fc640 10 monclient: get_auth_request con 0x7fdbf809bc30 auth_method 0 2023-10-30T12:17:48.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbf67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:17:48.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbf67fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:17:48.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.808+0000 7fdbf67fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:17:48.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:17:48.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:17:48.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:17:48.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbf67fc640 10 monclient(hunting): handle_auth_done global_id 204409 payload 824 2023-10-30T12:17:48.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbf67fc640 1 -- 172.21.15.50:0/3012974456 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdbf804e160 con 0x7fdbf809bc30 2023-10-30T12:17:48.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbf67fc640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809bc30 0x7fdbf81366c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0014000 tx=0x7fdbe0005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:17:48.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 10 client.224162.objecter ms_handle_connect 0x7fdbf809bc30 2023-10-30T12:17:48.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 10 client.224162.objecter resend_mon_ops 2023-10-30T12:17:48.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0020070 con 0x7fdbf809bc30 2023-10-30T12:17:48.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:17:48.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:17:48.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 10 monclient: dump: 2023-10-30T12:17:48.856 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:17:48.857 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:17:48.857 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.857 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:17:48.857 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:17:48.857 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:17:48.858 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:17:48.858 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:17:48.858 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:17:48.858 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:17:48.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7fdbe0010400 con 0x7fdbf809bc30 2023-10-30T12:17:48.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:17:48.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:17:48.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809bc30 msgr2=0x7fdbf81366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809bc30 0x7fdbf81366c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0014000 tx=0x7fdbe0005a20 comp rx=0 tx=0).stop 2023-10-30T12:17:48.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809bc30 msgr2=0x7fdbf81366c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:17:48.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.809+0000 7fdbef7fe640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbf809bc30 0x7fdbf81366c0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0014000 tx=0x7fdbe0005a20 comp rx=0 tx=0).stop 2023-10-30T12:17:48.860 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:17:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:48.860 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:17:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 178, 2023-10-30T12:17:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:17:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:17:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:17:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.861 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:17:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:17:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:17:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:48.862 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:48.863 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:48.864 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:48.864 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:48.864 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.864 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:17:48.864 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:17:48.865 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:48.865 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:48.865 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:48.865 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:48.865 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:48.866 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:48.866 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:48.866 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:48.866 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:48.866 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:48.866 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.867 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:17:48.867 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:17:48.867 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:17:48.867 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:17:48.868 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:17:48.868 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:17:48.868 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:48.868 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:17:48.868 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:17:48.868 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:17:48.869 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:17:48.869 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:17:48.869 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:17:48.869 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:17:48.869 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:17:48.870 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:17:48.870 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:17:48.870 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:17:48.870 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:17:48.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:17:48.870 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:17:48.871 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:17:48.871 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:17:48.871 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:17:48.871 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:17:48.871 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:17:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:17:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:17:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:17:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.873 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:17:48.873 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:17:48.873 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:48.873 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:48.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:48.874 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:17:48.874 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:48.875 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:17:48.875 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.875 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.875 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.876 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:17:48.876 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:17:48.876 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:48.876 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:48.876 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:48.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.877 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:17:48.877 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:17:48.877 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:48.877 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:48.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.878 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:48.878 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:17:48.878 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.879 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:48.879 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:17:48.879 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.879 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.879 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.880 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:17:48.880 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:17:48.880 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:48.880 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:17:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:17:48.881 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:17:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:17:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:17:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:17:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:17:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:17:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:17:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:17:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:17:48.885 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:17:48.885 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:17:48.885 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:17:48.885 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.885 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.886 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:17:48.886 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:17:48.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.886 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:48.886 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:48.887 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:17:48.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.887 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:17:48.887 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:17:48.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:17:48.887 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:17:48.888 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:17:48.888 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:17:48.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:17:48.888 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:17:48.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:17:48.889 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:17:48.889 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:17:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 10 librados: watch_flush enter 2023-10-30T12:17:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 10 librados: watch_flush exit 2023-10-30T12:17:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 20 client.224162.objecter shutdown clearing up homeless session... 2023-10-30T12:17:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 10 client.224162.objecter successfully canceled tick 2023-10-30T12:17:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbd80392a0 msgr2=0x7fdbd803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fdbd80392a0 0x7fdbd803b760 secure :-1 s=READY pgs=4257 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8070db0 tx=0x7fdbf0006bf0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 10 monclient: shutdown 2023-10-30T12:17:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:17:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 msgr2=0x7fdbf81ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:17:48.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbf77fe640 1 -- 172.21.15.50:0/3012974456 reap_dead start 2023-10-30T12:17:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf81ca4b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8036040 tx=0x7fdbe8004a40 comp rx=0 tx=0).stop 2023-10-30T12:17:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 shutdown_connections 2023-10-30T12:17:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 --2- 172.21.15.50:0/3012974456 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdbf809e7e0 0x7fdbf81ca4b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:17:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 >> 172.21.15.50:0/3012974456 conn(0x7fdbf80a4800 msgr2=0x7fdbf8133860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:17:48.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 shutdown_connections 2023-10-30T12:17:48.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 -- 172.21.15.50:0/3012974456 wait complete. 2023-10-30T12:17:48.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:17:48.810+0000 7fdbfd7a4640 1 librados: shutdown 2023-10-30T12:17:48.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:17:48.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:18:08.896 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:18:08.896 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:18:08.897 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:18:08.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:18:08.900 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:18:08.900 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:18:08.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:09.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.015+0000 7f1258c75640 10 monclient: get_monmap_and_config 2023-10-30T12:18:09.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.015+0000 7f1258c75640 10 monclient: build_initial_monmap 2023-10-30T12:18:09.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.015+0000 7f1258c75640 10 monclient: monmap: 2023-10-30T12:18:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:09.016935+0000 2023-10-30T12:18:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:09.016935+0000 2023-10-30T12:18:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:09.019 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:09.020 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:09.020 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:09.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.016+0000 7f1258c75640 1 Processor -- start 2023-10-30T12:18:09.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.016+0000 7f1258c75640 1 -- start start 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.016+0000 7f1258c75640 10 monclient: init 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1254135b90 0x7f1254135600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient(hunting): picked mon.noname-c con 0x7f1254135b90 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:09.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1254135f70 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:09.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 0x7f125413d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient(hunting): picked mon.noname-a con 0x7f12540a2b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:09.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1254099f90 con 0x7f1254135f70 2023-10-30T12:18:09.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125409a200 con 0x7f12540a2b60 2023-10-30T12:18:09.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12540a14c0 con 0x7f1254135b90 2023-10-30T12:18:09.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:09.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1258c75640 10 monclient(hunting): authenticate will time out at 1934.200739s 2023-10-30T12:18:09.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1252d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 0x7f125413d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1251d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1254135f70 0x7f12540a2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42710/0 (socket says 172.21.15.50:42710) 2023-10-30T12:18:09.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1252d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 0x7f125413d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54486/0 (socket says 172.21.15.50:54486) 2023-10-30T12:18:09.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1251d9b640 1 -- 172.21.15.50:0/1109777392 learned_addr learned my addr 172.21.15.50:0/1109777392 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:09.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1252d9d640 10 monclient(hunting): get_auth_request con 0x7f12540a2b60 auth_method 0 2023-10-30T12:18:09.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1252d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1252d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f1252d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f125259c640 1 --2- 172.21.15.50:0/1109777392 >> v2:172.21.15.118:3300/0 conn(0x7f1254135b90 0x7f1254135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f125259c640 10 monclient(hunting): get_auth_request con 0x7f1254135b90 auth_method 0 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f125259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f125259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.017+0000 7f125259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f1252d9d640 1 -- 172.21.15.50:0/1109777392 >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 msgr2=0x7f125413d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:18:09.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f1252d9d640 1 -- 172.21.15.50:0/1109777392 >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 msgr2=0x7f125413d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f1252d9d640 1 --2- 172.21.15.50:0/1109777392 >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 0x7f125413d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f1252d9d640 1 --2- 172.21.15.50:0/1109777392 >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 0x7f125413d9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 10 monclient(hunting): handle_auth_done global_id 204412 payload 274 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 1 -- 172.21.15.50:0/1109777392 >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 msgr2=0x7f125413d9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 1 --2- 172.21.15.50:0/1109777392 >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 0x7f125413d9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 1 -- 172.21.15.50:0/1109777392 >> v1:172.21.15.50:6789/0 conn(0x7f1254135f70 legacy=0x7f12540a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 1 monclient: found mon.noname-c 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.018+0000 7f125259c640 1 -- 172.21.15.50:0/1109777392 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125409a4e0 con 0x7f1254135b90 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.019+0000 7f125259c640 1 --2- 172.21.15.50:0/1109777392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135b90 0x7f1254135600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12540a4f50 tx=0x7f124402b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a27fdf62684f9a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 1 -- 172.21.15.50:0/1109777392 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244031710 con 0x7f1254135b90 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: dump: 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:09.034 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:09.035 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: _finish_auth 0 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:09.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f1258c75640 5 monclient: authenticate success, global_id 204412 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f1258c75640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 1 -- 172.21.15.50:0/1109777392 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124402f050 con 0x7f1254135b90 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:09.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f1258c75640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 1 -- 172.21.15.50:0/1109777392 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1244031a90 con 0x7f1254135b90 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f1258c75640 4 set_mon_vals no callback set 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:09.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.020+0000 7f125159a640 10 monclient: dump: 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.039 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:09.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 10 monclient: shutdown 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 1 -- 172.21.15.50:0/1109777392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135b90 msgr2=0x7f1254135600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 1 --2- 172.21.15.50:0/1109777392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135b90 0x7f1254135600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12540a4f50 tx=0x7f124402b7b0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 1 -- 172.21.15.50:0/1109777392 shutdown_connections 2023-10-30T12:18:09.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 1 --2- 172.21.15.50:0/1109777392 >> v2:172.21.15.50:3300/0 conn(0x7f12540a2b60 0x7f125413d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 1 --2- 172.21.15.50:0/1109777392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135b90 0x7f1254135600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.021+0000 7f1258c75640 1 -- 172.21.15.50:0/1109777392 >> 172.21.15.50:0/1109777392 conn(0x7f12540a82b0 msgr2=0x7f12540a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.022+0000 7f1258c75640 1 -- 172.21.15.50:0/1109777392 shutdown_connections 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.022+0000 7f1258c75640 1 -- 172.21.15.50:0/1109777392 wait complete. 2023-10-30T12:18:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 10 monclient: build_initial_monmap 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 10 monclient: monmap: 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:09.043 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:09.044 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:09.044 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:09.044 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:09.044 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:09.044 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:09.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 1 librados: starting msgr at 2023-10-30T12:18:09.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 1 librados: starting objecter 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 1 Processor -- start 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 1 -- start start 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 1 librados: setting wanted keys 2023-10-30T12:18:09.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 1 librados: calling monclient init 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.023+0000 7f1258c75640 10 monclient: init 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 0x7f12541ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): picked mon.noname-a con 0x7f12540a2b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:09.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1254135b90 0x7f12541cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): picked mon.noname-c con 0x7f1254135b90 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135f70 0x7f125409f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1254135b90 0x7f12541cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): picked mon.noname-b con 0x7f1254135f70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:09.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1254120010 con 0x7f12540a2b60 2023-10-30T12:18:09.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12540a16b0 con 0x7f1254135f70 2023-10-30T12:18:09.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:09.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125411f020 con 0x7f1254135b90 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 0x7f12541ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1258c75640 10 monclient(hunting): authenticate will time out at 1934.207725s 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1254135b90 0x7f12541cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33706/0 (socket says 172.21.15.50:33706) 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 1 -- 172.21.15.50:0/1245771856 learned_addr learned my addr 172.21.15.50:0/1245771856 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:09.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 10 monclient(hunting): get_auth_request con 0x7f1254135b90 auth_method 0 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 10 monclient(hunting): get_auth_request con 0x7f12540a2b60 auth_method 0 2023-10-30T12:18:09.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:09.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:09.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:09.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:09.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:09.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f1251d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:09.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 1 -- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 msgr2=0x7f12541ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:18:09.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 1 -- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 msgr2=0x7f12541ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:18:09.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 0x7f12541ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:09.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.024+0000 7f125259c640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 0x7f12541ca640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:09.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 10 monclient(hunting): handle_auth_done global_id 224165 payload 824 2023-10-30T12:18:09.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 1 -- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135f70 msgr2=0x7f125409f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1252d9d640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135f70 0x7f125409f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135f70 0x7f125409f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 1 -- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 msgr2=0x7f12541ca640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 0x7f12541ca640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 1 monclient: found mon.noname-c 2023-10-30T12:18:09.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:09.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:09.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 1 -- 172.21.15.50:0/1245771856 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244009400 con 0x7f1254135b90 2023-10-30T12:18:09.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1252d9d640 10 monclient: get_auth_request con 0x7f1254135f70 auth_method 0 2023-10-30T12:18:09.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1252d9d640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135f70 0x7f125409f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:18:09.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1251d9b640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1254135b90 0x7f12541cab80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f124800c430 tx=0x7f124800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 client.?.objecter ms_handle_connect 0x7f1254135b90 2023-10-30T12:18:09.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:09.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 1 -- 172.21.15.50:0/1245771856 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248005ce0 con 0x7f1254135b90 2023-10-30T12:18:09.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:09.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: dump: 2023-10-30T12:18:09.062 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:09.063 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:09.063 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.063 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.063 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.063 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:09.063 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:09.064 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:09.064 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:09.064 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:09.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:09.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:09.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 5 monclient: authenticate success, global_id 224165 2023-10-30T12:18:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 10 monclient: _renew_subs 2023-10-30T12:18:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 1 -- 172.21.15.50:0/1245771856 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1248005e40 con 0x7f1254135b90 2023-10-30T12:18:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1254134fc0 con 0x7f1254135b90 2023-10-30T12:18:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 1 -- 172.21.15.50:0/1245771856 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12480072f0 con 0x7f1254135b90 2023-10-30T12:18:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1242ffd640 10 monclient: dump: 2023-10-30T12:18:09.067 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:09.067 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:09.067 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.067 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:09.067 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:09.067 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:09.068 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:09.068 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:09.068 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:09.068 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:09.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1250d99640 4 set_mon_vals no callback set 2023-10-30T12:18:09.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 10 client.224165.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:09.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 10 monclient: _renew_subs 2023-10-30T12:18:09.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:09.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125411e3e0 con 0x7f1254135b90 2023-10-30T12:18:09.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.025+0000 7f1258c75640 1 librados: init done 2023-10-30T12:18:09.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1242ffd640 1 -- 172.21.15.50:0/1245771856 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1248007510 con 0x7f1254135b90 2023-10-30T12:18:09.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1242ffd640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12340392f0 0x7f123403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:09.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1242ffd640 1 -- 172.21.15.50:0/1245771856 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(543..543 src has 490..543) v4 ==== 6241+0+0 (secure 0 0 0) 0x7f124804a220 con 0x7f1254135b90 2023-10-30T12:18:09.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1242ffd640 10 client.224165.objecter ms_dispatch 0x7f1254000b90 osd_map(543..543 src has 490..543) v4 2023-10-30T12:18:09.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1242ffd640 3 client.224165.objecter handle_osd_map got epochs [543,543] > 0 2023-10-30T12:18:09.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1242ffd640 3 client.224165.objecter handle_osd_map decoding full epoch 543 2023-10-30T12:18:09.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1242ffd640 20 client.224165.objecter dump_active .. 0 homeless 2023-10-30T12:18:09.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1258c75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1258c75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1258c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:09.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12541205e0 con 0x7f1254135b90 2023-10-30T12:18:09.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f125259c640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12340392f0 0x7f123403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:09.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.026+0000 7f125259c640 10 monclient: get_auth_request con 0x7f12340392f0 auth_method 0 2023-10-30T12:18:09.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.027+0000 7f1242ffd640 10 client.224165.objecter ms_handle_connect 0x7f12340392f0 2023-10-30T12:18:09.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.027+0000 7f125259c640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12340392f0 0x7f123403b7b0 secure :-1 s=READY pgs=4259 cs=0 l=1 rev1=1 crypto rx=0x7f1244035000 tx=0x7f1244008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:09.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.031+0000 7f1242ffd640 1 -- 172.21.15.50:0/1245771856 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f124800e280 con 0x7f1254135b90 2023-10-30T12:18:09.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.031+0000 7f1242ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:09.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.031+0000 7f1242ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:09.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.281+0000 7f1258c75640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:09.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.281+0000 7f1258c75640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:09.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.281+0000 7f1258c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.281+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f125409a7f0 con 0x7f1254135b90 2023-10-30T12:18:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.282+0000 7f1242ffd640 1 -- 172.21.15.50:0/1245771856 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f124803b620 con 0x7f1254135b90 2023-10-30T12:18:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.282+0000 7f1242ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:09.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.282+0000 7f1242ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:09.285 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":178,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:09.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 10 librados: watch_flush enter 2023-10-30T12:18:09.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 10 librados: watch_flush exit 2023-10-30T12:18:09.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 20 client.224165.objecter shutdown clearing up homeless session... 2023-10-30T12:18:09.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 10 client.224165.objecter successfully canceled tick 2023-10-30T12:18:09.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12340392f0 msgr2=0x7f123403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12340392f0 0x7f123403b7b0 secure :-1 s=READY pgs=4259 cs=0 l=1 rev1=1 crypto rx=0x7f1244035000 tx=0x7f1244008030 comp rx=0 tx=0).stop 2023-10-30T12:18:09.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 10 monclient: shutdown 2023-10-30T12:18:09.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:09.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1254135b90 msgr2=0x7f12541cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:09.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1254135b90 0x7f12541cab80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f124800c430 tx=0x7f124800c8f0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 shutdown_connections 2023-10-30T12:18:09.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12340392f0 0x7f123403b7b0 unknown :-1 s=CLOSED pgs=4259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1254135f70 0x7f125409f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1254135b90 0x7f12541cab80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 --2- 172.21.15.50:0/1245771856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12540a2b60 0x7f12541ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 >> 172.21.15.50:0/1245771856 conn(0x7f12540a82b0 msgr2=0x7f1254135410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 shutdown_connections 2023-10-30T12:18:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 -- 172.21.15.50:0/1245771856 wait complete. 2023-10-30T12:18:09.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:09.284+0000 7f1258c75640 1 librados: shutdown 2023-10-30T12:18:12.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:12.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.412+0000 7fbaf51b6640 10 monclient: get_monmap_and_config 2023-10-30T12:18:12.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.413+0000 7fbaf51b6640 10 monclient: build_initial_monmap 2023-10-30T12:18:12.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.413+0000 7fbaf51b6640 10 monclient: monmap: 2023-10-30T12:18:12.414 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:12.414 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:12.414 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:12.413073+0000 2023-10-30T12:18:12.414 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:12.413073+0000 2023-10-30T12:18:12.415 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:12.415 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:12.415 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:12.415 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:12.415 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:12.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:12.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.413+0000 7fbaf51b6640 1 Processor -- start 2023-10-30T12:18:12.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.413+0000 7fbaf51b6640 1 -- start start 2023-10-30T12:18:12.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.413+0000 7fbaf51b6640 10 monclient: init 2023-10-30T12:18:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-10-30T12:18:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbaf009f5b0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf0096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient(hunting): picked mon.noname-b con 0x7fbaf0099b80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbaf0138b30 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf009f990 con 0x7fbaf0138b30 2023-10-30T12:18:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf0098810 con 0x7fbaf009f5b0 2023-10-30T12:18:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf006af10 con 0x7fbaf0099b80 2023-10-30T12:18:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaf51b6640 10 monclient(hunting): authenticate will time out at 1937.596388s 2023-10-30T12:18:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaef59e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbaf0138b30 0x7fbaf013af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42722/0 (socket says 172.21.15.50:42722) 2023-10-30T12:18:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf0096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbaef59e640 1 -- 172.21.15.50:0/3793514998 learned_addr learned my addr 172.21.15.50:0/3793514998 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): get_auth_request con 0x7fbaf0099b80 auth_method 0 2023-10-30T12:18:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient(hunting): handle_auth_done global_id 224168 payload 274 2023-10-30T12:18:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 1 -- 172.21.15.50:0/3793514998 >> v1:172.21.15.118:6789/0 conn(0x7fbaf009f5b0 legacy=0x7fbaf0096080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 1 -- 172.21.15.50:0/3793514998 >> v1:172.21.15.50:6789/0 conn(0x7fbaf0138b30 legacy=0x7fbaf013af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 1 monclient: found mon.noname-b 2023-10-30T12:18:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:18:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.414+0000 7fbae7fff640 1 -- 172.21.15.50:0/3793514998 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf009a1a0 con 0x7fbaf0099b80 2023-10-30T12:18:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbae7fff640 1 --2- 172.21.15.50:0/3793514998 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf0096790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbad80097d0 tx=0x7fbad802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42f5f36f36ab765e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 1 -- 172.21.15.50:0/3793514998 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8031770 con 0x7fbaf0099b80 2023-10-30T12:18:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: dump: 2023-10-30T12:18:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:12.433 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:12.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: _finish_auth 0 2023-10-30T12:18:12.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: _check_auth_tickets 2023-10-30T12:18:12.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:12.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 1 -- 172.21.15.50:0/3793514998 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad80318d0 con 0x7fbaf0099b80 2023-10-30T12:18:12.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaf51b6640 5 monclient: authenticate success, global_id 224168 2023-10-30T12:18:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaf51b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaf51b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 1 -- 172.21.15.50:0/3793514998 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8031c60 con 0x7fbaf0099b80 2023-10-30T12:18:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaf51b6640 4 set_mon_vals no callback set 2023-10-30T12:18:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.415+0000 7fbaee59c640 10 monclient: dump: 2023-10-30T12:18:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:12.437 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.437 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.437 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:12.437 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:12.437 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:12.438 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:12.438 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:12.438 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:12.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 10 monclient: shutdown 2023-10-30T12:18:12.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:12.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 1 -- 172.21.15.50:0/3793514998 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 msgr2=0x7fbaf0096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:12.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/3793514998 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf0096790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbad80097d0 tx=0x7fbad802b810 comp rx=0 tx=0).stop 2023-10-30T12:18:12.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 1 -- 172.21.15.50:0/3793514998 shutdown_connections 2023-10-30T12:18:12.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/3793514998 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf0096790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 1 -- 172.21.15.50:0/3793514998 >> 172.21.15.50:0/3793514998 conn(0x7fbaf00a05c0 msgr2=0x7fbaf009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:12.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 1 -- 172.21.15.50:0/3793514998 shutdown_connections 2023-10-30T12:18:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.416+0000 7fbaf51b6640 1 -- 172.21.15.50:0/3793514998 wait complete. 2023-10-30T12:18:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient: build_initial_monmap 2023-10-30T12:18:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient: monmap: 2023-10-30T12:18:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:12.442 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:12.442 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:12.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 librados: starting msgr at 2023-10-30T12:18:12.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 librados: starting objecter 2023-10-30T12:18:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 Processor -- start 2023-10-30T12:18:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 -- start start 2023-10-30T12:18:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 librados: setting wanted keys 2023-10-30T12:18:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 librados: calling monclient init 2023-10-30T12:18:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient: init 2023-10-30T12:18:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:18:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf01ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf0099b80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbaf009f5b0 0x7fbaf0095350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): picked mon.noname-b con 0x7fbaf009f5b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 0x7fbaf0095890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): picked mon.noname-a con 0x7fbaf0138b30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf008c8a0 con 0x7fbaf0138b30 2023-10-30T12:18:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:12.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf009c400 con 0x7fbaf009f5b0 2023-10-30T12:18:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf008c2e0 con 0x7fbaf0099b80 2023-10-30T12:18:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.417+0000 7fbaf51b6640 10 monclient(hunting): authenticate will time out at 1937.599988s 2023-10-30T12:18:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf01ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf01ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33764/0 (socket says 172.21.15.50:33764) 2023-10-30T12:18:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 -- 172.21.15.50:0/2251990653 learned_addr learned my addr 172.21.15.50:0/2251990653 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): get_auth_request con 0x7fbaf0099b80 auth_method 0 2023-10-30T12:18:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 0x7fbaf0095890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 10 monclient(hunting): get_auth_request con 0x7fbaf0138b30 auth_method 0 2023-10-30T12:18:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbaf009f5b0 0x7fbaf0095350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 10 monclient(hunting): get_auth_request con 0x7fbaf009f5b0 auth_method 0 2023-10-30T12:18:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 1 -- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 msgr2=0x7fbaf0095890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:18:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 1 -- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 msgr2=0x7fbaf0095890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:18:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 0x7fbaf0095890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaef59e640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 0x7fbaf0095890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient(hunting): handle_auth_done global_id 224171 payload 824 2023-10-30T12:18:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 -- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbaf009f5b0 msgr2=0x7fbaf0095350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbaf009f5b0 0x7fbaf0095350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 -- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 msgr2=0x7fbaf0095890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 0x7fbaf0095890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 monclient: found mon.noname-c 2023-10-30T12:18:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 -- 172.21.15.50:0/2251990653 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8009400 con 0x7fbaf0099b80 2023-10-30T12:18:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaeed9d640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf01ca750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbae000c400 tx=0x7fbae000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:12.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 client.?.objecter ms_handle_connect 0x7fbaf0099b80 2023-10-30T12:18:12.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:12.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 1 -- 172.21.15.50:0/2251990653 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae00052d0 con 0x7fbaf0099b80 2023-10-30T12:18:12.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:12.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:12.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: dump: 2023-10-30T12:18:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:12.456 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 1 -- 172.21.15.50:0/2251990653 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae0005430 con 0x7fbaf0099b80 2023-10-30T12:18:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 5 monclient: authenticate success, global_id 224171 2023-10-30T12:18:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 10 monclient: _renew_subs 2023-10-30T12:18:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf008c5c0 con 0x7fbaf0099b80 2023-10-30T12:18:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:12.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 1 -- 172.21.15.50:0/2251990653 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae00038a0 con 0x7fbaf0099b80 2023-10-30T12:18:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbae77fe640 10 monclient: dump: 2023-10-30T12:18:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:12.459 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:12.460 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:12.462 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 10 client.224171.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaedd9b640 4 set_mon_vals no callback set 2023-10-30T12:18:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 10 monclient: _renew_subs 2023-10-30T12:18:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf008b5d0 con 0x7fbaf0099b80 2023-10-30T12:18:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.418+0000 7fbaf51b6640 1 librados: init done 2023-10-30T12:18:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbae77fe640 1 -- 172.21.15.50:0/2251990653 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbae0003a00 con 0x7fbaf0099b80 2023-10-30T12:18:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbaf51b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbae77fe640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbad00392f0 0x7fbad003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbaf51b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbaf51b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbae77fe640 1 -- 172.21.15.50:0/2251990653 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(546..546 src has 490..546) v4 ==== 6622+0+0 (secure 0 0 0) 0x7fbae004a3f0 con 0x7fbaf0099b80 2023-10-30T12:18:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf00a1df0 con 0x7fbaf0099b80 2023-10-30T12:18:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbae77fe640 10 client.224171.objecter ms_dispatch 0x7fbaf0000b90 osd_map(546..546 src has 490..546) v4 2023-10-30T12:18:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbae77fe640 3 client.224171.objecter handle_osd_map got epochs [546,546] > 0 2023-10-30T12:18:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.419+0000 7fbae77fe640 3 client.224171.objecter handle_osd_map decoding full epoch 546 2023-10-30T12:18:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.420+0000 7fbae77fe640 20 client.224171.objecter dump_active .. 0 homeless 2023-10-30T12:18:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.420+0000 7fbae7fff640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbad00392f0 0x7fbad003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.420+0000 7fbae7fff640 10 monclient: get_auth_request con 0x7fbad00392f0 auth_method 0 2023-10-30T12:18:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.420+0000 7fbae77fe640 10 client.224171.objecter ms_handle_connect 0x7fbad00392f0 2023-10-30T12:18:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.420+0000 7fbae7fff640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbad00392f0 0x7fbad003b7b0 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 crypto rx=0x7fbad8036000 tx=0x7fbad803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.424+0000 7fbae77fe640 1 -- 172.21.15.50:0/2251990653 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbae0017070 con 0x7fbaf0099b80 2023-10-30T12:18:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.424+0000 7fbae77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:12.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.424+0000 7fbae77fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.671+0000 7fbaf51b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.671+0000 7fbaf51b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.671+0000 7fbaf51b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.671+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbaf004e160 con 0x7fbaf0099b80 2023-10-30T12:18:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.672+0000 7fbae77fe640 1 -- 172.21.15.50:0/2251990653 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fbae0019e60 con 0x7fbaf0099b80 2023-10-30T12:18:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.672+0000 7fbae77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.672+0000 7fbae77fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:12.674 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":178,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.673+0000 7fbaf51b6640 10 librados: watch_flush enter 2023-10-30T12:18:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 10 librados: watch_flush exit 2023-10-30T12:18:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 20 client.224171.objecter shutdown clearing up homeless session... 2023-10-30T12:18:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 10 client.224171.objecter successfully canceled tick 2023-10-30T12:18:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbad00392f0 msgr2=0x7fbad003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbad00392f0 0x7fbad003b7b0 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 crypto rx=0x7fbad8036000 tx=0x7fbad803d040 comp rx=0 tx=0).stop 2023-10-30T12:18:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 10 monclient: shutdown 2023-10-30T12:18:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 msgr2=0x7fbaf01ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf01ca750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbae000c400 tx=0x7fbae000c8c0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 shutdown_connections 2023-10-30T12:18:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbad00392f0 0x7fbad003b7b0 unknown :-1 s=CLOSED pgs=4260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbaf0138b30 0x7fbaf0095890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbaf009f5b0 0x7fbaf0095350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 --2- 172.21.15.50:0/2251990653 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbaf0099b80 0x7fbaf01ca750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 >> 172.21.15.50:0/2251990653 conn(0x7fbaf00a05c0 msgr2=0x7fbaf013a140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 shutdown_connections 2023-10-30T12:18:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 -- 172.21.15.50:0/2251990653 wait complete. 2023-10-30T12:18:12.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:12.674+0000 7fbaf51b6640 1 librados: shutdown 2023-10-30T12:18:15.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:15.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.800+0000 7fea38a52640 10 monclient: get_monmap_and_config 2023-10-30T12:18:15.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.800+0000 7fea38a52640 10 monclient: build_initial_monmap 2023-10-30T12:18:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.800+0000 7fea38a52640 10 monclient: monmap: 2023-10-30T12:18:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:15.801992+0000 2023-10-30T12:18:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:15.801992+0000 2023-10-30T12:18:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:15.805 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:15.805 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:15.805 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:15.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.801+0000 7fea38a52640 1 Processor -- start 2023-10-30T12:18:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.801+0000 7fea38a52640 1 -- start start 2023-10-30T12:18:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.801+0000 7fea38a52640 10 monclient: init 2023-10-30T12:18:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T12:18:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 0x7fea3409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient(hunting): picked mon.noname-a con 0x7fea34098520 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea34098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea341379f0 0x7fea34139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient(hunting): picked mon.noname-b con 0x7fea341379f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3409ad30 con 0x7fea34098900 2023-10-30T12:18:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3409c460 con 0x7fea34098520 2023-10-30T12:18:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3409b260 con 0x7fea341379f0 2023-10-30T12:18:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea38a52640 10 monclient(hunting): authenticate will time out at 1940.985280s 2023-10-30T12:18:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea341379f0 0x7fea34139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 0x7fea3409bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea341379f0 0x7fea34139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33784/0 (socket says 172.21.15.50:33784) 2023-10-30T12:18:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 0x7fea3409bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54496/0 (socket says 172.21.15.50:54496) 2023-10-30T12:18:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 -- 172.21.15.50:0/2913391375 learned_addr learned my addr 172.21.15.50:0/2913391375 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): get_auth_request con 0x7fea341379f0 auth_method 0 2023-10-30T12:18:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 10 monclient(hunting): get_auth_request con 0x7fea34098520 auth_method 0 2023-10-30T12:18:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 1 -- 172.21.15.50:0/2913391375 >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 msgr2=0x7fea3409bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:18:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 1 -- 172.21.15.50:0/2913391375 >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 msgr2=0x7fea3409bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 1 --2- 172.21.15.50:0/2913391375 >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 0x7fea3409bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea3259c640 1 --2- 172.21.15.50:0/2913391375 >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 0x7fea3409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient(hunting): handle_auth_done global_id 224174 payload 274 2023-10-30T12:18:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 -- 172.21.15.50:0/2913391375 >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 msgr2=0x7fea3409bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 --2- 172.21.15.50:0/2913391375 >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 0x7fea3409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 -- 172.21.15.50:0/2913391375 >> v1:172.21.15.118:6789/0 conn(0x7fea34098900 legacy=0x7fea34098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 monclient: found mon.noname-b 2023-10-30T12:18:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:18:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.802+0000 7fea32d9d640 1 -- 172.21.15.50:0/2913391375 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea34072120 con 0x7fea341379f0 2023-10-30T12:18:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea32d9d640 1 --2- 172.21.15.50:0/2913391375 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea341379f0 0x7fea34139de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009e50 tx=0x7fea2c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52de63afc1818cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 1 -- 172.21.15.50:0/2913391375 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c02c9e0 con 0x7fea341379f0 2023-10-30T12:18:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: dump: 2023-10-30T12:18:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:15.824 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: _finish_auth 0 2023-10-30T12:18:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea38a52640 5 monclient: authenticate success, global_id 224174 2023-10-30T12:18:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea38a52640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 1 -- 172.21.15.50:0/2913391375 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c02cb40 con 0x7fea341379f0 2023-10-30T12:18:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea38a52640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea38a52640 4 set_mon_vals no callback set 2023-10-30T12:18:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 1 -- 172.21.15.50:0/2913391375 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c0343f0 con 0x7fea341379f0 2023-10-30T12:18:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.803+0000 7fea3159a640 10 monclient: dump: 2023-10-30T12:18:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:15.829 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:15.829 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:15.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 10 monclient: shutdown 2023-10-30T12:18:15.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:15.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 -- 172.21.15.50:0/2913391375 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea341379f0 msgr2=0x7fea34139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 --2- 172.21.15.50:0/2913391375 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea341379f0 0x7fea34139de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009e50 tx=0x7fea2c02be40 comp rx=0 tx=0).stop 2023-10-30T12:18:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 -- 172.21.15.50:0/2913391375 shutdown_connections 2023-10-30T12:18:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 --2- 172.21.15.50:0/2913391375 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea341379f0 0x7fea34139de0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 --2- 172.21.15.50:0/2913391375 >> v2:172.21.15.50:3300/0 conn(0x7fea34098520 0x7fea3409bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 -- 172.21.15.50:0/2913391375 >> 172.21.15.50:0/2913391375 conn(0x7fea340a05c0 msgr2=0x7fea3409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 -- 172.21.15.50:0/2913391375 shutdown_connections 2023-10-30T12:18:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.804+0000 7fea38a52640 1 -- 172.21.15.50:0/2913391375 wait complete. 2023-10-30T12:18:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 10 monclient: build_initial_monmap 2023-10-30T12:18:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 10 monclient: monmap: 2023-10-30T12:18:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:15.833 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 1 librados: starting msgr at 2023-10-30T12:18:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 1 librados: starting objecter 2023-10-30T12:18:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 1 Processor -- start 2023-10-30T12:18:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 1 -- start start 2023-10-30T12:18:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 1 librados: setting wanted keys 2023-10-30T12:18:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 1 librados: calling monclient init 2023-10-30T12:18:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 10 monclient: init 2023-10-30T12:18:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.805+0000 7fea38a52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea34098520 0x7fea3413a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): picked mon.noname-c con 0x7fea34098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 0x7fea3413a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): picked mon.noname-a con 0x7fea34098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea341379f0 0x7fea3413ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): picked mon.noname-b con 0x7fea341379f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea34090370 con 0x7fea34098900 2023-10-30T12:18:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3409d610 con 0x7fea341379f0 2023-10-30T12:18:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea34096ce0 con 0x7fea34098520 2023-10-30T12:18:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient(hunting): authenticate will time out at 1940.989090s 2023-10-30T12:18:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea34098520 0x7fea3413a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 0x7fea3413a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea34098520 0x7fea3413a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33786/0 (socket says 172.21.15.50:33786) 2023-10-30T12:18:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 -- 172.21.15.50:0/543369295 learned_addr learned my addr 172.21.15.50:0/543369295 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): get_auth_request con 0x7fea34098520 auth_method 0 2023-10-30T12:18:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 10 monclient(hunting): get_auth_request con 0x7fea34098900 auth_method 0 2023-10-30T12:18:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 1 -- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 msgr2=0x7fea3413a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:18:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 1 -- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 msgr2=0x7fea3413a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:18:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 0x7fea3413a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea31d9b640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 0x7fea3413a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea32d9d640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea341379f0 0x7fea3413ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea32d9d640 10 monclient(hunting): get_auth_request con 0x7fea341379f0 auth_method 0 2023-10-30T12:18:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea32d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea32d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient(hunting): handle_auth_done global_id 224177 payload 824 2023-10-30T12:18:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 -- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea341379f0 msgr2=0x7fea3413ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea341379f0 0x7fea3413ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 -- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 msgr2=0x7fea3413a860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 0x7fea3413a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient: _finish_hunting 0 2023-10-30T12:18:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 monclient: found mon.noname-c 2023-10-30T12:18:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 -- 172.21.15.50:0/543369295 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c009b30 con 0x7fea34098520 2023-10-30T12:18:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea32d9d640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea341379f0 0x7fea3413ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:18:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea3259c640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea34098520 0x7fea3413a320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fea2400d830 tx=0x7fea2400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 client.?.objecter ms_handle_connect 0x7fea34098520 2023-10-30T12:18:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 1 -- 172.21.15.50:0/543369295 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24018070 con 0x7fea34098520 2023-10-30T12:18:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: dump: 2023-10-30T12:18:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:15.848 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 1 -- 172.21.15.50:0/543369295 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea24005990 con 0x7fea34098520 2023-10-30T12:18:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 5 monclient: authenticate success, global_id 224177 2023-10-30T12:18:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 1 -- 172.21.15.50:0/543369295 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24005b20 con 0x7fea34098520 2023-10-30T12:18:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient: _renew_subs 2023-10-30T12:18:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea34090090 con 0x7fea34098520 2023-10-30T12:18:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea30d99640 4 set_mon_vals no callback set 2023-10-30T12:18:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.806+0000 7fea22ffd640 10 monclient: dump: 2023-10-30T12:18:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:15.852 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:15.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 10 client.224177.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 10 monclient: _renew_subs 2023-10-30T12:18:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3408fd80 con 0x7fea34098520 2023-10-30T12:18:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 1 librados: init done 2023-10-30T12:18:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea22ffd640 1 -- 172.21.15.50:0/543369295 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fea2400b6c0 con 0x7fea34098520 2023-10-30T12:18:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.807+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3404e160 con 0x7fea34098520 2023-10-30T12:18:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea22ffd640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fea140392f0 0x7fea1403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea22ffd640 1 -- 172.21.15.50:0/543369295 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(549..549 src has 490..549) v4 ==== 6622+0+0 (secure 0 0 0) 0x7fea2404aed0 con 0x7fea34098520 2023-10-30T12:18:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea22ffd640 10 client.224177.objecter ms_dispatch 0x7fea34000b90 osd_map(549..549 src has 490..549) v4 2023-10-30T12:18:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea22ffd640 3 client.224177.objecter handle_osd_map got epochs [549,549] > 0 2023-10-30T12:18:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea22ffd640 3 client.224177.objecter handle_osd_map decoding full epoch 549 2023-10-30T12:18:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea22ffd640 20 client.224177.objecter dump_active .. 0 homeless 2023-10-30T12:18:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea31d9b640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fea140392f0 0x7fea1403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea31d9b640 10 monclient: get_auth_request con 0x7fea140392f0 auth_method 0 2023-10-30T12:18:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea22ffd640 10 client.224177.objecter ms_handle_connect 0x7fea140392f0 2023-10-30T12:18:15.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.808+0000 7fea31d9b640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fea140392f0 0x7fea1403b7b0 secure :-1 s=READY pgs=4261 cs=0 l=1 rev1=1 crypto rx=0x7fea34071bd0 tx=0x7fea28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.812+0000 7fea22ffd640 1 -- 172.21.15.50:0/543369295 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea2400b970 con 0x7fea34098520 2023-10-30T12:18:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.812+0000 7fea22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:15.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:15.812+0000 7fea22ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:16.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.063+0000 7fea38a52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:16.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.063+0000 7fea38a52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:16.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.063+0000 7fea38a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:16.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.063+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fea34096810 con 0x7fea34098520 2023-10-30T12:18:16.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.064+0000 7fea22ffd640 1 -- 172.21.15.50:0/543369295 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fea24013b80 con 0x7fea34098520 2023-10-30T12:18:16.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.064+0000 7fea22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:16.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.064+0000 7fea22ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:16.067 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":178,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:16.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 10 librados: watch_flush enter 2023-10-30T12:18:16.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 10 librados: watch_flush exit 2023-10-30T12:18:16.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 20 client.224177.objecter shutdown clearing up homeless session... 2023-10-30T12:18:16.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 10 client.224177.objecter successfully canceled tick 2023-10-30T12:18:16.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fea140392f0 msgr2=0x7fea1403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:16.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fea140392f0 0x7fea1403b7b0 secure :-1 s=READY pgs=4261 cs=0 l=1 rev1=1 crypto rx=0x7fea34071bd0 tx=0x7fea28009040 comp rx=0 tx=0).stop 2023-10-30T12:18:16.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 10 monclient: shutdown 2023-10-30T12:18:16.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:16.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea34098520 msgr2=0x7fea3413a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:16.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea34098520 0x7fea3413a320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fea2400d830 tx=0x7fea2400dcf0 comp rx=0 tx=0).stop 2023-10-30T12:18:16.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 shutdown_connections 2023-10-30T12:18:16.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fea140392f0 0x7fea1403b7b0 unknown :-1 s=CLOSED pgs=4261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:16.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea341379f0 0x7fea3413ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:16.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea34098900 0x7fea3413a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:16.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 --2- 172.21.15.50:0/543369295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea34098520 0x7fea3413a320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:16.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 >> 172.21.15.50:0/543369295 conn(0x7fea340a05c0 msgr2=0x7fea34138090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:16.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 shutdown_connections 2023-10-30T12:18:16.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 -- 172.21.15.50:0/543369295 wait complete. 2023-10-30T12:18:16.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:16.066+0000 7fea38a52640 1 librados: shutdown 2023-10-30T12:18:19.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:19.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.190+0000 7f1a8fb4c640 10 monclient: get_monmap_and_config 2023-10-30T12:18:19.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.190+0000 7f1a8fb4c640 10 monclient: build_initial_monmap 2023-10-30T12:18:19.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.190+0000 7f1a8fb4c640 10 monclient: monmap: 2023-10-30T12:18:19.192 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:19.191663+0000 2023-10-30T12:18:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:19.191663+0000 2023-10-30T12:18:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:19.195 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:19.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 1 Processor -- start 2023-10-30T12:18:19.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 1 -- start start 2023-10-30T12:18:19.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient: init 2023-10-30T12:18:19.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:19.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T12:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 0x7f1a8809f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a88131b20 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a88132270 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a88132670 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a88131f00 con 0x7f1a88132670 2023-10-30T12:18:19.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8809d920 con 0x7f1a88132270 2023-10-30T12:18:19.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:19.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8809a0b0 con 0x7f1a88131b20 2023-10-30T12:18:19.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:19.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.191+0000 7f1a8fb4c640 10 monclient(hunting): authenticate will time out at 1944.374998s 2023-10-30T12:18:19.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 0x7f1a8809f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8e0e9640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1a88132670 0x7f1a88139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53428/0 (socket says 172.21.15.50:53428) 2023-10-30T12:18:19.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8e0e9640 1 -- 172.21.15.50:0/3745065383 learned_addr learned my addr 172.21.15.50:0/3745065383 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:19.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 10 monclient(hunting): get_auth_request con 0x7f1a88131b20 auth_method 0 2023-10-30T12:18:19.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:19.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:19.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/3745065383 >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 msgr2=0x7f1a8809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:19.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/3745065383 >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 msgr2=0x7f1a8809f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:19.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/3745065383 >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 0x7f1a8809f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:19.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/3745065383 >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 0x7f1a8809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:19.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734628307 0 0) 0x7f1a8809d920 con 0x7f1a88132270 2023-10-30T12:18:19.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient(hunting): _negotiate 2023-10-30T12:18:19.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:19.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient(hunting): my global_id is 204427 2023-10-30T12:18:19.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a74003520 con 0x7f1a88132270 2023-10-30T12:18:19.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411178541 0 0) 0x7f1a74003520 con 0x7f1a88132270 2023-10-30T12:18:19.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 msgr2=0x7f1a8809f5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:19.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 --2- 172.21.15.50:0/3745065383 >> v2:172.21.15.50:3300/0 conn(0x7f1a88131b20 0x7f1a8809f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 >> v1:172.21.15.50:6789/0 conn(0x7f1a88132670 legacy=0x7f1a88139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:19.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: _finish_hunting 0 2023-10-30T12:18:19.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:18:19.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:19.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:18:19.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8809a390 con 0x7f1a88132270 2023-10-30T12:18:19.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: _finish_auth 0 2023-10-30T12:18:19.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: _check_auth_tickets 2023-10-30T12:18:19.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:19.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f1a78001670 con 0x7f1a88132270 2023-10-30T12:18:19.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8fb4c640 5 monclient: authenticate success, global_id 204427 2023-10-30T12:18:19.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8fb4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:19.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:19.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:18:19.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: dump: 2023-10-30T12:18:19.213 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:19.213 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:19.213 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:19.215 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:19.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:18:19.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:19.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 >> v1:172.21.15.118:6789/0 conn(0x7f1a88132270 legacy=0x7f1a88071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:19.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:18:19.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient(hunting): picked mon.c con 0x7f1a74003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:19.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a740047c0 0x7f1a74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 10 monclient(hunting): picked mon.b con 0x7f1a740047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:19.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.192+0000 7f1a8c8e6640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a74005380 0x7f1a7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 10 monclient(hunting): picked mon.a con 0x7f1a74005380 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:19.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:19.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8809a0b0 con 0x7f1a74005380 2023-10-30T12:18:19.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:19.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7400c210 con 0x7f1a740047c0 2023-10-30T12:18:19.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:19.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7400c3d0 con 0x7f1a74003970 2023-10-30T12:18:19.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:19.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8c8e6640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:18:19.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8fb4c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:19.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a74005380 0x7f1a7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): get_auth_request con 0x7f1a74003970 auth_method 0 2023-10-30T12:18:19.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:19.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:19.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 10 monclient(hunting): get_auth_request con 0x7f1a74005380 auth_method 0 2023-10-30T12:18:19.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:19.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:19.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a740047c0 0x7f1a74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:19.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:19.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:19.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:19.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:19.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:19.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 10 monclient(hunting): get_auth_request con 0x7f1a740047c0 auth_method 0 2023-10-30T12:18:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:19.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d0e7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_done global_id 204427 payload 293 2023-10-30T12:18:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/3745065383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a740047c0 msgr2=0x7f1a74004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a740047c0 0x7f1a74004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a74005380 msgr2=0x7f1a7400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:19.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a74005380 0x7f1a7400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient: _finish_hunting 0 2023-10-30T12:18:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 1 monclient: found mon.c 2023-10-30T12:18:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/3745065383 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74003520 con 0x7f1a74003970 2023-10-30T12:18:19.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient: _finish_auth 0 2023-10-30T12:18:19.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 10 monclient: _check_auth_tickets 2023-10-30T12:18:19.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8d8e8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:19.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a74005380 0x7f1a7400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:19.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.193+0000 7f1a8e0e9640 1 -- 172.21.15.50:0/3745065383 reap_dead start 2023-10-30T12:18:19.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a74003d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400b9e0 tx=0x7f1a740057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28a63579cd694e9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:19.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84004020 con 0x7f1a74003970 2023-10-30T12:18:19.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:19.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:19.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 10 monclient: dump: 2023-10-30T12:18:19.226 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:19.227 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:19.227 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.227 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.227 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:19.227 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:19.227 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:19.228 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a84004180 con 0x7f1a74003970 2023-10-30T12:18:19.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:19.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:19.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 4 set_mon_vals no callback set 2023-10-30T12:18:19.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 1 -- 172.21.15.50:0/3745065383 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84003610 con 0x7f1a74003970 2023-10-30T12:18:19.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:19.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8c8e6640 10 monclient: dump: 2023-10-30T12:18:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:19.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 10 monclient: shutdown 2023-10-30T12:18:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 msgr2=0x7f1a74003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a74003d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400b9e0 tx=0x7f1a740057f0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/3745065383 shutdown_connections 2023-10-30T12:18:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/3745065383 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a74003d90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.194+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/3745065383 >> 172.21.15.50:0/3745065383 conn(0x7f1a880a05c0 msgr2=0x7f1a8809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.195+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/3745065383 shutdown_connections 2023-10-30T12:18:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.195+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/3745065383 wait complete. 2023-10-30T12:18:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.195+0000 7f1a8fb4c640 10 monclient: build_initial_monmap 2023-10-30T12:18:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.195+0000 7f1a8fb4c640 10 monclient: monmap: 2023-10-30T12:18:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:19.235 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.195+0000 7f1a8fb4c640 1 librados: starting msgr at 2023-10-30T12:18:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.195+0000 7f1a8fb4c640 1 librados: starting objecter 2023-10-30T12:18:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.195+0000 7f1a8fb4c640 1 Processor -- start 2023-10-30T12:18:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 -- start start 2023-10-30T12:18:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 librados: setting wanted keys 2023-10-30T12:18:19.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 librados: calling monclient init 2023-10-30T12:18:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient: init 2023-10-30T12:18:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a88129a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a74003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 0x7f1a8806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a8806faa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a88070420 0x7f1a881d0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a88070420 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880993c0 con 0x7f1a8806faa0 2023-10-30T12:18:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a88071c30 con 0x7f1a88070420 2023-10-30T12:18:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880708b0 con 0x7f1a74003970 2023-10-30T12:18:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8fb4c640 10 monclient(hunting): authenticate will time out at 1944.379447s 2023-10-30T12:18:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a88129a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 0x7f1a8806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a88129a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:54208/0 (socket says 172.21.15.50:54208) 2023-10-30T12:18:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/2354424944 learned_addr learned my addr 172.21.15.50:0/2354424944 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): get_auth_request con 0x7f1a74003970 auth_method 0 2023-10-30T12:18:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:19.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 10 monclient(hunting): get_auth_request con 0x7f1a8806faa0 auth_method 0 2023-10-30T12:18:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 1 -- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 msgr2=0x7f1a8806fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:18:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 1 -- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 msgr2=0x7f1a8806fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:18:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 0x7f1a8806fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8e0e9640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a88070420 0x7f1a881d0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8e0e9640 10 monclient(hunting): get_auth_request con 0x7f1a88070420 auth_method 0 2023-10-30T12:18:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8e0e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8e0e9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8e0e9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d0e7640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 0x7f1a8806fee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient(hunting): handle_auth_done global_id 224180 payload 824 2023-10-30T12:18:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a88070420 msgr2=0x7f1a881d0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a88070420 0x7f1a881d0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 msgr2=0x7f1a8806fee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 0x7f1a8806fee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient: _finish_hunting 0 2023-10-30T12:18:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 monclient: found mon.noname-c 2023-10-30T12:18:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8d8e8640 1 -- 172.21.15.50:0/2354424944 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a88128960 con 0x7f1a74003970 2023-10-30T12:18:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.196+0000 7f1a8e0e9640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a88070420 0x7f1a881d0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:18:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8d8e8640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a88129a70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400bb00 tx=0x7f1a84004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 client.?.objecter ms_handle_connect 0x7f1a74003970 2023-10-30T12:18:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:19.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/2354424944 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84003a40 con 0x7f1a74003970 2023-10-30T12:18:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: dump: 2023-10-30T12:18:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:19.248 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:19.250 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 5 monclient: authenticate success, global_id 224180 2023-10-30T12:18:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/2354424944 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a84040070 con 0x7f1a74003970 2023-10-30T12:18:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 10 monclient: _renew_subs 2023-10-30T12:18:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/2354424944 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84003dc0 con 0x7f1a74003970 2023-10-30T12:18:19.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a88128c40 con 0x7f1a74003970 2023-10-30T12:18:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7e7fc640 10 monclient: dump: 2023-10-30T12:18:19.252 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:19.253 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:19.254 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a7ffff640 4 set_mon_vals no callback set 2023-10-30T12:18:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 10 client.224180.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 10 monclient: _renew_subs 2023-10-30T12:18:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a88131560 con 0x7f1a74003970 2023-10-30T12:18:19.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.197+0000 7f1a8fb4c640 1 librados: init done 2023-10-30T12:18:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/2354424944 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1a84042030 con 0x7f1a74003970 2023-10-30T12:18:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/2354424944 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(553..553 src has 490..553) v4 ==== 6627+0+0 (secure 0 0 0) 0x7f1a8406cda0 con 0x7f1a74003970 2023-10-30T12:18:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 10 client.224180.objecter ms_dispatch 0x7f1a88000b90 osd_map(553..553 src has 490..553) v4 2023-10-30T12:18:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 3 client.224180.objecter handle_osd_map got epochs [553,553] > 0 2023-10-30T12:18:19.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 3 client.224180.objecter handle_osd_map decoding full epoch 553 2023-10-30T12:18:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 20 client.224180.objecter dump_active .. 0 homeless 2023-10-30T12:18:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a8fb4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a8fb4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a8fb4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:19.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a880677b0 con 0x7f1a74003970 2023-10-30T12:18:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a8d0e7640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a8d0e7640 10 monclient: get_auth_request con 0x7f1a6c0392f0 auth_method 0 2023-10-30T12:18:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a7e7fc640 10 client.224180.objecter ms_handle_connect 0x7f1a6c0392f0 2023-10-30T12:18:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.198+0000 7f1a8d0e7640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 secure :-1 s=READY pgs=4262 cs=0 l=1 rev1=1 crypto rx=0x7f1a780010e0 tx=0x7f1a78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:19.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.202+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/2354424944 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1a8403c970 con 0x7f1a74003970 2023-10-30T12:18:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.203+0000 7f1a7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:19.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.203+0000 7f1a7e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:19.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.451+0000 7f1a8fb4c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.451+0000 7f1a8fb4c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.451+0000 7f1a8fb4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:19.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.451+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1a8804e160 con 0x7f1a74003970 2023-10-30T12:18:19.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.452+0000 7f1a7e7fc640 1 -- 172.21.15.50:0/2354424944 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f1a84044bc0 con 0x7f1a74003970 2023-10-30T12:18:19.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.452+0000 7f1a7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:19.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.452+0000 7f1a7e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:19.457 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":178,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 10 librados: watch_flush enter 2023-10-30T12:18:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 10 librados: watch_flush exit 2023-10-30T12:18:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 20 client.224180.objecter shutdown clearing up homeless session... 2023-10-30T12:18:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 10 client.224180.objecter successfully canceled tick 2023-10-30T12:18:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 msgr2=0x7f1a6c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 secure :-1 s=READY pgs=4262 cs=0 l=1 rev1=1 crypto rx=0x7f1a780010e0 tx=0x7f1a78009040 comp rx=0 tx=0).stop 2023-10-30T12:18:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 10 monclient: shutdown 2023-10-30T12:18:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 msgr2=0x7f1a88129a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a88129a70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400bb00 tx=0x7f1a84004730 comp rx=0 tx=0).stop 2023-10-30T12:18:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 shutdown_connections 2023-10-30T12:18:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1a6c0392f0 0x7f1a6c03b7b0 unknown :-1 s=CLOSED pgs=4262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a88070420 0x7f1a881d0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a8806faa0 0x7f1a8806fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 --2- 172.21.15.50:0/2354424944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1a74003970 0x7f1a88129a70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 >> 172.21.15.50:0/2354424944 conn(0x7f1a880a05c0 msgr2=0x7f1a881bac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 shutdown_connections 2023-10-30T12:18:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 -- 172.21.15.50:0/2354424944 wait complete. 2023-10-30T12:18:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:19.454+0000 7f1a8fb4c640 1 librados: shutdown 2023-10-30T12:18:22.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:22.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.582+0000 7f63083b6640 10 monclient: get_monmap_and_config 2023-10-30T12:18:22.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.582+0000 7f63083b6640 10 monclient: build_initial_monmap 2023-10-30T12:18:22.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.582+0000 7f63083b6640 10 monclient: monmap: 2023-10-30T12:18:22.584 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:22.584 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:22.583410+0000 2023-10-30T12:18:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:22.583410+0000 2023-10-30T12:18:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:22.586 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:22.586 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:22.586 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:22.586 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.582+0000 7f63083b6640 1 Processor -- start 2023-10-30T12:18:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.582+0000 7f63083b6640 1 -- start start 2023-10-30T12:18:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.582+0000 7f63083b6640 10 monclient: init 2023-10-30T12:18:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:18:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient(hunting): picked mon.noname-a con 0x7f63000707d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63000a0170 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63000710f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63000a11f0 con 0x7f63000a0170 2023-10-30T12:18:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6300135410 con 0x7f63000710f0 2023-10-30T12:18:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630009f4b0 con 0x7f63000707d0 2023-10-30T12:18:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f63083b6640 10 monclient(hunting): authenticate will time out at 1947.766663s 2023-10-30T12:18:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6305951640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f63000a0170 0x7f63001345f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53434/0 (socket says 172.21.15.50:53434) 2023-10-30T12:18:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48124/0 (socket says 172.21.15.50:48124) 2023-10-30T12:18:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 learned_addr learned my addr 172.21.15.50:0/2877798866 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f63000707d0 auth_method 0 2023-10-30T12:18:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:22.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:22.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:22.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.583+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:22.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:22.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f63000707d0 auth_method 0 2023-10-30T12:18:22.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:22.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:22.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:22.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:22.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:22.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:22.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:22.784+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:18:23.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:23.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f63000707d0 auth_method 0 2023-10-30T12:18:23.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:23.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:23.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:23.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:23.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:23.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:23.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.185+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:18:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f63000707d0 auth_method 0 2023-10-30T12:18:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.986+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:18:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.999+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> v1:172.21.15.50:6789/0 conn(0x7f63000a0170 legacy=0x7f63001345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-10-30T12:18:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.999+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> v1:172.21.15.50:6789/0 conn(0x7f63000a0170 legacy=0x7f63001345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:18:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.999+0000 7f6305951640 1 --1- 172.21.15.50:0/2877798866 >> v1:172.21.15.50:6789/0 conn(0x7f63000a0170 0x7f63001345f0 :-1 s=OPENED pgs=31 cs=1 l=1).handle_message read tag failed 2023-10-30T12:18:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.999+0000 7f6305951640 1 --1- 172.21.15.50:0/2877798866 >> v1:172.21.15.50:6789/0 conn(0x7f63000a0170 0x7f63001345f0 :-1 s=OPENED pgs=31 cs=1 l=1).fault on lossy channel, failing 2023-10-30T12:18:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:23.999+0000 7f6305150640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.50:6789/0 2023-10-30T12:18:25.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): tick 2023-10-30T12:18:25.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:18:25.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:18:25.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:18:25.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 msgr2=0x7f6300070bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:25.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.50:3300/0 conn(0x7f63000707d0 0x7f6300070bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:25.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 -- 172.21.15.50:0/2877798866 >> v1:172.21.15.118:6789/0 conn(0x7f63000710f0 legacy=0x7f6300134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:25.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 -- 172.21.15.50:0/2877798866 >> v1:172.21.15.50:6789/0 conn(0x7f63000a0170 legacy=0x7f63001345f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:25.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient: _add_conns ranks=[4,2] 2023-10-30T12:18:25.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:25.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f62e4003610 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:25.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:25.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f62e4006150 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:25.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:25.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 -- 172.21.15.50:0/2877798866 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e4008d10 con 0x7f62e4003610 2023-10-30T12:18:25.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:25.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 1 -- 172.21.15.50:0/2877798866 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4008e80 con 0x7f62e4006150 2023-10-30T12:18:25.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.583+0000 7f62f7fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:25.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:25.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f62e4003610 auth_method 0 2023-10-30T12:18:25.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:25.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:25.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:25.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:25.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 10 monclient(hunting): get_auth_request con 0x7f62e4006150 auth_method 0 2023-10-30T12:18:25.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:25.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:25.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:25.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 msgr2=0x7f62e4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:25.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 msgr2=0x7f62e4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:25.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:25.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:25.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:18:25.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:25.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:25.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.584+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:25.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:25.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 10 monclient(hunting): get_auth_request con 0x7f62e4006150 auth_method 0 2023-10-30T12:18:25.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:25.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:25.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:25.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:25.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:25.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:25.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:18:25.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:25.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f62e4003610 auth_method 0 2023-10-30T12:18:25.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:25.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:25.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:25.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 msgr2=0x7f62e4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:18:25.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 msgr2=0x7f62e4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:25.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:25.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:25.785+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:18:26.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:26.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 10 monclient(hunting): get_auth_request con 0x7f62e4006150 auth_method 0 2023-10-30T12:18:26.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:26.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:26.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:26.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:26.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:26.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:26.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:18:26.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:26.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f62e4003610 auth_method 0 2023-10-30T12:18:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 msgr2=0x7f62e4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:18:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 msgr2=0x7f62e4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:26.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:26.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.186+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:18:26.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:26.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 10 monclient(hunting): get_auth_request con 0x7f62e4006150 auth_method 0 2023-10-30T12:18:26.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6305951640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:18:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> v2:172.21.15.118:3300/0 conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f62e4003610 auth_method 0 2023-10-30T12:18:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.987+0000 7f6306152640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:26.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 10 monclient(hunting): handle_auth_done global_id 204433 payload 274 2023-10-30T12:18:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 msgr2=0x7f62e4008610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62e4006150 0x7f62e4008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 10 monclient: _finish_hunting 0 2023-10-30T12:18:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 1 monclient: found mon.noname-c 2023-10-30T12:18:26.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:18:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.988+0000 7f6306152640 1 -- 172.21.15.50:0/2877798866 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630009d9d0 con 0x7f62e4003610 2023-10-30T12:18:27.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6306152640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 0x7f62e4005a90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006fd0 tx=0x7f62fc02d6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b12fdc68b5b52e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:27.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 1 -- 172.21.15.50:0/2877798866 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc004020 con 0x7f62e4003610 2023-10-30T12:18:27.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:27.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:27.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: dump: 2023-10-30T12:18:27.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:27.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:27.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:27.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:27.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:27.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:27.004 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:27.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:27.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: _finish_auth 0 2023-10-30T12:18:27.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: _check_auth_tickets 2023-10-30T12:18:27.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 1 -- 172.21.15.50:0/2877798866 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc030070 con 0x7f62e4003610 2023-10-30T12:18:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 1 -- 172.21.15.50:0/2877798866 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc004200 con 0x7f62e4003610 2023-10-30T12:18:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f63083b6640 5 monclient: authenticate success, global_id 204433 2023-10-30T12:18:27.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f63083b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:27.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f63083b6640 4 set_mon_vals no callback set 2023-10-30T12:18:27.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:27.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:27.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.989+0000 7f6305150640 10 monclient: dump: 2023-10-30T12:18:27.007 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:27.007 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:27.007 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.007 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.007 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:27.007 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:27.008 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 10 monclient: shutdown 2023-10-30T12:18:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 1 -- 172.21.15.50:0/2877798866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 msgr2=0x7f62e4005a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 0x7f62e4005a90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006fd0 tx=0x7f62fc02d6c0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f6306953640 1 -- 172.21.15.50:0/2877798866 reap_dead start 2023-10-30T12:18:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 1 -- 172.21.15.50:0/2877798866 shutdown_connections 2023-10-30T12:18:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 1 --2- 172.21.15.50:0/2877798866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 0x7f62e4005a90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 1 -- 172.21.15.50:0/2877798866 >> 172.21.15.50:0/2877798866 conn(0x7f63000a47f0 msgr2=0x7f63000a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 1 -- 172.21.15.50:0/2877798866 shutdown_connections 2023-10-30T12:18:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.990+0000 7f63083b6640 1 -- 172.21.15.50:0/2877798866 wait complete. 2023-10-30T12:18:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 10 monclient: build_initial_monmap 2023-10-30T12:18:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 10 monclient: monmap: 2023-10-30T12:18:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:27.012 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 1 librados: starting msgr at 2023-10-30T12:18:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 1 librados: starting objecter 2023-10-30T12:18:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 1 Processor -- start 2023-10-30T12:18:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 1 -- start start 2023-10-30T12:18:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 1 librados: setting wanted keys 2023-10-30T12:18:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 1 librados: calling monclient init 2023-10-30T12:18:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.991+0000 7f63083b6640 10 monclient: init 2023-10-30T12:18:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 0x7f63001ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:27.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): picked mon.noname-b con 0x7f62e4003610 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:27.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63001cab60 0x7f630011fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:27.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): picked mon.noname-a con 0x7f63001cab60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:27.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 0x7f630011ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:27.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): picked mon.noname-c con 0x7f63001caf80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:27.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6300123750 con 0x7f63001cab60 2023-10-30T12:18:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63001239a0 con 0x7f62e4003610 2023-10-30T12:18:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6300121b50 con 0x7f63001caf80 2023-10-30T12:18:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:27.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f63083b6640 10 monclient(hunting): authenticate will time out at 1952.175316s 2023-10-30T12:18:27.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63001cab60 0x7f630011fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:27.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 0x7f630011ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:27.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63001cab60 0x7f630011fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48154/0 (socket says 172.21.15.50:48154) 2023-10-30T12:18:27.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 10 monclient(hunting): get_auth_request con 0x7f63001caf80 auth_method 0 2023-10-30T12:18:27.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 -- 172.21.15.50:0/2243632312 learned_addr learned my addr 172.21.15.50:0/2243632312 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:27.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:27.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:27.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:27.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 0x7f63001ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:27.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): get_auth_request con 0x7f63001cab60 auth_method 0 2023-10-30T12:18:27.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:27.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:27.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:27.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 10 monclient(hunting): get_auth_request con 0x7f62e4003610 auth_method 0 2023-10-30T12:18:27.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:27.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:27.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:27.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 1 -- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 msgr2=0x7f630011ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:18:27.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 1 -- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 msgr2=0x7f630011ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:18:27.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 0x7f630011ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:27.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:27.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:27.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306953640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 0x7f630011ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:27.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:27.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:27.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:27.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6306152640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:27.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient(hunting): handle_auth_done global_id 234099 payload 824 2023-10-30T12:18:27.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 -- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 msgr2=0x7f630011ff50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:27.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 0x7f630011ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 -- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 msgr2=0x7f63001ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:27.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 0x7f63001ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient: _finish_hunting 0 2023-10-30T12:18:27.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 monclient: found mon.noname-a 2023-10-30T12:18:27.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:27.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:27.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.992+0000 7f6305951640 1 -- 172.21.15.50:0/2243632312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc00e040 con 0x7f63001cab60 2023-10-30T12:18:27.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f6305951640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63001cab60 0x7f630011fa10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f62f0007890 tx=0x7f62f00017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:27.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 client.?.objecter ms_handle_connect 0x7f63001cab60 2023-10-30T12:18:27.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:27.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 1 -- 172.21.15.50:0/2243632312 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0016690 con 0x7f63001cab60 2023-10-30T12:18:27.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:27.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:27.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: dump: 2023-10-30T12:18:27.026 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:27.026 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:27.026 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.026 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.026 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:27.027 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 1 -- 172.21.15.50:0/2243632312 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f00167f0 con 0x7f63001cab60 2023-10-30T12:18:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 1 -- 172.21.15.50:0/2243632312 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0016ba0 con 0x7f63001cab60 2023-10-30T12:18:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f62f6ffd640 10 monclient: dump: 2023-10-30T12:18:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:27.031 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 5 monclient: authenticate success, global_id 234099 2023-10-30T12:18:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 10 monclient: _renew_subs 2023-10-30T12:18:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630009d6a0 con 0x7f63001cab60 2023-10-30T12:18:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f630494f640 4 set_mon_vals no callback set 2023-10-30T12:18:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 10 client.234099.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 10 monclient: _renew_subs 2023-10-30T12:18:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63001215d0 con 0x7f63001cab60 2023-10-30T12:18:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.993+0000 7f63083b6640 1 librados: init done 2023-10-30T12:18:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f62f6ffd640 1 -- 172.21.15.50:0/2243632312 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f62f0016d00 con 0x7f63001cab60 2023-10-30T12:18:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f62f6ffd640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62e403a310 0x7f62e403c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f63083b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f63083b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f63083b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f62f6ffd640 1 -- 172.21.15.50:0/2243632312 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(556..556 src has 490..556) v4 ==== 6622+0+0 (secure 0 0 0) 0x7f62f004baf0 con 0x7f63001cab60 2023-10-30T12:18:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f62f6ffd640 10 client.234099.objecter ms_dispatch 0x7f6300000b90 osd_map(556..556 src has 490..556) v4 2023-10-30T12:18:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f62f6ffd640 3 client.234099.objecter handle_osd_map got epochs [556,556] > 0 2023-10-30T12:18:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f62f6ffd640 3 client.234099.objecter handle_osd_map decoding full epoch 556 2023-10-30T12:18:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6300123220 con 0x7f63001cab60 2023-10-30T12:18:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f62f6ffd640 20 client.234099.objecter dump_active .. 0 homeless 2023-10-30T12:18:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.994+0000 7f6306152640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62e403a310 0x7f62e403c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.995+0000 7f6306152640 10 monclient: get_auth_request con 0x7f62e403a310 auth_method 0 2023-10-30T12:18:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.995+0000 7f6306152640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62e403a310 0x7f62e403c7d0 secure :-1 s=READY pgs=4263 cs=0 l=1 rev1=1 crypto rx=0x7f62fc036000 tx=0x7f62fc030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.995+0000 7f62f6ffd640 10 client.234099.objecter ms_handle_connect 0x7f62e403a310 2023-10-30T12:18:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.999+0000 7f62f6ffd640 1 -- 172.21.15.50:0/2243632312 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f62f001b030 con 0x7f63001cab60 2023-10-30T12:18:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.999+0000 7f62f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:26.999+0000 7f62f6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:27.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.251+0000 7f63083b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:27.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.251+0000 7f63083b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:27.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.251+0000 7f63083b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:27.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.251+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6300123ae0 con 0x7f63001cab60 2023-10-30T12:18:27.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.251+0000 7f62f6ffd640 1 -- 172.21.15.50:0/2243632312 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f6300123ae0 con 0x7f63001cab60 2023-10-30T12:18:27.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.251+0000 7f62f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:27.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.251+0000 7f62f6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:27.254 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":182,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:27.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 10 librados: watch_flush enter 2023-10-30T12:18:27.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 10 librados: watch_flush exit 2023-10-30T12:18:27.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 20 client.234099.objecter shutdown clearing up homeless session... 2023-10-30T12:18:27.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 10 client.234099.objecter successfully canceled tick 2023-10-30T12:18:27.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62e403a310 msgr2=0x7f62e403c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:27.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62e403a310 0x7f62e403c7d0 secure :-1 s=READY pgs=4263 cs=0 l=1 rev1=1 crypto rx=0x7f62fc036000 tx=0x7f62fc030000 comp rx=0 tx=0).stop 2023-10-30T12:18:27.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 10 monclient: shutdown 2023-10-30T12:18:27.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:27.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63001cab60 msgr2=0x7f630011fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:27.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63001cab60 0x7f630011fa10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f62f0007890 tx=0x7f62f00017a0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 shutdown_connections 2023-10-30T12:18:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62e403a310 0x7f62e403c7d0 unknown :-1 s=CLOSED pgs=4263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63001caf80 0x7f630011ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63001cab60 0x7f630011fa10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 --2- 172.21.15.50:0/2243632312 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62e4003610 0x7f63001ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 >> 172.21.15.50:0/2243632312 conn(0x7f63000a47f0 msgr2=0x7f6300134890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 shutdown_connections 2023-10-30T12:18:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 -- 172.21.15.50:0/2243632312 wait complete. 2023-10-30T12:18:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:27.253+0000 7f63083b6640 1 librados: shutdown 2023-10-30T12:18:30.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.381+0000 7f3074075640 10 monclient: get_monmap_and_config 2023-10-30T12:18:30.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.381+0000 7f3074075640 10 monclient: build_initial_monmap 2023-10-30T12:18:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.381+0000 7f3074075640 10 monclient: monmap: 2023-10-30T12:18:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:30.382794+0000 2023-10-30T12:18:30.384 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:30.382794+0000 2023-10-30T12:18:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:30.386 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 1 Processor -- start 2023-10-30T12:18:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 1 -- start start 2023-10-30T12:18:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 10 monclient: init 2023-10-30T12:18:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-10-30T12:18:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 10 monclient(hunting): picked mon.noname-a con 0x7f306c09f5b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f306c096080 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.382+0000 7f3074075640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f306c096ba0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3074075640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306c098810 con 0x7f306c096080 2023-10-30T12:18:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3074075640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306c06aef0 con 0x7f306c096ba0 2023-10-30T12:18:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3074075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3074075640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306c09a1a0 con 0x7f306c09f5b0 2023-10-30T12:18:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3074075640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3074075640 10 monclient(hunting): authenticate will time out at 1955.566030s 2023-10-30T12:18:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071610640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f306c096080 0x7f306c096490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59896/0 (socket says 172.21.15.50:59896) 2023-10-30T12:18:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071610640 1 -- 172.21.15.50:0/2746205462 learned_addr learned my addr 172.21.15.50:0/2746205462 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 10 monclient(hunting): get_auth_request con 0x7f306c09f5b0 auth_method 0 2023-10-30T12:18:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.383+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:30.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:30.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 10 monclient(hunting): get_auth_request con 0x7f306c09f5b0 auth_method 0 2023-10-30T12:18:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:30.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:30.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.584+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:18:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 10 monclient(hunting): get_auth_request con 0x7f306c09f5b0 auth_method 0 2023-10-30T12:18:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:30.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:30.985+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:18:31.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:31.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 10 monclient(hunting): get_auth_request con 0x7f306c09f5b0 auth_method 0 2023-10-30T12:18:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:31.786+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:18:32.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.650+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735425788 0 0) 0x7f306c06aef0 con 0x7f306c096ba0 2023-10-30T12:18:32.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.650+0000 7f3070e0f640 10 monclient(hunting): _negotiate 2023-10-30T12:18:32.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.650+0000 7f3070e0f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:32.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.650+0000 7f3070e0f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:32.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.650+0000 7f3070e0f640 10 monclient(hunting): my global_id is 204439 2023-10-30T12:18:32.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.650+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3058003520 con 0x7f306c096ba0 2023-10-30T12:18:32.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 117450908 0 0) 0x7f3058003520 con 0x7f306c096ba0 2023-10-30T12:18:32.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 msgr2=0x7f306c099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:32.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 --2- 172.21.15.50:0/2746205462 >> v2:172.21.15.50:3300/0 conn(0x7f306c09f5b0 0x7f306c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 >> v1:172.21.15.50:6789/0 conn(0x7f306c096080 legacy=0x7f306c096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:32.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: _finish_hunting 0 2023-10-30T12:18:32.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:18:32.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:32.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:18:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c098af0 con 0x7f306c096ba0 2023-10-30T12:18:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: _finish_auth 0 2023-10-30T12:18:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: _check_auth_tickets 2023-10-30T12:18:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f30640019f0 con 0x7f306c096ba0 2023-10-30T12:18:32.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:18:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: dump: 2023-10-30T12:18:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:32.660 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:32.661 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:32.662 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:18:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 >> v1:172.21.15.118:6789/0 conn(0x7f306c096ba0 legacy=0x7f306c13ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3058003970 0x7f3058003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): picked mon.c con 0x7f3058003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30580047c0 0x7f3058004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): picked mon.a con 0x7f30580047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058005380 0x7f305800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): picked mon.b con 0x7f3058005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c09a1a0 con 0x7f30580047c0 2023-10-30T12:18:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305800c210 con 0x7f3058005380 2023-10-30T12:18:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305800c3d0 con 0x7f3058003970 2023-10-30T12:18:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3070e0f640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:18:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30580047c0 0x7f3058004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3058003970 0x7f3058003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 10 monclient(hunting): get_auth_request con 0x7f30580047c0 auth_method 0 2023-10-30T12:18:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:32.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 10 monclient(hunting): get_auth_request con 0x7f3058003970 auth_method 0 2023-10-30T12:18:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:32.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3072612640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058005380 0x7f305800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:32.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071610640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3071e11640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:32.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3072612640 10 monclient(hunting): get_auth_request con 0x7f3058005380 auth_method 0 2023-10-30T12:18:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3072612640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3072612640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3072612640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3072612640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:32.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.651+0000 7f3072612640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3072612640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 10 monclient(hunting): handle_auth_done global_id 204439 payload 293 2023-10-30T12:18:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058005380 msgr2=0x7f305800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058005380 0x7f305800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30580047c0 msgr2=0x7f3058004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30580047c0 0x7f3058004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 10 monclient: _finish_hunting 0 2023-10-30T12:18:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 1 monclient: found mon.c 2023-10-30T12:18:32.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3072612640 1 -- 172.21.15.50:0/2746205462 reap_dead start 2023-10-30T12:18:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 1 -- 172.21.15.50:0/2746205462 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3058003520 con 0x7f3058003970 2023-10-30T12:18:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 10 monclient: _finish_auth 0 2023-10-30T12:18:32.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 10 monclient: _check_auth_tickets 2023-10-30T12:18:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071e11640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3074075640 5 monclient: authenticate success, global_id 204439 2023-10-30T12:18:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3074075640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.652+0000 7f3071610640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30580047c0 0x7f3058004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3071e11640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3058003970 0x7f3058003d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3068011b30 tx=0x7f30580057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4088596ced2da7a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068005990 con 0x7f3058003970 2023-10-30T12:18:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:32.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 10 monclient: dump: 2023-10-30T12:18:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.676 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:32.678 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3074075640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306802f030 con 0x7f3058003970 2023-10-30T12:18:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3074075640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 1 -- 172.21.15.50:0/2746205462 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068005d30 con 0x7f3058003970 2023-10-30T12:18:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:32.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3074075640 4 set_mon_vals no callback set 2023-10-30T12:18:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.653+0000 7f3070e0f640 10 monclient: dump: 2023-10-30T12:18:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:32.680 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:32.682 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:32.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 10 monclient: shutdown 2023-10-30T12:18:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 1 -- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3058003970 msgr2=0x7f3058003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3058003970 0x7f3058003d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3068011b30 tx=0x7f30580057f0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 1 -- 172.21.15.50:0/2746205462 shutdown_connections 2023-10-30T12:18:32.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 1 --2- 172.21.15.50:0/2746205462 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3058003970 0x7f3058003d90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 1 -- 172.21.15.50:0/2746205462 >> 172.21.15.50:0/2746205462 conn(0x7f306c0a05c0 msgr2=0x7f306c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 1 -- 172.21.15.50:0/2746205462 shutdown_connections 2023-10-30T12:18:32.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.654+0000 7f3074075640 1 -- 172.21.15.50:0/2746205462 wait complete. 2023-10-30T12:18:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 10 monclient: build_initial_monmap 2023-10-30T12:18:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 10 monclient: monmap: 2023-10-30T12:18:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:32.690 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:32.691 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:32.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 1 librados: starting msgr at 2023-10-30T12:18:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 1 librados: starting objecter 2023-10-30T12:18:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 1 Processor -- start 2023-10-30T12:18:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 1 -- start start 2023-10-30T12:18:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 1 librados: setting wanted keys 2023-10-30T12:18:32.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 1 librados: calling monclient init 2023-10-30T12:18:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.655+0000 7f3074075640 10 monclient: init 2023-10-30T12:18:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058003970 0x7f306c1caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): picked mon.noname-b con 0x7f3058003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 0x7f306c06f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): picked mon.noname-c con 0x7f306c1caf80 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c1cb3e0 0x7f306c06fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): picked mon.noname-a con 0x7f306c1cb3e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c1b5380 con 0x7f306c1cb3e0 2023-10-30T12:18:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c09c830 con 0x7f3058003970 2023-10-30T12:18:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306c09b370 con 0x7f306c1caf80 2023-10-30T12:18:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3074075640 10 monclient(hunting): authenticate will time out at 1957.839162s 2023-10-30T12:18:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c1cb3e0 0x7f306c06fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c1cb3e0 0x7f306c06fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34138/0 (socket says 172.21.15.50:34138) 2023-10-30T12:18:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 0x7f306c06f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:32.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 -- 172.21.15.50:0/1420461901 learned_addr learned my addr 172.21.15.50:0/1420461901 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): get_auth_request con 0x7f306c1cb3e0 auth_method 0 2023-10-30T12:18:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 10 monclient(hunting): get_auth_request con 0x7f306c1caf80 auth_method 0 2023-10-30T12:18:32.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:32.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 1 -- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 msgr2=0x7f306c06f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:18:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 1 -- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 msgr2=0x7f306c06f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:18:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 0x7f306c06f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071e11640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058003970 0x7f306c1caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:32.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071e11640 10 monclient(hunting): get_auth_request con 0x7f3058003970 auth_method 0 2023-10-30T12:18:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071e11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071e11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071e11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071610640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 0x7f306c06f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:32.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient(hunting): handle_auth_done global_id 234102 payload 824 2023-10-30T12:18:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 -- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 msgr2=0x7f306c06f830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 0x7f306c06f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 -- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058003970 msgr2=0x7f306c1caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058003970 0x7f306c1caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient: _finish_hunting 0 2023-10-30T12:18:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 monclient: found mon.noname-a 2023-10-30T12:18:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 -- 172.21.15.50:0/1420461901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c072010 con 0x7f306c1cb3e0 2023-10-30T12:18:32.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3071e11640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058003970 0x7f306c1caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:18:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f3072612640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c1cb3e0 0x7f306c06fd70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f306c073e10 tx=0x7f3064007f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f30627fc640 10 client.?.objecter ms_handle_connect 0x7f306c1cb3e0 2023-10-30T12:18:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.656+0000 7f30627fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:32.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 1 -- 172.21.15.50:0/1420461901 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064003b80 con 0x7f306c1cb3e0 2023-10-30T12:18:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: dump: 2023-10-30T12:18:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:32.704 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:32.705 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:32.706 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:32.706 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:32.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:32.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:32.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:32.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 1 -- 172.21.15.50:0/1420461901 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064003d00 con 0x7f306c1cb3e0 2023-10-30T12:18:32.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:32.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 1 -- 172.21.15.50:0/1420461901 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306400da40 con 0x7f306c1cb3e0 2023-10-30T12:18:32.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 5 monclient: authenticate success, global_id 234102 2023-10-30T12:18:32.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 10 monclient: _renew_subs 2023-10-30T12:18:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c09c1e0 con 0x7f306c1cb3e0 2023-10-30T12:18:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3063fff640 4 set_mon_vals no callback set 2023-10-30T12:18:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:32.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f30627fc640 10 monclient: dump: 2023-10-30T12:18:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:32.710 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:32.711 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 10 client.234102.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 10 monclient: _renew_subs 2023-10-30T12:18:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:32.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306c1b1ae0 con 0x7f306c1cb3e0 2023-10-30T12:18:32.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.666+0000 7f3074075640 1 librados: init done 2023-10-30T12:18:32.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f30627fc640 1 -- 172.21.15.50:0/1420461901 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f306400dba0 con 0x7f306c1cb3e0 2023-10-30T12:18:32.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f30627fc640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3050039370 0x7f305003b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:32.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f30627fc640 1 -- 172.21.15.50:0/1420461901 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(556..556 src has 490..556) v4 ==== 6622+0+0 (secure 0 0 0) 0x7f306404b0c0 con 0x7f306c1cb3e0 2023-10-30T12:18:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f30627fc640 10 client.234102.objecter ms_dispatch 0x7f306c000b90 osd_map(556..556 src has 490..556) v4 2023-10-30T12:18:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f30627fc640 3 client.234102.objecter handle_osd_map got epochs [556,556] > 0 2023-10-30T12:18:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f30627fc640 3 client.234102.objecter handle_osd_map decoding full epoch 556 2023-10-30T12:18:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f3074075640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.667+0000 7f3074075640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.668+0000 7f3074075640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.668+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c04e160 con 0x7f306c1cb3e0 2023-10-30T12:18:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.668+0000 7f30627fc640 20 client.234102.objecter dump_active .. 0 homeless 2023-10-30T12:18:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.668+0000 7f3071e11640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3050039370 0x7f305003b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:32.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.668+0000 7f3071e11640 10 monclient: get_auth_request con 0x7f3050039370 auth_method 0 2023-10-30T12:18:32.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.668+0000 7f30627fc640 10 client.234102.objecter ms_handle_connect 0x7f3050039370 2023-10-30T12:18:32.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.668+0000 7f3071e11640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3050039370 0x7f305003b830 secure :-1 s=READY pgs=4264 cs=0 l=1 rev1=1 crypto rx=0x7f3068036000 tx=0x7f3068030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:32.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.673+0000 7f30627fc640 1 -- 172.21.15.50:0/1420461901 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f306401b950 con 0x7f306c1cb3e0 2023-10-30T12:18:32.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.673+0000 7f30627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:32.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.673+0000 7f30627fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:32.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.920+0000 7f3074075640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:32.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.920+0000 7f3074075640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:32.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.920+0000 7f3074075640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:32.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.920+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f306c0a2b00 con 0x7f306c1cb3e0 2023-10-30T12:18:32.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.920+0000 7f30627fc640 1 -- 172.21.15.50:0/1420461901 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f306401b360 con 0x7f306c1cb3e0 2023-10-30T12:18:32.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.921+0000 7f30627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:32.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.921+0000 7f30627fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:32.924 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":186,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:32.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 10 librados: watch_flush enter 2023-10-30T12:18:32.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 10 librados: watch_flush exit 2023-10-30T12:18:32.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 20 client.234102.objecter shutdown clearing up homeless session... 2023-10-30T12:18:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 10 client.234102.objecter successfully canceled tick 2023-10-30T12:18:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3050039370 msgr2=0x7f305003b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3050039370 0x7f305003b830 secure :-1 s=READY pgs=4264 cs=0 l=1 rev1=1 crypto rx=0x7f3068036000 tx=0x7f3068030000 comp rx=0 tx=0).stop 2023-10-30T12:18:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 10 monclient: shutdown 2023-10-30T12:18:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c1cb3e0 msgr2=0x7f306c06fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c1cb3e0 0x7f306c06fd70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f306c073e10 tx=0x7f3064007f80 comp rx=0 tx=0).stop 2023-10-30T12:18:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 shutdown_connections 2023-10-30T12:18:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3050039370 0x7f305003b830 unknown :-1 s=CLOSED pgs=4264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f306c1cb3e0 0x7f306c06fd70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f306c1caf80 0x7f306c06f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 --2- 172.21.15.50:0/1420461901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3058003970 0x7f306c1caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:32.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 >> 172.21.15.50:0/1420461901 conn(0x7f306c0a05c0 msgr2=0x7f306c1b9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:32.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 shutdown_connections 2023-10-30T12:18:32.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 -- 172.21.15.50:0/1420461901 wait complete. 2023-10-30T12:18:32.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:32.922+0000 7f3074075640 1 librados: shutdown 2023-10-30T12:18:35.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:36.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.051+0000 7ff08d887640 10 monclient: get_monmap_and_config 2023-10-30T12:18:36.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.051+0000 7ff08d887640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.051+0000 7ff08d887640 10 monclient: monmap: 2023-10-30T12:18:36.054 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.054 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.054 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:36.052908+0000 2023-10-30T12:18:36.055 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:36.052908+0000 2023-10-30T12:18:36.055 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.055 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.055 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:36.056 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:36.056 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:36.056 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:36.056 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:36.056 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 Processor -- start 2023-10-30T12:18:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 -- start start 2023-10-30T12:18:36.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient: init 2023-10-30T12:18:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:18:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff0880a0080 0x7ff0880a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): picked mon.noname-a con 0x7ff0880a0080 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff0880720b0 0x7ff0880a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): picked mon.noname-b con 0x7ff0880720b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff088073a30 0x7ff088097320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): picked mon.noname-c con 0x7ff088073a30 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff088073620 con 0x7ff0880a0080 2023-10-30T12:18:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08809ccc0 con 0x7ff088073a30 2023-10-30T12:18:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880a0ee0 con 0x7ff0880720b0 2023-10-30T12:18:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.052+0000 7ff08d887640 10 monclient(hunting): authenticate will time out at 1961.235962s 2023-10-30T12:18:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff0880a0080 0x7ff0880a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff0880720b0 0x7ff0880a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff0880a0080 0x7ff0880a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34154/0 (socket says 172.21.15.50:34154) 2023-10-30T12:18:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff0880720b0 0x7ff0880a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40772/0 (socket says 172.21.15.50:40772) 2023-10-30T12:18:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 1 -- 172.21.15.50:0/3217102979 learned_addr learned my addr 172.21.15.50:0/3217102979 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 10 monclient(hunting): get_auth_request con 0x7ff0880a0080 auth_method 0 2023-10-30T12:18:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): get_auth_request con 0x7ff0880720b0 auth_method 0 2023-10-30T12:18:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 1 --2- 172.21.15.50:0/3217102979 >> v2:172.21.15.118:3300/0 conn(0x7ff088073a30 0x7ff088097320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 10 monclient(hunting): get_auth_request con 0x7ff088073a30 auth_method 0 2023-10-30T12:18:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient(hunting): handle_auth_done global_id 224183 payload 274 2023-10-30T12:18:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 -- 172.21.15.50:0/3217102979 >> v2:172.21.15.118:3300/0 conn(0x7ff088073a30 msgr2=0x7ff088097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 --2- 172.21.15.50:0/3217102979 >> v2:172.21.15.118:3300/0 conn(0x7ff088073a30 0x7ff088097320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 -- 172.21.15.50:0/3217102979 >> v2:172.21.15.50:3300/0 conn(0x7ff0880a0080 msgr2=0x7ff0880a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 --2- 172.21.15.50:0/3217102979 >> v2:172.21.15.50:3300/0 conn(0x7ff0880a0080 0x7ff0880a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 monclient: found mon.noname-b 2023-10-30T12:18:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff0867fc640 1 -- 172.21.15.50:0/3217102979 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088071e90 con 0x7ff0880720b0 2023-10-30T12:18:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.053+0000 7ff086ffd640 1 --2- 172.21.15.50:0/3217102979 >> v2:172.21.15.50:3300/0 conn(0x7ff0880a0080 0x7ff0880a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff0867fc640 1 --2- 172.21.15.50:0/3217102979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff0880720b0 0x7ff0880a09a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff08806d0f0 tx=0x7ff07c02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffbed8ef96315dd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 1 -- 172.21.15.50:0/3217102979 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c030710 con 0x7ff0880720b0 2023-10-30T12:18:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: dump: 2023-10-30T12:18:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.080 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 1 -- 172.21.15.50:0/3217102979 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c02e050 con 0x7ff0880720b0 2023-10-30T12:18:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 1 -- 172.21.15.50:0/3217102979 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c030a90 con 0x7ff0880720b0 2023-10-30T12:18:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff08d887640 5 monclient: authenticate success, global_id 224183 2023-10-30T12:18:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff08d887640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff08d887640 4 set_mon_vals no callback set 2023-10-30T12:18:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.054+0000 7ff085ffb640 10 monclient: dump: 2023-10-30T12:18:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.083 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.084 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 10 monclient: shutdown 2023-10-30T12:18:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 -- 172.21.15.50:0/3217102979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff0880720b0 msgr2=0x7ff0880a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 --2- 172.21.15.50:0/3217102979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff0880720b0 0x7ff0880a09a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff08806d0f0 tx=0x7ff07c02b710 comp rx=0 tx=0).stop 2023-10-30T12:18:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 -- 172.21.15.50:0/3217102979 shutdown_connections 2023-10-30T12:18:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 --2- 172.21.15.50:0/3217102979 >> v2:172.21.15.118:3300/0 conn(0x7ff088073a30 0x7ff088097320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 --2- 172.21.15.50:0/3217102979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff0880720b0 0x7ff0880a09a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 --2- 172.21.15.50:0/3217102979 >> v2:172.21.15.50:3300/0 conn(0x7ff0880a0080 0x7ff0880a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 -- 172.21.15.50:0/3217102979 >> 172.21.15.50:0/3217102979 conn(0x7ff0880a4870 msgr2=0x7ff0880a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 -- 172.21.15.50:0/3217102979 shutdown_connections 2023-10-30T12:18:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 -- 172.21.15.50:0/3217102979 wait complete. 2023-10-30T12:18:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 10 monclient: monmap: 2023-10-30T12:18:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:36.088 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 librados: starting msgr at 2023-10-30T12:18:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.055+0000 7ff08d887640 1 librados: starting objecter 2023-10-30T12:18:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 Processor -- start 2023-10-30T12:18:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 -- start start 2023-10-30T12:18:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 librados: setting wanted keys 2023-10-30T12:18:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 librados: calling monclient init 2023-10-30T12:18:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient: init 2023-10-30T12:18:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:18:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff088073a30 0x7ff0881ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): picked mon.noname-c con 0x7ff088073a30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0880a0080 0x7ff0880943c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): picked mon.noname-b con 0x7ff0880a0080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff088094900 0x7ff08808ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): picked mon.noname-a con 0x7ff088094900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880908b0 con 0x7ff088094900 2023-10-30T12:18:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880905d0 con 0x7ff0880a0080 2023-10-30T12:18:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff088073e90 con 0x7ff088073a30 2023-10-30T12:18:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff08d887640 10 monclient(hunting): authenticate will time out at 1961.239821s 2023-10-30T12:18:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff086ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff088073a30 0x7ff0881ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff086ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff088073a30 0x7ff0881ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40778/0 (socket says 172.21.15.50:40778) 2023-10-30T12:18:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff086ffd640 1 -- 172.21.15.50:0/1964158013 learned_addr learned my addr 172.21.15.50:0/1964158013 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff086ffd640 10 monclient(hunting): get_auth_request con 0x7ff088073a30 auth_method 0 2023-10-30T12:18:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff086ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff086ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff0877fe640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff088094900 0x7ff08808ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff0877fe640 10 monclient(hunting): get_auth_request con 0x7ff088094900 auth_method 0 2023-10-30T12:18:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff0877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff0877fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff0877fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.056+0000 7ff0867fc640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0880a0080 0x7ff0880943c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0867fc640 10 monclient(hunting): get_auth_request con 0x7ff0880a0080 auth_method 0 2023-10-30T12:18:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0867fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0867fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 10 monclient(hunting): handle_auth_done global_id 224186 payload 824 2023-10-30T12:18:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 1 -- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0880a0080 msgr2=0x7ff0880943c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0880a0080 0x7ff0880943c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 1 -- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff088094900 msgr2=0x7ff08808ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff088094900 0x7ff08808ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 1 monclient: found mon.noname-c 2023-10-30T12:18:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 1 -- 172.21.15.50:0/1964158013 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078009460 con 0x7ff088073a30 2023-10-30T12:18:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff086ffd640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff088073a30 0x7ff0881ca1d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff07800e9e0 tx=0x7ff07800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 client.?.objecter ms_handle_connect 0x7ff088073a30 2023-10-30T12:18:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:36.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 1 -- 172.21.15.50:0/1964158013 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800ca30 con 0x7ff088073a30 2023-10-30T12:18:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: dump: 2023-10-30T12:18:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.101 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.103 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 1 -- 172.21.15.50:0/1964158013 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07800cb90 con 0x7ff088073a30 2023-10-30T12:18:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 5 monclient: authenticate success, global_id 224186 2023-10-30T12:18:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 10 monclient: _renew_subs 2023-10-30T12:18:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c009400 con 0x7ff088073a30 2023-10-30T12:18:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 1 -- 172.21.15.50:0/1964158013 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078005990 con 0x7ff088073a30 2023-10-30T12:18:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:36.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff06ffff640 10 monclient: dump: 2023-10-30T12:18:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.107 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff0857fa640 4 set_mon_vals no callback set 2023-10-30T12:18:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 10 client.224186.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 10 monclient: _renew_subs 2023-10-30T12:18:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088090e70 con 0x7ff088073a30 2023-10-30T12:18:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.057+0000 7ff08d887640 1 librados: init done 2023-10-30T12:18:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff06ffff640 1 -- 172.21.15.50:0/1964158013 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff078005af0 con 0x7ff088073a30 2023-10-30T12:18:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff08d887640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff08d887640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff08d887640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08804e160 con 0x7ff088073a30 2023-10-30T12:18:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff06ffff640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff0740392f0 0x7ff07403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff06ffff640 1 -- 172.21.15.50:0/1964158013 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(558..558 src has 490..558) v4 ==== 6734+0+0 (secure 0 0 0) 0x7ff07804b270 con 0x7ff088073a30 2023-10-30T12:18:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff06ffff640 10 client.224186.objecter ms_dispatch 0x7ff088000b90 osd_map(558..558 src has 490..558) v4 2023-10-30T12:18:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff06ffff640 3 client.224186.objecter handle_osd_map got epochs [558,558] > 0 2023-10-30T12:18:36.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff06ffff640 3 client.224186.objecter handle_osd_map decoding full epoch 558 2023-10-30T12:18:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff06ffff640 20 client.224186.objecter dump_active .. 0 homeless 2023-10-30T12:18:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff0867fc640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff0740392f0 0x7ff07403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.058+0000 7ff0867fc640 10 monclient: get_auth_request con 0x7ff0740392f0 auth_method 0 2023-10-30T12:18:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.059+0000 7ff06ffff640 10 client.224186.objecter ms_handle_connect 0x7ff0740392f0 2023-10-30T12:18:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.059+0000 7ff0867fc640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff0740392f0 0x7ff07403b7b0 secure :-1 s=READY pgs=4265 cs=0 l=1 rev1=1 crypto rx=0x7ff07c034000 tx=0x7ff07c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.063+0000 7ff06ffff640 1 -- 172.21.15.50:0/1964158013 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff078006550 con 0x7ff088073a30 2023-10-30T12:18:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.063+0000 7ff06ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.063+0000 7ff06ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:36.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.307+0000 7ff08d887640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.307+0000 7ff08d887640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.307+0000 7ff08d887640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.307+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff08809c310 con 0x7ff088073a30 2023-10-30T12:18:36.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.307+0000 7ff06ffff640 1 -- 172.21.15.50:0/1964158013 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7ff078011350 con 0x7ff088073a30 2023-10-30T12:18:36.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.307+0000 7ff06ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:36.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.307+0000 7ff06ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:36.310 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 10 librados: watch_flush enter 2023-10-30T12:18:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 10 librados: watch_flush exit 2023-10-30T12:18:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 20 client.224186.objecter shutdown clearing up homeless session... 2023-10-30T12:18:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 10 client.224186.objecter successfully canceled tick 2023-10-30T12:18:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff0740392f0 msgr2=0x7ff07403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff0740392f0 0x7ff07403b7b0 secure :-1 s=READY pgs=4265 cs=0 l=1 rev1=1 crypto rx=0x7ff07c034000 tx=0x7ff07c02f000 comp rx=0 tx=0).stop 2023-10-30T12:18:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 10 monclient: shutdown 2023-10-30T12:18:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff088073a30 msgr2=0x7ff0881ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff088073a30 0x7ff0881ca1d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff07800e9e0 tx=0x7ff07800eea0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 shutdown_connections 2023-10-30T12:18:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff0740392f0 0x7ff07403b7b0 unknown :-1 s=CLOSED pgs=4265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff088094900 0x7ff08808ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0880a0080 0x7ff0880943c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 --2- 172.21.15.50:0/1964158013 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff088073a30 0x7ff0881ca1d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 >> 172.21.15.50:0/1964158013 conn(0x7ff0880a4870 msgr2=0x7ff0881bd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 shutdown_connections 2023-10-30T12:18:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 -- 172.21.15.50:0/1964158013 wait complete. 2023-10-30T12:18:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.309+0000 7ff08d887640 1 librados: shutdown 2023-10-30T12:18:36.321 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:18:36.321 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:18:36.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:18:36.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.426+0000 7f5a93ed7640 10 monclient: get_monmap_and_config 2023-10-30T12:18:36.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.426+0000 7f5a93ed7640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.426+0000 7f5a93ed7640 10 monclient: monmap: 2023-10-30T12:18:36.429 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.429 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.429 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:36.427812+0000 2023-10-30T12:18:36.429 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:36.427812+0000 2023-10-30T12:18:36.429 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.430 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.430 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:36.430 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:36.430 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:36.431 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:36.431 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:36.431 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 1 Processor -- start 2023-10-30T12:18:36.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 1 -- start start 2023-10-30T12:18:36.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient: init 2023-10-30T12:18:36.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:18:36.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a8c097780 0x7f5a8c097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8c097780 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:36.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5a8c073b10 0x7f5a8c0a0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8c073b10 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:36.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a8c09ccd0 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:36.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c071ec0 con 0x7f5a8c09ccd0 2023-10-30T12:18:36.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8c09bd10 con 0x7f5a8c097780 2023-10-30T12:18:36.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8c09bff0 con 0x7f5a8c073b10 2023-10-30T12:18:36.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.427+0000 7f5a93ed7640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a93ed7640 10 monclient(hunting): authenticate will time out at 1961.611013s 2023-10-30T12:18:36.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a8c097780 0x7f5a8c097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a92474640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5a8c09ccd0 0x7f5a8c0a05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59908/0 (socket says 172.21.15.50:59908) 2023-10-30T12:18:36.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): get_auth_request con 0x7f5a8c097780 auth_method 0 2023-10-30T12:18:36.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a92474640 1 -- 172.21.15.50:0/2791931520 learned_addr learned my addr 172.21.15.50:0/2791931520 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:36.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 1 --2- 172.21.15.50:0/2791931520 >> v2:172.21.15.118:3300/0 conn(0x7f5a8c073b10 0x7f5a8c0a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 10 monclient(hunting): get_auth_request con 0x7f5a8c073b10 auth_method 0 2023-10-30T12:18:36.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient(hunting): handle_auth_done global_id 234111 payload 274 2023-10-30T12:18:36.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 1 -- 172.21.15.50:0/2791931520 >> v2:172.21.15.118:3300/0 conn(0x7f5a8c073b10 msgr2=0x7f5a8c0a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:36.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 1 --2- 172.21.15.50:0/2791931520 >> v2:172.21.15.118:3300/0 conn(0x7f5a8c073b10 0x7f5a8c0a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 1 -- 172.21.15.50:0/2791931520 >> v1:172.21.15.50:6789/0 conn(0x7f5a8c09ccd0 legacy=0x7f5a8c0a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a90c71640 1 -- 172.21.15.50:0/2791931520 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186654860 0 0) 0x7f5a8c071ec0 con 0x7f5a8c09ccd0 2023-10-30T12:18:36.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 1 monclient: found mon.noname-a 2023-10-30T12:18:36.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a91c73640 1 -- 172.21.15.50:0/2791931520 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0980a0 con 0x7f5a8c097780 2023-10-30T12:18:36.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.428+0000 7f5a90c71640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:18:36.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a91c73640 1 --2- 172.21.15.50:0/2791931520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c097b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c06a230 tx=0x7f5a8802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336b4ae1c456035f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 1 -- 172.21.15.50:0/2791931520 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88031710 con 0x7f5a8c097780 2023-10-30T12:18:36.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:36.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: dump: 2023-10-30T12:18:36.452 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.452 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.452 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.452 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.452 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.453 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.453 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.453 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.453 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.453 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 1 -- 172.21.15.50:0/2791931520 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8802f050 con 0x7f5a8c097780 2023-10-30T12:18:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 5 monclient: authenticate success, global_id 234111 2023-10-30T12:18:36.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:36.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:36.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 4 set_mon_vals no callback set 2023-10-30T12:18:36.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 1 -- 172.21.15.50:0/2791931520 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88031a90 con 0x7f5a8c097780 2023-10-30T12:18:36.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:36.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a90c71640 10 monclient: dump: 2023-10-30T12:18:36.456 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.456 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.457 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.457 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.457 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.457 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.457 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.458 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.458 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.458 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 10 monclient: shutdown 2023-10-30T12:18:36.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 1 -- 172.21.15.50:0/2791931520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 msgr2=0x7f5a8c097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 1 --2- 172.21.15.50:0/2791931520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c097b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c06a230 tx=0x7f5a8802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 1 -- 172.21.15.50:0/2791931520 shutdown_connections 2023-10-30T12:18:36.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 1 --2- 172.21.15.50:0/2791931520 >> v2:172.21.15.118:3300/0 conn(0x7f5a8c073b10 0x7f5a8c0a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 1 --2- 172.21.15.50:0/2791931520 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c097b60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.429+0000 7f5a93ed7640 1 -- 172.21.15.50:0/2791931520 >> 172.21.15.50:0/2791931520 conn(0x7f5a8c0a1170 msgr2=0x7f5a8c0a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 -- 172.21.15.50:0/2791931520 shutdown_connections 2023-10-30T12:18:36.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 -- 172.21.15.50:0/2791931520 wait complete. 2023-10-30T12:18:36.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 10 monclient: monmap: 2023-10-30T12:18:36.461 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.461 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:36.461 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:36.461 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:36.461 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.462 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.462 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:36.462 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:36.462 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:36.463 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 librados: starting msgr at 2023-10-30T12:18:36.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 librados: starting objecter 2023-10-30T12:18:36.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 Processor -- start 2023-10-30T12:18:36.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 -- start start 2023-10-30T12:18:36.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 librados: setting wanted keys 2023-10-30T12:18:36.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 1 librados: calling monclient init 2023-10-30T12:18:36.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.430+0000 7f5a93ed7640 10 monclient: init 2023-10-30T12:18:36.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:36.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a8c073b10 0x7f5a8c1c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): picked mon.noname-b con 0x7f5a8c073b10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:36.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c095770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8c097780 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:36.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a8c09ccd0 0x7f5a8c1c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8c09ccd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:36.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c093330 con 0x7f5a8c097780 2023-10-30T12:18:36.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c090670 con 0x7f5a8c073b10 2023-10-30T12:18:36.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c092990 con 0x7f5a8c09ccd0 2023-10-30T12:18:36.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a93ed7640 10 monclient(hunting): authenticate will time out at 1961.614321s 2023-10-30T12:18:36.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c095770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c095770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34192/0 (socket says 172.21.15.50:34192) 2023-10-30T12:18:36.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a8c09ccd0 0x7f5a8c1c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 -- 172.21.15.50:0/868464530 learned_addr learned my addr 172.21.15.50:0/868464530 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:36.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): get_auth_request con 0x7f5a8c097780 auth_method 0 2023-10-30T12:18:36.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 10 monclient(hunting): get_auth_request con 0x7f5a8c09ccd0 auth_method 0 2023-10-30T12:18:36.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91c73640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a8c073b10 0x7f5a8c1c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a92474640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91c73640 10 monclient(hunting): get_auth_request con 0x7f5a8c073b10 auth_method 0 2023-10-30T12:18:36.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91c73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91c73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91c73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient(hunting): handle_auth_done global_id 234117 payload 824 2023-10-30T12:18:36.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a8c09ccd0 msgr2=0x7f5a8c1c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a8c09ccd0 0x7f5a8c1c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a8c073b10 msgr2=0x7f5a8c1c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a8c073b10 0x7f5a8c1c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 monclient: found mon.noname-a 2023-10-30T12:18:36.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 -- 172.21.15.50:0/868464530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a88009400 con 0x7f5a8c097780 2023-10-30T12:18:36.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91c73640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a8c073b10 0x7f5a8c1c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:18:36.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a91472640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c095770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00e9b0 tx=0x7f5a7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a827fc640 10 client.?.objecter ms_handle_connect 0x7f5a8c097780 2023-10-30T12:18:36.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.431+0000 7f5a827fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:36.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c009670 con 0x7f5a8c097780 2023-10-30T12:18:36.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:36.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: dump: 2023-10-30T12:18:36.476 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.476 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.477 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.477 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.477 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.477 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.477 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.478 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.478 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.478 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 5 monclient: authenticate success, global_id 234117 2023-10-30T12:18:36.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00c9a0 con 0x7f5a8c097780 2023-10-30T12:18:36.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 10 monclient: _renew_subs 2023-10-30T12:18:36.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00cd50 con 0x7f5a8c097780 2023-10-30T12:18:36.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c092c70 con 0x7f5a8c097780 2023-10-30T12:18:36.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:36.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a827fc640 10 monclient: dump: 2023-10-30T12:18:36.481 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.481 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.481 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.481 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.482 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.482 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.482 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.482 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.482 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.483 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a83fff640 4 set_mon_vals no callback set 2023-10-30T12:18:36.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 10 client.234117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:36.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 10 monclient: _renew_subs 2023-10-30T12:18:36.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c091aa0 con 0x7f5a8c097780 2023-10-30T12:18:36.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.432+0000 7f5a93ed7640 1 librados: init done 2023-10-30T12:18:36.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a93ed7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a93ed7640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c091de0 0x7f5a8c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a93ed7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c06a5a0 con 0x7f5a8c091de0 2023-10-30T12:18:36.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5a7c013470 con 0x7f5a8c097780 2023-10-30T12:18:36.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c091de0 0x7f5a8c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient: get_auth_request con 0x7f5a8c091de0 auth_method 0 2023-10-30T12:18:36.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a700392f0 0x7f5a7003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(558..558 src has 490..558) v4 ==== 6734+0+0 (secure 0 0 0) 0x7f5a7c04b1b0 con 0x7f5a8c097780 2023-10-30T12:18:36.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 10 client.234117.objecter ms_dispatch 0x7f5a8c000b90 osd_map(558..558 src has 490..558) v4 2023-10-30T12:18:36.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 3 client.234117.objecter handle_osd_map got epochs [558,558] > 0 2023-10-30T12:18:36.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 3 client.234117.objecter handle_osd_map decoding full epoch 558 2023-10-30T12:18:36.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 10 monclient(hunting): handle_auth_done global_id 234120 payload 824 2023-10-30T12:18:36.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 1 -- 172.21.15.50:0/868464530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a8c0a0cd0 con 0x7f5a8c091de0 2023-10-30T12:18:36.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 20 client.234117.objecter dump_active .. 0 homeless 2023-10-30T12:18:36.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 10 client.234117.objecter ms_handle_connect 0x7f5a8c091de0 2023-10-30T12:18:36.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 10 client.234117.objecter resend_mon_ops 2023-10-30T12:18:36.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a91c73640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c091de0 0x7f5a8c02b430 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c09e770 tx=0x7f5a880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a92474640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a700392f0 0x7f5a7003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a92474640 10 monclient: get_auth_request con 0x7f5a700392f0 auth_method 0 2023-10-30T12:18:36.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88033950 con 0x7f5a8c091de0 2023-10-30T12:18:36.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:36.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.433+0000 7f5a827fc640 10 monclient: dump: 2023-10-30T12:18:36.490 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.491 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.491 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.491 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.491 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.492 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.492 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.492 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.492 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.492 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5a88033ab0 con 0x7f5a8c091de0 2023-10-30T12:18:36.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:36.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c091de0 msgr2=0x7f5a8c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c091de0 0x7f5a8c02b430 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c09e770 tx=0x7f5a880046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c091de0 msgr2=0x7f5a8c02b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c091de0 0x7f5a8c02b430 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c09e770 tx=0x7f5a880046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a827fc640 10 client.234117.objecter ms_handle_connect 0x7f5a700392f0 2023-10-30T12:18:36.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.434+0000 7f5a92474640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a700392f0 0x7f5a7003b7b0 secure :-1 s=READY pgs=4266 cs=0 l=1 rev1=1 crypto rx=0x7f5a840012e0 tx=0x7f5a840016b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a93ed7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:36.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a93ed7640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c02b970 0x7f5a8c02bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a93ed7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c04e160 con 0x7f5a8c02b970 2023-10-30T12:18:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c02b970 0x7f5a8c02bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient: get_auth_request con 0x7f5a8c02b970 auth_method 0 2023-10-30T12:18:36.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 10 monclient(hunting): handle_auth_done global_id 234123 payload 824 2023-10-30T12:18:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 1 -- 172.21.15.50:0/868464530 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5a8c08c6e0 con 0x7f5a8c02b970 2023-10-30T12:18:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a827fc640 10 client.234117.objecter ms_handle_connect 0x7f5a8c02b970 2023-10-30T12:18:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a827fc640 10 client.234117.objecter resend_mon_ops 2023-10-30T12:18:36.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a91c73640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c02b970 0x7f5a8c02bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5a8802b7f0 tx=0x7f5a88009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.440+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88055070 con 0x7f5a8c02b970 2023-10-30T12:18:36.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:36.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 10 monclient: dump: 2023-10-30T12:18:36.501 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.501 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.501 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.501 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.501 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.502 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.502 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.502 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.502 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.502 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f5a8803d440 con 0x7f5a8c02b970 2023-10-30T12:18:36.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:36.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:36.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c02b970 msgr2=0x7f5a8c02bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c02b970 0x7f5a8c02bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5a8802b7f0 tx=0x7f5a88009190 comp rx=0 tx=0).stop 2023-10-30T12:18:36.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c02b970 msgr2=0x7f5a8c02bd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:36.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a827fc640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c02b970 0x7f5a8c02bd50 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5a8802b7f0 tx=0x7f5a88009190 comp rx=0 tx=0).stop 2023-10-30T12:18:36.504 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:36.505 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:36.505 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:36.505 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:18:36.505 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 190, 2023-10-30T12:18:36.505 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:36.506 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:18:36.506 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:36.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:36.506 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.507 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:18:36.507 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:36.507 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:36.507 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:36.507 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.508 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.508 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.508 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.508 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.508 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.509 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.509 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.509 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.509 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.509 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.510 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:36.510 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:36.510 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.510 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.511 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.511 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.511 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.511 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.511 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.512 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.512 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.512 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.512 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.513 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:36.513 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:36.513 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:36.513 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:36.513 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:36.514 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:36.514 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:36.514 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:36.514 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:36.514 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:36.515 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:36.515 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:36.515 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:36.515 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:36.516 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:36.516 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:36.516 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:36.516 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.516 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.517 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.517 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.517 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.517 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.517 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.518 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.518 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.518 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.518 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.518 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:36.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.519 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:36.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.519 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:36.520 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:36.520 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.520 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.521 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:36.521 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.521 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.522 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:36.522 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.522 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.523 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:36.523 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:36.523 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.523 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.523 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.524 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.525 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.525 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:36.525 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.526 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.526 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:36.526 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.527 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:36.527 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:36.527 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.528 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.528 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.528 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:36.529 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:36.529 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.529 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.529 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.530 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:36.530 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.530 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.531 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:36.531 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.531 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.531 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.532 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:36.532 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:36.532 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.532 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.532 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.533 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.533 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:36.533 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:36.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.534 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.534 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.534 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:36.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.535 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.535 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:36.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.535 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.535 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.536 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:18:36.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.536 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.536 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:36.537 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 10 librados: watch_flush enter 2023-10-30T12:18:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 10 librados: watch_flush exit 2023-10-30T12:18:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 20 client.234117.objecter shutdown clearing up homeless session... 2023-10-30T12:18:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 10 client.234117.objecter successfully canceled tick 2023-10-30T12:18:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a700392f0 msgr2=0x7f5a7003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a700392f0 0x7f5a7003b7b0 secure :-1 s=READY pgs=4266 cs=0 l=1 rev1=1 crypto rx=0x7f5a840012e0 tx=0x7f5a840016b0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 10 monclient: shutdown 2023-10-30T12:18:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 msgr2=0x7f5a8c095770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 --2- 172.21.15.50:0/868464530 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a8c097780 0x7f5a8c095770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00e9b0 tx=0x7f5a7c00ee70 comp rx=0 tx=0).stop 2023-10-30T12:18:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a92474640 1 -- 172.21.15.50:0/868464530 reap_dead start 2023-10-30T12:18:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a92474640 1 -- 172.21.15.50:0/868464530 reap_dead start 2023-10-30T12:18:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 shutdown_connections 2023-10-30T12:18:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 >> 172.21.15.50:0/868464530 conn(0x7f5a8c0a1170 msgr2=0x7f5a8c09af50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 shutdown_connections 2023-10-30T12:18:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 -- 172.21.15.50:0/868464530 wait complete. 2023-10-30T12:18:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.441+0000 7f5a93ed7640 1 librados: shutdown 2023-10-30T12:18:36.542 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:18:36.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:18:36.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:18:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.648+0000 7f6593d53640 10 monclient: get_monmap_and_config 2023-10-30T12:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.648+0000 7f6593d53640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.648+0000 7f6593d53640 10 monclient: monmap: 2023-10-30T12:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:36.649282+0000 2023-10-30T12:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:36.649282+0000 2023-10-30T12:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:36.652 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:36.652 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:36.652 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:36.652 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.648+0000 7f6593d53640 1 Processor -- start 2023-10-30T12:18:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 1 -- start start 2023-10-30T12:18:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient: init 2023-10-30T12:18:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-10-30T12:18:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f658c0a2ae0 0x7f658c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient(hunting): picked mon.noname-c con 0x7f658c0a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f658c0a2ec0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f658c135df0 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c0a4190 con 0x7f658c135df0 2023-10-30T12:18:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c09a2e0 con 0x7f658c0a2ec0 2023-10-30T12:18:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658c0a2990 con 0x7f658c0a2ae0 2023-10-30T12:18:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6593d53640 10 monclient(hunting): authenticate will time out at 1961.832603s 2023-10-30T12:18:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f65922f0640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f658c135df0 0x7f658c13da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59916/0 (socket says 172.21.15.50:59916) 2023-10-30T12:18:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f65922f0640 1 -- 172.21.15.50:0/3821817618 learned_addr learned my addr 172.21.15.50:0/3821817618 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 1 --2- 172.21.15.50:0/3821817618 >> v2:172.21.15.118:3300/0 conn(0x7f658c0a2ae0 0x7f658c0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 10 monclient(hunting): get_auth_request con 0x7f658c0a2ae0 auth_method 0 2023-10-30T12:18:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6590aed640 1 -- 172.21.15.50:0/3821817618 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204750214 0 0) 0x7f6584001490 con 0x7f658c135df0 2023-10-30T12:18:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6590aed640 10 monclient(hunting): _negotiate 2023-10-30T12:18:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6590aed640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6590aed640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.649+0000 7f6590aed640 10 monclient(hunting): my global_id is 234126 2023-10-30T12:18:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6590aed640 1 -- 172.21.15.50:0/3821817618 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578003520 con 0x7f658c135df0 2023-10-30T12:18:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 10 monclient(hunting): handle_auth_done global_id 204457 payload 274 2023-10-30T12:18:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 1 -- 172.21.15.50:0/3821817618 >> v1:172.21.15.118:6789/0 conn(0x7f658c0a2ec0 legacy=0x7f658c1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 1 -- 172.21.15.50:0/3821817618 >> v1:172.21.15.50:6789/0 conn(0x7f658c135df0 legacy=0x7f658c13da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6590aed640 1 -- 172.21.15.50:0/3821817618 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1593587839 0 0) 0x7f6578003520 con 0x7f658c135df0 2023-10-30T12:18:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 1 monclient: found mon.noname-c 2023-10-30T12:18:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:18:36.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 1 -- 172.21.15.50:0/3821817618 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c09a8a0 con 0x7f658c0a2ae0 2023-10-30T12:18:36.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6590aed640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:18:36.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.650+0000 7f6591aef640 1 --2- 172.21.15.50:0/3821817618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c0a2ae0 0x7f658c0a2450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f658c09a470 tx=0x7f658802b6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd2ec97540a07dc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 1 -- 172.21.15.50:0/3821817618 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584001b80 con 0x7f658c0a2ae0 2023-10-30T12:18:36.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: dump: 2023-10-30T12:18:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.671 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.671 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.671 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.671 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.671 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.672 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.672 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 1 -- 172.21.15.50:0/3821817618 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658802e050 con 0x7f658c0a2ae0 2023-10-30T12:18:36.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 5 monclient: authenticate success, global_id 204457 2023-10-30T12:18:36.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:36.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:36.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 4 set_mon_vals no callback set 2023-10-30T12:18:36.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 1 -- 172.21.15.50:0/3821817618 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588030710 con 0x7f658c0a2ae0 2023-10-30T12:18:36.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:36.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6590aed640 10 monclient: dump: 2023-10-30T12:18:36.674 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.675 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.675 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.675 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.675 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.675 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.675 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.676 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 10 monclient: shutdown 2023-10-30T12:18:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 1 -- 172.21.15.50:0/3821817618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c0a2ae0 msgr2=0x7f658c0a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 1 --2- 172.21.15.50:0/3821817618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c0a2ae0 0x7f658c0a2450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f658c09a470 tx=0x7f658802b6f0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 1 -- 172.21.15.50:0/3821817618 shutdown_connections 2023-10-30T12:18:36.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 1 --2- 172.21.15.50:0/3821817618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c0a2ae0 0x7f658c0a2450 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 1 -- 172.21.15.50:0/3821817618 >> 172.21.15.50:0/3821817618 conn(0x7f658c0a80b0 msgr2=0x7f658c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 1 -- 172.21.15.50:0/3821817618 shutdown_connections 2023-10-30T12:18:36.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.651+0000 7f6593d53640 1 -- 172.21.15.50:0/3821817618 wait complete. 2023-10-30T12:18:36.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 10 monclient: monmap: 2023-10-30T12:18:36.678 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.679 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:36.679 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:36.679 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:36.679 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.679 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.679 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:36.680 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:36.680 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:36.680 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 1 librados: starting msgr at 2023-10-30T12:18:36.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 1 librados: starting objecter 2023-10-30T12:18:36.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 1 Processor -- start 2023-10-30T12:18:36.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 1 -- start start 2023-10-30T12:18:36.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 1 librados: setting wanted keys 2023-10-30T12:18:36.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 1 librados: calling monclient init 2023-10-30T12:18:36.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.652+0000 7f6593d53640 10 monclient: init 2023-10-30T12:18:36.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:36.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c0a2ae0 0x7f658c11e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): picked mon.noname-c con 0x7f658c0a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:36.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f658c0a2ec0 0x7f658c11e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): picked mon.noname-a con 0x7f658c0a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:36.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c135df0 0x7f658c11eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): picked mon.noname-b con 0x7f658c135df0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:36.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c1216c0 con 0x7f658c0a2ec0 2023-10-30T12:18:36.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c120110 con 0x7f658c135df0 2023-10-30T12:18:36.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0a1790 con 0x7f658c0a2ae0 2023-10-30T12:18:36.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6593d53640 10 monclient(hunting): authenticate will time out at 1961.836335s 2023-10-30T12:18:36.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c0a2ae0 0x7f658c11e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f658c0a2ec0 0x7f658c11e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c0a2ae0 0x7f658c11e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40802/0 (socket says 172.21.15.50:40802) 2023-10-30T12:18:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 1 -- 172.21.15.50:0/661179896 learned_addr learned my addr 172.21.15.50:0/661179896 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 10 monclient(hunting): get_auth_request con 0x7f658c0a2ae0 auth_method 0 2023-10-30T12:18:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): get_auth_request con 0x7f658c0a2ec0 auth_method 0 2023-10-30T12:18:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65922f0640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c135df0 0x7f658c11eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65922f0640 10 monclient(hunting): get_auth_request con 0x7f658c135df0 auth_method 0 2023-10-30T12:18:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65922f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65922f0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65922f0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient(hunting): handle_auth_done global_id 234129 payload 824 2023-10-30T12:18:36.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c0a2ae0 msgr2=0x7f658c11e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c0a2ae0 0x7f658c11e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c135df0 msgr2=0x7f658c11eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f658c135df0 0x7f658c11eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 monclient: found mon.noname-a 2023-10-30T12:18:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 -- 172.21.15.50:0/661179896 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588009400 con 0x7f658c0a2ec0 2023-10-30T12:18:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65912ee640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f658c0a2ec0 0x7f658c11e5e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f657c00c390 tx=0x7f657c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65827fc640 10 client.?.objecter ms_handle_connect 0x7f658c0a2ec0 2023-10-30T12:18:36.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.653+0000 7f65827fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:36.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c0052d0 con 0x7f658c0a2ec0 2023-10-30T12:18:36.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:36.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: dump: 2023-10-30T12:18:36.692 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.692 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.692 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.692 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.693 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.693 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.693 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.693 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.693 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.693 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 5 monclient: authenticate success, global_id 234129 2023-10-30T12:18:36.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c005430 con 0x7f658c0a2ec0 2023-10-30T12:18:36.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 10 monclient: _renew_subs 2023-10-30T12:18:36.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c132470 con 0x7f658c0a2ec0 2023-10-30T12:18:36.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c003830 con 0x7f658c0a2ec0 2023-10-30T12:18:36.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:36.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f65827fc640 10 monclient: dump: 2023-10-30T12:18:36.696 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.696 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.697 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.697 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.697 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.697 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.697 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.697 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.698 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.698 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 10 client.234129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:36.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6583fff640 4 set_mon_vals no callback set 2023-10-30T12:18:36.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 10 monclient: _renew_subs 2023-10-30T12:18:36.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c130fc0 con 0x7f658c0a2ec0 2023-10-30T12:18:36.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 1 librados: init done 2023-10-30T12:18:36.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1ca5d0 0x7f658c1ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.654+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c073ff0 con 0x7f658c1ca5d0 2023-10-30T12:18:36.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f657c003990 con 0x7f658c0a2ec0 2023-10-30T12:18:36.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1ca5d0 0x7f658c1ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient: get_auth_request con 0x7f658c1ca5d0 auth_method 0 2023-10-30T12:18:36.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65700392f0 0x7f657003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(559..559 src has 490..559) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f657c04a320 con 0x7f658c0a2ec0 2023-10-30T12:18:36.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 10 client.234129.objecter ms_dispatch 0x7f658c000b90 osd_map(559..559 src has 490..559) v4 2023-10-30T12:18:36.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 3 client.234129.objecter handle_osd_map got epochs [559,559] > 0 2023-10-30T12:18:36.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 3 client.234129.objecter handle_osd_map decoding full epoch 559 2023-10-30T12:18:36.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 10 monclient(hunting): handle_auth_done global_id 224195 payload 824 2023-10-30T12:18:36.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 1 -- 172.21.15.50:0/661179896 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f658c09a5c0 con 0x7f658c1ca5d0 2023-10-30T12:18:36.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 20 client.234129.objecter dump_active .. 0 homeless 2023-10-30T12:18:36.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 10 client.234129.objecter ms_handle_connect 0x7f658c1ca5d0 2023-10-30T12:18:36.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 10 client.234129.objecter resend_mon_ops 2023-10-30T12:18:36.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f6591aef640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1ca5d0 0x7f658c1ca9b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f658802b730 tx=0x7f65880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65922f0640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65700392f0 0x7f657003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588032750 con 0x7f658c1ca5d0 2023-10-30T12:18:36.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:36.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65827fc640 10 monclient: dump: 2023-10-30T12:18:36.705 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.705 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.705 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.706 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.706 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.706 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.706 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.706 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.655+0000 7f65922f0640 10 monclient: get_auth_request con 0x7f65700392f0 auth_method 0 2023-10-30T12:18:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f65880328b0 con 0x7f658c1ca5d0 2023-10-30T12:18:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1ca5d0 msgr2=0x7f658c1ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1ca5d0 0x7f658c1ca9b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f658802b730 tx=0x7f65880046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1ca5d0 msgr2=0x7f658c1ca9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1ca5d0 0x7f658c1ca9b0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f658802b730 tx=0x7f65880046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65827fc640 10 client.234129.objecter ms_handle_connect 0x7f65700392f0 2023-10-30T12:18:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.656+0000 7f65922f0640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65700392f0 0x7f657003b7b0 secure :-1 s=READY pgs=4267 cs=0 l=1 rev1=1 crypto rx=0x7f6584001cb0 tx=0x7f6584009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6593d53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6593d53640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1caef0 0x7f658c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6593d53640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c099b40 con 0x7f658c1caef0 2023-10-30T12:18:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1caef0 0x7f658c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient: get_auth_request con 0x7f658c1caef0 auth_method 0 2023-10-30T12:18:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 10 monclient(hunting): handle_auth_done global_id 224198 payload 824 2023-10-30T12:18:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 1 -- 172.21.15.50:0/661179896 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f658c06b060 con 0x7f658c1caef0 2023-10-30T12:18:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f6591aef640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1caef0 0x7f658c09cd60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6588000c00 tx=0x7f6588030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f65827fc640 10 client.234129.objecter ms_handle_connect 0x7f658c1caef0 2023-10-30T12:18:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f65827fc640 10 client.234129.objecter resend_mon_ops 2023-10-30T12:18:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588050450 con 0x7f658c1caef0 2023-10-30T12:18:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f65827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f65827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.662+0000 7f65827fc640 10 monclient: dump: 2023-10-30T12:18:36.715 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.715 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.715 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.715 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.715 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.716 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.716 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.716 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.716 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.716 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f65880505b0 con 0x7f658c1caef0 2023-10-30T12:18:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65827fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1caef0 msgr2=0x7f658c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65827fc640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1caef0 0x7f658c09cd60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6588000c00 tx=0x7f6588030000 comp rx=0 tx=0).stop 2023-10-30T12:18:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65827fc640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1caef0 msgr2=0x7f658c09cd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:36.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65827fc640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f658c1caef0 0x7f658c09cd60 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6588000c00 tx=0x7f6588030000 comp rx=0 tx=0).stop 2023-10-30T12:18:36.718 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:36.718 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:36.719 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:36.719 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:18:36.719 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 190, 2023-10-30T12:18:36.719 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:36.719 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:18:36.720 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:36.720 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:36.720 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.720 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:18:36.720 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:36.721 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:36.721 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:36.721 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.721 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.721 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.721 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.722 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.722 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.722 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.722 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.722 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.723 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.723 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.723 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:36.723 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:36.723 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.723 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.724 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.724 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.724 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.724 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.724 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.725 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.725 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.725 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.725 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.726 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:36.726 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:36.726 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:36.726 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:36.726 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:36.727 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:36.727 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:36.727 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:36.727 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:36.727 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:36.728 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:36.728 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:36.728 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:36.728 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:36.728 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:36.729 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:36.729 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:36.729 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.729 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.729 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.729 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.730 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.730 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.730 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.730 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.730 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.731 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.731 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.731 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:36.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.731 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:36.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:36.732 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:36.732 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.732 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.733 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.733 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:36.733 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.734 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.734 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:36.734 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.734 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.734 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.735 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:36.735 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:36.735 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.735 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.735 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:36.736 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:36.736 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.737 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.737 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:36.737 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.738 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.738 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:36.738 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.739 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.739 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:36.739 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:36.739 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.740 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.740 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:36.741 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:36.741 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.741 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.741 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.742 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:36.742 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.742 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.743 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:36.743 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.743 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.743 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:36.744 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:36.744 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.744 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.744 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.745 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.745 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:36.745 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:36.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.745 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.746 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.746 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:36.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.746 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.746 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:18:36.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.747 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.747 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.747 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:18:36.747 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.748 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.748 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:36.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.748 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.748 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.748 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:18:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:18:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.750 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.750 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:36.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.750 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.750 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:36.751 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 10 librados: watch_flush enter 2023-10-30T12:18:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 10 librados: watch_flush exit 2023-10-30T12:18:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 20 client.234129.objecter shutdown clearing up homeless session... 2023-10-30T12:18:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 10 client.234129.objecter successfully canceled tick 2023-10-30T12:18:36.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65700392f0 msgr2=0x7f657003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65700392f0 0x7f657003b7b0 secure :-1 s=READY pgs=4267 cs=0 l=1 rev1=1 crypto rx=0x7f6584001cb0 tx=0x7f6584009040 comp rx=0 tx=0).stop 2023-10-30T12:18:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 10 monclient: shutdown 2023-10-30T12:18:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f658c0a2ec0 msgr2=0x7f658c11e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 --2- 172.21.15.50:0/661179896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f658c0a2ec0 0x7f658c11e5e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f657c00c390 tx=0x7f657c00c850 comp rx=0 tx=0).stop 2023-10-30T12:18:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65922f0640 1 -- 172.21.15.50:0/661179896 reap_dead start 2023-10-30T12:18:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f65922f0640 1 -- 172.21.15.50:0/661179896 reap_dead start 2023-10-30T12:18:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 shutdown_connections 2023-10-30T12:18:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 >> 172.21.15.50:0/661179896 conn(0x7f658c0a80b0 msgr2=0x7f658c131ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 shutdown_connections 2023-10-30T12:18:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 -- 172.21.15.50:0/661179896 wait complete. 2023-10-30T12:18:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.663+0000 7f6593d53640 1 librados: shutdown 2023-10-30T12:18:36.756 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:18:36.756 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:18:36.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:18:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.861+0000 7fa26d5a5640 10 monclient: get_monmap_and_config 2023-10-30T12:18:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.861+0000 7fa26d5a5640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.861+0000 7fa26d5a5640 10 monclient: monmap: 2023-10-30T12:18:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:36.862679+0000 2023-10-30T12:18:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:36.862679+0000 2023-10-30T12:18:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:36.865 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:36.865 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:36.865 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:36.865 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:36.865 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 1 Processor -- start 2023-10-30T12:18:36.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 1 -- start start 2023-10-30T12:18:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient: init 2023-10-30T12:18:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:18:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa268097780 0x7fa268097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient(hunting): picked mon.noname-a con 0x7fa268097780 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa268073b10 0x7fa2680a0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient(hunting): picked mon.noname-c con 0x7fa268073b10 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa26809ccd0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa268071ec0 con 0x7fa26809ccd0 2023-10-30T12:18:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26809bd10 con 0x7fa268097780 2023-10-30T12:18:36.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26809bff0 con 0x7fa268073b10 2023-10-30T12:18:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.862+0000 7fa26d5a5640 10 monclient(hunting): authenticate will time out at 1962.045907s 2023-10-30T12:18:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa268097780 0x7fa268097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa268097780 0x7fa268097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34224/0 (socket says 172.21.15.50:34224) 2023-10-30T12:18:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 -- 172.21.15.50:0/2799950654 learned_addr learned my addr 172.21.15.50:0/2799950654 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): get_auth_request con 0x7fa268097780 auth_method 0 2023-10-30T12:18:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient(hunting): handle_auth_done global_id 234132 payload 274 2023-10-30T12:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa25ffff640 1 --2- 172.21.15.50:0/2799950654 >> v2:172.21.15.118:3300/0 conn(0x7fa268073b10 0x7fa2680a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 -- 172.21.15.50:0/2799950654 >> v2:172.21.15.118:3300/0 conn(0x7fa268073b10 msgr2=0x7fa2680a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 --2- 172.21.15.50:0/2799950654 >> v2:172.21.15.118:3300/0 conn(0x7fa268073b10 0x7fa2680a0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 -- 172.21.15.50:0/2799950654 >> v1:172.21.15.118:6789/0 conn(0x7fa26809ccd0 legacy=0x7fa2680a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 monclient: found mon.noname-a 2023-10-30T12:18:36.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.863+0000 7fa266ffd640 1 -- 172.21.15.50:0/2799950654 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2680980a0 con 0x7fa268097780 2023-10-30T12:18:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa266ffd640 1 --2- 172.21.15.50:0/2799950654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268097780 0x7fa268097b60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa26806a230 tx=0x7fa25802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4173496269e1a60f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 1 -- 172.21.15.50:0/2799950654 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258031750 con 0x7fa268097780 2023-10-30T12:18:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: dump: 2023-10-30T12:18:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.886 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.886 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 5 monclient: authenticate success, global_id 234132 2023-10-30T12:18:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 1 -- 172.21.15.50:0/2799950654 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2580318b0 con 0x7fa268097780 2023-10-30T12:18:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 4 set_mon_vals no callback set 2023-10-30T12:18:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 1 -- 172.21.15.50:0/2799950654 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258031c40 con 0x7fa268097780 2023-10-30T12:18:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa2667fc640 10 monclient: dump: 2023-10-30T12:18:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.889 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.890 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.891 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 10 monclient: shutdown 2023-10-30T12:18:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 -- 172.21.15.50:0/2799950654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268097780 msgr2=0x7fa268097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 --2- 172.21.15.50:0/2799950654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268097780 0x7fa268097b60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa26806a230 tx=0x7fa25802b7f0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 -- 172.21.15.50:0/2799950654 shutdown_connections 2023-10-30T12:18:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 --2- 172.21.15.50:0/2799950654 >> v2:172.21.15.118:3300/0 conn(0x7fa268073b10 0x7fa2680a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 --2- 172.21.15.50:0/2799950654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268097780 0x7fa268097b60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 -- 172.21.15.50:0/2799950654 >> 172.21.15.50:0/2799950654 conn(0x7fa2680a1170 msgr2=0x7fa2680a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 -- 172.21.15.50:0/2799950654 shutdown_connections 2023-10-30T12:18:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.864+0000 7fa26d5a5640 1 -- 172.21.15.50:0/2799950654 wait complete. 2023-10-30T12:18:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 10 monclient: build_initial_monmap 2023-10-30T12:18:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 10 monclient: monmap: 2023-10-30T12:18:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:36.893 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:36.894 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:36.895 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 1 librados: starting msgr at 2023-10-30T12:18:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 1 librados: starting objecter 2023-10-30T12:18:36.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 1 Processor -- start 2023-10-30T12:18:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 1 -- start start 2023-10-30T12:18:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 1 librados: setting wanted keys 2023-10-30T12:18:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 1 librados: calling monclient init 2023-10-30T12:18:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.865+0000 7fa26d5a5640 10 monclient: init 2023-10-30T12:18:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:36.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268073b10 0x7fa2681ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): picked mon.noname-a con 0x7fa268073b10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa268097780 0x7fa2681caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): picked mon.noname-c con 0x7fa268097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:36.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa26809ccd0 0x7fa2681cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): picked mon.noname-b con 0x7fa26809ccd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa268093850 con 0x7fa268073b10 2023-10-30T12:18:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa268091ad0 con 0x7fa26809ccd0 2023-10-30T12:18:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa268091170 con 0x7fa268097780 2023-10-30T12:18:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient(hunting): authenticate will time out at 1962.049188s 2023-10-30T12:18:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268073b10 0x7fa2681ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa268097780 0x7fa2681caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268073b10 0x7fa2681ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34238/0 (socket says 172.21.15.50:34238) 2023-10-30T12:18:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): get_auth_request con 0x7fa268097780 auth_method 0 2023-10-30T12:18:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 1 -- 172.21.15.50:0/3772327223 learned_addr learned my addr 172.21.15.50:0/3772327223 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 10 monclient(hunting): get_auth_request con 0x7fa268073b10 auth_method 0 2023-10-30T12:18:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa2677fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa26809ccd0 0x7fa2681cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient(hunting): handle_auth_done global_id 224201 payload 824 2023-10-30T12:18:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa26809ccd0 msgr2=0x7fa2681cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa26809ccd0 0x7fa2681cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268073b10 msgr2=0x7fa2681ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268073b10 0x7fa2681ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 monclient: found mon.noname-c 2023-10-30T12:18:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:36.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 -- 172.21.15.50:0/3772327223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2580094d0 con 0x7fa268097780 2023-10-30T12:18:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa2677fe640 10 monclient: get_auth_request con 0x7fa26809ccd0 auth_method 0 2023-10-30T12:18:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa2677fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa26809ccd0 0x7fa2681cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:18:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa266ffd640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa268073b10 0x7fa2681ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25ffff640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa268097780 0x7fa2681caba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c5e0 tx=0x7fa25000caa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 client.?.objecter ms_handle_connect 0x7fa268097780 2023-10-30T12:18:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250004220 con 0x7fa268097780 2023-10-30T12:18:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: dump: 2023-10-30T12:18:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.908 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 5 monclient: authenticate success, global_id 224201 2023-10-30T12:18:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa250004380 con 0x7fa268097780 2023-10-30T12:18:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient: _renew_subs 2023-10-30T12:18:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa268073f70 con 0x7fa268097780 2023-10-30T12:18:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250003850 con 0x7fa268097780 2023-10-30T12:18:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa25f7fe640 10 monclient: dump: 2023-10-30T12:18:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.912 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.866+0000 7fa265ffb640 4 set_mon_vals no callback set 2023-10-30T12:18:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 10 client.224201.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 10 monclient: _renew_subs 2023-10-30T12:18:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa268092b90 con 0x7fa268097780 2023-10-30T12:18:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 1 librados: init done 2023-10-30T12:18:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268090540 0x7fa268060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa268092f30 con 0x7fa268090540 2023-10-30T12:18:36.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa2500039b0 con 0x7fa268097780 2023-10-30T12:18:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa266ffd640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268090540 0x7fa268060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.867+0000 7fa266ffd640 10 monclient: get_auth_request con 0x7fa268090540 auth_method 0 2023-10-30T12:18:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa2480392f0 0x7fa24803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(559..559 src has 490..559) v4 ==== 6678+0+0 (secure 0 0 0) 0x7fa25004af00 con 0x7fa268097780 2023-10-30T12:18:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 10 client.224201.objecter ms_dispatch 0x7fa268000b90 osd_map(559..559 src has 490..559) v4 2023-10-30T12:18:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 3 client.224201.objecter handle_osd_map got epochs [559,559] > 0 2023-10-30T12:18:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 3 client.224201.objecter handle_osd_map decoding full epoch 559 2023-10-30T12:18:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 20 client.224201.objecter dump_active .. 0 homeless 2023-10-30T12:18:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa2677fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa2480392f0 0x7fa24803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 10 monclient(hunting): handle_auth_done global_id 204466 payload 824 2023-10-30T12:18:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 1 -- 172.21.15.50:0/3772327223 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2680a0cd0 con 0x7fa268090540 2023-10-30T12:18:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa2677fe640 10 monclient: get_auth_request con 0x7fa2480392f0 auth_method 0 2023-10-30T12:18:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa266ffd640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268090540 0x7fa268060fc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa25802b830 tx=0x7fa2580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 10 client.224201.objecter ms_handle_connect 0x7fa268090540 2023-10-30T12:18:36.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 10 client.224201.objecter resend_mon_ops 2023-10-30T12:18:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25803d440 con 0x7fa268090540 2023-10-30T12:18:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa2677fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa2480392f0 0x7fa24803b7b0 secure :-1 s=READY pgs=4268 cs=0 l=1 rev1=1 crypto rx=0x7fa260001d40 tx=0x7fa260009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:36.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 10 monclient: dump: 2023-10-30T12:18:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.920 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.921 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.868+0000 7fa25f7fe640 10 client.224201.objecter ms_handle_connect 0x7fa2480392f0 2023-10-30T12:18:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.869+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa258041070 con 0x7fa268090540 2023-10-30T12:18:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.869+0000 7fa25f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.869+0000 7fa25f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.869+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268090540 msgr2=0x7fa268060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.869+0000 7fa25f7fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268090540 0x7fa268060fc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa25802b830 tx=0x7fa2580046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.869+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268090540 msgr2=0x7fa268060fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:36.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.869+0000 7fa25f7fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268090540 0x7fa268060fc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa26d5a5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:36.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa26d5a5640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268061500 0x7fa26802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:36.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa26d5a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:36.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26806a5a0 con 0x7fa268061500 2023-10-30T12:18:36.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268061500 0x7fa26802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:36.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient: get_auth_request con 0x7fa268061500 auth_method 0 2023-10-30T12:18:36.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:36.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:36.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:36.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:36.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:36.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:36.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 10 monclient(hunting): handle_auth_done global_id 204469 payload 824 2023-10-30T12:18:36.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.875+0000 7fa266ffd640 1 -- 172.21.15.50:0/3772327223 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa26804e160 con 0x7fa268061500 2023-10-30T12:18:36.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa266ffd640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268061500 0x7fa26802b430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa258045000 tx=0x7fa258031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:36.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 10 client.224201.objecter ms_handle_connect 0x7fa268061500 2023-10-30T12:18:36.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 10 client.224201.objecter resend_mon_ops 2023-10-30T12:18:36.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258050420 con 0x7fa268061500 2023-10-30T12:18:36.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:36.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:36.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 10 monclient: dump: 2023-10-30T12:18:36.929 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:36.929 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:36.929 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.929 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:36.929 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:36.929 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:36.930 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:36.930 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:36.930 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:36.930 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:36.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fa258050580 con 0x7fa268061500 2023-10-30T12:18:36.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:36.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:36.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268061500 msgr2=0x7fa26802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268061500 0x7fa26802b430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa258045000 tx=0x7fa258031000 comp rx=0 tx=0).stop 2023-10-30T12:18:36.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268061500 msgr2=0x7fa26802b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:36.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa25f7fe640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa268061500 0x7fa26802b430 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.932 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:18:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 190, 2023-10-30T12:18:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:18:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:18:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:18:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:36.961 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:36.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 10 librados: watch_flush enter 2023-10-30T12:18:36.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 10 librados: watch_flush exit 2023-10-30T12:18:36.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 20 client.224201.objecter shutdown clearing up homeless session... 2023-10-30T12:18:36.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 10 client.224201.objecter successfully canceled tick 2023-10-30T12:18:36.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa2480392f0 msgr2=0x7fa24803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa2480392f0 0x7fa24803b7b0 secure :-1 s=READY pgs=4268 cs=0 l=1 rev1=1 crypto rx=0x7fa260001d40 tx=0x7fa260009040 comp rx=0 tx=0).stop 2023-10-30T12:18:36.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 10 monclient: shutdown 2023-10-30T12:18:36.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:36.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa268097780 msgr2=0x7fa2681caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:36.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa26d5a5640 1 --2- 172.21.15.50:0/3772327223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa268097780 0x7fa2681caba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c5e0 tx=0x7fa25000caa0 comp rx=0 tx=0).stop 2023-10-30T12:18:36.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.876+0000 7fa2677fe640 1 -- 172.21.15.50:0/3772327223 reap_dead start 2023-10-30T12:18:36.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.877+0000 7fa2677fe640 1 -- 172.21.15.50:0/3772327223 reap_dead start 2023-10-30T12:18:36.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.877+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 shutdown_connections 2023-10-30T12:18:36.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.877+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 >> 172.21.15.50:0/3772327223 conn(0x7fa2680a1170 msgr2=0x7fa26809a580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:36.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.877+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 shutdown_connections 2023-10-30T12:18:36.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.877+0000 7fa26d5a5640 1 -- 172.21.15.50:0/3772327223 wait complete. 2023-10-30T12:18:36.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:36.877+0000 7fa26d5a5640 1 librados: shutdown 2023-10-30T12:18:36.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:18:36.967 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:18:36.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:18:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.070+0000 7f20870a9640 10 monclient: get_monmap_and_config 2023-10-30T12:18:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.070+0000 7f20870a9640 10 monclient: build_initial_monmap 2023-10-30T12:18:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.070+0000 7f20870a9640 10 monclient: monmap: 2023-10-30T12:18:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:37.071734+0000 2023-10-30T12:18:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:37.071734+0000 2023-10-30T12:18:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:37.075 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 1 Processor -- start 2023-10-30T12:18:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 1 -- start start 2023-10-30T12:18:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 10 monclient: init 2023-10-30T12:18:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-10-30T12:18:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20800707d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20800712c0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f208009e7e0 0x7f208013e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 10 monclient(hunting): picked mon.noname-a con 0x7f208009e7e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.071+0000 7f20870a9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208009ee40 con 0x7f20800707d0 2023-10-30T12:18:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f20870a9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208009f4f0 con 0x7f20800712c0 2023-10-30T12:18:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f20870a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f20870a9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208009ec30 con 0x7f208009e7e0 2023-10-30T12:18:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f20870a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f20870a9640 10 monclient(hunting): authenticate will time out at 1962.255031s 2023-10-30T12:18:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f208009e7e0 0x7f208013e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2084e45640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f20800707d0 0x7f2080070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59930/0 (socket says 172.21.15.50:59930) 2023-10-30T12:18:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2084e45640 1 -- 172.21.15.50:0/4172804793 learned_addr learned my addr 172.21.15.50:0/4172804793 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): get_auth_request con 0x7f208009e7e0 auth_method 0 2023-10-30T12:18:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:37.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:37.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:37.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:37.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient(hunting): handle_auth_done global_id 234138 payload 274 2023-10-30T12:18:37.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 1 -- 172.21.15.50:0/4172804793 >> v1:172.21.15.118:6789/0 conn(0x7f20800712c0 legacy=0x7f208013aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:37.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 1 -- 172.21.15.50:0/4172804793 >> v1:172.21.15.50:6789/0 conn(0x7f20800707d0 legacy=0x7f2080070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:37.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient: _finish_hunting 0 2023-10-30T12:18:37.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 1 monclient: found mon.noname-a 2023-10-30T12:18:37.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:37.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:37.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f20777fe640 1 -- 172.21.15.50:0/4172804793 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936982322 0 0) 0x7f208009ee40 con 0x7f20800707d0 2023-10-30T12:18:37.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f2085646640 1 -- 172.21.15.50:0/4172804793 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208009f7d0 con 0x7f208009e7e0 2023-10-30T12:18:37.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.072+0000 7f20777fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:18:37.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f2085646640 1 --2- 172.21.15.50:0/4172804793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f208013e2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2078009d90 tx=0x7f207802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d28b75963c392f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:37.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 1 -- 172.21.15.50:0/4172804793 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207802cb40 con 0x7f208009e7e0 2023-10-30T12:18:37.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:37.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:37.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: dump: 2023-10-30T12:18:37.091 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:37.091 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:37.091 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.092 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.092 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:37.092 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:37.092 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:37.092 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:37.093 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:37.093 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:37.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:37.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:37.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:37.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 5 monclient: authenticate success, global_id 234138 2023-10-30T12:18:37.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:37.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 1 -- 172.21.15.50:0/4172804793 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207802cca0 con 0x7f208009e7e0 2023-10-30T12:18:37.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:37.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 1 -- 172.21.15.50:0/4172804793 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078034660 con 0x7f208009e7e0 2023-10-30T12:18:37.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:37.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:37.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20777fe640 10 monclient: dump: 2023-10-30T12:18:37.095 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:37.095 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:37.095 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.096 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.096 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:37.096 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:37.096 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:37.096 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:37.097 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:37.097 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:37.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:37.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 4 set_mon_vals no callback set 2023-10-30T12:18:37.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 10 monclient: shutdown 2023-10-30T12:18:37.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:37.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 1 -- 172.21.15.50:0/4172804793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 msgr2=0x7f208013e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:37.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 1 --2- 172.21.15.50:0/4172804793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f208013e2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2078009d90 tx=0x7f207802bd90 comp rx=0 tx=0).stop 2023-10-30T12:18:37.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 1 -- 172.21.15.50:0/4172804793 shutdown_connections 2023-10-30T12:18:37.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 1 --2- 172.21.15.50:0/4172804793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f208013e2d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:37.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 1 -- 172.21.15.50:0/4172804793 >> 172.21.15.50:0/4172804793 conn(0x7f20800a47f0 msgr2=0x7f20800a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:37.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 1 -- 172.21.15.50:0/4172804793 shutdown_connections 2023-10-30T12:18:37.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.073+0000 7f20870a9640 1 -- 172.21.15.50:0/4172804793 wait complete. 2023-10-30T12:18:37.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 10 monclient: build_initial_monmap 2023-10-30T12:18:37.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 10 monclient: monmap: 2023-10-30T12:18:37.100 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:37.100 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:37.100 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:37.100 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:37.100 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:37.101 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:37.101 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:37.101 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:37.101 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:37.101 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:37.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 1 librados: starting msgr at 2023-10-30T12:18:37.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 1 librados: starting objecter 2023-10-30T12:18:37.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 1 Processor -- start 2023-10-30T12:18:37.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 1 -- start start 2023-10-30T12:18:37.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 1 librados: setting wanted keys 2023-10-30T12:18:37.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 1 librados: calling monclient init 2023-10-30T12:18:37.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.074+0000 7f20870a9640 10 monclient: init 2023-10-30T12:18:37.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:37.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:37.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20800707d0 0x7f20801c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:37.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): picked mon.noname-b con 0x7f20800707d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20800712c0 0x7f20801b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): picked mon.noname-c con 0x7f20800712c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f20801b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): picked mon.noname-a con 0x7f208009e7e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20801a6e70 con 0x7f208009e7e0 2023-10-30T12:18:37.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:37.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20801a45b0 con 0x7f20800707d0 2023-10-30T12:18:37.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:37.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20801b9200 con 0x7f20800712c0 2023-10-30T12:18:37.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:37.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f20870a9640 10 monclient(hunting): authenticate will time out at 1962.258452s 2023-10-30T12:18:37.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20800712c0 0x7f20801b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:37.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f20801b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:37.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20800712c0 0x7f20801b8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40836/0 (socket says 172.21.15.50:40836) 2023-10-30T12:18:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 1 -- 172.21.15.50:0/4044827066 learned_addr learned my addr 172.21.15.50:0/4044827066 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 10 monclient(hunting): get_auth_request con 0x7f20800712c0 auth_method 0 2023-10-30T12:18:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f20801b8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34258/0 (socket says 172.21.15.50:34258) 2023-10-30T12:18:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): get_auth_request con 0x7f208009e7e0 auth_method 0 2023-10-30T12:18:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2084e45640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20800707d0 0x7f20801c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2084e45640 10 monclient(hunting): get_auth_request con 0x7f20800707d0 auth_method 0 2023-10-30T12:18:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2084e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2084e45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2084e45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient(hunting): handle_auth_done global_id 234144 payload 824 2023-10-30T12:18:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 -- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20800712c0 msgr2=0x7f20801b8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20800712c0 0x7f20801b8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 -- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20800707d0 msgr2=0x7f20801c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20800707d0 0x7f20801c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:37.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient: _finish_hunting 0 2023-10-30T12:18:37.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 monclient: found mon.noname-a 2023-10-30T12:18:37.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:37.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:37.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2085646640 1 -- 172.21.15.50:0/4044827066 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078009a70 con 0x7f208009e7e0 2023-10-30T12:18:37.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2077fff640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20800712c0 0x7f20801b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:37.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.075+0000 7f2084e45640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20800707d0 0x7f20801c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:18:37.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f2085646640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f20801b8cc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f20800661a0 tx=0x7f2078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:37.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 client.?.objecter ms_handle_connect 0x7f208009e7e0 2023-10-30T12:18:37.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:37.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 1 -- 172.21.15.50:0/4044827066 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207803f420 con 0x7f208009e7e0 2023-10-30T12:18:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: dump: 2023-10-30T12:18:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:37.114 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:37.115 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:37.116 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: _finish_auth 0 2023-10-30T12:18:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: _check_auth_tickets 2023-10-30T12:18:37.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 1 -- 172.21.15.50:0/4044827066 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207803f580 con 0x7f208009e7e0 2023-10-30T12:18:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 5 monclient: authenticate success, global_id 234144 2023-10-30T12:18:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 1 -- 172.21.15.50:0/4044827066 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207803f8f0 con 0x7f208009e7e0 2023-10-30T12:18:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20757fa640 10 monclient: dump: 2023-10-30T12:18:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:37.119 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:37.120 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 10 monclient: _renew_subs 2023-10-30T12:18:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20801a5c40 con 0x7f208009e7e0 2023-10-30T12:18:37.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f2076ffd640 4 set_mon_vals no callback set 2023-10-30T12:18:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 10 client.234144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 10 monclient: _renew_subs 2023-10-30T12:18:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20801a4d80 con 0x7f208009e7e0 2023-10-30T12:18:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.076+0000 7f20870a9640 1 librados: init done 2023-10-30T12:18:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20757fa640 1 -- 172.21.15.50:0/4044827066 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f207803fa50 con 0x7f208009e7e0 2023-10-30T12:18:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20757fa640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f206c0392f0 0x7f206c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20870a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20870a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:37.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20757fa640 1 -- 172.21.15.50:0/4044827066 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(559..559 src has 490..559) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f207806de90 con 0x7f208009e7e0 2023-10-30T12:18:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20870a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20757fa640 10 client.234144.objecter ms_dispatch 0x7f2080000b90 osd_map(559..559 src has 490..559) v4 2023-10-30T12:18:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20757fa640 3 client.234144.objecter handle_osd_map got epochs [559,559] > 0 2023-10-30T12:18:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20757fa640 3 client.234144.objecter handle_osd_map decoding full epoch 559 2023-10-30T12:18:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20801a7020 con 0x7f208009e7e0 2023-10-30T12:18:37.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f20757fa640 20 client.234144.objecter dump_active .. 0 homeless 2023-10-30T12:18:37.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f2084e45640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f206c0392f0 0x7f206c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:37.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.077+0000 7f2084e45640 10 monclient: get_auth_request con 0x7f206c0392f0 auth_method 0 2023-10-30T12:18:37.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.078+0000 7f20757fa640 10 client.234144.objecter ms_handle_connect 0x7f206c0392f0 2023-10-30T12:18:37.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.078+0000 7f2084e45640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f206c0392f0 0x7f206c03b7b0 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 crypto rx=0x7f207c000f60 tx=0x7f207c007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:37.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.082+0000 7f20757fa640 1 -- 172.21.15.50:0/4044827066 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f207809aca0 con 0x7f208009e7e0 2023-10-30T12:18:37.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.082+0000 7f20757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:37.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.082+0000 7f20757fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:37.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.328+0000 7f20870a9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:18:37.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.328+0000 7f20870a9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:18:37.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.328+0000 7f20870a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:37.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.328+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f208004e160 con 0x7f208009e7e0 2023-10-30T12:18:37.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.329+0000 7f20757fa640 1 -- 172.21.15.50:0/4044827066 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f20801a7020 con 0x7f208009e7e0 2023-10-30T12:18:37.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.329+0000 7f20757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:18:37.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.329+0000 7f20757fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:37.331 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.329+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.329+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=1446189503,config=3140514009,health=3820569738,kv=2905431493,logm=1525907472}) 2023-10-30T12:18:37.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.329+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=9,logm=50} crc {auth=586912585,config=3140514009,health=815781731,kv=2905431493,logm=4040127870}) 2023-10-30T12:18:37.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.329+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.329+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=1446189503,config=3140514009,health=3820569738,kv=2905431493,logm=1525907472}) 2023-10-30T12:18:37.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.329+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=1,health=12,kv=8,logm=53} crc {auth=4186506248,config=1228700967,health=815781731,kv=1634934840,logm=2604674704}) 2023-10-30T12:18:37.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.331+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.333 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.331+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320922726}) 2023-10-30T12:18:37.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.331+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187184701}) 2023-10-30T12:18:37.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.331+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.334 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.331+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320922726}) 2023-10-30T12:18:37.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 10 librados: watch_flush enter 2023-10-30T12:18:37.335 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.331+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676520270}) 2023-10-30T12:18:37.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 10 librados: watch_flush exit 2023-10-30T12:18:37.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 20 client.234144.objecter shutdown clearing up homeless session... 2023-10-30T12:18:37.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 10 client.234144.objecter successfully canceled tick 2023-10-30T12:18:37.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f206c0392f0 msgr2=0x7f206c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:37.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f206c0392f0 0x7f206c03b7b0 secure :-1 s=READY pgs=4269 cs=0 l=1 rev1=1 crypto rx=0x7f207c000f60 tx=0x7f207c007c10 comp rx=0 tx=0).stop 2023-10-30T12:18:37.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 10 monclient: shutdown 2023-10-30T12:18:37.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:37.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 msgr2=0x7f20801b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:37.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f20801b8cc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f20800661a0 tx=0x7f2078004a40 comp rx=0 tx=0).stop 2023-10-30T12:18:37.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 shutdown_connections 2023-10-30T12:18:37.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f206c0392f0 0x7f206c03b7b0 unknown :-1 s=CLOSED pgs=4269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:37.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f208009e7e0 0x7f20801b8cc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:37.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20800712c0 0x7f20801b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:37.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 --2- 172.21.15.50:0/4044827066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20800707d0 0x7f20801c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:37.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 >> 172.21.15.50:0/4044827066 conn(0x7f20800a47f0 msgr2=0x7f20800a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:37.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 shutdown_connections 2023-10-30T12:18:37.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 -- 172.21.15.50:0/4044827066 wait complete. 2023-10-30T12:18:37.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:37.331+0000 7f20870a9640 1 librados: shutdown 2023-10-30T12:18:37.344 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.332+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.332+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705785161}) 2023-10-30T12:18:37.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.332+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=13233278}) 2023-10-30T12:18:37.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.332+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.332+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705785161}) 2023-10-30T12:18:37.345 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.332+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690724390}) 2023-10-30T12:18:37.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.333+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.333+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3649673272}) 2023-10-30T12:18:37.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.333+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1891627659}) 2023-10-30T12:18:37.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.333+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.333+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3649673272}) 2023-10-30T12:18:37.346 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.333+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854016100}) 2023-10-30T12:18:37.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.334+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.334+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960206768}) 2023-10-30T12:18:37.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.334+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1470986570}) 2023-10-30T12:18:37.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.334+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.347 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960206768}) 2023-10-30T12:18:37.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3549648799}) 2023-10-30T12:18:37.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999400441}) 2023-10-30T12:18:37.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1468132781}) 2023-10-30T12:18:37.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.348 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999400441}) 2023-10-30T12:18:37.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.335+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660580686}) 2023-10-30T12:18:37.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.336+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.336+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475799764}) 2023-10-30T12:18:37.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.336+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3208730372}) 2023-10-30T12:18:37.349 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.336+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.336+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475799764}) 2023-10-30T12:18:37.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.336+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767559280}) 2023-10-30T12:18:37.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.337+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.337+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593630273}) 2023-10-30T12:18:37.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.337+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4075944047}) 2023-10-30T12:18:37.350 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.337+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.337+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593630273}) 2023-10-30T12:18:37.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.337+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4140037498}) 2023-10-30T12:18:37.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.338+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.338+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758904045}) 2023-10-30T12:18:37.351 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.338+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1432255482}) 2023-10-30T12:18:37.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.338+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.338+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758904045}) 2023-10-30T12:18:37.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.338+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2984111292}) 2023-10-30T12:18:37.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.339+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.339+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2155593698}) 2023-10-30T12:18:37.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.339+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257194017}) 2023-10-30T12:18:37.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.339+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.339+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2155593698}) 2023-10-30T12:18:37.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.339+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=782467463}) 2023-10-30T12:18:37.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.340+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.353 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.340+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775720486}) 2023-10-30T12:18:37.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.340+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2955791304}) 2023-10-30T12:18:37.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.340+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.340+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775720486}) 2023-10-30T12:18:37.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.340+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163959580}) 2023-10-30T12:18:37.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.341+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.354 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.341+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320117287}) 2023-10-30T12:18:37.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.341+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2317567103}) 2023-10-30T12:18:37.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.341+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.341+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320117287}) 2023-10-30T12:18:37.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.341+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316330351}) 2023-10-30T12:18:37.355 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919575363}) 2023-10-30T12:18:37.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077952521}) 2023-10-30T12:18:37.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919575363}) 2023-10-30T12:18:37.356 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1000438272}) 2023-10-30T12:18:37.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:18:37.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931384460}) 2023-10-30T12:18:37.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.342+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1286938813}) 2023-10-30T12:18:37.377 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931384460}) 2023-10-30T12:18:37.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084793662}) 2023-10-30T12:18:37.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612622649}) 2023-10-30T12:18:37.378 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377159733}) 2023-10-30T12:18:37.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612622649}) 2023-10-30T12:18:37.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.343+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618004953}) 2023-10-30T12:18:37.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.344+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.379 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.344+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261720308}) 2023-10-30T12:18:37.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.344+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1992621942}) 2023-10-30T12:18:37.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.344+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.344+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261720308}) 2023-10-30T12:18:37.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.344+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563777602}) 2023-10-30T12:18:37.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.345+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.380 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.345+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927545708}) 2023-10-30T12:18:37.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.345+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321379699}) 2023-10-30T12:18:37.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.345+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.345+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927545708}) 2023-10-30T12:18:37.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.345+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477091349}) 2023-10-30T12:18:37.381 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.346+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.346+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065735455}) 2023-10-30T12:18:37.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.346+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1400344791}) 2023-10-30T12:18:37.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.346+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.346+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065735455}) 2023-10-30T12:18:37.382 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.346+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=485463247}) 2023-10-30T12:18:37.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.347+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.347+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890655126}) 2023-10-30T12:18:37.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.347+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639863163}) 2023-10-30T12:18:37.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.347+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.347+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890655126}) 2023-10-30T12:18:37.383 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.347+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744887203}) 2023-10-30T12:18:37.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.348+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.348+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367837537}) 2023-10-30T12:18:37.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.348+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230539957}) 2023-10-30T12:18:37.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.348+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.384 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.348+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367837537}) 2023-10-30T12:18:37.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.348+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1058696702}) 2023-10-30T12:18:37.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.349+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.349+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804152691}) 2023-10-30T12:18:37.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.349+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434400428}) 2023-10-30T12:18:37.385 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.349+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.349+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804152691}) 2023-10-30T12:18:37.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.349+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2153330221}) 2023-10-30T12:18:37.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.350+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.350+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612907904}) 2023-10-30T12:18:37.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.350+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042088279}) 2023-10-30T12:18:37.386 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.350+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.350+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1612907904}) 2023-10-30T12:18:37.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.350+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540902794}) 2023-10-30T12:18:37.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.351+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.351+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678385596}) 2023-10-30T12:18:37.387 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.351+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331970664}) 2023-10-30T12:18:37.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.351+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.351+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678385596}) 2023-10-30T12:18:37.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.351+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1214100379}) 2023-10-30T12:18:37.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.352+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.352+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=398283520}) 2023-10-30T12:18:37.388 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.352+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2122741663}) 2023-10-30T12:18:37.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.352+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.352+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=398283520}) 2023-10-30T12:18:37.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.352+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2872336108}) 2023-10-30T12:18:37.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.389 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232279144}) 2023-10-30T12:18:37.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=570491380}) 2023-10-30T12:18:37.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232279144}) 2023-10-30T12:18:37.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343198029}) 2023-10-30T12:18:37.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.390 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.353+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428731456}) 2023-10-30T12:18:37.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114375281}) 2023-10-30T12:18:37.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428731456}) 2023-10-30T12:18:37.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555610494}) 2023-10-30T12:18:37.391 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155811542}) 2023-10-30T12:18:37.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3459969673}) 2023-10-30T12:18:37.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155811542}) 2023-10-30T12:18:37.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.354+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678587191}) 2023-10-30T12:18:37.392 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.355+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.355+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3306533846}) 2023-10-30T12:18:37.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.355+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349771510}) 2023-10-30T12:18:37.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.355+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.355+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3306533846}) 2023-10-30T12:18:37.393 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.355+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=212036997}) 2023-10-30T12:18:37.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.356+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.356+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431789638}) 2023-10-30T12:18:37.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.356+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1475957201}) 2023-10-30T12:18:37.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.356+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.356+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431789638}) 2023-10-30T12:18:37.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.356+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045833766}) 2023-10-30T12:18:37.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.357+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.357+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398058848}) 2023-10-30T12:18:37.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.357+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1664830146}) 2023-10-30T12:18:37.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.357+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.357+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398058848}) 2023-10-30T12:18:37.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.357+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501313950}) 2023-10-30T12:18:37.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862026924}) 2023-10-30T12:18:37.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154889573}) 2023-10-30T12:18:37.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.396 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862026924}) 2023-10-30T12:18:37.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525340336}) 2023-10-30T12:18:37.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2910358945}) 2023-10-30T12:18:37.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596570215}) 2023-10-30T12:18:37.397 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2910358945}) 2023-10-30T12:18:37.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.358+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419676946}) 2023-10-30T12:18:37.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.359+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.359+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3650133142}) 2023-10-30T12:18:37.398 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.359+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1770486694}) 2023-10-30T12:18:37.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.359+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.359+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3650133142}) 2023-10-30T12:18:37.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.359+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827725960}) 2023-10-30T12:18:37.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.360+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.360+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97953198}) 2023-10-30T12:18:37.399 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.360+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115301333}) 2023-10-30T12:18:37.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.360+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.360+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97953198}) 2023-10-30T12:18:37.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.360+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321959060}) 2023-10-30T12:18:37.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.361+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.400 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.361+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2979282355}) 2023-10-30T12:18:37.401 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.361+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4072857715}) 2023-10-30T12:18:37.401 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.361+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.401 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.361+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2979282355}) 2023-10-30T12:18:37.401 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.361+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376381570}) 2023-10-30T12:18:37.401 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.362+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.401 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.362+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160588032}) 2023-10-30T12:18:37.402 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.362+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1546205048}) 2023-10-30T12:18:37.402 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.362+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.402 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.362+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160588032}) 2023-10-30T12:18:37.402 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.362+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821142500}) 2023-10-30T12:18:37.402 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.403 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270533391}) 2023-10-30T12:18:37.403 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1829620071}) 2023-10-30T12:18:37.403 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.403 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270533391}) 2023-10-30T12:18:37.403 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183142429}) 2023-10-30T12:18:37.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456202593}) 2023-10-30T12:18:37.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603574793}) 2023-10-30T12:18:37.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.404 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456202593}) 2023-10-30T12:18:37.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.363+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2924570564}) 2023-10-30T12:18:37.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.364+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.364+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851765885}) 2023-10-30T12:18:37.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.364+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3022597141}) 2023-10-30T12:18:37.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.364+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.405 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.364+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851765885}) 2023-10-30T12:18:37.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.364+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2056309430}) 2023-10-30T12:18:37.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.365+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.365+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1857161128}) 2023-10-30T12:18:37.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.365+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2332731453}) 2023-10-30T12:18:37.406 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.365+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.365+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1857161128}) 2023-10-30T12:18:37.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.365+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574729965}) 2023-10-30T12:18:37.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.366+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.366+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608507898}) 2023-10-30T12:18:37.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.366+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1432189231}) 2023-10-30T12:18:37.407 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.366+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.366+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608507898}) 2023-10-30T12:18:37.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.366+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309921950}) 2023-10-30T12:18:37.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.367+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.367+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195017571}) 2023-10-30T12:18:37.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.367+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1328063231}) 2023-10-30T12:18:37.408 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.367+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.367+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195017571}) 2023-10-30T12:18:37.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.367+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024057238}) 2023-10-30T12:18:37.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.368+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.368+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209442008}) 2023-10-30T12:18:37.409 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.368+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992493934}) 2023-10-30T12:18:37.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.368+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.368+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209442008}) 2023-10-30T12:18:37.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.368+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178925350}) 2023-10-30T12:18:37.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.369+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.369+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522786468}) 2023-10-30T12:18:37.410 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.369+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3139758823}) 2023-10-30T12:18:37.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.369+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.369+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522786468}) 2023-10-30T12:18:37.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.369+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1209041863}) 2023-10-30T12:18:37.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.370+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.370+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136210181}) 2023-10-30T12:18:37.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.370+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2471476422}) 2023-10-30T12:18:37.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.370+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.370+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136210181}) 2023-10-30T12:18:37.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.370+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52513733}) 2023-10-30T12:18:37.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.371+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.371+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573916272}) 2023-10-30T12:18:37.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.371+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618787650}) 2023-10-30T12:18:37.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.371+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.371+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573916272}) 2023-10-30T12:18:37.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.371+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613884275}) 2023-10-30T12:18:37.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.372+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.372+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2383976808}) 2023-10-30T12:18:37.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.372+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=786934601}) 2023-10-30T12:18:37.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.372+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.372+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2383976808}) 2023-10-30T12:18:37.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.372+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482022347}) 2023-10-30T12:18:37.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098584208}) 2023-10-30T12:18:37.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854363300}) 2023-10-30T12:18:37.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098584208}) 2023-10-30T12:18:37.415 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2313139161}) 2023-10-30T12:18:37.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408076077}) 2023-10-30T12:18:37.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350267196}) 2023-10-30T12:18:37.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.416 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408076077}) 2023-10-30T12:18:37.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.373+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3986973597}) 2023-10-30T12:18:37.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.374+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.374+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691774467}) 2023-10-30T12:18:37.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.374+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1179324574}) 2023-10-30T12:18:37.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.374+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.417 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.374+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691774467}) 2023-10-30T12:18:37.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.374+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2835210168}) 2023-10-30T12:18:37.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.375+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.375+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011799917}) 2023-10-30T12:18:37.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.375+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396115388}) 2023-10-30T12:18:37.418 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.375+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.375+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011799917}) 2023-10-30T12:18:37.419 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.375+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232844313}) 2023-10-30T12:18:37.442 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.376+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.442 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.376+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360868412}) 2023-10-30T12:18:37.443 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.376+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1726458037}) 2023-10-30T12:18:37.443 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.376+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.443 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.376+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360868412}) 2023-10-30T12:18:37.443 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.376+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628083025}) 2023-10-30T12:18:37.445 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.377+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.445 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.377+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407912649}) 2023-10-30T12:18:37.445 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.377+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912149702}) 2023-10-30T12:18:37.445 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.377+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.445 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.377+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407912649}) 2023-10-30T12:18:37.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.377+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434629727}) 2023-10-30T12:18:37.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3921800393}) 2023-10-30T12:18:37.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1239207222}) 2023-10-30T12:18:37.446 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3921800393}) 2023-10-30T12:18:37.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749706250}) 2023-10-30T12:18:37.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512006611}) 2023-10-30T12:18:37.447 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863317594}) 2023-10-30T12:18:37.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512006611}) 2023-10-30T12:18:37.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.378+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708007358}) 2023-10-30T12:18:37.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.379+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.379+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241317020}) 2023-10-30T12:18:37.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.379+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673405900}) 2023-10-30T12:18:37.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.379+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.379+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241317020}) 2023-10-30T12:18:37.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.379+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502439432}) 2023-10-30T12:18:37.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.380+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.380+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589577356}) 2023-10-30T12:18:37.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.380+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=336822540}) 2023-10-30T12:18:37.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.380+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.380+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589577356}) 2023-10-30T12:18:37.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.380+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=223651765}) 2023-10-30T12:18:37.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.381+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.381+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2675741568}) 2023-10-30T12:18:37.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.381+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3300882967}) 2023-10-30T12:18:37.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.381+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.381+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2675741568}) 2023-10-30T12:18:37.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.381+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3236759343}) 2023-10-30T12:18:37.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.382+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.382+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128605629}) 2023-10-30T12:18:37.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.382+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4139537419}) 2023-10-30T12:18:37.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.382+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.382+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128605629}) 2023-10-30T12:18:37.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.382+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1075954897}) 2023-10-30T12:18:37.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.383+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.383+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444702866}) 2023-10-30T12:18:37.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.383+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589524699}) 2023-10-30T12:18:37.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.383+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.383+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444702866}) 2023-10-30T12:18:37.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.383+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2983074676}) 2023-10-30T12:18:37.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857845013}) 2023-10-30T12:18:37.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86070861}) 2023-10-30T12:18:37.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857845013}) 2023-10-30T12:18:37.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731555081}) 2023-10-30T12:18:37.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2295594084}) 2023-10-30T12:18:37.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975204010}) 2023-10-30T12:18:37.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2295594084}) 2023-10-30T12:18:37.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.384+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=169339448}) 2023-10-30T12:18:37.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.385+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927218210}) 2023-10-30T12:18:37.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382634502}) 2023-10-30T12:18:37.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927218210}) 2023-10-30T12:18:37.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610073724}) 2023-10-30T12:18:37.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383334770}) 2023-10-30T12:18:37.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=331949818}) 2023-10-30T12:18:37.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383334770}) 2023-10-30T12:18:37.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.386+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302269689}) 2023-10-30T12:18:37.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.387+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.387+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163430644}) 2023-10-30T12:18:37.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.387+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1249838342}) 2023-10-30T12:18:37.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.387+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.387+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163430644}) 2023-10-30T12:18:37.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.387+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550325723}) 2023-10-30T12:18:37.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.388+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.388+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743123972}) 2023-10-30T12:18:37.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.388+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717527449}) 2023-10-30T12:18:37.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.388+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.388+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743123972}) 2023-10-30T12:18:37.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.388+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560869829}) 2023-10-30T12:18:37.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.389+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.389+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909506341}) 2023-10-30T12:18:37.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.389+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651330538}) 2023-10-30T12:18:37.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.389+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.389+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909506341}) 2023-10-30T12:18:37.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.389+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902449048}) 2023-10-30T12:18:37.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.390+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.390+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2284795085}) 2023-10-30T12:18:37.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.390+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101924886}) 2023-10-30T12:18:37.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.390+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.390+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2284795085}) 2023-10-30T12:18:37.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.390+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=58876722}) 2023-10-30T12:18:37.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.391+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.391+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900217170}) 2023-10-30T12:18:37.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.391+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4079404788}) 2023-10-30T12:18:37.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.391+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.391+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900217170}) 2023-10-30T12:18:37.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.391+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832930772}) 2023-10-30T12:18:37.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4277897956}) 2023-10-30T12:18:37.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343957560}) 2023-10-30T12:18:37.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4277897956}) 2023-10-30T12:18:37.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860426789}) 2023-10-30T12:18:37.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523226816}) 2023-10-30T12:18:37.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.392+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497208717}) 2023-10-30T12:18:37.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.393+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.393+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523226816}) 2023-10-30T12:18:37.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.393+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155209864}) 2023-10-30T12:18:37.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.393+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.393+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568818472}) 2023-10-30T12:18:37.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.393+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3167896856}) 2023-10-30T12:18:37.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.393+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568818472}) 2023-10-30T12:18:37.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1198289747}) 2023-10-30T12:18:37.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1961470043}) 2023-10-30T12:18:37.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2563202431}) 2023-10-30T12:18:37.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1961470043}) 2023-10-30T12:18:37.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.394+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771076285}) 2023-10-30T12:18:37.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.395+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.395+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074309693}) 2023-10-30T12:18:37.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.395+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3980190798}) 2023-10-30T12:18:37.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.395+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.395+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074309693}) 2023-10-30T12:18:37.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.395+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758833382}) 2023-10-30T12:18:37.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.396+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.396+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581908495}) 2023-10-30T12:18:37.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.396+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=261316602}) 2023-10-30T12:18:37.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.396+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.396+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581908495}) 2023-10-30T12:18:37.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.396+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3878908361}) 2023-10-30T12:18:37.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.397+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.397+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476977889}) 2023-10-30T12:18:37.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.397+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2804147553}) 2023-10-30T12:18:37.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.397+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.397+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476977889}) 2023-10-30T12:18:37.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.397+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186203083}) 2023-10-30T12:18:37.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.398+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.398+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2471957873}) 2023-10-30T12:18:37.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.398+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=528989238}) 2023-10-30T12:18:37.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.398+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.398+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2471957873}) 2023-10-30T12:18:37.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.398+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648972314}) 2023-10-30T12:18:37.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.399+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.399+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512075461}) 2023-10-30T12:18:37.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.399+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109675748}) 2023-10-30T12:18:37.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.399+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.399+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512075461}) 2023-10-30T12:18:37.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.399+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867343377}) 2023-10-30T12:18:37.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882909866}) 2023-10-30T12:18:37.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781157961}) 2023-10-30T12:18:37.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882909866}) 2023-10-30T12:18:37.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3690874988}) 2023-10-30T12:18:37.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414062847}) 2023-10-30T12:18:37.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654529855}) 2023-10-30T12:18:37.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414062847}) 2023-10-30T12:18:37.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.400+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065965671}) 2023-10-30T12:18:37.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.401+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975843218}) 2023-10-30T12:18:37.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=589029589}) 2023-10-30T12:18:37.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975843218}) 2023-10-30T12:18:37.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367941219}) 2023-10-30T12:18:37.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772176292}) 2023-10-30T12:18:37.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579083704}) 2023-10-30T12:18:37.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772176292}) 2023-10-30T12:18:37.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.402+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82172832}) 2023-10-30T12:18:37.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.403+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.403+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994873139}) 2023-10-30T12:18:37.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.403+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2286164518}) 2023-10-30T12:18:37.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.403+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.403+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1994873139}) 2023-10-30T12:18:37.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.403+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3755029189}) 2023-10-30T12:18:37.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.404+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.404+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120648600}) 2023-10-30T12:18:37.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.404+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3305002536}) 2023-10-30T12:18:37.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.404+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.404+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120648600}) 2023-10-30T12:18:37.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.404+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038721907}) 2023-10-30T12:18:37.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.405+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.405+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160491098}) 2023-10-30T12:18:37.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.405+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035324548}) 2023-10-30T12:18:37.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.405+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.405+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160491098}) 2023-10-30T12:18:37.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.405+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133671821}) 2023-10-30T12:18:37.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.406+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.406+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216271144}) 2023-10-30T12:18:37.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.406+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2683507507}) 2023-10-30T12:18:37.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.406+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.406+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216271144}) 2023-10-30T12:18:37.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.406+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278559366}) 2023-10-30T12:18:37.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.407+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.407+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81302063}) 2023-10-30T12:18:37.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.407+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906976199}) 2023-10-30T12:18:37.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.407+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.407+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81302063}) 2023-10-30T12:18:37.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.407+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3863575782}) 2023-10-30T12:18:37.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069914117}) 2023-10-30T12:18:37.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3799557153}) 2023-10-30T12:18:37.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069914117}) 2023-10-30T12:18:37.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2502842580}) 2023-10-30T12:18:37.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744482649}) 2023-10-30T12:18:37.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1207940011}) 2023-10-30T12:18:37.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744482649}) 2023-10-30T12:18:37.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.408+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4264518891}) 2023-10-30T12:18:37.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.409+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.409+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383878923}) 2023-10-30T12:18:37.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.409+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485526619}) 2023-10-30T12:18:37.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.409+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.409+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383878923}) 2023-10-30T12:18:37.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.409+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=727968084}) 2023-10-30T12:18:37.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784351453}) 2023-10-30T12:18:37.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731796440}) 2023-10-30T12:18:37.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784351453}) 2023-10-30T12:18:37.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=271802858}) 2023-10-30T12:18:37.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115891732}) 2023-10-30T12:18:37.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569888297}) 2023-10-30T12:18:37.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.489 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115891732}) 2023-10-30T12:18:37.489 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.410+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734392806}) 2023-10-30T12:18:37.489 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.411+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.489 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.411+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824838124}) 2023-10-30T12:18:37.489 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.411+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1269990672}) 2023-10-30T12:18:37.489 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.411+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.490 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.411+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824838124}) 2023-10-30T12:18:37.490 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.411+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3656446152}) 2023-10-30T12:18:37.490 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.490 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669178330}) 2023-10-30T12:18:37.490 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007816753}) 2023-10-30T12:18:37.491 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.491 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669178330}) 2023-10-30T12:18:37.491 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447429824}) 2023-10-30T12:18:37.491 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.491 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063165748}) 2023-10-30T12:18:37.492 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1797418183}) 2023-10-30T12:18:37.492 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.492 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063165748}) 2023-10-30T12:18:37.492 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.412+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143258334}) 2023-10-30T12:18:37.492 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.413+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.492 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.413+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67795144}) 2023-10-30T12:18:37.493 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.413+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2664737725}) 2023-10-30T12:18:37.493 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.413+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.493 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.413+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67795144}) 2023-10-30T12:18:37.493 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.413+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339010268}) 2023-10-30T12:18:37.493 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.414+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.494 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.414+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541490433}) 2023-10-30T12:18:37.494 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.414+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032191081}) 2023-10-30T12:18:37.494 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.414+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.494 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.414+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541490433}) 2023-10-30T12:18:37.494 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.414+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778174115}) 2023-10-30T12:18:37.494 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.415+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.495 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.415+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432755286}) 2023-10-30T12:18:37.495 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.415+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=813979399}) 2023-10-30T12:18:37.495 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.415+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.495 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.415+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432755286}) 2023-10-30T12:18:37.495 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.415+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1797805710}) 2023-10-30T12:18:37.496 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.416+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.496 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.416+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415659456}) 2023-10-30T12:18:37.496 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.416+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939980746}) 2023-10-30T12:18:37.496 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.416+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.496 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.416+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415659456}) 2023-10-30T12:18:37.496 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.416+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1139357220}) 2023-10-30T12:18:37.497 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.417+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.497 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.417+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1268864945}) 2023-10-30T12:18:37.497 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.417+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3510378336}) 2023-10-30T12:18:37.497 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.417+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.497 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.417+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1268864945}) 2023-10-30T12:18:37.498 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.417+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=952760905}) 2023-10-30T12:18:37.498 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.498 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013087060}) 2023-10-30T12:18:37.498 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3223534109}) 2023-10-30T12:18:37.498 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.499 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013087060}) 2023-10-30T12:18:37.499 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683829563}) 2023-10-30T12:18:37.499 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.499 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954344984}) 2023-10-30T12:18:37.499 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192192830}) 2023-10-30T12:18:37.499 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.500 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954344984}) 2023-10-30T12:18:37.500 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.418+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1213889600}) 2023-10-30T12:18:37.500 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.419+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.500 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.419+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=38,osdmap=20} crc {logm=1812818362,mdsmap=3984425268,mgr=413753915,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=4245497279,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2398621561,osd_pg_creating=2746269749,osd_snap=1461429948,osdmap=3355146738}) 2023-10-30T12:18:37.500 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.420+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=20} crc {logm=2361909562,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2579323660,mon_config_key=715115367,monmap=4204254258,osd_metadata=2479346626,osd_pg_creating=2746269749,osd_snap=3022349943,osdmap=3355146738}) 2023-10-30T12:18:37.501 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.420+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.501 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.420+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=38,osdmap=20} crc {logm=1812818362,mdsmap=3984425268,mgr=413753915,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=4245497279,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2398621561,osd_pg_creating=2746269749,osd_snap=1461429948,osdmap=3355146738}) 2023-10-30T12:18:37.501 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.420+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=39,osdmap=18} crc {logm=2361909562,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=244160203,mon_config_key=715115367,monmap=3140514009,osd_metadata=1267604196,osd_pg_creating=2746269749,osd_snap=3581021904,osdmap=1701910888}) 2023-10-30T12:18:37.501 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.425+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.501 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.425+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=98} crc {osdmap=3422037496}) 2023-10-30T12:18:37.501 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.425+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4134868759}) 2023-10-30T12:18:37.502 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.425+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:18:37.502 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.425+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=98} crc {osdmap=3422037496}) 2023-10-30T12:18:37.502 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:18:37.425+0000 7fec2d16e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=157312212}) 2023-10-30T12:18:38.359 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:18:38.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:38.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.475+0000 7f0c7cd9e640 10 monclient: get_monmap_and_config 2023-10-30T12:18:38.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.475+0000 7f0c7cd9e640 10 monclient: build_initial_monmap 2023-10-30T12:18:38.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.475+0000 7f0c7cd9e640 10 monclient: monmap: 2023-10-30T12:18:38.478 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:38.478 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.478 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:38.476890+0000 2023-10-30T12:18:38.479 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:38.476890+0000 2023-10-30T12:18:38.479 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:38.479 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.479 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:38.480 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:38.480 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:38.480 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:38.480 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:38.480 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.476+0000 7f0c7cd9e640 1 Processor -- start 2023-10-30T12:18:38.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.476+0000 7f0c7cd9e640 1 -- start start 2023-10-30T12:18:38.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.476+0000 7f0c7cd9e640 10 monclient: init 2023-10-30T12:18:38.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:38.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-10-30T12:18:38.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0c780a4b60 0x7f0c781326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient(hunting): picked mon.noname-a con 0x7f0c780a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c78099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0c78132be0 0x7f0c780a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient(hunting): picked mon.noname-c con 0x7f0c78132be0 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c780a4f40 con 0x7f0c78099a40 2023-10-30T12:18:38.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c780a3d70 con 0x7f0c780a4b60 2023-10-30T12:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c780a4150 con 0x7f0c78132be0 2023-10-30T12:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:38.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7cd9e640 10 monclient(hunting): authenticate will time out at 1963.660166s 2023-10-30T12:18:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0c78132be0 0x7f0c780a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0c780a4b60 0x7f0c781326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0c78132be0 0x7f0c780a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:57864/0 (socket says 172.21.15.50:57864) 2023-10-30T12:18:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c75d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0c78099a40 0x7f0c780a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53106/0 (socket says 172.21.15.50:53106) 2023-10-30T12:18:38.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 -- 172.21.15.50:0/2057815325 learned_addr learned my addr 172.21.15.50:0/2057815325 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): get_auth_request con 0x7f0c78132be0 auth_method 0 2023-10-30T12:18:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 10 monclient(hunting): get_auth_request con 0x7f0c780a4b60 auth_method 0 2023-10-30T12:18:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient(hunting): handle_auth_done global_id 204475 payload 274 2023-10-30T12:18:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 -- 172.21.15.50:0/2057815325 >> v2:172.21.15.50:3300/0 conn(0x7f0c780a4b60 msgr2=0x7f0c781326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 --2- 172.21.15.50:0/2057815325 >> v2:172.21.15.50:3300/0 conn(0x7f0c780a4b60 0x7f0c781326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 -- 172.21.15.50:0/2057815325 >> v1:172.21.15.50:6789/0 conn(0x7f0c78099a40 legacy=0x7f0c780a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 monclient: found mon.noname-c 2023-10-30T12:18:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:18:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.477+0000 7f0c76d9d640 1 -- 172.21.15.50:0/2057815325 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7809a440 con 0x7f0c78132be0 2023-10-30T12:18:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.478+0000 7f0c76d9d640 1 --2- 172.21.15.50:0/2057815325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c78132be0 0x7f0c780a2a60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0c780a3a20 tx=0x7f0c7002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce21c4605d438953 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.478+0000 7f0c7559a640 1 -- 172.21.15.50:0/2057815325 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7002c9f0 con 0x7f0c78132be0 2023-10-30T12:18:38.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.478+0000 7f0c7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: dump: 2023-10-30T12:18:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.498 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.499 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: _finish_auth 0 2023-10-30T12:18:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 1 -- 172.21.15.50:0/2057815325 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7002cb50 con 0x7f0c78132be0 2023-10-30T12:18:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 1 -- 172.21.15.50:0/2057815325 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c700343f0 con 0x7f0c78132be0 2023-10-30T12:18:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 5 monclient: authenticate success, global_id 204475 2023-10-30T12:18:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 4 set_mon_vals no callback set 2023-10-30T12:18:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7559a640 10 monclient: dump: 2023-10-30T12:18:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.503 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 10 monclient: shutdown 2023-10-30T12:18:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2057815325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c78132be0 msgr2=0x7f0c780a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2057815325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c78132be0 0x7f0c780a2a60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0c780a3a20 tx=0x7f0c7002be50 comp rx=0 tx=0).stop 2023-10-30T12:18:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2057815325 shutdown_connections 2023-10-30T12:18:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2057815325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c78132be0 0x7f0c780a2a60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2057815325 >> v2:172.21.15.50:3300/0 conn(0x7f0c780a4b60 0x7f0c781326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2057815325 >> 172.21.15.50:0/2057815325 conn(0x7f0c780a8060 msgr2=0x7f0c780a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2057815325 shutdown_connections 2023-10-30T12:18:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.479+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2057815325 wait complete. 2023-10-30T12:18:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 10 monclient: build_initial_monmap 2023-10-30T12:18:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 10 monclient: monmap: 2023-10-30T12:18:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:38.507 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 1 librados: starting msgr at 2023-10-30T12:18:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 1 librados: starting objecter 2023-10-30T12:18:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 1 Processor -- start 2023-10-30T12:18:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 1 -- start start 2023-10-30T12:18:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 1 librados: setting wanted keys 2023-10-30T12:18:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 1 librados: calling monclient init 2023-10-30T12:18:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.480+0000 7f0c7cd9e640 10 monclient: init 2023-10-30T12:18:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c780a4b60 0x7f0c7811e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): picked mon.noname-a con 0x7f0c780a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c78132be0 0x7f0c7811ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): picked mon.noname-c con 0x7f0c78132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c7811f2c0 0x7f0c781d0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): picked mon.noname-b con 0x7f0c7811f2c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78121fc0 con 0x7f0c780a4b60 2023-10-30T12:18:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78120e10 con 0x7f0c7811f2c0 2023-10-30T12:18:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780a18d0 con 0x7f0c78132be0 2023-10-30T12:18:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7cd9e640 10 monclient(hunting): authenticate will time out at 1963.664468s 2023-10-30T12:18:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c780a4b60 0x7f0c7811e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c78132be0 0x7f0c7811ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c780a4b60 0x7f0c7811e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55124/0 (socket says 172.21.15.50:55124) 2023-10-30T12:18:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 1 -- 172.21.15.50:0/2428525019 learned_addr learned my addr 172.21.15.50:0/2428525019 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 10 monclient(hunting): get_auth_request con 0x7f0c780a4b60 auth_method 0 2023-10-30T12:18:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): get_auth_request con 0x7f0c78132be0 auth_method 0 2023-10-30T12:18:38.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c7811f2c0 0x7f0c781d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 10 monclient(hunting): get_auth_request con 0x7f0c7811f2c0 auth_method 0 2023-10-30T12:18:38.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient(hunting): handle_auth_done global_id 224207 payload 824 2023-10-30T12:18:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 1 -- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c7811f2c0 msgr2=0x7f0c781d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c7811f2c0 0x7f0c781d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 1 -- 172.21.15.50:0/2428525019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c780a4b60 msgr2=0x7f0c7811e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c780a4b60 0x7f0c7811e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 1 monclient: found mon.noname-c 2023-10-30T12:18:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:38.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.481+0000 7f0c75d9b640 1 -- 172.21.15.50:0/2428525019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70009b30 con 0x7f0c78132be0 2023-10-30T12:18:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c75d9b640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c78132be0 0x7f0c7811ed80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800c390 tx=0x7f0c6800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 client.?.objecter ms_handle_connect 0x7f0c78132be0 2023-10-30T12:18:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 1 -- 172.21.15.50:0/2428525019 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680074e0 con 0x7f0c78132be0 2023-10-30T12:18:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: dump: 2023-10-30T12:18:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.520 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.521 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.522 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:38.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 5 monclient: authenticate success, global_id 224207 2023-10-30T12:18:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 1 -- 172.21.15.50:0/2428525019 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c68007640 con 0x7f0c78132be0 2023-10-30T12:18:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 1 -- 172.21.15.50:0/2428525019 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68004a00 con 0x7f0c78132be0 2023-10-30T12:18:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c66ffd640 10 monclient: dump: 2023-10-30T12:18:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.524 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.525 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.525 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c74d99640 4 set_mon_vals no callback set 2023-10-30T12:18:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 10 monclient: _renew_subs 2023-10-30T12:18:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c781229a0 con 0x7f0c78132be0 2023-10-30T12:18:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 10 client.224207.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 10 monclient: _renew_subs 2023-10-30T12:18:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c78131fa0 con 0x7f0c78132be0 2023-10-30T12:18:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.482+0000 7f0c7cd9e640 1 librados: init done 2023-10-30T12:18:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c66ffd640 1 -- 172.21.15.50:0/2428525019 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0c68004b60 con 0x7f0c78132be0 2023-10-30T12:18:38.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c7cd9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c7cd9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c7cd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7804e160 con 0x7f0c78132be0 2023-10-30T12:18:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c66ffd640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c580392f0 0x7f0c5803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c66ffd640 1 -- 172.21.15.50:0/2428525019 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(561..561 src has 490..561) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f0c6804a4a0 con 0x7f0c78132be0 2023-10-30T12:18:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c66ffd640 10 client.224207.objecter ms_dispatch 0x7f0c78000b90 osd_map(561..561 src has 490..561) v4 2023-10-30T12:18:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c66ffd640 3 client.224207.objecter handle_osd_map got epochs [561,561] > 0 2023-10-30T12:18:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c66ffd640 3 client.224207.objecter handle_osd_map decoding full epoch 561 2023-10-30T12:18:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c66ffd640 20 client.224207.objecter dump_active .. 0 homeless 2023-10-30T12:18:38.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c7659c640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c580392f0 0x7f0c5803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.483+0000 7f0c7659c640 10 monclient: get_auth_request con 0x7f0c580392f0 auth_method 0 2023-10-30T12:18:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.484+0000 7f0c7659c640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c580392f0 0x7f0c5803b7b0 secure :-1 s=READY pgs=4270 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009c00 tx=0x7f0c6c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.484+0000 7f0c66ffd640 10 client.224207.objecter ms_handle_connect 0x7f0c580392f0 2023-10-30T12:18:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.488+0000 7f0c66ffd640 1 -- 172.21.15.50:0/2428525019 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c680192c0 con 0x7f0c78132be0 2023-10-30T12:18:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.488+0000 7f0c66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:38.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.488+0000 7f0c66ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:38.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.731+0000 7f0c7cd9e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.731+0000 7f0c7cd9e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.731+0000 7f0c7cd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.731+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0c7809e5d0 con 0x7f0c78132be0 2023-10-30T12:18:38.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.731+0000 7f0c66ffd640 1 -- 172.21.15.50:0/2428525019 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f0c6801aac0 con 0x7f0c78132be0 2023-10-30T12:18:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.731+0000 7f0c66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.731+0000 7f0c66ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:38.734 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 10 librados: watch_flush enter 2023-10-30T12:18:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 10 librados: watch_flush exit 2023-10-30T12:18:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 20 client.224207.objecter shutdown clearing up homeless session... 2023-10-30T12:18:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 10 client.224207.objecter successfully canceled tick 2023-10-30T12:18:38.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c580392f0 msgr2=0x7f0c5803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c580392f0 0x7f0c5803b7b0 secure :-1 s=READY pgs=4270 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009c00 tx=0x7f0c6c009190 comp rx=0 tx=0).stop 2023-10-30T12:18:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 10 monclient: shutdown 2023-10-30T12:18:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c78132be0 msgr2=0x7f0c7811ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c78132be0 0x7f0c7811ed80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800c390 tx=0x7f0c6800c850 comp rx=0 tx=0).stop 2023-10-30T12:18:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 shutdown_connections 2023-10-30T12:18:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0c580392f0 0x7f0c5803b7b0 unknown :-1 s=CLOSED pgs=4270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0c7811f2c0 0x7f0c781d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c78132be0 0x7f0c7811ed80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 --2- 172.21.15.50:0/2428525019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c780a4b60 0x7f0c7811e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 >> 172.21.15.50:0/2428525019 conn(0x7f0c780a8060 msgr2=0x7f0c78134530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 shutdown_connections 2023-10-30T12:18:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 -- 172.21.15.50:0/2428525019 wait complete. 2023-10-30T12:18:38.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.733+0000 7f0c7cd9e640 1 librados: shutdown 2023-10-30T12:18:38.745 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:18:38.745 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:18:38.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:18:38.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:18:38.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.850+0000 7f7cbca39640 10 monclient: get_monmap_and_config 2023-10-30T12:18:38.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.850+0000 7f7cbca39640 10 monclient: build_initial_monmap 2023-10-30T12:18:38.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.850+0000 7f7cbca39640 10 monclient: monmap: 2023-10-30T12:18:38.853 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:38.853 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.853 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:38.851859+0000 2023-10-30T12:18:38.854 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:38.851859+0000 2023-10-30T12:18:38.854 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:38.854 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.854 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:38.854 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:38.855 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:38.855 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:38.855 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:38.855 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 1 Processor -- start 2023-10-30T12:18:38.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 1 -- start start 2023-10-30T12:18:38.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient: init 2023-10-30T12:18:38.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:38.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T12:18:38.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7cb8097780 0x7f7cb8097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb8097780 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:38.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7cb8073b10 0x7f7cb80a0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient(hunting): picked mon.noname-a con 0x7f7cb8073b10 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:38.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cb809ccd0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:38.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb8071ec0 con 0x7f7cb809ccd0 2023-10-30T12:18:38.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb809bd10 con 0x7f7cb8073b10 2023-10-30T12:18:38.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb809bff0 con 0x7f7cb8097780 2023-10-30T12:18:38.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.851+0000 7f7cbca39640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:38.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cbca39640 10 monclient(hunting): authenticate will time out at 1964.035013s 2023-10-30T12:18:38.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7cb8073b10 0x7f7cb80a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7cb8073b10 0x7f7cb80a0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55140/0 (socket says 172.21.15.50:55140) 2023-10-30T12:18:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 -- 172.21.15.50:0/552300388 learned_addr learned my addr 172.21.15.50:0/552300388 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): get_auth_request con 0x7f7cb8073b10 auth_method 0 2023-10-30T12:18:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb659c640 1 --2- 172.21.15.50:0/552300388 >> v2:172.21.15.118:3300/0 conn(0x7f7cb8097780 0x7f7cb8097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient(hunting): handle_auth_done global_id 234153 payload 274 2023-10-30T12:18:38.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 -- 172.21.15.50:0/552300388 >> v2:172.21.15.118:3300/0 conn(0x7f7cb8097780 msgr2=0x7f7cb8097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:38.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 --2- 172.21.15.50:0/552300388 >> v2:172.21.15.118:3300/0 conn(0x7f7cb8097780 0x7f7cb8097b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 -- 172.21.15.50:0/552300388 >> v1:172.21.15.118:6789/0 conn(0x7f7cb809ccd0 legacy=0x7f7cb80a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:38.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:38.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 monclient: found mon.noname-a 2023-10-30T12:18:38.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:38.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:38.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb5d9b640 1 -- 172.21.15.50:0/552300388 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80980a0 con 0x7f7cb8073b10 2023-10-30T12:18:38.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb659c640 10 monclient: get_auth_request con 0x7f7cb8097780 auth_method 0 2023-10-30T12:18:38.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.852+0000 7f7cb659c640 1 --2- 172.21.15.50:0/552300388 >> v2:172.21.15.118:3300/0 conn(0x7f7cb8097780 0x7f7cb8097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:18:38.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb5d9b640 1 --2- 172.21.15.50:0/552300388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb8073b10 0x7f7cb80a0080 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7cb806a280 tx=0x7f7cac02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf523d458323abbd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 1 -- 172.21.15.50:0/552300388 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac031750 con 0x7f7cb8073b10 2023-10-30T12:18:38.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:38.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: dump: 2023-10-30T12:18:38.875 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.875 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.875 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.876 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.876 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.876 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.876 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.876 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.876 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: _finish_auth 0 2023-10-30T12:18:38.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:38.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:38.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 5 monclient: authenticate success, global_id 234153 2023-10-30T12:18:38.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:38.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 1 -- 172.21.15.50:0/552300388 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac0318b0 con 0x7f7cb8073b10 2023-10-30T12:18:38.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:38.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:38.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 4 set_mon_vals no callback set 2023-10-30T12:18:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 1 -- 172.21.15.50:0/552300388 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac031c40 con 0x7f7cb8073b10 2023-10-30T12:18:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cb559a640 10 monclient: dump: 2023-10-30T12:18:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.879 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.880 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.881 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 10 monclient: shutdown 2023-10-30T12:18:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:38.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 1 -- 172.21.15.50:0/552300388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb8073b10 msgr2=0x7f7cb80a0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.853+0000 7f7cbca39640 1 --2- 172.21.15.50:0/552300388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb8073b10 0x7f7cb80a0080 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7cb806a280 tx=0x7f7cac02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 -- 172.21.15.50:0/552300388 shutdown_connections 2023-10-30T12:18:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 --2- 172.21.15.50:0/552300388 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb8073b10 0x7f7cb80a0080 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 --2- 172.21.15.50:0/552300388 >> v2:172.21.15.118:3300/0 conn(0x7f7cb8097780 0x7f7cb8097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 -- 172.21.15.50:0/552300388 >> 172.21.15.50:0/552300388 conn(0x7f7cb80a1170 msgr2=0x7f7cb80a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 -- 172.21.15.50:0/552300388 shutdown_connections 2023-10-30T12:18:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 -- 172.21.15.50:0/552300388 wait complete. 2023-10-30T12:18:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 10 monclient: build_initial_monmap 2023-10-30T12:18:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 10 monclient: monmap: 2023-10-30T12:18:38.883 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:38.884 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:38.884 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:38.884 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:38.884 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:38.884 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:38.885 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 librados: starting msgr at 2023-10-30T12:18:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.854+0000 7f7cbca39640 1 librados: starting objecter 2023-10-30T12:18:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 Processor -- start 2023-10-30T12:18:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 -- start start 2023-10-30T12:18:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 librados: setting wanted keys 2023-10-30T12:18:38.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 librados: calling monclient init 2023-10-30T12:18:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient: init 2023-10-30T12:18:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb8073b10 0x7f7cb808e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): picked mon.noname-b con 0x7f7cb8073b10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:38.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb8097780 0x7f7cb808eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb8097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:38.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb809ccd0 0x7f7cb808f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): picked mon.noname-a con 0x7f7cb809ccd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:38.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8092810 con 0x7f7cb809ccd0 2023-10-30T12:18:38.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8090a90 con 0x7f7cb8073b10 2023-10-30T12:18:38.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8073f70 con 0x7f7cb8097780 2023-10-30T12:18:38.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:38.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cbca39640 10 monclient(hunting): authenticate will time out at 1964.038534s 2023-10-30T12:18:38.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb809ccd0 0x7f7cb808f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb8097780 0x7f7cb808eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb809ccd0 0x7f7cb808f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55142/0 (socket says 172.21.15.50:55142) 2023-10-30T12:18:38.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 10 monclient(hunting): get_auth_request con 0x7f7cb8097780 auth_method 0 2023-10-30T12:18:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 -- 172.21.15.50:0/192307015 learned_addr learned my addr 172.21.15.50:0/192307015 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): get_auth_request con 0x7f7cb809ccd0 auth_method 0 2023-10-30T12:18:38.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb659c640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb8073b10 0x7f7cb808e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb659c640 10 monclient(hunting): get_auth_request con 0x7f7cb8073b10 auth_method 0 2023-10-30T12:18:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient(hunting): handle_auth_done global_id 234156 payload 824 2023-10-30T12:18:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb8097780 msgr2=0x7f7cb808eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7cb8097780 0x7f7cb808eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb8073b10 msgr2=0x7f7cb808e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb8073b10 0x7f7cb808e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 monclient: found mon.noname-a 2023-10-30T12:18:38.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb6d9d640 1 -- 172.21.15.50:0/192307015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0094d0 con 0x7f7cb809ccd0 2023-10-30T12:18:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.855+0000 7f7cb659c640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7cb8073b10 0x7f7cb808e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:18:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cb6d9d640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb809ccd0 0x7f7cb808f070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000c7b0 tx=0x7f7cb000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 client.?.objecter ms_handle_connect 0x7f7cb809ccd0 2023-10-30T12:18:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0004b60 con 0x7f7cb809ccd0 2023-10-30T12:18:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: dump: 2023-10-30T12:18:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.898 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.899 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.899 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb0004cc0 con 0x7f7cb809ccd0 2023-10-30T12:18:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 5 monclient: authenticate success, global_id 234156 2023-10-30T12:18:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 10 monclient: _renew_subs 2023-10-30T12:18:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb80a3b20 con 0x7f7cb809ccd0 2023-10-30T12:18:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb000aa90 con 0x7f7cb809ccd0 2023-10-30T12:18:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7ca6ffd640 10 monclient: dump: 2023-10-30T12:18:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.902 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.903 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.904 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cb4d99640 4 set_mon_vals no callback set 2023-10-30T12:18:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 10 client.234156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 10 monclient: _renew_subs 2023-10-30T12:18:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:18:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb8094db0 con 0x7f7cb809ccd0 2023-10-30T12:18:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.856+0000 7f7cbca39640 1 librados: init done 2023-10-30T12:18:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cbca39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cbca39640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808afd0 0x7f7cb808b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cbca39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb806a5a0 con 0x7f7cb808afd0 2023-10-30T12:18:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808afd0 0x7f7cb808b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient: get_auth_request con 0x7f7cb808afd0 auth_method 0 2023-10-30T12:18:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7cb000abf0 con 0x7f7cb809ccd0 2023-10-30T12:18:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 10 monclient(hunting): handle_auth_done global_id 234159 payload 824 2023-10-30T12:18:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 1 -- 172.21.15.50:0/192307015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7cb8091ec0 con 0x7f7cb808afd0 2023-10-30T12:18:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c980392f0 0x7f7c9803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(561..561 src has 490..561) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f7cb004ada0 con 0x7f7cb809ccd0 2023-10-30T12:18:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 10 client.234156.objecter ms_dispatch 0x7f7cb8000b90 osd_map(561..561 src has 490..561) v4 2023-10-30T12:18:38.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 3 client.234156.objecter handle_osd_map got epochs [561,561] > 0 2023-10-30T12:18:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 3 client.234156.objecter handle_osd_map decoding full epoch 561 2023-10-30T12:18:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb659c640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808afd0 0x7f7cb808b3b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800f510 tx=0x7f7ca800f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 20 client.234156.objecter dump_active .. 0 homeless 2023-10-30T12:18:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 10 client.234156.objecter ms_handle_connect 0x7f7cb808afd0 2023-10-30T12:18:38.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 10 client.234156.objecter resend_mon_ops 2023-10-30T12:18:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8004230 con 0x7f7cb808afd0 2023-10-30T12:18:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7ca6ffd640 10 monclient: dump: 2023-10-30T12:18:38.910 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.912 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb5d9b640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c980392f0 0x7f7c9803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.857+0000 7f7cb5d9b640 10 monclient: get_auth_request con 0x7f7c980392f0 auth_method 0 2023-10-30T12:18:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f7ca800c030 con 0x7f7cb808afd0 2023-10-30T12:18:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808afd0 msgr2=0x7f7cb808b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808afd0 0x7f7cb808b3b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800f510 tx=0x7f7ca800f9d0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808afd0 msgr2=0x7f7cb808b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808afd0 0x7f7cb808b3b0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800f510 tx=0x7f7ca800f9d0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7cb5d9b640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c980392f0 0x7f7c9803b7b0 secure :-1 s=READY pgs=4271 cs=0 l=1 rev1=1 crypto rx=0x7f7cac035000 tx=0x7f7cac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.858+0000 7f7ca6ffd640 10 client.234156.objecter ms_handle_connect 0x7f7c980392f0 2023-10-30T12:18:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cbca39640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:38.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cbca39640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808b8f0 0x7f7cb81d46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cbca39640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8092350 con 0x7f7cb808b8f0 2023-10-30T12:18:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808b8f0 0x7f7cb81d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient: get_auth_request con 0x7f7cb808b8f0 auth_method 0 2023-10-30T12:18:38.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 10 monclient(hunting): handle_auth_done global_id 234162 payload 824 2023-10-30T12:18:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.864+0000 7f7cb659c640 1 -- 172.21.15.50:0/192307015 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7cb80a0cd0 con 0x7f7cb808b8f0 2023-10-30T12:18:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 10 client.234156.objecter ms_handle_connect 0x7f7cb808b8f0 2023-10-30T12:18:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 10 client.234156.objecter resend_mon_ops 2023-10-30T12:18:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cb659c640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808b8f0 0x7f7cb81d46b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800f510 tx=0x7f7ca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:38.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8003df0 con 0x7f7cb808b8f0 2023-10-30T12:18:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 10 monclient: dump: 2023-10-30T12:18:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:38.922 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f7ca8004520 con 0x7f7cb808b8f0 2023-10-30T12:18:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808b8f0 msgr2=0x7f7cb81d46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808b8f0 0x7f7cb81d46b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800f510 tx=0x7f7ca80046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808b8f0 msgr2=0x7f7cb81d46b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7ca6ffd640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb808b8f0 0x7f7cb81d46b0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800f510 tx=0x7f7ca80046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:38.924 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:38.924 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:38.924 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:38.924 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:18:38.925 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 190, 2023-10-30T12:18:38.925 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:38.925 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:18:38.925 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:38.925 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:38.926 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:38.926 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:18:38.926 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:38.926 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:38.926 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:38.926 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:38.927 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:38.927 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:38.927 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:38.927 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:38.927 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:38.928 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:38.928 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:38.928 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:38.928 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:38.928 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:38.928 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:38.929 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:38.929 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:38.929 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:38.929 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:38.929 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:38.930 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:38.930 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:38.930 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:38.930 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:38.930 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:38.931 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:38.931 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:38.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.931 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:38.931 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:38.931 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:38.932 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:38.932 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:38.932 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:38.932 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:38.933 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:38.933 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:38.933 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:38.933 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:38.933 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:38.934 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:38.934 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:38.934 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:38.934 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:38.934 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:38.935 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:38.935 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:38.935 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:38.935 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:38.935 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:38.936 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:38.936 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:38.936 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:38.936 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:38.936 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:38.936 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:38.937 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:38.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:38.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.937 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:38.938 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:38.938 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:38.938 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:38.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:38.939 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:38.939 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:38.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:38.939 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:38.940 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:38.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:38.940 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:38.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.940 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:38.941 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:38.941 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:38.941 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:38.941 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:38.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.942 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:38.942 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:38.942 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:38.942 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:38.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.943 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:38.943 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:38.943 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:38.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:38.944 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:38.944 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:38.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:38.945 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.946 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.948 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:38.949 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:38.950 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:38.952 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:38.953 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:18:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:38.954 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:38.955 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 10 librados: watch_flush enter 2023-10-30T12:18:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 10 librados: watch_flush exit 2023-10-30T12:18:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 20 client.234156.objecter shutdown clearing up homeless session... 2023-10-30T12:18:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 10 client.234156.objecter successfully canceled tick 2023-10-30T12:18:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c980392f0 msgr2=0x7f7c9803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f7c980392f0 0x7f7c9803b7b0 secure :-1 s=READY pgs=4271 cs=0 l=1 rev1=1 crypto rx=0x7f7cac035000 tx=0x7f7cac008030 comp rx=0 tx=0).stop 2023-10-30T12:18:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 10 monclient: shutdown 2023-10-30T12:18:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb809ccd0 msgr2=0x7f7cb808f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 --2- 172.21.15.50:0/192307015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cb809ccd0 0x7f7cb808f070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000c7b0 tx=0x7f7cb000cc70 comp rx=0 tx=0).stop 2023-10-30T12:18:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cb6d9d640 1 -- 172.21.15.50:0/192307015 reap_dead start 2023-10-30T12:18:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cb6d9d640 1 -- 172.21.15.50:0/192307015 reap_dead start 2023-10-30T12:18:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 shutdown_connections 2023-10-30T12:18:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 >> 172.21.15.50:0/192307015 conn(0x7f7cb80a1170 msgr2=0x7f7cb80a2570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 shutdown_connections 2023-10-30T12:18:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 -- 172.21.15.50:0/192307015 wait complete. 2023-10-30T12:18:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:38.865+0000 7f7cbca39640 1 librados: shutdown 2023-10-30T12:18:38.960 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:18:38.960 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:18:38.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:18:39.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.080+0000 7fc45ba26640 10 monclient: get_monmap_and_config 2023-10-30T12:18:39.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.080+0000 7fc45ba26640 10 monclient: build_initial_monmap 2023-10-30T12:18:39.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.081+0000 7fc45ba26640 10 monclient: monmap: 2023-10-30T12:18:39.083 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:39.083 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.083 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:39.082047+0000 2023-10-30T12:18:39.083 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:39.082047+0000 2023-10-30T12:18:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:39.085 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:39.085 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.081+0000 7fc45ba26640 1 Processor -- start 2023-10-30T12:18:39.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.081+0000 7fc45ba26640 1 -- start start 2023-10-30T12:18:39.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.081+0000 7fc45ba26640 10 monclient: init 2023-10-30T12:18:39.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:39.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:18:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4540a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2450 0x7fc45413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient(hunting): picked mon.noname-b con 0x7fc4540a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc45413af40 0x7fc45413d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient(hunting): picked mon.noname-a con 0x7fc45413af40 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:39.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4540a4190 con 0x7fc4540a2ae0 2023-10-30T12:18:39.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4540a1630 con 0x7fc45413af40 2023-10-30T12:18:39.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45409a2e0 con 0x7fc4540a2450 2023-10-30T12:18:39.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:39.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc45ba26640 10 monclient(hunting): authenticate will time out at 1964.265280s 2023-10-30T12:18:39.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc45413af40 0x7fc45413d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2450 0x7fc45413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc4597c2640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc4540a2ae0 0x7fc4540a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53112/0 (socket says 172.21.15.50:53112) 2023-10-30T12:18:39.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc45413af40 0x7fc45413d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55164/0 (socket says 172.21.15.50:55164) 2023-10-30T12:18:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc4597c2640 1 -- 172.21.15.50:0/2985520391 learned_addr learned my addr 172.21.15.50:0/2985520391 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 10 monclient(hunting): get_auth_request con 0x7fc4540a2450 auth_method 0 2023-10-30T12:18:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): get_auth_request con 0x7fc45413af40 auth_method 0 2023-10-30T12:18:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient(hunting): handle_auth_done global_id 234165 payload 274 2023-10-30T12:18:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 1 -- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2450 msgr2=0x7fc45413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 1 --2- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2450 0x7fc45413aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 1 -- 172.21.15.50:0/2985520391 >> v1:172.21.15.50:6789/0 conn(0x7fc4540a2ae0 legacy=0x7fc4540a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient: _finish_hunting 0 2023-10-30T12:18:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 1 monclient: found mon.noname-a 2023-10-30T12:18:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc459fc3640 1 -- 172.21.15.50:0/2985520391 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45409a8a0 con 0x7fc45413af40 2023-10-30T12:18:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.082+0000 7fc458fc1640 1 --2- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2450 0x7fc45413aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc459fc3640 1 --2- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 0x7fc45413d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009d90 tx=0x7fc44c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7b183db6fe34df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 1 -- 172.21.15.50:0/2985520391 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c02cb30 con 0x7fc45413af40 2023-10-30T12:18:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: dump: 2023-10-30T12:18:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.102 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.104 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: _finish_auth 0 2023-10-30T12:18:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:39.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 1 -- 172.21.15.50:0/2985520391 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c02cc90 con 0x7fc45413af40 2023-10-30T12:18:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc45ba26640 5 monclient: authenticate success, global_id 234165 2023-10-30T12:18:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc45ba26640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc45ba26640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 1 -- 172.21.15.50:0/2985520391 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c034440 con 0x7fc45413af40 2023-10-30T12:18:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc45ba26640 4 set_mon_vals no callback set 2023-10-30T12:18:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.083+0000 7fc44bfff640 10 monclient: dump: 2023-10-30T12:18:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.107 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.108 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 10 monclient: shutdown 2023-10-30T12:18:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:39.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 -- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 msgr2=0x7fc45413d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 --2- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 0x7fc45413d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009d90 tx=0x7fc44c02bd80 comp rx=0 tx=0).stop 2023-10-30T12:18:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 -- 172.21.15.50:0/2985520391 shutdown_connections 2023-10-30T12:18:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 --2- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 0x7fc45413d330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 --2- 172.21.15.50:0/2985520391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2450 0x7fc45413aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 -- 172.21.15.50:0/2985520391 >> 172.21.15.50:0/2985520391 conn(0x7fc4540a80b0 msgr2=0x7fc4540a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 -- 172.21.15.50:0/2985520391 shutdown_connections 2023-10-30T12:18:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 1 -- 172.21.15.50:0/2985520391 wait complete. 2023-10-30T12:18:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 10 monclient: build_initial_monmap 2023-10-30T12:18:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.084+0000 7fc45ba26640 10 monclient: monmap: 2023-10-30T12:18:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:39.111 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:39.112 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 librados: starting msgr at 2023-10-30T12:18:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 librados: starting objecter 2023-10-30T12:18:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 Processor -- start 2023-10-30T12:18:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 -- start start 2023-10-30T12:18:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 librados: setting wanted keys 2023-10-30T12:18:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 librados: calling monclient init 2023-10-30T12:18:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient: init 2023-10-30T12:18:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:18:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4540a2450 0x7fc4541ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): picked mon.noname-b con 0x7fc4540a2450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2ae0 0x7fc454136e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): picked mon.noname-c con 0x7fc4540a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 0x7fc454137350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): picked mon.noname-a con 0x7fc45413af40 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45411e940 con 0x7fc45413af40 2023-10-30T12:18:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540a1790 con 0x7fc4540a2450 2023-10-30T12:18:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45413a670 con 0x7fc4540a2ae0 2023-10-30T12:18:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc45ba26640 10 monclient(hunting): authenticate will time out at 1964.268632s 2023-10-30T12:18:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2ae0 0x7fc454136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2ae0 0x7fc454136e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51622/0 (socket says 172.21.15.50:51622) 2023-10-30T12:18:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 0x7fc454137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 1 -- 172.21.15.50:0/1491600508 learned_addr learned my addr 172.21.15.50:0/1491600508 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): get_auth_request con 0x7fc4540a2ae0 auth_method 0 2023-10-30T12:18:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 10 monclient(hunting): get_auth_request con 0x7fc45413af40 auth_method 0 2023-10-30T12:18:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc459fc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.085+0000 7fc458fc1640 10 monclient(hunting): handle_auth_done global_id 224216 payload 824 2023-10-30T12:18:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc4597c2640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4540a2450 0x7fc4541ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4540a2450 msgr2=0x7fc4541ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4540a2450 0x7fc4541ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 msgr2=0x7fc454137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 0x7fc454137350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 10 monclient: _finish_hunting 0 2023-10-30T12:18:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 1 monclient: found mon.noname-c 2023-10-30T12:18:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 1 -- 172.21.15.50:0/1491600508 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444009400 con 0x7fc4540a2ae0 2023-10-30T12:18:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc4597c2640 10 monclient: get_auth_request con 0x7fc4540a2450 auth_method 0 2023-10-30T12:18:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc459fc3640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45413af40 0x7fc454137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:39.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc4597c2640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4540a2450 0x7fc4541ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:18:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc458fc1640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2ae0 0x7fc454136e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc44400ec10 tx=0x7fc44400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 client.?.objecter ms_handle_connect 0x7fc4540a2ae0 2023-10-30T12:18:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444009760 con 0x7fc4540a2ae0 2023-10-30T12:18:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: dump: 2023-10-30T12:18:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.124 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.125 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: _finish_auth 0 2023-10-30T12:18:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:18:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44400cc30 con 0x7fc4540a2ae0 2023-10-30T12:18:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 5 monclient: authenticate success, global_id 224216 2023-10-30T12:18:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 10 monclient: _renew_subs 2023-10-30T12:18:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c009a70 con 0x7fc4540a2ae0 2023-10-30T12:18:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444013470 con 0x7fc4540a2ae0 2023-10-30T12:18:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc449ffb640 10 monclient: dump: 2023-10-30T12:18:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.129 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.130 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc44b7fe640 4 set_mon_vals no callback set 2023-10-30T12:18:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 10 client.224216.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 10 monclient: _renew_subs 2023-10-30T12:18:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc454138ab0 con 0x7fc4540a2ae0 2023-10-30T12:18:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.086+0000 7fc45ba26640 1 librados: init done 2023-10-30T12:18:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc45ba26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc45ba26640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541cad00 0x7fc454137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc45ba26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc454073ff0 con 0x7fc4541cad00 2023-10-30T12:18:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc4440135d0 con 0x7fc4540a2ae0 2023-10-30T12:18:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541cad00 0x7fc454137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient: get_auth_request con 0x7fc4541cad00 auth_method 0 2023-10-30T12:18:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4380392f0 0x7fc43803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(562..562 src has 490..562) v4 ==== 6670+0+0 (secure 0 0 0) 0x7fc44404b0d0 con 0x7fc4540a2ae0 2023-10-30T12:18:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 10 client.224216.objecter ms_dispatch 0x7fc454000b90 osd_map(562..562 src has 490..562) v4 2023-10-30T12:18:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 3 client.224216.objecter handle_osd_map got epochs [562,562] > 0 2023-10-30T12:18:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 3 client.224216.objecter handle_osd_map decoding full epoch 562 2023-10-30T12:18:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 10 monclient(hunting): handle_auth_done global_id 224219 payload 824 2023-10-30T12:18:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 1 -- 172.21.15.50:0/1491600508 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc45409a5c0 con 0x7fc4541cad00 2023-10-30T12:18:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 20 client.224216.objecter dump_active .. 0 homeless 2023-10-30T12:18:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc4597c2640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541cad00 0x7fc454137890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc450000f60 tx=0x7fc45000d970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 10 client.224216.objecter ms_handle_connect 0x7fc4541cad00 2023-10-30T12:18:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.087+0000 7fc449ffb640 10 client.224216.objecter resend_mon_ops 2023-10-30T12:18:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450005cc0 con 0x7fc4541cad00 2023-10-30T12:18:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 10 monclient: dump: 2023-10-30T12:18:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.138 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc459fc3640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4380392f0 0x7fc43803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc459fc3640 10 monclient: get_auth_request con 0x7fc4380392f0 auth_method 0 2023-10-30T12:18:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc450005e20 con 0x7fc4541cad00 2023-10-30T12:18:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541cad00 msgr2=0x7fc454137890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541cad00 0x7fc454137890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc450000f60 tx=0x7fc45000d970 comp rx=0 tx=0).stop 2023-10-30T12:18:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541cad00 msgr2=0x7fc454137890 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541cad00 0x7fc454137890 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc450000f60 tx=0x7fc45000d970 comp rx=0 tx=0).stop 2023-10-30T12:18:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc449ffb640 10 client.224216.objecter ms_handle_connect 0x7fc4380392f0 2023-10-30T12:18:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.088+0000 7fc459fc3640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4380392f0 0x7fc43803b7b0 secure :-1 s=READY pgs=4272 cs=0 l=1 rev1=1 crypto rx=0x7fc44c036000 tx=0x7fc44c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc45ba26640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc45ba26640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541c0030 0x7fc454137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc45ba26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc454099b70 con 0x7fc4541c0030 2023-10-30T12:18:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541c0030 0x7fc454137dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient: get_auth_request con 0x7fc4541c0030 auth_method 0 2023-10-30T12:18:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 10 monclient(hunting): handle_auth_done global_id 224222 payload 824 2023-10-30T12:18:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.094+0000 7fc4597c2640 1 -- 172.21.15.50:0/1491600508 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc4540a2870 con 0x7fc4541c0030 2023-10-30T12:18:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 10 client.224216.objecter ms_handle_connect 0x7fc4541c0030 2023-10-30T12:18:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc4597c2640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541c0030 0x7fc454137dd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc450000f60 tx=0x7fc4500057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 10 client.224216.objecter resend_mon_ops 2023-10-30T12:18:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450023070 con 0x7fc4541c0030 2023-10-30T12:18:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 10 monclient: dump: 2023-10-30T12:18:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.145 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.147 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7fc45001e2e0 con 0x7fc4541c0030 2023-10-30T12:18:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541c0030 msgr2=0x7fc454137dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541c0030 0x7fc454137dd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc450000f60 tx=0x7fc4500057f0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541c0030 msgr2=0x7fc454137dd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:39.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc449ffb640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4541c0030 0x7fc454137dd0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc450000f60 tx=0x7fc4500057f0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.149 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:18:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 190, 2023-10-30T12:18:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:18:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:18:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:18:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:18:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:18:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:18:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:39.180 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 10 librados: watch_flush enter 2023-10-30T12:18:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 10 librados: watch_flush exit 2023-10-30T12:18:39.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 20 client.224216.objecter shutdown clearing up homeless session... 2023-10-30T12:18:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 10 client.224216.objecter successfully canceled tick 2023-10-30T12:18:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4380392f0 msgr2=0x7fc43803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc4380392f0 0x7fc43803b7b0 secure :-1 s=READY pgs=4272 cs=0 l=1 rev1=1 crypto rx=0x7fc44c036000 tx=0x7fc44c030000 comp rx=0 tx=0).stop 2023-10-30T12:18:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 10 monclient: shutdown 2023-10-30T12:18:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2ae0 msgr2=0x7fc454136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 1 --2- 172.21.15.50:0/1491600508 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc4540a2ae0 0x7fc454136e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc44400ec10 tx=0x7fc44400c520 comp rx=0 tx=0).stop 2023-10-30T12:18:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc459fc3640 1 -- 172.21.15.50:0/1491600508 reap_dead start 2023-10-30T12:18:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc459fc3640 1 -- 172.21.15.50:0/1491600508 reap_dead start 2023-10-30T12:18:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 shutdown_connections 2023-10-30T12:18:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.095+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 >> 172.21.15.50:0/1491600508 conn(0x7fc4540a80b0 msgr2=0x7fc4540a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.096+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 shutdown_connections 2023-10-30T12:18:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.096+0000 7fc45ba26640 1 -- 172.21.15.50:0/1491600508 wait complete. 2023-10-30T12:18:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.096+0000 7fc45ba26640 1 librados: shutdown 2023-10-30T12:18:39.185 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:18:39.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:18:39.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:18:39.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.291+0000 7fe9ea4d0640 10 monclient: get_monmap_and_config 2023-10-30T12:18:39.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.291+0000 7fe9ea4d0640 10 monclient: build_initial_monmap 2023-10-30T12:18:39.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.291+0000 7fe9ea4d0640 10 monclient: monmap: 2023-10-30T12:18:39.293 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:39.293 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:39.292412+0000 2023-10-30T12:18:39.294 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:39.292412+0000 2023-10-30T12:18:39.294 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:39.294 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.294 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:39.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:39.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:39.295 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:39.295 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:39.295 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 1 Processor -- start 2023-10-30T12:18:39.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 1 -- start start 2023-10-30T12:18:39.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient: init 2023-10-30T12:18:39.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:39.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-10-30T12:18:39.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4097fe0 0x7fe9e409a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e4097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9e4071c00 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe9e4137850 0x7fe9e4139c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e4137850 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e4097600 con 0x7fe9e4071c00 2023-10-30T12:18:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e409faa0 con 0x7fe9e4137850 2023-10-30T12:18:39.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e40978e0 con 0x7fe9e4097fe0 2023-10-30T12:18:39.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:39.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9ea4d0640 10 monclient(hunting): authenticate will time out at 1964.475659s 2023-10-30T12:18:39.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4097fe0 0x7fe9e409a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe9e4137850 0x7fe9e4139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe9e4137850 0x7fe9e4139c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55186/0 (socket says 172.21.15.50:55186) 2023-10-30T12:18:39.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4097fe0 0x7fe9e409a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51656/0 (socket says 172.21.15.50:51656) 2023-10-30T12:18:39.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/2348515488 learned_addr learned my addr 172.21.15.50:0/2348515488 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:39.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 10 monclient(hunting): get_auth_request con 0x7fe9e4137850 auth_method 0 2023-10-30T12:18:39.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 10 monclient(hunting): get_auth_request con 0x7fe9e4097fe0 auth_method 0 2023-10-30T12:18:39.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.292+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_done global_id 234174 payload 274 2023-10-30T12:18:39.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4097fe0 msgr2=0x7fe9e409a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:39.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 1 --2- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4097fe0 0x7fe9e409a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/2348515488 >> v1:172.21.15.118:6789/0 conn(0x7fe9e4071c00 legacy=0x7fe9e409f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 1 monclient: found mon.noname-a 2023-10-30T12:18:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:18:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/2348515488 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e409bf50 con 0x7fe9e4137850 2023-10-30T12:18:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e3fff640 1 --2- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4097fe0 0x7fe9e409a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.293+0000 7fe9e8a6d640 1 --2- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4137850 0x7fe9e4139c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00d150 tx=0x7fe9dc02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443f247009ea38d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 1 -- 172.21.15.50:0/2348515488 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc02e9e0 con 0x7fe9e4137850 2023-10-30T12:18:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:18:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: dump: 2023-10-30T12:18:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.314 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 1 -- 172.21.15.50:0/2348515488 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc02eb40 con 0x7fe9e4137850 2023-10-30T12:18:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 5 monclient: authenticate success, global_id 234174 2023-10-30T12:18:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 4 set_mon_vals no callback set 2023-10-30T12:18:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 1 -- 172.21.15.50:0/2348515488 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc009600 con 0x7fe9e4137850 2023-10-30T12:18:39.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:18:39.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9e2ffd640 10 monclient: dump: 2023-10-30T12:18:39.317 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.317 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.318 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.318 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.318 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.318 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.318 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.319 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 10 monclient: shutdown 2023-10-30T12:18:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4137850 msgr2=0x7fe9e4139c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4137850 0x7fe9e4139c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00d150 tx=0x7fe9dc02e040 comp rx=0 tx=0).stop 2023-10-30T12:18:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/2348515488 shutdown_connections 2023-10-30T12:18:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4137850 0x7fe9e4139c40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/2348515488 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4097fe0 0x7fe9e409a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/2348515488 >> 172.21.15.50:0/2348515488 conn(0x7fe9e40a3980 msgr2=0x7fe9e40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/2348515488 shutdown_connections 2023-10-30T12:18:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.294+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/2348515488 wait complete. 2023-10-30T12:18:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 10 monclient: build_initial_monmap 2023-10-30T12:18:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 10 monclient: monmap: 2023-10-30T12:18:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:39.321 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:39.323 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:39.323 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 1 librados: starting msgr at 2023-10-30T12:18:39.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 1 librados: starting objecter 2023-10-30T12:18:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 1 Processor -- start 2023-10-30T12:18:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 1 -- start start 2023-10-30T12:18:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 1 librados: setting wanted keys 2023-10-30T12:18:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 1 librados: calling monclient init 2023-10-30T12:18:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.295+0000 7fe9ea4d0640 10 monclient: init 2023-10-30T12:18:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4071c00 0x7fe9e408afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e4071c00 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4097fe0 0x7fe9e408b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e4097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4137850 0x7fe9e408ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e4137850 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e408d100 con 0x7fe9e4097fe0 2023-10-30T12:18:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e40a2fe0 con 0x7fe9e4071c00 2023-10-30T12:18:39.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e408cab0 con 0x7fe9e4137850 2023-10-30T12:18:39.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:39.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9ea4d0640 10 monclient(hunting): authenticate will time out at 1964.479254s 2023-10-30T12:18:39.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4097fe0 0x7fe9e408b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4137850 0x7fe9e408ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4097fe0 0x7fe9e408b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55190/0 (socket says 172.21.15.50:55190) 2023-10-30T12:18:39.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 1 -- 172.21.15.50:0/1505447112 learned_addr learned my addr 172.21.15.50:0/1505447112 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:39.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4137850 0x7fe9e408ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51660/0 (socket says 172.21.15.50:51660) 2023-10-30T12:18:39.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 10 monclient(hunting): get_auth_request con 0x7fe9e4097fe0 auth_method 0 2023-10-30T12:18:39.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): get_auth_request con 0x7fe9e4137850 auth_method 0 2023-10-30T12:18:39.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e3fff640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4071c00 0x7fe9e408afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e3fff640 10 monclient(hunting): get_auth_request con 0x7fe9e4071c00 auth_method 0 2023-10-30T12:18:39.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient(hunting): handle_auth_done global_id 224228 payload 824 2023-10-30T12:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4071c00 msgr2=0x7fe9e408afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4071c00 0x7fe9e408afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4097fe0 msgr2=0x7fe9e408b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4097fe0 0x7fe9e408b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 monclient: found mon.noname-c 2023-10-30T12:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/1505447112 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d40093a0 con 0x7fe9e4137850 2023-10-30T12:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e37fe640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e4097fe0 0x7fe9e408b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e8a6d640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4137850 0x7fe9e408ba50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc034000 tx=0x7fe9dc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9e4137850 2023-10-30T12:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e0ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.296+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc02e9e0 con 0x7fe9e4137850 2023-10-30T12:18:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: dump: 2023-10-30T12:18:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.336 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.337 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: _finish_auth 0 2023-10-30T12:18:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:18:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:39.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 5 monclient: authenticate success, global_id 224228 2023-10-30T12:18:39.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 10 monclient: _renew_subs 2023-10-30T12:18:39.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc00bd40 con 0x7fe9e4137850 2023-10-30T12:18:39.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc036820 con 0x7fe9e4137850 2023-10-30T12:18:39.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc036c10 con 0x7fe9e4137850 2023-10-30T12:18:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 10 monclient: dump: 2023-10-30T12:18:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 10 client.224228.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 10 monclient: _renew_subs 2023-10-30T12:18:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e408c760 con 0x7fe9e4137850 2023-10-30T12:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 1 librados: init done 2023-10-30T12:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e27fc640 4 set_mon_vals no callback set 2023-10-30T12:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe9dc04a460 con 0x7fe9e4137850 2023-10-30T12:18:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4136a30 0x7fe9e40954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4073ff0 con 0x7fe9e4136a30 2023-10-30T12:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.297+0000 7fe9e0ff9640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe9d00392f0 0x7fe9d003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4136a30 0x7fe9e40954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient: get_auth_request con 0x7fe9e4136a30 auth_method 0 2023-10-30T12:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(562..562 src has 490..562) v4 ==== 6670+0+0 (secure 0 0 0) 0x7fe9dc06e260 con 0x7fe9e4137850 2023-10-30T12:18:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 client.224228.objecter ms_dispatch 0x7fe9e4000b90 osd_map(562..562 src has 490..562) v4 2023-10-30T12:18:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 3 client.224228.objecter handle_osd_map got epochs [562,562] > 0 2023-10-30T12:18:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 3 client.224228.objecter handle_osd_map decoding full epoch 562 2023-10-30T12:18:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e37fe640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe9d00392f0 0x7fe9d003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e37fe640 10 monclient: get_auth_request con 0x7fe9d00392f0 auth_method 0 2023-10-30T12:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 20 client.224228.objecter dump_active .. 0 homeless 2023-10-30T12:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_done global_id 204487 payload 824 2023-10-30T12:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 1 -- 172.21.15.50:0/1505447112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9e408cc20 con 0x7fe9e4136a30 2023-10-30T12:18:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e3fff640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4136a30 0x7fe9e40954a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4011830 tx=0x7fe9d4011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 client.224228.objecter ms_handle_connect 0x7fe9e4136a30 2023-10-30T12:18:39.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 client.224228.objecter resend_mon_ops 2023-10-30T12:18:39.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4004230 con 0x7fe9e4136a30 2023-10-30T12:18:39.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:39.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 monclient: dump: 2023-10-30T12:18:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.349 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.349 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.349 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.349 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.349 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e37fe640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe9d00392f0 0x7fe9d003b7b0 secure :-1 s=READY pgs=4273 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40712f0 tx=0x7fe9d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 client.224228.objecter ms_handle_connect 0x7fe9d00392f0 2023-10-30T12:18:39.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fe9d4004390 con 0x7fe9e4136a30 2023-10-30T12:18:39.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:39.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:39.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4136a30 msgr2=0x7fe9e40954a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.298+0000 7fe9e0ff9640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4136a30 0x7fe9e40954a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4011830 tx=0x7fe9d4011cf0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.299+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4136a30 msgr2=0x7fe9e40954a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:39.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.299+0000 7fe9e0ff9640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e4136a30 0x7fe9e40954a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9ea4d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:39.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e40959e0 0x7fe9e4095df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9ea4d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4072020 con 0x7fe9e40959e0 2023-10-30T12:18:39.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9e3fff640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e40959e0 0x7fe9e4095df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9e3fff640 10 monclient: get_auth_request con 0x7fe9e40959e0 auth_method 0 2023-10-30T12:18:39.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9e3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.305+0000 7fe9e3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e3fff640 10 monclient(hunting): handle_auth_done global_id 204490 payload 824 2023-10-30T12:18:39.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e3fff640 1 -- 172.21.15.50:0/1505447112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe9e409d9b0 con 0x7fe9e40959e0 2023-10-30T12:18:39.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e0ff9640 10 client.224228.objecter ms_handle_connect 0x7fe9e40959e0 2023-10-30T12:18:39.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e3fff640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e40959e0 0x7fe9e4095df0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4011830 tx=0x7fe9d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e0ff9640 10 client.224228.objecter resend_mon_ops 2023-10-30T12:18:39.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4006800 con 0x7fe9e40959e0 2023-10-30T12:18:39.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:39.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.306+0000 7fe9e0ff9640 10 monclient: dump: 2023-10-30T12:18:39.357 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.357 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.357 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.357 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.357 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.357 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.358 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.358 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.358 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.358 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fe9d4003610 con 0x7fe9e40959e0 2023-10-30T12:18:39.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:39.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e0ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:39.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e40959e0 msgr2=0x7fe9e4095df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e0ff9640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e40959e0 0x7fe9e4095df0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4011830 tx=0x7fe9d40046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e0ff9640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e40959e0 msgr2=0x7fe9e4095df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:39.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e0ff9640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe9e40959e0 0x7fe9e4095df0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.360 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:18:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 190, 2023-10-30T12:18:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:18:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:18:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:18:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:39.389 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:39.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 10 librados: watch_flush enter 2023-10-30T12:18:39.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 10 librados: watch_flush exit 2023-10-30T12:18:39.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 20 client.224228.objecter shutdown clearing up homeless session... 2023-10-30T12:18:39.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 10 client.224228.objecter successfully canceled tick 2023-10-30T12:18:39.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe9d00392f0 msgr2=0x7fe9d003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe9d00392f0 0x7fe9d003b7b0 secure :-1 s=READY pgs=4273 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40712f0 tx=0x7fe9d8009040 comp rx=0 tx=0).stop 2023-10-30T12:18:39.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 10 monclient: shutdown 2023-10-30T12:18:39.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:39.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4137850 msgr2=0x7fe9e408ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9e8a6d640 1 -- 172.21.15.50:0/1505447112 reap_dead start 2023-10-30T12:18:39.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4137850 0x7fe9e408ba50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc034000 tx=0x7fe9dc004a40 comp rx=0 tx=0).stop 2023-10-30T12:18:39.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 shutdown_connections 2023-10-30T12:18:39.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 --2- 172.21.15.50:0/1505447112 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe9e4137850 0x7fe9e408ba50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 >> 172.21.15.50:0/1505447112 conn(0x7fe9e40a3980 msgr2=0x7fe9e409a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:39.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 shutdown_connections 2023-10-30T12:18:39.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 -- 172.21.15.50:0/1505447112 wait complete. 2023-10-30T12:18:39.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.307+0000 7fe9ea4d0640 1 librados: shutdown 2023-10-30T12:18:39.394 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:18:39.395 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:18:39.395 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:18:39.395 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:18:39.395 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:18:39.396 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:18:39.396 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:18:39.406 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:18:39.406 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:18:39.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:39.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.549+0000 7fca729a5640 10 monclient: get_monmap_and_config 2023-10-30T12:18:39.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.549+0000 7fca729a5640 10 monclient: build_initial_monmap 2023-10-30T12:18:39.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.549+0000 7fca729a5640 10 monclient: monmap: 2023-10-30T12:18:39.552 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:39.552 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.552 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:39.550802+0000 2023-10-30T12:18:39.552 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:39.550802+0000 2023-10-30T12:18:39.552 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:39.552 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.553 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:39.553 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:39.553 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:39.553 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:39.553 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:39.554 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 1 Processor -- start 2023-10-30T12:18:39.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 1 -- start start 2023-10-30T12:18:39.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient: init 2023-10-30T12:18:39.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:39.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T12:18:39.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca6c070bc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:39.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca6c070390 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:39.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca6c13c0b0 0x7fca6c13e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient(hunting): picked mon.noname-a con 0x7fca6c13c0b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:39.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6c070800 con 0x7fca6c070390 2023-10-30T12:18:39.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6c09d9e0 con 0x7fca6c070bc0 2023-10-30T12:18:39.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6c09d540 con 0x7fca6c13c0b0 2023-10-30T12:18:39.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:39.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca729a5640 10 monclient(hunting): authenticate will time out at 1964.733963s 2023-10-30T12:18:39.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca70f42640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fca6c13c0b0 msgr2=0x7fca6c13e4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:39.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.550+0000 7fca6b7fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fca6c070390 legacy=0x7fca6c13a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:18:39.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca70f42640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca6c13c0b0 0x7fca6c13e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6bfff640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fca6c070bc0 0x7fca6c070fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:51990/0 (socket says 172.21.15.50:51990) 2023-10-30T12:18:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6bfff640 1 -- 172.21.15.50:0/1654141578 learned_addr learned my addr 172.21.15.50:0/1654141578 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304850655 0 0) 0x7fca6c09d9e0 con 0x7fca6c070bc0 2023-10-30T12:18:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): _negotiate 2023-10-30T12:18:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): my global_id is 204493 2023-10-30T12:18:39.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54003520 con 0x7fca6c070bc0 2023-10-30T12:18:39.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730323106 0 0) 0x7fca54003520 con 0x7fca6c070bc0 2023-10-30T12:18:39.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 >> v2:172.21.15.50:3300/0 conn(0x7fca6c13c0b0 msgr2=0x7fca6c13e4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:39.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 --2- 172.21.15.50:0/1654141578 >> v2:172.21.15.50:3300/0 conn(0x7fca6c13c0b0 0x7fca6c13e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 >> v1:172.21.15.50:6789/0 conn(0x7fca6c070390 legacy=0x7fca6c13a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:18:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:18:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c099f60 con 0x7fca6c070bc0 2023-10-30T12:18:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: _finish_auth 0 2023-10-30T12:18:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:39.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:39.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fca60001670 con 0x7fca6c070bc0 2023-10-30T12:18:39.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:18:39.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: dump: 2023-10-30T12:18:39.572 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.572 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.572 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.572 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.572 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.573 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.573 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.573 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.574 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.574 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:18:39.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:39.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 >> v1:172.21.15.118:6789/0 conn(0x7fca6c070bc0 legacy=0x7fca6c070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:39.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca54003970 0x7fca54003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): picked mon.a con 0x7fca54003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:39.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): picked mon.c con 0x7fca540047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:39.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca54005380 0x7fca5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): picked mon.b con 0x7fca54005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:39.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c09d540 con 0x7fca54003970 2023-10-30T12:18:39.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400c220 con 0x7fca54005380 2023-10-30T12:18:39.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400c3e0 con 0x7fca540047c0 2023-10-30T12:18:39.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:39.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6affd640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:18:39.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6bfff640 1 -- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca54003970 msgr2=0x7fca54003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:39.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.551+0000 7fca6bfff640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca54003970 0x7fca54003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:39.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca54005380 0x7fca5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 10 monclient(hunting): get_auth_request con 0x7fca54005380 auth_method 0 2023-10-30T12:18:39.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:39.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): get_auth_request con 0x7fca540047c0 auth_method 0 2023-10-30T12:18:39.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:39.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:39.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:39.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient(hunting): handle_auth_done global_id 204493 payload 293 2023-10-30T12:18:39.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 1 -- 172.21.15.50:0/1654141578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca54005380 msgr2=0x7fca5400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:39.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca54005380 0x7fca5400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 1 -- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca54003970 msgr2=0x7fca54003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:39.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca54003970 0x7fca54003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:18:39.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 1 monclient: found mon.c 2023-10-30T12:18:39.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:39.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 1 -- 172.21.15.50:0/1654141578 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca54003520 con 0x7fca540047c0 2023-10-30T12:18:39.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:18:39.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:18:39.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:39.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca729a5640 5 monclient: authenticate success, global_id 204493 2023-10-30T12:18:39.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca54005380 0x7fca5400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:39.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca729a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:39.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.552+0000 7fca70f42640 1 -- 172.21.15.50:0/1654141578 reap_dead start 2023-10-30T12:18:39.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6b7fe640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca54004c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009900 tx=0x7fca540057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e1a73821ceb481 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c003bb0 con 0x7fca540047c0 2023-10-30T12:18:39.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:39.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 10 monclient: dump: 2023-10-30T12:18:39.586 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.586 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.586 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.587 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.587 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.587 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.587 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.587 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.588 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.588 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:39.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c003d10 con 0x7fca540047c0 2023-10-30T12:18:39.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:39.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:39.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 1 -- 172.21.15.50:0/1654141578 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0052b0 con 0x7fca540047c0 2023-10-30T12:18:39.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 4 set_mon_vals no callback set 2023-10-30T12:18:39.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:39.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca6affd640 10 monclient: dump: 2023-10-30T12:18:39.590 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.590 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.590 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.591 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.591 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.591 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.591 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.592 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.592 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.592 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 10 monclient: shutdown 2023-10-30T12:18:39.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:39.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 1 -- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 msgr2=0x7fca54004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca54004c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009900 tx=0x7fca540057f0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 1 -- 172.21.15.50:0/1654141578 shutdown_connections 2023-10-30T12:18:39.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 1 --2- 172.21.15.50:0/1654141578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca54004c40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 1 -- 172.21.15.50:0/1654141578 >> 172.21.15.50:0/1654141578 conn(0x7fca6c0a4670 msgr2=0x7fca6c0a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:39.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 1 -- 172.21.15.50:0/1654141578 shutdown_connections 2023-10-30T12:18:39.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.553+0000 7fca729a5640 1 -- 172.21.15.50:0/1654141578 wait complete. 2023-10-30T12:18:39.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 10 monclient: build_initial_monmap 2023-10-30T12:18:39.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 10 monclient: monmap: 2023-10-30T12:18:39.594 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:39.594 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:39.595 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:39.595 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:39.595 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:39.595 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.595 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:39.596 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:39.596 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:39.596 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 1 librados: starting msgr at 2023-10-30T12:18:39.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 1 librados: starting objecter 2023-10-30T12:18:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 1 Processor -- start 2023-10-30T12:18:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 1 -- start start 2023-10-30T12:18:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 1 librados: setting wanted keys 2023-10-30T12:18:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 1 librados: calling monclient init 2023-10-30T12:18:39.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.554+0000 7fca729a5640 10 monclient: init 2023-10-30T12:18:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca6c1ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): picked mon.noname-c con 0x7fca540047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:39.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca6c13b920 0x7fca6c1ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): picked mon.noname-a con 0x7fca6c13b920 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:39.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca6c133f70 0x7fca6c134420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): picked mon.noname-b con 0x7fca6c133f70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:39.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c1366a0 con 0x7fca6c13b920 2023-10-30T12:18:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c138070 con 0x7fca6c133f70 2023-10-30T12:18:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c13ac70 con 0x7fca540047c0 2023-10-30T12:18:39.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca729a5640 10 monclient(hunting): authenticate will time out at 1964.738331s 2023-10-30T12:18:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6b7fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca6c13b920 msgr2=0x7fca6c1ca9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca6c13b920 0x7fca6c1ca9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca6c1ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca6c1ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51680/0 (socket says 172.21.15.50:51680) 2023-10-30T12:18:39.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 -- 172.21.15.50:0/955281194 learned_addr learned my addr 172.21.15.50:0/955281194 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:39.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): get_auth_request con 0x7fca540047c0 auth_method 0 2023-10-30T12:18:39.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:39.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:39.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:39.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca70f42640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca6c133f70 0x7fca6c134420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca70f42640 10 monclient(hunting): get_auth_request con 0x7fca6c133f70 auth_method 0 2023-10-30T12:18:39.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca70f42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:39.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca70f42640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:39.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca70f42640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:39.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient(hunting): handle_auth_done global_id 224231 payload 824 2023-10-30T12:18:39.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 -- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca6c133f70 msgr2=0x7fca6c134420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca6c133f70 0x7fca6c134420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 -- 172.21.15.50:0/955281194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca6c13b920 msgr2=0x7fca6c1ca9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:39.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca6c13b920 0x7fca6c1ca9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:39.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 monclient: found mon.noname-c 2023-10-30T12:18:39.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:39.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 -- 172.21.15.50:0/955281194 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c09d6b0 con 0x7fca540047c0 2023-10-30T12:18:39.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca6bfff640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca6c1ca4a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c740 tx=0x7fca6000cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca68ff9640 10 client.?.objecter ms_handle_connect 0x7fca540047c0 2023-10-30T12:18:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca68ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.555+0000 7fca68ff9640 1 -- 172.21.15.50:0/955281194 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca600072d0 con 0x7fca540047c0 2023-10-30T12:18:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: dump: 2023-10-30T12:18:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.610 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.610 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: _finish_auth 0 2023-10-30T12:18:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:18:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 1 -- 172.21.15.50:0/955281194 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca60007450 con 0x7fca540047c0 2023-10-30T12:18:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 5 monclient: authenticate success, global_id 224231 2023-10-30T12:18:39.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 10 monclient: _renew_subs 2023-10-30T12:18:39.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6c0a36c0 con 0x7fca540047c0 2023-10-30T12:18:39.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:39.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 1 -- 172.21.15.50:0/955281194 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca600047e0 con 0x7fca540047c0 2023-10-30T12:18:39.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 10 monclient: dump: 2023-10-30T12:18:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:39.613 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:39.615 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:39.615 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:39.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 10 client.224231.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:39.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 10 monclient: _renew_subs 2023-10-30T12:18:39.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6c135c10 con 0x7fca540047c0 2023-10-30T12:18:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca729a5640 1 librados: init done 2023-10-30T12:18:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca6a7fc640 4 set_mon_vals no callback set 2023-10-30T12:18:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.556+0000 7fca68ff9640 1 -- 172.21.15.50:0/955281194 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fca60004960 con 0x7fca540047c0 2023-10-30T12:18:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca729a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca729a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca729a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca68ff9640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fca580392d0 0x7fca5803b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6c04e160 con 0x7fca540047c0 2023-10-30T12:18:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca68ff9640 1 -- 172.21.15.50:0/955281194 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(562..562 src has 490..562) v4 ==== 6670+0+0 (secure 0 0 0) 0x7fca6004a160 con 0x7fca540047c0 2023-10-30T12:18:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca68ff9640 10 client.224231.objecter ms_dispatch 0x7fca6c000b90 osd_map(562..562 src has 490..562) v4 2023-10-30T12:18:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca68ff9640 3 client.224231.objecter handle_osd_map got epochs [562,562] > 0 2023-10-30T12:18:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca68ff9640 3 client.224231.objecter handle_osd_map decoding full epoch 562 2023-10-30T12:18:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca68ff9640 20 client.224231.objecter dump_active .. 0 homeless 2023-10-30T12:18:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca6b7fe640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fca580392d0 0x7fca5803b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca6b7fe640 10 monclient: get_auth_request con 0x7fca580392d0 auth_method 0 2023-10-30T12:18:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca68ff9640 10 client.224231.objecter ms_handle_connect 0x7fca580392d0 2023-10-30T12:18:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.557+0000 7fca6b7fe640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fca580392d0 0x7fca5803b790 secure :-1 s=READY pgs=4274 cs=0 l=1 rev1=1 crypto rx=0x7fca5c033000 tx=0x7fca5c005000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.561+0000 7fca68ff9640 1 -- 172.21.15.50:0/955281194 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fca60018020 con 0x7fca540047c0 2023-10-30T12:18:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.561+0000 7fca68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.561+0000 7fca68ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:39.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.804+0000 7fca729a5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:39.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.804+0000 7fca729a5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:39.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.804+0000 7fca729a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:39.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.804+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fca6c069de0 con 0x7fca540047c0 2023-10-30T12:18:39.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.804+0000 7fca68ff9640 1 -- 172.21.15.50:0/955281194 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fca600119b0 con 0x7fca540047c0 2023-10-30T12:18:39.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.804+0000 7fca68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:39.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.804+0000 7fca68ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:39.807 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:39.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 10 librados: watch_flush enter 2023-10-30T12:18:39.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 10 librados: watch_flush exit 2023-10-30T12:18:39.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 20 client.224231.objecter shutdown clearing up homeless session... 2023-10-30T12:18:39.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 10 client.224231.objecter successfully canceled tick 2023-10-30T12:18:39.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fca580392d0 msgr2=0x7fca5803b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fca580392d0 0x7fca5803b790 secure :-1 s=READY pgs=4274 cs=0 l=1 rev1=1 crypto rx=0x7fca5c033000 tx=0x7fca5c005000 comp rx=0 tx=0).stop 2023-10-30T12:18:39.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 10 monclient: shutdown 2023-10-30T12:18:39.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:39.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 msgr2=0x7fca6c1ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:39.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca6c1ca4a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c740 tx=0x7fca6000cc00 comp rx=0 tx=0).stop 2023-10-30T12:18:39.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 shutdown_connections 2023-10-30T12:18:39.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fca580392d0 0x7fca5803b790 unknown :-1 s=CLOSED pgs=4274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca6c133f70 0x7fca6c134420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca6c13b920 0x7fca6c1ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 --2- 172.21.15.50:0/955281194 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca540047c0 0x7fca6c1ca4a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:39.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 >> 172.21.15.50:0/955281194 conn(0x7fca6c0a4670 msgr2=0x7fca6c09ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:39.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 shutdown_connections 2023-10-30T12:18:39.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 -- 172.21.15.50:0/955281194 wait complete. 2023-10-30T12:18:39.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:39.806+0000 7fca729a5640 1 librados: shutdown 2023-10-30T12:18:42.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:42.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.934+0000 7f080617c640 10 monclient: get_monmap_and_config 2023-10-30T12:18:42.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.934+0000 7f080617c640 10 monclient: build_initial_monmap 2023-10-30T12:18:42.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.934+0000 7f080617c640 10 monclient: monmap: 2023-10-30T12:18:42.936 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:42.936 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:42.936 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:42.935298+0000 2023-10-30T12:18:42.937 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:42.935298+0000 2023-10-30T12:18:42.937 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:42.937 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:42.937 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:42.937 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:42.938 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:42.938 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:42.938 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:42.938 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:42.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.934+0000 7f080617c640 1 Processor -- start 2023-10-30T12:18:42.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 1 -- start start 2023-10-30T12:18:42.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient: init 2023-10-30T12:18:42.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:42.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-10-30T12:18:42.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08000a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:42.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0800099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:42.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0800132db0 0x7f080013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient(hunting): picked mon.noname-a con 0x7f0800132db0 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:42.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f07ff7fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f08000a4b60 legacy=0x7f08001326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:18:42.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000a4f40 con 0x7f08000a4b60 2023-10-30T12:18:42.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000a4150 con 0x7f0800099a40 2023-10-30T12:18:42.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:42.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08000a3d70 con 0x7f0800132db0 2023-10-30T12:18:42.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:42.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f080617c640 10 monclient(hunting): authenticate will time out at 1968.118593s 2023-10-30T12:18:42.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f07fffff640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f0800132db0 msgr2=0x7f080013e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:42.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f07fffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0800132db0 0x7f080013e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:42.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f07feffd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f0800099a40 0x7f080013aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:51994/0 (socket says 172.21.15.50:51994) 2023-10-30T12:18:42.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.935+0000 7f07feffd640 1 -- 172.21.15.50:0/1639116132 learned_addr learned my addr 172.21.15.50:0/1639116132 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:42.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198329504 0 0) 0x7f08000a4150 con 0x7f0800099a40 2023-10-30T12:18:42.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:18:42.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:42.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:42.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): my global_id is 204499 2023-10-30T12:18:42.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e8003520 con 0x7f0800099a40 2023-10-30T12:18:42.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 310468255 0 0) 0x7f07e8003520 con 0x7f0800099a40 2023-10-30T12:18:42.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 >> v2:172.21.15.50:3300/0 conn(0x7f0800132db0 msgr2=0x7f080013e1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:42.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 --2- 172.21.15.50:0/1639116132 >> v2:172.21.15.50:3300/0 conn(0x7f0800132db0 0x7f080013e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:42.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 >> v1:172.21.15.50:6789/0 conn(0x7f08000a4b60 legacy=0x7f08001326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:42.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:18:42.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:18:42.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:42.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:18:42.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080009a440 con 0x7f0800099a40 2023-10-30T12:18:42.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:42.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:42.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:42.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f07f4001670 con 0x7f0800099a40 2023-10-30T12:18:42.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:42.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:18:42.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: dump: 2023-10-30T12:18:42.956 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:42.956 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:42.956 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.956 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:42.957 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:42.958 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:42.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:18:42.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:42.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 >> v1:172.21.15.118:6789/0 conn(0x7f0800099a40 legacy=0x7f080013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:42.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:42.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e8003970 0x7f07e8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): picked mon.a con 0x7f07e8003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07e80047c0 0x7f07e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): picked mon.c con 0x7f07e80047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e8005380 0x7f07e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): picked mon.b con 0x7f07e8005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e800c1f0 con 0x7f07e8003970 2023-10-30T12:18:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e800c360 con 0x7f07e8005380 2023-10-30T12:18:42.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e800c520 con 0x7f07e80047c0 2023-10-30T12:18:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07ff7fe640 1 -- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e8003970 msgr2=0x7f07e8003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fe7fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:18:42.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07ff7fe640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e8003970 0x7f07e8003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07e80047c0 0x7f07e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 10 monclient(hunting): get_auth_request con 0x7f07e80047c0 auth_method 0 2023-10-30T12:18:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:42.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e8005380 0x7f07e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 10 monclient(hunting): get_auth_request con 0x7f07e8005380 auth_method 0 2023-10-30T12:18:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:42.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:42.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07feffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:42.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:42.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:42.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.936+0000 7f07fffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:18:42.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 10 monclient(hunting): handle_auth_done global_id 204499 payload 293 2023-10-30T12:18:42.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 1 -- 172.21.15.50:0/1639116132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e8005380 msgr2=0x7f07e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:42.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f07e8005380 0x7f07e800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:42.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 1 -- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e8003970 msgr2=0x7f07e8003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:42.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e8003970 0x7f07e8003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:42.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:42.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 1 monclient: found mon.c 2023-10-30T12:18:42.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:42.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:42.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 1 -- 172.21.15.50:0/1639116132 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8003520 con 0x7f07e80047c0 2023-10-30T12:18:42.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:42.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:42.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:42.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f080617c640 5 monclient: authenticate success, global_id 204499 2023-10-30T12:18:42.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f080617c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:42.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07fffff640 1 -- 172.21.15.50:0/1639116132 reap_dead start 2023-10-30T12:18:42.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07feffd640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07e80047c0 0x7f07e8004c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0800070610 tx=0x7f07e80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6b3ecb98228b66e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:42.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.937+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4005ad0 con 0x7f07e80047c0 2023-10-30T12:18:42.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:42.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:42.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 10 monclient: dump: 2023-10-30T12:18:42.968 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:42.969 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:42.969 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.969 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.969 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:42.969 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:42.969 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:42.970 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:42.970 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:42.970 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:42.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:42.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f4005c30 con 0x7f07e80047c0 2023-10-30T12:18:42.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:42.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:42.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 4 set_mon_vals no callback set 2023-10-30T12:18:42.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 1 -- 172.21.15.50:0/1639116132 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f402c810 con 0x7f07e80047c0 2023-10-30T12:18:42.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:42.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:42.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f07fe7fc640 10 monclient: dump: 2023-10-30T12:18:42.972 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:42.972 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:42.972 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.972 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:42.973 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:42.973 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 10 monclient: shutdown 2023-10-30T12:18:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 1 -- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07e80047c0 msgr2=0x7f07e8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07e80047c0 0x7f07e8004c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0800070610 tx=0x7f07e80057f0 comp rx=0 tx=0).stop 2023-10-30T12:18:42.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 1 -- 172.21.15.50:0/1639116132 shutdown_connections 2023-10-30T12:18:42.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 1 --2- 172.21.15.50:0/1639116132 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07e80047c0 0x7f07e8004c40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:42.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 1 -- 172.21.15.50:0/1639116132 >> 172.21.15.50:0/1639116132 conn(0x7f08000a8060 msgr2=0x7f08000a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:42.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 1 -- 172.21.15.50:0/1639116132 shutdown_connections 2023-10-30T12:18:42.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.938+0000 7f080617c640 1 -- 172.21.15.50:0/1639116132 wait complete. 2023-10-30T12:18:42.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 10 monclient: build_initial_monmap 2023-10-30T12:18:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 10 monclient: monmap: 2023-10-30T12:18:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:42.976 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:42.977 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:42.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 1 librados: starting msgr at 2023-10-30T12:18:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 1 librados: starting objecter 2023-10-30T12:18:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 1 Processor -- start 2023-10-30T12:18:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 1 -- start start 2023-10-30T12:18:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 1 librados: setting wanted keys 2023-10-30T12:18:42.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 1 librados: calling monclient init 2023-10-30T12:18:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.939+0000 7f080617c640 10 monclient: init 2023-10-30T12:18:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e80047c0 0x7f08001c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): picked mon.noname-a con 0x7f07e80047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f080011e810 0x7f08001315d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): picked mon.noname-c con 0x7f080011e810 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0800131b10 0x7f08001c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): picked mon.noname-b con 0x7f0800131b10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:42.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0800131fb0 con 0x7f07e80047c0 2023-10-30T12:18:42.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:42.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080011f7e0 con 0x7f0800131b10 2023-10-30T12:18:42.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:42.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000a05e0 con 0x7f080011e810 2023-10-30T12:18:42.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:42.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f080617c640 10 monclient(hunting): authenticate will time out at 1968.123169s 2023-10-30T12:18:42.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07ff7fe640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e80047c0 msgr2=0x7f08001c9fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:42.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07ff7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e80047c0 0x7f08001c9fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:42.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f080011e810 0x7f08001315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:42.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f080011e810 0x7f08001315d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51694/0 (socket says 172.21.15.50:51694) 2023-10-30T12:18:42.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 -- 172.21.15.50:0/3399296735 learned_addr learned my addr 172.21.15.50:0/3399296735 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:42.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): get_auth_request con 0x7f080011e810 auth_method 0 2023-10-30T12:18:42.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:42.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:42.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:42.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:42.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:42.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:42.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07fffff640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0800131b10 0x7f08001c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:42.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07fffff640 10 monclient(hunting): get_auth_request con 0x7f0800131b10 auth_method 0 2023-10-30T12:18:42.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:42.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07fffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:42.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07fffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:42.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient(hunting): handle_auth_done global_id 224234 payload 824 2023-10-30T12:18:42.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 -- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0800131b10 msgr2=0x7f08001c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:42.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0800131b10 0x7f08001c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:42.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 -- 172.21.15.50:0/3399296735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e80047c0 msgr2=0x7f08001c9fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:42.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e80047c0 0x7f08001c9fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:42.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:42.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 monclient: found mon.noname-c 2023-10-30T12:18:42.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:42.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:42.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 -- 172.21.15.50:0/3399296735 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000a0e00 con 0x7f080011e810 2023-10-30T12:18:42.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07feffd640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f080011e810 0x7f08001315d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f07f4000c00 tx=0x7f07f4001a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:42.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07e7fff640 10 client.?.objecter ms_handle_connect 0x7f080011e810 2023-10-30T12:18:42.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07e7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:42.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07e7fff640 1 -- 172.21.15.50:0/3399296735 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f403c040 con 0x7f080011e810 2023-10-30T12:18:42.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:42.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:42.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.940+0000 7f07e7fff640 10 monclient: dump: 2023-10-30T12:18:42.988 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:42.988 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:42.988 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.988 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.989 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:42.989 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:42.989 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:42.989 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:42.989 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:42.990 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:42.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 10 monclient: _finish_auth 0 2023-10-30T12:18:42.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:42.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:42.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 1 -- 172.21.15.50:0/3399296735 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f402cc20 con 0x7f080011e810 2023-10-30T12:18:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 1 -- 172.21.15.50:0/3399296735 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4007070 con 0x7f080011e810 2023-10-30T12:18:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 10 monclient: dump: 2023-10-30T12:18:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:42.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:42.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:42.992 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:42.992 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:42.993 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:42.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:42.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:42.993 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:42.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 5 monclient: authenticate success, global_id 224234 2023-10-30T12:18:42.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 10 monclient: _renew_subs 2023-10-30T12:18:42.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:42.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07fdffb640 4 set_mon_vals no callback set 2023-10-30T12:18:42.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0800120000 con 0x7f080011e810 2023-10-30T12:18:42.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 10 client.224234.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:42.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 10 monclient: _renew_subs 2023-10-30T12:18:42.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:42.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08001caa20 con 0x7f080011e810 2023-10-30T12:18:42.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 1 librados: init done 2023-10-30T12:18:42.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f07e7fff640 1 -- 172.21.15.50:0/3399296735 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f07f4032070 con 0x7f080011e810 2023-10-30T12:18:42.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.941+0000 7f080617c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:42.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f080617c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:42.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f080617c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:42.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080004e160 con 0x7f080011e810 2023-10-30T12:18:42.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07e7fff640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f07ec0392a0 0x7f07ec03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:42.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07e7fff640 1 -- 172.21.15.50:0/3399296735 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(562..562 src has 490..562) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f07f406bfc0 con 0x7f080011e810 2023-10-30T12:18:42.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07e7fff640 10 client.224234.objecter ms_dispatch 0x7f0800000b90 osd_map(562..562 src has 490..562) v4 2023-10-30T12:18:42.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07e7fff640 3 client.224234.objecter handle_osd_map got epochs [562,562] > 0 2023-10-30T12:18:42.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07e7fff640 3 client.224234.objecter handle_osd_map decoding full epoch 562 2023-10-30T12:18:42.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07e7fff640 20 client.224234.objecter dump_active .. 0 homeless 2023-10-30T12:18:42.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07ff7fe640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f07ec0392a0 0x7f07ec03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:42.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07ff7fe640 10 monclient: get_auth_request con 0x7f07ec0392a0 auth_method 0 2023-10-30T12:18:42.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07ff7fe640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f07ec0392a0 0x7f07ec03b760 secure :-1 s=READY pgs=4275 cs=0 l=1 rev1=1 crypto rx=0x7f07f0006fd0 tx=0x7f07f0006de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:42.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.942+0000 7f07e7fff640 10 client.224234.objecter ms_handle_connect 0x7f07ec0392a0 2023-10-30T12:18:42.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.946+0000 7f07e7fff640 1 -- 172.21.15.50:0/3399296735 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f07f409eca0 con 0x7f080011e810 2023-10-30T12:18:42.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.946+0000 7f07e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:42.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:42.946+0000 7f07e7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:43.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.190+0000 7f080617c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:43.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.190+0000 7f080617c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:43.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.190+0000 7f080617c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:43.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.190+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f080011fa40 con 0x7f080011e810 2023-10-30T12:18:43.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.190+0000 7f07e7fff640 1 -- 172.21.15.50:0/3399296735 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f07f4046070 con 0x7f080011e810 2023-10-30T12:18:43.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.190+0000 7f07e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:43.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.190+0000 7f07e7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:43.193 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 10 librados: watch_flush enter 2023-10-30T12:18:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 10 librados: watch_flush exit 2023-10-30T12:18:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 20 client.224234.objecter shutdown clearing up homeless session... 2023-10-30T12:18:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 10 client.224234.objecter successfully canceled tick 2023-10-30T12:18:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f07ec0392a0 msgr2=0x7f07ec03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f07ec0392a0 0x7f07ec03b760 secure :-1 s=READY pgs=4275 cs=0 l=1 rev1=1 crypto rx=0x7f07f0006fd0 tx=0x7f07f0006de0 comp rx=0 tx=0).stop 2023-10-30T12:18:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 10 monclient: shutdown 2023-10-30T12:18:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f080011e810 msgr2=0x7f08001315d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f080011e810 0x7f08001315d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f07f4000c00 tx=0x7f07f4001a30 comp rx=0 tx=0).stop 2023-10-30T12:18:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 shutdown_connections 2023-10-30T12:18:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f07ec0392a0 0x7f07ec03b760 unknown :-1 s=CLOSED pgs=4275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0800131b10 0x7f08001c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f080011e810 0x7f08001315d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 --2- 172.21.15.50:0/3399296735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07e80047c0 0x7f08001c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 >> 172.21.15.50:0/3399296735 conn(0x7f08000a8060 msgr2=0x7f08000a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 shutdown_connections 2023-10-30T12:18:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 -- 172.21.15.50:0/3399296735 wait complete. 2023-10-30T12:18:43.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:43.192+0000 7f080617c640 1 librados: shutdown 2023-10-30T12:18:46.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:46.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.320+0000 7f51f4773640 10 monclient: get_monmap_and_config 2023-10-30T12:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.320+0000 7f51f4773640 10 monclient: build_initial_monmap 2023-10-30T12:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.320+0000 7f51f4773640 10 monclient: monmap: 2023-10-30T12:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:46.321251+0000 2023-10-30T12:18:46.323 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:46.321251+0000 2023-10-30T12:18:46.323 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:46.323 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:46.323 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:46.323 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:46.324 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:46.324 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:46.324 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:46.324 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:46.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.320+0000 7f51f4773640 1 Processor -- start 2023-10-30T12:18:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.320+0000 7f51f4773640 1 -- start start 2023-10-30T12:18:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.320+0000 7f51f4773640 10 monclient: init 2023-10-30T12:18:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:18:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f51ec0659d0 0x7f51ec12fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): picked mon.noname-a con 0x7f51ec0659d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): picked mon.noname-b con 0x7f51ec071410 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f51ec071850 0x7f51ec09ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): picked mon.noname-c con 0x7f51ec071850 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec09e090 con 0x7f51ec0659d0 2023-10-30T12:18:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec09fb30 con 0x7f51ec071850 2023-10-30T12:18:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec065db0 con 0x7f51ec071410 2023-10-30T12:18:46.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:46.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f4773640 10 monclient(hunting): authenticate will time out at 1971.504596s 2023-10-30T12:18:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f250f640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f51ec0659d0 msgr2=0x7f51ec12fe60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f250f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f51ec0659d0 0x7f51ec12fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec09e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51710/0 (socket says 172.21.15.50:51710) 2023-10-30T12:18:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 1 -- 172.21.15.50:0/3130488390 learned_addr learned my addr 172.21.15.50:0/3130488390 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 10 monclient(hunting): get_auth_request con 0x7f51ec071410 auth_method 0 2023-10-30T12:18:46.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:46.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:46.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:46.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:46.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:46.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:46.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f2d10640 1 --2- 172.21.15.50:0/3130488390 >> v2:172.21.15.118:3300/0 conn(0x7f51ec071850 0x7f51ec09ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:46.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f2d10640 10 monclient(hunting): get_auth_request con 0x7f51ec071850 auth_method 0 2023-10-30T12:18:46.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f2d10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:46.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f2d10640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.321+0000 7f51f2d10640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_done global_id 224237 payload 274 2023-10-30T12:18:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 1 -- 172.21.15.50:0/3130488390 >> v2:172.21.15.118:3300/0 conn(0x7f51ec071850 msgr2=0x7f51ec09ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 1 --2- 172.21.15.50:0/3130488390 >> v2:172.21.15.118:3300/0 conn(0x7f51ec071850 0x7f51ec09ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 1 -- 172.21.15.50:0/3130488390 >> v2:172.21.15.50:3300/0 conn(0x7f51ec0659d0 msgr2=0x7f51ec12fe60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 1 --2- 172.21.15.50:0/3130488390 >> v2:172.21.15.50:3300/0 conn(0x7f51ec0659d0 0x7f51ec12fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 10 monclient: _finish_hunting 0 2023-10-30T12:18:46.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 1 monclient: found mon.noname-b 2023-10-30T12:18:46.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:46.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:18:46.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 1 -- 172.21.15.50:0/3130488390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec09fd80 con 0x7f51ec071410 2023-10-30T12:18:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f1d0e640 1 --2- 172.21.15.50:0/3130488390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec09e7e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51ec06a510 tx=0x7f51dc02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1bd0f4c7f7e4b4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 1 -- 172.21.15.50:0/3130488390 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc031760 con 0x7f51ec071410 2023-10-30T12:18:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: dump: 2023-10-30T12:18:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:46.343 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: _finish_auth 0 2023-10-30T12:18:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: _check_auth_tickets 2023-10-30T12:18:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f4773640 5 monclient: authenticate success, global_id 224237 2023-10-30T12:18:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 1 -- 172.21.15.50:0/3130488390 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc0318c0 con 0x7f51ec071410 2023-10-30T12:18:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f4773640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:46.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f4773640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:46.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f4773640 4 set_mon_vals no callback set 2023-10-30T12:18:46.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 1 -- 172.21.15.50:0/3130488390 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc031c50 con 0x7f51ec071410 2023-10-30T12:18:46.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:46.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:46.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.322+0000 7f51f150d640 10 monclient: dump: 2023-10-30T12:18:46.346 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:46.346 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:46.346 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.346 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.347 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:46.347 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:46.347 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:46.347 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:46.347 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:46.348 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:46.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 10 monclient: shutdown 2023-10-30T12:18:46.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:46.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 -- 172.21.15.50:0/3130488390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 msgr2=0x7f51ec09e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:46.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 --2- 172.21.15.50:0/3130488390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec09e7e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51ec06a510 tx=0x7f51dc02b800 comp rx=0 tx=0).stop 2023-10-30T12:18:46.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 -- 172.21.15.50:0/3130488390 shutdown_connections 2023-10-30T12:18:46.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 --2- 172.21.15.50:0/3130488390 >> v2:172.21.15.118:3300/0 conn(0x7f51ec071850 0x7f51ec09ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 --2- 172.21.15.50:0/3130488390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec09e7e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 --2- 172.21.15.50:0/3130488390 >> v2:172.21.15.50:3300/0 conn(0x7f51ec0659d0 0x7f51ec12fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 -- 172.21.15.50:0/3130488390 >> 172.21.15.50:0/3130488390 conn(0x7f51ec0a47f0 msgr2=0x7f51ec0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:46.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 -- 172.21.15.50:0/3130488390 shutdown_connections 2023-10-30T12:18:46.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.323+0000 7f51f4773640 1 -- 172.21.15.50:0/3130488390 wait complete. 2023-10-30T12:18:46.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 10 monclient: build_initial_monmap 2023-10-30T12:18:46.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 10 monclient: monmap: 2023-10-30T12:18:46.350 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:46.350 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:46.351 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:46.351 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:46.351 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:46.351 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:46.351 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:46.351 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:46.352 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:46.352 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:46.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 1 librados: starting msgr at 2023-10-30T12:18:46.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 1 librados: starting objecter 2023-10-30T12:18:46.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 1 Processor -- start 2023-10-30T12:18:46.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 1 -- start start 2023-10-30T12:18:46.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 1 librados: setting wanted keys 2023-10-30T12:18:46.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 1 librados: calling monclient init 2023-10-30T12:18:46.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 10 monclient: init 2023-10-30T12:18:46.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:46.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:18:46.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ec0659d0 0x7f51ec11a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:46.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.324+0000 7f51f4773640 10 monclient(hunting): picked mon.noname-a con 0x7f51ec0659d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:46.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec11a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:46.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient(hunting): picked mon.noname-c con 0x7f51ec071410 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:46.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec071850 0x7f51ec11aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:46.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient(hunting): picked mon.noname-b con 0x7f51ec071850 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:46.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:46.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec11c2b0 con 0x7f51ec0659d0 2023-10-30T12:18:46.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:46.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0a3570 con 0x7f51ec071850 2023-10-30T12:18:46.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:46.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec11d3e0 con 0x7f51ec071410 2023-10-30T12:18:46.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:46.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient(hunting): authenticate will time out at 1971.508046s 2023-10-30T12:18:46.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f250f640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ec0659d0 msgr2=0x7f51ec11a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:46.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f250f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ec0659d0 0x7f51ec11a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:46.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec11a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:46.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec11a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51716/0 (socket says 172.21.15.50:51716) 2023-10-30T12:18:46.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 -- 172.21.15.50:0/2715543716 learned_addr learned my addr 172.21.15.50:0/2715543716 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:46.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): get_auth_request con 0x7f51ec071410 auth_method 0 2023-10-30T12:18:46.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:46.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:46.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:46.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:46.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:46.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:46.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient(hunting): handle_auth_done global_id 224240 payload 824 2023-10-30T12:18:46.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f2d10640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec071850 0x7f51ec11aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:46.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 -- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec071850 msgr2=0x7f51ec11aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:46.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec071850 0x7f51ec11aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 -- 172.21.15.50:0/2715543716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ec0659d0 msgr2=0x7f51ec11a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ec0659d0 0x7f51ec11a450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient: _finish_hunting 0 2023-10-30T12:18:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 monclient: found mon.noname-c 2023-10-30T12:18:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:46.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:46.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 -- 172.21.15.50:0/2715543716 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc0094d0 con 0x7f51ec071410 2023-10-30T12:18:46.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f1d0e640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec11a990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f51dc000c00 tx=0x7f51dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:46.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 client.?.objecter ms_handle_connect 0x7f51ec071410 2023-10-30T12:18:46.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:46.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 1 -- 172.21.15.50:0/2715543716 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc03b800 con 0x7f51ec071410 2023-10-30T12:18:46.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:46.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:46.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: dump: 2023-10-30T12:18:46.362 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:46.362 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:46.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.363 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.363 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:46.363 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:46.363 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:46.364 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:46.364 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:46.364 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:46.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:18:46.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:46.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 1 -- 172.21.15.50:0/2715543716 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc03b980 con 0x7f51ec071410 2023-10-30T12:18:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 5 monclient: authenticate success, global_id 224240 2023-10-30T12:18:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient: _renew_subs 2023-10-30T12:18:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec11d6c0 con 0x7f51ec071410 2023-10-30T12:18:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 1 -- 172.21.15.50:0/2715543716 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc03bd30 con 0x7f51ec071410 2023-10-30T12:18:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51e2ffd640 10 monclient: dump: 2023-10-30T12:18:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:46.368 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:46.368 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:46.368 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:46.368 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:46.368 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:46.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 client.224240.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient: _renew_subs 2023-10-30T12:18:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec11b840 con 0x7f51ec071410 2023-10-30T12:18:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f4773640 1 librados: init done 2023-10-30T12:18:46.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.325+0000 7f51f0d0c640 4 set_mon_vals no callback set 2023-10-30T12:18:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51e2ffd640 1 -- 172.21.15.50:0/2715543716 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f51dc0453f0 con 0x7f51ec071410 2023-10-30T12:18:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51f4773640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51f4773640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51f4773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:46.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec04e160 con 0x7f51ec071410 2023-10-30T12:18:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51e2ffd640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51d00392a0 0x7f51d003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51e2ffd640 1 -- 172.21.15.50:0/2715543716 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(562..562 src has 490..562) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f51dc02f050 con 0x7f51ec071410 2023-10-30T12:18:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51e2ffd640 10 client.224240.objecter ms_dispatch 0x7f51ec000b90 osd_map(562..562 src has 490..562) v4 2023-10-30T12:18:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51e2ffd640 3 client.224240.objecter handle_osd_map got epochs [562,562] > 0 2023-10-30T12:18:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.326+0000 7f51e2ffd640 3 client.224240.objecter handle_osd_map decoding full epoch 562 2023-10-30T12:18:46.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.327+0000 7f51e2ffd640 20 client.224240.objecter dump_active .. 0 homeless 2023-10-30T12:18:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.327+0000 7f51f250f640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51d00392a0 0x7f51d003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.327+0000 7f51f250f640 10 monclient: get_auth_request con 0x7f51d00392a0 auth_method 0 2023-10-30T12:18:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.327+0000 7f51f250f640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51d00392a0 0x7f51d003b760 secure :-1 s=READY pgs=4276 cs=0 l=1 rev1=1 crypto rx=0x7f51e8006fd0 tx=0x7f51e8006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.327+0000 7f51e2ffd640 10 client.224240.objecter ms_handle_connect 0x7f51d00392a0 2023-10-30T12:18:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.331+0000 7f51e2ffd640 1 -- 172.21.15.50:0/2715543716 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51dc04c6d0 con 0x7f51ec071410 2023-10-30T12:18:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.331+0000 7f51e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:46.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.331+0000 7f51e2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:46.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.576+0000 7f51f4773640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:46.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.576+0000 7f51f4773640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:46.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.576+0000 7f51f4773640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:46.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.576+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f51ec06a1b0 con 0x7f51ec071410 2023-10-30T12:18:46.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.576+0000 7f51e2ffd640 1 -- 172.21.15.50:0/2715543716 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f51dc04c310 con 0x7f51ec071410 2023-10-30T12:18:46.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.576+0000 7f51e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:46.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.576+0000 7f51e2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:46.579 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:46.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 10 librados: watch_flush enter 2023-10-30T12:18:46.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 10 librados: watch_flush exit 2023-10-30T12:18:46.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 20 client.224240.objecter shutdown clearing up homeless session... 2023-10-30T12:18:46.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 10 client.224240.objecter successfully canceled tick 2023-10-30T12:18:46.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51d00392a0 msgr2=0x7f51d003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:46.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51d00392a0 0x7f51d003b760 secure :-1 s=READY pgs=4276 cs=0 l=1 rev1=1 crypto rx=0x7f51e8006fd0 tx=0x7f51e8006d80 comp rx=0 tx=0).stop 2023-10-30T12:18:46.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 10 monclient: shutdown 2023-10-30T12:18:46.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:46.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 msgr2=0x7f51ec11a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:46.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec11a990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f51dc000c00 tx=0x7f51dc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 shutdown_connections 2023-10-30T12:18:46.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f51d00392a0 0x7f51d003b760 unknown :-1 s=CLOSED pgs=4276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f51ec071850 0x7f51ec11aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f51ec071410 0x7f51ec11a990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 --2- 172.21.15.50:0/2715543716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51ec0659d0 0x7f51ec11a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:46.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 >> 172.21.15.50:0/2715543716 conn(0x7f51ec0a47f0 msgr2=0x7f51ec12dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:46.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 shutdown_connections 2023-10-30T12:18:46.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 -- 172.21.15.50:0/2715543716 wait complete. 2023-10-30T12:18:46.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:46.578+0000 7f51f4773640 1 librados: shutdown 2023-10-30T12:18:49.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:18:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient: get_monmap_and_config 2023-10-30T12:18:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient: build_initial_monmap 2023-10-30T12:18:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient: monmap: 2023-10-30T12:18:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:49.708679+0000 2023-10-30T12:18:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:49.708679+0000 2023-10-30T12:18:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:49.712 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:49.712 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 1 Processor -- start 2023-10-30T12:18:49.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 1 -- start start 2023-10-30T12:18:49.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient: init 2023-10-30T12:18:49.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:49.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T12:18:49.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 0x7fa74409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:49.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient(hunting): picked mon.noname-a con 0x7fa744098520 addr v2:172.21.15.50:3300/0 2023-10-30T12:18:49.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa744098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:49.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7441379f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:49.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74409ad30 con 0x7fa7441379f0 2023-10-30T12:18:49.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74409c460 con 0x7fa744098900 2023-10-30T12:18:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74409b260 con 0x7fa744098520 2023-10-30T12:18:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa749b73640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.708+0000 7fa7437fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 msgr2=0x7fa74409bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.709+0000 7fa743fff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fa7441379f0 legacy=0x7fa744139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:18:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.709+0000 7fa749b73640 10 monclient(hunting): authenticate will time out at 1974.892014s 2023-10-30T12:18:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.709+0000 7fa7437fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 0x7fa74409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.709+0000 7fa742ffd640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fa744098900 0x7fa744098d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:37834/0 (socket says 172.21.15.50:37834) 2023-10-30T12:18:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.709+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 learned_addr learned my addr 172.21.15.50:0/1689518069 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:18:49.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.910+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 msgr2=0x7fa74409bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:49.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.910+0000 7fa743fff640 1 -- 172.21.15.50:0/1689518069 >> v1:172.21.15.50:6789/0 conn(0x7fa7441379f0 legacy=0x7fa744139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:18:49.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:49.910+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 0x7fa74409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:18:50.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:50.311+0000 7fa743fff640 1 -- 172.21.15.50:0/1689518069 >> v1:172.21.15.50:6789/0 conn(0x7fa7441379f0 legacy=0x7fa744139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:18:50.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:50.311+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 msgr2=0x7fa74409bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:50.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:50.311+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 0x7fa74409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:18:51.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:51.112+0000 7fa743fff640 1 -- 172.21.15.50:0/1689518069 >> v1:172.21.15.50:6789/0 conn(0x7fa7441379f0 legacy=0x7fa744139de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:18:51.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:51.112+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 msgr2=0x7fa74409bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:51.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:51.112+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 0x7fa74409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:18:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): tick 2023-10-30T12:18:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:18:52.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:18:52.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:18:52.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 msgr2=0x7fa74409bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:52.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.50:3300/0 conn(0x7fa744098520 0x7fa74409bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:52.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 -- 172.21.15.50:0/1689518069 >> v1:172.21.15.118:6789/0 conn(0x7fa744098900 legacy=0x7fa744098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:52.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 -- 172.21.15.50:0/1689518069 >> v1:172.21.15.50:6789/0 conn(0x7fa7441379f0 legacy=0x7fa744139de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:52.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient: _add_conns ranks=[2,4] 2023-10-30T12:18:52.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:52.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): picked mon.noname-b con 0x7fa7300036c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:52.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:52.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): picked mon.noname-c con 0x7fa730006240 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:52.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:52.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 -- 172.21.15.50:0/1689518069 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74409b260 con 0x7fa730006240 2023-10-30T12:18:52.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:52.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 1 -- 172.21.15.50:0/1689518069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa730008da0 con 0x7fa7300036c0 2023-10-30T12:18:52.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7417fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:52.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:52.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 10 monclient(hunting): get_auth_request con 0x7fa7300036c0 auth_method 0 2023-10-30T12:18:52.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:52.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:52.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:52.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 msgr2=0x7fa730005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 msgr2=0x7fa730005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 10 monclient(hunting): get_auth_request con 0x7fa730006240 auth_method 0 2023-10-30T12:18:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 msgr2=0x7fa7300086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:18:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 msgr2=0x7fa7300086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.709+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:52.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:52.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 10 monclient(hunting): get_auth_request con 0x7fa7300036c0 auth_method 0 2023-10-30T12:18:52.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:52.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:52.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:52.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 msgr2=0x7fa730005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:18:52.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 msgr2=0x7fa730005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:52.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:52.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:52.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:18:52.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 10 monclient(hunting): get_auth_request con 0x7fa730006240 auth_method 0 2023-10-30T12:18:52.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:52.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:52.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:52.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 msgr2=0x7fa7300086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:52.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 msgr2=0x7fa7300086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:52.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:52.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:52.910+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:18:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 10 monclient(hunting): get_auth_request con 0x7fa7300036c0 auth_method 0 2023-10-30T12:18:53.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 msgr2=0x7fa730005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:18:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 msgr2=0x7fa730005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:18:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 10 monclient(hunting): get_auth_request con 0x7fa730006240 auth_method 0 2023-10-30T12:18:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 msgr2=0x7fa7300086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:18:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 msgr2=0x7fa7300086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:18:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:18:53.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:53.311+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:18:54.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> v2:172.21.15.118:3300/0 conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): get_auth_request con 0x7fa730006240 auth_method 0 2023-10-30T12:18:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 10 monclient(hunting): get_auth_request con 0x7fa7300036c0 auth_method 0 2023-10-30T12:18:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:18:54.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa7437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient(hunting): handle_auth_done global_id 204508 payload 274 2023-10-30T12:18:54.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 msgr2=0x7fa730005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:54.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7300036c0 0x7fa730005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:18:54.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 1 monclient: found mon.noname-c 2023-10-30T12:18:54.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:54.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:18:54.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.112+0000 7fa742ffd640 1 -- 172.21.15.50:0/1689518069 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa744072120 con 0x7fa730006240 2023-10-30T12:18:54.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa742ffd640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa730006240 0x7fa7300086e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa74409c5f0 tx=0x7fa73802dce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d03eddec5dbf91f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 1 -- 172.21.15.50:0/1689518069 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73802ea10 con 0x7fa730006240 2023-10-30T12:18:54.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:54.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: dump: 2023-10-30T12:18:54.132 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.132 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.132 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.132 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.133 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.133 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.134 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.134 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.134 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:54.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:54.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:54.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 1 -- 172.21.15.50:0/1689518069 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa738032070 con 0x7fa730006240 2023-10-30T12:18:54.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa749b73640 5 monclient: authenticate success, global_id 204508 2023-10-30T12:18:54.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa749b73640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:54.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:54.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa749b73640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:54.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 1 -- 172.21.15.50:0/1689518069 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73802edb0 con 0x7fa730006240 2023-10-30T12:18:54.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa749b73640 4 set_mon_vals no callback set 2023-10-30T12:18:54.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:54.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.113+0000 7fa7427fc640 10 monclient: dump: 2023-10-30T12:18:54.140 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.140 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.140 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.141 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.141 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.141 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.142 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.142 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.142 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.143 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 10 monclient: shutdown 2023-10-30T12:18:54.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:54.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 1 -- 172.21.15.50:0/1689518069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa730006240 msgr2=0x7fa7300086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa730006240 0x7fa7300086e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa74409c5f0 tx=0x7fa73802dce0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa743fff640 1 -- 172.21.15.50:0/1689518069 reap_dead start 2023-10-30T12:18:54.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 1 -- 172.21.15.50:0/1689518069 shutdown_connections 2023-10-30T12:18:54.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 1 --2- 172.21.15.50:0/1689518069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa730006240 0x7fa7300086e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 1 -- 172.21.15.50:0/1689518069 >> 172.21.15.50:0/1689518069 conn(0x7fa7440a05c0 msgr2=0x7fa74409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:54.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 1 -- 172.21.15.50:0/1689518069 shutdown_connections 2023-10-30T12:18:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.114+0000 7fa749b73640 1 -- 172.21.15.50:0/1689518069 wait complete. 2023-10-30T12:18:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.115+0000 7fa749b73640 10 monclient: build_initial_monmap 2023-10-30T12:18:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.115+0000 7fa749b73640 10 monclient: monmap: 2023-10-30T12:18:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:54.147 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:54.147 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:54.147 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.115+0000 7fa749b73640 1 librados: starting msgr at 2023-10-30T12:18:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.115+0000 7fa749b73640 1 librados: starting objecter 2023-10-30T12:18:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.115+0000 7fa749b73640 1 Processor -- start 2023-10-30T12:18:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 -- start start 2023-10-30T12:18:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 librados: setting wanted keys 2023-10-30T12:18:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 librados: calling monclient init 2023-10-30T12:18:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient: init 2023-10-30T12:18:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:18:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa730006240 0x7fa7441c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): picked mon.noname-c con 0x7fa730006240 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7440961a0 0x7fa744096580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): picked mon.noname-a con 0x7fa7440961a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa744096ac0 0x7fa74408afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): picked mon.noname-b con 0x7fa744096ac0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74408ce30 con 0x7fa7440961a0 2023-10-30T12:18:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74408e370 con 0x7fa744096ac0 2023-10-30T12:18:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa744097f90 con 0x7fa730006240 2023-10-30T12:18:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa749b73640 10 monclient(hunting): authenticate will time out at 1979.299853s 2023-10-30T12:18:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa742ffd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7440961a0 msgr2=0x7fa744096580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa742ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7440961a0 0x7fa744096580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa7437fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa730006240 0x7fa7441c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.116+0000 7fa7437fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa730006240 0x7fa7441c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55060/0 (socket says 172.21.15.50:55060) 2023-10-30T12:18:54.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 1 -- 172.21.15.50:0/3087814015 learned_addr learned my addr 172.21.15.50:0/3087814015 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 10 monclient(hunting): get_auth_request con 0x7fa730006240 auth_method 0 2023-10-30T12:18:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa744096ac0 0x7fa74408afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): get_auth_request con 0x7fa744096ac0 auth_method 0 2023-10-30T12:18:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient(hunting): handle_auth_done global_id 204511 payload 824 2023-10-30T12:18:54.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 -- 172.21.15.50:0/3087814015 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa730006240 msgr2=0x7fa7441c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa730006240 0x7fa7441c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 -- 172.21.15.50:0/3087814015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7440961a0 msgr2=0x7fa744096580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:54.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7440961a0 0x7fa744096580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:54.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 monclient: found mon.noname-b 2023-10-30T12:18:54.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:54.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.118:3300/0 2023-10-30T12:18:54.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 -- 172.21.15.50:0/3087814015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73800f040 con 0x7fa744096ac0 2023-10-30T12:18:54.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa7437fe640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa730006240 0x7fa7441c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:18:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa743fff640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa744096ac0 0x7fa74408afd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa74406b9b0 tx=0x7fa73c00bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 client.?.objecter ms_handle_connect 0x7fa744096ac0 2023-10-30T12:18:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 1 -- 172.21.15.50:0/3087814015 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c016430 con 0x7fa744096ac0 2023-10-30T12:18:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:54.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: dump: 2023-10-30T12:18:54.165 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.165 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.165 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.165 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.166 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.166 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.166 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.166 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.166 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.167 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: _finish_auth 0 2023-10-30T12:18:54.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:18:54.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:54.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 1 -- 172.21.15.50:0/3087814015 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c0165b0 con 0x7fa744096ac0 2023-10-30T12:18:54.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 5 monclient: authenticate success, global_id 204511 2023-10-30T12:18:54.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 10 monclient: _renew_subs 2023-10-30T12:18:54.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:18:54.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa744071df0 con 0x7fa744096ac0 2023-10-30T12:18:54.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:54.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 1 -- 172.21.15.50:0/3087814015 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c016960 con 0x7fa744096ac0 2023-10-30T12:18:54.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:54.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa72bfff640 10 monclient: dump: 2023-10-30T12:18:54.170 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.170 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.170 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.171 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.171 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.171 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.171 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.171 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.172 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.172 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 10 client.204511.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:54.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa741ffb640 4 set_mon_vals no callback set 2023-10-30T12:18:54.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 10 monclient: _renew_subs 2023-10-30T12:18:54.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:18:54.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74408c880 con 0x7fa744096ac0 2023-10-30T12:18:54.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.117+0000 7fa749b73640 1 librados: init done 2023-10-30T12:18:54.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.118+0000 7fa72bfff640 1 -- 172.21.15.50:0/3087814015 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa73c012020 con 0x7fa744096ac0 2023-10-30T12:18:54.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.118+0000 7fa749b73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:54.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.118+0000 7fa749b73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:54.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.118+0000 7fa749b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:18:54.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa72bfff640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa73003a160 0x7fa73003c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.118+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa744067980 con 0x7fa744096ac0 2023-10-30T12:18:54.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa72bfff640 1 -- 172.21.15.50:0/3087814015 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(563..563 src has 490..563) v4 ==== 6683+0+0 (secure 0 0 0) 0x7fa73c04ae10 con 0x7fa744096ac0 2023-10-30T12:18:54.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa72bfff640 10 client.204511.objecter ms_dispatch 0x7fa744000b90 osd_map(563..563 src has 490..563) v4 2023-10-30T12:18:54.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa72bfff640 3 client.204511.objecter handle_osd_map got epochs [563,563] > 0 2023-10-30T12:18:54.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa72bfff640 3 client.204511.objecter handle_osd_map decoding full epoch 563 2023-10-30T12:18:54.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa72bfff640 20 client.204511.objecter dump_active .. 0 homeless 2023-10-30T12:18:54.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa7437fe640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa73003a160 0x7fa73003c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa7437fe640 10 monclient: get_auth_request con 0x7fa73003a160 auth_method 0 2023-10-30T12:18:54.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa72bfff640 10 client.204511.objecter ms_handle_connect 0x7fa73003a160 2023-10-30T12:18:54.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.119+0000 7fa7437fe640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa73003a160 0x7fa73003c620 secure :-1 s=READY pgs=4277 cs=0 l=1 rev1=1 crypto rx=0x7fa734009f50 tx=0x7fa734003eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.124+0000 7fa72bfff640 1 -- 172.21.15.50:0/3087814015 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa73c006e60 con 0x7fa744096ac0 2023-10-30T12:18:54.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.124+0000 7fa72bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:54.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.124+0000 7fa72bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.374+0000 7fa749b73640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:18:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.374+0000 7fa749b73640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.374+0000 7fa749b73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.118:3300/0 2023-10-30T12:18:54.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.374+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa74404e160 con 0x7fa744096ac0 2023-10-30T12:18:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.375+0000 7fa72bfff640 1 -- 172.21.15.50:0/3087814015 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fa73c0147b0 con 0x7fa744096ac0 2023-10-30T12:18:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.375+0000 7fa72bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:18:54.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.375+0000 7fa72bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:54.378 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:18:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 10 librados: watch_flush enter 2023-10-30T12:18:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 10 librados: watch_flush exit 2023-10-30T12:18:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 20 client.204511.objecter shutdown clearing up homeless session... 2023-10-30T12:18:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 10 client.204511.objecter successfully canceled tick 2023-10-30T12:18:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa73003a160 msgr2=0x7fa73003c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa73003a160 0x7fa73003c620 secure :-1 s=READY pgs=4277 cs=0 l=1 rev1=1 crypto rx=0x7fa734009f50 tx=0x7fa734003eb0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 10 monclient: shutdown 2023-10-30T12:18:54.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa744096ac0 msgr2=0x7fa74408afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa744096ac0 0x7fa74408afd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa74406b9b0 tx=0x7fa73c00bf80 comp rx=0 tx=0).stop 2023-10-30T12:18:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 shutdown_connections 2023-10-30T12:18:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa73003a160 0x7fa73003c620 unknown :-1 s=CLOSED pgs=4277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa744096ac0 0x7fa74408afd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7440961a0 0x7fa744096580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 --2- 172.21.15.50:0/3087814015 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa730006240 0x7fa7441c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 >> 172.21.15.50:0/3087814015 conn(0x7fa7440a05c0 msgr2=0x7fa74406bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 shutdown_connections 2023-10-30T12:18:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 -- 172.21.15.50:0/3087814015 wait complete. 2023-10-30T12:18:54.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.377+0000 7fa749b73640 1 librados: shutdown 2023-10-30T12:18:54.389 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:18:54.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:18:54.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:18:54.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.494+0000 7f4e58bb0640 10 monclient: get_monmap_and_config 2023-10-30T12:18:54.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.494+0000 7f4e58bb0640 10 monclient: build_initial_monmap 2023-10-30T12:18:54.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.494+0000 7f4e58bb0640 10 monclient: monmap: 2023-10-30T12:18:54.496 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:54.495789+0000 2023-10-30T12:18:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:54.495789+0000 2023-10-30T12:18:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:54.498 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:54.499 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 1 Processor -- start 2023-10-30T12:18:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 1 -- start start 2023-10-30T12:18:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient: init 2023-10-30T12:18:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:18:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4e5409bb70 0x7f4e5409bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5409bb70 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e5409c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5409d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e54096080 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:54.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e54098170 con 0x7f4e54096080 2023-10-30T12:18:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5409c9d0 con 0x7f4e5409bb70 2023-10-30T12:18:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5409cf60 con 0x7f4e5409d3f0 2023-10-30T12:18:54.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.495+0000 7f4e58bb0640 10 monclient(hunting): authenticate will time out at 1979.678913s 2023-10-30T12:18:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e5409c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e5409c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55072/0 (socket says 172.21.15.50:55072) 2023-10-30T12:18:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 -- 172.21.15.50:0/3337874440 learned_addr learned my addr 172.21.15.50:0/3337874440 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:54.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): get_auth_request con 0x7f4e5409d3f0 auth_method 0 2023-10-30T12:18:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3337874440 >> v2:172.21.15.118:3300/0 conn(0x7f4e5409bb70 0x7f4e5409bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient(hunting): handle_auth_done global_id 224249 payload 274 2023-10-30T12:18:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 -- 172.21.15.50:0/3337874440 >> v2:172.21.15.118:3300/0 conn(0x7f4e5409bb70 msgr2=0x7f4e5409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 --2- 172.21.15.50:0/3337874440 >> v2:172.21.15.118:3300/0 conn(0x7f4e5409bb70 0x7f4e5409bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 -- 172.21.15.50:0/3337874440 >> v1:172.21.15.118:6789/0 conn(0x7f4e54096080 legacy=0x7f4e540964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:18:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:18:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 monclient: found mon.noname-b 2023-10-30T12:18:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:18:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e4bfff640 1 -- 172.21.15.50:0/3337874440 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e54097b20 con 0x7f4e5409d3f0 2023-10-30T12:18:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e5259c640 10 monclient: get_auth_request con 0x7f4e5409bb70 auth_method 0 2023-10-30T12:18:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.496+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3337874440 >> v2:172.21.15.118:3300/0 conn(0x7f4e5409bb70 0x7f4e5409bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:18:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e4bfff640 1 --2- 172.21.15.50:0/3337874440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e5409c490 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00bb10 tx=0x7f4e3c02dfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=714e1a823825ec0b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 1 -- 172.21.15.50:0/3337874440 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c02ed50 con 0x7f4e5409d3f0 2023-10-30T12:18:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:18:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: dump: 2023-10-30T12:18:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.515 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.516 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.517 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: _finish_auth 0 2023-10-30T12:18:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:18:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 1 -- 172.21.15.50:0/3337874440 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c036440 con 0x7f4e5409d3f0 2023-10-30T12:18:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 5 monclient: authenticate success, global_id 224249 2023-10-30T12:18:54.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:54.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:54.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:54.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 1 -- 172.21.15.50:0/3337874440 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0367d0 con 0x7f4e5409d3f0 2023-10-30T12:18:54.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 4 set_mon_vals no callback set 2023-10-30T12:18:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e51d9b640 10 monclient: dump: 2023-10-30T12:18:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.519 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.520 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.521 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 10 monclient: shutdown 2023-10-30T12:18:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3337874440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 msgr2=0x7f4e5409c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 --2- 172.21.15.50:0/3337874440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e5409c490 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00bb10 tx=0x7f4e3c02dfa0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3337874440 shutdown_connections 2023-10-30T12:18:54.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 --2- 172.21.15.50:0/3337874440 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e5409c490 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 --2- 172.21.15.50:0/3337874440 >> v2:172.21.15.118:3300/0 conn(0x7f4e5409bb70 0x7f4e5409bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3337874440 >> 172.21.15.50:0/3337874440 conn(0x7f4e540a3980 msgr2=0x7f4e540a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:54.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3337874440 shutdown_connections 2023-10-30T12:18:54.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.497+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3337874440 wait complete. 2023-10-30T12:18:54.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.498+0000 7f4e58bb0640 10 monclient: build_initial_monmap 2023-10-30T12:18:54.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.498+0000 7f4e58bb0640 10 monclient: monmap: 2023-10-30T12:18:54.523 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:54.525 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.498+0000 7f4e58bb0640 1 librados: starting msgr at 2023-10-30T12:18:54.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.498+0000 7f4e58bb0640 1 librados: starting objecter 2023-10-30T12:18:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.498+0000 7f4e58bb0640 1 Processor -- start 2023-10-30T12:18:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.498+0000 7f4e58bb0640 1 -- start start 2023-10-30T12:18:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.498+0000 7f4e58bb0640 1 librados: setting wanted keys 2023-10-30T12:18:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 1 librados: calling monclient init 2023-10-30T12:18:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient: init 2023-10-30T12:18:54.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:18:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54096080 0x7f4e54095440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): picked mon.noname-b con 0x7f4e54096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5409bb70 0x7f4e54095980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5409bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e541c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5409d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5408efc0 con 0x7f4e5409bb70 2023-10-30T12:18:54.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54095ec0 con 0x7f4e54096080 2023-10-30T12:18:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e541b9530 con 0x7f4e5409d3f0 2023-10-30T12:18:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e58bb0640 10 monclient(hunting): authenticate will time out at 1979.682429s 2023-10-30T12:18:54.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e4bfff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5409bb70 msgr2=0x7f4e54095980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e4bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5409bb70 0x7f4e54095980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e541c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e541c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55084/0 (socket says 172.21.15.50:55084) 2023-10-30T12:18:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 -- 172.21.15.50:0/3040729771 learned_addr learned my addr 172.21.15.50:0/3040729771 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:54.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): get_auth_request con 0x7f4e5409d3f0 auth_method 0 2023-10-30T12:18:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54096080 0x7f4e54095440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient(hunting): handle_auth_done global_id 224252 payload 824 2023-10-30T12:18:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54096080 msgr2=0x7f4e54095440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54096080 0x7f4e54095440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5409bb70 msgr2=0x7f4e54095980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5409bb70 0x7f4e54095980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 monclient: found mon.noname-c 2023-10-30T12:18:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e52d9d640 1 -- 172.21.15.50:0/3040729771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c00b770 con 0x7f4e5409d3f0 2023-10-30T12:18:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e5259c640 10 monclient: get_auth_request con 0x7f4e54096080 auth_method 0 2023-10-30T12:18:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.499+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e54096080 0x7f4e54095440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:18:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e52d9d640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e541c5f50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00c780 tx=0x7f4e4c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 client.?.objecter ms_handle_connect 0x7f4e5409d3f0 2023-10-30T12:18:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c004b60 con 0x7f4e5409d3f0 2023-10-30T12:18:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: dump: 2023-10-30T12:18:54.536 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.536 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.536 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.538 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: _finish_auth 0 2023-10-30T12:18:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: _check_auth_tickets 2023-10-30T12:18:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:54.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c00ce90 con 0x7f4e5409d3f0 2023-10-30T12:18:54.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 5 monclient: authenticate success, global_id 224252 2023-10-30T12:18:54.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 10 monclient: _renew_subs 2023-10-30T12:18:54.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:54.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e540976e0 con 0x7f4e5409d3f0 2023-10-30T12:18:54.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:54.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c00a850 con 0x7f4e5409d3f0 2023-10-30T12:18:54.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:54.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 10 monclient: dump: 2023-10-30T12:18:54.540 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.540 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.542 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e5159a640 4 set_mon_vals no callback set 2023-10-30T12:18:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 10 client.224252.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 10 monclient: _renew_subs 2023-10-30T12:18:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5408e900 con 0x7f4e5409d3f0 2023-10-30T12:18:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 1 librados: init done 2023-10-30T12:18:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4e4c00a9b0 con 0x7f4e5409d3f0 2023-10-30T12:18:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.500+0000 7f4e58bb0640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e541c34a0 0x7f4e541c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e58bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54073ff0 con 0x7f4e541c34a0 2023-10-30T12:18:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e541c34a0 0x7f4e541c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient: get_auth_request con 0x7f4e541c34a0 auth_method 0 2023-10-30T12:18:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e340392a0 0x7f4e3403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(563..563 src has 490..563) v4 ==== 6683+0+0 (secure 0 0 0) 0x7f4e4c04b0a0 con 0x7f4e5409d3f0 2023-10-30T12:18:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 client.224252.objecter ms_dispatch 0x7f4e54000b90 osd_map(563..563 src has 490..563) v4 2023-10-30T12:18:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 3 client.224252.objecter handle_osd_map got epochs [563,563] > 0 2023-10-30T12:18:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 3 client.224252.objecter handle_osd_map decoding full epoch 563 2023-10-30T12:18:54.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 20 client.224252.objecter dump_active .. 0 homeless 2023-10-30T12:18:54.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 10 monclient(hunting): handle_auth_done global_id 224255 payload 824 2023-10-30T12:18:54.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 1 -- 172.21.15.50:0/3040729771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e54097560 con 0x7f4e541c34a0 2023-10-30T12:18:54.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4bfff640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e340392a0 0x7f4e3403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4bfff640 10 monclient: get_auth_request con 0x7f4e340392a0 auth_method 0 2023-10-30T12:18:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 client.224252.objecter ms_handle_connect 0x7f4e541c34a0 2023-10-30T12:18:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 client.224252.objecter resend_mon_ops 2023-10-30T12:18:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e541c34a0 0x7f4e541c6490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400d790 tx=0x7f4e4400dc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44005b90 con 0x7f4e541c34a0 2023-10-30T12:18:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 monclient: dump: 2023-10-30T12:18:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.549 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.550 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.551 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f4e4400c040 con 0x7f4e541c34a0 2023-10-30T12:18:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:54.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e541c34a0 msgr2=0x7f4e541c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e541c34a0 0x7f4e541c6490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400d790 tx=0x7f4e4400dc50 comp rx=0 tx=0).stop 2023-10-30T12:18:54.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e541c34a0 msgr2=0x7f4e541c6490 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:54.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.501+0000 7f4e4affd640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e541c34a0 0x7f4e541c6490 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400d790 tx=0x7f4e4400dc50 comp rx=0 tx=0).stop 2023-10-30T12:18:54.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.502+0000 7f4e4affd640 10 client.224252.objecter ms_handle_connect 0x7f4e340392a0 2023-10-30T12:18:54.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.502+0000 7f4e4bfff640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e340392a0 0x7f4e3403b760 secure :-1 s=READY pgs=4278 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c038000 tx=0x7f4e3c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e58bb0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:54.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e58bb0640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5408cb70 0x7f4e541c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e58bb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54097e00 con 0x7f4e5408cb70 2023-10-30T12:18:54.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5408cb70 0x7f4e541c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient: get_auth_request con 0x7f4e5408cb70 auth_method 0 2023-10-30T12:18:54.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 10 monclient(hunting): handle_auth_done global_id 224258 payload 824 2023-10-30T12:18:54.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 1 -- 172.21.15.50:0/3040729771 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4e5408ec40 con 0x7f4e5408cb70 2023-10-30T12:18:54.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e4affd640 10 client.224252.objecter ms_handle_connect 0x7f4e5408cb70 2023-10-30T12:18:54.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e5259c640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5408cb70 0x7f4e541c69d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400d790 tx=0x7f4e440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e4affd640 10 client.224252.objecter resend_mon_ops 2023-10-30T12:18:54.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44006810 con 0x7f4e5408cb70 2023-10-30T12:18:54.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:54.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.508+0000 7f4e4affd640 10 monclient: dump: 2023-10-30T12:18:54.558 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.558 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.560 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4791 (secure 0 0 0) 0x7f4e44012460 con 0x7f4e5408cb70 2023-10-30T12:18:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e4affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e4affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5408cb70 msgr2=0x7f4e541c69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e4affd640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5408cb70 0x7f4e541c69d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400d790 tx=0x7f4e440046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e4affd640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5408cb70 msgr2=0x7f4e541c69d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e4affd640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5408cb70 0x7f4e541c69d0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400d790 tx=0x7f4e440046c0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.562 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:54.562 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:54.562 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:54.562 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:18:54.563 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 194, 2023-10-30T12:18:54.563 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:54.563 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:54.563 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:54.563 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.563 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:18:54.564 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:54.564 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:54.564 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:54.564 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:54.564 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:54.565 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:54.565 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:54.565 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:54.565 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:54.565 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:54.566 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:54.566 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:54.566 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:54.566 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.566 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:54.567 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:54.567 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:54.567 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:54.567 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:54.567 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:54.567 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:54.568 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:54.568 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:54.568 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:54.568 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:54.568 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:54.569 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.569 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:54.569 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:54.569 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:54.569 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:54.570 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:54.570 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:54.570 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:54.570 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:54.570 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:54.571 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:54.571 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:54.571 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:54.571 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:54.571 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:54.572 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:54.572 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:54.572 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:54.572 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:54.572 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:54.573 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:54.573 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:54.573 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:54.573 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:54.573 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:54.573 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:54.574 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:54.574 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:54.574 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.574 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:54.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.575 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:54.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.575 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:54.575 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:54.575 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:54.576 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:54.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.576 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:54.576 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:54.576 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.577 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:54.577 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:54.577 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.577 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.578 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.578 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:54.578 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:54.578 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:54.579 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:54.579 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:54.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.579 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:54.579 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:54.580 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:54.580 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:54.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.580 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:54.580 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:54.581 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.581 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:54.581 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:54.581 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.582 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.582 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.582 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:54.582 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.584 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:54.584 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:54.584 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:54.584 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:54.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.584 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:18:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:18:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:18:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:18:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:54.594 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:54.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 10 librados: watch_flush enter 2023-10-30T12:18:54.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 10 librados: watch_flush exit 2023-10-30T12:18:54.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 20 client.224252.objecter shutdown clearing up homeless session... 2023-10-30T12:18:54.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 10 client.224252.objecter successfully canceled tick 2023-10-30T12:18:54.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e340392a0 msgr2=0x7f4e3403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4e340392a0 0x7f4e3403b760 secure :-1 s=READY pgs=4278 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c038000 tx=0x7f4e3c032000 comp rx=0 tx=0).stop 2023-10-30T12:18:54.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 10 monclient: shutdown 2023-10-30T12:18:54.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:54.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 msgr2=0x7f4e541c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 --2- 172.21.15.50:0/3040729771 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4e5409d3f0 0x7f4e541c5f50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00c780 tx=0x7f4e4c00cc40 comp rx=0 tx=0).stop 2023-10-30T12:18:54.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e52d9d640 1 -- 172.21.15.50:0/3040729771 reap_dead start 2023-10-30T12:18:54.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e52d9d640 1 -- 172.21.15.50:0/3040729771 reap_dead start 2023-10-30T12:18:54.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 shutdown_connections 2023-10-30T12:18:54.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 >> 172.21.15.50:0/3040729771 conn(0x7f4e540a3980 msgr2=0x7f4e5409a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:54.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 shutdown_connections 2023-10-30T12:18:54.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 -- 172.21.15.50:0/3040729771 wait complete. 2023-10-30T12:18:54.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.509+0000 7f4e58bb0640 1 librados: shutdown 2023-10-30T12:18:54.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:18:54.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:18:54.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:18:54.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.705+0000 7f09081d0640 10 monclient: get_monmap_and_config 2023-10-30T12:18:54.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.705+0000 7f09081d0640 10 monclient: build_initial_monmap 2023-10-30T12:18:54.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.705+0000 7f09081d0640 10 monclient: monmap: 2023-10-30T12:18:54.707 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:54.707 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.707 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:18:54.706314+0000 2023-10-30T12:18:54.708 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:18:54.706314+0000 2023-10-30T12:18:54.708 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:54.708 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.708 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:18:54.708 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:18:54.709 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:18:54.709 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:18:54.709 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:18:54.709 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.705+0000 7f09081d0640 1 Processor -- start 2023-10-30T12:18:54.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 1 -- start start 2023-10-30T12:18:54.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient: init 2023-10-30T12:18:54.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:54.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-10-30T12:18:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0900071c00 addr v1:172.21.15.118:6789/0 2023-10-30T12:18:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0900097fe0 0x7f090009a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient(hunting): picked mon.noname-c con 0x7f0900097fe0 addr v2:172.21.15.118:3300/0 2023-10-30T12:18:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f090009b170 addr v1:172.21.15.50:6789/0 2023-10-30T12:18:54.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0900097600 con 0x7f090009b170 2023-10-30T12:18:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09000978e0 con 0x7f0900071c00 2023-10-30T12:18:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090009bf50 con 0x7f0900097fe0 2023-10-30T12:18:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f09081d0640 10 monclient(hunting): authenticate will time out at 1979.889500s 2023-10-30T12:18:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090676d640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f090009b170 legacy=0x7f0900139760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:18:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0900097fe0 0x7f090009a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0900097fe0 0x7f090009a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:52396/0 (socket says 172.21.15.50:52396) 2023-10-30T12:18:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 1 -- 172.21.15.50:0/1929932610 learned_addr learned my addr 172.21.15.50:0/1929932610 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 10 monclient(hunting): get_auth_request con 0x7f0900097fe0 auth_method 0 2023-10-30T12:18:54.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.706+0000 7f090576b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 10 monclient(hunting): handle_auth_done global_id 204514 payload 274 2023-10-30T12:18:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 1 -- 172.21.15.50:0/1929932610 >> v1:172.21.15.118:6789/0 conn(0x7f0900071c00 legacy=0x7f090009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 1 -- 172.21.15.50:0/1929932610 >> v1:172.21.15.50:6789/0 conn(0x7f090009b170 legacy=0x7f0900139760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:18:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 10 monclient: _finish_hunting 0 2023-10-30T12:18:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 1 monclient: found mon.noname-c 2023-10-30T12:18:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:18:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 1 -- 172.21.15.50:0/1929932610 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090009d9b0 con 0x7f0900097fe0 2023-10-30T12:18:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.707+0000 7f090576b640 1 --2- 172.21.15.50:0/1929932610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 0x7f090009a150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009870 tx=0x7f08f402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e49a176d6e8b8ac7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 1 -- 172.21.15.50:0/1929932610 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f40317e0 con 0x7f0900097fe0 2023-10-30T12:18:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: dump: 2023-10-30T12:18:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.726 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: _finish_auth 0 2023-10-30T12:18:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: _check_auth_tickets 2023-10-30T12:18:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 5 monclient: authenticate success, global_id 204514 2023-10-30T12:18:54.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:18:54.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 1 -- 172.21.15.50:0/1929932610 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f4031940 con 0x7f0900097fe0 2023-10-30T12:18:54.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:54.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:18:54.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 4 set_mon_vals no callback set 2023-10-30T12:18:54.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 1 -- 172.21.15.50:0/1929932610 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4031cd0 con 0x7f0900097fe0 2023-10-30T12:18:54.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:54.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f0904f6a640 10 monclient: dump: 2023-10-30T12:18:54.728 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.728 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.730 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 10 monclient: shutdown 2023-10-30T12:18:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 1 -- 172.21.15.50:0/1929932610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 msgr2=0x7f090009a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 1 --2- 172.21.15.50:0/1929932610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 0x7f090009a150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009870 tx=0x7f08f402b880 comp rx=0 tx=0).stop 2023-10-30T12:18:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 1 -- 172.21.15.50:0/1929932610 shutdown_connections 2023-10-30T12:18:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 1 --2- 172.21.15.50:0/1929932610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 0x7f090009a150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 1 -- 172.21.15.50:0/1929932610 >> 172.21.15.50:0/1929932610 conn(0x7f09000a3980 msgr2=0x7f09000a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 1 -- 172.21.15.50:0/1929932610 shutdown_connections 2023-10-30T12:18:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.708+0000 7f09081d0640 1 -- 172.21.15.50:0/1929932610 wait complete. 2023-10-30T12:18:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 10 monclient: build_initial_monmap 2023-10-30T12:18:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 10 monclient: monmap: 2023-10-30T12:18:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:18:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:18:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:18:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:18:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:18:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:18:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:18:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:18:54.734 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 1 librados: starting msgr at 2023-10-30T12:18:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 1 librados: starting objecter 2023-10-30T12:18:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 1 Processor -- start 2023-10-30T12:18:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 1 -- start start 2023-10-30T12:18:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 1 librados: setting wanted keys 2023-10-30T12:18:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 1 librados: calling monclient init 2023-10-30T12:18:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.709+0000 7f09081d0640 10 monclient: init 2023-10-30T12:18:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:18:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:18:54.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0900071c00 0x7f0900094ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0900071c00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:18:54.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 0x7f090006bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0900097fe0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:18:54.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f090009b170 0x7f090006c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): picked mon.noname-c con 0x7f090009b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:18:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0900095850 con 0x7f0900071c00 2023-10-30T12:18:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0900071230 con 0x7f0900097fe0 2023-10-30T12:18:54.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0900095200 con 0x7f090009b170 2023-10-30T12:18:54.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:18:54.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f0905f6c640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0900071c00 msgr2=0x7f0900094ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:18:54.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient(hunting): authenticate will time out at 1979.893119s 2023-10-30T12:18:54.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f0905f6c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0900071c00 0x7f0900094ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:18:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f090009b170 0x7f090006c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f090009b170 0x7f090006c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55114/0 (socket says 172.21.15.50:55114) 2023-10-30T12:18:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 -- 172.21.15.50:0/2211583525 learned_addr learned my addr 172.21.15.50:0/2211583525 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:18:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): get_auth_request con 0x7f090009b170 auth_method 0 2023-10-30T12:18:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090576b640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 0x7f090006bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090576b640 10 monclient(hunting): get_auth_request con 0x7f0900097fe0 auth_method 0 2023-10-30T12:18:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090576b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090576b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090576b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient(hunting): handle_auth_done global_id 224261 payload 824 2023-10-30T12:18:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 msgr2=0x7f090006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900097fe0 0x7f090006bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0900071c00 msgr2=0x7f0900094ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:18:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0900071c00 0x7f0900094ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient: _finish_hunting 0 2023-10-30T12:18:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 monclient: found mon.noname-c 2023-10-30T12:18:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:18:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:18:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 -- 172.21.15.50:0/2211583525 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f40094d0 con 0x7f090009b170 2023-10-30T12:18:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f090676d640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f090009b170 0x7f090006c110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f08f800be50 tx=0x7f08f800be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 client.?.objecter ms_handle_connect 0x7f090009b170 2023-10-30T12:18:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:18:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8016430 con 0x7f090009b170 2023-10-30T12:18:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:18:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: dump: 2023-10-30T12:18:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.746 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: _finish_auth 0 2023-10-30T12:18:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:18:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:18:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f8016590 con 0x7f090009b170 2023-10-30T12:18:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 5 monclient: authenticate success, global_id 224261 2023-10-30T12:18:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient: _renew_subs 2023-10-30T12:18:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09000954e0 con 0x7f090009b170 2023-10-30T12:18:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:18:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8016980 con 0x7f090009b170 2023-10-30T12:18:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:18:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f27fc640 10 monclient: dump: 2023-10-30T12:18:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.751 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f08f3fff640 4 set_mon_vals no callback set 2023-10-30T12:18:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 client.224261.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:18:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient: _renew_subs 2023-10-30T12:18:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:18:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0900094310 con 0x7f090009b170 2023-10-30T12:18:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.710+0000 7f09081d0640 1 librados: init done 2023-10-30T12:18:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f09081d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f09081d0640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900095ec0 0x7f090002c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f09081d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0900073ff0 con 0x7f0900095ec0 2023-10-30T12:18:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f08f8012020 con 0x7f090009b170 2023-10-30T12:18:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f08f27fc640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08e40392a0 0x7f08e403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900095ec0 0x7f090002c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 10 monclient: get_auth_request con 0x7f0900095ec0 auth_method 0 2023-10-30T12:18:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(563..563 src has 490..563) v4 ==== 6683+0+0 (secure 0 0 0) 0x7f08f804b130 con 0x7f090009b170 2023-10-30T12:18:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f08f27fc640 10 client.224261.objecter ms_dispatch 0x7f0900000b90 osd_map(563..563 src has 490..563) v4 2023-10-30T12:18:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f08f27fc640 3 client.224261.objecter handle_osd_map got epochs [563,563] > 0 2023-10-30T12:18:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f08f27fc640 3 client.224261.objecter handle_osd_map decoding full epoch 563 2023-10-30T12:18:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f0905f6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.711+0000 7f08f27fc640 20 client.224261.objecter dump_active .. 0 homeless 2023-10-30T12:18:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f090576b640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08e40392a0 0x7f08e403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f090576b640 10 monclient: get_auth_request con 0x7f08e40392a0 auth_method 0 2023-10-30T12:18:54.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f0905f6c640 10 monclient(hunting): handle_auth_done global_id 204523 payload 824 2023-10-30T12:18:54.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f0905f6c640 1 -- 172.21.15.50:0/2211583525 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0900071fe0 con 0x7f0900095ec0 2023-10-30T12:18:54.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f0905f6c640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900095ec0 0x7f090002c5b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08fc001080 tx=0x7f08fc00c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 client.224261.objecter ms_handle_connect 0x7f0900095ec0 2023-10-30T12:18:54.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 client.224261.objecter resend_mon_ops 2023-10-30T12:18:54.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc0074e0 con 0x7f0900095ec0 2023-10-30T12:18:54.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f090576b640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08e40392a0 0x7f08e403b760 secure :-1 s=READY pgs=4279 cs=0 l=1 rev1=1 crypto rx=0x7f08f4036000 tx=0x7f08f403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:54.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 monclient: dump: 2023-10-30T12:18:54.758 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.758 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.758 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.758 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.758 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.759 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.759 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.759 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.759 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.759 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 client.224261.objecter ms_handle_connect 0x7f08e40392a0 2023-10-30T12:18:54.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f08fc007640 con 0x7f0900095ec0 2023-10-30T12:18:54.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:18:54.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:18:54.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900095ec0 msgr2=0x7f090002c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900095ec0 0x7f090002c5b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08fc001080 tx=0x7f08fc00c810 comp rx=0 tx=0).stop 2023-10-30T12:18:54.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900095ec0 msgr2=0x7f090002c5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:54.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.712+0000 7f08f27fc640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0900095ec0 0x7f090002c5b0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08fc001080 tx=0x7f08fc00c810 comp rx=0 tx=0).stop 2023-10-30T12:18:54.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f09081d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:18:54.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f09081d0640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f090006c650 0x7f0900060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:18:54.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f09081d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:18:54.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090006ca60 con 0x7f090006c650 2023-10-30T12:18:54.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f090006c650 0x7f0900060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:18:54.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient: get_auth_request con 0x7f090006c650 auth_method 0 2023-10-30T12:18:54.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:18:54.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:18:54.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:18:54.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:18:54.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:18:54.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:18:54.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 10 monclient(hunting): handle_auth_done global_id 204526 payload 824 2023-10-30T12:18:54.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 1 -- 172.21.15.50:0/2211583525 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f09000959c0 con 0x7f090006c650 2023-10-30T12:18:54.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f0905f6c640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f090006c650 0x7f0900060fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08fc001f00 tx=0x7f08fc000f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:18:54.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f08f27fc640 10 client.224261.objecter ms_handle_connect 0x7f090006c650 2023-10-30T12:18:54.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f08f27fc640 10 client.224261.objecter resend_mon_ops 2023-10-30T12:18:54.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.719+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc003b90 con 0x7f090006c650 2023-10-30T12:18:54.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:18:54.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:18:54.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 10 monclient: dump: 2023-10-30T12:18:54.767 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:18:54.767 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:18:54.767 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:18:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:18:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:18:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:18:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:18:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:18:54.769 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:18:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7f08fc003cf0 con 0x7f090006c650 2023-10-30T12:18:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:18:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:18:54.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f090006c650 msgr2=0x7f0900060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f090006c650 0x7f0900060fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08fc001f00 tx=0x7f08fc000f20 comp rx=0 tx=0).stop 2023-10-30T12:18:54.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f090006c650 msgr2=0x7f0900060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:18:54.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f08f27fc640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f090006c650 0x7f0900060fc0 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f08fc001f00 tx=0x7f08fc000f20 comp rx=0 tx=0).stop 2023-10-30T12:18:54.770 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:18:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:18:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 194, 2023-10-30T12:18:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:18:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:18:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:18:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:18:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:18:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:18:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:18:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:18:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:18:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:18:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:18:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:18:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:18:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:18:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:18:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:18:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:18:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:18:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:18:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:18:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:18:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:18:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:18:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:18:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:18:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:18:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:18:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:18:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:18:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:18:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:18:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:18:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:18:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:18:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:18:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:18:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:18:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:18:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:18:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:18:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:18:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:18:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:18:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:18:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:18:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:18:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:18:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:18:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:18:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:18:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:18:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:18:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:18:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:18:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:18:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:18:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:18:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:18:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:18:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:18:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:18:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:18:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:18:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:18:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:18:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:18:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:18:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:18:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:18:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:18:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:18:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:18:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:18:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:18:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:18:54.800 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:18:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 10 librados: watch_flush enter 2023-10-30T12:18:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 10 librados: watch_flush exit 2023-10-30T12:18:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 20 client.224261.objecter shutdown clearing up homeless session... 2023-10-30T12:18:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 10 client.224261.objecter successfully canceled tick 2023-10-30T12:18:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08e40392a0 msgr2=0x7f08e403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08e40392a0 0x7f08e403b760 secure :-1 s=READY pgs=4279 cs=0 l=1 rev1=1 crypto rx=0x7f08f4036000 tx=0x7f08f403d040 comp rx=0 tx=0).stop 2023-10-30T12:18:54.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 10 monclient: shutdown 2023-10-30T12:18:54.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:18:54.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f090009b170 msgr2=0x7f090006c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:18:54.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f090676d640 1 -- 172.21.15.50:0/2211583525 reap_dead start 2023-10-30T12:18:54.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f090009b170 0x7f090006c110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f08f800be50 tx=0x7f08f800be80 comp rx=0 tx=0).stop 2023-10-30T12:18:54.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 shutdown_connections 2023-10-30T12:18:54.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 --2- 172.21.15.50:0/2211583525 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f090009b170 0x7f090006c110 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:18:54.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 >> 172.21.15.50:0/2211583525 conn(0x7f09000a3980 msgr2=0x7f0900071440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:18:54.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 shutdown_connections 2023-10-30T12:18:54.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 -- 172.21.15.50:0/2211583525 wait complete. 2023-10-30T12:18:54.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:18:54.720+0000 7f09081d0640 1 librados: shutdown 2023-10-30T12:18:54.805 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:18:54.805 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:19:00.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:19:00.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (23353 ms) 2023-10-30T12:19:00.008 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-10-30T12:19:00.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:19:00.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5974 ms) 2023-10-30T12:19:00.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-10-30T12:19:00.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:19:00.009 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5021 ms) 2023-10-30T12:19:00.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-10-30T12:19:00.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:19:00.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (38798 ms) 2023-10-30T12:19:00.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-10-30T12:19:00.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:19:00.010 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14040 ms) 2023-10-30T12:19:00.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-10-30T12:19:00.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (46336 ms) 2023-10-30T12:19:00.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-10-30T12:19:00.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-10-30T12:19:00.011 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: 00000010 69 73 |is| 2023-10-30T12:19:00.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: 00000012 2023-10-30T12:19:00.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12038 ms) 2023-10-30T12:19:00.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-10-30T12:19:00.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: require_osd_release = reef 2023-10-30T12:19:00.012 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5484 ms) 2023-10-30T12:19:00.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (801364 ms total) 2023-10-30T12:19:00.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: 2023-10-30T12:19:00.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-10-30T12:19:00.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-10-30T12:19:00.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1014 ms) 2023-10-30T12:19:00.013 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-10-30T12:19:00.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (24501 ms) 2023-10-30T12:19:00.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-10-30T12:19:00.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17975 ms) 2023-10-30T12:19:00.014 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-10-30T12:19:00.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2023-10-30T12:19:00.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (43492 ms total) 2023-10-30T12:19:00.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: 2023-10-30T12:19:00.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-10-30T12:19:00.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-10-30T12:19:00.015 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (20894 ms) 2023-10-30T12:19:00.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-10-30T12:19:00.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8027 ms) 2023-10-30T12:19:00.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-10-30T12:19:00.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: waiting for scrub... 2023-10-30T12:19:00.016 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: done waiting 2023-10-30T12:19:00.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (39364 ms) 2023-10-30T12:19:00.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-10-30T12:19:00.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9754 ms) 2023-10-30T12:19:00.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-10-30T12:19:00.017 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8000 ms) 2023-10-30T12:19:00.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-10-30T12:19:00.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8027 ms) 2023-10-30T12:19:00.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-10-30T12:19:00.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (21416 ms) 2023-10-30T12:19:00.018 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-10-30T12:19:14.806 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:19:14.807 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:19:14.807 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:19:14.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:19:14.812 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:19:14.812 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:19:14.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:14.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.928+0000 7fc39a9c9640 10 monclient: get_monmap_and_config 2023-10-30T12:19:14.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.928+0000 7fc39a9c9640 10 monclient: build_initial_monmap 2023-10-30T12:19:14.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.928+0000 7fc39a9c9640 10 monclient: monmap: 2023-10-30T12:19:14.930 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:14.930 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:14.931 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:14.929382+0000 2023-10-30T12:19:14.933 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:14.929382+0000 2023-10-30T12:19:14.933 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:14.935 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:14.935 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:14.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 Processor -- start 2023-10-30T12:19:14.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 -- start start 2023-10-30T12:19:14.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient: init 2023-10-30T12:19:14.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:14.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T12:19:14.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc3940a6070 0x7fc3940a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:14.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc3940a6070 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:14.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 0x7fc394135590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:14.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc3940a6450 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:14.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc394135ad0 0x7fc394135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:14.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc394135ad0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:14.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:14.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc394136460 con 0x7fc3940a6450 2023-10-30T12:19:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc39409a270 con 0x7fc3940a6070 2023-10-30T12:19:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc394099f90 con 0x7fc394135ad0 2023-10-30T12:19:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc39a9c9640 10 monclient(hunting): authenticate will time out at 2000.112590s 2023-10-30T12:19:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc394135ad0 0x7fc394135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 0x7fc394135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc394135ad0 0x7fc394135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:58648/0 (socket says 172.21.15.50:58648) 2023-10-30T12:19:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 1 -- 172.21.15.50:0/2983381053 learned_addr learned my addr 172.21.15.50:0/2983381053 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): get_auth_request con 0x7fc394135ad0 auth_method 0 2023-10-30T12:19:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 0x7fc394135590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33232/0 (socket says 172.21.15.50:33232) 2023-10-30T12:19:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 10 monclient(hunting): get_auth_request con 0x7fc3940a6450 auth_method 0 2023-10-30T12:19:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc393fff640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.118:3300/0 conn(0x7fc3940a6070 0x7fc3940a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 1 -- 172.21.15.50:0/2983381053 >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 msgr2=0x7fc394135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:19:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 1 -- 172.21.15.50:0/2983381053 >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 msgr2=0x7fc394135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc393fff640 10 monclient(hunting): get_auth_request con 0x7fc3940a6070 auth_method 0 2023-10-30T12:19:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc393fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc393fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 0x7fc394135590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc3937fe640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 0x7fc394135590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.929+0000 7fc398f66640 10 monclient(hunting): handle_auth_done global_id 224264 payload 274 2023-10-30T12:19:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 1 -- 172.21.15.50:0/2983381053 >> v2:172.21.15.118:3300/0 conn(0x7fc3940a6070 msgr2=0x7fc3940a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.118:3300/0 conn(0x7fc3940a6070 0x7fc3940a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 1 -- 172.21.15.50:0/2983381053 >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 msgr2=0x7fc394135590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 0x7fc394135590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 10 monclient: _finish_hunting 0 2023-10-30T12:19:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 1 monclient: found mon.noname-b 2023-10-30T12:19:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 1 -- 172.21.15.50:0/2983381053 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39409a550 con 0x7fc394135ad0 2023-10-30T12:19:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc393fff640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.118:3300/0 conn(0x7fc3940a6070 0x7fc3940a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:19:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc398f66640 1 --2- 172.21.15.50:0/2983381053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc394135ad0 0x7fc394135f20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc3940a4ee0 tx=0x7fc38c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a03127a1757f890b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 1 -- 172.21.15.50:0/2983381053 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c02c950 con 0x7fc394135ad0 2023-10-30T12:19:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 10 monclient: dump: 2023-10-30T12:19:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:14.957 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 10 monclient: _finish_auth 0 2023-10-30T12:19:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:19:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc39a9c9640 5 monclient: authenticate success, global_id 224264 2023-10-30T12:19:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc39a9c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.930+0000 7fc392ffd640 1 -- 172.21.15.50:0/2983381053 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38c02cab0 con 0x7fc394135ad0 2023-10-30T12:19:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc392ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc392ffd640 1 -- 172.21.15.50:0/2983381053 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c02ce40 con 0x7fc394135ad0 2023-10-30T12:19:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 4 set_mon_vals no callback set 2023-10-30T12:19:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc392ffd640 10 monclient: dump: 2023-10-30T12:19:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:14.962 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 10 monclient: shutdown 2023-10-30T12:19:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 -- 172.21.15.50:0/2983381053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc394135ad0 msgr2=0x7fc394135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/2983381053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc394135ad0 0x7fc394135f20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc3940a4ee0 tx=0x7fc38c02bdb0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 -- 172.21.15.50:0/2983381053 shutdown_connections 2023-10-30T12:19:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/2983381053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc394135ad0 0x7fc394135f20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.50:3300/0 conn(0x7fc3940a6450 0x7fc394135590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/2983381053 >> v2:172.21.15.118:3300/0 conn(0x7fc3940a6070 0x7fc3940a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 -- 172.21.15.50:0/2983381053 >> 172.21.15.50:0/2983381053 conn(0x7fc3940a81d0 msgr2=0x7fc3940a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 -- 172.21.15.50:0/2983381053 shutdown_connections 2023-10-30T12:19:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.931+0000 7fc39a9c9640 1 -- 172.21.15.50:0/2983381053 wait complete. 2023-10-30T12:19:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 10 monclient: build_initial_monmap 2023-10-30T12:19:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 10 monclient: monmap: 2023-10-30T12:19:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:14.966 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 1 librados: starting msgr at 2023-10-30T12:19:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 1 librados: starting objecter 2023-10-30T12:19:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 1 Processor -- start 2023-10-30T12:19:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 1 -- start start 2023-10-30T12:19:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 1 librados: setting wanted keys 2023-10-30T12:19:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 1 librados: calling monclient init 2023-10-30T12:19:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.932+0000 7fc39a9c9640 10 monclient: init 2023-10-30T12:19:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:19:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3940a6070 0x7fc39406bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc3940a6070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3940a6450 0x7fc39406c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc3940a6450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 0x7fc39406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc394135ad0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39406ef70 con 0x7fc394135ad0 2023-10-30T12:19:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3940a1690 con 0x7fc3940a6070 2023-10-30T12:19:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc394132530 con 0x7fc3940a6450 2023-10-30T12:19:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc39a9c9640 10 monclient(hunting): authenticate will time out at 2000.116284s 2023-10-30T12:19:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 0x7fc39406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3940a6450 0x7fc39406c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3940a6450 0x7fc39406c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:58656/0 (socket says 172.21.15.50:58656) 2023-10-30T12:19:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 -- 172.21.15.50:0/335687546 learned_addr learned my addr 172.21.15.50:0/335687546 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 0x7fc39406ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33238/0 (socket says 172.21.15.50:33238) 2023-10-30T12:19:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): get_auth_request con 0x7fc3940a6450 auth_method 0 2023-10-30T12:19:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 10 monclient(hunting): get_auth_request con 0x7fc394135ad0 auth_method 0 2023-10-30T12:19:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 1 -- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 msgr2=0x7fc39406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:19:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 1 -- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 msgr2=0x7fc39406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:19:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 0x7fc39406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc398f66640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 0x7fc39406ca70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc393fff640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3940a6070 0x7fc39406bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc393fff640 10 monclient(hunting): get_auth_request con 0x7fc3940a6070 auth_method 0 2023-10-30T12:19:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc393fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc393fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient(hunting): handle_auth_done global_id 224267 payload 824 2023-10-30T12:19:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 -- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3940a6070 msgr2=0x7fc39406bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3940a6070 0x7fc39406bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 -- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 msgr2=0x7fc39406ca70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 0x7fc39406ca70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient: _finish_hunting 0 2023-10-30T12:19:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 monclient: found mon.noname-c 2023-10-30T12:19:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:19:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 -- 172.21.15.50:0/335687546 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c009b30 con 0x7fc3940a6450 2023-10-30T12:19:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc3937fe640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3940a6450 0x7fc39406c530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc38800d7d0 tx=0x7fc38800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc390ff9640 10 client.?.objecter ms_handle_connect 0x7fc3940a6450 2023-10-30T12:19:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc390ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.933+0000 7fc390ff9640 1 -- 172.21.15.50:0/335687546 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38800b6c0 con 0x7fc3940a6450 2023-10-30T12:19:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: dump: 2023-10-30T12:19:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:14.980 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: _finish_auth 0 2023-10-30T12:19:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:19:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 5 monclient: authenticate success, global_id 224267 2023-10-30T12:19:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 1 -- 172.21.15.50:0/335687546 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38800b840 con 0x7fc3940a6450 2023-10-30T12:19:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 monclient: _renew_subs 2023-10-30T12:19:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39406f810 con 0x7fc3940a6450 2023-10-30T12:19:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 1 -- 172.21.15.50:0/335687546 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38800bbf0 con 0x7fc3940a6450 2023-10-30T12:19:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:14.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:14.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 10 monclient: dump: 2023-10-30T12:19:14.983 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:14.983 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:14.983 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.984 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:14.985 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 client.224267.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 monclient: _renew_subs 2023-10-30T12:19:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39406d3d0 con 0x7fc3940a6450 2023-10-30T12:19:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc3927fc640 4 set_mon_vals no callback set 2023-10-30T12:19:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 1 librados: init done 2023-10-30T12:19:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc390ff9640 1 -- 172.21.15.50:0/335687546 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc38800bd50 con 0x7fc3940a6450 2023-10-30T12:19:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.934+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39404e160 con 0x7fc3940a6450 2023-10-30T12:19:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc390ff9640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3800392f0 0x7fc38003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc390ff9640 1 -- 172.21.15.50:0/335687546 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(578..578 src has 490..578) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fc38804b2d0 con 0x7fc3940a6450 2023-10-30T12:19:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc390ff9640 10 client.224267.objecter ms_dispatch 0x7fc394000b90 osd_map(578..578 src has 490..578) v4 2023-10-30T12:19:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc390ff9640 3 client.224267.objecter handle_osd_map got epochs [578,578] > 0 2023-10-30T12:19:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc390ff9640 3 client.224267.objecter handle_osd_map decoding full epoch 578 2023-10-30T12:19:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc390ff9640 20 client.224267.objecter dump_active .. 0 homeless 2023-10-30T12:19:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc393fff640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3800392f0 0x7fc38003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc393fff640 10 monclient: get_auth_request con 0x7fc3800392f0 auth_method 0 2023-10-30T12:19:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc390ff9640 10 client.224267.objecter ms_handle_connect 0x7fc3800392f0 2023-10-30T12:19:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.935+0000 7fc393fff640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3800392f0 0x7fc38003b7b0 secure :-1 s=READY pgs=4280 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:14.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.939+0000 7fc390ff9640 1 -- 172.21.15.50:0/335687546 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc38801e460 con 0x7fc3940a6450 2023-10-30T12:19:14.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.939+0000 7fc390ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:14.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:14.939+0000 7fc390ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:15.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.191+0000 7fc39a9c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:15.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.191+0000 7fc39a9c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:15.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.191+0000 7fc39a9c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:15.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.191+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3940a2dd0 con 0x7fc3940a6450 2023-10-30T12:19:15.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.191+0000 7fc390ff9640 1 -- 172.21.15.50:0/335687546 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fc388011340 con 0x7fc3940a6450 2023-10-30T12:19:15.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.191+0000 7fc390ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:15.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.191+0000 7fc390ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:15.194 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 10 librados: watch_flush enter 2023-10-30T12:19:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 10 librados: watch_flush exit 2023-10-30T12:19:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 20 client.224267.objecter shutdown clearing up homeless session... 2023-10-30T12:19:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 10 client.224267.objecter successfully canceled tick 2023-10-30T12:19:15.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3800392f0 msgr2=0x7fc38003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:15.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3800392f0 0x7fc38003b7b0 secure :-1 s=READY pgs=4280 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384008040 comp rx=0 tx=0).stop 2023-10-30T12:19:15.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 10 monclient: shutdown 2023-10-30T12:19:15.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:15.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3940a6450 msgr2=0x7fc39406c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3940a6450 0x7fc39406c530 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc38800d7d0 tx=0x7fc38800dc90 comp rx=0 tx=0).stop 2023-10-30T12:19:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 shutdown_connections 2023-10-30T12:19:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3800392f0 0x7fc38003b7b0 unknown :-1 s=CLOSED pgs=4280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc394135ad0 0x7fc39406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3940a6450 0x7fc39406c530 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 --2- 172.21.15.50:0/335687546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3940a6070 0x7fc39406bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:15.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 >> 172.21.15.50:0/335687546 conn(0x7fc3940a81d0 msgr2=0x7fc394131a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:15.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 shutdown_connections 2023-10-30T12:19:15.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 -- 172.21.15.50:0/335687546 wait complete. 2023-10-30T12:19:15.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:15.193+0000 7fc39a9c9640 1 librados: shutdown 2023-10-30T12:19:18.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.313+0000 7fced3a56640 10 monclient: get_monmap_and_config 2023-10-30T12:19:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.313+0000 7fced3a56640 10 monclient: build_initial_monmap 2023-10-30T12:19:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.313+0000 7fced3a56640 10 monclient: monmap: 2023-10-30T12:19:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:18.316 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:18.314882+0000 2023-10-30T12:19:18.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:18.314882+0000 2023-10-30T12:19:18.317 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:18.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:18.317 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:18.317 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:18.318 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:18.318 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:18.318 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:18.318 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:18.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.314+0000 7fced3a56640 1 Processor -- start 2023-10-30T12:19:18.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.314+0000 7fced3a56640 1 -- start start 2023-10-30T12:19:18.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.314+0000 7fced3a56640 10 monclient: init 2023-10-30T12:19:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-10-30T12:19:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcecc0a5100 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcecc0a5bf0 0x7fcecc09fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:18.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient(hunting): picked mon.noname-c con 0x7fcecc0a5bf0 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcecc138940 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc0a00b0 con 0x7fcecc0a5100 2023-10-30T12:19:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc09e8f0 con 0x7fcecc138940 2023-10-30T12:19:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:18.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcecc09e380 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced3a56640 10 monclient(hunting): authenticate will time out at 2003.498148s 2023-10-30T12:19:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced17f2640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcecc0a5100 0x7fcecc0a54e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49870/0 (socket says 172.21.15.50:49870) 2023-10-30T12:19:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced17f2640 1 -- 172.21.15.50:0/1410208212 learned_addr learned my addr 172.21.15.50:0/1410208212 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 1 --2- 172.21.15.50:0/1410208212 >> v2:172.21.15.118:3300/0 conn(0x7fcecc0a5bf0 0x7fcecc09fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): get_auth_request con 0x7fcecc0a5bf0 auth_method 0 2023-10-30T12:19:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fcec3fff640 1 -- 172.21.15.50:0/1410208212 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472916118 0 0) 0x7fcecc09e8f0 con 0x7fcecc138940 2023-10-30T12:19:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fcec3fff640 10 monclient(hunting): _negotiate 2023-10-30T12:19:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fcec3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fcec3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fcec3fff640 10 monclient(hunting): my global_id is 204538 2023-10-30T12:19:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fcec3fff640 1 -- 172.21.15.50:0/1410208212 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb8003520 con 0x7fcecc138940 2023-10-30T12:19:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient(hunting): handle_auth_done global_id 204535 payload 274 2023-10-30T12:19:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 1 -- 172.21.15.50:0/1410208212 >> v1:172.21.15.118:6789/0 conn(0x7fcecc138940 legacy=0x7fcecc13ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 1 -- 172.21.15.50:0/1410208212 >> v1:172.21.15.50:6789/0 conn(0x7fcecc0a5100 legacy=0x7fcecc0a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient: _finish_hunting 0 2023-10-30T12:19:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 1 monclient: found mon.noname-c 2023-10-30T12:19:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:19:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.315+0000 7fced0ff1640 1 -- 172.21.15.50:0/1410208212 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc09ebd0 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fced0ff1640 1 --2- 172.21.15.50:0/1410208212 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc0a5bf0 0x7fcecc09fb70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0097a0 tx=0x7fcebc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ae6b442f512359 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 1 -- 172.21.15.50:0/1410208212 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc02e070 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:18.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: dump: 2023-10-30T12:19:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:18.345 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:18.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: _finish_auth 0 2023-10-30T12:19:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 1 -- 172.21.15.50:0/1410208212 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc0315a0 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fced3a56640 5 monclient: authenticate success, global_id 204535 2023-10-30T12:19:18.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fced3a56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 1 -- 172.21.15.50:0/1410208212 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc031930 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:18.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fcec3fff640 10 monclient: dump: 2023-10-30T12:19:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:18.350 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fced3a56640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.316+0000 7fced3a56640 4 set_mon_vals no callback set 2023-10-30T12:19:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 10 monclient: shutdown 2023-10-30T12:19:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 1 -- 172.21.15.50:0/1410208212 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc0a5bf0 msgr2=0x7fcecc09fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 1 --2- 172.21.15.50:0/1410208212 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc0a5bf0 0x7fcecc09fb70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0097a0 tx=0x7fcebc02b840 comp rx=0 tx=0).stop 2023-10-30T12:19:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 1 -- 172.21.15.50:0/1410208212 shutdown_connections 2023-10-30T12:19:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 1 --2- 172.21.15.50:0/1410208212 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc0a5bf0 0x7fcecc09fb70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 1 -- 172.21.15.50:0/1410208212 >> 172.21.15.50:0/1410208212 conn(0x7fcecc0a3fc0 msgr2=0x7fcecc0a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 1 -- 172.21.15.50:0/1410208212 shutdown_connections 2023-10-30T12:19:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.317+0000 7fced3a56640 1 -- 172.21.15.50:0/1410208212 wait complete. 2023-10-30T12:19:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient: build_initial_monmap 2023-10-30T12:19:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient: monmap: 2023-10-30T12:19:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:18.354 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 librados: starting msgr at 2023-10-30T12:19:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 librados: starting objecter 2023-10-30T12:19:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 Processor -- start 2023-10-30T12:19:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 -- start start 2023-10-30T12:19:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 librados: setting wanted keys 2023-10-30T12:19:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 librados: calling monclient init 2023-10-30T12:19:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient: init 2023-10-30T12:19:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 0x7fcecc137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): picked mon.noname-a con 0x7fcecc0a5100 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcecc0a5bf0 0x7fcecc1379a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): picked mon.noname-c con 0x7fcecc0a5bf0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc138940 0x7fcecc09b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): picked mon.noname-b con 0x7fcecc138940 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc13ba40 con 0x7fcecc0a5100 2023-10-30T12:19:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc122cf0 con 0x7fcecc138940 2023-10-30T12:19:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc122730 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced3a56640 10 monclient(hunting): authenticate will time out at 2003.501862s 2023-10-30T12:19:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced17f2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 0x7fcecc137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced0ff1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcecc0a5bf0 0x7fcecc1379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced17f2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 0x7fcecc137460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35842/0 (socket says 172.21.15.50:35842) 2023-10-30T12:19:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced17f2640 1 -- 172.21.15.50:0/1839770278 learned_addr learned my addr 172.21.15.50:0/1839770278 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced17f2640 10 monclient(hunting): get_auth_request con 0x7fcecc0a5100 auth_method 0 2023-10-30T12:19:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced17f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced17f2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.318+0000 7fced17f2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): get_auth_request con 0x7fcecc0a5bf0 auth_method 0 2023-10-30T12:19:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc138940 0x7fcecc09b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 10 monclient(hunting): get_auth_request con 0x7fcecc138940 auth_method 0 2023-10-30T12:19:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced17f2640 1 -- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 msgr2=0x7fcecc137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:19:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced17f2640 1 -- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 msgr2=0x7fcecc137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:19:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced17f2640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 0x7fcecc137460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced17f2640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 0x7fcecc137460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced1ff3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient(hunting): handle_auth_done global_id 224270 payload 824 2023-10-30T12:19:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 1 -- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc138940 msgr2=0x7fcecc09b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc138940 0x7fcecc09b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 1 -- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 msgr2=0x7fcecc137460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 0x7fcecc137460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient: _finish_hunting 0 2023-10-30T12:19:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 1 monclient: found mon.noname-c 2023-10-30T12:19:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:19:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 1 -- 172.21.15.50:0/1839770278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc009400 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced0ff1640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcecc0a5bf0 0x7fcecc1379a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0a60e0 tx=0x7fcebc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 client.?.objecter ms_handle_connect 0x7fcecc0a5bf0 2023-10-30T12:19:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 1 -- 172.21.15.50:0/1839770278 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc004230 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: dump: 2023-10-30T12:19:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:18.369 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:19:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:19:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 5 monclient: authenticate success, global_id 224270 2023-10-30T12:19:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 1 -- 172.21.15.50:0/1839770278 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc031f00 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 1 -- 172.21.15.50:0/1839770278 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc032640 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec37fe640 4 set_mon_vals no callback set 2023-10-30T12:19:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fcec1ffb640 10 monclient: dump: 2023-10-30T12:19:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:18.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 10 monclient: _renew_subs 2023-10-30T12:19:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcecc122a10 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 10 client.224270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 10 monclient: _renew_subs 2023-10-30T12:19:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcecc1217d0 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.319+0000 7fced3a56640 1 librados: init done 2023-10-30T12:19:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.320+0000 7fcec1ffb640 1 -- 172.21.15.50:0/1839770278 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcebc003610 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.320+0000 7fced3a56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.320+0000 7fced3a56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.320+0000 7fced3a56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.320+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcecc04e160 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.320+0000 7fcec1ffb640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fceb00392f0 0x7fceb003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fcec1ffb640 1 -- 172.21.15.50:0/1839770278 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(582..582 src has 490..582) v4 ==== 6650+0+0 (secure 0 0 0) 0x7fcebc06c4d0 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fcec1ffb640 10 client.224270.objecter ms_dispatch 0x7fcecc000b90 osd_map(582..582 src has 490..582) v4 2023-10-30T12:19:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fcec1ffb640 3 client.224270.objecter handle_osd_map got epochs [582,582] > 0 2023-10-30T12:19:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fcec1ffb640 3 client.224270.objecter handle_osd_map decoding full epoch 582 2023-10-30T12:19:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fcec1ffb640 20 client.224270.objecter dump_active .. 0 homeless 2023-10-30T12:19:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fced17f2640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fceb00392f0 0x7fceb003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fced17f2640 10 monclient: get_auth_request con 0x7fceb00392f0 auth_method 0 2023-10-30T12:19:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fcec1ffb640 10 client.224270.objecter ms_handle_connect 0x7fceb00392f0 2023-10-30T12:19:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.321+0000 7fced17f2640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fceb00392f0 0x7fceb003b7b0 secure :-1 s=READY pgs=4282 cs=0 l=1 rev1=1 crypto rx=0x7fcec8005880 tx=0x7fcec8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.325+0000 7fcec1ffb640 1 -- 172.21.15.50:0/1839770278 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcebc0394f0 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.325+0000 7fcec1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.325+0000 7fcec1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:18.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.571+0000 7fced3a56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.571+0000 7fced3a56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.571+0000 7fced3a56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.571+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcecc122870 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.572+0000 7fcec1ffb640 1 -- 172.21.15.50:0/1839770278 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fcecc122870 con 0x7fcecc0a5bf0 2023-10-30T12:19:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.572+0000 7fcec1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.572+0000 7fcec1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:18.575 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 10 librados: watch_flush enter 2023-10-30T12:19:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 10 librados: watch_flush exit 2023-10-30T12:19:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 20 client.224270.objecter shutdown clearing up homeless session... 2023-10-30T12:19:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 10 client.224270.objecter successfully canceled tick 2023-10-30T12:19:18.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fceb00392f0 msgr2=0x7fceb003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:18.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fceb00392f0 0x7fceb003b7b0 secure :-1 s=READY pgs=4282 cs=0 l=1 rev1=1 crypto rx=0x7fcec8005880 tx=0x7fcec8009040 comp rx=0 tx=0).stop 2023-10-30T12:19:18.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 10 monclient: shutdown 2023-10-30T12:19:18.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:18.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcecc0a5bf0 msgr2=0x7fcecc1379a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:18.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcecc0a5bf0 0x7fcecc1379a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0a60e0 tx=0x7fcebc0046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 shutdown_connections 2023-10-30T12:19:18.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fceb00392f0 0x7fceb003b7b0 unknown :-1 s=CLOSED pgs=4282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcecc138940 0x7fcecc09b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcecc0a5bf0 0x7fcecc1379a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 --2- 172.21.15.50:0/1839770278 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcecc0a5100 0x7fcecc137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:18.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 >> 172.21.15.50:0/1839770278 conn(0x7fcecc0a3fc0 msgr2=0x7fcecc138ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:18.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 shutdown_connections 2023-10-30T12:19:18.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 -- 172.21.15.50:0/1839770278 wait complete. 2023-10-30T12:19:18.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:18.574+0000 7fced3a56640 1 librados: shutdown 2023-10-30T12:19:21.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.702+0000 7fe06a210640 10 monclient: get_monmap_and_config 2023-10-30T12:19:21.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.702+0000 7fe06a210640 10 monclient: build_initial_monmap 2023-10-30T12:19:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.702+0000 7fe06a210640 10 monclient: monmap: 2023-10-30T12:19:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:21.703945+0000 2023-10-30T12:19:21.705 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:21.703945+0000 2023-10-30T12:19:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:21.707 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:21.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:21.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.703+0000 7fe06a210640 1 Processor -- start 2023-10-30T12:19:21.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.703+0000 7fe06a210640 1 -- start start 2023-10-30T12:19:21.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.703+0000 7fe06a210640 10 monclient: init 2023-10-30T12:19:21.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:21.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T12:19:21.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe0640707d0 0x7fe06409e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:21.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient(hunting): picked mon.noname-c con 0x7fe0640707d0 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:21.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe064070bb0 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:21.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe06413aba0 0x7fe06413cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:21.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient(hunting): picked mon.noname-b con 0x7fe06413aba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:21.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06409ee40 con 0x7fe064070bb0 2023-10-30T12:19:21.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:21.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06409f4f0 con 0x7fe0640707d0 2023-10-30T12:19:21.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:21.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06409f7d0 con 0x7fe06413aba0 2023-10-30T12:19:21.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:21.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe06a210640 10 monclient(hunting): authenticate will time out at 2006.887315s 2023-10-30T12:19:21.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe062ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe064070bb0 0x7fe064071020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49884/0 (socket says 172.21.15.50:49884) 2023-10-30T12:19:21.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe062ffd640 1 -- 172.21.15.50:0/1034523790 learned_addr learned my addr 172.21.15.50:0/1034523790 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:21.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 1 --2- 172.21.15.50:0/1034523790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe06413aba0 0x7fe06413cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:21.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): get_auth_request con 0x7fe06413aba0 auth_method 0 2023-10-30T12:19:21.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:21.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:21.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:21.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:21.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:21.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:21.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 1 --2- 172.21.15.50:0/1034523790 >> v2:172.21.15.118:3300/0 conn(0x7fe0640707d0 0x7fe06409e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:21.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 10 monclient(hunting): get_auth_request con 0x7fe0640707d0 auth_method 0 2023-10-30T12:19:21.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:21.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:21.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:21.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:21.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:21.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe0637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:21.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient(hunting): handle_auth_done global_id 224273 payload 274 2023-10-30T12:19:21.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 1 -- 172.21.15.50:0/1034523790 >> v2:172.21.15.118:3300/0 conn(0x7fe0640707d0 msgr2=0x7fe06409e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:21.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 1 --2- 172.21.15.50:0/1034523790 >> v2:172.21.15.118:3300/0 conn(0x7fe0640707d0 0x7fe06409e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 1 -- 172.21.15.50:0/1034523790 >> v1:172.21.15.50:6789/0 conn(0x7fe064070bb0 legacy=0x7fe064071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient: _finish_hunting 0 2023-10-30T12:19:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 1 monclient: found mon.noname-b 2023-10-30T12:19:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.704+0000 7fe063fff640 1 -- 172.21.15.50:0/1034523790 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06409fab0 con 0x7fe06413aba0 2023-10-30T12:19:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe063fff640 1 --2- 172.21.15.50:0/1034523790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe06413aba0 0x7fe06413cf90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009d90 tx=0x7fe05c02bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58223750fc33d16a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 1 -- 172.21.15.50:0/1034523790 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c02cb30 con 0x7fe06413aba0 2023-10-30T12:19:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: dump: 2023-10-30T12:19:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:21.725 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: _finish_auth 0 2023-10-30T12:19:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: _check_auth_tickets 2023-10-30T12:19:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 1 -- 172.21.15.50:0/1034523790 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c02cc90 con 0x7fe06413aba0 2023-10-30T12:19:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe06a210640 5 monclient: authenticate success, global_id 224273 2023-10-30T12:19:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe06a210640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe06a210640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 1 -- 172.21.15.50:0/1034523790 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c034440 con 0x7fe06413aba0 2023-10-30T12:19:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe06a210640 4 set_mon_vals no callback set 2023-10-30T12:19:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.705+0000 7fe0627fc640 10 monclient: dump: 2023-10-30T12:19:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:21.730 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 10 monclient: shutdown 2023-10-30T12:19:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 -- 172.21.15.50:0/1034523790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe06413aba0 msgr2=0x7fe06413cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 --2- 172.21.15.50:0/1034523790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe06413aba0 0x7fe06413cf90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009d90 tx=0x7fe05c02bd80 comp rx=0 tx=0).stop 2023-10-30T12:19:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 -- 172.21.15.50:0/1034523790 shutdown_connections 2023-10-30T12:19:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 --2- 172.21.15.50:0/1034523790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe06413aba0 0x7fe06413cf90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 --2- 172.21.15.50:0/1034523790 >> v2:172.21.15.118:3300/0 conn(0x7fe0640707d0 0x7fe06409e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 -- 172.21.15.50:0/1034523790 >> 172.21.15.50:0/1034523790 conn(0x7fe0640a47f0 msgr2=0x7fe0640a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 -- 172.21.15.50:0/1034523790 shutdown_connections 2023-10-30T12:19:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 1 -- 172.21.15.50:0/1034523790 wait complete. 2023-10-30T12:19:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 10 monclient: build_initial_monmap 2023-10-30T12:19:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.706+0000 7fe06a210640 10 monclient: monmap: 2023-10-30T12:19:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:21.734 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 librados: starting msgr at 2023-10-30T12:19:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 librados: starting objecter 2023-10-30T12:19:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 Processor -- start 2023-10-30T12:19:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 -- start start 2023-10-30T12:19:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 librados: setting wanted keys 2023-10-30T12:19:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 librados: calling monclient init 2023-10-30T12:19:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 10 monclient: init 2023-10-30T12:19:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 0x7fe0641ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 10 monclient(hunting): picked mon.noname-a con 0x7fe0640707d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe064070bb0 0x7fe0641caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 10 monclient(hunting): picked mon.noname-c con 0x7fe064070bb0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe06413aba0 0x7fe06411c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 10 monclient(hunting): picked mon.noname-b con 0x7fe06413aba0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.707+0000 7fe06a210640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe064120170 con 0x7fe0640707d0 2023-10-30T12:19:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06409d6a0 con 0x7fe06413aba0 2023-10-30T12:19:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06409b590 con 0x7fe064070bb0 2023-10-30T12:19:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient(hunting): authenticate will time out at 2006.891029s 2023-10-30T12:19:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 0x7fe0641ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe064070bb0 0x7fe0641caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:21.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 0x7fe0641ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35858/0 (socket says 172.21.15.50:35858) 2023-10-30T12:19:21.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 1 -- 172.21.15.50:0/274330222 learned_addr learned my addr 172.21.15.50:0/274330222 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:21.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 10 monclient(hunting): get_auth_request con 0x7fe0640707d0 auth_method 0 2023-10-30T12:19:21.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:21.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:21.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:21.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): get_auth_request con 0x7fe064070bb0 auth_method 0 2023-10-30T12:19:21.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:21.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:21.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:21.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 1 -- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 msgr2=0x7fe0641ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:19:21.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:21.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 1 -- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 msgr2=0x7fe0641ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:19:21.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:21.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 0x7fe0641ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:21.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe06413aba0 0x7fe06411c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 10 monclient(hunting): get_auth_request con 0x7fe06413aba0 auth_method 0 2023-10-30T12:19:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe0637fe640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 0x7fe0641ca560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:21.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:21.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:21.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:21.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe063fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:21.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient(hunting): handle_auth_done global_id 224276 payload 824 2023-10-30T12:19:21.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 -- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe06413aba0 msgr2=0x7fe06411c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:21.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe06413aba0 0x7fe06411c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 -- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 msgr2=0x7fe0641ca560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:21.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 0x7fe0641ca560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:19:21.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 monclient: found mon.noname-c 2023-10-30T12:19:21.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:21.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:19:21.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 -- 172.21.15.50:0/274330222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c009a70 con 0x7fe064070bb0 2023-10-30T12:19:21.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe062ffd640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe064070bb0 0x7fe0641caaa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe05800c440 tx=0x7fe058001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:21.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 client.?.objecter ms_handle_connect 0x7fe064070bb0 2023-10-30T12:19:21.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:21.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 1 -- 172.21.15.50:0/274330222 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058005ce0 con 0x7fe064070bb0 2023-10-30T12:19:21.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:21.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:21.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: dump: 2023-10-30T12:19:21.746 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:21.746 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:21.747 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.747 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.747 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:21.747 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:21.747 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:21.748 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:21.748 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:21.748 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:21.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: _finish_auth 0 2023-10-30T12:19:21.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:21.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:21.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 1 -- 172.21.15.50:0/274330222 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe058005e40 con 0x7fe064070bb0 2023-10-30T12:19:21.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:21.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 1 -- 172.21.15.50:0/274330222 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580072d0 con 0x7fe064070bb0 2023-10-30T12:19:21.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 5 monclient: authenticate success, global_id 224276 2023-10-30T12:19:21.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient: _renew_subs 2023-10-30T12:19:21.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:21.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe064120a10 con 0x7fe064070bb0 2023-10-30T12:19:21.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:21.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:21.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe04bfff640 10 monclient: dump: 2023-10-30T12:19:21.750 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:21.751 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:21.751 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.751 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:21.751 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:21.751 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:21.752 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:21.752 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:21.752 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:21.752 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:21.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe061ffb640 4 set_mon_vals no callback set 2023-10-30T12:19:21.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 client.224276.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:21.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient: _renew_subs 2023-10-30T12:19:21.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:21.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06411e640 con 0x7fe064070bb0 2023-10-30T12:19:21.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.708+0000 7fe06a210640 1 librados: init done 2023-10-30T12:19:21.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.709+0000 7fe04bfff640 1 -- 172.21.15.50:0/274330222 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe0580074f0 con 0x7fe064070bb0 2023-10-30T12:19:21.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.709+0000 7fe06a210640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:21.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.709+0000 7fe06a210640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:21.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.709+0000 7fe06a210640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:21.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.709+0000 7fe04bfff640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0500392f0 0x7fe05003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:21.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.709+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0641205e0 con 0x7fe064070bb0 2023-10-30T12:19:21.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.709+0000 7fe04bfff640 1 -- 172.21.15.50:0/274330222 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(585..585 src has 490..585) v4 ==== 6269+0+0 (secure 0 0 0) 0x7fe05804a100 con 0x7fe064070bb0 2023-10-30T12:19:21.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe04bfff640 10 client.224276.objecter ms_dispatch 0x7fe064000b90 osd_map(585..585 src has 490..585) v4 2023-10-30T12:19:21.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe04bfff640 3 client.224276.objecter handle_osd_map got epochs [585,585] > 0 2023-10-30T12:19:21.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe04bfff640 3 client.224276.objecter handle_osd_map decoding full epoch 585 2023-10-30T12:19:21.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe04bfff640 20 client.224276.objecter dump_active .. 0 homeless 2023-10-30T12:19:21.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe0637fe640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0500392f0 0x7fe05003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:21.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe0637fe640 10 monclient: get_auth_request con 0x7fe0500392f0 auth_method 0 2023-10-30T12:19:21.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe04bfff640 10 client.224276.objecter ms_handle_connect 0x7fe0500392f0 2023-10-30T12:19:21.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.710+0000 7fe0637fe640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0500392f0 0x7fe05003b7b0 secure :-1 s=READY pgs=4283 cs=0 l=1 rev1=1 crypto rx=0x7fe0540095c0 tx=0x7fe054008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:21.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.714+0000 7fe04bfff640 1 -- 172.21.15.50:0/274330222 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe058019b20 con 0x7fe064070bb0 2023-10-30T12:19:21.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.714+0000 7fe04bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:21.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.714+0000 7fe04bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:21.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.964+0000 7fe06a210640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:21.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.964+0000 7fe06a210640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:21.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.964+0000 7fe06a210640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:21.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.964+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe06404e160 con 0x7fe064070bb0 2023-10-30T12:19:21.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.964+0000 7fe04bfff640 1 -- 172.21.15.50:0/274330222 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fe058017020 con 0x7fe064070bb0 2023-10-30T12:19:21.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.964+0000 7fe04bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:21.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.964+0000 7fe04bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:21.967 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:21.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 10 librados: watch_flush enter 2023-10-30T12:19:21.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 10 librados: watch_flush exit 2023-10-30T12:19:21.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 20 client.224276.objecter shutdown clearing up homeless session... 2023-10-30T12:19:21.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 10 client.224276.objecter successfully canceled tick 2023-10-30T12:19:21.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0500392f0 msgr2=0x7fe05003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:21.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0500392f0 0x7fe05003b7b0 secure :-1 s=READY pgs=4283 cs=0 l=1 rev1=1 crypto rx=0x7fe0540095c0 tx=0x7fe054008040 comp rx=0 tx=0).stop 2023-10-30T12:19:21.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 10 monclient: shutdown 2023-10-30T12:19:21.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:21.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe064070bb0 msgr2=0x7fe0641caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:21.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe064070bb0 0x7fe0641caaa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe05800c440 tx=0x7fe058001e70 comp rx=0 tx=0).stop 2023-10-30T12:19:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 shutdown_connections 2023-10-30T12:19:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe0500392f0 0x7fe05003b7b0 unknown :-1 s=CLOSED pgs=4283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe06413aba0 0x7fe06411c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe064070bb0 0x7fe0641caaa0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 --2- 172.21.15.50:0/274330222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0640707d0 0x7fe0641ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:21.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 >> 172.21.15.50:0/274330222 conn(0x7fe0640a47f0 msgr2=0x7fe0640a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 shutdown_connections 2023-10-30T12:19:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 -- 172.21.15.50:0/274330222 wait complete. 2023-10-30T12:19:21.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:21.966+0000 7fe06a210640 1 librados: shutdown 2023-10-30T12:19:24.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.095+0000 7f7a0a139640 10 monclient: get_monmap_and_config 2023-10-30T12:19:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.095+0000 7f7a0a139640 10 monclient: build_initial_monmap 2023-10-30T12:19:25.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.095+0000 7f7a0a139640 10 monclient: monmap: 2023-10-30T12:19:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:25.096809+0000 2023-10-30T12:19:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:25.096809+0000 2023-10-30T12:19:25.098 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:25.099 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:25.100 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.096+0000 7f7a0a139640 1 Processor -- start 2023-10-30T12:19:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.096+0000 7f7a0a139640 1 -- start start 2023-10-30T12:19:25.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.096+0000 7f7a0a139640 10 monclient: init 2023-10-30T12:19:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.096+0000 7f7a0a139640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-10-30T12:19:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7a0409f5b0 0x7f7a04099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient(hunting): picked mon.noname-c con 0x7f7a0409f5b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 0x7f7a04096490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:25.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient(hunting): picked mon.noname-a con 0x7f7a04096080 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:25.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a040969d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:25.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a04098810 con 0x7f7a040969d0 2023-10-30T12:19:25.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:25.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0406aef0 con 0x7f7a04096080 2023-10-30T12:19:25.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0409a1a0 con 0x7f7a0409f5b0 2023-10-30T12:19:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a0a139640 10 monclient(hunting): authenticate will time out at 2010.280126s 2023-10-30T12:19:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 0x7f7a04096490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:25.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 0x7f7a04096490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35876/0 (socket says 172.21.15.50:35876) 2023-10-30T12:19:25.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 1 -- 172.21.15.50:0/350275745 learned_addr learned my addr 172.21.15.50:0/350275745 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:25.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 10 monclient(hunting): get_auth_request con 0x7f7a04096080 auth_method 0 2023-10-30T12:19:25.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:25.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:25.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:25.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a03fff640 1 --1- 172.21.15.50:0/350275745 >> v1:172.21.15.50:6789/0 conn(0x7f7a040969d0 0x7f7a0409cb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49890/0 (socket says 172.21.15.50:49890) 2023-10-30T12:19:25.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 1 -- 172.21.15.50:0/350275745 >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 msgr2=0x7f7a04096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:19:25.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 1 -- 172.21.15.50:0/350275745 >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 msgr2=0x7f7a04096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:25.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 1 --2- 172.21.15.50:0/350275745 >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 0x7f7a04096490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:25.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 1 --2- 172.21.15.50:0/350275745 >> v2:172.21.15.118:3300/0 conn(0x7f7a0409f5b0 0x7f7a04099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:25.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a02ffd640 1 --2- 172.21.15.50:0/350275745 >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 0x7f7a04096490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:25.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): get_auth_request con 0x7f7a0409f5b0 auth_method 0 2023-10-30T12:19:25.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:25.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:25.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:25.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:25.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:25.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:25.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient(hunting): handle_auth_done global_id 204550 payload 274 2023-10-30T12:19:25.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 1 -- 172.21.15.50:0/350275745 >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 msgr2=0x7f7a04096490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:25.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 1 --2- 172.21.15.50:0/350275745 >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 0x7f7a04096490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 1 -- 172.21.15.50:0/350275745 >> v1:172.21.15.50:6789/0 conn(0x7f7a040969d0 legacy=0x7f7a0409cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient: _finish_hunting 0 2023-10-30T12:19:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 1 monclient: found mon.noname-c 2023-10-30T12:19:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:19:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.097+0000 7f7a037fe640 1 -- 172.21.15.50:0/350275745 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a04096e60 con 0x7f7a0409f5b0 2023-10-30T12:19:25.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a037fe640 1 --2- 172.21.15.50:0/350275745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a0409f5b0 0x7f7a04099b80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009790 tx=0x7f79f402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aef5d58cdf59a26a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:25.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 1 -- 172.21.15.50:0/350275745 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f402f050 con 0x7f7a0409f5b0 2023-10-30T12:19:25.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:25.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:25.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: dump: 2023-10-30T12:19:25.117 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:25.117 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:25.117 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.117 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.117 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:25.117 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:25.118 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:25.118 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:25.118 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:25.118 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:25.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: _finish_auth 0 2023-10-30T12:19:25.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: _check_auth_tickets 2023-10-30T12:19:25.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:25.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 1 -- 172.21.15.50:0/350275745 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f4031700 con 0x7f7a0409f5b0 2023-10-30T12:19:25.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a0a139640 5 monclient: authenticate success, global_id 204550 2023-10-30T12:19:25.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a0a139640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:25.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:25.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a0a139640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:25.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 1 -- 172.21.15.50:0/350275745 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4031a90 con 0x7f7a0409f5b0 2023-10-30T12:19:25.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:25.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a0a139640 4 set_mon_vals no callback set 2023-10-30T12:19:25.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:25.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.098+0000 7f7a027fc640 10 monclient: dump: 2023-10-30T12:19:25.121 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:25.121 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:25.121 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:25.123 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 10 monclient: shutdown 2023-10-30T12:19:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 -- 172.21.15.50:0/350275745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a0409f5b0 msgr2=0x7f7a04099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 --2- 172.21.15.50:0/350275745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a0409f5b0 0x7f7a04099b80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009790 tx=0x7f79f402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 -- 172.21.15.50:0/350275745 shutdown_connections 2023-10-30T12:19:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 --2- 172.21.15.50:0/350275745 >> v2:172.21.15.50:3300/0 conn(0x7f7a04096080 0x7f7a04096490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 --2- 172.21.15.50:0/350275745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a0409f5b0 0x7f7a04099b80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 -- 172.21.15.50:0/350275745 >> 172.21.15.50:0/350275745 conn(0x7f7a040a05c0 msgr2=0x7f7a0409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 -- 172.21.15.50:0/350275745 shutdown_connections 2023-10-30T12:19:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 1 -- 172.21.15.50:0/350275745 wait complete. 2023-10-30T12:19:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 10 monclient: build_initial_monmap 2023-10-30T12:19:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.099+0000 7f7a0a139640 10 monclient: monmap: 2023-10-30T12:19:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:25.126 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:25.126 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:25.126 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:25.126 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:25.126 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:25.127 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 librados: starting msgr at 2023-10-30T12:19:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 librados: starting objecter 2023-10-30T12:19:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 Processor -- start 2023-10-30T12:19:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 -- start start 2023-10-30T12:19:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 librados: setting wanted keys 2023-10-30T12:19:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 librados: calling monclient init 2023-10-30T12:19:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient: init 2023-10-30T12:19:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:19:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a04096080 0x7f7a041c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): picked mon.noname-b con 0x7f7a04096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a040969d0 0x7f7a041b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): picked mon.noname-c con 0x7f7a040969d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 0x7f7a041b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:25.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): picked mon.noname-a con 0x7f7a0409f5b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:25.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:25.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a04072010 con 0x7f7a0409f5b0 2023-10-30T12:19:25.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:25.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a041aeae0 con 0x7f7a04096080 2023-10-30T12:19:25.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:25.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a041ae800 con 0x7f7a040969d0 2023-10-30T12:19:25.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:25.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.100+0000 7f7a0a139640 10 monclient(hunting): authenticate will time out at 2010.283996s 2023-10-30T12:19:25.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 0x7f7a041b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:25.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a040969d0 0x7f7a041b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:25.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 0x7f7a041b5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35886/0 (socket says 172.21.15.50:35886) 2023-10-30T12:19:25.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 1 -- 172.21.15.50:0/3391640743 learned_addr learned my addr 172.21.15.50:0/3391640743 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:25.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 10 monclient(hunting): get_auth_request con 0x7f7a0409f5b0 auth_method 0 2023-10-30T12:19:25.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): get_auth_request con 0x7f7a040969d0 auth_method 0 2023-10-30T12:19:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:25.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 1 -- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 msgr2=0x7f7a041b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:19:25.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 1 -- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 msgr2=0x7f7a041b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:19:25.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 0x7f7a041b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:25.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a04096080 0x7f7a041c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:25.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:25.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:25.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a03fff640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 0x7f7a041b5b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:25.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:25.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 10 monclient(hunting): get_auth_request con 0x7f7a04096080 auth_method 0 2023-10-30T12:19:25.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient(hunting): handle_auth_done global_id 224279 payload 824 2023-10-30T12:19:25.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 -- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a04096080 msgr2=0x7f7a041c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:25.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a04096080 0x7f7a041c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 -- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 msgr2=0x7f7a041b5b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:25.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 0x7f7a041b5b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:19:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 monclient: found mon.noname-c 2023-10-30T12:19:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:19:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 -- 172.21.15.50:0/3391640743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f40093a0 con 0x7f7a040969d0 2023-10-30T12:19:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a02ffd640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a040969d0 0x7f7a041b5620 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f79f800e7e0 tx=0x7f79f800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 client.?.objecter ms_handle_connect 0x7f7a040969d0 2023-10-30T12:19:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 1 -- 172.21.15.50:0/3391640743 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8004020 con 0x7f7a040969d0 2023-10-30T12:19:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: dump: 2023-10-30T12:19:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:25.141 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:25.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: _finish_auth 0 2023-10-30T12:19:25.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:25.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:25.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 1 -- 172.21.15.50:0/3391640743 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f8004180 con 0x7f7a040969d0 2023-10-30T12:19:25.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a0a139640 5 monclient: authenticate success, global_id 224279 2023-10-30T12:19:25.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a0a139640 10 monclient: _renew_subs 2023-10-30T12:19:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a0a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a04071170 con 0x7f7a040969d0 2023-10-30T12:19:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 1 -- 172.21.15.50:0/3391640743 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f800c6d0 con 0x7f7a040969d0 2023-10-30T12:19:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f79ebfff640 10 monclient: dump: 2023-10-30T12:19:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:25.145 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:25.145 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:25.145 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:25.145 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:25.145 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:25.146 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a01ffb640 4 set_mon_vals no callback set 2023-10-30T12:19:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.101+0000 7f7a0a139640 10 client.224279.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 10 monclient: _renew_subs 2023-10-30T12:19:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a041c88d0 con 0x7f7a040969d0 2023-10-30T12:19:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 1 librados: init done 2023-10-30T12:19:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f79ebfff640 1 -- 172.21.15.50:0/3391640743 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f79f800c830 con 0x7f7a040969d0 2023-10-30T12:19:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.102+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0404e160 con 0x7f7a040969d0 2023-10-30T12:19:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f79ebfff640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f79f00392f0 0x7f79f003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f79ebfff640 1 -- 172.21.15.50:0/3391640743 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(589..589 src has 490..589) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f79f804a8b0 con 0x7f7a040969d0 2023-10-30T12:19:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f79ebfff640 10 client.224279.objecter ms_dispatch 0x7f7a04000b90 osd_map(589..589 src has 490..589) v4 2023-10-30T12:19:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f79ebfff640 3 client.224279.objecter handle_osd_map got epochs [589,589] > 0 2023-10-30T12:19:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f79ebfff640 3 client.224279.objecter handle_osd_map decoding full epoch 589 2023-10-30T12:19:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f79ebfff640 20 client.224279.objecter dump_active .. 0 homeless 2023-10-30T12:19:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f7a037fe640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f79f00392f0 0x7f79f003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f7a037fe640 10 monclient: get_auth_request con 0x7f79f00392f0 auth_method 0 2023-10-30T12:19:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f7a037fe640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f79f00392f0 0x7f79f003b7b0 secure :-1 s=READY pgs=4284 cs=0 l=1 rev1=1 crypto rx=0x7f79f4035000 tx=0x7f79f4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.103+0000 7f79ebfff640 10 client.224279.objecter ms_handle_connect 0x7f79f00392f0 2023-10-30T12:19:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.107+0000 7f79ebfff640 1 -- 172.21.15.50:0/3391640743 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f79f8009390 con 0x7f7a040969d0 2023-10-30T12:19:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.107+0000 7f79ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.107+0000 7f79ebfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.355+0000 7f7a0a139640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.355+0000 7f7a0a139640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.355+0000 7f7a0a139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.355+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7a040745c0 con 0x7f7a040969d0 2023-10-30T12:19:25.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.355+0000 7f79ebfff640 1 -- 172.21.15.50:0/3391640743 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f79f8012370 con 0x7f7a040969d0 2023-10-30T12:19:25.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.355+0000 7f79ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:25.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.355+0000 7f79ebfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:25.358 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 10 librados: watch_flush enter 2023-10-30T12:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 10 librados: watch_flush exit 2023-10-30T12:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 20 client.224279.objecter shutdown clearing up homeless session... 2023-10-30T12:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 10 client.224279.objecter successfully canceled tick 2023-10-30T12:19:25.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f79f00392f0 msgr2=0x7f79f003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f79f00392f0 0x7f79f003b7b0 secure :-1 s=READY pgs=4284 cs=0 l=1 rev1=1 crypto rx=0x7f79f4035000 tx=0x7f79f4008030 comp rx=0 tx=0).stop 2023-10-30T12:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 10 monclient: shutdown 2023-10-30T12:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a040969d0 msgr2=0x7f7a041b5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:25.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a040969d0 0x7f7a041b5620 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f79f800e7e0 tx=0x7f79f800eca0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 shutdown_connections 2023-10-30T12:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f79f00392f0 0x7f79f003b7b0 unknown :-1 s=CLOSED pgs=4284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a0409f5b0 0x7f7a041b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a040969d0 0x7f7a041b5620 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 --2- 172.21.15.50:0/3391640743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a04096080 0x7f7a041c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:25.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 >> 172.21.15.50:0/3391640743 conn(0x7f7a040a05c0 msgr2=0x7f7a041b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:25.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 shutdown_connections 2023-10-30T12:19:25.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 -- 172.21.15.50:0/3391640743 wait complete. 2023-10-30T12:19:25.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:25.357+0000 7f7a0a139640 1 librados: shutdown 2023-10-30T12:19:28.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.486+0000 7f8f65fd0640 10 monclient: get_monmap_and_config 2023-10-30T12:19:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.486+0000 7f8f65fd0640 10 monclient: build_initial_monmap 2023-10-30T12:19:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.486+0000 7f8f65fd0640 10 monclient: monmap: 2023-10-30T12:19:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:28.486351+0000 2023-10-30T12:19:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:28.486351+0000 2023-10-30T12:19:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:28.489 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:28.489 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:28.489 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:28.489 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 1 Processor -- start 2023-10-30T12:19:28.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 1 -- start start 2023-10-30T12:19:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient: init 2023-10-30T12:19:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:19:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f60131b20 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60071bc0 0x7f8f60071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f8f60071bc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f60136e40 0x7f8f60139230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f8f60136e40 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6009a9b0 con 0x7f8f60131b20 2023-10-30T12:19:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f60131f00 con 0x7f8f60136e40 2023-10-30T12:19:28.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60132980 con 0x7f8f60071bc0 2023-10-30T12:19:28.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:28.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f65fd0640 10 monclient(hunting): authenticate will time out at 2013.669550s 2023-10-30T12:19:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f60136e40 0x7f8f60139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60071bc0 0x7f8f60071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f60131b20 0x7f8f60132270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40388/0 (socket says 172.21.15.50:40388) 2023-10-30T12:19:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5f7fe640 1 -- 172.21.15.50:0/2372965977 learned_addr learned my addr 172.21.15.50:0/2372965977 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 10 monclient(hunting): get_auth_request con 0x7f8f60136e40 auth_method 0 2023-10-30T12:19:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 10 monclient(hunting): get_auth_request con 0x7f8f60071bc0 auth_method 0 2023-10-30T12:19:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.487+0000 7f8f5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 10 monclient(hunting): handle_auth_done global_id 224282 payload 274 2023-10-30T12:19:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5e7fc640 1 -- 172.21.15.50:0/2372965977 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176957544 0 0) 0x7f8f6009a9b0 con 0x7f8f60131b20 2023-10-30T12:19:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 1 -- 172.21.15.50:0/2372965977 >> v2:172.21.15.50:3300/0 conn(0x7f8f60136e40 msgr2=0x7f8f60139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 1 --2- 172.21.15.50:0/2372965977 >> v2:172.21.15.50:3300/0 conn(0x7f8f60136e40 0x7f8f60139230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 1 -- 172.21.15.50:0/2372965977 >> v1:172.21.15.50:6789/0 conn(0x7f8f60131b20 legacy=0x7f8f60132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 10 monclient: _finish_hunting 0 2023-10-30T12:19:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 1 monclient: found mon.noname-b 2023-10-30T12:19:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:28.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:28.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 1 -- 172.21.15.50:0/2372965977 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6009d920 con 0x7f8f60071bc0 2023-10-30T12:19:28.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:19:28.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5ffff640 1 --2- 172.21.15.50:0/2372965977 >> v2:172.21.15.50:3300/0 conn(0x7f8f60136e40 0x7f8f60139230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:28.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5effd640 1 --2- 172.21.15.50:0/2372965977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60071bc0 0x7f8f60071fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f540097a0 tx=0x7f8f5402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=616595d32309f424 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.488+0000 7f8f5e7fc640 1 -- 172.21.15.50:0/2372965977 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540316f0 con 0x7f8f60071bc0 2023-10-30T12:19:28.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:28.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: dump: 2023-10-30T12:19:28.507 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.509 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.509 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.509 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:19:28.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 5 monclient: authenticate success, global_id 224282 2023-10-30T12:19:28.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:28.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 1 -- 172.21.15.50:0/2372965977 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54031850 con 0x7f8f60071bc0 2023-10-30T12:19:28.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 4 set_mon_vals no callback set 2023-10-30T12:19:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 1 -- 172.21.15.50:0/2372965977 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54031be0 con 0x7f8f60071bc0 2023-10-30T12:19:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f5e7fc640 10 monclient: dump: 2023-10-30T12:19:28.512 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.512 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.512 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.512 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.512 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.513 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.513 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.513 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.513 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.513 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 10 monclient: shutdown 2023-10-30T12:19:28.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:28.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 -- 172.21.15.50:0/2372965977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60071bc0 msgr2=0x7f8f60071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/2372965977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60071bc0 0x7f8f60071fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f540097a0 tx=0x7f8f5402b790 comp rx=0 tx=0).stop 2023-10-30T12:19:28.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 -- 172.21.15.50:0/2372965977 shutdown_connections 2023-10-30T12:19:28.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/2372965977 >> v2:172.21.15.50:3300/0 conn(0x7f8f60136e40 0x7f8f60139230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/2372965977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60071bc0 0x7f8f60071fc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 -- 172.21.15.50:0/2372965977 >> 172.21.15.50:0/2372965977 conn(0x7f8f600a05c0 msgr2=0x7f8f6009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:28.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 -- 172.21.15.50:0/2372965977 shutdown_connections 2023-10-30T12:19:28.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.489+0000 7f8f65fd0640 1 -- 172.21.15.50:0/2372965977 wait complete. 2023-10-30T12:19:28.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 10 monclient: build_initial_monmap 2023-10-30T12:19:28.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 10 monclient: monmap: 2023-10-30T12:19:28.516 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:28.516 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:28.516 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:28.516 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:28.517 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:28.517 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.517 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:28.517 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:28.517 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:28.518 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 1 librados: starting msgr at 2023-10-30T12:19:28.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 1 librados: starting objecter 2023-10-30T12:19:28.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 1 Processor -- start 2023-10-30T12:19:28.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 1 -- start start 2023-10-30T12:19:28.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 1 librados: setting wanted keys 2023-10-30T12:19:28.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 1 librados: calling monclient init 2023-10-30T12:19:28.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.490+0000 7f8f65fd0640 10 monclient: init 2023-10-30T12:19:28.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:28.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:28.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f60071bc0 0x7f8f60070840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f8f60071bc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:28.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60131b20 0x7f8f60070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f8f60131b20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:28.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f60136e40 0x7f8f600712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f8f60136e40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:28.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600993c0 con 0x7f8f60071bc0 2023-10-30T12:19:28.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60136270 con 0x7f8f60136e40 2023-10-30T12:19:28.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6012a410 con 0x7f8f60131b20 2023-10-30T12:19:28.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:28.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f65fd0640 10 monclient(hunting): authenticate will time out at 2013.673253s 2023-10-30T12:19:28.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60131b20 0x7f8f60070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f60071bc0 0x7f8f60070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60131b20 0x7f8f60070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33728/0 (socket says 172.21.15.50:33728) 2023-10-30T12:19:28.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 1 -- 172.21.15.50:0/734698912 learned_addr learned my addr 172.21.15.50:0/734698912 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:28.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 10 monclient(hunting): get_auth_request con 0x7f8f60131b20 auth_method 0 2023-10-30T12:19:28.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): get_auth_request con 0x7f8f60071bc0 auth_method 0 2023-10-30T12:19:28.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5ffff640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f60136e40 0x7f8f600712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient(hunting): handle_auth_done global_id 244104 payload 824 2023-10-30T12:19:28.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 -- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60131b20 msgr2=0x7f8f60070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60131b20 0x7f8f60070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 -- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f60136e40 msgr2=0x7f8f600712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f60136e40 0x7f8f600712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:19:28.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 monclient: found mon.noname-a 2023-10-30T12:19:28.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:28.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:28.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 -- 172.21.15.50:0/734698912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f58009a70 con 0x7f8f60071bc0 2023-10-30T12:19:28.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5ffff640 10 monclient: get_auth_request con 0x7f8f60136e40 auth_method 0 2023-10-30T12:19:28.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5effd640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60131b20 0x7f8f60070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:28.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5ffff640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f60136e40 0x7f8f600712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:19:28.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f5f7fe640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f60071bc0 0x7f8f60070840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8f50001360 tx=0x7f8f5000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f47fff640 10 client.?.objecter ms_handle_connect 0x7f8f60071bc0 2023-10-30T12:19:28.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.491+0000 7f8f47fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:28.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 1 -- 172.21.15.50:0/734698912 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50013070 con 0x7f8f60071bc0 2023-10-30T12:19:28.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:28.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: dump: 2023-10-30T12:19:28.530 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.530 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.530 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.530 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.531 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.531 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.531 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.531 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.531 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.532 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: _finish_auth 0 2023-10-30T12:19:28.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 1 -- 172.21.15.50:0/734698912 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5000e2e0 con 0x7f8f60071bc0 2023-10-30T12:19:28.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 5 monclient: authenticate success, global_id 244104 2023-10-30T12:19:28.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:28.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 1 -- 172.21.15.50:0/734698912 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50007270 con 0x7f8f60071bc0 2023-10-30T12:19:28.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:28.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 10 monclient: dump: 2023-10-30T12:19:28.534 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.534 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.534 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.534 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.534 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.535 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.535 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.535 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.535 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.535 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 10 monclient: _renew_subs 2023-10-30T12:19:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f5dffb640 4 set_mon_vals no callback set 2023-10-30T12:19:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f54009400 con 0x7f8f60071bc0 2023-10-30T12:19:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 10 client.244104.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 10 monclient: _renew_subs 2023-10-30T12:19:28.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:28.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f601288e0 con 0x7f8f60071bc0 2023-10-30T12:19:28.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f65fd0640 1 librados: init done 2023-10-30T12:19:28.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.492+0000 7f8f47fff640 1 -- 172.21.15.50:0/734698912 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8f500073d0 con 0x7f8f60071bc0 2023-10-30T12:19:28.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f65fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:28.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f47fff640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f4c0392f0 0x7f8f4c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f65fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:28.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f65fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:28.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6004e160 con 0x7f8f60071bc0 2023-10-30T12:19:28.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f47fff640 1 -- 172.21.15.50:0/734698912 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(592..592 src has 490..592) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f8f5004e550 con 0x7f8f60071bc0 2023-10-30T12:19:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f47fff640 10 client.244104.objecter ms_dispatch 0x7f8f60000b90 osd_map(592..592 src has 490..592) v4 2023-10-30T12:19:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f47fff640 3 client.244104.objecter handle_osd_map got epochs [592,592] > 0 2023-10-30T12:19:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f47fff640 3 client.244104.objecter handle_osd_map decoding full epoch 592 2023-10-30T12:19:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f47fff640 20 client.244104.objecter dump_active .. 0 homeless 2023-10-30T12:19:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f5effd640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f4c0392f0 0x7f8f4c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.493+0000 7f8f5effd640 10 monclient: get_auth_request con 0x7f8f4c0392f0 auth_method 0 2023-10-30T12:19:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.494+0000 7f8f5effd640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f4c0392f0 0x7f8f4c03b7b0 secure :-1 s=READY pgs=4285 cs=0 l=1 rev1=1 crypto rx=0x7f8f5402b7d0 tx=0x7f8f54008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.494+0000 7f8f47fff640 10 client.244104.objecter ms_handle_connect 0x7f8f4c0392f0 2023-10-30T12:19:28.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.497+0000 7f8f47fff640 1 -- 172.21.15.50:0/734698912 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8f50010630 con 0x7f8f60071bc0 2023-10-30T12:19:28.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.497+0000 7f8f47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:28.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.497+0000 7f8f47fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:28.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.746+0000 7f8f65fd0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:28.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.746+0000 7f8f65fd0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:28.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.746+0000 7f8f65fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:28.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.746+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8f600988b0 con 0x7f8f60071bc0 2023-10-30T12:19:28.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.746+0000 7f8f47fff640 1 -- 172.21.15.50:0/734698912 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f8f5003f020 con 0x7f8f60071bc0 2023-10-30T12:19:28.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.746+0000 7f8f47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:28.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.746+0000 7f8f47fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:28.748 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":198,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:28.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 10 librados: watch_flush enter 2023-10-30T12:19:28.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 10 librados: watch_flush exit 2023-10-30T12:19:28.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 20 client.244104.objecter shutdown clearing up homeless session... 2023-10-30T12:19:28.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 10 client.244104.objecter successfully canceled tick 2023-10-30T12:19:28.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f4c0392f0 msgr2=0x7f8f4c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f4c0392f0 0x7f8f4c03b7b0 secure :-1 s=READY pgs=4285 cs=0 l=1 rev1=1 crypto rx=0x7f8f5402b7d0 tx=0x7f8f54008030 comp rx=0 tx=0).stop 2023-10-30T12:19:28.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 10 monclient: shutdown 2023-10-30T12:19:28.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:28.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f60071bc0 msgr2=0x7f8f60070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f60071bc0 0x7f8f60070840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8f50001360 tx=0x7f8f5000eef0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 shutdown_connections 2023-10-30T12:19:28.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f4c0392f0 0x7f8f4c03b7b0 unknown :-1 s=CLOSED pgs=4285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f60136e40 0x7f8f600712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f60131b20 0x7f8f60070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 --2- 172.21.15.50:0/734698912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f60071bc0 0x7f8f60070840 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 >> 172.21.15.50:0/734698912 conn(0x7f8f600a05c0 msgr2=0x7f8f60134e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:28.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 shutdown_connections 2023-10-30T12:19:28.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 -- 172.21.15.50:0/734698912 wait complete. 2023-10-30T12:19:28.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.748+0000 7f8f65fd0640 1 librados: shutdown 2023-10-30T12:19:28.759 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:19:28.759 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:19:28.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:19:28.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.865+0000 7f0223700640 10 monclient: get_monmap_and_config 2023-10-30T12:19:28.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.865+0000 7f0223700640 10 monclient: build_initial_monmap 2023-10-30T12:19:28.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.865+0000 7f0223700640 10 monclient: monmap: 2023-10-30T12:19:28.866 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:28.867 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.867 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:28.865734+0000 2023-10-30T12:19:28.867 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:28.865734+0000 2023-10-30T12:19:28.867 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:28.867 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.868 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:28.868 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:28.868 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:28.868 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:28.868 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:28.869 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 1 Processor -- start 2023-10-30T12:19:28.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 1 -- start start 2023-10-30T12:19:28.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient: init 2023-10-30T12:19:28.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:28.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-10-30T12:19:28.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f021c09bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:28.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f021c09d3f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:28.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09c660 0x7f021c13b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient(hunting): picked mon.noname-b con 0x7f021c09c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:28.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c098170 con 0x7f021c09d3f0 2023-10-30T12:19:28.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c09cf60 con 0x7f021c09bb70 2023-10-30T12:19:28.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c097b20 con 0x7f021c09c660 2023-10-30T12:19:28.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:28.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.866+0000 7f0223700640 10 monclient(hunting): authenticate will time out at 2014.048915s 2023-10-30T12:19:28.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09c660 0x7f021c13b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0220c9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f021c09d3f0 0x7f021c137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40398/0 (socket says 172.21.15.50:40398) 2023-10-30T12:19:28.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0220c9b640 1 -- 172.21.15.50:0/3094642472 learned_addr learned my addr 172.21.15.50:0/3094642472 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:28.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09c660 0x7f021c13b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33736/0 (socket says 172.21.15.50:33736) 2023-10-30T12:19:28.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): get_auth_request con 0x7f021c09c660 auth_method 0 2023-10-30T12:19:28.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient(hunting): handle_auth_done global_id 224288 payload 274 2023-10-30T12:19:28.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 1 -- 172.21.15.50:0/3094642472 >> v1:172.21.15.118:6789/0 conn(0x7f021c09bb70 legacy=0x7f021c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 1 -- 172.21.15.50:0/3094642472 >> v1:172.21.15.50:6789/0 conn(0x7f021c09d3f0 legacy=0x7f021c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient: _finish_hunting 0 2023-10-30T12:19:28.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 1 monclient: found mon.noname-b 2023-10-30T12:19:28.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:28.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:28.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f0221c9d640 1 -- 172.21.15.50:0/3094642472 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c097e00 con 0x7f021c09c660 2023-10-30T12:19:28.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f020ffff640 1 -- 172.21.15.50:0/3094642472 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041378902 0 0) 0x7f021c098170 con 0x7f021c09d3f0 2023-10-30T12:19:28.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.867+0000 7f020ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:19:28.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0221c9d640 1 --2- 172.21.15.50:0/3094642472 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09c660 0x7f021c13b090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0214009dc0 tx=0x7f021402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=609f6bbe02390a47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 1 -- 172.21.15.50:0/3094642472 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021402cbb0 con 0x7f021c09c660 2023-10-30T12:19:28.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:28.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: dump: 2023-10-30T12:19:28.889 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.889 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.889 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.889 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.889 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.890 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.890 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.890 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.890 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.890 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: _finish_auth 0 2023-10-30T12:19:28.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 5 monclient: authenticate success, global_id 224288 2023-10-30T12:19:28.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 1 -- 172.21.15.50:0/3094642472 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021402cd10 con 0x7f021c09c660 2023-10-30T12:19:28.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:28.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:28.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:28.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 4 set_mon_vals no callback set 2023-10-30T12:19:28.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 1 -- 172.21.15.50:0/3094642472 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214034660 con 0x7f021c09c660 2023-10-30T12:19:28.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:28.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f020ffff640 10 monclient: dump: 2023-10-30T12:19:28.893 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.894 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.894 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.894 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.894 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.894 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.895 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.895 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.895 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.895 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 10 monclient: shutdown 2023-10-30T12:19:28.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:28.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 1 -- 172.21.15.50:0/3094642472 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09c660 msgr2=0x7f021c13b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 1 --2- 172.21.15.50:0/3094642472 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09c660 0x7f021c13b090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0214009dc0 tx=0x7f021402be00 comp rx=0 tx=0).stop 2023-10-30T12:19:28.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 1 -- 172.21.15.50:0/3094642472 shutdown_connections 2023-10-30T12:19:28.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 1 --2- 172.21.15.50:0/3094642472 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09c660 0x7f021c13b090 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 1 -- 172.21.15.50:0/3094642472 >> 172.21.15.50:0/3094642472 conn(0x7f021c0a3980 msgr2=0x7f021c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:28.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 1 -- 172.21.15.50:0/3094642472 shutdown_connections 2023-10-30T12:19:28.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.868+0000 7f0223700640 1 -- 172.21.15.50:0/3094642472 wait complete. 2023-10-30T12:19:28.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 10 monclient: build_initial_monmap 2023-10-30T12:19:28.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 10 monclient: monmap: 2023-10-30T12:19:28.898 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:28.898 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:28.899 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:28.899 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:28.899 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:28.899 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.899 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:28.900 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:28.900 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:28.900 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 1 librados: starting msgr at 2023-10-30T12:19:28.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 1 librados: starting objecter 2023-10-30T12:19:28.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 1 Processor -- start 2023-10-30T12:19:28.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 1 -- start start 2023-10-30T12:19:28.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 1 librados: setting wanted keys 2023-10-30T12:19:28.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 1 librados: calling monclient init 2023-10-30T12:19:28.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.869+0000 7f0223700640 10 monclient: init 2023-10-30T12:19:28.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:28.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:19:28.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c09bb70 0x7f021c1c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): picked mon.noname-a con 0x7f021c09bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:28.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f021c09c660 0x7f021c094510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): picked mon.noname-b con 0x7f021c09c660 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:28.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09d3f0 0x7f021c094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): picked mon.noname-c con 0x7f021c09d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:28.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c08d480 con 0x7f021c09bb70 2023-10-30T12:19:28.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c08f710 con 0x7f021c09c660 2023-10-30T12:19:28.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c08f9f0 con 0x7f021c09d3f0 2023-10-30T12:19:28.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:28.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0223700640 10 monclient(hunting): authenticate will time out at 2014.052278s 2023-10-30T12:19:28.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c09bb70 0x7f021c1c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09d3f0 0x7f021c094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c09bb70 0x7f021c1c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48646/0 (socket says 172.21.15.50:48646) 2023-10-30T12:19:28.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 1 -- 172.21.15.50:0/1192969223 learned_addr learned my addr 172.21.15.50:0/1192969223 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:28.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09d3f0 0x7f021c094a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33748/0 (socket says 172.21.15.50:33748) 2023-10-30T12:19:28.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 10 monclient(hunting): get_auth_request con 0x7f021c09bb70 auth_method 0 2023-10-30T12:19:28.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): get_auth_request con 0x7f021c09d3f0 auth_method 0 2023-10-30T12:19:28.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0220c9b640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f021c09c660 0x7f021c094510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0220c9b640 10 monclient(hunting): get_auth_request con 0x7f021c09c660 auth_method 0 2023-10-30T12:19:28.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0220c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0220c9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0220c9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient(hunting): handle_auth_done global_id 224291 payload 824 2023-10-30T12:19:28.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f021c09c660 msgr2=0x7f021c094510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f021c09c660 0x7f021c094510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c09bb70 msgr2=0x7f021c1c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c09bb70 0x7f021c1c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient: _finish_hunting 0 2023-10-30T12:19:28.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 monclient: found mon.noname-c 2023-10-30T12:19:28.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:28.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:19:28.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 -- 172.21.15.50:0/1192969223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0214009a70 con 0x7f021c09d3f0 2023-10-30T12:19:28.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f022149c640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c09bb70 0x7f021c1c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:28.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f0221c9d640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09d3f0 0x7f021c094a50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f021c09fa70 tx=0x7f0214004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f020dffb640 10 client.?.objecter ms_handle_connect 0x7f021c09d3f0 2023-10-30T12:19:28.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.870+0000 7f020dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:28.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214040450 con 0x7f021c09d3f0 2023-10-30T12:19:28.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:28.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: dump: 2023-10-30T12:19:28.914 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.914 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.914 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.914 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.915 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.915 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.915 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.915 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.916 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.916 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: _finish_auth 0 2023-10-30T12:19:28.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:19:28.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:28.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02140405b0 con 0x7f021c09d3f0 2023-10-30T12:19:28.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 5 monclient: authenticate success, global_id 224291 2023-10-30T12:19:28.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 10 monclient: _renew_subs 2023-10-30T12:19:28.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:28.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c0976e0 con 0x7f021c09d3f0 2023-10-30T12:19:28.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:28.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214040920 con 0x7f021c09d3f0 2023-10-30T12:19:28.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:28.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020dffb640 10 monclient: dump: 2023-10-30T12:19:28.919 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.919 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.919 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.919 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.920 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.920 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.920 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.920 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.920 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.921 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 10 client.224291.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:28.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 10 monclient: _renew_subs 2023-10-30T12:19:28.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f020f7fe640 4 set_mon_vals no callback set 2023-10-30T12:19:28.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:28.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c08dc00 con 0x7f021c09d3f0 2023-10-30T12:19:28.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.871+0000 7f0223700640 1 librados: init done 2023-10-30T12:19:28.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f0223700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:28.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0214040a80 con 0x7f021c09d3f0 2023-10-30T12:19:28.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f0223700640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c49d0 0x7f021c1c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f0223700640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c073ff0 con 0x7f021c1c49d0 2023-10-30T12:19:28.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c49d0 0x7f021c1c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient: get_auth_request con 0x7f021c1c49d0 auth_method 0 2023-10-30T12:19:28.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02000392f0 0x7f020003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(593..593 src has 490..593) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f021406db80 con 0x7f021c09d3f0 2023-10-30T12:19:28.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 10 client.224291.objecter ms_dispatch 0x7f021c000b90 osd_map(593..593 src has 490..593) v4 2023-10-30T12:19:28.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 3 client.224291.objecter handle_osd_map got epochs [593,593] > 0 2023-10-30T12:19:28.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 3 client.224291.objecter handle_osd_map decoding full epoch 593 2023-10-30T12:19:28.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 10 monclient(hunting): handle_auth_done global_id 244113 payload 824 2023-10-30T12:19:28.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 1 -- 172.21.15.50:0/1192969223 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f021c097560 con 0x7f021c1c49d0 2023-10-30T12:19:28.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 20 client.224291.objecter dump_active .. 0 homeless 2023-10-30T12:19:28.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f0220c9b640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02000392f0 0x7f020003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f0220c9b640 10 monclient: get_auth_request con 0x7f02000392f0 auth_method 0 2023-10-30T12:19:28.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f022149c640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c49d0 0x7f021c1c4db0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f021800fb10 tx=0x7f0218001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 10 client.224291.objecter ms_handle_connect 0x7f021c1c49d0 2023-10-30T12:19:28.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 10 client.224291.objecter resend_mon_ops 2023-10-30T12:19:28.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02180052e0 con 0x7f021c1c49d0 2023-10-30T12:19:28.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:28.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.872+0000 7f020dffb640 10 monclient: dump: 2023-10-30T12:19:28.928 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.928 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.928 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.929 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.929 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.929 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.929 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.929 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.929 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.930 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 10 client.224291.objecter ms_handle_connect 0x7f02000392f0 2023-10-30T12:19:28.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f0220c9b640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02000392f0 0x7f020003b7b0 secure :-1 s=READY pgs=4286 cs=0 l=1 rev1=1 crypto rx=0x7f021c08d800 tx=0x7f0210007b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0218005440 con 0x7f021c1c49d0 2023-10-30T12:19:28.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:28.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:28.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c49d0 msgr2=0x7f021c1c4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c49d0 0x7f021c1c4db0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f021800fb10 tx=0x7f0218001e40 comp rx=0 tx=0).stop 2023-10-30T12:19:28.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c49d0 msgr2=0x7f021c1c4db0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:28.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.873+0000 7f020dffb640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c49d0 0x7f021c1c4db0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f021800fb10 tx=0x7f0218001e40 comp rx=0 tx=0).stop 2023-10-30T12:19:28.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f0223700640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:28.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f0223700640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c52f0 0x7f021c06fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:28.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f0223700640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:28.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c08e9d0 con 0x7f021c1c52f0 2023-10-30T12:19:28.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c52f0 0x7f021c06fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:28.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient: get_auth_request con 0x7f021c1c52f0 auth_method 0 2023-10-30T12:19:28.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:28.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:28.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:28.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:28.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:28.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:28.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 10 monclient(hunting): handle_auth_done global_id 244116 payload 824 2023-10-30T12:19:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 1 -- 172.21.15.50:0/1192969223 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f021c09f600 con 0x7f021c1c52f0 2023-10-30T12:19:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f020dffb640 10 client.224291.objecter ms_handle_connect 0x7f021c1c52f0 2023-10-30T12:19:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f022149c640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c52f0 0x7f021c06fdc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0218017000 tx=0x7f0218001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f020dffb640 10 client.224291.objecter resend_mon_ops 2023-10-30T12:19:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.879+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218024070 con 0x7f021c1c52f0 2023-10-30T12:19:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 10 monclient: dump: 2023-10-30T12:19:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:28.938 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:28.939 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f021801d460 con 0x7f021c1c52f0 2023-10-30T12:19:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c52f0 msgr2=0x7f021c06fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c52f0 0x7f021c06fdc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0218017000 tx=0x7f0218001170 comp rx=0 tx=0).stop 2023-10-30T12:19:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c52f0 msgr2=0x7f021c06fdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f020dffb640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f021c1c52f0 0x7f021c06fdc0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0218017000 tx=0x7f0218001170 comp rx=0 tx=0).stop 2023-10-30T12:19:28.941 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:28.941 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:28.941 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:28.941 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:19:28.941 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 198, 2023-10-30T12:19:28.942 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:28.942 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:28.942 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:19:28.942 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:19:28.943 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:28.943 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:19:28.943 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:28.943 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:28.944 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:28.944 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:28.944 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:28.944 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:28.944 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:28.945 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:28.945 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:28.945 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:28.945 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:28.945 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:28.946 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:28.946 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:28.946 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:28.946 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:28.947 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:28.947 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:28.947 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:28.947 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:28.948 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:28.948 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:28.948 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:28.948 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:28.948 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:28.949 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:28.949 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:28.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.949 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:28.949 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:28.950 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:28.950 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:28.950 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:28.950 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:28.950 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:28.951 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:28.951 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:28.951 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:28.951 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:28.951 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:28.952 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:28.952 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:28.952 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:28.952 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:28.952 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:28.953 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:28.953 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:28.953 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:28.953 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:28.954 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:28.954 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:28.954 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:28.954 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:28.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:28.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:28.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.959 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:28.960 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:28.960 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:28.960 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:28.960 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:28.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.961 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:28.961 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:28.961 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:28.961 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:28.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.962 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:28.962 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:28.962 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:28.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.963 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:28.963 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:28.963 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:28.963 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:28.964 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:28.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.964 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:28.964 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:28.964 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:28.965 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:28.965 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:28.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.965 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:28.966 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:28.966 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:28.966 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:28.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.967 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:28.967 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:28.967 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:28.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.968 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:28.968 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:28.968 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:28.968 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:28.968 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:28.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.969 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:28.969 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:28.969 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:28.969 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:28.970 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:28.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:28.970 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:28.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.971 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:28.971 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:28.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.971 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:28.971 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:28.972 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:28.972 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:28.972 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:28.972 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:28.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:28.972 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:28.973 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:28.973 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:28.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:28.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:28.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:28.974 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:28.974 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 10 librados: watch_flush enter 2023-10-30T12:19:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 10 librados: watch_flush exit 2023-10-30T12:19:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 20 client.224291.objecter shutdown clearing up homeless session... 2023-10-30T12:19:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 10 client.224291.objecter successfully canceled tick 2023-10-30T12:19:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02000392f0 msgr2=0x7f020003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f02000392f0 0x7f020003b7b0 secure :-1 s=READY pgs=4286 cs=0 l=1 rev1=1 crypto rx=0x7f021c08d800 tx=0x7f0210007b90 comp rx=0 tx=0).stop 2023-10-30T12:19:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 10 monclient: shutdown 2023-10-30T12:19:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09d3f0 msgr2=0x7f021c094a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09d3f0 0x7f021c094a50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f021c09fa70 tx=0x7f0214004a40 comp rx=0 tx=0).stop 2023-10-30T12:19:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0221c9d640 1 -- 172.21.15.50:0/1192969223 reap_dead start 2023-10-30T12:19:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 shutdown_connections 2023-10-30T12:19:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 --2- 172.21.15.50:0/1192969223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f021c09d3f0 0x7f021c094a50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 >> 172.21.15.50:0/1192969223 conn(0x7f021c0a3980 msgr2=0x7f021c096650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 shutdown_connections 2023-10-30T12:19:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 -- 172.21.15.50:0/1192969223 wait complete. 2023-10-30T12:19:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:28.880+0000 7f0223700640 1 librados: shutdown 2023-10-30T12:19:28.983 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:19:28.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:19:28.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:19:29.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.090+0000 7ffa795f5640 10 monclient: get_monmap_and_config 2023-10-30T12:19:29.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.090+0000 7ffa795f5640 10 monclient: build_initial_monmap 2023-10-30T12:19:29.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.090+0000 7ffa795f5640 10 monclient: monmap: 2023-10-30T12:19:29.091 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:29.091 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.091 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:29.090304+0000 2023-10-30T12:19:29.092 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:29.090304+0000 2023-10-30T12:19:29.092 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:29.092 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.092 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:29.092 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:29.093 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:29.093 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.090+0000 7ffa795f5640 1 Processor -- start 2023-10-30T12:19:29.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.090+0000 7ffa795f5640 1 -- start start 2023-10-30T12:19:29.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.090+0000 7ffa795f5640 10 monclient: init 2023-10-30T12:19:29.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:29.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-10-30T12:19:29.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ffa740a8010 0x7ffa74132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa740a8010 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:29.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa74099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:29.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa74132bd0 0x7ffa74135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient(hunting): picked mon.noname-a con 0x7ffa74132bd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:29.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa740a7190 con 0x7ffa74099a40 2023-10-30T12:19:29.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa740a83f0 con 0x7ffa74132bd0 2023-10-30T12:19:29.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa740a7600 con 0x7ffa740a8010 2023-10-30T12:19:29.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:29.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa795f5640 10 monclient(hunting): authenticate will time out at 2014.273465s 2023-10-30T12:19:29.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa74132bd0 0x7ffa74135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa727fc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ffa74099a40 0x7ffa741356d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40406/0 (socket says 172.21.15.50:40406) 2023-10-30T12:19:29.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa727fc640 1 -- 172.21.15.50:0/1498822362 learned_addr learned my addr 172.21.15.50:0/1498822362 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:29.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): get_auth_request con 0x7ffa74132bd0 auth_method 0 2023-10-30T12:19:29.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa72ffd640 1 --2- 172.21.15.50:0/1498822362 >> v2:172.21.15.118:3300/0 conn(0x7ffa740a8010 0x7ffa74132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa72ffd640 10 monclient(hunting): get_auth_request con 0x7ffa740a8010 auth_method 0 2023-10-30T12:19:29.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient(hunting): handle_auth_done global_id 244119 payload 274 2023-10-30T12:19:29.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 1 -- 172.21.15.50:0/1498822362 >> v2:172.21.15.118:3300/0 conn(0x7ffa740a8010 msgr2=0x7ffa74132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:29.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 1 --2- 172.21.15.50:0/1498822362 >> v2:172.21.15.118:3300/0 conn(0x7ffa740a8010 0x7ffa74132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 1 -- 172.21.15.50:0/1498822362 >> v1:172.21.15.50:6789/0 conn(0x7ffa74099a40 legacy=0x7ffa741356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient: _finish_hunting 0 2023-10-30T12:19:29.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 1 monclient: found mon.noname-a 2023-10-30T12:19:29.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:29.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa737fe640 1 -- 172.21.15.50:0/1498822362 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7409a440 con 0x7ffa74132bd0 2023-10-30T12:19:29.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa71ffb640 1 -- 172.21.15.50:0/1498822362 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695211073 0 0) 0x7ffa740a7190 con 0x7ffa74099a40 2023-10-30T12:19:29.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.091+0000 7ffa71ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:19:29.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa737fe640 1 --2- 172.21.15.50:0/1498822362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74132bd0 0x7ffa74135de0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa740a8640 tx=0x7ffa6402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdd4bb84bb7ab11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 1 -- 172.21.15.50:0/1498822362 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6402cb70 con 0x7ffa74132bd0 2023-10-30T12:19:29.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:29.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: dump: 2023-10-30T12:19:29.110 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.111 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.111 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.111 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.111 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.111 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.112 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.112 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.112 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.112 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: _finish_auth 0 2023-10-30T12:19:29.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:19:29.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:29.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa795f5640 5 monclient: authenticate success, global_id 244119 2023-10-30T12:19:29.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 1 -- 172.21.15.50:0/1498822362 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6402ccd0 con 0x7ffa74132bd0 2023-10-30T12:19:29.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa795f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:29.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa795f5640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa795f5640 4 set_mon_vals no callback set 2023-10-30T12:19:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 1 -- 172.21.15.50:0/1498822362 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64034660 con 0x7ffa74132bd0 2023-10-30T12:19:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:29.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.092+0000 7ffa71ffb640 10 monclient: dump: 2023-10-30T12:19:29.115 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.115 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.115 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.115 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.116 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.116 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.116 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.116 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.116 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.116 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 10 monclient: shutdown 2023-10-30T12:19:29.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:29.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 -- 172.21.15.50:0/1498822362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74132bd0 msgr2=0x7ffa74135de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 --2- 172.21.15.50:0/1498822362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74132bd0 0x7ffa74135de0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa740a8640 tx=0x7ffa6402bdc0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 -- 172.21.15.50:0/1498822362 shutdown_connections 2023-10-30T12:19:29.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 --2- 172.21.15.50:0/1498822362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74132bd0 0x7ffa74135de0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 --2- 172.21.15.50:0/1498822362 >> v2:172.21.15.118:3300/0 conn(0x7ffa740a8010 0x7ffa74132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 -- 172.21.15.50:0/1498822362 >> 172.21.15.50:0/1498822362 conn(0x7ffa740a4a10 msgr2=0x7ffa740a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:29.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 -- 172.21.15.50:0/1498822362 shutdown_connections 2023-10-30T12:19:29.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 1 -- 172.21.15.50:0/1498822362 wait complete. 2023-10-30T12:19:29.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 10 monclient: build_initial_monmap 2023-10-30T12:19:29.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.093+0000 7ffa795f5640 10 monclient: monmap: 2023-10-30T12:19:29.119 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:29.119 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:29.119 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:29.120 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:29.120 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:29.120 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.120 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:29.120 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:29.120 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:29.121 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 librados: starting msgr at 2023-10-30T12:19:29.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 librados: starting objecter 2023-10-30T12:19:29.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 Processor -- start 2023-10-30T12:19:29.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 -- start start 2023-10-30T12:19:29.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 librados: setting wanted keys 2023-10-30T12:19:29.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 librados: calling monclient init 2023-10-30T12:19:29.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient: init 2023-10-30T12:19:29.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:29.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:19:29.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa740a8010 0x7ffa7409f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): picked mon.noname-b con 0x7ffa740a8010 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:29.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74132bd0 0x7ffa7409faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa74132bd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:29.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa7409ffe0 0x7ffa741d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): picked mon.noname-a con 0x7ffa7409ffe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:29.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7411ec80 con 0x7ffa7409ffe0 2023-10-30T12:19:29.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7411fda0 con 0x7ffa740a8010 2023-10-30T12:19:29.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74120360 con 0x7ffa74132bd0 2023-10-30T12:19:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.094+0000 7ffa795f5640 10 monclient(hunting): authenticate will time out at 2014.276953s 2023-10-30T12:19:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa7409ffe0 0x7ffa741d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74132bd0 0x7ffa7409faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa7409ffe0 0x7ffa741d05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48684/0 (socket says 172.21.15.50:48684) 2023-10-30T12:19:29.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 1 -- 172.21.15.50:0/2664855585 learned_addr learned my addr 172.21.15.50:0/2664855585 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:29.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 10 monclient(hunting): get_auth_request con 0x7ffa7409ffe0 auth_method 0 2023-10-30T12:19:29.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): get_auth_request con 0x7ffa74132bd0 auth_method 0 2023-10-30T12:19:29.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa740a8010 0x7ffa7409f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 10 monclient(hunting): get_auth_request con 0x7ffa740a8010 auth_method 0 2023-10-30T12:19:29.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient(hunting): handle_auth_done global_id 224294 payload 824 2023-10-30T12:19:29.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa740a8010 msgr2=0x7ffa7409f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa740a8010 0x7ffa7409f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa7409ffe0 msgr2=0x7ffa741d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa7409ffe0 0x7ffa741d05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient: _finish_hunting 0 2023-10-30T12:19:29.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 monclient: found mon.noname-c 2023-10-30T12:19:29.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:29.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:19:29.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 -- 172.21.15.50:0/2664855585 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa64009a70 con 0x7ffa74132bd0 2023-10-30T12:19:29.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa737fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa7409ffe0 0x7ffa741d05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:29.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa727fc640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74132bd0 0x7ffa7409faa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffa740a5230 tx=0x7ffa6800c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 client.?.objecter ms_handle_connect 0x7ffa74132bd0 2023-10-30T12:19:29.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:29.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa680074e0 con 0x7ffa74132bd0 2023-10-30T12:19:29.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:19:29.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: dump: 2023-10-30T12:19:29.133 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.135 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.135 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.135 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.135 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:19:29.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:19:29.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:29.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 5 monclient: authenticate success, global_id 224294 2023-10-30T12:19:29.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa68007660 con 0x7ffa74132bd0 2023-10-30T12:19:29.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 10 monclient: _renew_subs 2023-10-30T12:19:29.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:29.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa740a18d0 con 0x7ffa74132bd0 2023-10-30T12:19:29.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:29.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa68004a20 con 0x7ffa74132bd0 2023-10-30T12:19:29.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:29.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa5b7fe640 10 monclient: dump: 2023-10-30T12:19:29.138 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.138 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.138 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.138 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.139 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.139 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.139 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.139 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.139 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.140 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa717fa640 4 set_mon_vals no callback set 2023-10-30T12:19:29.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 10 client.224294.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:29.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 10 monclient: _renew_subs 2023-10-30T12:19:29.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:19:29.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7411dfb0 con 0x7ffa74132bd0 2023-10-30T12:19:29.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.095+0000 7ffa795f5640 1 librados: init done 2023-10-30T12:19:29.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ffa68004b80 con 0x7ffa74132bd0 2023-10-30T12:19:29.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa795f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:29.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa795f5640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74099a40 0x7ffa7402c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa795f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74073fd0 con 0x7ffa74099a40 2023-10-30T12:19:29.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74099a40 0x7ffa7402c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa5b7fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa600392f0 0x7ffa6003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 10 monclient: get_auth_request con 0x7ffa74099a40 auth_method 0 2023-10-30T12:19:29.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(593..593 src has 490..593) v4 ==== 6650+0+0 (secure 0 0 0) 0x7ffa6804ab80 con 0x7ffa74132bd0 2023-10-30T12:19:29.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa5b7fe640 10 client.224294.objecter ms_dispatch 0x7ffa74000b90 osd_map(593..593 src has 490..593) v4 2023-10-30T12:19:29.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa5b7fe640 3 client.224294.objecter handle_osd_map got epochs [593,593] > 0 2023-10-30T12:19:29.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa5b7fe640 3 client.224294.objecter handle_osd_map decoding full epoch 593 2023-10-30T12:19:29.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.096+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 20 client.224294.objecter dump_active .. 0 homeless 2023-10-30T12:19:29.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_done global_id 224297 payload 824 2023-10-30T12:19:29.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa72ffd640 1 -- 172.21.15.50:0/2664855585 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7409aa00 con 0x7ffa74099a40 2023-10-30T12:19:29.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa737fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa600392f0 0x7ffa6003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa737fe640 10 monclient: get_auth_request con 0x7ffa600392f0 auth_method 0 2023-10-30T12:19:29.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa72ffd640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74099a40 0x7ffa7402c5b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c010780 tx=0x7ffa6c010c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 client.224294.objecter ms_handle_connect 0x7ffa74099a40 2023-10-30T12:19:29.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 client.224294.objecter resend_mon_ops 2023-10-30T12:19:29.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c004240 con 0x7ffa74099a40 2023-10-30T12:19:29.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:29.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 monclient: dump: 2023-10-30T12:19:29.147 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.147 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.147 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.147 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.147 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.147 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.148 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.148 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.148 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.148 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 client.224294.objecter ms_handle_connect 0x7ffa600392f0 2023-10-30T12:19:29.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa737fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa600392f0 0x7ffa6003b7b0 secure :-1 s=READY pgs=4287 cs=0 l=1 rev1=1 crypto rx=0x7ffa6402be00 tx=0x7ffa64030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ffa6c0043a0 con 0x7ffa74099a40 2023-10-30T12:19:29.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:29.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:29.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74099a40 msgr2=0x7ffa7402c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74099a40 0x7ffa7402c5b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c010780 tx=0x7ffa6c010c40 comp rx=0 tx=0).stop 2023-10-30T12:19:29.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74099a40 msgr2=0x7ffa7402c5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:29.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.097+0000 7ffa5b7fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74099a40 0x7ffa7402c5b0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c010780 tx=0x7ffa6c010c40 comp rx=0 tx=0).stop 2023-10-30T12:19:29.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.103+0000 7ffa795f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:29.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.103+0000 7ffa795f5640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa740a03f0 0x7ffa74060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.103+0000 7ffa795f5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.103+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7409a750 con 0x7ffa740a03f0 2023-10-30T12:19:29.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa740a03f0 0x7ffa74060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient: get_auth_request con 0x7ffa740a03f0 auth_method 0 2023-10-30T12:19:29.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 10 monclient(hunting): handle_auth_done global_id 224300 payload 824 2023-10-30T12:19:29.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 1 -- 172.21.15.50:0/2664855585 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa74133080 con 0x7ffa740a03f0 2023-10-30T12:19:29.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 10 client.224294.objecter ms_handle_connect 0x7ffa740a03f0 2023-10-30T12:19:29.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 10 client.224294.objecter resend_mon_ops 2023-10-30T12:19:29.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa72ffd640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa740a03f0 0x7ffa74060fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c010780 tx=0x7ffa6c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c025070 con 0x7ffa740a03f0 2023-10-30T12:19:29.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:29.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 10 monclient: dump: 2023-10-30T12:19:29.156 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.156 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.156 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.157 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.157 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.157 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.157 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.157 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.158 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.158 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7ffa6c003be0 con 0x7ffa740a03f0 2023-10-30T12:19:29.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:29.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:29.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa740a03f0 msgr2=0x7ffa74060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa740a03f0 0x7ffa74060fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c010780 tx=0x7ffa6c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa740a03f0 msgr2=0x7ffa74060fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:29.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.104+0000 7ffa5b7fe640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa740a03f0 0x7ffa74060fc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.159 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:29.160 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:29.160 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:29.160 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:19:29.160 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 198, 2023-10-30T12:19:29.161 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:29.161 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:29.161 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:19:29.161 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:19:29.161 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.161 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:19:29.162 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:29.162 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:29.162 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:29.162 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:29.162 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:29.163 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:29.163 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:29.163 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:29.163 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:29.163 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:29.164 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:29.164 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:29.164 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:29.164 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.164 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:29.164 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:29.165 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:29.165 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:29.165 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:29.165 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:29.165 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:29.166 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:29.166 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:29.166 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:29.166 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:29.166 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:29.167 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.167 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:29.167 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:29.167 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:29.168 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:29.168 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:29.168 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:29.168 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:29.168 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:29.168 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:29.169 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:29.169 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:29.169 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:29.169 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:29.169 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:29.170 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:29.170 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:29.170 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:29.170 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:29.170 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:29.171 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:29.171 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:29.171 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:29.171 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:29.171 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:29.172 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:29.172 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:29.172 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:29.172 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.172 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:29.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.173 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:29.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.173 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:29.173 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:29.173 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:29.174 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:29.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:29.174 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:29.174 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.175 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:29.175 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:29.175 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.176 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.176 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:29.176 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:29.176 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:29.177 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:29.177 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:29.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.177 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:29.178 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:29.178 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:29.178 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:29.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.178 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:29.179 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:29.179 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:29.179 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:29.180 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.180 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.180 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.180 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:29.181 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:29.181 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:29.181 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:29.181 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:29.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.182 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:29.182 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:29.182 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:29.182 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:29.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.183 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:29.183 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:29.183 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:29.184 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:29.184 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.184 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.185 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:29.185 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:29.185 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:29.185 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:29.186 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:29.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.186 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.186 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:29.187 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:29.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.187 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:29.187 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:29.187 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:29.188 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.188 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.188 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:19:29.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.188 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:29.188 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:29.189 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:19:29.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.189 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.189 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:29.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.190 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:29.190 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:29.190 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:19:29.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.190 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.191 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:19:29.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.191 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:29.191 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:29.191 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:29.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.192 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.192 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:29.192 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 10 librados: watch_flush enter 2023-10-30T12:19:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 10 librados: watch_flush exit 2023-10-30T12:19:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 20 client.224294.objecter shutdown clearing up homeless session... 2023-10-30T12:19:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 10 client.224294.objecter successfully canceled tick 2023-10-30T12:19:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa600392f0 msgr2=0x7ffa6003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffa600392f0 0x7ffa6003b7b0 secure :-1 s=READY pgs=4287 cs=0 l=1 rev1=1 crypto rx=0x7ffa6402be00 tx=0x7ffa64030000 comp rx=0 tx=0).stop 2023-10-30T12:19:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 10 monclient: shutdown 2023-10-30T12:19:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74132bd0 msgr2=0x7ffa7409faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 --2- 172.21.15.50:0/2664855585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa74132bd0 0x7ffa7409faa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffa740a5230 tx=0x7ffa6800c870 comp rx=0 tx=0).stop 2023-10-30T12:19:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa737fe640 1 -- 172.21.15.50:0/2664855585 reap_dead start 2023-10-30T12:19:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa737fe640 1 -- 172.21.15.50:0/2664855585 reap_dead start 2023-10-30T12:19:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 shutdown_connections 2023-10-30T12:19:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 >> 172.21.15.50:0/2664855585 conn(0x7ffa740a4a10 msgr2=0x7ffa74135970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 shutdown_connections 2023-10-30T12:19:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 -- 172.21.15.50:0/2664855585 wait complete. 2023-10-30T12:19:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.105+0000 7ffa795f5640 1 librados: shutdown 2023-10-30T12:19:29.198 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:19:29.198 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:19:29.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:19:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.302+0000 7f0e6aa21640 10 monclient: get_monmap_and_config 2023-10-30T12:19:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.302+0000 7f0e6aa21640 10 monclient: build_initial_monmap 2023-10-30T12:19:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.302+0000 7f0e6aa21640 10 monclient: monmap: 2023-10-30T12:19:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:29.303830+0000 2023-10-30T12:19:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:29.303830+0000 2023-10-30T12:19:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:29.307 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 1 Processor -- start 2023-10-30T12:19:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 1 -- start start 2023-10-30T12:19:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 10 monclient: init 2023-10-30T12:19:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-10-30T12:19:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64097780 0x7f0e64097b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 10 monclient(hunting): picked mon.noname-b con 0x7f0e64097780 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e64073b10 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0e6409ccd0 0x7f0e640a0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6409ccd0 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e64071ec0 con 0x7f0e64073b10 2023-10-30T12:19:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.303+0000 7f0e6aa21640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6409bd10 con 0x7f0e6409ccd0 2023-10-30T12:19:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e6aa21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e6aa21640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6409bff0 con 0x7f0e64097780 2023-10-30T12:19:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e6aa21640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e6aa21640 10 monclient(hunting): authenticate will time out at 2014.487022s 2023-10-30T12:19:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0e6409ccd0 0x7f0e640a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64097780 0x7f0e64097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64097780 0x7f0e64097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33798/0 (socket says 172.21.15.50:33798) 2023-10-30T12:19:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0e6409ccd0 0x7f0e640a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48686/0 (socket says 172.21.15.50:48686) 2023-10-30T12:19:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 -- 172.21.15.50:0/161606511 learned_addr learned my addr 172.21.15.50:0/161606511 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): get_auth_request con 0x7f0e64097780 auth_method 0 2023-10-30T12:19:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e637fe640 1 --1- 172.21.15.50:0/161606511 >> v1:172.21.15.50:6789/0 conn(0x7f0e64073b10 0x7f0e640a0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40418/0 (socket says 172.21.15.50:40418) 2023-10-30T12:19:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 10 monclient(hunting): get_auth_request con 0x7f0e6409ccd0 auth_method 0 2023-10-30T12:19:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient(hunting): handle_auth_done global_id 224303 payload 274 2023-10-30T12:19:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 -- 172.21.15.50:0/161606511 >> v2:172.21.15.50:3300/0 conn(0x7f0e6409ccd0 msgr2=0x7f0e640a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 --2- 172.21.15.50:0/161606511 >> v2:172.21.15.50:3300/0 conn(0x7f0e6409ccd0 0x7f0e640a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 -- 172.21.15.50:0/161606511 >> v1:172.21.15.50:6789/0 conn(0x7f0e64073b10 legacy=0x7f0e640a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient: _finish_hunting 0 2023-10-30T12:19:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 monclient: found mon.noname-b 2023-10-30T12:19:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e62ffd640 1 -- 172.21.15.50:0/161606511 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629778184 0 0) 0x7f0e64071ec0 con 0x7f0e64073b10 2023-10-30T12:19:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e63fff640 1 -- 172.21.15.50:0/161606511 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640980a0 con 0x7f0e64097780 2023-10-30T12:19:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e62ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:19:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.304+0000 7f0e68fbe640 1 --2- 172.21.15.50:0/161606511 >> v2:172.21.15.50:3300/0 conn(0x7f0e6409ccd0 0x7f0e640a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e63fff640 1 --2- 172.21.15.50:0/161606511 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64097780 0x7f0e64097b60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009870 tx=0x7f0e5402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16524abd9008ebd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 1 -- 172.21.15.50:0/161606511 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e540317c0 con 0x7f0e64097780 2023-10-30T12:19:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: dump: 2023-10-30T12:19:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.327 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: _finish_auth 0 2023-10-30T12:19:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:19:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 5 monclient: authenticate success, global_id 224303 2023-10-30T12:19:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 1 -- 172.21.15.50:0/161606511 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54031920 con 0x7f0e64097780 2023-10-30T12:19:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 1 -- 172.21.15.50:0/161606511 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54031cb0 con 0x7f0e64097780 2023-10-30T12:19:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 4 set_mon_vals no callback set 2023-10-30T12:19:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e62ffd640 10 monclient: dump: 2023-10-30T12:19:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.332 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 10 monclient: shutdown 2023-10-30T12:19:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 1 -- 172.21.15.50:0/161606511 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64097780 msgr2=0x7f0e64097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.305+0000 7f0e6aa21640 1 --2- 172.21.15.50:0/161606511 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64097780 0x7f0e64097b60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009870 tx=0x7f0e5402b860 comp rx=0 tx=0).stop 2023-10-30T12:19:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 -- 172.21.15.50:0/161606511 shutdown_connections 2023-10-30T12:19:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 --2- 172.21.15.50:0/161606511 >> v2:172.21.15.50:3300/0 conn(0x7f0e6409ccd0 0x7f0e640a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 --2- 172.21.15.50:0/161606511 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64097780 0x7f0e64097b60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 -- 172.21.15.50:0/161606511 >> 172.21.15.50:0/161606511 conn(0x7f0e640a1170 msgr2=0x7f0e640a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 -- 172.21.15.50:0/161606511 shutdown_connections 2023-10-30T12:19:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 -- 172.21.15.50:0/161606511 wait complete. 2023-10-30T12:19:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 10 monclient: build_initial_monmap 2023-10-30T12:19:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 10 monclient: monmap: 2023-10-30T12:19:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:29.336 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 librados: starting msgr at 2023-10-30T12:19:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.306+0000 7f0e6aa21640 1 librados: starting objecter 2023-10-30T12:19:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 Processor -- start 2023-10-30T12:19:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 -- start start 2023-10-30T12:19:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 librados: setting wanted keys 2023-10-30T12:19:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 librados: calling monclient init 2023-10-30T12:19:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient: init 2023-10-30T12:19:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:19:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64073b10 0x7f0e641ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): picked mon.noname-c con 0x7f0e64073b10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e64097780 0x7f0e641cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): picked mon.noname-a con 0x7f0e64097780 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6409ccd0 0x7f0e6408f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6409ccd0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64073f70 con 0x7f0e64097780 2023-10-30T12:19:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640a3b00 con 0x7f0e6409ccd0 2023-10-30T12:19:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64093520 con 0x7f0e64073b10 2023-10-30T12:19:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e6aa21640 10 monclient(hunting): authenticate will time out at 2014.490723s 2023-10-30T12:19:29.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64073b10 0x7f0e641ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e64097780 0x7f0e641cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64073b10 0x7f0e641ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33808/0 (socket says 172.21.15.50:33808) 2023-10-30T12:19:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 10 monclient(hunting): get_auth_request con 0x7f0e64097780 auth_method 0 2023-10-30T12:19:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 1 -- 172.21.15.50:0/2651059267 learned_addr learned my addr 172.21.15.50:0/2651059267 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 10 monclient(hunting): get_auth_request con 0x7f0e64073b10 auth_method 0 2023-10-30T12:19:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.307+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e68fbe640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6409ccd0 0x7f0e6408f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e68fbe640 10 monclient(hunting): get_auth_request con 0x7f0e6409ccd0 auth_method 0 2023-10-30T12:19:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e68fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e68fbe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e68fbe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 10 monclient(hunting): handle_auth_done global_id 244134 payload 824 2023-10-30T12:19:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64073b10 msgr2=0x7f0e641ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64073b10 0x7f0e641ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6409ccd0 msgr2=0x7f0e6408f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6409ccd0 0x7f0e6408f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 10 monclient: _finish_hunting 0 2023-10-30T12:19:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 1 monclient: found mon.noname-a 2023-10-30T12:19:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 1 -- 172.21.15.50:0/2651059267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c009a70 con 0x7f0e64097780 2023-10-30T12:19:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e63fff640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0e64073b10 0x7f0e641ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e637fe640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e64097780 0x7f0e641cae10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800c450 tx=0x7f0e5800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 client.?.objecter ms_handle_connect 0x7f0e64097780 2023-10-30T12:19:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e580074e0 con 0x7f0e64097780 2023-10-30T12:19:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: dump: 2023-10-30T12:19:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.350 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: _finish_auth 0 2023-10-30T12:19:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:19:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 5 monclient: authenticate success, global_id 244134 2023-10-30T12:19:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 10 monclient: _renew_subs 2023-10-30T12:19:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e58007640 con 0x7f0e64097780 2023-10-30T12:19:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e540094d0 con 0x7f0e64097780 2023-10-30T12:19:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e58004a20 con 0x7f0e64097780 2023-10-30T12:19:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e60ff9640 10 monclient: dump: 2023-10-30T12:19:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.354 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e627fc640 4 set_mon_vals no callback set 2023-10-30T12:19:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 10 client.244134.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 10 monclient: _renew_subs 2023-10-30T12:19:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e64092c80 con 0x7f0e64097780 2023-10-30T12:19:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.308+0000 7f0e6aa21640 1 librados: init done 2023-10-30T12:19:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e6aa21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e6aa21640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6408fa90 0x7f0e6408fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e6aa21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6406a5a0 con 0x7f0e6408fa90 2023-10-30T12:19:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0e58004b80 con 0x7f0e64097780 2023-10-30T12:19:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6408fa90 0x7f0e6408fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient: get_auth_request con 0x7f0e6408fa90 auth_method 0 2023-10-30T12:19:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e60ff9640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0e500392f0 0x7f0e5003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(593..593 src has 490..593) v4 ==== 6650+0+0 (secure 0 0 0) 0x7f0e58049f10 con 0x7f0e64097780 2023-10-30T12:19:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e60ff9640 10 client.244134.objecter ms_dispatch 0x7f0e64000b90 osd_map(593..593 src has 490..593) v4 2023-10-30T12:19:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e60ff9640 3 client.244134.objecter handle_osd_map got epochs [593,593] > 0 2023-10-30T12:19:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e60ff9640 3 client.244134.objecter handle_osd_map decoding full epoch 593 2023-10-30T12:19:29.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e60ff9640 20 client.244134.objecter dump_active .. 0 homeless 2023-10-30T12:19:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e68fbe640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0e500392f0 0x7f0e5003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e68fbe640 10 monclient: get_auth_request con 0x7f0e500392f0 auth_method 0 2023-10-30T12:19:29.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 10 monclient(hunting): handle_auth_done global_id 204571 payload 824 2023-10-30T12:19:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.309+0000 7f0e63fff640 1 -- 172.21.15.50:0/2651059267 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e640a0cd0 con 0x7f0e6408fa90 2023-10-30T12:19:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 client.244134.objecter ms_handle_connect 0x7f0e6408fa90 2023-10-30T12:19:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 client.244134.objecter resend_mon_ops 2023-10-30T12:19:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e63fff640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6408fa90 0x7f0e6408fe70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e54034040 tx=0x7f0e54004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 client.244134.objecter ms_handle_connect 0x7f0e500392f0 2023-10-30T12:19:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e68fbe640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0e500392f0 0x7f0e5003b7b0 secure :-1 s=READY pgs=4288 cs=0 l=1 rev1=1 crypto rx=0x7f0e64092bb0 tx=0x7f0e5c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54040040 con 0x7f0e6408fa90 2023-10-30T12:19:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 monclient: dump: 2023-10-30T12:19:29.361 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.363 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f0e54003810 con 0x7f0e6408fa90 2023-10-30T12:19:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6408fa90 msgr2=0x7f0e6408fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6408fa90 0x7f0e6408fe70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e54034040 tx=0x7f0e54004730 comp rx=0 tx=0).stop 2023-10-30T12:19:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6408fa90 msgr2=0x7f0e6408fe70 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:29.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.310+0000 7f0e60ff9640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e6408fa90 0x7f0e6408fe70 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e54034040 tx=0x7f0e54004730 comp rx=0 tx=0).stop 2023-10-30T12:19:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.316+0000 7f0e6aa21640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.316+0000 7f0e6aa21640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e640903b0 0x7f0e6408bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.316+0000 7f0e6aa21640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.316+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6404e160 con 0x7f0e640903b0 2023-10-30T12:19:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e640903b0 0x7f0e6408bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient: get_auth_request con 0x7f0e640903b0 auth_method 0 2023-10-30T12:19:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 10 monclient(hunting): handle_auth_done global_id 204574 payload 824 2023-10-30T12:19:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 1 -- 172.21.15.50:0/2651059267 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0e64096840 con 0x7f0e640903b0 2023-10-30T12:19:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e63fff640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e640903b0 0x7f0e6408bfe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009870 tx=0x7f0e540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e60ff9640 10 client.244134.objecter ms_handle_connect 0x7f0e640903b0 2023-10-30T12:19:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.317+0000 7f0e60ff9640 10 client.244134.objecter resend_mon_ops 2023-10-30T12:19:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54032e90 con 0x7f0e640903b0 2023-10-30T12:19:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 10 monclient: dump: 2023-10-30T12:19:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.372 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f0e5405a440 con 0x7f0e640903b0 2023-10-30T12:19:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e640903b0 msgr2=0x7f0e6408bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e640903b0 0x7f0e6408bfe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009870 tx=0x7f0e540046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e640903b0 msgr2=0x7f0e6408bfe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e60ff9640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e640903b0 0x7f0e6408bfe0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009870 tx=0x7f0e540046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.374 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:29.374 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:29.375 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:29.375 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:19:29.375 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 198, 2023-10-30T12:19:29.375 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:29.375 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:19:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:19:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:19:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:29.376 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:29.377 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:29.377 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:29.377 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:29.377 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:29.377 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:29.378 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:29.378 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:29.378 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:29.378 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:29.378 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:29.379 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:29.379 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.379 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:29.379 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:29.379 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:29.379 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:29.380 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:29.380 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:29.380 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:29.380 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:29.380 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:29.381 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:29.381 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:29.381 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:29.381 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.382 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:29.382 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:29.382 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:29.382 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:29.382 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:29.384 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:29.384 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:29.384 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:29.384 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:29.384 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:29.385 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:29.385 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:29.385 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:29.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:29.385 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:29.386 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:29.386 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:29.386 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:29.386 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:29.386 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:29.386 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:29.387 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.387 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:29.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:29.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:29.389 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:29.389 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.390 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:29.390 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:29.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.391 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:29.391 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:29.391 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:29.392 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:29.392 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:29.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.392 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:29.392 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:29.393 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:29.393 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:29.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:29.393 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:29.394 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.394 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:29.394 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:29.395 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.395 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:29.395 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:29.396 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:29.396 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:29.396 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:29.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.397 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:29.397 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:29.397 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:29.397 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:29.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:29.400 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:29.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:29.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:29.403 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:19:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:29.404 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:29.405 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 10 librados: watch_flush enter 2023-10-30T12:19:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 10 librados: watch_flush exit 2023-10-30T12:19:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 20 client.244134.objecter shutdown clearing up homeless session... 2023-10-30T12:19:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 10 client.244134.objecter successfully canceled tick 2023-10-30T12:19:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0e500392f0 msgr2=0x7f0e5003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0e500392f0 0x7f0e5003b7b0 secure :-1 s=READY pgs=4288 cs=0 l=1 rev1=1 crypto rx=0x7f0e64092bb0 tx=0x7f0e5c00b040 comp rx=0 tx=0).stop 2023-10-30T12:19:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 10 monclient: shutdown 2023-10-30T12:19:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e64097780 msgr2=0x7f0e641cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 --2- 172.21.15.50:0/2651059267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e64097780 0x7f0e641cae10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800c450 tx=0x7f0e5800c910 comp rx=0 tx=0).stop 2023-10-30T12:19:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e68fbe640 1 -- 172.21.15.50:0/2651059267 reap_dead start 2023-10-30T12:19:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e68fbe640 1 -- 172.21.15.50:0/2651059267 reap_dead start 2023-10-30T12:19:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 shutdown_connections 2023-10-30T12:19:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 >> 172.21.15.50:0/2651059267 conn(0x7f0e640a1170 msgr2=0x7f0e640a2570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 shutdown_connections 2023-10-30T12:19:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 -- 172.21.15.50:0/2651059267 wait complete. 2023-10-30T12:19:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.318+0000 7f0e6aa21640 1 librados: shutdown 2023-10-30T12:19:29.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:19:29.410 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:19:29.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:19:29.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.515+0000 7fd7b917c640 10 monclient: get_monmap_and_config 2023-10-30T12:19:29.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.515+0000 7fd7b917c640 10 monclient: build_initial_monmap 2023-10-30T12:19:29.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.515+0000 7fd7b917c640 10 monclient: monmap: 2023-10-30T12:19:29.517 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:29.517 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.517 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:29.516158+0000 2023-10-30T12:19:29.517 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:29.516158+0000 2023-10-30T12:19:29.518 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:29.518 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.518 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:29.518 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:29.518 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:29.519 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:29.519 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:29.519 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.515+0000 7fd7b917c640 1 Processor -- start 2023-10-30T12:19:29.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.515+0000 7fd7b917c640 1 -- start start 2023-10-30T12:19:29.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.515+0000 7fd7b917c640 10 monclient: init 2023-10-30T12:19:29.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:29.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-10-30T12:19:29.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd7b40a4b60 0x7fd7b41326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b40a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:29.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7b4099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:29.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b40a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b4132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:29.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b40a4f40 con 0x7fd7b4099a40 2023-10-30T12:19:29.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b40a3d70 con 0x7fd7b40a4b60 2023-10-30T12:19:29.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b40a4150 con 0x7fd7b4132be0 2023-10-30T12:19:29.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:29.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b917c640 10 monclient(hunting): authenticate will time out at 2014.699313s 2023-10-30T12:19:29.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd7b40a4b60 0x7fd7b41326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b40a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd7b40a4b60 0x7fd7b41326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48706/0 (socket says 172.21.15.50:48706) 2023-10-30T12:19:29.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 1 -- 172.21.15.50:0/2550466401 learned_addr learned my addr 172.21.15.50:0/2550466401 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:29.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 10 monclient(hunting): get_auth_request con 0x7fd7b40a4b60 auth_method 0 2023-10-30T12:19:29.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): get_auth_request con 0x7fd7b4132be0 auth_method 0 2023-10-30T12:19:29.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient(hunting): handle_auth_done global_id 224309 payload 274 2023-10-30T12:19:29.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 1 -- 172.21.15.50:0/2550466401 >> v2:172.21.15.50:3300/0 conn(0x7fd7b40a4b60 msgr2=0x7fd7b41326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:29.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 1 --2- 172.21.15.50:0/2550466401 >> v2:172.21.15.50:3300/0 conn(0x7fd7b40a4b60 0x7fd7b41326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 1 -- 172.21.15.50:0/2550466401 >> v1:172.21.15.118:6789/0 conn(0x7fd7b4099a40 legacy=0x7fd7b40a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient: _finish_hunting 0 2023-10-30T12:19:29.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 1 monclient: found mon.noname-b 2023-10-30T12:19:29.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:29.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:29.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b359e640 1 -- 172.21.15.50:0/2550466401 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b409a440 con 0x7fd7b4132be0 2023-10-30T12:19:29.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.516+0000 7fd7b2d9d640 1 --2- 172.21.15.50:0/2550466401 >> v2:172.21.15.50:3300/0 conn(0x7fd7b40a4b60 0x7fd7b41326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:29.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b359e640 1 --2- 172.21.15.50:0/2550466401 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b40a2a60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4009d90 tx=0x7fd7a402bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d781bc45aedbabc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 1 -- 172.21.15.50:0/2550466401 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a402cb40 con 0x7fd7b4132be0 2023-10-30T12:19:29.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:29.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: dump: 2023-10-30T12:19:29.536 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.537 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.537 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.537 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.537 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.537 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.538 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.538 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.538 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.538 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: _finish_auth 0 2023-10-30T12:19:29.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:19:29.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:29.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 1 -- 172.21.15.50:0/2550466401 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a402cca0 con 0x7fd7b4132be0 2023-10-30T12:19:29.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b917c640 5 monclient: authenticate success, global_id 224309 2023-10-30T12:19:29.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b917c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:29.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:29.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b917c640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:29.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 1 -- 172.21.15.50:0/2550466401 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4034660 con 0x7fd7b4132be0 2023-10-30T12:19:29.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b917c640 4 set_mon_vals no callback set 2023-10-30T12:19:29.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:29.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.517+0000 7fd7b1d9b640 10 monclient: dump: 2023-10-30T12:19:29.541 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.541 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.541 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.541 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.543 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 10 monclient: shutdown 2023-10-30T12:19:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 -- 172.21.15.50:0/2550466401 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 msgr2=0x7fd7b40a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 --2- 172.21.15.50:0/2550466401 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b40a2a60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4009d90 tx=0x7fd7a402bd90 comp rx=0 tx=0).stop 2023-10-30T12:19:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 -- 172.21.15.50:0/2550466401 shutdown_connections 2023-10-30T12:19:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 --2- 172.21.15.50:0/2550466401 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b40a2a60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 --2- 172.21.15.50:0/2550466401 >> v2:172.21.15.50:3300/0 conn(0x7fd7b40a4b60 0x7fd7b41326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 -- 172.21.15.50:0/2550466401 >> 172.21.15.50:0/2550466401 conn(0x7fd7b40a8060 msgr2=0x7fd7b40a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 -- 172.21.15.50:0/2550466401 shutdown_connections 2023-10-30T12:19:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 1 -- 172.21.15.50:0/2550466401 wait complete. 2023-10-30T12:19:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 10 monclient: build_initial_monmap 2023-10-30T12:19:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.518+0000 7fd7b917c640 10 monclient: monmap: 2023-10-30T12:19:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:29.545 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:29.546 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:29.546 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:29.546 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.546 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:29.546 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:29.547 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:29.547 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 librados: starting msgr at 2023-10-30T12:19:29.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 librados: starting objecter 2023-10-30T12:19:29.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 Processor -- start 2023-10-30T12:19:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 -- start start 2023-10-30T12:19:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 librados: setting wanted keys 2023-10-30T12:19:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 librados: calling monclient init 2023-10-30T12:19:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient: init 2023-10-30T12:19:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7b40a4b60 0x7fd7b41ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b40a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b41cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b4132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b411e490 0x7fd7b411e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b411e490 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b41222b0 con 0x7fd7b40a4b60 2023-10-30T12:19:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b4122500 con 0x7fd7b411e490 2023-10-30T12:19:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b4120d70 con 0x7fd7b4132be0 2023-10-30T12:19:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b917c640 10 monclient(hunting): authenticate will time out at 2014.702833s 2023-10-30T12:19:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7b40a4b60 0x7fd7b41ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b41cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7b40a4b60 0x7fd7b41ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48718/0 (socket says 172.21.15.50:48718) 2023-10-30T12:19:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b41cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33834/0 (socket says 172.21.15.50:33834) 2023-10-30T12:19:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b2d9d640 1 -- 172.21.15.50:0/352189838 learned_addr learned my addr 172.21.15.50:0/352189838 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b2d9d640 10 monclient(hunting): get_auth_request con 0x7fd7b40a4b60 auth_method 0 2023-10-30T12:19:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b259c640 10 monclient(hunting): get_auth_request con 0x7fd7b4132be0 auth_method 0 2023-10-30T12:19:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b359e640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b411e490 0x7fd7b411e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.519+0000 7fd7b259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b359e640 10 monclient(hunting): get_auth_request con 0x7fd7b411e490 auth_method 0 2023-10-30T12:19:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:29.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 10 monclient(hunting): handle_auth_done global_id 244140 payload 824 2023-10-30T12:19:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 1 -- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 msgr2=0x7fd7b41cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b41cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 1 -- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b411e490 msgr2=0x7fd7b411e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b411e490 0x7fd7b411e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:19:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 1 monclient: found mon.noname-a 2023-10-30T12:19:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 1 -- 172.21.15.50:0/352189838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac0094d0 con 0x7fd7b40a4b60 2023-10-30T12:19:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b259c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b41cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b2d9d640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7b40a4b60 0x7fd7b41ca8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00e9d0 tx=0x7fd7ac00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 client.?.objecter ms_handle_connect 0x7fd7b40a4b60 2023-10-30T12:19:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 1 -- 172.21.15.50:0/352189838 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac00ca60 con 0x7fd7b40a4b60 2023-10-30T12:19:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: dump: 2023-10-30T12:19:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.560 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.562 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: _finish_auth 0 2023-10-30T12:19:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:19:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:29.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 1 -- 172.21.15.50:0/352189838 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac00cbc0 con 0x7fd7b40a4b60 2023-10-30T12:19:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 5 monclient: authenticate success, global_id 244140 2023-10-30T12:19:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 10 monclient: _renew_subs 2023-10-30T12:19:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a4009a70 con 0x7fd7b40a4b60 2023-10-30T12:19:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 1 -- 172.21.15.50:0/352189838 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac013470 con 0x7fd7b40a4b60 2023-10-30T12:19:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7ab7fe640 10 monclient: dump: 2023-10-30T12:19:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:29.566 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b159a640 4 set_mon_vals no callback set 2023-10-30T12:19:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 10 client.244140.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 10 monclient: _renew_subs 2023-10-30T12:19:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b4121c60 con 0x7fd7b40a4b60 2023-10-30T12:19:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.520+0000 7fd7b917c640 1 librados: init done 2023-10-30T12:19:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7ab7fe640 1 -- 172.21.15.50:0/352189838 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd7ac0135d0 con 0x7fd7b40a4b60 2023-10-30T12:19:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7ab7fe640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd794039320 0x7fd79403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7b917c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7b917c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7b917c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7ab7fe640 1 -- 172.21.15.50:0/352189838 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(593..593 src has 490..593) v4 ==== 6650+0+0 (secure 0 0 0) 0x7fd7ac04b030 con 0x7fd7b40a4b60 2023-10-30T12:19:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b404e160 con 0x7fd7b40a4b60 2023-10-30T12:19:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7ab7fe640 10 client.244140.objecter ms_dispatch 0x7fd7b4000b90 osd_map(593..593 src has 490..593) v4 2023-10-30T12:19:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7ab7fe640 3 client.244140.objecter handle_osd_map got epochs [593,593] > 0 2023-10-30T12:19:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7ab7fe640 3 client.244140.objecter handle_osd_map decoding full epoch 593 2023-10-30T12:19:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7ab7fe640 20 client.244140.objecter dump_active .. 0 homeless 2023-10-30T12:19:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7b259c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd794039320 0x7fd79403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.521+0000 7fd7b259c640 10 monclient: get_auth_request con 0x7fd794039320 auth_method 0 2023-10-30T12:19:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.522+0000 7fd7ab7fe640 10 client.244140.objecter ms_handle_connect 0x7fd794039320 2023-10-30T12:19:29.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.522+0000 7fd7b259c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd794039320 0x7fd79403b7e0 secure :-1 s=READY pgs=4289 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4120d00 tx=0x7fd79c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.526+0000 7fd7ab7fe640 1 -- 172.21.15.50:0/352189838 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7ac017020 con 0x7fd7b40a4b60 2023-10-30T12:19:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.526+0000 7fd7ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.526+0000 7fd7ab7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:29.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.775+0000 7fd7b917c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:19:29.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.775+0000 7fd7b917c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:19:29.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.775+0000 7fd7b917c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:29.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.775+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd7b4122640 con 0x7fd7b40a4b60 2023-10-30T12:19:29.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.776+0000 7fd7ab7fe640 1 -- 172.21.15.50:0/352189838 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fd7b4122640 con 0x7fd7b40a4b60 2023-10-30T12:19:29.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.776+0000 7fd7ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:19:29.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.776+0000 7fd7ab7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:29.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.776+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.776+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=3721122613,config=3140514009,health=3347481906,kv=4169269627,logm=2895986403}) 2023-10-30T12:19:29.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.776+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=3721122612,config=3140514009,health=3347481906,kv=2905431493,logm=269444591}) 2023-10-30T12:19:29.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.776+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.776+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=3721122613,config=3140514009,health=3347481906,kv=4169269627,logm=2895986403}) 2023-10-30T12:19:29.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.776+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=10,kv=9,logm=53} crc {auth=1115238688,config=3140514009,health=1970216923,kv=2905431493,logm=3447052236}) 2023-10-30T12:19:29.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 10 librados: watch_flush enter 2023-10-30T12:19:29.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 10 librados: watch_flush exit 2023-10-30T12:19:29.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 20 client.244140.objecter shutdown clearing up homeless session... 2023-10-30T12:19:29.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 10 client.244140.objecter successfully canceled tick 2023-10-30T12:19:29.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd794039320 msgr2=0x7fd79403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd794039320 0x7fd79403b7e0 secure :-1 s=READY pgs=4289 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4120d00 tx=0x7fd79c009040 comp rx=0 tx=0).stop 2023-10-30T12:19:29.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 10 monclient: shutdown 2023-10-30T12:19:29.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:29.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7b40a4b60 msgr2=0x7fd7b41ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:29.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7b40a4b60 0x7fd7b41ca8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00e9d0 tx=0x7fd7ac00ee90 comp rx=0 tx=0).stop 2023-10-30T12:19:29.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 shutdown_connections 2023-10-30T12:19:29.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd794039320 0x7fd79403b7e0 unknown :-1 s=CLOSED pgs=4289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.783 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.778+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.783 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.778+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1862911608}) 2023-10-30T12:19:29.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7b411e490 0x7fd7b411e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7b4132be0 0x7fd7b41cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 --2- 172.21.15.50:0/352189838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7b40a4b60 0x7fd7b41ca8c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:29.785 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.778+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4267191929}) 2023-10-30T12:19:29.785 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.778+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.785 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.778+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1862911608}) 2023-10-30T12:19:29.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 >> 172.21.15.50:0/352189838 conn(0x7fd7b40a8060 msgr2=0x7fd7b41332b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:29.786 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.778+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=583395966}) 2023-10-30T12:19:29.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 shutdown_connections 2023-10-30T12:19:29.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 -- 172.21.15.50:0/352189838 wait complete. 2023-10-30T12:19:29.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:29.778+0000 7fd7b917c640 1 librados: shutdown 2023-10-30T12:19:29.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.779+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.779+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005099618}) 2023-10-30T12:19:29.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.779+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=724015049}) 2023-10-30T12:19:29.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.779+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.779+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005099618}) 2023-10-30T12:19:29.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.779+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232682392}) 2023-10-30T12:19:29.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.781+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.781+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1306191869}) 2023-10-30T12:19:29.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.781+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1751215668}) 2023-10-30T12:19:29.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.781+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.781+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1306191869}) 2023-10-30T12:19:29.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.781+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=520227857}) 2023-10-30T12:19:29.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.782+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.782+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594774881}) 2023-10-30T12:19:29.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.782+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3446281770}) 2023-10-30T12:19:29.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.782+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.782+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594774881}) 2023-10-30T12:19:29.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.782+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700978704}) 2023-10-30T12:19:29.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.783+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.783+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143408639}) 2023-10-30T12:19:29.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.783+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2902148297}) 2023-10-30T12:19:29.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.783+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.783+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143408639}) 2023-10-30T12:19:29.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.783+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972671900}) 2023-10-30T12:19:29.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.784+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.784+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008565092}) 2023-10-30T12:19:29.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.784+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616948533}) 2023-10-30T12:19:29.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.784+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.784+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008565092}) 2023-10-30T12:19:29.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.784+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898742463}) 2023-10-30T12:19:29.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.784+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817942731}) 2023-10-30T12:19:29.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1547224097}) 2023-10-30T12:19:29.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817942731}) 2023-10-30T12:19:29.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045931006}) 2023-10-30T12:19:29.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787069934}) 2023-10-30T12:19:29.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=316532973}) 2023-10-30T12:19:29.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787069934}) 2023-10-30T12:19:29.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.785+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3858390781}) 2023-10-30T12:19:29.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.786+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.786+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273255324}) 2023-10-30T12:19:29.800 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.786+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374176876}) 2023-10-30T12:19:29.800 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.786+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.800 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.786+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273255324}) 2023-10-30T12:19:29.800 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.786+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282028428}) 2023-10-30T12:19:29.800 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.787+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.801 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.787+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13931978}) 2023-10-30T12:19:29.801 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.787+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71550858}) 2023-10-30T12:19:29.801 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.787+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.787+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13931978}) 2023-10-30T12:19:29.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.787+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4144222055}) 2023-10-30T12:19:29.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.788+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.788+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778433797}) 2023-10-30T12:19:29.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.788+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939604070}) 2023-10-30T12:19:29.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.788+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.788+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778433797}) 2023-10-30T12:19:29.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.788+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501333047}) 2023-10-30T12:19:29.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.789+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.789+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333856740}) 2023-10-30T12:19:29.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.789+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1683761416}) 2023-10-30T12:19:29.804 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.789+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.804 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.789+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333856740}) 2023-10-30T12:19:29.804 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.789+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2869970098}) 2023-10-30T12:19:29.805 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:19:29.828 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.828 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511003445}) 2023-10-30T12:19:29.828 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113127667}) 2023-10-30T12:19:29.829 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.829 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511003445}) 2023-10-30T12:19:29.829 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154281405}) 2023-10-30T12:19:29.829 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.829 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2544718485}) 2023-10-30T12:19:29.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=624066950}) 2023-10-30T12:19:29.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2544718485}) 2023-10-30T12:19:29.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.790+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883878611}) 2023-10-30T12:19:29.830 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.791+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.791+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187075003}) 2023-10-30T12:19:29.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.791+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606354148}) 2023-10-30T12:19:29.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.791+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.791+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187075003}) 2023-10-30T12:19:29.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.791+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992199857}) 2023-10-30T12:19:29.831 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.792+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.792+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858584579}) 2023-10-30T12:19:29.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.792+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3819638136}) 2023-10-30T12:19:29.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.792+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.792+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858584579}) 2023-10-30T12:19:29.832 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.792+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140672765}) 2023-10-30T12:19:29.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.793+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.793+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1125682576}) 2023-10-30T12:19:29.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.793+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2471109812}) 2023-10-30T12:19:29.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.793+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.793+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1125682576}) 2023-10-30T12:19:29.833 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.793+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465299284}) 2023-10-30T12:19:29.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445222077}) 2023-10-30T12:19:29.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579698470}) 2023-10-30T12:19:29.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.834 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445222077}) 2023-10-30T12:19:29.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2105309677}) 2023-10-30T12:19:29.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652813693}) 2023-10-30T12:19:29.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148040738}) 2023-10-30T12:19:29.835 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652813693}) 2023-10-30T12:19:29.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.794+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885003840}) 2023-10-30T12:19:29.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.795+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.795+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419225396}) 2023-10-30T12:19:29.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.795+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3183588505}) 2023-10-30T12:19:29.836 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.795+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.795+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419225396}) 2023-10-30T12:19:29.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.795+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351861650}) 2023-10-30T12:19:29.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.796+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.796+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490521463}) 2023-10-30T12:19:29.837 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.796+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596260805}) 2023-10-30T12:19:29.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.796+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.796+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490521463}) 2023-10-30T12:19:29.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.796+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3870890235}) 2023-10-30T12:19:29.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.797+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.797+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459462013}) 2023-10-30T12:19:29.838 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.797+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727334979}) 2023-10-30T12:19:29.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.797+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.797+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459462013}) 2023-10-30T12:19:29.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.797+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182077185}) 2023-10-30T12:19:29.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.839 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=464669156}) 2023-10-30T12:19:29.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4041313518}) 2023-10-30T12:19:29.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=464669156}) 2023-10-30T12:19:29.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=381584768}) 2023-10-30T12:19:29.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34125335}) 2023-10-30T12:19:29.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1511232324}) 2023-10-30T12:19:29.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34125335}) 2023-10-30T12:19:29.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.798+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=423727845}) 2023-10-30T12:19:29.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.799+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.799+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042377258}) 2023-10-30T12:19:29.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.799+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957276821}) 2023-10-30T12:19:29.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.799+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.799+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042377258}) 2023-10-30T12:19:29.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.799+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119973207}) 2023-10-30T12:19:29.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.800+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.800+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662535765}) 2023-10-30T12:19:29.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.800+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2054868864}) 2023-10-30T12:19:29.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.800+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.800+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662535765}) 2023-10-30T12:19:29.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.800+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154739682}) 2023-10-30T12:19:29.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.801+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.801+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229695392}) 2023-10-30T12:19:29.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.801+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936827030}) 2023-10-30T12:19:29.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.801+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.801+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229695392}) 2023-10-30T12:19:29.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.801+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3878725600}) 2023-10-30T12:19:29.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.802+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.802+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134711474}) 2023-10-30T12:19:29.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.802+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136770301}) 2023-10-30T12:19:29.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.802+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.802+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134711474}) 2023-10-30T12:19:29.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.802+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777023739}) 2023-10-30T12:19:29.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.803+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.803+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146621760}) 2023-10-30T12:19:29.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.803+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677347651}) 2023-10-30T12:19:29.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.803+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.803+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146621760}) 2023-10-30T12:19:29.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.803+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3596239189}) 2023-10-30T12:19:29.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.804+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.804+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163098747}) 2023-10-30T12:19:29.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.804+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849722572}) 2023-10-30T12:19:29.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.804+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.804+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163098747}) 2023-10-30T12:19:29.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.804+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183724095}) 2023-10-30T12:19:29.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614665229}) 2023-10-30T12:19:29.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=255662117}) 2023-10-30T12:19:29.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614665229}) 2023-10-30T12:19:29.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854238408}) 2023-10-30T12:19:29.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134138766}) 2023-10-30T12:19:29.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=738269328}) 2023-10-30T12:19:29.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134138766}) 2023-10-30T12:19:29.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.805+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4099531057}) 2023-10-30T12:19:29.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.806+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.806+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188057086}) 2023-10-30T12:19:29.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.806+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3424046952}) 2023-10-30T12:19:29.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.806+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.806+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188057086}) 2023-10-30T12:19:29.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.806+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472923084}) 2023-10-30T12:19:29.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.807+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.807+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001870322}) 2023-10-30T12:19:29.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.807+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1330498783}) 2023-10-30T12:19:29.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.807+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.807+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2001870322}) 2023-10-30T12:19:29.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.807+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2578226493}) 2023-10-30T12:19:29.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806001230}) 2023-10-30T12:19:29.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=559095581}) 2023-10-30T12:19:29.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806001230}) 2023-10-30T12:19:29.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845623317}) 2023-10-30T12:19:29.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354258898}) 2023-10-30T12:19:29.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960312176}) 2023-10-30T12:19:29.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354258898}) 2023-10-30T12:19:29.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.808+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2005051487}) 2023-10-30T12:19:29.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.809+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.809+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382081143}) 2023-10-30T12:19:29.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.809+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227074125}) 2023-10-30T12:19:29.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.809+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.809+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382081143}) 2023-10-30T12:19:29.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.809+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224571957}) 2023-10-30T12:19:29.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.810+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.810+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934538045}) 2023-10-30T12:19:29.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.810+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1689791735}) 2023-10-30T12:19:29.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.810+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.810+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934538045}) 2023-10-30T12:19:29.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.810+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797268034}) 2023-10-30T12:19:29.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.811+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.811+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833845351}) 2023-10-30T12:19:29.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.811+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582505893}) 2023-10-30T12:19:29.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.811+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.811+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833845351}) 2023-10-30T12:19:29.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.811+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052630311}) 2023-10-30T12:19:29.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982362711}) 2023-10-30T12:19:29.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3223594108}) 2023-10-30T12:19:29.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982362711}) 2023-10-30T12:19:29.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013783205}) 2023-10-30T12:19:29.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956218840}) 2023-10-30T12:19:29.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190581490}) 2023-10-30T12:19:29.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.812+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956218840}) 2023-10-30T12:19:29.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3399791135}) 2023-10-30T12:19:29.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782362931}) 2023-10-30T12:19:29.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3291941310}) 2023-10-30T12:19:29.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782362931}) 2023-10-30T12:19:29.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.813+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794603533}) 2023-10-30T12:19:29.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.814+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.814+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247249933}) 2023-10-30T12:19:29.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.814+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638082678}) 2023-10-30T12:19:29.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.814+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.814+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247249933}) 2023-10-30T12:19:29.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.814+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945390173}) 2023-10-30T12:19:29.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3631132037}) 2023-10-30T12:19:29.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148994558}) 2023-10-30T12:19:29.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3631132037}) 2023-10-30T12:19:29.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039445855}) 2023-10-30T12:19:29.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068851918}) 2023-10-30T12:19:29.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=203339392}) 2023-10-30T12:19:29.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068851918}) 2023-10-30T12:19:29.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.815+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447746834}) 2023-10-30T12:19:29.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.816+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.816+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716059059}) 2023-10-30T12:19:29.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.816+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2180932630}) 2023-10-30T12:19:29.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.816+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.816+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716059059}) 2023-10-30T12:19:29.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.816+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894932590}) 2023-10-30T12:19:29.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.817+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.817+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497011968}) 2023-10-30T12:19:29.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.817+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=402932140}) 2023-10-30T12:19:29.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.817+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.817+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497011968}) 2023-10-30T12:19:29.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.817+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=700011070}) 2023-10-30T12:19:29.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264841408}) 2023-10-30T12:19:29.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807144133}) 2023-10-30T12:19:29.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264841408}) 2023-10-30T12:19:29.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2265183175}) 2023-10-30T12:19:29.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027294876}) 2023-10-30T12:19:29.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060174912}) 2023-10-30T12:19:29.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027294876}) 2023-10-30T12:19:29.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.818+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218271550}) 2023-10-30T12:19:29.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.819+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.819+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762380346}) 2023-10-30T12:19:29.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.819+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321629938}) 2023-10-30T12:19:29.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.819+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.819+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762380346}) 2023-10-30T12:19:29.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.819+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3709060171}) 2023-10-30T12:19:29.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16956668}) 2023-10-30T12:19:29.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1672719628}) 2023-10-30T12:19:29.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16956668}) 2023-10-30T12:19:29.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3694447780}) 2023-10-30T12:19:29.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482413409}) 2023-10-30T12:19:29.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.820+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717529999}) 2023-10-30T12:19:29.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482413409}) 2023-10-30T12:19:29.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020170316}) 2023-10-30T12:19:29.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162356456}) 2023-10-30T12:19:29.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379833783}) 2023-10-30T12:19:29.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162356456}) 2023-10-30T12:19:29.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.821+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472386114}) 2023-10-30T12:19:29.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.822+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.822+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232164318}) 2023-10-30T12:19:29.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.822+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2704192996}) 2023-10-30T12:19:29.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.822+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.822+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232164318}) 2023-10-30T12:19:29.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.822+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1995670896}) 2023-10-30T12:19:29.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244557808}) 2023-10-30T12:19:29.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2357741532}) 2023-10-30T12:19:29.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244557808}) 2023-10-30T12:19:29.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=153089587}) 2023-10-30T12:19:29.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117052132}) 2023-10-30T12:19:29.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3749405550}) 2023-10-30T12:19:29.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117052132}) 2023-10-30T12:19:29.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.823+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180504940}) 2023-10-30T12:19:29.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.824+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.824+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=978618567}) 2023-10-30T12:19:29.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.824+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062388751}) 2023-10-30T12:19:29.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.824+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.824+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=978618567}) 2023-10-30T12:19:29.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.824+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910718756}) 2023-10-30T12:19:29.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.825+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.825+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598985174}) 2023-10-30T12:19:29.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.825+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989667276}) 2023-10-30T12:19:29.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.825+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.825+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598985174}) 2023-10-30T12:19:29.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.825+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214460428}) 2023-10-30T12:19:29.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.825+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1015361445}) 2023-10-30T12:19:29.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880337979}) 2023-10-30T12:19:29.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1015361445}) 2023-10-30T12:19:29.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462829947}) 2023-10-30T12:19:29.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112918062}) 2023-10-30T12:19:29.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3920803325}) 2023-10-30T12:19:29.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112918062}) 2023-10-30T12:19:29.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.826+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745983294}) 2023-10-30T12:19:29.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.827+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.827+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884348712}) 2023-10-30T12:19:29.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.827+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3234020698}) 2023-10-30T12:19:29.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.827+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.827+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884348712}) 2023-10-30T12:19:29.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.827+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1804724795}) 2023-10-30T12:19:29.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150633682}) 2023-10-30T12:19:29.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232050346}) 2023-10-30T12:19:29.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150633682}) 2023-10-30T12:19:29.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3210160183}) 2023-10-30T12:19:29.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622426970}) 2023-10-30T12:19:29.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707424903}) 2023-10-30T12:19:29.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622426970}) 2023-10-30T12:19:29.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.828+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2817836105}) 2023-10-30T12:19:29.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.829+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.829+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391315641}) 2023-10-30T12:19:29.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.829+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1288449882}) 2023-10-30T12:19:29.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.829+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.829+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391315641}) 2023-10-30T12:19:29.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.829+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729875913}) 2023-10-30T12:19:29.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.830+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.830+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045810557}) 2023-10-30T12:19:29.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.830+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805532256}) 2023-10-30T12:19:29.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.830+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.830+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045810557}) 2023-10-30T12:19:29.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.830+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2250149197}) 2023-10-30T12:19:29.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799989261}) 2023-10-30T12:19:29.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011481186}) 2023-10-30T12:19:29.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799989261}) 2023-10-30T12:19:29.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1254512796}) 2023-10-30T12:19:29.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709593088}) 2023-10-30T12:19:29.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3760783113}) 2023-10-30T12:19:29.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709593088}) 2023-10-30T12:19:29.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.831+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179287505}) 2023-10-30T12:19:29.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.832+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.832+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722546746}) 2023-10-30T12:19:29.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.832+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3136841234}) 2023-10-30T12:19:29.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.832+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.832+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722546746}) 2023-10-30T12:19:29.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.832+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2848583380}) 2023-10-30T12:19:29.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.833+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.833+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786509109}) 2023-10-30T12:19:29.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.833+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1651765089}) 2023-10-30T12:19:29.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.833+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.833+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786509109}) 2023-10-30T12:19:29.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.833+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169284279}) 2023-10-30T12:19:29.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.833+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697609960}) 2023-10-30T12:19:29.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013789397}) 2023-10-30T12:19:29.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697609960}) 2023-10-30T12:19:29.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3453314442}) 2023-10-30T12:19:29.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412717377}) 2023-10-30T12:19:29.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=754788180}) 2023-10-30T12:19:29.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412717377}) 2023-10-30T12:19:29.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.834+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1568812572}) 2023-10-30T12:19:29.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.835+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.835+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296674979}) 2023-10-30T12:19:29.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.835+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917624169}) 2023-10-30T12:19:29.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.835+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.835+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296674979}) 2023-10-30T12:19:29.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.835+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873989895}) 2023-10-30T12:19:29.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798224257}) 2023-10-30T12:19:29.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423519419}) 2023-10-30T12:19:29.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798224257}) 2023-10-30T12:19:29.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3572244765}) 2023-10-30T12:19:29.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186337936}) 2023-10-30T12:19:29.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135440574}) 2023-10-30T12:19:29.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186337936}) 2023-10-30T12:19:29.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.836+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2875965703}) 2023-10-30T12:19:29.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.837+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.837+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511352928}) 2023-10-30T12:19:29.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.837+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=142352943}) 2023-10-30T12:19:29.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.837+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.837+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511352928}) 2023-10-30T12:19:29.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.837+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=862229270}) 2023-10-30T12:19:29.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.838+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.838+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=635521270}) 2023-10-30T12:19:29.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.838+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=547307514}) 2023-10-30T12:19:29.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.838+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.838+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=635521270}) 2023-10-30T12:19:29.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.838+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2408204461}) 2023-10-30T12:19:29.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.839+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.839+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568163174}) 2023-10-30T12:19:29.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.839+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1212985709}) 2023-10-30T12:19:29.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.839+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.839+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568163174}) 2023-10-30T12:19:29.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.839+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3939782594}) 2023-10-30T12:19:29.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.840+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.840+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494031382}) 2023-10-30T12:19:29.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.840+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3632131390}) 2023-10-30T12:19:29.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.840+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.840+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494031382}) 2023-10-30T12:19:29.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.840+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170516136}) 2023-10-30T12:19:29.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721845549}) 2023-10-30T12:19:29.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497312418}) 2023-10-30T12:19:29.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721845549}) 2023-10-30T12:19:29.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1297434426}) 2023-10-30T12:19:29.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271462052}) 2023-10-30T12:19:29.928 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2872482810}) 2023-10-30T12:19:29.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271462052}) 2023-10-30T12:19:29.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.841+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2850609720}) 2023-10-30T12:19:29.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.842+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.842+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3424001867}) 2023-10-30T12:19:29.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.842+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3575841164}) 2023-10-30T12:19:29.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.842+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.842+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3424001867}) 2023-10-30T12:19:29.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.842+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317441985}) 2023-10-30T12:19:29.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185760948}) 2023-10-30T12:19:29.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79440434}) 2023-10-30T12:19:29.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185760948}) 2023-10-30T12:19:29.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2347009682}) 2023-10-30T12:19:29.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342081210}) 2023-10-30T12:19:29.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=169574617}) 2023-10-30T12:19:29.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342081210}) 2023-10-30T12:19:29.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.843+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407055023}) 2023-10-30T12:19:29.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.844+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.844+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565920893}) 2023-10-30T12:19:29.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.844+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821251132}) 2023-10-30T12:19:29.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.844+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.844+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565920893}) 2023-10-30T12:19:29.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.844+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972414616}) 2023-10-30T12:19:29.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474890993}) 2023-10-30T12:19:29.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2879818043}) 2023-10-30T12:19:29.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474890993}) 2023-10-30T12:19:29.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038270489}) 2023-10-30T12:19:29.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057076838}) 2023-10-30T12:19:29.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=474589364}) 2023-10-30T12:19:29.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057076838}) 2023-10-30T12:19:29.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.845+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149915802}) 2023-10-30T12:19:29.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.846+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.846+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850973092}) 2023-10-30T12:19:29.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.846+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=103793831}) 2023-10-30T12:19:29.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.846+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.846+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850973092}) 2023-10-30T12:19:29.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.846+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100696213}) 2023-10-30T12:19:29.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839006191}) 2023-10-30T12:19:29.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2912606674}) 2023-10-30T12:19:29.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839006191}) 2023-10-30T12:19:29.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3961139070}) 2023-10-30T12:19:29.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142985950}) 2023-10-30T12:19:29.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459852538}) 2023-10-30T12:19:29.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142985950}) 2023-10-30T12:19:29.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.847+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4117540339}) 2023-10-30T12:19:29.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.848+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.848+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472222238}) 2023-10-30T12:19:29.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.848+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1906753351}) 2023-10-30T12:19:29.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.848+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.848+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472222238}) 2023-10-30T12:19:29.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.848+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995176005}) 2023-10-30T12:19:29.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.849+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.849+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717253147}) 2023-10-30T12:19:29.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.849+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939514989}) 2023-10-30T12:19:29.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.849+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.849+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717253147}) 2023-10-30T12:19:29.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.849+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51736015}) 2023-10-30T12:19:29.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2203389703}) 2023-10-30T12:19:29.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537863284}) 2023-10-30T12:19:29.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2203389703}) 2023-10-30T12:19:29.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137859393}) 2023-10-30T12:19:29.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029243408}) 2023-10-30T12:19:29.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2226835517}) 2023-10-30T12:19:29.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029243408}) 2023-10-30T12:19:29.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.850+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575903072}) 2023-10-30T12:19:29.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.851+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.851+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48750483}) 2023-10-30T12:19:29.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.851+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3265666793}) 2023-10-30T12:19:29.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.851+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.851+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48750483}) 2023-10-30T12:19:29.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.851+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903416677}) 2023-10-30T12:19:29.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875389898}) 2023-10-30T12:19:29.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35561045}) 2023-10-30T12:19:29.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875389898}) 2023-10-30T12:19:29.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4263836642}) 2023-10-30T12:19:29.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542475823}) 2023-10-30T12:19:29.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=692349008}) 2023-10-30T12:19:29.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542475823}) 2023-10-30T12:19:29.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.852+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=980982827}) 2023-10-30T12:19:29.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.853+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.853+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3729808138}) 2023-10-30T12:19:29.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.853+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120039356}) 2023-10-30T12:19:29.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.853+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.853+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3729808138}) 2023-10-30T12:19:29.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.853+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704701355}) 2023-10-30T12:19:29.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.854+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.854+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776428320}) 2023-10-30T12:19:29.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.854+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314900460}) 2023-10-30T12:19:29.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.854+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.854+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776428320}) 2023-10-30T12:19:29.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.854+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53882168}) 2023-10-30T12:19:29.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.854+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387733897}) 2023-10-30T12:19:29.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1898342532}) 2023-10-30T12:19:29.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387733897}) 2023-10-30T12:19:29.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=22974486}) 2023-10-30T12:19:29.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445260789}) 2023-10-30T12:19:29.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2704889797}) 2023-10-30T12:19:29.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445260789}) 2023-10-30T12:19:29.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.855+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=557094327}) 2023-10-30T12:19:29.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.856+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.856+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889144796}) 2023-10-30T12:19:29.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.856+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4167335882}) 2023-10-30T12:19:29.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.856+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.856+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889144796}) 2023-10-30T12:19:29.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.856+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644923}) 2023-10-30T12:19:29.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922360781}) 2023-10-30T12:19:29.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2381042543}) 2023-10-30T12:19:29.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922360781}) 2023-10-30T12:19:29.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1811378633}) 2023-10-30T12:19:29.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854702065}) 2023-10-30T12:19:29.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=543655670}) 2023-10-30T12:19:29.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854702065}) 2023-10-30T12:19:29.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.857+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1961732923}) 2023-10-30T12:19:29.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.858+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.858+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039937405}) 2023-10-30T12:19:29.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.858+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1068326053}) 2023-10-30T12:19:29.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.858+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.858+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039937405}) 2023-10-30T12:19:29.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.858+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3956014459}) 2023-10-30T12:19:29.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716858115}) 2023-10-30T12:19:29.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226415481}) 2023-10-30T12:19:29.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716858115}) 2023-10-30T12:19:29.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2204892281}) 2023-10-30T12:19:29.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776069685}) 2023-10-30T12:19:29.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3212418652}) 2023-10-30T12:19:29.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776069685}) 2023-10-30T12:19:29.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.859+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3292880946}) 2023-10-30T12:19:29.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.860+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.860+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704371664}) 2023-10-30T12:19:29.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.860+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3641922128}) 2023-10-30T12:19:29.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.860+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.860+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704371664}) 2023-10-30T12:19:29.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.860+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076331751}) 2023-10-30T12:19:29.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.861+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.861+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113912293}) 2023-10-30T12:19:29.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.861+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192185310}) 2023-10-30T12:19:29.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.861+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.861+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113912293}) 2023-10-30T12:19:29.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.861+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542164591}) 2023-10-30T12:19:29.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.862+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.862+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=512578672,mdsmap=3984425268,mgr=1598784600,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3990293227,mon_config_key=715115367,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=232572148,osd_snap=740260791}) 2023-10-30T12:19:29.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.862+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=17} crc {logm=1005936054,mdsmap=2806139572,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=917829455,mon_config_key=715115367,monmap=3140514009,osd_metadata=451695420,osd_pg_creating=232572148,osd_snap=3084502386}) 2023-10-30T12:19:29.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.862+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.862+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=512578672,mdsmap=3984425268,mgr=1598784600,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3990293227,mon_config_key=715115367,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=232572148,osd_snap=740260791}) 2023-10-30T12:19:29.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.862+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=1606097713,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=405837660,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=232572148,osd_snap=2172385760}) 2023-10-30T12:19:29.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.863+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.863+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=1427006927,osdmap=1783640469}) 2023-10-30T12:19:29.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.863+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=1486108288,osdmap=1599561258}) 2023-10-30T12:19:29.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.863+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.863+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=1427006927,osdmap=1783640469}) 2023-10-30T12:19:29.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.863+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=2047403190,osdmap=4122546241}) 2023-10-30T12:19:29.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.869+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.869+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=87} crc {osdmap=3850550868}) 2023-10-30T12:19:29.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.869+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=88} crc {osdmap=1815217104}) 2023-10-30T12:19:29.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.869+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:19:29.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.869+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=87} crc {osdmap=3850550868}) 2023-10-30T12:19:29.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:19:29.869+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=85} crc {osdmap=568010620}) 2023-10-30T12:19:30.807 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:19:30.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.922+0000 7fa11e832640 10 monclient: get_monmap_and_config 2023-10-30T12:19:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.922+0000 7fa11e832640 10 monclient: build_initial_monmap 2023-10-30T12:19:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.922+0000 7fa11e832640 10 monclient: monmap: 2023-10-30T12:19:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:30.923671+0000 2023-10-30T12:19:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:30.923671+0000 2023-10-30T12:19:30.926 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:30.928 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:30.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 1 Processor -- start 2023-10-30T12:19:30.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 1 -- start start 2023-10-30T12:19:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 10 monclient: init 2023-10-30T12:19:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:19:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa118098520 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:30.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa118099010 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa11809bf20 0x7fa11813b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 10 monclient(hunting): picked mon.noname-b con 0x7fa11809bf20 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.923+0000 7fa11e832640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11809ad30 con 0x7fa118098520 2023-10-30T12:19:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11e832640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11809b260 con 0x7fa118099010 2023-10-30T12:19:30.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11e832640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11e832640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa118072120 con 0x7fa11809bf20 2023-10-30T12:19:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11e832640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11e832640 10 monclient(hunting): authenticate will time out at 2016.107075s 2023-10-30T12:19:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa117fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa118098520 0x7fa118098900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40434/0 (socket says 172.21.15.50:40434) 2023-10-30T12:19:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa117fff640 1 -- 172.21.15.50:0/2839544255 learned_addr learned my addr 172.21.15.50:0/2839544255 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:30.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 1 --2- 172.21.15.50:0/2839544255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa11809bf20 0x7fa11813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): get_auth_request con 0x7fa11809bf20 auth_method 0 2023-10-30T12:19:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:30.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa116ffd640 1 -- 172.21.15.50:0/2839544255 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732908600 0 0) 0x7fa11809ad30 con 0x7fa118098520 2023-10-30T12:19:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa116ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:19:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa116ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:30.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa116ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:30.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa116ffd640 10 monclient(hunting): my global_id is 244143 2023-10-30T12:19:30.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa116ffd640 1 -- 172.21.15.50:0/2839544255 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa100003520 con 0x7fa118098520 2023-10-30T12:19:30.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient(hunting): handle_auth_done global_id 224315 payload 274 2023-10-30T12:19:30.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 1 -- 172.21.15.50:0/2839544255 >> v1:172.21.15.118:6789/0 conn(0x7fa118099010 legacy=0x7fa1181379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:30.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 1 -- 172.21.15.50:0/2839544255 >> v1:172.21.15.50:6789/0 conn(0x7fa118098520 legacy=0x7fa118098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:30.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient: _finish_hunting 0 2023-10-30T12:19:30.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 1 monclient: found mon.noname-b 2023-10-30T12:19:30.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:30.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:30.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.924+0000 7fa11cdcf640 1 -- 172.21.15.50:0/2839544255 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa118099610 con 0x7fa11809bf20 2023-10-30T12:19:30.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11cdcf640 1 --2- 172.21.15.50:0/2839544255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa11809bf20 0x7fa11813b120 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa110009dc0 tx=0x7fa11002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e2e1df832abe87a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:30.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 1 -- 172.21.15.50:0/2839544255 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11002cbb0 con 0x7fa11809bf20 2023-10-30T12:19:30.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:30.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:30.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: dump: 2023-10-30T12:19:30.946 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:30.946 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:30.947 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.947 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.947 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:30.947 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:30.947 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:30.947 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:30.948 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:30.948 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:30.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: _finish_auth 0 2023-10-30T12:19:30.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:19:30.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:30.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 1 -- 172.21.15.50:0/2839544255 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11002cd10 con 0x7fa11809bf20 2023-10-30T12:19:30.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11e832640 5 monclient: authenticate success, global_id 224315 2023-10-30T12:19:30.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11e832640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:30.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:30.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11e832640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:30.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 1 -- 172.21.15.50:0/2839544255 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110034660 con 0x7fa11809bf20 2023-10-30T12:19:30.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:30.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11e832640 4 set_mon_vals no callback set 2023-10-30T12:19:30.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:30.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa116ffd640 10 monclient: dump: 2023-10-30T12:19:30.950 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:30.951 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:30.951 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.951 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.951 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:30.951 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:30.952 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:30.952 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:30.952 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:30.952 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:30.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11e832640 10 monclient: shutdown 2023-10-30T12:19:30.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11e832640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:30.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.925+0000 7fa11e832640 1 -- 172.21.15.50:0/2839544255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa11809bf20 msgr2=0x7fa11813b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:30.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 --2- 172.21.15.50:0/2839544255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa11809bf20 0x7fa11813b120 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa110009dc0 tx=0x7fa11002be00 comp rx=0 tx=0).stop 2023-10-30T12:19:30.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 -- 172.21.15.50:0/2839544255 shutdown_connections 2023-10-30T12:19:30.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 --2- 172.21.15.50:0/2839544255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa11809bf20 0x7fa11813b120 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:30.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 -- 172.21.15.50:0/2839544255 >> 172.21.15.50:0/2839544255 conn(0x7fa1180a05c0 msgr2=0x7fa11809de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:30.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 -- 172.21.15.50:0/2839544255 shutdown_connections 2023-10-30T12:19:30.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 -- 172.21.15.50:0/2839544255 wait complete. 2023-10-30T12:19:30.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 10 monclient: build_initial_monmap 2023-10-30T12:19:30.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 10 monclient: monmap: 2023-10-30T12:19:30.955 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:30.955 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:30.955 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:30.955 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:30.955 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:30.956 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:30.956 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:30.956 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:30.956 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:30.956 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:30.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 librados: starting msgr at 2023-10-30T12:19:30.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.926+0000 7fa11e832640 1 librados: starting objecter 2023-10-30T12:19:30.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 Processor -- start 2023-10-30T12:19:30.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 -- start start 2023-10-30T12:19:30.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 librados: setting wanted keys 2023-10-30T12:19:30.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 librados: calling monclient init 2023-10-30T12:19:30.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient: init 2023-10-30T12:19:30.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:30.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:30.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa118098520 0x7fa1181c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:30.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): picked mon.noname-a con 0x7fa118098520 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:30.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa118099010 0x7fa1181b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:30.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): picked mon.noname-c con 0x7fa118099010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:30.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa11809bf20 0x7fa1181b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:30.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): picked mon.noname-b con 0x7fa11809bf20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:30.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:30.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181b19d0 con 0x7fa118098520 2023-10-30T12:19:30.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:30.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181b1cb0 con 0x7fa11809bf20 2023-10-30T12:19:30.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:30.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181b1f90 con 0x7fa118099010 2023-10-30T12:19:30.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:30.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa11e832640 10 monclient(hunting): authenticate will time out at 2016.110730s 2023-10-30T12:19:30.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa118098520 0x7fa1181c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:30.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa118099010 0x7fa1181b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:30.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa118098520 0x7fa1181c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48732/0 (socket says 172.21.15.50:48732) 2023-10-30T12:19:30.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 1 -- 172.21.15.50:0/756452755 learned_addr learned my addr 172.21.15.50:0/756452755 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:30.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 10 monclient(hunting): get_auth_request con 0x7fa118098520 auth_method 0 2023-10-30T12:19:30.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:30.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:30.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:30.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 10 monclient(hunting): get_auth_request con 0x7fa118099010 auth_method 0 2023-10-30T12:19:30.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:30.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:30.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:30.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:30.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:30.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa117fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:30.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:30.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:30.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.927+0000 7fa1177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:30.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11cdcf640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa11809bf20 0x7fa1181b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:30.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11cdcf640 10 monclient(hunting): get_auth_request con 0x7fa11809bf20 auth_method 0 2023-10-30T12:19:30.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11cdcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:30.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11cdcf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:30.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11cdcf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:30.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 10 monclient(hunting): handle_auth_done global_id 244146 payload 824 2023-10-30T12:19:30.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 1 -- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa118099010 msgr2=0x7fa1181b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:30.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa118099010 0x7fa1181b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:30.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 1 -- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa11809bf20 msgr2=0x7fa1181b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:30.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa11809bf20 0x7fa1181b7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:30.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 10 monclient: _finish_hunting 0 2023-10-30T12:19:30.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 1 monclient: found mon.noname-a 2023-10-30T12:19:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 1 -- 172.21.15.50:0/756452755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa110009a70 con 0x7fa118098520 2023-10-30T12:19:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa1177fe640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa118099010 0x7fa1181b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11cdcf640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa11809bf20 0x7fa1181b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:19:30.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa117fff640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa118098520 0x7fa1181c6d40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa10c001260 tx=0x7fa10c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 client.?.objecter ms_handle_connect 0x7fa118098520 2023-10-30T12:19:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 1 -- 172.21.15.50:0/756452755 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c007480 con 0x7fa118098520 2023-10-30T12:19:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: dump: 2023-10-30T12:19:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:30.970 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:30.970 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:30.970 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:30.970 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:30.970 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:30.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: _finish_auth 0 2023-10-30T12:19:30.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:19:30.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:30.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 1 -- 172.21.15.50:0/756452755 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c0075e0 con 0x7fa118098520 2023-10-30T12:19:30.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 5 monclient: authenticate success, global_id 244146 2023-10-30T12:19:30.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 10 monclient: _renew_subs 2023-10-30T12:19:30.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:30.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1181b2270 con 0x7fa118098520 2023-10-30T12:19:30.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:30.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 1 -- 172.21.15.50:0/756452755 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00ca70 con 0x7fa118098520 2023-10-30T12:19:30.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:30.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:30.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa114ff9640 10 monclient: dump: 2023-10-30T12:19:30.973 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:30.973 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:30.973 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.974 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:30.974 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:30.974 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:30.974 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:30.974 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:30.974 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:30.975 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:30.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 10 client.244146.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:30.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 10 monclient: _renew_subs 2023-10-30T12:19:30.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa1167fc640 4 set_mon_vals no callback set 2023-10-30T12:19:30.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:30.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1181b8260 con 0x7fa118098520 2023-10-30T12:19:30.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.928+0000 7fa11e832640 1 librados: init done 2023-10-30T12:19:30.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa114ff9640 1 -- 172.21.15.50:0/756452755 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa10c00cbd0 con 0x7fa118098520 2023-10-30T12:19:30.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa114ff9640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1040392f0 0x7fa10403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:30.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa114ff9640 1 -- 172.21.15.50:0/756452755 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(595..595 src has 490..595) v4 ==== 6762+0+0 (secure 0 0 0) 0x7fa10c04d530 con 0x7fa118098520 2023-10-30T12:19:30.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa114ff9640 10 client.244146.objecter ms_dispatch 0x7fa118000b90 osd_map(595..595 src has 490..595) v4 2023-10-30T12:19:30.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa114ff9640 3 client.244146.objecter handle_osd_map got epochs [595,595] > 0 2023-10-30T12:19:30.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa114ff9640 3 client.244146.objecter handle_osd_map decoding full epoch 595 2023-10-30T12:19:30.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa11e832640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:30.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa11e832640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:30.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa11e832640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:30.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11804e160 con 0x7fa118098520 2023-10-30T12:19:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa114ff9640 20 client.244146.objecter dump_active .. 0 homeless 2023-10-30T12:19:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa1177fe640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1040392f0 0x7fa10403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.929+0000 7fa1177fe640 10 monclient: get_auth_request con 0x7fa1040392f0 auth_method 0 2023-10-30T12:19:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.930+0000 7fa114ff9640 10 client.244146.objecter ms_handle_connect 0x7fa1040392f0 2023-10-30T12:19:30.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.930+0000 7fa1177fe640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1040392f0 0x7fa10403b7b0 secure :-1 s=READY pgs=4290 cs=0 l=1 rev1=1 crypto rx=0x7fa1181afd20 tx=0x7fa108009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.934+0000 7fa114ff9640 1 -- 172.21.15.50:0/756452755 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa10c018770 con 0x7fa118098520 2023-10-30T12:19:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.934+0000 7fa114ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:30.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:30.934+0000 7fa114ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:31.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.180+0000 7fa11e832640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:31.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.180+0000 7fa11e832640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:31.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.180+0000 7fa11e832640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.180+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa1180745c0 con 0x7fa118098520 2023-10-30T12:19:31.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.180+0000 7fa114ff9640 1 -- 172.21.15.50:0/756452755 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fa10c03e080 con 0x7fa118098520 2023-10-30T12:19:31.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.180+0000 7fa114ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:31.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.180+0000 7fa114ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:31.184 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":198,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:31.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 10 librados: watch_flush enter 2023-10-30T12:19:31.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 10 librados: watch_flush exit 2023-10-30T12:19:31.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 20 client.244146.objecter shutdown clearing up homeless session... 2023-10-30T12:19:31.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 10 client.244146.objecter successfully canceled tick 2023-10-30T12:19:31.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1040392f0 msgr2=0x7fa10403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1040392f0 0x7fa10403b7b0 secure :-1 s=READY pgs=4290 cs=0 l=1 rev1=1 crypto rx=0x7fa1181afd20 tx=0x7fa108009040 comp rx=0 tx=0).stop 2023-10-30T12:19:31.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 10 monclient: shutdown 2023-10-30T12:19:31.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:31.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa118098520 msgr2=0x7fa1181c6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa118098520 0x7fa1181c6d40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa10c001260 tx=0x7fa10c00eda0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 shutdown_connections 2023-10-30T12:19:31.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa1040392f0 0x7fa10403b7b0 unknown :-1 s=CLOSED pgs=4290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa11809bf20 0x7fa1181b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa118099010 0x7fa1181b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 --2- 172.21.15.50:0/756452755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa118098520 0x7fa1181c6d40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 >> 172.21.15.50:0/756452755 conn(0x7fa1180a05c0 msgr2=0x7fa1181b9910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:31.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 shutdown_connections 2023-10-30T12:19:31.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 -- 172.21.15.50:0/756452755 wait complete. 2023-10-30T12:19:31.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.182+0000 7fa11e832640 1 librados: shutdown 2023-10-30T12:19:31.194 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:19:31.194 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:19:31.194 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:19:31.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:19:31.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.299+0000 7fcc83852640 10 monclient: get_monmap_and_config 2023-10-30T12:19:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.299+0000 7fcc83852640 10 monclient: build_initial_monmap 2023-10-30T12:19:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.299+0000 7fcc83852640 10 monclient: monmap: 2023-10-30T12:19:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:31.300318+0000 2023-10-30T12:19:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:31.300318+0000 2023-10-30T12:19:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:31.303 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:31.303 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:31.303 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:31.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.299+0000 7fcc83852640 1 Processor -- start 2023-10-30T12:19:31.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 1 -- start start 2023-10-30T12:19:31.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient: init 2023-10-30T12:19:31.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:31.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-10-30T12:19:31.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c0a2ae0 0x7fcc7c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient(hunting): picked mon.noname-b con 0x7fcc7c0a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:31.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc7c0a2ec0 0x7fcc7c1356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7c0a2ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:31.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc7c0a32e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:31.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0a4190 con 0x7fcc7c0a32e0 2023-10-30T12:19:31.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c0a2990 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0a1630 con 0x7fcc7c0a2ae0 2023-10-30T12:19:31.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:31.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc83852640 10 monclient(hunting): authenticate will time out at 2016.483544s 2023-10-30T12:19:31.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c0a2ae0 0x7fcc7c0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc7c0a2ec0 0x7fcc7c1356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c0a2ae0 0x7fcc7c0a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33866/0 (socket says 172.21.15.50:33866) 2023-10-30T12:19:31.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc7c0a2ec0 0x7fcc7c1356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48738/0 (socket says 172.21.15.50:48738) 2023-10-30T12:19:31.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 1 -- 172.21.15.50:0/2142116567 learned_addr learned my addr 172.21.15.50:0/2142116567 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:31.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 10 monclient(hunting): get_auth_request con 0x7fcc7c0a2ae0 auth_method 0 2023-10-30T12:19:31.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): get_auth_request con 0x7fcc7c0a2ec0 auth_method 0 2023-10-30T12:19:31.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 10 monclient(hunting): handle_auth_done global_id 244149 payload 274 2023-10-30T12:19:31.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 1 -- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c0a2ae0 msgr2=0x7fcc7c0a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:31.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.300+0000 7fcc80ded640 1 --2- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c0a2ae0 0x7fcc7c0a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc80ded640 1 -- 172.21.15.50:0/2142116567 >> v1:172.21.15.50:6789/0 conn(0x7fcc7c0a32e0 legacy=0x7fcc7c135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc80ded640 10 monclient: _finish_hunting 0 2023-10-30T12:19:31.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc80ded640 1 monclient: found mon.noname-a 2023-10-30T12:19:31.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc80ded640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:31.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc80ded640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc80ded640 1 -- 172.21.15.50:0/2142116567 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c09a2e0 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc815ee640 1 --2- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c0a2ae0 0x7fcc7c0a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:31.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc80ded640 1 --2- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c1356e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc700097a0 tx=0x7fcc7002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df4c9490bc44322 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 1 -- 172.21.15.50:0/2142116567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc700316f0 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:31.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: dump: 2023-10-30T12:19:31.322 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.322 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.322 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.322 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.322 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.322 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.323 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.323 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.323 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.323 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: _finish_auth 0 2023-10-30T12:19:31.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:31.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:31.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 1 -- 172.21.15.50:0/2142116567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc70031850 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc83852640 5 monclient: authenticate success, global_id 244149 2023-10-30T12:19:31.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc83852640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:31.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:31.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc83852640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:31.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 1 -- 172.21.15.50:0/2142116567 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70031be0 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc83852640 4 set_mon_vals no callback set 2023-10-30T12:19:31.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:31.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.301+0000 7fcc7bfff640 10 monclient: dump: 2023-10-30T12:19:31.326 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.326 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.326 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.326 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.327 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.327 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.327 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.327 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.327 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.328 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 10 monclient: shutdown 2023-10-30T12:19:31.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:31.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 -- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 msgr2=0x7fcc7c1356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 --2- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c1356e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc700097a0 tx=0x7fcc7002b790 comp rx=0 tx=0).stop 2023-10-30T12:19:31.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 -- 172.21.15.50:0/2142116567 shutdown_connections 2023-10-30T12:19:31.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 --2- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c1356e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 --2- 172.21.15.50:0/2142116567 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c0a2ae0 0x7fcc7c0a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 -- 172.21.15.50:0/2142116567 >> 172.21.15.50:0/2142116567 conn(0x7fcc7c0a80b0 msgr2=0x7fcc7c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:31.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 -- 172.21.15.50:0/2142116567 shutdown_connections 2023-10-30T12:19:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.302+0000 7fcc83852640 1 -- 172.21.15.50:0/2142116567 wait complete. 2023-10-30T12:19:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient: build_initial_monmap 2023-10-30T12:19:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient: monmap: 2023-10-30T12:19:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:31.331 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:31.332 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.333 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:31.333 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:31.333 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:31.333 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 librados: starting msgr at 2023-10-30T12:19:31.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 librados: starting objecter 2023-10-30T12:19:31.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 Processor -- start 2023-10-30T12:19:31.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 -- start start 2023-10-30T12:19:31.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 librados: setting wanted keys 2023-10-30T12:19:31.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 librados: calling monclient init 2023-10-30T12:19:31.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient: init 2023-10-30T12:19:31.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:31.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:19:31.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc7c0a2ae0 0x7fcc7c1ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient(hunting): picked mon.noname-b con 0x7fcc7c0a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:31.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c132550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7c0a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:31.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c132a90 0x7fcc7c11e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.303+0000 7fcc83852640 10 monclient(hunting): picked mon.noname-c con 0x7fcc7c132a90 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:31.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c1ca870 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0a1790 con 0x7fcc7c0a2ae0 2023-10-30T12:19:31.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c120680 con 0x7fcc7c132a90 2023-10-30T12:19:31.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:31.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 10 monclient(hunting): authenticate will time out at 2016.487031s 2023-10-30T12:19:31.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c132550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c132a90 0x7fcc7c11e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c132550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48744/0 (socket says 172.21.15.50:48744) 2023-10-30T12:19:31.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 -- 172.21.15.50:0/1180891432 learned_addr learned my addr 172.21.15.50:0/1180891432 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:31.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): get_auth_request con 0x7fcc7c0a2ec0 auth_method 0 2023-10-30T12:19:31.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 10 monclient(hunting): get_auth_request con 0x7fcc7c132a90 auth_method 0 2023-10-30T12:19:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc815ee640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc7c0a2ae0 0x7fcc7c1ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc815ee640 10 monclient(hunting): get_auth_request con 0x7fcc7c0a2ae0 auth_method 0 2023-10-30T12:19:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc815ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc815ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc815ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient(hunting): handle_auth_done global_id 244155 payload 824 2023-10-30T12:19:31.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c132a90 msgr2=0x7fcc7c11e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c132a90 0x7fcc7c11e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc7c0a2ae0 msgr2=0x7fcc7c1ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc7c0a2ae0 0x7fcc7c1ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient: _finish_hunting 0 2023-10-30T12:19:31.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 monclient: found mon.noname-a 2023-10-30T12:19:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 -- 172.21.15.50:0/1180891432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c009b30 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc81def640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc7c132a90 0x7fcc7c11e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:31.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc815ee640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc7c0a2ae0 0x7fcc7c1ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:19:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc80ded640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c132550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc70000c00 tx=0x7fcc700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 client.?.objecter ms_handle_connect 0x7fcc7c0a2ec0 2023-10-30T12:19:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:31.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70034680 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: dump: 2023-10-30T12:19:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.347 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.348 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.349 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: _finish_auth 0 2023-10-30T12:19:31.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:19:31.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:31.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc70034820 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:31.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70034bd0 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 5 monclient: authenticate success, global_id 244155 2023-10-30T12:19:31.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 10 monclient: _renew_subs 2023-10-30T12:19:31.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc70009400 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc7b7fe640 4 set_mon_vals no callback set 2023-10-30T12:19:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.304+0000 7fcc79ffb640 10 monclient: dump: 2023-10-30T12:19:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.353 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.354 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 10 client.244155.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 10 monclient: _renew_subs 2023-10-30T12:19:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c09ff30 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 1 librados: init done 2023-10-30T12:19:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcc7003b040 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a32e0 0x7fcc7c11e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c073ff0 con 0x7fcc7c0a32e0 2023-10-30T12:19:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc815ee640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a32e0 0x7fcc7c11e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc815ee640 10 monclient: get_auth_request con 0x7fcc7c0a32e0 auth_method 0 2023-10-30T12:19:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc815ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc815ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc815ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc79ffb640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc680392f0 0x7fcc6803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.305+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(595..595 src has 490..595) v4 ==== 6762+0+0 (secure 0 0 0) 0x7fcc7006c760 con 0x7fcc7c0a2ec0 2023-10-30T12:19:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 client.244155.objecter ms_dispatch 0x7fcc7c000b90 osd_map(595..595 src has 490..595) v4 2023-10-30T12:19:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 3 client.244155.objecter handle_osd_map got epochs [595,595] > 0 2023-10-30T12:19:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 3 client.244155.objecter handle_osd_map decoding full epoch 595 2023-10-30T12:19:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 20 client.244155.objecter dump_active .. 0 homeless 2023-10-30T12:19:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc815ee640 10 monclient(hunting): handle_auth_done global_id 244158 payload 824 2023-10-30T12:19:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc815ee640 1 -- 172.21.15.50:0/1180891432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc7c09a5c0 con 0x7fcc7c0a32e0 2023-10-30T12:19:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc81def640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc680392f0 0x7fcc6803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc81def640 10 monclient: get_auth_request con 0x7fcc680392f0 auth_method 0 2023-10-30T12:19:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 client.244155.objecter ms_handle_connect 0x7fcc7c0a32e0 2023-10-30T12:19:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 client.244155.objecter resend_mon_ops 2023-10-30T12:19:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc815ee640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a32e0 0x7fcc7c11e5e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007dc0 tx=0x7fcc6c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c00e050 con 0x7fcc7c0a32e0 2023-10-30T12:19:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 monclient: dump: 2023-10-30T12:19:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.364 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 client.244155.objecter ms_handle_connect 0x7fcc680392f0 2023-10-30T12:19:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc81def640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc680392f0 0x7fcc6803b7b0 secure :-1 s=READY pgs=4291 cs=0 l=1 rev1=1 crypto rx=0x7fcc74000f60 tx=0x7fcc74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fcc6c00f040 con 0x7fcc7c0a32e0 2023-10-30T12:19:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a32e0 msgr2=0x7fcc7c11e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a32e0 0x7fcc7c11e5e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007dc0 tx=0x7fcc6c010590 comp rx=0 tx=0).stop 2023-10-30T12:19:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a32e0 msgr2=0x7fcc7c11e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.306+0000 7fcc79ffb640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a32e0 0x7fcc7c11e5e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.312+0000 7fcc83852640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.312+0000 7fcc83852640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c120850 0x7fcc7c11eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.312+0000 7fcc83852640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.312+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c099b40 con 0x7fcc7c120850 2023-10-30T12:19:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c120850 0x7fcc7c11eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient: get_auth_request con 0x7fcc7c120850 auth_method 0 2023-10-30T12:19:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 10 monclient(hunting): handle_auth_done global_id 244161 payload 824 2023-10-30T12:19:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 1 -- 172.21.15.50:0/1180891432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcc7c06b060 con 0x7fcc7c120850 2023-10-30T12:19:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc815ee640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c120850 0x7fcc7c11eb20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007dc0 tx=0x7fcc6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 10 client.244155.objecter ms_handle_connect 0x7fcc7c120850 2023-10-30T12:19:31.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 10 client.244155.objecter resend_mon_ops 2023-10-30T12:19:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0208c0 con 0x7fcc7c120850 2023-10-30T12:19:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 10 monclient: dump: 2023-10-30T12:19:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.374 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.374 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.374 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.374 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7fcc6c020a20 con 0x7fcc7c120850 2023-10-30T12:19:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c120850 msgr2=0x7fcc7c11eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c120850 0x7fcc7c11eb20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007dc0 tx=0x7fcc6c004a40 comp rx=0 tx=0).stop 2023-10-30T12:19:31.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c120850 msgr2=0x7fcc7c11eb20 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:31.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.313+0000 7fcc79ffb640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c120850 0x7fcc7c11eb20 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007dc0 tx=0x7fcc6c004a40 comp rx=0 tx=0).stop 2023-10-30T12:19:31.376 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:31.377 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:31.377 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:31.377 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:19:31.377 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 198, 2023-10-30T12:19:31.377 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:31.378 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:31.378 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:19:31.378 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:19:31.378 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.379 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:19:31.379 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:31.379 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:31.379 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:31.379 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.380 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.380 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.380 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.380 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.380 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.381 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.381 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.381 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.381 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.382 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.382 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:31.382 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:31.382 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.382 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.383 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.383 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.383 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.383 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.383 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.384 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.384 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.384 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.384 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.385 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:31.385 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:31.385 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:31.385 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:31.386 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:31.386 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:31.386 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:31.386 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:31.386 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:31.387 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:31.387 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:31.387 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:31.387 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:31.388 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:31.388 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:31.388 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:31.388 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:31.388 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.389 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.389 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.389 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.389 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.390 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.390 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.390 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.390 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.391 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:31.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:31.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.392 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:31.392 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:31.392 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.392 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.393 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:31.393 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:31.396 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.396 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.396 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.397 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:31.397 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:31.397 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.397 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.398 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.398 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:31.398 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.399 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.399 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:31.399 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.400 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.400 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:31.401 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:31.401 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.401 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.402 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.402 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:19:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.411 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:31.411 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:31.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 10 librados: watch_flush enter 2023-10-30T12:19:31.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 10 librados: watch_flush exit 2023-10-30T12:19:31.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 20 client.244155.objecter shutdown clearing up homeless session... 2023-10-30T12:19:31.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 10 client.244155.objecter successfully canceled tick 2023-10-30T12:19:31.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc680392f0 msgr2=0x7fcc6803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcc680392f0 0x7fcc6803b7b0 secure :-1 s=READY pgs=4291 cs=0 l=1 rev1=1 crypto rx=0x7fcc74000f60 tx=0x7fcc74009040 comp rx=0 tx=0).stop 2023-10-30T12:19:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 10 monclient: shutdown 2023-10-30T12:19:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 msgr2=0x7fcc7c132550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 --2- 172.21.15.50:0/1180891432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc7c0a2ec0 0x7fcc7c132550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc70000c00 tx=0x7fcc700046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc81def640 1 -- 172.21.15.50:0/1180891432 reap_dead start 2023-10-30T12:19:31.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc81def640 1 -- 172.21.15.50:0/1180891432 reap_dead start 2023-10-30T12:19:31.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 shutdown_connections 2023-10-30T12:19:31.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 >> 172.21.15.50:0/1180891432 conn(0x7fcc7c0a80b0 msgr2=0x7fcc7c1bbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:31.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 shutdown_connections 2023-10-30T12:19:31.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 -- 172.21.15.50:0/1180891432 wait complete. 2023-10-30T12:19:31.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.314+0000 7fcc83852640 1 librados: shutdown 2023-10-30T12:19:31.417 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:19:31.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:19:31.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:19:31.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.523+0000 7fbb4f861640 10 monclient: get_monmap_and_config 2023-10-30T12:19:31.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.523+0000 7fbb4f861640 10 monclient: build_initial_monmap 2023-10-30T12:19:31.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.523+0000 7fbb4f861640 10 monclient: monmap: 2023-10-30T12:19:31.525 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:31.525 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.525 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:31.524419+0000 2023-10-30T12:19:31.526 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:31.524419+0000 2023-10-30T12:19:31.526 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:31.526 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.526 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:31.526 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:31.527 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:31.527 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:31.527 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:31.527 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.523+0000 7fbb4f861640 1 Processor -- start 2023-10-30T12:19:31.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 1 -- start start 2023-10-30T12:19:31.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient: init 2023-10-30T12:19:31.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:31.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-10-30T12:19:31.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb48096540 0x7fbb48096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient(hunting): picked mon.noname-b con 0x7fbb48096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:31.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb48096e60 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:31.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbb4809b170 0x7fbb4809b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient(hunting): picked mon.noname-c con 0x7fbb4809b170 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:31.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4809bb20 con 0x7fbb48096e60 2023-10-30T12:19:31.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4809bd30 con 0x7fbb4809b170 2023-10-30T12:19:31.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4809c400 con 0x7fbb48096540 2023-10-30T12:19:31.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:31.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4f861640 10 monclient(hunting): authenticate will time out at 2016.707603s 2023-10-30T12:19:31.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb48096540 0x7fbb48096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4cdfc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbb48096e60 0x7fbb4809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40466/0 (socket says 172.21.15.50:40466) 2023-10-30T12:19:31.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4cdfc640 1 -- 172.21.15.50:0/1381620471 learned_addr learned my addr 172.21.15.50:0/1381620471 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:31.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 10 monclient(hunting): get_auth_request con 0x7fbb48096540 auth_method 0 2023-10-30T12:19:31.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.524+0000 7fbb4d5fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4ddfe640 1 --2- 172.21.15.50:0/1381620471 >> v2:172.21.15.118:3300/0 conn(0x7fbb4809b170 0x7fbb4809b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 10 monclient(hunting): handle_auth_done global_id 224327 payload 274 2023-10-30T12:19:31.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 1 -- 172.21.15.50:0/1381620471 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902154177 0 0) 0x7fbb4809bb20 con 0x7fbb48096e60 2023-10-30T12:19:31.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 1 -- 172.21.15.50:0/1381620471 >> v2:172.21.15.118:3300/0 conn(0x7fbb4809b170 msgr2=0x7fbb4809b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:31.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 1 --2- 172.21.15.50:0/1381620471 >> v2:172.21.15.118:3300/0 conn(0x7fbb4809b170 0x7fbb4809b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 1 -- 172.21.15.50:0/1381620471 >> v1:172.21.15.50:6789/0 conn(0x7fbb48096e60 legacy=0x7fbb4809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 10 monclient: _finish_hunting 0 2023-10-30T12:19:31.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 1 monclient: found mon.noname-b 2023-10-30T12:19:31.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:31.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:19:31.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 1 -- 172.21.15.50:0/1381620471 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4809c120 con 0x7fbb48096540 2023-10-30T12:19:31.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:19:31.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4ddfe640 10 monclient: get_auth_request con 0x7fbb4809b170 auth_method 0 2023-10-30T12:19:31.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4ddfe640 1 --2- 172.21.15.50:0/1381620471 >> v2:172.21.15.118:3300/0 conn(0x7fbb4809b170 0x7fbb4809b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:19:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4d5fd640 1 --2- 172.21.15.50:0/1381620471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb48096540 0x7fbb48096920 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009790 tx=0x7fbb4402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d12476331d09aac2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 1 -- 172.21.15.50:0/1381620471 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4402d040 con 0x7fbb48096540 2023-10-30T12:19:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:19:31.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: dump: 2023-10-30T12:19:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.548 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.548 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.548 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.548 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.548 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: _finish_auth 0 2023-10-30T12:19:31.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:31.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:31.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 1 -- 172.21.15.50:0/1381620471 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb44031810 con 0x7fbb48096540 2023-10-30T12:19:31.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4f861640 5 monclient: authenticate success, global_id 224327 2023-10-30T12:19:31.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4f861640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:31.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:31.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 1 -- 172.21.15.50:0/1381620471 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44031ba0 con 0x7fbb48096540 2023-10-30T12:19:31.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:31.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb3ffff640 10 monclient: dump: 2023-10-30T12:19:31.551 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.551 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.551 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.551 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.551 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.552 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.552 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.552 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.552 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.552 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4f861640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.525+0000 7fbb4f861640 4 set_mon_vals no callback set 2023-10-30T12:19:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 10 monclient: shutdown 2023-10-30T12:19:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 -- 172.21.15.50:0/1381620471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb48096540 msgr2=0x7fbb48096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 --2- 172.21.15.50:0/1381620471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb48096540 0x7fbb48096920 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009790 tx=0x7fbb4402b8b0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 -- 172.21.15.50:0/1381620471 shutdown_connections 2023-10-30T12:19:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 --2- 172.21.15.50:0/1381620471 >> v2:172.21.15.118:3300/0 conn(0x7fbb4809b170 0x7fbb4809b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 --2- 172.21.15.50:0/1381620471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb48096540 0x7fbb48096920 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 -- 172.21.15.50:0/1381620471 >> 172.21.15.50:0/1381620471 conn(0x7fbb480a3980 msgr2=0x7fbb480a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:31.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 -- 172.21.15.50:0/1381620471 shutdown_connections 2023-10-30T12:19:31.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.526+0000 7fbb4f861640 1 -- 172.21.15.50:0/1381620471 wait complete. 2023-10-30T12:19:31.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient: build_initial_monmap 2023-10-30T12:19:31.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient: monmap: 2023-10-30T12:19:31.555 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:31.555 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.556 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:31.557 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 librados: starting msgr at 2023-10-30T12:19:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 librados: starting objecter 2023-10-30T12:19:31.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 Processor -- start 2023-10-30T12:19:31.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 -- start start 2023-10-30T12:19:31.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 librados: setting wanted keys 2023-10-30T12:19:31.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 librados: calling monclient init 2023-10-30T12:19:31.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient: init 2023-10-30T12:19:31.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:31.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:19:31.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb48096540 0x7fbb481ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): picked mon.noname-b con 0x7fbb48096540 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:31.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb48096e60 0x7fbb480945f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): picked mon.noname-a con 0x7fbb48096e60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:31.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb4809b170 0x7fbb48094b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): picked mon.noname-c con 0x7fbb4809b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480989c0 con 0x7fbb48096e60 2023-10-30T12:19:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb481cad30 con 0x7fbb48096540 2023-10-30T12:19:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48091130 con 0x7fbb4809b170 2023-10-30T12:19:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.527+0000 7fbb4f861640 10 monclient(hunting): authenticate will time out at 2016.711004s 2023-10-30T12:19:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb4809b170 0x7fbb48094b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb48096e60 0x7fbb480945f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb48096e60 0x7fbb480945f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48774/0 (socket says 172.21.15.50:48774) 2023-10-30T12:19:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb4809b170 0x7fbb48094b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33886/0 (socket says 172.21.15.50:33886) 2023-10-30T12:19:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 -- 172.21.15.50:0/60191257 learned_addr learned my addr 172.21.15.50:0/60191257 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:31.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): get_auth_request con 0x7fbb48096e60 auth_method 0 2023-10-30T12:19:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 10 monclient(hunting): get_auth_request con 0x7fbb4809b170 auth_method 0 2023-10-30T12:19:31.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb48096540 0x7fbb481ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 10 monclient(hunting): get_auth_request con 0x7fbb48096540 auth_method 0 2023-10-30T12:19:31.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4d5fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient(hunting): handle_auth_done global_id 244167 payload 824 2023-10-30T12:19:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb4809b170 msgr2=0x7fbb48094b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb4809b170 0x7fbb48094b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb48096540 msgr2=0x7fbb481ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb48096540 0x7fbb481ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient: _finish_hunting 0 2023-10-30T12:19:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 monclient: found mon.noname-a 2023-10-30T12:19:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:31.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 -- 172.21.15.50:0/60191257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb440093a0 con 0x7fbb48096e60 2023-10-30T12:19:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4ddfe640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb4809b170 0x7fbb48094b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:19:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4cdfc640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb48096e60 0x7fbb480945f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800c470 tx=0x7fbb3800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 client.?.objecter ms_handle_connect 0x7fbb48096e60 2023-10-30T12:19:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380074e0 con 0x7fbb48096e60 2023-10-30T12:19:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: dump: 2023-10-30T12:19:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.571 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.571 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.571 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.571 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.571 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.572 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.572 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.572 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.572 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: _finish_auth 0 2023-10-30T12:19:31.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: _check_auth_tickets 2023-10-30T12:19:31.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:31.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4f861640 5 monclient: authenticate success, global_id 244167 2023-10-30T12:19:31.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4f861640 10 monclient: _renew_subs 2023-10-30T12:19:31.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4f861640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb48090860 con 0x7fbb48096e60 2023-10-30T12:19:31.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb38007640 con 0x7fbb48096e60 2023-10-30T12:19:31.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:31.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38004a20 con 0x7fbb48096e60 2023-10-30T12:19:31.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:31.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3dffb640 10 monclient: dump: 2023-10-30T12:19:31.575 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.575 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.576 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.576 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.576 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.576 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.577 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.577 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.577 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.577 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.528+0000 7fbb3f7fe640 4 set_mon_vals no callback set 2023-10-30T12:19:31.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 10 client.244167.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:31.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 10 monclient: _renew_subs 2023-10-30T12:19:31.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb48099060 con 0x7fbb48096e60 2023-10-30T12:19:31.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 1 librados: init done 2023-10-30T12:19:31.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbb38004b80 con 0x7fbb48096e60 2023-10-30T12:19:31.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb3dffb640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbb2c0392f0 0x7fbb2c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(595..595 src has 490..595) v4 ==== 6762+0+0 (secure 0 0 0) 0x7fbb3804a0d0 con 0x7fbb48096e60 2023-10-30T12:19:31.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb3dffb640 10 client.244167.objecter ms_dispatch 0x7fbb48000b90 osd_map(595..595 src has 490..595) v4 2023-10-30T12:19:31.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb3dffb640 3 client.244167.objecter handle_osd_map got epochs [595,595] > 0 2023-10-30T12:19:31.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb3dffb640 3 client.244167.objecter handle_osd_map decoding full epoch 595 2023-10-30T12:19:31.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:31.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c6fd0 0x7fbb4802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.529+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48090c80 con 0x7fbb481c6fd0 2023-10-30T12:19:31.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 20 client.244167.objecter dump_active .. 0 homeless 2023-10-30T12:19:31.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4d5fd640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbb2c0392f0 0x7fbb2c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c6fd0 0x7fbb4802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4d5fd640 10 monclient: get_auth_request con 0x7fbb2c0392f0 auth_method 0 2023-10-30T12:19:31.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient: get_auth_request con 0x7fbb481c6fd0 auth_method 0 2023-10-30T12:19:31.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_done global_id 224333 payload 824 2023-10-30T12:19:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 1 -- 172.21.15.50:0/60191257 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb4809d3a0 con 0x7fbb481c6fd0 2023-10-30T12:19:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 client.244167.objecter ms_handle_connect 0x7fbb481c6fd0 2023-10-30T12:19:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 client.244167.objecter resend_mon_ops 2023-10-30T12:19:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4ddfe640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c6fd0 0x7fbb4802b430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000dfd0 tx=0x7fbb4000cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 client.244167.objecter ms_handle_connect 0x7fbb2c0392f0 2023-10-30T12:19:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb4d5fd640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbb2c0392f0 0x7fbb2c03b7b0 secure :-1 s=READY pgs=4292 cs=0 l=1 rev1=1 crypto rx=0x7fbb44036000 tx=0x7fbb4403c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40012070 con 0x7fbb481c6fd0 2023-10-30T12:19:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 monclient: dump: 2023-10-30T12:19:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.586 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.586 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.586 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.586 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fbb4000ce70 con 0x7fbb481c6fd0 2023-10-30T12:19:31.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:31.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:31.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c6fd0 msgr2=0x7fbb4802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c6fd0 0x7fbb4802b430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000dfd0 tx=0x7fbb4000cc20 comp rx=0 tx=0).stop 2023-10-30T12:19:31.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c6fd0 msgr2=0x7fbb4802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.530+0000 7fbb3dffb640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c6fd0 0x7fbb4802b430 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000dfd0 tx=0x7fbb4000cc20 comp rx=0 tx=0).stop 2023-10-30T12:19:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4f861640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4f861640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c73b0 0x7fbb4802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4f861640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48073ff0 con 0x7fbb481c73b0 2023-10-30T12:19:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c73b0 0x7fbb4802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient: get_auth_request con 0x7fbb481c73b0 auth_method 0 2023-10-30T12:19:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 10 monclient(hunting): handle_auth_done global_id 224336 payload 824 2023-10-30T12:19:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 1 -- 172.21.15.50:0/60191257 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb4804e160 con 0x7fbb481c73b0 2023-10-30T12:19:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb4ddfe640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c73b0 0x7fbb4802b970 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000dfd0 tx=0x7fbb40004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb3dffb640 10 client.244167.objecter ms_handle_connect 0x7fbb481c73b0 2023-10-30T12:19:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb3dffb640 10 client.244167.objecter resend_mon_ops 2023-10-30T12:19:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40022630 con 0x7fbb481c73b0 2023-10-30T12:19:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:19:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.537+0000 7fbb3dffb640 10 monclient: dump: 2023-10-30T12:19:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.596 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7fbb40022790 con 0x7fbb481c73b0 2023-10-30T12:19:31.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb3dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:31.596 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:19:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 198, 2023-10-30T12:19:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:19:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:19:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:19:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:19:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:19:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:19:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:19:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:31.629 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb3dffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c73b0 msgr2=0x7fbb4802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb3dffb640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c73b0 0x7fbb4802b970 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000dfd0 tx=0x7fbb40004a40 comp rx=0 tx=0).stop 2023-10-30T12:19:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb3dffb640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c73b0 msgr2=0x7fbb4802b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb3dffb640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb481c73b0 0x7fbb4802b970 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 10 librados: watch_flush enter 2023-10-30T12:19:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 10 librados: watch_flush exit 2023-10-30T12:19:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 20 client.244167.objecter shutdown clearing up homeless session... 2023-10-30T12:19:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 10 client.244167.objecter successfully canceled tick 2023-10-30T12:19:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbb2c0392f0 msgr2=0x7fbb2c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbb2c0392f0 0x7fbb2c03b7b0 secure :-1 s=READY pgs=4292 cs=0 l=1 rev1=1 crypto rx=0x7fbb44036000 tx=0x7fbb4403c040 comp rx=0 tx=0).stop 2023-10-30T12:19:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 10 monclient: shutdown 2023-10-30T12:19:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb48096e60 msgr2=0x7fbb480945f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4ddfe640 1 -- 172.21.15.50:0/60191257 reap_dead start 2023-10-30T12:19:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb48096e60 0x7fbb480945f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800c470 tx=0x7fbb3800c930 comp rx=0 tx=0).stop 2023-10-30T12:19:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 shutdown_connections 2023-10-30T12:19:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 --2- 172.21.15.50:0/60191257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb48096e60 0x7fbb480945f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 >> 172.21.15.50:0/60191257 conn(0x7fbb480a3980 msgr2=0x7fbb481ba900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 shutdown_connections 2023-10-30T12:19:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 -- 172.21.15.50:0/60191257 wait complete. 2023-10-30T12:19:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.538+0000 7fbb4f861640 1 librados: shutdown 2023-10-30T12:19:31.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:19:31.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:19:31.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:19:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.741+0000 7fb91e04f640 10 monclient: get_monmap_and_config 2023-10-30T12:19:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.741+0000 7fb91e04f640 10 monclient: build_initial_monmap 2023-10-30T12:19:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.741+0000 7fb91e04f640 10 monclient: monmap: 2023-10-30T12:19:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:31.742718+0000 2023-10-30T12:19:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:31.742718+0000 2023-10-30T12:19:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:31.746 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 1 Processor -- start 2023-10-30T12:19:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 1 -- start start 2023-10-30T12:19:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient: init 2023-10-30T12:19:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:19:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb91809bb70 0x7fb91809bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient(hunting): picked mon.noname-a con 0x7fb91809bb70 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb91809d3f0 0x7fb91809c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient(hunting): picked mon.noname-c con 0x7fb91809d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb918096080 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb918098170 con 0x7fb918096080 2023-10-30T12:19:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91809c9d0 con 0x7fb91809bb70 2023-10-30T12:19:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91809cf60 con 0x7fb91809d3f0 2023-10-30T12:19:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.742+0000 7fb91e04f640 10 monclient(hunting): authenticate will time out at 2016.925948s 2023-10-30T12:19:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb91809bb70 0x7fb91809bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb91809bb70 0x7fb91809bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48776/0 (socket says 172.21.15.50:48776) 2023-10-30T12:19:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 -- 172.21.15.50:0/3371123183 learned_addr learned my addr 172.21.15.50:0/3371123183 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): get_auth_request con 0x7fb91809bb70 auth_method 0 2023-10-30T12:19:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb916ffd640 1 --2- 172.21.15.50:0/3371123183 >> v2:172.21.15.118:3300/0 conn(0x7fb91809d3f0 0x7fb91809c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb916ffd640 10 monclient(hunting): get_auth_request con 0x7fb91809d3f0 auth_method 0 2023-10-30T12:19:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb916ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb916ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient(hunting): handle_auth_done global_id 244170 payload 274 2023-10-30T12:19:31.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 -- 172.21.15.50:0/3371123183 >> v2:172.21.15.118:3300/0 conn(0x7fb91809d3f0 msgr2=0x7fb91809c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:31.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 --2- 172.21.15.50:0/3371123183 >> v2:172.21.15.118:3300/0 conn(0x7fb91809d3f0 0x7fb91809c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 -- 172.21.15.50:0/3371123183 >> v1:172.21.15.50:6789/0 conn(0x7fb918096080 legacy=0x7fb918096510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient: _finish_hunting 0 2023-10-30T12:19:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 monclient: found mon.noname-a 2023-10-30T12:19:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.743+0000 7fb9177fe640 1 -- 172.21.15.50:0/3371123183 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb918097b20 con 0x7fb91809bb70 2023-10-30T12:19:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9177fe640 1 --2- 172.21.15.50:0/3371123183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb91809bb70 0x7fb91809bf50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb91809d840 tx=0x7fb90802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbf01e4cb301398 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 1 -- 172.21.15.50:0/3371123183 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908004230 con 0x7fb91809bb70 2023-10-30T12:19:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: dump: 2023-10-30T12:19:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.765 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: _finish_auth 0 2023-10-30T12:19:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: _check_auth_tickets 2023-10-30T12:19:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 1 -- 172.21.15.50:0/3371123183 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb908004390 con 0x7fb91809bb70 2023-10-30T12:19:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 5 monclient: authenticate success, global_id 244170 2023-10-30T12:19:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:31.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:31.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 1 -- 172.21.15.50:0/3371123183 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908003810 con 0x7fb91809bb70 2023-10-30T12:19:31.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 4 set_mon_vals no callback set 2023-10-30T12:19:31.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb9167fc640 10 monclient: dump: 2023-10-30T12:19:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.769 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 10 monclient: shutdown 2023-10-30T12:19:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 -- 172.21.15.50:0/3371123183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb91809bb70 msgr2=0x7fb91809bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 --2- 172.21.15.50:0/3371123183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb91809bb70 0x7fb91809bf50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb91809d840 tx=0x7fb90802b860 comp rx=0 tx=0).stop 2023-10-30T12:19:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 -- 172.21.15.50:0/3371123183 shutdown_connections 2023-10-30T12:19:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 --2- 172.21.15.50:0/3371123183 >> v2:172.21.15.118:3300/0 conn(0x7fb91809d3f0 0x7fb91809c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 --2- 172.21.15.50:0/3371123183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb91809bb70 0x7fb91809bf50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 -- 172.21.15.50:0/3371123183 >> 172.21.15.50:0/3371123183 conn(0x7fb9180a3980 msgr2=0x7fb9180a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 -- 172.21.15.50:0/3371123183 shutdown_connections 2023-10-30T12:19:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.744+0000 7fb91e04f640 1 -- 172.21.15.50:0/3371123183 wait complete. 2023-10-30T12:19:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 10 monclient: build_initial_monmap 2023-10-30T12:19:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 10 monclient: monmap: 2023-10-30T12:19:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:31.773 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.773 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:31.773 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:31.773 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:31.773 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 1 librados: starting msgr at 2023-10-30T12:19:31.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 1 librados: starting objecter 2023-10-30T12:19:31.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 1 Processor -- start 2023-10-30T12:19:31.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 1 -- start start 2023-10-30T12:19:31.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 1 librados: setting wanted keys 2023-10-30T12:19:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 1 librados: calling monclient init 2023-10-30T12:19:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.745+0000 7fb91e04f640 10 monclient: init 2023-10-30T12:19:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb918096080 0x7fb9180712e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): picked mon.noname-a con 0x7fb918096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb91809bb70 0x7fb9181c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): picked mon.noname-c con 0x7fb91809bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91809d3f0 0x7fb9181c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): picked mon.noname-b con 0x7fb91809d3f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91806d490 con 0x7fb918096080 2023-10-30T12:19:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb918097170 con 0x7fb91809d3f0 2023-10-30T12:19:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9181c5640 con 0x7fb91809bb70 2023-10-30T12:19:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb91e04f640 10 monclient(hunting): authenticate will time out at 2016.929434s 2023-10-30T12:19:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb918096080 0x7fb9180712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb91809bb70 0x7fb9181c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb918096080 0x7fb9180712e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48782/0 (socket says 172.21.15.50:48782) 2023-10-30T12:19:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 -- 172.21.15.50:0/2874729663 learned_addr learned my addr 172.21.15.50:0/2874729663 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): get_auth_request con 0x7fb918096080 auth_method 0 2023-10-30T12:19:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 10 monclient(hunting): get_auth_request con 0x7fb91809bb70 auth_method 0 2023-10-30T12:19:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb917fff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91809d3f0 0x7fb9181c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb917fff640 10 monclient(hunting): get_auth_request con 0x7fb91809d3f0 auth_method 0 2023-10-30T12:19:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb917fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb917fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient(hunting): handle_auth_done global_id 244176 payload 824 2023-10-30T12:19:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb91809bb70 msgr2=0x7fb9181c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb91809bb70 0x7fb9181c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91809d3f0 msgr2=0x7fb9181c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91809d3f0 0x7fb9181c5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient: _finish_hunting 0 2023-10-30T12:19:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 monclient: found mon.noname-a 2023-10-30T12:19:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:31.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.746+0000 7fb9177fe640 1 -- 172.21.15.50:0/2874729663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb908009460 con 0x7fb918096080 2023-10-30T12:19:31.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb9177fe640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb918096080 0x7fb9180712e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb908009940 tx=0x7fb9080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 client.?.objecter ms_handle_connect 0x7fb918096080 2023-10-30T12:19:31.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:31.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908005ea0 con 0x7fb918096080 2023-10-30T12:19:31.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:31.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: dump: 2023-10-30T12:19:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.787 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: _finish_auth 0 2023-10-30T12:19:31.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:31.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:31.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb908004500 con 0x7fb918096080 2023-10-30T12:19:31.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 5 monclient: authenticate success, global_id 244176 2023-10-30T12:19:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 10 monclient: _renew_subs 2023-10-30T12:19:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9180976e0 con 0x7fb918096080 2023-10-30T12:19:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908006280 con 0x7fb918096080 2023-10-30T12:19:31.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb8fffff640 10 monclient: dump: 2023-10-30T12:19:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.791 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.791 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.791 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.791 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.791 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.792 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.792 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 10 client.244176.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:31.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 10 monclient: _renew_subs 2023-10-30T12:19:31.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:31.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb915ffb640 4 set_mon_vals no callback set 2023-10-30T12:19:31.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91806fdb0 con 0x7fb918096080 2023-10-30T12:19:31.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.747+0000 7fb91e04f640 1 librados: init done 2023-10-30T12:19:31.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb9080063e0 con 0x7fb918096080 2023-10-30T12:19:31.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb91e04f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:31.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb91e04f640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb918071820 0x7fb9181c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb91e04f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb918073ff0 con 0x7fb918071820 2023-10-30T12:19:31.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb9040392f0 0x7fb90403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(596..596 src has 490..596) v4 ==== 6706+0+0 (secure 0 0 0) 0x7fb90806cec0 con 0x7fb918096080 2023-10-30T12:19:31.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 10 client.244176.objecter ms_dispatch 0x7fb918000b90 osd_map(596..596 src has 490..596) v4 2023-10-30T12:19:31.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 3 client.244176.objecter handle_osd_map got epochs [596,596] > 0 2023-10-30T12:19:31.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 3 client.244176.objecter handle_osd_map decoding full epoch 596 2023-10-30T12:19:31.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb918071820 0x7fb9181c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient: get_auth_request con 0x7fb918071820 auth_method 0 2023-10-30T12:19:31.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 20 client.244176.objecter dump_active .. 0 homeless 2023-10-30T12:19:31.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb917fff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb9040392f0 0x7fb90403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb917fff640 10 monclient: get_auth_request con 0x7fb9040392f0 auth_method 0 2023-10-30T12:19:31.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 10 monclient(hunting): handle_auth_done global_id 204601 payload 824 2023-10-30T12:19:31.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 1 -- 172.21.15.50:0/2874729663 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb918097560 con 0x7fb918071820 2023-10-30T12:19:31.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb916ffd640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb918071820 0x7fb9181c88e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb91813ad20 tx=0x7fb90c010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 10 client.244176.objecter ms_handle_connect 0x7fb918071820 2023-10-30T12:19:31.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 10 client.244176.objecter resend_mon_ops 2023-10-30T12:19:31.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb8fffff640 10 client.244176.objecter ms_handle_connect 0x7fb9040392f0 2023-10-30T12:19:31.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.748+0000 7fb917fff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb9040392f0 0x7fb90403b7b0 secure :-1 s=READY pgs=4293 cs=0 l=1 rev1=1 crypto rx=0x7fb910007fd0 tx=0x7fb910009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c004240 con 0x7fb918071820 2023-10-30T12:19:31.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:31.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 10 monclient: dump: 2023-10-30T12:19:31.799 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.799 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.799 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.801 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.801 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fb90c0043a0 con 0x7fb918071820 2023-10-30T12:19:31.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:31.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:31.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb918071820 msgr2=0x7fb9181c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb918071820 0x7fb9181c88e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb91813ad20 tx=0x7fb90c010c10 comp rx=0 tx=0).stop 2023-10-30T12:19:31.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb918071820 msgr2=0x7fb9181c88e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:31.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.749+0000 7fb8fffff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb918071820 0x7fb9181c88e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb91e04f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:31.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb91e04f640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91802b430 0x7fb91802b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:31.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb91e04f640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:31.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb918097e00 con 0x7fb91802b430 2023-10-30T12:19:31.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb916ffd640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91802b430 0x7fb91802b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:31.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb916ffd640 10 monclient: get_auth_request con 0x7fb91802b430 auth_method 0 2023-10-30T12:19:31.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:31.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb916ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:31.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb916ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:31.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:31.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.755+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:31.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb916ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:31.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb916ffd640 10 monclient(hunting): handle_auth_done global_id 204604 payload 824 2023-10-30T12:19:31.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb916ffd640 1 -- 172.21.15.50:0/2874729663 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb91804e160 con 0x7fb91802b430 2023-10-30T12:19:31.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb916ffd640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91802b430 0x7fb91802b840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb90c019000 tx=0x7fb90c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:31.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 10 client.244176.objecter ms_handle_connect 0x7fb91802b430 2023-10-30T12:19:31.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 10 client.244176.objecter resend_mon_ops 2023-10-30T12:19:31.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c025070 con 0x7fb91802b430 2023-10-30T12:19:31.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:31.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:31.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 10 monclient: dump: 2023-10-30T12:19:31.808 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:31.808 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:31.808 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.809 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:31.809 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:31.809 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:31.809 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:31.809 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:31.810 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:31.810 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:31.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fb90c003df0 con 0x7fb91802b430 2023-10-30T12:19:31.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:31.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:31.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91802b430 msgr2=0x7fb91802b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91802b430 0x7fb91802b840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb90c019000 tx=0x7fb90c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91802b430 msgr2=0x7fb91802b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:31.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb8fffff640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb91802b430 0x7fb91802b840 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb90c019000 tx=0x7fb90c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.812 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:31.812 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:31.812 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:31.812 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:19:31.813 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 198, 2023-10-30T12:19:31.813 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:31.813 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:31.813 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:19:31.813 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:19:31.814 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.814 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:19:31.814 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:31.814 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:31.814 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:31.815 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.815 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.815 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.815 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.815 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.816 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.816 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.816 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.816 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.816 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.816 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.817 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:31.817 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:31.817 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.817 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.817 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.818 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.818 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.818 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.818 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.818 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.819 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.819 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.819 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.819 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:31.819 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:31.820 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:31.820 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:31.820 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:31.820 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:31.820 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:31.821 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:31.821 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:31.821 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:31.821 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:31.821 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:31.822 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:31.822 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:31.822 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:31.822 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:31.822 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:31.823 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:31.823 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:31.823 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:31.823 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:31.823 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:31.824 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:31.824 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:31.824 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:31.824 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:31.824 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:31.825 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.825 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:31.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.825 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:31.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.825 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:31.826 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:31.826 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.826 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.826 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.827 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:31.827 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.827 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.827 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:31.828 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.828 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.828 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.828 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:31.829 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:31.829 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.829 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.829 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.830 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:31.830 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:31.830 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.830 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.831 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:31.831 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.832 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.832 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:31.832 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.832 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.833 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.833 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:31.833 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:31.833 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.833 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.834 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.834 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:31.834 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:31.835 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:31.835 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:31.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.835 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:31.835 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:31.836 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.836 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:31.836 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:31.837 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:31.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.837 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.837 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:31.838 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:31.838 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:31.838 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:31.838 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:31.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.839 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.839 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:31.839 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:31.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.840 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.840 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.840 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:31.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.840 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:31.841 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:31.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:31.841 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:31.841 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:31.841 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:19:31.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:31.842 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:31.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:31.842 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:31.842 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:31.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb91e04f640 10 librados: watch_flush enter 2023-10-30T12:19:31.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb91e04f640 10 librados: watch_flush exit 2023-10-30T12:19:31.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb91e04f640 20 client.244176.objecter shutdown clearing up homeless session... 2023-10-30T12:19:31.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.756+0000 7fb91e04f640 10 client.244176.objecter successfully canceled tick 2023-10-30T12:19:31.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb9040392f0 msgr2=0x7fb90403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb9040392f0 0x7fb90403b7b0 secure :-1 s=READY pgs=4293 cs=0 l=1 rev1=1 crypto rx=0x7fb910007fd0 tx=0x7fb910009040 comp rx=0 tx=0).stop 2023-10-30T12:19:31.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 10 monclient: shutdown 2023-10-30T12:19:31.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:31.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb918096080 msgr2=0x7fb9180712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:31.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 --2- 172.21.15.50:0/2874729663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb918096080 0x7fb9180712e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb908009940 tx=0x7fb9080046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:31.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb917fff640 1 -- 172.21.15.50:0/2874729663 reap_dead start 2023-10-30T12:19:31.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb917fff640 1 -- 172.21.15.50:0/2874729663 reap_dead start 2023-10-30T12:19:31.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 shutdown_connections 2023-10-30T12:19:31.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 >> 172.21.15.50:0/2874729663 conn(0x7fb9180a3980 msgr2=0x7fb91813b080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:31.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 shutdown_connections 2023-10-30T12:19:31.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 -- 172.21.15.50:0/2874729663 wait complete. 2023-10-30T12:19:31.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:31.757+0000 7fb91e04f640 1 librados: shutdown 2023-10-30T12:19:31.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:19:31.848 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-10-30T12:19:31.848 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:19:31.849 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-10-30T12:19:31.849 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:19:31.849 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:19:31.849 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:19:31.862 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:19:31.863 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:19:31.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:32.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.004+0000 7f628bc56640 10 monclient: get_monmap_and_config 2023-10-30T12:19:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.004+0000 7f628bc56640 10 monclient: build_initial_monmap 2023-10-30T12:19:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.004+0000 7f628bc56640 10 monclient: monmap: 2023-10-30T12:19:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:32.005843+0000 2023-10-30T12:19:32.007 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:32.005843+0000 2023-10-30T12:19:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:32.009 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 1 Processor -- start 2023-10-30T12:19:32.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 1 -- start start 2023-10-30T12:19:32.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 10 monclient: init 2023-10-30T12:19:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:32.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-10-30T12:19:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62840a0080 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f62840a0460 0x7f62840a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 10 monclient(hunting): picked mon.noname-a con 0x7f62840a0460 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f6284138e60 0x7f628413b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:32.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 10 monclient(hunting): picked mon.noname-c con 0x7f6284138e60 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.005+0000 7f628bc56640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628409d8f0 con 0x7f62840a0080 2023-10-30T12:19:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628bc56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628bc56640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628409e9e0 con 0x7f62840a0460 2023-10-30T12:19:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628bc56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628bc56640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628409d610 con 0x7f6284138e60 2023-10-30T12:19:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628bc56640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628bc56640 10 monclient(hunting): authenticate will time out at 2017.189024s 2023-10-30T12:19:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f62840a0460 0x7f62840a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f62840a0460 0x7f62840a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48816/0 (socket says 172.21.15.50:48816) 2023-10-30T12:19:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 -- 172.21.15.50:0/3320425870 learned_addr learned my addr 172.21.15.50:0/3320425870 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): get_auth_request con 0x7f62840a0460 auth_method 0 2023-10-30T12:19:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:32.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:32.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628a1f3640 1 --2- 172.21.15.50:0/3320425870 >> v2:172.21.15.118:3300/0 conn(0x7f6284138e60 0x7f628413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient(hunting): handle_auth_done global_id 244179 payload 274 2023-10-30T12:19:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 -- 172.21.15.50:0/3320425870 >> v2:172.21.15.118:3300/0 conn(0x7f6284138e60 msgr2=0x7f628413b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 --2- 172.21.15.50:0/3320425870 >> v2:172.21.15.118:3300/0 conn(0x7f6284138e60 0x7f628413b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 -- 172.21.15.50:0/3320425870 >> v1:172.21.15.118:6789/0 conn(0x7f62840a0080 legacy=0x7f628409e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:32.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient: _finish_hunting 0 2023-10-30T12:19:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 monclient: found mon.noname-a 2023-10-30T12:19:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:32.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f62891f1640 1 -- 172.21.15.50:0/3320425870 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628409d050 con 0x7f62840a0460 2023-10-30T12:19:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628a1f3640 10 monclient: get_auth_request con 0x7f6284138e60 auth_method 0 2023-10-30T12:19:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.006+0000 7f628a1f3640 1 --2- 172.21.15.50:0/3320425870 >> v2:172.21.15.118:3300/0 conn(0x7f6284138e60 0x7f628413b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:19:32.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62891f1640 1 --2- 172.21.15.50:0/3320425870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840a08b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f628409c460 tx=0x7f627402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333b8e63c9150cc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 1 -- 172.21.15.50:0/3320425870 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740317e0 con 0x7f62840a0460 2023-10-30T12:19:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:32.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: dump: 2023-10-30T12:19:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:32.031 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: _finish_auth 0 2023-10-30T12:19:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: _check_auth_tickets 2023-10-30T12:19:32.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 5 monclient: authenticate success, global_id 244179 2023-10-30T12:19:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 1 -- 172.21.15.50:0/3320425870 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6274031940 con 0x7f62840a0460 2023-10-30T12:19:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 4 set_mon_vals no callback set 2023-10-30T12:19:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 1 -- 172.21.15.50:0/3320425870 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274031cd0 con 0x7f62840a0460 2023-10-30T12:19:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f62889f0640 10 monclient: dump: 2023-10-30T12:19:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:32.035 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 10 monclient: shutdown 2023-10-30T12:19:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 1 -- 172.21.15.50:0/3320425870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 msgr2=0x7f62840a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 1 --2- 172.21.15.50:0/3320425870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840a08b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f628409c460 tx=0x7f627402b880 comp rx=0 tx=0).stop 2023-10-30T12:19:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 1 -- 172.21.15.50:0/3320425870 shutdown_connections 2023-10-30T12:19:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 1 --2- 172.21.15.50:0/3320425870 >> v2:172.21.15.118:3300/0 conn(0x7f6284138e60 0x7f628413b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 1 --2- 172.21.15.50:0/3320425870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840a08b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.007+0000 7f628bc56640 1 -- 172.21.15.50:0/3320425870 >> 172.21.15.50:0/3320425870 conn(0x7f62840a4cc0 msgr2=0x7f6284073620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 -- 172.21.15.50:0/3320425870 shutdown_connections 2023-10-30T12:19:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 -- 172.21.15.50:0/3320425870 wait complete. 2023-10-30T12:19:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 10 monclient: build_initial_monmap 2023-10-30T12:19:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 10 monclient: monmap: 2023-10-30T12:19:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:32.038 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:32.039 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 librados: starting msgr at 2023-10-30T12:19:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 librados: starting objecter 2023-10-30T12:19:32.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 Processor -- start 2023-10-30T12:19:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 -- start start 2023-10-30T12:19:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 librados: setting wanted keys 2023-10-30T12:19:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 1 librados: calling monclient init 2023-10-30T12:19:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.008+0000 7f628bc56640 10 monclient: init 2023-10-30T12:19:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:19:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62840a0080 0x7f6284097390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): picked mon.noname-b con 0x7f62840a0080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): picked mon.noname-a con 0x7f62840a0460 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6284138e60 0x7f628412f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): picked mon.noname-c con 0x7f6284138e60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62840981c0 con 0x7f62840a0460 2023-10-30T12:19:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6284138940 con 0x7f62840a0080 2023-10-30T12:19:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6284132d50 con 0x7f6284138e60 2023-10-30T12:19:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628bc56640 10 monclient(hunting): authenticate will time out at 2017.192342s 2023-10-30T12:19:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628a1f3640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6284138e60 msgr2=0x7f628412f4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f628a1f3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6284138e60 0x7f628412f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48818/0 (socket says 172.21.15.50:48818) 2023-10-30T12:19:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 -- 172.21.15.50:0/4277773865 learned_addr learned my addr 172.21.15.50:0/4277773865 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): get_auth_request con 0x7f62840a0460 auth_method 0 2023-10-30T12:19:32.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62899f2640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62840a0080 0x7f6284097390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient(hunting): handle_auth_done global_id 244182 payload 824 2023-10-30T12:19:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 -- 172.21.15.50:0/4277773865 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6284138e60 msgr2=0x7f628412f4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6284138e60 0x7f628412f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 -- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62840a0080 msgr2=0x7f6284097390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62840a0080 0x7f6284097390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient: _finish_hunting 0 2023-10-30T12:19:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 monclient: found mon.noname-a 2023-10-30T12:19:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 -- 172.21.15.50:0/4277773865 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62740094d0 con 0x7f62840a0460 2023-10-30T12:19:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62899f2640 10 monclient: get_auth_request con 0x7f62840a0080 auth_method 0 2023-10-30T12:19:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62899f2640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62840a0080 0x7f6284097390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:19:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f62891f1640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840978d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6274000c00 tx=0x7f62740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f627a7fc640 10 client.?.objecter ms_handle_connect 0x7f62840a0460 2023-10-30T12:19:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.009+0000 7f627a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 1 -- 172.21.15.50:0/4277773865 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740329c0 con 0x7f62840a0460 2023-10-30T12:19:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:32.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: dump: 2023-10-30T12:19:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.052 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:32.053 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:32.054 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:19:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:19:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:32.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 1 -- 172.21.15.50:0/4277773865 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6274032b40 con 0x7f62840a0460 2023-10-30T12:19:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 5 monclient: authenticate success, global_id 244182 2023-10-30T12:19:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 10 monclient: _renew_subs 2023-10-30T12:19:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6284133030 con 0x7f62840a0460 2023-10-30T12:19:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 1 -- 172.21.15.50:0/4277773865 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740453f0 con 0x7f62840a0460 2023-10-30T12:19:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 10 monclient: dump: 2023-10-30T12:19:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:32.056 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:32.058 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 10 client.244182.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:32.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627bfff640 4 set_mon_vals no callback set 2023-10-30T12:19:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 10 monclient: _renew_subs 2023-10-30T12:19:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6284130f40 con 0x7f62840a0460 2023-10-30T12:19:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f628bc56640 1 librados: init done 2023-10-30T12:19:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.010+0000 7f627a7fc640 1 -- 172.21.15.50:0/4277773865 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6274045550 con 0x7f62840a0460 2023-10-30T12:19:32.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f627a7fc640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62680392a0 0x7f626803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f628bc56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f628bc56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f628bc56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f627a7fc640 1 -- 172.21.15.50:0/4277773865 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(596..596 src has 490..596) v4 ==== 6706+0+0 (secure 0 0 0) 0x7f627406d3a0 con 0x7f62840a0460 2023-10-30T12:19:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f627a7fc640 10 client.244182.objecter ms_dispatch 0x7f6284000b90 osd_map(596..596 src has 490..596) v4 2023-10-30T12:19:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f627a7fc640 3 client.244182.objecter handle_osd_map got epochs [596,596] > 0 2023-10-30T12:19:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f627a7fc640 3 client.244182.objecter handle_osd_map decoding full epoch 596 2023-10-30T12:19:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62840a0db0 con 0x7f62840a0460 2023-10-30T12:19:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f627a7fc640 20 client.244182.objecter dump_active .. 0 homeless 2023-10-30T12:19:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f62899f2640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62680392a0 0x7f626803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f62899f2640 10 monclient: get_auth_request con 0x7f62680392a0 auth_method 0 2023-10-30T12:19:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f627a7fc640 10 client.244182.objecter ms_handle_connect 0x7f62680392a0 2023-10-30T12:19:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.011+0000 7f62899f2640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62680392a0 0x7f626803b760 secure :-1 s=READY pgs=4294 cs=0 l=1 rev1=1 crypto rx=0x7f6280001700 tx=0x7f6280009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.015+0000 7f627a7fc640 1 -- 172.21.15.50:0/4277773865 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f627403b3f0 con 0x7f62840a0460 2023-10-30T12:19:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.015+0000 7f627a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.015+0000 7f627a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.258+0000 7f628bc56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.258+0000 7f628bc56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.258+0000 7f628bc56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.258+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f628404e160 con 0x7f62840a0460 2023-10-30T12:19:32.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.258+0000 7f627a7fc640 1 -- 172.21.15.50:0/4277773865 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f627404d360 con 0x7f62840a0460 2023-10-30T12:19:32.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.258+0000 7f627a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:32.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.258+0000 7f627a7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:32.266 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":198,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 10 librados: watch_flush enter 2023-10-30T12:19:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 10 librados: watch_flush exit 2023-10-30T12:19:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 20 client.244182.objecter shutdown clearing up homeless session... 2023-10-30T12:19:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 10 client.244182.objecter successfully canceled tick 2023-10-30T12:19:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62680392a0 msgr2=0x7f626803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:32.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62680392a0 0x7f626803b760 secure :-1 s=READY pgs=4294 cs=0 l=1 rev1=1 crypto rx=0x7f6280001700 tx=0x7f6280009040 comp rx=0 tx=0).stop 2023-10-30T12:19:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 10 monclient: shutdown 2023-10-30T12:19:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:32.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 msgr2=0x7f62840978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840978d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6274000c00 tx=0x7f62740046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 shutdown_connections 2023-10-30T12:19:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f62680392a0 0x7f626803b760 unknown :-1 s=CLOSED pgs=4294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6284138e60 0x7f628412f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62840a0460 0x7f62840978d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 --2- 172.21.15.50:0/4277773865 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62840a0080 0x7f6284097390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:32.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 >> 172.21.15.50:0/4277773865 conn(0x7f62840a4cc0 msgr2=0x7f6284071090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 shutdown_connections 2023-10-30T12:19:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 -- 172.21.15.50:0/4277773865 wait complete. 2023-10-30T12:19:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:32.260+0000 7f628bc56640 1 librados: shutdown 2023-10-30T12:19:35.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.391+0000 7f08e9a10640 10 monclient: get_monmap_and_config 2023-10-30T12:19:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.391+0000 7f08e9a10640 10 monclient: build_initial_monmap 2023-10-30T12:19:35.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.391+0000 7f08e9a10640 10 monclient: monmap: 2023-10-30T12:19:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:35.392857+0000 2023-10-30T12:19:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:35.392857+0000 2023-10-30T12:19:35.394 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:35.395 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:35.396 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:35.396 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 1 Processor -- start 2023-10-30T12:19:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 1 -- start start 2023-10-30T12:19:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient: init 2023-10-30T12:19:35.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-10-30T12:19:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f08e4098520 0x7f08e409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient(hunting): picked mon.noname-a con 0x7f08e4098520 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4098900 0x7f08e4098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient(hunting): picked mon.noname-b con 0x7f08e4098900 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08e4132740 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e409ad30 con 0x7f08e4132740 2023-10-30T12:19:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e409c460 con 0x7f08e4098520 2023-10-30T12:19:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e409b260 con 0x7f08e4098900 2023-10-30T12:19:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e9a10640 10 monclient(hunting): authenticate will time out at 2020.575923s 2023-10-30T12:19:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.392+0000 7f08e27fc640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4098900 msgr2=0x7f08e4098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e27fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4098900 0x7f08e4098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f08e4098520 0x7f08e409bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f08e4098520 0x7f08e409bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48824/0 (socket says 172.21.15.50:48824) 2023-10-30T12:19:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 -- 172.21.15.50:0/1480278068 learned_addr learned my addr 172.21.15.50:0/1480278068 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): get_auth_request con 0x7f08e4098520 auth_method 0 2023-10-30T12:19:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:35.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient(hunting): handle_auth_done global_id 244185 payload 274 2023-10-30T12:19:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 -- 172.21.15.50:0/1480278068 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4098900 msgr2=0x7f08e4098ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 --2- 172.21.15.50:0/1480278068 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4098900 0x7f08e4098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 -- 172.21.15.50:0/1480278068 >> v1:172.21.15.118:6789/0 conn(0x7f08e4132740 legacy=0x7f08e4132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:19:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 monclient: found mon.noname-a 2023-10-30T12:19:35.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.393+0000 7f08e2ffd640 1 -- 172.21.15.50:0/1480278068 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4072120 con 0x7f08e4098520 2023-10-30T12:19:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e2ffd640 1 --2- 172.21.15.50:0/1480278068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098520 0x7f08e409bf20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009850 tx=0x7f08d402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6905325404a67b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 1 -- 172.21.15.50:0/1480278068 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4004230 con 0x7f08e4098520 2023-10-30T12:19:35.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: dump: 2023-10-30T12:19:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.411 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:35.412 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:35.412 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:19:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:19:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 1 -- 172.21.15.50:0/1480278068 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d4004390 con 0x7f08e4098520 2023-10-30T12:19:35.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 5 monclient: authenticate success, global_id 244185 2023-10-30T12:19:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 1 -- 172.21.15.50:0/1480278068 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4003810 con 0x7f08e4098520 2023-10-30T12:19:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e1ffb640 10 monclient: dump: 2023-10-30T12:19:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.415 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:35.416 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:35.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 4 set_mon_vals no callback set 2023-10-30T12:19:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 10 monclient: shutdown 2023-10-30T12:19:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 1 -- 172.21.15.50:0/1480278068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098520 msgr2=0x7f08e409bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 1 --2- 172.21.15.50:0/1480278068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098520 0x7f08e409bf20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009850 tx=0x7f08d402b7e0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 1 -- 172.21.15.50:0/1480278068 shutdown_connections 2023-10-30T12:19:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 1 --2- 172.21.15.50:0/1480278068 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4098900 0x7f08e4098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 1 --2- 172.21.15.50:0/1480278068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098520 0x7f08e409bf20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.394+0000 7f08e9a10640 1 -- 172.21.15.50:0/1480278068 >> 172.21.15.50:0/1480278068 conn(0x7f08e40a05c0 msgr2=0x7f08e409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.395+0000 7f08e9a10640 1 -- 172.21.15.50:0/1480278068 shutdown_connections 2023-10-30T12:19:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.395+0000 7f08e9a10640 1 -- 172.21.15.50:0/1480278068 wait complete. 2023-10-30T12:19:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.395+0000 7f08e9a10640 10 monclient: build_initial_monmap 2023-10-30T12:19:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.395+0000 7f08e9a10640 10 monclient: monmap: 2023-10-30T12:19:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:35.419 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:35.421 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.395+0000 7f08e9a10640 1 librados: starting msgr at 2023-10-30T12:19:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.395+0000 7f08e9a10640 1 librados: starting objecter 2023-10-30T12:19:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.395+0000 7f08e9a10640 1 Processor -- start 2023-10-30T12:19:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 -- start start 2023-10-30T12:19:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 librados: setting wanted keys 2023-10-30T12:19:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 librados: calling monclient init 2023-10-30T12:19:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient: init 2023-10-30T12:19:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:19:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08e4098520 0x7f08e413ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): picked mon.noname-b con 0x7f08e4098520 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098900 0x7f08e413bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): picked mon.noname-a con 0x7f08e4098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4132740 0x7f08e413c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): picked mon.noname-c con 0x7f08e4132740 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e4139730 con 0x7f08e4098900 2023-10-30T12:19:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e413cef0 con 0x7f08e4098520 2023-10-30T12:19:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e4071df0 con 0x7f08e4132740 2023-10-30T12:19:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e9a10640 10 monclient(hunting): authenticate will time out at 2020.579739s 2023-10-30T12:19:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e37fe640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4132740 msgr2=0x7f08e413c4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e37fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4132740 0x7f08e413c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098900 0x7f08e413bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098900 0x7f08e413bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48838/0 (socket says 172.21.15.50:48838) 2023-10-30T12:19:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 1 -- 172.21.15.50:0/376457054 learned_addr learned my addr 172.21.15.50:0/376457054 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 10 monclient(hunting): get_auth_request con 0x7f08e4098900 auth_method 0 2023-10-30T12:19:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.396+0000 7f08e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:35.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e2ffd640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08e4098520 0x7f08e413ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 10 monclient(hunting): handle_auth_done global_id 244188 payload 824 2023-10-30T12:19:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 1 -- 172.21.15.50:0/376457054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4132740 msgr2=0x7f08e413c4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4132740 0x7f08e413c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 1 -- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08e4098520 msgr2=0x7f08e413ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08e4098520 0x7f08e413ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 10 monclient: _finish_hunting 0 2023-10-30T12:19:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 1 monclient: found mon.noname-a 2023-10-30T12:19:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 1 -- 172.21.15.50:0/376457054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d4009460 con 0x7f08e4098900 2023-10-30T12:19:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e2ffd640 10 monclient: get_auth_request con 0x7f08e4098520 auth_method 0 2023-10-30T12:19:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e2ffd640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08e4098520 0x7f08e413ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:19:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e27fc640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098900 0x7f08e413bf60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f08d800b580 tx=0x7f08d800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 client.?.objecter ms_handle_connect 0x7f08e4098900 2023-10-30T12:19:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 1 -- 172.21.15.50:0/376457054 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d8003dc0 con 0x7f08e4098900 2023-10-30T12:19:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: dump: 2023-10-30T12:19:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:35.433 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:35.433 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:35.433 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:35.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: _finish_auth 0 2023-10-30T12:19:35.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:19:35.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 5 monclient: authenticate success, global_id 244188 2023-10-30T12:19:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 1 -- 172.21.15.50:0/376457054 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d8005070 con 0x7f08e4098900 2023-10-30T12:19:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 10 monclient: _renew_subs 2023-10-30T12:19:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e40a3180 con 0x7f08e4098900 2023-10-30T12:19:35.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:35.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 1 -- 172.21.15.50:0/376457054 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d8005420 con 0x7f08e4098900 2023-10-30T12:19:35.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:35.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:35.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08cbfff640 10 monclient: dump: 2023-10-30T12:19:35.435 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:35.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:35.436 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.436 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:35.436 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:35.436 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:35.436 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:35.437 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:35.437 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:35.437 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:35.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 10 client.244188.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:35.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 10 monclient: _renew_subs 2023-10-30T12:19:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e17fa640 4 set_mon_vals no callback set 2023-10-30T12:19:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e4139fd0 con 0x7f08e4098900 2023-10-30T12:19:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.397+0000 7f08e9a10640 1 librados: init done 2023-10-30T12:19:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08cbfff640 1 -- 172.21.15.50:0/376457054 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f08d8005580 con 0x7f08e4098900 2023-10-30T12:19:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08e9a10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:35.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08e9a10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:35.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08e9a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:35.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08cbfff640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08d00392a0 0x7f08d003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:35.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e404e160 con 0x7f08e4098900 2023-10-30T12:19:35.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08cbfff640 1 -- 172.21.15.50:0/376457054 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(596..596 src has 490..596) v4 ==== 6706+0+0 (secure 0 0 0) 0x7f08d804ade0 con 0x7f08e4098900 2023-10-30T12:19:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08cbfff640 10 client.244188.objecter ms_dispatch 0x7f08e4000b90 osd_map(596..596 src has 490..596) v4 2023-10-30T12:19:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08cbfff640 3 client.244188.objecter handle_osd_map got epochs [596,596] > 0 2023-10-30T12:19:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08cbfff640 3 client.244188.objecter handle_osd_map decoding full epoch 596 2023-10-30T12:19:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08cbfff640 20 client.244188.objecter dump_active .. 0 homeless 2023-10-30T12:19:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08e2ffd640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08d00392a0 0x7f08d003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.398+0000 7f08e2ffd640 10 monclient: get_auth_request con 0x7f08d00392a0 auth_method 0 2023-10-30T12:19:35.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.399+0000 7f08cbfff640 10 client.244188.objecter ms_handle_connect 0x7f08d00392a0 2023-10-30T12:19:35.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.399+0000 7f08e2ffd640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08d00392a0 0x7f08d003b760 secure :-1 s=READY pgs=4295 cs=0 l=1 rev1=1 crypto rx=0x7f08d4035000 tx=0x7f08d4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:35.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.403+0000 7f08cbfff640 1 -- 172.21.15.50:0/376457054 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f08d8005830 con 0x7f08e4098900 2023-10-30T12:19:35.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.403+0000 7f08cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:35.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.403+0000 7f08cbfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:35.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.645+0000 7f08e9a10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:35.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.645+0000 7f08e9a10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:35.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.645+0000 7f08e9a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:35.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.645+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f08e409f2b0 con 0x7f08e4098900 2023-10-30T12:19:35.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.645+0000 7f08cbfff640 1 -- 172.21.15.50:0/376457054 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f08d8013030 con 0x7f08e4098900 2023-10-30T12:19:35.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.645+0000 7f08cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:35.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.645+0000 7f08cbfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:35.649 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":198,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:35.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 10 librados: watch_flush enter 2023-10-30T12:19:35.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 10 librados: watch_flush exit 2023-10-30T12:19:35.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 20 client.244188.objecter shutdown clearing up homeless session... 2023-10-30T12:19:35.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 10 client.244188.objecter successfully canceled tick 2023-10-30T12:19:35.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08d00392a0 msgr2=0x7f08d003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:35.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08d00392a0 0x7f08d003b760 secure :-1 s=READY pgs=4295 cs=0 l=1 rev1=1 crypto rx=0x7f08d4035000 tx=0x7f08d4008030 comp rx=0 tx=0).stop 2023-10-30T12:19:35.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 10 monclient: shutdown 2023-10-30T12:19:35.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:35.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098900 msgr2=0x7f08e413bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:35.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098900 0x7f08e413bf60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f08d800b580 tx=0x7f08d800ba40 comp rx=0 tx=0).stop 2023-10-30T12:19:35.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 shutdown_connections 2023-10-30T12:19:35.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08d00392a0 0x7f08d003b760 unknown :-1 s=CLOSED pgs=4295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f08e4132740 0x7f08e413c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08e4098900 0x7f08e413bf60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 --2- 172.21.15.50:0/376457054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08e4098520 0x7f08e413ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:35.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 >> 172.21.15.50:0/376457054 conn(0x7f08e40a05c0 msgr2=0x7f08e413b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:35.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 shutdown_connections 2023-10-30T12:19:35.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.647+0000 7f08e9a10640 1 -- 172.21.15.50:0/376457054 wait complete. 2023-10-30T12:19:35.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:35.648+0000 7f08e9a10640 1 librados: shutdown 2023-10-30T12:19:38.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:38.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.777+0000 7f9d2fb16640 10 monclient: get_monmap_and_config 2023-10-30T12:19:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.777+0000 7f9d2fb16640 10 monclient: build_initial_monmap 2023-10-30T12:19:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.777+0000 7f9d2fb16640 10 monclient: monmap: 2023-10-30T12:19:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:38.779 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:38.778098+0000 2023-10-30T12:19:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:38.778098+0000 2023-10-30T12:19:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:38.780 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:38.781 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:38.781 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:38.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.777+0000 7f9d2fb16640 1 Processor -- start 2023-10-30T12:19:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.777+0000 7f9d2fb16640 1 -- start start 2023-10-30T12:19:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.777+0000 7f9d2fb16640 10 monclient: init 2023-10-30T12:19:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-10-30T12:19:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): picked mon.noname-b con 0x7f9d280a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9d28099a40 0x7f9d28132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:38.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): picked mon.noname-c con 0x7f9d28099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9d280a2350 0x7f9d280a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): picked mon.noname-a con 0x7f9d280a2350 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d280a3d70 con 0x7f9d280a2350 2023-10-30T12:19:38.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d28133120 con 0x7f9d28099a40 2023-10-30T12:19:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d280a4f40 con 0x7f9d280a4b60 2023-10-30T12:19:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2fb16640 10 monclient(hunting): authenticate will time out at 2023.961397s 2023-10-30T12:19:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2d8b2640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 msgr2=0x7f9d281326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2d8b2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9d280a2350 0x7f9d280a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9d280a2350 0x7f9d280a27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33126/0 (socket says 172.21.15.50:33126) 2023-10-30T12:19:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/4049741962 learned_addr learned my addr 172.21.15.50:0/4049741962 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:38.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): get_auth_request con 0x7f9d280a2350 auth_method 0 2023-10-30T12:19:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:38.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2d0b1640 1 --2- 172.21.15.50:0/4049741962 >> v2:172.21.15.118:3300/0 conn(0x7f9d28099a40 0x7f9d28132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2d0b1640 10 monclient(hunting): get_auth_request con 0x7f9d28099a40 auth_method 0 2023-10-30T12:19:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2d0b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2d0b1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2d0b1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_done global_id 244191 payload 274 2023-10-30T12:19:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/4049741962 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 msgr2=0x7f9d281326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 --2- 172.21.15.50:0/4049741962 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/4049741962 >> v2:172.21.15.118:3300/0 conn(0x7f9d28099a40 msgr2=0x7f9d28132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 --2- 172.21.15.50:0/4049741962 >> v2:172.21.15.118:3300/0 conn(0x7f9d28099a40 0x7f9d28132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient: _finish_hunting 0 2023-10-30T12:19:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 monclient: found mon.noname-a 2023-10-30T12:19:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.778+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/4049741962 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d280a4150 con 0x7f9d280a2350 2023-10-30T12:19:38.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2e0b3640 1 --2- 172.21.15.50:0/4049741962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d280a2350 0x7f9d280a27c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d280a5210 tx=0x7f9d2002bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da89efcfc1b669 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 1 -- 172.21.15.50:0/4049741962 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2002c980 con 0x7f9d280a2350 2023-10-30T12:19:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: dump: 2023-10-30T12:19:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:38.800 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:38.801 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:38.802 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: _finish_auth 0 2023-10-30T12:19:38.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: _check_auth_tickets 2023-10-30T12:19:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2fb16640 5 monclient: authenticate success, global_id 244191 2023-10-30T12:19:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 1 -- 172.21.15.50:0/4049741962 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2002cae0 con 0x7f9d280a2350 2023-10-30T12:19:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2fb16640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:38.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2fb16640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2fb16640 4 set_mon_vals no callback set 2023-10-30T12:19:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 1 -- 172.21.15.50:0/4049741962 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2002ce70 con 0x7f9d280a2350 2023-10-30T12:19:38.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.779+0000 7f9d2c8b0640 10 monclient: dump: 2023-10-30T12:19:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:38.805 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:38.807 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:38.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 10 monclient: shutdown 2023-10-30T12:19:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 -- 172.21.15.50:0/4049741962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d280a2350 msgr2=0x7f9d280a27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/4049741962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d280a2350 0x7f9d280a27c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d280a5210 tx=0x7f9d2002bde0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 -- 172.21.15.50:0/4049741962 shutdown_connections 2023-10-30T12:19:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/4049741962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d280a2350 0x7f9d280a27c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/4049741962 >> v2:172.21.15.118:3300/0 conn(0x7f9d28099a40 0x7f9d28132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/4049741962 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 -- 172.21.15.50:0/4049741962 >> 172.21.15.50:0/4049741962 conn(0x7f9d280a8060 msgr2=0x7f9d280a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 -- 172.21.15.50:0/4049741962 shutdown_connections 2023-10-30T12:19:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.780+0000 7f9d2fb16640 1 -- 172.21.15.50:0/4049741962 wait complete. 2023-10-30T12:19:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient: build_initial_monmap 2023-10-30T12:19:38.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient: monmap: 2023-10-30T12:19:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:38.811 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:38.812 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:38.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 librados: starting msgr at 2023-10-30T12:19:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 librados: starting objecter 2023-10-30T12:19:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 Processor -- start 2023-10-30T12:19:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 -- start start 2023-10-30T12:19:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 librados: setting wanted keys 2023-10-30T12:19:38.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 librados: calling monclient init 2023-10-30T12:19:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient: init 2023-10-30T12:19:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:19:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d280a2350 0x7f9d281ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): picked mon.noname-b con 0x7f9d280a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:38.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): picked mon.noname-c con 0x7f9d280a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2809e470 0x7f9d2809e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): picked mon.noname-a con 0x7f9d2809e470 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:38.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2811e100 con 0x7f9d2809e470 2023-10-30T12:19:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2811ecb0 con 0x7f9d280a2350 2023-10-30T12:19:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2811f550 con 0x7f9d280a4b60 2023-10-30T12:19:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2fb16640 10 monclient(hunting): authenticate will time out at 2023.964848s 2023-10-30T12:19:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2d0b1640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 msgr2=0x7f9d281cae00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2d0b1640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2e0b3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2809e470 0x7f9d2809e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2e0b3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2809e470 0x7f9d2809e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33130/0 (socket says 172.21.15.50:33130) 2023-10-30T12:19:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/1582566699 learned_addr learned my addr 172.21.15.50:0/1582566699 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2e0b3640 10 monclient(hunting): get_auth_request con 0x7f9d2809e470 auth_method 0 2023-10-30T12:19:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2e0b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2e0b3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:38.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.781+0000 7f9d2e0b3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d280a2350 0x7f9d281ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 10 monclient(hunting): get_auth_request con 0x7f9d280a2350 auth_method 0 2023-10-30T12:19:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:38.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2d8b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 10 monclient(hunting): handle_auth_done global_id 244194 payload 824 2023-10-30T12:19:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/1582566699 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 msgr2=0x7f9d281cae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d280a2350 msgr2=0x7f9d281ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d280a2350 0x7f9d281ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:38.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 10 monclient: _finish_hunting 0 2023-10-30T12:19:38.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 1 monclient: found mon.noname-a 2023-10-30T12:19:38.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:38.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:38.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 1 -- 172.21.15.50:0/1582566699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d20009b30 con 0x7f9d2809e470 2023-10-30T12:19:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2e0b3640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2809e470 0x7f9d2809e920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d200093e0 tx=0x7f9d20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 client.?.objecter ms_handle_connect 0x7f9d2809e470 2023-10-30T12:19:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 1 -- 172.21.15.50:0/1582566699 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d200079b0 con 0x7f9d2809e470 2023-10-30T12:19:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: dump: 2023-10-30T12:19:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.824 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:38.825 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:38.826 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:19:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:19:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 1 -- 172.21.15.50:0/1582566699 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d20007b30 con 0x7f9d2809e470 2023-10-30T12:19:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 5 monclient: authenticate success, global_id 244194 2023-10-30T12:19:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 10 monclient: _renew_subs 2023-10-30T12:19:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d280a06f0 con 0x7f9d2809e470 2023-10-30T12:19:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 1 -- 172.21.15.50:0/1582566699 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2003e450 con 0x7f9d2809e470 2023-10-30T12:19:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:38.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:38.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1e7fc640 10 monclient: dump: 2023-10-30T12:19:38.828 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:38.828 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:38.829 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.829 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:38.829 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:38.829 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:38.830 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:38.830 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:38.830 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:38.830 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:38.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 10 client.244194.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:38.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 10 monclient: _renew_subs 2023-10-30T12:19:38.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:38.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d280a0d50 con 0x7f9d2809e470 2023-10-30T12:19:38.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d1ffff640 4 set_mon_vals no callback set 2023-10-30T12:19:38.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.782+0000 7f9d2fb16640 1 librados: init done 2023-10-30T12:19:38.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d1e7fc640 1 -- 172.21.15.50:0/1582566699 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9d2003e5b0 con 0x7f9d2809e470 2023-10-30T12:19:38.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d2fb16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:38.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d2fb16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:38.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d2fb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:38.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2811ee20 con 0x7f9d2809e470 2023-10-30T12:19:38.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d1e7fc640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9d0c0392a0 0x7f9d0c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:38.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d1e7fc640 1 -- 172.21.15.50:0/1582566699 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(596..596 src has 490..596) v4 ==== 6706+0+0 (secure 0 0 0) 0x7f9d2006d410 con 0x7f9d2809e470 2023-10-30T12:19:38.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d1e7fc640 10 client.244194.objecter ms_dispatch 0x7f9d28000b90 osd_map(596..596 src has 490..596) v4 2023-10-30T12:19:38.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d1e7fc640 3 client.244194.objecter handle_osd_map got epochs [596,596] > 0 2023-10-30T12:19:38.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.783+0000 7f9d1e7fc640 3 client.244194.objecter handle_osd_map decoding full epoch 596 2023-10-30T12:19:38.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.784+0000 7f9d1e7fc640 20 client.244194.objecter dump_active .. 0 homeless 2023-10-30T12:19:38.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.784+0000 7f9d2d8b2640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9d0c0392a0 0x7f9d0c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:38.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.784+0000 7f9d2d8b2640 10 monclient: get_auth_request con 0x7f9d0c0392a0 auth_method 0 2023-10-30T12:19:38.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.784+0000 7f9d1e7fc640 10 client.244194.objecter ms_handle_connect 0x7f9d0c0392a0 2023-10-30T12:19:38.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.784+0000 7f9d2d8b2640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9d0c0392a0 0x7f9d0c03b760 secure :-1 s=READY pgs=4296 cs=0 l=1 rev1=1 crypto rx=0x7f9d240093d0 tx=0x7f9d24009360 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:38.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.788+0000 7f9d1e7fc640 1 -- 172.21.15.50:0/1582566699 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d200477c0 con 0x7f9d2809e470 2023-10-30T12:19:38.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.788+0000 7f9d1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:38.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:38.788+0000 7f9d1e7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.037+0000 7f9d2fb16640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.037+0000 7f9d2fb16640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.037+0000 7f9d2fb16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.037+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9d280a06f0 con 0x7f9d2809e470 2023-10-30T12:19:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.038+0000 7f9d1e7fc640 1 -- 172.21.15.50:0/1582566699 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7f9d20043620 con 0x7f9d2809e470 2023-10-30T12:19:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.038+0000 7f9d1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.038+0000 7f9d1e7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:39.041 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":198,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.039+0000 7f9d2fb16640 10 librados: watch_flush enter 2023-10-30T12:19:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.039+0000 7f9d2fb16640 10 librados: watch_flush exit 2023-10-30T12:19:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.039+0000 7f9d2fb16640 20 client.244194.objecter shutdown clearing up homeless session... 2023-10-30T12:19:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.039+0000 7f9d2fb16640 10 client.244194.objecter successfully canceled tick 2023-10-30T12:19:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9d0c0392a0 msgr2=0x7f9d0c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9d0c0392a0 0x7f9d0c03b760 secure :-1 s=READY pgs=4296 cs=0 l=1 rev1=1 crypto rx=0x7f9d240093d0 tx=0x7f9d24009360 comp rx=0 tx=0).stop 2023-10-30T12:19:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 10 monclient: shutdown 2023-10-30T12:19:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2809e470 msgr2=0x7f9d2809e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2809e470 0x7f9d2809e920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d200093e0 tx=0x7f9d20004a40 comp rx=0 tx=0).stop 2023-10-30T12:19:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 shutdown_connections 2023-10-30T12:19:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9d0c0392a0 0x7f9d0c03b760 unknown :-1 s=CLOSED pgs=4296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2809e470 0x7f9d2809e920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d280a4b60 0x7f9d281cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:39.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 --2- 172.21.15.50:0/1582566699 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d280a2350 0x7f9d281ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 >> 172.21.15.50:0/1582566699 conn(0x7f9d280a8060 msgr2=0x7f9d28131af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 shutdown_connections 2023-10-30T12:19:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 -- 172.21.15.50:0/1582566699 wait complete. 2023-10-30T12:19:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:39.040+0000 7f9d2fb16640 1 librados: shutdown 2023-10-30T12:19:42.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:19:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.168+0000 7f65fd0fc640 10 monclient: get_monmap_and_config 2023-10-30T12:19:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.168+0000 7f65fd0fc640 10 monclient: build_initial_monmap 2023-10-30T12:19:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.168+0000 7f65fd0fc640 10 monclient: monmap: 2023-10-30T12:19:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:42.169237+0000 2023-10-30T12:19:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:42.169237+0000 2023-10-30T12:19:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:42.172 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.168+0000 7f65fd0fc640 1 Processor -- start 2023-10-30T12:19:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.168+0000 7f65fd0fc640 1 -- start start 2023-10-30T12:19:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.168+0000 7f65fd0fc640 10 monclient: init 2023-10-30T12:19:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-10-30T12:19:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 0x7f65f81326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient(hunting): picked mon.noname-b con 0x7f65f80a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65f8099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient(hunting): picked mon.noname-a con 0x7f65f8132be0 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f80a4f40 con 0x7f65f8099a40 2023-10-30T12:19:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f80a3d70 con 0x7f65f8132be0 2023-10-30T12:19:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80a4150 con 0x7f65f80a4b60 2023-10-30T12:19:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f6d9d640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 msgr2=0x7f65f81326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65fd0fc640 10 monclient(hunting): authenticate will time out at 2027.352445s 2023-10-30T12:19:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f6d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 0x7f65f81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33142/0 (socket says 172.21.15.50:33142) 2023-10-30T12:19:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 learned_addr learned my addr 172.21.15.50:0/434562304 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 10 monclient(hunting): get_auth_request con 0x7f65f8132be0 auth_method 0 2023-10-30T12:19:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:19:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.169+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:42.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f6d9d640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 msgr2=0x7f65f81326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 0x7f65f81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:19:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 10 monclient(hunting): get_auth_request con 0x7f65f8132be0 auth_method 0 2023-10-30T12:19:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:19:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:42.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.370+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:19:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f6d9d640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 msgr2=0x7f65f81326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 0x7f65f81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:19:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 10 monclient(hunting): get_auth_request con 0x7f65f8132be0 auth_method 0 2023-10-30T12:19:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:42.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:19:42.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:42.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:42.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:42.771+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:19:43.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f6d9d640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 msgr2=0x7f65f81326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:43.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 0x7f65f81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:19:43.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:43.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 10 monclient(hunting): get_auth_request con 0x7f65f8132be0 auth_method 0 2023-10-30T12:19:43.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:43.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:43.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:43.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:19:43.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:43.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:43.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:43.572+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:19:45.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient(hunting): tick 2023-10-30T12:19:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:19:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 monclient(hunting): continuing hunt 2023-10-30T12:19:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:19:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 msgr2=0x7f65f81326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65f80a4b60 0x7f65f81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 msgr2=0x7f65f80a2a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.50:3300/0 conn(0x7f65f8132be0 0x7f65f80a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 -- 172.21.15.50:0/434562304 >> v1:172.21.15.118:6789/0 conn(0x7f65f8099a40 legacy=0x7f65f80a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient: _add_conns ranks=[0,4] 2023-10-30T12:19:45.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65e40036e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient(hunting): picked mon.noname-c con 0x7f65e4007410 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 -- 172.21.15.50:0/434562304 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f809a440 con 0x7f65e40036e0 2023-10-30T12:19:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:45.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 1 -- 172.21.15.50:0/434562304 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f80a4150 con 0x7f65e4007410 2023-10-30T12:19:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.169+0000 7f65f4d99640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 10 monclient(hunting): get_auth_request con 0x7f65e4007410 auth_method 0 2023-10-30T12:19:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:19:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:45.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.170+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:45.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:45.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 10 monclient(hunting): get_auth_request con 0x7f65e4007410 auth_method 0 2023-10-30T12:19:45.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:45.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:45.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:45.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:19:45.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:45.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:45.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.371+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:19:45.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:45.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 10 monclient(hunting): get_auth_request con 0x7f65e4007410 auth_method 0 2023-10-30T12:19:45.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:45.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:45.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:45.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:19:45.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:45.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:45.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:45.772+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:19:46.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:46.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 10 monclient(hunting): get_auth_request con 0x7f65e4007410 auth_method 0 2023-10-30T12:19:46.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:46.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:46.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:46.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:19:46.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:19:46.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:19:46.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.573+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:19:46.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.693+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608023738 0 0) 0x7f65f809a440 con 0x7f65e40036e0 2023-10-30T12:19:46.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.693+0000 7f65f5d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:19:46.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.693+0000 7f65f5d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:46.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.693+0000 7f65f5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:46.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.693+0000 7f65f5d9b640 10 monclient(hunting): my global_id is 244197 2023-10-30T12:19:46.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.693+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e0003520 con 0x7f65e40036e0 2023-10-30T12:19:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415044408 0 0) 0x7f65e0003520 con 0x7f65e40036e0 2023-10-30T12:19:46.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 msgr2=0x7f65e40098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 --2- 172.21.15.50:0/434562304 >> v2:172.21.15.118:3300/0 conn(0x7f65e4007410 0x7f65e40098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:19:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:19:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:19:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e400b040 con 0x7f65e40036e0 2023-10-30T12:19:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: _finish_auth 0 2023-10-30T12:19:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:19:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f65e8004340 con 0x7f65e40036e0 2023-10-30T12:19:46.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:46.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:19:46.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: dump: 2023-10-30T12:19:46.702 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:46.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:46.706 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.706 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.706 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:46.707 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:46.707 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:46.707 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:46.707 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:46.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:19:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 >> v1:172.21.15.50:6789/0 conn(0x7f65e40036e0 legacy=0x7f65e4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 1 -- 172.21.15.50:0/434562304 reap_dead start 2023-10-30T12:19:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65e0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): picked mon.a con 0x7f65e0003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): picked mon.c con 0x7f65e00047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): picked mon.b con 0x7f65e00051d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80a4150 con 0x7f65e0003970 2023-10-30T12:19:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e000c100 con 0x7f65e00051d0 2023-10-30T12:19:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e000c2c0 con 0x7f65e00047c0 2023-10-30T12:19:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f5d9b640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:19:46.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f659c640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 msgr2=0x7f65e0004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f659c640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65e0004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65e0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): get_auth_request con 0x7f65e0003970 auth_method 0 2023-10-30T12:19:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:46.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:46.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:19:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 10 monclient(hunting): get_auth_request con 0x7f65e00051d0 auth_method 0 2023-10-30T12:19:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:46.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:46.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:19:46.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_done global_id 244197 payload 293 2023-10-30T12:19:46.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 msgr2=0x7f65e0004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:46.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65e0004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 msgr2=0x7f65e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:46.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65e000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:19:46.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 1 monclient: found mon.a 2023-10-30T12:19:46.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:46.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:46.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 1 -- 172.21.15.50:0/434562304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0003520 con 0x7f65e0003970 2023-10-30T12:19:46.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient: _finish_auth 0 2023-10-30T12:19:46.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 10 monclient: _check_auth_tickets 2023-10-30T12:19:46.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65f6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:46.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65fd0fc640 5 monclient: authenticate success, global_id 244197 2023-10-30T12:19:46.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.696+0000 7f65fd0fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:46.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65e0003d70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65e8003980 tx=0x7f65e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad28f47702f24e35 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:46.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e8006270 con 0x7f65e0003970 2023-10-30T12:19:46.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:46.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:46.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 10 monclient: dump: 2023-10-30T12:19:46.719 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:46.720 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:46.720 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.720 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.720 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:46.720 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:46.720 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:46.721 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:46.721 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:46.721 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:46.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e80063f0 con 0x7f65e0003970 2023-10-30T12:19:46.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65fd0fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:46.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:46.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 1 -- 172.21.15.50:0/434562304 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e802cbd0 con 0x7f65e0003970 2023-10-30T12:19:46.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65fd0fc640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:46.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65fd0fc640 4 set_mon_vals no callback set 2023-10-30T12:19:46.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:46.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:46.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.698+0000 7f65f5d9b640 10 monclient: dump: 2023-10-30T12:19:46.723 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:46.723 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:46.723 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.723 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.724 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:46.724 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:46.724 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:46.724 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:46.724 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:46.725 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 10 monclient: shutdown 2023-10-30T12:19:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 -- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 msgr2=0x7f65e0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65e0003d70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65e8003980 tx=0x7f65e00057f0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 -- 172.21.15.50:0/434562304 shutdown_connections 2023-10-30T12:19:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65e000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65e0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/434562304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65e0003d70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 -- 172.21.15.50:0/434562304 >> 172.21.15.50:0/434562304 conn(0x7f65f80a8060 msgr2=0x7f65f80a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 -- 172.21.15.50:0/434562304 shutdown_connections 2023-10-30T12:19:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.699+0000 7f65fd0fc640 1 -- 172.21.15.50:0/434562304 wait complete. 2023-10-30T12:19:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.700+0000 7f65fd0fc640 10 monclient: build_initial_monmap 2023-10-30T12:19:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.700+0000 7f65fd0fc640 10 monclient: monmap: 2023-10-30T12:19:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:46.734 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:46.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.701+0000 7f65fd0fc640 1 librados: starting msgr at 2023-10-30T12:19:46.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.701+0000 7f65fd0fc640 1 librados: starting objecter 2023-10-30T12:19:46.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.701+0000 7f65fd0fc640 1 Processor -- start 2023-10-30T12:19:46.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.701+0000 7f65fd0fc640 1 -- start start 2023-10-30T12:19:46.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.701+0000 7f65fd0fc640 1 librados: setting wanted keys 2023-10-30T12:19:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.701+0000 7f65fd0fc640 1 librados: calling monclient init 2023-10-30T12:19:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.701+0000 7f65fd0fc640 10 monclient: init 2023-10-30T12:19:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:46.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65f81c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:46.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): picked mon.noname-a con 0x7f65e0003970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:46.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65f81316f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:46.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): picked mon.noname-c con 0x7f65e00047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:46.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65f81c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:46.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): picked mon.noname-b con 0x7f65e00051d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:46.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:46.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f811fde0 con 0x7f65e0003970 2023-10-30T12:19:46.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:46.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80a0560 con 0x7f65e00051d0 2023-10-30T12:19:46.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:46.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80a1940 con 0x7f65e00047c0 2023-10-30T12:19:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65fd0fc640 10 monclient(hunting): authenticate will time out at 2031.885514s 2023-10-30T12:19:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f659c640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 msgr2=0x7f65f81316f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f659c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65f81316f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:46.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65f81c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65f81c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33184/0 (socket says 172.21.15.50:33184) 2023-10-30T12:19:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 1 -- 172.21.15.50:0/2087634690 learned_addr learned my addr 172.21.15.50:0/2087634690 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 10 monclient(hunting): get_auth_request con 0x7f65e0003970 auth_method 0 2023-10-30T12:19:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:46.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:46.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:46.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:46.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:46.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:46.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f759e640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65f81c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:46.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f759e640 10 monclient(hunting): get_auth_request con 0x7f65e00051d0 auth_method 0 2023-10-30T12:19:46.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:46.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:46.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.702+0000 7f65f759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:46.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:46.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:46.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:46.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 10 monclient(hunting): handle_auth_done global_id 244200 payload 824 2023-10-30T12:19:46.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 1 -- 172.21.15.50:0/2087634690 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 msgr2=0x7f65f81316f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:46.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65f81316f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 1 -- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 msgr2=0x7f65f81c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:46.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65f81c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:19:46.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 1 monclient: found mon.noname-a 2023-10-30T12:19:46.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:46.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:46.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 1 -- 172.21.15.50:0/2087634690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f809fcc0 con 0x7f65e0003970 2023-10-30T12:19:46.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65f6d9d640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65f81c6a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65e800c3b0 tx=0x7f65e802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:46.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65df7fe640 10 client.?.objecter ms_handle_connect 0x7f65e0003970 2023-10-30T12:19:46.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.703+0000 7f65df7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:46.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 1 -- 172.21.15.50:0/2087634690 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e803c040 con 0x7f65e0003970 2023-10-30T12:19:46.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:46.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:46.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: dump: 2023-10-30T12:19:46.747 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:46.747 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:46.747 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.747 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.748 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:46.748 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:46.748 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:46.748 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:46.748 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:46.748 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: _finish_auth 0 2023-10-30T12:19:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:19:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 1 -- 172.21.15.50:0/2087634690 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e8004930 con 0x7f65e0003970 2023-10-30T12:19:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 5 monclient: authenticate success, global_id 244200 2023-10-30T12:19:46.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 10 monclient: _renew_subs 2023-10-30T12:19:46.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:46.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f811f880 con 0x7f65e0003970 2023-10-30T12:19:46.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:46.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 1 -- 172.21.15.50:0/2087634690 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e8004ce0 con 0x7f65e0003970 2023-10-30T12:19:46.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:46.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:46.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65df7fe640 10 monclient: dump: 2023-10-30T12:19:46.751 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:46.751 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:46.752 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.752 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:46.752 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:46.752 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:46.752 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:46.753 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:46.753 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:46.753 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:46.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 10 client.244200.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:46.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 10 monclient: _renew_subs 2023-10-30T12:19:46.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65f559a640 4 set_mon_vals no callback set 2023-10-30T12:19:46.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:46.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f81c7640 con 0x7f65e0003970 2023-10-30T12:19:46.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.704+0000 7f65fd0fc640 1 librados: init done 2023-10-30T12:19:46.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.705+0000 7f65df7fe640 1 -- 172.21.15.50:0/2087634690 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f65e8032070 con 0x7f65e0003970 2023-10-30T12:19:46.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.705+0000 7f65fd0fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:46.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.705+0000 7f65df7fe640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65e4039ac0 0x7f65e403bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:46.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.705+0000 7f65fd0fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:46.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.705+0000 7f65fd0fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:46.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.705+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f804e160 con 0x7f65e0003970 2023-10-30T12:19:46.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65df7fe640 1 -- 172.21.15.50:0/2087634690 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(597..597 src has 490..597) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f65e8004e40 con 0x7f65e0003970 2023-10-30T12:19:46.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65df7fe640 10 client.244200.objecter ms_dispatch 0x7f65f8000b90 osd_map(597..597 src has 490..597) v4 2023-10-30T12:19:46.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65df7fe640 3 client.244200.objecter handle_osd_map got epochs [597,597] > 0 2023-10-30T12:19:46.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65df7fe640 3 client.244200.objecter handle_osd_map decoding full epoch 597 2023-10-30T12:19:46.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65df7fe640 20 client.244200.objecter dump_active .. 0 homeless 2023-10-30T12:19:46.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65f659c640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65e4039ac0 0x7f65e403bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:46.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65f659c640 10 monclient: get_auth_request con 0x7f65e4039ac0 auth_method 0 2023-10-30T12:19:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65f659c640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65e4039ac0 0x7f65e403bf80 secure :-1 s=READY pgs=4297 cs=0 l=1 rev1=1 crypto rx=0x7f65ec007fd0 tx=0x7f65ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.706+0000 7f65df7fe640 10 client.244200.objecter ms_handle_connect 0x7f65e4039ac0 2023-10-30T12:19:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.711+0000 7f65df7fe640 1 -- 172.21.15.50:0/2087634690 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65e8007e40 con 0x7f65e0003970 2023-10-30T12:19:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.711+0000 7f65df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.711+0000 7f65df7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:46.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.961+0000 7f65fd0fc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:19:46.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.961+0000 7f65fd0fc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:46.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.961+0000 7f65fd0fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:46.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.961+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f65f80a71a0 con 0x7f65e0003970 2023-10-30T12:19:46.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.961+0000 7f65df7fe640 1 -- 172.21.15.50:0/2087634690 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f65e804a3e0 con 0x7f65e0003970 2023-10-30T12:19:46.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.961+0000 7f65df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:19:46.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.961+0000 7f65df7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:46.965 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":202,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:19:46.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 10 librados: watch_flush enter 2023-10-30T12:19:46.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 10 librados: watch_flush exit 2023-10-30T12:19:46.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 20 client.244200.objecter shutdown clearing up homeless session... 2023-10-30T12:19:46.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 10 client.244200.objecter successfully canceled tick 2023-10-30T12:19:46.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65e4039ac0 msgr2=0x7f65e403bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:46.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65e4039ac0 0x7f65e403bf80 secure :-1 s=READY pgs=4297 cs=0 l=1 rev1=1 crypto rx=0x7f65ec007fd0 tx=0x7f65ec009040 comp rx=0 tx=0).stop 2023-10-30T12:19:46.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 10 monclient: shutdown 2023-10-30T12:19:46.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:46.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 msgr2=0x7f65f81c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:46.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.963+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65f81c6a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65e800c3b0 tx=0x7f65e802e070 comp rx=0 tx=0).stop 2023-10-30T12:19:46.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 shutdown_connections 2023-10-30T12:19:46.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65e4039ac0 0x7f65e403bf80 unknown :-1 s=CLOSED pgs=4297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65e00051d0 0x7f65f81c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f65e00047c0 0x7f65f81316f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 --2- 172.21.15.50:0/2087634690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e0003970 0x7f65f81c6a00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:46.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 >> 172.21.15.50:0/2087634690 conn(0x7f65f80a8060 msgr2=0x7f65f80a0960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:46.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 shutdown_connections 2023-10-30T12:19:46.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 -- 172.21.15.50:0/2087634690 wait complete. 2023-10-30T12:19:46.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:46.964+0000 7f65fd0fc640 1 librados: shutdown 2023-10-30T12:19:46.976 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:19:46.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:19:46.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:19:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.082+0000 7f9550f01640 10 monclient: get_monmap_and_config 2023-10-30T12:19:47.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.082+0000 7f9550f01640 10 monclient: build_initial_monmap 2023-10-30T12:19:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.082+0000 7f9550f01640 10 monclient: monmap: 2023-10-30T12:19:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:47.083168+0000 2023-10-30T12:19:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:47.083168+0000 2023-10-30T12:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.082+0000 7f9550f01640 1 Processor -- start 2023-10-30T12:19:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.082+0000 7f9550f01640 1 -- start start 2023-10-30T12:19:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.082+0000 7f9550f01640 10 monclient: init 2023-10-30T12:19:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T12:19:47.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f954c0a2ae0 0x7f954c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient(hunting): picked mon.noname-c con 0x7f954c0a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 0x7f954c1356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient(hunting): picked mon.noname-b con 0x7f954c0a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f954c0a32e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:19:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c0a4190 con 0x7f954c0a32e0 2023-10-30T12:19:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c0a2990 con 0x7f954c0a2ae0 2023-10-30T12:19:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0a1630 con 0x7f954c0a2ec0 2023-10-30T12:19:47.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9550f01640 10 monclient(hunting): authenticate will time out at 2032.266378s 2023-10-30T12:19:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9549d9b640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 msgr2=0x7f954c1356e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f9549d9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 0x7f954c1356e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954ad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f954c0a32e0 0x7f954c135c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58728/0 (socket says 172.21.15.50:58728) 2023-10-30T12:19:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954ad9d640 1 -- 172.21.15.50:0/2919819421 learned_addr learned my addr 172.21.15.50:0/2919819421 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:19:47.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533723433 0 0) 0x7f953c001110 con 0x7f954c0a32e0 2023-10-30T12:19:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient(hunting): _negotiate 2023-10-30T12:19:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient(hunting): my global_id is 244203 2023-10-30T12:19:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9534003520 con 0x7f954c0a32e0 2023-10-30T12:19:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954a59c640 1 --2- 172.21.15.50:0/2919819421 >> v2:172.21.15.118:3300/0 conn(0x7f954c0a2ae0 0x7f954c0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954a59c640 10 monclient(hunting): get_auth_request con 0x7f954c0a2ae0 auth_method 0 2023-10-30T12:19:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3703858737 0 0) 0x7f9534003520 con 0x7f954c0a32e0 2023-10-30T12:19:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 msgr2=0x7f954c1356e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 0x7f954c1356e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 >> v2:172.21.15.118:3300/0 conn(0x7f954c0a2ae0 msgr2=0x7f954c0a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 --2- 172.21.15.50:0/2919819421 >> v2:172.21.15.118:3300/0 conn(0x7f954c0a2ae0 0x7f954c0a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient: _finish_hunting 0 2023-10-30T12:19:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:19:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:19:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c09a2e0 con 0x7f954c0a32e0 2023-10-30T12:19:47.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient: _finish_auth 0 2023-10-30T12:19:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient: _check_auth_tickets 2023-10-30T12:19:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f953c001800 con 0x7f954c0a32e0 2023-10-30T12:19:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.083+0000 7f954959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:19:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient: dump: 2023-10-30T12:19:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.106 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.107 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:19:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 >> v1:172.21.15.50:6789/0 conn(0x7f954c0a32e0 legacy=0x7f954c135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9534003970 0x7f9534003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): picked mon.b con 0x7f9534003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 0x7f95340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): picked mon.c con 0x7f954c0a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9534004d00 0x7f9534005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): picked mon.a con 0x7f9534004d00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0a1630 con 0x7f9534004d00 2023-10-30T12:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953400bb90 con 0x7f9534003970 2023-10-30T12:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953400bd50 con 0x7f954c0a2ec0 2023-10-30T12:19:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954959a640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:19:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f9549d9b640 1 -- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 msgr2=0x7f95340047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f9549d9b640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 0x7f95340047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9534004d00 0x7f9534005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): get_auth_request con 0x7f9534004d00 auth_method 0 2023-10-30T12:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954a59c640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9534003970 0x7f9534003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954a59c640 10 monclient(hunting): get_auth_request con 0x7f9534003970 auth_method 0 2023-10-30T12:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:19:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient(hunting): handle_auth_done global_id 244203 payload 293 2023-10-30T12:19:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 1 -- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 msgr2=0x7f95340047c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:19:47.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f954c0a2ec0 0x7f95340047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 1 -- 172.21.15.50:0/2919819421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9534003970 msgr2=0x7f9534003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9534003970 0x7f9534003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:19:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 1 monclient: found mon.a 2023-10-30T12:19:47.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:47.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 1 -- 172.21.15.50:0/2919819421 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9534003520 con 0x7f9534004d00 2023-10-30T12:19:47.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient: _finish_auth 0 2023-10-30T12:19:47.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 10 monclient: _check_auth_tickets 2023-10-30T12:19:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954ad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f9550f01640 5 monclient: authenticate success, global_id 244203 2023-10-30T12:19:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f9550f01640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.084+0000 7f954a59c640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9534003970 0x7f9534003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:19:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954ad9d640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9534004d00 0x7f9534005160 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f954c072170 tx=0x7f95340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177ef7bc43796bbe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c005ad0 con 0x7f9534004d00 2023-10-30T12:19:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 10 monclient: dump: 2023-10-30T12:19:47.120 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.121 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.122 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c005c30 con 0x7f9534004d00 2023-10-30T12:19:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 1 -- 172.21.15.50:0/2919819421 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c007070 con 0x7f9534004d00 2023-10-30T12:19:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 4 set_mon_vals no callback set 2023-10-30T12:19:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954959a640 10 monclient: dump: 2023-10-30T12:19:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.124 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.126 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 10 monclient: shutdown 2023-10-30T12:19:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 1 -- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9534004d00 msgr2=0x7f9534005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 1 --2- 172.21.15.50:0/2919819421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9534004d00 0x7f9534005160 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f954c072170 tx=0x7f95340057f0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f954ad9d640 1 -- 172.21.15.50:0/2919819421 reap_dead start 2023-10-30T12:19:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 1 -- 172.21.15.50:0/2919819421 shutdown_connections 2023-10-30T12:19:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 1 -- 172.21.15.50:0/2919819421 >> 172.21.15.50:0/2919819421 conn(0x7f954c0a80b0 msgr2=0x7f954c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 1 -- 172.21.15.50:0/2919819421 shutdown_connections 2023-10-30T12:19:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.085+0000 7f9550f01640 1 -- 172.21.15.50:0/2919819421 wait complete. 2023-10-30T12:19:47.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.086+0000 7f9550f01640 10 monclient: build_initial_monmap 2023-10-30T12:19:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.086+0000 7f9550f01640 10 monclient: monmap: 2023-10-30T12:19:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:47.128 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:47.130 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.086+0000 7f9550f01640 1 librados: starting msgr at 2023-10-30T12:19:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.086+0000 7f9550f01640 1 librados: starting objecter 2023-10-30T12:19:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.086+0000 7f9550f01640 1 Processor -- start 2023-10-30T12:19:47.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 -- start start 2023-10-30T12:19:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 librados: setting wanted keys 2023-10-30T12:19:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 librados: calling monclient init 2023-10-30T12:19:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient: init 2023-10-30T12:19:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:19:47.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9534004d00 0x7f954c0a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): picked mon.noname-c con 0x7f9534004d00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1ca5b0 0x7f954c132850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): picked mon.noname-a con 0x7f954c1ca5b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f954c132d90 0x7f954c1c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): picked mon.noname-b con 0x7f954c132d90 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c1caa30 con 0x7f954c1ca5b0 2023-10-30T12:19:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c120f40 con 0x7f954c132d90 2023-10-30T12:19:47.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0a0cc0 con 0x7f9534004d00 2023-10-30T12:19:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f954a59c640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9534004d00 msgr2=0x7f954c0a0030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9550f01640 10 monclient(hunting): authenticate will time out at 2032.270466s 2023-10-30T12:19:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f954a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9534004d00 0x7f954c0a0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:47.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1ca5b0 0x7f954c132850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1ca5b0 0x7f954c132850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33212/0 (socket says 172.21.15.50:33212) 2023-10-30T12:19:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 -- 172.21.15.50:0/47116946 learned_addr learned my addr 172.21.15.50:0/47116946 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): get_auth_request con 0x7f954c1ca5b0 auth_method 0 2023-10-30T12:19:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:47.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient(hunting): handle_auth_done global_id 244206 payload 824 2023-10-30T12:19:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9534004d00 msgr2=0x7f954c0a0030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9534004d00 0x7f954c0a0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f954c132d90 msgr2=0x7f954c1c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f954c132d90 0x7f954c1c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:19:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 monclient: found mon.noname-a 2023-10-30T12:19:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 -- 172.21.15.50:0/47116946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c120980 con 0x7f954c1ca5b0 2023-10-30T12:19:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f9549d9b640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1ca5b0 0x7f954c132850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f954000ba10 tx=0x7f954000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f953affd640 10 client.?.objecter ms_handle_connect 0x7f954c1ca5b0 2023-10-30T12:19:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.087+0000 7f953affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954000cb50 con 0x7f954c1ca5b0 2023-10-30T12:19:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:47.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: dump: 2023-10-30T12:19:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.140 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.141 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: _finish_auth 0 2023-10-30T12:19:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: _check_auth_tickets 2023-10-30T12:19:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 5 monclient: authenticate success, global_id 244206 2023-10-30T12:19:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954000ccd0 con 0x7f954c1ca5b0 2023-10-30T12:19:47.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 10 monclient: _renew_subs 2023-10-30T12:19:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954c120c60 con 0x7f954c1ca5b0 2023-10-30T12:19:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95400156b0 con 0x7f954c1ca5b0 2023-10-30T12:19:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 10 monclient: dump: 2023-10-30T12:19:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.144 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.145 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.145 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9548d99640 4 set_mon_vals no callback set 2023-10-30T12:19:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 10 client.244206.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 10 monclient: _renew_subs 2023-10-30T12:19:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c1c7130 con 0x7f954c1ca5b0 2023-10-30T12:19:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 1 librados: init done 2023-10-30T12:19:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c73b0 0x7f954c09cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c073ff0 con 0x7f954c1c73b0 2023-10-30T12:19:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.088+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9540012070 con 0x7f954c1ca5b0 2023-10-30T12:19:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c73b0 0x7f954c09cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient: get_auth_request con 0x7f954c1c73b0 auth_method 0 2023-10-30T12:19:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f952c0392a0 0x7f952c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(597..597 src has 490..597) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f954004acb0 con 0x7f954c1ca5b0 2023-10-30T12:19:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 client.244206.objecter ms_dispatch 0x7f954c000b90 osd_map(597..597 src has 490..597) v4 2023-10-30T12:19:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 3 client.244206.objecter handle_osd_map got epochs [597,597] > 0 2023-10-30T12:19:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 3 client.244206.objecter handle_osd_map decoding full epoch 597 2023-10-30T12:19:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 10 monclient(hunting): handle_auth_done global_id 244209 payload 824 2023-10-30T12:19:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 1 -- 172.21.15.50:0/47116946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f954c11ff90 con 0x7f954c1c73b0 2023-10-30T12:19:47.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 20 client.244206.objecter dump_active .. 0 homeless 2023-10-30T12:19:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 client.244206.objecter ms_handle_connect 0x7f954c1c73b0 2023-10-30T12:19:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954a59c640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c73b0 0x7f954c09cd90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9544003fd0 tx=0x7f954400f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 client.244206.objecter resend_mon_ops 2023-10-30T12:19:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954ad9d640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f952c0392a0 0x7f952c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f954ad9d640 10 monclient: get_auth_request con 0x7f952c0392a0 auth_method 0 2023-10-30T12:19:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9544014870 con 0x7f954c1c73b0 2023-10-30T12:19:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 monclient: dump: 2023-10-30T12:19:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.152 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.153 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.154 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.154 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.154 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f95440149d0 con 0x7f954c1c73b0 2023-10-30T12:19:47.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:47.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:47.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c73b0 msgr2=0x7f954c09cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c73b0 0x7f954c09cd90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9544003fd0 tx=0x7f954400f8a0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c73b0 msgr2=0x7f954c09cd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:47.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.089+0000 7f953affd640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c73b0 0x7f954c09cd90 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9544003fd0 tx=0x7f954400f8a0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.090+0000 7f953affd640 10 client.244206.objecter ms_handle_connect 0x7f952c0392a0 2023-10-30T12:19:47.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.090+0000 7f954ad9d640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f952c0392a0 0x7f952c03b760 secure :-1 s=READY pgs=4298 cs=0 l=1 rev1=1 crypto rx=0x7f953c034000 tx=0x7f953c039040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f9550f01640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:47.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f9550f01640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c7790 0x7f954c09d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f9550f01640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c09a5c0 con 0x7f954c1c7790 2023-10-30T12:19:47.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c7790 0x7f954c09d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient: get_auth_request con 0x7f954c1c7790 auth_method 0 2023-10-30T12:19:47.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:47.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 10 monclient(hunting): handle_auth_done global_id 244212 payload 824 2023-10-30T12:19:47.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 1 -- 172.21.15.50:0/47116946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f954c099b40 con 0x7f954c1c7790 2023-10-30T12:19:47.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f954a59c640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c7790 0x7f954c09d2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9544003fd0 tx=0x7f95440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f953affd640 10 client.244206.objecter ms_handle_connect 0x7f954c1c7790 2023-10-30T12:19:47.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f953affd640 10 client.244206.objecter resend_mon_ops 2023-10-30T12:19:47.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9544024070 con 0x7f954c1c7790 2023-10-30T12:19:47.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f953affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f953affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:47.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.096+0000 7f953affd640 10 monclient: dump: 2023-10-30T12:19:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.163 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.163 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.163 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.163 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4793 (secure 0 0 0) 0x7f954401d420 con 0x7f954c1c7790 2023-10-30T12:19:47.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f953affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:47.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f953affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:47.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c7790 msgr2=0x7f954c09d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f953affd640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c7790 0x7f954c09d2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9544003fd0 tx=0x7f95440046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f953affd640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c7790 msgr2=0x7f954c09d2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:47.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f953affd640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1c7790 0x7f954c09d2d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.165 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:47.165 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:47.165 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:47.165 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:19:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 202, 2023-10-30T12:19:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:19:47.166 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:19:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:47.167 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:47.168 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:47.169 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:47.170 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:47.171 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:47.172 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:47.173 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:47.174 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:47.175 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:47.176 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:47.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:47.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:47.182 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.184 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:47.185 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:47.186 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:47.187 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:47.188 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:47.188 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.188 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.189 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.189 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:47.189 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:47.189 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:47.189 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:47.190 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:47.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.190 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.190 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:47.191 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:47.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.191 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:47.191 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:47.191 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:47.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.192 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.192 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:19:47.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.192 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:47.192 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:47.193 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:19:47.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.193 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.193 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:47.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.193 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:47.194 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:47.194 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:19:47.194 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.194 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.194 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:19:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.196 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:47.196 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:47.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 10 librados: watch_flush enter 2023-10-30T12:19:47.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 10 librados: watch_flush exit 2023-10-30T12:19:47.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 20 client.244206.objecter shutdown clearing up homeless session... 2023-10-30T12:19:47.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 10 client.244206.objecter successfully canceled tick 2023-10-30T12:19:47.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f952c0392a0 msgr2=0x7f952c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f952c0392a0 0x7f952c03b760 secure :-1 s=READY pgs=4298 cs=0 l=1 rev1=1 crypto rx=0x7f953c034000 tx=0x7f953c039040 comp rx=0 tx=0).stop 2023-10-30T12:19:47.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 10 monclient: shutdown 2023-10-30T12:19:47.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:47.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1ca5b0 msgr2=0x7f954c132850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 --2- 172.21.15.50:0/47116946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c1ca5b0 0x7f954c132850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f954000ba10 tx=0x7f954000bed0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f954ad9d640 1 -- 172.21.15.50:0/47116946 reap_dead start 2023-10-30T12:19:47.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f954ad9d640 1 -- 172.21.15.50:0/47116946 reap_dead start 2023-10-30T12:19:47.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 shutdown_connections 2023-10-30T12:19:47.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 >> 172.21.15.50:0/47116946 conn(0x7f954c0a80b0 msgr2=0x7f954c1bcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:47.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 shutdown_connections 2023-10-30T12:19:47.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 -- 172.21.15.50:0/47116946 wait complete. 2023-10-30T12:19:47.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.097+0000 7f9550f01640 1 librados: shutdown 2023-10-30T12:19:47.201 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:19:47.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:19:47.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:19:47.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.307+0000 7faf816a9640 10 monclient: get_monmap_and_config 2023-10-30T12:19:47.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.307+0000 7faf816a9640 10 monclient: build_initial_monmap 2023-10-30T12:19:47.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.307+0000 7faf816a9640 10 monclient: monmap: 2023-10-30T12:19:47.309 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:47.310 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.310 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:19:47.308662+0000 2023-10-30T12:19:47.310 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:19:47.308662+0000 2023-10-30T12:19:47.310 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:19:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:19:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:19:47.311 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:19:47.312 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:19:47.312 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 1 Processor -- start 2023-10-30T12:19:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 1 -- start start 2023-10-30T12:19:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient: init 2023-10-30T12:19:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-10-30T12:19:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf7c09e7e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:19:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf7c09f2d0 0x7faf7c06c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient(hunting): picked mon.noname-a con 0x7faf7c09f2d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:19:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7faf7c138b60 0x7faf7c13af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient(hunting): picked mon.noname-c con 0x7faf7c138b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:19:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c09fbf0 con 0x7faf7c09e7e0 2023-10-30T12:19:47.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c09d5f0 con 0x7faf7c09f2d0 2023-10-30T12:19:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c09fed0 con 0x7faf7c138b60 2023-10-30T12:19:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.308+0000 7faf816a9640 10 monclient(hunting): authenticate will time out at 2032.491944s 2023-10-30T12:19:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf7c09f2d0 0x7faf7c06c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf7c09f2d0 0x7faf7c06c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33224/0 (socket says 172.21.15.50:33224) 2023-10-30T12:19:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 -- 172.21.15.50:0/2558379712 learned_addr learned my addr 172.21.15.50:0/2558379712 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): get_auth_request con 0x7faf7c09f2d0 auth_method 0 2023-10-30T12:19:47.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf7b7fe640 1 --2- 172.21.15.50:0/2558379712 >> v2:172.21.15.118:3300/0 conn(0x7faf7c138b60 0x7faf7c13af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf7b7fe640 10 monclient(hunting): get_auth_request con 0x7faf7c138b60 auth_method 0 2023-10-30T12:19:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf7b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf7b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient(hunting): handle_auth_done global_id 244215 payload 274 2023-10-30T12:19:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 -- 172.21.15.50:0/2558379712 >> v2:172.21.15.118:3300/0 conn(0x7faf7c138b60 msgr2=0x7faf7c13af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 --2- 172.21.15.50:0/2558379712 >> v2:172.21.15.118:3300/0 conn(0x7faf7c138b60 0x7faf7c13af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 -- 172.21.15.50:0/2558379712 >> v1:172.21.15.118:6789/0 conn(0x7faf7c09e7e0 legacy=0x7faf7c09ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:19:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient: _finish_hunting 0 2023-10-30T12:19:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 monclient: found mon.noname-a 2023-10-30T12:19:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.309+0000 7faf73fff640 1 -- 172.21.15.50:0/2558379712 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0a01b0 con 0x7faf7c09f2d0 2023-10-30T12:19:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf73fff640 1 --2- 172.21.15.50:0/2558379712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09f2d0 0x7faf7c06c3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faf7c071280 tx=0x7faf6402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7dc681e0f00d02b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 1 -- 172.21.15.50:0/2558379712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64031750 con 0x7faf7c09f2d0 2023-10-30T12:19:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: dump: 2023-10-30T12:19:47.334 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.335 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.336 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:19:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:19:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 5 monclient: authenticate success, global_id 244215 2023-10-30T12:19:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:19:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 1 -- 172.21.15.50:0/2558379712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf640318b0 con 0x7faf7c09f2d0 2023-10-30T12:19:47.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:19:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 4 set_mon_vals no callback set 2023-10-30T12:19:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 1 -- 172.21.15.50:0/2558379712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64031c40 con 0x7faf7c09f2d0 2023-10-30T12:19:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf7a7fc640 10 monclient: dump: 2023-10-30T12:19:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.339 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.340 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.341 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 10 monclient: shutdown 2023-10-30T12:19:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 -- 172.21.15.50:0/2558379712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09f2d0 msgr2=0x7faf7c06c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 --2- 172.21.15.50:0/2558379712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09f2d0 0x7faf7c06c3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faf7c071280 tx=0x7faf6402b7f0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 -- 172.21.15.50:0/2558379712 shutdown_connections 2023-10-30T12:19:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 --2- 172.21.15.50:0/2558379712 >> v2:172.21.15.118:3300/0 conn(0x7faf7c138b60 0x7faf7c13af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 --2- 172.21.15.50:0/2558379712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09f2d0 0x7faf7c06c3c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 -- 172.21.15.50:0/2558379712 >> 172.21.15.50:0/2558379712 conn(0x7faf7c0a4800 msgr2=0x7faf7c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 -- 172.21.15.50:0/2558379712 shutdown_connections 2023-10-30T12:19:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.310+0000 7faf816a9640 1 -- 172.21.15.50:0/2558379712 wait complete. 2023-10-30T12:19:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 10 monclient: build_initial_monmap 2023-10-30T12:19:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 10 monclient: monmap: 2023-10-30T12:19:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:19:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:19:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:19:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:19:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:19:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:19:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:19:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:19:47.345 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 1 librados: starting msgr at 2023-10-30T12:19:47.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 1 librados: starting objecter 2023-10-30T12:19:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 1 Processor -- start 2023-10-30T12:19:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 1 -- start start 2023-10-30T12:19:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 1 librados: setting wanted keys 2023-10-30T12:19:47.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 1 librados: calling monclient init 2023-10-30T12:19:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.311+0000 7faf816a9640 10 monclient: init 2023-10-30T12:19:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:19:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:19:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09e7e0 0x7faf7c1c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): picked mon.noname-a con 0x7faf7c09e7e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:19:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf7c09f2d0 0x7faf7c132710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): picked mon.noname-c con 0x7faf7c09f2d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:19:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c138b60 0x7faf7c132c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): picked mon.noname-b con 0x7faf7c138b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:19:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c070e70 con 0x7faf7c09e7e0 2023-10-30T12:19:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c1c8f40 con 0x7faf7c138b60 2023-10-30T12:19:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0a32f0 con 0x7faf7c09f2d0 2023-10-30T12:19:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:19:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf816a9640 10 monclient(hunting): authenticate will time out at 2032.495342s 2023-10-30T12:19:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf73fff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf7c09f2d0 msgr2=0x7faf7c132710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:19:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf73fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf7c09f2d0 0x7faf7c132710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:19:47.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09e7e0 0x7faf7c1c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09e7e0 0x7faf7c1c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33234/0 (socket says 172.21.15.50:33234) 2023-10-30T12:19:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 -- 172.21.15.50:0/3255010651 learned_addr learned my addr 172.21.15.50:0/3255010651 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:19:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): get_auth_request con 0x7faf7c09e7e0 auth_method 0 2023-10-30T12:19:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient(hunting): handle_auth_done global_id 244218 payload 824 2023-10-30T12:19:47.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7b7fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c138b60 0x7faf7c132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf7c09f2d0 msgr2=0x7faf7c132710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:19:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7faf7c09f2d0 0x7faf7c132710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c138b60 msgr2=0x7faf7c132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c138b60 0x7faf7c132c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient: _finish_hunting 0 2023-10-30T12:19:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 monclient: found mon.noname-a 2023-10-30T12:19:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:19:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 -- 172.21.15.50:0/3255010651 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf640094d0 con 0x7faf7c09e7e0 2023-10-30T12:19:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf7affd640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09e7e0 0x7faf7c1c8250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b420 tx=0x7faf6c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf737fe640 10 client.?.objecter ms_handle_connect 0x7faf7c09e7e0 2023-10-30T12:19:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.312+0000 7faf737fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:19:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c011830 con 0x7faf7c09e7e0 2023-10-30T12:19:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:19:47.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: dump: 2023-10-30T12:19:47.357 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.357 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.357 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.357 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.358 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.358 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.358 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.358 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.358 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.359 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: _finish_auth 0 2023-10-30T12:19:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: _check_auth_tickets 2023-10-30T12:19:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:19:47.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 5 monclient: authenticate success, global_id 244218 2023-10-30T12:19:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 10 monclient: _renew_subs 2023-10-30T12:19:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c011990 con 0x7faf7c09e7e0 2023-10-30T12:19:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c11e820 con 0x7faf7c09e7e0 2023-10-30T12:19:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:19:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c011d40 con 0x7faf7c09e7e0 2023-10-30T12:19:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:19:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 10 monclient: dump: 2023-10-30T12:19:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.361 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.362 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.363 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf79ffb640 4 set_mon_vals no callback set 2023-10-30T12:19:47.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 10 client.244218.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:19:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 10 monclient: _renew_subs 2023-10-30T12:19:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:19:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c11cd80 con 0x7faf7c09e7e0 2023-10-30T12:19:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 1 librados: init done 2023-10-30T12:19:47.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c11db50 0x7faf7c11b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0a1280 con 0x7faf7c11db50 2023-10-30T12:19:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.313+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7faf6c01b460 con 0x7faf7c09e7e0 2023-10-30T12:19:47.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c11db50 0x7faf7c11b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient: get_auth_request con 0x7faf7c11db50 auth_method 0 2023-10-30T12:19:47.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf5c0392a0 0x7faf5c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(597..597 src has 490..597) v4 ==== 6698+0+0 (secure 0 0 0) 0x7faf6c04a950 con 0x7faf7c09e7e0 2023-10-30T12:19:47.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 10 client.244218.objecter ms_dispatch 0x7faf7c000b90 osd_map(597..597 src has 490..597) v4 2023-10-30T12:19:47.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 3 client.244218.objecter handle_osd_map got epochs [597,597] > 0 2023-10-30T12:19:47.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 3 client.244218.objecter handle_osd_map decoding full epoch 597 2023-10-30T12:19:47.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:47.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 20 client.244218.objecter dump_active .. 0 homeless 2023-10-30T12:19:47.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf7b7fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf5c0392a0 0x7faf5c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf7b7fe640 10 monclient: get_auth_request con 0x7faf5c0392a0 auth_method 0 2023-10-30T12:19:47.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 10 monclient(hunting): handle_auth_done global_id 204622 payload 824 2023-10-30T12:19:47.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 1 -- 172.21.15.50:0/3255010651 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf7c09e060 con 0x7faf7c11db50 2023-10-30T12:19:47.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 10 client.244218.objecter ms_handle_connect 0x7faf7c11db50 2023-10-30T12:19:47.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 10 client.244218.objecter resend_mon_ops 2023-10-30T12:19:47.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf73fff640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c11db50 0x7faf7c11b860 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0a17a0 tx=0x7faf640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 10 client.244218.objecter ms_handle_connect 0x7faf5c0392a0 2023-10-30T12:19:47.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf7b7fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf5c0392a0 0x7faf5c03b760 secure :-1 s=READY pgs=4299 cs=0 l=1 rev1=1 crypto rx=0x7faf7c06c950 tx=0x7faf7400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64033a10 con 0x7faf7c11db50 2023-10-30T12:19:47.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:47.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.314+0000 7faf737fe640 10 monclient: dump: 2023-10-30T12:19:47.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.371 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.371 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.371 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.372 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.372 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.372 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.372 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.373 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.373 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.315+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7faf64033b70 con 0x7faf7c11db50 2023-10-30T12:19:47.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.315+0000 7faf737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:19:47.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.315+0000 7faf737fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:19:47.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.315+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c11db50 msgr2=0x7faf7c11b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.315+0000 7faf737fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c11db50 0x7faf7c11b860 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0a17a0 tx=0x7faf640046c0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.315+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c11db50 msgr2=0x7faf7c11b860 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:47.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.315+0000 7faf737fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c11db50 0x7faf7c11b860 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf816a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:19:47.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf816a9640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c02b430 0x7faf7c02b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:19:47.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf816a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:19:47.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c074800 con 0x7faf7c02b430 2023-10-30T12:19:47.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c02b430 0x7faf7c02b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:19:47.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient: get_auth_request con 0x7faf7c02b430 auth_method 0 2023-10-30T12:19:47.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:19:47.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:19:47.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:19:47.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:19:47.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:19:47.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:19:47.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 10 monclient(hunting): handle_auth_done global_id 204625 payload 824 2023-10-30T12:19:47.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.321+0000 7faf73fff640 1 -- 172.21.15.50:0/3255010651 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faf7c11f1e0 con 0x7faf7c02b430 2023-10-30T12:19:47.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf73fff640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c02b430 0x7faf7c02b840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faf64000c00 tx=0x7faf64009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:19:47.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 10 client.244218.objecter ms_handle_connect 0x7faf7c02b430 2023-10-30T12:19:47.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 10 client.244218.objecter resend_mon_ops 2023-10-30T12:19:47.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6403dbf0 con 0x7faf7c02b430 2023-10-30T12:19:47.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:19:47.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:19:47.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 10 monclient: dump: 2023-10-30T12:19:47.379 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:19:47.380 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:19:47.380 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.380 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:19:47.380 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:19:47.380 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:19:47.381 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:19:47.381 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:19:47.381 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:19:47.381 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:19:47.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4621 (secure 0 0 0) 0x7faf64047410 con 0x7faf7c02b430 2023-10-30T12:19:47.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:19:47.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:19:47.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c02b430 msgr2=0x7faf7c02b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c02b430 0x7faf7c02b840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faf64000c00 tx=0x7faf64009240 comp rx=0 tx=0).stop 2023-10-30T12:19:47.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c02b430 msgr2=0x7faf7c02b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:19:47.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf737fe640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faf7c02b430 0x7faf7c02b840 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faf64000c00 tx=0x7faf64009240 comp rx=0 tx=0).stop 2023-10-30T12:19:47.385 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:19:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:47.385 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:19:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 202, 2023-10-30T12:19:47.386 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:19:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:19:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:19:47.387 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:19:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:19:47.388 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:19:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:47.389 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:47.390 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:19:47.391 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:19:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:47.392 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:47.393 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:19:47.394 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:19:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:19:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:19:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:19:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:19:47.395 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:19:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:19:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:19:47.396 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:19:47.397 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:19:47.397 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:19:47.397 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:19:47.397 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:19:47.397 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:19:47.398 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:19:47.398 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:19:47.398 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:19:47.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:19:47.399 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:19:47.399 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:19:47.399 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:19:47.400 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:19:47.400 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:19:47.401 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:19:47.401 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:19:47.401 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.402 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:19:47.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:19:47.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.403 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:19:47.404 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:19:47.404 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:47.404 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:47.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.405 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:47.405 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:19:47.405 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.407 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:47.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:19:47.407 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:19:47.408 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:19:47.408 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:47.408 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:47.409 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:47.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.409 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:19:47.409 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:19:47.409 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:47.410 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:47.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.410 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:47.410 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:19:47.411 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:47.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:19:47.412 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.412 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:19:47.412 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:19:47.413 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:47.413 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:47.413 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:47.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.414 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:19:47.414 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:19:47.414 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:19:47.414 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:19:47.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.415 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:19:47.415 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:19:47.415 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:19:47.416 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:19:47.416 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:19:47.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.417 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:19:47.417 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:19:47.417 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:19:47.418 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:19:47.418 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:19:47.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.418 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.419 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:19:47.419 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:19:47.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.420 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:47.420 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:47.421 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:19:47.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.421 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.421 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:19:47.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.422 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:47.422 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:47.422 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:19:47.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.423 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:19:47.423 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:19:47.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:19:47.423 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:19:47.424 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:19:47.424 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:19:47.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:19:47.424 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:19:47.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:19:47.425 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:19:47.425 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:19:47.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 10 librados: watch_flush enter 2023-10-30T12:19:47.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 10 librados: watch_flush exit 2023-10-30T12:19:47.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 20 client.244218.objecter shutdown clearing up homeless session... 2023-10-30T12:19:47.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 10 client.244218.objecter successfully canceled tick 2023-10-30T12:19:47.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf5c0392a0 msgr2=0x7faf5c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7faf5c0392a0 0x7faf5c03b760 secure :-1 s=READY pgs=4299 cs=0 l=1 rev1=1 crypto rx=0x7faf7c06c950 tx=0x7faf7400b040 comp rx=0 tx=0).stop 2023-10-30T12:19:47.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 10 monclient: shutdown 2023-10-30T12:19:47.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:19:47.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09e7e0 msgr2=0x7faf7c1c8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:19:47.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 1 --2- 172.21.15.50:0/3255010651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf7c09e7e0 0x7faf7c1c8250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b420 tx=0x7faf6c00b8e0 comp rx=0 tx=0).stop 2023-10-30T12:19:47.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf7b7fe640 1 -- 172.21.15.50:0/3255010651 reap_dead start 2023-10-30T12:19:47.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf7b7fe640 1 -- 172.21.15.50:0/3255010651 reap_dead start 2023-10-30T12:19:47.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 shutdown_connections 2023-10-30T12:19:47.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.322+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 >> 172.21.15.50:0/3255010651 conn(0x7faf7c0a4800 msgr2=0x7faf7c1337d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:19:47.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.323+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 shutdown_connections 2023-10-30T12:19:47.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.323+0000 7faf816a9640 1 -- 172.21.15.50:0/3255010651 wait complete. 2023-10-30T12:19:47.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:19:47.323+0000 7faf816a9640 1 librados: shutdown 2023-10-30T12:19:47.431 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:19:47.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:20:07.432 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:20:07.433 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-10-30T12:20:07.433 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:20:07.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:20:07.438 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:20:07.438 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:20:07.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.555+0000 7fece3ec2640 10 monclient: get_monmap_and_config 2023-10-30T12:20:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.555+0000 7fece3ec2640 10 monclient: build_initial_monmap 2023-10-30T12:20:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.555+0000 7fece3ec2640 10 monclient: monmap: 2023-10-30T12:20:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:07.556209+0000 2023-10-30T12:20:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:07.556209+0000 2023-10-30T12:20:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:07.559 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:07.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.555+0000 7fece3ec2640 1 Processor -- start 2023-10-30T12:20:07.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.555+0000 7fece3ec2640 1 -- start start 2023-10-30T12:20:07.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.555+0000 7fece3ec2640 10 monclient: init 2023-10-30T12:20:07.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-10-30T12:20:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient(hunting): picked mon.noname-b con 0x7fecdc0a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecdc099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecdc132be0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc0a4f40 con 0x7fecdc099a40 2023-10-30T12:20:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc0a4150 con 0x7fecdc132be0 2023-10-30T12:20:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc0a3d70 con 0x7fecdc0a4b60 2023-10-30T12:20:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece3ec2640 10 monclient(hunting): authenticate will time out at 2052.739519s 2023-10-30T12:20:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece145d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fecdc099a40 0x7fecdc0a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56024/0 (socket says 172.21.15.50:56024) 2023-10-30T12:20:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece145d640 1 -- 172.21.15.50:0/4209647019 learned_addr learned my addr 172.21.15.50:0/4209647019 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:20:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 10 monclient(hunting): get_auth_request con 0x7fecdc0a4b60 auth_method 0 2023-10-30T12:20:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 1 -- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 msgr2=0x7fecdc1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 1 -- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 msgr2=0x7fecdc1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece1c5e640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927298279 0 0) 0x7fecdc0a4f40 con 0x7fecdc099a40 2023-10-30T12:20:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece0c5c640 10 monclient(hunting): _negotiate 2023-10-30T12:20:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece0c5c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece0c5c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece0c5c640 10 monclient(hunting): my global_id is 244221 2023-10-30T12:20:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.556+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc8003520 con 0x7fecdc099a40 2023-10-30T12:20:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524190051 0 0) 0x7fecdc0a4150 con 0x7fecdc132be0 2023-10-30T12:20:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): _negotiate 2023-10-30T12:20:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): my global_id is 204628 2023-10-30T12:20:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecdc0a4f40 con 0x7fecdc132be0 2023-10-30T12:20:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 466019292 0 0) 0x7fecc8003520 con 0x7fecdc099a40 2023-10-30T12:20:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 msgr2=0x7fecdc1326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 >> v1:172.21.15.118:6789/0 conn(0x7fecdc132be0 legacy=0x7fecdc0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: _finish_hunting 0 2023-10-30T12:20:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:20:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:20:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc0a3170 con 0x7fecdc099a40 2023-10-30T12:20:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: _finish_auth 0 2023-10-30T12:20:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: _check_auth_tickets 2023-10-30T12:20:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fecd8001690 con 0x7fecdc099a40 2023-10-30T12:20:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece3ec2640 5 monclient: authenticate success, global_id 244221 2023-10-30T12:20:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece3ec2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:20:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: dump: 2023-10-30T12:20:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:07.581 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:20:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 >> v1:172.21.15.50:6789/0 conn(0x7fecdc099a40 legacy=0x7fecdc0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:20:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecc8003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): picked mon.c con 0x7fecdc0a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc8004090 0x7fecc80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): picked mon.b con 0x7fecc8004090 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecc8004d50 0x7fecc80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): picked mon.a con 0x7fecc8004d50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc800bc00 con 0x7fecc8004d50 2023-10-30T12:20:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc800bd70 con 0x7fecc8004090 2023-10-30T12:20:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc800bf30 con 0x7fecdc0a4b60 2023-10-30T12:20:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece0c5c640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:20:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece3ec2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecc8003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): get_auth_request con 0x7fecdc0a4b60 auth_method 0 2023-10-30T12:20:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecc8004d50 0x7fecc80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 10 monclient(hunting): get_auth_request con 0x7fecc8004d50 auth_method 0 2023-10-30T12:20:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc8004090 0x7fecc80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:20:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 10 monclient(hunting): get_auth_request con 0x7fecc8004090 auth_method 0 2023-10-30T12:20:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:20:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece145d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:20:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient(hunting): handle_auth_done global_id 244221 payload 293 2023-10-30T12:20:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 1 -- 172.21.15.50:0/4209647019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc8004090 msgr2=0x7fecc80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc8004090 0x7fecc80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 1 -- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecc8004d50 msgr2=0x7fecc80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecc8004d50 0x7fecc80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient: _finish_hunting 0 2023-10-30T12:20:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 1 monclient: found mon.c 2023-10-30T12:20:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 1 -- 172.21.15.50:0/4209647019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8003520 con 0x7fecdc0a4b60 2023-10-30T12:20:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient: _finish_auth 0 2023-10-30T12:20:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 10 monclient: _check_auth_tickets 2023-10-30T12:20:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece1c5e640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.557+0000 7fece245f640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecc8004d50 0x7fecc80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:20:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece1c5e640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecc8003b30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feccc00b9e0 tx=0x7fecc80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f735a1e7e75e54c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc032630 con 0x7fecdc0a4b60 2023-10-30T12:20:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 10 monclient: dump: 2023-10-30T12:20:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:07.594 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece3ec2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc032790 con 0x7fecdc0a4b60 2023-10-30T12:20:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece3ec2640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece3ec2640 4 set_mon_vals no callback set 2023-10-30T12:20:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 1 -- 172.21.15.50:0/4209647019 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc032b40 con 0x7fecdc0a4b60 2023-10-30T12:20:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.558+0000 7fece0c5c640 10 monclient: dump: 2023-10-30T12:20:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:07.598 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 10 monclient: shutdown 2023-10-30T12:20:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 1 -- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 msgr2=0x7fecc8003b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 1 --2- 172.21.15.50:0/4209647019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecc8003b30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feccc00b9e0 tx=0x7fecc80057f0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece245f640 1 -- 172.21.15.50:0/4209647019 reap_dead start 2023-10-30T12:20:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 1 -- 172.21.15.50:0/4209647019 shutdown_connections 2023-10-30T12:20:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 1 -- 172.21.15.50:0/4209647019 >> 172.21.15.50:0/4209647019 conn(0x7fecdc0a8060 msgr2=0x7fecdc0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 1 -- 172.21.15.50:0/4209647019 shutdown_connections 2023-10-30T12:20:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 1 -- 172.21.15.50:0/4209647019 wait complete. 2023-10-30T12:20:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 10 monclient: build_initial_monmap 2023-10-30T12:20:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.559+0000 7fece3ec2640 10 monclient: monmap: 2023-10-30T12:20:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:07.602 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 librados: starting msgr at 2023-10-30T12:20:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 librados: starting objecter 2023-10-30T12:20:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 Processor -- start 2023-10-30T12:20:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 -- start start 2023-10-30T12:20:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 librados: setting wanted keys 2023-10-30T12:20:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 librados: calling monclient init 2023-10-30T12:20:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient: init 2023-10-30T12:20:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:20:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): picked mon.noname-c con 0x7fecdc0a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecdc1caac0 0x7fecdc1217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): picked mon.noname-a con 0x7fecdc1caac0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc1caee0 0x7fecdc121d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): picked mon.noname-b con 0x7fecdc1caee0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc1255e0 con 0x7fecdc1caac0 2023-10-30T12:20:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc125830 con 0x7fecdc1caee0 2023-10-30T12:20:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc0a0760 con 0x7fecdc0a4b60 2023-10-30T12:20:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece3ec2640 10 monclient(hunting): authenticate will time out at 2052.743701s 2023-10-30T12:20:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecdc1caac0 0x7fecdc1217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47598/0 (socket says 172.21.15.50:47598) 2023-10-30T12:20:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecdc1caac0 0x7fecdc1217c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44874/0 (socket says 172.21.15.50:44874) 2023-10-30T12:20:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 1 -- 172.21.15.50:0/110793624 learned_addr learned my addr 172.21.15.50:0/110793624 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 10 monclient(hunting): get_auth_request con 0x7fecdc0a4b60 auth_method 0 2023-10-30T12:20:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 10 monclient(hunting): get_auth_request con 0x7fecdc1caac0 auth_method 0 2023-10-30T12:20:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 1 -- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 msgr2=0x7fecdc1ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:20:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 1 -- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 msgr2=0x7fecdc1ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece1c5e640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece145d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.560+0000 7fece245f640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc1caee0 0x7fecdc121d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece1c5e640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1ca580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece245f640 10 monclient(hunting): get_auth_request con 0x7fecdc1caee0 auth_method 0 2023-10-30T12:20:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece245f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece245f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece245f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece245f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece245f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece245f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 10 monclient(hunting): handle_auth_done global_id 244224 payload 824 2023-10-30T12:20:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 1 -- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 msgr2=0x7fecdc1ca580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1ca580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 1 -- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc1caee0 msgr2=0x7fecdc121d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc1caee0 0x7fecdc121d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 10 monclient: _finish_hunting 0 2023-10-30T12:20:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 1 monclient: found mon.noname-a 2023-10-30T12:20:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 1 -- 172.21.15.50:0/110793624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc0a18d0 con 0x7fecdc1caac0 2023-10-30T12:20:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece145d640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecdc1caac0 0x7fecdc1217c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0a50d0 tx=0x7fecd8007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 client.?.objecter ms_handle_connect 0x7fecdc1caac0 2023-10-30T12:20:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 1 -- 172.21.15.50:0/110793624 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd800f600 con 0x7fecdc1caac0 2023-10-30T12:20:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: dump: 2023-10-30T12:20:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:07.617 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: _finish_auth 0 2023-10-30T12:20:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:20:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 1 -- 172.21.15.50:0/110793624 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd800f780 con 0x7fecdc1caac0 2023-10-30T12:20:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 1 -- 172.21.15.50:0/110793624 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd800fb30 con 0x7fecdc1caac0 2023-10-30T12:20:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 5 monclient: authenticate success, global_id 244224 2023-10-30T12:20:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd27fc640 10 monclient: dump: 2023-10-30T12:20:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:07.620 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 10 monclient: _renew_subs 2023-10-30T12:20:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fecd3fff640 4 set_mon_vals no callback set 2023-10-30T12:20:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecdc123e20 con 0x7fecdc1caac0 2023-10-30T12:20:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 10 client.244224.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 10 monclient: _renew_subs 2023-10-30T12:20:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecdc126210 con 0x7fecdc1caac0 2023-10-30T12:20:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.561+0000 7fece3ec2640 1 librados: init done 2023-10-30T12:20:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.562+0000 7fecd27fc640 1 -- 172.21.15.50:0/110793624 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fecd800fc90 con 0x7fecdc1caac0 2023-10-30T12:20:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.562+0000 7fece3ec2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.562+0000 7fece3ec2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.562+0000 7fece3ec2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.562+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecdc1250b0 con 0x7fecdc1caac0 2023-10-30T12:20:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.562+0000 7fecd27fc640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fecc00392f0 0x7fecc003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fecd27fc640 1 -- 172.21.15.50:0/110793624 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(618..618 src has 490..618) v4 ==== 6678+0+0 (secure 0 0 0) 0x7fecd804acc0 con 0x7fecdc1caac0 2023-10-30T12:20:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fecd27fc640 10 client.244224.objecter ms_dispatch 0x7fecdc000b90 osd_map(618..618 src has 490..618) v4 2023-10-30T12:20:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fecd27fc640 3 client.244224.objecter handle_osd_map got epochs [618,618] > 0 2023-10-30T12:20:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fecd27fc640 3 client.244224.objecter handle_osd_map decoding full epoch 618 2023-10-30T12:20:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fecd27fc640 20 client.244224.objecter dump_active .. 0 homeless 2023-10-30T12:20:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fece1c5e640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fecc00392f0 0x7fecc003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fece1c5e640 10 monclient: get_auth_request con 0x7fecc00392f0 auth_method 0 2023-10-30T12:20:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fecd27fc640 10 client.244224.objecter ms_handle_connect 0x7fecc00392f0 2023-10-30T12:20:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.563+0000 7fece1c5e640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fecc00392f0 0x7fecc003b7b0 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 crypto rx=0x7feccc000c00 tx=0x7feccc03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.567+0000 7fecd27fc640 1 -- 172.21.15.50:0/110793624 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fecd801ddb0 con 0x7fecdc1caac0 2023-10-30T12:20:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.567+0000 7fecd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.567+0000 7fecd27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:07.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.815+0000 7fece3ec2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:07.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.815+0000 7fece3ec2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:07.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.815+0000 7fece3ec2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:07.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.815+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fecdc125970 con 0x7fecdc1caac0 2023-10-30T12:20:07.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.816+0000 7fecd27fc640 1 -- 172.21.15.50:0/110793624 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fecdc125970 con 0x7fecdc1caac0 2023-10-30T12:20:07.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.816+0000 7fecd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:07.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.816+0000 7fecd27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:07.819 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":202,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:07.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.817+0000 7fece3ec2640 10 librados: watch_flush enter 2023-10-30T12:20:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.817+0000 7fece3ec2640 10 librados: watch_flush exit 2023-10-30T12:20:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.817+0000 7fece3ec2640 20 client.244224.objecter shutdown clearing up homeless session... 2023-10-30T12:20:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.817+0000 7fece3ec2640 10 client.244224.objecter successfully canceled tick 2023-10-30T12:20:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fecc00392f0 msgr2=0x7fecc003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fecc00392f0 0x7fecc003b7b0 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 crypto rx=0x7feccc000c00 tx=0x7feccc03d040 comp rx=0 tx=0).stop 2023-10-30T12:20:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 10 monclient: shutdown 2023-10-30T12:20:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecdc1caac0 msgr2=0x7fecdc1217c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecdc1caac0 0x7fecdc1217c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0a50d0 tx=0x7fecd8007430 comp rx=0 tx=0).stop 2023-10-30T12:20:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 shutdown_connections 2023-10-30T12:20:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fecc00392f0 0x7fecc003b7b0 unknown :-1 s=CLOSED pgs=4301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecdc1caee0 0x7fecdc121d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fecdc1caac0 0x7fecdc1217c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 --2- 172.21.15.50:0/110793624 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fecdc0a4b60 0x7fecdc1ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 >> 172.21.15.50:0/110793624 conn(0x7fecdc0a8060 msgr2=0x7fecdc0a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 shutdown_connections 2023-10-30T12:20:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 -- 172.21.15.50:0/110793624 wait complete. 2023-10-30T12:20:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:07.818+0000 7fece3ec2640 1 librados: shutdown 2023-10-30T12:20:10.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:10.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.947+0000 7f86334da640 10 monclient: get_monmap_and_config 2023-10-30T12:20:10.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.947+0000 7f86334da640 10 monclient: build_initial_monmap 2023-10-30T12:20:10.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.947+0000 7f86334da640 10 monclient: monmap: 2023-10-30T12:20:10.950 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:10.950 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:10.950 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:10.948685+0000 2023-10-30T12:20:10.950 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:10.948685+0000 2023-10-30T12:20:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:10.952 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:10.952 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:10.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.948+0000 7f86334da640 1 Processor -- start 2023-10-30T12:20:10.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.948+0000 7f86334da640 1 -- start start 2023-10-30T12:20:10.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.948+0000 7f86334da640 10 monclient: init 2023-10-30T12:20:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-10-30T12:20:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f862c0a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f862c099a40 0x7f862c13aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient(hunting): picked mon.noname-a con 0x7f862c099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c13cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient(hunting): picked mon.noname-b con 0x7f862c132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862c0a4f40 con 0x7f862c0a4b60 2023-10-30T12:20:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c0a3d70 con 0x7f862c099a40 2023-10-30T12:20:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0a4150 con 0x7f862c132db0 2023-10-30T12:20:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:10.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f86334da640 10 monclient(hunting): authenticate will time out at 2056.132245s 2023-10-30T12:20:10.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f862c099a40 0x7f862c13aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:10.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c13cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:10.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f862c099a40 0x7f862c13aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37980/0 (socket says 172.21.15.50:37980) 2023-10-30T12:20:10.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 -- 172.21.15.50:0/1326531485 learned_addr learned my addr 172.21.15.50:0/1326531485 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:10.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): get_auth_request con 0x7f862c099a40 auth_method 0 2023-10-30T12:20:10.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:10.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:10.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:10.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 10 monclient(hunting): get_auth_request con 0x7f862c132db0 auth_method 0 2023-10-30T12:20:10.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:10.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:10.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:10.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:10.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:10.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:10.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 1 -- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 msgr2=0x7f862c13cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:20:10.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 1 -- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 msgr2=0x7f862c13cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:10.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 1 --2- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c13cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:10.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8631a77640 1 --2- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c13cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:10.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient(hunting): handle_auth_done global_id 244227 payload 274 2023-10-30T12:20:10.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 -- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 msgr2=0x7f862c13cfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:10.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 --2- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c13cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:10.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 -- 172.21.15.50:0/1326531485 >> v1:172.21.15.50:6789/0 conn(0x7f862c0a4b60 legacy=0x7f862c1326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:10.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient: _finish_hunting 0 2023-10-30T12:20:10.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 monclient: found mon.noname-a 2023-10-30T12:20:10.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:10.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:10.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.949+0000 7f8630a75640 1 -- 172.21.15.50:0/1326531485 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c09a440 con 0x7f862c099a40 2023-10-30T12:20:10.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8630a75640 1 --2- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c099a40 0x7f862c13aa80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f861c009870 tx=0x7f861c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54e131ebc5cd911 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:10.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 1 -- 172.21.15.50:0/1326531485 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c0317c0 con 0x7f862c099a40 2023-10-30T12:20:10.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:10.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:10.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: dump: 2023-10-30T12:20:10.971 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:10.971 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:10.971 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:10.972 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:10.972 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:10.972 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:10.972 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:10.972 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:10.973 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:10.973 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:10.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: _finish_auth 0 2023-10-30T12:20:10.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: _check_auth_tickets 2023-10-30T12:20:10.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:10.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 1 -- 172.21.15.50:0/1326531485 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c031920 con 0x7f862c099a40 2023-10-30T12:20:10.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:10.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 1 -- 172.21.15.50:0/1326531485 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c031cb0 con 0x7f862c099a40 2023-10-30T12:20:10.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f86334da640 5 monclient: authenticate success, global_id 244227 2023-10-30T12:20:10.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f86334da640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:10.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f86334da640 4 set_mon_vals no callback set 2023-10-30T12:20:10.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:10.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:10.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.950+0000 7f8623fff640 10 monclient: dump: 2023-10-30T12:20:10.976 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:10.976 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:10.976 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:10.976 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:10.976 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:10.977 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:10.977 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:10.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:10.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:10.977 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 10 monclient: shutdown 2023-10-30T12:20:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 1 -- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c099a40 msgr2=0x7f862c13aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 1 --2- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c099a40 0x7f862c13aa80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f861c009870 tx=0x7f861c02b860 comp rx=0 tx=0).stop 2023-10-30T12:20:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 1 -- 172.21.15.50:0/1326531485 shutdown_connections 2023-10-30T12:20:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 1 --2- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c13cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 1 --2- 172.21.15.50:0/1326531485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c099a40 0x7f862c13aa80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.951+0000 7f86334da640 1 -- 172.21.15.50:0/1326531485 >> 172.21.15.50:0/1326531485 conn(0x7f862c0a8060 msgr2=0x7f862c0a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.952+0000 7f86334da640 1 -- 172.21.15.50:0/1326531485 shutdown_connections 2023-10-30T12:20:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.952+0000 7f86334da640 1 -- 172.21.15.50:0/1326531485 wait complete. 2023-10-30T12:20:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.952+0000 7f86334da640 10 monclient: build_initial_monmap 2023-10-30T12:20:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.952+0000 7f86334da640 10 monclient: monmap: 2023-10-30T12:20:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:10.982 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 librados: starting msgr at 2023-10-30T12:20:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 librados: starting objecter 2023-10-30T12:20:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 Processor -- start 2023-10-30T12:20:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 -- start start 2023-10-30T12:20:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 librados: setting wanted keys 2023-10-30T12:20:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 librados: calling monclient init 2023-10-30T12:20:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient: init 2023-10-30T12:20:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:20:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f862c0a4b60 0x7f862c1ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): picked mon.noname-b con 0x7f862c0a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c1cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): picked mon.noname-c con 0x7f862c132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c135b70 0x7f862c136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): picked mon.noname-a con 0x7f862c135b70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:10.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c13caa0 con 0x7f862c135b70 2023-10-30T12:20:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0a18d0 con 0x7f862c0a4b60 2023-10-30T12:20:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:10.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c1381d0 con 0x7f862c132db0 2023-10-30T12:20:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f86334da640 10 monclient(hunting): authenticate will time out at 2056.136750s 2023-10-30T12:20:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c1cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c135b70 0x7f862c136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c1cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39434/0 (socket says 172.21.15.50:39434) 2023-10-30T12:20:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c135b70 0x7f862c136000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37982/0 (socket says 172.21.15.50:37982) 2023-10-30T12:20:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 1 -- 172.21.15.50:0/649424186 learned_addr learned my addr 172.21.15.50:0/649424186 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 10 monclient(hunting): get_auth_request con 0x7f862c132db0 auth_method 0 2023-10-30T12:20:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 10 monclient(hunting): get_auth_request con 0x7f862c135b70 auth_method 0 2023-10-30T12:20:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:10.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8631a77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.953+0000 7f8630a75640 1 -- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 msgr2=0x7f862c1cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:20:10.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8630a75640 1 -- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 msgr2=0x7f862c1cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:10.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8630a75640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c1cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:10.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f862c0a4b60 0x7f862c1ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:10.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8630a75640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c1cad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:10.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 10 monclient(hunting): get_auth_request con 0x7f862c0a4b60 auth_method 0 2023-10-30T12:20:10.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631276640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 10 monclient(hunting): handle_auth_done global_id 244230 payload 824 2023-10-30T12:20:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 1 -- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 msgr2=0x7f862c1cad40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c1cad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 1 -- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f862c0a4b60 msgr2=0x7f862c1ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f862c0a4b60 0x7f862c1ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 10 monclient: _finish_hunting 0 2023-10-30T12:20:10.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 1 monclient: found mon.noname-a 2023-10-30T12:20:10.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:10.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:10.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 1 -- 172.21.15.50:0/649424186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c0094d0 con 0x7f862c135b70 2023-10-30T12:20:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8631a77640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c135b70 0x7f862c136000 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8624007cf0 tx=0x7f862400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 client.?.objecter ms_handle_connect 0x7f862c135b70 2023-10-30T12:20:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 1 -- 172.21.15.50:0/649424186 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624018070 con 0x7f862c135b70 2023-10-30T12:20:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:10.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:10.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: dump: 2023-10-30T12:20:10.995 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:10.995 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:10.995 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:10.996 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:10.996 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:10.996 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:10.996 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:10.996 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:10.997 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: _finish_auth 0 2023-10-30T12:20:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:20:10.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 1 -- 172.21.15.50:0/649424186 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862400edd0 con 0x7f862c135b70 2023-10-30T12:20:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 5 monclient: authenticate success, global_id 244230 2023-10-30T12:20:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 10 monclient: _renew_subs 2023-10-30T12:20:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:10.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862c139340 con 0x7f862c135b70 2023-10-30T12:20:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 1 -- 172.21.15.50:0/649424186 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624003bd0 con 0x7f862c135b70 2023-10-30T12:20:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f8621ffb640 10 monclient: dump: 2023-10-30T12:20:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:11.001 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86237fe640 4 set_mon_vals no callback set 2023-10-30T12:20:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 10 client.244230.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 10 monclient: _renew_subs 2023-10-30T12:20:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c137810 con 0x7f862c135b70 2023-10-30T12:20:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.954+0000 7f86334da640 1 librados: init done 2023-10-30T12:20:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f8621ffb640 1 -- 172.21.15.50:0/649424186 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8624014420 con 0x7f862c135b70 2023-10-30T12:20:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f8621ffb640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86100392f0 0x7f861003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f8621ffb640 1 -- 172.21.15.50:0/649424186 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(621..621 src has 490..621) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f862404af10 con 0x7f862c135b70 2023-10-30T12:20:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f8621ffb640 10 client.244230.objecter ms_dispatch 0x7f862c000b90 osd_map(621..621 src has 490..621) v4 2023-10-30T12:20:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f86334da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f8621ffb640 3 client.244230.objecter handle_osd_map got epochs [621,621] > 0 2023-10-30T12:20:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f8621ffb640 3 client.244230.objecter handle_osd_map decoding full epoch 621 2023-10-30T12:20:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f86334da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f86334da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862c04e160 con 0x7f862c135b70 2023-10-30T12:20:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.955+0000 7f8621ffb640 20 client.244230.objecter dump_active .. 0 homeless 2023-10-30T12:20:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.956+0000 7f8631276640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86100392f0 0x7f861003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:11.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.956+0000 7f8631276640 10 monclient: get_auth_request con 0x7f86100392f0 auth_method 0 2023-10-30T12:20:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.956+0000 7f8631276640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86100392f0 0x7f861003b7b0 secure :-1 s=READY pgs=4302 cs=0 l=1 rev1=1 crypto rx=0x7f8628000f60 tx=0x7f8628009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.956+0000 7f8621ffb640 10 client.244230.objecter ms_handle_connect 0x7f86100392f0 2023-10-30T12:20:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.960+0000 7f8621ffb640 1 -- 172.21.15.50:0/649424186 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f86240146d0 con 0x7f862c135b70 2023-10-30T12:20:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.960+0000 7f8621ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:11.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:10.960+0000 7f8621ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:11.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.206+0000 7f86334da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:11.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.206+0000 7f86334da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:11.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.206+0000 7f86334da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:11.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.206+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f862c0745c0 con 0x7f862c135b70 2023-10-30T12:20:11.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.206+0000 7f8621ffb640 1 -- 172.21.15.50:0/649424186 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f862401b310 con 0x7f862c135b70 2023-10-30T12:20:11.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.206+0000 7f8621ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:11.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.206+0000 7f8621ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:11.210 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":202,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:11.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 10 librados: watch_flush enter 2023-10-30T12:20:11.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 10 librados: watch_flush exit 2023-10-30T12:20:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 20 client.244230.objecter shutdown clearing up homeless session... 2023-10-30T12:20:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 10 client.244230.objecter successfully canceled tick 2023-10-30T12:20:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86100392f0 msgr2=0x7f861003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86100392f0 0x7f861003b7b0 secure :-1 s=READY pgs=4302 cs=0 l=1 rev1=1 crypto rx=0x7f8628000f60 tx=0x7f8628009040 comp rx=0 tx=0).stop 2023-10-30T12:20:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 10 monclient: shutdown 2023-10-30T12:20:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.208+0000 7f86334da640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c135b70 msgr2=0x7f862c136000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c135b70 0x7f862c136000 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8624007cf0 tx=0x7f862400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:20:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 shutdown_connections 2023-10-30T12:20:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f86100392f0 0x7f861003b7b0 unknown :-1 s=CLOSED pgs=4302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f862c135b70 0x7f862c136000 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f862c132db0 0x7f862c1cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 --2- 172.21.15.50:0/649424186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f862c0a4b60 0x7f862c1ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 >> 172.21.15.50:0/649424186 conn(0x7f862c0a8060 msgr2=0x7f862c0a5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:11.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 shutdown_connections 2023-10-30T12:20:11.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 -- 172.21.15.50:0/649424186 wait complete. 2023-10-30T12:20:11.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:11.209+0000 7f86334da640 1 librados: shutdown 2023-10-30T12:20:14.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.336+0000 7fcef367c640 10 monclient: get_monmap_and_config 2023-10-30T12:20:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.336+0000 7fcef367c640 10 monclient: build_initial_monmap 2023-10-30T12:20:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.336+0000 7fcef367c640 10 monclient: monmap: 2023-10-30T12:20:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:14.337341+0000 2023-10-30T12:20:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:14.337341+0000 2023-10-30T12:20:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:14.340 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.336+0000 7fcef367c640 1 Processor -- start 2023-10-30T12:20:14.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 1 -- start start 2023-10-30T12:20:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient: init 2023-10-30T12:20:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-10-30T12:20:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fceec0a0080 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:14.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 0x7fceec13ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient(hunting): picked mon.noname-b con 0x7fceec0720b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fceec073a30 0x7fceec13d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient(hunting): picked mon.noname-a con 0x7fceec073a30 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:14.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceec09ccc0 con 0x7fceec0a0080 2023-10-30T12:20:14.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceec073620 con 0x7fceec073a30 2023-10-30T12:20:14.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec071e90 con 0x7fceec0720b0 2023-10-30T12:20:14.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:14.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef367c640 10 monclient(hunting): authenticate will time out at 2059.520732s 2023-10-30T12:20:14.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef0c17640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 0x7fceec13ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef1c19640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fceec073a30 0x7fceec13d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef1c19640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fceec073a30 0x7fceec13d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37990/0 (socket says 172.21.15.50:37990) 2023-10-30T12:20:14.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef0c17640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 0x7fceec13ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39454/0 (socket says 172.21.15.50:39454) 2023-10-30T12:20:14.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef1c19640 1 -- 172.21.15.50:0/4230986503 learned_addr learned my addr 172.21.15.50:0/4230986503 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:14.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef1c19640 10 monclient(hunting): get_auth_request con 0x7fceec073a30 auth_method 0 2023-10-30T12:20:14.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef1c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef1c19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef1c19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef0c17640 10 monclient(hunting): get_auth_request con 0x7fceec0720b0 auth_method 0 2023-10-30T12:20:14.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef0c17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef0c17640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.337+0000 7fcef0c17640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef0c17640 1 -- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 msgr2=0x7fceec13ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:20:14.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef0c17640 1 -- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 msgr2=0x7fceec13ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:14.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef0c17640 1 --2- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 0x7fceec13ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:14.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef0c17640 1 --2- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 0x7fceec13ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:14.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 10 monclient(hunting): handle_auth_done global_id 244233 payload 274 2023-10-30T12:20:14.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 1 -- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 msgr2=0x7fceec13ad30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:14.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 1 --2- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 0x7fceec13ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 1 -- 172.21.15.50:0/4230986503 >> v1:172.21.15.50:6789/0 conn(0x7fceec0a0080 legacy=0x7fceec0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 10 monclient: _finish_hunting 0 2023-10-30T12:20:14.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 1 monclient: found mon.noname-a 2023-10-30T12:20:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.338+0000 7fcef1c19640 1 -- 172.21.15.50:0/4230986503 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec09c2b0 con 0x7fceec073a30 2023-10-30T12:20:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef1c19640 1 --2- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec13d280 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009e50 tx=0x7fcee402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64314aff2fa1d9ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 1 -- 172.21.15.50:0/4230986503 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee402c9e0 con 0x7fceec073a30 2023-10-30T12:20:14.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:14.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: dump: 2023-10-30T12:20:14.358 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.358 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.358 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.359 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.360 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: _finish_auth 0 2023-10-30T12:20:14.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:20:14.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:14.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 1 -- 172.21.15.50:0/4230986503 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee402cb40 con 0x7fceec073a30 2023-10-30T12:20:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 5 monclient: authenticate success, global_id 244233 2023-10-30T12:20:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 4 set_mon_vals no callback set 2023-10-30T12:20:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 1 -- 172.21.15.50:0/4230986503 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee40343f0 con 0x7fceec073a30 2023-10-30T12:20:14.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:14.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcee3fff640 10 monclient: dump: 2023-10-30T12:20:14.362 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.363 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.363 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.363 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.363 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.363 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.364 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.365 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.365 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 10 monclient: shutdown 2023-10-30T12:20:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 -- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 msgr2=0x7fceec13d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 --2- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec13d280 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009e50 tx=0x7fcee402be40 comp rx=0 tx=0).stop 2023-10-30T12:20:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 -- 172.21.15.50:0/4230986503 shutdown_connections 2023-10-30T12:20:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 --2- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec13d280 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 --2- 172.21.15.50:0/4230986503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec0720b0 0x7fceec13ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 -- 172.21.15.50:0/4230986503 >> 172.21.15.50:0/4230986503 conn(0x7fceec0a4870 msgr2=0x7fceec0a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 -- 172.21.15.50:0/4230986503 shutdown_connections 2023-10-30T12:20:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.339+0000 7fcef367c640 1 -- 172.21.15.50:0/4230986503 wait complete. 2023-10-30T12:20:14.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 10 monclient: build_initial_monmap 2023-10-30T12:20:14.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 10 monclient: monmap: 2023-10-30T12:20:14.368 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:14.368 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:14.370 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 1 librados: starting msgr at 2023-10-30T12:20:14.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 1 librados: starting objecter 2023-10-30T12:20:14.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 1 Processor -- start 2023-10-30T12:20:14.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 1 -- start start 2023-10-30T12:20:14.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 1 librados: setting wanted keys 2023-10-30T12:20:14.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 1 librados: calling monclient init 2023-10-30T12:20:14.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.340+0000 7fcef367c640 10 monclient: init 2023-10-30T12:20:14.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:14.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:20:14.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec1ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): picked mon.noname-a con 0x7fceec073a30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:14.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceec0a0080 0x7fceec1b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): picked mon.noname-b con 0x7fceec0a0080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:14.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 0x7fceec1b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): picked mon.noname-c con 0x7fceec1ca670 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:14.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec09b1d0 con 0x7fceec073a30 2023-10-30T12:20:14.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec073e90 con 0x7fceec0a0080 2023-10-30T12:20:14.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec1afab0 con 0x7fceec1ca670 2023-10-30T12:20:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef367c640 10 monclient(hunting): authenticate will time out at 2059.524366s 2023-10-30T12:20:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec1ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec1ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37992/0 (socket says 172.21.15.50:37992) 2023-10-30T12:20:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 -- 172.21.15.50:0/3200810150 learned_addr learned my addr 172.21.15.50:0/3200810150 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 0x7fceec1b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): get_auth_request con 0x7fceec073a30 auth_method 0 2023-10-30T12:20:14.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 10 monclient(hunting): get_auth_request con 0x7fceec1ca670 auth_method 0 2023-10-30T12:20:14.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 1 -- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 msgr2=0x7fceec1b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:20:14.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 1 -- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 msgr2=0x7fceec1b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:14.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 0x7fceec1b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:14.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1c19640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 0x7fceec1b7e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:14.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient(hunting): handle_auth_done global_id 244239 payload 824 2023-10-30T12:20:14.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 -- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 msgr2=0x7fceec1b7e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:14.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 0x7fceec1b7e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 -- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceec0a0080 msgr2=0x7fceec1b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef0c17640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceec0a0080 0x7fceec1b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceec0a0080 0x7fceec1b78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient: _finish_hunting 0 2023-10-30T12:20:14.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 monclient: found mon.noname-a 2023-10-30T12:20:14.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:14.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 -- 172.21.15.50:0/3200810150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee4009b30 con 0x7fceec073a30 2023-10-30T12:20:14.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcef1418640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec1ca130 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcee800c4d0 tx=0x7fcee800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcee1ffb640 10 client.?.objecter ms_handle_connect 0x7fceec073a30 2023-10-30T12:20:14.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.341+0000 7fcee1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:14.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 1 -- 172.21.15.50:0/3200810150 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8005480 con 0x7fceec073a30 2023-10-30T12:20:14.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:14.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: dump: 2023-10-30T12:20:14.387 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.387 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.387 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.387 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.388 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.388 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.388 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.388 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.388 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.388 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:20:14.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:20:14.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:14.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 5 monclient: authenticate success, global_id 244239 2023-10-30T12:20:14.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 10 monclient: _renew_subs 2023-10-30T12:20:14.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 1 -- 172.21.15.50:0/3200810150 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee80035f0 con 0x7fceec073a30 2023-10-30T12:20:14.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceec1afd90 con 0x7fceec073a30 2023-10-30T12:20:14.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:14.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 1 -- 172.21.15.50:0/3200810150 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee8003960 con 0x7fceec073a30 2023-10-30T12:20:14.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:14.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee1ffb640 10 monclient: dump: 2023-10-30T12:20:14.392 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.392 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.392 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.392 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.392 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.393 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.393 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.393 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.393 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.394 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcee37fe640 4 set_mon_vals no callback set 2023-10-30T12:20:14.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 10 client.244239.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:14.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 10 monclient: _renew_subs 2023-10-30T12:20:14.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceec1b8360 con 0x7fceec073a30 2023-10-30T12:20:14.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.342+0000 7fcef367c640 1 librados: init done 2023-10-30T12:20:14.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcee1ffb640 1 -- 172.21.15.50:0/3200810150 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcee8003ac0 con 0x7fceec073a30 2023-10-30T12:20:14.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcee1ffb640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fced00392f0 0x7fced003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcee1ffb640 1 -- 172.21.15.50:0/3200810150 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(624..624 src has 490..624) v4 ==== 6683+0+0 (secure 0 0 0) 0x7fcee804a360 con 0x7fceec073a30 2023-10-30T12:20:14.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcee1ffb640 10 client.244239.objecter ms_dispatch 0x7fceec000b90 osd_map(624..624 src has 490..624) v4 2023-10-30T12:20:14.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcee1ffb640 3 client.244239.objecter handle_osd_map got epochs [624,624] > 0 2023-10-30T12:20:14.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcee1ffb640 3 client.244239.objecter handle_osd_map decoding full epoch 624 2023-10-30T12:20:14.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcef367c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:14.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcef367c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:14.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcef367c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcee1ffb640 20 client.244239.objecter dump_active .. 0 homeless 2023-10-30T12:20:14.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceec04e160 con 0x7fceec073a30 2023-10-30T12:20:14.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcef0c17640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fced00392f0 0x7fced003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.343+0000 7fcef0c17640 10 monclient: get_auth_request con 0x7fced00392f0 auth_method 0 2023-10-30T12:20:14.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.344+0000 7fcef0c17640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fced00392f0 0x7fced003b7b0 secure :-1 s=READY pgs=4303 cs=0 l=1 rev1=1 crypto rx=0x7fceec1ae8e0 tx=0x7fcedc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.344+0000 7fcee1ffb640 10 client.244239.objecter ms_handle_connect 0x7fced00392f0 2023-10-30T12:20:14.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.348+0000 7fcee1ffb640 1 -- 172.21.15.50:0/3200810150 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcee801b3b0 con 0x7fceec073a30 2023-10-30T12:20:14.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.348+0000 7fcee1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:14.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.348+0000 7fcee1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:14.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.593+0000 7fcef367c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.593+0000 7fcef367c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.593+0000 7fcef367c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.593+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fceec1b0400 con 0x7fceec073a30 2023-10-30T12:20:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.593+0000 7fcee1ffb640 1 -- 172.21.15.50:0/3200810150 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fcee8017070 con 0x7fceec073a30 2023-10-30T12:20:14.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.593+0000 7fcee1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:14.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.593+0000 7fcee1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:14.596 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:14.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 10 librados: watch_flush enter 2023-10-30T12:20:14.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 10 librados: watch_flush exit 2023-10-30T12:20:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 20 client.244239.objecter shutdown clearing up homeless session... 2023-10-30T12:20:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 10 client.244239.objecter successfully canceled tick 2023-10-30T12:20:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fced00392f0 msgr2=0x7fced003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fced00392f0 0x7fced003b7b0 secure :-1 s=READY pgs=4303 cs=0 l=1 rev1=1 crypto rx=0x7fceec1ae8e0 tx=0x7fcedc008040 comp rx=0 tx=0).stop 2023-10-30T12:20:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 10 monclient: shutdown 2023-10-30T12:20:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 msgr2=0x7fceec1ca130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec1ca130 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcee800c4d0 tx=0x7fcee800c990 comp rx=0 tx=0).stop 2023-10-30T12:20:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 shutdown_connections 2023-10-30T12:20:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fced00392f0 0x7fced003b7b0 unknown :-1 s=CLOSED pgs=4303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fceec1ca670 0x7fceec1b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fceec0a0080 0x7fceec1b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 --2- 172.21.15.50:0/3200810150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fceec073a30 0x7fceec1ca130 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 >> 172.21.15.50:0/3200810150 conn(0x7fceec0a4870 msgr2=0x7fceec09ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 shutdown_connections 2023-10-30T12:20:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 -- 172.21.15.50:0/3200810150 wait complete. 2023-10-30T12:20:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.595+0000 7fcef367c640 1 librados: shutdown 2023-10-30T12:20:14.607 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:20:14.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:20:14.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:20:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.712+0000 7f64e1052640 10 monclient: get_monmap_and_config 2023-10-30T12:20:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.712+0000 7f64e1052640 10 monclient: build_initial_monmap 2023-10-30T12:20:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.712+0000 7f64e1052640 10 monclient: monmap: 2023-10-30T12:20:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:14.713455+0000 2023-10-30T12:20:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:14.713455+0000 2023-10-30T12:20:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:14.716 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:14.716 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:14.716 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:14.716 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:14.716 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 1 Processor -- start 2023-10-30T12:20:14.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 1 -- start start 2023-10-30T12:20:14.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient: init 2023-10-30T12:20:14.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:14.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-10-30T12:20:14.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 0x7f64dc096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient(hunting): picked mon.noname-b con 0x7f64dc096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:14.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64dc096e60 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:14.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f64dc09b170 0x7f64dc09b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient(hunting): picked mon.noname-a con 0x7f64dc09b170 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:14.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc09bb20 con 0x7f64dc096e60 2023-10-30T12:20:14.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64dc09bd30 con 0x7f64dc09b170 2023-10-30T12:20:14.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc09c400 con 0x7f64dc096540 2023-10-30T12:20:14.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:14.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64e1052640 10 monclient(hunting): authenticate will time out at 2059.896722s 2023-10-30T12:20:14.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 0x7f64dc096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f64dc09b170 0x7f64dc09b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 0x7f64dc096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39462/0 (socket says 172.21.15.50:39462) 2023-10-30T12:20:14.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 1 -- 172.21.15.50:0/3115069526 learned_addr learned my addr 172.21.15.50:0/3115069526 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:14.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 10 monclient(hunting): get_auth_request con 0x7f64dc096540 auth_method 0 2023-10-30T12:20:14.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 10 monclient(hunting): get_auth_request con 0x7f64dc09b170 auth_method 0 2023-10-30T12:20:14.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64db59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 1 -- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 msgr2=0x7f64dc096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:14.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 1 -- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 msgr2=0x7f64dc096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:14.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.713+0000 7f64dad9d640 1 --2- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 0x7f64dc096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:14.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64dad9d640 1 --2- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 0x7f64dc096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:14.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 10 monclient(hunting): handle_auth_done global_id 244242 payload 274 2023-10-30T12:20:14.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 1 -- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 msgr2=0x7f64dc096920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:14.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 1 --2- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 0x7f64dc096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 1 -- 172.21.15.50:0/3115069526 >> v1:172.21.15.118:6789/0 conn(0x7f64dc096e60 legacy=0x7f64dc09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:14.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 10 monclient: _finish_hunting 0 2023-10-30T12:20:14.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 1 monclient: found mon.noname-a 2023-10-30T12:20:14.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:14.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 1 -- 172.21.15.50:0/3115069526 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc09c120 con 0x7f64dc09b170 2023-10-30T12:20:14.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64db59e640 1 --2- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc09b170 0x7f64dc09b5e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009d90 tx=0x7f64d402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1e16e6d26ceb58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.714+0000 7f64da59c640 1 -- 172.21.15.50:0/3115069526 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d402cbc0 con 0x7f64dc09b170 2023-10-30T12:20:14.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:14.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: dump: 2023-10-30T12:20:14.735 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.735 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.735 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.735 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.736 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.736 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.736 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.736 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.736 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.736 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: _finish_auth 0 2023-10-30T12:20:14.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: _check_auth_tickets 2023-10-30T12:20:14.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:14.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 5 monclient: authenticate success, global_id 244242 2023-10-30T12:20:14.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:14.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 1 -- 172.21.15.50:0/3115069526 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d402cd20 con 0x7f64dc09b170 2023-10-30T12:20:14.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:14.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:14.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 4 set_mon_vals no callback set 2023-10-30T12:20:14.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 1 -- 172.21.15.50:0/3115069526 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4034660 con 0x7f64dc09b170 2023-10-30T12:20:14.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:14.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64da59c640 10 monclient: dump: 2023-10-30T12:20:14.739 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.739 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.740 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.740 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.740 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.740 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.740 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.740 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.741 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.741 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 10 monclient: shutdown 2023-10-30T12:20:14.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:14.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 -- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc09b170 msgr2=0x7f64dc09b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 --2- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc09b170 0x7f64dc09b5e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009d90 tx=0x7f64d402be10 comp rx=0 tx=0).stop 2023-10-30T12:20:14.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 -- 172.21.15.50:0/3115069526 shutdown_connections 2023-10-30T12:20:14.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 --2- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc09b170 0x7f64dc09b5e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 --2- 172.21.15.50:0/3115069526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc096540 0x7f64dc096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 -- 172.21.15.50:0/3115069526 >> 172.21.15.50:0/3115069526 conn(0x7f64dc0a3980 msgr2=0x7f64dc0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:14.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 -- 172.21.15.50:0/3115069526 shutdown_connections 2023-10-30T12:20:14.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.715+0000 7f64e1052640 1 -- 172.21.15.50:0/3115069526 wait complete. 2023-10-30T12:20:14.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 10 monclient: build_initial_monmap 2023-10-30T12:20:14.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 10 monclient: monmap: 2023-10-30T12:20:14.743 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:14.743 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:14.744 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:14.744 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:14.744 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:14.744 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.744 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:14.744 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:14.745 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:14.745 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 1 librados: starting msgr at 2023-10-30T12:20:14.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 1 librados: starting objecter 2023-10-30T12:20:14.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 1 Processor -- start 2023-10-30T12:20:14.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 1 -- start start 2023-10-30T12:20:14.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 1 librados: setting wanted keys 2023-10-30T12:20:14.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 1 librados: calling monclient init 2023-10-30T12:20:14.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.716+0000 7f64e1052640 10 monclient: init 2023-10-30T12:20:14.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:14.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:20:14.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc096540 0x7f64dc1ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): picked mon.noname-a con 0x7f64dc096540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:14.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64dc096e60 0x7f64dc0943b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): picked mon.noname-b con 0x7f64dc096e60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:14.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 0x7f64dc0948f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): picked mon.noname-c con 0x7f64dc09b170 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:14.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc090e60 con 0x7f64dc096540 2023-10-30T12:20:14.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc0911b0 con 0x7f64dc096e60 2023-10-30T12:20:14.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc0908a0 con 0x7f64dc09b170 2023-10-30T12:20:14.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:14.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64e1052640 10 monclient(hunting): authenticate will time out at 2059.900286s 2023-10-30T12:20:14.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc096540 0x7f64dc1ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 0x7f64dc0948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc096540 0x7f64dc1ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38014/0 (socket says 172.21.15.50:38014) 2023-10-30T12:20:14.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 -- 172.21.15.50:0/3262198614 learned_addr learned my addr 172.21.15.50:0/3262198614 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:14.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): get_auth_request con 0x7f64dc096540 auth_method 0 2023-10-30T12:20:14.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 10 monclient(hunting): get_auth_request con 0x7f64dc09b170 auth_method 0 2023-10-30T12:20:14.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 msgr2=0x7f64dc0948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:20:14.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 msgr2=0x7f64dc0948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:14.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 0x7f64dc0948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:14.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64db59e640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 0x7f64dc0948f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:14.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64dc096e60 0x7f64dc0943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 10 monclient(hunting): get_auth_request con 0x7f64dc096e60 auth_method 0 2023-10-30T12:20:14.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient(hunting): handle_auth_done global_id 244245 payload 824 2023-10-30T12:20:14.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 msgr2=0x7f64dc0948f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:14.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f64dc09b170 0x7f64dc0948f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64dc096e60 msgr2=0x7f64dc0943b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64dc096e60 0x7f64dc0943b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient: _finish_hunting 0 2023-10-30T12:20:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 monclient: found mon.noname-a 2023-10-30T12:20:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 -- 172.21.15.50:0/3262198614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d4009a70 con 0x7f64dc096540 2023-10-30T12:20:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64dad9d640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc096540 0x7f64dc1ca170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00e840 tx=0x7f64cc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d37fe640 10 client.?.objecter ms_handle_connect 0x7f64dc096540 2023-10-30T12:20:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.717+0000 7f64d37fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc00c6d0 con 0x7f64dc096540 2023-10-30T12:20:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: dump: 2023-10-30T12:20:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.759 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: _finish_auth 0 2023-10-30T12:20:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:20:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc00c830 con 0x7f64dc096540 2023-10-30T12:20:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 5 monclient: authenticate success, global_id 244245 2023-10-30T12:20:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 10 monclient: _renew_subs 2023-10-30T12:20:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc0905c0 con 0x7f64dc096540 2023-10-30T12:20:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc00cbe0 con 0x7f64dc096540 2023-10-30T12:20:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 10 monclient: dump: 2023-10-30T12:20:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.763 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d9d9b640 4 set_mon_vals no callback set 2023-10-30T12:20:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 10 client.244245.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 10 monclient: _renew_subs 2023-10-30T12:20:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc098e10 con 0x7f64dc096540 2023-10-30T12:20:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 1 librados: init done 2023-10-30T12:20:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097560 0x7f64dc097940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc073ff0 con 0x7f64dc097560 2023-10-30T12:20:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.718+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f64cc00cd40 con 0x7f64dc096540 2023-10-30T12:20:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097560 0x7f64dc097940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient: get_auth_request con 0x7f64dc097560 auth_method 0 2023-10-30T12:20:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64bc0392f0 0x7f64bc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(624..624 src has 490..624) v4 ==== 6683+0+0 (secure 0 0 0) 0x7f64cc04ac70 con 0x7f64dc096540 2023-10-30T12:20:14.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 client.244245.objecter ms_dispatch 0x7f64dc000b90 osd_map(624..624 src has 490..624) v4 2023-10-30T12:20:14.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 3 client.244245.objecter handle_osd_map got epochs [624,624] > 0 2023-10-30T12:20:14.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 3 client.244245.objecter handle_osd_map decoding full epoch 624 2023-10-30T12:20:14.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 10 monclient(hunting): handle_auth_done global_id 244248 payload 824 2023-10-30T12:20:14.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 20 client.244245.objecter dump_active .. 0 homeless 2023-10-30T12:20:14.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 1 -- 172.21.15.50:0/3262198614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64dc09d3a0 con 0x7f64dc097560 2023-10-30T12:20:14.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 client.244245.objecter ms_handle_connect 0x7f64dc097560 2023-10-30T12:20:14.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d3fff640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097560 0x7f64dc097940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0742b0 tx=0x7f64c400fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 client.244245.objecter resend_mon_ops 2023-10-30T12:20:14.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64db59e640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64bc0392f0 0x7f64bc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40067d0 con 0x7f64dc097560 2023-10-30T12:20:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 monclient: dump: 2023-10-30T12:20:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.772 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64db59e640 10 monclient: get_auth_request con 0x7f64bc0392f0 auth_method 0 2023-10-30T12:20:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f64c4006930 con 0x7f64dc097560 2023-10-30T12:20:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097560 msgr2=0x7f64dc097940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097560 0x7f64dc097940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0742b0 tx=0x7f64c400fc00 comp rx=0 tx=0).stop 2023-10-30T12:20:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097560 msgr2=0x7f64dc097940 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.719+0000 7f64d37fe640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097560 0x7f64dc097940 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0742b0 tx=0x7f64c400fc00 comp rx=0 tx=0).stop 2023-10-30T12:20:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.720+0000 7f64d37fe640 10 client.244245.objecter ms_handle_connect 0x7f64bc0392f0 2023-10-30T12:20:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.720+0000 7f64db59e640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64bc0392f0 0x7f64bc03b7b0 secure :-1 s=READY pgs=4304 cs=0 l=1 rev1=1 crypto rx=0x7f64d4036000 tx=0x7f64d4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64e1052640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64e1052640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097e80 0x7f64dc06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64e1052640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc090c90 con 0x7f64dc097e80 2023-10-30T12:20:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097e80 0x7f64dc06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient: get_auth_request con 0x7f64dc097e80 auth_method 0 2023-10-30T12:20:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 10 monclient(hunting): handle_auth_done global_id 244251 payload 824 2023-10-30T12:20:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 1 -- 172.21.15.50:0/3262198614 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f64dc099090 con 0x7f64dc097e80 2023-10-30T12:20:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d37fe640 10 client.244245.objecter ms_handle_connect 0x7f64dc097e80 2023-10-30T12:20:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d3fff640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097e80 0x7f64dc06dbb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0742b0 tx=0x7f64c4003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d37fe640 10 client.244245.objecter resend_mon_ops 2023-10-30T12:20:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4024070 con 0x7f64dc097e80 2023-10-30T12:20:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.726+0000 7f64d37fe640 10 monclient: dump: 2023-10-30T12:20:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.782 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7f64c400daa0 con 0x7f64dc097e80 2023-10-30T12:20:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64d37fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097e80 msgr2=0x7f64dc06dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64d37fe640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097e80 0x7f64dc06dbb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0742b0 tx=0x7f64c4003760 comp rx=0 tx=0).stop 2023-10-30T12:20:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64d37fe640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097e80 msgr2=0x7f64dc06dbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64d37fe640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc097e80 0x7f64dc06dbb0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0742b0 tx=0x7f64c4003760 comp rx=0 tx=0).stop 2023-10-30T12:20:14.784 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:14.784 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:14.784 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:14.784 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:20:14.784 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 204, 2023-10-30T12:20:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:20:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:20:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:14.789 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:14.789 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:14.789 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:14.789 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:14.789 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.806 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:14.806 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:14.806 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:14.806 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.806 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:14.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.807 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:14.807 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:14.807 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:14.807 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:14.808 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:14.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.808 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:14.808 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:14.809 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:14.809 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:14.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.809 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:14.809 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:14.810 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:14.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.810 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:14.810 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:14.811 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:14.811 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.811 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:14.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.811 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:14.811 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:14.812 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:14.812 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:14.812 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:14.812 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.812 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:14.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.813 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:14.813 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:14.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.813 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:14.814 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:14.814 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:14.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.814 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:14.814 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:20:14.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.815 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:14.815 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:14.815 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:20:14.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.815 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:14.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:14.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.816 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:14.816 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:14.816 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:20:14.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.817 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:14.817 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:20:14.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:14.817 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:14.817 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:14.818 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:14.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:14.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:14.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:14.819 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:14.819 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:14.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 10 librados: watch_flush enter 2023-10-30T12:20:14.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 10 librados: watch_flush exit 2023-10-30T12:20:14.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 20 client.244245.objecter shutdown clearing up homeless session... 2023-10-30T12:20:14.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 10 client.244245.objecter successfully canceled tick 2023-10-30T12:20:14.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64bc0392f0 msgr2=0x7f64bc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f64bc0392f0 0x7f64bc03b7b0 secure :-1 s=READY pgs=4304 cs=0 l=1 rev1=1 crypto rx=0x7f64d4036000 tx=0x7f64d4030000 comp rx=0 tx=0).stop 2023-10-30T12:20:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 10 monclient: shutdown 2023-10-30T12:20:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc096540 msgr2=0x7f64dc1ca170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc096540 0x7f64dc1ca170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00e840 tx=0x7f64cc00ed00 comp rx=0 tx=0).stop 2023-10-30T12:20:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64db59e640 1 -- 172.21.15.50:0/3262198614 reap_dead start 2023-10-30T12:20:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 shutdown_connections 2023-10-30T12:20:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 --2- 172.21.15.50:0/3262198614 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64dc096540 0x7f64dc1ca170 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 >> 172.21.15.50:0/3262198614 conn(0x7f64dc0a3980 msgr2=0x7f64dc1bb840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:14.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 shutdown_connections 2023-10-30T12:20:14.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 -- 172.21.15.50:0/3262198614 wait complete. 2023-10-30T12:20:14.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.727+0000 7f64e1052640 1 librados: shutdown 2023-10-30T12:20:14.825 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:20:14.825 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:20:14.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:20:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.933+0000 7f2eb132c640 10 monclient: get_monmap_and_config 2023-10-30T12:20:14.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.933+0000 7f2eb132c640 10 monclient: build_initial_monmap 2023-10-30T12:20:14.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.933+0000 7f2eb132c640 10 monclient: monmap: 2023-10-30T12:20:14.935 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:14.936 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.936 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:14.934578+0000 2023-10-30T12:20:14.936 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:14.934578+0000 2023-10-30T12:20:14.936 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:14.936 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.937 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:14.937 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:14.937 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:14.937 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:14.937 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:14.938 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 1 Processor -- start 2023-10-30T12:20:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 1 -- start start 2023-10-30T12:20:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient: init 2023-10-30T12:20:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-10-30T12:20:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2eac0a2ae0 0x7f2eac0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient(hunting): picked mon.noname-c con 0x7f2eac0a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 0x7f2eac1356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient(hunting): picked mon.noname-b con 0x7f2eac0a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2eac0a32e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eac0a4190 con 0x7f2eac0a32e0 2023-10-30T12:20:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eac0a2990 con 0x7f2eac0a2ae0 2023-10-30T12:20:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac0a1630 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eb132c640 10 monclient(hunting): authenticate will time out at 2060.117695s 2023-10-30T12:20:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eab7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2eac0a32e0 0x7f2eac135c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52160/0 (socket says 172.21.15.50:52160) 2023-10-30T12:20:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eab7fe640 1 -- 172.21.15.50:0/2113162621 learned_addr learned my addr 172.21.15.50:0/2113162621 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:20:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eaa7fc640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 0x7f2eac1356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eaa7fc640 10 monclient(hunting): get_auth_request con 0x7f2eac0a2ec0 auth_method 0 2023-10-30T12:20:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eaa7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.934+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/2113162621 >> v2:172.21.15.118:3300/0 conn(0x7f2eac0a2ae0 0x7f2eac0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request con 0x7f2eac0a2ae0 auth_method 0 2023-10-30T12:20:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaa7fc640 1 -- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 msgr2=0x7f2eac1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:20:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaa7fc640 1 -- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 msgr2=0x7f2eac1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaa7fc640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 0x7f2eac1356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2ea9ffb640 1 -- 172.21.15.50:0/2113162621 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505636685 0 0) 0x7f2ea4001490 con 0x7f2eac0a32e0 2023-10-30T12:20:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2ea9ffb640 10 monclient(hunting): _negotiate 2023-10-30T12:20:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaa7fc640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 0x7f2eac1356e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2ea9ffb640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2ea9ffb640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2ea9ffb640 10 monclient(hunting): my global_id is 244254 2023-10-30T12:20:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2ea9ffb640 1 -- 172.21.15.50:0/2113162621 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e94003520 con 0x7f2eac0a32e0 2023-10-30T12:20:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_done global_id 204640 payload 274 2023-10-30T12:20:14.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 1 -- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 msgr2=0x7f2eac1356e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 0x7f2eac1356e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 1 -- 172.21.15.50:0/2113162621 >> v1:172.21.15.50:6789/0 conn(0x7f2eac0a32e0 legacy=0x7f2eac135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:20:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 1 monclient: found mon.noname-c 2023-10-30T12:20:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:20:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.935+0000 7f2eaaffd640 1 -- 172.21.15.50:0/2113162621 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac09a2e0 con 0x7f2eac0a2ae0 2023-10-30T12:20:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 0x7f2eac0a2450 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0097a0 tx=0x7f2e9c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=803b998710ccc51a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 1 -- 172.21.15.50:0/2113162621 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0316f0 con 0x7f2eac0a2ae0 2023-10-30T12:20:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: dump: 2023-10-30T12:20:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.958 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.959 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:20:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:20:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2eb132c640 5 monclient: authenticate success, global_id 204640 2023-10-30T12:20:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2eb132c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 1 -- 172.21.15.50:0/2113162621 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9c031850 con 0x7f2eac0a2ae0 2023-10-30T12:20:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 1 -- 172.21.15.50:0/2113162621 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c031bc0 con 0x7f2eac0a2ae0 2023-10-30T12:20:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2eb132c640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2eb132c640 4 set_mon_vals no callback set 2023-10-30T12:20:14.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.936+0000 7f2ea9ffb640 10 monclient: dump: 2023-10-30T12:20:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.964 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 10 monclient: shutdown 2023-10-30T12:20:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 -- 172.21.15.50:0/2113162621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 msgr2=0x7f2eac0a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 0x7f2eac0a2450 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0097a0 tx=0x7f2e9c02b790 comp rx=0 tx=0).stop 2023-10-30T12:20:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 -- 172.21.15.50:0/2113162621 shutdown_connections 2023-10-30T12:20:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a2ec0 0x7f2eac1356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 --2- 172.21.15.50:0/2113162621 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 0x7f2eac0a2450 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 -- 172.21.15.50:0/2113162621 >> 172.21.15.50:0/2113162621 conn(0x7f2eac0a80b0 msgr2=0x7f2eac0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 -- 172.21.15.50:0/2113162621 shutdown_connections 2023-10-30T12:20:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.937+0000 7f2eb132c640 1 -- 172.21.15.50:0/2113162621 wait complete. 2023-10-30T12:20:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.938+0000 7f2eb132c640 10 monclient: build_initial_monmap 2023-10-30T12:20:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.938+0000 7f2eb132c640 10 monclient: monmap: 2023-10-30T12:20:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:14.969 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.938+0000 7f2eb132c640 1 librados: starting msgr at 2023-10-30T12:20:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.938+0000 7f2eb132c640 1 librados: starting objecter 2023-10-30T12:20:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 Processor -- start 2023-10-30T12:20:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 -- start start 2023-10-30T12:20:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 librados: setting wanted keys 2023-10-30T12:20:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 librados: calling monclient init 2023-10-30T12:20:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient: init 2023-10-30T12:20:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:20:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 0x7f2eac09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): picked mon.noname-b con 0x7f2eac0a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eac0a2ec0 0x7f2eac132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): picked mon.noname-a con 0x7f2eac0a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 0x7f2eac1c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): picked mon.noname-c con 0x7f2eac132a50 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac120720 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac120ee0 con 0x7f2eac0a2ae0 2023-10-30T12:20:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac11f080 con 0x7f2eac132a50 2023-10-30T12:20:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.939+0000 7f2eb132c640 10 monclient(hunting): authenticate will time out at 2060.122943s 2023-10-30T12:20:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eac0a2ec0 0x7f2eac132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 0x7f2eac1c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eac0a2ec0 0x7f2eac132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38032/0 (socket says 172.21.15.50:38032) 2023-10-30T12:20:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 -- 172.21.15.50:0/717447324 learned_addr learned my addr 172.21.15.50:0/717447324 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): get_auth_request con 0x7f2eac0a2ec0 auth_method 0 2023-10-30T12:20:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 10 monclient(hunting): get_auth_request con 0x7f2eac132a50 auth_method 0 2023-10-30T12:20:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 0x7f2eac09f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request con 0x7f2eac0a2ae0 auth_method 0 2023-10-30T12:20:14.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 msgr2=0x7f2eac1c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:20:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 msgr2=0x7f2eac1c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 0x7f2eac1c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:14.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eab7fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 0x7f2eac1c9d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient(hunting): handle_auth_done global_id 244257 payload 824 2023-10-30T12:20:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 msgr2=0x7f2eac1c9d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac132a50 0x7f2eac1c9d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 msgr2=0x7f2eac09f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2eac0a2ae0 0x7f2eac09f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:20:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 monclient: found mon.noname-a 2023-10-30T12:20:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 -- 172.21.15.50:0/717447324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c009400 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eaa7fc640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eac0a2ec0 0x7f2eac132510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000d7d0 tx=0x7f2ea000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 client.?.objecter ms_handle_connect 0x7f2eac0a2ec0 2023-10-30T12:20:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea000b6c0 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: dump: 2023-10-30T12:20:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.983 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.983 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.984 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.984 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.984 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.984 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.984 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: _finish_auth 0 2023-10-30T12:20:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: _check_auth_tickets 2023-10-30T12:20:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea000b840 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea000bbf0 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2e937fe640 10 monclient: dump: 2023-10-30T12:20:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:14.989 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 5 monclient: authenticate success, global_id 244257 2023-10-30T12:20:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 10 monclient: _renew_subs 2023-10-30T12:20:14.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eac120c00 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2ea97fa640 4 set_mon_vals no callback set 2023-10-30T12:20:14.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 10 client.244257.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 10 monclient: _renew_subs 2023-10-30T12:20:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.940+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac11f7e0 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.941+0000 7f2eb132c640 1 librados: init done 2023-10-30T12:20:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.941+0000 7f2eb132c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.941+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2ea000bd50 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.941+0000 7f2eb132c640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.941+0000 7f2eb132c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.941+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac073ff0 con 0x7f2eac0a32e0 2023-10-30T12:20:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2e937fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2e980392f0 0x7f2e9803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(624..624 src has 490..624) v4 ==== 6683+0+0 (secure 0 0 0) 0x7f2ea004b2c0 con 0x7f2eac0a2ec0 2023-10-30T12:20:14.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2e937fe640 10 client.244257.objecter ms_dispatch 0x7f2eac000b90 osd_map(624..624 src has 490..624) v4 2023-10-30T12:20:14.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2e937fe640 3 client.244257.objecter handle_osd_map got epochs [624,624] > 0 2023-10-30T12:20:14.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2e937fe640 3 client.244257.objecter handle_osd_map decoding full epoch 624 2023-10-30T12:20:14.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2e937fe640 20 client.244257.objecter dump_active .. 0 homeless 2023-10-30T12:20:14.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 10 monclient: get_auth_request con 0x7f2eac0a32e0 auth_method 0 2023-10-30T12:20:14.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:14.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:14.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:14.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:20:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eab7fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2e980392f0 0x7f2e9803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eab7fe640 10 monclient: get_auth_request con 0x7f2e980392f0 auth_method 0 2023-10-30T12:20:14.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2e937fe640 10 client.244257.objecter ms_handle_connect 0x7f2e980392f0 2023-10-30T12:20:14.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:14.942+0000 7f2eab7fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2e980392f0 0x7f2e9803b7b0 secure :-1 s=READY pgs=4305 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40015f0 tx=0x7f2ea4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 10 monclient: get_auth_request con 0x7f2eac0a32e0 auth_method 0 2023-10-30T12:20:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:20:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.143+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:20:15.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:15.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 10 monclient: get_auth_request con 0x7f2eac0a32e0 auth_method 0 2023-10-30T12:20:15.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:15.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:15.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:15.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:20:15.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:15.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:15.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:15.544+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:20:16.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:16.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 10 monclient: get_auth_request con 0x7f2eac0a32e0 auth_method 0 2023-10-30T12:20:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:20:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:16.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:16.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:16.345+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:20:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.940+0000 7f2e927fc640 10 monclient: tick 2023-10-30T12:20:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.940+0000 7f2e927fc640 10 monclient: _check_auth_tickets 2023-10-30T12:20:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.940+0000 7f2e927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.940+0000 7f2e927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:17.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:17.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 10 monclient: get_auth_request con 0x7f2eac0a32e0 auth_method 0 2023-10-30T12:20:17.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:17.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:17.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.947+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_done global_id 254099 payload 824 2023-10-30T12:20:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2eac09a5c0 con 0x7f2eac0a32e0 2023-10-30T12:20:17.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0a5500 tx=0x7f2e9c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2e937fe640 10 client.244257.objecter ms_handle_connect 0x7f2eac0a32e0 2023-10-30T12:20:17.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2e937fe640 10 client.244257.objecter resend_mon_ops 2023-10-30T12:20:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c044050 con 0x7f2eac0a32e0 2023-10-30T12:20:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:17.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2e937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.948+0000 7f2e937fe640 10 monclient: dump: 2023-10-30T12:20:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:17.959 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:17.959 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.949+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f2e9c045040 con 0x7f2eac0a32e0 2023-10-30T12:20:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.949+0000 7f2e937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:17.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.949+0000 7f2e937fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.949+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.949+0000 7f2e937fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0a5500 tx=0x7f2e9c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.949+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 msgr2=0x7f2eac1ca2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.949+0000 7f2e937fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac0a32e0 0x7f2eac1ca2a0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0a5500 tx=0x7f2e9c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eb132c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eb132c640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac1ca7e0 0x7f2eac09c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eb132c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac099b40 con 0x7f2eac1ca7e0 2023-10-30T12:20:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac1ca7e0 0x7f2eac09c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:17.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient: get_auth_request con 0x7f2eac1ca7e0 auth_method 0 2023-10-30T12:20:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:17.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 10 monclient(hunting): handle_auth_done global_id 254102 payload 824 2023-10-30T12:20:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 1 -- 172.21.15.50:0/717447324 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2eac06b060 con 0x7f2eac1ca7e0 2023-10-30T12:20:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2eaaffd640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac1ca7e0 0x7f2eac09c420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c02b7d0 tx=0x7f2e9c031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2e937fe640 10 client.244257.objecter ms_handle_connect 0x7f2eac1ca7e0 2023-10-30T12:20:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.956+0000 7f2e937fe640 10 client.244257.objecter resend_mon_ops 2023-10-30T12:20:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c034680 con 0x7f2eac1ca7e0 2023-10-30T12:20:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 10 monclient: dump: 2023-10-30T12:20:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:17.969 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:17.969 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:17.969 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:17.969 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:17.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f2e9c003e50 con 0x7f2eac1ca7e0 2023-10-30T12:20:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac1ca7e0 msgr2=0x7f2eac09c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac1ca7e0 0x7f2eac09c420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c02b7d0 tx=0x7f2e9c031000 comp rx=0 tx=0).stop 2023-10-30T12:20:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac1ca7e0 msgr2=0x7f2eac09c420 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:17.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2e937fe640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2eac1ca7e0 0x7f2eac09c420 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c02b7d0 tx=0x7f2e9c031000 comp rx=0 tx=0).stop 2023-10-30T12:20:17.971 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:17.971 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:17.971 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:20:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 204, 2023-10-30T12:20:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:17.972 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:20:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 4, 2023-10-30T12:20:17.973 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:17.974 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:17.975 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:17.976 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:17.977 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:17.978 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:17.979 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:17.980 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:17.980 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:17.980 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:17.980 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:17.980 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:17.981 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:17.981 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:17.981 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:17.981 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:17.981 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:17.982 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:17.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:17.982 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:17.982 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:17.982 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:17.983 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:17.983 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:17.983 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:17.983 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:17.983 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:17.984 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:17.984 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:17.984 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:17.984 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:17.984 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:17.984 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:17.986 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:17.986 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:17.986 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:17.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.986 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:17.987 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:17.987 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:17.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.987 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:17.987 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:17.988 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:17.988 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:17.988 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:17.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.988 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:17.989 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:17.989 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:17.989 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:17.989 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:17.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.990 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:17.990 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:17.990 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:17.990 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:17.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:18.002 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 10 librados: watch_flush enter 2023-10-30T12:20:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 10 librados: watch_flush exit 2023-10-30T12:20:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 20 client.244257.objecter shutdown clearing up homeless session... 2023-10-30T12:20:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 10 client.244257.objecter successfully canceled tick 2023-10-30T12:20:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2e980392f0 msgr2=0x7f2e9803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2e980392f0 0x7f2e9803b7b0 secure :-1 s=READY pgs=4305 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40015f0 tx=0x7f2ea4009040 comp rx=0 tx=0).stop 2023-10-30T12:20:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 10 monclient: shutdown 2023-10-30T12:20:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:18.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eac0a2ec0 msgr2=0x7f2eac132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 1 --2- 172.21.15.50:0/717447324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2eac0a2ec0 0x7f2eac132510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000d7d0 tx=0x7f2ea000dc90 comp rx=0 tx=0).stop 2023-10-30T12:20:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eab7fe640 1 -- 172.21.15.50:0/717447324 reap_dead start 2023-10-30T12:20:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eab7fe640 1 -- 172.21.15.50:0/717447324 reap_dead start 2023-10-30T12:20:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 shutdown_connections 2023-10-30T12:20:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 >> 172.21.15.50:0/717447324 conn(0x7f2eac0a80b0 msgr2=0x7f2eac09b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:18.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.957+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 shutdown_connections 2023-10-30T12:20:18.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.958+0000 7f2eb132c640 1 -- 172.21.15.50:0/717447324 wait complete. 2023-10-30T12:20:18.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:17.958+0000 7f2eb132c640 1 librados: shutdown 2023-10-30T12:20:18.007 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:20:18.008 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:20:18.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:20:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.121+0000 7eff0c19e640 10 monclient: get_monmap_and_config 2023-10-30T12:20:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.121+0000 7eff0c19e640 10 monclient: build_initial_monmap 2023-10-30T12:20:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.121+0000 7eff0c19e640 10 monclient: monmap: 2023-10-30T12:20:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:18.122157+0000 2023-10-30T12:20:18.123 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:18.122157+0000 2023-10-30T12:20:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:18.124 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:18.125 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:18.125 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:18.125 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.121+0000 7eff0c19e640 1 Processor -- start 2023-10-30T12:20:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.121+0000 7eff0c19e640 1 -- start start 2023-10-30T12:20:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.121+0000 7eff0c19e640 10 monclient: init 2023-10-30T12:20:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T12:20:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff040a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2450 0x7eff0413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient(hunting): picked mon.noname-b con 0x7eff040a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff0413af40 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040a4190 con 0x7eff0413af40 2023-10-30T12:20:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0409a2e0 con 0x7eff040a2ae0 2023-10-30T12:20:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff040a1630 con 0x7eff040a2450 2023-10-30T12:20:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0c19e640 10 monclient(hunting): authenticate will time out at 2063.305409s 2023-10-30T12:20:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2450 0x7eff0413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2450 0x7eff0413aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52380/0 (socket says 172.21.15.50:52380) 2023-10-30T12:20:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff0a73b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7eff0413af40 0x7eff0413d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48876/0 (socket says 172.21.15.50:48876) 2023-10-30T12:20:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 1 -- 172.21.15.50:0/2056192291 learned_addr learned my addr 172.21.15.50:0/2056192291 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): get_auth_request con 0x7eff040a2450 auth_method 0 2023-10-30T12:20:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff08f38640 1 -- 172.21.15.50:0/2056192291 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953891841 0 0) 0x7efefc001490 con 0x7eff0413af40 2023-10-30T12:20:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient(hunting): handle_auth_done global_id 254105 payload 274 2023-10-30T12:20:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 1 -- 172.21.15.50:0/2056192291 >> v1:172.21.15.118:6789/0 conn(0x7eff040a2ae0 legacy=0x7eff040a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 1 -- 172.21.15.50:0/2056192291 >> v1:172.21.15.50:6789/0 conn(0x7eff0413af40 legacy=0x7eff0413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient: _finish_hunting 0 2023-10-30T12:20:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 1 monclient: found mon.noname-b 2023-10-30T12:20:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:20:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff09739640 1 -- 172.21.15.50:0/2056192291 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0409a8a0 con 0x7eff040a2450 2023-10-30T12:20:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.122+0000 7eff08f38640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:20:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff09739640 1 --2- 172.21.15.50:0/2056192291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2450 0x7eff0413aa00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efef40098a0 tx=0x7efef402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f0bc2fabe2d9186 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 1 -- 172.21.15.50:0/2056192291 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4031640 con 0x7eff040a2450 2023-10-30T12:20:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:20:18.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: dump: 2023-10-30T12:20:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.143 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: _finish_auth 0 2023-10-30T12:20:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: _check_auth_tickets 2023-10-30T12:20:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 1 -- 172.21.15.50:0/2056192291 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef40317a0 con 0x7eff040a2450 2023-10-30T12:20:18.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff0c19e640 5 monclient: authenticate success, global_id 254105 2023-10-30T12:20:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff0c19e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff0c19e640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 1 -- 172.21.15.50:0/2056192291 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4031b30 con 0x7eff040a2450 2023-10-30T12:20:18.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff0c19e640 4 set_mon_vals no callback set 2023-10-30T12:20:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.123+0000 7eff08f38640 10 monclient: dump: 2023-10-30T12:20:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.145 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.147 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 10 monclient: shutdown 2023-10-30T12:20:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056192291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2450 msgr2=0x7eff0413aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 1 --2- 172.21.15.50:0/2056192291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2450 0x7eff0413aa00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efef40098a0 tx=0x7efef402b8f0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056192291 shutdown_connections 2023-10-30T12:20:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 1 --2- 172.21.15.50:0/2056192291 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2450 0x7eff0413aa00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056192291 >> 172.21.15.50:0/2056192291 conn(0x7eff040a80b0 msgr2=0x7eff040a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056192291 shutdown_connections 2023-10-30T12:20:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.124+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056192291 wait complete. 2023-10-30T12:20:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient: build_initial_monmap 2023-10-30T12:20:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient: monmap: 2023-10-30T12:20:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:18.151 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 librados: starting msgr at 2023-10-30T12:20:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 librados: starting objecter 2023-10-30T12:20:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 Processor -- start 2023-10-30T12:20:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 -- start start 2023-10-30T12:20:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 librados: setting wanted keys 2023-10-30T12:20:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 librados: calling monclient init 2023-10-30T12:20:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient: init 2023-10-30T12:20:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:20:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff040a2450 0x7eff041ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): picked mon.noname-a con 0x7eff040a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2ae0 0x7eff041cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): picked mon.noname-c con 0x7eff040a2ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff0413af40 0x7eff04136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): picked mon.noname-b con 0x7eff0413af40 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff04139660 con 0x7eff040a2450 2023-10-30T12:20:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff040a1790 con 0x7eff0413af40 2023-10-30T12:20:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0413a6e0 con 0x7eff040a2ae0 2023-10-30T12:20:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:18.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.125+0000 7eff0c19e640 10 monclient(hunting): authenticate will time out at 2063.308976s 2023-10-30T12:20:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff040a2450 0x7eff041ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2ae0 0x7eff041cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff040a2450 0x7eff041ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57966/0 (socket says 172.21.15.50:57966) 2023-10-30T12:20:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 1 -- 172.21.15.50:0/2056267162 learned_addr learned my addr 172.21.15.50:0/2056267162 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 10 monclient(hunting): get_auth_request con 0x7eff040a2450 auth_method 0 2023-10-30T12:20:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): get_auth_request con 0x7eff040a2ae0 auth_method 0 2023-10-30T12:20:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0a73b640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff0413af40 0x7eff04136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0a73b640 10 monclient(hunting): get_auth_request con 0x7eff0413af40 auth_method 0 2023-10-30T12:20:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0a73b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0a73b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0a73b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient(hunting): handle_auth_done global_id 254108 payload 824 2023-10-30T12:20:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff0413af40 msgr2=0x7eff04136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff0413af40 0x7eff04136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff040a2450 msgr2=0x7eff041ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff040a2450 0x7eff041ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient: _finish_hunting 0 2023-10-30T12:20:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 monclient: found mon.noname-c 2023-10-30T12:20:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 -- 172.21.15.50:0/2056267162 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef40094d0 con 0x7eff040a2ae0 2023-10-30T12:20:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09f3a640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff040a2450 0x7eff041ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:20:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff09739640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2ae0 0x7eff041cab90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efef4033000 tx=0x7efef40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 client.?.objecter ms_handle_connect 0x7eff040a2ae0 2023-10-30T12:20:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef403ac40 con 0x7eff040a2ae0 2023-10-30T12:20:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: dump: 2023-10-30T12:20:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.165 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: _finish_auth 0 2023-10-30T12:20:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:20:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef403ada0 con 0x7eff040a2ae0 2023-10-30T12:20:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 5 monclient: authenticate success, global_id 254108 2023-10-30T12:20:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 10 monclient: _renew_subs 2023-10-30T12:20:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0411e8b0 con 0x7eff040a2ae0 2023-10-30T12:20:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4044630 con 0x7eff040a2ae0 2023-10-30T12:20:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefa7fc640 10 monclient: dump: 2023-10-30T12:20:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.169 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7efefbfff640 4 set_mon_vals no callback set 2023-10-30T12:20:18.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 10 client.254108.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:18.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 10 monclient: _renew_subs 2023-10-30T12:20:18.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:18.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff04132510 con 0x7eff040a2ae0 2023-10-30T12:20:18.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.126+0000 7eff0c19e640 1 librados: init done 2023-10-30T12:20:18.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7efef402f050 con 0x7eff040a2ae0 2023-10-30T12:20:18.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff0c19e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:18.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff0c19e640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137320 0x7eff04137700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff0c19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff04073ff0 con 0x7eff04137320 2023-10-30T12:20:18.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7efefa7fc640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efee80392f0 0x7efee803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(628..628 src has 490..628) v4 ==== 6704+0+0 (secure 0 0 0) 0x7efef406d040 con 0x7eff040a2ae0 2023-10-30T12:20:18.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7efefa7fc640 10 client.254108.objecter ms_dispatch 0x7eff04000b90 osd_map(628..628 src has 490..628) v4 2023-10-30T12:20:18.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7efefa7fc640 3 client.254108.objecter handle_osd_map got epochs [628,628] > 0 2023-10-30T12:20:18.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7efefa7fc640 3 client.254108.objecter handle_osd_map decoding full epoch 628 2023-10-30T12:20:18.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7efefa7fc640 20 client.254108.objecter dump_active .. 0 homeless 2023-10-30T12:20:18.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff09f3a640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137320 0x7eff04137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff09f3a640 10 monclient: get_auth_request con 0x7eff04137320 auth_method 0 2023-10-30T12:20:18.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff09f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff09f3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff09f3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff0a73b640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efee80392f0 0x7efee803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.127+0000 7eff0a73b640 10 monclient: get_auth_request con 0x7efee80392f0 auth_method 0 2023-10-30T12:20:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7eff09f3a640 10 monclient(hunting): handle_auth_done global_id 204652 payload 824 2023-10-30T12:20:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7eff09f3a640 1 -- 172.21.15.50:0/2056267162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff0409a5c0 con 0x7eff04137320 2023-10-30T12:20:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 client.254108.objecter ms_handle_connect 0x7efee80392f0 2023-10-30T12:20:18.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7eff0a73b640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efee80392f0 0x7efee803b7b0 secure :-1 s=READY pgs=4306 cs=0 l=1 rev1=1 crypto rx=0x7efefc005fd0 tx=0x7efefc007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 client.254108.objecter ms_handle_connect 0x7eff04137320 2023-10-30T12:20:18.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 client.254108.objecter resend_mon_ops 2023-10-30T12:20:18.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7eff09f3a640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137320 0x7eff04137700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7eff0000fa30 tx=0x7eff0000fef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff000052e0 con 0x7eff04137320 2023-10-30T12:20:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 monclient: dump: 2023-10-30T12:20:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.179 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.179 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.179 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7eff00005440 con 0x7eff04137320 2023-10-30T12:20:18.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137320 msgr2=0x7eff04137700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137320 0x7eff04137700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7eff0000fa30 tx=0x7eff0000fef0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137320 msgr2=0x7eff04137700 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.128+0000 7efefa7fc640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137320 0x7eff04137700 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7eff0000fa30 tx=0x7eff0000fef0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff0c19e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff0c19e640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137c40 0x7eff041332b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff0c19e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0411ea40 con 0x7eff04137c40 2023-10-30T12:20:18.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137c40 0x7eff041332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient: get_auth_request con 0x7eff04137c40 auth_method 0 2023-10-30T12:20:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 10 monclient(hunting): handle_auth_done global_id 204655 payload 824 2023-10-30T12:20:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.135+0000 7eff09f3a640 1 -- 172.21.15.50:0/2056267162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7eff04099b40 con 0x7eff04137c40 2023-10-30T12:20:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff09f3a640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137c40 0x7eff041332b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff0000fa30 tx=0x7eff00001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 10 client.254108.objecter ms_handle_connect 0x7eff04137c40 2023-10-30T12:20:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 10 client.254108.objecter resend_mon_ops 2023-10-30T12:20:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00006be0 con 0x7eff04137c40 2023-10-30T12:20:18.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:18.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 10 monclient: dump: 2023-10-30T12:20:18.189 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.190 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.190 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.190 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.191 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.191 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.191 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.191 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.192 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.192 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7eff00006d40 con 0x7eff04137c40 2023-10-30T12:20:18.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:18.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:18.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137c40 msgr2=0x7eff041332b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137c40 0x7eff041332b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff0000fa30 tx=0x7eff00001120 comp rx=0 tx=0).stop 2023-10-30T12:20:18.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137c40 msgr2=0x7eff041332b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:18.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7efefa7fc640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff04137c40 0x7eff041332b0 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff0000fa30 tx=0x7eff00001120 comp rx=0 tx=0).stop 2023-10-30T12:20:18.194 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:20:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 204, 2023-10-30T12:20:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:20:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 5, 2023-10-30T12:20:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:18.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.216 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:18.216 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:18.216 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:18.216 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:18.216 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:20:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:20:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:20:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:18.226 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:18.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:18.226 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:18.226 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:18.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 10 librados: watch_flush enter 2023-10-30T12:20:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 10 librados: watch_flush exit 2023-10-30T12:20:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 20 client.254108.objecter shutdown clearing up homeless session... 2023-10-30T12:20:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 10 client.254108.objecter successfully canceled tick 2023-10-30T12:20:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efee80392f0 msgr2=0x7efee803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7efee80392f0 0x7efee803b7b0 secure :-1 s=READY pgs=4306 cs=0 l=1 rev1=1 crypto rx=0x7efefc005fd0 tx=0x7efefc007c90 comp rx=0 tx=0).stop 2023-10-30T12:20:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 10 monclient: shutdown 2023-10-30T12:20:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2ae0 msgr2=0x7eff041cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 --2- 172.21.15.50:0/2056267162 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7eff040a2ae0 0x7eff041cab90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efef4033000 tx=0x7efef40046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0a73b640 1 -- 172.21.15.50:0/2056267162 reap_dead start 2023-10-30T12:20:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0a73b640 1 -- 172.21.15.50:0/2056267162 reap_dead start 2023-10-30T12:20:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 shutdown_connections 2023-10-30T12:20:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 >> 172.21.15.50:0/2056267162 conn(0x7eff040a80b0 msgr2=0x7eff040a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 shutdown_connections 2023-10-30T12:20:18.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.136+0000 7eff0c19e640 1 -- 172.21.15.50:0/2056267162 wait complete. 2023-10-30T12:20:18.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.137+0000 7eff0c19e640 1 librados: shutdown 2023-10-30T12:20:18.232 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:20:18.232 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:20:18.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:20:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.336+0000 7fc6e0bb7640 10 monclient: get_monmap_and_config 2023-10-30T12:20:18.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.336+0000 7fc6e0bb7640 10 monclient: build_initial_monmap 2023-10-30T12:20:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.337+0000 7fc6e0bb7640 10 monclient: monmap: 2023-10-30T12:20:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:18.338054+0000 2023-10-30T12:20:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:18.338054+0000 2023-10-30T12:20:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:18.340 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:18.341 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:18.341 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.337+0000 7fc6e0bb7640 1 Processor -- start 2023-10-30T12:20:18.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.337+0000 7fc6e0bb7640 1 -- start start 2023-10-30T12:20:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.337+0000 7fc6e0bb7640 10 monclient: init 2023-10-30T12:20:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T12:20:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc6dc131b20 0x7fc6dc132270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient(hunting): picked mon.noname-a con 0x7fc6dc131b20 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6dc071bc0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6dc071fc0 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc09a9b0 con 0x7fc6dc071fc0 2023-10-30T12:20:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc131f00 con 0x7fc6dc071bc0 2023-10-30T12:20:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc098230 con 0x7fc6dc131b20 2023-10-30T12:20:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6e0bb7640 10 monclient(hunting): authenticate will time out at 2063.521210s 2023-10-30T12:20:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc6dc131b20 0x7fc6dc132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc6dc131b20 0x7fc6dc132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57976/0 (socket says 172.21.15.50:57976) 2023-10-30T12:20:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6dad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc6dc071fc0 0x7fc6dc097b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48880/0 (socket says 172.21.15.50:48880) 2023-10-30T12:20:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 1 -- 172.21.15.50:0/2984456063 learned_addr learned my addr 172.21.15.50:0/2984456063 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): get_auth_request con 0x7fc6dc131b20 auth_method 0 2023-10-30T12:20:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient(hunting): handle_auth_done global_id 244266 payload 274 2023-10-30T12:20:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 1 -- 172.21.15.50:0/2984456063 >> v1:172.21.15.118:6789/0 conn(0x7fc6dc071bc0 legacy=0x7fc6dc097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 1 -- 172.21.15.50:0/2984456063 >> v1:172.21.15.50:6789/0 conn(0x7fc6dc071fc0 legacy=0x7fc6dc097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient: _finish_hunting 0 2023-10-30T12:20:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 1 monclient: found mon.noname-a 2023-10-30T12:20:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.338+0000 7fc6da59c640 1 -- 172.21.15.50:0/2984456063 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc09d920 con 0x7fc6dc131b20 2023-10-30T12:20:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6da59c640 1 --2- 172.21.15.50:0/2984456063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc131b20 0x7fc6dc132270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc09db10 tx=0x7fc6d402cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa453876c3eb0a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 1 -- 172.21.15.50:0/2984456063 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4004230 con 0x7fc6dc131b20 2023-10-30T12:20:18.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: dump: 2023-10-30T12:20:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.356 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.358 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: _finish_auth 0 2023-10-30T12:20:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: _check_auth_tickets 2023-10-30T12:20:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 1 -- 172.21.15.50:0/2984456063 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4004390 con 0x7fc6dc131b20 2023-10-30T12:20:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 5 monclient: authenticate success, global_id 244266 2023-10-30T12:20:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 1 -- 172.21.15.50:0/2984456063 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d402db50 con 0x7fc6dc131b20 2023-10-30T12:20:18.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 4 set_mon_vals no callback set 2023-10-30T12:20:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6d959a640 10 monclient: dump: 2023-10-30T12:20:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.360 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.361 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.362 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 10 monclient: shutdown 2023-10-30T12:20:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/2984456063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc131b20 msgr2=0x7fc6dc132270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.339+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/2984456063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc131b20 0x7fc6dc132270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc09db10 tx=0x7fc6d402cda0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.340+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/2984456063 shutdown_connections 2023-10-30T12:20:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.340+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/2984456063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc131b20 0x7fc6dc132270 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.340+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/2984456063 >> 172.21.15.50:0/2984456063 conn(0x7fc6dc0a05c0 msgr2=0x7fc6dc09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:18.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.340+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/2984456063 shutdown_connections 2023-10-30T12:20:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.340+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/2984456063 wait complete. 2023-10-30T12:20:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.340+0000 7fc6e0bb7640 10 monclient: build_initial_monmap 2023-10-30T12:20:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.340+0000 7fc6e0bb7640 10 monclient: monmap: 2023-10-30T12:20:18.364 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.365 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:18.366 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 librados: starting msgr at 2023-10-30T12:20:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 librados: starting objecter 2023-10-30T12:20:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 Processor -- start 2023-10-30T12:20:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 -- start start 2023-10-30T12:20:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 librados: setting wanted keys 2023-10-30T12:20:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 librados: calling monclient init 2023-10-30T12:20:18.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient: init 2023-10-30T12:20:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:20:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc071bc0 0x7fc6dc1c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): picked mon.noname-a con 0x7fc6dc071bc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:18.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6dc071fc0 0x7fc6dc096640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): picked mon.noname-c con 0x7fc6dc071fc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6dc131b20 0x7fc6dc096b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): picked mon.noname-b con 0x7fc6dc131b20 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc06d4c0 con 0x7fc6dc071bc0 2023-10-30T12:20:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc06c150 con 0x7fc6dc131b20 2023-10-30T12:20:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1ca990 con 0x7fc6dc071fc0 2023-10-30T12:20:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6e0bb7640 10 monclient(hunting): authenticate will time out at 2063.524652s 2023-10-30T12:20:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc071bc0 0x7fc6dc1c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc071bc0 0x7fc6dc1c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57980/0 (socket says 172.21.15.50:57980) 2023-10-30T12:20:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 1 -- 172.21.15.50:0/1178592158 learned_addr learned my addr 172.21.15.50:0/1178592158 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): get_auth_request con 0x7fc6dc071bc0 auth_method 0 2023-10-30T12:20:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6dc071fc0 0x7fc6dc096640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 10 monclient(hunting): get_auth_request con 0x7fc6dc071fc0 auth_method 0 2023-10-30T12:20:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6dad9d640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6dc131b20 0x7fc6dc096b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6dad9d640 10 monclient(hunting): get_auth_request con 0x7fc6dc131b20 auth_method 0 2023-10-30T12:20:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6dad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6dad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.341+0000 7fc6da59c640 10 monclient(hunting): handle_auth_done global_id 244272 payload 824 2023-10-30T12:20:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 1 -- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6dc071fc0 msgr2=0x7fc6dc096640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6dc071fc0 0x7fc6dc096640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 1 -- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6dc131b20 msgr2=0x7fc6dc096b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6dc131b20 0x7fc6dc096b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 10 monclient: _finish_hunting 0 2023-10-30T12:20:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 1 monclient: found mon.noname-a 2023-10-30T12:20:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 1 -- 172.21.15.50:0/1178592158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d400a600 con 0x7fc6dc071bc0 2023-10-30T12:20:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6dad9d640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6dc131b20 0x7fc6dc096b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:20:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6da59c640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc071bc0 0x7fc6dc1c9f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000c00 tx=0x7fc6d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 client.?.objecter ms_handle_connect 0x7fc6dc071bc0 2023-10-30T12:20:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:18.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 1 -- 172.21.15.50:0/1178592158 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4003810 con 0x7fc6dc071bc0 2023-10-30T12:20:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: dump: 2023-10-30T12:20:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.381 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:20:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:20:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 1 -- 172.21.15.50:0/1178592158 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4003970 con 0x7fc6dc071bc0 2023-10-30T12:20:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 5 monclient: authenticate success, global_id 244272 2023-10-30T12:20:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 1 -- 172.21.15.50:0/1178592158 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4003d20 con 0x7fc6dc071bc0 2023-10-30T12:20:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 10 monclient: _renew_subs 2023-10-30T12:20:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc0993c0 con 0x7fc6dc071bc0 2023-10-30T12:20:18.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d8d99640 4 set_mon_vals no callback set 2023-10-30T12:20:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6d2ffd640 10 monclient: dump: 2023-10-30T12:20:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:18.385 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.385 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:18.385 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:18.386 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:18.386 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:18.386 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:18.386 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:18.387 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:18.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 10 client.244272.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:18.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 10 monclient: _renew_subs 2023-10-30T12:20:18.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:18.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc06da80 con 0x7fc6dc071bc0 2023-10-30T12:20:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.342+0000 7fc6e0bb7640 1 librados: init done 2023-10-30T12:20:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d2ffd640 1 -- 172.21.15.50:0/1178592158 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc6d4006040 con 0x7fc6dc071bc0 2023-10-30T12:20:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d2ffd640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6bc0392f0 0x7fc6bc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d2ffd640 1 -- 172.21.15.50:0/1178592158 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(628..628 src has 490..628) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fc6d406d770 con 0x7fc6dc071bc0 2023-10-30T12:20:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d2ffd640 10 client.244272.objecter ms_dispatch 0x7fc6dc000b90 osd_map(628..628 src has 490..628) v4 2023-10-30T12:20:18.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6e0bb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:18.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d2ffd640 3 client.244272.objecter handle_osd_map got epochs [628,628] > 0 2023-10-30T12:20:18.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d2ffd640 3 client.244272.objecter handle_osd_map decoding full epoch 628 2023-10-30T12:20:18.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6e0bb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:18.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6e0bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:18.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc04e160 con 0x7fc6dc071bc0 2023-10-30T12:20:18.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d2ffd640 20 client.244272.objecter dump_active .. 0 homeless 2023-10-30T12:20:18.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d9d9b640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6bc0392f0 0x7fc6bc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:18.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.343+0000 7fc6d9d9b640 10 monclient: get_auth_request con 0x7fc6bc0392f0 auth_method 0 2023-10-30T12:20:18.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.344+0000 7fc6d2ffd640 10 client.244272.objecter ms_handle_connect 0x7fc6bc0392f0 2023-10-30T12:20:18.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.344+0000 7fc6d9d9b640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6bc0392f0 0x7fc6bc03b7b0 secure :-1 s=READY pgs=4307 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc06d3f0 tx=0x7fc6c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:18.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.348+0000 7fc6d2ffd640 1 -- 172.21.15.50:0/1178592158 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc6d403e440 con 0x7fc6dc071bc0 2023-10-30T12:20:18.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.348+0000 7fc6d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:18.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.348+0000 7fc6d2ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:18.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.592+0000 7fc6e0bb7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:20:18.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.592+0000 7fc6e0bb7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:20:18.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.592+0000 7fc6e0bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:18.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.592+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc6dc0745c0 con 0x7fc6dc071bc0 2023-10-30T12:20:18.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.593+0000 7fc6d2ffd640 1 -- 172.21.15.50:0/1178592158 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc6d4043020 con 0x7fc6dc071bc0 2023-10-30T12:20:18.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.593+0000 7fc6d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:20:18.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.593+0000 7fc6d2ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:18.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.594+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.594+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=489907606,config=55734535,health=556162595,kv=1634934840,logm=1258165878}) 2023-10-30T12:20:18.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.594+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=4040173763,config=3140514009,health=556162595,kv=109951669,logm=1996476233}) 2023-10-30T12:20:18.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.594+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.594+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=489907606,config=55734535,health=556162595,kv=1634934840,logm=1258165878}) 2023-10-30T12:20:18.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.594+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=1170590282,config=3140514009,health=556162595,kv=3201645878,logm=3298488873}) 2023-10-30T12:20:18.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.595+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.595+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552698226}) 2023-10-30T12:20:18.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.595+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1857038530}) 2023-10-30T12:20:18.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.595+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.595+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552698226}) 2023-10-30T12:20:18.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.595+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419794499}) 2023-10-30T12:20:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 10 librados: watch_flush enter 2023-10-30T12:20:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 10 librados: watch_flush exit 2023-10-30T12:20:18.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 20 client.244272.objecter shutdown clearing up homeless session... 2023-10-30T12:20:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 10 client.244272.objecter successfully canceled tick 2023-10-30T12:20:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6bc0392f0 msgr2=0x7fc6bc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6bc0392f0 0x7fc6bc03b7b0 secure :-1 s=READY pgs=4307 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc06d3f0 tx=0x7fc6c4009040 comp rx=0 tx=0).stop 2023-10-30T12:20:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 10 monclient: shutdown 2023-10-30T12:20:18.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc071bc0 msgr2=0x7fc6dc1c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.595+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc071bc0 0x7fc6dc1c9f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000c00 tx=0x7fc6d40046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 shutdown_connections 2023-10-30T12:20:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc6bc0392f0 0x7fc6bc03b7b0 unknown :-1 s=CLOSED pgs=4307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc6dc131b20 0x7fc6dc096b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6dc071fc0 0x7fc6dc096640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 --2- 172.21.15.50:0/1178592158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6dc071bc0 0x7fc6dc1c9f30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 >> 172.21.15.50:0/1178592158 conn(0x7fc6dc0a05c0 msgr2=0x7fc6dc0710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 shutdown_connections 2023-10-30T12:20:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 -- 172.21.15.50:0/1178592158 wait complete. 2023-10-30T12:20:18.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:18.596+0000 7fc6e0bb7640 1 librados: shutdown 2023-10-30T12:20:18.608 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.597+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.608 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.597+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105797156}) 2023-10-30T12:20:18.608 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.597+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1996260736}) 2023-10-30T12:20:18.608 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.597+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.609 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.597+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105797156}) 2023-10-30T12:20:18.609 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.597+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746429896}) 2023-10-30T12:20:18.609 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.598+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.609 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.598+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684377579}) 2023-10-30T12:20:18.609 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.598+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2731001829}) 2023-10-30T12:20:18.610 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.598+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.610 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.598+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684377579}) 2023-10-30T12:20:18.610 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.598+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4217685157}) 2023-10-30T12:20:18.610 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.600+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.610 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.600+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365744309}) 2023-10-30T12:20:18.611 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.600+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353369292}) 2023-10-30T12:20:18.611 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.600+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.611 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.600+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365744309}) 2023-10-30T12:20:18.611 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.600+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260662990}) 2023-10-30T12:20:18.611 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.612 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769214731}) 2023-10-30T12:20:18.612 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896068768}) 2023-10-30T12:20:18.612 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.612 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769214731}) 2023-10-30T12:20:18.612 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3596442934}) 2023-10-30T12:20:18.612 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.613 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871368497}) 2023-10-30T12:20:18.613 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2597721836}) 2023-10-30T12:20:18.613 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.613 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871368497}) 2023-10-30T12:20:18.614 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.601+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288284846}) 2023-10-30T12:20:18.614 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.602+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.614 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.602+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601353815}) 2023-10-30T12:20:18.614 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.602+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546217082}) 2023-10-30T12:20:18.614 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.602+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.615 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.602+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601353815}) 2023-10-30T12:20:18.615 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.602+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3725205822}) 2023-10-30T12:20:18.615 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.603+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.615 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.603+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237290511}) 2023-10-30T12:20:18.616 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.603+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635441192}) 2023-10-30T12:20:18.616 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.603+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.616 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.603+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237290511}) 2023-10-30T12:20:18.616 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.603+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3092966282}) 2023-10-30T12:20:18.616 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.604+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.617 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.604+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809664020}) 2023-10-30T12:20:18.617 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.604+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3267261244}) 2023-10-30T12:20:18.617 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.604+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.617 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.604+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809664020}) 2023-10-30T12:20:18.617 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.604+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1510658999}) 2023-10-30T12:20:18.617 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.618 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=433012658}) 2023-10-30T12:20:18.618 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553295178}) 2023-10-30T12:20:18.618 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.618 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=433012658}) 2023-10-30T12:20:18.619 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=671325796}) 2023-10-30T12:20:18.619 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.619 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218620056}) 2023-10-30T12:20:18.619 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295742427}) 2023-10-30T12:20:18.619 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.620 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218620056}) 2023-10-30T12:20:18.620 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.605+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894445435}) 2023-10-30T12:20:18.620 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.606+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.620 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.606+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1205890340}) 2023-10-30T12:20:18.621 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.606+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398709074}) 2023-10-30T12:20:18.621 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.606+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.621 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.606+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1205890340}) 2023-10-30T12:20:18.621 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.606+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=485358988}) 2023-10-30T12:20:18.622 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:20:18.674 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.607+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.674 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.607+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185738901}) 2023-10-30T12:20:18.674 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.607+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1206160695}) 2023-10-30T12:20:18.675 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.607+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.675 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.607+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185738901}) 2023-10-30T12:20:18.675 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.607+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740553764}) 2023-10-30T12:20:18.675 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.608+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.675 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.608+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3417498552}) 2023-10-30T12:20:18.676 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.608+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195609321}) 2023-10-30T12:20:18.676 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.608+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.676 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.608+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3417498552}) 2023-10-30T12:20:18.676 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.608+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=540088551}) 2023-10-30T12:20:18.676 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.677 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804067734}) 2023-10-30T12:20:18.677 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2218474222}) 2023-10-30T12:20:18.677 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.677 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804067734}) 2023-10-30T12:20:18.677 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=698874464}) 2023-10-30T12:20:18.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473201631}) 2023-10-30T12:20:18.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659957806}) 2023-10-30T12:20:18.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473201631}) 2023-10-30T12:20:18.678 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.609+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395192169}) 2023-10-30T12:20:18.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.610+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.610+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014382568}) 2023-10-30T12:20:18.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.610+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2574074219}) 2023-10-30T12:20:18.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.610+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.679 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.610+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014382568}) 2023-10-30T12:20:18.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.610+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257219611}) 2023-10-30T12:20:18.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.611+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.611+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3182901644}) 2023-10-30T12:20:18.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.611+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751538298}) 2023-10-30T12:20:18.680 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.611+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.611+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3182901644}) 2023-10-30T12:20:18.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.611+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2627889983}) 2023-10-30T12:20:18.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617535274}) 2023-10-30T12:20:18.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2446132589}) 2023-10-30T12:20:18.681 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617535274}) 2023-10-30T12:20:18.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1497369461}) 2023-10-30T12:20:18.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.682 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12127337}) 2023-10-30T12:20:18.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=479366767}) 2023-10-30T12:20:18.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.683 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12127337}) 2023-10-30T12:20:18.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.612+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293134881}) 2023-10-30T12:20:18.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.613+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.684 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.613+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489756976}) 2023-10-30T12:20:18.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.613+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3317826278}) 2023-10-30T12:20:18.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.613+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.685 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.613+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489756976}) 2023-10-30T12:20:18.686 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.613+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251488825}) 2023-10-30T12:20:18.686 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.614+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.686 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.614+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635223018}) 2023-10-30T12:20:18.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.614+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4288513446}) 2023-10-30T12:20:18.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.614+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.687 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.614+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635223018}) 2023-10-30T12:20:18.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.614+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550518873}) 2023-10-30T12:20:18.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.615+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.688 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.615+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111582078}) 2023-10-30T12:20:18.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.615+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930525960}) 2023-10-30T12:20:18.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.615+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.689 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.615+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111582078}) 2023-10-30T12:20:18.690 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.615+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1390940318}) 2023-10-30T12:20:18.690 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.615+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2905055611}) 2023-10-30T12:20:18.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4121843045}) 2023-10-30T12:20:18.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2905055611}) 2023-10-30T12:20:18.691 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2111354212}) 2023-10-30T12:20:18.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767960138}) 2023-10-30T12:20:18.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1286343755}) 2023-10-30T12:20:18.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.692 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767960138}) 2023-10-30T12:20:18.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.616+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609078086}) 2023-10-30T12:20:18.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.617+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.617+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075418240}) 2023-10-30T12:20:18.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.617+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=161897714}) 2023-10-30T12:20:18.693 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.617+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.694 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.617+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075418240}) 2023-10-30T12:20:18.694 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.617+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1455890549}) 2023-10-30T12:20:18.694 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.618+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.694 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.618+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955115379}) 2023-10-30T12:20:18.695 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.618+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3580307653}) 2023-10-30T12:20:18.695 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.618+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.695 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.618+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955115379}) 2023-10-30T12:20:18.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.618+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708804330}) 2023-10-30T12:20:18.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.619+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.619+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2566062485}) 2023-10-30T12:20:18.696 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.619+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1301290564}) 2023-10-30T12:20:18.697 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.619+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.698 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.619+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2566062485}) 2023-10-30T12:20:18.698 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.619+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387325547}) 2023-10-30T12:20:18.699 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.699 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719853087}) 2023-10-30T12:20:18.699 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=756789949}) 2023-10-30T12:20:18.699 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.699 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719853087}) 2023-10-30T12:20:18.700 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172511437}) 2023-10-30T12:20:18.700 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.700 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187164688}) 2023-10-30T12:20:18.700 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1906067948}) 2023-10-30T12:20:18.700 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.701 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187164688}) 2023-10-30T12:20:18.701 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.620+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860078702}) 2023-10-30T12:20:18.701 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.621+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.701 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.621+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071397383}) 2023-10-30T12:20:18.701 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.621+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1667495078}) 2023-10-30T12:20:18.702 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.621+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.702 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.621+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071397383}) 2023-10-30T12:20:18.702 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.621+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2547356283}) 2023-10-30T12:20:18.702 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.622+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.622+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117496283}) 2023-10-30T12:20:18.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.622+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748166200}) 2023-10-30T12:20:18.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.622+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.622+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117496283}) 2023-10-30T12:20:18.704 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.622+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2038963476}) 2023-10-30T12:20:18.704 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.623+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.704 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.624+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161025274}) 2023-10-30T12:20:18.704 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.624+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2613652805}) 2023-10-30T12:20:18.704 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.624+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.704 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.624+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161025274}) 2023-10-30T12:20:18.705 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.624+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=637232183}) 2023-10-30T12:20:18.705 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.624+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.705 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.624+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198043401}) 2023-10-30T12:20:18.705 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907559869}) 2023-10-30T12:20:18.705 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.706 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198043401}) 2023-10-30T12:20:18.706 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4260277200}) 2023-10-30T12:20:18.706 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.706 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865932477}) 2023-10-30T12:20:18.706 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974029434}) 2023-10-30T12:20:18.707 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.707 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865932477}) 2023-10-30T12:20:18.707 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.625+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2526540320}) 2023-10-30T12:20:18.707 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.626+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.707 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.626+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246303740}) 2023-10-30T12:20:18.708 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.626+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285041039}) 2023-10-30T12:20:18.708 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.626+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.708 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.626+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246303740}) 2023-10-30T12:20:18.708 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.626+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3393981187}) 2023-10-30T12:20:18.708 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.627+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.708 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.627+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1302316130}) 2023-10-30T12:20:18.709 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.627+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591134434}) 2023-10-30T12:20:18.709 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.627+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.709 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.627+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1302316130}) 2023-10-30T12:20:18.709 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.627+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396591287}) 2023-10-30T12:20:18.709 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.709 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468387334}) 2023-10-30T12:20:18.710 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078523192}) 2023-10-30T12:20:18.710 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.710 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468387334}) 2023-10-30T12:20:18.710 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4088087959}) 2023-10-30T12:20:18.710 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754315766}) 2023-10-30T12:20:18.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334025923}) 2023-10-30T12:20:18.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754315766}) 2023-10-30T12:20:18.711 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.628+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061546726}) 2023-10-30T12:20:18.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.629+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.629+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805079005}) 2023-10-30T12:20:18.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.629+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548369583}) 2023-10-30T12:20:18.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.629+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.629+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805079005}) 2023-10-30T12:20:18.712 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.629+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4099369340}) 2023-10-30T12:20:18.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.630+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.630+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457727515}) 2023-10-30T12:20:18.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.630+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=866383025}) 2023-10-30T12:20:18.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.630+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.630+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457727515}) 2023-10-30T12:20:18.713 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.630+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636755660}) 2023-10-30T12:20:18.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265564666}) 2023-10-30T12:20:18.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2580471798}) 2023-10-30T12:20:18.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.714 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265564666}) 2023-10-30T12:20:18.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035050200}) 2023-10-30T12:20:18.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076945485}) 2023-10-30T12:20:18.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354132348}) 2023-10-30T12:20:18.715 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076945485}) 2023-10-30T12:20:18.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.631+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3640395154}) 2023-10-30T12:20:18.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.632+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.632+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2497064110}) 2023-10-30T12:20:18.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.632+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534835689}) 2023-10-30T12:20:18.716 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.632+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.632+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2497064110}) 2023-10-30T12:20:18.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.632+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660803546}) 2023-10-30T12:20:18.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.633+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.633+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835399913}) 2023-10-30T12:20:18.717 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.633+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57870534}) 2023-10-30T12:20:18.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.633+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.633+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835399913}) 2023-10-30T12:20:18.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.633+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950874516}) 2023-10-30T12:20:18.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.634+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.718 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.634+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408653379}) 2023-10-30T12:20:18.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.634+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545190670}) 2023-10-30T12:20:18.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.634+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.634+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408653379}) 2023-10-30T12:20:18.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.634+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259619661}) 2023-10-30T12:20:18.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.719 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59228055}) 2023-10-30T12:20:18.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092864734}) 2023-10-30T12:20:18.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=59228055}) 2023-10-30T12:20:18.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708481589}) 2023-10-30T12:20:18.720 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090319722}) 2023-10-30T12:20:18.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244307512}) 2023-10-30T12:20:18.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090319722}) 2023-10-30T12:20:18.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.635+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657267036}) 2023-10-30T12:20:18.721 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.636+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.636+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583380155}) 2023-10-30T12:20:18.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.636+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449775554}) 2023-10-30T12:20:18.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.636+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.636+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583380155}) 2023-10-30T12:20:18.722 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.636+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595767902}) 2023-10-30T12:20:18.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704394582}) 2023-10-30T12:20:18.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513634174}) 2023-10-30T12:20:18.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.723 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704394582}) 2023-10-30T12:20:18.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1105343092}) 2023-10-30T12:20:18.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100145834}) 2023-10-30T12:20:18.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1698497927}) 2023-10-30T12:20:18.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.724 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100145834}) 2023-10-30T12:20:18.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.637+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597977705}) 2023-10-30T12:20:18.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.638+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.638+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576419623}) 2023-10-30T12:20:18.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.638+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173964799}) 2023-10-30T12:20:18.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.638+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.725 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.638+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576419623}) 2023-10-30T12:20:18.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.638+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=420062246}) 2023-10-30T12:20:18.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.639+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.639+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397240281}) 2023-10-30T12:20:18.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.639+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3040596443}) 2023-10-30T12:20:18.726 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.639+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.639+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397240281}) 2023-10-30T12:20:18.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.639+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3902050611}) 2023-10-30T12:20:18.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.640+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.640+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888647533}) 2023-10-30T12:20:18.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.640+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1049056441}) 2023-10-30T12:20:18.727 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.640+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.640+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888647533}) 2023-10-30T12:20:18.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.640+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1263553954}) 2023-10-30T12:20:18.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1138504849}) 2023-10-30T12:20:18.728 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690696373}) 2023-10-30T12:20:18.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1138504849}) 2023-10-30T12:20:18.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=619132892}) 2023-10-30T12:20:18.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996361006}) 2023-10-30T12:20:18.729 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013486429}) 2023-10-30T12:20:18.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996361006}) 2023-10-30T12:20:18.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.641+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2175561664}) 2023-10-30T12:20:18.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.642+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.730 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.642+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914226141}) 2023-10-30T12:20:18.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.642+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2107404349}) 2023-10-30T12:20:18.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.642+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.642+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914226141}) 2023-10-30T12:20:18.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.642+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=458415615}) 2023-10-30T12:20:18.731 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035168073}) 2023-10-30T12:20:18.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1702201275}) 2023-10-30T12:20:18.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035168073}) 2023-10-30T12:20:18.732 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=355318456}) 2023-10-30T12:20:18.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.643+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136967613}) 2023-10-30T12:20:18.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=928717732}) 2023-10-30T12:20:18.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136967613}) 2023-10-30T12:20:18.733 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3116925805}) 2023-10-30T12:20:18.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055465221}) 2023-10-30T12:20:18.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3893374911}) 2023-10-30T12:20:18.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.734 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055465221}) 2023-10-30T12:20:18.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.644+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448148513}) 2023-10-30T12:20:18.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.645+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.645+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281298506}) 2023-10-30T12:20:18.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.645+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996212528}) 2023-10-30T12:20:18.735 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.645+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.645+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281298506}) 2023-10-30T12:20:18.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.645+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172127791}) 2023-10-30T12:20:18.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604194827}) 2023-10-30T12:20:18.736 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093003065}) 2023-10-30T12:20:18.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604194827}) 2023-10-30T12:20:18.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435134298}) 2023-10-30T12:20:18.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.737 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41259729}) 2023-10-30T12:20:18.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2635872231}) 2023-10-30T12:20:18.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41259729}) 2023-10-30T12:20:18.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.646+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186591080}) 2023-10-30T12:20:18.738 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.647+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.647+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201430245}) 2023-10-30T12:20:18.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.647+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2379836341}) 2023-10-30T12:20:18.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.647+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.647+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201430245}) 2023-10-30T12:20:18.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.647+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56517539}) 2023-10-30T12:20:18.739 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.648+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.648+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909914812}) 2023-10-30T12:20:18.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.648+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=196685410}) 2023-10-30T12:20:18.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.648+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.648+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909914812}) 2023-10-30T12:20:18.740 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.648+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604114840}) 2023-10-30T12:20:18.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987746092}) 2023-10-30T12:20:18.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=203518972}) 2023-10-30T12:20:18.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987746092}) 2023-10-30T12:20:18.741 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678029116}) 2023-10-30T12:20:18.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459319301}) 2023-10-30T12:20:18.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3185288395}) 2023-10-30T12:20:18.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.742 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459319301}) 2023-10-30T12:20:18.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.649+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903990950}) 2023-10-30T12:20:18.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.650+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.650+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549286273}) 2023-10-30T12:20:18.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.650+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922664238}) 2023-10-30T12:20:18.743 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.650+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.650+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549286273}) 2023-10-30T12:20:18.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.650+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765675881}) 2023-10-30T12:20:18.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225922840}) 2023-10-30T12:20:18.744 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4053905467}) 2023-10-30T12:20:18.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225922840}) 2023-10-30T12:20:18.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860188589}) 2023-10-30T12:20:18.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98903222}) 2023-10-30T12:20:18.745 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3963407260}) 2023-10-30T12:20:18.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98903222}) 2023-10-30T12:20:18.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.651+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=30574299}) 2023-10-30T12:20:18.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.652+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.746 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.652+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811642500}) 2023-10-30T12:20:18.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.652+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3889397687}) 2023-10-30T12:20:18.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.652+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.747 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.652+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811642500}) 2023-10-30T12:20:18.748 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.652+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122616945}) 2023-10-30T12:20:18.748 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.653+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.653+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276942385}) 2023-10-30T12:20:18.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.653+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2596063641}) 2023-10-30T12:20:18.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.653+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.749 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.653+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276942385}) 2023-10-30T12:20:18.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.653+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2092579227}) 2023-10-30T12:20:18.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.750 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79363509}) 2023-10-30T12:20:18.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2340475973}) 2023-10-30T12:20:18.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79363509}) 2023-10-30T12:20:18.751 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4152234376}) 2023-10-30T12:20:18.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696567412}) 2023-10-30T12:20:18.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163077292}) 2023-10-30T12:20:18.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696567412}) 2023-10-30T12:20:18.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.654+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3471734179}) 2023-10-30T12:20:18.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.655+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.655+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723738283}) 2023-10-30T12:20:18.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.655+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457919796}) 2023-10-30T12:20:18.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.655+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.655+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723738283}) 2023-10-30T12:20:18.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.655+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782264652}) 2023-10-30T12:20:18.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.656+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.656+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521358890}) 2023-10-30T12:20:18.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.656+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3202692785}) 2023-10-30T12:20:18.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.656+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.656+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521358890}) 2023-10-30T12:20:18.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.656+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1197605084}) 2023-10-30T12:20:18.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587537106}) 2023-10-30T12:20:18.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3699516630}) 2023-10-30T12:20:18.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.758 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587537106}) 2023-10-30T12:20:18.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2878063990}) 2023-10-30T12:20:18.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.759 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.657+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793332633}) 2023-10-30T12:20:18.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3333244772}) 2023-10-30T12:20:18.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793332633}) 2023-10-30T12:20:18.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003754129}) 2023-10-30T12:20:18.760 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332131610}) 2023-10-30T12:20:18.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572175787}) 2023-10-30T12:20:18.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332131610}) 2023-10-30T12:20:18.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.658+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429970713}) 2023-10-30T12:20:18.761 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.659+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.659+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3282100460}) 2023-10-30T12:20:18.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.659+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701896185}) 2023-10-30T12:20:18.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.659+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.659+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3282100460}) 2023-10-30T12:20:18.762 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.659+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3720963190}) 2023-10-30T12:20:18.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.660+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.660+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327023858}) 2023-10-30T12:20:18.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.660+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=951936162}) 2023-10-30T12:20:18.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.660+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.763 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.660+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327023858}) 2023-10-30T12:20:18.764 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.660+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090273464}) 2023-10-30T12:20:18.764 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.661+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.764 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.661+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932092861}) 2023-10-30T12:20:18.764 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.661+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3139654759}) 2023-10-30T12:20:18.764 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.661+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.764 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.661+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932092861}) 2023-10-30T12:20:18.765 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.661+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062206679}) 2023-10-30T12:20:18.765 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.765 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159884031}) 2023-10-30T12:20:18.765 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4085629621}) 2023-10-30T12:20:18.765 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.766 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159884031}) 2023-10-30T12:20:18.766 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789898832}) 2023-10-30T12:20:18.766 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.766 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975779769}) 2023-10-30T12:20:18.766 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802479091}) 2023-10-30T12:20:18.767 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.767 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975779769}) 2023-10-30T12:20:18.767 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.662+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1719332810}) 2023-10-30T12:20:18.767 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.663+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.767 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.663+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038029168}) 2023-10-30T12:20:18.768 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.663+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346072137}) 2023-10-30T12:20:18.768 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.663+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.768 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.663+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038029168}) 2023-10-30T12:20:18.768 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.663+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828721295}) 2023-10-30T12:20:18.768 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.664+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.664+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349446624}) 2023-10-30T12:20:18.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.664+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078305543}) 2023-10-30T12:20:18.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.664+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.664+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349446624}) 2023-10-30T12:20:18.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.664+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653433618}) 2023-10-30T12:20:18.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.665+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.665+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234410941}) 2023-10-30T12:20:18.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.665+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3806234728}) 2023-10-30T12:20:18.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.665+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.665+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234410941}) 2023-10-30T12:20:18.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.665+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169260063}) 2023-10-30T12:20:18.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797831526}) 2023-10-30T12:20:18.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072517647}) 2023-10-30T12:20:18.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797831526}) 2023-10-30T12:20:18.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972853863}) 2023-10-30T12:20:18.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088211034}) 2023-10-30T12:20:18.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1378396880}) 2023-10-30T12:20:18.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088211034}) 2023-10-30T12:20:18.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.666+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3390304803}) 2023-10-30T12:20:18.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.667+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.667+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304207569}) 2023-10-30T12:20:18.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.667+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=944557613}) 2023-10-30T12:20:18.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.667+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.667+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304207569}) 2023-10-30T12:20:18.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.667+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=322041830}) 2023-10-30T12:20:18.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.668+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.668+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482982288}) 2023-10-30T12:20:18.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.668+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1321679104}) 2023-10-30T12:20:18.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.668+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.668+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482982288}) 2023-10-30T12:20:18.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.668+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952433470}) 2023-10-30T12:20:18.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.669+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.669+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365452577}) 2023-10-30T12:20:18.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.669+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3353963877}) 2023-10-30T12:20:18.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.669+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.669+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365452577}) 2023-10-30T12:20:18.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.669+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459088485}) 2023-10-30T12:20:18.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864749588}) 2023-10-30T12:20:18.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904999315}) 2023-10-30T12:20:18.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864749588}) 2023-10-30T12:20:18.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782009747}) 2023-10-30T12:20:18.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672161737}) 2023-10-30T12:20:18.778 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=316505483}) 2023-10-30T12:20:18.778 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.778 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672161737}) 2023-10-30T12:20:18.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.670+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429879076}) 2023-10-30T12:20:18.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.671+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.671+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005924943}) 2023-10-30T12:20:18.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.671+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3206543800}) 2023-10-30T12:20:18.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.671+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.671+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005924943}) 2023-10-30T12:20:18.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.671+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679619368}) 2023-10-30T12:20:18.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.672+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.672+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658755676}) 2023-10-30T12:20:18.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.672+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=116085108}) 2023-10-30T12:20:18.781 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.672+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.781 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.672+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658755676}) 2023-10-30T12:20:18.781 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.672+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3894049352}) 2023-10-30T12:20:18.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.673+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.673+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349766747}) 2023-10-30T12:20:18.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.673+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747709997}) 2023-10-30T12:20:18.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.673+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.673+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349766747}) 2023-10-30T12:20:18.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.673+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1962068992}) 2023-10-30T12:20:18.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.674+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.674+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800884596}) 2023-10-30T12:20:18.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.674+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3143274503}) 2023-10-30T12:20:18.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.674+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.674+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800884596}) 2023-10-30T12:20:18.794 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.674+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864864019}) 2023-10-30T12:20:18.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1657638264}) 2023-10-30T12:20:18.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=559460266}) 2023-10-30T12:20:18.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1657638264}) 2023-10-30T12:20:18.795 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1105516063}) 2023-10-30T12:20:18.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364240408}) 2023-10-30T12:20:18.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=261099232}) 2023-10-30T12:20:18.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.796 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364240408}) 2023-10-30T12:20:18.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.675+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1049262806}) 2023-10-30T12:20:18.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.676+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.676+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2829103428}) 2023-10-30T12:20:18.797 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.676+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1708666247}) 2023-10-30T12:20:18.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.676+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.676+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2829103428}) 2023-10-30T12:20:18.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.676+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640943675}) 2023-10-30T12:20:18.798 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.677+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.677+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604525142}) 2023-10-30T12:20:18.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.677+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417782637}) 2023-10-30T12:20:18.799 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.677+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.800 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.677+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604525142}) 2023-10-30T12:20:18.800 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.677+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3652656686}) 2023-10-30T12:20:18.801 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.801 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648350980}) 2023-10-30T12:20:18.801 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3442180171}) 2023-10-30T12:20:18.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648350980}) 2023-10-30T12:20:18.802 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1731663290}) 2023-10-30T12:20:18.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516545428}) 2023-10-30T12:20:18.803 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3439134968}) 2023-10-30T12:20:18.804 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.804 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516545428}) 2023-10-30T12:20:18.804 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.678+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234014613}) 2023-10-30T12:20:18.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.679+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.679+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2361751477}) 2023-10-30T12:20:18.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.679+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=765606481}) 2023-10-30T12:20:18.805 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.679+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.806 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.679+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2361751477}) 2023-10-30T12:20:18.806 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.679+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569346089}) 2023-10-30T12:20:18.806 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.680+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.680+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711158087}) 2023-10-30T12:20:18.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.680+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=966653865}) 2023-10-30T12:20:18.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.680+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.680+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711158087}) 2023-10-30T12:20:18.807 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.680+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931606385}) 2023-10-30T12:20:18.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955421799}) 2023-10-30T12:20:18.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234729334}) 2023-10-30T12:20:18.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.808 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955421799}) 2023-10-30T12:20:18.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1914812313}) 2023-10-30T12:20:18.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914817774}) 2023-10-30T12:20:18.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.681+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594131731}) 2023-10-30T12:20:18.809 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914817774}) 2023-10-30T12:20:18.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076242387}) 2023-10-30T12:20:18.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390282397}) 2023-10-30T12:20:18.810 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874341749}) 2023-10-30T12:20:18.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390282397}) 2023-10-30T12:20:18.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.682+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697782514}) 2023-10-30T12:20:18.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.683+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.683+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206260795}) 2023-10-30T12:20:18.811 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.683+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1712902248}) 2023-10-30T12:20:18.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.683+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.683+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206260795}) 2023-10-30T12:20:18.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.683+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2164113280}) 2023-10-30T12:20:18.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.684+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.684+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481913141}) 2023-10-30T12:20:18.812 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.684+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2054560554}) 2023-10-30T12:20:18.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.684+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.684+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481913141}) 2023-10-30T12:20:18.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.684+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731341426}) 2023-10-30T12:20:18.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.813 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272425530}) 2023-10-30T12:20:18.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1756803545}) 2023-10-30T12:20:18.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272425530}) 2023-10-30T12:20:18.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890301283}) 2023-10-30T12:20:18.814 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386352522}) 2023-10-30T12:20:18.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3160814345}) 2023-10-30T12:20:18.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.815 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386352522}) 2023-10-30T12:20:18.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.685+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=806736238}) 2023-10-30T12:20:18.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.686+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.686+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443772490}) 2023-10-30T12:20:18.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.686+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173845758}) 2023-10-30T12:20:18.816 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.686+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.686+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443772490}) 2023-10-30T12:20:18.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.686+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742370559}) 2023-10-30T12:20:18.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.687+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.687+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491281614}) 2023-10-30T12:20:18.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.687+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198419683}) 2023-10-30T12:20:18.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.687+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.687+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491281614}) 2023-10-30T12:20:18.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.687+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825695071}) 2023-10-30T12:20:18.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.818 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438760984}) 2023-10-30T12:20:18.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818388337}) 2023-10-30T12:20:18.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438760984}) 2023-10-30T12:20:18.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1960044451}) 2023-10-30T12:20:18.819 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017074014}) 2023-10-30T12:20:18.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654476443}) 2023-10-30T12:20:18.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017074014}) 2023-10-30T12:20:18.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.688+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1424636648}) 2023-10-30T12:20:18.820 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.689+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.821 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.689+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=41,osdmap=6} crc {logm=3063900546,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3783691950,mon_config_key=3217606506,monmap=3324057927,osd_metadata=3009730626,osd_pg_creating=2739455718,osd_snap=4091493920,osdmap=1959544754}) 2023-10-30T12:20:18.821 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.689+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=3} crc {logm=1307715178,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3783691950,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=2739455718,osd_snap=3866942305,osdmap=3638547978}) 2023-10-30T12:20:18.821 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.690+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.821 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.690+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=41,osdmap=6} crc {logm=3063900546,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3783691950,mon_config_key=3217606506,monmap=3324057927,osd_metadata=3009730626,osd_pg_creating=2739455718,osd_snap=4091493920,osdmap=1959544754}) 2023-10-30T12:20:18.822 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.690+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=15,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=7} crc {logm=1307715178,mdsmap=3984425268,mgr=1715906815,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3783691950,mon_config_key=3217606506,monmap=365899282,osd_metadata=2479346626,osd_pg_creating=2739455718,osd_snap=799199680,osdmap=2965646512}) 2023-10-30T12:20:18.822 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.691+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.822 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.691+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1001423101}) 2023-10-30T12:20:18.822 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.691+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=261601048}) 2023-10-30T12:20:18.823 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.691+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.823 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.691+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1001423101}) 2023-10-30T12:20:18.823 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.691+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3883856764}) 2023-10-30T12:20:18.823 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.693+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.823 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.693+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3432863913}) 2023-10-30T12:20:18.824 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.693+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1270698450}) 2023-10-30T12:20:18.824 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.693+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.824 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.693+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3432863913}) 2023-10-30T12:20:18.824 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.693+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2954210910}) 2023-10-30T12:20:18.824 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.697+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.824 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.697+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=9} crc {osdmap=1294104127}) 2023-10-30T12:20:18.825 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.697+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=3256317225}) 2023-10-30T12:20:18.825 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.697+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:20:18.825 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.697+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=9} crc {osdmap=1294104127}) 2023-10-30T12:20:18.825 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:20:18.697+0000 7f0071b6e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=10} crc {osdmap=3256317225}) 2023-10-30T12:20:19.624 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:20:19.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:19.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.735+0000 7fbf025a0640 10 monclient: get_monmap_and_config 2023-10-30T12:20:19.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.735+0000 7fbf025a0640 10 monclient: build_initial_monmap 2023-10-30T12:20:19.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.735+0000 7fbf025a0640 10 monclient: monmap: 2023-10-30T12:20:19.737 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:19.738 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:19.738 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:19.736490+0000 2023-10-30T12:20:19.738 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:19.736490+0000 2023-10-30T12:20:19.738 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:19.738 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:19.739 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:19.739 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:19.739 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:19.739 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:19.739 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:19.739 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:19.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 1 Processor -- start 2023-10-30T12:20:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 1 -- start start 2023-10-30T12:20:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient: init 2023-10-30T12:20:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-10-30T12:20:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbefc1312b0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbefc131690 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbefc131ae0 0x7fbefc13dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient(hunting): picked mon.noname-a con 0x7fbefc131ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc06bda0 con 0x7fbefc131690 2023-10-30T12:20:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc0a3c20 con 0x7fbefc1312b0 2023-10-30T12:20:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbefc06a330 con 0x7fbefc131ae0 2023-10-30T12:20:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf025a0640 10 monclient(hunting): authenticate will time out at 2064.919883s 2023-10-30T12:20:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.736+0000 7fbf01d9f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbefc131ae0 0x7fbefc13dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbefc131ae0 0x7fbefc13dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57984/0 (socket says 172.21.15.50:57984) 2023-10-30T12:20:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 1 -- 172.21.15.50:0/2835971251 learned_addr learned my addr 172.21.15.50:0/2835971251 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): get_auth_request con 0x7fbefc131ae0 auth_method 0 2023-10-30T12:20:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient(hunting): handle_auth_done global_id 244275 payload 274 2023-10-30T12:20:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 1 -- 172.21.15.50:0/2835971251 >> v1:172.21.15.118:6789/0 conn(0x7fbefc1312b0 legacy=0x7fbefc06b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 1 -- 172.21.15.50:0/2835971251 >> v1:172.21.15.50:6789/0 conn(0x7fbefc131690 legacy=0x7fbefc13a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient: _finish_hunting 0 2023-10-30T12:20:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 1 monclient: found mon.noname-a 2023-10-30T12:20:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.737+0000 7fbf01d9f640 1 -- 172.21.15.50:0/2835971251 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc06a610 con 0x7fbefc131ae0 2023-10-30T12:20:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf01d9f640 1 --2- 172.21.15.50:0/2835971251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc131ae0 0x7fbefc13dfa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009e50 tx=0x7fbef402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863ef4b0056b201f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 1 -- 172.21.15.50:0/2835971251 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef402ca80 con 0x7fbefc131ae0 2023-10-30T12:20:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: dump: 2023-10-30T12:20:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.756 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.756 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:19.756 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:19.756 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:19.756 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:19.757 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:19.757 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:19.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: _finish_auth 0 2023-10-30T12:20:19.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:20:19.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:19.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 5 monclient: authenticate success, global_id 244275 2023-10-30T12:20:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 1 -- 172.21.15.50:0/2835971251 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef402cbe0 con 0x7fbefc131ae0 2023-10-30T12:20:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 4 set_mon_vals no callback set 2023-10-30T12:20:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 1 -- 172.21.15.50:0/2835971251 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef40343f0 con 0x7fbefc131ae0 2023-10-30T12:20:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbef3fff640 10 monclient: dump: 2023-10-30T12:20:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:19.761 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 10 monclient: shutdown 2023-10-30T12:20:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 1 -- 172.21.15.50:0/2835971251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc131ae0 msgr2=0x7fbefc13dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 1 --2- 172.21.15.50:0/2835971251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc131ae0 0x7fbefc13dfa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009e50 tx=0x7fbef402bee0 comp rx=0 tx=0).stop 2023-10-30T12:20:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 1 -- 172.21.15.50:0/2835971251 shutdown_connections 2023-10-30T12:20:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 1 --2- 172.21.15.50:0/2835971251 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc131ae0 0x7fbefc13dfa0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 1 -- 172.21.15.50:0/2835971251 >> 172.21.15.50:0/2835971251 conn(0x7fbefc0a4200 msgr2=0x7fbefc0a4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:19.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 1 -- 172.21.15.50:0/2835971251 shutdown_connections 2023-10-30T12:20:19.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.738+0000 7fbf025a0640 1 -- 172.21.15.50:0/2835971251 wait complete. 2023-10-30T12:20:19.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 10 monclient: build_initial_monmap 2023-10-30T12:20:19.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 10 monclient: monmap: 2023-10-30T12:20:19.763 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:19.763 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:19.764 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:19.764 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:19.764 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:19.764 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:19.764 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:19.765 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:19.765 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:19.765 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:19.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 1 librados: starting msgr at 2023-10-30T12:20:19.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 1 librados: starting objecter 2023-10-30T12:20:19.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 1 Processor -- start 2023-10-30T12:20:19.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 1 -- start start 2023-10-30T12:20:19.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 1 librados: setting wanted keys 2023-10-30T12:20:19.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 1 librados: calling monclient init 2023-10-30T12:20:19.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.739+0000 7fbf025a0640 10 monclient: init 2023-10-30T12:20:19.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:19.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:20:19.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc1312b0 0x7fbefc09dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:19.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): picked mon.noname-a con 0x7fbefc1312b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:19.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbefc131690 0x7fbefc133120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:19.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): picked mon.noname-c con 0x7fbefc131690 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:19.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbefc131ae0 0x7fbefc133660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:19.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): picked mon.noname-b con 0x7fbefc131ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:19.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:19.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc0a38f0 con 0x7fbefc1312b0 2023-10-30T12:20:19.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc134f10 con 0x7fbefc131ae0 2023-10-30T12:20:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc0a0ce0 con 0x7fbefc131690 2023-10-30T12:20:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf025a0640 10 monclient(hunting): authenticate will time out at 2064.923270s 2023-10-30T12:20:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc1312b0 0x7fbefc09dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc1312b0 0x7fbefc09dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57990/0 (socket says 172.21.15.50:57990) 2023-10-30T12:20:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 -- 172.21.15.50:0/1199303458 learned_addr learned my addr 172.21.15.50:0/1199303458 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): get_auth_request con 0x7fbefc1312b0 auth_method 0 2023-10-30T12:20:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:19.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbefc131690 0x7fbefc133120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:19.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:19.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:19.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 10 monclient(hunting): get_auth_request con 0x7fbefc131690 auth_method 0 2023-10-30T12:20:19.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:19.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:19.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:19.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:19.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:19.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:19.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:19.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:19.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:19.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf01d9f640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbefc131ae0 0x7fbefc133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient(hunting): handle_auth_done global_id 244281 payload 824 2023-10-30T12:20:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 -- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbefc131690 msgr2=0x7fbefc133120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbefc131690 0x7fbefc133120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:19.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 -- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbefc131ae0 msgr2=0x7fbefc133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:19.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbefc131ae0 0x7fbefc133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:19.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient: _finish_hunting 0 2023-10-30T12:20:19.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 monclient: found mon.noname-a 2023-10-30T12:20:19.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:19.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:19.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 -- 172.21.15.50:0/1199303458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef4009b30 con 0x7fbefc1312b0 2023-10-30T12:20:19.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf01d9f640 10 monclient: get_auth_request con 0x7fbefc131ae0 auth_method 0 2023-10-30T12:20:19.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf01d9f640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbefc131ae0 0x7fbefc133660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:20:19.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf00d9d640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbefc131690 0x7fbefc133120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:20:19.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbf0159e640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc1312b0 0x7fbefc09dc60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbef8004640 tx=0x7fbef800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:19.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbef1ffb640 10 client.?.objecter ms_handle_connect 0x7fbefc1312b0 2023-10-30T12:20:19.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.740+0000 7fbef1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:19.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 1 -- 172.21.15.50:0/1199303458 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8011790 con 0x7fbefc1312b0 2023-10-30T12:20:19.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:19.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: dump: 2023-10-30T12:20:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:19.783 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:19.783 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:19.783 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:19.783 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:19.783 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:19.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:20:19.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:20:19.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:19.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 1 -- 172.21.15.50:0/1199303458 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef80118f0 con 0x7fbefc1312b0 2023-10-30T12:20:19.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 5 monclient: authenticate success, global_id 244281 2023-10-30T12:20:19.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 10 monclient: _renew_subs 2023-10-30T12:20:19.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:19.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbefc12dc00 con 0x7fbefc1312b0 2023-10-30T12:20:19.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:19.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 1 -- 172.21.15.50:0/1199303458 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8011ce0 con 0x7fbefc1312b0 2023-10-30T12:20:19.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:19.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:19.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef1ffb640 10 monclient: dump: 2023-10-30T12:20:19.786 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:19.786 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:19.787 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.787 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:19.787 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:19.787 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:19.787 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:19.787 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:19.788 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:19.788 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:19.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbef37fe640 4 set_mon_vals no callback set 2023-10-30T12:20:19.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 10 client.244281.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:19.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 10 monclient: _renew_subs 2023-10-30T12:20:19.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:19.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbefc134810 con 0x7fbefc1312b0 2023-10-30T12:20:19.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.741+0000 7fbf025a0640 1 librados: init done 2023-10-30T12:20:19.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbef1ffb640 1 -- 172.21.15.50:0/1199303458 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fbef801c460 con 0x7fbefc1312b0 2023-10-30T12:20:19.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbef1ffb640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbee0039320 0x7fbee003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:19.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbef1ffb640 1 -- 172.21.15.50:0/1199303458 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(629..629 src has 490..629) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fbef804ac10 con 0x7fbefc1312b0 2023-10-30T12:20:19.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbef1ffb640 10 client.244281.objecter ms_dispatch 0x7fbefc000b90 osd_map(629..629 src has 490..629) v4 2023-10-30T12:20:19.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbef1ffb640 3 client.244281.objecter handle_osd_map got epochs [629,629] > 0 2023-10-30T12:20:19.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbef1ffb640 3 client.244281.objecter handle_osd_map decoding full epoch 629 2023-10-30T12:20:19.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbf025a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:19.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbf025a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:19.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbf025a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:19.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbef1ffb640 20 client.244281.objecter dump_active .. 0 homeless 2023-10-30T12:20:19.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbefc04e160 con 0x7fbefc1312b0 2023-10-30T12:20:19.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbf00d9d640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbee0039320 0x7fbee003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:19.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.742+0000 7fbf00d9d640 10 monclient: get_auth_request con 0x7fbee0039320 auth_method 0 2023-10-30T12:20:19.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.743+0000 7fbf00d9d640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbee0039320 0x7fbee003b7e0 secure :-1 s=READY pgs=4308 cs=0 l=1 rev1=1 crypto rx=0x7fbefc136430 tx=0x7fbeec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:19.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.743+0000 7fbef1ffb640 10 client.244281.objecter ms_handle_connect 0x7fbee0039320 2023-10-30T12:20:19.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.747+0000 7fbef1ffb640 1 -- 172.21.15.50:0/1199303458 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbef80104c0 con 0x7fbefc1312b0 2023-10-30T12:20:19.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.747+0000 7fbef1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:19.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.747+0000 7fbef1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:19.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.996+0000 7fbf025a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.996+0000 7fbf025a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.996+0000 7fbf025a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.996+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbefc074390 con 0x7fbefc1312b0 2023-10-30T12:20:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.996+0000 7fbef1ffb640 1 -- 172.21.15.50:0/1199303458 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fbef801a360 con 0x7fbefc1312b0 2023-10-30T12:20:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.996+0000 7fbef1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:20.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.996+0000 7fbef1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:20.000 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:20.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.998+0000 7fbf025a0640 10 librados: watch_flush enter 2023-10-30T12:20:20.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.998+0000 7fbf025a0640 10 librados: watch_flush exit 2023-10-30T12:20:20.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.998+0000 7fbf025a0640 20 client.244281.objecter shutdown clearing up homeless session... 2023-10-30T12:20:20.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.998+0000 7fbf025a0640 10 client.244281.objecter successfully canceled tick 2023-10-30T12:20:20.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbee0039320 msgr2=0x7fbee003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbee0039320 0x7fbee003b7e0 secure :-1 s=READY pgs=4308 cs=0 l=1 rev1=1 crypto rx=0x7fbefc136430 tx=0x7fbeec009040 comp rx=0 tx=0).stop 2023-10-30T12:20:20.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 10 monclient: shutdown 2023-10-30T12:20:20.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc1312b0 msgr2=0x7fbefc09dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc1312b0 0x7fbefc09dc60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbef8004640 tx=0x7fbef800c8e0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 shutdown_connections 2023-10-30T12:20:20.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fbee0039320 0x7fbee003b7e0 unknown :-1 s=CLOSED pgs=4308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbefc131ae0 0x7fbefc133660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbefc131690 0x7fbefc133120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 --2- 172.21.15.50:0/1199303458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbefc1312b0 0x7fbefc09dc60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 >> 172.21.15.50:0/1199303458 conn(0x7fbefc0a4200 msgr2=0x7fbefc0a6150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 shutdown_connections 2023-10-30T12:20:20.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 -- 172.21.15.50:0/1199303458 wait complete. 2023-10-30T12:20:20.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:19.999+0000 7fbf025a0640 1 librados: shutdown 2023-10-30T12:20:20.010 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:20:20.011 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:20:20.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:20:20.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:20:20.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.116+0000 7fe548b00640 10 monclient: get_monmap_and_config 2023-10-30T12:20:20.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.116+0000 7fe548b00640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.116+0000 7fe548b00640 10 monclient: monmap: 2023-10-30T12:20:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.118 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:20.117133+0000 2023-10-30T12:20:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:20.117133+0000 2023-10-30T12:20:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.119 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:20.120 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:20.120 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.116+0000 7fe548b00640 1 Processor -- start 2023-10-30T12:20:20.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.116+0000 7fe548b00640 1 -- start start 2023-10-30T12:20:20.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.116+0000 7fe548b00640 10 monclient: init 2023-10-30T12:20:20.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-10-30T12:20:20.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5440a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:20.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fe5440a2450 0x7fe54413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient(hunting): picked mon.noname-c con 0x7fe5440a2450 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:20.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe54413af40 0x7fe54413d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient(hunting): picked mon.noname-a con 0x7fe54413af40 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:20.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5440a4190 con 0x7fe5440a2ae0 2023-10-30T12:20:20.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5440a1630 con 0x7fe54413af40 2023-10-30T12:20:20.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54409a2e0 con 0x7fe5440a2450 2023-10-30T12:20:20.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe548b00640 10 monclient(hunting): authenticate will time out at 2065.300319s 2023-10-30T12:20:20.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe54413af40 0x7fe54413d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe54413af40 0x7fe54413d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58004/0 (socket says 172.21.15.50:58004) 2023-10-30T12:20:20.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 -- 172.21.15.50:0/1832709885 learned_addr learned my addr 172.21.15.50:0/1832709885 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): get_auth_request con 0x7fe54413af40 auth_method 0 2023-10-30T12:20:20.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe541d9b640 1 --2- 172.21.15.50:0/1832709885 >> v2:172.21.15.118:3300/0 conn(0x7fe5440a2450 0x7fe54413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe541d9b640 10 monclient(hunting): get_auth_request con 0x7fe5440a2450 auth_method 0 2023-10-30T12:20:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe541d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe541d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe541d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient(hunting): handle_auth_done global_id 244284 payload 274 2023-10-30T12:20:20.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 -- 172.21.15.50:0/1832709885 >> v2:172.21.15.118:3300/0 conn(0x7fe5440a2450 msgr2=0x7fe54413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:20.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 --2- 172.21.15.50:0/1832709885 >> v2:172.21.15.118:3300/0 conn(0x7fe5440a2450 0x7fe54413aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 -- 172.21.15.50:0/1832709885 >> v1:172.21.15.118:6789/0 conn(0x7fe5440a2ae0 legacy=0x7fe5440a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:20.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 monclient: found mon.noname-a 2023-10-30T12:20:20.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:20.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.117+0000 7fe542d9d640 1 -- 172.21.15.50:0/1832709885 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54409a8a0 con 0x7fe54413af40 2023-10-30T12:20:20.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe542d9d640 1 --2- 172.21.15.50:0/1832709885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe54413af40 0x7fe54413d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe5440a4c80 tx=0x7fe53402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b56d3b2356d94bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 1 -- 172.21.15.50:0/1832709885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534031750 con 0x7fe54413af40 2023-10-30T12:20:20.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:20.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: dump: 2023-10-30T12:20:20.140 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.140 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.140 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.140 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.140 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.140 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.141 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.141 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.141 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.141 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 1 -- 172.21.15.50:0/1832709885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5340318b0 con 0x7fe54413af40 2023-10-30T12:20:20.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe548b00640 5 monclient: authenticate success, global_id 244284 2023-10-30T12:20:20.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe548b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:20.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe548b00640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:20.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 1 -- 172.21.15.50:0/1832709885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534031c40 con 0x7fe54413af40 2023-10-30T12:20:20.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe548b00640 4 set_mon_vals no callback set 2023-10-30T12:20:20.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:20.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.118+0000 7fe54159a640 10 monclient: dump: 2023-10-30T12:20:20.145 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.145 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.145 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.146 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.146 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.146 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.146 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.147 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.147 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.147 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 10 monclient: shutdown 2023-10-30T12:20:20.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 -- 172.21.15.50:0/1832709885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe54413af40 msgr2=0x7fe54413d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 --2- 172.21.15.50:0/1832709885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe54413af40 0x7fe54413d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe5440a4c80 tx=0x7fe53402b7f0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 -- 172.21.15.50:0/1832709885 shutdown_connections 2023-10-30T12:20:20.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 --2- 172.21.15.50:0/1832709885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe54413af40 0x7fe54413d330 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 --2- 172.21.15.50:0/1832709885 >> v2:172.21.15.118:3300/0 conn(0x7fe5440a2450 0x7fe54413aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 -- 172.21.15.50:0/1832709885 >> 172.21.15.50:0/1832709885 conn(0x7fe5440a80b0 msgr2=0x7fe5440a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 -- 172.21.15.50:0/1832709885 shutdown_connections 2023-10-30T12:20:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.119+0000 7fe548b00640 1 -- 172.21.15.50:0/1832709885 wait complete. 2023-10-30T12:20:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient: monmap: 2023-10-30T12:20:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.151 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.151 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:20.151 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:20.151 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:20.151 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 librados: starting msgr at 2023-10-30T12:20:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 librados: starting objecter 2023-10-30T12:20:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 Processor -- start 2023-10-30T12:20:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 -- start start 2023-10-30T12:20:20.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 librados: setting wanted keys 2023-10-30T12:20:20.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 librados: calling monclient init 2023-10-30T12:20:20.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient: init 2023-10-30T12:20:20.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:20:20.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5440a2450 0x7fe5441ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): picked mon.noname-a con 0x7fe5440a2450 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:20.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5440a2ae0 0x7fe544136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): picked mon.noname-b con 0x7fe5440a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe54413af40 0x7fe544137280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): picked mon.noname-c con 0x7fe54413af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54411e5b0 con 0x7fe5440a2450 2023-10-30T12:20:20.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54411e800 con 0x7fe5440a2ae0 2023-10-30T12:20:20.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe544139620 con 0x7fe54413af40 2023-10-30T12:20:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe548b00640 10 monclient(hunting): authenticate will time out at 2065.303850s 2023-10-30T12:20:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe54259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5440a2450 0x7fe5441ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe542d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe54413af40 0x7fe544137280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe54259c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5440a2450 0x7fe5441ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58020/0 (socket says 172.21.15.50:58020) 2023-10-30T12:20:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe54259c640 1 -- 172.21.15.50:0/1265714145 learned_addr learned my addr 172.21.15.50:0/1265714145 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe54259c640 10 monclient(hunting): get_auth_request con 0x7fe5440a2450 auth_method 0 2023-10-30T12:20:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe54259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe54259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.120+0000 7fe54259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe542d9d640 10 monclient(hunting): get_auth_request con 0x7fe54413af40 auth_method 0 2023-10-30T12:20:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe542d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe542d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe542d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5440a2ae0 0x7fe544136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe542d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe542d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe542d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 10 monclient(hunting): get_auth_request con 0x7fe5440a2ae0 auth_method 0 2023-10-30T12:20:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 10 monclient(hunting): handle_auth_done global_id 244287 payload 824 2023-10-30T12:20:20.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe54413af40 msgr2=0x7fe544137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe54413af40 0x7fe544137280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5440a2ae0 msgr2=0x7fe544136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe5440a2ae0 0x7fe544136d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 1 monclient: found mon.noname-a 2023-10-30T12:20:20.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:20.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 1 -- 172.21.15.50:0/1265714145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5340094d0 con 0x7fe5440a2450 2023-10-30T12:20:20.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe54259c640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5440a2450 0x7fe5441ca750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00c7c0 tx=0x7fe53c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 client.?.objecter ms_handle_connect 0x7fe5440a2450 2023-10-30T12:20:20.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:20.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c00aa50 con 0x7fe5440a2450 2023-10-30T12:20:20.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:20.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: dump: 2023-10-30T12:20:20.166 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.166 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.168 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.168 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.168 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c00abb0 con 0x7fe5440a2450 2023-10-30T12:20:20.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 5 monclient: authenticate success, global_id 244287 2023-10-30T12:20:20.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 10 monclient: _renew_subs 2023-10-30T12:20:20.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:20.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5440a17f0 con 0x7fe5440a2450 2023-10-30T12:20:20.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c012420 con 0x7fe5440a2450 2023-10-30T12:20:20.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:20.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe53affd640 10 monclient: dump: 2023-10-30T12:20:20.171 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.171 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.171 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.171 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.172 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.172 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.172 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.172 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.172 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.173 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe540d99640 4 set_mon_vals no callback set 2023-10-30T12:20:20.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 10 client.244287.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:20.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 10 monclient: _renew_subs 2023-10-30T12:20:20.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:20:20.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5441389e0 con 0x7fe5440a2450 2023-10-30T12:20:20.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.121+0000 7fe548b00640 1 librados: init done 2023-10-30T12:20:20.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe548b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe548b00640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5441cac90 0x7fe5441377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe548b00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54411e990 con 0x7fe5441cac90 2023-10-30T12:20:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe53c012580 con 0x7fe5440a2450 2023-10-30T12:20:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5441cac90 0x7fe5441377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient: get_auth_request con 0x7fe5441cac90 auth_method 0 2023-10-30T12:20:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe53affd640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe5240392f0 0x7fe52403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 10 monclient(hunting): handle_auth_done global_id 244290 payload 824 2023-10-30T12:20:20.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe541d9b640 1 -- 172.21.15.50:0/1265714145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe54409a5c0 con 0x7fe5441cac90 2023-10-30T12:20:20.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(630..630 src has 490..630) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fe53c04afb0 con 0x7fe5440a2450 2023-10-30T12:20:20.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe53affd640 10 client.244287.objecter ms_dispatch 0x7fe544000b90 osd_map(630..630 src has 490..630) v4 2023-10-30T12:20:20.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe53affd640 3 client.244287.objecter handle_osd_map got epochs [630,630] > 0 2023-10-30T12:20:20.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.122+0000 7fe53affd640 3 client.244287.objecter handle_osd_map decoding full epoch 630 2023-10-30T12:20:20.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 20 client.244287.objecter dump_active .. 0 homeless 2023-10-30T12:20:20.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe542d9d640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe5240392f0 0x7fe52403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 client.244287.objecter ms_handle_connect 0x7fe5441cac90 2023-10-30T12:20:20.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 client.244287.objecter resend_mon_ops 2023-10-30T12:20:20.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe541d9b640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5441cac90 0x7fe5441377c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe5441385e0 tx=0x7fe52c010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe542d9d640 10 monclient: get_auth_request con 0x7fe5240392f0 auth_method 0 2023-10-30T12:20:20.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c004240 con 0x7fe5441cac90 2023-10-30T12:20:20.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:20.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 monclient: dump: 2023-10-30T12:20:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.182 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.182 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.182 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.182 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.183 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fe52c0043a0 con 0x7fe5441cac90 2023-10-30T12:20:20.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:20.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5441cac90 msgr2=0x7fe5441377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5441cac90 0x7fe5441377c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe5441385e0 tx=0x7fe52c010c10 comp rx=0 tx=0).stop 2023-10-30T12:20:20.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5441cac90 msgr2=0x7fe5441377c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:20.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5441cac90 0x7fe5441377c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe542d9d640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe5240392f0 0x7fe52403b7b0 secure :-1 s=READY pgs=4309 cs=0 l=1 rev1=1 crypto rx=0x7fe534035000 tx=0x7fe534008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.123+0000 7fe53affd640 10 client.244287.objecter ms_handle_connect 0x7fe5240392f0 2023-10-30T12:20:20.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe548b00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:20.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe548b00640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe544137d00 0x7fe5441332d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe548b00640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe544073ff0 con 0x7fe544137d00 2023-10-30T12:20:20.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe541d9b640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe544137d00 0x7fe5441332d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe541d9b640 10 monclient: get_auth_request con 0x7fe544137d00 auth_method 0 2023-10-30T12:20:20.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe541d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe541d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.129+0000 7fe541d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe541d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe541d9b640 10 monclient(hunting): handle_auth_done global_id 244293 payload 824 2023-10-30T12:20:20.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe541d9b640 1 -- 172.21.15.50:0/1265714145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe5440a28a0 con 0x7fe544137d00 2023-10-30T12:20:20.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe541d9b640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe544137d00 0x7fe5441332d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe5441385e0 tx=0x7fe52c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 10 client.244287.objecter ms_handle_connect 0x7fe544137d00 2023-10-30T12:20:20.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 10 client.244287.objecter resend_mon_ops 2023-10-30T12:20:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c025070 con 0x7fe544137d00 2023-10-30T12:20:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 10 monclient: dump: 2023-10-30T12:20:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.192 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.192 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.192 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.192 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.192 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.192 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.193 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4804 (secure 0 0 0) 0x7fe52c004510 con 0x7fe544137d00 2023-10-30T12:20:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:20.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe544137d00 msgr2=0x7fe5441332d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe544137d00 0x7fe5441332d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe5441385e0 tx=0x7fe52c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe544137d00 msgr2=0x7fe5441332d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe53affd640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe544137d00 0x7fe5441332d0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe5441385e0 tx=0x7fe52c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.195 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:20.195 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:20.196 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:20.196 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:20:20.196 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 204, 2023-10-30T12:20:20.197 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:20.197 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:20:20.197 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:20.197 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:20.197 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.198 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:20:20.198 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:20.198 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:20.198 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:20.199 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.199 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.199 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.199 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.199 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.199 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.200 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.200 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.200 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.200 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.200 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.200 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:20.201 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:20.201 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.201 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.201 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.201 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.202 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.202 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.202 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.202 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.202 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.203 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.203 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.203 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:20.204 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:20.204 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:20.204 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:20.204 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:20.204 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:20.205 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:20.205 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:20.205 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:20.205 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:20.206 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:20.206 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:20.206 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:20.207 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:20.207 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:20.207 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:20.207 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:20.207 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.208 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.208 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.209 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.209 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.209 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.209 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.209 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.210 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.210 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.210 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:20.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.210 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:20.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.211 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:20.211 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:20.211 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.211 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.212 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.212 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:20.212 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.213 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.213 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:20.213 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.214 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.214 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:20.214 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:20.214 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.215 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.215 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.215 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:20.216 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:20.216 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.216 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:20.217 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.218 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.218 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:20.218 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.219 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.219 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.219 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:20.220 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:20.220 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.220 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.220 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.221 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:20.221 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:20.221 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.221 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.222 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.222 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:20.222 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.223 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:20.223 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.223 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.223 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.224 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:20.224 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:20.224 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.224 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.225 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.225 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.225 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.225 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:20.226 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:20.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.226 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.227 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.227 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:20.227 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.227 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.227 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:20:20.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.228 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.228 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.228 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:20:20.228 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.229 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.229 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:20.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.230 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.230 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.230 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:20:20.230 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.230 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:20:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.232 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:20.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.232 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.232 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:20.233 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:20.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 10 librados: watch_flush enter 2023-10-30T12:20:20.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 10 librados: watch_flush exit 2023-10-30T12:20:20.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 20 client.244287.objecter shutdown clearing up homeless session... 2023-10-30T12:20:20.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 10 client.244287.objecter successfully canceled tick 2023-10-30T12:20:20.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe5240392f0 msgr2=0x7fe52403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe5240392f0 0x7fe52403b7b0 secure :-1 s=READY pgs=4309 cs=0 l=1 rev1=1 crypto rx=0x7fe534035000 tx=0x7fe534008030 comp rx=0 tx=0).stop 2023-10-30T12:20:20.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 10 monclient: shutdown 2023-10-30T12:20:20.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5440a2450 msgr2=0x7fe5441ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.130+0000 7fe548b00640 1 --2- 172.21.15.50:0/1265714145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5440a2450 0x7fe5441ca750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00c7c0 tx=0x7fe53c00cc80 comp rx=0 tx=0).stop 2023-10-30T12:20:20.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.131+0000 7fe542d9d640 1 -- 172.21.15.50:0/1265714145 reap_dead start 2023-10-30T12:20:20.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.131+0000 7fe542d9d640 1 -- 172.21.15.50:0/1265714145 reap_dead start 2023-10-30T12:20:20.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.131+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 shutdown_connections 2023-10-30T12:20:20.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.131+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 >> 172.21.15.50:0/1265714145 conn(0x7fe5440a80b0 msgr2=0x7fe54409dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.131+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 shutdown_connections 2023-10-30T12:20:20.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.131+0000 7fe548b00640 1 -- 172.21.15.50:0/1265714145 wait complete. 2023-10-30T12:20:20.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.131+0000 7fe548b00640 1 librados: shutdown 2023-10-30T12:20:20.239 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:20:20.239 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:20:20.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:20:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.344+0000 7f5a7981a640 10 monclient: get_monmap_and_config 2023-10-30T12:20:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.344+0000 7f5a7981a640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.344+0000 7f5a7981a640 10 monclient: monmap: 2023-10-30T12:20:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:20.345946+0000 2023-10-30T12:20:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:20.345946+0000 2023-10-30T12:20:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:20.349 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:20.349 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 1 Processor -- start 2023-10-30T12:20:20.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 1 -- start start 2023-10-30T12:20:20.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient: init 2023-10-30T12:20:20.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T12:20:20.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74096920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient(hunting): picked mon.noname-b con 0x7f5a74096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:20.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a74096e60 0x7f5a7409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient(hunting): picked mon.noname-a con 0x7f5a74096e60 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:20.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a7409afa0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:20.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7409bb20 con 0x7f5a7409afa0 2023-10-30T12:20:20.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7409bd30 con 0x7f5a74096e60 2023-10-30T12:20:20.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7409c400 con 0x7f5a74096540 2023-10-30T12:20:20.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.345+0000 7f5a7981a640 10 monclient(hunting): authenticate will time out at 2065.528976s 2023-10-30T12:20:20.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a74096e60 0x7f5a7409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a74096e60 0x7f5a7409aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58044/0 (socket says 172.21.15.50:58044) 2023-10-30T12:20:20.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52442/0 (socket says 172.21.15.50:52442) 2023-10-30T12:20:20.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 -- 172.21.15.50:0/3012619693 learned_addr learned my addr 172.21.15.50:0/3012619693 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): get_auth_request con 0x7f5a74096e60 auth_method 0 2023-10-30T12:20:20.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 10 monclient(hunting): get_auth_request con 0x7f5a74096540 auth_method 0 2023-10-30T12:20:20.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient(hunting): handle_auth_done global_id 244296 payload 274 2023-10-30T12:20:20.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 -- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 msgr2=0x7f5a74096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:20.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 --2- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 -- 172.21.15.50:0/3012619693 >> v1:172.21.15.118:6789/0 conn(0x7f5a7409afa0 legacy=0x7f5a7409b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:20.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 monclient: found mon.noname-a 2023-10-30T12:20:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a727fc640 1 -- 172.21.15.50:0/3012619693 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7409c120 con 0x7f5a74096e60 2023-10-30T12:20:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.346+0000 7f5a72ffd640 1 --2- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:20:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a727fc640 1 --2- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a74096e60 0x7f5a7409aa60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009870 tx=0x7f5a6802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d7a585220a940d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 1 -- 172.21.15.50:0/3012619693 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68031640 con 0x7f5a74096e60 2023-10-30T12:20:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: dump: 2023-10-30T12:20:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.377 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.377 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.377 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.377 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 5 monclient: authenticate success, global_id 244296 2023-10-30T12:20:20.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:20.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 1 -- 172.21.15.50:0/3012619693 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a680317a0 con 0x7f5a74096e60 2023-10-30T12:20:20.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:20.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 4 set_mon_vals no callback set 2023-10-30T12:20:20.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 1 -- 172.21.15.50:0/3012619693 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68031b30 con 0x7f5a74096e60 2023-10-30T12:20:20.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:20.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a71ffb640 10 monclient: dump: 2023-10-30T12:20:20.380 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.381 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.381 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.381 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.381 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.382 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.382 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.382 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.382 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.383 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 10 monclient: shutdown 2023-10-30T12:20:20.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 1 -- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a74096e60 msgr2=0x7f5a7409aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 1 --2- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a74096e60 0x7f5a7409aa60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009870 tx=0x7f5a6802b8f0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 1 -- 172.21.15.50:0/3012619693 shutdown_connections 2023-10-30T12:20:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 1 --2- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a74096e60 0x7f5a7409aa60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 1 --2- 172.21.15.50:0/3012619693 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.347+0000 7f5a7981a640 1 -- 172.21.15.50:0/3012619693 >> 172.21.15.50:0/3012619693 conn(0x7f5a740a3980 msgr2=0x7f5a740a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.348+0000 7f5a7981a640 1 -- 172.21.15.50:0/3012619693 shutdown_connections 2023-10-30T12:20:20.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.348+0000 7f5a7981a640 1 -- 172.21.15.50:0/3012619693 wait complete. 2023-10-30T12:20:20.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.348+0000 7f5a7981a640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.348+0000 7f5a7981a640 10 monclient: monmap: 2023-10-30T12:20:20.385 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.385 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:20.385 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:20.386 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:20.386 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.386 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.386 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:20.386 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:20.387 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:20.387 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.348+0000 7f5a7981a640 1 librados: starting msgr at 2023-10-30T12:20:20.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.348+0000 7f5a7981a640 1 librados: starting objecter 2023-10-30T12:20:20.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.348+0000 7f5a7981a640 1 Processor -- start 2023-10-30T12:20:20.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 -- start start 2023-10-30T12:20:20.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 librados: setting wanted keys 2023-10-30T12:20:20.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 librados: calling monclient init 2023-10-30T12:20:20.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient: init 2023-10-30T12:20:20.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:20:20.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74092c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): picked mon.noname-c con 0x7f5a74096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:20.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a74096e60 0x7f5a7408e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): picked mon.noname-b con 0x7f5a74096e60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:20.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a7409afa0 0x7f5a7408e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): picked mon.noname-a con 0x7f5a7409afa0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:20.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74091bf0 con 0x7f5a7409afa0 2023-10-30T12:20:20.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74095440 con 0x7f5a74096e60 2023-10-30T12:20:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7409a080 con 0x7f5a74096540 2023-10-30T12:20:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a7981a640 10 monclient(hunting): authenticate will time out at 2065.532513s 2023-10-30T12:20:20.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74092c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a7409afa0 0x7f5a7408e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74092c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52448/0 (socket says 172.21.15.50:52448) 2023-10-30T12:20:20.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 10 monclient(hunting): get_auth_request con 0x7f5a7409afa0 auth_method 0 2023-10-30T12:20:20.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 -- 172.21.15.50:0/1483228160 learned_addr learned my addr 172.21.15.50:0/1483228160 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): get_auth_request con 0x7f5a74096540 auth_method 0 2023-10-30T12:20:20.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a727fc640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a74096e60 0x7f5a7408e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient(hunting): handle_auth_done global_id 254123 payload 824 2023-10-30T12:20:20.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a74096e60 msgr2=0x7f5a7408e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a74096e60 0x7f5a7408e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a7409afa0 msgr2=0x7f5a7408e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a7409afa0 0x7f5a7408e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 monclient: found mon.noname-c 2023-10-30T12:20:20.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a72ffd640 1 -- 172.21.15.50:0/1483228160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a640093a0 con 0x7f5a74096540 2023-10-30T12:20:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a727fc640 10 monclient: get_auth_request con 0x7f5a74096e60 auth_method 0 2023-10-30T12:20:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a727fc640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a74096e60 0x7f5a7408e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:20:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.349+0000 7f5a737fe640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a7409afa0 0x7f5a7408e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:20:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a72ffd640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74092c20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400ebe0 tx=0x7f5a6400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 client.?.objecter ms_handle_connect 0x7f5a74096540 2023-10-30T12:20:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64009750 con 0x7f5a74096540 2023-10-30T12:20:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: dump: 2023-10-30T12:20:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.403 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.403 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.404 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.404 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6400cc30 con 0x7f5a74096540 2023-10-30T12:20:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 5 monclient: authenticate success, global_id 254123 2023-10-30T12:20:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 10 monclient: _renew_subs 2023-10-30T12:20:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a680094d0 con 0x7f5a74096540 2023-10-30T12:20:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400eee0 con 0x7f5a74096540 2023-10-30T12:20:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a5bfff640 10 monclient: dump: 2023-10-30T12:20:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.409 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 10 client.254123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 10 monclient: _renew_subs 2023-10-30T12:20:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a717fa640 4 set_mon_vals no callback set 2023-10-30T12:20:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a74092490 con 0x7f5a74096540 2023-10-30T12:20:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.350+0000 7f5a7981a640 1 librados: init done 2023-10-30T12:20:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5a64005990 con 0x7f5a74096540 2023-10-30T12:20:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a7981a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a7981a640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a7408eea0 0x7f5a741ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a7981a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74073ff0 con 0x7f5a7408eea0 2023-10-30T12:20:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a7408eea0 0x7f5a741ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a60039320 0x7f5a6003b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient: get_auth_request con 0x7f5a7408eea0 auth_method 0 2023-10-30T12:20:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(630..630 src has 490..630) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f5a6404b370 con 0x7f5a74096540 2023-10-30T12:20:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 10 client.254123.objecter ms_dispatch 0x7f5a74000b90 osd_map(630..630 src has 490..630) v4 2023-10-30T12:20:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 3 client.254123.objecter handle_osd_map got epochs [630,630] > 0 2023-10-30T12:20:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 3 client.254123.objecter handle_osd_map decoding full epoch 630 2023-10-30T12:20:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 20 client.254123.objecter dump_active .. 0 homeless 2023-10-30T12:20:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a737fe640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a60039320 0x7f5a6003b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 10 monclient(hunting): handle_auth_done global_id 254126 payload 824 2023-10-30T12:20:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 1 -- 172.21.15.50:0/1483228160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a7409d3a0 con 0x7f5a7408eea0 2023-10-30T12:20:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a737fe640 10 monclient: get_auth_request con 0x7f5a60039320 auth_method 0 2023-10-30T12:20:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a727fc640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a7408eea0 0x7f5a741ca560 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a7409c8f0 tx=0x7f5a68004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 10 client.254123.objecter ms_handle_connect 0x7f5a7408eea0 2023-10-30T12:20:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 10 client.254123.objecter resend_mon_ops 2023-10-30T12:20:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68031ea0 con 0x7f5a7408eea0 2023-10-30T12:20:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 10 monclient: dump: 2023-10-30T12:20:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.419 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a5bfff640 10 client.254123.objecter ms_handle_connect 0x7f5a60039320 2023-10-30T12:20:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.351+0000 7f5a737fe640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a60039320 0x7f5a6003b7e0 secure :-1 s=READY pgs=4310 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00f1d0 tx=0x7f5a6c003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.352+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f5a6803aa00 con 0x7f5a7408eea0 2023-10-30T12:20:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.352+0000 7f5a5bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.352+0000 7f5a5bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.352+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a7408eea0 msgr2=0x7f5a741ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.352+0000 7f5a5bfff640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a7408eea0 0x7f5a741ca560 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a7409c8f0 tx=0x7f5a68004730 comp rx=0 tx=0).stop 2023-10-30T12:20:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.352+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a7408eea0 msgr2=0x7f5a741ca560 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.352+0000 7f5a5bfff640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a7408eea0 0x7f5a741ca560 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a7409c8f0 tx=0x7f5a68004730 comp rx=0 tx=0).stop 2023-10-30T12:20:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a7981a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a7981a640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a741caaa0 0x7f5a741caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a7981a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a741cb3f0 con 0x7f5a741caaa0 2023-10-30T12:20:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a741caaa0 0x7f5a741caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient: get_auth_request con 0x7f5a741caaa0 auth_method 0 2023-10-30T12:20:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 10 monclient(hunting): handle_auth_done global_id 254129 payload 824 2023-10-30T12:20:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 1 -- 172.21.15.50:0/1483228160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5a7408f280 con 0x7f5a741caaa0 2023-10-30T12:20:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a727fc640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a741caaa0 0x7f5a741caeb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5a68000c00 tx=0x7f5a680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a5bfff640 10 client.254123.objecter ms_handle_connect 0x7f5a741caaa0 2023-10-30T12:20:20.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a5bfff640 10 client.254123.objecter resend_mon_ops 2023-10-30T12:20:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.358+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6804e670 con 0x7f5a741caaa0 2023-10-30T12:20:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 10 monclient: dump: 2023-10-30T12:20:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.428 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.429 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.430 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.430 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f5a6804e7d0 con 0x7f5a741caaa0 2023-10-30T12:20:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a741caaa0 msgr2=0x7f5a741caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a741caaa0 0x7f5a741caeb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5a68000c00 tx=0x7f5a680046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a741caaa0 msgr2=0x7f5a741caeb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:20.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a5bfff640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a741caaa0 0x7f5a741caeb0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5a68000c00 tx=0x7f5a680046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.432 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:20:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 204, 2023-10-30T12:20:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:20:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:20:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:20.470 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:20.470 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.470 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:20.471 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.472 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:20.473 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.474 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:20.474 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:20.474 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.475 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.475 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.476 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:20.476 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:20.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.476 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.476 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.477 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.478 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.478 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:20:20.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.478 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.479 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:20.479 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:20.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 10 librados: watch_flush enter 2023-10-30T12:20:20.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 10 librados: watch_flush exit 2023-10-30T12:20:20.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 20 client.254123.objecter shutdown clearing up homeless session... 2023-10-30T12:20:20.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 10 client.254123.objecter successfully canceled tick 2023-10-30T12:20:20.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a60039320 msgr2=0x7f5a6003b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5a60039320 0x7f5a6003b7e0 secure :-1 s=READY pgs=4310 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00f1d0 tx=0x7f5a6c003d10 comp rx=0 tx=0).stop 2023-10-30T12:20:20.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 10 monclient: shutdown 2023-10-30T12:20:20.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 msgr2=0x7f5a74092c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 --2- 172.21.15.50:0/1483228160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a74096540 0x7f5a74092c20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400ebe0 tx=0x7f5a6400c520 comp rx=0 tx=0).stop 2023-10-30T12:20:20.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a737fe640 1 -- 172.21.15.50:0/1483228160 reap_dead start 2023-10-30T12:20:20.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a737fe640 1 -- 172.21.15.50:0/1483228160 reap_dead start 2023-10-30T12:20:20.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 shutdown_connections 2023-10-30T12:20:20.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 >> 172.21.15.50:0/1483228160 conn(0x7f5a740a3980 msgr2=0x7f5a7406ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 shutdown_connections 2023-10-30T12:20:20.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 -- 172.21.15.50:0/1483228160 wait complete. 2023-10-30T12:20:20.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.359+0000 7f5a7981a640 1 librados: shutdown 2023-10-30T12:20:20.484 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:20:20.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:20:20.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:20:20.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.590+0000 7f6f59409640 10 monclient: get_monmap_and_config 2023-10-30T12:20:20.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.590+0000 7f6f59409640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.590+0000 7f6f59409640 10 monclient: monmap: 2023-10-30T12:20:20.593 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.593 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.593 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:20.591924+0000 2023-10-30T12:20:20.593 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:20.591924+0000 2023-10-30T12:20:20.594 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.594 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.594 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:20.594 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:20.594 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:20.595 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:20.595 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:20.595 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 1 Processor -- start 2023-10-30T12:20:20.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 1 -- start start 2023-10-30T12:20:20.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 10 monclient: init 2023-10-30T12:20:20.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-10-30T12:20:20.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f5406afc0 0x7f6f540a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 10 monclient(hunting): picked mon.noname-a con 0x7f6f5406afc0 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f540a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.591+0000 7f6f59409640 10 monclient(hunting): picked mon.noname-b con 0x7f6f5406b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f540a2710 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5406bde0 con 0x7f6f540a2710 2023-10-30T12:20:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f540a2b40 con 0x7f6f5406afc0 2023-10-30T12:20:20.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5406b3a0 con 0x7f6f5406b640 2023-10-30T12:20:20.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f59409640 10 monclient(hunting): authenticate will time out at 2065.775168s 2023-10-30T12:20:20.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f5406afc0 0x7f6f540a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f5406afc0 0x7f6f540a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58076/0 (socket says 172.21.15.50:58076) 2023-10-30T12:20:20.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 -- 172.21.15.50:0/3568092940 learned_addr learned my addr 172.21.15.50:0/3568092940 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): get_auth_request con 0x7f6f5406afc0 auth_method 0 2023-10-30T12:20:20.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 1 --2- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f540a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 10 monclient(hunting): get_auth_request con 0x7f6f5406b640 auth_method 0 2023-10-30T12:20:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_done global_id 244302 payload 274 2023-10-30T12:20:20.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 -- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 msgr2=0x7f6f540a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:20.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f540a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 -- 172.21.15.50:0/3568092940 >> v1:172.21.15.50:6789/0 conn(0x7f6f540a2710 legacy=0x7f6f5413c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 monclient: found mon.noname-a 2023-10-30T12:20:20.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:20:20.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f52ffd640 1 -- 172.21.15.50:0/3568092940 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5409e540 con 0x7f6f5406afc0 2023-10-30T12:20:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.592+0000 7f6f527fc640 1 --2- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f540a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:20:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f5406afc0 0x7f6f540a1c90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009870 tx=0x7f6f4402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51481ec166974268 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 1 -- 172.21.15.50:0/3568092940 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f440317c0 con 0x7f6f5406afc0 2023-10-30T12:20:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:20:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: dump: 2023-10-30T12:20:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.614 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 1 -- 172.21.15.50:0/3568092940 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f44031920 con 0x7f6f5406afc0 2023-10-30T12:20:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 5 monclient: authenticate success, global_id 244302 2023-10-30T12:20:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:20.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:20.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 4 set_mon_vals no callback set 2023-10-30T12:20:20.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 1 -- 172.21.15.50:0/3568092940 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44031cb0 con 0x7f6f5406afc0 2023-10-30T12:20:20.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:20:20.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f51ffb640 10 monclient: dump: 2023-10-30T12:20:20.617 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.617 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.617 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.618 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.618 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.618 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.618 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.618 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.618 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.619 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 10 monclient: shutdown 2023-10-30T12:20:20.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 1 -- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f5406afc0 msgr2=0x7f6f540a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 1 --2- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f5406afc0 0x7f6f540a1c90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009870 tx=0x7f6f4402b860 comp rx=0 tx=0).stop 2023-10-30T12:20:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 1 -- 172.21.15.50:0/3568092940 shutdown_connections 2023-10-30T12:20:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 1 --2- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f540a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 1 --2- 172.21.15.50:0/3568092940 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f5406afc0 0x7f6f540a1c90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.593+0000 7f6f59409640 1 -- 172.21.15.50:0/3568092940 >> 172.21.15.50:0/3568092940 conn(0x7f6f540a6200 msgr2=0x7f6f540a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.594+0000 7f6f59409640 1 -- 172.21.15.50:0/3568092940 shutdown_connections 2023-10-30T12:20:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.594+0000 7f6f59409640 1 -- 172.21.15.50:0/3568092940 wait complete. 2023-10-30T12:20:20.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.594+0000 7f6f59409640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.594+0000 7f6f59409640 10 monclient: monmap: 2023-10-30T12:20:20.621 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.621 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:20.621 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:20.622 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:20.622 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.622 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.622 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:20.622 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:20.623 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:20.623 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.594+0000 7f6f59409640 1 librados: starting msgr at 2023-10-30T12:20:20.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.594+0000 7f6f59409640 1 librados: starting objecter 2023-10-30T12:20:20.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 Processor -- start 2023-10-30T12:20:20.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 -- start start 2023-10-30T12:20:20.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 librados: setting wanted keys 2023-10-30T12:20:20.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 librados: calling monclient init 2023-10-30T12:20:20.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient: init 2023-10-30T12:20:20.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:20:20.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f5406afc0 0x7f6f541bdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): picked mon.noname-b con 0x7f6f5406afc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:20.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f541be1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): picked mon.noname-c con 0x7f6f5406b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:20.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f540a2710 0x7f6f541a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): picked mon.noname-a con 0x7f6f540a2710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:20.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f541a9cb0 con 0x7f6f540a2710 2023-10-30T12:20:20.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f541ab160 con 0x7f6f5406afc0 2023-10-30T12:20:20.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f541ab3b0 con 0x7f6f5406b640 2023-10-30T12:20:20.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f59409640 10 monclient(hunting): authenticate will time out at 2065.778505s 2023-10-30T12:20:20.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f541be1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f540a2710 0x7f6f541a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f541be1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52482/0 (socket says 172.21.15.50:52482) 2023-10-30T12:20:20.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 -- 172.21.15.50:0/2135608363 learned_addr learned my addr 172.21.15.50:0/2135608363 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): get_auth_request con 0x7f6f5406b640 auth_method 0 2023-10-30T12:20:20.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 10 monclient(hunting): get_auth_request con 0x7f6f540a2710 auth_method 0 2023-10-30T12:20:20.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f5406afc0 0x7f6f541bdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f52ffd640 10 monclient(hunting): get_auth_request con 0x7f6f5406afc0 auth_method 0 2023-10-30T12:20:20.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f52ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient(hunting): handle_auth_done global_id 254135 payload 824 2023-10-30T12:20:20.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f5406afc0 msgr2=0x7f6f541bdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f5406afc0 0x7f6f541bdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f540a2710 msgr2=0x7f6f541a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f540a2710 0x7f6f541a73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 monclient: found mon.noname-c 2023-10-30T12:20:20.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f527fc640 1 -- 172.21.15.50:0/2135608363 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48009400 con 0x7f6f5406b640 2023-10-30T12:20:20.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f5406afc0 0x7f6f541bdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:20:20.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.595+0000 7f6f537fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f540a2710 0x7f6f541a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:20:20.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f527fc640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f541be1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800ec10 tx=0x7f6f4800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 client.?.objecter ms_handle_connect 0x7f6f5406b640 2023-10-30T12:20:20.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:20.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48009760 con 0x7f6f5406b640 2023-10-30T12:20:20.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:20.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: dump: 2023-10-30T12:20:20.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.640 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.640 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.640 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.640 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.641 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.641 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4800cc30 con 0x7f6f5406b640 2023-10-30T12:20:20.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 5 monclient: authenticate success, global_id 254135 2023-10-30T12:20:20.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48013470 con 0x7f6f5406b640 2023-10-30T12:20:20.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:20.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f3b7fe640 10 monclient: dump: 2023-10-30T12:20:20.643 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.643 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.644 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.644 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.644 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.644 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.645 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.645 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.645 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.645 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 10 monclient: _renew_subs 2023-10-30T12:20:20.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f440094d0 con 0x7f6f5406b640 2023-10-30T12:20:20.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f517fa640 4 set_mon_vals no callback set 2023-10-30T12:20:20.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 10 client.254135.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:20.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 10 monclient: _renew_subs 2023-10-30T12:20:20.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f541a99a0 con 0x7f6f5406b640 2023-10-30T12:20:20.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.596+0000 7f6f59409640 1 librados: init done 2023-10-30T12:20:20.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6f480135d0 con 0x7f6f5406b640 2023-10-30T12:20:20.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f59409640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f59409640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a80e0 0x7f6f5402b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f59409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f541ab540 con 0x7f6f541a80e0 2023-10-30T12:20:20.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f3b7fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6f400392f0 0x7f6f4003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(630..630 src has 490..630) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f6f4804b080 con 0x7f6f5406b640 2023-10-30T12:20:20.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f3b7fe640 10 client.254135.objecter ms_dispatch 0x7f6f54000b90 osd_map(630..630 src has 490..630) v4 2023-10-30T12:20:20.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f3b7fe640 3 client.254135.objecter handle_osd_map got epochs [630,630] > 0 2023-10-30T12:20:20.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f3b7fe640 3 client.254135.objecter handle_osd_map decoding full epoch 630 2023-10-30T12:20:20.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a80e0 0x7f6f5402b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f3b7fe640 20 client.254135.objecter dump_active .. 0 homeless 2023-10-30T12:20:20.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 10 monclient: get_auth_request con 0x7f6f541a80e0 auth_method 0 2023-10-30T12:20:20.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f537fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6f400392f0 0x7f6f4003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f537fe640 10 monclient: get_auth_request con 0x7f6f400392f0 auth_method 0 2023-10-30T12:20:20.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.597+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_done global_id 204673 payload 824 2023-10-30T12:20:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f52ffd640 1 -- 172.21.15.50:0/2135608363 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f5409df80 con 0x7f6f541a80e0 2023-10-30T12:20:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 client.254135.objecter ms_handle_connect 0x7f6f400392f0 2023-10-30T12:20:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f537fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6f400392f0 0x7f6f4003b7b0 secure :-1 s=READY pgs=4311 cs=0 l=1 rev1=1 crypto rx=0x7f6f541a8cd0 tx=0x7f6f4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 client.254135.objecter ms_handle_connect 0x7f6f541a80e0 2023-10-30T12:20:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 client.254135.objecter resend_mon_ops 2023-10-30T12:20:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a80e0 0x7f6f5402b430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6f540712a0 tx=0x7f6f44004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44032970 con 0x7f6f541a80e0 2023-10-30T12:20:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 monclient: dump: 2023-10-30T12:20:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.655 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f6f44032ad0 con 0x7f6f541a80e0 2023-10-30T12:20:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a80e0 msgr2=0x7f6f5402b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a80e0 0x7f6f5402b430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6f540712a0 tx=0x7f6f44004730 comp rx=0 tx=0).stop 2023-10-30T12:20:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a80e0 msgr2=0x7f6f5402b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.598+0000 7f6f3b7fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a80e0 0x7f6f5402b430 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.604+0000 7f6f59409640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.604+0000 7f6f59409640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a84c0 0x7f6f5402b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.604+0000 7f6f59409640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.604+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f54073ff0 con 0x7f6f541a84c0 2023-10-30T12:20:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a84c0 0x7f6f5402b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient: get_auth_request con 0x7f6f541a84c0 auth_method 0 2023-10-30T12:20:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 10 monclient(hunting): handle_auth_done global_id 204676 payload 824 2023-10-30T12:20:20.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 1 -- 172.21.15.50:0/2135608363 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6f5406a6f0 con 0x7f6f541a84c0 2023-10-30T12:20:20.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f52ffd640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a84c0 0x7f6f5402b970 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009870 tx=0x7f6f440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f3b7fe640 10 client.254135.objecter ms_handle_connect 0x7f6f541a84c0 2023-10-30T12:20:20.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f3b7fe640 10 client.254135.objecter resend_mon_ops 2023-10-30T12:20:20.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4404f660 con 0x7f6f541a84c0 2023-10-30T12:20:20.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:20.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.605+0000 7f6f3b7fe640 10 monclient: dump: 2023-10-30T12:20:20.663 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.663 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.663 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.663 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.663 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.664 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.664 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.664 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.664 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.664 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f6f44053070 con 0x7f6f541a84c0 2023-10-30T12:20:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f3b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f3b7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a84c0 msgr2=0x7f6f5402b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f3b7fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a84c0 0x7f6f5402b970 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009870 tx=0x7f6f440046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f3b7fe640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a84c0 msgr2=0x7f6f5402b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:20.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f3b7fe640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6f541a84c0 0x7f6f5402b970 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.666 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:20.666 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:20.667 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:20.667 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:20:20.667 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 204, 2023-10-30T12:20:20.667 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:20.667 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:20:20.668 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:20.668 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:20.668 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.668 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 7, 2023-10-30T12:20:20.668 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:20.669 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:20.669 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:20.669 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.669 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.670 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.670 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.670 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.670 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.670 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.671 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.671 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.671 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.671 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:20.671 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:20.671 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.672 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.672 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.672 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.672 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.673 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.673 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.673 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.673 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.673 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.674 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:20.674 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:20.674 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:20.675 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:20.675 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:20.675 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:20.675 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:20.675 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:20.676 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:20.676 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:20.676 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:20.676 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:20.676 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:20.677 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:20.677 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:20.677 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:20.677 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:20.677 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:20.678 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:20.679 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:20.679 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:20.679 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:20.679 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:20.679 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.680 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:20.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:20.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.680 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:20.681 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:20.681 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.681 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.682 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:20.682 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.682 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.683 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:20.683 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.683 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.683 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.684 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:20.684 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:20.684 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.684 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.685 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.685 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:20.685 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:20.686 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.686 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.686 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.687 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:20.687 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.688 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.689 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:20.689 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.689 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.690 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:20.690 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:20.691 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.691 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.691 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.692 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:20.692 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:20.692 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:20.693 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:20.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.693 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:20.693 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:20.693 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.694 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:20.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:20.694 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:20.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.695 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:20.695 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:20.696 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:20.696 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:20.696 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:20.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.697 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.698 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:20.698 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.698 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:20:20.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.699 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.699 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.699 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:20:20.699 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.700 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:20.700 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:20.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:20.700 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:20.700 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:20.701 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:20:20.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:20.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:20.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:20.701 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:20.701 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:20.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 10 librados: watch_flush enter 2023-10-30T12:20:20.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 10 librados: watch_flush exit 2023-10-30T12:20:20.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 20 client.254135.objecter shutdown clearing up homeless session... 2023-10-30T12:20:20.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 10 client.254135.objecter successfully canceled tick 2023-10-30T12:20:20.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6f400392f0 msgr2=0x7f6f4003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6f400392f0 0x7f6f4003b7b0 secure :-1 s=READY pgs=4311 cs=0 l=1 rev1=1 crypto rx=0x7f6f541a8cd0 tx=0x7f6f4c009040 comp rx=0 tx=0).stop 2023-10-30T12:20:20.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 10 monclient: shutdown 2023-10-30T12:20:20.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 msgr2=0x7f6f541be1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 --2- 172.21.15.50:0/2135608363 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6f5406b640 0x7f6f541be1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800ec10 tx=0x7f6f4800c520 comp rx=0 tx=0).stop 2023-10-30T12:20:20.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f537fe640 1 -- 172.21.15.50:0/2135608363 reap_dead start 2023-10-30T12:20:20.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f537fe640 1 -- 172.21.15.50:0/2135608363 reap_dead start 2023-10-30T12:20:20.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 shutdown_connections 2023-10-30T12:20:20.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 >> 172.21.15.50:0/2135608363 conn(0x7f6f540a6200 msgr2=0x7f6f540a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 shutdown_connections 2023-10-30T12:20:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 -- 172.21.15.50:0/2135608363 wait complete. 2023-10-30T12:20:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.606+0000 7f6f59409640 1 librados: shutdown 2023-10-30T12:20:20.707 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:20:20.707 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-10-30T12:20:20.708 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-10-30T12:20:20.708 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-10-30T12:20:20.708 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:20:20.708 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:20:20.709 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:20:20.721 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:20:20.721 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:20:20.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.865+0000 7fd3adf7c640 10 monclient: get_monmap_and_config 2023-10-30T12:20:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.865+0000 7fd3adf7c640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.865+0000 7fd3adf7c640 10 monclient: monmap: 2023-10-30T12:20:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:20.866176+0000 2023-10-30T12:20:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:20.866176+0000 2023-10-30T12:20:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:20.870 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.865+0000 7fd3adf7c640 1 Processor -- start 2023-10-30T12:20:20.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.865+0000 7fd3adf7c640 1 -- start start 2023-10-30T12:20:20.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.865+0000 7fd3adf7c640 10 monclient: init 2023-10-30T12:20:20.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-10-30T12:20:20.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3a80a4b60 0x7fd3a81326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a80a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:20.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8099a40 0x7fd3a8132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): picked mon.noname-b con 0x7fd3a8099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:20.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd3a80a2350 0x7fd3a80a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): picked mon.noname-c con 0x7fd3a80a2350 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:20.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80a3d70 con 0x7fd3a80a4b60 2023-10-30T12:20:20.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a8133120 con 0x7fd3a80a2350 2023-10-30T12:20:20.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80a4f40 con 0x7fd3a8099a40 2023-10-30T12:20:20.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3adf7c640 10 monclient(hunting): authenticate will time out at 2066.049365s 2023-10-30T12:20:20.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a77fe640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fd3a80a4b60 msgr2=0x7fd3a81326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:20.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a77fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3a80a4b60 0x7fd3a81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:20.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8099a40 0x7fd3a8132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8099a40 0x7fd3a8132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52516/0 (socket says 172.21.15.50:52516) 2023-10-30T12:20:20.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 -- 172.21.15.50:0/1513565062 learned_addr learned my addr 172.21.15.50:0/1513565062 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): get_auth_request con 0x7fd3a8099a40 auth_method 0 2023-10-30T12:20:20.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a7fff640 1 --2- 172.21.15.50:0/1513565062 >> v2:172.21.15.118:3300/0 conn(0x7fd3a80a2350 0x7fd3a80a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a7fff640 10 monclient(hunting): get_auth_request con 0x7fd3a80a2350 auth_method 0 2023-10-30T12:20:20.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient(hunting): handle_auth_done global_id 254138 payload 274 2023-10-30T12:20:20.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 -- 172.21.15.50:0/1513565062 >> v2:172.21.15.118:3300/0 conn(0x7fd3a80a2350 msgr2=0x7fd3a80a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:20.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 --2- 172.21.15.50:0/1513565062 >> v2:172.21.15.118:3300/0 conn(0x7fd3a80a2350 0x7fd3a80a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 -- 172.21.15.50:0/1513565062 >> v2:172.21.15.50:3300/0 conn(0x7fd3a80a4b60 msgr2=0x7fd3a81326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:20.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 --2- 172.21.15.50:0/1513565062 >> v2:172.21.15.50:3300/0 conn(0x7fd3a80a4b60 0x7fd3a81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 monclient: found mon.noname-b 2023-10-30T12:20:20.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a6ffd640 1 -- 172.21.15.50:0/1513565062 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80a4150 con 0x7fd3a8099a40 2023-10-30T12:20:20.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.866+0000 7fd3a7fff640 1 --2- 172.21.15.50:0/1513565062 >> v2:172.21.15.118:3300/0 conn(0x7fd3a80a2350 0x7fd3a80a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:20:20.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a6ffd640 1 --2- 172.21.15.50:0/1513565062 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8099a40 0x7fd3a8132be0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80a53f0 tx=0x7fd39c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d786352dfa1f7f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 1 -- 172.21.15.50:0/1513565062 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c031770 con 0x7fd3a8099a40 2023-10-30T12:20:20.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:20:20.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: dump: 2023-10-30T12:20:20.892 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.892 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.892 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.893 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.893 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.893 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.893 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.894 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.894 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.894 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3adf7c640 5 monclient: authenticate success, global_id 254138 2023-10-30T12:20:20.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3adf7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:20.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 1 -- 172.21.15.50:0/1513565062 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c0318d0 con 0x7fd3a8099a40 2023-10-30T12:20:20.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3adf7c640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:20.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3adf7c640 4 set_mon_vals no callback set 2023-10-30T12:20:20.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 1 -- 172.21.15.50:0/1513565062 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c031c60 con 0x7fd3a8099a40 2023-10-30T12:20:20.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:20.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.867+0000 7fd3a67fc640 10 monclient: dump: 2023-10-30T12:20:20.897 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.897 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.897 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.897 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.898 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.898 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.898 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.898 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.898 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.899 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 10 monclient: shutdown 2023-10-30T12:20:20.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:20.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1513565062 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8099a40 msgr2=0x7fd3a8132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1513565062 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8099a40 0x7fd3a8132be0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80a53f0 tx=0x7fd39c02b810 comp rx=0 tx=0).stop 2023-10-30T12:20:20.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1513565062 shutdown_connections 2023-10-30T12:20:20.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1513565062 >> v2:172.21.15.118:3300/0 conn(0x7fd3a80a2350 0x7fd3a80a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1513565062 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8099a40 0x7fd3a8132be0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1513565062 >> v2:172.21.15.50:3300/0 conn(0x7fd3a80a4b60 0x7fd3a81326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1513565062 >> 172.21.15.50:0/1513565062 conn(0x7fd3a80a8060 msgr2=0x7fd3a80a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:20.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1513565062 shutdown_connections 2023-10-30T12:20:20.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1513565062 wait complete. 2023-10-30T12:20:20.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 10 monclient: build_initial_monmap 2023-10-30T12:20:20.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.868+0000 7fd3adf7c640 10 monclient: monmap: 2023-10-30T12:20:20.901 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:20.901 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:20.902 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:20.902 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:20.902 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:20.902 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.902 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:20.903 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:20.903 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:20.903 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 librados: starting msgr at 2023-10-30T12:20:20.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 librados: starting objecter 2023-10-30T12:20:20.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 Processor -- start 2023-10-30T12:20:20.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 -- start start 2023-10-30T12:20:20.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 librados: setting wanted keys 2023-10-30T12:20:20.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 librados: calling monclient init 2023-10-30T12:20:20.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient: init 2023-10-30T12:20:20.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:20.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:20:20.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3a80a2350 0x7fd3a811e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): picked mon.noname-b con 0x7fd3a80a2350 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:20.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3a80a4b60 0x7fd3a811ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a80a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:20.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8131d90 0x7fd3a811f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): picked mon.noname-c con 0x7fd3a8131d90 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:20.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8131060 con 0x7fd3a80a4b60 2023-10-30T12:20:20.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8122390 con 0x7fd3a80a2350 2023-10-30T12:20:20.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:20.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80a18d0 con 0x7fd3a8131d90 2023-10-30T12:20:20.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:20.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3adf7c640 10 monclient(hunting): authenticate will time out at 2066.052833s 2023-10-30T12:20:20.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a6ffd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3a80a4b60 msgr2=0x7fd3a811ebb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:20.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a6ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3a80a4b60 0x7fd3a811ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:20.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8131d90 0x7fd3a811f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8131d90 0x7fd3a811f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52522/0 (socket says 172.21.15.50:52522) 2023-10-30T12:20:20.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a7fff640 1 -- 172.21.15.50:0/1722332377 learned_addr learned my addr 172.21.15.50:0/1722332377 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:20.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a7fff640 10 monclient(hunting): get_auth_request con 0x7fd3a8131d90 auth_method 0 2023-10-30T12:20:20.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.869+0000 7fd3a7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:20.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:20.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:20.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a77fe640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3a80a2350 0x7fd3a811e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a77fe640 10 monclient(hunting): get_auth_request con 0x7fd3a80a2350 auth_method 0 2023-10-30T12:20:20.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:20.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:20.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:20.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 10 monclient(hunting): handle_auth_done global_id 254141 payload 824 2023-10-30T12:20:20.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 1 -- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3a80a2350 msgr2=0x7fd3a811e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:20.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3a80a2350 0x7fd3a811e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 1 -- 172.21.15.50:0/1722332377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3a80a4b60 msgr2=0x7fd3a811ebb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:20.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3a80a4b60 0x7fd3a811ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:20.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:20:20.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 1 monclient: found mon.noname-c 2023-10-30T12:20:20.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:20.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 1 -- 172.21.15.50:0/1722332377 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c0094d0 con 0x7fd3a8131d90 2023-10-30T12:20:20.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a7fff640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8131d90 0x7fd3a811f0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009fd0 tx=0x7fd3a000d560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 client.?.objecter ms_handle_connect 0x7fd3a8131d90 2023-10-30T12:20:20.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:20.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 1 -- 172.21.15.50:0/1722332377 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a000dee0 con 0x7fd3a8131d90 2023-10-30T12:20:20.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:20.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: dump: 2023-10-30T12:20:20.923 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.924 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.924 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.924 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.925 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.925 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.925 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.926 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.926 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.927 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: _finish_auth 0 2023-10-30T12:20:20.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:20:20.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:20.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 1 -- 172.21.15.50:0/1722332377 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a000f040 con 0x7fd3a8131d90 2023-10-30T12:20:20.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 5 monclient: authenticate success, global_id 254141 2023-10-30T12:20:20.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 10 monclient: _renew_subs 2023-10-30T12:20:20.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a8135bf0 con 0x7fd3a8131d90 2023-10-30T12:20:20.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:20.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 1 -- 172.21.15.50:0/1722332377 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0003df0 con 0x7fd3a8131d90 2023-10-30T12:20:20.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:20.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:20.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd38ffff640 10 monclient: dump: 2023-10-30T12:20:20.932 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:20.932 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:20.932 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.933 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:20.933 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:20.934 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:20.934 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:20.934 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:20.934 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:20.935 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:20.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3a5ffb640 4 set_mon_vals no callback set 2023-10-30T12:20:20.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 10 client.254141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:20.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 10 monclient: _renew_subs 2023-10-30T12:20:20.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a811ff30 con 0x7fd3a8131d90 2023-10-30T12:20:20.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.870+0000 7fd3adf7c640 1 librados: init done 2023-10-30T12:20:20.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd38ffff640 1 -- 172.21.15.50:0/1722332377 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd3a0004030 con 0x7fd3a8131d90 2023-10-30T12:20:20.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd3adf7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd3adf7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd3adf7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:20.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a804e160 con 0x7fd3a8131d90 2023-10-30T12:20:20.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd38ffff640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3940392a0 0x7fd39403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:20.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd38ffff640 1 -- 172.21.15.50:0/1722332377 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(630..630 src has 490..630) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fd3a004b1f0 con 0x7fd3a8131d90 2023-10-30T12:20:20.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd38ffff640 10 client.254141.objecter ms_dispatch 0x7fd3a8000b90 osd_map(630..630 src has 490..630) v4 2023-10-30T12:20:20.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd38ffff640 3 client.254141.objecter handle_osd_map got epochs [630,630] > 0 2023-10-30T12:20:20.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd38ffff640 3 client.254141.objecter handle_osd_map decoding full epoch 630 2023-10-30T12:20:20.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd38ffff640 20 client.254141.objecter dump_active .. 0 homeless 2023-10-30T12:20:20.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd3a77fe640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3940392a0 0x7fd39403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:20.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.871+0000 7fd3a77fe640 10 monclient: get_auth_request con 0x7fd3940392a0 auth_method 0 2023-10-30T12:20:20.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.872+0000 7fd3a77fe640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3940392a0 0x7fd39403b760 secure :-1 s=READY pgs=4312 cs=0 l=1 rev1=1 crypto rx=0x7fd398003980 tx=0x7fd398008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:20.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.872+0000 7fd38ffff640 10 client.254141.objecter ms_handle_connect 0x7fd3940392a0 2023-10-30T12:20:20.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.876+0000 7fd38ffff640 1 -- 172.21.15.50:0/1722332377 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd3a00136c0 con 0x7fd3a8131d90 2023-10-30T12:20:20.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.876+0000 7fd38ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:20.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:20.876+0000 7fd38ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:21.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.120+0000 7fd3adf7c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:21.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.120+0000 7fd3adf7c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:21.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.120+0000 7fd3adf7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:21.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.120+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd3a80745c0 con 0x7fd3a8131d90 2023-10-30T12:20:21.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.120+0000 7fd38ffff640 1 -- 172.21.15.50:0/1722332377 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fd3a0012350 con 0x7fd3a8131d90 2023-10-30T12:20:21.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.120+0000 7fd38ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:21.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.120+0000 7fd38ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:21.124 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:21.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 10 librados: watch_flush enter 2023-10-30T12:20:21.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 10 librados: watch_flush exit 2023-10-30T12:20:21.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 20 client.254141.objecter shutdown clearing up homeless session... 2023-10-30T12:20:21.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 10 client.254141.objecter successfully canceled tick 2023-10-30T12:20:21.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3940392a0 msgr2=0x7fd39403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:21.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3940392a0 0x7fd39403b760 secure :-1 s=READY pgs=4312 cs=0 l=1 rev1=1 crypto rx=0x7fd398003980 tx=0x7fd398008040 comp rx=0 tx=0).stop 2023-10-30T12:20:21.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 10 monclient: shutdown 2023-10-30T12:20:21.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:21.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8131d90 msgr2=0x7fd3a811f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:21.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8131d90 0x7fd3a811f0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009fd0 tx=0x7fd3a000d560 comp rx=0 tx=0).stop 2023-10-30T12:20:21.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 shutdown_connections 2023-10-30T12:20:21.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd3940392a0 0x7fd39403b760 unknown :-1 s=CLOSED pgs=4312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:21.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3a8131d90 0x7fd3a811f0f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:21.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3a80a4b60 0x7fd3a811ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:21.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 --2- 172.21.15.50:0/1722332377 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd3a80a2350 0x7fd3a811e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:21.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 >> 172.21.15.50:0/1722332377 conn(0x7fd3a80a8060 msgr2=0x7fd3a8132510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:21.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 shutdown_connections 2023-10-30T12:20:21.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 -- 172.21.15.50:0/1722332377 wait complete. 2023-10-30T12:20:21.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:21.122+0000 7fd3adf7c640 1 librados: shutdown 2023-10-30T12:20:24.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:24.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.252+0000 7fe7686e7640 10 monclient: get_monmap_and_config 2023-10-30T12:20:24.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.252+0000 7fe7686e7640 10 monclient: build_initial_monmap 2023-10-30T12:20:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.252+0000 7fe7686e7640 10 monclient: monmap: 2023-10-30T12:20:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:24.253807+0000 2023-10-30T12:20:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:24.253807+0000 2023-10-30T12:20:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:24.257 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:24.258 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:24.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.253+0000 7fe7686e7640 1 Processor -- start 2023-10-30T12:20:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.253+0000 7fe7686e7640 1 -- start start 2023-10-30T12:20:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.253+0000 7fe7686e7640 10 monclient: init 2023-10-30T12:20:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-10-30T12:20:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe760098520 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:24.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe760099010 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe76009bf20 0x7fe76013b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient(hunting): picked mon.noname-a con 0x7fe76009bf20 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76009ad30 con 0x7fe760099010 2023-10-30T12:20:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76009b260 con 0x7fe760098520 2023-10-30T12:20:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe760072120 con 0x7fe76009bf20 2023-10-30T12:20:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe765c82640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fe760099010 legacy=0x7fe7601379f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe7686e7640 10 monclient(hunting): authenticate will time out at 2069.437172s 2023-10-30T12:20:24.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe766c84640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7fe76009bf20 msgr2=0x7fe76013b120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe766c84640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe76009bf20 0x7fe76013b120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe766483640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fe760098520 0x7fe760098900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:59262/0 (socket says 172.21.15.50:59262) 2023-10-30T12:20:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe766483640 1 -- 172.21.15.50:0/1571640928 learned_addr learned my addr 172.21.15.50:0/1571640928 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:20:24.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471072001 0 0) 0x7fe76009b260 con 0x7fe760098520 2023-10-30T12:20:24.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe765481640 10 monclient(hunting): _negotiate 2023-10-30T12:20:24.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe765481640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:24.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe765481640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:24.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe765481640 10 monclient(hunting): my global_id is 204685 2023-10-30T12:20:24.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.254+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c003520 con 0x7fe760098520 2023-10-30T12:20:24.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1651888277 0 0) 0x7fe74c003520 con 0x7fe760098520 2023-10-30T12:20:24.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 >> v2:172.21.15.50:3300/0 conn(0x7fe76009bf20 msgr2=0x7fe76013b120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:24.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 --2- 172.21.15.50:0/1571640928 >> v2:172.21.15.50:3300/0 conn(0x7fe76009bf20 0x7fe76013b120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 >> v1:172.21.15.50:6789/0 conn(0x7fe760099010 legacy=0x7fe7601379f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:24.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: _finish_hunting 0 2023-10-30T12:20:24.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:20:24.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:24.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:20:24.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760099610 con 0x7fe760098520 2023-10-30T12:20:24.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: _finish_auth 0 2023-10-30T12:20:24.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: _check_auth_tickets 2023-10-30T12:20:24.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:24.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fe75c0016c0 con 0x7fe760098520 2023-10-30T12:20:24.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe7686e7640 5 monclient: authenticate success, global_id 204685 2023-10-30T12:20:24.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:24.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:20:24.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: dump: 2023-10-30T12:20:24.279 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:24.279 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:24.279 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.279 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.279 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:24.280 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:24.280 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:24.280 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:24.280 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:24.280 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:24.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:20:24.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:24.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 >> v1:172.21.15.118:6789/0 conn(0x7fe760098520 legacy=0x7fe760098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:24.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:20:24.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe74c003970 0x7fe74c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): picked mon.b con 0x7fe74c003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:24.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe74c0047c0 0x7fe74c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): picked mon.c con 0x7fe74c0047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:24.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c005380 0x7fe74c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): picked mon.a con 0x7fe74c005380 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:24.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:24.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760072120 con 0x7fe74c005380 2023-10-30T12:20:24.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:24.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c00c220 con 0x7fe74c003970 2023-10-30T12:20:24.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:24.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c00c3e0 con 0x7fe74c0047c0 2023-10-30T12:20:24.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:24.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765481640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:20:24.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe7686e7640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:24.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766c84640 1 -- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c005380 msgr2=0x7fe74c00b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:24.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766c84640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c005380 0x7fe74c00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:24.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe74c0047c0 0x7fe74c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:24.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): get_auth_request con 0x7fe74c0047c0 auth_method 0 2023-10-30T12:20:24.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:24.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:24.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:24.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe74c003970 0x7fe74c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:24.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 10 monclient(hunting): get_auth_request con 0x7fe74c003970 auth_method 0 2023-10-30T12:20:24.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:24.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:24.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:20:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:24.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766483640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:20:24.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient(hunting): handle_auth_done global_id 204685 payload 293 2023-10-30T12:20:24.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 1 -- 172.21.15.50:0/1571640928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe74c003970 msgr2=0x7fe74c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:24.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe74c003970 0x7fe74c003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 1 -- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c005380 msgr2=0x7fe74c00b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:24.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c005380 0x7fe74c00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient: _finish_hunting 0 2023-10-30T12:20:24.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 1 monclient: found mon.c 2023-10-30T12:20:24.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:24.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:24.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe766c84640 1 -- 172.21.15.50:0/1571640928 reap_dead start 2023-10-30T12:20:24.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 1 -- 172.21.15.50:0/1571640928 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c003520 con 0x7fe74c0047c0 2023-10-30T12:20:24.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient: _finish_auth 0 2023-10-30T12:20:24.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 10 monclient: _check_auth_tickets 2023-10-30T12:20:24.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.255+0000 7fe765c82640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:24.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765c82640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe74c0047c0 0x7fe74c004c40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe750009900 tx=0x7fe74c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86868958bb2e529d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:24.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750003bb0 con 0x7fe74c0047c0 2023-10-30T12:20:24.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:24.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:24.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 10 monclient: dump: 2023-10-30T12:20:24.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:24.292 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:24.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:24.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:24.293 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:24.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:24.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:24.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:24.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe7686e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:24.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe750003d10 con 0x7fe74c0047c0 2023-10-30T12:20:24.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:24.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe7686e7640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:24.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 1 -- 172.21.15.50:0/1571640928 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7500052b0 con 0x7fe74c0047c0 2023-10-30T12:20:24.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe7686e7640 4 set_mon_vals no callback set 2023-10-30T12:20:24.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:24.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:24.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.256+0000 7fe765481640 10 monclient: dump: 2023-10-30T12:20:24.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:24.296 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:24.296 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.297 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.297 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:24.297 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:24.297 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:24.297 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:24.297 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:24.298 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:24.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 10 monclient: shutdown 2023-10-30T12:20:24.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:24.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 -- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe74c0047c0 msgr2=0x7fe74c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:24.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe74c0047c0 0x7fe74c004c40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe750009900 tx=0x7fe74c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 -- 172.21.15.50:0/1571640928 shutdown_connections 2023-10-30T12:20:24.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 --2- 172.21.15.50:0/1571640928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe74c0047c0 0x7fe74c004c40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 -- 172.21.15.50:0/1571640928 >> 172.21.15.50:0/1571640928 conn(0x7fe7600a05c0 msgr2=0x7fe76009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:24.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 -- 172.21.15.50:0/1571640928 shutdown_connections 2023-10-30T12:20:24.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 -- 172.21.15.50:0/1571640928 wait complete. 2023-10-30T12:20:24.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 10 monclient: build_initial_monmap 2023-10-30T12:20:24.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 10 monclient: monmap: 2023-10-30T12:20:24.300 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:24.300 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:24.300 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:24.300 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:24.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:24.301 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:24.301 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:24.301 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:24.301 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:24.302 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:24.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 librados: starting msgr at 2023-10-30T12:20:24.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.257+0000 7fe7686e7640 1 librados: starting objecter 2023-10-30T12:20:24.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 Processor -- start 2023-10-30T12:20:24.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 -- start start 2023-10-30T12:20:24.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 librados: setting wanted keys 2023-10-30T12:20:24.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 librados: calling monclient init 2023-10-30T12:20:24.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient: init 2023-10-30T12:20:24.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:24.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:20:24.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c0047c0 0x7fe7601c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): picked mon.noname-a con 0x7fe74c0047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:24.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7601b0780 0x7fe7601b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): picked mon.noname-b con 0x7fe7601b0780 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:24.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7601b89d0 0x7fe7601ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): picked mon.noname-c con 0x7fe7601b89d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:24.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:24.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601afc60 con 0x7fe74c0047c0 2023-10-30T12:20:24.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:24.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760070d80 con 0x7fe7601b0780 2023-10-30T12:20:24.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:24.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601b1670 con 0x7fe7601b89d0 2023-10-30T12:20:24.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:24.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe7686e7640 10 monclient(hunting): authenticate will time out at 2069.441604s 2023-10-30T12:20:24.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766483640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c0047c0 msgr2=0x7fe7601c7f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:24.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766483640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c0047c0 0x7fe7601c7f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:24.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7601b89d0 0x7fe7601ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:24.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7601b89d0 0x7fe7601ae390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52534/0 (socket says 172.21.15.50:52534) 2023-10-30T12:20:24.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 -- 172.21.15.50:0/467839038 learned_addr learned my addr 172.21.15.50:0/467839038 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:24.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): get_auth_request con 0x7fe7601b89d0 auth_method 0 2023-10-30T12:20:24.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:24.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient(hunting): handle_auth_done global_id 254144 payload 824 2023-10-30T12:20:24.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 -- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7601b0780 msgr2=0x7fe7601b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7601b0780 0x7fe7601b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 -- 172.21.15.50:0/467839038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c0047c0 msgr2=0x7fe7601c7f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c0047c0 0x7fe7601c7f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient: _finish_hunting 0 2023-10-30T12:20:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 monclient: found mon.noname-c 2023-10-30T12:20:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.258+0000 7fe766c84640 1 -- 172.21.15.50:0/467839038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7601b1950 con 0x7fe7601b89d0 2023-10-30T12:20:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe766c84640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7601b89d0 0x7fe7601ae390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe76006baf0 tx=0x7fe758009e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:24.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 client.?.objecter ms_handle_connect 0x7fe7601b89d0 2023-10-30T12:20:24.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:24.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 1 -- 172.21.15.50:0/467839038 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75800cd30 con 0x7fe7601b89d0 2023-10-30T12:20:24.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:24.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:24.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: dump: 2023-10-30T12:20:24.311 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:24.312 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:24.312 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.312 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.312 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:24.312 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:24.313 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:24.313 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:24.313 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:24.313 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:24.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: _finish_auth 0 2023-10-30T12:20:24.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:20:24.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:24.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 5 monclient: authenticate success, global_id 254144 2023-10-30T12:20:24.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 1 -- 172.21.15.50:0/467839038 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758015470 con 0x7fe7601b89d0 2023-10-30T12:20:24.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:24.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 1 -- 172.21.15.50:0/467839038 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758015820 con 0x7fe7601b89d0 2023-10-30T12:20:24.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:24.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:24.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe764c80640 4 set_mon_vals no callback set 2023-10-30T12:20:24.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe756ffd640 10 monclient: dump: 2023-10-30T12:20:24.315 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:24.317 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 10 monclient: _renew_subs 2023-10-30T12:20:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:24.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7601b8e50 con 0x7fe7601b89d0 2023-10-30T12:20:24.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 10 client.254144.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:24.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 10 monclient: _renew_subs 2023-10-30T12:20:24.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:24.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7601aeab0 con 0x7fe7601b89d0 2023-10-30T12:20:24.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.259+0000 7fe7686e7640 1 librados: init done 2023-10-30T12:20:24.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe756ffd640 1 -- 172.21.15.50:0/467839038 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe758012070 con 0x7fe7601b89d0 2023-10-30T12:20:24.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe7686e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:24.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe7686e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:24.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe7686e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:24.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76004e160 con 0x7fe7601b89d0 2023-10-30T12:20:24.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe756ffd640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe7440392a0 0x7fe74403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:24.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe756ffd640 1 -- 172.21.15.50:0/467839038 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(630..630 src has 490..630) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fe75804ac20 con 0x7fe7601b89d0 2023-10-30T12:20:24.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe756ffd640 10 client.254144.objecter ms_dispatch 0x7fe760000b90 osd_map(630..630 src has 490..630) v4 2023-10-30T12:20:24.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe756ffd640 3 client.254144.objecter handle_osd_map got epochs [630,630] > 0 2023-10-30T12:20:24.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe756ffd640 3 client.254144.objecter handle_osd_map decoding full epoch 630 2023-10-30T12:20:24.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe756ffd640 20 client.254144.objecter dump_active .. 0 homeless 2023-10-30T12:20:24.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe766483640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe7440392a0 0x7fe74403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:24.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.260+0000 7fe766483640 10 monclient: get_auth_request con 0x7fe7440392a0 auth_method 0 2023-10-30T12:20:24.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.261+0000 7fe766483640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe7440392a0 0x7fe74403b760 secure :-1 s=READY pgs=4313 cs=0 l=1 rev1=1 crypto rx=0x7fe760072c20 tx=0x7fe75c001ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:24.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.261+0000 7fe756ffd640 10 client.254144.objecter ms_handle_connect 0x7fe7440392a0 2023-10-30T12:20:24.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.265+0000 7fe756ffd640 1 -- 172.21.15.50:0/467839038 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe758005070 con 0x7fe7601b89d0 2023-10-30T12:20:24.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.265+0000 7fe756ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:24.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.265+0000 7fe756ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:24.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.505+0000 7fe7686e7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:24.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.505+0000 7fe7686e7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:24.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.505+0000 7fe7686e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:24.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.505+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe7600745c0 con 0x7fe7601b89d0 2023-10-30T12:20:24.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.506+0000 7fe756ffd640 1 -- 172.21.15.50:0/467839038 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7fe75801a070 con 0x7fe7601b89d0 2023-10-30T12:20:24.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.506+0000 7fe756ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:24.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.506+0000 7fe756ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:24.509 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.507+0000 7fe7686e7640 10 librados: watch_flush enter 2023-10-30T12:20:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.507+0000 7fe7686e7640 10 librados: watch_flush exit 2023-10-30T12:20:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.507+0000 7fe7686e7640 20 client.254144.objecter shutdown clearing up homeless session... 2023-10-30T12:20:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.507+0000 7fe7686e7640 10 client.254144.objecter successfully canceled tick 2023-10-30T12:20:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.507+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe7440392a0 msgr2=0x7fe74403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe7440392a0 0x7fe74403b760 secure :-1 s=READY pgs=4313 cs=0 l=1 rev1=1 crypto rx=0x7fe760072c20 tx=0x7fe75c001ba0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 10 monclient: shutdown 2023-10-30T12:20:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7601b89d0 msgr2=0x7fe7601ae390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7601b89d0 0x7fe7601ae390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe76006baf0 tx=0x7fe758009e10 comp rx=0 tx=0).stop 2023-10-30T12:20:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 shutdown_connections 2023-10-30T12:20:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fe7440392a0 0x7fe74403b760 unknown :-1 s=CLOSED pgs=4313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe7601b89d0 0x7fe7601ae390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7601b0780 0x7fe7601b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 --2- 172.21.15.50:0/467839038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe74c0047c0 0x7fe7601c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 >> 172.21.15.50:0/467839038 conn(0x7fe7600a05c0 msgr2=0x7fe760097410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 shutdown_connections 2023-10-30T12:20:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 -- 172.21.15.50:0/467839038 wait complete. 2023-10-30T12:20:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:24.508+0000 7fe7686e7640 1 librados: shutdown 2023-10-30T12:20:27.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.635+0000 7fa071d2c640 10 monclient: get_monmap_and_config 2023-10-30T12:20:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.635+0000 7fa071d2c640 10 monclient: build_initial_monmap 2023-10-30T12:20:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.636+0000 7fa071d2c640 10 monclient: monmap: 2023-10-30T12:20:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:27.637058+0000 2023-10-30T12:20:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:27.637058+0000 2023-10-30T12:20:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:27.640 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.636+0000 7fa071d2c640 1 Processor -- start 2023-10-30T12:20:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.636+0000 7fa071d2c640 1 -- start start 2023-10-30T12:20:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.636+0000 7fa071d2c640 10 monclient: init 2023-10-30T12:20:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-10-30T12:20:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa06c0659d0 0x7fa06c1334d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient(hunting): picked mon.noname-c con 0x7fa06c0659d0 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fa06c071410 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa06c071850 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06c09fb50 con 0x7fa06c071850 2023-10-30T12:20:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06c065db0 con 0x7fa06c0659d0 2023-10-30T12:20:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c09dd70 con 0x7fa06c071410 2023-10-30T12:20:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa071d2c640 10 monclient(hunting): authenticate will time out at 2072.820398s 2023-10-30T12:20:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06bfff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7fa06c071850 legacy=0x7fa06c09ed20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:27.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c09e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56064/0 (socket says 172.21.15.50:56064) 2023-10-30T12:20:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 -- 172.21.15.50:0/503687892 learned_addr learned my addr 172.21.15.50:0/503687892 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): get_auth_request con 0x7fa06c071410 auth_method 0 2023-10-30T12:20:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:27.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06b7fe640 1 --2- 172.21.15.50:0/503687892 >> v2:172.21.15.118:3300/0 conn(0x7fa06c0659d0 0x7fa06c1334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06b7fe640 10 monclient(hunting): get_auth_request con 0x7fa06c0659d0 auth_method 0 2023-10-30T12:20:27.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:27.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:27.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:27.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient(hunting): handle_auth_done global_id 254147 payload 274 2023-10-30T12:20:27.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 -- 172.21.15.50:0/503687892 >> v2:172.21.15.118:3300/0 conn(0x7fa06c0659d0 msgr2=0x7fa06c1334d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:27.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 --2- 172.21.15.50:0/503687892 >> v2:172.21.15.118:3300/0 conn(0x7fa06c0659d0 0x7fa06c1334d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 -- 172.21.15.50:0/503687892 >> v1:172.21.15.50:6789/0 conn(0x7fa06c071850 legacy=0x7fa06c09ed20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:27.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient: _finish_hunting 0 2023-10-30T12:20:27.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 monclient: found mon.noname-b 2023-10-30T12:20:27.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:27.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:20:27.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06affd640 1 -- 172.21.15.50:0/503687892 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c133b50 con 0x7fa06c071410 2023-10-30T12:20:27.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.637+0000 7fa06b7fe640 1 --2- 172.21.15.50:0/503687892 >> v2:172.21.15.118:3300/0 conn(0x7fa06c0659d0 0x7fa06c1334d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:20:27.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06affd640 1 --2- 172.21.15.50:0/503687892 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c09e7e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa060009870 tx=0x7fa06002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3d4723131dc230e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:27.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 1 -- 172.21.15.50:0/503687892 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0600317c0 con 0x7fa06c071410 2023-10-30T12:20:27.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:27.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:20:27.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: dump: 2023-10-30T12:20:27.657 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:27.657 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:27.657 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.657 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.658 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:27.658 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:27.658 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:27.658 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:27.658 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:27.659 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:20:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:20:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:27.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa071d2c640 5 monclient: authenticate success, global_id 254147 2023-10-30T12:20:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 1 -- 172.21.15.50:0/503687892 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060031920 con 0x7fa06c071410 2023-10-30T12:20:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa071d2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa071d2c640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa071d2c640 4 set_mon_vals no callback set 2023-10-30T12:20:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 1 -- 172.21.15.50:0/503687892 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060031cb0 con 0x7fa06c071410 2023-10-30T12:20:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.638+0000 7fa06a7fc640 10 monclient: dump: 2023-10-30T12:20:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:27.662 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:27.663 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:27.663 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:27.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 10 monclient: shutdown 2023-10-30T12:20:27.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:27.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 -- 172.21.15.50:0/503687892 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 msgr2=0x7fa06c09e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:27.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 --2- 172.21.15.50:0/503687892 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c09e7e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa060009870 tx=0x7fa06002b860 comp rx=0 tx=0).stop 2023-10-30T12:20:27.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 -- 172.21.15.50:0/503687892 shutdown_connections 2023-10-30T12:20:27.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 --2- 172.21.15.50:0/503687892 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c09e7e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 --2- 172.21.15.50:0/503687892 >> v2:172.21.15.118:3300/0 conn(0x7fa06c0659d0 0x7fa06c1334d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 -- 172.21.15.50:0/503687892 >> 172.21.15.50:0/503687892 conn(0x7fa06c0a47f0 msgr2=0x7fa06c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:27.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 -- 172.21.15.50:0/503687892 shutdown_connections 2023-10-30T12:20:27.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.639+0000 7fa071d2c640 1 -- 172.21.15.50:0/503687892 wait complete. 2023-10-30T12:20:27.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 10 monclient: build_initial_monmap 2023-10-30T12:20:27.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 10 monclient: monmap: 2023-10-30T12:20:27.665 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:27.665 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:27.666 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:27.666 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:27.666 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:27.666 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:27.666 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:27.667 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:27.667 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:27.667 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:27.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 1 librados: starting msgr at 2023-10-30T12:20:27.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 1 librados: starting objecter 2023-10-30T12:20:27.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 1 Processor -- start 2023-10-30T12:20:27.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 1 -- start start 2023-10-30T12:20:27.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 1 librados: setting wanted keys 2023-10-30T12:20:27.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 1 librados: calling monclient init 2023-10-30T12:20:27.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.640+0000 7fa071d2c640 10 monclient: init 2023-10-30T12:20:27.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:27.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:20:27.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa06c0659d0 0x7fa06c1c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:27.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fa06c0659d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:27.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c06da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:27.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): picked mon.noname-c con 0x7fa06c071410 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:27.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa06c071850 0x7fa06c06df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:27.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): picked mon.noname-a con 0x7fa06c071850 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:27.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:27.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c070170 con 0x7fa06c071850 2023-10-30T12:20:27.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:27.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c11a8f0 con 0x7fa06c0659d0 2023-10-30T12:20:27.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:27.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c11ab40 con 0x7fa06c071410 2023-10-30T12:20:27.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:27.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa071d2c640 10 monclient(hunting): authenticate will time out at 2072.824091s 2023-10-30T12:20:27.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06bfff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa06c071850 msgr2=0x7fa06c06df80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:27.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa06c071850 0x7fa06c06df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:27.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c06da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:27.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c06da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56072/0 (socket says 172.21.15.50:56072) 2023-10-30T12:20:27.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 -- 172.21.15.50:0/2978055524 learned_addr learned my addr 172.21.15.50:0/2978055524 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:27.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): get_auth_request con 0x7fa06c071410 auth_method 0 2023-10-30T12:20:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa06c0659d0 0x7fa06c1c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 10 monclient(hunting): get_auth_request con 0x7fa06c0659d0 auth_method 0 2023-10-30T12:20:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient(hunting): handle_auth_done global_id 254150 payload 824 2023-10-30T12:20:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 -- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa06c0659d0 msgr2=0x7fa06c1c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa06c0659d0 0x7fa06c1c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 -- 172.21.15.50:0/2978055524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa06c071850 msgr2=0x7fa06c06df80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa06c071850 0x7fa06c06df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient: _finish_hunting 0 2023-10-30T12:20:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 monclient: found mon.noname-c 2023-10-30T12:20:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 -- 172.21.15.50:0/2978055524 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0600094d0 con 0x7fa06c071410 2023-10-30T12:20:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa06affd640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c06da40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa060000c00 tx=0x7fa0600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa053fff640 10 client.?.objecter ms_handle_connect 0x7fa06c071410 2023-10-30T12:20:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa053fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa053fff640 1 -- 172.21.15.50:0/2978055524 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060032860 con 0x7fa06c071410 2023-10-30T12:20:27.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:27.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa053fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:27.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.641+0000 7fa053fff640 10 monclient: dump: 2023-10-30T12:20:27.678 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:27.678 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:27.680 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 10 monclient: _finish_auth 0 2023-10-30T12:20:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 10 monclient: _check_auth_tickets 2023-10-30T12:20:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 1 -- 172.21.15.50:0/2978055524 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0600329c0 con 0x7fa06c071410 2023-10-30T12:20:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 5 monclient: authenticate success, global_id 254150 2023-10-30T12:20:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 10 monclient: _renew_subs 2023-10-30T12:20:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06c06fc60 con 0x7fa06c071410 2023-10-30T12:20:27.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:27.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 1 -- 172.21.15.50:0/2978055524 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060032d70 con 0x7fa06c071410 2023-10-30T12:20:27.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:27.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:27.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 10 monclient: dump: 2023-10-30T12:20:27.683 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:27.683 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:27.683 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.683 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:27.683 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:27.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:27.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:27.684 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:27.684 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:27.684 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:27.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 10 client.254150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:27.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa069ffb640 4 set_mon_vals no callback set 2023-10-30T12:20:27.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 10 monclient: _renew_subs 2023-10-30T12:20:27.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:27.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06c09e100 con 0x7fa06c071410 2023-10-30T12:20:27.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa071d2c640 1 librados: init done 2023-10-30T12:20:27.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.642+0000 7fa053fff640 1 -- 172.21.15.50:0/2978055524 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa06004e450 con 0x7fa06c071410 2023-10-30T12:20:27.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa071d2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:27.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa071d2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:27.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa071d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:27.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa053fff640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa0580392d0 0x7fa05803b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:27.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06c11ac80 con 0x7fa06c071410 2023-10-30T12:20:27.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa053fff640 1 -- 172.21.15.50:0/2978055524 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(630..630 src has 490..630) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fa06006cf90 con 0x7fa06c071410 2023-10-30T12:20:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa053fff640 10 client.254150.objecter ms_dispatch 0x7fa06c000b90 osd_map(630..630 src has 490..630) v4 2023-10-30T12:20:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa053fff640 3 client.254150.objecter handle_osd_map got epochs [630,630] > 0 2023-10-30T12:20:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa053fff640 3 client.254150.objecter handle_osd_map decoding full epoch 630 2023-10-30T12:20:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa053fff640 20 client.254150.objecter dump_active .. 0 homeless 2023-10-30T12:20:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa06b7fe640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa0580392d0 0x7fa05803b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa06b7fe640 10 monclient: get_auth_request con 0x7fa0580392d0 auth_method 0 2023-10-30T12:20:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa053fff640 10 client.254150.objecter ms_handle_connect 0x7fa0580392d0 2023-10-30T12:20:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.643+0000 7fa06b7fe640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa0580392d0 0x7fa05803b790 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 crypto rx=0x7fa06c06f380 tx=0x7fa05c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.647+0000 7fa053fff640 1 -- 172.21.15.50:0/2978055524 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa06c11ac80 con 0x7fa06c071410 2023-10-30T12:20:27.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.647+0000 7fa053fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:27.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.647+0000 7fa053fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:27.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.892+0000 7fa071d2c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:27.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.892+0000 7fa071d2c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:27.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.892+0000 7fa071d2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:27.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.892+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa06c11ac80 con 0x7fa06c071410 2023-10-30T12:20:27.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.892+0000 7fa053fff640 1 -- 172.21.15.50:0/2978055524 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1545 (secure 0 0 0) 0x7fa06c11ac80 con 0x7fa06c071410 2023-10-30T12:20:27.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.892+0000 7fa053fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:27.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.892+0000 7fa053fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:27.896 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":204,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:27.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 10 librados: watch_flush enter 2023-10-30T12:20:27.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 10 librados: watch_flush exit 2023-10-30T12:20:27.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 20 client.254150.objecter shutdown clearing up homeless session... 2023-10-30T12:20:27.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 10 client.254150.objecter successfully canceled tick 2023-10-30T12:20:27.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa0580392d0 msgr2=0x7fa05803b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:27.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa0580392d0 0x7fa05803b790 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 crypto rx=0x7fa06c06f380 tx=0x7fa05c008040 comp rx=0 tx=0).stop 2023-10-30T12:20:27.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 10 monclient: shutdown 2023-10-30T12:20:27.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:27.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 msgr2=0x7fa06c06da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:27.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c06da40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa060000c00 tx=0x7fa0600046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 shutdown_connections 2023-10-30T12:20:27.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa0580392d0 0x7fa05803b790 unknown :-1 s=CLOSED pgs=4314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa06c071850 0x7fa06c06df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa06c071410 0x7fa06c06da40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 --2- 172.21.15.50:0/2978055524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa06c0659d0 0x7fa06c1c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:27.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 >> 172.21.15.50:0/2978055524 conn(0x7fa06c0a47f0 msgr2=0x7fa06c0a1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:27.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 shutdown_connections 2023-10-30T12:20:27.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 -- 172.21.15.50:0/2978055524 wait complete. 2023-10-30T12:20:27.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:27.894+0000 7fa071d2c640 1 librados: shutdown 2023-10-30T12:20:30.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:31.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.023+0000 7f875d452640 10 monclient: get_monmap_and_config 2023-10-30T12:20:31.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.023+0000 7f875d452640 10 monclient: build_initial_monmap 2023-10-30T12:20:31.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.023+0000 7f875d452640 10 monclient: monmap: 2023-10-30T12:20:31.025 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:31.025 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:31.026 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:31.024470+0000 2023-10-30T12:20:31.026 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:31.024470+0000 2023-10-30T12:20:31.026 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:31.026 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:31.027 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:31.027 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:31.027 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:31.027 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:31.027 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:31.027 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:31.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 1 Processor -- start 2023-10-30T12:20:31.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 1 -- start start 2023-10-30T12:20:31.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient: init 2023-10-30T12:20:31.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:31.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-10-30T12:20:31.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:31.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient(hunting): picked mon.noname-c con 0x7f87580a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:31.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8758099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:31.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:31.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient(hunting): picked mon.noname-b con 0x7f8758132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:31.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87580a4f40 con 0x7f8758099a40 2023-10-30T12:20:31.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:31.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87580a3d70 con 0x7f87580a4b60 2023-10-30T12:20:31.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:31.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87580a4150 con 0x7f8758132be0 2023-10-30T12:20:31.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:31.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f875d452640 10 monclient(hunting): authenticate will time out at 2076.207748s 2023-10-30T12:20:31.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f87577fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:31.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f87577fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56078/0 (socket says 172.21.15.50:56078) 2023-10-30T12:20:31.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 learned_addr learned my addr 172.21.15.50:0/2318551355 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:31.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f87577fe640 10 monclient(hunting): get_auth_request con 0x7f8758132be0 auth_method 0 2023-10-30T12:20:31.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f87577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:31.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f87577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:31.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f87577fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:31.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:31.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f8756ffd640 10 monclient(hunting): get_auth_request con 0x7f87580a4b60 auth_method 0 2023-10-30T12:20:31.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f8756ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:31.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f8756ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:31.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.024+0000 7f8756ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:31.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:20:31.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:31.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:31.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:31.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:31.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:31.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:31.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.025+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:31.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:31.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 10 monclient(hunting): get_auth_request con 0x7f8758132be0 auth_method 0 2023-10-30T12:20:31.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:31.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:31.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:31.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:20:31.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:31.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:31.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:20:31.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:31.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 10 monclient(hunting): get_auth_request con 0x7f87580a4b60 auth_method 0 2023-10-30T12:20:31.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:31.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:31.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:31.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:31.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:31.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:31.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.226+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:20:31.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:31.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 10 monclient(hunting): get_auth_request con 0x7f8758132be0 auth_method 0 2023-10-30T12:20:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:20:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 10 monclient(hunting): get_auth_request con 0x7f87580a4b60 auth_method 0 2023-10-30T12:20:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:20:31.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:31.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:31.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:31.627+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:20:32.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 10 monclient(hunting): get_auth_request con 0x7f8758132be0 auth_method 0 2023-10-30T12:20:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f87577fe640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:20:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f8756ffd640 10 monclient(hunting): get_auth_request con 0x7f87580a4b60 auth_method 0 2023-10-30T12:20:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f8756ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f8756ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.428+0000 7f8756ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.429+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.429+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.429+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:32.429+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:20:34.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient(hunting): tick 2023-10-30T12:20:34.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:20:34.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:20:34.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:20:34.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 msgr2=0x7f87580a2a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:34.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8758132be0 0x7f87580a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:34.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 msgr2=0x7f87581326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:34.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f87581326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:34.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.118:6789/0 conn(0x7f8758099a40 legacy=0x7f87580a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:34.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient: _add_conns ranks=[3,0] 2023-10-30T12:20:34.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87440036e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8744006230 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875809a440 con 0x7f8744006230 2023-10-30T12:20:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87580a4150 con 0x7f87440036e0 2023-10-30T12:20:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8754ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 msgr2=0x7f8744005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f87567fc640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.50:6789/0 conn(0x7f8744006230 legacy=0x7f87440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.025+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:34.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.226+0000 7f87567fc640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.50:6789/0 conn(0x7f8744006230 legacy=0x7f87440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:34.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.226+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 msgr2=0x7f8744005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:34.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.226+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:20:34.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.627+0000 7f87567fc640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.50:6789/0 conn(0x7f8744006230 legacy=0x7f87440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:34.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.627+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 msgr2=0x7f8744005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:34.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:34.627+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:20:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:35.428+0000 7f87567fc640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.50:6789/0 conn(0x7f8744006230 legacy=0x7f87440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:35.428+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 msgr2=0x7f8744005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:35.428+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:20:37.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): tick 2023-10-30T12:20:37.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:20:37.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 monclient(hunting): continuing hunt 2023-10-30T12:20:37.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:20:37.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 msgr2=0x7f8744005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:37.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.50:6789/0 conn(0x7f8744006230 legacy=0x7f87440086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:37.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-10-30T12:20:37.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 reap_dead start 2023-10-30T12:20:37.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f8744009f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f87580a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:37.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87440036e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:37.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8744006230 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:37.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874400b040 con 0x7f8744006230 2023-10-30T12:20:37.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87580a4150 con 0x7f87440036e0 2023-10-30T12:20:37.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 1 -- 172.21.15.50:0/2318551355 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874400a580 con 0x7f87580a4b60 2023-10-30T12:20:37.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8754ff9640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:37.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f87567fc640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 msgr2=0x7f8744005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:37.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f87577fe640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.50:6789/0 conn(0x7f8744006230 legacy=0x7f87440086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:37.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f87567fc640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:37.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.118:3300/0 conn(0x7f87580a4b60 0x7f8744009f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): get_auth_request con 0x7f87580a4b60 auth_method 0 2023-10-30T12:20:37.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 10 monclient(hunting): handle_auth_done global_id 204694 payload 274 2023-10-30T12:20:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 msgr2=0x7f8744005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.025+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 >> v1:172.21.15.50:6789/0 conn(0x7f8744006230 legacy=0x7f87440086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.026+0000 7f8756ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:20:37.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.026+0000 7f8756ffd640 1 monclient: found mon.noname-c 2023-10-30T12:20:37.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.026+0000 7f8756ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:37.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.026+0000 7f8756ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:20:37.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.026+0000 7f8756ffd640 1 -- 172.21.15.50:0/2318551355 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875809a440 con 0x7f87580a4b60 2023-10-30T12:20:37.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.026+0000 7f8756ffd640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87580a4b60 0x7f8744009f80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f875806af70 tx=0x7f874802c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99af1ef089766c34 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 1 -- 172.21.15.50:0/2318551355 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748035410 con 0x7f87580a4b60 2023-10-30T12:20:37.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:37.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: dump: 2023-10-30T12:20:37.039 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.039 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.039 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.039 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.040 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.040 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.040 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: _finish_auth 0 2023-10-30T12:20:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:20:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 1 -- 172.21.15.50:0/2318551355 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874802f030 con 0x7f87580a4b60 2023-10-30T12:20:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f875d452640 5 monclient: authenticate success, global_id 204694 2023-10-30T12:20:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f875d452640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:37.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:37.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f875d452640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:37.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 1 -- 172.21.15.50:0/2318551355 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87480357b0 con 0x7f87580a4b60 2023-10-30T12:20:37.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f875d452640 4 set_mon_vals no callback set 2023-10-30T12:20:37.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:37.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f8755ffb640 10 monclient: dump: 2023-10-30T12:20:37.043 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.043 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.043 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.044 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.044 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.044 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.045 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.045 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.045 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.046 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f875d452640 10 monclient: shutdown 2023-10-30T12:20:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f875d452640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.027+0000 7f875d452640 1 -- 172.21.15.50:0/2318551355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87580a4b60 msgr2=0x7f8744009f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87580a4b60 0x7f8744009f80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f875806af70 tx=0x7f874802c450 comp rx=0 tx=0).stop 2023-10-30T12:20:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 1 -- 172.21.15.50:0/2318551355 shutdown_connections 2023-10-30T12:20:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 1 --2- 172.21.15.50:0/2318551355 >> v2:172.21.15.50:3300/0 conn(0x7f87440036e0 0x7f8744005b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 1 --2- 172.21.15.50:0/2318551355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f87580a4b60 0x7f8744009f80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 1 -- 172.21.15.50:0/2318551355 >> 172.21.15.50:0/2318551355 conn(0x7f87580a8060 msgr2=0x7f87580a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 1 -- 172.21.15.50:0/2318551355 shutdown_connections 2023-10-30T12:20:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 1 -- 172.21.15.50:0/2318551355 wait complete. 2023-10-30T12:20:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 10 monclient: build_initial_monmap 2023-10-30T12:20:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.028+0000 7f875d452640 10 monclient: monmap: 2023-10-30T12:20:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:37.059 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.029+0000 7f875d452640 1 librados: starting msgr at 2023-10-30T12:20:37.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.029+0000 7f875d452640 1 librados: starting objecter 2023-10-30T12:20:37.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.029+0000 7f875d452640 1 Processor -- start 2023-10-30T12:20:37.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.029+0000 7f875d452640 1 -- start start 2023-10-30T12:20:37.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.029+0000 7f875d452640 1 librados: setting wanted keys 2023-10-30T12:20:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.029+0000 7f875d452640 1 librados: calling monclient init 2023-10-30T12:20:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.029+0000 7f875d452640 10 monclient: init 2023-10-30T12:20:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:20:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87440036e0 0x7f875811dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): picked mon.noname-a con 0x7f87440036e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8744006230 0x7f875811e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): picked mon.noname-b con 0x7f8744006230 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87580a4b60 0x7f875811ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): picked mon.noname-c con 0x7f87580a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758120e80 con 0x7f87440036e0 2023-10-30T12:20:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758121520 con 0x7f8744006230 2023-10-30T12:20:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8758120000 con 0x7f87580a4b60 2023-10-30T12:20:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f875d452640 10 monclient(hunting): authenticate will time out at 2082.213320s 2023-10-30T12:20:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f8756ffd640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87440036e0 msgr2=0x7f875811dfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f8756ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87440036e0 0x7f875811dfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:37.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8744006230 0x7f875811e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87580a4b60 0x7f875811ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87580a4b60 0x7f875811ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56124/0 (socket says 172.21.15.50:56124) 2023-10-30T12:20:37.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8744006230 0x7f875811e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.50:37766/0 (socket says 172.21.15.50:37766) 2023-10-30T12:20:37.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 1 -- 172.21.15.50:0/186560358 learned_addr learned my addr 172.21.15.50:0/186560358 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:37.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 10 monclient(hunting): get_auth_request con 0x7f87580a4b60 auth_method 0 2023-10-30T12:20:37.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 10 monclient(hunting): get_auth_request con 0x7f8744006230 auth_method 0 2023-10-30T12:20:37.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.030+0000 7f87567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 10 monclient(hunting): handle_auth_done global_id 254153 payload 824 2023-10-30T12:20:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 1 -- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8744006230 msgr2=0x7f875811e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8744006230 0x7f875811e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 1 -- 172.21.15.50:0/186560358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87440036e0 msgr2=0x7f875811dfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87440036e0 0x7f875811dfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 10 monclient: _finish_hunting 0 2023-10-30T12:20:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 1 monclient: found mon.noname-c 2023-10-30T12:20:37.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:37.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 1 -- 172.21.15.50:0/186560358 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874800e040 con 0x7f87580a4b60 2023-10-30T12:20:37.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87577fe640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87580a4b60 0x7f875811ea40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8758099f00 tx=0x7f8750012db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 client.?.objecter ms_handle_connect 0x7f87580a4b60 2023-10-30T12:20:37.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:37.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 1 -- 172.21.15.50:0/186560358 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875000b5a0 con 0x7f87580a4b60 2023-10-30T12:20:37.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:37.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: dump: 2023-10-30T12:20:37.070 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.071 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.071 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.071 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.071 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.071 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.072 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:20:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:20:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 1 -- 172.21.15.50:0/186560358 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875000f040 con 0x7f87580a4b60 2023-10-30T12:20:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f875d452640 5 monclient: authenticate success, global_id 254153 2023-10-30T12:20:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:37.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 1 -- 172.21.15.50:0/186560358 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87500079e0 con 0x7f87580a4b60 2023-10-30T12:20:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f873f7fe640 10 monclient: dump: 2023-10-30T12:20:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.074 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.075 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.075 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.075 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.075 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.075 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.076 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.076 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.076 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f875d452640 10 monclient: _renew_subs 2023-10-30T12:20:37.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f875d452640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87580a18d0 con 0x7f87580a4b60 2023-10-30T12:20:37.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.031+0000 7f87557fa640 4 set_mon_vals no callback set 2023-10-30T12:20:37.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.032+0000 7f875d452640 10 client.254153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:37.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.032+0000 7f875d452640 10 monclient: _renew_subs 2023-10-30T12:20:37.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.032+0000 7f875d452640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.032+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875811f350 con 0x7f87580a4b60 2023-10-30T12:20:37.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.032+0000 7f875d452640 1 librados: init done 2023-10-30T12:20:37.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.032+0000 7f873f7fe640 1 -- 172.21.15.50:0/186560358 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8750007b40 con 0x7f87580a4b60 2023-10-30T12:20:37.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f873f7fe640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f874403a130 0x7f874403c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f873f7fe640 1 -- 172.21.15.50:0/186560358 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(632..632 src has 520..632) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f875004b560 con 0x7f87580a4b60 2023-10-30T12:20:37.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f873f7fe640 10 client.254153.objecter ms_dispatch 0x7f8758000b90 osd_map(632..632 src has 520..632) v4 2023-10-30T12:20:37.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f873f7fe640 3 client.254153.objecter handle_osd_map got epochs [632,632] > 0 2023-10-30T12:20:37.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f873f7fe640 3 client.254153.objecter handle_osd_map decoding full epoch 632 2023-10-30T12:20:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f873f7fe640 20 client.254153.objecter dump_active .. 0 homeless 2023-10-30T12:20:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f8756ffd640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f874403a130 0x7f874403c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f8756ffd640 10 monclient: get_auth_request con 0x7f874403a130 auth_method 0 2023-10-30T12:20:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f875d452640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f875d452640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f875d452640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f873f7fe640 10 client.254153.objecter ms_handle_connect 0x7f874403a130 2023-10-30T12:20:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f8756ffd640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f874403a130 0x7f874403c5f0 secure :-1 s=READY pgs=4315 cs=0 l=1 rev1=1 crypto rx=0x7f8748037000 tx=0x7f8748030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.033+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875804e160 con 0x7f87580a4b60 2023-10-30T12:20:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.038+0000 7f873f7fe640 1 -- 172.21.15.50:0/186560358 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8750014320 con 0x7f87580a4b60 2023-10-30T12:20:37.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.038+0000 7f873f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.038+0000 7f873f7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.292+0000 7f875d452640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.292+0000 7f875d452640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.292+0000 7f875d452640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.292+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f875809a7b0 con 0x7f87580a4b60 2023-10-30T12:20:37.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.292+0000 7f873f7fe640 1 -- 172.21.15.50:0/186560358 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f875001eba0 con 0x7f87580a4b60 2023-10-30T12:20:37.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.292+0000 7f873f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:37.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.292+0000 7f873f7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:37.295 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":206,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:37.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 10 librados: watch_flush enter 2023-10-30T12:20:37.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 10 librados: watch_flush exit 2023-10-30T12:20:37.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 20 client.254153.objecter shutdown clearing up homeless session... 2023-10-30T12:20:37.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 10 client.254153.objecter successfully canceled tick 2023-10-30T12:20:37.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f874403a130 msgr2=0x7f874403c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f874403a130 0x7f874403c5f0 secure :-1 s=READY pgs=4315 cs=0 l=1 rev1=1 crypto rx=0x7f8748037000 tx=0x7f8748030000 comp rx=0 tx=0).stop 2023-10-30T12:20:37.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 10 monclient: shutdown 2023-10-30T12:20:37.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:37.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87580a4b60 msgr2=0x7f875811ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87580a4b60 0x7f875811ea40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8758099f00 tx=0x7f8750012db0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 shutdown_connections 2023-10-30T12:20:37.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f874403a130 0x7f874403c5f0 unknown :-1 s=CLOSED pgs=4315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87580a4b60 0x7f875811ea40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8744006230 0x7f875811e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 --2- 172.21.15.50:0/186560358 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87440036e0 0x7f875811dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.294+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 >> 172.21.15.50:0/186560358 conn(0x7f87580a8060 msgr2=0x7f87580a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:37.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.295+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 shutdown_connections 2023-10-30T12:20:37.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.295+0000 7f875d452640 1 -- 172.21.15.50:0/186560358 wait complete. 2023-10-30T12:20:37.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.295+0000 7f875d452640 1 librados: shutdown 2023-10-30T12:20:37.307 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:20:37.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:20:37.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:20:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.412+0000 7f1b430e7640 10 monclient: get_monmap_and_config 2023-10-30T12:20:37.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.412+0000 7f1b430e7640 10 monclient: build_initial_monmap 2023-10-30T12:20:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.412+0000 7f1b430e7640 10 monclient: monmap: 2023-10-30T12:20:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:37.413094+0000 2023-10-30T12:20:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:37.413094+0000 2023-10-30T12:20:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:37.416 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:37.416 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:37.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.412+0000 7f1b430e7640 1 Processor -- start 2023-10-30T12:20:37.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.412+0000 7f1b430e7640 1 -- start start 2023-10-30T12:20:37.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.412+0000 7f1b430e7640 10 monclient: init 2023-10-30T12:20:37.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:37.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-10-30T12:20:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b3c0a2ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1b3c0a2450 0x7f1b3c13aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c0a2450 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b3c13af40 addr v1:172.21.15.50:6789/0 2023-10-30T12:20:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0a4190 con 0x7f1b3c13af40 2023-10-30T12:20:37.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c09a2e0 con 0x7f1b3c0a2ae0 2023-10-30T12:20:37.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c0a1630 con 0x7f1b3c0a2450 2023-10-30T12:20:37.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:37.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b430e7640 10 monclient(hunting): authenticate will time out at 2082.596247s 2023-10-30T12:20:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b41684640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f1b3c13af40 legacy=0x7f1b3c13d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2023-10-30T12:20:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b33fff640 1 -- >> v2:172.21.15.50:3300/0 conn(0x7f1b3c0a2450 msgr2=0x7f1b3c13aa00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b33fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1b3c0a2450 0x7f1b3c13aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b40e83640 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f1b3c0a2ae0 0x7f1b3c0a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.50:34302/0 (socket says 172.21.15.50:34302) 2023-10-30T12:20:37.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b40e83640 1 -- 172.21.15.50:0/137659997 learned_addr learned my addr 172.21.15.50:0/137659997 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:20:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993810962 0 0) 0x7f1b3c09a2e0 con 0x7f1b3c0a2ae0 2023-10-30T12:20:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b337fe640 10 monclient(hunting): _negotiate 2023-10-30T12:20:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b337fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b337fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b337fe640 10 monclient(hunting): my global_id is 204700 2023-10-30T12:20:37.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.413+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24003520 con 0x7f1b3c0a2ae0 2023-10-30T12:20:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 <== mon.1 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1123990191 0 0) 0x7f1b24003520 con 0x7f1b3c0a2ae0 2023-10-30T12:20:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 >> v2:172.21.15.50:3300/0 conn(0x7f1b3c0a2450 msgr2=0x7f1b3c13aa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 --2- 172.21.15.50:0/137659997 >> v2:172.21.15.50:3300/0 conn(0x7f1b3c0a2450 0x7f1b3c13aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 >> v1:172.21.15.50:6789/0 conn(0x7f1b3c13af40 legacy=0x7f1b3c13d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:37.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: _finish_hunting 0 2023-10-30T12:20:37.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 monclient: found mon.noname-c-legacy 2023-10-30T12:20:37.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:37.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.118:6789/0 2023-10-30T12:20:37.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c09a8a0 con 0x7f1b3c0a2ae0 2023-10-30T12:20:37.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: _finish_auth 0 2023-10-30T12:20:37.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: _check_auth_tickets 2023-10-30T12:20:37.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:37.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 <== mon.1 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f1b38001670 con 0x7f1b3c0a2ae0 2023-10-30T12:20:37.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-10-30T12:20:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: dump: 2023-10-30T12:20:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.438 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.438 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.438 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.438 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.438 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] but i'm connected to v1:172.21.15.118:6789/0, reconnecting 2023-10-30T12:20:37.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:37.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 >> v1:172.21.15.118:6789/0 conn(0x7f1b3c0a2ae0 legacy=0x7f1b3c0a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:20:37.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b24003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): picked mon.c con 0x7f1b24003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): picked mon.b con 0x7f1b240047c0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b24005380 0x7f1b2400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): picked mon.a con 0x7f1b24005380 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0a1630 con 0x7f1b24005380 2023-10-30T12:20:37.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2400c220 con 0x7f1b240047c0 2023-10-30T12:20:37.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2400c3e0 con 0x7f1b24003970 2023-10-30T12:20:37.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:37.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b337fe640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:20:37.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b41684640 1 -- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b24005380 msgr2=0x7f1b2400b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:37.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b41684640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b24005380 0x7f1b2400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:37.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b24003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): get_auth_request con 0x7f1b24003970 auth_method 0 2023-10-30T12:20:37.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:37.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:20:37.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 10 monclient(hunting): get_auth_request con 0x7f1b240047c0 auth_method 0 2023-10-30T12:20:37.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:20:37.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:20:37.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient(hunting): handle_auth_done global_id 204700 payload 293 2023-10-30T12:20:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 1 -- 172.21.15.50:0/137659997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240047c0 msgr2=0x7f1b24004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 1 -- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b24005380 msgr2=0x7f1b2400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b24005380 0x7f1b2400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient: _finish_hunting 0 2023-10-30T12:20:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 1 monclient: found mon.c 2023-10-30T12:20:37.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:37.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b41684640 1 -- 172.21.15.50:0/137659997 reap_dead start 2023-10-30T12:20:37.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 1 -- 172.21.15.50:0/137659997 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b24003520 con 0x7f1b24003970 2023-10-30T12:20:37.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient: _finish_auth 0 2023-10-30T12:20:37.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 10 monclient: _check_auth_tickets 2023-10-30T12:20:37.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b40e83640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:37.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b430e7640 5 monclient: authenticate success, global_id 204700 2023-10-30T12:20:37.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.414+0000 7f1b430e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:37.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b40e83640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b24003d70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b3800c210 tx=0x7f1b240057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de032bb7cc4d3f68 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b38005ad0 con 0x7f1b24003970 2023-10-30T12:20:37.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 10 monclient: dump: 2023-10-30T12:20:37.449 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.450 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.450 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.450 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.450 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.450 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.450 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.451 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.451 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.451 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b38005c30 con 0x7f1b24003970 2023-10-30T12:20:37.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b430e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:37.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:37.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 1 -- 172.21.15.50:0/137659997 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3802c870 con 0x7f1b24003970 2023-10-30T12:20:37.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b430e7640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:37.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b430e7640 4 set_mon_vals no callback set 2023-10-30T12:20:37.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.415+0000 7f1b337fe640 10 monclient: dump: 2023-10-30T12:20:37.453 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.453 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.453 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.453 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.454 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.454 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.454 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.454 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.454 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.455 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 10 monclient: shutdown 2023-10-30T12:20:37.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:37.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 -- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 msgr2=0x7f1b24003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b24003d70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b3800c210 tx=0x7f1b240057f0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 -- 172.21.15.50:0/137659997 shutdown_connections 2023-10-30T12:20:37.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 --2- 172.21.15.50:0/137659997 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b24003d70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 -- 172.21.15.50:0/137659997 >> 172.21.15.50:0/137659997 conn(0x7f1b3c0a80b0 msgr2=0x7f1b3c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:37.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 -- 172.21.15.50:0/137659997 shutdown_connections 2023-10-30T12:20:37.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 -- 172.21.15.50:0/137659997 wait complete. 2023-10-30T12:20:37.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 10 monclient: build_initial_monmap 2023-10-30T12:20:37.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 10 monclient: monmap: 2023-10-30T12:20:37.457 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:37.457 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:37.457 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:37.457 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:37.458 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:37.458 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.458 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:37.458 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:37.458 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:37.458 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 librados: starting msgr at 2023-10-30T12:20:37.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 librados: starting objecter 2023-10-30T12:20:37.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.416+0000 7f1b430e7640 1 Processor -- start 2023-10-30T12:20:37.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 -- start start 2023-10-30T12:20:37.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 librados: setting wanted keys 2023-10-30T12:20:37.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 librados: calling monclient init 2023-10-30T12:20:37.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient: init 2023-10-30T12:20:37.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:37.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:20:37.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b3c1ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): picked mon.noname-c con 0x7f1b24003970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b3c1ca8f0 0x7f1b3c09d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c1ca8f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b3c1cad10 0x7f1b3c09d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): picked mon.noname-b con 0x7f1b3c1cad10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0a0c10 con 0x7f1b3c1ca8f0 2023-10-30T12:20:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0a1790 con 0x7f1b3c1cad10 2023-10-30T12:20:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c135f60 con 0x7f1b24003970 2023-10-30T12:20:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b430e7640 10 monclient(hunting): authenticate will time out at 2082.600597s 2023-10-30T12:20:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b33fff640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b3c1ca8f0 msgr2=0x7f1b3c09d440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b3c1ca8f0 0x7f1b3c09d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b3c1ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b3c1ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56142/0 (socket says 172.21.15.50:56142) 2023-10-30T12:20:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 -- 172.21.15.50:0/3882817515 learned_addr learned my addr 172.21.15.50:0/3882817515 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): get_auth_request con 0x7f1b24003970 auth_method 0 2023-10-30T12:20:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient(hunting): handle_auth_done global_id 254156 payload 824 2023-10-30T12:20:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b41684640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b3c1cad10 0x7f1b3c09d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b3c1cad10 msgr2=0x7f1b3c09d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b3c1cad10 0x7f1b3c09d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b3c1ca8f0 msgr2=0x7f1b3c09d440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b3c1ca8f0 0x7f1b3c09d440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient: _finish_hunting 0 2023-10-30T12:20:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 monclient: found mon.noname-c 2023-10-30T12:20:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b40e83640 1 -- 172.21.15.50:0/3882817515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c09faa0 con 0x7f1b24003970 2023-10-30T12:20:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b41684640 10 monclient: get_auth_request con 0x7f1b3c1cad10 auth_method 0 2023-10-30T12:20:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.417+0000 7f1b41684640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b3c1cad10 0x7f1b3c09d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:20:37.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b40e83640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b3c1ca3b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b380018f0 tx=0x7f1b3c0a58e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 client.?.objecter ms_handle_connect 0x7f1b24003970 2023-10-30T12:20:37.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:37.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3803c040 con 0x7f1b24003970 2023-10-30T12:20:37.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: dump: 2023-10-30T12:20:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.471 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: _finish_auth 0 2023-10-30T12:20:37.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: _check_auth_tickets 2023-10-30T12:20:37.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:37.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3802cc80 con 0x7f1b24003970 2023-10-30T12:20:37.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:37.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b38007070 con 0x7f1b24003970 2023-10-30T12:20:37.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 5 monclient: authenticate success, global_id 254156 2023-10-30T12:20:37.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 10 monclient: _renew_subs 2023-10-30T12:20:37.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c11e700 con 0x7f1b24003970 2023-10-30T12:20:37.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b32ffd640 4 set_mon_vals no callback set 2023-10-30T12:20:37.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b317fa640 10 monclient: dump: 2023-10-30T12:20:37.473 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.473 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.474 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.474 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.474 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.474 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.474 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.475 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 10 client.254156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 10 monclient: _renew_subs 2023-10-30T12:20:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c11eb20 con 0x7f1b24003970 2023-10-30T12:20:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.418+0000 7f1b430e7640 1 librados: init done 2023-10-30T12:20:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1b38032070 con 0x7f1b24003970 2023-10-30T12:20:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b430e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b430e7640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02b430 0x7f1b3c02b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b430e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c073ff0 con 0x7f1b3c02b430 2023-10-30T12:20:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02b430 0x7f1b3c02b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient: get_auth_request con 0x7f1b3c02b430 auth_method 0 2023-10-30T12:20:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b280392d0 0x7f1b2803b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(633..633 src has 520..633) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f1b3806ba50 con 0x7f1b24003970 2023-10-30T12:20:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 10 client.254156.objecter ms_dispatch 0x7f1b3c000b90 osd_map(633..633 src has 520..633) v4 2023-10-30T12:20:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 3 client.254156.objecter handle_osd_map got epochs [633,633] > 0 2023-10-30T12:20:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 3 client.254156.objecter handle_osd_map decoding full epoch 633 2023-10-30T12:20:37.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 20 client.254156.objecter dump_active .. 0 homeless 2023-10-30T12:20:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 10 monclient(hunting): handle_auth_done global_id 254159 payload 824 2023-10-30T12:20:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 1 -- 172.21.15.50:0/3882817515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b3c09a5c0 con 0x7f1b3c02b430 2023-10-30T12:20:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b41684640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b280392d0 0x7f1b2803b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b41684640 10 monclient: get_auth_request con 0x7f1b280392d0 auth_method 0 2023-10-30T12:20:37.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 10 client.254156.objecter ms_handle_connect 0x7f1b3c02b430 2023-10-30T12:20:37.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 10 client.254156.objecter resend_mon_ops 2023-10-30T12:20:37.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b33fff640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02b430 0x7f1b3c02b810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c003980 tx=0x7f1b2c009a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c012600 con 0x7f1b3c02b430 2023-10-30T12:20:37.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.419+0000 7f1b317fa640 10 monclient: dump: 2023-10-30T12:20:37.482 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.482 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.482 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.482 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.482 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.483 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.483 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.483 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.483 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.483 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 10 client.254156.objecter ms_handle_connect 0x7f1b280392d0 2023-10-30T12:20:37.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b41684640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b280392d0 0x7f1b2803b790 secure :-1 s=READY pgs=4316 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0a3c80 tx=0x7f1b340072a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1b2c012760 con 0x7f1b3c02b430 2023-10-30T12:20:37.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:37.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:37.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02b430 msgr2=0x7f1b3c02b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02b430 0x7f1b3c02b810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c003980 tx=0x7f1b2c009a80 comp rx=0 tx=0).stop 2023-10-30T12:20:37.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02b430 msgr2=0x7f1b3c02b810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:37.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.420+0000 7f1b317fa640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02b430 0x7f1b3c02b810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b430e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:37.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b430e7640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02bd50 0x7f1b3c02c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b430e7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c099b40 con 0x7f1b3c02bd50 2023-10-30T12:20:37.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02bd50 0x7f1b3c02c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient: get_auth_request con 0x7f1b3c02bd50 auth_method 0 2023-10-30T12:20:37.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 10 monclient(hunting): handle_auth_done global_id 254162 payload 824 2023-10-30T12:20:37.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.426+0000 7f1b33fff640 1 -- 172.21.15.50:0/3882817515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1b3c0a28a0 con 0x7f1b3c02bd50 2023-10-30T12:20:37.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 10 client.254156.objecter ms_handle_connect 0x7f1b3c02bd50 2023-10-30T12:20:37.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b33fff640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02bd50 0x7f1b3c02c160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c003980 tx=0x7f1b2c005a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 10 client.254156.objecter resend_mon_ops 2023-10-30T12:20:37.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c024070 con 0x7f1b3c02bd50 2023-10-30T12:20:37.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 10 monclient: dump: 2023-10-30T12:20:37.489 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.489 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.489 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.489 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.490 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.490 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.490 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.490 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.490 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.491 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4791 (secure 0 0 0) 0x7f1b2c01f320 con 0x7f1b3c02bd50 2023-10-30T12:20:37.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:37.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:37.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02bd50 msgr2=0x7f1b3c02c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02bd50 0x7f1b3c02c160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c003980 tx=0x7f1b2c005a80 comp rx=0 tx=0).stop 2023-10-30T12:20:37.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02bd50 msgr2=0x7f1b3c02c160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:37.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b317fa640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b3c02bd50 0x7f1b3c02c160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.492 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:37.493 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:37.493 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:37.493 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:20:37.493 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 206, 2023-10-30T12:20:37.493 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:37.493 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:37.494 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:37.494 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.494 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:20:37.494 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:37.494 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:37.495 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:37.495 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:37.495 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:37.495 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:37.495 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:37.495 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:37.496 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:37.496 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:37.496 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:37.496 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:37.496 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:37.496 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.497 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:37.497 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:37.497 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:37.497 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:37.497 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:37.498 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:37.498 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:37.498 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:37.498 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:37.498 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:37.498 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:37.499 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:37.499 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.499 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:37.499 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:37.500 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:37.500 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:37.500 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:37.500 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:37.500 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:37.500 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:37.501 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:37.501 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:37.501 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:37.501 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:37.501 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:37.501 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:37.502 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:37.502 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:37.502 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:37.502 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:37.502 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:37.503 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:37.503 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:37.503 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:37.503 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:37.503 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:37.504 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:37.504 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:37.504 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:37.504 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.504 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:37.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.505 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:37.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.505 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:37.505 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:37.505 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:37.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:37.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:37.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:37.506 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.508 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:37.508 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:37.508 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:37.509 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:37.509 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:37.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.509 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:37.509 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:37.510 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:37.510 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:37.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.510 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:37.510 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:37.511 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:37.511 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:37.511 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.512 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.512 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:37.512 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.513 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:37.514 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:37.514 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:37.514 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:37.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:37.515 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:37.515 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.515 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:37.516 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:37.516 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.516 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.516 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.516 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:37.517 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:37.517 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:37.517 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:37.517 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:37.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.518 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.518 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:37.518 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:37.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.518 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:37.519 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:37.519 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:37.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.519 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.519 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:20:37.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.520 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:37.520 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:37.520 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:20:37.520 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.520 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:37.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.521 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:37.521 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:37.521 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:20:37.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.522 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.522 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:20:37.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.522 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:37.522 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:37.523 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:37.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.523 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.523 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:37.524 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:37.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 10 librados: watch_flush enter 2023-10-30T12:20:37.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 10 librados: watch_flush exit 2023-10-30T12:20:37.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 20 client.254156.objecter shutdown clearing up homeless session... 2023-10-30T12:20:37.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 10 client.254156.objecter successfully canceled tick 2023-10-30T12:20:37.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b280392d0 msgr2=0x7f1b2803b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b280392d0 0x7f1b2803b790 secure :-1 s=READY pgs=4316 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0a3c80 tx=0x7f1b340072a0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 10 monclient: shutdown 2023-10-30T12:20:37.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:37.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 msgr2=0x7f1b3c1ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b3c1ca3b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b380018f0 tx=0x7f1b3c0a58e0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b41684640 1 -- 172.21.15.50:0/3882817515 reap_dead start 2023-10-30T12:20:37.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 shutdown_connections 2023-10-30T12:20:37.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 --2- 172.21.15.50:0/3882817515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b24003970 0x7f1b3c1ca3b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 >> 172.21.15.50:0/3882817515 conn(0x7f1b3c0a80b0 msgr2=0x7f1b3c135780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:37.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 shutdown_connections 2023-10-30T12:20:37.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 -- 172.21.15.50:0/3882817515 wait complete. 2023-10-30T12:20:37.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.427+0000 7f1b430e7640 1 librados: shutdown 2023-10-30T12:20:37.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:20:37.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:20:37.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:20:37.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.634+0000 7f78dc1b7640 10 monclient: get_monmap_and_config 2023-10-30T12:20:37.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.634+0000 7f78dc1b7640 10 monclient: build_initial_monmap 2023-10-30T12:20:37.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.634+0000 7f78dc1b7640 10 monclient: monmap: 2023-10-30T12:20:37.637 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:37.637 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.637 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:37.635784+0000 2023-10-30T12:20:37.637 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:37.635784+0000 2023-10-30T12:20:37.638 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:37.638 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.638 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:37.638 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:37.638 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:37.638 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:37.639 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:37.639 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 1 Processor -- start 2023-10-30T12:20:37.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 1 -- start start 2023-10-30T12:20:37.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient: init 2023-10-30T12:20:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-10-30T12:20:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4071c00 0x7f78d409a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient(hunting): picked mon.noname-b con 0x7f78d4071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78d4097fe0 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f78d409b170 0x7f78d4139760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient(hunting): picked mon.noname-c con 0x7f78d409b170 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d4097600 con 0x7f78d4097fe0 2023-10-30T12:20:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d40978e0 con 0x7f78d409b170 2023-10-30T12:20:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d409bf50 con 0x7f78d4071c00 2023-10-30T12:20:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.635+0000 7f78dc1b7640 10 monclient(hunting): authenticate will time out at 2082.818957s 2023-10-30T12:20:37.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4071c00 0x7f78d409a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4071c00 0x7f78d409a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34494/0 (socket says 172.21.15.50:34494) 2023-10-30T12:20:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 -- 172.21.15.50:0/2928218910 learned_addr learned my addr 172.21.15.50:0/2928218910 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): get_auth_request con 0x7f78d4071c00 auth_method 0 2023-10-30T12:20:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78da754640 1 --2- 172.21.15.50:0/2928218910 >> v2:172.21.15.118:3300/0 conn(0x7f78d409b170 0x7f78d4139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78da754640 10 monclient(hunting): get_auth_request con 0x7f78d409b170 auth_method 0 2023-10-30T12:20:37.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78da754640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78da754640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78da754640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient(hunting): handle_auth_done global_id 254165 payload 274 2023-10-30T12:20:37.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 -- 172.21.15.50:0/2928218910 >> v2:172.21.15.118:3300/0 conn(0x7f78d409b170 msgr2=0x7f78d4139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:37.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 --2- 172.21.15.50:0/2928218910 >> v2:172.21.15.118:3300/0 conn(0x7f78d409b170 0x7f78d4139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 -- 172.21.15.50:0/2928218910 >> v1:172.21.15.118:6789/0 conn(0x7f78d4097fe0 legacy=0x7f78d409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:20:37.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient: _finish_hunting 0 2023-10-30T12:20:37.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 monclient: found mon.noname-b 2023-10-30T12:20:37.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:37.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.636+0000 7f78d9f53640 1 -- 172.21.15.50:0/2928218910 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d409d9b0 con 0x7f78d4071c00 2023-10-30T12:20:37.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d9f53640 1 --2- 172.21.15.50:0/2928218910 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4071c00 0x7f78d409a150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009850 tx=0x7f78d002b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f807a91b9de414c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 1 -- 172.21.15.50:0/2928218910 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d0004230 con 0x7f78d4071c00 2023-10-30T12:20:37.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:20:37.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: dump: 2023-10-30T12:20:37.656 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.656 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.656 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.656 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.657 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.657 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.657 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.657 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.657 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.658 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: _finish_auth 0 2023-10-30T12:20:37.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: _check_auth_tickets 2023-10-30T12:20:37.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:37.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 5 monclient: authenticate success, global_id 254165 2023-10-30T12:20:37.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:20:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 1 -- 172.21.15.50:0/2928218910 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d002f050 con 0x7f78d4071c00 2023-10-30T12:20:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 4 set_mon_vals no callback set 2023-10-30T12:20:37.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 1 -- 172.21.15.50:0/2928218910 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d0003610 con 0x7f78d4071c00 2023-10-30T12:20:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78d8f51640 10 monclient: dump: 2023-10-30T12:20:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.660 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.661 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.662 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 10 monclient: shutdown 2023-10-30T12:20:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 -- 172.21.15.50:0/2928218910 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4071c00 msgr2=0x7f78d409a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 --2- 172.21.15.50:0/2928218910 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4071c00 0x7f78d409a150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009850 tx=0x7f78d002b7b0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 -- 172.21.15.50:0/2928218910 shutdown_connections 2023-10-30T12:20:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 --2- 172.21.15.50:0/2928218910 >> v2:172.21.15.118:3300/0 conn(0x7f78d409b170 0x7f78d4139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 --2- 172.21.15.50:0/2928218910 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4071c00 0x7f78d409a150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 -- 172.21.15.50:0/2928218910 >> 172.21.15.50:0/2928218910 conn(0x7f78d40a3980 msgr2=0x7f78d40a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:37.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 -- 172.21.15.50:0/2928218910 shutdown_connections 2023-10-30T12:20:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.637+0000 7f78dc1b7640 1 -- 172.21.15.50:0/2928218910 wait complete. 2023-10-30T12:20:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 10 monclient: build_initial_monmap 2023-10-30T12:20:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 10 monclient: monmap: 2023-10-30T12:20:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:37.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:37.666 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 1 librados: starting msgr at 2023-10-30T12:20:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 1 librados: starting objecter 2023-10-30T12:20:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 1 Processor -- start 2023-10-30T12:20:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 1 -- start start 2023-10-30T12:20:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 1 librados: setting wanted keys 2023-10-30T12:20:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 1 librados: calling monclient init 2023-10-30T12:20:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.638+0000 7f78dc1b7640 10 monclient: init 2023-10-30T12:20:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:20:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78d4071c00 0x7f78d4096930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): picked mon.noname-a con 0x7f78d4071c00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4097fe0 0x7f78d406bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): picked mon.noname-c con 0x7f78d4097fe0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:37.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d409b170 0x7f78d406c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): picked mon.noname-b con 0x7f78d409b170 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d406ed60 con 0x7f78d4071c00 2023-10-30T12:20:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9f53640 1 -- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78d4071c00 msgr2=0x7f78d4096930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3300/0 2023-10-30T12:20:37.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d406d160 con 0x7f78d409b170 2023-10-30T12:20:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d406d9d0 con 0x7f78d4097fe0 2023-10-30T12:20:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:37.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78dc1b7640 10 monclient(hunting): authenticate will time out at 2082.822367s 2023-10-30T12:20:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9f53640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78d4071c00 0x7f78d4096930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4097fe0 0x7f78d406bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4097fe0 0x7f78d406bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34504/0 (socket says 172.21.15.50:34504) 2023-10-30T12:20:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 -- 172.21.15.50:0/938296509 learned_addr learned my addr 172.21.15.50:0/938296509 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): get_auth_request con 0x7f78d4097fe0 auth_method 0 2023-10-30T12:20:37.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78da754640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d409b170 0x7f78d406c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78da754640 10 monclient(hunting): get_auth_request con 0x7f78d409b170 auth_method 0 2023-10-30T12:20:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78da754640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78da754640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78da754640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient(hunting): handle_auth_done global_id 254168 payload 824 2023-10-30T12:20:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d409b170 msgr2=0x7f78d406c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d409b170 0x7f78d406c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78d4071c00 msgr2=0x7f78d4096930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:37.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f78d4071c00 0x7f78d4096930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient: _finish_hunting 0 2023-10-30T12:20:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 monclient: found mon.noname-c 2023-10-30T12:20:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 -- 172.21.15.50:0/938296509 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0009460 con 0x7f78d4097fe0 2023-10-30T12:20:37.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78d9752640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4097fe0 0x7f78d406bba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78c800c460 tx=0x7f78c800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78c67fc640 10 client.?.objecter ms_handle_connect 0x7f78d4097fe0 2023-10-30T12:20:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.639+0000 7f78c67fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8012830 con 0x7f78d4097fe0 2023-10-30T12:20:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: dump: 2023-10-30T12:20:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.677 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.678 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: _finish_auth 0 2023-10-30T12:20:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:20:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c80129d0 con 0x7f78d4097fe0 2023-10-30T12:20:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 5 monclient: authenticate success, global_id 254168 2023-10-30T12:20:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8012d80 con 0x7f78d4097fe0 2023-10-30T12:20:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 10 monclient: _renew_subs 2023-10-30T12:20:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d406dde0 con 0x7f78d4097fe0 2023-10-30T12:20:37.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c7fff640 4 set_mon_vals no callback set 2023-10-30T12:20:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 10 monclient: dump: 2023-10-30T12:20:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.681 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.683 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 10 client.254168.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 10 monclient: _renew_subs 2023-10-30T12:20:37.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d406ef40 con 0x7f78d4097fe0 2023-10-30T12:20:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 1 librados: init done 2023-10-30T12:20:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d406f360 0x7f78d406c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d4073ff0 con 0x7f78d406f360 2023-10-30T12:20:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.640+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f78c801b400 con 0x7f78d4097fe0 2023-10-30T12:20:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d406f360 0x7f78d406c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient: get_auth_request con 0x7f78d406f360 auth_method 0 2023-10-30T12:20:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78b80392a0 0x7f78b803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(633..633 src has 520..633) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f78c804ad10 con 0x7f78d4097fe0 2023-10-30T12:20:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 10 client.254168.objecter ms_dispatch 0x7f78d4000b90 osd_map(633..633 src has 520..633) v4 2023-10-30T12:20:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 3 client.254168.objecter handle_osd_map got epochs [633,633] > 0 2023-10-30T12:20:37.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 3 client.254168.objecter handle_osd_map decoding full epoch 633 2023-10-30T12:20:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 20 client.254168.objecter dump_active .. 0 homeless 2023-10-30T12:20:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78da754640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78b80392a0 0x7f78b803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78da754640 10 monclient: get_auth_request con 0x7f78b80392a0 auth_method 0 2023-10-30T12:20:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 10 monclient(hunting): handle_auth_done global_id 204709 payload 824 2023-10-30T12:20:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 1 -- 172.21.15.50:0/938296509 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78d4071fe0 con 0x7f78d406f360 2023-10-30T12:20:37.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78d9f53640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d406f360 0x7f78d406c620 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f78d409fe20 tx=0x7f78d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 10 client.254168.objecter ms_handle_connect 0x7f78d406f360 2023-10-30T12:20:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 10 client.254168.objecter resend_mon_ops 2023-10-30T12:20:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 10 client.254168.objecter ms_handle_connect 0x7f78b80392a0 2023-10-30T12:20:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78da754640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78b80392a0 0x7f78b803b760 secure :-1 s=READY pgs=4317 cs=0 l=1 rev1=1 crypto rx=0x7f78d4096ea0 tx=0x7f78cc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d0003c00 con 0x7f78d406f360 2023-10-30T12:20:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.641+0000 7f78c67fc640 10 monclient: dump: 2023-10-30T12:20:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.690 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.691 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.692 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.642+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f78d0003610 con 0x7f78d406f360 2023-10-30T12:20:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.642+0000 7f78c67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:37.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.642+0000 7f78c67fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.642+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d406f360 msgr2=0x7f78d406c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.642+0000 7f78c67fc640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d406f360 0x7f78d406c620 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f78d409fe20 tx=0x7f78d00046c0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.642+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d406f360 msgr2=0x7f78d406c620 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.642+0000 7f78c67fc640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d406f360 0x7f78d406c620 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:37.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78dc1b7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:20:37.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78dc1b7640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d408afd0 0x7f78d408b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:37.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78dc1b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d404e160 con 0x7f78d408afd0 2023-10-30T12:20:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d408afd0 0x7f78d408b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 10 monclient: get_auth_request con 0x7f78d408afd0 auth_method 0 2023-10-30T12:20:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.648+0000 7f78d9f53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78d9f53640 10 monclient(hunting): handle_auth_done global_id 204712 payload 824 2023-10-30T12:20:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78d9f53640 1 -- 172.21.15.50:0/938296509 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f78d4099790 con 0x7f78d408afd0 2023-10-30T12:20:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78d9f53640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d408afd0 0x7f78d408b3e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009850 tx=0x7f78d0009210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 10 client.254168.objecter ms_handle_connect 0x7f78d408afd0 2023-10-30T12:20:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 10 client.254168.objecter resend_mon_ops 2023-10-30T12:20:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d0031d30 con 0x7f78d408afd0 2023-10-30T12:20:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 10 monclient: dump: 2023-10-30T12:20:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:37.699 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:37.700 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:37.701 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4623 (secure 0 0 0) 0x7f78d004b040 con 0x7f78d408afd0 2023-10-30T12:20:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:20:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:37.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d408afd0 msgr2=0x7f78d408b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d408afd0 0x7f78d408b3e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009850 tx=0x7f78d0009210 comp rx=0 tx=0).stop 2023-10-30T12:20:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d408afd0 msgr2=0x7f78d408b3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:20:37.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78c67fc640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78d408afd0 0x7f78d408b3e0 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009850 tx=0x7f78d0009210 comp rx=0 tx=0).stop 2023-10-30T12:20:37.702 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:20:37.703 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:37.703 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:37.703 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:20:37.703 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 206, 2023-10-30T12:20:37.703 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:20:37.704 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:20:37.704 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:20:37.704 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.704 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:20:37.704 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:37.705 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:20:37.705 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:20:37.705 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:37.705 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:37.705 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:37.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:37.706 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:37.706 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:37.706 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:37.706 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:37.707 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:37.707 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:37.707 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.707 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:20:37.707 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:20:37.707 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:20:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:20:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:20:37.711 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:20:37.711 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:20:37.711 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:20:37.711 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:37.711 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:20:37.712 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:20:37.712 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:20:37.712 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:20:37.712 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:20:37.712 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:20:37.712 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:20:37.713 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:20:37.713 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:20:37.713 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:20:37.713 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:20:37.713 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:20:37.714 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:20:37.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:20:37.714 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:20:37.714 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:20:37.714 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:20:37.715 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:20:37.715 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:20:37.715 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:20:37.715 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.715 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:20:37.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.716 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:20:37.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.716 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:20:37.716 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:20:37.716 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:37.717 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:37.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.717 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:37.717 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:20:37.717 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:37.718 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:20:37.718 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.719 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.719 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:20:37.719 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:20:37.719 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:37.720 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:37.720 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:37.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.720 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:20:37.721 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:20:37.721 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:37.721 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:37.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.721 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:37.722 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:20:37.722 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:37.723 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:20:37.723 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.723 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.723 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.723 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:20:37.724 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:20:37.724 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:37.724 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:37.724 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:37.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:20:37.725 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:20:37.725 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:20:37.725 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:20:37.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.726 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:20:37.726 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:20:37.726 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.727 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:20:37.727 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:20:37.727 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:20:37.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.727 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.728 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:20:37.728 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:20:37.728 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:20:37.728 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:20:37.728 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:20:37.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.729 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.729 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:20:37.729 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:20:37.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.730 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:37.730 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:37.730 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:20:37.730 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.731 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.731 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:20:37.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.731 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:37.731 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:37.731 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:20:37.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.732 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:20:37.732 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:20:37.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:20:37.732 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:20:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:20:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:20:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:20:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:20:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:20:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:20:37.734 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:20:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 10 librados: watch_flush enter 2023-10-30T12:20:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 10 librados: watch_flush exit 2023-10-30T12:20:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 20 client.254168.objecter shutdown clearing up homeless session... 2023-10-30T12:20:37.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 10 client.254168.objecter successfully canceled tick 2023-10-30T12:20:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78b80392a0 msgr2=0x7f78b803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f78b80392a0 0x7f78b803b760 secure :-1 s=READY pgs=4317 cs=0 l=1 rev1=1 crypto rx=0x7f78d4096ea0 tx=0x7f78cc00b040 comp rx=0 tx=0).stop 2023-10-30T12:20:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 10 monclient: shutdown 2023-10-30T12:20:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:37.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4097fe0 msgr2=0x7f78d406bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 1 --2- 172.21.15.50:0/938296509 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f78d4097fe0 0x7f78d406bba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78c800c460 tx=0x7f78c800c920 comp rx=0 tx=0).stop 2023-10-30T12:20:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78da754640 1 -- 172.21.15.50:0/938296509 reap_dead start 2023-10-30T12:20:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78da754640 1 -- 172.21.15.50:0/938296509 reap_dead start 2023-10-30T12:20:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 shutdown_connections 2023-10-30T12:20:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 >> 172.21.15.50:0/938296509 conn(0x7f78d40a3980 msgr2=0x7f78d4133930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.649+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 shutdown_connections 2023-10-30T12:20:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.650+0000 7f78dc1b7640 1 -- 172.21.15.50:0/938296509 wait complete. 2023-10-30T12:20:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:37.650+0000 7f78dc1b7640 1 librados: shutdown 2023-10-30T12:20:37.739 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:20:37.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:20:57.740 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-10-30T12:20:57.741 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-10-30T12:20:57.741 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:20:57.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:20:57.745 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:20:57.745 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:20:57.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:20:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.859+0000 7fc400075640 10 monclient: get_monmap_and_config 2023-10-30T12:20:57.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.859+0000 7fc400075640 10 monclient: build_initial_monmap 2023-10-30T12:20:57.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.859+0000 7fc400075640 10 monclient: monmap: 2023-10-30T12:20:57.861 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:57.861 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:57.861 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:20:57.860352+0000 2023-10-30T12:20:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:20:57.860352+0000 2023-10-30T12:20:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:20:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:20:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:20:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:20:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:20:57.863 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.859+0000 7fc400075640 1 Processor -- start 2023-10-30T12:20:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 1 -- start start 2023-10-30T12:20:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient: init 2023-10-30T12:20:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-10-30T12:20:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 0x7fc3f81326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient(hunting): picked mon.noname-a con 0x7fc3f80a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:20:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc3f8099a40 0x7fc3f8132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient(hunting): picked mon.noname-c con 0x7fc3f8099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:20:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3f80a2350 addr v1:172.21.15.118:6789/0 2023-10-30T12:20:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f8133120 con 0x7fc3f80a2350 2023-10-30T12:20:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f80a3d70 con 0x7fc3f80a4b60 2023-10-30T12:20:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:57.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f80a4f40 con 0x7fc3f8099a40 2023-10-30T12:20:57.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:57.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc400075640 10 monclient(hunting): authenticate will time out at 2103.043716s 2023-10-30T12:20:57.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 0x7fc3f81326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:57.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 0x7fc3f81326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37286/0 (socket says 172.21.15.50:37286) 2023-10-30T12:20:57.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 1 -- 172.21.15.50:0/1849763542 learned_addr learned my addr 172.21.15.50:0/1849763542 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:57.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 10 monclient(hunting): get_auth_request con 0x7fc3f80a4b60 auth_method 0 2023-10-30T12:20:57.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:57.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:57.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:57.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 1 -- 172.21.15.50:0/1849763542 >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 msgr2=0x7fc3f81326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:20:57.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 1 -- 172.21.15.50:0/1849763542 >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 msgr2=0x7fc3f81326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:20:57.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.860+0000 7fc3fde11640 1 --2- 172.21.15.50:0/1849763542 >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 0x7fc3f81326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:57.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fde11640 1 --2- 172.21.15.50:0/1849763542 >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 0x7fc3f81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:57.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 1 --2- 172.21.15.50:0/1849763542 >> v2:172.21.15.118:3300/0 conn(0x7fc3f8099a40 0x7fc3f8132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:57.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): get_auth_request con 0x7fc3f8099a40 auth_method 0 2023-10-30T12:20:57.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:57.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:57.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:57.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:57.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:57.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:57.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient(hunting): handle_auth_done global_id 204715 payload 274 2023-10-30T12:20:57.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 1 -- 172.21.15.50:0/1849763542 >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 msgr2=0x7fc3f81326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:20:57.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 1 --2- 172.21.15.50:0/1849763542 >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 0x7fc3f81326a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:57.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 1 -- 172.21.15.50:0/1849763542 >> v1:172.21.15.118:6789/0 conn(0x7fc3f80a2350 legacy=0x7fc3f80a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:57.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient: _finish_hunting 0 2023-10-30T12:20:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 1 monclient: found mon.noname-c 2023-10-30T12:20:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fce0f640 1 -- 172.21.15.50:0/1849763542 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325811384 0 0) 0x7fc3f8133120 con 0x7fc3f80a2350 2023-10-30T12:20:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:20:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fd610640 1 -- 172.21.15.50:0/1849763542 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80a4150 con 0x7fc3f8099a40 2023-10-30T12:20:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.861+0000 7fc3fce0f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:20:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fd610640 1 --2- 172.21.15.50:0/1849763542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f8099a40 0x7fc3f8132be0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8099ed0 tx=0x7fc3e802b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1db9be20a431d14 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 1 -- 172.21.15.50:0/1849763542 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8031710 con 0x7fc3f8099a40 2023-10-30T12:20:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: dump: 2023-10-30T12:20:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:57.882 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:57.882 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:57.882 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:57.882 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:57.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: _finish_auth 0 2023-10-30T12:20:57.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: _check_auth_tickets 2023-10-30T12:20:57.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:57.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 1 -- 172.21.15.50:0/1849763542 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e802f050 con 0x7fc3f8099a40 2023-10-30T12:20:57.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:57.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 5 monclient: authenticate success, global_id 204715 2023-10-30T12:20:57.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 10 monclient: get_monmap_and_config success 2023-10-30T12:20:57.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 4 set_mon_vals no callback set 2023-10-30T12:20:57.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 1 -- 172.21.15.50:0/1849763542 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8031a90 con 0x7fc3f8099a40 2023-10-30T12:20:57.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:57.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:20:57.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc3fce0f640 10 monclient: dump: 2023-10-30T12:20:57.885 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:57.885 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:57.885 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.885 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.885 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:57.885 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:57.886 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:57.886 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:57.886 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:57.886 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:57.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 10 monclient: shutdown 2023-10-30T12:20:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 1 -- 172.21.15.50:0/1849763542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f8099a40 msgr2=0x7fc3f8132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 1 --2- 172.21.15.50:0/1849763542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f8099a40 0x7fc3f8132be0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8099ed0 tx=0x7fc3e802b7b0 comp rx=0 tx=0).stop 2023-10-30T12:20:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 1 -- 172.21.15.50:0/1849763542 shutdown_connections 2023-10-30T12:20:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 1 --2- 172.21.15.50:0/1849763542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f8099a40 0x7fc3f8132be0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 1 --2- 172.21.15.50:0/1849763542 >> v2:172.21.15.50:3300/0 conn(0x7fc3f80a4b60 0x7fc3f81326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:57.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.862+0000 7fc400075640 1 -- 172.21.15.50:0/1849763542 >> 172.21.15.50:0/1849763542 conn(0x7fc3f80a8060 msgr2=0x7fc3f80a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:57.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.863+0000 7fc400075640 1 -- 172.21.15.50:0/1849763542 shutdown_connections 2023-10-30T12:20:57.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.863+0000 7fc400075640 1 -- 172.21.15.50:0/1849763542 wait complete. 2023-10-30T12:20:57.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.863+0000 7fc400075640 10 monclient: build_initial_monmap 2023-10-30T12:20:57.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.863+0000 7fc400075640 10 monclient: monmap: 2023-10-30T12:20:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:20:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:20:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:20:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:20:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:20:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:20:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:20:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:20:57.890 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.863+0000 7fc400075640 1 librados: starting msgr at 2023-10-30T12:20:57.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.863+0000 7fc400075640 1 librados: starting objecter 2023-10-30T12:20:57.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.863+0000 7fc400075640 1 Processor -- start 2023-10-30T12:20:57.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 -- start start 2023-10-30T12:20:57.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 librados: setting wanted keys 2023-10-30T12:20:57.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 librados: calling monclient init 2023-10-30T12:20:57.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient: init 2023-10-30T12:20:57.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient: _reopen_session rank -1 2023-10-30T12:20:57.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:20:57.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3f80a2350 0x7fc3f81c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:57.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): picked mon.noname-c con 0x7fc3f80a2350 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:20:57.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f80a4b60 0x7fc3f81ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:57.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): picked mon.noname-b con 0x7fc3f80a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:20:57.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 0x7fc3f809cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:57.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): picked mon.noname-a con 0x7fc3f81cab90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:20:57.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:57.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80a1970 con 0x7fc3f81cab90 2023-10-30T12:20:57.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:57.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80a0990 con 0x7fc3f80a4b60 2023-10-30T12:20:57.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): start opening mon connection 2023-10-30T12:20:57.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f8131440 con 0x7fc3f80a2350 2023-10-30T12:20:57.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): _renew_subs 2023-10-30T12:20:57.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc400075640 10 monclient(hunting): authenticate will time out at 2103.047585s 2023-10-30T12:20:57.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3f80a2350 0x7fc3f81c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:57.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 0x7fc3f809cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:57.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3f80a2350 0x7fc3f81c9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39004/0 (socket says 172.21.15.50:39004) 2023-10-30T12:20:57.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 -- 172.21.15.50:0/1345023210 learned_addr learned my addr 172.21.15.50:0/1345023210 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:20:57.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): get_auth_request con 0x7fc3f80a2350 auth_method 0 2023-10-30T12:20:57.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:57.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:57.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:57.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 10 monclient(hunting): get_auth_request con 0x7fc3f81cab90 auth_method 0 2023-10-30T12:20:57.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:20:57.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:20:57.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:20:57.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:20:57.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:20:57.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:20:57.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 1 -- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 msgr2=0x7fc3f809cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:20:57.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 1 -- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 msgr2=0x7fc3f809cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:20:57.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 0x7fc3f809cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:20:57.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fe612640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 0x7fc3f809cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:20:57.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fd610640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f80a4b60 0x7fc3f81ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:57.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient(hunting): handle_auth_done global_id 254171 payload 824 2023-10-30T12:20:57.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 -- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f80a4b60 msgr2=0x7fc3f81ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:57.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f80a4b60 0x7fc3f81ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 -- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 msgr2=0x7fc3f809cc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:20:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 0x7fc3f809cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 10 monclient: _finish_hunting 0 2023-10-30T12:20:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 1 monclient: found mon.noname-c 2023-10-30T12:20:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.864+0000 7fc3fde11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:20:57.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3fde11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:20:57.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3fde11640 1 -- 172.21.15.50:0/1345023210 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8009400 con 0x7fc3f80a2350 2023-10-30T12:20:57.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3fd610640 10 monclient: get_auth_request con 0x7fc3f80a4b60 auth_method 0 2023-10-30T12:20:57.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3fd610640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f80a4b60 0x7fc3f81ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:20:57.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3fde11640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3f80a2350 0x7fc3f81c9f20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400e810 tx=0x7fc3f400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:57.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 client.?.objecter ms_handle_connect 0x7fc3f80a2350 2023-10-30T12:20:57.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:20:57.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 1 -- 172.21.15.50:0/1345023210 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4004020 con 0x7fc3f80a2350 2023-10-30T12:20:57.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:20:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: dump: 2023-10-30T12:20:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.904 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:57.904 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:57.904 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:57.904 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:57.904 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:57.904 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:57.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: _finish_auth 0 2023-10-30T12:20:57.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:20:57.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:20:57.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 1 -- 172.21.15.50:0/1345023210 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f4004180 con 0x7fc3f80a2350 2023-10-30T12:20:57.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:20:57.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 1 -- 172.21.15.50:0/1345023210 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f400c6d0 con 0x7fc3f80a2350 2023-10-30T12:20:57.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 5 monclient: authenticate success, global_id 254171 2023-10-30T12:20:57.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 10 monclient: _renew_subs 2023-10-30T12:20:57.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:57.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f80a06b0 con 0x7fc3f80a2350 2023-10-30T12:20:57.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3effff640 4 set_mon_vals no callback set 2023-10-30T12:20:57.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:20:57.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:20:57.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc3ee7fc640 10 monclient: dump: 2023-10-30T12:20:57.907 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:20:57.907 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:20:57.908 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.908 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:20:57.908 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:20:57.908 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:20:57.908 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:20:57.909 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:20:57.909 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:20:57.909 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:20:57.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 10 client.254171.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:20:57.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 10 monclient: _renew_subs 2023-10-30T12:20:57.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:57.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f809f4e0 con 0x7fc3f80a2350 2023-10-30T12:20:57.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.865+0000 7fc400075640 1 librados: init done 2023-10-30T12:20:57.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3ee7fc640 1 -- 172.21.15.50:0/1345023210 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc3f400c830 con 0x7fc3f80a2350 2023-10-30T12:20:57.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc400075640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:57.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc400075640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:57.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc400075640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:57.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3ee7fc640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3dc0392f0 0x7fc3dc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:20:57.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f80a0b20 con 0x7fc3f80a2350 2023-10-30T12:20:57.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3ee7fc640 1 -- 172.21.15.50:0/1345023210 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(652..652 src has 520..652) v4 ==== 6678+0+0 (secure 0 0 0) 0x7fc3f404a960 con 0x7fc3f80a2350 2023-10-30T12:20:57.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3ee7fc640 10 client.254171.objecter ms_dispatch 0x7fc3f8000b90 osd_map(652..652 src has 520..652) v4 2023-10-30T12:20:57.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3ee7fc640 3 client.254171.objecter handle_osd_map got epochs [652,652] > 0 2023-10-30T12:20:57.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3ee7fc640 3 client.254171.objecter handle_osd_map decoding full epoch 652 2023-10-30T12:20:57.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3ee7fc640 20 client.254171.objecter dump_active .. 0 homeless 2023-10-30T12:20:57.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3fd610640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3dc0392f0 0x7fc3dc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:20:57.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.866+0000 7fc3fd610640 10 monclient: get_auth_request con 0x7fc3dc0392f0 auth_method 0 2023-10-30T12:20:57.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.867+0000 7fc3ee7fc640 10 client.254171.objecter ms_handle_connect 0x7fc3dc0392f0 2023-10-30T12:20:57.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.867+0000 7fc3fd610640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3dc0392f0 0x7fc3dc03b7b0 secure :-1 s=READY pgs=4319 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8035000 tx=0x7fc3e8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:20:57.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.871+0000 7fc3ee7fc640 1 -- 172.21.15.50:0/1345023210 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc3f400ce60 con 0x7fc3f80a2350 2023-10-30T12:20:57.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.871+0000 7fc3ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:20:57.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:57.871+0000 7fc3ee7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:20:58.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.116+0000 7fc400075640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:20:58.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.116+0000 7fc400075640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:58.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.116+0000 7fc400075640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:20:58.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.116+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3f80a04c0 con 0x7fc3f80a2350 2023-10-30T12:20:58.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.116+0000 7fc3ee7fc640 1 -- 172.21.15.50:0/1345023210 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fc3f4012370 con 0x7fc3f80a2350 2023-10-30T12:20:58.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.116+0000 7fc3ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:20:58.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.116+0000 7fc3ee7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:20:58.119 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":206,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:20:58.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 10 librados: watch_flush enter 2023-10-30T12:20:58.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 10 librados: watch_flush exit 2023-10-30T12:20:58.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 20 client.254171.objecter shutdown clearing up homeless session... 2023-10-30T12:20:58.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 10 client.254171.objecter successfully canceled tick 2023-10-30T12:20:58.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3dc0392f0 msgr2=0x7fc3dc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:58.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3dc0392f0 0x7fc3dc03b7b0 secure :-1 s=READY pgs=4319 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8035000 tx=0x7fc3e8008030 comp rx=0 tx=0).stop 2023-10-30T12:20:58.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 10 monclient: shutdown 2023-10-30T12:20:58.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:20:58.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3f80a2350 msgr2=0x7fc3f81c9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:20:58.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3f80a2350 0x7fc3f81c9f20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400e810 tx=0x7fc3f400ecd0 comp rx=0 tx=0).stop 2023-10-30T12:20:58.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 shutdown_connections 2023-10-30T12:20:58.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc3dc0392f0 0x7fc3dc03b7b0 unknown :-1 s=CLOSED pgs=4319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:58.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3f81cab90 0x7fc3f809cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:58.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3f80a4b60 0x7fc3f81ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:58.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 --2- 172.21.15.50:0/1345023210 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3f80a2350 0x7fc3f81c9f20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:20:58.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 >> 172.21.15.50:0/1345023210 conn(0x7fc3f80a8060 msgr2=0x7fc3f81bd970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:20:58.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 shutdown_connections 2023-10-30T12:20:58.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 -- 172.21.15.50:0/1345023210 wait complete. 2023-10-30T12:20:58.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:20:58.118+0000 7fc400075640 1 librados: shutdown 2023-10-30T12:21:01.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:21:01.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.247+0000 7fcdfb39e640 10 monclient: get_monmap_and_config 2023-10-30T12:21:01.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.247+0000 7fcdfb39e640 10 monclient: build_initial_monmap 2023-10-30T12:21:01.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.247+0000 7fcdfb39e640 10 monclient: monmap: 2023-10-30T12:21:01.250 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:01.250 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:01.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:01.248853+0000 2023-10-30T12:21:01.250 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:01.248853+0000 2023-10-30T12:21:01.251 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:01.251 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:01.251 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:01.251 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:01.251 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:01.252 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:01.252 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:01.252 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:01.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.248+0000 7fcdfb39e640 1 Processor -- start 2023-10-30T12:21:01.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.248+0000 7fcdfb39e640 1 -- start start 2023-10-30T12:21:01.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.248+0000 7fcdfb39e640 10 monclient: init 2023-10-30T12:21:01.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:01.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-10-30T12:21:01.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcdf41312b0 0x7fcdf406b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:01.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf41312b0 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:01.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 0x7fcdf40a2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:01.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf4131690 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:01.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdf4131ae0 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:01.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf40a3c20 con 0x7fcdf4131ae0 2023-10-30T12:21:01.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:01.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf406a330 con 0x7fcdf4131690 2023-10-30T12:21:01.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:01.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf406a610 con 0x7fcdf41312b0 2023-10-30T12:21:01.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:01.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfb39e640 10 monclient(hunting): authenticate will time out at 2106.432112s 2023-10-30T12:21:01.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 0x7fcdf40a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:01.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 0x7fcdf40a2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37306/0 (socket says 172.21.15.50:37306) 2023-10-30T12:21:01.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 1 -- 172.21.15.50:0/4223464936 learned_addr learned my addr 172.21.15.50:0/4223464936 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:01.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 10 monclient(hunting): get_auth_request con 0x7fcdf4131690 auth_method 0 2023-10-30T12:21:01.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:01.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:01.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:01.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 1 -- 172.21.15.50:0/4223464936 >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 msgr2=0x7fcdf40a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:21:01.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 1 -- 172.21.15.50:0/4223464936 >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 msgr2=0x7fcdf40a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:01.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 1 --2- 172.21.15.50:0/4223464936 >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 0x7fcdf40a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:01.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 1 --2- 172.21.15.50:0/4223464936 >> v2:172.21.15.118:3300/0 conn(0x7fcdf41312b0 0x7fcdf406b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:01.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdf9b9b640 1 --2- 172.21.15.50:0/4223464936 >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 0x7fcdf40a2100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:01.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): get_auth_request con 0x7fcdf41312b0 auth_method 0 2023-10-30T12:21:01.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:01.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:01.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:01.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:01.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:01.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:01.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient(hunting): handle_auth_done global_id 204721 payload 274 2023-10-30T12:21:01.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 1 -- 172.21.15.50:0/4223464936 >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 msgr2=0x7fcdf40a2100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:01.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 1 --2- 172.21.15.50:0/4223464936 >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 0x7fcdf40a2100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 1 -- 172.21.15.50:0/4223464936 >> v1:172.21.15.118:6789/0 conn(0x7fcdf4131ae0 legacy=0x7fcdf40a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:01.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient: _finish_hunting 0 2023-10-30T12:21:01.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 1 monclient: found mon.noname-c 2023-10-30T12:21:01.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:01.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:21:01.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.249+0000 7fcdfa39c640 1 -- 172.21.15.50:0/4223464936 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf406a8f0 con 0x7fcdf41312b0 2023-10-30T12:21:01.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdfa39c640 1 --2- 172.21.15.50:0/4223464936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 0x7fcdf406b690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4073aa0 tx=0x7fcdf002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8890961976cf8df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:01.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 1 -- 172.21.15.50:0/4223464936 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf002e070 con 0x7fcdf41312b0 2023-10-30T12:21:01.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:01.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:01.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: dump: 2023-10-30T12:21:01.269 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:01.270 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:01.270 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.270 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.270 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:01.271 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:01.271 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:01.271 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:01.271 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:01.271 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:01.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: _finish_auth 0 2023-10-30T12:21:01.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: _check_auth_tickets 2023-10-30T12:21:01.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:01.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 1 -- 172.21.15.50:0/4223464936 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf00315a0 con 0x7fcdf41312b0 2023-10-30T12:21:01.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdfb39e640 5 monclient: authenticate success, global_id 204721 2023-10-30T12:21:01.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdfb39e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:01.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:01.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 1 -- 172.21.15.50:0/4223464936 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0031930 con 0x7fcdf41312b0 2023-10-30T12:21:01.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdfb39e640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:01.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdfb39e640 4 set_mon_vals no callback set 2023-10-30T12:21:01.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:01.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:21:01.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.250+0000 7fcdf939a640 10 monclient: dump: 2023-10-30T12:21:01.274 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:01.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:01.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.275 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.275 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:01.275 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:01.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:01.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:01.275 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:01.276 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:01.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 10 monclient: shutdown 2023-10-30T12:21:01.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:01.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 -- 172.21.15.50:0/4223464936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 msgr2=0x7fcdf406b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:01.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/4223464936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 0x7fcdf406b690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4073aa0 tx=0x7fcdf002b840 comp rx=0 tx=0).stop 2023-10-30T12:21:01.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 -- 172.21.15.50:0/4223464936 shutdown_connections 2023-10-30T12:21:01.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/4223464936 >> v2:172.21.15.50:3300/0 conn(0x7fcdf4131690 0x7fcdf40a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/4223464936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 0x7fcdf406b690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 -- 172.21.15.50:0/4223464936 >> 172.21.15.50:0/4223464936 conn(0x7fcdf40a4200 msgr2=0x7fcdf40a4610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:01.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 -- 172.21.15.50:0/4223464936 shutdown_connections 2023-10-30T12:21:01.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.251+0000 7fcdfb39e640 1 -- 172.21.15.50:0/4223464936 wait complete. 2023-10-30T12:21:01.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 10 monclient: build_initial_monmap 2023-10-30T12:21:01.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 10 monclient: monmap: 2023-10-30T12:21:01.278 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:01.278 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:01.279 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:01.279 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:01.279 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:01.279 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:01.279 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:01.279 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:01.280 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:01.280 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:01.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 1 librados: starting msgr at 2023-10-30T12:21:01.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 1 librados: starting objecter 2023-10-30T12:21:01.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 1 Processor -- start 2023-10-30T12:21:01.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 1 -- start start 2023-10-30T12:21:01.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 1 librados: setting wanted keys 2023-10-30T12:21:01.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 1 librados: calling monclient init 2023-10-30T12:21:01.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 10 monclient: init 2023-10-30T12:21:01.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:01.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:21:01.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.252+0000 7fcdfb39e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 0x7fcdf409d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:01.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf41312b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:01.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 0x7fcdf4130050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:01.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf4131690 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:01.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdf4131ae0 0x7fcdf4132e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:01.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf4131ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:01.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:01.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf4135d10 con 0x7fcdf4131690 2023-10-30T12:21:01.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:01.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf4134c90 con 0x7fcdf41312b0 2023-10-30T12:21:01.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:01.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41356c0 con 0x7fcdf4131ae0 2023-10-30T12:21:01.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:01.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfb39e640 10 monclient(hunting): authenticate will time out at 2106.436078s 2023-10-30T12:21:01.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 0x7fcdf4130050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 0x7fcdf4130050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37320/0 (socket says 172.21.15.50:37320) 2023-10-30T12:21:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 1 -- 172.21.15.50:0/3286752550 learned_addr learned my addr 172.21.15.50:0/3286752550 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdf4131ae0 0x7fcdf4132e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 10 monclient(hunting): get_auth_request con 0x7fcdf4131690 auth_method 0 2023-10-30T12:21:01.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): get_auth_request con 0x7fcdf4131ae0 auth_method 0 2023-10-30T12:21:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:01.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 0x7fcdf409d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 1 -- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 msgr2=0x7fcdf4130050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:21:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 1 -- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 msgr2=0x7fcdf4130050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:21:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 0x7fcdf4130050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:01.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 10 monclient(hunting): get_auth_request con 0x7fcdf41312b0 auth_method 0 2023-10-30T12:21:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:01.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdf9b9b640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 0x7fcdf4130050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfa39c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:01.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient(hunting): handle_auth_done global_id 254174 payload 824 2023-10-30T12:21:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 -- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 msgr2=0x7fcdf409d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 0x7fcdf409d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 -- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 msgr2=0x7fcdf4130050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:21:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 0x7fcdf4130050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient: _finish_hunting 0 2023-10-30T12:21:01.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 monclient: found mon.noname-c 2023-10-30T12:21:01.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:01.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:21:01.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 -- 172.21.15.50:0/3286752550 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf00093a0 con 0x7fcdf4131ae0 2023-10-30T12:21:01.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdfab9d640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdf4131ae0 0x7fcdf4132e10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c690 tx=0x7fcdec00cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:01.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdeaffd640 10 client.?.objecter ms_handle_connect 0x7fcdf4131ae0 2023-10-30T12:21:01.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdeaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:01.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdeaffd640 1 -- 172.21.15.50:0/3286752550 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec003b70 con 0x7fcdf4131ae0 2023-10-30T12:21:01.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:01.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:01.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.253+0000 7fcdeaffd640 10 monclient: dump: 2023-10-30T12:21:01.292 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:01.293 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:01.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:01.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:01.294 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:01.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:01.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:01.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:01.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 10 monclient: _finish_auth 0 2023-10-30T12:21:01.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:21:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 1 -- 172.21.15.50:0/3286752550 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec003cd0 con 0x7fcdf4131ae0 2023-10-30T12:21:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 5 monclient: authenticate success, global_id 254174 2023-10-30T12:21:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 10 monclient: _renew_subs 2023-10-30T12:21:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:01.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf41359a0 con 0x7fcdf4131ae0 2023-10-30T12:21:01.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:01.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 1 -- 172.21.15.50:0/3286752550 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec005310 con 0x7fcdf4131ae0 2023-10-30T12:21:01.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:01.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:01.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 10 monclient: dump: 2023-10-30T12:21:01.297 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:01.297 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:01.297 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.297 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:01.297 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:01.298 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:01.298 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:01.298 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:01.298 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:01.298 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:01.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdf8b99640 4 set_mon_vals no callback set 2023-10-30T12:21:01.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 10 client.254174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:01.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 10 monclient: _renew_subs 2023-10-30T12:21:01.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:01.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf409e4a0 con 0x7fcdf4131ae0 2023-10-30T12:21:01.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 1 librados: init done 2023-10-30T12:21:01.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdeaffd640 1 -- 172.21.15.50:0/3286752550 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcdec005470 con 0x7fcdf4131ae0 2023-10-30T12:21:01.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.254+0000 7fcdfb39e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:01.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdfb39e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:01.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdfb39e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:01.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdeaffd640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdd80392f0 0x7fcdd803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:01.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf409a830 con 0x7fcdf4131ae0 2023-10-30T12:21:01.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdeaffd640 1 -- 172.21.15.50:0/3286752550 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(652..652 src has 520..652) v4 ==== 6678+0+0 (secure 0 0 0) 0x7fcdec04aaf0 con 0x7fcdf4131ae0 2023-10-30T12:21:01.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdeaffd640 10 client.254174.objecter ms_dispatch 0x7fcdf4000b90 osd_map(652..652 src has 520..652) v4 2023-10-30T12:21:01.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdeaffd640 3 client.254174.objecter handle_osd_map got epochs [652,652] > 0 2023-10-30T12:21:01.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdeaffd640 3 client.254174.objecter handle_osd_map decoding full epoch 652 2023-10-30T12:21:01.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdeaffd640 20 client.254174.objecter dump_active .. 0 homeless 2023-10-30T12:21:01.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdfa39c640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdd80392f0 0x7fcdd803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:01.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdfa39c640 10 monclient: get_auth_request con 0x7fcdd80392f0 auth_method 0 2023-10-30T12:21:01.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdeaffd640 10 client.254174.objecter ms_handle_connect 0x7fcdd80392f0 2023-10-30T12:21:01.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.255+0000 7fcdfa39c640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdd80392f0 0x7fcdd803b7b0 secure :-1 s=READY pgs=4321 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0036000 tx=0x7fcdf003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:01.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.259+0000 7fcdeaffd640 1 -- 172.21.15.50:0/3286752550 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcdec003e40 con 0x7fcdf4131ae0 2023-10-30T12:21:01.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.259+0000 7fcdeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:01.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.259+0000 7fcdeaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:01.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.505+0000 7fcdfb39e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:21:01.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.505+0000 7fcdfb39e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:01.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.505+0000 7fcdfb39e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:01.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.505+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcdf4135e80 con 0x7fcdf4131ae0 2023-10-30T12:21:01.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.506+0000 7fcdeaffd640 1 -- 172.21.15.50:0/3286752550 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fcdec011380 con 0x7fcdf4131ae0 2023-10-30T12:21:01.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.506+0000 7fcdeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:01.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.506+0000 7fcdeaffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:01.508 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":206,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":26,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:21:01.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.507+0000 7fcdfb39e640 10 librados: watch_flush enter 2023-10-30T12:21:01.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.507+0000 7fcdfb39e640 10 librados: watch_flush exit 2023-10-30T12:21:01.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 20 client.254174.objecter shutdown clearing up homeless session... 2023-10-30T12:21:01.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 10 client.254174.objecter successfully canceled tick 2023-10-30T12:21:01.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdd80392f0 msgr2=0x7fcdd803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:01.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdd80392f0 0x7fcdd803b7b0 secure :-1 s=READY pgs=4321 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0036000 tx=0x7fcdf003d040 comp rx=0 tx=0).stop 2023-10-30T12:21:01.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 10 monclient: shutdown 2023-10-30T12:21:01.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:01.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdf4131ae0 msgr2=0x7fcdf4132e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:01.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdf4131ae0 0x7fcdf4132e10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c690 tx=0x7fcdec00cb50 comp rx=0 tx=0).stop 2023-10-30T12:21:01.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 shutdown_connections 2023-10-30T12:21:01.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcdd80392f0 0x7fcdd803b7b0 unknown :-1 s=CLOSED pgs=4321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcdf4131ae0 0x7fcdf4132e10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcdf4131690 0x7fcdf4130050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 --2- 172.21.15.50:0/3286752550 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdf41312b0 0x7fcdf409d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:01.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 >> 172.21.15.50:0/3286752550 conn(0x7fcdf40a4200 msgr2=0x7fcdf40a6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:01.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 shutdown_connections 2023-10-30T12:21:01.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 -- 172.21.15.50:0/3286752550 wait complete. 2023-10-30T12:21:01.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:01.508+0000 7fcdfb39e640 1 librados: shutdown 2023-10-30T12:21:04.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:21:04.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.635+0000 7f3e3dcf2640 10 monclient: get_monmap_and_config 2023-10-30T12:21:04.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.635+0000 7f3e3dcf2640 10 monclient: build_initial_monmap 2023-10-30T12:21:04.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.635+0000 7f3e3dcf2640 10 monclient: monmap: 2023-10-30T12:21:04.637 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:04.637 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:04.638 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:04.636321+0000 2023-10-30T12:21:04.638 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:04.636321+0000 2023-10-30T12:21:04.638 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:04.638 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:04.638 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:04.639 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:04.639 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:04.639 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:04.639 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:04.639 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:04.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 1 Processor -- start 2023-10-30T12:21:04.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 1 -- start start 2023-10-30T12:21:04.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient: init 2023-10-30T12:21:04.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:04.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-10-30T12:21:04.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e380a4b60 0x7f3e381326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:04.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380a4b60 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:04.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e38099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:04.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e38132be0 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:04.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380a4f40 con 0x7f3e38099a40 2023-10-30T12:21:04.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380a4150 con 0x7f3e38132be0 2023-10-30T12:21:04.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:04.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380a3d70 con 0x7f3e380a4b60 2023-10-30T12:21:04.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:04.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e3dcf2640 10 monclient(hunting): authenticate will time out at 2109.819691s 2023-10-30T12:21:04.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e380a4b60 0x7f3e381326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:04.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e36ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3e38099a40 0x7f3e380a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48568/0 (socket says 172.21.15.50:48568) 2023-10-30T12:21:04.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e380a4b60 0x7f3e381326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37334/0 (socket says 172.21.15.50:37334) 2023-10-30T12:21:04.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e36ffd640 1 -- 172.21.15.50:0/2866167442 learned_addr learned my addr 172.21.15.50:0/2866167442 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:04.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 10 monclient(hunting): get_auth_request con 0x7f3e380a4b60 auth_method 0 2023-10-30T12:21:04.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:04.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:04.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:04.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:04.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:04.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.636+0000 7f3e377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:04.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 10 monclient(hunting): handle_auth_done global_id 264099 payload 274 2023-10-30T12:21:04.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 1 -- 172.21.15.50:0/2866167442 >> v1:172.21.15.118:6789/0 conn(0x7f3e38132be0 legacy=0x7f3e380a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:04.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 1 -- 172.21.15.50:0/2866167442 >> v1:172.21.15.50:6789/0 conn(0x7f3e38099a40 legacy=0x7f3e380a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:04.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 10 monclient: _finish_hunting 0 2023-10-30T12:21:04.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 1 monclient: found mon.noname-a 2023-10-30T12:21:04.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:04.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:04.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 1 -- 172.21.15.50:0/2866167442 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380a3170 con 0x7f3e380a4b60 2023-10-30T12:21:04.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.637+0000 7f3e377fe640 1 --2- 172.21.15.50:0/2866167442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e380a4b60 0x7f3e381326a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3e280097a0 tx=0x7f3e2802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ade0965c32b9a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 1 -- 172.21.15.50:0/2866167442 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2802f050 con 0x7f3e380a4b60 2023-10-30T12:21:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: dump: 2023-10-30T12:21:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:04.656 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:04.656 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:04.656 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:04.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: _finish_auth 0 2023-10-30T12:21:04.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: _check_auth_tickets 2023-10-30T12:21:04.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:04.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 5 monclient: authenticate success, global_id 264099 2023-10-30T12:21:04.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 1 -- 172.21.15.50:0/2866167442 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28031700 con 0x7f3e380a4b60 2023-10-30T12:21:04.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:04.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:04.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 4 set_mon_vals no callback set 2023-10-30T12:21:04.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 1 -- 172.21.15.50:0/2866167442 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28031a90 con 0x7f3e380a4b60 2023-10-30T12:21:04.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:04.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:04.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e367fc640 10 monclient: dump: 2023-10-30T12:21:04.658 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:04.659 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:04.659 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.659 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.659 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:04.659 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:04.659 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:04.660 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:04.661 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:04.661 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:04.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 10 monclient: shutdown 2023-10-30T12:21:04.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:04.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/2866167442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e380a4b60 msgr2=0x7f3e381326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:04.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/2866167442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e380a4b60 0x7f3e381326a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3e280097a0 tx=0x7f3e2802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/2866167442 shutdown_connections 2023-10-30T12:21:04.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/2866167442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e380a4b60 0x7f3e381326a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/2866167442 >> 172.21.15.50:0/2866167442 conn(0x7f3e380a8060 msgr2=0x7f3e380a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/2866167442 shutdown_connections 2023-10-30T12:21:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.638+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/2866167442 wait complete. 2023-10-30T12:21:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 10 monclient: build_initial_monmap 2023-10-30T12:21:04.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 10 monclient: monmap: 2023-10-30T12:21:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:04.664 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:04.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:04.665 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:04.665 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:04.665 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:04.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 1 librados: starting msgr at 2023-10-30T12:21:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 1 librados: starting objecter 2023-10-30T12:21:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 1 Processor -- start 2023-10-30T12:21:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 1 -- start start 2023-10-30T12:21:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 1 librados: setting wanted keys 2023-10-30T12:21:04.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 1 librados: calling monclient init 2023-10-30T12:21:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.639+0000 7f3e3dcf2640 10 monclient: init 2023-10-30T12:21:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:21:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e380a4b60 0x7f3e38136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): picked mon.noname-b con 0x7f3e380a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:04.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e38132be0 0x7f3e38137440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): picked mon.noname-a con 0x7f3e38132be0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e3813b120 0x7f3e38137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): picked mon.noname-c con 0x7f3e3813b120 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:04.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380a0640 con 0x7f3e38132be0 2023-10-30T12:21:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380a18d0 con 0x7f3e380a4b60 2023-10-30T12:21:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3813ad00 con 0x7f3e3813b120 2023-10-30T12:21:04.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:04.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e3dcf2640 10 monclient(hunting): authenticate will time out at 2109.823350s 2023-10-30T12:21:04.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e38132be0 0x7f3e38137440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:04.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e38132be0 0x7f3e38137440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37344/0 (socket says 172.21.15.50:37344) 2023-10-30T12:21:04.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e3813b120 0x7f3e38137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:04.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 -- 172.21.15.50:0/71033885 learned_addr learned my addr 172.21.15.50:0/71033885 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:04.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): get_auth_request con 0x7f3e38132be0 auth_method 0 2023-10-30T12:21:04.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:04.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:04.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:04.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 10 monclient(hunting): get_auth_request con 0x7f3e3813b120 auth_method 0 2023-10-30T12:21:04.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:04.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:04.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:04.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:04.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:04.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:04.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:04.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:04.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:04.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e380a4b60 0x7f3e38136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:04.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 10 monclient(hunting): get_auth_request con 0x7f3e380a4b60 auth_method 0 2023-10-30T12:21:04.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:04.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:04.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:04.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:04.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:04.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:04.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient(hunting): handle_auth_done global_id 264105 payload 824 2023-10-30T12:21:04.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 -- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e3813b120 msgr2=0x7f3e38137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:04.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e3813b120 0x7f3e38137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 -- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e380a4b60 msgr2=0x7f3e38136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:04.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e380a4b60 0x7f3e38136f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:21:04.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 monclient: found mon.noname-a 2023-10-30T12:21:04.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:04.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:04.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e36ffd640 1 -- 172.21.15.50:0/71033885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28009400 con 0x7f3e38132be0 2023-10-30T12:21:04.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.640+0000 7f3e37fff640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e3813b120 0x7f3e38137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:04.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e36ffd640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e38132be0 0x7f3e38137440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c3c0 tx=0x7f3e2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:04.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 client.?.objecter ms_handle_connect 0x7f3e38132be0 2023-10-30T12:21:04.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:04.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 1 -- 172.21.15.50:0/71033885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c005ce0 con 0x7f3e38132be0 2023-10-30T12:21:04.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:04.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:04.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: dump: 2023-10-30T12:21:04.678 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:04.678 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:04.678 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.679 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.679 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:04.679 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:04.679 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:04.679 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:04.679 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:04.680 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:04.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: _finish_auth 0 2023-10-30T12:21:04.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:21:04.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:04.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 1 -- 172.21.15.50:0/71033885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c005e40 con 0x7f3e38132be0 2023-10-30T12:21:04.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 5 monclient: authenticate success, global_id 264105 2023-10-30T12:21:04.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 10 monclient: _renew_subs 2023-10-30T12:21:04.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:04.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3813a460 con 0x7f3e38132be0 2023-10-30T12:21:04.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:04.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 1 -- 172.21.15.50:0/71033885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0072d0 con 0x7f3e38132be0 2023-10-30T12:21:04.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:04.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:04.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e1ffff640 10 monclient: dump: 2023-10-30T12:21:04.682 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:04.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:04.683 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.683 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:04.683 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:04.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:04.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:04.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:04.684 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:04.684 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:04.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 10 client.264105.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:04.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e35ffb640 4 set_mon_vals no callback set 2023-10-30T12:21:04.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 10 monclient: _renew_subs 2023-10-30T12:21:04.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:04.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e38137ec0 con 0x7f3e38132be0 2023-10-30T12:21:04.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.641+0000 7f3e3dcf2640 1 librados: init done 2023-10-30T12:21:04.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e1ffff640 1 -- 172.21.15.50:0/71033885 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3e2c007430 con 0x7f3e38132be0 2023-10-30T12:21:04.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e1ffff640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e24039320 0x7f3e2403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:04.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e3dcf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:04.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e3dcf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:04.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e3dcf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:04.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e1ffff640 1 -- 172.21.15.50:0/71033885 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(652..652 src has 520..652) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f3e2c04a210 con 0x7f3e38132be0 2023-10-30T12:21:04.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e1ffff640 10 client.264105.objecter ms_dispatch 0x7f3e38000b90 osd_map(652..652 src has 520..652) v4 2023-10-30T12:21:04.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e1ffff640 3 client.264105.objecter handle_osd_map got epochs [652,652] > 0 2023-10-30T12:21:04.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e1ffff640 3 client.264105.objecter handle_osd_map decoding full epoch 652 2023-10-30T12:21:04.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3813a6a0 con 0x7f3e38132be0 2023-10-30T12:21:04.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e1ffff640 20 client.264105.objecter dump_active .. 0 homeless 2023-10-30T12:21:04.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e377fe640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e24039320 0x7f3e2403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:04.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e377fe640 10 monclient: get_auth_request con 0x7f3e24039320 auth_method 0 2023-10-30T12:21:04.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.643+0000 7f3e1ffff640 10 client.264105.objecter ms_handle_connect 0x7f3e24039320 2023-10-30T12:21:04.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.642+0000 7f3e377fe640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e24039320 0x7f3e2403b7e0 secure :-1 s=READY pgs=4322 cs=0 l=1 rev1=1 crypto rx=0x7f3e28035000 tx=0x7f3e28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:04.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.647+0000 7f3e1ffff640 1 -- 172.21.15.50:0/71033885 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e2c017070 con 0x7f3e38132be0 2023-10-30T12:21:04.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.647+0000 7f3e1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:04.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.647+0000 7f3e1ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:04.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.895+0000 7f3e3dcf2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:21:04.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.895+0000 7f3e3dcf2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:04.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.895+0000 7f3e3dcf2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:04.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.895+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3e3804e160 con 0x7f3e38132be0 2023-10-30T12:21:04.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.896+0000 7f3e1ffff640 1 -- 172.21.15.50:0/71033885 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f3e2c012400 con 0x7f3e38132be0 2023-10-30T12:21:04.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.896+0000 7f3e1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:04.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.896+0000 7f3e1ffff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":210,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:21:04.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 10 librados: watch_flush enter 2023-10-30T12:21:04.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 10 librados: watch_flush exit 2023-10-30T12:21:04.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 20 client.264105.objecter shutdown clearing up homeless session... 2023-10-30T12:21:04.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 10 client.264105.objecter successfully canceled tick 2023-10-30T12:21:04.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e24039320 msgr2=0x7f3e2403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:04.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e24039320 0x7f3e2403b7e0 secure :-1 s=READY pgs=4322 cs=0 l=1 rev1=1 crypto rx=0x7f3e28035000 tx=0x7f3e28008030 comp rx=0 tx=0).stop 2023-10-30T12:21:04.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 10 monclient: shutdown 2023-10-30T12:21:04.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:04.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e38132be0 msgr2=0x7f3e38137440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:04.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e38132be0 0x7f3e38137440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c3c0 tx=0x7f3e2c00c880 comp rx=0 tx=0).stop 2023-10-30T12:21:04.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 shutdown_connections 2023-10-30T12:21:04.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e24039320 0x7f3e2403b7e0 unknown :-1 s=CLOSED pgs=4322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e3813b120 0x7f3e38137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e38132be0 0x7f3e38137440 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 --2- 172.21.15.50:0/71033885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e380a4b60 0x7f3e38136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:04.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 >> 172.21.15.50:0/71033885 conn(0x7f3e380a8060 msgr2=0x7f3e380a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:04.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 shutdown_connections 2023-10-30T12:21:04.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 -- 172.21.15.50:0/71033885 wait complete. 2023-10-30T12:21:04.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:04.898+0000 7f3e3dcf2640 1 librados: shutdown 2023-10-30T12:21:04.910 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:21:04.910 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:21:04.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:21:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.015+0000 7f484aca2640 10 monclient: get_monmap_and_config 2023-10-30T12:21:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.015+0000 7f484aca2640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.015+0000 7f484aca2640 10 monclient: monmap: 2023-10-30T12:21:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:05.016335+0000 2023-10-30T12:21:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:05.016335+0000 2023-10-30T12:21:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:05.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.015+0000 7f484aca2640 1 Processor -- start 2023-10-30T12:21:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 1 -- start start 2023-10-30T12:21:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient: init 2023-10-30T12:21:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-10-30T12:21:05.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844132970 0x7f4844132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient(hunting): picked mon.noname-b con 0x7f4844132970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48440a03d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4844070ee0 0x7f484413db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient(hunting): picked mon.noname-a con 0x7f4844070ee0 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48440a17d0 con 0x7f48440a03d0 2023-10-30T12:21:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48440a1510 con 0x7f4844070ee0 2023-10-30T12:21:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484409f680 con 0x7f4844132970 2023-10-30T12:21:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484aca2640 10 monclient(hunting): authenticate will time out at 2110.199516s 2023-10-30T12:21:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844132970 0x7f4844132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4844070ee0 0x7f484413db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4843fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f48440a03d0 0x7f48440707d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48570/0 (socket says 172.21.15.50:48570) 2023-10-30T12:21:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844132970 0x7f4844132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39100/0 (socket says 172.21.15.50:39100) 2023-10-30T12:21:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4843fff640 1 -- 172.21.15.50:0/3069237921 learned_addr learned my addr 172.21.15.50:0/3069237921 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): get_auth_request con 0x7f4844070ee0 auth_method 0 2023-10-30T12:21:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 10 monclient(hunting): get_auth_request con 0x7f4844132970 auth_method 0 2023-10-30T12:21:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 10 monclient(hunting): handle_auth_done global_id 264108 payload 274 2023-10-30T12:21:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 1 -- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844132970 msgr2=0x7f4844132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 1 --2- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844132970 0x7f4844132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.016+0000 7f484923f640 1 -- 172.21.15.50:0/3069237921 >> v1:172.21.15.50:6789/0 conn(0x7f48440a03d0 legacy=0x7f48440707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484923f640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484923f640 1 monclient: found mon.noname-a 2023-10-30T12:21:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484923f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484923f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 1 -- 172.21.15.50:0/3069237921 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227064304 0 0) 0x7f48440a17d0 con 0x7f48440a03d0 2023-10-30T12:21:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484923f640 1 -- 172.21.15.50:0/3069237921 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484409f960 con 0x7f4844070ee0 2023-10-30T12:21:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:21:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844132970 0x7f4844132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484923f640 1 --2- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844070ee0 0x7f484413db00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f482c0097a0 tx=0x7f482c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4702d145f7e89fac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 1 -- 172.21.15.50:0/3069237921 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c0316f0 con 0x7f4844070ee0 2023-10-30T12:21:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: dump: 2023-10-30T12:21:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.037 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.038 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484aca2640 5 monclient: authenticate success, global_id 264108 2023-10-30T12:21:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484aca2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 1 -- 172.21.15.50:0/3069237921 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c031850 con 0x7f4844070ee0 2023-10-30T12:21:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 1 -- 172.21.15.50:0/3069237921 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c031be0 con 0x7f4844070ee0 2023-10-30T12:21:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f48437fe640 10 monclient: dump: 2023-10-30T12:21:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.040 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.041 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.042 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484aca2640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.017+0000 7f484aca2640 4 set_mon_vals no callback set 2023-10-30T12:21:05.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 10 monclient: shutdown 2023-10-30T12:21:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 -- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844070ee0 msgr2=0x7f484413db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 --2- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844070ee0 0x7f484413db00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f482c0097a0 tx=0x7f482c02b790 comp rx=0 tx=0).stop 2023-10-30T12:21:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 -- 172.21.15.50:0/3069237921 shutdown_connections 2023-10-30T12:21:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 --2- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844070ee0 0x7f484413db00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 --2- 172.21.15.50:0/3069237921 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844132970 0x7f4844132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 -- 172.21.15.50:0/3069237921 >> 172.21.15.50:0/3069237921 conn(0x7f48440a4800 msgr2=0x7f48440a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 -- 172.21.15.50:0/3069237921 shutdown_connections 2023-10-30T12:21:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.018+0000 7f484aca2640 1 -- 172.21.15.50:0/3069237921 wait complete. 2023-10-30T12:21:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 10 monclient: monmap: 2023-10-30T12:21:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.045 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:05.046 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 1 librados: starting msgr at 2023-10-30T12:21:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 1 librados: starting objecter 2023-10-30T12:21:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 1 Processor -- start 2023-10-30T12:21:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 1 -- start start 2023-10-30T12:21:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 1 librados: setting wanted keys 2023-10-30T12:21:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 1 librados: calling monclient init 2023-10-30T12:21:05.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.019+0000 7f484aca2640 10 monclient: init 2023-10-30T12:21:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:21:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844070ee0 0x7f484412fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): picked mon.noname-c con 0x7f4844070ee0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844132970 0x7f48441c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): picked mon.noname-a con 0x7f4844132970 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f48441c7290 0x7f48441ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): picked mon.noname-b con 0x7f48441c7290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484411bbc0 con 0x7f4844132970 2023-10-30T12:21:05.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484411c370 con 0x7f48441c7290 2023-10-30T12:21:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484409d7c0 con 0x7f4844070ee0 2023-10-30T12:21:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient(hunting): authenticate will time out at 2110.203131s 2023-10-30T12:21:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844070ee0 0x7f484412fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844132970 0x7f48441c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844070ee0 0x7f484412fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39108/0 (socket says 172.21.15.50:39108) 2023-10-30T12:21:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844132970 0x7f48441c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37366/0 (socket says 172.21.15.50:37366) 2023-10-30T12:21:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 1 -- 172.21.15.50:0/3557923007 learned_addr learned my addr 172.21.15.50:0/3557923007 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 10 monclient(hunting): get_auth_request con 0x7f4844070ee0 auth_method 0 2023-10-30T12:21:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): get_auth_request con 0x7f4844132970 auth_method 0 2023-10-30T12:21:05.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient(hunting): handle_auth_done global_id 264114 payload 824 2023-10-30T12:21:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844070ee0 msgr2=0x7f484412fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4844070ee0 0x7f484412fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f48441c7290 msgr2=0x7f48441ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f48441c7290 0x7f48441ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 monclient: found mon.noname-a 2023-10-30T12:21:05.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 -- 172.21.15.50:0/3557923007 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c009a70 con 0x7f4844132970 2023-10-30T12:21:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4843fff640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844132970 0x7f48441c6d50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f483400c450 tx=0x7f483400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 client.?.objecter ms_handle_connect 0x7f4844132970 2023-10-30T12:21:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:05.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48340074e0 con 0x7f4844132970 2023-10-30T12:21:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: dump: 2023-10-30T12:21:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.057 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.058 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.059 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4834007640 con 0x7f4844132970 2023-10-30T12:21:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 5 monclient: authenticate success, global_id 264114 2023-10-30T12:21:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient: _renew_subs 2023-10-30T12:21:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c009400 con 0x7f4844132970 2023-10-30T12:21:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834004a20 con 0x7f4844132970 2023-10-30T12:21:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f48417fa640 10 monclient: dump: 2023-10-30T12:21:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.061 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.062 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.063 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f4842ffd640 4 set_mon_vals no callback set 2023-10-30T12:21:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 client.264114.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:05.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient: _renew_subs 2023-10-30T12:21:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484411e360 con 0x7f4844132970 2023-10-30T12:21:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.020+0000 7f484aca2640 1 librados: init done 2023-10-30T12:21:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f484aca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4834004b80 con 0x7f4844132970 2023-10-30T12:21:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f484aca2640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f484aca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484411e700 con 0x7f48440a03d0 2023-10-30T12:21:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient: get_auth_request con 0x7f48440a03d0 auth_method 0 2023-10-30T12:21:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f48417fa640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48240392f0 0x7f482403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(652..652 src has 520..652) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f4834049f20 con 0x7f4844132970 2023-10-30T12:21:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f48417fa640 10 client.264114.objecter ms_dispatch 0x7f4844000b90 osd_map(652..652 src has 520..652) v4 2023-10-30T12:21:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f48417fa640 3 client.264114.objecter handle_osd_map got epochs [652,652] > 0 2023-10-30T12:21:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f48417fa640 3 client.264114.objecter handle_osd_map decoding full epoch 652 2023-10-30T12:21:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f4848a3e640 10 monclient(hunting): handle_auth_done global_id 264117 payload 824 2023-10-30T12:21:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.021+0000 7f48417fa640 20 client.264114.objecter dump_active .. 0 homeless 2023-10-30T12:21:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f4848a3e640 1 -- 172.21.15.50:0/3557923007 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4844133290 con 0x7f48440a03d0 2023-10-30T12:21:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f484923f640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48240392f0 0x7f482403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f484923f640 10 monclient: get_auth_request con 0x7f48240392f0 auth_method 0 2023-10-30T12:21:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f4848a3e640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 msgr2=0x7f48441ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-10-30T12:21:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f4848a3e640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 msgr2=0x7f48441ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:21:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f483c011490 tx=0x7f483c011950 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f48417fa640 10 client.264114.objecter ms_handle_connect 0x7f48240392f0 2023-10-30T12:21:05.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.022+0000 7f484923f640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48240392f0 0x7f482403b7b0 secure :-1 s=READY pgs=4323 cs=0 l=1 rev1=1 crypto rx=0x7f482c036000 tx=0x7f482c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: get_auth_request con 0x7f48440a03d0 auth_method 0 2023-10-30T12:21:05.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: _init_auth method 2 2023-10-30T12:21:05.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: _init_auth already have auth, reseting 2023-10-30T12:21:05.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: handle_auth_reply_more payload 9 2023-10-30T12:21:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: handle_auth_reply_more responding with 132 bytes 2023-10-30T12:21:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 10 monclient: handle_auth_done global_id 264117 payload 293 2023-10-30T12:21:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f483c00f010 tx=0x7f483c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 10 client.264114.objecter ms_handle_connect 0x7f48440a03d0 2023-10-30T12:21:05.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 10 client.264114.objecter resend_mon_ops 2023-10-30T12:21:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c00e040 con 0x7f48440a03d0 2023-10-30T12:21:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 10 monclient: dump: 2023-10-30T12:21:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.229 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.230 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f483c00f040 con 0x7f48440a03d0 2023-10-30T12:21:05.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 msgr2=0x7f48441ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f483c00f010 tx=0x7f483c00b040 comp rx=0 tx=0).stop 2023-10-30T12:21:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.223+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 msgr2=0x7f48441ca7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:05.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.224+0000 7f48417fa640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48440a03d0 0x7f48441ca7f0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f483c00f010 tx=0x7f483c00b040 comp rx=0 tx=0).stop 2023-10-30T12:21:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f484aca2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f484aca2640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48441cad30 0x7f48440a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f484aca2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4844071a70 con 0x7f48441cad30 2023-10-30T12:21:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48441cad30 0x7f48440a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient: get_auth_request con 0x7f48441cad30 auth_method 0 2023-10-30T12:21:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 10 monclient(hunting): handle_auth_done global_id 264120 payload 824 2023-10-30T12:21:05.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.230+0000 7f4848a3e640 1 -- 172.21.15.50:0/3557923007 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f48441cb140 con 0x7f48441cad30 2023-10-30T12:21:05.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 10 client.264114.objecter ms_handle_connect 0x7f48441cad30 2023-10-30T12:21:05.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 10 client.264114.objecter resend_mon_ops 2023-10-30T12:21:05.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f4848a3e640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48441cad30 0x7f48440a22a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f483c024000 tx=0x7f483c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c030480 con 0x7f48441cad30 2023-10-30T12:21:05.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:05.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 10 monclient: dump: 2023-10-30T12:21:05.237 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.237 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.237 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.238 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.238 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.238 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.238 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.238 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.239 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f483c01f420 con 0x7f48441cad30 2023-10-30T12:21:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:05.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48441cad30 msgr2=0x7f48440a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48441cad30 0x7f48440a22a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f483c024000 tx=0x7f483c004ab0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48441cad30 msgr2=0x7f48440a22a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:05.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f48417fa640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48441cad30 0x7f48440a22a0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f483c024000 tx=0x7f483c004ab0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.240 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:05.241 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:05.241 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:05.241 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:21:05.241 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 210, 2023-10-30T12:21:05.242 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:05.242 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:05.242 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:21:05.242 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:05.242 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.242 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:21:05.243 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:05.243 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:05.243 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:05.243 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.243 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.244 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.244 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.244 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.244 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.244 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.245 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.245 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.245 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.245 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.245 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:05.246 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:05.246 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.246 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.246 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.246 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.247 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.247 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.247 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.247 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.247 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.247 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.248 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:05.248 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:05.248 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:05.249 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:05.249 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:05.249 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:05.249 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:05.249 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:05.250 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:05.250 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:05.250 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:05.250 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:05.250 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:05.251 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:05.251 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:05.251 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:05.251 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:05.251 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.252 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.252 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.252 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.252 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.252 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.253 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.253 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.253 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.253 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.253 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.254 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:05.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.254 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:05.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.254 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:05.255 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:05.255 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.255 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.255 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.255 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:05.256 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.256 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:05.257 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.257 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.257 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.257 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:05.258 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:05.258 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.258 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.258 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.259 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:05.259 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:05.259 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.259 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.260 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.260 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:05.260 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.261 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.261 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:05.261 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.261 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.262 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:05.262 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:05.262 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.262 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.264 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.264 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.264 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:05.264 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.265 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:05.265 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.266 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.266 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:05.266 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:05.267 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.267 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.267 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.267 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.267 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.269 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.269 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:05.269 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.269 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.269 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:05.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.270 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.270 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.270 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:21:05.270 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.271 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:05.271 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 10 librados: watch_flush enter 2023-10-30T12:21:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 10 librados: watch_flush exit 2023-10-30T12:21:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 20 client.264114.objecter shutdown clearing up homeless session... 2023-10-30T12:21:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 10 client.264114.objecter successfully canceled tick 2023-10-30T12:21:05.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48240392f0 msgr2=0x7f482403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f48240392f0 0x7f482403b7b0 secure :-1 s=READY pgs=4323 cs=0 l=1 rev1=1 crypto rx=0x7f482c036000 tx=0x7f482c030000 comp rx=0 tx=0).stop 2023-10-30T12:21:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 10 monclient: shutdown 2023-10-30T12:21:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844132970 msgr2=0x7f48441c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 --2- 172.21.15.50:0/3557923007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4844132970 0x7f48441c6d50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f483400c450 tx=0x7f483400c910 comp rx=0 tx=0).stop 2023-10-30T12:21:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484923f640 1 -- 172.21.15.50:0/3557923007 reap_dead start 2023-10-30T12:21:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484923f640 1 -- 172.21.15.50:0/3557923007 reap_dead start 2023-10-30T12:21:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 shutdown_connections 2023-10-30T12:21:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 >> 172.21.15.50:0/3557923007 conn(0x7f48440a4800 msgr2=0x7f4844131ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 shutdown_connections 2023-10-30T12:21:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 -- 172.21.15.50:0/3557923007 wait complete. 2023-10-30T12:21:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.231+0000 7f484aca2640 1 librados: shutdown 2023-10-30T12:21:05.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:21:05.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:21:05.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:21:05.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.382+0000 7f249e925640 10 monclient: get_monmap_and_config 2023-10-30T12:21:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.382+0000 7f249e925640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.382+0000 7f249e925640 10 monclient: monmap: 2023-10-30T12:21:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.384 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:05.383478+0000 2023-10-30T12:21:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:05.383478+0000 2023-10-30T12:21:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:05.386 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:05.386 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 1 Processor -- start 2023-10-30T12:21:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 1 -- start start 2023-10-30T12:21:05.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient: init 2023-10-30T12:21:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-10-30T12:21:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2498071c00 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2498097fe0 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249809a150 0x7f249813a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient(hunting): picked mon.noname-b con 0x7f249809a150 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249809b880 con 0x7f2498097fe0 2023-10-30T12:21:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2498097600 con 0x7f2498071c00 2023-10-30T12:21:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249809a5e0 con 0x7f249809a150 2023-10-30T12:21:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249e925640 10 monclient(hunting): authenticate will time out at 2110.566614s 2023-10-30T12:21:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249809a150 0x7f249813a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f24977fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2498097fe0 0x7f249809b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48586/0 (socket says 172.21.15.50:48586) 2023-10-30T12:21:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f24977fe640 1 -- 172.21.15.50:0/8750530 learned_addr learned my addr 172.21.15.50:0/8750530 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 10 monclient(hunting): get_auth_request con 0x7f249809a150 auth_method 0 2023-10-30T12:21:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.383+0000 7f249cec2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 10 monclient(hunting): handle_auth_done global_id 254186 payload 274 2023-10-30T12:21:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 1 -- 172.21.15.50:0/8750530 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707478353 0 0) 0x7f249809b880 con 0x7f2498097fe0 2023-10-30T12:21:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 1 -- 172.21.15.50:0/8750530 >> v1:172.21.15.118:6789/0 conn(0x7f2498071c00 legacy=0x7f249809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 1 -- 172.21.15.50:0/8750530 >> v1:172.21.15.50:6789/0 conn(0x7f2498097fe0 legacy=0x7f249809b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 1 monclient: found mon.noname-b 2023-10-30T12:21:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 1 -- 172.21.15.50:0/8750530 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980978e0 con 0x7f249809a150 2023-10-30T12:21:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:21:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249cec2640 1 --2- 172.21.15.50:0/8750530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249809a150 0x7f249813a770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2490009dc0 tx=0x7f249002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c081845dec386b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 1 -- 172.21.15.50:0/8750530 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249002cbb0 con 0x7f249809a150 2023-10-30T12:21:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:21:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: dump: 2023-10-30T12:21:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.403 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249e925640 5 monclient: authenticate success, global_id 254186 2023-10-30T12:21:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 1 -- 172.21.15.50:0/8750530 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249002cd10 con 0x7f249809a150 2023-10-30T12:21:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249e925640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f249e925640 4 set_mon_vals no callback set 2023-10-30T12:21:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 1 -- 172.21.15.50:0/8750530 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490034660 con 0x7f249809a150 2023-10-30T12:21:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.384+0000 7f2496ffd640 10 monclient: dump: 2023-10-30T12:21:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.407 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 10 monclient: shutdown 2023-10-30T12:21:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 1 -- 172.21.15.50:0/8750530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249809a150 msgr2=0x7f249813a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 1 --2- 172.21.15.50:0/8750530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249809a150 0x7f249813a770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2490009dc0 tx=0x7f249002be00 comp rx=0 tx=0).stop 2023-10-30T12:21:05.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 1 -- 172.21.15.50:0/8750530 shutdown_connections 2023-10-30T12:21:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 1 --2- 172.21.15.50:0/8750530 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249809a150 0x7f249813a770 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 1 -- 172.21.15.50:0/8750530 >> 172.21.15.50:0/8750530 conn(0x7f24980a3980 msgr2=0x7f24980a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 1 -- 172.21.15.50:0/8750530 shutdown_connections 2023-10-30T12:21:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.385+0000 7f249e925640 1 -- 172.21.15.50:0/8750530 wait complete. 2023-10-30T12:21:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient: monmap: 2023-10-30T12:21:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:05.411 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:05.411 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 librados: starting msgr at 2023-10-30T12:21:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 librados: starting objecter 2023-10-30T12:21:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 Processor -- start 2023-10-30T12:21:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 -- start start 2023-10-30T12:21:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 librados: setting wanted keys 2023-10-30T12:21:05.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 librados: calling monclient init 2023-10-30T12:21:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient: init 2023-10-30T12:21:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:21:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2498071c00 0x7f24981ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): picked mon.noname-c con 0x7f2498071c00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2498097fe0 0x7f24981b7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): picked mon.noname-b con 0x7f2498097fe0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249809a150 0x7f24981b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): picked mon.noname-a con 0x7f249809a150 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24981ca920 con 0x7f249809a150 2023-10-30T12:21:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249806db60 con 0x7f2498097fe0 2023-10-30T12:21:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24981b1a20 con 0x7f2498071c00 2023-10-30T12:21:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.386+0000 7f249e925640 10 monclient(hunting): authenticate will time out at 2110.570004s 2023-10-30T12:21:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2498071c00 0x7f24981ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249809a150 0x7f24981b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2498071c00 0x7f24981ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39130/0 (socket says 172.21.15.50:39130) 2023-10-30T12:21:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249809a150 0x7f24981b7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37400/0 (socket says 172.21.15.50:37400) 2023-10-30T12:21:05.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 -- 172.21.15.50:0/1039326433 learned_addr learned my addr 172.21.15.50:0/1039326433 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): get_auth_request con 0x7f2498071c00 auth_method 0 2023-10-30T12:21:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 10 monclient(hunting): get_auth_request con 0x7f249809a150 auth_method 0 2023-10-30T12:21:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f24977fe640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2498097fe0 0x7f24981b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f24977fe640 10 monclient(hunting): get_auth_request con 0x7f2498097fe0 auth_method 0 2023-10-30T12:21:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f24977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f24977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f24977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient(hunting): handle_auth_done global_id 254189 payload 824 2023-10-30T12:21:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2498097fe0 msgr2=0x7f24981b7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2498097fe0 0x7f24981b7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249809a150 msgr2=0x7f24981b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249809a150 0x7f24981b7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 monclient: found mon.noname-c 2023-10-30T12:21:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 -- 172.21.15.50:0/1039326433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2490009a70 con 0x7f2498071c00 2023-10-30T12:21:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249cec2640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249809a150 0x7f24981b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2497fff640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2498071c00 0x7f24981ca3e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f248800c400 tx=0x7f248800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 client.?.objecter ms_handle_connect 0x7f2498071c00 2023-10-30T12:21:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24880052d0 con 0x7f2498071c00 2023-10-30T12:21:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: dump: 2023-10-30T12:21:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488005470 con 0x7f2498071c00 2023-10-30T12:21:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 5 monclient: authenticate success, global_id 254189 2023-10-30T12:21:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 10 monclient: _renew_subs 2023-10-30T12:21:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24981b2910 con 0x7f2498071c00 2023-10-30T12:21:05.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488006a60 con 0x7f2498071c00 2023-10-30T12:21:05.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:05.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f2494ff9640 10 monclient: dump: 2023-10-30T12:21:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.429 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f24967fc640 4 set_mon_vals no callback set 2023-10-30T12:21:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 10 client.254189.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 10 monclient: _renew_subs 2023-10-30T12:21:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24981b2ca0 con 0x7f2498071c00 2023-10-30T12:21:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.387+0000 7f249e925640 1 librados: init done 2023-10-30T12:21:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f249e925640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2488006bc0 con 0x7f2498071c00 2023-10-30T12:21:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f249e925640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806bba0 0x7f249806bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f249e925640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2498073ff0 con 0x7f249806bba0 2023-10-30T12:21:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806bba0 0x7f249806bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient: get_auth_request con 0x7f249806bba0 auth_method 0 2023-10-30T12:21:05.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f2494ff9640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24840392f0 0x7f248403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(652..652 src has 520..652) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f248804a1f0 con 0x7f2498071c00 2023-10-30T12:21:05.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f2494ff9640 10 client.254189.objecter ms_dispatch 0x7f2498000b90 osd_map(652..652 src has 520..652) v4 2023-10-30T12:21:05.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f2494ff9640 3 client.254189.objecter handle_osd_map got epochs [652,652] > 0 2023-10-30T12:21:05.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f2494ff9640 3 client.254189.objecter handle_osd_map decoding full epoch 652 2023-10-30T12:21:05.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 10 monclient(hunting): handle_auth_done global_id 254192 payload 824 2023-10-30T12:21:05.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f2494ff9640 20 client.254189.objecter dump_active .. 0 homeless 2023-10-30T12:21:05.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.388+0000 7f24977fe640 1 -- 172.21.15.50:0/1039326433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f249809d9b0 con 0x7f249806bba0 2023-10-30T12:21:05.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f249cec2640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24840392f0 0x7f248403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 client.254189.objecter ms_handle_connect 0x7f249806bba0 2023-10-30T12:21:05.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f24977fe640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806bba0 0x7f249806bf80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2498070bf0 tx=0x7f248c001970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 client.254189.objecter resend_mon_ops 2023-10-30T12:21:05.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f249cec2640 10 monclient: get_auth_request con 0x7f24840392f0 auth_method 0 2023-10-30T12:21:05.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c012590 con 0x7f249806bba0 2023-10-30T12:21:05.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 monclient: dump: 2023-10-30T12:21:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.437 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.437 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.437 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.437 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.437 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f248c0126f0 con 0x7f249806bba0 2023-10-30T12:21:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806bba0 msgr2=0x7f249806bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806bba0 0x7f249806bf80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2498070bf0 tx=0x7f248c001970 comp rx=0 tx=0).stop 2023-10-30T12:21:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806bba0 msgr2=0x7f249806bf80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:05.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806bba0 0x7f249806bf80 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2498070bf0 tx=0x7f248c001970 comp rx=0 tx=0).stop 2023-10-30T12:21:05.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f249cec2640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24840392f0 0x7f248403b7b0 secure :-1 s=READY pgs=4324 cs=0 l=1 rev1=1 crypto rx=0x7f2490036000 tx=0x7f2490030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.389+0000 7f2494ff9640 10 client.254189.objecter ms_handle_connect 0x7f24840392f0 2023-10-30T12:21:05.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f249e925640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:05.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f249e925640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806c4c0 0x7f249806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f249e925640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2498073a20 con 0x7f249806c4c0 2023-10-30T12:21:05.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806c4c0 0x7f249806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 10 monclient: get_auth_request con 0x7f249806c4c0 auth_method 0 2023-10-30T12:21:05.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.395+0000 7f24977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f24977fe640 10 monclient(hunting): handle_auth_done global_id 254195 payload 824 2023-10-30T12:21:05.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f24977fe640 1 -- 172.21.15.50:0/1039326433 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f249809bf50 con 0x7f249806c4c0 2023-10-30T12:21:05.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f24977fe640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806c4c0 0x7f249806ddb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f248c017000 tx=0x7f248c007a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 10 client.254189.objecter ms_handle_connect 0x7f249806c4c0 2023-10-30T12:21:05.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 10 client.254189.objecter resend_mon_ops 2023-10-30T12:21:05.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c022070 con 0x7f249806c4c0 2023-10-30T12:21:05.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:05.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 10 monclient: dump: 2023-10-30T12:21:05.445 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.445 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.445 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.445 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.445 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.446 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.446 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.446 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.446 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.446 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f248c01d9c0 con 0x7f249806c4c0 2023-10-30T12:21:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806c4c0 msgr2=0x7f249806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806c4c0 0x7f249806ddb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f248c017000 tx=0x7f248c007a10 comp rx=0 tx=0).stop 2023-10-30T12:21:05.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806c4c0 msgr2=0x7f249806ddb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:05.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f2494ff9640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f249806c4c0 0x7f249806ddb0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f248c017000 tx=0x7f248c007a10 comp rx=0 tx=0).stop 2023-10-30T12:21:05.448 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 210, 2023-10-30T12:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:05.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 10 librados: watch_flush enter 2023-10-30T12:21:05.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 10 librados: watch_flush exit 2023-10-30T12:21:05.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 20 client.254189.objecter shutdown clearing up homeless session... 2023-10-30T12:21:05.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 10 client.254189.objecter successfully canceled tick 2023-10-30T12:21:05.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24840392f0 msgr2=0x7f248403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f24840392f0 0x7f248403b7b0 secure :-1 s=READY pgs=4324 cs=0 l=1 rev1=1 crypto rx=0x7f2490036000 tx=0x7f2490030000 comp rx=0 tx=0).stop 2023-10-30T12:21:05.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 10 monclient: shutdown 2023-10-30T12:21:05.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:05.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2498071c00 msgr2=0x7f24981ca3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 1 --2- 172.21.15.50:0/1039326433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2498071c00 0x7f24981ca3e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f248800c400 tx=0x7f248800c8c0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249cec2640 1 -- 172.21.15.50:0/1039326433 reap_dead start 2023-10-30T12:21:05.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249cec2640 1 -- 172.21.15.50:0/1039326433 reap_dead start 2023-10-30T12:21:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 shutdown_connections 2023-10-30T12:21:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.396+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 >> 172.21.15.50:0/1039326433 conn(0x7f24980a3980 msgr2=0x7f24980a1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.397+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 shutdown_connections 2023-10-30T12:21:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.397+0000 7f249e925640 1 -- 172.21.15.50:0/1039326433 wait complete. 2023-10-30T12:21:05.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.397+0000 7f249e925640 1 librados: shutdown 2023-10-30T12:21:05.487 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:21:05.487 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:21:05.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:21:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.592+0000 7ffbee3cc640 10 monclient: get_monmap_and_config 2023-10-30T12:21:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.592+0000 7ffbee3cc640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.592+0000 7ffbee3cc640 10 monclient: monmap: 2023-10-30T12:21:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.594 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:05.593466+0000 2023-10-30T12:21:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:05.593466+0000 2023-10-30T12:21:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.595 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:05.596 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:05.596 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 1 Processor -- start 2023-10-30T12:21:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 1 -- start start 2023-10-30T12:21:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient: init 2023-10-30T12:21:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-10-30T12:21:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbe80a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbe80a2450 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe813c120 0x7ffbe813e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe813c120 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe80a4190 con 0x7ffbe80a2ae0 2023-10-30T12:21:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe809a2e0 con 0x7ffbe80a2450 2023-10-30T12:21:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80a1630 con 0x7ffbe813c120 2023-10-30T12:21:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbee3cc640 10 monclient(hunting): authenticate will time out at 2110.776631s 2023-10-30T12:21:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbe7fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ffbe80a2ae0 0x7ffbe80a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48590/0 (socket says 172.21.15.50:48590) 2023-10-30T12:21:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbe7fff640 1 -- 172.21.15.50:0/1354176417 learned_addr learned my addr 172.21.15.50:0/1354176417 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 1 --2- 172.21.15.50:0/1354176417 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe813c120 0x7ffbe813e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 10 monclient(hunting): get_auth_request con 0x7ffbe813c120 auth_method 0 2023-10-30T12:21:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.593+0000 7ffbec969640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 10 monclient(hunting): handle_auth_done global_id 254198 payload 274 2023-10-30T12:21:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 1 -- 172.21.15.50:0/1354176417 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810268963 0 0) 0x7ffbd8001110 con 0x7ffbe80a2ae0 2023-10-30T12:21:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 1 -- 172.21.15.50:0/1354176417 >> v1:172.21.15.118:6789/0 conn(0x7ffbe80a2450 legacy=0x7ffbe813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 1 -- 172.21.15.50:0/1354176417 >> v1:172.21.15.50:6789/0 conn(0x7ffbe80a2ae0 legacy=0x7ffbe80a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 1 monclient: found mon.noname-b 2023-10-30T12:21:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 1 -- 172.21.15.50:0/1354176417 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe809a8a0 con 0x7ffbe813c120 2023-10-30T12:21:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:21:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbec969640 1 --2- 172.21.15.50:0/1354176417 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe813c120 0x7ffbe813e510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009dc0 tx=0x7ffbe002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=733c0e590ab56f2c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 1 -- 172.21.15.50:0/1354176417 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe002cb20 con 0x7ffbe813c120 2023-10-30T12:21:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:21:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 10 monclient: dump: 2023-10-30T12:21:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.613 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbee3cc640 5 monclient: authenticate success, global_id 254198 2023-10-30T12:21:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.594+0000 7ffbe6ffd640 1 -- 172.21.15.50:0/1354176417 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe002cc80 con 0x7ffbe813c120 2023-10-30T12:21:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbe6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 4 set_mon_vals no callback set 2023-10-30T12:21:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbe6ffd640 1 -- 172.21.15.50:0/1354176417 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0034440 con 0x7ffbe813c120 2023-10-30T12:21:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbe6ffd640 10 monclient: dump: 2023-10-30T12:21:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.617 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 10 monclient: shutdown 2023-10-30T12:21:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 1 -- 172.21.15.50:0/1354176417 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe813c120 msgr2=0x7ffbe813e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 1 --2- 172.21.15.50:0/1354176417 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe813c120 0x7ffbe813e510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009dc0 tx=0x7ffbe002bd70 comp rx=0 tx=0).stop 2023-10-30T12:21:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 1 -- 172.21.15.50:0/1354176417 shutdown_connections 2023-10-30T12:21:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 1 --2- 172.21.15.50:0/1354176417 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe813c120 0x7ffbe813e510 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 1 -- 172.21.15.50:0/1354176417 >> 172.21.15.50:0/1354176417 conn(0x7ffbe80a80b0 msgr2=0x7ffbe80a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 1 -- 172.21.15.50:0/1354176417 shutdown_connections 2023-10-30T12:21:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.595+0000 7ffbee3cc640 1 -- 172.21.15.50:0/1354176417 wait complete. 2023-10-30T12:21:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 10 monclient: monmap: 2023-10-30T12:21:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.620 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:05.620 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:05.620 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:05.620 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.620 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:05.621 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 1 librados: starting msgr at 2023-10-30T12:21:05.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 1 librados: starting objecter 2023-10-30T12:21:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 1 Processor -- start 2023-10-30T12:21:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 1 -- start start 2023-10-30T12:21:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 1 librados: setting wanted keys 2023-10-30T12:21:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 1 librados: calling monclient init 2023-10-30T12:21:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.596+0000 7ffbee3cc640 10 monclient: init 2023-10-30T12:21:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:21:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe80a2450 0x7ffbe8131990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe80a2450 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbe80a2ae0 0x7ffbe8131ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe80a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe813c120 0x7ffbe809c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe813c120 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe809fd50 con 0x7ffbe80a2ae0 2023-10-30T12:21:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe809ebe0 con 0x7ffbe813c120 2023-10-30T12:21:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80a1790 con 0x7ffbe80a2450 2023-10-30T12:21:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbee3cc640 10 monclient(hunting): authenticate will time out at 2110.780357s 2023-10-30T12:21:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe80a2450 0x7ffbe8131990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe80a2450 0x7ffbe8131990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39182/0 (socket says 172.21.15.50:39182) 2023-10-30T12:21:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 -- 172.21.15.50:0/3359828648 learned_addr learned my addr 172.21.15.50:0/3359828648 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbe80a2ae0 0x7ffbe8131ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): get_auth_request con 0x7ffbe80a2450 auth_method 0 2023-10-30T12:21:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 10 monclient(hunting): get_auth_request con 0x7ffbe80a2ae0 auth_method 0 2023-10-30T12:21:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbec969640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe813c120 0x7ffbe809c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient(hunting): handle_auth_done global_id 254201 payload 824 2023-10-30T12:21:05.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe813c120 msgr2=0x7ffbe809c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe813c120 0x7ffbe809c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbe80a2ae0 msgr2=0x7ffbe8131ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbe80a2ae0 0x7ffbe8131ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 monclient: found mon.noname-c 2023-10-30T12:21:05.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 -- 172.21.15.50:0/3359828648 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe0009a70 con 0x7ffbe80a2450 2023-10-30T12:21:05.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbec969640 10 monclient: get_auth_request con 0x7ffbe813c120 auth_method 0 2023-10-30T12:21:05.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbec969640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe813c120 0x7ffbe809c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:21:05.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe77fe640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbe80a2ae0 0x7ffbe8131ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:05.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe7fff640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe80a2450 0x7ffbe8131990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800c780 tx=0x7ffbd800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe4ff9640 10 client.?.objecter ms_handle_connect 0x7ffbe80a2450 2023-10-30T12:21:05.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.597+0000 7ffbe4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:05.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80072d0 con 0x7ffbe80a2450 2023-10-30T12:21:05.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:05.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: dump: 2023-10-30T12:21:05.633 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.633 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.633 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.633 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.633 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.634 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.634 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.634 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.634 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.634 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd8007450 con 0x7ffbe80a2450 2023-10-30T12:21:05.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 5 monclient: authenticate success, global_id 254201 2023-10-30T12:21:05.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 10 monclient: _renew_subs 2023-10-30T12:21:05.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe813b820 con 0x7ffbe80a2450 2023-10-30T12:21:05.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80047e0 con 0x7ffbe80a2450 2023-10-30T12:21:05.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:05.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 10 monclient: dump: 2023-10-30T12:21:05.637 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.637 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.637 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.638 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.638 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.638 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.638 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.638 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.639 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.639 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 10 client.254201.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:05.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe67fc640 4 set_mon_vals no callback set 2023-10-30T12:21:05.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 10 monclient: _renew_subs 2023-10-30T12:21:05.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe809e220 con 0x7ffbe80a2450 2023-10-30T12:21:05.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 1 librados: init done 2023-10-30T12:21:05.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ffbd8004960 con 0x7ffbe80a2450 2023-10-30T12:21:05.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d310 0x7ffbe802b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.598+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe8073ff0 con 0x7ffbe809d310 2023-10-30T12:21:05.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd40392f0 0x7ffbd403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(652..652 src has 520..652) v4 ==== 6678+0+0 (secure 0 0 0) 0x7ffbd804a1a0 con 0x7ffbe80a2450 2023-10-30T12:21:05.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 10 client.254201.objecter ms_dispatch 0x7ffbe8000b90 osd_map(652..652 src has 520..652) v4 2023-10-30T12:21:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 3 client.254201.objecter handle_osd_map got epochs [652,652] > 0 2023-10-30T12:21:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 3 client.254201.objecter handle_osd_map decoding full epoch 652 2023-10-30T12:21:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 20 client.254201.objecter dump_active .. 0 homeless 2023-10-30T12:21:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d310 0x7ffbe802b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient: get_auth_request con 0x7ffbe809d310 auth_method 0 2023-10-30T12:21:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbec969640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd40392f0 0x7ffbd403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbec969640 10 monclient: get_auth_request con 0x7ffbd40392f0 auth_method 0 2023-10-30T12:21:05.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_done global_id 204736 payload 824 2023-10-30T12:21:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 1 -- 172.21.15.50:0/3359828648 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffbe809a5c0 con 0x7ffbe809d310 2023-10-30T12:21:05.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 10 client.254201.objecter ms_handle_connect 0x7ffbd40392f0 2023-10-30T12:21:05.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbec969640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd40392f0 0x7ffbd403b7b0 secure :-1 s=READY pgs=4325 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0036000 tx=0x7ffbe0030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 10 client.254201.objecter ms_handle_connect 0x7ffbe809d310 2023-10-30T12:21:05.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe4ff9640 10 client.254201.objecter resend_mon_ops 2023-10-30T12:21:05.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.599+0000 7ffbe77fe640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d310 0x7ffbe802b430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ffbe809deb0 tx=0x7ffbdc00ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00da70 con 0x7ffbe809d310 2023-10-30T12:21:05.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:21:05.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 10 monclient: dump: 2023-10-30T12:21:05.646 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.646 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.646 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.647 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.648 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7ffbdc00dbd0 con 0x7ffbe809d310 2023-10-30T12:21:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d310 msgr2=0x7ffbe802b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d310 0x7ffbe802b430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ffbe809deb0 tx=0x7ffbdc00ff70 comp rx=0 tx=0).stop 2023-10-30T12:21:05.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d310 msgr2=0x7ffbe802b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:05.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.600+0000 7ffbe4ff9640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d310 0x7ffbe802b430 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ffbe809deb0 tx=0x7ffbdc00ff70 comp rx=0 tx=0).stop 2023-10-30T12:21:05.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.606+0000 7ffbee3cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:05.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.606+0000 7ffbee3cc640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d6f0 0x7ffbe802b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.606+0000 7ffbee3cc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.606+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe806b060 con 0x7ffbe809d6f0 2023-10-30T12:21:05.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d6f0 0x7ffbe802b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient: get_auth_request con 0x7ffbe809d6f0 auth_method 0 2023-10-30T12:21:05.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 10 monclient(hunting): handle_auth_done global_id 204739 payload 824 2023-10-30T12:21:05.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 1 -- 172.21.15.50:0/3359828648 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffbe80a2880 con 0x7ffbe809d6f0 2023-10-30T12:21:05.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe77fe640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d6f0 0x7ffbe802b970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc018000 tx=0x7ffbdc003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe4ff9640 10 client.254201.objecter ms_handle_connect 0x7ffbe809d6f0 2023-10-30T12:21:05.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe4ff9640 10 client.254201.objecter resend_mon_ops 2023-10-30T12:21:05.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc024070 con 0x7ffbe809d6f0 2023-10-30T12:21:05.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:21:05.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.607+0000 7ffbe4ff9640 10 monclient: dump: 2023-10-30T12:21:05.655 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.655 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.655 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.655 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.655 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.656 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7ffbdc004e40 con 0x7ffbe809d6f0 2023-10-30T12:21:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbe4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbe4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d6f0 msgr2=0x7ffbe802b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbe4ff9640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d6f0 0x7ffbe802b970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc018000 tx=0x7ffbdc003760 comp rx=0 tx=0).stop 2023-10-30T12:21:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbe4ff9640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d6f0 msgr2=0x7ffbe802b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:05.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbe4ff9640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffbe809d6f0 0x7ffbe802b970 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc018000 tx=0x7ffbdc003760 comp rx=0 tx=0).stop 2023-10-30T12:21:05.658 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:05.658 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:21:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 210, 2023-10-30T12:21:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:05.659 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:21:05.660 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:05.660 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.660 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:21:05.660 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:05.660 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:05.661 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:05.661 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.661 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.661 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.661 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.662 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.662 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.662 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.662 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.662 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.663 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.663 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:05.663 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:05.663 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.663 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.664 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.664 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.664 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.664 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.664 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.665 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.665 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.665 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.665 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:05.666 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:05.666 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:05.666 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:05.666 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:05.666 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:05.667 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:05.667 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:05.667 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:05.667 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:05.667 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:05.667 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:05.668 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:05.668 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:05.668 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:05.668 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:05.668 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:05.669 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:05.669 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:05.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:05.669 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:05.669 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:05.669 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:05.670 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:05.670 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:05.670 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:05.670 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:05.670 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.671 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:05.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:05.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.671 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:05.671 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:05.672 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.672 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.672 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.672 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:05.672 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.673 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.673 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:05.673 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.674 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.674 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:05.674 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:05.674 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.675 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.675 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.675 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:05.675 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:05.676 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.676 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.676 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.676 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:05.676 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.677 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:05.677 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.678 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:05.678 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:05.678 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.679 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.679 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.679 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:05.680 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:05.680 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:05.680 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:05.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.680 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:05.680 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:05.681 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:05.681 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:05.681 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:05.682 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.682 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.682 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:05.682 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:05.683 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:05.683 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:05.683 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:05.683 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.683 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:21:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.686 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.686 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.686 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:21:05.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.686 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:05.689 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:05.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:05.690 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:05.690 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:05.690 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:21:05.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:05.691 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:05.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:05.691 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:05.691 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:05.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 10 librados: watch_flush enter 2023-10-30T12:21:05.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 10 librados: watch_flush exit 2023-10-30T12:21:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 20 client.254201.objecter shutdown clearing up homeless session... 2023-10-30T12:21:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 10 client.254201.objecter successfully canceled tick 2023-10-30T12:21:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd40392f0 msgr2=0x7ffbd403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ffbd40392f0 0x7ffbd403b7b0 secure :-1 s=READY pgs=4325 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0036000 tx=0x7ffbe0030000 comp rx=0 tx=0).stop 2023-10-30T12:21:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 10 monclient: shutdown 2023-10-30T12:21:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe80a2450 msgr2=0x7ffbe8131990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 --2- 172.21.15.50:0/3359828648 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffbe80a2450 0x7ffbe8131990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800c780 tx=0x7ffbd800cc40 comp rx=0 tx=0).stop 2023-10-30T12:21:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbec969640 1 -- 172.21.15.50:0/3359828648 reap_dead start 2023-10-30T12:21:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbec969640 1 -- 172.21.15.50:0/3359828648 reap_dead start 2023-10-30T12:21:05.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 shutdown_connections 2023-10-30T12:21:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 >> 172.21.15.50:0/3359828648 conn(0x7ffbe80a80b0 msgr2=0x7ffbe8135ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 shutdown_connections 2023-10-30T12:21:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 -- 172.21.15.50:0/3359828648 wait complete. 2023-10-30T12:21:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.608+0000 7ffbee3cc640 1 librados: shutdown 2023-10-30T12:21:05.697 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:21:05.697 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:21:05.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:21:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.801+0000 7fa9bc297640 10 monclient: get_monmap_and_config 2023-10-30T12:21:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.801+0000 7fa9bc297640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.801+0000 7fa9bc297640 10 monclient: monmap: 2023-10-30T12:21:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:05.802424+0000 2023-10-30T12:21:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:05.802424+0000 2023-10-30T12:21:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:05.805 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.801+0000 7fa9bc297640 1 Processor -- start 2023-10-30T12:21:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.801+0000 7fa9bc297640 1 -- start start 2023-10-30T12:21:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.801+0000 7fa9bc297640 10 monclient: init 2023-10-30T12:21:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-10-30T12:21:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b409bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b4098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9b4098900 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9b41379f0 0x7fa9b4139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient(hunting): picked mon.noname-a con 0x7fa9b41379f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:05.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b409ad30 con 0x7fa9b4098900 2023-10-30T12:21:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b40993f0 con 0x7fa9b41379f0 2023-10-30T12:21:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b409c460 con 0x7fa9b4098520 2023-10-30T12:21:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9bc297640 10 monclient(hunting): authenticate will time out at 2110.985596s 2023-10-30T12:21:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b409bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9b41379f0 0x7fa9b4139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b409bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39190/0 (socket says 172.21.15.50:39190) 2023-10-30T12:21:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 1 -- 172.21.15.50:0/2469075290 learned_addr learned my addr 172.21.15.50:0/2469075290 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:05.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 10 monclient(hunting): get_auth_request con 0x7fa9b4098520 auth_method 0 2023-10-30T12:21:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9b9832640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa9b4098900 0x7fa9b4098ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48596/0 (socket says 172.21.15.50:48596) 2023-10-30T12:21:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 10 monclient(hunting): get_auth_request con 0x7fa9b41379f0 auth_method 0 2023-10-30T12:21:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba033640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.802+0000 7fa9ba834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 10 monclient(hunting): handle_auth_done global_id 264135 payload 274 2023-10-30T12:21:05.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 1 -- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 msgr2=0x7fa9b409bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 1 --2- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b409bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 1 -- 172.21.15.50:0/2469075290 >> v1:172.21.15.50:6789/0 conn(0x7fa9b4098900 legacy=0x7fa9b4098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 1 monclient: found mon.noname-a 2023-10-30T12:21:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9b9031640 1 -- 172.21.15.50:0/2469075290 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157875023 0 0) 0x7fa9b409ad30 con 0x7fa9b4098900 2023-10-30T12:21:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 1 -- 172.21.15.50:0/2469075290 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b409b260 con 0x7fa9b41379f0 2023-10-30T12:21:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9b9031640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:21:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba033640 1 --2- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b409bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9ba834640 1 --2- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4139de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009d90 tx=0x7fa9ac02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470bc26e8de60eea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9b9031640 1 -- 172.21.15.50:0/2469075290 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac02cb30 con 0x7fa9b41379f0 2023-10-30T12:21:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9b9031640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9b9031640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.803+0000 7fa9b9031640 10 monclient: dump: 2023-10-30T12:21:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.823 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.824 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 1 -- 172.21.15.50:0/2469075290 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac02cc90 con 0x7fa9b41379f0 2023-10-30T12:21:05.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 5 monclient: authenticate success, global_id 264135 2023-10-30T12:21:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 1 -- 172.21.15.50:0/2469075290 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac034440 con 0x7fa9b41379f0 2023-10-30T12:21:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 4 set_mon_vals no callback set 2023-10-30T12:21:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9b9031640 10 monclient: dump: 2023-10-30T12:21:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.827 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.828 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 10 monclient: shutdown 2023-10-30T12:21:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 -- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 msgr2=0x7fa9b4139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 --2- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4139de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009d90 tx=0x7fa9ac02bd80 comp rx=0 tx=0).stop 2023-10-30T12:21:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 -- 172.21.15.50:0/2469075290 shutdown_connections 2023-10-30T12:21:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 --2- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4139de0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 --2- 172.21.15.50:0/2469075290 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b409bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 -- 172.21.15.50:0/2469075290 >> 172.21.15.50:0/2469075290 conn(0x7fa9b40a05c0 msgr2=0x7fa9b409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:05.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 -- 172.21.15.50:0/2469075290 shutdown_connections 2023-10-30T12:21:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.804+0000 7fa9bc297640 1 -- 172.21.15.50:0/2469075290 wait complete. 2023-10-30T12:21:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient: build_initial_monmap 2023-10-30T12:21:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient: monmap: 2023-10-30T12:21:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:05.832 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 librados: starting msgr at 2023-10-30T12:21:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 librados: starting objecter 2023-10-30T12:21:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 Processor -- start 2023-10-30T12:21:05.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 -- start start 2023-10-30T12:21:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 librados: setting wanted keys 2023-10-30T12:21:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 librados: calling monclient init 2023-10-30T12:21:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient: init 2023-10-30T12:21:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:21:05.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b41ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): picked mon.noname-c con 0x7fa9b4098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9b4098900 0x7fa9b4096440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): picked mon.noname-b con 0x7fa9b4098900 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4096980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): picked mon.noname-a con 0x7fa9b41379f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b41378b0 con 0x7fa9b41379f0 2023-10-30T12:21:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4094950 con 0x7fa9b4098900 2023-10-30T12:21:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b41cace0 con 0x7fa9b4098520 2023-10-30T12:21:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.805+0000 7fa9bc297640 10 monclient(hunting): authenticate will time out at 2110.988981s 2023-10-30T12:21:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b41ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4096980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b41ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39192/0 (socket says 172.21.15.50:39192) 2023-10-30T12:21:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 -- 172.21.15.50:0/1626094476 learned_addr learned my addr 172.21.15.50:0/1626094476 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 10 monclient(hunting): get_auth_request con 0x7fa9b41379f0 auth_method 0 2023-10-30T12:21:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): get_auth_request con 0x7fa9b4098520 auth_method 0 2023-10-30T12:21:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9b9832640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9b4098900 0x7fa9b4096440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9b9832640 10 monclient(hunting): get_auth_request con 0x7fa9b4098900 auth_method 0 2023-10-30T12:21:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9b9832640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9b9832640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9b9832640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient(hunting): handle_auth_done global_id 254207 payload 824 2023-10-30T12:21:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 -- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9b4098900 msgr2=0x7fa9b4096440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9b4098900 0x7fa9b4096440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 -- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 msgr2=0x7fa9b4096980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4096980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient: _finish_hunting 0 2023-10-30T12:21:05.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 monclient: found mon.noname-c 2023-10-30T12:21:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 -- 172.21.15.50:0/1626094476 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b0009460 con 0x7fa9b4098520 2023-10-30T12:21:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9b9832640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9b4098900 0x7fa9b4096440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:21:05.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba834640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4096980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9ba033640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b41ca000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000e9b0 tx=0x7fa9b000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 client.?.objecter ms_handle_connect 0x7fa9b4098520 2023-10-30T12:21:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 1 -- 172.21.15.50:0/1626094476 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000ca00 con 0x7fa9b4098520 2023-10-30T12:21:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: dump: 2023-10-30T12:21:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.845 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: _finish_auth 0 2023-10-30T12:21:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: _check_auth_tickets 2023-10-30T12:21:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 5 monclient: authenticate success, global_id 254207 2023-10-30T12:21:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 1 -- 172.21.15.50:0/1626094476 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b000cb60 con 0x7fa9b4098520 2023-10-30T12:21:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 10 monclient: _renew_subs 2023-10-30T12:21:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac009a70 con 0x7fa9b4098520 2023-10-30T12:21:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 1 -- 172.21.15.50:0/1626094476 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0005990 con 0x7fa9b4098520 2023-10-30T12:21:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9aaffd640 10 monclient: dump: 2023-10-30T12:21:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.848 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:05.849 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:05.849 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9b8830640 4 set_mon_vals no callback set 2023-10-30T12:21:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 10 client.254207.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 10 monclient: _renew_subs 2023-10-30T12:21:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b4093060 con 0x7fa9b4098520 2023-10-30T12:21:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.806+0000 7fa9bc297640 1 librados: init done 2023-10-30T12:21:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9aaffd640 1 -- 172.21.15.50:0/1626094476 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa9b0005af0 con 0x7fa9b4098520 2023-10-30T12:21:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9bc297640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9bc297640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9bc297640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b4095030 con 0x7fa9b4098520 2023-10-30T12:21:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9aaffd640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9980392f0 0x7fa99803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9aaffd640 1 -- 172.21.15.50:0/1626094476 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(652..652 src has 520..652) v4 ==== 6678+0+0 (secure 0 0 0) 0x7fa9b004b2d0 con 0x7fa9b4098520 2023-10-30T12:21:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9aaffd640 10 client.254207.objecter ms_dispatch 0x7fa9b4000b90 osd_map(652..652 src has 520..652) v4 2023-10-30T12:21:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9aaffd640 3 client.254207.objecter handle_osd_map got epochs [652,652] > 0 2023-10-30T12:21:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9aaffd640 3 client.254207.objecter handle_osd_map decoding full epoch 652 2023-10-30T12:21:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.807+0000 7fa9aaffd640 20 client.254207.objecter dump_active .. 0 homeless 2023-10-30T12:21:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.808+0000 7fa9b9832640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9980392f0 0x7fa99803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.808+0000 7fa9b9832640 10 monclient: get_auth_request con 0x7fa9980392f0 auth_method 0 2023-10-30T12:21:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.808+0000 7fa9b9832640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9980392f0 0x7fa99803b7b0 secure :-1 s=READY pgs=4326 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4072040 tx=0x7fa9a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.808+0000 7fa9aaffd640 10 client.254207.objecter ms_handle_connect 0x7fa9980392f0 2023-10-30T12:21:05.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.812+0000 7fa9aaffd640 1 -- 172.21.15.50:0/1626094476 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa9b0022710 con 0x7fa9b4098520 2023-10-30T12:21:05.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.812+0000 7fa9aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:05.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:05.812+0000 7fa9aaffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:06.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.054+0000 7fa9bc297640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:21:06.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.054+0000 7fa9bc297640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:21:06.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.054+0000 7fa9bc297640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.054+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa9b4097ed0 con 0x7fa9b4098520 2023-10-30T12:21:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.056+0000 7fa9aaffd640 1 -- 172.21.15.50:0/1626094476 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fa9b0011350 con 0x7fa9b4098520 2023-10-30T12:21:06.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.056+0000 7fa9aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:21:06.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.056+0000 7fa9aaffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:06.059 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.056+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.059 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.056+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=2242284902,config=3140514009,health=670360021,kv=2849054840,logm=1005989193}) 2023-10-30T12:21:06.059 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.056+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=8,logm=52} crc {auth=2505243191,config=3140514009,health=3346127888,kv=535555768,logm=3578097461}) 2023-10-30T12:21:06.060 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.056+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.060 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.056+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=2242284902,config=3140514009,health=670360021,kv=2849054840,logm=1005989193}) 2023-10-30T12:21:06.060 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.056+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=7,logm=53} crc {auth=95612808,config=3140514009,health=670360021,kv=982468352,logm=2468865742}) 2023-10-30T12:21:06.060 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.057+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.060 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.057+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629158266}) 2023-10-30T12:21:06.061 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.057+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1932550293}) 2023-10-30T12:21:06.061 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.057+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.061 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.057+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629158266}) 2023-10-30T12:21:06.061 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.057+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542096988}) 2023-10-30T12:21:06.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 10 librados: watch_flush enter 2023-10-30T12:21:06.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 10 librados: watch_flush exit 2023-10-30T12:21:06.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 20 client.254207.objecter shutdown clearing up homeless session... 2023-10-30T12:21:06.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 10 client.254207.objecter successfully canceled tick 2023-10-30T12:21:06.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9980392f0 msgr2=0x7fa99803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:06.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9980392f0 0x7fa99803b7b0 secure :-1 s=READY pgs=4326 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4072040 tx=0x7fa9a4009040 comp rx=0 tx=0).stop 2023-10-30T12:21:06.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 10 monclient: shutdown 2023-10-30T12:21:06.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:06.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 msgr2=0x7fa9b41ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:06.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b41ca000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000e9b0 tx=0x7fa9b000ee70 comp rx=0 tx=0).stop 2023-10-30T12:21:06.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 shutdown_connections 2023-10-30T12:21:06.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa9980392f0 0x7fa99803b7b0 unknown :-1 s=CLOSED pgs=4326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:06.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9b41379f0 0x7fa9b4096980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:06.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9b4098900 0x7fa9b4096440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:06.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 --2- 172.21.15.50:0/1626094476 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9b4098520 0x7fa9b41ca000 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:06.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 >> 172.21.15.50:0/1626094476 conn(0x7fa9b40a05c0 msgr2=0x7fa9b409dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:06.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 shutdown_connections 2023-10-30T12:21:06.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 -- 172.21.15.50:0/1626094476 wait complete. 2023-10-30T12:21:06.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:06.058+0000 7fa9bc297640 1 librados: shutdown 2023-10-30T12:21:06.070 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.059+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.070 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.059+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3970484254}) 2023-10-30T12:21:06.070 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.059+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748095750}) 2023-10-30T12:21:06.070 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.059+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.070 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.059+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3970484254}) 2023-10-30T12:21:06.071 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.059+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734650502}) 2023-10-30T12:21:06.071 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.060+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.071 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.060+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515795683}) 2023-10-30T12:21:06.071 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.060+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723554945}) 2023-10-30T12:21:06.071 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.060+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.072 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.060+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515795683}) 2023-10-30T12:21:06.072 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.060+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340521868}) 2023-10-30T12:21:06.072 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.061+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.072 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.061+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552979112}) 2023-10-30T12:21:06.072 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.061+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372246490}) 2023-10-30T12:21:06.073 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.061+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.073 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.061+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552979112}) 2023-10-30T12:21:06.073 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.061+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216876196}) 2023-10-30T12:21:06.073 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.062+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.073 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.062+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136887462}) 2023-10-30T12:21:06.073 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.062+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572843982}) 2023-10-30T12:21:06.074 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.062+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.074 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.062+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136887462}) 2023-10-30T12:21:06.074 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.062+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3297570924}) 2023-10-30T12:21:06.074 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.063+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.074 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.063+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855669767}) 2023-10-30T12:21:06.075 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.063+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806513064}) 2023-10-30T12:21:06.075 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.063+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.075 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.063+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855669767}) 2023-10-30T12:21:06.075 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.063+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1325247753}) 2023-10-30T12:21:06.075 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.075 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775640296}) 2023-10-30T12:21:06.076 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3033180368}) 2023-10-30T12:21:06.076 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.076 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775640296}) 2023-10-30T12:21:06.076 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376711427}) 2023-10-30T12:21:06.076 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.077 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3779515589}) 2023-10-30T12:21:06.077 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3718505667}) 2023-10-30T12:21:06.077 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.077 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.064+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3779515589}) 2023-10-30T12:21:06.077 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.065+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991840512}) 2023-10-30T12:21:06.077 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.065+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.078 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.065+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148675821}) 2023-10-30T12:21:06.078 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.065+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912285282}) 2023-10-30T12:21:06.078 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.065+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.078 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.065+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148675821}) 2023-10-30T12:21:06.078 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.065+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4190111621}) 2023-10-30T12:21:06.079 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.066+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.079 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.066+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102412900}) 2023-10-30T12:21:06.079 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.066+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=366903022}) 2023-10-30T12:21:06.079 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.066+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.079 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.066+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102412900}) 2023-10-30T12:21:06.079 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.066+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2612894220}) 2023-10-30T12:21:06.080 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.080 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840620487}) 2023-10-30T12:21:06.080 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1802874219}) 2023-10-30T12:21:06.080 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.080 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840620487}) 2023-10-30T12:21:06.081 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040293787}) 2023-10-30T12:21:06.081 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.081 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90912736}) 2023-10-30T12:21:06.081 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3263756807}) 2023-10-30T12:21:06.081 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.082 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90912736}) 2023-10-30T12:21:06.082 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.067+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518667374}) 2023-10-30T12:21:06.082 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.068+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.082 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.068+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=508523529}) 2023-10-30T12:21:06.082 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.068+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1180442215}) 2023-10-30T12:21:06.083 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.068+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.083 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.068+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=508523529}) 2023-10-30T12:21:06.083 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.068+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2469747396}) 2023-10-30T12:21:06.084 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.069+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.085 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.069+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107138197}) 2023-10-30T12:21:06.085 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.069+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096004533}) 2023-10-30T12:21:06.085 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.069+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.085 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.069+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107138197}) 2023-10-30T12:21:06.085 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.069+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681575148}) 2023-10-30T12:21:06.086 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.086 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242954722}) 2023-10-30T12:21:06.086 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98665295}) 2023-10-30T12:21:06.086 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.086 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242954722}) 2023-10-30T12:21:06.087 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3892639729}) 2023-10-30T12:21:06.087 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.087 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822967266}) 2023-10-30T12:21:06.087 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1871212078}) 2023-10-30T12:21:06.087 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.088 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822967266}) 2023-10-30T12:21:06.088 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.070+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2847543499}) 2023-10-30T12:21:06.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:21:06.106 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.071+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.107 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.071+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529343351}) 2023-10-30T12:21:06.107 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.071+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615104321}) 2023-10-30T12:21:06.107 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.071+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.108 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.071+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529343351}) 2023-10-30T12:21:06.108 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.071+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965269205}) 2023-10-30T12:21:06.108 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.072+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.108 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.072+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284473912}) 2023-10-30T12:21:06.108 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.072+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=204867848}) 2023-10-30T12:21:06.109 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.072+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.109 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.072+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284473912}) 2023-10-30T12:21:06.109 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.072+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937653445}) 2023-10-30T12:21:06.109 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.072+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.109 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303430426}) 2023-10-30T12:21:06.110 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4075831142}) 2023-10-30T12:21:06.110 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.110 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303430426}) 2023-10-30T12:21:06.110 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015602194}) 2023-10-30T12:21:06.110 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.111 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372855949}) 2023-10-30T12:21:06.111 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1709701171}) 2023-10-30T12:21:06.111 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.111 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372855949}) 2023-10-30T12:21:06.111 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.073+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421167853}) 2023-10-30T12:21:06.111 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.074+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.112 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.074+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559786349}) 2023-10-30T12:21:06.112 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.074+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494612837}) 2023-10-30T12:21:06.112 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.074+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.112 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.074+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559786349}) 2023-10-30T12:21:06.112 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.074+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1497347231}) 2023-10-30T12:21:06.113 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.075+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.113 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.075+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092640031}) 2023-10-30T12:21:06.113 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.075+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489626283}) 2023-10-30T12:21:06.113 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.075+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.113 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.075+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092640031}) 2023-10-30T12:21:06.113 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.075+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1030927104}) 2023-10-30T12:21:06.114 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.114 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876173470}) 2023-10-30T12:21:06.114 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3961126511}) 2023-10-30T12:21:06.114 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.114 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876173470}) 2023-10-30T12:21:06.115 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347833255}) 2023-10-30T12:21:06.115 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.115 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108961717}) 2023-10-30T12:21:06.115 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2342121400}) 2023-10-30T12:21:06.115 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.116 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108961717}) 2023-10-30T12:21:06.116 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.076+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239211084}) 2023-10-30T12:21:06.116 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.077+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.116 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.077+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257530972}) 2023-10-30T12:21:06.116 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.077+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105121423}) 2023-10-30T12:21:06.116 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.077+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.117 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.077+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2257530972}) 2023-10-30T12:21:06.117 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.077+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1112593373}) 2023-10-30T12:21:06.117 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.117 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017144264}) 2023-10-30T12:21:06.117 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4175882856}) 2023-10-30T12:21:06.118 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.118 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017144264}) 2023-10-30T12:21:06.118 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682795058}) 2023-10-30T12:21:06.118 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.118 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705737654}) 2023-10-30T12:21:06.119 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515329119}) 2023-10-30T12:21:06.119 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.119 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705737654}) 2023-10-30T12:21:06.119 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.078+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303302498}) 2023-10-30T12:21:06.119 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.079+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.119 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.079+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271909589}) 2023-10-30T12:21:06.120 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.079+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247076867}) 2023-10-30T12:21:06.120 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.079+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.120 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.079+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271909589}) 2023-10-30T12:21:06.120 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.079+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2922109009}) 2023-10-30T12:21:06.120 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.120 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366984925}) 2023-10-30T12:21:06.121 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3814704733}) 2023-10-30T12:21:06.121 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.121 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366984925}) 2023-10-30T12:21:06.121 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715123822}) 2023-10-30T12:21:06.121 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.121 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3566378582}) 2023-10-30T12:21:06.122 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=855290376}) 2023-10-30T12:21:06.122 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.122 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3566378582}) 2023-10-30T12:21:06.122 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.080+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31593989}) 2023-10-30T12:21:06.122 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.081+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.123 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.081+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261867101}) 2023-10-30T12:21:06.123 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.081+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1978310156}) 2023-10-30T12:21:06.123 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.081+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.123 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.081+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261867101}) 2023-10-30T12:21:06.123 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.081+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798873596}) 2023-10-30T12:21:06.123 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.082+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.124 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.082+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214597207}) 2023-10-30T12:21:06.124 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.082+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509467310}) 2023-10-30T12:21:06.124 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.082+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.124 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.082+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214597207}) 2023-10-30T12:21:06.124 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.082+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3578377509}) 2023-10-30T12:21:06.124 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.125 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557191176}) 2023-10-30T12:21:06.125 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3351452319}) 2023-10-30T12:21:06.125 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.125 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557191176}) 2023-10-30T12:21:06.125 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3200272444}) 2023-10-30T12:21:06.125 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.126 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262089260}) 2023-10-30T12:21:06.126 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712051527}) 2023-10-30T12:21:06.126 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.126 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262089260}) 2023-10-30T12:21:06.126 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.083+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647950535}) 2023-10-30T12:21:06.127 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.084+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.127 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.084+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10977517}) 2023-10-30T12:21:06.127 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.084+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4142154124}) 2023-10-30T12:21:06.127 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.084+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.127 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.084+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10977517}) 2023-10-30T12:21:06.127 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.084+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995161632}) 2023-10-30T12:21:06.128 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.128 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772399797}) 2023-10-30T12:21:06.128 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607802740}) 2023-10-30T12:21:06.128 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.128 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772399797}) 2023-10-30T12:21:06.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=675164181}) 2023-10-30T12:21:06.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2514779630}) 2023-10-30T12:21:06.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3136521006}) 2023-10-30T12:21:06.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.129 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2514779630}) 2023-10-30T12:21:06.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.085+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3739203561}) 2023-10-30T12:21:06.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.086+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.086+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178291573}) 2023-10-30T12:21:06.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.086+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3624660910}) 2023-10-30T12:21:06.130 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.086+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.086+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178291573}) 2023-10-30T12:21:06.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.086+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=65027894}) 2023-10-30T12:21:06.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826772008}) 2023-10-30T12:21:06.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163731083}) 2023-10-30T12:21:06.131 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.132 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826772008}) 2023-10-30T12:21:06.132 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1842591044}) 2023-10-30T12:21:06.132 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.132 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867823762}) 2023-10-30T12:21:06.132 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2321905402}) 2023-10-30T12:21:06.132 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.133 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867823762}) 2023-10-30T12:21:06.133 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.087+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804506850}) 2023-10-30T12:21:06.133 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.088+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.133 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.088+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349416845}) 2023-10-30T12:21:06.133 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.088+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396689200}) 2023-10-30T12:21:06.133 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.088+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.134 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.088+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349416845}) 2023-10-30T12:21:06.134 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.088+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789053545}) 2023-10-30T12:21:06.134 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.134 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360239661}) 2023-10-30T12:21:06.134 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389825221}) 2023-10-30T12:21:06.135 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.135 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360239661}) 2023-10-30T12:21:06.135 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983554246}) 2023-10-30T12:21:06.135 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.135 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99530394}) 2023-10-30T12:21:06.135 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2511742464}) 2023-10-30T12:21:06.136 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.136 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99530394}) 2023-10-30T12:21:06.136 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.089+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728103638}) 2023-10-30T12:21:06.136 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.136 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3757191871}) 2023-10-30T12:21:06.136 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293391668}) 2023-10-30T12:21:06.137 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.137 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3757191871}) 2023-10-30T12:21:06.137 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772528136}) 2023-10-30T12:21:06.137 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.137 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760519262}) 2023-10-30T12:21:06.137 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.090+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863369544}) 2023-10-30T12:21:06.138 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.138 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760519262}) 2023-10-30T12:21:06.138 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648821943}) 2023-10-30T12:21:06.138 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.138 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330201100}) 2023-10-30T12:21:06.139 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556893166}) 2023-10-30T12:21:06.139 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.139 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330201100}) 2023-10-30T12:21:06.139 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.091+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057393056}) 2023-10-30T12:21:06.139 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.092+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.139 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.092+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1880640647}) 2023-10-30T12:21:06.140 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.092+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639853793}) 2023-10-30T12:21:06.140 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.092+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.140 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.092+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1880640647}) 2023-10-30T12:21:06.140 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.092+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1032475501}) 2023-10-30T12:21:06.140 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.141 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=978298948}) 2023-10-30T12:21:06.141 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232681957}) 2023-10-30T12:21:06.141 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.141 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=978298948}) 2023-10-30T12:21:06.141 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320097769}) 2023-10-30T12:21:06.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460342031}) 2023-10-30T12:21:06.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1554332399}) 2023-10-30T12:21:06.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460342031}) 2023-10-30T12:21:06.142 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.093+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105462920}) 2023-10-30T12:21:06.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.094+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.094+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939889603}) 2023-10-30T12:21:06.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.094+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962532580}) 2023-10-30T12:21:06.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.094+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.094+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939889603}) 2023-10-30T12:21:06.143 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.094+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=199841540}) 2023-10-30T12:21:06.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673570768}) 2023-10-30T12:21:06.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937546437}) 2023-10-30T12:21:06.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.144 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673570768}) 2023-10-30T12:21:06.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2500290504}) 2023-10-30T12:21:06.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909994125}) 2023-10-30T12:21:06.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1923702765}) 2023-10-30T12:21:06.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.145 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909994125}) 2023-10-30T12:21:06.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.095+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1238105219}) 2023-10-30T12:21:06.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.096+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.096+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746973897}) 2023-10-30T12:21:06.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.096+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1322966472}) 2023-10-30T12:21:06.146 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.096+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.096+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746973897}) 2023-10-30T12:21:06.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.096+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587656004}) 2023-10-30T12:21:06.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821001347}) 2023-10-30T12:21:06.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262960234}) 2023-10-30T12:21:06.147 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821001347}) 2023-10-30T12:21:06.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=527944246}) 2023-10-30T12:21:06.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255386927}) 2023-10-30T12:21:06.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1087681180}) 2023-10-30T12:21:06.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255386927}) 2023-10-30T12:21:06.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.097+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2108362483}) 2023-10-30T12:21:06.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.098+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.098+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686074347}) 2023-10-30T12:21:06.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.098+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1178124977}) 2023-10-30T12:21:06.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.098+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.098+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686074347}) 2023-10-30T12:21:06.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.098+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=856816358}) 2023-10-30T12:21:06.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028817408}) 2023-10-30T12:21:06.150 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2881978333}) 2023-10-30T12:21:06.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028817408}) 2023-10-30T12:21:06.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1441958158}) 2023-10-30T12:21:06.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993022414}) 2023-10-30T12:21:06.151 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2936433902}) 2023-10-30T12:21:06.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993022414}) 2023-10-30T12:21:06.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.099+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465589816}) 2023-10-30T12:21:06.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.100+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.100+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83133426}) 2023-10-30T12:21:06.152 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.100+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285579177}) 2023-10-30T12:21:06.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.100+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.100+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83133426}) 2023-10-30T12:21:06.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.100+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380875861}) 2023-10-30T12:21:06.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805992773}) 2023-10-30T12:21:06.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4287409573}) 2023-10-30T12:21:06.153 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805992773}) 2023-10-30T12:21:06.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705592748}) 2023-10-30T12:21:06.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191629678}) 2023-10-30T12:21:06.154 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497204222}) 2023-10-30T12:21:06.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191629678}) 2023-10-30T12:21:06.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.101+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4091617629}) 2023-10-30T12:21:06.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.102+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.155 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.102+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155122484}) 2023-10-30T12:21:06.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.102+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=172762809}) 2023-10-30T12:21:06.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.102+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.102+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155122484}) 2023-10-30T12:21:06.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.102+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82991874}) 2023-10-30T12:21:06.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.156 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213935592}) 2023-10-30T12:21:06.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2253267375}) 2023-10-30T12:21:06.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213935592}) 2023-10-30T12:21:06.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1295194865}) 2023-10-30T12:21:06.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.157 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660881919}) 2023-10-30T12:21:06.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727174815}) 2023-10-30T12:21:06.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660881919}) 2023-10-30T12:21:06.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.103+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343383906}) 2023-10-30T12:21:06.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.104+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.158 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.104+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4073918422}) 2023-10-30T12:21:06.159 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.104+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3652414718}) 2023-10-30T12:21:06.159 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.104+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.159 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.104+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4073918422}) 2023-10-30T12:21:06.159 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.104+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1002893057}) 2023-10-30T12:21:06.159 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.105+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.159 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.105+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225553200}) 2023-10-30T12:21:06.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.105+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3146485164}) 2023-10-30T12:21:06.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.105+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.105+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225553200}) 2023-10-30T12:21:06.160 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.105+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850107748}) 2023-10-30T12:21:06.185 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.185 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498884868}) 2023-10-30T12:21:06.185 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572806081}) 2023-10-30T12:21:06.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498884868}) 2023-10-30T12:21:06.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624872233}) 2023-10-30T12:21:06.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.186 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057532874}) 2023-10-30T12:21:06.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778170740}) 2023-10-30T12:21:06.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057532874}) 2023-10-30T12:21:06.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.106+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110868838}) 2023-10-30T12:21:06.187 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.107+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.107+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567337564}) 2023-10-30T12:21:06.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.107+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1652072170}) 2023-10-30T12:21:06.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.107+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.107+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567337564}) 2023-10-30T12:21:06.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.107+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795539637}) 2023-10-30T12:21:06.188 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447744744}) 2023-10-30T12:21:06.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640308612}) 2023-10-30T12:21:06.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447744744}) 2023-10-30T12:21:06.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233515446}) 2023-10-30T12:21:06.189 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824228889}) 2023-10-30T12:21:06.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=859730855}) 2023-10-30T12:21:06.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824228889}) 2023-10-30T12:21:06.190 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.108+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417422205}) 2023-10-30T12:21:06.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.109+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.109+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=80969910}) 2023-10-30T12:21:06.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.109+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224998249}) 2023-10-30T12:21:06.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.109+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.191 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.109+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=80969910}) 2023-10-30T12:21:06.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.109+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3596015636}) 2023-10-30T12:21:06.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=69565113}) 2023-10-30T12:21:06.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451116693}) 2023-10-30T12:21:06.192 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=69565113}) 2023-10-30T12:21:06.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660694110}) 2023-10-30T12:21:06.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694433720}) 2023-10-30T12:21:06.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232199868}) 2023-10-30T12:21:06.193 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694433720}) 2023-10-30T12:21:06.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.110+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=717230011}) 2023-10-30T12:21:06.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.111+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.111+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853706395}) 2023-10-30T12:21:06.194 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.111+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173519133}) 2023-10-30T12:21:06.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.111+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.111+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853706395}) 2023-10-30T12:21:06.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.111+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785910685}) 2023-10-30T12:21:06.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.112+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.112+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863007989}) 2023-10-30T12:21:06.195 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.112+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=173025404}) 2023-10-30T12:21:06.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.112+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.112+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863007989}) 2023-10-30T12:21:06.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.112+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=200765742}) 2023-10-30T12:21:06.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.112+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.196 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134523003}) 2023-10-30T12:21:06.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821970643}) 2023-10-30T12:21:06.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134523003}) 2023-10-30T12:21:06.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992945884}) 2023-10-30T12:21:06.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.197 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539927976}) 2023-10-30T12:21:06.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129685189}) 2023-10-30T12:21:06.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539927976}) 2023-10-30T12:21:06.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.113+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3825332603}) 2023-10-30T12:21:06.198 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.114+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.114+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989649405}) 2023-10-30T12:21:06.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.114+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292269045}) 2023-10-30T12:21:06.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.114+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.114+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989649405}) 2023-10-30T12:21:06.199 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.114+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174317566}) 2023-10-30T12:21:06.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447616409}) 2023-10-30T12:21:06.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018100332}) 2023-10-30T12:21:06.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447616409}) 2023-10-30T12:21:06.200 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93762907}) 2023-10-30T12:21:06.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354714809}) 2023-10-30T12:21:06.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294260157}) 2023-10-30T12:21:06.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.201 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354714809}) 2023-10-30T12:21:06.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.115+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2535919381}) 2023-10-30T12:21:06.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.116+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.116+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556210700}) 2023-10-30T12:21:06.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.116+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079019676}) 2023-10-30T12:21:06.202 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.116+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.116+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556210700}) 2023-10-30T12:21:06.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.116+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050082552}) 2023-10-30T12:21:06.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994665388}) 2023-10-30T12:21:06.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2593117066}) 2023-10-30T12:21:06.203 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994665388}) 2023-10-30T12:21:06.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2447814825}) 2023-10-30T12:21:06.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498126199}) 2023-10-30T12:21:06.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3967143060}) 2023-10-30T12:21:06.204 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498126199}) 2023-10-30T12:21:06.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.117+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=838320590}) 2023-10-30T12:21:06.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.118+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.118+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120919932}) 2023-10-30T12:21:06.205 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.118+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949144973}) 2023-10-30T12:21:06.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.118+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.118+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120919932}) 2023-10-30T12:21:06.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.118+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143138166}) 2023-10-30T12:21:06.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501413998}) 2023-10-30T12:21:06.206 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=261390789}) 2023-10-30T12:21:06.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501413998}) 2023-10-30T12:21:06.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440651596}) 2023-10-30T12:21:06.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.207 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731236124}) 2023-10-30T12:21:06.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3840758238}) 2023-10-30T12:21:06.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731236124}) 2023-10-30T12:21:06.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.119+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949016071}) 2023-10-30T12:21:06.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.120+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.208 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.120+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247538317}) 2023-10-30T12:21:06.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.120+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930045944}) 2023-10-30T12:21:06.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.120+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.120+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247538317}) 2023-10-30T12:21:06.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.120+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278267540}) 2023-10-30T12:21:06.209 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472063428}) 2023-10-30T12:21:06.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763687104}) 2023-10-30T12:21:06.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472063428}) 2023-10-30T12:21:06.210 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711890739}) 2023-10-30T12:21:06.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1623613722}) 2023-10-30T12:21:06.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=735536868}) 2023-10-30T12:21:06.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1623613722}) 2023-10-30T12:21:06.211 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.121+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4088885018}) 2023-10-30T12:21:06.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.122+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.122+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207033148}) 2023-10-30T12:21:06.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.122+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119676814}) 2023-10-30T12:21:06.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.122+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.212 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.122+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207033148}) 2023-10-30T12:21:06.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.122+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185534787}) 2023-10-30T12:21:06.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026858289}) 2023-10-30T12:21:06.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3502572775}) 2023-10-30T12:21:06.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.213 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026858289}) 2023-10-30T12:21:06.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102901251}) 2023-10-30T12:21:06.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515917113}) 2023-10-30T12:21:06.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1932327105}) 2023-10-30T12:21:06.214 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515917113}) 2023-10-30T12:21:06.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.123+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3917374556}) 2023-10-30T12:21:06.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.124+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.124+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778046643}) 2023-10-30T12:21:06.215 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.124+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939319237}) 2023-10-30T12:21:06.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.124+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.124+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778046643}) 2023-10-30T12:21:06.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.124+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914191828}) 2023-10-30T12:21:06.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.125+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.125+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077287981}) 2023-10-30T12:21:06.216 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.125+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585583107}) 2023-10-30T12:21:06.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.125+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.125+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077287981}) 2023-10-30T12:21:06.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.125+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=126607379}) 2023-10-30T12:21:06.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.217 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273716517}) 2023-10-30T12:21:06.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=367411815}) 2023-10-30T12:21:06.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273716517}) 2023-10-30T12:21:06.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1448311832}) 2023-10-30T12:21:06.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.218 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2457951061}) 2023-10-30T12:21:06.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44566867}) 2023-10-30T12:21:06.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2457951061}) 2023-10-30T12:21:06.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.126+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2123797125}) 2023-10-30T12:21:06.219 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.127+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.127+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845526911}) 2023-10-30T12:21:06.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.127+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2282927912}) 2023-10-30T12:21:06.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.127+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.127+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845526911}) 2023-10-30T12:21:06.220 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.127+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1234609138}) 2023-10-30T12:21:06.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219116299}) 2023-10-30T12:21:06.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3054705893}) 2023-10-30T12:21:06.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219116299}) 2023-10-30T12:21:06.221 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762485874}) 2023-10-30T12:21:06.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485740350}) 2023-10-30T12:21:06.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813684486}) 2023-10-30T12:21:06.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.222 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485740350}) 2023-10-30T12:21:06.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.128+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116070556}) 2023-10-30T12:21:06.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.129+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.129+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661951506}) 2023-10-30T12:21:06.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.129+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086650933}) 2023-10-30T12:21:06.223 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.129+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.129+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661951506}) 2023-10-30T12:21:06.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.129+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027358567}) 2023-10-30T12:21:06.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809756926}) 2023-10-30T12:21:06.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092579009}) 2023-10-30T12:21:06.224 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809756926}) 2023-10-30T12:21:06.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082391586}) 2023-10-30T12:21:06.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905829764}) 2023-10-30T12:21:06.225 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962835923}) 2023-10-30T12:21:06.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905829764}) 2023-10-30T12:21:06.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.130+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239198271}) 2023-10-30T12:21:06.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.131+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.131+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733603117}) 2023-10-30T12:21:06.226 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.131+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541290513}) 2023-10-30T12:21:06.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.131+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.131+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733603117}) 2023-10-30T12:21:06.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.131+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629373369}) 2023-10-30T12:21:06.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.227 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075218669}) 2023-10-30T12:21:06.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2708602962}) 2023-10-30T12:21:06.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075218669}) 2023-10-30T12:21:06.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360821548}) 2023-10-30T12:21:06.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.228 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409534448}) 2023-10-30T12:21:06.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2928696846}) 2023-10-30T12:21:06.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409534448}) 2023-10-30T12:21:06.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.132+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=880843824}) 2023-10-30T12:21:06.229 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.133+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.230 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.133+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3927361994}) 2023-10-30T12:21:06.230 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.133+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770432094}) 2023-10-30T12:21:06.230 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.133+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.230 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.133+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3927361994}) 2023-10-30T12:21:06.230 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.133+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=28043020}) 2023-10-30T12:21:06.230 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.134+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.134+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244378017}) 2023-10-30T12:21:06.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.134+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=475943241}) 2023-10-30T12:21:06.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.134+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.134+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244378017}) 2023-10-30T12:21:06.231 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.134+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2154647153}) 2023-10-30T12:21:06.232 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.232 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350302265}) 2023-10-30T12:21:06.232 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3116558573}) 2023-10-30T12:21:06.232 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.232 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350302265}) 2023-10-30T12:21:06.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389346886}) 2023-10-30T12:21:06.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068542567}) 2023-10-30T12:21:06.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879541672}) 2023-10-30T12:21:06.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.233 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068542567}) 2023-10-30T12:21:06.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.135+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3405977099}) 2023-10-30T12:21:06.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.136+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.136+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355592394}) 2023-10-30T12:21:06.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.136+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828939909}) 2023-10-30T12:21:06.234 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.136+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.136+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355592394}) 2023-10-30T12:21:06.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.136+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=505072559}) 2023-10-30T12:21:06.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147772180}) 2023-10-30T12:21:06.235 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391798366}) 2023-10-30T12:21:06.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147772180}) 2023-10-30T12:21:06.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2914070242}) 2023-10-30T12:21:06.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228706822}) 2023-10-30T12:21:06.236 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978614258}) 2023-10-30T12:21:06.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228706822}) 2023-10-30T12:21:06.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.137+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213434030}) 2023-10-30T12:21:06.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.138+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.237 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.138+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609672917}) 2023-10-30T12:21:06.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.138+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=89428106}) 2023-10-30T12:21:06.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.138+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.138+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609672917}) 2023-10-30T12:21:06.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.138+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853148341}) 2023-10-30T12:21:06.238 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3069182635}) 2023-10-30T12:21:06.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=484511200}) 2023-10-30T12:21:06.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3069182635}) 2023-10-30T12:21:06.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2938045699}) 2023-10-30T12:21:06.239 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102611150}) 2023-10-30T12:21:06.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101772640}) 2023-10-30T12:21:06.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102611150}) 2023-10-30T12:21:06.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.139+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121689014}) 2023-10-30T12:21:06.240 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.140+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.140+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502361054}) 2023-10-30T12:21:06.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.140+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533822231}) 2023-10-30T12:21:06.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.140+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.140+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502361054}) 2023-10-30T12:21:06.241 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.140+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904669333}) 2023-10-30T12:21:06.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.141+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.141+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061447582}) 2023-10-30T12:21:06.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.141+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083128764}) 2023-10-30T12:21:06.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.141+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.242 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.141+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061447582}) 2023-10-30T12:21:06.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.141+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553734629}) 2023-10-30T12:21:06.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753753912}) 2023-10-30T12:21:06.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766768790}) 2023-10-30T12:21:06.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.243 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753753912}) 2023-10-30T12:21:06.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913547515}) 2023-10-30T12:21:06.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418337803}) 2023-10-30T12:21:06.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983285227}) 2023-10-30T12:21:06.244 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418337803}) 2023-10-30T12:21:06.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.142+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083094666}) 2023-10-30T12:21:06.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.143+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.143+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=968929734}) 2023-10-30T12:21:06.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.143+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291197674}) 2023-10-30T12:21:06.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.143+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.143+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=968929734}) 2023-10-30T12:21:06.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.143+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942537187}) 2023-10-30T12:21:06.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492814862}) 2023-10-30T12:21:06.246 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603890455}) 2023-10-30T12:21:06.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492814862}) 2023-10-30T12:21:06.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072558518}) 2023-10-30T12:21:06.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297729554}) 2023-10-30T12:21:06.247 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2105871560}) 2023-10-30T12:21:06.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297729554}) 2023-10-30T12:21:06.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.144+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=697339753}) 2023-10-30T12:21:06.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.145+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.248 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.145+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903407666}) 2023-10-30T12:21:06.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.145+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=666940919}) 2023-10-30T12:21:06.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.145+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.145+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903407666}) 2023-10-30T12:21:06.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.145+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1537612906}) 2023-10-30T12:21:06.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.146+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.249 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.146+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1563119343,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=46904685}) 2023-10-30T12:21:06.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.146+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=3985088333,mdsmap=909706845,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=629142163}) 2023-10-30T12:21:06.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.146+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.146+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1563119343,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=46904685}) 2023-10-30T12:21:06.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.146+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=87,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=2322198436,mdsmap=2441318933,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=46904685}) 2023-10-30T12:21:06.250 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.147+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.147+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=43,osdmap=32} crc {mgrstat=1202406674,mon_config_key=3217606506,monmap=3324057927,osd_metadata=451695420,osd_pg_creating=1660984007,osd_snap=3204665652,osdmap=1775006639}) 2023-10-30T12:21:06.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.147+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=36} crc {mgrstat=122961121,mon_config_key=3217606506,monmap=4204254258,osd_metadata=2479346626,osd_pg_creating=1660984007,osd_snap=1879609884,osdmap=3738204123}) 2023-10-30T12:21:06.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.147+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.147+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=43,osdmap=32} crc {mgrstat=1202406674,mon_config_key=3217606506,monmap=3324057927,osd_metadata=451695420,osd_pg_creating=1660984007,osd_snap=3204665652,osdmap=1775006639}) 2023-10-30T12:21:06.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.147+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=32} crc {mgrstat=122961121,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1660984007,osd_snap=2695686481,osdmap=3002549307}) 2023-10-30T12:21:06.251 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.147+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.148+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=819165993}) 2023-10-30T12:21:06.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.148+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=598884627}) 2023-10-30T12:21:06.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.148+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.148+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=819165993}) 2023-10-30T12:21:06.252 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.148+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1006163659}) 2023-10-30T12:21:06.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.153+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.153+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=2051085353}) 2023-10-30T12:21:06.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.153+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=71} crc {osdmap=1672026671}) 2023-10-30T12:21:06.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.153+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:21:06.253 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.153+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=2051085353}) 2023-10-30T12:21:06.254 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:21:06.153+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=73} crc {osdmap=1673764618}) 2023-10-30T12:21:07.090 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:21:07.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:21:07.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.206+0000 7f5e1be0e640 10 monclient: get_monmap_and_config 2023-10-30T12:21:07.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.206+0000 7f5e1be0e640 10 monclient: build_initial_monmap 2023-10-30T12:21:07.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.206+0000 7f5e1be0e640 10 monclient: monmap: 2023-10-30T12:21:07.209 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:07.209 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.209 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:07.207490+0000 2023-10-30T12:21:07.209 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:07.207490+0000 2023-10-30T12:21:07.209 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:07.209 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.210 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:07.210 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:07.210 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:07.210 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:07.210 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:07.211 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 1 Processor -- start 2023-10-30T12:21:07.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 1 -- start start 2023-10-30T12:21:07.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient: init 2023-10-30T12:21:07.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:07.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-10-30T12:21:07.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e140a0080 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:07.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5e140720b0 0x7f5e1413ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient(hunting): picked mon.noname-a con 0x7f5e140720b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:07.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e14073a30 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:07.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1409ccc0 con 0x7f5e14073a30 2023-10-30T12:21:07.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e14071e90 con 0x7f5e140a0080 2023-10-30T12:21:07.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e14073620 con 0x7f5e140720b0 2023-10-30T12:21:07.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:07.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1be0e640 10 monclient(hunting): authenticate will time out at 2112.390732s 2023-10-30T12:21:07.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5e140720b0 0x7f5e1413ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1a3ab640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5e14073a30 0x7f5e1413d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48612/0 (socket says 172.21.15.50:48612) 2023-10-30T12:21:07.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e1a3ab640 1 -- 172.21.15.50:0/2626123000 learned_addr learned my addr 172.21.15.50:0/2626123000 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:07.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 10 monclient(hunting): get_auth_request con 0x7f5e140720b0 auth_method 0 2023-10-30T12:21:07.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.207+0000 7f5e193a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 10 monclient(hunting): handle_auth_done global_id 264144 payload 274 2023-10-30T12:21:07.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 1 -- 172.21.15.50:0/2626123000 >> v1:172.21.15.118:6789/0 conn(0x7f5e140a0080 legacy=0x7f5e140a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:07.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 1 -- 172.21.15.50:0/2626123000 >> v1:172.21.15.50:6789/0 conn(0x7f5e14073a30 legacy=0x7f5e1413d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 10 monclient: _finish_hunting 0 2023-10-30T12:21:07.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 1 monclient: found mon.noname-a 2023-10-30T12:21:07.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:07.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 1 -- 172.21.15.50:0/2626123000 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1409c2b0 con 0x7f5e140720b0 2023-10-30T12:21:07.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e18ba8640 1 -- 172.21.15.50:0/2626123000 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048441864 0 0) 0x7f5e1409ccc0 con 0x7f5e14073a30 2023-10-30T12:21:07.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e18ba8640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:21:07.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.208+0000 7f5e193a9640 1 --2- 172.21.15.50:0/2626123000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e140720b0 0x7f5e1413ad30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5e14073760 tx=0x7f5e0402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f982c96a10f7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 1 -- 172.21.15.50:0/2626123000 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0402f050 con 0x7f5e140720b0 2023-10-30T12:21:07.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:07.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: dump: 2023-10-30T12:21:07.225 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.225 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.225 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.226 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.226 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.226 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.226 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.226 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.227 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.227 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: _finish_auth 0 2023-10-30T12:21:07.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: _check_auth_tickets 2023-10-30T12:21:07.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:07.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 1 -- 172.21.15.50:0/2626123000 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e04031700 con 0x7f5e140720b0 2023-10-30T12:21:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 1 -- 172.21.15.50:0/2626123000 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04031a90 con 0x7f5e140720b0 2023-10-30T12:21:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 5 monclient: authenticate success, global_id 264144 2023-10-30T12:21:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:07.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 4 set_mon_vals no callback set 2023-10-30T12:21:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e18ba8640 10 monclient: dump: 2023-10-30T12:21:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.229 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.230 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.230 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.230 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.230 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.230 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.231 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 10 monclient: shutdown 2023-10-30T12:21:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2626123000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e140720b0 msgr2=0x7f5e1413ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2626123000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e140720b0 0x7f5e1413ad30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5e14073760 tx=0x7f5e0402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2626123000 shutdown_connections 2023-10-30T12:21:07.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2626123000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e140720b0 0x7f5e1413ad30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2626123000 >> 172.21.15.50:0/2626123000 conn(0x7f5e140a4870 msgr2=0x7f5e140a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:07.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2626123000 shutdown_connections 2023-10-30T12:21:07.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.209+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2626123000 wait complete. 2023-10-30T12:21:07.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 10 monclient: build_initial_monmap 2023-10-30T12:21:07.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 10 monclient: monmap: 2023-10-30T12:21:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:07.233 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:07.234 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.234 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:07.234 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:07.234 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:07.234 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 1 librados: starting msgr at 2023-10-30T12:21:07.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 1 librados: starting objecter 2023-10-30T12:21:07.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 1 Processor -- start 2023-10-30T12:21:07.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 1 -- start start 2023-10-30T12:21:07.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 1 librados: setting wanted keys 2023-10-30T12:21:07.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 1 librados: calling monclient init 2023-10-30T12:21:07.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.210+0000 7f5e1be0e640 10 monclient: init 2023-10-30T12:21:07.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:07.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:21:07.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e14073a30 0x7f5e141ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): picked mon.noname-c con 0x7f5e14073a30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:07.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e140a0080 0x7f5e140944c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): picked mon.noname-b con 0x7f5e140a0080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:07.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e141ca6c0 0x7f5e14094a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): picked mon.noname-a con 0x7f5e141ca6c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:07.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e14094f40 con 0x7f5e141ca6c0 2023-10-30T12:21:07.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e14073e90 con 0x7f5e140a0080 2023-10-30T12:21:07.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1408e6b0 con 0x7f5e14073a30 2023-10-30T12:21:07.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:07.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1be0e640 10 monclient(hunting): authenticate will time out at 2112.394248s 2023-10-30T12:21:07.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e14073a30 0x7f5e141ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e14073a30 0x7f5e141ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39198/0 (socket says 172.21.15.50:39198) 2023-10-30T12:21:07.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 1 -- 172.21.15.50:0/2206703957 learned_addr learned my addr 172.21.15.50:0/2206703957 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:07.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e141ca6c0 0x7f5e14094a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 10 monclient(hunting): get_auth_request con 0x7f5e14073a30 auth_method 0 2023-10-30T12:21:07.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): get_auth_request con 0x7f5e141ca6c0 auth_method 0 2023-10-30T12:21:07.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e193a9640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e140a0080 0x7f5e140944c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient(hunting): handle_auth_done global_id 264150 payload 824 2023-10-30T12:21:07.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 -- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e14073a30 msgr2=0x7f5e141ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e14073a30 0x7f5e141ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 -- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e140a0080 msgr2=0x7f5e140944c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e140a0080 0x7f5e140944c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient: _finish_hunting 0 2023-10-30T12:21:07.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 monclient: found mon.noname-a 2023-10-30T12:21:07.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:07.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 -- 172.21.15.50:0/2206703957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e04009400 con 0x7f5e141ca6c0 2023-10-30T12:21:07.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e193a9640 10 monclient: get_auth_request con 0x7f5e140a0080 auth_method 0 2023-10-30T12:21:07.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e19baa640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e14073a30 0x7f5e141ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:07.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e193a9640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e140a0080 0x7f5e140944c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:21:07.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e1a3ab640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e141ca6c0 0x7f5e14094a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5e14073e60 tx=0x7f5e0c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e0a7fc640 10 client.?.objecter ms_handle_connect 0x7f5e141ca6c0 2023-10-30T12:21:07.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.211+0000 7f5e0a7fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:07.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 1 -- 172.21.15.50:0/2206703957 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c004d60 con 0x7f5e141ca6c0 2023-10-30T12:21:07.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:07.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: dump: 2023-10-30T12:21:07.246 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.246 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.246 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.247 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.247 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.247 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.247 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.247 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.247 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.248 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:21:07.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:21:07.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:07.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 1 -- 172.21.15.50:0/2206703957 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0c00a640 con 0x7f5e141ca6c0 2023-10-30T12:21:07.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:07.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 1 -- 172.21.15.50:0/2206703957 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c00a9f0 con 0x7f5e141ca6c0 2023-10-30T12:21:07.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:07.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 10 monclient: dump: 2023-10-30T12:21:07.249 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.251 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.251 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.251 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.251 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 5 monclient: authenticate success, global_id 264150 2023-10-30T12:21:07.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 10 monclient: _renew_subs 2023-10-30T12:21:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0bfff640 4 set_mon_vals no callback set 2023-10-30T12:21:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1408e990 con 0x7f5e141ca6c0 2023-10-30T12:21:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 10 client.264150.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 10 monclient: _renew_subs 2023-10-30T12:21:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1408cc10 con 0x7f5e141ca6c0 2023-10-30T12:21:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e1be0e640 1 librados: init done 2023-10-30T12:21:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.212+0000 7f5e0a7fc640 1 -- 172.21.15.50:0/2206703957 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5e0c00ab50 con 0x7f5e141ca6c0 2023-10-30T12:21:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e0a7fc640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5df80392f0 0x7f5df803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e0a7fc640 1 -- 172.21.15.50:0/2206703957 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(653..653 src has 520..653) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f5e0c04a140 con 0x7f5e141ca6c0 2023-10-30T12:21:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e0a7fc640 10 client.264150.objecter ms_dispatch 0x7f5e14000b90 osd_map(653..653 src has 520..653) v4 2023-10-30T12:21:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e0a7fc640 3 client.264150.objecter handle_osd_map got epochs [653,653] > 0 2023-10-30T12:21:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e0a7fc640 3 client.264150.objecter handle_osd_map decoding full epoch 653 2023-10-30T12:21:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e1be0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e1be0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e1be0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1404e160 con 0x7f5e141ca6c0 2023-10-30T12:21:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e0a7fc640 20 client.264150.objecter dump_active .. 0 homeless 2023-10-30T12:21:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e19baa640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5df80392f0 0x7f5df803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e19baa640 10 monclient: get_auth_request con 0x7f5df80392f0 auth_method 0 2023-10-30T12:21:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e19baa640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5df80392f0 0x7f5df803b7b0 secure :-1 s=READY pgs=4327 cs=0 l=1 rev1=1 crypto rx=0x7f5e10007fd0 tx=0x7f5e10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.213+0000 7f5e0a7fc640 10 client.264150.objecter ms_handle_connect 0x7f5df80392f0 2023-10-30T12:21:07.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.217+0000 7f5e0a7fc640 1 -- 172.21.15.50:0/2206703957 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5e0c018080 con 0x7f5e141ca6c0 2023-10-30T12:21:07.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.217+0000 7f5e0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:07.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.217+0000 7f5e0a7fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:07.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.464+0000 7f5e1be0e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:21:07.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.464+0000 7f5e1be0e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:07.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.464+0000 7f5e1be0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.464+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e1406ab70 con 0x7f5e141ca6c0 2023-10-30T12:21:07.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.464+0000 7f5e0a7fc640 1 -- 172.21.15.50:0/2206703957 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f5e0c00ae00 con 0x7f5e141ca6c0 2023-10-30T12:21:07.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.464+0000 7f5e0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:07.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.464+0000 7f5e0a7fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:07.468 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":210,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:21:07.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 10 librados: watch_flush enter 2023-10-30T12:21:07.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 10 librados: watch_flush exit 2023-10-30T12:21:07.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 20 client.264150.objecter shutdown clearing up homeless session... 2023-10-30T12:21:07.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 10 client.264150.objecter successfully canceled tick 2023-10-30T12:21:07.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5df80392f0 msgr2=0x7f5df803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5df80392f0 0x7f5df803b7b0 secure :-1 s=READY pgs=4327 cs=0 l=1 rev1=1 crypto rx=0x7f5e10007fd0 tx=0x7f5e10009040 comp rx=0 tx=0).stop 2023-10-30T12:21:07.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 10 monclient: shutdown 2023-10-30T12:21:07.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:07.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e141ca6c0 msgr2=0x7f5e14094a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e141ca6c0 0x7f5e14094a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5e14073e60 tx=0x7f5e0c00cbe0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 shutdown_connections 2023-10-30T12:21:07.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f5df80392f0 0x7f5df803b7b0 unknown :-1 s=CLOSED pgs=4327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e141ca6c0 0x7f5e14094a00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e140a0080 0x7f5e140944c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 --2- 172.21.15.50:0/2206703957 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e14073a30 0x7f5e141ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 >> 172.21.15.50:0/2206703957 conn(0x7f5e140a4870 msgr2=0x7f5e141bbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:07.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.466+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 shutdown_connections 2023-10-30T12:21:07.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.467+0000 7f5e1be0e640 1 -- 172.21.15.50:0/2206703957 wait complete. 2023-10-30T12:21:07.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.467+0000 7f5e1be0e640 1 librados: shutdown 2023-10-30T12:21:07.478 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:21:07.478 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:21:07.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:21:07.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:21:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.583+0000 7f84f05e2640 10 monclient: get_monmap_and_config 2023-10-30T12:21:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.583+0000 7f84f05e2640 10 monclient: build_initial_monmap 2023-10-30T12:21:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.583+0000 7f84f05e2640 10 monclient: monmap: 2023-10-30T12:21:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:07.584814+0000 2023-10-30T12:21:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:07.584814+0000 2023-10-30T12:21:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:07.588 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 1 Processor -- start 2023-10-30T12:21:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 1 -- start start 2023-10-30T12:21:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient: init 2023-10-30T12:21:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:21:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84e809bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient(hunting): picked mon.noname-b con 0x7f84e809d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f84e809c660 0x7f84e8139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient(hunting): picked mon.noname-c con 0x7f84e809c660 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e8098170 con 0x7f84e809bb70 2023-10-30T12:21:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e809cf60 con 0x7f84e809c660 2023-10-30T12:21:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8097b20 con 0x7f84e809d3f0 2023-10-30T12:21:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.584+0000 7f84f05e2640 10 monclient(hunting): authenticate will time out at 2112.767978s 2023-10-30T12:21:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8137960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39208/0 (socket says 172.21.15.50:39208) 2023-10-30T12:21:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 -- 172.21.15.50:0/400005181 learned_addr learned my addr 172.21.15.50:0/400005181 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): get_auth_request con 0x7f84e809d3f0 auth_method 0 2023-10-30T12:21:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 1 --2- 172.21.15.50:0/400005181 >> v2:172.21.15.118:3300/0 conn(0x7f84e809c660 0x7f84e8139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 10 monclient(hunting): get_auth_request con 0x7f84e809c660 auth_method 0 2023-10-30T12:21:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84eeb7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient(hunting): handle_auth_done global_id 254213 payload 274 2023-10-30T12:21:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 -- 172.21.15.50:0/400005181 >> v2:172.21.15.118:3300/0 conn(0x7f84e809c660 msgr2=0x7f84e8139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 --2- 172.21.15.50:0/400005181 >> v2:172.21.15.118:3300/0 conn(0x7f84e809c660 0x7f84e8139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 -- 172.21.15.50:0/400005181 >> v1:172.21.15.118:6789/0 conn(0x7f84e809bb70 legacy=0x7f84e809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient: _finish_hunting 0 2023-10-30T12:21:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 monclient: found mon.noname-b 2023-10-30T12:21:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:21:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.585+0000 7f84edb7d640 1 -- 172.21.15.50:0/400005181 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8097e00 con 0x7f84e809d3f0 2023-10-30T12:21:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84edb7d640 1 --2- 172.21.15.50:0/400005181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8137960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c5a853204547bc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 1 -- 172.21.15.50:0/400005181 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80317c0 con 0x7f84e809d3f0 2023-10-30T12:21:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:21:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: dump: 2023-10-30T12:21:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.615 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: _finish_auth 0 2023-10-30T12:21:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: _check_auth_tickets 2023-10-30T12:21:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 5 monclient: authenticate success, global_id 254213 2023-10-30T12:21:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 1 -- 172.21.15.50:0/400005181 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d8031920 con 0x7f84e809d3f0 2023-10-30T12:21:07.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 4 set_mon_vals no callback set 2023-10-30T12:21:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 1 -- 172.21.15.50:0/400005181 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8031cb0 con 0x7f84e809d3f0 2023-10-30T12:21:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84ed37c640 10 monclient: dump: 2023-10-30T12:21:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.618 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.619 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.620 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 10 monclient: shutdown 2023-10-30T12:21:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 1 -- 172.21.15.50:0/400005181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 msgr2=0x7f84e8137960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 1 --2- 172.21.15.50:0/400005181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8137960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d802b860 comp rx=0 tx=0).stop 2023-10-30T12:21:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 1 -- 172.21.15.50:0/400005181 shutdown_connections 2023-10-30T12:21:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 1 --2- 172.21.15.50:0/400005181 >> v2:172.21.15.118:3300/0 conn(0x7f84e809c660 0x7f84e8139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 1 --2- 172.21.15.50:0/400005181 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8137960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.586+0000 7f84f05e2640 1 -- 172.21.15.50:0/400005181 >> 172.21.15.50:0/400005181 conn(0x7f84e80a3980 msgr2=0x7f84e80a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.587+0000 7f84f05e2640 1 -- 172.21.15.50:0/400005181 shutdown_connections 2023-10-30T12:21:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.587+0000 7f84f05e2640 1 -- 172.21.15.50:0/400005181 wait complete. 2023-10-30T12:21:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.587+0000 7f84f05e2640 10 monclient: build_initial_monmap 2023-10-30T12:21:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.587+0000 7f84f05e2640 10 monclient: monmap: 2023-10-30T12:21:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:07.623 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:07.624 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.587+0000 7f84f05e2640 1 librados: starting msgr at 2023-10-30T12:21:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.587+0000 7f84f05e2640 1 librados: starting objecter 2023-10-30T12:21:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.587+0000 7f84f05e2640 1 Processor -- start 2023-10-30T12:21:07.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 -- start start 2023-10-30T12:21:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 librados: setting wanted keys 2023-10-30T12:21:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 librados: calling monclient init 2023-10-30T12:21:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient: init 2023-10-30T12:21:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:21:07.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e809bb70 0x7f84e80962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): picked mon.noname-a con 0x7f84e809bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:07.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84e809c660 0x7f84e8096830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): picked mon.noname-b con 0x7f84e809c660 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:07.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): picked mon.noname-c con 0x7f84e809d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:07.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e813a2f0 con 0x7f84e809bb70 2023-10-30T12:21:07.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8097190 con 0x7f84e809c660 2023-10-30T12:21:07.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8139c30 con 0x7f84e809d3f0 2023-10-30T12:21:07.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:07.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84f05e2640 10 monclient(hunting): authenticate will time out at 2112.771481s 2023-10-30T12:21:07.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e809bb70 0x7f84e80962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39220/0 (socket says 172.21.15.50:39220) 2023-10-30T12:21:07.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 1 -- 172.21.15.50:0/4061237270 learned_addr learned my addr 172.21.15.50:0/4061237270 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:07.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): get_auth_request con 0x7f84e809bb70 auth_method 0 2023-10-30T12:21:07.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 10 monclient(hunting): get_auth_request con 0x7f84e809d3f0 auth_method 0 2023-10-30T12:21:07.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84eeb7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84edb7d640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84e809c660 0x7f84e8096830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84edb7d640 10 monclient(hunting): get_auth_request con 0x7f84e809c660 auth_method 0 2023-10-30T12:21:07.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84edb7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84edb7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84edb7d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient(hunting): handle_auth_done global_id 264153 payload 824 2023-10-30T12:21:07.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 msgr2=0x7f84e8136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f84e809d3f0 0x7f84e8136180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84e809c660 msgr2=0x7f84e8096830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84e809c660 0x7f84e8096830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient: _finish_hunting 0 2023-10-30T12:21:07.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 1 monclient: found mon.noname-a 2023-10-30T12:21:07.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:07.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.588+0000 7f84ee37e640 1 -- 172.21.15.50:0/4061237270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80094d0 con 0x7f84e809bb70 2023-10-30T12:21:07.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84ee37e640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e809bb70 0x7f84e80962f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000fb0 tx=0x7f84e400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 client.?.objecter ms_handle_connect 0x7f84e809bb70 2023-10-30T12:21:07.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:07.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e40052d0 con 0x7f84e809bb70 2023-10-30T12:21:07.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:07.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: dump: 2023-10-30T12:21:07.636 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.637 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.637 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.637 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.637 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.637 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.638 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.638 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.638 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.638 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: _finish_auth 0 2023-10-30T12:21:07.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: _check_auth_tickets 2023-10-30T12:21:07.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:07.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e4005430 con 0x7f84e809bb70 2023-10-30T12:21:07.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:07.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e4003880 con 0x7f84e809bb70 2023-10-30T12:21:07.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:07.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84deffd640 10 monclient: dump: 2023-10-30T12:21:07.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.640 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.641 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.641 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.641 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.642 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 5 monclient: authenticate success, global_id 264153 2023-10-30T12:21:07.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 10 monclient: _renew_subs 2023-10-30T12:21:07.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84ecb7b640 4 set_mon_vals no callback set 2023-10-30T12:21:07.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e80976e0 con 0x7f84e809bb70 2023-10-30T12:21:07.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 10 client.264153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:07.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 10 monclient: _renew_subs 2023-10-30T12:21:07.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e8139640 con 0x7f84e809bb70 2023-10-30T12:21:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.589+0000 7f84f05e2640 1 librados: init done 2023-10-30T12:21:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84f05e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84f05e2640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8136e80 0x7f84e8137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84f05e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8073ff0 con 0x7f84e8136e80 2023-10-30T12:21:07.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f84e40039e0 con 0x7f84e809bb70 2023-10-30T12:21:07.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8136e80 0x7f84e8137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient: get_auth_request con 0x7f84e8136e80 auth_method 0 2023-10-30T12:21:07.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84cc039320 0x7f84cc03b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(653..653 src has 520..653) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f84e404a340 con 0x7f84e809bb70 2023-10-30T12:21:07.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 10 client.264153.objecter ms_dispatch 0x7f84e8000b90 osd_map(653..653 src has 520..653) v4 2023-10-30T12:21:07.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 3 client.264153.objecter handle_osd_map got epochs [653,653] > 0 2023-10-30T12:21:07.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 3 client.264153.objecter handle_osd_map decoding full epoch 653 2023-10-30T12:21:07.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 10 monclient(hunting): handle_auth_done global_id 264156 payload 824 2023-10-30T12:21:07.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 1 -- 172.21.15.50:0/4061237270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84e8097560 con 0x7f84e8136e80 2023-10-30T12:21:07.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 20 client.264153.objecter dump_active .. 0 homeless 2023-10-30T12:21:07.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84edb7d640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8136e80 0x7f84e8137260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84e806b140 tx=0x7f84d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 10 client.264153.objecter ms_handle_connect 0x7f84e8136e80 2023-10-30T12:21:07.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 10 client.264153.objecter resend_mon_ops 2023-10-30T12:21:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84eeb7f640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84cc039320 0x7f84cc03b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84eeb7f640 10 monclient: get_auth_request con 0x7f84cc039320 auth_method 0 2023-10-30T12:21:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8049070 con 0x7f84e8136e80 2023-10-30T12:21:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.590+0000 7f84deffd640 10 monclient: dump: 2023-10-30T12:21:07.650 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.650 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.650 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.650 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.651 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.651 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.651 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.651 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f84d8032660 con 0x7f84e8136e80 2023-10-30T12:21:07.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:07.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:07.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8136e80 msgr2=0x7f84e8137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8136e80 0x7f84e8137260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84e806b140 tx=0x7f84d8004730 comp rx=0 tx=0).stop 2023-10-30T12:21:07.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8136e80 msgr2=0x7f84e8137260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:07.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8136e80 0x7f84e8137260 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84deffd640 10 client.264153.objecter ms_handle_connect 0x7f84cc039320 2023-10-30T12:21:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.591+0000 7f84eeb7f640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84cc039320 0x7f84cc03b7e0 secure :-1 s=READY pgs=4328 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009a40 tx=0x7f84e000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84f05e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84f05e2640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e802b430 0x7f84e802b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84f05e2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8139e70 con 0x7f84e802b430 2023-10-30T12:21:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e802b430 0x7f84e802b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient: get_auth_request con 0x7f84e802b430 auth_method 0 2023-10-30T12:21:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 10 monclient(hunting): handle_auth_done global_id 264159 payload 824 2023-10-30T12:21:07.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.597+0000 7f84edb7d640 1 -- 172.21.15.50:0/4061237270 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f84e809f600 con 0x7f84e802b430 2023-10-30T12:21:07.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 10 client.264153.objecter ms_handle_connect 0x7f84e802b430 2023-10-30T12:21:07.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 10 client.264153.objecter resend_mon_ops 2023-10-30T12:21:07.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84edb7d640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e802b430 0x7f84e802b840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84d80090b0 tx=0x7f84d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8058440 con 0x7f84e802b430 2023-10-30T12:21:07.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:07.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 10 monclient: dump: 2023-10-30T12:21:07.657 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.657 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.657 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.658 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.658 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.658 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.658 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.658 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.659 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4464 (secure 0 0 0) 0x7f84d80585a0 con 0x7f84e802b430 2023-10-30T12:21:07.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:07.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:07.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e802b430 msgr2=0x7f84e802b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e802b430 0x7f84e802b840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84d80090b0 tx=0x7f84d80046c0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e802b430 msgr2=0x7f84e802b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:07.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84deffd640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e802b430 0x7f84e802b840 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84d80090b0 tx=0x7f84d80046c0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.661 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:07.661 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:07.661 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:07.661 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:21:07.662 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 210, 2023-10-30T12:21:07.662 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:07.662 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:07.662 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:21:07.663 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:07.663 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.663 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:21:07.663 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:07.663 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:07.664 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:07.664 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:07.664 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:07.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:07.665 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:07.665 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:07.665 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:07.665 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:07.665 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:07.665 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:07.666 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:07.666 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.666 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:07.666 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:07.666 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:07.667 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:07.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:07.667 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:07.667 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:07.667 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:07.668 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:07.668 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:07.668 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:07.668 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:07.668 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.669 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:07.669 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:07.669 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:07.670 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:07.670 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:07.670 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:07.670 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:07.670 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:07.671 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:07.671 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:07.671 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:07.671 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:07.671 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:07.671 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:07.672 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:07.672 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:07.672 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:07.672 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:07.672 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:07.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:07.674 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:07.674 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.674 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:07.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:07.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.675 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:07.675 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:07.675 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:07.676 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:07.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.676 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:07.676 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:07.676 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.677 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:07.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:07.677 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.678 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:07.678 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:07.678 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:07.679 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:07.679 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:07.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.680 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:07.680 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:07.680 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:07.680 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:07.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:07.681 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:07.681 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.682 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:07.682 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:07.682 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.682 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.683 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.683 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:07.683 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:07.684 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:07.684 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:07.684 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:07.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.685 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:07.685 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:07.685 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:07.685 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:07.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.686 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:07.686 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:07.686 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.687 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:07.687 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:07.687 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.687 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.687 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.688 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:07.688 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:07.689 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:07.689 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:07.690 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:07.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.690 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.691 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:07.691 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:07.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.692 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:07.692 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:07.693 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:07.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.693 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.694 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:07.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.695 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:07.695 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:07.695 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:21:07.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.696 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:07.697 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:07.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 10 librados: watch_flush enter 2023-10-30T12:21:07.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 10 librados: watch_flush exit 2023-10-30T12:21:07.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 20 client.264153.objecter shutdown clearing up homeless session... 2023-10-30T12:21:07.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 10 client.264153.objecter successfully canceled tick 2023-10-30T12:21:07.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84cc039320 msgr2=0x7f84cc03b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84cc039320 0x7f84cc03b7e0 secure :-1 s=READY pgs=4328 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009a40 tx=0x7f84e000b040 comp rx=0 tx=0).stop 2023-10-30T12:21:07.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 10 monclient: shutdown 2023-10-30T12:21:07.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:07.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e809bb70 msgr2=0x7f84e80962f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 --2- 172.21.15.50:0/4061237270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e809bb70 0x7f84e80962f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000fb0 tx=0x7f84e400c890 comp rx=0 tx=0).stop 2023-10-30T12:21:07.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84eeb7f640 1 -- 172.21.15.50:0/4061237270 reap_dead start 2023-10-30T12:21:07.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84eeb7f640 1 -- 172.21.15.50:0/4061237270 reap_dead start 2023-10-30T12:21:07.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 shutdown_connections 2023-10-30T12:21:07.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 >> 172.21.15.50:0/4061237270 conn(0x7f84e80a3980 msgr2=0x7f84e806dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:07.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 shutdown_connections 2023-10-30T12:21:07.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 -- 172.21.15.50:0/4061237270 wait complete. 2023-10-30T12:21:07.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.598+0000 7f84f05e2640 1 librados: shutdown 2023-10-30T12:21:07.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:21:07.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:21:07.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:21:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.808+0000 7f5323ee5640 10 monclient: get_monmap_and_config 2023-10-30T12:21:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.808+0000 7f5323ee5640 10 monclient: build_initial_monmap 2023-10-30T12:21:07.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.808+0000 7f5323ee5640 10 monclient: monmap: 2023-10-30T12:21:07.810 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:07.810 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.810 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:07.809196+0000 2023-10-30T12:21:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:07.809196+0000 2023-10-30T12:21:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:07.812 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:07.812 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:07.812 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:07.812 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.808+0000 7f5323ee5640 1 Processor -- start 2023-10-30T12:21:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.808+0000 7f5323ee5640 1 -- start start 2023-10-30T12:21:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.808+0000 7f5323ee5640 10 monclient: init 2023-10-30T12:21:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:21:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f531c0a2ae0 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f531c0a2450 0x7f531c13aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient(hunting): picked mon.noname-a con 0x7f531c0a2450 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f531c13af40 0x7f531c13d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient(hunting): picked mon.noname-c con 0x7f531c13af40 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c0a4190 con 0x7f531c0a2ae0 2023-10-30T12:21:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531c0a1630 con 0x7f531c0a2450 2023-10-30T12:21:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531c09a2e0 con 0x7f531c13af40 2023-10-30T12:21:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5323ee5640 10 monclient(hunting): authenticate will time out at 2112.992415s 2023-10-30T12:21:07.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321c81640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f531c0a2ae0 0x7f531c0a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38898/0 (socket says 172.21.15.50:38898) 2023-10-30T12:21:07.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f531c0a2450 0x7f531c13aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321c81640 1 -- 172.21.15.50:0/93700680 learned_addr learned my addr 172.21.15.50:0/93700680 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:07.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): get_auth_request con 0x7f531c0a2450 auth_method 0 2023-10-30T12:21:07.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5322482640 1 --2- 172.21.15.50:0/93700680 >> v2:172.21.15.118:3300/0 conn(0x7f531c13af40 0x7f531c13d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5322482640 10 monclient(hunting): get_auth_request con 0x7f531c13af40 auth_method 0 2023-10-30T12:21:07.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5322482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5322482640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5322482640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient(hunting): handle_auth_done global_id 264162 payload 274 2023-10-30T12:21:07.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 1 -- 172.21.15.50:0/93700680 >> v2:172.21.15.118:3300/0 conn(0x7f531c13af40 msgr2=0x7f531c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:07.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 1 --2- 172.21.15.50:0/93700680 >> v2:172.21.15.118:3300/0 conn(0x7f531c13af40 0x7f531c13d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 1 -- 172.21.15.50:0/93700680 >> v1:172.21.15.50:6789/0 conn(0x7f531c0a2ae0 legacy=0x7f531c0a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient: _finish_hunting 0 2023-10-30T12:21:07.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 1 monclient: found mon.noname-a 2023-10-30T12:21:07.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5320c7f640 1 -- 172.21.15.50:0/93700680 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717311796 0 0) 0x7f5318001110 con 0x7f531c0a2ae0 2023-10-30T12:21:07.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:07.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5321480640 1 -- 172.21.15.50:0/93700680 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c09a8a0 con 0x7f531c0a2450 2023-10-30T12:21:07.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.809+0000 7f5320c7f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:21:07.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5321480640 1 --2- 172.21.15.50:0/93700680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2450 0x7f531c13aa00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f531c0705b0 tx=0x7f530c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1451a66ea7c4fdef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 1 -- 172.21.15.50:0/93700680 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c031710 con 0x7f531c0a2450 2023-10-30T12:21:07.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:07.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: dump: 2023-10-30T12:21:07.831 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.831 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.831 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.832 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.832 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.832 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.832 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.832 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.832 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.833 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: _finish_auth 0 2023-10-30T12:21:07.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: _check_auth_tickets 2023-10-30T12:21:07.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:07.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 1 -- 172.21.15.50:0/93700680 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c02f050 con 0x7f531c0a2450 2023-10-30T12:21:07.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:07.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 1 -- 172.21.15.50:0/93700680 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c031a90 con 0x7f531c0a2450 2023-10-30T12:21:07.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5323ee5640 5 monclient: authenticate success, global_id 264162 2023-10-30T12:21:07.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5323ee5640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:07.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5323ee5640 4 set_mon_vals no callback set 2023-10-30T12:21:07.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:07.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.810+0000 7f5320c7f640 10 monclient: dump: 2023-10-30T12:21:07.835 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.835 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.836 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.836 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.836 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.836 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.836 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.837 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.837 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.837 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 10 monclient: shutdown 2023-10-30T12:21:07.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:07.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 -- 172.21.15.50:0/93700680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2450 msgr2=0x7f531c13aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 --2- 172.21.15.50:0/93700680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2450 0x7f531c13aa00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f531c0705b0 tx=0x7f530c02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 -- 172.21.15.50:0/93700680 shutdown_connections 2023-10-30T12:21:07.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 --2- 172.21.15.50:0/93700680 >> v2:172.21.15.118:3300/0 conn(0x7f531c13af40 0x7f531c13d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 --2- 172.21.15.50:0/93700680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2450 0x7f531c13aa00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 -- 172.21.15.50:0/93700680 >> 172.21.15.50:0/93700680 conn(0x7f531c0a80b0 msgr2=0x7f531c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:07.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 -- 172.21.15.50:0/93700680 shutdown_connections 2023-10-30T12:21:07.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 1 -- 172.21.15.50:0/93700680 wait complete. 2023-10-30T12:21:07.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 10 monclient: build_initial_monmap 2023-10-30T12:21:07.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.811+0000 7f5323ee5640 10 monclient: monmap: 2023-10-30T12:21:07.840 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:07.840 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:07.840 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:07.840 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:07.840 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:07.841 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.841 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:07.841 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:07.841 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:07.841 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 librados: starting msgr at 2023-10-30T12:21:07.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 librados: starting objecter 2023-10-30T12:21:07.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 Processor -- start 2023-10-30T12:21:07.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 -- start start 2023-10-30T12:21:07.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 librados: setting wanted keys 2023-10-30T12:21:07.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 librados: calling monclient init 2023-10-30T12:21:07.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient: init 2023-10-30T12:21:07.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:07.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:21:07.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f531c0a2450 0x7f531c1ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): picked mon.noname-b con 0x7f531c0a2450 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:07.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2ae0 0x7f531c1320c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): picked mon.noname-a con 0x7f531c0a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:07.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c13af40 0x7f531c09d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): picked mon.noname-c con 0x7f531c13af40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:07.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c11e4b0 con 0x7f531c0a2ae0 2023-10-30T12:21:07.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0a1790 con 0x7f531c0a2450 2023-10-30T12:21:07.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c13a810 con 0x7f531c13af40 2023-10-30T12:21:07.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:07.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5323ee5640 10 monclient(hunting): authenticate will time out at 2112.995858s 2023-10-30T12:21:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321480640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2ae0 0x7f531c1320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321480640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2ae0 0x7f531c1320c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49230/0 (socket says 172.21.15.50:49230) 2023-10-30T12:21:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321480640 1 -- 172.21.15.50:0/2375621222 learned_addr learned my addr 172.21.15.50:0/2375621222 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321480640 10 monclient(hunting): get_auth_request con 0x7f531c0a2ae0 auth_method 0 2023-10-30T12:21:07.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321480640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321480640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321480640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5322482640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c13af40 0x7f531c09d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.812+0000 7f5321c81640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f531c0a2450 0x7f531c1ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5322482640 10 monclient(hunting): get_auth_request con 0x7f531c13af40 auth_method 0 2023-10-30T12:21:07.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5322482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5322482640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5322482640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321c81640 10 monclient(hunting): get_auth_request con 0x7f531c0a2450 auth_method 0 2023-10-30T12:21:07.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321c81640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321c81640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5322482640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5322482640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5322482640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 10 monclient(hunting): handle_auth_done global_id 264168 payload 824 2023-10-30T12:21:07.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c13af40 msgr2=0x7f531c09d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c13af40 0x7f531c09d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f531c0a2450 msgr2=0x7f531c1ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f531c0a2450 0x7f531c1ca140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 10 monclient: _finish_hunting 0 2023-10-30T12:21:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 1 monclient: found mon.noname-a 2023-10-30T12:21:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 1 -- 172.21.15.50:0/2375621222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c009400 con 0x7f531c0a2ae0 2023-10-30T12:21:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5321480640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2ae0 0x7f531c1320c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f530c000c00 tx=0x7f530c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 client.?.objecter ms_handle_connect 0x7f531c0a2ae0 2023-10-30T12:21:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c033910 con 0x7f531c0a2ae0 2023-10-30T12:21:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: dump: 2023-10-30T12:21:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.859 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.859 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.859 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.859 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.859 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: _finish_auth 0 2023-10-30T12:21:07.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: _check_auth_tickets 2023-10-30T12:21:07.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:07.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c033a70 con 0x7f531c0a2ae0 2023-10-30T12:21:07.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 5 monclient: authenticate success, global_id 264168 2023-10-30T12:21:07.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:07.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 10 monclient: _renew_subs 2023-10-30T12:21:07.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c033e20 con 0x7f531c0a2ae0 2023-10-30T12:21:07.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c0a0430 con 0x7f531c0a2ae0 2023-10-30T12:21:07.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:07.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5313fff640 4 set_mon_vals no callback set 2023-10-30T12:21:07.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f53127fc640 10 monclient: dump: 2023-10-30T12:21:07.862 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.862 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.862 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.863 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.863 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.863 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.863 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.863 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.863 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.864 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 10 client.264168.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:07.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 10 monclient: _renew_subs 2023-10-30T12:21:07.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:07.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c1cafd0 con 0x7f531c0a2ae0 2023-10-30T12:21:07.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.813+0000 7f5323ee5640 1 librados: init done 2023-10-30T12:21:07.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5323ee5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:07.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5323ee5640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09dd50 0x7f531c09e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5323ee5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c073ff0 con 0x7f531c09dd50 2023-10-30T12:21:07.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f530c03a040 con 0x7f531c0a2ae0 2023-10-30T12:21:07.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09dd50 0x7f531c09e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient: get_auth_request con 0x7f531c09dd50 auth_method 0 2023-10-30T12:21:07.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f53000392f0 0x7f530003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 10 monclient(hunting): handle_auth_done global_id 254222 payload 824 2023-10-30T12:21:07.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 1 -- 172.21.15.50:0/2375621222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f531c09a5c0 con 0x7f531c09dd50 2023-10-30T12:21:07.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(653..653 src has 520..653) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f530c06c3e0 con 0x7f531c0a2ae0 2023-10-30T12:21:07.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 10 client.264168.objecter ms_dispatch 0x7f531c000b90 osd_map(653..653 src has 520..653) v4 2023-10-30T12:21:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 3 client.264168.objecter handle_osd_map got epochs [653,653] > 0 2023-10-30T12:21:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 3 client.264168.objecter handle_osd_map decoding full epoch 653 2023-10-30T12:21:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 20 client.264168.objecter dump_active .. 0 homeless 2023-10-30T12:21:07.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 10 client.264168.objecter ms_handle_connect 0x7f531c09dd50 2023-10-30T12:21:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f53127fc640 10 client.264168.objecter resend_mon_ops 2023-10-30T12:21:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5321c81640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09dd50 0x7f531c09e130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0080 tx=0x7f5318001b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.814+0000 7f5322482640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f53000392f0 0x7f530003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f5322482640 10 monclient: get_auth_request con 0x7f53000392f0 auth_method 0 2023-10-30T12:21:07.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318007680 con 0x7f531c09dd50 2023-10-30T12:21:07.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:07.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 10 monclient: dump: 2023-10-30T12:21:07.874 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.875 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.875 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.875 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.875 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.876 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.876 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.876 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.877 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f531800f820 con 0x7f531c09dd50 2023-10-30T12:21:07.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:07.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:07.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09dd50 msgr2=0x7f531c09e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09dd50 0x7f531c09e130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0080 tx=0x7f5318001b00 comp rx=0 tx=0).stop 2023-10-30T12:21:07.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09dd50 msgr2=0x7f531c09e130 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:07.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09dd50 0x7f531c09e130 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0080 tx=0x7f5318001b00 comp rx=0 tx=0).stop 2023-10-30T12:21:07.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f53127fc640 10 client.264168.objecter ms_handle_connect 0x7f53000392f0 2023-10-30T12:21:07.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.815+0000 7f5322482640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f53000392f0 0x7f530003b7b0 secure :-1 s=READY pgs=4329 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0d80 tx=0x7f5314009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5323ee5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5323ee5640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09ad50 0x7f531c09b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5323ee5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:07.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531c073a20 con 0x7f531c09ad50 2023-10-30T12:21:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09ad50 0x7f531c09b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 10 monclient: get_auth_request con 0x7f531c09ad50 auth_method 0 2023-10-30T12:21:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.821+0000 7f5321c81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5321c81640 10 monclient(hunting): handle_auth_done global_id 254225 payload 824 2023-10-30T12:21:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5321c81640 1 -- 172.21.15.50:0/2375621222 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f531c0a28a0 con 0x7f531c09ad50 2023-10-30T12:21:07.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5321c81640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09ad50 0x7f531c09b160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0080 tx=0x7f5318001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 10 client.264168.objecter ms_handle_connect 0x7f531c09ad50 2023-10-30T12:21:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 10 client.264168.objecter resend_mon_ops 2023-10-30T12:21:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318023070 con 0x7f531c09ad50 2023-10-30T12:21:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 10 monclient: dump: 2023-10-30T12:21:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.886 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:07.887 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:07.888 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f53180133f0 con 0x7f531c09ad50 2023-10-30T12:21:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09ad50 msgr2=0x7f531c09b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09ad50 0x7f531c09b160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0080 tx=0x7f5318001080 comp rx=0 tx=0).stop 2023-10-30T12:21:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09ad50 msgr2=0x7f531c09b160 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f53127fc640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f531c09ad50 0x7f531c09b160 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0080 tx=0x7f5318001080 comp rx=0 tx=0).stop 2023-10-30T12:21:07.889 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:07.890 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:07.890 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:07.890 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:21:07.890 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 210, 2023-10-30T12:21:07.891 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:07.891 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:07.891 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:21:07.891 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:07.891 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.891 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:21:07.892 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:07.892 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:07.892 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:07.893 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:07.893 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:07.893 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:07.893 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:07.893 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:07.893 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:07.894 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:07.894 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:07.894 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:07.894 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:07.895 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:07.896 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:07.896 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:07.896 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:07.896 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:07.896 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:07.897 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:07.897 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:07.897 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.897 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:07.898 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:07.898 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:07.898 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:07.898 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:07.898 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:07.898 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:07.899 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:07.899 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:07.899 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:07.899 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:07.899 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:07.899 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:07.900 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:07.900 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:07.900 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:07.900 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:07.900 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:07.900 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:07.901 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:07.901 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:07.901 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:07.901 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:07.902 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:07.902 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:07.903 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:07.903 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:07.903 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.904 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:07.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.904 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:07.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.905 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:07.905 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:07.905 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:07.906 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:07.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.906 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:07.906 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:07.907 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:07.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:07.908 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.909 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:07.909 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:07.909 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:07.909 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:07.909 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:07.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.910 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:07.910 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:07.910 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:07.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:07.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:07.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:07.911 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:07.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:07.912 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.913 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.913 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:07.913 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:07.913 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:07.913 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:07.914 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:07.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.914 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:07.914 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:07.915 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:07.915 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:07.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.915 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:07.915 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:07.916 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.916 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:07.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:07.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:07.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.917 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.917 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:07.918 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:07.918 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:07.918 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:07.918 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:07.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.919 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.919 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:07.919 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:07.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.920 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:07.920 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:07.920 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:07.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.920 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.920 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:21:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:21:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.921 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.922 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:07.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.922 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:07.922 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:07.922 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:07.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.923 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:07.923 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:21:07.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:07.923 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:07.923 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:07.924 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:07.924 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:07.924 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:07.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:07.924 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:07.924 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:07.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 10 librados: watch_flush enter 2023-10-30T12:21:07.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 10 librados: watch_flush exit 2023-10-30T12:21:07.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 20 client.264168.objecter shutdown clearing up homeless session... 2023-10-30T12:21:07.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 10 client.264168.objecter successfully canceled tick 2023-10-30T12:21:07.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f53000392f0 msgr2=0x7f530003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f53000392f0 0x7f530003b7b0 secure :-1 s=READY pgs=4329 cs=0 l=1 rev1=1 crypto rx=0x7f531c0a0d80 tx=0x7f5314009820 comp rx=0 tx=0).stop 2023-10-30T12:21:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 10 monclient: shutdown 2023-10-30T12:21:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2ae0 msgr2=0x7f531c1320c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2ae0 0x7f531c1320c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f530c000c00 tx=0x7f530c0046c0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5322482640 1 -- 172.21.15.50:0/2375621222 reap_dead start 2023-10-30T12:21:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 shutdown_connections 2023-10-30T12:21:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 1 --2- 172.21.15.50:0/2375621222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f531c0a2ae0 0x7f531c1320c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.822+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 >> 172.21.15.50:0/2375621222 conn(0x7f531c0a80b0 msgr2=0x7f531c1347b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.823+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 shutdown_connections 2023-10-30T12:21:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.823+0000 7f5323ee5640 1 -- 172.21.15.50:0/2375621222 wait complete. 2023-10-30T12:21:07.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:07.823+0000 7f5323ee5640 1 librados: shutdown 2023-10-30T12:21:07.930 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:21:07.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:21:07.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:21:08.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.036+0000 7f8bcab73640 10 monclient: get_monmap_and_config 2023-10-30T12:21:08.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.036+0000 7f8bcab73640 10 monclient: build_initial_monmap 2023-10-30T12:21:08.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.036+0000 7f8bcab73640 10 monclient: monmap: 2023-10-30T12:21:08.038 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:08.039 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:08.039 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:08.037744+0000 2023-10-30T12:21:08.039 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:08.037744+0000 2023-10-30T12:21:08.039 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:08.039 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:08.040 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:08.040 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:08.040 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:08.040 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:08.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 1 Processor -- start 2023-10-30T12:21:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 1 -- start start 2023-10-30T12:21:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient: init 2023-10-30T12:21:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-10-30T12:21:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bc40a0080 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bc40a0460 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8bc4139bd0 0x7f8bc413bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc4139bd0 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc409e440 con 0x7f8bc40a0080 2023-10-30T12:21:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc4073c30 con 0x7f8bc40a0460 2023-10-30T12:21:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc409e610 con 0x7f8bc4139bd0 2023-10-30T12:21:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.037+0000 7f8bcab73640 10 monclient(hunting): authenticate will time out at 2113.221000s 2023-10-30T12:21:08.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc890f640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8bc40a0080 0x7f8bc4098f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38908/0 (socket says 172.21.15.50:38908) 2023-10-30T12:21:08.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc890f640 1 -- 172.21.15.50:0/2950390814 learned_addr learned my addr 172.21.15.50:0/2950390814 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:08.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 1 --2- 172.21.15.50:0/2950390814 >> v2:172.21.15.118:3300/0 conn(0x7f8bc4139bd0 0x7f8bc413bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:08.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): get_auth_request con 0x7f8bc4139bd0 auth_method 0 2023-10-30T12:21:08.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:08.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:08.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:08.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 1 -- 172.21.15.50:0/2950390814 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141872997 0 0) 0x7f8bc409e440 con 0x7f8bc40a0080 2023-10-30T12:21:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): _negotiate 2023-10-30T12:21:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): my global_id is 264171 2023-10-30T12:21:08.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 1 -- 172.21.15.50:0/2950390814 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb4003520 con 0x7f8bc40a0080 2023-10-30T12:21:08.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 1 -- 172.21.15.50:0/2950390814 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288748877 0 0) 0x7f8bc4073c30 con 0x7f8bc40a0460 2023-10-30T12:21:08.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): _negotiate 2023-10-30T12:21:08.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 10 monclient(hunting): my global_id is 204763 2023-10-30T12:21:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc37fe640 1 -- 172.21.15.50:0/2950390814 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc409e440 con 0x7f8bc40a0460 2023-10-30T12:21:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient(hunting): handle_auth_done global_id 204760 payload 274 2023-10-30T12:21:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 1 -- 172.21.15.50:0/2950390814 >> v1:172.21.15.118:6789/0 conn(0x7f8bc40a0460 legacy=0x7f8bc40a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 1 -- 172.21.15.50:0/2950390814 >> v1:172.21.15.50:6789/0 conn(0x7f8bc40a0080 legacy=0x7f8bc4098f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient: _finish_hunting 0 2023-10-30T12:21:08.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 1 monclient: found mon.noname-c 2023-10-30T12:21:08.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:08.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:21:08.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.038+0000 7f8bc9110640 1 -- 172.21.15.50:0/2950390814 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc4074380 con 0x7f8bc4139bd0 2023-10-30T12:21:08.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc9110640 1 --2- 172.21.15.50:0/2950390814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc4139bd0 0x7f8bc413bfc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc009d90 tx=0x7f8bbc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae04e1aacd1e638 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 1 -- 172.21.15.50:0/2950390814 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc02cc60 con 0x7f8bc4139bd0 2023-10-30T12:21:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: dump: 2023-10-30T12:21:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.060 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:08.060 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:08.060 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:08.060 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: _finish_auth 0 2023-10-30T12:21:08.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:21:08.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:08.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 1 -- 172.21.15.50:0/2950390814 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc02cdc0 con 0x7f8bc4139bd0 2023-10-30T12:21:08.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bcab73640 5 monclient: authenticate success, global_id 204760 2023-10-30T12:21:08.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bcab73640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:08.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:08.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bcab73640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:08.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 1 -- 172.21.15.50:0/2950390814 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc034660 con 0x7f8bc4139bd0 2023-10-30T12:21:08.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:08.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bcab73640 4 set_mon_vals no callback set 2023-10-30T12:21:08.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:21:08.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.039+0000 7f8bc37fe640 10 monclient: dump: 2023-10-30T12:21:08.063 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:08.063 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:08.063 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.063 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.064 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:08.064 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.064 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:08.064 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:08.064 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:08.064 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 10 monclient: shutdown 2023-10-30T12:21:08.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:08.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 -- 172.21.15.50:0/2950390814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc4139bd0 msgr2=0x7f8bc413bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 --2- 172.21.15.50:0/2950390814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc4139bd0 0x7f8bc413bfc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc009d90 tx=0x7f8bbc02beb0 comp rx=0 tx=0).stop 2023-10-30T12:21:08.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 -- 172.21.15.50:0/2950390814 shutdown_connections 2023-10-30T12:21:08.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 --2- 172.21.15.50:0/2950390814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc4139bd0 0x7f8bc413bfc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:08.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 -- 172.21.15.50:0/2950390814 >> 172.21.15.50:0/2950390814 conn(0x7f8bc40a1170 msgr2=0x7f8bc40a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:08.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 -- 172.21.15.50:0/2950390814 shutdown_connections 2023-10-30T12:21:08.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 -- 172.21.15.50:0/2950390814 wait complete. 2023-10-30T12:21:08.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 10 monclient: build_initial_monmap 2023-10-30T12:21:08.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 10 monclient: monmap: 2023-10-30T12:21:08.067 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:08.067 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:08.067 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:08.067 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:08.067 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:08.067 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.068 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:08.068 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:08.068 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:08.068 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 librados: starting msgr at 2023-10-30T12:21:08.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.040+0000 7f8bcab73640 1 librados: starting objecter 2023-10-30T12:21:08.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 Processor -- start 2023-10-30T12:21:08.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 -- start start 2023-10-30T12:21:08.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 librados: setting wanted keys 2023-10-30T12:21:08.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 librados: calling monclient init 2023-10-30T12:21:08.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient: init 2023-10-30T12:21:08.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:08.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:21:08.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc40a0080 0x7f8bc41ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:08.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): picked mon.noname-b con 0x7f8bc40a0080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:08.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc40a0460 0x7f8bc41cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:08.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): picked mon.noname-a con 0x7f8bc40a0460 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:08.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc4139bd0 0x7f8bc409bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:08.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc4139bd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:08.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:08.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4073e90 con 0x7f8bc40a0460 2023-10-30T12:21:08.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:08.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40922e0 con 0x7f8bc40a0080 2023-10-30T12:21:08.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:08.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc413b800 con 0x7f8bc4139bd0 2023-10-30T12:21:08.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:08.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bcab73640 10 monclient(hunting): authenticate will time out at 2113.224771s 2023-10-30T12:21:08.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc40a0460 0x7f8bc41cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:08.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc9110640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc4139bd0 0x7f8bc409bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:08.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc40a0460 0x7f8bc41cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49238/0 (socket says 172.21.15.50:49238) 2023-10-30T12:21:08.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc9110640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc4139bd0 0x7f8bc409bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40998/0 (socket says 172.21.15.50:40998) 2023-10-30T12:21:08.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 1 -- 172.21.15.50:0/745005852 learned_addr learned my addr 172.21.15.50:0/745005852 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:08.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 10 monclient(hunting): get_auth_request con 0x7f8bc40a0460 auth_method 0 2023-10-30T12:21:08.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:08.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc9110640 10 monclient(hunting): get_auth_request con 0x7f8bc4139bd0 auth_method 0 2023-10-30T12:21:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc9110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc9110640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc9110640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc890f640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc40a0080 0x7f8bc41ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc890f640 10 monclient(hunting): get_auth_request con 0x7f8bc40a0080 auth_method 0 2023-10-30T12:21:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc890f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc890f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc890f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.041+0000 7f8bc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:08.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc9110640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:08.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc9110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:08.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc9110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:08.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:08.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:08.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:08.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 10 monclient(hunting): handle_auth_done global_id 264174 payload 824 2023-10-30T12:21:08.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc4139bd0 msgr2=0x7f8bc409bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc4139bd0 0x7f8bc409bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:08.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc40a0080 msgr2=0x7f8bc41ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc40a0080 0x7f8bc41ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:08.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 10 monclient: _finish_hunting 0 2023-10-30T12:21:08.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 1 monclient: found mon.noname-a 2023-10-30T12:21:08.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:08.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:08.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 1 -- 172.21.15.50:0/745005852 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bbc009a70 con 0x7f8bc40a0460 2023-10-30T12:21:08.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc3fff640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc40a0460 0x7f8bc41cab70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000c500 tx=0x7f8bb000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:08.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 client.?.objecter ms_handle_connect 0x7f8bc40a0460 2023-10-30T12:21:08.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:08.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb00052d0 con 0x7f8bc40a0460 2023-10-30T12:21:08.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:08.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:08.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: dump: 2023-10-30T12:21:08.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:08.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:08.081 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.081 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.082 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:08.082 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.082 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:08.082 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:08.082 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:08.083 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: _finish_auth 0 2023-10-30T12:21:08.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: _check_auth_tickets 2023-10-30T12:21:08.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:08.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 5 monclient: authenticate success, global_id 264174 2023-10-30T12:21:08.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb0005470 con 0x7f8bc40a0460 2023-10-30T12:21:08.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 10 monclient: _renew_subs 2023-10-30T12:21:08.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:08.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc4091d40 con 0x7f8bc40a0460 2023-10-30T12:21:08.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:08.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0003830 con 0x7f8bc40a0460 2023-10-30T12:21:08.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:08.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:08.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc17fa640 10 monclient: dump: 2023-10-30T12:21:08.085 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:08.085 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:08.086 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.086 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.086 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:08.086 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.086 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:08.087 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:08.087 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:08.087 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bc2ffd640 4 set_mon_vals no callback set 2023-10-30T12:21:08.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 10 client.264174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:08.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 10 monclient: _renew_subs 2023-10-30T12:21:08.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:08.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc4092000 con 0x7f8bc40a0460 2023-10-30T12:21:08.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.042+0000 7f8bcab73640 1 librados: init done 2023-10-30T12:21:08.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bcab73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:08.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bcab73640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409c0e0 0x7f8bc409c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:08.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bcab73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:08.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc406a5a0 con 0x7f8bc409c0e0 2023-10-30T12:21:08.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8bb0003990 con 0x7f8bc40a0460 2023-10-30T12:21:08.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409c0e0 0x7f8bc409c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:08.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc17fa640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ba8039320 0x7f8ba803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:08.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 10 monclient: get_auth_request con 0x7f8bc409c0e0 auth_method 0 2023-10-30T12:21:08.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:08.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:08.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:08.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(654..654 src has 520..654) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f8bb004a2a0 con 0x7f8bc40a0460 2023-10-30T12:21:08.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc17fa640 10 client.264174.objecter ms_dispatch 0x7f8bc4000b90 osd_map(654..654 src has 520..654) v4 2023-10-30T12:21:08.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc17fa640 3 client.264174.objecter handle_osd_map got epochs [654,654] > 0 2023-10-30T12:21:08.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc17fa640 3 client.264174.objecter handle_osd_map decoding full epoch 654 2023-10-30T12:21:08.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:08.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc9110640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ba8039320 0x7f8ba803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:08.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.043+0000 7f8bc17fa640 20 client.264174.objecter dump_active .. 0 homeless 2023-10-30T12:21:08.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc9110640 10 monclient: get_auth_request con 0x7f8ba8039320 auth_method 0 2023-10-30T12:21:08.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc890f640 10 monclient(hunting): handle_auth_done global_id 204769 payload 824 2023-10-30T12:21:08.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc890f640 1 -- 172.21.15.50:0/745005852 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8bc4098880 con 0x7f8bc409c0e0 2023-10-30T12:21:08.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 client.264174.objecter ms_handle_connect 0x7f8bc409c0e0 2023-10-30T12:21:08.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 client.264174.objecter resend_mon_ops 2023-10-30T12:21:08.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc890f640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409c0e0 0x7f8bc409c4c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000f60 tx=0x7f8bb8001990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:08.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8013600 con 0x7f8bc409c0e0 2023-10-30T12:21:08.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:08.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:21:08.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 monclient: dump: 2023-10-30T12:21:08.094 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:08.094 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:08.094 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.095 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.095 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:08.095 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.095 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:08.095 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:08.096 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:08.096 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 client.264174.objecter ms_handle_connect 0x7f8ba8039320 2023-10-30T12:21:08.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc9110640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ba8039320 0x7f8ba803b7e0 secure :-1 s=READY pgs=4330 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc036000 tx=0x7f8bbc030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:08.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8bb8013760 con 0x7f8bc409c0e0 2023-10-30T12:21:08.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:08.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:08.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409c0e0 msgr2=0x7f8bc409c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409c0e0 0x7f8bc409c4c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000f60 tx=0x7f8bb8001990 comp rx=0 tx=0).stop 2023-10-30T12:21:08.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409c0e0 msgr2=0x7f8bc409c4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:08.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.044+0000 7f8bc17fa640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409c0e0 0x7f8bc409c4c0 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000f60 tx=0x7f8bb8001990 comp rx=0 tx=0).stop 2023-10-30T12:21:08.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.050+0000 7f8bcab73640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:08.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.050+0000 7f8bcab73640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409ca00 0x7f8bc40704c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:08.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.050+0000 7f8bcab73640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:08.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.050+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc404e160 con 0x7f8bc409ca00 2023-10-30T12:21:08.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409ca00 0x7f8bc40704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:08.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient: get_auth_request con 0x7f8bc409ca00 auth_method 0 2023-10-30T12:21:08.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:08.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:08.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:08.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:08.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:08.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:08.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 10 monclient(hunting): handle_auth_done global_id 204772 payload 824 2023-10-30T12:21:08.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 1 -- 172.21.15.50:0/745005852 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8bc4092420 con 0x7f8bc409ca00 2023-10-30T12:21:08.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc890f640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409ca00 0x7f8bc40704c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000f60 tx=0x7f8bb8001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:08.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc17fa640 10 client.264174.objecter ms_handle_connect 0x7f8bc409ca00 2023-10-30T12:21:08.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc17fa640 10 client.264174.objecter resend_mon_ops 2023-10-30T12:21:08.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8024070 con 0x7f8bc409ca00 2023-10-30T12:21:08.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:08.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:21:08.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.051+0000 7f8bc17fa640 10 monclient: dump: 2023-10-30T12:21:08.103 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:08.104 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:08.104 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.104 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:08.104 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:08.104 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:08.104 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:08.105 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:08.105 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:08.105 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:08.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7f8bb8006110 con 0x7f8bc409ca00 2023-10-30T12:21:08.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:08.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:08.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409ca00 msgr2=0x7f8bc40704c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc17fa640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409ca00 0x7f8bc40704c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000f60 tx=0x7f8bb8001150 comp rx=0 tx=0).stop 2023-10-30T12:21:08.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc17fa640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409ca00 msgr2=0x7f8bc40704c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:08.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc17fa640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8bc409ca00 0x7f8bc40704c0 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000f60 tx=0x7f8bb8001150 comp rx=0 tx=0).stop 2023-10-30T12:21:08.107 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:08.107 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:08.107 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:08.107 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:21:08.108 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 210, 2023-10-30T12:21:08.108 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:08.108 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:08.108 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:21:08.108 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:08.109 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:08.109 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:21:08.109 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:08.109 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:08.109 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:08.110 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:08.110 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:08.110 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:08.110 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:08.110 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:08.111 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:08.111 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:08.111 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:08.111 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:08.111 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:08.112 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:08.112 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:08.112 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:08.112 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:08.112 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:08.112 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:08.113 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:08.113 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:08.113 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:08.113 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:08.113 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:08.113 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:08.114 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:08.114 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:08.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.114 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:08.114 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:08.115 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:08.115 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:08.115 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:08.115 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:08.115 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:08.116 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:08.116 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:08.116 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:08.116 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:08.116 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:08.117 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:08.117 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:08.117 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:08.117 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:08.117 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:08.117 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:08.118 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:08.118 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:08.118 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:08.118 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:08.118 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:08.119 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:08.119 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:08.119 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:08.119 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:08.119 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:08.120 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:08.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.120 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:08.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.120 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:08.120 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:08.121 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:08.121 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:08.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.121 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:08.121 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:08.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:08.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:08.122 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:08.123 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:08.123 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:08.123 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:08.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.123 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:08.123 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:08.124 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:08.124 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:08.124 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:08.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.125 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:08.125 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:08.125 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:08.125 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:08.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.126 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:08.126 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:08.126 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:08.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.127 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:08.127 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:08.127 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:08.127 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:08.127 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:08.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.128 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:08.128 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:08.128 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:08.128 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:08.128 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:08.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.129 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:08.129 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:08.129 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:08.130 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:08.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:08.130 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:08.130 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:08.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.131 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:08.131 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:08.131 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:08.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:08.132 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:08.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.132 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:08.132 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:08.132 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:08.132 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:08.133 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:08.133 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:08.133 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:08.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.133 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:08.134 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:08.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.134 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:08.134 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:08.134 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:08.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:08.135 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:08.135 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:21:08.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.135 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:08.136 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:08.136 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:21:08.136 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:08.136 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:08.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:08.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:08.137 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:08.137 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:08.137 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:21:08.137 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:08.137 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:08.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:08.138 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:08.138 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 10 librados: watch_flush enter 2023-10-30T12:21:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 10 librados: watch_flush exit 2023-10-30T12:21:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 20 client.264174.objecter shutdown clearing up homeless session... 2023-10-30T12:21:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 10 client.264174.objecter successfully canceled tick 2023-10-30T12:21:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ba8039320 msgr2=0x7f8ba803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8ba8039320 0x7f8ba803b7e0 secure :-1 s=READY pgs=4330 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc036000 tx=0x7f8bbc030000 comp rx=0 tx=0).stop 2023-10-30T12:21:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 10 monclient: shutdown 2023-10-30T12:21:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc40a0460 msgr2=0x7f8bc41cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 --2- 172.21.15.50:0/745005852 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc40a0460 0x7f8bc41cab70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000c500 tx=0x7f8bb000c9c0 comp rx=0 tx=0).stop 2023-10-30T12:21:08.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc9110640 1 -- 172.21.15.50:0/745005852 reap_dead start 2023-10-30T12:21:08.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bc9110640 1 -- 172.21.15.50:0/745005852 reap_dead start 2023-10-30T12:21:08.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 shutdown_connections 2023-10-30T12:21:08.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 >> 172.21.15.50:0/745005852 conn(0x7f8bc40a1170 msgr2=0x7f8bc40a2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:08.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 shutdown_connections 2023-10-30T12:21:08.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 -- 172.21.15.50:0/745005852 wait complete. 2023-10-30T12:21:08.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:08.052+0000 7f8bcab73640 1 librados: shutdown 2023-10-30T12:21:08.143 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:21:08.144 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-10-30T12:21:08.144 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2023-10-30T12:21:08.144 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-10-30T12:21:08.144 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:21:08.144 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:21:08.145 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:21:08.162 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:21:08.162 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-10-30T12:21:08.163 INFO:tasks.ceph.mon.a:Sent signal 19 2023-10-30T12:21:08.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-10-30T12:21:23.164 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-10-30T12:21:23.165 INFO:tasks.ceph.mon.a:Sent signal 18 2023-10-30T12:21:23.165 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:21:23.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:21:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.280+0000 7f95521c5640 10 monclient: get_monmap_and_config 2023-10-30T12:21:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.280+0000 7f95521c5640 10 monclient: build_initial_monmap 2023-10-30T12:21:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.280+0000 7f95521c5640 10 monclient: monmap: 2023-10-30T12:21:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:23.281484+0000 2023-10-30T12:21:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:23.281484+0000 2023-10-30T12:21:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:23.283 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:23.284 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:23.284 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:23.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 1 Processor -- start 2023-10-30T12:21:23.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 1 -- start start 2023-10-30T12:21:23.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient: init 2023-10-30T12:21:23.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-10-30T12:21:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient(hunting): picked mon.noname-a con 0x7f954c09f5b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f954c096080 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f954c096ba0 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c098810 con 0x7f954c096ba0 2023-10-30T12:21:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c06aef0 con 0x7f954c096080 2023-10-30T12:21:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c09a1a0 con 0x7f954c09f5b0 2023-10-30T12:21:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f95521c5640 10 monclient(hunting): authenticate will time out at 2128.464739s 2023-10-30T12:21:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954affd640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f954c096080 legacy=0x7f954c096490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:21:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954b7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f954c096ba0 0x7f954c13ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55278/0 (socket says 172.21.15.50:55278) 2023-10-30T12:21:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954bfff640 1 -- 172.21.15.50:0/551396145 learned_addr learned my addr 172.21.15.50:0/551396145 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f954c09f5b0 auth_method 0 2023-10-30T12:21:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.281+0000 7f954b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.282+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:21:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.282+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.282+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.282+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:23.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.482+0000 7f954affd640 1 -- 172.21.15.50:0/551396145 >> v1:172.21.15.118:6789/0 conn(0x7f954c096080 legacy=0x7f954c096490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:21:23.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f954c09f5b0 auth_method 0 2023-10-30T12:21:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:21:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:21:23.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:23.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:23.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.483+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:21:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.883+0000 7f954affd640 1 -- 172.21.15.50:0/551396145 >> v1:172.21.15.118:6789/0 conn(0x7f954c096080 legacy=0x7f954c096490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:21:23.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f954c09f5b0 auth_method 0 2023-10-30T12:21:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:21:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:21:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:23.884+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:21:24.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.684+0000 7f954affd640 1 -- 172.21.15.50:0/551396145 >> v1:172.21.15.118:6789/0 conn(0x7f954c096080 legacy=0x7f954c096490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:21:24.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f954c09f5b0 auth_method 0 2023-10-30T12:21:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:21:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:21:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:24.685+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:21:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 10 monclient(hunting): tick 2023-10-30T12:21:26.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:21:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:21:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:21:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f954c099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f954c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:26.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 >> v1:172.21.15.118:6789/0 conn(0x7f954c096080 legacy=0x7f954c096490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 >> v1:172.21.15.50:6789/0 conn(0x7f954c096ba0 legacy=0x7f954c13ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 10 monclient: _add_conns ranks=[2,4] 2023-10-30T12:21:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.281+0000 7f95497fa640 10 monclient(hunting): picked mon.noname-b con 0x7f95380036d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:26.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f95497fa640 10 monclient(hunting): picked mon.noname-c con 0x7f9538006250 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f95497fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c09a1a0 con 0x7f9538006250 2023-10-30T12:21:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f95497fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9538008db0 con 0x7f95380036d0 2023-10-30T12:21:26.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f95497fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f95380036d0 auth_method 0 2023-10-30T12:21:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:26.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954affd640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 msgr2=0x7f95380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954affd640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:21:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:26.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.282+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954affd640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 msgr2=0x7f95380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954affd640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:21:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:26.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f95380036d0 auth_method 0 2023-10-30T12:21:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:21:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:21:26.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:26.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:26.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.483+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:21:26.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954affd640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 msgr2=0x7f95380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:26.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:26.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954affd640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:21:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f95380036d0 auth_method 0 2023-10-30T12:21:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:21:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:21:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:26.884+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:21:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954affd640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 msgr2=0x7f95380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954affd640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:21:27.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 10 monclient(hunting): get_auth_request con 0x7f95380036d0 auth_method 0 2023-10-30T12:21:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:21:27.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:21:27.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:21:27.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:21:27.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:27.685+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:21:29.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): tick 2023-10-30T12:21:29.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:21:29.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:21:29.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:21:29.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:29.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 msgr2=0x7f95380086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:29.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-10-30T12:21:29.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 1 -- 172.21.15.50:0/551396145 reap_dead start 2023-10-30T12:21:29.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): picked mon.noname-c con 0x7f9538006250 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f953800b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): picked mon.noname-a con 0x7f954c09f5b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): picked mon.noname-b con 0x7f95380036d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:29.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c09a1a0 con 0x7f954c09f5b0 2023-10-30T12:21:29.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9538008db0 con 0x7f9538006250 2023-10-30T12:21:29.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 1 -- 172.21.15.50:0/551396145 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953800b6e0 con 0x7f95380036d0 2023-10-30T12:21:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f95497fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954b7fe640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 msgr2=0x7f95380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954b7fe640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f953800b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): get_auth_request con 0x7f95380036d0 auth_method 0 2023-10-30T12:21:29.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 10 monclient(hunting): get_auth_request con 0x7f954c09f5b0 auth_method 0 2023-10-30T12:21:29.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 10 monclient(hunting): handle_auth_done global_id 254231 payload 274 2023-10-30T12:21:29.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 msgr2=0x7f95380086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:29.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.282+0000 7f954bfff640 1 -- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 msgr2=0x7f953800b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:21:29.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.283+0000 7f954bfff640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f953800b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.283+0000 7f954bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:21:29.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.283+0000 7f954bfff640 1 monclient: found mon.noname-b 2023-10-30T12:21:29.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.283+0000 7f954bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:29.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.283+0000 7f954bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.283+0000 7f954bfff640 1 -- 172.21.15.50:0/551396145 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c098af0 con 0x7f95380036d0 2023-10-30T12:21:29.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.283+0000 7f954bfff640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f954c096460 tx=0x7f954402bc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e85cd226aa7ad16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 1 -- 172.21.15.50:0/551396145 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954402c9c0 con 0x7f95380036d0 2023-10-30T12:21:29.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:21:29.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: dump: 2023-10-30T12:21:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.304 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:21:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:21:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 1 -- 172.21.15.50:0/551396145 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954402cb20 con 0x7f95380036d0 2023-10-30T12:21:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:29.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 1 -- 172.21.15.50:0/551396145 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954402ce90 con 0x7f95380036d0 2023-10-30T12:21:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f95521c5640 5 monclient: authenticate success, global_id 254231 2023-10-30T12:21:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f95521c5640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f95521c5640 4 set_mon_vals no callback set 2023-10-30T12:21:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f954a7fc640 10 monclient: dump: 2023-10-30T12:21:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.308 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.308 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.308 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.308 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.308 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.309 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f95521c5640 10 monclient: shutdown 2023-10-30T12:21:29.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f95521c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:29.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f95521c5640 1 -- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 msgr2=0x7f9538005b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.284+0000 7f95521c5640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f954c096460 tx=0x7f954402bc10 comp rx=0 tx=0).stop 2023-10-30T12:21:29.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 1 -- 172.21.15.50:0/551396145 shutdown_connections 2023-10-30T12:21:29.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 1 --2- 172.21.15.50:0/551396145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95380036d0 0x7f9538005b20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.50:3300/0 conn(0x7f954c09f5b0 0x7f953800b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 1 --2- 172.21.15.50:0/551396145 >> v2:172.21.15.118:3300/0 conn(0x7f9538006250 0x7f95380086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 1 -- 172.21.15.50:0/551396145 >> 172.21.15.50:0/551396145 conn(0x7f954c0a05c0 msgr2=0x7f954c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:29.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 1 -- 172.21.15.50:0/551396145 shutdown_connections 2023-10-30T12:21:29.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 1 -- 172.21.15.50:0/551396145 wait complete. 2023-10-30T12:21:29.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 10 monclient: build_initial_monmap 2023-10-30T12:21:29.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.285+0000 7f95521c5640 10 monclient: monmap: 2023-10-30T12:21:29.311 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:29.311 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:29.311 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:29.312 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:29.312 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:29.312 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.312 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:29.312 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:29.313 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:29.313 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.286+0000 7f95521c5640 1 librados: starting msgr at 2023-10-30T12:21:29.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.286+0000 7f95521c5640 1 librados: starting objecter 2023-10-30T12:21:29.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.286+0000 7f95521c5640 1 Processor -- start 2023-10-30T12:21:29.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.286+0000 7f95521c5640 1 -- start start 2023-10-30T12:21:29.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.286+0000 7f95521c5640 1 librados: setting wanted keys 2023-10-30T12:21:29.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.286+0000 7f95521c5640 1 librados: calling monclient init 2023-10-30T12:21:29.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.286+0000 7f95521c5640 10 monclient: init 2023-10-30T12:21:29.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:21:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95380036d0 0x7f954c1b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): picked mon.noname-b con 0x7f95380036d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9538006250 0x7f954c1cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): picked mon.noname-c con 0x7f9538006250 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:29.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c09f5b0 0x7f954c1cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): picked mon.noname-a con 0x7f954c09f5b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c1cb860 con 0x7f954c09f5b0 2023-10-30T12:21:29.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c1b2500 con 0x7f95380036d0 2023-10-30T12:21:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c1b27e0 con 0x7f9538006250 2023-10-30T12:21:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:29.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f95521c5640 10 monclient(hunting): authenticate will time out at 2134.470309s 2023-10-30T12:21:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954b7fe640 1 -- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95380036d0 msgr2=0x7f954c1b37a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954b7fe640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95380036d0 0x7f954c1b37a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9538006250 0x7f954c1cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c09f5b0 0x7f954c1cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9538006250 0x7f954c1cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45250/0 (socket says 172.21.15.50:45250) 2023-10-30T12:21:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 10 monclient(hunting): get_auth_request con 0x7f954c09f5b0 auth_method 0 2023-10-30T12:21:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 -- 172.21.15.50:0/4245478698 learned_addr learned my addr 172.21.15.50:0/4245478698 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): get_auth_request con 0x7f9538006250 auth_method 0 2023-10-30T12:21:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient(hunting): handle_auth_done global_id 254234 payload 824 2023-10-30T12:21:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 -- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95380036d0 msgr2=0x7f954c1b37a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:21:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95380036d0 0x7f954c1b37a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 -- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c09f5b0 msgr2=0x7f954c1cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c09f5b0 0x7f954c1cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient: _finish_hunting 0 2023-10-30T12:21:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 monclient: found mon.noname-c 2023-10-30T12:21:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954affd640 1 -- 172.21.15.50:0/4245478698 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c098810 con 0x7f9538006250 2023-10-30T12:21:29.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.287+0000 7f954bfff640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c09f5b0 0x7f954c1cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f954affd640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9538006250 0x7f954c1cade0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9540000c00 tx=0x7f954000d5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 client.?.objecter ms_handle_connect 0x7f9538006250 2023-10-30T12:21:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 1 -- 172.21.15.50:0/4245478698 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954000de70 con 0x7f9538006250 2023-10-30T12:21:29.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: dump: 2023-10-30T12:21:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.325 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.326 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.327 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: _finish_auth 0 2023-10-30T12:21:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: _check_auth_tickets 2023-10-30T12:21:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:29.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 1 -- 172.21.15.50:0/4245478698 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9540007e40 con 0x7f9538006250 2023-10-30T12:21:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 5 monclient: authenticate success, global_id 254234 2023-10-30T12:21:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 1 -- 172.21.15.50:0/4245478698 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540012610 con 0x7f9538006250 2023-10-30T12:21:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9533fff640 10 monclient: dump: 2023-10-30T12:21:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.329 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.330 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.331 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 10 monclient: _renew_subs 2023-10-30T12:21:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954c1b2ac0 con 0x7f9538006250 2023-10-30T12:21:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f9549ffb640 4 set_mon_vals no callback set 2023-10-30T12:21:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 10 client.254234.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 10 monclient: _renew_subs 2023-10-30T12:21:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c09bd30 con 0x7f9538006250 2023-10-30T12:21:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.288+0000 7f95521c5640 1 librados: init done 2023-10-30T12:21:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f9533fff640 1 -- 172.21.15.50:0/4245478698 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9540012770 con 0x7f9538006250 2023-10-30T12:21:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f95521c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f95521c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f95521c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f9533fff640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f953803a1d0 0x7f953803c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954c1b1750 con 0x7f9538006250 2023-10-30T12:21:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f9533fff640 1 -- 172.21.15.50:0/4245478698 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(655..655 src has 520..655) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f954004b560 con 0x7f9538006250 2023-10-30T12:21:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f9533fff640 10 client.254234.objecter ms_dispatch 0x7f954c000b90 osd_map(655..655 src has 520..655) v4 2023-10-30T12:21:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f9533fff640 3 client.254234.objecter handle_osd_map got epochs [655,655] > 0 2023-10-30T12:21:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.289+0000 7f9533fff640 3 client.254234.objecter handle_osd_map decoding full epoch 655 2023-10-30T12:21:29.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.290+0000 7f9533fff640 20 client.254234.objecter dump_active .. 0 homeless 2023-10-30T12:21:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.290+0000 7f954b7fe640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f953803a1d0 0x7f953803c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.290+0000 7f954b7fe640 10 monclient: get_auth_request con 0x7f953803a1d0 auth_method 0 2023-10-30T12:21:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.290+0000 7f954b7fe640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f953803a1d0 0x7f953803c690 secure :-1 s=READY pgs=4331 cs=0 l=1 rev1=1 crypto rx=0x7f953c006c80 tx=0x7f953c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.290+0000 7f9533fff640 10 client.254234.objecter ms_handle_connect 0x7f953803a1d0 2023-10-30T12:21:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.295+0000 7f9533fff640 1 -- 172.21.15.50:0/4245478698 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9540016030 con 0x7f9538006250 2023-10-30T12:21:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.295+0000 7f9533fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.295+0000 7f9533fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:29.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.539+0000 7f95521c5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:21:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.539+0000 7f95521c5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.539+0000 7f95521c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.539+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f954c097090 con 0x7f9538006250 2023-10-30T12:21:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.540+0000 7f9533fff640 1 -- 172.21.15.50:0/4245478698 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f954003c080 con 0x7f9538006250 2023-10-30T12:21:29.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.540+0000 7f9533fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:21:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.540+0000 7f9533fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:29.544 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":214,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:21:29.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 10 librados: watch_flush enter 2023-10-30T12:21:29.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 10 librados: watch_flush exit 2023-10-30T12:21:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 20 client.254234.objecter shutdown clearing up homeless session... 2023-10-30T12:21:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 10 client.254234.objecter successfully canceled tick 2023-10-30T12:21:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f953803a1d0 msgr2=0x7f953803c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f953803a1d0 0x7f953803c690 secure :-1 s=READY pgs=4331 cs=0 l=1 rev1=1 crypto rx=0x7f953c006c80 tx=0x7f953c008040 comp rx=0 tx=0).stop 2023-10-30T12:21:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 10 monclient: shutdown 2023-10-30T12:21:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.541+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9538006250 msgr2=0x7f954c1cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9538006250 0x7f954c1cade0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9540000c00 tx=0x7f954000d5c0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 shutdown_connections 2023-10-30T12:21:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f953803a1d0 0x7f953803c690 unknown :-1 s=CLOSED pgs=4331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f954c09f5b0 0x7f954c1cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9538006250 0x7f954c1cade0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 --2- 172.21.15.50:0/4245478698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95380036d0 0x7f954c1b37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 >> 172.21.15.50:0/4245478698 conn(0x7f954c0a05c0 msgr2=0x7f954c1b98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 shutdown_connections 2023-10-30T12:21:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 -- 172.21.15.50:0/4245478698 wait complete. 2023-10-30T12:21:29.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.542+0000 7f95521c5640 1 librados: shutdown 2023-10-30T12:21:29.555 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:21:29.555 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:21:29.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:21:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.661+0000 7f986a046640 10 monclient: get_monmap_and_config 2023-10-30T12:21:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.661+0000 7f986a046640 10 monclient: build_initial_monmap 2023-10-30T12:21:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.661+0000 7f986a046640 10 monclient: monmap: 2023-10-30T12:21:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:29.662154+0000 2023-10-30T12:21:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:29.662154+0000 2023-10-30T12:21:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:29.667 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.661+0000 7f986a046640 1 Processor -- start 2023-10-30T12:21:29.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.661+0000 7f986a046640 1 -- start start 2023-10-30T12:21:29.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.661+0000 7f986a046640 10 monclient: init 2023-10-30T12:21:29.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:29.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-10-30T12:21:29.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f98640a2ae0 0x7f98640a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient(hunting): picked mon.noname-c con 0x7f98640a2ae0 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:29.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98640a2ec0 0x7f98641356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient(hunting): picked mon.noname-b con 0x7f98640a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:29.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98640a32e0 addr v1:172.21.15.118:6789/0 2023-10-30T12:21:29.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98640a4190 con 0x7f98640a32e0 2023-10-30T12:21:29.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98640a2990 con 0x7f98640a2ae0 2023-10-30T12:21:29.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98640a1630 con 0x7f98640a2ec0 2023-10-30T12:21:29.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:29.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f986a046640 10 monclient(hunting): authenticate will time out at 2134.845367s 2023-10-30T12:21:29.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9863fff640 1 -- >> v1:172.21.15.118:6789/0 conn(0x7f98640a32e0 legacy=0x7f9864135c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.118:6789/0 2023-10-30T12:21:29.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98640a2ec0 0x7f98641356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f98637fe640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f98640a2ae0 msgr2=0x7f98640a2450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:29.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f98637fe640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f98640a2ae0 0x7f98640a2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:29.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98640a2ec0 0x7f98641356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45254/0 (socket says 172.21.15.50:45254) 2023-10-30T12:21:29.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 -- 172.21.15.50:0/3741463206 learned_addr learned my addr 172.21.15.50:0/3741463206 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:29.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): get_auth_request con 0x7f98640a2ec0 auth_method 0 2023-10-30T12:21:29.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient(hunting): handle_auth_done global_id 254237 payload 274 2023-10-30T12:21:29.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 -- 172.21.15.50:0/3741463206 >> v2:172.21.15.118:3300/0 conn(0x7f98640a2ae0 msgr2=0x7f98640a2450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:29.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 --2- 172.21.15.50:0/3741463206 >> v2:172.21.15.118:3300/0 conn(0x7f98640a2ae0 0x7f98640a2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 -- 172.21.15.50:0/3741463206 >> v1:172.21.15.118:6789/0 conn(0x7f98640a32e0 legacy=0x7f9864135c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:29.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:21:29.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 monclient: found mon.noname-b 2023-10-30T12:21:29.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:29.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.662+0000 7f9862ffd640 1 -- 172.21.15.50:0/3741463206 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986409a2e0 con 0x7f98640a2ec0 2023-10-30T12:21:29.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f9862ffd640 1 --2- 172.21.15.50:0/3741463206 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98640a2ec0 0x7f98641356e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f984c009870 tx=0x7f984c02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd7e9e0f917fb6c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 1 -- 172.21.15.50:0/3741463206 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c031750 con 0x7f98640a2ec0 2023-10-30T12:21:29.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:21:29.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: dump: 2023-10-30T12:21:29.702 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.702 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.703 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.703 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.703 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.704 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.704 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.704 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.705 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.705 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: _finish_auth 0 2023-10-30T12:21:29.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: _check_auth_tickets 2023-10-30T12:21:29.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:29.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f986a046640 5 monclient: authenticate success, global_id 254237 2023-10-30T12:21:29.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 1 -- 172.21.15.50:0/3741463206 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984c0318b0 con 0x7f98640a2ec0 2023-10-30T12:21:29.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f986a046640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:29.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:29.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f986a046640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:29.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f986a046640 4 set_mon_vals no callback set 2023-10-30T12:21:29.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 1 -- 172.21.15.50:0/3741463206 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c031c40 con 0x7f98640a2ec0 2023-10-30T12:21:29.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:29.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.663+0000 7f98627fc640 10 monclient: dump: 2023-10-30T12:21:29.710 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.711 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.711 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.711 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.712 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.713 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.713 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.713 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 10 monclient: shutdown 2023-10-30T12:21:29.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:29.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 -- 172.21.15.50:0/3741463206 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98640a2ec0 msgr2=0x7f98641356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 --2- 172.21.15.50:0/3741463206 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98640a2ec0 0x7f98641356e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f984c009870 tx=0x7f984c02b7f0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 -- 172.21.15.50:0/3741463206 shutdown_connections 2023-10-30T12:21:29.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 --2- 172.21.15.50:0/3741463206 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98640a2ec0 0x7f98641356e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 --2- 172.21.15.50:0/3741463206 >> v2:172.21.15.118:3300/0 conn(0x7f98640a2ae0 0x7f98640a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 -- 172.21.15.50:0/3741463206 >> 172.21.15.50:0/3741463206 conn(0x7f98640a80b0 msgr2=0x7f98640a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:29.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 -- 172.21.15.50:0/3741463206 shutdown_connections 2023-10-30T12:21:29.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 1 -- 172.21.15.50:0/3741463206 wait complete. 2023-10-30T12:21:29.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 10 monclient: build_initial_monmap 2023-10-30T12:21:29.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.664+0000 7f986a046640 10 monclient: monmap: 2023-10-30T12:21:29.718 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:29.719 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:29.719 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:29.719 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:29.720 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:29.720 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.720 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:29.721 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:29.721 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:29.721 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 librados: starting msgr at 2023-10-30T12:21:29.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 librados: starting objecter 2023-10-30T12:21:29.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 Processor -- start 2023-10-30T12:21:29.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 -- start start 2023-10-30T12:21:29.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 librados: setting wanted keys 2023-10-30T12:21:29.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 librados: calling monclient init 2023-10-30T12:21:29.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient: init 2023-10-30T12:21:29.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:29.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:21:29.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98640a2ae0 0x7f98641ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): picked mon.noname-b con 0x7f98640a2ae0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:29.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a2ec0 0x7f986411e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): picked mon.noname-a con 0x7f98640a2ec0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:29.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98641caed0 0x7f986411ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): picked mon.noname-c con 0x7f98641caed0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:29.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98641210e0 con 0x7f98640a2ec0 2023-10-30T12:21:29.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98640a1790 con 0x7f98640a2ae0 2023-10-30T12:21:29.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98641327c0 con 0x7f98641caed0 2023-10-30T12:21:29.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:29.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f986a046640 10 monclient(hunting): authenticate will time out at 2134.848821s 2023-10-30T12:21:29.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9863fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98641caed0 0x7f986411ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9862ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a2ec0 0x7f986411e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9863fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98641caed0 0x7f986411ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45256/0 (socket says 172.21.15.50:45256) 2023-10-30T12:21:29.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9863fff640 1 -- 172.21.15.50:0/2064278807 learned_addr learned my addr 172.21.15.50:0/2064278807 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:29.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9863fff640 10 monclient(hunting): get_auth_request con 0x7f98641caed0 auth_method 0 2023-10-30T12:21:29.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9863fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9862ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a2ec0 0x7f986411e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34828/0 (socket says 172.21.15.50:34828) 2023-10-30T12:21:29.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9863fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f98637fe640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98640a2ae0 msgr2=0x7f98641ca990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:29.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9862ffd640 10 monclient(hunting): get_auth_request con 0x7f98640a2ec0 auth_method 0 2023-10-30T12:21:29.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9862ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f9862ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.665+0000 7f98637fe640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98640a2ae0 0x7f98641ca990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:29.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 10 monclient(hunting): handle_auth_done global_id 254240 payload 824 2023-10-30T12:21:29.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98640a2ae0 msgr2=0x7f98641ca990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:21:29.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98640a2ae0 0x7f98641ca990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a2ec0 msgr2=0x7f986411e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a2ec0 0x7f986411e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 10 monclient: _finish_hunting 0 2023-10-30T12:21:29.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 1 monclient: found mon.noname-c 2023-10-30T12:21:29.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:29.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 1 -- 172.21.15.50:0/2064278807 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c0094d0 con 0x7f98641caed0 2023-10-30T12:21:29.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9862ffd640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a2ec0 0x7f986411e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:21:29.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9863fff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98641caed0 0x7f986411ed40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f985c00bd60 tx=0x7f985c00be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 client.?.objecter ms_handle_connect 0x7f98641caed0 2023-10-30T12:21:29.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:29.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c015470 con 0x7f98641caed0 2023-10-30T12:21:29.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:21:29.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: dump: 2023-10-30T12:21:29.732 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.733 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.733 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.733 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.733 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.733 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.734 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.734 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: _finish_auth 0 2023-10-30T12:21:29.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:21:29.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:29.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 5 monclient: authenticate success, global_id 254240 2023-10-30T12:21:29.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c0155d0 con 0x7f98641caed0 2023-10-30T12:21:29.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:29.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 10 monclient: _renew_subs 2023-10-30T12:21:29.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c015a00 con 0x7f98641caed0 2023-10-30T12:21:29.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9864121fd0 con 0x7f98641caed0 2023-10-30T12:21:29.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f9861ffb640 4 set_mon_vals no callback set 2023-10-30T12:21:29.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:29.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f985bfff640 10 monclient: dump: 2023-10-30T12:21:29.737 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.737 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.737 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.737 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.737 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.738 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.738 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.738 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.738 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.738 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 10 client.254240.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:29.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 10 monclient: _renew_subs 2023-10-30T12:21:29.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:21:29.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98641204a0 con 0x7f98641caed0 2023-10-30T12:21:29.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.666+0000 7f986a046640 1 librados: init done 2023-10-30T12:21:29.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f985c012070 con 0x7f98641caed0 2023-10-30T12:21:29.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f986a046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:29.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f986a046640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a32e0 0x7f986402b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f986a046640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9864073ff0 con 0x7f98640a32e0 2023-10-30T12:21:29.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f985bfff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98440392a0 0x7f984403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(655..655 src has 520..655) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f985c04aee0 con 0x7f98641caed0 2023-10-30T12:21:29.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f985bfff640 10 client.254240.objecter ms_dispatch 0x7f9864000b90 osd_map(655..655 src has 520..655) v4 2023-10-30T12:21:29.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f985bfff640 3 client.254240.objecter handle_osd_map got epochs [655,655] > 0 2023-10-30T12:21:29.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f985bfff640 3 client.254240.objecter handle_osd_map decoding full epoch 655 2023-10-30T12:21:29.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a32e0 0x7f986402b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient: get_auth_request con 0x7f98640a32e0 auth_method 0 2023-10-30T12:21:29.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f985bfff640 20 client.254240.objecter dump_active .. 0 homeless 2023-10-30T12:21:29.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f9862ffd640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98440392a0 0x7f984403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f9862ffd640 10 monclient: get_auth_request con 0x7f98440392a0 auth_method 0 2023-10-30T12:21:29.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 10 monclient(hunting): handle_auth_done global_id 264186 payload 824 2023-10-30T12:21:29.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.667+0000 7f98637fe640 1 -- 172.21.15.50:0/2064278807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f986409a5c0 con 0x7f98640a32e0 2023-10-30T12:21:29.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 client.254240.objecter ms_handle_connect 0x7f98640a32e0 2023-10-30T12:21:29.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f98637fe640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a32e0 0x7f986402b430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9854003930 tx=0x7f985400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 client.254240.objecter resend_mon_ops 2023-10-30T12:21:29.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854003e10 con 0x7f98640a32e0 2023-10-30T12:21:29.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:29.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 monclient: dump: 2023-10-30T12:21:29.745 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.745 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.745 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.746 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.746 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.746 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.746 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.746 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.746 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.747 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 client.254240.objecter ms_handle_connect 0x7f98440392a0 2023-10-30T12:21:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f9862ffd640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98440392a0 0x7f984403b760 secure :-1 s=READY pgs=4332 cs=0 l=1 rev1=1 crypto rx=0x7f984c035000 tx=0x7f984c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f9854005070 con 0x7f98640a32e0 2023-10-30T12:21:29.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a32e0 msgr2=0x7f986402b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a32e0 0x7f986402b430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9854003930 tx=0x7f985400b8c0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a32e0 msgr2=0x7f986402b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.668+0000 7f985bfff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98640a32e0 0x7f986402b430 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.674+0000 7f986a046640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:29.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.674+0000 7f986a046640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f986402b970 0x7f986402bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.674+0000 7f986a046640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.674+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986406b060 con 0x7f986402b970 2023-10-30T12:21:29.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.674+0000 7f98637fe640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f986402b970 0x7f986402bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient: get_auth_request con 0x7f986402b970 auth_method 0 2023-10-30T12:21:29.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 10 monclient(hunting): handle_auth_done global_id 264189 payload 824 2023-10-30T12:21:29.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 1 -- 172.21.15.50:0/2064278807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9864099b20 con 0x7f986402b970 2023-10-30T12:21:29.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f98637fe640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f986402b970 0x7f986402bd80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9854003930 tx=0x7f9854005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 10 client.254240.objecter ms_handle_connect 0x7f986402b970 2023-10-30T12:21:29.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 10 client.254240.objecter resend_mon_ops 2023-10-30T12:21:29.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854020070 con 0x7f986402b970 2023-10-30T12:21:29.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:29.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 10 monclient: dump: 2023-10-30T12:21:29.752 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.752 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.753 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.753 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.753 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.753 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.753 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.753 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.754 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.754 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4623 (secure 0 0 0) 0x7f9854010610 con 0x7f986402b970 2023-10-30T12:21:29.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:29.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:29.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f986402b970 msgr2=0x7f986402bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f986402b970 0x7f986402bd80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9854003930 tx=0x7f9854005a20 comp rx=0 tx=0).stop 2023-10-30T12:21:29.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f986402b970 msgr2=0x7f986402bd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:29.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f985bfff640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f986402b970 0x7f986402bd80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.756 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:21:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 214, 2023-10-30T12:21:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:21:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.772 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:29.772 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:29.772 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:29.772 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:29.772 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:29.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.773 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:29.773 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:29.773 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:29.773 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:29.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:21:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:21:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:29.786 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:29.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f986a046640 10 librados: watch_flush enter 2023-10-30T12:21:29.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f986a046640 10 librados: watch_flush exit 2023-10-30T12:21:29.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f986a046640 20 client.254240.objecter shutdown clearing up homeless session... 2023-10-30T12:21:29.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.675+0000 7f986a046640 10 client.254240.objecter successfully canceled tick 2023-10-30T12:21:29.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98440392a0 msgr2=0x7f984403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f98440392a0 0x7f984403b760 secure :-1 s=READY pgs=4332 cs=0 l=1 rev1=1 crypto rx=0x7f984c035000 tx=0x7f984c008030 comp rx=0 tx=0).stop 2023-10-30T12:21:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 10 monclient: shutdown 2023-10-30T12:21:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98641caed0 msgr2=0x7f986411ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98641caed0 0x7f986411ed40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f985c00bd60 tx=0x7f985c00be50 comp rx=0 tx=0).stop 2023-10-30T12:21:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f9863fff640 1 -- 172.21.15.50:0/2064278807 reap_dead start 2023-10-30T12:21:29.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 shutdown_connections 2023-10-30T12:21:29.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 --2- 172.21.15.50:0/2064278807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98641caed0 0x7f986411ed40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 >> 172.21.15.50:0/2064278807 conn(0x7f98640a80b0 msgr2=0x7f9864134c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:29.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 shutdown_connections 2023-10-30T12:21:29.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 -- 172.21.15.50:0/2064278807 wait complete. 2023-10-30T12:21:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.676+0000 7f986a046640 1 librados: shutdown 2023-10-30T12:21:29.791 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:21:29.791 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-10-30T12:21:29.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:21:29.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.899+0000 7f8f92bd0640 10 monclient: get_monmap_and_config 2023-10-30T12:21:29.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.899+0000 7f8f92bd0640 10 monclient: build_initial_monmap 2023-10-30T12:21:29.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.899+0000 7f8f92bd0640 10 monclient: monmap: 2023-10-30T12:21:29.901 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:29.902 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.902 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:21:29.900552+0000 2023-10-30T12:21:29.902 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:21:29.900552+0000 2023-10-30T12:21:29.902 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:29.902 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.903 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:21:29.903 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:21:29.903 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:21:29.903 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:21:29.903 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:21:29.904 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 1 Processor -- start 2023-10-30T12:21:29.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 1 -- start start 2023-10-30T12:21:29.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient: init 2023-10-30T12:21:29.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:29.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-10-30T12:21:29.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f8c096540 addr v1:172.21.15.50:6789/0 2023-10-30T12:21:29.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8f8c136960 0x7f8f8c138d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f8f8c136960 addr v2:172.21.15.118:3300/0 2023-10-30T12:21:29.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f8c139290 0x7f8f8c13b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f8f8c139290 addr v2:172.21.15.50:3300/0 2023-10-30T12:21:29.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c09bd30 con 0x7f8f8c096540 2023-10-30T12:21:29.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8c09c400 con 0x7f8f8c139290 2023-10-30T12:21:29.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8c09c120 con 0x7f8f8c136960 2023-10-30T12:21:29.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:29.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f92bd0640 10 monclient(hunting): authenticate will time out at 2135.083709s 2023-10-30T12:21:29.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9116d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f8c139290 0x7f8f8c13b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f83fff640 1 -- >> v2:172.21.15.118:3300/0 conn(0x7f8f8c136960 msgr2=0x7f8f8c138d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:29.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9096c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f8c096540 0x7f8f8c096920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56694/0 (socket says 172.21.15.50:56694) 2023-10-30T12:21:29.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9096c640 1 -- 172.21.15.50:0/3111860425 learned_addr learned my addr 172.21.15.50:0/3111860425 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:21:29.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f83fff640 1 --2- 172.21.15.50:0/3111860425 >> v2:172.21.15.118:3300/0 conn(0x7f8f8c136960 0x7f8f8c138d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:29.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9116d640 10 monclient(hunting): get_auth_request con 0x7f8f8c139290 auth_method 0 2023-10-30T12:21:29.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9116d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9116d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9116d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9116d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.900+0000 7f8f9116d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 10 monclient(hunting): handle_auth_done global_id 264192 payload 274 2023-10-30T12:21:29.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 1 -- 172.21.15.50:0/3111860425 >> v2:172.21.15.118:3300/0 conn(0x7f8f8c136960 msgr2=0x7f8f8c138d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:21:29.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 1 --2- 172.21.15.50:0/3111860425 >> v2:172.21.15.118:3300/0 conn(0x7f8f8c136960 0x7f8f8c138d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 1 -- 172.21.15.50:0/3111860425 >> v1:172.21.15.50:6789/0 conn(0x7f8f8c096540 legacy=0x7f8f8c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 10 monclient: _finish_hunting 0 2023-10-30T12:21:29.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 1 monclient: found mon.noname-a 2023-10-30T12:21:29.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:29.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:29.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f837fe640 1 -- 172.21.15.50:0/3111860425 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315468207 0 0) 0x7f8f8c09bd30 con 0x7f8f8c096540 2023-10-30T12:21:29.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 1 -- 172.21.15.50:0/3111860425 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c097030 con 0x7f8f8c139290 2023-10-30T12:21:29.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f837fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:21:29.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.901+0000 7f8f9116d640 1 --2- 172.21.15.50:0/3111860425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c139290 0x7f8f8c13b680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c074290 tx=0x7f8f8402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8aa5524ead145e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 1 -- 172.21.15.50:0/3111860425 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8402cc00 con 0x7f8f8c139290 2023-10-30T12:21:29.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:29.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: dump: 2023-10-30T12:21:29.921 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.921 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.921 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.921 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.921 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.922 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.922 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.922 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.922 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.922 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: _finish_auth 0 2023-10-30T12:21:29.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: _check_auth_tickets 2023-10-30T12:21:29.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:29.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 1 -- 172.21.15.50:0/3111860425 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8402cd60 con 0x7f8f8c139290 2023-10-30T12:21:29.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 5 monclient: authenticate success, global_id 264192 2023-10-30T12:21:29.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:21:29.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:29.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 1 -- 172.21.15.50:0/3111860425 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84034660 con 0x7f8f8c139290 2023-10-30T12:21:29.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 10 monclient: get_monmap_and_config success 2023-10-30T12:21:29.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 4 set_mon_vals no callback set 2023-10-30T12:21:29.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:29.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f837fe640 10 monclient: dump: 2023-10-30T12:21:29.925 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.925 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.925 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.925 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.926 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.926 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.926 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.926 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.926 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.927 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 10 monclient: shutdown 2023-10-30T12:21:29.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:29.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 -- 172.21.15.50:0/3111860425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c139290 msgr2=0x7f8f8c13b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 --2- 172.21.15.50:0/3111860425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c139290 0x7f8f8c13b680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c074290 tx=0x7f8f8402be50 comp rx=0 tx=0).stop 2023-10-30T12:21:29.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 -- 172.21.15.50:0/3111860425 shutdown_connections 2023-10-30T12:21:29.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 --2- 172.21.15.50:0/3111860425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c139290 0x7f8f8c13b680 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 --2- 172.21.15.50:0/3111860425 >> v2:172.21.15.118:3300/0 conn(0x7f8f8c136960 0x7f8f8c138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 -- 172.21.15.50:0/3111860425 >> 172.21.15.50:0/3111860425 conn(0x7f8f8c0a3980 msgr2=0x7f8f8c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:29.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 -- 172.21.15.50:0/3111860425 shutdown_connections 2023-10-30T12:21:29.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.902+0000 7f8f92bd0640 1 -- 172.21.15.50:0/3111860425 wait complete. 2023-10-30T12:21:29.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 10 monclient: build_initial_monmap 2023-10-30T12:21:29.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 10 monclient: monmap: 2023-10-30T12:21:29.929 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:21:29.929 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:21:29.929 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:21:29.930 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:21:29.930 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:21:29.930 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.930 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:21:29.931 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:21:29.931 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:21:29.931 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 1 librados: starting msgr at 2023-10-30T12:21:29.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 1 librados: starting objecter 2023-10-30T12:21:29.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 1 Processor -- start 2023-10-30T12:21:29.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 1 -- start start 2023-10-30T12:21:29.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 1 librados: setting wanted keys 2023-10-30T12:21:29.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 1 librados: calling monclient init 2023-10-30T12:21:29.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.903+0000 7f8f92bd0640 10 monclient: init 2023-10-30T12:21:29.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient: _reopen_session rank -1 2023-10-30T12:21:29.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:21:29.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c096540 0x7f8f8c1c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f8f8c096540 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:21:29.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c136960 0x7f8f8c093770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f8f8c136960 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:21:29.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f8c139290 0x7f8f8c1c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f8f8c139290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:21:29.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c08d280 con 0x7f8f8c136960 2023-10-30T12:21:29.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c1c5000 con 0x7f8f8c139290 2023-10-30T12:21:29.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c09a0b0 con 0x7f8f8c096540 2023-10-30T12:21:29.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): _renew_subs 2023-10-30T12:21:29.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient(hunting): authenticate will time out at 2135.087194s 2023-10-30T12:21:29.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c136960 0x7f8f8c093770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c096540 0x7f8f8c1c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c096540 0x7f8f8c1c4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45264/0 (socket says 172.21.15.50:45264) 2023-10-30T12:21:29.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c136960 0x7f8f8c093770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34876/0 (socket says 172.21.15.50:34876) 2023-10-30T12:21:29.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 1 -- 172.21.15.50:0/1078133019 learned_addr learned my addr 172.21.15.50:0/1078133019 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:21:29.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 10 monclient(hunting): get_auth_request con 0x7f8f8c096540 auth_method 0 2023-10-30T12:21:29.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): get_auth_request con 0x7f8f8c136960 auth_method 0 2023-10-30T12:21:29.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9116d640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f8c139290 msgr2=0x7f8f8c1c7a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:3300/0 2023-10-30T12:21:29.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9116d640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f8c139290 0x7f8f8c1c7a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:21:29.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient(hunting): handle_auth_done global_id 264198 payload 824 2023-10-30T12:21:29.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c096540 msgr2=0x7f8f8c1c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c096540 0x7f8f8c1c4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f8c139290 msgr2=0x7f8f8c1c7a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:21:29.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f8c139290 0x7f8f8c1c7a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient: _finish_hunting 0 2023-10-30T12:21:29.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 monclient: found mon.noname-a 2023-10-30T12:21:29.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:21:29.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:21:29.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 -- 172.21.15.50:0/1078133019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f84009a70 con 0x7f8f8c136960 2023-10-30T12:21:29.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f83fff640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c136960 0x7f8f8c093770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009fd0 tx=0x7f8f7c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 10 client.?.objecter ms_handle_connect 0x7f8f8c136960 2023-10-30T12:21:29.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:21:29.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c003db0 con 0x7f8f8c136960 2023-10-30T12:21:29.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:21:29.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 10 monclient: dump: 2023-10-30T12:21:29.943 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.943 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.943 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.943 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.944 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.944 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.944 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.944 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.944 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.944 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 10 monclient: _finish_auth 0 2023-10-30T12:21:29.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 10 monclient: _check_auth_tickets 2023-10-30T12:21:29.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:21:29.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c005070 con 0x7f8f8c136960 2023-10-30T12:21:29.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 5 monclient: authenticate success, global_id 264198 2023-10-30T12:21:29.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient: _renew_subs 2023-10-30T12:21:29.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:29.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.904+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c08ba60 con 0x7f8f8c136960 2023-10-30T12:21:29.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f817fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:21:29.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c005420 con 0x7f8f8c136960 2023-10-30T12:21:29.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:21:29.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f817fa640 10 monclient: dump: 2023-10-30T12:21:29.947 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.947 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.948 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.948 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.948 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.948 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.948 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.949 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.949 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.949 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 10 client.264198.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:21:29.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f82ffd640 4 set_mon_vals no callback set 2023-10-30T12:21:29.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 10 monclient: _renew_subs 2023-10-30T12:21:29.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:21:29.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8c08bda0 con 0x7f8f8c136960 2023-10-30T12:21:29.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 1 librados: init done 2023-10-30T12:21:29.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:29.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c071210 0x7f8f8c02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8f7c005580 con 0x7f8f8c136960 2023-10-30T12:21:29.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c073ff0 con 0x7f8f8c071210 2023-10-30T12:21:29.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f9096c640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c071210 0x7f8f8c02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f9096c640 10 monclient: get_auth_request con 0x7f8f8c071210 auth_method 0 2023-10-30T12:21:29.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f9096c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f9096c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.905+0000 7f8f9096c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f780392a0 0x7f8f7803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(655..655 src has 520..655) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f8f7c04a080 con 0x7f8f8c136960 2023-10-30T12:21:29.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 client.264198.objecter ms_dispatch 0x7f8f8c000b90 osd_map(655..655 src has 520..655) v4 2023-10-30T12:21:29.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 3 client.264198.objecter handle_osd_map got epochs [655,655] > 0 2023-10-30T12:21:29.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 3 client.264198.objecter handle_osd_map decoding full epoch 655 2023-10-30T12:21:29.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 20 client.264198.objecter dump_active .. 0 homeless 2023-10-30T12:21:29.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9096c640 10 monclient(hunting): handle_auth_done global_id 254246 payload 824 2023-10-30T12:21:29.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9096c640 1 -- 172.21.15.50:0/1078133019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f8c09ba70 con 0x7f8f8c071210 2023-10-30T12:21:29.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 client.264198.objecter ms_handle_connect 0x7f8f8c071210 2023-10-30T12:21:29.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 client.264198.objecter resend_mon_ops 2023-10-30T12:21:29.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9096c640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c071210 0x7f8f8c02b430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8f88001360 tx=0x7f8f8800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9116d640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f780392a0 0x7f8f7803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880070d0 con 0x7f8f8c071210 2023-10-30T12:21:29.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 monclient: dump: 2023-10-30T12:21:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.957 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.957 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.957 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.957 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.957 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f9116d640 10 monclient: get_auth_request con 0x7f8f780392a0 auth_method 0 2023-10-30T12:21:29.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f8f88007230 con 0x7f8f8c071210 2023-10-30T12:21:29.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:21:29.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:21:29.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c071210 msgr2=0x7f8f8c02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c071210 0x7f8f8c02b430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8f88001360 tx=0x7f8f8800fc00 comp rx=0 tx=0).stop 2023-10-30T12:21:29.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c071210 msgr2=0x7f8f8c02b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:29.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.906+0000 7f8f817fa640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c071210 0x7f8f8c02b430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:21:29.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.907+0000 7f8f817fa640 10 client.264198.objecter ms_handle_connect 0x7f8f780392a0 2023-10-30T12:21:29.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.907+0000 7f8f9116d640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f780392a0 0x7f8f7803b760 secure :-1 s=READY pgs=4333 cs=0 l=1 rev1=1 crypto rx=0x7f8f84036000 tx=0x7f8f84030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f92bd0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:21:29.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f92bd0640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c02b970 0x7f8f8c02bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:21:29.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f92bd0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:21:29.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c08ca60 con 0x7f8f8c02b970 2023-10-30T12:21:29.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c02b970 0x7f8f8c02bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:21:29.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient: get_auth_request con 0x7f8f8c02b970 auth_method 0 2023-10-30T12:21:29.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:21:29.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:21:29.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:21:29.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:21:29.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:21:29.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:21:29.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 10 monclient(hunting): handle_auth_done global_id 254249 payload 824 2023-10-30T12:21:29.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 1 -- 172.21.15.50:0/1078133019 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f8c04e160 con 0x7f8f8c02b970 2023-10-30T12:21:29.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f9096c640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c02b970 0x7f8f8c02bd80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8f88001360 tx=0x7f8f88001270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:21:29.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 10 client.264198.objecter ms_handle_connect 0x7f8f8c02b970 2023-10-30T12:21:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 10 client.264198.objecter resend_mon_ops 2023-10-30T12:21:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88024070 con 0x7f8f8c02b970 2023-10-30T12:21:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:21:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:21:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 10 monclient: dump: 2023-10-30T12:21:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:21:29.966 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:21:29.966 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.966 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:21:29.966 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:21:29.967 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:21:29.967 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:21:29.967 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:21:29.967 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:21:29.967 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:21:29.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4791 (secure 0 0 0) 0x7f8f8800dd40 con 0x7f8f8c02b970 2023-10-30T12:21:29.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:21:29.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:21:29.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c02b970 msgr2=0x7f8f8c02bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:29.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c02b970 0x7f8f8c02bd80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8f88001360 tx=0x7f8f88001270 comp rx=0 tx=0).stop 2023-10-30T12:21:29.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c02b970 msgr2=0x7f8f8c02bd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:21:29.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.913+0000 7f8f817fa640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f8c02b970 0x7f8f8c02bd80 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8f88001360 tx=0x7f8f88001270 comp rx=0 tx=0).stop 2023-10-30T12:21:29.969 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:21:29.969 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:29.970 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:29.970 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:21:29.970 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 214, 2023-10-30T12:21:29.970 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:21:29.970 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:21:29.971 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:21:29.971 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.971 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 1, 2023-10-30T12:21:29.971 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:29.971 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:21:29.972 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:21:29.972 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:29.972 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:29.972 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:29.972 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:29.973 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:29.973 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:29.973 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:29.973 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:29.973 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:29.974 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:29.974 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.974 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:21:29.974 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:21:29.974 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:29.975 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:29.975 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:29.975 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:29.975 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:29.975 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:29.976 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:29.976 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:29.976 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:29.976 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:29.976 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.977 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:21:29.977 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:21:29.977 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:21:29.977 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:21:29.978 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:21:29.978 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:21:29.978 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:29.978 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:21:29.978 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:21:29.979 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:21:29.979 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:21:29.979 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:21:29.979 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:21:29.979 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:21:29.980 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:21:29.980 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:21:29.980 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:21:29.980 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:21:29.980 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:21:29.981 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:21:29.981 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:21:29.981 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:21:29.981 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:21:29.981 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:21:29.981 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:21:29.982 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:21:29.982 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:21:29.982 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:29.982 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:21:29.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.983 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:21:29.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.983 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:21:29.983 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:21:29.984 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:29.984 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:29.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.984 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:29.984 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:21:29.984 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.985 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:29.985 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:21:29.985 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.986 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.986 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:21:29.986 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:21:29.987 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:29.987 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:29.987 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:29.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.988 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:21:29.988 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:21:29.988 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:29.989 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:29.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.989 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:29.989 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:21:29.989 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.990 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:29.991 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:21:29.991 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.991 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.991 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.992 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:21:29.992 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:21:29.992 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:29.992 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:29.992 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:29.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.993 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:21:29.993 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:21:29.994 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:21:29.994 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:21:29.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.994 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:21:29.994 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:21:29.995 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.995 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:21:29.995 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:21:29.996 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:21:29.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.996 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.996 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:21:29.997 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:21:29.997 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:21:29.997 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:21:29.997 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:21:29.997 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:29.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:29.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:29.998 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:21:29.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:21:29.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:29.999 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:29.999 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:29.999 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:29.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:30.000 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:30.000 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:21:30.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:30.000 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:30.000 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:30.001 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:21:30.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:30.001 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:30.001 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:21:30.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:30.002 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:30.002 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:30.002 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:21:30.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:30.002 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:21:30.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:21:30.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:21:30.003 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:21:30.003 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:21:30.003 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:21:30.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:21:30.004 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:21:30.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:21:30.004 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:21:30.004 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:21:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 10 librados: watch_flush enter 2023-10-30T12:21:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 10 librados: watch_flush exit 2023-10-30T12:21:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 20 client.264198.objecter shutdown clearing up homeless session... 2023-10-30T12:21:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 10 client.264198.objecter successfully canceled tick 2023-10-30T12:21:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f780392a0 msgr2=0x7f8f7803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f8f780392a0 0x7f8f7803b760 secure :-1 s=READY pgs=4333 cs=0 l=1 rev1=1 crypto rx=0x7f8f84036000 tx=0x7f8f84030000 comp rx=0 tx=0).stop 2023-10-30T12:21:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 10 monclient: shutdown 2023-10-30T12:21:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:21:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c136960 msgr2=0x7f8f8c093770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:21:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 --2- 172.21.15.50:0/1078133019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f8c136960 0x7f8f8c093770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009fd0 tx=0x7f8f7c00b8e0 comp rx=0 tx=0).stop 2023-10-30T12:21:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f9116d640 1 -- 172.21.15.50:0/1078133019 reap_dead start 2023-10-30T12:21:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f9116d640 1 -- 172.21.15.50:0/1078133019 reap_dead start 2023-10-30T12:21:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 shutdown_connections 2023-10-30T12:21:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 >> 172.21.15.50:0/1078133019 conn(0x7f8f8c0a3980 msgr2=0x7f8f8c0a1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:21:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 shutdown_connections 2023-10-30T12:21:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 -- 172.21.15.50:0/1078133019 wait complete. 2023-10-30T12:21:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:21:29.914+0000 7f8f92bd0640 1 librados: shutdown 2023-10-30T12:21:30.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:21:30.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:21:50.011 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-10-30T12:21:50.012 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-10-30T12:21:50.012 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:21:50.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:21:50.016 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:21:50.016 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-10-30T12:21:50.017 INFO:tasks.ceph.mon.a:Sent signal 19 2023-10-30T12:21:50.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-10-30T12:22:05.018 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-10-30T12:22:05.018 INFO:tasks.ceph.mon.a:Sent signal 18 2023-10-30T12:22:05.019 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:22:05.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:22:05.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.139+0000 7f6548597640 10 monclient: get_monmap_and_config 2023-10-30T12:22:05.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.139+0000 7f6548597640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.139+0000 7f6548597640 10 monclient: monmap: 2023-10-30T12:22:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:05.140168+0000 2023-10-30T12:22:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:05.140168+0000 2023-10-30T12:22:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:05.144 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.139+0000 7f6548597640 1 Processor -- start 2023-10-30T12:22:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.139+0000 7f6548597640 1 -- start start 2023-10-30T12:22:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.139+0000 7f6548597640 10 monclient: init 2023-10-30T12:22:05.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-10-30T12:22:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f65400a4b60 0x7f65401326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient(hunting): picked mon.noname-c con 0x7f65400a4b60 addr v2:172.21.15.118:3300/0 2023-10-30T12:22:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6540099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6540132be0 0x7f65400a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient(hunting): picked mon.noname-a con 0x7f6540132be0 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400a4f40 con 0x7f6540099a40 2023-10-30T12:22:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65400a3d70 con 0x7f6540132be0 2023-10-30T12:22:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65400a4150 con 0x7f65400a4b60 2023-10-30T12:22:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6548597640 10 monclient(hunting): authenticate will time out at 2170.323441s 2023-10-30T12:22:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6540132be0 0x7f65400a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6545b32640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6540099a40 0x7f65400a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46012/0 (socket says 172.21.15.50:46012) 2023-10-30T12:22:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6545b32640 1 -- 172.21.15.50:0/797816650 learned_addr learned my addr 172.21.15.50:0/797816650 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:22:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): get_auth_request con 0x7f6540132be0 auth_method 0 2023-10-30T12:22:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546333640 1 --2- 172.21.15.50:0/797816650 >> v2:172.21.15.118:3300/0 conn(0x7f65400a4b60 0x7f65401326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546333640 10 monclient(hunting): get_auth_request con 0x7f65400a4b60 auth_method 0 2023-10-30T12:22:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546333640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546333640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient(hunting): handle_auth_done global_id 264201 payload 274 2023-10-30T12:22:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 1 -- 172.21.15.50:0/797816650 >> v2:172.21.15.118:3300/0 conn(0x7f65400a4b60 msgr2=0x7f65401326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 1 --2- 172.21.15.50:0/797816650 >> v2:172.21.15.118:3300/0 conn(0x7f65400a4b60 0x7f65401326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 1 -- 172.21.15.50:0/797816650 >> v1:172.21.15.50:6789/0 conn(0x7f6540099a40 legacy=0x7f65400a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 1 monclient: found mon.noname-a 2023-10-30T12:22:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6545331640 1 -- 172.21.15.50:0/797816650 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797797841 0 0) 0x7f65400a4f40 con 0x7f6540099a40 2023-10-30T12:22:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546b34640 1 -- 172.21.15.50:0/797816650 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654009a440 con 0x7f6540132be0 2023-10-30T12:22:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6546333640 1 --2- 172.21.15.50:0/797816650 >> v2:172.21.15.118:3300/0 conn(0x7f65400a4b60 0x7f65401326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:22:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.140+0000 7f6545331640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:22:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6546b34640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 0x7f65400a2a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65400a3820 tx=0x7f653802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b833fb3ed89aa759 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6546b34640 1 -- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 msgr2=0x7f65400a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2023-10-30T12:22:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6546b34640 1 -- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 msgr2=0x7f65400a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:22:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6546b34640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 0x7f65400a2a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65400a3820 tx=0x7f653802bdb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6546b34640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 0x7f65400a2a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65400a3820 tx=0x7f653802bdb0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient: ms_handle_reset current mon [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 1 -- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 msgr2=0x7f65400a2a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 0x7f65400a2a60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient: _add_conns ranks=[1,2] 2023-10-30T12:22:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f652c0034c0 addr v1:172.21.15.118:6789/0 2023-10-30T12:22:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f652c007360 0x7f652c009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient(hunting): picked mon.noname-b con 0x7f652c007360 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 1 -- 172.21.15.50:0/797816650 --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400a4f40 con 0x7f652c0034c0 2023-10-30T12:22:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 1 -- 172.21.15.50:0/797816650 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f652c00b0f0 con 0x7f652c007360 2023-10-30T12:22:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.141+0000 7f6545331640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f652c007360 0x7f652c009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): get_auth_request con 0x7f652c007360 auth_method 0 2023-10-30T12:22:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:22:05.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient(hunting): handle_auth_done global_id 264201 payload 293 2023-10-30T12:22:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 1 -- 172.21.15.50:0/797816650 >> v1:172.21.15.118:6789/0 conn(0x7f652c0034c0 legacy=0x7f652c005940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 1 monclient: found mon.noname-b 2023-10-30T12:22:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 1 -- 172.21.15.50:0/797816650 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c00b320 con 0x7f652c007360 2023-10-30T12:22:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545b32640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f652c007360 0x7f652c009820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6540070610 tx=0x7f6538009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af0227ee31111580 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 1 -- 172.21.15.50:0/797816650 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65300014d0 con 0x7f652c007360 2023-10-30T12:22:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:22:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: dump: 2023-10-30T12:22:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.169 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.169 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.169 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.169 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.169 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.169 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6548597640 5 monclient: authenticate success, global_id 264201 2023-10-30T12:22:05.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 1 -- 172.21.15.50:0/797816650 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653000cb00 con 0x7f652c007360 2023-10-30T12:22:05.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6548597640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:05.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6548597640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:05.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6548597640 4 set_mon_vals no callback set 2023-10-30T12:22:05.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 1 -- 172.21.15.50:0/797816650 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65300058d0 con 0x7f652c007360 2023-10-30T12:22:05.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:05.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.142+0000 7f6545331640 10 monclient: dump: 2023-10-30T12:22:05.172 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.173 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.174 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.174 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.174 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 10 monclient: shutdown 2023-10-30T12:22:05.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:05.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 1 -- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f652c007360 msgr2=0x7f652c009820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f652c007360 0x7f652c009820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6540070610 tx=0x7f6538009750 comp rx=0 tx=0).stop 2023-10-30T12:22:05.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6546b34640 1 -- 172.21.15.50:0/797816650 reap_dead start 2023-10-30T12:22:05.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 1 -- 172.21.15.50:0/797816650 shutdown_connections 2023-10-30T12:22:05.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 1 --2- 172.21.15.50:0/797816650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6540132be0 0x7f65400a2a60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 1 -- 172.21.15.50:0/797816650 >> 172.21.15.50:0/797816650 conn(0x7f65400a8060 msgr2=0x7f65400a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:05.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 1 -- 172.21.15.50:0/797816650 shutdown_connections 2023-10-30T12:22:05.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 1 -- 172.21.15.50:0/797816650 wait complete. 2023-10-30T12:22:05.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.143+0000 7f6548597640 10 monclient: monmap: 2023-10-30T12:22:05.176 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.177 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:05.177 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:05.177 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:05.177 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.177 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:05.178 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 librados: starting msgr at 2023-10-30T12:22:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 librados: starting objecter 2023-10-30T12:22:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 Processor -- start 2023-10-30T12:22:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 -- start start 2023-10-30T12:22:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 librados: setting wanted keys 2023-10-30T12:22:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 librados: calling monclient init 2023-10-30T12:22:05.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient: init 2023-10-30T12:22:05.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:22:05.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6540132be0 0x7f65401ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): picked mon.noname-c con 0x7f6540132be0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f652c007360 0x7f6540131710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): picked mon.noname-b con 0x7f652c007360 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:05.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65401ca6a0 0x7f6540131c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): picked mon.noname-a con 0x7f65401ca6a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:05.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400a1120 con 0x7f65401ca6a0 2023-10-30T12:22:05.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400a18d0 con 0x7f652c007360 2023-10-30T12:22:05.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654011e6f0 con 0x7f6540132be0 2023-10-30T12:22:05.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6548597640 10 monclient(hunting): authenticate will time out at 2170.327695s 2023-10-30T12:22:05.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6540132be0 0x7f65401ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6540132be0 0x7f65401ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55100/0 (socket says 172.21.15.50:55100) 2023-10-30T12:22:05.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 1 -- 172.21.15.50:0/161843307 learned_addr learned my addr 172.21.15.50:0/161843307 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:05.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 10 monclient(hunting): get_auth_request con 0x7f6540132be0 auth_method 0 2023-10-30T12:22:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65401ca6a0 0x7f6540131c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 10 monclient(hunting): get_auth_request con 0x7f65401ca6a0 auth_method 0 2023-10-30T12:22:05.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6545b32640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f652c007360 0x7f6540131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6546b34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6545b32640 10 monclient(hunting): get_auth_request con 0x7f652c007360 auth_method 0 2023-10-30T12:22:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6545b32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6545b32640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.144+0000 7f6545b32640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 10 monclient(hunting): handle_auth_done global_id 254252 payload 824 2023-10-30T12:22:05.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 1 -- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f652c007360 msgr2=0x7f6540131710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f652c007360 0x7f6540131710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 1 -- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65401ca6a0 msgr2=0x7f6540131c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65401ca6a0 0x7f6540131c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 1 monclient: found mon.noname-c 2023-10-30T12:22:05.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 1 -- 172.21.15.50:0/161843307 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654011f860 con 0x7f6540132be0 2023-10-30T12:22:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6546333640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6540132be0 0x7f65401ca160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f653c00e7f0 tx=0x7f653c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 client.?.objecter ms_handle_connect 0x7f6540132be0 2023-10-30T12:22:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 1 -- 172.21.15.50:0/161843307 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c00c8d0 con 0x7f6540132be0 2023-10-30T12:22:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:22:05.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: dump: 2023-10-30T12:22:05.190 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.190 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.190 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.191 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.191 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.191 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.191 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.191 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.191 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.192 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 1 -- 172.21.15.50:0/161843307 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c00ca50 con 0x7f6540132be0 2023-10-30T12:22:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 5 monclient: authenticate success, global_id 254252 2023-10-30T12:22:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 10 monclient: _renew_subs 2023-10-30T12:22:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 1 -- 172.21.15.50:0/161843307 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c00ce00 con 0x7f6540132be0 2023-10-30T12:22:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654011fb40 con 0x7f6540132be0 2023-10-30T12:22:05.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:05.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6544b30640 4 set_mon_vals no callback set 2023-10-30T12:22:05.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6536ffd640 10 monclient: dump: 2023-10-30T12:22:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.196 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 10 client.254252.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 10 monclient: _renew_subs 2023-10-30T12:22:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654011ed70 con 0x7f6540132be0 2023-10-30T12:22:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.145+0000 7f6548597640 1 librados: init done 2023-10-30T12:22:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6536ffd640 1 -- 172.21.15.50:0/161843307 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f653c013410 con 0x7f6540132be0 2023-10-30T12:22:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6548597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6548597640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6548597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6536ffd640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65240392f0 0x7f652403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65400675d0 con 0x7f6540132be0 2023-10-30T12:22:05.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6536ffd640 1 -- 172.21.15.50:0/161843307 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(666..666 src has 520..666) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f653c04aa30 con 0x7f6540132be0 2023-10-30T12:22:05.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6536ffd640 10 client.254252.objecter ms_dispatch 0x7f6540000b90 osd_map(666..666 src has 520..666) v4 2023-10-30T12:22:05.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6536ffd640 3 client.254252.objecter handle_osd_map got epochs [666,666] > 0 2023-10-30T12:22:05.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6536ffd640 3 client.254252.objecter handle_osd_map decoding full epoch 666 2023-10-30T12:22:05.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6536ffd640 20 client.254252.objecter dump_active .. 0 homeless 2023-10-30T12:22:05.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6545b32640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65240392f0 0x7f652403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.146+0000 7f6545b32640 10 monclient: get_auth_request con 0x7f65240392f0 auth_method 0 2023-10-30T12:22:05.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.147+0000 7f6536ffd640 10 client.254252.objecter ms_handle_connect 0x7f65240392f0 2023-10-30T12:22:05.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.147+0000 7f6545b32640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65240392f0 0x7f652403b7b0 secure :-1 s=READY pgs=4335 cs=0 l=1 rev1=1 crypto rx=0x7f653000c5c0 tx=0x7f653000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.151+0000 7f6536ffd640 1 -- 172.21.15.50:0/161843307 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f653c01b9a0 con 0x7f6540132be0 2023-10-30T12:22:05.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.151+0000 7f6536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.151+0000 7f6536ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.398+0000 7f6548597640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:22:05.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.398+0000 7f6548597640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:22:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.398+0000 7f6548597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.398+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f654011ed70 con 0x7f6540132be0 2023-10-30T12:22:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.399+0000 7f6536ffd640 1 -- 172.21.15.50:0/161843307 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f654011ed70 con 0x7f6540132be0 2023-10-30T12:22:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.399+0000 7f6536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:22:05.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.399+0000 7f6536ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:05.402 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:22:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 10 librados: watch_flush enter 2023-10-30T12:22:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 10 librados: watch_flush exit 2023-10-30T12:22:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 20 client.254252.objecter shutdown clearing up homeless session... 2023-10-30T12:22:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 10 client.254252.objecter successfully canceled tick 2023-10-30T12:22:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65240392f0 msgr2=0x7f652403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65240392f0 0x7f652403b7b0 secure :-1 s=READY pgs=4335 cs=0 l=1 rev1=1 crypto rx=0x7f653000c5c0 tx=0x7f653000a000 comp rx=0 tx=0).stop 2023-10-30T12:22:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 10 monclient: shutdown 2023-10-30T12:22:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6540132be0 msgr2=0x7f65401ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6540132be0 0x7f65401ca160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f653c00e7f0 tx=0x7f653c00ecb0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 shutdown_connections 2023-10-30T12:22:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f65240392f0 0x7f652403b7b0 unknown :-1 s=CLOSED pgs=4335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65401ca6a0 0x7f6540131c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f652c007360 0x7f6540131710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 --2- 172.21.15.50:0/161843307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6540132be0 0x7f65401ca160 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 >> 172.21.15.50:0/161843307 conn(0x7f65400a8060 msgr2=0x7f65400a5750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 shutdown_connections 2023-10-30T12:22:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 -- 172.21.15.50:0/161843307 wait complete. 2023-10-30T12:22:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.401+0000 7f6548597640 1 librados: shutdown 2023-10-30T12:22:05.413 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:22:05.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:22:05.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:22:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.518+0000 7f1cc5939640 10 monclient: get_monmap_and_config 2023-10-30T12:22:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.518+0000 7f1cc5939640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.518+0000 7f1cc5939640 10 monclient: monmap: 2023-10-30T12:22:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:05.519676+0000 2023-10-30T12:22:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:05.519676+0000 2023-10-30T12:22:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.521 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:05.522 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:05.523 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 1 Processor -- start 2023-10-30T12:22:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 1 -- start start 2023-10-30T12:22:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient: init 2023-10-30T12:22:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-10-30T12:22:05.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006afc0 0x7f1cc00a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc006afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cc006b640 addr v1:172.21.15.118:6789/0 2023-10-30T12:22:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1cc013b070 0x7f1cc013d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc013b070 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc006bde0 con 0x7f1cc006b640 2023-10-30T12:22:05.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc006b3a0 con 0x7f1cc013b070 2023-10-30T12:22:05.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc009e540 con 0x7f1cc006afc0 2023-10-30T12:22:05.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.519+0000 7f1cc5939640 10 monclient(hunting): authenticate will time out at 2170.702938s 2023-10-30T12:22:05.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006afc0 0x7f1cc00a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006afc0 0x7f1cc00a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55102/0 (socket says 172.21.15.50:55102) 2023-10-30T12:22:05.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1cc013b070 0x7f1cc013d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 1 -- 172.21.15.50:0/770105127 learned_addr learned my addr 172.21.15.50:0/770105127 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:05.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 10 monclient(hunting): get_auth_request con 0x7f1cc006afc0 auth_method 0 2023-10-30T12:22:05.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): get_auth_request con 0x7f1cc013b070 auth_method 0 2023-10-30T12:22:05.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient(hunting): handle_auth_done global_id 264210 payload 274 2023-10-30T12:22:05.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 1 -- 172.21.15.50:0/770105127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006afc0 msgr2=0x7f1cc00a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:05.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 1 --2- 172.21.15.50:0/770105127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006afc0 0x7f1cc00a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 1 -- 172.21.15.50:0/770105127 >> v1:172.21.15.118:6789/0 conn(0x7f1cc006b640 legacy=0x7f1cc00a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 1 monclient: found mon.noname-a 2023-10-30T12:22:05.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:05.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.520+0000 7f1cbf7fe640 1 -- 172.21.15.50:0/770105127 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc009fa10 con 0x7f1cc013b070 2023-10-30T12:22:05.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbf7fe640 1 --2- 172.21.15.50:0/770105127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc013b070 0x7f1cc013d460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009d90 tx=0x7f1cb802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2afb0ab0e3356e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 1 -- 172.21.15.50:0/770105127 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb802cb30 con 0x7f1cc013b070 2023-10-30T12:22:05.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:05.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: dump: 2023-10-30T12:22:05.541 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.541 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.541 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.541 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.541 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.542 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.542 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.542 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.542 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.542 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 5 monclient: authenticate success, global_id 264210 2023-10-30T12:22:05.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 1 -- 172.21.15.50:0/770105127 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb802cc90 con 0x7f1cc013b070 2023-10-30T12:22:05.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:05.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:05.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 4 set_mon_vals no callback set 2023-10-30T12:22:05.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 1 -- 172.21.15.50:0/770105127 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8034440 con 0x7f1cc013b070 2023-10-30T12:22:05.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:05.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cbe7fc640 10 monclient: dump: 2023-10-30T12:22:05.545 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.545 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.545 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.545 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.546 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.546 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.546 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.546 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.546 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.547 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 10 monclient: shutdown 2023-10-30T12:22:05.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:05.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 -- 172.21.15.50:0/770105127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc013b070 msgr2=0x7f1cc013d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 --2- 172.21.15.50:0/770105127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc013b070 0x7f1cc013d460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009d90 tx=0x7f1cb802bd80 comp rx=0 tx=0).stop 2023-10-30T12:22:05.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 -- 172.21.15.50:0/770105127 shutdown_connections 2023-10-30T12:22:05.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 --2- 172.21.15.50:0/770105127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc013b070 0x7f1cc013d460 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 --2- 172.21.15.50:0/770105127 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006afc0 0x7f1cc00a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 -- 172.21.15.50:0/770105127 >> 172.21.15.50:0/770105127 conn(0x7f1cc00a6200 msgr2=0x7f1cc00a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:05.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 -- 172.21.15.50:0/770105127 shutdown_connections 2023-10-30T12:22:05.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.521+0000 7f1cc5939640 1 -- 172.21.15.50:0/770105127 wait complete. 2023-10-30T12:22:05.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 10 monclient: monmap: 2023-10-30T12:22:05.549 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.549 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:05.549 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:05.549 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:05.550 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.550 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.550 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:05.550 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:05.550 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:05.551 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 1 librados: starting msgr at 2023-10-30T12:22:05.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 1 librados: starting objecter 2023-10-30T12:22:05.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 1 Processor -- start 2023-10-30T12:22:05.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 1 -- start start 2023-10-30T12:22:05.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 1 librados: setting wanted keys 2023-10-30T12:22:05.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 1 librados: calling monclient init 2023-10-30T12:22:05.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.522+0000 7f1cc5939640 10 monclient: init 2023-10-30T12:22:05.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:22:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc006afc0 0x7f1cc01ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc006afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006b640 0x7f1cc01ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc006b640 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc013b070 0x7f1cc011c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc013b070 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:05.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0120250 con 0x7f1cc006afc0 2023-10-30T12:22:05.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc01204a0 con 0x7f1cc013b070 2023-10-30T12:22:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0138630 con 0x7f1cc006b640 2023-10-30T12:22:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cc5939640 10 monclient(hunting): authenticate will time out at 2170.706235s 2023-10-30T12:22:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc006afc0 0x7f1cc01ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006b640 0x7f1cc01ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc006afc0 0x7f1cc01ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48182/0 (socket says 172.21.15.50:48182) 2023-10-30T12:22:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006b640 0x7f1cc01ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55112/0 (socket says 172.21.15.50:55112) 2023-10-30T12:22:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 -- 172.21.15.50:0/214042578 learned_addr learned my addr 172.21.15.50:0/214042578 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): get_auth_request con 0x7f1cc006afc0 auth_method 0 2023-10-30T12:22:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 10 monclient(hunting): get_auth_request con 0x7f1cc006b640 auth_method 0 2023-10-30T12:22:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbf7fe640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc013b070 0x7f1cc011c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbf7fe640 10 monclient(hunting): get_auth_request con 0x7f1cc013b070 auth_method 0 2023-10-30T12:22:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient(hunting): handle_auth_done global_id 264213 payload 824 2023-10-30T12:22:05.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006b640 msgr2=0x7f1cc01ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cc006b640 0x7f1cc01ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc013b070 msgr2=0x7f1cc011c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc013b070 0x7f1cc011c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 monclient: found mon.noname-a 2023-10-30T12:22:05.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:05.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 -- 172.21.15.50:0/214042578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8009a70 con 0x7f1cc006afc0 2023-10-30T12:22:05.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbf7fe640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1cc013b070 0x7f1cc011c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:22:05.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbeffd640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc006afc0 0x7f1cc01ca470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009fd0 tx=0x7f1cb000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 10 client.?.objecter ms_handle_connect 0x7f1cc006afc0 2023-10-30T12:22:05.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:05.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000c940 con 0x7f1cc006afc0 2023-10-30T12:22:05.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:05.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 10 monclient: dump: 2023-10-30T12:22:05.563 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.563 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.563 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.565 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.523+0000 7f1cbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 5 monclient: authenticate success, global_id 264213 2023-10-30T12:22:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb000caa0 con 0x7f1cc006afc0 2023-10-30T12:22:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cbcff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000ce30 con 0x7f1cc006afc0 2023-10-30T12:22:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 10 monclient: _renew_subs 2023-10-30T12:22:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc011ea60 con 0x7f1cc006afc0 2023-10-30T12:22:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cbdffb640 4 set_mon_vals no callback set 2023-10-30T12:22:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cbcff9640 10 monclient: dump: 2023-10-30T12:22:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.569 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 10 client.264213.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 10 monclient: _renew_subs 2023-10-30T12:22:05.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc011e0a0 con 0x7f1cc006afc0 2023-10-30T12:22:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 1 librados: init done 2023-10-30T12:22:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d100 0x7f1cc002b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.524+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0073ff0 con 0x7f1cc011d100 2023-10-30T12:22:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1cb0013470 con 0x7f1cc006afc0 2023-10-30T12:22:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d100 0x7f1cc002b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient: get_auth_request con 0x7f1cc011d100 auth_method 0 2023-10-30T12:22:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ca00392f0 0x7f1ca003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_done global_id 264216 payload 824 2023-10-30T12:22:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 1 -- 172.21.15.50:0/214042578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cc01205e0 con 0x7f1cc011d100 2023-10-30T12:22:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(666..666 src has 520..666) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f1cb004af80 con 0x7f1cc006afc0 2023-10-30T12:22:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 client.264213.objecter ms_dispatch 0x7f1cc0000b90 osd_map(666..666 src has 520..666) v4 2023-10-30T12:22:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 3 client.264213.objecter handle_osd_map got epochs [666,666] > 0 2023-10-30T12:22:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 3 client.264213.objecter handle_osd_map decoding full epoch 666 2023-10-30T12:22:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 20 client.264213.objecter dump_active .. 0 homeless 2023-10-30T12:22:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 client.264213.objecter ms_handle_connect 0x7f1cc011d100 2023-10-30T12:22:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 client.264213.objecter resend_mon_ops 2023-10-30T12:22:05.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cb7fff640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d100 0x7f1cc002b430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1cc011dca0 tx=0x7f1ca800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbf7fe640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ca00392f0 0x7f1ca003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbf7fe640 10 monclient: get_auth_request con 0x7f1ca00392f0 auth_method 0 2023-10-30T12:22:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca80050d0 con 0x7f1cc011d100 2023-10-30T12:22:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 monclient: dump: 2023-10-30T12:22:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.577 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.577 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1ca8005230 con 0x7f1cc011d100 2023-10-30T12:22:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d100 msgr2=0x7f1cc002b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d100 0x7f1cc002b430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1cc011dca0 tx=0x7f1ca800fc00 comp rx=0 tx=0).stop 2023-10-30T12:22:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d100 msgr2=0x7f1cc002b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.525+0000 7f1cbcff9640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d100 0x7f1cc002b430 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1cc011dca0 tx=0x7f1ca800fc00 comp rx=0 tx=0).stop 2023-10-30T12:22:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.526+0000 7f1cbcff9640 10 client.264213.objecter ms_handle_connect 0x7f1ca00392f0 2023-10-30T12:22:05.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.526+0000 7f1cbf7fe640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ca00392f0 0x7f1ca003b7b0 secure :-1 s=READY pgs=4336 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8036000 tx=0x7f1cb8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cc5939640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cc5939640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d4e0 0x7f1cc002b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cc5939640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0120040 con 0x7f1cc011d4e0 2023-10-30T12:22:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d4e0 0x7f1cc002b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient: get_auth_request con 0x7f1cc011d4e0 auth_method 0 2023-10-30T12:22:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 10 monclient(hunting): handle_auth_done global_id 264219 payload 824 2023-10-30T12:22:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 1 -- 172.21.15.50:0/214042578 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1cc009f0b0 con 0x7f1cc011d4e0 2023-10-30T12:22:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cbcff9640 10 client.264213.objecter ms_handle_connect 0x7f1cc011d4e0 2023-10-30T12:22:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cbcff9640 10 client.264213.objecter resend_mon_ops 2023-10-30T12:22:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cb7fff640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d4e0 0x7f1cc002b970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8000fb0 tx=0x7f1ca8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8024070 con 0x7f1cc011d4e0 2023-10-30T12:22:05.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.532+0000 7f1cbcff9640 10 monclient: dump: 2023-10-30T12:22:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.585 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.587 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4634 (secure 0 0 0) 0x7f1ca8007300 con 0x7f1cc011d4e0 2023-10-30T12:22:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbcff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d4e0 msgr2=0x7f1cc002b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbcff9640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d4e0 0x7f1cc002b970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8000fb0 tx=0x7f1ca8001120 comp rx=0 tx=0).stop 2023-10-30T12:22:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbcff9640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d4e0 msgr2=0x7f1cc002b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:05.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbcff9640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc011d4e0 0x7f1cc002b970 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8000fb0 tx=0x7f1ca8001120 comp rx=0 tx=0).stop 2023-10-30T12:22:05.589 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:05.589 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:05.589 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:05.589 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:22:05.589 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 218, 2023-10-30T12:22:05.590 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:05.590 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:05.590 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:22:05.590 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:22:05.590 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.591 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:22:05.591 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:05.591 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:05.591 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:05.591 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:05.593 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:05.593 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:05.593 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:05.593 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:05.593 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.594 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:05.594 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:05.594 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:05.594 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:05.594 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:05.595 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:05.595 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:05.595 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:05.595 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:05.595 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:05.595 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:05.596 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:05.596 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.596 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:05.596 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:05.597 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:05.597 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:05.597 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:05.597 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:05.597 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:05.598 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:05.598 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:05.598 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:05.598 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:05.598 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:05.599 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:05.599 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:05.599 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:05.599 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:05.599 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:05.599 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:05.600 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:05.600 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:05.600 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:05.600 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:05.600 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:05.601 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:05.601 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:05.601 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:05.601 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:05.601 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.601 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:05.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.602 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:05.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.602 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:05.602 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:05.603 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:05.603 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:05.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.603 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:05.603 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:05.604 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.604 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:05.604 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:05.604 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.605 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.605 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.605 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:05.605 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:05.606 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:05.606 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:05.606 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:05.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.607 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:05.607 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:05.607 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:05.607 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:05.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.608 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:05.608 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:05.608 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.609 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:05.609 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:05.609 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.609 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.609 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.610 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:05.610 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:05.610 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:05.610 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:05.611 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:05.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.611 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:05.611 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:05.612 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:05.612 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:05.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.612 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:05.612 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:05.612 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:05.613 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:05.613 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.614 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.614 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.614 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:05.614 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.616 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:05.616 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:05.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.616 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:05.616 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:05.617 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:05.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.617 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.617 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:22:05.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.618 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:05.618 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:05.618 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:22:05.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.618 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:05.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.619 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:05.619 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:05.619 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:22:05.620 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.620 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.620 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:05.620 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 10 librados: watch_flush enter 2023-10-30T12:22:05.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 10 librados: watch_flush exit 2023-10-30T12:22:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 20 client.264213.objecter shutdown clearing up homeless session... 2023-10-30T12:22:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 10 client.264213.objecter successfully canceled tick 2023-10-30T12:22:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ca00392f0 msgr2=0x7f1ca003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1ca00392f0 0x7f1ca003b7b0 secure :-1 s=READY pgs=4336 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8036000 tx=0x7f1cb8030000 comp rx=0 tx=0).stop 2023-10-30T12:22:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 10 monclient: shutdown 2023-10-30T12:22:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc006afc0 msgr2=0x7f1cc01ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc006afc0 0x7f1cc01ca470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009fd0 tx=0x7f1cb000ed40 comp rx=0 tx=0).stop 2023-10-30T12:22:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cbf7fe640 1 -- 172.21.15.50:0/214042578 reap_dead start 2023-10-30T12:22:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 shutdown_connections 2023-10-30T12:22:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 --2- 172.21.15.50:0/214042578 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc006afc0 0x7f1cc01ca470 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 >> 172.21.15.50:0/214042578 conn(0x7f1cc00a6200 msgr2=0x7f1cc0137df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 shutdown_connections 2023-10-30T12:22:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 -- 172.21.15.50:0/214042578 wait complete. 2023-10-30T12:22:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.533+0000 7f1cc5939640 1 librados: shutdown 2023-10-30T12:22:05.626 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:22:05.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:22:05.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:22:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.731+0000 7f04b37b7640 10 monclient: get_monmap_and_config 2023-10-30T12:22:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.731+0000 7f04b37b7640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.731+0000 7f04b37b7640 10 monclient: monmap: 2023-10-30T12:22:05.733 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:05.732334+0000 2023-10-30T12:22:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:05.732334+0000 2023-10-30T12:22:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.734 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:05.735 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:05.736 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.731+0000 7f04b37b7640 1 Processor -- start 2023-10-30T12:22:05.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 1 -- start start 2023-10-30T12:22:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient: init 2023-10-30T12:22:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-10-30T12:22:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f04ac0a2ae0 0x7f04ac0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient(hunting): picked mon.noname-a con 0x7f04ac0a2ae0 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient(hunting): picked mon.noname-b con 0x7f04ac0a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04ac0a32e0 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac0a4190 con 0x7f04ac0a32e0 2023-10-30T12:22:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac0a2990 con 0x7f04ac0a2ae0 2023-10-30T12:22:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0a1630 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b37b7640 10 monclient(hunting): authenticate will time out at 2170.915571s 2023-10-30T12:22:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f04ac0a2ae0 0x7f04ac0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55124/0 (socket says 172.21.15.50:55124) 2023-10-30T12:22:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f04ac0a2ae0 0x7f04ac0a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48210/0 (socket says 172.21.15.50:48210) 2023-10-30T12:22:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 1 -- 172.21.15.50:0/1341810172 learned_addr learned my addr 172.21.15.50:0/1341810172 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 10 monclient(hunting): get_auth_request con 0x7f04ac0a2ec0 auth_method 0 2023-10-30T12:22:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1d54640 1 --1- 172.21.15.50:0/1341810172 >> v1:172.21.15.50:6789/0 conn(0x7f04ac0a32e0 0x7f04ac135c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46028/0 (socket says 172.21.15.50:46028) 2023-10-30T12:22:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 10 monclient(hunting): get_auth_request con 0x7f04ac0a2ae0 auth_method 0 2023-10-30T12:22:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.732+0000 7f04b0d52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 10 monclient(hunting): handle_auth_done global_id 264222 payload 274 2023-10-30T12:22:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 1 -- 172.21.15.50:0/1341810172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 msgr2=0x7f04ac1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:05.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 1 --2- 172.21.15.50:0/1341810172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 1 -- 172.21.15.50:0/1341810172 >> v1:172.21.15.50:6789/0 conn(0x7f04ac0a32e0 legacy=0x7f04ac135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 1 monclient: found mon.noname-a 2023-10-30T12:22:05.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 1 -- 172.21.15.50:0/1341810172 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625537503 0 0) 0x7f04a4001490 con 0x7f04ac0a32e0 2023-10-30T12:22:05.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:05.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 1 -- 172.21.15.50:0/1341810172 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac09a2e0 con 0x7f04ac0a2ae0 2023-10-30T12:22:05.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:22:05.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04b1553640 1 --2- 172.21.15.50:0/1341810172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac0a2450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04a80097a0 tx=0x7f04a802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccf8c5688cfb71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 1 -- 172.21.15.50:0/1341810172 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a802f050 con 0x7f04ac0a2ae0 2023-10-30T12:22:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 10 monclient: dump: 2023-10-30T12:22:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.755 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.755 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.755 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.755 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.755 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.756 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.733+0000 7f04a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04a3fff640 1 -- 172.21.15.50:0/1341810172 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8031700 con 0x7f04ac0a2ae0 2023-10-30T12:22:05.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 5 monclient: authenticate success, global_id 264222 2023-10-30T12:22:05.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:05.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04a3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04a3fff640 1 -- 172.21.15.50:0/1341810172 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8031a90 con 0x7f04ac0a2ae0 2023-10-30T12:22:05.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:05.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04a3fff640 10 monclient: dump: 2023-10-30T12:22:05.758 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.758 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.759 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.759 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.759 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.759 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.759 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.759 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.760 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.760 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:05.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 4 set_mon_vals no callback set 2023-10-30T12:22:05.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 10 monclient: shutdown 2023-10-30T12:22:05.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:05.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 -- 172.21.15.50:0/1341810172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 msgr2=0x7f04ac0a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 --2- 172.21.15.50:0/1341810172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac0a2450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f04a80097a0 tx=0x7f04a802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 -- 172.21.15.50:0/1341810172 shutdown_connections 2023-10-30T12:22:05.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 --2- 172.21.15.50:0/1341810172 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 --2- 172.21.15.50:0/1341810172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac0a2450 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 -- 172.21.15.50:0/1341810172 >> 172.21.15.50:0/1341810172 conn(0x7f04ac0a80b0 msgr2=0x7f04ac0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:05.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 -- 172.21.15.50:0/1341810172 shutdown_connections 2023-10-30T12:22:05.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.734+0000 7f04b37b7640 1 -- 172.21.15.50:0/1341810172 wait complete. 2023-10-30T12:22:05.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient: monmap: 2023-10-30T12:22:05.763 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.763 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:05.763 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:05.763 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:05.763 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.764 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.764 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:05.764 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:05.764 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:05.764 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 librados: starting msgr at 2023-10-30T12:22:05.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 librados: starting objecter 2023-10-30T12:22:05.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 Processor -- start 2023-10-30T12:22:05.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 -- start start 2023-10-30T12:22:05.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 librados: setting wanted keys 2023-10-30T12:22:05.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 librados: calling monclient init 2023-10-30T12:22:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient: init 2023-10-30T12:22:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:22:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac1ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): picked mon.noname-a con 0x7f04ac0a2ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): picked mon.noname-c con 0x7f04ac0a2ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ac11e590 0x7f04ac11ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): picked mon.noname-b con 0x7f04ac11e590 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac121d60 con 0x7f04ac0a2ae0 2023-10-30T12:22:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac120e70 con 0x7f04ac11e590 2023-10-30T12:22:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0a1790 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.735+0000 7f04b37b7640 10 monclient(hunting): authenticate will time out at 2170.918954s 2023-10-30T12:22:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac1ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac1ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48222/0 (socket says 172.21.15.50:48222) 2023-10-30T12:22:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55130/0 (socket says 172.21.15.50:55130) 2023-10-30T12:22:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 1 -- 172.21.15.50:0/1586167839 learned_addr learned my addr 172.21.15.50:0/1586167839 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 10 monclient(hunting): get_auth_request con 0x7f04ac0a2ae0 auth_method 0 2023-10-30T12:22:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): get_auth_request con 0x7f04ac0a2ec0 auth_method 0 2023-10-30T12:22:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1d54640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ac11e590 0x7f04ac11ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient(hunting): handle_auth_done global_id 254264 payload 824 2023-10-30T12:22:05.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ac11e590 msgr2=0x7f04ac11ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ac11e590 0x7f04ac11ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 msgr2=0x7f04ac1ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac1ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 monclient: found mon.noname-c 2023-10-30T12:22:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 -- 172.21.15.50:0/1586167839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a8009400 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1d54640 10 monclient: get_auth_request con 0x7f04ac11e590 auth_method 0 2023-10-30T12:22:05.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1d54640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04ac11e590 0x7f04ac11ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:22:05.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b1553640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04ac0a2ae0 0x7f04ac1ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:22:05.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b0d52640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1cabf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f049c00e850 tx=0x7f049c00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 client.?.objecter ms_handle_connect 0x7f04ac0a2ec0 2023-10-30T12:22:05.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:05.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c00c940 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:22:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: dump: 2023-10-30T12:22:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.778 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b37b7640 5 monclient: authenticate success, global_id 254264 2023-10-30T12:22:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b37b7640 10 monclient: _renew_subs 2023-10-30T12:22:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b37b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac132750 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049c00caa0 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c00ce10 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a1ffb640 10 monclient: dump: 2023-10-30T12:22:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.782 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 10 client.254264.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.736+0000 7f04a37fe640 4 set_mon_vals no callback set 2023-10-30T12:22:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 10 monclient: _renew_subs 2023-10-30T12:22:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac120230 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 1 librados: init done 2023-10-30T12:22:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a32e0 0x7f04ac02b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac073ff0 con 0x7f04ac0a32e0 2023-10-30T12:22:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f049c013470 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b1553640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a32e0 0x7f04ac02b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b1553640 10 monclient: get_auth_request con 0x7f04ac0a32e0 auth_method 0 2023-10-30T12:22:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b1553640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b1553640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b1553640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.737+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04900392f0 0x7f049003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(666..666 src has 520..666) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f049c04afe0 con 0x7f04ac0a2ec0 2023-10-30T12:22:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 client.254264.objecter ms_dispatch 0x7f04ac000b90 osd_map(666..666 src has 520..666) v4 2023-10-30T12:22:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 3 client.254264.objecter handle_osd_map got epochs [666,666] > 0 2023-10-30T12:22:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04b1553640 10 monclient(hunting): handle_auth_done global_id 254267 payload 824 2023-10-30T12:22:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 3 client.254264.objecter handle_osd_map decoding full epoch 666 2023-10-30T12:22:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04b1553640 1 -- 172.21.15.50:0/1586167839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04ac09a5c0 con 0x7f04ac0a32e0 2023-10-30T12:22:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 20 client.254264.objecter dump_active .. 0 homeless 2023-10-30T12:22:05.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04b1d54640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04900392f0 0x7f049003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04b1d54640 10 monclient: get_auth_request con 0x7f04900392f0 auth_method 0 2023-10-30T12:22:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 client.254264.objecter ms_handle_connect 0x7f04ac0a32e0 2023-10-30T12:22:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 client.254264.objecter resend_mon_ops 2023-10-30T12:22:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04b1553640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a32e0 0x7f04ac02b430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f04a80097a0 tx=0x7f04a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a803d640 con 0x7f04ac0a32e0 2023-10-30T12:22:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 monclient: dump: 2023-10-30T12:22:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.791 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 client.254264.objecter ms_handle_connect 0x7f04900392f0 2023-10-30T12:22:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04b1d54640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04900392f0 0x7f049003b7b0 secure :-1 s=READY pgs=4337 cs=0 l=1 rev1=1 crypto rx=0x7f04a40012e0 tx=0x7f04a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f04a8041070 con 0x7f04ac0a32e0 2023-10-30T12:22:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a32e0 msgr2=0x7f04ac02b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a32e0 0x7f04ac02b430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f04a80097a0 tx=0x7f04a80046c0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a32e0 msgr2=0x7f04ac02b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:05.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.738+0000 7f04a1ffb640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a32e0 0x7f04ac02b430 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b37b7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b37b7640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac11f720 0x7f04ac02b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b37b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac073a20 con 0x7f04ac11f720 2023-10-30T12:22:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac11f720 0x7f04ac02b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient: get_auth_request con 0x7f04ac11f720 auth_method 0 2023-10-30T12:22:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 10 monclient(hunting): handle_auth_done global_id 254270 payload 824 2023-10-30T12:22:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 1 -- 172.21.15.50:0/1586167839 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04ac099b40 con 0x7f04ac11f720 2023-10-30T12:22:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 10 client.254264.objecter ms_handle_connect 0x7f04ac11f720 2023-10-30T12:22:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 10 client.254264.objecter resend_mon_ops 2023-10-30T12:22:05.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04b1553640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac11f720 0x7f04ac02b970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04a8034000 tx=0x7f04a8031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a802f040 con 0x7f04ac11f720 2023-10-30T12:22:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 10 monclient: dump: 2023-10-30T12:22:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.800 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f04a8052450 con 0x7f04ac11f720 2023-10-30T12:22:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac11f720 msgr2=0x7f04ac02b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac11f720 0x7f04ac02b970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04a8034000 tx=0x7f04a8031000 comp rx=0 tx=0).stop 2023-10-30T12:22:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac11f720 msgr2=0x7f04ac02b970 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.745+0000 7f04a1ffb640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac11f720 0x7f04ac02b970 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04a8034000 tx=0x7f04a8031000 comp rx=0 tx=0).stop 2023-10-30T12:22:05.802 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:05.802 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:05.802 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:05.802 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:22:05.803 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 218, 2023-10-30T12:22:05.803 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:05.803 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:05.803 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:22:05.803 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:22:05.804 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.804 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:22:05.804 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:05.804 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:05.804 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:05.804 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:05.805 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:05.805 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:05.805 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:05.805 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:05.805 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:05.805 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:05.807 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:05.807 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:05.807 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:05.807 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:05.807 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:05.808 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:05.808 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:05.808 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:05.808 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:05.808 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:05.808 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:05.809 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.809 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:05.809 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:05.809 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:05.810 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:05.810 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:05.810 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:05.810 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:05.810 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:05.810 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:05.811 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:05.811 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:05.811 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:05.811 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:05.811 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:05.812 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:05.812 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:05.812 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:05.812 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:05.812 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:05.812 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:05.813 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:05.813 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:05.813 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:05.813 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:05.813 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:05.814 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:05.814 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:05.814 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.814 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:05.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:05.815 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:05.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.816 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:05.816 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:05.816 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.817 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:05.817 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:05.817 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.817 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.818 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:05.818 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:05.818 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:05.818 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:05.819 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:05.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.819 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:05.819 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:05.820 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:05.820 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:05.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.820 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:05.820 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:05.820 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.821 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:05.821 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:05.821 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.822 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.822 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:05.822 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:05.823 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:05.823 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:05.823 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:05.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.823 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:05.824 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:05.824 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:05.824 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:05.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.824 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:05.825 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:05.825 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.825 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:05.825 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:05.826 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:05.826 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.826 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.826 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:05.827 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:05.827 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:05.827 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:05.827 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:05.827 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.827 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.828 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:05.828 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:05.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.828 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:22:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:22:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:22:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:22:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:05.833 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:05.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:05.833 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:05.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:05.833 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:05.834 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:05.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 10 librados: watch_flush enter 2023-10-30T12:22:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 10 librados: watch_flush exit 2023-10-30T12:22:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 20 client.254264.objecter shutdown clearing up homeless session... 2023-10-30T12:22:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 10 client.254264.objecter successfully canceled tick 2023-10-30T12:22:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04900392f0 msgr2=0x7f049003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f04900392f0 0x7f049003b7b0 secure :-1 s=READY pgs=4337 cs=0 l=1 rev1=1 crypto rx=0x7f04a40012e0 tx=0x7f04a4009040 comp rx=0 tx=0).stop 2023-10-30T12:22:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 10 monclient: shutdown 2023-10-30T12:22:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 msgr2=0x7f04ac1cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 --2- 172.21.15.50:0/1586167839 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f04ac0a2ec0 0x7f04ac1cabf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f049c00e850 tx=0x7f049c00ed10 comp rx=0 tx=0).stop 2023-10-30T12:22:05.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b1d54640 1 -- 172.21.15.50:0/1586167839 reap_dead start 2023-10-30T12:22:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b1d54640 1 -- 172.21.15.50:0/1586167839 reap_dead start 2023-10-30T12:22:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 shutdown_connections 2023-10-30T12:22:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 >> 172.21.15.50:0/1586167839 conn(0x7f04ac0a80b0 msgr2=0x7f04ac134cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 shutdown_connections 2023-10-30T12:22:05.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 -- 172.21.15.50:0/1586167839 wait complete. 2023-10-30T12:22:05.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.746+0000 7f04b37b7640 1 librados: shutdown 2023-10-30T12:22:05.839 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:22:05.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:22:05.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:22:05.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.943+0000 7f3d02105640 10 monclient: get_monmap_and_config 2023-10-30T12:22:05.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.943+0000 7f3d02105640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.943+0000 7f3d02105640 10 monclient: monmap: 2023-10-30T12:22:05.946 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.946 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.946 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:05.945016+0000 2023-10-30T12:22:05.947 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:05.945016+0000 2023-10-30T12:22:05.947 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.947 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.947 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:05.947 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:05.947 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:05.948 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:05.948 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:05.948 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.944+0000 7f3d02105640 1 Processor -- start 2023-10-30T12:22:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.944+0000 7f3d02105640 1 -- start start 2023-10-30T12:22:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.944+0000 7f3d02105640 10 monclient: init 2023-10-30T12:22:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-10-30T12:22:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cfc096540 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3cfc136960 0x7f3cfc138d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient(hunting): picked mon.noname-c con 0x7f3cfc136960 addr v2:172.21.15.118:3300/0 2023-10-30T12:22:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc139290 0x7f3cfc13b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient(hunting): picked mon.noname-b con 0x7f3cfc139290 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc09bd30 con 0x7f3cfc096540 2023-10-30T12:22:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc09c400 con 0x7f3cfc136960 2023-10-30T12:22:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc09c120 con 0x7f3cfc139290 2023-10-30T12:22:05.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3d02105640 10 monclient(hunting): authenticate will time out at 2171.128241s 2023-10-30T12:22:05.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc139290 0x7f3cfc13b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfb7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3cfc096540 0x7f3cfc096920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46036/0 (socket says 172.21.15.50:46036) 2023-10-30T12:22:05.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfb7fe640 1 -- 172.21.15.50:0/4231776165 learned_addr learned my addr 172.21.15.50:0/4231776165 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:22:05.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): get_auth_request con 0x7f3cfc139290 auth_method 0 2023-10-30T12:22:05.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 1 --2- 172.21.15.50:0/4231776165 >> v2:172.21.15.118:3300/0 conn(0x7f3cfc136960 0x7f3cfc138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 10 monclient(hunting): get_auth_request con 0x7f3cfc136960 auth_method 0 2023-10-30T12:22:05.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfa7fc640 1 -- 172.21.15.50:0/4231776165 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538127603 0 0) 0x7f3cfc09bd30 con 0x7f3cfc096540 2023-10-30T12:22:05.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfa7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:22:05.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfa7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfa7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfa7fc640 10 monclient(hunting): my global_id is 264231 2023-10-30T12:22:05.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfa7fc640 1 -- 172.21.15.50:0/4231776165 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce4003520 con 0x7f3cfc096540 2023-10-30T12:22:05.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient(hunting): handle_auth_done global_id 254273 payload 274 2023-10-30T12:22:05.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 1 -- 172.21.15.50:0/4231776165 >> v2:172.21.15.118:3300/0 conn(0x7f3cfc136960 msgr2=0x7f3cfc138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:05.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 1 --2- 172.21.15.50:0/4231776165 >> v2:172.21.15.118:3300/0 conn(0x7f3cfc136960 0x7f3cfc138d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 1 -- 172.21.15.50:0/4231776165 >> v1:172.21.15.50:6789/0 conn(0x7f3cfc096540 legacy=0x7f3cfc096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 1 monclient: found mon.noname-b 2023-10-30T12:22:05.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.945+0000 7f3cfbfff640 1 -- 172.21.15.50:0/4231776165 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc097030 con 0x7f3cfc139290 2023-10-30T12:22:05.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfbfff640 1 --2- 172.21.15.50:0/4231776165 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc139290 0x7f3cfc13b680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009d90 tx=0x7f3cf402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=344353418f114e57 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 1 -- 172.21.15.50:0/4231776165 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf402cbc0 con 0x7f3cfc139290 2023-10-30T12:22:05.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:22:05.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: dump: 2023-10-30T12:22:05.966 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.966 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.967 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.967 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.967 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.967 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.967 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.967 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.968 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.968 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3d02105640 5 monclient: authenticate success, global_id 254273 2023-10-30T12:22:05.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 1 -- 172.21.15.50:0/4231776165 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf402cd20 con 0x7f3cfc139290 2023-10-30T12:22:05.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3d02105640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:05.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3d02105640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:05.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 1 -- 172.21.15.50:0/4231776165 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4034660 con 0x7f3cfc139290 2023-10-30T12:22:05.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3d02105640 4 set_mon_vals no callback set 2023-10-30T12:22:05.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:05.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.946+0000 7f3cfa7fc640 10 monclient: dump: 2023-10-30T12:22:05.971 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.971 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.971 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.971 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.971 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.971 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.972 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.972 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.972 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.972 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 10 monclient: shutdown 2023-10-30T12:22:05.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:05.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 -- 172.21.15.50:0/4231776165 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc139290 msgr2=0x7f3cfc13b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 --2- 172.21.15.50:0/4231776165 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc139290 0x7f3cfc13b680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009d90 tx=0x7f3cf402be10 comp rx=0 tx=0).stop 2023-10-30T12:22:05.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 -- 172.21.15.50:0/4231776165 shutdown_connections 2023-10-30T12:22:05.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 --2- 172.21.15.50:0/4231776165 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc139290 0x7f3cfc13b680 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 --2- 172.21.15.50:0/4231776165 >> v2:172.21.15.118:3300/0 conn(0x7f3cfc136960 0x7f3cfc138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 -- 172.21.15.50:0/4231776165 >> 172.21.15.50:0/4231776165 conn(0x7f3cfc0a3980 msgr2=0x7f3cfc0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:05.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 -- 172.21.15.50:0/4231776165 shutdown_connections 2023-10-30T12:22:05.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 -- 172.21.15.50:0/4231776165 wait complete. 2023-10-30T12:22:05.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 10 monclient: build_initial_monmap 2023-10-30T12:22:05.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 10 monclient: monmap: 2023-10-30T12:22:05.975 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:05.975 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:05.975 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:05.975 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:05.975 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:05.976 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.976 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:05.976 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:05.976 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:05.976 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 librados: starting msgr at 2023-10-30T12:22:05.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.947+0000 7f3d02105640 1 librados: starting objecter 2023-10-30T12:22:05.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 Processor -- start 2023-10-30T12:22:05.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 -- start start 2023-10-30T12:22:05.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 librados: setting wanted keys 2023-10-30T12:22:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 librados: calling monclient init 2023-10-30T12:22:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient: init 2023-10-30T12:22:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:22:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cfc096540 0x7f3cfc1c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): picked mon.noname-a con 0x7f3cfc096540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc136960 0x7f3cfc1c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): picked mon.noname-c con 0x7f3cfc136960 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc139290 0x7f3cfc070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): picked mon.noname-b con 0x7f3cfc139290 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:05.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc093380 con 0x7f3cfc096540 2023-10-30T12:22:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc08d6a0 con 0x7f3cfc139290 2023-10-30T12:22:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0a28e0 con 0x7f3cfc136960 2023-10-30T12:22:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3d02105640 10 monclient(hunting): authenticate will time out at 2171.131685s 2023-10-30T12:22:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc136960 0x7f3cfc1c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cfc096540 0x7f3cfc1c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc136960 0x7f3cfc1c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55150/0 (socket says 172.21.15.50:55150) 2023-10-30T12:22:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 1 -- 172.21.15.50:0/50464851 learned_addr learned my addr 172.21.15.50:0/50464851 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): get_auth_request con 0x7f3cfc136960 auth_method 0 2023-10-30T12:22:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 10 monclient(hunting): get_auth_request con 0x7f3cfc096540 auth_method 0 2023-10-30T12:22:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfaffd640 10 monclient(hunting): handle_auth_done global_id 254276 payload 824 2023-10-30T12:22:05.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.948+0000 7f3cfbfff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc139290 0x7f3cfc070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc139290 msgr2=0x7f3cfc070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc139290 0x7f3cfc070680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cfc096540 msgr2=0x7f3cfc1c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:05.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cfc096540 0x7f3cfc1c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:05.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 10 monclient: _finish_hunting 0 2023-10-30T12:22:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 1 monclient: found mon.noname-c 2023-10-30T12:22:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 1 -- 172.21.15.50:0/50464851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf4009a70 con 0x7f3cfc136960 2023-10-30T12:22:05.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cfaffd640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc136960 0x7f3cfc1c8bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000e980 tx=0x7f3cf000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 client.?.objecter ms_handle_connect 0x7f3cfc136960 2023-10-30T12:22:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0009670 con 0x7f3cfc136960 2023-10-30T12:22:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:22:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: dump: 2023-10-30T12:22:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.988 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.989 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.989 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.989 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.989 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.989 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: _finish_auth 0 2023-10-30T12:22:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:22:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 5 monclient: authenticate success, global_id 254276 2023-10-30T12:22:05.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf000ca00 con 0x7f3cfc136960 2023-10-30T12:22:05.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 10 monclient: _renew_subs 2023-10-30T12:22:05.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc08ce60 con 0x7f3cfc136960 2023-10-30T12:22:05.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:05.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf000cd90 con 0x7f3cfc136960 2023-10-30T12:22:05.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:05.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:05.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3ce3fff640 10 monclient: dump: 2023-10-30T12:22:05.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:05.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:05.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.993 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:05.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:05.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:05.993 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:05.993 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:05.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:05.994 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:05.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 10 client.254276.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:05.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 10 monclient: _renew_subs 2023-10-30T12:22:05.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:05.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3cf9ffb640 4 set_mon_vals no callback set 2023-10-30T12:22:05.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc1c9110 con 0x7f3cfc136960 2023-10-30T12:22:05.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.949+0000 7f3d02105640 1 librados: init done 2023-10-30T12:22:05.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3cf0013470 con 0x7f3cfc136960 2023-10-30T12:22:05.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3d02105640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:05.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3d02105640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc08b830 0x7f3cfc070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3d02105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc073ff0 con 0x7f3cfc08b830 2023-10-30T12:22:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3ce3fff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3ce80392f0 0x7f3ce803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc08b830 0x7f3cfc070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient: get_auth_request con 0x7f3cfc08b830 auth_method 0 2023-10-30T12:22:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:05.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:05.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:05.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(666..666 src has 520..666) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f3cf004b000 con 0x7f3cfc136960 2023-10-30T12:22:05.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3ce3fff640 10 client.254276.objecter ms_dispatch 0x7f3cfc000b90 osd_map(666..666 src has 520..666) v4 2023-10-30T12:22:05.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3ce3fff640 3 client.254276.objecter handle_osd_map got epochs [666,666] > 0 2023-10-30T12:22:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3ce3fff640 3 client.254276.objecter handle_osd_map decoding full epoch 666 2023-10-30T12:22:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfbfff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3ce80392f0 0x7f3ce803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfbfff640 10 monclient: get_auth_request con 0x7f3ce80392f0 auth_method 0 2023-10-30T12:22:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3ce3fff640 20 client.254276.objecter dump_active .. 0 homeless 2023-10-30T12:22:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_done global_id 274114 payload 824 2023-10-30T12:22:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.950+0000 7f3cfb7fe640 1 -- 172.21.15.50:0/50464851 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cfc09ba70 con 0x7f3cfc08b830 2023-10-30T12:22:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 client.254276.objecter ms_handle_connect 0x7f3cfc08b830 2023-10-30T12:22:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 client.254276.objecter resend_mon_ops 2023-10-30T12:22:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3cfb7fe640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc08b830 0x7f3cfc070bc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001000 tx=0x7f3cec001a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec006e90 con 0x7f3cfc08b830 2023-10-30T12:22:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:22:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 monclient: dump: 2023-10-30T12:22:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:06.002 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 client.254276.objecter ms_handle_connect 0x7f3ce80392f0 2023-10-30T12:22:06.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3cfbfff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3ce80392f0 0x7f3ce803b7b0 secure :-1 s=READY pgs=4338 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4036000 tx=0x7f3cf4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:06.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f3cec00f7a0 con 0x7f3cfc08b830 2023-10-30T12:22:06.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:06.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:06.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc08b830 msgr2=0x7f3cfc070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc08b830 0x7f3cfc070bc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001000 tx=0x7f3cec001a50 comp rx=0 tx=0).stop 2023-10-30T12:22:06.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc08b830 msgr2=0x7f3cfc070bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:06.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.951+0000 7f3ce3fff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc08b830 0x7f3cfc070bc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.957+0000 7f3d02105640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:06.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.957+0000 7f3d02105640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc071100 0x7f3cfc071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:06.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.957+0000 7f3d02105640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:06.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.957+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0678d0 con 0x7f3cfc071100 2023-10-30T12:22:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc071100 0x7f3cfc071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient: get_auth_request con 0x7f3cfc071100 auth_method 0 2023-10-30T12:22:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 10 monclient(hunting): handle_auth_done global_id 274117 payload 824 2023-10-30T12:22:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 1 -- 172.21.15.50:0/50464851 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3cfc071a50 con 0x7f3cfc071100 2023-10-30T12:22:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3cfb7fe640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc071100 0x7f3cfc071510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001000 tx=0x7f3cec001050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 10 client.254276.objecter ms_handle_connect 0x7f3cfc071100 2023-10-30T12:22:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 10 client.254276.objecter resend_mon_ops 2023-10-30T12:22:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec024070 con 0x7f3cfc071100 2023-10-30T12:22:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:22:06.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 10 monclient: dump: 2023-10-30T12:22:06.010 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:06.010 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:06.010 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:06.012 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7f3cec01c630 con 0x7f3cfc071100 2023-10-30T12:22:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.958+0000 7f3ce3fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc071100 msgr2=0x7f3cfc071510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3ce3fff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc071100 0x7f3cfc071510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001000 tx=0x7f3cec001050 comp rx=0 tx=0).stop 2023-10-30T12:22:06.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3ce3fff640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc071100 msgr2=0x7f3cfc071510 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:06.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3ce3fff640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3cfc071100 0x7f3cfc071510 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001000 tx=0x7f3cec001050 comp rx=0 tx=0).stop 2023-10-30T12:22:06.013 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:06.014 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:06.014 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:06.014 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:22:06.014 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 218, 2023-10-30T12:22:06.014 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:06.015 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:06.015 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:22:06.015 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:22:06.015 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:06.015 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:22:06.016 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:06.016 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:06.016 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:06.016 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:06.016 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:06.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:06.017 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:06.017 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:06.017 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:06.017 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:06.018 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:06.018 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:06.018 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:06.018 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:06.018 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:06.018 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:06.019 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:06.019 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:06.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:06.019 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:06.019 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:06.020 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:06.020 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:06.020 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:06.020 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:06.020 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:06.021 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:06.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.021 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:06.021 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:06.021 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:06.022 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:06.022 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:06.022 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:06.022 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:06.022 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:06.023 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:06.023 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:06.023 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:06.023 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:06.023 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:06.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:06.025 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:06.025 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:06.025 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:06.025 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:06.026 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:06.026 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:06.026 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:06.026 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:06.026 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:06.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.027 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:06.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.027 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:06.027 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:06.027 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:06.028 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:06.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.028 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:06.028 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:06.028 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:06.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:06.029 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:06.029 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:06.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:06.030 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:06.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.030 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:06.030 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:06.030 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:06.031 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:06.031 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:06.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.031 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:06.032 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:06.032 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:06.032 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:06.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:06.032 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:06.033 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:06.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.033 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:06.033 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:06.034 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:06.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:06.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:06.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.034 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:06.035 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:06.035 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:06.035 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:06.035 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:06.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.036 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:06.036 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:06.036 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:06.036 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:06.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:06.037 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:06.037 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:06.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:06.038 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:06.038 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:06.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:06.038 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:06.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.039 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:06.039 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:06.039 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:06.039 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:06.039 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:06.040 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:06.040 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:06.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.040 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:06.040 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:06.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.041 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:06.041 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:06.041 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:06.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:06.041 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:06.042 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:06.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:06.042 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:06.042 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:06.042 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:06.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:06.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:06.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:06.043 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:06.043 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:06.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 10 librados: watch_flush enter 2023-10-30T12:22:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 10 librados: watch_flush exit 2023-10-30T12:22:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 20 client.254276.objecter shutdown clearing up homeless session... 2023-10-30T12:22:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 10 client.254276.objecter successfully canceled tick 2023-10-30T12:22:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3ce80392f0 msgr2=0x7f3ce803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3ce80392f0 0x7f3ce803b7b0 secure :-1 s=READY pgs=4338 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4036000 tx=0x7f3cf4030000 comp rx=0 tx=0).stop 2023-10-30T12:22:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 10 monclient: shutdown 2023-10-30T12:22:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc136960 msgr2=0x7f3cfc1c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 --2- 172.21.15.50:0/50464851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3cfc136960 0x7f3cfc1c8bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000e980 tx=0x7f3cf000ee40 comp rx=0 tx=0).stop 2023-10-30T12:22:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3cfbfff640 1 -- 172.21.15.50:0/50464851 reap_dead start 2023-10-30T12:22:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3cfbfff640 1 -- 172.21.15.50:0/50464851 reap_dead start 2023-10-30T12:22:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 shutdown_connections 2023-10-30T12:22:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 >> 172.21.15.50:0/50464851 conn(0x7f3cfc0a3980 msgr2=0x7f3cfc09a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 shutdown_connections 2023-10-30T12:22:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 -- 172.21.15.50:0/50464851 wait complete. 2023-10-30T12:22:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:05.959+0000 7f3d02105640 1 librados: shutdown 2023-10-30T12:22:06.049 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:22:06.049 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:22:06.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:22:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.153+0000 7f0b531b7640 10 monclient: get_monmap_and_config 2023-10-30T12:22:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.153+0000 7f0b531b7640 10 monclient: build_initial_monmap 2023-10-30T12:22:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.153+0000 7f0b531b7640 10 monclient: monmap: 2023-10-30T12:22:06.156 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:06.156 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:06.156 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:06.154861+0000 2023-10-30T12:22:06.156 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:06.154861+0000 2023-10-30T12:22:06.156 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:06.158 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:06.158 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 1 Processor -- start 2023-10-30T12:22:06.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 1 -- start start 2023-10-30T12:22:06.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 10 monclient: init 2023-10-30T12:22:06.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:06.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-10-30T12:22:06.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c0707d0 0x7f0b4c09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:06.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 10 monclient(hunting): picked mon.noname-b con 0x7f0b4c0707d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:06.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b4c070bb0 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:06.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f0b4c13aba0 0x7f0b4c13cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:06.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 10 monclient(hunting): picked mon.noname-c con 0x7f0b4c13aba0 addr v2:172.21.15.118:3300/0 2023-10-30T12:22:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c09ee40 con 0x7f0b4c070bb0 2023-10-30T12:22:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.154+0000 7f0b531b7640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c09f4f0 con 0x7f0b4c13aba0 2023-10-30T12:22:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b531b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b531b7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c09f7d0 con 0x7f0b4c0707d0 2023-10-30T12:22:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b531b7640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b531b7640 10 monclient(hunting): authenticate will time out at 2171.338015s 2023-10-30T12:22:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c0707d0 0x7f0b4c09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b4bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0b4c070bb0 0x7f0b4c070ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46040/0 (socket says 172.21.15.50:46040) 2023-10-30T12:22:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b4bfff640 1 -- 172.21.15.50:0/469267224 learned_addr learned my addr 172.21.15.50:0/469267224 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:22:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): get_auth_request con 0x7f0b4c0707d0 auth_method 0 2023-10-30T12:22:06.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:06.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:06.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:06.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:06.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b51754640 1 --2- 172.21.15.50:0/469267224 >> v2:172.21.15.118:3300/0 conn(0x7f0b4c13aba0 0x7f0b4c13cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b51754640 10 monclient(hunting): get_auth_request con 0x7f0b4c13aba0 auth_method 0 2023-10-30T12:22:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b51754640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b51754640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b51754640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient(hunting): handle_auth_done global_id 254279 payload 274 2023-10-30T12:22:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 1 -- 172.21.15.50:0/469267224 >> v2:172.21.15.118:3300/0 conn(0x7f0b4c13aba0 msgr2=0x7f0b4c13cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 1 --2- 172.21.15.50:0/469267224 >> v2:172.21.15.118:3300/0 conn(0x7f0b4c13aba0 0x7f0b4c13cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b4b7fe640 1 -- 172.21.15.50:0/469267224 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306151101 0 0) 0x7f0b4c09ee40 con 0x7f0b4c070bb0 2023-10-30T12:22:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 1 -- 172.21.15.50:0/469267224 >> v1:172.21.15.50:6789/0 conn(0x7f0b4c070bb0 legacy=0x7f0b4c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient: _finish_hunting 0 2023-10-30T12:22:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 1 monclient: found mon.noname-b 2023-10-30T12:22:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:22:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b50f53640 1 -- 172.21.15.50:0/469267224 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c09fab0 con 0x7f0b4c0707d0 2023-10-30T12:22:06.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b51754640 1 --2- 172.21.15.50:0/469267224 >> v2:172.21.15.118:3300/0 conn(0x7f0b4c13aba0 0x7f0b4c13cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:22:06.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.155+0000 7f0b4b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:22:06.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b50f53640 1 --2- 172.21.15.50:0/469267224 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c0707d0 0x7f0b4c09e7e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0b400097a0 tx=0x7f0b4002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4e48edbdf33d7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:06.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 1 -- 172.21.15.50:0/469267224 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400316f0 con 0x7f0b4c0707d0 2023-10-30T12:22:06.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:22:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: dump: 2023-10-30T12:22:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:06.176 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:22:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:22:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 5 monclient: authenticate success, global_id 254279 2023-10-30T12:22:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 1 -- 172.21.15.50:0/469267224 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b40031850 con 0x7f0b4c0707d0 2023-10-30T12:22:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 4 set_mon_vals no callback set 2023-10-30T12:22:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 1 -- 172.21.15.50:0/469267224 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b40031be0 con 0x7f0b4c0707d0 2023-10-30T12:22:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b4b7fe640 10 monclient: dump: 2023-10-30T12:22:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:06.179 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.179 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.179 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:06.179 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.179 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:06.179 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:06.180 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:06.180 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 10 monclient: shutdown 2023-10-30T12:22:06.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:06.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 -- 172.21.15.50:0/469267224 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c0707d0 msgr2=0x7f0b4c09e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 --2- 172.21.15.50:0/469267224 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c0707d0 0x7f0b4c09e7e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0b400097a0 tx=0x7f0b4002b790 comp rx=0 tx=0).stop 2023-10-30T12:22:06.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 -- 172.21.15.50:0/469267224 shutdown_connections 2023-10-30T12:22:06.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 --2- 172.21.15.50:0/469267224 >> v2:172.21.15.118:3300/0 conn(0x7f0b4c13aba0 0x7f0b4c13cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 --2- 172.21.15.50:0/469267224 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c0707d0 0x7f0b4c09e7e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 -- 172.21.15.50:0/469267224 >> 172.21.15.50:0/469267224 conn(0x7f0b4c0a47f0 msgr2=0x7f0b4c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:06.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 -- 172.21.15.50:0/469267224 shutdown_connections 2023-10-30T12:22:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.156+0000 7f0b531b7640 1 -- 172.21.15.50:0/469267224 wait complete. 2023-10-30T12:22:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 10 monclient: build_initial_monmap 2023-10-30T12:22:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 10 monclient: monmap: 2023-10-30T12:22:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:06.183 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:06.183 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:06.183 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:06.183 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.183 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:06.183 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:06.184 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:06.184 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 1 librados: starting msgr at 2023-10-30T12:22:06.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 1 librados: starting objecter 2023-10-30T12:22:06.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 1 Processor -- start 2023-10-30T12:22:06.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 1 -- start start 2023-10-30T12:22:06.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 1 librados: setting wanted keys 2023-10-30T12:22:06.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 1 librados: calling monclient init 2023-10-30T12:22:06.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.157+0000 7f0b531b7640 10 monclient: init 2023-10-30T12:22:06.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:06.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:22:06.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b4c0707d0 0x7f0b4c1ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:06.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): picked mon.noname-b con 0x7f0b4c0707d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:06.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b4c070bb0 0x7f0b4c1caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:06.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): picked mon.noname-a con 0x7f0b4c070bb0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:06.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c13aba0 0x7f0b4c11b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:06.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): picked mon.noname-c con 0x7f0b4c13aba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:06.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:06.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c135760 con 0x7f0b4c070bb0 2023-10-30T12:22:06.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:06.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c11fda0 con 0x7f0b4c0707d0 2023-10-30T12:22:06.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:06.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c11e020 con 0x7f0b4c13aba0 2023-10-30T12:22:06.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:06.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b531b7640 10 monclient(hunting): authenticate will time out at 2171.341541s 2023-10-30T12:22:06.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b4c070bb0 0x7f0b4c1caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:06.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b4c070bb0 0x7f0b4c1caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48246/0 (socket says 172.21.15.50:48246) 2023-10-30T12:22:06.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 -- 172.21.15.50:0/3462996723 learned_addr learned my addr 172.21.15.50:0/3462996723 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:06.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): get_auth_request con 0x7f0b4c070bb0 auth_method 0 2023-10-30T12:22:06.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:06.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:06.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:06.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c13aba0 0x7f0b4c11b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:06.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 10 monclient(hunting): get_auth_request con 0x7f0b4c13aba0 auth_method 0 2023-10-30T12:22:06.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:06.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:06.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:06.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:06.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:06.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:06.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:06.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:06.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:06.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b50f53640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b4c0707d0 0x7f0b4c1ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:06.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b50f53640 10 monclient(hunting): get_auth_request con 0x7f0b4c0707d0 auth_method 0 2023-10-30T12:22:06.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b50f53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:06.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b50f53640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:06.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b50f53640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:06.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient(hunting): handle_auth_done global_id 264240 payload 824 2023-10-30T12:22:06.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 -- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c13aba0 msgr2=0x7f0b4c11b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c13aba0 0x7f0b4c11b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 -- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b4c0707d0 msgr2=0x7f0b4c1ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b4c0707d0 0x7f0b4c1ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:22:06.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 monclient: found mon.noname-a 2023-10-30T12:22:06.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:06.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:06.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b4bfff640 1 -- 172.21.15.50:0/3462996723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b40009400 con 0x7f0b4c070bb0 2023-10-30T12:22:06.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.158+0000 7f0b51754640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c13aba0 0x7f0b4c11b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:22:06.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b4bfff640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b4c070bb0 0x7f0b4c1caaa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800c4d0 tx=0x7f0b3800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:06.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 client.?.objecter ms_handle_connect 0x7f0b4c070bb0 2023-10-30T12:22:06.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:06.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 1 -- 172.21.15.50:0/3462996723 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b380074e0 con 0x7f0b4c070bb0 2023-10-30T12:22:06.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:06.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:06.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: dump: 2023-10-30T12:22:06.196 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:06.196 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:06.196 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.196 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.197 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:06.197 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.197 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:06.197 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:06.197 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:06.197 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: _finish_auth 0 2023-10-30T12:22:06.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: _check_auth_tickets 2023-10-30T12:22:06.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:06.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 5 monclient: authenticate success, global_id 264240 2023-10-30T12:22:06.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 1 -- 172.21.15.50:0/3462996723 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b38007660 con 0x7f0b4c070bb0 2023-10-30T12:22:06.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:06.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 1 -- 172.21.15.50:0/3462996723 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38004a20 con 0x7f0b4c070bb0 2023-10-30T12:22:06.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:06.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:06.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b497fa640 10 monclient: dump: 2023-10-30T12:22:06.199 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:06.200 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:06.200 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.200 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:06.200 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:06.200 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:06.200 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:06.201 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:06.201 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:06.201 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:06.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b4affd640 4 set_mon_vals no callback set 2023-10-30T12:22:06.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 10 monclient: _renew_subs 2023-10-30T12:22:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c09d6a0 con 0x7f0b4c070bb0 2023-10-30T12:22:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 10 client.264240.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 10 monclient: _renew_subs 2023-10-30T12:22:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c11f0e0 con 0x7f0b4c070bb0 2023-10-30T12:22:06.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.159+0000 7f0b531b7640 1 librados: init done 2023-10-30T12:22:06.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b497fa640 1 -- 172.21.15.50:0/3462996723 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0b38004b80 con 0x7f0b4c070bb0 2023-10-30T12:22:06.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b531b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:06.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b497fa640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0b300392f0 0x7f0b3003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:06.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b531b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:06.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b531b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:06.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c04e160 con 0x7f0b4c070bb0 2023-10-30T12:22:06.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b497fa640 1 -- 172.21.15.50:0/3462996723 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(667..667 src has 520..667) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f0b38049f40 con 0x7f0b4c070bb0 2023-10-30T12:22:06.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b497fa640 10 client.264240.objecter ms_dispatch 0x7f0b4c000b90 osd_map(667..667 src has 520..667) v4 2023-10-30T12:22:06.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b497fa640 3 client.264240.objecter handle_osd_map got epochs [667,667] > 0 2023-10-30T12:22:06.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b497fa640 3 client.264240.objecter handle_osd_map decoding full epoch 667 2023-10-30T12:22:06.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b497fa640 20 client.264240.objecter dump_active .. 0 homeless 2023-10-30T12:22:06.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b50f53640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0b300392f0 0x7f0b3003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:06.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.160+0000 7f0b50f53640 10 monclient: get_auth_request con 0x7f0b300392f0 auth_method 0 2023-10-30T12:22:06.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.161+0000 7f0b497fa640 10 client.264240.objecter ms_handle_connect 0x7f0b300392f0 2023-10-30T12:22:06.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.161+0000 7f0b50f53640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0b300392f0 0x7f0b3003b7b0 secure :-1 s=READY pgs=4339 cs=0 l=1 rev1=1 crypto rx=0x7f0b40036000 tx=0x7f0b40008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:06.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.165+0000 7f0b497fa640 1 -- 172.21.15.50:0/3462996723 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0b38076ca0 con 0x7f0b4c070bb0 2023-10-30T12:22:06.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.165+0000 7f0b497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:06.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.165+0000 7f0b497fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:06.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.407+0000 7f0b531b7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:22:06.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.407+0000 7f0b531b7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:22:06.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.407+0000 7f0b531b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:06.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.407+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0b4c0008d0 con 0x7f0b4c070bb0 2023-10-30T12:22:06.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.408+0000 7f0b497fa640 1 -- 172.21.15.50:0/3462996723 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0b38018080 con 0x7f0b4c070bb0 2023-10-30T12:22:06.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.408+0000 7f0b497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:22:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.408+0000 7f0b497fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:06.411 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.409+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.409+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=2899230517,config=3140514009,health=513986523,kv=2905431493,logm=1262027291}) 2023-10-30T12:22:06.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.409+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=9,kv=9,logm=53} crc {auth=1235974427,config=3140514009,health=311090877,kv=2905431493,logm=2373005695}) 2023-10-30T12:22:06.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.409+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.412 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.409+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=2899230517,config=3140514009,health=513986523,kv=2905431493,logm=1262027291}) 2023-10-30T12:22:06.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.409+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=662631691,config=3140514009,health=298569200,kv=4169269627,logm=461201998}) 2023-10-30T12:22:06.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=335996685}) 2023-10-30T12:22:06.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1541322199}) 2023-10-30T12:22:06.413 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=335996685}) 2023-10-30T12:22:06.414 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2921112282}) 2023-10-30T12:22:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 10 librados: watch_flush enter 2023-10-30T12:22:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 10 librados: watch_flush exit 2023-10-30T12:22:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 20 client.264240.objecter shutdown clearing up homeless session... 2023-10-30T12:22:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 10 client.264240.objecter successfully canceled tick 2023-10-30T12:22:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0b300392f0 msgr2=0x7f0b3003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0b300392f0 0x7f0b3003b7b0 secure :-1 s=READY pgs=4339 cs=0 l=1 rev1=1 crypto rx=0x7f0b40036000 tx=0x7f0b40008030 comp rx=0 tx=0).stop 2023-10-30T12:22:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 10 monclient: shutdown 2023-10-30T12:22:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b4c070bb0 msgr2=0x7f0b4c1caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.410+0000 7f0b531b7640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b4c070bb0 0x7f0b4c1caaa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800c4d0 tx=0x7f0b3800c990 comp rx=0 tx=0).stop 2023-10-30T12:22:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 shutdown_connections 2023-10-30T12:22:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f0b300392f0 0x7f0b3003b7b0 unknown :-1 s=CLOSED pgs=4339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b4c13aba0 0x7f0b4c11b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b4c070bb0 0x7f0b4c1caaa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 --2- 172.21.15.50:0/3462996723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b4c0707d0 0x7f0b4c1ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 >> 172.21.15.50:0/3462996723 conn(0x7f0b4c0a47f0 msgr2=0x7f0b4c09ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 shutdown_connections 2023-10-30T12:22:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 -- 172.21.15.50:0/3462996723 wait complete. 2023-10-30T12:22:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:06.411+0000 7f0b531b7640 1 librados: shutdown 2023-10-30T12:22:06.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.412+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.412+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419201670}) 2023-10-30T12:22:06.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.412+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252368809}) 2023-10-30T12:22:06.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.412+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.422 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.412+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419201670}) 2023-10-30T12:22:06.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.412+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284584707}) 2023-10-30T12:22:06.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.413+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.413+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140088632}) 2023-10-30T12:22:06.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.413+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915017868}) 2023-10-30T12:22:06.423 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.413+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.413+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140088632}) 2023-10-30T12:22:06.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.413+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559105275}) 2023-10-30T12:22:06.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.414+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.414+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420073051}) 2023-10-30T12:22:06.424 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.414+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4085158837}) 2023-10-30T12:22:06.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.414+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.414+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420073051}) 2023-10-30T12:22:06.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.414+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831417813}) 2023-10-30T12:22:06.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.415+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.425 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.415+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986803301}) 2023-10-30T12:22:06.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.415+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2931570122}) 2023-10-30T12:22:06.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.415+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.415+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986803301}) 2023-10-30T12:22:06.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.415+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=627373988}) 2023-10-30T12:22:06.426 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184684090}) 2023-10-30T12:22:06.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2817693071}) 2023-10-30T12:22:06.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184684090}) 2023-10-30T12:22:06.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473643147}) 2023-10-30T12:22:06.427 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235827450}) 2023-10-30T12:22:06.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151185844}) 2023-10-30T12:22:06.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.416+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235827450}) 2023-10-30T12:22:06.428 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401117044}) 2023-10-30T12:22:06.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248842240}) 2023-10-30T12:22:06.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1964285286}) 2023-10-30T12:22:06.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248842240}) 2023-10-30T12:22:06.429 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.417+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163278943}) 2023-10-30T12:22:06.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.418+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.418+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4235226902}) 2023-10-30T12:22:06.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.418+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927430311}) 2023-10-30T12:22:06.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.418+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.430 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.418+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4235226902}) 2023-10-30T12:22:06.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.418+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3406796530}) 2023-10-30T12:22:06.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605875665}) 2023-10-30T12:22:06.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=381007247}) 2023-10-30T12:22:06.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.431 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605875665}) 2023-10-30T12:22:06.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2141124311}) 2023-10-30T12:22:06.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172136158}) 2023-10-30T12:22:06.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748531214}) 2023-10-30T12:22:06.432 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172136158}) 2023-10-30T12:22:06.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.419+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4071472261}) 2023-10-30T12:22:06.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.420+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.420+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788663924}) 2023-10-30T12:22:06.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.420+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1682431882}) 2023-10-30T12:22:06.433 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.420+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.434 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.420+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788663924}) 2023-10-30T12:22:06.434 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.420+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166207627}) 2023-10-30T12:22:06.435 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.421+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.435 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.421+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92338731}) 2023-10-30T12:22:06.435 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.421+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1795680632}) 2023-10-30T12:22:06.436 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.421+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.436 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.421+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92338731}) 2023-10-30T12:22:06.436 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.421+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=625812197}) 2023-10-30T12:22:06.436 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.436 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096352806}) 2023-10-30T12:22:06.437 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1903026499}) 2023-10-30T12:22:06.437 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.437 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096352806}) 2023-10-30T12:22:06.437 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264909763}) 2023-10-30T12:22:06.438 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:22:06.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302915966}) 2023-10-30T12:22:06.448 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4269698444}) 2023-10-30T12:22:06.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302915966}) 2023-10-30T12:22:06.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.422+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457974122}) 2023-10-30T12:22:06.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.423+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.449 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.423+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=691132529}) 2023-10-30T12:22:06.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.423+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749029009}) 2023-10-30T12:22:06.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.423+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.423+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=691132529}) 2023-10-30T12:22:06.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.423+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022578598}) 2023-10-30T12:22:06.450 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830435967}) 2023-10-30T12:22:06.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2091527015}) 2023-10-30T12:22:06.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830435967}) 2023-10-30T12:22:06.451 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218096930}) 2023-10-30T12:22:06.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.424+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2209046716}) 2023-10-30T12:22:06.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.425+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517499015}) 2023-10-30T12:22:06.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.425+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.425+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2209046716}) 2023-10-30T12:22:06.452 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.425+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1526909175}) 2023-10-30T12:22:06.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.425+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.425+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653167484}) 2023-10-30T12:22:06.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960280667}) 2023-10-30T12:22:06.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.453 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653167484}) 2023-10-30T12:22:06.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907837924}) 2023-10-30T12:22:06.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3661829911}) 2023-10-30T12:22:06.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159898441}) 2023-10-30T12:22:06.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.454 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3661829911}) 2023-10-30T12:22:06.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.426+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2211713771}) 2023-10-30T12:22:06.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.427+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.427+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448941100}) 2023-10-30T12:22:06.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.427+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2740528875}) 2023-10-30T12:22:06.455 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.427+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.427+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448941100}) 2023-10-30T12:22:06.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.427+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=930333250}) 2023-10-30T12:22:06.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935396848}) 2023-10-30T12:22:06.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3498503254}) 2023-10-30T12:22:06.456 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935396848}) 2023-10-30T12:22:06.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372514148}) 2023-10-30T12:22:06.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765946781}) 2023-10-30T12:22:06.457 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2395661442}) 2023-10-30T12:22:06.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765946781}) 2023-10-30T12:22:06.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.428+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218210107}) 2023-10-30T12:22:06.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.429+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.429+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076969409}) 2023-10-30T12:22:06.458 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.429+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3935719822}) 2023-10-30T12:22:06.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.429+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.429+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1076969409}) 2023-10-30T12:22:06.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.429+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=115881924}) 2023-10-30T12:22:06.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.459 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516072256}) 2023-10-30T12:22:06.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693471839}) 2023-10-30T12:22:06.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516072256}) 2023-10-30T12:22:06.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206002188}) 2023-10-30T12:22:06.460 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782215918}) 2023-10-30T12:22:06.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966001455}) 2023-10-30T12:22:06.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782215918}) 2023-10-30T12:22:06.461 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.430+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239129519}) 2023-10-30T12:22:06.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.431+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.431+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188098128}) 2023-10-30T12:22:06.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.431+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1595970411}) 2023-10-30T12:22:06.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.431+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.462 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.431+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188098128}) 2023-10-30T12:22:06.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.431+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=519441928}) 2023-10-30T12:22:06.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529183977}) 2023-10-30T12:22:06.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963279516}) 2023-10-30T12:22:06.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529183977}) 2023-10-30T12:22:06.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3877007532}) 2023-10-30T12:22:06.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797804238}) 2023-10-30T12:22:06.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1923917360}) 2023-10-30T12:22:06.464 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797804238}) 2023-10-30T12:22:06.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.432+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2928171917}) 2023-10-30T12:22:06.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.433+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.433+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652571047}) 2023-10-30T12:22:06.465 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.433+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751227607}) 2023-10-30T12:22:06.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.433+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.433+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652571047}) 2023-10-30T12:22:06.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.433+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337309844}) 2023-10-30T12:22:06.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.466 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730849267}) 2023-10-30T12:22:06.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2218143173}) 2023-10-30T12:22:06.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730849267}) 2023-10-30T12:22:06.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=228961898}) 2023-10-30T12:22:06.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.467 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.434+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448134029}) 2023-10-30T12:22:06.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50510851}) 2023-10-30T12:22:06.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448134029}) 2023-10-30T12:22:06.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2838077552}) 2023-10-30T12:22:06.468 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742090014}) 2023-10-30T12:22:06.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027777701}) 2023-10-30T12:22:06.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742090014}) 2023-10-30T12:22:06.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.435+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622167400}) 2023-10-30T12:22:06.469 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.436+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.436+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405415351}) 2023-10-30T12:22:06.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.436+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=324193334}) 2023-10-30T12:22:06.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.436+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.436+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405415351}) 2023-10-30T12:22:06.470 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.436+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457962834}) 2023-10-30T12:22:06.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963423806}) 2023-10-30T12:22:06.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029479304}) 2023-10-30T12:22:06.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963423806}) 2023-10-30T12:22:06.471 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722936914}) 2023-10-30T12:22:06.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616396475}) 2023-10-30T12:22:06.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2704402730}) 2023-10-30T12:22:06.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.472 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616396475}) 2023-10-30T12:22:06.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.437+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253730564}) 2023-10-30T12:22:06.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.438+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.438+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140284707}) 2023-10-30T12:22:06.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.438+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164853350}) 2023-10-30T12:22:06.473 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.438+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.438+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140284707}) 2023-10-30T12:22:06.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.438+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3468044193}) 2023-10-30T12:22:06.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442761020}) 2023-10-30T12:22:06.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174528280}) 2023-10-30T12:22:06.474 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442761020}) 2023-10-30T12:22:06.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010134248}) 2023-10-30T12:22:06.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624981507}) 2023-10-30T12:22:06.475 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485112678}) 2023-10-30T12:22:06.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624981507}) 2023-10-30T12:22:06.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.439+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110105148}) 2023-10-30T12:22:06.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.440+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.440+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084030002}) 2023-10-30T12:22:06.476 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.440+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1335003724}) 2023-10-30T12:22:06.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.440+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.440+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084030002}) 2023-10-30T12:22:06.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.440+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881711645}) 2023-10-30T12:22:06.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.477 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110392553}) 2023-10-30T12:22:06.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3677835750}) 2023-10-30T12:22:06.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110392553}) 2023-10-30T12:22:06.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094725891}) 2023-10-30T12:22:06.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.478 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741778875}) 2023-10-30T12:22:06.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872818744}) 2023-10-30T12:22:06.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741778875}) 2023-10-30T12:22:06.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.441+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42314310}) 2023-10-30T12:22:06.479 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.442+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.442+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864883895}) 2023-10-30T12:22:06.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.442+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084936897}) 2023-10-30T12:22:06.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.442+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.442+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864883895}) 2023-10-30T12:22:06.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.442+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4089005771}) 2023-10-30T12:22:06.480 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826718323}) 2023-10-30T12:22:06.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830379420}) 2023-10-30T12:22:06.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826718323}) 2023-10-30T12:22:06.481 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2345765443}) 2023-10-30T12:22:06.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217250766}) 2023-10-30T12:22:06.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854744134}) 2023-10-30T12:22:06.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217250766}) 2023-10-30T12:22:06.482 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.443+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2050543602}) 2023-10-30T12:22:06.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.444+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.444+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097902226}) 2023-10-30T12:22:06.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.444+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285989175}) 2023-10-30T12:22:06.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.444+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.483 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.444+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097902226}) 2023-10-30T12:22:06.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.444+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2985060217}) 2023-10-30T12:22:06.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460248747}) 2023-10-30T12:22:06.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=226480926}) 2023-10-30T12:22:06.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.484 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460248747}) 2023-10-30T12:22:06.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1491891392}) 2023-10-30T12:22:06.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805721231}) 2023-10-30T12:22:06.485 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742923549}) 2023-10-30T12:22:06.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805721231}) 2023-10-30T12:22:06.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.445+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2878415442}) 2023-10-30T12:22:06.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.446+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.486 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.446+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285567568}) 2023-10-30T12:22:06.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.446+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3501937370}) 2023-10-30T12:22:06.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.446+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.446+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285567568}) 2023-10-30T12:22:06.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.446+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496991097}) 2023-10-30T12:22:06.487 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466878228}) 2023-10-30T12:22:06.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=240771091}) 2023-10-30T12:22:06.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466878228}) 2023-10-30T12:22:06.488 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495319824}) 2023-10-30T12:22:06.515 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.515 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803786635}) 2023-10-30T12:22:06.515 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2474484099}) 2023-10-30T12:22:06.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803786635}) 2023-10-30T12:22:06.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.447+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=852185171}) 2023-10-30T12:22:06.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.448+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.516 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.448+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284164512}) 2023-10-30T12:22:06.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.448+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196454092}) 2023-10-30T12:22:06.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.448+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.448+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284164512}) 2023-10-30T12:22:06.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.448+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1530570092}) 2023-10-30T12:22:06.517 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480864864}) 2023-10-30T12:22:06.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335783653}) 2023-10-30T12:22:06.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480864864}) 2023-10-30T12:22:06.518 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785966223}) 2023-10-30T12:22:06.519 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.519 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183570259}) 2023-10-30T12:22:06.519 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119175706}) 2023-10-30T12:22:06.519 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.519 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183570259}) 2023-10-30T12:22:06.519 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.449+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046109481}) 2023-10-30T12:22:06.520 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.450+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.520 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.450+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986706721}) 2023-10-30T12:22:06.520 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.450+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456095956}) 2023-10-30T12:22:06.520 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.450+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.520 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.450+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986706721}) 2023-10-30T12:22:06.520 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.450+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532615264}) 2023-10-30T12:22:06.521 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.521 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486930380}) 2023-10-30T12:22:06.521 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=637805140}) 2023-10-30T12:22:06.521 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.521 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486930380}) 2023-10-30T12:22:06.521 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952313755}) 2023-10-30T12:22:06.522 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.522 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1380858188}) 2023-10-30T12:22:06.522 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3139006296}) 2023-10-30T12:22:06.522 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.522 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1380858188}) 2023-10-30T12:22:06.523 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.451+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485953983}) 2023-10-30T12:22:06.523 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.452+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.523 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.452+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612799907}) 2023-10-30T12:22:06.523 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.452+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669598194}) 2023-10-30T12:22:06.523 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.452+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.524 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.452+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612799907}) 2023-10-30T12:22:06.524 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.452+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4134482301}) 2023-10-30T12:22:06.524 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.524 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930943937}) 2023-10-30T12:22:06.524 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684478204}) 2023-10-30T12:22:06.524 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.525 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930943937}) 2023-10-30T12:22:06.525 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404989454}) 2023-10-30T12:22:06.525 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.525 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082978291}) 2023-10-30T12:22:06.525 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1040317341}) 2023-10-30T12:22:06.526 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.526 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082978291}) 2023-10-30T12:22:06.526 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.453+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=489217320}) 2023-10-30T12:22:06.526 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.454+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.526 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.454+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=602969948}) 2023-10-30T12:22:06.527 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.454+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1933519584}) 2023-10-30T12:22:06.527 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.454+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.527 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.454+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=602969948}) 2023-10-30T12:22:06.527 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.454+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1489153934}) 2023-10-30T12:22:06.527 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.454+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.527 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787605064}) 2023-10-30T12:22:06.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985243793}) 2023-10-30T12:22:06.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787605064}) 2023-10-30T12:22:06.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2573531041}) 2023-10-30T12:22:06.528 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357734129}) 2023-10-30T12:22:06.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1192830312}) 2023-10-30T12:22:06.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357734129}) 2023-10-30T12:22:06.529 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.455+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003755151}) 2023-10-30T12:22:06.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.456+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.456+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485185001}) 2023-10-30T12:22:06.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.456+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032178170}) 2023-10-30T12:22:06.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.456+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.456+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485185001}) 2023-10-30T12:22:06.530 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.456+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1449213276}) 2023-10-30T12:22:06.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061803458}) 2023-10-30T12:22:06.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491403056}) 2023-10-30T12:22:06.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.531 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061803458}) 2023-10-30T12:22:06.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=928846645}) 2023-10-30T12:22:06.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166921589}) 2023-10-30T12:22:06.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590121}) 2023-10-30T12:22:06.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.532 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166921589}) 2023-10-30T12:22:06.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.457+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3762091913}) 2023-10-30T12:22:06.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.458+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.458+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233475463}) 2023-10-30T12:22:06.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.458+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1951791340}) 2023-10-30T12:22:06.533 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.458+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.458+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233475463}) 2023-10-30T12:22:06.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.458+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435704574}) 2023-10-30T12:22:06.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739428606}) 2023-10-30T12:22:06.534 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565072370}) 2023-10-30T12:22:06.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739428606}) 2023-10-30T12:22:06.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941120243}) 2023-10-30T12:22:06.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687799642}) 2023-10-30T12:22:06.535 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2808070559}) 2023-10-30T12:22:06.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687799642}) 2023-10-30T12:22:06.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.459+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963734461}) 2023-10-30T12:22:06.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.460+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.536 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.460+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406989350}) 2023-10-30T12:22:06.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.460+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2339794553}) 2023-10-30T12:22:06.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.460+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.460+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406989350}) 2023-10-30T12:22:06.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.460+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642437094}) 2023-10-30T12:22:06.537 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278888889}) 2023-10-30T12:22:06.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3929992093}) 2023-10-30T12:22:06.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278888889}) 2023-10-30T12:22:06.538 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953555053}) 2023-10-30T12:22:06.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471092169}) 2023-10-30T12:22:06.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950080416}) 2023-10-30T12:22:06.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471092169}) 2023-10-30T12:22:06.539 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.461+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282513090}) 2023-10-30T12:22:06.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.462+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.462+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593262998}) 2023-10-30T12:22:06.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.462+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453959807}) 2023-10-30T12:22:06.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.462+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.540 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.462+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593262998}) 2023-10-30T12:22:06.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.462+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3570916809}) 2023-10-30T12:22:06.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645985585}) 2023-10-30T12:22:06.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669681566}) 2023-10-30T12:22:06.541 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645985585}) 2023-10-30T12:22:06.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860720276}) 2023-10-30T12:22:06.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875439614}) 2023-10-30T12:22:06.542 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1633806597}) 2023-10-30T12:22:06.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875439614}) 2023-10-30T12:22:06.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.463+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3236547665}) 2023-10-30T12:22:06.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.464+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.543 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.464+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043844402}) 2023-10-30T12:22:06.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.464+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930923074}) 2023-10-30T12:22:06.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.464+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.464+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043844402}) 2023-10-30T12:22:06.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.464+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626971625}) 2023-10-30T12:22:06.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.544 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250862902}) 2023-10-30T12:22:06.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882805394}) 2023-10-30T12:22:06.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250862902}) 2023-10-30T12:22:06.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=292316216}) 2023-10-30T12:22:06.545 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598844111}) 2023-10-30T12:22:06.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3684555996}) 2023-10-30T12:22:06.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598844111}) 2023-10-30T12:22:06.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.465+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290059763}) 2023-10-30T12:22:06.546 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.466+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.466+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3667163110}) 2023-10-30T12:22:06.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.466+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4224139682}) 2023-10-30T12:22:06.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.466+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.466+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3667163110}) 2023-10-30T12:22:06.547 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.466+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080683321}) 2023-10-30T12:22:06.548 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.467+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.548 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.467+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390029696}) 2023-10-30T12:22:06.548 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.467+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117030285}) 2023-10-30T12:22:06.548 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.467+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.548 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.467+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390029696}) 2023-10-30T12:22:06.548 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.467+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=978146974}) 2023-10-30T12:22:06.549 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.467+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.549 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607492012}) 2023-10-30T12:22:06.549 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=891313335}) 2023-10-30T12:22:06.549 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.549 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607492012}) 2023-10-30T12:22:06.550 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2101047614}) 2023-10-30T12:22:06.550 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.550 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406186469}) 2023-10-30T12:22:06.550 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819534461}) 2023-10-30T12:22:06.550 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.550 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406186469}) 2023-10-30T12:22:06.551 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.468+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4019146126}) 2023-10-30T12:22:06.551 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.469+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.551 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.469+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492589216}) 2023-10-30T12:22:06.551 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.469+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519606841}) 2023-10-30T12:22:06.551 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.469+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.552 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.469+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492589216}) 2023-10-30T12:22:06.552 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.469+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1498772205}) 2023-10-30T12:22:06.552 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.552 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477714315}) 2023-10-30T12:22:06.552 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050874289}) 2023-10-30T12:22:06.552 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.553 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477714315}) 2023-10-30T12:22:06.553 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=675125894}) 2023-10-30T12:22:06.553 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.553 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147590642}) 2023-10-30T12:22:06.553 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319836470}) 2023-10-30T12:22:06.554 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.554 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147590642}) 2023-10-30T12:22:06.554 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.470+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941237942}) 2023-10-30T12:22:06.554 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.471+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.554 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.471+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517269748}) 2023-10-30T12:22:06.555 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.471+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714628168}) 2023-10-30T12:22:06.555 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.471+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.555 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.471+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517269748}) 2023-10-30T12:22:06.555 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.471+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826082255}) 2023-10-30T12:22:06.555 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.556 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945881538}) 2023-10-30T12:22:06.556 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461591062}) 2023-10-30T12:22:06.556 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.556 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945881538}) 2023-10-30T12:22:06.556 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618172689}) 2023-10-30T12:22:06.556 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.557 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311841374}) 2023-10-30T12:22:06.557 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3873846824}) 2023-10-30T12:22:06.557 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.557 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311841374}) 2023-10-30T12:22:06.557 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.472+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892063102}) 2023-10-30T12:22:06.557 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.473+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.558 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.473+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304885448}) 2023-10-30T12:22:06.558 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.473+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3905991914}) 2023-10-30T12:22:06.558 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.473+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.558 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.473+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304885448}) 2023-10-30T12:22:06.558 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.473+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3982954192}) 2023-10-30T12:22:06.559 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.559 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178136882}) 2023-10-30T12:22:06.559 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=290661262}) 2023-10-30T12:22:06.559 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.559 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178136882}) 2023-10-30T12:22:06.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136454881}) 2023-10-30T12:22:06.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750987380}) 2023-10-30T12:22:06.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=811178613}) 2023-10-30T12:22:06.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.560 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750987380}) 2023-10-30T12:22:06.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.474+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046729003}) 2023-10-30T12:22:06.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.475+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037724034}) 2023-10-30T12:22:06.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816591209}) 2023-10-30T12:22:06.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.561 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037724034}) 2023-10-30T12:22:06.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=267480551}) 2023-10-30T12:22:06.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286149937}) 2023-10-30T12:22:06.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1905688184}) 2023-10-30T12:22:06.562 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286149937}) 2023-10-30T12:22:06.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.476+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639309784}) 2023-10-30T12:22:06.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.477+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.477+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202191372}) 2023-10-30T12:22:06.563 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.477+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722723562}) 2023-10-30T12:22:06.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.477+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.477+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202191372}) 2023-10-30T12:22:06.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.477+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422355979}) 2023-10-30T12:22:06.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1721915857}) 2023-10-30T12:22:06.564 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729572143}) 2023-10-30T12:22:06.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1721915857}) 2023-10-30T12:22:06.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3230973828}) 2023-10-30T12:22:06.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.565 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536271960}) 2023-10-30T12:22:06.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597284633}) 2023-10-30T12:22:06.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536271960}) 2023-10-30T12:22:06.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.478+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380661488}) 2023-10-30T12:22:06.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.479+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.566 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.479+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792657966}) 2023-10-30T12:22:06.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.479+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925333564}) 2023-10-30T12:22:06.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.479+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.479+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792657966}) 2023-10-30T12:22:06.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.479+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146680954}) 2023-10-30T12:22:06.567 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960380351}) 2023-10-30T12:22:06.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3430167192}) 2023-10-30T12:22:06.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960380351}) 2023-10-30T12:22:06.568 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3963045873}) 2023-10-30T12:22:06.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098687770}) 2023-10-30T12:22:06.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=889053901}) 2023-10-30T12:22:06.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098687770}) 2023-10-30T12:22:06.569 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.480+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318698784}) 2023-10-30T12:22:06.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.481+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.481+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391411679}) 2023-10-30T12:22:06.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.481+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2652046082}) 2023-10-30T12:22:06.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.481+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.570 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.481+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391411679}) 2023-10-30T12:22:06.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.481+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715722327}) 2023-10-30T12:22:06.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283311336}) 2023-10-30T12:22:06.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679716442}) 2023-10-30T12:22:06.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.571 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283311336}) 2023-10-30T12:22:06.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4089444771}) 2023-10-30T12:22:06.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159808669}) 2023-10-30T12:22:06.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1604109898}) 2023-10-30T12:22:06.572 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159808669}) 2023-10-30T12:22:06.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.482+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628489778}) 2023-10-30T12:22:06.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604349157}) 2023-10-30T12:22:06.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018774502}) 2023-10-30T12:22:06.573 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604349157}) 2023-10-30T12:22:06.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595370053}) 2023-10-30T12:22:06.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.483+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961092231}) 2023-10-30T12:22:06.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989663559}) 2023-10-30T12:22:06.574 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961092231}) 2023-10-30T12:22:06.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103157845}) 2023-10-30T12:22:06.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1590413613}) 2023-10-30T12:22:06.575 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=548814139}) 2023-10-30T12:22:06.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1590413613}) 2023-10-30T12:22:06.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.484+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1691692120}) 2023-10-30T12:22:06.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.485+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.576 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.485+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664457520}) 2023-10-30T12:22:06.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.485+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3435853713}) 2023-10-30T12:22:06.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.485+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.485+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664457520}) 2023-10-30T12:22:06.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.485+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3733147995}) 2023-10-30T12:22:06.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.577 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650574324}) 2023-10-30T12:22:06.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3878461084}) 2023-10-30T12:22:06.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650574324}) 2023-10-30T12:22:06.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419919246}) 2023-10-30T12:22:06.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135842304}) 2023-10-30T12:22:06.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942654181}) 2023-10-30T12:22:06.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135842304}) 2023-10-30T12:22:06.579 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.486+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4231097450}) 2023-10-30T12:22:06.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.487+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.487+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689735370}) 2023-10-30T12:22:06.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.487+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=148824950}) 2023-10-30T12:22:06.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.487+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.580 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.487+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689735370}) 2023-10-30T12:22:06.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.487+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1219327259}) 2023-10-30T12:22:06.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=739277599}) 2023-10-30T12:22:06.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221465798}) 2023-10-30T12:22:06.581 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=739277599}) 2023-10-30T12:22:06.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708134646}) 2023-10-30T12:22:06.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2562712333}) 2023-10-30T12:22:06.582 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1065569223}) 2023-10-30T12:22:06.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2562712333}) 2023-10-30T12:22:06.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.488+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303778209}) 2023-10-30T12:22:06.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.489+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.489+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818233953}) 2023-10-30T12:22:06.583 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.489+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057695846}) 2023-10-30T12:22:06.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.489+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.489+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818233953}) 2023-10-30T12:22:06.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.489+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1275360207}) 2023-10-30T12:22:06.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.584 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440290202}) 2023-10-30T12:22:06.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1067412927}) 2023-10-30T12:22:06.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440290202}) 2023-10-30T12:22:06.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=164307997}) 2023-10-30T12:22:06.585 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797770685}) 2023-10-30T12:22:06.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.490+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3966217291}) 2023-10-30T12:22:06.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797770685}) 2023-10-30T12:22:06.586 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2549637843}) 2023-10-30T12:22:06.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663612065}) 2023-10-30T12:22:06.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=275567677}) 2023-10-30T12:22:06.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663612065}) 2023-10-30T12:22:06.587 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.491+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541405978}) 2023-10-30T12:22:06.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.492+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.492+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2650989487}) 2023-10-30T12:22:06.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.492+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1134100152}) 2023-10-30T12:22:06.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.492+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.588 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.492+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2650989487}) 2023-10-30T12:22:06.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.492+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872283462}) 2023-10-30T12:22:06.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.493+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.493+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928007617}) 2023-10-30T12:22:06.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.493+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2282482807}) 2023-10-30T12:22:06.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.493+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.589 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.493+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928007617}) 2023-10-30T12:22:06.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.493+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=541930074}) 2023-10-30T12:22:06.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282610856}) 2023-10-30T12:22:06.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384421886}) 2023-10-30T12:22:06.590 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282610856}) 2023-10-30T12:22:06.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2962416084}) 2023-10-30T12:22:06.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995365382}) 2023-10-30T12:22:06.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=500734121}) 2023-10-30T12:22:06.591 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995365382}) 2023-10-30T12:22:06.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.494+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2760833566}) 2023-10-30T12:22:06.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.495+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.495+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449126454}) 2023-10-30T12:22:06.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.495+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2722257620}) 2023-10-30T12:22:06.592 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.495+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.495+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449126454}) 2023-10-30T12:22:06.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.495+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2507554167}) 2023-10-30T12:22:06.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157212539}) 2023-10-30T12:22:06.593 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4260009665}) 2023-10-30T12:22:06.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157212539}) 2023-10-30T12:22:06.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38006439}) 2023-10-30T12:22:06.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.594 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172834450}) 2023-10-30T12:22:06.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899783408}) 2023-10-30T12:22:06.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172834450}) 2023-10-30T12:22:06.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.496+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961659617}) 2023-10-30T12:22:06.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.497+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.595 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.497+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106521955}) 2023-10-30T12:22:06.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.497+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1708120462}) 2023-10-30T12:22:06.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.497+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.497+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106521955}) 2023-10-30T12:22:06.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.497+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067610160}) 2023-10-30T12:22:06.596 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171024043}) 2023-10-30T12:22:06.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034718860}) 2023-10-30T12:22:06.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171024043}) 2023-10-30T12:22:06.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1637295466}) 2023-10-30T12:22:06.597 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747537413}) 2023-10-30T12:22:06.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1926516716}) 2023-10-30T12:22:06.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747537413}) 2023-10-30T12:22:06.598 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.498+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667915738}) 2023-10-30T12:22:06.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.499+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.499+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4138472686}) 2023-10-30T12:22:06.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.499+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=634447875}) 2023-10-30T12:22:06.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.499+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.499+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4138472686}) 2023-10-30T12:22:06.599 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.499+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2578177168}) 2023-10-30T12:22:06.600 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.600 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886066368}) 2023-10-30T12:22:06.600 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52052043}) 2023-10-30T12:22:06.600 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.600 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886066368}) 2023-10-30T12:22:06.601 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316633055}) 2023-10-30T12:22:06.601 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.601 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2316545300,mdsmap=361818223}) 2023-10-30T12:22:06.601 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271500827}) 2023-10-30T12:22:06.601 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.601 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2316545300,mdsmap=361818223}) 2023-10-30T12:22:06.602 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.500+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470287790}) 2023-10-30T12:22:06.602 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.501+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.602 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.501+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=3,osd_pg_creating=1,osd_snap=41,osdmap=26} crc {mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1083336090,mon_config_key=3206958671,monmap=3324057927,osd_metadata=3779642462,osd_pg_creating=1742268318,osd_snap=1814644193,osdmap=4051408049}) 2023-10-30T12:22:06.602 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.501+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=39,osdmap=27} crc {mdsmap=3140514009,mgr=413753915,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3163335290,mon_config_key=3217606506,monmap=3324057927,osd_metadata=3009730626,osd_snap=3791642825,osdmap=853348735}) 2023-10-30T12:22:06.602 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.501+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.603 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.501+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=3,osd_pg_creating=1,osd_snap=41,osdmap=26} crc {mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=1083336090,mon_config_key=3206958671,monmap=3324057927,osd_metadata=3779642462,osd_pg_creating=1742268318,osd_snap=1814644193,osdmap=4051408049}) 2023-10-30T12:22:06.603 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.501+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39,osdmap=25} crc {mdsmap=3140514009,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2307986955,mon_config_key=3965234311,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=1742268318,osd_snap=3760159354,osdmap=76525002}) 2023-10-30T12:22:06.603 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.502+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.603 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.502+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2762399809}) 2023-10-30T12:22:06.603 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.502+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1285509041}) 2023-10-30T12:22:06.604 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.502+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.604 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.502+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2762399809}) 2023-10-30T12:22:06.604 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.502+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3004020210}) 2023-10-30T12:22:06.604 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.508+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.604 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.508+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=1888653973}) 2023-10-30T12:22:06.604 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.508+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=99} crc {osdmap=3126696057}) 2023-10-30T12:22:06.605 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.508+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:22:06.605 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.508+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=1888653973}) 2023-10-30T12:22:06.605 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:22:06.508+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=96} crc {osdmap=3673028467}) 2023-10-30T12:22:07.439 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:22:07.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:22:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.553+0000 7f54bc133640 10 monclient: get_monmap_and_config 2023-10-30T12:22:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.553+0000 7f54bc133640 10 monclient: build_initial_monmap 2023-10-30T12:22:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.553+0000 7f54bc133640 10 monclient: monmap: 2023-10-30T12:22:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:07.554182+0000 2023-10-30T12:22:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:07.554182+0000 2023-10-30T12:22:07.557 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:07.558 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:07.559 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:07.559 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.553+0000 7f54bc133640 1 Processor -- start 2023-10-30T12:22:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.553+0000 7f54bc133640 1 -- start start 2023-10-30T12:22:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.553+0000 7f54bc133640 10 monclient: init 2023-10-30T12:22:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-10-30T12:22:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a4b60 0x7f54b41326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient(hunting): picked mon.noname-b con 0x7f54b40a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f54b4099a40 0x7f54b4132be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient(hunting): picked mon.noname-a con 0x7f54b4099a40 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54b40a2350 addr v1:172.21.15.118:6789/0 2023-10-30T12:22:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b4133120 con 0x7f54b40a2350 2023-10-30T12:22:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b40a3d70 con 0x7f54b4099a40 2023-10-30T12:22:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40a4f40 con 0x7f54b40a4b60 2023-10-30T12:22:07.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54bc133640 10 monclient(hunting): authenticate will time out at 2172.737434s 2023-10-30T12:22:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a4b60 0x7f54b41326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f54b4099a40 0x7f54b4132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a4b60 0x7f54b41326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55184/0 (socket says 172.21.15.50:55184) 2023-10-30T12:22:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f54b4099a40 0x7f54b4132be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48266/0 (socket says 172.21.15.50:48266) 2023-10-30T12:22:07.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 1 -- 172.21.15.50:0/3410695000 learned_addr learned my addr 172.21.15.50:0/3410695000 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 10 monclient(hunting): get_auth_request con 0x7f54b40a4b60 auth_method 0 2023-10-30T12:22:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): get_auth_request con 0x7f54b4099a40 auth_method 0 2023-10-30T12:22:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient(hunting): handle_auth_done global_id 264243 payload 274 2023-10-30T12:22:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 1 -- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a4b60 msgr2=0x7f54b41326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:07.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 1 --2- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a4b60 0x7f54b41326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 1 -- 172.21.15.50:0/3410695000 >> v1:172.21.15.118:6789/0 conn(0x7f54b40a2350 legacy=0x7f54b40a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient: _finish_hunting 0 2023-10-30T12:22:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 1 monclient: found mon.noname-a 2023-10-30T12:22:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:07.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b96ce640 1 -- 172.21.15.50:0/3410695000 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b40a4150 con 0x7f54b4099a40 2023-10-30T12:22:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.554+0000 7f54b9ecf640 1 --2- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a4b60 0x7f54b41326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:22:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b96ce640 1 --2- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b4099a40 0x7f54b4132be0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f54a80097a0 tx=0x7f54a802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89db2aba803a225c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 1 -- 172.21.15.50:0/3410695000 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8030710 con 0x7f54b4099a40 2023-10-30T12:22:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: dump: 2023-10-30T12:22:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.580 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.581 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.581 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: _finish_auth 0 2023-10-30T12:22:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: _check_auth_tickets 2023-10-30T12:22:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 1 -- 172.21.15.50:0/3410695000 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a802e050 con 0x7f54b4099a40 2023-10-30T12:22:07.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54bc133640 5 monclient: authenticate success, global_id 264243 2023-10-30T12:22:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 1 -- 172.21.15.50:0/3410695000 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8030a90 con 0x7f54b4099a40 2023-10-30T12:22:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:07.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54b8ecd640 10 monclient: dump: 2023-10-30T12:22:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.584 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.585 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.585 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54bc133640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.555+0000 7f54bc133640 4 set_mon_vals no callback set 2023-10-30T12:22:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 10 monclient: shutdown 2023-10-30T12:22:07.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 -- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b4099a40 msgr2=0x7f54b4132be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 --2- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b4099a40 0x7f54b4132be0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f54a80097a0 tx=0x7f54a802b710 comp rx=0 tx=0).stop 2023-10-30T12:22:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 -- 172.21.15.50:0/3410695000 shutdown_connections 2023-10-30T12:22:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 --2- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b4099a40 0x7f54b4132be0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 --2- 172.21.15.50:0/3410695000 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a4b60 0x7f54b41326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 -- 172.21.15.50:0/3410695000 >> 172.21.15.50:0/3410695000 conn(0x7f54b40a8060 msgr2=0x7f54b40a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 -- 172.21.15.50:0/3410695000 shutdown_connections 2023-10-30T12:22:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.556+0000 7f54bc133640 1 -- 172.21.15.50:0/3410695000 wait complete. 2023-10-30T12:22:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient: build_initial_monmap 2023-10-30T12:22:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient: monmap: 2023-10-30T12:22:07.588 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:07.589 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:07.590 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 librados: starting msgr at 2023-10-30T12:22:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 librados: starting objecter 2023-10-30T12:22:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 Processor -- start 2023-10-30T12:22:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 -- start start 2023-10-30T12:22:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 librados: setting wanted keys 2023-10-30T12:22:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 librados: calling monclient init 2023-10-30T12:22:07.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient: init 2023-10-30T12:22:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:22:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a2350 0x7f54b41ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): picked mon.noname-c con 0x7f54b40a2350 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:07.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54b40a4b60 0x7f54b41cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): picked mon.noname-b con 0x7f54b40a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b411e490 0x7f54b411e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): picked mon.noname-a con 0x7f54b411e490 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4122500 con 0x7f54b411e490 2023-10-30T12:22:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4122290 con 0x7f54b40a4b60 2023-10-30T12:22:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4120d70 con 0x7f54b40a2350 2023-10-30T12:22:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54bc133640 10 monclient(hunting): authenticate will time out at 2172.740863s 2023-10-30T12:22:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54b9ecf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a2350 0x7f54b41ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54b9ecf640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a2350 0x7f54b41ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55198/0 (socket says 172.21.15.50:55198) 2023-10-30T12:22:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54b9ecf640 1 -- 172.21.15.50:0/2779496033 learned_addr learned my addr 172.21.15.50:0/2779496033 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54b9ecf640 10 monclient(hunting): get_auth_request con 0x7f54b40a2350 auth_method 0 2023-10-30T12:22:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54b9ecf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54b9ecf640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54b9ecf640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.557+0000 7f54ba6d0640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b411e490 0x7f54b411e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54ba6d0640 10 monclient(hunting): get_auth_request con 0x7f54b411e490 auth_method 0 2023-10-30T12:22:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54ba6d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54ba6d0640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54ba6d0640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54ba6d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54ba6d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54ba6d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b96ce640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54b40a4b60 0x7f54b41cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b96ce640 10 monclient(hunting): get_auth_request con 0x7f54b40a4b60 auth_method 0 2023-10-30T12:22:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b96ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b96ce640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b96ce640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 10 monclient(hunting): handle_auth_done global_id 254288 payload 824 2023-10-30T12:22:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 1 -- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54b40a4b60 msgr2=0x7f54b41cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54b40a4b60 0x7f54b41cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 1 -- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b411e490 msgr2=0x7f54b411e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b411e490 0x7f54b411e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 10 monclient: _finish_hunting 0 2023-10-30T12:22:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 1 monclient: found mon.noname-c 2023-10-30T12:22:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 1 -- 172.21.15.50:0/2779496033 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac009b30 con 0x7f54b40a2350 2023-10-30T12:22:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b9ecf640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a2350 0x7f54b41ca8c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00efc0 tx=0x7f54ac00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 client.?.objecter ms_handle_connect 0x7f54b40a2350 2023-10-30T12:22:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 1 -- 172.21.15.50:0/2779496033 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac00ce60 con 0x7f54b40a2350 2023-10-30T12:22:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:22:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: dump: 2023-10-30T12:22:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.604 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.605 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: _finish_auth 0 2023-10-30T12:22:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: _check_auth_tickets 2023-10-30T12:22:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 1 -- 172.21.15.50:0/2779496033 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac00f040 con 0x7f54b40a2350 2023-10-30T12:22:07.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 1 -- 172.21.15.50:0/2779496033 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac014680 con 0x7f54b40a2350 2023-10-30T12:22:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b27fc640 10 monclient: dump: 2023-10-30T12:22:07.606 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.607 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.608 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.608 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 5 monclient: authenticate success, global_id 254288 2023-10-30T12:22:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 10 monclient: _renew_subs 2023-10-30T12:22:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a8009400 con 0x7f54b40a2350 2023-10-30T12:22:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54b3fff640 4 set_mon_vals no callback set 2023-10-30T12:22:07.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 10 client.254288.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 10 monclient: _renew_subs 2023-10-30T12:22:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b4121c60 con 0x7f54b40a2350 2023-10-30T12:22:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.558+0000 7f54bc133640 1 librados: init done 2023-10-30T12:22:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b27fc640 1 -- 172.21.15.50:0/2779496033 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f54ac0147e0 con 0x7f54b40a2350 2023-10-30T12:22:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b27fc640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a00392f0 0x7f54a003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54bc133640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54bc133640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54bc133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b40a1930 con 0x7f54b40a2350 2023-10-30T12:22:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b27fc640 1 -- 172.21.15.50:0/2779496033 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(668..668 src has 520..668) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f54ac04b660 con 0x7f54b40a2350 2023-10-30T12:22:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b27fc640 10 client.254288.objecter ms_dispatch 0x7f54b4000b90 osd_map(668..668 src has 520..668) v4 2023-10-30T12:22:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b27fc640 3 client.254288.objecter handle_osd_map got epochs [668,668] > 0 2023-10-30T12:22:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b27fc640 3 client.254288.objecter handle_osd_map decoding full epoch 668 2023-10-30T12:22:07.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b27fc640 20 client.254288.objecter dump_active .. 0 homeless 2023-10-30T12:22:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b96ce640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a00392f0 0x7f54a003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.559+0000 7f54b96ce640 10 monclient: get_auth_request con 0x7f54a00392f0 auth_method 0 2023-10-30T12:22:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.560+0000 7f54b27fc640 10 client.254288.objecter ms_handle_connect 0x7f54a00392f0 2023-10-30T12:22:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.560+0000 7f54b96ce640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a00392f0 0x7f54a003b7b0 secure :-1 s=READY pgs=4340 cs=0 l=1 rev1=1 crypto rx=0x7f54a8034000 tx=0x7f54a802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:07.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.564+0000 7f54b27fc640 1 -- 172.21.15.50:0/2779496033 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f54ac01b310 con 0x7f54b40a2350 2023-10-30T12:22:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.564+0000 7f54b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.564+0000 7f54b27fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:07.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.811+0000 7f54bc133640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:22:07.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.811+0000 7f54bc133640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:22:07.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.811+0000 7f54bc133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.811+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f54b404e160 con 0x7f54b40a2350 2023-10-30T12:22:07.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.811+0000 7f54b27fc640 1 -- 172.21.15.50:0/2779496033 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f54ac012b30 con 0x7f54b40a2350 2023-10-30T12:22:07.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.811+0000 7f54b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:22:07.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.811+0000 7f54b27fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:07.816 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:22:07.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.813+0000 7f54bc133640 10 librados: watch_flush enter 2023-10-30T12:22:07.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.813+0000 7f54bc133640 10 librados: watch_flush exit 2023-10-30T12:22:07.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.813+0000 7f54bc133640 20 client.254288.objecter shutdown clearing up homeless session... 2023-10-30T12:22:07.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.813+0000 7f54bc133640 10 client.254288.objecter successfully canceled tick 2023-10-30T12:22:07.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a00392f0 msgr2=0x7f54a003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a00392f0 0x7f54a003b7b0 secure :-1 s=READY pgs=4340 cs=0 l=1 rev1=1 crypto rx=0x7f54a8034000 tx=0x7f54a802f000 comp rx=0 tx=0).stop 2023-10-30T12:22:07.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 10 monclient: shutdown 2023-10-30T12:22:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a2350 msgr2=0x7f54b41ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a2350 0x7f54b41ca8c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00efc0 tx=0x7f54ac00c490 comp rx=0 tx=0).stop 2023-10-30T12:22:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 shutdown_connections 2023-10-30T12:22:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f54a00392f0 0x7f54a003b7b0 unknown :-1 s=CLOSED pgs=4340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b411e490 0x7f54b411e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54b40a4b60 0x7f54b41cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 --2- 172.21.15.50:0/2779496033 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f54b40a2350 0x7f54b41ca8c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 >> 172.21.15.50:0/2779496033 conn(0x7f54b40a8060 msgr2=0x7f54b4134160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 shutdown_connections 2023-10-30T12:22:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 -- 172.21.15.50:0/2779496033 wait complete. 2023-10-30T12:22:07.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.814+0000 7f54bc133640 1 librados: shutdown 2023-10-30T12:22:07.826 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:22:07.827 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-10-30T12:22:07.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:22:07.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:22:07.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.932+0000 7fc4603a9640 10 monclient: get_monmap_and_config 2023-10-30T12:22:07.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.932+0000 7fc4603a9640 10 monclient: build_initial_monmap 2023-10-30T12:22:07.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.932+0000 7fc4603a9640 10 monclient: monmap: 2023-10-30T12:22:07.934 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:07.934 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.935 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:07.933464+0000 2023-10-30T12:22:07.935 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:07.933464+0000 2023-10-30T12:22:07.935 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:07.935 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.935 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:07.936 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:07.936 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:07.936 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:07.936 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:07.936 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.932+0000 7fc4603a9640 1 Processor -- start 2023-10-30T12:22:07.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 1 -- start start 2023-10-30T12:22:07.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient: init 2023-10-30T12:22:07.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-10-30T12:22:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc458097fe0 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc45809a150 0x7fc458071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient(hunting): picked mon.noname-a con 0x7fc45809a150 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc458137850 addr v1:172.21.15.118:6789/0 2023-10-30T12:22:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc458072140 con 0x7fc458097fe0 2023-10-30T12:22:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45809a580 con 0x7fc458137850 2023-10-30T12:22:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45809faa0 con 0x7fc45809a150 2023-10-30T12:22:07.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc4603a9640 10 monclient(hunting): authenticate will time out at 2173.116616s 2023-10-30T12:22:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc45809a150 0x7fc458071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45e145640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc458097fe0 0x7fc45809f390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36540/0 (socket says 172.21.15.50:36540) 2023-10-30T12:22:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 10 monclient(hunting): get_auth_request con 0x7fc45809a150 auth_method 0 2023-10-30T12:22:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45e145640 1 -- 172.21.15.50:0/277770770 learned_addr learned my addr 172.21.15.50:0/277770770 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:22:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.933+0000 7fc45d944640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 10 monclient(hunting): handle_auth_done global_id 264249 payload 274 2023-10-30T12:22:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 1 -- 172.21.15.50:0/277770770 >> v1:172.21.15.118:6789/0 conn(0x7fc458137850 legacy=0x7fc458139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 1 -- 172.21.15.50:0/277770770 >> v1:172.21.15.50:6789/0 conn(0x7fc458097fe0 legacy=0x7fc45809f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 10 monclient: _finish_hunting 0 2023-10-30T12:22:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 1 monclient: found mon.noname-a 2023-10-30T12:22:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 1 -- 172.21.15.50:0/277770770 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc458097600 con 0x7fc45809a150 2023-10-30T12:22:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d944640 1 --2- 172.21.15.50:0/277770770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45809a150 0x7fc458071c00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4480097a0 tx=0x7fc44802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c11e7aae6ba28ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 1 -- 172.21.15.50:0/277770770 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44802e070 con 0x7fc45809a150 2023-10-30T12:22:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: dump: 2023-10-30T12:22:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.953 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.953 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.953 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.953 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.953 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.954 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: _finish_auth 0 2023-10-30T12:22:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: _check_auth_tickets 2023-10-30T12:22:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 1 -- 172.21.15.50:0/277770770 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4480315a0 con 0x7fc45809a150 2023-10-30T12:22:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 1 -- 172.21.15.50:0/277770770 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448031930 con 0x7fc45809a150 2023-10-30T12:22:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc4603a9640 5 monclient: authenticate success, global_id 264249 2023-10-30T12:22:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc4603a9640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc4603a9640 4 set_mon_vals no callback set 2023-10-30T12:22:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.934+0000 7fc45d143640 10 monclient: dump: 2023-10-30T12:22:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.957 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.957 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.957 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.957 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.957 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.957 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.958 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.958 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 10 monclient: shutdown 2023-10-30T12:22:07.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:07.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 1 -- 172.21.15.50:0/277770770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45809a150 msgr2=0x7fc458071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 1 --2- 172.21.15.50:0/277770770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45809a150 0x7fc458071c00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4480097a0 tx=0x7fc44802b840 comp rx=0 tx=0).stop 2023-10-30T12:22:07.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 1 -- 172.21.15.50:0/277770770 shutdown_connections 2023-10-30T12:22:07.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 1 --2- 172.21.15.50:0/277770770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45809a150 0x7fc458071c00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 1 -- 172.21.15.50:0/277770770 >> 172.21.15.50:0/277770770 conn(0x7fc4580a3980 msgr2=0x7fc4580a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:07.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 1 -- 172.21.15.50:0/277770770 shutdown_connections 2023-10-30T12:22:07.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.935+0000 7fc4603a9640 1 -- 172.21.15.50:0/277770770 wait complete. 2023-10-30T12:22:07.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient: build_initial_monmap 2023-10-30T12:22:07.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient: monmap: 2023-10-30T12:22:07.960 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:07.960 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:07.961 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:07.961 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:07.961 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:07.962 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.962 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:07.962 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:07.962 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:07.963 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 librados: starting msgr at 2023-10-30T12:22:07.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 librados: starting objecter 2023-10-30T12:22:07.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 Processor -- start 2023-10-30T12:22:07.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 -- start start 2023-10-30T12:22:07.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 librados: setting wanted keys 2023-10-30T12:22:07.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 librados: calling monclient init 2023-10-30T12:22:07.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient: init 2023-10-30T12:22:07.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:07.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:22:07.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc458097fe0 0x7fc4581ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient(hunting): picked mon.noname-a con 0x7fc458097fe0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:07.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc45809a150 0x7fc458096540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient(hunting): picked mon.noname-c con 0x7fc45809a150 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:07.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc458137850 0x7fc458096a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient(hunting): picked mon.noname-b con 0x7fc458137850 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:07.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458090800 con 0x7fc458097fe0 2023-10-30T12:22:07.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458090520 con 0x7fc458137850 2023-10-30T12:22:07.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.936+0000 7fc4603a9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc458098f80 con 0x7fc45809a150 2023-10-30T12:22:07.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:07.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 10 monclient(hunting): authenticate will time out at 2173.120024s 2023-10-30T12:22:07.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc458097fe0 0x7fc4581ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc45809a150 0x7fc458096540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc458097fe0 0x7fc4581ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34566/0 (socket says 172.21.15.50:34566) 2023-10-30T12:22:07.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 1 -- 172.21.15.50:0/3545499429 learned_addr learned my addr 172.21.15.50:0/3545499429 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:07.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 10 monclient(hunting): get_auth_request con 0x7fc458097fe0 auth_method 0 2023-10-30T12:22:07.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): get_auth_request con 0x7fc45809a150 auth_method 0 2023-10-30T12:22:07.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e946640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc458137850 0x7fc458096a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e946640 10 monclient(hunting): get_auth_request con 0x7fc458137850 auth_method 0 2023-10-30T12:22:07.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e946640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e946640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient(hunting): handle_auth_done global_id 254291 payload 824 2023-10-30T12:22:07.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc458137850 msgr2=0x7fc458096a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc458137850 0x7fc458096a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc458097fe0 msgr2=0x7fc4581ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc458097fe0 0x7fc4581ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:07.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient: _finish_hunting 0 2023-10-30T12:22:07.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 monclient: found mon.noname-c 2023-10-30T12:22:07.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:07.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 -- 172.21.15.50:0/3545499429 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448009400 con 0x7fc45809a150 2023-10-30T12:22:07.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45e145640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc458097fe0 0x7fc4581ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:22:07.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45d944640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc45809a150 0x7fc458096540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc45806afe0 tx=0x7fc4480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:07.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 client.?.objecter ms_handle_connect 0x7fc45809a150 2023-10-30T12:22:07.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:07.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448032980 con 0x7fc45809a150 2023-10-30T12:22:07.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:22:07.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: dump: 2023-10-30T12:22:07.975 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.977 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: _finish_auth 0 2023-10-30T12:22:07.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: _check_auth_tickets 2023-10-30T12:22:07.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:07.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448032b00 con 0x7fc45809a150 2023-10-30T12:22:07.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 5 monclient: authenticate success, global_id 254291 2023-10-30T12:22:07.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 10 monclient: _renew_subs 2023-10-30T12:22:07.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45809cbc0 con 0x7fc45809a150 2023-10-30T12:22:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448045400 con 0x7fc45809a150 2023-10-30T12:22:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc44effd640 10 monclient: dump: 2023-10-30T12:22:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.981 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc45c942640 4 set_mon_vals no callback set 2023-10-30T12:22:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 10 client.254291.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 10 monclient: _renew_subs 2023-10-30T12:22:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc458090dc0 con 0x7fc45809a150 2023-10-30T12:22:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.937+0000 7fc4603a9640 1 librados: init done 2023-10-30T12:22:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc4603a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc44804e430 con 0x7fc45809a150 2023-10-30T12:22:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc4603a9640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808ce70 0x7fc45808d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc4603a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458090c80 con 0x7fc45808ce70 2023-10-30T12:22:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808ce70 0x7fc45808d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient: get_auth_request con 0x7fc45808ce70 auth_method 0 2023-10-30T12:22:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc44effd640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc43c0392f0 0x7fc43c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:07.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:07.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(668..668 src has 520..668) v4 ==== 6670+0+0 (secure 0 0 0) 0x7fc44806ce40 con 0x7fc45809a150 2023-10-30T12:22:07.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc44effd640 10 client.254291.objecter ms_dispatch 0x7fc458000b90 osd_map(668..668 src has 520..668) v4 2023-10-30T12:22:07.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc44effd640 3 client.254291.objecter handle_osd_map got epochs [668,668] > 0 2023-10-30T12:22:07.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc44effd640 3 client.254291.objecter handle_osd_map decoding full epoch 668 2023-10-30T12:22:07.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 10 monclient(hunting): handle_auth_done global_id 264258 payload 824 2023-10-30T12:22:07.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e145640 1 -- 172.21.15.50:0/3545499429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc45809d9b0 con 0x7fc45808ce70 2023-10-30T12:22:07.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc44effd640 20 client.254291.objecter dump_active .. 0 homeless 2023-10-30T12:22:07.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e946640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc43c0392f0 0x7fc43c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.938+0000 7fc45e946640 10 monclient: get_auth_request con 0x7fc43c0392f0 auth_method 0 2023-10-30T12:22:07.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 client.254291.objecter ms_handle_connect 0x7fc45808ce70 2023-10-30T12:22:07.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 client.254291.objecter resend_mon_ops 2023-10-30T12:22:07.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc45e145640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808ce70 0x7fc45808d250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc454001000 tx=0x7fc45400ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:07.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4540072e0 con 0x7fc45808ce70 2023-10-30T12:22:07.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:07.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:07.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 monclient: dump: 2023-10-30T12:22:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:07.991 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:07.991 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:07.991 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:07.993 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:07.993 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:07.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 client.254291.objecter ms_handle_connect 0x7fc43c0392f0 2023-10-30T12:22:07.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc45e946640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc43c0392f0 0x7fc43c03b7b0 secure :-1 s=READY pgs=4341 cs=0 l=1 rev1=1 crypto rx=0x7fc450007fd0 tx=0x7fc450009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:07.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fc454007440 con 0x7fc45808ce70 2023-10-30T12:22:07.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:07.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:07.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808ce70 msgr2=0x7fc45808d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:07.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808ce70 0x7fc45808d250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc454001000 tx=0x7fc45400ff00 comp rx=0 tx=0).stop 2023-10-30T12:22:07.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808ce70 msgr2=0x7fc45808d250 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:07.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.939+0000 7fc44effd640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808ce70 0x7fc45808d250 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc454001000 tx=0x7fc45400ff00 comp rx=0 tx=0).stop 2023-10-30T12:22:07.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc4603a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:07.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc4603a9640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808d790 0x7fc458138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:07.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc4603a9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:07.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458073ff0 con 0x7fc45808d790 2023-10-30T12:22:07.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808d790 0x7fc458138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:07.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 10 monclient: get_auth_request con 0x7fc45808d790 auth_method 0 2023-10-30T12:22:07.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:07.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:07.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:07.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:07.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.945+0000 7fc45e145640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc45e145640 10 monclient(hunting): handle_auth_done global_id 264261 payload 824 2023-10-30T12:22:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc45e145640 1 -- 172.21.15.50:0/3545499429 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc45809bf50 con 0x7fc45808d790 2023-10-30T12:22:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 10 client.254291.objecter ms_handle_connect 0x7fc45808d790 2023-10-30T12:22:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 10 client.254291.objecter resend_mon_ops 2023-10-30T12:22:08.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc45e145640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808d790 0x7fc458138c40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc454001000 tx=0x7fc454001050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454024070 con 0x7fc45808d790 2023-10-30T12:22:08.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:08.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 10 monclient: dump: 2023-10-30T12:22:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.003 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4634 (secure 0 0 0) 0x7fc454005d70 con 0x7fc45808d790 2023-10-30T12:22:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808d790 msgr2=0x7fc458138c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808d790 0x7fc458138c40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc454001000 tx=0x7fc454001050 comp rx=0 tx=0).stop 2023-10-30T12:22:08.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808d790 msgr2=0x7fc458138c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:08.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc44effd640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45808d790 0x7fc458138c40 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc454001000 tx=0x7fc454001050 comp rx=0 tx=0).stop 2023-10-30T12:22:08.005 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:08.006 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:08.006 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:08.006 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:22:08.006 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 218, 2023-10-30T12:22:08.006 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:08.007 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:08.007 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:22:08.007 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:22:08.007 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.008 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 2, 2023-10-30T12:22:08.008 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:08.008 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:08.008 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:08.008 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.009 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.009 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.009 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.009 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.010 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.010 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.010 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.010 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.011 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.011 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:08.011 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:08.011 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.011 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.012 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.012 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.012 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.012 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.013 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.013 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.013 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.013 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.014 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:08.014 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:08.014 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:08.014 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:08.015 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:08.015 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:08.015 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:08.015 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:08.015 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:08.016 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:08.016 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:08.016 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:08.016 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:08.016 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:08.017 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:08.017 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:08.017 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:08.017 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.018 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.018 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.018 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.019 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.019 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.019 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.019 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.019 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.020 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.020 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:08.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:08.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.021 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:08.022 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:08.022 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.022 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.022 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.023 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:08.023 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.024 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:08.024 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.024 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.024 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.025 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:08.025 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:08.025 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.025 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.026 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.026 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:08.027 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:08.027 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.027 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.027 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.028 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:08.028 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.028 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.029 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:08.029 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.029 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.030 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:08.030 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:08.030 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.030 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.030 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.031 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:08.031 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:08.032 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.032 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.032 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:08.033 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.033 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.033 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:08.034 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.034 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:08.035 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:08.035 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.035 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.035 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.036 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.036 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.036 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:08.036 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:08.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.037 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.037 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.037 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:08.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.038 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:22:08.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.038 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.039 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.039 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2023-10-30T12:22:08.039 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.039 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.040 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:08.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.040 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.040 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.040 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:22:08.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.041 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.041 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:08.041 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:08.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 10 librados: watch_flush enter 2023-10-30T12:22:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 10 librados: watch_flush exit 2023-10-30T12:22:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 20 client.254291.objecter shutdown clearing up homeless session... 2023-10-30T12:22:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 10 client.254291.objecter successfully canceled tick 2023-10-30T12:22:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc43c0392f0 msgr2=0x7fc43c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc43c0392f0 0x7fc43c03b7b0 secure :-1 s=READY pgs=4341 cs=0 l=1 rev1=1 crypto rx=0x7fc450007fd0 tx=0x7fc450009040 comp rx=0 tx=0).stop 2023-10-30T12:22:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 10 monclient: shutdown 2023-10-30T12:22:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc45809a150 msgr2=0x7fc458096540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 1 --2- 172.21.15.50:0/3545499429 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc45809a150 0x7fc458096540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc45806afe0 tx=0x7fc4480046c0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc45e946640 1 -- 172.21.15.50:0/3545499429 reap_dead start 2023-10-30T12:22:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc45e946640 1 -- 172.21.15.50:0/3545499429 reap_dead start 2023-10-30T12:22:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 shutdown_connections 2023-10-30T12:22:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.946+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 >> 172.21.15.50:0/3545499429 conn(0x7fc4580a3980 msgr2=0x7fc458136970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.947+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 shutdown_connections 2023-10-30T12:22:08.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.947+0000 7fc4603a9640 1 -- 172.21.15.50:0/3545499429 wait complete. 2023-10-30T12:22:08.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:07.947+0000 7fc4603a9640 1 librados: shutdown 2023-10-30T12:22:08.047 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:22:08.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:22:08.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:22:08.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.161+0000 7f1ebaea9640 10 monclient: get_monmap_and_config 2023-10-30T12:22:08.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.161+0000 7f1ebaea9640 10 monclient: build_initial_monmap 2023-10-30T12:22:08.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.162+0000 7f1ebaea9640 10 monclient: monmap: 2023-10-30T12:22:08.164 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:08.164 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.164 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:08.163044+0000 2023-10-30T12:22:08.165 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:08.163044+0000 2023-10-30T12:22:08.165 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:08.165 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.165 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:08.166 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:08.166 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:08.166 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:08.166 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:08.166 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.162+0000 7f1ebaea9640 1 Processor -- start 2023-10-30T12:22:08.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.162+0000 7f1ebaea9640 1 -- start start 2023-10-30T12:22:08.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.162+0000 7f1ebaea9640 10 monclient: init 2023-10-30T12:22:08.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:08.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-10-30T12:22:08.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1eb406afc0 0x7f1eb40a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb406afc0 addr v2:172.21.15.118:3300/0 2023-10-30T12:22:08.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1eb406b640 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:08.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb413b070 0x7f1eb413d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb413b070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:08.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb406b3a0 con 0x7f1eb406b640 2023-10-30T12:22:08.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb409e540 con 0x7f1eb406afc0 2023-10-30T12:22:08.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb409fa10 con 0x7f1eb413b070 2023-10-30T12:22:08.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:08.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1ebaea9640 10 monclient(hunting): authenticate will time out at 2173.346207s 2023-10-30T12:22:08.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb413b070 0x7f1eb413d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb3fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1eb406b640 0x7f1eb40a21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36556/0 (socket says 172.21.15.50:36556) 2023-10-30T12:22:08.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb3fff640 1 -- 172.21.15.50:0/3199769410 learned_addr learned my addr 172.21.15.50:0/3199769410 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:22:08.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb413b070 0x7f1eb413d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34824/0 (socket says 172.21.15.50:34824) 2023-10-30T12:22:08.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): get_auth_request con 0x7f1eb413b070 auth_method 0 2023-10-30T12:22:08.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 1 --2- 172.21.15.50:0/3199769410 >> v2:172.21.15.118:3300/0 conn(0x7f1eb406afc0 0x7f1eb40a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 10 monclient(hunting): get_auth_request con 0x7f1eb406afc0 auth_method 0 2023-10-30T12:22:08.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb37fe640 1 -- 172.21.15.50:0/3199769410 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290009252 0 0) 0x7f1eb406b3a0 con 0x7f1eb406b640 2023-10-30T12:22:08.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb37fe640 10 monclient(hunting): _negotiate 2023-10-30T12:22:08.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb37fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb37fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb37fe640 10 monclient(hunting): my global_id is 264264 2023-10-30T12:22:08.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb37fe640 1 -- 172.21.15.50:0/3199769410 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0003520 con 0x7f1eb406b640 2023-10-30T12:22:08.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb8c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient(hunting): handle_auth_done global_id 254294 payload 274 2023-10-30T12:22:08.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 1 -- 172.21.15.50:0/3199769410 >> v2:172.21.15.118:3300/0 conn(0x7f1eb406afc0 msgr2=0x7f1eb40a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:08.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 1 --2- 172.21.15.50:0/3199769410 >> v2:172.21.15.118:3300/0 conn(0x7f1eb406afc0 0x7f1eb40a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 1 -- 172.21.15.50:0/3199769410 >> v1:172.21.15.50:6789/0 conn(0x7f1eb406b640 legacy=0x7f1eb40a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient: _finish_hunting 0 2023-10-30T12:22:08.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 1 monclient: found mon.noname-b 2023-10-30T12:22:08.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:08.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.163+0000 7f1eb9446640 1 -- 172.21.15.50:0/3199769410 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb406a6f0 con 0x7f1eb413b070 2023-10-30T12:22:08.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb9446640 1 --2- 172.21.15.50:0/3199769410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb413b070 0x7f1eb413d460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40097a0 tx=0x7f1ea402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fdb9d14d6ddebdf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 1 -- 172.21.15.50:0/3199769410 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea402f050 con 0x7f1eb413b070 2023-10-30T12:22:08.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:22:08.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: dump: 2023-10-30T12:22:08.187 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.188 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.188 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.188 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.188 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.188 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.189 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.189 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.189 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.189 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: _finish_auth 0 2023-10-30T12:22:08.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: _check_auth_tickets 2023-10-30T12:22:08.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:08.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 1 -- 172.21.15.50:0/3199769410 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea4031700 con 0x7f1eb413b070 2023-10-30T12:22:08.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1ebaea9640 5 monclient: authenticate success, global_id 254294 2023-10-30T12:22:08.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1ebaea9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:08.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:08.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1ebaea9640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:08.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 1 -- 172.21.15.50:0/3199769410 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4031a90 con 0x7f1eb413b070 2023-10-30T12:22:08.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1ebaea9640 4 set_mon_vals no callback set 2023-10-30T12:22:08.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:08.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.164+0000 7f1eb37fe640 10 monclient: dump: 2023-10-30T12:22:08.192 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.192 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.193 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.193 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.193 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.193 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.193 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.194 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.194 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.194 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 10 monclient: shutdown 2023-10-30T12:22:08.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:08.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 -- 172.21.15.50:0/3199769410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb413b070 msgr2=0x7f1eb413d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 --2- 172.21.15.50:0/3199769410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb413b070 0x7f1eb413d460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40097a0 tx=0x7f1ea402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 -- 172.21.15.50:0/3199769410 shutdown_connections 2023-10-30T12:22:08.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 --2- 172.21.15.50:0/3199769410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb413b070 0x7f1eb413d460 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 --2- 172.21.15.50:0/3199769410 >> v2:172.21.15.118:3300/0 conn(0x7f1eb406afc0 0x7f1eb40a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 -- 172.21.15.50:0/3199769410 >> 172.21.15.50:0/3199769410 conn(0x7f1eb40a6200 msgr2=0x7f1eb40a0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:08.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 -- 172.21.15.50:0/3199769410 shutdown_connections 2023-10-30T12:22:08.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 -- 172.21.15.50:0/3199769410 wait complete. 2023-10-30T12:22:08.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 10 monclient: build_initial_monmap 2023-10-30T12:22:08.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 10 monclient: monmap: 2023-10-30T12:22:08.197 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:08.197 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:08.197 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:08.198 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:08.198 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:08.198 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.198 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:08.198 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:08.199 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:08.199 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 librados: starting msgr at 2023-10-30T12:22:08.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.165+0000 7f1ebaea9640 1 librados: starting objecter 2023-10-30T12:22:08.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 Processor -- start 2023-10-30T12:22:08.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 -- start start 2023-10-30T12:22:08.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 librados: setting wanted keys 2023-10-30T12:22:08.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 librados: calling monclient init 2023-10-30T12:22:08.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient: init 2023-10-30T12:22:08.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:08.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:22:08.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb406afc0 0x7f1eb411e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb406afc0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:08.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb406b640 0x7f1eb411eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb406b640 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:08.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb413b070 0x7f1eb411f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb413b070 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:08.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4122770 con 0x7f1eb413b070 2023-10-30T12:22:08.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4070de0 con 0x7f1eb406b640 2023-10-30T12:22:08.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4122350 con 0x7f1eb406afc0 2023-10-30T12:22:08.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:08.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1ebaea9640 10 monclient(hunting): authenticate will time out at 2173.349705s 2023-10-30T12:22:08.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb406afc0 0x7f1eb411e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb413b070 0x7f1eb411f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb406afc0 0x7f1eb411e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34828/0 (socket says 172.21.15.50:34828) 2023-10-30T12:22:08.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 1 -- 172.21.15.50:0/1987705526 learned_addr learned my addr 172.21.15.50:0/1987705526 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:08.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 10 monclient(hunting): get_auth_request con 0x7f1eb406afc0 auth_method 0 2023-10-30T12:22:08.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 10 monclient(hunting): get_auth_request con 0x7f1eb413b070 auth_method 0 2023-10-30T12:22:08.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb8c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb3fff640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb406b640 0x7f1eb411eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb9446640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb3fff640 10 monclient(hunting): get_auth_request con 0x7f1eb406b640 auth_method 0 2023-10-30T12:22:08.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.166+0000 7f1eb3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 10 monclient(hunting): handle_auth_done global_id 254297 payload 824 2023-10-30T12:22:08.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb406b640 msgr2=0x7f1eb411eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1eb406b640 0x7f1eb411eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb413b070 msgr2=0x7f1eb411f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb413b070 0x7f1eb411f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 10 monclient: _finish_hunting 0 2023-10-30T12:22:08.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 1 monclient: found mon.noname-c 2023-10-30T12:22:08.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:08.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 1 -- 172.21.15.50:0/1987705526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4009400 con 0x7f1eb406afc0 2023-10-30T12:22:08.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb8c45640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb406afc0 0x7f1eb411e5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009db0 tx=0x7f1eac00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 client.?.objecter ms_handle_connect 0x7f1eb406afc0 2023-10-30T12:22:08.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:08.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac014480 con 0x7f1eb406afc0 2023-10-30T12:22:08.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:22:08.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: dump: 2023-10-30T12:22:08.213 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.213 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.213 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.214 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.214 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.214 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.214 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.214 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.215 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: _finish_auth 0 2023-10-30T12:22:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: _check_auth_tickets 2023-10-30T12:22:08.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:08.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00f040 con 0x7f1eb406afc0 2023-10-30T12:22:08.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 5 monclient: authenticate success, global_id 254297 2023-10-30T12:22:08.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:08.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac014800 con 0x7f1eb406afc0 2023-10-30T12:22:08.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:08.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb17fa640 10 monclient: dump: 2023-10-30T12:22:08.217 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.217 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.218 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.219 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 10 monclient: _renew_subs 2023-10-30T12:22:08.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1eb2ffd640 4 set_mon_vals no callback set 2023-10-30T12:22:08.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb41382f0 con 0x7f1eb406afc0 2023-10-30T12:22:08.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 10 client.254297.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:08.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 10 monclient: _renew_subs 2023-10-30T12:22:08.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb411ff80 con 0x7f1eb406afc0 2023-10-30T12:22:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.167+0000 7f1ebaea9640 1 librados: init done 2023-10-30T12:22:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1eac014960 con 0x7f1eb406afc0 2023-10-30T12:22:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1ebaea9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1ebaea9640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138480 0x7f1eb4138860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1ebaea9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4073ff0 con 0x7f1eb4138480 2023-10-30T12:22:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138480 0x7f1eb4138860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient: get_auth_request con 0x7f1eb4138480 auth_method 0 2023-10-30T12:22:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e940392f0 0x7f1e9403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(669..669 src has 520..669) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f1eac04b860 con 0x7f1eb406afc0 2023-10-30T12:22:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 10 client.254297.objecter ms_dispatch 0x7f1eb4000b90 osd_map(669..669 src has 520..669) v4 2023-10-30T12:22:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 3 client.254297.objecter handle_osd_map got epochs [669,669] > 0 2023-10-30T12:22:08.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 3 client.254297.objecter handle_osd_map decoding full epoch 669 2023-10-30T12:22:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 20 client.254297.objecter dump_active .. 0 homeless 2023-10-30T12:22:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_done global_id 254300 payload 824 2023-10-30T12:22:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb9446640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e940392f0 0x7f1e9403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 1 -- 172.21.15.50:0/1987705526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1eb409f0b0 con 0x7f1eb4138480 2023-10-30T12:22:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb9446640 10 monclient: get_auth_request con 0x7f1e940392f0 auth_method 0 2023-10-30T12:22:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 10 client.254297.objecter ms_handle_connect 0x7f1eb4138480 2023-10-30T12:22:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb17fa640 10 client.254297.objecter resend_mon_ops 2023-10-30T12:22:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.168+0000 7f1eb3fff640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138480 0x7f1eb4138860 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c005880 tx=0x7f1e9c00fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0058b0 con 0x7f1eb4138480 2023-10-30T12:22:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 10 monclient: dump: 2023-10-30T12:22:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.229 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 10 client.254297.objecter ms_handle_connect 0x7f1e940392f0 2023-10-30T12:22:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb9446640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e940392f0 0x7f1e9403b7b0 secure :-1 s=READY pgs=4342 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4035000 tx=0x7f1ea4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f1e9c005a10 con 0x7f1eb4138480 2023-10-30T12:22:08.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:08.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:08.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138480 msgr2=0x7f1eb4138860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138480 0x7f1eb4138860 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c005880 tx=0x7f1e9c00fc00 comp rx=0 tx=0).stop 2023-10-30T12:22:08.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138480 msgr2=0x7f1eb4138860 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:08.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.169+0000 7f1eb17fa640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138480 0x7f1eb4138860 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c005880 tx=0x7f1e9c00fc00 comp rx=0 tx=0).stop 2023-10-30T12:22:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1ebaea9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1ebaea9640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138da0 0x7f1eb41d4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1ebaea9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb409df80 con 0x7f1eb4138da0 2023-10-30T12:22:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138da0 0x7f1eb41d4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient: get_auth_request con 0x7f1eb4138da0 auth_method 0 2023-10-30T12:22:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.175+0000 7f1eb3fff640 10 monclient(hunting): handle_auth_done global_id 254303 payload 824 2023-10-30T12:22:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb3fff640 1 -- 172.21.15.50:0/1987705526 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1eb406aa80 con 0x7f1eb4138da0 2023-10-30T12:22:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 10 client.254297.objecter ms_handle_connect 0x7f1eb4138da0 2023-10-30T12:22:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 10 client.254297.objecter resend_mon_ops 2023-10-30T12:22:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb3fff640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138da0 0x7f1eb41d4250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c005880 tx=0x7f1e9c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c024070 con 0x7f1eb4138da0 2023-10-30T12:22:08.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:08.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 10 monclient: dump: 2023-10-30T12:22:08.237 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.237 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.239 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4802 (secure 0 0 0) 0x7f1e9c00da70 con 0x7f1eb4138da0 2023-10-30T12:22:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138da0 msgr2=0x7f1eb41d4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138da0 0x7f1eb41d4250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c005880 tx=0x7f1e9c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138da0 msgr2=0x7f1eb41d4250 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb17fa640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb4138da0 0x7f1eb41d4250 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c005880 tx=0x7f1e9c0057f0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.241 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:08.241 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:08.241 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:08.242 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:22:08.242 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 218, 2023-10-30T12:22:08.242 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:08.242 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:08.242 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:22:08.243 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:22:08.243 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.243 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:22:08.243 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:08.243 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:08.244 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:08.244 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.244 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.244 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.244 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.245 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.245 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.245 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.245 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.245 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.245 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.246 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.246 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:08.246 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:08.246 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.246 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.247 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.247 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.247 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.247 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.247 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.248 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.248 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.248 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.249 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:08.249 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:08.249 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:08.249 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:08.249 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:08.250 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:08.250 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:08.250 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:08.250 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:08.251 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:08.251 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:08.251 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:08.251 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:08.251 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:08.252 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:08.252 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:08.252 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:08.252 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.252 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.253 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.253 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.253 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.253 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.253 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.254 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.254 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.254 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.254 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.254 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:08.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.255 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:08.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.255 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:08.255 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:08.256 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.256 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.256 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:08.257 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.257 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.258 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:08.258 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.258 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.259 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:08.259 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:08.259 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.259 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.260 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.260 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:08.261 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:08.261 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.261 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.262 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.262 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:08.262 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.263 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.263 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:08.263 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.264 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.264 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:08.264 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:08.265 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.265 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.265 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.266 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:08.266 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:08.266 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.266 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.267 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.267 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:08.267 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.268 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.268 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:08.269 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.269 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.269 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.270 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:08.270 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:08.270 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.270 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.271 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.271 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:08.272 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:08.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.272 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.272 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.272 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:08.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.273 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.273 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:22:08.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.273 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.274 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.274 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2023-10-30T12:22:08.274 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.274 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.274 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:08.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.275 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.275 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.275 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2023-10-30T12:22:08.275 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.276 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.276 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:22:08.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.276 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.276 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.277 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:08.277 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.277 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.277 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:08.278 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:08.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 10 librados: watch_flush enter 2023-10-30T12:22:08.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 10 librados: watch_flush exit 2023-10-30T12:22:08.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 20 client.254297.objecter shutdown clearing up homeless session... 2023-10-30T12:22:08.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 10 client.254297.objecter successfully canceled tick 2023-10-30T12:22:08.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e940392f0 msgr2=0x7f1e9403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1e940392f0 0x7f1e9403b7b0 secure :-1 s=READY pgs=4342 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4035000 tx=0x7f1ea4008030 comp rx=0 tx=0).stop 2023-10-30T12:22:08.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 10 monclient: shutdown 2023-10-30T12:22:08.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:08.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb406afc0 msgr2=0x7f1eb411e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1ebaea9640 1 --2- 172.21.15.50:0/1987705526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1eb406afc0 0x7f1eb411e5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009db0 tx=0x7f1eac00c490 comp rx=0 tx=0).stop 2023-10-30T12:22:08.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb9446640 1 -- 172.21.15.50:0/1987705526 reap_dead start 2023-10-30T12:22:08.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.176+0000 7f1eb9446640 1 -- 172.21.15.50:0/1987705526 reap_dead start 2023-10-30T12:22:08.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.177+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 shutdown_connections 2023-10-30T12:22:08.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.177+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 >> 172.21.15.50:0/1987705526 conn(0x7f1eb40a6200 msgr2=0x7f1eb40a4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:08.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.177+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 shutdown_connections 2023-10-30T12:22:08.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.177+0000 7f1ebaea9640 1 -- 172.21.15.50:0/1987705526 wait complete. 2023-10-30T12:22:08.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.177+0000 7f1ebaea9640 1 librados: shutdown 2023-10-30T12:22:08.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:22:08.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:22:08.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:22:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.389+0000 7fd707ec9640 10 monclient: get_monmap_and_config 2023-10-30T12:22:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.389+0000 7fd707ec9640 10 monclient: build_initial_monmap 2023-10-30T12:22:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.389+0000 7fd707ec9640 10 monclient: monmap: 2023-10-30T12:22:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:08.390792+0000 2023-10-30T12:22:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:08.390792+0000 2023-10-30T12:22:08.393 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:08.393 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.393 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:08.393 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:08.393 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:08.394 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:08.394 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:08.394 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 1 Processor -- start 2023-10-30T12:22:08.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 1 -- start start 2023-10-30T12:22:08.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient: init 2023-10-30T12:22:08.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:08.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-10-30T12:22:08.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd700132970 0x7fd700132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fd700132970 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7000a03d0 0x7fd7000707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient(hunting): picked mon.noname-b con 0x7fd7000a03d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd700070d10 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7000a17d0 con 0x7fd700070d10 2023-10-30T12:22:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7000a1510 con 0x7fd700132970 2023-10-30T12:22:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70009f680 con 0x7fd7000a03d0 2023-10-30T12:22:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:08.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.390+0000 7fd707ec9640 10 monclient(hunting): authenticate will time out at 2173.573972s 2023-10-30T12:22:08.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7000a03d0 0x7fd7000707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7000a03d0 0x7fd7000707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34844/0 (socket says 172.21.15.50:34844) 2023-10-30T12:22:08.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 -- 172.21.15.50:0/1175084140 learned_addr learned my addr 172.21.15.50:0/1175084140 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:08.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): get_auth_request con 0x7fd7000a03d0 auth_method 0 2023-10-30T12:22:08.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 1 --2- 172.21.15.50:0/1175084140 >> v2:172.21.15.50:3300/0 conn(0x7fd700132970 0x7fd700132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 10 monclient(hunting): get_auth_request con 0x7fd700132970 auth_method 0 2023-10-30T12:22:08.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient(hunting): handle_auth_done global_id 254306 payload 274 2023-10-30T12:22:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 -- 172.21.15.50:0/1175084140 >> v2:172.21.15.50:3300/0 conn(0x7fd700132970 msgr2=0x7fd700132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 --2- 172.21.15.50:0/1175084140 >> v2:172.21.15.50:3300/0 conn(0x7fd700132970 0x7fd700132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 -- 172.21.15.50:0/1175084140 >> v1:172.21.15.50:6789/0 conn(0x7fd700070d10 legacy=0x7fd70013caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient: _finish_hunting 0 2023-10-30T12:22:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 monclient: found mon.noname-b 2023-10-30T12:22:08.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:08.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.391+0000 7fd705464640 1 -- 172.21.15.50:0/1175084140 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70009f960 con 0x7fd7000a03d0 2023-10-30T12:22:08.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd705464640 1 --2- 172.21.15.50:0/1175084140 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7000a03d0 0x7fd7000707d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009870 tx=0x7fd6f002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e37de6ad2f76c94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 1 -- 172.21.15.50:0/1175084140 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f00317c0 con 0x7fd7000a03d0 2023-10-30T12:22:08.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:22:08.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: dump: 2023-10-30T12:22:08.418 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.418 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.419 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.419 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.419 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.419 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.419 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.420 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.420 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.420 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: _finish_auth 0 2023-10-30T12:22:08.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: _check_auth_tickets 2023-10-30T12:22:08.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:08.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 5 monclient: authenticate success, global_id 254306 2023-10-30T12:22:08.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:08.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 1 -- 172.21.15.50:0/1175084140 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f0031920 con 0x7fd7000a03d0 2023-10-30T12:22:08.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:08.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:08.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 1 -- 172.21.15.50:0/1175084140 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0031cb0 con 0x7fd7000a03d0 2023-10-30T12:22:08.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 4 set_mon_vals no callback set 2023-10-30T12:22:08.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:08.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd704c63640 10 monclient: dump: 2023-10-30T12:22:08.423 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.423 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.423 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.424 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.424 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.424 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.424 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.424 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.425 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.425 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 10 monclient: shutdown 2023-10-30T12:22:08.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:08.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 1 -- 172.21.15.50:0/1175084140 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7000a03d0 msgr2=0x7fd7000707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 1 --2- 172.21.15.50:0/1175084140 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7000a03d0 0x7fd7000707d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009870 tx=0x7fd6f002b860 comp rx=0 tx=0).stop 2023-10-30T12:22:08.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 1 -- 172.21.15.50:0/1175084140 shutdown_connections 2023-10-30T12:22:08.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 1 --2- 172.21.15.50:0/1175084140 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7000a03d0 0x7fd7000707d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 1 --2- 172.21.15.50:0/1175084140 >> v2:172.21.15.50:3300/0 conn(0x7fd700132970 0x7fd700132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.392+0000 7fd707ec9640 1 -- 172.21.15.50:0/1175084140 >> 172.21.15.50:0/1175084140 conn(0x7fd7000a4800 msgr2=0x7fd7000a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:08.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 -- 172.21.15.50:0/1175084140 shutdown_connections 2023-10-30T12:22:08.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 -- 172.21.15.50:0/1175084140 wait complete. 2023-10-30T12:22:08.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 10 monclient: build_initial_monmap 2023-10-30T12:22:08.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 10 monclient: monmap: 2023-10-30T12:22:08.427 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:08.428 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:08.428 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:08.428 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:08.428 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:08.428 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.429 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:08.429 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:08.429 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:08.429 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 librados: starting msgr at 2023-10-30T12:22:08.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 librados: starting objecter 2023-10-30T12:22:08.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 Processor -- start 2023-10-30T12:22:08.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 -- start start 2023-10-30T12:22:08.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 librados: setting wanted keys 2023-10-30T12:22:08.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 1 librados: calling monclient init 2023-10-30T12:22:08.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.393+0000 7fd707ec9640 10 monclient: init 2023-10-30T12:22:08.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:08.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:22:08.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd700070d10 0x7fd7001ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): picked mon.noname-c con 0x7fd700070d10 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:08.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700132970 0x7fd70012fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): picked mon.noname-b con 0x7fd700132970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:08.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd70012fff0 0x7fd70011a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fd70012fff0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:08.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7001caf00 con 0x7fd70012fff0 2023-10-30T12:22:08.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70009d7c0 con 0x7fd700132970 2023-10-30T12:22:08.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70011c860 con 0x7fd700070d10 2023-10-30T12:22:08.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:08.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd707ec9640 10 monclient(hunting): authenticate will time out at 2173.577486s 2023-10-30T12:22:08.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd700070d10 0x7fd7001ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd700070d10 0x7fd7001ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34854/0 (socket says 172.21.15.50:34854) 2023-10-30T12:22:08.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 -- 172.21.15.50:0/2576442561 learned_addr learned my addr 172.21.15.50:0/2576442561 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:08.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): get_auth_request con 0x7fd700070d10 auth_method 0 2023-10-30T12:22:08.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd70012fff0 0x7fd70011a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 10 monclient(hunting): get_auth_request con 0x7fd70012fff0 auth_method 0 2023-10-30T12:22:08.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd706466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient(hunting): handle_auth_done global_id 254309 payload 824 2023-10-30T12:22:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700132970 msgr2=0x7fd70012fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700132970 0x7fd70012fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd70012fff0 msgr2=0x7fd70011a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd70012fff0 0x7fd70011a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient: _finish_hunting 0 2023-10-30T12:22:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 monclient: found mon.noname-c 2023-10-30T12:22:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.394+0000 7fd705c65640 1 -- 172.21.15.50:0/2576442561 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f00094d0 con 0x7fd700070d10 2023-10-30T12:22:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd705c65640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd700070d10 0x7fd7001ca250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00e9b0 tx=0x7fd6fc00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 client.?.objecter ms_handle_connect 0x7fd700070d10 2023-10-30T12:22:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc009670 con 0x7fd700070d10 2023-10-30T12:22:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:22:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: dump: 2023-10-30T12:22:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.443 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.443 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.443 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.443 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: _finish_auth 0 2023-10-30T12:22:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:22:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc00c9a0 con 0x7fd700070d10 2023-10-30T12:22:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc00cd50 con 0x7fd700070d10 2023-10-30T12:22:08.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:22:08.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 10 monclient: dump: 2023-10-30T12:22:08.445 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.447 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 5 monclient: authenticate success, global_id 254309 2023-10-30T12:22:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 10 monclient: _renew_subs 2023-10-30T12:22:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70011d9d0 con 0x7fd700070d10 2023-10-30T12:22:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f7fff640 4 set_mon_vals no callback set 2023-10-30T12:22:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 10 client.254309.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 10 monclient: _renew_subs 2023-10-30T12:22:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:22:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70011bf10 con 0x7fd700070d10 2023-10-30T12:22:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 1 librados: init done 2023-10-30T12:22:08.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:08.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd6fc013470 con 0x7fd700070d10 2023-10-30T12:22:08.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7000a03d0 0x7fd70011a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.395+0000 7fd707ec9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd700071a70 con 0x7fd7000a03d0 2023-10-30T12:22:08.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd6e40392f0 0x7fd6e403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(669..669 src has 520..669) v4 ==== 6297+0+0 (secure 0 0 0) 0x7fd6fc04af00 con 0x7fd700070d10 2023-10-30T12:22:08.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 10 client.254309.objecter ms_dispatch 0x7fd700000b90 osd_map(669..669 src has 520..669) v4 2023-10-30T12:22:08.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 3 client.254309.objecter handle_osd_map got epochs [669,669] > 0 2023-10-30T12:22:08.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 3 client.254309.objecter handle_osd_map decoding full epoch 669 2023-10-30T12:22:08.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 20 client.254309.objecter dump_active .. 0 homeless 2023-10-30T12:22:08.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7000a03d0 0x7fd70011a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient: get_auth_request con 0x7fd7000a03d0 auth_method 0 2023-10-30T12:22:08.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd706466640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd6e40392f0 0x7fd6e403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd706466640 10 monclient: get_auth_request con 0x7fd6e40392f0 auth_method 0 2023-10-30T12:22:08.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 10 monclient(hunting): handle_auth_done global_id 274141 payload 824 2023-10-30T12:22:08.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 1 -- 172.21.15.50:0/2576442561 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd700133290 con 0x7fd7000a03d0 2023-10-30T12:22:08.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 10 client.254309.objecter ms_handle_connect 0x7fd6e40392f0 2023-10-30T12:22:08.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd706466640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd6e40392f0 0x7fd6e403b7b0 secure :-1 s=READY pgs=4343 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80012e0 tx=0x7fd6f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 10 client.254309.objecter ms_handle_connect 0x7fd7000a03d0 2023-10-30T12:22:08.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd6f67fc640 10 client.254309.objecter resend_mon_ops 2023-10-30T12:22:08.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.396+0000 7fd705464640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7000a03d0 0x7fd70011a970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd7000a1c90 tx=0x7fd6f0031cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0032860 con 0x7fd7000a03d0 2023-10-30T12:22:08.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:22:08.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 10 monclient: dump: 2023-10-30T12:22:08.456 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.457 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.457 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.457 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.457 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.457 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.458 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.458 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.458 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.458 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fd6f0031ed0 con 0x7fd7000a03d0 2023-10-30T12:22:08.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:08.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:08.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7000a03d0 msgr2=0x7fd70011a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7000a03d0 0x7fd70011a970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd7000a1c90 tx=0x7fd6f0031cc0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7000a03d0 msgr2=0x7fd70011a970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:08.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.397+0000 7fd6f67fc640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd7000a03d0 0x7fd70011a970 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.403+0000 7fd707ec9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:08.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.403+0000 7fd707ec9640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd70011aeb0 0x7fd7000a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:08.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.403+0000 7fd707ec9640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.403+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70011e190 con 0x7fd70011aeb0 2023-10-30T12:22:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd70011aeb0 0x7fd7000a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient: get_auth_request con 0x7fd70011aeb0 auth_method 0 2023-10-30T12:22:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 10 monclient(hunting): handle_auth_done global_id 274144 payload 824 2023-10-30T12:22:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 1 -- 172.21.15.50:0/2576442561 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd70004e160 con 0x7fd70011aeb0 2023-10-30T12:22:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd705464640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd70011aeb0 0x7fd7000a24a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009870 tx=0x7fd6f0031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd6f67fc640 10 client.254309.objecter ms_handle_connect 0x7fd70011aeb0 2023-10-30T12:22:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd6f67fc640 10 client.254309.objecter resend_mon_ops 2023-10-30T12:22:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0006840 con 0x7fd70011aeb0 2023-10-30T12:22:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd6f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd6f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:22:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.404+0000 7fd6f67fc640 10 monclient: dump: 2023-10-30T12:22:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:08.466 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fd6f0050440 con 0x7fd70011aeb0 2023-10-30T12:22:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd6f67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd6f67fc640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd70011aeb0 msgr2=0x7fd7000a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd6f67fc640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd70011aeb0 0x7fd7000a24a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009870 tx=0x7fd6f0031000 comp rx=0 tx=0).stop 2023-10-30T12:22:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd6f67fc640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd70011aeb0 msgr2=0x7fd7000a24a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd6f67fc640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd70011aeb0 0x7fd7000a24a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.468 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:08.468 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:08.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:08.469 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:22:08.469 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 218, 2023-10-30T12:22:08.469 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:08.469 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:08.470 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:22:08.470 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:22:08.470 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.470 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 3, 2023-10-30T12:22:08.470 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:08.471 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:08.471 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:08.471 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.471 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.471 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.472 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.472 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.472 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.472 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.472 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.472 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.473 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.473 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.473 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:08.473 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:08.474 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.474 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.474 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.474 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.474 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.475 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.475 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.475 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.475 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.475 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.476 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:08.476 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:08.476 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:08.477 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:08.477 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:08.477 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:08.477 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:08.477 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:08.478 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:08.478 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:08.478 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:08.478 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:08.479 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:08.479 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:08.479 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:08.479 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:08.479 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:08.480 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:08.480 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:08.480 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:08.480 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:08.480 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:08.481 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:08.481 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:08.481 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:08.481 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:08.481 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:08.482 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.482 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:08.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.482 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:08.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.483 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:08.483 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:08.483 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.484 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.484 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:08.484 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.485 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:08.485 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.485 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.485 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.486 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:08.486 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:08.486 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.487 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.487 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:08.487 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:08.488 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.488 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.488 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.489 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:08.489 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.489 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.490 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:08.490 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.490 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:08.501 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:08.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 10 librados: watch_flush enter 2023-10-30T12:22:08.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 10 librados: watch_flush exit 2023-10-30T12:22:08.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 20 client.254309.objecter shutdown clearing up homeless session... 2023-10-30T12:22:08.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 10 client.254309.objecter successfully canceled tick 2023-10-30T12:22:08.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd6e40392f0 msgr2=0x7fd6e403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd6e40392f0 0x7fd6e403b7b0 secure :-1 s=READY pgs=4343 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80012e0 tx=0x7fd6f8009040 comp rx=0 tx=0).stop 2023-10-30T12:22:08.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 10 monclient: shutdown 2023-10-30T12:22:08.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:08.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd700070d10 msgr2=0x7fd7001ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:08.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd700070d10 0x7fd7001ca250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00e9b0 tx=0x7fd6fc00ee70 comp rx=0 tx=0).stop 2023-10-30T12:22:08.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd706466640 1 -- 172.21.15.50:0/2576442561 reap_dead start 2023-10-30T12:22:08.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 shutdown_connections 2023-10-30T12:22:08.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 --2- 172.21.15.50:0/2576442561 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd700070d10 0x7fd7001ca250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:08.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 >> 172.21.15.50:0/2576442561 conn(0x7fd7000a4800 msgr2=0x7fd700131d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:08.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 shutdown_connections 2023-10-30T12:22:08.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 -- 172.21.15.50:0/2576442561 wait complete. 2023-10-30T12:22:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:08.405+0000 7fd707ec9640 1 librados: shutdown 2023-10-30T12:22:08.507 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:22:08.507 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-10-30T12:22:08.508 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2023-10-30T12:22:08.508 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-10-30T12:22:08.508 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:22:08.508 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:22:08.509 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:22:08.521 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:22:08.521 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-10-30T12:22:08.522 INFO:tasks.ceph.mon.a:Sent signal 19 2023-10-30T12:22:08.522 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-10-30T12:22:23.523 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-10-30T12:22:23.524 INFO:tasks.ceph.mon.a:Sent signal 18 2023-10-30T12:22:23.524 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-10-30T12:22:23.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:22:23.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.639+0000 7f6ea4439640 10 monclient: get_monmap_and_config 2023-10-30T12:22:23.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.639+0000 7f6ea4439640 10 monclient: build_initial_monmap 2023-10-30T12:22:23.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.639+0000 7f6ea4439640 10 monclient: monmap: 2023-10-30T12:22:23.641 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:23.642 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:23.642 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:23.640618+0000 2023-10-30T12:22:23.642 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:23.640618+0000 2023-10-30T12:22:23.642 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:23.643 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:23.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:23.643 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:23.643 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:23.644 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:23.644 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:23.644 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:23.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 1 Processor -- start 2023-10-30T12:22:23.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 1 -- start start 2023-10-30T12:22:23.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient: init 2023-10-30T12:22:23.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:23.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-10-30T12:22:23.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:23.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient(hunting): picked mon.noname-a con 0x7f6e9c09f5b0 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:23.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e9c096460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:23.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient(hunting): picked mon.noname-b con 0x7f6e9c096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:23.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e9c0969a0 addr v1:172.21.15.118:6789/0 2023-10-30T12:22:23.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c098810 con 0x7f6e9c0969a0 2023-10-30T12:22:23.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:23.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c06aef0 con 0x7f6e9c09f5b0 2023-10-30T12:22:23.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c09a1a0 con 0x7f6e9c096080 2023-10-30T12:22:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea4439640 10 monclient(hunting): authenticate will time out at 2188.823803s 2023-10-30T12:22:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea19d4640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 msgr2=0x7f6e9c096460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea19d4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e9c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.640+0000 7f6ea21d5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44092/0 (socket says 172.21.15.50:44092) 2023-10-30T12:22:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 learned_addr learned my addr 172.21.15.50:0/1060216712 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e9c09f5b0 auth_method 0 2023-10-30T12:22:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:23.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:23.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:23.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:23.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:23.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:23.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.641+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:23.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.841+0000 7f6ea19d4640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 msgr2=0x7f6e9c096460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.841+0000 7f6ea19d4640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e9c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:22:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e9c09f5b0 auth_method 0 2023-10-30T12:22:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:23.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:23.842+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:22:24.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.242+0000 7f6ea19d4640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 msgr2=0x7f6e9c096460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.242+0000 7f6ea19d4640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e9c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:22:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e9c09f5b0 auth_method 0 2023-10-30T12:22:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:24.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:24.243+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:22:25.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.043+0000 7f6ea19d4640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 msgr2=0x7f6e9c096460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:25.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.043+0000 7f6ea19d4640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e9c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:22:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e9c09f5b0 auth_method 0 2023-10-30T12:22:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:25.044+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:22:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 10 monclient(hunting): tick 2023-10-30T12:22:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:22:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 1 monclient(hunting): continuing hunt 2023-10-30T12:22:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:22:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 msgr2=0x7f6e9c096460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:22:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e9c096460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 msgr2=0x7f6e9c099b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:22:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.50:3300/0 conn(0x7f6e9c09f5b0 0x7f6e9c099b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:26.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.640+0000 7f6e93fff640 1 -- 172.21.15.50:0/1060216712 >> v1:172.21.15.118:6789/0 conn(0x7f6e9c0969a0 legacy=0x7f6e9c09cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:26.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 10 monclient: _add_conns ranks=[4,0] 2023-10-30T12:22:26.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:26.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6e80003710 addr v2:172.21.15.118:3300/0 2023-10-30T12:22:26.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e80006280 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 1 -- 172.21.15.50:0/1060216712 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c098af0 con 0x7f6e80006280 2023-10-30T12:22:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 1 -- 172.21.15.50:0/1060216712 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c06aef0 con 0x7f6e80003710 2023-10-30T12:22:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6e93fff640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:26.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e80003710 auth_method 0 2023-10-30T12:22:26.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:26.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:26.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:26.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.641+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e80003710 auth_method 0 2023-10-30T12:22:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:26.842+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:22:27.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:27.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e80003710 auth_method 0 2023-10-30T12:22:27.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:27.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:27.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:27.243+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:22:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e80003710 auth_method 0 2023-10-30T12:22:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:22:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:22:28.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:22:28.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.045+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:22:28.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.557+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291345612 0 0) 0x7f6e9c098af0 con 0x7f6e80006280 2023-10-30T12:22:28.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.557+0000 7f6ea11d3640 10 monclient(hunting): _negotiate 2023-10-30T12:22:28.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.557+0000 7f6ea11d3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.557+0000 7f6ea11d3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:28.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.557+0000 7f6ea11d3640 10 monclient(hunting): my global_id is 264279 2023-10-30T12:22:28.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.558+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e88003520 con 0x7f6e80006280 2023-10-30T12:22:28.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4064847658 0 0) 0x7f6e88003520 con 0x7f6e80006280 2023-10-30T12:22:28.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 msgr2=0x7f6e80005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:22:28.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 --2- 172.21.15.50:0/1060216712 >> v2:172.21.15.118:3300/0 conn(0x7f6e80003710 0x7f6e80005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: _finish_hunting 0 2023-10-30T12:22:28.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:22:28.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:28.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:22:28.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8000afd0 con 0x7f6e80006280 2023-10-30T12:22:28.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: _finish_auth 0 2023-10-30T12:22:28.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: _check_auth_tickets 2023-10-30T12:22:28.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:28.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7f6e8c004340 con 0x7f6e80006280 2023-10-30T12:22:28.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:22:28.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: dump: 2023-10-30T12:22:28.565 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.565 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.566 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.566 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.566 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.566 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.566 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.567 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.567 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.567 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:22:28.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:28.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 >> v1:172.21.15.50:6789/0 conn(0x7f6e80006280 legacy=0x7f6e80008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:22:28.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e88003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): picked mon.b con 0x7f6e88003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): picked mon.c con 0x7f6e9c096080 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e88004d00 0x7f6e88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): picked mon.a con 0x7f6e88004d00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c06aef0 con 0x7f6e88004d00 2023-10-30T12:22:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8800bb40 con 0x7f6e88003970 2023-10-30T12:22:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8800bd00 con 0x7f6e9c096080 2023-10-30T12:22:28.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:28.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea11d3640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:22:28.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea29d6640 1 -- 172.21.15.50:0/1060216712 reap_dead start 2023-10-30T12:22:28.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea19d4640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 msgr2=0x7f6e880047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.561+0000 7f6ea19d4640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e880047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e88004d00 0x7f6e88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 10 monclient(hunting): get_auth_request con 0x7f6e88004d00 auth_method 0 2023-10-30T12:22:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:22:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e88003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e88003970 auth_method 0 2023-10-30T12:22:28.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.562+0000 7f6ea21d5640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:28.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea21d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:28.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea21d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:28.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea21d5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:22:28.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_done global_id 264279 payload 293 2023-10-30T12:22:28.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 msgr2=0x7f6e880047c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:22:28.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e880047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 msgr2=0x7f6e88003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:28.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e88003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 10 monclient: _finish_hunting 0 2023-10-30T12:22:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 1 monclient: found mon.a 2023-10-30T12:22:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:28.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 1 -- 172.21.15.50:0/1060216712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8000afd0 con 0x7f6e88004d00 2023-10-30T12:22:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 10 monclient: _finish_auth 0 2023-10-30T12:22:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 10 monclient: _check_auth_tickets 2023-10-30T12:22:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.563+0000 7f6ea29d6640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.564+0000 7f6ea4439640 5 monclient: authenticate success, global_id 264279 2023-10-30T12:22:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.564+0000 7f6ea4439640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.565+0000 7f6ea29d6640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e88004d00 0x7f6e88005180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c073e10 tx=0x7f6e880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9ebeda818d59ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e94005d40 con 0x7f6e88004d00 2023-10-30T12:22:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 10 monclient: dump: 2023-10-30T12:22:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.580 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.581 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea4439640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9402e070 con 0x7f6e88004d00 2023-10-30T12:22:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 1 -- 172.21.15.50:0/1060216712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e940072b0 con 0x7f6e88004d00 2023-10-30T12:22:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea4439640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea4439640 4 set_mon_vals no callback set 2023-10-30T12:22:28.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.567+0000 7f6ea11d3640 10 monclient: dump: 2023-10-30T12:22:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.583 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.584 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.585 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.585 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.568+0000 7f6ea4439640 10 monclient: shutdown 2023-10-30T12:22:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.568+0000 7f6ea4439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.568+0000 7f6ea4439640 1 -- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e88004d00 msgr2=0x7f6e88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.568+0000 7f6ea4439640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e88004d00 0x7f6e88005180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c073e10 tx=0x7f6e880057f0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.568+0000 7f6ea4439640 1 -- 172.21.15.50:0/1060216712 shutdown_connections 2023-10-30T12:22:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e88004d00 0x7f6e88005180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e9c096080 0x7f6e880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 1 --2- 172.21.15.50:0/1060216712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e88003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 1 -- 172.21.15.50:0/1060216712 >> 172.21.15.50:0/1060216712 conn(0x7f6e9c0a05c0 msgr2=0x7f6e9c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 1 -- 172.21.15.50:0/1060216712 shutdown_connections 2023-10-30T12:22:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 1 -- 172.21.15.50:0/1060216712 wait complete. 2023-10-30T12:22:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 10 monclient: build_initial_monmap 2023-10-30T12:22:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.569+0000 7f6ea4439640 10 monclient: monmap: 2023-10-30T12:22:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:28.594 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 librados: starting msgr at 2023-10-30T12:22:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 librados: starting objecter 2023-10-30T12:22:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 Processor -- start 2023-10-30T12:22:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 -- start start 2023-10-30T12:22:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 librados: setting wanted keys 2023-10-30T12:22:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 librados: calling monclient init 2023-10-30T12:22:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient: init 2023-10-30T12:22:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:22:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e9c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): picked mon.noname-b con 0x7f6e88003970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e88004d00 0x7f6e9c095480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): picked mon.noname-c con 0x7f6e88004d00 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e9c096080 0x7f6e9c0959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): picked mon.noname-a con 0x7f6e9c096080 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c095f00 con 0x7f6e9c096080 2023-10-30T12:22:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0909f0 con 0x7f6e88003970 2023-10-30T12:22:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0a2730 con 0x7f6e88004d00 2023-10-30T12:22:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea4439640 10 monclient(hunting): authenticate will time out at 2193.753770s 2023-10-30T12:22:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea19d4640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e88004d00 msgr2=0x7f6e9c095480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea19d4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e88004d00 0x7f6e9c095480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea29d6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e9c096080 0x7f6e9c0959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea29d6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e9c096080 0x7f6e9c0959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60730/0 (socket says 172.21.15.50:60730) 2023-10-30T12:22:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea29d6640 1 -- 172.21.15.50:0/3832140237 learned_addr learned my addr 172.21.15.50:0/3832140237 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea29d6640 10 monclient(hunting): get_auth_request con 0x7f6e9c096080 auth_method 0 2023-10-30T12:22:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea29d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea29d6640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.570+0000 7f6ea29d6640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e9c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request con 0x7f6e88003970 auth_method 0 2023-10-30T12:22:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea21d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 10 monclient(hunting): handle_auth_done global_id 264282 payload 824 2023-10-30T12:22:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 1 -- 172.21.15.50:0/3832140237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e88004d00 msgr2=0x7f6e9c095480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:22:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e88004d00 0x7f6e9c095480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 1 -- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 msgr2=0x7f6e9c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e9c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 10 monclient: _finish_hunting 0 2023-10-30T12:22:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 1 monclient: found mon.noname-a 2023-10-30T12:22:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 1 -- 172.21.15.50:0/3832140237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c09c430 con 0x7f6e9c096080 2023-10-30T12:22:28.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea29d6640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e9c096080 0x7f6e9c0959c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0a0e10 tx=0x7f6e940014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 client.?.objecter ms_handle_connect 0x7f6e9c096080 2023-10-30T12:22:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 1 -- 172.21.15.50:0/3832140237 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e94003970 con 0x7f6e9c096080 2023-10-30T12:22:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: dump: 2023-10-30T12:22:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.606 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: _finish_auth 0 2023-10-30T12:22:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:22:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 5 monclient: authenticate success, global_id 264282 2023-10-30T12:22:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 1 -- 172.21.15.50:0/3832140237 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e94003ad0 con 0x7f6e9c096080 2023-10-30T12:22:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 10 monclient: _renew_subs 2023-10-30T12:22:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c0906e0 con 0x7f6e9c096080 2023-10-30T12:22:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 1 -- 172.21.15.50:0/3832140237 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e94003e40 con 0x7f6e9c096080 2023-10-30T12:22:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6e92ffd640 10 monclient: dump: 2023-10-30T12:22:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.609 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.610 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.610 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea09d2640 4 set_mon_vals no callback set 2023-10-30T12:22:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 10 client.264282.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 10 monclient: _renew_subs 2023-10-30T12:22:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c070e60 con 0x7f6e9c096080 2023-10-30T12:22:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.571+0000 7f6ea4439640 1 librados: init done 2023-10-30T12:22:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.572+0000 7f6e92ffd640 1 -- 172.21.15.50:0/3832140237 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f6e94030030 con 0x7f6e9c096080 2023-10-30T12:22:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6ea4439640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6ea4439640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6ea4439640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6e92ffd640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6e80039fa0 0x7f6e8003c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c090b80 con 0x7f6e9c096080 2023-10-30T12:22:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6e92ffd640 1 -- 172.21.15.50:0/3832140237 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(670..670 src has 520..670) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f6e9406b8a0 con 0x7f6e9c096080 2023-10-30T12:22:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6e92ffd640 10 client.264282.objecter ms_dispatch 0x7f6e9c000b90 osd_map(670..670 src has 520..670) v4 2023-10-30T12:22:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6e92ffd640 3 client.264282.objecter handle_osd_map got epochs [670,670] > 0 2023-10-30T12:22:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6e92ffd640 3 client.264282.objecter handle_osd_map decoding full epoch 670 2023-10-30T12:22:28.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6e80039fa0 0x7f6e8003c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6e92ffd640 20 client.264282.objecter dump_active .. 0 homeless 2023-10-30T12:22:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.573+0000 7f6ea21d5640 10 monclient: get_auth_request con 0x7f6e80039fa0 auth_method 0 2023-10-30T12:22:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.574+0000 7f6e92ffd640 10 client.264282.objecter ms_handle_connect 0x7f6e80039fa0 2023-10-30T12:22:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.574+0000 7f6ea21d5640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6e80039fa0 0x7f6e8003c460 secure :-1 s=READY pgs=4344 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004650 tx=0x7f6e9801b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.578+0000 7f6e92ffd640 1 -- 172.21.15.50:0/3832140237 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e940469a0 con 0x7f6e9c096080 2023-10-30T12:22:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.578+0000 7f6e92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.578+0000 7f6e92ffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.824+0000 7f6ea4439640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:22:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.824+0000 7f6ea4439640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:22:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.824+0000 7f6ea4439640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.824+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e9c04e160 con 0x7f6e9c096080 2023-10-30T12:22:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.824+0000 7f6e92ffd640 1 -- 172.21.15.50:0/3832140237 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f6e9403e420 con 0x7f6e9c096080 2023-10-30T12:22:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.824+0000 7f6e92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:22:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.824+0000 7f6e92ffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:28.827 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":222,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:22:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 10 librados: watch_flush enter 2023-10-30T12:22:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 10 librados: watch_flush exit 2023-10-30T12:22:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 20 client.264282.objecter shutdown clearing up homeless session... 2023-10-30T12:22:28.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 10 client.264282.objecter successfully canceled tick 2023-10-30T12:22:28.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6e80039fa0 msgr2=0x7f6e8003c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6e80039fa0 0x7f6e8003c460 secure :-1 s=READY pgs=4344 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004650 tx=0x7f6e9801b040 comp rx=0 tx=0).stop 2023-10-30T12:22:28.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 10 monclient: shutdown 2023-10-30T12:22:28.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:28.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e9c096080 msgr2=0x7f6e9c0959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e9c096080 0x7f6e9c0959c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0a0e10 tx=0x7f6e940014d0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 shutdown_connections 2023-10-30T12:22:28.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f6e80039fa0 0x7f6e8003c460 unknown :-1 s=CLOSED pgs=4344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e9c096080 0x7f6e9c0959c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e88004d00 0x7f6e9c095480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 --2- 172.21.15.50:0/3832140237 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e88003970 0x7f6e9c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 >> 172.21.15.50:0/3832140237 conn(0x7f6e9c0a05c0 msgr2=0x7f6e9c1bb920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:28.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 shutdown_connections 2023-10-30T12:22:28.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 -- 172.21.15.50:0/3832140237 wait complete. 2023-10-30T12:22:28.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.826+0000 7f6ea4439640 1 librados: shutdown 2023-10-30T12:22:28.838 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-10-30T12:22:28.838 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-10-30T12:22:28.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:22:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.943+0000 7fd841fb0640 10 monclient: get_monmap_and_config 2023-10-30T12:22:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.943+0000 7fd841fb0640 10 monclient: build_initial_monmap 2023-10-30T12:22:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.943+0000 7fd841fb0640 10 monclient: monmap: 2023-10-30T12:22:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:28.944521+0000 2023-10-30T12:22:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:28.944521+0000 2023-10-30T12:22:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:28.947 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 1 Processor -- start 2023-10-30T12:22:28.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 1 -- start start 2023-10-30T12:22:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient: init 2023-10-30T12:22:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-10-30T12:22:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd83c071c00 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd83c097fe0 addr v1:172.21.15.118:6789/0 2023-10-30T12:22:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 0x7fd83c13a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient(hunting): picked mon.noname-b con 0x7fd83c09a150 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c09b880 con 0x7fd83c071c00 2023-10-30T12:22:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c097600 con 0x7fd83c097fe0 2023-10-30T12:22:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c09a5e0 con 0x7fd83c09a150 2023-10-30T12:22:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd841fb0640 10 monclient(hunting): authenticate will time out at 2194.127669s 2023-10-30T12:22:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd83bfff640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 msgr2=0x7fd83c13a770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd83bfff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 0x7fd83c13a770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd83b7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd83c071c00 0x7fd83c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56884/0 (socket says 172.21.15.50:56884) 2023-10-30T12:22:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.944+0000 7fd83b7fe640 1 -- 172.21.15.50:0/4235124897 learned_addr learned my addr 172.21.15.50:0/4235124897 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:22:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994820707 0 0) 0x7fd83c09b880 con 0x7fd83c071c00 2023-10-30T12:22:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:22:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): my global_id is 264285 2023-10-30T12:22:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd824005730 con 0x7fd83c071c00 2023-10-30T12:22:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989872015 0 0) 0x7fd83c097600 con 0x7fd83c097fe0 2023-10-30T12:22:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:22:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): my global_id is 274150 2023-10-30T12:22:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd83c09b880 con 0x7fd83c097fe0 2023-10-30T12:22:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1806767465 0 0) 0x7fd824005730 con 0x7fd83c071c00 2023-10-30T12:22:28.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 msgr2=0x7fd83c13a770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:22:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 0x7fd83c13a770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 >> v1:172.21.15.118:6789/0 conn(0x7fd83c097fe0 legacy=0x7fd83c09b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:22:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:22:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:22:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0978e0 con 0x7fd83c071c00 2023-10-30T12:22:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:22:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:22:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fd82c0017f0 con 0x7fd83c071c00 2023-10-30T12:22:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:22:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: dump: 2023-10-30T12:22:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.971 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:22:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 >> v1:172.21.15.50:6789/0 conn(0x7fd83c071c00 legacy=0x7fd83c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:22:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 0x7fd824005d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): picked mon.c con 0x7fd83c09a150 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd8240069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): picked mon.a con 0x7fd8240062c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd824007030 0x7fd82400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): picked mon.b con 0x7fd824007030 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c09a5e0 con 0x7fd8240062c0 2023-10-30T12:22:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd824007850 con 0x7fd824007030 2023-10-30T12:22:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83b7fe640 1 -- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 msgr2=0x7fd824005d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82400e100 con 0x7fd83c09a150 2023-10-30T12:22:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83b7fe640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 0x7fd824005d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83a7fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:22:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd8240069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): get_auth_request con 0x7fd8240062c0 auth_method 0 2023-10-30T12:22:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd824007030 0x7fd82400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 10 monclient(hunting): get_auth_request con 0x7fd824007030 auth_method 0 2023-10-30T12:22:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:22:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:22:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:22:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient(hunting): handle_auth_done global_id 264285 payload 293 2023-10-30T12:22:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 1 -- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 msgr2=0x7fd824005d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:22:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c09a150 0x7fd824005d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 1 -- 172.21.15.50:0/4235124897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd824007030 msgr2=0x7fd82400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:22:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd824007030 0x7fd82400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient: _finish_hunting 0 2023-10-30T12:22:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 1 monclient: found mon.a 2023-10-30T12:22:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 1 -- 172.21.15.50:0/4235124897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824005730 con 0x7fd8240062c0 2023-10-30T12:22:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient: _finish_auth 0 2023-10-30T12:22:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 10 monclient: _check_auth_tickets 2023-10-30T12:22:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd83affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd841fb0640 5 monclient: authenticate success, global_id 264285 2023-10-30T12:22:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.945+0000 7fd841fb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83affd640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd8240069d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8300018a0 tx=0x7fd824007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5681cb836cd7e1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8300052d0 con 0x7fd8240062c0 2023-10-30T12:22:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 10 monclient: dump: 2023-10-30T12:22:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.983 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd841fb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd830005430 con 0x7fd8240062c0 2023-10-30T12:22:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd841fb0640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd841fb0640 4 set_mon_vals no callback set 2023-10-30T12:22:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 1 -- 172.21.15.50:0/4235124897 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830003810 con 0x7fd8240062c0 2023-10-30T12:22:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.946+0000 7fd83a7fc640 10 monclient: dump: 2023-10-30T12:22:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:28.987 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 10 monclient: shutdown 2023-10-30T12:22:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 1 -- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 msgr2=0x7fd8240069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 1 --2- 172.21.15.50:0/4235124897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd8240069d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8300018a0 tx=0x7fd824007a00 comp rx=0 tx=0).stop 2023-10-30T12:22:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd83bfff640 1 -- 172.21.15.50:0/4235124897 reap_dead start 2023-10-30T12:22:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 1 -- 172.21.15.50:0/4235124897 shutdown_connections 2023-10-30T12:22:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 1 -- 172.21.15.50:0/4235124897 >> 172.21.15.50:0/4235124897 conn(0x7fd83c0a3980 msgr2=0x7fd83c0a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 1 -- 172.21.15.50:0/4235124897 shutdown_connections 2023-10-30T12:22:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 1 -- 172.21.15.50:0/4235124897 wait complete. 2023-10-30T12:22:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 10 monclient: build_initial_monmap 2023-10-30T12:22:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.947+0000 7fd841fb0640 10 monclient: monmap: 2023-10-30T12:22:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:28.989 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:28.990 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:28.991 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 librados: starting msgr at 2023-10-30T12:22:28.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 librados: starting objecter 2023-10-30T12:22:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 Processor -- start 2023-10-30T12:22:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 -- start start 2023-10-30T12:22:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 librados: setting wanted keys 2023-10-30T12:22:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 librados: calling monclient init 2023-10-30T12:22:28.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient: init 2023-10-30T12:22:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:22:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd83c095fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): picked mon.noname-a con 0x7fd8240062c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c091aa0 0x7fd83c091ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): picked mon.noname-c con 0x7fd83c091aa0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd83c092420 0x7fd83c1ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): picked mon.noname-b con 0x7fd83c092420 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c095290 con 0x7fd8240062c0 2023-10-30T12:22:28.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c094fb0 con 0x7fd83c092420 2023-10-30T12:22:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0928d0 con 0x7fd83c091aa0 2023-10-30T12:22:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd841fb0640 10 monclient(hunting): authenticate will time out at 2194.131813s 2023-10-30T12:22:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83affd640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c091aa0 msgr2=0x7fd83c091ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c091aa0 0x7fd83c091ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd83c095fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd83c095fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60752/0 (socket says 172.21.15.50:60752) 2023-10-30T12:22:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 1 -- 172.21.15.50:0/4294563749 learned_addr learned my addr 172.21.15.50:0/4294563749 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:28.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 10 monclient(hunting): get_auth_request con 0x7fd8240062c0 auth_method 0 2023-10-30T12:22:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:28.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.948+0000 7fd83b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83bfff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd83c092420 0x7fd83c1ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83bfff640 10 monclient(hunting): get_auth_request con 0x7fd83c092420 auth_method 0 2023-10-30T12:22:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 10 monclient(hunting): handle_auth_done global_id 264288 payload 824 2023-10-30T12:22:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c091aa0 msgr2=0x7fd83c091ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:22:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd83c091aa0 0x7fd83c091ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd83c092420 msgr2=0x7fd83c1ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd83c092420 0x7fd83c1ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:22:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:22:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:29.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 1 -- 172.21.15.50:0/4294563749 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c09cbc0 con 0x7fd8240062c0 2023-10-30T12:22:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83bfff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd83c092420 0x7fd83c1ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:22:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd83b7fe640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd83c095fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c4e0 tx=0x7fd82c00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 client.?.objecter ms_handle_connect 0x7fd8240062c0 2023-10-30T12:22:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:29.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c007480 con 0x7fd8240062c0 2023-10-30T12:22:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: dump: 2023-10-30T12:22:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.002 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.004 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: _finish_auth 0 2023-10-30T12:22:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: _check_auth_tickets 2023-10-30T12:22:29.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c0075e0 con 0x7fd8240062c0 2023-10-30T12:22:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0059c0 con 0x7fd8240062c0 2023-10-30T12:22:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 5 monclient: authenticate success, global_id 264288 2023-10-30T12:22:29.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 10 monclient: _renew_subs 2023-10-30T12:22:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83c0a2bb0 con 0x7fd8240062c0 2023-10-30T12:22:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd839ffb640 4 set_mon_vals no callback set 2023-10-30T12:22:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd823fff640 10 monclient: dump: 2023-10-30T12:22:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.007 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.008 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 10 client.264288.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 10 monclient: _renew_subs 2023-10-30T12:22:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c093410 con 0x7fd8240062c0 2023-10-30T12:22:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.949+0000 7fd841fb0640 1 librados: init done 2023-10-30T12:22:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd841fb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd841fb0640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02b430 0x7fd83c02b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd841fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c073ff0 con 0x7fd83c02b430 2023-10-30T12:22:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd82c005b20 con 0x7fd8240062c0 2023-10-30T12:22:29.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02b430 0x7fd83c02b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient: get_auth_request con 0x7fd83c02b430 auth_method 0 2023-10-30T12:22:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:29.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd8280392d0 0x7fd82803b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 10 monclient(hunting): handle_auth_done global_id 264291 payload 824 2023-10-30T12:22:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(670..670 src has 520..670) v4 ==== 6670+0+0 (secure 0 0 0) 0x7fd82c04a370 con 0x7fd8240062c0 2023-10-30T12:22:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 10 client.264288.objecter ms_dispatch 0x7fd83c000b90 osd_map(670..670 src has 520..670) v4 2023-10-30T12:22:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 3 client.264288.objecter handle_osd_map got epochs [670,670] > 0 2023-10-30T12:22:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 3 client.264288.objecter handle_osd_map decoding full epoch 670 2023-10-30T12:22:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 1 -- 172.21.15.50:0/4294563749 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd83c09d9b0 con 0x7fd83c02b430 2023-10-30T12:22:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 20 client.264288.objecter dump_active .. 0 homeless 2023-10-30T12:22:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83affd640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02b430 0x7fd83c02b810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd830001870 tx=0x7fd8300012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 10 client.264288.objecter ms_handle_connect 0x7fd83c02b430 2023-10-30T12:22:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd823fff640 10 client.264288.objecter resend_mon_ops 2023-10-30T12:22:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83bfff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd8280392d0 0x7fd82803b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.950+0000 7fd83bfff640 10 monclient: get_auth_request con 0x7fd8280392d0 auth_method 0 2023-10-30T12:22:29.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830045600 con 0x7fd83c02b430 2023-10-30T12:22:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 10 monclient: dump: 2023-10-30T12:22:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.015 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.017 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fd8300457c0 con 0x7fd83c02b430 2023-10-30T12:22:29.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02b430 msgr2=0x7fd83c02b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02b430 0x7fd83c02b810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd830001870 tx=0x7fd8300012b0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02b430 msgr2=0x7fd83c02b810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02b430 0x7fd83c02b810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd83bfff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd8280392d0 0x7fd82803b790 secure :-1 s=READY pgs=4345 cs=0 l=1 rev1=1 crypto rx=0x7fd83c09cdd0 tx=0x7fd834009520 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.951+0000 7fd823fff640 10 client.264288.objecter ms_handle_connect 0x7fd8280392d0 2023-10-30T12:22:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd841fb0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd841fb0640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02bd50 0x7fd83c02c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd841fb0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c09bf50 con 0x7fd83c02bd50 2023-10-30T12:22:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02bd50 0x7fd83c02c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient: get_auth_request con 0x7fd83c02bd50 auth_method 0 2023-10-30T12:22:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:29.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:29.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:29.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:29.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:29.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 10 monclient(hunting): handle_auth_done global_id 264294 payload 824 2023-10-30T12:22:29.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.957+0000 7fd83affd640 1 -- 172.21.15.50:0/4294563749 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd83c071fe0 con 0x7fd83c02bd50 2023-10-30T12:22:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 10 client.264288.objecter ms_handle_connect 0x7fd83c02bd50 2023-10-30T12:22:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd83affd640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02bd50 0x7fd83c02c160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd830005620 tx=0x7fd830007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 10 client.264288.objecter resend_mon_ops 2023-10-30T12:22:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830032ba0 con 0x7fd83c02bd50 2023-10-30T12:22:29.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 10 monclient: dump: 2023-10-30T12:22:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.023 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.025 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4623 (secure 0 0 0) 0x7fd83003d3f0 con 0x7fd83c02bd50 2023-10-30T12:22:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:29.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02bd50 msgr2=0x7fd83c02c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02bd50 0x7fd83c02c160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd830005620 tx=0x7fd830007c40 comp rx=0 tx=0).stop 2023-10-30T12:22:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02bd50 msgr2=0x7fd83c02c160 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd823fff640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd83c02bd50 0x7fd83c02c160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.026 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:29.027 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:29.027 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:29.027 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:22:29.027 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 222, 2023-10-30T12:22:29.027 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:29.028 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:29.028 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:22:29.028 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.028 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:22:29.028 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:29.029 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:29.029 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:29.029 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:29.029 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:29.029 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:29.030 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:29.031 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:29.032 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:29.033 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:29.034 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:29.035 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:29.036 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:29.037 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:29.038 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.039 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:29.040 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:29.046 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:29.047 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:29.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:29.052 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:29.053 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:29.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:22:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:29.055 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:22:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.056 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7 2023-10-30T12:22:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.058 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:29.058 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 10 librados: watch_flush enter 2023-10-30T12:22:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 10 librados: watch_flush exit 2023-10-30T12:22:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 20 client.264288.objecter shutdown clearing up homeless session... 2023-10-30T12:22:29.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 10 client.264288.objecter successfully canceled tick 2023-10-30T12:22:29.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd8280392d0 msgr2=0x7fd82803b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd8280392d0 0x7fd82803b790 secure :-1 s=READY pgs=4345 cs=0 l=1 rev1=1 crypto rx=0x7fd83c09cdd0 tx=0x7fd834009520 comp rx=0 tx=0).stop 2023-10-30T12:22:29.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 10 monclient: shutdown 2023-10-30T12:22:29.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:29.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 msgr2=0x7fd83c095fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd83c095fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c4e0 tx=0x7fd82c00c9a0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd83bfff640 1 -- 172.21.15.50:0/4294563749 reap_dead start 2023-10-30T12:22:29.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 shutdown_connections 2023-10-30T12:22:29.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 --2- 172.21.15.50:0/4294563749 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8240062c0 0x7fd83c095fe0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 >> 172.21.15.50:0/4294563749 conn(0x7fd83c0a3980 msgr2=0x7fd83c0a2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:29.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 shutdown_connections 2023-10-30T12:22:29.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 -- 172.21.15.50:0/4294563749 wait complete. 2023-10-30T12:22:29.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:28.958+0000 7fd841fb0640 1 librados: shutdown 2023-10-30T12:22:29.063 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:22:29.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-10-30T12:22:29.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:22:29.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.169+0000 7f3f69a56640 10 monclient: get_monmap_and_config 2023-10-30T12:22:29.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.169+0000 7f3f69a56640 10 monclient: build_initial_monmap 2023-10-30T12:22:29.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.169+0000 7f3f69a56640 10 monclient: monmap: 2023-10-30T12:22:29.171 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:22:29.170215+0000 2023-10-30T12:22:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:22:29.170215+0000 2023-10-30T12:22:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:22:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:22:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:22:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:22:29.173 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:22:29.173 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.169+0000 7f3f69a56640 1 Processor -- start 2023-10-30T12:22:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.169+0000 7f3f69a56640 1 -- start start 2023-10-30T12:22:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.169+0000 7f3f69a56640 10 monclient: init 2023-10-30T12:22:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-10-30T12:22:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f640a8010 addr v1:172.21.15.50:6789/0 2023-10-30T12:22:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f64099a40 addr v1:172.21.15.118:6789/0 2023-10-30T12:22:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3f64132da0 0x7f3f6413e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient(hunting): picked mon.noname-a con 0x7f3f64132da0 addr v2:172.21.15.50:3300/0 2023-10-30T12:22:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640a7190 con 0x7f3f640a8010 2023-10-30T12:22:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640a83f0 con 0x7f3f64099a40 2023-10-30T12:22:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f640a7600 con 0x7f3f64132da0 2023-10-30T12:22:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f69a56640 10 monclient(hunting): authenticate will time out at 2194.353273s 2023-10-30T12:22:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3f64132da0 0x7f3f6413e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f62ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3f640a8010 0x7f3f64132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56896/0 (socket says 172.21.15.50:56896) 2023-10-30T12:22:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f62ffd640 1 -- 172.21.15.50:0/1260549357 learned_addr learned my addr 172.21.15.50:0/1260549357 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:22:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): get_auth_request con 0x7f3f64132da0 auth_method 0 2023-10-30T12:22:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:29.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient(hunting): handle_auth_done global_id 264297 payload 274 2023-10-30T12:22:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 1 -- 172.21.15.50:0/1260549357 >> v1:172.21.15.118:6789/0 conn(0x7f3f64099a40 legacy=0x7f3f6413aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 1 -- 172.21.15.50:0/1260549357 >> v1:172.21.15.50:6789/0 conn(0x7f3f640a8010 legacy=0x7f3f64132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient: _finish_hunting 0 2023-10-30T12:22:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 1 monclient: found mon.noname-a 2023-10-30T12:22:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f61ffb640 1 -- 172.21.15.50:0/1260549357 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827596601 0 0) 0x7f3f640a83f0 con 0x7f3f64099a40 2023-10-30T12:22:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f637fe640 1 -- 172.21.15.50:0/1260549357 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6409a440 con 0x7f3f64132da0 2023-10-30T12:22:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.170+0000 7f3f61ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:22:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f637fe640 1 --2- 172.21.15.50:0/1260549357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f64132da0 0x7f3f6413e140 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009d90 tx=0x7f3f5402bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1014fbc862e8b39b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 1 -- 172.21.15.50:0/1260549357 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5402c9c0 con 0x7f3f64132da0 2023-10-30T12:22:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: dump: 2023-10-30T12:22:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.192 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: _finish_auth 0 2023-10-30T12:22:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:22:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f69a56640 5 monclient: authenticate success, global_id 264297 2023-10-30T12:22:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f69a56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:22:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 1 -- 172.21.15.50:0/1260549357 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5402cb20 con 0x7f3f64132da0 2023-10-30T12:22:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f69a56640 10 monclient: get_monmap_and_config success 2023-10-30T12:22:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f69a56640 4 set_mon_vals no callback set 2023-10-30T12:22:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 1 -- 172.21.15.50:0/1260549357 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54034440 con 0x7f3f64132da0 2023-10-30T12:22:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.171+0000 7f3f61ffb640 10 monclient: dump: 2023-10-30T12:22:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.197 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 10 monclient: shutdown 2023-10-30T12:22:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 -- 172.21.15.50:0/1260549357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f64132da0 msgr2=0x7f3f6413e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 --2- 172.21.15.50:0/1260549357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f64132da0 0x7f3f6413e140 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009d90 tx=0x7f3f5402bc90 comp rx=0 tx=0).stop 2023-10-30T12:22:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 -- 172.21.15.50:0/1260549357 shutdown_connections 2023-10-30T12:22:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 --2- 172.21.15.50:0/1260549357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f64132da0 0x7f3f6413e140 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 -- 172.21.15.50:0/1260549357 >> 172.21.15.50:0/1260549357 conn(0x7f3f640a4a10 msgr2=0x7f3f640a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 -- 172.21.15.50:0/1260549357 shutdown_connections 2023-10-30T12:22:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 -- 172.21.15.50:0/1260549357 wait complete. 2023-10-30T12:22:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 10 monclient: build_initial_monmap 2023-10-30T12:22:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 10 monclient: monmap: 2023-10-30T12:22:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:22:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:22:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:22:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:22:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:22:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:22:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:22:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:22:29.201 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 librados: starting msgr at 2023-10-30T12:22:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.172+0000 7f3f69a56640 1 librados: starting objecter 2023-10-30T12:22:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 Processor -- start 2023-10-30T12:22:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 -- start start 2023-10-30T12:22:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 librados: setting wanted keys 2023-10-30T12:22:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 librados: calling monclient init 2023-10-30T12:22:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient: init 2023-10-30T12:22:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient: _reopen_session rank -1 2023-10-30T12:22:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:22:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f640a8010 0x7f3f641367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): picked mon.noname-c con 0x7f3f640a8010 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:22:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64132da0 0x7f3f64136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): picked mon.noname-b con 0x7f3f64132da0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:22:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f6413a9d0 0x7f3f64137240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6413a9d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:22:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64139ac0 con 0x7f3f6413a9d0 2023-10-30T12:22:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640a18d0 con 0x7f3f64132da0 2023-10-30T12:22:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6413db50 con 0x7f3f640a8010 2023-10-30T12:22:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): _renew_subs 2023-10-30T12:22:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f69a56640 10 monclient(hunting): authenticate will time out at 2194.356677s 2023-10-30T12:22:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f62ffd640 1 -- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f640a8010 msgr2=0x7f3f641367c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:3301/0 2023-10-30T12:22:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f62ffd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f640a8010 0x7f3f641367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:22:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f6413a9d0 0x7f3f64137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f6413a9d0 0x7f3f64137240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60788/0 (socket says 172.21.15.50:60788) 2023-10-30T12:22:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 1 -- 172.21.15.50:0/559849753 learned_addr learned my addr 172.21.15.50:0/559849753 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:22:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 10 monclient(hunting): get_auth_request con 0x7f3f6413a9d0 auth_method 0 2023-10-30T12:22:29.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.173+0000 7f3f637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f627fc640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64132da0 0x7f3f64136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f627fc640 10 monclient(hunting): get_auth_request con 0x7f3f64132da0 auth_method 0 2023-10-30T12:22:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f627fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:29.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f627fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 10 monclient(hunting): handle_auth_done global_id 264303 payload 824 2023-10-30T12:22:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f640a8010 msgr2=0x7f3f641367c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:22:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f640a8010 0x7f3f641367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64132da0 msgr2=0x7f3f64136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64132da0 0x7f3f64136d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 10 monclient: _finish_hunting 0 2023-10-30T12:22:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 1 monclient: found mon.noname-a 2023-10-30T12:22:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:22:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:22:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 1 -- 172.21.15.50:0/559849753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f54009a70 con 0x7f3f6413a9d0 2023-10-30T12:22:29.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f637fe640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f6413a9d0 0x7f3f64137240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009d90 tx=0x7f3f54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 client.?.objecter ms_handle_connect 0x7f3f6413a9d0 2023-10-30T12:22:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:22:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5403e610 con 0x7f3f6413a9d0 2023-10-30T12:22:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:22:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: dump: 2023-10-30T12:22:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.212 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.213 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.213 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: _finish_auth 0 2023-10-30T12:22:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:22:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:22:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 5 monclient: authenticate success, global_id 264303 2023-10-30T12:22:29.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5403e770 con 0x7f3f6413a9d0 2023-10-30T12:22:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 10 monclient: _renew_subs 2023-10-30T12:22:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f64139ff0 con 0x7f3f6413a9d0 2023-10-30T12:22:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:22:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5403eb20 con 0x7f3f6413a9d0 2023-10-30T12:22:29.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:22:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f4bfff640 10 monclient: dump: 2023-10-30T12:22:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.216 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.217 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.218 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 10 client.264303.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:22:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 10 monclient: _renew_subs 2023-10-30T12:22:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f617fa640 4 set_mon_vals no callback set 2023-10-30T12:22:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:22:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f64137bb0 con 0x7f3f6413a9d0 2023-10-30T12:22:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.174+0000 7f3f69a56640 1 librados: init done 2023-10-30T12:22:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3f5403b050 con 0x7f3f6413a9d0 2023-10-30T12:22:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f4bfff640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f500392a0 0x7f3f5003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f69a56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f69a56640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64099a40 0x7f3f641ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f69a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64137d40 con 0x7f3f64099a40 2023-10-30T12:22:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(670..670 src has 520..670) v4 ==== 6670+0+0 (secure 0 0 0) 0x7f3f5406d560 con 0x7f3f6413a9d0 2023-10-30T12:22:29.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f4bfff640 10 client.264303.objecter ms_dispatch 0x7f3f64000b90 osd_map(670..670 src has 520..670) v4 2023-10-30T12:22:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f4bfff640 3 client.264303.objecter handle_osd_map got epochs [670,670] > 0 2023-10-30T12:22:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f4bfff640 3 client.264303.objecter handle_osd_map decoding full epoch 670 2023-10-30T12:22:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f4bfff640 20 client.264303.objecter dump_active .. 0 homeless 2023-10-30T12:22:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64099a40 0x7f3f641ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f62ffd640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f500392a0 0x7f3f5003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient: get_auth_request con 0x7f3f64099a40 auth_method 0 2023-10-30T12:22:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:29.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f62ffd640 10 monclient: get_auth_request con 0x7f3f500392a0 auth_method 0 2023-10-30T12:22:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.175+0000 7f3f627fc640 10 monclient(hunting): handle_auth_done global_id 274162 payload 824 2023-10-30T12:22:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f627fc640 1 -- 172.21.15.50:0/559849753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f6409aa00 con 0x7f3f64099a40 2023-10-30T12:22:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 client.264303.objecter ms_handle_connect 0x7f3f500392a0 2023-10-30T12:22:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f62ffd640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f500392a0 0x7f3f5003b760 secure :-1 s=READY pgs=4346 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c004810 tx=0x7f3f5c007b00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 client.264303.objecter ms_handle_connect 0x7f3f64099a40 2023-10-30T12:22:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 client.264303.objecter resend_mon_ops 2023-10-30T12:22:29.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f627fc640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64099a40 0x7f3f641ca710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3f58000f60 tx=0x7f3f5800e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580052e0 con 0x7f3f64099a40 2023-10-30T12:22:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:22:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 monclient: dump: 2023-10-30T12:22:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.226 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.227 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7f3f58005440 con 0x7f3f64099a40 2023-10-30T12:22:29.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:22:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:22:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64099a40 msgr2=0x7f3f641ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64099a40 0x7f3f641ca710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3f58000f60 tx=0x7f3f5800e960 comp rx=0 tx=0).stop 2023-10-30T12:22:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64099a40 msgr2=0x7f3f641ca710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:29.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.176+0000 7f3f4bfff640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f64099a40 0x7f3f641ca710 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.182+0000 7f3f69a56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:22:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.182+0000 7f3f69a56640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f641cac50 0x7f3f641cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:22:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.182+0000 7f3f69a56640 10 monclient(hunting): start opening mon connection 2023-10-30T12:22:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.182+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64073fd0 con 0x7f3f641cac50 2023-10-30T12:22:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f641cac50 0x7f3f641cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:22:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient: get_auth_request con 0x7f3f641cac50 auth_method 0 2023-10-30T12:22:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:22:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:22:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:22:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:22:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:22:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:22:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 10 monclient(hunting): handle_auth_done global_id 274165 payload 824 2023-10-30T12:22:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 1 -- 172.21.15.50:0/559849753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3f6409a720 con 0x7f3f641cac50 2023-10-30T12:22:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f627fc640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f641cac50 0x7f3f641cb060 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f6413d1a0 tx=0x7f3f58001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:22:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f4bfff640 10 client.264303.objecter ms_handle_connect 0x7f3f641cac50 2023-10-30T12:22:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f4bfff640 10 client.264303.objecter resend_mon_ops 2023-10-30T12:22:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58023070 con 0x7f3f641cac50 2023-10-30T12:22:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:22:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:22:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.183+0000 7f3f4bfff640 10 monclient: dump: 2023-10-30T12:22:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:22:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:22:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:22:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:22:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:22:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:22:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:22:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:22:29.236 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:22:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4621 (secure 0 0 0) 0x7f3f58013420 con 0x7f3f641cac50 2023-10-30T12:22:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f4bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:22:29.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f4bfff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:22:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f641cac50 msgr2=0x7f3f641cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f4bfff640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f641cac50 0x7f3f641cb060 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3f6413d1a0 tx=0x7f3f58001080 comp rx=0 tx=0).stop 2023-10-30T12:22:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f4bfff640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f641cac50 msgr2=0x7f3f641cb060 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:22:29.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f4bfff640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f641cac50 0x7f3f641cb060 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:22:29.238 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:22:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:22:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 222, 2023-10-30T12:22:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:22:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:22:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2023-10-30T12:22:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:22:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:22:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:22:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:29.241 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:29.242 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:29.242 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:29.242 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.242 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:22:29.242 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:22:29.243 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:29.243 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:29.243 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:29.243 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:29.243 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:29.243 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:29.244 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:29.244 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:29.244 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:29.244 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:29.244 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.245 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:22:29.245 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:22:29.245 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:22:29.245 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:22:29.246 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:22:29.246 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:22:29.246 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:29.246 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:22:29.246 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:22:29.247 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:22:29.247 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:22:29.247 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:22:29.247 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:22:29.248 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:22:29.248 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:22:29.248 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:22:29.248 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:22:29.248 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:22:29.249 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:22:29.249 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:22:29.249 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:22:29.249 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:22:29.249 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:22:29.250 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:22:29.250 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:22:29.250 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:22:29.250 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:22:29.251 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.251 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:22:29.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.251 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:22:29.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.252 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:22:29.252 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:22:29.252 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:29.252 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:29.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.253 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:29.253 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:22:29.253 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.253 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:29.254 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:22:29.254 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.254 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:22:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:22:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:29.255 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:29.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.256 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:22:29.256 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:22:29.256 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:29.257 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:29.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.257 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:29.257 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:22:29.257 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.258 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:29.258 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:22:29.258 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.259 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.259 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:22:29.259 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:22:29.259 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:29.260 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:29.260 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:29.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.260 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:22:29.261 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:22:29.261 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:22:29.261 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:22:29.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.261 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:22:29.262 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:22:29.262 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.262 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:22:29.262 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:22:29.263 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:22:29.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.263 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.263 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:22:29.264 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:22:29.264 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:22:29.264 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:22:29.264 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:22:29.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.265 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.265 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:22:29.265 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:22:29.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.265 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:29.266 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:29.266 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:29.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.266 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.267 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:22:29.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.267 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:29.267 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:29.267 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:29.268 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.268 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:22:29.268 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:22:29.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:22:29.268 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:22:29.269 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:22:29.269 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:22:29.269 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:22:29.269 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:22:29.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:22:29.269 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:22:29.270 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:22:29.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 10 librados: watch_flush enter 2023-10-30T12:22:29.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 10 librados: watch_flush exit 2023-10-30T12:22:29.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 20 client.264303.objecter shutdown clearing up homeless session... 2023-10-30T12:22:29.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 10 client.264303.objecter successfully canceled tick 2023-10-30T12:22:29.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f500392a0 msgr2=0x7f3f5003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3f500392a0 0x7f3f5003b760 secure :-1 s=READY pgs=4346 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c004810 tx=0x7f3f5c007b00 comp rx=0 tx=0).stop 2023-10-30T12:22:29.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 10 monclient: shutdown 2023-10-30T12:22:29.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:22:29.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f6413a9d0 msgr2=0x7f3f64137240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:22:29.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 --2- 172.21.15.50:0/559849753 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f6413a9d0 0x7f3f64137240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009d90 tx=0x7f3f54004a40 comp rx=0 tx=0).stop 2023-10-30T12:22:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f637fe640 1 -- 172.21.15.50:0/559849753 reap_dead start 2023-10-30T12:22:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f637fe640 1 -- 172.21.15.50:0/559849753 reap_dead start 2023-10-30T12:22:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 shutdown_connections 2023-10-30T12:22:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 >> 172.21.15.50:0/559849753 conn(0x7f3f640a4a10 msgr2=0x7f3f640a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:22:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 shutdown_connections 2023-10-30T12:22:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 -- 172.21.15.50:0/559849753 wait complete. 2023-10-30T12:22:29.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:22:29.184+0000 7f3f69a56640 1 librados: shutdown 2023-10-30T12:22:29.275 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-10-30T12:22:29.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-10-30T12:22:44.610 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (21542 ms) 2023-10-30T12:22:44.610 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-10-30T12:22:44.611 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13035 ms) 2023-10-30T12:22:44.611 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-10-30T12:22:44.611 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8031 ms) 2023-10-30T12:22:44.611 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-10-30T12:22:44.611 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (26626 ms) 2023-10-30T12:22:44.612 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-10-30T12:22:44.612 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7223 ms) 2023-10-30T12:22:44.612 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-10-30T12:22:44.612 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8036 ms) 2023-10-30T12:22:44.612 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-10-30T12:22:44.613 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: hmm, no HitSet yet 2023-10-30T12:22:44.613 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2023-10-30T12:22:44.613 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8991 ms) 2023-10-30T12:22:44.613 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-10-30T12:22:44.613 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,0 2023-10-30T12:22:44.613 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: first is 1698668419 2023-10-30T12:22:44.614 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,0 2023-10-30T12:22:44.614 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,0 2023-10-30T12:22:44.614 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,0 2023-10-30T12:22:44.614 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,0 2023-10-30T12:22:44.615 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,0 2023-10-30T12:22:44.615 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,1698668425,1698668426,0 2023-10-30T12:22:44.615 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,1698668425,1698668426,0 2023-10-30T12:22:44.615 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,1698668425,1698668426,0 2023-10-30T12:22:44.615 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,1698668425,1698668426,1698668428,1698668429,0 2023-10-30T12:22:44.616 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,1698668425,1698668426,1698668428,1698668429,0 2023-10-30T12:22:44.616 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668419,1698668421,1698668422,1698668425,1698668426,1698668428,1698668429,0 2023-10-30T12:22:44.616 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: got ls 1698668422,1698668425,1698668426,1698668428,1698668429,1698668431,1698668432,0 2023-10-30T12:22:44.616 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: first now 1698668422, trimmed 2023-10-30T12:22:44.616 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (24134 ms) 2023-10-30T12:22:44.617 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-10-30T12:22:44.617 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: foo0 2023-10-30T12:22:44.617 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-10-30T12:22:44.617 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14954 ms) 2023-10-30T12:22:44.617 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-10-30T12:22:44.618 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (38154 ms) 2023-10-30T12:22:44.618 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-10-30T12:22:44.618 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (38866 ms) 2023-10-30T12:22:44.618 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-10-30T12:22:44.619 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (24503 ms) 2023-10-30T12:22:44.619 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-10-30T12:22:44.619 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3008 ms) 2023-10-30T12:22:44.619 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-10-30T12:22:44.619 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3010 ms) 2023-10-30T12:22:44.620 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-10-30T12:22:44.620 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3008 ms) 2023-10-30T12:22:44.620 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (358604 ms total) 2023-10-30T12:22:44.620 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: 2023-10-30T12:22:44.621 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-10-30T12:22:44.621 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1250329 ms total) 2023-10-30T12:22:44.621 INFO:tasks.workunit.client.0.smithi118.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-10-30T12:22:44.622 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.623 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87234 2023-10-30T12:22:44.623 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87234 2023-10-30T12:22:44.623 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.623 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87382 2023-10-30T12:22:44.624 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87382 2023-10-30T12:22:44.624 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.624 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87836 2023-10-30T12:22:44.624 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87836 2023-10-30T12:22:44.624 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.625 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87619 2023-10-30T12:22:44.625 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87619 2023-10-30T12:22:44.625 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.625 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87467 2023-10-30T12:22:44.626 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87467 2023-10-30T12:22:44.626 INFO:tasks.workunit.client.0.smithi118.stdout:error in api_stat_pp (87467) 2023-10-30T12:22:44.627 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'error in api_stat_pp (87467)' 2023-10-30T12:22:44.627 INFO:tasks.workunit.client.0.smithi118.stderr:+ ret=1 2023-10-30T12:22:44.628 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.628 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87314 2023-10-30T12:22:44.628 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87314 2023-10-30T12:22:44.628 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.629 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87575 2023-10-30T12:22:44.629 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87575 2023-10-30T12:22:44.629 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.629 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87331 2023-10-30T12:22:44.629 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87331 2023-10-30T12:22:44.630 INFO:tasks.workunit.client.0.smithi118.stdout:error in api_misc_pp (87331) 2023-10-30T12:22:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:22:44.635 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'error in api_misc_pp (87331)' 2023-10-30T12:22:44.635 INFO:tasks.workunit.client.0.smithi118.stderr:+ ret=1 2023-10-30T12:22:44.635 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.636 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87764 2023-10-30T12:22:44.636 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87764 2023-10-30T12:22:44.636 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.636 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87209 2023-10-30T12:22:44.636 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87209 2023-10-30T12:22:44.637 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.637 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87272 2023-10-30T12:22:44.637 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87272 2023-10-30T12:22:44.637 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.637 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87788 2023-10-30T12:22:44.638 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87788 2023-10-30T12:22:44.638 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.638 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87200 2023-10-30T12:22:44.638 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87200 2023-10-30T12:22:44.638 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.639 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87221 2023-10-30T12:22:44.639 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87221 2023-10-30T12:22:44.639 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.639 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87407 2023-10-30T12:22:44.639 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87407 2023-10-30T12:22:44.640 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.640 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87497 2023-10-30T12:22:44.640 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87497 2023-10-30T12:22:44.640 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.640 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87441 2023-10-30T12:22:44.641 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87441 2023-10-30T12:22:44.641 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.641 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87803 2023-10-30T12:22:44.641 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87803 2023-10-30T12:22:44.641 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.642 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87744 2023-10-30T12:22:44.642 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87744 2023-10-30T12:22:44.642 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.642 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87367 2023-10-30T12:22:44.642 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87367 2023-10-30T12:22:44.643 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.643 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87647 2023-10-30T12:22:44.643 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87647 2023-10-30T12:22:44.643 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.643 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87193 2023-10-30T12:22:44.644 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87193 2023-10-30T12:22:44.644 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.644 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87251 2023-10-30T12:22:44.644 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87251 2023-10-30T12:22:44.644 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.645 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87598 2023-10-30T12:22:44.645 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87598 2023-10-30T12:22:44.645 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.645 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87291 2023-10-30T12:22:44.645 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87291 2023-10-30T12:22:44.646 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.646 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87690 2023-10-30T12:22:44.646 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87690 2023-10-30T12:22:44.647 INFO:tasks.workunit.client.0.smithi118.stderr:+ for t in "${!pids[@]}" 2023-10-30T12:22:44.647 INFO:tasks.workunit.client.0.smithi118.stderr:+ pid=87711 2023-10-30T12:22:44.647 INFO:tasks.workunit.client.0.smithi118.stderr:+ wait 87711 2023-10-30T12:22:44.648 INFO:tasks.workunit.client.0.smithi118.stderr:+ exit 1 2023-10-30T12:22:44.648 INFO:tasks.workunit.client.0.smithi118.stderr:+ cleanup 2023-10-30T12:22:44.648 INFO:tasks.workunit.client.0.smithi118.stderr:+ pkill -P 87188 2023-10-30T12:22:44.649 INFO:tasks.workunit.client.0.smithi118.stderr:+ true 2023-10-30T12:22:45.894 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2023-10-30T12:22:45.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T12:22:46.168 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/remote.py", line 529, in run_unit_test r = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=397e85a5f8ee6b984077e94184404370497237a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa/tasks/workunit.py", line 426, in _run_tests remote.run_unit_test( File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/remote.py", line 534, in run_unit_test raise UnitTestError( teuthology.exceptions.UnitTestError: Unit test failed (workunit test rados/test.sh) on smithi118 with status 1: '(total 7 failed) FAILURE: Test `LongNamePP` of `LibRadosMiscPP`. Reason: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0....' 2023-10-30T12:22:47.268 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cb081291f18544ecbcfbc902f0f11aca Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/remote.py", line 529, in run_unit_test r = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=397e85a5f8ee6b984077e94184404370497237a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_VallariAg_ceph_397e85a5f8ee6b984077e94184404370497237a9/qa/tasks/workunit.py", line 426, in _run_tests remote.run_unit_test( File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/remote.py", line 534, in run_unit_test raise UnitTestError( teuthology.exceptions.UnitTestError: Unit test failed (workunit test rados/test.sh) on smithi118 with status 1: '(total 7 failed) FAILURE: Test `LongNamePP` of `LibRadosMiscPP`. Reason: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0....' 2023-10-30T12:22:47.285 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2023-10-30T12:22:47.294 INFO:tasks.mon_thrash:joining mon_thrasher 2023-10-30T12:22:49.276 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-10-30T12:22:49.277 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-10-30T12:22:49.277 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:22:49.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:22:49.281 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:22:49.281 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-10-30T12:22:49.282 INFO:tasks.ceph.mon.a:Sent signal 19 2023-10-30T12:22:49.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-10-30T12:23:04.283 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-10-30T12:23:04.284 INFO:tasks.ceph.mon.a:Sent signal 18 2023-10-30T12:23:04.284 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:23:04.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:23:04.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.395+0000 7fc041cd4640 10 monclient: get_monmap_and_config 2023-10-30T12:23:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.395+0000 7fc041cd4640 10 monclient: build_initial_monmap 2023-10-30T12:23:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.395+0000 7fc041cd4640 10 monclient: monmap: 2023-10-30T12:23:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:04.396399+0000 2023-10-30T12:23:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:04.396399+0000 2023-10-30T12:23:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:04.399 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:04.401 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:04.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 1 Processor -- start 2023-10-30T12:23:04.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 1 -- start start 2023-10-30T12:23:04.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient: init 2023-10-30T12:23:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-10-30T12:23:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:04.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient(hunting): picked mon.noname-b con 0x7fc03c098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:04.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc03c098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:04.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:04.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient(hunting): picked mon.noname-a con 0x7fc03c1379f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:04.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09ad30 con 0x7fc03c098900 2023-10-30T12:23:04.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:04.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c0993f0 con 0x7fc03c1379f0 2023-10-30T12:23:04.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:04.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c09c460 con 0x7fc03c098520 2023-10-30T12:23:04.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:04.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc041cd4640 10 monclient(hunting): authenticate will time out at 2229.579667s 2023-10-30T12:23:04.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:04.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55074/0 (socket says 172.21.15.50:55074) 2023-10-30T12:23:04.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:04.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 learned_addr learned my addr 172.21.15.50:0/4117656211 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:04.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request con 0x7fc03c098520 auth_method 0 2023-10-30T12:23:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:04.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 10 monclient(hunting): get_auth_request con 0x7fc03c1379f0 auth_method 0 2023-10-30T12:23:04.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:04.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:23:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:23:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:04.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:23:04.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.396+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:23:04.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:04.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:04.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request con 0x7fc03c098520 auth_method 0 2023-10-30T12:23:04.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:04.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:04.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:04.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 10 monclient(hunting): get_auth_request con 0x7fc03c1379f0 auth_method 0 2023-10-30T12:23:04.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:04.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:04.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:04.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:23:04.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:04.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:04.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:23:04.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:04.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:04.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:23:04.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.597+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:23:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:05.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request con 0x7fc03c098520 auth_method 0 2023-10-30T12:23:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:05.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 10 monclient(hunting): get_auth_request con 0x7fc03c1379f0 auth_method 0 2023-10-30T12:23:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:23:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:05.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:23:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:05.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:23:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:04.998+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:23:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request con 0x7fc03c098520 auth_method 0 2023-10-30T12:23:05.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:05.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 10 monclient(hunting): get_auth_request con 0x7fc03c1379f0 auth_method 0 2023-10-30T12:23:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:23:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:23:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:23:05.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:05.800+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:23:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient(hunting): tick 2023-10-30T12:23:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient(hunting): _check_auth_tickets 2023-10-30T12:23:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 monclient(hunting): continuing hunt 2023-10-30T12:23:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient(hunting): _reopen_session rank -1 2023-10-30T12:23:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 msgr2=0x7fc03c09bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:23:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc03c098520 0x7fc03c09bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 msgr2=0x7fc03c139de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:23:07.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.50:3300/0 conn(0x7fc03c1379f0 0x7fc03c139de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:07.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 -- 172.21.15.50:0/4117656211 >> v1:172.21.15.118:6789/0 conn(0x7fc03c098900 legacy=0x7fc03c098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:07.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient: _add_conns ranks=[0,4] 2023-10-30T12:23:07.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0280036c0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:07.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:07.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc028007410 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:07.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 -- 172.21.15.50:0/4117656211 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09b260 con 0x7fc0280036c0 2023-10-30T12:23:07.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:07.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 1 -- 172.21.15.50:0/4117656211 --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c0993f0 con 0x7fc028007410 2023-10-30T12:23:07.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc0397fa640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:07.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:07.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 10 monclient(hunting): get_auth_request con 0x7fc028007410 auth_method 0 2023-10-30T12:23:07.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:07.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:07.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:07.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:23:07.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:07.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:07.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.397+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:23:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 10 monclient(hunting): get_auth_request con 0x7fc028007410 auth_method 0 2023-10-30T12:23:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:07.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:23:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.598+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:23:08.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 10 monclient(hunting): get_auth_request con 0x7fc028007410 auth_method 0 2023-10-30T12:23:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:08.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:23:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:07.999+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:23:08.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:08.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 10 monclient(hunting): get_auth_request con 0x7fc028007410 auth_method 0 2023-10-30T12:23:08.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:08.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:08.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:08.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:23:08.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:23:08.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:08.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:08.801+0000 7fc03affd640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:23:09.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.314+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280801787 0 0) 0x7fc03c09b260 con 0x7fc0280036c0 2023-10-30T12:23:09.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.314+0000 7fc03a7fc640 10 monclient(hunting): _negotiate 2023-10-30T12:23:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.314+0000 7fc03a7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.314+0000 7fc03a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.314+0000 7fc03a7fc640 10 monclient(hunting): my global_id is 264306 2023-10-30T12:23:09.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.314+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc024003520 con 0x7fc0280036c0 2023-10-30T12:23:09.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1051989057 0 0) 0x7fc024003520 con 0x7fc0280036c0 2023-10-30T12:23:09.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 msgr2=0x7fc0280098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:23:09.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 --2- 172.21.15.50:0/4117656211 >> v2:172.21.15.118:3300/0 conn(0x7fc028007410 0x7fc0280098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:23:09.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 monclient: found mon.noname-a-legacy 2023-10-30T12:23:09.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:09.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.50:6789/0 2023-10-30T12:23:09.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02800afc0 con 0x7fc0280036c0 2023-10-30T12:23:09.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: _finish_auth 0 2023-10-30T12:23:09.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:23:09.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:09.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3888465108 0 0) 0x7fc02c00e750 con 0x7fc0280036c0 2023-10-30T12:23:09.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-10-30T12:23:09.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: dump: 2023-10-30T12:23:09.343 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.344 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.344 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.344 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.345 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.345 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.345 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.346 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.346 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.346 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-10-30T12:23:09.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:09.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 >> v1:172.21.15.50:6789/0 conn(0x7fc0280036c0 legacy=0x7fc028005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03bfff640 1 -- 172.21.15.50:0/4117656211 reap_dead start 2023-10-30T12:23:09.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:23:09.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240039c0 0x7fc024003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): picked mon.c con 0x7fc0240039c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:09.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240047c0 0x7fc024004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): picked mon.a con 0x7fc0240047c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:09.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc02400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): picked mon.b con 0x7fc024005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:09.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0993f0 con 0x7fc0240047c0 2023-10-30T12:23:09.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02400c240 con 0x7fc024005380 2023-10-30T12:23:09.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02400c400 con 0x7fc0240039c0 2023-10-30T12:23:09.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:09.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03a7fc640 10 monclient(hunting): _finish_auth 0 2023-10-30T12:23:09.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240039c0 0x7fc024003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request con 0x7fc0240039c0 auth_method 0 2023-10-30T12:23:09.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:09.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc02400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03bfff640 10 monclient(hunting): get_auth_request con 0x7fc024005380 auth_method 0 2023-10-30T12:23:09.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-10-30T12:23:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.335+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:23:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-10-30T12:23:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_done global_id 264306 payload 293 2023-10-30T12:23:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 msgr2=0x7fc02400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc02400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240047c0 msgr2=0x7fc024004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:09.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240047c0 0x7fc024004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:09.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 1 monclient: found mon.c 2023-10-30T12:23:09.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:09.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 1 -- 172.21.15.50:0/4117656211 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024003520 con 0x7fc0240039c0 2023-10-30T12:23:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 10 monclient: _finish_auth 0 2023-10-30T12:23:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:23:09.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:09.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc041cd4640 5 monclient: authenticate success, global_id 264306 2023-10-30T12:23:09.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc041cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:09.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.336+0000 7fc03bfff640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc02400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:09.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.337+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240039c0 0x7fc024003dc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc02c006f90 tx=0x7fc0240057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0866f36b1c8c1b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c037450 con 0x7fc0240039c0 2023-10-30T12:23:09.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:09.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 10 monclient: dump: 2023-10-30T12:23:09.371 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.372 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.372 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.372 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.373 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.373 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.373 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.374 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.374 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.374 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c0375b0 con 0x7fc0240039c0 2023-10-30T12:23:09.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc041cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:09.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:09.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc041cd4640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:09.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 1 -- 172.21.15.50:0/4117656211 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c0379a0 con 0x7fc0240039c0 2023-10-30T12:23:09.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc041cd4640 4 set_mon_vals no callback set 2023-10-30T12:23:09.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:09.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.346+0000 7fc03a7fc640 10 monclient: dump: 2023-10-30T12:23:09.377 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.378 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.378 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.378 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.379 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.379 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.379 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.379 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.380 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.380 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 10 monclient: shutdown 2023-10-30T12:23:09.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:09.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 -- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240039c0 msgr2=0x7fc024003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240039c0 0x7fc024003dc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc02c006f90 tx=0x7fc0240057f0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 -- 172.21.15.50:0/4117656211 shutdown_connections 2023-10-30T12:23:09.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc02400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240047c0 0x7fc024004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 --2- 172.21.15.50:0/4117656211 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240039c0 0x7fc024003dc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 -- 172.21.15.50:0/4117656211 >> 172.21.15.50:0/4117656211 conn(0x7fc03c0a05c0 msgr2=0x7fc03c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:09.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 -- 172.21.15.50:0/4117656211 shutdown_connections 2023-10-30T12:23:09.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.347+0000 7fc041cd4640 1 -- 172.21.15.50:0/4117656211 wait complete. 2023-10-30T12:23:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.348+0000 7fc041cd4640 10 monclient: build_initial_monmap 2023-10-30T12:23:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.348+0000 7fc041cd4640 10 monclient: monmap: 2023-10-30T12:23:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:09.385 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:09.385 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:09.385 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:09.385 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:09.386 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.386 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:09.386 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:09.387 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:09.387 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.348+0000 7fc041cd4640 1 librados: starting msgr at 2023-10-30T12:23:09.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.348+0000 7fc041cd4640 1 librados: starting objecter 2023-10-30T12:23:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.349+0000 7fc041cd4640 1 Processor -- start 2023-10-30T12:23:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.349+0000 7fc041cd4640 1 -- start start 2023-10-30T12:23:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.349+0000 7fc041cd4640 1 librados: setting wanted keys 2023-10-30T12:23:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.349+0000 7fc041cd4640 1 librados: calling monclient init 2023-10-30T12:23:09.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.349+0000 7fc041cd4640 10 monclient: init 2023-10-30T12:23:09.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:09.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:23:09.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240039c0 0x7fc03c1c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): picked mon.noname-a con 0x7fc0240039c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 0x7fc03c096320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): picked mon.noname-c con 0x7fc0240047c0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc03c096860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): picked mon.noname-b con 0x7fc024005380 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c094550 con 0x7fc0240039c0 2023-10-30T12:23:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0a2a60 con 0x7fc024005380 2023-10-30T12:23:09.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c093a10 con 0x7fc0240047c0 2023-10-30T12:23:09.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:09.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc041cd4640 10 monclient(hunting): authenticate will time out at 2234.533186s 2023-10-30T12:23:09.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240039c0 0x7fc03c1c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 0x7fc03c096320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240039c0 0x7fc03c1c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59632/0 (socket says 172.21.15.50:59632) 2023-10-30T12:23:09.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 -- 172.21.15.50:0/953527548 learned_addr learned my addr 172.21.15.50:0/953527548 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:09.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request con 0x7fc0240039c0 auth_method 0 2023-10-30T12:23:09.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 10 monclient(hunting): get_auth_request con 0x7fc0240047c0 auth_method 0 2023-10-30T12:23:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc03c096860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 10 monclient(hunting): get_auth_request con 0x7fc024005380 auth_method 0 2023-10-30T12:23:09.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 1 -- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 msgr2=0x7fc03c096320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:23:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 1 -- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 msgr2=0x7fc03c096320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:23:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 0x7fc03c096320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03affd640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 0x7fc03c096320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:23:09.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient(hunting): handle_auth_done global_id 264309 payload 824 2023-10-30T12:23:09.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 -- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 msgr2=0x7fc03c096320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:23:09.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 0x7fc03c096320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 -- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 msgr2=0x7fc03c096860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc03c096860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 monclient: found mon.noname-a 2023-10-30T12:23:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.350+0000 7fc03b7fe640 1 -- 172.21.15.50:0/953527548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c092fa0 con 0x7fc0240039c0 2023-10-30T12:23:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc03b7fe640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240039c0 0x7fc03c1c9d90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc02c03a040 tx=0x7fc02c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 client.?.objecter ms_handle_connect 0x7fc0240039c0 2023-10-30T12:23:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 1 -- 172.21.15.50:0/953527548 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c037450 con 0x7fc0240039c0 2023-10-30T12:23:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: dump: 2023-10-30T12:23:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.409 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: _finish_auth 0 2023-10-30T12:23:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:09.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 5 monclient: authenticate success, global_id 264309 2023-10-30T12:23:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 1 -- 172.21.15.50:0/953527548 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c037b30 con 0x7fc0240039c0 2023-10-30T12:23:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 10 monclient: _renew_subs 2023-10-30T12:23:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c071df0 con 0x7fc0240039c0 2023-10-30T12:23:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 1 -- 172.21.15.50:0/953527548 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c0090c0 con 0x7fc0240039c0 2023-10-30T12:23:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:09.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc023fff640 10 monclient: dump: 2023-10-30T12:23:09.414 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.414 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.414 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.414 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.415 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.415 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.415 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.416 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.416 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.416 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 10 client.264309.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 10 monclient: _renew_subs 2023-10-30T12:23:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c092a20 con 0x7fc0240039c0 2023-10-30T12:23:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc039ffb640 4 set_mon_vals no callback set 2023-10-30T12:23:09.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.351+0000 7fc041cd4640 1 librados: init done 2023-10-30T12:23:09.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc023fff640 1 -- 172.21.15.50:0/953527548 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc02c03e070 con 0x7fc0240039c0 2023-10-30T12:23:09.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc041cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:09.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc041cd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:09.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc041cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:09.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc023fff640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc028039b70 0x7fc02803c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c04e160 con 0x7fc0240039c0 2023-10-30T12:23:09.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc023fff640 1 -- 172.21.15.50:0/953527548 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7fc02c000e40 con 0x7fc0240039c0 2023-10-30T12:23:09.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc023fff640 10 client.264309.objecter ms_dispatch 0x7fc03c000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:09.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc023fff640 3 client.264309.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:09.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc023fff640 3 client.264309.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:09.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc023fff640 20 client.264309.objecter dump_active .. 0 homeless 2023-10-30T12:23:09.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc03affd640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc028039b70 0x7fc02803c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.352+0000 7fc03affd640 10 monclient: get_auth_request con 0x7fc028039b70 auth_method 0 2023-10-30T12:23:09.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.353+0000 7fc03affd640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc028039b70 0x7fc02803c030 secure :-1 s=READY pgs=4348 cs=0 l=1 rev1=1 crypto rx=0x7fc030007b80 tx=0x7fc030007b10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.353+0000 7fc023fff640 10 client.264309.objecter ms_handle_connect 0x7fc028039b70 2023-10-30T12:23:09.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.357+0000 7fc023fff640 1 -- 172.21.15.50:0/953527548 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc02c00e9e0 con 0x7fc0240039c0 2023-10-30T12:23:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.357+0000 7fc023fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:09.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.357+0000 7fc023fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:09.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.609+0000 7fc041cd4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:23:09.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.609+0000 7fc041cd4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:23:09.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.609+0000 7fc041cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:09.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.609+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc03c0a26f0 con 0x7fc0240039c0 2023-10-30T12:23:09.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.609+0000 7fc023fff640 1 -- 172.21.15.50:0/953527548 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fc02c03c080 con 0x7fc0240039c0 2023-10-30T12:23:09.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.609+0000 7fc023fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:23:09.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.609+0000 7fc023fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:23:09.613 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":228,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:23:09.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 10 librados: watch_flush enter 2023-10-30T12:23:09.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 10 librados: watch_flush exit 2023-10-30T12:23:09.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 20 client.264309.objecter shutdown clearing up homeless session... 2023-10-30T12:23:09.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 10 client.264309.objecter successfully canceled tick 2023-10-30T12:23:09.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc028039b70 msgr2=0x7fc02803c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc028039b70 0x7fc02803c030 secure :-1 s=READY pgs=4348 cs=0 l=1 rev1=1 crypto rx=0x7fc030007b80 tx=0x7fc030007b10 comp rx=0 tx=0).stop 2023-10-30T12:23:09.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 10 monclient: shutdown 2023-10-30T12:23:09.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:09.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240039c0 msgr2=0x7fc03c1c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240039c0 0x7fc03c1c9d90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc02c03a040 tx=0x7fc02c004730 comp rx=0 tx=0).stop 2023-10-30T12:23:09.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 shutdown_connections 2023-10-30T12:23:09.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fc028039b70 0x7fc02803c030 unknown :-1 s=CLOSED pgs=4348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc024005380 0x7fc03c096860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc0240047c0 0x7fc03c096320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 --2- 172.21.15.50:0/953527548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0240039c0 0x7fc03c1c9d90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 >> 172.21.15.50:0/953527548 conn(0x7fc03c0a05c0 msgr2=0x7fc03c09dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:09.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 shutdown_connections 2023-10-30T12:23:09.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 -- 172.21.15.50:0/953527548 wait complete. 2023-10-30T12:23:09.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.611+0000 7fc041cd4640 1 librados: shutdown 2023-10-30T12:23:09.623 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:23:09.624 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-10-30T12:23:09.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-10-30T12:23:09.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.728+0000 7fa3f59d0640 10 monclient: get_monmap_and_config 2023-10-30T12:23:09.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.728+0000 7fa3f59d0640 10 monclient: build_initial_monmap 2023-10-30T12:23:09.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.728+0000 7fa3f59d0640 10 monclient: monmap: 2023-10-30T12:23:09.731 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:09.732 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.732 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:09.729605+0000 2023-10-30T12:23:09.732 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:09.729605+0000 2023-10-30T12:23:09.733 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:09.733 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.733 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:09.733 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:09.734 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:09.734 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:09.735 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:09.735 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 1 Processor -- start 2023-10-30T12:23:09.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 1 -- start start 2023-10-30T12:23:09.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient: init 2023-10-30T12:23:09.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:09.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-10-30T12:23:09.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3f009bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:09.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fa3f009d3f0 0x7fa3f0137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f009d3f0 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:09.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f009c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:09.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f0098170 con 0x7fa3f009bb70 2023-10-30T12:23:09.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f009cf60 con 0x7fa3f009d3f0 2023-10-30T12:23:09.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0097b20 con 0x7fa3f009c660 2023-10-30T12:23:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3f59d0640 10 monclient(hunting): authenticate will time out at 2234.912646s 2023-10-30T12:23:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60506/0 (socket says 172.21.15.50:60506) 2023-10-30T12:23:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 1 -- 172.21.15.50:0/2565021471 learned_addr learned my addr 172.21.15.50:0/2565021471 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa3f009c660 auth_method 0 2023-10-30T12:23:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ee7fc640 1 --2- 172.21.15.50:0/2565021471 >> v2:172.21.15.118:3300/0 conn(0x7fa3f009d3f0 0x7fa3f0137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa3f009d3f0 auth_method 0 2023-10-30T12:23:09.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ee7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.729+0000 7fa3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 10 monclient(hunting): handle_auth_done global_id 284099 payload 274 2023-10-30T12:23:09.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 1 -- 172.21.15.50:0/2565021471 >> v2:172.21.15.118:3300/0 conn(0x7fa3f009d3f0 msgr2=0x7fa3f0137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:09.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 1 --2- 172.21.15.50:0/2565021471 >> v2:172.21.15.118:3300/0 conn(0x7fa3f009d3f0 0x7fa3f0137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 1 -- 172.21.15.50:0/2565021471 >> v1:172.21.15.118:6789/0 conn(0x7fa3f009bb70 legacy=0x7fa3f009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:09.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 1 monclient: found mon.noname-b 2023-10-30T12:23:09.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:09.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:09.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 1 -- 172.21.15.50:0/2565021471 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0097e00 con 0x7fa3f009c660 2023-10-30T12:23:09.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3ef7fe640 1 --2- 172.21.15.50:0/2565021471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0139eb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009e50 tx=0x7fa3e802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7125913dee30723a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.730+0000 7fa3edffb640 1 -- 172.21.15.50:0/2565021471 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e802c9e0 con 0x7fa3f009c660 2023-10-30T12:23:09.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:09.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: dump: 2023-10-30T12:23:09.760 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.762 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.762 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.762 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.762 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.763 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: _finish_auth 0 2023-10-30T12:23:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: _check_auth_tickets 2023-10-30T12:23:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:09.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 1 -- 172.21.15.50:0/2565021471 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e802cb40 con 0x7fa3f009c660 2023-10-30T12:23:09.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 5 monclient: authenticate success, global_id 284099 2023-10-30T12:23:09.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 1 -- 172.21.15.50:0/2565021471 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e80343f0 con 0x7fa3f009c660 2023-10-30T12:23:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 4 set_mon_vals no callback set 2023-10-30T12:23:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3edffb640 10 monclient: dump: 2023-10-30T12:23:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.769 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.769 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 10 monclient: shutdown 2023-10-30T12:23:09.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:09.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 -- 172.21.15.50:0/2565021471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 msgr2=0x7fa3f0139eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 --2- 172.21.15.50:0/2565021471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0139eb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009e50 tx=0x7fa3e802be40 comp rx=0 tx=0).stop 2023-10-30T12:23:09.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 -- 172.21.15.50:0/2565021471 shutdown_connections 2023-10-30T12:23:09.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 --2- 172.21.15.50:0/2565021471 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0139eb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 --2- 172.21.15.50:0/2565021471 >> v2:172.21.15.118:3300/0 conn(0x7fa3f009d3f0 0x7fa3f0137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 -- 172.21.15.50:0/2565021471 >> 172.21.15.50:0/2565021471 conn(0x7fa3f00a3980 msgr2=0x7fa3f00a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:09.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 -- 172.21.15.50:0/2565021471 shutdown_connections 2023-10-30T12:23:09.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.731+0000 7fa3f59d0640 1 -- 172.21.15.50:0/2565021471 wait complete. 2023-10-30T12:23:09.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 10 monclient: build_initial_monmap 2023-10-30T12:23:09.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 10 monclient: monmap: 2023-10-30T12:23:09.772 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:09.773 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:09.773 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:09.773 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:09.773 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:09.774 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.774 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:09.774 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:09.775 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 1 librados: starting msgr at 2023-10-30T12:23:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 1 librados: starting objecter 2023-10-30T12:23:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 1 Processor -- start 2023-10-30T12:23:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 1 -- start start 2023-10-30T12:23:09.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 1 librados: setting wanted keys 2023-10-30T12:23:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 1 librados: calling monclient init 2023-10-30T12:23:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 10 monclient: init 2023-10-30T12:23:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:09.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:23:09.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.732+0000 7fa3f59d0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f009bb70 0x7fa3f01c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f009bb70 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:09.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0094710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f009c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:09.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3f009d3f0 0x7fa3f01c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f009d3f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:09.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f008de90 con 0x7fa3f009bb70 2023-10-30T12:23:09.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f01c59a0 con 0x7fa3f009d3f0 2023-10-30T12:23:09.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f008fbf0 con 0x7fa3f009c660 2023-10-30T12:23:09.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:09.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient(hunting): authenticate will time out at 2234.916066s 2023-10-30T12:23:09.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0094710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f009bb70 0x7fa3f01c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0094710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60518/0 (socket says 172.21.15.50:60518) 2023-10-30T12:23:09.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 -- 172.21.15.50:0/1643505982 learned_addr learned my addr 172.21.15.50:0/1643505982 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:09.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f009bb70 0x7fa3f01c4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59636/0 (socket says 172.21.15.50:59636) 2023-10-30T12:23:09.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa3f009c660 auth_method 0 2023-10-30T12:23:09.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 10 monclient(hunting): get_auth_request con 0x7fa3f009bb70 auth_method 0 2023-10-30T12:23:09.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3f009d3f0 0x7fa3f01c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa3f009d3f0 auth_method 0 2023-10-30T12:23:09.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient(hunting): handle_auth_done global_id 284102 payload 824 2023-10-30T12:23:09.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3f009d3f0 msgr2=0x7fa3f01c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3f009d3f0 0x7fa3f01c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f009bb70 msgr2=0x7fa3f01c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f009bb70 0x7fa3f01c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:09.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient: _finish_hunting 0 2023-10-30T12:23:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 monclient: found mon.noname-c 2023-10-30T12:23:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:09.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 -- 172.21.15.50:0/1643505982 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e8009b30 con 0x7fa3f009c660 2023-10-30T12:23:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3eeffd640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f009bb70 0x7fa3f01c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ee7fc640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0094710 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400e9b0 tx=0x7fa3e400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 client.?.objecter ms_handle_connect 0x7fa3f009c660 2023-10-30T12:23:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4009670 con 0x7fa3f009c660 2023-10-30T12:23:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:09.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: dump: 2023-10-30T12:23:09.794 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.794 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.794 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.795 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.795 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.795 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.796 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.796 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.796 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.796 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: _finish_auth 0 2023-10-30T12:23:09.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:09.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:09.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 5 monclient: authenticate success, global_id 284102 2023-10-30T12:23:09.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e400c9a0 con 0x7fa3f009c660 2023-10-30T12:23:09.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:09.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient: _renew_subs 2023-10-30T12:23:09.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e400cd50 con 0x7fa3f009c660 2023-10-30T12:23:09.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:09.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f00976e0 con 0x7fa3f009c660 2023-10-30T12:23:09.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:09.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3d7fff640 10 monclient: dump: 2023-10-30T12:23:09.800 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.800 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.801 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.801 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.801 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.801 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.802 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.802 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.802 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.803 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3ed7fa640 4 set_mon_vals no callback set 2023-10-30T12:23:09.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 client.284102.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:09.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient: _renew_subs 2023-10-30T12:23:09.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:09.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f00901b0 con 0x7fa3f009c660 2023-10-30T12:23:09.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.733+0000 7fa3f59d0640 1 librados: init done 2023-10-30T12:23:09.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fa3e4013470 con 0x7fa3f009c660 2023-10-30T12:23:09.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3f59d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:09.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3f59d0640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f0094c50 0x7fa3f0095030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3f59d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0073ff0 con 0x7fa3f0094c50 2023-10-30T12:23:09.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3d7fff640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3dc0392f0 0x7fa3dc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7fa3e404acf0 con 0x7fa3f009c660 2023-10-30T12:23:09.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3d7fff640 10 client.284102.objecter ms_dispatch 0x7fa3f0000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:09.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3d7fff640 3 client.284102.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:09.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3d7fff640 3 client.284102.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:09.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.734+0000 7fa3eeffd640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f0094c50 0x7fa3f0095030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient: get_auth_request con 0x7fa3f0094c50 auth_method 0 2023-10-30T12:23:09.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 20 client.284102.objecter dump_active .. 0 homeless 2023-10-30T12:23:09.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3ef7fe640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3dc0392f0 0x7fa3dc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3ef7fe640 10 monclient: get_auth_request con 0x7fa3dc0392f0 auth_method 0 2023-10-30T12:23:09.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_done global_id 264315 payload 824 2023-10-30T12:23:09.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 1 -- 172.21.15.50:0/1643505982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3f0097560 con 0x7fa3f0094c50 2023-10-30T12:23:09.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 client.284102.objecter ms_handle_connect 0x7fa3f0094c50 2023-10-30T12:23:09.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 client.284102.objecter resend_mon_ops 2023-10-30T12:23:09.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3eeffd640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f0094c50 0x7fa3f0095030 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0000fb0 tx=0x7fa3e000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e00052e0 con 0x7fa3f0094c50 2023-10-30T12:23:09.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:09.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 monclient: dump: 2023-10-30T12:23:09.812 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.813 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.813 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.813 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.814 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.814 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.814 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.814 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.815 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.815 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 client.284102.objecter ms_handle_connect 0x7fa3dc0392f0 2023-10-30T12:23:09.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3ef7fe640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3dc0392f0 0x7fa3dc03b7b0 secure :-1 s=READY pgs=4349 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8035000 tx=0x7fa3e8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fa3e0005440 con 0x7fa3f0094c50 2023-10-30T12:23:09.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:09.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:09.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f0094c50 msgr2=0x7fa3f0095030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f0094c50 0x7fa3f0095030 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0000fb0 tx=0x7fa3e000ff70 comp rx=0 tx=0).stop 2023-10-30T12:23:09.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f0094c50 msgr2=0x7fa3f0095030 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:23:09.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.735+0000 7fa3d7fff640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f0094c50 0x7fa3f0095030 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0000fb0 tx=0x7fa3e000ff70 comp rx=0 tx=0).stop 2023-10-30T12:23:09.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3f59d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:23:09.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3f59d0640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f006fdc0 0x7fa3f00701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:09.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3f59d0640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:09.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f009f600 con 0x7fa3f006fdc0 2023-10-30T12:23:09.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f006fdc0 0x7fa3f00701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:09.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient: get_auth_request con 0x7fa3f006fdc0 auth_method 0 2023-10-30T12:23:09.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:09.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:09.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:09.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:09.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:09.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:09.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 10 monclient(hunting): handle_auth_done global_id 264318 payload 824 2023-10-30T12:23:09.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 1 -- 172.21.15.50:0/1643505982 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa3f004e160 con 0x7fa3f006fdc0 2023-10-30T12:23:09.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 10 client.284102.objecter ms_handle_connect 0x7fa3f006fdc0 2023-10-30T12:23:09.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 10 client.284102.objecter resend_mon_ops 2023-10-30T12:23:09.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3eeffd640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f006fdc0 0x7fa3f00701d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0017000 tx=0x7fa3e0001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:09.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0024070 con 0x7fa3f006fdc0 2023-10-30T12:23:09.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:09.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:09.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 10 monclient: dump: 2023-10-30T12:23:09.823 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:09.823 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.824 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.824 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:09.824 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:09.824 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.825 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:09.825 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:09.825 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:09.825 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4634 (secure 0 0 0) 0x7fa3e0003c20 con 0x7fa3f006fdc0 2023-10-30T12:23:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:23:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:23:09.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f006fdc0 msgr2=0x7fa3f00701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f006fdc0 0x7fa3f00701d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0017000 tx=0x7fa3e0001170 comp rx=0 tx=0).stop 2023-10-30T12:23:09.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f006fdc0 msgr2=0x7fa3f00701d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:23:09.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.742+0000 7fa3d7fff640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3f006fdc0 0x7fa3f00701d0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0017000 tx=0x7fa3e0001170 comp rx=0 tx=0).stop 2023-10-30T12:23:09.828 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:09.828 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:23:09.829 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:23:09.829 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2023-10-30T12:23:09.829 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 228, 2023-10-30T12:23:09.829 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:23:09.830 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:23:09.830 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:23:09.830 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:23:09.830 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:09.831 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:23:09.831 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:23:09.831 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:23:09.831 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:23:09.832 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:09.832 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:09.832 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:09.833 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:09.833 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:09.833 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:09.834 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:09.834 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:09.834 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:09.834 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:09.835 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:09.835 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:23:09.835 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:23:09.835 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:09.836 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:09.836 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:09.836 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:09.837 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:09.837 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:09.837 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:09.837 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:09.838 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:09.838 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:09.838 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:09.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.839 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:23:09.839 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:23:09.839 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:23:09.839 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:23:09.840 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:23:09.840 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:23:09.840 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:23:09.840 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:23:09.841 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:23:09.841 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:23:09.842 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:23:09.842 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:23:09.842 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:23:09.842 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:23:09.843 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:23:09.843 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:23:09.843 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:23:09.843 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:09.844 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:09.844 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:09.844 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:09.845 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:09.845 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:09.845 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:09.845 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:09.846 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:09.846 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:09.846 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:09.846 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:23:09.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.847 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:23:09.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.847 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:23:09.848 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:23:09.848 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:09.848 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:09.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:09.849 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:23:09.849 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:09.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.850 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:09.850 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:23:09.851 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:09.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:09.851 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:09.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.852 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:23:09.852 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:23:09.852 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:09.852 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:09.853 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:09.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.854 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:23:09.854 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:23:09.854 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:09.854 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:09.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.855 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:09.855 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:23:09.855 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:09.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:09.857 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:23:09.857 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:09.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:09.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:09.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.858 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:23:09.858 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:23:09.859 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:09.859 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:09.859 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:09.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.860 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:23:09.860 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:23:09.860 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:09.861 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:09.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.861 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:09.862 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:23:09.862 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:09.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.863 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:09.863 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:23:09.863 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:09.863 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:09.864 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:09.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.864 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:23:09.865 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:23:09.865 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:09.865 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:09.865 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:09.866 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:09.866 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:09.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.867 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:23:09.867 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:23:09.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.867 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:09.868 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:09.868 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:23:09.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:09.868 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:09.869 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2023-10-30T12:23:09.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.869 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:09.869 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:09.870 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6 2023-10-30T12:23:09.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:09.870 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:09.871 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:23:09.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:09.871 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:09.871 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:09.872 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5 2023-10-30T12:23:09.872 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:09.872 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:09.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:09.873 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:23:09.873 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:09.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 10 librados: watch_flush enter 2023-10-30T12:23:09.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 10 librados: watch_flush exit 2023-10-30T12:23:09.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 20 client.284102.objecter shutdown clearing up homeless session... 2023-10-30T12:23:09.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 10 client.284102.objecter successfully canceled tick 2023-10-30T12:23:09.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3dc0392f0 msgr2=0x7fa3dc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fa3dc0392f0 0x7fa3dc03b7b0 secure :-1 s=READY pgs=4349 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8035000 tx=0x7fa3e8030000 comp rx=0 tx=0).stop 2023-10-30T12:23:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 10 monclient: shutdown 2023-10-30T12:23:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 msgr2=0x7fa3f0094710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:09.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 --2- 172.21.15.50:0/1643505982 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa3f009c660 0x7fa3f0094710 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400e9b0 tx=0x7fa3e400ee70 comp rx=0 tx=0).stop 2023-10-30T12:23:09.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3ef7fe640 1 -- 172.21.15.50:0/1643505982 reap_dead start 2023-10-30T12:23:09.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3ef7fe640 1 -- 172.21.15.50:0/1643505982 reap_dead start 2023-10-30T12:23:09.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 shutdown_connections 2023-10-30T12:23:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 >> 172.21.15.50:0/1643505982 conn(0x7fa3f00a3980 msgr2=0x7fa3f0096590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 shutdown_connections 2023-10-30T12:23:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 -- 172.21.15.50:0/1643505982 wait complete. 2023-10-30T12:23:09.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.743+0000 7fa3f59d0640 1 librados: shutdown 2023-10-30T12:23:09.880 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:23:09.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-10-30T12:23:09.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-10-30T12:23:09.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.985+0000 7fefae67c640 10 monclient: get_monmap_and_config 2023-10-30T12:23:09.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.985+0000 7fefae67c640 10 monclient: build_initial_monmap 2023-10-30T12:23:09.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.985+0000 7fefae67c640 10 monclient: monmap: 2023-10-30T12:23:09.988 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:09.988 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:09.988 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:09.986576+0000 2023-10-30T12:23:09.989 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:09.986576+0000 2023-10-30T12:23:09.989 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:09.989 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:09.989 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:09.990 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:09.990 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:09.990 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:09.990 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:09.991 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:09.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 1 Processor -- start 2023-10-30T12:23:10.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 1 -- start start 2023-10-30T12:23:10.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient: init 2023-10-30T12:23:10.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:10.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-10-30T12:23:10.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fefa809bb70 0x7fefa809bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient(hunting): picked mon.noname-c con 0x7fefa809bb70 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:10.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fefa809d3f0 0x7fefa809c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient(hunting): picked mon.noname-a con 0x7fefa809d3f0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:10.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefa8096080 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:10.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa8098170 con 0x7fefa8096080 2023-10-30T12:23:10.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa809c9d0 con 0x7fefa809d3f0 2023-10-30T12:23:10.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa809cf60 con 0x7fefa809bb70 2023-10-30T12:23:10.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:10.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefae67c640 10 monclient(hunting): authenticate will time out at 2235.169753s 2023-10-30T12:23:10.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefa77fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fefa809d3f0 0x7fefa809c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefa77fe640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fefa809d3f0 0x7fefa809c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59674/0 (socket says 172.21.15.50:59674) 2023-10-30T12:23:10.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefa77fe640 1 -- 172.21.15.50:0/1176236755 learned_addr learned my addr 172.21.15.50:0/1176236755 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:10.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefa77fe640 10 monclient(hunting): get_auth_request con 0x7fefa809d3f0 auth_method 0 2023-10-30T12:23:10.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefa77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.986+0000 7fefa77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa7fff640 1 --2- 172.21.15.50:0/1176236755 >> v2:172.21.15.118:3300/0 conn(0x7fefa809bb70 0x7fefa809bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa7fff640 10 monclient(hunting): get_auth_request con 0x7fefa809bb70 auth_method 0 2023-10-30T12:23:10.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 10 monclient(hunting): handle_auth_done global_id 264321 payload 274 2023-10-30T12:23:10.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 1 -- 172.21.15.50:0/1176236755 >> v2:172.21.15.118:3300/0 conn(0x7fefa809bb70 msgr2=0x7fefa809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:10.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 1 --2- 172.21.15.50:0/1176236755 >> v2:172.21.15.118:3300/0 conn(0x7fefa809bb70 0x7fefa809bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 1 -- 172.21.15.50:0/1176236755 >> v1:172.21.15.118:6789/0 conn(0x7fefa8096080 legacy=0x7fefa8096510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:10.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 1 monclient: found mon.noname-a 2023-10-30T12:23:10.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:10.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 1 -- 172.21.15.50:0/1176236755 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8097b20 con 0x7fefa809d3f0 2023-10-30T12:23:10.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa7fff640 1 --2- 172.21.15.50:0/1176236755 >> v2:172.21.15.118:3300/0 conn(0x7fefa809bb70 0x7fefa809bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:23:10.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.987+0000 7fefa77fe640 1 --2- 172.21.15.50:0/1176236755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 0x7fefa809c490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fefa806b140 tx=0x7fef9c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721be49cc20e29ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 1 -- 172.21.15.50:0/1176236755 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0317e0 con 0x7fefa809d3f0 2023-10-30T12:23:10.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:10.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: dump: 2023-10-30T12:23:10.013 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.013 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.013 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.014 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.014 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.014 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.015 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.015 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.015 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.015 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:23:10.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:10.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:10.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 1 -- 172.21.15.50:0/1176236755 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9c031940 con 0x7fefa809d3f0 2023-10-30T12:23:10.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 5 monclient: authenticate success, global_id 264321 2023-10-30T12:23:10.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:10.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:10.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:10.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 1 -- 172.21.15.50:0/1176236755 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c031cd0 con 0x7fefa809d3f0 2023-10-30T12:23:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 4 set_mon_vals no callback set 2023-10-30T12:23:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefa6ffd640 10 monclient: dump: 2023-10-30T12:23:10.020 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.020 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.020 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.021 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.021 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.021 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.021 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.022 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.022 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.022 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 10 monclient: shutdown 2023-10-30T12:23:10.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:10.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 -- 172.21.15.50:0/1176236755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 msgr2=0x7fefa809c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 --2- 172.21.15.50:0/1176236755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 0x7fefa809c490 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fefa806b140 tx=0x7fef9c02b880 comp rx=0 tx=0).stop 2023-10-30T12:23:10.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 -- 172.21.15.50:0/1176236755 shutdown_connections 2023-10-30T12:23:10.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 --2- 172.21.15.50:0/1176236755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 0x7fefa809c490 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 --2- 172.21.15.50:0/1176236755 >> v2:172.21.15.118:3300/0 conn(0x7fefa809bb70 0x7fefa809bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 -- 172.21.15.50:0/1176236755 >> 172.21.15.50:0/1176236755 conn(0x7fefa80a3980 msgr2=0x7fefa80a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:10.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 -- 172.21.15.50:0/1176236755 shutdown_connections 2023-10-30T12:23:10.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.988+0000 7fefae67c640 1 -- 172.21.15.50:0/1176236755 wait complete. 2023-10-30T12:23:10.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 10 monclient: build_initial_monmap 2023-10-30T12:23:10.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 10 monclient: monmap: 2023-10-30T12:23:10.026 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:10.026 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:10.026 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:10.027 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:10.027 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:10.028 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.028 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:10.028 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:10.028 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:10.029 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 1 librados: starting msgr at 2023-10-30T12:23:10.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 1 librados: starting objecter 2023-10-30T12:23:10.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 1 Processor -- start 2023-10-30T12:23:10.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 1 -- start start 2023-10-30T12:23:10.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 1 librados: setting wanted keys 2023-10-30T12:23:10.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 1 librados: calling monclient init 2023-10-30T12:23:10.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.989+0000 7fefae67c640 10 monclient: init 2023-10-30T12:23:10.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:10.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:23:10.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8096080 0x7fefa81ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): picked mon.noname-b con 0x7fefa8096080 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:10.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 0x7fefa8094630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): picked mon.noname-c con 0x7fefa809bb70 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:10.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 0x7fefa8094b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): picked mon.noname-a con 0x7fefa809d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:10.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa81ca640 con 0x7fefa809d3f0 2023-10-30T12:23:10.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8091dc0 con 0x7fefa8096080 2023-10-30T12:23:10.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8092860 con 0x7fefa809bb70 2023-10-30T12:23:10.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:10.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefae67c640 10 monclient(hunting): authenticate will time out at 2235.173453s 2023-10-30T12:23:10.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 0x7fefa8094630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 0x7fefa8094b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 0x7fefa8094630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60532/0 (socket says 172.21.15.50:60532) 2023-10-30T12:23:10.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 1 -- 172.21.15.50:0/2088641304 learned_addr learned my addr 172.21.15.50:0/2088641304 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:10.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 10 monclient(hunting): get_auth_request con 0x7fefa809bb70 auth_method 0 2023-10-30T12:23:10.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): get_auth_request con 0x7fefa809d3f0 auth_method 0 2023-10-30T12:23:10.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8096080 0x7fefa81ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 msgr2=0x7fefa8094630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-10-30T12:23:10.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 msgr2=0x7fefa8094630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:23:10.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 0x7fefa8094630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:10.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 10 monclient(hunting): get_auth_request con 0x7fefa8096080 auth_method 0 2023-10-30T12:23:10.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa77fe640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 0x7fefa8094630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:23:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient(hunting): handle_auth_done global_id 264324 payload 824 2023-10-30T12:23:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 msgr2=0x7fefa8094630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:23:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa809bb70 0x7fefa8094630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8096080 msgr2=0x7fefa81ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8096080 0x7fefa81ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient: _finish_hunting 0 2023-10-30T12:23:10.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 1 monclient: found mon.noname-a 2023-10-30T12:23:10.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:10.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.990+0000 7fefacc19640 1 -- 172.21.15.50:0/2088641304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0094d0 con 0x7fefa809d3f0 2023-10-30T12:23:10.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefacc19640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 0x7fefa8094b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fefa000c690 tx=0x7fefa000cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 client.?.objecter ms_handle_connect 0x7fefa809d3f0 2023-10-30T12:23:10.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:10.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0003b70 con 0x7fefa809d3f0 2023-10-30T12:23:10.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:10.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: dump: 2023-10-30T12:23:10.050 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.050 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.053 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:23:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:23:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa0003cd0 con 0x7fefa809d3f0 2023-10-30T12:23:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 5 monclient: authenticate success, global_id 264324 2023-10-30T12:23:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 10 monclient: _renew_subs 2023-10-30T12:23:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa80976e0 con 0x7fefa809d3f0 2023-10-30T12:23:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa0005310 con 0x7fefa809d3f0 2023-10-30T12:23:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa4ff9640 10 monclient: dump: 2023-10-30T12:23:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.058 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.058 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.058 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.058 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.059 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.059 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefa67fc640 4 set_mon_vals no callback set 2023-10-30T12:23:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 10 client.264324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 10 monclient: _renew_subs 2023-10-30T12:23:10.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa80950b0 con 0x7fefa809d3f0 2023-10-30T12:23:10.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.991+0000 7fefae67c640 1 librados: init done 2023-10-30T12:23:10.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefae67c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:10.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefae67c640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa8090d00 0x7fefa808f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefae67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8073ff0 con 0x7fefa8090d00 2023-10-30T12:23:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fefa0005470 con 0x7fefa809d3f0 2023-10-30T12:23:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa8090d00 0x7fefa808f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient: get_auth_request con 0x7fefa8090d00 auth_method 0 2023-10-30T12:23:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef940392f0 0x7fef9403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7fefa004a660 con 0x7fefa809d3f0 2023-10-30T12:23:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 10 client.264324.objecter ms_dispatch 0x7fefa8000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 3 client.264324.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 3 client.264324.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 10 monclient(hunting): handle_auth_done global_id 284105 payload 824 2023-10-30T12:23:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 1 -- 172.21.15.50:0/2088641304 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefa8097560 con 0x7fefa8090d00 2023-10-30T12:23:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 20 client.264324.objecter dump_active .. 0 homeless 2023-10-30T12:23:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa77fe640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef940392f0 0x7fef9403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa77fe640 10 monclient: get_auth_request con 0x7fef940392f0 auth_method 0 2023-10-30T12:23:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 10 client.264324.objecter ms_handle_connect 0x7fefa8090d00 2023-10-30T12:23:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 10 client.264324.objecter resend_mon_ops 2023-10-30T12:23:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa7fff640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa8090d00 0x7fefa808f3e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef98011600 tx=0x7fef98011ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98004300 con 0x7fefa8090d00 2023-10-30T12:23:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:10.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.992+0000 7fefa4ff9640 10 monclient: dump: 2023-10-30T12:23:10.070 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.070 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.071 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.071 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.071 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.071 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.072 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.072 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.072 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.072 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 10 client.264324.objecter ms_handle_connect 0x7fef940392f0 2023-10-30T12:23:10.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa77fe640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef940392f0 0x7fef9403b7b0 secure :-1 s=READY pgs=4350 cs=0 l=1 rev1=1 crypto rx=0x7fefa81bda00 tx=0x7fef9c03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7fef98004460 con 0x7fefa8090d00 2023-10-30T12:23:10.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:10.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:10.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa8090d00 msgr2=0x7fefa808f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa8090d00 0x7fefa808f3e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef98011600 tx=0x7fef98011ac0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa8090d00 msgr2=0x7fefa808f3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:23:10.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.993+0000 7fefa4ff9640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa8090d00 0x7fefa808f3e0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef98011600 tx=0x7fef98011ac0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefae67c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:23:10.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefae67c640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa808f920 0x7fefa808fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefae67c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8092ba0 con 0x7fefa808f920 2023-10-30T12:23:10.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa808f920 0x7fefa808fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient: get_auth_request con 0x7fefa808f920 auth_method 0 2023-10-30T12:23:10.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 10 monclient(hunting): handle_auth_done global_id 284108 payload 824 2023-10-30T12:23:10.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:09.999+0000 7fefa7fff640 1 -- 172.21.15.50:0/2088641304 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fefa8097e00 con 0x7fefa808f920 2023-10-30T12:23:10.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa7fff640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa808f920 0x7fefa808fd30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fef98011600 tx=0x7fef980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 10 client.264324.objecter ms_handle_connect 0x7fefa808f920 2023-10-30T12:23:10.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 10 client.264324.objecter resend_mon_ops 2023-10-30T12:23:10.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98025070 con 0x7fefa808f920 2023-10-30T12:23:10.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:10.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 10 monclient: dump: 2023-10-30T12:23:10.082 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.082 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.083 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.083 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.083 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.084 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.084 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.084 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.084 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.085 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4462 (secure 0 0 0) 0x7fef98003610 con 0x7fefa808f920 2023-10-30T12:23:10.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:23:10.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:23:10.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa808f920 msgr2=0x7fefa808fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa808f920 0x7fefa808fd30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fef98011600 tx=0x7fef980046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa808f920 msgr2=0x7fefa808fd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:23:10.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefa4ff9640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fefa808f920 0x7fefa808fd30 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fef98011600 tx=0x7fef980046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.087 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:10.087 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:23:10.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:23:10.088 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:23:10.088 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 228, 2023-10-30T12:23:10.088 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:23:10.089 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:23:10.089 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:23:10.089 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:23:10.089 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.090 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:23:10.090 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:23:10.090 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:23:10.091 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:23:10.091 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:10.091 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:10.091 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:10.092 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:10.092 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:10.092 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:10.092 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:10.093 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:10.093 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:10.093 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:10.093 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.094 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:23:10.094 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:23:10.094 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:10.095 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:10.095 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:10.095 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:10.095 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:10.096 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:10.096 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:10.096 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:10.096 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:10.097 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:10.097 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.097 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:23:10.098 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:23:10.098 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:23:10.098 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:23:10.098 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:23:10.099 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:23:10.099 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:23:10.099 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:23:10.099 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:23:10.100 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:23:10.100 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:23:10.100 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:23:10.100 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:23:10.101 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:23:10.101 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:23:10.101 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:23:10.101 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:23:10.102 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:10.102 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:10.102 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:10.102 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:10.103 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:10.103 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:10.103 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:10.104 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:10.104 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:10.104 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:10.104 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.105 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:23:10.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.105 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:23:10.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.106 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:23:10.106 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:23:10.106 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:10.106 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:10.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.107 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:10.107 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:23:10.107 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.108 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:10.109 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:23:10.110 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.110 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.110 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.111 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:23:10.111 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:23:10.111 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:10.111 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:10.112 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:10.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.112 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:23:10.113 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:23:10.113 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:10.113 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:10.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:10.114 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:23:10.114 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.115 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:10.115 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:23:10.116 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.116 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.116 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.117 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:23:10.117 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:23:10.117 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:10.117 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:10.118 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:10.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.118 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:23:10.119 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:23:10.119 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:10.119 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:10.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.120 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:10.120 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:23:10.120 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.121 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:10.121 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:23:10.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.122 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.123 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:23:10.123 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:23:10.123 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:10.123 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:10.124 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:10.124 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.124 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.125 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:23:10.125 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:23:10.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.126 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:10.126 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:10.126 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:23:10.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.127 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.127 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:23:10.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.127 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:10.128 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:10.128 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:23:10.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.128 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.129 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:23:10.129 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:10.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 10 librados: watch_flush enter 2023-10-30T12:23:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 10 librados: watch_flush exit 2023-10-30T12:23:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 20 client.264324.objecter shutdown clearing up homeless session... 2023-10-30T12:23:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 10 client.264324.objecter successfully canceled tick 2023-10-30T12:23:10.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef940392f0 msgr2=0x7fef9403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fef940392f0 0x7fef9403b7b0 secure :-1 s=READY pgs=4350 cs=0 l=1 rev1=1 crypto rx=0x7fefa81bda00 tx=0x7fef9c03d040 comp rx=0 tx=0).stop 2023-10-30T12:23:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 10 monclient: shutdown 2023-10-30T12:23:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:10.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 msgr2=0x7fefa8094b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 --2- 172.21.15.50:0/2088641304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fefa809d3f0 0x7fefa8094b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fefa000c690 tx=0x7fefa000cb50 comp rx=0 tx=0).stop 2023-10-30T12:23:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefacc19640 1 -- 172.21.15.50:0/2088641304 reap_dead start 2023-10-30T12:23:10.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefacc19640 1 -- 172.21.15.50:0/2088641304 reap_dead start 2023-10-30T12:23:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 shutdown_connections 2023-10-30T12:23:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 >> 172.21.15.50:0/2088641304 conn(0x7fefa80a3980 msgr2=0x7fefa809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 shutdown_connections 2023-10-30T12:23:10.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 -- 172.21.15.50:0/2088641304 wait complete. 2023-10-30T12:23:10.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.000+0000 7fefae67c640 1 librados: shutdown 2023-10-30T12:23:10.136 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:23:10.136 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-10-30T12:23:10.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-10-30T12:23:10.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.243+0000 7febe031d640 10 monclient: get_monmap_and_config 2023-10-30T12:23:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.243+0000 7febe031d640 10 monclient: build_initial_monmap 2023-10-30T12:23:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.243+0000 7febe031d640 10 monclient: monmap: 2023-10-30T12:23:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:10.247 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.247 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:10.244699+0000 2023-10-30T12:23:10.247 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:10.244699+0000 2023-10-30T12:23:10.247 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:10.248 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.248 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:10.248 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:10.249 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 1 Processor -- start 2023-10-30T12:23:10.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 1 -- start start 2023-10-30T12:23:10.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient: init 2023-10-30T12:23:10.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:10.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-10-30T12:23:10.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7febd8132970 0x7febd8132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient(hunting): picked mon.noname-a con 0x7febd8132970 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:10.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd80a03d0 0x7febd80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient(hunting): picked mon.noname-b con 0x7febd80a03d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:10.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febd8070d10 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:10.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd80a17d0 con 0x7febd8070d10 2023-10-30T12:23:10.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd80a1510 con 0x7febd8132970 2023-10-30T12:23:10.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd809f680 con 0x7febd80a03d0 2023-10-30T12:23:10.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:10.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.244+0000 7febe031d640 10 monclient(hunting): authenticate will time out at 2235.427990s 2023-10-30T12:23:10.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7febd8132970 0x7febd8132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd80a03d0 0x7febd80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7febd8132970 0x7febd8132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59690/0 (socket says 172.21.15.50:59690) 2023-10-30T12:23:10.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd80a03d0 0x7febd80707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60570/0 (socket says 172.21.15.50:60570) 2023-10-30T12:23:10.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 1 -- 172.21.15.50:0/576193418 learned_addr learned my addr 172.21.15.50:0/576193418 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:10.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 10 monclient(hunting): get_auth_request con 0x7febd8132970 auth_method 0 2023-10-30T12:23:10.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde8ba640 1 --1- 172.21.15.50:0/576193418 >> v1:172.21.15.50:6789/0 conn(0x7febd8070d10 0x7febd813caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49550/0 (socket says 172.21.15.50:49550) 2023-10-30T12:23:10.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): get_auth_request con 0x7febd80a03d0 auth_method 0 2023-10-30T12:23:10.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient(hunting): handle_auth_done global_id 284111 payload 274 2023-10-30T12:23:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 1 -- 172.21.15.50:0/576193418 >> v2:172.21.15.50:3300/0 conn(0x7febd8132970 msgr2=0x7febd8132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:10.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 1 --2- 172.21.15.50:0/576193418 >> v2:172.21.15.50:3300/0 conn(0x7febd8132970 0x7febd8132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 1 -- 172.21.15.50:0/576193418 >> v1:172.21.15.50:6789/0 conn(0x7febd8070d10 legacy=0x7febd813caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient: _finish_hunting 0 2023-10-30T12:23:10.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 1 monclient: found mon.noname-b 2023-10-30T12:23:10.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:10.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:10.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd8b8640 1 -- 172.21.15.50:0/576193418 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd809f960 con 0x7febd80a03d0 2023-10-30T12:23:10.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd0b7640 1 -- 172.21.15.50:0/576193418 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221107484 0 0) 0x7febd80a17d0 con 0x7febd8070d10 2023-10-30T12:23:10.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febdd0b7640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:10.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.245+0000 7febde0b9640 1 --2- 172.21.15.50:0/576193418 >> v2:172.21.15.50:3300/0 conn(0x7febd8132970 0x7febd8132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:10.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd8b8640 1 --2- 172.21.15.50:0/576193418 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd80a03d0 0x7febd80707d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7febc80097a0 tx=0x7febc802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8d906d4fd7a06c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 1 -- 172.21.15.50:0/576193418 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc80316f0 con 0x7febd80a03d0 2023-10-30T12:23:10.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:10.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: dump: 2023-10-30T12:23:10.274 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.274 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.274 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.274 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.276 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.276 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: _finish_auth 0 2023-10-30T12:23:10.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: _check_auth_tickets 2023-10-30T12:23:10.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:10.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 5 monclient: authenticate success, global_id 284111 2023-10-30T12:23:10.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:10.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 1 -- 172.21.15.50:0/576193418 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc8031850 con 0x7febd80a03d0 2023-10-30T12:23:10.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:10.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:10.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 4 set_mon_vals no callback set 2023-10-30T12:23:10.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 1 -- 172.21.15.50:0/576193418 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8031be0 con 0x7febd80a03d0 2023-10-30T12:23:10.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:10.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febdd0b7640 10 monclient: dump: 2023-10-30T12:23:10.279 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.279 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.280 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.280 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.280 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.280 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.281 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.281 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.281 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.281 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 10 monclient: shutdown 2023-10-30T12:23:10.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:10.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 1 -- 172.21.15.50:0/576193418 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd80a03d0 msgr2=0x7febd80707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 1 --2- 172.21.15.50:0/576193418 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd80a03d0 0x7febd80707d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7febc80097a0 tx=0x7febc802b790 comp rx=0 tx=0).stop 2023-10-30T12:23:10.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 1 -- 172.21.15.50:0/576193418 shutdown_connections 2023-10-30T12:23:10.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 1 --2- 172.21.15.50:0/576193418 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd80a03d0 0x7febd80707d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 1 --2- 172.21.15.50:0/576193418 >> v2:172.21.15.50:3300/0 conn(0x7febd8132970 0x7febd8132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.246+0000 7febe031d640 1 -- 172.21.15.50:0/576193418 >> 172.21.15.50:0/576193418 conn(0x7febd80a4800 msgr2=0x7febd80a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:10.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 -- 172.21.15.50:0/576193418 shutdown_connections 2023-10-30T12:23:10.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 -- 172.21.15.50:0/576193418 wait complete. 2023-10-30T12:23:10.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 10 monclient: build_initial_monmap 2023-10-30T12:23:10.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 10 monclient: monmap: 2023-10-30T12:23:10.285 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:10.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:10.285 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:10.285 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:10.286 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:10.286 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:10.286 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:10.287 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:10.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 librados: starting msgr at 2023-10-30T12:23:10.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 librados: starting objecter 2023-10-30T12:23:10.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 Processor -- start 2023-10-30T12:23:10.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 -- start start 2023-10-30T12:23:10.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 librados: setting wanted keys 2023-10-30T12:23:10.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 1 librados: calling monclient init 2023-10-30T12:23:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.247+0000 7febe031d640 10 monclient: init 2023-10-30T12:23:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:10.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:23:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd8070d10 0x7febd81ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): picked mon.noname-a con 0x7febd8070d10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8132970 0x7febd812fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): picked mon.noname-b con 0x7febd8132970 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd81ca710 0x7febd8130040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): picked mon.noname-c con 0x7febd81ca710 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd811b890 con 0x7febd8070d10 2023-10-30T12:23:10.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd809d7c0 con 0x7febd8132970 2023-10-30T12:23:10.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd811da00 con 0x7febd81ca710 2023-10-30T12:23:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febe031d640 10 monclient(hunting): authenticate will time out at 2235.431348s 2023-10-30T12:23:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd8070d10 0x7febd81ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd81ca710 0x7febd8130040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd8070d10 0x7febd81ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59696/0 (socket says 172.21.15.50:59696) 2023-10-30T12:23:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 -- 172.21.15.50:0/2848275934 learned_addr learned my addr 172.21.15.50:0/2848275934 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): get_auth_request con 0x7febd8070d10 auth_method 0 2023-10-30T12:23:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 10 monclient(hunting): get_auth_request con 0x7febd81ca710 auth_method 0 2023-10-30T12:23:10.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febdd8b8640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8132970 0x7febd812fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febdd8b8640 10 monclient(hunting): get_auth_request con 0x7febd8132970 auth_method 0 2023-10-30T12:23:10.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febdd8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febdd8b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febdd8b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient(hunting): handle_auth_done global_id 264333 payload 824 2023-10-30T12:23:10.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd81ca710 msgr2=0x7febd8130040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd81ca710 0x7febd8130040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8132970 msgr2=0x7febd812fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8132970 0x7febd812fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient: _finish_hunting 0 2023-10-30T12:23:10.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 monclient: found mon.noname-a 2023-10-30T12:23:10.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:10.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 -- 172.21.15.50:0/2848275934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd40094d0 con 0x7febd8070d10 2023-10-30T12:23:10.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde8ba640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7febd81ca710 0x7febd8130040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febde0b9640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd8070d10 0x7febd81ca1d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7febd400e9d0 tx=0x7febd400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febceffd640 10 client.?.objecter ms_handle_connect 0x7febd8070d10 2023-10-30T12:23:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.248+0000 7febceffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd400ca60 con 0x7febd8070d10 2023-10-30T12:23:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: dump: 2023-10-30T12:23:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.305 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.305 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.305 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.305 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.306 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: _finish_auth 0 2023-10-30T12:23:10.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd400cbc0 con 0x7febd8070d10 2023-10-30T12:23:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 5 monclient: authenticate success, global_id 264333 2023-10-30T12:23:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 10 monclient: _renew_subs 2023-10-30T12:23:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc8009400 con 0x7febd8070d10 2023-10-30T12:23:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4013470 con 0x7febd8070d10 2023-10-30T12:23:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 10 monclient: dump: 2023-10-30T12:23:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.312 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 10 client.264333.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febdc8b6640 4 set_mon_vals no callback set 2023-10-30T12:23:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 10 monclient: _renew_subs 2023-10-30T12:23:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd811dfc0 con 0x7febd8070d10 2023-10-30T12:23:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 1 librados: init done 2023-10-30T12:23:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:10.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd80a03d0 0x7febd811a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd8071a70 con 0x7febd80a03d0 2023-10-30T12:23:10.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.249+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7febd40135d0 con 0x7febd8070d10 2023-10-30T12:23:10.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd80a03d0 0x7febd811a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient: get_auth_request con 0x7febd80a03d0 auth_method 0 2023-10-30T12:23:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7febbc0392f0 0x7febbc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7febd404ace0 con 0x7febd8070d10 2023-10-30T12:23:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 10 client.264333.objecter ms_dispatch 0x7febd8000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 3 client.264333.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 3 client.264333.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 20 client.264333.objecter dump_active .. 0 homeless 2023-10-30T12:23:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 10 monclient(hunting): handle_auth_done global_id 274192 payload 824 2023-10-30T12:23:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 1 -- 172.21.15.50:0/2848275934 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7febd8133290 con 0x7febd80a03d0 2023-10-30T12:23:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febde8ba640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7febbc0392f0 0x7febbc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febde8ba640 10 monclient: get_auth_request con 0x7febbc0392f0 auth_method 0 2023-10-30T12:23:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 10 client.264333.objecter ms_handle_connect 0x7febd80a03d0 2023-10-30T12:23:10.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 10 client.264333.objecter resend_mon_ops 2023-10-30T12:23:10.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febdd8b8640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd80a03d0 0x7febd811a430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febd80a1c90 tx=0x7febc80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8034680 con 0x7febd80a03d0 2023-10-30T12:23:10.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:23:10.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 10 monclient: dump: 2023-10-30T12:23:10.323 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.324 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.324 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.324 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.324 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.326 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febceffd640 10 client.264333.objecter ms_handle_connect 0x7febbc0392f0 2023-10-30T12:23:10.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.250+0000 7febde8ba640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7febbc0392f0 0x7febbc03b7b0 secure :-1 s=READY pgs=4351 cs=0 l=1 rev1=1 crypto rx=0x7febd81cab10 tx=0x7febd00016b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.251+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8460 (secure 0 0 0) 0x7febc80347e0 con 0x7febd80a03d0 2023-10-30T12:23:10.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.251+0000 7febceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:10.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.251+0000 7febceffd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:10.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.251+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd80a03d0 msgr2=0x7febd811a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.251+0000 7febceffd640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd80a03d0 0x7febd811a430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febd80a1c90 tx=0x7febc80046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.251+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd80a03d0 msgr2=0x7febd811a430 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:23:10.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.251+0000 7febceffd640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd80a03d0 0x7febd811a430 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febd80a1c90 tx=0x7febc80046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febe031d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2023-10-30T12:23:10.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febe031d640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd811a970 0x7febd811ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febe031d640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd8074800 con 0x7febd811a970 2023-10-30T12:23:10.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd811a970 0x7febd811ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient: get_auth_request con 0x7febd811a970 auth_method 0 2023-10-30T12:23:10.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 10 monclient(hunting): handle_auth_done global_id 274195 payload 824 2023-10-30T12:23:10.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.257+0000 7febdd8b8640 1 -- 172.21.15.50:0/2848275934 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7febd811db70 con 0x7febd811a970 2023-10-30T12:23:10.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 10 client.264333.objecter ms_handle_connect 0x7febd811a970 2023-10-30T12:23:10.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 10 client.264333.objecter resend_mon_ops 2023-10-30T12:23:10.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febdd8b8640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd811a970 0x7febd811ad50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febc802b7d0 tx=0x7febc8009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8048d10 con 0x7febd811a970 2023-10-30T12:23:10.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:23:10.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 10 monclient: dump: 2023-10-30T12:23:10.333 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.334 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.334 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.334 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.335 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.335 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.335 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.335 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.336 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.336 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 <== mon.1 v2:172.21.15.118:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4632 (secure 0 0 0) 0x7febc8048e70 con 0x7febd811a970 2023-10-30T12:23:10.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2023-10-30T12:23:10.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:23:10.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd811a970 msgr2=0x7febd811ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd811a970 0x7febd811ad50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febc802b7d0 tx=0x7febc8009190 comp rx=0 tx=0).stop 2023-10-30T12:23:10.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd811a970 msgr2=0x7febd811ad50 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:23:10.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febceffd640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd811a970 0x7febd811ad50 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febc802b7d0 tx=0x7febc8009190 comp rx=0 tx=0).stop 2023-10-30T12:23:10.338 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:10.339 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:23:10.339 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:23:10.339 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2023-10-30T12:23:10.339 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 228, 2023-10-30T12:23:10.340 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2023-10-30T12:23:10.340 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2023-10-30T12:23:10.340 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2023-10-30T12:23:10.340 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2023-10-30T12:23:10.341 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.341 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 0, 2023-10-30T12:23:10.341 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:23:10.341 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2023-10-30T12:23:10.342 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2023-10-30T12:23:10.342 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:10.342 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:10.342 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:10.344 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:10.344 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:10.344 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:10.344 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.345 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2023-10-30T12:23:10.345 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2023-10-30T12:23:10.345 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:10.345 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:10.346 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:10.346 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:10.346 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:10.346 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:10.347 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:10.347 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:10.347 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:10.347 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:10.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.348 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2023-10-30T12:23:10.348 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2023-10-30T12:23:10.349 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2023-10-30T12:23:10.349 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2023-10-30T12:23:10.349 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2023-10-30T12:23:10.349 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "87b22564-9727-4955-ac7b-d9a79464690c", 2023-10-30T12:23:10.350 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:23:10.350 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2023-10-30T12:00:17.894366Z", 2023-10-30T12:23:10.350 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2023-10-30T12:23:10.350 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2023-10-30T12:23:10.351 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2023-10-30T12:23:10.351 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2023-10-30T12:23:10.351 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2023-10-30T12:23:10.351 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2023-10-30T12:23:10.352 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2023-10-30T12:23:10.352 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2023-10-30T12:23:10.352 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2023-10-30T12:23:10.352 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2023-10-30T12:23:10.353 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2023-10-30T12:23:10.353 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2023-10-30T12:23:10.353 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2023-10-30T12:23:10.353 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2023-10-30T12:23:10.354 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2023-10-30T12:23:10.354 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2023-10-30T12:23:10.355 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2023-10-30T12:23:10.355 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2023-10-30T12:23:10.355 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2023-10-30T12:23:10.355 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.356 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2023-10-30T12:23:10.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.356 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2023-10-30T12:23:10.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.357 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2023-10-30T12:23:10.357 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2023-10-30T12:23:10.357 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:10.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:10.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.358 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:10.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3300", 2023-10-30T12:23:10.359 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.360 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:10.360 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2023-10-30T12:23:10.360 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.361 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2023-10-30T12:23:10.361 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2023-10-30T12:23:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.363 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2023-10-30T12:23:10.363 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2023-10-30T12:23:10.364 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:10.364 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:10.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:10.365 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:3300", 2023-10-30T12:23:10.365 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.366 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:10.366 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789", 2023-10-30T12:23:10.366 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.367 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.367 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.118:6789/0", 2023-10-30T12:23:10.368 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.118:6789/0", 2023-10-30T12:23:10.368 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:10.368 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:10.369 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:10.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.369 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2023-10-30T12:23:10.370 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2023-10-30T12:23:10.370 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2023-10-30T12:23:10.370 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2023-10-30T12:23:10.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2023-10-30T12:23:10.371 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:3301", 2023-10-30T12:23:10.371 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.372 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-10-30T12:23:10.372 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2023-10-30T12:23:10.373 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2023-10-30T12:23:10.373 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.373 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.374 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2023-10-30T12:23:10.374 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2023-10-30T12:23:10.374 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2023-10-30T12:23:10.374 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2023-10-30T12:23:10.375 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2023-10-30T12:23:10.375 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.375 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.376 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2023-10-30T12:23:10.376 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2023-10-30T12:23:10.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.376 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:10.377 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:10.377 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:23:10.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.378 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.378 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2023-10-30T12:23:10.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.378 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:10.379 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:10.379 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:23:10.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.380 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2023-10-30T12:23:10.380 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2023-10-30T12:23:10.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-10-30T12:23:10.380 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2023-10-30T12:23:10.381 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "luminous", 2023-10-30T12:23:10.381 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2023-10-30T12:23:10.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-10-30T12:23:10.381 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-10-30T12:23:10.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-10-30T12:23:10.382 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2023-10-30T12:23:10.382 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:10.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 10 librados: watch_flush enter 2023-10-30T12:23:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 10 librados: watch_flush exit 2023-10-30T12:23:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 20 client.264333.objecter shutdown clearing up homeless session... 2023-10-30T12:23:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 10 client.264333.objecter successfully canceled tick 2023-10-30T12:23:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7febbc0392f0 msgr2=0x7febbc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7febbc0392f0 0x7febbc03b7b0 secure :-1 s=READY pgs=4351 cs=0 l=1 rev1=1 crypto rx=0x7febd81cab10 tx=0x7febd00016b0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 10 monclient: shutdown 2023-10-30T12:23:10.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:10.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd8070d10 msgr2=0x7febd81ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febe031d640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd8070d10 0x7febd81ca1d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7febd400e9d0 tx=0x7febd400ee90 comp rx=0 tx=0).stop 2023-10-30T12:23:10.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.258+0000 7febde8ba640 1 -- 172.21.15.50:0/2848275934 reap_dead start 2023-10-30T12:23:10.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.259+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 shutdown_connections 2023-10-30T12:23:10.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.259+0000 7febe031d640 1 --2- 172.21.15.50:0/2848275934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd8070d10 0x7febd81ca1d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.259+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 >> 172.21.15.50:0/2848275934 conn(0x7febd80a4800 msgr2=0x7febd8131ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:10.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.259+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 shutdown_connections 2023-10-30T12:23:10.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.259+0000 7febe031d640 1 -- 172.21.15.50:0/2848275934 wait complete. 2023-10-30T12:23:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.259+0000 7febe031d640 1 librados: shutdown 2023-10-30T12:23:10.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-10-30T12:23:10.390 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-10-30T12:23:10.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-10-30T12:23:10.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.493+0000 7f1b72d55640 10 monclient: get_monmap_and_config 2023-10-30T12:23:10.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.493+0000 7f1b72d55640 10 monclient: build_initial_monmap 2023-10-30T12:23:10.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.493+0000 7f1b72d55640 10 monclient: monmap: 2023-10-30T12:23:10.496 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:10.496 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.497 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:10.494958+0000 2023-10-30T12:23:10.497 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:10.494958+0000 2023-10-30T12:23:10.497 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:10.497 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.498 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:10.498 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:10.498 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:10.498 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:10.499 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:10.499 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.494+0000 7f1b72d55640 1 Processor -- start 2023-10-30T12:23:10.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.494+0000 7f1b72d55640 1 -- start start 2023-10-30T12:23:10.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.494+0000 7f1b72d55640 10 monclient: init 2023-10-30T12:23:10.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.494+0000 7f1b72d55640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:10.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-10-30T12:23:10.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b6c09f5b0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:10.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1b6c099b80 0x7f1b6c096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient(hunting): picked mon.noname-c con 0x7f1b6c099b80 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:10.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1b6c138b30 0x7f1b6c13af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient(hunting): picked mon.noname-a con 0x7f1b6c138b30 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:10.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c098810 con 0x7f1b6c09f5b0 2023-10-30T12:23:10.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c06aef0 con 0x7f1b6c138b30 2023-10-30T12:23:10.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c09a1a0 con 0x7f1b6c099b80 2023-10-30T12:23:10.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:10.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b72d55640 10 monclient(hunting): authenticate will time out at 2235.678077s 2023-10-30T12:23:10.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1b6c138b30 0x7f1b6c13af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b70af1640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1b6c09f5b0 0x7f1b6c096080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49554/0 (socket says 172.21.15.50:49554) 2023-10-30T12:23:10.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1b6c138b30 0x7f1b6c13af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59712/0 (socket says 172.21.15.50:59712) 2023-10-30T12:23:10.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b70af1640 1 -- 172.21.15.50:0/439623703 learned_addr learned my addr 172.21.15.50:0/439623703 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:10.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): get_auth_request con 0x7f1b6c138b30 auth_method 0 2023-10-30T12:23:10.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b63fff640 1 --2- 172.21.15.50:0/439623703 >> v2:172.21.15.118:3300/0 conn(0x7f1b6c099b80 0x7f1b6c096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b63fff640 10 monclient(hunting): get_auth_request con 0x7f1b6c099b80 auth_method 0 2023-10-30T12:23:10.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient(hunting): handle_auth_done global_id 264336 payload 274 2023-10-30T12:23:10.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 1 -- 172.21.15.50:0/439623703 >> v2:172.21.15.118:3300/0 conn(0x7f1b6c099b80 msgr2=0x7f1b6c096790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:10.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 1 --2- 172.21.15.50:0/439623703 >> v2:172.21.15.118:3300/0 conn(0x7f1b6c099b80 0x7f1b6c096790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 1 -- 172.21.15.50:0/439623703 >> v1:172.21.15.50:6789/0 conn(0x7f1b6c09f5b0 legacy=0x7f1b6c096080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient: _finish_hunting 0 2023-10-30T12:23:10.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 1 monclient: found mon.noname-a 2023-10-30T12:23:10.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:10.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:10.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b712f2640 1 -- 172.21.15.50:0/439623703 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c098af0 con 0x7f1b6c138b30 2023-10-30T12:23:10.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b637fe640 1 -- 172.21.15.50:0/439623703 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315674274 0 0) 0x7f1b6c098810 con 0x7f1b6c09f5b0 2023-10-30T12:23:10.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.495+0000 7f1b637fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:10.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b712f2640 1 --2- 172.21.15.50:0/439623703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c138b30 0x7f1b6c13af20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009d90 tx=0x7f1b6402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d99491c3eb97a39 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 1 -- 172.21.15.50:0/439623703 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6402cb40 con 0x7f1b6c138b30 2023-10-30T12:23:10.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:10.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: dump: 2023-10-30T12:23:10.521 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.521 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.521 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.522 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.522 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.522 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.523 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.523 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.523 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.523 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: _finish_auth 0 2023-10-30T12:23:10.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: _check_auth_tickets 2023-10-30T12:23:10.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:10.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 5 monclient: authenticate success, global_id 264336 2023-10-30T12:23:10.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:10.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 1 -- 172.21.15.50:0/439623703 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6402cca0 con 0x7f1b6c138b30 2023-10-30T12:23:10.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:10.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:10.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 1 -- 172.21.15.50:0/439623703 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64034660 con 0x7f1b6c138b30 2023-10-30T12:23:10.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 4 set_mon_vals no callback set 2023-10-30T12:23:10.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b637fe640 10 monclient: dump: 2023-10-30T12:23:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.528 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.528 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.528 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.528 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.529 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.529 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.529 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.529 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.530 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 10 monclient: shutdown 2023-10-30T12:23:10.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:10.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 1 -- 172.21.15.50:0/439623703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c138b30 msgr2=0x7f1b6c13af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 1 --2- 172.21.15.50:0/439623703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c138b30 0x7f1b6c13af20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009d90 tx=0x7f1b6402bd90 comp rx=0 tx=0).stop 2023-10-30T12:23:10.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 1 -- 172.21.15.50:0/439623703 shutdown_connections 2023-10-30T12:23:10.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 1 --2- 172.21.15.50:0/439623703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c138b30 0x7f1b6c13af20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 1 --2- 172.21.15.50:0/439623703 >> v2:172.21.15.118:3300/0 conn(0x7f1b6c099b80 0x7f1b6c096790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.496+0000 7f1b72d55640 1 -- 172.21.15.50:0/439623703 >> 172.21.15.50:0/439623703 conn(0x7f1b6c0a05c0 msgr2=0x7f1b6c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:10.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 -- 172.21.15.50:0/439623703 shutdown_connections 2023-10-30T12:23:10.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 -- 172.21.15.50:0/439623703 wait complete. 2023-10-30T12:23:10.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 10 monclient: build_initial_monmap 2023-10-30T12:23:10.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 10 monclient: monmap: 2023-10-30T12:23:10.533 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:10.533 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:10.534 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:10.534 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:10.534 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:10.534 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.535 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:10.535 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:10.535 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:10.536 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 librados: starting msgr at 2023-10-30T12:23:10.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 librados: starting objecter 2023-10-30T12:23:10.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 Processor -- start 2023-10-30T12:23:10.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 -- start start 2023-10-30T12:23:10.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 librados: setting wanted keys 2023-10-30T12:23:10.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 1 librados: calling monclient init 2023-10-30T12:23:10.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.497+0000 7f1b72d55640 10 monclient: init 2023-10-30T12:23:10.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:10.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:23:10.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b6c099b80 0x7f1b6c1ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): picked mon.noname-b con 0x7f1b6c099b80 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:10.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c09f5b0 0x7f1b6c095430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): picked mon.noname-a con 0x7f1b6c09f5b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:10.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b6c138b30 0x7f1b6c095970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): picked mon.noname-c con 0x7f1b6c138b30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c091f70 con 0x7f1b6c09f5b0 2023-10-30T12:23:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c095eb0 con 0x7f1b6c099b80 2023-10-30T12:23:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c072010 con 0x7f1b6c138b30 2023-10-30T12:23:10.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:10.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b72d55640 10 monclient(hunting): authenticate will time out at 2235.681608s 2023-10-30T12:23:10.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c09f5b0 0x7f1b6c095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b6c138b30 0x7f1b6c095970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c09f5b0 0x7f1b6c095430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59724/0 (socket says 172.21.15.50:59724) 2023-10-30T12:23:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 1 -- 172.21.15.50:0/4186392026 learned_addr learned my addr 172.21.15.50:0/4186392026 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b6c138b30 0x7f1b6c095970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60604/0 (socket says 172.21.15.50:60604) 2023-10-30T12:23:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 10 monclient(hunting): get_auth_request con 0x7f1b6c09f5b0 auth_method 0 2023-10-30T12:23:10.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 10 monclient(hunting): get_auth_request con 0x7f1b6c138b30 auth_method 0 2023-10-30T12:23:10.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b70af1640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b6c099b80 0x7f1b6c1ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b712f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b70af1640 10 monclient(hunting): get_auth_request con 0x7f1b6c099b80 auth_method 0 2023-10-30T12:23:10.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b70af1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:10.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b70af1640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:10.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b70af1640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.498+0000 7f1b70af1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b70af1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b70af1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 10 monclient(hunting): handle_auth_done global_id 284117 payload 824 2023-10-30T12:23:10.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 1 -- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b6c099b80 msgr2=0x7f1b6c1ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b6c099b80 0x7f1b6c1ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 1 -- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c09f5b0 msgr2=0x7f1b6c095430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c09f5b0 0x7f1b6c095430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 10 monclient: _finish_hunting 0 2023-10-30T12:23:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 1 monclient: found mon.noname-c 2023-10-30T12:23:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 1 -- 172.21.15.50:0/4186392026 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009a70 con 0x7f1b6c138b30 2023-10-30T12:23:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b63fff640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c09f5b0 0x7f1b6c095430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b712f2640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b6c138b30 0x7f1b6c095970 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0a1070 tx=0x7f1b64004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 client.?.objecter ms_handle_connect 0x7f1b6c138b30 2023-10-30T12:23:10.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:10.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 1 -- 172.21.15.50:0/4186392026 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64003b90 con 0x7f1b6c138b30 2023-10-30T12:23:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: dump: 2023-10-30T12:23:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.557 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.557 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: _finish_auth 0 2023-10-30T12:23:10.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: _check_auth_tickets 2023-10-30T12:23:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 5 monclient: authenticate success, global_id 284117 2023-10-30T12:23:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 1 -- 172.21.15.50:0/4186392026 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64003cf0 con 0x7f1b6c138b30 2023-10-30T12:23:10.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:10.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 1 -- 172.21.15.50:0/4186392026 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64004270 con 0x7f1b6c138b30 2023-10-30T12:23:10.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 10 monclient: _renew_subs 2023-10-30T12:23:10.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:10.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c13ac80 con 0x7f1b6c138b30 2023-10-30T12:23:10.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:10.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b62ffd640 4 set_mon_vals no callback set 2023-10-30T12:23:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b617fa640 10 monclient: dump: 2023-10-30T12:23:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:10.562 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.562 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:10.562 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:10.562 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:10.564 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 10 client.284117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 10 monclient: _renew_subs 2023-10-30T12:23:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:10.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c091920 con 0x7f1b6c138b30 2023-10-30T12:23:10.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.499+0000 7f1b72d55640 1 librados: init done 2023-10-30T12:23:10.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b617fa640 1 -- 172.21.15.50:0/4186392026 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f1b6403b050 con 0x7f1b6c138b30 2023-10-30T12:23:10.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b72d55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:10.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b72d55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:10.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b72d55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:10.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b617fa640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b580392f0 0x7f1b5803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:10.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c04e160 con 0x7f1b6c138b30 2023-10-30T12:23:10.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b617fa640 1 -- 172.21.15.50:0/4186392026 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f1b6406cde0 con 0x7f1b6c138b30 2023-10-30T12:23:10.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b617fa640 10 client.284117.objecter ms_dispatch 0x7f1b6c000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:10.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b617fa640 3 client.284117.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:10.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b617fa640 3 client.284117.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:10.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b617fa640 20 client.284117.objecter dump_active .. 0 homeless 2023-10-30T12:23:10.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b70af1640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b580392f0 0x7f1b5803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:10.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.500+0000 7f1b70af1640 10 monclient: get_auth_request con 0x7f1b580392f0 auth_method 0 2023-10-30T12:23:10.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.501+0000 7f1b617fa640 10 client.284117.objecter ms_handle_connect 0x7f1b580392f0 2023-10-30T12:23:10.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.501+0000 7f1b70af1640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b580392f0 0x7f1b5803b7b0 secure :-1 s=READY pgs=4352 cs=0 l=1 rev1=1 crypto rx=0x7f1b68007fd0 tx=0x7f1b68009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:10.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.505+0000 7f1b617fa640 1 -- 172.21.15.50:0/4186392026 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1b64039030 con 0x7f1b6c138b30 2023-10-30T12:23:10.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.505+0000 7f1b617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:10.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.505+0000 7f1b617fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:10.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.746+0000 7f1b72d55640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2023-10-30T12:23:10.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.746+0000 7f1b72d55640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2023-10-30T12:23:10.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.746+0000 7f1b72d55640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:10.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.746+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1b6c0a2ef0 con 0x7f1b6c138b30 2023-10-30T12:23:10.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f1b617fa640 1 -- 172.21.15.50:0/4186392026 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f1b64039030 con 0x7f1b6c138b30 2023-10-30T12:23:10.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f1b617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2023-10-30T12:23:10.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f1b617fa640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:23:10.752 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=9,logm=51} crc {auth=3890086685,config=3140514009,health=3197116582,kv=2905431493,logm=3252408975}) 2023-10-30T12:23:10.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=2604412262,config=3140514009,health=2789545438,kv=2905431493,logm=2772733812}) 2023-10-30T12:23:10.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.753 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=9,logm=51} crc {auth=3890086685,config=3140514009,health=3197116582,kv=2905431493,logm=3252408975}) 2023-10-30T12:23:10.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.748+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=777919783,config=3140514009,health=2789545439,kv=4037166911,logm=1891511190}) 2023-10-30T12:23:10.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.749+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.754 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.749+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056931146}) 2023-10-30T12:23:10.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.749+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859118965}) 2023-10-30T12:23:10.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.749+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.749+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056931146}) 2023-10-30T12:23:10.755 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.749+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323499173}) 2023-10-30T12:23:10.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.750+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.750+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603912850}) 2023-10-30T12:23:10.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.750+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649313718}) 2023-10-30T12:23:10.756 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603912850}) 2023-10-30T12:23:10.757 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159819502}) 2023-10-30T12:23:10.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 10 librados: watch_flush enter 2023-10-30T12:23:10.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 10 librados: watch_flush exit 2023-10-30T12:23:10.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 20 client.284117.objecter shutdown clearing up homeless session... 2023-10-30T12:23:10.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 10 client.284117.objecter successfully canceled tick 2023-10-30T12:23:10.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b580392f0 msgr2=0x7f1b5803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b580392f0 0x7f1b5803b7b0 secure :-1 s=READY pgs=4352 cs=0 l=1 rev1=1 crypto rx=0x7f1b68007fd0 tx=0x7f1b68009040 comp rx=0 tx=0).stop 2023-10-30T12:23:10.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 10 monclient: shutdown 2023-10-30T12:23:10.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:10.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b6c138b30 msgr2=0x7f1b6c095970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:10.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b6c138b30 0x7f1b6c095970 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0a1070 tx=0x7f1b64004a40 comp rx=0 tx=0).stop 2023-10-30T12:23:10.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 shutdown_connections 2023-10-30T12:23:10.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f1b580392f0 0x7f1b5803b7b0 unknown :-1 s=CLOSED pgs=4352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1b6c138b30 0x7f1b6c095970 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b6c09f5b0 0x7f1b6c095430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 --2- 172.21.15.50:0/4186392026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b6c099b80 0x7f1b6c1ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:10.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 >> 172.21.15.50:0/4186392026 conn(0x7f1b6c0a05c0 msgr2=0x7f1b6c138f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 shutdown_connections 2023-10-30T12:23:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 -- 172.21.15.50:0/4186392026 wait complete. 2023-10-30T12:23:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:10.751+0000 7f1b72d55640 1 librados: shutdown 2023-10-30T12:23:10.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.752+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.752+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614727139}) 2023-10-30T12:23:10.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.752+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3192560965}) 2023-10-30T12:23:10.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.752+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.770 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.752+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614727139}) 2023-10-30T12:23:10.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.752+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2059975422}) 2023-10-30T12:23:10.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.753+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.753+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931347664}) 2023-10-30T12:23:10.771 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.753+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071378595}) 2023-10-30T12:23:10.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.753+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.753+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931347664}) 2023-10-30T12:23:10.772 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.753+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3631095478}) 2023-10-30T12:23:10.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.754+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.754+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731527614}) 2023-10-30T12:23:10.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.754+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374246140}) 2023-10-30T12:23:10.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.754+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.754+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731527614}) 2023-10-30T12:23:10.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.754+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4144152464}) 2023-10-30T12:23:10.774 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.755+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.755+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573673432}) 2023-10-30T12:23:10.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.755+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2014548977}) 2023-10-30T12:23:10.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.755+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.775 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.755+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573673432}) 2023-10-30T12:23:10.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.755+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876669389}) 2023-10-30T12:23:10.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.756+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.776 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.756+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593868596}) 2023-10-30T12:23:10.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.756+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1496855416}) 2023-10-30T12:23:10.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.756+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.756+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593868596}) 2023-10-30T12:23:10.777 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.756+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=430025412}) 2023-10-30T12:23:10.778 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.757+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.778 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.757+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028047273}) 2023-10-30T12:23:10.778 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.757+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314920132}) 2023-10-30T12:23:10.778 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.757+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.757+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028047273}) 2023-10-30T12:23:10.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.757+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095634067}) 2023-10-30T12:23:10.779 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991466887}) 2023-10-30T12:23:10.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=786693760}) 2023-10-30T12:23:10.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.780 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991466887}) 2023-10-30T12:23:10.781 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171807355}) 2023-10-30T12:23:10.781 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.781 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416391023}) 2023-10-30T12:23:10.781 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604823897}) 2023-10-30T12:23:10.782 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.782 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416391023}) 2023-10-30T12:23:10.782 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.758+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824722754}) 2023-10-30T12:23:10.782 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.759+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.783 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.759+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289954364}) 2023-10-30T12:23:10.783 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.759+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3012396254}) 2023-10-30T12:23:10.783 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.759+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.784 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.759+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289954364}) 2023-10-30T12:23:10.784 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.759+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1809125958}) 2023-10-30T12:23:10.784 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.760+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.784 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.760+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1530872512}) 2023-10-30T12:23:10.785 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.760+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467210590}) 2023-10-30T12:23:10.785 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.760+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.785 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.760+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1530872512}) 2023-10-30T12:23:10.785 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.760+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478785786}) 2023-10-30T12:23:10.786 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.761+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.786 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.761+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613350907}) 2023-10-30T12:23:10.786 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.761+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816025425}) 2023-10-30T12:23:10.786 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.761+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.787 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.761+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613350907}) 2023-10-30T12:23:10.787 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.761+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771765316}) 2023-10-30T12:23:10.787 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.762+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.788 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.762+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327294836}) 2023-10-30T12:23:10.788 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.762+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818211483}) 2023-10-30T12:23:10.788 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.762+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.788 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.762+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327294836}) 2023-10-30T12:23:10.789 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.762+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010790071}) 2023-10-30T12:23:10.789 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.763+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.789 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.763+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118727964}) 2023-10-30T12:23:10.789 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.763+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449418377}) 2023-10-30T12:23:10.790 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.763+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.790 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.763+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118727964}) 2023-10-30T12:23:10.790 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.763+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3497429293}) 2023-10-30T12:23:10.790 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102205215}) 2023-10-30T12:23:10.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239288851}) 2023-10-30T12:23:10.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.791 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102205215}) 2023-10-30T12:23:10.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=700457512}) 2023-10-30T12:23:10.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981323260}) 2023-10-30T12:23:10.792 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=890002143}) 2023-10-30T12:23:10.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981323260}) 2023-10-30T12:23:10.793 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.764+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662293042}) 2023-10-30T12:23:10.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-10-30T12:23:10.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.765+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.765+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881555329}) 2023-10-30T12:23:10.840 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.765+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125849509}) 2023-10-30T12:23:10.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.765+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.765+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881555329}) 2023-10-30T12:23:10.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.765+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=823482190}) 2023-10-30T12:23:10.841 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273488099}) 2023-10-30T12:23:10.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434416380}) 2023-10-30T12:23:10.842 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273488099}) 2023-10-30T12:23:10.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3738153504}) 2023-10-30T12:23:10.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.843 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870313513}) 2023-10-30T12:23:10.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1236501291}) 2023-10-30T12:23:10.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870313513}) 2023-10-30T12:23:10.844 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.766+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098898127}) 2023-10-30T12:23:10.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.767+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.767+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357418665}) 2023-10-30T12:23:10.845 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.767+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529677962}) 2023-10-30T12:23:10.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.767+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.767+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357418665}) 2023-10-30T12:23:10.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.767+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474469941}) 2023-10-30T12:23:10.846 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175198555}) 2023-10-30T12:23:10.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2527835282}) 2023-10-30T12:23:10.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.847 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175198555}) 2023-10-30T12:23:10.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1336974249}) 2023-10-30T12:23:10.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364605406}) 2023-10-30T12:23:10.848 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319963511}) 2023-10-30T12:23:10.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364605406}) 2023-10-30T12:23:10.849 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.768+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256114754}) 2023-10-30T12:23:10.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438149926}) 2023-10-30T12:23:10.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967343154}) 2023-10-30T12:23:10.850 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438149926}) 2023-10-30T12:23:10.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3565334946}) 2023-10-30T12:23:10.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.851 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836280281}) 2023-10-30T12:23:10.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151859170}) 2023-10-30T12:23:10.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.769+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.852 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3836280281}) 2023-10-30T12:23:10.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320731941}) 2023-10-30T12:23:10.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166139011}) 2023-10-30T12:23:10.853 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3102310485}) 2023-10-30T12:23:10.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166139011}) 2023-10-30T12:23:10.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.770+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499973899}) 2023-10-30T12:23:10.854 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64770161}) 2023-10-30T12:23:10.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328195802}) 2023-10-30T12:23:10.855 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64770161}) 2023-10-30T12:23:10.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360425641}) 2023-10-30T12:23:10.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.856 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224851958}) 2023-10-30T12:23:10.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367600075}) 2023-10-30T12:23:10.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.857 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224851958}) 2023-10-30T12:23:10.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.771+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2763157415}) 2023-10-30T12:23:10.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.772+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.772+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595751750}) 2023-10-30T12:23:10.858 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.772+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801118984}) 2023-10-30T12:23:10.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.772+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.772+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595751750}) 2023-10-30T12:23:10.859 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.772+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596004561}) 2023-10-30T12:23:10.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298276430}) 2023-10-30T12:23:10.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3480008239}) 2023-10-30T12:23:10.860 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298276430}) 2023-10-30T12:23:10.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576476602}) 2023-10-30T12:23:10.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.861 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483489432}) 2023-10-30T12:23:10.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3520167658}) 2023-10-30T12:23:10.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.862 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483489432}) 2023-10-30T12:23:10.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.773+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=10092806}) 2023-10-30T12:23:10.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.774+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.774+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994087213}) 2023-10-30T12:23:10.863 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.774+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866724473}) 2023-10-30T12:23:10.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.774+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.774+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994087213}) 2023-10-30T12:23:10.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.774+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3236131287}) 2023-10-30T12:23:10.864 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782239108}) 2023-10-30T12:23:10.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918223424}) 2023-10-30T12:23:10.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.865 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782239108}) 2023-10-30T12:23:10.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941234393}) 2023-10-30T12:23:10.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.866 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860222757}) 2023-10-30T12:23:10.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=574487980}) 2023-10-30T12:23:10.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860222757}) 2023-10-30T12:23:10.867 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.775+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195675131}) 2023-10-30T12:23:10.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159490795}) 2023-10-30T12:23:10.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409239224}) 2023-10-30T12:23:10.868 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159490795}) 2023-10-30T12:23:10.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913845707}) 2023-10-30T12:23:10.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.869 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.776+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293949268}) 2023-10-30T12:23:10.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8628818}) 2023-10-30T12:23:10.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.870 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293949268}) 2023-10-30T12:23:10.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862048691}) 2023-10-30T12:23:10.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170937021}) 2023-10-30T12:23:10.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807297251}) 2023-10-30T12:23:10.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170937021}) 2023-10-30T12:23:10.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.777+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205370692}) 2023-10-30T12:23:10.872 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988885341}) 2023-10-30T12:23:10.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=567823378}) 2023-10-30T12:23:10.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.873 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988885341}) 2023-10-30T12:23:10.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=803684399}) 2023-10-30T12:23:10.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.874 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076749157}) 2023-10-30T12:23:10.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=560600234}) 2023-10-30T12:23:10.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076749157}) 2023-10-30T12:23:10.875 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.778+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1674932018}) 2023-10-30T12:23:10.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.779+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.779+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768197602}) 2023-10-30T12:23:10.876 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.779+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185299069}) 2023-10-30T12:23:10.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.779+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.779+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768197602}) 2023-10-30T12:23:10.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.779+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327746614}) 2023-10-30T12:23:10.877 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584465587}) 2023-10-30T12:23:10.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260371486}) 2023-10-30T12:23:10.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.878 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584465587}) 2023-10-30T12:23:10.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055933182}) 2023-10-30T12:23:10.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.879 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525431212}) 2023-10-30T12:23:10.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257022286}) 2023-10-30T12:23:10.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525431212}) 2023-10-30T12:23:10.880 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.780+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647931849}) 2023-10-30T12:23:10.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551088560}) 2023-10-30T12:23:10.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915491732}) 2023-10-30T12:23:10.881 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551088560}) 2023-10-30T12:23:10.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498022675}) 2023-10-30T12:23:10.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.882 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1692786987}) 2023-10-30T12:23:10.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2270033864}) 2023-10-30T12:23:10.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.883 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1692786987}) 2023-10-30T12:23:10.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.781+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363346475}) 2023-10-30T12:23:10.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.782+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.782+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195550696}) 2023-10-30T12:23:10.884 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.782+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556201901}) 2023-10-30T12:23:10.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.782+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.782+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195550696}) 2023-10-30T12:23:10.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.782+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2415146653}) 2023-10-30T12:23:10.885 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887405181}) 2023-10-30T12:23:10.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663171721}) 2023-10-30T12:23:10.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.886 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887405181}) 2023-10-30T12:23:10.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=785553135}) 2023-10-30T12:23:10.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.887 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647075041}) 2023-10-30T12:23:10.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4076654835}) 2023-10-30T12:23:10.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647075041}) 2023-10-30T12:23:10.888 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.783+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822685385}) 2023-10-30T12:23:10.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.784+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.784+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464071635}) 2023-10-30T12:23:10.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.784+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3375677387}) 2023-10-30T12:23:10.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.784+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.784+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464071635}) 2023-10-30T12:23:10.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.784+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476515921}) 2023-10-30T12:23:10.890 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026667096}) 2023-10-30T12:23:10.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757027653}) 2023-10-30T12:23:10.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.891 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026667096}) 2023-10-30T12:23:10.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011432873}) 2023-10-30T12:23:10.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730730158}) 2023-10-30T12:23:10.892 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2664850287}) 2023-10-30T12:23:10.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=730730158}) 2023-10-30T12:23:10.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.785+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256994977}) 2023-10-30T12:23:10.893 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084503420}) 2023-10-30T12:23:10.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3959714873}) 2023-10-30T12:23:10.894 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084503420}) 2023-10-30T12:23:10.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429066045}) 2023-10-30T12:23:10.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.895 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664529332}) 2023-10-30T12:23:10.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123046853}) 2023-10-30T12:23:10.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664529332}) 2023-10-30T12:23:10.896 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.786+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214890708}) 2023-10-30T12:23:10.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.787+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.787+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075483073}) 2023-10-30T12:23:10.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.787+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1442169675}) 2023-10-30T12:23:10.897 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.787+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.787+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075483073}) 2023-10-30T12:23:10.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.787+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483214419}) 2023-10-30T12:23:10.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.898 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331849939}) 2023-10-30T12:23:10.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=213950360}) 2023-10-30T12:23:10.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.899 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331849939}) 2023-10-30T12:23:10.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830982077}) 2023-10-30T12:23:10.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592392721}) 2023-10-30T12:23:10.900 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2909854532}) 2023-10-30T12:23:10.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592392721}) 2023-10-30T12:23:10.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.788+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706814022}) 2023-10-30T12:23:10.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179764942}) 2023-10-30T12:23:10.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=116691095}) 2023-10-30T12:23:10.902 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179764942}) 2023-10-30T12:23:10.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127588065}) 2023-10-30T12:23:10.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.903 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146741561}) 2023-10-30T12:23:10.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3487548786}) 2023-10-30T12:23:10.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146741561}) 2023-10-30T12:23:10.904 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.789+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223557645}) 2023-10-30T12:23:10.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.790+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.790+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169477627}) 2023-10-30T12:23:10.905 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.790+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3096930514}) 2023-10-30T12:23:10.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.790+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.790+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169477627}) 2023-10-30T12:23:10.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.790+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948747089}) 2023-10-30T12:23:10.906 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555684221}) 2023-10-30T12:23:10.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2783158369}) 2023-10-30T12:23:10.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.907 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555684221}) 2023-10-30T12:23:10.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584648394}) 2023-10-30T12:23:10.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.908 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=292951916}) 2023-10-30T12:23:10.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603855374}) 2023-10-30T12:23:10.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=292951916}) 2023-10-30T12:23:10.909 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.791+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2164279035}) 2023-10-30T12:23:10.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.792+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.792+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377133295}) 2023-10-30T12:23:10.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.792+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2026624699}) 2023-10-30T12:23:10.910 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.792+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.792+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377133295}) 2023-10-30T12:23:10.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.792+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267783885}) 2023-10-30T12:23:10.911 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223658538}) 2023-10-30T12:23:10.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3959975395}) 2023-10-30T12:23:10.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.912 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223658538}) 2023-10-30T12:23:10.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=417678143}) 2023-10-30T12:23:10.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562636244}) 2023-10-30T12:23:10.913 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832504411}) 2023-10-30T12:23:10.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562636244}) 2023-10-30T12:23:10.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.793+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417224293}) 2023-10-30T12:23:10.914 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258175609}) 2023-10-30T12:23:10.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1753615899}) 2023-10-30T12:23:10.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.915 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258175609}) 2023-10-30T12:23:10.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257806176}) 2023-10-30T12:23:10.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.916 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2325412462}) 2023-10-30T12:23:10.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=290054078}) 2023-10-30T12:23:10.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.794+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2325412462}) 2023-10-30T12:23:10.917 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2181882909}) 2023-10-30T12:23:10.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148324332}) 2023-10-30T12:23:10.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855564311}) 2023-10-30T12:23:10.918 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148324332}) 2023-10-30T12:23:10.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.795+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1178512414}) 2023-10-30T12:23:10.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.919 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553709110}) 2023-10-30T12:23:10.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1618202726}) 2023-10-30T12:23:10.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.920 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553709110}) 2023-10-30T12:23:10.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182152560}) 2023-10-30T12:23:10.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094022277}) 2023-10-30T12:23:10.921 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043134588}) 2023-10-30T12:23:10.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094022277}) 2023-10-30T12:23:10.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.796+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=303237477}) 2023-10-30T12:23:10.922 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.797+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.797+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470913380}) 2023-10-30T12:23:10.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.797+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544732459}) 2023-10-30T12:23:10.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.797+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.797+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470913380}) 2023-10-30T12:23:10.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.797+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143697099}) 2023-10-30T12:23:10.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219705648}) 2023-10-30T12:23:10.924 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3672223007}) 2023-10-30T12:23:10.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219705648}) 2023-10-30T12:23:10.925 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3901922074}) 2023-10-30T12:23:10.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056222108}) 2023-10-30T12:23:10.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703016173}) 2023-10-30T12:23:10.926 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056222108}) 2023-10-30T12:23:10.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.798+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681000946}) 2023-10-30T12:23:10.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.799+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.927 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.799+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887650360}) 2023-10-30T12:23:10.929 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.799+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2396694479}) 2023-10-30T12:23:10.930 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.799+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.799+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887650360}) 2023-10-30T12:23:10.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.799+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1303292507}) 2023-10-30T12:23:10.931 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950050392}) 2023-10-30T12:23:10.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964218725}) 2023-10-30T12:23:10.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.932 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950050392}) 2023-10-30T12:23:10.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3833747174}) 2023-10-30T12:23:10.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965583943}) 2023-10-30T12:23:10.933 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1514825746}) 2023-10-30T12:23:10.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965583943}) 2023-10-30T12:23:10.934 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.800+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3904041680}) 2023-10-30T12:23:10.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.801+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.801+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2383459592}) 2023-10-30T12:23:10.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.801+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2474926526}) 2023-10-30T12:23:10.935 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.801+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.801+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2383459592}) 2023-10-30T12:23:10.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.801+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173428002}) 2023-10-30T12:23:10.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.936 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686515262}) 2023-10-30T12:23:10.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3889702969}) 2023-10-30T12:23:10.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686515262}) 2023-10-30T12:23:10.937 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746030032}) 2023-10-30T12:23:10.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712330483}) 2023-10-30T12:23:10.938 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061597242}) 2023-10-30T12:23:10.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712330483}) 2023-10-30T12:23:10.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.802+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684435312}) 2023-10-30T12:23:10.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683402924}) 2023-10-30T12:23:10.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626530397}) 2023-10-30T12:23:10.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.940 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683402924}) 2023-10-30T12:23:10.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222357816}) 2023-10-30T12:23:10.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032531812}) 2023-10-30T12:23:10.941 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741469276}) 2023-10-30T12:23:10.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.803+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032531812}) 2023-10-30T12:23:10.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166058590}) 2023-10-30T12:23:10.942 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587381308}) 2023-10-30T12:23:10.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676519235}) 2023-10-30T12:23:10.943 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587381308}) 2023-10-30T12:23:10.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.804+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247320970}) 2023-10-30T12:23:10.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.944 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778550266}) 2023-10-30T12:23:10.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=250341039}) 2023-10-30T12:23:10.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778550266}) 2023-10-30T12:23:10.945 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278138055}) 2023-10-30T12:23:10.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228836715}) 2023-10-30T12:23:10.946 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505330392}) 2023-10-30T12:23:10.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228836715}) 2023-10-30T12:23:10.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.805+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2806189570}) 2023-10-30T12:23:10.947 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.806+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.806+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226660914}) 2023-10-30T12:23:10.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.806+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915353408}) 2023-10-30T12:23:10.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.806+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.948 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.806+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226660914}) 2023-10-30T12:23:10.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.806+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939256650}) 2023-10-30T12:23:10.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609662135}) 2023-10-30T12:23:10.949 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584567795}) 2023-10-30T12:23:10.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609662135}) 2023-10-30T12:23:10.950 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803248045}) 2023-10-30T12:23:10.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2838708437}) 2023-10-30T12:23:10.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806808136}) 2023-10-30T12:23:10.951 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2838708437}) 2023-10-30T12:23:10.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.807+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2625343168}) 2023-10-30T12:23:10.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.808+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.952 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.808+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456637160}) 2023-10-30T12:23:10.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.808+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1304023715}) 2023-10-30T12:23:10.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.808+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.953 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.808+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456637160}) 2023-10-30T12:23:10.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.808+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2694114370}) 2023-10-30T12:23:10.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797598057}) 2023-10-30T12:23:10.954 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3575250631}) 2023-10-30T12:23:10.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797598057}) 2023-10-30T12:23:10.955 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180668000}) 2023-10-30T12:23:10.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151514883}) 2023-10-30T12:23:10.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4062237058}) 2023-10-30T12:23:10.956 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151514883}) 2023-10-30T12:23:10.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.809+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4030483129}) 2023-10-30T12:23:10.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.810+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.957 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.810+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010671070}) 2023-10-30T12:23:10.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.810+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=730583964}) 2023-10-30T12:23:10.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.810+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.958 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.810+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010671070}) 2023-10-30T12:23:10.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.810+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2150305415}) 2023-10-30T12:23:10.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141398347}) 2023-10-30T12:23:10.959 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575253183}) 2023-10-30T12:23:10.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141398347}) 2023-10-30T12:23:10.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2624315009}) 2023-10-30T12:23:10.960 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356567976}) 2023-10-30T12:23:10.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436000088}) 2023-10-30T12:23:10.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.961 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356567976}) 2023-10-30T12:23:10.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.811+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068502466}) 2023-10-30T12:23:10.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.962 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259072033}) 2023-10-30T12:23:10.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034605884}) 2023-10-30T12:23:10.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259072033}) 2023-10-30T12:23:10.963 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2045158354}) 2023-10-30T12:23:10.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2803551655}) 2023-10-30T12:23:10.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579766563}) 2023-10-30T12:23:10.964 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.812+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2803551655}) 2023-10-30T12:23:10.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.813+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=840847794}) 2023-10-30T12:23:10.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.813+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.965 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.813+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2471847647}) 2023-10-30T12:23:10.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.813+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931677410}) 2023-10-30T12:23:10.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.813+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.966 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.813+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2471847647}) 2023-10-30T12:23:10.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.813+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348043193}) 2023-10-30T12:23:10.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289591414}) 2023-10-30T12:23:10.967 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=632826119}) 2023-10-30T12:23:10.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289591414}) 2023-10-30T12:23:10.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4093540526}) 2023-10-30T12:23:10.968 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749515473}) 2023-10-30T12:23:10.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1653443732}) 2023-10-30T12:23:10.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.969 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749515473}) 2023-10-30T12:23:10.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.814+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1368446012}) 2023-10-30T12:23:10.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.815+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.970 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.815+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952133304}) 2023-10-30T12:23:10.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.815+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3923772183}) 2023-10-30T12:23:10.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.815+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.815+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952133304}) 2023-10-30T12:23:10.971 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.815+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1430074877}) 2023-10-30T12:23:10.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235120983}) 2023-10-30T12:23:10.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3960541439}) 2023-10-30T12:23:10.972 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235120983}) 2023-10-30T12:23:10.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255114447}) 2023-10-30T12:23:10.973 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846549055}) 2023-10-30T12:23:10.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3047842613}) 2023-10-30T12:23:10.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.974 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846549055}) 2023-10-30T12:23:10.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.816+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060851430}) 2023-10-30T12:23:10.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.817+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.817+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253312139}) 2023-10-30T12:23:10.975 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.817+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2194725753}) 2023-10-30T12:23:10.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.817+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.817+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253312139}) 2023-10-30T12:23:10.976 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.817+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122192914}) 2023-10-30T12:23:10.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2502462228}) 2023-10-30T12:23:10.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3923425657}) 2023-10-30T12:23:10.977 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2502462228}) 2023-10-30T12:23:10.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360106020}) 2023-10-30T12:23:10.978 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338334661}) 2023-10-30T12:23:10.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176068761}) 2023-10-30T12:23:10.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.979 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338334661}) 2023-10-30T12:23:10.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.818+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2576862940}) 2023-10-30T12:23:10.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.819+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.819+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=222211792}) 2023-10-30T12:23:10.980 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.819+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3810514455}) 2023-10-30T12:23:10.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.819+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.819+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=222211792}) 2023-10-30T12:23:10.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.819+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291706327}) 2023-10-30T12:23:10.981 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554268751}) 2023-10-30T12:23:10.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1860243765}) 2023-10-30T12:23:10.982 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554268751}) 2023-10-30T12:23:10.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542345773}) 2023-10-30T12:23:10.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.983 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225773817}) 2023-10-30T12:23:10.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3171055568}) 2023-10-30T12:23:10.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225773817}) 2023-10-30T12:23:10.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.820+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3764528042}) 2023-10-30T12:23:10.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303780645}) 2023-10-30T12:23:10.985 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2822336435}) 2023-10-30T12:23:10.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303780645}) 2023-10-30T12:23:10.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2404553948}) 2023-10-30T12:23:10.986 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658968869}) 2023-10-30T12:23:10.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290509854}) 2023-10-30T12:23:10.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.987 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658968869}) 2023-10-30T12:23:10.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.821+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052931436}) 2023-10-30T12:23:10.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.822+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.988 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.822+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272975318}) 2023-10-30T12:23:10.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.822+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343908854}) 2023-10-30T12:23:10.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.822+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.822+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272975318}) 2023-10-30T12:23:10.989 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.822+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2673137879}) 2023-10-30T12:23:10.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887913663}) 2023-10-30T12:23:10.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3623848605}) 2023-10-30T12:23:10.990 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887913663}) 2023-10-30T12:23:10.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247902389}) 2023-10-30T12:23:10.991 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066969596}) 2023-10-30T12:23:10.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2909033637}) 2023-10-30T12:23:10.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066969596}) 2023-10-30T12:23:10.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.823+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108869429}) 2023-10-30T12:23:10.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.824+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.824+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592058236}) 2023-10-30T12:23:10.993 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.824+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1609032670}) 2023-10-30T12:23:10.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.824+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.824+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592058236}) 2023-10-30T12:23:10.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.824+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942434090}) 2023-10-30T12:23:10.994 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.825+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.825+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778296525}) 2023-10-30T12:23:10.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.825+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3977849269}) 2023-10-30T12:23:10.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.825+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.995 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.825+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778296525}) 2023-10-30T12:23:10.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.825+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3852968844}) 2023-10-30T12:23:10.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333673849}) 2023-10-30T12:23:10.996 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1152573877}) 2023-10-30T12:23:10.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333673849}) 2023-10-30T12:23:10.997 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026113606}) 2023-10-30T12:23:10.998 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.998 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345712552}) 2023-10-30T12:23:10.998 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=535902981}) 2023-10-30T12:23:10.998 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.999 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345712552}) 2023-10-30T12:23:10.999 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.826+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639039101}) 2023-10-30T12:23:10.999 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.827+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:10.999 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.827+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606682348}) 2023-10-30T12:23:11.000 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.827+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880889690}) 2023-10-30T12:23:11.000 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.827+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.000 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.827+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606682348}) 2023-10-30T12:23:11.000 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.827+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071202250}) 2023-10-30T12:23:11.001 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.827+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.001 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2576785031}) 2023-10-30T12:23:11.001 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193392330}) 2023-10-30T12:23:11.001 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.002 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2576785031}) 2023-10-30T12:23:11.002 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241329487}) 2023-10-30T12:23:11.002 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.003 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=729476562}) 2023-10-30T12:23:11.003 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718262878}) 2023-10-30T12:23:11.003 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.003 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=729476562}) 2023-10-30T12:23:11.004 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.828+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3073785383}) 2023-10-30T12:23:11.004 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.004 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168428541}) 2023-10-30T12:23:11.004 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153466576}) 2023-10-30T12:23:11.005 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.005 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168428541}) 2023-10-30T12:23:11.005 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3000664093}) 2023-10-30T12:23:11.005 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.006 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175289449}) 2023-10-30T12:23:11.006 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2853602281}) 2023-10-30T12:23:11.006 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.007 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175289449}) 2023-10-30T12:23:11.007 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.829+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426527511}) 2023-10-30T12:23:11.007 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.830+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.007 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.830+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3082916205}) 2023-10-30T12:23:11.008 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.830+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482056853}) 2023-10-30T12:23:11.008 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.830+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.008 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.830+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3082916205}) 2023-10-30T12:23:11.008 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.830+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536672269}) 2023-10-30T12:23:11.009 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.009 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796243788}) 2023-10-30T12:23:11.009 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586728935}) 2023-10-30T12:23:11.009 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.010 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796243788}) 2023-10-30T12:23:11.010 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1677277790}) 2023-10-30T12:23:11.010 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292933386}) 2023-10-30T12:23:11.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986811546}) 2023-10-30T12:23:11.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.011 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292933386}) 2023-10-30T12:23:11.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.831+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3624203723}) 2023-10-30T12:23:11.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.832+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.832+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2375652279}) 2023-10-30T12:23:11.012 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.832+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514863965}) 2023-10-30T12:23:11.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.832+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.832+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2375652279}) 2023-10-30T12:23:11.013 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.832+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932528523}) 2023-10-30T12:23:11.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2659368905}) 2023-10-30T12:23:11.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032887142}) 2023-10-30T12:23:11.014 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2659368905}) 2023-10-30T12:23:11.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300167377}) 2023-10-30T12:23:11.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.015 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458080945}) 2023-10-30T12:23:11.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454033822}) 2023-10-30T12:23:11.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458080945}) 2023-10-30T12:23:11.016 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.833+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2736351188}) 2023-10-30T12:23:11.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912875124}) 2023-10-30T12:23:11.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1399106264}) 2023-10-30T12:23:11.017 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912875124}) 2023-10-30T12:23:11.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575659337}) 2023-10-30T12:23:11.018 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206194626}) 2023-10-30T12:23:11.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429525675}) 2023-10-30T12:23:11.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.019 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206194626}) 2023-10-30T12:23:11.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.834+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320597969}) 2023-10-30T12:23:11.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.835+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.835+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199986062}) 2023-10-30T12:23:11.020 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.835+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=320994510}) 2023-10-30T12:23:11.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.835+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.835+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199986062}) 2023-10-30T12:23:11.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.835+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4012827562}) 2023-10-30T12:23:11.021 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.836+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.836+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=2384060344,mdsmap=3984425268,mgr=3474422946,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2297444390}) 2023-10-30T12:23:11.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.836+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1175600076,mdsmap=3984425268,mgr=2906314846,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=3438708115}) 2023-10-30T12:23:11.022 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.836+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.836+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=2384060344,mdsmap=3984425268,mgr=3474422946,mgr_command_descs=3900280244,mgr_metadata=2083152833,mgrstat=2297444390}) 2023-10-30T12:23:11.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.836+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=543930193,mdsmap=3984425268,mgr=2119566901}) 2023-10-30T12:23:11.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.837+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.023 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.837+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=34} crc {mgrstat=2338404398,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=723956144,osd_snap=2239320510,osdmap=3091649098}) 2023-10-30T12:23:11.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.837+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=35} crc {mgrstat=1463808021,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=723956144,osd_snap=548972925,osdmap=1727732494}) 2023-10-30T12:23:11.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.837+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.837+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=34} crc {mgrstat=2338404398,mon_config_key=3217606506,monmap=3324057927,osd_metadata=2479346626,osd_pg_creating=723956144,osd_snap=2239320510,osdmap=3091649098}) 2023-10-30T12:23:11.024 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.837+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=33} crc {mgrstat=1463808021,mon_config_key=3217606506,monmap=3324057927,osd_metadata=3578860750,osd_pg_creating=723956144,osd_snap=3350230443,osdmap=134842867}) 2023-10-30T12:23:11.025 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.842+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.842+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=1608298746}) 2023-10-30T12:23:11.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.842+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=39} crc {osdmap=1917696675}) 2023-10-30T12:23:11.026 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.842+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-10-30T12:23:11.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.842+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=1608298746}) 2023-10-30T12:23:11.027 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-30T12:23:10.842+0000 7f8e5c211640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=41} crc {osdmap=3694351226}) 2023-10-30T12:23:11.796 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-10-30T12:23:11.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-10-30T12:23:11.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.911+0000 7f79a0b83640 10 monclient: get_monmap_and_config 2023-10-30T12:23:11.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.911+0000 7f79a0b83640 10 monclient: build_initial_monmap 2023-10-30T12:23:11.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.911+0000 7f79a0b83640 10 monclient: monmap: 2023-10-30T12:23:11.914 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:11.914 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:11.914 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:11.912707+0000 2023-10-30T12:23:11.915 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:11.912707+0000 2023-10-30T12:23:11.915 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:11.915 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:11.916 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:11.916 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:11.916 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:11.916 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:11.917 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:11.917 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:11.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 1 Processor -- start 2023-10-30T12:23:11.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 1 -- start start 2023-10-30T12:23:11.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient: init 2023-10-30T12:23:11.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:11.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:23:11.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f799c0707d0 0x7f799c09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:11.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient(hunting): picked mon.noname-a con 0x7f799c0707d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:11.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f799c070bb0 0x7f799c071020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:11.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient(hunting): picked mon.noname-c con 0x7f799c070bb0 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:11.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f799c1345f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:11.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c09ee40 con 0x7f799c1345f0 2023-10-30T12:23:11.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:11.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799c09f4f0 con 0x7f799c0707d0 2023-10-30T12:23:11.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:11.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799c09f7d0 con 0x7f799c070bb0 2023-10-30T12:23:11.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:11.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f79a0b83640 10 monclient(hunting): authenticate will time out at 2237.095831s 2023-10-30T12:23:11.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f799a59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f799c0707d0 0x7f799c09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:11.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f799ad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f799c1345f0 0x7f799c134a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49570/0 (socket says 172.21.15.50:49570) 2023-10-30T12:23:11.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f799a59c640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f799c0707d0 0x7f799c09e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59732/0 (socket says 172.21.15.50:59732) 2023-10-30T12:23:11.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f799ad9d640 1 -- 172.21.15.50:0/2646787995 learned_addr learned my addr 172.21.15.50:0/2646787995 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:11.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.912+0000 7f799a59c640 10 monclient(hunting): get_auth_request con 0x7f799c0707d0 auth_method 0 2023-10-30T12:23:11.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:11.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:11.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:11.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:11.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:11.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:11.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7991d9b640 1 --2- 172.21.15.50:0/2646787995 >> v2:172.21.15.118:3300/0 conn(0x7f799c070bb0 0x7f799c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:11.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7991d9b640 10 monclient(hunting): get_auth_request con 0x7f799c070bb0 auth_method 0 2023-10-30T12:23:11.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7991d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:11.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7991d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:11.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7991d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:11.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7999d9b640 1 -- 172.21.15.50:0/2646787995 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659319369 0 0) 0x7f799c09ee40 con 0x7f799c1345f0 2023-10-30T12:23:11.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7999d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:23:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7999d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7999d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:11.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7999d9b640 10 monclient(hunting): my global_id is 264348 2023-10-30T12:23:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f7999d9b640 1 -- 172.21.15.50:0/2646787995 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7988003520 con 0x7f799c1345f0 2023-10-30T12:23:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient(hunting): handle_auth_done global_id 264345 payload 274 2023-10-30T12:23:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 1 -- 172.21.15.50:0/2646787995 >> v2:172.21.15.118:3300/0 conn(0x7f799c070bb0 msgr2=0x7f799c071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:11.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 1 --2- 172.21.15.50:0/2646787995 >> v2:172.21.15.118:3300/0 conn(0x7f799c070bb0 0x7f799c071020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:11.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 1 -- 172.21.15.50:0/2646787995 >> v1:172.21.15.50:6789/0 conn(0x7f799c1345f0 legacy=0x7f799c134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:11.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:11.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 1 monclient: found mon.noname-a 2023-10-30T12:23:11.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:11.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:11.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.913+0000 7f799a59c640 1 -- 172.21.15.50:0/2646787995 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c09fab0 con 0x7f799c0707d0 2023-10-30T12:23:11.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f799a59c640 1 --2- 172.21.15.50:0/2646787995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c0707d0 0x7f799c09e7e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f799c09af70 tx=0x7f798c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c350285f79d7e2f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:11.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 1 -- 172.21.15.50:0/2646787995 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c031710 con 0x7f799c0707d0 2023-10-30T12:23:11.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:11.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:11.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: dump: 2023-10-30T12:23:11.940 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:11.940 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:11.941 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.941 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.941 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:11.941 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:11.942 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:11.942 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:11.942 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:11.943 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:11.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: _finish_auth 0 2023-10-30T12:23:11.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:23:11.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:11.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 1 -- 172.21.15.50:0/2646787995 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c02f050 con 0x7f799c0707d0 2023-10-30T12:23:11.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:11.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 1 -- 172.21.15.50:0/2646787995 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c031a90 con 0x7f799c0707d0 2023-10-30T12:23:11.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 5 monclient: authenticate success, global_id 264345 2023-10-30T12:23:11.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:11.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 4 set_mon_vals no callback set 2023-10-30T12:23:11.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:11.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:11.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f7999d9b640 10 monclient: dump: 2023-10-30T12:23:11.946 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:11.946 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:11.946 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.947 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.947 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:11.947 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:11.948 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:11.948 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:11.948 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:11.948 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:11.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 10 monclient: shutdown 2023-10-30T12:23:11.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:11.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 -- 172.21.15.50:0/2646787995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c0707d0 msgr2=0x7f799c09e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:11.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 --2- 172.21.15.50:0/2646787995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c0707d0 0x7f799c09e7e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f799c09af70 tx=0x7f798c02b7b0 comp rx=0 tx=0).stop 2023-10-30T12:23:11.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 -- 172.21.15.50:0/2646787995 shutdown_connections 2023-10-30T12:23:11.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 --2- 172.21.15.50:0/2646787995 >> v2:172.21.15.118:3300/0 conn(0x7f799c070bb0 0x7f799c071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:11.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 --2- 172.21.15.50:0/2646787995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c0707d0 0x7f799c09e7e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:11.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 -- 172.21.15.50:0/2646787995 >> 172.21.15.50:0/2646787995 conn(0x7f799c0a47f0 msgr2=0x7f799c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:11.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 -- 172.21.15.50:0/2646787995 shutdown_connections 2023-10-30T12:23:11.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.914+0000 7f79a0b83640 1 -- 172.21.15.50:0/2646787995 wait complete. 2023-10-30T12:23:11.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.915+0000 7f79a0b83640 10 monclient: build_initial_monmap 2023-10-30T12:23:11.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.915+0000 7f79a0b83640 10 monclient: monmap: 2023-10-30T12:23:11.952 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:11.952 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:11.952 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:11.953 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:11.953 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:11.953 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:11.953 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:11.954 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:11.954 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:11.954 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:11.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.915+0000 7f79a0b83640 1 librados: starting msgr at 2023-10-30T12:23:11.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.915+0000 7f79a0b83640 1 librados: starting objecter 2023-10-30T12:23:11.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 Processor -- start 2023-10-30T12:23:11.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 -- start start 2023-10-30T12:23:11.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 librados: setting wanted keys 2023-10-30T12:23:11.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 librados: calling monclient init 2023-10-30T12:23:11.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient: init 2023-10-30T12:23:11.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:11.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:23:11.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f799c0707d0 0x7f799c1ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:11.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): picked mon.noname-c con 0x7f799c0707d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:11.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c070bb0 0x7f799c131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:11.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): picked mon.noname-a con 0x7f799c070bb0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:11.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f799c1345f0 0x7f799c131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:11.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): picked mon.noname-b con 0x7f799c1345f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:11.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:11.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c120410 con 0x7f799c070bb0 2023-10-30T12:23:11.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:11.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c1cae00 con 0x7f799c1345f0 2023-10-30T12:23:11.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:11.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c120130 con 0x7f799c0707d0 2023-10-30T12:23:11.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:11.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f79a0b83640 10 monclient(hunting): authenticate will time out at 2237.099725s 2023-10-30T12:23:11.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c070bb0 0x7f799c131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:11.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799a59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f799c0707d0 0x7f799c1ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:11.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c070bb0 0x7f799c131a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59746/0 (socket says 172.21.15.50:59746) 2023-10-30T12:23:11.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 1 -- 172.21.15.50:0/3690745962 learned_addr learned my addr 172.21.15.50:0/3690745962 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:11.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 10 monclient(hunting): get_auth_request con 0x7f799c070bb0 auth_method 0 2023-10-30T12:23:11.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799a59c640 10 monclient(hunting): get_auth_request con 0x7f799c0707d0 auth_method 0 2023-10-30T12:23:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:11.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799a59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:11.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799a59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:11.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:11.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:11.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f7991d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:11.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:11.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:11.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.916+0000 7f799ad9d640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f799c1345f0 0x7f799c131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:11.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:11.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799ad9d640 10 monclient(hunting): get_auth_request con 0x7f799c1345f0 auth_method 0 2023-10-30T12:23:11.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:11.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799ad9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:11.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:11.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:11.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:11.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:11.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 10 monclient(hunting): handle_auth_done global_id 264351 payload 824 2023-10-30T12:23:11.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 1 -- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f799c0707d0 msgr2=0x7f799c1ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:11.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f799c0707d0 0x7f799c1ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:11.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 1 -- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f799c1345f0 msgr2=0x7f799c131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:11.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f799c1345f0 0x7f799c131f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:11.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:23:11.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 1 monclient: found mon.noname-a 2023-10-30T12:23:11.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:11.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:11.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 1 -- 172.21.15.50:0/3690745962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c009400 con 0x7f799c070bb0 2023-10-30T12:23:11.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799a59c640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f799c0707d0 0x7f799c1ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:11.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f7991d9b640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c070bb0 0x7f799c131a40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f798400d800 tx=0x7f798400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:11.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 client.?.objecter ms_handle_connect 0x7f799c070bb0 2023-10-30T12:23:11.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:11.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 1 -- 172.21.15.50:0/3690745962 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400b6c0 con 0x7f799c070bb0 2023-10-30T12:23:11.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:11.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:11.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: dump: 2023-10-30T12:23:11.973 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:11.973 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:11.973 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.973 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.974 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:11.974 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:11.974 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:11.974 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:11.975 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:11.975 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:11.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: _finish_auth 0 2023-10-30T12:23:11.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: _check_auth_tickets 2023-10-30T12:23:11.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:11.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 5 monclient: authenticate success, global_id 264351 2023-10-30T12:23:11.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 10 monclient: _renew_subs 2023-10-30T12:23:11.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 1 -- 172.21.15.50:0/3690745962 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798400b820 con 0x7f799c070bb0 2023-10-30T12:23:11.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:11.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c11fb70 con 0x7f799c070bb0 2023-10-30T12:23:11.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:11.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 1 -- 172.21.15.50:0/3690745962 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400bbb0 con 0x7f799c070bb0 2023-10-30T12:23:11.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:11.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:11.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79937fe640 10 monclient: dump: 2023-10-30T12:23:11.979 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:11.979 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:11.979 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.980 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:11.980 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:11.980 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:11.980 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:11.981 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:11.981 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:11.981 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:11.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f799959a640 4 set_mon_vals no callback set 2023-10-30T12:23:11.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 10 client.264351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:11.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 10 monclient: _renew_subs 2023-10-30T12:23:11.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:11.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c1caaa0 con 0x7f799c070bb0 2023-10-30T12:23:11.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.917+0000 7f79a0b83640 1 librados: init done 2023-10-30T12:23:11.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79937fe640 1 -- 172.21.15.50:0/3690745962 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f798400bd10 con 0x7f799c070bb0 2023-10-30T12:23:11.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79a0b83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:11.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79a0b83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:11.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79a0b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:11.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c0714e0 con 0x7f799c070bb0 2023-10-30T12:23:11.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79937fe640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f797c0392f0 0x7f797c03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:11.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79937fe640 1 -- 172.21.15.50:0/3690745962 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f798404ade0 con 0x7f799c070bb0 2023-10-30T12:23:11.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79937fe640 10 client.264351.objecter ms_dispatch 0x7f799c000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:11.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79937fe640 3 client.264351.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:11.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79937fe640 3 client.264351.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:11.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f79937fe640 20 client.264351.objecter dump_active .. 0 homeless 2023-10-30T12:23:11.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f799a59c640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f797c0392f0 0x7f797c03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:11.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.918+0000 7f799a59c640 10 monclient: get_auth_request con 0x7f797c0392f0 auth_method 0 2023-10-30T12:23:11.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.919+0000 7f799a59c640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f797c0392f0 0x7f797c03b7b0 secure :-1 s=READY pgs=4353 cs=0 l=1 rev1=1 crypto rx=0x7f798c035000 tx=0x7f798c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:11.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.919+0000 7f79937fe640 10 client.264351.objecter ms_handle_connect 0x7f797c0392f0 2023-10-30T12:23:11.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.923+0000 7f79937fe640 1 -- 172.21.15.50:0/3690745962 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7984014430 con 0x7f799c070bb0 2023-10-30T12:23:11.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.923+0000 7f79937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:11.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:11.923+0000 7f79937fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:12.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.170+0000 7f79a0b83640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2023-10-30T12:23:12.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.170+0000 7f79a0b83640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2023-10-30T12:23:12.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.170+0000 7f79a0b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.170+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f799c04e160 con 0x7f799c070bb0 2023-10-30T12:23:12.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.171+0000 7f79937fe640 1 -- 172.21.15.50:0/3690745962 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f798401d3c0 con 0x7f799c070bb0 2023-10-30T12:23:12.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.171+0000 7f79937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2023-10-30T12:23:12.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.171+0000 7f79937fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:23:12.175 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":228,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","modified":"2023-10-30T12:00:17.894366Z","created":"2023-10-30T12:00:17.894366Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-10-30T12:23:12.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 10 librados: watch_flush enter 2023-10-30T12:23:12.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 10 librados: watch_flush exit 2023-10-30T12:23:12.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 20 client.264351.objecter shutdown clearing up homeless session... 2023-10-30T12:23:12.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 10 client.264351.objecter successfully canceled tick 2023-10-30T12:23:12.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f797c0392f0 msgr2=0x7f797c03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f797c0392f0 0x7f797c03b7b0 secure :-1 s=READY pgs=4353 cs=0 l=1 rev1=1 crypto rx=0x7f798c035000 tx=0x7f798c008030 comp rx=0 tx=0).stop 2023-10-30T12:23:12.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 10 monclient: shutdown 2023-10-30T12:23:12.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:12.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c070bb0 msgr2=0x7f799c131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c070bb0 0x7f799c131a40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f798400d800 tx=0x7f798400dcc0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 shutdown_connections 2023-10-30T12:23:12.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f797c0392f0 0x7f797c03b7b0 unknown :-1 s=CLOSED pgs=4353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f799c1345f0 0x7f799c131f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f799c070bb0 0x7f799c131a40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 --2- 172.21.15.50:0/3690745962 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f799c0707d0 0x7f799c1ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 >> 172.21.15.50:0/3690745962 conn(0x7f799c0a47f0 msgr2=0x7f799c09c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:12.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 shutdown_connections 2023-10-30T12:23:12.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 -- 172.21.15.50:0/3690745962 wait complete. 2023-10-30T12:23:12.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.173+0000 7f79a0b83640 1 librados: shutdown 2023-10-30T12:23:12.185 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-10-30T12:23:12.185 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T12:23:12.195 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:23:12.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:23:12.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.301+0000 7f32657b6640 10 monclient: get_monmap_and_config 2023-10-30T12:23:12.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.301+0000 7f32657b6640 10 monclient: build_initial_monmap 2023-10-30T12:23:12.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.301+0000 7f32657b6640 10 monclient: monmap: 2023-10-30T12:23:12.304 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:12.304 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:12.302615+0000 2023-10-30T12:23:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:12.302615+0000 2023-10-30T12:23:12.305 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:12.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:12.307 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:12.307 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 1 Processor -- start 2023-10-30T12:23:12.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 1 -- start start 2023-10-30T12:23:12.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient: init 2023-10-30T12:23:12.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:12.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-10-30T12:23:12.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32600707d0 0x7f326009e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient(hunting): picked mon.noname-a con 0x7f32600707d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:12.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3260070bb0 0x7f3260070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient(hunting): picked mon.noname-b con 0x7f3260070bb0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:12.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32601345f0 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:12.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326009ee40 con 0x7f32601345f0 2023-10-30T12:23:12.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326009f4f0 con 0x7f32600707d0 2023-10-30T12:23:12.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326009f7d0 con 0x7f3260070bb0 2023-10-30T12:23:12.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:12.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.302+0000 7f32657b6640 10 monclient(hunting): authenticate will time out at 2237.485919s 2023-10-30T12:23:12.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3260070bb0 0x7f3260070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32600707d0 0x7f326009e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32600707d0 0x7f326009e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59748/0 (socket says 172.21.15.50:59748) 2023-10-30T12:23:12.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3260070bb0 0x7f3260070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60620/0 (socket says 172.21.15.50:60620) 2023-10-30T12:23:12.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 -- 172.21.15.50:0/1090219225 learned_addr learned my addr 172.21.15.50:0/1090219225 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:12.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): get_auth_request con 0x7f32600707d0 auth_method 0 2023-10-30T12:23:12.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 10 monclient(hunting): get_auth_request con 0x7f3260070bb0 auth_method 0 2023-10-30T12:23:12.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient(hunting): handle_auth_done global_id 264354 payload 274 2023-10-30T12:23:12.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 -- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3260070bb0 msgr2=0x7f3260070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:12.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 --2- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3260070bb0 0x7f3260070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 -- 172.21.15.50:0/1090219225 >> v1:172.21.15.118:6789/0 conn(0x7f32601345f0 legacy=0x7f3260134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:12.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient: _finish_hunting 0 2023-10-30T12:23:12.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 monclient: found mon.noname-a 2023-10-30T12:23:12.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:12.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f325effd640 1 -- 172.21.15.50:0/1090219225 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326009fab0 con 0x7f32600707d0 2023-10-30T12:23:12.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.303+0000 7f3257fff640 1 --2- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3260070bb0 0x7f3260070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:12.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325effd640 1 --2- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32600707d0 0x7f326009e7e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3250009870 tx=0x7f325002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7782bb227591ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 1 -- 172.21.15.50:0/1090219225 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250031740 con 0x7f32600707d0 2023-10-30T12:23:12.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:12.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: dump: 2023-10-30T12:23:12.331 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.331 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.332 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.332 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.332 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.332 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.333 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.333 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.333 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.333 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: _finish_auth 0 2023-10-30T12:23:12.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: _check_auth_tickets 2023-10-30T12:23:12.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:12.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 1 -- 172.21.15.50:0/1090219225 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32500318a0 con 0x7f32600707d0 2023-10-30T12:23:12.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 5 monclient: authenticate success, global_id 264354 2023-10-30T12:23:12.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:12.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:12.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:12.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 1 -- 172.21.15.50:0/1090219225 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250031c30 con 0x7f32600707d0 2023-10-30T12:23:12.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 4 set_mon_vals no callback set 2023-10-30T12:23:12.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:12.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f325e7fc640 10 monclient: dump: 2023-10-30T12:23:12.338 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.338 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.339 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.339 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.340 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.340 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.340 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.340 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.341 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.341 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 10 monclient: shutdown 2023-10-30T12:23:12.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:12.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 -- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32600707d0 msgr2=0x7f326009e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 --2- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32600707d0 0x7f326009e7e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3250009870 tx=0x7f325002b7e0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 -- 172.21.15.50:0/1090219225 shutdown_connections 2023-10-30T12:23:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 --2- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3260070bb0 0x7f3260070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 --2- 172.21.15.50:0/1090219225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32600707d0 0x7f326009e7e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 -- 172.21.15.50:0/1090219225 >> 172.21.15.50:0/1090219225 conn(0x7f32600a47f0 msgr2=0x7f32600a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:12.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 -- 172.21.15.50:0/1090219225 shutdown_connections 2023-10-30T12:23:12.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.304+0000 7f32657b6640 1 -- 172.21.15.50:0/1090219225 wait complete. 2023-10-30T12:23:12.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 10 monclient: build_initial_monmap 2023-10-30T12:23:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 10 monclient: monmap: 2023-10-30T12:23:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:12.347 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 1 librados: starting msgr at 2023-10-30T12:23:12.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 1 librados: starting objecter 2023-10-30T12:23:12.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 1 Processor -- start 2023-10-30T12:23:12.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 1 -- start start 2023-10-30T12:23:12.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 1 librados: setting wanted keys 2023-10-30T12:23:12.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 1 librados: calling monclient init 2023-10-30T12:23:12.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.305+0000 7f32657b6640 10 monclient: init 2023-10-30T12:23:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:23:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32600707d0 0x7f32601ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): picked mon.noname-c con 0x7f32600707d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3260070bb0 0x7f32601cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): picked mon.noname-a con 0x7f3260070bb0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32601345f0 0x7f326011c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): picked mon.noname-b con 0x7f32601345f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32601200e0 con 0x7f3260070bb0 2023-10-30T12:23:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326011f1f0 con 0x7f32601345f0 2023-10-30T12:23:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326009d6a0 con 0x7f32600707d0 2023-10-30T12:23:12.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:12.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient(hunting): authenticate will time out at 2237.489243s 2023-10-30T12:23:12.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32600707d0 0x7f32601ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3260070bb0 0x7f32601cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32600707d0 0x7f32601ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60622/0 (socket says 172.21.15.50:60622) 2023-10-30T12:23:12.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 -- 172.21.15.50:0/3900789205 learned_addr learned my addr 172.21.15.50:0/3900789205 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3260070bb0 0x7f32601cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59758/0 (socket says 172.21.15.50:59758) 2023-10-30T12:23:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): get_auth_request con 0x7f32600707d0 auth_method 0 2023-10-30T12:23:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 10 monclient(hunting): get_auth_request con 0x7f3260070bb0 auth_method 0 2023-10-30T12:23:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient(hunting): handle_auth_done global_id 284126 payload 824 2023-10-30T12:23:12.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 -- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32601345f0 msgr2=0x7f326011c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32601345f0 0x7f326011c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 -- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3260070bb0 msgr2=0x7f32601cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3260070bb0 0x7f32601cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient: _finish_hunting 0 2023-10-30T12:23:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 monclient: found mon.noname-c 2023-10-30T12:23:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:12.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 -- 172.21.15.50:0/3900789205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3248009400 con 0x7f32600707d0 2023-10-30T12:23:12.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f3257fff640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3260070bb0 0x7f32601cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:12.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325effd640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32600707d0 0x7f32601ca7d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3250009870 tx=0x7f32500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 client.?.objecter ms_handle_connect 0x7f32600707d0 2023-10-30T12:23:12.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:12.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 1 -- 172.21.15.50:0/3900789205 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250033a40 con 0x7f32600707d0 2023-10-30T12:23:12.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:12.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 monclient: dump: 2023-10-30T12:23:12.368 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.369 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.369 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.369 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.371 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.371 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.371 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.372 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 monclient: _finish_auth 0 2023-10-30T12:23:12.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 monclient: _check_auth_tickets 2023-10-30T12:23:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 5 monclient: authenticate success, global_id 284126 2023-10-30T12:23:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 1 -- 172.21.15.50:0/3900789205 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3250033ba0 con 0x7f32600707d0 2023-10-30T12:23:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient: _renew_subs 2023-10-30T12:23:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32500094d0 con 0x7f32600707d0 2023-10-30T12:23:12.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:12.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 1 -- 172.21.15.50:0/3900789205 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325003c400 con 0x7f32600707d0 2023-10-30T12:23:12.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f32577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 10 monclient: dump: 2023-10-30T12:23:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.378 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.378 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.378 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.379 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 10 client.284126.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.306+0000 7f325dffb640 4 set_mon_vals no callback set 2023-10-30T12:23:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 10 monclient: _renew_subs 2023-10-30T12:23:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3260120980 con 0x7f32600707d0 2023-10-30T12:23:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 1 librados: init done 2023-10-30T12:23:12.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 1 -- 172.21.15.50:0/3900789205 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f325003a040 con 0x7f32600707d0 2023-10-30T12:23:12.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32400392f0 0x7f324003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 1 -- 172.21.15.50:0/3900789205 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f325006c420 con 0x7f32600707d0 2023-10-30T12:23:12.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 10 client.284126.objecter ms_dispatch 0x7f3260000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:12.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 3 client.284126.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:12.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32577fe640 3 client.284126.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:12.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:12.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.307+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326004e160 con 0x7f32600707d0 2023-10-30T12:23:12.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.308+0000 7f32577fe640 20 client.284126.objecter dump_active .. 0 homeless 2023-10-30T12:23:12.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.308+0000 7f3257fff640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32400392f0 0x7f324003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.308+0000 7f3257fff640 10 monclient: get_auth_request con 0x7f32400392f0 auth_method 0 2023-10-30T12:23:12.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.308+0000 7f32577fe640 10 client.284126.objecter ms_handle_connect 0x7f32400392f0 2023-10-30T12:23:12.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.308+0000 7f3257fff640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32400392f0 0x7f324003b7b0 secure :-1 s=READY pgs=4354 cs=0 l=1 rev1=1 crypto rx=0x7f326009ded0 tx=0x7f3248009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.312+0000 7f32577fe640 1 -- 172.21.15.50:0/3900789205 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3250038020 con 0x7f32600707d0 2023-10-30T12:23:12.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.312+0000 7f32577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.312+0000 7f32577fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:12.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.488+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3260071530 con 0x7f32400392f0 2023-10-30T12:23:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.493+0000 7f32577fe640 1 -- 172.21.15.50:0/3900789205 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133807 (secure 0 0 0) 0x7f3248014440 con 0x7f32400392f0 2023-10-30T12:23:12.497 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:23:12.498 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:23:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 10 librados: watch_flush enter 2023-10-30T12:23:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 10 librados: watch_flush exit 2023-10-30T12:23:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 20 client.284126.objecter shutdown clearing up homeless session... 2023-10-30T12:23:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 10 client.284126.objecter successfully canceled tick 2023-10-30T12:23:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32400392f0 msgr2=0x7f324003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32400392f0 0x7f324003b7b0 secure :-1 s=READY pgs=4354 cs=0 l=1 rev1=1 crypto rx=0x7f326009ded0 tx=0x7f3248009190 comp rx=0 tx=0).stop 2023-10-30T12:23:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 10 monclient: shutdown 2023-10-30T12:23:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32600707d0 msgr2=0x7f32601ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32600707d0 0x7f32601ca7d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3250009870 tx=0x7f32500046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 shutdown_connections 2023-10-30T12:23:12.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f32400392f0 0x7f324003b7b0 unknown :-1 s=CLOSED pgs=4354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32601345f0 0x7f326011c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3260070bb0 0x7f32601cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 --2- 172.21.15.50:0/3900789205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32600707d0 0x7f32601ca7d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 >> 172.21.15.50:0/3900789205 conn(0x7f32600a47f0 msgr2=0x7f32600a3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 shutdown_connections 2023-10-30T12:23:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 -- 172.21.15.50:0/3900789205 wait complete. 2023-10-30T12:23:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.495+0000 7f32657b6640 1 librados: shutdown 2023-10-30T12:23:12.507 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1351,"stamp":"2023-10-30T12:23:10.735684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":429848,"kb_used_data":10840,"kb_used_omap":9,"kb_used_meta":418806,"kb_avail":565801192,"statfs":{"total":579820584960,"available":579380420608,"internally_reserved":0,"allocated":11100160,"data_stored":3863898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":428858023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002525"},"pg_stats":[{"pgid":"15.13","version":"0'0","reported_seq":650,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711339+0000","last_change":"2023-10-30T12:01:59.232802+0000","last_active":"2023-10-30T12:22:45.711339+0000","last_peered":"2023-10-30T12:22:45.711339+0000","last_clean":"2023-10-30T12:22:45.711339+0000","last_became_active":"2023-10-30T12:01:59.223072+0000","last_became_peered":"2023-10-30T12:01:59.223072+0000","last_unstale":"2023-10-30T12:22:45.711339+0000","last_undegraded":"2023-10-30T12:22:45.711339+0000","last_fullsized":"2023-10-30T12:22:45.711339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:39.226354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":643,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711074+0000","last_change":"2023-10-30T12:01:59.344564+0000","last_active":"2023-10-30T12:22:45.711074+0000","last_peered":"2023-10-30T12:22:45.711074+0000","last_clean":"2023-10-30T12:22:45.711074+0000","last_became_active":"2023-10-30T12:01:59.344017+0000","last_became_peered":"2023-10-30T12:01:59.344017+0000","last_unstale":"2023-10-30T12:22:45.711074+0000","last_undegraded":"2023-10-30T12:22:45.711074+0000","last_fullsized":"2023-10-30T12:22:45.711074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.878167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610023+0000","last_change":"2023-10-30T12:01:59.312117+0000","last_active":"2023-10-30T12:22:44.610023+0000","last_peered":"2023-10-30T12:22:44.610023+0000","last_clean":"2023-10-30T12:22:44.610023+0000","last_became_active":"2023-10-30T12:01:59.308994+0000","last_became_peered":"2023-10-30T12:01:59.308994+0000","last_unstale":"2023-10-30T12:22:44.610023+0000","last_undegraded":"2023-10-30T12:22:44.610023+0000","last_fullsized":"2023-10-30T12:22:44.610023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:12.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710486+0000","last_change":"2023-10-30T12:01:59.280468+0000","last_active":"2023-10-30T12:22:45.710486+0000","last_peered":"2023-10-30T12:22:45.710486+0000","last_clean":"2023-10-30T12:22:45.710486+0000","last_became_active":"2023-10-30T12:01:59.277217+0000","last_became_peered":"2023-10-30T12:01:59.277217+0000","last_unstale":"2023-10-30T12:22:45.710486+0000","last_undegraded":"2023-10-30T12:22:45.710486+0000","last_fullsized":"2023-10-30T12:22:45.710486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:39:50.184703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110225+0000","last_change":"2023-10-30T12:01:59.246329+0000","last_active":"2023-10-30T12:22:44.110225+0000","last_peered":"2023-10-30T12:22:44.110225+0000","last_clean":"2023-10-30T12:22:44.110225+0000","last_became_active":"2023-10-30T12:01:59.246101+0000","last_became_peered":"2023-10-30T12:01:59.246101+0000","last_unstale":"2023-10-30T12:22:44.110225+0000","last_undegraded":"2023-10-30T12:22:44.110225+0000","last_fullsized":"2023-10-30T12:22:44.110225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:17.822160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987154+0000","last_change":"2023-10-30T12:01:59.270798+0000","last_active":"2023-10-30T12:22:43.987154+0000","last_peered":"2023-10-30T12:22:43.987154+0000","last_clean":"2023-10-30T12:22:43.987154+0000","last_became_active":"2023-10-30T12:01:59.269001+0000","last_became_peered":"2023-10-30T12:01:59.269001+0000","last_unstale":"2023-10-30T12:22:43.987154+0000","last_undegraded":"2023-10-30T12:22:43.987154+0000","last_fullsized":"2023-10-30T12:22:43.987154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:36:36.641287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110448+0000","last_change":"2023-10-30T12:01:59.315912+0000","last_active":"2023-10-30T12:22:44.110448+0000","last_peered":"2023-10-30T12:22:44.110448+0000","last_clean":"2023-10-30T12:22:44.110448+0000","last_became_active":"2023-10-30T12:01:59.314812+0000","last_became_peered":"2023-10-30T12:01:59.314812+0000","last_unstale":"2023-10-30T12:22:44.110448+0000","last_undegraded":"2023-10-30T12:22:44.110448+0000","last_fullsized":"2023-10-30T12:22:44.110448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:11:29.969594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":664,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602031+0000","last_change":"2023-10-30T12:01:59.339356+0000","last_active":"2023-10-30T12:22:44.602031+0000","last_peered":"2023-10-30T12:22:44.602031+0000","last_clean":"2023-10-30T12:22:44.602031+0000","last_became_active":"2023-10-30T12:01:59.338820+0000","last_became_peered":"2023-10-30T12:01:59.338820+0000","last_unstale":"2023-10-30T12:22:44.602031+0000","last_undegraded":"2023-10-30T12:22:44.602031+0000","last_fullsized":"2023-10-30T12:22:44.602031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:18.746605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":663,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599775+0000","last_change":"2023-10-30T12:01:59.305090+0000","last_active":"2023-10-30T12:22:43.599775+0000","last_peered":"2023-10-30T12:22:43.599775+0000","last_clean":"2023-10-30T12:22:43.599775+0000","last_became_active":"2023-10-30T12:01:59.296246+0000","last_became_peered":"2023-10-30T12:01:59.296246+0000","last_unstale":"2023-10-30T12:22:43.599775+0000","last_undegraded":"2023-10-30T12:22:43.599775+0000","last_fullsized":"2023-10-30T12:22:43.599775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:40.407666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110494+0000","last_change":"2023-10-30T12:01:59.297023+0000","last_active":"2023-10-30T12:22:44.110494+0000","last_peered":"2023-10-30T12:22:44.110494+0000","last_clean":"2023-10-30T12:22:44.110494+0000","last_became_active":"2023-10-30T12:01:59.296511+0000","last_became_peered":"2023-10-30T12:01:59.296511+0000","last_unstale":"2023-10-30T12:22:44.110494+0000","last_undegraded":"2023-10-30T12:22:44.110494+0000","last_fullsized":"2023-10-30T12:22:44.110494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:45.585836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109889+0000","last_change":"2023-10-30T12:01:59.286076+0000","last_active":"2023-10-30T12:22:44.109889+0000","last_peered":"2023-10-30T12:22:44.109889+0000","last_clean":"2023-10-30T12:22:44.109889+0000","last_became_active":"2023-10-30T12:01:59.282069+0000","last_became_peered":"2023-10-30T12:01:59.282069+0000","last_unstale":"2023-10-30T12:22:44.109889+0000","last_undegraded":"2023-10-30T12:22:44.109889+0000","last_fullsized":"2023-10-30T12:22:44.109889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:33.906969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.986878+0000","last_change":"2023-10-30T12:01:59.267278+0000","last_active":"2023-10-30T12:22:43.986878+0000","last_peered":"2023-10-30T12:22:43.986878+0000","last_clean":"2023-10-30T12:22:43.986878+0000","last_became_active":"2023-10-30T12:01:59.266725+0000","last_became_peered":"2023-10-30T12:01:59.266725+0000","last_unstale":"2023-10-30T12:22:43.986878+0000","last_undegraded":"2023-10-30T12:22:43.986878+0000","last_fullsized":"2023-10-30T12:22:43.986878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:13.026323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599780+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:22:43.599780+0000","last_peered":"2023-10-30T12:22:43.599780+0000","last_clean":"2023-10-30T12:22:43.599780+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:22:43.599780+0000","last_undegraded":"2023-10-30T12:22:43.599780+0000","last_fullsized":"2023-10-30T12:22:43.599780+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609116+0000","last_change":"2023-10-30T12:01:59.274700+0000","last_active":"2023-10-30T12:22:43.609116+0000","last_peered":"2023-10-30T12:22:43.609116+0000","last_clean":"2023-10-30T12:22:43.609116+0000","last_became_active":"2023-10-30T12:01:59.274444+0000","last_became_peered":"2023-10-30T12:01:59.274444+0000","last_unstale":"2023-10-30T12:22:43.609116+0000","last_undegraded":"2023-10-30T12:22:43.609116+0000","last_fullsized":"2023-10-30T12:22:43.609116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:14.789781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110011+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:22:44.110011+0000","last_peered":"2023-10-30T12:22:44.110011+0000","last_clean":"2023-10-30T12:22:44.110011+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:22:44.110011+0000","last_undegraded":"2023-10-30T12:22:44.110011+0000","last_fullsized":"2023-10-30T12:22:44.110011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609261+0000","last_change":"2023-10-30T12:01:59.272386+0000","last_active":"2023-10-30T12:22:43.609261+0000","last_peered":"2023-10-30T12:22:43.609261+0000","last_clean":"2023-10-30T12:22:43.609261+0000","last_became_active":"2023-10-30T12:01:59.272125+0000","last_became_peered":"2023-10-30T12:01:59.272125+0000","last_unstale":"2023-10-30T12:22:43.609261+0000","last_undegraded":"2023-10-30T12:22:43.609261+0000","last_fullsized":"2023-10-30T12:22:43.609261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:15.708175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599879+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:22:43.599879+0000","last_peered":"2023-10-30T12:22:43.599879+0000","last_clean":"2023-10-30T12:22:43.599879+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:22:43.599879+0000","last_undegraded":"2023-10-30T12:22:43.599879+0000","last_fullsized":"2023-10-30T12:22:43.599879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609202+0000","last_change":"2023-10-30T12:01:59.269863+0000","last_active":"2023-10-30T12:22:43.609202+0000","last_peered":"2023-10-30T12:22:43.609202+0000","last_clean":"2023-10-30T12:22:43.609202+0000","last_became_active":"2023-10-30T12:01:59.269127+0000","last_became_peered":"2023-10-30T12:01:59.269127+0000","last_unstale":"2023-10-30T12:22:43.609202+0000","last_undegraded":"2023-10-30T12:22:43.609202+0000","last_fullsized":"2023-10-30T12:22:43.609202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:31:20.350152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110004+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:22:44.110004+0000","last_peered":"2023-10-30T12:22:44.110004+0000","last_clean":"2023-10-30T12:22:44.110004+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:22:44.110004+0000","last_undegraded":"2023-10-30T12:22:44.110004+0000","last_fullsized":"2023-10-30T12:22:44.110004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110202+0000","last_change":"2023-10-30T12:01:59.306784+0000","last_active":"2023-10-30T12:22:44.110202+0000","last_peered":"2023-10-30T12:22:44.110202+0000","last_clean":"2023-10-30T12:22:44.110202+0000","last_became_active":"2023-10-30T12:01:59.306293+0000","last_became_peered":"2023-10-30T12:01:59.306293+0000","last_unstale":"2023-10-30T12:22:44.110202+0000","last_undegraded":"2023-10-30T12:22:44.110202+0000","last_fullsized":"2023-10-30T12:22:44.110202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:26:17.267254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110433+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:22:44.110433+0000","last_peered":"2023-10-30T12:22:44.110433+0000","last_clean":"2023-10-30T12:22:44.110433+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:22:44.110433+0000","last_undegraded":"2023-10-30T12:22:44.110433+0000","last_fullsized":"2023-10-30T12:22:44.110433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":664,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599745+0000","last_change":"2023-10-30T12:01:59.314586+0000","last_active":"2023-10-30T12:22:43.599745+0000","last_peered":"2023-10-30T12:22:43.599745+0000","last_clean":"2023-10-30T12:22:43.599745+0000","last_became_active":"2023-10-30T12:01:59.313414+0000","last_became_peered":"2023-10-30T12:01:59.313414+0000","last_unstale":"2023-10-30T12:22:43.599745+0000","last_undegraded":"2023-10-30T12:22:43.599745+0000","last_fullsized":"2023-10-30T12:22:43.599745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:05.821606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609150+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:22:43.609150+0000","last_peered":"2023-10-30T12:22:43.609150+0000","last_clean":"2023-10-30T12:22:43.609150+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:22:43.609150+0000","last_undegraded":"2023-10-30T12:22:43.609150+0000","last_fullsized":"2023-10-30T12:22:43.609150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110338+0000","last_change":"2023-10-30T12:01:59.289541+0000","last_active":"2023-10-30T12:22:44.110338+0000","last_peered":"2023-10-30T12:22:44.110338+0000","last_clean":"2023-10-30T12:22:44.110338+0000","last_became_active":"2023-10-30T12:01:59.289175+0000","last_became_peered":"2023-10-30T12:01:59.289175+0000","last_unstale":"2023-10-30T12:22:44.110338+0000","last_undegraded":"2023-10-30T12:22:44.110338+0000","last_fullsized":"2023-10-30T12:22:44.110338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:01.620219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":668,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601903+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:22:44.601903+0000","last_peered":"2023-10-30T12:22:44.601903+0000","last_clean":"2023-10-30T12:22:44.601903+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:22:44.601903+0000","last_undegraded":"2023-10-30T12:22:44.601903+0000","last_fullsized":"2023-10-30T12:22:44.601903+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110134+0000","last_change":"2023-10-30T12:01:59.307590+0000","last_active":"2023-10-30T12:22:44.110134+0000","last_peered":"2023-10-30T12:22:44.110134+0000","last_clean":"2023-10-30T12:22:44.110134+0000","last_became_active":"2023-10-30T12:01:59.302155+0000","last_became_peered":"2023-10-30T12:01:59.302155+0000","last_unstale":"2023-10-30T12:22:44.110134+0000","last_undegraded":"2023-10-30T12:22:44.110134+0000","last_fullsized":"2023-10-30T12:22:44.110134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:48.321197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110344+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:22:44.110344+0000","last_peered":"2023-10-30T12:22:44.110344+0000","last_clean":"2023-10-30T12:22:44.110344+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:22:44.110344+0000","last_undegraded":"2023-10-30T12:22:44.110344+0000","last_fullsized":"2023-10-30T12:22:44.110344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":820,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602127+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:22:44.602127+0000","last_peered":"2023-10-30T12:22:44.602127+0000","last_clean":"2023-10-30T12:22:44.602127+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:22:44.602127+0000","last_undegraded":"2023-10-30T12:22:44.602127+0000","last_fullsized":"2023-10-30T12:22:44.602127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609190+0000","last_change":"2023-10-30T12:01:59.256285+0000","last_active":"2023-10-30T12:22:43.609190+0000","last_peered":"2023-10-30T12:22:43.609190+0000","last_clean":"2023-10-30T12:22:43.609190+0000","last_became_active":"2023-10-30T12:01:59.253931+0000","last_became_peered":"2023-10-30T12:01:59.253931+0000","last_unstale":"2023-10-30T12:22:43.609190+0000","last_undegraded":"2023-10-30T12:22:43.609190+0000","last_fullsized":"2023-10-30T12:22:43.609190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:21.312914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711364+0000","last_change":"2023-10-30T12:01:59.296719+0000","last_active":"2023-10-30T12:22:45.711364+0000","last_peered":"2023-10-30T12:22:45.711364+0000","last_clean":"2023-10-30T12:22:45.711364+0000","last_became_active":"2023-10-30T12:01:59.277183+0000","last_became_peered":"2023-10-30T12:01:59.277183+0000","last_unstale":"2023-10-30T12:22:45.711364+0000","last_undegraded":"2023-10-30T12:22:45.711364+0000","last_fullsized":"2023-10-30T12:22:45.711364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:49.956478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":665,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602043+0000","last_change":"2023-10-30T12:01:59.339853+0000","last_active":"2023-10-30T12:22:44.602043+0000","last_peered":"2023-10-30T12:22:44.602043+0000","last_clean":"2023-10-30T12:22:44.602043+0000","last_became_active":"2023-10-30T12:01:59.339614+0000","last_became_peered":"2023-10-30T12:01:59.339614+0000","last_unstale":"2023-10-30T12:22:44.602043+0000","last_undegraded":"2023-10-30T12:22:44.602043+0000","last_fullsized":"2023-10-30T12:22:44.602043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:21.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987106+0000","last_change":"2023-10-30T12:01:59.303724+0000","last_active":"2023-10-30T12:22:43.987106+0000","last_peered":"2023-10-30T12:22:43.987106+0000","last_clean":"2023-10-30T12:22:43.987106+0000","last_became_active":"2023-10-30T12:01:59.303216+0000","last_became_peered":"2023-10-30T12:01:59.303216+0000","last_unstale":"2023-10-30T12:22:43.987106+0000","last_undegraded":"2023-10-30T12:22:43.987106+0000","last_fullsized":"2023-10-30T12:22:43.987106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:35.112009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109953+0000","last_change":"2023-10-30T12:01:59.209818+0000","last_active":"2023-10-30T12:22:44.109953+0000","last_peered":"2023-10-30T12:22:44.109953+0000","last_clean":"2023-10-30T12:22:44.109953+0000","last_became_active":"2023-10-30T12:01:59.209551+0000","last_became_peered":"2023-10-30T12:01:59.209551+0000","last_unstale":"2023-10-30T12:22:44.109953+0000","last_undegraded":"2023-10-30T12:22:44.109953+0000","last_fullsized":"2023-10-30T12:22:44.109953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:49.423499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":653,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609027+0000","last_change":"2023-10-30T12:01:59.312819+0000","last_active":"2023-10-30T12:22:43.609027+0000","last_peered":"2023-10-30T12:22:43.609027+0000","last_clean":"2023-10-30T12:22:43.609027+0000","last_became_active":"2023-10-30T12:01:59.310341+0000","last_became_peered":"2023-10-30T12:01:59.310341+0000","last_unstale":"2023-10-30T12:22:43.609027+0000","last_undegraded":"2023-10-30T12:22:43.609027+0000","last_fullsized":"2023-10-30T12:22:43.609027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:52.592573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"18'7","reported_seq":678,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601944+0000","last_change":"2023-10-30T12:01:59.261094+0000","last_active":"2023-10-30T12:22:44.601944+0000","last_peered":"2023-10-30T12:22:44.601944+0000","last_clean":"2023-10-30T12:22:44.601944+0000","last_became_active":"2023-10-30T12:01:59.260849+0000","last_became_peered":"2023-10-30T12:01:59.260849+0000","last_unstale":"2023-10-30T12:22:44.601944+0000","last_undegraded":"2023-10-30T12:22:44.601944+0000","last_fullsized":"2023-10-30T12:22:44.601944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:36.395779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610042+0000","last_change":"2023-10-30T12:08:57.463451+0000","last_active":"2023-10-30T12:22:44.610042+0000","last_peered":"2023-10-30T12:22:44.610042+0000","last_clean":"2023-10-30T12:22:44.610042+0000","last_became_active":"2023-10-30T12:08:57.463102+0000","last_became_peered":"2023-10-30T12:08:57.463102+0000","last_unstale":"2023-10-30T12:22:44.610042+0000","last_undegraded":"2023-10-30T12:22:44.610042+0000","last_fullsized":"2023-10-30T12:22:44.610042+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:48.599858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710660+0000","last_change":"2023-10-30T12:01:59.247352+0000","last_active":"2023-10-30T12:22:45.710660+0000","last_peered":"2023-10-30T12:22:45.710660+0000","last_clean":"2023-10-30T12:22:45.710660+0000","last_became_active":"2023-10-30T12:01:59.246868+0000","last_became_peered":"2023-10-30T12:01:59.246868+0000","last_unstale":"2023-10-30T12:22:45.710660+0000","last_undegraded":"2023-10-30T12:22:45.710660+0000","last_fullsized":"2023-10-30T12:22:45.710660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:50.392311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":649,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109897+0000","last_change":"2023-10-30T12:01:59.276982+0000","last_active":"2023-10-30T12:22:44.109897+0000","last_peered":"2023-10-30T12:22:44.109897+0000","last_clean":"2023-10-30T12:22:44.109897+0000","last_became_active":"2023-10-30T12:01:59.239117+0000","last_became_peered":"2023-10-30T12:01:59.239117+0000","last_unstale":"2023-10-30T12:22:44.109897+0000","last_undegraded":"2023-10-30T12:22:44.109897+0000","last_fullsized":"2023-10-30T12:22:44.109897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.157663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110047+0000","last_change":"2023-10-30T12:01:59.297769+0000","last_active":"2023-10-30T12:22:44.110047+0000","last_peered":"2023-10-30T12:22:44.110047+0000","last_clean":"2023-10-30T12:22:44.110047+0000","last_became_active":"2023-10-30T12:01:59.293055+0000","last_became_peered":"2023-10-30T12:01:59.293055+0000","last_unstale":"2023-10-30T12:22:44.110047+0000","last_undegraded":"2023-10-30T12:22:44.110047+0000","last_fullsized":"2023-10-30T12:22:44.110047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:53.053266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609249+0000","last_change":"2023-10-30T12:01:59.310309+0000","last_active":"2023-10-30T12:22:43.609249+0000","last_peered":"2023-10-30T12:22:43.609249+0000","last_clean":"2023-10-30T12:22:43.609249+0000","last_became_active":"2023-10-30T12:01:59.307440+0000","last_became_peered":"2023-10-30T12:01:59.307440+0000","last_unstale":"2023-10-30T12:22:43.609249+0000","last_undegraded":"2023-10-30T12:22:43.609249+0000","last_fullsized":"2023-10-30T12:22:43.609249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:39.454237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711399+0000","last_change":"2023-10-30T12:01:59.249544+0000","last_active":"2023-10-30T12:22:45.711399+0000","last_peered":"2023-10-30T12:22:45.711399+0000","last_clean":"2023-10-30T12:22:45.711399+0000","last_became_active":"2023-10-30T12:01:59.239219+0000","last_became_peered":"2023-10-30T12:01:59.239219+0000","last_unstale":"2023-10-30T12:22:45.711399+0000","last_undegraded":"2023-10-30T12:22:45.711399+0000","last_fullsized":"2023-10-30T12:22:45.711399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:17.868254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640526,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77256,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294584,"statfs":{"total":96636764160,"available":96557654016,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.82699999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.65000000000000002,"5min":0.81599999999999995,"15min":1.056},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65400000000000003,"5min":0.83399999999999996,"15min":2.173},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.28299999999999997},"max":{"1min":0.64800000000000002,"5min":0.72799999999999998,"15min":0.78100000000000003},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.67400000000000004,"5min":0.74299999999999999,"15min":0.79900000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.41499999999999998,"15min":0.32900000000000001},"max":{"1min":0.65000000000000002,"5min":0.79900000000000004,"15min":0.84699999999999998},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.36899999999999999,"15min":0.373},"min":{"1min":0.25,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":0.499,"5min":0.58799999999999997,"15min":0.70399999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.223},"max":{"1min":0.54500000000000004,"5min":0.69699999999999995,"15min":0.95199999999999996},"last":0.377}]},{"osd":4,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.31,"5min":0.252,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.63700000000000001,"15min":0.63800000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.435,"15min":0.45300000000000001},"min":{"1min":0.28100000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.60299999999999998,"5min":0.68400000000000005,"15min":0.72099999999999997},"last":0.54500000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81416,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94290424,"statfs":{"total":96636764160,"available":96553394176,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53900000000000003,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.51400000000000001},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.84899999999999998},"last":0.60699999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.79000000000000004,"15min":0.94099999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.36299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.67700000000000005,"5min":0.749,"15min":0.79800000000000004},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27300000000000002},"max":{"1min":0.627,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.246},"max":{"1min":0.59399999999999997,"5min":0.627,"15min":0.63600000000000001},"last":0.376},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.25,"5min":0.249,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.67000000000000004,"15min":0.77100000000000002},"last":0.38700000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.435,"15min":0.40999999999999998},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.25600000000000001},"max":{"1min":0.621,"5min":0.64000000000000001,"15min":0.69999999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.67400000000000004,"15min":0.89700000000000002},"last":0.54700000000000004}]}]},{"osd":3,"up_from":11,"seq":47244640527,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69444,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302396,"statfs":{"total":96636764160,"available":96565653504,"internally_reserved":0,"allocated":2207744,"data_stored":996605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.503,"15min":0.501},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72799999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.61899999999999999,"5min":0.71399999999999997,"15min":1.2490000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.68999999999999995,"5min":0.81100000000000005,"15min":1.2689999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69999999999999996,"5min":0.82599999999999996,"15min":1.337},"last":0.67900000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":0.88900000000000001,"15min":1.302},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":0.746,"15min":0.76900000000000002},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.37,"15min":0.38},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":0.50700000000000001,"5min":0.61199999999999999,"15min":1.371},"last":0.495},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.30599999999999999},"max":{"1min":0.56599999999999995,"5min":0.66300000000000003,"15min":1.444},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39600000000000002,"15min":0.38700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":0.502,"5min":0.64200000000000002,"15min":0.876},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.375,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53500000000000003,"5min":0.84099999999999997,"15min":1.3540000000000001},"last":0.627}]}]},{"osd":2,"up_from":11,"seq":47244640526,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64764,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94307076,"statfs":{"total":96636764160,"available":96570445824,"internally_reserved":0,"allocated":1675264,"data_stored":467677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33500000000000002},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44800000000000001,"5min":0.50700000000000001,"15min":2.2730000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.38400000000000001},"min":{"1min":0.27500000000000002,"5min":0.24199999999999999,"15min":0.22800000000000001},"max":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":2.141},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33700000000000002,"15min":0.34200000000000003},"min":{"1min":0.24399999999999999,"5min":0.21299999999999999,"15min":0.20799999999999999},"max":{"1min":0.46999999999999997,"5min":0.502,"15min":1.087},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.24399999999999999},"max":{"1min":0.495,"5min":0.63900000000000001,"15min":3.8260000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.42199999999999999,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":0.749,"5min":0.75700000000000001,"15min":1.143},"last":0.5},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.42899999999999999,"5min":0.33600000000000002,"15min":0.314},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":3.4729999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.34399999999999997,"15min":0.27300000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.3240000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.45600000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":1.8340000000000001},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.32200000000000001},"max":{"1min":0.81699999999999995,"5min":0.81999999999999995,"15min":1.893},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.48499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75900000000000001,"5min":0.81599999999999995,"15min":2.2269999999999999},"last":0.65400000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640528,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59208,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312632,"statfs":{"total":96636764160,"available":96576135168,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.34499999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":0.46899999999999997,"5min":0.745,"15min":0.745},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.374},"min":{"1min":0.253,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.38200000000000001},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.223},"max":{"1min":0.54000000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.255},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.51800000000000002,"5min":0.621,"15min":0.93200000000000005},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.443,"5min":0.375,"15min":0.28499999999999998},"max":{"1min":0.69999999999999996,"5min":0.86599999999999999,"15min":0.879},"last":0.435},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.79900000000000004,"5min":0.85399999999999998,"15min":0.88900000000000001},"last":0.501}]},{"osd":4,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.46400000000000002,"5min":0.41699999999999998,"15min":0.29899999999999999},"max":{"1min":0.86799999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.36799999999999999},"max":{"1min":0.71299999999999997,"5min":0.94799999999999995,"15min":0.95599999999999996},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.58899999999999997},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.39300000000000002},"max":{"1min":0.84499999999999997,"5min":1,"15min":1},"last":0.46600000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640528,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77760,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294080,"statfs":{"total":96636764160,"available":96557137920,"internally_reserved":0,"allocated":2203648,"data_stored":996600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.35199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23899999999999999},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.21299999999999999},"max":{"1min":1.802,"5min":1.802,"15min":3.8519999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26700000000000002,"15min":0.20100000000000001},"max":{"1min":0.58699999999999997,"5min":0.623,"15min":2.0590000000000002},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.27800000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":1.5489999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":1.78},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.29399999999999998},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59099999999999997,"15min":0.57899999999999996},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":1.6020000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:23:12.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:23:12.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.616+0000 7fcb62421640 10 monclient: get_monmap_and_config 2023-10-30T12:23:12.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.616+0000 7fcb62421640 10 monclient: build_initial_monmap 2023-10-30T12:23:12.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.616+0000 7fcb62421640 10 monclient: monmap: 2023-10-30T12:23:12.619 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:12.619 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.619 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:12.617080+0000 2023-10-30T12:23:12.620 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:12.617080+0000 2023-10-30T12:23:12.620 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:12.620 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.621 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:12.621 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:12.621 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:12.622 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:12.622 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:12.622 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.616+0000 7fcb62421640 1 Processor -- start 2023-10-30T12:23:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.616+0000 7fcb62421640 1 -- start start 2023-10-30T12:23:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.616+0000 7fcb62421640 10 monclient: init 2023-10-30T12:23:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:12.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-10-30T12:23:12.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcb5c0659d0 0x7fcb5c1334d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient(hunting): picked mon.noname-a con 0x7fcb5c0659d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fcb5c071410 0x7fcb5c09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5c071410 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb5c071850 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c09fb50 con 0x7fcb5c071850 2023-10-30T12:23:12.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5c065db0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5c09dd70 con 0x7fcb5c071410 2023-10-30T12:23:12.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:12.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb62421640 10 monclient(hunting): authenticate will time out at 2237.800341s 2023-10-30T12:23:12.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcb5c0659d0 0x7fcb5c1334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb609be640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcb5c071850 0x7fcb5c09ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49586/0 (socket says 172.21.15.50:49586) 2023-10-30T12:23:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcb5c0659d0 0x7fcb5c1334d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59774/0 (socket says 172.21.15.50:59774) 2023-10-30T12:23:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb609be640 1 -- 172.21.15.50:0/3310681676 learned_addr learned my addr 172.21.15.50:0/3310681676 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): get_auth_request con 0x7fcb5c0659d0 auth_method 0 2023-10-30T12:23:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5b7fe640 1 --2- 172.21.15.50:0/3310681676 >> v2:172.21.15.118:3300/0 conn(0x7fcb5c071410 0x7fcb5c09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb5c071410 auth_method 0 2023-10-30T12:23:12.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_done global_id 264360 payload 274 2023-10-30T12:23:12.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 1 -- 172.21.15.50:0/3310681676 >> v2:172.21.15.118:3300/0 conn(0x7fcb5c071410 msgr2=0x7fcb5c09e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:12.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 1 --2- 172.21.15.50:0/3310681676 >> v2:172.21.15.118:3300/0 conn(0x7fcb5c071410 0x7fcb5c09e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 1 -- 172.21.15.50:0/3310681676 >> v1:172.21.15.50:6789/0 conn(0x7fcb5c071850 legacy=0x7fcb5c09ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:23:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 1 monclient: found mon.noname-a 2023-10-30T12:23:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:12.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5affd640 1 -- 172.21.15.50:0/3310681676 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427085734 0 0) 0x7fcb5c09fb50 con 0x7fcb5c071850 2023-10-30T12:23:12.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5bfff640 1 -- 172.21.15.50:0/3310681676 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c133b50 con 0x7fcb5c0659d0 2023-10-30T12:23:12.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.617+0000 7fcb5affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5bfff640 1 --2- 172.21.15.50:0/3310681676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c1334d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c066230 tx=0x7fcb5002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9b784fff858a52 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 1 -- 172.21.15.50:0/3310681676 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5002e070 con 0x7fcb5c0659d0 2023-10-30T12:23:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: dump: 2023-10-30T12:23:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.646 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.646 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.646 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.646 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.647 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.647 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.647 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.648 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: _finish_auth 0 2023-10-30T12:23:12.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:12.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:12.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 1 -- 172.21.15.50:0/3310681676 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb500315a0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb62421640 5 monclient: authenticate success, global_id 264360 2023-10-30T12:23:12.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb62421640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:12.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:12.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb62421640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:12.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 1 -- 172.21.15.50:0/3310681676 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50031930 con 0x7fcb5c0659d0 2023-10-30T12:23:12.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb62421640 4 set_mon_vals no callback set 2023-10-30T12:23:12.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:12.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.618+0000 7fcb5affd640 10 monclient: dump: 2023-10-30T12:23:12.651 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.652 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.652 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.652 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.652 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.653 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.653 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.653 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.653 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.654 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 10 monclient: shutdown 2023-10-30T12:23:12.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:12.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 -- 172.21.15.50:0/3310681676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 msgr2=0x7fcb5c1334d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 --2- 172.21.15.50:0/3310681676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c1334d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c066230 tx=0x7fcb5002b840 comp rx=0 tx=0).stop 2023-10-30T12:23:12.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 -- 172.21.15.50:0/3310681676 shutdown_connections 2023-10-30T12:23:12.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 --2- 172.21.15.50:0/3310681676 >> v2:172.21.15.118:3300/0 conn(0x7fcb5c071410 0x7fcb5c09e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 --2- 172.21.15.50:0/3310681676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c1334d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 -- 172.21.15.50:0/3310681676 >> 172.21.15.50:0/3310681676 conn(0x7fcb5c0a47f0 msgr2=0x7fcb5c0a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:12.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 -- 172.21.15.50:0/3310681676 shutdown_connections 2023-10-30T12:23:12.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 1 -- 172.21.15.50:0/3310681676 wait complete. 2023-10-30T12:23:12.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 10 monclient: build_initial_monmap 2023-10-30T12:23:12.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.619+0000 7fcb62421640 10 monclient: monmap: 2023-10-30T12:23:12.657 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:12.657 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:12.658 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:12.658 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:12.658 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:12.659 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.659 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:12.659 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:12.659 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:12.660 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 librados: starting msgr at 2023-10-30T12:23:12.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 librados: starting objecter 2023-10-30T12:23:12.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 Processor -- start 2023-10-30T12:23:12.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 -- start start 2023-10-30T12:23:12.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 librados: setting wanted keys 2023-10-30T12:23:12.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 librados: calling monclient init 2023-10-30T12:23:12.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient: init 2023-10-30T12:23:12.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:12.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:23:12.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c131cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): picked mon.noname-a con 0x7fcb5c0659d0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:12.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb5c071410 0x7fcb5c129b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5c071410 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:12.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb5c071850 0x7fcb5c12a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): picked mon.noname-b con 0x7fcb5c071850 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:12.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c12cdb0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c09e100 con 0x7fcb5c071850 2023-10-30T12:23:12.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c131520 con 0x7fcb5c071410 2023-10-30T12:23:12.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb62421640 10 monclient(hunting): authenticate will time out at 2237.803835s 2023-10-30T12:23:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c131cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5b7fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb5c071410 0x7fcb5c129b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5bfff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c131cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59782/0 (socket says 172.21.15.50:59782) 2023-10-30T12:23:12.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5bfff640 1 -- 172.21.15.50:0/2459068001 learned_addr learned my addr 172.21.15.50:0/2459068001 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:12.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5bfff640 10 monclient(hunting): get_auth_request con 0x7fcb5c0659d0 auth_method 0 2023-10-30T12:23:12.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5bfff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5bfff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb5c071410 auth_method 0 2023-10-30T12:23:12.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5b7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.620+0000 7fcb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb5c071850 0x7fcb5c12a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 10 monclient(hunting): get_auth_request con 0x7fcb5c071850 auth_method 0 2023-10-30T12:23:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb609be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 10 monclient(hunting): handle_auth_done global_id 264366 payload 824 2023-10-30T12:23:12.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 1 -- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb5c071410 msgr2=0x7fcb5c129b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb5c071410 0x7fcb5c129b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 1 -- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb5c071850 msgr2=0x7fcb5c12a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb5c071850 0x7fcb5c12a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 10 monclient: _finish_hunting 0 2023-10-30T12:23:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 1 monclient: found mon.noname-a 2023-10-30T12:23:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 1 -- 172.21.15.50:0/2459068001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb50009400 con 0x7fcb5c0659d0 2023-10-30T12:23:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5b7fe640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb5c071410 0x7fcb5c129b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5bfff640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c131cb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009890 tx=0x7fcb500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 client.?.objecter ms_handle_connect 0x7fcb5c0659d0 2023-10-30T12:23:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 1 -- 172.21.15.50:0/2459068001 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50032840 con 0x7fcb5c0659d0 2023-10-30T12:23:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:12.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: dump: 2023-10-30T12:23:12.679 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.679 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.679 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.680 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.680 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.680 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.681 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.681 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.681 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.681 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: _finish_auth 0 2023-10-30T12:23:12.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:23:12.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 1 -- 172.21.15.50:0/2459068001 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb500329a0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 1 -- 172.21.15.50:0/2459068001 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50032d50 con 0x7fcb5c0659d0 2023-10-30T12:23:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 5 monclient: authenticate success, global_id 264366 2023-10-30T12:23:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 10 monclient: _renew_subs 2023-10-30T12:23:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c12fb10 con 0x7fcb5c0659d0 2023-10-30T12:23:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb5a7fc640 4 set_mon_vals no callback set 2023-10-30T12:23:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb58ff9640 10 monclient: dump: 2023-10-30T12:23:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.687 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.687 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.687 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.688 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.688 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.688 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.688 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 10 client.264366.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:12.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 10 monclient: _renew_subs 2023-10-30T12:23:12.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c09cfb0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.621+0000 7fcb62421640 1 librados: init done 2023-10-30T12:23:12.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb58ff9640 1 -- 172.21.15.50:0/2459068001 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fcb5004d3f0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb58ff9640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb480392f0 0x7fcb4803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb62421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb62421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb62421640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb58ff9640 1 -- 172.21.15.50:0/2459068001 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7fcb5006c4f0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb58ff9640 10 client.264366.objecter ms_dispatch 0x7fcb5c000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:12.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb58ff9640 3 client.264366.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:12.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb58ff9640 3 client.264366.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:12.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c04e160 con 0x7fcb5c0659d0 2023-10-30T12:23:12.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb58ff9640 20 client.264366.objecter dump_active .. 0 homeless 2023-10-30T12:23:12.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb5b7fe640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb480392f0 0x7fcb4803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.622+0000 7fcb5b7fe640 10 monclient: get_auth_request con 0x7fcb480392f0 auth_method 0 2023-10-30T12:23:12.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.623+0000 7fcb58ff9640 10 client.264366.objecter ms_handle_connect 0x7fcb480392f0 2023-10-30T12:23:12.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.623+0000 7fcb5b7fe640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb480392f0 0x7fcb4803b7b0 secure :-1 s=READY pgs=4355 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c09cbb0 tx=0x7fcb4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.627+0000 7fcb58ff9640 1 -- 172.21.15.50:0/2459068001 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcb500308e0 con 0x7fcb5c0659d0 2023-10-30T12:23:12.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.627+0000 7fcb58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.627+0000 7fcb58ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:12.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.800+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb5c131690 con 0x7fcb480392f0 2023-10-30T12:23:12.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.805+0000 7fcb58ff9640 1 -- 172.21.15.50:0/2459068001 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133807 (secure 0 0 0) 0x7fcb5c131690 con 0x7fcb480392f0 2023-10-30T12:23:12.807 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:23:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:23:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 10 librados: watch_flush enter 2023-10-30T12:23:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 10 librados: watch_flush exit 2023-10-30T12:23:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 20 client.264366.objecter shutdown clearing up homeless session... 2023-10-30T12:23:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 10 client.264366.objecter successfully canceled tick 2023-10-30T12:23:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb480392f0 msgr2=0x7fcb4803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb480392f0 0x7fcb4803b7b0 secure :-1 s=READY pgs=4355 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c09cbb0 tx=0x7fcb4c008040 comp rx=0 tx=0).stop 2023-10-30T12:23:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 10 monclient: shutdown 2023-10-30T12:23:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 msgr2=0x7fcb5c131cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c131cb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009890 tx=0x7fcb500046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 shutdown_connections 2023-10-30T12:23:12.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fcb480392f0 0x7fcb4803b7b0 unknown :-1 s=CLOSED pgs=4355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb5c071850 0x7fcb5c12a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb5c071410 0x7fcb5c129b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 --2- 172.21.15.50:0/2459068001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb5c0659d0 0x7fcb5c131cb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 >> 172.21.15.50:0/2459068001 conn(0x7fcb5c0a47f0 msgr2=0x7fcb5c12b510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 shutdown_connections 2023-10-30T12:23:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 -- 172.21.15.50:0/2459068001 wait complete. 2023-10-30T12:23:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.807+0000 7fcb62421640 1 librados: shutdown 2023-10-30T12:23:12.819 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1352,"stamp":"2023-10-30T12:23:12.736157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":429848,"kb_used_data":10840,"kb_used_omap":9,"kb_used_meta":418806,"kb_avail":565801192,"statfs":{"total":579820584960,"available":579380420608,"internally_reserved":0,"allocated":11100160,"data_stored":3863898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":428858023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002423"},"pg_stats":[{"pgid":"15.13","version":"0'0","reported_seq":650,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711339+0000","last_change":"2023-10-30T12:01:59.232802+0000","last_active":"2023-10-30T12:22:45.711339+0000","last_peered":"2023-10-30T12:22:45.711339+0000","last_clean":"2023-10-30T12:22:45.711339+0000","last_became_active":"2023-10-30T12:01:59.223072+0000","last_became_peered":"2023-10-30T12:01:59.223072+0000","last_unstale":"2023-10-30T12:22:45.711339+0000","last_undegraded":"2023-10-30T12:22:45.711339+0000","last_fullsized":"2023-10-30T12:22:45.711339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:39.226354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":643,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711074+0000","last_change":"2023-10-30T12:01:59.344564+0000","last_active":"2023-10-30T12:22:45.711074+0000","last_peered":"2023-10-30T12:22:45.711074+0000","last_clean":"2023-10-30T12:22:45.711074+0000","last_became_active":"2023-10-30T12:01:59.344017+0000","last_became_peered":"2023-10-30T12:01:59.344017+0000","last_unstale":"2023-10-30T12:22:45.711074+0000","last_undegraded":"2023-10-30T12:22:45.711074+0000","last_fullsized":"2023-10-30T12:22:45.711074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.878167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610023+0000","last_change":"2023-10-30T12:01:59.312117+0000","last_active":"2023-10-30T12:22:44.610023+0000","last_peered":"2023-10-30T12:22:44.610023+0000","last_clean":"2023-10-30T12:22:44.610023+0000","last_became_active":"2023-10-30T12:01:59.308994+0000","last_became_peered":"2023-10-30T12:01:59.308994+0000","last_unstale":"2023-10-30T12:22:44.610023+0000","last_undegraded":"2023-10-30T12:22:44.610023+0000","last_fullsized":"2023-10-30T12:22:44.610023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:12.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710486+0000","last_change":"2023-10-30T12:01:59.280468+0000","last_active":"2023-10-30T12:22:45.710486+0000","last_peered":"2023-10-30T12:22:45.710486+0000","last_clean":"2023-10-30T12:22:45.710486+0000","last_became_active":"2023-10-30T12:01:59.277217+0000","last_became_peered":"2023-10-30T12:01:59.277217+0000","last_unstale":"2023-10-30T12:22:45.710486+0000","last_undegraded":"2023-10-30T12:22:45.710486+0000","last_fullsized":"2023-10-30T12:22:45.710486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:39:50.184703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110225+0000","last_change":"2023-10-30T12:01:59.246329+0000","last_active":"2023-10-30T12:22:44.110225+0000","last_peered":"2023-10-30T12:22:44.110225+0000","last_clean":"2023-10-30T12:22:44.110225+0000","last_became_active":"2023-10-30T12:01:59.246101+0000","last_became_peered":"2023-10-30T12:01:59.246101+0000","last_unstale":"2023-10-30T12:22:44.110225+0000","last_undegraded":"2023-10-30T12:22:44.110225+0000","last_fullsized":"2023-10-30T12:22:44.110225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:17.822160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987154+0000","last_change":"2023-10-30T12:01:59.270798+0000","last_active":"2023-10-30T12:22:43.987154+0000","last_peered":"2023-10-30T12:22:43.987154+0000","last_clean":"2023-10-30T12:22:43.987154+0000","last_became_active":"2023-10-30T12:01:59.269001+0000","last_became_peered":"2023-10-30T12:01:59.269001+0000","last_unstale":"2023-10-30T12:22:43.987154+0000","last_undegraded":"2023-10-30T12:22:43.987154+0000","last_fullsized":"2023-10-30T12:22:43.987154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:36:36.641287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110448+0000","last_change":"2023-10-30T12:01:59.315912+0000","last_active":"2023-10-30T12:22:44.110448+0000","last_peered":"2023-10-30T12:22:44.110448+0000","last_clean":"2023-10-30T12:22:44.110448+0000","last_became_active":"2023-10-30T12:01:59.314812+0000","last_became_peered":"2023-10-30T12:01:59.314812+0000","last_unstale":"2023-10-30T12:22:44.110448+0000","last_undegraded":"2023-10-30T12:22:44.110448+0000","last_fullsized":"2023-10-30T12:22:44.110448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:11:29.969594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":664,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602031+0000","last_change":"2023-10-30T12:01:59.339356+0000","last_active":"2023-10-30T12:22:44.602031+0000","last_peered":"2023-10-30T12:22:44.602031+0000","last_clean":"2023-10-30T12:22:44.602031+0000","last_became_active":"2023-10-30T12:01:59.338820+0000","last_became_peered":"2023-10-30T12:01:59.338820+0000","last_unstale":"2023-10-30T12:22:44.602031+0000","last_undegraded":"2023-10-30T12:22:44.602031+0000","last_fullsized":"2023-10-30T12:22:44.602031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:18.746605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":663,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599775+0000","last_change":"2023-10-30T12:01:59.305090+0000","last_active":"2023-10-30T12:22:43.599775+0000","last_peered":"2023-10-30T12:22:43.599775+0000","last_clean":"2023-10-30T12:22:43.599775+0000","last_became_active":"2023-10-30T12:01:59.296246+0000","last_became_peered":"2023-10-30T12:01:59.296246+0000","last_unstale":"2023-10-30T12:22:43.599775+0000","last_undegraded":"2023-10-30T12:22:43.599775+0000","last_fullsized":"2023-10-30T12:22:43.599775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:40.407666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110494+0000","last_change":"2023-10-30T12:01:59.297023+0000","last_active":"2023-10-30T12:22:44.110494+0000","last_peered":"2023-10-30T12:22:44.110494+0000","last_clean":"2023-10-30T12:22:44.110494+0000","last_became_active":"2023-10-30T12:01:59.296511+0000","last_became_peered":"2023-10-30T12:01:59.296511+0000","last_unstale":"2023-10-30T12:22:44.110494+0000","last_undegraded":"2023-10-30T12:22:44.110494+0000","last_fullsized":"2023-10-30T12:22:44.110494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:45.585836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109889+0000","last_change":"2023-10-30T12:01:59.286076+0000","last_active":"2023-10-30T12:22:44.109889+0000","last_peered":"2023-10-30T12:22:44.109889+0000","last_clean":"2023-10-30T12:22:44.109889+0000","last_became_active":"2023-10-30T12:01:59.282069+0000","last_became_peered":"2023-10-30T12:01:59.282069+0000","last_unstale":"2023-10-30T12:22:44.109889+0000","last_undegraded":"2023-10-30T12:22:44.109889+0000","last_fullsized":"2023-10-30T12:22:44.109889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:33.906969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.986878+0000","last_change":"2023-10-30T12:01:59.267278+0000","last_active":"2023-10-30T12:22:43.986878+0000","last_peered":"2023-10-30T12:22:43.986878+0000","last_clean":"2023-10-30T12:22:43.986878+0000","last_became_active":"2023-10-30T12:01:59.266725+0000","last_became_peered":"2023-10-30T12:01:59.266725+0000","last_unstale":"2023-10-30T12:22:43.986878+0000","last_undegraded":"2023-10-30T12:22:43.986878+0000","last_fullsized":"2023-10-30T12:22:43.986878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:13.026323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599780+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:22:43.599780+0000","last_peered":"2023-10-30T12:22:43.599780+0000","last_clean":"2023-10-30T12:22:43.599780+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:22:43.599780+0000","last_undegraded":"2023-10-30T12:22:43.599780+0000","last_fullsized":"2023-10-30T12:22:43.599780+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609116+0000","last_change":"2023-10-30T12:01:59.274700+0000","last_active":"2023-10-30T12:22:43.609116+0000","last_peered":"2023-10-30T12:22:43.609116+0000","last_clean":"2023-10-30T12:22:43.609116+0000","last_became_active":"2023-10-30T12:01:59.274444+0000","last_became_peered":"2023-10-30T12:01:59.274444+0000","last_unstale":"2023-10-30T12:22:43.609116+0000","last_undegraded":"2023-10-30T12:22:43.609116+0000","last_fullsized":"2023-10-30T12:22:43.609116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:14.789781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110011+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:22:44.110011+0000","last_peered":"2023-10-30T12:22:44.110011+0000","last_clean":"2023-10-30T12:22:44.110011+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:22:44.110011+0000","last_undegraded":"2023-10-30T12:22:44.110011+0000","last_fullsized":"2023-10-30T12:22:44.110011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609261+0000","last_change":"2023-10-30T12:01:59.272386+0000","last_active":"2023-10-30T12:22:43.609261+0000","last_peered":"2023-10-30T12:22:43.609261+0000","last_clean":"2023-10-30T12:22:43.609261+0000","last_became_active":"2023-10-30T12:01:59.272125+0000","last_became_peered":"2023-10-30T12:01:59.272125+0000","last_unstale":"2023-10-30T12:22:43.609261+0000","last_undegraded":"2023-10-30T12:22:43.609261+0000","last_fullsized":"2023-10-30T12:22:43.609261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:15.708175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599879+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:22:43.599879+0000","last_peered":"2023-10-30T12:22:43.599879+0000","last_clean":"2023-10-30T12:22:43.599879+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:22:43.599879+0000","last_undegraded":"2023-10-30T12:22:43.599879+0000","last_fullsized":"2023-10-30T12:22:43.599879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609202+0000","last_change":"2023-10-30T12:01:59.269863+0000","last_active":"2023-10-30T12:22:43.609202+0000","last_peered":"2023-10-30T12:22:43.609202+0000","last_clean":"2023-10-30T12:22:43.609202+0000","last_became_active":"2023-10-30T12:01:59.269127+0000","last_became_peered":"2023-10-30T12:01:59.269127+0000","last_unstale":"2023-10-30T12:22:43.609202+0000","last_undegraded":"2023-10-30T12:22:43.609202+0000","last_fullsized":"2023-10-30T12:22:43.609202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:31:20.350152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110004+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:22:44.110004+0000","last_peered":"2023-10-30T12:22:44.110004+0000","last_clean":"2023-10-30T12:22:44.110004+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:22:44.110004+0000","last_undegraded":"2023-10-30T12:22:44.110004+0000","last_fullsized":"2023-10-30T12:22:44.110004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110202+0000","last_change":"2023-10-30T12:01:59.306784+0000","last_active":"2023-10-30T12:22:44.110202+0000","last_peered":"2023-10-30T12:22:44.110202+0000","last_clean":"2023-10-30T12:22:44.110202+0000","last_became_active":"2023-10-30T12:01:59.306293+0000","last_became_peered":"2023-10-30T12:01:59.306293+0000","last_unstale":"2023-10-30T12:22:44.110202+0000","last_undegraded":"2023-10-30T12:22:44.110202+0000","last_fullsized":"2023-10-30T12:22:44.110202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:26:17.267254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110433+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:22:44.110433+0000","last_peered":"2023-10-30T12:22:44.110433+0000","last_clean":"2023-10-30T12:22:44.110433+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:22:44.110433+0000","last_undegraded":"2023-10-30T12:22:44.110433+0000","last_fullsized":"2023-10-30T12:22:44.110433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":664,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599745+0000","last_change":"2023-10-30T12:01:59.314586+0000","last_active":"2023-10-30T12:22:43.599745+0000","last_peered":"2023-10-30T12:22:43.599745+0000","last_clean":"2023-10-30T12:22:43.599745+0000","last_became_active":"2023-10-30T12:01:59.313414+0000","last_became_peered":"2023-10-30T12:01:59.313414+0000","last_unstale":"2023-10-30T12:22:43.599745+0000","last_undegraded":"2023-10-30T12:22:43.599745+0000","last_fullsized":"2023-10-30T12:22:43.599745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:05.821606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609150+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:22:43.609150+0000","last_peered":"2023-10-30T12:22:43.609150+0000","last_clean":"2023-10-30T12:22:43.609150+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:22:43.609150+0000","last_undegraded":"2023-10-30T12:22:43.609150+0000","last_fullsized":"2023-10-30T12:22:43.609150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110338+0000","last_change":"2023-10-30T12:01:59.289541+0000","last_active":"2023-10-30T12:22:44.110338+0000","last_peered":"2023-10-30T12:22:44.110338+0000","last_clean":"2023-10-30T12:22:44.110338+0000","last_became_active":"2023-10-30T12:01:59.289175+0000","last_became_peered":"2023-10-30T12:01:59.289175+0000","last_unstale":"2023-10-30T12:22:44.110338+0000","last_undegraded":"2023-10-30T12:22:44.110338+0000","last_fullsized":"2023-10-30T12:22:44.110338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:01.620219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":668,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601903+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:22:44.601903+0000","last_peered":"2023-10-30T12:22:44.601903+0000","last_clean":"2023-10-30T12:22:44.601903+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:22:44.601903+0000","last_undegraded":"2023-10-30T12:22:44.601903+0000","last_fullsized":"2023-10-30T12:22:44.601903+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110134+0000","last_change":"2023-10-30T12:01:59.307590+0000","last_active":"2023-10-30T12:22:44.110134+0000","last_peered":"2023-10-30T12:22:44.110134+0000","last_clean":"2023-10-30T12:22:44.110134+0000","last_became_active":"2023-10-30T12:01:59.302155+0000","last_became_peered":"2023-10-30T12:01:59.302155+0000","last_unstale":"2023-10-30T12:22:44.110134+0000","last_undegraded":"2023-10-30T12:22:44.110134+0000","last_fullsized":"2023-10-30T12:22:44.110134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:48.321197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110344+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:22:44.110344+0000","last_peered":"2023-10-30T12:22:44.110344+0000","last_clean":"2023-10-30T12:22:44.110344+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:22:44.110344+0000","last_undegraded":"2023-10-30T12:22:44.110344+0000","last_fullsized":"2023-10-30T12:22:44.110344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":820,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602127+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:22:44.602127+0000","last_peered":"2023-10-30T12:22:44.602127+0000","last_clean":"2023-10-30T12:22:44.602127+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:22:44.602127+0000","last_undegraded":"2023-10-30T12:22:44.602127+0000","last_fullsized":"2023-10-30T12:22:44.602127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609190+0000","last_change":"2023-10-30T12:01:59.256285+0000","last_active":"2023-10-30T12:22:43.609190+0000","last_peered":"2023-10-30T12:22:43.609190+0000","last_clean":"2023-10-30T12:22:43.609190+0000","last_became_active":"2023-10-30T12:01:59.253931+0000","last_became_peered":"2023-10-30T12:01:59.253931+0000","last_unstale":"2023-10-30T12:22:43.609190+0000","last_undegraded":"2023-10-30T12:22:43.609190+0000","last_fullsized":"2023-10-30T12:22:43.609190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:21.312914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711364+0000","last_change":"2023-10-30T12:01:59.296719+0000","last_active":"2023-10-30T12:22:45.711364+0000","last_peered":"2023-10-30T12:22:45.711364+0000","last_clean":"2023-10-30T12:22:45.711364+0000","last_became_active":"2023-10-30T12:01:59.277183+0000","last_became_peered":"2023-10-30T12:01:59.277183+0000","last_unstale":"2023-10-30T12:22:45.711364+0000","last_undegraded":"2023-10-30T12:22:45.711364+0000","last_fullsized":"2023-10-30T12:22:45.711364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:49.956478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":665,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602043+0000","last_change":"2023-10-30T12:01:59.339853+0000","last_active":"2023-10-30T12:22:44.602043+0000","last_peered":"2023-10-30T12:22:44.602043+0000","last_clean":"2023-10-30T12:22:44.602043+0000","last_became_active":"2023-10-30T12:01:59.339614+0000","last_became_peered":"2023-10-30T12:01:59.339614+0000","last_unstale":"2023-10-30T12:22:44.602043+0000","last_undegraded":"2023-10-30T12:22:44.602043+0000","last_fullsized":"2023-10-30T12:22:44.602043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:21.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987106+0000","last_change":"2023-10-30T12:01:59.303724+0000","last_active":"2023-10-30T12:22:43.987106+0000","last_peered":"2023-10-30T12:22:43.987106+0000","last_clean":"2023-10-30T12:22:43.987106+0000","last_became_active":"2023-10-30T12:01:59.303216+0000","last_became_peered":"2023-10-30T12:01:59.303216+0000","last_unstale":"2023-10-30T12:22:43.987106+0000","last_undegraded":"2023-10-30T12:22:43.987106+0000","last_fullsized":"2023-10-30T12:22:43.987106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:35.112009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109953+0000","last_change":"2023-10-30T12:01:59.209818+0000","last_active":"2023-10-30T12:22:44.109953+0000","last_peered":"2023-10-30T12:22:44.109953+0000","last_clean":"2023-10-30T12:22:44.109953+0000","last_became_active":"2023-10-30T12:01:59.209551+0000","last_became_peered":"2023-10-30T12:01:59.209551+0000","last_unstale":"2023-10-30T12:22:44.109953+0000","last_undegraded":"2023-10-30T12:22:44.109953+0000","last_fullsized":"2023-10-30T12:22:44.109953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:49.423499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":653,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609027+0000","last_change":"2023-10-30T12:01:59.312819+0000","last_active":"2023-10-30T12:22:43.609027+0000","last_peered":"2023-10-30T12:22:43.609027+0000","last_clean":"2023-10-30T12:22:43.609027+0000","last_became_active":"2023-10-30T12:01:59.310341+0000","last_became_peered":"2023-10-30T12:01:59.310341+0000","last_unstale":"2023-10-30T12:22:43.609027+0000","last_undegraded":"2023-10-30T12:22:43.609027+0000","last_fullsized":"2023-10-30T12:22:43.609027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:52.592573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"18'7","reported_seq":678,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601944+0000","last_change":"2023-10-30T12:01:59.261094+0000","last_active":"2023-10-30T12:22:44.601944+0000","last_peered":"2023-10-30T12:22:44.601944+0000","last_clean":"2023-10-30T12:22:44.601944+0000","last_became_active":"2023-10-30T12:01:59.260849+0000","last_became_peered":"2023-10-30T12:01:59.260849+0000","last_unstale":"2023-10-30T12:22:44.601944+0000","last_undegraded":"2023-10-30T12:22:44.601944+0000","last_fullsized":"2023-10-30T12:22:44.601944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:36.395779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610042+0000","last_change":"2023-10-30T12:08:57.463451+0000","last_active":"2023-10-30T12:22:44.610042+0000","last_peered":"2023-10-30T12:22:44.610042+0000","last_clean":"2023-10-30T12:22:44.610042+0000","last_became_active":"2023-10-30T12:08:57.463102+0000","last_became_peered":"2023-10-30T12:08:57.463102+0000","last_unstale":"2023-10-30T12:22:44.610042+0000","last_undegraded":"2023-10-30T12:22:44.610042+0000","last_fullsized":"2023-10-30T12:22:44.610042+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:48.599858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710660+0000","last_change":"2023-10-30T12:01:59.247352+0000","last_active":"2023-10-30T12:22:45.710660+0000","last_peered":"2023-10-30T12:22:45.710660+0000","last_clean":"2023-10-30T12:22:45.710660+0000","last_became_active":"2023-10-30T12:01:59.246868+0000","last_became_peered":"2023-10-30T12:01:59.246868+0000","last_unstale":"2023-10-30T12:22:45.710660+0000","last_undegraded":"2023-10-30T12:22:45.710660+0000","last_fullsized":"2023-10-30T12:22:45.710660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:50.392311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":649,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109897+0000","last_change":"2023-10-30T12:01:59.276982+0000","last_active":"2023-10-30T12:22:44.109897+0000","last_peered":"2023-10-30T12:22:44.109897+0000","last_clean":"2023-10-30T12:22:44.109897+0000","last_became_active":"2023-10-30T12:01:59.239117+0000","last_became_peered":"2023-10-30T12:01:59.239117+0000","last_unstale":"2023-10-30T12:22:44.109897+0000","last_undegraded":"2023-10-30T12:22:44.109897+0000","last_fullsized":"2023-10-30T12:22:44.109897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.157663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110047+0000","last_change":"2023-10-30T12:01:59.297769+0000","last_active":"2023-10-30T12:22:44.110047+0000","last_peered":"2023-10-30T12:22:44.110047+0000","last_clean":"2023-10-30T12:22:44.110047+0000","last_became_active":"2023-10-30T12:01:59.293055+0000","last_became_peered":"2023-10-30T12:01:59.293055+0000","last_unstale":"2023-10-30T12:22:44.110047+0000","last_undegraded":"2023-10-30T12:22:44.110047+0000","last_fullsized":"2023-10-30T12:22:44.110047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:53.053266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609249+0000","last_change":"2023-10-30T12:01:59.310309+0000","last_active":"2023-10-30T12:22:43.609249+0000","last_peered":"2023-10-30T12:22:43.609249+0000","last_clean":"2023-10-30T12:22:43.609249+0000","last_became_active":"2023-10-30T12:01:59.307440+0000","last_became_peered":"2023-10-30T12:01:59.307440+0000","last_unstale":"2023-10-30T12:22:43.609249+0000","last_undegraded":"2023-10-30T12:22:43.609249+0000","last_fullsized":"2023-10-30T12:22:43.609249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:39.454237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711399+0000","last_change":"2023-10-30T12:01:59.249544+0000","last_active":"2023-10-30T12:22:45.711399+0000","last_peered":"2023-10-30T12:22:45.711399+0000","last_clean":"2023-10-30T12:22:45.711399+0000","last_became_active":"2023-10-30T12:01:59.239219+0000","last_became_peered":"2023-10-30T12:01:59.239219+0000","last_unstale":"2023-10-30T12:22:45.711399+0000","last_undegraded":"2023-10-30T12:22:45.711399+0000","last_fullsized":"2023-10-30T12:22:45.711399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:17.868254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640526,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77256,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294584,"statfs":{"total":96636764160,"available":96557654016,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.82699999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.65000000000000002,"5min":0.81599999999999995,"15min":1.056},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65400000000000003,"5min":0.83399999999999996,"15min":2.173},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.28299999999999997},"max":{"1min":0.64800000000000002,"5min":0.72799999999999998,"15min":0.78100000000000003},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.67400000000000004,"5min":0.74299999999999999,"15min":0.79900000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.41499999999999998,"15min":0.32900000000000001},"max":{"1min":0.65000000000000002,"5min":0.79900000000000004,"15min":0.84699999999999998},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.36899999999999999,"15min":0.373},"min":{"1min":0.25,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":0.499,"5min":0.58799999999999997,"15min":0.70399999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.223},"max":{"1min":0.54500000000000004,"5min":0.69699999999999995,"15min":0.95199999999999996},"last":0.377}]},{"osd":4,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.31,"5min":0.252,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.63700000000000001,"15min":0.63800000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.435,"15min":0.45300000000000001},"min":{"1min":0.28100000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.60299999999999998,"5min":0.68400000000000005,"15min":0.72099999999999997},"last":0.54500000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81416,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94290424,"statfs":{"total":96636764160,"available":96553394176,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53900000000000003,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.51400000000000001},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.84899999999999998},"last":0.60699999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.79000000000000004,"15min":0.94099999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.36299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.67700000000000005,"5min":0.749,"15min":0.79800000000000004},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27300000000000002},"max":{"1min":0.627,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.246},"max":{"1min":0.59399999999999997,"5min":0.627,"15min":0.63600000000000001},"last":0.376},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.25,"5min":0.249,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.67000000000000004,"15min":0.77100000000000002},"last":0.38700000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.435,"15min":0.40999999999999998},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.25600000000000001},"max":{"1min":0.621,"5min":0.64000000000000001,"15min":0.69999999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.67400000000000004,"15min":0.89700000000000002},"last":0.54700000000000004}]}]},{"osd":3,"up_from":11,"seq":47244640527,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69444,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302396,"statfs":{"total":96636764160,"available":96565653504,"internally_reserved":0,"allocated":2207744,"data_stored":996605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.503,"15min":0.501},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72799999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.61899999999999999,"5min":0.71399999999999997,"15min":1.2490000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.68999999999999995,"5min":0.81100000000000005,"15min":1.2689999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69999999999999996,"5min":0.82599999999999996,"15min":1.337},"last":0.67900000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":0.88900000000000001,"15min":1.302},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":0.746,"15min":0.76900000000000002},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.37,"15min":0.38},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":0.50700000000000001,"5min":0.61199999999999999,"15min":1.371},"last":0.495},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.30599999999999999},"max":{"1min":0.56599999999999995,"5min":0.66300000000000003,"15min":1.444},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39600000000000002,"15min":0.38700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":0.502,"5min":0.64200000000000002,"15min":0.876},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.375,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53500000000000003,"5min":0.84099999999999997,"15min":1.3540000000000001},"last":0.627}]}]},{"osd":2,"up_from":11,"seq":47244640526,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64764,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94307076,"statfs":{"total":96636764160,"available":96570445824,"internally_reserved":0,"allocated":1675264,"data_stored":467677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33500000000000002},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44800000000000001,"5min":0.50700000000000001,"15min":2.2730000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.38400000000000001},"min":{"1min":0.27500000000000002,"5min":0.24199999999999999,"15min":0.22800000000000001},"max":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":2.141},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33700000000000002,"15min":0.34200000000000003},"min":{"1min":0.24399999999999999,"5min":0.21299999999999999,"15min":0.20799999999999999},"max":{"1min":0.46999999999999997,"5min":0.502,"15min":1.087},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.24399999999999999},"max":{"1min":0.495,"5min":0.63900000000000001,"15min":3.8260000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.42199999999999999,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":0.749,"5min":0.75700000000000001,"15min":1.143},"last":0.5},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.42899999999999999,"5min":0.33600000000000002,"15min":0.314},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":3.4729999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.34399999999999997,"15min":0.27300000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.3240000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.45600000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":1.8340000000000001},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.32200000000000001},"max":{"1min":0.81699999999999995,"5min":0.81999999999999995,"15min":1.893},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.48499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75900000000000001,"5min":0.81599999999999995,"15min":2.2269999999999999},"last":0.65400000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640528,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59208,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312632,"statfs":{"total":96636764160,"available":96576135168,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.34499999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":0.46899999999999997,"5min":0.745,"15min":0.745},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.374},"min":{"1min":0.253,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.38200000000000001},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.223},"max":{"1min":0.54000000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.255},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.51800000000000002,"5min":0.621,"15min":0.93200000000000005},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.443,"5min":0.375,"15min":0.28499999999999998},"max":{"1min":0.69999999999999996,"5min":0.86599999999999999,"15min":0.879},"last":0.435},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.79900000000000004,"5min":0.85399999999999998,"15min":0.88900000000000001},"last":0.501}]},{"osd":4,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.46400000000000002,"5min":0.41699999999999998,"15min":0.29899999999999999},"max":{"1min":0.86799999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.36799999999999999},"max":{"1min":0.71299999999999997,"5min":0.94799999999999995,"15min":0.95599999999999996},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.58899999999999997},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.39300000000000002},"max":{"1min":0.84499999999999997,"5min":1,"15min":1},"last":0.46600000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640528,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77760,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294080,"statfs":{"total":96636764160,"available":96557137920,"internally_reserved":0,"allocated":2203648,"data_stored":996600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.35199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23899999999999999},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.21299999999999999},"max":{"1min":1.802,"5min":1.802,"15min":3.8519999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26700000000000002,"15min":0.20100000000000001},"max":{"1min":0.58699999999999997,"5min":0.623,"15min":2.0590000000000002},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.27800000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":1.5489999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":1.78},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.29399999999999998},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59099999999999997,"15min":0.57899999999999996},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":1.6020000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:23:12.822 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:23:12.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:23:12.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.925+0000 7f12d3490640 10 monclient: get_monmap_and_config 2023-10-30T12:23:12.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.925+0000 7f12d3490640 10 monclient: build_initial_monmap 2023-10-30T12:23:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.925+0000 7f12d3490640 10 monclient: monmap: 2023-10-30T12:23:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:12.926863+0000 2023-10-30T12:23:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:12.926863+0000 2023-10-30T12:23:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:12.931 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.926+0000 7f12d3490640 1 Processor -- start 2023-10-30T12:23:12.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.926+0000 7f12d3490640 1 -- start start 2023-10-30T12:23:12.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.926+0000 7f12d3490640 10 monclient: init 2023-10-30T12:23:12.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.926+0000 7f12d3490640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:12.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.926+0000 7f12d3490640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-10-30T12:23:12.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.926+0000 7f12d3490640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f12cc098520 0x7f12cc09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 10 monclient(hunting): picked mon.noname-c con 0x7f12cc098520 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:12.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12cc098900 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:12.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12cc1379f0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:12.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc09ad30 con 0x7f12cc1379f0 2023-10-30T12:23:12.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12cc09c460 con 0x7f12cc098900 2023-10-30T12:23:12.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12cc09b260 con 0x7f12cc098520 2023-10-30T12:23:12.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:12.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d3490640 10 monclient(hunting): authenticate will time out at 2238.110068s 2023-10-30T12:23:12.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d1a2d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f12cc1379f0 0x7f12cc139de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49598/0 (socket says 172.21.15.50:49598) 2023-10-30T12:23:12.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d1a2d640 1 -- 172.21.15.50:0/1082520905 learned_addr learned my addr 172.21.15.50:0/1082520905 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:12.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 1 --2- 172.21.15.50:0/1082520905 >> v2:172.21.15.118:3300/0 conn(0x7f12cc098520 0x7f12cc09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): get_auth_request con 0x7f12cc098520 auth_method 0 2023-10-30T12:23:12.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216374950 0 0) 0x7f12cc09ad30 con 0x7f12cc1379f0 2023-10-30T12:23:12.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): _negotiate 2023-10-30T12:23:12.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): my global_id is 264369 2023-10-30T12:23:12.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b8003520 con 0x7f12cc1379f0 2023-10-30T12:23:12.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711282456 0 0) 0x7f12cc09c460 con 0x7f12cc098900 2023-10-30T12:23:12.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): _negotiate 2023-10-30T12:23:12.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient(hunting): my global_id is 274219 2023-10-30T12:23:12.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12cc09ad30 con 0x7f12cc098900 2023-10-30T12:23:12.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224584775 0 0) 0x7f12b8003520 con 0x7f12cc1379f0 2023-10-30T12:23:12.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient(hunting): handle_auth_done global_id 274216 payload 274 2023-10-30T12:23:12.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 1 -- 172.21.15.50:0/1082520905 >> v1:172.21.15.118:6789/0 conn(0x7f12cc098900 legacy=0x7f12cc098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 1 -- 172.21.15.50:0/1082520905 >> v1:172.21.15.50:6789/0 conn(0x7f12cc1379f0 legacy=0x7f12cc139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:12.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 1 monclient: found mon.noname-c 2023-10-30T12:23:12.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:12.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:23:12.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12d122c640 1 -- 172.21.15.50:0/1082520905 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc072120 con 0x7f12cc098520 2023-10-30T12:23:12.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.927+0000 7f12c3fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:12.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12d122c640 1 --2- 172.21.15.50:0/1082520905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc098520 0x7f12cc09bf20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009790 tx=0x7f12c802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8748783730793b35 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c40019f0 con 0x7f12cc098520 2023-10-30T12:23:12.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:12.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: dump: 2023-10-30T12:23:12.954 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.954 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.954 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.955 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.955 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.955 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.955 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.956 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.956 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.956 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: _finish_auth 0 2023-10-30T12:23:12.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:12.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:12.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c802e050 con 0x7f12cc098520 2023-10-30T12:23:12.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12d3490640 5 monclient: authenticate success, global_id 274216 2023-10-30T12:23:12.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12d3490640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:12.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:12.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12d3490640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:12.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12d3490640 4 set_mon_vals no callback set 2023-10-30T12:23:12.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 1 -- 172.21.15.50:0/1082520905 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8030730 con 0x7f12cc098520 2023-10-30T12:23:12.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:23:12.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.928+0000 7f12c3fff640 10 monclient: dump: 2023-10-30T12:23:12.960 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.960 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.960 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.961 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.961 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.961 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.962 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.962 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.962 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.962 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 10 monclient: shutdown 2023-10-30T12:23:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 1 -- 172.21.15.50:0/1082520905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc098520 msgr2=0x7f12cc09bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 1 --2- 172.21.15.50:0/1082520905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc098520 0x7f12cc09bf20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009790 tx=0x7f12c802b710 comp rx=0 tx=0).stop 2023-10-30T12:23:12.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 1 -- 172.21.15.50:0/1082520905 shutdown_connections 2023-10-30T12:23:12.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 1 --2- 172.21.15.50:0/1082520905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc098520 0x7f12cc09bf20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 1 -- 172.21.15.50:0/1082520905 >> 172.21.15.50:0/1082520905 conn(0x7f12cc0a05c0 msgr2=0x7f12cc09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 1 -- 172.21.15.50:0/1082520905 shutdown_connections 2023-10-30T12:23:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 1 -- 172.21.15.50:0/1082520905 wait complete. 2023-10-30T12:23:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 10 monclient: build_initial_monmap 2023-10-30T12:23:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.929+0000 7f12d3490640 10 monclient: monmap: 2023-10-30T12:23:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:12.968 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:12.968 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 librados: starting msgr at 2023-10-30T12:23:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 librados: starting objecter 2023-10-30T12:23:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 Processor -- start 2023-10-30T12:23:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 -- start start 2023-10-30T12:23:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 librados: setting wanted keys 2023-10-30T12:23:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 librados: calling monclient init 2023-10-30T12:23:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient: init 2023-10-30T12:23:12.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:23:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12cc098520 0x7f12cc1ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): picked mon.noname-c con 0x7f12cc098520 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12cc098900 0x7f12cc096670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): picked mon.noname-a con 0x7f12cc098900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:12.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc1379f0 0x7f12cc096bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): picked mon.noname-b con 0x7f12cc1379f0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1378b0 con 0x7f12cc098900 2023-10-30T12:23:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc1cae40 con 0x7f12cc1379f0 2023-10-30T12:23:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc071e90 con 0x7f12cc098520 2023-10-30T12:23:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:12.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d3490640 10 monclient(hunting): authenticate will time out at 2238.113897s 2023-10-30T12:23:12.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d0a2b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12cc098900 0x7f12cc096670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.930+0000 7f12d122c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12cc098520 0x7f12cc1ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12cc098900 0x7f12cc096670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59788/0 (socket says 172.21.15.50:59788) 2023-10-30T12:23:12.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 -- 172.21.15.50:0/4163043058 learned_addr learned my addr 172.21.15.50:0/4163043058 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:12.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12cc098520 0x7f12cc1ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60644/0 (socket says 172.21.15.50:60644) 2023-10-30T12:23:12.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): get_auth_request con 0x7f12cc098900 auth_method 0 2023-10-30T12:23:12.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 10 monclient(hunting): get_auth_request con 0x7f12cc098520 auth_method 0 2023-10-30T12:23:12.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:12.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:12.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d122c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:12.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d1a2d640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc1379f0 0x7f12cc096bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:12.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d1a2d640 10 monclient(hunting): get_auth_request con 0x7f12cc1379f0 auth_method 0 2023-10-30T12:23:12.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d1a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:12.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d1a2d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:12.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d1a2d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:12.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient(hunting): handle_auth_done global_id 264372 payload 824 2023-10-30T12:23:12.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 -- 172.21.15.50:0/4163043058 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12cc098520 msgr2=0x7f12cc1ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12cc098520 0x7f12cc1ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 -- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc1379f0 msgr2=0x7f12cc096bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:12.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc1379f0 0x7f12cc096bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:12.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient: _finish_hunting 0 2023-10-30T12:23:12.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 monclient: found mon.noname-a 2023-10-30T12:23:12.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:12.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 -- 172.21.15.50:0/4163043058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80093a0 con 0x7f12cc098900 2023-10-30T12:23:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d0a2b640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12cc098900 0x7f12cc096670 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00c500 tx=0x7f12bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 client.?.objecter ms_handle_connect 0x7f12cc098900 2023-10-30T12:23:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:12.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 1 -- 172.21.15.50:0/4163043058 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc0052d0 con 0x7f12cc098900 2023-10-30T12:23:12.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:12.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: dump: 2023-10-30T12:23:12.985 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.986 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.986 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.986 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.987 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.987 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.987 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.987 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.988 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.988 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: _finish_auth 0 2023-10-30T12:23:12.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:23:12.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:12.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 5 monclient: authenticate success, global_id 264372 2023-10-30T12:23:12.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 1 -- 172.21.15.50:0/4163043058 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc005450 con 0x7f12cc098900 2023-10-30T12:23:12.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 10 monclient: _renew_subs 2023-10-30T12:23:12.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12cc094b80 con 0x7f12cc098900 2023-10-30T12:23:12.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:12.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 1 -- 172.21.15.50:0/4163043058 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc003830 con 0x7f12cc098900 2023-10-30T12:23:12.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:12.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:12.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c1ffb640 10 monclient: dump: 2023-10-30T12:23:12.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:12.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:12.993 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.993 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:12.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:12.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:12.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:12.994 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:12.994 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:12.994 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:12.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12c37fe640 4 set_mon_vals no callback set 2023-10-30T12:23:12.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 10 client.264372.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:12.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 10 monclient: _renew_subs 2023-10-30T12:23:12.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12cc093ec0 con 0x7f12cc098900 2023-10-30T12:23:12.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.931+0000 7f12d3490640 1 librados: init done 2023-10-30T12:23:12.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12c1ffb640 1 -- 172.21.15.50:0/4163043058 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f12bc003990 con 0x7f12cc098900 2023-10-30T12:23:12.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12d3490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12d3490640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:12.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12d3490640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:12.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12cc04e160 con 0x7f12cc098900 2023-10-30T12:23:12.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12c1ffb640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12b00392f0 0x7f12b003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:12.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12c1ffb640 1 -- 172.21.15.50:0/4163043058 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f12bc04a090 con 0x7f12cc098900 2023-10-30T12:23:12.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12c1ffb640 10 client.264372.objecter ms_dispatch 0x7f12cc000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:12.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12c1ffb640 3 client.264372.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:12.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.932+0000 7f12c1ffb640 3 client.264372.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:12.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.933+0000 7f12c1ffb640 20 client.264372.objecter dump_active .. 0 homeless 2023-10-30T12:23:13.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.933+0000 7f12d122c640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12b00392f0 0x7f12b003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:13.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.933+0000 7f12d122c640 10 monclient: get_auth_request con 0x7f12b00392f0 auth_method 0 2023-10-30T12:23:13.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.933+0000 7f12c1ffb640 10 client.264372.objecter ms_handle_connect 0x7f12b00392f0 2023-10-30T12:23:13.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.933+0000 7f12d122c640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12b00392f0 0x7f12b003b7b0 secure :-1 s=READY pgs=4356 cs=0 l=1 rev1=1 crypto rx=0x7f12c8034000 tx=0x7f12c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:13.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.937+0000 7f12c1ffb640 1 -- 172.21.15.50:0/4163043058 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12bc00e280 con 0x7f12cc098900 2023-10-30T12:23:13.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.937+0000 7f12c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:13.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:12.937+0000 7f12c1ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:13.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.112+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12cc06ae80 con 0x7f12b00392f0 2023-10-30T12:23:13.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.117+0000 7f12c1ffb640 1 -- 172.21.15.50:0/4163043058 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133807 (secure 0 0 0) 0x7f12c8040070 con 0x7f12b00392f0 2023-10-30T12:23:13.120 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:23:13.121 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:23:13.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 10 librados: watch_flush enter 2023-10-30T12:23:13.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 10 librados: watch_flush exit 2023-10-30T12:23:13.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 20 client.264372.objecter shutdown clearing up homeless session... 2023-10-30T12:23:13.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 10 client.264372.objecter successfully canceled tick 2023-10-30T12:23:13.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12b00392f0 msgr2=0x7f12b003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12b00392f0 0x7f12b003b7b0 secure :-1 s=READY pgs=4356 cs=0 l=1 rev1=1 crypto rx=0x7f12c8034000 tx=0x7f12c8008030 comp rx=0 tx=0).stop 2023-10-30T12:23:13.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 10 monclient: shutdown 2023-10-30T12:23:13.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:13.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12cc098900 msgr2=0x7f12cc096670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12cc098900 0x7f12cc096670 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00c500 tx=0x7f12bc00c9c0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 shutdown_connections 2023-10-30T12:23:13.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12b00392f0 0x7f12b003b7b0 unknown :-1 s=CLOSED pgs=4356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12cc1379f0 0x7f12cc096bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12cc098900 0x7f12cc096670 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 --2- 172.21.15.50:0/4163043058 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f12cc098520 0x7f12cc1ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 >> 172.21.15.50:0/4163043058 conn(0x7f12cc0a05c0 msgr2=0x7f12cc09dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:13.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 shutdown_connections 2023-10-30T12:23:13.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.118+0000 7f12d3490640 1 -- 172.21.15.50:0/4163043058 wait complete. 2023-10-30T12:23:13.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.119+0000 7f12d3490640 1 librados: shutdown 2023-10-30T12:23:13.130 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1352,"stamp":"2023-10-30T12:23:12.736157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":429848,"kb_used_data":10840,"kb_used_omap":9,"kb_used_meta":418806,"kb_avail":565801192,"statfs":{"total":579820584960,"available":579380420608,"internally_reserved":0,"allocated":11100160,"data_stored":3863898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":428858023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002423"},"pg_stats":[{"pgid":"15.13","version":"0'0","reported_seq":650,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711339+0000","last_change":"2023-10-30T12:01:59.232802+0000","last_active":"2023-10-30T12:22:45.711339+0000","last_peered":"2023-10-30T12:22:45.711339+0000","last_clean":"2023-10-30T12:22:45.711339+0000","last_became_active":"2023-10-30T12:01:59.223072+0000","last_became_peered":"2023-10-30T12:01:59.223072+0000","last_unstale":"2023-10-30T12:22:45.711339+0000","last_undegraded":"2023-10-30T12:22:45.711339+0000","last_fullsized":"2023-10-30T12:22:45.711339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:39.226354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":643,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711074+0000","last_change":"2023-10-30T12:01:59.344564+0000","last_active":"2023-10-30T12:22:45.711074+0000","last_peered":"2023-10-30T12:22:45.711074+0000","last_clean":"2023-10-30T12:22:45.711074+0000","last_became_active":"2023-10-30T12:01:59.344017+0000","last_became_peered":"2023-10-30T12:01:59.344017+0000","last_unstale":"2023-10-30T12:22:45.711074+0000","last_undegraded":"2023-10-30T12:22:45.711074+0000","last_fullsized":"2023-10-30T12:22:45.711074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.878167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610023+0000","last_change":"2023-10-30T12:01:59.312117+0000","last_active":"2023-10-30T12:22:44.610023+0000","last_peered":"2023-10-30T12:22:44.610023+0000","last_clean":"2023-10-30T12:22:44.610023+0000","last_became_active":"2023-10-30T12:01:59.308994+0000","last_became_peered":"2023-10-30T12:01:59.308994+0000","last_unstale":"2023-10-30T12:22:44.610023+0000","last_undegraded":"2023-10-30T12:22:44.610023+0000","last_fullsized":"2023-10-30T12:22:44.610023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:12.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710486+0000","last_change":"2023-10-30T12:01:59.280468+0000","last_active":"2023-10-30T12:22:45.710486+0000","last_peered":"2023-10-30T12:22:45.710486+0000","last_clean":"2023-10-30T12:22:45.710486+0000","last_became_active":"2023-10-30T12:01:59.277217+0000","last_became_peered":"2023-10-30T12:01:59.277217+0000","last_unstale":"2023-10-30T12:22:45.710486+0000","last_undegraded":"2023-10-30T12:22:45.710486+0000","last_fullsized":"2023-10-30T12:22:45.710486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:39:50.184703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110225+0000","last_change":"2023-10-30T12:01:59.246329+0000","last_active":"2023-10-30T12:22:44.110225+0000","last_peered":"2023-10-30T12:22:44.110225+0000","last_clean":"2023-10-30T12:22:44.110225+0000","last_became_active":"2023-10-30T12:01:59.246101+0000","last_became_peered":"2023-10-30T12:01:59.246101+0000","last_unstale":"2023-10-30T12:22:44.110225+0000","last_undegraded":"2023-10-30T12:22:44.110225+0000","last_fullsized":"2023-10-30T12:22:44.110225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:17.822160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987154+0000","last_change":"2023-10-30T12:01:59.270798+0000","last_active":"2023-10-30T12:22:43.987154+0000","last_peered":"2023-10-30T12:22:43.987154+0000","last_clean":"2023-10-30T12:22:43.987154+0000","last_became_active":"2023-10-30T12:01:59.269001+0000","last_became_peered":"2023-10-30T12:01:59.269001+0000","last_unstale":"2023-10-30T12:22:43.987154+0000","last_undegraded":"2023-10-30T12:22:43.987154+0000","last_fullsized":"2023-10-30T12:22:43.987154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:36:36.641287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110448+0000","last_change":"2023-10-30T12:01:59.315912+0000","last_active":"2023-10-30T12:22:44.110448+0000","last_peered":"2023-10-30T12:22:44.110448+0000","last_clean":"2023-10-30T12:22:44.110448+0000","last_became_active":"2023-10-30T12:01:59.314812+0000","last_became_peered":"2023-10-30T12:01:59.314812+0000","last_unstale":"2023-10-30T12:22:44.110448+0000","last_undegraded":"2023-10-30T12:22:44.110448+0000","last_fullsized":"2023-10-30T12:22:44.110448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:11:29.969594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":664,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602031+0000","last_change":"2023-10-30T12:01:59.339356+0000","last_active":"2023-10-30T12:22:44.602031+0000","last_peered":"2023-10-30T12:22:44.602031+0000","last_clean":"2023-10-30T12:22:44.602031+0000","last_became_active":"2023-10-30T12:01:59.338820+0000","last_became_peered":"2023-10-30T12:01:59.338820+0000","last_unstale":"2023-10-30T12:22:44.602031+0000","last_undegraded":"2023-10-30T12:22:44.602031+0000","last_fullsized":"2023-10-30T12:22:44.602031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:18.746605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":663,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599775+0000","last_change":"2023-10-30T12:01:59.305090+0000","last_active":"2023-10-30T12:22:43.599775+0000","last_peered":"2023-10-30T12:22:43.599775+0000","last_clean":"2023-10-30T12:22:43.599775+0000","last_became_active":"2023-10-30T12:01:59.296246+0000","last_became_peered":"2023-10-30T12:01:59.296246+0000","last_unstale":"2023-10-30T12:22:43.599775+0000","last_undegraded":"2023-10-30T12:22:43.599775+0000","last_fullsized":"2023-10-30T12:22:43.599775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:40.407666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110494+0000","last_change":"2023-10-30T12:01:59.297023+0000","last_active":"2023-10-30T12:22:44.110494+0000","last_peered":"2023-10-30T12:22:44.110494+0000","last_clean":"2023-10-30T12:22:44.110494+0000","last_became_active":"2023-10-30T12:01:59.296511+0000","last_became_peered":"2023-10-30T12:01:59.296511+0000","last_unstale":"2023-10-30T12:22:44.110494+0000","last_undegraded":"2023-10-30T12:22:44.110494+0000","last_fullsized":"2023-10-30T12:22:44.110494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:45.585836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109889+0000","last_change":"2023-10-30T12:01:59.286076+0000","last_active":"2023-10-30T12:22:44.109889+0000","last_peered":"2023-10-30T12:22:44.109889+0000","last_clean":"2023-10-30T12:22:44.109889+0000","last_became_active":"2023-10-30T12:01:59.282069+0000","last_became_peered":"2023-10-30T12:01:59.282069+0000","last_unstale":"2023-10-30T12:22:44.109889+0000","last_undegraded":"2023-10-30T12:22:44.109889+0000","last_fullsized":"2023-10-30T12:22:44.109889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:33.906969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.986878+0000","last_change":"2023-10-30T12:01:59.267278+0000","last_active":"2023-10-30T12:22:43.986878+0000","last_peered":"2023-10-30T12:22:43.986878+0000","last_clean":"2023-10-30T12:22:43.986878+0000","last_became_active":"2023-10-30T12:01:59.266725+0000","last_became_peered":"2023-10-30T12:01:59.266725+0000","last_unstale":"2023-10-30T12:22:43.986878+0000","last_undegraded":"2023-10-30T12:22:43.986878+0000","last_fullsized":"2023-10-30T12:22:43.986878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:13.026323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599780+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:22:43.599780+0000","last_peered":"2023-10-30T12:22:43.599780+0000","last_clean":"2023-10-30T12:22:43.599780+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:22:43.599780+0000","last_undegraded":"2023-10-30T12:22:43.599780+0000","last_fullsized":"2023-10-30T12:22:43.599780+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609116+0000","last_change":"2023-10-30T12:01:59.274700+0000","last_active":"2023-10-30T12:22:43.609116+0000","last_peered":"2023-10-30T12:22:43.609116+0000","last_clean":"2023-10-30T12:22:43.609116+0000","last_became_active":"2023-10-30T12:01:59.274444+0000","last_became_peered":"2023-10-30T12:01:59.274444+0000","last_unstale":"2023-10-30T12:22:43.609116+0000","last_undegraded":"2023-10-30T12:22:43.609116+0000","last_fullsized":"2023-10-30T12:22:43.609116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:14.789781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110011+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:22:44.110011+0000","last_peered":"2023-10-30T12:22:44.110011+0000","last_clean":"2023-10-30T12:22:44.110011+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:22:44.110011+0000","last_undegraded":"2023-10-30T12:22:44.110011+0000","last_fullsized":"2023-10-30T12:22:44.110011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609261+0000","last_change":"2023-10-30T12:01:59.272386+0000","last_active":"2023-10-30T12:22:43.609261+0000","last_peered":"2023-10-30T12:22:43.609261+0000","last_clean":"2023-10-30T12:22:43.609261+0000","last_became_active":"2023-10-30T12:01:59.272125+0000","last_became_peered":"2023-10-30T12:01:59.272125+0000","last_unstale":"2023-10-30T12:22:43.609261+0000","last_undegraded":"2023-10-30T12:22:43.609261+0000","last_fullsized":"2023-10-30T12:22:43.609261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:15.708175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599879+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:22:43.599879+0000","last_peered":"2023-10-30T12:22:43.599879+0000","last_clean":"2023-10-30T12:22:43.599879+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:22:43.599879+0000","last_undegraded":"2023-10-30T12:22:43.599879+0000","last_fullsized":"2023-10-30T12:22:43.599879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609202+0000","last_change":"2023-10-30T12:01:59.269863+0000","last_active":"2023-10-30T12:22:43.609202+0000","last_peered":"2023-10-30T12:22:43.609202+0000","last_clean":"2023-10-30T12:22:43.609202+0000","last_became_active":"2023-10-30T12:01:59.269127+0000","last_became_peered":"2023-10-30T12:01:59.269127+0000","last_unstale":"2023-10-30T12:22:43.609202+0000","last_undegraded":"2023-10-30T12:22:43.609202+0000","last_fullsized":"2023-10-30T12:22:43.609202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:31:20.350152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110004+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:22:44.110004+0000","last_peered":"2023-10-30T12:22:44.110004+0000","last_clean":"2023-10-30T12:22:44.110004+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:22:44.110004+0000","last_undegraded":"2023-10-30T12:22:44.110004+0000","last_fullsized":"2023-10-30T12:22:44.110004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110202+0000","last_change":"2023-10-30T12:01:59.306784+0000","last_active":"2023-10-30T12:22:44.110202+0000","last_peered":"2023-10-30T12:22:44.110202+0000","last_clean":"2023-10-30T12:22:44.110202+0000","last_became_active":"2023-10-30T12:01:59.306293+0000","last_became_peered":"2023-10-30T12:01:59.306293+0000","last_unstale":"2023-10-30T12:22:44.110202+0000","last_undegraded":"2023-10-30T12:22:44.110202+0000","last_fullsized":"2023-10-30T12:22:44.110202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:26:17.267254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110433+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:22:44.110433+0000","last_peered":"2023-10-30T12:22:44.110433+0000","last_clean":"2023-10-30T12:22:44.110433+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:22:44.110433+0000","last_undegraded":"2023-10-30T12:22:44.110433+0000","last_fullsized":"2023-10-30T12:22:44.110433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":664,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599745+0000","last_change":"2023-10-30T12:01:59.314586+0000","last_active":"2023-10-30T12:22:43.599745+0000","last_peered":"2023-10-30T12:22:43.599745+0000","last_clean":"2023-10-30T12:22:43.599745+0000","last_became_active":"2023-10-30T12:01:59.313414+0000","last_became_peered":"2023-10-30T12:01:59.313414+0000","last_unstale":"2023-10-30T12:22:43.599745+0000","last_undegraded":"2023-10-30T12:22:43.599745+0000","last_fullsized":"2023-10-30T12:22:43.599745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:05.821606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609150+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:22:43.609150+0000","last_peered":"2023-10-30T12:22:43.609150+0000","last_clean":"2023-10-30T12:22:43.609150+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:22:43.609150+0000","last_undegraded":"2023-10-30T12:22:43.609150+0000","last_fullsized":"2023-10-30T12:22:43.609150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110338+0000","last_change":"2023-10-30T12:01:59.289541+0000","last_active":"2023-10-30T12:22:44.110338+0000","last_peered":"2023-10-30T12:22:44.110338+0000","last_clean":"2023-10-30T12:22:44.110338+0000","last_became_active":"2023-10-30T12:01:59.289175+0000","last_became_peered":"2023-10-30T12:01:59.289175+0000","last_unstale":"2023-10-30T12:22:44.110338+0000","last_undegraded":"2023-10-30T12:22:44.110338+0000","last_fullsized":"2023-10-30T12:22:44.110338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:01.620219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":668,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601903+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:22:44.601903+0000","last_peered":"2023-10-30T12:22:44.601903+0000","last_clean":"2023-10-30T12:22:44.601903+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:22:44.601903+0000","last_undegraded":"2023-10-30T12:22:44.601903+0000","last_fullsized":"2023-10-30T12:22:44.601903+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110134+0000","last_change":"2023-10-30T12:01:59.307590+0000","last_active":"2023-10-30T12:22:44.110134+0000","last_peered":"2023-10-30T12:22:44.110134+0000","last_clean":"2023-10-30T12:22:44.110134+0000","last_became_active":"2023-10-30T12:01:59.302155+0000","last_became_peered":"2023-10-30T12:01:59.302155+0000","last_unstale":"2023-10-30T12:22:44.110134+0000","last_undegraded":"2023-10-30T12:22:44.110134+0000","last_fullsized":"2023-10-30T12:22:44.110134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:48.321197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110344+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:22:44.110344+0000","last_peered":"2023-10-30T12:22:44.110344+0000","last_clean":"2023-10-30T12:22:44.110344+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:22:44.110344+0000","last_undegraded":"2023-10-30T12:22:44.110344+0000","last_fullsized":"2023-10-30T12:22:44.110344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":820,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602127+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:22:44.602127+0000","last_peered":"2023-10-30T12:22:44.602127+0000","last_clean":"2023-10-30T12:22:44.602127+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:22:44.602127+0000","last_undegraded":"2023-10-30T12:22:44.602127+0000","last_fullsized":"2023-10-30T12:22:44.602127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609190+0000","last_change":"2023-10-30T12:01:59.256285+0000","last_active":"2023-10-30T12:22:43.609190+0000","last_peered":"2023-10-30T12:22:43.609190+0000","last_clean":"2023-10-30T12:22:43.609190+0000","last_became_active":"2023-10-30T12:01:59.253931+0000","last_became_peered":"2023-10-30T12:01:59.253931+0000","last_unstale":"2023-10-30T12:22:43.609190+0000","last_undegraded":"2023-10-30T12:22:43.609190+0000","last_fullsized":"2023-10-30T12:22:43.609190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:21.312914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711364+0000","last_change":"2023-10-30T12:01:59.296719+0000","last_active":"2023-10-30T12:22:45.711364+0000","last_peered":"2023-10-30T12:22:45.711364+0000","last_clean":"2023-10-30T12:22:45.711364+0000","last_became_active":"2023-10-30T12:01:59.277183+0000","last_became_peered":"2023-10-30T12:01:59.277183+0000","last_unstale":"2023-10-30T12:22:45.711364+0000","last_undegraded":"2023-10-30T12:22:45.711364+0000","last_fullsized":"2023-10-30T12:22:45.711364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:49.956478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":665,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602043+0000","last_change":"2023-10-30T12:01:59.339853+0000","last_active":"2023-10-30T12:22:44.602043+0000","last_peered":"2023-10-30T12:22:44.602043+0000","last_clean":"2023-10-30T12:22:44.602043+0000","last_became_active":"2023-10-30T12:01:59.339614+0000","last_became_peered":"2023-10-30T12:01:59.339614+0000","last_unstale":"2023-10-30T12:22:44.602043+0000","last_undegraded":"2023-10-30T12:22:44.602043+0000","last_fullsized":"2023-10-30T12:22:44.602043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:21.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987106+0000","last_change":"2023-10-30T12:01:59.303724+0000","last_active":"2023-10-30T12:22:43.987106+0000","last_peered":"2023-10-30T12:22:43.987106+0000","last_clean":"2023-10-30T12:22:43.987106+0000","last_became_active":"2023-10-30T12:01:59.303216+0000","last_became_peered":"2023-10-30T12:01:59.303216+0000","last_unstale":"2023-10-30T12:22:43.987106+0000","last_undegraded":"2023-10-30T12:22:43.987106+0000","last_fullsized":"2023-10-30T12:22:43.987106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:35.112009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109953+0000","last_change":"2023-10-30T12:01:59.209818+0000","last_active":"2023-10-30T12:22:44.109953+0000","last_peered":"2023-10-30T12:22:44.109953+0000","last_clean":"2023-10-30T12:22:44.109953+0000","last_became_active":"2023-10-30T12:01:59.209551+0000","last_became_peered":"2023-10-30T12:01:59.209551+0000","last_unstale":"2023-10-30T12:22:44.109953+0000","last_undegraded":"2023-10-30T12:22:44.109953+0000","last_fullsized":"2023-10-30T12:22:44.109953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:49.423499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":653,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609027+0000","last_change":"2023-10-30T12:01:59.312819+0000","last_active":"2023-10-30T12:22:43.609027+0000","last_peered":"2023-10-30T12:22:43.609027+0000","last_clean":"2023-10-30T12:22:43.609027+0000","last_became_active":"2023-10-30T12:01:59.310341+0000","last_became_peered":"2023-10-30T12:01:59.310341+0000","last_unstale":"2023-10-30T12:22:43.609027+0000","last_undegraded":"2023-10-30T12:22:43.609027+0000","last_fullsized":"2023-10-30T12:22:43.609027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:52.592573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"18'7","reported_seq":678,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601944+0000","last_change":"2023-10-30T12:01:59.261094+0000","last_active":"2023-10-30T12:22:44.601944+0000","last_peered":"2023-10-30T12:22:44.601944+0000","last_clean":"2023-10-30T12:22:44.601944+0000","last_became_active":"2023-10-30T12:01:59.260849+0000","last_became_peered":"2023-10-30T12:01:59.260849+0000","last_unstale":"2023-10-30T12:22:44.601944+0000","last_undegraded":"2023-10-30T12:22:44.601944+0000","last_fullsized":"2023-10-30T12:22:44.601944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:36.395779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610042+0000","last_change":"2023-10-30T12:08:57.463451+0000","last_active":"2023-10-30T12:22:44.610042+0000","last_peered":"2023-10-30T12:22:44.610042+0000","last_clean":"2023-10-30T12:22:44.610042+0000","last_became_active":"2023-10-30T12:08:57.463102+0000","last_became_peered":"2023-10-30T12:08:57.463102+0000","last_unstale":"2023-10-30T12:22:44.610042+0000","last_undegraded":"2023-10-30T12:22:44.610042+0000","last_fullsized":"2023-10-30T12:22:44.610042+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:48.599858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710660+0000","last_change":"2023-10-30T12:01:59.247352+0000","last_active":"2023-10-30T12:22:45.710660+0000","last_peered":"2023-10-30T12:22:45.710660+0000","last_clean":"2023-10-30T12:22:45.710660+0000","last_became_active":"2023-10-30T12:01:59.246868+0000","last_became_peered":"2023-10-30T12:01:59.246868+0000","last_unstale":"2023-10-30T12:22:45.710660+0000","last_undegraded":"2023-10-30T12:22:45.710660+0000","last_fullsized":"2023-10-30T12:22:45.710660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:50.392311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":649,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109897+0000","last_change":"2023-10-30T12:01:59.276982+0000","last_active":"2023-10-30T12:22:44.109897+0000","last_peered":"2023-10-30T12:22:44.109897+0000","last_clean":"2023-10-30T12:22:44.109897+0000","last_became_active":"2023-10-30T12:01:59.239117+0000","last_became_peered":"2023-10-30T12:01:59.239117+0000","last_unstale":"2023-10-30T12:22:44.109897+0000","last_undegraded":"2023-10-30T12:22:44.109897+0000","last_fullsized":"2023-10-30T12:22:44.109897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.157663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110047+0000","last_change":"2023-10-30T12:01:59.297769+0000","last_active":"2023-10-30T12:22:44.110047+0000","last_peered":"2023-10-30T12:22:44.110047+0000","last_clean":"2023-10-30T12:22:44.110047+0000","last_became_active":"2023-10-30T12:01:59.293055+0000","last_became_peered":"2023-10-30T12:01:59.293055+0000","last_unstale":"2023-10-30T12:22:44.110047+0000","last_undegraded":"2023-10-30T12:22:44.110047+0000","last_fullsized":"2023-10-30T12:22:44.110047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:53.053266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609249+0000","last_change":"2023-10-30T12:01:59.310309+0000","last_active":"2023-10-30T12:22:43.609249+0000","last_peered":"2023-10-30T12:22:43.609249+0000","last_clean":"2023-10-30T12:22:43.609249+0000","last_became_active":"2023-10-30T12:01:59.307440+0000","last_became_peered":"2023-10-30T12:01:59.307440+0000","last_unstale":"2023-10-30T12:22:43.609249+0000","last_undegraded":"2023-10-30T12:22:43.609249+0000","last_fullsized":"2023-10-30T12:22:43.609249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:39.454237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711399+0000","last_change":"2023-10-30T12:01:59.249544+0000","last_active":"2023-10-30T12:22:45.711399+0000","last_peered":"2023-10-30T12:22:45.711399+0000","last_clean":"2023-10-30T12:22:45.711399+0000","last_became_active":"2023-10-30T12:01:59.239219+0000","last_became_peered":"2023-10-30T12:01:59.239219+0000","last_unstale":"2023-10-30T12:22:45.711399+0000","last_undegraded":"2023-10-30T12:22:45.711399+0000","last_fullsized":"2023-10-30T12:22:45.711399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:17.868254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640526,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77256,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294584,"statfs":{"total":96636764160,"available":96557654016,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.82699999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.65000000000000002,"5min":0.81599999999999995,"15min":1.056},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65400000000000003,"5min":0.83399999999999996,"15min":2.173},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.28299999999999997},"max":{"1min":0.64800000000000002,"5min":0.72799999999999998,"15min":0.78100000000000003},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.67400000000000004,"5min":0.74299999999999999,"15min":0.79900000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.41499999999999998,"15min":0.32900000000000001},"max":{"1min":0.65000000000000002,"5min":0.79900000000000004,"15min":0.84699999999999998},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.36899999999999999,"15min":0.373},"min":{"1min":0.25,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":0.499,"5min":0.58799999999999997,"15min":0.70399999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.223},"max":{"1min":0.54500000000000004,"5min":0.69699999999999995,"15min":0.95199999999999996},"last":0.377}]},{"osd":4,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.31,"5min":0.252,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.63700000000000001,"15min":0.63800000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.435,"15min":0.45300000000000001},"min":{"1min":0.28100000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.60299999999999998,"5min":0.68400000000000005,"15min":0.72099999999999997},"last":0.54500000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640527,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81416,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94290424,"statfs":{"total":96636764160,"available":96553394176,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53900000000000003,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.51400000000000001},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.84899999999999998},"last":0.60699999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.79000000000000004,"15min":0.94099999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.36299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.67700000000000005,"5min":0.749,"15min":0.79800000000000004},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27300000000000002},"max":{"1min":0.627,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.246},"max":{"1min":0.59399999999999997,"5min":0.627,"15min":0.63600000000000001},"last":0.376},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.25,"5min":0.249,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.67000000000000004,"15min":0.77100000000000002},"last":0.38700000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.435,"15min":0.40999999999999998},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.25600000000000001},"max":{"1min":0.621,"5min":0.64000000000000001,"15min":0.69999999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.67400000000000004,"15min":0.89700000000000002},"last":0.54700000000000004}]}]},{"osd":3,"up_from":11,"seq":47244640527,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69444,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302396,"statfs":{"total":96636764160,"available":96565653504,"internally_reserved":0,"allocated":2207744,"data_stored":996605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.503,"15min":0.501},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72799999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.61899999999999999,"5min":0.71399999999999997,"15min":1.2490000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.68999999999999995,"5min":0.81100000000000005,"15min":1.2689999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69999999999999996,"5min":0.82599999999999996,"15min":1.337},"last":0.67900000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":0.88900000000000001,"15min":1.302},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":0.746,"15min":0.76900000000000002},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.37,"15min":0.38},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":0.50700000000000001,"5min":0.61199999999999999,"15min":1.371},"last":0.495},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.30599999999999999},"max":{"1min":0.56599999999999995,"5min":0.66300000000000003,"15min":1.444},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39600000000000002,"15min":0.38700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":0.502,"5min":0.64200000000000002,"15min":0.876},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.375,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53500000000000003,"5min":0.84099999999999997,"15min":1.3540000000000001},"last":0.627}]}]},{"osd":2,"up_from":11,"seq":47244640526,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64764,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94307076,"statfs":{"total":96636764160,"available":96570445824,"internally_reserved":0,"allocated":1675264,"data_stored":467677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33500000000000002},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44800000000000001,"5min":0.50700000000000001,"15min":2.2730000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.38400000000000001},"min":{"1min":0.27500000000000002,"5min":0.24199999999999999,"15min":0.22800000000000001},"max":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":2.141},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33700000000000002,"15min":0.34200000000000003},"min":{"1min":0.24399999999999999,"5min":0.21299999999999999,"15min":0.20799999999999999},"max":{"1min":0.46999999999999997,"5min":0.502,"15min":1.087},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.24399999999999999},"max":{"1min":0.495,"5min":0.63900000000000001,"15min":3.8260000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.42199999999999999,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":0.749,"5min":0.75700000000000001,"15min":1.143},"last":0.5},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.42899999999999999,"5min":0.33600000000000002,"15min":0.314},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":3.4729999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.34399999999999997,"15min":0.27300000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.3240000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.45600000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":1.8340000000000001},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.32200000000000001},"max":{"1min":0.81699999999999995,"5min":0.81999999999999995,"15min":1.893},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.48499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75900000000000001,"5min":0.81599999999999995,"15min":2.2269999999999999},"last":0.65400000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640528,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59208,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312632,"statfs":{"total":96636764160,"available":96576135168,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.34499999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":0.46899999999999997,"5min":0.745,"15min":0.745},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.374},"min":{"1min":0.253,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.38200000000000001},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.223},"max":{"1min":0.54000000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.255},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.51800000000000002,"5min":0.621,"15min":0.93200000000000005},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.443,"5min":0.375,"15min":0.28499999999999998},"max":{"1min":0.69999999999999996,"5min":0.86599999999999999,"15min":0.879},"last":0.435},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.79900000000000004,"5min":0.85399999999999998,"15min":0.88900000000000001},"last":0.501}]},{"osd":4,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.46400000000000002,"5min":0.41699999999999998,"15min":0.29899999999999999},"max":{"1min":0.86799999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.36799999999999999},"max":{"1min":0.71299999999999997,"5min":0.94799999999999995,"15min":0.95599999999999996},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.58899999999999997},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.39300000000000002},"max":{"1min":0.84499999999999997,"5min":1,"15min":1},"last":0.46600000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640528,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77760,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294080,"statfs":{"total":96636764160,"available":96557137920,"internally_reserved":0,"allocated":2203648,"data_stored":996600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.35199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23899999999999999},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.21299999999999999},"max":{"1min":1.802,"5min":1.802,"15min":3.8519999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26700000000000002,"15min":0.20100000000000001},"max":{"1min":0.58699999999999997,"5min":0.623,"15min":2.0590000000000002},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.27800000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":1.5489999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":1.78},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.29399999999999998},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59099999999999997,"15min":0.57899999999999996},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":1.6020000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:23:13.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:23:13.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.237+0000 7f4aae875640 10 monclient: get_monmap_and_config 2023-10-30T12:23:13.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.238+0000 7f4aae875640 10 monclient: build_initial_monmap 2023-10-30T12:23:13.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.238+0000 7f4aae875640 10 monclient: monmap: 2023-10-30T12:23:13.240 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:13.241 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:13.241 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:13.239070+0000 2023-10-30T12:23:13.241 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:13.239070+0000 2023-10-30T12:23:13.242 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:13.242 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:13.242 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:13.242 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:13.244 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:13.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.238+0000 7f4aae875640 1 Processor -- start 2023-10-30T12:23:13.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.238+0000 7f4aae875640 1 -- start start 2023-10-30T12:23:13.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.238+0000 7f4aae875640 10 monclient: init 2023-10-30T12:23:13.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:13.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-10-30T12:23:13.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4aa80a2dd0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:13.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa806a8e0 0x7f4aa80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:13.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa806a8e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:13.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4aa8071400 0x7f4aa813cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:13.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa8071400 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:13.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa80a31b0 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:13.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa806a6a0 con 0x7f4aa8071400 2023-10-30T12:23:13.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:13.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa806be90 con 0x7f4aa806a8e0 2023-10-30T12:23:13.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:13.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aae875640 10 monclient(hunting): authenticate will time out at 2238.422251s 2023-10-30T12:23:13.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa806a8e0 0x7f4aa80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:13.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa7fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4aa80a2dd0 0x7f4aa80a46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49606/0 (socket says 172.21.15.50:49606) 2023-10-30T12:23:13.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa7fff640 1 -- 172.21.15.50:0/2272350854 learned_addr learned my addr 172.21.15.50:0/2272350854 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:13.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): get_auth_request con 0x7f4aa806a8e0 auth_method 0 2023-10-30T12:23:13.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:13.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:13.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:13.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:13.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:13.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:13.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aace12640 1 --2- 172.21.15.50:0/2272350854 >> v2:172.21.15.118:3300/0 conn(0x7f4aa8071400 0x7f4aa813cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:13.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aace12640 10 monclient(hunting): get_auth_request con 0x7f4aa8071400 auth_method 0 2023-10-30T12:23:13.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aace12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:13.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aace12640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:13.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aace12640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:13.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa6ffd640 1 -- 172.21.15.50:0/2272350854 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25487634 0 0) 0x7f4aa80a31b0 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_done global_id 284135 payload 274 2023-10-30T12:23:13.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 1 -- 172.21.15.50:0/2272350854 >> v2:172.21.15.118:3300/0 conn(0x7f4aa8071400 msgr2=0x7f4aa813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:13.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 1 --2- 172.21.15.50:0/2272350854 >> v2:172.21.15.118:3300/0 conn(0x7f4aa8071400 0x7f4aa813cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 1 -- 172.21.15.50:0/2272350854 >> v1:172.21.15.50:6789/0 conn(0x7f4aa80a2dd0 legacy=0x7f4aa80a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 1 monclient: found mon.noname-b 2023-10-30T12:23:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:13.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:13.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa77fe640 1 -- 172.21.15.50:0/2272350854 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa809e190 con 0x7f4aa806a8e0 2023-10-30T12:23:13.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.239+0000 7f4aa6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:13.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa77fe640 1 --2- 172.21.15.50:0/2272350854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa806a8e0 0x7f4aa80a4dd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4a980097a0 tx=0x7f4a9802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62efdac6564a3cf6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:13.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 1 -- 172.21.15.50:0/2272350854 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9802f050 con 0x7f4aa806a8e0 2023-10-30T12:23:13.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:13.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:13.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: dump: 2023-10-30T12:23:13.265 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:13.265 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:13.265 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.265 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.266 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:13.266 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:13.266 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:13.266 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:13.267 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:13.267 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:13.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: _finish_auth 0 2023-10-30T12:23:13.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:13.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:13.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aae875640 5 monclient: authenticate success, global_id 284135 2023-10-30T12:23:13.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aae875640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:13.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 1 -- 172.21.15.50:0/2272350854 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a98031700 con 0x7f4aa806a8e0 2023-10-30T12:23:13.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:13.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aae875640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:13.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aae875640 4 set_mon_vals no callback set 2023-10-30T12:23:13.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 1 -- 172.21.15.50:0/2272350854 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98031a90 con 0x7f4aa806a8e0 2023-10-30T12:23:13.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:13.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:13.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.240+0000 7f4aa6ffd640 10 monclient: dump: 2023-10-30T12:23:13.271 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:13.271 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:13.271 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.271 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.272 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:13.272 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:13.272 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:13.272 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:13.273 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:13.273 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:13.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 10 monclient: shutdown 2023-10-30T12:23:13.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:13.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 -- 172.21.15.50:0/2272350854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa806a8e0 msgr2=0x7f4aa80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 --2- 172.21.15.50:0/2272350854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa806a8e0 0x7f4aa80a4dd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4a980097a0 tx=0x7f4a9802b7a0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 -- 172.21.15.50:0/2272350854 shutdown_connections 2023-10-30T12:23:13.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 --2- 172.21.15.50:0/2272350854 >> v2:172.21.15.118:3300/0 conn(0x7f4aa8071400 0x7f4aa813cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 --2- 172.21.15.50:0/2272350854 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa806a8e0 0x7f4aa80a4dd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 -- 172.21.15.50:0/2272350854 >> 172.21.15.50:0/2272350854 conn(0x7f4aa80a22b0 msgr2=0x7f4aa80a26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:13.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 -- 172.21.15.50:0/2272350854 shutdown_connections 2023-10-30T12:23:13.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 -- 172.21.15.50:0/2272350854 wait complete. 2023-10-30T12:23:13.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 10 monclient: build_initial_monmap 2023-10-30T12:23:13.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 10 monclient: monmap: 2023-10-30T12:23:13.276 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:13.277 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:13.277 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:13.277 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:13.277 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:13.278 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:13.278 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:13.278 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:13.279 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:13.279 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:13.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 librados: starting msgr at 2023-10-30T12:23:13.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.241+0000 7f4aae875640 1 librados: starting objecter 2023-10-30T12:23:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 Processor -- start 2023-10-30T12:23:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 -- start start 2023-10-30T12:23:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 librados: setting wanted keys 2023-10-30T12:23:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 librados: calling monclient init 2023-10-30T12:23:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient: init 2023-10-30T12:23:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:23:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa8071400 0x7f4aa81c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:13.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa8071400 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:13.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa80a2dd0 0x7f4aa81c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:13.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa80a2dd0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4aa81c6e30 0x7f4aa81c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa81c6e30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa806b130 con 0x7f4aa81c6e30 2023-10-30T12:23:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa81a8a30 con 0x7f4aa8071400 2023-10-30T12:23:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8070640 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aae875640 10 monclient(hunting): authenticate will time out at 2238.425724s 2023-10-30T12:23:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa80a2dd0 0x7f4aa81c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4aa81c6e30 0x7f4aa81c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:13.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa80a2dd0 0x7f4aa81c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60664/0 (socket says 172.21.15.50:60664) 2023-10-30T12:23:13.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 1 -- 172.21.15.50:0/4053572223 learned_addr learned my addr 172.21.15.50:0/4053572223 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:13.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4aa81c6e30 0x7f4aa81c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59800/0 (socket says 172.21.15.50:59800) 2023-10-30T12:23:13.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 10 monclient(hunting): get_auth_request con 0x7f4aa80a2dd0 auth_method 0 2023-10-30T12:23:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 10 monclient(hunting): get_auth_request con 0x7f4aa81c6e30 auth_method 0 2023-10-30T12:23:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:13.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa7fff640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa8071400 0x7f4aa81c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:13.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:13.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:13.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aace12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:13.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:13.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:13.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:13.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa7fff640 10 monclient(hunting): get_auth_request con 0x7f4aa8071400 auth_method 0 2023-10-30T12:23:13.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:13.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:13.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.242+0000 7f4aa7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:13.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:13.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:13.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:13.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 10 monclient(hunting): handle_auth_done global_id 284138 payload 824 2023-10-30T12:23:13.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 1 -- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa8071400 msgr2=0x7f4aa81c3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa8071400 0x7f4aa81c3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 1 -- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4aa81c6e30 msgr2=0x7f4aa81c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4aa81c6e30 0x7f4aa81c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:13.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 1 monclient: found mon.noname-c 2023-10-30T12:23:13.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:13.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 1 -- 172.21.15.50:0/4053572223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a98009400 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aace12640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4aa81c6e30 0x7f4aa81c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa77fe640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa80a2dd0 0x7f4aa81c68f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a980098c0 tx=0x7f4a980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 client.?.objecter ms_handle_connect 0x7f4aa80a2dd0 2023-10-30T12:23:13.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:13.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 1 -- 172.21.15.50:0/4053572223 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98004030 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:13.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:13.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: dump: 2023-10-30T12:23:13.298 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:13.298 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:13.299 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.299 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.299 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:13.299 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:13.300 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:13.300 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:13.300 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:13.300 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:13.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: _finish_auth 0 2023-10-30T12:23:13.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:23:13.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:13.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 1 -- 172.21.15.50:0/4053572223 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a98004190 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 5 monclient: authenticate success, global_id 284138 2023-10-30T12:23:13.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 10 monclient: _renew_subs 2023-10-30T12:23:13.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:13.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa81aad90 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:13.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 1 -- 172.21.15.50:0/4053572223 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98033930 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:13.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:13.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa4ff9640 10 monclient: dump: 2023-10-30T12:23:13.304 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:13.304 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:13.305 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.305 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:13.305 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:13.305 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:13.306 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:13.306 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:13.306 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:13.307 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:13.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aa67fc640 4 set_mon_vals no callback set 2023-10-30T12:23:13.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 10 client.284138.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:13.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 10 monclient: _renew_subs 2023-10-30T12:23:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa81a92b0 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.243+0000 7f4aae875640 1 librados: init done 2023-10-30T12:23:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa4ff9640 1 -- 172.21.15.50:0/4053572223 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4a98033a90 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa4ff9640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a940392f0 0x7f4a9403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aae875640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa4ff9640 1 -- 172.21.15.50:0/4053572223 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f4a9806ca10 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aae875640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:13.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa4ff9640 10 client.284138.objecter ms_dispatch 0x7f4aa8000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:13.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aae875640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:13.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa4ff9640 3 client.284138.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:13.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa4ff9640 3 client.284138.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:13.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa80a46c0 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa4ff9640 20 client.284138.objecter dump_active .. 0 homeless 2023-10-30T12:23:13.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa7fff640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a940392f0 0x7f4a9403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:13.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.244+0000 7f4aa7fff640 10 monclient: get_auth_request con 0x7f4a940392f0 auth_method 0 2023-10-30T12:23:13.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.245+0000 7f4aa4ff9640 10 client.284138.objecter ms_handle_connect 0x7f4a940392f0 2023-10-30T12:23:13.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.245+0000 7f4aa7fff640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a940392f0 0x7f4a9403b7b0 secure :-1 s=READY pgs=4357 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000f60 tx=0x7f4a9c007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:13.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.249+0000 7f4aa4ff9640 1 -- 172.21.15.50:0/4053572223 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4a98038070 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.249+0000 7f4aa4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:13.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.249+0000 7f4aa4ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:13.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.427+0000 7f4aae875640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:23:13.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.427+0000 7f4aae875640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:23:13.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.427+0000 7f4aae875640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:13.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.427+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4aa804e160 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.428+0000 7f4aa4ff9640 1 -- 172.21.15.50:0/4053572223 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v686) v1 ==== 74+0+15139 (secure 0 0 0) 0x7f4a98003610 con 0x7f4aa80a2dd0 2023-10-30T12:23:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.428+0000 7f4aa4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-10-30T12:23:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.428+0000 7f4aa4ff9640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:23:13.432 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:23:13.432 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":686,"fsid":"87b22564-9727-4955-ac7b-d9a79464690c","created":"2023-10-30T12:00:40.280079+0000","modified":"2023-10-30T12:22:44.598315+0000","last_up_change":"2023-10-30T12:00:48.401197+0000","last_in_change":"2023-10-30T12:00:41.981126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":101,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:00:48.447131+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-30T12:00:50.466441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-10-30T12:01:57.401478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dba8cbf1-bc2e-4420-957f-11ec607e814e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":681,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6817","nonce":47190190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6819","nonce":47190190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6823","nonce":47190190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":47190190},{"type":"v1","addr":"172.21.15.50:6821","nonce":47190190}]},"public_addr":"172.21.15.50:6817/47190190","cluster_addr":"172.21.15.50:6819/47190190","heartbeat_back_addr":"172.21.15.50:6823/47190190","heartbeat_front_addr":"172.21.15.50:6821/47190190","state":["exists","up"]},{"osd":1,"uuid":"c13ebbef-aa1a-4da7-b5f6-26b02e43955c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":681,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6809","nonce":659398918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6811","nonce":659398918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6815","nonce":659398918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":659398918},{"type":"v1","addr":"172.21.15.50:6813","nonce":659398918}]},"public_addr":"172.21.15.50:6809/659398918","cluster_addr":"172.21.15.50:6811/659398918","heartbeat_back_addr":"172.21.15.50:6815/659398918","heartbeat_front_addr":"172.21.15.50:6813/659398918","state":["exists","up"]},{"osd":2,"uuid":"23116074-425b-4664-a1be-0e40bba62fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":681,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6801","nonce":3518075216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6803","nonce":3518075216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6807","nonce":3518075216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3518075216},{"type":"v1","addr":"172.21.15.50:6805","nonce":3518075216}]},"public_addr":"172.21.15.50:6801/3518075216","cluster_addr":"172.21.15.50:6803/3518075216","heartbeat_back_addr":"172.21.15.50:6807/3518075216","heartbeat_front_addr":"172.21.15.50:6805/3518075216","state":["exists","up"]},{"osd":3,"uuid":"515e513c-cb67-4495-ba25-68cb35215c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":681,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6810","nonce":1009584072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6814","nonce":1009584072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6822","nonce":1009584072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1009584072},{"type":"v1","addr":"172.21.15.118:6818","nonce":1009584072}]},"public_addr":"172.21.15.118:6810/1009584072","cluster_addr":"172.21.15.118:6814/1009584072","heartbeat_back_addr":"172.21.15.118:6822/1009584072","heartbeat_front_addr":"172.21.15.118:6818/1009584072","state":["exists","up"]},{"osd":4,"uuid":"fa9df8af-676c-49c6-9888-0e330fbbc4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":681,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6811","nonce":4272127357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6815","nonce":4272127357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6823","nonce":4272127357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6817","nonce":4272127357},{"type":"v1","addr":"172.21.15.118:6819","nonce":4272127357}]},"public_addr":"172.21.15.118:6811/4272127357","cluster_addr":"172.21.15.118:6815/4272127357","heartbeat_back_addr":"172.21.15.118:6823/4272127357","heartbeat_front_addr":"172.21.15.118:6819/4272127357","state":["exists","up"]},{"osd":5,"uuid":"79b07bf0-2944-4413-b6e1-23ecbc533add","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":681,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6801","nonce":1399177807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6803","nonce":1399177807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6807","nonce":1399177807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1399177807},{"type":"v1","addr":"172.21.15.118:6805","nonce":1399177807}]},"public_addr":"172.21.15.118:6801/1399177807","cluster_addr":"172.21.15.118:6803/1399177807","heartbeat_back_addr":"172.21.15.118:6807/1399177807","heartbeat_front_addr":"172.21.15.118:6805/1399177807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.083230+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.120938+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.049176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.192079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:00:47.216462+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"15.2","mappings":[{"from":4,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":21,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":25,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":54,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":11,"length":2}]},{"pool":71,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":76,"snaps":[{"begin":2,"length":1}]},{"pool":86,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":87,"snaps":[{"begin":4,"length":4}]},{"pool":108,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":115,"snaps":[{"begin":12,"length":1}]},{"pool":122,"snaps":[{"begin":2,"length":1},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":140,"snaps":[{"begin":14,"length":1}]},{"pool":151,"snaps":[{"begin":17,"length":1}]},{"pool":157,"snaps":[{"begin":19,"length":2}]},{"pool":172,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:23:13.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 10 librados: watch_flush enter 2023-10-30T12:23:13.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 10 librados: watch_flush exit 2023-10-30T12:23:13.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 20 client.284138.objecter shutdown clearing up homeless session... 2023-10-30T12:23:13.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 10 client.284138.objecter successfully canceled tick 2023-10-30T12:23:13.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a940392f0 msgr2=0x7f4a9403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a940392f0 0x7f4a9403b7b0 secure :-1 s=READY pgs=4357 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000f60 tx=0x7f4a9c007d50 comp rx=0 tx=0).stop 2023-10-30T12:23:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 10 monclient: shutdown 2023-10-30T12:23:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa80a2dd0 msgr2=0x7f4aa81c68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa80a2dd0 0x7f4aa81c68f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a980098c0 tx=0x7f4a980046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 shutdown_connections 2023-10-30T12:23:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4a940392f0 0x7f4a9403b7b0 unknown :-1 s=CLOSED pgs=4357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4aa81c6e30 0x7f4aa81c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4aa80a2dd0 0x7f4aa81c68f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 --2- 172.21.15.50:0/4053572223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4aa8071400 0x7f4aa81c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:13.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 >> 172.21.15.50:0/4053572223 conn(0x7f4aa80a22b0 msgr2=0x7f4aa806e820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:13.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.430+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 shutdown_connections 2023-10-30T12:23:13.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.431+0000 7f4aae875640 1 -- 172.21.15.50:0/4053572223 wait complete. 2023-10-30T12:23:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:13.431+0000 7f4aae875640 1 librados: shutdown 2023-10-30T12:23:14.443 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T12:23:14.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:23:14.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.559+0000 7ff8e8983640 10 monclient: get_monmap_and_config 2023-10-30T12:23:14.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.559+0000 7ff8e8983640 10 monclient: build_initial_monmap 2023-10-30T12:23:14.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.559+0000 7ff8e8983640 10 monclient: monmap: 2023-10-30T12:23:14.562 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:14.562 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.562 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:14.560530+0000 2023-10-30T12:23:14.562 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:14.560530+0000 2023-10-30T12:23:14.563 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:14.563 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.563 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:14.563 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:14.564 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:14.564 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:14.564 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:14.564 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 1 Processor -- start 2023-10-30T12:23:14.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 1 -- start start 2023-10-30T12:23:14.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient: init 2023-10-30T12:23:14.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:14.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-10-30T12:23:14.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8e40965d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:14.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff8e409d210 0x7ff8e41389f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e409d210 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:14.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7ff8e4138f30 0x7ff8e413b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e4138f30 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:14.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40970c0 con 0x7ff8e40965d0 2023-10-30T12:23:14.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e4099c30 con 0x7ff8e409d210 2023-10-30T12:23:14.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e409bf90 con 0x7ff8e4138f30 2023-10-30T12:23:14.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:14.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e8983640 10 monclient(hunting): authenticate will time out at 2239.743711s 2023-10-30T12:23:14.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff8e409d210 0x7ff8e41389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff8e409d210 0x7ff8e41389f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59812/0 (socket says 172.21.15.50:59812) 2023-10-30T12:23:14.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 1 -- 172.21.15.50:0/2123401399 learned_addr learned my addr 172.21.15.50:0/2123401399 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:14.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 10 monclient(hunting): get_auth_request con 0x7ff8e409d210 auth_method 0 2023-10-30T12:23:14.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e259c640 1 --1- 172.21.15.50:0/2123401399 >> v1:172.21.15.50:6789/0 conn(0x7ff8e40965d0 0x7ff8e40969b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49616/0 (socket says 172.21.15.50:49616) 2023-10-30T12:23:14.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.560+0000 7ff8e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e2d9d640 1 --2- 172.21.15.50:0/2123401399 >> v2:172.21.15.118:3300/0 conn(0x7ff8e4138f30 0x7ff8e413b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e2d9d640 10 monclient(hunting): get_auth_request con 0x7ff8e4138f30 auth_method 0 2023-10-30T12:23:14.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 10 monclient(hunting): handle_auth_done global_id 264381 payload 274 2023-10-30T12:23:14.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 1 -- 172.21.15.50:0/2123401399 >> v2:172.21.15.118:3300/0 conn(0x7ff8e4138f30 msgr2=0x7ff8e413b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:14.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 1 --2- 172.21.15.50:0/2123401399 >> v2:172.21.15.118:3300/0 conn(0x7ff8e4138f30 0x7ff8e413b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 1 -- 172.21.15.50:0/2123401399 >> v1:172.21.15.50:6789/0 conn(0x7ff8e40965d0 legacy=0x7ff8e40969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 10 monclient: _finish_hunting 0 2023-10-30T12:23:14.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 1 monclient: found mon.noname-a 2023-10-30T12:23:14.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e159a640 1 -- 172.21.15.50:0/2123401399 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150567219 0 0) 0x7ff8e40970c0 con 0x7ff8e40965d0 2023-10-30T12:23:14.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:14.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:14.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 1 -- 172.21.15.50:0/2123401399 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e409c270 con 0x7ff8e409d210 2023-10-30T12:23:14.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:14.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.561+0000 7ff8e1d9b640 1 --2- 172.21.15.50:0/2123401399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e409d210 0x7ff8e41389f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009870 tx=0x7ff8d802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210407019ac07019 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:14.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 1 -- 172.21.15.50:0/2123401399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80317e0 con 0x7ff8e409d210 2023-10-30T12:23:14.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:14.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: dump: 2023-10-30T12:23:14.586 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.586 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.586 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.586 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.587 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.587 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.587 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.587 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.588 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.588 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: _finish_auth 0 2023-10-30T12:23:14.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: _check_auth_tickets 2023-10-30T12:23:14.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:14.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 5 monclient: authenticate success, global_id 264381 2023-10-30T12:23:14.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:14.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 1 -- 172.21.15.50:0/2123401399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8031940 con 0x7ff8e409d210 2023-10-30T12:23:14.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:14.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:14.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 4 set_mon_vals no callback set 2023-10-30T12:23:14.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 1 -- 172.21.15.50:0/2123401399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8031cd0 con 0x7ff8e409d210 2023-10-30T12:23:14.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:14.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e159a640 10 monclient: dump: 2023-10-30T12:23:14.592 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.592 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.592 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.593 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.593 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.593 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.593 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.594 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.594 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.594 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 10 monclient: shutdown 2023-10-30T12:23:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 -- 172.21.15.50:0/2123401399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e409d210 msgr2=0x7ff8e41389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 --2- 172.21.15.50:0/2123401399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e409d210 0x7ff8e41389f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009870 tx=0x7ff8d802b880 comp rx=0 tx=0).stop 2023-10-30T12:23:14.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 -- 172.21.15.50:0/2123401399 shutdown_connections 2023-10-30T12:23:14.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 --2- 172.21.15.50:0/2123401399 >> v2:172.21.15.118:3300/0 conn(0x7ff8e4138f30 0x7ff8e413b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 --2- 172.21.15.50:0/2123401399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e409d210 0x7ff8e41389f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 -- 172.21.15.50:0/2123401399 >> 172.21.15.50:0/2123401399 conn(0x7ff8e40a0480 msgr2=0x7ff8e40a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:14.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 -- 172.21.15.50:0/2123401399 shutdown_connections 2023-10-30T12:23:14.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.562+0000 7ff8e8983640 1 -- 172.21.15.50:0/2123401399 wait complete. 2023-10-30T12:23:14.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 10 monclient: build_initial_monmap 2023-10-30T12:23:14.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 10 monclient: monmap: 2023-10-30T12:23:14.598 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:14.598 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:14.598 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:14.598 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:14.599 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:14.600 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 1 librados: starting msgr at 2023-10-30T12:23:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 1 librados: starting objecter 2023-10-30T12:23:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 1 Processor -- start 2023-10-30T12:23:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 1 -- start start 2023-10-30T12:23:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 1 librados: setting wanted keys 2023-10-30T12:23:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 1 librados: calling monclient init 2023-10-30T12:23:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.563+0000 7ff8e8983640 10 monclient: init 2023-10-30T12:23:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:23:14.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8e40965d0 0x7ff8e41367e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e40965d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:14.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8e409d210 0x7ff8e4136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e409d210 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:14.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e4138f30 0x7ff8e4137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e4138f30 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:14.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4099530 con 0x7ff8e4138f30 2023-10-30T12:23:14.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e409a000 con 0x7ff8e409d210 2023-10-30T12:23:14.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4095120 con 0x7ff8e40965d0 2023-10-30T12:23:14.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:14.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient(hunting): authenticate will time out at 2239.747197s 2023-10-30T12:23:14.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8e40965d0 0x7ff8e41367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e4138f30 0x7ff8e4137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8e40965d0 0x7ff8e41367e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60676/0 (socket says 172.21.15.50:60676) 2023-10-30T12:23:14.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 1 -- 172.21.15.50:0/3617921724 learned_addr learned my addr 172.21.15.50:0/3617921724 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:14.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 10 monclient(hunting): get_auth_request con 0x7ff8e40965d0 auth_method 0 2023-10-30T12:23:14.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): get_auth_request con 0x7ff8e4138f30 auth_method 0 2023-10-30T12:23:14.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8e409d210 0x7ff8e4136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 10 monclient(hunting): get_auth_request con 0x7ff8e409d210 auth_method 0 2023-10-30T12:23:14.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient(hunting): handle_auth_done global_id 264387 payload 824 2023-10-30T12:23:14.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 -- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8e40965d0 msgr2=0x7ff8e41367e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff8e40965d0 0x7ff8e41367e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 -- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8e409d210 msgr2=0x7ff8e4136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8e409d210 0x7ff8e4136d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:23:14.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 monclient: found mon.noname-a 2023-10-30T12:23:14.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:14.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:14.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 -- 172.21.15.50:0/3617921724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d80094d0 con 0x7ff8e4138f30 2023-10-30T12:23:14.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e2d9d640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e4138f30 0x7ff8e4137260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007d60 tx=0x7ff8dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:14.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8e4138f30 2023-10-30T12:23:14.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:14.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc018070 con 0x7ff8e4138f30 2023-10-30T12:23:14.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:14.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 monclient: dump: 2023-10-30T12:23:14.618 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.618 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.619 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.619 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.619 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.619 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.620 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.620 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.620 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.620 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 monclient: _finish_auth 0 2023-10-30T12:23:14.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:14.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:14.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 5 monclient: authenticate success, global_id 264387 2023-10-30T12:23:14.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc00edd0 con 0x7ff8e4138f30 2023-10-30T12:23:14.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:14.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc003bb0 con 0x7ff8e4138f30 2023-10-30T12:23:14.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient: _renew_subs 2023-10-30T12:23:14.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:14.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e40905e0 con 0x7ff8e4138f30 2023-10-30T12:23:14.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.564+0000 7ff8e0d99640 4 set_mon_vals no callback set 2023-10-30T12:23:14.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:14.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8d2ffd640 10 monclient: dump: 2023-10-30T12:23:14.624 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.625 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.627 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.627 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.627 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.627 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.628 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 10 client.264387.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:14.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 10 monclient: _renew_subs 2023-10-30T12:23:14.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:14.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e40988a0 con 0x7ff8e4138f30 2023-10-30T12:23:14.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 1 librados: init done 2023-10-30T12:23:14.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 10 client.264387.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:14.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 15 client.264387.objecter _session_command_op_assign -1 1 2023-10-30T12:23:14.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 20 client.264387.objecter _get_session osd=-1 returning homeless 2023-10-30T12:23:14.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 10 client.264387.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:14.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8e409ba70 con 0x7ff8e4138f30 2023-10-30T12:23:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.565+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7ff8dc014420 con 0x7ff8e4138f30 2023-10-30T12:23:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8c40392f0 0x7ff8c403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7ff8dc04aac0 con 0x7ff8e4138f30 2023-10-30T12:23:14.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 10 client.264387.objecter ms_dispatch 0x7ff8e4000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:14.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 3 client.264387.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:14.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 3 client.264387.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:14.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 10 client.264387.objecter checking command 1 2023-10-30T12:23:14.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7ff8c4040680 0x7ff8c4042b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 20 client.264387.objecter _get_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 20 client.264387.objecter put_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 15 client.264387.objecter _session_command_op_remove -1 1 2023-10-30T12:23:14.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 20 client.264387.objecter _get_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 20 client.264387.objecter get_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 15 client.264387.objecter _session_command_op_assign 0 1 2023-10-30T12:23:14.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 20 client.264387.objecter put_session s=0x7ff8c403ffb0 osd=0 4 2023-10-30T12:23:14.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 10 client.264387.objecter _send_command 1 2023-10-30T12:23:14.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 --> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8c40431f0 con 0x7ff8c4040680 2023-10-30T12:23:14.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 20 client.264387.objecter dump_active .. 0 homeless 2023-10-30T12:23:14.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=686) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8dc04f2c0 con 0x7ff8e4138f30 2023-10-30T12:23:14.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 10 monclient: handle_get_version_reply finishing 1 version 686 2023-10-30T12:23:14.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8e259c640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8c40392f0 0x7ff8c403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8e259c640 10 monclient: get_auth_request con 0x7ff8c40392f0 auth_method 0 2023-10-30T12:23:14.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8e1d9b640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7ff8c4040680 0x7ff8c4042b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8e1d9b640 10 monclient: get_auth_request con 0x7ff8c4040680 auth_method 0 2023-10-30T12:23:14.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 10 client.264387.objecter ms_handle_connect 0x7ff8c4040680 2023-10-30T12:23:14.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8e1d9b640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7ff8c4040680 0x7ff8c4042b20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:14.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8d2ffd640 10 client.264387.objecter ms_handle_connect 0x7ff8c40392f0 2023-10-30T12:23:14.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.566+0000 7ff8e259c640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8c40392f0 0x7ff8c403b7b0 secure :-1 s=READY pgs=4358 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4000f60 tx=0x7ff8d4007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:14.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.567+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== osd.0 v2:172.21.15.50:6816/47190190 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ff8c40431f0 con 0x7ff8c4040680 2023-10-30T12:23:14.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.567+0000 7ff8d2ffd640 10 client.264387.objecter ms_dispatch 0x7ff8e4000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:23:14.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.567+0000 7ff8d2ffd640 10 client.264387.objecter _finish_command 1 = system:0 2023-10-30T12:23:14.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.567+0000 7ff8d2ffd640 20 client.264387.objecter put_session s=0x7ff8c403ffb0 osd=0 4 2023-10-30T12:23:14.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.567+0000 7ff8d2ffd640 15 client.264387.objecter _session_command_op_remove 0 1 2023-10-30T12:23:14.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 10 client.264387.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:23:14.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 15 client.264387.objecter _session_command_op_assign -1 2 2023-10-30T12:23:14.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 client.264387.objecter _get_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 client.264387.objecter put_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 client.264387.objecter _get_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 15 client.264387.objecter _session_command_op_remove -1 2 2023-10-30T12:23:14.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 client.264387.objecter get_session s=0x7ff8c403ffb0 osd=0 3 2023-10-30T12:23:14.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 15 client.264387.objecter _session_command_op_assign 0 2 2023-10-30T12:23:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 client.264387.objecter put_session s=0x7ff8c403ffb0 osd=0 4 2023-10-30T12:23:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 10 client.264387.objecter _send_command 2 2023-10-30T12:23:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 --> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff8e409c550 con 0x7ff8c4040680 2023-10-30T12:23:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8d2ffd640 1 -- 172.21.15.50:0/3617921724 <== osd.0 v2:172.21.15.50:6816/47190190 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff8d80315e0 con 0x7ff8c4040680 2023-10-30T12:23:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8d2ffd640 10 client.264387.objecter ms_dispatch 0x7ff8e4000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:23:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8d2ffd640 10 client.264387.objecter _finish_command 2 = system:0 2023-10-30T12:23:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8d2ffd640 20 client.264387.objecter put_session s=0x7ff8c403ffb0 osd=0 4 2023-10-30T12:23:14.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8d2ffd640 15 client.264387.objecter _session_command_op_remove 0 2 2023-10-30T12:23:14.645 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:14.645 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:23:14.646 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:14.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 10 librados: watch_flush enter 2023-10-30T12:23:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 10 librados: watch_flush exit 2023-10-30T12:23:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 10 client.264387.objecter close_session for osd.0 2023-10-30T12:23:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7ff8c4040680 msgr2=0x7ff8c4042b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:6816/47190190,v1:172.21.15.50:6817/47190190] conn(0x7ff8c4040680 0x7ff8c4042b20 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 client.264387.objecter put_session s=0x7ff8c403ffb0 osd=0 1 2023-10-30T12:23:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 client.264387.objecter shutdown clearing up homeless session... 2023-10-30T12:23:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 10 client.264387.objecter successfully canceled tick 2023-10-30T12:23:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8c40392f0 msgr2=0x7ff8c403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7ff8c40392f0 0x7ff8c403b7b0 secure :-1 s=READY pgs=4358 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4000f60 tx=0x7ff8d4007e10 comp rx=0 tx=0).stop 2023-10-30T12:23:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 10 monclient: shutdown 2023-10-30T12:23:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e4138f30 msgr2=0x7ff8e4137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 --2- 172.21.15.50:0/3617921724 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e4138f30 0x7ff8e4137260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007d60 tx=0x7ff8dc00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e2d9d640 1 -- 172.21.15.50:0/3617921724 reap_dead start 2023-10-30T12:23:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 shutdown_connections 2023-10-30T12:23:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 >> 172.21.15.50:0/3617921724 conn(0x7ff8e40a0480 msgr2=0x7ff8e409a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 shutdown_connections 2023-10-30T12:23:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 -- 172.21.15.50:0/3617921724 wait complete. 2023-10-30T12:23:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.580+0000 7ff8e8983640 1 librados: shutdown 2023-10-30T12:23:14.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T12:23:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.757+0000 7f0664fa5640 10 monclient: get_monmap_and_config 2023-10-30T12:23:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.757+0000 7f0664fa5640 10 monclient: build_initial_monmap 2023-10-30T12:23:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.757+0000 7f0664fa5640 10 monclient: monmap: 2023-10-30T12:23:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:14.758950+0000 2023-10-30T12:23:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:14.758950+0000 2023-10-30T12:23:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:14.763 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.758+0000 7f0664fa5640 1 Processor -- start 2023-10-30T12:23:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.758+0000 7f0664fa5640 1 -- start start 2023-10-30T12:23:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.758+0000 7f0664fa5640 10 monclient: init 2023-10-30T12:23:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-10-30T12:23:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f066009bb70 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009d3f0 0x7f0660137960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient(hunting): picked mon.noname-b con 0x7f066009d3f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f066009c660 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0660098170 con 0x7f066009bb70 2023-10-30T12:23:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066009cf60 con 0x7f066009c660 2023-10-30T12:23:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0660097b20 con 0x7f066009d3f0 2023-10-30T12:23:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:14.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f0664fa5640 10 monclient(hunting): authenticate will time out at 2239.942234s 2023-10-30T12:23:14.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065e59c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f066009bb70 0x7f066009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49628/0 (socket says 172.21.15.50:49628) 2023-10-30T12:23:14.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009d3f0 0x7f0660137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065e59c640 1 -- 172.21.15.50:0/3812721490 learned_addr learned my addr 172.21.15.50:0/3812721490 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): get_auth_request con 0x7f066009d3f0 auth_method 0 2023-10-30T12:23:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065d59a640 1 -- 172.21.15.50:0/3812721490 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159863091 0 0) 0x7f0660098170 con 0x7f066009bb70 2023-10-30T12:23:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient(hunting): handle_auth_done global_id 284144 payload 274 2023-10-30T12:23:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 1 -- 172.21.15.50:0/3812721490 >> v1:172.21.15.118:6789/0 conn(0x7f066009c660 legacy=0x7f0660139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 1 -- 172.21.15.50:0/3812721490 >> v1:172.21.15.50:6789/0 conn(0x7f066009bb70 legacy=0x7f066009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:23:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 1 monclient: found mon.noname-b 2023-10-30T12:23:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065dd9b640 1 -- 172.21.15.50:0/3812721490 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0660097e00 con 0x7f066009d3f0 2023-10-30T12:23:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.759+0000 7f065d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065dd9b640 1 --2- 172.21.15.50:0/3812721490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009d3f0 0x7f0660137960 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f06540097d0 tx=0x7f065402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48c5070fdbce0a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 1 -- 172.21.15.50:0/3812721490 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654031770 con 0x7f066009d3f0 2023-10-30T12:23:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: dump: 2023-10-30T12:23:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.784 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.784 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: _finish_auth 0 2023-10-30T12:23:14.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: _check_auth_tickets 2023-10-30T12:23:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 1 -- 172.21.15.50:0/3812721490 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06540318d0 con 0x7f066009d3f0 2023-10-30T12:23:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f0664fa5640 5 monclient: authenticate success, global_id 284144 2023-10-30T12:23:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f0664fa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 1 -- 172.21.15.50:0/3812721490 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654031c60 con 0x7f066009d3f0 2023-10-30T12:23:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f0664fa5640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:14.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f0664fa5640 4 set_mon_vals no callback set 2023-10-30T12:23:14.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:14.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.760+0000 7f065d59a640 10 monclient: dump: 2023-10-30T12:23:14.788 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.788 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.788 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.788 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.789 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.789 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.789 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.789 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.790 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.790 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 10 monclient: shutdown 2023-10-30T12:23:14.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:14.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 1 -- 172.21.15.50:0/3812721490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009d3f0 msgr2=0x7f0660137960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 1 --2- 172.21.15.50:0/3812721490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009d3f0 0x7f0660137960 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f06540097d0 tx=0x7f065402b810 comp rx=0 tx=0).stop 2023-10-30T12:23:14.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 1 -- 172.21.15.50:0/3812721490 shutdown_connections 2023-10-30T12:23:14.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 1 --2- 172.21.15.50:0/3812721490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009d3f0 0x7f0660137960 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 1 -- 172.21.15.50:0/3812721490 >> 172.21.15.50:0/3812721490 conn(0x7f06600a3980 msgr2=0x7f06600a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:14.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 1 -- 172.21.15.50:0/3812721490 shutdown_connections 2023-10-30T12:23:14.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 1 -- 172.21.15.50:0/3812721490 wait complete. 2023-10-30T12:23:14.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 10 monclient: build_initial_monmap 2023-10-30T12:23:14.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.761+0000 7f0664fa5640 10 monclient: monmap: 2023-10-30T12:23:14.793 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:14.793 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:14.794 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:14.794 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:14.794 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:14.795 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.795 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:14.795 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:14.795 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:14.796 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 librados: starting msgr at 2023-10-30T12:23:14.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 librados: starting objecter 2023-10-30T12:23:14.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 Processor -- start 2023-10-30T12:23:14.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 -- start start 2023-10-30T12:23:14.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 librados: setting wanted keys 2023-10-30T12:23:14.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 librados: calling monclient init 2023-10-30T12:23:14.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient: init 2023-10-30T12:23:14.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:14.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:23:14.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f066009bb70 0x7f0660070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): picked mon.noname-b con 0x7f066009bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:14.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009c660 0x7f06600946a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): picked mon.noname-c con 0x7f066009c660 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:14.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f066009d3f0 0x7f066006bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): picked mon.noname-a con 0x7f066009d3f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:14.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660071960 con 0x7f066009d3f0 2023-10-30T12:23:14.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066008f8a0 con 0x7f066009bb70 2023-10-30T12:23:14.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:14.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066008fb80 con 0x7f066009c660 2023-10-30T12:23:14.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:14.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f0664fa5640 10 monclient(hunting): authenticate will time out at 2239.945814s 2023-10-30T12:23:14.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009c660 0x7f06600946a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f066009d3f0 0x7f066006bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065dd9b640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009c660 0x7f06600946a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60700/0 (socket says 172.21.15.50:60700) 2023-10-30T12:23:14.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065dd9b640 1 -- 172.21.15.50:0/2119935041 learned_addr learned my addr 172.21.15.50:0/2119935041 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:14.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065dd9b640 10 monclient(hunting): get_auth_request con 0x7f066009c660 auth_method 0 2023-10-30T12:23:14.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065dd9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065ed9d640 10 monclient(hunting): get_auth_request con 0x7f066009d3f0 auth_method 0 2023-10-30T12:23:14.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.762+0000 7f065e59c640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f066009bb70 0x7f0660070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065e59c640 10 monclient(hunting): get_auth_request con 0x7f066009bb70 auth_method 0 2023-10-30T12:23:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:14.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:14.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 10 monclient(hunting): handle_auth_done global_id 284147 payload 824 2023-10-30T12:23:14.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 1 -- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f066009bb70 msgr2=0x7f0660070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f066009bb70 0x7f0660070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 1 -- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f066009d3f0 msgr2=0x7f066006bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f066009d3f0 0x7f066006bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 10 monclient: _finish_hunting 0 2023-10-30T12:23:14.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 1 monclient: found mon.noname-c 2023-10-30T12:23:14.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:14.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:14.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 1 -- 172.21.15.50:0/2119935041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654009400 con 0x7f066009c660 2023-10-30T12:23:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065ed9d640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f066009d3f0 0x7f066006bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065dd9b640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009c660 0x7f06600946a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0654034040 tx=0x7f06540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 client.?.objecter ms_handle_connect 0x7f066009c660 2023-10-30T12:23:14.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:14.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 1 -- 172.21.15.50:0/2119935041 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654032950 con 0x7f066009c660 2023-10-30T12:23:14.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:14.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: dump: 2023-10-30T12:23:14.814 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.814 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.814 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.816 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.816 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.816 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: _finish_auth 0 2023-10-30T12:23:14.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:14.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:14.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 1 -- 172.21.15.50:0/2119935041 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654032ab0 con 0x7f066009c660 2023-10-30T12:23:14.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 5 monclient: authenticate success, global_id 284147 2023-10-30T12:23:14.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 10 monclient: _renew_subs 2023-10-30T12:23:14.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:14.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06600976e0 con 0x7f066009c660 2023-10-30T12:23:14.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:14.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 1 -- 172.21.15.50:0/2119935041 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654032e60 con 0x7f066009c660 2023-10-30T12:23:14.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:14.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:14.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f064effd640 10 monclient: dump: 2023-10-30T12:23:14.820 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:14.820 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:14.820 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.821 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:14.821 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:14.821 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:14.822 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f065cd99640 4 set_mon_vals no callback set 2023-10-30T12:23:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 10 client.284147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 10 monclient: _renew_subs 2023-10-30T12:23:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:14.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06601cb030 con 0x7f066009c660 2023-10-30T12:23:14.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.763+0000 7f0664fa5640 1 librados: init done 2023-10-30T12:23:14.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f064effd640 1 -- 172.21.15.50:0/2119935041 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f065404e450 con 0x7f066009c660 2023-10-30T12:23:14.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f0664fa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:14.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f0664fa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:14.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f0664fa5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:14.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066004e160 con 0x7f066009c660 2023-10-30T12:23:14.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f064effd640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06400392f0 0x7f064003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:14.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f064effd640 1 -- 172.21.15.50:0/2119935041 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f065406cb00 con 0x7f066009c660 2023-10-30T12:23:14.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f064effd640 10 client.284147.objecter ms_dispatch 0x7f0660000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:14.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f064effd640 3 client.284147.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:14.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.764+0000 7f064effd640 3 client.284147.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:14.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.765+0000 7f064effd640 20 client.284147.objecter dump_active .. 0 homeless 2023-10-30T12:23:14.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.765+0000 7f065e59c640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06400392f0 0x7f064003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:14.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.765+0000 7f065e59c640 10 monclient: get_auth_request con 0x7f06400392f0 auth_method 0 2023-10-30T12:23:14.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.765+0000 7f064effd640 10 client.284147.objecter ms_handle_connect 0x7f06400392f0 2023-10-30T12:23:14.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.765+0000 7f065e59c640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06400392f0 0x7f064003b7b0 secure :-1 s=READY pgs=4359 cs=0 l=1 rev1=1 crypto rx=0x7f0650000c00 tx=0x7f0650009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:14.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.769+0000 7f064effd640 1 -- 172.21.15.50:0/2119935041 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f065404c940 con 0x7f066009c660 2023-10-30T12:23:14.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.769+0000 7f064effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:14.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.769+0000 7f064effd640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:14.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.944+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f0660097560 con 0x7f06400392f0 2023-10-30T12:23:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.945+0000 7f064effd640 1 -- 172.21.15.50:0/2119935041 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f065000f7d0 con 0x7f06400392f0 2023-10-30T12:23:14.947 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T12:23:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 10 librados: watch_flush enter 2023-10-30T12:23:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 10 librados: watch_flush exit 2023-10-30T12:23:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 20 client.284147.objecter shutdown clearing up homeless session... 2023-10-30T12:23:14.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 10 client.284147.objecter successfully canceled tick 2023-10-30T12:23:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06400392f0 msgr2=0x7f064003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06400392f0 0x7f064003b7b0 secure :-1 s=READY pgs=4359 cs=0 l=1 rev1=1 crypto rx=0x7f0650000c00 tx=0x7f0650009040 comp rx=0 tx=0).stop 2023-10-30T12:23:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 10 monclient: shutdown 2023-10-30T12:23:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009c660 msgr2=0x7f06600946a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009c660 0x7f06600946a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0654034040 tx=0x7f06540046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 shutdown_connections 2023-10-30T12:23:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06400392f0 0x7f064003b7b0 unknown :-1 s=CLOSED pgs=4359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f066009d3f0 0x7f066006bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f066009c660 0x7f06600946a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 --2- 172.21.15.50:0/2119935041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f066009bb70 0x7f0660070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 >> 172.21.15.50:0/2119935041 conn(0x7f06600a3980 msgr2=0x7f0660096630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 shutdown_connections 2023-10-30T12:23:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 -- 172.21.15.50:0/2119935041 wait complete. 2023-10-30T12:23:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:14.947+0000 7f0664fa5640 1 librados: shutdown 2023-10-30T12:23:14.959 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T12:23:14.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:23:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.066+0000 7f5a20497640 10 monclient: get_monmap_and_config 2023-10-30T12:23:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.066+0000 7f5a20497640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.066+0000 7f5a20497640 10 monclient: monmap: 2023-10-30T12:23:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:15.067930+0000 2023-10-30T12:23:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:15.067930+0000 2023-10-30T12:23:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:15.072 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.067+0000 7f5a20497640 1 Processor -- start 2023-10-30T12:23:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.067+0000 7f5a20497640 1 -- start start 2023-10-30T12:23:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.067+0000 7f5a20497640 10 monclient: init 2023-10-30T12:23:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.067+0000 7f5a20497640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.067+0000 7f5a20497640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-10-30T12:23:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.067+0000 7f5a20497640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a180965d0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a1809d210 0x7f5a181389f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 10 monclient(hunting): picked mon.noname-b con 0x7f5a1809d210 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a18138f30 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180970c0 con 0x7f5a180965d0 2023-10-30T12:23:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a18099c30 con 0x7f5a18138f30 2023-10-30T12:23:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1809bf90 con 0x7f5a1809d210 2023-10-30T12:23:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a20497640 10 monclient(hunting): authenticate will time out at 2240.251090s 2023-10-30T12:23:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a1809d210 0x7f5a181389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1e233640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5a180965d0 0x7f5a180969b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49636/0 (socket says 172.21.15.50:49636) 2023-10-30T12:23:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1e233640 1 -- 172.21.15.50:0/1613349202 learned_addr learned my addr 172.21.15.50:0/1613349202 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): get_auth_request con 0x7f5a1809d210 auth_method 0 2023-10-30T12:23:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient(hunting): handle_auth_done global_id 284150 payload 274 2023-10-30T12:23:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 1 -- 172.21.15.50:0/1613349202 >> v1:172.21.15.118:6789/0 conn(0x7f5a18138f30 legacy=0x7f5a1813b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 1 -- 172.21.15.50:0/1613349202 >> v1:172.21.15.50:6789/0 conn(0x7f5a180965d0 legacy=0x7f5a180969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1d231640 1 -- 172.21.15.50:0/1613349202 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846628672 0 0) 0x7f5a180970c0 con 0x7f5a180965d0 2023-10-30T12:23:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 1 monclient: found mon.noname-b 2023-10-30T12:23:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1da32640 1 -- 172.21.15.50:0/1613349202 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1809c270 con 0x7f5a1809d210 2023-10-30T12:23:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.068+0000 7f5a1d231640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1da32640 1 --2- 172.21.15.50:0/1613349202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a1809d210 0x7f5a181389f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5a080097d0 tx=0x7f5a0802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea834a38f92a8b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 1 -- 172.21.15.50:0/1613349202 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a080316f0 con 0x7f5a1809d210 2023-10-30T12:23:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: dump: 2023-10-30T12:23:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.092 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 1 -- 172.21.15.50:0/1613349202 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08031850 con 0x7f5a1809d210 2023-10-30T12:23:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 5 monclient: authenticate success, global_id 284150 2023-10-30T12:23:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 1 -- 172.21.15.50:0/1613349202 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08031be0 con 0x7f5a1809d210 2023-10-30T12:23:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 4 set_mon_vals no callback set 2023-10-30T12:23:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a1d231640 10 monclient: dump: 2023-10-30T12:23:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.098 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 10 monclient: shutdown 2023-10-30T12:23:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 1 -- 172.21.15.50:0/1613349202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a1809d210 msgr2=0x7f5a181389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 1 --2- 172.21.15.50:0/1613349202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a1809d210 0x7f5a181389f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5a080097d0 tx=0x7f5a0802b790 comp rx=0 tx=0).stop 2023-10-30T12:23:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 1 -- 172.21.15.50:0/1613349202 shutdown_connections 2023-10-30T12:23:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 1 --2- 172.21.15.50:0/1613349202 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a1809d210 0x7f5a181389f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.069+0000 7f5a20497640 1 -- 172.21.15.50:0/1613349202 >> 172.21.15.50:0/1613349202 conn(0x7f5a180a0480 msgr2=0x7f5a180a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 -- 172.21.15.50:0/1613349202 shutdown_connections 2023-10-30T12:23:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 -- 172.21.15.50:0/1613349202 wait complete. 2023-10-30T12:23:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 10 monclient: monmap: 2023-10-30T12:23:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:15.104 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 librados: starting msgr at 2023-10-30T12:23:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 librados: starting objecter 2023-10-30T12:23:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 Processor -- start 2023-10-30T12:23:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 -- start start 2023-10-30T12:23:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 librados: setting wanted keys 2023-10-30T12:23:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 1 librados: calling monclient init 2023-10-30T12:23:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.070+0000 7f5a20497640 10 monclient: init 2023-10-30T12:23:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:23:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a180965d0 0x7f5a18137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): picked mon.noname-b con 0x7f5a180965d0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a1809d210 0x7f5a181380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): picked mon.noname-a con 0x7f5a1809d210 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a18138f30 0x7f5a1808da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): picked mon.noname-c con 0x7f5a18138f30 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18091470 con 0x7f5a1809d210 2023-10-30T12:23:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1809a000 con 0x7f5a180965d0 2023-10-30T12:23:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18090020 con 0x7f5a18138f30 2023-10-30T12:23:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a20497640 10 monclient(hunting): authenticate will time out at 2240.254398s 2023-10-30T12:23:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a1809d210 0x7f5a181380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a18138f30 0x7f5a1808da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a1809d210 0x7f5a181380c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59838/0 (socket says 172.21.15.50:59838) 2023-10-30T12:23:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 1 -- 172.21.15.50:0/2221048973 learned_addr learned my addr 172.21.15.50:0/2221048973 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a18138f30 0x7f5a1808da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60720/0 (socket says 172.21.15.50:60720) 2023-10-30T12:23:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 10 monclient(hunting): get_auth_request con 0x7f5a1809d210 auth_method 0 2023-10-30T12:23:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): get_auth_request con 0x7f5a18138f30 auth_method 0 2023-10-30T12:23:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1e233640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a180965d0 0x7f5a18137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1e233640 10 monclient(hunting): get_auth_request con 0x7f5a180965d0 auth_method 0 2023-10-30T12:23:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1e233640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1e233640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1e233640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient(hunting): handle_auth_done global_id 284153 payload 824 2023-10-30T12:23:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 -- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a180965d0 msgr2=0x7f5a18137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5a180965d0 0x7f5a18137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 -- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a1809d210 msgr2=0x7f5a181380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a1809d210 0x7f5a181380c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 monclient: found mon.noname-c 2023-10-30T12:23:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1ea34640 1 -- 172.21.15.50:0/2221048973 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a08009400 con 0x7f5a18138f30 2023-10-30T12:23:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.071+0000 7f5a1da32640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a1809d210 0x7f5a181380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a1ea34640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a18138f30 0x7f5a1808da10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a18074140 tx=0x7f5a1000cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 client.?.objecter ms_handle_connect 0x7f5a18138f30 2023-10-30T12:23:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000a840 con 0x7f5a18138f30 2023-10-30T12:23:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: dump: 2023-10-30T12:23:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.122 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.124 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 5 monclient: authenticate success, global_id 284153 2023-10-30T12:23:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1000a9c0 con 0x7f5a18138f30 2023-10-30T12:23:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000ad70 con 0x7f5a18138f30 2023-10-30T12:23:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 10 monclient: _renew_subs 2023-10-30T12:23:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a18091190 con 0x7f5a18138f30 2023-10-30T12:23:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a1ca30640 4 set_mon_vals no callback set 2023-10-30T12:23:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a0effd640 10 monclient: dump: 2023-10-30T12:23:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.129 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.129 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.129 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.130 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.130 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.130 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.130 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.131 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 10 client.284153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 10 monclient: _renew_subs 2023-10-30T12:23:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a18138600 con 0x7f5a18138f30 2023-10-30T12:23:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.072+0000 7f5a20497640 1 librados: init done 2023-10-30T12:23:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a20497640 10 client.284153.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a20497640 15 client.284153.objecter _session_command_op_assign -1 1 2023-10-30T12:23:15.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a20497640 20 client.284153.objecter _get_session osd=-1 returning homeless 2023-10-30T12:23:15.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a20497640 10 client.284153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:15.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a20497640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5a1809ba70 con 0x7f5a18138f30 2023-10-30T12:23:15.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5a10012420 con 0x7f5a18138f30 2023-10-30T12:23:15.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59fc0392f0 0x7f59fc03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f5a1004a970 con 0x7f5a18138f30 2023-10-30T12:23:15.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 10 client.284153.objecter ms_dispatch 0x7f5a18000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:15.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 3 client.284153.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:15.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 3 client.284153.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:15.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 10 client.284153.objecter checking command 1 2023-10-30T12:23:15.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f59fc040680 0x7f59fc042b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 20 client.284153.objecter _get_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 20 client.284153.objecter put_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 15 client.284153.objecter _session_command_op_remove -1 1 2023-10-30T12:23:15.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 20 client.284153.objecter _get_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 20 client.284153.objecter get_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 15 client.284153.objecter _session_command_op_assign 1 1 2023-10-30T12:23:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 20 client.284153.objecter put_session s=0x7f59fc03ffb0 osd=1 4 2023-10-30T12:23:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 10 client.284153.objecter _send_command 1 2023-10-30T12:23:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 --> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59fc0431f0 con 0x7f59fc040680 2023-10-30T12:23:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 20 client.284153.objecter dump_active .. 0 homeless 2023-10-30T12:23:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=686) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a1004f2c0 con 0x7f5a18138f30 2023-10-30T12:23:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a0effd640 10 monclient: handle_get_version_reply finishing 1 version 686 2023-10-30T12:23:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a1e233640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59fc0392f0 0x7f59fc03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a1e233640 10 monclient: get_auth_request con 0x7f59fc0392f0 auth_method 0 2023-10-30T12:23:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a1da32640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f59fc040680 0x7f59fc042b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.073+0000 7f5a1da32640 10 monclient: get_auth_request con 0x7f59fc040680 auth_method 0 2023-10-30T12:23:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a0effd640 10 client.284153.objecter ms_handle_connect 0x7f59fc040680 2023-10-30T12:23:15.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a1da32640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f59fc040680 0x7f59fc042b20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a0effd640 10 client.284153.objecter ms_handle_connect 0x7f59fc0392f0 2023-10-30T12:23:15.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a1e233640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59fc0392f0 0x7f59fc03b7b0 secure :-1 s=READY pgs=4360 cs=0 l=1 rev1=1 crypto rx=0x7f5a14000f60 tx=0x7f5a14007b50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== osd.1 v2:172.21.15.50:6808/659398918 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f59fc0431f0 con 0x7f59fc040680 2023-10-30T12:23:15.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a0effd640 10 client.284153.objecter ms_dispatch 0x7f5a18000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:23:15.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a0effd640 10 client.284153.objecter _finish_command 1 = system:0 2023-10-30T12:23:15.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a0effd640 20 client.284153.objecter put_session s=0x7f59fc03ffb0 osd=1 4 2023-10-30T12:23:15.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.074+0000 7f5a0effd640 15 client.284153.objecter _session_command_op_remove 1 1 2023-10-30T12:23:15.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 10 client.284153.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:23:15.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 15 client.284153.objecter _session_command_op_assign -1 2 2023-10-30T12:23:15.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 20 client.284153.objecter _get_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 20 client.284153.objecter put_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 20 client.284153.objecter _get_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 15 client.284153.objecter _session_command_op_remove -1 2 2023-10-30T12:23:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 20 client.284153.objecter get_session s=0x7f59fc03ffb0 osd=1 3 2023-10-30T12:23:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 15 client.284153.objecter _session_command_op_assign 1 2 2023-10-30T12:23:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 20 client.284153.objecter put_session s=0x7f59fc03ffb0 osd=1 4 2023-10-30T12:23:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 10 client.284153.objecter _send_command 2 2023-10-30T12:23:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 --> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5a1809c550 con 0x7f59fc040680 2023-10-30T12:23:15.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a0effd640 1 -- 172.21.15.50:0/2221048973 <== osd.1 v2:172.21.15.50:6808/659398918 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5a08031690 con 0x7f59fc040680 2023-10-30T12:23:15.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a0effd640 10 client.284153.objecter ms_dispatch 0x7f5a18000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:23:15.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a0effd640 10 client.284153.objecter _finish_command 2 = system:0 2023-10-30T12:23:15.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a0effd640 20 client.284153.objecter put_session s=0x7f59fc03ffb0 osd=1 4 2023-10-30T12:23:15.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a0effd640 15 client.284153.objecter _session_command_op_remove 1 2 2023-10-30T12:23:15.148 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:15.148 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:23:15.149 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:15.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 10 librados: watch_flush enter 2023-10-30T12:23:15.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 10 librados: watch_flush exit 2023-10-30T12:23:15.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 10 client.284153.objecter close_session for osd.1 2023-10-30T12:23:15.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f59fc040680 msgr2=0x7f59fc042b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:6808/659398918,v1:172.21.15.50:6809/659398918] conn(0x7f59fc040680 0x7f59fc042b20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 20 client.284153.objecter put_session s=0x7f59fc03ffb0 osd=1 1 2023-10-30T12:23:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 20 client.284153.objecter shutdown clearing up homeless session... 2023-10-30T12:23:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.087+0000 7f5a20497640 10 client.284153.objecter successfully canceled tick 2023-10-30T12:23:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59fc0392f0 msgr2=0x7f59fc03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f59fc0392f0 0x7f59fc03b7b0 secure :-1 s=READY pgs=4360 cs=0 l=1 rev1=1 crypto rx=0x7f5a14000f60 tx=0x7f5a14007b50 comp rx=0 tx=0).stop 2023-10-30T12:23:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 10 monclient: shutdown 2023-10-30T12:23:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a18138f30 msgr2=0x7f5a1808da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 --2- 172.21.15.50:0/2221048973 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a18138f30 0x7f5a1808da10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a18074140 tx=0x7f5a1000cbe0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a1ea34640 1 -- 172.21.15.50:0/2221048973 reap_dead start 2023-10-30T12:23:15.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 shutdown_connections 2023-10-30T12:23:15.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 >> 172.21.15.50:0/2221048973 conn(0x7f5a180a0480 msgr2=0x7f5a1809b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 shutdown_connections 2023-10-30T12:23:15.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 -- 172.21.15.50:0/2221048973 wait complete. 2023-10-30T12:23:15.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.088+0000 7f5a20497640 1 librados: shutdown 2023-10-30T12:23:15.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T12:23:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient: get_monmap_and_config 2023-10-30T12:23:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient: monmap: 2023-10-30T12:23:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:15.262376+0000 2023-10-30T12:23:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:15.262376+0000 2023-10-30T12:23:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:15.266 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 1 Processor -- start 2023-10-30T12:23:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 1 -- start start 2023-10-30T12:23:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient: init 2023-10-30T12:23:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-10-30T12:23:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa000132970 0x7fa000132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient(hunting): picked mon.noname-a con 0x7fa000132970 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0000a03d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa000070ee0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000a17d0 con 0x7fa000070ee0 2023-10-30T12:23:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000a1510 con 0x7fa0000a03d0 2023-10-30T12:23:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa00009f680 con 0x7fa000132970 2023-10-30T12:23:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa007040640 10 monclient(hunting): authenticate will time out at 2240.445538s 2023-10-30T12:23:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa000132970 0x7fa000132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa0055dd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa000070ee0 0x7fa00013db00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49644/0 (socket says 172.21.15.50:49644) 2023-10-30T12:23:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa000132970 0x7fa000132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59844/0 (socket says 172.21.15.50:59844) 2023-10-30T12:23:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa0055dd640 1 -- 172.21.15.50:0/1774562090 learned_addr learned my addr 172.21.15.50:0/1774562090 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): get_auth_request con 0x7fa000132970 auth_method 0 2023-10-30T12:23:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 10 monclient(hunting): handle_auth_done global_id 264402 payload 274 2023-10-30T12:23:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.262+0000 7fa004ddc640 1 -- 172.21.15.50:0/1774562090 >> v1:172.21.15.118:6789/0 conn(0x7fa0000a03d0 legacy=0x7fa0000707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 1 -- 172.21.15.50:0/1774562090 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55756043 0 0) 0x7fa0000a17d0 con 0x7fa000070ee0 2023-10-30T12:23:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa004ddc640 1 -- 172.21.15.50:0/1774562090 >> v1:172.21.15.50:6789/0 conn(0x7fa000070ee0 legacy=0x7fa00013db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa004ddc640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa004ddc640 1 monclient: found mon.noname-a 2023-10-30T12:23:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa004ddc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa004ddc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa004ddc640 1 -- 172.21.15.50:0/1774562090 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00009f960 con 0x7fa000132970 2023-10-30T12:23:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa004ddc640 1 --2- 172.21.15.50:0/1774562090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000132970 0x7fa000132d50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00097a0 tx=0x7f9ff002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f17042783ee21f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 1 -- 172.21.15.50:0/1774562090 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff002e070 con 0x7fa000132970 2023-10-30T12:23:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: dump: 2023-10-30T12:23:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.287 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa007040640 5 monclient: authenticate success, global_id 264402 2023-10-30T12:23:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa007040640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 1 -- 172.21.15.50:0/1774562090 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff00315a0 con 0x7fa000132970 2023-10-30T12:23:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa007040640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7fa007040640 4 set_mon_vals no callback set 2023-10-30T12:23:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 1 -- 172.21.15.50:0/1774562090 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0031930 con 0x7fa000132970 2023-10-30T12:23:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.263+0000 7f9fff7fe640 10 monclient: dump: 2023-10-30T12:23:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.293 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 10 monclient: shutdown 2023-10-30T12:23:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 1 -- 172.21.15.50:0/1774562090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000132970 msgr2=0x7fa000132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 1 --2- 172.21.15.50:0/1774562090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000132970 0x7fa000132d50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00097a0 tx=0x7f9ff002b840 comp rx=0 tx=0).stop 2023-10-30T12:23:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 1 -- 172.21.15.50:0/1774562090 shutdown_connections 2023-10-30T12:23:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 1 --2- 172.21.15.50:0/1774562090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000132970 0x7fa000132d50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 1 -- 172.21.15.50:0/1774562090 >> 172.21.15.50:0/1774562090 conn(0x7fa0000a4800 msgr2=0x7fa0000a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 1 -- 172.21.15.50:0/1774562090 shutdown_connections 2023-10-30T12:23:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.264+0000 7fa007040640 1 -- 172.21.15.50:0/1774562090 wait complete. 2023-10-30T12:23:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient: monmap: 2023-10-30T12:23:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:15.298 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:15.299 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 librados: starting msgr at 2023-10-30T12:23:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 librados: starting objecter 2023-10-30T12:23:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 Processor -- start 2023-10-30T12:23:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 -- start start 2023-10-30T12:23:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 librados: setting wanted keys 2023-10-30T12:23:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 librados: calling monclient init 2023-10-30T12:23:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient: init 2023-10-30T12:23:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:23:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000070ee0 0x7fa0001c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): picked mon.noname-a con 0x7fa000070ee0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa000132970 0x7fa00012fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): picked mon.noname-c con 0x7fa000132970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa000130100 0x7fa0001ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): picked mon.noname-b con 0x7fa000130100 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00011e460 con 0x7fa000070ee0 2023-10-30T12:23:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00009d830 con 0x7fa000130100 2023-10-30T12:23:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0001c78c0 con 0x7fa000132970 2023-10-30T12:23:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.265+0000 7fa007040640 10 monclient(hunting): authenticate will time out at 2240.448996s 2023-10-30T12:23:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000070ee0 0x7fa0001c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa000132970 0x7fa00012fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000070ee0 0x7fa0001c6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59856/0 (socket says 172.21.15.50:59856) 2023-10-30T12:23:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): get_auth_request con 0x7fa000132970 auth_method 0 2023-10-30T12:23:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 1 -- 172.21.15.50:0/956114691 learned_addr learned my addr 172.21.15.50:0/956114691 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 10 monclient(hunting): get_auth_request con 0x7fa000070ee0 auth_method 0 2023-10-30T12:23:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa0055dd640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa000130100 0x7fa0001ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa0055dd640 10 monclient(hunting): get_auth_request con 0x7fa000130100 auth_method 0 2023-10-30T12:23:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa0055dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa0055dd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa0055dd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient(hunting): handle_auth_done global_id 284156 payload 824 2023-10-30T12:23:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 -- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa000130100 msgr2=0x7fa0001ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa000130100 0x7fa0001ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 -- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000070ee0 msgr2=0x7fa0001c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000070ee0 0x7fa0001c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 monclient: found mon.noname-c 2023-10-30T12:23:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 -- 172.21.15.50:0/956114691 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff0009400 con 0x7fa000132970 2023-10-30T12:23:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa004ddc640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000070ee0 0x7fa0001c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffffff640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa000132970 0x7fa00012fbc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa00006c370 tx=0x7f9ff400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 client.?.objecter ms_handle_connect 0x7fa000132970 2023-10-30T12:23:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 1 -- 172.21.15.50:0/956114691 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40052d0 con 0x7fa000132970 2023-10-30T12:23:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: dump: 2023-10-30T12:23:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.318 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.318 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.318 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.319 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 5 monclient: authenticate success, global_id 284156 2023-10-30T12:23:15.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 1 -- 172.21.15.50:0/956114691 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff4005430 con 0x7fa000132970 2023-10-30T12:23:15.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 10 monclient: _renew_subs 2023-10-30T12:23:15.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00011cdf0 con 0x7fa000132970 2023-10-30T12:23:15.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 1 -- 172.21.15.50:0/956114691 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4003810 con 0x7fa000132970 2023-10-30T12:23:15.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:15.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffd7fa640 10 monclient: dump: 2023-10-30T12:23:15.322 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.325 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 10 client.284156.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 10 monclient: _renew_subs 2023-10-30T12:23:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7f9ffeffd640 4 set_mon_vals no callback set 2023-10-30T12:23:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00011dda0 con 0x7fa000132970 2023-10-30T12:23:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.266+0000 7fa007040640 1 librados: init done 2023-10-30T12:23:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.267+0000 7f9ffd7fa640 1 -- 172.21.15.50:0/956114691 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9ff4003970 con 0x7fa000132970 2023-10-30T12:23:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.267+0000 7fa007040640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.267+0000 7fa007040640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.267+0000 7fa007040640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7f9ffd7fa640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9fec0392f0 0x7f9fec03b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.267+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00011df30 con 0x7fa000132970 2023-10-30T12:23:15.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7f9ffd7fa640 1 -- 172.21.15.50:0/956114691 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f9ff4049e90 con 0x7fa000132970 2023-10-30T12:23:15.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7f9ffd7fa640 10 client.284156.objecter ms_dispatch 0x7fa000000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:15.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7f9ffd7fa640 3 client.284156.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:15.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7f9ffd7fa640 3 client.284156.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:15.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7f9ffd7fa640 20 client.284156.objecter dump_active .. 0 homeless 2023-10-30T12:23:15.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7fa004ddc640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9fec0392f0 0x7f9fec03b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7fa004ddc640 10 monclient: get_auth_request con 0x7f9fec0392f0 auth_method 0 2023-10-30T12:23:15.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7f9ffd7fa640 10 client.284156.objecter ms_handle_connect 0x7f9fec0392f0 2023-10-30T12:23:15.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.268+0000 7fa004ddc640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9fec0392f0 0x7f9fec03b7b0 secure :-1 s=READY pgs=4361 cs=0 l=1 rev1=1 crypto rx=0x7f9ff002b880 tx=0x7f9ff003e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.272+0000 7f9ffd7fa640 1 -- 172.21.15.50:0/956114691 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9ff4019c50 con 0x7fa000132970 2023-10-30T12:23:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.272+0000 7f9ffd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.272+0000 7f9ffd7fa640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.447+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa000133290 con 0x7f9fec0392f0 2023-10-30T12:23:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.448+0000 7f9ffd7fa640 1 -- 172.21.15.50:0/956114691 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9ff0031710 con 0x7f9fec0392f0 2023-10-30T12:23:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T12:23:15.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 10 librados: watch_flush enter 2023-10-30T12:23:15.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 10 librados: watch_flush exit 2023-10-30T12:23:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 20 client.284156.objecter shutdown clearing up homeless session... 2023-10-30T12:23:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 10 client.284156.objecter successfully canceled tick 2023-10-30T12:23:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9fec0392f0 msgr2=0x7f9fec03b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9fec0392f0 0x7f9fec03b7b0 secure :-1 s=READY pgs=4361 cs=0 l=1 rev1=1 crypto rx=0x7f9ff002b880 tx=0x7f9ff003e040 comp rx=0 tx=0).stop 2023-10-30T12:23:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 10 monclient: shutdown 2023-10-30T12:23:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa000132970 msgr2=0x7fa00012fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa000132970 0x7fa00012fbc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa00006c370 tx=0x7f9ff400c820 comp rx=0 tx=0).stop 2023-10-30T12:23:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 shutdown_connections 2023-10-30T12:23:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f9fec0392f0 0x7f9fec03b7b0 unknown :-1 s=CLOSED pgs=4361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa000130100 0x7fa0001ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa000132970 0x7fa00012fbc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 --2- 172.21.15.50:0/956114691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa000070ee0 0x7fa0001c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 >> 172.21.15.50:0/956114691 conn(0x7fa0000a4800 msgr2=0x7fa000131ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 shutdown_connections 2023-10-30T12:23:15.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 -- 172.21.15.50:0/956114691 wait complete. 2023-10-30T12:23:15.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.451+0000 7fa007040640 1 librados: shutdown 2023-10-30T12:23:15.462 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T12:23:15.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:23:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.570+0000 7f091b6a2640 10 monclient: get_monmap_and_config 2023-10-30T12:23:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.570+0000 7f091b6a2640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.570+0000 7f091b6a2640 10 monclient: monmap: 2023-10-30T12:23:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:15.571649+0000 2023-10-30T12:23:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:15.571649+0000 2023-10-30T12:23:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:15.576 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:15.576 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 1 Processor -- start 2023-10-30T12:23:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 1 -- start start 2023-10-30T12:23:15.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient: init 2023-10-30T12:23:15.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-10-30T12:23:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0914099ee0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141347b0 0x7f0914136ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient(hunting): picked mon.noname-b con 0x7f09141347b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f09141370e0 0x7f09141394d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient(hunting): picked mon.noname-a con 0x7f09141370e0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091409cdd0 con 0x7f0914099ee0 2023-10-30T12:23:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091409a580 con 0x7f09141370e0 2023-10-30T12:23:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09140723f0 con 0x7f09141347b0 2023-10-30T12:23:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091b6a2640 10 monclient(hunting): authenticate will time out at 2240.754882s 2023-10-30T12:23:15.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091943e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0914099ee0 0x7f0914097d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49656/0 (socket says 172.21.15.50:49656) 2023-10-30T12:23:15.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f091943e640 1 -- 172.21.15.50:0/856658084 learned_addr learned my addr 172.21.15.50:0/856658084 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:15.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f0919c3f640 1 --2- 172.21.15.50:0/856658084 >> v2:172.21.15.50:3300/0 conn(0x7f09141370e0 0x7f09141394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.571+0000 7f0918c3d640 1 --2- 172.21.15.50:0/856658084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141347b0 0x7f0914136ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): get_auth_request con 0x7f09141347b0 auth_method 0 2023-10-30T12:23:15.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 10 monclient(hunting): get_auth_request con 0x7f09141370e0 auth_method 0 2023-10-30T12:23:15.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient(hunting): handle_auth_done global_id 284159 payload 274 2023-10-30T12:23:15.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 1 -- 172.21.15.50:0/856658084 >> v2:172.21.15.50:3300/0 conn(0x7f09141370e0 msgr2=0x7f09141394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:15.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 1 --2- 172.21.15.50:0/856658084 >> v2:172.21.15.50:3300/0 conn(0x7f09141370e0 0x7f09141394d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 1 -- 172.21.15.50:0/856658084 >> v1:172.21.15.50:6789/0 conn(0x7f0914099ee0 legacy=0x7f0914097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 1 monclient: found mon.noname-b 2023-10-30T12:23:15.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0918c3d640 1 -- 172.21.15.50:0/856658084 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091409a2c0 con 0x7f09141347b0 2023-10-30T12:23:15.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f090bfff640 1 -- 172.21.15.50:0/856658084 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398991384 0 0) 0x7f091409cdd0 con 0x7f0914099ee0 2023-10-30T12:23:15.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f090bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:15.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.572+0000 7f0919c3f640 1 --2- 172.21.15.50:0/856658084 >> v2:172.21.15.50:3300/0 conn(0x7f09141370e0 0x7f09141394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:15.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f0918c3d640 1 --2- 172.21.15.50:0/856658084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141347b0 0x7f0914136ba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09040097a0 tx=0x7f090402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee26190855e2ab8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 1 -- 172.21.15.50:0/856658084 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090402f050 con 0x7f09141347b0 2023-10-30T12:23:15.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:15.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: dump: 2023-10-30T12:23:15.598 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.599 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.599 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.599 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.599 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.600 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.600 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.600 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.601 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.601 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 5 monclient: authenticate success, global_id 284159 2023-10-30T12:23:15.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:15.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 1 -- 172.21.15.50:0/856658084 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0904031700 con 0x7f09141347b0 2023-10-30T12:23:15.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:15.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 4 set_mon_vals no callback set 2023-10-30T12:23:15.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 1 -- 172.21.15.50:0/856658084 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904031a90 con 0x7f09141347b0 2023-10-30T12:23:15.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:15.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f090bfff640 10 monclient: dump: 2023-10-30T12:23:15.605 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.605 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.605 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.605 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.606 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.606 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.606 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.607 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.607 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.607 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 10 monclient: shutdown 2023-10-30T12:23:15.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 -- 172.21.15.50:0/856658084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141347b0 msgr2=0x7f0914136ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 --2- 172.21.15.50:0/856658084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141347b0 0x7f0914136ba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09040097a0 tx=0x7f090402b7a0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 -- 172.21.15.50:0/856658084 shutdown_connections 2023-10-30T12:23:15.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 --2- 172.21.15.50:0/856658084 >> v2:172.21.15.50:3300/0 conn(0x7f09141370e0 0x7f09141394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 --2- 172.21.15.50:0/856658084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141347b0 0x7f0914136ba0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 -- 172.21.15.50:0/856658084 >> 172.21.15.50:0/856658084 conn(0x7f09140a0480 msgr2=0x7f09140a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 -- 172.21.15.50:0/856658084 shutdown_connections 2023-10-30T12:23:15.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.573+0000 7f091b6a2640 1 -- 172.21.15.50:0/856658084 wait complete. 2023-10-30T12:23:15.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 10 monclient: monmap: 2023-10-30T12:23:15.611 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.611 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:15.611 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:15.612 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:15.612 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.612 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.612 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:15.613 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:15.613 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:15.613 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 1 librados: starting msgr at 2023-10-30T12:23:15.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 1 librados: starting objecter 2023-10-30T12:23:15.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 1 Processor -- start 2023-10-30T12:23:15.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 1 -- start start 2023-10-30T12:23:15.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 1 librados: setting wanted keys 2023-10-30T12:23:15.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 1 librados: calling monclient init 2023-10-30T12:23:15.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.574+0000 7f091b6a2640 10 monclient: init 2023-10-30T12:23:15.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:23:15.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0914099ee0 0x7f0914133080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): picked mon.noname-a con 0x7f0914099ee0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:15.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09141347b0 0x7f0914134b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): picked mon.noname-b con 0x7f09141347b0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:15.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141370e0 0x7f09141350d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): picked mon.noname-c con 0x7f09141370e0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:15.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091406f300 con 0x7f0914099ee0 2023-10-30T12:23:15.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091409ea90 con 0x7f09141347b0 2023-10-30T12:23:15.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091406f7c0 con 0x7f09141370e0 2023-10-30T12:23:15.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient(hunting): authenticate will time out at 2240.758235s 2023-10-30T12:23:15.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0914099ee0 0x7f0914133080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0914099ee0 0x7f0914133080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59876/0 (socket says 172.21.15.50:59876) 2023-10-30T12:23:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141370e0 0x7f09141350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 -- 172.21.15.50:0/4280633557 learned_addr learned my addr 172.21.15.50:0/4280633557 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:15.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): get_auth_request con 0x7f0914099ee0 auth_method 0 2023-10-30T12:23:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 10 monclient(hunting): get_auth_request con 0x7f09141370e0 auth_method 0 2023-10-30T12:23:15.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient(hunting): handle_auth_done global_id 264417 payload 824 2023-10-30T12:23:15.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0918c3d640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09141347b0 0x7f0914134b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 -- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141370e0 msgr2=0x7f09141350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141370e0 0x7f09141350d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 -- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09141347b0 msgr2=0x7f0914134b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09141347b0 0x7f0914134b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 monclient: found mon.noname-a 2023-10-30T12:23:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 -- 172.21.15.50:0/4280633557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c009a70 con 0x7f0914099ee0 2023-10-30T12:23:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0919c3f640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f09141370e0 0x7f09141350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:15.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091943e640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0914099ee0 0x7f0914133080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f091000c4d0 tx=0x7f091000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 10 client.?.objecter ms_handle_connect 0x7f0914099ee0 2023-10-30T12:23:15.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09100074e0 con 0x7f0914099ee0 2023-10-30T12:23:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 10 monclient: dump: 2023-10-30T12:23:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.632 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.632 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.632 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.633 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.633 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.633 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0910007660 con 0x7f0914099ee0 2023-10-30T12:23:15.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 5 monclient: authenticate success, global_id 264417 2023-10-30T12:23:15.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient: _renew_subs 2023-10-30T12:23:15.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:15.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.575+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0904009400 con 0x7f0914099ee0 2023-10-30T12:23:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f0909ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910004a20 con 0x7f0914099ee0 2023-10-30T12:23:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f0909ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f0909ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:15.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f0909ffb640 10 monclient: dump: 2023-10-30T12:23:15.637 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.637 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.638 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.638 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.638 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.638 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.639 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.639 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.639 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.640 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f090b7fe640 4 set_mon_vals no callback set 2023-10-30T12:23:15.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f091b6a2640 10 client.264417.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f091b6a2640 10 monclient: _renew_subs 2023-10-30T12:23:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f091b6a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091406fbe0 con 0x7f0914099ee0 2023-10-30T12:23:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f091b6a2640 1 librados: init done 2023-10-30T12:23:15.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.576+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f0910004b80 con 0x7f0914099ee0 2023-10-30T12:23:15.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f091b6a2640 10 client.264417.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f091b6a2640 15 client.264417.objecter _session_command_op_assign -1 1 2023-10-30T12:23:15.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f091b6a2640 20 client.264417.objecter _get_session osd=-1 returning homeless 2023-10-30T12:23:15.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f091b6a2640 10 client.264417.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:15.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f091b6a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:15.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f091409d7b0 con 0x7f0914099ee0 2023-10-30T12:23:15.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08f80392f0 0x7f08f803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f0910049e60 con 0x7f0914099ee0 2023-10-30T12:23:15.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 10 client.264417.objecter ms_dispatch 0x7f0914000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:15.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 3 client.264417.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:15.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 3 client.264417.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:15.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 10 client.264417.objecter checking command 1 2023-10-30T12:23:15.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f08f8040680 0x7f08f8042b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 20 client.264417.objecter _get_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 20 client.264417.objecter put_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 15 client.264417.objecter _session_command_op_remove -1 1 2023-10-30T12:23:15.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 20 client.264417.objecter _get_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 20 client.264417.objecter get_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 15 client.264417.objecter _session_command_op_assign 2 1 2023-10-30T12:23:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 20 client.264417.objecter put_session s=0x7f08f803ffb0 osd=2 4 2023-10-30T12:23:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 10 client.264417.objecter _send_command 1 2023-10-30T12:23:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 --> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f80431f0 con 0x7f08f8040680 2023-10-30T12:23:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 20 client.264417.objecter dump_active .. 0 homeless 2023-10-30T12:23:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=686) v2 ==== 24+0+0 (secure 0 0 0) 0x7f091001ab40 con 0x7f0914099ee0 2023-10-30T12:23:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 10 monclient: handle_get_version_reply finishing 1 version 686 2023-10-30T12:23:15.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0919c3f640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f08f8040680 0x7f08f8042b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0918c3d640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08f80392f0 0x7f08f803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0918c3d640 10 monclient: get_auth_request con 0x7f08f80392f0 auth_method 0 2023-10-30T12:23:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0919c3f640 10 monclient: get_auth_request con 0x7f08f8040680 auth_method 0 2023-10-30T12:23:15.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0909ffb640 10 client.264417.objecter ms_handle_connect 0x7f08f8040680 2023-10-30T12:23:15.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.577+0000 7f0919c3f640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f08f8040680 0x7f08f8042b20 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.578+0000 7f0909ffb640 10 client.264417.objecter ms_handle_connect 0x7f08f80392f0 2023-10-30T12:23:15.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.578+0000 7f0918c3d640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08f80392f0 0x7f08f803b7b0 secure :-1 s=READY pgs=4362 cs=0 l=1 rev1=1 crypto rx=0x7f09040097a0 tx=0x7f0904008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.578+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== osd.2 v2:172.21.15.50:6800/3518075216 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f08f80431f0 con 0x7f08f8040680 2023-10-30T12:23:15.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.578+0000 7f0909ffb640 10 client.264417.objecter ms_dispatch 0x7f0914000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:23:15.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.578+0000 7f0909ffb640 10 client.264417.objecter _finish_command 1 = system:0 2023-10-30T12:23:15.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.578+0000 7f0909ffb640 20 client.264417.objecter put_session s=0x7f08f803ffb0 osd=2 4 2023-10-30T12:23:15.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.578+0000 7f0909ffb640 15 client.264417.objecter _session_command_op_remove 2 1 2023-10-30T12:23:15.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 10 client.264417.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:23:15.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 15 client.264417.objecter _session_command_op_assign -1 2 2023-10-30T12:23:15.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 client.264417.objecter _get_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 client.264417.objecter put_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 client.264417.objecter _get_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 15 client.264417.objecter _session_command_op_remove -1 2 2023-10-30T12:23:15.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 client.264417.objecter get_session s=0x7f08f803ffb0 osd=2 3 2023-10-30T12:23:15.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 15 client.264417.objecter _session_command_op_assign 2 2 2023-10-30T12:23:15.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 client.264417.objecter put_session s=0x7f08f803ffb0 osd=2 4 2023-10-30T12:23:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 10 client.264417.objecter _send_command 2 2023-10-30T12:23:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 --> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f091409c340 con 0x7f08f8040680 2023-10-30T12:23:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f0909ffb640 1 -- 172.21.15.50:0/4280633557 <== osd.2 v2:172.21.15.50:6800/3518075216 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f090c00f040 con 0x7f08f8040680 2023-10-30T12:23:15.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f0909ffb640 10 client.264417.objecter ms_dispatch 0x7f0914000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:23:15.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f0909ffb640 10 client.264417.objecter _finish_command 2 = system:0 2023-10-30T12:23:15.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f0909ffb640 20 client.264417.objecter put_session s=0x7f08f803ffb0 osd=2 4 2023-10-30T12:23:15.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f0909ffb640 15 client.264417.objecter _session_command_op_remove 2 2 2023-10-30T12:23:15.666 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:15.666 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:23:15.667 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 10 librados: watch_flush enter 2023-10-30T12:23:15.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 10 librados: watch_flush exit 2023-10-30T12:23:15.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 10 client.264417.objecter close_session for osd.2 2023-10-30T12:23:15.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f08f8040680 msgr2=0x7f08f8042b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:6800/3518075216,v1:172.21.15.50:6801/3518075216] conn(0x7f08f8040680 0x7f08f8042b20 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 client.264417.objecter put_session s=0x7f08f803ffb0 osd=2 1 2023-10-30T12:23:15.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 client.264417.objecter shutdown clearing up homeless session... 2023-10-30T12:23:15.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 10 client.264417.objecter successfully canceled tick 2023-10-30T12:23:15.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08f80392f0 msgr2=0x7f08f803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f08f80392f0 0x7f08f803b7b0 secure :-1 s=READY pgs=4362 cs=0 l=1 rev1=1 crypto rx=0x7f09040097a0 tx=0x7f0904008030 comp rx=0 tx=0).stop 2023-10-30T12:23:15.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 10 monclient: shutdown 2023-10-30T12:23:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0914099ee0 msgr2=0x7f0914133080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 --2- 172.21.15.50:0/4280633557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0914099ee0 0x7f0914133080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f091000c4d0 tx=0x7f091000c990 comp rx=0 tx=0).stop 2023-10-30T12:23:15.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f0919c3f640 1 -- 172.21.15.50:0/4280633557 reap_dead start 2023-10-30T12:23:15.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 shutdown_connections 2023-10-30T12:23:15.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.591+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 >> 172.21.15.50:0/4280633557 conn(0x7f09140a0480 msgr2=0x7f0914133cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.592+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 shutdown_connections 2023-10-30T12:23:15.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.592+0000 7f091b6a2640 1 -- 172.21.15.50:0/4280633557 wait complete. 2023-10-30T12:23:15.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.592+0000 7f091b6a2640 1 librados: shutdown 2023-10-30T12:23:15.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T12:23:15.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.780+0000 7f06fdd8a640 10 monclient: get_monmap_and_config 2023-10-30T12:23:15.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.780+0000 7f06fdd8a640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.780+0000 7f06fdd8a640 10 monclient: monmap: 2023-10-30T12:23:15.782 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.782 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.782 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:15.780694+0000 2023-10-30T12:23:15.783 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:15.780694+0000 2023-10-30T12:23:15.783 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.783 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.783 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:15.785 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 1 Processor -- start 2023-10-30T12:23:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 1 -- start start 2023-10-30T12:23:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient: init 2023-10-30T12:23:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-10-30T12:23:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06f8132970 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f06f80a03d0 0x7f06f813aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient(hunting): picked mon.noname-a con 0x7f06f80a03d0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f813b020 0x7f06f813d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient(hunting): picked mon.noname-b con 0x7f06f813b020 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:15.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80a17d0 con 0x7f06f8132970 2023-10-30T12:23:15.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f8133460 con 0x7f06f80a03d0 2023-10-30T12:23:15.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80a1510 con 0x7f06f813b020 2023-10-30T12:23:15.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06fdd8a640 10 monclient(hunting): authenticate will time out at 2240.963852s 2023-10-30T12:23:15.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f06f80a03d0 0x7f06f813aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f813b020 0x7f06f813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f06f80a03d0 0x7f06f813aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59886/0 (socket says 172.21.15.50:59886) 2023-10-30T12:23:15.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f813b020 0x7f06f813d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60762/0 (socket says 172.21.15.50:60762) 2023-10-30T12:23:15.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 -- 172.21.15.50:0/410420396 learned_addr learned my addr 172.21.15.50:0/410420396 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:15.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): get_auth_request con 0x7f06f80a03d0 auth_method 0 2023-10-30T12:23:15.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 10 monclient(hunting): get_auth_request con 0x7f06f813b020 auth_method 0 2023-10-30T12:23:15.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient(hunting): handle_auth_done global_id 264420 payload 274 2023-10-30T12:23:15.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 -- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f813b020 msgr2=0x7f06f813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:15.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 --2- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f813b020 0x7f06f813d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 -- 172.21.15.50:0/410420396 >> v1:172.21.15.118:6789/0 conn(0x7f06f8132970 legacy=0x7f06f8132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 monclient: found mon.noname-a 2023-10-30T12:23:15.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f6ffd640 1 -- 172.21.15.50:0/410420396 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f809f680 con 0x7f06f80a03d0 2023-10-30T12:23:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.781+0000 7f06f7fff640 1 --2- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f813b020 0x7f06f813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:15.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f6ffd640 1 --2- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f80a03d0 0x7f06f813aae0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009870 tx=0x7f06ec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b77231b477b033 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 1 -- 172.21.15.50:0/410420396 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0317c0 con 0x7f06f80a03d0 2023-10-30T12:23:15.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: dump: 2023-10-30T12:23:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.808 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.808 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.808 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 5 monclient: authenticate success, global_id 264420 2023-10-30T12:23:15.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 1 -- 172.21.15.50:0/410420396 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec031920 con 0x7f06f80a03d0 2023-10-30T12:23:15.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:15.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 4 set_mon_vals no callback set 2023-10-30T12:23:15.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 1 -- 172.21.15.50:0/410420396 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec031cb0 con 0x7f06f80a03d0 2023-10-30T12:23:15.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:15.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06f67fc640 10 monclient: dump: 2023-10-30T12:23:15.812 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.812 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.812 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.812 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.814 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 10 monclient: shutdown 2023-10-30T12:23:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 -- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f80a03d0 msgr2=0x7f06f813aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f80a03d0 0x7f06f813aae0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009870 tx=0x7f06ec02b860 comp rx=0 tx=0).stop 2023-10-30T12:23:15.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 -- 172.21.15.50:0/410420396 shutdown_connections 2023-10-30T12:23:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f813b020 0x7f06f813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/410420396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f80a03d0 0x7f06f813aae0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 -- 172.21.15.50:0/410420396 >> 172.21.15.50:0/410420396 conn(0x7f06f80a4800 msgr2=0x7f06f80a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 -- 172.21.15.50:0/410420396 shutdown_connections 2023-10-30T12:23:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.782+0000 7f06fdd8a640 1 -- 172.21.15.50:0/410420396 wait complete. 2023-10-30T12:23:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 10 monclient: build_initial_monmap 2023-10-30T12:23:15.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 10 monclient: monmap: 2023-10-30T12:23:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.819 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:15.820 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 1 librados: starting msgr at 2023-10-30T12:23:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 1 librados: starting objecter 2023-10-30T12:23:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 1 Processor -- start 2023-10-30T12:23:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 1 -- start start 2023-10-30T12:23:15.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 1 librados: setting wanted keys 2023-10-30T12:23:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 1 librados: calling monclient init 2023-10-30T12:23:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.783+0000 7f06fdd8a640 10 monclient: init 2023-10-30T12:23:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:23:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f8132970 0x7f06f8139ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): picked mon.noname-c con 0x7f06f8132970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:15.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f813b020 0x7f06f81313d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): picked mon.noname-b con 0x7f06f813b020 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f811a430 0x7f06f811a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): picked mon.noname-a con 0x7f06f811a430 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:15.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f811d7c0 con 0x7f06f811a430 2023-10-30T12:23:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f811c420 con 0x7f06f813b020 2023-10-30T12:23:15.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f811d4e0 con 0x7f06f8132970 2023-10-30T12:23:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:15.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06fdd8a640 10 monclient(hunting): authenticate will time out at 2240.967341s 2023-10-30T12:23:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f8132970 0x7f06f8139ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f811a430 0x7f06f811a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f8132970 0x7f06f8139ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60764/0 (socket says 172.21.15.50:60764) 2023-10-30T12:23:15.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 -- 172.21.15.50:0/2097057765 learned_addr learned my addr 172.21.15.50:0/2097057765 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): get_auth_request con 0x7f06f8132970 auth_method 0 2023-10-30T12:23:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f811a430 0x7f06f811a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59888/0 (socket says 172.21.15.50:59888) 2023-10-30T12:23:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 10 monclient(hunting): get_auth_request con 0x7f06f811a430 auth_method 0 2023-10-30T12:23:15.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:15.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f6ffd640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f813b020 0x7f06f81313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f6ffd640 10 monclient(hunting): get_auth_request con 0x7f06f813b020 auth_method 0 2023-10-30T12:23:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:15.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f6ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient(hunting): handle_auth_done global_id 284168 payload 824 2023-10-30T12:23:15.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 -- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f813b020 msgr2=0x7f06f81313d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f813b020 0x7f06f81313d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 -- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f811a430 msgr2=0x7f06f811a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f811a430 0x7f06f811a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 monclient: found mon.noname-c 2023-10-30T12:23:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 -- 172.21.15.50:0/2097057765 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0009a70 con 0x7f06f8132970 2023-10-30T12:23:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f7fff640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f811a430 0x7f06f811a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06f77fe640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f8132970 0x7f06f8139ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f06e800c480 tx=0x7f06e800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06dffff640 10 client.?.objecter ms_handle_connect 0x7f06f8132970 2023-10-30T12:23:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.784+0000 7f06dffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 1 -- 172.21.15.50:0/2097057765 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80052d0 con 0x7f06f8132970 2023-10-30T12:23:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: dump: 2023-10-30T12:23:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.838 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.839 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.840 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.840 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: _finish_auth 0 2023-10-30T12:23:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 1 -- 172.21.15.50:0/2097057765 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e8005430 con 0x7f06f8132970 2023-10-30T12:23:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 5 monclient: authenticate success, global_id 284168 2023-10-30T12:23:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 10 monclient: _renew_subs 2023-10-30T12:23:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec0094d0 con 0x7f06f8132970 2023-10-30T12:23:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 1 -- 172.21.15.50:0/2097057765 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8012870 con 0x7f06f8132970 2023-10-30T12:23:15.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 10 monclient: dump: 2023-10-30T12:23:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:15.844 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:15.845 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:15.846 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:15.846 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06f5ffb640 4 set_mon_vals no callback set 2023-10-30T12:23:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 10 client.284168.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:15.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 10 monclient: _renew_subs 2023-10-30T12:23:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f811dd80 con 0x7f06f8132970 2023-10-30T12:23:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06fdd8a640 1 librados: init done 2023-10-30T12:23:15.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.785+0000 7f06dffff640 1 -- 172.21.15.50:0/2097057765 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f06e80129d0 con 0x7f06f8132970 2023-10-30T12:23:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06fdd8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06fdd8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06fdd8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:15.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06dffff640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06e40392f0 0x7f06e403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f804e160 con 0x7f06f8132970 2023-10-30T12:23:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06dffff640 1 -- 172.21.15.50:0/2097057765 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f06e804a530 con 0x7f06f8132970 2023-10-30T12:23:15.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06dffff640 10 client.284168.objecter ms_dispatch 0x7f06f8000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06dffff640 3 client.284168.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06dffff640 3 client.284168.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06dffff640 20 client.284168.objecter dump_active .. 0 homeless 2023-10-30T12:23:15.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06f6ffd640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06e40392f0 0x7f06e403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06f6ffd640 10 monclient: get_auth_request con 0x7f06e40392f0 auth_method 0 2023-10-30T12:23:15.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06dffff640 10 client.284168.objecter ms_handle_connect 0x7f06e40392f0 2023-10-30T12:23:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.786+0000 7f06f6ffd640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06e40392f0 0x7f06e403b7b0 secure :-1 s=READY pgs=4363 cs=0 l=1 rev1=1 crypto rx=0x7f06ec036000 tx=0x7f06ec03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.791+0000 7f06dffff640 1 -- 172.21.15.50:0/2097057765 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f06e8012e60 con 0x7f06f8132970 2023-10-30T12:23:15.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.791+0000 7f06dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.791+0000 7f06dffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:15.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.967+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f06f809f960 con 0x7f06e40392f0 2023-10-30T12:23:15.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.968+0000 7f06dffff640 1 -- 172.21.15.50:0/2097057765 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f06ec031b10 con 0x7f06e40392f0 2023-10-30T12:23:15.969 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T12:23:15.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 10 librados: watch_flush enter 2023-10-30T12:23:15.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 10 librados: watch_flush exit 2023-10-30T12:23:15.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 20 client.284168.objecter shutdown clearing up homeless session... 2023-10-30T12:23:15.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 10 client.284168.objecter successfully canceled tick 2023-10-30T12:23:15.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06e40392f0 msgr2=0x7f06e403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06e40392f0 0x7f06e403b7b0 secure :-1 s=READY pgs=4363 cs=0 l=1 rev1=1 crypto rx=0x7f06ec036000 tx=0x7f06ec03d040 comp rx=0 tx=0).stop 2023-10-30T12:23:15.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 10 monclient: shutdown 2023-10-30T12:23:15.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:15.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f8132970 msgr2=0x7f06f8139ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:15.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f8132970 0x7f06f8139ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f06e800c480 tx=0x7f06e800c940 comp rx=0 tx=0).stop 2023-10-30T12:23:15.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 shutdown_connections 2023-10-30T12:23:15.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f06e40392f0 0x7f06e403b7b0 unknown :-1 s=CLOSED pgs=4363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06f811a430 0x7f06f811a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f813b020 0x7f06f81313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.969+0000 7f06fdd8a640 1 --2- 172.21.15.50:0/2097057765 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06f8132970 0x7f06f8139ee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:15.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.970+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 >> 172.21.15.50:0/2097057765 conn(0x7f06f80a4800 msgr2=0x7f06f80a26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:15.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.970+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 shutdown_connections 2023-10-30T12:23:15.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.970+0000 7f06fdd8a640 1 -- 172.21.15.50:0/2097057765 wait complete. 2023-10-30T12:23:15.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:15.970+0000 7f06fdd8a640 1 librados: shutdown 2023-10-30T12:23:15.980 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T12:23:15.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:23:16.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.089+0000 7fb57d0c2640 10 monclient: get_monmap_and_config 2023-10-30T12:23:16.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.089+0000 7fb57d0c2640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.089+0000 7fb57d0c2640 10 monclient: monmap: 2023-10-30T12:23:16.091 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.091 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.092 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:16.089650+0000 2023-10-30T12:23:16.092 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:16.089650+0000 2023-10-30T12:23:16.092 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.092 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.093 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:16.093 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:16.093 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:16.093 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:16.094 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:16.094 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 1 Processor -- start 2023-10-30T12:23:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 1 -- start start 2023-10-30T12:23:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient: init 2023-10-30T12:23:16.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-10-30T12:23:16.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb57809bb70 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:16.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57809c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient(hunting): picked mon.noname-b con 0x7fb57809bf50 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb578137ac0 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57809d720 con 0x7fb578137ac0 2023-10-30T12:23:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57809d040 con 0x7fb57809bb70 2023-10-30T12:23:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb578097c60 con 0x7fb57809bf50 2023-10-30T12:23:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57d0c2640 10 monclient(hunting): authenticate will time out at 2241.272696s 2023-10-30T12:23:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57809c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57759e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb578137ac0 0x7fb578139eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49666/0 (socket says 172.21.15.50:49666) 2023-10-30T12:23:16.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb57759e640 1 -- 172.21.15.50:0/487476199 learned_addr learned my addr 172.21.15.50:0/487476199 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:16.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 10 monclient(hunting): get_auth_request con 0x7fb57809bf50 auth_method 0 2023-10-30T12:23:16.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.090+0000 7fb56e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 10 monclient(hunting): handle_auth_done global_id 284171 payload 274 2023-10-30T12:23:16.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb57659c640 1 -- 172.21.15.50:0/487476199 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885164554 0 0) 0x7fb57809d720 con 0x7fb578137ac0 2023-10-30T12:23:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 1 -- 172.21.15.50:0/487476199 >> v1:172.21.15.118:6789/0 conn(0x7fb57809bb70 legacy=0x7fb578098c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 1 -- 172.21.15.50:0/487476199 >> v1:172.21.15.50:6789/0 conn(0x7fb578137ac0 legacy=0x7fb578139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 1 monclient: found mon.noname-b 2023-10-30T12:23:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 1 -- 172.21.15.50:0/487476199 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578097980 con 0x7fb57809bf50 2023-10-30T12:23:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb57659c640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:16.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.091+0000 7fb56e59c640 1 --2- 172.21.15.50:0/487476199 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57809c370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5600097d0 tx=0x7fb56002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aefbe0e778e83a19 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 1 -- 172.21.15.50:0/487476199 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560031770 con 0x7fb57809bf50 2023-10-30T12:23:16.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:16.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: dump: 2023-10-30T12:23:16.113 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.113 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.114 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.114 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.114 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.114 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.115 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.115 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.115 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.115 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 5 monclient: authenticate success, global_id 284171 2023-10-30T12:23:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 1 -- 172.21.15.50:0/487476199 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5600318d0 con 0x7fb57809bf50 2023-10-30T12:23:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 1 -- 172.21.15.50:0/487476199 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560031c60 con 0x7fb57809bf50 2023-10-30T12:23:16.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:16.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 4 set_mon_vals no callback set 2023-10-30T12:23:16.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57659c640 10 monclient: dump: 2023-10-30T12:23:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.121 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.122 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 10 monclient: shutdown 2023-10-30T12:23:16.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:16.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 1 -- 172.21.15.50:0/487476199 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 msgr2=0x7fb57809c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 1 --2- 172.21.15.50:0/487476199 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57809c370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5600097d0 tx=0x7fb56002b810 comp rx=0 tx=0).stop 2023-10-30T12:23:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 1 -- 172.21.15.50:0/487476199 shutdown_connections 2023-10-30T12:23:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 1 --2- 172.21.15.50:0/487476199 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57809c370 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 1 -- 172.21.15.50:0/487476199 >> 172.21.15.50:0/487476199 conn(0x7fb5780a0480 msgr2=0x7fb5780a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 1 -- 172.21.15.50:0/487476199 shutdown_connections 2023-10-30T12:23:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.092+0000 7fb57d0c2640 1 -- 172.21.15.50:0/487476199 wait complete. 2023-10-30T12:23:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 10 monclient: monmap: 2023-10-30T12:23:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.126 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.127 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:16.127 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:16.127 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:16.127 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 1 librados: starting msgr at 2023-10-30T12:23:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 1 librados: starting objecter 2023-10-30T12:23:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 1 Processor -- start 2023-10-30T12:23:16.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 1 -- start start 2023-10-30T12:23:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 1 librados: setting wanted keys 2023-10-30T12:23:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 1 librados: calling monclient init 2023-10-30T12:23:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.093+0000 7fb57d0c2640 10 monclient: init 2023-10-30T12:23:16.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:23:16.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb57809bb70 0x7fb578136e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): picked mon.noname-b con 0x7fb57809bb70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57808efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): picked mon.noname-c con 0x7fb57809bf50 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb578137ac0 0x7fb57808f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): picked mon.noname-a con 0x7fb578137ac0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb578137880 con 0x7fb578137ac0 2023-10-30T12:23:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb578092640 con 0x7fb57809bb70 2023-10-30T12:23:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57809d380 con 0x7fb57809bf50 2023-10-30T12:23:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57d0c2640 10 monclient(hunting): authenticate will time out at 2241.276376s 2023-10-30T12:23:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57808efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb578137ac0 0x7fb57808f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57808efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60776/0 (socket says 172.21.15.50:60776) 2023-10-30T12:23:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 -- 172.21.15.50:0/4072905667 learned_addr learned my addr 172.21.15.50:0/4072905667 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): get_auth_request con 0x7fb57809bf50 auth_method 0 2023-10-30T12:23:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 10 monclient(hunting): get_auth_request con 0x7fb578137ac0 auth_method 0 2023-10-30T12:23:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb57809bb70 0x7fb578136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 10 monclient(hunting): get_auth_request con 0x7fb57809bb70 auth_method 0 2023-10-30T12:23:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb576d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient(hunting): handle_auth_done global_id 284174 payload 824 2023-10-30T12:23:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb57809bb70 msgr2=0x7fb578136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb57809bb70 0x7fb578136e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb578137ac0 msgr2=0x7fb57808f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb578137ac0 0x7fb57808f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 monclient: found mon.noname-c 2023-10-30T12:23:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb56e59c640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb560009400 con 0x7fb57809bf50 2023-10-30T12:23:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.094+0000 7fb57759e640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb578137ac0 0x7fb57808f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56e59c640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57808efc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb560034040 tx=0x7fb5600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 client.?.objecter ms_handle_connect 0x7fb57809bf50 2023-10-30T12:23:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560004230 con 0x7fb57809bf50 2023-10-30T12:23:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: dump: 2023-10-30T12:23:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.146 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.147 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.148 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb560004390 con 0x7fb57809bf50 2023-10-30T12:23:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560032910 con 0x7fb57809bf50 2023-10-30T12:23:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 5 monclient: authenticate success, global_id 284174 2023-10-30T12:23:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 10 monclient: _renew_subs 2023-10-30T12:23:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb578092b70 con 0x7fb57809bf50 2023-10-30T12:23:16.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb56ffff640 10 monclient: dump: 2023-10-30T12:23:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.152 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.153 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.153 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.153 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.153 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.155 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb575d9b640 4 set_mon_vals no callback set 2023-10-30T12:23:16.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 10 client.284174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:16.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 10 monclient: _renew_subs 2023-10-30T12:23:16.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb578090730 con 0x7fb57809bf50 2023-10-30T12:23:16.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.095+0000 7fb57d0c2640 1 librados: init done 2023-10-30T12:23:16.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb560032a70 con 0x7fb57809bf50 2023-10-30T12:23:16.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57d0c2640 10 client.284174.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57d0c2640 15 client.284174.objecter _session_command_op_assign -1 1 2023-10-30T12:23:16.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57d0c2640 20 client.284174.objecter _get_session osd=-1 returning homeless 2023-10-30T12:23:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57d0c2640 10 client.284174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57d0c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb57809c8b0 con 0x7fb57809bf50 2023-10-30T12:23:16.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb558039320 0x7fb55803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7fb56006cc40 con 0x7fb57809bf50 2023-10-30T12:23:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 10 client.284174.objecter ms_dispatch 0x7fb578000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 3 client.284174.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:16.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 3 client.284174.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 10 client.284174.objecter checking command 1 2023-10-30T12:23:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 0x7fb558042b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 20 client.284174.objecter _get_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 20 client.284174.objecter put_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 15 client.284174.objecter _session_command_op_remove -1 1 2023-10-30T12:23:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 20 client.284174.objecter _get_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 20 client.284174.objecter get_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 15 client.284174.objecter _session_command_op_assign 3 1 2023-10-30T12:23:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 20 client.284174.objecter put_session s=0x7fb55803ffe0 osd=3 4 2023-10-30T12:23:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 10 client.284174.objecter _send_command 1 2023-10-30T12:23:16.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb558043220 con 0x7fb5580406b0 2023-10-30T12:23:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 20 client.284174.objecter dump_active .. 0 homeless 2023-10-30T12:23:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=686) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5600722a0 con 0x7fb57809bf50 2023-10-30T12:23:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb56ffff640 10 monclient: handle_get_version_reply finishing 1 version 686 2023-10-30T12:23:16.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb576d9d640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb558039320 0x7fb55803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb576d9d640 10 monclient: get_auth_request con 0x7fb558039320 auth_method 0 2023-10-30T12:23:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57759e640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 0x7fb558042b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.096+0000 7fb57759e640 10 monclient: get_auth_request con 0x7fb5580406b0 auth_method 0 2023-10-30T12:23:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb56ffff640 10 client.284174.objecter ms_handle_connect 0x7fb558039320 2023-10-30T12:23:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb576d9d640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb558039320 0x7fb55803b7e0 secure :-1 s=READY pgs=4364 cs=0 l=1 rev1=1 crypto rx=0x7fb568000f60 tx=0x7fb568009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb56ffff640 10 client.284174.objecter ms_handle_connect 0x7fb5580406b0 2023-10-30T12:23:16.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb57759e640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 0x7fb558042b50 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== osd.3 v2:172.21.15.118:6808/1009584072 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb558043220 con 0x7fb5580406b0 2023-10-30T12:23:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb56ffff640 10 client.284174.objecter ms_dispatch 0x7fb578000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:23:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb56ffff640 10 client.284174.objecter _finish_command 1 = system:0 2023-10-30T12:23:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb56ffff640 20 client.284174.objecter put_session s=0x7fb55803ffe0 osd=3 4 2023-10-30T12:23:16.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.097+0000 7fb56ffff640 15 client.284174.objecter _session_command_op_remove 3 1 2023-10-30T12:23:16.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 10 client.284174.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:23:16.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 15 client.284174.objecter _session_command_op_assign -1 2 2023-10-30T12:23:16.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 20 client.284174.objecter _get_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 20 client.284174.objecter put_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 20 client.284174.objecter _get_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 15 client.284174.objecter _session_command_op_remove -1 2 2023-10-30T12:23:16.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 20 client.284174.objecter get_session s=0x7fb55803ffe0 osd=3 3 2023-10-30T12:23:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 15 client.284174.objecter _session_command_op_assign 3 2 2023-10-30T12:23:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 20 client.284174.objecter put_session s=0x7fb55803ffe0 osd=3 4 2023-10-30T12:23:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 10 client.284174.objecter _send_command 2 2023-10-30T12:23:16.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb578074530 con 0x7fb5580406b0 2023-10-30T12:23:16.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57759e640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 msgr2=0x7fb558042b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T12:23:16.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57759e640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 msgr2=0x7fb558042b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T12:23:16.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.110+0000 7fb57759e640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 0x7fb558042b50 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:23:16.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb57759e640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 0x7fb558042b50 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 1 client.284174.objecter ms_handle_reset 0x7fb5580406b0 session 0x7fb55803ffe0 osd.3 2023-10-30T12:23:16.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 client.284174.objecter reopen_session osd.3 session, addr now [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] 2023-10-30T12:23:16.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 msgr2=0x7fb558042b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-30T12:23:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb5580406b0 0x7fb558042b50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb558043490 0x7fb558045880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 client.284174.objecter _send_command 2 2023-10-30T12:23:16.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb558043220 con 0x7fb558043490 2023-10-30T12:23:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 client.284174.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 monclient: _renew_subs 2023-10-30T12:23:16.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=687}) v3 -- 0x7fb558045e60 con 0x7fb57809bf50 2023-10-30T12:23:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb57759e640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb558043490 0x7fb558045880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb57759e640 10 monclient: get_auth_request con 0x7fb558043490 auth_method 0 2023-10-30T12:23:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 client.284174.objecter ms_handle_connect 0x7fb558043490 2023-10-30T12:23:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb57759e640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb558043490 0x7fb558045880 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 1 -- 172.21.15.50:0/4072905667 <== osd.3 v2:172.21.15.118:6808/1009584072 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+22 (crc 0 0 0) 0x7fb558043220 con 0x7fb558043490 2023-10-30T12:23:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 client.284174.objecter ms_dispatch 0x7fb578000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:23:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 10 client.284174.objecter _finish_command 2 = system:0 2023-10-30T12:23:16.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 20 client.284174.objecter put_session s=0x7fb55803ffe0 osd=3 4 2023-10-30T12:23:16.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.111+0000 7fb56ffff640 15 client.284174.objecter _session_command_op_remove 3 2 2023-10-30T12:23:16.179 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "" 2023-10-30T12:23:16.179 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:16.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 10 librados: watch_flush enter 2023-10-30T12:23:16.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 10 librados: watch_flush exit 2023-10-30T12:23:16.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 10 client.284174.objecter close_session for osd.3 2023-10-30T12:23:16.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb558043490 msgr2=0x7fb558045880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6808/1009584072,v1:172.21.15.118:6810/1009584072] conn(0x7fb558043490 0x7fb558045880 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 20 client.284174.objecter put_session s=0x7fb55803ffe0 osd=3 1 2023-10-30T12:23:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 20 client.284174.objecter shutdown clearing up homeless session... 2023-10-30T12:23:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 10 client.284174.objecter successfully canceled tick 2023-10-30T12:23:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb558039320 msgr2=0x7fb55803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb558039320 0x7fb55803b7e0 secure :-1 s=READY pgs=4364 cs=0 l=1 rev1=1 crypto rx=0x7fb568000f60 tx=0x7fb568009040 comp rx=0 tx=0).stop 2023-10-30T12:23:16.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 10 monclient: shutdown 2023-10-30T12:23:16.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 msgr2=0x7fb57808efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 --2- 172.21.15.50:0/4072905667 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb57809bf50 0x7fb57808efc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb560034040 tx=0x7fb5600046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57759e640 1 -- 172.21.15.50:0/4072905667 reap_dead start 2023-10-30T12:23:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 shutdown_connections 2023-10-30T12:23:16.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 >> 172.21.15.50:0/4072905667 conn(0x7fb5780a0480 msgr2=0x7fb57809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:16.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 shutdown_connections 2023-10-30T12:23:16.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 -- 172.21.15.50:0/4072905667 wait complete. 2023-10-30T12:23:16.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.112+0000 7fb57d0c2640 1 librados: shutdown 2023-10-30T12:23:16.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T12:23:16.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.292+0000 7f849ea26640 10 monclient: get_monmap_and_config 2023-10-30T12:23:16.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.292+0000 7f849ea26640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.293+0000 7f849ea26640 10 monclient: monmap: 2023-10-30T12:23:16.294 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.294 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.295 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:16.293050+0000 2023-10-30T12:23:16.295 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:16.293050+0000 2023-10-30T12:23:16.295 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.296 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.296 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:16.296 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:16.296 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:16.297 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:16.297 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:16.297 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.293+0000 7f849ea26640 1 Processor -- start 2023-10-30T12:23:16.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.293+0000 7f849ea26640 1 -- start start 2023-10-30T12:23:16.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.293+0000 7f849ea26640 10 monclient: init 2023-10-30T12:23:16.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-10-30T12:23:16.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8498096540 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:16.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498136960 0x7f8498138d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient(hunting): picked mon.noname-b con 0x7f8498136960 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8498139290 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:16.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849809bd30 con 0x7f8498096540 2023-10-30T12:23:16.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849809c400 con 0x7f8498139290 2023-10-30T12:23:16.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849809c120 con 0x7f8498136960 2023-10-30T12:23:16.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f849ea26640 10 monclient(hunting): authenticate will time out at 2241.476206s 2023-10-30T12:23:16.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8497fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8498096540 0x7f8498096920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49682/0 (socket says 172.21.15.50:49682) 2023-10-30T12:23:16.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8497fff640 1 -- 172.21.15.50:0/4170750905 learned_addr learned my addr 172.21.15.50:0/4170750905 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:16.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 1 --2- 172.21.15.50:0/4170750905 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498136960 0x7f8498138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): get_auth_request con 0x7f8498136960 auth_method 0 2023-10-30T12:23:16.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8496ffd640 1 -- 172.21.15.50:0/4170750905 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498153161 0 0) 0x7f849809bd30 con 0x7f8498096540 2023-10-30T12:23:16.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8496ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:23:16.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8496ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8496ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8496ffd640 10 monclient(hunting): my global_id is 264432 2023-10-30T12:23:16.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f8496ffd640 1 -- 172.21.15.50:0/4170750905 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8480003520 con 0x7f8498096540 2023-10-30T12:23:16.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient(hunting): handle_auth_done global_id 284177 payload 274 2023-10-30T12:23:16.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 1 -- 172.21.15.50:0/4170750905 >> v1:172.21.15.118:6789/0 conn(0x7f8498139290 legacy=0x7f849813b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 1 -- 172.21.15.50:0/4170750905 >> v1:172.21.15.50:6789/0 conn(0x7f8498096540 legacy=0x7f8498096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 1 monclient: found mon.noname-b 2023-10-30T12:23:16.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.294+0000 7f84977fe640 1 -- 172.21.15.50:0/4170750905 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8498097030 con 0x7f8498136960 2023-10-30T12:23:16.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f84977fe640 1 --2- 172.21.15.50:0/4170750905 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498136960 0x7f8498138d50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84880097d0 tx=0x7f848802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa9cdf8b97f762c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 1 -- 172.21.15.50:0/4170750905 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488031800 con 0x7f8498136960 2023-10-30T12:23:16.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:16.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: dump: 2023-10-30T12:23:16.317 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.317 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.317 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.318 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.318 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.318 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.318 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.319 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.319 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.319 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 1 -- 172.21.15.50:0/4170750905 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488031960 con 0x7f8498136960 2023-10-30T12:23:16.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 1 -- 172.21.15.50:0/4170750905 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488031cf0 con 0x7f8498136960 2023-10-30T12:23:16.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f849ea26640 5 monclient: authenticate success, global_id 284177 2023-10-30T12:23:16.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f849ea26640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:16.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f849ea26640 4 set_mon_vals no callback set 2023-10-30T12:23:16.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:16.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f8496ffd640 10 monclient: dump: 2023-10-30T12:23:16.323 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.323 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.323 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.323 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.325 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.325 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f849ea26640 10 monclient: shutdown 2023-10-30T12:23:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f849ea26640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f849ea26640 1 -- 172.21.15.50:0/4170750905 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498136960 msgr2=0x7f8498138d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.295+0000 7f849ea26640 1 --2- 172.21.15.50:0/4170750905 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498136960 0x7f8498138d50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f84880097d0 tx=0x7f848802b8a0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 -- 172.21.15.50:0/4170750905 shutdown_connections 2023-10-30T12:23:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 --2- 172.21.15.50:0/4170750905 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498136960 0x7f8498138d50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 -- 172.21.15.50:0/4170750905 >> 172.21.15.50:0/4170750905 conn(0x7f84980a3980 msgr2=0x7f84980a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 -- 172.21.15.50:0/4170750905 shutdown_connections 2023-10-30T12:23:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 -- 172.21.15.50:0/4170750905 wait complete. 2023-10-30T12:23:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 10 monclient: monmap: 2023-10-30T12:23:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.329 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:16.329 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:16.329 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:16.329 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:16.331 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:16.331 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 librados: starting msgr at 2023-10-30T12:23:16.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 librados: starting objecter 2023-10-30T12:23:16.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.296+0000 7f849ea26640 1 Processor -- start 2023-10-30T12:23:16.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 -- start start 2023-10-30T12:23:16.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 librados: setting wanted keys 2023-10-30T12:23:16.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 librados: calling monclient init 2023-10-30T12:23:16.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient: init 2023-10-30T12:23:16.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:23:16.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8498096540 0x7f84981359c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): picked mon.noname-a con 0x7f8498096540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:16.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8498136960 0x7f8498135f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): picked mon.noname-b con 0x7f8498136960 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498139290 0x7f84980704b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): picked mon.noname-c con 0x7f8498139290 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84980a2990 con 0x7f8498096540 2023-10-30T12:23:16.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8498099130 con 0x7f8498136960 2023-10-30T12:23:16.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849809a080 con 0x7f8498139290 2023-10-30T12:23:16.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849ea26640 10 monclient(hunting): authenticate will time out at 2241.479586s 2023-10-30T12:23:16.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8498096540 0x7f84981359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8498096540 0x7f84981359c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59914/0 (socket says 172.21.15.50:59914) 2023-10-30T12:23:16.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498139290 0x7f84980704b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 1 -- 172.21.15.50:0/597142767 learned_addr learned my addr 172.21.15.50:0/597142767 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:16.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): get_auth_request con 0x7f8498096540 auth_method 0 2023-10-30T12:23:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 10 monclient(hunting): get_auth_request con 0x7f8498139290 auth_method 0 2023-10-30T12:23:16.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f849cfc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f84977fe640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8498136960 0x7f8498135f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f84977fe640 10 monclient(hunting): get_auth_request con 0x7f8498136960 auth_method 0 2023-10-30T12:23:16.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f84977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f84977fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f84977fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 10 monclient(hunting): handle_auth_done global_id 264435 payload 824 2023-10-30T12:23:16.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.297+0000 7f8497fff640 1 -- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498139290 msgr2=0x7f84980704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498139290 0x7f84980704b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 1 -- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8498136960 msgr2=0x7f8498135f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8498136960 0x7f8498135f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 1 monclient: found mon.noname-a 2023-10-30T12:23:16.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 1 -- 172.21.15.50:0/597142767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8488009400 con 0x7f8498096540 2023-10-30T12:23:16.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849cfc3640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498139290 0x7f84980704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:16.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f84977fe640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8498136960 0x7f8498135f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:23:16.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8497fff640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8498096540 0x7f84981359c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f848c001260 tx=0x7f848c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 client.?.objecter ms_handle_connect 0x7f8498096540 2023-10-30T12:23:16.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:16.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 1 -- 172.21.15.50:0/597142767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c0074e0 con 0x7f8498096540 2023-10-30T12:23:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:16.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: dump: 2023-10-30T12:23:16.349 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.349 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.349 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.349 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.350 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.350 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.350 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.350 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.351 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.351 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 1 -- 172.21.15.50:0/597142767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c007640 con 0x7f8498096540 2023-10-30T12:23:16.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 1 -- 172.21.15.50:0/597142767 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c0059a0 con 0x7f8498096540 2023-10-30T12:23:16.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 5 monclient: authenticate success, global_id 264435 2023-10-30T12:23:16.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 10 monclient: _renew_subs 2023-10-30T12:23:16.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849808f910 con 0x7f8498096540 2023-10-30T12:23:16.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f84967fc640 4 set_mon_vals no callback set 2023-10-30T12:23:16.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:16.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f8494ff9640 10 monclient: dump: 2023-10-30T12:23:16.355 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.355 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.356 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.356 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.356 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.356 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.357 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.357 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 10 client.264435.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:16.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 10 monclient: _renew_subs 2023-10-30T12:23:16.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84980984e0 con 0x7f8498096540 2023-10-30T12:23:16.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.298+0000 7f849ea26640 1 librados: init done 2023-10-30T12:23:16.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f8494ff9640 1 -- 172.21.15.50:0/597142767 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f848c005b00 con 0x7f8498096540 2023-10-30T12:23:16.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f849ea26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f849ea26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f849ea26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f8494ff9640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84840392f0 0x7f848403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849804e160 con 0x7f8498096540 2023-10-30T12:23:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f8494ff9640 1 -- 172.21.15.50:0/597142767 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f848c049ed0 con 0x7f8498096540 2023-10-30T12:23:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f8494ff9640 10 client.264435.objecter ms_dispatch 0x7f8498000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f8494ff9640 3 client.264435.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:16.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f8494ff9640 3 client.264435.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f8494ff9640 20 client.264435.objecter dump_active .. 0 homeless 2023-10-30T12:23:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f84977fe640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84840392f0 0x7f848403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.299+0000 7f84977fe640 10 monclient: get_auth_request con 0x7f84840392f0 auth_method 0 2023-10-30T12:23:16.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.300+0000 7f8494ff9640 10 client.264435.objecter ms_handle_connect 0x7f84840392f0 2023-10-30T12:23:16.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.300+0000 7f84977fe640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84840392f0 0x7f848403b7b0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 crypto rx=0x7f8488036000 tx=0x7f848803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.303+0000 7f8494ff9640 1 -- 172.21.15.50:0/597142767 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f848c017020 con 0x7f8498096540 2023-10-30T12:23:16.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.303+0000 7f8494ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.303+0000 7f8494ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:16.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.482+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f849809ba70 con 0x7f84840392f0 2023-10-30T12:23:16.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.483+0000 7f8494ff9640 1 -- 172.21.15.50:0/597142767 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f848803a420 con 0x7f84840392f0 2023-10-30T12:23:16.484 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T12:23:16.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 10 librados: watch_flush enter 2023-10-30T12:23:16.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 10 librados: watch_flush exit 2023-10-30T12:23:16.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 20 client.264435.objecter shutdown clearing up homeless session... 2023-10-30T12:23:16.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 10 client.264435.objecter successfully canceled tick 2023-10-30T12:23:16.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84840392f0 msgr2=0x7f848403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84840392f0 0x7f848403b7b0 secure :-1 s=READY pgs=4365 cs=0 l=1 rev1=1 crypto rx=0x7f8488036000 tx=0x7f848803c040 comp rx=0 tx=0).stop 2023-10-30T12:23:16.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 10 monclient: shutdown 2023-10-30T12:23:16.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:16.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8498096540 msgr2=0x7f84981359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8498096540 0x7f84981359c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f848c001260 tx=0x7f848c00c960 comp rx=0 tx=0).stop 2023-10-30T12:23:16.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 shutdown_connections 2023-10-30T12:23:16.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f84840392f0 0x7f848403b7b0 unknown :-1 s=CLOSED pgs=4365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8498139290 0x7f84980704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8498136960 0x7f8498135f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 --2- 172.21.15.50:0/597142767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8498096540 0x7f84981359c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.484+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 >> 172.21.15.50:0/597142767 conn(0x7f84980a3980 msgr2=0x7f849809a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:16.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.485+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 shutdown_connections 2023-10-30T12:23:16.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.485+0000 7f849ea26640 1 -- 172.21.15.50:0/597142767 wait complete. 2023-10-30T12:23:16.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.485+0000 7f849ea26640 1 librados: shutdown 2023-10-30T12:23:16.495 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T12:23:16.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:23:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.604+0000 7f423bd83640 10 monclient: get_monmap_and_config 2023-10-30T12:23:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.604+0000 7f423bd83640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.604+0000 7f423bd83640 10 monclient: monmap: 2023-10-30T12:23:16.606 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.606 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.606 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:16.604645+0000 2023-10-30T12:23:16.607 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:16.604645+0000 2023-10-30T12:23:16.607 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.607 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.608 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:16.608 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:16.608 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:16.608 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:16.609 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:16.609 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 1 Processor -- start 2023-10-30T12:23:16.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 1 -- start start 2023-10-30T12:23:16.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient: init 2023-10-30T12:23:16.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-10-30T12:23:16.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42340965d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:16.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423409d210 0x7f42341389f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient(hunting): picked mon.noname-b con 0x7f423409d210 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f4234138f30 0x7f423413b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient(hunting): picked mon.noname-c con 0x7f4234138f30 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:16.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42340970c0 con 0x7f42340965d0 2023-10-30T12:23:16.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4234099c30 con 0x7f4234138f30 2023-10-30T12:23:16.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423409bf90 con 0x7f423409d210 2023-10-30T12:23:16.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423bd83640 10 monclient(hunting): authenticate will time out at 2241.787827s 2023-10-30T12:23:16.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.605+0000 7f423931e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423409d210 0x7f42341389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423409d210 0x7f42341389f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60796/0 (socket says 172.21.15.50:60796) 2023-10-30T12:23:16.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 1 -- 172.21.15.50:0/3801223351 learned_addr learned my addr 172.21.15.50:0/3801223351 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:16.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): get_auth_request con 0x7f423409d210 auth_method 0 2023-10-30T12:23:16.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 1 --2- 172.21.15.50:0/3801223351 >> v2:172.21.15.118:3300/0 conn(0x7f4234138f30 0x7f423413b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 10 monclient(hunting): get_auth_request con 0x7f4234138f30 auth_method 0 2023-10-30T12:23:16.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423a320640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient(hunting): handle_auth_done global_id 284183 payload 274 2023-10-30T12:23:16.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 1 -- 172.21.15.50:0/3801223351 >> v2:172.21.15.118:3300/0 conn(0x7f4234138f30 msgr2=0x7f423413b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:16.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 1 --2- 172.21.15.50:0/3801223351 >> v2:172.21.15.118:3300/0 conn(0x7f4234138f30 0x7f423413b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 1 -- 172.21.15.50:0/3801223351 >> v1:172.21.15.118:6789/0 conn(0x7f42340965d0 legacy=0x7f42340969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 1 monclient: found mon.noname-b 2023-10-30T12:23:16.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.606+0000 7f423931e640 1 -- 172.21.15.50:0/3801223351 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423409c270 con 0x7f423409d210 2023-10-30T12:23:16.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423931e640 1 --2- 172.21.15.50:0/3801223351 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423409d210 0x7f42341389f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4228009870 tx=0x7f422802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f7a20c8b814c774 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 1 -- 172.21.15.50:0/3801223351 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280317c0 con 0x7f423409d210 2023-10-30T12:23:16.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:16.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: dump: 2023-10-30T12:23:16.632 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.632 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.633 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.633 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.633 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.634 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.634 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.634 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.635 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.635 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 5 monclient: authenticate success, global_id 284183 2023-10-30T12:23:16.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:16.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 1 -- 172.21.15.50:0/3801223351 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228031920 con 0x7f423409d210 2023-10-30T12:23:16.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:16.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 4 set_mon_vals no callback set 2023-10-30T12:23:16.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 1 -- 172.21.15.50:0/3801223351 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228031cb0 con 0x7f423409d210 2023-10-30T12:23:16.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:16.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f4238b1d640 10 monclient: dump: 2023-10-30T12:23:16.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.641 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.641 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.642 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.642 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.643 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.644 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.644 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.645 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 10 monclient: shutdown 2023-10-30T12:23:16.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:16.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 -- 172.21.15.50:0/3801223351 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423409d210 msgr2=0x7f42341389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 --2- 172.21.15.50:0/3801223351 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423409d210 0x7f42341389f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4228009870 tx=0x7f422802b860 comp rx=0 tx=0).stop 2023-10-30T12:23:16.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 -- 172.21.15.50:0/3801223351 shutdown_connections 2023-10-30T12:23:16.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 --2- 172.21.15.50:0/3801223351 >> v2:172.21.15.118:3300/0 conn(0x7f4234138f30 0x7f423413b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 --2- 172.21.15.50:0/3801223351 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423409d210 0x7f42341389f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 -- 172.21.15.50:0/3801223351 >> 172.21.15.50:0/3801223351 conn(0x7f42340a0480 msgr2=0x7f42340a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:16.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 -- 172.21.15.50:0/3801223351 shutdown_connections 2023-10-30T12:23:16.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.607+0000 7f423bd83640 1 -- 172.21.15.50:0/3801223351 wait complete. 2023-10-30T12:23:16.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 10 monclient: monmap: 2023-10-30T12:23:16.650 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.650 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:16.650 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:16.650 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:16.651 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.651 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.651 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:16.652 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:16.652 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:16.652 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 1 librados: starting msgr at 2023-10-30T12:23:16.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 1 librados: starting objecter 2023-10-30T12:23:16.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 1 Processor -- start 2023-10-30T12:23:16.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 1 -- start start 2023-10-30T12:23:16.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 1 librados: setting wanted keys 2023-10-30T12:23:16.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 1 librados: calling monclient init 2023-10-30T12:23:16.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.608+0000 7f423bd83640 10 monclient: init 2023-10-30T12:23:16.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:23:16.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42340965d0 0x7f42341c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): picked mon.noname-c con 0x7f42340965d0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423409d210 0x7f4234094210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): picked mon.noname-a con 0x7f423409d210 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:16.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4234138f30 0x7f4234094750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): picked mon.noname-b con 0x7f4234138f30 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:16.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423408ced0 con 0x7f423409d210 2023-10-30T12:23:16.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423408e790 con 0x7f4234138f30 2023-10-30T12:23:16.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4234136870 con 0x7f42340965d0 2023-10-30T12:23:16.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423bd83640 10 monclient(hunting): authenticate will time out at 2241.791210s 2023-10-30T12:23:16.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42340965d0 0x7f42341c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42340965d0 0x7f42341c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60812/0 (socket says 172.21.15.50:60812) 2023-10-30T12:23:16.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 1 -- 172.21.15.50:0/3914425265 learned_addr learned my addr 172.21.15.50:0/3914425265 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:16.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 10 monclient(hunting): get_auth_request con 0x7f42340965d0 auth_method 0 2023-10-30T12:23:16.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423409d210 0x7f4234094210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): get_auth_request con 0x7f423409d210 auth_method 0 2023-10-30T12:23:16.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423a320640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4234138f30 0x7f4234094750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423a320640 10 monclient(hunting): get_auth_request con 0x7f4234138f30 auth_method 0 2023-10-30T12:23:16.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423a320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423a320640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423a320640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient(hunting): handle_auth_done global_id 264438 payload 824 2023-10-30T12:23:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 -- 172.21.15.50:0/3914425265 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42340965d0 msgr2=0x7f42341c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42340965d0 0x7f42341c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 -- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4234138f30 msgr2=0x7f4234094750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4234138f30 0x7f4234094750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 monclient: found mon.noname-a 2023-10-30T12:23:16.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 -- 172.21.15.50:0/3914425265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42280094d0 con 0x7f423409d210 2023-10-30T12:23:16.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f4239b1f640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f42340965d0 0x7f42341c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:16.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f423931e640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423409d210 0x7f4234094210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4228033000 tx=0x7f42280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f42267fc640 10 client.?.objecter ms_handle_connect 0x7f423409d210 2023-10-30T12:23:16.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.609+0000 7f42267fc640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:16.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228032860 con 0x7f423409d210 2023-10-30T12:23:16.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:16.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: dump: 2023-10-30T12:23:16.673 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.673 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.675 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.675 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.675 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.675 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42280329e0 con 0x7f423409d210 2023-10-30T12:23:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 5 monclient: authenticate success, global_id 264438 2023-10-30T12:23:16.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228032d90 con 0x7f423409d210 2023-10-30T12:23:16.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:16.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 10 monclient: dump: 2023-10-30T12:23:16.680 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.680 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.681 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.681 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.681 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.682 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.682 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.683 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.683 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 monclient: _renew_subs 2023-10-30T12:23:16.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423409a000 con 0x7f423409d210 2023-10-30T12:23:16.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f4227fff640 4 set_mon_vals no callback set 2023-10-30T12:23:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 client.264438.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 monclient: _renew_subs 2023-10-30T12:23:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423408e4b0 con 0x7f423409d210 2023-10-30T12:23:16.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 1 librados: init done 2023-10-30T12:23:16.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 client.264438.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 15 client.264438.objecter _session_command_op_assign -1 1 2023-10-30T12:23:16.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 20 client.264438.objecter _get_session osd=-1 returning homeless 2023-10-30T12:23:16.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 client.264438.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:16.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:16.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f423409ba70 con 0x7f423409d210 2023-10-30T12:23:16.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.610+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f422804e450 con 0x7f423409d210 2023-10-30T12:23:16.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4218039320 0x7f421803b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f422806ca80 con 0x7f423409d210 2023-10-30T12:23:16.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 10 client.264438.objecter ms_dispatch 0x7f4234000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:16.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 3 client.264438.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:16.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 3 client.264438.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:16.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 10 client.264438.objecter checking command 1 2023-10-30T12:23:16.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f42180406b0 0x7f4218042b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 20 client.264438.objecter _get_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 20 client.264438.objecter put_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 15 client.264438.objecter _session_command_op_remove -1 1 2023-10-30T12:23:16.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 20 client.264438.objecter _get_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 20 client.264438.objecter get_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 15 client.264438.objecter _session_command_op_assign 4 1 2023-10-30T12:23:16.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 20 client.264438.objecter put_session s=0x7f421803ffe0 osd=4 4 2023-10-30T12:23:16.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 10 client.264438.objecter _send_command 1 2023-10-30T12:23:16.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 --> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4218043220 con 0x7f42180406b0 2023-10-30T12:23:16.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 20 client.264438.objecter dump_active .. 0 homeless 2023-10-30T12:23:16.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=686) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42280712c0 con 0x7f423409d210 2023-10-30T12:23:16.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 10 monclient: handle_get_version_reply finishing 1 version 686 2023-10-30T12:23:16.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f4239b1f640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4218039320 0x7f421803b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f4239b1f640 10 monclient: get_auth_request con 0x7f4218039320 auth_method 0 2023-10-30T12:23:16.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f423a320640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f42180406b0 0x7f4218042b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f423a320640 10 monclient: get_auth_request con 0x7f42180406b0 auth_method 0 2023-10-30T12:23:16.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f42267fc640 10 client.264438.objecter ms_handle_connect 0x7f4218039320 2023-10-30T12:23:16.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.611+0000 7f4239b1f640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4218039320 0x7f421803b7e0 secure :-1 s=READY pgs=4366 cs=0 l=1 rev1=1 crypto rx=0x7f4230007fd0 tx=0x7f4230009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.612+0000 7f42267fc640 10 client.264438.objecter ms_handle_connect 0x7f42180406b0 2023-10-30T12:23:16.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.612+0000 7f423a320640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f42180406b0 0x7f4218042b50 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.612+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== osd.4 v2:172.21.15.118:6809/4272127357 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4218043220 con 0x7f42180406b0 2023-10-30T12:23:16.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.612+0000 7f42267fc640 10 client.264438.objecter ms_dispatch 0x7f4234000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:23:16.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.612+0000 7f42267fc640 10 client.264438.objecter _finish_command 1 = system:0 2023-10-30T12:23:16.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.612+0000 7f42267fc640 20 client.264438.objecter put_session s=0x7f421803ffe0 osd=4 4 2023-10-30T12:23:16.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.612+0000 7f42267fc640 15 client.264438.objecter _session_command_op_remove 4 1 2023-10-30T12:23:16.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 10 client.264438.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:23:16.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 15 client.264438.objecter _session_command_op_assign -1 2 2023-10-30T12:23:16.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 20 client.264438.objecter _get_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 20 client.264438.objecter put_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 20 client.264438.objecter _get_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 15 client.264438.objecter _session_command_op_remove -1 2 2023-10-30T12:23:16.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 20 client.264438.objecter get_session s=0x7f421803ffe0 osd=4 3 2023-10-30T12:23:16.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 15 client.264438.objecter _session_command_op_assign 4 2 2023-10-30T12:23:16.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 20 client.264438.objecter put_session s=0x7f421803ffe0 osd=4 4 2023-10-30T12:23:16.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 10 client.264438.objecter _send_command 2 2023-10-30T12:23:16.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.625+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 --> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f423409c5b0 con 0x7f42180406b0 2023-10-30T12:23:16.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f42267fc640 1 -- 172.21.15.50:0/3914425265 <== osd.4 v2:172.21.15.118:6809/4272127357 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f422c00f040 con 0x7f42180406b0 2023-10-30T12:23:16.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f42267fc640 10 client.264438.objecter ms_dispatch 0x7f4234000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:23:16.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f42267fc640 10 client.264438.objecter _finish_command 2 = system:0 2023-10-30T12:23:16.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f42267fc640 20 client.264438.objecter put_session s=0x7f421803ffe0 osd=4 4 2023-10-30T12:23:16.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f42267fc640 15 client.264438.objecter _session_command_op_remove 4 2 2023-10-30T12:23:16.707 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:23:16.708 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:16.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 10 librados: watch_flush enter 2023-10-30T12:23:16.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 10 librados: watch_flush exit 2023-10-30T12:23:16.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 10 client.264438.objecter close_session for osd.4 2023-10-30T12:23:16.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f42180406b0 msgr2=0x7f4218042b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6809/4272127357,v1:172.21.15.118:6811/4272127357] conn(0x7f42180406b0 0x7f4218042b50 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 20 client.264438.objecter put_session s=0x7f421803ffe0 osd=4 1 2023-10-30T12:23:16.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 20 client.264438.objecter shutdown clearing up homeless session... 2023-10-30T12:23:16.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 10 client.264438.objecter successfully canceled tick 2023-10-30T12:23:16.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4218039320 msgr2=0x7f421803b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f4218039320 0x7f421803b7e0 secure :-1 s=READY pgs=4366 cs=0 l=1 rev1=1 crypto rx=0x7f4230007fd0 tx=0x7f4230009040 comp rx=0 tx=0).stop 2023-10-30T12:23:16.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 10 monclient: shutdown 2023-10-30T12:23:16.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:16.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423409d210 msgr2=0x7f4234094210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 --2- 172.21.15.50:0/3914425265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423409d210 0x7f4234094210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4228033000 tx=0x7f42280046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423a320640 1 -- 172.21.15.50:0/3914425265 reap_dead start 2023-10-30T12:23:16.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 shutdown_connections 2023-10-30T12:23:16.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 >> 172.21.15.50:0/3914425265 conn(0x7f42340a0480 msgr2=0x7f42340a1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:16.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 shutdown_connections 2023-10-30T12:23:16.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 -- 172.21.15.50:0/3914425265 wait complete. 2023-10-30T12:23:16.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.626+0000 7f423bd83640 1 librados: shutdown 2023-10-30T12:23:16.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T12:23:16.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.824+0000 7fd5eb87c640 10 monclient: get_monmap_and_config 2023-10-30T12:23:16.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.824+0000 7fd5eb87c640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.824+0000 7fd5eb87c640 10 monclient: monmap: 2023-10-30T12:23:16.826 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.826 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.826 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:16.824696+0000 2023-10-30T12:23:16.827 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:16.824696+0000 2023-10-30T12:23:16.827 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.827 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.827 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:16.828 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:16.828 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:16.828 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:16.829 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:16.829 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 1 Processor -- start 2023-10-30T12:23:16.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 1 -- start start 2023-10-30T12:23:16.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient: init 2023-10-30T12:23:16.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-10-30T12:23:16.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5e4135ba0 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:16.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd5e4135670 0x7fd5e413a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e4135670 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:16.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e413aed0 0x7fd5e413d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e413aed0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e409a870 con 0x7fd5e4135ba0 2023-10-30T12:23:16.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e409d550 con 0x7fd5e4135670 2023-10-30T12:23:16.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e409a040 con 0x7fd5e413aed0 2023-10-30T12:23:16.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.825+0000 7fd5eb87c640 10 monclient(hunting): authenticate will time out at 2242.007874s 2023-10-30T12:23:16.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e413aed0 0x7fd5e413d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e413aed0 0x7fd5e413d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60820/0 (socket says 172.21.15.50:60820) 2023-10-30T12:23:16.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 -- 172.21.15.50:0/1272575166 learned_addr learned my addr 172.21.15.50:0/1272575166 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:16.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): get_auth_request con 0x7fd5e413aed0 auth_method 0 2023-10-30T12:23:16.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 1 --2- 172.21.15.50:0/1272575166 >> v2:172.21.15.118:3300/0 conn(0x7fd5e4135670 0x7fd5e413a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 10 monclient(hunting): get_auth_request con 0x7fd5e4135670 auth_method 0 2023-10-30T12:23:16.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e8e17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient(hunting): handle_auth_done global_id 284189 payload 274 2023-10-30T12:23:16.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 -- 172.21.15.50:0/1272575166 >> v2:172.21.15.118:3300/0 conn(0x7fd5e4135670 msgr2=0x7fd5e413a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:16.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 --2- 172.21.15.50:0/1272575166 >> v2:172.21.15.118:3300/0 conn(0x7fd5e4135670 0x7fd5e413a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 -- 172.21.15.50:0/1272575166 >> v1:172.21.15.118:6789/0 conn(0x7fd5e4135ba0 legacy=0x7fd5e4135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 monclient: found mon.noname-b 2023-10-30T12:23:16.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.826+0000 7fd5e9e19640 1 -- 172.21.15.50:0/1272575166 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e409a2b0 con 0x7fd5e413aed0 2023-10-30T12:23:16.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5e9e19640 1 --2- 172.21.15.50:0/1272575166 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e413aed0 0x7fd5e413d2c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009e50 tx=0x7fd5dc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7a0c1d337aabfb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 1 -- 172.21.15.50:0/1272575166 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc02cb00 con 0x7fd5e413aed0 2023-10-30T12:23:16.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:16.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: dump: 2023-10-30T12:23:16.850 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.851 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.851 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.851 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.851 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.852 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.852 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.852 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.852 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.853 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 5 monclient: authenticate success, global_id 284189 2023-10-30T12:23:16.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 1 -- 172.21.15.50:0/1272575166 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc02cc60 con 0x7fd5e413aed0 2023-10-30T12:23:16.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 4 set_mon_vals no callback set 2023-10-30T12:23:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 1 -- 172.21.15.50:0/1272575166 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0343f0 con 0x7fd5e413aed0 2023-10-30T12:23:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5dbfff640 10 monclient: dump: 2023-10-30T12:23:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.858 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.858 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.858 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.858 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.859 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 10 monclient: shutdown 2023-10-30T12:23:16.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:16.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 -- 172.21.15.50:0/1272575166 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e413aed0 msgr2=0x7fd5e413d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/1272575166 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e413aed0 0x7fd5e413d2c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009e50 tx=0x7fd5dc02bf60 comp rx=0 tx=0).stop 2023-10-30T12:23:16.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 -- 172.21.15.50:0/1272575166 shutdown_connections 2023-10-30T12:23:16.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/1272575166 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e413aed0 0x7fd5e413d2c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/1272575166 >> v2:172.21.15.118:3300/0 conn(0x7fd5e4135670 0x7fd5e413a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 -- 172.21.15.50:0/1272575166 >> 172.21.15.50:0/1272575166 conn(0x7fd5e40a4500 msgr2=0x7fd5e40a0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 -- 172.21.15.50:0/1272575166 shutdown_connections 2023-10-30T12:23:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.827+0000 7fd5eb87c640 1 -- 172.21.15.50:0/1272575166 wait complete. 2023-10-30T12:23:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 10 monclient: build_initial_monmap 2023-10-30T12:23:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 10 monclient: monmap: 2023-10-30T12:23:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:16.865 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 1 librados: starting msgr at 2023-10-30T12:23:16.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 1 librados: starting objecter 2023-10-30T12:23:16.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 1 Processor -- start 2023-10-30T12:23:16.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 1 -- start start 2023-10-30T12:23:16.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 1 librados: setting wanted keys 2023-10-30T12:23:16.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 1 librados: calling monclient init 2023-10-30T12:23:16.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.828+0000 7fd5eb87c640 10 monclient: init 2023-10-30T12:23:16.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:16.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient: _add_conns ranks=[1,2,0] 2023-10-30T12:23:16.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5e4135670 0x7fd5e41344e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e4135670 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:16.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e4135ba0 0x7fd5e411ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e4135ba0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:16.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5e413aed0 0x7fd5e411d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e413aed0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:16.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e41cb010 con 0x7fd5e413aed0 2023-10-30T12:23:16.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e409d700 con 0x7fd5e4135670 2023-10-30T12:23:16.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:16.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e409bb80 con 0x7fd5e4135ba0 2023-10-30T12:23:16.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:16.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5eb87c640 10 monclient(hunting): authenticate will time out at 2242.011366s 2023-10-30T12:23:16.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e4135ba0 0x7fd5e411ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e4135ba0 0x7fd5e411ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60826/0 (socket says 172.21.15.50:60826) 2023-10-30T12:23:16.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 -- 172.21.15.50:0/2219619007 learned_addr learned my addr 172.21.15.50:0/2219619007 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:16.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): get_auth_request con 0x7fd5e4135ba0 auth_method 0 2023-10-30T12:23:16.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5e413aed0 0x7fd5e411d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 10 monclient(hunting): get_auth_request con 0x7fd5e413aed0 auth_method 0 2023-10-30T12:23:16.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5e4135670 0x7fd5e41344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 10 monclient(hunting): get_auth_request con 0x7fd5e4135670 auth_method 0 2023-10-30T12:23:16.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:16.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:16.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:16.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9e19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:16.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:16.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e9618640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:16.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient(hunting): handle_auth_done global_id 284192 payload 824 2023-10-30T12:23:16.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 -- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5e4135670 msgr2=0x7fd5e41344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5e4135670 0x7fd5e41344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 -- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5e413aed0 msgr2=0x7fd5e411d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:16.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5e413aed0 0x7fd5e411d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:16.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient: _finish_hunting 0 2023-10-30T12:23:16.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 monclient: found mon.noname-c 2023-10-30T12:23:16.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:16.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 -- 172.21.15.50:0/2219619007 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc009b30 con 0x7fd5e4135ba0 2023-10-30T12:23:16.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5e8e17640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e4135ba0 0x7fd5e411ce20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400e9b0 tx=0x7fd5d400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.829+0000 7fd5d9ffb640 10 client.?.objecter ms_handle_connect 0x7fd5e4135ba0 2023-10-30T12:23:16.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:16.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 1 -- 172.21.15.50:0/2219619007 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4009670 con 0x7fd5e4135ba0 2023-10-30T12:23:16.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: dump: 2023-10-30T12:23:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.885 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.885 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.886 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.886 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.886 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.886 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.887 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: _finish_auth 0 2023-10-30T12:23:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:23:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:16.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 5 monclient: authenticate success, global_id 284192 2023-10-30T12:23:16.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 1 -- 172.21.15.50:0/2219619007 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d400c9a0 con 0x7fd5e4135ba0 2023-10-30T12:23:16.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 10 monclient: _renew_subs 2023-10-30T12:23:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e41205f0 con 0x7fd5e4135ba0 2023-10-30T12:23:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 1 -- 172.21.15.50:0/2219619007 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d400cd50 con 0x7fd5e4135ba0 2023-10-30T12:23:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5d9ffb640 10 monclient: dump: 2023-10-30T12:23:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:16.891 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:16.891 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.891 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:16.891 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:16.893 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:16.893 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:16.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 10 client.284192.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:16.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 10 monclient: _renew_subs 2023-10-30T12:23:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5db7fe640 4 set_mon_vals no callback set 2023-10-30T12:23:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e411eac0 con 0x7fd5e4135ba0 2023-10-30T12:23:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.830+0000 7fd5eb87c640 1 librados: init done 2023-10-30T12:23:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5d9ffb640 1 -- 172.21.15.50:0/2219619007 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fd5d4013470 con 0x7fd5e4135ba0 2023-10-30T12:23:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5eb87c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5eb87c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5eb87c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:16.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5d9ffb640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5c80392f0 0x7fd5c803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:16.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e404e160 con 0x7fd5e4135ba0 2023-10-30T12:23:16.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5d9ffb640 1 -- 172.21.15.50:0/2219619007 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7fd5d404ac00 con 0x7fd5e4135ba0 2023-10-30T12:23:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5d9ffb640 10 client.284192.objecter ms_dispatch 0x7fd5e4000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5d9ffb640 3 client.284192.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5d9ffb640 3 client.284192.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5d9ffb640 20 client.284192.objecter dump_active .. 0 homeless 2023-10-30T12:23:16.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5e9618640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5c80392f0 0x7fd5c803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:16.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.831+0000 7fd5e9618640 10 monclient: get_auth_request con 0x7fd5c80392f0 auth_method 0 2023-10-30T12:23:16.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.832+0000 7fd5d9ffb640 10 client.284192.objecter ms_handle_connect 0x7fd5c80392f0 2023-10-30T12:23:16.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.832+0000 7fd5e9618640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5c80392f0 0x7fd5c803b7b0 secure :-1 s=READY pgs=4367 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0000f60 tx=0x7fd5e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.836+0000 7fd5d9ffb640 1 -- 172.21.15.50:0/2219619007 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd5d40232f0 con 0x7fd5e4135ba0 2023-10-30T12:23:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.836+0000 7fd5d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:16.836+0000 7fd5d9ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:17.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.011+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5e409a590 con 0x7fd5c80392f0 2023-10-30T12:23:17.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.011+0000 7fd5d9ffb640 1 -- 172.21.15.50:0/2219619007 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd5e00014a0 con 0x7fd5c80392f0 2023-10-30T12:23:17.012 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T12:23:17.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 10 librados: watch_flush enter 2023-10-30T12:23:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 10 librados: watch_flush exit 2023-10-30T12:23:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 20 client.284192.objecter shutdown clearing up homeless session... 2023-10-30T12:23:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 10 client.284192.objecter successfully canceled tick 2023-10-30T12:23:17.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5c80392f0 msgr2=0x7fd5c803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5c80392f0 0x7fd5c803b7b0 secure :-1 s=READY pgs=4367 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0000f60 tx=0x7fd5e0009040 comp rx=0 tx=0).stop 2023-10-30T12:23:17.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 10 monclient: shutdown 2023-10-30T12:23:17.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:17.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e4135ba0 msgr2=0x7fd5e411ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e4135ba0 0x7fd5e411ce20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400e9b0 tx=0x7fd5d400ee70 comp rx=0 tx=0).stop 2023-10-30T12:23:17.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 shutdown_connections 2023-10-30T12:23:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fd5c80392f0 0x7fd5c803b7b0 unknown :-1 s=CLOSED pgs=4367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5e413aed0 0x7fd5e411d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd5e4135ba0 0x7fd5e411ce20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 --2- 172.21.15.50:0/2219619007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5e4135670 0x7fd5e41344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 >> 172.21.15.50:0/2219619007 conn(0x7fd5e40a4500 msgr2=0x7fd5e40a2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:17.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.013+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 shutdown_connections 2023-10-30T12:23:17.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.014+0000 7fd5eb87c640 1 -- 172.21.15.50:0/2219619007 wait complete. 2023-10-30T12:23:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.014+0000 7fd5eb87c640 1 librados: shutdown 2023-10-30T12:23:17.024 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T12:23:17.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:23:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient: get_monmap_and_config 2023-10-30T12:23:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient: build_initial_monmap 2023-10-30T12:23:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient: monmap: 2023-10-30T12:23:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:17.132799+0000 2023-10-30T12:23:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:17.132799+0000 2023-10-30T12:23:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:17.137 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 1 Processor -- start 2023-10-30T12:23:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 1 -- start start 2023-10-30T12:23:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient: init 2023-10-30T12:23:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-10-30T12:23:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2f40099ee0 0x7f2f40097d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient(hunting): picked mon.noname-c con 0x7f2f40099ee0 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f4009aa60 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f4009b5b0 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4009cdd0 con 0x7f2f4009aa60 2023-10-30T12:23:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400723f0 con 0x7f2f4009b5b0 2023-10-30T12:23:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4009a2c0 con 0x7f2f40099ee0 2023-10-30T12:23:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.132+0000 7f2f46d90640 10 monclient(hunting): authenticate will time out at 2242.315922s 2023-10-30T12:23:17.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3ffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2f4009aa60 0x7f2f4009aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49698/0 (socket says 172.21.15.50:49698) 2023-10-30T12:23:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3ffff640 1 -- 172.21.15.50:0/2972117467 learned_addr learned my addr 172.21.15.50:0/2972117467 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 1 --2- 172.21.15.50:0/2972117467 >> v2:172.21.15.118:3300/0 conn(0x7f2f40099ee0 0x7f2f40097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): get_auth_request con 0x7f2f40099ee0 auth_method 0 2023-10-30T12:23:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 1 -- 172.21.15.50:0/2972117467 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053153747 0 0) 0x7f2f4009cdd0 con 0x7f2f4009aa60 2023-10-30T12:23:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:23:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): my global_id is 264444 2023-10-30T12:23:17.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 1 -- 172.21.15.50:0/2972117467 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f30004630 con 0x7f2f4009aa60 2023-10-30T12:23:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 1 -- 172.21.15.50:0/2972117467 <== mon.1 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717165996 0 0) 0x7f2f38001490 con 0x7f2f4009b5b0 2023-10-30T12:23:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): _negotiate 2023-10-30T12:23:17.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 10 monclient(hunting): my global_id is 274285 2023-10-30T12:23:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f3f7fe640 1 -- 172.21.15.50:0/2972117467 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f4009cdd0 con 0x7f2f4009b5b0 2023-10-30T12:23:17.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_done global_id 274282 payload 274 2023-10-30T12:23:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 1 -- 172.21.15.50:0/2972117467 >> v1:172.21.15.118:6789/0 conn(0x7f2f4009b5b0 legacy=0x7f2f401397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 1 -- 172.21.15.50:0/2972117467 >> v1:172.21.15.50:6789/0 conn(0x7f2f4009aa60 legacy=0x7f2f4009aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:17.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 1 monclient: found mon.noname-c 2023-10-30T12:23:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.118:3300/0 2023-10-30T12:23:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.133+0000 7f2f44b2c640 1 -- 172.21.15.50:0/2972117467 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4009c300 con 0x7f2f40099ee0 2023-10-30T12:23:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f44b2c640 1 --2- 172.21.15.50:0/2972117467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f40099ee0 0x7f2f40097d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f340097a0 tx=0x7f2f3402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a76af6ac2148ddb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 1 -- 172.21.15.50:0/2972117467 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3402e070 con 0x7f2f40099ee0 2023-10-30T12:23:17.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:17.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: dump: 2023-10-30T12:23:17.159 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.159 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.159 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.160 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.160 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.160 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.160 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.161 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.161 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.161 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: _finish_auth 0 2023-10-30T12:23:17.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:23:17.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:17.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 1 -- 172.21.15.50:0/2972117467 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f340315a0 con 0x7f2f40099ee0 2023-10-30T12:23:17.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 5 monclient: authenticate success, global_id 274282 2023-10-30T12:23:17.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:17.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:17.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:17.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 4 set_mon_vals no callback set 2023-10-30T12:23:17.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 1 -- 172.21.15.50:0/2972117467 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34031910 con 0x7f2f40099ee0 2023-10-30T12:23:17.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-10-30T12:23:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f3f7fe640 10 monclient: dump: 2023-10-30T12:23:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.167 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 10 monclient: shutdown 2023-10-30T12:23:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 1 -- 172.21.15.50:0/2972117467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f40099ee0 msgr2=0x7f2f40097d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.134+0000 7f2f46d90640 1 --2- 172.21.15.50:0/2972117467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f40099ee0 0x7f2f40097d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f340097a0 tx=0x7f2f3402b840 comp rx=0 tx=0).stop 2023-10-30T12:23:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 -- 172.21.15.50:0/2972117467 shutdown_connections 2023-10-30T12:23:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 --2- 172.21.15.50:0/2972117467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f40099ee0 0x7f2f40097d90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 -- 172.21.15.50:0/2972117467 >> 172.21.15.50:0/2972117467 conn(0x7f2f400a0480 msgr2=0x7f2f400a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 -- 172.21.15.50:0/2972117467 shutdown_connections 2023-10-30T12:23:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 -- 172.21.15.50:0/2972117467 wait complete. 2023-10-30T12:23:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 10 monclient: build_initial_monmap 2023-10-30T12:23:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 10 monclient: monmap: 2023-10-30T12:23:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:17.173 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 librados: starting msgr at 2023-10-30T12:23:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 librados: starting objecter 2023-10-30T12:23:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 Processor -- start 2023-10-30T12:23:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 -- start start 2023-10-30T12:23:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 librados: setting wanted keys 2023-10-30T12:23:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 1 librados: calling monclient init 2023-10-30T12:23:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.135+0000 7f2f46d90640 10 monclient: init 2023-10-30T12:23:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient: _add_conns ranks=[0,1,2] 2023-10-30T12:23:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f40099ee0 0x7f2f40095d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): picked mon.noname-a con 0x7f2f40099ee0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4009aa60 0x7f2f400915e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4009aa60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2f4009b5b0 0x7f2f40091b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): picked mon.noname-c con 0x7f2f4009b5b0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40095390 con 0x7f2f40099ee0 2023-10-30T12:23:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40094cd0 con 0x7f2f4009aa60 2023-10-30T12:23:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4009ca30 con 0x7f2f4009b5b0 2023-10-30T12:23:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f46d90640 10 monclient(hunting): authenticate will time out at 2242.319533s 2023-10-30T12:23:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f40099ee0 0x7f2f40095d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f40099ee0 0x7f2f40095d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59936/0 (socket says 172.21.15.50:59936) 2023-10-30T12:23:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 -- 172.21.15.50:0/497356677 learned_addr learned my addr 172.21.15.50:0/497356677 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:17.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): get_auth_request con 0x7f2f40099ee0 auth_method 0 2023-10-30T12:23:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2f4009b5b0 0x7f2f40091b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 10 monclient(hunting): get_auth_request con 0x7f2f4009b5b0 auth_method 0 2023-10-30T12:23:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f3ffff640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4009aa60 0x7f2f400915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f4532d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f3ffff640 10 monclient(hunting): get_auth_request con 0x7f2f4009aa60 auth_method 0 2023-10-30T12:23:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f3ffff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f3ffff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient(hunting): handle_auth_done global_id 264447 payload 824 2023-10-30T12:23:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 -- 172.21.15.50:0/497356677 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2f4009b5b0 msgr2=0x7f2f40091b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2f4009b5b0 0x7f2f40091b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 -- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4009aa60 msgr2=0x7f2f400915e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4009aa60 0x7f2f400915e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 1 monclient: found mon.noname-a 2023-10-30T12:23:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.136+0000 7f2f44b2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f44b2c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f44b2c640 1 -- 172.21.15.50:0/497356677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f34009400 con 0x7f2f40099ee0 2023-10-30T12:23:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3ffff640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f4009aa60 0x7f2f400915e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:23:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f44b2c640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f40099ee0 0x7f2f40095d00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f34034040 tx=0x7f2f340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 client.?.objecter ms_handle_connect 0x7f2f40099ee0 2023-10-30T12:23:17.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34032630 con 0x7f2f40099ee0 2023-10-30T12:23:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: dump: 2023-10-30T12:23:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.193 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: _finish_auth 0 2023-10-30T12:23:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: _check_auth_tickets 2023-10-30T12:23:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f34032790 con 0x7f2f40099ee0 2023-10-30T12:23:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 5 monclient: authenticate success, global_id 264447 2023-10-30T12:23:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34032b00 con 0x7f2f40099ee0 2023-10-30T12:23:17.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 10 monclient: _renew_subs 2023-10-30T12:23:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f400a2b60 con 0x7f2f40099ee0 2023-10-30T12:23:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3d7fa640 10 monclient: dump: 2023-10-30T12:23:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.199 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f3effd640 4 set_mon_vals no callback set 2023-10-30T12:23:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 10 client.264447.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 10 monclient: _renew_subs 2023-10-30T12:23:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f40095570 con 0x7f2f40099ee0 2023-10-30T12:23:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.137+0000 7f2f46d90640 1 librados: init done 2023-10-30T12:23:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f2f34032c60 con 0x7f2f40099ee0 2023-10-30T12:23:17.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f46d90640 10 client.264447.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f46d90640 15 client.264447.objecter _session_command_op_assign -1 1 2023-10-30T12:23:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f46d90640 20 client.264447.objecter _get_session osd=-1 returning homeless 2023-10-30T12:23:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f46d90640 10 client.264447.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f46d90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f40097690 con 0x7f2f40099ee0 2023-10-30T12:23:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2f24039320 0x7f2f2403b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f2f3406c340 con 0x7f2f40099ee0 2023-10-30T12:23:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 10 client.264447.objecter ms_dispatch 0x7f2f40000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 3 client.264447.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 3 client.264447.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 10 client.264447.objecter checking command 1 2023-10-30T12:23:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7f2f240406b0 0x7f2f24042b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 20 client.264447.objecter _get_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 20 client.264447.objecter put_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 15 client.264447.objecter _session_command_op_remove -1 1 2023-10-30T12:23:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 20 client.264447.objecter _get_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 20 client.264447.objecter get_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 15 client.264447.objecter _session_command_op_assign 5 1 2023-10-30T12:23:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 20 client.264447.objecter put_session s=0x7f2f2403ffe0 osd=5 4 2023-10-30T12:23:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 10 client.264447.objecter _send_command 1 2023-10-30T12:23:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 --> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f24043220 con 0x7f2f240406b0 2023-10-30T12:23:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 20 client.264447.objecter dump_active .. 0 homeless 2023-10-30T12:23:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=686) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f3406e9d0 con 0x7f2f40099ee0 2023-10-30T12:23:17.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3d7fa640 10 monclient: handle_get_version_reply finishing 1 version 686 2023-10-30T12:23:17.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3ffff640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2f24039320 0x7f2f2403b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f3ffff640 10 monclient: get_auth_request con 0x7f2f24039320 auth_method 0 2023-10-30T12:23:17.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f4532d640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7f2f240406b0 0x7f2f24042b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.138+0000 7f2f4532d640 10 monclient: get_auth_request con 0x7f2f240406b0 auth_method 0 2023-10-30T12:23:17.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3d7fa640 10 client.264447.objecter ms_handle_connect 0x7f2f24039320 2023-10-30T12:23:17.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3d7fa640 10 client.264447.objecter ms_handle_connect 0x7f2f240406b0 2023-10-30T12:23:17.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3ffff640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2f24039320 0x7f2f2403b7e0 secure :-1 s=READY pgs=4368 cs=0 l=1 rev1=1 crypto rx=0x7f2f40092e30 tx=0x7f2f2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f4532d640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7f2f240406b0 0x7f2f24042b50 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== osd.5 v2:172.21.15.118:6800/1399177807 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f2f24043220 con 0x7f2f240406b0 2023-10-30T12:23:17.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3d7fa640 10 client.264447.objecter ms_dispatch 0x7f2f40000b90 command_reply(tid 1: 0 ) v1 2023-10-30T12:23:17.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3d7fa640 10 client.264447.objecter _finish_command 1 = system:0 2023-10-30T12:23:17.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3d7fa640 20 client.264447.objecter put_session s=0x7f2f2403ffe0 osd=5 4 2023-10-30T12:23:17.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.139+0000 7f2f3d7fa640 15 client.264447.objecter _session_command_op_remove 5 1 2023-10-30T12:23:17.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 10 client.264447.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-10-30T12:23:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 15 client.264447.objecter _session_command_op_assign -1 2 2023-10-30T12:23:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 20 client.264447.objecter _get_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 20 client.264447.objecter put_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 20 client.264447.objecter _get_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 15 client.264447.objecter _session_command_op_remove -1 2 2023-10-30T12:23:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 20 client.264447.objecter get_session s=0x7f2f2403ffe0 osd=5 3 2023-10-30T12:23:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 15 client.264447.objecter _session_command_op_assign 5 2 2023-10-30T12:23:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 20 client.264447.objecter put_session s=0x7f2f2403ffe0 osd=5 4 2023-10-30T12:23:17.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 10 client.264447.objecter _send_command 2 2023-10-30T12:23:17.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.152+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 --> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2f4009d7b0 con 0x7f2f240406b0 2023-10-30T12:23:17.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f3d7fa640 1 -- 172.21.15.50:0/497356677 <== osd.5 v2:172.21.15.118:6800/1399177807 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2f38001920 con 0x7f2f240406b0 2023-10-30T12:23:17.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f3d7fa640 10 client.264447.objecter ms_dispatch 0x7f2f40000b90 command_reply(tid 2: 0 ) v1 2023-10-30T12:23:17.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f3d7fa640 10 client.264447.objecter _finish_command 2 = system:0 2023-10-30T12:23:17.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f3d7fa640 20 client.264447.objecter put_session s=0x7f2f2403ffe0 osd=5 4 2023-10-30T12:23:17.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f3d7fa640 15 client.264447.objecter _session_command_op_remove 5 2 2023-10-30T12:23:17.217 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-10-30T12:23:17.217 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:23:17.217 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-10-30T12:23:17.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 10 librados: watch_flush enter 2023-10-30T12:23:17.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 10 librados: watch_flush exit 2023-10-30T12:23:17.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 10 client.264447.objecter close_session for osd.5 2023-10-30T12:23:17.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7f2f240406b0 msgr2=0x7f2f24042b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6800/1399177807,v1:172.21.15.118:6801/1399177807] conn(0x7f2f240406b0 0x7f2f24042b50 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 20 client.264447.objecter put_session s=0x7f2f2403ffe0 osd=5 1 2023-10-30T12:23:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 20 client.264447.objecter shutdown clearing up homeless session... 2023-10-30T12:23:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 10 client.264447.objecter successfully canceled tick 2023-10-30T12:23:17.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2f24039320 msgr2=0x7f2f2403b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f2f24039320 0x7f2f2403b7e0 secure :-1 s=READY pgs=4368 cs=0 l=1 rev1=1 crypto rx=0x7f2f40092e30 tx=0x7f2f2c009040 comp rx=0 tx=0).stop 2023-10-30T12:23:17.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 10 monclient: shutdown 2023-10-30T12:23:17.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:17.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f40099ee0 msgr2=0x7f2f40095d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 --2- 172.21.15.50:0/497356677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f40099ee0 0x7f2f40095d00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f34034040 tx=0x7f2f340046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f4532d640 1 -- 172.21.15.50:0/497356677 reap_dead start 2023-10-30T12:23:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 shutdown_connections 2023-10-30T12:23:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 >> 172.21.15.50:0/497356677 conn(0x7f2f400a0480 msgr2=0x7f2f40070860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 shutdown_connections 2023-10-30T12:23:17.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 -- 172.21.15.50:0/497356677 wait complete. 2023-10-30T12:23:17.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.153+0000 7f2f46d90640 1 librados: shutdown 2023-10-30T12:23:17.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T12:23:17.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.330+0000 7f3e49ba4640 10 monclient: get_monmap_and_config 2023-10-30T12:23:17.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.330+0000 7f3e49ba4640 10 monclient: build_initial_monmap 2023-10-30T12:23:17.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.330+0000 7f3e49ba4640 10 monclient: monmap: 2023-10-30T12:23:17.333 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:17.333 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.334 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:17.331932+0000 2023-10-30T12:23:17.334 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:17.331932+0000 2023-10-30T12:23:17.334 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:17.334 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.335 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:17.335 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:17.335 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:17.336 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:17.336 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:17.336 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.331+0000 7f3e49ba4640 1 Processor -- start 2023-10-30T12:23:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.331+0000 7f3e49ba4640 1 -- start start 2023-10-30T12:23:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.331+0000 7f3e49ba4640 10 monclient: init 2023-10-30T12:23:17.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:17.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-10-30T12:23:17.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e44132970 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:17.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e440a03d0 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:17.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e4413c200 0x7f3e4413e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient(hunting): picked mon.noname-a con 0x7f3e4413c200 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:17.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e440a17d0 con 0x7f3e44132970 2023-10-30T12:23:17.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e440a1510 con 0x7f3e440a03d0 2023-10-30T12:23:17.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e44133460 con 0x7f3e4413c200 2023-10-30T12:23:17.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:17.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e49ba4640 10 monclient(hunting): authenticate will time out at 2242.515106s 2023-10-30T12:23:17.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e4413c200 0x7f3e4413e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e437fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3e44132970 0x7f3e44132d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49702/0 (socket says 172.21.15.50:49702) 2023-10-30T12:23:17.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e4413c200 0x7f3e4413e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59942/0 (socket says 172.21.15.50:59942) 2023-10-30T12:23:17.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e437fe640 1 -- 172.21.15.50:0/1915119630 learned_addr learned my addr 172.21.15.50:0/1915119630 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:17.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): get_auth_request con 0x7f3e4413c200 auth_method 0 2023-10-30T12:23:17.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient(hunting): handle_auth_done global_id 264450 payload 274 2023-10-30T12:23:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 1 -- 172.21.15.50:0/1915119630 >> v1:172.21.15.118:6789/0 conn(0x7f3e440a03d0 legacy=0x7f3e4413aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 1 -- 172.21.15.50:0/1915119630 >> v1:172.21.15.50:6789/0 conn(0x7f3e44132970 legacy=0x7f3e44132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient: _finish_hunting 0 2023-10-30T12:23:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 1 monclient: found mon.noname-a 2023-10-30T12:23:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.332+0000 7f3e43fff640 1 -- 172.21.15.50:0/1915119630 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4409f680 con 0x7f3e4413c200 2023-10-30T12:23:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e43fff640 1 --2- 172.21.15.50:0/1915119630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413c200 0x7f3e4413e5f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009d90 tx=0x7f3e3c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53255c84a725e7af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 1 -- 172.21.15.50:0/1915119630 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c02cbd0 con 0x7f3e4413c200 2023-10-30T12:23:17.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: dump: 2023-10-30T12:23:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.369 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.370 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.371 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.371 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.371 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.372 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: _finish_auth 0 2023-10-30T12:23:17.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: _check_auth_tickets 2023-10-30T12:23:17.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:17.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 1 -- 172.21.15.50:0/1915119630 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c02cd30 con 0x7f3e4413c200 2023-10-30T12:23:17.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 5 monclient: authenticate success, global_id 264450 2023-10-30T12:23:17.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:17.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 1 -- 172.21.15.50:0/1915119630 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c02cea0 con 0x7f3e4413c200 2023-10-30T12:23:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e427fc640 10 monclient: dump: 2023-10-30T12:23:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.375 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.376 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.377 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.377 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.377 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:17.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 4 set_mon_vals no callback set 2023-10-30T12:23:17.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 10 monclient: shutdown 2023-10-30T12:23:17.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:17.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1915119630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413c200 msgr2=0x7f3e4413e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1915119630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413c200 0x7f3e4413e5f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009d90 tx=0x7f3e3c02be20 comp rx=0 tx=0).stop 2023-10-30T12:23:17.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1915119630 shutdown_connections 2023-10-30T12:23:17.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1915119630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413c200 0x7f3e4413e5f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.333+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1915119630 >> 172.21.15.50:0/1915119630 conn(0x7f3e440a4800 msgr2=0x7f3e440a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:17.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.334+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1915119630 shutdown_connections 2023-10-30T12:23:17.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.334+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1915119630 wait complete. 2023-10-30T12:23:17.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.334+0000 7f3e49ba4640 10 monclient: build_initial_monmap 2023-10-30T12:23:17.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.334+0000 7f3e49ba4640 10 monclient: monmap: 2023-10-30T12:23:17.384 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:17.384 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:17.385 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:17.385 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:17.386 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:17.386 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.387 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:17.387 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:17.388 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:17.388 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.334+0000 7f3e49ba4640 1 librados: starting msgr at 2023-10-30T12:23:17.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.334+0000 7f3e49ba4640 1 librados: starting objecter 2023-10-30T12:23:17.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.334+0000 7f3e49ba4640 1 Processor -- start 2023-10-30T12:23:17.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 -- start start 2023-10-30T12:23:17.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 librados: setting wanted keys 2023-10-30T12:23:17.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 librados: calling monclient init 2023-10-30T12:23:17.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient: init 2023-10-30T12:23:17.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:17.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient: _add_conns ranks=[2,1,0] 2023-10-30T12:23:17.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e44132970 0x7f3e44131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): picked mon.noname-c con 0x7f3e44132970 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:17.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4413c200 0x7f3e4413a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4413c200 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:17.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413a560 0x7f3e4413a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): picked mon.noname-a con 0x7f3e4413a560 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:17.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4411c390 con 0x7f3e4413a560 2023-10-30T12:23:17.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4409d7c0 con 0x7f3e4413c200 2023-10-30T12:23:17.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4413b890 con 0x7f3e44132970 2023-10-30T12:23:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e49ba4640 10 monclient(hunting): authenticate will time out at 2242.518524s 2023-10-30T12:23:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e44132970 0x7f3e44131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413a560 0x7f3e4413a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413a560 0x7f3e4413a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59954/0 (socket says 172.21.15.50:59954) 2023-10-30T12:23:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e44132970 0x7f3e44131290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60842/0 (socket says 172.21.15.50:60842) 2023-10-30T12:23:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 -- 172.21.15.50:0/1099789942 learned_addr learned my addr 172.21.15.50:0/1099789942 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:17.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): get_auth_request con 0x7f3e4413a560 auth_method 0 2023-10-30T12:23:17.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 10 monclient(hunting): get_auth_request con 0x7f3e44132970 auth_method 0 2023-10-30T12:23:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4413c200 0x7f3e4413a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 10 monclient(hunting): get_auth_request con 0x7f3e4413c200 auth_method 0 2023-10-30T12:23:17.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient(hunting): handle_auth_done global_id 264456 payload 824 2023-10-30T12:23:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 -- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e44132970 msgr2=0x7f3e44131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e44132970 0x7f3e44131290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 -- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4413c200 msgr2=0x7f3e4413a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4413c200 0x7f3e4413a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient: _finish_hunting 0 2023-10-30T12:23:17.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 monclient: found mon.noname-a 2023-10-30T12:23:17.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:17.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.335+0000 7f3e43fff640 1 -- 172.21.15.50:0/1099789942 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c009a70 con 0x7f3e4413a560 2023-10-30T12:23:17.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e437fe640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e44132970 0x7f3e44131290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:17.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e43fff640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413a560 0x7f3e4413a9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009d90 tx=0x7f3e3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 client.?.objecter ms_handle_connect 0x7f3e4413a560 2023-10-30T12:23:17.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:17.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 1 -- 172.21.15.50:0/1099789942 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c02cea0 con 0x7f3e4413a560 2023-10-30T12:23:17.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:17.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: dump: 2023-10-30T12:23:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.409 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.409 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.409 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.409 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.410 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.410 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.410 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: _finish_auth 0 2023-10-30T12:23:17.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:17.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:17.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 1 -- 172.21.15.50:0/1099789942 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c040650 con 0x7f3e4413a560 2023-10-30T12:23:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 1 -- 172.21.15.50:0/1099789942 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c040a00 con 0x7f3e4413a560 2023-10-30T12:23:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e2bfff640 10 monclient: dump: 2023-10-30T12:23:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.413 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.414 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.415 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.415 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 5 monclient: authenticate success, global_id 264456 2023-10-30T12:23:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 10 monclient: _renew_subs 2023-10-30T12:23:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e441cb150 con 0x7f3e4413a560 2023-10-30T12:23:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e41ffb640 4 set_mon_vals no callback set 2023-10-30T12:23:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 10 client.264456.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 10 monclient: _renew_subs 2023-10-30T12:23:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4411b220 con 0x7f3e4413a560 2023-10-30T12:23:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.336+0000 7f3e49ba4640 1 librados: init done 2023-10-30T12:23:17.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e2bfff640 1 -- 172.21.15.50:0/1099789942 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f3e3c040b60 con 0x7f3e4413a560 2023-10-30T12:23:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e49ba4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e49ba4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e49ba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4404e160 con 0x7f3e4413a560 2023-10-30T12:23:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e2bfff640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e300392f0 0x7f3e3003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e2bfff640 1 -- 172.21.15.50:0/1099789942 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f3e3c06d840 con 0x7f3e4413a560 2023-10-30T12:23:17.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e2bfff640 10 client.264456.objecter ms_dispatch 0x7f3e44000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e2bfff640 3 client.264456.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e2bfff640 3 client.264456.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e2bfff640 20 client.264456.objecter dump_active .. 0 homeless 2023-10-30T12:23:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e437fe640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e300392f0 0x7f3e3003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.337+0000 7f3e437fe640 10 monclient: get_auth_request con 0x7f3e300392f0 auth_method 0 2023-10-30T12:23:17.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.338+0000 7f3e2bfff640 10 client.264456.objecter ms_handle_connect 0x7f3e300392f0 2023-10-30T12:23:17.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.338+0000 7f3e437fe640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e300392f0 0x7f3e3003b7b0 secure :-1 s=READY pgs=4369 cs=0 l=1 rev1=1 crypto rx=0x7f3e34005880 tx=0x7f3e34009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.342+0000 7f3e2bfff640 1 -- 172.21.15.50:0/1099789942 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e3c03e400 con 0x7f3e4413a560 2023-10-30T12:23:17.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.342+0000 7f3e2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:17.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.342+0000 7f3e2bfff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:17.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.519+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e4409f960 con 0x7f3e300392f0 2023-10-30T12:23:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.519+0000 7f3e2bfff640 1 -- 172.21.15.50:0/1099789942 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3e34001970 con 0x7f3e300392f0 2023-10-30T12:23:17.521 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T12:23:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 10 librados: watch_flush enter 2023-10-30T12:23:17.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 10 librados: watch_flush exit 2023-10-30T12:23:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 20 client.264456.objecter shutdown clearing up homeless session... 2023-10-30T12:23:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 10 client.264456.objecter successfully canceled tick 2023-10-30T12:23:17.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e300392f0 msgr2=0x7f3e3003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e300392f0 0x7f3e3003b7b0 secure :-1 s=READY pgs=4369 cs=0 l=1 rev1=1 crypto rx=0x7f3e34005880 tx=0x7f3e34009040 comp rx=0 tx=0).stop 2023-10-30T12:23:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 10 monclient: shutdown 2023-10-30T12:23:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:17.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413a560 msgr2=0x7f3e4413a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413a560 0x7f3e4413a9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009d90 tx=0x7f3e3c004a40 comp rx=0 tx=0).stop 2023-10-30T12:23:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 shutdown_connections 2023-10-30T12:23:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f3e300392f0 0x7f3e3003b7b0 unknown :-1 s=CLOSED pgs=4369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e4413a560 0x7f3e4413a9f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4413c200 0x7f3e4413a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 --2- 172.21.15.50:0/1099789942 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3e44132970 0x7f3e44131290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 >> 172.21.15.50:0/1099789942 conn(0x7f3e440a4800 msgr2=0x7f3e4409bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 shutdown_connections 2023-10-30T12:23:17.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 -- 172.21.15.50:0/1099789942 wait complete. 2023-10-30T12:23:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.521+0000 7f3e49ba4640 1 librados: shutdown 2023-10-30T12:23:17.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:23:17.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.637+0000 7f1265083640 10 monclient: get_monmap_and_config 2023-10-30T12:23:17.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.637+0000 7f1265083640 10 monclient: build_initial_monmap 2023-10-30T12:23:17.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.637+0000 7f1265083640 10 monclient: monmap: 2023-10-30T12:23:17.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:17.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.640 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:17.638253+0000 2023-10-30T12:23:17.640 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:17.638253+0000 2023-10-30T12:23:17.641 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:17.641 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.641 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:17.641 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:17.642 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:17.642 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:17.642 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:17.643 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.637+0000 7f1265083640 1 Processor -- start 2023-10-30T12:23:17.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.637+0000 7f1265083640 1 -- start start 2023-10-30T12:23:17.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.637+0000 7f1265083640 10 monclient: init 2023-10-30T12:23:17.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:17.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-10-30T12:23:17.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12600a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:17.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260099a40 0x7f126013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient(hunting): picked mon.noname-b con 0x7f1260099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:17.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1260132db0 0x7f126013cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient(hunting): picked mon.noname-a con 0x7f1260132db0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:17.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600a4f40 con 0x7f12600a4b60 2023-10-30T12:23:17.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600a3d70 con 0x7f1260132db0 2023-10-30T12:23:17.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600a4150 con 0x7f1260099a40 2023-10-30T12:23:17.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:17.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f1265083640 10 monclient(hunting): authenticate will time out at 2242.821326s 2023-10-30T12:23:17.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1260132db0 0x7f126013cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260099a40 0x7f126013aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260099a40 0x7f126013aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59068/0 (socket says 172.21.15.50:59068) 2023-10-30T12:23:17.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 -- 172.21.15.50:0/64717330 learned_addr learned my addr 172.21.15.50:0/64717330 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:17.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): get_auth_request con 0x7f1260099a40 auth_method 0 2023-10-30T12:23:17.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1260132db0 0x7f126013cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39478/0 (socket says 172.21.15.50:39478) 2023-10-30T12:23:17.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 10 monclient(hunting): get_auth_request con 0x7f1260132db0 auth_method 0 2023-10-30T12:23:17.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient(hunting): handle_auth_done global_id 284201 payload 274 2023-10-30T12:23:17.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 -- 172.21.15.50:0/64717330 >> v2:172.21.15.50:3300/0 conn(0x7f1260132db0 msgr2=0x7f126013cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:17.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 --2- 172.21.15.50:0/64717330 >> v2:172.21.15.50:3300/0 conn(0x7f1260132db0 0x7f126013cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 -- 172.21.15.50:0/64717330 >> v1:172.21.15.118:6789/0 conn(0x7f12600a4b60 legacy=0x7f12601326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:17.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:17.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 monclient: found mon.noname-b 2023-10-30T12:23:17.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:17.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:17.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125e59c640 1 -- 172.21.15.50:0/64717330 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126009a440 con 0x7f1260099a40 2023-10-30T12:23:17.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.638+0000 7f125f59e640 1 --2- 172.21.15.50:0/64717330 >> v2:172.21.15.50:3300/0 conn(0x7f1260132db0 0x7f126013cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:17.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125e59c640 1 --2- 172.21.15.50:0/64717330 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260099a40 0x7f126013aa80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1248009870 tx=0x7f124802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=424e349daf8918a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 1 -- 172.21.15.50:0/64717330 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12480317c0 con 0x7f1260099a40 2023-10-30T12:23:17.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:17.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: dump: 2023-10-30T12:23:17.664 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.664 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.665 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.665 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.665 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.665 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.666 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.666 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.666 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.666 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: _finish_auth 0 2023-10-30T12:23:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: _check_auth_tickets 2023-10-30T12:23:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f1265083640 5 monclient: authenticate success, global_id 284201 2023-10-30T12:23:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f1265083640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 1 -- 172.21.15.50:0/64717330 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1248031920 con 0x7f1260099a40 2023-10-30T12:23:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f1265083640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f1265083640 4 set_mon_vals no callback set 2023-10-30T12:23:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 1 -- 172.21.15.50:0/64717330 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248031cb0 con 0x7f1260099a40 2023-10-30T12:23:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.639+0000 7f125dd9b640 10 monclient: dump: 2023-10-30T12:23:17.671 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.671 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.671 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.672 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.672 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.672 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.672 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.673 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 10 monclient: shutdown 2023-10-30T12:23:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 -- 172.21.15.50:0/64717330 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260099a40 msgr2=0x7f126013aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 --2- 172.21.15.50:0/64717330 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260099a40 0x7f126013aa80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1248009870 tx=0x7f124802b860 comp rx=0 tx=0).stop 2023-10-30T12:23:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 -- 172.21.15.50:0/64717330 shutdown_connections 2023-10-30T12:23:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 --2- 172.21.15.50:0/64717330 >> v2:172.21.15.50:3300/0 conn(0x7f1260132db0 0x7f126013cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 --2- 172.21.15.50:0/64717330 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260099a40 0x7f126013aa80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 -- 172.21.15.50:0/64717330 >> 172.21.15.50:0/64717330 conn(0x7f12600a8060 msgr2=0x7f12600a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 -- 172.21.15.50:0/64717330 shutdown_connections 2023-10-30T12:23:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 1 -- 172.21.15.50:0/64717330 wait complete. 2023-10-30T12:23:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 10 monclient: build_initial_monmap 2023-10-30T12:23:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.640+0000 7f1265083640 10 monclient: monmap: 2023-10-30T12:23:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:17.678 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:17.678 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:17.678 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:17.679 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 librados: starting msgr at 2023-10-30T12:23:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 librados: starting objecter 2023-10-30T12:23:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 Processor -- start 2023-10-30T12:23:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 -- start start 2023-10-30T12:23:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 librados: setting wanted keys 2023-10-30T12:23:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 librados: calling monclient init 2023-10-30T12:23:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient: init 2023-10-30T12:23:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient: _add_conns ranks=[0,2,1] 2023-10-30T12:23:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12600a4b60 0x7f126009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): picked mon.noname-a con 0x7f12600a4b60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:17.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260132db0 0x7f126009b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): picked mon.noname-c con 0x7f1260132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:17.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f126009b960 0x7f12601ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): picked mon.noname-b con 0x7f126009b960 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:17.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126009e410 con 0x7f12600a4b60 2023-10-30T12:23:17.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126009dd50 con 0x7f126009b960 2023-10-30T12:23:17.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:17.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126013ca40 con 0x7f1260132db0 2023-10-30T12:23:17.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f1265083640 10 monclient(hunting): authenticate will time out at 2242.824905s 2023-10-30T12:23:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f125e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260132db0 0x7f126009b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.641+0000 7f125ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12600a4b60 0x7f126009aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260132db0 0x7f126009b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59072/0 (socket says 172.21.15.50:59072) 2023-10-30T12:23:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12600a4b60 0x7f126009aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39492/0 (socket says 172.21.15.50:39492) 2023-10-30T12:23:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 -- 172.21.15.50:0/3119478464 learned_addr learned my addr 172.21.15.50:0/3119478464 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): get_auth_request con 0x7f1260132db0 auth_method 0 2023-10-30T12:23:17.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 10 monclient(hunting): get_auth_request con 0x7f12600a4b60 auth_method 0 2023-10-30T12:23:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:17.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125f59e640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f126009b960 0x7f12601ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125f59e640 10 monclient(hunting): get_auth_request con 0x7f126009b960 auth_method 0 2023-10-30T12:23:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125f59e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125f59e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient(hunting): handle_auth_done global_id 284204 payload 824 2023-10-30T12:23:17.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 -- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f126009b960 msgr2=0x7f12601ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f126009b960 0x7f12601ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 -- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12600a4b60 msgr2=0x7f126009aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12600a4b60 0x7f126009aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient: _finish_hunting 0 2023-10-30T12:23:17.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 monclient: found mon.noname-c 2023-10-30T12:23:17.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:17.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:17.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 -- 172.21.15.50:0/3119478464 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1250009400 con 0x7f1260132db0 2023-10-30T12:23:17.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125f59e640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f126009b960 0x7f12601ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:23:17.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125ed9d640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12600a4b60 0x7f126009aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:17.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125e59c640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260132db0 0x7f126009b420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f124802b8a0 tx=0x7f12480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 client.?.objecter ms_handle_connect 0x7f1260132db0 2023-10-30T12:23:17.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:17.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 1 -- 172.21.15.50:0/3119478464 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248032860 con 0x7f1260132db0 2023-10-30T12:23:17.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:17.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: dump: 2023-10-30T12:23:17.698 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.698 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.698 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.699 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.699 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.700 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.700 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.700 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.703 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: _finish_auth 0 2023-10-30T12:23:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: _check_auth_tickets 2023-10-30T12:23:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 5 monclient: authenticate success, global_id 284204 2023-10-30T12:23:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 1 -- 172.21.15.50:0/3119478464 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12480329c0 con 0x7f1260132db0 2023-10-30T12:23:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 10 monclient: _renew_subs 2023-10-30T12:23:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12480094d0 con 0x7f1260132db0 2023-10-30T12:23:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 1 -- 172.21.15.50:0/3119478464 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248032d70 con 0x7f1260132db0 2023-10-30T12:23:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f125d59a640 4 set_mon_vals no callback set 2023-10-30T12:23:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f12577fe640 10 monclient: dump: 2023-10-30T12:23:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:17.708 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.708 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:17.708 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:17.710 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:17.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 10 client.284204.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:17.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 10 monclient: _renew_subs 2023-10-30T12:23:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126009e5f0 con 0x7f1260132db0 2023-10-30T12:23:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.642+0000 7f1265083640 1 librados: init done 2023-10-30T12:23:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f12577fe640 1 -- 172.21.15.50:0/3119478464 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f124804d3f0 con 0x7f1260132db0 2023-10-30T12:23:17.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f1265083640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:17.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f1265083640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:17.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f1265083640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126004e160 con 0x7f1260132db0 2023-10-30T12:23:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f12577fe640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12400392f0 0x7f124003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f12577fe640 1 -- 172.21.15.50:0/3119478464 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f124806c670 con 0x7f1260132db0 2023-10-30T12:23:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f12577fe640 10 client.284204.objecter ms_dispatch 0x7f1260000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f12577fe640 3 client.284204.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f12577fe640 3 client.284204.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f12577fe640 20 client.284204.objecter dump_active .. 0 homeless 2023-10-30T12:23:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f125ed9d640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12400392f0 0x7f124003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.643+0000 7f125ed9d640 10 monclient: get_auth_request con 0x7f12400392f0 auth_method 0 2023-10-30T12:23:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.644+0000 7f12577fe640 10 client.284204.objecter ms_handle_connect 0x7f12400392f0 2023-10-30T12:23:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.644+0000 7f125ed9d640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12400392f0 0x7f124003b7b0 secure :-1 s=READY pgs=4370 cs=0 l=1 rev1=1 crypto rx=0x7f12580012e0 tx=0x7f1258009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.648+0000 7f12577fe640 1 -- 172.21.15.50:0/3119478464 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f124804c530 con 0x7f1260132db0 2023-10-30T12:23:17.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.648+0000 7f12577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:17.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.648+0000 7f12577fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.821+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f126009aa00 con 0x7f12400392f0 2023-10-30T12:23:17.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.825+0000 7f12577fe640 1 -- 172.21.15.50:0/3119478464 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133837 (secure 0 0 0) 0x7f1258001f90 con 0x7f12400392f0 2023-10-30T12:23:17.828 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:23:17.829 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:23:17.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 10 librados: watch_flush enter 2023-10-30T12:23:17.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 10 librados: watch_flush exit 2023-10-30T12:23:17.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 20 client.284204.objecter shutdown clearing up homeless session... 2023-10-30T12:23:17.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 10 client.284204.objecter successfully canceled tick 2023-10-30T12:23:17.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12400392f0 msgr2=0x7f124003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12400392f0 0x7f124003b7b0 secure :-1 s=READY pgs=4370 cs=0 l=1 rev1=1 crypto rx=0x7f12580012e0 tx=0x7f1258009040 comp rx=0 tx=0).stop 2023-10-30T12:23:17.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 10 monclient: shutdown 2023-10-30T12:23:17.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:17.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260132db0 msgr2=0x7f126009b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:17.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260132db0 0x7f126009b420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f124802b8a0 tx=0x7f12480046c0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 shutdown_connections 2023-10-30T12:23:17.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f12400392f0 0x7f124003b7b0 unknown :-1 s=CLOSED pgs=4370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f126009b960 0x7f12601ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1260132db0 0x7f126009b420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 --2- 172.21.15.50:0/3119478464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12600a4b60 0x7f126009aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:17.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 >> 172.21.15.50:0/3119478464 conn(0x7f12600a8060 msgr2=0x7f12600a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:17.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 shutdown_connections 2023-10-30T12:23:17.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 -- 172.21.15.50:0/3119478464 wait complete. 2023-10-30T12:23:17.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:17.827+0000 7f1265083640 1 librados: shutdown 2023-10-30T12:23:17.839 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1354,"stamp":"2023-10-30T12:23:16.737099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":429848,"kb_used_data":10840,"kb_used_omap":9,"kb_used_meta":418806,"kb_avail":565801192,"statfs":{"total":579820584960,"available":579380420608,"internally_reserved":0,"allocated":11100160,"data_stored":3863898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":428858023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002567"},"pg_stats":[{"pgid":"15.13","version":"0'0","reported_seq":650,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711339+0000","last_change":"2023-10-30T12:01:59.232802+0000","last_active":"2023-10-30T12:22:45.711339+0000","last_peered":"2023-10-30T12:22:45.711339+0000","last_clean":"2023-10-30T12:22:45.711339+0000","last_became_active":"2023-10-30T12:01:59.223072+0000","last_became_peered":"2023-10-30T12:01:59.223072+0000","last_unstale":"2023-10-30T12:22:45.711339+0000","last_undegraded":"2023-10-30T12:22:45.711339+0000","last_fullsized":"2023-10-30T12:22:45.711339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:39.226354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":643,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711074+0000","last_change":"2023-10-30T12:01:59.344564+0000","last_active":"2023-10-30T12:22:45.711074+0000","last_peered":"2023-10-30T12:22:45.711074+0000","last_clean":"2023-10-30T12:22:45.711074+0000","last_became_active":"2023-10-30T12:01:59.344017+0000","last_became_peered":"2023-10-30T12:01:59.344017+0000","last_unstale":"2023-10-30T12:22:45.711074+0000","last_undegraded":"2023-10-30T12:22:45.711074+0000","last_fullsized":"2023-10-30T12:22:45.711074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.878167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610023+0000","last_change":"2023-10-30T12:01:59.312117+0000","last_active":"2023-10-30T12:22:44.610023+0000","last_peered":"2023-10-30T12:22:44.610023+0000","last_clean":"2023-10-30T12:22:44.610023+0000","last_became_active":"2023-10-30T12:01:59.308994+0000","last_became_peered":"2023-10-30T12:01:59.308994+0000","last_unstale":"2023-10-30T12:22:44.610023+0000","last_undegraded":"2023-10-30T12:22:44.610023+0000","last_fullsized":"2023-10-30T12:22:44.610023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:12.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710486+0000","last_change":"2023-10-30T12:01:59.280468+0000","last_active":"2023-10-30T12:22:45.710486+0000","last_peered":"2023-10-30T12:22:45.710486+0000","last_clean":"2023-10-30T12:22:45.710486+0000","last_became_active":"2023-10-30T12:01:59.277217+0000","last_became_peered":"2023-10-30T12:01:59.277217+0000","last_unstale":"2023-10-30T12:22:45.710486+0000","last_undegraded":"2023-10-30T12:22:45.710486+0000","last_fullsized":"2023-10-30T12:22:45.710486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:39:50.184703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110225+0000","last_change":"2023-10-30T12:01:59.246329+0000","last_active":"2023-10-30T12:22:44.110225+0000","last_peered":"2023-10-30T12:22:44.110225+0000","last_clean":"2023-10-30T12:22:44.110225+0000","last_became_active":"2023-10-30T12:01:59.246101+0000","last_became_peered":"2023-10-30T12:01:59.246101+0000","last_unstale":"2023-10-30T12:22:44.110225+0000","last_undegraded":"2023-10-30T12:22:44.110225+0000","last_fullsized":"2023-10-30T12:22:44.110225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:17.822160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987154+0000","last_change":"2023-10-30T12:01:59.270798+0000","last_active":"2023-10-30T12:22:43.987154+0000","last_peered":"2023-10-30T12:22:43.987154+0000","last_clean":"2023-10-30T12:22:43.987154+0000","last_became_active":"2023-10-30T12:01:59.269001+0000","last_became_peered":"2023-10-30T12:01:59.269001+0000","last_unstale":"2023-10-30T12:22:43.987154+0000","last_undegraded":"2023-10-30T12:22:43.987154+0000","last_fullsized":"2023-10-30T12:22:43.987154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:36:36.641287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110448+0000","last_change":"2023-10-30T12:01:59.315912+0000","last_active":"2023-10-30T12:22:44.110448+0000","last_peered":"2023-10-30T12:22:44.110448+0000","last_clean":"2023-10-30T12:22:44.110448+0000","last_became_active":"2023-10-30T12:01:59.314812+0000","last_became_peered":"2023-10-30T12:01:59.314812+0000","last_unstale":"2023-10-30T12:22:44.110448+0000","last_undegraded":"2023-10-30T12:22:44.110448+0000","last_fullsized":"2023-10-30T12:22:44.110448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:11:29.969594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":664,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602031+0000","last_change":"2023-10-30T12:01:59.339356+0000","last_active":"2023-10-30T12:22:44.602031+0000","last_peered":"2023-10-30T12:22:44.602031+0000","last_clean":"2023-10-30T12:22:44.602031+0000","last_became_active":"2023-10-30T12:01:59.338820+0000","last_became_peered":"2023-10-30T12:01:59.338820+0000","last_unstale":"2023-10-30T12:22:44.602031+0000","last_undegraded":"2023-10-30T12:22:44.602031+0000","last_fullsized":"2023-10-30T12:22:44.602031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:18.746605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":663,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599775+0000","last_change":"2023-10-30T12:01:59.305090+0000","last_active":"2023-10-30T12:22:43.599775+0000","last_peered":"2023-10-30T12:22:43.599775+0000","last_clean":"2023-10-30T12:22:43.599775+0000","last_became_active":"2023-10-30T12:01:59.296246+0000","last_became_peered":"2023-10-30T12:01:59.296246+0000","last_unstale":"2023-10-30T12:22:43.599775+0000","last_undegraded":"2023-10-30T12:22:43.599775+0000","last_fullsized":"2023-10-30T12:22:43.599775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:40.407666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110494+0000","last_change":"2023-10-30T12:01:59.297023+0000","last_active":"2023-10-30T12:22:44.110494+0000","last_peered":"2023-10-30T12:22:44.110494+0000","last_clean":"2023-10-30T12:22:44.110494+0000","last_became_active":"2023-10-30T12:01:59.296511+0000","last_became_peered":"2023-10-30T12:01:59.296511+0000","last_unstale":"2023-10-30T12:22:44.110494+0000","last_undegraded":"2023-10-30T12:22:44.110494+0000","last_fullsized":"2023-10-30T12:22:44.110494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:57:45.585836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109889+0000","last_change":"2023-10-30T12:01:59.286076+0000","last_active":"2023-10-30T12:22:44.109889+0000","last_peered":"2023-10-30T12:22:44.109889+0000","last_clean":"2023-10-30T12:22:44.109889+0000","last_became_active":"2023-10-30T12:01:59.282069+0000","last_became_peered":"2023-10-30T12:01:59.282069+0000","last_unstale":"2023-10-30T12:22:44.109889+0000","last_undegraded":"2023-10-30T12:22:44.109889+0000","last_fullsized":"2023-10-30T12:22:44.109889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:33.906969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":642,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.986878+0000","last_change":"2023-10-30T12:01:59.267278+0000","last_active":"2023-10-30T12:22:43.986878+0000","last_peered":"2023-10-30T12:22:43.986878+0000","last_clean":"2023-10-30T12:22:43.986878+0000","last_became_active":"2023-10-30T12:01:59.266725+0000","last_became_peered":"2023-10-30T12:01:59.266725+0000","last_unstale":"2023-10-30T12:22:43.986878+0000","last_undegraded":"2023-10-30T12:22:43.986878+0000","last_fullsized":"2023-10-30T12:22:43.986878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:13.026323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599780+0000","last_change":"2023-10-30T12:00:53.075967+0000","last_active":"2023-10-30T12:22:43.599780+0000","last_peered":"2023-10-30T12:22:43.599780+0000","last_clean":"2023-10-30T12:22:43.599780+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:22:43.599780+0000","last_undegraded":"2023-10-30T12:22:43.599780+0000","last_fullsized":"2023-10-30T12:22:43.599780+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:48.465755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609116+0000","last_change":"2023-10-30T12:01:59.274700+0000","last_active":"2023-10-30T12:22:43.609116+0000","last_peered":"2023-10-30T12:22:43.609116+0000","last_clean":"2023-10-30T12:22:43.609116+0000","last_became_active":"2023-10-30T12:01:59.274444+0000","last_became_peered":"2023-10-30T12:01:59.274444+0000","last_unstale":"2023-10-30T12:22:43.609116+0000","last_undegraded":"2023-10-30T12:22:43.609116+0000","last_fullsized":"2023-10-30T12:22:43.609116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:18:14.789781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110011+0000","last_change":"2023-10-30T12:00:52.480624+0000","last_active":"2023-10-30T12:22:44.110011+0000","last_peered":"2023-10-30T12:22:44.110011+0000","last_clean":"2023-10-30T12:22:44.110011+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:22:44.110011+0000","last_undegraded":"2023-10-30T12:22:44.110011+0000","last_fullsized":"2023-10-30T12:22:44.110011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:05.291701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609261+0000","last_change":"2023-10-30T12:01:59.272386+0000","last_active":"2023-10-30T12:22:43.609261+0000","last_peered":"2023-10-30T12:22:43.609261+0000","last_clean":"2023-10-30T12:22:43.609261+0000","last_became_active":"2023-10-30T12:01:59.272125+0000","last_became_peered":"2023-10-30T12:01:59.272125+0000","last_unstale":"2023-10-30T12:22:43.609261+0000","last_undegraded":"2023-10-30T12:22:43.609261+0000","last_fullsized":"2023-10-30T12:22:43.609261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:15.708175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":668,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599879+0000","last_change":"2023-10-30T12:00:52.476934+0000","last_active":"2023-10-30T12:22:43.599879+0000","last_peered":"2023-10-30T12:22:43.599879+0000","last_clean":"2023-10-30T12:22:43.599879+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:22:43.599879+0000","last_undegraded":"2023-10-30T12:22:43.599879+0000","last_fullsized":"2023-10-30T12:22:43.599879+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:36.339475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609202+0000","last_change":"2023-10-30T12:01:59.269863+0000","last_active":"2023-10-30T12:22:43.609202+0000","last_peered":"2023-10-30T12:22:43.609202+0000","last_clean":"2023-10-30T12:22:43.609202+0000","last_became_active":"2023-10-30T12:01:59.269127+0000","last_became_peered":"2023-10-30T12:01:59.269127+0000","last_unstale":"2023-10-30T12:22:43.609202+0000","last_undegraded":"2023-10-30T12:22:43.609202+0000","last_fullsized":"2023-10-30T12:22:43.609202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:31:20.350152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":654,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110004+0000","last_change":"2023-10-30T12:00:52.479941+0000","last_active":"2023-10-30T12:22:44.110004+0000","last_peered":"2023-10-30T12:22:44.110004+0000","last_clean":"2023-10-30T12:22:44.110004+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:22:44.110004+0000","last_undegraded":"2023-10-30T12:22:44.110004+0000","last_fullsized":"2023-10-30T12:22:44.110004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:27.759194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110202+0000","last_change":"2023-10-30T12:01:59.306784+0000","last_active":"2023-10-30T12:22:44.110202+0000","last_peered":"2023-10-30T12:22:44.110202+0000","last_clean":"2023-10-30T12:22:44.110202+0000","last_became_active":"2023-10-30T12:01:59.306293+0000","last_became_peered":"2023-10-30T12:01:59.306293+0000","last_unstale":"2023-10-30T12:22:44.110202+0000","last_undegraded":"2023-10-30T12:22:44.110202+0000","last_fullsized":"2023-10-30T12:22:44.110202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:26:17.267254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110433+0000","last_change":"2023-10-30T12:00:52.480712+0000","last_active":"2023-10-30T12:22:44.110433+0000","last_peered":"2023-10-30T12:22:44.110433+0000","last_clean":"2023-10-30T12:22:44.110433+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:22:44.110433+0000","last_undegraded":"2023-10-30T12:22:44.110433+0000","last_fullsized":"2023-10-30T12:22:44.110433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:30.194847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":664,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.599745+0000","last_change":"2023-10-30T12:01:59.314586+0000","last_active":"2023-10-30T12:22:43.599745+0000","last_peered":"2023-10-30T12:22:43.599745+0000","last_clean":"2023-10-30T12:22:43.599745+0000","last_became_active":"2023-10-30T12:01:59.313414+0000","last_became_peered":"2023-10-30T12:01:59.313414+0000","last_unstale":"2023-10-30T12:22:43.599745+0000","last_undegraded":"2023-10-30T12:22:43.599745+0000","last_fullsized":"2023-10-30T12:22:43.599745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:05.821606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609150+0000","last_change":"2023-10-30T12:00:52.480385+0000","last_active":"2023-10-30T12:22:43.609150+0000","last_peered":"2023-10-30T12:22:43.609150+0000","last_clean":"2023-10-30T12:22:43.609150+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:22:43.609150+0000","last_undegraded":"2023-10-30T12:22:43.609150+0000","last_fullsized":"2023-10-30T12:22:43.609150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:54:57.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":644,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110338+0000","last_change":"2023-10-30T12:01:59.289541+0000","last_active":"2023-10-30T12:22:44.110338+0000","last_peered":"2023-10-30T12:22:44.110338+0000","last_clean":"2023-10-30T12:22:44.110338+0000","last_became_active":"2023-10-30T12:01:59.289175+0000","last_became_peered":"2023-10-30T12:01:59.289175+0000","last_unstale":"2023-10-30T12:22:44.110338+0000","last_undegraded":"2023-10-30T12:22:44.110338+0000","last_fullsized":"2023-10-30T12:22:44.110338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:01.620219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":668,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601903+0000","last_change":"2023-10-30T12:00:52.479825+0000","last_active":"2023-10-30T12:22:44.601903+0000","last_peered":"2023-10-30T12:22:44.601903+0000","last_clean":"2023-10-30T12:22:44.601903+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:22:44.601903+0000","last_undegraded":"2023-10-30T12:22:44.601903+0000","last_fullsized":"2023-10-30T12:22:44.601903+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:18.465900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110134+0000","last_change":"2023-10-30T12:01:59.307590+0000","last_active":"2023-10-30T12:22:44.110134+0000","last_peered":"2023-10-30T12:22:44.110134+0000","last_clean":"2023-10-30T12:22:44.110134+0000","last_became_active":"2023-10-30T12:01:59.302155+0000","last_became_peered":"2023-10-30T12:01:59.302155+0000","last_unstale":"2023-10-30T12:22:44.110134+0000","last_undegraded":"2023-10-30T12:22:44.110134+0000","last_fullsized":"2023-10-30T12:22:44.110134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:48.321197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":648,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110344+0000","last_change":"2023-10-30T12:00:52.479093+0000","last_active":"2023-10-30T12:22:44.110344+0000","last_peered":"2023-10-30T12:22:44.110344+0000","last_clean":"2023-10-30T12:22:44.110344+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:22:44.110344+0000","last_undegraded":"2023-10-30T12:22:44.110344+0000","last_fullsized":"2023-10-30T12:22:44.110344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:51.469554+0000","last_clean_scrub_stamp":"2023-10-30T12:00:51.469554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:23.366435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":820,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602127+0000","last_change":"2023-10-30T12:00:50.418228+0000","last_active":"2023-10-30T12:22:44.602127+0000","last_peered":"2023-10-30T12:22:44.602127+0000","last_clean":"2023-10-30T12:22:44.602127+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:22:44.602127+0000","last_undegraded":"2023-10-30T12:22:44.602127+0000","last_fullsized":"2023-10-30T12:22:44.602127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:00:49.409444+0000","last_clean_scrub_stamp":"2023-10-30T12:00:49.409444+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:44.043013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":651,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609190+0000","last_change":"2023-10-30T12:01:59.256285+0000","last_active":"2023-10-30T12:22:43.609190+0000","last_peered":"2023-10-30T12:22:43.609190+0000","last_clean":"2023-10-30T12:22:43.609190+0000","last_became_active":"2023-10-30T12:01:59.253931+0000","last_became_peered":"2023-10-30T12:01:59.253931+0000","last_unstale":"2023-10-30T12:22:43.609190+0000","last_undegraded":"2023-10-30T12:22:43.609190+0000","last_fullsized":"2023-10-30T12:22:43.609190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:32:21.312914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711364+0000","last_change":"2023-10-30T12:01:59.296719+0000","last_active":"2023-10-30T12:22:45.711364+0000","last_peered":"2023-10-30T12:22:45.711364+0000","last_clean":"2023-10-30T12:22:45.711364+0000","last_became_active":"2023-10-30T12:01:59.277183+0000","last_became_peered":"2023-10-30T12:01:59.277183+0000","last_unstale":"2023-10-30T12:22:45.711364+0000","last_undegraded":"2023-10-30T12:22:45.711364+0000","last_fullsized":"2023-10-30T12:22:45.711364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:49.956478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":665,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.602043+0000","last_change":"2023-10-30T12:01:59.339853+0000","last_active":"2023-10-30T12:22:44.602043+0000","last_peered":"2023-10-30T12:22:44.602043+0000","last_clean":"2023-10-30T12:22:44.602043+0000","last_became_active":"2023-10-30T12:01:59.339614+0000","last_became_peered":"2023-10-30T12:01:59.339614+0000","last_unstale":"2023-10-30T12:22:44.602043+0000","last_undegraded":"2023-10-30T12:22:44.602043+0000","last_fullsized":"2023-10-30T12:22:44.602043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:21.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":643,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.987106+0000","last_change":"2023-10-30T12:01:59.303724+0000","last_active":"2023-10-30T12:22:43.987106+0000","last_peered":"2023-10-30T12:22:43.987106+0000","last_clean":"2023-10-30T12:22:43.987106+0000","last_became_active":"2023-10-30T12:01:59.303216+0000","last_became_peered":"2023-10-30T12:01:59.303216+0000","last_unstale":"2023-10-30T12:22:43.987106+0000","last_undegraded":"2023-10-30T12:22:43.987106+0000","last_fullsized":"2023-10-30T12:22:43.987106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:35.112009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109953+0000","last_change":"2023-10-30T12:01:59.209818+0000","last_active":"2023-10-30T12:22:44.109953+0000","last_peered":"2023-10-30T12:22:44.109953+0000","last_clean":"2023-10-30T12:22:44.109953+0000","last_became_active":"2023-10-30T12:01:59.209551+0000","last_became_peered":"2023-10-30T12:01:59.209551+0000","last_unstale":"2023-10-30T12:22:44.109953+0000","last_undegraded":"2023-10-30T12:22:44.109953+0000","last_fullsized":"2023-10-30T12:22:44.109953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:49.423499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":653,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609027+0000","last_change":"2023-10-30T12:01:59.312819+0000","last_active":"2023-10-30T12:22:43.609027+0000","last_peered":"2023-10-30T12:22:43.609027+0000","last_clean":"2023-10-30T12:22:43.609027+0000","last_became_active":"2023-10-30T12:01:59.310341+0000","last_became_peered":"2023-10-30T12:01:59.310341+0000","last_unstale":"2023-10-30T12:22:43.609027+0000","last_undegraded":"2023-10-30T12:22:43.609027+0000","last_fullsized":"2023-10-30T12:22:43.609027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:52.592573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"18'7","reported_seq":678,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.601944+0000","last_change":"2023-10-30T12:01:59.261094+0000","last_active":"2023-10-30T12:22:44.601944+0000","last_peered":"2023-10-30T12:22:44.601944+0000","last_clean":"2023-10-30T12:22:44.601944+0000","last_became_active":"2023-10-30T12:01:59.260849+0000","last_became_peered":"2023-10-30T12:01:59.260849+0000","last_unstale":"2023-10-30T12:22:44.601944+0000","last_undegraded":"2023-10-30T12:22:44.601944+0000","last_fullsized":"2023-10-30T12:22:44.601944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:36.395779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610042+0000","last_change":"2023-10-30T12:08:57.463451+0000","last_active":"2023-10-30T12:22:44.610042+0000","last_peered":"2023-10-30T12:22:44.610042+0000","last_clean":"2023-10-30T12:22:44.610042+0000","last_became_active":"2023-10-30T12:08:57.463102+0000","last_became_peered":"2023-10-30T12:08:57.463102+0000","last_unstale":"2023-10-30T12:22:44.610042+0000","last_undegraded":"2023-10-30T12:22:44.610042+0000","last_fullsized":"2023-10-30T12:22:44.610042+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:48.599858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.710660+0000","last_change":"2023-10-30T12:01:59.247352+0000","last_active":"2023-10-30T12:22:45.710660+0000","last_peered":"2023-10-30T12:22:45.710660+0000","last_clean":"2023-10-30T12:22:45.710660+0000","last_became_active":"2023-10-30T12:01:59.246868+0000","last_became_peered":"2023-10-30T12:01:59.246868+0000","last_unstale":"2023-10-30T12:22:45.710660+0000","last_undegraded":"2023-10-30T12:22:45.710660+0000","last_fullsized":"2023-10-30T12:22:45.710660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:50.392311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":649,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.109897+0000","last_change":"2023-10-30T12:01:59.276982+0000","last_active":"2023-10-30T12:22:44.109897+0000","last_peered":"2023-10-30T12:22:44.109897+0000","last_clean":"2023-10-30T12:22:44.109897+0000","last_became_active":"2023-10-30T12:01:59.239117+0000","last_became_peered":"2023-10-30T12:01:59.239117+0000","last_unstale":"2023-10-30T12:22:44.109897+0000","last_undegraded":"2023-10-30T12:22:44.109897+0000","last_fullsized":"2023-10-30T12:22:44.109897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.157663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":650,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.110047+0000","last_change":"2023-10-30T12:01:59.297769+0000","last_active":"2023-10-30T12:22:44.110047+0000","last_peered":"2023-10-30T12:22:44.110047+0000","last_clean":"2023-10-30T12:22:44.110047+0000","last_became_active":"2023-10-30T12:01:59.293055+0000","last_became_peered":"2023-10-30T12:01:59.293055+0000","last_unstale":"2023-10-30T12:22:44.110047+0000","last_undegraded":"2023-10-30T12:22:44.110047+0000","last_fullsized":"2023-10-30T12:22:44.110047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:53.053266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":652,"reported_epoch":685,"state":"active+clean","last_fresh":"2023-10-30T12:22:43.609249+0000","last_change":"2023-10-30T12:01:59.310309+0000","last_active":"2023-10-30T12:22:43.609249+0000","last_peered":"2023-10-30T12:22:43.609249+0000","last_clean":"2023-10-30T12:22:43.609249+0000","last_became_active":"2023-10-30T12:01:59.307440+0000","last_became_peered":"2023-10-30T12:01:59.307440+0000","last_unstale":"2023-10-30T12:22:43.609249+0000","last_undegraded":"2023-10-30T12:22:43.609249+0000","last_fullsized":"2023-10-30T12:22:43.609249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:39.454237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.711399+0000","last_change":"2023-10-30T12:01:59.249544+0000","last_active":"2023-10-30T12:22:45.711399+0000","last_peered":"2023-10-30T12:22:45.711399+0000","last_clean":"2023-10-30T12:22:45.711399+0000","last_became_active":"2023-10-30T12:01:59.239219+0000","last_became_peered":"2023-10-30T12:01:59.239219+0000","last_unstale":"2023-10-30T12:22:45.711399+0000","last_undegraded":"2023-10-30T12:22:45.711399+0000","last_fullsized":"2023-10-30T12:22:45.711399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:17.868254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640527,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77256,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294584,"statfs":{"total":96636764160,"available":96557654016,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.82699999999999996},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.65000000000000002,"5min":0.81599999999999995,"15min":1.056},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65400000000000003,"5min":0.83399999999999996,"15min":2.173},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.28299999999999997},"max":{"1min":0.64800000000000002,"5min":0.72799999999999998,"15min":0.78100000000000003},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.67400000000000004,"5min":0.74299999999999999,"15min":0.79900000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.41499999999999998,"15min":0.32900000000000001},"max":{"1min":0.65000000000000002,"5min":0.79900000000000004,"15min":0.84699999999999998},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.36899999999999999,"15min":0.373},"min":{"1min":0.25,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":0.499,"5min":0.58799999999999997,"15min":0.70399999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.223},"max":{"1min":0.54500000000000004,"5min":0.69699999999999995,"15min":0.95199999999999996},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.31,"5min":0.252,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.63700000000000001,"15min":0.63800000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.435,"15min":0.45300000000000001},"min":{"1min":0.28100000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.60299999999999998,"5min":0.68400000000000005,"15min":0.72099999999999997},"last":0.55200000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81416,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94290424,"statfs":{"total":96636764160,"available":96553394176,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53900000000000003,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.51400000000000001},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.84899999999999998},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.79000000000000004,"15min":0.94099999999999995},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.36299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.67700000000000005,"5min":0.749,"15min":0.79800000000000004},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27300000000000002},"max":{"1min":0.627,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.246},"max":{"1min":0.59399999999999997,"5min":0.627,"15min":0.63600000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.25,"5min":0.249,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.67000000000000004,"15min":0.77100000000000002},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.435,"15min":0.40999999999999998},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.25600000000000001},"max":{"1min":0.621,"5min":0.64000000000000001,"15min":0.69999999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.67400000000000004,"15min":0.89700000000000002},"last":0.58299999999999996}]}]},{"osd":3,"up_from":11,"seq":47244640528,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69444,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302396,"statfs":{"total":96636764160,"available":96565653504,"internally_reserved":0,"allocated":2207744,"data_stored":996605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.503,"15min":0.501},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72799999999999998},"last":0.432},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.61899999999999999,"5min":0.71399999999999997,"15min":1.2490000000000001},"last":0.53000000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.33100000000000002,"15min":0.315},"max":{"1min":0.68999999999999995,"5min":0.81100000000000005,"15min":1.2689999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69999999999999996,"5min":0.82599999999999996,"15min":1.337},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.68899999999999995,"5min":0.88900000000000001,"15min":1.302},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":0.746,"15min":0.76900000000000002},"last":0.48699999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.37,"15min":0.38},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23300000000000001},"max":{"1min":0.50700000000000001,"5min":0.61199999999999999,"15min":1.371},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.30599999999999999},"max":{"1min":0.56599999999999995,"5min":0.66300000000000003,"15min":1.444},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Oct 30 12:22:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39600000000000002,"15min":0.38700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":0.502,"5min":0.64200000000000002,"15min":0.876},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53500000000000003,"5min":0.84099999999999997,"15min":1.3540000000000001},"last":0.51000000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640527,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64764,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94307076,"statfs":{"total":96636764160,"available":96570445824,"internally_reserved":0,"allocated":1675264,"data_stored":467677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33500000000000002},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.188},"max":{"1min":0.44800000000000001,"5min":0.50700000000000001,"15min":2.2730000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.38400000000000001},"min":{"1min":0.27500000000000002,"5min":0.24199999999999999,"15min":0.22800000000000001},"max":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":2.141},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33700000000000002,"15min":0.34200000000000003},"min":{"1min":0.24399999999999999,"5min":0.21299999999999999,"15min":0.20799999999999999},"max":{"1min":0.46999999999999997,"5min":0.502,"15min":1.087},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.40600000000000003},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.24399999999999999},"max":{"1min":0.495,"5min":0.63900000000000001,"15min":3.8260000000000001},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54500000000000004},"min":{"1min":0.42199999999999999,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":0.749,"5min":0.75700000000000001,"15min":1.143},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.42899999999999999,"5min":0.33600000000000002,"15min":0.314},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":3.4729999999999999},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.34399999999999997,"15min":0.27300000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.3240000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.45600000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":1.8340000000000001},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Oct 30 12:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.497,"5min":0.39000000000000001,"15min":0.32200000000000001},"max":{"1min":0.81699999999999995,"5min":0.81999999999999995,"15min":1.893},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.48499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75900000000000001,"5min":0.81599999999999995,"15min":2.2269999999999999},"last":0.56899999999999995}]}]},{"osd":1,"up_from":11,"seq":47244640529,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59208,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312632,"statfs":{"total":96636764160,"available":96576135168,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.34499999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":0.46899999999999997,"5min":0.745,"15min":0.745},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.374},"min":{"1min":0.253,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.38200000000000001},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.223},"max":{"1min":0.54000000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.51800000000000002,"5min":0.621,"15min":0.93200000000000005},"last":0.379}]},{"osd":3,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.443,"5min":0.375,"15min":0.28499999999999998},"max":{"1min":0.69999999999999996,"5min":0.86599999999999999,"15min":0.879},"last":0.495},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.79900000000000004,"5min":0.85399999999999998,"15min":0.88900000000000001},"last":0.436}]},{"osd":4,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.46400000000000002,"5min":0.41699999999999998,"15min":0.29899999999999999},"max":{"1min":0.86799999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.36799999999999999},"max":{"1min":0.71299999999999997,"5min":0.94799999999999995,"15min":0.95599999999999996},"last":0.51000000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.58899999999999997},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.39300000000000002},"max":{"1min":0.84499999999999997,"5min":1,"15min":1},"last":0.53800000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640529,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77760,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294080,"statfs":{"total":96636764160,"available":96557137920,"internally_reserved":0,"allocated":2203648,"data_stored":996600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.35199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23899999999999999},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.29899999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.21299999999999999},"max":{"1min":1.802,"5min":1.802,"15min":3.8519999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26700000000000002,"15min":0.20100000000000001},"max":{"1min":0.58699999999999997,"5min":0.623,"15min":2.0590000000000002},"last":0.23499999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.27800000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":1.5489999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":1.78},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.29399999999999998},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59099999999999997,"15min":0.57899999999999996},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":1.6020000000000001},"last":0.433},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:23:17.844 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.845 INFO:tasks.ceph:pgid 15.12 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.845 INFO:tasks.ceph:pgid 15.11 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.846 INFO:tasks.ceph:pgid 15.10 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.846 INFO:tasks.ceph:pgid 15.17 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.846 INFO:tasks.ceph:pgid 15.16 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.847 INFO:tasks.ceph:pgid 15.15 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.847 INFO:tasks.ceph:pgid 15.14 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.847 INFO:tasks.ceph:pgid 15.1b last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.848 INFO:tasks.ceph:pgid 15.1a last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.848 INFO:tasks.ceph:pgid 15.19 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.849 INFO:tasks.ceph:pgid 15.18 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.849 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.849 INFO:tasks.ceph:pgid 15.a last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.850 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.850 INFO:tasks.ceph:pgid 15.b last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.851 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.851 INFO:tasks.ceph:pgid 15.8 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.851 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.852 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.852 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.852 INFO:tasks.ceph:pgid 15.f last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.853 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.853 INFO:tasks.ceph:pgid 15.c last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.853 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.854 INFO:tasks.ceph:pgid 15.d last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.854 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T12:00:51.469554+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=51, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.855 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T12:00:49.409444+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=0, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.855 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.855 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.856 INFO:tasks.ceph:pgid 15.5 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.856 INFO:tasks.ceph:pgid 15.6 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.856 INFO:tasks.ceph:pgid 15.7 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.857 INFO:tasks.ceph:pgid 15.0 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.857 INFO:tasks.ceph:pgid 15.1 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.858 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.858 INFO:tasks.ceph:pgid 15.3 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.858 INFO:tasks.ceph:pgid 15.1c last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.859 INFO:tasks.ceph:pgid 15.1d last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.859 INFO:tasks.ceph:pgid 15.1e last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.859 INFO:tasks.ceph:pgid 15.1f last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:17.860 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:23:37.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:23:37.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.978+0000 7f916e139640 10 monclient: get_monmap_and_config 2023-10-30T12:23:37.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.978+0000 7f916e139640 10 monclient: build_initial_monmap 2023-10-30T12:23:37.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.978+0000 7f916e139640 10 monclient: monmap: 2023-10-30T12:23:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:37.979357+0000 2023-10-30T12:23:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:37.979357+0000 2023-10-30T12:23:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:37.983 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:37.983 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:37.983 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:37.983 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:37.984 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:37.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 1 Processor -- start 2023-10-30T12:23:37.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 1 -- start start 2023-10-30T12:23:37.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient: init 2023-10-30T12:23:37.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:37.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-10-30T12:23:37.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91680a4b60 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:37.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f9168099a40 0x7f916813aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:37.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient(hunting): picked mon.noname-c con 0x7f9168099a40 addr v2:172.21.15.118:3300/0 2023-10-30T12:23:37.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9168132db0 0x7f916813cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:37.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient(hunting): picked mon.noname-a con 0x7f9168132db0 addr v2:172.21.15.50:3300/0 2023-10-30T12:23:37.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91680a4f40 con 0x7f91680a4b60 2023-10-30T12:23:37.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:37.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91680a3d70 con 0x7f9168132db0 2023-10-30T12:23:37.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:37.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91680a4150 con 0x7f9168099a40 2023-10-30T12:23:37.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:37.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f916e139640 10 monclient(hunting): authenticate will time out at 2263.162714s 2023-10-30T12:23:37.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f91677fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f91680a4b60 0x7f91681326a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58326/0 (socket says 172.21.15.50:58326) 2023-10-30T12:23:37.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9168132db0 0x7f916813cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:37.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f91677fe640 1 -- 172.21.15.50:0/3052100661 learned_addr learned my addr 172.21.15.50:0/3052100661 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:37.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 10 monclient(hunting): get_auth_request con 0x7f9168132db0 auth_method 0 2023-10-30T12:23:37.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:37.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:37.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:37.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:37.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:37.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.979+0000 7f9167fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:37.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9166ffd640 1 --2- 172.21.15.50:0/3052100661 >> v2:172.21.15.118:3300/0 conn(0x7f9168099a40 0x7f916813aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:37.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 10 monclient(hunting): handle_auth_done global_id 264465 payload 274 2023-10-30T12:23:38.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 1 -- 172.21.15.50:0/3052100661 >> v2:172.21.15.118:3300/0 conn(0x7f9168099a40 msgr2=0x7f916813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:23:38.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 1 --2- 172.21.15.50:0/3052100661 >> v2:172.21.15.118:3300/0 conn(0x7f9168099a40 0x7f916813aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 1 -- 172.21.15.50:0/3052100661 >> v1:172.21.15.50:6789/0 conn(0x7f91680a4b60 legacy=0x7f91681326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:38.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 10 monclient: _finish_hunting 0 2023-10-30T12:23:38.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 1 monclient: found mon.noname-a 2023-10-30T12:23:38.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:38.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:38.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 1 -- 172.21.15.50:0/3052100661 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916809a440 con 0x7f9168132db0 2023-10-30T12:23:38.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f91667fc640 1 -- 172.21.15.50:0/3052100661 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353642241 0 0) 0x7f91680a4f40 con 0x7f91680a4b60 2023-10-30T12:23:38.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f91667fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-10-30T12:23:38.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9166ffd640 10 monclient: get_auth_request con 0x7f9168099a40 auth_method 0 2023-10-30T12:23:38.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9166ffd640 1 --2- 172.21.15.50:0/3052100661 >> v2:172.21.15.118:3300/0 conn(0x7f9168099a40 0x7f916813aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:23:38.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.980+0000 7f9167fff640 1 --2- 172.21.15.50:0/3052100661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f916813cfd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9168099ed0 tx=0x7f916002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f0bc43afee9a83 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:38.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 1 -- 172.21.15.50:0/3052100661 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916002cb60 con 0x7f9168132db0 2023-10-30T12:23:38.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:38.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:38.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: dump: 2023-10-30T12:23:38.004 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:38.004 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:38.005 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.005 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.005 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:38.006 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:38.006 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:38.006 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:38.006 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:38.007 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:38.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: _finish_auth 0 2023-10-30T12:23:38.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: _check_auth_tickets 2023-10-30T12:23:38.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:38.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 5 monclient: authenticate success, global_id 264465 2023-10-30T12:23:38.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:23:38.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 1 -- 172.21.15.50:0/3052100661 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916002ccc0 con 0x7f9168132db0 2023-10-30T12:23:38.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:38.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:38.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 1 -- 172.21.15.50:0/3052100661 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160034660 con 0x7f9168132db0 2023-10-30T12:23:38.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 4 set_mon_vals no callback set 2023-10-30T12:23:38.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:38.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:38.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f91667fc640 10 monclient: dump: 2023-10-30T12:23:38.010 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:38.011 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:38.011 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.011 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.012 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:38.012 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:38.012 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:38.012 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:38.013 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:38.013 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:38.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 10 monclient: shutdown 2023-10-30T12:23:38.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:38.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 -- 172.21.15.50:0/3052100661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 msgr2=0x7f916813cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:38.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 --2- 172.21.15.50:0/3052100661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f916813cfd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9168099ed0 tx=0x7f916002bdb0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 -- 172.21.15.50:0/3052100661 shutdown_connections 2023-10-30T12:23:38.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 --2- 172.21.15.50:0/3052100661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f916813cfd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 --2- 172.21.15.50:0/3052100661 >> v2:172.21.15.118:3300/0 conn(0x7f9168099a40 0x7f916813aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 -- 172.21.15.50:0/3052100661 >> 172.21.15.50:0/3052100661 conn(0x7f91680a8060 msgr2=0x7f91680a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:38.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 -- 172.21.15.50:0/3052100661 shutdown_connections 2023-10-30T12:23:38.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.981+0000 7f916e139640 1 -- 172.21.15.50:0/3052100661 wait complete. 2023-10-30T12:23:38.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 10 monclient: build_initial_monmap 2023-10-30T12:23:38.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 10 monclient: monmap: 2023-10-30T12:23:38.017 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:38.017 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:38.017 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:38.017 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:38.018 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:38.018 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:38.018 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:38.019 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:38.019 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:38.019 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:38.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 1 librados: starting msgr at 2023-10-30T12:23:38.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 1 librados: starting objecter 2023-10-30T12:23:38.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 1 Processor -- start 2023-10-30T12:23:38.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 1 -- start start 2023-10-30T12:23:38.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 1 librados: setting wanted keys 2023-10-30T12:23:38.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 1 librados: calling monclient init 2023-10-30T12:23:38.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.982+0000 7f916e139640 10 monclient: init 2023-10-30T12:23:38.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:38.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:23:38.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91680a4b60 0x7f91681ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:38.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): picked mon.noname-c con 0x7f91680a4b60 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:38.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f91681caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:38.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): picked mon.noname-a con 0x7f9168132db0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:38.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91681cafc0 0x7f9168136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:38.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): picked mon.noname-b con 0x7f91681cafc0 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:38.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:38.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916813c9d0 con 0x7f9168132db0 2023-10-30T12:23:38.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:38.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916813bbc0 con 0x7f91681cafc0 2023-10-30T12:23:38.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:38.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9168138820 con 0x7f91680a4b60 2023-10-30T12:23:38.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:38.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f916e139640 10 monclient(hunting): authenticate will time out at 2263.166301s 2023-10-30T12:23:38.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f91681caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:38.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91680a4b60 0x7f91681ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:38.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f91681caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32914/0 (socket says 172.21.15.50:32914) 2023-10-30T12:23:38.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91680a4b60 0x7f91681ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50578/0 (socket says 172.21.15.50:50578) 2023-10-30T12:23:38.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 1 -- 172.21.15.50:0/2704358409 learned_addr learned my addr 172.21.15.50:0/2704358409 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:38.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 10 monclient(hunting): get_auth_request con 0x7f9168132db0 auth_method 0 2023-10-30T12:23:38.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:38.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:38.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:38.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): get_auth_request con 0x7f91680a4b60 auth_method 0 2023-10-30T12:23:38.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:38.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:38.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:38.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:38.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:38.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:38.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:38.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:38.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:38.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91681cafc0 0x7f9168136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:38.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 10 monclient(hunting): get_auth_request con 0x7f91681cafc0 auth_method 0 2023-10-30T12:23:38.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:38.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:38.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:38.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:38.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:38.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9167fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:38.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient(hunting): handle_auth_done global_id 284207 payload 824 2023-10-30T12:23:38.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 -- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91681cafc0 msgr2=0x7f9168136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:38.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91681cafc0 0x7f9168136850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 -- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 msgr2=0x7f91681caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:38.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f91681caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:38.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 monclient: found mon.noname-c 2023-10-30T12:23:38.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:38.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:23:38.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f91677fe640 1 -- 172.21.15.50:0/2704358409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9160009a70 con 0x7f91680a4b60 2023-10-30T12:23:38.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.983+0000 7f9166ffd640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f91681caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:23:38.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f91677fe640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91680a4b60 0x7f91681ca540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f915800c450 tx=0x7f915800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:38.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 client.?.objecter ms_handle_connect 0x7f91680a4b60 2023-10-30T12:23:38.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:38.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 1 -- 172.21.15.50:0/2704358409 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91580074e0 con 0x7f91680a4b60 2023-10-30T12:23:38.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:38.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:23:38.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: dump: 2023-10-30T12:23:38.038 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:38.038 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:38.039 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.039 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.039 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:38.040 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:38.040 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:38.040 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:38.040 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:38.041 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:38.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: _finish_auth 0 2023-10-30T12:23:38.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: _check_auth_tickets 2023-10-30T12:23:38.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:38.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 5 monclient: authenticate success, global_id 284207 2023-10-30T12:23:38.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 1 -- 172.21.15.50:0/2704358409 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158007640 con 0x7f91680a4b60 2023-10-30T12:23:38.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:38.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 1 -- 172.21.15.50:0/2704358409 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158004a20 con 0x7f91680a4b60 2023-10-30T12:23:38.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:38.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:38.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f914ffff640 10 monclient: dump: 2023-10-30T12:23:38.044 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:38.044 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:38.044 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.044 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:38.045 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:38.045 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:38.045 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:38.045 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:38.046 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:38.046 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:38.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 10 monclient: _renew_subs 2023-10-30T12:23:38.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f9165ffb640 4 set_mon_vals no callback set 2023-10-30T12:23:38.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:38.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91680a18d0 con 0x7f91680a4b60 2023-10-30T12:23:38.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 10 client.284207.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:38.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 10 monclient: _renew_subs 2023-10-30T12:23:38.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:38.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9168139f40 con 0x7f91680a4b60 2023-10-30T12:23:38.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.984+0000 7f916e139640 1 librados: init done 2023-10-30T12:23:38.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f914ffff640 1 -- 172.21.15.50:0/2704358409 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f9158004b80 con 0x7f91680a4b60 2023-10-30T12:23:38.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f916e139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:38.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f916e139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:38.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f916e139640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:23:38.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916804e160 con 0x7f91680a4b60 2023-10-30T12:23:38.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f914ffff640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91540392f0 0x7f915403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:38.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f914ffff640 1 -- 172.21.15.50:0/2704358409 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f9158049a10 con 0x7f91680a4b60 2023-10-30T12:23:38.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f914ffff640 10 client.284207.objecter ms_dispatch 0x7f9168000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:38.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f914ffff640 3 client.284207.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:38.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f914ffff640 3 client.284207.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:38.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f914ffff640 20 client.284207.objecter dump_active .. 0 homeless 2023-10-30T12:23:38.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f9166ffd640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91540392f0 0x7f915403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:38.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.985+0000 7f9166ffd640 10 monclient: get_auth_request con 0x7f91540392f0 auth_method 0 2023-10-30T12:23:38.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.986+0000 7f914ffff640 10 client.284207.objecter ms_handle_connect 0x7f91540392f0 2023-10-30T12:23:38.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.986+0000 7f9166ffd640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91540392f0 0x7f915403b7b0 secure :-1 s=READY pgs=4371 cs=0 l=1 rev1=1 crypto rx=0x7f91680a0f70 tx=0x7f915c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.990+0000 7f914ffff640 1 -- 172.21.15.50:0/2704358409 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f915801b880 con 0x7f91680a4b60 2023-10-30T12:23:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.990+0000 7f914ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:37.990+0000 7f914ffff640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:38.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.160+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f91681396a0 con 0x7f91540392f0 2023-10-30T12:23:38.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.165+0000 7f914ffff640 1 -- 172.21.15.50:0/2704358409 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134173 (secure 0 0 0) 0x7f91681396a0 con 0x7f91540392f0 2023-10-30T12:23:38.168 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:23:38.169 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:23:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 10 librados: watch_flush enter 2023-10-30T12:23:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 10 librados: watch_flush exit 2023-10-30T12:23:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 20 client.284207.objecter shutdown clearing up homeless session... 2023-10-30T12:23:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 10 client.284207.objecter successfully canceled tick 2023-10-30T12:23:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91540392f0 msgr2=0x7f915403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91540392f0 0x7f915403b7b0 secure :-1 s=READY pgs=4371 cs=0 l=1 rev1=1 crypto rx=0x7f91680a0f70 tx=0x7f915c008040 comp rx=0 tx=0).stop 2023-10-30T12:23:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 10 monclient: shutdown 2023-10-30T12:23:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91680a4b60 msgr2=0x7f91681ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91680a4b60 0x7f91681ca540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f915800c450 tx=0x7f915800c910 comp rx=0 tx=0).stop 2023-10-30T12:23:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 shutdown_connections 2023-10-30T12:23:38.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f91540392f0 0x7f915403b7b0 unknown :-1 s=CLOSED pgs=4371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91681cafc0 0x7f9168136850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9168132db0 0x7f91681caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 --2- 172.21.15.50:0/2704358409 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f91680a4b60 0x7f91681ca540 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 >> 172.21.15.50:0/2704358409 conn(0x7f91680a8060 msgr2=0x7f91680a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:38.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 shutdown_connections 2023-10-30T12:23:38.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 -- 172.21.15.50:0/2704358409 wait complete. 2023-10-30T12:23:38.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:38.167+0000 7f916e139640 1 librados: shutdown 2023-10-30T12:23:38.179 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1364,"stamp":"2023-10-30T12:23:36.742238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":429704,"kb_used_data":10840,"kb_used_omap":9,"kb_used_meta":418806,"kb_avail":565801336,"statfs":{"total":579820584960,"available":579380568064,"internally_reserved":0,"allocated":11100160,"data_stored":3863898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":428858023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003005"},"pg_stats":[{"pgid":"15.13","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:59.578693+0000","last_change":"2023-10-30T12:01:59.232802+0000","last_active":"2023-10-30T12:22:59.578693+0000","last_peered":"2023-10-30T12:22:59.578693+0000","last_clean":"2023-10-30T12:22:59.578693+0000","last_became_active":"2023-10-30T12:01:59.223072+0000","last_became_peered":"2023-10-30T12:01:59.223072+0000","last_unstale":"2023-10-30T12:22:59.578693+0000","last_undegraded":"2023-10-30T12:22:59.578693+0000","last_fullsized":"2023-10-30T12:22:59.578693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.679786+0000","last_change":"2023-10-30T12:23:22.679842+0000","last_active":"2023-10-30T12:23:22.679786+0000","last_peered":"2023-10-30T12:23:22.679786+0000","last_clean":"2023-10-30T12:23:22.679786+0000","last_became_active":"2023-10-30T12:01:59.344017+0000","last_became_peered":"2023-10-30T12:01:59.344017+0000","last_unstale":"2023-10-30T12:23:22.679786+0000","last_undegraded":"2023-10-30T12:23:22.679786+0000","last_fullsized":"2023-10-30T12:23:22.679786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.679707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.679707+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.679707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:10.004012+0000","scrub_duration":0.0020254040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:27.841929+0000","last_change":"2023-10-30T12:23:27.842001+0000","last_active":"2023-10-30T12:23:27.841929+0000","last_peered":"2023-10-30T12:23:27.841929+0000","last_clean":"2023-10-30T12:23:27.841929+0000","last_became_active":"2023-10-30T12:01:59.308994+0000","last_became_peered":"2023-10-30T12:01:59.308994+0000","last_unstale":"2023-10-30T12:23:27.841929+0000","last_undegraded":"2023-10-30T12:23:27.841929+0000","last_fullsized":"2023-10-30T12:23:27.841929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:27.841846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:27.841846+0000","last_clean_scrub_stamp":"2023-10-30T12:23:27.841846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:25.854306+0000","scrub_duration":0.0022774079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.051009+0000","last_change":"2023-10-30T12:23:22.051049+0000","last_active":"2023-10-30T12:23:22.051009+0000","last_peered":"2023-10-30T12:23:22.051009+0000","last_clean":"2023-10-30T12:23:22.051009+0000","last_became_active":"2023-10-30T12:01:59.277217+0000","last_became_peered":"2023-10-30T12:01:59.277217+0000","last_unstale":"2023-10-30T12:23:22.051009+0000","last_undegraded":"2023-10-30T12:23:22.051009+0000","last_fullsized":"2023-10-30T12:23:22.051009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.050938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.050938+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.050938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:06.125027+0000","scrub_duration":0.0012565440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:26.057976+0000","last_change":"2023-10-30T12:23:26.058042+0000","last_active":"2023-10-30T12:23:26.057976+0000","last_peered":"2023-10-30T12:23:26.057976+0000","last_clean":"2023-10-30T12:23:26.057976+0000","last_became_active":"2023-10-30T12:01:59.246101+0000","last_became_peered":"2023-10-30T12:01:59.246101+0000","last_unstale":"2023-10-30T12:23:26.057976+0000","last_undegraded":"2023-10-30T12:23:26.057976+0000","last_fullsized":"2023-10-30T12:23:26.057976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:26.057897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:26.057897+0000","last_clean_scrub_stamp":"2023-10-30T12:23:26.057897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:25.272602+0000","scrub_duration":0.0028676180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:16.731281+0000","last_change":"2023-10-30T12:23:16.731316+0000","last_active":"2023-10-30T12:23:16.731281+0000","last_peered":"2023-10-30T12:23:16.731281+0000","last_clean":"2023-10-30T12:23:16.731281+0000","last_became_active":"2023-10-30T12:01:59.269001+0000","last_became_peered":"2023-10-30T12:01:59.269001+0000","last_unstale":"2023-10-30T12:23:16.731281+0000","last_undegraded":"2023-10-30T12:23:16.731281+0000","last_fullsized":"2023-10-30T12:23:16.731281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:16.731211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:16.731211+0000","last_clean_scrub_stamp":"2023-10-30T12:23:16.731211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:54:46.097638+0000","scrub_duration":0.001926624,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:24.294887+0000","last_change":"2023-10-30T12:23:24.294957+0000","last_active":"2023-10-30T12:23:24.294887+0000","last_peered":"2023-10-30T12:23:24.294887+0000","last_clean":"2023-10-30T12:23:24.294887+0000","last_became_active":"2023-10-30T12:01:59.314812+0000","last_became_peered":"2023-10-30T12:01:59.314812+0000","last_unstale":"2023-10-30T12:23:24.294887+0000","last_undegraded":"2023-10-30T12:23:24.294887+0000","last_fullsized":"2023-10-30T12:23:24.294887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:24.294737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:24.294737+0000","last_clean_scrub_stamp":"2023-10-30T12:23:24.294737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:52.025084+0000","scrub_duration":0.0040212520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":673,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:28.583622+0000","last_change":"2023-10-30T12:23:28.583669+0000","last_active":"2023-10-30T12:23:28.583622+0000","last_peered":"2023-10-30T12:23:28.583622+0000","last_clean":"2023-10-30T12:23:28.583622+0000","last_became_active":"2023-10-30T12:01:59.338820+0000","last_became_peered":"2023-10-30T12:01:59.338820+0000","last_unstale":"2023-10-30T12:23:28.583622+0000","last_undegraded":"2023-10-30T12:23:28.583622+0000","last_fullsized":"2023-10-30T12:23:28.583622+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:28.583537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:28.583537+0000","last_clean_scrub_stamp":"2023-10-30T12:23:28.583537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:58.563507+0000","scrub_duration":0.0020370190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":672,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:25.572225+0000","last_change":"2023-10-30T12:23:25.572281+0000","last_active":"2023-10-30T12:23:25.572225+0000","last_peered":"2023-10-30T12:23:25.572225+0000","last_clean":"2023-10-30T12:23:25.572225+0000","last_became_active":"2023-10-30T12:01:59.296246+0000","last_became_peered":"2023-10-30T12:01:59.296246+0000","last_unstale":"2023-10-30T12:23:25.572225+0000","last_undegraded":"2023-10-30T12:23:25.572225+0000","last_fullsized":"2023-10-30T12:23:25.572225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:25.572139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:25.572139+0000","last_clean_scrub_stamp":"2023-10-30T12:23:25.572139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:30:05.572263+0000","scrub_duration":0.003219356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:23.339792+0000","last_change":"2023-10-30T12:23:23.339848+0000","last_active":"2023-10-30T12:23:23.339792+0000","last_peered":"2023-10-30T12:23:23.339792+0000","last_clean":"2023-10-30T12:23:23.339792+0000","last_became_active":"2023-10-30T12:01:59.296511+0000","last_became_peered":"2023-10-30T12:01:59.296511+0000","last_unstale":"2023-10-30T12:23:23.339792+0000","last_undegraded":"2023-10-30T12:23:23.339792+0000","last_fullsized":"2023-10-30T12:23:23.339792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:23.339704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:23.339704+0000","last_clean_scrub_stamp":"2023-10-30T12:23:23.339704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:23.941804+0000","scrub_duration":0.001707178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:21.036654+0000","last_change":"2023-10-30T12:23:21.036694+0000","last_active":"2023-10-30T12:23:21.036654+0000","last_peered":"2023-10-30T12:23:21.036654+0000","last_clean":"2023-10-30T12:23:21.036654+0000","last_became_active":"2023-10-30T12:01:59.282069+0000","last_became_peered":"2023-10-30T12:01:59.282069+0000","last_unstale":"2023-10-30T12:23:21.036654+0000","last_undegraded":"2023-10-30T12:23:21.036654+0000","last_fullsized":"2023-10-30T12:23:21.036654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:21.036559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:21.036559+0000","last_clean_scrub_stamp":"2023-10-30T12:23:21.036559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:12:55.972365+0000","scrub_duration":0.0022463069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":651,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.663274+0000","last_change":"2023-10-30T12:23:19.663318+0000","last_active":"2023-10-30T12:23:19.663274+0000","last_peered":"2023-10-30T12:23:19.663274+0000","last_clean":"2023-10-30T12:23:19.663274+0000","last_became_active":"2023-10-30T12:01:59.266725+0000","last_became_peered":"2023-10-30T12:01:59.266725+0000","last_unstale":"2023-10-30T12:23:19.663274+0000","last_undegraded":"2023-10-30T12:23:19.663274+0000","last_fullsized":"2023-10-30T12:23:19.663274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.663197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.663197+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.663197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:06.339497+0000","scrub_duration":0.001409296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:23.544975+0000","last_change":"2023-10-30T12:23:23.545019+0000","last_active":"2023-10-30T12:23:23.544975+0000","last_peered":"2023-10-30T12:23:23.544975+0000","last_clean":"2023-10-30T12:23:23.544975+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:23:23.544975+0000","last_undegraded":"2023-10-30T12:23:23.544975+0000","last_fullsized":"2023-10-30T12:23:23.544975+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:23.544892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:23.544892+0000","last_clean_scrub_stamp":"2023-10-30T12:23:23.544892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:12:55.039855+0000","scrub_duration":0.0014902439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.866613+0000","last_change":"2023-10-30T12:23:22.866657+0000","last_active":"2023-10-30T12:23:22.866613+0000","last_peered":"2023-10-30T12:23:22.866613+0000","last_clean":"2023-10-30T12:23:22.866613+0000","last_became_active":"2023-10-30T12:01:59.274444+0000","last_became_peered":"2023-10-30T12:01:59.274444+0000","last_unstale":"2023-10-30T12:23:22.866613+0000","last_undegraded":"2023-10-30T12:23:22.866613+0000","last_fullsized":"2023-10-30T12:23:22.866613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.866525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.866525+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.866525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:36:46.228780+0000","scrub_duration":0.001319738,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":663,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:28.133407+0000","last_change":"2023-10-30T12:23:28.133453+0000","last_active":"2023-10-30T12:23:28.133407+0000","last_peered":"2023-10-30T12:23:28.133407+0000","last_clean":"2023-10-30T12:23:28.133407+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:23:28.133407+0000","last_undegraded":"2023-10-30T12:23:28.133407+0000","last_fullsized":"2023-10-30T12:23:28.133407+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:28.133331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:28.133331+0000","last_clean_scrub_stamp":"2023-10-30T12:23:28.133331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:03.511823+0000","scrub_duration":0.0019102469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:32.895617+0000","last_change":"2023-10-30T12:23:32.895661+0000","last_active":"2023-10-30T12:23:32.895617+0000","last_peered":"2023-10-30T12:23:32.895617+0000","last_clean":"2023-10-30T12:23:32.895617+0000","last_became_active":"2023-10-30T12:01:59.272125+0000","last_became_peered":"2023-10-30T12:01:59.272125+0000","last_unstale":"2023-10-30T12:23:32.895617+0000","last_undegraded":"2023-10-30T12:23:32.895617+0000","last_fullsized":"2023-10-30T12:23:32.895617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:32.895528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:32.895528+0000","last_clean_scrub_stamp":"2023-10-30T12:23:32.895528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:25:50.392514+0000","scrub_duration":0.0030611589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":677,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:20.556737+0000","last_change":"2023-10-30T12:23:20.556804+0000","last_active":"2023-10-30T12:23:20.556737+0000","last_peered":"2023-10-30T12:23:20.556737+0000","last_clean":"2023-10-30T12:23:20.556737+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:23:20.556737+0000","last_undegraded":"2023-10-30T12:23:20.556737+0000","last_fullsized":"2023-10-30T12:23:20.556737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:20.556644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:20.556644+0000","last_clean_scrub_stamp":"2023-10-30T12:23:20.556644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:07.027218+0000","scrub_duration":0.0020766999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610002+0000","last_change":"2023-10-30T12:01:59.269863+0000","last_active":"2023-10-30T12:22:44.610002+0000","last_peered":"2023-10-30T12:22:44.610002+0000","last_clean":"2023-10-30T12:22:44.610002+0000","last_became_active":"2023-10-30T12:01:59.269127+0000","last_became_peered":"2023-10-30T12:01:59.269127+0000","last_unstale":"2023-10-30T12:22:44.610002+0000","last_undegraded":"2023-10-30T12:22:44.610002+0000","last_fullsized":"2023-10-30T12:22:44.610002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":663,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:20.230983+0000","last_change":"2023-10-30T12:23:20.231029+0000","last_active":"2023-10-30T12:23:20.230983+0000","last_peered":"2023-10-30T12:23:20.230983+0000","last_clean":"2023-10-30T12:23:20.230983+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:23:20.230983+0000","last_undegraded":"2023-10-30T12:23:20.230983+0000","last_fullsized":"2023-10-30T12:23:20.230983+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:20.230896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:20.230896+0000","last_clean_scrub_stamp":"2023-10-30T12:23:20.230896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:39.228622+0000","scrub_duration":0.00164442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":645,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:45.448850+0000","last_change":"2023-10-30T12:01:59.306784+0000","last_active":"2023-10-30T12:22:45.448850+0000","last_peered":"2023-10-30T12:22:45.448850+0000","last_clean":"2023-10-30T12:22:45.448850+0000","last_became_active":"2023-10-30T12:01:59.306293+0000","last_became_peered":"2023-10-30T12:01:59.306293+0000","last_unstale":"2023-10-30T12:22:45.448850+0000","last_undegraded":"2023-10-30T12:22:45.448850+0000","last_fullsized":"2023-10-30T12:22:45.448850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":657,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.382974+0000","last_change":"2023-10-30T12:23:19.383016+0000","last_active":"2023-10-30T12:23:19.382974+0000","last_peered":"2023-10-30T12:23:19.382974+0000","last_clean":"2023-10-30T12:23:19.382974+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:23:19.382974+0000","last_undegraded":"2023-10-30T12:23:19.382974+0000","last_fullsized":"2023-10-30T12:23:19.382974+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.382889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.382889+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.382889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:44:05.674822+0000","scrub_duration":0.001638839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":673,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:32.492426+0000","last_change":"2023-10-30T12:23:32.492472+0000","last_active":"2023-10-30T12:23:32.492426+0000","last_peered":"2023-10-30T12:23:32.492426+0000","last_clean":"2023-10-30T12:23:32.492426+0000","last_became_active":"2023-10-30T12:01:59.313414+0000","last_became_peered":"2023-10-30T12:01:59.313414+0000","last_unstale":"2023-10-30T12:23:32.492426+0000","last_undegraded":"2023-10-30T12:23:32.492426+0000","last_fullsized":"2023-10-30T12:23:32.492426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:32.492334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:32.492334+0000","last_clean_scrub_stamp":"2023-10-30T12:23:32.492334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:37.769000+0000","scrub_duration":0.0032407759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":665,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.873204+0000","last_change":"2023-10-30T12:23:18.873251+0000","last_active":"2023-10-30T12:23:18.873204+0000","last_peered":"2023-10-30T12:23:18.873204+0000","last_clean":"2023-10-30T12:23:18.873204+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:23:18.873204+0000","last_undegraded":"2023-10-30T12:23:18.873204+0000","last_fullsized":"2023-10-30T12:23:18.873204+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.873126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.873126+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.873126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:12.606571+0000","scrub_duration":0.0015593950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.250540+0000","last_change":"2023-10-30T12:23:29.250608+0000","last_active":"2023-10-30T12:23:29.250540+0000","last_peered":"2023-10-30T12:23:29.250540+0000","last_clean":"2023-10-30T12:23:29.250540+0000","last_became_active":"2023-10-30T12:01:59.289175+0000","last_became_peered":"2023-10-30T12:01:59.289175+0000","last_unstale":"2023-10-30T12:23:29.250540+0000","last_undegraded":"2023-10-30T12:23:29.250540+0000","last_fullsized":"2023-10-30T12:23:29.250540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:29.250439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:29.250439+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.250439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:10.722207+0000","scrub_duration":0.0013662959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":677,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:21.542207+0000","last_change":"2023-10-30T12:23:21.542272+0000","last_active":"2023-10-30T12:23:21.542207+0000","last_peered":"2023-10-30T12:23:21.542207+0000","last_clean":"2023-10-30T12:23:21.542207+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:23:21.542207+0000","last_undegraded":"2023-10-30T12:23:21.542207+0000","last_fullsized":"2023-10-30T12:23:21.542207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:21.542130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:21.542130+0000","last_clean_scrub_stamp":"2023-10-30T12:23:21.542130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:12.658987+0000","scrub_duration":0.0020392639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:30.095199+0000","last_change":"2023-10-30T12:23:30.095249+0000","last_active":"2023-10-30T12:23:30.095199+0000","last_peered":"2023-10-30T12:23:30.095199+0000","last_clean":"2023-10-30T12:23:30.095199+0000","last_became_active":"2023-10-30T12:01:59.302155+0000","last_became_peered":"2023-10-30T12:01:59.302155+0000","last_unstale":"2023-10-30T12:23:30.095199+0000","last_undegraded":"2023-10-30T12:23:30.095199+0000","last_fullsized":"2023-10-30T12:23:30.095199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:30.095111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:30.095111+0000","last_clean_scrub_stamp":"2023-10-30T12:23:30.095111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:02.227834+0000","scrub_duration":0.0025979240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":657,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.413486+0000","last_change":"2023-10-30T12:23:18.413538+0000","last_active":"2023-10-30T12:23:18.413486+0000","last_peered":"2023-10-30T12:23:18.413486+0000","last_clean":"2023-10-30T12:23:18.413486+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:23:18.413486+0000","last_undegraded":"2023-10-30T12:23:18.413486+0000","last_fullsized":"2023-10-30T12:23:18.413486+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.413381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.413381+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.413381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:15.717044+0000","scrub_duration":0.00185736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"686'119","reported_seq":831,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.536942+0000","last_change":"2023-10-30T12:23:19.537004+0000","last_active":"2023-10-30T12:23:19.536942+0000","last_peered":"2023-10-30T12:23:19.536942+0000","last_clean":"2023-10-30T12:23:19.536942+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:23:19.536942+0000","last_undegraded":"2023-10-30T12:23:19.536942+0000","last_fullsized":"2023-10-30T12:23:19.536942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'119","last_scrub_stamp":"2023-10-30T12:23:19.536853+0000","last_deep_scrub":"686'119","last_deep_scrub_stamp":"2023-10-30T12:23:19.536853+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.536853+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:12.041107+0000","scrub_duration":0.0043842680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610169+0000","last_change":"2023-10-30T12:01:59.256285+0000","last_active":"2023-10-30T12:22:44.610169+0000","last_peered":"2023-10-30T12:22:44.610169+0000","last_clean":"2023-10-30T12:22:44.610169+0000","last_became_active":"2023-10-30T12:01:59.253931+0000","last_became_peered":"2023-10-30T12:01:59.253931+0000","last_unstale":"2023-10-30T12:22:44.610169+0000","last_undegraded":"2023-10-30T12:22:44.610169+0000","last_fullsized":"2023-10-30T12:22:44.610169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:49.577767+0000","last_change":"2023-10-30T12:01:59.296719+0000","last_active":"2023-10-30T12:22:49.577767+0000","last_peered":"2023-10-30T12:22:49.577767+0000","last_clean":"2023-10-30T12:22:49.577767+0000","last_became_active":"2023-10-30T12:01:59.277183+0000","last_became_peered":"2023-10-30T12:01:59.277183+0000","last_unstale":"2023-10-30T12:22:49.577767+0000","last_undegraded":"2023-10-30T12:22:49.577767+0000","last_fullsized":"2023-10-30T12:22:49.577767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":674,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:27.569483+0000","last_change":"2023-10-30T12:23:27.569531+0000","last_active":"2023-10-30T12:23:27.569483+0000","last_peered":"2023-10-30T12:23:27.569483+0000","last_clean":"2023-10-30T12:23:27.569483+0000","last_became_active":"2023-10-30T12:01:59.339614+0000","last_became_peered":"2023-10-30T12:01:59.339614+0000","last_unstale":"2023-10-30T12:23:27.569483+0000","last_undegraded":"2023-10-30T12:23:27.569483+0000","last_fullsized":"2023-10-30T12:23:27.569483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:27.569393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:27.569393+0000","last_clean_scrub_stamp":"2023-10-30T12:23:27.569393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:10:08.233929+0000","scrub_duration":0.002727185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.692790+0000","last_change":"2023-10-30T12:23:18.692837+0000","last_active":"2023-10-30T12:23:18.692790+0000","last_peered":"2023-10-30T12:23:18.692790+0000","last_clean":"2023-10-30T12:23:18.692790+0000","last_became_active":"2023-10-30T12:01:59.303216+0000","last_became_peered":"2023-10-30T12:01:59.303216+0000","last_unstale":"2023-10-30T12:23:18.692790+0000","last_undegraded":"2023-10-30T12:23:18.692790+0000","last_fullsized":"2023-10-30T12:23:18.692790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.692709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.692709+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.692709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:02.768827+0000","scrub_duration":0.002027278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.106769+0000","last_change":"2023-10-30T12:23:29.106812+0000","last_active":"2023-10-30T12:23:29.106769+0000","last_peered":"2023-10-30T12:23:29.106769+0000","last_clean":"2023-10-30T12:23:29.106769+0000","last_became_active":"2023-10-30T12:01:59.209551+0000","last_became_peered":"2023-10-30T12:01:59.209551+0000","last_unstale":"2023-10-30T12:23:29.106769+0000","last_undegraded":"2023-10-30T12:23:29.106769+0000","last_fullsized":"2023-10-30T12:23:29.106769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:29.106681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:29.106681+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.106681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:50.945615+0000","scrub_duration":0.0032893250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":654,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.609993+0000","last_change":"2023-10-30T12:01:59.312819+0000","last_active":"2023-10-30T12:22:44.609993+0000","last_peered":"2023-10-30T12:22:44.609993+0000","last_clean":"2023-10-30T12:22:44.609993+0000","last_became_active":"2023-10-30T12:01:59.310341+0000","last_became_peered":"2023-10-30T12:01:59.310341+0000","last_unstale":"2023-10-30T12:22:44.609993+0000","last_undegraded":"2023-10-30T12:22:44.609993+0000","last_fullsized":"2023-10-30T12:22:44.609993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"686'8","reported_seq":688,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.543719+0000","last_change":"2023-10-30T12:23:29.543806+0000","last_active":"2023-10-30T12:23:29.543719+0000","last_peered":"2023-10-30T12:23:29.543719+0000","last_clean":"2023-10-30T12:23:29.543719+0000","last_became_active":"2023-10-30T12:01:59.260849+0000","last_became_peered":"2023-10-30T12:01:59.260849+0000","last_unstale":"2023-10-30T12:23:29.543719+0000","last_undegraded":"2023-10-30T12:23:29.543719+0000","last_fullsized":"2023-10-30T12:23:29.543719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'8","last_scrub_stamp":"2023-10-30T12:23:29.543628+0000","last_deep_scrub":"686'8","last_deep_scrub_stamp":"2023-10-30T12:23:29.543628+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.543628+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:42.277331+0000","scrub_duration":0.0034138010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:49.579990+0000","last_change":"2023-10-30T12:08:57.463451+0000","last_active":"2023-10-30T12:22:49.579990+0000","last_peered":"2023-10-30T12:22:49.579990+0000","last_clean":"2023-10-30T12:22:49.579990+0000","last_became_active":"2023-10-30T12:08:57.463102+0000","last_became_peered":"2023-10-30T12:08:57.463102+0000","last_unstale":"2023-10-30T12:22:49.579990+0000","last_undegraded":"2023-10-30T12:22:49.579990+0000","last_fullsized":"2023-10-30T12:22:49.579990+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.082953+0000","last_change":"2023-10-30T12:23:19.082999+0000","last_active":"2023-10-30T12:23:19.082953+0000","last_peered":"2023-10-30T12:23:19.082953+0000","last_clean":"2023-10-30T12:23:19.082953+0000","last_became_active":"2023-10-30T12:01:59.246868+0000","last_became_peered":"2023-10-30T12:01:59.246868+0000","last_unstale":"2023-10-30T12:23:19.082953+0000","last_undegraded":"2023-10-30T12:23:19.082953+0000","last_fullsized":"2023-10-30T12:23:19.082953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.082868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.082868+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.082868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:50.129280+0000","scrub_duration":0.0020950420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":658,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:31.113426+0000","last_change":"2023-10-30T12:23:31.113469+0000","last_active":"2023-10-30T12:23:31.113426+0000","last_peered":"2023-10-30T12:23:31.113426+0000","last_clean":"2023-10-30T12:23:31.113426+0000","last_became_active":"2023-10-30T12:01:59.239117+0000","last_became_peered":"2023-10-30T12:01:59.239117+0000","last_unstale":"2023-10-30T12:23:31.113426+0000","last_undegraded":"2023-10-30T12:23:31.113426+0000","last_fullsized":"2023-10-30T12:23:31.113426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:31.113340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:31.113340+0000","last_clean_scrub_stamp":"2023-10-30T12:23:31.113340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:02:44.961562+0000","scrub_duration":0.0016503100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:31.058690+0000","last_change":"2023-10-30T12:23:31.058765+0000","last_active":"2023-10-30T12:23:31.058690+0000","last_peered":"2023-10-30T12:23:31.058690+0000","last_clean":"2023-10-30T12:23:31.058690+0000","last_became_active":"2023-10-30T12:01:59.293055+0000","last_became_peered":"2023-10-30T12:01:59.293055+0000","last_unstale":"2023-10-30T12:23:31.058690+0000","last_undegraded":"2023-10-30T12:23:31.058690+0000","last_fullsized":"2023-10-30T12:23:31.058690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:31.058583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:31.058583+0000","last_clean_scrub_stamp":"2023-10-30T12:23:31.058583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:48.548044+0000","scrub_duration":0.002423377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:44.610218+0000","last_change":"2023-10-30T12:01:59.310309+0000","last_active":"2023-10-30T12:22:44.610218+0000","last_peered":"2023-10-30T12:22:44.610218+0000","last_clean":"2023-10-30T12:22:44.610218+0000","last_became_active":"2023-10-30T12:01:59.307440+0000","last_became_peered":"2023-10-30T12:01:59.307440+0000","last_unstale":"2023-10-30T12:22:44.610218+0000","last_undegraded":"2023-10-30T12:22:44.610218+0000","last_fullsized":"2023-10-30T12:22:44.610218+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:30.140984+0000","last_change":"2023-10-30T12:23:30.141028+0000","last_active":"2023-10-30T12:23:30.140984+0000","last_peered":"2023-10-30T12:23:30.140984+0000","last_clean":"2023-10-30T12:23:30.140984+0000","last_became_active":"2023-10-30T12:01:59.239219+0000","last_became_peered":"2023-10-30T12:01:59.239219+0000","last_unstale":"2023-10-30T12:23:30.140984+0000","last_undegraded":"2023-10-30T12:23:30.140984+0000","last_fullsized":"2023-10-30T12:23:30.140984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:30.140907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:30.140907+0000","last_clean_scrub_stamp":"2023-10-30T12:23:30.140907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:52.739662+0000","scrub_duration":0.0037158099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640531,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77224,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294616,"statfs":{"total":96636764160,"available":96557686784,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.82699999999999996},"last":0.629},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52000000000000002,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.65000000000000002,"5min":0.81599999999999995,"15min":1.056},"last":0.54500000000000004}]},{"osd":1,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65400000000000003,"5min":0.83399999999999996,"15min":2.173},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.28299999999999997},"max":{"1min":0.64800000000000002,"5min":0.72799999999999998,"15min":0.78100000000000003},"last":0.52800000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.67400000000000004,"5min":0.74299999999999999,"15min":0.79900000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.41499999999999998,"15min":0.32900000000000001},"max":{"1min":0.65000000000000002,"5min":0.79900000000000004,"15min":0.84699999999999998},"last":0.501}]},{"osd":3,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.36899999999999999,"15min":0.373},"min":{"1min":0.25,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":0.499,"5min":0.58799999999999997,"15min":0.70399999999999996},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.223},"max":{"1min":0.54500000000000004,"5min":0.69699999999999995,"15min":0.95199999999999996},"last":0.29199999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.31,"5min":0.252,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.63700000000000001,"15min":0.63800000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.435,"15min":0.45300000000000001},"min":{"1min":0.28100000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.60299999999999998,"5min":0.68400000000000005,"15min":0.72099999999999997},"last":0.318}]}]},{"osd":4,"up_from":11,"seq":47244640532,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81384,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94290456,"statfs":{"total":96636764160,"available":96553426944,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53900000000000003,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.504,"15min":0.51400000000000001},"min":{"1min":0.36599999999999999,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.84899999999999998},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.29299999999999998},"max":{"1min":0.65000000000000002,"5min":0.79000000000000004,"15min":0.94099999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.36299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.67700000000000005,"5min":0.749,"15min":0.79800000000000004},"last":0.53400000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27300000000000002},"max":{"1min":0.627,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.499}]},{"osd":3,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.246},"max":{"1min":0.59399999999999997,"5min":0.627,"15min":0.63600000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38800000000000001},"min":{"1min":0.25,"5min":0.249,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.67000000000000004,"15min":0.77100000000000002},"last":0.34100000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.435,"15min":0.40999999999999998},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.25600000000000001},"max":{"1min":0.621,"5min":0.64000000000000001,"15min":0.69999999999999996},"last":0.311},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":0.64400000000000002,"5min":0.67400000000000004,"15min":0.89700000000000002},"last":0.32200000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640532,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69428,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302412,"statfs":{"total":96636764160,"available":96565669888,"internally_reserved":0,"allocated":2207744,"data_stored":996605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.501},"min":{"1min":0.39800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.498,"15min":0.5},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.307},"max":{"1min":0.66800000000000004,"5min":0.70099999999999996,"15min":1.2490000000000001},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.55900000000000005},"min":{"1min":0.41199999999999998,"5min":0.33100000000000002,"15min":0.32000000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.2689999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.58999999999999997},"min":{"1min":0.44400000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":1.337},"last":0.70199999999999996}]},{"osd":2,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.52000000000000002},"min":{"1min":0.377,"5min":0.314,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.88900000000000001,"15min":1.302},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53300000000000003,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.56599999999999995}]},{"osd":4,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.377},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55100000000000005,"5min":0.61199999999999999,"15min":1.371},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":1.444},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.25600000000000001},"max":{"1min":0.61899999999999999,"5min":0.64200000000000002,"15min":0.876},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.627,"5min":0.84099999999999997,"15min":1.3540000000000001},"last":0.376}]}]},{"osd":2,"up_from":11,"seq":47244640531,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64748,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94307092,"statfs":{"total":96636764160,"available":96570462208,"internally_reserved":0,"allocated":1675264,"data_stored":467677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33900000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.192},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":2.2730000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.22800000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":2.141},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20799999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":1.087},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.39400000000000002,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.27300000000000002,"15min":0.24399999999999999},"max":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":3.8260000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":1.143},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.78800000000000003,"15min":3.4729999999999999},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.34399999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.3240000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82099999999999995,"5min":0.85999999999999999,"15min":1.8340000000000001},"last":0.496}]},{"osd":5,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":0.81599999999999995,"5min":0.81699999999999995,"15min":1.893},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.61499999999999999,"15min":0.60299999999999998},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.81599999999999995,"15min":2.2269999999999999},"last":0.432}]}]},{"osd":1,"up_from":11,"seq":47244640533,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59176,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312664,"statfs":{"total":96636764160,"available":96576167936,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.34499999999999997},"min":{"1min":0.23100000000000001,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":0.46899999999999997,"5min":0.745,"15min":0.745},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.373,"15min":0.374},"min":{"1min":0.253,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.26000000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.38200000000000001},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.223},"max":{"1min":0.54000000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.51800000000000002,"5min":0.621,"15min":0.93200000000000005},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.443,"5min":0.375,"15min":0.28499999999999998},"max":{"1min":0.69999999999999996,"5min":0.86599999999999999,"15min":0.879},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.315},"max":{"1min":0.79900000000000004,"5min":0.85399999999999998,"15min":0.88900000000000001},"last":0.315}]},{"osd":4,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58899999999999997,"15min":0.59199999999999997},"min":{"1min":0.46400000000000002,"5min":0.41699999999999998,"15min":0.29899999999999999},"max":{"1min":0.86799999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.36799999999999999},"max":{"1min":0.71299999999999997,"5min":0.94799999999999995,"15min":0.95599999999999996},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.58899999999999997},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.33800000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.39300000000000002},"max":{"1min":0.84499999999999997,"5min":1,"15min":1},"last":0.375}]}]},{"osd":0,"up_from":11,"seq":47244640533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77744,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294096,"statfs":{"total":96636764160,"available":96557154304,"internally_reserved":0,"allocated":2203648,"data_stored":996600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.36499999999999999,"15min":0.35199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23899999999999999},"max":{"1min":3.1440000000000001,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.45800000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.21299999999999999},"max":{"1min":1.802,"5min":1.802,"15min":3.8519999999999999},"last":0.314},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26700000000000002,"15min":0.20100000000000001},"max":{"1min":0.58699999999999997,"5min":0.623,"15min":2.0590000000000002},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.27800000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":1.5489999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.56699999999999995},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.28199999999999997},"max":{"1min":0.875,"5min":0.875,"15min":1.78},"last":0.57999999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.626,"15min":0.61299999999999999},"min":{"1min":0.46899999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.29399999999999998},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59099999999999997,"15min":0.57899999999999996},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":1.6020000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:23:38.183 INFO:tasks.ceph:pgid 15.13 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.184 INFO:tasks.ceph:pgid 15.8 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.184 INFO:tasks.ceph:pgid 15.9 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.184 INFO:tasks.ceph:pgid 15.e last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.185 INFO:tasks.ceph:pgid 15.4 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.185 INFO:tasks.ceph:pgid 15.0 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.186 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.186 INFO:tasks.ceph:pgid 15.1e last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:38.187 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:23:58.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:23:58.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.305+0000 7f856e440640 10 monclient: get_monmap_and_config 2023-10-30T12:23:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.305+0000 7f856e440640 10 monclient: build_initial_monmap 2023-10-30T12:23:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.305+0000 7f856e440640 10 monclient: monmap: 2023-10-30T12:23:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:23:58.306246+0000 2023-10-30T12:23:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:23:58.306246+0000 2023-10-30T12:23:58.308 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:58.309 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:58.309 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:23:58.309 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:23:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:23:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:23:58.310 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:23:58.310 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.305+0000 7f856e440640 1 Processor -- start 2023-10-30T12:23:58.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 1 -- start start 2023-10-30T12:23:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient: init 2023-10-30T12:23:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-10-30T12:23:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85680a4b60 addr v1:172.21.15.118:6789/0 2023-10-30T12:23:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8568099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:23:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8568132db0 0x7f856813e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:58.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient(hunting): picked mon.noname-b con 0x7f8568132db0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85680a4f40 con 0x7f8568099a40 2023-10-30T12:23:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85680a4150 con 0x7f85680a4b60 2023-10-30T12:23:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85680a3d70 con 0x7f8568132db0 2023-10-30T12:23:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856e440640 10 monclient(hunting): authenticate will time out at 2283.489592s 2023-10-30T12:23:58.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f85677fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8568099a40 0x7f856813aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45450/0 (socket says 172.21.15.50:45450) 2023-10-30T12:23:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f85677fe640 1 -- 172.21.15.50:0/87764046 learned_addr learned my addr 172.21.15.50:0/87764046 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:23:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 1 --2- 172.21.15.50:0/87764046 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8568132db0 0x7f856813e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 10 monclient(hunting): get_auth_request con 0x7f8568132db0 auth_method 0 2023-10-30T12:23:58.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f856c9dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f8566ffd640 1 -- 172.21.15.50:0/87764046 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152940906 0 0) 0x7f85680a4f40 con 0x7f8568099a40 2023-10-30T12:23:58.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f8566ffd640 10 monclient(hunting): _negotiate 2023-10-30T12:23:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f8566ffd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f8566ffd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:58.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f8566ffd640 10 monclient(hunting): my global_id is 264474 2023-10-30T12:23:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.306+0000 7f8566ffd640 1 -- 172.21.15.50:0/87764046 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8550003520 con 0x7f8568099a40 2023-10-30T12:23:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 10 monclient(hunting): handle_auth_done global_id 284210 payload 274 2023-10-30T12:23:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 1 -- 172.21.15.50:0/87764046 >> v1:172.21.15.118:6789/0 conn(0x7f85680a4b60 legacy=0x7f85681326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:58.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 1 -- 172.21.15.50:0/87764046 >> v1:172.21.15.50:6789/0 conn(0x7f8568099a40 legacy=0x7f856813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 10 monclient: _finish_hunting 0 2023-10-30T12:23:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 1 monclient: found mon.noname-b 2023-10-30T12:23:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:58.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:23:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 1 -- 172.21.15.50:0/87764046 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856809a440 con 0x7f8568132db0 2023-10-30T12:23:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856c9dd640 1 --2- 172.21.15.50:0/87764046 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8568132db0 0x7f856813e1b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8560009dc0 tx=0x7f856002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0844403d50cc979 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:58.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 1 -- 172.21.15.50:0/87764046 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856002cbb0 con 0x7f8568132db0 2023-10-30T12:23:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:23:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 10 monclient: dump: 2023-10-30T12:23:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.331 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:58.332 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:58.333 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 10 monclient: _finish_auth 0 2023-10-30T12:23:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 10 monclient: _check_auth_tickets 2023-10-30T12:23:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856e440640 5 monclient: authenticate success, global_id 284210 2023-10-30T12:23:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 1 -- 172.21.15.50:0/87764046 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856002cd10 con 0x7f8568132db0 2023-10-30T12:23:58.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856e440640 10 monclient: get_monmap_and_config success 2023-10-30T12:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f856e440640 4 set_mon_vals no callback set 2023-10-30T12:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 1 -- 172.21.15.50:0/87764046 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8560034660 con 0x7f8568132db0 2023-10-30T12:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.307+0000 7f8566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f8566ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f8566ffd640 10 monclient: dump: 2023-10-30T12:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:58.338 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:58.339 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 10 monclient: shutdown 2023-10-30T12:23:58.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 1 -- 172.21.15.50:0/87764046 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8568132db0 msgr2=0x7f856813e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 1 --2- 172.21.15.50:0/87764046 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8568132db0 0x7f856813e1b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8560009dc0 tx=0x7f856002be00 comp rx=0 tx=0).stop 2023-10-30T12:23:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 1 -- 172.21.15.50:0/87764046 shutdown_connections 2023-10-30T12:23:58.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 1 --2- 172.21.15.50:0/87764046 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8568132db0 0x7f856813e1b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 1 -- 172.21.15.50:0/87764046 >> 172.21.15.50:0/87764046 conn(0x7f85680a8060 msgr2=0x7f85680a4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 1 -- 172.21.15.50:0/87764046 shutdown_connections 2023-10-30T12:23:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.308+0000 7f856e440640 1 -- 172.21.15.50:0/87764046 wait complete. 2023-10-30T12:23:58.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 10 monclient: build_initial_monmap 2023-10-30T12:23:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 10 monclient: monmap: 2023-10-30T12:23:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:23:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:23:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:23:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:23:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:23:58.343 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:23:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:23:58.344 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:23:58.344 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 1 librados: starting msgr at 2023-10-30T12:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 1 librados: starting objecter 2023-10-30T12:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 1 Processor -- start 2023-10-30T12:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 1 -- start start 2023-10-30T12:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 1 librados: setting wanted keys 2023-10-30T12:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 1 librados: calling monclient init 2023-10-30T12:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.309+0000 7f856e440640 10 monclient: init 2023-10-30T12:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient: _reopen_session rank -1 2023-10-30T12:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85680a4b60 0x7f8568135140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): picked mon.noname-b con 0x7f85680a4b60 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:23:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8568132db0 0x7f856812db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): picked mon.noname-a con 0x7f8568132db0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:23:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f856812e070 0x7f856812e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): picked mon.noname-c con 0x7f856812e070 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:23:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856812fad0 con 0x7f8568132db0 2023-10-30T12:23:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85680a18d0 con 0x7f85680a4b60 2023-10-30T12:23:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): start opening mon connection 2023-10-30T12:23:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856813d950 con 0x7f856812e070 2023-10-30T12:23:58.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): _renew_subs 2023-10-30T12:23:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856e440640 10 monclient(hunting): authenticate will time out at 2283.493280s 2023-10-30T12:23:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8568132db0 0x7f856812db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:58.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8568132db0 0x7f856812db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34288/0 (socket says 172.21.15.50:34288) 2023-10-30T12:23:58.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f856812e070 0x7f856812e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:58.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 -- 172.21.15.50:0/1024227532 learned_addr learned my addr 172.21.15.50:0/1024227532 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:23:58.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): get_auth_request con 0x7f8568132db0 auth_method 0 2023-10-30T12:23:58.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:58.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:58.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:58.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 10 monclient(hunting): get_auth_request con 0x7f856812e070 auth_method 0 2023-10-30T12:23:58.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85680a4b60 0x7f8568135140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:58.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:58.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:58.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:58.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 10 monclient(hunting): get_auth_request con 0x7f85680a4b60 auth_method 0 2023-10-30T12:23:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:23:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:23:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:23:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f856c9dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:23:58.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:23:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:23:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient(hunting): handle_auth_done global_id 264477 payload 824 2023-10-30T12:23:58.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 -- 172.21.15.50:0/1024227532 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f856812e070 msgr2=0x7f856812e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f856812e070 0x7f856812e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 -- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85680a4b60 msgr2=0x7f8568135140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85680a4b60 0x7f8568135140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient: _finish_hunting 0 2023-10-30T12:23:58.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 monclient: found mon.noname-a 2023-10-30T12:23:58.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:23:58.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.50:3300/0 2023-10-30T12:23:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 -- 172.21.15.50:0/1024227532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560009a70 con 0x7f8568132db0 2023-10-30T12:23:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f85677fe640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8568132db0 0x7f856812db30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f855800c3c0 tx=0x7f855800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8564ff9640 10 client.?.objecter ms_handle_connect 0x7f8568132db0 2023-10-30T12:23:58.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.310+0000 7f8564ff9640 10 client.?.objecter resend_mon_ops 2023-10-30T12:23:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 1 -- 172.21.15.50:0/1024227532 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558005ce0 con 0x7f8568132db0 2023-10-30T12:23:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-10-30T12:23:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: dump: 2023-10-30T12:23:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:58.364 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:58.365 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: _finish_auth 0 2023-10-30T12:23:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: _check_auth_tickets 2023-10-30T12:23:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:23:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 1 -- 172.21.15.50:0/1024227532 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8558005e60 con 0x7f8568132db0 2023-10-30T12:23:58.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 5 monclient: authenticate success, global_id 264477 2023-10-30T12:23:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 10 monclient: _renew_subs 2023-10-30T12:23:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:58.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8568135e70 con 0x7f8568132db0 2023-10-30T12:23:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:23:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 1 -- 172.21.15.50:0/1024227532 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580072f0 con 0x7f8568132db0 2023-10-30T12:23:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:23:58.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-10-30T12:23:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 10 monclient: dump: 2023-10-30T12:23:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:23:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:23:58.369 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:23:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:23:58.370 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:23:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:23:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:23:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:23:58.371 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:23:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f85667fc640 4 set_mon_vals no callback set 2023-10-30T12:23:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 10 client.264477.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:23:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 10 monclient: _renew_subs 2023-10-30T12:23:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8568130040 con 0x7f8568132db0 2023-10-30T12:23:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f856e440640 1 librados: init done 2023-10-30T12:23:58.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.311+0000 7f8564ff9640 1 -- 172.21.15.50:0/1024227532 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f8558007470 con 0x7f8568132db0 2023-10-30T12:23:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8564ff9640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85540392f0 0x7f855403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:23:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8564ff9640 1 -- 172.21.15.50:0/1024227532 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f8558049f50 con 0x7f8568132db0 2023-10-30T12:23:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8564ff9640 10 client.264477.objecter ms_dispatch 0x7f8568000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:23:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8564ff9640 3 client.264477.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:23:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8564ff9640 3 client.264477.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:23:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f856e440640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f856e440640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f856e440640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.50:3300/0 2023-10-30T12:23:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856804e160 con 0x7f8568132db0 2023-10-30T12:23:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8564ff9640 20 client.264477.objecter dump_active .. 0 homeless 2023-10-30T12:23:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8567fff640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85540392f0 0x7f855403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:23:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.312+0000 7f8567fff640 10 monclient: get_auth_request con 0x7f85540392f0 auth_method 0 2023-10-30T12:23:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.313+0000 7f8567fff640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85540392f0 0x7f855403b7b0 secure :-1 s=READY pgs=4372 cs=0 l=1 rev1=1 crypto rx=0x7f855c000f60 tx=0x7f855c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:23:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.313+0000 7f8564ff9640 10 client.264477.objecter ms_handle_connect 0x7f85540392f0 2023-10-30T12:23:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.316+0000 7f8564ff9640 1 -- 172.21.15.50:0/1024227532 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8558017020 con 0x7f8568132db0 2023-10-30T12:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.316+0000 7f8564ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.316+0000 7f8564ff9640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:23:58.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.493+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f856809aa00 con 0x7f85540392f0 2023-10-30T12:23:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.498+0000 7f8564ff9640 1 -- 172.21.15.50:0/1024227532 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134463 (secure 0 0 0) 0x7f855c0014a0 con 0x7f85540392f0 2023-10-30T12:23:58.501 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:23:58.502 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:23:58.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 10 librados: watch_flush enter 2023-10-30T12:23:58.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 10 librados: watch_flush exit 2023-10-30T12:23:58.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 20 client.264477.objecter shutdown clearing up homeless session... 2023-10-30T12:23:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 10 client.264477.objecter successfully canceled tick 2023-10-30T12:23:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85540392f0 msgr2=0x7f855403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85540392f0 0x7f855403b7b0 secure :-1 s=READY pgs=4372 cs=0 l=1 rev1=1 crypto rx=0x7f855c000f60 tx=0x7f855c009040 comp rx=0 tx=0).stop 2023-10-30T12:23:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 10 monclient: shutdown 2023-10-30T12:23:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:23:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8568132db0 msgr2=0x7f856812db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:23:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8568132db0 0x7f856812db30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f855800c3c0 tx=0x7f855800c880 comp rx=0 tx=0).stop 2023-10-30T12:23:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 shutdown_connections 2023-10-30T12:23:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f85540392f0 0x7f855403b7b0 unknown :-1 s=CLOSED pgs=4372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f856812e070 0x7f856812e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8568132db0 0x7f856812db30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 --2- 172.21.15.50:0/1024227532 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85680a4b60 0x7f8568135140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:23:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 >> 172.21.15.50:0/1024227532 conn(0x7f85680a8060 msgr2=0x7f85680a7150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:23:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 shutdown_connections 2023-10-30T12:23:58.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.500+0000 7f856e440640 1 -- 172.21.15.50:0/1024227532 wait complete. 2023-10-30T12:23:58.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:23:58.501+0000 7f856e440640 1 librados: shutdown 2023-10-30T12:23:58.512 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1374,"stamp":"2023-10-30T12:23:56.746743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":429704,"kb_used_data":10840,"kb_used_omap":9,"kb_used_meta":418806,"kb_avail":565801336,"statfs":{"total":579820584960,"available":579380568064,"internally_reserved":0,"allocated":11100160,"data_stored":3863898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":428858023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"15.13","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:35.097222+0000","last_change":"2023-10-30T12:23:35.097268+0000","last_active":"2023-10-30T12:23:35.097222+0000","last_peered":"2023-10-30T12:23:35.097222+0000","last_clean":"2023-10-30T12:23:35.097222+0000","last_became_active":"2023-10-30T12:01:59.223072+0000","last_became_peered":"2023-10-30T12:01:59.223072+0000","last_unstale":"2023-10-30T12:23:35.097222+0000","last_undegraded":"2023-10-30T12:23:35.097222+0000","last_fullsized":"2023-10-30T12:23:35.097222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:35.097131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:35.097131+0000","last_clean_scrub_stamp":"2023-10-30T12:23:35.097131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:42:45.018845+0000","scrub_duration":0.0027366370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.679786+0000","last_change":"2023-10-30T12:23:22.679842+0000","last_active":"2023-10-30T12:23:22.679786+0000","last_peered":"2023-10-30T12:23:22.679786+0000","last_clean":"2023-10-30T12:23:22.679786+0000","last_became_active":"2023-10-30T12:01:59.344017+0000","last_became_peered":"2023-10-30T12:01:59.344017+0000","last_unstale":"2023-10-30T12:23:22.679786+0000","last_undegraded":"2023-10-30T12:23:22.679786+0000","last_fullsized":"2023-10-30T12:23:22.679786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.679707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.679707+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.679707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:10.004012+0000","scrub_duration":0.0020254040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:27.841929+0000","last_change":"2023-10-30T12:23:27.842001+0000","last_active":"2023-10-30T12:23:27.841929+0000","last_peered":"2023-10-30T12:23:27.841929+0000","last_clean":"2023-10-30T12:23:27.841929+0000","last_became_active":"2023-10-30T12:01:59.308994+0000","last_became_peered":"2023-10-30T12:01:59.308994+0000","last_unstale":"2023-10-30T12:23:27.841929+0000","last_undegraded":"2023-10-30T12:23:27.841929+0000","last_fullsized":"2023-10-30T12:23:27.841929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:27.841846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:27.841846+0000","last_clean_scrub_stamp":"2023-10-30T12:23:27.841846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:25.854306+0000","scrub_duration":0.0022774079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.051009+0000","last_change":"2023-10-30T12:23:22.051049+0000","last_active":"2023-10-30T12:23:22.051009+0000","last_peered":"2023-10-30T12:23:22.051009+0000","last_clean":"2023-10-30T12:23:22.051009+0000","last_became_active":"2023-10-30T12:01:59.277217+0000","last_became_peered":"2023-10-30T12:01:59.277217+0000","last_unstale":"2023-10-30T12:23:22.051009+0000","last_undegraded":"2023-10-30T12:23:22.051009+0000","last_fullsized":"2023-10-30T12:23:22.051009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.050938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.050938+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.050938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:06.125027+0000","scrub_duration":0.0012565440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:26.058042+0000","last_change":"2023-10-30T12:23:26.058042+0000","last_active":"2023-10-30T12:23:26.058042+0000","last_peered":"2023-10-30T12:23:26.058042+0000","last_clean":"2023-10-30T12:23:26.058042+0000","last_became_active":"2023-10-30T12:01:59.246101+0000","last_became_peered":"2023-10-30T12:01:59.246101+0000","last_unstale":"2023-10-30T12:23:26.058042+0000","last_undegraded":"2023-10-30T12:23:26.058042+0000","last_fullsized":"2023-10-30T12:23:26.058042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:26.057897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:26.057897+0000","last_clean_scrub_stamp":"2023-10-30T12:23:26.057897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:25.272602+0000","scrub_duration":0.0028676180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:16.731316+0000","last_change":"2023-10-30T12:23:16.731316+0000","last_active":"2023-10-30T12:23:16.731316+0000","last_peered":"2023-10-30T12:23:16.731316+0000","last_clean":"2023-10-30T12:23:16.731316+0000","last_became_active":"2023-10-30T12:01:59.269001+0000","last_became_peered":"2023-10-30T12:01:59.269001+0000","last_unstale":"2023-10-30T12:23:16.731316+0000","last_undegraded":"2023-10-30T12:23:16.731316+0000","last_fullsized":"2023-10-30T12:23:16.731316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:16.731211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:16.731211+0000","last_clean_scrub_stamp":"2023-10-30T12:23:16.731211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:54:46.097638+0000","scrub_duration":0.001926624,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:24.294957+0000","last_change":"2023-10-30T12:23:24.294957+0000","last_active":"2023-10-30T12:23:24.294957+0000","last_peered":"2023-10-30T12:23:24.294957+0000","last_clean":"2023-10-30T12:23:24.294957+0000","last_became_active":"2023-10-30T12:01:59.314812+0000","last_became_peered":"2023-10-30T12:01:59.314812+0000","last_unstale":"2023-10-30T12:23:24.294957+0000","last_undegraded":"2023-10-30T12:23:24.294957+0000","last_fullsized":"2023-10-30T12:23:24.294957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:24.294737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:24.294737+0000","last_clean_scrub_stamp":"2023-10-30T12:23:24.294737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:52.025084+0000","scrub_duration":0.0040212520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":673,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:28.583622+0000","last_change":"2023-10-30T12:23:28.583669+0000","last_active":"2023-10-30T12:23:28.583622+0000","last_peered":"2023-10-30T12:23:28.583622+0000","last_clean":"2023-10-30T12:23:28.583622+0000","last_became_active":"2023-10-30T12:01:59.338820+0000","last_became_peered":"2023-10-30T12:01:59.338820+0000","last_unstale":"2023-10-30T12:23:28.583622+0000","last_undegraded":"2023-10-30T12:23:28.583622+0000","last_fullsized":"2023-10-30T12:23:28.583622+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:28.583537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:28.583537+0000","last_clean_scrub_stamp":"2023-10-30T12:23:28.583537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:58.563507+0000","scrub_duration":0.0020370190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":672,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:25.572225+0000","last_change":"2023-10-30T12:23:25.572281+0000","last_active":"2023-10-30T12:23:25.572225+0000","last_peered":"2023-10-30T12:23:25.572225+0000","last_clean":"2023-10-30T12:23:25.572225+0000","last_became_active":"2023-10-30T12:01:59.296246+0000","last_became_peered":"2023-10-30T12:01:59.296246+0000","last_unstale":"2023-10-30T12:23:25.572225+0000","last_undegraded":"2023-10-30T12:23:25.572225+0000","last_fullsized":"2023-10-30T12:23:25.572225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:25.572139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:25.572139+0000","last_clean_scrub_stamp":"2023-10-30T12:23:25.572139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:30:05.572263+0000","scrub_duration":0.003219356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:23.339792+0000","last_change":"2023-10-30T12:23:23.339848+0000","last_active":"2023-10-30T12:23:23.339792+0000","last_peered":"2023-10-30T12:23:23.339792+0000","last_clean":"2023-10-30T12:23:23.339792+0000","last_became_active":"2023-10-30T12:01:59.296511+0000","last_became_peered":"2023-10-30T12:01:59.296511+0000","last_unstale":"2023-10-30T12:23:23.339792+0000","last_undegraded":"2023-10-30T12:23:23.339792+0000","last_fullsized":"2023-10-30T12:23:23.339792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:23.339704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:23.339704+0000","last_clean_scrub_stamp":"2023-10-30T12:23:23.339704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:23.941804+0000","scrub_duration":0.001707178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:21.036694+0000","last_change":"2023-10-30T12:23:21.036694+0000","last_active":"2023-10-30T12:23:21.036694+0000","last_peered":"2023-10-30T12:23:21.036694+0000","last_clean":"2023-10-30T12:23:21.036694+0000","last_became_active":"2023-10-30T12:01:59.282069+0000","last_became_peered":"2023-10-30T12:01:59.282069+0000","last_unstale":"2023-10-30T12:23:21.036694+0000","last_undegraded":"2023-10-30T12:23:21.036694+0000","last_fullsized":"2023-10-30T12:23:21.036694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:21.036559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:21.036559+0000","last_clean_scrub_stamp":"2023-10-30T12:23:21.036559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:12:55.972365+0000","scrub_duration":0.0022463069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.663318+0000","last_change":"2023-10-30T12:23:19.663318+0000","last_active":"2023-10-30T12:23:19.663318+0000","last_peered":"2023-10-30T12:23:19.663318+0000","last_clean":"2023-10-30T12:23:19.663318+0000","last_became_active":"2023-10-30T12:01:59.266725+0000","last_became_peered":"2023-10-30T12:01:59.266725+0000","last_unstale":"2023-10-30T12:23:19.663318+0000","last_undegraded":"2023-10-30T12:23:19.663318+0000","last_fullsized":"2023-10-30T12:23:19.663318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.663197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.663197+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.663197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:06.339497+0000","scrub_duration":0.001409296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:23.544975+0000","last_change":"2023-10-30T12:23:23.545019+0000","last_active":"2023-10-30T12:23:23.544975+0000","last_peered":"2023-10-30T12:23:23.544975+0000","last_clean":"2023-10-30T12:23:23.544975+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:23:23.544975+0000","last_undegraded":"2023-10-30T12:23:23.544975+0000","last_fullsized":"2023-10-30T12:23:23.544975+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:23.544892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:23.544892+0000","last_clean_scrub_stamp":"2023-10-30T12:23:23.544892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:12:55.039855+0000","scrub_duration":0.0014902439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.866657+0000","last_change":"2023-10-30T12:23:22.866657+0000","last_active":"2023-10-30T12:23:22.866657+0000","last_peered":"2023-10-30T12:23:22.866657+0000","last_clean":"2023-10-30T12:23:22.866657+0000","last_became_active":"2023-10-30T12:01:59.274444+0000","last_became_peered":"2023-10-30T12:01:59.274444+0000","last_unstale":"2023-10-30T12:23:22.866657+0000","last_undegraded":"2023-10-30T12:23:22.866657+0000","last_fullsized":"2023-10-30T12:23:22.866657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.866525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.866525+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.866525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:36:46.228780+0000","scrub_duration":0.001319738,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":663,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:28.133407+0000","last_change":"2023-10-30T12:23:28.133453+0000","last_active":"2023-10-30T12:23:28.133407+0000","last_peered":"2023-10-30T12:23:28.133407+0000","last_clean":"2023-10-30T12:23:28.133407+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:23:28.133407+0000","last_undegraded":"2023-10-30T12:23:28.133407+0000","last_fullsized":"2023-10-30T12:23:28.133407+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:28.133331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:28.133331+0000","last_clean_scrub_stamp":"2023-10-30T12:23:28.133331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:03.511823+0000","scrub_duration":0.0019102469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:32.895617+0000","last_change":"2023-10-30T12:23:32.895661+0000","last_active":"2023-10-30T12:23:32.895617+0000","last_peered":"2023-10-30T12:23:32.895617+0000","last_clean":"2023-10-30T12:23:32.895617+0000","last_became_active":"2023-10-30T12:01:59.272125+0000","last_became_peered":"2023-10-30T12:01:59.272125+0000","last_unstale":"2023-10-30T12:23:32.895617+0000","last_undegraded":"2023-10-30T12:23:32.895617+0000","last_fullsized":"2023-10-30T12:23:32.895617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:32.895528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:32.895528+0000","last_clean_scrub_stamp":"2023-10-30T12:23:32.895528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:25:50.392514+0000","scrub_duration":0.0030611589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":678,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:20.556804+0000","last_change":"2023-10-30T12:23:20.556804+0000","last_active":"2023-10-30T12:23:20.556804+0000","last_peered":"2023-10-30T12:23:20.556804+0000","last_clean":"2023-10-30T12:23:20.556804+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:23:20.556804+0000","last_undegraded":"2023-10-30T12:23:20.556804+0000","last_fullsized":"2023-10-30T12:23:20.556804+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:20.556644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:20.556644+0000","last_clean_scrub_stamp":"2023-10-30T12:23:20.556644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:07.027218+0000","scrub_duration":0.0020766999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":654,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:15.968271+0000","last_change":"2023-10-30T12:01:59.269863+0000","last_active":"2023-10-30T12:23:15.968271+0000","last_peered":"2023-10-30T12:23:15.968271+0000","last_clean":"2023-10-30T12:23:15.968271+0000","last_became_active":"2023-10-30T12:01:59.269127+0000","last_became_peered":"2023-10-30T12:01:59.269127+0000","last_unstale":"2023-10-30T12:23:15.968271+0000","last_undegraded":"2023-10-30T12:23:15.968271+0000","last_fullsized":"2023-10-30T12:23:15.968271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":664,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:20.231029+0000","last_change":"2023-10-30T12:23:20.231029+0000","last_active":"2023-10-30T12:23:20.231029+0000","last_peered":"2023-10-30T12:23:20.231029+0000","last_clean":"2023-10-30T12:23:20.231029+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:23:20.231029+0000","last_undegraded":"2023-10-30T12:23:20.231029+0000","last_fullsized":"2023-10-30T12:23:20.231029+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:20.230896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:20.230896+0000","last_clean_scrub_stamp":"2023-10-30T12:23:20.230896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:39.228622+0000","scrub_duration":0.00164442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":654,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:36.180099+0000","last_change":"2023-10-30T12:23:36.180099+0000","last_active":"2023-10-30T12:23:36.180099+0000","last_peered":"2023-10-30T12:23:36.180099+0000","last_clean":"2023-10-30T12:23:36.180099+0000","last_became_active":"2023-10-30T12:01:59.306293+0000","last_became_peered":"2023-10-30T12:01:59.306293+0000","last_unstale":"2023-10-30T12:23:36.180099+0000","last_undegraded":"2023-10-30T12:23:36.180099+0000","last_fullsized":"2023-10-30T12:23:36.180099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:36.179942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:36.179942+0000","last_clean_scrub_stamp":"2023-10-30T12:23:36.179942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:12.386250+0000","scrub_duration":0.002831882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":658,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.383016+0000","last_change":"2023-10-30T12:23:19.383016+0000","last_active":"2023-10-30T12:23:19.383016+0000","last_peered":"2023-10-30T12:23:19.383016+0000","last_clean":"2023-10-30T12:23:19.383016+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:23:19.383016+0000","last_undegraded":"2023-10-30T12:23:19.383016+0000","last_fullsized":"2023-10-30T12:23:19.383016+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.382889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.382889+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.382889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:44:05.674822+0000","scrub_duration":0.001638839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":673,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:32.492426+0000","last_change":"2023-10-30T12:23:32.492472+0000","last_active":"2023-10-30T12:23:32.492426+0000","last_peered":"2023-10-30T12:23:32.492426+0000","last_clean":"2023-10-30T12:23:32.492426+0000","last_became_active":"2023-10-30T12:01:59.313414+0000","last_became_peered":"2023-10-30T12:01:59.313414+0000","last_unstale":"2023-10-30T12:23:32.492426+0000","last_undegraded":"2023-10-30T12:23:32.492426+0000","last_fullsized":"2023-10-30T12:23:32.492426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:32.492334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:32.492334+0000","last_clean_scrub_stamp":"2023-10-30T12:23:32.492334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:37.769000+0000","scrub_duration":0.0032407759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":666,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.873251+0000","last_change":"2023-10-30T12:23:18.873251+0000","last_active":"2023-10-30T12:23:18.873251+0000","last_peered":"2023-10-30T12:23:18.873251+0000","last_clean":"2023-10-30T12:23:18.873251+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:23:18.873251+0000","last_undegraded":"2023-10-30T12:23:18.873251+0000","last_fullsized":"2023-10-30T12:23:18.873251+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.873126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.873126+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.873126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:12.606571+0000","scrub_duration":0.0015593950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.250540+0000","last_change":"2023-10-30T12:23:29.250608+0000","last_active":"2023-10-30T12:23:29.250540+0000","last_peered":"2023-10-30T12:23:29.250540+0000","last_clean":"2023-10-30T12:23:29.250540+0000","last_became_active":"2023-10-30T12:01:59.289175+0000","last_became_peered":"2023-10-30T12:01:59.289175+0000","last_unstale":"2023-10-30T12:23:29.250540+0000","last_undegraded":"2023-10-30T12:23:29.250540+0000","last_fullsized":"2023-10-30T12:23:29.250540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:29.250439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:29.250439+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.250439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:10.722207+0000","scrub_duration":0.0013662959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":677,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:21.542207+0000","last_change":"2023-10-30T12:23:21.542272+0000","last_active":"2023-10-30T12:23:21.542207+0000","last_peered":"2023-10-30T12:23:21.542207+0000","last_clean":"2023-10-30T12:23:21.542207+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:23:21.542207+0000","last_undegraded":"2023-10-30T12:23:21.542207+0000","last_fullsized":"2023-10-30T12:23:21.542207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:21.542130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:21.542130+0000","last_clean_scrub_stamp":"2023-10-30T12:23:21.542130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:12.658987+0000","scrub_duration":0.0020392639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:30.095199+0000","last_change":"2023-10-30T12:23:30.095249+0000","last_active":"2023-10-30T12:23:30.095199+0000","last_peered":"2023-10-30T12:23:30.095199+0000","last_clean":"2023-10-30T12:23:30.095199+0000","last_became_active":"2023-10-30T12:01:59.302155+0000","last_became_peered":"2023-10-30T12:01:59.302155+0000","last_unstale":"2023-10-30T12:23:30.095199+0000","last_undegraded":"2023-10-30T12:23:30.095199+0000","last_fullsized":"2023-10-30T12:23:30.095199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:30.095111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:30.095111+0000","last_clean_scrub_stamp":"2023-10-30T12:23:30.095111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:02.227834+0000","scrub_duration":0.0025979240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":658,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.413538+0000","last_change":"2023-10-30T12:23:18.413538+0000","last_active":"2023-10-30T12:23:18.413538+0000","last_peered":"2023-10-30T12:23:18.413538+0000","last_clean":"2023-10-30T12:23:18.413538+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:23:18.413538+0000","last_undegraded":"2023-10-30T12:23:18.413538+0000","last_fullsized":"2023-10-30T12:23:18.413538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.413381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.413381+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.413381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:15.717044+0000","scrub_duration":0.00185736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"686'119","reported_seq":831,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.536942+0000","last_change":"2023-10-30T12:23:19.537004+0000","last_active":"2023-10-30T12:23:19.536942+0000","last_peered":"2023-10-30T12:23:19.536942+0000","last_clean":"2023-10-30T12:23:19.536942+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:23:19.536942+0000","last_undegraded":"2023-10-30T12:23:19.536942+0000","last_fullsized":"2023-10-30T12:23:19.536942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'119","last_scrub_stamp":"2023-10-30T12:23:19.536853+0000","last_deep_scrub":"686'119","last_deep_scrub_stamp":"2023-10-30T12:23:19.536853+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.536853+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:12.041107+0000","scrub_duration":0.0043842680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:44.858009+0000","last_change":"2023-10-30T12:23:44.858062+0000","last_active":"2023-10-30T12:23:44.858009+0000","last_peered":"2023-10-30T12:23:44.858009+0000","last_clean":"2023-10-30T12:23:44.858009+0000","last_became_active":"2023-10-30T12:01:59.253931+0000","last_became_peered":"2023-10-30T12:01:59.253931+0000","last_unstale":"2023-10-30T12:23:44.858009+0000","last_undegraded":"2023-10-30T12:23:44.858009+0000","last_fullsized":"2023-10-30T12:23:44.858009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:44.857928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:44.857928+0000","last_clean_scrub_stamp":"2023-10-30T12:23:44.857928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:08.641471+0000","scrub_duration":0.0016835999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:40.071963+0000","last_change":"2023-10-30T12:23:40.072011+0000","last_active":"2023-10-30T12:23:40.071963+0000","last_peered":"2023-10-30T12:23:40.071963+0000","last_clean":"2023-10-30T12:23:40.071963+0000","last_became_active":"2023-10-30T12:01:59.277183+0000","last_became_peered":"2023-10-30T12:01:59.277183+0000","last_unstale":"2023-10-30T12:23:40.071963+0000","last_undegraded":"2023-10-30T12:23:40.071963+0000","last_fullsized":"2023-10-30T12:23:40.071963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:40.071880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:40.071880+0000","last_clean_scrub_stamp":"2023-10-30T12:23:40.071880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:48.102089+0000","scrub_duration":0.001972892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":674,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:27.569483+0000","last_change":"2023-10-30T12:23:27.569531+0000","last_active":"2023-10-30T12:23:27.569483+0000","last_peered":"2023-10-30T12:23:27.569483+0000","last_clean":"2023-10-30T12:23:27.569483+0000","last_became_active":"2023-10-30T12:01:59.339614+0000","last_became_peered":"2023-10-30T12:01:59.339614+0000","last_unstale":"2023-10-30T12:23:27.569483+0000","last_undegraded":"2023-10-30T12:23:27.569483+0000","last_fullsized":"2023-10-30T12:23:27.569483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:27.569393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:27.569393+0000","last_clean_scrub_stamp":"2023-10-30T12:23:27.569393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:10:08.233929+0000","scrub_duration":0.002727185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.692837+0000","last_change":"2023-10-30T12:23:18.692837+0000","last_active":"2023-10-30T12:23:18.692837+0000","last_peered":"2023-10-30T12:23:18.692837+0000","last_clean":"2023-10-30T12:23:18.692837+0000","last_became_active":"2023-10-30T12:01:59.303216+0000","last_became_peered":"2023-10-30T12:01:59.303216+0000","last_unstale":"2023-10-30T12:23:18.692837+0000","last_undegraded":"2023-10-30T12:23:18.692837+0000","last_fullsized":"2023-10-30T12:23:18.692837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.692709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.692709+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.692709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:02.768827+0000","scrub_duration":0.002027278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.106812+0000","last_change":"2023-10-30T12:23:29.106812+0000","last_active":"2023-10-30T12:23:29.106812+0000","last_peered":"2023-10-30T12:23:29.106812+0000","last_clean":"2023-10-30T12:23:29.106812+0000","last_became_active":"2023-10-30T12:01:59.209551+0000","last_became_peered":"2023-10-30T12:01:59.209551+0000","last_unstale":"2023-10-30T12:23:29.106812+0000","last_undegraded":"2023-10-30T12:23:29.106812+0000","last_fullsized":"2023-10-30T12:23:29.106812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:29.106681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:29.106681+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.106681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:50.945615+0000","scrub_duration":0.0032893250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":662,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:47.873563+0000","last_change":"2023-10-30T12:23:47.873611+0000","last_active":"2023-10-30T12:23:47.873563+0000","last_peered":"2023-10-30T12:23:47.873563+0000","last_clean":"2023-10-30T12:23:47.873563+0000","last_became_active":"2023-10-30T12:01:59.310341+0000","last_became_peered":"2023-10-30T12:01:59.310341+0000","last_unstale":"2023-10-30T12:23:47.873563+0000","last_undegraded":"2023-10-30T12:23:47.873563+0000","last_fullsized":"2023-10-30T12:23:47.873563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:47.873470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:47.873470+0000","last_clean_scrub_stamp":"2023-10-30T12:23:47.873470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:48.784611+0000","scrub_duration":0.0019736110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"686'8","reported_seq":688,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.543719+0000","last_change":"2023-10-30T12:23:29.543806+0000","last_active":"2023-10-30T12:23:29.543719+0000","last_peered":"2023-10-30T12:23:29.543719+0000","last_clean":"2023-10-30T12:23:29.543719+0000","last_became_active":"2023-10-30T12:01:59.260849+0000","last_became_peered":"2023-10-30T12:01:59.260849+0000","last_unstale":"2023-10-30T12:23:29.543719+0000","last_undegraded":"2023-10-30T12:23:29.543719+0000","last_fullsized":"2023-10-30T12:23:29.543719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'8","last_scrub_stamp":"2023-10-30T12:23:29.543628+0000","last_deep_scrub":"686'8","last_deep_scrub_stamp":"2023-10-30T12:23:29.543628+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.543628+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:42.277331+0000","scrub_duration":0.0034138010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:22:49.579990+0000","last_change":"2023-10-30T12:08:57.463451+0000","last_active":"2023-10-30T12:22:49.579990+0000","last_peered":"2023-10-30T12:22:49.579990+0000","last_clean":"2023-10-30T12:22:49.579990+0000","last_became_active":"2023-10-30T12:08:57.463102+0000","last_became_peered":"2023-10-30T12:08:57.463102+0000","last_unstale":"2023-10-30T12:22:49.579990+0000","last_undegraded":"2023-10-30T12:22:49.579990+0000","last_fullsized":"2023-10-30T12:22:49.579990+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:01:58.122184+0000","last_clean_scrub_stamp":"2023-10-30T12:01:58.122184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.082953+0000","last_change":"2023-10-30T12:23:19.082999+0000","last_active":"2023-10-30T12:23:19.082953+0000","last_peered":"2023-10-30T12:23:19.082953+0000","last_clean":"2023-10-30T12:23:19.082953+0000","last_became_active":"2023-10-30T12:01:59.246868+0000","last_became_peered":"2023-10-30T12:01:59.246868+0000","last_unstale":"2023-10-30T12:23:19.082953+0000","last_undegraded":"2023-10-30T12:23:19.082953+0000","last_fullsized":"2023-10-30T12:23:19.082953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.082868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.082868+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.082868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:50.129280+0000","scrub_duration":0.0020950420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":658,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:31.113426+0000","last_change":"2023-10-30T12:23:31.113469+0000","last_active":"2023-10-30T12:23:31.113426+0000","last_peered":"2023-10-30T12:23:31.113426+0000","last_clean":"2023-10-30T12:23:31.113426+0000","last_became_active":"2023-10-30T12:01:59.239117+0000","last_became_peered":"2023-10-30T12:01:59.239117+0000","last_unstale":"2023-10-30T12:23:31.113426+0000","last_undegraded":"2023-10-30T12:23:31.113426+0000","last_fullsized":"2023-10-30T12:23:31.113426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:31.113340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:31.113340+0000","last_clean_scrub_stamp":"2023-10-30T12:23:31.113340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:02:44.961562+0000","scrub_duration":0.0016503100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:31.058690+0000","last_change":"2023-10-30T12:23:31.058765+0000","last_active":"2023-10-30T12:23:31.058690+0000","last_peered":"2023-10-30T12:23:31.058690+0000","last_clean":"2023-10-30T12:23:31.058690+0000","last_became_active":"2023-10-30T12:01:59.293055+0000","last_became_peered":"2023-10-30T12:01:59.293055+0000","last_unstale":"2023-10-30T12:23:31.058690+0000","last_undegraded":"2023-10-30T12:23:31.058690+0000","last_fullsized":"2023-10-30T12:23:31.058690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:31.058583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:31.058583+0000","last_clean_scrub_stamp":"2023-10-30T12:23:31.058583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:48.548044+0000","scrub_duration":0.002423377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:42.894646+0000","last_change":"2023-10-30T12:23:42.894694+0000","last_active":"2023-10-30T12:23:42.894646+0000","last_peered":"2023-10-30T12:23:42.894646+0000","last_clean":"2023-10-30T12:23:42.894646+0000","last_became_active":"2023-10-30T12:01:59.307440+0000","last_became_peered":"2023-10-30T12:01:59.307440+0000","last_unstale":"2023-10-30T12:23:42.894646+0000","last_undegraded":"2023-10-30T12:23:42.894646+0000","last_fullsized":"2023-10-30T12:23:42.894646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:42.894562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:42.894562+0000","last_clean_scrub_stamp":"2023-10-30T12:23:42.894562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:54:50.034244+0000","scrub_duration":0.001821285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:30.140984+0000","last_change":"2023-10-30T12:23:30.141028+0000","last_active":"2023-10-30T12:23:30.140984+0000","last_peered":"2023-10-30T12:23:30.140984+0000","last_clean":"2023-10-30T12:23:30.140984+0000","last_became_active":"2023-10-30T12:01:59.239219+0000","last_became_peered":"2023-10-30T12:01:59.239219+0000","last_unstale":"2023-10-30T12:23:30.140984+0000","last_undegraded":"2023-10-30T12:23:30.140984+0000","last_fullsized":"2023-10-30T12:23:30.140984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:30.140907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:30.140907+0000","last_clean_scrub_stamp":"2023-10-30T12:23:30.140907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:52.739662+0000","scrub_duration":0.0037158099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77224,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294616,"statfs":{"total":96636764160,"available":96557686784,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":0.63700000000000001,"5min":0.71999999999999997,"15min":0.82699999999999996},"last":0.627},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.35499999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.66800000000000004,"5min":0.71799999999999997,"15min":1.056},"last":0.80700000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.371,"5min":0.246,"15min":0.246},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":2.173},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.35199999999999998,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":0.70899999999999996,"15min":0.78100000000000003},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.72699999999999998,"5min":0.74299999999999999,"15min":0.79900000000000004},"last":0.746},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.48999999999999999,"5min":0.41499999999999998,"15min":0.34999999999999998},"max":{"1min":0.66200000000000003,"5min":0.74099999999999999,"15min":0.84699999999999998},"last":0.64800000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36599999999999999,"15min":0.372},"min":{"1min":0.26400000000000001,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":0.52100000000000002,"5min":0.58799999999999997,"15min":0.70399999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39900000000000002},"min":{"1min":0.28299999999999997,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58299999999999996,"5min":0.69699999999999995,"15min":0.95199999999999996},"last":0.505}]},{"osd":4,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.40799999999999997},"min":{"1min":0.30499999999999999,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":0.56799999999999995,"5min":0.63700000000000001,"15min":0.63800000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.42799999999999999,"15min":0.45100000000000001},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.55200000000000005,"5min":0.68400000000000005,"15min":0.72099999999999997},"last":0.55300000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640536,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81384,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94290456,"statfs":{"total":96636764160,"available":96553426944,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.41399999999999998,"5min":0.35099999999999998,"15min":0.24099999999999999},"max":{"1min":0.73399999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.371,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":0.70299999999999996,"5min":0.78700000000000003,"15min":0.84899999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.29299999999999998},"max":{"1min":0.65900000000000003,"5min":0.78000000000000003,"15min":0.94099999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53400000000000003},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79800000000000004},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.51500000000000001},"min":{"1min":0.40100000000000002,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.68100000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.40000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.627}]},{"osd":3,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.27900000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53900000000000003,"5min":0.627,"15min":0.63600000000000001},"last":0.317},{"interface":"front","average":{"1min":0.378,"5min":0.374,"15min":0.38400000000000001},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.23799999999999999},"max":{"1min":0.60199999999999998,"5min":0.67000000000000004,"15min":0.77100000000000002},"last":0.33700000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.434,"15min":0.41399999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.25600000000000001},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.69999999999999996},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.442,"15min":0.45200000000000001},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":0.58599999999999997,"5min":0.67400000000000004,"15min":0.89700000000000002},"last":0.38600000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640536,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69428,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302412,"statfs":{"total":96636764160,"available":96565669888,"internally_reserved":0,"allocated":2207744,"data_stored":996605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.501},"min":{"1min":0.39800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.498,"15min":0.5},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.307},"max":{"1min":0.66800000000000004,"5min":0.70099999999999996,"15min":1.2490000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.55900000000000005},"min":{"1min":0.41199999999999998,"5min":0.33100000000000002,"15min":0.32000000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.2689999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.58999999999999997},"min":{"1min":0.44400000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":1.337},"last":0.628}]},{"osd":2,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.52000000000000002},"min":{"1min":0.377,"5min":0.314,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.88900000000000001,"15min":1.302},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53300000000000003,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.377},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55100000000000005,"5min":0.61199999999999999,"15min":1.371},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":1.444},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.25600000000000001},"max":{"1min":0.61899999999999999,"5min":0.64200000000000002,"15min":0.876},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.627,"5min":0.84099999999999997,"15min":1.3540000000000001},"last":0.38600000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640535,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64748,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94307092,"statfs":{"total":96636764160,"available":96570462208,"internally_reserved":0,"allocated":1675264,"data_stored":467677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33900000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.192},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":2.2730000000000001},"last":0.371},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.22800000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":2.141},"last":0.48499999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20799999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":1.087},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.39400000000000002,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.27300000000000002,"15min":0.24399999999999999},"max":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":3.8260000000000001},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":1.143},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.78800000000000003,"15min":3.4729999999999999},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.34399999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.3240000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82099999999999995,"5min":0.85999999999999999,"15min":1.8340000000000001},"last":0.753}]},{"osd":5,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":0.81599999999999995,"5min":0.81699999999999995,"15min":1.893},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.61499999999999999,"15min":0.60299999999999998},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.81599999999999995,"15min":2.2269999999999999},"last":0.63600000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640537,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59176,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312664,"statfs":{"total":96636764160,"available":96576167936,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"min":{"1min":0.22900000000000001,"5min":0.22700000000000001,"15min":0.20499999999999999},"max":{"1min":0.53900000000000003,"5min":0.745,"15min":0.745},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.435,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.378,"15min":0.376},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.497,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.52600000000000002,"5min":0.621,"15min":0.621},"last":0.434}]},{"osd":3,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.57199999999999995},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75600000000000001,"5min":0.86599999999999999,"15min":0.879},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.85799999999999998},"last":0.745}]},{"osd":4,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68300000000000005,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.629},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.754,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60299999999999998,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.38900000000000001,"15min":0.38100000000000001},"max":{"1min":0.77900000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78600000000000003,"5min":1,"15min":1},"last":0.70599999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640537,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77744,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294096,"statfs":{"total":96636764160,"available":96557154304,"internally_reserved":0,"allocated":2203648,"data_stored":996600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.248,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":0.51100000000000001,"5min":1.865,"15min":1.865},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42199999999999999,"15min":0.41399999999999998},"min":{"1min":0.29599999999999999,"5min":0.27900000000000003,"15min":0.23899999999999999},"max":{"1min":0.55600000000000005,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38500000000000001},"min":{"1min":0.26300000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51000000000000001,"5min":1.802,"15min":3.8519999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.38600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":0.57099999999999995,"5min":0.623,"15min":2.0590000000000002},"last":0.53600000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.57499999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.27800000000000002},"max":{"1min":0.83999999999999997,"5min":0.95999999999999996,"15min":1.5489999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.35399999999999998,"15min":0.28199999999999997},"max":{"1min":0.74099999999999999,"5min":0.875,"15min":1.78},"last":0.61699999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.624,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79800000000000004,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.622,"15min":0.61699999999999999},"min":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.29399999999999998},"max":{"1min":0.81399999999999995,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.65900000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.33900000000000002},"max":{"1min":0.79200000000000004,"5min":0.88800000000000001,"15min":1.6020000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":0.75800000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:23:58.516 INFO:tasks.ceph:pgid 15.8 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:58.517 INFO:tasks.ceph:pgid 15.2 last_scrub_stamp 2023-10-30T12:01:58.122184+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=23, tm_sec=13, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:23:58.517 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:24:18.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:24:18.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.635+0000 7f52636f2640 10 monclient: get_monmap_and_config 2023-10-30T12:24:18.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.635+0000 7f52636f2640 10 monclient: build_initial_monmap 2023-10-30T12:24:18.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.635+0000 7f52636f2640 10 monclient: monmap: 2023-10-30T12:24:18.640 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:24:18.640 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.641 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:24:18.636973+0000 2023-10-30T12:24:18.641 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:24:18.636973+0000 2023-10-30T12:24:18.642 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:24:18.642 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.643 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:24:18.643 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:24:18.644 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:24:18.644 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:24:18.645 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:24:18.645 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.636+0000 7f52636f2640 1 Processor -- start 2023-10-30T12:24:18.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.636+0000 7f52636f2640 1 -- start start 2023-10-30T12:24:18.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.636+0000 7f52636f2640 10 monclient: init 2023-10-30T12:24:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:24:18.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-10-30T12:24:18.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ae0 0x7f525c097e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient(hunting): picked mon.noname-b con 0x7f525c135ae0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:24:18.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f525c135ec0 0x7f525c1362a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient(hunting): picked mon.noname-a con 0x7f525c135ec0 addr v2:172.21.15.50:3300/0 2023-10-30T12:24:18.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f525c06ff70 addr v1:172.21.15.118:6789/0 2023-10-30T12:24:18.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c071cf0 con 0x7f525c06ff70 2023-10-30T12:24:18.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:18.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c072380 con 0x7f525c135ec0 2023-10-30T12:24:18.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:18.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c097790 con 0x7f525c135ae0 2023-10-30T12:24:18.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:24:18.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f52636f2640 10 monclient(hunting): authenticate will time out at 2303.820299s 2023-10-30T12:24:18.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ae0 0x7f525c097e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f525c135ec0 0x7f525c1362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ae0 0x7f525c097e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40706/0 (socket says 172.21.15.50:40706) 2023-10-30T12:24:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 -- 172.21.15.50:0/3093947147 learned_addr learned my addr 172.21.15.50:0/3093947147 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:24:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): get_auth_request con 0x7f525c135ae0 auth_method 0 2023-10-30T12:24:18.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:18.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f525c135ec0 0x7f525c1362a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44394/0 (socket says 172.21.15.50:44394) 2023-10-30T12:24:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 10 monclient(hunting): get_auth_request con 0x7f525c135ec0 auth_method 0 2023-10-30T12:24:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:18.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:18.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:18.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:18.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:18.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:18.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient(hunting): handle_auth_done global_id 284216 payload 274 2023-10-30T12:24:18.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 -- 172.21.15.50:0/3093947147 >> v2:172.21.15.50:3300/0 conn(0x7f525c135ec0 msgr2=0x7f525c1362a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:24:18.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 --2- 172.21.15.50:0/3093947147 >> v2:172.21.15.50:3300/0 conn(0x7f525c135ec0 0x7f525c1362a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 -- 172.21.15.50:0/3093947147 >> v1:172.21.15.118:6789/0 conn(0x7f525c06ff70 legacy=0x7f525c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient: _finish_hunting 0 2023-10-30T12:24:18.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 monclient: found mon.noname-b 2023-10-30T12:24:18.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:24:18.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:24:18.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f526148e640 1 -- 172.21.15.50:0/3093947147 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c097a70 con 0x7f525c135ae0 2023-10-30T12:24:18.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.637+0000 7f5260c8d640 1 --2- 172.21.15.50:0/3093947147 >> v2:172.21.15.50:3300/0 conn(0x7f525c135ec0 0x7f525c1362a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:24:18.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f526148e640 1 --2- 172.21.15.50:0/3093947147 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ae0 0x7f525c097e90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f524c009850 tx=0x7f524c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=120e87aea7a1d81d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:24:18.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 1 -- 172.21.15.50:0/3093947147 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c02e070 con 0x7f525c135ae0 2023-10-30T12:24:18.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:24:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: dump: 2023-10-30T12:24:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.683 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:18.683 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.683 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:18.683 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:18.684 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:18.684 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: _finish_auth 0 2023-10-30T12:24:18.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: _check_auth_tickets 2023-10-30T12:24:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:24:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f52636f2640 5 monclient: authenticate success, global_id 284216 2023-10-30T12:24:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f52636f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:24:18.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 1 -- 172.21.15.50:0/3093947147 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524c004030 con 0x7f525c135ae0 2023-10-30T12:24:18.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:24:18.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f52636f2640 10 monclient: get_monmap_and_config success 2023-10-30T12:24:18.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f52636f2640 4 set_mon_vals no callback set 2023-10-30T12:24:18.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 1 -- 172.21.15.50:0/3093947147 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c004420 con 0x7f525c135ae0 2023-10-30T12:24:18.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:18.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:24:18.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.638+0000 7f5253fff640 10 monclient: dump: 2023-10-30T12:24:18.688 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:18.688 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.689 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.689 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.689 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:18.690 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.690 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:18.690 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:18.691 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:18.691 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 10 monclient: shutdown 2023-10-30T12:24:18.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:24:18.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 -- 172.21.15.50:0/3093947147 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ae0 msgr2=0x7f525c097e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 --2- 172.21.15.50:0/3093947147 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ae0 0x7f525c097e90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f524c009850 tx=0x7f524c02b840 comp rx=0 tx=0).stop 2023-10-30T12:24:18.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 -- 172.21.15.50:0/3093947147 shutdown_connections 2023-10-30T12:24:18.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 --2- 172.21.15.50:0/3093947147 >> v2:172.21.15.50:3300/0 conn(0x7f525c135ec0 0x7f525c1362a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 --2- 172.21.15.50:0/3093947147 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ae0 0x7f525c097e90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 -- 172.21.15.50:0/3093947147 >> 172.21.15.50:0/3093947147 conn(0x7f525c0a05c0 msgr2=0x7f525c09de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:24:18.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 -- 172.21.15.50:0/3093947147 shutdown_connections 2023-10-30T12:24:18.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.639+0000 7f52636f2640 1 -- 172.21.15.50:0/3093947147 wait complete. 2023-10-30T12:24:18.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient: build_initial_monmap 2023-10-30T12:24:18.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient: monmap: 2023-10-30T12:24:18.694 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:24:18.695 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:24:18.695 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:24:18.695 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:24:18.695 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:24:18.696 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.696 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:24:18.696 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:24:18.696 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:24:18.697 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 librados: starting msgr at 2023-10-30T12:24:18.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 librados: starting objecter 2023-10-30T12:24:18.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 Processor -- start 2023-10-30T12:24:18.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 -- start start 2023-10-30T12:24:18.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 librados: setting wanted keys 2023-10-30T12:24:18.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 librados: calling monclient init 2023-10-30T12:24:18.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient: init 2023-10-30T12:24:18.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient: _reopen_session rank -1 2023-10-30T12:24:18.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient: _add_conns ranks=[1,0,2] 2023-10-30T12:24:18.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f525c06ff70 0x7f525c1ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): picked mon.noname-b con 0x7f525c06ff70 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:24:18.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f525c135ae0 0x7f525c096340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): picked mon.noname-a con 0x7f525c135ae0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:24:18.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ec0 0x7f525c096880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): picked mon.noname-c con 0x7f525c135ec0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:24:18.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:18.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c093550 con 0x7f525c135ae0 2023-10-30T12:24:18.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:18.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c094be0 con 0x7f525c06ff70 2023-10-30T12:24:18.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:18.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c09d560 con 0x7f525c135ec0 2023-10-30T12:24:18.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): _renew_subs 2023-10-30T12:24:18.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.640+0000 7f52636f2640 10 monclient(hunting): authenticate will time out at 2303.823921s 2023-10-30T12:24:18.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ec0 0x7f525c096880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f525c135ae0 0x7f525c096340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ec0 0x7f525c096880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40714/0 (socket says 172.21.15.50:40714) 2023-10-30T12:24:18.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f525c135ae0 0x7f525c096340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44410/0 (socket says 172.21.15.50:44410) 2023-10-30T12:24:18.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 -- 172.21.15.50:0/1709557971 learned_addr learned my addr 172.21.15.50:0/1709557971 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:24:18.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): get_auth_request con 0x7f525c135ec0 auth_method 0 2023-10-30T12:24:18.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:18.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 10 monclient(hunting): get_auth_request con 0x7f525c135ae0 auth_method 0 2023-10-30T12:24:18.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:18.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:18.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:18.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:18.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:18.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:18.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:18.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f526148e640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f525c06ff70 0x7f525c1ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f526148e640 10 monclient(hunting): get_auth_request con 0x7f525c06ff70 auth_method 0 2023-10-30T12:24:18.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f526148e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:18.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f526148e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f526148e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient(hunting): handle_auth_done global_id 284219 payload 824 2023-10-30T12:24:18.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 -- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f525c06ff70 msgr2=0x7f525c1ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f525c06ff70 0x7f525c1ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 -- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f525c135ae0 msgr2=0x7f525c096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f525c135ae0 0x7f525c096340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient: _finish_hunting 0 2023-10-30T12:24:18.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 monclient: found mon.noname-c 2023-10-30T12:24:18.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:24:18.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:24:18.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 -- 172.21.15.50:0/1709557971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258009a70 con 0x7f525c135ec0 2023-10-30T12:24:18.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f526148e640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f525c06ff70 0x7f525c1ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:24:18.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5260c8d640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f525c135ae0 0x7f525c096340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:24:18.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5261c8f640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ec0 0x7f525c096880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5254001eb0 tx=0x7f525400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:24:18.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 client.?.objecter ms_handle_connect 0x7f525c135ec0 2023-10-30T12:24:18.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 client.?.objecter resend_mon_ops 2023-10-30T12:24:18.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 1 -- 172.21.15.50:0/1709557971 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52540052d0 con 0x7f525c135ec0 2023-10-30T12:24:18.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:18.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:24:18.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: dump: 2023-10-30T12:24:18.714 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:18.715 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.715 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.715 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.715 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:18.716 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.716 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:18.716 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:18.716 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:18.717 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: _finish_auth 0 2023-10-30T12:24:18.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: _check_auth_tickets 2023-10-30T12:24:18.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:24:18.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f52636f2640 5 monclient: authenticate success, global_id 284219 2023-10-30T12:24:18.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 1 -- 172.21.15.50:0/1709557971 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5254005430 con 0x7f525c135ec0 2023-10-30T12:24:18.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:24:18.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 1 -- 172.21.15.50:0/1709557971 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254003810 con 0x7f525c135ec0 2023-10-30T12:24:18.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f52636f2640 10 monclient: _renew_subs 2023-10-30T12:24:18.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f52636f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:24:18.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524c009460 con 0x7f525c135ec0 2023-10-30T12:24:18.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f52537fe640 4 set_mon_vals no callback set 2023-10-30T12:24:18.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:18.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:24:18.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.641+0000 7f5251ffb640 10 monclient: dump: 2023-10-30T12:24:18.721 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:18.721 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.721 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.721 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.722 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:18.722 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.722 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:18.722 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:18.723 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:18.723 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 10 client.284219.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:24:18.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 10 monclient: _renew_subs 2023-10-30T12:24:18.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:24:18.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c094620 con 0x7f525c135ec0 2023-10-30T12:24:18.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 1 librados: init done 2023-10-30T12:24:18.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f5251ffb640 1 -- 172.21.15.50:0/1709557971 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f5254003970 con 0x7f525c135ec0 2023-10-30T12:24:18.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f5251ffb640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f52400392f0 0x7f524003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:24:18.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:24:18.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:24:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f5251ffb640 1 -- 172.21.15.50:0/1709557971 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7f5254049ee0 con 0x7f525c135ec0 2023-10-30T12:24:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f5251ffb640 10 client.284219.objecter ms_dispatch 0x7f525c000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:24:18.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f5251ffb640 3 client.284219.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:24:18.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f5251ffb640 3 client.284219.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:24:18.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.642+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c04e160 con 0x7f525c135ec0 2023-10-30T12:24:18.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f5251ffb640 20 client.284219.objecter dump_active .. 0 homeless 2023-10-30T12:24:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f526148e640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f52400392f0 0x7f524003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f526148e640 10 monclient: get_auth_request con 0x7f52400392f0 auth_method 0 2023-10-30T12:24:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f5251ffb640 10 client.284219.objecter ms_handle_connect 0x7f52400392f0 2023-10-30T12:24:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.643+0000 7f526148e640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f52400392f0 0x7f524003b7b0 secure :-1 s=READY pgs=4373 cs=0 l=1 rev1=1 crypto rx=0x7f524c034000 tx=0x7f524c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:24:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.648+0000 7f5251ffb640 1 -- 172.21.15.50:0/1709557971 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5254017020 con 0x7f525c135ec0 2023-10-30T12:24:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.648+0000 7f5251ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:24:18.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.648+0000 7f5251ffb640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:24:18.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.820+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 --> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f525c0983d0 con 0x7f52400392f0 2023-10-30T12:24:18.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.824+0000 7f5251ffb640 1 -- 172.21.15.50:0/1709557971 <== mgr.4102 v2:172.21.15.118:6824/1859404137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134567 (secure 0 0 0) 0x7f524c0043c0 con 0x7f52400392f0 2023-10-30T12:24:18.827 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-30T12:24:18.828 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-30T12:24:18.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 10 librados: watch_flush enter 2023-10-30T12:24:18.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 10 librados: watch_flush exit 2023-10-30T12:24:18.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 20 client.284219.objecter shutdown clearing up homeless session... 2023-10-30T12:24:18.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 10 client.284219.objecter successfully canceled tick 2023-10-30T12:24:18.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f52400392f0 msgr2=0x7f524003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f52400392f0 0x7f524003b7b0 secure :-1 s=READY pgs=4373 cs=0 l=1 rev1=1 crypto rx=0x7f524c034000 tx=0x7f524c008030 comp rx=0 tx=0).stop 2023-10-30T12:24:18.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 10 monclient: shutdown 2023-10-30T12:24:18.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:24:18.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ec0 msgr2=0x7f525c096880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ec0 0x7f525c096880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5254001eb0 tx=0x7f525400c890 comp rx=0 tx=0).stop 2023-10-30T12:24:18.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 shutdown_connections 2023-10-30T12:24:18.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7f52400392f0 0x7f524003b7b0 unknown :-1 s=CLOSED pgs=4373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f525c135ec0 0x7f525c096880 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f525c135ae0 0x7f525c096340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 --2- 172.21.15.50:0/1709557971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f525c06ff70 0x7f525c1ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.826+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 >> 172.21.15.50:0/1709557971 conn(0x7f525c0a05c0 msgr2=0x7f525c08afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:24:18.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.827+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 shutdown_connections 2023-10-30T12:24:18.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.827+0000 7f52636f2640 1 -- 172.21.15.50:0/1709557971 wait complete. 2023-10-30T12:24:18.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.827+0000 7f52636f2640 1 librados: shutdown 2023-10-30T12:24:18.838 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1385,"stamp":"2023-10-30T12:24:18.751638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":429704,"kb_used_data":10840,"kb_used_omap":9,"kb_used_meta":418806,"kb_avail":565801336,"statfs":{"total":579820584960,"available":579380568064,"internally_reserved":0,"allocated":11100160,"data_stored":3863898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":428858023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002470"},"pg_stats":[{"pgid":"15.13","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:35.097222+0000","last_change":"2023-10-30T12:23:35.097268+0000","last_active":"2023-10-30T12:23:35.097222+0000","last_peered":"2023-10-30T12:23:35.097222+0000","last_clean":"2023-10-30T12:23:35.097222+0000","last_became_active":"2023-10-30T12:01:59.223072+0000","last_became_peered":"2023-10-30T12:01:59.223072+0000","last_unstale":"2023-10-30T12:23:35.097222+0000","last_undegraded":"2023-10-30T12:23:35.097222+0000","last_fullsized":"2023-10-30T12:23:35.097222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:35.097131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:35.097131+0000","last_clean_scrub_stamp":"2023-10-30T12:23:35.097131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:42:45.018845+0000","scrub_duration":0.0027366370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.679786+0000","last_change":"2023-10-30T12:23:22.679842+0000","last_active":"2023-10-30T12:23:22.679786+0000","last_peered":"2023-10-30T12:23:22.679786+0000","last_clean":"2023-10-30T12:23:22.679786+0000","last_became_active":"2023-10-30T12:01:59.344017+0000","last_became_peered":"2023-10-30T12:01:59.344017+0000","last_unstale":"2023-10-30T12:23:22.679786+0000","last_undegraded":"2023-10-30T12:23:22.679786+0000","last_fullsized":"2023-10-30T12:23:22.679786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.679707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.679707+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.679707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:10.004012+0000","scrub_duration":0.0020254040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:27.841929+0000","last_change":"2023-10-30T12:23:27.842001+0000","last_active":"2023-10-30T12:23:27.841929+0000","last_peered":"2023-10-30T12:23:27.841929+0000","last_clean":"2023-10-30T12:23:27.841929+0000","last_became_active":"2023-10-30T12:01:59.308994+0000","last_became_peered":"2023-10-30T12:01:59.308994+0000","last_unstale":"2023-10-30T12:23:27.841929+0000","last_undegraded":"2023-10-30T12:23:27.841929+0000","last_fullsized":"2023-10-30T12:23:27.841929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:27.841846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:27.841846+0000","last_clean_scrub_stamp":"2023-10-30T12:23:27.841846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:25.854306+0000","scrub_duration":0.0022774079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.051009+0000","last_change":"2023-10-30T12:23:22.051049+0000","last_active":"2023-10-30T12:23:22.051009+0000","last_peered":"2023-10-30T12:23:22.051009+0000","last_clean":"2023-10-30T12:23:22.051009+0000","last_became_active":"2023-10-30T12:01:59.277217+0000","last_became_peered":"2023-10-30T12:01:59.277217+0000","last_unstale":"2023-10-30T12:23:22.051009+0000","last_undegraded":"2023-10-30T12:23:22.051009+0000","last_fullsized":"2023-10-30T12:23:22.051009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.050938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.050938+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.050938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:06.125027+0000","scrub_duration":0.0012565440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:26.058042+0000","last_change":"2023-10-30T12:23:26.058042+0000","last_active":"2023-10-30T12:23:26.058042+0000","last_peered":"2023-10-30T12:23:26.058042+0000","last_clean":"2023-10-30T12:23:26.058042+0000","last_became_active":"2023-10-30T12:01:59.246101+0000","last_became_peered":"2023-10-30T12:01:59.246101+0000","last_unstale":"2023-10-30T12:23:26.058042+0000","last_undegraded":"2023-10-30T12:23:26.058042+0000","last_fullsized":"2023-10-30T12:23:26.058042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:26.057897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:26.057897+0000","last_clean_scrub_stamp":"2023-10-30T12:23:26.057897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:25.272602+0000","scrub_duration":0.0028676180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:16.731316+0000","last_change":"2023-10-30T12:23:16.731316+0000","last_active":"2023-10-30T12:23:16.731316+0000","last_peered":"2023-10-30T12:23:16.731316+0000","last_clean":"2023-10-30T12:23:16.731316+0000","last_became_active":"2023-10-30T12:01:59.269001+0000","last_became_peered":"2023-10-30T12:01:59.269001+0000","last_unstale":"2023-10-30T12:23:16.731316+0000","last_undegraded":"2023-10-30T12:23:16.731316+0000","last_fullsized":"2023-10-30T12:23:16.731316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:16.731211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:16.731211+0000","last_clean_scrub_stamp":"2023-10-30T12:23:16.731211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:54:46.097638+0000","scrub_duration":0.001926624,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:24.294957+0000","last_change":"2023-10-30T12:23:24.294957+0000","last_active":"2023-10-30T12:23:24.294957+0000","last_peered":"2023-10-30T12:23:24.294957+0000","last_clean":"2023-10-30T12:23:24.294957+0000","last_became_active":"2023-10-30T12:01:59.314812+0000","last_became_peered":"2023-10-30T12:01:59.314812+0000","last_unstale":"2023-10-30T12:23:24.294957+0000","last_undegraded":"2023-10-30T12:23:24.294957+0000","last_fullsized":"2023-10-30T12:23:24.294957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:24.294737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:24.294737+0000","last_clean_scrub_stamp":"2023-10-30T12:23:24.294737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:52.025084+0000","scrub_duration":0.0040212520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":673,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:28.583622+0000","last_change":"2023-10-30T12:23:28.583669+0000","last_active":"2023-10-30T12:23:28.583622+0000","last_peered":"2023-10-30T12:23:28.583622+0000","last_clean":"2023-10-30T12:23:28.583622+0000","last_became_active":"2023-10-30T12:01:59.338820+0000","last_became_peered":"2023-10-30T12:01:59.338820+0000","last_unstale":"2023-10-30T12:23:28.583622+0000","last_undegraded":"2023-10-30T12:23:28.583622+0000","last_fullsized":"2023-10-30T12:23:28.583622+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:28.583537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:28.583537+0000","last_clean_scrub_stamp":"2023-10-30T12:23:28.583537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:58.563507+0000","scrub_duration":0.0020370190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":672,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:25.572225+0000","last_change":"2023-10-30T12:23:25.572281+0000","last_active":"2023-10-30T12:23:25.572225+0000","last_peered":"2023-10-30T12:23:25.572225+0000","last_clean":"2023-10-30T12:23:25.572225+0000","last_became_active":"2023-10-30T12:01:59.296246+0000","last_became_peered":"2023-10-30T12:01:59.296246+0000","last_unstale":"2023-10-30T12:23:25.572225+0000","last_undegraded":"2023-10-30T12:23:25.572225+0000","last_fullsized":"2023-10-30T12:23:25.572225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:25.572139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:25.572139+0000","last_clean_scrub_stamp":"2023-10-30T12:23:25.572139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:30:05.572263+0000","scrub_duration":0.003219356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:23.339792+0000","last_change":"2023-10-30T12:23:23.339848+0000","last_active":"2023-10-30T12:23:23.339792+0000","last_peered":"2023-10-30T12:23:23.339792+0000","last_clean":"2023-10-30T12:23:23.339792+0000","last_became_active":"2023-10-30T12:01:59.296511+0000","last_became_peered":"2023-10-30T12:01:59.296511+0000","last_unstale":"2023-10-30T12:23:23.339792+0000","last_undegraded":"2023-10-30T12:23:23.339792+0000","last_fullsized":"2023-10-30T12:23:23.339792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:23.339704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:23.339704+0000","last_clean_scrub_stamp":"2023-10-30T12:23:23.339704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:23.941804+0000","scrub_duration":0.001707178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:21.036694+0000","last_change":"2023-10-30T12:23:21.036694+0000","last_active":"2023-10-30T12:23:21.036694+0000","last_peered":"2023-10-30T12:23:21.036694+0000","last_clean":"2023-10-30T12:23:21.036694+0000","last_became_active":"2023-10-30T12:01:59.282069+0000","last_became_peered":"2023-10-30T12:01:59.282069+0000","last_unstale":"2023-10-30T12:23:21.036694+0000","last_undegraded":"2023-10-30T12:23:21.036694+0000","last_fullsized":"2023-10-30T12:23:21.036694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:21.036559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:21.036559+0000","last_clean_scrub_stamp":"2023-10-30T12:23:21.036559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:12:55.972365+0000","scrub_duration":0.0022463069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":652,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.663318+0000","last_change":"2023-10-30T12:23:19.663318+0000","last_active":"2023-10-30T12:23:19.663318+0000","last_peered":"2023-10-30T12:23:19.663318+0000","last_clean":"2023-10-30T12:23:19.663318+0000","last_became_active":"2023-10-30T12:01:59.266725+0000","last_became_peered":"2023-10-30T12:01:59.266725+0000","last_unstale":"2023-10-30T12:23:19.663318+0000","last_undegraded":"2023-10-30T12:23:19.663318+0000","last_fullsized":"2023-10-30T12:23:19.663318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.663197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.663197+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.663197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:06.339497+0000","scrub_duration":0.001409296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:23.544975+0000","last_change":"2023-10-30T12:23:23.545019+0000","last_active":"2023-10-30T12:23:23.544975+0000","last_peered":"2023-10-30T12:23:23.544975+0000","last_clean":"2023-10-30T12:23:23.544975+0000","last_became_active":"2023-10-30T12:00:53.075649+0000","last_became_peered":"2023-10-30T12:00:53.075649+0000","last_unstale":"2023-10-30T12:23:23.544975+0000","last_undegraded":"2023-10-30T12:23:23.544975+0000","last_fullsized":"2023-10-30T12:23:23.544975+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:23.544892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:23.544892+0000","last_clean_scrub_stamp":"2023-10-30T12:23:23.544892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:12:55.039855+0000","scrub_duration":0.0014902439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:22.866657+0000","last_change":"2023-10-30T12:23:22.866657+0000","last_active":"2023-10-30T12:23:22.866657+0000","last_peered":"2023-10-30T12:23:22.866657+0000","last_clean":"2023-10-30T12:23:22.866657+0000","last_became_active":"2023-10-30T12:01:59.274444+0000","last_became_peered":"2023-10-30T12:01:59.274444+0000","last_unstale":"2023-10-30T12:23:22.866657+0000","last_undegraded":"2023-10-30T12:23:22.866657+0000","last_fullsized":"2023-10-30T12:23:22.866657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:22.866525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:22.866525+0000","last_clean_scrub_stamp":"2023-10-30T12:23:22.866525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:36:46.228780+0000","scrub_duration":0.001319738,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":663,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:28.133407+0000","last_change":"2023-10-30T12:23:28.133453+0000","last_active":"2023-10-30T12:23:28.133407+0000","last_peered":"2023-10-30T12:23:28.133407+0000","last_clean":"2023-10-30T12:23:28.133407+0000","last_became_active":"2023-10-30T12:00:52.480330+0000","last_became_peered":"2023-10-30T12:00:52.480330+0000","last_unstale":"2023-10-30T12:23:28.133407+0000","last_undegraded":"2023-10-30T12:23:28.133407+0000","last_fullsized":"2023-10-30T12:23:28.133407+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:28.133331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:28.133331+0000","last_clean_scrub_stamp":"2023-10-30T12:23:28.133331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:03.511823+0000","scrub_duration":0.0019102469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:32.895617+0000","last_change":"2023-10-30T12:23:32.895661+0000","last_active":"2023-10-30T12:23:32.895617+0000","last_peered":"2023-10-30T12:23:32.895617+0000","last_clean":"2023-10-30T12:23:32.895617+0000","last_became_active":"2023-10-30T12:01:59.272125+0000","last_became_peered":"2023-10-30T12:01:59.272125+0000","last_unstale":"2023-10-30T12:23:32.895617+0000","last_undegraded":"2023-10-30T12:23:32.895617+0000","last_fullsized":"2023-10-30T12:23:32.895617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:32.895528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:32.895528+0000","last_clean_scrub_stamp":"2023-10-30T12:23:32.895528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:25:50.392514+0000","scrub_duration":0.0030611589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":678,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:20.556804+0000","last_change":"2023-10-30T12:23:20.556804+0000","last_active":"2023-10-30T12:23:20.556804+0000","last_peered":"2023-10-30T12:23:20.556804+0000","last_clean":"2023-10-30T12:23:20.556804+0000","last_became_active":"2023-10-30T12:00:52.476594+0000","last_became_peered":"2023-10-30T12:00:52.476594+0000","last_unstale":"2023-10-30T12:23:20.556804+0000","last_undegraded":"2023-10-30T12:23:20.556804+0000","last_fullsized":"2023-10-30T12:23:20.556804+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:20.556644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:20.556644+0000","last_clean_scrub_stamp":"2023-10-30T12:23:20.556644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:07.027218+0000","scrub_duration":0.0020766999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":662,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:58.799412+0000","last_change":"2023-10-30T12:23:58.799449+0000","last_active":"2023-10-30T12:23:58.799412+0000","last_peered":"2023-10-30T12:23:58.799412+0000","last_clean":"2023-10-30T12:23:58.799412+0000","last_became_active":"2023-10-30T12:01:59.269127+0000","last_became_peered":"2023-10-30T12:01:59.269127+0000","last_unstale":"2023-10-30T12:23:58.799412+0000","last_undegraded":"2023-10-30T12:23:58.799412+0000","last_fullsized":"2023-10-30T12:23:58.799412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:58.799341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:58.799341+0000","last_clean_scrub_stamp":"2023-10-30T12:23:58.799341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:53:48.914562+0000","scrub_duration":0.0011465900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":664,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:20.231029+0000","last_change":"2023-10-30T12:23:20.231029+0000","last_active":"2023-10-30T12:23:20.231029+0000","last_peered":"2023-10-30T12:23:20.231029+0000","last_clean":"2023-10-30T12:23:20.231029+0000","last_became_active":"2023-10-30T12:00:52.479619+0000","last_became_peered":"2023-10-30T12:00:52.479619+0000","last_unstale":"2023-10-30T12:23:20.231029+0000","last_undegraded":"2023-10-30T12:23:20.231029+0000","last_fullsized":"2023-10-30T12:23:20.231029+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:20.230896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:20.230896+0000","last_clean_scrub_stamp":"2023-10-30T12:23:20.230896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:39.228622+0000","scrub_duration":0.00164442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":654,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:36.180099+0000","last_change":"2023-10-30T12:23:36.180099+0000","last_active":"2023-10-30T12:23:36.180099+0000","last_peered":"2023-10-30T12:23:36.180099+0000","last_clean":"2023-10-30T12:23:36.180099+0000","last_became_active":"2023-10-30T12:01:59.306293+0000","last_became_peered":"2023-10-30T12:01:59.306293+0000","last_unstale":"2023-10-30T12:23:36.180099+0000","last_undegraded":"2023-10-30T12:23:36.180099+0000","last_fullsized":"2023-10-30T12:23:36.180099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:36.179942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:36.179942+0000","last_clean_scrub_stamp":"2023-10-30T12:23:36.179942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:12.386250+0000","scrub_duration":0.002831882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":658,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.383016+0000","last_change":"2023-10-30T12:23:19.383016+0000","last_active":"2023-10-30T12:23:19.383016+0000","last_peered":"2023-10-30T12:23:19.383016+0000","last_clean":"2023-10-30T12:23:19.383016+0000","last_became_active":"2023-10-30T12:00:52.480392+0000","last_became_peered":"2023-10-30T12:00:52.480392+0000","last_unstale":"2023-10-30T12:23:19.383016+0000","last_undegraded":"2023-10-30T12:23:19.383016+0000","last_fullsized":"2023-10-30T12:23:19.383016+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.382889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.382889+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.382889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:44:05.674822+0000","scrub_duration":0.001638839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":673,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:32.492426+0000","last_change":"2023-10-30T12:23:32.492472+0000","last_active":"2023-10-30T12:23:32.492426+0000","last_peered":"2023-10-30T12:23:32.492426+0000","last_clean":"2023-10-30T12:23:32.492426+0000","last_became_active":"2023-10-30T12:01:59.313414+0000","last_became_peered":"2023-10-30T12:01:59.313414+0000","last_unstale":"2023-10-30T12:23:32.492426+0000","last_undegraded":"2023-10-30T12:23:32.492426+0000","last_fullsized":"2023-10-30T12:23:32.492426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:32.492334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:32.492334+0000","last_clean_scrub_stamp":"2023-10-30T12:23:32.492334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:32:37.769000+0000","scrub_duration":0.0032407759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":666,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.873251+0000","last_change":"2023-10-30T12:23:18.873251+0000","last_active":"2023-10-30T12:23:18.873251+0000","last_peered":"2023-10-30T12:23:18.873251+0000","last_clean":"2023-10-30T12:23:18.873251+0000","last_became_active":"2023-10-30T12:00:52.480096+0000","last_became_peered":"2023-10-30T12:00:52.480096+0000","last_unstale":"2023-10-30T12:23:18.873251+0000","last_undegraded":"2023-10-30T12:23:18.873251+0000","last_fullsized":"2023-10-30T12:23:18.873251+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.873126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.873126+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.873126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:12.606571+0000","scrub_duration":0.0015593950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.250540+0000","last_change":"2023-10-30T12:23:29.250608+0000","last_active":"2023-10-30T12:23:29.250540+0000","last_peered":"2023-10-30T12:23:29.250540+0000","last_clean":"2023-10-30T12:23:29.250540+0000","last_became_active":"2023-10-30T12:01:59.289175+0000","last_became_peered":"2023-10-30T12:01:59.289175+0000","last_unstale":"2023-10-30T12:23:29.250540+0000","last_undegraded":"2023-10-30T12:23:29.250540+0000","last_fullsized":"2023-10-30T12:23:29.250540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:29.250439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:29.250439+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.250439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:10.722207+0000","scrub_duration":0.0013662959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":677,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:21.542207+0000","last_change":"2023-10-30T12:23:21.542272+0000","last_active":"2023-10-30T12:23:21.542207+0000","last_peered":"2023-10-30T12:23:21.542207+0000","last_clean":"2023-10-30T12:23:21.542207+0000","last_became_active":"2023-10-30T12:00:52.479548+0000","last_became_peered":"2023-10-30T12:00:52.479548+0000","last_unstale":"2023-10-30T12:23:21.542207+0000","last_undegraded":"2023-10-30T12:23:21.542207+0000","last_fullsized":"2023-10-30T12:23:21.542207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:21.542130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:21.542130+0000","last_clean_scrub_stamp":"2023-10-30T12:23:21.542130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:12.658987+0000","scrub_duration":0.0020392639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:30.095199+0000","last_change":"2023-10-30T12:23:30.095249+0000","last_active":"2023-10-30T12:23:30.095199+0000","last_peered":"2023-10-30T12:23:30.095199+0000","last_clean":"2023-10-30T12:23:30.095199+0000","last_became_active":"2023-10-30T12:01:59.302155+0000","last_became_peered":"2023-10-30T12:01:59.302155+0000","last_unstale":"2023-10-30T12:23:30.095199+0000","last_undegraded":"2023-10-30T12:23:30.095199+0000","last_fullsized":"2023-10-30T12:23:30.095199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:30.095111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:30.095111+0000","last_clean_scrub_stamp":"2023-10-30T12:23:30.095111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:02.227834+0000","scrub_duration":0.0025979240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":658,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.413538+0000","last_change":"2023-10-30T12:23:18.413538+0000","last_active":"2023-10-30T12:23:18.413538+0000","last_peered":"2023-10-30T12:23:18.413538+0000","last_clean":"2023-10-30T12:23:18.413538+0000","last_became_active":"2023-10-30T12:00:52.478834+0000","last_became_peered":"2023-10-30T12:00:52.478834+0000","last_unstale":"2023-10-30T12:23:18.413538+0000","last_undegraded":"2023-10-30T12:23:18.413538+0000","last_fullsized":"2023-10-30T12:23:18.413538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.413381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.413381+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.413381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:15.717044+0000","scrub_duration":0.00185736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"686'119","reported_seq":831,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.536942+0000","last_change":"2023-10-30T12:23:19.537004+0000","last_active":"2023-10-30T12:23:19.536942+0000","last_peered":"2023-10-30T12:23:19.536942+0000","last_clean":"2023-10-30T12:23:19.536942+0000","last_became_active":"2023-10-30T12:00:50.417951+0000","last_became_peered":"2023-10-30T12:00:50.417951+0000","last_unstale":"2023-10-30T12:23:19.536942+0000","last_undegraded":"2023-10-30T12:23:19.536942+0000","last_fullsized":"2023-10-30T12:23:19.536942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'119","last_scrub_stamp":"2023-10-30T12:23:19.536853+0000","last_deep_scrub":"686'119","last_deep_scrub_stamp":"2023-10-30T12:23:19.536853+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.536853+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:12.041107+0000","scrub_duration":0.0043842680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:44.858009+0000","last_change":"2023-10-30T12:23:44.858062+0000","last_active":"2023-10-30T12:23:44.858009+0000","last_peered":"2023-10-30T12:23:44.858009+0000","last_clean":"2023-10-30T12:23:44.858009+0000","last_became_active":"2023-10-30T12:01:59.253931+0000","last_became_peered":"2023-10-30T12:01:59.253931+0000","last_unstale":"2023-10-30T12:23:44.858009+0000","last_undegraded":"2023-10-30T12:23:44.858009+0000","last_fullsized":"2023-10-30T12:23:44.858009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:44.857928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:44.857928+0000","last_clean_scrub_stamp":"2023-10-30T12:23:44.857928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:08.641471+0000","scrub_duration":0.0016835999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:40.071963+0000","last_change":"2023-10-30T12:23:40.072011+0000","last_active":"2023-10-30T12:23:40.071963+0000","last_peered":"2023-10-30T12:23:40.071963+0000","last_clean":"2023-10-30T12:23:40.071963+0000","last_became_active":"2023-10-30T12:01:59.277183+0000","last_became_peered":"2023-10-30T12:01:59.277183+0000","last_unstale":"2023-10-30T12:23:40.071963+0000","last_undegraded":"2023-10-30T12:23:40.071963+0000","last_fullsized":"2023-10-30T12:23:40.071963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:40.071880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:40.071880+0000","last_clean_scrub_stamp":"2023-10-30T12:23:40.071880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:48.102089+0000","scrub_duration":0.001972892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":674,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:27.569483+0000","last_change":"2023-10-30T12:23:27.569531+0000","last_active":"2023-10-30T12:23:27.569483+0000","last_peered":"2023-10-30T12:23:27.569483+0000","last_clean":"2023-10-30T12:23:27.569483+0000","last_became_active":"2023-10-30T12:01:59.339614+0000","last_became_peered":"2023-10-30T12:01:59.339614+0000","last_unstale":"2023-10-30T12:23:27.569483+0000","last_undegraded":"2023-10-30T12:23:27.569483+0000","last_fullsized":"2023-10-30T12:23:27.569483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:27.569393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:27.569393+0000","last_clean_scrub_stamp":"2023-10-30T12:23:27.569393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:10:08.233929+0000","scrub_duration":0.002727185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":653,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:18.692837+0000","last_change":"2023-10-30T12:23:18.692837+0000","last_active":"2023-10-30T12:23:18.692837+0000","last_peered":"2023-10-30T12:23:18.692837+0000","last_clean":"2023-10-30T12:23:18.692837+0000","last_became_active":"2023-10-30T12:01:59.303216+0000","last_became_peered":"2023-10-30T12:01:59.303216+0000","last_unstale":"2023-10-30T12:23:18.692837+0000","last_undegraded":"2023-10-30T12:23:18.692837+0000","last_fullsized":"2023-10-30T12:23:18.692837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:18.692709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:18.692709+0000","last_clean_scrub_stamp":"2023-10-30T12:23:18.692709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:02.768827+0000","scrub_duration":0.002027278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.106812+0000","last_change":"2023-10-30T12:23:29.106812+0000","last_active":"2023-10-30T12:23:29.106812+0000","last_peered":"2023-10-30T12:23:29.106812+0000","last_clean":"2023-10-30T12:23:29.106812+0000","last_became_active":"2023-10-30T12:01:59.209551+0000","last_became_peered":"2023-10-30T12:01:59.209551+0000","last_unstale":"2023-10-30T12:23:29.106812+0000","last_undegraded":"2023-10-30T12:23:29.106812+0000","last_fullsized":"2023-10-30T12:23:29.106812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:29.106681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:29.106681+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.106681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:30:50.945615+0000","scrub_duration":0.0032893250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":663,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:47.873611+0000","last_change":"2023-10-30T12:23:47.873611+0000","last_active":"2023-10-30T12:23:47.873611+0000","last_peered":"2023-10-30T12:23:47.873611+0000","last_clean":"2023-10-30T12:23:47.873611+0000","last_became_active":"2023-10-30T12:01:59.310341+0000","last_became_peered":"2023-10-30T12:01:59.310341+0000","last_unstale":"2023-10-30T12:23:47.873611+0000","last_undegraded":"2023-10-30T12:23:47.873611+0000","last_fullsized":"2023-10-30T12:23:47.873611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:47.873470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:47.873470+0000","last_clean_scrub_stamp":"2023-10-30T12:23:47.873470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:48.784611+0000","scrub_duration":0.0019736110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1","version":"686'8","reported_seq":688,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:29.543719+0000","last_change":"2023-10-30T12:23:29.543806+0000","last_active":"2023-10-30T12:23:29.543719+0000","last_peered":"2023-10-30T12:23:29.543719+0000","last_clean":"2023-10-30T12:23:29.543719+0000","last_became_active":"2023-10-30T12:01:59.260849+0000","last_became_peered":"2023-10-30T12:01:59.260849+0000","last_unstale":"2023-10-30T12:23:29.543719+0000","last_undegraded":"2023-10-30T12:23:29.543719+0000","last_fullsized":"2023-10-30T12:23:29.543719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'8","last_scrub_stamp":"2023-10-30T12:23:29.543628+0000","last_deep_scrub":"686'8","last_deep_scrub_stamp":"2023-10-30T12:23:29.543628+0000","last_clean_scrub_stamp":"2023-10-30T12:23:29.543628+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:22:42.277331+0000","scrub_duration":0.0034138010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":668,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:24:02.730409+0000","last_change":"2023-10-30T12:24:02.730460+0000","last_active":"2023-10-30T12:24:02.730409+0000","last_peered":"2023-10-30T12:24:02.730409+0000","last_clean":"2023-10-30T12:24:02.730409+0000","last_became_active":"2023-10-30T12:08:57.463102+0000","last_became_peered":"2023-10-30T12:08:57.463102+0000","last_unstale":"2023-10-30T12:24:02.730409+0000","last_undegraded":"2023-10-30T12:24:02.730409+0000","last_fullsized":"2023-10-30T12:24:02.730409+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:24:02.730318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:24:02.730318+0000","last_clean_scrub_stamp":"2023-10-30T12:24:02.730318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:50:02.145135+0000","scrub_duration":0.001389326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":661,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:19.082953+0000","last_change":"2023-10-30T12:23:19.082999+0000","last_active":"2023-10-30T12:23:19.082953+0000","last_peered":"2023-10-30T12:23:19.082953+0000","last_clean":"2023-10-30T12:23:19.082953+0000","last_became_active":"2023-10-30T12:01:59.246868+0000","last_became_peered":"2023-10-30T12:01:59.246868+0000","last_unstale":"2023-10-30T12:23:19.082953+0000","last_undegraded":"2023-10-30T12:23:19.082953+0000","last_fullsized":"2023-10-30T12:23:19.082953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:19.082868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:19.082868+0000","last_clean_scrub_stamp":"2023-10-30T12:23:19.082868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:50.129280+0000","scrub_duration":0.0020950420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":658,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:31.113426+0000","last_change":"2023-10-30T12:23:31.113469+0000","last_active":"2023-10-30T12:23:31.113426+0000","last_peered":"2023-10-30T12:23:31.113426+0000","last_clean":"2023-10-30T12:23:31.113426+0000","last_became_active":"2023-10-30T12:01:59.239117+0000","last_became_peered":"2023-10-30T12:01:59.239117+0000","last_unstale":"2023-10-30T12:23:31.113426+0000","last_undegraded":"2023-10-30T12:23:31.113426+0000","last_fullsized":"2023-10-30T12:23:31.113426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:31.113340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:31.113340+0000","last_clean_scrub_stamp":"2023-10-30T12:23:31.113340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:02:44.961562+0000","scrub_duration":0.0016503100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":659,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:31.058690+0000","last_change":"2023-10-30T12:23:31.058765+0000","last_active":"2023-10-30T12:23:31.058690+0000","last_peered":"2023-10-30T12:23:31.058690+0000","last_clean":"2023-10-30T12:23:31.058690+0000","last_became_active":"2023-10-30T12:01:59.293055+0000","last_became_peered":"2023-10-30T12:01:59.293055+0000","last_unstale":"2023-10-30T12:23:31.058690+0000","last_undegraded":"2023-10-30T12:23:31.058690+0000","last_fullsized":"2023-10-30T12:23:31.058690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:31.058583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:31.058583+0000","last_clean_scrub_stamp":"2023-10-30T12:23:31.058583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:48.548044+0000","scrub_duration":0.002423377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":662,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:42.894694+0000","last_change":"2023-10-30T12:23:42.894694+0000","last_active":"2023-10-30T12:23:42.894694+0000","last_peered":"2023-10-30T12:23:42.894694+0000","last_clean":"2023-10-30T12:23:42.894694+0000","last_became_active":"2023-10-30T12:01:59.307440+0000","last_became_peered":"2023-10-30T12:01:59.307440+0000","last_unstale":"2023-10-30T12:23:42.894694+0000","last_undegraded":"2023-10-30T12:23:42.894694+0000","last_fullsized":"2023-10-30T12:23:42.894694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:42.894562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:42.894562+0000","last_clean_scrub_stamp":"2023-10-30T12:23:42.894562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:54:50.034244+0000","scrub_duration":0.001821285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":660,"reported_epoch":686,"state":"active+clean","last_fresh":"2023-10-30T12:23:30.140984+0000","last_change":"2023-10-30T12:23:30.141028+0000","last_active":"2023-10-30T12:23:30.140984+0000","last_peered":"2023-10-30T12:23:30.140984+0000","last_clean":"2023-10-30T12:23:30.140984+0000","last_became_active":"2023-10-30T12:01:59.239219+0000","last_became_peered":"2023-10-30T12:01:59.239219+0000","last_unstale":"2023-10-30T12:23:30.140984+0000","last_undegraded":"2023-10-30T12:23:30.140984+0000","last_fullsized":"2023-10-30T12:23:30.140984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:23:30.140907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:23:30.140907+0000","last_clean_scrub_stamp":"2023-10-30T12:23:30.140907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:52.739662+0000","scrub_duration":0.0037158099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640539,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77224,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294616,"statfs":{"total":96636764160,"available":96557686784,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":0.63700000000000001,"5min":0.71999999999999997,"15min":0.82699999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.35499999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.66800000000000004,"5min":0.71799999999999997,"15min":1.056},"last":0.57099999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.371,"5min":0.246,"15min":0.246},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":2.173},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.35199999999999998,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":0.70899999999999996,"15min":0.78100000000000003},"last":0.624}]},{"osd":2,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53300000000000003,"15min":0.54000000000000004},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.72699999999999998,"5min":0.74299999999999999,"15min":0.79900000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.48999999999999999,"5min":0.41499999999999998,"15min":0.34999999999999998},"max":{"1min":0.66200000000000003,"5min":0.74099999999999999,"15min":0.84699999999999998},"last":0.66300000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36599999999999999,"15min":0.372},"min":{"1min":0.26400000000000001,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":0.52100000000000002,"5min":0.58799999999999997,"15min":0.70399999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39900000000000002},"min":{"1min":0.28299999999999997,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58299999999999996,"5min":0.69699999999999995,"15min":0.95199999999999996},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.40799999999999997},"min":{"1min":0.30499999999999999,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":0.56799999999999995,"5min":0.63700000000000001,"15min":0.63800000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.42799999999999999,"15min":0.45100000000000001},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.55200000000000005,"5min":0.68400000000000005,"15min":0.72099999999999997},"last":0.432}]}]},{"osd":4,"up_from":11,"seq":47244640540,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81384,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94290456,"statfs":{"total":96636764160,"available":96553426944,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.41399999999999998,"5min":0.35099999999999998,"15min":0.24099999999999999},"max":{"1min":0.73399999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.371,"5min":0.30299999999999999,"15min":0.29799999999999999},"max":{"1min":0.70299999999999996,"5min":0.78700000000000003,"15min":0.84899999999999998},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.29299999999999998},"max":{"1min":0.65900000000000003,"5min":0.78000000000000003,"15min":0.94099999999999995},"last":0.627},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53300000000000003,"15min":0.53400000000000003},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79800000000000004},"last":0.67800000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.51500000000000001},"min":{"1min":0.40100000000000002,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.68100000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.40000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.27900000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53900000000000003,"5min":0.627,"15min":0.63600000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.374,"15min":0.38400000000000001},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.23799999999999999},"max":{"1min":0.60199999999999998,"5min":0.67000000000000004,"15min":0.77100000000000002},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.434,"15min":0.41399999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.25600000000000001},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.69999999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.442,"15min":0.45200000000000001},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":0.58599999999999997,"5min":0.67400000000000004,"15min":0.89700000000000002},"last":0.57599999999999996}]}]},{"osd":3,"up_from":11,"seq":47244640540,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69428,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302412,"statfs":{"total":96636764160,"available":96565669888,"internally_reserved":0,"allocated":2207744,"data_stored":996605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.501},"min":{"1min":0.39800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.69899999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.497},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.498,"15min":0.5},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.307},"max":{"1min":0.66800000000000004,"5min":0.70099999999999996,"15min":1.2490000000000001},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.55900000000000005},"min":{"1min":0.41199999999999998,"5min":0.33100000000000002,"15min":0.32000000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.2689999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58099999999999996,"15min":0.58999999999999997},"min":{"1min":0.44400000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":1.337},"last":0.376}]},{"osd":2,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.52000000000000002},"min":{"1min":0.377,"5min":0.314,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.88900000000000001,"15min":1.302},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53300000000000003,"15min":0.51300000000000001},"min":{"1min":0.38300000000000001,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.377},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55100000000000005,"5min":0.61199999999999999,"15min":1.371},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":1.444},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:23:31 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.25600000000000001},"max":{"1min":0.61899999999999999,"5min":0.64200000000000002,"15min":0.876},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.627,"5min":0.84099999999999997,"15min":1.3540000000000001},"last":0.38900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640539,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64748,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94307092,"statfs":{"total":96636764160,"available":96570462208,"internally_reserved":0,"allocated":1675264,"data_stored":467677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33900000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.192},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":2.2730000000000001},"last":0.314},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.22800000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":2.141},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20799999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":1.087},"last":0.376},{"interface":"front","average":{"1min":0.433,"5min":0.39400000000000002,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.27300000000000002,"15min":0.24399999999999999},"max":{"1min":0.59099999999999997,"5min":0.63900000000000001,"15min":3.8260000000000001},"last":0.443}]},{"osd":3,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":1.143},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.78800000000000003,"15min":3.4729999999999999},"last":0.52400000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.34399999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.3240000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.625,"15min":0.61599999999999999},"min":{"1min":0.46200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82099999999999995,"5min":0.85999999999999999,"15min":1.8340000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Oct 30 12:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61499999999999999,"15min":0.59399999999999997},"min":{"1min":0.44800000000000001,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":0.81599999999999995,"5min":0.81699999999999995,"15min":1.893},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.61499999999999999,"15min":0.60299999999999998},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.81599999999999995,"15min":2.2269999999999999},"last":0.65000000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640541,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59176,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312664,"statfs":{"total":96636764160,"available":96576167936,"internally_reserved":0,"allocated":1671168,"data_stored":467672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.35299999999999998,"15min":0.34000000000000002},"min":{"1min":0.22900000000000001,"5min":0.22700000000000001,"15min":0.20499999999999999},"max":{"1min":0.53900000000000003,"5min":0.745,"15min":0.745},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.435,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.378,"15min":0.376},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.497,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.312},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.52600000000000002,"5min":0.621,"15min":0.621},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.57199999999999995},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75600000000000001,"5min":0.86599999999999999,"15min":0.879},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.85799999999999998},"last":0.60899999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68300000000000005,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.754,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.625}]},{"osd":5,"last update":"Mon Oct 30 12:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60299999999999998,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.38900000000000001,"15min":0.38100000000000001},"max":{"1min":0.77900000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78600000000000003,"5min":1,"15min":1},"last":0.70199999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640541,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77744,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94294096,"statfs":{"total":96636764160,"available":96557154304,"internally_reserved":0,"allocated":2203648,"data_stored":996600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35399999999999998,"15min":0.35199999999999998},"min":{"1min":0.248,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":0.51100000000000001,"5min":1.865,"15min":1.865},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42199999999999999,"15min":0.41399999999999998},"min":{"1min":0.29599999999999999,"5min":0.27900000000000003,"15min":0.23899999999999999},"max":{"1min":0.55600000000000005,"5min":3.1440000000000001,"15min":3.1440000000000001},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38500000000000001},"min":{"1min":0.26300000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51000000000000001,"5min":1.802,"15min":3.8519999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.38600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20100000000000001},"max":{"1min":0.57099999999999995,"5min":0.623,"15min":2.0590000000000002},"last":0.27900000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.57499999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.27800000000000002},"max":{"1min":0.83999999999999997,"5min":0.95999999999999996,"15min":1.5489999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.35399999999999998,"15min":0.28199999999999997},"max":{"1min":0.74099999999999999,"5min":0.875,"15min":1.78},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.624,"15min":0.61599999999999999},"min":{"1min":0.41899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79800000000000004,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.627},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.622,"15min":0.61699999999999999},"min":{"1min":0.45100000000000001,"5min":0.42299999999999999,"15min":0.29399999999999998},"max":{"1min":0.81399999999999995,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Mon Oct 30 12:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.33900000000000002},"max":{"1min":0.79200000000000004,"5min":0.88800000000000001,"15min":1.6020000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":0.75800000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:24:18.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T12:24:18.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.946+0000 7fb1d907c640 10 monclient: get_monmap_and_config 2023-10-30T12:24:18.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.946+0000 7fb1d907c640 10 monclient: build_initial_monmap 2023-10-30T12:24:18.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.946+0000 7fb1d907c640 10 monclient: monmap: 2023-10-30T12:24:18.949 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:24:18.949 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.949 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:24:18.947343+0000 2023-10-30T12:24:18.950 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:24:18.947343+0000 2023-10-30T12:24:18.950 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:24:18.950 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.950 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a-legacy 2023-10-30T12:24:18.951 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.118:6789/0 mon.noname-c-legacy 2023-10-30T12:24:18.951 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-b 2023-10-30T12:24:18.951 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.50:3300/0 mon.noname-a 2023-10-30T12:24:18.952 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.118:3300/0 mon.noname-c 2023-10-30T12:24:18.952 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.946+0000 7fb1d907c640 1 Processor -- start 2023-10-30T12:24:18.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.946+0000 7fb1d907c640 1 -- start start 2023-10-30T12:24:18.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.946+0000 7fb1d907c640 10 monclient: init 2023-10-30T12:24:18.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient: _reopen_session rank -1 2023-10-30T12:24:18.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-10-30T12:24:18.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d41327f0 0x7fb1d4132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d41327f0 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:24:18.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1d4099a40 addr v1:172.21.15.50:6789/0 2023-10-30T12:24:18.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb1d4135e10 0x7fb1d413da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:18.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d4135e10 addr v2:172.21.15.118:3300/0 2023-10-30T12:24:18.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d40a1620 con 0x7fb1d4099a40 2023-10-30T12:24:18.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:18.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d409a6b0 con 0x7fb1d4135e10 2023-10-30T12:24:18.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:18.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d40a1b20 con 0x7fb1d41327f0 2023-10-30T12:24:18.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient(hunting): _renew_subs 2023-10-30T12:24:18.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d907c640 10 monclient(hunting): authenticate will time out at 2304.130376s 2023-10-30T12:24:18.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d41327f0 0x7fb1d4132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d259c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb1d4099a40 0x7fb1d4135700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50174/0 (socket says 172.21.15.50:50174) 2023-10-30T12:24:18.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): get_auth_request con 0x7fb1d41327f0 auth_method 0 2023-10-30T12:24:18.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d259c640 1 -- 172.21.15.50:0/3747444353 learned_addr learned my addr 172.21.15.50:0/3747444353 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-10-30T12:24:18.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:18.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 1 --2- 172.21.15.50:0/3747444353 >> v2:172.21.15.118:3300/0 conn(0x7fb1d4135e10 0x7fb1d413da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:18.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 10 monclient(hunting): get_auth_request con 0x7fb1d4135e10 auth_method 0 2023-10-30T12:24:18.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:18.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:18.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:18.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:18.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:18.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:18.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:18.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d1d9b640 1 -- 172.21.15.50:0/3747444353 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74920921 0 0) 0x7fb1d40a1620 con 0x7fb1d4099a40 2023-10-30T12:24:18.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d1d9b640 10 monclient(hunting): _negotiate 2023-10-30T12:24:18.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d1d9b640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:18.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d1d9b640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:18.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d1d9b640 10 monclient(hunting): my global_id is 264486 2023-10-30T12:24:18.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d1d9b640 1 -- 172.21.15.50:0/3747444353 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c0003520 con 0x7fb1d4099a40 2023-10-30T12:24:18.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_done global_id 284222 payload 274 2023-10-30T12:24:18.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 1 -- 172.21.15.50:0/3747444353 >> v2:172.21.15.118:3300/0 conn(0x7fb1d4135e10 msgr2=0x7fb1d413da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:24:18.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 1 --2- 172.21.15.50:0/3747444353 >> v2:172.21.15.118:3300/0 conn(0x7fb1d4135e10 0x7fb1d413da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 1 -- 172.21.15.50:0/3747444353 >> v1:172.21.15.50:6789/0 conn(0x7fb1d4099a40 legacy=0x7fb1d4135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:24:18.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 1 monclient: found mon.noname-b 2023-10-30T12:24:18.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:24:18.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3301/0 2023-10-30T12:24:18.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.947+0000 7fb1d2d9d640 1 -- 172.21.15.50:0/3747444353 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4133110 con 0x7fb1d41327f0 2023-10-30T12:24:18.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d2d9d640 1 --2- 172.21.15.50:0/3747444353 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d41327f0 0x7fb1d4132bd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009d90 tx=0x7fb1c402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73b4571abade009 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:24:18.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 1 -- 172.21.15.50:0/3747444353 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c402cb30 con 0x7fb1d41327f0 2023-10-30T12:24:18.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:18.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-10-30T12:24:18.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: dump: 2023-10-30T12:24:18.977 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:18.978 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.978 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.979 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.979 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:18.979 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:18.980 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: _finish_auth 0 2023-10-30T12:24:18.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: _check_auth_tickets 2023-10-30T12:24:18.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:24:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 1 -- 172.21.15.50:0/3747444353 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c402cc90 con 0x7fb1d41327f0 2023-10-30T12:24:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d907c640 5 monclient: authenticate success, global_id 284222 2023-10-30T12:24:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d907c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-10-30T12:24:18.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:24:18.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 1 -- 172.21.15.50:0/3747444353 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4034440 con 0x7fb1d41327f0 2023-10-30T12:24:18.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:24:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d1d9b640 10 monclient: dump: 2023-10-30T12:24:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:18.987 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d907c640 10 monclient: get_monmap_and_config success 2023-10-30T12:24:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.948+0000 7fb1d907c640 4 set_mon_vals no callback set 2023-10-30T12:24:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 10 monclient: shutdown 2023-10-30T12:24:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:24:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 -- 172.21.15.50:0/3747444353 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d41327f0 msgr2=0x7fb1d4132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3747444353 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d41327f0 0x7fb1d4132bd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009d90 tx=0x7fb1c402bd80 comp rx=0 tx=0).stop 2023-10-30T12:24:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 -- 172.21.15.50:0/3747444353 shutdown_connections 2023-10-30T12:24:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3747444353 >> v2:172.21.15.118:3300/0 conn(0x7fb1d4135e10 0x7fb1d413da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3747444353 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d41327f0 0x7fb1d4132bd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:18.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 -- 172.21.15.50:0/3747444353 >> 172.21.15.50:0/3747444353 conn(0x7fb1d40a82f0 msgr2=0x7fb1d40a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:24:18.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 -- 172.21.15.50:0/3747444353 shutdown_connections 2023-10-30T12:24:18.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 1 -- 172.21.15.50:0/3747444353 wait complete. 2023-10-30T12:24:18.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.949+0000 7fb1d907c640 10 monclient: build_initial_monmap 2023-10-30T12:24:18.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient: monmap: 2023-10-30T12:24:18.991 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-10-30T12:24:18.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-10-30T12:24:18.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-10-30T12:24:18.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-10-30T12:24:18.993 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-10-30T12:24:18.993 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:18.994 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-a 2023-10-30T12:24:18.994 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.noname-b 2023-10-30T12:24:18.995 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.noname-c 2023-10-30T12:24:18.995 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:18.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 librados: starting msgr at 2023-10-30T12:24:18.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 librados: starting objecter 2023-10-30T12:24:18.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 Processor -- start 2023-10-30T12:24:18.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 -- start start 2023-10-30T12:24:18.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 librados: setting wanted keys 2023-10-30T12:24:18.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 librados: calling monclient init 2023-10-30T12:24:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient: init 2023-10-30T12:24:18.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient: _reopen_session rank -1 2023-10-30T12:24:19.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient: _add_conns ranks=[2,0,1] 2023-10-30T12:24:19.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d4099a40 0x7fb1d41ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:19.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d4099a40 addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2023-10-30T12:24:19.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1d41327f0 0x7fb1d41ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:19.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d41327f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-10-30T12:24:19.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d4135e10 0x7fb1d411e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:19.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d4135e10 addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2023-10-30T12:24:19.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4122500 con 0x7fb1d41327f0 2023-10-30T12:24:19.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:19.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4122750 con 0x7fb1d4135e10 2023-10-30T12:24:19.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): start opening mon connection 2023-10-30T12:24:19.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4120fc0 con 0x7fb1d4099a40 2023-10-30T12:24:19.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): _renew_subs 2023-10-30T12:24:19.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.950+0000 7fb1d907c640 10 monclient(hunting): authenticate will time out at 2304.133997s 2023-10-30T12:24:19.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d4099a40 0x7fb1d41ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:19.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d4099a40 0x7fb1d41ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40732/0 (socket says 172.21.15.50:40732) 2023-10-30T12:24:19.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 -- 172.21.15.50:0/3054673082 learned_addr learned my addr 172.21.15.50:0/3054673082 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-10-30T12:24:19.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): get_auth_request con 0x7fb1d4099a40 auth_method 0 2023-10-30T12:24:19.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:19.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:19.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:19.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1d41327f0 0x7fb1d41ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:19.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 10 monclient(hunting): get_auth_request con 0x7fb1d41327f0 auth_method 0 2023-10-30T12:24:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d4135e10 0x7fb1d411e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 10 monclient(hunting): get_auth_request con 0x7fb1d4135e10 auth_method 0 2023-10-30T12:24:19.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-10-30T12:24:19.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 10 monclient(hunting): _init_auth method 2 2023-10-30T12:24:19.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 10 monclient(hunting): _init_auth creating new auth 2023-10-30T12:24:19.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:19.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:19.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:19.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:19.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-10-30T12:24:19.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-10-30T12:24:19.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-10-30T12:24:19.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient(hunting): handle_auth_done global_id 284225 payload 824 2023-10-30T12:24:19.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 -- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d4135e10 msgr2=0x7fb1d411e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:19.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d4135e10 0x7fb1d411e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:19.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 -- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1d41327f0 msgr2=0x7fb1d41ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:19.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1d41327f0 0x7fb1d41ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:19.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient: _finish_hunting 0 2023-10-30T12:24:19.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 monclient: found mon.noname-c 2023-10-30T12:24:19.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-10-30T12:24:19.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3301/0 2023-10-30T12:24:19.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 -- 172.21.15.50:0/3054673082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c4009a70 con 0x7fb1d4099a40 2023-10-30T12:24:19.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d259c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1d41327f0 0x7fb1d41ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:24:19.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d2d9d640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d4099a40 0x7fb1d41ca2d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4037040 tx=0x7fb1c4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:24:19.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb1d4099a40 2023-10-30T12:24:19.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 client.?.objecter resend_mon_ops 2023-10-30T12:24:19.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4034e50 con 0x7fb1d4099a40 2023-10-30T12:24:19.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:19.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-10-30T12:24:19.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: dump: 2023-10-30T12:24:19.027 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:19.027 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:19.028 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:19.028 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:19.028 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:19.028 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:19.029 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:19.029 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:19.029 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:19.029 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: _finish_auth 0 2023-10-30T12:24:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: _check_auth_tickets 2023-10-30T12:24:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-10-30T12:24:19.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c4040450 con 0x7fb1d4099a40 2023-10-30T12:24:19.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 5 monclient: authenticate success, global_id 284225 2023-10-30T12:24:19.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 10 monclient: _renew_subs 2023-10-30T12:24:19.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:24:19.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d40a1850 con 0x7fb1d4099a40 2023-10-30T12:24:19.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: handle_config config(0 keys) v1 2023-10-30T12:24:19.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4040800 con 0x7fb1d4099a40 2023-10-30T12:24:19.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-10-30T12:24:19.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-10-30T12:24:19.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1cb7fe640 10 monclient: dump: 2023-10-30T12:24:19.033 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-10-30T12:24:19.033 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 87b22564-9727-4955-ac7b-d9a79464690c 2023-10-30T12:24:19.034 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:19.034 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-10-30T12:00:17.894366+0000 2023-10-30T12:24:19.034 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 18 (reef) 2023-10-30T12:24:19.035 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-10-30T12:24:19.035 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-30T12:24:19.035 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2023-10-30T12:24:19.035 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2023-10-30T12:24:19.036 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:19.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 10 client.284225.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-10-30T12:24:19.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 10 monclient: _renew_subs 2023-10-30T12:24:19.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d159a640 4 set_mon_vals no callback set 2023-10-30T12:24:19.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:24:19.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d4120380 con 0x7fb1d4099a40 2023-10-30T12:24:19.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.951+0000 7fb1d907c640 1 librados: init done 2023-10-30T12:24:19.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.952+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 5) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fb1c4040960 con 0x7fb1d4099a40 2023-10-30T12:24:19.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.952+0000 7fb1d907c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-10-30T12:24:19.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.952+0000 7fb1d907c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:24:19.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.952+0000 7fb1d907c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:24:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.952+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d404e1d0 con 0x7fb1d4099a40 2023-10-30T12:24:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1cb7fe640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1b40392f0 0x7fb1b403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:24:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(686..686 src has 649..686) v4 ==== 5510+0+0 (secure 0 0 0) 0x7fb1c406d470 con 0x7fb1d4099a40 2023-10-30T12:24:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1cb7fe640 10 client.284225.objecter ms_dispatch 0x7fb1d4000b90 osd_map(686..686 src has 649..686) v4 2023-10-30T12:24:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1cb7fe640 3 client.284225.objecter handle_osd_map got epochs [686,686] > 0 2023-10-30T12:24:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1cb7fe640 3 client.284225.objecter handle_osd_map decoding full epoch 686 2023-10-30T12:24:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1cb7fe640 20 client.284225.objecter dump_active .. 0 homeless 2023-10-30T12:24:19.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1d259c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1b40392f0 0x7fb1b403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:24:19.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1d259c640 10 monclient: get_auth_request con 0x7fb1b40392f0 auth_method 0 2023-10-30T12:24:19.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1cb7fe640 10 client.284225.objecter ms_handle_connect 0x7fb1b40392f0 2023-10-30T12:24:19.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.953+0000 7fb1d259c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1b40392f0 0x7fb1b403b7b0 secure :-1 s=READY pgs=4374 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4131f80 tx=0x7fb1bc007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:24:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.957+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb1c4033990 con 0x7fb1d4099a40 2023-10-30T12:24:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.957+0000 7fb1cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:24:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:18.957+0000 7fb1cb7fe640 10 monclient: _finish_command 1 = system:0 2023-10-30T12:24:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.132+0000 7fb1d907c640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:24:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.132+0000 7fb1d907c640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:24:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.132+0000 7fb1d907c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.50:3301/0 2023-10-30T12:24:19.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.132+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fb1d4122890 con 0x7fb1d4099a40 2023-10-30T12:24:19.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.136+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb1c404ae90 con 0x7fb1d4099a40 2023-10-30T12:24:19.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.136+0000 7fb1cb7fe640 10 monclient: handle_config config(1 keys) v1 2023-10-30T12:24:19.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.136+0000 7fb1d159a640 4 set_mon_vals no callback set 2023-10-30T12:24:19.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.136+0000 7fb1d159a640 10 set_mon_vals mon_health_to_clog = false 2023-10-30T12:24:19.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.136+0000 7fb1cb7fe640 1 -- 172.21.15.50:0/3054673082 <== mon.2 v2:172.21.15.50:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fb1d4122890 con 0x7fb1d4099a40 2023-10-30T12:24:19.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.136+0000 7fb1cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-10-30T12:24:19.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.136+0000 7fb1cb7fe640 10 monclient: _finish_command 2 = system:0 2023-10-30T12:24:19.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 10 librados: watch_flush enter 2023-10-30T12:24:19.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 10 librados: watch_flush exit 2023-10-30T12:24:19.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 20 client.284225.objecter shutdown clearing up homeless session... 2023-10-30T12:24:19.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 10 client.284225.objecter successfully canceled tick 2023-10-30T12:24:19.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1b40392f0 msgr2=0x7fb1b403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:19.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1b40392f0 0x7fb1b403b7b0 secure :-1 s=READY pgs=4374 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4131f80 tx=0x7fb1bc007d30 comp rx=0 tx=0).stop 2023-10-30T12:24:19.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 10 monclient: shutdown 2023-10-30T12:24:19.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 20 monclient: shutdown discarding 0 pending message(s) 2023-10-30T12:24:19.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d4099a40 msgr2=0x7fb1d41ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:24:19.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d4099a40 0x7fb1d41ca2d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4037040 tx=0x7fb1c4004a40 comp rx=0 tx=0).stop 2023-10-30T12:24:19.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 shutdown_connections 2023-10-30T12:24:19.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:6824/1859404137,v1:172.21.15.118:6825/1859404137] conn(0x7fb1b40392f0 0x7fb1b403b7b0 unknown :-1 s=CLOSED pgs=4374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:19.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d4135e10 0x7fb1d411e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:19.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1d41327f0 0x7fb1d41ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:19.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 --2- 172.21.15.50:0/3054673082 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1d4099a40 0x7fb1d41ca2d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:24:19.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 >> 172.21.15.50:0/3054673082 conn(0x7fb1d40a82f0 msgr2=0x7fb1d4134020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:24:19.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 shutdown_connections 2023-10-30T12:24:19.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.138+0000 7fb1d907c640 1 -- 172.21.15.50:0/3054673082 wait complete. 2023-10-30T12:24:19.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-30T12:24:19.139+0000 7fb1d907c640 1 librados: shutdown 2023-10-30T12:24:19.150 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T12:24:19.151 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T12:24:19.151 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T12:24:19.152 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.188 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T12:24:19.189 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T12:24:19.189 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.262 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T12:24:19.263 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T12:24:19.263 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.339 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T12:24:19.340 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T12:24:19.340 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.378 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T12:24:19.379 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T12:24:19.379 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.451 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T12:24:19.452 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T12:24:19.452 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.520 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T12:24:19.521 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T12:24:19.521 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T12:24:19.521 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.587 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T12:24:19.587 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T12:24:19.588 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:24:19.588 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.601 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:24:19.602 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:24:19.602 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.652 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:24:19.653 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:24:19.654 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:24:19.666 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:24:19.667 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T12:24:19.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2023-10-30T12:24:19.733 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:24:19.734 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T12:24:19.911 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:24:19.912 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T12:24:20.008 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:24:20.009 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T12:24:20.068 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:24:20.069 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T12:24:20.285 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:24:20.286 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T12:24:20.385 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:24:20.386 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T12:24:20.450 INFO:tasks.ceph:Archiving mon data... 2023-10-30T12:24:20.452 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/data/mon.a.tgz 2023-10-30T12:24:20.452 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-10-30T12:24:20.469 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.vbcV73kzrH 2023-10-30T12:24:20.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.vbcV73kzrH 2023-10-30T12:24:20.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.vbcV73kzrH 2023-10-30T12:24:20.808 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.vbcV73kzrH is 978KB 2023-10-30T12:24:20.913 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.vbcV73kzrH 2023-10-30T12:24:20.928 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/data/mon.c.tgz 2023-10-30T12:24:20.928 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-10-30T12:24:20.982 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.uUrbvqT8vO 2023-10-30T12:24:20.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uUrbvqT8vO 2023-10-30T12:24:21.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.uUrbvqT8vO 2023-10-30T12:24:21.301 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.uUrbvqT8vO is 975KB 2023-10-30T12:24:21.405 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.uUrbvqT8vO 2023-10-30T12:24:21.420 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/data/mon.b.tgz 2023-10-30T12:24:21.420 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2023-10-30T12:24:21.438 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.UPvlRO7kAl 2023-10-30T12:24:21.439 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.UPvlRO7kAl 2023-10-30T12:24:21.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.UPvlRO7kAl 2023-10-30T12:24:21.761 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.UPvlRO7kAl is 975KB 2023-10-30T12:24:21.867 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.UPvlRO7kAl 2023-10-30T12:24:21.882 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T12:24:21.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:24:21.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:24:21.971 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-10-30T12:24:22.003 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-10-30T12:24:22.004 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T12:24:22.006 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/remote/smithi050/crash 2023-10-30T12:24:22.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:24:22.042 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/remote/smithi118/crash 2023-10-30T12:24:22.043 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:24:22.070 INFO:tasks.ceph:Compressing logs... 2023-10-30T12:24:22.070 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:24:22.088 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:24:22.203 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88573.log 2023-10-30T12:24:22.204 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88660.log 2023-10-30T12:24:22.204 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87747.log 2023-10-30T12:24:22.204 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88573.log: /var/log/ceph/ceph-client.admin.88660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87537.log 2023-10-30T12:24:22.205 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87615.log 2023-10-30T12:24:22.205 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87580.log 2023-10-30T12:24:22.206 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87537.log: /var/log/ceph/ceph-client.admin.87615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84190.log 2023-10-30T12:24:22.206 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87542.log 2023-10-30T12:24:22.207 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.84190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87397.log 2023-10-30T12:24:22.207 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87542.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84190.log.gz 2023-10-30T12:24:22.207 INFO:teuthology.orchestra.run.smithi118.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.88573.log.gz 2023-10-30T12:24:22.207 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87510.log 2023-10-30T12:24:22.208 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87397.log: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87846.log 2023-10-30T12:24:22.208 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88660.log.gz 2023-10-30T12:24:22.209 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84110.log 2023-10-30T12:24:22.209 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87846.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.87542.log.gz 2023-10-30T12:24:22.209 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87828.log 2023-10-30T12:24:22.210 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.84110.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.87615.log.gz 2023-10-30T12:24:22.210 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88524.log 2023-10-30T12:24:22.210 INFO:teuthology.orchestra.run.smithi118.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84110.log.gz/var/log/ceph/ceph-client.admin.87828.log: 2023-10-30T12:24:22.211 INFO:teuthology.orchestra.run.smithi118.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.87397.log.gz 2023-10-30T12:24:22.211 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:24:22.211 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88524.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88575.log 2023-10-30T12:24:22.212 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87537.log.gz 2023-10-30T12:24:22.212 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87929.log 2023-10-30T12:24:22.213 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88575.log: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.87846.log.gz 2023-10-30T12:24:22.213 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87417.log 2023-10-30T12:24:22.214 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87929.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.88524.log.gz 2023-10-30T12:24:22.214 INFO:teuthology.orchestra.run.smithi118.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.87580.log.gz 2023-10-30T12:24:22.215 INFO:teuthology.orchestra.run.smithi118.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.87510.log.gz 2023-10-30T12:24:22.215 INFO:teuthology.orchestra.run.smithi118.stderr:gzip 91.3% -- replaced with /var/log/ceph/ceph-client.admin.87747.log.gz 2023-10-30T12:24:22.216 INFO:teuthology.orchestra.run.smithi118.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.84070.log 2023-10-30T12:24:22.216 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87426.log 2023-10-30T12:24:22.216 INFO:teuthology.orchestra.run.smithi118.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.88575.log.gz 2023-10-30T12:24:22.217 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.84070.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T12:24:22.217 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87333.log 2023-10-30T12:24:22.218 INFO:teuthology.orchestra.run.smithi118.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84070.log.gz/var/log/ceph/ceph-osd.4.log: 2023-10-30T12:24:22.218 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87715.log 2023-10-30T12:24:22.219 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87307.log 2023-10-30T12:24:22.219 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87715.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.87929.log.gz 2023-10-30T12:24:22.219 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88471.log 2023-10-30T12:24:22.219 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87307.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.87333.log.gz 2023-10-30T12:24:22.220 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87917.log 2023-10-30T12:24:22.220 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88471.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.87426.log.gz 2023-10-30T12:24:22.220 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87577.log 2023-10-30T12:24:22.221 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87917.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.87917.log.gz 2023-10-30T12:24:22.221 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T12:24:22.221 INFO:teuthology.orchestra.run.smithi118.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.88471.log.gz 2023-10-30T12:24:22.221 INFO:teuthology.orchestra.run.smithi118.stderr: 92.0%/var/log/ceph/ceph-client.admin.87577.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T12:24:22.222 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88685.log 2023-10-30T12:24:22.222 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87261.log 2023-10-30T12:24:22.222 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88685.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.87715.log.gz 2023-10-30T12:24:22.222 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88473.log 2023-10-30T12:24:22.223 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87261.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.87577.log.gz 2023-10-30T12:24:22.223 INFO:teuthology.orchestra.run.smithi118.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.88685.log.gzgzip 2023-10-30T12:24:22.223 INFO:teuthology.orchestra.run.smithi118.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87473.log 2023-10-30T12:24:22.223 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88473.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88574.log 2023-10-30T12:24:22.224 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87307.log.gz 2023-10-30T12:24:22.224 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87473.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T12:24:22.224 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88574.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.88473.log.gz 2023-10-30T12:24:22.225 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88658.log 2023-10-30T12:24:22.225 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.3.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.88574.log.gz 2023-10-30T12:24:22.225 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87804.log 2023-10-30T12:24:22.225 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88658.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.88658.log.gz 2023-10-30T12:24:22.226 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88680.log 2023-10-30T12:24:22.226 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87804.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.87804.log.gz 2023-10-30T12:24:22.226 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87349.log 2023-10-30T12:24:22.227 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88680.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.88680.log.gz 2023-10-30T12:24:22.227 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88522.log 2023-10-30T12:24:22.227 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87349.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T12:24:22.227 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88522.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.88522.log.gz 2023-10-30T12:24:22.228 INFO:teuthology.orchestra.run.smithi118.stderr: 92.4% 90.1% -- replaced with /var/log/ceph/ceph-client.admin.87261.log.gz 2023-10-30T12:24:22.228 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87828.log.gz 2023-10-30T12:24:22.228 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88684.log 2023-10-30T12:24:22.229 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87870.log 2023-10-30T12:24:22.229 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88684.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88684.log.gz 2023-10-30T12:24:22.229 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87908.log 2023-10-30T12:24:22.230 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87870.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.87870.log.gz 2023-10-30T12:24:22.230 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87930.log 2023-10-30T12:24:22.230 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87908.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.87908.log.gz 2023-10-30T12:24:22.231 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87673.log 2023-10-30T12:24:22.231 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87930.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.87930.log.gz 2023-10-30T12:24:22.231 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88683.log 2023-10-30T12:24:22.234 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87673.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.87673.log.gz 2023-10-30T12:24:22.235 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84150.log 2023-10-30T12:24:22.235 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88683.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.88683.log.gz 2023-10-30T12:24:22.236 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:24:22.236 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.84150.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84150.log.gz 2023-10-30T12:24:22.236 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87914.log 2023-10-30T12:24:22.236 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87909.log 2023-10-30T12:24:22.237 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87914.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.87914.log.gz 2023-10-30T12:24:22.239 INFO:teuthology.orchestra.run.smithi118.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.87349.log.gz 2023-10-30T12:24:22.247 INFO:teuthology.orchestra.run.smithi118.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:24:22.247 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88472.log 2023-10-30T12:24:22.247 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84230.log 2023-10-30T12:24:22.248 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87909.log: /var/log/ceph/ceph-client.admin.88472.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87909.log.gz 2023-10-30T12:24:22.248 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T12:24:22.248 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.84230.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84230.log.gz 2023-10-30T12:24:22.249 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88576.log 2023-10-30T12:24:22.249 INFO:teuthology.orchestra.run.smithi118.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.88472.log.gz 2023-10-30T12:24:22.258 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87326.log 2023-10-30T12:24:22.265 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88576.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.88576.log.gz 2023-10-30T12:24:22.272 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88523.log 2023-10-30T12:24:22.279 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87326.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.87326.log.gz 2023-10-30T12:24:22.286 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87578.log 2023-10-30T12:24:22.286 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88523.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.88523.log.gz 2023-10-30T12:24:22.286 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87282.log 2023-10-30T12:24:22.304 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87772.log 2023-10-30T12:24:22.311 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88577.log 2023-10-30T12:24:22.318 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84270.log 2023-10-30T12:24:22.332 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88577.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88659.log 2023-10-30T12:24:22.333 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88577.log.gz 2023-10-30T12:24:22.333 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.84270.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84270.log.gz 2023-10-30T12:24:22.333 INFO:teuthology.orchestra.run.smithi118.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.87772.log.gz 2023-10-30T12:24:22.350 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87902.log 2023-10-30T12:24:22.350 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88659.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.88659.log.gz 2023-10-30T12:24:22.363 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88682.log 2023-10-30T12:24:22.363 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.87902.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87902.log.gz 2023-10-30T12:24:22.372 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104870.log 2023-10-30T12:24:22.373 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98217.log 2023-10-30T12:24:22.373 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105080.log 2023-10-30T12:24:22.373 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107225.log 2023-10-30T12:24:22.374 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104870.log.gz 2023-10-30T12:24:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89942.log 2023-10-30T12:24:22.374 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105080.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98217.log.gz 2023-10-30T12:24:22.375 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99401.log 2023-10-30T12:24:22.375 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107225.log: gzip/var/log/ceph/ceph-client.admin.89942.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.89193.log 2023-10-30T12:24:22.375 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.105080.log.gz 2023-10-30T12:24:22.375 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90158.log 2023-10-30T12:24:22.376 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99401.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107225.log.gz 2023-10-30T12:24:22.376 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89942.log.gz 2023-10-30T12:24:22.376 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91294.log 2023-10-30T12:24:22.377 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90158.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.99401.log.gz 2023-10-30T12:24:22.377 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101025.log 2023-10-30T12:24:22.377 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89193.log.gz 2023-10-30T12:24:22.377 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91294.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T12:24:22.378 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90158.log.gz 2023-10-30T12:24:22.378 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91847.log 2023-10-30T12:24:22.378 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91294.log.gz 2023-10-30T12:24:22.378 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100259.log 2023-10-30T12:24:22.379 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101025.log.gz 2023-10-30T12:24:22.379 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106973.log 2023-10-30T12:24:22.379 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106679.log 2023-10-30T12:24:22.379 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91847.log.gz 2023-10-30T12:24:22.380 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104255.log 2023-10-30T12:24:22.380 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100259.log.gz 2023-10-30T12:24:22.380 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103102.log 2023-10-30T12:24:22.380 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106679.log: /var/log/ceph/ceph-client.admin.104255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91065.log 2023-10-30T12:24:22.381 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106973.log.gz 2023-10-30T12:24:22.381 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88624.log 2023-10-30T12:24:22.381 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.106679.log.gz 2023-10-30T12:24:22.382 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97483.log 2023-10-30T12:24:22.382 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91065.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104255.log.gz 2023-10-30T12:24:22.382 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.103102.log.gz 2023-10-30T12:24:22.382 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98723.log 2023-10-30T12:24:22.383 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86689.log 2023-10-30T12:24:22.383 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97483.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88624.log.gz 2023-10-30T12:24:22.383 INFO:teuthology.orchestra.run.smithi050.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.91065.log.gz 2023-10-30T12:24:22.384 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94016.log 2023-10-30T12:24:22.384 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90979.log 2023-10-30T12:24:22.384 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97483.log.gz 2023-10-30T12:24:22.384 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98723.log.gz 2023-10-30T12:24:22.385 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99191.log 2023-10-30T12:24:22.385 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86689.log.gz 2023-10-30T12:24:22.385 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90638.log 2023-10-30T12:24:22.386 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90979.log: /var/log/ceph/ceph-client.admin.99191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92757.log 2023-10-30T12:24:22.386 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.94016.log.gz 2023-10-30T12:24:22.386 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101464.log 2023-10-30T12:24:22.386 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90979.log.gz 2023-10-30T12:24:22.387 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99191.log.gz 2023-10-30T12:24:22.387 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96668.log 2023-10-30T12:24:22.387 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90638.log.gz 2023-10-30T12:24:22.387 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99317.log 2023-10-30T12:24:22.388 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104744.log 2023-10-30T12:24:22.388 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92757.log.gz 2023-10-30T12:24:22.388 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99317.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101464.log.gz 2023-10-30T12:24:22.388 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93468.log 2023-10-30T12:24:22.389 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96668.log.gz 2023-10-30T12:24:22.389 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88750.log 2023-10-30T12:24:22.389 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99317.log.gz 2023-10-30T12:24:22.389 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99443.log 2023-10-30T12:24:22.390 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104744.log.gz 2023-10-30T12:24:22.390 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94326.log 2023-10-30T12:24:22.390 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93468.log.gz 2023-10-30T12:24:22.391 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87519.log 2023-10-30T12:24:22.391 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88750.log.gz 2023-10-30T12:24:22.391 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91250.log 2023-10-30T12:24:22.391 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94326.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99443.log.gz 2023-10-30T12:24:22.392 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106805.log 2023-10-30T12:24:22.392 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97214.log 2023-10-30T12:24:22.392 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94326.log.gz 2023-10-30T12:24:22.392 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87519.log.gz 2023-10-30T12:24:22.393 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104828.log 2023-10-30T12:24:22.393 INFO:teuthology.orchestra.run.smithi050.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.91250.log.gz 2023-10-30T12:24:22.393 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97904.log 2023-10-30T12:24:22.393 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97214.log: /var/log/ceph/ceph-client.admin.104828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94410.log 2023-10-30T12:24:22.394 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106805.log.gz 2023-10-30T12:24:22.394 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103186.log 2023-10-30T12:24:22.394 INFO:teuthology.orchestra.run.smithi050.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.97214.log.gz 2023-10-30T12:24:22.395 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93183.log 2023-10-30T12:24:22.395 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94410.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.104828.log.gz 2023-10-30T12:24:22.395 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106275.log 2023-10-30T12:24:22.396 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103186.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97904.log.gz 2023-10-30T12:24:22.396 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93183.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94410.log.gz 2023-10-30T12:24:22.396 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101590.log 2023-10-30T12:24:22.396 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.103186.log.gz -5 2023-10-30T12:24:22.397 INFO:teuthology.orchestra.run.smithi050.stderr: --verbose -- /var/log/ceph/ceph-client.admin.88792.log 2023-10-30T12:24:22.397 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106275.log: /var/log/ceph/ceph-client.admin.101590.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93183.log.gz 2023-10-30T12:24:22.397 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90068.log 2023-10-30T12:24:22.397 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88792.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106275.log.gz 2023-10-30T12:24:22.398 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88003.log 2023-10-30T12:24:22.398 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106191.log 2023-10-30T12:24:22.398 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90068.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.101590.log.gz 2023-10-30T12:24:22.399 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88792.log.gz 2023-10-30T12:24:22.399 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107351.log 2023-10-30T12:24:22.399 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90068.log.gz 2023-10-30T12:24:22.399 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93099.log 2023-10-30T12:24:22.400 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106191.log: /var/log/ceph/ceph-client.admin.107351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87347.log 2023-10-30T12:24:22.400 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.88003.log.gz 2023-10-30T12:24:22.400 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100091.log 2023-10-30T12:24:22.400 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106191.log.gz 2023-10-30T12:24:22.401 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107351.log.gz 2023-10-30T12:24:22.401 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100217.log 2023-10-30T12:24:22.401 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87347.log: /var/log/ceph/ceph-client.admin.100091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105122.log 2023-10-30T12:24:22.401 INFO:teuthology.orchestra.run.smithi050.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.93099.log.gz 2023-10-30T12:24:22.402 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.87347.log.gz 2023-10-30T12:24:22.402 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95988.log 2023-10-30T12:24:22.402 INFO:teuthology.orchestra.run.smithi050.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.100091.log.gz 2023-10-30T12:24:22.403 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102810.log 2023-10-30T12:24:22.403 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105122.log: /var/log/ceph/ceph-client.admin.95988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104171.log 2023-10-30T12:24:22.403 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100217.log.gz 2023-10-30T12:24:22.403 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99839.log 2023-10-30T12:24:22.404 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105122.log.gz 2023-10-30T12:24:22.404 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95988.log.gz 2023-10-30T12:24:22.404 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106847.log 2023-10-30T12:24:22.404 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104171.log: /var/log/ceph/ceph-client.admin.99839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95440.log 2023-10-30T12:24:22.405 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102810.log.gz 2023-10-30T12:24:22.405 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106847.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104171.log.gz 2023-10-30T12:24:22.405 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88498.log 2023-10-30T12:24:22.405 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91965.log 2023-10-30T12:24:22.406 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95440.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106847.log.gz 2023-10-30T12:24:22.406 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98467.log 2023-10-30T12:24:22.406 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96282.log 2023-10-30T12:24:22.407 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95440.log.gz 2023-10-30T12:24:22.407 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88374.log 2023-10-30T12:24:22.407 INFO:teuthology.orchestra.run.smithi050.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.99839.log.gz 2023-10-30T12:24:22.407 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91965.log.gz 2023-10-30T12:24:22.408 INFO:teuthology.orchestra.run.smithi050.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.88498.log.gz 2023-10-30T12:24:22.408 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99715.log 2023-10-30T12:24:22.408 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96282.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98467.log.gz 2023-10-30T12:24:22.408 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105643.log 2023-10-30T12:24:22.409 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88374.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.99715.log: /var/log/ceph/ceph-client.admin.87919.log 2023-10-30T12:24:22.409 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96282.log.gz 2023-10-30T12:24:22.409 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93267.log 2023-10-30T12:24:22.409 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88374.log.gz 2023-10-30T12:24:22.410 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99715.log.gz 2023-10-30T12:24:22.410 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94453.log 2023-10-30T12:24:22.410 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105643.log.gz 2023-10-30T12:24:22.411 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89451.log 2023-10-30T12:24:22.411 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93267.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87919.log.gz 2023-10-30T12:24:22.411 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93932.log 2023-10-30T12:24:22.411 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93267.log.gz 2023-10-30T12:24:22.412 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100857.log 2023-10-30T12:24:22.412 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89451.log: /var/log/ceph/ceph-client.admin.93932.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94453.log.gz 2023-10-30T12:24:22.412 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102284.log 2023-10-30T12:24:22.412 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87095.log 2023-10-30T12:24:22.413 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93932.log.gz 2023-10-30T12:24:22.413 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89451.log.gz 2023-10-30T12:24:22.413 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103876.log 2023-10-30T12:24:22.413 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95001.log 2023-10-30T12:24:22.414 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87095.log: /var/log/ceph/ceph-client.admin.103876.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.100857.log.gz 2023-10-30T12:24:22.414 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98030.log 2023-10-30T12:24:22.414 INFO:teuthology.orchestra.run.smithi050.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.102284.log.gz 2023-10-30T12:24:22.415 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107309.log 2023-10-30T12:24:22.415 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95001.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.87095.log.gz 2023-10-30T12:24:22.415 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103876.log.gz 2023-10-30T12:24:22.415 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106595.log 2023-10-30T12:24:22.416 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92179.log 2023-10-30T12:24:22.416 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95001.log.gz 87.6% 2023-10-30T12:24:22.416 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.98030.log.gz 2023-10-30T12:24:22.417 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93015.log 2023-10-30T12:24:22.417 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107309.log.gz/var/log/ceph/ceph-client.admin.92179.log: 2023-10-30T12:24:22.417 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99065.log 2023-10-30T12:24:22.417 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93015.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92179.log.gz 2023-10-30T12:24:22.418 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87877.log 2023-10-30T12:24:22.418 INFO:teuthology.orchestra.run.smithi050.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.106595.log.gz 2023-10-30T12:24:22.418 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94495.log 2023-10-30T12:24:22.418 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99065.log: /var/log/ceph/ceph-client.admin.87877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101109.log 2023-10-30T12:24:22.419 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93015.log.gz 2023-10-30T12:24:22.419 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84131.log 2023-10-30T12:24:22.419 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99065.log.gz 2023-10-30T12:24:22.420 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87877.log.gz 2023-10-30T12:24:22.420 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92477.log 2023-10-30T12:24:22.420 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.84131.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.94495.log.gz 2023-10-30T12:24:22.420 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106233.log 2023-10-30T12:24:22.421 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90895.log 2023-10-30T12:24:22.421 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101109.log.gz 2023-10-30T12:24:22.421 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99023.log 2023-10-30T12:24:22.421 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92477.log.gz 2023-10-30T12:24:22.422 INFO:teuthology.orchestra.run.smithi050.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.84131.log.gz 2023-10-30T12:24:22.422 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100601.log 2023-10-30T12:24:22.422 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106233.log.gz 2023-10-30T12:24:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87340.log 2023-10-30T12:24:22.423 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90895.log.gz 2023-10-30T12:24:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88582.log 2023-10-30T12:24:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87340.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99023.log.gz 2023-10-30T12:24:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86953.log 2023-10-30T12:24:22.424 INFO:teuthology.orchestra.run.smithi050.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.100601.log.gz 2023-10-30T12:24:22.424 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93594.log 2023-10-30T12:24:22.425 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87340.log.gz 2023-10-30T12:24:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100983.log 2023-10-30T12:24:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86953.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88582.log.gz 2023-10-30T12:24:22.425 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89888.log 2023-10-30T12:24:22.426 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86953.log.gz 2023-10-30T12:24:22.426 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97694.log 2023-10-30T12:24:22.426 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93594.log.gz 2023-10-30T12:24:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99107.log 2023-10-30T12:24:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89888.log: /var/log/ceph/ceph-client.admin.97694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89322.log 2023-10-30T12:24:22.427 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100983.log.gz 2023-10-30T12:24:22.427 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98933.log 2023-10-30T12:24:22.428 INFO:teuthology.orchestra.run.smithi050.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.89888.log.gz 2023-10-30T12:24:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89322.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97694.log.gz 2023-10-30T12:24:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100301.log 2023-10-30T12:24:22.428 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99107.log.gz 2023-10-30T12:24:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87337.log 2023-10-30T12:24:22.429 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89322.log.gz 2023-10-30T12:24:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95860.log 2023-10-30T12:24:22.429 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98933.log.gz 2023-10-30T12:24:22.430 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96710.log 2023-10-30T12:24:22.430 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100301.log.gz 2023-10-30T12:24:22.430 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101336.log 2023-10-30T12:24:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95860.log: /var/log/ceph/ceph-client.admin.96710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95356.log 2023-10-30T12:24:22.431 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87337.log.gz 2023-10-30T12:24:22.431 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104297.log 2023-10-30T12:24:22.431 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95860.log.gz 2023-10-30T12:24:22.432 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96710.log.gz 2023-10-30T12:24:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95776.log 2023-10-30T12:24:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95356.log: /var/log/ceph/ceph-client.admin.104297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87583.log 2023-10-30T12:24:22.432 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95356.log.gz 2023-10-30T12:24:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94747.log 2023-10-30T12:24:22.433 INFO:teuthology.orchestra.run.smithi050.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.101336.log.gz 2023-10-30T12:24:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107099.log 2023-10-30T12:24:22.434 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104297.log.gz 2023-10-30T12:24:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97568.log 2023-10-30T12:24:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94747.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.95776.log.gz 2023-10-30T12:24:22.434 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104465.log 2023-10-30T12:24:22.435 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87583.log.gz 2023-10-30T12:24:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104213.log 2023-10-30T12:24:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97568.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94747.log.gz 2023-10-30T12:24:22.435 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107099.log.gz 2023-10-30T12:24:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104129.log 2023-10-30T12:24:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87336.log 2023-10-30T12:24:22.436 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104213.log: /var/log/ceph/ceph-client.admin.104129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96578.log 2023-10-30T12:24:22.437 INFO:teuthology.orchestra.run.smithi050.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.97568.log.gz 2023-10-30T12:24:22.437 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87336.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.104465.log.gz 2023-10-30T12:24:22.437 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96492.log 2023-10-30T12:24:22.437 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.104213.log.gz 2023-10-30T12:24:22.438 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104129.log.gz 2023-10-30T12:24:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104381.log 2023-10-30T12:24:22.438 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87336.log.gz 2023-10-30T12:24:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96836.log 2023-10-30T12:24:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102536.log 2023-10-30T12:24:22.439 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96578.log.gz 2023-10-30T12:24:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96836.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104381.log.gz 2023-10-30T12:24:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101420.log 2023-10-30T12:24:22.440 INFO:teuthology.orchestra.run.smithi050.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.96492.log.gz 2023-10-30T12:24:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98156.log/var/log/ceph/ceph-client.admin.102536.log: 2023-10-30T12:24:22.440 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96836.log.gz 2023-10-30T12:24:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92435.log 2023-10-30T12:24:22.441 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.102536.log.gz 2023-10-30T12:24:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.98156.log: /var/log/ceph/ceph-client.admin.93351.log 2023-10-30T12:24:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95650.log 2023-10-30T12:24:22.442 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101420.log.gz 2023-10-30T12:24:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93351.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98156.log.gz 2023-10-30T12:24:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91805.log 2023-10-30T12:24:22.443 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92435.log.gz 2023-10-30T12:24:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89493.log 2023-10-30T12:24:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95650.log: /var/log/ceph/ceph-client.admin.91805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90937.log 2023-10-30T12:24:22.443 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.93351.log.gz 2023-10-30T12:24:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89493.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95650.log.gz 2023-10-30T12:24:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90811.log 2023-10-30T12:24:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87751.log 2023-10-30T12:24:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90937.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91805.log.gz 2023-10-30T12:24:22.445 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89493.log.gz 2023-10-30T12:24:22.445 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104423.log 2023-10-30T12:24:22.445 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90937.log.gz 2023-10-30T12:24:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94621.log 2023-10-30T12:24:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87751.log: /var/log/ceph/ceph-client.admin.104423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93890.log 2023-10-30T12:24:22.446 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90811.log.gz 2023-10-30T12:24:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92971.log 2023-10-30T12:24:22.447 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87751.log.gz 2023-10-30T12:24:22.447 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104423.log.gz 2023-10-30T12:24:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89703.log 2023-10-30T12:24:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93890.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94621.log.gz 2023-10-30T12:24:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95524.log 2023-10-30T12:24:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96366.log 2023-10-30T12:24:22.448 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92971.log.gz 2023-10-30T12:24:22.449 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93890.log.gz 2023-10-30T12:24:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106487.log 2023-10-30T12:24:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96156.log 2023-10-30T12:24:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96366.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89703.log.gz 2023-10-30T12:24:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106487.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95524.log.gz 2023-10-30T12:24:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97004.log 2023-10-30T12:24:22.450 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93848.log 2023-10-30T12:24:22.451 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96366.log.gz 2023-10-30T12:24:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96156.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106487.log.gz 2023-10-30T12:24:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94184.log 2023-10-30T12:24:22.451 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104087.log 2023-10-30T12:24:22.452 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96156.log.gz 2023-10-30T12:24:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105290.log 2023-10-30T12:24:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94184.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.97004.log.gz 2023-10-30T12:24:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91420.log 2023-10-30T12:24:22.453 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.93848.log.gz 2023-10-30T12:24:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91919.log 2023-10-30T12:24:22.453 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94184.log.gz 2023-10-30T12:24:22.454 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104087.log.gz 2023-10-30T12:24:22.454 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98807.log 2023-10-30T12:24:22.454 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.105290.log.gz 2023-10-30T12:24:22.454 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92393.log 2023-10-30T12:24:22.455 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91919.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91420.log.gz 2023-10-30T12:24:22.455 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89984.log 2023-10-30T12:24:22.455 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99612.log 2023-10-30T12:24:22.456 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91919.log.gz 2023-10-30T12:24:22.456 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.98807.log.gz 2023-10-30T12:24:22.456 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102117.log 2023-10-30T12:24:22.456 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92393.log.gz 2023-10-30T12:24:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87093.log 2023-10-30T12:24:22.457 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99612.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.102117.log: -- /var/log/ceph/ceph-client.admin.104549.log 2023-10-30T12:24:22.457 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89984.log.gz 2023-10-30T12:24:22.458 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106318.log 2023-10-30T12:24:22.458 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99612.log.gz 2023-10-30T12:24:22.458 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104549.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102117.log.gzgzip 2023-10-30T12:24:22.458 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90727.log 2023-10-30T12:24:22.459 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87093.log.gz 2023-10-30T12:24:22.459 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107685.log 2023-10-30T12:24:22.459 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104549.log.gz 2023-10-30T12:24:22.459 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98072.log 2023-10-30T12:24:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90727.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106318.log.gz 2023-10-30T12:24:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107685.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:24:22.460 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103650.log 2023-10-30T12:24:22.461 INFO:teuthology.orchestra.run.smithi050.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.90727.log.gz 2023-10-30T12:24:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.audit.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107685.log.gz 2023-10-30T12:24:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96878.log 2023-10-30T12:24:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94875.log 2023-10-30T12:24:22.462 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103650.log: /var/log/ceph/ceph-client.admin.96878.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.98072.log.gz 2023-10-30T12:24:22.462 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107477.log 2023-10-30T12:24:22.462 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103650.log.gz 2023-10-30T12:24:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98298.log 2023-10-30T12:24:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.102054.log -- replaced with /var/log/ceph/ceph-client.admin.96878.log.gz 2023-10-30T12:24:22.463 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.463 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107477.log: /var/log/ceph/ceph-client.admin.98298.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94875.log.gz 2023-10-30T12:24:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105685.log 2023-10-30T12:24:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92929.log 2023-10-30T12:24:22.464 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6%/var/log/ceph/ceph-client.admin.102054.log: -- replaced with /var/log/ceph/ceph-client.admin.107477.log.gz 2023-10-30T12:24:22.464 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98298.log.gz 2023-10-30T12:24:22.465 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107645.log 2023-10-30T12:24:22.465 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106889.log 2023-10-30T12:24:22.465 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102054.log.gz 2023-10-30T12:24:22.465 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105685.log.gz 2023-10-30T12:24:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99796.log 2023-10-30T12:24:22.466 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92929.log.gz 2023-10-30T12:24:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100133.log 2023-10-30T12:24:22.467 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107645.log.gz 2023-10-30T12:24:22.467 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96624.log 2023-10-30T12:24:22.467 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99796.log: /var/log/ceph/ceph-client.admin.100133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99486.log 2023-10-30T12:24:22.467 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106889.log.gz 2023-10-30T12:24:22.468 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103524.log 2023-10-30T12:24:22.468 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99796.log.gz 2023-10-30T12:24:22.468 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.100133.log.gz 2023-10-30T12:24:22.468 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88918.log 2023-10-30T12:24:22.469 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103524.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.96624.log.gz 2023-10-30T12:24:22.469 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103398.log 2023-10-30T12:24:22.469 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99486.log.gz 2023-10-30T12:24:22.469 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103482.log 2023-10-30T12:24:22.470 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.103524.log.gz 2023-10-30T12:24:22.470 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97299.log 2023-10-30T12:24:22.470 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96920.log 2023-10-30T12:24:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103482.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.88918.log.gz 2023-10-30T12:24:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97610.log 2023-10-30T12:24:22.471 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103398.log.gz 2023-10-30T12:24:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96920.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103482.log.gz -5 2023-10-30T12:24:22.472 INFO:teuthology.orchestra.run.smithi050.stderr: --verbose -- /var/log/ceph/ceph-client.admin.105981.log 2023-10-30T12:24:22.472 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95185.log 2023-10-30T12:24:22.472 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97299.log.gz 2023-10-30T12:24:22.472 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97610.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96920.log.gz 2023-10-30T12:24:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87793.log 2023-10-30T12:24:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90026.log 2023-10-30T12:24:22.473 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95185.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97610.log.gz 2023-10-30T12:24:22.474 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105981.log.gz 2023-10-30T12:24:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89063.log 2023-10-30T12:24:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107183.log 2023-10-30T12:24:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90026.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95185.log.gz 2023-10-30T12:24:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104003.log 2023-10-30T12:24:22.475 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87793.log.gz 2023-10-30T12:24:22.475 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105038.log 2023-10-30T12:24:22.476 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90026.log.gz 2023-10-30T12:24:22.476 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104045.log 2023-10-30T12:24:22.476 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89063.log.gz 2023-10-30T12:24:22.476 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104003.log: /var/log/ceph/ceph-client.admin.105038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88186.log 2023-10-30T12:24:22.477 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107183.log.gz 2023-10-30T12:24:22.477 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104003.log.gz 2023-10-30T12:24:22.477 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98849.log 2023-10-30T12:24:22.477 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 89.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.105038.log.gz /var/log/ceph/ceph-client.admin.93636.log 2023-10-30T12:24:22.478 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.478 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88186.log: /var/log/ceph/ceph-client.admin.98849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102933.log 2023-10-30T12:24:22.478 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104045.log.gz 2023-10-30T12:24:22.479 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88186.log.gz 2023-10-30T12:24:22.479 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91107.log 2023-10-30T12:24:22.479 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89845.log 2023-10-30T12:24:22.479 INFO:teuthology.orchestra.run.smithi050.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.98849.log.gz 2023-10-30T12:24:22.480 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91107.log: gzip -5 --verbose 89.8% -- replaced with /var/log/ceph/ceph-client.admin.93636.log.gz 2023-10-30T12:24:22.480 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- /var/log/ceph/ceph-client.admin.91721.log -- replaced with /var/log/ceph/ceph-client.admin.102933.log.gz 2023-10-30T12:24:22.480 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.480 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90681.log 2023-10-30T12:24:22.481 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.91107.log.gz 2023-10-30T12:24:22.481 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102578.log 2023-10-30T12:24:22.481 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91721.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89845.log.gz 2023-10-30T12:24:22.481 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101151.log 2023-10-30T12:24:22.482 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91721.log.gz 2023-10-30T12:24:22.482 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99654.log 2023-10-30T12:24:22.482 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101151.log: gzip -5 91.0% -- replaced with /var/log/ceph/ceph-client.admin.90681.log.gz 2023-10-30T12:24:22.483 INFO:teuthology.orchestra.run.smithi050.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90244.log 2023-10-30T12:24:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99654.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.102578.log.gz 2023-10-30T12:24:22.483 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98981.log 2023-10-30T12:24:22.483 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101151.log.gz 2023-10-30T12:24:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102199.log 2023-10-30T12:24:22.484 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99654.log.gz 2023-10-30T12:24:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99233.log 2023-10-30T12:24:22.484 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103228.log 2023-10-30T12:24:22.485 INFO:teuthology.orchestra.run.smithi050.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T12:24:22.485 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102199.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.90244.log.gz 2023-10-30T12:24:22.485 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100559.log 2023-10-30T12:24:22.486 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98981.log.gz 2023-10-30T12:24:22.486 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102199.log.gz 2023-10-30T12:24:22.486 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91504.log 2023-10-30T12:24:22.486 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103228.log: /var/log/ceph/ceph-client.admin.100559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95314.log 2023-10-30T12:24:22.487 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99233.log.gz 2023-10-30T12:24:22.487 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87092.log 2023-10-30T12:24:22.487 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103228.log.gz 2023-10-30T12:24:22.487 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100559.log.gz 2023-10-30T12:24:22.488 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93057.log 2023-10-30T12:24:22.488 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95314.log: /var/log/ceph/ceph-client.admin.87092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102891.log 2023-10-30T12:24:22.488 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91504.log.gz 2023-10-30T12:24:22.489 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100771.log 2023-10-30T12:24:22.489 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95314.log.gz 2023-10-30T12:24:22.489 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107057.log 2023-10-30T12:24:22.489 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102891.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87092.log.gz 2023-10-30T12:24:22.490 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.93057.log.gz 2023-10-30T12:24:22.490 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92137.log 2023-10-30T12:24:22.490 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102891.log.gz 2023-10-30T12:24:22.490 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93678.log 2023-10-30T12:24:22.491 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107057.log: /var/log/ceph/ceph-client.admin.92137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95482.log 2023-10-30T12:24:22.491 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100771.log.gz 2023-10-30T12:24:22.491 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93678.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107057.log.gz 2023-10-30T12:24:22.491 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87667.log 2023-10-30T12:24:22.492 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.92137.log.gz 2023-10-30T12:24:22.492 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97988.log 2023-10-30T12:24:22.492 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95482.log: /var/log/ceph/ceph-client.admin.87667.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T12:24:22.493 INFO:teuthology.orchestra.run.smithi050.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.93678.log.gz 2023-10-30T12:24:22.493 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97988.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95482.log.gz 2023-10-30T12:24:22.493 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84223.log 2023-10-30T12:24:22.493 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87667.log.gz 2023-10-30T12:24:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97088.log 2023-10-30T12:24:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.1.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.97988.log.gz 2023-10-30T12:24:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.84223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100815.log 2023-10-30T12:24:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95043.log 2023-10-30T12:24:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97088.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84223.log.gz 2023-10-30T12:24:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94959.log 2023-10-30T12:24:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106107.log 2023-10-30T12:24:22.496 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.97088.log.gz 2023-10-30T12:24:22.496 INFO:teuthology.orchestra.run.smithi050.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.100815.log.gz 2023-10-30T12:24:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89661.log 2023-10-30T12:24:22.496 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106107.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95043.log.gz 2023-10-30T12:24:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98597.log 2023-10-30T12:24:22.497 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105206.log 2023-10-30T12:24:22.497 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.94959.log.gz 2023-10-30T12:24:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106721.log 2023-10-30T12:24:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98597.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106107.log.gz 2023-10-30T12:24:22.498 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89661.log.gz 2023-10-30T12:24:22.498 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92887.log 2023-10-30T12:24:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103356.log 2023-10-30T12:24:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106721.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.98597.log.gz 2023-10-30T12:24:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101506.log 2023-10-30T12:24:22.499 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105206.log.gz 2023-10-30T12:24:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107519.log 2023-10-30T12:24:22.500 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.106721.log.gz 2023-10-30T12:24:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95228.log 2023-10-30T12:24:22.501 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.92887.log.gz 2023-10-30T12:24:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107519.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.103356.log.gz 2023-10-30T12:24:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103753.log 2023-10-30T12:24:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89407.log 2023-10-30T12:24:22.502 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101506.log.gz 2023-10-30T12:24:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103753.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107519.log.gz 2023-10-30T12:24:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101212.log 2023-10-30T12:24:22.503 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104996.log 2023-10-30T12:24:22.503 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.103753.log.gz 2023-10-30T12:24:22.503 INFO:teuthology.orchestra.run.smithi050.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.95228.log.gz 2023-10-30T12:24:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105811.log 2023-10-30T12:24:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:gzip/var/log/ceph/ceph-client.admin.104996.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.93806.log 2023-10-30T12:24:22.504 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89407.log.gz 2023-10-30T12:24:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92095.log 2023-10-30T12:24:22.505 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101212.log.gz 2023-10-30T12:24:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103918.log 2023-10-30T12:24:22.505 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.104996.log.gz 2023-10-30T12:24:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102975.log 2023-10-30T12:24:22.506 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105811.log.gz 2023-10-30T12:24:22.506 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.93806.log.gz 2023-10-30T12:24:22.506 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97441.log 2023-10-30T12:24:22.507 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92095.log.gz 2023-10-30T12:24:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100049.log 2023-10-30T12:24:22.507 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103918.log.gz 2023-10-30T12:24:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91149.log 2023-10-30T12:24:22.508 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102975.log.gz 2023-10-30T12:24:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92845.log 2023-10-30T12:24:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91149.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97441.log.gz 2023-10-30T12:24:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92664.log 2023-10-30T12:24:22.509 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100049.log.gz 2023-10-30T12:24:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95946.log 2023-10-30T12:24:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103018.log 2023-10-30T12:24:22.510 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0%/var/log/ceph/ceph-client.admin.92664.log: -- replaced with /var/log/ceph/ceph-client.admin.91149.log.gz 2023-10-30T12:24:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93764.log 2023-10-30T12:24:22.510 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.92845.log.gz 2023-10-30T12:24:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102242.log 2023-10-30T12:24:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103018.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92664.log.gz 2023-10-30T12:24:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107141.log 2023-10-30T12:24:22.511 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95946.log.gz 2023-10-30T12:24:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102242.log: gzip 90.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.103018.log.gz -- 2023-10-30T12:24:22.512 INFO:teuthology.orchestra.run.smithi050.stderr: /var/log/ceph/ceph-client.admin.102620.log 2023-10-30T12:24:22.512 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93764.log.gz 2023-10-30T12:24:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90201.log 2023-10-30T12:24:22.513 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103312.log 2023-10-30T12:24:22.513 INFO:teuthology.orchestra.run.smithi050.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.102242.log.gz 2023-10-30T12:24:22.513 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2%/var/log/ceph/ceph-client.admin.90201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88708.log 2023-10-30T12:24:22.513 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107141.log.gz 2023-10-30T12:24:22.514 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102620.log.gz 2023-10-30T12:24:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106445.log 2023-10-30T12:24:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88708.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90201.log.gz 2023-10-30T12:24:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106637.log 2023-10-30T12:24:22.515 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103312.log.gz 2023-10-30T12:24:22.515 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97736.log 2023-10-30T12:24:22.517 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93309.log 2023-10-30T12:24:22.517 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.88708.log.gz 2023-10-30T12:24:22.518 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97736.log: gzip -5 --verbose -- 89.5% -- replaced with /var/log/ceph/ceph-client.admin.106445.log.gz 2023-10-30T12:24:22.518 INFO:teuthology.orchestra.run.smithi050.stderr: /var/log/ceph/ceph-client.admin.101293.log 2023-10-30T12:24:22.518 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.106637.log.gz 2023-10-30T12:24:22.519 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90286.log 2023-10-30T12:24:22.519 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97736.log.gz 2023-10-30T12:24:22.519 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96072.log 2023-10-30T12:24:22.519 INFO:teuthology.orchestra.run.smithi050.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.93309.log.gz 2023-10-30T12:24:22.520 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104507.log 2023-10-30T12:24:22.520 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9%/var/log/ceph/ceph-client.admin.96072.log: -- replaced with /var/log/ceph/ceph-client.admin.101293.log.gz 2023-10-30T12:24:22.520 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87835.log 2023-10-30T12:24:22.520 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90286.log.gz 2023-10-30T12:24:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94579.log 2023-10-30T12:24:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87835.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96072.log.gz 2023-10-30T12:24:22.521 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107603.log 2023-10-30T12:24:22.521 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94579.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87835.log.gz 2023-10-30T12:24:22.522 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.104507.log.gz 2023-10-30T12:24:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103834.log 2023-10-30T12:24:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107603.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.94579.log.gz 2023-10-30T12:24:22.523 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.94833.log 2023-10-30T12:24:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91588.log 2023-10-30T12:24:22.523 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107603.log.gz 2023-10-30T12:24:22.523 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92267.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103834.log.gz 2023-10-30T12:24:22.524 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92221.log 2023-10-30T12:24:22.524 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94833.log.gz 2023-10-30T12:24:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93393.log 2023-10-30T12:24:22.525 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91588.log.gz 2023-10-30T12:24:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88228.log 2023-10-30T12:24:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93393.log: gzip 89.7% -- replaced with /var/log/ceph/ceph-client.admin.92267.log.gz 2023-10-30T12:24:22.526 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.98383.log 2023-10-30T12:24:22.526 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92221.log.gz 2023-10-30T12:24:22.526 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96450.log 2023-10-30T12:24:22.526 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93393.log.gz 2023-10-30T12:24:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88876.log 2023-10-30T12:24:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98383.log: /var/log/ceph/ceph-client.admin.96450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99923.log 2023-10-30T12:24:22.527 INFO:teuthology.orchestra.run.smithi050.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.88228.log.gz 2023-10-30T12:24:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:gzip/var/log/ceph/ceph-client.admin.88876.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.87085.log 2023-10-30T12:24:22.528 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98383.log.gz 2023-10-30T12:24:22.528 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96450.log.gz 2023-10-30T12:24:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89577.log 2023-10-30T12:24:22.529 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88876.log.gz 2023-10-30T12:24:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102410.log 2023-10-30T12:24:22.529 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99923.log.gz 2023-10-30T12:24:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104786.log 2023-10-30T12:24:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89577.log: /var/log/ceph/ceph-client.admin.102410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95734.log 2023-10-30T12:24:22.530 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87085.log.gz 2023-10-30T12:24:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106931.log 2023-10-30T12:24:22.531 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89577.log.gz 2023-10-30T12:24:22.531 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102410.log.gz 2023-10-30T12:24:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94245.log 2023-10-30T12:24:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106931.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.104786.log.gz 2023-10-30T12:24:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94789.log 2023-10-30T12:24:22.532 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95734.log.gz 2023-10-30T12:24:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93551.log 2023-10-30T12:24:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.106931.log.gz --verbose 2023-10-30T12:24:22.533 INFO:teuthology.orchestra.run.smithi050.stderr: -- /var/log/ceph/ceph-client.admin.89277.log 2023-10-30T12:24:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94789.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94245.log.gz 2023-10-30T12:24:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101378.log 2023-10-30T12:24:22.534 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101800.log 2023-10-30T12:24:22.534 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89277.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94789.log.gz 2023-10-30T12:24:22.534 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88144.log 2023-10-30T12:24:22.534 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93551.log.gz 2023-10-30T12:24:22.535 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104617.log 2023-10-30T12:24:22.535 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89277.log.gz 2023-10-30T12:24:22.535 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101378.log.gz 2023-10-30T12:24:22.536 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88270.log 2023-10-30T12:24:22.536 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88144.log: /var/log/ceph/ceph-client.admin.104617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97652.log 2023-10-30T12:24:22.536 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.101800.log.gz 2023-10-30T12:24:22.536 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88144.log.gz 2023-10-30T12:24:22.537 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94917.log 2023-10-30T12:24:22.537 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104617.log.gz/var/log/ceph/ceph-client.admin.97652.log: 2023-10-30T12:24:22.537 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98425.log 2023-10-30T12:24:22.537 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88270.log.gz 2023-10-30T12:24:22.538 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106361.log 2023-10-30T12:24:22.538 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97652.log.gz 2023-10-30T12:24:22.538 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100473.log 2023-10-30T12:24:22.539 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94917.log.gz 2023-10-30T12:24:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88834.log 2023-10-30T12:24:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106361.log: /var/log/ceph/ceph-client.admin.100473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100385.log 2023-10-30T12:24:22.539 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.98425.log.gz 2023-10-30T12:24:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87094.log 2023-10-30T12:24:22.540 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106361.log.gz 2023-10-30T12:24:22.540 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100473.log.gz 2023-10-30T12:24:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90557.log 2023-10-30T12:24:22.541 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88834.log.gz 2023-10-30T12:24:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87094.log: gzip -5 --verbose -- 89.4% /var/log/ceph/ceph-client.admin.91462.log 2023-10-30T12:24:22.541 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.100385.log.gz 2023-10-30T12:24:22.541 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104954.log 2023-10-30T12:24:22.542 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87094.log.gz 2023-10-30T12:24:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91462.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90557.log.gz 2023-10-30T12:24:22.542 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107015.log 2023-10-30T12:24:22.543 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105517.log 2023-10-30T12:24:22.543 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91462.log.gz 2023-10-30T12:24:22.543 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86911.log 2023-10-30T12:24:22.543 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104954.log.gz 2023-10-30T12:24:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86745.log 2023-10-30T12:24:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105517.log: /var/log/ceph/ceph-client.admin.86911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100643.log 2023-10-30T12:24:22.544 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107015.log.gz 2023-10-30T12:24:22.544 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105517.log.gz 2023-10-30T12:24:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103144.log 2023-10-30T12:24:22.545 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86911.log.gz 2023-10-30T12:24:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91378.log 2023-10-30T12:24:22.545 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86745.log.gz 2023-10-30T12:24:22.546 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93722.log 2023-10-30T12:24:22.546 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91378.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.100643.log.gz 2023-10-30T12:24:22.546 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105393.log 2023-10-30T12:24:22.547 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103144.log.gz 2023-10-30T12:24:22.547 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98681.log 2023-10-30T12:24:22.547 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93722.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.91378.log.gz 2023-10-30T12:24:22.547 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94537.log 2023-10-30T12:24:22.548 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102704.log 2023-10-30T12:24:22.548 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98681.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.93722.log.gz 2023-10-30T12:24:22.548 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105393.log.gz 2023-10-30T12:24:22.548 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97778.log 2023-10-30T12:24:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5/var/log/ceph/ceph-client.admin.102704.log: --verbose -- /var/log/ceph/ceph-client.admin.100941.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.98681.log.gz 2023-10-30T12:24:22.549 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.549 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97778.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.94537.log.gz 2023-10-30T12:24:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100343.log 2023-10-30T12:24:22.550 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102704.log.gz 2023-10-30T12:24:22.550 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102452.log 2023-10-30T12:24:22.550 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97778.log.gz 2023-10-30T12:24:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89235.log 2023-10-30T12:24:22.551 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100941.log.gz 2023-10-30T12:24:22.551 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92603.log 2023-10-30T12:24:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89235.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.100343.log.gz 2023-10-30T12:24:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94142.log 2023-10-30T12:24:22.552 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.102452.log.gz 2023-10-30T12:24:22.552 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5/var/log/ceph/ceph-client.admin.92603.log: --verbose -- /var/log/ceph/ceph-client.admin.93974.log 2023-10-30T12:24:22.553 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89235.log.gz 2023-10-30T12:24:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90116.log 2023-10-30T12:24:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93974.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92603.log.gz 2023-10-30T12:24:22.553 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.102662.log 2023-10-30T12:24:22.554 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94142.log.gz 2023-10-30T12:24:22.554 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90116.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93974.log.gz 2023-10-30T12:24:22.554 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87346.log 2023-10-30T12:24:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106023.log 2023-10-30T12:24:22.555 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90116.log.gz 2023-10-30T12:24:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87961.log 2023-10-30T12:24:22.555 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106023.log: gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87346.log.gz 2023-10-30T12:24:22.556 INFO:teuthology.orchestra.run.smithi050.stderr: /var/log/ceph/ceph-client.admin.88540.log 2023-10-30T12:24:22.556 INFO:teuthology.orchestra.run.smithi050.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.102662.log.gz 2023-10-30T12:24:22.556 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87961.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106023.log.gz 2023-10-30T12:24:22.556 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105939.log 2023-10-30T12:24:22.557 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93141.log 2023-10-30T12:24:22.557 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87961.log.gz 2023-10-30T12:24:22.557 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107267.log 2023-10-30T12:24:22.557 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93141.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.88540.log.gz 2023-10-30T12:24:22.558 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105939.log.gz 2023-10-30T12:24:22.558 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88666.log 2023-10-30T12:24:22.558 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100515.log 2023-10-30T12:24:22.559 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.93141.log.gz 2023-10-30T12:24:22.559 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95904.log 2023-10-30T12:24:22.559 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107267.log.gz 2023-10-30T12:24:22.559 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97862.log 2023-10-30T12:24:22.560 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90328.log 2023-10-30T12:24:22.560 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95904.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88666.log.gz 2023-10-30T12:24:22.560 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87709.log 2023-10-30T12:24:22.560 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100515.log.gz 2023-10-30T12:24:22.561 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99149.log 2023-10-30T12:24:22.561 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95904.log.gz 2023-10-30T12:24:22.561 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97862.log.gz 2023-10-30T12:24:22.562 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100427.log 2023-10-30T12:24:22.562 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99149.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90328.log.gz 2023-10-30T12:24:22.562 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106149.log 2023-10-30T12:24:22.562 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87709.log.gz 2023-10-30T12:24:22.563 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94663.log 2023-10-30T12:24:22.563 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100427.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99149.log.gz 2023-10-30T12:24:22.563 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99881.log 2023-10-30T12:24:22.563 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97820.log 2023-10-30T12:24:22.564 INFO:teuthology.orchestra.run.smithi050.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.100427.log.gz 2023-10-30T12:24:22.564 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106149.log.gz 2023-10-30T12:24:22.564 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101548.log 2023-10-30T12:24:22.565 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94705.log 2023-10-30T12:24:22.565 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94663.log.gz 2023-10-30T12:24:22.565 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97820.log: /var/log/ceph/ceph-client.admin.101548.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.99881.log.gz 2023-10-30T12:24:22.565 INFO:teuthology.orchestra.run.smithi050.stderr: --verbose -- /var/log/ceph/ceph-client.admin.96030.log 2023-10-30T12:24:22.566 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96962.log 2023-10-30T12:24:22.566 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97820.log.gz 2023-10-30T12:24:22.566 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.94705.log: /var/log/ceph/ceph-client.admin.96030.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101548.log.gz 2023-10-30T12:24:22.567 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107561.log 2023-10-30T12:24:22.567 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96962.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.94705.log.gz 2023-10-30T12:24:22.567 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79507.log 2023-10-30T12:24:22.567 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.96030.log.gz 2023-10-30T12:24:22.568 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98114.log/var/log/ceph/ceph-client.admin.107561.log: 2023-10-30T12:24:22.568 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.tmp-client.admin.79507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101674.log 0.0% 2023-10-30T12:24:22.568 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.79507.log.gz 2023-10-30T12:24:22.568 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96962.log.gz 2023-10-30T12:24:22.569 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101716.log 2023-10-30T12:24:22.569 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107561.log.gz 2023-10-30T12:24:22.569 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98114.log: /var/log/ceph/ceph-client.admin.101674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89365.log 2023-10-30T12:24:22.570 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98114.log.gz 2023-10-30T12:24:22.570 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96408.log 2023-10-30T12:24:22.570 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.101674.log.gz -5 2023-10-30T12:24:22.570 INFO:teuthology.orchestra.run.smithi050.stderr: --verbose -- /var/log/ceph/ceph-client.admin.93225.log 2023-10-30T12:24:22.571 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89365.log: /var/log/ceph/ceph-client.admin.96408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94058.log 2023-10-30T12:24:22.571 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101716.log.gz 2023-10-30T12:24:22.571 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89365.log.gz 2023-10-30T12:24:22.571 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96198.log 2023-10-30T12:24:22.572 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96408.log.gz 2023-10-30T12:24:22.572 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.93225.log: /var/log/ceph/ceph-client.admin.94058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102746.log 2023-10-30T12:24:22.572 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91763.log 2023-10-30T12:24:22.573 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.93225.log.gz 2023-10-30T12:24:22.573 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94058.log.gz 2023-10-30T12:24:22.573 INFO:teuthology.orchestra.run.smithi050.stderr:gzip/var/log/ceph/ceph-client.admin.102746.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.96324.log 2023-10-30T12:24:22.573 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90412.log 2023-10-30T12:24:22.574 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91763.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96198.log.gz 2023-10-30T12:24:22.574 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89151.log 2023-10-30T12:24:22.574 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102746.log.gz 2023-10-30T12:24:22.574 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91679.log 2023-10-30T12:24:22.575 INFO:teuthology.orchestra.run.smithi050.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.96324.log.gz 2023-10-30T12:24:22.575 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89151.log: 90.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83899.log 2023-10-30T12:24:22.575 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91763.log.gz 2023-10-30T12:24:22.575 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100899.log 2023-10-30T12:24:22.576 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91679.log: /var/log/ceph/ceph-client.admin.83899.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.90412.log.gz 2023-10-30T12:24:22.576 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89151.log.gz 2023-10-30T12:24:22.576 INFO:teuthology.orchestra.run.smithi050.stderr: /var/log/ceph/ceph-client.admin.104699.log 2023-10-30T12:24:22.576 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100899.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91679.log.gz 2023-10-30T12:24:22.577 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95692.log 2023-10-30T12:24:22.577 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83899.log.gz 2023-10-30T12:24:22.577 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.104699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98891.log 2023-10-30T12:24:22.577 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100899.log.gz 2023-10-30T12:24:22.578 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104699.log.gz 2023-10-30T12:24:22.578 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88312.log 2023-10-30T12:24:22.578 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95818.log 2023-10-30T12:24:22.579 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98891.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95692.log.gz 2023-10-30T12:24:22.579 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92561.log 2023-10-30T12:24:22.579 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101968.log 2023-10-30T12:24:22.579 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95818.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.98891.log.gz 2023-10-30T12:24:22.580 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92351.log 2023-10-30T12:24:22.580 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88312.log.gz 2023-10-30T12:24:22.580 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92309.log 2023-10-30T12:24:22.580 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95818.log.gz 2023-10-30T12:24:22.581 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92561.log.gz 2023-10-30T12:24:22.581 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91636.log/var/log/ceph/ceph-client.admin.92351.log: 2023-10-30T12:24:22.581 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91023.log 2023-10-30T12:24:22.582 INFO:teuthology.orchestra.run.smithi050.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.101968.log.gz 2023-10-30T12:24:22.582 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92351.log.gz 2023-10-30T12:24:22.582 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99965.log 2023-10-30T12:24:22.582 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5/var/log/ceph/ceph-client.admin.91023.log: --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.92309.log.gz 2023-10-30T12:24:22.583 INFO:teuthology.orchestra.run.smithi050.stderr: -- /var/log/ceph/ceph-client.admin.106763.log 2023-10-30T12:24:22.583 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99965.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91636.log.gz 2023-10-30T12:24:22.583 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105601.log 2023-10-30T12:24:22.584 INFO:teuthology.orchestra.run.smithi050.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.91023.log.gz 2023-10-30T12:24:22.584 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106763.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.99965.log.gz -5 2023-10-30T12:24:22.584 INFO:teuthology.orchestra.run.smithi050.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T12:24:22.584 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88456.log 2023-10-30T12:24:22.585 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106763.log.gz 2023-10-30T12:24:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92800.log 2023-10-30T12:24:22.585 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105601.log.gz 2023-10-30T12:24:22.585 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105164.log 2023-10-30T12:24:22.586 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102012.log 2023-10-30T12:24:22.586 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92800.log: /var/log/ceph/ceph-client.admin.105164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105895.log 2023-10-30T12:24:22.586 INFO:teuthology.orchestra.run.smithi050.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.88456.log.gz 2023-10-30T12:24:22.587 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106065.log 2023-10-30T12:24:22.587 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102012.log: 90.8%/var/log/ceph/ceph-client.admin.105895.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105164.log.gz 2023-10-30T12:24:22.587 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106403.log 2023-10-30T12:24:22.587 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92800.log.gz 2023-10-30T12:24:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89535.log 2023-10-30T12:24:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.106403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105853.log 2023-10-30T12:24:22.588 INFO:teuthology.orchestra.run.smithi050.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.105895.log.gz 2023-10-30T12:24:22.588 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89535.log: 89.5% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.102012.log.gz 2023-10-30T12:24:22.589 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106065.log.gz 2023-10-30T12:24:22.589 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103566.log 2023-10-30T12:24:22.589 INFO:teuthology.orchestra.run.smithi050.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.106403.log.gz 2023-10-30T12:24:22.589 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.89535.log.gz 2023-10-30T12:24:22.590 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.95272.log 2023-10-30T12:24:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105853.log: /var/log/ceph/ceph-client.admin.103566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98555.log 2023-10-30T12:24:22.590 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95272.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103566.log.gz 2023-10-30T12:24:22.591 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105853.log.gz 2023-10-30T12:24:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99359.log 2023-10-30T12:24:22.591 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89764.log 2023-10-30T12:24:22.591 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95272.log.gz 2023-10-30T12:24:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97360.log 2023-10-30T12:24:22.592 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.98555.log.gz 2023-10-30T12:24:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94368.log 2023-10-30T12:24:22.592 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.99359.log.gz 2023-10-30T12:24:22.593 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97360.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:24:22.593 INFO:teuthology.orchestra.run.smithi050.stderr:gzip/var/log/ceph/ceph-client.admin.94368.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.95398.log 2023-10-30T12:24:22.593 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89764.log.gz 2023-10-30T12:24:22.593 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97360.log.gz 2023-10-30T12:24:22.594 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95566.log 2023-10-30T12:24:22.594 INFO:teuthology.orchestra.run.smithi050.stderr:gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94368.log.gz 2023-10-30T12:24:22.594 INFO:teuthology.orchestra.run.smithi050.stderr: -5/var/log/ceph/ceph-client.admin.95398.log: --verbose -- /var/log/ceph/ceph-client.admin.102326.log 2023-10-30T12:24:22.595 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86869.log 2023-10-30T12:24:22.595 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102326.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95398.log.gz 2023-10-30T12:24:22.595 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95566.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92053.log 2023-10-30T12:24:22.595 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.596 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105727.log 2023-10-30T12:24:22.596 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102326.log.gz 2023-10-30T12:24:22.596 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105332.log 2023-10-30T12:24:22.596 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92053.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86869.log.gz 2023-10-30T12:24:22.597 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97257.log 2023-10-30T12:24:22.597 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107393.log 2023-10-30T12:24:22.597 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97257.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105727.log.gz 2023-10-30T12:24:22.597 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.92053.log.gz 2023-10-30T12:24:22.598 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105332.log.gz 2023-10-30T12:24:22.598 INFO:teuthology.orchestra.run.smithi050.stderr: -- /var/log/ceph/ceph-osd.2.log 2023-10-30T12:24:22.598 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107393.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97257.log.gz 2023-10-30T12:24:22.599 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97130.log 2023-10-30T12:24:22.599 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.103270.log 2023-10-30T12:24:22.599 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107393.log.gz 2023-10-30T12:24:22.599 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103060.log 2023-10-30T12:24:22.600 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103270.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100007.log 2023-10-30T12:24:22.600 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.97130.log.gz 2023-10-30T12:24:22.600 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103060.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103270.log.gz 2023-10-30T12:24:22.600 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98511.log 2023-10-30T12:24:22.601 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97946.log 2023-10-30T12:24:22.601 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98511.log: 89.5% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100007.log.gz 2023-10-30T12:24:22.601 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.103060.log.gz 2023-10-30T12:24:22.601 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89619.log 2023-10-30T12:24:22.602 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.98511.log.gz 2023-10-30T12:24:22.602 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105474.log 2023-10-30T12:24:22.602 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94100.log 2023-10-30T12:24:22.603 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97946.log.gz 2023-10-30T12:24:22.603 INFO:teuthology.orchestra.run.smithi050.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.89619.log.gz 2023-10-30T12:24:22.603 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97046.log 2023-10-30T12:24:22.603 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105474.log: /var/log/ceph/ceph-client.admin.94100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90769.log 2023-10-30T12:24:22.604 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97046.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105474.log.gz 2023-10-30T12:24:22.604 INFO:teuthology.orchestra.run.smithi050.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.94100.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99570.log 2023-10-30T12:24:22.604 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.604 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97046.log.gz 2023-10-30T12:24:22.605 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99528.log 2023-10-30T12:24:22.605 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90769.log: /var/log/ceph/ceph-client.admin.99570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96114.log 2023-10-30T12:24:22.605 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99528.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90769.log.gz 2023-10-30T12:24:22.605 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.99570.log.gz 2023-10-30T12:24:22.606 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99528.log.gzgzip 2023-10-30T12:24:22.606 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.100175.log 2023-10-30T12:24:22.606 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5/var/log/ceph/ceph-client.admin.96114.log: --verbose -- /var/log/ceph/ceph-client.admin.98341.log 2023-10-30T12:24:22.607 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96240.log 2023-10-30T12:24:22.607 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.96114.log.gz 2023-10-30T12:24:22.607 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98341.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.100175.log.gz 2023-10-30T12:24:22.607 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103440.log 2023-10-30T12:24:22.608 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88960.log 2023-10-30T12:24:22.608 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103440.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.98341.log.gz 2023-10-30T12:24:22.608 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98765.log 2023-10-30T12:24:22.608 INFO:teuthology.orchestra.run.smithi050.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.96240.log.gz 2023-10-30T12:24:22.609 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96536.log 2023-10-30T12:24:22.609 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.103440.log.gz 2023-10-30T12:24:22.609 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88960.log: /var/log/ceph/ceph-client.admin.98765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102494.log 2023-10-30T12:24:22.610 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96536.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88960.log.gz 2023-10-30T12:24:22.610 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98765.log.gz 2023-10-30T12:24:22.610 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105769.log 2023-10-30T12:24:22.610 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.96536.log.gz 2023-10-30T12:24:22.611 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.101067.log 2023-10-30T12:24:22.611 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102494.log: /var/log/ceph/ceph-client.admin.105769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103692.log 2023-10-30T12:24:22.611 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101067.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.105769.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.102494.log.gz 2023-10-30T12:24:22.611 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.612 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104339.log 2023-10-30T12:24:22.612 INFO:teuthology.orchestra.run.smithi050.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.101067.log.gz 2023-10-30T12:24:22.612 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101632.log 2023-10-30T12:24:22.613 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103692.log: gzip/var/log/ceph/ceph-client.admin.104339.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.88045.log 2023-10-30T12:24:22.613 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101632.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104339.log.gz 2023-10-30T12:24:22.613 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103692.log.gz 2023-10-30T12:24:22.613 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101632.log.gz 2023-10-30T12:24:22.614 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99275.log 2023-10-30T12:24:22.614 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101842.log 2023-10-30T12:24:22.614 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.99275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92519.log 2023-10-30T12:24:22.614 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88045.log.gz 2023-10-30T12:24:22.615 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101842.log: gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99275.log.gz 2023-10-30T12:24:22.615 INFO:teuthology.orchestra.run.smithi050.stderr: -- /var/log/ceph/ceph-client.admin.105248.log 2023-10-30T12:24:22.615 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101926.log 2023-10-30T12:24:22.616 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101842.log.gz 2023-10-30T12:24:22.616 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105248.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.92519.log.gz 2023-10-30T12:24:22.616 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86819.log 2023-10-30T12:24:22.616 INFO:teuthology.orchestra.run.smithi050.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.105248.log.gz/var/log/ceph/ceph-client.admin.101926.log: 2023-10-30T12:24:22.617 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91336.log 2023-10-30T12:24:22.617 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89002.log 2023-10-30T12:24:22.617 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101926.log.gz 2023-10-30T12:24:22.617 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86819.log: /var/log/ceph/ceph-client.admin.91336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102368.log 2023-10-30T12:24:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.89002.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86819.log.gz 2023-10-30T12:24:22.618 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91336.log.gz 2023-10-30T12:24:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95104.log 2023-10-30T12:24:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90370.log 2023-10-30T12:24:22.619 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89002.log.gz 2023-10-30T12:24:22.619 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.102368.log: /var/log/ceph/ceph-client.admin.95104.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102368.log.gz 2023-10-30T12:24:22.619 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87345.log 2023-10-30T12:24:22.619 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95104.log.gz 2023-10-30T12:24:22.620 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100729.log 2023-10-30T12:24:22.620 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90370.log: /var/log/ceph/ceph-client.admin.87345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90496.log 2023-10-30T12:24:22.620 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100729.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90370.log.gz 2023-10-30T12:24:22.620 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87345.log.gz 2023-10-30T12:24:22.621 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100729.log.gz 2023-10-30T12:24:22.621 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107435.log 2023-10-30T12:24:22.621 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97172.log 2023-10-30T12:24:22.622 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.107435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101884.log 2023-10-30T12:24:22.622 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90496.log.gz 2023-10-30T12:24:22.622 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.97172.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.107435.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.95608.log 2023-10-30T12:24:22.622 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101884.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97172.log.gz 2023-10-30T12:24:22.623 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101884.log.gz 2023-10-30T12:24:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98639.log 2023-10-30T12:24:22.623 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.95608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97525.log 2023-10-30T12:24:22.624 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105559.log 2023-10-30T12:24:22.624 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.98639.log: /var/log/ceph/ceph-client.admin.97525.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95608.log.gz 2023-10-30T12:24:22.624 INFO:teuthology.orchestra.run.smithi050.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.98639.log.gz 2023-10-30T12:24:22.624 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96794.log 2023-10-30T12:24:22.625 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.105559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92011.log 2023-10-30T12:24:22.625 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97525.log.gz 2023-10-30T12:24:22.625 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96794.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105559.log.gz 2023-10-30T12:24:22.626 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96794.log.gz 2023-10-30T12:24:22.626 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96752.log 2023-10-30T12:24:22.626 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.92011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104912.log 2023-10-30T12:24:22.626 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.96752.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92011.log.gz 2023-10-30T12:24:22.627 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87625.log 2023-10-30T12:24:22.627 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88087.log 2023-10-30T12:24:22.627 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96752.log.gz 2023-10-30T12:24:22.627 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87625.log: /var/log/ceph/ceph-client.admin.104912.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87625.log.gz 2023-10-30T12:24:22.628 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104912.log.gz 2023-10-30T12:24:22.628 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103961.log 2023-10-30T12:24:22.628 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.88087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90853.log 2023-10-30T12:24:22.629 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101758.log 2023-10-30T12:24:22.629 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90853.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88087.log.gz 2023-10-30T12:24:22.629 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103961.log.gz 2023-10-30T12:24:22.629 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90853.log.gz 2023-10-30T12:24:22.630 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87499.log 2023-10-30T12:24:22.630 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.101758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100686.log 2023-10-30T12:24:22.630 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103608.log 2023-10-30T12:24:22.630 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101758.log.gz 2023-10-30T12:24:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.100686.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87499.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T12:24:22.631 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.103608.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100686.log.gz 2023-10-30T12:24:22.632 INFO:teuthology.orchestra.run.smithi050.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.103608.log.gz 2023-10-30T12:24:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90454.log 2023-10-30T12:24:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87091.log 2023-10-30T12:24:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.90454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91546.log 2023-10-30T12:24:22.633 INFO:teuthology.orchestra.run.smithi050.stderr: 91.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:24:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.87091.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90454.log.gz 2023-10-30T12:24:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.91546.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87091.log.gz 2023-10-30T12:24:22.633 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91546.log.gz 2023-10-30T12:24:22.635 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.88682.log: 94.8% -- replaced with /var/log/ceph/ceph-client.admin.87578.log.gz 2023-10-30T12:24:22.635 INFO:teuthology.orchestra.run.smithi118.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88682.log.gz 2023-10-30T12:24:22.636 INFO:teuthology.orchestra.run.smithi118.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.87282.log.gz 2023-10-30T12:24:22.636 INFO:teuthology.orchestra.run.smithi118.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T12:24:22.670 INFO:teuthology.orchestra.run.smithi118.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.87473.log.gz 2023-10-30T12:24:23.233 INFO:teuthology.orchestra.run.smithi050.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T12:24:23.456 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T12:24:23.497 INFO:teuthology.orchestra.run.smithi118.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.87417.log.gz 2023-10-30T12:24:24.182 INFO:teuthology.orchestra.run.smithi118.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T12:24:27.385 INFO:teuthology.orchestra.run.smithi118.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T12:24:27.475 INFO:teuthology.orchestra.run.smithi050.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T12:24:28.374 INFO:teuthology.orchestra.run.smithi118.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T12:24:28.400 INFO:teuthology.orchestra.run.smithi050.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T12:24:28.430 INFO:teuthology.orchestra.run.smithi050.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T12:24:28.432 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-30T12:24:28.433 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m6.332s 2023-10-30T12:24:28.433 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m19.698s 2023-10-30T12:24:28.433 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m0.898s 2023-10-30T12:24:28.527 INFO:teuthology.orchestra.run.smithi118.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T12:24:28.529 INFO:teuthology.orchestra.run.smithi118.stderr: 2023-10-30T12:24:28.530 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m6.403s 2023-10-30T12:24:28.530 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m21.673s 2023-10-30T12:24:28.530 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.760s 2023-10-30T12:24:28.531 INFO:tasks.ceph:Archiving logs... 2023-10-30T12:24:28.531 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/remote/smithi050/log 2023-10-30T12:24:28.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:24:31.872 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/remote/smithi118/log 2023-10-30T12:24:31.874 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:24:35.220 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T12:24:35.229 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T12:24:35.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:24:35.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:24:35.323 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T12:24:35.334 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T12:24:35.334 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:24:35.338 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:24:35.352 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:24:35.353 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2023-10-30T12:24:35.357 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:24:35.357 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-10-30T12:24:35.358 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +2339us[+2342us] +/- 87ms 2023-10-30T12:24:35.358 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 7 377 53 -1762us[-1759us] +/- 70ms 2023-10-30T12:24:35.358 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +137us[ +137us] +/- 75ms 2023-10-30T12:24:35.358 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:24:35.359 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-30T12:24:35.360 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2023-10-30T12:24:35.360 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +2246us[+2216us] +/- 87ms 2023-10-30T12:24:35.360 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -1848us[-1877us] +/- 70ms 2023-10-30T12:24:35.361 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +54us[ +25us] +/- 75ms 2023-10-30T12:24:35.361 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-30T12:24:35.361 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T12:24:35.373 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T12:24:35.375 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T12:24:35.389 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T12:24:35.514 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-30T12:24:35.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T12:24:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:24:35.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-30T12:24:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:24:35.682 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T12:24:35.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T12:24:35.701 INFO:teuthology.task.internal:Duration was 1951.559521 seconds 2023-10-30T12:24:35.702 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T12:24:35.711 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T12:24:35.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:24:35.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:24:35.763 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T12:24:35.766 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-30T12:24:36.124 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T12:24:36.124 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2023-10-30T12:24:36.125 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:24:36.146 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2023-10-30T12:24:36.146 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:24:36.171 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T12:24:36.172 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:24:36.189 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:24:36.240 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T12:24:36.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:24:36.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:24:36.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T12:24:36.388 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T12:24:36.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:24:36.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:24:36.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T12:24:36.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:24:36.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:24:36.489 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2023-10-30T12:24:36.512 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2023-10-30T12:24:36.535 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:24:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:24:36.573 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:24:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:24:36.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T12:24:36.600 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T12:24:36.601 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/remote/smithi050 2023-10-30T12:24:36.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:24:36.658 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989/remote/smithi118 2023-10-30T12:24:36.658 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:24:36.714 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T12:24:36.715 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:24:36.717 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:24:36.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T12:24:36.757 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T12:24:36.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T12:24:36.767 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T12:24:36.768 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:24:36.773 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:24:36.785 INFO:teuthology.orchestra.run.smithi050.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:24 /home/ubuntu/cephtest 2023-10-30T12:24:36.799 INFO:teuthology.orchestra.run.smithi118.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 30 12:24 /home/ubuntu/cephtest 2023-10-30T12:24:36.799 INFO:teuthology.orchestra.run.smithi118.stdout: 397151 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 30 12:00 /home/ubuntu/cephtest/mnt.0 2023-10-30T12:24:36.800 INFO:teuthology.orchestra.run.smithi118.stdout: 397152 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 30 12:01 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-30T12:24:36.800 INFO:teuthology.orchestra.run.smithi118.stdout: 411057 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:24:36.800 INFO:teuthology.orchestra.run.smithi118.stdout: 411067 20 -rw-r--r-- 1 ubuntu ubuntu 19628 Oct 30 12:22 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_tier_pp.log 2023-10-30T12:24:36.800 INFO:teuthology.orchestra.run.smithi118.stdout: 411071 4 -rw-r--r-- 1 ubuntu ubuntu 2434 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat_pp.log 2023-10-30T12:24:36.801 INFO:teuthology.orchestra.run.smithi118.stdout: 411064 4 -rw-r--r-- 1 ubuntu ubuntu 1841 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock.log 2023-10-30T12:24:36.801 INFO:teuthology.orchestra.run.smithi118.stdout: 411081 4 -rw-r--r-- 1 ubuntu ubuntu 1673 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_read_operations.log 2023-10-30T12:24:36.801 INFO:teuthology.orchestra.run.smithi118.stdout: 411085 8 -rw-r--r-- 1 ubuntu ubuntu 5559 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_list_parallel.log 2023-10-30T12:24:36.801 INFO:teuthology.orchestra.run.smithi118.stdout: 411072 8 -rw-r--r-- 1 ubuntu ubuntu 5917 Oct 30 12:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc_pp.log 2023-10-30T12:24:36.802 INFO:teuthology.orchestra.run.smithi118.stdout: 411058 4 -rw-r--r-- 1 ubuntu ubuntu 4043 Oct 30 12:07 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio.log 2023-10-30T12:24:36.802 INFO:teuthology.orchestra.run.smithi118.stdout: 411065 4 -rw-r--r-- 1 ubuntu ubuntu 2118 Oct 30 12:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc.log 2023-10-30T12:24:36.802 INFO:teuthology.orchestra.run.smithi118.stdout: 411082 4 -rw-r--r-- 1 ubuntu ubuntu 420 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cls_remote_reads.log 2023-10-30T12:24:36.802 INFO:teuthology.orchestra.run.smithi118.stdout: 411076 4 -rw-r--r-- 1 ubuntu ubuntu 589 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd_pp.log 2023-10-30T12:24:36.803 INFO:teuthology.orchestra.run.smithi118.stdout: 411078 24 -rw-r--r-- 1 ubuntu ubuntu 20684 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd.log 2023-10-30T12:24:36.803 INFO:teuthology.orchestra.run.smithi118.stdout: 411073 4 -rw-r--r-- 1 ubuntu ubuntu 2708 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify.log 2023-10-30T12:24:36.803 INFO:teuthology.orchestra.run.smithi118.stdout: 411068 4 -rw-r--r-- 1 ubuntu ubuntu 2065 Oct 30 12:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots.log 2023-10-30T12:24:36.804 INFO:teuthology.orchestra.run.smithi118.stdout: 411062 88 -rw-r--r-- 1 ubuntu ubuntu 88105 Oct 30 12:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_asio.log 2023-10-30T12:24:36.804 INFO:teuthology.orchestra.run.smithi118.stdout: 411070 4 -rw-r--r-- 1 ubuntu ubuntu 1151 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat.log 2023-10-30T12:24:36.804 INFO:teuthology.orchestra.run.smithi118.stdout: 411084 4 -rw-r--r-- 1 ubuntu ubuntu 1295 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_delete_pools_parallel.log 2023-10-30T12:24:36.804 INFO:teuthology.orchestra.run.smithi118.stdout: 411079 4 -rw-r--r-- 1 ubuntu ubuntu 713 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service_pp.log 2023-10-30T12:24:36.805 INFO:teuthology.orchestra.run.smithi118.stdout: 411080 4 -rw-r--r-- 1 ubuntu ubuntu 1196 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_write_operations.log 2023-10-30T12:24:36.805 INFO:teuthology.orchestra.run.smithi118.stdout: 411077 4 -rw-r--r-- 1 ubuntu ubuntu 2666 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service.log 2023-10-30T12:24:36.805 INFO:teuthology.orchestra.run.smithi118.stdout: 411059 8 -rw-r--r-- 1 ubuntu ubuntu 5861 Oct 30 12:09 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio_pp.log 2023-10-30T12:24:36.805 INFO:teuthology.orchestra.run.smithi118.stdout: 411061 4 -rw-r--r-- 1 ubuntu ubuntu 3823 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io_pp.log 2023-10-30T12:24:36.806 INFO:teuthology.orchestra.run.smithi118.stdout: 411083 4 -rw-r--r-- 1 ubuntu ubuntu 1029 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_open_pools_parallel.log 2023-10-30T12:24:36.806 INFO:teuthology.orchestra.run.smithi118.stdout: 411075 8 -rw-r--r-- 1 ubuntu ubuntu 4195 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify_pp.log 2023-10-30T12:24:36.806 INFO:teuthology.orchestra.run.smithi118.stdout: 411066 4 -rw-r--r-- 1 ubuntu ubuntu 1988 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock_pp.log 2023-10-30T12:24:36.806 INFO:teuthology.orchestra.run.smithi118.stdout: 411069 4 -rw-r--r-- 1 ubuntu ubuntu 1243 Oct 30 12:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_pool.log 2023-10-30T12:24:36.807 INFO:teuthology.orchestra.run.smithi118.stdout: 411063 8 -rw-r--r-- 1 ubuntu ubuntu 7015 Oct 30 12:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_list.log 2023-10-30T12:24:36.807 INFO:teuthology.orchestra.run.smithi118.stdout: 411074 4 -rw-r--r-- 1 ubuntu ubuntu 3368 Oct 30 12:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots_pp.log 2023-10-30T12:24:36.807 INFO:teuthology.orchestra.run.smithi118.stdout: 411060 4 -rw-r--r-- 1 ubuntu ubuntu 2522 Oct 30 12:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io.log 2023-10-30T12:24:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:24:36.809 INFO:teuthology.orchestra.run.smithi118.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-30T12:24:36.809 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_bef9b3ee0aeb26a72d4f9af72f05102582ed658c/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-30T12:24:36.817 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T12:24:36.828 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T12:24:36.839 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi118 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-30T12:24:36.839 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T12:24:36.873 DEBUG:teuthology.nuke:shortname: smithi050 2023-10-30T12:24:36.874 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:24:36.895 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:42:17.500390', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/jATN70vGGBL54z9Cu8g6a1rsSO8iHlokG3RH+faOTkoEZuvVRihXNsRF42NBzTBt0Pz6tSWC90Tt7c2Ko0kA='} 2023-10-30T12:24:36.901 DEBUG:teuthology.nuke:shortname: smithi118 2023-10-30T12:24:36.901 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:24:36.923 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2023-10-30_03:55:12-rados:monthrash-wip-vallariag-centos9-only-distro-default-smithi/7440989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 11:42:17.501905', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZKfXPJ1kTzkgnwWEh9xKGaopLAhExAfdfXrJKba4ufD5+QyJGrpSOAwDeMmT0mp2LxAeaX0erLyhdetBDjU1o='} 2023-10-30T12:24:36.943 INFO:teuthology.orchestra.console.smithi050:Power off 2023-10-30T12:24:36.944 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:24:36.967 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2023-10-30T12:24:36.968 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:24:36.984 INFO:teuthology.orchestra.console.smithi118:Power off 2023-10-30T12:24:36.985 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:24:36.994 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-10-30T12:24:37.007 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2023-10-30T12:24:37.008 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:24:37.031 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-30T12:24:40.996 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:24:41.033 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:24:41.107 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-10-30T12:24:41.145 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-30T12:24:45.110 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:24:45.147 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:24:45.220 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2023-10-30T12:24:45.258 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-30T12:24:45.322 INFO:teuthology.orchestra.console.smithi050:Power off completed 2023-10-30T12:24:49.261 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:24:49.371 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2023-10-30T12:24:49.472 INFO:teuthology.orchestra.console.smithi118:Power off completed 2023-10-30T12:24:49.577 INFO:teuthology.run:Summary data: description: rados:monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_api_tests} duration: 1951.5595207214355 failure_reason: 'Unit test failed (workunit test rados/test.sh) on smithi118 with status 1: ''(total 7 failed) FAILURE: Test `LongNamePP` of `LibRadosMiscPP`. Reason: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6851-g003372b1/rpm/el9/BUILD/ceph-18.0....''' flavor: default owner: scheduled_vallariag@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cb081291f18544ecbcfbc902f0f11aca status: fail success: false 2023-10-30T12:24:49.578 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T12:24:49.647 INFO:teuthology.run:FAIL